Thanks for your reply Bogdan,
>>I have been testing the performance of an IWILL H8501 8-way opteron
>>system and have been getting some surprisingly bad results.
>>
>>
>
>Given that a multiprocessor Opteron is a NUMA machine, I would suggest
>playing with BIOS memory settings. In the Tyan BIOS that I have, they
>are called "Bank Interleaving" and "Node Interleaving", with the
>latter being the more important for this case in my opinion. You
>should be able to find details about what they mean with a web search
>engine...
>
>
>
Bank interleaving will only alter local access times and should be on by
default. Node interleaving will make the entire system slow down as a
large percentage of memory access becomes non-local, basically negating
the advantage of NUMA awareness. I think node interleaving might improve
comms, but at an unacceptable cost in overall performance.
>>3. Would a hyper transport specific RPI improve matters or is this
>>more likely to be a capacity/latency issue?
>>
>>
>
>I don't think that a HyperTransport RPI could be written, simply
>because the details of HyperTransport are not available to user
>applications. I think that it is a memory access latency issue and
>this can be changed with the BIOS options. I don't know if even the
>kernel can easily change the memory interleaving settings mentioned
>above.
>
>
Sorry my knowledge of RPIs is very limited. I suppose a more accurate
wording would have been a NUMA aware RPI? Currently when I start a mpi
run, LAM loads the usysv shared memory RPI. As far as I can tell the
shared memory RPIs are by their very nature not optimised for the
asymmetrical nature of NUMA systems, since all comms pass through a
single block of physical memory (at least this is my understanding,
please correct me if I am wrong). With the hypertransport architecture
on the current generation 8-way boards being peer-to-peer, having a
single central comms cache just doesnt make sense to me.
>>I would be glad to perform and post more tests if it would help.
>>
>>
>
>Please post the speedup figures that you obtain with different memory
>interleaving settings.
>
>
I cannot alter the BIOS settings at present, but will post these stats
as soon as I am able.
|
