On Tue, 19 Apr 2005, Eugene de Villiers wrote:
>> I don't think that a HyperTransport RPI could be written, simply because
>> the details of HyperTransport are not available to user applications. I
>> think that it is a memory access latency issue and this can be changed with
>> the BIOS options. I don't know if even the kernel can easily change the
>> memory interleaving settings mentioned above.
>>
> Sorry my knowledge of RPIs is very limited. I suppose a more accurate wording
> would have been a NUMA aware RPI? Currently when I start a mpi run, LAM loads
> the usysv shared memory RPI. As far as I can tell the shared memory RPIs are
> by their very nature not optimised for the asymmetrical nature of NUMA
> systems, since all comms pass through a single block of physical memory (at
> least this is my understanding, please correct me if I am wrong). With the
> hypertransport architecture on the current generation 8-way boards being
> peer-to-peer, having a single central comms cache just doesnt make sense to
> me.
>
Shared memory does use a single block of memory space (whatever the
interleaving) to share the messages. Of course, moving messages across
the HyperTransport bus in-to and out-of a common memory block is far
faster than moving them down the HT bus to memory, then down the HT bus to
the PCI bus, out the PCI to Infiniband, out over the Infiniband, and then
back the whole way on another node. Even if possible, a NUMA aware RPI
with multilple (physical) shared memory areas just wouldn't improve
performance.
It would certainly be nice to somehow guarauntee that each task ran with
its memory locally allocated, but this would require some sort of user
space interface in the NUMA kernel code, as I believe there is in IRIX.
------------------------------------------------------------
Anthony Ciani (aciani1_at_[hidden])
Computational Condensed Matter Physics
Department of Physics, University of Illinois, Chicago
http://ciani.phy.uic.edu/~tony
------------------------------------------------------------
|
