On Tue, 28 Oct 2003, Pablo Milano wrote:
> Does anybody know if there is a MPI implementation capable of make real
> "progress" of nonblocking communication in the background (i.e,
> overlapping computation with data transfer).
Yes, there are several.
LAM, for example, has two, and has plans to add more.
lamd: The lamd RPI uses LAM's out-of-band communications channel which
funnels all communication through the LAM daemon. While this method is
slow (consider that it adds multiple hops to the message journey), the LAM
daemon is running in a separate process space and therefore makes progress
regardless of what the user program is doing. Some LAM users have found
significant speedup in their applications when using the lamd RPI module.
gm: Myrinet networks have NICs with on-board processors such that when LAM
hands off the message to the hardware (which is almost always
immediately), the NIC and Myrinet switch do all the processing in the
background. This results in very low latency and high bandwidth.
LAM will be adding support for more networks like Myrinet in the
not-distant future. We may also be investigating having TCP progress in
the background (e.g., via threads or other mechanisms), but it's not at
the top of our priority list.
> I tried doing calls to MPI_Test in the loop to help this progress in
> both LAM_MPI and MPICH, buth I got no good results with big ( >
> 25Kbytes) buffers.
LAM defaults to 64k eager send limits -- so if the message is less than
64k, it should be sent more-or-less "right away" (pending OS buffering,
that is). As such, you should actually see your best performance when you
use blocking sends -- this allows LAM to block in the underlying write()
instead of traversing the entire communications stack (MPI_Send all the
way down to the underlying write()) multiple times in a non-blocking
fashion, each time writing out a few more bytes (as OS buffering allows).
I realize that this is LAM-specific behavior, but I believe that other
implementations do it almost exactly the same way (although the exact
boundary size for eager/renedesvouz sending protocols may differ).
> I was told MPI/Pro and ChaMPIon/Pro do this overlap in a better way. Has
> anybody tried this?
The *MPI*/Pro products are fine MPI implementations and do some neat stuff
to get progress in the background. I have not seen performance numbers
for them to know exactly how well they work with their threaded approach
to background progress. You'll have to consult their web pages for
pricing and other information.
--
{+} Jeff Squyres
{+} jsquyres_at_[hidden]
{+} http://www.lam-mpi.org/
|
