On Tue, 4 Nov 2003, Avi Purkayastha wrote:
> I am looking for some pointers for obtaining optimal performance on the
> lam-mpi installation on a single 2p G5 system. The default install
> installed the rpi module sysv. However I am getting very poor latency
> numbers from two mpi tasks on a single node, which leads me to believe
> the message-passing is not using the shared-memory and rather going out
> to the switch for on-node communication.
Can you be a little more specific? What kind of numbers are you getting?
What kind of test are you running? Are you overloading your machine? How
many CPU's does it have? Have you tried the usysv RPI module?
You can use the "-ssi" command line switches to force a different RPI
module. For example:
mpirun -ssi rpi sysv my_mpi_app
or mpirun -ssi rpi tcp my_mpi_app
or mpirun -ssi rpi usysv my_mpi_app
Note that the sysv RPI module uses System V semaphores for coordination
between MPI processes, and the usysv RPI module uses spin locks. Hence,
if you have more processes than processors, you should use the sysv RPI
module (so that the OS can swap out processes when they block on a
semaphore). Otherwise, you should use the usysv RPI module so that your
processes minimize their overhead of locking.
I ran NetPIPE on a dual 2GHz G5 here in our lab for tcp, sysv, and usysv.
I attached the bandwidth and round-trip time graphs. The usysv line in
the time graph is not a mistake -- it's actually 1 us latency (!) all the
way up to about 1k messages. These numbers are with zero tuning
whatsoever -- I took LAM's default compile options (-O3) and with the
default memory pool sizes (hence, the larger message sizes taper off in
performance because of the memory pool restrictions -- see the LAM User
Guide for more details). All I'm trying to show with these graphs is that
out of the box, LAM should give good performance. With tuning, you can
probably get even better performance.
> When I am forcing lamboot to boot up the module sysv, I get the following:
>
> leapfrog<6>lamboot -v -ssi rpi sysv bhost
Note that you don't pick the RPI module during lamboot. Lamboot is only
used to launch the LAM run-time environment -- is technically doesn't even
have anything to do with MPI (think of it in layers -- MPI is layered on
top of LAM). See the LAM User's Guide and the lamssi(7) man page (and
related man pages, e.g., lamssi_rpi(7)) for more details.
Hope that helps.
--
{+} Jeff Squyres
{+} jsquyres_at_[hidden]
{+} http://www.lam-mpi.org/
|
