Hello folks,
I'm trying out lam-7.1b18 on my infiniband test cluster. Generally LAM
runs fine on this machine but the latency for small packages is not
comparable to the other MPI-implementation on the cluster (ScaMPI).
observation1: on tcp LAM and ScaMPI are very close, and both push my Gig-E
network to its limits.
observation2: LAM on ib wins a small victory on bandwidh
observation3: while tcp->ib divides ScaMPI latency by three it only
reduces LAM latency by 20%
Does anyone know if this is expected behaviour for this beta? Here are the
numbers (and yes, the bandwidth is veeery low for ib (chipset issue)):
--- lam, ib ---
[cap_at_n1 mpi]$ /usr/local/lam-7.1.b18-intel/bin/mpirun -np 2
/home/cap/mpi/mpibench.lam71b18ib_intel
Using Zero pattern.
starting lat-bw test.
Latency: 17.5 µsec Bandwidth: 0.0 bytes/s (0 x 10000)
Latency: 17.6 µsec Bandwidth: 57.0 kbytes/s (1 x 10000)
Latency: 17.5 µsec Bandwidth: 114.1 kbytes/s (2 x 10000)
Latency: 17.5 µsec Bandwidth: 228.0 kbytes/s (4 x 10000)
Latency: 17.6 µsec Bandwidth: 455.7 kbytes/s (8 x 10000)
Latency: 17.6 µsec Bandwidth: 909.0 kbytes/s (16 x 10000)
Latency: 17.7 µsec Bandwidth: 1.8 Mbytes/s (32 x 10000)
Latency: 17.8 µsec Bandwidth: 3.6 Mbytes/s (64 x 10000)
Latency: 18.4 µsec Bandwidth: 6.9 Mbytes/s (128 x 10000)
Latency: 19.8 µsec Bandwidth: 13.0 Mbytes/s (256 x 10000)
Latency: 22.4 µsec Bandwidth: 22.8 Mbytes/s (512 x 10000)
Latency: 27.5 µsec Bandwidth: 37.2 Mbytes/s (1024 x 10000)
Latency: 64.3 µsec Bandwidth: 31.8 Mbytes/s (2048 x 10000)
Latency: 75.1 µsec Bandwidth: 54.5 Mbytes/s (4096 x 10000)
Latency: 94.6 µsec Bandwidth: 86.6 Mbytes/s (8192 x 10000)
Latency: 125.5 µsec Bandwidth: 130.5 Mbytes/s (16384 x 6400)
Latency: 212.7 µsec Bandwidth: 154.0 Mbytes/s (32768 x 3200)
Latency: 362.1 µsec Bandwidth: 181.0 Mbytes/s (65536 x 1600)
Latency: 686.4 µsec Bandwidth: 191.0 Mbytes/s (131072 x 800)
Latency: 1.3 msec Bandwidth: 199.9 Mbytes/s (262144 x 400)
Latency: 2.6 msec Bandwidth: 203.2 Mbytes/s (524288 x 200)
Latency: 5.1 msec Bandwidth: 205.3 Mbytes/s (1048576 x 100)
--- lam, tcp ---
[cap_at_n1 mpi]$ /usr/local/lam-7.1.b18-intel/bin/mpirun -ssi rpi tcp -np 2
/home/cap/mpi/mpibench.lam71b18ib_intel
Using Zero pattern.
starting lat-bw test.
Latency: 21.7 µsec Bandwidth: 0.0 bytes/s (0 x 10000)
Latency: 23.5 µsec Bandwidth: 42.6 kbytes/s (1 x 10000)
Latency: 23.5 µsec Bandwidth: 85.3 kbytes/s (2 x 10000)
Latency: 23.5 µsec Bandwidth: 170.0 kbytes/s (4 x 10000)
Latency: 23.5 µsec Bandwidth: 340.4 kbytes/s (8 x 10000)
Latency: 23.9 µsec Bandwidth: 669.8 kbytes/s (16 x 10000)
Latency: 24.3 µsec Bandwidth: 1.3 Mbytes/s (32 x 10000)
Latency: 25.3 µsec Bandwidth: 2.5 Mbytes/s (64 x 10000)
Latency: 27.5 µsec Bandwidth: 4.7 Mbytes/s (128 x 10000)
Latency: 31.5 µsec Bandwidth: 8.1 Mbytes/s (256 x 10000)
Latency: 40.8 µsec Bandwidth: 12.5 Mbytes/s (512 x 10000)
Latency: 55.3 µsec Bandwidth: 18.5 Mbytes/s (1024 x 10000)
Latency: 73.8 µsec Bandwidth: 27.7 Mbytes/s (2048 x 10000)
Latency: 90.6 µsec Bandwidth: 45.2 Mbytes/s (4096 x 10000)
Latency: 126.0 µsec Bandwidth: 65.0 Mbytes/s (8192 x 10000)
Latency: 196.0 µsec Bandwidth: 83.6 Mbytes/s (16384 x 6400)
Latency: 335.1 µsec Bandwidth: 97.8 Mbytes/s (32768 x 3200)
Latency: 616.2 µsec Bandwidth: 106.4 Mbytes/s (65536 x 1600)
Latency: 1.2 msec Bandwidth: 107.0 Mbytes/s (131072 x 800)
Latency: 2.4 msec Bandwidth: 110.2 Mbytes/s (262144 x 400)
Latency: 4.7 msec Bandwidth: 111.3 Mbytes/s (524288 x 200)
Latency: 9.3 msec Bandwidth: 112.3 Mbytes/s (1048576 x 100)
--- scampi, ib ---
[cap_at_n1 mpi]$ /opt/scali/bin/mpimon mpibench.scampi -- n1 n2
starting lat-bw test.
Latency: 6.8 µsec Bandwidth: 0.0 bytes/s (0 x 10000)
Latency: 6.8 µsec Bandwidth: 146.1 kbytes/s (1 x 10000)
Latency: 6.8 µsec Bandwidth: 292.3 kbytes/s (2 x 10000)
Latency: 6.8 µsec Bandwidth: 587.1 kbytes/s (4 x 10000)
Latency: 7.0 µsec Bandwidth: 1.2 Mbytes/s (8 x 10000)
Latency: 7.1 µsec Bandwidth: 2.3 Mbytes/s (16 x 10000)
Latency: 7.3 µsec Bandwidth: 4.4 Mbytes/s (32 x 10000)
Latency: 7.7 µsec Bandwidth: 8.3 Mbytes/s (64 x 10000)
Latency: 8.5 µsec Bandwidth: 15.0 Mbytes/s (128 x 10000)
Latency: 10.3 µsec Bandwidth: 24.9 Mbytes/s (256 x 10000)
Latency: 13.9 µsec Bandwidth: 36.7 Mbytes/s (512 x 10000)
Latency: 19.6 µsec Bandwidth: 52.3 Mbytes/s (1024 x 10000)
Latency: 27.8 µsec Bandwidth: 73.7 Mbytes/s (2048 x 10000)
Latency: 39.3 µsec Bandwidth: 104.2 Mbytes/s (4096 x 10000)
Latency: 64.0 µsec Bandwidth: 127.9 Mbytes/s (8192 x 10000)
Latency: 110.1 µsec Bandwidth: 148.8 Mbytes/s (16384 x 6400)
Latency: 202.4 µsec Bandwidth: 161.9 Mbytes/s (32768 x 3200)
Latency: 378.4 µsec Bandwidth: 173.2 Mbytes/s (65536 x 1600)
Latency: 720.8 µsec Bandwidth: 181.8 Mbytes/s (131072 x 800)
Latency: 1.4 msec Bandwidth: 186.4 Mbytes/s (262144 x 400)
Latency: 2.8 msec Bandwidth: 188.8 Mbytes/s (524288 x 200)
Latency: 5.5 msec Bandwidth: 189.9 Mbytes/s (1048576 x 100)
--- scampi, tcp ---
[cap_at_n1 mpi]$ /opt/scali/bin/mpimon -networks tcp mpibench.scampi -- n1 n2
starting lat-bw test.
Latency: 21.7 µsec Bandwidth: 0.0 bytes/s (0 x 10000)
Latency: 22.3 µsec Bandwidth: 44.8 kbytes/s (1 x 10000)
Latency: 22.3 µsec Bandwidth: 89.8 kbytes/s (2 x 10000)
Latency: 22.4 µsec Bandwidth: 178.5 kbytes/s (4 x 10000)
Latency: 22.4 µsec Bandwidth: 356.7 kbytes/s (8 x 10000)
Latency: 22.6 µsec Bandwidth: 707.8 kbytes/s (16 x 10000)
Latency: 23.1 µsec Bandwidth: 1.4 Mbytes/s (32 x 10000)
Latency: 24.1 µsec Bandwidth: 2.7 Mbytes/s (64 x 10000)
Latency: 26.2 µsec Bandwidth: 4.9 Mbytes/s (128 x 10000)
Latency: 30.4 µsec Bandwidth: 8.4 Mbytes/s (256 x 10000)
Latency: 41.0 µsec Bandwidth: 12.5 Mbytes/s (512 x 10000)
Latency: 55.0 µsec Bandwidth: 18.6 Mbytes/s (1024 x 10000)
Latency: 75.2 µsec Bandwidth: 27.2 Mbytes/s (2048 x 10000)
Latency: 89.9 µsec Bandwidth: 45.5 Mbytes/s (4096 x 10000)
Latency: 126.5 µsec Bandwidth: 64.8 Mbytes/s (8192 x 10000)
Latency: 196.1 µsec Bandwidth: 83.5 Mbytes/s (16384 x 6400)
Latency: 335.4 µsec Bandwidth: 97.7 Mbytes/s (32768 x 3200)
Latency: 616.3 µsec Bandwidth: 106.3 Mbytes/s (65536 x 1600)
Latency: 1.2 msec Bandwidth: 111.1 Mbytes/s (131072 x 800)
Latency: 2.3 msec Bandwidth: 113.3 Mbytes/s (262144 x 400)
Latency: 4.6 msec Bandwidth: 113.2 Mbytes/s (524288 x 200)
Latency: 9.3 msec Bandwidth: 113.1 Mbytes/s (1048576 x 100)
--
------------------------------------------------------------
Peter Kjellstroem | E-mail: cap_at_[hidden]
National Supercomputer Centre | Office: +46(0)13 281492
Linkoeping University | Fax : +46(0)13 282535
SE-581 83 Linkoeping |
Sweden | http://www.nsc.liu.se
|
