We have only stress tested on Mellanox thca-linux-3.2-build-024; I
don't know enough about their version numbering to know how linux-3.2
compares to darwin-1.05.
What *could* be happening (although I would be surprised) is that LAM
is attempting to register too much IB-addressable memory, and the
system is simply running out (and therefore failing). LAM has a
definite scalability problem with its IB stack in that each MPI process
tries to allocate IB-addressable memory for every other MPI peer
process. From the LAM/MPI User Guide (pardon the LaTeX markup):
-----
Note that the 7.1.x versions of the \rpi{ib} \kind{rpi} will not scale
well to large numbers of nodes because they register a fixed number of
buffers ($M$ bytes) for each process peer during
\mpifunc{MPI\_\-INIT}. Hence, for an $N$-process
\mpiconst{MPI\_\-COMM\_\-WORLD}, the total memory registered by each
process during \mpifunc{MPI\_\-INIT} is $(N - 1) \times M$ bytes.
This can be prohibitive as $N$ grows large.
This effect can be limited, however, by decreasing the number and size
of buffers that the \rpi{ib} \kind{rpi} module via SSI parameters at
run-time. See the Section~\ref{sec:mpi-ssi-ib}
(page~\pageref{sec:mpi-ssi-ib}) for more details.
-----
You can easily check if this is the issue by decreasing the number and
size of pre-allocated buffers by decreasing the default values of 2 SSI
parameters:
ssi_rpi_ib_num_envelopes default value: 64
ssi_rpi_ib_tinymsglen default value: 32768
Try decreasing these values and let us know what you find.
On Nov 27, 2004, at 5:36 PM, Eugene Ingerman wrote:
> Hello,
>
> I need help figuring out the problem I have with LAM 7.1.1. We just
> got a 16
> node dual-G5 cluster with mellanox infiniband (build_id
> thca-darwin-1.05-build-001).
>
> Even when I try to run examples/cpi program I get the following error:
>
> xserve100:~/lampi/bin eu$ mpirun C -ssi rpi ib /nfs/home/eu/cpi
>
> VIP(1): ERROR: VIPKL_create_mr failed !
> -----------------------------------------------------------------------
> ------
> It seems registration of memory for Infiniband operation failed. We
> can not register any more memory. Aborting ..
> -----------------------------------------------------------------------
> ------
> rank 24 REGISTER FAILED!
> Process 5 of 32 on xserve103
>
> Thanks,
> Eugene
>
>
>
> -----------------------------------------------------------------------
> -------------
> xserve100:~/lampi/bin eu$ ./laminfo
> LAM/MPI: 7.1.1
> Prefix: /nfs/home/eu/lampi
> Architecture: powerpc-apple-darwin7.3.1
> Configured by: eu
> Configured on: Wed Nov 24 20:07:50 PST 2004
> Configure host: xserve100
> Memory manager: darwin7malloc
> C bindings: yes
> C++ bindings: yes
> Fortran bindings: no
> C compiler: gcc
> C++ compiler: g++
> Fortran compiler: false
> Fortran symbols: none
> C profiling: yes
> C++ profiling: yes
> Fortran profiling: no
> C++ exceptions: no
> Thread support: yes
> ROMIO support: yes
> IMPI support: no
> Debug support: no
> Purify clean: no
> SSI boot: globus (API v1.1, Module v0.6)
> SSI boot: rsh (API v1.1, Module v1.1)
> SSI boot: slurm (API v1.1, Module v1.0)
> SSI coll: lam_basic (API v1.1, Module v7.1)
> SSI coll: shmem (API v1.1, Module v1.0)
> SSI coll: smp (API v1.1, Module v1.2)
> SSI rpi: crtcp (API v1.1, Module v1.1)
> SSI rpi: ib (API v1.1, Module v1.0)
> SSI rpi: lamd (API v1.0, Module v7.1)
> SSI rpi: sysv (API v1.0, Module v7.1)
> SSI rpi: tcp (API v1.0, Module v7.1)
> SSI rpi: usysv (API v1.0, Module v7.1)
> SSI cr: self (API v1.0, Module v1.0)
>
>
> _______________________________________________
> This list is archived at http://www.lam-mpi.org/MailArchives/lam/
>
--
{+} Jeff Squyres
{+} jsquyres_at_[hidden]
{+} http://www.lam-mpi.org/
|
