LAM/MPI logo

LAM 6.5.9 Release Notes

  |   Home   |   Download   |   Documentation   |   FAQ   |   all just 6.5

This file contains the release notes for LAM/MPI version 6.5.9. Here's a brief table of contents (### indicates updates for this version):

  1. Installation instructions
  2. ### Supported systems
  3. System Notes
  4. Caveats
  5. Backward compatibility
  6. LAM and Linux
  7. ### New feature overview
  8. ### PBS/LSF support
  9. ### LAM help file
  10. Zeroing out LAM buffers before use
  11. ### Trillium executables / header files / man pages
  12. ### Shared library support
  13. ### mpirun behavior

  14. ### lamexec
  15. hcc / hcp / hf77 / mpicc / mpiCC / mpif77
  16. ### lamboot / recon / wipe / lamhalt
  17. ### lamnodes
  18. Root execution disallowed
  19. RPI transport layers

  20. IMPI extensions
  21. Signal catching
  22. Internal signal
  23. New basic datatypes
  24. MPI-2 Support

Installation instructions

Please see the installation guide for instructions on installing LAM 6.5.9 in the INSTALL file.

Supported systems

LAM 6.5.9 has been tested on the following systems:

  • AIX 5.1
  • IRIX 6.5
  • Linux 2.4.18
  • Mac OS X 10.2.1
  • OpenBSD 3.1
  • Solaris 8

Even if your operating system is not listed here, LAM will likely compile/run properly. The source code is fairly portable -- it should work properly under most POSIX-like systems.

We request that you also download the "lamtest" package from the LAM web site and run it on your system. If any errors are encountered, please send us all the output (see the README file in that package).

System Notes

  • Mac OS X
    • HFS+ is case-insensitive - mpicc and mpiCC will both refer to the same file. This obviously makes calling the wrapper compilers mpicc and mpiCC impossible. LAM will attempt to determine if you are building on an HFS+ partition. If you are, the C++ compiler will be called mpic++. Otherwise, the C++ compiler will be called mpiCC (although mpic++ can also be used).

      To control this behavior, you can use the configure arguments:

            --with-cs-fs    (to force creation of mpiCC)
            --without-cf-fs (to not create mpiCC / other case conflicting files)
    • The Apple developers CD does not include a Fortan compiler. If you do not wish to use Fortran, this isn't a problem - but you need to tell LAM not to build the Fortran bindings. Simply add the argument

      to configure and LAM will skip the Fortran bindings.
  • AIX
    • If you are using the GNU Compiler suite and the AIX text utilities (grep, sed, etc.), configure will fail at the end with an error message about too long of a line. The build will succeed, but dependency tracking will be broken. Configuring with


      will disable dependency tracking (which you only really need if you intend to work on the LAM source code itself).

  • IRIX
    • The C++ examples and C++ tests in the test suite will fail unless CXXFLAGS is set to -LANG:std when using the SGI compilers.


LAM is fully MPI-1 complaint with the exception of MPI_CANCEL. MPI_CANCEL works properly for receives, but will almost never work on sends. MPI_CANCEL is most frequently used with unmatched MPI_IRECV's that were made "in case" a matching message arrived. This simply entails removing the receive request from the local queue, and is fairly straightforward to implement.

Actually canceling a send operation is much more difficult because some meta information about a message is usually sent immediately. As such, the message is usually at least partially sent before an MPI_CANCEL is issued. Trying to chase down all the particular cases is a nightmare, to say the least.

As such, the LAM Team decided not to implement MPI_CANCEL on sends, and instead concentrate on other features.

But in true MPI Forum tradition, we would be happy to discuss any code that someone would like to submit that fully implements MPI_CANCEL. :-)

Backward compatibility

LAM provides full source code backward compatibility with previous versions of LAM. Old applications that compiled properly with older versions of LAM can simply be recompiled with this version of LAM.

Binary compatibility, however, is not provided -- applications that have been compiled with previous versions of LAM will need to be recompiled in order to run properly with this version of LAM. If applications are not re-compiled with this LAM, their behavior will be unpredictable (read: they just won't work).

Specifically -- users need to ensure that they use the new versions of all the LAM commands in order to run properly. For example, mixing LAM 6.3.2 and 6.5.9 commands (e.g., running the "wrong" version of lamd) will result in unpredictable behavior.

LAM and Linux

LAM is frequently used on Linux-based machines (iX86 and otherwise). It works correctly under 2.0.36 (we didn't test under 2.0.37, but we have no reason to believe that it wouldn't work under that version as well, since it is really only minor changes from 2.0.36) and 2.2.x. We haven't explicitly tested under 2.4, but we don't expect any problems with it.

However, versions 2.2.0 through 2.2.9 had some TCP/IP performance problems. It seems that version 2.2.10 fixed these problems; if you are using a Linux version between 2.2.0 and 2.2.9, LAM may exhibit poor C2C performance due to the Linux TCP/IP kernel bugs. We recommend that you upgrade to 2.2.10 (or the latest version).

See for a full discussion of the problem.

New feature overview

LAM 6.5

  • Added man pages for all MPI-1 and MPI-2 functions.
  • Added some primitive syslog messages to the lamd (when launched via "lamboot -d ...") to help the troubleshooting process when the lamd seems to be involved.
  • lamboot, recon, and [deprecated] wipe are noticably faster when the "-b" switch is used. "-b" can be used in situations where the user's shell is the on the remote nodes as on the local node.
  • Allow a single user to have multiple LAM instances on a single node; this is useful for batch kinds of systems. LAM does this automatically when runnning LAM in a PBS or LSF job, but users can also do this manually.
  • New SMP-aware "mpirun" command line syntax and boot schema syntax for "lamboot".
  • "-pty", pseudo tty support, is now enabled by default with mpirun. Use the new option "-npty" to disable pseudo tty support.
  • MPI_UNIVERSE_SIZE now reports total number of CPUs, not nodes.
  • Slightly changed the internal and install directory structures; now fully using automake and libtool for building LAM.
  • Full support for VPATH builds.
  • Shared library support (only if not using ROMIO or the C++ bindings).
  • Added MPI_Info key to MPI_COMM_SPAWN to specify a schema file for MPMD spawns.
  • Added "lamhalt" command to quickly shut down LAM
  • Added "lamnodes" command to retrieve hostnames from nX and cX nomenclature.
  • Added MPI_ALLOC_MEM and MPI_FREE_MEM, and 2 optional RPI calls to support them, mainly in anticipation of Myrinet and VIA RPIs.
  • Added "-s" option to lamboot to allow "rsh somenode lamboot -s hostfile" to complete normally.
  • Expanded many error messages to be more descriptive and generally user-friendly.
  • Updated MPI 2 C++ bindings distribution.
  • Updated and patched ROMIO distribution. Building ROMIO is now the default.
  • Building the profiling layer is now the default.
  • Supplemental Trillium executables and .h files that are not necessary for MPI operation are no longer built/installed by default.
  • Made the lamtests package easier to port to other implementations of MPI.
  • Oodles of bug fixes.

LAM 6.3 series

  • Fixed critical bug in sysv and usysv RPIs
  • Added ROMIO MPI I/O package (chapter 9 from the MPI-2 standard) from the Argonne National Laboratory
  • Added the MPI-2 C++ bindings package (chapter 10 from the MPI-2 standard) from the Open Systems Laboratory at Indiana University
  • Pseudo-tty support for remote IO (e.g., line buffered output).
  • Ability to pass environment variables through mpirun.
  • Ability to mpirun shell scripts/debuggers/etc. (that eventually run LAM/MPI programs).
  • New default present working directory behavior for mpirun
  • Ability to execute non-MPI programs across the multicomputer.
  • Added configurable ability to zero-fill internal LAM buffers before they are used (for development tools such as Purify).
  • Greatly expanded error messages; provided for customizable local help files.
  • Expanded and updated documentation.
  • Wrapper compilers now use same compilers as in ./configure.
  • Made -lmpi implicit in wrapper compilers.
  • Fixed compatibility problems with some compilers.
  • Fixed problem with some compilers not getting prototypes properly from .
  • Various bug fixes (see the HISTORY file).

LAM 6.2 series

  • GNU ./configure script for installing LAM.
  • Full standard conforming support for MPI-2 dynamic processes.
  • Support for MPI-2 one-sided communication.
  • Many MPI-2 support functions implemented.
  • More basic datatypes supported (including MPI-2 datatypes such as MPI_TYPE_DARRAY).
  • Direct TCP and shared-memory message passing transports.
  • Optimized the common send and receive cases.
  • All command line arguments are transparently passed to the user program when main() is invoked (no additional command line arguments are introduced by LAM).
  • New user level locking shared memory RPI transport.
  • Signal catching to aid the termination of batch jobs.
  • Support for shared libraries on Linux and Solaris.
  • Various small bug fixes.

PBS / LSF support

LAM is now "somewhat aware" of the Portable Batch System (PBS) and Load Sharing Facility (LSF) batch queue schedulers. That is, the following problems have been solved:

  • The "multiple LAM" problem has been fixed. When running under PBS or LSF, LAM will take precautions to isolate itself from other instances of LAM in concurrent PBS/LSF jobs. That is, the multiple LAM instances from the same user can exist on the same machine when executing under PBS/LSF. This allows a user to submit as many LAM jobs as necessary, and not have any fear that jobs will be killed.

    This "multiple LAM" behavior is *only* exhibited under a batch environment (currently, only PBS and LSF are supported). Other batch systems can easily be supported -- let the LAM Team know if you'd like to see support for others included. Additionally, manually setting the environment variable LAM_MPI_SOCKET_SUFFIX on the node where lamboot is run achieves the same ends.

  • PBS will generate a $PBS_NODEFILE with the same hostname multiple times when multiple vnodes from the same node are allocated by the scheduler. LAM will now correctly parse this file, and take it to mean that the node has multiple CPUs available. See the section on lamboot and mpirun, below.

One major feature that is still lacking with respect to PBS is to use PBS's implementation of the Task Management (TM) extensions. Once LAM uses the TM extensions:

  • "epilog" scripts will no longer be necessary to guarantee the cleanup of LAM jobs
  • accounting information will accurately reflect all usage (including on nodes other than where the mother superior resides)
  • the world will be a better place

LAM help file

The following LAM binaries have had their help messages greatly expanded:

  • mpicc / mpiCC / mpif77
  • lamboot
  • lamhalt
  • recon

The messages should be much more helpful in trying to diagnose problems, especially for first-time users. The help messages generally try to identify the problem and suggest solutions. It is possible for multiple cascading error messages to be printed; one failure may cause other failures. As such, the first error message is generally (but not always) the most relevant message -- solving that error may solve the rest.

Additionally, much more information is now output when the "-d" switch is used on all of these program (which enables debugging output).

The help messages are all contained in a single ASCII file which is initially installed into the following file (where $prefix is the option supplied to --prefix in the ./configure script):


The format of the file is simple; simple delimiter lines separate help topic messages. It should be quite obvious which message corresponds to which program/topic name combination.

This file allows system administrators to customize help messages for their users according to the local environment. When LAM tries to find the helpfile to print out a help message, it actually searches for the file in the following order:

  • $HOME/lam-helpfile
  • $HOME/lam-6.5.9-helpfile
  • $HOME/etc/lam-helpfile
  • $HOME/etc/lam-6.5.9-helpfile
  • $LAMHELPDIR/lam-helpfile
  • $LAMHELPDIR/lam-6.5.9-helpfile
  • $LAMHOME/etc/lam-helpfile
  • $LAMHOME/etc/lam-6.5.9-helpfile
  • $TROLLIUSHOME/etc/lam-helpfile
  • $TROLLIUSHOME/etc/lam-6.5.9-helpfile
  • $prefix/etc/lam-helpfile
  • $prefix/etc/lam-6.5.9-helpfile

This seemingly-over complicated scheme will allow for maximum flexibility for system administrators and/or users to define the location of customized help files.

Zeroing out LAM buffers before use

LAM has several structures that are used in many different situations. One example is LAM's internal "struct nmsg" -- one of the underlying message constructs used to pass data between LAM entities. But since the "struct nmsg" is used in so many places, it is a generalized structure and contains fields that are not used in every situation.

By default, LAM only initializes relevant struct members before using a structure. Using a structure may involve sending the entire structure (including uninitialized members) to a remote host. This is not a problem, because the remote host will also ignore the irrelevant struct members (depending on the specific function being invoked). More to the point -- LAM was designed this way to avoid setting variables that will not be used; this is a slight optimization in run-time performance.

Memory-checking debuggers are quite popular (such as purify and the Solaris Workshop/Forte bcheck program), and quite useful to find memory leaks, indexing past the end of arrays, and other types of Heisenbugs. Since LAM "uses" uninitialized memory, it tends to generate many warnings with these types of debuggers.

The --with-purify option has been added to the ./configure script that will force LAM to zero out *all* memory before it is used. This will eliminate the "read from uninitialized" types of warnings that memory-checking debuggers will identify deep inside LAM. However, this option invokes a slight overhead penalty in the run-time performance of LAM, so it is not the default.

Trillium executables / header files / man pages

In prior versions of LAM/MPI, several additional Trillium executables were built during the default build (Trillium is the built-in lower layer of the LAM/MPI run-time system). Additionally, several Trillium-specific header files and man pages were installed by default.

These executables, header files, and man pages are actually intended for LAM/Trillium developers, and are not necessary for normal MPI use. That is -- these executables, header files, and man pages are not necessary to develop, compile, and run normal MPI programs under LAM/MPI. Third party packages may require the Trillium header files and/or executables, however. For example, compiling XMPI will require that the Trillium header files were previously installed.

As such, these executables have been removed from the default build. Correspondingly, the header files and man pages have been removed from the default install. There is an option to configure to enable both the Trillium executables to be built as well as the header files and man pages to be installed. See the INSTALL file for details.

Shared library support

Since LAM/MPI is now build with automake/libtool, the two libraries necessary for MPI programs (liblam and libmpi) can be built as shared or static libraries. See the INSTALL file for details.

However, the two third party packages that LAM/MPI includes -- the MPI 2 C++ bindings and ROMIO -- do *not* support shared library builds. If you select to build either of these packages (both are enabled by default; see the INSTALL file for how to disable them), building LAM/MPI with shared library support is disallowed.

Additionally, GNU libtool also does not appear to build shared libraries correctly on all systems. For example, libtool fails to build the LAM shared libraries properly under AIX 4.3.3. Hence, LAM can only be built statically on such platforms. This is a limitation of libtool, not LAM. Since LAM uses libtool to build itself, libtool limitations can constrain the build options for LAM/MPI. libtool cannot also reliably build C++ libraries (it *usually* works for static linking of C++ libraries that have no templates, but shared libraries typically don't work with non-GNU compilers).

It is hoped that future versions of the GNU libtool package will allow buildings C++ libraries and correct building of shared libraries on all platforms. Write to your congressman/local government representative! ;-)

mpirun behavior

Several new features have been added to mpirun.

SMP-aware command line syntax

New "C" syntax has been added to mpirun (note that this section also applies to the "lamexec" command). When running on SMP machines, it is frequently desirable to group as many adjoining ranks as possible on a single node in order to maximize shared memory message passing. When used in conjunction with the extended bootschema syntax (that allows the specification of number of CPUs available on each host), the mpirun "C" syntax will run one executable on each available CPU, and will group adjoining MPI_COMM_WORLD ranks on the same nodes. For example, when running on two SMPs, the first having four CPUs, and the second having two CPUs, the following command:

      % mpirun C foo

will run four copies of "foo" on the four-way SMP (MPI_COMM_WORLD ranks 0 through 3), and will run two copies of "foo" on the two-way SMP (MPI_COMM_WORLD ranks 4 and 5).

Just like the "N" syntax in mpirun, the "C" syntax can also be used to indicate specific CPUs. For example:

      % mpirun c4,5 foo

runs "foo" on the fourth and fifth CPUs (i.e., the two-way SMP from the previous examples). "C" and "cX" syntax can also be combined:

      % mpirun c0 C master-slave

could be used to launch a "master" process (i.e., rank 0 in MPI_COMM_WORLD) on CPU zero, a slave on each CPU (including rank 0. This may be desirable, for example, in situations where the master rank does very little computation).

The behavior of "-np" has been altered to match the "C" semantics. "-np" now schedules across CPUs, not nodes. Using "-np 6" in the previous example would be the same as "C"; using "-np 4" would run one four copies of "foo" on the four-way SMP.

Also note that "N", "nX", C", and "cX" syntax can all be used simultaneously, although it is not clear that this is really useful.

Ability to pass environment variables

All environment variables named LAM_MPI_* are now automatically passed to remote notes (unless disabled via the "-nx" option to mpirun). The "-x" option enabled exporting of specific environment variables to the remote nodes:

     % LAM_MPI_FOO="green eggs and ham"
     % export LAM_MPI_FOO
     % mpirun C -x DISPLAY,SEUSS=author samIam

This will launch the "samIam" application on all available CPUs. The LAM_MPI_FOO, DISPLAY, and SEUSS variables will be created on all nodes before the user's program is invoked.

Note that the parser for the "-x" option is currently not very sophisticated -- it cannot even handle quoted values when defining new environment variables. Users are advised to set variables in the environment prior to invoking mpirun, and only use "-x" to export the variables to the remote nodes (not to define new variables), if possible.

Pseudo-tty support

The "-pty" option to mpirun enabled pseudo tty support. Among other things, this gives line-buffered output from the remote nodes (which is probably what you want). "-pty" is now the default mode for mpirun. Use the "-npty" option to disable pseudo tty support.

Current working directory behavior

The default behavior of mpirun has changed with respect to the directory that processes will be started in.

The "-wd" option to mpirun allows the user to change to an arbitrary directory before their program is invoked. It can also be used in application schema files to specify working directories on specific nodes and/or for specific applications.

If the "-wd" option appears both in a schema file and on the command line, the schema file directory will override the command line value. "-wd" is mutually exclusive with "-D".

If neither "-wd" nor "-D" are specified, the local node will send the directory name where mpirun was invoked from to each of the remote nodes. The remote nodes will then try to change to that directory. If they fail (e.g., if the directory does not exist on that node), they will start from the user's home directory.

All directory changing occurs before the user's program is invoked; it does not wait until MPI_INIT is called.

Ability to run shell scripts/debuggers/etc.

mpirun can now also run non-LAM/MPI programs. That is, one can mpirun a shell script, debugger, or any other program that will eventually either exec a LAM/MPI program or spawn a LAM/MPI program as a child.

This is extremely helpful for batch systems and debugging environments. For example:

     % mpirun C gdb

However, one must be careful when running programs on remote nodes that expect the use of stdin. Recall that stdin on remote nodes is redirected to /dev/null. It is probably more advantageous to export the DISPLAY environment variable, and run a shell script that invokes an xterm with "gdb" running in it on each node. For example:

     % mpirun C -x DISPLAY xterm-gdb.csh

The default behavior of mpirun has changed

The default options now correspond to "-w -c2c -nger". That is:

  • wait for the application to terminate
  • use the fast client-to-client communication mode
  • disable GER.

To get the old behavior, use the options "-lamd -ger -nw".

mpirun now recognizes command lines of the form

     % mpirun -np  {LAM specific mpirun args}  {program args}

For example,

     % mpirun -np 4 -lamd n0 n1 /bin/foobar 12 a b c

runs 4 copies of the /bin/foobar program on nodes n0 and n1 (2 copies on each node), passing the arguments, 12 a b c, to the program. The new syntax is equivalent to the following in the "-c" syntax which is still supported.

     % mpirun -c  {LAM specific mpirun args}  -- {program args}


The lamexec command has been added to LAM/MPI's reporitore. It is an "mpirun clone", but is specifically for running non-MPI programs. That is, one can do the following:

     % lamexec N hostname

which will run "hostname" on all nodes in the multicomputer. It can take most of the same command line arguments as mpirun; it does not support the flags that do not make sense for non-MPI programs (e.g., -c2c, -lamd, etc.). See lamexec(1) for more details.

Note that the "C" syntax for mpirun also applies to lamexec.

hcc / hcp / hf77 / mpicc / mpiCC / mpif77

The hcc, hcp, and hf77 wrapper compilers have previously not automatically passed the "-lmpi" option to the underlying compiler. The rationale behind this decision was that the "mpicc" and "mpif77" wrapper compilers added this functionality; the "h" wrappers were intended as Trollius compilers, not LAM/MPI compilers.

But hcc, hcp, and hf77 have become the de facto wrapper compilers (vs. mpicc and mpif77). Indeed, some users have been confused about why -lmpi is not implicit to the "h" wrapper compilers.

Hence, "-lmpi" is now automatically passed to the underlying compiler in the hcc, hcp, and hf77 wrapper compilers. The mpicc and mpif77 compilers are now symbolic links to hcc and hf77, respectively.

For symmetry, mpiCC has been created as a symbolic link to hcp.

lamboot / recon / wipe / lamhalt

The help messages in the lamboot, lamhalt, and wipe commands have been greatly enhanced. It is hoped that this will be very useful to new users of LAM.

  • lamboot, recon, and wipe are now noticably faster when used with the "-b" switch. Without "-b", these commands each use two remote shell invocations (e.g., rsh or ssh) to each node. The first remote shell invocation determines if the user's default shell was the Bourne/Korn shell or not (in which case $HOME/.profile must be invoked manually). The second remote shell invocation actually runs the desired command.

    Since many users do not use the Bourne or Korn shells, an alternative algorithm that does not invoke the extra remote shell can be utilized with the "-b" switch, resulting in a nice speedup. Bourne and Korn shell users cannot use the "-b" switch. It should be noted that Bash, even though it is a Bourne shell derivative, *can* use the "-b" switch for faster execution.

  • The wipe command is now deprecated in favor of the lamhalt command. The lamhalt command is identical in functionality to wipe except that it does not require a hostfile argument to shut down an entire LAM. That is, to shut down LAM, you can simply execute:
          % lamhalt

    lamhalt is considerably faster and does not require a hostfile. wipe should only be used if something happened to "damage" the currently running LAM such that lamhalt fails to shut it down properly.

  • lamboot has been extended to understand multiple CPUs on a single host, and is intended to be used in conjunction with the new "C" mpirun syntax for running on SMP machines (see the section on mpirun). Multiple CPUs can be indicated in two ways: list a hostname multiple times, or add a "cpu=N" phrase to the host line (where "N" is the number of CPUs available on that host). For example, the following hostfile:
    	pinky cpu=2

    indicates that there are four CPUs available on the "blinky" host, and that there are two CPUs available on the "pinky" host. Note that this works nicely in a PBS environment, because PBS will list a host multiple times when multiple vnodes on a single node have been allocated by the scheduler.

    Note that different usernames can be specified for specific hosts as well. For example:

           blinky cpu=2 user=lamguest

    specifies that the username "lamguest" should be used to login to the machine "blinky". This is different than previous syntax for specifying usernames for remote nodes; although its use is deprecated, the "-o" option to recon, lamboot, and wipe can be used to obtain previous parsing behavior.

  • The "-s" option to lamboot will now close the stdout/stderr of the locally-launched LAM daemon. Normally, the stdout/stderr of the local lamd is left open so that LAM's tstdio(3) emulation package works properly. Using "-s" does not affect the behavior of mpirun; for example:
            # csh-based example
            mpirun -np 4 foo >& foo.out

    still works as expected, by redirecting the output of the parallel run of "foo" to the file foo.out. The "-s" option does allow the lamboot command to be executed remotely by a remote agent such as rsh:

    	rsh somenode lamboot -s hostfile

    rsh will wait for stdout/stderr to close before returning. So even if lamboot completes, rsh will hang while waiting for stdout/stderr of all launched processes to close. The "-s" option will force lamboot to close stdout/stderr and allow rsh to complete normally. While most programs will not be affected, be sure to note that "-s" will break tstdio functionality.

  • Setting the LAM_MPI_SOCKET_SUFFIX environment variable before running lamboot will a user to have multiple instances of the LAM/MPI run-time environment on a single machine. This is useful for batch and/or automated execution types of environments where the same node may be used multiple times.

  • The wipe command is now symmetric to lamboot in that if you do not specify a hostfile on the command line, wipe will use the default boot schema ($prefix/etc/lam-bhost.def, where $prefix is the top-level directory where LAM was installed).

    Previous versions of wipe would ask for a filename if none were specified on the command line; this option has been removed.

  • Additionally, the debugging capability of lamboot has been enhanced. If the "-d" switch is given to lamboot, lamboot (and its helper programs) become quite verbose, showing exactly what is being executed (or not) -- which is a great help when attempting to determine the exact source of user problems. When started up in debugging mode, the lamd will now output some diagnostic messages to the syslog when starting and when launching user programs via mpirun or lamexec. The messages are output at the LOG_INFO severity, and will be prefixed with "lamd" and the lamd's PID.

  • Finally, the LAMRSH environment variable can be set at invocation time to override the default remote shell command that recon/lamboot/wipe/etc. use to invoke commands on remote nodes (e.g., rsh, ssh, etc.). The default value is wherever rsh is found in the path at ./configure time (or specified with the --with-rsh option); setting the LAMRSH environment variable will override this setting.

    The LAMRSH environment variable can contain a binary name and any number of arguments. For example (for the C shell and its derivatives):

         % setenv LAMRSH "ssh -x"


    LAM was specifically designed to abstract away hostnames once lamboot has completed successfully. However, for various reasons (usually related to system-administration concerns, and/or for creating human-readable reports), it can be desirable to retrieve the hostnames of LAM nodes long after lamboot.

    A new command, "lamnodes", can be used for this purpose. It accepts both the "N" and "C" syntax from mpirun, and will return the corresponding names of the specified nodes. For example:

          % lamnodes C

    will return the node that each CPU is located on, the hostname of that node, and the total number of CPUs on each (lamnodes' command line arguments can be used to customize its output).

          % lamnodes n0,3

    will return the node, hostname, and number of CPUs on n0 and n3.

    Root execution disallowed

    It is a Very Bad Idea to run the LAM executables as root.

    LAM was designed to be run by individual users; it was *not* designed to be run as a root-level service where multiple users use the same LAM daemons in a client-server fashion (see "Typical Usage" in the INSTALL file). LAM should be booted by each individual user who wishes to run MPI programs. There are a wide array of security issues when root runs a service-level daemon; LAM does not even attempt to address any of these issues.

    Especially with today's propensity for hackers to scan for root-owned network daemons, it could be tragic to run this program as root. While LAM is known to be quite stable, and LAM does not leave network sockets open for random connections after the initial setup, several factors should strike fear into system administrator's hearts if LAM were to be constantly running for all users to utilize:

    1. LAM leaves a unix domain socket open on each machine in the /tmp directory. So if someone breaks into root on one machine, they effectively have root on all machines that are connected via LAM.
    2. Indeed, there must have been a .rhosts (or some other trust mechanism) for root which must have allowed you to run LAM on remote nodes. Depending on your local setup, this may not be safe.
    3. LAM has never been checked for buffer overflows and other malicious input types of errors. We don't *think* that there are any buffer-overflow types of situations in LAM, we've never checked explicitly (hence, per Mr. Murphy, there are certainly some hiding somewhere).
    4. LAM programs are not audited or tracked in any way. This could present a sneaky way to execute binaries without log trails (especially as root).

    Hence, it's a Very Bad Idea to run LAM as root. LAM binaries will quit immediately if root runs them. Login as a different user to run LAM.

    RPI transport layers

    LAM currently provides three client-to-client transport layers which implement the request progression interface (RPI). As in LAM 6.1, the LAM daemon RPI transport (lamd) is always available. It is no longer the default transport and must be explicitly invoked via the -lamd option to mpirun.

    The three client-to-client transports are:


    The tcp transport uses TCP sockets for all interprocess communication.


    The usysv transport is multi-protocol. Processes on the same node communicate via SYSV shared memory and processes on different nodes communicate via TCP sockets. It uses spin-locks for shared memory synchronization as well as a SYSV semaphore or pthread mutex for synchronizing access to a per node global shared memory pool.

    The spin-locks require that the architecture has strongly ordered writes and this transport is only supported on such platforms. It should be relatively easy to modify this transport to work on systems with weakly ordered writes by adding memory barriers in appropriate places.

    The usysv transport should give the best performance on SMPs. Please refer to the tuning notes for more on the performance of the various transports and on tuning them.


    The sysv transport is the same as the usysv transport except that SYSV semaphores are used for message synchronization rather than spin-locks. On some uniprocessor systems (e.g. Linux), the blocking nature of semaphores can lead to better performance than when using spin-locks.

    IMPI extensions

    The IMPI extensions are still considered experimental, and are disabled by default in LAM. They must be enabled when LAM is configured and built -- see the INSTALL file for details.

    Since the IMPI extensions are still experimental, relevant instructions and release notes are in the top-level README.impi file in the LAM distribution.

    Signal catching

    LAM MPI now catches the signals SEGV, BUS, FPE, and ILL. The signal handler terminates the application. This is useful in batch jobs to help ensure that mpirun returns if an application process dies. To disable the catching of signals use the -nsigs option to mpirun.

    Internal signal

    The signal used internally by LAM has been changed from SIGUSR1 to SIGUSR2 to reduce the chance of conflicts with the Linux pthreads library. The signal used is configurable. See the installation guide for the specific ./configure flag that can be used to change the internal signal.

    New basic datatypes

    Support has been added for the MPI_LONG_LONG_INT, MPI_UNSIGNED_LONG_LONG and MPI_WCHAR basic datatypes.

    MPI-2 Support

    LAM 6.5.9 has support for many MPI-2 features.


    MPI-IO support has been added by including the ROMIO package from Argonne National Labs (, version 1.0.3. The MPI-IO functions are described in chapter 9 of the MPI-2 standard, which can be found at

    The ROMIO package can be compiled with LAM, and the LAM wrapper compilers (mpicc/mpiCC/mpif77) will automatically do "the right things" to compile/link user programs that use ROMIO function calls.

    The ROMIO group at Argonne has now included support for LAM in their package. However, there are still "one or two" things that the LAM Team had to do to make ROMIO behave nicely with LAM/MPI. As such, if you try to install the ROMIO package manually with LAM/MPI, you will experience some difficulties.

    ROMIO support can be disabled via the LAM ./configure script; see the INSTALL file for specific instructions.

    There are some important limitations to ROMIO that are discussed in the romio/README file.

    One limitation that is not currently listed in the ROMIO README file is that atomic file access will not work with AFS. This is because of file locking problems with AFS (i.e., AFS does not support file locking). The ROMIO test program "atomicity" will fail if you specify an output file on AFS.

    Since building ROMIO is now the default for LAM/MPI, building LAM's profiling layer is also the default. You cannot build ROMIO without profiling support (i.e., if you attempt to configure LAM to not build the profiling layer and still build ROMIO, the configure script will complain).

    Additionally, ROMIO does not [yet] support shared libraries. Even though LAM can now support shared libraries, if you select to build ROMIO (the default), shared library support is disabled.

    Additionally, ROMIO does not support the following LAM functionality:

    • LAM MPI-2 datatypes cannot be used with ROMIO; ROMIO makes the fundamental assumption that MPI-2 datatypes are built upon MPI-1 datatypes. LAM builds MPI-2 datatypes natively -- ROMIO cannot presently handle this case. This will hopefully be fixed in some future release of ROMIO. The ROMIO test programs "coll_test", "fcoll_test", "large_array", and "coll_perf" will fail because they use the MPI-2 datatype MPI_DARRAY.

    Please see the sections "ROMIO Users Mailing List" and "Reporting Bugs" in romio/README for how to submit questions and bug reports about ROMIO (that do not specifically pertain to LAM).

    C++ bindings

    C++ bindings for MPI-1 are provided from the MPI-2 C++ bindings package from Indiana University (, version 1.0.6. The MPI-1 C++ bindings are described in Chapter 10 and Appendix B of the MPI-2 standard, which can be found at

    The C++ bindings package is compiled, by default, with LAM, and the LAM wrapper compilers (mpicc/mpiCC/mpif77) will automatically do "the right things" to compile/link user programs that use MPI C++ bindings function calls.

    Note that the C++ bindings have requirements on the degree of conformance that your C++ compiler supports; see the file mpi2c++/README for more details. If your C++ compiler cannot support the requirements of the C++ bindings package, it is safest just to disable MPI C++ bindings support in LAM (however, most modern C++ compilers now support all the required functionality).

    MPI C++ bindings support can be disabled via the LAM ./configure script; see the INSTALL file for specific instructions.

    Please see the "Contact Information" section of the mpi2c++/README file for how to submit questions and bug reports about the MPI 2 C++ bindings package (that do not specifically pertain to LAM).

    Inter-language interoperability

    Inter-language interoperability is supported. It is now possible to initialize LAM MPI from either C or Fortran and mix MPI calls from both languages.

    One-sided communication

    Support is provided for get/put/accumulate data transfer operations and for the post/wait/start/complete and fence synchronization operations. No support is provided for window locking.

    The datatypes used in the get/put/accumulate operations are restricted to being basic datatypes or single level contiguous/vectors of basic datatypes.

    The implementation of the one-sided operations is layered on top of the point-to-point functions, and will thus perform no better than them. Nevertheless it is hoped that providing this support will aid developers in developing and debugging codes using one-sided communication.

    The following functions related to one-sided communication have been implemented.

    • MPI_Win_create
    • MPI_Win_free
    • MPI_Win_get_group

    • MPI_Get
    • MPI_Put
    • MPI_Accumulate

    • MPI_Win_fence
    • MPI_Win_post
    • MPI_Win_wait
    • MPI_Win_start
    • MPI_Win_complete

    Dynamic processes

    The dynamic process support provided in LAM 6.2b has been extended and the function names changed to conform to the final MPI-2 standard. The following functions related to dynamic process support are provided.

    • MPI_Comm_spawn
    • MPI_Comm_spawn_multiple
    • MPI_Comm_get_parent
    • MPI_Comm_accept
    • MPI_Comm_connect
    • MPI_Comm_disconnect
    • MPI_Comm_join

    • MPI_Lookup_name
    • MPI_Publish_name
    • MPI_Unpublish_name

    • MPI_Open_port
    • MPI_Close_port

    As requested by LAM users, MPI_Comm_spawn and MPI_Comm_spawn_multiple can now take an MPI_Info argument specifying an appschema file for MPMD spawns. The key name is "file", and the value should be the filename of the appschema (relative to the present working directory on the root, or an absolute pathname accessible on the root).

    These functions supersede the MPIL_Comm_spawn function that LAM/MPI introduced in version 6.2b. MPIL_Comm_spawn is considered depreciated and should not be used (although it is still available for backwards compatibility).


    Full support for info objects is provided.

    • MPI_Info_create
    • MPI_Info_free
    • MPI_Info_delete
    • MPI_Info_dup
    • MPI_Info_get
    • MPI_Info_get_nkeys
    • MPI_Info_get_nthkey
    • MPI_Info_get_valuelen
    • MPI_Info_set

    Communicator and window error handling

    The new communicator error handler functions are supported, and window error handlers are also supported.

    • MPI_Comm_create_errhandler
    • MPI_Comm_get_errhandler
    • MPI_Comm_set_errhandler

    • MPI_Win_create_errhandler
    • MPI_Win_get_errhandler
    • MPI_Win_set_errhandler

    Handle conversions

    Handle conversions for inter-language interoperability are supported.

    • MPI_Comm_f2c
    • MPI_Comm_c2f

    • MPI_Group_f2c
    • MPI_Group_c2f

    • MPI_Type_f2c
    • MPI_Type_c2f

    • MPI_Request_f2c
    • MPI_Request_c2f

    • MPI_Info_f2c
    • MPI_Info_c2f

    • MPI_Win_f2c
    • MPI_Win_c2f

    • MPI_Status_f2c
    • MPI_Status_c2f

    Attributes on communicators, datatypes, and windows

    Attributes may now be set on and retrieved from datatypes and windows. The new communicator attribute handling functions are also supported.

    • MPI_Comm_create_keyval
    • MPI_Comm_free_keyval
    • MPI_Comm_delete_attr
    • MPI_Comm_get_attr
    • MPI_Comm_set_attr

    • MPI_Type_create_keyval
    • MPI_Type_free_keyval
    • MPI_Type_delete_attr
    • MPI_Type_get_attr
    • MPI_Type_set_attr

    • MPI_Win_create_keyval
    • MPI_Win_free_keyval
    • MPI_Win_delete_attr
    • MPI_Win_get_attr
    • MPI_Win_set_attr

    New derived type constructors and type inquiry functions

    Support has been added for the following new derived type constructors.

    • MPI_Type_create_struct
    • MPI_Type_create_hindexed
    • MPI_Type_create_hvector
    • MPI_Type_dup
    • MPI_Type_create_resized
    • MPI_Type_create_subarray
    • MPI_Type_create_darray

    Support has also been added for the new type inquiry functions.

    • MPI_Type_get_contents
    • MPI_Type_get_envelope
    • MPI_Type_get_extent
    • MPI_Type_get_true_extent


    Implementations of the following functions are provided. LAM 6.5.9 reports its MPI version as 1.2.

    • MPI_Alloc_mem
    • MPI_Finalized
    • MPI_Free_mem
    • MPI_Get_version
    • MPI_Get_address

    The MPI_Alloc_mem and MPI_Free_mem functions (and their two corresponding RPI functions) have been added in anticipation of RPIs that can take advantage of "special" memory for fast message passing. They are simply wrappers to malloc() and free() (respectively) in the currently provided RPIs, but can be used for portability and future performance gains.