Hello,

I am working with LAM MPI and BLCR to checkpoint applications. I replicate the check pointed files across selected LAM nodes and then try to restart application on them.

So when I tested in just a single node the application gets checkpointed with BLCR and the corresponding context file is produced. But when I tested it with two nodes the mpirun itself gets hung, so that it doesnt even gets checkpointed.

After debugging, I found that when an mpirun command is issued, MPI_init( ) gets blocked. After going deeper I found the problem in the following snippet of code in /share/mpi/laminit.c -

/*
* If spawned or started by mpirun, receive the list of GPS. Local
* world GPS's are first followed by the parents (if any). Otherwise if
* the number of processes is one assume a singleton init, else assume one
* process per node and pids are not needed.
*/

printf("kli_init() - Receiving GPS list \n");
if ((_kio.ki_parent > 0) || (_kio.ki_rtf & RTF_MPIRUN)) {

nhead.nh_event = (-lam_getpid()) & 0xBFFFFFFF;
nhead.nh_type = BLKMPIINIT;
nhead.nh_flags = DINT4DATA;
nhead.nh_length = procs_n * sizeof(struct _gps);
nhead.nh_msg = (char *) procs;

printf("Gonna do nrecv \n"); // Debugging statement added by me

if (nrecv(&nhead)) { // nrecv blocks
free((char *) procs);
return(LAMERROR);
}

printf("Did nrecv \n"); // added by me

The nrecv gets blocked on the remote node, consequently MPI_Init() does not proceed further.

As this is happening even before the checkpoint module is called I am wondering will this be related to some network issues or due to some mismatch in directory structures across the nodes.

--

Mukuntan V Viswanathan

Visit My webpage at
www.buffalo.edu/~mvv2