I developed a C++ app w/ a master/slave architecture for splitting an
array in the master and having the slaves each process part of the
array. The master sets up the slaves, including the communicator, when
the master object is constructed. The Master->ProcessArray() member
splits the array and, depending on input params either does or does not
create a separate thread to send and receive the subarrays from the
previously-established slaves. In either case, I use nonblocking
MPI_Irecv() and MPI_Isend() calls for master/slave communication. When I
do not use a separate thread, all is well. However, if I do use a
separate thread (so the client code can do other work while the thread
spawned by Master->ProcessArray() is running), then MPI_Irecv() blocks,
with the effect that only 1 slave ever gets work and all processing is
sequential. I put 1 second sleeps in the slaves and put fprintfs around
the code to verify the it is the MPI_Irecv() that is causing the
problem:
fprintf(
stderr,
"V: Starting AsyncRecvSubarrayParamsOutFromSlave %d @T %8.3f\n",
TileJob->SlaveStatus->Rank,
GetSecs()
);
MPI_Irecv(
& TileJob->SubarrayParamsOut,
sizeof(SubarrayParamsOutType) / sizeof(int),
MPI_INT,
TileJob->SlaveStatus->Rank, // Source
SubarrayParamsOutTag,
AllComms,
& TileJob->SlaveStatus->MPIRequest
);
fprintf(
stderr,
"V: Done w/ AsyncRecvSubarrayParamsOutFromSlave %d @T %8.3f\n",
TileJob->SlaveStatus->Rank,
GetSecs()
);
Here is the output:
V: Starting AsyncRecvSubarrayParamsOutFromSlave 1 @T 1.755
V: Done w/ AsyncRecvSubarrayParamsOutFromSlave 1 @T 2.763
The same output from a non-threaded run shows that MPI_Irecv() is
nonblocking.
I am running LAM v. 7.0 compiled w/ gcc 2.95.3 on a Sun workstation OS
5.7. The thread is created with pthread_create(). I tried initializing
MPI 3 different ways with the same result:
MPI_Init(NULL, NULL);
MPI_Init_thread(NULL, NULL, MPI_THREAD_SINGLE, &DontCare);
MPI_Init_thread(NULL, NULL, MPI_THREAD_SERIALIZED, &DontCare);
What do I need to do to get this to work in the multithreaded mode?
R. T. Bailey
|
