On Nov 8, 2004, at 4:57 AM, Vinod Kannan wrote:
> I am presently trying to create a threaded + MPI
> application. I went through the previous postings
> related to the subject. I have a few questions, could
> someone please help answer them?
>
> 1) I saw some "teasers" which lead me to believe that
> LAM MPI might be thread safe in an immediate future
> version. I haven't seen any further postings regarding
> the same. Is there a thread safe LAM-MPI version?
No.
The teasers were in reference to Open MPI (http://www.open-mpi.org/),
the new, next-generation MPI implementation that we're working on.
Open MPI will support MPI_THREAD_MULTIPLE.
Have you looked at LA-MPI? They support MPI_THREAD_MULTIPLE already
(they are one of our collaborators on Open MPI; we used a lot of their
technology and ideas for MPI_THREAD_MULTIPLE in Open MPI).
> 2) What would I need to be aware of with regards to
> the portablity of a code ( using threads+ Lam/MPI) to
> other Operating Systems?
Not sure what you mean here... Threads packages are different on many
platforms. So if you use threads, you may need to have #if kinds of
directives to do different things on different platforms. Your MPI
calls should be the same on all platforms; your code should compile and
run with Open MPI on all different platforms.
> 3) One of the postings had a sample code
> ("http://www.lam-mpi.org/MailArchives/lam/msg04037.php").
> I tried running the code on my machine, requesting and
> getting MPI_THREAD_SERIALIZE. It hangs ( the non MPI
> thread comes out, the other one just hangs). Could
> someone tell me what I need to do to make sure the
> code executes correctly? SERIALIZE would take care of
> the locking and unlocking for every MPI call - is that
> correct?
Correct. It ensures that only one thread is in the MPI library at a
time -- there is no concurrency of MPI actions between threads, so
deadlocking is still possible.
> So it is not a deadlock issue, correct? Why
> then does it hang? The configuration Im using is:
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
> 4) Is reinstalling LinuxThreads with SIGUSR1 &
> SIGUSR2 turned off mandatory? (The posts regarding the
> same were mainly from the year 1998)
Not sure what this is in reference to. LAM uses SIGUSR2 internally, by
default. I wasn't aware that the Linux 2.4 threads used that signal
internally...? I don't think that Linux 2.6 threads use that signal
internally, either.
> 5) I am using LAM 7.0.6. Do I still need to go in and
> add the thread flags for compilation ( -D_REENTRANT
> and -lpthreads )? It seems the flags for threads are
> included by default in version 7.0.6. Also if I need
> to reinstall linuxthreads with SIGUSR1&2 turned off,
> then do I need to point to the different library
> (-pthread vx -lpthread?
Correct -- LAM should be putting in all the -D_REENTRANT (and friends)
for you.
The only difference between -pthread and -lpthread (for gcc) is that
-pthread is a "macro"-like command line argument that turns into
whatever the "right" flags are for pthreads (i.e., gcc does a
substitution on -pthread). We prefer using -pthread instead of
-lpthread because on some platforms, the pthreads library is named
something other that libpthread.* (IIRC, some of the BSD's were in this
category).
> 6) I am wondering if the following design can be
> implemented with the present version of LAM without
> any ( little) impact of performance. What issues would
> I have to be careful of in terms of performance?
>
> Why I need threads is I am creating is a framework
> for application-level fault tolerance/ dynamic
> addition( & removal) of nodes at the application
> level. LAMs fault tolerant capabilities are
> considerable ( especially with regards to node
> failure) but I need additional functionality.
>
> Right now I have 2 MPI execs B, C launched via a
> appschema file. To add ( robust) fault tolerance and
> other capabilities I plan to redesign by adding a
> daemon app A as follows.
>
> I will have 3 execs A,B,C. B & C's existing logic
> will be forked off as their business-logic thread(
> only MPI calls used in this thread will be Non
> blocking sends and receives + test + wait) and a
> control thread( for all dynamic MPI calls, fault
> tolerance via regular messaging, communication of
> new-node addition( /new instance creation on the added
> node) via lamgrow etc). I want to launch B & C from
> A's control thread using MPISpawn. The control threads
> in B & C will MPIConnect to a name published by A's
> control thread. B's control thread will publish and
> accept a connect from C's control thread. B & C will
> communicate with each other on the Businesslogic
> thread, and communicate with A on the control thread.
>
> I plan to use MPI_THREAD_SERIALIZE. Can the above
> design be accomplished
Yes. But you'll need to carefully synchronize your MPI usage between
the two threads -- it's easy to imagine a deadlock situation where one
blocking MPI call is waiting indefinitely and the second thread then
also blocks trying to get into the MPI library (because the 1st thread
has it locked). Recall that hold-and-wait is one of the four necessary
criteria for deadlock; try to avoid those situations in your app.
> with little or no performance
> degradation?
This is relative. You're going to incur a little extra overhead
because you're locking and unlocking the MPI library in every MPI
function. And threads themselves add a small amount of overhead to a
process -- especially if you have more threads than CPU's. If you're
oversubscribing the node with more threads than CPUs, then it becomes a
game of how you "schedule" your threads -- if you can guarantee that
some threads are blocking most of the time while the other is running.
There's also cache effects to consider, etc. (i.e., the pre-empting of
one thread with another may cause cache thrashing, which can noticeably
impact your performance).
Don't get me wrong -- I think threads are great, but be sure that you
understand the implications and potential performance issues before
assuming that they can solve all your problems.
> Could there be issues with scaling the
> above design?
Yes. If you want all processes to be able to talk to each other, then
every time you add a new "worker" (B, C, D, E, F, ...etc.), you'll need
to have an intricate dance of CONNECT / ACCEPT calls to make this
happen. You might want to ask yourself if your set of workers really
needs to be fully connected or not.
> What issues will I have to be aware of
> considering Im using MPI_THREAD_SERIALIZE?
Just that this is implemented with one big lock around the MPI library.
As I mentioned above, be careful of deadlock situations (most will
involve fairly obvious hold-and-wait issues).
> Could
> someone with knowledge/experience with using threads
> with MPI point me in the right direction?
Good luck!
--
{+} Jeff Squyres
{+} jsquyres_at_[hidden]
{+} http://www.lam-mpi.org/
|
