On Fri, 9 Jan 2004, [big5] Marty - ½²¨Ê¨j wrote:
> I am developing a job scheduling system for PC Cluster system. So, I
> want to control parallel job (MPI), and I must get PIDs of all MPI
> processes. Unfortunately, I can't get PID of all MPI processes in remote
> nodes.
Is the Maui scheduler and/or PBS not sufficient for what you want to do?
> For example, I execute "mpirun -np 3 ./cpi" at server node and then the
> MPI program will run in node 1, 2 and 3.
>
> How do I get the PID in node1, 2 and 3 ?
We actually do not make this information available to the user; there's
rarely any reason for it.
> I tried using "ps" command to get PID by match user ID and program name,
> but there are still some problem is that, if a user run several jobs
> with the same program name, then I can't identify the program.
>
> I traced source code of OpenPBS, but I can't the solution.!
> ( OpenPBS can handle this well. )
It only works well in PBS when you use the tm boot module in LAM/MPI; if
you use the traditional rsh/ssh startup mechanism, PBS won't be able to
know the PIDs of any of your processes.
Also, more specifically, PBS does not know the PIDs of your MPI
application -- it actually only knows the PIDs of the LAM daemons. It
makes a separate process group for the LAM daemons; therefore, any process
that the daemon forks (including your MPI application) are all in the same
process group. So if PBS wants, it can kill the entire process group (see
the kill(2) man page).
This works because we use the "native" job launch mechanism under PBS --
the TM interface -- to launch all the LAM daemons. Hence, a PBS MOM
daemon will eventually fork/exec each LAM daemon (one per node), and
therefore know its PID.
Hence, for each run-time environment, you'll probably need to have a LAM
SSI boot module that uses the job scheduling system's native job launching
mechanism, and use a mechanism similar to the one described above (the PBS
MOM) to fork/exec the LAM daemons. LAM's boot SSI API is documented at
http://www.lam-mpi.org/using/docs/.
Hope that helps.
--
{+} Jeff Squyres
{+} jsquyres_at_[hidden]
{+} http://www.lam-mpi.org/
|
