Hello folks,
I am trying to benchmark a cluster using linpack. Sometimes I get this
message
MPI_Send: Process in local group is dead (rank 1, comm 7)
Rank (2, MPI_COMM_WORLD): Call stack within LAM:
Rank (2, MPI_COMM_WORLD): - MPI_Send()
Rank (2, MPI
Rank (2, MPI_COMM_WORLD): - main()
---------------------------------------------------
One of the processess started by mpirun has exited with a nonzero exity
code. This is typically indicates that the process finished in error.
If your process did not finish in error, be sure to include a "return
0" or "exit(0)" in your C code before exiting the application.
PID 14983 failed on noded n0 (10.0.0.2) with exit status 1.
-----------------------------------------------------------
there are 7 nodes on the cluster and linpack runs on all of them. the
cluster is 6 (slaves) + 1 (master).
I didn't have this problem when I ran on 5 slaves.
I tried to trace by
lamtrace -mpi -v -k n6 p28973
lamtrace (open): def.lamtr: File exists
and the file size remains zero. When I try run the benchmark and then
try tracing it like this, it complains even though i run
mpirun -t c0-13 xhpl
lamtrace -mpi -v -k n4-6 trace
searching for an MPI world, done
-----------------------------------------------------------------------------
lamtrace was unable to find any MPI traces.
This may mean that there is a problem with the lam daemon or lamtrace.
More than likely, however, it means that tracing was not enabled
during any of the MPI jobs run since the last lamboot. In order to
enable tracing, you should specify the '-t' flag to mpirun.
------------------------------
I would appreciate pointers to get trace running and other ways to find
out the process is going dead .
regards
harshu
=====
Never underestimate the predictibitly of stupidity!
__________________________________
Do you Yahoo!?
Free Pop-Up Blocker - Get it now
http://companion.yahoo.com/
|
