Hi all,
My mpi program did work until 14 processors. But when i used 16 it gaves
me the following message.
Why is that ?
nuthatch% mpirun -v -np 16 out A2000.txt
5134 out running on n0 (o)
22456 out running on n1
21401 out running on n2
22179 out running on n3
20861 out running on n4
21264 out running on n5
21180 out running on n6
21031 out running on n7
20921 out running on n8
21398 out running on n9
22133 out running on n10
21685 out running on n11
7177 out running on n12
21030 out running on n13
20397 out running on n14
20071 out running on n15
-----------------------------------------------------------------------------
It seems that rank 15 was not able to open a TCP client socket for
some reason. LAM is likely to abort your program shortly. :-(
Perhaps this unix error message will help:
Unix errno: 111
Connection refused
-----------------------------------------------------------------------------
----------------------------------------------------------------------------
-----------------------------------------------------------------------------
The selected RPI failed to initialize during MPI_INIT. This is a
fatal error; I must abort.
This occurred on host alarqas (n15).
The PID of failed process was 20071 (MPI_COMM_WORLD rank: 15)
-----------------------------------------------------------------------------
MPI_Recv: process in local group is dead (rank 4, MPI_COMM_WORLD)
Rank (4, MPI_COMM_WORLD): Call stack within LAM:
Rank (4, MPI_COMM_WORLD): - MPI_Recv()
Rank (4, MPI_COMM_WORLD): - MPI_Barrier()
Rank (4, MPI_COMM_WORLD): - MPI_Init()
Rank (4, MPI_COMM_WORLD): - main()
MPI_Recv: process in local group is dead (rank 8, MPI_COMM_WORLD)
Rank (8, MPI_COMM_WORLD): Call stack within LAM:
Rank (8, MPI_COMM_WORLD): - MPI_Recv()
Rank (8, MPI_COMM_WORLD): - MPI_Barrier()
Rank (8, MPI_COMM_WORLD): - MPI_Init()
Rank (8, MPI_COMM_WORLD): - main()
MPI_Recv: process in local group is dead (rank 5, MPI_COMM_WORLD)
Rank (5, MPI_COMM_WORLD): Call stack within LAM:
Rank (5, MPI_COMM_WORLD): - MPI_Recv()
Rank (5, MPI_COMM_WORLD): - MPI_Barrier()
Rank (5, MPI_COMM_WORLD): - MPI_Init()
Rank (5, MPI_COMM_WORLD): - main()
MPI_Recv: process in local group is dead (rank 9, MPI_COMM_WORLD)
Rank (9, MPI_COMM_WORLD): Call stack within LAM:
Rank (9, MPI_COMM_WORLD): - MPI_Recv()
Rank (9, MPI_COMM_WORLD): - MPI_Barrier()
Rank (9, MPI_COMM_WORLD): - MPI_Init()
Rank (9, MPI_COMM_WORLD): - main()
MPI_Recv: process in local group is dead (rank 10, MPI_COMM_WORLD)
Rank (10, MPI_COMM_WORLD): Call stack within LAM:
Rank (10, MPI_COMM_WORLD): - MPI_Recv()
Rank (10, MPI_COMM_WORLD): - MPI_Barrier()
Rank (10, MPI_COMM_WORLD): - MPI_Init()
Rank (10, MPI_COMM_WORLD): - main()
MPI_Recv: process in local group is dead (rank 11, MPI_COMM_WORLD)
Rank (11, MPI_COMM_WORLD): Call stack within LAM:
Rank (11, MPI_COMM_WORLD): - MPI_Recv()
Rank (11, MPI_COMM_WORLD): - MPI_Barrier()
Rank (11, MPI_COMM_WORLD): - MPI_Init()
Rank (11, MPI_COMM_WORLD): - main()
MPI_Recv: process in local group is dead (rank 12, MPI_COMM_WORLD)
Rank (12, MPI_COMM_WORLD): Call stack within LAM:
Rank (12, MPI_COMM_WORLD): - MPI_Recv()
Rank (12, MPI_COMM_WORLD): - MPI_Barrier()
Rank (12, MPI_COMM_WORLD): - MPI_Init()
Rank (12, MPI_COMM_WORLD): - main()
MPI_Recv: process in local group is dead (rank 13, MPI_COMM_WORLD)
Rank (13, MPI_COMM_WORLD): Call stack within LAM:
Rank (13, MPI_COMM_WORLD): - MPI_Recv()
Rank (13, MPI_COMM_WORLD): - MPI_Barrier()
Rank (13, MPI_COMM_WORLD): - MPI_Init()
Rank (13, MPI_COMM_WORLD): - main()
-----------------------------------------------------------------------------
One of the processes started by mpirun has exited with a nonzero exit
code. This typically indicates that the process finished in error.
If your process did not finish in error, be sure to include a "return
0" or "exit(0)" in your C code before exiting the application.
PID 5134 failed on node n0 (134.153.50.235) with exit status 1.
-----------------------------------------------------------------------------
nuthatch% ---------------------------------------
<
/bin/ksh:
-------------------------------------------------------------------------:
not found
nuthatch% MPI_Recv: process in local group is dead (rank 4, MPI_COMM_WORLD)
/bin/ksh: syntax error: `(' unexpected
nuthatch% Rank (4, MPI_COMM_WORLD): Call stack within LAM:
/bin/ksh: syntax error: `4,' unexpected
nuthatch% Rank (4, MPI_COMM_WORLD): - MPI_Recv()
/bin/ksh: syntax error: `4,' unexpected
nuthatch% Rank (4, MPI_COMM_WORLD): - MPI_Barrier()
/bin/ksh: syntax error: `4,' unexpected
nuthatch% Rank (4, MPI_COMM_WORLD): - MPI_Init()
/bin/ksh: syntax error: `4,' unexpected
nuthatch% Rank (4, MPI_COMM_WORLD): - main()
/bin/ksh: syntax error: `4,' unexpected
nuthatch% MPI_Recv: process in local group is dead (rank 8, MPI_COMM_WORLD)
/bin/ksh: syntax error: `(' unexpected
nuthatch% -------------------------------------------------
<
/bin/ksh: syntax error:
`-----------------------------------------------------------------------------'
unexpected
nuthatch% -------------------------------------------
<
/bin/ksh:
-----------------------------------------------------------------------------:
not found
nuthatch% The selected RPI failed to initialize during MPI_INIT. This is a
/bin/ksh: The: not found
nuthatch% fatal error; I must abort.
/bin/ksh: fatal: not found
/bin/ksh: I: not found
nuthatch%
nuthatch% This occurred on host alarqas (n15).
/bin/ksh: syntax error: `(' unexpected
nuthatch% The PID of failed process was 20071 (MPI_COMM_WORLD rank: 15)
/bin/ksh: syntax error: `(' unexpected
nuthatch% -------------------------------------------
<
/bin/ksh:
-----------------------------------------------------------------------------:
not found
nuthatch% MPI_Recv: process in local group is dead (rank 4, MPI_COMM_WORLD)
/bin/ksh: syntax error: `(' unexpected
nuthatch% Rank (4, MPI_COMM_WORLD): Call stack within LAM:
/bin/ksh: syntax error: `4,' unexpected
nuthatch% Rank (4, MPI_COMM_WORLD): - MPI_Recv()
/bin/ksh: syntax error: `4,' unexpected
nuthatch% Rank (4, MPI_COMM_WORLD): - MPI_Barrier()
/bin/ksh: syntax error: `4,' unexpected
nuthatch% Rank (4, MPI_COMM_WORLD): - MPI_Init()
/bin/ksh: syntax error: `4,' unexpected
nuthatch% Rank (4, MPI_COMM_WORLD): - main()
/bin/ksh: syntax error: `4,' unexpected
nuthatch% MPI_Recv: process in local group is dead (rank 8, MPI_COMM_WORLD)
/bin/ksh: syntax error: `(' unexpected
nuthatch% Rank (8, MPI_COMM_WORLD): Call stack within LAM:
/bin/ksh: syntax error: `8,' unexpected
nuthatch% Rank (8, MPI_COMM_WORLD): - MPI_Recv()
/bin/ksh: syntax error: `8,' unexpected
nuthatch% Rank (8, MPI_COMM_WORLD): - MPI_Barrier()
/bin/ksh: syntax error: `8,' unexpected
nuthatch% Rank (8, MPI_COMM_WORLD): - MPI_Init()
/bin/ksh: syntax error: `8,' unexpected
nuthatch% Rank (8, MPI_COMM_WORLD): - main()
/bin/ksh: syntax error: `8,' unexpected
nuthatch% MPI_Recv: process in local group is dead (rank 5, MPI_COMM_WORLD)
/bin/ksh: syntax error: `(' unexpected
nuthatch% Rank (5, MPI_COMM_WORLD): Call stack within LAM:
/bin/ksh: syntax error: `5,' unexpected
nuthatch% Rank (5, MPI_COMM_WORLD): - MPI_Recv()
/bin/ksh: syntax error: `5,' unexpected
nuthatch% Rank (5, MPI_COMM_WORLD): - MPI_Barrier()
/bin/ksh: syntax error: `5,' unexpected
nuthatch% Rank (5, MPI_COMM_WORLD): - MPI_Init()
/bin/ksh: syntax error: `5,' unexpected
nuthatch% Rank (5, MPI_COMM_WORLD): - main()
/bin/ksh: syntax error: `5,' unexpected
nuthatch% MPI_Recv: process in local group is dead (rank 9, MPI_COMM_WORLD)
/bin/ksh: syntax error: `(' unexpected
nuthatch% Rank (9, MPI_COMM_WORLD): Call stack within LAM:
/bin/ksh: syntax error: `9,' unexpected
nuthatch% Rank (9, MPI_COMM_WORLD): - MPI_Recv()
/bin/ksh: syntax error: `9,' unexpected
nuthatch% Rank (9, MPI_COMM_WORLD): - MPI_Barrier()
/bin/ksh: syntax error: `9,' unexpected
nuthatch% Rank (9, MPI_COMM_WORLD): - MPI_Init()
/bin/ksh: syntax error: `9,' unexpected
nuthatch% Rank (9, MPI_COMM_WORLD): - main()
/bin/ksh: syntax error: `9,' unexpected
nuthatch% MPI_Recv: process in local group is dead (rank 10, MPI_COMM_WORLD)
/bin/ksh: syntax error: `(' unexpected
nuthatch% Rank (10, MPI_COMM_WORLD): Call stack within LAM:
/bin/ksh: syntax error: `10,' unexpected
nuthatch% Rank (10, MPI_COMM_WORLD): - MPI_Recv()
/bin/ksh: syntax error: `10,' unexpected
nuthatch% Rank (10, MPI_COMM_WORLD): - MPI_Barrier()
/bin/ksh: syntax error: `10,' unexpected
nuthatch% Rank (10, MPI_COMM_WORLD): - MPI_Init()
/bin/ksh: syntax error: `10,' unexpected
nuthatch% Rank (10, MPI_COMM_WORLD): - main()
/bin/ksh: syntax error: `10,' unexpected
nuthatch% MPI_Recv: process in local group is dead (rank 11, MPI_COMM_WORLD)
/bin/ksh: syntax error: `(' unexpected
nuthatch% Rank (11, MPI_COMM_WORLD): Call stack within LAM:
/bin/ksh: syntax error: `11,' unexpected
nuthatch% Rank (11, MPI_COMM_WORLD): - MPI_Recv()
/bin/ksh: syntax error: `11,' unexpected
nuthatch% Rank (11, MPI_COMM_WORLD): - MPI_Barrier()
/bin/ksh: syntax error: `11,' unexpected
nuthatch% Rank (11, MPI_COMM_WORLD): - MPI_Init()
/bin/ksh: syntax error: `11,' unexpected
nuthatch% Rank cd '/users/cs/grad/dilani/Research_2005'
/bin/ksh: Rank: not found
nuthatch%
Thanks.
Dilani Perera.
(MSC Candidate for Computational Sciences)
Department of Computer Science,
St. John's, NL
Canada,A1B 3X5
Tel: 709-737-6142 (office)
email : dilani_at_[hidden]
Visit me at : www.cs.mun.ca/~dilani
|
