Brian,
Thank you so much for the quick response. I just attached the whole
code here, it should be compilable on any cluster because it is so
simple. Can you do me favor? just compile it and run on your cluster to
see what happen? I suspected that maybe something was wrong with my
cluster setting or configuration, the results doesn't make sense at all
!!! If MPI by itself is so inefficient, it would be used so widely.
Here is my cluster's information:
Beowulf linux cluster with redhat 8.0 3.2-7
7 slave nodes, each has two CPU (AMD MP1800+)
1 server, two CPU, but only one is used in lam
here is what I did for this test:
>mpiCC -o TestSpeed TestSpeed.c (TestSpeed can be accessed by all
nodes)
>mpirun C TestSpeed
results (I reduced dimensionality for quick test, it was 20000X1000,
now it is 200*100):
node 0 comutime 0.001150 totaltime 0.016289 allgather time 0.000621
node 1 comutime 0.005046 totaltime 0.223765 allgather time 0.000671
node 2 comutime 0.005082 totaltime 0.261913 allgather time 0.001139
node 3 comutime 0.005048 totaltime 0.267254 allgather time 0.000817
node 4 comutime 0.004983 totaltime 0.267995 allgather time 0.000689
node 5 comutime 0.004646 totaltime 0.277137 allgather time 0.000699
node 6 comutime 0.004730 totaltime 0.276793 allgather time 0.001246
node 7 comutime 0.004709 totaltime 0.284526 allgather time 0.000781
node 8 comutime 0.004652 totaltime 0.285085 allgather time 0.000642
node 9 comutime 0.004655 totaltime 0.290738 allgather time 0.000693
node 10 comutime 0.004716 totaltime 0.290280 allgather time 0.001134
node 11 comutime 0.004751 totaltime 0.294677 allgather time 0.000773
node 12 comutime 0.004716 totaltime 0.295332 allgather time 0.000697
node 13 comutime 0.004707 totaltime 0.297164 allgather time 0.000708
node 14 comutime 0.004680 totaltime 0.296572 allgather time 0.001253
Guoli Wang, Ph.D.
Bioinformatics Group
Fox Chase Cancer Center
Phone: (215)-214-4261
On Nov 2, 2005, at 8:48 AM, Brian Barrett wrote:
> On Nov 1, 2005, at 4:59 PM, wgl wrote:
>
>> I have a small test code here trying to test the performance of
>> MPI_Allgatherv, I was shocked by the waiting time that MPI_Allgatherv
>> consumed. I'm sure something wrong with my code. Could someone help me
>> to modify it? Thanks a lot!!!!
>
> These numbers are unexpected, but I really can't give any answers as
> to why (based on the information you gave). Our Allgatherv
> implementation is really simple - it's basically:
>
> for (int i = 0 ; i < nprocs ; ++i) {
> MPI_Gatherv(sendbuf, sendcount, sendtype, recvbuf, recvcounts,
> displs, recvtype, i, comm);
> }
>
> Which means that each process is going to get slammed with every
> other proc's data, one at a time. Not exactly the best use of the
> network :/. You're sending approximately .5MB of data from each
> process, so there's a fairly sizable spike in network traffic all at
> once.
>
> Just out of curiosity, how are you running this code? What network,
> how many processors per node, how many processes per node, etc?
>
> Brian
>
>
>> +++++++++++++++++++++++++++++++++++++++++++
>> results
>>
>> node 0 comutime 0.000001 totaltime 13.694220
>> node 1 comutime 5.649959 totaltime 95.133599
>> node 2 comutime 5.464476 totaltime 95.928307
>> node 3 comutime 5.464566 totaltime 96.483795
>> node 4 comutime 5.540017 totaltime 97.016238
>> node 5 comutime 5.466977 totaltime 97.538457
>> node 6 comutime 5.618389 totaltime 98.086424
>> node 7 comutime 5.490053 totaltime 98.649446
>> node 8 comutime 5.541841 totaltime 99.408869
>> node 9 comutime 5.466705 totaltime 99.961623
>> node 10 comutime 5.466315 totaltime 100.550240
>> node 11 comutime 5.515423 totaltime 101.133764
>> node 12 comutime 5.516169 totaltime 101.655122
>> node 13 comutime 5.480149 totaltime 101.737988
>> node 14 comutime 5.386903 totaltime 101.799232
>> ++++++++++++++++++++++++++++++++++++++++++
>>
>> code:
>>
>> #include <stdio.h>
>> #include <mpi.h>
>>
>> const int Nrows = 20000;
>> const int Ncolumns = 1000;
>>
>> int main (int argc, char **argv) {
>> // int matrix[Nrows][Ncolumns];
>> int myRank, Nprocessors;
>>
>> MPI_Status stat;
>> MPI_Datatype columntype, rowtype;
>>
>> MPI_Init(&argc, &argv);
>> MPI_Comm_rank(MPI_COMM_WORLD, &myRank);
>> MPI_Comm_size(MPI_COMM_WORLD, &Nprocessors);
>>
>>
>> int counts[Nprocessors];
>> int displacements[Nprocessors];
>> int offsets[Nprocessors];
>>
>> counts[0] = 0;
>> displacements[0] = 0;
>> displacements[1] = 0;
>>
>> int bin = Nrows / (Nprocessors-1);
>> int bins[Nprocessors];
>>
>> int *matrix = new int[Nrows*Ncolumns];
>>
>> for (int i=1; i<Nprocessors; i++) bins[i] = bin;
>>
>> if( Nrows > (Nprocessors - 1) * bin )
>> for (int i=1; i<=Nrows-(Nprocessors-1)*bin; i++)
>> bins[i] ++;
>>
>> for( int i=1; i<Nprocessors; i++ ) {
>> displacements[i] = displacements[i - 1] + counts
>> [i-1];
>> counts[i] = bins[i] * Ncolumns;
>> }
>>
>> MPI_Allgather( &counts[myRank], 1, MPI_INT, counts, 1,
>> MPI_INT, MPI_COMM_WORLD );
>>
>> int my_offset = displacements[myRank];
>>
>> MPI_Gather( &my_offset, 1, MPI_INT, offsets, 1, MPI_INT, 0,
>> MPI_COMM_WORLD );
>>
>> double start_time = MPI_Wtime();
>> for (int i=0; i<counts[myRank]; i++) {
>> matrix[my_offset + i] = myRank*1000 + i;
>> for (int j=0; j<1000; j++) {}
>> }
>> double end_time = MPI_Wtime();
>>
>> double computation = end_time - start_time;
>>
>> // MPI_Barrier(MPI_COMM_WORLD);
>> MPI_Allgatherv ( &matrix[my_offset], counts[myRank],
>> MPI_INT,
>> matrix, counts, displacements, MPI_INT, MPI_COMM_WORLD);
>>
>> // MPI_Barrier(MPI_COMM_WORLD);
>>
>> end_time = MPI_Wtime();
>> double totaltime = end_time - start_time;
>>
>> printf("node %d comutime %lf totaltime %lf\n", myRank,
>> computation, totaltime);
>> MPI_Finalize();
>> return(0);
>> }
>>
>>
>>
>>
>> Guoli Wang, Ph.D.
>> Bioinformatics Group
>> Fox Chase Cancer Center
>>
>> Phone: (215)-214-4261
>>
>> _______________________________________________
>> This list is archived at http://www.lam-mpi.org/MailArchives/lam/
>>
>
> --
> Brian Barrett
> LAM/MPI developer and all around nice guy
> Have a LAM/MPI day: http://www.lam-mpi.org/
>
>
> _______________________________________________
> This list is archived at http://www.lam-mpi.org/MailArchives/lam/
|
