As almost a new comer for MPI, I am confused on a specific application: hierarchical communication.

After building a hierarchical bi-tree of processes, I try MPI_Reduce (or MPI_Bcast) to communicate between the two processes at each level.

Surprisingly, the performance gets worse sometimes. For example,

Mpirun –np 16 executive

The first invoking of hierarchical-MPI_Reduce may costs merely 0.01 second, but the second one may cost up to 0.38 second (for some processes), or 38 times.

The question is, what’s the real reason, and tried to find it in a lot of ways. Now I want your kindly help.

The following is the brief code:

       MPI_Init(&argc, &argv);

       MPI_Comm_rank(MPI_COMM_WORLD,&my_rank);

       MPI_Comm_size(MPI_COMM_WORLD,&np)

        level = 0;

        key = my_rank;                              //in MPI_COMM_WORLD

        color = key/power_of_2[level+1];     //finally, colr = rank / 2^(levelno +1)

//color = 0, 0, 1, 1, 2, 2, 3, 3...

        MPI_Comm_split(MPI_COMM_WORLD, color, key, &subComm[levelno][key]);

       //go on splitting, until construct a bi-tree of level ceiling(log(np)/log(2));

    //build a hierarchichal bi-tree of processes using the MPI_COMM_SPLIT

    //                                          0

    //                                  0                   8

    //                              0       4           8          12

    //                           0    2   4   6     8     10    12    14

    //                          0 1  2 3 4 5 6 7  8   9  10 11 12 13 14 15

         for(levelno=1; levelno<iLevel; levelno++)

        {

            color = MPI_UNDEFINED;

            if( key % power_of_2[levelno] == 0)     //2 to some power

                //if level=1, key is 0, 2, 4, 6, 8, ...

            {

                color = key/power_of_2[levelno+1];  //finally, colr = rank / 2^(levelno+1)

                    //key= 0, 1, 2, 3, 4, 5, 6, 7, 8, ...      //in MPI_COMM_WORLD

                    //  = 0, 0, 0, 0, 1, 1, 1, 1, 2, ...  //in new communicator

            }

            MPI_Comm_split(MPI_COMM_WORLD, color, key, &subComm[levelno][key]);

        }

    for(loop=0; loop < 2; loop ++)

    {

        // level 0, all invoke

        MPI_Reduce(buff1, buff2,size,MPI_DOUBLE,MPI_SUM,0,subComm[levelno][my_rank]);

        for(levelno=0; levelno<iLevel; levelno++ )

        {

            if( rankIsInLevel(my_rank, power_of_2[levelno]))              //this process is in this level, i.e. my_rank %2^level == 0

                MPI_Reduce(buff1, buff2, size, MPI_DOUBLE, MPI_SUM,0, subComm[levelno][my_rank]);

        }//for

}

   The running time of different processes varies greatly, for example, for process 0, the time is 0.01 * 2 = 0.02 seconds, but for process 1, the time may be 0.38*2.

By the way, the machine is a linux cluster composed by Dell processors.

Thank you very much!

Xiren