Hi,
Sorry for the delay in replying to your post.
The high CPU usage in case of usysv is due to the fact that
synchronization of access to shared memory is implemented using spin
locks. Alternately, you can use the sysv rpi in which synchronization is
done using System V semaphores; CPU utilization should be *significantly*
lesser in this case.
I ran my version of the same test on a very similar setup (2 nodes, 2 cpus
each, 1GB memory running Solaris 8), and I see that tcp and usysv have
comparable execution times. For a message size of 100MB, I see that tcp
rpi takes approximately 40 seconds while usysv takes 41 seconds.
Approximate CPU usage values obtained from 'top' are given below:
tcp:
CPU states: 90.0% idle, 0.0% user, 10.0% kernel, 0.0% iowait, 0.0% swap
usysv:
CPU states: 0.0% idle, 90.0% user, 10.0% kernel, 0.0% iowait, 0.0% swap
Considering that you were seeing 50% cpu cycles being spent on iowait, it
is possible that there are other processes that are utilizing resources
heavily on those nodes. Try running your example program when the
machines are lightly loaded; you might see better results.
To find out the RPIs that were compiled to be used with your LAM
installation, use the 'laminfo' command in 7.0. To determine the default
RPI used in a run, you can do one of the following:
* Each rpi has an associated priority which can be overridden by using the
'-ssi rpi <name>' option to mpirun. The priority values for each rpi
can be found in the LAM user doc. So you can determine the default rpi
used by finding the highest priority rpi from among those that have been
compiled into your current LAM installation.
* You can turn on debugging info for the RPI SSI modules by using the
-ssi rpi_verbose level:X
flag to mpirun. X can take on any value in the range -1 to 50.
X = -1 (no debugging)
X = 0 (minimal debugging info)
X = 50 (maximum debugging info)
The debug messages will indicate the rpi being used.
Hope this helps.
--
Sriram Sankaran
email: ssankara_at_[hidden]
http://www.lam-mpi.org/
Thus spake Karl Hahn, on Jul 4:
>Date: Fri, 04 Jul 2003 11:18:15 +0200
>From: Karl Hahn <hahnk_at_[hidden]>
>Reply-To: hahnk_at_[hidden],
> General LAM/MPI mailing list <lam_at_[hidden]>
>To: General LAM/MPI mailing list <lam_at_[hidden]>
>Subject: LAM: tcp vs. usysv (do I have to care about that?)
>
>Hello,
>
>I have done some tests with LAM 7.0. My programm just sends
>large messages to slaves. Each slave returns the message
>(unchanged) to the master.
>With this little programm I tested the new LAM 7.0
>feature to give the RPI with mpirun.
>What really surprises me is that TCP seems so be faster
>than usysv on my Sun with two processors (SMP).
>
>2 LAM nodes on a 2 CPU Sun, 1.2 GB memory:
>
>1) tcp
>
> > mpirun -c 5 -ssi rpi tcp program
>Master sending message of size 100MB to 4 processes.
>Master received value from 1
>Master received value from 2
>Master received value from 3
>Master received value from 4
>Time: 125.16
>
>top shows that CPU state is about 50% iowait, 10% kernel, 40% idle
>
>2) usysv
>
> > mpirun -c 5 -ssi rpi usysv program
>Master sending message of size 100MB to 4 processes.
>Master received value from 1
>Master received value from 2
>Master received value from 3
>Master received value from 4
>Time: 208.538
>
>top shows that CPU state is about 80% user, 20% kernel, 0% idle
>
>
>Can anybody explain that? Why is tcp so much faster? Why is
>usysv using so much CPU (I just send / receive messages, doing
>nothing else which is CPU-bound)?
>
>By the way: what RPI is the default? Having a look at top when
>executing 'mpirun -c 5 program' I would say tcp. I haven't
>found that info in the mpirun manpage.
>
>Thanks for help,
>Charlie
>
>_______________________________________________
>This list is archived at http://www.lam-mpi.org/MailArchives/lam/
>
|
