On Nov 22, 2005, at 1:33 AM, Angel Tsankov wrote:
> I decided to transfer a single element at each step in the second case
> in order to measure the startup time. It can be seen that startup time
> (i.e. overhead) is longer than shared memory communication. If one can
> completely overlap communications with computations (as is the case
> with larger systems when I use Ethernet) then this could be another
> possible reason for the solver finishing work in less time when
> communication is performed over Ethernet, right?
You also can't forget that the OS's TCP stack may do some buffering and
asynchronous progress outside of the process. Hence, the writev() at
the bottom of MPI_Send may return before the message is actually sent.
With shared memory, progress only occurs when the MPI library is
actively working, so MPI_Send will not return until the memcpy() at the
bottom has completed copying the message to shared memory.
> Some more results in case someone is interested:
> Transferring 32*32 (=1K) doubles using shared memory takes 0.557143s.
Run a program like NetPIPE and check out the log-log plots of the
results -- you'll see the characteristic knee-bend plots showing that
sending times are generally flat until you reach a specific size (which
is different for every network transport) and then it grows more or
less in relation to the size of the message.
> It seems that if shared memory is used communication time grows
> (almost) linearly with amount of data. I must admit, that I believed
> this time is fairly constant.
This is not possible. We have to copy data into shared memory, which,
by definition, means that we have to copy each byte from
process-specific memory to shared memory. At the root of this is the
memcpy() function, which, although it is usually highly tuned and not
like the simplistic pseudocode shown below, can be abstracted as the
following operation:
for (i = 0; i < size; ++i) {
*(dest++) = *(src++);
}
Which is, by definition, O(n).
Make sense?
--
{+} Jeff Squyres
{+} The Open MPI Project
{+} http://www.open-mpi.org/
|
