On May 28, 2004, at 11:19 AM, Mike Arnold wrote:
> thanks Brian for your reply. Could I get you to please clarify the
> following:
>
> 1. MPI_BSend() returns when the buffer is free to use, but I'm still
> confused as to whether this also implies (in the lam implementation)
> that the sending process has completed with the communication. For
> example with a rendezvous protocol, that both the negotiation has been
> completed (the ack has been received from the receiving process), and
> the data has been passed to the OS to send, ie there is nothing more
> for the sending process to do. Am I correct in assuming that progress
> threads are not being used in lam?
A MPI_Bsend completing says nothing about the message's progress
towards being received. The message may not be completely sent until
the next call to an MPI function. This kind of makes Bsend a weird
function for single threaded MPI implementations like LAM.
> If the above is true, then for lam, MPI_Bsend should not offer any
> advantage in terms of apparent latency over MPI_Send?
No, not really. MPI_Bsend really doesn't make sense to use for a
point-to-point network. MPI_Bsend *requires* an extra memory copy on
the local node, so that never really never helps with performance.
Also, since it is a local operation, implementations such as LAM that
are single threaded aren't going to make much progress until the next
call into an MPI implementation.
Brian
> Brian W. Barrett wrote:
>
>> On Mon, 24 May 2004 mikea_at_[hidden] wrote:
>>> would somebody please be able to clarify for me the finer
>>> distinctions
>>> between the different send and receive calls (buffered and standard,
>>> blocking and non- blocking) in terms of the time taken by the calling
>>> process. That is, I'm not concerned about the time taken to complete
>>> the
>>> entire communication, but only the time that the calling process must
>>> spend inside the mpi calls. I'm aware of what the mpi standard
>>> states,
>>> but its not clear to me how it is interpreted within lam-mpi given
>>> that
>>> it is single-threaded and presumably does not have internal
>>> "progress"
>>> threads. My understanding of the lam documentation is that the
>>> communication only progresses when the calling process is inside an
>>> mpi
>>> call, hence some of the advantages of non-blocking and buffered
>>> disappear. Specifically I'm guessing that:
>>>
>>> 1. Buffering should ensure that send() can return as soon as the
>>> message
>>> is buffered, but with lam I presume that a buffered send() should
>>> take
>>> at least as long as a non-buffered send() to return, as in both cases
>>> the call can not return until the sending process has fully completed
>>> with the message.
>> So this is a really hard question to answer, but just some general
>> notes... MPI_Send and MPI_BSend both use an eager prototcol for
>> sending
>> short messages. After some point, a rendezvous protocol is used that
>> requires an ACK from the receiving side before sending the actual
>> data.
>> MPI_Ssend always uses the rendezvous protocol, and MPI_Isend does
>> something similar to MPI_Send and MPI_Bsend.
>> Note that just because a send has returned does *NOT* mean the
>> message has
>> been sent. For MPI_Send and MPI_Bsend, a return means that the
>> supplied
>> buffer can be reused by the user. For MPI_Ssend, a return means that
>> the
>> receiving side has *started* to receive the message. For MPI_Isend,
>> all
>> that the function returns means is that the send has started.
>>> 2. Similarly for lam, the combined time spent in a non-blocking
>>> send()
>>> and its matching wait() will be at least that spent in a blocking
>>> send(), regardless of the time interval between isend() and wait(),
>>> unless lam uses "progress" threads. The same applies for receive().
>> Not necessarily. It could be more, the same, or less. Which isn't
>> really
>> a useful answer, but let's look at the TCP case:
>> same: this is fairly trivial - you guessed how this would work...
>> less: remember that the OS does some buffering for us. Let's say
>> that we
>> are sending a rendezvous protocol send. During the ISend call, we
>> send
>> the RTS and return. During the Wait, we grab the CTS that is waiting
>> for
>> us already and start sending data. We manage to skip out on an entire
>> round trip for the RTS/CTS protocol. There are also some buffering
>> cases
>> that make Isends faster as well...
>> mori: If you have lots of Isends posted, you can actually slow things
>> down
>> in the bookkeeping as LAM tries to progress all the messages. This is
>> especially true on the shared memory protocols, if I recall correctly.
>>> 3. Using the lamd rpi module, will a standard blocking send() return
>>> once the message has been passed to the local daemon, or only when a
>>> matching receive has also been posted? I'm guessing the former, and
>>> that
>>> the above points also apply to lamd. I can see an advantage in using
>>> lamd, as communication with a process on the local node is cheaper
>>> than
>>> with one on a remote node, but will there be a difference whether the
>>> send is blocking or non-blocking, buffered or standard?
>> Remember that the standard talks about when it is safe to reuse the
>> buffer
>> more than anything. Since you can reuse the buffer as soon as the
>> message
>> is sent to the local lamd, the apparent latency for medium sized
>> messages
>> is often lower than the TCP rpi. However, the realized bandwidth
>> will be
>> much lower, so it isn't necessarily going to be a win.
>> Hope this helps...
>> Brian
>
> --
> -----------------------------------------------------------------------
> Computational Neurobiology Laboratory : Phone: 858 453-4100x1455
> The Salk Institute of Biological Studies : Fax: 858 587-0417
> 10010 N. Torrey Pines Rd, La Jolla, CA 92037 : Email: mikea_at_[hidden]
> : www.cnl.salk.edu/~mikea
> _______________________________________________
> This list is archived at http://www.lam-mpi.org/MailArchives/lam/
>
--
Brian Barrett
LAM/MPI developer and all around nice guy
Have a LAM/MPI day: http://www.lam-mpi.org/
|
