[ Please disable HTML content of your messages. Please wrap your lines
to less than 80 characters. ]
On Thu, 1 Apr 2004, Ross Torkington wrote:
> I've been timing how long it takes to send various sized messages
Send over what ? Fast Ethernet, Gigabit Ethernet, Myrinet (GM or IP
over GM), Infiniband, shared memory - these are all supported by LAM
(and sorry if I forgot some) and have quite different communication
paramaters.
> and found that the cost per kb begins high for short messages, drops
> sharply
This is quite counterintuitive. But without a good definition of
"short messages", it's still unclear. Latency tests usually start with
a MPI payload of zero bytes (which means a message that includes the
MPI envelope but no data). Have you included this in your tests ? If
you are talking about kilobyte sized messages and if you use Ethernet,
some effect of going over the MTU of the Ethernet interface might be
visible as the TCP streams have to be chunked at transmitter and
reassembled at receiver.
> and remains at around 50us for messages between 10 and 100kb
>From this, I assume that it is indeed Ethernet, but what kind ? Are
you using Jumbo frames ?
> but then increases in a staircase fashion beyond that.
If it's indeed TCP over Ethernet, most likely it's the network drivers
and/or TCP/IP stack that have a non-linear behaviour. For example,
recently some Linux network drivers have gotten "NAPI" capabilities
which means that at a high interrupt rate generated for receiving
packets, they switch to an interrupt-less mode where some polling is
done to detect whether or not new packets have arrived; if the load is
then reduced, it goes back to interrupt mode - this can kick in and
out without any control from user processes and can introduce serious
measuring artifacts. I'm not saying that this is what happened during
your tests, but this is just an example of how things can go out of
control.
--
Bogdan Costescu
IWR - Interdisziplinaeres Zentrum fuer Wissenschaftliches Rechnen
Universitaet Heidelberg, INF 368, D-69120 Heidelberg, GERMANY
Telephone: +49 6221 54 8869, Telefax: +49 6221 54 8868
E-mail: Bogdan.Costescu_at_[hidden]
|
