On Sun, 8 May 2005, Guanhua Yan wrote:
> My hunch is that network contention caused this problem but am not
> quite sure of the real reason behind it.
There can be _many_ reasons for delayed receiving and network
contention (vague definition, but I won't try to define it :-)) is one
of them. But you talk only about one receiver - is this a 1to1
communication and is there any other communication taking place at the
same time between other nodes ?
One thing that can influence quite a lot the latency is context
switches. There is activity on the system that needs some CPU time and
this will take the CPU away from the MPI process. For example, on a
Linux system, it's typical to have an ext3 filesystem access the disk
every several seconds, cron runs every 1 minute, etc. If these
deviations appear at regular times, it's most probably caused by
something like this.
If there is any other network traffic on the same network segment
(f.e. if you have only one network card that is used for both NFS and
MPI traffic), the Linux kernel network stack might be responsible for
such delays, especially as you are using data sizes that do not fit
into one Ethernet packet and require the kernel to split and then
assemble them.
If you are using SMP computers, even if the other CPU(s) are idle, the
processes can "jump" from one CPU to another, which require cache
flushes which induce delays.
I can go on... but the idea is that delays can appear. If you need a
guaranteed response time, you should look at a real-time capable OS,
buy a better communication hardware+software solution or maybe a large
SMP (where delays can still appear, but they won't depend on network
anymore...)
--
Bogdan Costescu
IWR - Interdisziplinaeres Zentrum fuer Wissenschaftliches Rechnen
Universitaet Heidelberg, INF 368, D-69120 Heidelberg, GERMANY
Telephone: +49 6221 54 8869, Telefax: +49 6221 54 8868
E-mail: Bogdan.Costescu_at_[hidden]
|
