Hello Tim and All,
There are some network parameters which MAY help MPI jobs run faster over
TCP/IP.
First, for pre 2.4.x kernels (which doesn't apply to RH EL3_U3), the
default maximum packet size could be increased from 32K to 256K. 2.4.x
and later kernels can increase the maximum packet size automatically
in increments of 32K.
Second, there was some talk when GigE was first introduced of increasing
the maximum ethernet frame size from 1500 bytes to 9000 bytes (aka Jumbo
Frames). In the early days of slow GigE switches, this did help reduce
fragmentation, and improved throughput a good 90% (because clients were
talking longer before the switch switched). With modern "wire-speed"
switches, there is no switching delay, so jumbo and normal frame sizes
transfer at about the same rate. Jumbo frames DO reduce the computational
overhead of TCP/IP communication, cutting the amount of processor time
spent for the actual transmission in half.
As an example, let's say a program which uses a total CPU time of 7200
seconds transfers 100GB of data (this is a lot for such a short running
program). The actual transfer of data takes 850 seconds, and uses 250
seconds on the CPUs (assuming 3.0 GHz) for system time. This means the
actual computations took 6950 seconds of CPU time. Using jumbo frames,
the system time is halved to 125 seconds, so now the job takes 6950 + 125
= 7075 seconds, about 2% faster than with normal frames.
Also, if the program used non-blocking calls, then that 850 seconds to
transfer the data isn't really noticed, but if the calls are blocking,
then that 850 seconds adds to the elapsed time (7200 + 850 = 8050). If
the switching equipment is old, then that 850 seconds becomes 1700
seconds (7200 + 1700 = 8900), which is a good 10% slower.
If you are using a modern >2.4.x kernel, a modern LAM >7.0, then there
really isn't any tuning that can be done to improve performance (outside
of LAM), apart from jumbo frames, which the above example shows to be
quite a marginal improvement on modern "wire-speed" switches. If your
switches are old, then it might help, if the switches and network drivers
support jumbo frames.
On Wed, 8 Dec 2004, Tim Prince wrote:
> We have information that lam application performance on a RH EL3_U3 giganet
> cluster can be improved by more than 10% with kernel tuning parameters. Can
> someone provide pointers on how to do this?
------------------------------------------------------------
Anthony Ciani (aciani1_at_[hidden])
Computational Condensed Matter Physics
Department of Physics, University of Illinois, Chicago
http://ciani.phy.uic.edu/~tony
------------------------------------------------------------
|
