Hi,
We are having similar problems with a 3C940 Gigabit LOM using the
sk98lin module. Testing with netpipe without modifying the module, the
max throughput attained was around 450 Mbps. Taking the idea suggested
by Jess Michelesen I've modified the module to have static interrupt
moderation and variable number of interrupts processed by second. The
best numbers I've obtained are a bandwidth of about 720 Mbps with 5000
interrupts per second. However it seems that the card has "magic
numbers" for message sizes where the bandwidth drops dramatically.
Does anyone have information on tunning this card on linux for a beowulf
type cluster ?
Thank you in advance for any help.
On Wed, 2004-04-14 at 11:01, jess michelsen wrote:
> Hi Andy!
>
> Seems like the problems, we initially had with the e1000. All problems,
> both latency/bandwidth and stability, were resolved by installing the
> newest Intel e1000 driver version. It is now 5.2.30.1. In our
> applications, the communications are ultimately controlled by the
> computations, so we can safely increase InterruptThrottleRate all the
> way. With NetPipe, we have recorded latencies as low as 16-32
> microseconds and bandwidth up to just below 900 Mbit/sec. We have gained
> a little bit of performance by setting both Initial and Absolute write
> delay to 32 microseconds, so several sends can be performed at one
> interrupt.
>
> Best regards, Jess Michelsen
>
>
> On Tue, 2004-04-13 at 18:44, Andy Manners wrote:
> > We have just installed a small gigabit ethernet cluster using Linux
> > 2.4.21 and an e1000 gigabit ethernet driver. Performance was woeful
> > using the default configuration with 250 usec. small message latencies
> > and only 200-300 Mbit/sec throughput for large messages. Some
> > experimentation with the driver parameters improved things somewhat to
> > 50usec. and 900 Mbit/sec. However, things do not look quite right and
> > the controller hangs for larger message sizes without interrupt
> > throttling.
> >
> > Can someone point us towards information on tuning gigabit ethernet on
> > linux for numerical cluster purposes? That is, where low latencies are
> > important and maximum throughput less so?
> >
> > In particular, is it possible to set minimum latencies for small
> > messages (blocking mpi calls) and coalescing for message sizes
> > exceeding the packet size?
> >
> > Pointers for parameters to prod and tune in the 2.4 kernel also
> > gratefully received.
> >
> > Apologies if the subject is slightly off topic.
> >
> > _______________________________________________
> > This list is archived at http://www.lam-mpi.org/MailArchives/lam/
>
>
> _______________________________________________
> This list is archived at http://www.lam-mpi.org/MailArchives/lam/
|
