Hello Pierre,
I've been doing work on "multiple NIC" per node
networks for awhile, see our work on FNNs:
http://aggregate.org/FNN/
As others have stated, using round-robin (mode 0) channel
bonding with GigE doesn't help, and probably slows
things down. If you have fast enough hardware (PCI-X),
and are talking to multiple destinations, the
balance-alb mode may help out when using GigE.
However, with current economics of small vs. large GigE
switches, it seems to still be beneficial to use multiple
GigE NICs per node for connectivity with an FNN, even
if you can't utilize all the bandwidth of the NICs due to
PCI and/or software limitations. Complicated topologies
with GigE doesn't apply to clusters of 24 nodes or less.
It's just amazing how cheap you can get a 24 port
GigE switch today. I can remember the day when... ;-)
I'm looking forward to playing with OpenMPI's multi-path
support. I am also still working on my own channel bonding
mode to better support FNNs and hide their complexities
from network layers 3 & higher.
It may be a race between OpenMPI and my "FNN routing"
bonding mode as to which code gets released first... I have
the advantage of only having a tiny bit of code to write...
but also the disadvantage that I have a dissertation
to write that is higher priority than "code".
I think we are all targeting SC2004 for release...
Now, I must disappear and get back to work on
that dissertation.
On Tue, 5 Oct 2004 19:11:44 +0200 (CEST), Pierre Valiron
<pierre.valiron_at_[hidden]> wrote:
> Hi,
>
> I am not surprised of poor performance improvements with 2 e1000 NICs.
>
> You already need a very good harware to feed 1 e1000 at full speed under
> TCP-IP (the theoretical limit is 120 MB/s, the usual limit is about 80-90
> MB/s with close to 100% CPU utilization on most platforms for real
> applications).
>
> Bonding multiple NICs should be more effective if they eat less CPU
> cycles. Has anybody some experience bonding Myrinet, Infiniband, etc,
> between powerful SMP 64-bit machines (quad Opterons or Power5) ?
>
> Another application of multi-NICs would be the support for cheaper
> topologies. For instance, it may prove very expensive to buy a flat switch
> over 64 nodes. Supporting multiple NICs might allow to interconnect
> several switches via the nodes themselves, or several nodes to a single
> switch port in a chained fashion, or even to build small rings with NO
> switches.
>
> For instance, assuming all nodes possess 2 NICs, one might create a
> cheap ring connecting
>
> node 1 to nodes N and 2
> node i to nodes i-1 and i+1
> node N to nodes N-1 and 1
>
> which might be usable provided the network is fast (Myrinet, Infiniband,
> etc) and N is small enough. This is the topology used by the SGI Altix 350
> (involving a maximum of 8 nodes) with excellent performances.
>
> I doubt this type of configuration is supported in present LAM/MPI.
>
> What is the future with Open-MPI ?
>
> Best.
> Pierre.
--
Tim Mattox - tmattox_at_[hidden] - http://homepage.mac.com/tmattox/
|
