Daniel,
We were able to get the performance fixes for subarray and darray
into the latest release of MPICH2 (1.0.2) that just went out a few seconds
ago. Please try the new version, and if you have problems, send mail to
mpich2-maint_at_mcs.anl.gov.
Regards,
Rajeev
> -----Original Message-----
> From: Rajeev Thakur [mailto:thakur_at_[hidden]]
> Sent: Thursday, June 09, 2005 12:33 PM
> To: 'daniel.egloff_at_[hidden]'
> Cc: 'lam_at_[hidden]'; 'trueter_at_[hidden]';
> 'michael.christian.gauckler_at_[hidden]'; 'til-zkb_at_[hidden]'; 'Rob Ross'
> Subject: RE: LAM: Performance of MPI-IO / ROMIO / HDF5
>
> Daniel,
> Are you using the subarray datatype? If so, there is a
> known performance problem with it if you are using MPICH2,
> but not with MPICH-1. So you can try using MPICH-1 for now,
> and it should work fine. The problem with MPICH2 will be
> fixed soon, but not in the upcoming release of MPICH2 1.0.2
> that will happen in the next day or so. There will perhaps be
> a separate release of ROMIO that will follow.
>
> I am not sure if the subarray type will work at all with LAM,
> because LAM (correctly) presents it as a first-class
> datatype, whereas the older version of ROMIO that is in LAM
> expects it to be built out of MPI-1 types internally. This
> will also be fixed soon.
>
> Rajeev
>
>
>
>
> > Date: Thu, 9 Jun 2005 08:20:32 +0200
> > From: daniel.egloff_at_[hidden]
> > Subject: LAM: Performance of MPI-IO / ROMIO / HDF5
> > To: jsquyres_at_[hidden], lam_at_[hidden]
> > Cc: trueter_at_[hidden], michael.christian.gauckler_at_[hidden],
> > til-zkb_at_[hidden]
> > Message-ID:
> >
> > <OFC312CFE7.027052BB-ONC125701B.001FEE91-C125701B.0022D734_at_[hidden]>
> > Content-Type: text/plain; charset=iso-8859-2
> >
> > Dear LAM list
> >
> > We have severe performance problems with MPI-IO / ROMIO on a 100 CPU
> > Linux Intel Cluster with GigE interconnects and a reasonable switch.
> >
> > We tried several setups:
> >
> > LAM / ROMIO with pvfs2 (has some limitations with MPI-2
> > datatypes such as subarrays in combination mit MPI_File_set_view)
> > MPICH2 / ROMIO with pvfs2
> >
> > Both setups face similar problems, most likely because the parallel
> > file io layer is ROMIO for both setups.
> >
> > I guess that our "problem" is rather standard:
> >
> > Simulation phase:
> > Write a 4 dimensional cube to a file.
> > The cube is generated by different processes in blocks,
> > The blocks are generated by slicing the cube along the last
> > dimension, i.e. the dimension with stride 1 (C array order)
> >
> > Aggregation phase:
> > Read back the cube from file, this time sliced along the first
> > dimension, i.e. the dimension with the larges stride. (C
> array order
> > Every process reads a slice which is now just a
> threedimensional
> > subcube and processes it, until no more slices are to be processed.
> > (fortuantely one slice fits entierly in memory)
> >
> > The dimensionality of the cube is 1000 x50x50x1000000 doubles = 2 x
> > 10^13 bytes or in the range of 20 Tera bytes.
> >
> > Questions:
> > -Is such a problem doable with MPI-IO? Our tests indicate that
> > the "data sieving optimizaiton" done by ROMIO
> > leads to an unacceptable performance degradation. A lot of
> > communication happends inside MPI,
> > probably done by ROMIO to do some "optimization"? The
> > theoretical IO bandwidth of our infrastructure is not
> really stressed.
> > - What IO bandwidth can we expect?
> > - Has the community some indications about possible IO
> > bandwidth?
> > - Has the community some experience with other setups, i.e.
> > other parallel file systems, or ev. commerical MPI implementations,
> > Lustre, GPFS....?
> > - Shall we do it entierly different, i.e. another
> memory layout,
> > or drop MPI-IO and do it "by hand" e.g. by sending the
> chunks of data
> > to suitable storage locations via MPI send/recv, thereby
> > bypassing MPI-IO / ROMIO?
> >
> > Best regards,
> >
> > Daniel Egloff
> > Z|rcher Kantonalbank, VFEF
> > Head Financial Computing
> > Josefstrasse 222, 8005 Z|rich
> > Tel. +41 (0) 44 292 45 33, Fax +41 (0) 44 292 45 93
> > Briefadresse: Postfach, 8010 Z|rich, http://www.zkb.ch
>
|
