(renamed subject; was "Next generation LAM/MPI: Open MPI")
Very cool stuff!
Did you, perchance, ever make this a standalone library that others
could use?
On Sep 27, 2004, at 7:29 AM, daniel.egloff_at_[hidden] wrote:
> Jeff Squyres wrote:
>
> So any system that moves objects would have to either be layered
> on top of MPI or be a different system (e.g., CORBA and
> friends).
>
> It looks like that sending C++ objects around via MPI is a task which
> interests many MPI programmers. Often a system like Corba and friends
> does not provide the performance or is just to heavy because it comes
> with additional features that might not be needed. I would like to
> indicate how we approached this problem.
>
> We evaluated OOMPI and decided that it does not fullfill our needs
> (not all, but obviously some of them). The major problem is that we
> wanted to be able to "memory map" instances of classes, including
> heterogeneous containers (something that STL does not provide as such
> directly) and create a MPI memory map once the objects are avialable
> in memory. Next we wanted to send the whole object with a single
> MPI_send (blocking / nonblocking ...), no repeating send / receive for
> every member variable, no recursive descents down the object
> hierarchy, nor packing the object into a temp buffer. Because our
> application is sending terabytes of data around we really cannot
> affort any uncecessary copy operations. Last but not least we wanted
> to use the memory mapping feature to implement generic object
> persistency (target formats: database blobs, binary files, hdf5 files)
>
> However, to make things work the receiver of such an object needs to
> be able to create an "empty clone" of an object he is going to
> receive, which has the proper "size" to receive all the data from the
> sender.
>
> We separated the problem as follows: A "type map" and a "memory map".
> The overall procedure is described in the following steps:
>
> 1) Every "distributable" class needs to be able to produce a type map,
> which includes all the type AND size information to recreate a
> properly sized class from this string
>
> Example:
>
> class A
> {
> private
> vector<int> a;
> };
>
> The type map string "A{vector<int>[9]}" would correspond to an object
> of class A with 9 int values stored in A::a.
>
> 2) Every "distributable" class needs to be able to generate its memory
> map. We choose an abstract memory map independent of MPI: It is
> defined as
>
> struct MemoryMapDescr
> {
> MemoryMapDescr(DataType t, void* a, int l) : type(t), addr(a),
> len(l) {}
> DataType type;
> void* addr;
> int len;
> };
>
> std::vector<MemoryMapDescr>& memorymap;
>
> We then use a specific function to get from our memory map an MPI
> memory map.
>
> int createMPIDataType(std::vector<MemoryMapDescr>& memorymap,
> MPI_Datatype &datatype);
>
> Putting things together: we defined the interface "Distributable"
> which looks like this
>
> // create type map string, including all "size information"
> std::string typeMap();
>
> // create a memory map of a "properly sized" object
> std::vector<MemoryMapDescr>&
> memoryMap(std::vector<MemoryMapDescr>& memoryMap);
>
> 3) Finally, every class inheriting from "Distributable" must provide a
> constructor from a type map string:
>
> class A
> {
> A();
>
> A(std::string typemap);
> private
> vector<int> a;
> };
>
> A(std::string typemap)
> {
> // parse typemap, find the size value and resize A::a
> }
>
> The constructor A::A(std::string typemap) parses the type map string
> typemap and properly resizes
> the int vector A::a. To do this we used the Spirit parser from
> boost.org, which is a very nice piece of software.
>
> We implemented the factory pattern to map strings for class names to
> object constructor functors. With this tool at hand we can easily
> handle heterogeneous containers and pointer member variables.
>
> 4) Objects are now distributed with MPI as follows:
>
> The sender creates type map and memory map.
> Then it sends the type map string and aferwards the MPI_datatype
> constructed from the memory map.
> The reciever constructs from the received type ma string the object
> and then the memory map and MPI datatypeand posts a receive on that
> datatype.
>
> That's basically all.
>
> 5) Note that the whole type map and memory map business allows you to
> implement object persisentcy: Imaging the scenarion where a storage
> server opens a memory mapped file and maps an object directly into the
> memory mapped part of the file. Then MPI_recv directly stores the data
> into the file without copying.
>
> You can also use code like
>
> std::vector<MemoryMapDescr> memoryMap;
> MPI_Datatype datatype;
> obj->memoryMap(memorymap);
> createMPIDataType(datatype, MPI_Datatype &datatype);
> MPI_File fh;
> MPI_File_open(MPI_COMM_WORLD, "/tmp/mytestfile",
> /*MPI_MODE_CREATE | */MPI_MODE_RDWR, 0 /*info*/, &fh);
> MPI_File_set_atomicity (fh,1);
> MPI_File_read(fh, 0,1,datatype, &status);
> MPI_File_close(&fh);
>
> to store objects via MPI. A similar strategy also works for hdf5
> formats (which supports MPI directly).
>
> Hope that helps.
>
> Daniel Egloff
>
>
>
>
>
>
>
>
>
>
>>>>>> On Fri, 24 Sep 2004 14:22:34 -0400, Jeff Squyres
> <jsquyres_at_[hidden]> said:
> jsquyres> That's all fine and good, but it's just not
> jsquyres> part of MPI. :-) This would have to be
>
> I know that it is not MPI, but rather something that I wish
> would be there:)
>
> One problem with the C++ interface of MPI is that it was
> done before STL and as it stands today I can not see what it
> really adds over the C API. Sure namespaces and exceptions
> are used and it is object based, but why all the virtual
> functions in the interface? I would assume that a library
> builder of today would aggregate MPI objects rather that
> inheriting the interface. Furthermore, using MPI with data
> stored in a e.g. std::vector is awkward, and there are also
> sometimes odd data-types in the interface: the count
> argument is an int and not a size_t. In short, I think that
> the C++ interface could use an overhaul; maybe a mpi2
> namespace?
>
> jsquyres> something layered on top of MPI. Hence, your
> jsquyres> function will probably eventually resolve down
> jsquyres> to an MPI_SEND that will do the actual sending
> jsquyres> of the correct datatype. With a bit of
> jsquyres> cleverness, the function could probably be
> jsquyres> inlined, and there would be little performance
> jsquyres> overhead (although going through the iterators
> jsquyres> might take some time... you'd have to evaluate
> jsquyres> that for your own requirements/needs).
>
> Sending the data takes eons anyway, at least compared with a
> function call or going through iterators.
>
> jsquyres> The OOMPI project took a few steps in the
> jsquyres> direction of a class library for MPI several
> jsquyres> years ago (1996-ish?), but didn't do anything
> jsquyres> like this. Both the C++ language itself and
> jsquyres> the quality of compilers available have
> jsquyres> improved significantly since then.
>
> Yep I have seen it, but never gotten around to using it.
>
> [...]
>
> --
> Fredrik Hedman
>
>
> _______________________________________________
> This list is archived at http://www.lam-mpi.org/MailArchives/lam/
>
>
>
> ___________________________________________________________________
>
> Disclaimer:
>
>
> Diese Mitteilung ist nur fuer die Empfaengerin / den Empfaenger
> bestimmt.
>
> Fuer den Fall, dass sie von nichtberechtigten Personen empfangen wird,
> bitten wir diese hoeflich, die Mitteilung an die ZKB zurueckzusenden
> und anschliessend die Mitteilung mit allen Anhaengen sowie allfaellige
> Kopien zu vernichten bzw. zu loeschen. Der Gebrauch der Information
> ist verboten.
>
>
> This message is intended only for the named recipient and may contain
> confidential or privileged information.
>
> If you have received it in error, please advise the sender by return
> e-mail and delete this message and any attachments. Any unauthorised
> use or dissemination of this information is strictly prohibited.
>
> _______________________________________________
> This list is archived at http://www.lam-mpi.org/MailArchives/lam/
>
--
{+} Jeff Squyres
{+} jsquyres_at_[hidden]
{+} http://www.lam-mpi.org/
|
