Currently it is more a design pattern that is used throughout our
application.
However, we plan to isolate the main features into a separate library
early next year.
At least we would like to provide a running example which delivers a
starting point
to integrate our approach into other MPI based C++ applications.
Daniel
Jeff Squyres
<jsquyres_at_lam-mpi An: General LAM/MPI mailing list <lam_at_[hidden]>
.org> Kopie:
Gesendet von: Thema: LAM: C++ objects and MPI
lam-bounces_at_lam-m
pi.org
27.09.04 15:57
Bitte antworten
an General
LAM/MPI mailing
list
(renamed subject; was "Next generation LAM/MPI: Open MPI")
Very cool stuff!
Did you, perchance, ever make this a standalone library that others
could use?
On Sep 27, 2004, at 7:29 AM, daniel.egloff_at_[hidden] wrote:
> Jeff Squyres wrote:
>
> So any system that moves objects would have to either be
layered
> on top of MPI or be a different system (e.g., CORBA and
> friends).
>
> It looks like that sending C++ objects around via MPI is a task
which
> interests many MPI programmers. Often a system like Corba and
friends
> does not provide the performance or is just to heavy because it
comes
> with additional features that might not be needed. I would like to
> indicate how we approached this problem.
>
> We evaluated OOMPI and decided that it does not fullfill our needs
> (not all, but obviously some of them). The major problem is that we
> wanted to be able to "memory map" instances of classes, including
> heterogeneous containers (something that STL does not provide as
such
> directly) and create a MPI memory map once the objects are avialable
> in memory. Next we wanted to send the whole object with a single
> MPI_send (blocking / nonblocking ...), no repeating send / receive
for
> every member variable, no recursive descents down the object
> hierarchy, nor packing the object into a temp buffer. Because our
> application is sending terabytes of data around we really cannot
> affort any uncecessary copy operations. Last but not least we wanted
> to use the memory mapping feature to implement generic object
> persistency (target formats: database blobs, binary files, hdf5
files)
>
> However, to make things work the receiver of such an object needs to
> be able to create an "empty clone" of an object he is going to
> receive, which has the proper "size" to receive all the data from
the
> sender.
>
> We separated the problem as follows: A "type map" and a "memory
map".
> The overall procedure is described in the following steps:
>
> 1) Every "distributable" class needs to be able to produce a type
map,
> which includes all the type AND size information to recreate a
> properly sized class from this string
>
> Example:
>
> class A
> {
> private
> vector<int> a;
> };
>
> The type map string "A{vector<int>[9]}" would correspond to an
object
> of class A with 9 int values stored in A::a.
>
> 2) Every "distributable" class needs to be able to generate its
memory
> map. We choose an abstract memory map independent of MPI: It is
> defined as
>
> struct MemoryMapDescr
> {
> MemoryMapDescr(DataType t, void* a, int l) : type(t), addr(a),
> len(l) {}
> DataType type;
> void* addr;
> int len;
> };
>
> std::vector<MemoryMapDescr>& memorymap;
>
> We then use a specific function to get from our memory map an MPI
> memory map.
>
> int createMPIDataType(std::vector<MemoryMapDescr>& memorymap,
> MPI_Datatype &datatype);
>
> Putting things together: we defined the interface "Distributable"
> which looks like this
>
> // create type map string, including all "size information"
> std::string typeMap();
>
> // create a memory map of a "properly sized" object
> std::vector<MemoryMapDescr>&
> memoryMap(std::vector<MemoryMapDescr>& memoryMap);
>
> 3) Finally, every class inheriting from "Distributable" must provide
a
> constructor from a type map string:
>
> class A
> {
> A();
>
> A(std::string typemap);
> private
> vector<int> a;
> };
>
> A(std::string typemap)
> {
> // parse typemap, find the size value and resize A::a
> }
>
> The constructor A::A(std::string typemap) parses the type map string
> typemap and properly resizes
> the int vector A::a. To do this we used the Spirit parser from
> boost.org, which is a very nice piece of software.
>
> We implemented the factory pattern to map strings for class names to
> object constructor functors. With this tool at hand we can easily
> handle heterogeneous containers and pointer member variables.
>
> 4) Objects are now distributed with MPI as follows:
>
> The sender creates type map and memory map.
> Then it sends the type map string and aferwards the MPI_datatype
> constructed from the memory map.
> The reciever constructs from the received type ma string the object
> and then the memory map and MPI datatypeand posts a receive on that
> datatype.
>
> That's basically all.
>
> 5) Note that the whole type map and memory map business allows you
to
> implement object persisentcy: Imaging the scenarion where a storage
> server opens a memory mapped file and maps an object directly into
the
> memory mapped part of the file. Then MPI_recv directly stores the
data
> into the file without copying.
>
> You can also use code like
>
> std::vector<MemoryMapDescr> memoryMap;
> MPI_Datatype datatype;
> obj->memoryMap(memorymap);
> createMPIDataType(datatype, MPI_Datatype &datatype);
> MPI_File fh;
> MPI_File_open(MPI_COMM_WORLD, "/tmp/mytestfile",
> /*MPI_MODE_CREATE | */MPI_MODE_RDWR, 0 /*info*/, &fh);
> MPI_File_set_atomicity (fh,1);
> MPI_File_read(fh, 0,1,datatype, &status);
> MPI_File_close(&fh);
>
> to store objects via MPI. A similar strategy also works for hdf5
> formats (which supports MPI directly).
>
> Hope that helps.
>
> Daniel Egloff
>
>
>
>
>
>
>
>
>
>
>>>>>> On Fri, 24 Sep 2004 14:22:34 -0400, Jeff Squyres
> <jsquyres_at_[hidden]> said:
> jsquyres> That's all fine and good, but it's just not
> jsquyres> part of MPI. :-) This would have to be
>
> I know that it is not MPI, but rather something that I wish
> would be there:)
>
> One problem with the C++ interface of MPI is that it was
> done before STL and as it stands today I can not see what it
> really adds over the C API. Sure namespaces and exceptions
> are used and it is object based, but why all the virtual
> functions in the interface? I would assume that a library
> builder of today would aggregate MPI objects rather that
> inheriting the interface. Furthermore, using MPI with data
> stored in a e.g. std::vector is awkward, and there are also
> sometimes odd data-types in the interface: the count
> argument is an int and not a size_t. In short, I think that
> the C++ interface could use an overhaul; maybe a mpi2
> namespace?
>
> jsquyres> something layered on top of MPI. Hence, your
> jsquyres> function will probably eventually resolve down
> jsquyres> to an MPI_SEND that will do the actual sending
> jsquyres> of the correct datatype. With a bit of
> jsquyres> cleverness, the function could probably be
> jsquyres> inlined, and there would be little performance
> jsquyres> overhead (although going through the iterators
> jsquyres> might take some time... you'd have to evaluate
> jsquyres> that for your own requirements/needs).
>
> Sending the data takes eons anyway, at least compared with a
> function call or going through iterators.
>
> jsquyres> The OOMPI project took a few steps in the
> jsquyres> direction of a class library for MPI several
> jsquyres> years ago (1996-ish?), but didn't do anything
> jsquyres> like this. Both the C++ language itself and
> jsquyres> the quality of compilers available have
> jsquyres> improved significantly since then.
>
> Yep I have seen it, but never gotten around to using it.
>
> [...]
>
> --
> Fredrik Hedman
>
>
> _______________________________________________
> This list is archived at http://www.lam-mpi.org/MailArchives/lam/
>
>
>
> ___________________________________________________________________
>
> Disclaimer:
>
>
> Diese Mitteilung ist nur fuer die Empfaengerin / den Empfaenger
> bestimmt.
>
> Fuer den Fall, dass sie von nichtberechtigten Personen empfangen
wird,
> bitten wir diese hoeflich, die Mitteilung an die ZKB zurueckzusenden
> und anschliessend die Mitteilung mit allen Anhaengen sowie
allfaellige
> Kopien zu vernichten bzw. zu loeschen. Der Gebrauch der Information
> ist verboten.
>
>
> This message is intended only for the named recipient and may
contain
> confidential or privileged information.
>
> If you have received it in error, please advise the sender by return
> e-mail and delete this message and any attachments. Any unauthorised
> use or dissemination of this information is strictly prohibited.
>
> _______________________________________________
> This list is archived at http://www.lam-mpi.org/MailArchives/lam/
>
--
{+} Jeff Squyres
{+} jsquyres_at_[hidden]
{+} http://www.lam-mpi.org/
_______________________________________________
This list is archived at http://www.lam-mpi.org/MailArchives/lam/
___________________________________________________________________
Disclaimer:
Diese Mitteilung ist nur fuer die Empfaengerin / den Empfaenger
bestimmt.
Fuer den Fall, dass sie von nichtberechtigten Personen empfangen wird,
bitten wir diese hoeflich, die Mitteilung an die ZKB zurueckzusenden
und anschliessend die Mitteilung mit allen Anhaengen sowie allfaellige
Kopien zu vernichten bzw. zu loeschen. Der Gebrauch der Information
ist verboten.
This message is intended only for the named recipient and may contain
confidential or privileged information.
If you have received it in error, please advise the sender by return
e-mail and delete this message and any attachments. Any unauthorised
use or dissemination of this information is strictly prohibited.
|
