Jeff Squyres wrote:
So any system that moves objects would have to either be layered
on top of MPI or be a different system (e.g., CORBA and
friends).
It looks like that sending C++ objects around via MPI is a task which
interests many MPI programmers. Often a system like Corba and friends
does not provide the performance or is just to heavy because it comes
with additional features that might not be needed. I would like to
indicate how we approached this problem.
We evaluated OOMPI and decided that it does not fullfill our needs
(not all, but obviously some of them). The major problem is that we
wanted to be able to "memory map" instances of classes, including
heterogeneous containers (something that STL does not provide as such
directly) and create a MPI memory map once the objects are avialable
in memory. Next we wanted to send the whole object with a single
MPI_send (blocking / nonblocking ...), no repeating send / receive for
every member variable, no recursive descents down the object
hierarchy, nor packing the object into a temp buffer. Because our
application is sending terabytes of data around we really cannot
affort any uncecessary copy operations. Last but not least we wanted
to use the memory mapping feature to implement generic object
persistency (target formats: database blobs, binary files, hdf5 files)
However, to make things work the receiver of such an object needs to
be able to create an "empty clone" of an object he is going to
receive, which has the proper "size" to receive all the data from the
sender.
We separated the problem as follows: A "type map" and a "memory map".
The overall procedure is described in the following steps:
1) Every "distributable" class needs to be able to produce a type map,
which includes all the type AND size information to recreate a
properly sized class from this string
Example:
class A
{
private
vector<int> a;
};
The type map string "A{vector<int>[9]}" would correspond to an object
of class A with 9 int values stored in A::a.
2) Every "distributable" class needs to be able to generate its memory
map. We choose an abstract memory map independent of MPI: It is
defined as
struct MemoryMapDescr
{
MemoryMapDescr(DataType t, void* a, int l) : type(t), addr(a),
len(l) {}
DataType type;
void* addr;
int len;
};
std::vector<MemoryMapDescr>& memorymap;
We then use a specific function to get from our memory map an MPI
memory map.
int createMPIDataType(std::vector<MemoryMapDescr>& memorymap,
MPI_Datatype &datatype);
Putting things together: we defined the interface "Distributable"
which looks like this
// create type map string, including all "size information"
std::string typeMap();
// create a memory map of a "properly sized" object
std::vector<MemoryMapDescr>&
memoryMap(std::vector<MemoryMapDescr>& memoryMap);
3) Finally, every class inheriting from "Distributable" must provide a
constructor from a type map string:
class A
{
A();
A(std::string typemap);
private
vector<int> a;
};
A(std::string typemap)
{
// parse typemap, find the size value and resize A::a
}
The constructor A::A(std::string typemap) parses the type map string
typemap and properly resizes
the int vector A::a. To do this we used the Spirit parser from
boost.org, which is a very nice piece of software.
We implemented the factory pattern to map strings for class names to
object constructor functors. With this tool at hand we can easily
handle heterogeneous containers and pointer member variables.
4) Objects are now distributed with MPI as follows:
The sender creates type map and memory map.
Then it sends the type map string and aferwards the MPI_datatype
constructed from the memory map.
The reciever constructs from the received type ma string the object
and then the memory map and MPI datatypeand posts a receive on that
datatype.
That's basically all.
5) Note that the whole type map and memory map business allows you to
implement object persisentcy: Imaging the scenarion where a storage
server opens a memory mapped file and maps an object directly into the
memory mapped part of the file. Then MPI_recv directly stores the data
into the file without copying.
You can also use code like
std::vector<MemoryMapDescr> memoryMap;
MPI_Datatype datatype;
obj->memoryMap(memorymap);
createMPIDataType(datatype, MPI_Datatype &datatype);
MPI_File fh;
MPI_File_open(MPI_COMM_WORLD, "/tmp/mytestfile",
/*MPI_MODE_CREATE | */MPI_MODE_RDWR, 0 /*info*/, &fh);
MPI_File_set_atomicity (fh,1);
MPI_File_read(fh, 0,1,datatype, &status);
MPI_File_close(&fh);
to store objects via MPI. A similar strategy also works for hdf5
formats (which supports MPI directly).
Hope that helps.
Daniel Egloff
>>>>> On Fri, 24 Sep 2004 14:22:34 -0400, Jeff Squyres
<jsquyres_at_[hidden]> said:
jsquyres> That's all fine and good, but it's just not
jsquyres> part of MPI. :-) This would have to be
I know that it is not MPI, but rather something that I wish
would be there:)
One problem with the C++ interface of MPI is that it was
done before STL and as it stands today I can not see what it
really adds over the C API. Sure namespaces and exceptions
are used and it is object based, but why all the virtual
functions in the interface? I would assume that a library
builder of today would aggregate MPI objects rather that
inheriting the interface. Furthermore, using MPI with data
stored in a e.g. std::vector is awkward, and there are also
sometimes odd data-types in the interface: the count
argument is an int and not a size_t. In short, I think that
the C++ interface could use an overhaul; maybe a mpi2
namespace?
jsquyres> something layered on top of MPI. Hence, your
jsquyres> function will probably eventually resolve down
jsquyres> to an MPI_SEND that will do the actual sending
jsquyres> of the correct datatype. With a bit of
jsquyres> cleverness, the function could probably be
jsquyres> inlined, and there would be little performance
jsquyres> overhead (although going through the iterators
jsquyres> might take some time... you'd have to evaluate
jsquyres> that for your own requirements/needs).
Sending the data takes eons anyway, at least compared with a
function call or going through iterators.
jsquyres> The OOMPI project took a few steps in the
jsquyres> direction of a class library for MPI several
jsquyres> years ago (1996-ish?), but didn't do anything
jsquyres> like this. Both the C++ language itself and
jsquyres> the quality of compilers available have
jsquyres> improved significantly since then.
Yep I have seen it, but never gotten around to using it.
[...]
--
Fredrik Hedman
_______________________________________________
This list is archived at http://www.lam-mpi.org/MailArchives/lam/
___________________________________________________________________
Disclaimer:
Diese Mitteilung ist nur fuer die Empfaengerin / den Empfaenger
bestimmt.
Fuer den Fall, dass sie von nichtberechtigten Personen empfangen wird,
bitten wir diese hoeflich, die Mitteilung an die ZKB zurueckzusenden
und anschliessend die Mitteilung mit allen Anhaengen sowie allfaellige
Kopien zu vernichten bzw. zu loeschen. Der Gebrauch der Information
ist verboten.
This message is intended only for the named recipient and may contain
confidential or privileged information.
If you have received it in error, please advise the sender by return
e-mail and delete this message and any attachments. Any unauthorised
use or dissemination of this information is strictly prohibited.
|
