On Nov 9, 2005, at 11:19 AM, Michael Lees wrote:
>> So posting a lot of receives is not going to dramatically increase
>> the
>> memory usage in LAM itself -- it's going to increase your process'
>> memory usage, because assumedly you have unique message buffers
>> associated with each of those requests.
>
> The system is a multi-threaded design and I'm trying to determine the
> limits of the design.
> ie., with x number of messages from the slaves the master thread won't
> be able to service its MPI recv thread fast enough and frequently
> enough
> to ensure all x*16 messages are received before the next x*16 messages
> are sent (ie., within 100 milliseconds).
>
> So really I'm interested in the CPU overhead associated with context
> switching, buffering etc.
> Presumably x does exist, there must be some number of messages which
> will overload the system so as to cause a 'backlog' in the MPI buffer?
It depends on which network you are using. If you're using TCP, we
rely on the OS's socket implementation to do all buffering for us.
All other networks (including shared memory) have flow control
implemented at the MPI layer because either the network doesn't
provide it, or doesn't provide enough information to enforce MPI
semantics.
For GM, for example, if we run out of GM resources, the sender will
simply stop sending until more resources become available (e.g., the
receiver starts receiving).
There's more detail to it than that, but you get the general idea.
Still, LAM will try to send and receive directly from the user's
buffer, so queueing up a million Isends will only cause a constant
amount of overhead per message (i.e., it's not dependent upon the
size of the message).
As for measuring CPU context switching, etc., getrusage and
gettimeofday are probably your best bets, but those in themselves are
system calls, so they're expensive (e.g., when using a low latency
network). Running under profiling tools can also be helpful here,
but that also slows performance down dramatically, so the information
cannot be taken as absolute (e.g., use the relative percentages as a
guide -- not an absolute "this is what will happen every time I run"
kind of measure).
>> You can't really monitor the internals of LAM/MPI itself, but you can
>> monitor the entire process with getrusage(). See the man page for
>> details.
>
> Is getrusage only milliseconds resolution? GetTimeOfDay seems to be
> better resolution but it doesn't do CPU time?
Yep. Aren't computers fun? :-)
--
{+} Jeff Squyres
{+} The Open MPI Project
{+} http://www.open-mpi.org/
|
