Hi Jeff,
Thanks for your suggestion.
I did some more work on it yesterday and came up with a solution which
seems to work well and I think is safe....
I changed the inOut thread into two separate threads in and out.
The in thread looks something like...
So the out thread looks like this...
{
Semaphore.P();
Lock(mpimutex) //prevents more than one thread accessing mpi
MPI_ISend()
Unlock(mpimutex)
while(flag==0){
Lock(mpimutex)
MPI_Test(flag)
Unlock(mpimutex)
sched_yield()
}
}
and the in thread....
{
Lock(mpimutex)
MPI_Irecv()
Unlock(mpimutex)
while(flag==0){
Lock(mpimutex)
MPI_Test(flag)
Unlock(mpimutex)
sched_yield()
}
}
This way,
1. only one thread at a time is making and mpi call
2. The send thread only executes when signalled (which is everytime
something is put in the sendQ)
3. It's safe - deadlock won't occur on two processes sending as long as
each thread is given some scheduler time.
It's seems fairly efficient - I've also managed to fix the slowdown
problem now.
I don't know how the locking unlocking will affect performance?
Thanks for all your help.
ps. I don't know if this should be an faq, I'm sure quite a few people
will be using MPI in this way and there isn't much information about it.
Cheers
-Mike
Jeff Squyres wrote:
> On Aug 17, 2005, at 7:45 AM, Michael Lees wrote:
>
>
>>So I've been trying to think of a safe solution to the problem above. I
>>came up with
>>
>>inOutThread
>>{
>>MPI_IRecv
>>if(!outQ.empty())
>> MPI_ISend
>>
>>MPI_Waitany(index)
>>
>>if(index = recv)
>>inQ.push()
>>}
>>
>>This is incomplete - I think I'd have to cancel whichever of the Irecv
>>and Isend didn't complete at the beginning of each loop? Would this be
>>inefficient? The other option would be to check which of the operations
>>completed and post a new one.
>
>
> The latter would be better. Plus, you don't want to cancel the Isend
> -- you want to let it complete. Hence, you want to post a new Isend
> every time there's something in the outQ and then just complete them,
> but post a new Irecv every time it completes. Perhaps something like
> this:
>
> -----
> inOutThread(...) {
> MPI_Recv_init(recv_buffer, ..., &array_req[0]);
> MPI_Start(
> while (!done) {
> while (!outQ.empty()) {
> foo = outQ.pop();
> MPI_Isend(foo, ..., &array_req[some_free_index]);
> }
> do {
> MPI_Waitany(size, array_req, &index, &status);
> if (MPI_UNDEFINED != index) {
> if (0 == index) {
> inQ.push(recv_buffer);
> } else {
> // ...index/array management...
> }
> }
> }
> }
> }
> -----
>
> This is off the top of my head, so it's kinda rough (and not guaranteed
> to be 100% correct!). Clearly, too, you'll need to add some request
> array management in there (e.g., to find a free index in the array and
> to maintain the correct value of "size"). I made it loop to send until
> the outQ is drained -- an optimization that might help, but that adds
> some complexity to the request array management.
>
> This also assumes that your queue management stuff is
> thread-safe/concurrent (empty, push, pop).
>
> Make sense?
>
This message has been checked for viruses but the contents of an attachment
may still contain software viruses, which could damage your computer system:
you are advised to perform your own checks. Email communications with the
University of Nottingham may be monitored as permitted by UK legislation.
|
