Hello Jeff and thanx for the feedback. I just figured out a solution today
(I m sure there are many other ways to do the trick) and as it turned out
the problem was with the mpi_type_vector resulting in a consecutive
placement
of the recieved data in the recieve buffer. Following your advice I read a
little about the mpi
datatypes and employed the mpi_type_struct to solve the problem. Things are
working now :)
here is a little piece of code I wrote to gather data from proccesses in a
3D topology and place them in
one big matrix at the root process
program gather_global
implicit none
!## input
integer, parameter :: nx=6, ny=3, nz=6,& ! global matrix dims
lenx=3, leny=3, lenz=3,& ! small
matrices dims
dims=3 ! 3D topology
integer i,k,tmp
!## mpi integers
integer err,nump,rank,size_of_integers,reorder,cartcomm
!## mpi datatypes
integer stride,vstride,hstride
integer, allocatable :: recvbuf(:,:,:),sendbuf(:,:,:),&
rcounts(:),displ(:),&
periods(:),coords(:),numprocs(:)
integer displs(2), blklen(2), t(2)
include "mpif.h"
!## allocate storage
allocate(recvbuf(nx,ny,nz), sendbuf(lenx,leny,lenz))
allocate(periods(dims), coords(dims), numprocs(dims))
numprocs=(/ 2,1,2 /)
periods=0; reorder=1
call mpi_init(err)
call mpi_comm_size(mpi_comm_world,nump,err)
!## create 3d topology
call mpi_cart_create(mpi_comm_world,dims,numprocs,periods,reorder,&
cartcomm,err)
call mpi_comm_rank(cartcomm,rank,err)
call mpi_cart_coords(cartcomm,rank,dims,coords,err)
allocate(rcounts(nump),displ(nump))
!## set the displacement array for mpi_gatherv based
!## on the process position within the topology
tmp=coords(1)+nx*coords(2)+nx*ny*coords(3)
call mpi_gather(tmp,1,mpi_integer,displ,1,mpi_integer,0,&
cartcomm,err)
rcounts=1
recvbuf=-1
!## fill in the local matrices
sendbuf=rank
call mpi_type_extent(mpi_integer,size_of_integers,err)
call mpi_type_vector(leny,lenx,nx,mpi_integer,stride,err)
call mpi_type_hvector(lenz,1,nx*ny*size_of_integers,stride,hstride)
t(1)=hstride
t(2)=mpi_ub
displs(1)=0
displs(2)=lenz*size_of_integers
blklen(1)=1
blklen(2)=1
call mpi_type_struct(2,blklen,displs,t,vstride,err)
call mpi_type_commit(vstride,err)
!## gather the matrices
call mpi_gatherv(sendbuf,lenx*leny*lenz,mpi_integer,&
recvbuf,&
rcounts,displ,vstride,&
0,cartcomm,err)
call mpi_type_free(vstride,err)
!## output
if(rank==0) then
do k=1,nz
print*, 'this is z level ',k
do i=1,ny
write(*,'(6i3)') recvbuf(1:nx,i,k)
end do
end do
end if
call mpi_finalize(err)
end program gather_global
here is the output
this is z level 1
0 0 0 2 2 2
0 0 0 2 2 2
0 0 0 2 2 2
this is z level 2
0 0 0 2 2 2
0 0 0 2 2 2
0 0 0 2 2 2
this is z level 3
0 0 0 2 2 2
0 0 0 2 2 2
0 0 0 2 2 2
this is z level 4
1 1 1 3 3 3
1 1 1 3 3 3
1 1 1 3 3 3
this is z level 5
1 1 1 3 3 3
1 1 1 3 3 3
1 1 1 3 3 3
this is z level 6
1 1 1 3 3 3
1 1 1 3 3 3
1 1 1 3 3 3
just what I wanted :).
----- Original Message -----
From: "Jeff Squyres" <jsquyres_at_[hidden]>
To: "General LAM/MPI mailing list" <lam_at_[hidden]>
Sent: Monday, April 18, 2005 5:55 PM
Subject: Re: LAM: MPI_gather for 3D topology
> This is because you're receiving the data as 25 consecutive MPI_INTEGERs,
> so MPI will fill them in consecutively. You need to receive them with an
> appropriate datatype such that MPI will fill them in in the right order
> (similar to how you're sending the "stride" type; you need to receive into
> an analogous type).
>
> Make sense?
>
> On Apr 16, 2005, at 11:08 AM, Khaled Al-Salem wrote:
>
>> Thanks Jeff for the quick response.
>>
>> my knowledge of mpi datatypes is actually limited to mpi_type_vector and
>> mpi_type_hvector. I'm sure there is more to datatypes than just that.
>> I'll have to look into it, I appreciate the advice.
>>
>> Here is a snippet of a test code that illustrates where I'm stuck right
>> now
>>
>> integer recvbuf(10,5), sendbuf(5,5), rcounts(2), displ(2)
>> include "mpif.h"
>> displ=(/ 0,1 /)
>> rcounts=1
>> ...
>> ...
>> sendbuf=rank
>>
>> call mpi_type_vector(5,5,5,mpi_integer,stride,err)
>> call mpi_type_commit(stride,err)
>> call mpi_gatherv(sendbuf,25,mpi_integer,&
>> recvbuf, &
>> rcounts,displ,stride, &
>> 0,mpi_comm_world,err)
>>
>> and here is the output
>>
>> 0 0 0 0 0 0 0 0 0 0
>> 0 0 0 0 0 0 0 0 0 0
>> 0 0 0 0 0 1 1 1 1 1
>> 1 1 1 1 1 1 1 1 1 1
>> 1 1 1 1 1 1 1 1 1 1
>>
>> any idea?
>>
>> ----- Original Message ----- From: "Jeff Squyres" <jsquyres_at_[hidden]>
>> To: "General LAM/MPI mailing list" <lam_at_[hidden]>
>> Sent: Friday, April 15, 2005 8:47 PM
>> Subject: Re: LAM: MPI_gather for 3D topology
>>
>>
>>> MPI_GATHER can be used to put data in this order, but you need to think
>>> about your data layout in memory, and structure your use datatypes and
>>> MPI_GATHER accordingly.
>>>
>>> Keep in mind that the datatype that you gather *from* does not need to
>>> be the same datatype that you gather *to*. Specifically, the datatypes
>>> provided on the root and non-root processes do not need to be the same.
>>> They must be equivalent -- essentially meaning that the resulting number
>>> of bytes and basic data elements are the same, but the layout in memory
>>> could be different.
>>>
>>> I realize that I'm not giving you much of an answer :-(, but datatypes
>>> are a rather complex issue; I would strongly suggest spending a little
>>> time with an MPI book or a tutorial to get at least the basics of
>>> complex datatypes. They can give much more insight and detail than I
>>> can in an e-mail.
>>>
>>> Hope that helps!
>>>
>>>
>>> On Apr 15, 2005, at 7:05 PM, Khaled Al Salem wrote:
>>>
>>>> Hello,
>>>> I wrote an f90 mpi code to solve a 3D problem. The topology is
>>>> obtained using mpi_cart_creat. At the end of the calculations I wish
>>>> to get the results gathered from all processors into one big 3
>>>> dimensional array that can be directly visualized without going
>>>> through post-processing. The problem is that mpi_gather doesn't
>>>> gather the results from the different proccessors as blocks.
>>>> for example, in 2D, if proc. 1 has the following
>>>>
>>>> 1 1 1
>>>> 1 1 1
>>>> 1 1 1
>>>>
>>>> and proc. 2
>>>>
>>>> 2 2 2
>>>> 2 2 2
>>>> 2 2 2
>>>>
>>>> then mpi_gather results in the following
>>>>
>>>> 1 1 1 1 1 1
>>>> 1 1 1 2 2 2
>>>> 2 2 2 2 2 2
>>>>
>>>> the results that I'm after would be
>>>>
>>>> 1 1 1 2 2 2
>>>> 1 1 1 2 2 2
>>>> 1 1 1 2 2 2
>>>>
>>>> is there an easy way to do this?
>>>> _______________________________________________
>>>> This list is archived at http://www.lam-mpi.org/MailArchives/lam/
>>>>
>>>
>>> --
>>> {+} Jeff Squyres
>>> {+} jsquyres_at_[hidden]
>>> {+} http://www.lam-mpi.org/
>>>
>>> _______________________________________________
>>> This list is archived at http://www.lam-mpi.org/MailArchives/lam/
>>>
>>
>> _______________________________________________
>> This list is archived at http://www.lam-mpi.org/MailArchives/lam/
>>
>
> --
> {+} Jeff Squyres
> {+} jsquyres_at_[hidden]
> {+} http://www.lam-mpi.org/
>
> _______________________________________________
> This list is archived at http://www.lam-mpi.org/MailArchives/lam/
>
>
|
