When you use dynamic windows, you must use the virtual address of the remote memory as the offset, That means you must attach a buffer and then get the address with MPI_GET_ADDRESS. Then you must share that address with any processes that target that memory, perhaps using MPI_SEND/MPI_RECV or MPI_ALLGATHER of an address-sized integer (MPI_AINT is the MPI type corresponding to the MPI_Aint C type). It appears you are not doing this. This issue should affect you whether you use automatic arrays or heap data... It does not appear to be a problem here, but if you use automatic arrays with RMA, you must guarentee that they remain in scope throughout the duration of when they will be accessed remotely. I think you are doing this sufficiently with a barrier. However, at the point at which you are calling barrier to ensure they stay in scope, you lose all of the benefits of fine-grain synchronization from PSCW. You might as well just use MPI_Win_fence. There is a sentence in the MPI spec that says that, strictly speaking, using memory not allocated by MPI_Alloc_mem (or MPI_Win_allocate(_shared), of course) in RMA is not portable, but I don't know any implementation that actually behaves this way. MPICH has an active-message implementation of RMA, which does not care what storage class is involved, up to performance differences (interprocess shared memory is faster in some cases). This is a fairly complicated topic and it is possible that I have been a bit crude in summarizing the MPI standard, so I apologize to any MPI Forum experts who can find fault in what I've written :-) Jeff On Tue, Apr 19, 2016 at 5:51 AM, Maciej Szpindler <[email protected]> wrote:

This is simplified version of my routine. It may look odd but I am trying to migrate from send/recv scheme to one sided pscw and that is the reason for buffers etc. As long as dynamic windows are not used, it works fine (I believe). When I switch to dynamic windows it fails with segmentation fault. I would appreciate any comment and suggestion how to improve this.

SUBROUTINE swap_simple_rma(field, row_length, rows, levels, halo_size)

USE mpi

IMPLICIT NONE

INTEGER, INTENT(IN) :: row_length INTEGER, INTENT(IN) :: rows INTEGER, INTENT(IN) :: levels INTEGER, INTENT(IN) :: halo_size REAL(KIND=8), INTENT(INOUT) :: field(1:row_length, 1:rows, levels) REAL(KIND=8) :: send_buffer(halo_size, rows, levels) REAL(KIND=8) :: recv_buffer(halo_size, rows, levels) INTEGER :: buffer_size INTEGER :: i,j,k INTEGER(kind=MPI_INTEGER_KIND) :: ierror INTEGER(kind=MPI_INTEGER_KIND) :: my_rank, comm_size Integer(kind=MPI_INTEGER_KIND) :: win, win_info Integer(kind=MPI_INTEGER_KIND) :: my_group, origin_group, target_group Integer(kind=MPI_INTEGER_KIND), DIMENSION(1) :: target_rank, origin_rank Integer(kind=MPI_ADDRESS_KIND) :: win_size, disp

CALL MPI_Comm_Rank(MPI_COMM_WORLD, my_rank, ierror) CALL MPI_Comm_Size(MPI_COMM_WORLD, comm_size, ierror)

buffer_size = halo_size * rows * levels

CALL MPI_Info_create(win_info, ierror) CALL MPI_Info_set(win_info, "no_locks", "true", ierror)

CALL MPI_Comm_group(MPI_COMM_WORLD, my_group, ierror)

If (my_rank /= comm_size - 1) Then origin_rank = my_rank + 1 CALL MPI_Group_incl(my_group, 1, origin_rank, origin_group, ierror) win_size = 8*buffer_size Else origin_group = MPI_GROUP_EMPTY win_size = 0 End If

CALL MPI_Win_create_dynamic(win_info, MPI_COMM_WORLD, win, ierror) !! CALL MPI_Win_create(recv_buffer, win_size, & !! 8, win_info, MPI_COMM_WORLD, win, ierror) CALL MPI_Win_attach(win, recv_buffer, win_size, ierror)

CALL MPI_Barrier(MPI_COMM_WORLD, ierror)

CALL MPI_Win_post(origin_group, MPI_MODE_NOSTORE, win, ierror)

! Prepare buffer DO k=1,levels DO j=1,rows DO i=1,halo_size send_buffer(i,j,k)=field(i,j,k) END DO ! I END DO ! J END DO ! K

If (my_rank /= 0 ) Then target_rank = my_rank - 1 CALL MPI_Group_incl(my_group, 1, target_rank, target_group, ierror) Else target_group = MPI_GROUP_EMPTY End If

CALL MPI_Win_start(target_group, 0, win, ierror)

disp = 0

If (my_rank /= 0) Then CALL MPI_Put(send_buffer, buffer_size, MPI_REAL8, & my_rank - 1, disp, buffer_size, MPI_REAL8, win, ierror) End If CALL MPI_Win_complete(win, ierror)

CALL MPI_Barrier(MPI_COMM_WORLD, ierror) write (0,*) 'Put OK' CALL MPI_Barrier(MPI_COMM_WORLD, ierror)

CALL MPI_Win_wait(win, ierror)

! Read from buffer If (my_rank /= comm_size -1 ) Then DO k=1,levels DO j=1,rows DO i=1,halo_size field(row_length+i,j,k) = recv_buffer(i,j,k) END DO END DO END DO End if

CALL MPI_Win_detach(win, recv_buffer, ierror) CALL MPI_Win_free(win, ierror)

END SUBROUTINE swap_simple_rma

Best Regards, Maciej

W dniu 14.04.2016 o 19:21, Thakur, Rajeev pisze:

After the Win_attach, did you add a barrier or some other form of

...

synchronization? The put shouldn’t happen before Win_attach returns.

Rajeev

On Apr 14, 2016, at 10:56 AM, Maciej Szpindler <[email protected]>

...

wrote:

Dear All,

I am trying to use dynamic RMA windows in fortran. In my case I would like to attach automatic array to dynamic window. The question is if it is correct and allowed in MPICH. I feel that it is not working, at least in cray-mpich/7.3.2.

I have a subroutine that use RMA windows:

SUBROUTINE foo(x, y, z , ...)

USE mpi ...

INTEGER, INTENT(IN) :: x, y, z REAL(KIND=8) :: buffer(x, y, z) INTEGER(kind=MPI_INTEGER_KIND) :: win_info, win, comm INTEGER(kind=MPI_INTEGER_KIND) :: buff_size ...

buff_size = x*y*z*8

CALL MPI_Info_create(win_info, ierror) CALL MPI_Info_set(win_info, "no_locks", "true", ierror)

CALL MPI_Win_create_dynamic(win_info, comm, win, ierror)

CALL MPI_Win_attach(win, buffer, buff_size, ierror)

...

This produces segmentation fault when MPI_Put is called on a window, while exactly the same routine code with static MPI_Win_create on buffer instead of create_dynamic+attach works fine. As far as I understand buffer is in this case "simply contiguous" in a sense of the MPI Standard. Any help would be appreciated!

Best Regards, Maciej _______________________________________________ discuss mailing list [email protected] To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

_______________________________________________ discuss mailing list [email protected] To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

_______________________________________________ discuss mailing list [email protected] To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

-- Jeff Hammond [email protected] http://jeffhammond.github.io/ _______________________________________________ discuss mailing list [email protected] To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss