April 2016 - discuss - mailman3.cels.anl.gov

Is it allowed to attach automatic array for remote access with MPI_Win_attach?
by Maciej Szpindler 14 Apr '16

14 Apr '16

Dear All, I am trying to use dynamic RMA windows in fortran. In my case I would like to attach automatic array to dynamic window. The question is if it is correct and allowed in MPICH. I feel that it is not working, at least in cray-mpich/7.3.2. I have a subroutine that use RMA windows: SUBROUTINE foo(x, y, z , ...) USE mpi ... INTEGER, INTENT(IN) :: x, y, z REAL(KIND=8) :: buffer(x, y, z) INTEGER(kind=MPI_INTEGER_KIND) :: win_info, win, comm INTEGER(kind=MPI_INTEGER_KIND) :: buff_size ... buff_size = x*y*z*8 CALL MPI_Info_create(win_info, ierror) CALL MPI_Info_set(win_info, "no_locks", "true", ierror) CALL MPI_Win_create_dynamic(win_info, comm, win, ierror) CALL MPI_Win_attach(win, buffer, buff_size, ierror) ... This produces segmentation fault when MPI_Put is called on a window, while exactly the same routine code with static MPI_Win_create on buffer instead of create_dynamic+attach works fine. As far as I understand buffer is in this case "simply contiguous" in a sense of the MPI Standard. Any help would be appreciated! Best Regards, Maciej _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Have constants available in shared library
by Erik Schnetter 14 Apr '16

14 Apr '16

I am one of the maintainers of the MPI wrapper package for Julia <https://github.com/JuliaParallel/MPI.jl>. This package provides Julia language bindings for MPI, essentially wrapping various MPI functions, comparable to what mpi4py <http://pythonhosted.org/mpi4py/> does for Python. It is in Julia straightforward to call C functions in a shared library. The MPI API is standardized, so that part is straightforward. What is not standardized are the values of various constants, such as e.g. MPI_INT, MPI_MAX, or MPI_ANY_TAG. We currently use a cmake function that builds and runs a small executable that outputs the (implementation-specific) values of these constants, and auto-generates Julia code that defines these constants. We do this when the Julia MPI package is installed or updated. This works fine, but I think there could be a better way. The HDF5 library <https://www.hdfgroup.org/HDF5/> puts global constants into the data section of their library, with names derived from the preprocessor constants (with a _g suffix, I believe). This makes it straightforward to access the values of these constants at run time. For example, there exists a constant H5T_NATIVE_INT_g. Could you add similar constants to the OpenMPI shared library? This would simplify wrapping the library. Thanks, -erik -- Erik Schnetter <schnetter(a)gmail.com> http://www.perimeterinstitute.ca/personal/eschnetter/ _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Race condition in MPI3 (Fortran, shared memory)
by Oden, Lena 13 Apr '16

13 Apr '16

Actually, afaik, Win_fence is not the right synchronization method for you approach. As Halim said, you should use passive-target synchronization here. Fence guarantees you that all RMA operations on win originating at a given process and started before the fence call will complete at that process before the fence call return - so it only synchronizes RMA calls - but not multiple accesses to the same are. However, in you example that does NOT avoid that two processes access a at the same time - that why you get your race-condition. Lena On Apr 13, 2016, at 3:08 PM, Halim Amer <aamer(a)anl.gov<mailto:[email protected]>> wrote: Win_sync can only be used within passive target epochs. You could replace the win_fences with win_locks. --Halim On 4/13/16 11:28 AM, C.Friedrich wrote: Thanks for the suggestion! Adding MPI_WIN_SYNC in this way: do i = 1,10000 call mpe_win_fence(a,0,Nwin,err) a = a + 1 call mpi_win_sync(Nwin,err) call mpe_win_fence(a,0,Nwin,err) enddo does not work. The compiled executable either issues the error Fatal error in MPI_Win_sync: Wrong synchronization of RMA calls , error stack: MPI_Win_sync(115)..: MPI_Win_sync(win=0xa0000000) failed MPID_Win_sync(1805): Wrong synchronization of RMA calls or simply dies (without writing the numbers). Perhaps I use it incorrectly... Christoph ------------------------------------------------------------------------------------------------ ------------------------------------------------------------------------------------------------ Forschungszentrum Juelich GmbH 52425 Juelich Sitz der Gesellschaft: Juelich Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498 Vorsitzender des Aufsichtsrats: MinDir Dr. Karl Eugen Huthmacher Geschaeftsfuehrung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender), Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt, Prof. Dr. Sebastian M. Schmidt ------------------------------------------------------------------------------------------------ ------------------------------------------------------------------------------------------------ _______________________________________________ discuss mailing list discuss(a)mpich.org<mailto:[email protected]> To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss _______________________________________________ discuss mailing list discuss(a)mpich.org<mailto:[email protected]> To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Race condition in MPI3 (Fortran, shared memory)
by Halim Amer 13 Apr '16

13 Apr '16

Win_sync can only be used within passive target epochs. You could replace the win_fences with win_locks. --Halim On 4/13/16 11:28 AM, C.Friedrich wrote: > Thanks for the suggestion! Adding MPI_WIN_SYNC in this way: > do i = 1,10000 > call mpe_win_fence(a,0,Nwin,err) > a = a + 1 > call mpi_win_sync(Nwin,err) > call mpe_win_fence(a,0,Nwin,err) > enddo > does not work. The compiled executable either issues the error > Fatal error in MPI_Win_sync: Wrong synchronization of RMA calls , error > stack: > MPI_Win_sync(115)..: MPI_Win_sync(win=0xa0000000) failed > MPID_Win_sync(1805): Wrong synchronization of RMA calls > or simply dies (without writing the numbers). Perhaps I use it > incorrectly... > > Christoph > > > > ------------------------------------------------------------------------------------------------ > > ------------------------------------------------------------------------------------------------ > > Forschungszentrum Juelich GmbH > 52425 Juelich > Sitz der Gesellschaft: Juelich > Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498 > Vorsitzender des Aufsichtsrats: MinDir Dr. Karl Eugen Huthmacher > Geschaeftsfuehrung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender), > Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt, > Prof. Dr. Sebastian M. Schmidt > ------------------------------------------------------------------------------------------------ > > ------------------------------------------------------------------------------------------------ > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Race condition in MPI3 (Fortran, shared memory)
by C.Friedrich 13 Apr '16

13 Apr '16

Thanks for the suggestion! Adding MPI_WIN_SYNC in this way: do i = 1,10000 call mpe_win_fence(a,0,Nwin,err) a = a + 1 call mpi_win_sync(Nwin,err) call mpe_win_fence(a,0,Nwin,err) enddo does not work. The compiled executable either issues the error Fatal error in MPI_Win_sync: Wrong synchronization of RMA calls , error stack: MPI_Win_sync(115)..: MPI_Win_sync(win=0xa0000000) failed MPID_Win_sync(1805): Wrong synchronization of RMA calls or simply dies (without writing the numbers). Perhaps I use it incorrectly... Christoph ------------------------------------------------------------------------------------------------ ------------------------------------------------------------------------------------------------ Forschungszentrum Juelich GmbH 52425 Juelich Sitz der Gesellschaft: Juelich Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498 Vorsitzender des Aufsichtsrats: MinDir Dr. Karl Eugen Huthmacher Geschaeftsfuehrung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender), Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt, Prof. Dr. Sebastian M. Schmidt ------------------------------------------------------------------------------------------------ ------------------------------------------------------------------------------------------------ _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Race condition in MPI3 (Fortran, shared memory)
by Halim Amer 13 Apr '16

13 Apr '16

Can you try adding MPI_Win_sync after the array updates. It should insure ordering and visibility of the load/store operations. --Halim www.mcs.anl.gov/~aamer On 4/13/16 10:43 AM, C.Friedrich wrote: > Sorry, there was a typo: a(*) should be integer rather than real(8). > (ifort accepts it, gfortran does not.) >> subroutine mpe_win_fence(a,assert,win,err) >> implicit none >> integer, intent(inout) :: a(*) >> integer, intent(in) :: assert,win >> integer, intent(out) :: err >> call mpi_win_fence(assert,win,err) >> end > > > > ------------------------------------------------------------------------------------------------ > > ------------------------------------------------------------------------------------------------ > > Forschungszentrum Juelich GmbH > 52425 Juelich > Sitz der Gesellschaft: Juelich > Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498 > Vorsitzender des Aufsichtsrats: MinDir Dr. Karl Eugen Huthmacher > Geschaeftsfuehrung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender), > Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt, > Prof. Dr. Sebastian M. Schmidt > ------------------------------------------------------------------------------------------------ > > ------------------------------------------------------------------------------------------------ > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Race condition in MPI3 (Fortran, shared memory)
by C.Friedrich 13 Apr '16

13 Apr '16

Sorry, there was a typo: a(*) should be integer rather than real(8). (ifort accepts it, gfortran does not.) > subroutine mpe_win_fence(a,assert,win,err) > implicit none > integer, intent(inout) :: a(*) > integer, intent(in) :: assert,win > integer, intent(out) :: err > call mpi_win_fence(assert,win,err) > end ------------------------------------------------------------------------------------------------ ------------------------------------------------------------------------------------------------ Forschungszentrum Juelich GmbH 52425 Juelich Sitz der Gesellschaft: Juelich Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498 Vorsitzender des Aufsichtsrats: MinDir Dr. Karl Eugen Huthmacher Geschaeftsfuehrung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender), Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt, Prof. Dr. Sebastian M. Schmidt ------------------------------------------------------------------------------------------------ ------------------------------------------------------------------------------------------------ _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Race condition in MPI3 (Fortran, shared memory)
by C.Friedrich 13 Apr '16

13 Apr '16

Hello all, I have a problem with race conditions when several processes on the same node calculate an array a(:) held in shared memory. I understand that in the case of Fortran one has to make sure that the call to MPI_WIN_FENCE is accompanied with a call to MPI_F_SYNC_REG(a) to make sure that the memory is updated. However, as the routine MPI_F_SYNC_REG does not seem to be contained in the library I have (MPICH 3.2), I have used a different approach, namely that I move MPI_WIN_FENCE into an external subroutine MPE_WIN_FENCE that contains the array a(:) as an additional argument. In this way, the Fortran compiler should be forced to guarantee memory consistency, to my understanding. See the following test case. The program allocates the integer array a(100) in shared memory. Then, each process does a = a + 1 10000 times. When running the program with two processes, the final result of the array should thus be 20000, 20000, ... In order to protect "a=a+1" I have added calls to MPE_WIN_FENCE on both sides. However, race conflicts seem to persist, because the result is something like 18442, 18883, 19126, ... I could not find anything appropriate in the archives. Thanks in advance Christoph --- program test use iso_c_binding implicit none interface mpe_win_fence subroutine mpe_win_fence(a,assert,win,err) implicit none integer, intent(inout) :: a(*) integer, intent(in) :: assert,win integer, intent(out) :: err end subroutine mpe_win_fence end interface include '../include/mpif.h' integer :: i,j integer :: err,disp integer :: Nsize,Nrank,Ncomm,Nwin integer(kind=mpi_address_kind) :: size type(c_ptr) :: aptr integer, pointer :: a(:) call mpi_init(err) call mpi_comm_split_type(mpi_comm_world,mpi_comm_type_shared,0, & mpi_info_null,Ncomm,err) call mpi_comm_size(Ncomm,Nsize,err) call mpi_comm_rank(Ncomm,Nrank,err) size = 0 if(Nrank.eq.0) size = 100 * 4 disp = 1 call mpi_win_allocate_shared(size,disp,mpi_info_null,Ncomm,aptr, & Nwin,err) if(Nrank.ne.0) then call mpi_win_shared_query(Nwin,0,size,disp,aptr,err) endif call c_f_pointer(aptr,a,(/100/)) if(Nrank.eq.0) a = 0 call mpe_win_fence(a,0,Nwin,err) do i = 1,10000 call mpe_win_fence(a,0,Nwin,err) a = a + 1 call mpe_win_fence(a,0,Nwin,err) enddo if(Nrank.eq.0) then write(*,*) a endif call mpi_win_free(Nwin,err) call mpi_comm_free(Ncomm,err) call mpi_finalize(err) end subroutine mpe_win_fence(a,assert,win,err) implicit none real(8), intent(inout) :: a(*) integer, intent(in) :: assert,win integer, intent(out) :: err call mpi_win_fence(assert,win,err) end ------------------------------------------------------------------------------------------------ ------------------------------------------------------------------------------------------------ Forschungszentrum Juelich GmbH 52425 Juelich Sitz der Gesellschaft: Juelich Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498 Vorsitzender des Aufsichtsrats: MinDir Dr. Karl Eugen Huthmacher Geschaeftsfuehrung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender), Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt, Prof. Dr. Sebastian M. Schmidt ------------------------------------------------------------------------------------------------ ------------------------------------------------------------------------------------------------ _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] MPICH 3.2 failing in MPI_Init
by Andrew Wood 08 Apr '16

08 Apr '16

That fixes it and the job now runs successfully. So it looks like the problem is associated with mpiexec 0.84. Thanks, Andy. On 07/04/16 18:52, Kenneth Raffenetti wrote: > It's probably missing support for your job launcher on the cluster. You can > provide a flag (--with-pbs=<install_dir>) to tell the build where to look. > > Ken > > On 04/07/2016 10:03 AM, Andrew Wood wrote: >> I've now tried using the MPICH-built version of mpiexec, but using our desktops >> as nodes, instead of our cluster. The job runs successfully in that case. >> >> Andy. >> >> On 07/04/16 16:59, Andrew Wood wrote: >>> Thanks for the response. >>> >>> Using the MPICH-built mpiexec gives this error message: >>> >>> ssh_askpass: exec(/usr/lib/ssh/ksshaskpass): No such file or directory >>> Host key verification failed. >>> >>> >>> which presumably means I'd have to do something with the ssh configuration on >>> the nodes. The same error occurs with MPICH 3.1.4, but like I said, MPICH 3.1.4 >>> works fine with mpiexec 0.84. >>> >>> Andy. >>> >>> >>> >>> On 07/04/16 15:55, Kenneth Raffenetti wrote: >>>> Just to be sure, can you use the mpiexec that is built/installed with MPICH >>>> 3.2? >>>> You mention mpiexec version 0.84 below, so that's the first thing I would try. >>>> >>>> Ken >>>> >>>> On 04/07/2016 05:49 AM, Andrew Wood wrote: >>>>> Hi, >>>>> >>>>> I'm trying to get MPICH 3.2 working on our cluster, but jobs are failing in >>>>> MPI_Init with the following output if they are run on two or more nodes (4 >>>>> processes per node): >>>>> >>>>> >>>>> Fatal error in MPI_Init: Other MPI error, error stack: >>>>> MPIR_Init_thread(490).................: >>>>> MPID_Init(201)........................: channel initialization failed >>>>> MPIDI_CH3_Init(93)....................: >>>>> MPID_nem_init(285)....................: >>>>> MPIDI_CH3I_Seg_commit(366)............: >>>>> MPIU_SHMW_Hnd_deserialize(324)........: >>>>> MPIU_SHMW_Seg_open(867)...............: >>>>> MPIU_SHMW_Seg_create_attach_templ(638): open failed - No such file or >>>>> directory >>>>> mpiexec: Error: handle_pmi: unknown cmd abort. >>>>> >>>>> >>>>> >>>>> The full output above only occurs intermittently. Sometimes only the last line >>>>> appears (job aborted before stderr is flushed?). >>>>> >>>>> >>>>> >>>>> Our cluster uses Torque 2.5.13 and Maui 3.3.1, and the jobs are launched with >>>>> mpiexec 0.84. >>>>> >>>>> >>>>> >>>>> I've configured MPICH as follows. >>>>> >>>>> ./configure --enable-error-checking=all --enable-error-messages=all >>>>> --enable-g=all --disable-fast --enable-check-compiler-flags >>>>> --enable-fortran=all >>>>> --enable-cxx --enable-romio --enable-debuginfo --enable-versioning >>>>> --enable-strict >>>>> >>>>> >>>>> >>>>> I've found the problem goes away if I include the option >>>>> '--enable-nemesis-dbg-nolocal', but presumably that could have an impact on >>>>> performance. >>>>> >>>>> >>>>> The problem doesn't occur with MPICH 3.1.4, configured with the same options. >>>>> >>>>> >>>>> I've found this message in the mailing list archives, reporting the same >>>>> problem >>>>> http://lists.mpich.org/pipermail/discuss/2015-December/004352.html >>>>> However, that was on a system using SLURM, and the replies suggest that the >>>>> problem was with SLURM rather than MPICH, and we're not using SLURM on our >>>>> system. >>>>> >>>>> Can anyone help? >>>>> >>>>> >>>>> Regards, >>>>> Andy. >>>>> >>>>> >>>> . >>>> >>> >>> >> >> > . > -- Dr Andrew Wood Fluid Gravity Engineering Ltd. 83 Market Street St Andrews Fife KY16 9NX Tel: +44 (0)1334 460805 Fax: +44 (0)1334 460813 Fluid Gravity Engineering Ltd is registered in the UK with registration number 1674369. The registered address is Fluid Gravity Engineering Ltd, Unit 1, The Old Coach House, 1 West Street, Emsworth, Hampshire, PO10 7DX.

1 0

Re: [mpich-discuss] MPICH 3.2 failing in MPI_Init
by Kenneth Raffenetti 07 Apr '16

07 Apr '16

It's probably missing support for your job launcher on the cluster. You can provide a flag (--with-pbs=<install_dir>) to tell the build where to look. Ken On 04/07/2016 10:03 AM, Andrew Wood wrote: > I've now tried using the MPICH-built version of mpiexec, but using our desktops > as nodes, instead of our cluster. The job runs successfully in that case. > > Andy. > > On 07/04/16 16:59, Andrew Wood wrote: >> Thanks for the response. >> >> Using the MPICH-built mpiexec gives this error message: >> >> ssh_askpass: exec(/usr/lib/ssh/ksshaskpass): No such file or directory >> Host key verification failed. >> >> >> which presumably means I'd have to do something with the ssh configuration on >> the nodes. The same error occurs with MPICH 3.1.4, but like I said, MPICH 3.1.4 >> works fine with mpiexec 0.84. >> >> Andy. >> >> >> >> On 07/04/16 15:55, Kenneth Raffenetti wrote: >>> Just to be sure, can you use the mpiexec that is built/installed with MPICH 3.2? >>> You mention mpiexec version 0.84 below, so that's the first thing I would try. >>> >>> Ken >>> >>> On 04/07/2016 05:49 AM, Andrew Wood wrote: >>>> Hi, >>>> >>>> I'm trying to get MPICH 3.2 working on our cluster, but jobs are failing in >>>> MPI_Init with the following output if they are run on two or more nodes (4 >>>> processes per node): >>>> >>>> >>>> Fatal error in MPI_Init: Other MPI error, error stack: >>>> MPIR_Init_thread(490).................: >>>> MPID_Init(201)........................: channel initialization failed >>>> MPIDI_CH3_Init(93)....................: >>>> MPID_nem_init(285)....................: >>>> MPIDI_CH3I_Seg_commit(366)............: >>>> MPIU_SHMW_Hnd_deserialize(324)........: >>>> MPIU_SHMW_Seg_open(867)...............: >>>> MPIU_SHMW_Seg_create_attach_templ(638): open failed - No such file or directory >>>> mpiexec: Error: handle_pmi: unknown cmd abort. >>>> >>>> >>>> >>>> The full output above only occurs intermittently. Sometimes only the last line >>>> appears (job aborted before stderr is flushed?). >>>> >>>> >>>> >>>> Our cluster uses Torque 2.5.13 and Maui 3.3.1, and the jobs are launched with >>>> mpiexec 0.84. >>>> >>>> >>>> >>>> I've configured MPICH as follows. >>>> >>>> ./configure --enable-error-checking=all --enable-error-messages=all >>>> --enable-g=all --disable-fast --enable-check-compiler-flags --enable-fortran=all >>>> --enable-cxx --enable-romio --enable-debuginfo --enable-versioning >>>> --enable-strict >>>> >>>> >>>> >>>> I've found the problem goes away if I include the option >>>> '--enable-nemesis-dbg-nolocal', but presumably that could have an impact on >>>> performance. >>>> >>>> >>>> The problem doesn't occur with MPICH 3.1.4, configured with the same options. >>>> >>>> >>>> I've found this message in the mailing list archives, reporting the same problem >>>> http://lists.mpich.org/pipermail/discuss/2015-December/004352.html >>>> However, that was on a system using SLURM, and the replies suggest that the >>>> problem was with SLURM rather than MPICH, and we're not using SLURM on our >>>> system. >>>> >>>> Can anyone help? >>>> >>>> >>>> Regards, >>>> Andy. >>>> >>>> >>> . >>> >> >> > > _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0