Thank you Jeff and Halim, Halim, I have tried 3.1.4 but it does not return 0 (error) when the program is finished, e.g. for a helloworld program ================================================================== program hello include 'mpif.h' integer rank, size, ierror, tag, status(MPI_STATUS_SIZE) call MPI_INIT(ierror) call MPI_COMM_SIZE(MPI_COMM_WORLD, size, ierror) call MPI_COMM_RANK(MPI_COMM_WORLD, rank, ierror) print*, 'node', rank, ': Hello world' call MPI_FINALIZE(ierror) end ================================================================== Using MPI 3.1.rc4 ================================================================== [chiensh@cumulus test]$ which mpif90 ~/apps/mpich/3.1.rc4/bin/mpif90 [chiensh@cumulus test]$ srun -n 2 ./a.out node 1 : Hello world node 0 : Hello world [chiensh@cumulus test]$ ================================================================== Using MPI 3.1.4 ================================================================== [chiensh@cumulus test]$ which mpif90 ~/apps/mpich/3.1.4/bin/mpif90 [chiensh@cumulus test]$ srun -n 2 ./a.out node 1 : Hello world node 0 : Hello world 2016-01-11 14:24:25.968 (WARN ) [0xfff7ef48b10] 75532:ibm.runjob.client.Job: terminated by signal 11 2016-01-11 14:24:25.968 (WARN ) [0xfff7ef48b10] 75532:ibm.runjob.client.Job: abnormal termination by signal 11 from rank 1 ================================================================== Jeff, after I set PAMID_THREAD_MULTIPLE=1 and PAMID_ASYNC_PROGRESS=1, it seems to have some "improvement": nonblocking test can run for up to 4 processes sometime, but sometime it just get a "deadlock", (see below) ========================================================== [chiensh@cumulus coll.bak]$ srun --nodes=4 --ntasks-per-node=1 nonblocking MPIDI_Process.* verbose : 2 statistics : 1 contexts : 32 async_progress : 1 context_post : 1 pt2pt.limits application eager remote, local : 4097, 4097 short remote, local : 113, 113 internal eager remote, local : 4097, 4097 short remote, local : 113, 113 rma_pending : 1000 shmem_pt2pt : 1 disable_internal_eager_scale : 524288 optimized.collectives : 0 optimized.select_colls: 2 optimized.subcomms : 1 optimized.memory : 0 optimized.num_requests: 1 mpir_nbc : 1 numTasks : 4 mpi thread level : 'MPI_THREAD_SINGLE' MPIU_THREAD_GRANULARITY : 'per object' ASSERT_LEVEL : 0 MPICH_LIBDIR : not defined The following MPICH_* environment variables were specified: The following PAMID_* environment variables were specified: PAMID_STATISTICS=1 PAMID_ASYNC_PROGRESS=1 PAMID_THREAD_MULTIPLE=1 PAMID_VERBOSE=2 The following PAMI_* environment variables were specified: The following COMMAGENT_* environment variables were specified: The following MUSPI_* environment variables were specified: The following BG_* environment variables were specified: No errors ========================================================== [chiensh@cumulus coll.bak]$ srun --nodes=4 --ntasks-per-node=1 nonblocking MPIDI_Process.* verbose : 2 statistics : 1 contexts : 32 async_progress : 1 context_post : 1 pt2pt.limits application eager remote, local : 4097, 4097 short remote, local : 113, 113 internal eager remote, local : 4097, 4097 short remote, local : 113, 113 rma_pending : 1000 shmem_pt2pt : 1 disable_internal_eager_scale : 524288 optimized.collectives : 0 optimized.select_colls: 2 optimized.subcomms : 1 optimized.memory : 0 optimized.num_requests: 1 mpir_nbc : 1 numTasks : 4 mpi thread level : 'MPI_THREAD_SINGLE' MPIU_THREAD_GRANULARITY : 'per object' ASSERT_LEVEL : 0 MPICH_LIBDIR : not defined The following MPICH_* environment variables were specified: The following PAMID_* environment variables were specified: PAMID_STATISTICS=1 PAMID_ASYNC_PROGRESS=1 PAMID_THREAD_MULTIPLE=1 PAMID_VERBOSE=2 The following PAMI_* environment variables were specified: The following COMMAGENT_* environment variables were specified: The following MUSPI_* environment variables were specified: The following BG_* environment variables were specified: (never return from here) ========================================================== Thanks! Regards, Dominic On 11 Jan, 2016, at 12:08 pm, Halim Amer <[email protected]> wrote:
Dominic,
There were a bunch of fixes that went to PAMID since v3.1rc4. You could try a release from the 3.1 series (i.e. from 3.1 through 3.1.4).
Regards, --Halim
www.mcs.anl.gov/~aamer On 11 Jan, 2016, at 11:30 am, Jeff Hammond <[email protected]> wrote: I recall MPI-3 RMA on BGQ deadlocks if you set PAMID_THREAD_MULTIPLE (please see ALCF MPI docs to verify exact name), which is required for async progress.
ARMCI-MPI test suite is one good way to validate MPI-3 RMA is working.
Jeff
_______________________________________________ discuss mailing list [email protected] To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss