January 2016 - discuss - mailman3.cels.anl.gov

Re: [mpich-discuss] MPICH 3.2 on BlueGene/Q
by Dominic Chien 11 Jan '16

11 Jan '16

Hi All, Here is an update: MPICH 3.1.3 is the last version that passed the nonblocking test, even without setting PAMID_THREAD_MULTIPLE. However, setting PAMID_ASYNC_PROGRESS=1 will cause error.(Abort(1) on node 7: 'locking' async progress not applicable...) [chiensh@cumulus coll.bak]$ which mpif90 ~/apps/mpich/3.1.3/bin/mpif90 [chiensh@cumulus coll.bak]$ make nonblocking CC nonblocking.o CCLD nonblocking [chiensh@cumulus coll.bak]$ srun -n 2 ./nonblocking No errors [chiensh@cumulus coll.bak]$ srun -n 4 ./nonblocking No errors [chiensh@cumulus coll.bak]$ srun -n 16 ./nonblocking No errors Thanks all! Regards, Dominic On 11 Jan, 2016, at 2:29 pm, Dominic Chien <chiensh.acrc(a)gmail.com> wrote: > Thank you Jeff and Halim, > > Halim, I have tried 3.1.4 but it does not return 0 (error) when the program is finished, e.g. for a helloworld program > ================================================================== > program hello > include 'mpif.h' > integer rank, size, ierror, tag, status(MPI_STATUS_SIZE) > > call MPI_INIT(ierror) > call MPI_COMM_SIZE(MPI_COMM_WORLD, size, ierror) > call MPI_COMM_RANK(MPI_COMM_WORLD, rank, ierror) > print*, 'node', rank, ': Hello world' > call MPI_FINALIZE(ierror) > end > ================================================================== > > Using MPI 3.1.rc4 > ================================================================== > [chiensh@cumulus test]$ which mpif90 > ~/apps/mpich/3.1.rc4/bin/mpif90 > [chiensh@cumulus test]$ srun -n 2 ./a.out > node 1 : Hello world > node 0 : Hello world > [chiensh@cumulus test]$ > ================================================================== > Using MPI 3.1.4 > ================================================================== > [chiensh@cumulus test]$ which mpif90 > ~/apps/mpich/3.1.4/bin/mpif90 > [chiensh@cumulus test]$ srun -n 2 ./a.out > node 1 : Hello world > node 0 : Hello world > 2016-01-11 14:24:25.968 (WARN ) [0xfff7ef48b10] 75532:ibm.runjob.client.Job: terminated by signal 11 > 2016-01-11 14:24:25.968 (WARN ) [0xfff7ef48b10] 75532:ibm.runjob.client.Job: abnormal termination by signal 11 from rank 1 > ================================================================== > > > Jeff, after I set PAMID_THREAD_MULTIPLE=1 and PAMID_ASYNC_PROGRESS=1, it seems to have some "improvement": nonblocking test can run for up to 4 processes sometime, but sometime it just get a "deadlock", (see below) > ========================================================== > [chiensh@cumulus coll.bak]$ srun --nodes=4 --ntasks-per-node=1 nonblocking > MPIDI_Process.* > verbose : 2 > statistics : 1 > contexts : 32 > async_progress : 1 > context_post : 1 > pt2pt.limits > application > eager > remote, local : 4097, 4097 > short > remote, local : 113, 113 > internal > eager > remote, local : 4097, 4097 > short > remote, local : 113, 113 > rma_pending : 1000 > shmem_pt2pt : 1 > disable_internal_eager_scale : 524288 > optimized.collectives : 0 > optimized.select_colls: 2 > optimized.subcomms : 1 > optimized.memory : 0 > optimized.num_requests: 1 > mpir_nbc : 1 > numTasks : 4 > mpi thread level : 'MPI_THREAD_SINGLE' > MPIU_THREAD_GRANULARITY : 'per object' > ASSERT_LEVEL : 0 > MPICH_LIBDIR : not defined > The following MPICH_* environment variables were specified: > The following PAMID_* environment variables were specified: > PAMID_STATISTICS=1 > PAMID_ASYNC_PROGRESS=1 > PAMID_THREAD_MULTIPLE=1 > PAMID_VERBOSE=2 > The following PAMI_* environment variables were specified: > The following COMMAGENT_* environment variables were specified: > The following MUSPI_* environment variables were specified: > The following BG_* environment variables were specified: > No errors > ========================================================== > [chiensh@cumulus coll.bak]$ srun --nodes=4 --ntasks-per-node=1 nonblocking > MPIDI_Process.* > verbose : 2 > statistics : 1 > contexts : 32 > async_progress : 1 > context_post : 1 > pt2pt.limits > application > eager > remote, local : 4097, 4097 > short > remote, local : 113, 113 > internal > eager > remote, local : 4097, 4097 > short > remote, local : 113, 113 > rma_pending : 1000 > shmem_pt2pt : 1 > disable_internal_eager_scale : 524288 > optimized.collectives : 0 > optimized.select_colls: 2 > optimized.subcomms : 1 > optimized.memory : 0 > optimized.num_requests: 1 > mpir_nbc : 1 > numTasks : 4 > mpi thread level : 'MPI_THREAD_SINGLE' > MPIU_THREAD_GRANULARITY : 'per object' > ASSERT_LEVEL : 0 > MPICH_LIBDIR : not defined > The following MPICH_* environment variables were specified: > The following PAMID_* environment variables were specified: > PAMID_STATISTICS=1 > PAMID_ASYNC_PROGRESS=1 > PAMID_THREAD_MULTIPLE=1 > PAMID_VERBOSE=2 > The following PAMI_* environment variables were specified: > The following COMMAGENT_* environment variables were specified: > The following MUSPI_* environment variables were specified: > The following BG_* environment variables were specified: > (never return from here) > ========================================================== > > Thanks! > > Regards, > Dominic > > > On 11 Jan, 2016, at 12:08 pm, Halim Amer <aamer(a)anl.gov> wrote: > >> Dominic, >> >> There were a bunch of fixes that went to PAMID since v3.1rc4. You could try a release from the 3.1 series (i.e. from 3.1 through 3.1.4). >> >> Regards, >> --Halim >> >> www.mcs.anl.gov/~aamer > On 11 Jan, 2016, at 11:30 am, Jeff Hammond <jeff.science(a)gmail.com> wrote: >> I recall MPI-3 RMA on BGQ deadlocks if you set PAMID_THREAD_MULTIPLE (please see ALCF MPI docs to verify exact name), which is required for async progress. >> >> ARMCI-MPI test suite is one good way to validate MPI-3 RMA is working. >> >> Jeff > _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] MPICH 3.2 on BlueGene/Q
by Dominic Chien 11 Jan '16

11 Jan '16

Thank you Jeff and Halim, Halim, I have tried 3.1.4 but it does not return 0 (error) when the program is finished, e.g. for a helloworld program ================================================================== program hello include 'mpif.h' integer rank, size, ierror, tag, status(MPI_STATUS_SIZE) call MPI_INIT(ierror) call MPI_COMM_SIZE(MPI_COMM_WORLD, size, ierror) call MPI_COMM_RANK(MPI_COMM_WORLD, rank, ierror) print*, 'node', rank, ': Hello world' call MPI_FINALIZE(ierror) end ================================================================== Using MPI 3.1.rc4 ================================================================== [chiensh@cumulus test]$ which mpif90 ~/apps/mpich/3.1.rc4/bin/mpif90 [chiensh@cumulus test]$ srun -n 2 ./a.out node 1 : Hello world node 0 : Hello world [chiensh@cumulus test]$ ================================================================== Using MPI 3.1.4 ================================================================== [chiensh@cumulus test]$ which mpif90 ~/apps/mpich/3.1.4/bin/mpif90 [chiensh@cumulus test]$ srun -n 2 ./a.out node 1 : Hello world node 0 : Hello world 2016-01-11 14:24:25.968 (WARN ) [0xfff7ef48b10] 75532:ibm.runjob.client.Job: terminated by signal 11 2016-01-11 14:24:25.968 (WARN ) [0xfff7ef48b10] 75532:ibm.runjob.client.Job: abnormal termination by signal 11 from rank 1 ================================================================== Jeff, after I set PAMID_THREAD_MULTIPLE=1 and PAMID_ASYNC_PROGRESS=1, it seems to have some "improvement": nonblocking test can run for up to 4 processes sometime, but sometime it just get a "deadlock", (see below) ========================================================== [chiensh@cumulus coll.bak]$ srun --nodes=4 --ntasks-per-node=1 nonblocking MPIDI_Process.* verbose : 2 statistics : 1 contexts : 32 async_progress : 1 context_post : 1 pt2pt.limits application eager remote, local : 4097, 4097 short remote, local : 113, 113 internal eager remote, local : 4097, 4097 short remote, local : 113, 113 rma_pending : 1000 shmem_pt2pt : 1 disable_internal_eager_scale : 524288 optimized.collectives : 0 optimized.select_colls: 2 optimized.subcomms : 1 optimized.memory : 0 optimized.num_requests: 1 mpir_nbc : 1 numTasks : 4 mpi thread level : 'MPI_THREAD_SINGLE' MPIU_THREAD_GRANULARITY : 'per object' ASSERT_LEVEL : 0 MPICH_LIBDIR : not defined The following MPICH_* environment variables were specified: The following PAMID_* environment variables were specified: PAMID_STATISTICS=1 PAMID_ASYNC_PROGRESS=1 PAMID_THREAD_MULTIPLE=1 PAMID_VERBOSE=2 The following PAMI_* environment variables were specified: The following COMMAGENT_* environment variables were specified: The following MUSPI_* environment variables were specified: The following BG_* environment variables were specified: No errors ========================================================== [chiensh@cumulus coll.bak]$ srun --nodes=4 --ntasks-per-node=1 nonblocking MPIDI_Process.* verbose : 2 statistics : 1 contexts : 32 async_progress : 1 context_post : 1 pt2pt.limits application eager remote, local : 4097, 4097 short remote, local : 113, 113 internal eager remote, local : 4097, 4097 short remote, local : 113, 113 rma_pending : 1000 shmem_pt2pt : 1 disable_internal_eager_scale : 524288 optimized.collectives : 0 optimized.select_colls: 2 optimized.subcomms : 1 optimized.memory : 0 optimized.num_requests: 1 mpir_nbc : 1 numTasks : 4 mpi thread level : 'MPI_THREAD_SINGLE' MPIU_THREAD_GRANULARITY : 'per object' ASSERT_LEVEL : 0 MPICH_LIBDIR : not defined The following MPICH_* environment variables were specified: The following PAMID_* environment variables were specified: PAMID_STATISTICS=1 PAMID_ASYNC_PROGRESS=1 PAMID_THREAD_MULTIPLE=1 PAMID_VERBOSE=2 The following PAMI_* environment variables were specified: The following COMMAGENT_* environment variables were specified: The following MUSPI_* environment variables were specified: The following BG_* environment variables were specified: (never return from here) ========================================================== Thanks! Regards, Dominic On 11 Jan, 2016, at 12:08 pm, Halim Amer <aamer(a)anl.gov> wrote: > Dominic, > > There were a bunch of fixes that went to PAMID since v3.1rc4. You could try a release from the 3.1 series (i.e. from 3.1 through 3.1.4). > > Regards, > --Halim > > www.mcs.anl.gov/~aamer On 11 Jan, 2016, at 11:30 am, Jeff Hammond <jeff.science(a)gmail.com> wrote: > I recall MPI-3 RMA on BGQ deadlocks if you set PAMID_THREAD_MULTIPLE (please see ALCF MPI docs to verify exact name), which is required for async progress. > > ARMCI-MPI test suite is one good way to validate MPI-3 RMA is working. > > Jeff _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] valgrind warning in MPIR_Process_status
by Jeff Hammond 10 Jan '16

10 Jan '16

To be explicit, the other thread is http://lists.mpich.org/pipermail/devel/2016-January/000677.html and it is indeed the same issue. I pushed dibanez-helper_fns-valgrind to mpich-review just now. https://trac.mpich.org/projects/mpich/ticket/2328 was created for tracking purposes. Jeff On Sun, Jan 10, 2016 at 5:04 PM, Jeff Hammond <jeff.science(a)gmail.com> wrote: > I think this is the same issue reported on the devel list yesterday. If it > is, Daniel Ibanez provided a patch that I've merged into my (completely > unofficial fork on github) and will push to the official MPICH review repo > tonight. > > If you want to try and/or verify Daniel's patch, see > github/jeffhammond/mpich. The branch you want should be obvious from the > name (I can't remember what it is off hand). > > Best, > > Jeff > > > On Sunday, January 10, 2016, VandeVondele Joost < > joost.vandevondele(a)mat.ethz.ch> wrote: > >> Hi, >> >> we upgraded from mpich 3.1.2 to 3.2 and our nightly builds under valgrind >> show a new valgrind warning, which I suppose originates from the mpi >> library, rather than our code (given the stack trace below and the fact >> that they are absent with 3.1): >> >> There are two slightly distinct traces (alltoall and alltoallv): >> >> ==30165== Conditional jump or move depends on uninitialised value(s) >> ==30165== at 0x6E00056: MPIR_Process_status (mpiimpl.h:4394) >> ==30165== by 0x6E00056: MPIC_Waitall (helper_fns.c:774) >> ==30165== by 0x6D5A853: MPIR_Alltoall_intra (alltoall.c:369) >> ==30165== by 0x6D5B63C: MPIR_Alltoall (alltoall.c:564) >> ==30165== by 0x6D5B6EF: MPIR_Alltoall_impl (alltoall.c:599) >> ==30165== by 0x6D5BF07: PMPI_Alltoall (alltoall.c:722) >> ==30165== by 0x6AFF0DC: PMPI_ALLTOALL (alltoallf.c:276) >> ==30165== by 0x1D04699: __message_passing_MOD_mp_alltoall_i22 >> >> ==30201== Conditional jump or move depends on uninitialised value(s) >> ==30201== at 0x6E00056: MPIR_Process_status (mpiimpl.h:4394) >> ==30201== by 0x6E00056: MPIC_Waitall (helper_fns.c:774) >> ==30201== by 0x6D5C4EA: MPIR_Alltoallv_intra (alltoallv.c:194) >> ==30201== by 0x6D5CEC9: MPIR_Alltoallv (alltoallv.c:339) >> ==30201== by 0x6D5CFAA: MPIR_Alltoallv_impl (alltoallv.c:376) >> ==30201== by 0x6D5D8CA: PMPI_Alltoallv (alltoallv.c:527) >> ==30201== by 0x6AFF13E: PMPI_ALLTOALLV (alltoallvf.c:276) >> ==30201== by 0x1CCE180: __message_passing_MOD_mp_alltoall_z22v >> (message_passing_z.f90:223) >> >> Is this issue already known/fixed ? >> >> I saw there was a somewhat related change (which is part of 3.2): >> >> >> https://trac.mpich.org/projects/mpich/changeset/be7c17b3177ccb0ede49761b44a… >> >> but maybe that change was not fully complete ? >> >> Best regards, >> >> Joost VandeVondele >> >> ------------------------------ >> Prof. Dr. Joost VandeVondele >> Nanoscale Simulations >> >> ETH Zurich >> HIT G 41.4 >> Wolfgang-Pauli-Strasse 27 >> 8093 Zurich, Switzerland >> >> Joost.VandeVondele(a)mat.ethz.ch >> http://www.nanosim.mat.ethz.ch/ >> Tel.: +41 (0)44 6338451 >> ------------------------------ >> > > > -- > Jeff Hammond > jeff.science(a)gmail.com > http://jeffhammond.github.io/ > -- Jeff Hammond jeff.science(a)gmail.com http://jeffhammond.github.io/ _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] MPICH 3.2 on BlueGene/Q
by Halim Amer 10 Jan '16

10 Jan '16

Dominic, There were a bunch of fixes that went to PAMID since v3.1rc4. You could try a release from the 3.1 series (i.e. from 3.1 through 3.1.4). Regards, --Halim www.mcs.anl.gov/~aamer On 1/10/16 9:04 PM, Dominic Chien wrote: > Hi Rob and Pavan, > > Thank you for your clarification. > > If the MPI-3 functionalities works correctly for version 3.1rc4 on BG/Q, I may not need version 3.2. Have all MPI-3 features being tested and verified on BGQ? I just found that 3.1rc4 hang like a "deadlock" for for nonblocking test when n >2 . (maybe I should open another ticket for this issue.) > > [chiensh@cumulus coll.bak]$ srun -n 2 nonblocking > No errors > [chiensh@cumulus coll.bak]$ srun -n 4 nonblocking > (never return) > > Thanks! > > Regards, > Dominic > > On 9 Jan, 2016, at 12:14 am, Rob Latham <robl(a)mcs.anl.gov> wrote: > >> >> >> On 01/08/2016 10:13 AM, Rob Latham wrote: >> >>> If there are 3.2 features you'd like to cherry pick into this fork, and >>> they don't touch the device interface, or depend on patches that do, >>> then we can definitely do that. >>> >>> If you are working on Argonne's Blue Gene, I built and installed this >>> fork to /soft/libraries/unsupported/mpich >>> >>> Let me know if you run into any problems. >> >> Oh, and as Pavan says, this is unsupported which means I'd like to know if you have problems, but I cannot promise that I'll be able to spend a lot of time debugging them. >> >> ==rob >> >> >>>> Regards, >>>> Dominic >>>> >>>> Here is the environment variables >>>> ==================================================================================== >>>> >>>> export CC=/opt/ibmcmp/vac/bg/12.1/bin/bgxlc_r >>>> export CXX=/opt/ibmcmp/vacpp/bg/12.1/bin/bgxlC_r >>>> export F77=/opt/ibmcmp/xlf/bg/14.1/bin/bgxlf_r >>>> export FC=/opt/ibmcmp/xlf/bg/14.1/bin/bgxlf90_r >>>> export >>>> AR=/bgsys/drivers/V1R2M0/ppc64/gnu-linux/powerpc64-bgq-linux/bin/ar >>>> export >>>> LD=/bgsys/drivers/V1R2M0/ppc64/gnu-linux/powerpc64-bgq-linux/bin/ld >>>> export >>>> RANLIB=/bgsys/drivers/V1R2M0/ppc64/gnu-linux/powerpc64-bgq-linux/bin/ranlib >>>> >>>> export MPICHLIB_CXXFLAGS="-qhot -qinline=800 -qflag=i:i -qsaveopt >>>> -qsuppress=1506-236" >>>> export MPICHLIB_CFLAGS=${MPICHLIB_CXXFLAGS} >>>> export MPICHLIB_FFLAGS=${MPICHLIB_CXXFLAGS} >>>> export MPICHLIB_F90FLAGS=${MPICHLIB_CXXFLAGS} >>>> ==================================================================================== >>>> >>>> >>>> Here is the configure >>>> ==================================================================================== >>>> >>>> ../configure --host=powerpc64-bgq-linux --with-device=pamid >>>> --with-file-system=gpfs:BGQ >>>> --with-bgq-install-dir=/bgsys/drivers/V1R2M0/ppc64 >>>> --with-pami=/bgsys/drivers/V1R2M0/ppc64/comm/sys >>>> --with-pami-include=/bgsys/drivers/V1R2M0/ppc64/comm/sys/include >>>> --with-pami-lib=/bgsys/drivers/V1R2M0/ppc64/comm/sys/lib >>>> --disable-wrapper-rpath --enable-fast=nochkmsg,notiming,O3 >>>> --with-assert-level=0 --disable-error-messages --disable-debuginfo >>>> --enable-thread-cs=per-object --with-atomic-primitives >>>> --enable-handle-allocation=tls --enable-refcount=lock-free >>>> --disable-predefined-refcount >>>> --with-cross-file=src/mpid/pamid/cross/bgq8 >>>> --prefix=/scratch/home/chiensh/apps/mpich/3.2.rc2 >>>> ==================================================================================== >>>> >>>> >>>> Here is the make log >>>> ==================================================================================== >>>> >>>> ... >>>> CC src/mpi/attr/lib_libmpi_la-attr_delete.lo >>>> "/scratch/home/chiensh/mpich/mpich-3.2rc2/src/mpid/pamid/include/mpidi_thread.h", >>>> line 64.9: 1506-358 (I) "MPIU_THREAD_CS_ENTER" is defined on line 80 >>>> of ../src/util/thread/mpiu_thread_multiple.h. >>>> "/scratch/home/chiensh/mpich/mpich-3.2rc2/src/mpid/pamid/include/mpidi_thread.h", >>>> line 65.9: 1506-358 (I) "MPIU_THREAD_CS_EXIT" is defined on line 81 of >>>> ../src/util/thread/mpiu_thread_multiple.h. >>>> "/scratch/home/chiensh/mpich/mpich-3.2rc2/src/mpid/pamid/include/mpidi_thread.h", >>>> line 66.9: 1506-358 (I) "MPIU_THREAD_CS_YIELD" is defined on line 82 >>>> of ../src/util/thread/mpiu_thread_multiple.h. >>>> "../src/include/mpiimpl.h", line 1184.5: 1506-046 (S) Syntax error. >>>> "../src/include/mpiimpl.h", line 1481.5: 1506-046 (S) Syntax error. >>>> "../src/include/mpiimpl.h", line 1636.5: 1506-046 (S) Syntax error. >>>> "../src/include/mpiimpl.h", line 2785.5: 1506-343 (S) Redeclaration of >>>> MPID_Send differs from previous declaration on line 2760 of >>>> "../src/include/mpiimpl.h". >>>> "../src/include/mpiimpl.h", line 2785.5: 1506-377 (I) The type "int" >>>> of parameter 2 differs from the previous type "long". >>>> "../src/include/mpiimpl.h", line 2884.5: 1506-343 (S) Redeclaration of >>>> MPID_Isend differs from previous declaration on line 2865 of >>>> "../src/include/mpiimpl.h". >>>> "../src/include/mpiimpl.h", line 2884.5: 1506-377 (I) The type "int" >>>> of parameter 2 differs from the previous type "long". >>>> "../src/include/mpitimpl.h", line 245.5: 1506-046 (S) Syntax error. >>>> "../src/include/mpitimpl.h", line 900.40: 1506-022 (S) "total" is not >>>> a member of "struct {...}". >>>> "/scratch/home/chiensh/mpich/mpich-3.2rc2/src/mpid/pamid/include/../src/mpid_request.h", >>>> line 156.21: 1506-022 (S) "cc" is not a member of "struct MPID_Request". >>>> "/scratch/home/chiensh/mpich/mpich-3.2rc2/src/mpid/pamid/include/../src/mpid_request.h", >>>> line 157.8: 1506-022 (S) "cc_ptr" is not a member of "struct >>>> MPID_Request". >>>> "/scratch/home/chiensh/mpich/mpich-3.2rc2/src/mpid/pamid/include/../src/mpid_request.h", >>>> line 157.23: 1506-022 (S) "cc" is not a member of "struct MPID_Request". >>>> "/scratch/home/chiensh/mpich/mpich-3.2rc2/src/mpid/pamid/include/../src/mpid_request.h", >>>> line 283.62: 1506-099 (S) Unexpected argument. >>>> "/scratch/home/chiensh/mpich/mpich-3.2rc2/src/mpid/pamid/include/../src/mpid_request.h", >>>> line 326.23: 1506-022 (S) "cc_ptr" is not a member of "struct >>>> MPID_Request". >>>> "/scratch/home/chiensh/mpich/mpich-3.2rc2/src/mpid/pamid/include/../src/mpid_request.h", >>>> line 341.23: 1506-022 (S) "cc_ptr" is not a member of "struct >>>> MPID_Request". >>>> "/scratch/home/chiensh/mpich/mpich-3.2rc2/src/mpid/pamid/include/mpidpost.h", >>>> line 56.44: 1506-022 (S) "vcr" is not a member of "struct MPID_Comm". >>>> "/scratch/home/chiensh/mpich/mpich-3.2rc2/src/mpid/pamid/include/mpidpost.h", >>>> line 83.43: 1506-022 (S) "vcrt" is not a member of "struct MPID_Comm". >>>> "/scratch/home/chiensh/mpich/mpich-3.2rc2/src/mpid/pamid/include/mpidpost.h", >>>> line 84.37: 1506-022 (S) "vcrt" is not a member of "struct MPID_Comm". >>>> "/scratch/home/chiensh/mpich/mpich-3.2rc2/src/mpid/pamid/include/mpidpost.h", >>>> line 84.57: 1506-022 (S) "vcr" is not a member of "struct MPID_Comm". >>>> "/scratch/home/chiensh/mpich/mpich-3.2rc2/src/mpid/pamid/include/mpidpost.h", >>>> line 91.42: 1506-022 (S) "vcr" is not a member of "struct MPID_Comm". >>>> "/scratch/home/chiensh/mpich/mpich-3.2rc2/src/mpid/pamid/include/mpidpost.h", >>>> line 92.41: 1506-022 (S) "vcr" is not a member of "struct MPID_Comm". >>>> "../src/mpi/attr/attr_delete.c", line 65.26: 1506-045 (S) Undeclared >>>> identifier GLOBAL. >>>> make[2]: *** [src/mpi/attr/lib_libmpi_la-attr_delete.lo] Error 1 >>>> make[2]: Leaving directory >>>> `/scratch/home/chiensh/mpich/mpich-3.2rc2/build' >>>> make[1]: *** [all-recursive] Error 1 >>>> make[1]: Leaving directory >>>> `/scratch/home/chiensh/mpich/mpich-3.2rc2/build' >>>> make: *** [all] Error 2 >>>> ==================================================================================== >>>> >>>> _______________________________________________ >>>> discuss mailing list discuss(a)mpich.org >>>> To manage subscription options or unsubscribe: >>>> https://lists.mpich.org/mailman/listinfo/discuss >>>> >>> _______________________________________________ >>> discuss mailing list discuss(a)mpich.org >>> To manage subscription options or unsubscribe: >>> https://lists.mpich.org/mailman/listinfo/discuss >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] MPICH 3.2 on BlueGene/Q
by Jeff Hammond 10 Jan '16

10 Jan '16

I recall MPI-3 RMA on BGQ deadlocks if you set PAMID_THREAD_MULTIPLE (please see ALCF MPI docs to verify exact name), which is required for async progress. ARMCI-MPI test suite is one good way to validate MPI-3 RMA is working. Jeff On Sunday, January 10, 2016, Dominic Chien <chiensh.acrc(a)gmail.com> wrote: > Hi Rob and Pavan, > > Thank you for your clarification. > > If the MPI-3 functionalities works correctly for version 3.1rc4 on BG/Q, > I may not need version 3.2. Have all MPI-3 features being tested and > verified on BGQ? I just found that 3.1rc4 hang like a "deadlock" for for > nonblocking test when n >2 . (maybe I should open another ticket for this > issue.) > > [chiensh@cumulus coll.bak]$ srun -n 2 nonblocking > No errors > [chiensh@cumulus coll.bak]$ srun -n 4 nonblocking > (never return) > > Thanks! > > Regards, > Dominic > > On 9 Jan, 2016, at 12:14 am, Rob Latham <robl(a)mcs.anl.gov <javascript:;>> > wrote: > > > > > > > On 01/08/2016 10:13 AM, Rob Latham wrote: > > > >> If there are 3.2 features you'd like to cherry pick into this fork, and > >> they don't touch the device interface, or depend on patches that do, > >> then we can definitely do that. > >> > >> If you are working on Argonne's Blue Gene, I built and installed this > >> fork to /soft/libraries/unsupported/mpich > >> > >> Let me know if you run into any problems. > > > > Oh, and as Pavan says, this is unsupported which means I'd like to know > if you have problems, but I cannot promise that I'll be able to spend a lot > of time debugging them. > > > > ==rob > > > > > >>> Regards, > >>> Dominic > >>> > >>> Here is the environment variables > >>> > ==================================================================================== > >>> > >>> export CC=/opt/ibmcmp/vac/bg/12.1/bin/bgxlc_r > >>> export CXX=/opt/ibmcmp/vacpp/bg/12.1/bin/bgxlC_r > >>> export F77=/opt/ibmcmp/xlf/bg/14.1/bin/bgxlf_r > >>> export FC=/opt/ibmcmp/xlf/bg/14.1/bin/bgxlf90_r > >>> export > >>> AR=/bgsys/drivers/V1R2M0/ppc64/gnu-linux/powerpc64-bgq-linux/bin/ar > >>> export > >>> LD=/bgsys/drivers/V1R2M0/ppc64/gnu-linux/powerpc64-bgq-linux/bin/ld > >>> export > >>> > RANLIB=/bgsys/drivers/V1R2M0/ppc64/gnu-linux/powerpc64-bgq-linux/bin/ranlib > >>> > >>> export MPICHLIB_CXXFLAGS="-qhot -qinline=800 -qflag=i:i -qsaveopt > >>> -qsuppress=1506-236" > >>> export MPICHLIB_CFLAGS=${MPICHLIB_CXXFLAGS} > >>> export MPICHLIB_FFLAGS=${MPICHLIB_CXXFLAGS} > >>> export MPICHLIB_F90FLAGS=${MPICHLIB_CXXFLAGS} > >>> > ==================================================================================== > >>> > >>> > >>> Here is the configure > >>> > ==================================================================================== > >>> > >>> ../configure --host=powerpc64-bgq-linux --with-device=pamid > >>> --with-file-system=gpfs:BGQ > >>> --with-bgq-install-dir=/bgsys/drivers/V1R2M0/ppc64 > >>> --with-pami=/bgsys/drivers/V1R2M0/ppc64/comm/sys > >>> --with-pami-include=/bgsys/drivers/V1R2M0/ppc64/comm/sys/include > >>> --with-pami-lib=/bgsys/drivers/V1R2M0/ppc64/comm/sys/lib > >>> --disable-wrapper-rpath --enable-fast=nochkmsg,notiming,O3 > >>> --with-assert-level=0 --disable-error-messages --disable-debuginfo > >>> --enable-thread-cs=per-object --with-atomic-primitives > >>> --enable-handle-allocation=tls --enable-refcount=lock-free > >>> --disable-predefined-refcount > >>> --with-cross-file=src/mpid/pamid/cross/bgq8 > >>> --prefix=/scratch/home/chiensh/apps/mpich/3.2.rc2 > >>> > ==================================================================================== > >>> > >>> > >>> Here is the make log > >>> > ==================================================================================== > >>> > >>> ... > >>> CC src/mpi/attr/lib_libmpi_la-attr_delete.lo > >>> > "/scratch/home/chiensh/mpich/mpich-3.2rc2/src/mpid/pamid/include/mpidi_thread.h", > >>> line 64.9: 1506-358 (I) "MPIU_THREAD_CS_ENTER" is defined on line 80 > >>> of ../src/util/thread/mpiu_thread_multiple.h. > >>> > "/scratch/home/chiensh/mpich/mpich-3.2rc2/src/mpid/pamid/include/mpidi_thread.h", > >>> line 65.9: 1506-358 (I) "MPIU_THREAD_CS_EXIT" is defined on line 81 of > >>> ../src/util/thread/mpiu_thread_multiple.h. > >>> > "/scratch/home/chiensh/mpich/mpich-3.2rc2/src/mpid/pamid/include/mpidi_thread.h", > >>> line 66.9: 1506-358 (I) "MPIU_THREAD_CS_YIELD" is defined on line 82 > >>> of ../src/util/thread/mpiu_thread_multiple.h. > >>> "../src/include/mpiimpl.h", line 1184.5: 1506-046 (S) Syntax error. > >>> "../src/include/mpiimpl.h", line 1481.5: 1506-046 (S) Syntax error. > >>> "../src/include/mpiimpl.h", line 1636.5: 1506-046 (S) Syntax error. > >>> "../src/include/mpiimpl.h", line 2785.5: 1506-343 (S) Redeclaration of > >>> MPID_Send differs from previous declaration on line 2760 of > >>> "../src/include/mpiimpl.h". > >>> "../src/include/mpiimpl.h", line 2785.5: 1506-377 (I) The type "int" > >>> of parameter 2 differs from the previous type "long". > >>> "../src/include/mpiimpl.h", line 2884.5: 1506-343 (S) Redeclaration of > >>> MPID_Isend differs from previous declaration on line 2865 of > >>> "../src/include/mpiimpl.h". > >>> "../src/include/mpiimpl.h", line 2884.5: 1506-377 (I) The type "int" > >>> of parameter 2 differs from the previous type "long". > >>> "../src/include/mpitimpl.h", line 245.5: 1506-046 (S) Syntax error. > >>> "../src/include/mpitimpl.h", line 900.40: 1506-022 (S) "total" is not > >>> a member of "struct {...}". > >>> > "/scratch/home/chiensh/mpich/mpich-3.2rc2/src/mpid/pamid/include/../src/mpid_request.h", > >>> line 156.21: 1506-022 (S) "cc" is not a member of "struct > MPID_Request". > >>> > "/scratch/home/chiensh/mpich/mpich-3.2rc2/src/mpid/pamid/include/../src/mpid_request.h", > >>> line 157.8: 1506-022 (S) "cc_ptr" is not a member of "struct > >>> MPID_Request". > >>> > "/scratch/home/chiensh/mpich/mpich-3.2rc2/src/mpid/pamid/include/../src/mpid_request.h", > >>> line 157.23: 1506-022 (S) "cc" is not a member of "struct > MPID_Request". > >>> > "/scratch/home/chiensh/mpich/mpich-3.2rc2/src/mpid/pamid/include/../src/mpid_request.h", > >>> line 283.62: 1506-099 (S) Unexpected argument. > >>> > "/scratch/home/chiensh/mpich/mpich-3.2rc2/src/mpid/pamid/include/../src/mpid_request.h", > >>> line 326.23: 1506-022 (S) "cc_ptr" is not a member of "struct > >>> MPID_Request". > >>> > "/scratch/home/chiensh/mpich/mpich-3.2rc2/src/mpid/pamid/include/../src/mpid_request.h", > >>> line 341.23: 1506-022 (S) "cc_ptr" is not a member of "struct > >>> MPID_Request". > >>> > "/scratch/home/chiensh/mpich/mpich-3.2rc2/src/mpid/pamid/include/mpidpost.h", > >>> line 56.44: 1506-022 (S) "vcr" is not a member of "struct MPID_Comm". > >>> > "/scratch/home/chiensh/mpich/mpich-3.2rc2/src/mpid/pamid/include/mpidpost.h", > >>> line 83.43: 1506-022 (S) "vcrt" is not a member of "struct MPID_Comm". > >>> > "/scratch/home/chiensh/mpich/mpich-3.2rc2/src/mpid/pamid/include/mpidpost.h", > >>> line 84.37: 1506-022 (S) "vcrt" is not a member of "struct MPID_Comm". > >>> > "/scratch/home/chiensh/mpich/mpich-3.2rc2/src/mpid/pamid/include/mpidpost.h", > >>> line 84.57: 1506-022 (S) "vcr" is not a member of "struct MPID_Comm". > >>> > "/scratch/home/chiensh/mpich/mpich-3.2rc2/src/mpid/pamid/include/mpidpost.h", > >>> line 91.42: 1506-022 (S) "vcr" is not a member of "struct MPID_Comm". > >>> > "/scratch/home/chiensh/mpich/mpich-3.2rc2/src/mpid/pamid/include/mpidpost.h", > >>> line 92.41: 1506-022 (S) "vcr" is not a member of "struct MPID_Comm". > >>> "../src/mpi/attr/attr_delete.c", line 65.26: 1506-045 (S) Undeclared > >>> identifier GLOBAL. > >>> make[2]: *** [src/mpi/attr/lib_libmpi_la-attr_delete.lo] Error 1 > >>> make[2]: Leaving directory > >>> `/scratch/home/chiensh/mpich/mpich-3.2rc2/build' > >>> make[1]: *** [all-recursive] Error 1 > >>> make[1]: Leaving directory > >>> `/scratch/home/chiensh/mpich/mpich-3.2rc2/build' > >>> make: *** [all] Error 2 > >>> > ==================================================================================== > >>> > >>> _______________________________________________ > >>> discuss mailing list discuss(a)mpich.org <javascript:;> > >>> To manage subscription options or unsubscribe: > >>> https://lists.mpich.org/mailman/listinfo/discuss > >>> > >> _______________________________________________ > >> discuss mailing list discuss(a)mpich.org <javascript:;> > >> To manage subscription options or unsubscribe: > >> https://lists.mpich.org/mailman/listinfo/discuss > > _______________________________________________ > > discuss mailing list discuss(a)mpich.org <javascript:;> > > To manage subscription options or unsubscribe: > > https://lists.mpich.org/mailman/listinfo/discuss > > _______________________________________________ > discuss mailing list discuss(a)mpich.org <javascript:;> > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > -- Jeff Hammond jeff.science(a)gmail.com http://jeffhammond.github.io/ _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] MPICH 3.2 on BlueGene/Q
by Dominic Chien 10 Jan '16

10 Jan '16

Hi Rob and Pavan, Thank you for your clarification. If the MPI-3 functionalities works correctly for version 3.1rc4 on BG/Q, I may not need version 3.2. Have all MPI-3 features being tested and verified on BGQ? I just found that 3.1rc4 hang like a "deadlock" for for nonblocking test when n >2 . (maybe I should open another ticket for this issue.) [chiensh@cumulus coll.bak]$ srun -n 2 nonblocking No errors [chiensh@cumulus coll.bak]$ srun -n 4 nonblocking (never return) Thanks! Regards, Dominic On 9 Jan, 2016, at 12:14 am, Rob Latham <robl(a)mcs.anl.gov> wrote: > > > On 01/08/2016 10:13 AM, Rob Latham wrote: > >> If there are 3.2 features you'd like to cherry pick into this fork, and >> they don't touch the device interface, or depend on patches that do, >> then we can definitely do that. >> >> If you are working on Argonne's Blue Gene, I built and installed this >> fork to /soft/libraries/unsupported/mpich >> >> Let me know if you run into any problems. > > Oh, and as Pavan says, this is unsupported which means I'd like to know if you have problems, but I cannot promise that I'll be able to spend a lot of time debugging them. > > ==rob > > >>> Regards, >>> Dominic >>> >>> Here is the environment variables >>> ==================================================================================== >>> >>> export CC=/opt/ibmcmp/vac/bg/12.1/bin/bgxlc_r >>> export CXX=/opt/ibmcmp/vacpp/bg/12.1/bin/bgxlC_r >>> export F77=/opt/ibmcmp/xlf/bg/14.1/bin/bgxlf_r >>> export FC=/opt/ibmcmp/xlf/bg/14.1/bin/bgxlf90_r >>> export >>> AR=/bgsys/drivers/V1R2M0/ppc64/gnu-linux/powerpc64-bgq-linux/bin/ar >>> export >>> LD=/bgsys/drivers/V1R2M0/ppc64/gnu-linux/powerpc64-bgq-linux/bin/ld >>> export >>> RANLIB=/bgsys/drivers/V1R2M0/ppc64/gnu-linux/powerpc64-bgq-linux/bin/ranlib >>> >>> export MPICHLIB_CXXFLAGS="-qhot -qinline=800 -qflag=i:i -qsaveopt >>> -qsuppress=1506-236" >>> export MPICHLIB_CFLAGS=${MPICHLIB_CXXFLAGS} >>> export MPICHLIB_FFLAGS=${MPICHLIB_CXXFLAGS} >>> export MPICHLIB_F90FLAGS=${MPICHLIB_CXXFLAGS} >>> ==================================================================================== >>> >>> >>> Here is the configure >>> ==================================================================================== >>> >>> ../configure --host=powerpc64-bgq-linux --with-device=pamid >>> --with-file-system=gpfs:BGQ >>> --with-bgq-install-dir=/bgsys/drivers/V1R2M0/ppc64 >>> --with-pami=/bgsys/drivers/V1R2M0/ppc64/comm/sys >>> --with-pami-include=/bgsys/drivers/V1R2M0/ppc64/comm/sys/include >>> --with-pami-lib=/bgsys/drivers/V1R2M0/ppc64/comm/sys/lib >>> --disable-wrapper-rpath --enable-fast=nochkmsg,notiming,O3 >>> --with-assert-level=0 --disable-error-messages --disable-debuginfo >>> --enable-thread-cs=per-object --with-atomic-primitives >>> --enable-handle-allocation=tls --enable-refcount=lock-free >>> --disable-predefined-refcount >>> --with-cross-file=src/mpid/pamid/cross/bgq8 >>> --prefix=/scratch/home/chiensh/apps/mpich/3.2.rc2 >>> ==================================================================================== >>> >>> >>> Here is the make log >>> ==================================================================================== >>> >>> ... >>> CC src/mpi/attr/lib_libmpi_la-attr_delete.lo >>> "/scratch/home/chiensh/mpich/mpich-3.2rc2/src/mpid/pamid/include/mpidi_thread.h", >>> line 64.9: 1506-358 (I) "MPIU_THREAD_CS_ENTER" is defined on line 80 >>> of ../src/util/thread/mpiu_thread_multiple.h. >>> "/scratch/home/chiensh/mpich/mpich-3.2rc2/src/mpid/pamid/include/mpidi_thread.h", >>> line 65.9: 1506-358 (I) "MPIU_THREAD_CS_EXIT" is defined on line 81 of >>> ../src/util/thread/mpiu_thread_multiple.h. >>> "/scratch/home/chiensh/mpich/mpich-3.2rc2/src/mpid/pamid/include/mpidi_thread.h", >>> line 66.9: 1506-358 (I) "MPIU_THREAD_CS_YIELD" is defined on line 82 >>> of ../src/util/thread/mpiu_thread_multiple.h. >>> "../src/include/mpiimpl.h", line 1184.5: 1506-046 (S) Syntax error. >>> "../src/include/mpiimpl.h", line 1481.5: 1506-046 (S) Syntax error. >>> "../src/include/mpiimpl.h", line 1636.5: 1506-046 (S) Syntax error. >>> "../src/include/mpiimpl.h", line 2785.5: 1506-343 (S) Redeclaration of >>> MPID_Send differs from previous declaration on line 2760 of >>> "../src/include/mpiimpl.h". >>> "../src/include/mpiimpl.h", line 2785.5: 1506-377 (I) The type "int" >>> of parameter 2 differs from the previous type "long". >>> "../src/include/mpiimpl.h", line 2884.5: 1506-343 (S) Redeclaration of >>> MPID_Isend differs from previous declaration on line 2865 of >>> "../src/include/mpiimpl.h". >>> "../src/include/mpiimpl.h", line 2884.5: 1506-377 (I) The type "int" >>> of parameter 2 differs from the previous type "long". >>> "../src/include/mpitimpl.h", line 245.5: 1506-046 (S) Syntax error. >>> "../src/include/mpitimpl.h", line 900.40: 1506-022 (S) "total" is not >>> a member of "struct {...}". >>> "/scratch/home/chiensh/mpich/mpich-3.2rc2/src/mpid/pamid/include/../src/mpid_request.h", >>> line 156.21: 1506-022 (S) "cc" is not a member of "struct MPID_Request". >>> "/scratch/home/chiensh/mpich/mpich-3.2rc2/src/mpid/pamid/include/../src/mpid_request.h", >>> line 157.8: 1506-022 (S) "cc_ptr" is not a member of "struct >>> MPID_Request". >>> "/scratch/home/chiensh/mpich/mpich-3.2rc2/src/mpid/pamid/include/../src/mpid_request.h", >>> line 157.23: 1506-022 (S) "cc" is not a member of "struct MPID_Request". >>> "/scratch/home/chiensh/mpich/mpich-3.2rc2/src/mpid/pamid/include/../src/mpid_request.h", >>> line 283.62: 1506-099 (S) Unexpected argument. >>> "/scratch/home/chiensh/mpich/mpich-3.2rc2/src/mpid/pamid/include/../src/mpid_request.h", >>> line 326.23: 1506-022 (S) "cc_ptr" is not a member of "struct >>> MPID_Request". >>> "/scratch/home/chiensh/mpich/mpich-3.2rc2/src/mpid/pamid/include/../src/mpid_request.h", >>> line 341.23: 1506-022 (S) "cc_ptr" is not a member of "struct >>> MPID_Request". >>> "/scratch/home/chiensh/mpich/mpich-3.2rc2/src/mpid/pamid/include/mpidpost.h", >>> line 56.44: 1506-022 (S) "vcr" is not a member of "struct MPID_Comm". >>> "/scratch/home/chiensh/mpich/mpich-3.2rc2/src/mpid/pamid/include/mpidpost.h", >>> line 83.43: 1506-022 (S) "vcrt" is not a member of "struct MPID_Comm". >>> "/scratch/home/chiensh/mpich/mpich-3.2rc2/src/mpid/pamid/include/mpidpost.h", >>> line 84.37: 1506-022 (S) "vcrt" is not a member of "struct MPID_Comm". >>> "/scratch/home/chiensh/mpich/mpich-3.2rc2/src/mpid/pamid/include/mpidpost.h", >>> line 84.57: 1506-022 (S) "vcr" is not a member of "struct MPID_Comm". >>> "/scratch/home/chiensh/mpich/mpich-3.2rc2/src/mpid/pamid/include/mpidpost.h", >>> line 91.42: 1506-022 (S) "vcr" is not a member of "struct MPID_Comm". >>> "/scratch/home/chiensh/mpich/mpich-3.2rc2/src/mpid/pamid/include/mpidpost.h", >>> line 92.41: 1506-022 (S) "vcr" is not a member of "struct MPID_Comm". >>> "../src/mpi/attr/attr_delete.c", line 65.26: 1506-045 (S) Undeclared >>> identifier GLOBAL. >>> make[2]: *** [src/mpi/attr/lib_libmpi_la-attr_delete.lo] Error 1 >>> make[2]: Leaving directory >>> `/scratch/home/chiensh/mpich/mpich-3.2rc2/build' >>> make[1]: *** [all-recursive] Error 1 >>> make[1]: Leaving directory >>> `/scratch/home/chiensh/mpich/mpich-3.2rc2/build' >>> make: *** [all] Error 2 >>> ==================================================================================== >>> >>> _______________________________________________ >>> discuss mailing list discuss(a)mpich.org >>> To manage subscription options or unsubscribe: >>> https://lists.mpich.org/mailman/listinfo/discuss >>> >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] valgrind warning in MPIR_Process_status
by Jeff Hammond 10 Jan '16

10 Jan '16

I think this is the same issue reported on the devel list yesterday. If it is, Daniel Ibanez provided a patch that I've merged into my (completely unofficial fork on github) and will push to the official MPICH review repo tonight. If you want to try and/or verify Daniel's patch, see github/jeffhammond/mpich. The branch you want should be obvious from the name (I can't remember what it is off hand). Best, Jeff On Sunday, January 10, 2016, VandeVondele Joost < joost.vandevondele(a)mat.ethz.ch> wrote: > Hi, > > we upgraded from mpich 3.1.2 to 3.2 and our nightly builds under valgrind > show a new valgrind warning, which I suppose originates from the mpi > library, rather than our code (given the stack trace below and the fact > that they are absent with 3.1): > > There are two slightly distinct traces (alltoall and alltoallv): > > ==30165== Conditional jump or move depends on uninitialised value(s) > ==30165== at 0x6E00056: MPIR_Process_status (mpiimpl.h:4394) > ==30165== by 0x6E00056: MPIC_Waitall (helper_fns.c:774) > ==30165== by 0x6D5A853: MPIR_Alltoall_intra (alltoall.c:369) > ==30165== by 0x6D5B63C: MPIR_Alltoall (alltoall.c:564) > ==30165== by 0x6D5B6EF: MPIR_Alltoall_impl (alltoall.c:599) > ==30165== by 0x6D5BF07: PMPI_Alltoall (alltoall.c:722) > ==30165== by 0x6AFF0DC: PMPI_ALLTOALL (alltoallf.c:276) > ==30165== by 0x1D04699: __message_passing_MOD_mp_alltoall_i22 > > ==30201== Conditional jump or move depends on uninitialised value(s) > ==30201== at 0x6E00056: MPIR_Process_status (mpiimpl.h:4394) > ==30201== by 0x6E00056: MPIC_Waitall (helper_fns.c:774) > ==30201== by 0x6D5C4EA: MPIR_Alltoallv_intra (alltoallv.c:194) > ==30201== by 0x6D5CEC9: MPIR_Alltoallv (alltoallv.c:339) > ==30201== by 0x6D5CFAA: MPIR_Alltoallv_impl (alltoallv.c:376) > ==30201== by 0x6D5D8CA: PMPI_Alltoallv (alltoallv.c:527) > ==30201== by 0x6AFF13E: PMPI_ALLTOALLV (alltoallvf.c:276) > ==30201== by 0x1CCE180: __message_passing_MOD_mp_alltoall_z22v > (message_passing_z.f90:223) > > Is this issue already known/fixed ? > > I saw there was a somewhat related change (which is part of 3.2): > > > https://trac.mpich.org/projects/mpich/changeset/be7c17b3177ccb0ede49761b44a… > > but maybe that change was not fully complete ? > > Best regards, > > Joost VandeVondele > > ------------------------------ > Prof. Dr. Joost VandeVondele > Nanoscale Simulations > > ETH Zurich > HIT G 41.4 > Wolfgang-Pauli-Strasse 27 > 8093 Zurich, Switzerland > > Joost.VandeVondele(a)mat.ethz.ch > <javascript:_e(%7B%7D,'cvml','Joost.VandeVondele(a)mat.ethz.ch');> > http://www.nanosim.mat.ethz.ch/ > Tel.: +41 (0)44 6338451 > ------------------------------ > -- Jeff Hammond jeff.science(a)gmail.com http://jeffhammond.github.io/ _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

valgrind warning in MPIR_Process_status
by VandeVondele Joost 10 Jan '16

10 Jan '16

Hi, we upgraded from mpich 3.1.2 to 3.2 and our nightly builds under valgrind show a new valgrind warning, which I suppose originates from the mpi library, rather than our code (given the stack trace below and the fact that they are absent with 3.1): There are two slightly distinct traces (alltoall and alltoallv): ==30165== Conditional jump or move depends on uninitialised value(s) ==30165== at 0x6E00056: MPIR_Process_status (mpiimpl.h:4394) ==30165== by 0x6E00056: MPIC_Waitall (helper_fns.c:774) ==30165== by 0x6D5A853: MPIR_Alltoall_intra (alltoall.c:369) ==30165== by 0x6D5B63C: MPIR_Alltoall (alltoall.c:564) ==30165== by 0x6D5B6EF: MPIR_Alltoall_impl (alltoall.c:599) ==30165== by 0x6D5BF07: PMPI_Alltoall (alltoall.c:722) ==30165== by 0x6AFF0DC: PMPI_ALLTOALL (alltoallf.c:276) ==30165== by 0x1D04699: __message_passing_MOD_mp_alltoall_i22 ==30201== Conditional jump or move depends on uninitialised value(s) ==30201== at 0x6E00056: MPIR_Process_status (mpiimpl.h:4394) ==30201== by 0x6E00056: MPIC_Waitall (helper_fns.c:774) ==30201== by 0x6D5C4EA: MPIR_Alltoallv_intra (alltoallv.c:194) ==30201== by 0x6D5CEC9: MPIR_Alltoallv (alltoallv.c:339) ==30201== by 0x6D5CFAA: MPIR_Alltoallv_impl (alltoallv.c:376) ==30201== by 0x6D5D8CA: PMPI_Alltoallv (alltoallv.c:527) ==30201== by 0x6AFF13E: PMPI_ALLTOALLV (alltoallvf.c:276) ==30201== by 0x1CCE180: __message_passing_MOD_mp_alltoall_z22v (message_passing_z.f90:223) Is this issue already known/fixed ? I saw there was a somewhat related change (which is part of 3.2): https://trac.mpich.org/projects/mpich/changeset/be7c17b3177ccb0ede49761b44a… but maybe that change was not fully complete ? Best regards, Joost VandeVondele ________________________________ Prof. Dr. Joost VandeVondele Nanoscale Simulations ETH Zurich HIT G 41.4 Wolfgang-Pauli-Strasse 27 8093 Zurich, Switzerland Joost.VandeVondele(a)mat.ethz.ch http://www.nanosim.mat.ethz.ch/ Tel.: +41 (0)44 6338451 ________________________________ _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] MPICH 3.2 on BlueGene/Q
by Rob Latham 08 Jan '16

08 Jan '16

On 01/08/2016 10:13 AM, Rob Latham wrote: > If there are 3.2 features you'd like to cherry pick into this fork, and > they don't touch the device interface, or depend on patches that do, > then we can definitely do that. > > If you are working on Argonne's Blue Gene, I built and installed this > fork to /soft/libraries/unsupported/mpich > > Let me know if you run into any problems. Oh, and as Pavan says, this is unsupported which means I'd like to know if you have problems, but I cannot promise that I'll be able to spend a lot of time debugging them. ==rob >> Regards, >> Dominic >> >> Here is the environment variables >> ==================================================================================== >> >> export CC=/opt/ibmcmp/vac/bg/12.1/bin/bgxlc_r >> export CXX=/opt/ibmcmp/vacpp/bg/12.1/bin/bgxlC_r >> export F77=/opt/ibmcmp/xlf/bg/14.1/bin/bgxlf_r >> export FC=/opt/ibmcmp/xlf/bg/14.1/bin/bgxlf90_r >> export >> AR=/bgsys/drivers/V1R2M0/ppc64/gnu-linux/powerpc64-bgq-linux/bin/ar >> export >> LD=/bgsys/drivers/V1R2M0/ppc64/gnu-linux/powerpc64-bgq-linux/bin/ld >> export >> RANLIB=/bgsys/drivers/V1R2M0/ppc64/gnu-linux/powerpc64-bgq-linux/bin/ranlib >> >> export MPICHLIB_CXXFLAGS="-qhot -qinline=800 -qflag=i:i -qsaveopt >> -qsuppress=1506-236" >> export MPICHLIB_CFLAGS=${MPICHLIB_CXXFLAGS} >> export MPICHLIB_FFLAGS=${MPICHLIB_CXXFLAGS} >> export MPICHLIB_F90FLAGS=${MPICHLIB_CXXFLAGS} >> ==================================================================================== >> >> >> Here is the configure >> ==================================================================================== >> >> ../configure --host=powerpc64-bgq-linux --with-device=pamid >> --with-file-system=gpfs:BGQ >> --with-bgq-install-dir=/bgsys/drivers/V1R2M0/ppc64 >> --with-pami=/bgsys/drivers/V1R2M0/ppc64/comm/sys >> --with-pami-include=/bgsys/drivers/V1R2M0/ppc64/comm/sys/include >> --with-pami-lib=/bgsys/drivers/V1R2M0/ppc64/comm/sys/lib >> --disable-wrapper-rpath --enable-fast=nochkmsg,notiming,O3 >> --with-assert-level=0 --disable-error-messages --disable-debuginfo >> --enable-thread-cs=per-object --with-atomic-primitives >> --enable-handle-allocation=tls --enable-refcount=lock-free >> --disable-predefined-refcount >> --with-cross-file=src/mpid/pamid/cross/bgq8 >> --prefix=/scratch/home/chiensh/apps/mpich/3.2.rc2 >> ==================================================================================== >> >> >> Here is the make log >> ==================================================================================== >> >> ... >> CC src/mpi/attr/lib_libmpi_la-attr_delete.lo >> "/scratch/home/chiensh/mpich/mpich-3.2rc2/src/mpid/pamid/include/mpidi_thread.h", >> line 64.9: 1506-358 (I) "MPIU_THREAD_CS_ENTER" is defined on line 80 >> of ../src/util/thread/mpiu_thread_multiple.h. >> "/scratch/home/chiensh/mpich/mpich-3.2rc2/src/mpid/pamid/include/mpidi_thread.h", >> line 65.9: 1506-358 (I) "MPIU_THREAD_CS_EXIT" is defined on line 81 of >> ../src/util/thread/mpiu_thread_multiple.h. >> "/scratch/home/chiensh/mpich/mpich-3.2rc2/src/mpid/pamid/include/mpidi_thread.h", >> line 66.9: 1506-358 (I) "MPIU_THREAD_CS_YIELD" is defined on line 82 >> of ../src/util/thread/mpiu_thread_multiple.h. >> "../src/include/mpiimpl.h", line 1184.5: 1506-046 (S) Syntax error. >> "../src/include/mpiimpl.h", line 1481.5: 1506-046 (S) Syntax error. >> "../src/include/mpiimpl.h", line 1636.5: 1506-046 (S) Syntax error. >> "../src/include/mpiimpl.h", line 2785.5: 1506-343 (S) Redeclaration of >> MPID_Send differs from previous declaration on line 2760 of >> "../src/include/mpiimpl.h". >> "../src/include/mpiimpl.h", line 2785.5: 1506-377 (I) The type "int" >> of parameter 2 differs from the previous type "long". >> "../src/include/mpiimpl.h", line 2884.5: 1506-343 (S) Redeclaration of >> MPID_Isend differs from previous declaration on line 2865 of >> "../src/include/mpiimpl.h". >> "../src/include/mpiimpl.h", line 2884.5: 1506-377 (I) The type "int" >> of parameter 2 differs from the previous type "long". >> "../src/include/mpitimpl.h", line 245.5: 1506-046 (S) Syntax error. >> "../src/include/mpitimpl.h", line 900.40: 1506-022 (S) "total" is not >> a member of "struct {...}". >> "/scratch/home/chiensh/mpich/mpich-3.2rc2/src/mpid/pamid/include/../src/mpid_request.h", >> line 156.21: 1506-022 (S) "cc" is not a member of "struct MPID_Request". >> "/scratch/home/chiensh/mpich/mpich-3.2rc2/src/mpid/pamid/include/../src/mpid_request.h", >> line 157.8: 1506-022 (S) "cc_ptr" is not a member of "struct >> MPID_Request". >> "/scratch/home/chiensh/mpich/mpich-3.2rc2/src/mpid/pamid/include/../src/mpid_request.h", >> line 157.23: 1506-022 (S) "cc" is not a member of "struct MPID_Request". >> "/scratch/home/chiensh/mpich/mpich-3.2rc2/src/mpid/pamid/include/../src/mpid_request.h", >> line 283.62: 1506-099 (S) Unexpected argument. >> "/scratch/home/chiensh/mpich/mpich-3.2rc2/src/mpid/pamid/include/../src/mpid_request.h", >> line 326.23: 1506-022 (S) "cc_ptr" is not a member of "struct >> MPID_Request". >> "/scratch/home/chiensh/mpich/mpich-3.2rc2/src/mpid/pamid/include/../src/mpid_request.h", >> line 341.23: 1506-022 (S) "cc_ptr" is not a member of "struct >> MPID_Request". >> "/scratch/home/chiensh/mpich/mpich-3.2rc2/src/mpid/pamid/include/mpidpost.h", >> line 56.44: 1506-022 (S) "vcr" is not a member of "struct MPID_Comm". >> "/scratch/home/chiensh/mpich/mpich-3.2rc2/src/mpid/pamid/include/mpidpost.h", >> line 83.43: 1506-022 (S) "vcrt" is not a member of "struct MPID_Comm". >> "/scratch/home/chiensh/mpich/mpich-3.2rc2/src/mpid/pamid/include/mpidpost.h", >> line 84.37: 1506-022 (S) "vcrt" is not a member of "struct MPID_Comm". >> "/scratch/home/chiensh/mpich/mpich-3.2rc2/src/mpid/pamid/include/mpidpost.h", >> line 84.57: 1506-022 (S) "vcr" is not a member of "struct MPID_Comm". >> "/scratch/home/chiensh/mpich/mpich-3.2rc2/src/mpid/pamid/include/mpidpost.h", >> line 91.42: 1506-022 (S) "vcr" is not a member of "struct MPID_Comm". >> "/scratch/home/chiensh/mpich/mpich-3.2rc2/src/mpid/pamid/include/mpidpost.h", >> line 92.41: 1506-022 (S) "vcr" is not a member of "struct MPID_Comm". >> "../src/mpi/attr/attr_delete.c", line 65.26: 1506-045 (S) Undeclared >> identifier GLOBAL. >> make[2]: *** [src/mpi/attr/lib_libmpi_la-attr_delete.lo] Error 1 >> make[2]: Leaving directory >> `/scratch/home/chiensh/mpich/mpich-3.2rc2/build' >> make[1]: *** [all-recursive] Error 1 >> make[1]: Leaving directory >> `/scratch/home/chiensh/mpich/mpich-3.2rc2/build' >> make: *** [all] Error 2 >> ==================================================================================== >> >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss >> > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] MPICH 3.2 on BlueGene/Q
by Rob Latham 08 Jan '16

08 Jan '16

On 01/08/2016 04:35 AM, Dominic Chien wrote: > Hi, > > I have been trying to install MPICH 3 on a BlueGene/Q system to evaluate the MPI-3 functionalities that are missed in the IBM supported MPI, and I also want to use it to build ARMCI-MPI3, GA and NWChem on this system. > > Based on the instruction prepared by Argonne National Lab (https://wiki.mpich.org/mpich/index.php/BGQ) I managed to build the MPICH 3.1rc4 on this system. Although I am still evaluating the stability and correctness of this version on BGQ, it seem promising so far. > > However, I got problem to build MPICH 3.2 with the same instruction, can anyone shade me some light to let me get it compile? Yes, the MPICH developers made some changes to the device interface and the PAMI device (which blue gene uses) did not get those updates. What I've done is forked MPICH from the last working version (somewhere around last July) and then cherry-picked a dozen or so "important" patches (where 'important' meant mostly I/O related, since that's my primary area of focus) https://xgitlab.cels.anl.gov/robl/MPICH-BlueGene (I just now noticed that our README doesn't look so good. I'll clean it up but wanted you to get an answer as soon as possible) If there are 3.2 features you'd like to cherry pick into this fork, and they don't touch the device interface, or depend on patches that do, then we can definitely do that. If you are working on Argonne's Blue Gene, I built and installed this fork to /soft/libraries/unsupported/mpich Let me know if you run into any problems. ==rob > Regards, > Dominic > > Here is the environment variables > ==================================================================================== > export CC=/opt/ibmcmp/vac/bg/12.1/bin/bgxlc_r > export CXX=/opt/ibmcmp/vacpp/bg/12.1/bin/bgxlC_r > export F77=/opt/ibmcmp/xlf/bg/14.1/bin/bgxlf_r > export FC=/opt/ibmcmp/xlf/bg/14.1/bin/bgxlf90_r > export AR=/bgsys/drivers/V1R2M0/ppc64/gnu-linux/powerpc64-bgq-linux/bin/ar > export LD=/bgsys/drivers/V1R2M0/ppc64/gnu-linux/powerpc64-bgq-linux/bin/ld > export RANLIB=/bgsys/drivers/V1R2M0/ppc64/gnu-linux/powerpc64-bgq-linux/bin/ranlib > export MPICHLIB_CXXFLAGS="-qhot -qinline=800 -qflag=i:i -qsaveopt -qsuppress=1506-236" > export MPICHLIB_CFLAGS=${MPICHLIB_CXXFLAGS} > export MPICHLIB_FFLAGS=${MPICHLIB_CXXFLAGS} > export MPICHLIB_F90FLAGS=${MPICHLIB_CXXFLAGS} > ==================================================================================== > > Here is the configure > ==================================================================================== > ../configure --host=powerpc64-bgq-linux --with-device=pamid --with-file-system=gpfs:BGQ --with-bgq-install-dir=/bgsys/drivers/V1R2M0/ppc64 --with-pami=/bgsys/drivers/V1R2M0/ppc64/comm/sys --with-pami-include=/bgsys/drivers/V1R2M0/ppc64/comm/sys/include --with-pami-lib=/bgsys/drivers/V1R2M0/ppc64/comm/sys/lib --disable-wrapper-rpath --enable-fast=nochkmsg,notiming,O3 --with-assert-level=0 --disable-error-messages --disable-debuginfo --enable-thread-cs=per-object --with-atomic-primitives --enable-handle-allocation=tls --enable-refcount=lock-free --disable-predefined-refcount --with-cross-file=src/mpid/pamid/cross/bgq8 --prefix=/scratch/home/chiensh/apps/mpich/3.2.rc2 > ==================================================================================== > > Here is the make log > ==================================================================================== > ... > CC src/mpi/attr/lib_libmpi_la-attr_delete.lo > "/scratch/home/chiensh/mpich/mpich-3.2rc2/src/mpid/pamid/include/mpidi_thread.h", line 64.9: 1506-358 (I) "MPIU_THREAD_CS_ENTER" is defined on line 80 of ../src/util/thread/mpiu_thread_multiple.h. > "/scratch/home/chiensh/mpich/mpich-3.2rc2/src/mpid/pamid/include/mpidi_thread.h", line 65.9: 1506-358 (I) "MPIU_THREAD_CS_EXIT" is defined on line 81 of ../src/util/thread/mpiu_thread_multiple.h. > "/scratch/home/chiensh/mpich/mpich-3.2rc2/src/mpid/pamid/include/mpidi_thread.h", line 66.9: 1506-358 (I) "MPIU_THREAD_CS_YIELD" is defined on line 82 of ../src/util/thread/mpiu_thread_multiple.h. > "../src/include/mpiimpl.h", line 1184.5: 1506-046 (S) Syntax error. > "../src/include/mpiimpl.h", line 1481.5: 1506-046 (S) Syntax error. > "../src/include/mpiimpl.h", line 1636.5: 1506-046 (S) Syntax error. > "../src/include/mpiimpl.h", line 2785.5: 1506-343 (S) Redeclaration of MPID_Send differs from previous declaration on line 2760 of "../src/include/mpiimpl.h". > "../src/include/mpiimpl.h", line 2785.5: 1506-377 (I) The type "int" of parameter 2 differs from the previous type "long". > "../src/include/mpiimpl.h", line 2884.5: 1506-343 (S) Redeclaration of MPID_Isend differs from previous declaration on line 2865 of "../src/include/mpiimpl.h". > "../src/include/mpiimpl.h", line 2884.5: 1506-377 (I) The type "int" of parameter 2 differs from the previous type "long". > "../src/include/mpitimpl.h", line 245.5: 1506-046 (S) Syntax error. > "../src/include/mpitimpl.h", line 900.40: 1506-022 (S) "total" is not a member of "struct {...}". > "/scratch/home/chiensh/mpich/mpich-3.2rc2/src/mpid/pamid/include/../src/mpid_request.h", line 156.21: 1506-022 (S) "cc" is not a member of "struct MPID_Request". > "/scratch/home/chiensh/mpich/mpich-3.2rc2/src/mpid/pamid/include/../src/mpid_request.h", line 157.8: 1506-022 (S) "cc_ptr" is not a member of "struct MPID_Request". > "/scratch/home/chiensh/mpich/mpich-3.2rc2/src/mpid/pamid/include/../src/mpid_request.h", line 157.23: 1506-022 (S) "cc" is not a member of "struct MPID_Request". > "/scratch/home/chiensh/mpich/mpich-3.2rc2/src/mpid/pamid/include/../src/mpid_request.h", line 283.62: 1506-099 (S) Unexpected argument. > "/scratch/home/chiensh/mpich/mpich-3.2rc2/src/mpid/pamid/include/../src/mpid_request.h", line 326.23: 1506-022 (S) "cc_ptr" is not a member of "struct MPID_Request". > "/scratch/home/chiensh/mpich/mpich-3.2rc2/src/mpid/pamid/include/../src/mpid_request.h", line 341.23: 1506-022 (S) "cc_ptr" is not a member of "struct MPID_Request". > "/scratch/home/chiensh/mpich/mpich-3.2rc2/src/mpid/pamid/include/mpidpost.h", line 56.44: 1506-022 (S) "vcr" is not a member of "struct MPID_Comm". > "/scratch/home/chiensh/mpich/mpich-3.2rc2/src/mpid/pamid/include/mpidpost.h", line 83.43: 1506-022 (S) "vcrt" is not a member of "struct MPID_Comm". > "/scratch/home/chiensh/mpich/mpich-3.2rc2/src/mpid/pamid/include/mpidpost.h", line 84.37: 1506-022 (S) "vcrt" is not a member of "struct MPID_Comm". > "/scratch/home/chiensh/mpich/mpich-3.2rc2/src/mpid/pamid/include/mpidpost.h", line 84.57: 1506-022 (S) "vcr" is not a member of "struct MPID_Comm". > "/scratch/home/chiensh/mpich/mpich-3.2rc2/src/mpid/pamid/include/mpidpost.h", line 91.42: 1506-022 (S) "vcr" is not a member of "struct MPID_Comm". > "/scratch/home/chiensh/mpich/mpich-3.2rc2/src/mpid/pamid/include/mpidpost.h", line 92.41: 1506-022 (S) "vcr" is not a member of "struct MPID_Comm". > "../src/mpi/attr/attr_delete.c", line 65.26: 1506-045 (S) Undeclared identifier GLOBAL. > make[2]: *** [src/mpi/attr/lib_libmpi_la-attr_delete.lo] Error 1 > make[2]: Leaving directory `/scratch/home/chiensh/mpich/mpich-3.2rc2/build' > make[1]: *** [all-recursive] Error 1 > make[1]: Leaving directory `/scratch/home/chiensh/mpich/mpich-3.2rc2/build' > make: *** [all] Error 2 > ==================================================================================== > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0