March 2014 - discuss - mailman3.cels.anl.gov

Re: [mpich-discuss] Increasing MPI ranks
by Lu, Huiwei 12 Mar '14

12 Mar '14

Hi, Jeffrey, The example in mpi4py manual can not run correctly with more than 16 processes. So I have a reason to believe mpi4py library is not calling MPI correctly. It would be more appropriate to ask mpi4py developers to support their manual example to run more than 16 processes first. — Huiwei On Mar 11, 2014, at 8:40 PM, Jeffrey Larson <jmlarson(a)anl.gov> wrote: > I was using the example from the mpi4py manual: > http://mpi4py.scipy.org/docs/mpi4py.pdf > (See the lines directly below "Worker (or child, or server) side:" in Section 4.3: Dynamic Process Management)

1 0

Re: [mpich-discuss] MPI_Address()
by Rajeev Thakur 12 Mar '14

12 Mar '14

That got added by mistake then (using an old patch first created in MPI 2.2 days I guess). Rajeev On Mar 12, 2014, at 5:31 AM, Jeff Squyres (jsquyres) <jsquyres(a)cisco.com> wrote: > On Mar 11, 2014, at 11:32 PM, Orion Poplawski <orion(a)cora.nwra.com> wrote: > >>> There are a whole bunch of such functions that were deprecated in MPI 2.0 (in 1997) and removed in MPI 3.0 (in 2012). We discussed whether to keep or remove them in MPICH, and decided to keep them so as not to gratuitously break user codes that still use the old functions. >> >> Ah, okay. It's a little curious that their interfaces definitions were >> updated with the const additions of MPI-3, but I suppose that leads to a >> compiler warning at most. > > (This thread originated on the OMPI list) > > The point is that the old/deprecated functions (e.g., MPI_ADDRESS, MPI_TYPE_STRUCT, MPI_TYPE_HINDEXED, etc.) should not be updated to have const parameters -- those interfaces were not updated in MPI-3. Seems like a minor bug that should be easy to fix. > > -- > Jeff Squyres > jsquyres(a)cisco.com > For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/ > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] MPI_Address()
by Jeff Squyres (jsquyres) 12 Mar '14

12 Mar '14

On Mar 11, 2014, at 11:32 PM, Orion Poplawski <orion(a)cora.nwra.com> wrote: >> There are a whole bunch of such functions that were deprecated in MPI 2.0 (in 1997) and removed in MPI 3.0 (in 2012). We discussed whether to keep or remove them in MPICH, and decided to keep them so as not to gratuitously break user codes that still use the old functions. > > Ah, okay. It's a little curious that their interfaces definitions were > updated with the const additions of MPI-3, but I suppose that leads to a > compiler warning at most. (This thread originated on the OMPI list) The point is that the old/deprecated functions (e.g., MPI_ADDRESS, MPI_TYPE_STRUCT, MPI_TYPE_HINDEXED, etc.) should not be updated to have const parameters -- those interfaces were not updated in MPI-3. Seems like a minor bug that should be easy to fix. -- Jeff Squyres jsquyres(a)cisco.com For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/

1 0

Re: [mpich-discuss] MPI_Address()
by Orion Poplawski 11 Mar '14

11 Mar '14

On 03/11/2014 08:51 PM, Rajeev Thakur wrote: > There are a whole bunch of such functions that were deprecated in MPI 2.0 (in 1997) and removed in MPI 3.0 (in 2012). We discussed whether to keep or remove them in MPICH, and decided to keep them so as not to gratuitously break user codes that still use the old functions. > > Rajeev Ah, okay. It's a little curious that their interfaces definitions were updated with the const additions of MPI-3, but I suppose that leads to a compiler warning at most. Orion > > On Mar 11, 2014, at 9:28 PM, Orion Poplawski <orion(a)cora.nwra.com> > wrote: > >> MPI_Address() was removed from the MPI API in the MPI-3 specification so >> it should be removed from mpich's mpi.h as well. Still present in 3.1 >> and current repository. >> >> - Orion >> >> -- >> Orion Poplawski >> Technical Manager 303-415-9701 x222 >> NWRA/CoRA Division FAX: 303-415-9702 >> 3380 Mitchell Lane orion(a)cora.nwra.com >> Boulder, CO 80301 http://www.cora.nwra.com >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > -- Orion Poplawski Technical Manager 303-415-9701 x222 NWRA/CoRA Division FAX: 303-415-9702 3380 Mitchell Lane orion(a)cora.nwra.com Boulder, CO 80301 http://www.cora.nwra.com

1 0

Re: [mpich-discuss] MPI_Address()
by Rajeev Thakur 11 Mar '14

11 Mar '14

There are a whole bunch of such functions that were deprecated in MPI 2.0 (in 1997) and removed in MPI 3.0 (in 2012). We discussed whether to keep or remove them in MPICH, and decided to keep them so as not to gratuitously break user codes that still use the old functions. Rajeev On Mar 11, 2014, at 9:28 PM, Orion Poplawski <orion(a)cora.nwra.com> wrote: > MPI_Address() was removed from the MPI API in the MPI-3 specification so > it should be removed from mpich's mpi.h as well. Still present in 3.1 > and current repository. > > - Orion > > -- > Orion Poplawski > Technical Manager 303-415-9701 x222 > NWRA/CoRA Division FAX: 303-415-9702 > 3380 Mitchell Lane orion(a)cora.nwra.com > Boulder, CO 80301 http://www.cora.nwra.com > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss

1 0

MPI_Address()
by Orion Poplawski 11 Mar '14

11 Mar '14

MPI_Address() was removed from the MPI API in the MPI-3 specification so it should be removed from mpich's mpi.h as well. Still present in 3.1 and current repository. - Orion -- Orion Poplawski Technical Manager 303-415-9701 x222 NWRA/CoRA Division FAX: 303-415-9702 3380 Mitchell Lane orion(a)cora.nwra.com Boulder, CO 80301 http://www.cora.nwra.com

1 0

Re: [mpich-discuss] Increasing MPI ranks
by Jeffrey Larson 11 Mar '14

11 Mar '14

Thank you both for your time. I respond to each question below: Huiwei Lu, the outputs from the two commands are below: [jlarson@mintthinkpad test]$ mpiexec --version HYDRA build details: Version: 3.1 Release Date: Thu Feb 20 11:41:13 CST 2014 CC: gcc CXX: g++ F77: gfortran F90: gfortran Configure options: '--disable-option-checking' '--prefix=/home/jlarson/software/mpich-install' '--cache-file=/dev/null' '--srcdir=.' 'CC=gcc' 'CFLAGS= -O2' 'LDFLAGS= ' 'LIBS=-lrt -lpthread ' 'CPPFLAGS= -I/home/jlarson/software/mpich-3.1/src/mpl/include -I/home/jlarson/software/mpich-3.1/src/mpl/include -I/home/jlarson/software/mpich-3.1/src/openpa/src -I/home/jlarson/software/mpich-3.1/src/openpa/src -I/home/jlarson/software/mpich-3.1/src/mpi/romio/include' Process Manager: pmi Launchers available: ssh rsh fork slurm ll lsf sge manual persist Topology libraries available: hwloc Resource management kernels available: user slurm ll lsf sge pbs cobalt Checkpointing libraries available: blcr Demux engines available: poll select [jlarson@mintthinkpad test]$ mpicc -v mpicc for MPICH version 3.1 Using built-in specs. COLLECT_GCC=gcc COLLECT_LTO_WRAPPER=/usr/lib/gcc/x86_64-linux-gnu/4.6/lto-wrapper Target: x86_64-linux-gnu Configured with: ../src/configure -v --with-pkgversion='Ubuntu/Linaro 4.6.3-1ubuntu5' --with-bugurl=file:///usr/share/doc/gcc-4.6/README.Bugs --enable-languages=c,c++,fortran,objc,obj-c++ --prefix=/usr --program-suffix=-4.6 --enable-shared --enable-linker-build-id --with-system-zlib --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --with-gxx-include-dir=/usr/include/c++/4.6 --libdir=/usr/lib --enable-nls --with-sysroot=/ --enable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcxx-time=yes --enable-gnu-unique-object --enable-plugin --enable-objc-gc --disable-werror --with-arch-32=i686 --with-tune=generic --enable-checking=release --build=x86_64-linux-gnu --host=x86_64-linux-gnu --target=x86_64-linux-gnu Thread model: posix gcc version 4.6.3 (Ubuntu/Linaro 4.6.3-1ubuntu5) Jed Brown, I don't know how else to have the function communicate with the worker that spawned it. If I replace the lines: ---- comm_to_function_wrapper = MPI.Comm.Get_parent() rank = comm_to_function_wrapper.Get_rank() ---- with: ---- comm_to_function_wrapper = MPI.COMM_WORLD.Get_parent() rank = comm_to_function_wrapper.Get_rank() ---- then I still get valid output for 3 ranks, but not 30. Is there a valid way for a spawned task to communicate with the worker that spawned it? I was using the example from the mpi4py manual: http://mpi4py.scipy.org/docs/mpi4py.pdf (See the lines directly below "Worker (or child, or server) side:" in Section 4.3: Dynamic Process Management) Thank you again! Jeff On Tue, Mar 11, 2014 at 5:52 PM, Jed Brown <jed(a)jedbrown.org> wrote: > Jeffrey Larson <jmlarson(a)anl.gov> writes: > > > Thank you for your response. Attached are the script and the simple > > function that it calls. > > > > The command: > > $ mpiexec -n 3 python script.py > > works great, but 30 crashes. > > I don't think this is valid: > > | comm_to_function_wrapper = MPI.Comm.Get_parent() > | rank = comm_to_function_wrapper.Get_rank() > > $ make intercomm CC=/opt/mpich/bin/mpicc && /opt/mpich/bin/mpiexec -n 2 > ./intercomm > /opt/mpich/bin/mpicc intercomm.c -o intercomm > Fatal error in PMPI_Comm_rank: Invalid communicator, error stack: > PMPI_Comm_rank(108): MPI_Comm_rank(MPI_COMM_NULL, rank=0x7fff9fe6a118) > failed > PMPI_Comm_rank(66).: Null communicator > Fatal error in PMPI_Comm_rank: Invalid communicator, error stack: > PMPI_Comm_rank(108): MPI_Comm_rank(MPI_COMM_NULL, rank=0x7fff5e484d48) > failed > PMPI_Comm_rank(66).: Null communicator > > > =================================================================================== > = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES > = PID 8474 RUNNING AT batura > = EXIT CODE: 1 > = CLEANING UP REMAINING PROCESSES > = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES > > =================================================================================== > $ /opt/mpich/bin/mpichversion > MPICH Version: 3.1 > MPICH Release date: Thu Feb 20 11:41:13 CST 2014 > MPICH Device: ch3:nemesis > MPICH configure: --prefix=/opt/mpich --enable-shared > --enable-sharedlibs=gcc --enable-error-checking=runtime > --enable-error-messages=all --enable-timer-type=clock_gettime > --with-python=python2 > MPICH CC: gcc -march=x86-64 -mtune=generic -O2 -pipe > -fstack-protector --param=ssp-buffer-size=4 -O2 > MPICH CXX: g++ -march=x86-64 -mtune=generic -O2 -pipe > -fstack-protector --param=ssp-buffer-size=4 > MPICH F77: gfortran -O2 > MPICH FC: gfortran > >

1 0

Re: [mpich-discuss] Increasing MPI ranks
by Jed Brown 11 Mar '14

11 Mar '14

Jeffrey Larson <jmlarson(a)anl.gov> writes: > Thank you for your response. Attached are the script and the simple > function that it calls. > > The command: > $ mpiexec -n 3 python script.py > works great, but 30 crashes. I don't think this is valid: | comm_to_function_wrapper = MPI.Comm.Get_parent() | rank = comm_to_function_wrapper.Get_rank() $ make intercomm CC=/opt/mpich/bin/mpicc && /opt/mpich/bin/mpiexec -n 2 ./intercomm /opt/mpich/bin/mpicc intercomm.c -o intercomm Fatal error in PMPI_Comm_rank: Invalid communicator, error stack: PMPI_Comm_rank(108): MPI_Comm_rank(MPI_COMM_NULL, rank=0x7fff9fe6a118) failed PMPI_Comm_rank(66).: Null communicator Fatal error in PMPI_Comm_rank: Invalid communicator, error stack: PMPI_Comm_rank(108): MPI_Comm_rank(MPI_COMM_NULL, rank=0x7fff5e484d48) failed PMPI_Comm_rank(66).: Null communicator =================================================================================== = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = PID 8474 RUNNING AT batura = EXIT CODE: 1 = CLEANING UP REMAINING PROCESSES = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES =================================================================================== $ /opt/mpich/bin/mpichversion MPICH Version: 3.1 MPICH Release date: Thu Feb 20 11:41:13 CST 2014 MPICH Device: ch3:nemesis MPICH configure: --prefix=/opt/mpich --enable-shared --enable-sharedlibs=gcc --enable-error-checking=runtime --enable-error-messages=all --enable-timer-type=clock_gettime --with-python=python2 MPICH CC: gcc -march=x86-64 -mtune=generic -O2 -pipe -fstack-protector --param=ssp-buffer-size=4 -O2 MPICH CXX: g++ -march=x86-64 -mtune=generic -O2 -pipe -fstack-protector --param=ssp-buffer-size=4 MPICH F77: gfortran -O2 MPICH FC: gfortran

1 0

Re: [mpich-discuss] Increasing MPI ranks
by Lu, Huiwei 11 Mar '14

11 Mar '14

Can you give us the output of the following: $ mpicc -v $ mpiexec —version — Huiwei On Mar 11, 2014, at 4:49 PM, Jeffrey Larson <jmlarson(a)anl.gov> wrote: > Now that I look closer, I don't believe that the version 1.4.1 is actually being used. Just the mpich2version command refers to the old version: > > [jlarson@mintthinkpad ~]$ which mpiexec > /home/jlarson/software/mpich-install/bin/mpiexec > [jlarson@mintthinkpad ~]$ which mpicc > /home/jlarson/software/mpich-install/bin/mpicc > [jlarson@mintthinkpad ~]$ which mpich2version > /usr/bin/mpich2version > > > > On Tue, Mar 11, 2014 at 4:41 PM, Lu, Huiwei <huiweilu(a)mcs.anl.gov> wrote: > A quick question before we digging into the script: why are you using different version of MPICH2 and mpiexec? > > — > Huiwei > > On Mar 11, 2014, at 4:37 PM, Jeffrey Larson <jmlarson(a)anl.gov> wrote: > > > Thank you for your response. Attached are the script and the simple function that it calls. > > > > The command: > > $ mpiexec -n 3 python script.py > > works great, but 30 crashes. > > > > I am using MPICH2 Version: 1.4.1. and mpiexec version 3.1. > > > > > > > > On Tue, Mar 11, 2014 at 4:31 PM, Lu, Huiwei <huiweilu(a)mcs.anl.gov> wrote: > > Hi, Jeffrey, > > > > Can you send us a minimum example of your script.py? > > Simple python scripts seem to work fine for me on both Mac and Ubuntu machines. > > > > Also, what version of MPICH are you using? > > — > > Huiwei Lu > > Postdoc Appointee > > Mathematics and Computer Science Division, Argonne National Laboratory > > http://www.mcs.anl.gov/~huiweilu/ > > > > On Mar 11, 2014, at 4:08 PM, Jeffrey Larson <jmlarson(a)anl.gov> wrote: > > > > > Hello, > > > > > > I am using mpi4py to call MPI from python, and when I increase the number of ranks at the command line so something like: > > > > > > [jlarson@mintthinkpad test]$ mpiexec -n 30 python script.py > > > > > > I receive the following error messages: > > > > > > [mpiexec@mintthinkpad] handle_pmi_cmd (pm/pmiserv/pmiserv_cb.c:52): Unrecognized PMI command: | cleaning up processes > > > [mpiexec@mintthinkpad] control_cb (pm/pmiserv/pmiserv_cb.c:280): unable to process PMI command > > > [mpiexec@mintthinkpad] HYDT_dmxu_poll_wait_for_event (tools/demux/demux_poll.c:76): callback returned error status > > > [mpiexec@mintthinkpad] HYD_pmci_wait_for_completion (pm/pmiserv/pmiserv_pmci.c:198): error waiting for event > > > [mpiexec@mintthinkpad] main (ui/mpich/mpiexec.c:336): process manager error waiting for completion > > > > > > When I search online for help with these error messages, I am only pointed towards the repository containing the files that are themselves throwing the error. For example: Googling for "handle_pmi_cmd Unrecognized PMI command" only returns: > > > > > > https://svn.mcs.anl.gov/repos/mpi/mpich2/tags/release/mpich2-1.3a1/src/pm/h… > > > > > > > > > Note that my script is using very few resources; each rank is spawning a simple external function. The function takes a vector and then sums all the entries, squares them, and returns the number to each rank. The script works as expected when the number of initial ranks is small. > > > > > > Thank you for your help. > > > Jeffrey Larson > > > _______________________________________________ > > > discuss mailing list discuss(a)mpich.org > > > To manage subscription options or unsubscribe: > > > https://lists.mpich.org/mailman/listinfo/discuss > > > > _______________________________________________ > > discuss mailing list discuss(a)mpich.org > > To manage subscription options or unsubscribe: > > https://lists.mpich.org/mailman/listinfo/discuss > > > > <function.py><script.py>_______________________________________________ > > discuss mailing list discuss(a)mpich.org > > To manage subscription options or unsubscribe: > > https://lists.mpich.org/mailman/listinfo/discuss > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Increasing MPI ranks
by Jeffrey Larson 11 Mar '14

11 Mar '14

Now that I look closer, I don't believe that the version 1.4.1 is actually being used. Just the mpich2version command refers to the old version: [jlarson@mintthinkpad ~]$ which mpiexec /home/jlarson/software/mpich-install/bin/mpiexec [jlarson@mintthinkpad ~]$ which mpicc /home/jlarson/software/mpich-install/bin/mpicc [jlarson@mintthinkpad ~]$ which mpich2version /usr/bin/mpich2version On Tue, Mar 11, 2014 at 4:41 PM, Lu, Huiwei <huiweilu(a)mcs.anl.gov> wrote: > A quick question before we digging into the script: why are you using > different version of MPICH2 and mpiexec? > > -- > Huiwei > > On Mar 11, 2014, at 4:37 PM, Jeffrey Larson <jmlarson(a)anl.gov> wrote: > > > Thank you for your response. Attached are the script and the simple > function that it calls. > > > > The command: > > $ mpiexec -n 3 python script.py > > works great, but 30 crashes. > > > > I am using MPICH2 Version: 1.4.1. and mpiexec version 3.1. > > > > > > > > On Tue, Mar 11, 2014 at 4:31 PM, Lu, Huiwei <huiweilu(a)mcs.anl.gov> > wrote: > > Hi, Jeffrey, > > > > Can you send us a minimum example of your script.py? > > Simple python scripts seem to work fine for me on both Mac and Ubuntu > machines. > > > > Also, what version of MPICH are you using? > > -- > > Huiwei Lu > > Postdoc Appointee > > Mathematics and Computer Science Division, Argonne National Laboratory > > http://www.mcs.anl.gov/~huiweilu/ > > > > On Mar 11, 2014, at 4:08 PM, Jeffrey Larson <jmlarson(a)anl.gov> wrote: > > > > > Hello, > > > > > > I am using mpi4py to call MPI from python, and when I increase the > number of ranks at the command line so something like: > > > > > > [jlarson@mintthinkpad test]$ mpiexec -n 30 python script.py > > > > > > I receive the following error messages: > > > > > > [mpiexec@mintthinkpad] handle_pmi_cmd (pm/pmiserv/pmiserv_cb.c:52): > Unrecognized PMI command: | cleaning up processes > > > [mpiexec@mintthinkpad] control_cb (pm/pmiserv/pmiserv_cb.c:280): > unable to process PMI command > > > [mpiexec@mintthinkpad] HYDT_dmxu_poll_wait_for_event > (tools/demux/demux_poll.c:76): callback returned error status > > > [mpiexec@mintthinkpad] HYD_pmci_wait_for_completion > (pm/pmiserv/pmiserv_pmci.c:198): error waiting for event > > > [mpiexec@mintthinkpad] main (ui/mpich/mpiexec.c:336): process manager > error waiting for completion > > > > > > When I search online for help with these error messages, I am only > pointed towards the repository containing the files that are themselves > throwing the error. For example: Googling for "handle_pmi_cmd Unrecognized > PMI command" only returns: > > > > > > > https://svn.mcs.anl.gov/repos/mpi/mpich2/tags/release/mpich2-1.3a1/src/pm/h… > > > > > > > > > Note that my script is using very few resources; each rank is spawning > a simple external function. The function takes a vector and then sums all > the entries, squares them, and returns the number to each rank. The script > works as expected when the number of initial ranks is small. > > > > > > Thank you for your help. > > > Jeffrey Larson > > > _______________________________________________ > > > discuss mailing list discuss(a)mpich.org > > > To manage subscription options or unsubscribe: > > > https://lists.mpich.org/mailman/listinfo/discuss > > > > _______________________________________________ > > discuss mailing list discuss(a)mpich.org > > To manage subscription options or unsubscribe: > > https://lists.mpich.org/mailman/listinfo/discuss > > > > <function.py><script.py>_______________________________________________ > > discuss mailing list discuss(a)mpich.org > > To manage subscription options or unsubscribe: > > https://lists.mpich.org/mailman/listinfo/discuss > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss >

1 0