January 2022 - discuss - mailman3.cels.anl.gov

Re: [mpich-discuss] Slurm and MPI_Comm_spawn
by Mccall, Kurt E. (MSFC-EV41) 07 Jan '22

07 Jan '22

Hui, "Failing" == spinning in place, deep in the MPI library inside the MPI_Comm_spawn call. I can provide the stack trace if you need it. Slurm says this, but not sure why it mentions srun since I am calling that nowhere. srun: error: Unable to create step for job 34378: Requested node configuration is not available I tried ntasks = 4, 20 and 48 and the results were always the same. Kurt From: Zhou, Hui <zhouh(a)anl.gov> Sent: Friday, January 7, 2022 4:13 PM To: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov>; discuss(a)mpich.org Subject: [EXTERNAL] Re: Slurm and MPI_Comm_spawn Hi Kurt, Thanks for the details. When you say the job is failing, is it process hanging or abort? Are there any error messages? My suspicion is that slurm is preventing extra process to be launched since you have assigned all the resources to the first two MPI processes. Could you try increase the ntasks in the batch command? -- Hui Zhou ________________________________ From: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov<mailto:[email protected]>> Sent: Friday, January 7, 2022 3:33 PM To: Zhou, Hui <zhouh(a)anl.gov<mailto:[email protected]>>; discuss(a)mpich.org<mailto:[email protected]> <discuss(a)mpich.org<mailto:[email protected]>> Cc: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov<mailto:[email protected]>> Subject: Re: Slurm and MPI_Comm_spawn Thanks for the reply, Hui. configure --prefix=/home/kmccall/mpich-install-4.0b1 --with-device=ch3:nemesis --disable-fortran -enable-debuginfo --enable-g=debug The program is run via sbatch, which is given a bash script as an argument. sbatch --nodes=2 --ntasks=2 --cpus-per-task=24 <bash_script> The bash script calls mpiexec: mpiexec -print-all-exitcodes -enable-x -np 2 -wdir ${work_dir} -env DISPLAY localhost:10.0 --ppn 1 <cmd> From: Zhou, Hui <zhouh(a)anl.gov<mailto:[email protected]>> Sent: Friday, January 7, 2022 2:39 PM To: discuss(a)mpich.org<mailto:[email protected]> Cc: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov<mailto:[email protected]>> Subject: [EXTERNAL] Re: Slurm and MPI_Comm_spawn MPICH uses PMI 1 by default. How is your MPICH configured? And how do you run your program, is it via srun? -- Hui Zhou From: Mccall, Kurt E. (MSFC-EV41) via discuss <discuss(a)mpich.org<mailto:[email protected]>> Date: Friday, January 7, 2022 at 2:21 PM To: discuss(a)mpich.org<mailto:[email protected]> <discuss(a)mpich.org<mailto:[email protected]>> Cc: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov<mailto:[email protected]>> Subject: [mpich-discuss] Slurm and MPI_Comm_spawn My MPICH/Slurm job is failing when the call to MPI_Comm_spawn is made. The Slurm MPI guide https://slurm.schedmd.com/mpi_guide.html#mpich2<https://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fslurm.sch…> specifically states that MPI_Comm_spawn will work going through Hydra's PMI 1.1 interface. How do I ensure that it goes through that interface? Maybe we'll have to rebuild Slurm to support PMI 1.1. This Slurm command yields the following and PMI 1.1 is not mentioned, although PMI 2 is. $ srun -mpi=list srun: MPI types are... srun: cray_shasta srun: pmi2 srun: none

1 0

Re: [mpich-discuss] Slurm and MPI_Comm_spawn
by Zhou, Hui 07 Jan '22

07 Jan '22

Hi Kurt, Thanks for the details. When you say the job is failing, is it process hanging or abort? Are there any error messages? My suspicion is that slurm is preventing extra process to be launched since you have assigned all the resources to the first two MPI processes. Could you try increase the ntasks in the batch command? -- Hui Zhou ________________________________ From: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov> Sent: Friday, January 7, 2022 3:33 PM To: Zhou, Hui <zhouh(a)anl.gov>; discuss(a)mpich.org <discuss(a)mpich.org> Cc: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov> Subject: Re: Slurm and MPI_Comm_spawn Thanks for the reply, Hui. configure --prefix=/home/kmccall/mpich-install-4.0b1 --with-device=ch3:nemesis --disable-fortran -enable-debuginfo --enable-g=debug The program is run via sbatch, which is given a bash script as an argument. sbatch --nodes=2 --ntasks=2 --cpus-per-task=24 <bash_script> The bash script calls mpiexec: mpiexec -print-all-exitcodes -enable-x -np 2 -wdir ${work_dir} -env DISPLAY localhost:10.0 --ppn 1 <cmd> From: Zhou, Hui <zhouh(a)anl.gov> Sent: Friday, January 7, 2022 2:39 PM To: discuss(a)mpich.org Cc: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov> Subject: [EXTERNAL] Re: Slurm and MPI_Comm_spawn MPICH uses PMI 1 by default. How is your MPICH configured? And how do you run your program, is it via srun? -- Hui Zhou From: Mccall, Kurt E. (MSFC-EV41) via discuss <discuss(a)mpich.org<mailto:[email protected]>> Date: Friday, January 7, 2022 at 2:21 PM To: discuss(a)mpich.org<mailto:[email protected]> <discuss(a)mpich.org<mailto:[email protected]>> Cc: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov<mailto:[email protected]>> Subject: [mpich-discuss] Slurm and MPI_Comm_spawn My MPICH/Slurm job is failing when the call to MPI_Comm_spawn is made. The Slurm MPI guide https://slurm.schedmd.com/mpi_guide.html#mpich2<https://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fslurm.sch…> specifically states that MPI_Comm_spawn will work going through Hydra’s PMI 1.1 interface. How do I ensure that it goes through that interface? Maybe we’ll have to rebuild Slurm to support PMI 1.1. This Slurm command yields the following and PMI 1.1 is not mentioned, although PMI 2 is. $ srun –mpi=list srun: MPI types are... srun: cray_shasta srun: pmi2 srun: none

1 0

Re: [mpich-discuss] Slurm and MPI_Comm_spawn
by Mccall, Kurt E. (MSFC-EV41) 07 Jan '22

07 Jan '22

Thanks for the reply, Hui. configure --prefix=/home/kmccall/mpich-install-4.0b1 --with-device=ch3:nemesis --disable-fortran -enable-debuginfo --enable-g=debug The program is run via sbatch, which is given a bash script as an argument. sbatch --nodes=2 --ntasks=2 --cpus-per-task=24 <bash_script> The bash script calls mpiexec: mpiexec -print-all-exitcodes -enable-x -np 2 -wdir ${work_dir} -env DISPLAY localhost:10.0 --ppn 1 <cmd> From: Zhou, Hui <zhouh(a)anl.gov> Sent: Friday, January 7, 2022 2:39 PM To: discuss(a)mpich.org Cc: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov> Subject: [EXTERNAL] Re: Slurm and MPI_Comm_spawn MPICH uses PMI 1 by default. How is your MPICH configured? And how do you run your program, is it via srun? -- Hui Zhou From: Mccall, Kurt E. (MSFC-EV41) via discuss <discuss(a)mpich.org<mailto:[email protected]>> Date: Friday, January 7, 2022 at 2:21 PM To: discuss(a)mpich.org<mailto:[email protected]> <discuss(a)mpich.org<mailto:[email protected]>> Cc: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov<mailto:[email protected]>> Subject: [mpich-discuss] Slurm and MPI_Comm_spawn My MPICH/Slurm job is failing when the call to MPI_Comm_spawn is made. The Slurm MPI guide https://slurm.schedmd.com/mpi_guide.html#mpich2<https://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fslurm.sch…> specifically states that MPI_Comm_spawn will work going through Hydra's PMI 1.1 interface. How do I ensure that it goes through that interface? Maybe we'll have to rebuild Slurm to support PMI 1.1. This Slurm command yields the following and PMI 1.1 is not mentioned, although PMI 2 is. $ srun -mpi=list srun: MPI types are... srun: cray_shasta srun: pmi2 srun: none

1 0

Re: [mpich-discuss] Slurm and MPI_Comm_spawn
by Zhou, Hui 07 Jan '22

07 Jan '22

MPICH uses PMI 1 by default. How is your MPICH configured? And how do you run your program, is it via srun? -- Hui Zhou From: Mccall, Kurt E. (MSFC-EV41) via discuss <discuss(a)mpich.org> Date: Friday, January 7, 2022 at 2:21 PM To: discuss(a)mpich.org <discuss(a)mpich.org> Cc: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov> Subject: [mpich-discuss] Slurm and MPI_Comm_spawn My MPICH/Slurm job is failing when the call to MPI_Comm_spawn is made. The Slurm MPI guide https://slurm.schedmd.com/mpi_guide.html#mpich2<https://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fslurm.sch…> specifically states that MPI_Comm_spawn will work going through Hydra’s PMI 1.1 interface. How do I ensure that it goes through that interface? Maybe we’ll have to rebuild Slurm to support PMI 1.1. This Slurm command yields the following and PMI 1.1 is not mentioned, although PMI 2 is. $ srun –mpi=list srun: MPI types are... srun: cray_shasta srun: pmi2 srun: none

1 0

Slurm and MPI_Comm_spawn
by Mccall, Kurt E. (MSFC-EV41) 07 Jan '22

07 Jan '22

My MPICH/Slurm job is failing when the call to MPI_Comm_spawn is made. The Slurm MPI guide https://slurm.schedmd.com/mpi_guide.html#mpich2<https://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fslurm.sch…> specifically states that MPI_Comm_spawn will work going through Hydra's PMI 1.1 interface. How do I ensure that it goes through that interface? Maybe we'll have to rebuild Slurm to support PMI 1.1. This Slurm command yields the following and PMI 1.1 is not mentioned, although PMI 2 is. $ srun -mpi=list srun: MPI types are... srun: cray_shasta srun: pmi2 srun: none

1 0

Re: [mpich-discuss] Building for Heterogeneous Clusters
by Zhou, Hui 07 Jan '22

07 Jan '22

MPICH doesn't support building with both libfabric and ucx at this point. We understand this use case and we may support it in the future. -- Hui Zhou ________________________________ From: Thompson, Matt (GSFC-610.1)[SCIENCE SYSTEMS AND APPLICATIONS INC] via discuss <discuss(a)mpich.org> Sent: Friday, January 7, 2022 11:16 AM To: discuss(a)mpich.org <discuss(a)mpich.org> Cc: Thompson, Matt (GSFC-610.1)[SCIENCE SYSTEMS AND APPLICATIONS INC] <matthew.thompson(a)nasa.gov> Subject: [mpich-discuss] Building for Heterogeneous Clusters Dear MPICH Gurus, I recently built MPICH 4.0b1 on a cluster I work on that happens to have a couple of different interconnects you can run on. One part is Omnipath and the other is Infiniband. Now, when I built MPICH, I did so on the Infiniband cluster and a user recently tried my module and: $ mpifort -o helloWorld.mpi3.MPICH.PSM2.exe helloWorld.mpi3.F90 /usr/bin/ld: /discover/swdev/gmao_SIteam/MPI/mpich/4.0b1/gcc-11.2.0/lib/libmpi.so: undefined reference to `rdma_establish(a)RDMACM_1.0' collect2: error: ld returned 1 exit status Turns out, you try to use that MPICH on the Omnipath cluster, and boom. My "solution" for him was "Build and run on the Infiniband cluster" and that's fine for now as we don't use MPICH in production. But it got me thinking. Is there any way of building MPICH so that it would "nicely" support both? My first build was a "no extra arguments" type of build, but the configure output did say to maybe try the ch4:ucx device (I think it chose ch4:ofi first). So, I built with "--with-device=ch4:ucx" (still on Infiniband) and this then does build on the Omnipath system...but throws a message on running: $ mpirun -np 4 ./helloWorld.mpi3.MPICH-UCX.PSM2.exe libibcm: couldn't read ABI version libibcm: couldn't read ABI version libibcm: couldn't read ABI version libibcm: couldn't read ABI version I mean, it still *ran* fine, it just prints out one of those for each process. So I thought I'd ask the experts: is there any way to build an MPICH that is "happy" on both PSM2 and Infiniband? Or should I just tell my users that want to try out MPICH "Stick with Infiniband" Note: There is no way for us to run a job on both clusters simultaneously, so it's not like I need something that will work on both at the same time. Just something that doesn't through warnings/messages if possible. Thanks, Matt -- Matt Thompson, SSAI, Ld Scientific Programmer/Analyst NASA GSFC, Global Modeling and Assimilation Office Code 610.1, 8800 Greenbelt Rd, Greenbelt, MD 20771 Phone: 301-614-6712 Fax: 301-614-6246 http://science.gsfc.nasa.gov/sed/bio/matthew.thompson

1 0

Building for Heterogeneous Clusters
by Thompson, Matt (GSFC-610.1)[SCIENCE SYSTEMS AND APPLICATIONS INC] 07 Jan '22

07 Jan '22

Dear MPICH Gurus, I recently built MPICH 4.0b1 on a cluster I work on that happens to have a couple of different interconnects you can run on. One part is Omnipath and the other is Infiniband. Now, when I built MPICH, I did so on the Infiniband cluster and a user recently tried my module and: $ mpifort -o helloWorld.mpi3.MPICH.PSM2.exe helloWorld.mpi3.F90 /usr/bin/ld: /discover/swdev/gmao_SIteam/MPI/mpich/4.0b1/gcc-11.2.0/lib/libmpi.so: undefined reference to `rdma_establish(a)RDMACM_1.0' collect2: error: ld returned 1 exit status Turns out, you try to use that MPICH on the Omnipath cluster, and boom. My "solution" for him was "Build and run on the Infiniband cluster" and that's fine for now as we don't use MPICH in production. But it got me thinking. Is there any way of building MPICH so that it would "nicely" support both? My first build was a "no extra arguments" type of build, but the configure output did say to maybe try the ch4:ucx device (I think it chose ch4:ofi first). So, I built with "--with-device=ch4:ucx" (still on Infiniband) and this then does build on the Omnipath system...but throws a message on running: $ mpirun -np 4 ./helloWorld.mpi3.MPICH-UCX.PSM2.exe libibcm: couldn't read ABI version libibcm: couldn't read ABI version libibcm: couldn't read ABI version libibcm: couldn't read ABI version I mean, it still *ran* fine, it just prints out one of those for each process. So I thought I'd ask the experts: is there any way to build an MPICH that is "happy" on both PSM2 and Infiniband? Or should I just tell my users that want to try out MPICH "Stick with Infiniband" Note: There is no way for us to run a job on both clusters simultaneously, so it's not like I need something that will work on both at the same time. Just something that doesn't through warnings/messages if possible. Thanks, Matt -- Matt Thompson, SSAI, Ld Scientific Programmer/Analyst NASA GSFC, Global Modeling and Assimilation Office Code 610.1, 8800 Greenbelt Rd, Greenbelt, MD 20771 Phone: 301-614-6712 Fax: 301-614-6246 http://science.gsfc.nasa.gov/sed/bio/matthew.thompson

1 0

Re: [mpich-discuss] Fortran 08 tests with 'use mpi'
by Raffenetti, Ken 07 Jan '22

07 Jan '22

We would consider a patch, yes. First, you would need to sign and return the contributors agreement. https://www.mpich.org/documentation/contributor-docs/ As for whether or not automatic conversion is practical, I think if we structured the tests such that the types needing conversion were easily identifiable, then it could be done. It would require some churn of the existing tests. It may also complicate the addition of new, as the author would need to adhere to guidelines that may not be obvious without prior knowledge. I gave myself a task to support automatic conversion of the tests the `use mpi_f08`, but it is not very high priority at the moment. Ken On 1/7/22, 3:10 AM, "Gregor Corbin via discuss" <discuss(a)mpich.org> wrote: Hi MPICH, I noticed that some of the tests in test/mpi/f08 still work with 'use mpi' instead of 'use mpi_f08', mostly in the pt2pt folder. These seem to be copies of the tests in f90 which in turn have been generated by f77tof90. Shouldn't these tests be converted to 'use mpi_f08'? If so, would you be interested in a PR that fixes this? I would need to do this anyway. Kind regards, Gregor Corbin PS.: The interfaces for 'use mpi' and 'use mpi_f08' are probably too different for an automatic conversion to be practical (how to change all the integers to type(mpi_xxx) for example?).

1 0

Fortran 08 tests with 'use mpi'
by Gregor Corbin 07 Jan '22

07 Jan '22

Hi MPICH, I noticed that some of the tests in test/mpi/f08 still work with 'use mpi' instead of 'use mpi_f08', mostly in the pt2pt folder. These seem to be copies of the tests in f90 which in turn have been generated by f77tof90. Shouldn't these tests be converted to 'use mpi_f08'? If so, would you be interested in a PR that fixes this? I would need to do this anyway. Kind regards, Gregor Corbin PS.: The interfaces for 'use mpi' and 'use mpi_f08' are probably too different for an automatic conversion to be practical (how to change all the integers to type(mpi_xxx) for example?).

1 0

Re: [mpich-discuss] Installation error: size of array element is not a multiple of its alignment
by Raffenetti, Ken 04 Jan '22

04 Jan '22

This appears to be fixed in 4.0.x and above. A full 4.0 release should be coming very soon. Ken On 1/3/22, 8:47 PM, "Raffenetti, Ken via discuss" <discuss(a)mpich.org> wrote: Thanks for reporting. I have created https://github.com/pmodels/mpich/issues/5744 to track the issue. If you have a chance, could you confirm the issue also exists with the latest MPICH release candidate? http://www.mpich.org/static/downloads/4.0rc1/mpich-4.0rc1.tar.gz Ken On 12/29/21, 2:51 PM, "Douglas G Dommermuth via discuss" <discuss(a)mpich.org> wrote: Dear mpich.org <http://mpich.org>, I was able to compile with gcc 10.2.0. Thank you, Doug. On Dec 29, 2021, at 9:40 AM, Douglas G Dommermuth <dgd(a)mit.edu> wrote: Dear mpich.org <http://mpich.org/>, I apologize. I sent this email out with the wrong subject matter. Here is the correct subject. Thank you. I am getting an error installing mpich-3.4.3 on a MacBook Air (M1,2020) with macOS Monterey 12.1, gcc 11.2.0, g++ 11.2.0, and gfortran 11.2.0. I used the following commands: ./configure FFLAGS=-fallow-argument-mismatch --prefix=/Users/douglasdommermuth/mpich2/mpich2-install | & tee c.txt make clean make V=1 | & tee m.txt I get the following error: CC src/mpi/attr/lib_libpmpi_la-attr_delete.lo In file included from ./src/mpid/ch4/netmod/include/../ofi/ofi_impl.h:10, from ./src/mpid/ch4/netmod/include/../ofi/ofi_am.h:8, from ./src/mpid/ch4/netmod/include/../ofi/netmod_inline.h:9, from ./src/mpid/ch4/netmod/include/netmod_impl.h:1644, from ./src/mpid/ch4/include/mpidch4.h:322, from ./src/mpid/ch4/include/mpidpost.h:10, from ./src/include/mpiimpl.h:241, from src/mpi/attr/attr_delete.c:6: ./src/mpid/ch4/netmod/include/../ofi/ofi_types.h:326:5: error: size of array element is not a multiple of its alignment 326 | MPIDI_OFI_cacheline_mutex_t mutexes[MAX_OFI_MUTEXES]; | ^~~~~~~~~~~~~~~~~~~~~~~~~~~ make[2]: *** [src/mpi/attr/lib_libpmpi_la-attr_delete.lo] Error 1 make[1]: *** [all-recursive] Error 1 make: *** [all] Error 2 An online search indicates that others are experiencing this error with gcc 11. For example, here https://github.com/JayDDee/cpuminer-opt/issues/313, https://gitlab.com/xonotic/darkplaces/-/issues/363, and https://github.com/relic-toolkit/relic/issues/202. I have not tried an earlier version of gcc. Compressed versions of c.txt, m.txt, and config log files are attached. Thank you, Doug. <m.txt.gz><c.txt.gz><config.log.gz><src_pm_hydra_config.log.gz><src_mpl_config.log.gz> _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0