December 2021 - discuss - mailman3.cels.anl.gov

make testing fails when building from git(main)
by Gregor Corbin 16 Dec '21

16 Dec '21

Dear MPICH devs, I want to use the MPI test suite provided with MPICH to test an MPI wrapper library I am developing (the MPI wrapper for Score-P). As a first step, I tried to check out the latest version from git, build everything and then run the test suite with `make testing`. I can build MPICH itself without a problem and the example cpi program runs just fine. However, during `make testing` I run into lots of errors of the form In file included from dtpools.c:6: ../include/dtpools_internal.h:12:10: fatal error: mpi.h: No such file or directory 12 | #include "mpi.h" | ^~~~~~~ compilation terminated. This happens only with the git version, not with the tarball version(mpich-4.0b1). I attached all the program outputs and log files. Any help is appreciated. As I workaround I first built MPICH itself, then did './configure CC=mpicc ...', in the test/mpi directory and then directly use the runtests script. This works for now, but is surely not how it was intended. Kind regards, Gregor Corbin

1 0

Re: [mpich-discuss] Question regarding fault tolerance difference 3.1 & 3.4.2
by Zhou, Hui 15 Dec '21

15 Dec '21

Hi Anatoly, I think the failure behavior is yet to be standardized, thus the behavior falls into the undefined behavior category. I suspect the different outcomes with MPICH 3.1 and 3.4.2 are from changes in the process manager. Could you try using hydra from 3.1 in your test with 3.4.2? -- Hui Zhou ________________________________ From: Anatoly G via discuss <discuss(a)mpich.org> Sent: Monday, December 13, 2021 7:18 AM To: mpich-discuss(a)mcs.anl.gov <mpich-discuss(a)mcs.anl.gov> Cc: Anatoly G <anatolyrishon(a)gmail.com> Subject: [mpich-discuss] Question regarding fault tolerance difference 3.1 & 3.4.2 Hi MPICH, I have a small program which has different outputs on MPICH 3.1 & MPICH 3.4.2 The program (code attached): 1. Master executes ping-pong with each one of the slaves (separately) 2. Each slave replies to master when it gets a message from master 3. One of the slaves simulates failre with command abort(). 4. Master recognizes that slave fails and continues to work with survived slaves only. I execute my program on ubuntu18. I use TCP as a transport layer. Execute command: mpiexec.hydra -genvall -disable-auto-cleanup -f MpiConfigMachines.txt -launcher=ssh -n 3 ft_ping_pong I expect that "ping pong" will continue till all 20 iterations will finish. MPICH 3.1 hydra works as expected only slave 1 fails and ping-pong continues between Master and slave2. MPICH 3.4.2 hydra process slave 2 fails together with slave 1. MPICH 3.1 configuration: ./configure --prefix="my directory" --enable-error-checking=runtime --enable-g=dbg CFLAGS=-fPIC CXXFLAGS=-fPIC FFLAGS=-fpic --enable-threads=runtime --enable-totalview --enable-static --disable-f77 --disable-fc MPICH 3.4.2 configuration: $ ./configure --prefix="my directory" --enable-error-checking=all --enable-g=dbg CFLAGS=-fPIC CXXFLAGS=-fPIC FFLAGS=-fpic --enable-threads=runtime --enable-totalview --enable-static --disable-f77 --disable-fc --disable-fortran --with-device=ch3:nemesis --enable-error-messages=all Should I use another device or the behavior was modified between versions? Regards, Anatoly.

1 0

Re: [mpich-discuss] [EXTERNAL] Re: Help Building MPICH 4.0b1: undefined references to hwloc
by Zhou, Hui 15 Dec '21

15 Dec '21

Hi Matt, Thanks for posting config.log. I found the issue. Both mpich the library and hydra the program uses hwloc and independently tries to probe hwloc. Because hydra is configured as a subdir inside mpich configure, it inherits the CFLAGS from mpich, resulting in using embedded hwloc.h while linking with external libhwloc. We'll patch mpich to strengthen its config check. Meanwhile, you can pass --with-hwloc=embedded to the mpich configure to force both mpich and hydra to use embedded hwloc, which should bypass your issue. Thanks for reporting! -- Hui Zhou ________________________________ From: Thompson, Matt (GSFC-610.1)[SCIENCE SYSTEMS AND APPLICATIONS INC] <matthew.thompson(a)nasa.gov> Sent: Wednesday, December 15, 2021 7:13 AM To: discuss(a)mpich.org <discuss(a)mpich.org> Cc: Zhou, Hui <zhouh(a)anl.gov> Subject: Re: [EXTERNAL] Re: [mpich-discuss] Help Building MPICH 4.0b1: undefined references to hwloc Hui, Please find attached the requested config.log Matt -- Matt Thompson, SSAI, Ld Scientific Programmer/Analyst NASA GSFC, Global Modeling and Assimilation Office Code 610.1, 8800 Greenbelt Rd, Greenbelt, MD 20771 Phone: 301-614-6712 Fax: 301-614-6246 http://science.gsfc.nasa.gov/sed/bio/matthew.thompson From: "Zhou, Hui via discuss" <discuss(a)mpich.org> Reply-To: "discuss(a)mpich.org" <discuss(a)mpich.org> Date: Tuesday, December 14, 2021 at 6:14 PM To: "discuss(a)mpich.org" <discuss(a)mpich.org> Cc: "Zhou, Hui" <zhouh(a)anl.gov> Subject: [EXTERNAL] Re: [mpich-discuss] Help Building MPICH 4.0b1: undefined references to hwloc Hi Matt, Could post your src/pm/hydra/config.log? -- Hui Zhou ________________________________ From: Thompson, Matt (GSFC-610.1)[SCIENCE SYSTEMS AND APPLICATIONS INC] via discuss <discuss(a)mpich.org> Sent: Tuesday, December 14, 2021 2:40 PM To: discuss(a)mpich.org <discuss(a)mpich.org> Cc: Thompson, Matt (GSFC-610.1)[SCIENCE SYSTEMS AND APPLICATIONS INC] <matthew.thompson(a)nasa.gov> Subject: [mpich-discuss] Help Building MPICH 4.0b1: undefined references to hwloc MPICH Discuss, I'm sure I've done something stupid, but I'm not sure what. I tried searching google, but my google-fu failed me. Today I decided to try and build MPICH 4.0b1 on a cluster I'm on (NCCS Discover) using GCC 11.2. My configure line was quite boring: ./configure CC=gcc CXX=g++ FC=gfortran \ FCFLAGS=-fallow-argument-mismatch \ FFLAGS=-fallow-argument-mismatch \ --prefix=/discover/swdev/gmao_SIteam/MPI/mpich/4.0b1/gcc-11.2.0 The MPICH configure step was nice enough to let me know I needed those FFLAGS and FCFLAGS. And it configured just happily.[1] When I went to the make step I eventually it errored out with: make[3]: Entering directory '/gpfsm/dswdev/gmao_SIteam/MPI/src/mpich-4.0b1/src/pm/hydra' CCLD libpm.la CCLD hydra_persist /usr/bin/ld: /discover/swdev/gmao_SIteam/MPI/src/mpich-4.0b1/src/pm/hydra/.libs/libhydra.a(topo_hwloc.o): in function `HYDT_topo_hwloc_init': /discover/swdev/gmao_SIteam/MPI/src/mpich-4.0b1/src/pm/hydra/tools/topo/hwloc/topo_hwloc.c:577: undefined reference to `hwloc_topology_set_io_types_filter' collect2: error: ld returned 1 exit status Makefile:1286: recipe for target 'hydra_persist' failed make[3]: *** [hydra_persist] Error 1 Now, as far as I can tell from the configure output, I'm using the embedded hwloc: checking if hwloc meets minimum version requirement... no configure: ===== configuring modules/hwloc ===== ... ### Configuring hwloc core ### checking hwloc building mode... embedded configure: hwloc builddir: /discover/swdev/gmao_SIteam/MPI/src/mpich-4.0b1/modules/hwloc configure: hwloc srcdir: /discover/swdev/gmao_SIteam/MPI/src/mpich-4.0b1/modules/hwloc ... And I can't see anything in the output that it's somehow picking up on an hwloc I might have installed somewhere else (be it Open MPI or just hwloc by hand). Maybe there are flags I need to pass? I do see reference to --with-hwloc=embedded, but it looks like configure decided to use that itself. Thanks for any help, Matt [1] Note: I'm building on an Infiniband system so maybe I need to do a build with --with-device=ch4:ucx eventually, but that's a runtime thing. I figure it can't be involved with this error...right? -- Matt Thompson, SSAI, Ld Scientific Programmer/Analyst NASA GSFC, Global Modeling and Assimilation Office Code 610.1, 8800 Greenbelt Rd, Greenbelt, MD 20771 Phone: 301-614-6712 Fax: 301-614-6246 http://science.gsfc.nasa.gov/sed/bio/matthew.thompson

1 0

Re: [mpich-discuss] [EXTERNAL] Re: Help Building MPICH 4.0b1: undefined references to hwloc
by Thompson, Matt (GSFC-610.1)[SCIENCE SYSTEMS AND APPLICATIONS INC] 15 Dec '21

15 Dec '21

Hui, Please find attached the requested config.log Matt -- Matt Thompson, SSAI, Ld Scientific Programmer/Analyst NASA GSFC, Global Modeling and Assimilation Office Code 610.1, 8800 Greenbelt Rd, Greenbelt, MD 20771 Phone: 301-614-6712 Fax: 301-614-6246 http://science.gsfc.nasa.gov/sed/bio/matthew.thompson From: "Zhou, Hui via discuss" <discuss(a)mpich.org> Reply-To: "discuss(a)mpich.org" <discuss(a)mpich.org> Date: Tuesday, December 14, 2021 at 6:14 PM To: "discuss(a)mpich.org" <discuss(a)mpich.org> Cc: "Zhou, Hui" <zhouh(a)anl.gov> Subject: [EXTERNAL] Re: [mpich-discuss] Help Building MPICH 4.0b1: undefined references to hwloc Hi Matt, Could post your src/pm/hydra/config.log? -- Hui Zhou ________________________________ From: Thompson, Matt (GSFC-610.1)[SCIENCE SYSTEMS AND APPLICATIONS INC] via discuss <discuss(a)mpich.org> Sent: Tuesday, December 14, 2021 2:40 PM To: discuss(a)mpich.org <discuss(a)mpich.org> Cc: Thompson, Matt (GSFC-610.1)[SCIENCE SYSTEMS AND APPLICATIONS INC] <matthew.thompson(a)nasa.gov> Subject: [mpich-discuss] Help Building MPICH 4.0b1: undefined references to hwloc MPICH Discuss, I'm sure I've done something stupid, but I'm not sure what. I tried searching google, but my google-fu failed me. Today I decided to try and build MPICH 4.0b1 on a cluster I'm on (NCCS Discover) using GCC 11.2. My configure line was quite boring: ./configure CC=gcc CXX=g++ FC=gfortran \ FCFLAGS=-fallow-argument-mismatch \ FFLAGS=-fallow-argument-mismatch \ --prefix=/discover/swdev/gmao_SIteam/MPI/mpich/4.0b1/gcc-11.2.0 The MPICH configure step was nice enough to let me know I needed those FFLAGS and FCFLAGS. And it configured just happily.[1] When I went to the make step I eventually it errored out with: make[3]: Entering directory '/gpfsm/dswdev/gmao_SIteam/MPI/src/mpich-4.0b1/src/pm/hydra' CCLD libpm.la CCLD hydra_persist /usr/bin/ld: /discover/swdev/gmao_SIteam/MPI/src/mpich-4.0b1/src/pm/hydra/.libs/libhydra.a(topo_hwloc.o): in function `HYDT_topo_hwloc_init': /discover/swdev/gmao_SIteam/MPI/src/mpich-4.0b1/src/pm/hydra/tools/topo/hwloc/topo_hwloc.c:577: undefined reference to `hwloc_topology_set_io_types_filter' collect2: error: ld returned 1 exit status Makefile:1286: recipe for target 'hydra_persist' failed make[3]: *** [hydra_persist] Error 1 Now, as far as I can tell from the configure output, I'm using the embedded hwloc: checking if hwloc meets minimum version requirement... no configure: ===== configuring modules/hwloc ===== ... ### Configuring hwloc core ### checking hwloc building mode... embedded configure: hwloc builddir: /discover/swdev/gmao_SIteam/MPI/src/mpich-4.0b1/modules/hwloc configure: hwloc srcdir: /discover/swdev/gmao_SIteam/MPI/src/mpich-4.0b1/modules/hwloc ... And I can't see anything in the output that it's somehow picking up on an hwloc I might have installed somewhere else (be it Open MPI or just hwloc by hand). Maybe there are flags I need to pass? I do see reference to --with-hwloc=embedded, but it looks like configure decided to use that itself. Thanks for any help, Matt [1] Note: I'm building on an Infiniband system so maybe I need to do a build with --with-device=ch4:ucx eventually, but that's a runtime thing. I figure it can't be involved with this error...right? -- Matt Thompson, SSAI, Ld Scientific Programmer/Analyst NASA GSFC, Global Modeling and Assimilation Office Code 610.1, 8800 Greenbelt Rd, Greenbelt, MD 20771 Phone: 301-614-6712 Fax: 301-614-6246 http://science.gsfc.nasa.gov/sed/bio/matthew.thompson

1 0

Re: [mpich-discuss] [EXTERNAL] Re: Help Building MPICH 4.0b1: undefined references to hwloc
by Thompson, Matt (GSFC-610.1)[SCIENCE SYSTEMS AND APPLICATIONS INC] 15 Dec '21

15 Dec '21

Brice, I'll take a look. Maybe I built lstopo or something at some point and it's hanging around... Matt -- Matt Thompson, SSAI, Ld Scientific Programmer/Analyst NASA GSFC, Global Modeling and Assimilation Office Code 610.1, 8800 Greenbelt Rd, Greenbelt, MD 20771 Phone: 301-614-6712 Fax: 301-614-6246 http://science.gsfc.nasa.gov/sed/bio/matthew.thompson From: Brice Goglin <Brice.Goglin(a)inria.fr> Date: Tuesday, December 14, 2021 at 5:37 PM To: <discuss(a)mpich.org> Cc: "Thompson, Matt (GSFC-610.1)[SCIENCE SYSTEMS AND APPLICATIONS INC]" <matthew.thompson(a)nasa.gov> Subject: [EXTERNAL] Re: [mpich-discuss] Help Building MPICH 4.0b1: undefined references to hwloc Le 14/12/2021 à 21:40, Thompson, Matt (GSFC-610.1)[SCIENCE SYSTEMS AND APPLICATIONS INC] via discuss a écrit : MPICH Discuss, I'm sure I've done something stupid, but I'm not sure what. I tried searching google, but my google-fu failed me. Today I decided to try and build MPICH 4.0b1 on a cluster I'm on (NCCS Discover) using GCC 11.2. My configure line was quite boring: ./configure CC=gcc CXX=g++ FC=gfortran \ FCFLAGS=-fallow-argument-mismatch \ FFLAGS=-fallow-argument-mismatch \ --prefix=/discover/swdev/gmao_SIteam/MPI/mpich/4.0b1/gcc-11.2.0 The MPICH configure step was nice enough to let me know I needed those FFLAGS and FCFLAGS. And it configured just happily.[1] When I went to the make step I eventually it errored out with: make[3]: Entering directory '/gpfsm/dswdev/gmao_SIteam/MPI/src/mpich-4.0b1/src/pm/hydra' CCLD libpm.la CCLD hydra_persist /usr/bin/ld: /discover/swdev/gmao_SIteam/MPI/src/mpich-4.0b1/src/pm/hydra/.libs/libhydra.a(topo_hwloc.o): in function `HYDT_topo_hwloc_init': /discover/swdev/gmao_SIteam/MPI/src/mpich-4.0b1/src/pm/hydra/tools/topo/hwloc/topo_hwloc.c:577: undefined reference to `hwloc_topology_set_io_types_filter' Hello I think this error usually occurs when compiling against hwloc 2.x (such as the one embedded inside MPICH) but linking against an old hwloc 1.x. Do you have such an old hwloc installed on this machine? The first line below likely says yes. I don't know why configure would mix them like this, but you should be able to avoid the error by uninstalling the old hwloc (or with a chmod 0 on its libhwloc.so during your configure). Brice checking if hwloc meets minimum version requirement... no configure: ===== configuring modules/hwloc ===== ... ### Configuring hwloc core ### checking hwloc building mode... embedded configure: hwloc builddir: /discover/swdev/gmao_SIteam/MPI/src/mpich-4.0b1/modules/hwloc configure: hwloc srcdir: /discover/swdev/gmao_SIteam/MPI/src/mpich-4.0b1/modules/hwloc ... And I can't see anything in the output that it's somehow picking up on an hwloc I might have installed somewhere else (be it Open MPI or just hwloc by hand). Maybe there are flags I need to pass? I do see reference to --with-hwloc=embedded, but it looks like configure decided to use that itself. Thanks for any help, Matt [1] Note: I'm building on an Infiniband system so maybe I need to do a build with --with-device=ch4:ucx eventually, but that's a runtime thing. I figure it can't be involved with this error...right? -- Matt Thompson, SSAI, Ld Scientific Programmer/Analyst NASA GSFC, Global Modeling and Assimilation Office Code 610.1, 8800 Greenbelt Rd, Greenbelt, MD 20771 Phone: 301-614-6712 Fax: 301-614-6246 http://science.gsfc.nasa.gov/sed/bio/matthew.thompson _______________________________________________ discuss mailing list discuss(a)mpich.org<mailto:[email protected]> To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Help Building MPICH 4.0b1: undefined references to hwloc
by Zhou, Hui 14 Dec '21

14 Dec '21

Hi Matt, Could post your src/pm/hydra/config.log? -- Hui Zhou ________________________________ From: Thompson, Matt (GSFC-610.1)[SCIENCE SYSTEMS AND APPLICATIONS INC] via discuss <discuss(a)mpich.org> Sent: Tuesday, December 14, 2021 2:40 PM To: discuss(a)mpich.org <discuss(a)mpich.org> Cc: Thompson, Matt (GSFC-610.1)[SCIENCE SYSTEMS AND APPLICATIONS INC] <matthew.thompson(a)nasa.gov> Subject: [mpich-discuss] Help Building MPICH 4.0b1: undefined references to hwloc MPICH Discuss, I'm sure I've done something stupid, but I'm not sure what. I tried searching google, but my google-fu failed me. Today I decided to try and build MPICH 4.0b1 on a cluster I'm on (NCCS Discover) using GCC 11.2. My configure line was quite boring: ./configure CC=gcc CXX=g++ FC=gfortran \ FCFLAGS=-fallow-argument-mismatch \ FFLAGS=-fallow-argument-mismatch \ --prefix=/discover/swdev/gmao_SIteam/MPI/mpich/4.0b1/gcc-11.2.0 The MPICH configure step was nice enough to let me know I needed those FFLAGS and FCFLAGS. And it configured just happily.[1] When I went to the make step I eventually it errored out with: make[3]: Entering directory '/gpfsm/dswdev/gmao_SIteam/MPI/src/mpich-4.0b1/src/pm/hydra' CCLD libpm.la CCLD hydra_persist /usr/bin/ld: /discover/swdev/gmao_SIteam/MPI/src/mpich-4.0b1/src/pm/hydra/.libs/libhydra.a(topo_hwloc.o): in function `HYDT_topo_hwloc_init': /discover/swdev/gmao_SIteam/MPI/src/mpich-4.0b1/src/pm/hydra/tools/topo/hwloc/topo_hwloc.c:577: undefined reference to `hwloc_topology_set_io_types_filter' collect2: error: ld returned 1 exit status Makefile:1286: recipe for target 'hydra_persist' failed make[3]: *** [hydra_persist] Error 1 Now, as far as I can tell from the configure output, I'm using the embedded hwloc: checking if hwloc meets minimum version requirement... no configure: ===== configuring modules/hwloc ===== ... ### Configuring hwloc core ### checking hwloc building mode... embedded configure: hwloc builddir: /discover/swdev/gmao_SIteam/MPI/src/mpich-4.0b1/modules/hwloc configure: hwloc srcdir: /discover/swdev/gmao_SIteam/MPI/src/mpich-4.0b1/modules/hwloc ... And I can't see anything in the output that it's somehow picking up on an hwloc I might have installed somewhere else (be it Open MPI or just hwloc by hand). Maybe there are flags I need to pass? I do see reference to --with-hwloc=embedded, but it looks like configure decided to use that itself. Thanks for any help, Matt [1] Note: I'm building on an Infiniband system so maybe I need to do a build with --with-device=ch4:ucx eventually, but that's a runtime thing. I figure it can't be involved with this error...right? -- Matt Thompson, SSAI, Ld Scientific Programmer/Analyst NASA GSFC, Global Modeling and Assimilation Office Code 610.1, 8800 Greenbelt Rd, Greenbelt, MD 20771 Phone: 301-614-6712 Fax: 301-614-6246 http://science.gsfc.nasa.gov/sed/bio/matthew.thompson

1 0

Re: [mpich-discuss] Help Building MPICH 4.0b1: undefined references to hwloc
by Brice Goglin 14 Dec '21

14 Dec '21

Le 14/12/2021 à 21:40, Thompson, Matt (GSFC-610.1)[SCIENCE SYSTEMS AND APPLICATIONS INC] via discuss a écrit : > > MPICH Discuss, > > I'm sure I've done something stupid, but I'm not sure what. I tried > searching google, but my google-fu failed me. > > Today I decided to try and build MPICH 4.0b1 on a cluster I'm on (NCCS > Discover) using GCC 11.2. My configure line was quite boring: > > ./configure CC=gcc CXX=g++ FC=gfortran \ > > FCFLAGS=-fallow-argument-mismatch \ > > FFLAGS=-fallow-argument-mismatch \ > > --prefix=/discover/swdev/gmao_SIteam/MPI/mpich/4.0b1/gcc-11.2.0 > > The MPICH configure step was nice enough to let me know I needed those > FFLAGS and FCFLAGS. And it configured just happily.[1] > > When I went to the make step I eventually it errored out with: > > make[3]: Entering directory > '/gpfsm/dswdev/gmao_SIteam/MPI/src/mpich-4.0b1/src/pm/hydra' > > CCLD libpm.la > > CCLD hydra_persist > > /usr/bin/ld: > /discover/swdev/gmao_SIteam/MPI/src/mpich-4.0b1/src/pm/hydra/.libs/libhydra.a(topo_hwloc.o): > in function `HYDT_topo_hwloc_init': > > /discover/swdev/gmao_SIteam/MPI/src/mpich-4.0b1/src/pm/hydra/tools/topo/hwloc/topo_hwloc.c:577: > undefined reference to `hwloc_topology_set_io_types_filter' > Hello I think this error usually occurs when compiling against hwloc 2.x (such as the one embedded inside MPICH) but linking against an old hwloc 1.x. Do you have such an old hwloc installed on this machine? The first line below likely says yes. I don't know why configure would mix them like this, but you should be able to avoid the error by uninstalling the old hwloc (or with a chmod 0 on its libhwloc.so during your configure). Brice > checking if hwloc meets minimum version requirement... no > > configure: ===== configuring modules/hwloc ===== > > ... > > ### Configuring hwloc core > > ### > > checking hwloc building mode... embedded > > configure: hwloc builddir: > /discover/swdev/gmao_SIteam/MPI/src/mpich-4.0b1/modules/hwloc > > configure: hwloc srcdir: > /discover/swdev/gmao_SIteam/MPI/src/mpich-4.0b1/modules/hwloc > > ... > > > And I can't see anything in the output that it's somehow picking up on > an hwloc I might have installed somewhere else (be it Open MPI or just > hwloc by hand). > > Maybe there are flags I need to pass? I do see reference to > --with-hwloc=embedded, but it looks like configure decided to use that > itself. > > Thanks for any help, > > Matt > > [1] Note: I'm building on an Infiniband system so maybe I need to do a > build with --with-device=ch4:ucx eventually, but that's a runtime > thing. I figure it can't be involved with this error...right? > > -- > > Matt Thompson, SSAI, Ld Scientific Programmer/Analyst > > NASA GSFC, Global Modeling and Assimilation Office > > Code 610.1, 8800 Greenbelt Rd, Greenbelt, MD 20771 > > Phone: 301-614-6712 Fax: 301-614-6246 > > _http://science.gsfc.nasa.gov/sed/bio/matthew.thompson_ > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss

1 0

Help Building MPICH 4.0b1: undefined references to hwloc
by Thompson, Matt (GSFC-610.1)[SCIENCE SYSTEMS AND APPLICATIONS INC] 14 Dec '21

14 Dec '21

MPICH Discuss, I'm sure I've done something stupid, but I'm not sure what. I tried searching google, but my google-fu failed me. Today I decided to try and build MPICH 4.0b1 on a cluster I'm on (NCCS Discover) using GCC 11.2. My configure line was quite boring: ./configure CC=gcc CXX=g++ FC=gfortran \ FCFLAGS=-fallow-argument-mismatch \ FFLAGS=-fallow-argument-mismatch \ --prefix=/discover/swdev/gmao_SIteam/MPI/mpich/4.0b1/gcc-11.2.0 The MPICH configure step was nice enough to let me know I needed those FFLAGS and FCFLAGS. And it configured just happily.[1] When I went to the make step I eventually it errored out with: make[3]: Entering directory '/gpfsm/dswdev/gmao_SIteam/MPI/src/mpich-4.0b1/src/pm/hydra' CCLD libpm.la CCLD hydra_persist /usr/bin/ld: /discover/swdev/gmao_SIteam/MPI/src/mpich-4.0b1/src/pm/hydra/.libs/libhydra.a(topo_hwloc.o): in function `HYDT_topo_hwloc_init': /discover/swdev/gmao_SIteam/MPI/src/mpich-4.0b1/src/pm/hydra/tools/topo/hwloc/topo_hwloc.c:577: undefined reference to `hwloc_topology_set_io_types_filter' collect2: error: ld returned 1 exit status Makefile:1286: recipe for target 'hydra_persist' failed make[3]: *** [hydra_persist] Error 1 Now, as far as I can tell from the configure output, I'm using the embedded hwloc: checking if hwloc meets minimum version requirement... no configure: ===== configuring modules/hwloc ===== ... ### Configuring hwloc core ### checking hwloc building mode... embedded configure: hwloc builddir: /discover/swdev/gmao_SIteam/MPI/src/mpich-4.0b1/modules/hwloc configure: hwloc srcdir: /discover/swdev/gmao_SIteam/MPI/src/mpich-4.0b1/modules/hwloc ... And I can't see anything in the output that it's somehow picking up on an hwloc I might have installed somewhere else (be it Open MPI or just hwloc by hand). Maybe there are flags I need to pass? I do see reference to --with-hwloc=embedded, but it looks like configure decided to use that itself. Thanks for any help, Matt [1] Note: I'm building on an Infiniband system so maybe I need to do a build with --with-device=ch4:ucx eventually, but that's a runtime thing. I figure it can't be involved with this error...right? -- Matt Thompson, SSAI, Ld Scientific Programmer/Analyst NASA GSFC, Global Modeling and Assimilation Office Code 610.1, 8800 Greenbelt Rd, Greenbelt, MD 20771 Phone: 301-614-6712 Fax: 301-614-6246 http://science.gsfc.nasa.gov/sed/bio/matthew.thompson

1 0

Question regarding fault tolerance difference 3.1 & 3.4.2
by Anatoly G 13 Dec '21

13 Dec '21

Hi MPICH, I have a small program which has different outputs on MPICH 3.1 & MPICH 3.4.2 The program (code attached): 1. Master executes ping-pong with each one of the slaves (separately) 2. Each slave replies to master when it gets a message from master 3. One of the slaves simulates failre with command abort(). 4. Master recognizes that slave fails and continues to work with survived slaves only. *I execute my program on ubuntu18.* I use TCP as a transport layer. *Execute command*: mpiexec.hydra -genvall -disable-auto-cleanup -f MpiConfigMachines.txt -launcher=ssh -n 3 ft_ping_pong I expect that "ping pong" will continue till all 20 iterations will finish. MPICH 3.1 hydra works as expected only slave 1 fails and ping-pong continues between Master and slave2. MPICH 3.4.2 hydra process slave 2 fails together with slave 1. *MPICH 3.1 configuration:* ./configure --prefix="my directory" --enable-error-checking=runtime --enable-g=dbg CFLAGS=-fPIC CXXFLAGS=-fPIC FFLAGS=-fpic --enable-threads=runtime --enable-totalview --enable-static --disable-f77 --disable-fc *MPICH 3.4.2 configuration:* $ ./configure --prefix="my directory" --enable-error-checking=all --enable-g=dbg CFLAGS=-fPIC CXXFLAGS=-fPIC FFLAGS=-fpic --enable-threads=runtime --enable-totalview --enable-static --disable-f77 --disable-fc --disable-fortran *--with-device=ch3:nemesis* --enable-error-messages=all Should I use another device or the behavior was modified between versions? Regards, Anatoly.

1 0

Re: [mpich-discuss] Why stuck in MPI_Finalize?
by Erik Schnetter 09 Dec '21

09 Dec '21

Thanks! -erik On Thu, Dec 9, 2021 at 12:06 PM Thomas Jahns via discuss <discuss(a)mpich.org> wrote: > > Hi, > > On 12/1/21 1:31 PM, Joachim Protze via discuss wrote: > > I think, calling MPI_Comm_free from the garbage collector in random > > ordering is not valid MPI code, since this function is a collective > > operation. The non-normative advice to implementers reads: > > > > Though collective, it is anticipated that this operation will normally > > be implemented to be local, though a debugging version of an MPI library > > might choose to synchronize. > > just an additional thought: for MPI_Win_free, ordering also matters, > _and_ it must be called for each and every corresponding window creation > before calling MPI_Finalize. > > Regards, Thomas > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss -- Erik Schnetter <schnetter(a)gmail.com> http://www.perimeterinstitute.ca/personal/eschnetter/

1 0