- discuss - mailman3.cels.anl.gov

Issue with MPICH 4.0b1 and MPI I/O and ROMIO...maybe?
by Thompson, Matt (GSFC-610.1)[SCIENCE SYSTEMS AND APPLICATIONS INC] 17 Dec '21

17 Dec '21

MPICH Discuss, So this is an odd one. Thanks to Hui Zhou from this list, I was able to build MPICH 4.0b1. I proceeded to build libraries and my application that I work on. And all seemed to compile just well. I then ran my model and...crash: Error opening file species.data -115 NetCDF: Error initializing for parallel access I did a bit of debugging and found that the crash was due to an environment variable that was set because my application mistakenly thought I was running Intel MPI (mainly because we didn't have detection for MPICH, so it defaulted to our "default" on this cluster of Intel MPI). When it sees Intel MPI, it sets: ROMIO_FSTYPE_FORCE="gpfs:" which we've found is useful when running with Intel MPI on our GPFS system. I whipped up a little netCDF reader executable that essentially does: call check( nf90_open(FILE_NAME, IOR(NF90_NOWRITE, NF90_MPIIO), comm=MPI_COMM_WORLD, info=MPI_INFO_NULL, ncid=ncid) ) and then: $ mpirun -np 2 ./simple_xy_rd_mpiio.exe OX_in(1,1:5): 7.1448361893544643E-008 1.7177123368128377E-007 2.8831971121690003E-007 3.7688394627366506E-007 4.6076729631749913E-007 OX_in(1,1:5): 7.1448361893544643E-008 1.7177123368128377E-007 2.8831971121690003E-007 3.7688394627366506E-007 4.6076729631749913E-007 *** SUCCESS reading example file species.data! *** SUCCESS reading example file species.data! $ ROMIO_FSTYPE_FORCE="gpfs:" mpirun -np 2 ./simple_xy_rd_mpiio.exe NetCDF: Error initializing for parallel access NetCDF: Error initializing for parallel access STOP Stopped STOP Stopped So, of course the "right" thing to do is not to set that. (Doctor, it hurts when I do this. So stop doing that.) But it got me wondering, is there perhaps a "better" way I should be building MPICH? Should this flag cause this sort of crash? Or does it mean I build MPICH/ROMIO incorrectly or incompletely (no GPFS support, say)? Thanks, Matt -- Matt Thompson, SSAI, Ld Scientific Programmer/Analyst NASA GSFC, Global Modeling and Assimilation Office Code 610.1, 8800 Greenbelt Rd, Greenbelt, MD 20771 Phone: 301-614-6712 Fax: 301-614-6246 http://science.gsfc.nasa.gov/sed/bio/matthew.thompson

1 0

Re: [mpich-discuss] [EXTERNAL] Re: Help Building MPICH 4.0b1: undefined references to hwloc
by Thompson, Matt (GSFC-610.1)[SCIENCE SYSTEMS AND APPLICATIONS INC] 17 Dec '21

17 Dec '21

Hui, Sorry for the late reply, but yep, that worked! I have MPICH! And it did helloworld. Now for complex things (aka hoping that ESMF's "mpich3" settings work with MPICH4!). Matt -- Matt Thompson, SSAI, Ld Scientific Programmer/Analyst NASA GSFC, Global Modeling and Assimilation Office Code 610.1, 8800 Greenbelt Rd, Greenbelt, MD 20771 Phone: 301-614-6712 Fax: 301-614-6246 http://science.gsfc.nasa.gov/sed/bio/matthew.thompson From: "Zhou, Hui" <zhouh(a)anl.gov> Date: Wednesday, December 15, 2021 at 10:44 AM To: "Thompson, Matt (GSFC-610.1)[SCIENCE SYSTEMS AND APPLICATIONS INC]" <matthew.thompson(a)nasa.gov>, "discuss(a)mpich.org" <discuss(a)mpich.org> Subject: Re: [EXTERNAL] Re: [mpich-discuss] Help Building MPICH 4.0b1: undefined references to hwloc Hi Matt, Thanks for posting config.log. I found the issue. Both mpich the library and hydra the program uses hwloc and independently tries to probe hwloc. Because hydra is configured as a subdir inside mpich configure, it inherits the CFLAGS from mpich, resulting in using embedded hwloc.h while linking with external libhwloc. We'll patch mpich to strengthen its config check. Meanwhile, you can pass --with-hwloc=embedded to the mpich configure to force both mpich and hydra to use embedded hwloc, which should bypass your issue. Thanks for reporting! -- Hui Zhou ________________________________ From: Thompson, Matt (GSFC-610.1)[SCIENCE SYSTEMS AND APPLICATIONS INC] <matthew.thompson(a)nasa.gov> Sent: Wednesday, December 15, 2021 7:13 AM To: discuss(a)mpich.org <discuss(a)mpich.org> Cc: Zhou, Hui <zhouh(a)anl.gov> Subject: Re: [EXTERNAL] Re: [mpich-discuss] Help Building MPICH 4.0b1: undefined references to hwloc Hui, Please find attached the requested config.log Matt -- Matt Thompson, SSAI, Ld Scientific Programmer/Analyst NASA GSFC, Global Modeling and Assimilation Office Code 610.1, 8800 Greenbelt Rd, Greenbelt, MD 20771 Phone: 301-614-6712 Fax: 301-614-6246 http://science.gsfc.nasa.gov/sed/bio/matthew.thompson From: "Zhou, Hui via discuss" <discuss(a)mpich.org> Reply-To: "discuss(a)mpich.org" <discuss(a)mpich.org> Date: Tuesday, December 14, 2021 at 6:14 PM To: "discuss(a)mpich.org" <discuss(a)mpich.org> Cc: "Zhou, Hui" <zhouh(a)anl.gov> Subject: [EXTERNAL] Re: [mpich-discuss] Help Building MPICH 4.0b1: undefined references to hwloc Hi Matt, Could post your src/pm/hydra/config.log? -- Hui Zhou ________________________________ From: Thompson, Matt (GSFC-610.1)[SCIENCE SYSTEMS AND APPLICATIONS INC] via discuss <discuss(a)mpich.org> Sent: Tuesday, December 14, 2021 2:40 PM To: discuss(a)mpich.org <discuss(a)mpich.org> Cc: Thompson, Matt (GSFC-610.1)[SCIENCE SYSTEMS AND APPLICATIONS INC] <matthew.thompson(a)nasa.gov> Subject: [mpich-discuss] Help Building MPICH 4.0b1: undefined references to hwloc MPICH Discuss, I'm sure I've done something stupid, but I'm not sure what. I tried searching google, but my google-fu failed me. Today I decided to try and build MPICH 4.0b1 on a cluster I'm on (NCCS Discover) using GCC 11.2. My configure line was quite boring: ./configure CC=gcc CXX=g++ FC=gfortran \ FCFLAGS=-fallow-argument-mismatch \ FFLAGS=-fallow-argument-mismatch \ --prefix=/discover/swdev/gmao_SIteam/MPI/mpich/4.0b1/gcc-11.2.0 The MPICH configure step was nice enough to let me know I needed those FFLAGS and FCFLAGS. And it configured just happily.[1] When I went to the make step I eventually it errored out with: make[3]: Entering directory '/gpfsm/dswdev/gmao_SIteam/MPI/src/mpich-4.0b1/src/pm/hydra' CCLD libpm.la CCLD hydra_persist /usr/bin/ld: /discover/swdev/gmao_SIteam/MPI/src/mpich-4.0b1/src/pm/hydra/.libs/libhydra.a(topo_hwloc.o): in function `HYDT_topo_hwloc_init': /discover/swdev/gmao_SIteam/MPI/src/mpich-4.0b1/src/pm/hydra/tools/topo/hwloc/topo_hwloc.c:577: undefined reference to `hwloc_topology_set_io_types_filter' collect2: error: ld returned 1 exit status Makefile:1286: recipe for target 'hydra_persist' failed make[3]: *** [hydra_persist] Error 1 Now, as far as I can tell from the configure output, I'm using the embedded hwloc: checking if hwloc meets minimum version requirement... no configure: ===== configuring modules/hwloc ===== ... ### Configuring hwloc core ### checking hwloc building mode... embedded configure: hwloc builddir: /discover/swdev/gmao_SIteam/MPI/src/mpich-4.0b1/modules/hwloc configure: hwloc srcdir: /discover/swdev/gmao_SIteam/MPI/src/mpich-4.0b1/modules/hwloc ... And I can't see anything in the output that it's somehow picking up on an hwloc I might have installed somewhere else (be it Open MPI or just hwloc by hand). Maybe there are flags I need to pass? I do see reference to --with-hwloc=embedded, but it looks like configure decided to use that itself. Thanks for any help, Matt [1] Note: I'm building on an Infiniband system so maybe I need to do a build with --with-device=ch4:ucx eventually, but that's a runtime thing. I figure it can't be involved with this error...right? -- Matt Thompson, SSAI, Ld Scientific Programmer/Analyst NASA GSFC, Global Modeling and Assimilation Office Code 610.1, 8800 Greenbelt Rd, Greenbelt, MD 20771 Phone: 301-614-6712 Fax: 301-614-6246 http://science.gsfc.nasa.gov/sed/bio/matthew.thompson

1 0

Re: [mpich-discuss] make testing fails when building from git(main)
by Zhou, Hui 16 Dec '21

16 Dec '21

Could you try with the latest git repo? We just merged this PR https://github.com/pmodels/mpich/pull/5722 to fix this issue. FYI, we have changed the way we configure test suite. In the main branch, test suite, i.e. test/mpi, is now configured independently from the mpich configure. To run test suite, after build and installing mpich, go to test/mpi, run ./autogen.sh ./configure make testing We added rule in the mpich Makefile so make testing will do above as a convenience. But if you encounter issues or need customized test configuration, try configure test/mpi separately. -- Hui Zhou [https://opengraph.githubassets.com/30d95a98b05bdcdc5c065f2271e7272120785210…]<https://github.com/pmodels/mpich/pull/5722> test/config: minor fixes by raffenet · Pull Request #5722 · pmodels/mpich<https://github.com/pmodels/mpich/pull/5722> Pull Request Description Improve configure output Fix dtpools build issue Author Checklist Provide Description Particularly focus on why, not what. Reference background, issues, test failures, ... github.com ________________________________ From: Gregor Corbin via discuss <discuss(a)mpich.org> Sent: Thursday, December 16, 2021 11:48 AM To: discuss(a)mpich.org <discuss(a)mpich.org> Cc: Gregor Corbin <g.corbin(a)fz-juelich.de> Subject: [mpich-discuss] make testing fails when building from git(main) Dear MPICH devs, I want to use the MPI test suite provided with MPICH to test an MPI wrapper library I am developing (the MPI wrapper for Score-P). As a first step, I tried to check out the latest version from git, build everything and then run the test suite with `make testing`. I can build MPICH itself without a problem and the example cpi program runs just fine. However, during `make testing` I run into lots of errors of the form In file included from dtpools.c:6: ../include/dtpools_internal.h:12:10: fatal error: mpi.h: No such file or directory 12 | #include "mpi.h" | ^~~~~~~ compilation terminated. This happens only with the git version, not with the tarball version(mpich-4.0b1). I attached all the program outputs and log files. Any help is appreciated. As I workaround I first built MPICH itself, then did './configure CC=mpicc ...', in the test/mpi directory and then directly use the runtests script. This works for now, but is surely not how it was intended. Kind regards, Gregor Corbin

1 0

make testing fails when building from git(main)
by Gregor Corbin 16 Dec '21

16 Dec '21

Dear MPICH devs, I want to use the MPI test suite provided with MPICH to test an MPI wrapper library I am developing (the MPI wrapper for Score-P). As a first step, I tried to check out the latest version from git, build everything and then run the test suite with `make testing`. I can build MPICH itself without a problem and the example cpi program runs just fine. However, during `make testing` I run into lots of errors of the form In file included from dtpools.c:6: ../include/dtpools_internal.h:12:10: fatal error: mpi.h: No such file or directory 12 | #include "mpi.h" | ^~~~~~~ compilation terminated. This happens only with the git version, not with the tarball version(mpich-4.0b1). I attached all the program outputs and log files. Any help is appreciated. As I workaround I first built MPICH itself, then did './configure CC=mpicc ...', in the test/mpi directory and then directly use the runtests script. This works for now, but is surely not how it was intended. Kind regards, Gregor Corbin

1 0

Re: [mpich-discuss] Question regarding fault tolerance difference 3.1 & 3.4.2
by Zhou, Hui 15 Dec '21

15 Dec '21

Hi Anatoly, I think the failure behavior is yet to be standardized, thus the behavior falls into the undefined behavior category. I suspect the different outcomes with MPICH 3.1 and 3.4.2 are from changes in the process manager. Could you try using hydra from 3.1 in your test with 3.4.2? -- Hui Zhou ________________________________ From: Anatoly G via discuss <discuss(a)mpich.org> Sent: Monday, December 13, 2021 7:18 AM To: mpich-discuss(a)mcs.anl.gov <mpich-discuss(a)mcs.anl.gov> Cc: Anatoly G <anatolyrishon(a)gmail.com> Subject: [mpich-discuss] Question regarding fault tolerance difference 3.1 & 3.4.2 Hi MPICH, I have a small program which has different outputs on MPICH 3.1 & MPICH 3.4.2 The program (code attached): 1. Master executes ping-pong with each one of the slaves (separately) 2. Each slave replies to master when it gets a message from master 3. One of the slaves simulates failre with command abort(). 4. Master recognizes that slave fails and continues to work with survived slaves only. I execute my program on ubuntu18. I use TCP as a transport layer. Execute command: mpiexec.hydra -genvall -disable-auto-cleanup -f MpiConfigMachines.txt -launcher=ssh -n 3 ft_ping_pong I expect that "ping pong" will continue till all 20 iterations will finish. MPICH 3.1 hydra works as expected only slave 1 fails and ping-pong continues between Master and slave2. MPICH 3.4.2 hydra process slave 2 fails together with slave 1. MPICH 3.1 configuration: ./configure --prefix="my directory" --enable-error-checking=runtime --enable-g=dbg CFLAGS=-fPIC CXXFLAGS=-fPIC FFLAGS=-fpic --enable-threads=runtime --enable-totalview --enable-static --disable-f77 --disable-fc MPICH 3.4.2 configuration: $ ./configure --prefix="my directory" --enable-error-checking=all --enable-g=dbg CFLAGS=-fPIC CXXFLAGS=-fPIC FFLAGS=-fpic --enable-threads=runtime --enable-totalview --enable-static --disable-f77 --disable-fc --disable-fortran --with-device=ch3:nemesis --enable-error-messages=all Should I use another device or the behavior was modified between versions? Regards, Anatoly.

1 0

Re: [mpich-discuss] [EXTERNAL] Re: Help Building MPICH 4.0b1: undefined references to hwloc
by Zhou, Hui 15 Dec '21

15 Dec '21

Hi Matt, Thanks for posting config.log. I found the issue. Both mpich the library and hydra the program uses hwloc and independently tries to probe hwloc. Because hydra is configured as a subdir inside mpich configure, it inherits the CFLAGS from mpich, resulting in using embedded hwloc.h while linking with external libhwloc. We'll patch mpich to strengthen its config check. Meanwhile, you can pass --with-hwloc=embedded to the mpich configure to force both mpich and hydra to use embedded hwloc, which should bypass your issue. Thanks for reporting! -- Hui Zhou ________________________________ From: Thompson, Matt (GSFC-610.1)[SCIENCE SYSTEMS AND APPLICATIONS INC] <matthew.thompson(a)nasa.gov> Sent: Wednesday, December 15, 2021 7:13 AM To: discuss(a)mpich.org <discuss(a)mpich.org> Cc: Zhou, Hui <zhouh(a)anl.gov> Subject: Re: [EXTERNAL] Re: [mpich-discuss] Help Building MPICH 4.0b1: undefined references to hwloc Hui, Please find attached the requested config.log Matt -- Matt Thompson, SSAI, Ld Scientific Programmer/Analyst NASA GSFC, Global Modeling and Assimilation Office Code 610.1, 8800 Greenbelt Rd, Greenbelt, MD 20771 Phone: 301-614-6712 Fax: 301-614-6246 http://science.gsfc.nasa.gov/sed/bio/matthew.thompson From: "Zhou, Hui via discuss" <discuss(a)mpich.org> Reply-To: "discuss(a)mpich.org" <discuss(a)mpich.org> Date: Tuesday, December 14, 2021 at 6:14 PM To: "discuss(a)mpich.org" <discuss(a)mpich.org> Cc: "Zhou, Hui" <zhouh(a)anl.gov> Subject: [EXTERNAL] Re: [mpich-discuss] Help Building MPICH 4.0b1: undefined references to hwloc Hi Matt, Could post your src/pm/hydra/config.log? -- Hui Zhou ________________________________ From: Thompson, Matt (GSFC-610.1)[SCIENCE SYSTEMS AND APPLICATIONS INC] via discuss <discuss(a)mpich.org> Sent: Tuesday, December 14, 2021 2:40 PM To: discuss(a)mpich.org <discuss(a)mpich.org> Cc: Thompson, Matt (GSFC-610.1)[SCIENCE SYSTEMS AND APPLICATIONS INC] <matthew.thompson(a)nasa.gov> Subject: [mpich-discuss] Help Building MPICH 4.0b1: undefined references to hwloc MPICH Discuss, I'm sure I've done something stupid, but I'm not sure what. I tried searching google, but my google-fu failed me. Today I decided to try and build MPICH 4.0b1 on a cluster I'm on (NCCS Discover) using GCC 11.2. My configure line was quite boring: ./configure CC=gcc CXX=g++ FC=gfortran \ FCFLAGS=-fallow-argument-mismatch \ FFLAGS=-fallow-argument-mismatch \ --prefix=/discover/swdev/gmao_SIteam/MPI/mpich/4.0b1/gcc-11.2.0 The MPICH configure step was nice enough to let me know I needed those FFLAGS and FCFLAGS. And it configured just happily.[1] When I went to the make step I eventually it errored out with: make[3]: Entering directory '/gpfsm/dswdev/gmao_SIteam/MPI/src/mpich-4.0b1/src/pm/hydra' CCLD libpm.la CCLD hydra_persist /usr/bin/ld: /discover/swdev/gmao_SIteam/MPI/src/mpich-4.0b1/src/pm/hydra/.libs/libhydra.a(topo_hwloc.o): in function `HYDT_topo_hwloc_init': /discover/swdev/gmao_SIteam/MPI/src/mpich-4.0b1/src/pm/hydra/tools/topo/hwloc/topo_hwloc.c:577: undefined reference to `hwloc_topology_set_io_types_filter' collect2: error: ld returned 1 exit status Makefile:1286: recipe for target 'hydra_persist' failed make[3]: *** [hydra_persist] Error 1 Now, as far as I can tell from the configure output, I'm using the embedded hwloc: checking if hwloc meets minimum version requirement... no configure: ===== configuring modules/hwloc ===== ... ### Configuring hwloc core ### checking hwloc building mode... embedded configure: hwloc builddir: /discover/swdev/gmao_SIteam/MPI/src/mpich-4.0b1/modules/hwloc configure: hwloc srcdir: /discover/swdev/gmao_SIteam/MPI/src/mpich-4.0b1/modules/hwloc ... And I can't see anything in the output that it's somehow picking up on an hwloc I might have installed somewhere else (be it Open MPI or just hwloc by hand). Maybe there are flags I need to pass? I do see reference to --with-hwloc=embedded, but it looks like configure decided to use that itself. Thanks for any help, Matt [1] Note: I'm building on an Infiniband system so maybe I need to do a build with --with-device=ch4:ucx eventually, but that's a runtime thing. I figure it can't be involved with this error...right? -- Matt Thompson, SSAI, Ld Scientific Programmer/Analyst NASA GSFC, Global Modeling and Assimilation Office Code 610.1, 8800 Greenbelt Rd, Greenbelt, MD 20771 Phone: 301-614-6712 Fax: 301-614-6246 http://science.gsfc.nasa.gov/sed/bio/matthew.thompson

1 0

Re: [mpich-discuss] [EXTERNAL] Re: Help Building MPICH 4.0b1: undefined references to hwloc
by Thompson, Matt (GSFC-610.1)[SCIENCE SYSTEMS AND APPLICATIONS INC] 15 Dec '21

15 Dec '21

Hui, Please find attached the requested config.log Matt -- Matt Thompson, SSAI, Ld Scientific Programmer/Analyst NASA GSFC, Global Modeling and Assimilation Office Code 610.1, 8800 Greenbelt Rd, Greenbelt, MD 20771 Phone: 301-614-6712 Fax: 301-614-6246 http://science.gsfc.nasa.gov/sed/bio/matthew.thompson From: "Zhou, Hui via discuss" <discuss(a)mpich.org> Reply-To: "discuss(a)mpich.org" <discuss(a)mpich.org> Date: Tuesday, December 14, 2021 at 6:14 PM To: "discuss(a)mpich.org" <discuss(a)mpich.org> Cc: "Zhou, Hui" <zhouh(a)anl.gov> Subject: [EXTERNAL] Re: [mpich-discuss] Help Building MPICH 4.0b1: undefined references to hwloc Hi Matt, Could post your src/pm/hydra/config.log? -- Hui Zhou ________________________________ From: Thompson, Matt (GSFC-610.1)[SCIENCE SYSTEMS AND APPLICATIONS INC] via discuss <discuss(a)mpich.org> Sent: Tuesday, December 14, 2021 2:40 PM To: discuss(a)mpich.org <discuss(a)mpich.org> Cc: Thompson, Matt (GSFC-610.1)[SCIENCE SYSTEMS AND APPLICATIONS INC] <matthew.thompson(a)nasa.gov> Subject: [mpich-discuss] Help Building MPICH 4.0b1: undefined references to hwloc MPICH Discuss, I'm sure I've done something stupid, but I'm not sure what. I tried searching google, but my google-fu failed me. Today I decided to try and build MPICH 4.0b1 on a cluster I'm on (NCCS Discover) using GCC 11.2. My configure line was quite boring: ./configure CC=gcc CXX=g++ FC=gfortran \ FCFLAGS=-fallow-argument-mismatch \ FFLAGS=-fallow-argument-mismatch \ --prefix=/discover/swdev/gmao_SIteam/MPI/mpich/4.0b1/gcc-11.2.0 The MPICH configure step was nice enough to let me know I needed those FFLAGS and FCFLAGS. And it configured just happily.[1] When I went to the make step I eventually it errored out with: make[3]: Entering directory '/gpfsm/dswdev/gmao_SIteam/MPI/src/mpich-4.0b1/src/pm/hydra' CCLD libpm.la CCLD hydra_persist /usr/bin/ld: /discover/swdev/gmao_SIteam/MPI/src/mpich-4.0b1/src/pm/hydra/.libs/libhydra.a(topo_hwloc.o): in function `HYDT_topo_hwloc_init': /discover/swdev/gmao_SIteam/MPI/src/mpich-4.0b1/src/pm/hydra/tools/topo/hwloc/topo_hwloc.c:577: undefined reference to `hwloc_topology_set_io_types_filter' collect2: error: ld returned 1 exit status Makefile:1286: recipe for target 'hydra_persist' failed make[3]: *** [hydra_persist] Error 1 Now, as far as I can tell from the configure output, I'm using the embedded hwloc: checking if hwloc meets minimum version requirement... no configure: ===== configuring modules/hwloc ===== ... ### Configuring hwloc core ### checking hwloc building mode... embedded configure: hwloc builddir: /discover/swdev/gmao_SIteam/MPI/src/mpich-4.0b1/modules/hwloc configure: hwloc srcdir: /discover/swdev/gmao_SIteam/MPI/src/mpich-4.0b1/modules/hwloc ... And I can't see anything in the output that it's somehow picking up on an hwloc I might have installed somewhere else (be it Open MPI or just hwloc by hand). Maybe there are flags I need to pass? I do see reference to --with-hwloc=embedded, but it looks like configure decided to use that itself. Thanks for any help, Matt [1] Note: I'm building on an Infiniband system so maybe I need to do a build with --with-device=ch4:ucx eventually, but that's a runtime thing. I figure it can't be involved with this error...right? -- Matt Thompson, SSAI, Ld Scientific Programmer/Analyst NASA GSFC, Global Modeling and Assimilation Office Code 610.1, 8800 Greenbelt Rd, Greenbelt, MD 20771 Phone: 301-614-6712 Fax: 301-614-6246 http://science.gsfc.nasa.gov/sed/bio/matthew.thompson

1 0

Re: [mpich-discuss] [EXTERNAL] Re: Help Building MPICH 4.0b1: undefined references to hwloc
by Thompson, Matt (GSFC-610.1)[SCIENCE SYSTEMS AND APPLICATIONS INC] 15 Dec '21

15 Dec '21

Brice, I'll take a look. Maybe I built lstopo or something at some point and it's hanging around... Matt -- Matt Thompson, SSAI, Ld Scientific Programmer/Analyst NASA GSFC, Global Modeling and Assimilation Office Code 610.1, 8800 Greenbelt Rd, Greenbelt, MD 20771 Phone: 301-614-6712 Fax: 301-614-6246 http://science.gsfc.nasa.gov/sed/bio/matthew.thompson From: Brice Goglin <Brice.Goglin(a)inria.fr> Date: Tuesday, December 14, 2021 at 5:37 PM To: <discuss(a)mpich.org> Cc: "Thompson, Matt (GSFC-610.1)[SCIENCE SYSTEMS AND APPLICATIONS INC]" <matthew.thompson(a)nasa.gov> Subject: [EXTERNAL] Re: [mpich-discuss] Help Building MPICH 4.0b1: undefined references to hwloc Le 14/12/2021 à 21:40, Thompson, Matt (GSFC-610.1)[SCIENCE SYSTEMS AND APPLICATIONS INC] via discuss a écrit : MPICH Discuss, I'm sure I've done something stupid, but I'm not sure what. I tried searching google, but my google-fu failed me. Today I decided to try and build MPICH 4.0b1 on a cluster I'm on (NCCS Discover) using GCC 11.2. My configure line was quite boring: ./configure CC=gcc CXX=g++ FC=gfortran \ FCFLAGS=-fallow-argument-mismatch \ FFLAGS=-fallow-argument-mismatch \ --prefix=/discover/swdev/gmao_SIteam/MPI/mpich/4.0b1/gcc-11.2.0 The MPICH configure step was nice enough to let me know I needed those FFLAGS and FCFLAGS. And it configured just happily.[1] When I went to the make step I eventually it errored out with: make[3]: Entering directory '/gpfsm/dswdev/gmao_SIteam/MPI/src/mpich-4.0b1/src/pm/hydra' CCLD libpm.la CCLD hydra_persist /usr/bin/ld: /discover/swdev/gmao_SIteam/MPI/src/mpich-4.0b1/src/pm/hydra/.libs/libhydra.a(topo_hwloc.o): in function `HYDT_topo_hwloc_init': /discover/swdev/gmao_SIteam/MPI/src/mpich-4.0b1/src/pm/hydra/tools/topo/hwloc/topo_hwloc.c:577: undefined reference to `hwloc_topology_set_io_types_filter' Hello I think this error usually occurs when compiling against hwloc 2.x (such as the one embedded inside MPICH) but linking against an old hwloc 1.x. Do you have such an old hwloc installed on this machine? The first line below likely says yes. I don't know why configure would mix them like this, but you should be able to avoid the error by uninstalling the old hwloc (or with a chmod 0 on its libhwloc.so during your configure). Brice checking if hwloc meets minimum version requirement... no configure: ===== configuring modules/hwloc ===== ... ### Configuring hwloc core ### checking hwloc building mode... embedded configure: hwloc builddir: /discover/swdev/gmao_SIteam/MPI/src/mpich-4.0b1/modules/hwloc configure: hwloc srcdir: /discover/swdev/gmao_SIteam/MPI/src/mpich-4.0b1/modules/hwloc ... And I can't see anything in the output that it's somehow picking up on an hwloc I might have installed somewhere else (be it Open MPI or just hwloc by hand). Maybe there are flags I need to pass? I do see reference to --with-hwloc=embedded, but it looks like configure decided to use that itself. Thanks for any help, Matt [1] Note: I'm building on an Infiniband system so maybe I need to do a build with --with-device=ch4:ucx eventually, but that's a runtime thing. I figure it can't be involved with this error...right? -- Matt Thompson, SSAI, Ld Scientific Programmer/Analyst NASA GSFC, Global Modeling and Assimilation Office Code 610.1, 8800 Greenbelt Rd, Greenbelt, MD 20771 Phone: 301-614-6712 Fax: 301-614-6246 http://science.gsfc.nasa.gov/sed/bio/matthew.thompson _______________________________________________ discuss mailing list discuss(a)mpich.org<mailto:[email protected]> To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Help Building MPICH 4.0b1: undefined references to hwloc
by Zhou, Hui 14 Dec '21

14 Dec '21

Hi Matt, Could post your src/pm/hydra/config.log? -- Hui Zhou ________________________________ From: Thompson, Matt (GSFC-610.1)[SCIENCE SYSTEMS AND APPLICATIONS INC] via discuss <discuss(a)mpich.org> Sent: Tuesday, December 14, 2021 2:40 PM To: discuss(a)mpich.org <discuss(a)mpich.org> Cc: Thompson, Matt (GSFC-610.1)[SCIENCE SYSTEMS AND APPLICATIONS INC] <matthew.thompson(a)nasa.gov> Subject: [mpich-discuss] Help Building MPICH 4.0b1: undefined references to hwloc MPICH Discuss, I'm sure I've done something stupid, but I'm not sure what. I tried searching google, but my google-fu failed me. Today I decided to try and build MPICH 4.0b1 on a cluster I'm on (NCCS Discover) using GCC 11.2. My configure line was quite boring: ./configure CC=gcc CXX=g++ FC=gfortran \ FCFLAGS=-fallow-argument-mismatch \ FFLAGS=-fallow-argument-mismatch \ --prefix=/discover/swdev/gmao_SIteam/MPI/mpich/4.0b1/gcc-11.2.0 The MPICH configure step was nice enough to let me know I needed those FFLAGS and FCFLAGS. And it configured just happily.[1] When I went to the make step I eventually it errored out with: make[3]: Entering directory '/gpfsm/dswdev/gmao_SIteam/MPI/src/mpich-4.0b1/src/pm/hydra' CCLD libpm.la CCLD hydra_persist /usr/bin/ld: /discover/swdev/gmao_SIteam/MPI/src/mpich-4.0b1/src/pm/hydra/.libs/libhydra.a(topo_hwloc.o): in function `HYDT_topo_hwloc_init': /discover/swdev/gmao_SIteam/MPI/src/mpich-4.0b1/src/pm/hydra/tools/topo/hwloc/topo_hwloc.c:577: undefined reference to `hwloc_topology_set_io_types_filter' collect2: error: ld returned 1 exit status Makefile:1286: recipe for target 'hydra_persist' failed make[3]: *** [hydra_persist] Error 1 Now, as far as I can tell from the configure output, I'm using the embedded hwloc: checking if hwloc meets minimum version requirement... no configure: ===== configuring modules/hwloc ===== ... ### Configuring hwloc core ### checking hwloc building mode... embedded configure: hwloc builddir: /discover/swdev/gmao_SIteam/MPI/src/mpich-4.0b1/modules/hwloc configure: hwloc srcdir: /discover/swdev/gmao_SIteam/MPI/src/mpich-4.0b1/modules/hwloc ... And I can't see anything in the output that it's somehow picking up on an hwloc I might have installed somewhere else (be it Open MPI or just hwloc by hand). Maybe there are flags I need to pass? I do see reference to --with-hwloc=embedded, but it looks like configure decided to use that itself. Thanks for any help, Matt [1] Note: I'm building on an Infiniband system so maybe I need to do a build with --with-device=ch4:ucx eventually, but that's a runtime thing. I figure it can't be involved with this error...right? -- Matt Thompson, SSAI, Ld Scientific Programmer/Analyst NASA GSFC, Global Modeling and Assimilation Office Code 610.1, 8800 Greenbelt Rd, Greenbelt, MD 20771 Phone: 301-614-6712 Fax: 301-614-6246 http://science.gsfc.nasa.gov/sed/bio/matthew.thompson

1 0

Re: [mpich-discuss] Help Building MPICH 4.0b1: undefined references to hwloc
by Brice Goglin 14 Dec '21

14 Dec '21

Le 14/12/2021 à 21:40, Thompson, Matt (GSFC-610.1)[SCIENCE SYSTEMS AND APPLICATIONS INC] via discuss a écrit : > > MPICH Discuss, > > I'm sure I've done something stupid, but I'm not sure what. I tried > searching google, but my google-fu failed me. > > Today I decided to try and build MPICH 4.0b1 on a cluster I'm on (NCCS > Discover) using GCC 11.2. My configure line was quite boring: > > ./configure CC=gcc CXX=g++ FC=gfortran \ > > FCFLAGS=-fallow-argument-mismatch \ > > FFLAGS=-fallow-argument-mismatch \ > > --prefix=/discover/swdev/gmao_SIteam/MPI/mpich/4.0b1/gcc-11.2.0 > > The MPICH configure step was nice enough to let me know I needed those > FFLAGS and FCFLAGS. And it configured just happily.[1] > > When I went to the make step I eventually it errored out with: > > make[3]: Entering directory > '/gpfsm/dswdev/gmao_SIteam/MPI/src/mpich-4.0b1/src/pm/hydra' > > CCLD libpm.la > > CCLD hydra_persist > > /usr/bin/ld: > /discover/swdev/gmao_SIteam/MPI/src/mpich-4.0b1/src/pm/hydra/.libs/libhydra.a(topo_hwloc.o): > in function `HYDT_topo_hwloc_init': > > /discover/swdev/gmao_SIteam/MPI/src/mpich-4.0b1/src/pm/hydra/tools/topo/hwloc/topo_hwloc.c:577: > undefined reference to `hwloc_topology_set_io_types_filter' > Hello I think this error usually occurs when compiling against hwloc 2.x (such as the one embedded inside MPICH) but linking against an old hwloc 1.x. Do you have such an old hwloc installed on this machine? The first line below likely says yes. I don't know why configure would mix them like this, but you should be able to avoid the error by uninstalling the old hwloc (or with a chmod 0 on its libhwloc.so during your configure). Brice > checking if hwloc meets minimum version requirement... no > > configure: ===== configuring modules/hwloc ===== > > ... > > ### Configuring hwloc core > > ### > > checking hwloc building mode... embedded > > configure: hwloc builddir: > /discover/swdev/gmao_SIteam/MPI/src/mpich-4.0b1/modules/hwloc > > configure: hwloc srcdir: > /discover/swdev/gmao_SIteam/MPI/src/mpich-4.0b1/modules/hwloc > > ... > > > And I can't see anything in the output that it's somehow picking up on > an hwloc I might have installed somewhere else (be it Open MPI or just > hwloc by hand). > > Maybe there are flags I need to pass? I do see reference to > --with-hwloc=embedded, but it looks like configure decided to use that > itself. > > Thanks for any help, > > Matt > > [1] Note: I'm building on an Infiniband system so maybe I need to do a > build with --with-device=ch4:ucx eventually, but that's a runtime > thing. I figure it can't be involved with this error...right? > > -- > > Matt Thompson, SSAI, Ld Scientific Programmer/Analyst > > NASA GSFC, Global Modeling and Assimilation Office > > Code 610.1, 8800 Greenbelt Rd, Greenbelt, MD 20771 > > Phone: 301-614-6712 Fax: 301-614-6246 > > _http://science.gsfc.nasa.gov/sed/bio/matthew.thompson_ > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss

1 0