September 2020 - discuss - mailman3.cels.anl.gov

Problem configuring mpich-3.4a3 and 3.4a2 --with-device=ch4:ucx
by Audet, Martin 22 Sep '20

22 Sep '20

Hello MPICH_Developers, I am having trouble running the configure script on two latest mpich release (3.4a2 and 3.4a3) when I use the flag --with-device=ch4:ucx (no such problems with ch3:sock or ch3:nemesis). It seems that the generated Makefile often contains two strange compilation and link flags: -Iyes/include and -Lyes/lib. To be able to compile the library I had to use the following command to remove those strange flags: find . -name Makefile -exec sed -i -e 's/-Iyes\/include\>//g' -e 's/-Lyes\/lib\>//g' {} \; After this step I am able to compile and the resulting library seems ok and appears to performs well (not extensive testing). For your information I call the configure script like this: ./configure --with-device=ch4:ucx --with-hcoll=/opt/mellanox/hcoll --with-pmix --prefix=/home/publique/depot/mpi/mpich-ch4_ucx-3.4a3 --enable-fast=all --enable-romio --with-file-system=ufs+nfs --enable-shared --enable-sharedlibs=gcc I had experienced this problem on the following three configurations: mpich-3.4a2 CentOS 7.6 MOFED 4.7 mpich-3.4a3 CentOS 7.6 MOFED 4.7 mpich-3.4a3 CentOS 7.8 MOFED 4.9 The architecture is x86_64. Could someone take a look at this problem ? Thanks, Martin Audet

1 0

Re: [mpich-discuss] Intermittent hang in MPI_Finalize with PGI 20.1
by Zhou, Hui 21 Sep '20

21 Sep '20

Hi Kent, I just tried with PGI 20.1 on mpich v3.3.2. I think I hit a hang once when I was checking it manually, but then I couldn’t reproduce it even after 1000 times repeat. Anyway, we have made some changes to the ch3 header structures that potentially makes the code more standard compliant. Could you try with latest development on github and see if the issue still occur on your end? -- Hui Zhou From: Kent Cheung via discuss <discuss(a)mpich.org> Reply-To: "discuss(a)mpich.org" <discuss(a)mpich.org> Date: Monday, September 21, 2020 at 4:52 AM To: "discuss(a)mpich.org" <discuss(a)mpich.org> Cc: Kent Cheung <Kent.Cheung(a)arm.com> Subject: Re: [mpich-discuss] Intermittent hang in MPI_Finalize with PGI 20.1 Are there any updates on this issue? Thanks. Kent ________________________________ From: Raffenetti, Kenneth J. <raffenet(a)mcs.anl.gov> Sent: 24 June 2020 17:04 To: discuss(a)mpich.org <discuss(a)mpich.org> Cc: Kent Cheung <Kent.Cheung(a)arm.com> Subject: Re: [mpich-discuss] Intermittent hang in MPI_Finalize with PGI 20.1 Hi Kent, Thanks for your report. We have not seen this issue with any compiler/OS combination in our nightly tests. We are using PGI 19.4 at this time. I will request 20.1 be installed so we can investigate further. Ken On 6/23/20, 8:26 AM, "Kent Cheung via discuss" <discuss(a)mpich.org> wrote: I'm running into an issue where processes sometimes hang when calling MPI_Finalize. This happens with both versions 3.3.2 and 3.4a2 on a single node RedHat 7.5 x86-64 machine, when MPICH is compiled with PGI 20.1 with these configuration flags --enable-debug --enable-shared --enable-debuginfo --enable-sharedlib=gcc If I change the default optimization level (-O2) by configuring with --enable-fast=O1 as well, the hang doesn't occur. Another data point is that the hang does not occur with PGI 19.5 with either optimization levels. I have been testing with the cpi.c code in the examples folder built with just mpicc cpi.c mpiexec -n 3 ./a.out Here is a the backtrace from one of the processes that is hanging (gdb) bt #0 MPID_nem_mpich_blocking_recv () at /tmp/mpich-3.3.2/build/../src/mpid/ch3/channels/nemesis/include/mpid_nem_inline.h:1038 #1 MPIDI_CH3I_Progress () at ../src/mpid/ch3/channels/nemesis/src/ch3_progress.c:506 #2 0x00000000004fc88d in MPIDI_CH3U_VC_WaitForClose () at ../src/mpid/ch3/src/ch3u_handle_connection.c:383 #3 0x0000000000442364 in MPID_Finalize () at ../src/mpid/ch3/src/mpid_finalize.c:110 #4 0x0000000000408621 in PMPI_Finalize () at ../src/mpi/init/finalize.c:260 #5 0x00000000004023e5 in main () at cpi.c:59 Is there a potential fix to be made to MPICH to prevent processes hanging when MPICH is compiled with the default optimization level? Thanks, Kent IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you. IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.

1 0

Re: [mpich-discuss] Intermittent hang in MPI_Finalize with PGI 20.1
by Kent Cheung 21 Sep '20

21 Sep '20

Are there any updates on this issue? Thanks. Kent ________________________________ From: Raffenetti, Kenneth J. <raffenet(a)mcs.anl.gov> Sent: 24 June 2020 17:04 To: discuss(a)mpich.org <discuss(a)mpich.org> Cc: Kent Cheung <Kent.Cheung(a)arm.com> Subject: Re: [mpich-discuss] Intermittent hang in MPI_Finalize with PGI 20.1 Hi Kent, Thanks for your report. We have not seen this issue with any compiler/OS combination in our nightly tests. We are using PGI 19.4 at this time. I will request 20.1 be installed so we can investigate further. Ken On 6/23/20, 8:26 AM, "Kent Cheung via discuss" <discuss(a)mpich.org> wrote: I'm running into an issue where processes sometimes hang when calling MPI_Finalize. This happens with both versions 3.3.2 and 3.4a2 on a single node RedHat 7.5 x86-64 machine, when MPICH is compiled with PGI 20.1 with these configuration flags --enable-debug --enable-shared --enable-debuginfo --enable-sharedlib=gcc If I change the default optimization level (-O2) by configuring with --enable-fast=O1 as well, the hang doesn't occur. Another data point is that the hang does not occur with PGI 19.5 with either optimization levels. I have been testing with the cpi.c code in the examples folder built with just mpicc cpi.c mpiexec -n 3 ./a.out Here is a the backtrace from one of the processes that is hanging (gdb) bt #0 MPID_nem_mpich_blocking_recv () at /tmp/mpich-3.3.2/build/../src/mpid/ch3/channels/nemesis/include/mpid_nem_inline.h:1038 #1 MPIDI_CH3I_Progress () at ../src/mpid/ch3/channels/nemesis/src/ch3_progress.c:506 #2 0x00000000004fc88d in MPIDI_CH3U_VC_WaitForClose () at ../src/mpid/ch3/src/ch3u_handle_connection.c:383 #3 0x0000000000442364 in MPID_Finalize () at ../src/mpid/ch3/src/mpid_finalize.c:110 #4 0x0000000000408621 in PMPI_Finalize () at ../src/mpi/init/finalize.c:260 #5 0x00000000004023e5 in main () at cpi.c:59 Is there a potential fix to be made to MPICH to prevent processes hanging when MPICH is compiled with the default optimization level? Thanks, Kent IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you. IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.

1 0

Re: [mpich-discuss] Help resolving hwloc_pci_compare_busids: Assertion `0' failed error
by David Alexander 04 Sep '20

04 Sep '20

Thanks! Setting HWLOC_COMPONENTS=-pci,-linuxio worked! BTW, I am not absolutely sure it was in the MPICH code. I do know that $ mpiexec -np 2 hostname worked and did not give the error, but of course that doesn’t do an of the MPI operations in the target executable “hostname". Thanks again! dave > On Sep 4, 2020, at 10:11 AM, Brice Goglin via discuss <discuss(a)mpich.org> wrote: > > Hello > > I think we've seen this issue once in the past, but I couldn't find it > in the archives yet. > > Anyway, it's not MPICH's fault. Can you open an issue on > github/open-mpi/hwloc, with the output of lspci -vt ? If you can build > hwloc, run lstopo to check whether the issue occurs, and get a gdb > backtrace, it'd be great. I am going to add some debug printf before > this assert to avoid having to run gdb next time. > > In the meantime, try setting HWLOC_COMPONENTS=-pci,-linuxio in the > environment so that hwloc's PCI backends are disabled. Hopefully, MPICH > won't fail because of this. > > Brice > > > > Le 04/09/2020 à 17:47, David Alexander via discuss a écrit : >> When I execute a distributed copy of MPICH 3.3.2 with a program >> built against that same copy of MPICH I see the following error on >> the target machine: >> >> $ mpiexec -np 2 myprogram >> >> myprogram: /builds/mpich-3.3.2/src/hwloc/hwloc/pci-common.c:259: hwloc_pci_compare_busids: Assertion `0' failed. >> myprogram: /builds/mpich-3.3.2/src/hwloc/hwloc/pci-common.c:259: hwloc_pci_compare_busids: Assertion `0' failed. >> >> The target machine where mpiexec is running on is an SGI with SLES12 and the >> machine that I built on is Centos7. >> >> The library dependency of mpiexec.hydra and myprogram are below: >> >> $ lddtree /installation/bin/mpiexec.hydra >> mpiexec.hydra => /installation/bin/mpiexec.hydra (interpreter => /lib64/ld-linux-x86-64.so.2) >> libm.so.6 => /lib64/libm.so.6 >> ld-linux-x86-64.so.2 => /lib64/ld-linux-x86-64.so.2 >> libudev.so.1 => /lib64/libudev.so.1 >> librt.so.1 => /lib64/librt.so.1 >> libcap.so.2 => /lib64/libcap.so.2 >> libattr.so.1 => /lib64/libattr.so.1 >> libdw.so.1 => /lib64/libdw.so.1 >> libelf.so.1 => /lib64/libelf.so.1 >> libz.so.1 => /lib64/libz.so.1 >> liblzma.so.5 => /lib64/liblzma.so.5 >> libbz2.so.1 => /lib64/libbz2.so.1 >> libdl.so.2 => /lib64/libdl.so.2 >> libgcc_s.so.1 => /lib64/libgcc_s.so.1 >> libpciaccess.so.0 => /lib64/libpciaccess.so.0 >> libxml2.so.2 => /lib64/libxml2.so.2 >> libpthread.so.0 => /lib64/libpthread.so.0 >> libc.so.6 => /lib64/libc.so.6 >> >> $ lddtree /installation/bin/myprogram >> myprogram => /installation/bin/myprogram (interpreter => /lib64/ld-linux-x86-64.so.2) >> libHYPRE.so => /installation/bin/../../lib/libHYPRE.so >> libz.so.1 => /installation/bin/../../lib/libz.so.1 >> libgfortran.so.5 => /installation/bin/../../lib/libgfortran.so.5 >> libquadmath.so.0 => /installation/bin/../../lib/libquadmath.so.0 >> libpthread.so.0 => /lib64/libpthread.so.0 >> libutil.so.1 => /lib64/libutil.so.1 >> libdl.so.2 => /lib64/libdl.so.2 >> librt.so.1 => /lib64/librt.so.1 >> libcurand.so.10 => /installation/bin/../../lib/libcurand.so.10 >> libmpicxx.so.12 => /installation/bin/../../lib/libmpicxx.so.12 >> libudev.so.1 => /lib64/libudev.so.1 >> libcap.so.2 => /lib64/libcap.so.2 >> libattr.so.1 => /lib64/libattr.so.1 >> libdw.so.1 => /lib64/libdw.so.1 >> libelf.so.1 => /lib64/libelf.so.1 >> liblzma.so.5 => /lib64/liblzma.so.5 >> libbz2.so.1 => /lib64/libbz2.so.1 >> libpciaccess.so.0 => /lib64/libpciaccess.so.0 >> libxml2.so.2 => /lib64/libxml2.so.2 >> libmpi.so.12 => /installation/bin/../../lib/libmpi.so.12 >> libpython2.7.so.1.0 => /installation/bin/../../lib/libpython2.7.so.1.0 >> libstdc++.so.6 => /installation/bin/../../lib/libstdc++.so.6 >> libm.so.6 => /lib64/libm.so.6 >> libgomp.so.1 => /installation/bin/../../lib/libgomp.so.1 >> libgcc_s.so.1 => /installation/bin/../../lib/libgcc_s.so.1 >> libc.so.6 => /lib64/libc.so.6 >> ld-linux-x86-64.so.2 => /lib64/ld-linux-x86-64.so.2 >> >> Thanks! >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Unable to use --enable-fast=O3,ndebug
by Raffenetti, Kenneth J. 04 Sep '20

04 Sep '20

I updated the "Report a bug" link to go to GitHub. I also created https://github.com/pmodels/mpich/issues/4773 to capture this specific issue. Ken On 8/26/20, 12:54 AM, "Jeff Hammond via discuss" <discuss(a)mpich.org> wrote: A GitHub issue seems appropriate here (https://github.com/pmodels/mpich/issues) Jeff On Tue, Aug 25, 2020 at 2:09 PM William Gropp via discuss <discuss(a)mpich.org> wrote: (If there is a better way to submit these, let me know. The “bug report” link doesn’t work from mpich.org <http://mpich.org>) —enable-fast in the top level configure accepts multiple, comma-separated items. The configure in modules/yaksa/configure.ac <http://configure.ac> expects (without checking) an O level only. This causes the configure step to fail with an unrelated error (yaksa thinks that it can’t fine pthreads). Bill William Gropp Director and Chief Scientist, NCSA Thomas M. Siebel Chair in Computer Science University of Illinois Urbana-Champaign _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss -- Jeff Hammond jeff.science(a)gmail.com http://jeffhammond.github.io/

1 0

Re: [mpich-discuss] Help resolving hwloc_pci_compare_busids: Assertion `0' failed error
by Brice Goglin 04 Sep '20

04 Sep '20

Hello I think we've seen this issue once in the past, but I couldn't find it in the archives yet. Anyway, it's not MPICH's fault. Can you open an issue on github/open-mpi/hwloc, with the output of lspci -vt ? If you can build hwloc, run lstopo to check whether the issue occurs, and get a gdb backtrace, it'd be great. I am going to add some debug printf before this assert to avoid having to run gdb next time. In the meantime, try setting HWLOC_COMPONENTS=-pci,-linuxio in the environment so that hwloc's PCI backends are disabled. Hopefully, MPICH won't fail because of this. Brice Le 04/09/2020 à 17:47, David Alexander via discuss a écrit : > When I execute a distributed copy of MPICH 3.3.2 with a program > built against that same copy of MPICH I see the following error on > the target machine: > > $ mpiexec -np 2 myprogram > > myprogram: /builds/mpich-3.3.2/src/hwloc/hwloc/pci-common.c:259: hwloc_pci_compare_busids: Assertion `0' failed. > myprogram: /builds/mpich-3.3.2/src/hwloc/hwloc/pci-common.c:259: hwloc_pci_compare_busids: Assertion `0' failed. > > The target machine where mpiexec is running on is an SGI with SLES12 and the > machine that I built on is Centos7. > > The library dependency of mpiexec.hydra and myprogram are below: > > $ lddtree /installation/bin/mpiexec.hydra > mpiexec.hydra => /installation/bin/mpiexec.hydra (interpreter => /lib64/ld-linux-x86-64.so.2) > libm.so.6 => /lib64/libm.so.6 > ld-linux-x86-64.so.2 => /lib64/ld-linux-x86-64.so.2 > libudev.so.1 => /lib64/libudev.so.1 > librt.so.1 => /lib64/librt.so.1 > libcap.so.2 => /lib64/libcap.so.2 > libattr.so.1 => /lib64/libattr.so.1 > libdw.so.1 => /lib64/libdw.so.1 > libelf.so.1 => /lib64/libelf.so.1 > libz.so.1 => /lib64/libz.so.1 > liblzma.so.5 => /lib64/liblzma.so.5 > libbz2.so.1 => /lib64/libbz2.so.1 > libdl.so.2 => /lib64/libdl.so.2 > libgcc_s.so.1 => /lib64/libgcc_s.so.1 > libpciaccess.so.0 => /lib64/libpciaccess.so.0 > libxml2.so.2 => /lib64/libxml2.so.2 > libpthread.so.0 => /lib64/libpthread.so.0 > libc.so.6 => /lib64/libc.so.6 > > $ lddtree /installation/bin/myprogram > myprogram => /installation/bin/myprogram (interpreter => /lib64/ld-linux-x86-64.so.2) > libHYPRE.so => /installation/bin/../../lib/libHYPRE.so > libz.so.1 => /installation/bin/../../lib/libz.so.1 > libgfortran.so.5 => /installation/bin/../../lib/libgfortran.so.5 > libquadmath.so.0 => /installation/bin/../../lib/libquadmath.so.0 > libpthread.so.0 => /lib64/libpthread.so.0 > libutil.so.1 => /lib64/libutil.so.1 > libdl.so.2 => /lib64/libdl.so.2 > librt.so.1 => /lib64/librt.so.1 > libcurand.so.10 => /installation/bin/../../lib/libcurand.so.10 > libmpicxx.so.12 => /installation/bin/../../lib/libmpicxx.so.12 > libudev.so.1 => /lib64/libudev.so.1 > libcap.so.2 => /lib64/libcap.so.2 > libattr.so.1 => /lib64/libattr.so.1 > libdw.so.1 => /lib64/libdw.so.1 > libelf.so.1 => /lib64/libelf.so.1 > liblzma.so.5 => /lib64/liblzma.so.5 > libbz2.so.1 => /lib64/libbz2.so.1 > libpciaccess.so.0 => /lib64/libpciaccess.so.0 > libxml2.so.2 => /lib64/libxml2.so.2 > libmpi.so.12 => /installation/bin/../../lib/libmpi.so.12 > libpython2.7.so.1.0 => /installation/bin/../../lib/libpython2.7.so.1.0 > libstdc++.so.6 => /installation/bin/../../lib/libstdc++.so.6 > libm.so.6 => /lib64/libm.so.6 > libgomp.so.1 => /installation/bin/../../lib/libgomp.so.1 > libgcc_s.so.1 => /installation/bin/../../lib/libgcc_s.so.1 > libc.so.6 => /lib64/libc.so.6 > ld-linux-x86-64.so.2 => /lib64/ld-linux-x86-64.so.2 > > Thanks! > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss

1 0

Help resolving hwloc_pci_compare_busids: Assertion `0' failed error
by David Alexander 04 Sep '20

04 Sep '20

When I execute a distributed copy of MPICH 3.3.2 with a program built against that same copy of MPICH I see the following error on the target machine: $ mpiexec -np 2 myprogram myprogram: /builds/mpich-3.3.2/src/hwloc/hwloc/pci-common.c:259: hwloc_pci_compare_busids: Assertion `0' failed. myprogram: /builds/mpich-3.3.2/src/hwloc/hwloc/pci-common.c:259: hwloc_pci_compare_busids: Assertion `0' failed. The target machine where mpiexec is running on is an SGI with SLES12 and the machine that I built on is Centos7. The library dependency of mpiexec.hydra and myprogram are below: $ lddtree /installation/bin/mpiexec.hydra mpiexec.hydra => /installation/bin/mpiexec.hydra (interpreter => /lib64/ld-linux-x86-64.so.2) libm.so.6 => /lib64/libm.so.6 ld-linux-x86-64.so.2 => /lib64/ld-linux-x86-64.so.2 libudev.so.1 => /lib64/libudev.so.1 librt.so.1 => /lib64/librt.so.1 libcap.so.2 => /lib64/libcap.so.2 libattr.so.1 => /lib64/libattr.so.1 libdw.so.1 => /lib64/libdw.so.1 libelf.so.1 => /lib64/libelf.so.1 libz.so.1 => /lib64/libz.so.1 liblzma.so.5 => /lib64/liblzma.so.5 libbz2.so.1 => /lib64/libbz2.so.1 libdl.so.2 => /lib64/libdl.so.2 libgcc_s.so.1 => /lib64/libgcc_s.so.1 libpciaccess.so.0 => /lib64/libpciaccess.so.0 libxml2.so.2 => /lib64/libxml2.so.2 libpthread.so.0 => /lib64/libpthread.so.0 libc.so.6 => /lib64/libc.so.6 $ lddtree /installation/bin/myprogram myprogram => /installation/bin/myprogram (interpreter => /lib64/ld-linux-x86-64.so.2) libHYPRE.so => /installation/bin/../../lib/libHYPRE.so libz.so.1 => /installation/bin/../../lib/libz.so.1 libgfortran.so.5 => /installation/bin/../../lib/libgfortran.so.5 libquadmath.so.0 => /installation/bin/../../lib/libquadmath.so.0 libpthread.so.0 => /lib64/libpthread.so.0 libutil.so.1 => /lib64/libutil.so.1 libdl.so.2 => /lib64/libdl.so.2 librt.so.1 => /lib64/librt.so.1 libcurand.so.10 => /installation/bin/../../lib/libcurand.so.10 libmpicxx.so.12 => /installation/bin/../../lib/libmpicxx.so.12 libudev.so.1 => /lib64/libudev.so.1 libcap.so.2 => /lib64/libcap.so.2 libattr.so.1 => /lib64/libattr.so.1 libdw.so.1 => /lib64/libdw.so.1 libelf.so.1 => /lib64/libelf.so.1 liblzma.so.5 => /lib64/liblzma.so.5 libbz2.so.1 => /lib64/libbz2.so.1 libpciaccess.so.0 => /lib64/libpciaccess.so.0 libxml2.so.2 => /lib64/libxml2.so.2 libmpi.so.12 => /installation/bin/../../lib/libmpi.so.12 libpython2.7.so.1.0 => /installation/bin/../../lib/libpython2.7.so.1.0 libstdc++.so.6 => /installation/bin/../../lib/libstdc++.so.6 libm.so.6 => /lib64/libm.so.6 libgomp.so.1 => /installation/bin/../../lib/libgomp.so.1 libgcc_s.so.1 => /installation/bin/../../lib/libgcc_s.so.1 libc.so.6 => /lib64/libc.so.6 ld-linux-x86-64.so.2 => /lib64/ld-linux-x86-64.so.2 Thanks!

1 0