June 2021 - discuss - mailman3.cels.anl.gov

Re: [mpich-discuss] libfabric+psm2 performance
by Bland, Wesley 16 Jun '21

16 Jun '21

I see. My guess (and I’ll let Hui confirm) is that you’re not actually using the PSM2 capabilities when you compile with —with-device=ch4:ofi. Instead, you’re getting a backup capability set that is supposed to work with any provider. When you switch to —with-device=ch4:ofi:psm2, you’re forcing MPICH to use the PSM2 capabilities (as we currently think of them for OFI 1.11 or whatever we expect right now) and your version of PSM2 doesn’t support that, so it crashes during initialization because it can’t find a provider that meets its requirements. If you set the environment variable MPIR_CVAR_CH4_OFI_CAPABILITY_SETS_DEBUG, it will print out the capability set (and provider) that MPICH is using so you can confirm. Assuming that’s the case, there’s probably some manual set of CVARs that will get you the right set of capabilities, but I’m not sure what it would be off the top of my head. 1.5 is pretty old at this point so it’s disappeared from my brain. :) I’m not that surprised that MVAPICH might be winning with an older version of OFI. My understanding is that it’s still on CH3 (I might be wrong here) and isn’t using the CH4 capability set code. I think the capability sets probably improve things in MPICH when matching our expected versions, but could cause these sorts of legacy issues. Just a guess though. Good luck! Wes On Jun 16, 2021, at 2:48 AM, Antonio J. Peña <antonio.pena(a)bsc.es<mailto:[email protected]>> wrote: Hi Wesley, Happy to hear from you. With that setting I cannot get out from this runtime error at init: Abort(69832847) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init: Other MPI error, error stack: PMPI_Init(98)............: MPI_Init(argc=0x7ffd4776159c, argv=0x7ffd47761590) failed MPII_Init_thread(196)....: MPID_Init(472)...........: MPID_Init_local(585).....: MPIDI_OFI_init_local(633): open_fabric(1360)........: OFI fi_getinfo() failed (ofi_init.c:1360:open_fabric:Function not implemented) There must be something in MPICH, at least since v. 3.4.2, or I'm doing something wrong at the MPICH side. My libfabric is working fine: I get good performance with fi_pingpong, but I've also just tried MVAPICH (with psm2 netmod, not under libfabric) and it gave good performance out of the box. Although I'd rather prefer to tweak MPICH because I'm far more comfortable with that code, I'm okay moving ahead with MVAPICH, so unless this is interesting from your side (I guess you don't care that much about psm2 now), we can close this thread. Thanks a lot for your help. Toni El 15/6/21 a las 15:03, Wesley Bland via discuss escribió: Hey Toni, I’d be surprised that the performance drops that much, but you can try —with-device=ch4:ofi:psm2 to convert at least some of the branches to be compile-time instead of runtime. After that, I don’t remember enough about OFI 1.5. There might have been some changes in MPICH over the last year or two that makes that version not perform as well… Thanks, Wes On Jun 15, 2021, at 4:46 AM, Antonio Peña via discuss <discuss(a)mpich.org<mailto:[email protected]>> wrote: Hi folks, I'm setting up an MPICH over libfabric over psm2 for MareNostrum (Omni-Path), to try out some ideas. I've compiled libfabric 1.5 (last one that compiles in this machine) over opa-psm2-11.2.185, and mpich-3.4.2 + mpich-4.0a1 in both ch3 and ch4 (yes 4 MPICH variants). There's only psm2 support in libfabric, so no danger of falling back to other providers. ldd confirms my libfabric is linked. ./fi_info provider: psm2 fabric: psm2 domain: psm2 version: 1.5 type: FI_EP_RDM protocol: FI_PROTO_PSMX2 I'm comparing 2-node pt2pt performance against impi/2017.4 using osu microbenchmarks. While both fi_pingong and impi give me a max. BW of ~10 MB/s, all mpich versions stick at ~3 MB/s. Is this expected? I mean, is there so much secret sauce in impi? Or, am likely doing something wrong? I'm doing fairly plain configures, nothing fancy, e.g.: ./configure --prefix=... --with-device=ch4:ofi --with-libfabric=... I'd appreciate some guidance - my MPICH tweaking is a little rusted :) Best, Toni _______________________________________________ discuss mailing list discuss(a)mpich.org<mailto:[email protected]> To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss _______________________________________________ discuss mailing list discuss(a)mpich.org<mailto:[email protected]> To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss -- Antonio J. Peña (PhD) Team Lead, Accelerators and Communications for HPC | Teaching and Research Staff Sr. Researcher, Computer Sciences Department | Computer Architecture Department Barcelona Supercomputing Center (BSC) | Universitat Politècnica de Catalunya (UPC) http://www.bsc.es/pena-antonio =============================== Looking for job opportunities? Open positions in my team. Please contact me.

1 0

the MPI daemon triggers an assertion on an ARM-based Linux system
by Fabrice Ducos 16 Jun '21

16 Jun '21

Greetings, when running an application (atmospheric science) with MPICH on an AWS ARM (not x86) instance (with Linux Ubuntu Server 20.04), our process crashes at the end of the processing. MPICH was installed precompiled with the Ubuntu apt provisioning tool: $ sudo apt install -y mpich $ apt list [only relevant lines displayed for brevity] libmpich-dev/focal,now 3.3.2-2build1 arm64 [residual-config] libmpich12/focal,now 3.3.2-2build1 arm64 [installed,auto-removable] mpich-doc/focal 3.3.2-2build1 all mpich/focal 3.3.2-2build1 arm64 Luckily, we got some debug information in mpid that can be valuable: Assertion failed in file src/mpid/ch3/channels/nemesis/src/ch3_progress.c at line 530: payload_len >= sizeof (MPIDI_CH3_Pkt_t) 0xffff86833f5f ??? ???:0 0xffff86881eef ??? ???:0 0xffff8683793f ??? ???:0 0xffff8686c543 ??? ???:0 0xffff8676d4b3 ??? ???:0 0xffff8637e6eb ??? ???:0 0xffff8637e85b ??? ???:0 0xffff86369093 ??? ???:0 0xaaaac5491e6f ??? ???:0 internal ABORT - process 22 The same application has been used for years with several MPI implementations (MPICH, OpenMPI, Intel MPI) on x86 systems without problem. It was successfully tested with MPICH on Linux Ubuntu Server 20.04 x86 shortly alongside the ARM test. We also tested the application with another MPI implementation (namely, OpenMPI) on the same ARM instance and it did work. We are perfectly fine using another MPI implementation in this specific case, but we thought that this issue would be of some interest to the MPICH maintenance team. Best regards Fabrice Ducos Ingénieur d’études CNRS Laboratoire d’Optique Atmosphérique - UMR CNRS 8518 Faculté des Sciences et Technologies Bâtiment P5 - Bureau 325 Université de Lille - Cité Scientifique 59655 Villeneuve d’Ascq

1 0

Re: [mpich-discuss] libfabric+psm2 performance
by Antonio J. Peña 16 Jun '21

16 Jun '21

Hi Wesley, Happy to hear from you. With that setting I cannot get out from this runtime error at init: Abort(69832847) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init: Other MPI error, error stack: PMPI_Init(98)............: MPI_Init(argc=0x7ffd4776159c, argv=0x7ffd47761590) failed MPII_Init_thread(196)....: MPID_Init(472)...........: MPID_Init_local(585).....: MPIDI_OFI_init_local(633): open_fabric(1360)........: OFI fi_getinfo() failed (ofi_init.c:1360:open_fabric:Function not implemented) There must be something in MPICH, at least since v. 3.4.2, or I'm doing something wrong at the MPICH side. My libfabric is working fine: I get good performance with fi_pingpong, but I've also just tried MVAPICH (with psm2 netmod, not under libfabric) and it gave good performance out of the box. Although I'd rather prefer to tweak MPICH because I'm far more comfortable with that code, I'm okay moving ahead with MVAPICH, so unless this is interesting from your side (I guess you don't care that much about psm2 now), we can close this thread. Thanks a lot for your help. Toni El 15/6/21 a las 15:03, Wesley Bland via discuss escribió: > Hey Toni, > > I’d be surprised that the performance drops that much, but you can try —with-device=ch4:ofi:psm2 to convert at least some of the branches to be compile-time instead of runtime. After that, I don’t remember enough about OFI 1.5. There might have been some changes in MPICH over the last year or two that makes that version not perform as well… > > Thanks, > Wes > >> On Jun 15, 2021, at 4:46 AM, Antonio Peña via discuss <discuss(a)mpich.org> wrote: >> >> >> Hi folks, >> >> I'm setting up an MPICH over libfabric over psm2 for MareNostrum (Omni-Path), to try out some ideas. >> >> I've compiled libfabric 1.5 (last one that compiles in this machine) over opa-psm2-11.2.185, and mpich-3.4.2 + mpich-4.0a1 in both ch3 and ch4 (yes 4 MPICH variants). There's only psm2 support in libfabric, so no danger of falling back to other providers. ldd confirms my libfabric is linked. >> >> ./fi_info >> provider: psm2 >> fabric: psm2 >> domain: psm2 >> version: 1.5 >> type: FI_EP_RDM >> protocol: FI_PROTO_PSMX2 >> >> I'm comparing 2-node pt2pt performance against impi/2017.4 using osu microbenchmarks. >> >> While both fi_pingong and impi give me a max. BW of ~10 MB/s, all mpich versions stick at ~3 MB/s. >> >> Is this expected? I mean, is there so much secret sauce in impi? Or, am likely doing something wrong? >> >> I'm doing fairly plain configures, nothing fancy, e.g.: >> ./configure --prefix=... --with-device=ch4:ofi --with-libfabric=... >> >> I'd appreciate some guidance - my MPICH tweaking is a little rusted :) >> >> Best, >> Toni >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss -- Antonio J. Peña (PhD) Team Lead, Accelerators and Communications for HPC | Teaching and Research Staff Sr. Researcher, Computer Sciences Department | Computer Architecture Department Barcelona Supercomputing Center (BSC) | Universitat Politècnica de Catalunya (UPC) http://www.bsc.es/pena-antonio =============================== Looking for job opportunities? Open positions in my team. Please contact me.

1 0

mpiexec (MPICH2) launch too slow on Windows10.
by ZJ CHEN 15 Jun '21

15 Jun '21

My OS: windows10 mpich2: 1.3-32bit and 1.5-64bit I install both versions successfully, and I test the MPIEXEC -N 10 CPI.EXE example. It works, but with a very slow launching duration of about 4 minutes. How can I fix this problem? Thanks for any help! ZHIJUN

1 0

Re: [mpich-discuss] libfabric+psm2 performance
by Wesley Bland 15 Jun '21

15 Jun '21

Hey Toni, I’d be surprised that the performance drops that much, but you can try —with-device=ch4:ofi:psm2 to convert at least some of the branches to be compile-time instead of runtime. After that, I don’t remember enough about OFI 1.5. There might have been some changes in MPICH over the last year or two that makes that version not perform as well… Thanks, Wes > On Jun 15, 2021, at 4:46 AM, Antonio Peña via discuss <discuss(a)mpich.org> wrote: > > > Hi folks, > > I'm setting up an MPICH over libfabric over psm2 for MareNostrum (Omni-Path), to try out some ideas. > > I've compiled libfabric 1.5 (last one that compiles in this machine) over opa-psm2-11.2.185, and mpich-3.4.2 + mpich-4.0a1 in both ch3 and ch4 (yes 4 MPICH variants). There's only psm2 support in libfabric, so no danger of falling back to other providers. ldd confirms my libfabric is linked. > > ./fi_info > provider: psm2 > fabric: psm2 > domain: psm2 > version: 1.5 > type: FI_EP_RDM > protocol: FI_PROTO_PSMX2 > > I'm comparing 2-node pt2pt performance against impi/2017.4 using osu microbenchmarks. > > While both fi_pingong and impi give me a max. BW of ~10 MB/s, all mpich versions stick at ~3 MB/s. > > Is this expected? I mean, is there so much secret sauce in impi? Or, am likely doing something wrong? > > I'm doing fairly plain configures, nothing fancy, e.g.: > ./configure --prefix=... --with-device=ch4:ofi --with-libfabric=... > > I'd appreciate some guidance - my MPICH tweaking is a little rusted :) > > Best, > Toni > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss

1 0

libfabric+psm2 performance
by Antonio Peña 15 Jun '21

15 Jun '21

Hi folks, I'm setting up an MPICH over libfabric over psm2 for MareNostrum (Omni-Path), to try out some ideas. I've compiled libfabric 1.5 (last one that compiles in this machine) over opa-psm2-11.2.185, and mpich-3.4.2 + mpich-4.0a1 in both ch3 and ch4 (yes 4 MPICH variants). There's only psm2 support in libfabric, so no danger of falling back to other providers. ldd confirms my libfabric is linked. ./fi_info provider: psm2 fabric: psm2 domain: psm2 version: 1.5 type: FI_EP_RDM protocol: FI_PROTO_PSMX2 I'm comparing 2-node pt2pt performance against impi/2017.4 using osu microbenchmarks. While both fi_pingong and impi give me a max. BW of ~10 MB/s, all mpich versions stick at ~3 MB/s. Is this expected? I mean, is there so much secret sauce in impi? Or, am likely doing something wrong? I'm doing fairly plain configures, nothing fancy, e.g.: ./configure --prefix=... --with-device=ch4:ofi --with-libfabric=... I'd appreciate some guidance - my MPICH tweaking is a little rusted :) Best, Toni

1 0

Re: [mpich-discuss] Building CUDA-aware MPICH fails on Ubuntu 20.04
by Omlin Samuel 07 Jun '21

07 Jun '21

Hi Kenneth, applying this patch made mpich build successful. Thanks a lot!! Cheers, Sam ________________________________ From: Raffenetti, Kenneth J. <raffenet(a)mcs.anl.gov> Sent: Wednesday, June 2, 2021 11:04:28 PM To: discuss(a)mpich.org Cc: Omlin Samuel Subject: Re: [mpich-discuss] Building CUDA-aware MPICH fails on Ubuntu 20.04 Hi Omlin, This is a known issue that was fixed but unfortunately not included in MPICH 3.4.2. A workaround would be to change /bin/sh to link to /bin/bash on your system. Or you could apply this patch to /home/omlins/soft/mpich-3.4.2/modules/yaksa/src/backend/cuda/cudalt.sh: https://github.com/pmodels/yaksa/pull/181/commits/eed193d9775dd0f33cbd8caa0… Ken On 6/2/21, 12:54 PM, "Omlin Samuel via discuss" <discuss(a)mpich.org> wrote: Dear all, Building CUDA-aware MPICH fails for me after a few seconds on Ubuntu 20.04 following the MPICH installation guide (CUDA Driver Version: 465.19.01, CUDA Version: 11.3). Reproducer: mkdir ~/soft cd ~/soft tar -xzf mpich-3.4.2.tar.gz mkdir /tmp/mpich-3.4.2 cd /tmp/mpich-3.4.2 ~/soft/mpich-3.4.2/configure --with-cuda=/usr/local/cuda --with-device=ch4:ucx 2>&1 | tee configure.log make 2>&1 | tee make.log Error: /home/omlins/soft/mpich-3.4.2/modules/yaksa/src/backend/cuda/cudalt.sh: 35: Bad substitution make[2]: *** [Makefile:8697: src/backend/cuda/pup/yaksuri_cudai_pup_hvector__Bool.lo] Error 2 make[2]: Leaving directory '/tmp/mpich-3.4.2/modules/yaksa' make[1]: *** [Makefile:43560: all-recursive] Error 1 make[1]: Leaving directory '/tmp/mpich-3.4.2' make: *** [Makefile:11141: all] Error 2 The with tee created log files 'configure.log' and 'make.log', as well as 'config.log', are attached. Am I missing something or is there an issue with MPICH here? Thanks a lot in advance for helping me solving this issue! Cheers, Sam -- Samuel Omlin, PhD Computational Scientist CSCS - Swiss National Supercomputing Centre ETH Zurich Via Trevano 131 CH-6900 Lugano Switzerland

1 0

Re: [mpich-discuss] Building CUDA-aware MPICH fails on Ubuntu 20.04
by Raffenetti, Kenneth J. 02 Jun '21

02 Jun '21

Hi Omlin, This is a known issue that was fixed but unfortunately not included in MPICH 3.4.2. A workaround would be to change /bin/sh to link to /bin/bash on your system. Or you could apply this patch to /home/omlins/soft/mpich-3.4.2/modules/yaksa/src/backend/cuda/cudalt.sh: https://github.com/pmodels/yaksa/pull/181/commits/eed193d9775dd0f33cbd8caa0… Ken On 6/2/21, 12:54 PM, "Omlin Samuel via discuss" <discuss(a)mpich.org> wrote: Dear all, Building CUDA-aware MPICH fails for me after a few seconds on Ubuntu 20.04 following the MPICH installation guide (CUDA Driver Version: 465.19.01, CUDA Version: 11.3). Reproducer: mkdir ~/soft cd ~/soft tar -xzf mpich-3.4.2.tar.gz mkdir /tmp/mpich-3.4.2 cd /tmp/mpich-3.4.2 ~/soft/mpich-3.4.2/configure --with-cuda=/usr/local/cuda --with-device=ch4:ucx 2>&1 | tee configure.log make 2>&1 | tee make.log Error: /home/omlins/soft/mpich-3.4.2/modules/yaksa/src/backend/cuda/cudalt.sh: 35: Bad substitution make[2]: *** [Makefile:8697: src/backend/cuda/pup/yaksuri_cudai_pup_hvector__Bool.lo] Error 2 make[2]: Leaving directory '/tmp/mpich-3.4.2/modules/yaksa' make[1]: *** [Makefile:43560: all-recursive] Error 1 make[1]: Leaving directory '/tmp/mpich-3.4.2' make: *** [Makefile:11141: all] Error 2 The with tee created log files 'configure.log' and 'make.log', as well as 'config.log', are attached. Am I missing something or is there an issue with MPICH here? Thanks a lot in advance for helping me solving this issue! Cheers, Sam -- Samuel Omlin, PhD Computational Scientist CSCS - Swiss National Supercomputing Centre ETH Zurich Via Trevano 131 CH-6900 Lugano Switzerland

1 0

Building CUDA-aware MPICH fails on Ubuntu 20.04
by Omlin Samuel 02 Jun '21

02 Jun '21

Dear all, Building CUDA-aware MPICH fails for me after a few seconds on Ubuntu 20.04 following the MPICH installation guide (CUDA Driver Version: 465.19.01, CUDA Version: 11.3). Reproducer: mkdir ~/soft cd ~/soft tar -xzf mpich-3.4.2.tar.gz mkdir /tmp/mpich-3.4.2 cd /tmp/mpich-3.4.2 ~/soft/mpich-3.4.2/configure --with-cuda=/usr/local/cuda --with-device=ch4:ucx 2>&1 | tee configure.log make 2>&1 | tee make.log Error: /home/omlins/soft/mpich-3.4.2/modules/yaksa/src/backend/cuda/cudalt.sh: 35: Bad substitution make[2]: *** [Makefile:8697: src/backend/cuda/pup/yaksuri_cudai_pup_hvector__Bool.lo] Error 2 make[2]: Leaving directory '/tmp/mpich-3.4.2/modules/yaksa' make[1]: *** [Makefile:43560: all-recursive] Error 1 make[1]: Leaving directory '/tmp/mpich-3.4.2' make: *** [Makefile:11141: all] Error 2 The with tee created log files 'configure.log' and 'make.log', as well as 'config.log', are attached. Am I missing something or is there an issue with MPICH here? Thanks a lot in advance for helping me solving this issue! Cheers, Sam -- Samuel Omlin, PhD Computational Scientist CSCS - Swiss National Supercomputing Centre ETH Zurich Via Trevano 131 CH-6900 Lugano Switzerland

1 0

Re: [mpich-discuss] Q: Why doesn't "use mpi, only : mpi_bcast" (or similar) work in MPICH?
by Zhou, Hui 01 Jun '21

01 Jun '21

MPICH’s `use mpi` module does not define function interfaces. Functions such as `MPI_Send` and `MPI_Bcast` are directly linked to the C symbols. Fortran 90 used to have very poor support for “choice buffers”, which probably resulted in the decision not to define the function interface in our legacy “use mpi” module. We recommend to try “use mpi_f08” instead. The `mpi_f08` interface has the full interface defined and adopts the C interoperability features. -- Hui Zhou From: Brian Dobbins via discuss <discuss(a)mpich.org> Date: Tuesday, June 1, 2021 at 7:53 PM To: discuss(a)mpich.org <discuss(a)mpich.org> Cc: Brian Dobbins <bdobbins(a)gmail.com> Subject: [mpich-discuss] Q: Why doesn't "use mpi, only : mpi_bcast" (or similar) work in MPICH? Hi all, I figure this is one of those things in the standard, but I figured I'd get a better explanation asking here than digging through that myself. Long story short, if I try the following code: program foo use mpi, only: mpi_bcast end program foo ... This fails to compile in MPICH (3.4.2, at least), IntelMPI and MVAPICH, but does work on OpenMPI and SGI MPT, at least on systems I've tried. Replacing the 'use, only' with just a 'use' directive works fine. And MPI_BCAST is just an example here; things like 'MPI_Finalize' and 'MPI_Integer' appear to be in the module, but MPI_Send also fails. Is this a scenario where the standard doesn't support this, but OpenMPI and MPT do it as a matter of convenience vs strict adherence found in the MPICH-based libraries? Or am I missing some simple issue? Any insight would be greatly appreciated! Thanks, - Brian

1 0