- discuss - mailman3.cels.anl.gov

Re: [mpich-discuss] Building CUDA-aware MPICH fails on Ubuntu 20.04
by Raffenetti, Kenneth J. 02 Jun '21

02 Jun '21

Hi Omlin, This is a known issue that was fixed but unfortunately not included in MPICH 3.4.2. A workaround would be to change /bin/sh to link to /bin/bash on your system. Or you could apply this patch to /home/omlins/soft/mpich-3.4.2/modules/yaksa/src/backend/cuda/cudalt.sh: https://github.com/pmodels/yaksa/pull/181/commits/eed193d9775dd0f33cbd8caa0… Ken On 6/2/21, 12:54 PM, "Omlin Samuel via discuss" <discuss(a)mpich.org> wrote: Dear all, Building CUDA-aware MPICH fails for me after a few seconds on Ubuntu 20.04 following the MPICH installation guide (CUDA Driver Version: 465.19.01, CUDA Version: 11.3). Reproducer: mkdir ~/soft cd ~/soft tar -xzf mpich-3.4.2.tar.gz mkdir /tmp/mpich-3.4.2 cd /tmp/mpich-3.4.2 ~/soft/mpich-3.4.2/configure --with-cuda=/usr/local/cuda --with-device=ch4:ucx 2>&1 | tee configure.log make 2>&1 | tee make.log Error: /home/omlins/soft/mpich-3.4.2/modules/yaksa/src/backend/cuda/cudalt.sh: 35: Bad substitution make[2]: *** [Makefile:8697: src/backend/cuda/pup/yaksuri_cudai_pup_hvector__Bool.lo] Error 2 make[2]: Leaving directory '/tmp/mpich-3.4.2/modules/yaksa' make[1]: *** [Makefile:43560: all-recursive] Error 1 make[1]: Leaving directory '/tmp/mpich-3.4.2' make: *** [Makefile:11141: all] Error 2 The with tee created log files 'configure.log' and 'make.log', as well as 'config.log', are attached. Am I missing something or is there an issue with MPICH here? Thanks a lot in advance for helping me solving this issue! Cheers, Sam -- Samuel Omlin, PhD Computational Scientist CSCS - Swiss National Supercomputing Centre ETH Zurich Via Trevano 131 CH-6900 Lugano Switzerland

1 0

Building CUDA-aware MPICH fails on Ubuntu 20.04
by Omlin Samuel 02 Jun '21

02 Jun '21

Dear all, Building CUDA-aware MPICH fails for me after a few seconds on Ubuntu 20.04 following the MPICH installation guide (CUDA Driver Version: 465.19.01, CUDA Version: 11.3). Reproducer: mkdir ~/soft cd ~/soft tar -xzf mpich-3.4.2.tar.gz mkdir /tmp/mpich-3.4.2 cd /tmp/mpich-3.4.2 ~/soft/mpich-3.4.2/configure --with-cuda=/usr/local/cuda --with-device=ch4:ucx 2>&1 | tee configure.log make 2>&1 | tee make.log Error: /home/omlins/soft/mpich-3.4.2/modules/yaksa/src/backend/cuda/cudalt.sh: 35: Bad substitution make[2]: *** [Makefile:8697: src/backend/cuda/pup/yaksuri_cudai_pup_hvector__Bool.lo] Error 2 make[2]: Leaving directory '/tmp/mpich-3.4.2/modules/yaksa' make[1]: *** [Makefile:43560: all-recursive] Error 1 make[1]: Leaving directory '/tmp/mpich-3.4.2' make: *** [Makefile:11141: all] Error 2 The with tee created log files 'configure.log' and 'make.log', as well as 'config.log', are attached. Am I missing something or is there an issue with MPICH here? Thanks a lot in advance for helping me solving this issue! Cheers, Sam -- Samuel Omlin, PhD Computational Scientist CSCS - Swiss National Supercomputing Centre ETH Zurich Via Trevano 131 CH-6900 Lugano Switzerland

1 0

Re: [mpich-discuss] Q: Why doesn't "use mpi, only : mpi_bcast" (or similar) work in MPICH?
by Zhou, Hui 01 Jun '21

01 Jun '21

MPICH’s `use mpi` module does not define function interfaces. Functions such as `MPI_Send` and `MPI_Bcast` are directly linked to the C symbols. Fortran 90 used to have very poor support for “choice buffers”, which probably resulted in the decision not to define the function interface in our legacy “use mpi” module. We recommend to try “use mpi_f08” instead. The `mpi_f08` interface has the full interface defined and adopts the C interoperability features. -- Hui Zhou From: Brian Dobbins via discuss <discuss(a)mpich.org> Date: Tuesday, June 1, 2021 at 7:53 PM To: discuss(a)mpich.org <discuss(a)mpich.org> Cc: Brian Dobbins <bdobbins(a)gmail.com> Subject: [mpich-discuss] Q: Why doesn't "use mpi, only : mpi_bcast" (or similar) work in MPICH? Hi all, I figure this is one of those things in the standard, but I figured I'd get a better explanation asking here than digging through that myself. Long story short, if I try the following code: program foo use mpi, only: mpi_bcast end program foo ... This fails to compile in MPICH (3.4.2, at least), IntelMPI and MVAPICH, but does work on OpenMPI and SGI MPT, at least on systems I've tried. Replacing the 'use, only' with just a 'use' directive works fine. And MPI_BCAST is just an example here; things like 'MPI_Finalize' and 'MPI_Integer' appear to be in the module, but MPI_Send also fails. Is this a scenario where the standard doesn't support this, but OpenMPI and MPT do it as a matter of convenience vs strict adherence found in the MPICH-based libraries? Or am I missing some simple issue? Any insight would be greatly appreciated! Thanks, - Brian

1 0

Q: Why doesn't "use mpi, only : mpi_bcast" (or similar) work in MPICH?
by Brian Dobbins 01 Jun '21

01 Jun '21

Hi all, I figure this is one of those things in the standard, but I figured I'd get a better explanation asking here than digging through that myself. Long story short, if I try the following code: *program foo* * use mpi, only: mpi_bcast* *end program foo* ... This fails to compile in MPICH (3.4.2, at least), IntelMPI and MVAPICH, but *does* work on OpenMPI and SGI MPT, at least on systems I've tried. Replacing the 'use, only' with just a 'use' directive works fine. And MPI_BCAST is just an example here; things like 'MPI_Finalize' and 'MPI_Integer' appear to be in the module, but MPI_Send also fails. Is this a scenario where the standard doesn't support this, but OpenMPI and MPT do it as a matter of convenience vs strict adherence found in the MPICH-based libraries? Or am I missing some simple issue? Any insight would be greatly appreciated! Thanks, - Brian

1 0

Re: [mpich-discuss] mpich3 error with ch3
by Zhou, Hui 27 May '21

27 May '21

Hi Alim, Thanks for the details. That looks like data corruptions, probably due to some of the atomics used are not strong enough on the particular CPU architecture. Ch3 in some place uses weaker atomics that runs fine on the known architecture, but you may have just provided an example that the weaker atomics is insufficient. If running Ch3 is important for you, please file an issue over https://github.com/pmodels/mpich and we will track it down. -- Hui Zhou From: Alim Akhtar <alim.akhtar(a)gmail.com> Date: Thursday, May 27, 2021 at 9:20 AM To: Zhou, Hui <zhouh(a)anl.gov> Cc: discuss(a)mpich.org <discuss(a)mpich.org> Subject: Re: [mpich-discuss] mpich3 error with ch3 Hi Hui On Thu, May 27, 2021 at 7:28 PM Zhou, Hui <zhouh(a)anl.gov> wrote: > > Similar issues can be very different in causes. Checking the referenced discussion, I wasn’t sure what was the original issues. We suggested to try ch4 as to get more data points rather than as a solution. Nevertheless, ch4 is the current recommended device as it is more actively developed. > > > > Ch3 is not broken as far as we know. Could you describe your issue in more details? > I get Assertion failed in file src/mpid/ch3/channels/nemesis/src/ch3_progress.c at line 530: payload_len >= sizeof (MPIDI_CH3_Pkt_t) after some loop. or sometime I see like : Assertion failed in file src/mpid/ch3/channels/nemesis/src/ch3_progress.c at line 567: !vc_ch->recv->active The number of pass loop does depends on number of CPUs used. (more number of CPUs more failure). With One CPU, no failure. > > > -- > Hui Zhou > > > > > > From: Alim Akhtar via discuss <discuss(a)mpich.org> > Date: Wednesday, May 26, 2021 at 10:51 PM > To: discuss(a)mpich.org <discuss(a)mpich.org> > Cc: Alim Akhtar <alim.akhtar(a)gmail.com> > Subject: [mpich-discuss] mpich3 error with ch3 > > Hi mpich dev team, > > I am facing one issue similar to discussed in below discussion > > https://lists.mpich.org/pipermail/devel/2021-January/000826.html > > Someone in the mailing list suggested recompiling the mpi bench using > CH4. as below: > > https://lists.mpich.org/pipermail/devel/2021-January/000828.html > > "MPICH with ch4, with `--with-device=ch4:ofi`" > > Actually this fixes the failure on this CPU architecture. > > Questions: > 1. Is this a known issue with MPI bench on the recent CPU > architecture? (I am running on ARM's Cortex -A 75), like ch3 is > broken? > 2. With no error after using CH4, does this mean the CPU is all good? > > Note: using ch3 was working fine on our previous CPU (A-72) > > Any input will be really appreciated. > > > -- > Regards, > Alim > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss -- Regards, Alim

1 0

Re: [mpich-discuss] mpich3 error with ch3
by Alim Akhtar 27 May '21

27 May '21

Hi Hui On Thu, May 27, 2021 at 7:28 PM Zhou, Hui <zhouh(a)anl.gov> wrote: > > Similar issues can be very different in causes. Checking the referenced discussion, I wasn’t sure what was the original issues. We suggested to try ch4 as to get more data points rather than as a solution. Nevertheless, ch4 is the current recommended device as it is more actively developed. > > > > Ch3 is not broken as far as we know. Could you describe your issue in more details? > I get Assertion failed in file src/mpid/ch3/channels/nemesis/src/ch3_progress.c at line 530: payload_len >= sizeof (MPIDI_CH3_Pkt_t) after some loop. or sometime I see like : Assertion failed in file src/mpid/ch3/channels/nemesis/src/ch3_progress.c at line 567: !vc_ch->recv->active The number of pass loop does depends on number of CPUs used. (more number of CPUs more failure). With One CPU, no failure. > > > -- > Hui Zhou > > > > > > From: Alim Akhtar via discuss <discuss(a)mpich.org> > Date: Wednesday, May 26, 2021 at 10:51 PM > To: discuss(a)mpich.org <discuss(a)mpich.org> > Cc: Alim Akhtar <alim.akhtar(a)gmail.com> > Subject: [mpich-discuss] mpich3 error with ch3 > > Hi mpich dev team, > > I am facing one issue similar to discussed in below discussion > > https://lists.mpich.org/pipermail/devel/2021-January/000826.html > > Someone in the mailing list suggested recompiling the mpi bench using > CH4. as below: > > https://lists.mpich.org/pipermail/devel/2021-January/000828.html > > "MPICH with ch4, with `--with-device=ch4:ofi`" > > Actually this fixes the failure on this CPU architecture. > > Questions: > 1. Is this a known issue with MPI bench on the recent CPU > architecture? (I am running on ARM's Cortex -A 75), like ch3 is > broken? > 2. With no error after using CH4, does this mean the CPU is all good? > > Note: using ch3 was working fine on our previous CPU (A-72) > > Any input will be really appreciated. > > > -- > Regards, > Alim > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss -- Regards, Alim

1 0

Re: [mpich-discuss] mpich3 error with ch3
by Zhou, Hui 27 May '21

27 May '21

Similar issues can be very different in causes. Checking the referenced discussion, I wasn’t sure what was the original issues. We suggested to try ch4 as to get more data points rather than as a solution. Nevertheless, ch4 is the current recommended device as it is more actively developed. Ch3 is not broken as far as we know. Could you describe your issue in more details? -- Hui Zhou From: Alim Akhtar via discuss <discuss(a)mpich.org> Date: Wednesday, May 26, 2021 at 10:51 PM To: discuss(a)mpich.org <discuss(a)mpich.org> Cc: Alim Akhtar <alim.akhtar(a)gmail.com> Subject: [mpich-discuss] mpich3 error with ch3 Hi mpich dev team, I am facing one issue similar to discussed in below discussion https://lists.mpich.org/pipermail/devel/2021-January/000826.html Someone in the mailing list suggested recompiling the mpi bench using CH4. as below: https://lists.mpich.org/pipermail/devel/2021-January/000828.html "MPICH with ch4, with `--with-device=ch4:ofi`" Actually this fixes the failure on this CPU architecture. Questions: 1. Is this a known issue with MPI bench on the recent CPU architecture? (I am running on ARM's Cortex -A 75), like ch3 is broken? 2. With no error after using CH4, does this mean the CPU is all good? Note: using ch3 was working fine on our previous CPU (A-72) Any input will be really appreciated. -- Regards, Alim _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

mpich3 error with ch3
by Alim Akhtar 26 May '21

26 May '21

Hi mpich dev team, I am facing one issue similar to discussed in below discussion https://lists.mpich.org/pipermail/devel/2021-January/000826.html Someone in the mailing list suggested recompiling the mpi bench using CH4. as below: https://lists.mpich.org/pipermail/devel/2021-January/000828.html "MPICH with ch4, with `--with-device=ch4:ofi`" Actually this fixes the failure on this CPU architecture. Questions: 1. Is this a known issue with MPI bench on the recent CPU architecture? (I am running on ARM's Cortex -A 75), like ch3 is broken? 2. With no error after using CH4, does this mean the CPU is all good? Note: using ch3 was working fine on our previous CPU (A-72) Any input will be really appreciated. -- Regards, Alim

1 0

Re: [mpich-discuss] Build MPICH Error
by Raffenetti, Kenneth J. 25 May '21

25 May '21

How did you add the Intel compilers to your environment? Did you source the compilervars.sh script provided by the installation? That is more reliable than adjusting PATH(s) on your own, in my experience. Ken On 5/24/21, 12:31 PM, "Omid Bagheri via discuss" <discuss(a)mpich.org> wrote: Hello, I am installing MPICH Version 3.4.1 on CentOS Linux 8 and I get an error in building MPICH while the configuration was successful. I was wondering if you can help me with this. The m.txt and c.txt are attached to this email and I used the following to configure: cd /storage/bagherio/tmp/mpich-3.4.1 /storage/bagherio/mpich-3.4.1/configure \ -prefix=/storage/bagherio/mpich-install --with-device=ch4:ofi 2>&1 | tee c.txt Regards, Omid

1 0

Re: [mpich-discuss] Build MPICH Error
by Zhou, Hui 25 May '21

25 May '21

Not sure this is related to MPICH. One suggestion from StackOverflow (https://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=&ved=2ahUKEwj…) seems to suggest this is an upgrade/install issue. -- Hui Zhou From: Omid Bagheri via discuss <discuss(a)mpich.org> Date: Monday, May 24, 2021 at 12:31 PM To: discuss(a)mpich.org <discuss(a)mpich.org> Cc: Omid Bagheri <obgcee(a)gmail.com> Subject: [mpich-discuss] Build MPICH Error Hello, I am installing MPICH Version 3.4.1 on CentOS Linux 8 and I get an error in building MPICH while the configuration was successful. I was wondering if you can help me with this. The m.txt and c.txt are attached to this email and I used the following to configure: cd /storage/bagherio/tmp/mpich-3.4.1 /storage/bagherio/mpich-3.4.1/configure \ -prefix=/storage/bagherio/mpich-install --with-device=ch4:ofi 2>&1 | tee c.txt Regards, Omid

1 0