May 2021 - discuss - mailman3.cels.anl.gov

Re: [mpich-discuss] mpich3 error with ch3
by Zhou, Hui 27 May '21

27 May '21

Hi Alim, Thanks for the details. That looks like data corruptions, probably due to some of the atomics used are not strong enough on the particular CPU architecture. Ch3 in some place uses weaker atomics that runs fine on the known architecture, but you may have just provided an example that the weaker atomics is insufficient. If running Ch3 is important for you, please file an issue over https://github.com/pmodels/mpich and we will track it down. -- Hui Zhou From: Alim Akhtar <alim.akhtar(a)gmail.com> Date: Thursday, May 27, 2021 at 9:20 AM To: Zhou, Hui <zhouh(a)anl.gov> Cc: discuss(a)mpich.org <discuss(a)mpich.org> Subject: Re: [mpich-discuss] mpich3 error with ch3 Hi Hui On Thu, May 27, 2021 at 7:28 PM Zhou, Hui <zhouh(a)anl.gov> wrote: > > Similar issues can be very different in causes. Checking the referenced discussion, I wasn’t sure what was the original issues. We suggested to try ch4 as to get more data points rather than as a solution. Nevertheless, ch4 is the current recommended device as it is more actively developed. > > > > Ch3 is not broken as far as we know. Could you describe your issue in more details? > I get Assertion failed in file src/mpid/ch3/channels/nemesis/src/ch3_progress.c at line 530: payload_len >= sizeof (MPIDI_CH3_Pkt_t) after some loop. or sometime I see like : Assertion failed in file src/mpid/ch3/channels/nemesis/src/ch3_progress.c at line 567: !vc_ch->recv->active The number of pass loop does depends on number of CPUs used. (more number of CPUs more failure). With One CPU, no failure. > > > -- > Hui Zhou > > > > > > From: Alim Akhtar via discuss <discuss(a)mpich.org> > Date: Wednesday, May 26, 2021 at 10:51 PM > To: discuss(a)mpich.org <discuss(a)mpich.org> > Cc: Alim Akhtar <alim.akhtar(a)gmail.com> > Subject: [mpich-discuss] mpich3 error with ch3 > > Hi mpich dev team, > > I am facing one issue similar to discussed in below discussion > > https://lists.mpich.org/pipermail/devel/2021-January/000826.html > > Someone in the mailing list suggested recompiling the mpi bench using > CH4. as below: > > https://lists.mpich.org/pipermail/devel/2021-January/000828.html > > "MPICH with ch4, with `--with-device=ch4:ofi`" > > Actually this fixes the failure on this CPU architecture. > > Questions: > 1. Is this a known issue with MPI bench on the recent CPU > architecture? (I am running on ARM's Cortex -A 75), like ch3 is > broken? > 2. With no error after using CH4, does this mean the CPU is all good? > > Note: using ch3 was working fine on our previous CPU (A-72) > > Any input will be really appreciated. > > > -- > Regards, > Alim > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss -- Regards, Alim

1 0

Re: [mpich-discuss] mpich3 error with ch3
by Alim Akhtar 27 May '21

27 May '21

Hi Hui On Thu, May 27, 2021 at 7:28 PM Zhou, Hui <zhouh(a)anl.gov> wrote: > > Similar issues can be very different in causes. Checking the referenced discussion, I wasn’t sure what was the original issues. We suggested to try ch4 as to get more data points rather than as a solution. Nevertheless, ch4 is the current recommended device as it is more actively developed. > > > > Ch3 is not broken as far as we know. Could you describe your issue in more details? > I get Assertion failed in file src/mpid/ch3/channels/nemesis/src/ch3_progress.c at line 530: payload_len >= sizeof (MPIDI_CH3_Pkt_t) after some loop. or sometime I see like : Assertion failed in file src/mpid/ch3/channels/nemesis/src/ch3_progress.c at line 567: !vc_ch->recv->active The number of pass loop does depends on number of CPUs used. (more number of CPUs more failure). With One CPU, no failure. > > > -- > Hui Zhou > > > > > > From: Alim Akhtar via discuss <discuss(a)mpich.org> > Date: Wednesday, May 26, 2021 at 10:51 PM > To: discuss(a)mpich.org <discuss(a)mpich.org> > Cc: Alim Akhtar <alim.akhtar(a)gmail.com> > Subject: [mpich-discuss] mpich3 error with ch3 > > Hi mpich dev team, > > I am facing one issue similar to discussed in below discussion > > https://lists.mpich.org/pipermail/devel/2021-January/000826.html > > Someone in the mailing list suggested recompiling the mpi bench using > CH4. as below: > > https://lists.mpich.org/pipermail/devel/2021-January/000828.html > > "MPICH with ch4, with `--with-device=ch4:ofi`" > > Actually this fixes the failure on this CPU architecture. > > Questions: > 1. Is this a known issue with MPI bench on the recent CPU > architecture? (I am running on ARM's Cortex -A 75), like ch3 is > broken? > 2. With no error after using CH4, does this mean the CPU is all good? > > Note: using ch3 was working fine on our previous CPU (A-72) > > Any input will be really appreciated. > > > -- > Regards, > Alim > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss -- Regards, Alim

1 0

Re: [mpich-discuss] mpich3 error with ch3
by Zhou, Hui 27 May '21

27 May '21

Similar issues can be very different in causes. Checking the referenced discussion, I wasn’t sure what was the original issues. We suggested to try ch4 as to get more data points rather than as a solution. Nevertheless, ch4 is the current recommended device as it is more actively developed. Ch3 is not broken as far as we know. Could you describe your issue in more details? -- Hui Zhou From: Alim Akhtar via discuss <discuss(a)mpich.org> Date: Wednesday, May 26, 2021 at 10:51 PM To: discuss(a)mpich.org <discuss(a)mpich.org> Cc: Alim Akhtar <alim.akhtar(a)gmail.com> Subject: [mpich-discuss] mpich3 error with ch3 Hi mpich dev team, I am facing one issue similar to discussed in below discussion https://lists.mpich.org/pipermail/devel/2021-January/000826.html Someone in the mailing list suggested recompiling the mpi bench using CH4. as below: https://lists.mpich.org/pipermail/devel/2021-January/000828.html "MPICH with ch4, with `--with-device=ch4:ofi`" Actually this fixes the failure on this CPU architecture. Questions: 1. Is this a known issue with MPI bench on the recent CPU architecture? (I am running on ARM's Cortex -A 75), like ch3 is broken? 2. With no error after using CH4, does this mean the CPU is all good? Note: using ch3 was working fine on our previous CPU (A-72) Any input will be really appreciated. -- Regards, Alim _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

mpich3 error with ch3
by Alim Akhtar 26 May '21

26 May '21

Hi mpich dev team, I am facing one issue similar to discussed in below discussion https://lists.mpich.org/pipermail/devel/2021-January/000826.html Someone in the mailing list suggested recompiling the mpi bench using CH4. as below: https://lists.mpich.org/pipermail/devel/2021-January/000828.html "MPICH with ch4, with `--with-device=ch4:ofi`" Actually this fixes the failure on this CPU architecture. Questions: 1. Is this a known issue with MPI bench on the recent CPU architecture? (I am running on ARM's Cortex -A 75), like ch3 is broken? 2. With no error after using CH4, does this mean the CPU is all good? Note: using ch3 was working fine on our previous CPU (A-72) Any input will be really appreciated. -- Regards, Alim

1 0

Re: [mpich-discuss] Build MPICH Error
by Raffenetti, Kenneth J. 25 May '21

25 May '21

How did you add the Intel compilers to your environment? Did you source the compilervars.sh script provided by the installation? That is more reliable than adjusting PATH(s) on your own, in my experience. Ken On 5/24/21, 12:31 PM, "Omid Bagheri via discuss" <discuss(a)mpich.org> wrote: Hello, I am installing MPICH Version 3.4.1 on CentOS Linux 8 and I get an error in building MPICH while the configuration was successful. I was wondering if you can help me with this. The m.txt and c.txt are attached to this email and I used the following to configure: cd /storage/bagherio/tmp/mpich-3.4.1 /storage/bagherio/mpich-3.4.1/configure \ -prefix=/storage/bagherio/mpich-install --with-device=ch4:ofi 2>&1 | tee c.txt Regards, Omid

1 0

Re: [mpich-discuss] Build MPICH Error
by Zhou, Hui 25 May '21

25 May '21

Not sure this is related to MPICH. One suggestion from StackOverflow (https://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=&ved=2ahUKEwj…) seems to suggest this is an upgrade/install issue. -- Hui Zhou From: Omid Bagheri via discuss <discuss(a)mpich.org> Date: Monday, May 24, 2021 at 12:31 PM To: discuss(a)mpich.org <discuss(a)mpich.org> Cc: Omid Bagheri <obgcee(a)gmail.com> Subject: [mpich-discuss] Build MPICH Error Hello, I am installing MPICH Version 3.4.1 on CentOS Linux 8 and I get an error in building MPICH while the configuration was successful. I was wondering if you can help me with this. The m.txt and c.txt are attached to this email and I used the following to configure: cd /storage/bagherio/tmp/mpich-3.4.1 /storage/bagherio/mpich-3.4.1/configure \ -prefix=/storage/bagherio/mpich-install --with-device=ch4:ofi 2>&1 | tee c.txt Regards, Omid

1 0

Build MPICH Error
by Omid Bagheri 24 May '21

24 May '21

Hello, I am installing MPICH Version 3.4.1 on CentOS Linux 8 and I get an error in building MPICH while the configuration was successful. I was wondering if you can help me with this. The m.txt and c.txt are attached to this email and I used the following to configure: cd /storage/bagherio/tmp/mpich-3.4.1 /storage/bagherio/mpich-3.4.1/configure \ -prefix=/storage/bagherio/mpich-install --with-device=ch4:ofi 2>&1 | tee c.txt Regards, Omid

1 0

Re: [mpich-discuss] “Invalid rank” error message MPICH2
by Parviz Ghahramani 18 May '21

18 May '21

Thanks. That helped to figure out the issue. Parviz -----Original Message----- From: Raffenetti, Kenneth J. <raffenet(a)mcs.anl.gov> Sent: Tuesday, May 18, 2021 3:54 PM To: discuss(a)mpich.org Cc: Parviz Ghahramani <Parviz.Ghahramani(a)inncelerex.com> Subject: Re: [mpich-discuss] “Invalid rank” error message MPICH2 The output indicates you have 2 MPI processes in your application. The ranks of those processes in MPI_COMM_WORLD would be 0 and 1, so 2 is not a valid destination rank for your message. If your application requires more than 2 processes, you'll need to amend your launch settings to start more. Ken On 5/18/21, 2:47 PM, "Parviz Ghahramani via discuss" <discuss(a)mpich.org> wrote: Hi, I am trying to run a process for parallelization over 4 cores. This is run on Windows EC2 instance on AWS. Initially I was getting error message that MPI rejected credentials. I entered and registered the credentials. Now I get the following message “Invalid rank”. USING PARALLEL PROFILE C:\nm75g64\run\para300.pnm MPI TRANSFER TYPE SELECTED Building NONMEM Executable Starting MPI version of nonmem execution ... Manager Location EC2AMAZ-EP7B8E1\\c:\nm75g64\run Fatal error in MPI_Send: Invalid rank, error stack: MPI_Send(173): MPI_Send(buf=00000000037FFD40, count=1, MPI_INTEGER, dest=2, tag=2, MPI_COMM_WORLD) failed MPI_Send(98).: Invalid rank has value 2 but must be nonnegative and less than 2 job aborted: rank: node: exit code[: error message] 0: localhost: 1: process 0 exited without calling finalize 1: localhost: 123 Can anyone provide guidance what this message is and how I can resolve it? Kind regards Parviz

1 0

Re: [mpich-discuss] “Invalid rank” error message MPICH2
by Raffenetti, Kenneth J. 18 May '21

18 May '21

The output indicates you have 2 MPI processes in your application. The ranks of those processes in MPI_COMM_WORLD would be 0 and 1, so 2 is not a valid destination rank for your message. If your application requires more than 2 processes, you'll need to amend your launch settings to start more. Ken On 5/18/21, 2:47 PM, "Parviz Ghahramani via discuss" <discuss(a)mpich.org> wrote: Hi, I am trying to run a process for parallelization over 4 cores. This is run on Windows EC2 instance on AWS. Initially I was getting error message that MPI rejected credentials. I entered and registered the credentials. Now I get the following message “Invalid rank”. USING PARALLEL PROFILE C:\nm75g64\run\para300.pnm MPI TRANSFER TYPE SELECTED Building NONMEM Executable Starting MPI version of nonmem execution ... Manager Location EC2AMAZ-EP7B8E1\\c:\nm75g64\run Fatal error in MPI_Send: Invalid rank, error stack: MPI_Send(173): MPI_Send(buf=00000000037FFD40, count=1, MPI_INTEGER, dest=2, tag=2, MPI_COMM_WORLD) failed MPI_Send(98).: Invalid rank has value 2 but must be nonnegative and less than 2 job aborted: rank: node: exit code[: error message] 0: localhost: 1: process 0 exited without calling finalize 1: localhost: 123 Can anyone provide guidance what this message is and how I can resolve it? Kind regards Parviz

1 0

“Invalid rank” error message MPICH2
by Parviz Ghahramani 18 May '21

18 May '21

Hi, I am trying to run a process for parallelization over 4 cores. This is run on Windows EC2 instance on AWS. Initially I was getting error message that MPI rejected credentials. I entered and registered the credentials. Now I get the following message “Invalid rank”. USING PARALLEL PROFILE C:\nm75g64\run\para300.pnm MPI TRANSFER TYPE SELECTED Building NONMEM Executable Starting MPI version of nonmem execution ... Manager Location EC2AMAZ-EP7B8E1\\c:\nm75g64\run Fatal error in MPI_Send: Invalid rank, error stack: MPI_Send(173): MPI_Send(buf=00000000037FFD40, count=1, MPI_INTEGER, dest=2, tag=2, MPI_COMM_WORLD) failed MPI_Send(98).: Invalid rank has value 2 but must be nonnegative and less than 2 job aborted: rank: node: exit code[: error message] 0: localhost: 1: process 0 exited without calling finalize 1: localhost: 123 Can anyone provide guidance what this message is and how I can resolve it? Kind regards Parviz

1 0