November 2022 - discuss - mailman3.cels.anl.gov

Re: [mpich-discuss] wrong number of processes on host
by Zhou, Hui 17 Nov '22

17 Nov '22

Bruno, No worries. We always like to hear use cases. We are glad that you found your bug. -- Hui ________________________________ From: bruno.guerraz(a)orange.com <bruno.guerraz(a)orange.com> Sent: Thursday, November 17, 2022 9:41 AM To: Zhou, Hui <zhouh(a)anl.gov>; discuss(a)mpich.org <discuss(a)mpich.org> Subject: Re: [mpich-discuss] wrong number of processes on host I am verry sorry, it is not a mpich bug, the bug is in my app Next time, I'll wait a couple of days before bothering you... On 17/11/2022 16:32, Zhou, Hui wrote: Bruno, No, I was looking for the earlier part of the log where mpiexec launches proxies. You may cut out the dump of environment variables if those are sensitive. -- Hui ________________________________ From: bruno.guerraz(a)orange.com<mailto:[email protected]> <bruno.guerraz(a)orange.com><mailto:[email protected]> Sent: Thursday, November 17, 2022 8:35 AM To: Zhou, Hui <zhouh(a)anl.gov><mailto:[email protected]>; discuss(a)mpich.org<mailto:[email protected]> <discuss(a)mpich.org><mailto:[email protected]> Subject: Re: [mpich-discuss] wrong number of processes on host Is this log OK for you ? [pgid: 0] got PMI command: cmd=barrier_in [pgid: 0] got PMI command: cmd=put sharedFilename[0]=/dev/shm/mpich_shar_tmpAOPuur [pgid: 0] got PMI command: cmd=barrier_in [pgid: 0] got PMI command: cmd=put sharedFilename[4]=/dev/shm/mpich_shar_tmp76zj4w [pgid: 0] got PMI command: cmd=barrier_in PMI response to fd 13 pid 4: cmd=keyval_cache sharedFilename[2]=/dev/shm/mpich_shar_tmppFa2Wf sharedFilename[0]=/dev/shm/mpich_shar_tmpAOPuur sharedFilename[4]=/dev/shm/mpich_shar_tmp76zj4w PMI response to fd 12 pid 4: cmd=keyval_cache sharedFilename[2]=/dev/shm/mpich_shar_tmppFa2Wf sharedFilename[0]=/dev/shm/mpich_shar_tmpAOPuur sharedFilename[4]=/dev/shm/mpich_shar_tmp76zj4w PMI response to fd 14 pid 4: cmd=keyval_cache sharedFilename[2]=/dev/shm/mpich_shar_tmppFa2Wf sharedFilename[0]=/dev/shm/mpich_shar_tmpAOPuur sharedFilename[4]=/dev/shm/mpich_shar_tmp76zj4w PMI response to fd 13 pid 4: cmd=barrier_out PMI response to fd 12 pid 4: cmd=barrier_out PMI response to fd 14 pid 4: cmd=barrier_out [pgid: 0] got PMI command: cmd=put P2-businesscard=description#l-neobi-1$port#39551$ifname#10.193.21.24$ P3-businesscard=description#l-neobi-1$port#49083$ifname#10.193.21.24$ [pgid: 0] got PMI command: cmd=barrier_in [pgid: 0] got PMI command: cmd=put P0-businesscard=description#l-neobi-4$port#57647$ifname#10.193.21.65$ P1-businesscard=description#l-neobi-4$port#46287$ifname#10.193.21.65$ [pgid: 0] got PMI command: cmd=barrier_in [pgid: 0] got PMI command: cmd=put P4-businesscard=description#l-neobi-3$port#48597$ifname#10.193.21.26$ P5-businesscard=description#l-neobi-3$port#47573$ifname#10.193.21.26$ [pgid: 0] got PMI command: cmd=barrier_in PMI response to fd 13 pid 5: cmd=keyval_cache P2-businesscard=description#l-neobi-1$port#39551$ifname#10.193.21.24$ P3-businesscard=description#l-neobi-1$port#49083$ifname#10.193.21.24$ P0-businesscard=description#l-neobi-4$port#57647$ifname#10.193.21.65$ P1-businesscard=description#l-neobi-4$port#46287$ifname#10.193.21.65$ P4-businesscard=description#l-neobi-3$port#48597$ifname#10.193.21.26$ P5-businesscard=description#l-neobi-3$port#47573$ifname#10.193.21.26$ PMI response to fd 12 pid 5: cmd=keyval_cache P2-businesscard=description#l-neobi-1$port#39551$ifname#10.193.21.24$ P3-businesscard=description#l-neobi-1$port#49083$ifname#10.193.21.24$ P0-businesscard=description#l-neobi-4$port#57647$ifname#10.193.21.65$ P1-businesscard=description#l-neobi-4$port#46287$ifname#10.193.21.65$ P4-businesscard=description#l-neobi-3$port#48597$ifname#10.193.21.26$ P5-businesscard=description#l-neobi-3$port#47573$ifname#10.193.21.26$ PMI response to fd 14 pid 5: cmd=keyval_cache P2-businesscard=description#l-neobi-1$port#39551$ifname#10.193.21.24$ P3-businesscard=description#l-neobi-1$port#49083$ifname#10.193.21.24$ P0-businesscard=description#l-neobi-4$port#57647$ifname#10.193.21.65$ P1-businesscard=description#l-neobi-4$port#46287$ifname#10.193.21.65$ P4-businesscard=description#l-neobi-3$port#48597$ifname#10.193.21.26$ P5-businesscard=description#l-neobi-3$port#47573$ifname#10.193.21.26$ PMI response to fd 13 pid 5: cmd=barrier_out PMI response to fd 12 pid 5: cmd=barrier_out PMI response to fd 14 pid 5: cmd=barrier_out On 17/11/2022 15:30, Zhou, Hui wrote: That is strange. Try pass -v option (i.e. mpiexec -v ...) to obtain a console log when that happens. -- Hui ________________________________ From: bruno via discuss <discuss(a)mpich.org><mailto:[email protected]> Sent: Thursday, November 17, 2022 8:11 AM To: discuss(a)mpich.org<mailto:[email protected]> <discuss(a)mpich.org><mailto:[email protected]> Cc: bruno.guerraz(a)orange.com<mailto:[email protected]> <bruno.guerraz(a)orange.com><mailto:[email protected]> Subject: [mpich-discuss] wrong number of processes on host Hi, I am using mpich on a hadoop cluster with yarn. It is not a smooth integration but it is working. Flowing an old post, I am using the manual launcher and the option -disable-hostname-propagation (https://lists.mpich.org/mailman/htdig/devel/2016-July/000717.html) The command line to launch by binary is : mpiexec -launcher manual -disable-hostname-propagation -n 6 -f ./hosts /path/to/my/bin And the file hosts contains host1:2 host2:2 host3:2 I expect to have 2 processes on each host but most of the time it launches 3 processes on host1, 1 on host2 and 2 on host3 Any ideas? Bruno _________________________________________________________________________________________________________________________ Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration, Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci. This message and its attachments may contain confidential or privileged information that may be protected by law; they should not be distributed, used or copied without authorisation. If you have received this email in error, please notify the sender and delete this message and its attachments. As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified. Thank you. _______________________________________________ discuss mailing list discuss(a)mpich.org<mailto:[email protected]> To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss _________________________________________________________________________________________________________________________ Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration, Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci. This message and its attachments may contain confidential or privileged information that may be protected by law; they should not be distributed, used or copied without authorisation. If you have received this email in error, please notify the sender and delete this message and its attachments. As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified. Thank you. _________________________________________________________________________________________________________________________ Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration, Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci. This message and its attachments may contain confidential or privileged information that may be protected by law; they should not be distributed, used or copied without authorisation. If you have received this email in error, please notify the sender and delete this message and its attachments. As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified. Thank you.

1 0

Re: [mpich-discuss] wrong number of processes on host
by bruno.guerraz＠orange.com 17 Nov '22

17 Nov '22

I am verry sorry, it is not a mpich bug, the bug is in my app Next time, I'll wait a couple of days before bothering you... On 17/11/2022 16:32, Zhou, Hui wrote: > Bruno, > > No, I was looking for the earlier part of the log where mpiexec > launches proxies. You may cut out the dump of environment variables if > those are sensitive. > > -- > Hui > ------------------------------------------------------------------------ > *From:* bruno.guerraz(a)orange.com <bruno.guerraz(a)orange.com> > *Sent:* Thursday, November 17, 2022 8:35 AM > *To:* Zhou, Hui <zhouh(a)anl.gov>; discuss(a)mpich.org <discuss(a)mpich.org> > *Subject:* Re: [mpich-discuss] wrong number of processes on host > Is this log OK for you ? > > [pgid: 0] got PMI command: cmd=barrier_in > [pgid: 0] got PMI command: cmd=put > sharedFilename[0]=/dev/shm/mpich_shar_tmpAOPuur > [pgid: 0] got PMI command: cmd=barrier_in > [pgid: 0] got PMI command: cmd=put > sharedFilename[4]=/dev/shm/mpich_shar_tmp76zj4w > [pgid: 0] got PMI command: cmd=barrier_in > PMI response to fd 13 pid 4: cmd=keyval_cache > sharedFilename[2]=/dev/shm/mpich_shar_tmppFa2Wf > sharedFilename[0]=/dev/shm/mpich_shar_tmpAOPuur > sharedFilename[4]=/dev/shm/mpich_shar_tmp76zj4w > PMI response to fd 12 pid 4: cmd=keyval_cache > sharedFilename[2]=/dev/shm/mpich_shar_tmppFa2Wf > sharedFilename[0]=/dev/shm/mpich_shar_tmpAOPuur > sharedFilename[4]=/dev/shm/mpich_shar_tmp76zj4w > PMI response to fd 14 pid 4: cmd=keyval_cache > sharedFilename[2]=/dev/shm/mpich_shar_tmppFa2Wf > sharedFilename[0]=/dev/shm/mpich_shar_tmpAOPuur > sharedFilename[4]=/dev/shm/mpich_shar_tmp76zj4w > PMI response to fd 13 pid 4: cmd=barrier_out > PMI response to fd 12 pid 4: cmd=barrier_out > PMI response to fd 14 pid 4: cmd=barrier_out > [pgid: 0] got PMI command: cmd=put > P2-businesscard=description#l-neobi-1$port#39551$ifname#10.193.21.24$ > P3-businesscard=description#l-neobi-1$port#49083$ifname#10.193.21.24$ > [pgid: 0] got PMI command: cmd=barrier_in > [pgid: 0] got PMI command: cmd=put > P0-businesscard=description#l-neobi-4$port#57647$ifname#10.193.21.65$ > P1-businesscard=description#l-neobi-4$port#46287$ifname#10.193.21.65$ > [pgid: 0] got PMI command: cmd=barrier_in > [pgid: 0] got PMI command: cmd=put > P4-businesscard=description#l-neobi-3$port#48597$ifname#10.193.21.26$ > P5-businesscard=description#l-neobi-3$port#47573$ifname#10.193.21.26$ > [pgid: 0] got PMI command: cmd=barrier_in > PMI response to fd 13 pid 5: cmd=keyval_cache > P2-businesscard=description#l-neobi-1$port#39551$ifname#10.193.21.24$ > P3-businesscard=description#l-neobi-1$port#49083$ifname#10.193.21.24$ > P0-businesscard=description#l-neobi-4$port#57647$ifname#10.193.21.65$ > P1-businesscard=description#l-neobi-4$port#46287$ifname#10.193.21.65$ > P4-businesscard=description#l-neobi-3$port#48597$ifname#10.193.21.26$ > P5-businesscard=description#l-neobi-3$port#47573$ifname#10.193.21.26$ > PMI response to fd 12 pid 5: cmd=keyval_cache > P2-businesscard=description#l-neobi-1$port#39551$ifname#10.193.21.24$ > P3-businesscard=description#l-neobi-1$port#49083$ifname#10.193.21.24$ > P0-businesscard=description#l-neobi-4$port#57647$ifname#10.193.21.65$ > P1-businesscard=description#l-neobi-4$port#46287$ifname#10.193.21.65$ > P4-businesscard=description#l-neobi-3$port#48597$ifname#10.193.21.26$ > P5-businesscard=description#l-neobi-3$port#47573$ifname#10.193.21.26$ > PMI response to fd 14 pid 5: cmd=keyval_cache > P2-businesscard=description#l-neobi-1$port#39551$ifname#10.193.21.24$ > P3-businesscard=description#l-neobi-1$port#49083$ifname#10.193.21.24$ > P0-businesscard=description#l-neobi-4$port#57647$ifname#10.193.21.65$ > P1-businesscard=description#l-neobi-4$port#46287$ifname#10.193.21.65$ > P4-businesscard=description#l-neobi-3$port#48597$ifname#10.193.21.26$ > P5-businesscard=description#l-neobi-3$port#47573$ifname#10.193.21.26$ > PMI response to fd 13 pid 5: cmd=barrier_out > PMI response to fd 12 pid 5: cmd=barrier_out > PMI response to fd 14 pid 5: cmd=barrier_out > > On 17/11/2022 15:30, Zhou, Hui wrote: >> That is strange. Try pass |-v| option (i.e. |mpiexec -v| ...) to >> obtain a console log when that happens. >> >> -- >> Hui >> ------------------------------------------------------------------------ >> *From:* bruno via discuss <discuss(a)mpich.org> <mailto:[email protected]> >> *Sent:* Thursday, November 17, 2022 8:11 AM >> *To:* discuss(a)mpich.org <mailto:[email protected]> >> <discuss(a)mpich.org> <mailto:[email protected]> >> *Cc:* bruno.guerraz(a)orange.com <mailto:[email protected]> >> <bruno.guerraz(a)orange.com> <mailto:[email protected]> >> *Subject:* [mpich-discuss] wrong number of processes on host >> Hi, I am using mpich on a hadoop cluster with yarn. It is not a smooth >> integration but it is working. >> Flowing an old post, I am using the manual launcher and the option >> -disable-hostname-propagation >> (https://lists.mpich.org/mailman/htdig/devel/2016-July/000717.html >> <https://lists.mpich.org/mailman/htdig/devel/2016-July/000717.html>) >> >> The command line to launch by binary is : >> >> mpiexec -launcher manual -disable-hostname-propagation -n 6 -f ./hosts >> /path/to/my/bin >> >> And the file hosts contains >> >> host1:2 >> host2:2 >> host3:2 >> >> I expect to have 2 processes on each host but most of the time it >> launches 3 processes on host1, 1 on host2 and 2 on host3 >> Any ideas? >> >> Bruno >> >> _________________________________________________________________________________________________________________________ >> >> Ce message et ses pieces jointes peuvent contenir des informations >> confidentielles ou privilegiees et ne doivent donc >> pas etre diffuses, exploites ou copies sans autorisation. Si vous >> avez recu ce message par erreur, veuillez le signaler >> a l'expediteur et le detruire ainsi que les pieces jointes. Les >> messages electroniques etant susceptibles d'alteration, >> Orange decline toute responsabilite si ce message a ete altere, >> deforme ou falsifie. Merci. >> >> This message and its attachments may contain confidential or >> privileged information that may be protected by law; >> they should not be distributed, used or copied without authorisation. >> If you have received this email in error, please notify the sender >> and delete this message and its attachments. >> As emails may be altered, Orange is not liable for messages that have >> been modified, changed or falsified. >> Thank you. >> >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org <mailto:[email protected]> >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss >> <https://lists.mpich.org/mailman/listinfo/discuss> > > _________________________________________________________________________________________________________________________ > > Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc > pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler > a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration, > Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci. > > This message and its attachments may contain confidential or privileged information that may be protected by law; > they should not be distributed, used or copied without authorisation. > If you have received this email in error, please notify the sender and delete this message and its attachments. > As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified. > Thank you. _________________________________________________________________________________________________________________________ Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration, Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci. This message and its attachments may contain confidential or privileged information that may be protected by law; they should not be distributed, used or copied without authorisation. If you have received this email in error, please notify the sender and delete this message and its attachments. As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified. Thank you.

1 0

Re: [mpich-discuss] wrong number of processes on host
by Zhou, Hui 17 Nov '22

17 Nov '22

Bruno, No, I was looking for the earlier part of the log where mpiexec launches proxies. You may cut out the dump of environment variables if those are sensitive. -- Hui ________________________________ From: bruno.guerraz(a)orange.com <bruno.guerraz(a)orange.com> Sent: Thursday, November 17, 2022 8:35 AM To: Zhou, Hui <zhouh(a)anl.gov>; discuss(a)mpich.org <discuss(a)mpich.org> Subject: Re: [mpich-discuss] wrong number of processes on host Is this log OK for you ? [pgid: 0] got PMI command: cmd=barrier_in [pgid: 0] got PMI command: cmd=put sharedFilename[0]=/dev/shm/mpich_shar_tmpAOPuur [pgid: 0] got PMI command: cmd=barrier_in [pgid: 0] got PMI command: cmd=put sharedFilename[4]=/dev/shm/mpich_shar_tmp76zj4w [pgid: 0] got PMI command: cmd=barrier_in PMI response to fd 13 pid 4: cmd=keyval_cache sharedFilename[2]=/dev/shm/mpich_shar_tmppFa2Wf sharedFilename[0]=/dev/shm/mpich_shar_tmpAOPuur sharedFilename[4]=/dev/shm/mpich_shar_tmp76zj4w PMI response to fd 12 pid 4: cmd=keyval_cache sharedFilename[2]=/dev/shm/mpich_shar_tmppFa2Wf sharedFilename[0]=/dev/shm/mpich_shar_tmpAOPuur sharedFilename[4]=/dev/shm/mpich_shar_tmp76zj4w PMI response to fd 14 pid 4: cmd=keyval_cache sharedFilename[2]=/dev/shm/mpich_shar_tmppFa2Wf sharedFilename[0]=/dev/shm/mpich_shar_tmpAOPuur sharedFilename[4]=/dev/shm/mpich_shar_tmp76zj4w PMI response to fd 13 pid 4: cmd=barrier_out PMI response to fd 12 pid 4: cmd=barrier_out PMI response to fd 14 pid 4: cmd=barrier_out [pgid: 0] got PMI command: cmd=put P2-businesscard=description#l-neobi-1$port#39551$ifname#10.193.21.24$ P3-businesscard=description#l-neobi-1$port#49083$ifname#10.193.21.24$ [pgid: 0] got PMI command: cmd=barrier_in [pgid: 0] got PMI command: cmd=put P0-businesscard=description#l-neobi-4$port#57647$ifname#10.193.21.65$ P1-businesscard=description#l-neobi-4$port#46287$ifname#10.193.21.65$ [pgid: 0] got PMI command: cmd=barrier_in [pgid: 0] got PMI command: cmd=put P4-businesscard=description#l-neobi-3$port#48597$ifname#10.193.21.26$ P5-businesscard=description#l-neobi-3$port#47573$ifname#10.193.21.26$ [pgid: 0] got PMI command: cmd=barrier_in PMI response to fd 13 pid 5: cmd=keyval_cache P2-businesscard=description#l-neobi-1$port#39551$ifname#10.193.21.24$ P3-businesscard=description#l-neobi-1$port#49083$ifname#10.193.21.24$ P0-businesscard=description#l-neobi-4$port#57647$ifname#10.193.21.65$ P1-businesscard=description#l-neobi-4$port#46287$ifname#10.193.21.65$ P4-businesscard=description#l-neobi-3$port#48597$ifname#10.193.21.26$ P5-businesscard=description#l-neobi-3$port#47573$ifname#10.193.21.26$ PMI response to fd 12 pid 5: cmd=keyval_cache P2-businesscard=description#l-neobi-1$port#39551$ifname#10.193.21.24$ P3-businesscard=description#l-neobi-1$port#49083$ifname#10.193.21.24$ P0-businesscard=description#l-neobi-4$port#57647$ifname#10.193.21.65$ P1-businesscard=description#l-neobi-4$port#46287$ifname#10.193.21.65$ P4-businesscard=description#l-neobi-3$port#48597$ifname#10.193.21.26$ P5-businesscard=description#l-neobi-3$port#47573$ifname#10.193.21.26$ PMI response to fd 14 pid 5: cmd=keyval_cache P2-businesscard=description#l-neobi-1$port#39551$ifname#10.193.21.24$ P3-businesscard=description#l-neobi-1$port#49083$ifname#10.193.21.24$ P0-businesscard=description#l-neobi-4$port#57647$ifname#10.193.21.65$ P1-businesscard=description#l-neobi-4$port#46287$ifname#10.193.21.65$ P4-businesscard=description#l-neobi-3$port#48597$ifname#10.193.21.26$ P5-businesscard=description#l-neobi-3$port#47573$ifname#10.193.21.26$ PMI response to fd 13 pid 5: cmd=barrier_out PMI response to fd 12 pid 5: cmd=barrier_out PMI response to fd 14 pid 5: cmd=barrier_out On 17/11/2022 15:30, Zhou, Hui wrote: That is strange. Try pass -v option (i.e. mpiexec -v ...) to obtain a console log when that happens. -- Hui ________________________________ From: bruno via discuss <discuss(a)mpich.org><mailto:[email protected]> Sent: Thursday, November 17, 2022 8:11 AM To: discuss(a)mpich.org<mailto:[email protected]> <discuss(a)mpich.org><mailto:[email protected]> Cc: bruno.guerraz(a)orange.com<mailto:[email protected]> <bruno.guerraz(a)orange.com><mailto:[email protected]> Subject: [mpich-discuss] wrong number of processes on host Hi, I am using mpich on a hadoop cluster with yarn. It is not a smooth integration but it is working. Flowing an old post, I am using the manual launcher and the option -disable-hostname-propagation (https://lists.mpich.org/mailman/htdig/devel/2016-July/000717.html) The command line to launch by binary is : mpiexec -launcher manual -disable-hostname-propagation -n 6 -f ./hosts /path/to/my/bin And the file hosts contains host1:2 host2:2 host3:2 I expect to have 2 processes on each host but most of the time it launches 3 processes on host1, 1 on host2 and 2 on host3 Any ideas? Bruno _________________________________________________________________________________________________________________________ Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration, Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci. This message and its attachments may contain confidential or privileged information that may be protected by law; they should not be distributed, used or copied without authorisation. If you have received this email in error, please notify the sender and delete this message and its attachments. As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified. Thank you. _______________________________________________ discuss mailing list discuss(a)mpich.org<mailto:[email protected]> To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss _________________________________________________________________________________________________________________________ Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration, Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci. This message and its attachments may contain confidential or privileged information that may be protected by law; they should not be distributed, used or copied without authorisation. If you have received this email in error, please notify the sender and delete this message and its attachments. As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified. Thank you.

1 0

Re: [mpich-discuss] wrong number of processes on host
by bruno.guerraz＠orange.com 17 Nov '22

17 Nov '22

Is this log OK for you ? [pgid: 0] got PMI command: cmd=barrier_in [pgid: 0] got PMI command: cmd=put sharedFilename[0]=/dev/shm/mpich_shar_tmpAOPuur [pgid: 0] got PMI command: cmd=barrier_in [pgid: 0] got PMI command: cmd=put sharedFilename[4]=/dev/shm/mpich_shar_tmp76zj4w [pgid: 0] got PMI command: cmd=barrier_in PMI response to fd 13 pid 4: cmd=keyval_cache sharedFilename[2]=/dev/shm/mpich_shar_tmppFa2Wf sharedFilename[0]=/dev/shm/mpich_shar_tmpAOPuur sharedFilename[4]=/dev/shm/mpich_shar_tmp76zj4w PMI response to fd 12 pid 4: cmd=keyval_cache sharedFilename[2]=/dev/shm/mpich_shar_tmppFa2Wf sharedFilename[0]=/dev/shm/mpich_shar_tmpAOPuur sharedFilename[4]=/dev/shm/mpich_shar_tmp76zj4w PMI response to fd 14 pid 4: cmd=keyval_cache sharedFilename[2]=/dev/shm/mpich_shar_tmppFa2Wf sharedFilename[0]=/dev/shm/mpich_shar_tmpAOPuur sharedFilename[4]=/dev/shm/mpich_shar_tmp76zj4w PMI response to fd 13 pid 4: cmd=barrier_out PMI response to fd 12 pid 4: cmd=barrier_out PMI response to fd 14 pid 4: cmd=barrier_out [pgid: 0] got PMI command: cmd=put P2-businesscard=description#l-neobi-1$port#39551$ifname#10.193.21.24$ P3-businesscard=description#l-neobi-1$port#49083$ifname#10.193.21.24$ [pgid: 0] got PMI command: cmd=barrier_in [pgid: 0] got PMI command: cmd=put P0-businesscard=description#l-neobi-4$port#57647$ifname#10.193.21.65$ P1-businesscard=description#l-neobi-4$port#46287$ifname#10.193.21.65$ [pgid: 0] got PMI command: cmd=barrier_in [pgid: 0] got PMI command: cmd=put P4-businesscard=description#l-neobi-3$port#48597$ifname#10.193.21.26$ P5-businesscard=description#l-neobi-3$port#47573$ifname#10.193.21.26$ [pgid: 0] got PMI command: cmd=barrier_in PMI response to fd 13 pid 5: cmd=keyval_cache P2-businesscard=description#l-neobi-1$port#39551$ifname#10.193.21.24$ P3-businesscard=description#l-neobi-1$port#49083$ifname#10.193.21.24$ P0-businesscard=description#l-neobi-4$port#57647$ifname#10.193.21.65$ P1-businesscard=description#l-neobi-4$port#46287$ifname#10.193.21.65$ P4-businesscard=description#l-neobi-3$port#48597$ifname#10.193.21.26$ P5-businesscard=description#l-neobi-3$port#47573$ifname#10.193.21.26$ PMI response to fd 12 pid 5: cmd=keyval_cache P2-businesscard=description#l-neobi-1$port#39551$ifname#10.193.21.24$ P3-businesscard=description#l-neobi-1$port#49083$ifname#10.193.21.24$ P0-businesscard=description#l-neobi-4$port#57647$ifname#10.193.21.65$ P1-businesscard=description#l-neobi-4$port#46287$ifname#10.193.21.65$ P4-businesscard=description#l-neobi-3$port#48597$ifname#10.193.21.26$ P5-businesscard=description#l-neobi-3$port#47573$ifname#10.193.21.26$ PMI response to fd 14 pid 5: cmd=keyval_cache P2-businesscard=description#l-neobi-1$port#39551$ifname#10.193.21.24$ P3-businesscard=description#l-neobi-1$port#49083$ifname#10.193.21.24$ P0-businesscard=description#l-neobi-4$port#57647$ifname#10.193.21.65$ P1-businesscard=description#l-neobi-4$port#46287$ifname#10.193.21.65$ P4-businesscard=description#l-neobi-3$port#48597$ifname#10.193.21.26$ P5-businesscard=description#l-neobi-3$port#47573$ifname#10.193.21.26$ PMI response to fd 13 pid 5: cmd=barrier_out PMI response to fd 12 pid 5: cmd=barrier_out PMI response to fd 14 pid 5: cmd=barrier_out On 17/11/2022 15:30, Zhou, Hui wrote: > That is strange. Try pass |-v| option (i.e. |mpiexec -v| ...) to > obtain a console log when that happens. > > -- > Hui > ------------------------------------------------------------------------ > *From:* bruno via discuss <discuss(a)mpich.org> > *Sent:* Thursday, November 17, 2022 8:11 AM > *To:* discuss(a)mpich.org <discuss(a)mpich.org> > *Cc:* bruno.guerraz(a)orange.com <bruno.guerraz(a)orange.com> > *Subject:* [mpich-discuss] wrong number of processes on host > Hi, I am using mpich on a hadoop cluster with yarn. It is not a smooth > integration but it is working. > Flowing an old post, I am using the manual launcher and the option > -disable-hostname-propagation > (https://lists.mpich.org/mailman/htdig/devel/2016-July/000717.html) > > The command line to launch by binary is : > > mpiexec -launcher manual -disable-hostname-propagation -n 6 -f ./hosts > /path/to/my/bin > > And the file hosts contains > > host1:2 > host2:2 > host3:2 > > I expect to have 2 processes on each host but most of the time it > launches 3 processes on host1, 1 on host2 and 2 on host3 > Any ideas? > > Bruno > > _________________________________________________________________________________________________________________________ > > Ce message et ses pieces jointes peuvent contenir des informations > confidentielles ou privilegiees et ne doivent donc > pas etre diffuses, exploites ou copies sans autorisation. Si vous avez > recu ce message par erreur, veuillez le signaler > a l'expediteur et le detruire ainsi que les pieces jointes. Les > messages electroniques etant susceptibles d'alteration, > Orange decline toute responsabilite si ce message a ete altere, > deforme ou falsifie. Merci. > > This message and its attachments may contain confidential or > privileged information that may be protected by law; > they should not be distributed, used or copied without authorisation. > If you have received this email in error, please notify the sender and > delete this message and its attachments. > As emails may be altered, Orange is not liable for messages that have > been modified, changed or falsified. > Thank you. > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss _________________________________________________________________________________________________________________________ Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration, Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci. This message and its attachments may contain confidential or privileged information that may be protected by law; they should not be distributed, used or copied without authorisation. If you have received this email in error, please notify the sender and delete this message and its attachments. As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified. Thank you.

1 0

Re: [mpich-discuss] wrong number of processes on host
by Zhou, Hui 17 Nov '22

17 Nov '22

That is strange. Try pass -v option (i.e. mpiexec -v ...) to obtain a console log when that happens. -- Hui ________________________________ From: bruno via discuss <discuss(a)mpich.org> Sent: Thursday, November 17, 2022 8:11 AM To: discuss(a)mpich.org <discuss(a)mpich.org> Cc: bruno.guerraz(a)orange.com <bruno.guerraz(a)orange.com> Subject: [mpich-discuss] wrong number of processes on host Hi, I am using mpich on a hadoop cluster with yarn. It is not a smooth integration but it is working. Flowing an old post, I am using the manual launcher and the option -disable-hostname-propagation (https://lists.mpich.org/mailman/htdig/devel/2016-July/000717.html) The command line to launch by binary is : mpiexec -launcher manual -disable-hostname-propagation -n 6 -f ./hosts /path/to/my/bin And the file hosts contains host1:2 host2:2 host3:2 I expect to have 2 processes on each host but most of the time it launches 3 processes on host1, 1 on host2 and 2 on host3 Any ideas? Bruno _________________________________________________________________________________________________________________________ Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration, Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci. This message and its attachments may contain confidential or privileged information that may be protected by law; they should not be distributed, used or copied without authorisation. If you have received this email in error, please notify the sender and delete this message and its attachments. As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified. Thank you. _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

wrong number of processes on host
by bruno.guerraz＠orange.com 17 Nov '22

17 Nov '22

Hi, I am using mpich on a hadoop cluster with yarn. It is not a smooth integration but it is working. Flowing an old post, I am using the manual launcher and the option -disable-hostname-propagation (https://lists.mpich.org/mailman/htdig/devel/2016-July/000717.html) The command line to launch by binary is : mpiexec -launcher manual -disable-hostname-propagation -n 6 -f ./hosts /path/to/my/bin And the file hosts contains host1:2 host2:2 host3:2 I expect to have 2 processes on each host but most of the time it launches 3 processes on host1, 1 on host2 and 2 on host3 Any ideas? Bruno _________________________________________________________________________________________________________________________ Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration, Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci. This message and its attachments may contain confidential or privileged information that may be protected by law; they should not be distributed, used or copied without authorisation. If you have received this email in error, please notify the sender and delete this message and its attachments. As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified. Thank you.

1 0

Join the MPICH BoF this Wednesday evening at SC22
by Raffenetti, Ken 14 Nov '22

14 Nov '22

All, The MPICH team will host a Birds-of-a-Feather session this Wednesday evening at 5:15pm CST as part of the SC22 proceedings. We hope to see you there to discuss the latest developments in MPICH, and to hear from partners Intel, HPE, ParTec, MVAPICH, NVIDIA, and Cornelis Networks. Visit the link below for more info. https://github.com/pmodels/mpich/wiki/2022-Supercomputing-BoF Cheers, The MPICH team

1 0

Re: [mpich-discuss] Crash on MPI_Rput
by Palmer, Bruce J 10 Nov '22

10 Nov '22

Oops. I forgot to hit send on this after I wrote it. Sorry about the delay. You can download the latest ga release at https://github.com/GlobalArrays/ga/releases/tag/v5.8.2 (this is what I’ve been testing with lately). I’ve been configuring with ../configure --enable-i4 --enable-cxx --with-mpi3\ --prefix=/people/d3g293/tmp/ga-hotfiix/build_rma_314 CC=mpicc CXX=mpicxx FC=mpif90 \ CFLAGS="-g" CXXFLAG="-g" FFLAGS="-g" After configuring and building, you can run the test suite using make check-ga MPIEXEC="mpirun -n 4 " Our standard configuration for running the test suite is 4 processors on 2 nodes. From: Zhou, Hui <zhouh(a)anl.gov> Date: Friday, November 4, 2022 at 2:09 PM To: discuss(a)mpich.org <discuss(a)mpich.org> Cc: Palmer, Bruce J <Bruce.Palmer(a)pnnl.gov> Subject: Re: [mpich-discuss] Crash on MPI_Rput Hi Bruce, Is the test suite available for us to checkout and test? -- Hui ________________________________ From: Palmer, Bruce J via discuss <discuss(a)mpich.org> Sent: Friday, November 4, 2022 4:03 PM To: discuss(a)mpich.org <discuss(a)mpich.org> Cc: Palmer, Bruce J <Bruce.Palmer(a)pnnl.gov> Subject: Re: [mpich-discuss] Crash on MPI_Rput I kind of dropped this for a while but I’d like to pick it back up. I did some more testing using different versions of mpich and got the following results for the RMA runtime MPICH-3.1.4 configured with ./configure --prefix=/people/d3g293/mpich/mpich-3.1.4/install --with-libfabric=embedded --enable-threads=multiple --with-slurm CC=gcc CXX=g++ 2/80 tests fail in GA test suite MPICH-4.0.2 configured with unset F90 ./configure --prefix=/people/d3g293/mpich/mpich-4.0.2/install --with-device=ch4:ofi:sockets --with-libfabric=embedded --enable-threads=multiple --with-slurm CC=gcc CXX=g++ 25/80 tests fail in GA test suite Running with MPICH-3.3.2 seems to lead to around 8 failures, but my notes on this aren’t that good. If I run with OpenMPI 4.1.4, everything passes. Any reason for why I’m seeing this? I haven’t really done much to this runtime in the last few years. Bruce From: Palmer, Bruce J via discuss <discuss(a)mpich.org> Date: Wednesday, September 28, 2022 at 12:30 PM To: 'Thakur, Rajeev' <thakur(a)anl.gov>, discuss(a)mpich.org <discuss(a)mpich.org>, Zhou, Hui <zhouh(a)anl.gov> Cc: Palmer, Bruce J <Bruce.Palmer(a)pnnl.gov> Subject: Re: [mpich-discuss] Crash on MPI_Rput Check twice before you click! This email originated from outside PNNL. I think the MPI-RMA runtime was mostly (maybe completely) working with 3.2-3.4. It may have even been working earlier with 4.0. I think there is a pretty good chance that the problem is a system configuration problem at our end and I was hoping that you might have some insight into what it might be based on the errors I’m seeing. I can try running with a few earlier versions of mpich and see if any of them work better. Bruce From: Thakur, Rajeev <thakur(a)anl.gov> Sent: Wednesday, September 28, 2022 12:24 PM To: discuss(a)mpich.org; Zhou, Hui <zhouh(a)anl.gov> Cc: Palmer, Bruce J <Bruce.Palmer(a)pnnl.gov> Subject: Re: [mpich-discuss] Crash on MPI_Rput Was it working with an earlier version of MPICH? If so, which one? Rajeev From: "Palmer, Bruce J via discuss" <discuss(a)mpich.org<mailto:[email protected]>> Reply-To: "discuss(a)mpich.org<mailto:[email protected]>" <discuss(a)mpich.org<mailto:[email protected]>> Date: Wednesday, September 28, 2022 at 2:20 PM To: "Zhou, Hui" <zhouh(a)anl.gov<mailto:[email protected]>>, "discuss(a)mpich.org<mailto:[email protected]>" <discuss(a)mpich.org<mailto:[email protected]>> Cc: "Palmer, Bruce J" <Bruce.Palmer(a)pnnl.gov<mailto:[email protected]>> Subject: Re: [mpich-discuss] Crash on MPI_Rput I upgraded to mpich-4.0.2 (latest stable release) and get pretty much the same result. This failure is reproducible, I get the same error on multiple runs so it doesn’t look like an unexpected process failure. One other feature that I forgot to mention earlier is that I’m running this test on 4 processors distributed over 2 nodes. If I run 4 processes on 1 node, the code runs without error. Bruce From: Zhou, Hui <zhouh(a)anl.gov<mailto:[email protected]>> Date: Tuesday, September 27, 2022 at 2:55 PM To: discuss(a)mpich.org<mailto:[email protected]> <discuss(a)mpich.org<mailto:[email protected]>> Cc: Palmer, Bruce J <Bruce.Palmer(a)pnnl.gov<mailto:[email protected]>> Subject: Re: Crash on MPI_Rput Hi Bruce, * srun: error: node003: task 1: Exited with exit code 7 Looks like one of the process crashed unexpectedly. -- Hui Zhou From: Palmer, Bruce J via discuss <discuss(a)mpich.org<mailto:[email protected]>> Date: Tuesday, September 27, 2022 at 3:32 PM To: discuss(a)mpich.org<mailto:[email protected]> <discuss(a)mpich.org<mailto:[email protected]>> Cc: Palmer, Bruce J <Bruce.Palmer(a)pnnl.gov<mailto:[email protected]>> Subject: [mpich-discuss] Crash on MPI_Rput Hi, I’m testing the MPI-RMA runtime in Global Arrays and I’m getting a lot more crashes than I’ve seen in the past. The MPI-RMA runtime code is fairly stable and hasn’t been modified much recently and all the tests used to pass using one of the more recent MPICH releases. However, I’m getting significant crashes at this point. One of them occurs in a program designed to test non-blocking communication. It creates an MPI window, using MPI_Alloc_mem followed by MPI_Win_create and then calls MPI_Win_lock_all on the window. The code currently crashes when it gets to an MPI_Rput call. I’m trying to see if there is something different in the environment that might be causing this. I’m currently up to MPICH-4.0b1 configured with ./configure --prefix=/people/d3g293/mpich/mpich-4.0b1/install --with-device=ch4:ofi:sockets --with-libfabric=embedded --enable-threads=multiple --with-slurm CC=gcc CXX=g++ #./configure --prefix=/people/d3g293/mpich/mpich-3.4.1/install-newell-nocuda --with-device=ch4:ofi:sockets --with-libfabric=embedded --enable-threads=multiple --with-slurm CC=gcc CXX=g++ I’ve tried other recent vintages of MPICH, but I get similar results. The error I’m seeing when the program crashes is [proxy:0:[email protected]] HYD_pmcd_pmip_control_cmd_cb (pm/pmiserv/pmip_cb.c:899): assert (!closed) failed [proxy:0:[email protected]] HYDT_dmxu_poll_wait_for_event (tools/demux/demux_poll.c:76): callback returned error status srun: error: node003: task 1: Exited with exit code 7 [proxy:0:[email protected]] main (pm/pmiserv/pmip.c:169): demux engine error waiting for event [mpiexec(a)node002.local] HYDT_bscu_wait_for_completion (tools/bootstrap/utils/bscu_wait.c:74): one of the processes terminated badly; aborting [mpiexec(a)node002.local] HYDT_bsci_wait_for_completion (tools/bootstrap/src/bsci_wait.c:21): launcher returned error waiting for completion [mpiexec(a)node002.local] HYD_pmci_wait_for_completion (pm/pmiserv/pmiserv_pmci.c:179): launcher returned error waiting for completion [mpiexec(a)node002.local] main (ui/mpich/mpiexec.c:325): process manager error waiting for completion Any suggestions about what might be going wrong here? It could be a problem with the machine configuration, since this code seemed to be running fine a while ago and has not been modified since then. I’ll try building the latest stable release and see if that fixes anything, but as I mentioned none of the recent releases seems to work. Bruce Palmer Computer Scientist Pacific Northwest National Laboratory (509) 375-3899

1 0

Re: [mpich-discuss] Crash on MPI_Rput
by Zhou, Hui 04 Nov '22

04 Nov '22

Hi Bruce, Is the test suite available for us to checkout and test? -- Hui ________________________________ From: Palmer, Bruce J via discuss <discuss(a)mpich.org> Sent: Friday, November 4, 2022 4:03 PM To: discuss(a)mpich.org <discuss(a)mpich.org> Cc: Palmer, Bruce J <Bruce.Palmer(a)pnnl.gov> Subject: Re: [mpich-discuss] Crash on MPI_Rput I kind of dropped this for a while but I’d like to pick it back up. I did some more testing using different versions of mpich and got the following results for the RMA runtime MPICH-3.1.4 configured with ./configure --prefix=/people/d3g293/mpich/mpich-3.1.4/install --with-libfabric=embedded --enable-threads=multiple --with-slurm CC=gcc CXX=g++ 2/80 tests fail in GA test suite MPICH-4.0.2 configured with unset F90 ./configure --prefix=/people/d3g293/mpich/mpich-4.0.2/install --with-device=ch4:ofi:sockets --with-libfabric=embedded --enable-threads=multiple --with-slurm CC=gcc CXX=g++ 25/80 tests fail in GA test suite Running with MPICH-3.3.2 seems to lead to around 8 failures, but my notes on this aren’t that good. If I run with OpenMPI 4.1.4, everything passes. Any reason for why I’m seeing this? I haven’t really done much to this runtime in the last few years. Bruce From: Palmer, Bruce J via discuss <discuss(a)mpich.org> Date: Wednesday, September 28, 2022 at 12:30 PM To: 'Thakur, Rajeev' <thakur(a)anl.gov>, discuss(a)mpich.org <discuss(a)mpich.org>, Zhou, Hui <zhouh(a)anl.gov> Cc: Palmer, Bruce J <Bruce.Palmer(a)pnnl.gov> Subject: Re: [mpich-discuss] Crash on MPI_Rput Check twice before you click! This email originated from outside PNNL. I think the MPI-RMA runtime was mostly (maybe completely) working with 3.2-3.4. It may have even been working earlier with 4.0. I think there is a pretty good chance that the problem is a system configuration problem at our end and I was hoping that you might have some insight into what it might be based on the errors I’m seeing. I can try running with a few earlier versions of mpich and see if any of them work better. Bruce From: Thakur, Rajeev <thakur(a)anl.gov> Sent: Wednesday, September 28, 2022 12:24 PM To: discuss(a)mpich.org; Zhou, Hui <zhouh(a)anl.gov> Cc: Palmer, Bruce J <Bruce.Palmer(a)pnnl.gov> Subject: Re: [mpich-discuss] Crash on MPI_Rput Was it working with an earlier version of MPICH? If so, which one? Rajeev From: "Palmer, Bruce J via discuss" <discuss(a)mpich.org<mailto:[email protected]>> Reply-To: "discuss(a)mpich.org<mailto:[email protected]>" <discuss(a)mpich.org<mailto:[email protected]>> Date: Wednesday, September 28, 2022 at 2:20 PM To: "Zhou, Hui" <zhouh(a)anl.gov<mailto:[email protected]>>, "discuss(a)mpich.org<mailto:[email protected]>" <discuss(a)mpich.org<mailto:[email protected]>> Cc: "Palmer, Bruce J" <Bruce.Palmer(a)pnnl.gov<mailto:[email protected]>> Subject: Re: [mpich-discuss] Crash on MPI_Rput I upgraded to mpich-4.0.2 (latest stable release) and get pretty much the same result. This failure is reproducible, I get the same error on multiple runs so it doesn’t look like an unexpected process failure. One other feature that I forgot to mention earlier is that I’m running this test on 4 processors distributed over 2 nodes. If I run 4 processes on 1 node, the code runs without error. Bruce From: Zhou, Hui <zhouh(a)anl.gov<mailto:[email protected]>> Date: Tuesday, September 27, 2022 at 2:55 PM To: discuss(a)mpich.org<mailto:[email protected]> <discuss(a)mpich.org<mailto:[email protected]>> Cc: Palmer, Bruce J <Bruce.Palmer(a)pnnl.gov<mailto:[email protected]>> Subject: Re: Crash on MPI_Rput Hi Bruce, * srun: error: node003: task 1: Exited with exit code 7 Looks like one of the process crashed unexpectedly. -- Hui Zhou From: Palmer, Bruce J via discuss <discuss(a)mpich.org<mailto:[email protected]>> Date: Tuesday, September 27, 2022 at 3:32 PM To: discuss(a)mpich.org<mailto:[email protected]> <discuss(a)mpich.org<mailto:[email protected]>> Cc: Palmer, Bruce J <Bruce.Palmer(a)pnnl.gov<mailto:[email protected]>> Subject: [mpich-discuss] Crash on MPI_Rput Hi, I’m testing the MPI-RMA runtime in Global Arrays and I’m getting a lot more crashes than I’ve seen in the past. The MPI-RMA runtime code is fairly stable and hasn’t been modified much recently and all the tests used to pass using one of the more recent MPICH releases. However, I’m getting significant crashes at this point. One of them occurs in a program designed to test non-blocking communication. It creates an MPI window, using MPI_Alloc_mem followed by MPI_Win_create and then calls MPI_Win_lock_all on the window. The code currently crashes when it gets to an MPI_Rput call. I’m trying to see if there is something different in the environment that might be causing this. I’m currently up to MPICH-4.0b1 configured with ./configure --prefix=/people/d3g293/mpich/mpich-4.0b1/install --with-device=ch4:ofi:sockets --with-libfabric=embedded --enable-threads=multiple --with-slurm CC=gcc CXX=g++ #./configure --prefix=/people/d3g293/mpich/mpich-3.4.1/install-newell-nocuda --with-device=ch4:ofi:sockets --with-libfabric=embedded --enable-threads=multiple --with-slurm CC=gcc CXX=g++ I’ve tried other recent vintages of MPICH, but I get similar results. The error I’m seeing when the program crashes is [proxy:0:[email protected]] HYD_pmcd_pmip_control_cmd_cb (pm/pmiserv/pmip_cb.c:899): assert (!closed) failed [proxy:0:[email protected]] HYDT_dmxu_poll_wait_for_event (tools/demux/demux_poll.c:76): callback returned error status srun: error: node003: task 1: Exited with exit code 7 [proxy:0:[email protected]] main (pm/pmiserv/pmip.c:169): demux engine error waiting for event [mpiexec(a)node002.local] HYDT_bscu_wait_for_completion (tools/bootstrap/utils/bscu_wait.c:74): one of the processes terminated badly; aborting [mpiexec(a)node002.local] HYDT_bsci_wait_for_completion (tools/bootstrap/src/bsci_wait.c:21): launcher returned error waiting for completion [mpiexec(a)node002.local] HYD_pmci_wait_for_completion (pm/pmiserv/pmiserv_pmci.c:179): launcher returned error waiting for completion [mpiexec(a)node002.local] main (ui/mpich/mpiexec.c:325): process manager error waiting for completion Any suggestions about what might be going wrong here? It could be a problem with the machine configuration, since this code seemed to be running fine a while ago and has not been modified since then. I’ll try building the latest stable release and see if that fixes anything, but as I mentioned none of the recent releases seems to work. Bruce Palmer Computer Scientist Pacific Northwest National Laboratory (509) 375-3899

1 0

Re: [mpich-discuss] Crash on MPI_Rput
by Palmer, Bruce J 04 Nov '22

04 Nov '22

I kind of dropped this for a while but I’d like to pick it back up. I did some more testing using different versions of mpich and got the following results for the RMA runtime MPICH-3.1.4 configured with ./configure --prefix=/people/d3g293/mpich/mpich-3.1.4/install --with-libfabric=embedded --enable-threads=multiple --with-slurm CC=gcc CXX=g++ 2/80 tests fail in GA test suite MPICH-4.0.2 configured with unset F90 ./configure --prefix=/people/d3g293/mpich/mpich-4.0.2/install --with-device=ch4:ofi:sockets --with-libfabric=embedded --enable-threads=multiple --with-slurm CC=gcc CXX=g++ 25/80 tests fail in GA test suite Running with MPICH-3.3.2 seems to lead to around 8 failures, but my notes on this aren’t that good. If I run with OpenMPI 4.1.4, everything passes. Any reason for why I’m seeing this? I haven’t really done much to this runtime in the last few years. Bruce From: Palmer, Bruce J via discuss <discuss(a)mpich.org> Date: Wednesday, September 28, 2022 at 12:30 PM To: 'Thakur, Rajeev' <thakur(a)anl.gov>, discuss(a)mpich.org <discuss(a)mpich.org>, Zhou, Hui <zhouh(a)anl.gov> Cc: Palmer, Bruce J <Bruce.Palmer(a)pnnl.gov> Subject: Re: [mpich-discuss] Crash on MPI_Rput Check twice before you click! This email originated from outside PNNL. I think the MPI-RMA runtime was mostly (maybe completely) working with 3.2-3.4. It may have even been working earlier with 4.0. I think there is a pretty good chance that the problem is a system configuration problem at our end and I was hoping that you might have some insight into what it might be based on the errors I’m seeing. I can try running with a few earlier versions of mpich and see if any of them work better. Bruce From: Thakur, Rajeev <thakur(a)anl.gov> Sent: Wednesday, September 28, 2022 12:24 PM To: discuss(a)mpich.org; Zhou, Hui <zhouh(a)anl.gov> Cc: Palmer, Bruce J <Bruce.Palmer(a)pnnl.gov> Subject: Re: [mpich-discuss] Crash on MPI_Rput Was it working with an earlier version of MPICH? If so, which one? Rajeev From: "Palmer, Bruce J via discuss" <discuss(a)mpich.org<mailto:[email protected]>> Reply-To: "discuss(a)mpich.org<mailto:[email protected]>" <discuss(a)mpich.org<mailto:[email protected]>> Date: Wednesday, September 28, 2022 at 2:20 PM To: "Zhou, Hui" <zhouh(a)anl.gov<mailto:[email protected]>>, "discuss(a)mpich.org<mailto:[email protected]>" <discuss(a)mpich.org<mailto:[email protected]>> Cc: "Palmer, Bruce J" <Bruce.Palmer(a)pnnl.gov<mailto:[email protected]>> Subject: Re: [mpich-discuss] Crash on MPI_Rput I upgraded to mpich-4.0.2 (latest stable release) and get pretty much the same result. This failure is reproducible, I get the same error on multiple runs so it doesn’t look like an unexpected process failure. One other feature that I forgot to mention earlier is that I’m running this test on 4 processors distributed over 2 nodes. If I run 4 processes on 1 node, the code runs without error. Bruce From: Zhou, Hui <zhouh(a)anl.gov<mailto:[email protected]>> Date: Tuesday, September 27, 2022 at 2:55 PM To: discuss(a)mpich.org<mailto:[email protected]> <discuss(a)mpich.org<mailto:[email protected]>> Cc: Palmer, Bruce J <Bruce.Palmer(a)pnnl.gov<mailto:[email protected]>> Subject: Re: Crash on MPI_Rput Hi Bruce, * srun: error: node003: task 1: Exited with exit code 7 Looks like one of the process crashed unexpectedly. -- Hui Zhou From: Palmer, Bruce J via discuss <discuss(a)mpich.org<mailto:[email protected]>> Date: Tuesday, September 27, 2022 at 3:32 PM To: discuss(a)mpich.org<mailto:[email protected]> <discuss(a)mpich.org<mailto:[email protected]>> Cc: Palmer, Bruce J <Bruce.Palmer(a)pnnl.gov<mailto:[email protected]>> Subject: [mpich-discuss] Crash on MPI_Rput Hi, I’m testing the MPI-RMA runtime in Global Arrays and I’m getting a lot more crashes than I’ve seen in the past. The MPI-RMA runtime code is fairly stable and hasn’t been modified much recently and all the tests used to pass using one of the more recent MPICH releases. However, I’m getting significant crashes at this point. One of them occurs in a program designed to test non-blocking communication. It creates an MPI window, using MPI_Alloc_mem followed by MPI_Win_create and then calls MPI_Win_lock_all on the window. The code currently crashes when it gets to an MPI_Rput call. I’m trying to see if there is something different in the environment that might be causing this. I’m currently up to MPICH-4.0b1 configured with ./configure --prefix=/people/d3g293/mpich/mpich-4.0b1/install --with-device=ch4:ofi:sockets --with-libfabric=embedded --enable-threads=multiple --with-slurm CC=gcc CXX=g++ #./configure --prefix=/people/d3g293/mpich/mpich-3.4.1/install-newell-nocuda --with-device=ch4:ofi:sockets --with-libfabric=embedded --enable-threads=multiple --with-slurm CC=gcc CXX=g++ I’ve tried other recent vintages of MPICH, but I get similar results. The error I’m seeing when the program crashes is [proxy:0:[email protected]] HYD_pmcd_pmip_control_cmd_cb (pm/pmiserv/pmip_cb.c:899): assert (!closed) failed [proxy:0:[email protected]] HYDT_dmxu_poll_wait_for_event (tools/demux/demux_poll.c:76): callback returned error status srun: error: node003: task 1: Exited with exit code 7 [proxy:0:[email protected]] main (pm/pmiserv/pmip.c:169): demux engine error waiting for event [mpiexec(a)node002.local] HYDT_bscu_wait_for_completion (tools/bootstrap/utils/bscu_wait.c:74): one of the processes terminated badly; aborting [mpiexec(a)node002.local] HYDT_bsci_wait_for_completion (tools/bootstrap/src/bsci_wait.c:21): launcher returned error waiting for completion [mpiexec(a)node002.local] HYD_pmci_wait_for_completion (pm/pmiserv/pmiserv_pmci.c:179): launcher returned error waiting for completion [mpiexec(a)node002.local] main (ui/mpich/mpiexec.c:325): process manager error waiting for completion Any suggestions about what might be going wrong here? It could be a problem with the machine configuration, since this code seemed to be running fine a while ago and has not been modified since then. I’ll try building the latest stable release and see if that fixes anything, but as I mentioned none of the recent releases seems to work. Bruce Palmer Computer Scientist Pacific Northwest National Laboratory (509) 375-3899

1 0