- discuss - mailman3.cels.anl.gov

Re: [mpich-discuss] MPICH Test Suite - limit number of processes
by Max R. Dechantsreiter 15 Jan '23

15 Jan '23

Thank you for your suggestion. However, I did not see how to carry it through; perhaps one day I will comb through the User's Guide. Instead I ran 'make testing' to completion on four cores: elapsed time was 2h36m53s on Intel(R) Xeon(R) CPU E5-2620 v4 (2.10GHz); all but two tests passed, with the two that did not pass simply timing out (after 180s). Then I ran it again after modifying "runtests" thus: sed -i 's/^$ppnMax = -1;/$ppnMax = 4;/' $RUNTESTS sed -i 's/^$np_max = -1;/$np_max = 4;/' $RUNTESTS where RUNTESTS=~test/mpi/runtests. (Alternatively these variables could be modified in $RUNTESTS.in before prior to building MPICH.) Changing those variables limited the maximum number of processes to 4 as desired. The salient issues were the warnings 'Number of processors must divide 10' that I presume could be avoided by setting $np_max=2 or 5; the number of tests affected was relatively small: 13 of 2524 tests failed, with no test timing out. Elapsed time was 1h04m28s - so the savings in time was significant but not spectacular, suggesting that relatively few tests in the original run oversubscribed the cores available in my VPS. On Thu, Jan 12, 2023 at 04:29:07PM +0000, Zhou, Hui wrote: > Unfortunately, the number of processes for each test is hardcoded in the testlist files. There is no mechanism for filtering or dynamically modifying them. You could either maintain a custom patch to modify the testlist, or have a script applying a separate xfail list. An example of such xfail list that we used in our internal testing is in test/mpi/maint/jenkins/xfail.conf. If the custom patch solution won't work in your situation, please open an github issue describing your use case, and we can discuss to add a filtering option to the runtests script. > > -- > Hui > ________________________________ > From: Max R. Dechantsreiter via discuss <discuss(a)mpich.org> > Sent: Thursday, January 12, 2023 9:58 AM > To: discuss(a)mpich.org <discuss(a)mpich.org> > Cc: Max R. Dechantsreiter <max(a)performancejones.com> > Subject: [mpich-discuss] MPICH Test Suite - limit number of processes > > I would like to limit the number of processes used by the > MPICH Test Suite, which I run by "make testing" after > building MPICH. > > I tried > > export MPITEST_PPNMAX=<n> # The maximum number of processes per node. > > but this had no apparent effect: some tests still ran > using more than 4 processes, when I replaced <n> by 4. > How could I restrict these tests to run with maximum 4 > processes, for example? > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] MPICH Test Suite - limit number of processes
by Gregor Corbin 13 Jan '23

13 Jan '23

Dear Max, I have a script that recursively walks through the existing teslist files and filters them according to various criteria (name, regex of the testlist line, contents of associated source code). You can find it in my fork of mpich: https://github.com/gcorbin/mpich/ at /test/mpi/scorep/generate_filtered_testlist.py Feel free to use + adapt that if it is helpful for you. Cheers, Gregor On 12.01.23 17:29, Zhou, Hui via discuss wrote: > Unfortunately, the number of processes for each test is hardcoded in > the testlist files. There is no mechanism for filtering or dynamically > modifying them. You could either maintain a custom patch to modify the > testlist, or have a script applying a separate xfail list. An example > of such xfail list that we used in our internal testing is in > test/mpi/maint/jenkins/xfail.conf. If the custom patch solution won't > work in your situation, please open an github issue describing your > use case, and we can discuss to add a filtering option to the runtests > script. > > -- > Hui > ------------------------------------------------------------------------ > *From:* Max R. Dechantsreiter via discuss <discuss(a)mpich.org> > *Sent:* Thursday, January 12, 2023 9:58 AM > *To:* discuss(a)mpich.org <discuss(a)mpich.org> > *Cc:* Max R. Dechantsreiter <max(a)performancejones.com> > *Subject:* [mpich-discuss] MPICH Test Suite - limit number of processes > I would like to limit the number of processes used by the > MPICH Test Suite, which I run by "make testing" after > building MPICH. > > I tried > > export MPITEST_PPNMAX=<n> # The maximum number of processes per node. > > but this had no apparent effect: some tests still ran > using more than 4 processes, when I replaced <n> by 4. > How could I restrict these tests to run with maximum 4 > processes, for example? > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > > _______________________________________________ > discuss mailing listdiscuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] MPICH Test Suite - limit number of processes
by Zhou, Hui 12 Jan '23

12 Jan '23

Unfortunately, the number of processes for each test is hardcoded in the testlist files. There is no mechanism for filtering or dynamically modifying them. You could either maintain a custom patch to modify the testlist, or have a script applying a separate xfail list. An example of such xfail list that we used in our internal testing is in test/mpi/maint/jenkins/xfail.conf. If the custom patch solution won't work in your situation, please open an github issue describing your use case, and we can discuss to add a filtering option to the runtests script. -- Hui ________________________________ From: Max R. Dechantsreiter via discuss <discuss(a)mpich.org> Sent: Thursday, January 12, 2023 9:58 AM To: discuss(a)mpich.org <discuss(a)mpich.org> Cc: Max R. Dechantsreiter <max(a)performancejones.com> Subject: [mpich-discuss] MPICH Test Suite - limit number of processes I would like to limit the number of processes used by the MPICH Test Suite, which I run by "make testing" after building MPICH. I tried export MPITEST_PPNMAX=<n> # The maximum number of processes per node. but this had no apparent effect: some tests still ran using more than 4 processes, when I replaced <n> by 4. How could I restrict these tests to run with maximum 4 processes, for example? _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

MPICH Test Suite - limit number of processes
by Max R. Dechantsreiter 12 Jan '23

12 Jan '23

I would like to limit the number of processes used by the MPICH Test Suite, which I run by "make testing" after building MPICH. I tried export MPITEST_PPNMAX=<n> # The maximum number of processes per node. but this had no apparent effect: some tests still ran using more than 4 processes, when I replaced <n> by 4. How could I restrict these tests to run with maximum 4 processes, for example?

1 0

Slides from SC22 MPICH BoF posted
by Raffenetti, Ken 15 Dec '22

15 Dec '22

Hi all, With permission from the presenters, we have posted slides from the SC22 MPICH BoF to the MPICH Github wiki. https://github.com/pmodels/mpich/wiki/2022-Supercomputing-BoF#presentations Thanks to all the presenters and the MPICH community for participating! Cheers, The MPICH team

1 0

Re: [mpich-discuss] wrong number of processes on host
by Zhou, Hui 17 Nov '22

17 Nov '22

Bruno, No worries. We always like to hear use cases. We are glad that you found your bug. -- Hui ________________________________ From: bruno.guerraz(a)orange.com <bruno.guerraz(a)orange.com> Sent: Thursday, November 17, 2022 9:41 AM To: Zhou, Hui <zhouh(a)anl.gov>; discuss(a)mpich.org <discuss(a)mpich.org> Subject: Re: [mpich-discuss] wrong number of processes on host I am verry sorry, it is not a mpich bug, the bug is in my app Next time, I'll wait a couple of days before bothering you... On 17/11/2022 16:32, Zhou, Hui wrote: Bruno, No, I was looking for the earlier part of the log where mpiexec launches proxies. You may cut out the dump of environment variables if those are sensitive. -- Hui ________________________________ From: bruno.guerraz(a)orange.com<mailto:[email protected]> <bruno.guerraz(a)orange.com><mailto:[email protected]> Sent: Thursday, November 17, 2022 8:35 AM To: Zhou, Hui <zhouh(a)anl.gov><mailto:[email protected]>; discuss(a)mpich.org<mailto:[email protected]> <discuss(a)mpich.org><mailto:[email protected]> Subject: Re: [mpich-discuss] wrong number of processes on host Is this log OK for you ? [pgid: 0] got PMI command: cmd=barrier_in [pgid: 0] got PMI command: cmd=put sharedFilename[0]=/dev/shm/mpich_shar_tmpAOPuur [pgid: 0] got PMI command: cmd=barrier_in [pgid: 0] got PMI command: cmd=put sharedFilename[4]=/dev/shm/mpich_shar_tmp76zj4w [pgid: 0] got PMI command: cmd=barrier_in PMI response to fd 13 pid 4: cmd=keyval_cache sharedFilename[2]=/dev/shm/mpich_shar_tmppFa2Wf sharedFilename[0]=/dev/shm/mpich_shar_tmpAOPuur sharedFilename[4]=/dev/shm/mpich_shar_tmp76zj4w PMI response to fd 12 pid 4: cmd=keyval_cache sharedFilename[2]=/dev/shm/mpich_shar_tmppFa2Wf sharedFilename[0]=/dev/shm/mpich_shar_tmpAOPuur sharedFilename[4]=/dev/shm/mpich_shar_tmp76zj4w PMI response to fd 14 pid 4: cmd=keyval_cache sharedFilename[2]=/dev/shm/mpich_shar_tmppFa2Wf sharedFilename[0]=/dev/shm/mpich_shar_tmpAOPuur sharedFilename[4]=/dev/shm/mpich_shar_tmp76zj4w PMI response to fd 13 pid 4: cmd=barrier_out PMI response to fd 12 pid 4: cmd=barrier_out PMI response to fd 14 pid 4: cmd=barrier_out [pgid: 0] got PMI command: cmd=put P2-businesscard=description#l-neobi-1$port#39551$ifname#10.193.21.24$ P3-businesscard=description#l-neobi-1$port#49083$ifname#10.193.21.24$ [pgid: 0] got PMI command: cmd=barrier_in [pgid: 0] got PMI command: cmd=put P0-businesscard=description#l-neobi-4$port#57647$ifname#10.193.21.65$ P1-businesscard=description#l-neobi-4$port#46287$ifname#10.193.21.65$ [pgid: 0] got PMI command: cmd=barrier_in [pgid: 0] got PMI command: cmd=put P4-businesscard=description#l-neobi-3$port#48597$ifname#10.193.21.26$ P5-businesscard=description#l-neobi-3$port#47573$ifname#10.193.21.26$ [pgid: 0] got PMI command: cmd=barrier_in PMI response to fd 13 pid 5: cmd=keyval_cache P2-businesscard=description#l-neobi-1$port#39551$ifname#10.193.21.24$ P3-businesscard=description#l-neobi-1$port#49083$ifname#10.193.21.24$ P0-businesscard=description#l-neobi-4$port#57647$ifname#10.193.21.65$ P1-businesscard=description#l-neobi-4$port#46287$ifname#10.193.21.65$ P4-businesscard=description#l-neobi-3$port#48597$ifname#10.193.21.26$ P5-businesscard=description#l-neobi-3$port#47573$ifname#10.193.21.26$ PMI response to fd 12 pid 5: cmd=keyval_cache P2-businesscard=description#l-neobi-1$port#39551$ifname#10.193.21.24$ P3-businesscard=description#l-neobi-1$port#49083$ifname#10.193.21.24$ P0-businesscard=description#l-neobi-4$port#57647$ifname#10.193.21.65$ P1-businesscard=description#l-neobi-4$port#46287$ifname#10.193.21.65$ P4-businesscard=description#l-neobi-3$port#48597$ifname#10.193.21.26$ P5-businesscard=description#l-neobi-3$port#47573$ifname#10.193.21.26$ PMI response to fd 14 pid 5: cmd=keyval_cache P2-businesscard=description#l-neobi-1$port#39551$ifname#10.193.21.24$ P3-businesscard=description#l-neobi-1$port#49083$ifname#10.193.21.24$ P0-businesscard=description#l-neobi-4$port#57647$ifname#10.193.21.65$ P1-businesscard=description#l-neobi-4$port#46287$ifname#10.193.21.65$ P4-businesscard=description#l-neobi-3$port#48597$ifname#10.193.21.26$ P5-businesscard=description#l-neobi-3$port#47573$ifname#10.193.21.26$ PMI response to fd 13 pid 5: cmd=barrier_out PMI response to fd 12 pid 5: cmd=barrier_out PMI response to fd 14 pid 5: cmd=barrier_out On 17/11/2022 15:30, Zhou, Hui wrote: That is strange. Try pass -v option (i.e. mpiexec -v ...) to obtain a console log when that happens. -- Hui ________________________________ From: bruno via discuss <discuss(a)mpich.org><mailto:[email protected]> Sent: Thursday, November 17, 2022 8:11 AM To: discuss(a)mpich.org<mailto:[email protected]> <discuss(a)mpich.org><mailto:[email protected]> Cc: bruno.guerraz(a)orange.com<mailto:[email protected]> <bruno.guerraz(a)orange.com><mailto:[email protected]> Subject: [mpich-discuss] wrong number of processes on host Hi, I am using mpich on a hadoop cluster with yarn. It is not a smooth integration but it is working. Flowing an old post, I am using the manual launcher and the option -disable-hostname-propagation (https://lists.mpich.org/mailman/htdig/devel/2016-July/000717.html) The command line to launch by binary is : mpiexec -launcher manual -disable-hostname-propagation -n 6 -f ./hosts /path/to/my/bin And the file hosts contains host1:2 host2:2 host3:2 I expect to have 2 processes on each host but most of the time it launches 3 processes on host1, 1 on host2 and 2 on host3 Any ideas? Bruno _________________________________________________________________________________________________________________________ Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration, Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci. This message and its attachments may contain confidential or privileged information that may be protected by law; they should not be distributed, used or copied without authorisation. If you have received this email in error, please notify the sender and delete this message and its attachments. As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified. Thank you. _______________________________________________ discuss mailing list discuss(a)mpich.org<mailto:[email protected]> To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss _________________________________________________________________________________________________________________________ Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration, Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci. This message and its attachments may contain confidential or privileged information that may be protected by law; they should not be distributed, used or copied without authorisation. If you have received this email in error, please notify the sender and delete this message and its attachments. As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified. Thank you. _________________________________________________________________________________________________________________________ Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration, Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci. This message and its attachments may contain confidential or privileged information that may be protected by law; they should not be distributed, used or copied without authorisation. If you have received this email in error, please notify the sender and delete this message and its attachments. As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified. Thank you.

1 0

Re: [mpich-discuss] wrong number of processes on host
by bruno.guerraz＠orange.com 17 Nov '22

17 Nov '22

I am verry sorry, it is not a mpich bug, the bug is in my app Next time, I'll wait a couple of days before bothering you... On 17/11/2022 16:32, Zhou, Hui wrote: > Bruno, > > No, I was looking for the earlier part of the log where mpiexec > launches proxies. You may cut out the dump of environment variables if > those are sensitive. > > -- > Hui > ------------------------------------------------------------------------ > *From:* bruno.guerraz(a)orange.com <bruno.guerraz(a)orange.com> > *Sent:* Thursday, November 17, 2022 8:35 AM > *To:* Zhou, Hui <zhouh(a)anl.gov>; discuss(a)mpich.org <discuss(a)mpich.org> > *Subject:* Re: [mpich-discuss] wrong number of processes on host > Is this log OK for you ? > > [pgid: 0] got PMI command: cmd=barrier_in > [pgid: 0] got PMI command: cmd=put > sharedFilename[0]=/dev/shm/mpich_shar_tmpAOPuur > [pgid: 0] got PMI command: cmd=barrier_in > [pgid: 0] got PMI command: cmd=put > sharedFilename[4]=/dev/shm/mpich_shar_tmp76zj4w > [pgid: 0] got PMI command: cmd=barrier_in > PMI response to fd 13 pid 4: cmd=keyval_cache > sharedFilename[2]=/dev/shm/mpich_shar_tmppFa2Wf > sharedFilename[0]=/dev/shm/mpich_shar_tmpAOPuur > sharedFilename[4]=/dev/shm/mpich_shar_tmp76zj4w > PMI response to fd 12 pid 4: cmd=keyval_cache > sharedFilename[2]=/dev/shm/mpich_shar_tmppFa2Wf > sharedFilename[0]=/dev/shm/mpich_shar_tmpAOPuur > sharedFilename[4]=/dev/shm/mpich_shar_tmp76zj4w > PMI response to fd 14 pid 4: cmd=keyval_cache > sharedFilename[2]=/dev/shm/mpich_shar_tmppFa2Wf > sharedFilename[0]=/dev/shm/mpich_shar_tmpAOPuur > sharedFilename[4]=/dev/shm/mpich_shar_tmp76zj4w > PMI response to fd 13 pid 4: cmd=barrier_out > PMI response to fd 12 pid 4: cmd=barrier_out > PMI response to fd 14 pid 4: cmd=barrier_out > [pgid: 0] got PMI command: cmd=put > P2-businesscard=description#l-neobi-1$port#39551$ifname#10.193.21.24$ > P3-businesscard=description#l-neobi-1$port#49083$ifname#10.193.21.24$ > [pgid: 0] got PMI command: cmd=barrier_in > [pgid: 0] got PMI command: cmd=put > P0-businesscard=description#l-neobi-4$port#57647$ifname#10.193.21.65$ > P1-businesscard=description#l-neobi-4$port#46287$ifname#10.193.21.65$ > [pgid: 0] got PMI command: cmd=barrier_in > [pgid: 0] got PMI command: cmd=put > P4-businesscard=description#l-neobi-3$port#48597$ifname#10.193.21.26$ > P5-businesscard=description#l-neobi-3$port#47573$ifname#10.193.21.26$ > [pgid: 0] got PMI command: cmd=barrier_in > PMI response to fd 13 pid 5: cmd=keyval_cache > P2-businesscard=description#l-neobi-1$port#39551$ifname#10.193.21.24$ > P3-businesscard=description#l-neobi-1$port#49083$ifname#10.193.21.24$ > P0-businesscard=description#l-neobi-4$port#57647$ifname#10.193.21.65$ > P1-businesscard=description#l-neobi-4$port#46287$ifname#10.193.21.65$ > P4-businesscard=description#l-neobi-3$port#48597$ifname#10.193.21.26$ > P5-businesscard=description#l-neobi-3$port#47573$ifname#10.193.21.26$ > PMI response to fd 12 pid 5: cmd=keyval_cache > P2-businesscard=description#l-neobi-1$port#39551$ifname#10.193.21.24$ > P3-businesscard=description#l-neobi-1$port#49083$ifname#10.193.21.24$ > P0-businesscard=description#l-neobi-4$port#57647$ifname#10.193.21.65$ > P1-businesscard=description#l-neobi-4$port#46287$ifname#10.193.21.65$ > P4-businesscard=description#l-neobi-3$port#48597$ifname#10.193.21.26$ > P5-businesscard=description#l-neobi-3$port#47573$ifname#10.193.21.26$ > PMI response to fd 14 pid 5: cmd=keyval_cache > P2-businesscard=description#l-neobi-1$port#39551$ifname#10.193.21.24$ > P3-businesscard=description#l-neobi-1$port#49083$ifname#10.193.21.24$ > P0-businesscard=description#l-neobi-4$port#57647$ifname#10.193.21.65$ > P1-businesscard=description#l-neobi-4$port#46287$ifname#10.193.21.65$ > P4-businesscard=description#l-neobi-3$port#48597$ifname#10.193.21.26$ > P5-businesscard=description#l-neobi-3$port#47573$ifname#10.193.21.26$ > PMI response to fd 13 pid 5: cmd=barrier_out > PMI response to fd 12 pid 5: cmd=barrier_out > PMI response to fd 14 pid 5: cmd=barrier_out > > On 17/11/2022 15:30, Zhou, Hui wrote: >> That is strange. Try pass |-v| option (i.e. |mpiexec -v| ...) to >> obtain a console log when that happens. >> >> -- >> Hui >> ------------------------------------------------------------------------ >> *From:* bruno via discuss <discuss(a)mpich.org> <mailto:[email protected]> >> *Sent:* Thursday, November 17, 2022 8:11 AM >> *To:* discuss(a)mpich.org <mailto:[email protected]> >> <discuss(a)mpich.org> <mailto:[email protected]> >> *Cc:* bruno.guerraz(a)orange.com <mailto:[email protected]> >> <bruno.guerraz(a)orange.com> <mailto:[email protected]> >> *Subject:* [mpich-discuss] wrong number of processes on host >> Hi, I am using mpich on a hadoop cluster with yarn. It is not a smooth >> integration but it is working. >> Flowing an old post, I am using the manual launcher and the option >> -disable-hostname-propagation >> (https://lists.mpich.org/mailman/htdig/devel/2016-July/000717.html >> <https://lists.mpich.org/mailman/htdig/devel/2016-July/000717.html>) >> >> The command line to launch by binary is : >> >> mpiexec -launcher manual -disable-hostname-propagation -n 6 -f ./hosts >> /path/to/my/bin >> >> And the file hosts contains >> >> host1:2 >> host2:2 >> host3:2 >> >> I expect to have 2 processes on each host but most of the time it >> launches 3 processes on host1, 1 on host2 and 2 on host3 >> Any ideas? >> >> Bruno >> >> _________________________________________________________________________________________________________________________ >> >> Ce message et ses pieces jointes peuvent contenir des informations >> confidentielles ou privilegiees et ne doivent donc >> pas etre diffuses, exploites ou copies sans autorisation. Si vous >> avez recu ce message par erreur, veuillez le signaler >> a l'expediteur et le detruire ainsi que les pieces jointes. Les >> messages electroniques etant susceptibles d'alteration, >> Orange decline toute responsabilite si ce message a ete altere, >> deforme ou falsifie. Merci. >> >> This message and its attachments may contain confidential or >> privileged information that may be protected by law; >> they should not be distributed, used or copied without authorisation. >> If you have received this email in error, please notify the sender >> and delete this message and its attachments. >> As emails may be altered, Orange is not liable for messages that have >> been modified, changed or falsified. >> Thank you. >> >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org <mailto:[email protected]> >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss >> <https://lists.mpich.org/mailman/listinfo/discuss> > > _________________________________________________________________________________________________________________________ > > Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc > pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler > a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration, > Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci. > > This message and its attachments may contain confidential or privileged information that may be protected by law; > they should not be distributed, used or copied without authorisation. > If you have received this email in error, please notify the sender and delete this message and its attachments. > As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified. > Thank you. _________________________________________________________________________________________________________________________ Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration, Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci. This message and its attachments may contain confidential or privileged information that may be protected by law; they should not be distributed, used or copied without authorisation. If you have received this email in error, please notify the sender and delete this message and its attachments. As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified. Thank you.

1 0

Re: [mpich-discuss] wrong number of processes on host
by Zhou, Hui 17 Nov '22

17 Nov '22

Bruno, No, I was looking for the earlier part of the log where mpiexec launches proxies. You may cut out the dump of environment variables if those are sensitive. -- Hui ________________________________ From: bruno.guerraz(a)orange.com <bruno.guerraz(a)orange.com> Sent: Thursday, November 17, 2022 8:35 AM To: Zhou, Hui <zhouh(a)anl.gov>; discuss(a)mpich.org <discuss(a)mpich.org> Subject: Re: [mpich-discuss] wrong number of processes on host Is this log OK for you ? [pgid: 0] got PMI command: cmd=barrier_in [pgid: 0] got PMI command: cmd=put sharedFilename[0]=/dev/shm/mpich_shar_tmpAOPuur [pgid: 0] got PMI command: cmd=barrier_in [pgid: 0] got PMI command: cmd=put sharedFilename[4]=/dev/shm/mpich_shar_tmp76zj4w [pgid: 0] got PMI command: cmd=barrier_in PMI response to fd 13 pid 4: cmd=keyval_cache sharedFilename[2]=/dev/shm/mpich_shar_tmppFa2Wf sharedFilename[0]=/dev/shm/mpich_shar_tmpAOPuur sharedFilename[4]=/dev/shm/mpich_shar_tmp76zj4w PMI response to fd 12 pid 4: cmd=keyval_cache sharedFilename[2]=/dev/shm/mpich_shar_tmppFa2Wf sharedFilename[0]=/dev/shm/mpich_shar_tmpAOPuur sharedFilename[4]=/dev/shm/mpich_shar_tmp76zj4w PMI response to fd 14 pid 4: cmd=keyval_cache sharedFilename[2]=/dev/shm/mpich_shar_tmppFa2Wf sharedFilename[0]=/dev/shm/mpich_shar_tmpAOPuur sharedFilename[4]=/dev/shm/mpich_shar_tmp76zj4w PMI response to fd 13 pid 4: cmd=barrier_out PMI response to fd 12 pid 4: cmd=barrier_out PMI response to fd 14 pid 4: cmd=barrier_out [pgid: 0] got PMI command: cmd=put P2-businesscard=description#l-neobi-1$port#39551$ifname#10.193.21.24$ P3-businesscard=description#l-neobi-1$port#49083$ifname#10.193.21.24$ [pgid: 0] got PMI command: cmd=barrier_in [pgid: 0] got PMI command: cmd=put P0-businesscard=description#l-neobi-4$port#57647$ifname#10.193.21.65$ P1-businesscard=description#l-neobi-4$port#46287$ifname#10.193.21.65$ [pgid: 0] got PMI command: cmd=barrier_in [pgid: 0] got PMI command: cmd=put P4-businesscard=description#l-neobi-3$port#48597$ifname#10.193.21.26$ P5-businesscard=description#l-neobi-3$port#47573$ifname#10.193.21.26$ [pgid: 0] got PMI command: cmd=barrier_in PMI response to fd 13 pid 5: cmd=keyval_cache P2-businesscard=description#l-neobi-1$port#39551$ifname#10.193.21.24$ P3-businesscard=description#l-neobi-1$port#49083$ifname#10.193.21.24$ P0-businesscard=description#l-neobi-4$port#57647$ifname#10.193.21.65$ P1-businesscard=description#l-neobi-4$port#46287$ifname#10.193.21.65$ P4-businesscard=description#l-neobi-3$port#48597$ifname#10.193.21.26$ P5-businesscard=description#l-neobi-3$port#47573$ifname#10.193.21.26$ PMI response to fd 12 pid 5: cmd=keyval_cache P2-businesscard=description#l-neobi-1$port#39551$ifname#10.193.21.24$ P3-businesscard=description#l-neobi-1$port#49083$ifname#10.193.21.24$ P0-businesscard=description#l-neobi-4$port#57647$ifname#10.193.21.65$ P1-businesscard=description#l-neobi-4$port#46287$ifname#10.193.21.65$ P4-businesscard=description#l-neobi-3$port#48597$ifname#10.193.21.26$ P5-businesscard=description#l-neobi-3$port#47573$ifname#10.193.21.26$ PMI response to fd 14 pid 5: cmd=keyval_cache P2-businesscard=description#l-neobi-1$port#39551$ifname#10.193.21.24$ P3-businesscard=description#l-neobi-1$port#49083$ifname#10.193.21.24$ P0-businesscard=description#l-neobi-4$port#57647$ifname#10.193.21.65$ P1-businesscard=description#l-neobi-4$port#46287$ifname#10.193.21.65$ P4-businesscard=description#l-neobi-3$port#48597$ifname#10.193.21.26$ P5-businesscard=description#l-neobi-3$port#47573$ifname#10.193.21.26$ PMI response to fd 13 pid 5: cmd=barrier_out PMI response to fd 12 pid 5: cmd=barrier_out PMI response to fd 14 pid 5: cmd=barrier_out On 17/11/2022 15:30, Zhou, Hui wrote: That is strange. Try pass -v option (i.e. mpiexec -v ...) to obtain a console log when that happens. -- Hui ________________________________ From: bruno via discuss <discuss(a)mpich.org><mailto:[email protected]> Sent: Thursday, November 17, 2022 8:11 AM To: discuss(a)mpich.org<mailto:[email protected]> <discuss(a)mpich.org><mailto:[email protected]> Cc: bruno.guerraz(a)orange.com<mailto:[email protected]> <bruno.guerraz(a)orange.com><mailto:[email protected]> Subject: [mpich-discuss] wrong number of processes on host Hi, I am using mpich on a hadoop cluster with yarn. It is not a smooth integration but it is working. Flowing an old post, I am using the manual launcher and the option -disable-hostname-propagation (https://lists.mpich.org/mailman/htdig/devel/2016-July/000717.html) The command line to launch by binary is : mpiexec -launcher manual -disable-hostname-propagation -n 6 -f ./hosts /path/to/my/bin And the file hosts contains host1:2 host2:2 host3:2 I expect to have 2 processes on each host but most of the time it launches 3 processes on host1, 1 on host2 and 2 on host3 Any ideas? Bruno _________________________________________________________________________________________________________________________ Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration, Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci. This message and its attachments may contain confidential or privileged information that may be protected by law; they should not be distributed, used or copied without authorisation. If you have received this email in error, please notify the sender and delete this message and its attachments. As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified. Thank you. _______________________________________________ discuss mailing list discuss(a)mpich.org<mailto:[email protected]> To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss _________________________________________________________________________________________________________________________ Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration, Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci. This message and its attachments may contain confidential or privileged information that may be protected by law; they should not be distributed, used or copied without authorisation. If you have received this email in error, please notify the sender and delete this message and its attachments. As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified. Thank you.

1 0

Re: [mpich-discuss] wrong number of processes on host
by bruno.guerraz＠orange.com 17 Nov '22

17 Nov '22

Is this log OK for you ? [pgid: 0] got PMI command: cmd=barrier_in [pgid: 0] got PMI command: cmd=put sharedFilename[0]=/dev/shm/mpich_shar_tmpAOPuur [pgid: 0] got PMI command: cmd=barrier_in [pgid: 0] got PMI command: cmd=put sharedFilename[4]=/dev/shm/mpich_shar_tmp76zj4w [pgid: 0] got PMI command: cmd=barrier_in PMI response to fd 13 pid 4: cmd=keyval_cache sharedFilename[2]=/dev/shm/mpich_shar_tmppFa2Wf sharedFilename[0]=/dev/shm/mpich_shar_tmpAOPuur sharedFilename[4]=/dev/shm/mpich_shar_tmp76zj4w PMI response to fd 12 pid 4: cmd=keyval_cache sharedFilename[2]=/dev/shm/mpich_shar_tmppFa2Wf sharedFilename[0]=/dev/shm/mpich_shar_tmpAOPuur sharedFilename[4]=/dev/shm/mpich_shar_tmp76zj4w PMI response to fd 14 pid 4: cmd=keyval_cache sharedFilename[2]=/dev/shm/mpich_shar_tmppFa2Wf sharedFilename[0]=/dev/shm/mpich_shar_tmpAOPuur sharedFilename[4]=/dev/shm/mpich_shar_tmp76zj4w PMI response to fd 13 pid 4: cmd=barrier_out PMI response to fd 12 pid 4: cmd=barrier_out PMI response to fd 14 pid 4: cmd=barrier_out [pgid: 0] got PMI command: cmd=put P2-businesscard=description#l-neobi-1$port#39551$ifname#10.193.21.24$ P3-businesscard=description#l-neobi-1$port#49083$ifname#10.193.21.24$ [pgid: 0] got PMI command: cmd=barrier_in [pgid: 0] got PMI command: cmd=put P0-businesscard=description#l-neobi-4$port#57647$ifname#10.193.21.65$ P1-businesscard=description#l-neobi-4$port#46287$ifname#10.193.21.65$ [pgid: 0] got PMI command: cmd=barrier_in [pgid: 0] got PMI command: cmd=put P4-businesscard=description#l-neobi-3$port#48597$ifname#10.193.21.26$ P5-businesscard=description#l-neobi-3$port#47573$ifname#10.193.21.26$ [pgid: 0] got PMI command: cmd=barrier_in PMI response to fd 13 pid 5: cmd=keyval_cache P2-businesscard=description#l-neobi-1$port#39551$ifname#10.193.21.24$ P3-businesscard=description#l-neobi-1$port#49083$ifname#10.193.21.24$ P0-businesscard=description#l-neobi-4$port#57647$ifname#10.193.21.65$ P1-businesscard=description#l-neobi-4$port#46287$ifname#10.193.21.65$ P4-businesscard=description#l-neobi-3$port#48597$ifname#10.193.21.26$ P5-businesscard=description#l-neobi-3$port#47573$ifname#10.193.21.26$ PMI response to fd 12 pid 5: cmd=keyval_cache P2-businesscard=description#l-neobi-1$port#39551$ifname#10.193.21.24$ P3-businesscard=description#l-neobi-1$port#49083$ifname#10.193.21.24$ P0-businesscard=description#l-neobi-4$port#57647$ifname#10.193.21.65$ P1-businesscard=description#l-neobi-4$port#46287$ifname#10.193.21.65$ P4-businesscard=description#l-neobi-3$port#48597$ifname#10.193.21.26$ P5-businesscard=description#l-neobi-3$port#47573$ifname#10.193.21.26$ PMI response to fd 14 pid 5: cmd=keyval_cache P2-businesscard=description#l-neobi-1$port#39551$ifname#10.193.21.24$ P3-businesscard=description#l-neobi-1$port#49083$ifname#10.193.21.24$ P0-businesscard=description#l-neobi-4$port#57647$ifname#10.193.21.65$ P1-businesscard=description#l-neobi-4$port#46287$ifname#10.193.21.65$ P4-businesscard=description#l-neobi-3$port#48597$ifname#10.193.21.26$ P5-businesscard=description#l-neobi-3$port#47573$ifname#10.193.21.26$ PMI response to fd 13 pid 5: cmd=barrier_out PMI response to fd 12 pid 5: cmd=barrier_out PMI response to fd 14 pid 5: cmd=barrier_out On 17/11/2022 15:30, Zhou, Hui wrote: > That is strange. Try pass |-v| option (i.e. |mpiexec -v| ...) to > obtain a console log when that happens. > > -- > Hui > ------------------------------------------------------------------------ > *From:* bruno via discuss <discuss(a)mpich.org> > *Sent:* Thursday, November 17, 2022 8:11 AM > *To:* discuss(a)mpich.org <discuss(a)mpich.org> > *Cc:* bruno.guerraz(a)orange.com <bruno.guerraz(a)orange.com> > *Subject:* [mpich-discuss] wrong number of processes on host > Hi, I am using mpich on a hadoop cluster with yarn. It is not a smooth > integration but it is working. > Flowing an old post, I am using the manual launcher and the option > -disable-hostname-propagation > (https://lists.mpich.org/mailman/htdig/devel/2016-July/000717.html) > > The command line to launch by binary is : > > mpiexec -launcher manual -disable-hostname-propagation -n 6 -f ./hosts > /path/to/my/bin > > And the file hosts contains > > host1:2 > host2:2 > host3:2 > > I expect to have 2 processes on each host but most of the time it > launches 3 processes on host1, 1 on host2 and 2 on host3 > Any ideas? > > Bruno > > _________________________________________________________________________________________________________________________ > > Ce message et ses pieces jointes peuvent contenir des informations > confidentielles ou privilegiees et ne doivent donc > pas etre diffuses, exploites ou copies sans autorisation. Si vous avez > recu ce message par erreur, veuillez le signaler > a l'expediteur et le detruire ainsi que les pieces jointes. Les > messages electroniques etant susceptibles d'alteration, > Orange decline toute responsabilite si ce message a ete altere, > deforme ou falsifie. Merci. > > This message and its attachments may contain confidential or > privileged information that may be protected by law; > they should not be distributed, used or copied without authorisation. > If you have received this email in error, please notify the sender and > delete this message and its attachments. > As emails may be altered, Orange is not liable for messages that have > been modified, changed or falsified. > Thank you. > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss _________________________________________________________________________________________________________________________ Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration, Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci. This message and its attachments may contain confidential or privileged information that may be protected by law; they should not be distributed, used or copied without authorisation. If you have received this email in error, please notify the sender and delete this message and its attachments. As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified. Thank you.

1 0

Re: [mpich-discuss] wrong number of processes on host
by Zhou, Hui 17 Nov '22

17 Nov '22

That is strange. Try pass -v option (i.e. mpiexec -v ...) to obtain a console log when that happens. -- Hui ________________________________ From: bruno via discuss <discuss(a)mpich.org> Sent: Thursday, November 17, 2022 8:11 AM To: discuss(a)mpich.org <discuss(a)mpich.org> Cc: bruno.guerraz(a)orange.com <bruno.guerraz(a)orange.com> Subject: [mpich-discuss] wrong number of processes on host Hi, I am using mpich on a hadoop cluster with yarn. It is not a smooth integration but it is working. Flowing an old post, I am using the manual launcher and the option -disable-hostname-propagation (https://lists.mpich.org/mailman/htdig/devel/2016-July/000717.html) The command line to launch by binary is : mpiexec -launcher manual -disable-hostname-propagation -n 6 -f ./hosts /path/to/my/bin And the file hosts contains host1:2 host2:2 host3:2 I expect to have 2 processes on each host but most of the time it launches 3 processes on host1, 1 on host2 and 2 on host3 Any ideas? Bruno _________________________________________________________________________________________________________________________ Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration, Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci. This message and its attachments may contain confidential or privileged information that may be protected by law; they should not be distributed, used or copied without authorisation. If you have received this email in error, please notify the sender and delete this message and its attachments. As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified. Thank you. _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0