January 2023 - discuss - mailman3.cels.anl.gov

Re: [mpich-discuss] MPICH Test Suite - limit number of processes
by Max R. Dechantsreiter 16 Jan '23

16 Jan '23

Hi Joachim, Thank you, I will try that overnight. This setting might be useful in practical situations. Interesting that some of the tests require 33 processes: that is high even for most modern processors, so I presume the idea was to run the suite on multiple nodes. Best regards, Max --- On Sun, Jan 15, 2023 at 09:19:58PM +0000, Jenke, Joachim wrote: > For our unit testing (not mpich specific) I found it useful to set the following environmental variable to reduce the time wasted in busy waiting while we run our tests massively oversubscribed: > export MPIR_CVAR_POLLS_BEFORE_YIELD=1 > > - Joachim > ________________________________ > From: Max R. Dechantsreiter via discuss <discuss(a)mpich.org> > Sent: Sunday, January 15, 2023 3:09:19 PM > To: Zhou, Hui <zhouh(a)anl.gov> > Cc: Max R. Dechantsreiter <max(a)performancejones.com>; discuss(a)mpich.org <discuss(a)mpich.org> > Subject: Re: [mpich-discuss] MPICH Test Suite - limit number of processes > > Thank you for your suggestion. > > However, I did not see how to carry it through; perhaps one > day I will comb through the User's Guide. > > Instead I ran 'make testing' to completion on four cores: > elapsed time was 2h36m53s on Intel(R) Xeon(R) CPU E5-2620 v4 > (2.10GHz); all but two tests passed, with the two that did > not pass simply timing out (after 180s). > > Then I ran it again after modifying "runtests" thus: > > sed -i 's/^$ppnMax = -1;/$ppnMax = 4;/' $RUNTESTS > sed -i 's/^$np_max = -1;/$np_max = 4;/' $RUNTESTS > > where RUNTESTS=~test/mpi/runtests. (Alternatively these > variables could be modified in $RUNTESTS.in before prior to > building MPICH.) > > Changing those variables limited the maximum number of > processes to 4 as desired. The salient issues were the > warnings 'Number of processors must divide 10' that I > presume could be avoided by setting $np_max=2 or 5; the > number of tests affected was relatively small: 13 of 2524 > tests failed, with no test timing out. Elapsed time was > 1h04m28s - so the savings in time was significant but not > spectacular, suggesting that relatively few tests in the > original run oversubscribed the cores available in my VPS. > > > On Thu, Jan 12, 2023 at 04:29:07PM +0000, Zhou, Hui wrote: > > Unfortunately, the number of processes for each test is hardcoded in the testlist files. There is no mechanism for filtering or dynamically modifying them. You could either maintain a custom patch to modify the testlist, or have a script applying a separate xfail list. An example of such xfail list that we used in our internal testing is in test/mpi/maint/jenkins/xfail.conf. If the custom patch solution won't work in your situation, please open an github issue describing your use case, and we can discuss to add a filtering option to the runtests script. > > > > -- > > Hui > > ________________________________ > > From: Max R. Dechantsreiter via discuss <discuss(a)mpich.org> > > Sent: Thursday, January 12, 2023 9:58 AM > > To: discuss(a)mpich.org <discuss(a)mpich.org> > > Cc: Max R. Dechantsreiter <max(a)performancejones.com> > > Subject: [mpich-discuss] MPICH Test Suite - limit number of processes > > > > I would like to limit the number of processes used by the > > MPICH Test Suite, which I run by "make testing" after > > building MPICH. > > > > I tried > > > > export MPITEST_PPNMAX=<n> # The maximum number of processes per node. > > > > but this had no apparent effect: some tests still ran > > using more than 4 processes, when I replaced <n> by 4. > > How could I restrict these tests to run with maximum 4 > > processes, for example? > > > > _______________________________________________ > > discuss mailing list discuss(a)mpich.org > > To manage subscription options or unsubscribe: > > https://lists.mpich.org/mailman/listinfo/discuss > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] MPICH Test Suite - limit number of processes
by Max R. Dechantsreiter 15 Jan '23

15 Jan '23

Thank you for your suggestion. However, I did not see how to carry it through; perhaps one day I will comb through the User's Guide. Instead I ran 'make testing' to completion on four cores: elapsed time was 2h36m53s on Intel(R) Xeon(R) CPU E5-2620 v4 (2.10GHz); all but two tests passed, with the two that did not pass simply timing out (after 180s). Then I ran it again after modifying "runtests" thus: sed -i 's/^$ppnMax = -1;/$ppnMax = 4;/' $RUNTESTS sed -i 's/^$np_max = -1;/$np_max = 4;/' $RUNTESTS where RUNTESTS=~test/mpi/runtests. (Alternatively these variables could be modified in $RUNTESTS.in before prior to building MPICH.) Changing those variables limited the maximum number of processes to 4 as desired. The salient issues were the warnings 'Number of processors must divide 10' that I presume could be avoided by setting $np_max=2 or 5; the number of tests affected was relatively small: 13 of 2524 tests failed, with no test timing out. Elapsed time was 1h04m28s - so the savings in time was significant but not spectacular, suggesting that relatively few tests in the original run oversubscribed the cores available in my VPS. On Thu, Jan 12, 2023 at 04:29:07PM +0000, Zhou, Hui wrote: > Unfortunately, the number of processes for each test is hardcoded in the testlist files. There is no mechanism for filtering or dynamically modifying them. You could either maintain a custom patch to modify the testlist, or have a script applying a separate xfail list. An example of such xfail list that we used in our internal testing is in test/mpi/maint/jenkins/xfail.conf. If the custom patch solution won't work in your situation, please open an github issue describing your use case, and we can discuss to add a filtering option to the runtests script. > > -- > Hui > ________________________________ > From: Max R. Dechantsreiter via discuss <discuss(a)mpich.org> > Sent: Thursday, January 12, 2023 9:58 AM > To: discuss(a)mpich.org <discuss(a)mpich.org> > Cc: Max R. Dechantsreiter <max(a)performancejones.com> > Subject: [mpich-discuss] MPICH Test Suite - limit number of processes > > I would like to limit the number of processes used by the > MPICH Test Suite, which I run by "make testing" after > building MPICH. > > I tried > > export MPITEST_PPNMAX=<n> # The maximum number of processes per node. > > but this had no apparent effect: some tests still ran > using more than 4 processes, when I replaced <n> by 4. > How could I restrict these tests to run with maximum 4 > processes, for example? > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] MPICH Test Suite - limit number of processes
by Gregor Corbin 13 Jan '23

13 Jan '23

Dear Max, I have a script that recursively walks through the existing teslist files and filters them according to various criteria (name, regex of the testlist line, contents of associated source code). You can find it in my fork of mpich: https://github.com/gcorbin/mpich/ at /test/mpi/scorep/generate_filtered_testlist.py Feel free to use + adapt that if it is helpful for you. Cheers, Gregor On 12.01.23 17:29, Zhou, Hui via discuss wrote: > Unfortunately, the number of processes for each test is hardcoded in > the testlist files. There is no mechanism for filtering or dynamically > modifying them. You could either maintain a custom patch to modify the > testlist, or have a script applying a separate xfail list. An example > of such xfail list that we used in our internal testing is in > test/mpi/maint/jenkins/xfail.conf. If the custom patch solution won't > work in your situation, please open an github issue describing your > use case, and we can discuss to add a filtering option to the runtests > script. > > -- > Hui > ------------------------------------------------------------------------ > *From:* Max R. Dechantsreiter via discuss <discuss(a)mpich.org> > *Sent:* Thursday, January 12, 2023 9:58 AM > *To:* discuss(a)mpich.org <discuss(a)mpich.org> > *Cc:* Max R. Dechantsreiter <max(a)performancejones.com> > *Subject:* [mpich-discuss] MPICH Test Suite - limit number of processes > I would like to limit the number of processes used by the > MPICH Test Suite, which I run by "make testing" after > building MPICH. > > I tried > > export MPITEST_PPNMAX=<n> # The maximum number of processes per node. > > but this had no apparent effect: some tests still ran > using more than 4 processes, when I replaced <n> by 4. > How could I restrict these tests to run with maximum 4 > processes, for example? > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > > _______________________________________________ > discuss mailing listdiscuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] MPICH Test Suite - limit number of processes
by Zhou, Hui 12 Jan '23

12 Jan '23

Unfortunately, the number of processes for each test is hardcoded in the testlist files. There is no mechanism for filtering or dynamically modifying them. You could either maintain a custom patch to modify the testlist, or have a script applying a separate xfail list. An example of such xfail list that we used in our internal testing is in test/mpi/maint/jenkins/xfail.conf. If the custom patch solution won't work in your situation, please open an github issue describing your use case, and we can discuss to add a filtering option to the runtests script. -- Hui ________________________________ From: Max R. Dechantsreiter via discuss <discuss(a)mpich.org> Sent: Thursday, January 12, 2023 9:58 AM To: discuss(a)mpich.org <discuss(a)mpich.org> Cc: Max R. Dechantsreiter <max(a)performancejones.com> Subject: [mpich-discuss] MPICH Test Suite - limit number of processes I would like to limit the number of processes used by the MPICH Test Suite, which I run by "make testing" after building MPICH. I tried export MPITEST_PPNMAX=<n> # The maximum number of processes per node. but this had no apparent effect: some tests still ran using more than 4 processes, when I replaced <n> by 4. How could I restrict these tests to run with maximum 4 processes, for example? _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

MPICH Test Suite - limit number of processes
by Max R. Dechantsreiter 12 Jan '23

12 Jan '23

I would like to limit the number of processes used by the MPICH Test Suite, which I run by "make testing" after building MPICH. I tried export MPITEST_PPNMAX=<n> # The maximum number of processes per node. but this had no apparent effect: some tests still ran using more than 4 processes, when I replaced <n> by 4. How could I restrict these tests to run with maximum 4 processes, for example?

1 0