April 2018 - discuss - mailman3.cels.anl.gov

Infiniband
by Mayank Jain 27 Apr '18

27 Apr '18

Hello, I am new to MPICH and I have cluster which MPICH installed. I wanted to check if MPICH is using Infiniband or sockets for transmission. Is their any way I can check same? Thanks, Mayank _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Using MPICH and Hydra to manage apruns
by Justin M Wozniak 27 Apr '18

27 Apr '18

Hi I am trying to use an MPI program to manage an ensemble of apruns under Cobalt on Theta. I compiled an MPICH 3.2 for the login node. Then, in an interactive allocation, I do: $ hostname thetamom2 $ mpiexec -launcher fork -n 1 echo hi [mpiexec@thetamom2] HYDU_parse_hostfile (utils/args/args.c:319): unable to open host file: /var/tmp/cobalt.222331 [mpiexec@thetamom2] HYDT_bscd_cobalt_query_node_list (tools/bootstrap/external/cobalt_query_node_list.c:27): error parsing hostfile [mpiexec@thetamom2] HYDT_bsci_query_node_list (tools/bootstrap/src/bsci_query_node_list.c:19): RMK returned error while querying node list [mpiexec@thetamom2] main (ui/mpich/mpiexec.c:186): unable to query the RMK for a node list It looks like Hydra has detected that it is under Cobalt. How do I force it to ignore Cobalt? Ultimately, I plan to replace 'echo' with a shell script that does an aprun. Thanks -- Justin M Wozniak _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Debugging / Stopping on error
by Kenneth Raffenetti 26 Apr '18

26 Apr '18

Breaking at MPID_Abort is a good suggestion for now. There have been discussions to add environment variables to control the internal crash behavior better, e.g. call actual abort(), wait for debug attach, etc., but have not yet been implemented. Ken On 04/23/2018 02:25 AM, Joseph Schuchart wrote: > Florian, > > I just came across your question and it seems like you haven't received > an answer yet so I thought I throw in my 2 cents. I usually use a > combination of xterm and gdb for local debugging, i.e., > > mpirun -n 4 xterm -e gdb -ex r --args ./app -arg1 ... > > To break at MPI errors you have to add a breakpoint for the > (implementation specific) abort routine. IIRC in MPICH that is > MPID_Abort (it's been a while since I last used MPICH for debugging > though). > > HTH, > Joseph > > On 04/12/2018 10:05 AM, Florian Lindner wrote: >> Hello, >> >> I am using mpich on Arch on my own desktop computer. Unfortunatly, I >> don't have Totalview or DDT at hand. >> >> OpenMPI has a useful option to pause on an MPI error: >> >> mpirun --mca opal_abort_delay 600 -n 4 ... >> >> so that I can attach with a debugger and backtrace up into my app. Is >> there something like that for mpich? >> >> What other recommendation could you give for debugging MPI >> applications (not mpich itself), especially for MPI Ports. I >> have re-compiled with --enable-g=all. >> >> setenv MPICH_DBG FILE >> setenv MPICH_DBG_LEVEL VERBOSE >> >> works, but produces too detailed output, even when $MPICH_DBG_TYPICAL >> is defined, and also too far away from my >> application code. >> >> Thanks, >> Florian >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss >> > > _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Debugging / Stopping on error
by Joseph Schuchart 23 Apr '18

23 Apr '18

Florian, I just came across your question and it seems like you haven't received an answer yet so I thought I throw in my 2 cents. I usually use a combination of xterm and gdb for local debugging, i.e., mpirun -n 4 xterm -e gdb -ex r --args ./app -arg1 ... To break at MPI errors you have to add a breakpoint for the (implementation specific) abort routine. IIRC in MPICH that is MPID_Abort (it's been a while since I last used MPICH for debugging though). HTH, Joseph On 04/12/2018 10:05 AM, Florian Lindner wrote: > Hello, > > I am using mpich on Arch on my own desktop computer. Unfortunatly, I don't have Totalview or DDT at hand. > > OpenMPI has a useful option to pause on an MPI error: > > mpirun --mca opal_abort_delay 600 -n 4 ... > > so that I can attach with a debugger and backtrace up into my app. Is there something like that for mpich? > > What other recommendation could you give for debugging MPI applications (not mpich itself), especially for MPI Ports. I > have re-compiled with --enable-g=all. > > setenv MPICH_DBG FILE > setenv MPICH_DBG_LEVEL VERBOSE > > works, but produces too detailed output, even when $MPICH_DBG_TYPICAL is defined, and also too far away from my > application code. > > Thanks, > Florian > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > -- Dipl.-Inf. Joseph Schuchart High Performance Computing Center Stuttgart (HLRS) Nobelstr. 19 D-70569 Stuttgart Tel.: +49(0)711-68565890 Fax: +49(0)711-6856832 E-Mail: schuchart(a)hlrs.de _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] [petsc-dev] Is mpich/master:a8a2b30fd21 tested with Petsc?
by Min Si 19 Apr '18

19 Apr '18

Hi Junchao, This is a great idea. We will add large tag tests in our test suite ! Min On 2018/04/17 18:17, Junchao Zhang wrote: > Min, > I suggest MPICH add tests to play with the maximal MPI tag (through > attribute MPI_TAG_UB). > PETSc uses tags from the maximal and downwards. I guess MPICH tests > use small tags. That is why the bug only showed up with PETSc. > > --Junchao Zhang > > On Tue, Apr 17, 2018 at 3:58 PM, Min Si <msi(a)anl.gov > <mailto:[email protected]>> wrote: > > Hi all, > > Thanks for narrowing down the problem. I checked the MPICH code > and believe this is a bug in MPICH. I just created a PR to fix it: > https://github.com/pmodels/mpich/pull/3097 > <https://github.com/pmodels/mpich/pull/3097> > > It should be merged into MPICH master branch soon. > > Thanks, > Min > > > On 2018/04/17 14:10, Eric Chamberland wrote: > > Hi, > > are we talking about the "tag" passed to MPI_Isend for example? > > but does that mean there is something to change for any MPI > call which involves tags usage or is it only a PETSc "bad" tag > usage? > > thanks Satish for your finding! > > Eric > > On 16/04/18 11:31 PM, Satish Balay wrote: > > On Tue, 13 Mar 2018, Eric Chamberland wrote: > > Hi, > > each night we are testing mpich/master with our > petsc-based code. I don't > know if PETSc team is doing the same thing with > mpich/master? (Maybe it is a > good idea?) > > Everything was fine (except the issue > https://github.com/pmodels/mpich/issues/2892 > <https://github.com/pmodels/mpich/issues/2892>) up to > commit 7b8d64debd, but > since commit mpich:a8a2b30fd21), I have a segfault on > a any parallel nightly > test. > > > I attempted a bisect of the above range of commits - and > narrowed down to: > > > db11d4c4a70e39a28be88ed32f00542301699e08 is the first bad > commit > <<<<<<< > > > balay@asterix /home/balay/soft/build/mpich > ((db11d4c4a...)|BISECTING) > $ git show db11d4c4a70e39a28be88ed32f00542301699e08 > commit db11d4c4a70e39a28be88ed32f00542301699e08 (HEAD, > refs/bisect/bad) > Author: Ken Raffenetti <raffenet(a)mcs.anl.gov > <mailto:[email protected]>> > Date: Thu Feb 15 11:37:59 2018 -0600 > > init: Fix tag upper limit initialization > The starting point for this value is equivalent > to the usable tag bits > macro. This value should be set before device > initialization, > otherwise devices will assume they have more bits > than are actually > available. > Signed-off-by: Wesley Bland > <wesley.bland(a)intel.com <mailto:[email protected]>> > > diff --git a/src/mpi/init/initthread.c > b/src/mpi/init/initthread.c > index cbc41f4d5..b31ae2f07 100644 > --- a/src/mpi/init/initthread.c > +++ b/src/mpi/init/initthread.c > @@ -403,7 +403,7 @@ int MPIR_Init_thread(int *argc, char > ***argv, int required, int *provided) > MPIR_Process.attrs.host = MPI_PROC_NULL; > MPIR_Process.attrs.io <http://MPIR_Process.attrs.io> = > MPI_PROC_NULL; > MPIR_Process.attrs.lastusedcode = MPI_ERR_LASTCODE; > - MPIR_Process.attrs.tag_ub = 0; > + MPIR_Process.attrs.tag_ub = MPIR_TAG_USABLE_BITS; > MPIR_Process.attrs.universe = > MPIR_UNIVERSE_SIZE_NOT_SET; > MPIR_Process.attrs.wtime_is_global = 0; > @@ -531,13 +531,6 @@ int MPIR_Init_thread(int *argc, > char ***argv, int required, int *provided) > MPIR_Assert(((unsigned) MPIR_Process. > attrs.tag_ub & ((unsigned) > MPIR_Process.attrs.tag_ub + 1)) == 0); > - /* Set aside tag space for tagged collectives and > failure notification */ > -#ifdef HAVE_TAG_ERROR_BITS > - MPIR_Process.attrs.tag_ub >>= 3; > -#else > - MPIR_Process.attrs.tag_ub >>= 1; > -#endif > - > /* Assert: tag_ub is at least the minimum asked for > in the MPI spec */ > MPIR_Assert(MPIR_Process.attrs.tag_ub >= 32767); > <<<<<<<<<<<<<<<<< > > Reverthing this patch gets mpich-3.3b2 working with petsc > > Satish > > > _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] [petsc-dev] Is mpich/master:a8a2b30fd21 tested with Petsc?
by Eric Chamberland 19 Apr '18

19 Apr '18

Hi, this morning, mpich/master with PETSc is 100% working again for us. Thanks to both commits: https://github.com/pmodels/mpich/commit/c597c8d79deea220a42751fda0f01ce7076… https://github.com/pmodels/mpich/commit/8edabc7373b82dd660019e53d2461317658… and thanks to everybody who helped: Satish Min Wesley Ken Rob Jed :) Eric On 17/04/18 04:58 PM, Min Si wrote: > Hi all, > > Thanks for narrowing down the problem. I checked the MPICH code and > believe this is a bug in MPICH. I just created a PR to fix it: > https://github.com/pmodels/mpich/pull/3097 > > It should be merged into MPICH master branch soon. > > Thanks, > Min > > On 2018/04/17 14:10, Eric Chamberland wrote: >> Hi, >> >> are we talking about the "tag" passed to MPI_Isend for example? >> >> but does that mean there is something to change for any MPI call which >> involves tags usage or is it only a PETSc "bad" tag usage? >> >> thanks Satish for your finding! >> >> Eric >> >> On 16/04/18 11:31 PM, Satish Balay wrote: >>> On Tue, 13 Mar 2018, Eric Chamberland wrote: >>> >>>> Hi, >>>> >>>> each night we are testing mpich/master with our petsc-based code. I >>>> don't >>>> know if PETSc team is doing the same thing with mpich/master? (Maybe >>>> it is a >>>> good idea?) >>>> >>>> Everything was fine (except the issue >>>> https://github.com/pmodels/mpich/issues/2892) up to commit >>>> 7b8d64debd, but >>>> since commit mpich:a8a2b30fd21), I have a segfault on a any parallel >>>> nightly >>>> test. >>> >>> I attempted a bisect of the above range of commits - and narrowed >>> down to: >>> >>>>>>>>>> >>> db11d4c4a70e39a28be88ed32f00542301699e08 is the first bad commit >>> <<<<<<< >>>>>>>>>>> >>> balay@asterix /home/balay/soft/build/mpich ((db11d4c4a...)|BISECTING) >>> $ git show db11d4c4a70e39a28be88ed32f00542301699e08 >>> commit db11d4c4a70e39a28be88ed32f00542301699e08 (HEAD, refs/bisect/bad) >>> Author: Ken Raffenetti <raffenet(a)mcs.anl.gov> >>> Date: Thu Feb 15 11:37:59 2018 -0600 >>> >>> init: Fix tag upper limit initialization >>> The starting point for this value is equivalent to the >>> usable tag bits >>> macro. This value should be set before device initialization, >>> otherwise devices will assume they have more bits than are actually >>> available. >>> Signed-off-by: Wesley Bland <wesley.bland(a)intel.com> >>> >>> diff --git a/src/mpi/init/initthread.c b/src/mpi/init/initthread.c >>> index cbc41f4d5..b31ae2f07 100644 >>> --- a/src/mpi/init/initthread.c >>> +++ b/src/mpi/init/initthread.c >>> @@ -403,7 +403,7 @@ int MPIR_Init_thread(int *argc, char ***argv, int >>> required, int *provided) >>> MPIR_Process.attrs.host = MPI_PROC_NULL; >>> MPIR_Process.attrs.io = MPI_PROC_NULL; >>> MPIR_Process.attrs.lastusedcode = MPI_ERR_LASTCODE; >>> - MPIR_Process.attrs.tag_ub = 0; >>> + MPIR_Process.attrs.tag_ub = MPIR_TAG_USABLE_BITS; >>> MPIR_Process.attrs.universe = MPIR_UNIVERSE_SIZE_NOT_SET; >>> MPIR_Process.attrs.wtime_is_global = 0; >>> @@ -531,13 +531,6 @@ int MPIR_Init_thread(int *argc, char ***argv, >>> int required, int *provided) >>> MPIR_Assert(((unsigned) MPIR_Process. >>> attrs.tag_ub & ((unsigned) >>> MPIR_Process.attrs.tag_ub + 1)) == 0); >>> - /* Set aside tag space for tagged collectives and failure >>> notification */ >>> -#ifdef HAVE_TAG_ERROR_BITS >>> - MPIR_Process.attrs.tag_ub >>= 3; >>> -#else >>> - MPIR_Process.attrs.tag_ub >>= 1; >>> -#endif >>> - >>> /* Assert: tag_ub is at least the minimum asked for in the MPI >>> spec */ >>> MPIR_Assert(MPIR_Process.attrs.tag_ub >= 32767); >>> <<<<<<<<<<<<<<<<< >>> >>> Reverthing this patch gets mpich-3.3b2 working with petsc >>> >>> Satish >>> _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] [petsc-dev] Is mpich/master:a8a2b30fd21 tested with Petsc?
by Junchao Zhang 17 Apr '18

17 Apr '18

Min, I suggest MPICH add tests to play with the maximal MPI tag (through attribute MPI_TAG_UB). PETSc uses tags from the maximal and downwards. I guess MPICH tests use small tags. That is why the bug only showed up with PETSc. --Junchao Zhang On Tue, Apr 17, 2018 at 3:58 PM, Min Si <msi(a)anl.gov> wrote: > Hi all, > > Thanks for narrowing down the problem. I checked the MPICH code and > believe this is a bug in MPICH. I just created a PR to fix it: > https://github.com/pmodels/mpich/pull/3097 > > It should be merged into MPICH master branch soon. > > Thanks, > Min > > > On 2018/04/17 14:10, Eric Chamberland wrote: > >> Hi, >> >> are we talking about the "tag" passed to MPI_Isend for example? >> >> but does that mean there is something to change for any MPI call which >> involves tags usage or is it only a PETSc "bad" tag usage? >> >> thanks Satish for your finding! >> >> Eric >> >> On 16/04/18 11:31 PM, Satish Balay wrote: >> >>> On Tue, 13 Mar 2018, Eric Chamberland wrote: >>> >>> Hi, >>>> >>>> each night we are testing mpich/master with our petsc-based code. I >>>> don't >>>> know if PETSc team is doing the same thing with mpich/master? (Maybe >>>> it is a >>>> good idea?) >>>> >>>> Everything was fine (except the issue >>>> https://github.com/pmodels/mpich/issues/2892) up to commit 7b8d64debd, >>>> but >>>> since commit mpich:a8a2b30fd21), I have a segfault on a any parallel >>>> nightly >>>> test. >>>> >>> >>> I attempted a bisect of the above range of commits - and narrowed down >>> to: >>> >>> >>>>>>>>>> db11d4c4a70e39a28be88ed32f00542301699e08 is the first bad commit >>> <<<<<<< >>> >>>> >>>>>>>>>>> balay@asterix /home/balay/soft/build/mpich >>> ((db11d4c4a...)|BISECTING) >>> $ git show db11d4c4a70e39a28be88ed32f00542301699e08 >>> commit db11d4c4a70e39a28be88ed32f00542301699e08 (HEAD, refs/bisect/bad) >>> Author: Ken Raffenetti <raffenet(a)mcs.anl.gov> >>> Date: Thu Feb 15 11:37:59 2018 -0600 >>> >>> init: Fix tag upper limit initialization >>> The starting point for this value is equivalent to the usable >>> tag bits >>> macro. This value should be set before device initialization, >>> otherwise devices will assume they have more bits than are actually >>> available. >>> Signed-off-by: Wesley Bland <wesley.bland(a)intel.com> >>> >>> diff --git a/src/mpi/init/initthread.c b/src/mpi/init/initthread.c >>> index cbc41f4d5..b31ae2f07 100644 >>> --- a/src/mpi/init/initthread.c >>> +++ b/src/mpi/init/initthread.c >>> @@ -403,7 +403,7 @@ int MPIR_Init_thread(int *argc, char ***argv, int >>> required, int *provided) >>> MPIR_Process.attrs.host = MPI_PROC_NULL; >>> MPIR_Process.attrs.io = MPI_PROC_NULL; >>> MPIR_Process.attrs.lastusedcode = MPI_ERR_LASTCODE; >>> - MPIR_Process.attrs.tag_ub = 0; >>> + MPIR_Process.attrs.tag_ub = MPIR_TAG_USABLE_BITS; >>> MPIR_Process.attrs.universe = MPIR_UNIVERSE_SIZE_NOT_SET; >>> MPIR_Process.attrs.wtime_is_global = 0; >>> @@ -531,13 +531,6 @@ int MPIR_Init_thread(int *argc, char ***argv, int >>> required, int *provided) >>> MPIR_Assert(((unsigned) MPIR_Process. >>> attrs.tag_ub & ((unsigned) MPIR_Process.attrs.tag_ub >>> + 1)) == 0); >>> - /* Set aside tag space for tagged collectives and failure >>> notification */ >>> -#ifdef HAVE_TAG_ERROR_BITS >>> - MPIR_Process.attrs.tag_ub >>= 3; >>> -#else >>> - MPIR_Process.attrs.tag_ub >>= 1; >>> -#endif >>> - >>> /* Assert: tag_ub is at least the minimum asked for in the MPI >>> spec */ >>> MPIR_Assert(MPIR_Process.attrs.tag_ub >= 32767); >>> <<<<<<<<<<<<<<<<< >>> >>> Reverthing this patch gets mpich-3.3b2 working with petsc >>> >>> Satish >>> >>> > _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] [petsc-dev] Is mpich/master:a8a2b30fd21 tested with Petsc?
by Satish Balay 17 Apr '18

17 Apr '18

Thanks! I tried the patch - and this testcase doesn't hang anymore.. Satish On Tue, 17 Apr 2018, Min Si wrote: > Hi all, > > Thanks for narrowing down the problem. I checked the MPICH code and believe > this is a bug in MPICH. I just created a PR to fix it: > https://github.com/pmodels/mpich/pull/3097 > > It should be merged into MPICH master branch soon. > > Thanks, > Min > > On 2018/04/17 14:10, Eric Chamberland wrote: > > Hi, > > > > are we talking about the "tag" passed to MPI_Isend for example? > > > > but does that mean there is something to change for any MPI call which > > involves tags usage or is it only a PETSc "bad" tag usage? > > > > thanks Satish for your finding! > > > > Eric > > > > On 16/04/18 11:31 PM, Satish Balay wrote: > >> On Tue, 13 Mar 2018, Eric Chamberland wrote: > >> > >>> Hi, > >>> > >>> each night we are testing mpich/master with our petsc-based code.ï¿œ I don't > >>> know if PETSc team is doing the same thing with mpich/master?ï¿œï¿œ (Maybe it > >>> is a > >>> good idea?) > >>> > >>> Everything was fine (except the issue > >>> https://github.com/pmodels/mpich/issues/2892) up to commit 7b8d64debd, but > >>> since commit mpich:a8a2b30fd21), I have a segfault on a any parallel > >>> nightly > >>> test. > >> > >> I attempted a bisect of the above range of commits - and narrowed down to: > >> > >>>>>>>>> > >> db11d4c4a70e39a28be88ed32f00542301699e08 is the first bad commit > >> <<<<<<< > >>>>>>>>>> > >> balay@asterix /home/balay/soft/build/mpich ((db11d4c4a...)|BISECTING) > >> $ git show db11d4c4a70e39a28be88ed32f00542301699e08 > >> commit db11d4c4a70e39a28be88ed32f00542301699e08 (HEAD, refs/bisect/bad) > >> Author: Ken Raffenetti <raffenet(a)mcs.anl.gov> > >> Date:ï¿œï¿œ Thu Feb 15 11:37:59 2018 -0600 > >> > >> ï¿œï¿œï¿œï¿œ init: Fix tag upper limit initialization > >> ï¿œï¿œï¿œï¿œ ï¿œï¿œï¿œï¿œ The starting point for this value is equivalent to the usable tag > >> bits > >> ï¿œï¿œï¿œï¿œ macro. This value should be set before device initialization, > >> ï¿œï¿œï¿œï¿œ otherwise devices will assume they have more bits than are actually > >> ï¿œï¿œï¿œï¿œ available. > >> ï¿œï¿œï¿œï¿œ ï¿œï¿œï¿œï¿œ Signed-off-by: Wesley Bland <wesley.bland(a)intel.com> > >> > >> diff --git a/src/mpi/init/initthread.c b/src/mpi/init/initthread.c > >> index cbc41f4d5..b31ae2f07 100644 > >> --- a/src/mpi/init/initthread.c > >> +++ b/src/mpi/init/initthread.c > >> @@ -403,7 +403,7 @@ int MPIR_Init_thread(int *argc, char ***argv, int > >> required, int *provided) > >> ï¿œï¿œï¿œï¿œï¿œ MPIR_Process.attrs.host = MPI_PROC_NULL; > >> ï¿œï¿œï¿œï¿œï¿œ MPIR_Process.attrs.io = MPI_PROC_NULL; > >> ï¿œï¿œï¿œï¿œï¿œ MPIR_Process.attrs.lastusedcode = MPI_ERR_LASTCODE; > >> -ï¿œï¿œï¿œ MPIR_Process.attrs.tag_ub = 0; > >> +ï¿œï¿œï¿œ MPIR_Process.attrs.tag_ub = MPIR_TAG_USABLE_BITS; > >> ï¿œï¿œï¿œï¿œï¿œ MPIR_Process.attrs.universe = MPIR_UNIVERSE_SIZE_NOT_SET; > >> ï¿œï¿œï¿œï¿œï¿œ MPIR_Process.attrs.wtime_is_global = 0; > >> ï¿œ @@ -531,13 +531,6 @@ int MPIR_Init_thread(int *argc, char ***argv, int > >> required, int *provided) > >> ï¿œï¿œï¿œï¿œï¿œ MPIR_Assert(((unsigned) MPIR_Process. > >> ï¿œï¿œï¿œï¿œï¿œï¿œï¿œï¿œï¿œï¿œï¿œï¿œï¿œï¿œï¿œï¿œï¿œï¿œ attrs.tag_ub & ((unsigned) MPIR_Process.attrs.tag_ub + > >> 1)) == 0); > >> ï¿œ -ï¿œï¿œï¿œ /* Set aside tag space for tagged collectives and failure > >> notification */ > >> -#ifdef HAVE_TAG_ERROR_BITS > >> -ï¿œï¿œï¿œ MPIR_Process.attrs.tag_ub >>= 3; > >> -#else > >> -ï¿œï¿œï¿œ MPIR_Process.attrs.tag_ub >>= 1; > >> -#endif > >> - > >> ï¿œï¿œï¿œï¿œï¿œ /* Assert: tag_ub is at least the minimum asked for in the MPI spec > >> */ > >> ï¿œï¿œï¿œï¿œï¿œ MPIR_Assert(MPIR_Process.attrs.tag_ub >= 32767); > >> <<<<<<<<<<<<<<<<< > >> > >> Reverthing this patch gets mpich-3.3b2 working with petsc > >> > >> Satish > >> > > > _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] [petsc-dev] Is mpich/master:a8a2b30fd21 tested with Petsc?
by Min Si 17 Apr '18

17 Apr '18

Hi all, Thanks for narrowing down the problem. I checked the MPICH code and believe this is a bug in MPICH. I just created a PR to fix it: https://github.com/pmodels/mpich/pull/3097 It should be merged into MPICH master branch soon. Thanks, Min On 2018/04/17 14:10, Eric Chamberland wrote: > Hi, > > are we talking about the "tag" passed to MPI_Isend for example? > > but does that mean there is something to change for any MPI call which > involves tags usage or is it only a PETSc "bad" tag usage? > > thanks Satish for your finding! > > Eric > > On 16/04/18 11:31 PM, Satish Balay wrote: >> On Tue, 13 Mar 2018, Eric Chamberland wrote: >> >>> Hi, >>> >>> each night we are testing mpich/master with our petsc-based code. I >>> don't >>> know if PETSc team is doing the same thing with mpich/master? >>> (Maybe it is a >>> good idea?) >>> >>> Everything was fine (except the issue >>> https://github.com/pmodels/mpich/issues/2892) up to commit >>> 7b8d64debd, but >>> since commit mpich:a8a2b30fd21), I have a segfault on a any parallel >>> nightly >>> test. >> >> I attempted a bisect of the above range of commits - and narrowed >> down to: >> >>>>>>>>> >> db11d4c4a70e39a28be88ed32f00542301699e08 is the first bad commit >> <<<<<<< >>>>>>>>>> >> balay@asterix /home/balay/soft/build/mpich ((db11d4c4a...)|BISECTING) >> $ git show db11d4c4a70e39a28be88ed32f00542301699e08 >> commit db11d4c4a70e39a28be88ed32f00542301699e08 (HEAD, refs/bisect/bad) >> Author: Ken Raffenetti <raffenet(a)mcs.anl.gov> >> Date: Thu Feb 15 11:37:59 2018 -0600 >> >> init: Fix tag upper limit initialization >> The starting point for this value is equivalent to the >> usable tag bits >> macro. This value should be set before device initialization, >> otherwise devices will assume they have more bits than are actually >> available. >> Signed-off-by: Wesley Bland <wesley.bland(a)intel.com> >> >> diff --git a/src/mpi/init/initthread.c b/src/mpi/init/initthread.c >> index cbc41f4d5..b31ae2f07 100644 >> --- a/src/mpi/init/initthread.c >> +++ b/src/mpi/init/initthread.c >> @@ -403,7 +403,7 @@ int MPIR_Init_thread(int *argc, char ***argv, int >> required, int *provided) >> MPIR_Process.attrs.host = MPI_PROC_NULL; >> MPIR_Process.attrs.io = MPI_PROC_NULL; >> MPIR_Process.attrs.lastusedcode = MPI_ERR_LASTCODE; >> - MPIR_Process.attrs.tag_ub = 0; >> + MPIR_Process.attrs.tag_ub = MPIR_TAG_USABLE_BITS; >> MPIR_Process.attrs.universe = MPIR_UNIVERSE_SIZE_NOT_SET; >> MPIR_Process.attrs.wtime_is_global = 0; >> @@ -531,13 +531,6 @@ int MPIR_Init_thread(int *argc, char ***argv, >> int required, int *provided) >> MPIR_Assert(((unsigned) MPIR_Process. >> attrs.tag_ub & ((unsigned) >> MPIR_Process.attrs.tag_ub + 1)) == 0); >> - /* Set aside tag space for tagged collectives and failure >> notification */ >> -#ifdef HAVE_TAG_ERROR_BITS >> - MPIR_Process.attrs.tag_ub >>= 3; >> -#else >> - MPIR_Process.attrs.tag_ub >>= 1; >> -#endif >> - >> /* Assert: tag_ub is at least the minimum asked for in the MPI >> spec */ >> MPIR_Assert(MPIR_Process.attrs.tag_ub >= 32767); >> <<<<<<<<<<<<<<<<< >> >> Reverthing this patch gets mpich-3.3b2 working with petsc >> >> Satish >> _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] [petsc-dev] Is mpich/master:a8a2b30fd21 tested with Petsc?
by Eric Chamberland 17 Apr '18

17 Apr '18

Hi, are we talking about the "tag" passed to MPI_Isend for example? but does that mean there is something to change for any MPI call which involves tags usage or is it only a PETSc "bad" tag usage? thanks Satish for your finding! Eric On 16/04/18 11:31 PM, Satish Balay wrote: > On Tue, 13 Mar 2018, Eric Chamberland wrote: > >> Hi, >> >> each night we are testing mpich/master with our petsc-based code. I don't >> know if PETSc team is doing the same thing with mpich/master? (Maybe it is a >> good idea?) >> >> Everything was fine (except the issue >> https://github.com/pmodels/mpich/issues/2892) up to commit 7b8d64debd, but >> since commit mpich:a8a2b30fd21), I have a segfault on a any parallel nightly >> test. > > I attempted a bisect of the above range of commits - and narrowed down to: > >>>>>>>> > db11d4c4a70e39a28be88ed32f00542301699e08 is the first bad commit > <<<<<<< >>>>>>>>> > balay@asterix /home/balay/soft/build/mpich ((db11d4c4a...)|BISECTING) > $ git show db11d4c4a70e39a28be88ed32f00542301699e08 > commit db11d4c4a70e39a28be88ed32f00542301699e08 (HEAD, refs/bisect/bad) > Author: Ken Raffenetti <raffenet(a)mcs.anl.gov> > Date: Thu Feb 15 11:37:59 2018 -0600 > > init: Fix tag upper limit initialization > > The starting point for this value is equivalent to the usable tag bits > macro. This value should be set before device initialization, > otherwise devices will assume they have more bits than are actually > available. > > Signed-off-by: Wesley Bland <wesley.bland(a)intel.com> > > diff --git a/src/mpi/init/initthread.c b/src/mpi/init/initthread.c > index cbc41f4d5..b31ae2f07 100644 > --- a/src/mpi/init/initthread.c > +++ b/src/mpi/init/initthread.c > @@ -403,7 +403,7 @@ int MPIR_Init_thread(int *argc, char ***argv, int required, int *provided) > MPIR_Process.attrs.host = MPI_PROC_NULL; > MPIR_Process.attrs.io = MPI_PROC_NULL; > MPIR_Process.attrs.lastusedcode = MPI_ERR_LASTCODE; > - MPIR_Process.attrs.tag_ub = 0; > + MPIR_Process.attrs.tag_ub = MPIR_TAG_USABLE_BITS; > MPIR_Process.attrs.universe = MPIR_UNIVERSE_SIZE_NOT_SET; > MPIR_Process.attrs.wtime_is_global = 0; > > @@ -531,13 +531,6 @@ int MPIR_Init_thread(int *argc, char ***argv, int required, int *provided) > MPIR_Assert(((unsigned) MPIR_Process. > attrs.tag_ub & ((unsigned) MPIR_Process.attrs.tag_ub + 1)) == 0); > > - /* Set aside tag space for tagged collectives and failure notification */ > -#ifdef HAVE_TAG_ERROR_BITS > - MPIR_Process.attrs.tag_ub >>= 3; > -#else > - MPIR_Process.attrs.tag_ub >>= 1; > -#endif > - > /* Assert: tag_ub is at least the minimum asked for in the MPI spec */ > MPIR_Assert(MPIR_Process.attrs.tag_ub >= 32767); > > <<<<<<<<<<<<<<<<< > > Reverthing this patch gets mpich-3.3b2 working with petsc > > Satish > _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0