January 2021 - petsc-dev - mailman3.cels.anl.gov

Re: [petsc-dev] bin/sh error while running tests
by Pierre Jolivet 20 Jan '21

20 Jan '21

> On 20 Jan 2021, at 12:11 PM, Stefano Zampini <stefano.zampini(a)gmail.com> wrote: > > This is an issue with the default shell used by the makefile. Below is my fix. We should probably have a CI machine that checks for these shell-related errors. I second this. Just spent too much time finding this other bash-ism in gmakefile.test… https://gitlab.com/petsc/petsc/-/commit/e4b11943e93779206a0e5f2091646de2e86… <https://gitlab.com/petsc/petsc/-/commit/e4b11943e93779206a0e5f2091646de2e86…> 2021-01-20T11:21:22.5942304Z /usr/bin/sh: 1: test: false: unexpected operator 2021-01-20T11:21:22.5981176Z make: *** [gmakefile.test:270: check-test-errors] Error 1 Thanks, Pierre > diff --git a/gmakefile.test b/gmakefile.test > index c38e37f..ffd7bdb 100644 > --- a/gmakefile.test > +++ b/gmakefile.test > @@ -379,10 +379,11 @@ starttime: pre-clean $(libpetscall) > @$(eval STARTTIME := $(shell date +%s)) > > report_tests: starttime $(TESTTARGETS) > +ifeq ($(showreport),true) > @$(eval ENDTIME := $(shell date +%s)) > - -@if test ${showreport} == "true"; then elapsed_time=$$(($(ENDTIME)- $(STARTTIME))) && \ > - $(PYTHON) $(CONFIGDIR)/report_tests.py -m $(MAKE) -d $(TESTDIR)/counts -t 5 -e $${elapsed_time};\ > - fi > + elapsed_time=$$(($(ENDTIME)- $(STARTTIME))) && \ > + $(PYTHON) $(CONFIGDIR)/report_tests.py -m $(MAKE) -d $(TESTDIR)/counts -t 5 -e $${elapsed_time}; > +endif > > Il giorno mar 19 gen 2021 alle ore 20:41 Scott Kruger <kruger(a)txcorp.com <mailto:[email protected]>> ha scritto: > > > I can't reproduce this with the latest master: > > hip 1261: git pull > Already up to date. > hip 1262: make -f gmakefile.test test search='notatest' > Using MAKEFLAGS: -- search=notatest > hip 1263: > > > > On 1/19/21 8:19 AM, Stefano Zampini wrote: >> Just rebased over latest master and got this >> >> zampins@vulture:~/Devel/petsc$ make -f gmakefile.test test search='notatest' >> Using MAKEFLAGS: -- search=notatest >> /bin/sh: 1: test: false: unexpected operator >> >> -- >> Stefano > > -- > Tech-X Corporation kruger(a)txcorp.com <mailto:[email protected]> > 5621 Arapahoe Ave, Suite A Phone: (720) 466-3196 > Boulder, CO 80303 Fax: (303) 448-7756 > > > -- > Stefano

1 0

Re: [petsc-dev] bin/sh error while running tests
by Stefano Zampini 20 Jan '21

20 Jan '21

This has been introduced in f42deef95efdb32c74e20b044efbd18de6bff042. That commit also broke another feature that I find very useful, which is to rerun only the failed tests. Now, the echofailures.sh file is no longer updated unless we do full testing. Can we please revert that commit? Il giorno mer 20 gen 2021 alle ore 14:11 Stefano Zampini < stefano.zampini(a)gmail.com> ha scritto: > This is an issue with the default shell used by the makefile. Below is my > fix. We should probably have a CI machine that checks for these > shell-related errors. > > diff --git a/gmakefile.test b/gmakefile.test > index c38e37f..ffd7bdb 100644 > --- a/gmakefile.test > +++ b/gmakefile.test > @@ -379,10 +379,11 @@ starttime: pre-clean $(libpetscall) > @$(eval STARTTIME := $(shell date +%s)) > > report_tests: starttime $(TESTTARGETS) > +ifeq ($(showreport),true) > @$(eval ENDTIME := $(shell date +%s)) > - -@if test ${showreport} == "true"; then > elapsed_time=$$(($(ENDTIME)- $(STARTTIME))) && \ > - $(PYTHON) $(CONFIGDIR)/report_tests.py -m $(MAKE) -d > $(TESTDIR)/counts -t 5 -e $${elapsed_time};\ > - fi > + elapsed_time=$$(($(ENDTIME)- $(STARTTIME))) && \ > + $(PYTHON) $(CONFIGDIR)/report_tests.py -m $(MAKE) -d > $(TESTDIR)/counts -t 5 -e $${elapsed_time}; > +endif > > Il giorno mar 19 gen 2021 alle ore 20:41 Scott Kruger <kruger(a)txcorp.com> > ha scritto: > >> >> >> I can't reproduce this with the latest master: >> >> hip 1261: git pull >> Already up to date. >> hip 1262: make -f gmakefile.test test search='notatest' >> Using MAKEFLAGS: -- search=notatest >> hip 1263: >> >> >> >> On 1/19/21 8:19 AM, Stefano Zampini wrote: >> >> Just rebased over latest master and got this >> >> zampins@vulture:~/Devel/petsc$ make -f gmakefile.test test >> search='notatest' >> Using MAKEFLAGS: -- search=notatest >> /bin/sh: 1: test: false: unexpected operator >> >> -- >> Stefano >> >> >> -- >> Tech-X Corporation kruger(a)txcorp.com >> 5621 Arapahoe Ave, Suite A Phone: (720) 466-3196 >> Boulder, CO 80303 Fax: (303) 448-7756 >> >> > > -- > Stefano > -- Stefano

1 0

Re: [petsc-dev] bin/sh error while running tests
by Stefano Zampini 20 Jan '21

20 Jan '21

This is an issue with the default shell used by the makefile. Below is my fix. We should probably have a CI machine that checks for these shell-related errors. diff --git a/gmakefile.test b/gmakefile.test index c38e37f..ffd7bdb 100644 --- a/gmakefile.test +++ b/gmakefile.test @@ -379,10 +379,11 @@ starttime: pre-clean $(libpetscall) @$(eval STARTTIME := $(shell date +%s)) report_tests: starttime $(TESTTARGETS) +ifeq ($(showreport),true) @$(eval ENDTIME := $(shell date +%s)) - -@if test ${showreport} == "true"; then elapsed_time=$$(($(ENDTIME)- $(STARTTIME))) && \ - $(PYTHON) $(CONFIGDIR)/report_tests.py -m $(MAKE) -d $(TESTDIR)/counts -t 5 -e $${elapsed_time};\ - fi + elapsed_time=$$(($(ENDTIME)- $(STARTTIME))) && \ + $(PYTHON) $(CONFIGDIR)/report_tests.py -m $(MAKE) -d $(TESTDIR)/counts -t 5 -e $${elapsed_time}; +endif Il giorno mar 19 gen 2021 alle ore 20:41 Scott Kruger <kruger(a)txcorp.com> ha scritto: > > > I can't reproduce this with the latest master: > > hip 1261: git pull > Already up to date. > hip 1262: make -f gmakefile.test test search='notatest' > Using MAKEFLAGS: -- search=notatest > hip 1263: > > > > On 1/19/21 8:19 AM, Stefano Zampini wrote: > > Just rebased over latest master and got this > > zampins@vulture:~/Devel/petsc$ make -f gmakefile.test test > search='notatest' > Using MAKEFLAGS: -- search=notatest > /bin/sh: 1: test: false: unexpected operator > > -- > Stefano > > > -- > Tech-X Corporation kruger(a)txcorp.com > 5621 Arapahoe Ave, Suite A Phone: (720) 466-3196 > Boulder, CO 80303 Fax: (303) 448-7756 > > -- Stefano

1 0

Re: [petsc-dev] bin/sh error while running tests
by Scott Kruger 19 Jan '21

19 Jan '21

I can't reproduce this with the latest master: hip 1261: git pull Already up to date. hip 1262: make -f gmakefile.test test search='notatest' Using MAKEFLAGS: -- search=notatest hip 1263: On 1/19/21 8:19 AM, Stefano Zampini wrote: > Just rebased over latest master and got this > > zampins@vulture:~/Devel/petsc$ make -f gmakefile.test test > search='notatest' > Using MAKEFLAGS: -- search=notatest > /bin/sh: 1: test: false: unexpected operator > > -- > Stefano -- Tech-X Corporation kruger(a)txcorp.com 5621 Arapahoe Ave, Suite A Phone: (720) 466-3196 Boulder, CO 80303 Fax: (303) 448-7756

1 0

Re: [petsc-dev] Memory problem with OpenMP and Fieldsplit sub solvers
by Mark Adams 19 Jan '21

19 Jan '21

Well, Summit is down for the day so I moved to NERSc and OMP/FieldSplit/cuSparse/[ILU,LU] seem to be working there. I added PetscInfo lines, below, and the answers with 10 species/threads are perfect. It looks like OMP threads are serialized (see below) but in random order. You can see that field 0 ("e") calls solve 14 times and converged in 13 iterations. I'm not sure what to make of this. I think I'll try to see if I can see any difference in total run time with 1 and 10 OMP threads. 9 SNES Function norm 4.741380472654e-13 [0] MatCUSPARSEGetDeviceMatWrite(): Assemble more than once already [0] PCSetUp(): Setting up PC with same nonzero pattern [0] PCApply_FieldSplit(): thread 2 in field 2 [0] PCSetUp(): Setting up PC with same nonzero pattern [0] MatSolve_SeqAIJCUSPARSE_NaturalOrdering(): Cuda solve (NO) [0] MatSolve_SeqAIJCUSPARSE_NaturalOrdering(): Cuda solve (NO) [0] MatSolve_SeqAIJCUSPARSE_NaturalOrdering(): Cuda solve (NO) [0] MatSolve_SeqAIJCUSPARSE_NaturalOrdering(): Cuda solve (NO) [0] MatSolve_SeqAIJCUSPARSE_NaturalOrdering(): Cuda solve (NO) [0] MatSolve_SeqAIJCUSPARSE_NaturalOrdering(): Cuda solve (NO) [0] MatSolve_SeqAIJCUSPARSE_NaturalOrdering(): Cuda solve (NO) [0] PCApply_FieldSplit(): thread 7 in field 7 [0] PCSetUp(): Setting up PC with same nonzero pattern [0] MatSolve_SeqAIJCUSPARSE_NaturalOrdering(): Cuda solve (NO) [0] MatSolve_SeqAIJCUSPARSE_NaturalOrdering(): Cuda solve (NO) [0] MatSolve_SeqAIJCUSPARSE_NaturalOrdering(): Cuda solve (NO) [0] MatSolve_SeqAIJCUSPARSE_NaturalOrdering(): Cuda solve (NO) .... *[0] PCApply_FieldSplit(): thread 0 in field 0[0] PCSetUp(): Setting up PC with same nonzero pattern[0] MatSolve_SeqAIJCUSPARSE_NaturalOrdering(): Cuda solve (NO)[0] MatSolve_SeqAIJCUSPARSE_NaturalOrdering(): Cuda solve (NO)[0] MatSolve_SeqAIJCUSPARSE_NaturalOrdering(): Cuda solve (NO)[0] MatSolve_SeqAIJCUSPARSE_NaturalOrdering(): Cuda solve (NO)[0] MatSolve_SeqAIJCUSPARSE_NaturalOrdering(): Cuda solve (NO)[0] MatSolve_SeqAIJCUSPARSE_NaturalOrdering(): Cuda solve (NO)[0] MatSolve_SeqAIJCUSPARSE_NaturalOrdering(): Cuda solve (NO)[0] MatSolve_SeqAIJCUSPARSE_NaturalOrdering(): Cuda solve (NO)[0] MatSolve_SeqAIJCUSPARSE_NaturalOrdering(): Cuda solve (NO)[0] MatSolve_SeqAIJCUSPARSE_NaturalOrdering(): Cuda solve (NO)[0] MatSolve_SeqAIJCUSPARSE_NaturalOrdering(): Cuda solve (NO)[0] MatSolve_SeqAIJCUSPARSE_NaturalOrdering(): Cuda solve (NO)[0] MatSolve_SeqAIJCUSPARSE_NaturalOrdering(): Cuda solve (NO)[0] MatSolve_SeqAIJCUSPARSE_NaturalOrdering(): Cuda solve (NO) Linear fieldsplit_e_ solve converged due to CONVERGED_RTOL iterations 13* [0] PCApply_FieldSplit(): thread 5 in field 5 [0] PCSetUp(): Setting up PC with same nonzero pattern [0] MatSolve_SeqAIJCUSPARSE_NaturalOrdering(): Cuda solve (NO) [0] MatSolve_SeqAIJCUSPARSE_NaturalOrdering(): Cuda solve (NO) ... On Tue, Jan 19, 2021 at 8:07 AM Mark Adams <mfadams(a)lbl.gov> wrote: > > > On Mon, Jan 18, 2021 at 11:06 PM Barry Smith <bsmith(a)petsc.dev> wrote: > >> >> Can valgrind run and help with OpenMP? >> > > I am pretty sure. There is also cuda-memcheck that has the same semantics > that works on GPU code, but I'm not sure how good it is for CPU code. > > >> >> You can run in the debugger and find any calls to the options checking >> inside your code block and comment them all out to see if that eliminates >> the problem. >> > > The stack trace does give me the method that it calls the fatal Free in, > so I will try a breakpoint in there. DDT does work with threads but not GPU > code. > > >> >> Also generically how safe is CUDA inside OpenMP? That is with multiple >> threads calling CUDA stuff? >> > > I recall that the XGC code, which has a lot of OMP, Cuda (and Kokkos) does > this. Not 100% sure. > > I know that they recently had to tear out some OMP loops that they > Kokkos'ized because they had some problem mixing Kokkos-OMP and Cuda so > they reverted back to pure OMP. > > >> >> >> Barry >> >> >> >> On Jan 18, 2021, at 7:04 PM, Mark Adams <mfadams(a)lbl.gov> wrote: >> >> >> Added this w/o luck: >> >> #if defined(PETSC_HAVE_CUDA) >> ierr = PetscOptionsCheckCUDA(logView);CHKERRQ(ierr); >> #if defined(PETSC_HAVE_THREADSAFETY) >> ierr = PetscCUPMInitializeCheck();CHKERRQ(ierr); >> #endif >> #endif >> >> Do you think I should keep this in or take it out? Seems like a good idea >> and when it all works we can see if we can make it lazy. >> >> 1) Calling PetscOptions inside threads. I looked quickly at the code and >>> it seems like it should be ok but perhaps not. This is one reason why >>> having stuff like PetscOptionsBegin inside a low-level creation >>> VecCreate_SeqCUDA_Private is normally not done in PETSc. Eventually this >>> needs to be moved or reworked. >>> >>> >> I will try this next. It is hard to see the stack here. I think I will >> put it in ddt and put a breakpoint PetscOptionsEnd_Private. Other ideas >> welcome. >> >> Mark >> >> >>> 2) PetscCUDAInitializeCheck is not thread safe.If it is being call for >>> the first timeby multiple threads there can be trouble. So edit init.c and >>> under >>> >>> #if defined(PETSC_HAVE_CUDA) >>> ierr = PetscOptionsCheckCUDA(logView);CHKERRQ(ierr); >>> #endif >>> >>> #if defined(PETSC_HAVE_HIP) >>> ierr = PetscOptionsCheckHIP(logView);CHKERRQ(ierr); >>> #endif >>> >>> put in >>> #if defined thread safety >>> PetscCUPMInitializeCheck >>> #endif >>> >>> this will force the initialize to be done before any threads are used >>> >> >>> >>

1 0

bin/sh error while running tests
by Stefano Zampini 19 Jan '21

19 Jan '21

Just rebased over latest master and got this zampins@vulture:~/Devel/petsc$ make -f gmakefile.test test search='notatest' Using MAKEFLAGS: -- search=notatest /bin/sh: 1: test: false: unexpected operator -- Stefano

1 0

Re: [petsc-dev] Memory problem with OpenMP and Fieldsplit sub solvers
by Mark Adams 19 Jan '21

19 Jan '21

On Mon, Jan 18, 2021 at 11:06 PM Barry Smith <bsmith(a)petsc.dev> wrote: > > Can valgrind run and help with OpenMP? > I am pretty sure. There is also cuda-memcheck that has the same semantics that works on GPU code, but I'm not sure how good it is for CPU code. > > You can run in the debugger and find any calls to the options checking > inside your code block and comment them all out to see if that eliminates > the problem. > The stack trace does give me the method that it calls the fatal Free in, so I will try a breakpoint in there. DDT does work with threads but not GPU code. > > Also generically how safe is CUDA inside OpenMP? That is with multiple > threads calling CUDA stuff? > I recall that the XGC code, which has a lot of OMP, Cuda (and Kokkos) does this. Not 100% sure. I know that they recently had to tear out some OMP loops that they Kokkos'ized because they had some problem mixing Kokkos-OMP and Cuda so they reverted back to pure OMP. > > > Barry > > > > On Jan 18, 2021, at 7:04 PM, Mark Adams <mfadams(a)lbl.gov> wrote: > > > Added this w/o luck: > > #if defined(PETSC_HAVE_CUDA) > ierr = PetscOptionsCheckCUDA(logView);CHKERRQ(ierr); > #if defined(PETSC_HAVE_THREADSAFETY) > ierr = PetscCUPMInitializeCheck();CHKERRQ(ierr); > #endif > #endif > > Do you think I should keep this in or take it out? Seems like a good idea > and when it all works we can see if we can make it lazy. > > 1) Calling PetscOptions inside threads. I looked quickly at the code and >> it seems like it should be ok but perhaps not. This is one reason why >> having stuff like PetscOptionsBegin inside a low-level creation >> VecCreate_SeqCUDA_Private is normally not done in PETSc. Eventually this >> needs to be moved or reworked. >> >> > I will try this next. It is hard to see the stack here. I think I will put > it in ddt and put a breakpoint PetscOptionsEnd_Private. Other ideas welcome. > > Mark > > >> 2) PetscCUDAInitializeCheck is not thread safe.If it is being call for >> the first timeby multiple threads there can be trouble. So edit init.c and >> under >> >> #if defined(PETSC_HAVE_CUDA) >> ierr = PetscOptionsCheckCUDA(logView);CHKERRQ(ierr); >> #endif >> >> #if defined(PETSC_HAVE_HIP) >> ierr = PetscOptionsCheckHIP(logView);CHKERRQ(ierr); >> #endif >> >> put in >> #if defined thread safety >> PetscCUPMInitializeCheck >> #endif >> >> this will force the initialize to be done before any threads are used >> > >> >

1 0

Re: [petsc-dev] Memory problem with OpenMP and Fieldsplit sub solvers
by Barry Smith 18 Jan '21

18 Jan '21

Can valgrind run and help with OpenMP? You can run in the debugger and find any calls to the options checking inside your code block and comment them all out to see if that eliminates the problem. Also generically how safe is CUDA inside OpenMP? That is with multiple threads calling CUDA stuff? Barry > On Jan 18, 2021, at 7:04 PM, Mark Adams <mfadams(a)lbl.gov> wrote: > > > Added this w/o luck: > > #if defined(PETSC_HAVE_CUDA) > ierr = PetscOptionsCheckCUDA(logView);CHKERRQ(ierr); > #if defined(PETSC_HAVE_THREADSAFETY) > ierr = PetscCUPMInitializeCheck();CHKERRQ(ierr); > #endif > #endif > > Do you think I should keep this in or take it out? Seems like a good idea and when it all works we can see if we can make it lazy. > > 1) Calling PetscOptions inside threads. I looked quickly at the code and it seems like it should be ok but perhaps not. This is one reason why having stuff like PetscOptionsBegin inside a low-level creation VecCreate_SeqCUDA_Private is normally not done in PETSc. Eventually this needs to be moved or reworked. > > > I will try this next. It is hard to see the stack here. I think I will put it in ddt and put a breakpoint PetscOptionsEnd_Private. Other ideas welcome. > > Mark > > 2) PetscCUDAInitializeCheck is not thread safe.If it is being call for the first timeby multiple threads there can be trouble. So edit init.c and under > > #if defined(PETSC_HAVE_CUDA) > ierr = PetscOptionsCheckCUDA(logView);CHKERRQ(ierr); > #endif > > #if defined(PETSC_HAVE_HIP) > ierr = PetscOptionsCheckHIP(logView);CHKERRQ(ierr); > #endif > > put in > #if defined thread safety > PetscCUPMInitializeCheck > #endif > > this will force the initialize to be done before any threads are used >

1 0

Re: [petsc-dev] Memory problem with OpenMP and Fieldsplit sub solvers
by Mark Adams 18 Jan '21

18 Jan '21

Added this w/o luck: #if defined(PETSC_HAVE_CUDA) ierr = PetscOptionsCheckCUDA(logView);CHKERRQ(ierr); #if defined(PETSC_HAVE_THREADSAFETY) ierr = PetscCUPMInitializeCheck();CHKERRQ(ierr); #endif #endif Do you think I should keep this in or take it out? Seems like a good idea and when it all works we can see if we can make it lazy. 1) Calling PetscOptions inside threads. I looked quickly at the code and > it seems like it should be ok but perhaps not. This is one reason why > having stuff like PetscOptionsBegin inside a low-level creation > VecCreate_SeqCUDA_Private is normally not done in PETSc. Eventually this > needs to be moved or reworked. > > I will try this next. It is hard to see the stack here. I think I will put it in ddt and put a breakpoint PetscOptionsEnd_Private. Other ideas welcome. Mark > 2) PetscCUDAInitializeCheck is not thread safe.If it is being call for the > first timeby multiple threads there can be trouble. So edit init.c and under > > #if defined(PETSC_HAVE_CUDA) > ierr = PetscOptionsCheckCUDA(logView);CHKERRQ(ierr); > #endif > > #if defined(PETSC_HAVE_HIP) > ierr = PetscOptionsCheckHIP(logView);CHKERRQ(ierr); > #endif > > put in > #if defined thread safety > PetscCUPMInitializeCheck > #endif > > this will force the initialize to be done before any threads are used > >

1 0

Re: [petsc-dev] navigating gitlab
by Mark Adams 18 Jan '21

18 Jan '21

Thanks, this works. gitk 7ba7a8179202d83658a309df68e2ebc2037284d4 > > git show 7ba7a8179202d83658a309df68e2ebc2037284d4 > > Satish >

1 0