November 2021 - petsc-dev - mailman3.cels.anl.gov

Re: [petsc-dev] [DocTip!] #2: Aiming for self-updating docs
by Mark Adams 08 Nov '21

08 Nov '21

> > > For those who love literate programming and fortran, PPPL developed this > in the 80's: > https://w3.pppl.gov/~krommes/fweb.html > and I believe it is still maintained. > I've been working with PPPL people 15+ years and have not seen a code that uses it. However I did once get caught by a crazy, brilliant but crazy, Russian at PPPL that yammered on to me about a Latex programming language, that sounded insane but that may have been the delivery, that may very well have been this project. I am sure it is dead.

1 0

Re: [petsc-dev] [DocTip!] #2: Aiming for self-updating docs
by Matthew Knepley 08 Nov '21

08 Nov '21

On Mon, Nov 8, 2021 at 12:36 PM Scott Kruger <kruger(a)txcorp.com> wrote: > On 2021-11-06 11:12, Matthew Knepley did write: > > On Sat, Nov 6, 2021 at 10:34 AM Patrick Sanan <patrick.sanan(a)gmail.com> > > wrote: > > > I don't know what WEB is, but if you're saying that this is kinda > clunky, > > > yes it definitely is - my only contention is that it's better than > > > copy-pasting code and output. I'm not sure if there's an easier and/or > > > better way with Sphinx. > > > > > > > WEB was the futuristic documentation idea of Don Knuth. > > > > It never caught on (for good reasons IMO), but it is important > historically and programmers should be aware of it: > https://en.wikipedia.org/wiki/Literate_programming Threadjack: My major problem with WEB was that it in essence assumed that everything was One Big Source File. I tried documenting parts of PETSc with it in the 90s and I found that it did not perform as well as sowing because it really wanted one gigantic control flow, just like the algorithms Knuth was presenting. Maybe a combination of sowing and Web could bridge the gap. Thanks, Matt > > For those who love literate programming and fortran, PPPL developed this > in the 80's: > https://w3.pppl.gov/~krommes/fweb.html > and I believe it is still maintained. > > I dealt with a code written with this tool. Interesting, but I never > wanted to follow it myself. > > Scott > > > > > Thanks, > > > > Matt > > > > > > > > > >> Doing actual literate documentation of key tutorial programs would be > a > > >> nice way of doing this, but I realise that's a lot more effort. > > >> > > > This is still a hope/plan to go into doc/tutorials - follow the deal.ii > > > model for a small number of key examples. Matt has done a couple of > pages > > > there already, in this direction. > > > > > > Lawrence > > > > > > > > > > -- > > What most experimenters take for granted before they begin their > > experiments is infinitely more interesting than any results to which > their > > experiments lead. > > -- Norbert Wiener > > > > https://www.cse.buffalo.edu/~knepley/ < > http://www.cse.buffalo.edu/~knepley/> > > -- > Scott Kruger > Tech-X Corporation kruger(a)txcorp.com > 5621 Arapahoe Ave, Suite A Phone: (720) 466-3196 > Boulder, CO 80303 Fax: (303) 448-7756 > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>

1 0

Re: [petsc-dev] [DocTip!] #3 CI docs build and preview
by Scott Kruger 08 Nov '21

08 Nov '21

On 2021-11-06 15:09, Patrick Sanan did write: > Unfortunately, I don't know a simple way to reliably preview a single .rst > file (as you might be used to from working with some Markdown-based tools). rst2html is the standard python docutils command. The problem is that Sphinx is really rst+(extensions for book-like documentation) so the extensions won't be handled, But that will be true for MyST well. Scott -- Scott Kruger Tech-X Corporation kruger(a)txcorp.com 5621 Arapahoe Ave, Suite A Phone: (720) 466-3196 Boulder, CO 80303 Fax: (303) 448-7756

1 0

Re: [petsc-dev] I am getting this error ...
by Satish Balay 08 Nov '21

08 Nov '21

Better yet.. [from Junchao] export CRAY_ACCEL_TARGET=host Satish On Fri, 5 Nov 2021, Mark Adams wrote: > Bingo! > > On Fri, Nov 5, 2021 at 12:54 PM Satish Balay <balay(a)mcs.anl.gov> wrote: > > > Yeah remove [C,CPP,CXX,CXXPP,F] FLAGS > > > > CPP defaults to '$CC -E' - so with "--with-cc='cc -mp=gpu'" - it should > > use "cc -mp=gpu -E" > > > > Satish > > > > On Fri, 5 Nov 2021, Mark Adams wrote: > > > > > How about CPP flags? > > > > > > On Fri, Nov 5, 2021 at 12:29 PM Satish Balay <balay(a)mcs.anl.gov> wrote: > > > > > > > I guess another way to deal with this is: not use CFLAGS etc.. > > > > > > > > > > > > --with-cc='cc -mp=gpu' --with-cxx='CC -mp=gpu' --with-fc='ftn -mp=gpu' > > > > > > > > Satish > > > > > > > > > > > > On Fri, 5 Nov 2021, Mark Adams wrote: > > > > > > > > > Yes, thanks. > > > > > I emailed the NERSc person and told him where we are and that we > > could > > > > fix > > > > > this manually, but I don't have this issue on Summit ... > > > > > He did not understand for the longest time, not sure he does now, > > that we > > > > > do not add -gpu. > > > > > And I gave him your reproducer. > > > > > With any luck, with the reproducer, they can figure this out at some > > > > point. > > > > > MPI is not working well for my scaling studies and my app has also > > given > > > > up > > > > > on nvhpc, so we are good. > > > > > > > > > > > > > > > On Fri, Nov 5, 2021 at 9:49 AM Satish Balay <balay(a)mcs.anl.gov> > > wrote: > > > > > > > > > > > For now - you could manually edit petscvariables and remove > > -mp=gpu > > > > from > > > > > > it. > > > > > > > > > > > > Its primarily required to make configure happy. > > > > > > > > > > > > Satish > > > > > > > > > > > > > > > > > > > > > > > > On Thu, 4 Nov 2021, Barry Smith wrote: > > > > > > > > > > > > > > > > > > > > This comes from the persistent problem with PETSc's make > > system > > > > using > > > > > > too many flags for compiling CUDA that have not been tested by > > > > configure. > > > > > > See below the -mp=gpu is provided probably from the CPPFLAGS or > > > > > > CXXPPFLAGS(sp) that is improperly used by the PETSc makefiles to > > > > compile > > > > > > CUDA code! > > > > > > > > > > > > > > > > > > > > > Using CUDA compile: > > > > > > /global/common/software/nersc/cos1.3/cuda/11.3.0/bin/nvcc -o .o > > > > > > -I/opt/cray/pe/mpich/8.1.10/ofi/nvidia/20.7/include > > > > -I/opt/cray/pe/libsci/ > > > > > > 21.08.1.2/NVIDIA/20.7/x86_64/include > > -I/opt/cray/pe/pmi/6.0.14/include > > > > > > -I/opt/cray/pe/dsmml/0.2.2/dsmml//include > > > > > > -I/opt/cray/xpmem/2.2.40-7.0.1.0_3.1__g1d7a24d.shasta/include -g > > > > > > -Xcompiler -rdynamic -DLANDAU_DIM=2 -DLANDAU_MAX_SPECIES=10 > > > > > > -DLANDAU_MAX_Q=4 -Xcompiler -fPIC -std=c++17 -gencode > > > > > > arch=compute_80,code=sm_80 -Wno-deprecated-gpu-targets -c > > > > > > --compiler-options=-I/global/homes/m/madams/petsc/include > > > > > > > > -I/global/homes/m/madams/petsc/arch-perlmutter-opt-nvidia-cuda/include > > > > > > -I/global/common/software/nersc/cos1.3/cuda/11.3.0/include -mp=gpu > > > > > > -I/global/homes/m/madams/petsc/include > > > > > > > > -I/global/homes/m/madams/petsc/arch-perlmutter-opt-nvidia-cuda/include > > > > > > -I/global/common/software/nersc/cos1.3/cuda/11.3.0/include > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Nov 4, 2021, at 7:31 PM, Mark Adams <mfadams(a)lbl.gov> > > wrote: > > > > > > > > > > > > > > > > OK, configure is done. > > > > > > > > Maybe I have too many -mp=gpu > > > > > > > > > > > > > > > > CUDAC > > > > > > > > > > > > arch-perlmutter-opt-nvidia-cuda/obj/sys/classes/random/impls/curand/curand2.o > > > > > > > > gcc: error: unrecognized command line option οΏ½-mp=gpuοΏ½; did you > > > > mean > > > > > > οΏ½-mpkuοΏ½? > > > > > > > > > > > > > > > > On Thu, Nov 4, 2021 at 5:51 PM Barry Smith <bsmith(a)petsc.dev > > > > <mailto: > > > > > > bsmith(a)petsc.dev>> wrote: > > > > > > > > > > > > > > > > Need the same thing for the C++ preprocessor flag > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> On Nov 4, 2021, at 5:44 PM, Mark Adams <mfadams(a)lbl.gov > > <mailto: > > > > > > mfadams(a)lbl.gov>> wrote: > > > > > > > >> > > > > > > > >> It gets a lot further. > > > > > > > >> > > > > > > > >> On Thu, Nov 4, 2021 at 5:32 PM Mark Adams <mfadams(a)lbl.gov > > > > <mailto: > > > > > > mfadams(a)lbl.gov>> wrote: > > > > > > > >> OK, sorry I missed the CPPFLAGS. It is running now. > > > > > > > >> Thanks, > > > > > > > >> > > > > > > > >> On Thu, Nov 4, 2021 at 4:43 PM Satish Balay < > > balay(a)mcs.anl.gov > > > > > > <mailto:[email protected]>> wrote: > > > > > > > >> Multiple e-mail threads on the same issue (: > > > > > > > >> > > > > > > > >> As suggested in my earlier thread - add -mp=gpu to both > > CPPFLAGS > > > > and > > > > > > CFLAGS [or LDFLAGS] > > > > > > > >> > > > > > > > >> Satish > > > > > > > >> > > > > > > > >> ------- > > > > > > > >> Executing: cc -o > > /tmp/petsc-Vvs8_T/config.setCompilers/conftest > > > > -g > > > > > > -DLANDAU_DIM=2 -DLANDAU_MAX_SPECIES=10 -DLANDAU_MAX_Q=4 > > > > > > /tmp/petsc-Vvs8_T/config.setCompilers/conftest.o > > > > > > > >> Possible ERROR while running linker: > > > > > > > >> stderr: > > > > > > > >> nvc-Warning-The -gpu option has no effect unless a > > > > language-specific > > > > > > option to enable GPU code generation is used (e.g.: -acc, -mp=gpu, > > > > -stdpar, > > > > > > -cuda) > > > > > > > >> > > > > > > > >> > > > > > > > >> On Thu, 4 Nov 2021, Mark Adams wrote: > > > > > > > >> > > > > > > > >> > It is CPPFLAGS. I seem to get the same behavior. > > > > > > > >> > > > > > > > > >> > FWIW, I did get this response from NERSc but I don't know > > how to > > > > > > interpret > > > > > > > >> > it. > > > > > > > >> > > > > > > > > >> > He seems to be saying that I don't need -mp=gpu for the > > device > > > > > > compiler > > > > > > > >> > (nvcc). He seems to think that I am adding -gpu. > > > > > > > >> > > > > > > > > >> > nvcc -- The device compiler does not need any of those flags > > > > > > because it > > > > > > > >> > already knows that it's being fed cuda code. The warning > > you're > > > > > > seeing is > > > > > > > >> > coming from nvc (which is the host / CPU side compiler) if > > > > you're > > > > > > in the > > > > > > > >> > PrgEnv-nvidia environment. You should not need to add > > -mp=gpu > > > > and > > > > > > -cuda, > > > > > > > >> > please just add the -cuda flag (to your host code) not to > > the > > > > > > device code. > > > > > > > >> > > > > > > > > >> > I will try to talk with this guy again. > > > > > > > >> > > > > > > > > >> > Thanks, > > > > > > > >> > > > > > > > > >> > > > > > > > > >> > On Thu, Nov 4, 2021 at 4:11 PM Barry Smith < > > bsmith(a)petsc.dev > > > > > > <mailto:[email protected]>> wrote: > > > > > > > >> > > > > > > > > >> > > > > > > > > > >> > > Yes, you need to use the CPPFLAGS which maybe called > > > > CPPCFLAGS > > > > > > I am not > > > > > > > >> > > sure > > > > > > > >> > > > > > > > > > >> > > > > > > > > > >> > > On Nov 4, 2021, at 3:23 PM, Mark Adams <mfadams(a)lbl.gov > > > > <mailto: > > > > > > mfadams(a)lbl.gov>> wrote: > > > > > > > >> > > > > > > > > > >> > > Ah, CCFLAGS does not seem to work. > > > > > > > >> > > > > > > > > > >> > > On Thu, Nov 4, 2021 at 3:07 PM Barry Smith < > > bsmith(a)petsc.dev > > > > > > <mailto:[email protected]>> wrote: > > > > > > > >> > > > > > > > > > >> > >> > > > > > > > >> > >> You have to pass in the flag to turn off the bitching > > about > > > > > > -gpu to the > > > > > > > >> > >> C preprocessor, not the C compiler. > > > > > > > >> > >> > > > > > > > >> > >> > > > > > > > >> > >> stderr: > > > > > > > >> > >> nvc-Warning-The -gpu option has no effect unless a > > > > > > language-specific > > > > > > > >> > >> option to enable GPU code generation is used (e.g.: -acc, > > > > > > -mp=gpu, -stdpar, > > > > > > > >> > >> -cuda) > > > > > > > >> > >> Source: > > > > > > > >> > >> #include "confdefs.h" > > > > > > > >> > >> #include "conffix.h" > > > > > > > >> > >> #include <stdlib.h> > > > > > > > >> > >> > > > > > > > >> > >> > > > > > > > >> > >> > > > > > > > >> > >> > On Nov 4, 2021, at 2:49 PM, Mark Adams < > > mfadams(a)lbl.gov > > > > > > <mailto:[email protected]>> wrote: > > > > > > > >> > >> > > > > > > > > >> > >> > on Perlmutter with nvhpc: > > > > > > > >> > >> > > > > > > > > >> > >> > Defined make macro "CPP" to "cc --use > > cpp32" > > > > > > > >> > >> > Preprocessing source: > > > > > > > >> > >> > #include "confdefs.h" > > > > > > > >> > >> > #include "conffix.h" > > > > > > > >> > >> > #include <stdlib.h> > > > > > > > >> > >> > > > > > > > > >> > >> > Executing: cc --use cpp32 > > > > > > -I/tmp/petsc-jV9U1b/config.setCompilers > > > > > > > >> > >> /tmp/petsc-jV9U1b/config.setCompilers/conftest.c > > > > > > > >> > >> > Possible ERROR while running preprocessor: exit code 1 > > > > > > > >> > >> > stderr: > > > > > > > >> > >> > nvc-Error-Unknown switch: --use > > > > > > > >> > >> > Source: > > > > > > > >> > >> > <configure.log> > > > > > > > >> > >> > > > > > > > >> > >> <configure.log> > > > > > > > >> > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> <configure.log> > > > > > > > > > > > > > > > > <make.log><configure.log> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >

1 0

Re: [petsc-dev] [DocTip!] #2: Aiming for self-updating docs
by Scott Kruger 08 Nov '21

08 Nov '21

On 2021-11-06 11:12, Matthew Knepley did write: > On Sat, Nov 6, 2021 at 10:34 AM Patrick Sanan <patrick.sanan(a)gmail.com> > wrote: > > I don't know what WEB is, but if you're saying that this is kinda clunky, > > yes it definitely is - my only contention is that it's better than > > copy-pasting code and output. I'm not sure if there's an easier and/or > > better way with Sphinx. > > > > WEB was the futuristic documentation idea of Don Knuth. > It never caught on (for good reasons IMO), but it is important historically and programmers should be aware of it: https://en.wikipedia.org/wiki/Literate_programming For those who love literate programming and fortran, PPPL developed this in the 80's: https://w3.pppl.gov/~krommes/fweb.html and I believe it is still maintained. I dealt with a code written with this tool. Interesting, but I never wanted to follow it myself. Scott > Thanks, > > Matt > > > > > >> Doing actual literate documentation of key tutorial programs would be a > >> nice way of doing this, but I realise that's a lot more effort. > >> > > This is still a hope/plan to go into doc/tutorials - follow the deal.ii > > model for a small number of key examples. Matt has done a couple of pages > > there already, in this direction. > > > > Lawrence > > > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/> -- Scott Kruger Tech-X Corporation kruger(a)txcorp.com 5621 Arapahoe Ave, Suite A Phone: (720) 466-3196 Boulder, CO 80303 Fax: (303) 448-7756

1 0

Re: [petsc-dev] CI fail
by Pierre Jolivet 07 Nov '21

07 Nov '21

You need to rebase your branch, you are missing https://gitlab.com/petsc/petsc/-/commit/16494f630f598d90bd39b2bdce6fb796c5f… <https://gitlab.com/petsc/petsc/-/commit/16494f630f598d90bd39b2bdce6fb796c5f…> Thanks, Pierre > On 7 Nov 2021, at 1:54 PM, Mark Adams <mfadams(a)lbl.gov> wrote: > > The pipeline seems to be failing with something in Slepc > https://gitlab.com/petsc/petsc/-/jobs/1756604732 <https://gitlab.com/petsc/petsc/-/jobs/1756604732>

1 0

CI fail
by Mark Adams 07 Nov '21

07 Nov '21

The pipeline seems to be failing with something in Slepc https://gitlab.com/petsc/petsc/-/jobs/1756604732

1 0

Re: [petsc-dev] OpenMP
by Mark Adams 07 Nov '21

07 Nov '21

Ah, great. I guessed right (that was the page I was looking at but I still don't see this) #if defined(_OPENMP) && _OPENMP >= 201811 On Sat, Nov 6, 2021 at 8:47 PM Junchao Zhang <junchao.zhang(a)gmail.com> wrote: > > > On Sat, Nov 6, 2021 at 3:51 PM Mark Adams <mfadams(a)lbl.gov> wrote: > >> Yea, that is a bit inscrutable, but I see mumps is the main/only user of >> this: >> >> /* if using PETSc OpenMP support, we only call MUMPS on master ranks. >> Before/after the call, we change/restore CPUs the master ranks can run on */ >> >> And I see _OPENMP is a macro for the release date (yyyymm) of the OMP >> version. It's not clear what the v5.0 is ( >> https://www.openmp.org/specifications/) >> > {200505,"2.5"},{200805,"3.0"},{201107,"3.1"},{201307,"4.0"},{201511,"4.5"},{201811,"5.0"},{202011,"5.1"} > > On Sat, Nov 6, 2021 at 4:27 PM Junchao Zhang <junchao.zhang(a)gmail.com> >> wrote: >> >>> >>> >>> On Sat, Nov 6, 2021 at 5:51 AM Mark Adams <mfadams(a)lbl.gov> wrote: >>> >>>> Two questions on OMP: >>>> >>>> * Can I test for the version of OMP? I want >= 5 and I see this, which >>>> looks promising: >>>> include/petscsys.h:#elif defined(_OPENMP) && *_OPENMP >= 201307* && >>>> !defined(_WIN32) >>>> >>>> * What is the difference between HAVE_OPENMP and >>>> HAVE_OPENMP_SUPPORT. >>>> >>>> # this is different from HAVE_OPENMP. HAVE_OPENMP_SUPPORT checks if we >>> have facilities to support >>> # running PETSc in flat-MPI mode and third party libraries in MPI+OpenMP >>> hybrid mode >>> if self.mpi.found and self.mpi.support_mpi3_shm and self.pthread.found >>> and self.hwloc.found: >>> # Apple pthread does not provide this functionality >>> if self.function.check('pthread_barrier_init', libraries = 'pthread'): >>> self.addDefine('HAVE_OPENMP_SUPPORT', 1) >>> >>> >>>> Thanks, >>>> Mark >>>> >>>

1 0

Re: [petsc-dev] OpenMP
by Junchao Zhang 06 Nov '21

06 Nov '21

On Sat, Nov 6, 2021 at 3:51 PM Mark Adams <mfadams(a)lbl.gov> wrote: > Yea, that is a bit inscrutable, but I see mumps is the main/only user of > this: > > /* if using PETSc OpenMP support, we only call MUMPS on master ranks. > Before/after the call, we change/restore CPUs the master ranks can run on */ > > And I see _OPENMP is a macro for the release date (yyyymm) of the OMP > version. It's not clear what the v5.0 is ( > https://www.openmp.org/specifications/) > {200505,"2.5"},{200805,"3.0"},{201107,"3.1"},{201307,"4.0"},{201511,"4.5"},{201811,"5.0"},{202011,"5.1"} On Sat, Nov 6, 2021 at 4:27 PM Junchao Zhang <junchao.zhang(a)gmail.com> > wrote: > >> >> >> On Sat, Nov 6, 2021 at 5:51 AM Mark Adams <mfadams(a)lbl.gov> wrote: >> >>> Two questions on OMP: >>> >>> * Can I test for the version of OMP? I want >= 5 and I see this, which >>> looks promising: >>> include/petscsys.h:#elif defined(_OPENMP) && *_OPENMP >= 201307* && >>> !defined(_WIN32) >>> >>> * What is the difference between HAVE_OPENMP and >>> HAVE_OPENMP_SUPPORT. >>> >>> # this is different from HAVE_OPENMP. HAVE_OPENMP_SUPPORT checks if we >> have facilities to support >> # running PETSc in flat-MPI mode and third party libraries in MPI+OpenMP >> hybrid mode >> if self.mpi.found and self.mpi.support_mpi3_shm and self.pthread.found >> and self.hwloc.found: >> # Apple pthread does not provide this functionality >> if self.function.check('pthread_barrier_init', libraries = 'pthread'): >> self.addDefine('HAVE_OPENMP_SUPPORT', 1) >> >> >>> Thanks, >>> Mark >>> >>

1 0

Re: [petsc-dev] OpenMP
by Mark Adams 06 Nov '21

06 Nov '21

Yea, that is a bit inscrutable, but I see mumps is the main/only user of this: /* if using PETSc OpenMP support, we only call MUMPS on master ranks. Before/after the call, we change/restore CPUs the master ranks can run on */ And I see _OPENMP is a macro for the release date (yyyymm) of the OMP version. It's not clear what the v5.0 is ( https://www.openmp.org/specifications/) On Sat, Nov 6, 2021 at 4:27 PM Junchao Zhang <junchao.zhang(a)gmail.com> wrote: > > > On Sat, Nov 6, 2021 at 5:51 AM Mark Adams <mfadams(a)lbl.gov> wrote: > >> Two questions on OMP: >> >> * Can I test for the version of OMP? I want >= 5 and I see this, which >> looks promising: >> include/petscsys.h:#elif defined(_OPENMP) && *_OPENMP >= 201307* && >> !defined(_WIN32) >> >> * What is the difference between HAVE_OPENMP and >> HAVE_OPENMP_SUPPORT. >> >> # this is different from HAVE_OPENMP. HAVE_OPENMP_SUPPORT checks if we > have facilities to support > # running PETSc in flat-MPI mode and third party libraries in MPI+OpenMP > hybrid mode > if self.mpi.found and self.mpi.support_mpi3_shm and self.pthread.found and > self.hwloc.found: > # Apple pthread does not provide this functionality > if self.function.check('pthread_barrier_init', libraries = 'pthread'): > self.addDefine('HAVE_OPENMP_SUPPORT', 1) > > >> Thanks, >> Mark >> >

1 0