November 2021 - petsc-dev - mailman3.cels.anl.gov

Re: [petsc-dev] OpenMP
by Junchao Zhang 06 Nov '21

06 Nov '21

On Sat, Nov 6, 2021 at 5:51 AM Mark Adams <mfadams(a)lbl.gov> wrote: > Two questions on OMP: > > * Can I test for the version of OMP? I want >= 5 and I see this, which > looks promising: > include/petscsys.h:#elif defined(_OPENMP) && *_OPENMP >= 201307* && > !defined(_WIN32) > > * What is the difference between HAVE_OPENMP and > HAVE_OPENMP_SUPPORT. > > # this is different from HAVE_OPENMP. HAVE_OPENMP_SUPPORT checks if we have facilities to support # running PETSc in flat-MPI mode and third party libraries in MPI+OpenMP hybrid mode if self.mpi.found and self.mpi.support_mpi3_shm and self.pthread.found and self.hwloc.found: # Apple pthread does not provide this functionality if self.function.check('pthread_barrier_init', libraries = 'pthread'): self.addDefine('HAVE_OPENMP_SUPPORT', 1) > Thanks, > Mark >

1 0

Re: [petsc-dev] [DocTip!] #2: Aiming for self-updating docs
by Matthew Knepley 06 Nov '21

06 Nov '21

On Sat, Nov 6, 2021 at 10:34 AM Patrick Sanan <patrick.sanan(a)gmail.com> wrote: > > > Am Fr., 5. Nov. 2021 um 19:18 Uhr schrieb Lawrence Mitchell <wence(a)gmx.li > >: > >> >> > On 5 Nov 2021, at 15:25, Patrick Sanan <patrick.sanan(a)gmail.com> wrote: >> > >> > Good question. We don't have any that start at specific line numbers, >> currently, as it is indeed too brittle - I suspect I removed some that were >> there at one point. >> > >> > The inclusions essentially all use :start-at: to specify a line to >> match. Ideally you'd also use :end-at: to specify where to stop, but some >> short snippets do instead specify a number of lines. Note that a >> practically handy option is :append:, as in this example where you are >> trying to excerpt the function PetscError(), and you can match on the >> PetscFunctionReturn(0) and then re-add the closing brace. >> > >> > .. literalinclude::/../src/sys/error/err.c >> > :start-at: PetscErrorCode PetscError( >> > :end-at: PetscFunctionReturn(0) >> > :append: } >> >> FWIW, this feels a bit like trying to reinvent WEB. >> > I don't know what WEB is, but if you're saying that this is kinda clunky, > yes it definitely is - my only contention is that it's better than > copy-pasting code and output. I'm not sure if there's an easier and/or > better way with Sphinx. > WEB was the futuristic documentation idea of Don Knuth. Thanks, Matt > >> Doing actual literate documentation of key tutorial programs would be a >> nice way of doing this, but I realise that's a lot more effort. >> > This is still a hope/plan to go into doc/tutorials - follow the deal.ii > model for a small number of key examples. Matt has done a couple of pages > there already, in this direction. > > Lawrence > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>

1 0

Re: [petsc-dev] [DocTip!] #3 CI docs build and preview
by Mark Adams 06 Nov '21

06 Nov '21

This is nice. I see "view latest app" in the pipeline box and you can edit/commit in the browser also. On Sat, Nov 6, 2021 at 10:10 AM Patrick Sanan <patrick.sanan(a)gmail.com> wrote: > All MRs now build all the documentation. This is done with the > "docs-review" job (defined in .gitlab-ci.yml ) . There is special logic > associated with the "docs-only" label: if you add this to your MR, the > usual heavy library tests will be skipped, and the docs will be built > without you having to manually "un-pause" the pipelines each time you > update the MR. > > Jed showed us a nice approach where the results of this build are deployed > to GitLab pages, allowing us to use a "Review App" to see the documentation > build corresponding to the MR branch. > > Practically, this allows you to simply click "view app" on the MR page and > you'll be taken to a complete version of the site (as on petsc.org) > > You may still want to build the docs locally if you want to iterate (more) > quickly, as this allows you to quickly regenerate them after making changes > (as opposed to the many minutes docs-review takes), and keep open a web > browser pointing to your local build. > > Unfortunately, I don't know a simple way to reliably preview a single .rst > file (as you might be used to from working with some Markdown-based tools). >

1 0

Re: [petsc-dev] [DocTip!] #2: Aiming for self-updating docs
by Patrick Sanan 06 Nov '21

06 Nov '21

Am Fr., 5. Nov. 2021 um 19:18 Uhr schrieb Lawrence Mitchell <wence(a)gmx.li>: > > > On 5 Nov 2021, at 15:25, Patrick Sanan <patrick.sanan(a)gmail.com> wrote: > > > > Good question. We don't have any that start at specific line numbers, > currently, as it is indeed too brittle - I suspect I removed some that were > there at one point. > > > > The inclusions essentially all use :start-at: to specify a line to > match. Ideally you'd also use :end-at: to specify where to stop, but some > short snippets do instead specify a number of lines. Note that a > practically handy option is :append:, as in this example where you are > trying to excerpt the function PetscError(), and you can match on the > PetscFunctionReturn(0) and then re-add the closing brace. > > > > .. literalinclude::/../src/sys/error/err.c > > :start-at: PetscErrorCode PetscError( > > :end-at: PetscFunctionReturn(0) > > :append: } > > FWIW, this feels a bit like trying to reinvent WEB. > I don't know what WEB is, but if you're saying that this is kinda clunky, yes it definitely is - my only contention is that it's better than copy-pasting code and output. I'm not sure if there's an easier and/or better way with Sphinx. > > Doing actual literate documentation of key tutorial programs would be a > nice way of doing this, but I realise that's a lot more effort. > This is still a hope/plan to go into doc/tutorials - follow the deal.ii model for a small number of key examples. Matt has done a couple of pages there already, in this direction. Lawrence

1 0

[DocTip!] #3 CI docs build and preview
by Patrick Sanan 06 Nov '21

06 Nov '21

All MRs now build all the documentation. This is done with the "docs-review" job (defined in .gitlab-ci.yml ) . There is special logic associated with the "docs-only" label: if you add this to your MR, the usual heavy library tests will be skipped, and the docs will be built without you having to manually "un-pause" the pipelines each time you update the MR. Jed showed us a nice approach where the results of this build are deployed to GitLab pages, allowing us to use a "Review App" to see the documentation build corresponding to the MR branch. Practically, this allows you to simply click "view app" on the MR page and you'll be taken to a complete version of the site (as on petsc.org) You may still want to build the docs locally if you want to iterate (more) quickly, as this allows you to quickly regenerate them after making changes (as opposed to the many minutes docs-review takes), and keep open a web browser pointing to your local build. Unfortunately, I don't know a simple way to reliably preview a single .rst file (as you might be used to from working with some Markdown-based tools).

1 0

OpenMP
by Mark Adams 06 Nov '21

06 Nov '21

Two questions on OMP: * Can I test for the version of OMP? I want >= 5 and I see this, which looks promising: include/petscsys.h:#elif defined(_OPENMP) && *_OPENMP >= 201307* && !defined(_WIN32) * What is the difference between HAVE_OPENMP and HAVE_OPENMP_SUPPORT. Thanks, Mark

1 0

Re: [petsc-dev] I am getting this error ...
by Mark Adams 05 Nov '21

05 Nov '21

Bingo! On Fri, Nov 5, 2021 at 12:54 PM Satish Balay <balay(a)mcs.anl.gov> wrote: > Yeah remove [C,CPP,CXX,CXXPP,F] FLAGS > > CPP defaults to '$CC -E' - so with "--with-cc='cc -mp=gpu'" - it should > use "cc -mp=gpu -E" > > Satish > > On Fri, 5 Nov 2021, Mark Adams wrote: > > > How about CPP flags? > > > > On Fri, Nov 5, 2021 at 12:29 PM Satish Balay <balay(a)mcs.anl.gov> wrote: > > > > > I guess another way to deal with this is: not use CFLAGS etc.. > > > > > > > > > --with-cc='cc -mp=gpu' --with-cxx='CC -mp=gpu' --with-fc='ftn -mp=gpu' > > > > > > Satish > > > > > > > > > On Fri, 5 Nov 2021, Mark Adams wrote: > > > > > > > Yes, thanks. > > > > I emailed the NERSc person and told him where we are and that we > could > > > fix > > > > this manually, but I don't have this issue on Summit ... > > > > He did not understand for the longest time, not sure he does now, > that we > > > > do not add -gpu. > > > > And I gave him your reproducer. > > > > With any luck, with the reproducer, they can figure this out at some > > > point. > > > > MPI is not working well for my scaling studies and my app has also > given > > > up > > > > on nvhpc, so we are good. > > > > > > > > > > > > On Fri, Nov 5, 2021 at 9:49 AM Satish Balay <balay(a)mcs.anl.gov> > wrote: > > > > > > > > > For now - you could manually edit petscvariables and remove > -mp=gpu > > > from > > > > > it. > > > > > > > > > > Its primarily required to make configure happy. > > > > > > > > > > Satish > > > > > > > > > > > > > > > > > > > > On Thu, 4 Nov 2021, Barry Smith wrote: > > > > > > > > > > > > > > > > > This comes from the persistent problem with PETSc's make > system > > > using > > > > > too many flags for compiling CUDA that have not been tested by > > > configure. > > > > > See below the -mp=gpu is provided probably from the CPPFLAGS or > > > > > CXXPPFLAGS(sp) that is improperly used by the PETSc makefiles to > > > compile > > > > > CUDA code! > > > > > > > > > > > > > > > > > > Using CUDA compile: > > > > > /global/common/software/nersc/cos1.3/cuda/11.3.0/bin/nvcc -o .o > > > > > -I/opt/cray/pe/mpich/8.1.10/ofi/nvidia/20.7/include > > > -I/opt/cray/pe/libsci/ > > > > > 21.08.1.2/NVIDIA/20.7/x86_64/include > -I/opt/cray/pe/pmi/6.0.14/include > > > > > -I/opt/cray/pe/dsmml/0.2.2/dsmml//include > > > > > -I/opt/cray/xpmem/2.2.40-7.0.1.0_3.1__g1d7a24d.shasta/include -g > > > > > -Xcompiler -rdynamic -DLANDAU_DIM=2 -DLANDAU_MAX_SPECIES=10 > > > > > -DLANDAU_MAX_Q=4 -Xcompiler -fPIC -std=c++17 -gencode > > > > > arch=compute_80,code=sm_80 -Wno-deprecated-gpu-targets -c > > > > > --compiler-options=-I/global/homes/m/madams/petsc/include > > > > > > -I/global/homes/m/madams/petsc/arch-perlmutter-opt-nvidia-cuda/include > > > > > -I/global/common/software/nersc/cos1.3/cuda/11.3.0/include -mp=gpu > > > > > -I/global/homes/m/madams/petsc/include > > > > > > -I/global/homes/m/madams/petsc/arch-perlmutter-opt-nvidia-cuda/include > > > > > -I/global/common/software/nersc/cos1.3/cuda/11.3.0/include > > > > > > > > > > > > > > > > > > > > > > > > > On Nov 4, 2021, at 7:31 PM, Mark Adams <mfadams(a)lbl.gov> > wrote: > > > > > > > > > > > > > > OK, configure is done. > > > > > > > Maybe I have too many -mp=gpu > > > > > > > > > > > > > > CUDAC > > > > > > > > > arch-perlmutter-opt-nvidia-cuda/obj/sys/classes/random/impls/curand/curand2.o > > > > > > > gcc: error: unrecognized command line option ‘-mp=gpu’; did you > > > mean > > > > > ‘-mpku’? > > > > > > > > > > > > > > On Thu, Nov 4, 2021 at 5:51 PM Barry Smith <bsmith(a)petsc.dev > > > <mailto: > > > > > bsmith(a)petsc.dev>> wrote: > > > > > > > > > > > > > > Need the same thing for the C++ preprocessor flag > > > > > > > > > > > > > > > > > > > > > > > > > > > >> On Nov 4, 2021, at 5:44 PM, Mark Adams <mfadams(a)lbl.gov > <mailto: > > > > > mfadams(a)lbl.gov>> wrote: > > > > > > >> > > > > > > >> It gets a lot further. > > > > > > >> > > > > > > >> On Thu, Nov 4, 2021 at 5:32 PM Mark Adams <mfadams(a)lbl.gov > > > <mailto: > > > > > mfadams(a)lbl.gov>> wrote: > > > > > > >> OK, sorry I missed the CPPFLAGS. It is running now. > > > > > > >> Thanks, > > > > > > >> > > > > > > >> On Thu, Nov 4, 2021 at 4:43 PM Satish Balay < > balay(a)mcs.anl.gov > > > > > <mailto:[email protected]>> wrote: > > > > > > >> Multiple e-mail threads on the same issue (: > > > > > > >> > > > > > > >> As suggested in my earlier thread - add -mp=gpu to both > CPPFLAGS > > > and > > > > > CFLAGS [or LDFLAGS] > > > > > > >> > > > > > > >> Satish > > > > > > >> > > > > > > >> ------- > > > > > > >> Executing: cc -o > /tmp/petsc-Vvs8_T/config.setCompilers/conftest > > > -g > > > > > -DLANDAU_DIM=2 -DLANDAU_MAX_SPECIES=10 -DLANDAU_MAX_Q=4 > > > > > /tmp/petsc-Vvs8_T/config.setCompilers/conftest.o > > > > > > >> Possible ERROR while running linker: > > > > > > >> stderr: > > > > > > >> nvc-Warning-The -gpu option has no effect unless a > > > language-specific > > > > > option to enable GPU code generation is used (e.g.: -acc, -mp=gpu, > > > -stdpar, > > > > > -cuda) > > > > > > >> > > > > > > >> > > > > > > >> On Thu, 4 Nov 2021, Mark Adams wrote: > > > > > > >> > > > > > > >> > It is CPPFLAGS. I seem to get the same behavior. > > > > > > >> > > > > > > > >> > FWIW, I did get this response from NERSc but I don't know > how to > > > > > interpret > > > > > > >> > it. > > > > > > >> > > > > > > > >> > He seems to be saying that I don't need -mp=gpu for the > device > > > > > compiler > > > > > > >> > (nvcc). He seems to think that I am adding -gpu. > > > > > > >> > > > > > > > >> > nvcc -- The device compiler does not need any of those flags > > > > > because it > > > > > > >> > already knows that it's being fed cuda code. The warning > you're > > > > > seeing is > > > > > > >> > coming from nvc (which is the host / CPU side compiler) if > > > you're > > > > > in the > > > > > > >> > PrgEnv-nvidia environment. You should not need to add > -mp=gpu > > > and > > > > > -cuda, > > > > > > >> > please just add the -cuda flag (to your host code) not to > the > > > > > device code. > > > > > > >> > > > > > > > >> > I will try to talk with this guy again. > > > > > > >> > > > > > > > >> > Thanks, > > > > > > >> > > > > > > > >> > > > > > > > >> > On Thu, Nov 4, 2021 at 4:11 PM Barry Smith < > bsmith(a)petsc.dev > > > > > <mailto:[email protected]>> wrote: > > > > > > >> > > > > > > > >> > > > > > > > > >> > > Yes, you need to use the CPPFLAGS which maybe called > > > CPPCFLAGS > > > > > I am not > > > > > > >> > > sure > > > > > > >> > > > > > > > > >> > > > > > > > > >> > > On Nov 4, 2021, at 3:23 PM, Mark Adams <mfadams(a)lbl.gov > > > <mailto: > > > > > mfadams(a)lbl.gov>> wrote: > > > > > > >> > > > > > > > > >> > > Ah, CCFLAGS does not seem to work. > > > > > > >> > > > > > > > > >> > > On Thu, Nov 4, 2021 at 3:07 PM Barry Smith < > bsmith(a)petsc.dev > > > > > <mailto:[email protected]>> wrote: > > > > > > >> > > > > > > > > >> > >> > > > > > > >> > >> You have to pass in the flag to turn off the bitching > about > > > > > -gpu to the > > > > > > >> > >> C preprocessor, not the C compiler. > > > > > > >> > >> > > > > > > >> > >> > > > > > > >> > >> stderr: > > > > > > >> > >> nvc-Warning-The -gpu option has no effect unless a > > > > > language-specific > > > > > > >> > >> option to enable GPU code generation is used (e.g.: -acc, > > > > > -mp=gpu, -stdpar, > > > > > > >> > >> -cuda) > > > > > > >> > >> Source: > > > > > > >> > >> #include "confdefs.h" > > > > > > >> > >> #include "conffix.h" > > > > > > >> > >> #include <stdlib.h> > > > > > > >> > >> > > > > > > >> > >> > > > > > > >> > >> > > > > > > >> > >> > On Nov 4, 2021, at 2:49 PM, Mark Adams < > mfadams(a)lbl.gov > > > > > <mailto:[email protected]>> wrote: > > > > > > >> > >> > > > > > > > >> > >> > on Perlmutter with nvhpc: > > > > > > >> > >> > > > > > > > >> > >> > Defined make macro "CPP" to "cc --use > cpp32" > > > > > > >> > >> > Preprocessing source: > > > > > > >> > >> > #include "confdefs.h" > > > > > > >> > >> > #include "conffix.h" > > > > > > >> > >> > #include <stdlib.h> > > > > > > >> > >> > > > > > > > >> > >> > Executing: cc --use cpp32 > > > > > -I/tmp/petsc-jV9U1b/config.setCompilers > > > > > > >> > >> /tmp/petsc-jV9U1b/config.setCompilers/conftest.c > > > > > > >> > >> > Possible ERROR while running preprocessor: exit code 1 > > > > > > >> > >> > stderr: > > > > > > >> > >> > nvc-Error-Unknown switch: --use > > > > > > >> > >> > Source: > > > > > > >> > >> > <configure.log> > > > > > > >> > >> > > > > > > >> > >> <configure.log> > > > > > > >> > > > > > > > > >> > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> <configure.log> > > > > > > > > > > > > > > <make.log><configure.log> > > > > > > > > > > > > > > > > > > > > > > > > > > >

1 0

Re: [petsc-dev] [DocTip!] #2: Aiming for self-updating docs
by Lawrence Mitchell 05 Nov '21

05 Nov '21

> On 5 Nov 2021, at 15:25, Patrick Sanan <patrick.sanan(a)gmail.com> wrote: > > Good question. We don't have any that start at specific line numbers, currently, as it is indeed too brittle - I suspect I removed some that were there at one point. > > The inclusions essentially all use :start-at: to specify a line to match. Ideally you'd also use :end-at: to specify where to stop, but some short snippets do instead specify a number of lines. Note that a practically handy option is :append:, as in this example where you are trying to excerpt the function PetscError(), and you can match on the PetscFunctionReturn(0) and then re-add the closing brace. > > .. literalinclude::/../src/sys/error/err.c > :start-at: PetscErrorCode PetscError( > :end-at: PetscFunctionReturn(0) > :append: } FWIW, this feels a bit like trying to reinvent WEB. Doing actual literate documentation of key tutorial programs would be a nice way of doing this, but I realise that's a lot more effort. Lawrence

1 0

Re: [petsc-dev] I am getting this error ...
by Satish Balay 05 Nov '21

05 Nov '21

Yeah remove [C,CPP,CXX,CXXPP,F] FLAGS CPP defaults to '$CC -E' - so with "--with-cc='cc -mp=gpu'" - it should use "cc -mp=gpu -E" Satish On Fri, 5 Nov 2021, Mark Adams wrote: > How about CPP flags? > > On Fri, Nov 5, 2021 at 12:29 PM Satish Balay <balay(a)mcs.anl.gov> wrote: > > > I guess another way to deal with this is: not use CFLAGS etc.. > > > > > > --with-cc='cc -mp=gpu' --with-cxx='CC -mp=gpu' --with-fc='ftn -mp=gpu' > > > > Satish > > > > > > On Fri, 5 Nov 2021, Mark Adams wrote: > > > > > Yes, thanks. > > > I emailed the NERSc person and told him where we are and that we could > > fix > > > this manually, but I don't have this issue on Summit ... > > > He did not understand for the longest time, not sure he does now, that we > > > do not add -gpu. > > > And I gave him your reproducer. > > > With any luck, with the reproducer, they can figure this out at some > > point. > > > MPI is not working well for my scaling studies and my app has also given > > up > > > on nvhpc, so we are good. > > > > > > > > > On Fri, Nov 5, 2021 at 9:49 AM Satish Balay <balay(a)mcs.anl.gov> wrote: > > > > > > > For now - you could manually edit petscvariables and remove -mp=gpu > > from > > > > it. > > > > > > > > Its primarily required to make configure happy. > > > > > > > > Satish > > > > > > > > > > > > > > > > On Thu, 4 Nov 2021, Barry Smith wrote: > > > > > > > > > > > > > > This comes from the persistent problem with PETSc's make system > > using > > > > too many flags for compiling CUDA that have not been tested by > > configure. > > > > See below the -mp=gpu is provided probably from the CPPFLAGS or > > > > CXXPPFLAGS(sp) that is improperly used by the PETSc makefiles to > > compile > > > > CUDA code! > > > > > > > > > > > > > > > Using CUDA compile: > > > > /global/common/software/nersc/cos1.3/cuda/11.3.0/bin/nvcc -o .o > > > > -I/opt/cray/pe/mpich/8.1.10/ofi/nvidia/20.7/include > > -I/opt/cray/pe/libsci/ > > > > 21.08.1.2/NVIDIA/20.7/x86_64/include -I/opt/cray/pe/pmi/6.0.14/include > > > > -I/opt/cray/pe/dsmml/0.2.2/dsmml//include > > > > -I/opt/cray/xpmem/2.2.40-7.0.1.0_3.1__g1d7a24d.shasta/include -g > > > > -Xcompiler -rdynamic -DLANDAU_DIM=2 -DLANDAU_MAX_SPECIES=10 > > > > -DLANDAU_MAX_Q=4 -Xcompiler -fPIC -std=c++17 -gencode > > > > arch=compute_80,code=sm_80 -Wno-deprecated-gpu-targets -c > > > > --compiler-options=-I/global/homes/m/madams/petsc/include > > > > -I/global/homes/m/madams/petsc/arch-perlmutter-opt-nvidia-cuda/include > > > > -I/global/common/software/nersc/cos1.3/cuda/11.3.0/include -mp=gpu > > > > -I/global/homes/m/madams/petsc/include > > > > -I/global/homes/m/madams/petsc/arch-perlmutter-opt-nvidia-cuda/include > > > > -I/global/common/software/nersc/cos1.3/cuda/11.3.0/include > > > > > > > > > > > > > > > > > > > > > On Nov 4, 2021, at 7:31 PM, Mark Adams <mfadams(a)lbl.gov> wrote: > > > > > > > > > > > > OK, configure is done. > > > > > > Maybe I have too many -mp=gpu > > > > > > > > > > > > CUDAC > > > > > > arch-perlmutter-opt-nvidia-cuda/obj/sys/classes/random/impls/curand/curand2.o > > > > > > gcc: error: unrecognized command line option ‘-mp=gpu’; did you > > mean > > > > ‘-mpku’? > > > > > > > > > > > > On Thu, Nov 4, 2021 at 5:51 PM Barry Smith <bsmith(a)petsc.dev > > <mailto: > > > > bsmith(a)petsc.dev>> wrote: > > > > > > > > > > > > Need the same thing for the C++ preprocessor flag > > > > > > > > > > > > > > > > > > > > > > > >> On Nov 4, 2021, at 5:44 PM, Mark Adams <mfadams(a)lbl.gov <mailto: > > > > mfadams(a)lbl.gov>> wrote: > > > > > >> > > > > > >> It gets a lot further. > > > > > >> > > > > > >> On Thu, Nov 4, 2021 at 5:32 PM Mark Adams <mfadams(a)lbl.gov > > <mailto: > > > > mfadams(a)lbl.gov>> wrote: > > > > > >> OK, sorry I missed the CPPFLAGS. It is running now. > > > > > >> Thanks, > > > > > >> > > > > > >> On Thu, Nov 4, 2021 at 4:43 PM Satish Balay <balay(a)mcs.anl.gov > > > > <mailto:[email protected]>> wrote: > > > > > >> Multiple e-mail threads on the same issue (: > > > > > >> > > > > > >> As suggested in my earlier thread - add -mp=gpu to both CPPFLAGS > > and > > > > CFLAGS [or LDFLAGS] > > > > > >> > > > > > >> Satish > > > > > >> > > > > > >> ------- > > > > > >> Executing: cc -o /tmp/petsc-Vvs8_T/config.setCompilers/conftest > > -g > > > > -DLANDAU_DIM=2 -DLANDAU_MAX_SPECIES=10 -DLANDAU_MAX_Q=4 > > > > /tmp/petsc-Vvs8_T/config.setCompilers/conftest.o > > > > > >> Possible ERROR while running linker: > > > > > >> stderr: > > > > > >> nvc-Warning-The -gpu option has no effect unless a > > language-specific > > > > option to enable GPU code generation is used (e.g.: -acc, -mp=gpu, > > -stdpar, > > > > -cuda) > > > > > >> > > > > > >> > > > > > >> On Thu, 4 Nov 2021, Mark Adams wrote: > > > > > >> > > > > > >> > It is CPPFLAGS. I seem to get the same behavior. > > > > > >> > > > > > > >> > FWIW, I did get this response from NERSc but I don't know how to > > > > interpret > > > > > >> > it. > > > > > >> > > > > > > >> > He seems to be saying that I don't need -mp=gpu for the device > > > > compiler > > > > > >> > (nvcc). He seems to think that I am adding -gpu. > > > > > >> > > > > > > >> > nvcc -- The device compiler does not need any of those flags > > > > because it > > > > > >> > already knows that it's being fed cuda code. The warning you're > > > > seeing is > > > > > >> > coming from nvc (which is the host / CPU side compiler) if > > you're > > > > in the > > > > > >> > PrgEnv-nvidia environment. You should not need to add -mp=gpu > > and > > > > -cuda, > > > > > >> > please just add the -cuda flag (to your host code) not to the > > > > device code. > > > > > >> > > > > > > >> > I will try to talk with this guy again. > > > > > >> > > > > > > >> > Thanks, > > > > > >> > > > > > > >> > > > > > > >> > On Thu, Nov 4, 2021 at 4:11 PM Barry Smith <bsmith(a)petsc.dev > > > > <mailto:[email protected]>> wrote: > > > > > >> > > > > > > >> > > > > > > > >> > > Yes, you need to use the CPPFLAGS which maybe called > > CPPCFLAGS > > > > I am not > > > > > >> > > sure > > > > > >> > > > > > > > >> > > > > > > > >> > > On Nov 4, 2021, at 3:23 PM, Mark Adams <mfadams(a)lbl.gov > > <mailto: > > > > mfadams(a)lbl.gov>> wrote: > > > > > >> > > > > > > > >> > > Ah, CCFLAGS does not seem to work. > > > > > >> > > > > > > > >> > > On Thu, Nov 4, 2021 at 3:07 PM Barry Smith <bsmith(a)petsc.dev > > > > <mailto:[email protected]>> wrote: > > > > > >> > > > > > > > >> > >> > > > > > >> > >> You have to pass in the flag to turn off the bitching about > > > > -gpu to the > > > > > >> > >> C preprocessor, not the C compiler. > > > > > >> > >> > > > > > >> > >> > > > > > >> > >> stderr: > > > > > >> > >> nvc-Warning-The -gpu option has no effect unless a > > > > language-specific > > > > > >> > >> option to enable GPU code generation is used (e.g.: -acc, > > > > -mp=gpu, -stdpar, > > > > > >> > >> -cuda) > > > > > >> > >> Source: > > > > > >> > >> #include "confdefs.h" > > > > > >> > >> #include "conffix.h" > > > > > >> > >> #include <stdlib.h> > > > > > >> > >> > > > > > >> > >> > > > > > >> > >> > > > > > >> > >> > On Nov 4, 2021, at 2:49 PM, Mark Adams <mfadams(a)lbl.gov > > > > <mailto:[email protected]>> wrote: > > > > > >> > >> > > > > > > >> > >> > on Perlmutter with nvhpc: > > > > > >> > >> > > > > > > >> > >> > Defined make macro "CPP" to "cc --use cpp32" > > > > > >> > >> > Preprocessing source: > > > > > >> > >> > #include "confdefs.h" > > > > > >> > >> > #include "conffix.h" > > > > > >> > >> > #include <stdlib.h> > > > > > >> > >> > > > > > > >> > >> > Executing: cc --use cpp32 > > > > -I/tmp/petsc-jV9U1b/config.setCompilers > > > > > >> > >> /tmp/petsc-jV9U1b/config.setCompilers/conftest.c > > > > > >> > >> > Possible ERROR while running preprocessor: exit code 1 > > > > > >> > >> > stderr: > > > > > >> > >> > nvc-Error-Unknown switch: --use > > > > > >> > >> > Source: > > > > > >> > >> > <configure.log> > > > > > >> > >> > > > > > >> > >> <configure.log> > > > > > >> > > > > > > > >> > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> <configure.log> > > > > > > > > > > > > <make.log><configure.log> > > > > > > > > > > > > > > > > > > > >

1 0

Re: [petsc-dev] I am getting this error ...
by Mark Adams 05 Nov '21

05 Nov '21

How about CPP flags? On Fri, Nov 5, 2021 at 12:29 PM Satish Balay <balay(a)mcs.anl.gov> wrote: > I guess another way to deal with this is: not use CFLAGS etc.. > > > --with-cc='cc -mp=gpu' --with-cxx='CC -mp=gpu' --with-fc='ftn -mp=gpu' > > Satish > > > On Fri, 5 Nov 2021, Mark Adams wrote: > > > Yes, thanks. > > I emailed the NERSc person and told him where we are and that we could > fix > > this manually, but I don't have this issue on Summit ... > > He did not understand for the longest time, not sure he does now, that we > > do not add -gpu. > > And I gave him your reproducer. > > With any luck, with the reproducer, they can figure this out at some > point. > > MPI is not working well for my scaling studies and my app has also given > up > > on nvhpc, so we are good. > > > > > > On Fri, Nov 5, 2021 at 9:49 AM Satish Balay <balay(a)mcs.anl.gov> wrote: > > > > > For now - you could manually edit petscvariables and remove -mp=gpu > from > > > it. > > > > > > Its primarily required to make configure happy. > > > > > > Satish > > > > > > > > > > > > On Thu, 4 Nov 2021, Barry Smith wrote: > > > > > > > > > > > This comes from the persistent problem with PETSc's make system > using > > > too many flags for compiling CUDA that have not been tested by > configure. > > > See below the -mp=gpu is provided probably from the CPPFLAGS or > > > CXXPPFLAGS(sp) that is improperly used by the PETSc makefiles to > compile > > > CUDA code! > > > > > > > > > > > > Using CUDA compile: > > > /global/common/software/nersc/cos1.3/cuda/11.3.0/bin/nvcc -o .o > > > -I/opt/cray/pe/mpich/8.1.10/ofi/nvidia/20.7/include > -I/opt/cray/pe/libsci/ > > > 21.08.1.2/NVIDIA/20.7/x86_64/include -I/opt/cray/pe/pmi/6.0.14/include > > > -I/opt/cray/pe/dsmml/0.2.2/dsmml//include > > > -I/opt/cray/xpmem/2.2.40-7.0.1.0_3.1__g1d7a24d.shasta/include -g > > > -Xcompiler -rdynamic -DLANDAU_DIM=2 -DLANDAU_MAX_SPECIES=10 > > > -DLANDAU_MAX_Q=4 -Xcompiler -fPIC -std=c++17 -gencode > > > arch=compute_80,code=sm_80 -Wno-deprecated-gpu-targets -c > > > --compiler-options=-I/global/homes/m/madams/petsc/include > > > -I/global/homes/m/madams/petsc/arch-perlmutter-opt-nvidia-cuda/include > > > -I/global/common/software/nersc/cos1.3/cuda/11.3.0/include -mp=gpu > > > -I/global/homes/m/madams/petsc/include > > > -I/global/homes/m/madams/petsc/arch-perlmutter-opt-nvidia-cuda/include > > > -I/global/common/software/nersc/cos1.3/cuda/11.3.0/include > > > > > > > > > > > > > > > > > On Nov 4, 2021, at 7:31 PM, Mark Adams <mfadams(a)lbl.gov> wrote: > > > > > > > > > > OK, configure is done. > > > > > Maybe I have too many -mp=gpu > > > > > > > > > > CUDAC > > > > arch-perlmutter-opt-nvidia-cuda/obj/sys/classes/random/impls/curand/curand2.o > > > > > gcc: error: unrecognized command line option ‘-mp=gpu’; did you > mean > > > ‘-mpku’? > > > > > > > > > > On Thu, Nov 4, 2021 at 5:51 PM Barry Smith <bsmith(a)petsc.dev > <mailto: > > > bsmith(a)petsc.dev>> wrote: > > > > > > > > > > Need the same thing for the C++ preprocessor flag > > > > > > > > > > > > > > > > > > > >> On Nov 4, 2021, at 5:44 PM, Mark Adams <mfadams(a)lbl.gov <mailto: > > > mfadams(a)lbl.gov>> wrote: > > > > >> > > > > >> It gets a lot further. > > > > >> > > > > >> On Thu, Nov 4, 2021 at 5:32 PM Mark Adams <mfadams(a)lbl.gov > <mailto: > > > mfadams(a)lbl.gov>> wrote: > > > > >> OK, sorry I missed the CPPFLAGS. It is running now. > > > > >> Thanks, > > > > >> > > > > >> On Thu, Nov 4, 2021 at 4:43 PM Satish Balay <balay(a)mcs.anl.gov > > > <mailto:[email protected]>> wrote: > > > > >> Multiple e-mail threads on the same issue (: > > > > >> > > > > >> As suggested in my earlier thread - add -mp=gpu to both CPPFLAGS > and > > > CFLAGS [or LDFLAGS] > > > > >> > > > > >> Satish > > > > >> > > > > >> ------- > > > > >> Executing: cc -o /tmp/petsc-Vvs8_T/config.setCompilers/conftest > -g > > > -DLANDAU_DIM=2 -DLANDAU_MAX_SPECIES=10 -DLANDAU_MAX_Q=4 > > > /tmp/petsc-Vvs8_T/config.setCompilers/conftest.o > > > > >> Possible ERROR while running linker: > > > > >> stderr: > > > > >> nvc-Warning-The -gpu option has no effect unless a > language-specific > > > option to enable GPU code generation is used (e.g.: -acc, -mp=gpu, > -stdpar, > > > -cuda) > > > > >> > > > > >> > > > > >> On Thu, 4 Nov 2021, Mark Adams wrote: > > > > >> > > > > >> > It is CPPFLAGS. I seem to get the same behavior. > > > > >> > > > > > >> > FWIW, I did get this response from NERSc but I don't know how to > > > interpret > > > > >> > it. > > > > >> > > > > > >> > He seems to be saying that I don't need -mp=gpu for the device > > > compiler > > > > >> > (nvcc). He seems to think that I am adding -gpu. > > > > >> > > > > > >> > nvcc -- The device compiler does not need any of those flags > > > because it > > > > >> > already knows that it's being fed cuda code. The warning you're > > > seeing is > > > > >> > coming from nvc (which is the host / CPU side compiler) if > you're > > > in the > > > > >> > PrgEnv-nvidia environment. You should not need to add -mp=gpu > and > > > -cuda, > > > > >> > please just add the -cuda flag (to your host code) not to the > > > device code. > > > > >> > > > > > >> > I will try to talk with this guy again. > > > > >> > > > > > >> > Thanks, > > > > >> > > > > > >> > > > > > >> > On Thu, Nov 4, 2021 at 4:11 PM Barry Smith <bsmith(a)petsc.dev > > > <mailto:[email protected]>> wrote: > > > > >> > > > > > >> > > > > > > >> > > Yes, you need to use the CPPFLAGS which maybe called > CPPCFLAGS > > > I am not > > > > >> > > sure > > > > >> > > > > > > >> > > > > > > >> > > On Nov 4, 2021, at 3:23 PM, Mark Adams <mfadams(a)lbl.gov > <mailto: > > > mfadams(a)lbl.gov>> wrote: > > > > >> > > > > > > >> > > Ah, CCFLAGS does not seem to work. > > > > >> > > > > > > >> > > On Thu, Nov 4, 2021 at 3:07 PM Barry Smith <bsmith(a)petsc.dev > > > <mailto:[email protected]>> wrote: > > > > >> > > > > > > >> > >> > > > > >> > >> You have to pass in the flag to turn off the bitching about > > > -gpu to the > > > > >> > >> C preprocessor, not the C compiler. > > > > >> > >> > > > > >> > >> > > > > >> > >> stderr: > > > > >> > >> nvc-Warning-The -gpu option has no effect unless a > > > language-specific > > > > >> > >> option to enable GPU code generation is used (e.g.: -acc, > > > -mp=gpu, -stdpar, > > > > >> > >> -cuda) > > > > >> > >> Source: > > > > >> > >> #include "confdefs.h" > > > > >> > >> #include "conffix.h" > > > > >> > >> #include <stdlib.h> > > > > >> > >> > > > > >> > >> > > > > >> > >> > > > > >> > >> > On Nov 4, 2021, at 2:49 PM, Mark Adams <mfadams(a)lbl.gov > > > <mailto:[email protected]>> wrote: > > > > >> > >> > > > > > >> > >> > on Perlmutter with nvhpc: > > > > >> > >> > > > > > >> > >> > Defined make macro "CPP" to "cc --use cpp32" > > > > >> > >> > Preprocessing source: > > > > >> > >> > #include "confdefs.h" > > > > >> > >> > #include "conffix.h" > > > > >> > >> > #include <stdlib.h> > > > > >> > >> > > > > > >> > >> > Executing: cc --use cpp32 > > > -I/tmp/petsc-jV9U1b/config.setCompilers > > > > >> > >> /tmp/petsc-jV9U1b/config.setCompilers/conftest.c > > > > >> > >> > Possible ERROR while running preprocessor: exit code 1 > > > > >> > >> > stderr: > > > > >> > >> > nvc-Error-Unknown switch: --use > > > > >> > >> > Source: > > > > >> > >> > <configure.log> > > > > >> > >> > > > > >> > >> <configure.log> > > > > >> > > > > > > >> > > > > > > >> > > > > > > >> > > > > > >> > > > > >> <configure.log> > > > > > > > > > > <make.log><configure.log> > > > > > > > > > > > > > >

1 0