August 2021 - petsc-dev - mailman3.cels.anl.gov

Is this a bug in test generation?
by Stefano Zampini 20 Aug '21

20 Aug '21

Scott This test is specified as testset: suffix: expl nsize: {{1 2}} filter: grep -v "MPI processes" | grep -v " type:" | grep -v "Mat Object" args: -ksp_converged_reason -view_explicit_mat -pc_type none -ksp_type {{cg gmres}} test: args: -mat_type aij test: requires: hypre args: -mat_type hypre It triggers an error here https://gitlab.com/petsc/petsc/-/jobs/1519558357 <https://gitlab.com/petsc/petsc/-/jobs/1519558357>, with a CI configuration without hypre. Other testset instances works fine if any “require:" is both specified in the testset body and in the specific test. Is this supposed to work? Thanks Stefano

1 0

I think PetscPreStage() should pass the stage number, not just the time
by Matthew Knepley 18 Aug '21

18 Aug '21

Are there any objections to this before I code it up? Thanks, Matt -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>

1 0

Re: [petsc-dev] PETSC_COMM_WORLD not a PETSc communicator
by Junchao Zhang 16 Aug '21

16 Aug '21

Barry, Thanks for the PetscRegisterFinalize() suggestion. I made an MR at https://gitlab.com/petsc/petsc/-/merge_requests/4238 In rare cases, if I do need to duplicate communicators, I now free them through PetscRegisterFinalize(). --Junchao Zhang On Sun, Aug 15, 2021 at 12:50 PM Barry Smith <bsmith(a)petsc.dev> wrote: > > Junchao, > > Thanks for checking this. > > Could you use PetscRegisterFinalize()? > > Barry > > > On Aug 14, 2021, at 10:46 PM, Junchao Zhang <junchao.zhang(a)gmail.com> > wrote: > > > > On Thu, Aug 12, 2021 at 11:22 AM Barry Smith <bsmith(a)petsc.dev> wrote: > >> >> User visible communicators generally do not have a keyval attached. >> Rather the keyval is attached to the inner communicator; because we don't >> want both PETSc and the user doing MPI operations on the same communicator >> (to prevent mixing up tags). >> >> I think PetscShmCommGet() is wrong. I think it should not call >> MPI_Comm_get_attr(globcomm,Petsc_Counter_keyval,&counter,&flg); but should >> call PetscCommDuplicate() and use that communicator to stash the pshmcomm; >> then you would not have the problem you are having. >> > Barry, I think it over and find the problem is: if I PetscCommDuplicate() > the outer comm in PetscShmCommGet(), then I can not find a place to destroy > the inner communicator (note usually petsc inner communciators are > destroyed with petsc objects. Doing what you said breaks the rule) > > PetscShmCommGet() was designed to help doing OpenMP multithreading on a > communicator that some petsc objects live in. So requiring the input > communicator to be petsc comm is not totally nonsense. > > I tried another approach: Don't check whether the input comm in > PetscShmCommGet(globcomm,&pshmcomm) is an outer comm or an inner comm. We > instead check its Petsc_ShmComm_keyval. If it does not have one, we just > create one for it (along with a new shared memory communicator) > With that, one is able to call PetscShmCommGet(PETSC_COMM_WORLD, ...). > The problem is we attached an attribute to PETSC_COMM_WORLD. It is deleted > inside MPI_Finalize(). PETSc -malloc_dump complains of unfreed memory > (since I used PetscMalloc inside PetscShmCommGet). I could > bypass PetscMalloc and directly use malloc() to avoid this situation. Is > it worthy? > > >> >> >> Barry >> >> >> >> > On Aug 12, 2021, at 11:05 AM, Pierre Jolivet <pierre(a)joliv.et> wrote: >> > >> > Hello, >> > Is there a specific reason why PETSC_COMM_WORLD is not a PETSc >> communicator, i.e., has no Petsc_Counter_keyval attached? >> > ierr = >> PetscOmpCtrlCreate(PETSC_COMM_WORLD,nthreads,&ctrl);CHKERRQ(ierr); >> > yields >> > [0]PETSC ERROR: Bad MPI communicator supplied must be a PETSc >> communicator >> > [0]PETSC ERROR: #1 PetscShmCommGet() at src/sys/utils/mpishm.c:60 >> > [0]PETSC ERROR: #2 PetscOmpCtrlCreate() at src/sys/utils/mpishm.c:340 >> > >> > Thanks, >> > Pierre >> >> >

1 0

Re: [petsc-dev] PETSC_COMM_WORLD not a PETSc communicator
by Barry Smith 15 Aug '21

15 Aug '21

Junchao, Thanks for checking this. Could you use PetscRegisterFinalize()? Barry > On Aug 14, 2021, at 10:46 PM, Junchao Zhang <junchao.zhang(a)gmail.com> wrote: > > > > On Thu, Aug 12, 2021 at 11:22 AM Barry Smith <bsmith(a)petsc.dev <mailto:[email protected]>> wrote: > > User visible communicators generally do not have a keyval attached. Rather the keyval is attached to the inner communicator; because we don't want both PETSc and the user doing MPI operations on the same communicator (to prevent mixing up tags). > > I think PetscShmCommGet() is wrong. I think it should not call MPI_Comm_get_attr(globcomm,Petsc_Counter_keyval,&counter,&flg); but should call PetscCommDuplicate() and use that communicator to stash the pshmcomm; then you would not have the problem you are having. > Barry, I think it over and find the problem is: if I PetscCommDuplicate() the outer comm in PetscShmCommGet(), then I can not find a place to destroy the inner communicator (note usually petsc inner communciators are destroyed with petsc objects. Doing what you said breaks the rule) > > PetscShmCommGet() was designed to help doing OpenMP multithreading on a communicator that some petsc objects live in. So requiring the input communicator to be petsc comm is not totally nonsense. > > I tried another approach: Don't check whether the input comm in PetscShmCommGet(globcomm,&pshmcomm) is an outer comm or an inner comm. We instead check its Petsc_ShmComm_keyval. If it does not have one, we just create one for it (along with a new shared memory communicator) > With that, one is able to call PetscShmCommGet(PETSC_COMM_WORLD, ...). The problem is we attached an attribute to PETSC_COMM_WORLD. It is deleted inside MPI_Finalize(). PETSc -malloc_dump complains of unfreed memory (since I used PetscMalloc inside PetscShmCommGet). I could > bypass PetscMalloc and directly use malloc() to avoid this situation. Is it worthy? > > > > Barry > > > > > On Aug 12, 2021, at 11:05 AM, Pierre Jolivet <pierre(a)joliv.et <mailto:[email protected]>> wrote: > > > > Hello, > > Is there a specific reason why PETSC_COMM_WORLD is not a PETSc communicator, i.e., has no Petsc_Counter_keyval attached? > > ierr = PetscOmpCtrlCreate(PETSC_COMM_WORLD,nthreads,&ctrl);CHKERRQ(ierr); > > yields > > [0]PETSC ERROR: Bad MPI communicator supplied must be a PETSc communicator > > [0]PETSC ERROR: #1 PetscShmCommGet() at src/sys/utils/mpishm.c:60 > > [0]PETSC ERROR: #2 PetscOmpCtrlCreate() at src/sys/utils/mpishm.c:340 > > > > Thanks, > > Pierre >

1 0

Re: [petsc-dev] PETSC_COMM_WORLD not a PETSc communicator
by Junchao Zhang 14 Aug '21

14 Aug '21

On Thu, Aug 12, 2021 at 11:22 AM Barry Smith <bsmith(a)petsc.dev> wrote: > > User visible communicators generally do not have a keyval attached. > Rather the keyval is attached to the inner communicator; because we don't > want both PETSc and the user doing MPI operations on the same communicator > (to prevent mixing up tags). > > I think PetscShmCommGet() is wrong. I think it should not call > MPI_Comm_get_attr(globcomm,Petsc_Counter_keyval,&counter,&flg); but should > call PetscCommDuplicate() and use that communicator to stash the pshmcomm; > then you would not have the problem you are having. > Barry, I think it over and find the problem is: if I PetscCommDuplicate() the outer comm in PetscShmCommGet(), then I can not find a place to destroy the inner communicator (note usually petsc inner communciators are destroyed with petsc objects. Doing what you said breaks the rule) PetscShmCommGet() was designed to help doing OpenMP multithreading on a communicator that some petsc objects live in. So requiring the input communicator to be petsc comm is not totally nonsense. I tried another approach: Don't check whether the input comm in PetscShmCommGet(globcomm,&pshmcomm) is an outer comm or an inner comm. We instead check its Petsc_ShmComm_keyval. If it does not have one, we just create one for it (along with a new shared memory communicator) With that, one is able to call PetscShmCommGet(PETSC_COMM_WORLD, ...). The problem is we attached an attribute to PETSC_COMM_WORLD. It is deleted inside MPI_Finalize(). PETSc -malloc_dump complains of unfreed memory (since I used PetscMalloc inside PetscShmCommGet). I could bypass PetscMalloc and directly use malloc() to avoid this situation. Is it worthy? > > > Barry > > > > > On Aug 12, 2021, at 11:05 AM, Pierre Jolivet <pierre(a)joliv.et> wrote: > > > > Hello, > > Is there a specific reason why PETSC_COMM_WORLD is not a PETSc > communicator, i.e., has no Petsc_Counter_keyval attached? > > ierr = PetscOmpCtrlCreate(PETSC_COMM_WORLD,nthreads,&ctrl);CHKERRQ(ierr); > > yields > > [0]PETSC ERROR: Bad MPI communicator supplied must be a PETSc > communicator > > [0]PETSC ERROR: #1 PetscShmCommGet() at src/sys/utils/mpishm.c:60 > > [0]PETSC ERROR: #2 PetscOmpCtrlCreate() at src/sys/utils/mpishm.c:340 > > > > Thanks, > > Pierre > >

1 0

Re: [petsc-dev] building on Spock
by Junchao Zhang 13 Aug '21

13 Aug '21

Yes, the two files are just petsc headers. --Junchao Zhang On Fri, Aug 13, 2021 at 5:17 PM Mark Adams <mfadams(a)lbl.gov> wrote: > I seem to be getting Kokkos includes in my install but there is no kokkos > in the configure and I started with a clean PETSc arch directory and > install directory. Does this make sense? > > 18:12 main= /gpfs/alpine/csc314/scratch/adams/petsc$ ll > /gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-opt-cray/include/ > | grep -i kokko > -rw-rw-r-- 1 adams adams 7074 May 27 06:57 petscdmda_kokkos.hpp > -rw-rw-r-- 1 adams adams 1636 May 27 06:57 petscvec_kokkos.hpp > 18:12 main= /gpfs/alpine/csc314/scratch/adams/petsc$ > > > On Fri, Aug 13, 2021 at 2:27 PM Mark Adams <mfadams(a)lbl.gov> wrote: > >> >> >> On Fri, Aug 13, 2021 at 2:01 PM Matthew Knepley <knepley(a)gmail.com> >> wrote: >> >>> On Fri, Aug 13, 2021 at 1:49 PM Mark Adams <mfadams(a)lbl.gov> wrote: >>> >>>> I was building on Spock a few weeks ago, but am getting these errors >>>> now. >>>> I have a setup with this environment and get this error. >>>> >>>> Any ideas? >>>> >>> >>> Here is the error: >>> >>> Executing: cc -o /tmp/petsc-6izfgpkr/config.setCompilers/conftest >>> -L/opt/rocm-4.2.0/lib -lhsa-runtime64 -L${ROCM_PATH}/lib -lamdhip64 >>> -lhsa-runtime64 /tmp/petsc-6izfgpkr/config.setCompilers/conftest.o >>> Possible ERROR while running linker: exit code 1 >>> stderr: >>> ld.lld: error: unable to find library -lmpi_gtl_hsa >>> clang-12: error: linker command failed with exit code 1 (use -v to see >>> invocation) >>> Linker output before filtering: >>> >>> ld.lld: error: unable to find library -lmpi_gtl_hsa >>> clang-12: error: linker command failed with exit code 1 (use -v to see >>> invocation) >>> : >>> Linker output after filtering: >>> >>> ld.lld: error: unable to find library -lmpi_gtl_hsa >>> clang-12: error: linker command failed with exit code 1 (use -v to see >>> invocation): >>> Error testing C compiler: Cannot compile/link C with cc. >>> Deleting "CC" >>> >>> That -l look like it is put in directly by 'cc'. How does anything work >>> on that machine? >>> >> >> It looks like this comes from an env variable that I have: >> >> export PE_MPICH_GTL_LIBS_amd_gfx908="-lmpi_gtl_hsa" >> >> >>> >>> Matt >>> >>> >>>> Thanks, >>>> Mark >>>> >>>> module load craype-accel-amd-gfx908 >>>> module load rocm >>>> module load emacs >>>> module load zlib >>>> module load autoconf automake libtool >>>> >>>> ## These must be set before compiling so the executable picks up GTL >>>> export >>>> PE_MPICH_GTL_DIR_amd_gfx908="-L/opt/cray/pe/mpich/8.1.4/gtl/lib" >>>> export PE_MPICH_GTL_LIBS_amd_gfx908="-lmpi_gtl_hsa" >>>> >>>> ## These must be set before running >>>> export MPIR_CVAR_GPU_EAGER_DEVICE_MEM=0 >>>> export MPICH_GPU_SUPPORT_ENABLED=1 >>>> export MPICH_SMP_SINGLE_COPY_MODE=CMA >>>> >>> >>> >>> -- >>> What most experimenters take for granted before they begin their >>> experiments is infinitely more interesting than any results to which their >>> experiments lead. >>> -- Norbert Wiener >>> >>> https://www.cse.buffalo.edu/~knepley/ >>> <http://www.cse.buffalo.edu/~knepley/> >>> >>

1 0

Re: [petsc-dev] building on Spock
by Mark Adams 13 Aug '21

13 Aug '21

I seem to be getting Kokkos includes in my install but there is no kokkos in the configure and I started with a clean PETSc arch directory and install directory. Does this make sense? 18:12 main= /gpfs/alpine/csc314/scratch/adams/petsc$ ll /gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-opt-cray/include/ | grep -i kokko -rw-rw-r-- 1 adams adams 7074 May 27 06:57 petscdmda_kokkos.hpp -rw-rw-r-- 1 adams adams 1636 May 27 06:57 petscvec_kokkos.hpp 18:12 main= /gpfs/alpine/csc314/scratch/adams/petsc$ On Fri, Aug 13, 2021 at 2:27 PM Mark Adams <mfadams(a)lbl.gov> wrote: > > > On Fri, Aug 13, 2021 at 2:01 PM Matthew Knepley <knepley(a)gmail.com> wrote: > >> On Fri, Aug 13, 2021 at 1:49 PM Mark Adams <mfadams(a)lbl.gov> wrote: >> >>> I was building on Spock a few weeks ago, but am getting these errors >>> now. >>> I have a setup with this environment and get this error. >>> >>> Any ideas? >>> >> >> Here is the error: >> >> Executing: cc -o /tmp/petsc-6izfgpkr/config.setCompilers/conftest >> -L/opt/rocm-4.2.0/lib -lhsa-runtime64 -L${ROCM_PATH}/lib -lamdhip64 >> -lhsa-runtime64 /tmp/petsc-6izfgpkr/config.setCompilers/conftest.o >> Possible ERROR while running linker: exit code 1 >> stderr: >> ld.lld: error: unable to find library -lmpi_gtl_hsa >> clang-12: error: linker command failed with exit code 1 (use -v to see >> invocation) >> Linker output before filtering: >> >> ld.lld: error: unable to find library -lmpi_gtl_hsa >> clang-12: error: linker command failed with exit code 1 (use -v to see >> invocation) >> : >> Linker output after filtering: >> >> ld.lld: error: unable to find library -lmpi_gtl_hsa >> clang-12: error: linker command failed with exit code 1 (use -v to see >> invocation): >> Error testing C compiler: Cannot compile/link C with cc. >> Deleting "CC" >> >> That -l look like it is put in directly by 'cc'. How does anything work >> on that machine? >> > > It looks like this comes from an env variable that I have: > > export PE_MPICH_GTL_LIBS_amd_gfx908="-lmpi_gtl_hsa" > > >> >> Matt >> >> >>> Thanks, >>> Mark >>> >>> module load craype-accel-amd-gfx908 >>> module load rocm >>> module load emacs >>> module load zlib >>> module load autoconf automake libtool >>> >>> ## These must be set before compiling so the executable picks up GTL >>> export >>> PE_MPICH_GTL_DIR_amd_gfx908="-L/opt/cray/pe/mpich/8.1.4/gtl/lib" >>> export PE_MPICH_GTL_LIBS_amd_gfx908="-lmpi_gtl_hsa" >>> >>> ## These must be set before running >>> export MPIR_CVAR_GPU_EAGER_DEVICE_MEM=0 >>> export MPICH_GPU_SUPPORT_ENABLED=1 >>> export MPICH_SMP_SINGLE_COPY_MODE=CMA >>> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> >> https://www.cse.buffalo.edu/~knepley/ >> <http://www.cse.buffalo.edu/~knepley/> >> >

1 0

Re: [petsc-dev] building on Spock
by Mark Adams 13 Aug '21

13 Aug '21

On Fri, Aug 13, 2021 at 2:25 PM Aaron Scheinberg <aaron(a)jubileedev.com> wrote: > Hey Mark, I believe the problem is in this line: > > export PE_MPICH_GTL_DIR_amd_gfx908="-L/opt/cray/pe/mpich/8.1.4/gtl/lib" > "8.1.4" should be replaced with "default" since I think that version is in > flux. > Good catch. Its running now. > > On Fri, Aug 13, 2021 at 1:49 PM Mark Adams <mfadams(a)lbl.gov> wrote: > >> I was building on Spock a few weeks ago, but am getting these errors now. >> I have a setup with this environment and get this error. >> >> Any ideas? >> >> Thanks, >> Mark >> >> module load craype-accel-amd-gfx908 >> module load rocm >> module load emacs >> module load zlib >> module load autoconf automake libtool >> >> ## These must be set before compiling so the executable picks up GTL >> export >> PE_MPICH_GTL_DIR_amd_gfx908="-L/opt/cray/pe/mpich/8.1.4/gtl/lib" >> export PE_MPICH_GTL_LIBS_amd_gfx908="-lmpi_gtl_hsa" >> >> ## These must be set before running >> export MPIR_CVAR_GPU_EAGER_DEVICE_MEM=0 >> export MPICH_GPU_SUPPORT_ENABLED=1 >> export MPICH_SMP_SINGLE_COPY_MODE=CMA >> >

1 0

Re: [petsc-dev] building on Spock
by Mark Adams 13 Aug '21

13 Aug '21

On Fri, Aug 13, 2021 at 2:01 PM Matthew Knepley <knepley(a)gmail.com> wrote: > On Fri, Aug 13, 2021 at 1:49 PM Mark Adams <mfadams(a)lbl.gov> wrote: > >> I was building on Spock a few weeks ago, but am getting these errors now. >> I have a setup with this environment and get this error. >> >> Any ideas? >> > > Here is the error: > > Executing: cc -o /tmp/petsc-6izfgpkr/config.setCompilers/conftest > -L/opt/rocm-4.2.0/lib -lhsa-runtime64 -L${ROCM_PATH}/lib -lamdhip64 > -lhsa-runtime64 /tmp/petsc-6izfgpkr/config.setCompilers/conftest.o > Possible ERROR while running linker: exit code 1 > stderr: > ld.lld: error: unable to find library -lmpi_gtl_hsa > clang-12: error: linker command failed with exit code 1 (use -v to see > invocation) > Linker output before filtering: > > ld.lld: error: unable to find library -lmpi_gtl_hsa > clang-12: error: linker command failed with exit code 1 (use -v to see > invocation) > : > Linker output after filtering: > > ld.lld: error: unable to find library -lmpi_gtl_hsa > clang-12: error: linker command failed with exit code 1 (use -v to see > invocation): > Error testing C compiler: Cannot compile/link C with cc. > Deleting "CC" > > That -l look like it is put in directly by 'cc'. How does anything work on > that machine? > It looks like this comes from an env variable that I have: export PE_MPICH_GTL_LIBS_amd_gfx908="-lmpi_gtl_hsa" > > Matt > > >> Thanks, >> Mark >> >> module load craype-accel-amd-gfx908 >> module load rocm >> module load emacs >> module load zlib >> module load autoconf automake libtool >> >> ## These must be set before compiling so the executable picks up GTL >> export >> PE_MPICH_GTL_DIR_amd_gfx908="-L/opt/cray/pe/mpich/8.1.4/gtl/lib" >> export PE_MPICH_GTL_LIBS_amd_gfx908="-lmpi_gtl_hsa" >> >> ## These must be set before running >> export MPIR_CVAR_GPU_EAGER_DEVICE_MEM=0 >> export MPICH_GPU_SUPPORT_ENABLED=1 >> export MPICH_SMP_SINGLE_COPY_MODE=CMA >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > <http://www.cse.buffalo.edu/~knepley/> >

1 0

Re: [petsc-dev] building on Spock
by Aaron Scheinberg 13 Aug '21

13 Aug '21

Hey Mark, I believe the problem is in this line: export PE_MPICH_GTL_DIR_amd_gfx908="-L/opt/cray/pe/mpich/8.1.4/gtl/lib" "8.1.4" should be replaced with "default" since I think that version is in flux. On Fri, Aug 13, 2021 at 1:49 PM Mark Adams <mfadams(a)lbl.gov> wrote: > I was building on Spock a few weeks ago, but am getting these errors now. > I have a setup with this environment and get this error. > > Any ideas? > > Thanks, > Mark > > module load craype-accel-amd-gfx908 > module load rocm > module load emacs > module load zlib > module load autoconf automake libtool > > ## These must be set before compiling so the executable picks up GTL > export PE_MPICH_GTL_DIR_amd_gfx908="-L/opt/cray/pe/mpich/8.1.4/gtl/lib" > export PE_MPICH_GTL_LIBS_amd_gfx908="-lmpi_gtl_hsa" > > ## These must be set before running > export MPIR_CVAR_GPU_EAGER_DEVICE_MEM=0 > export MPICH_GPU_SUPPORT_ENABLED=1 > export MPICH_SMP_SINGLE_COPY_MODE=CMA >

1 0