September 2021 - petsc-dev - mailman3.cels.anl.gov

Re: [petsc-dev] Can someone help with this failure?
by Jacob Faibussowitsch 27 Sep '21

27 Sep '21

TL;DR: don’t use generalized unpacking here: !2, 3.5 /home/glci/builds-stage1/ytsywSZu/0/petsc/petsc/config/BuildSystem/config/packages/tchem.py generalized unpacking requires 3.5+ ——— vermin is a python package that checks which python version is required to run a particular python script. Petsc targets python 2.6 or lower and 3.4 or lower, however your change would make python 3.5 the new minimum: Minimum required versions: 3.5 Incompatible versions: 2 Target versions not met: 2.6-, 3.4- ctrl-F for “3.5” will find the file and reason in the output. Best regards, Jacob Faibussowitsch (Jacob Fai - booss - oh - vitch) > On Sep 27, 2021, at 10:04, Matthew Knepley <knepley(a)gmail.com> wrote: > > I do not understand what is being tested or where the failure lies? > > https://gitlab.com/petsc/petsc/-/jobs/1626183341 <https://gitlab.com/petsc/petsc/-/jobs/1626183341> > > Thanks, > > Matt > > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>

1 0

Re: [petsc-dev] Can someone help with this failure?
by Satish Balay 27 Sep '21

27 Sep '21

>>> $ vermin -vv -t=2.6- -t=3.4- config Minimum required versions: 3.5 !2, 3.5 /home/glci/builds-stage1/ytsywSZu/0/petsc/petsc/config/BuildSystem/config/packages/tchem.py generalized unpacking requires 3.5+ <<< Our current min requirements are python-2.6 & 3.4 And tchem.py appears to have python-3.5 code - so this test is failing [for both python2 and python3 requirements] Satish On Mon, 27 Sep 2021, Matthew Knepley wrote: > I do not understand what is being tested or where the failure lies? > > https://gitlab.com/petsc/petsc/-/jobs/1626183341 > > Thanks, > > Matt > >

1 0

Can someone help with this failure?
by Matthew Knepley 27 Sep '21

27 Sep '21

I do not understand what is being tested or where the failure lies? https://gitlab.com/petsc/petsc/-/jobs/1626183341 Thanks, Matt -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>

1 0

Re: [petsc-dev] cuda-memcheck finds an error on Summit
by Mark Adams 26 Sep '21

26 Sep '21

The test do run, but I noticed in my logs that VecCopy was not done on the GPU. I looked at the code and VecCopy_SeqCUDA is there. It looks like it should be used. So I put a breakpoint in DDT but ddt failed when it called some simple CUDA method in PetscInitialize. cuda-memcheck fails as I showed. Valgrind also has all kinds of errors but it runs. I am thinking the libs are messed up. I am trying an older CUDA now. I might try nvhpc next. Thanks, Mark On Sun, Sep 26, 2021 at 4:53 PM Junchao Zhang <junchao.zhang(a)gmail.com> wrote: > Mark, > without cuda-memcheck, did the test run? > --Junchao Zhang > > > On Sun, Sep 26, 2021 at 12:38 PM Mark Adams <mfadams(a)lbl.gov> wrote: > >> FYI, I am getting this with cuda-memcheck on Summit with CUDA 11.0.3: >> >> jsrun -n 48 -a 6 -c 6 -g 1 -r 6 --smpiargs -gpu cuda-memcheck ../ex13-cu >> -dm_plex_box_faces 4,6,12 -petscpartitioner_simple_node_grid 2,2,2 >> -dm_plex_box_upper 2,3,6 -petscpartitioner_simple_process_grid 2,3,6 >> -dm_refine 3 -dm_mat_type aijcusparse -dm_vec_type cuda -dm_view >> >> This job runs with Kokkos and it runs with these 8 nodes with -dm_refine >> 2 instead of 3. And it runs with the 1 node version of this test. >> >> Thanks, >> Mark >> >> [8]PETSC ERROR: --------------------- Error Message >> -------------------------------------------------------------- >> [8]PETSC ERROR: GPU error >> [8]PETSC ERROR: cuda error 715 (cudaErrorIllegalInstruction) : an illegal >> instruction was encountered >> [8]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting. >> [8]PETSC ERROR: Petsc Development GIT revision: v3.15.4-943-g83e1f11c26 >> GIT Date: 2021-09-25 18:33:40 -0400 >> [8]PETSC ERROR: ../ex13-cu on a arch-summit-opt-gnu-cuda named f10n17 by >> Unknown Sun Sep 26 13:29:03 2021 >> [8]PETSC ERROR: Configure options --with-fc=mpifort --with-cc=mpicc >> --with-cxx=mpiCC --CFLAGS="-fPIC -g -DLANDAU_DIM=2 -DLANDAU_MAX_SPECIES=10 >> -DLANDAU_MAX_Q=4" --CXXFLAGS="-fPIC -g -DLANDAU_DIM=2 >> -DLANDAU_MAX_SPECIES=10 -DLANDAU_MAX_Q=4" --CUDAFLAGS="-g -Xcompiler >> -DLANDAU_DIM=2 -DLANDAU_MAX_SPECIES=10 -DLANDAU_MAX_Q=4" --FCFLAGS="-fPIC >> -g" --COPTFLAGS=-O3 --CXXOPTFLAGS=-O3 --with-ssl=0 --with-batch=0 >> --with-mpiexec="jsrun -g1" --with-cuda=1 --with-cudac=nvcc >> --with-cuda-arch=70 --download-p4est=1 --download-zlib >> --with-blaslapack-lib="-L/sw/summit/spack-envs/base/opt/linux-rhel8-ppc64le/gcc-9.1.0/netlib-lapack-3.9.1-e6vxode53ghsjrop2dfwlq77s3dvkr7t/lib64 >> -lblas -llapack" --with-x=0 --with-64-bit-indices=0 --with-debugging=0 >> PETSC_ARCH=arch-summit-opt-gnu-cuda >> [8]PETSC ERROR: #1 PetscSFLinkSyncStream_CUDA() at >> /gpfs/alpine/csc314/scratch/adams/petsc/src/vec/is/sf/impls/basic/cuda/ >> sfcuda.cu:872 >> [8]PETSC ERROR: #2 PetscSFLinkSyncStreamBeforeCallMPI() at >> /gpfs/alpine/csc314/scratch/adams/petsc/include/../src/vec/is/sf/impls/basic/sfpack.h:344 >> [8]PETSC ERROR: #3 PetscSFLinkStartRequests_MPI() at >> /gpfs/alpine/csc314/scratch/adams/petsc/src/vec/is/sf/impls/basic/sfmpi.c:40 >> [8]PETSC ERROR: #4 PetscSFLinkStartCommunication() at >> /gpfs/alpine/csc314/scratch/adams/petsc/include/../src/vec/is/sf/impls/basic/sfpack.h:267 >> [8]PETSC ERROR: #5 PetscSFBcastBegin_Basic() at >> /gpfs/alpine/csc314/scratch/adams/petsc/src/vec/is/sf/impls/basic/sfbasic.c:191 >> [8]PETSC ERROR: #6 PetscSFBcastWithMemTypeBegin() at >> /gpfs/alpine/csc314/scratch/adams/petsc/src/vec/is/sf/interface/sf.c:1493 >> [8]PETSC ERROR: #7 DMGlobalToLocalBegin() at >> /gpfs/alpine/csc314/scratch/adams/petsc/src/dm/interface/dm.c:2613 >> [8]PETSC ERROR: #8 SNESComputeJacobian_DMLocal() at >> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/utils/dmlocalsnes.c:119 >> [8]PETSC ERROR: #9 SNESComputeJacobian() at >> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/interface/snes.c:2824 >> [8]PETSC ERROR: #10 SNESSolve_KSPONLY() at >> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/impls/ksponly/ksponly.c:43 >> [8]PETSC ERROR: #11 SNESSolve() at >> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/interface/snes.c:4769 >> [8]PETSC ERROR: #12 main() at ex13.c:169 >> >

1 0

Re: [petsc-dev] cuda-memcheck finds an error on Summit
by Junchao Zhang 26 Sep '21

26 Sep '21

Mark, without cuda-memcheck, did the test run? --Junchao Zhang On Sun, Sep 26, 2021 at 12:38 PM Mark Adams <mfadams(a)lbl.gov> wrote: > FYI, I am getting this with cuda-memcheck on Summit with CUDA 11.0.3: > > jsrun -n 48 -a 6 -c 6 -g 1 -r 6 --smpiargs -gpu cuda-memcheck ../ex13-cu > -dm_plex_box_faces 4,6,12 -petscpartitioner_simple_node_grid 2,2,2 > -dm_plex_box_upper 2,3,6 -petscpartitioner_simple_process_grid 2,3,6 > -dm_refine 3 -dm_mat_type aijcusparse -dm_vec_type cuda -dm_view > > This job runs with Kokkos and it runs with these 8 nodes with -dm_refine 2 > instead of 3. And it runs with the 1 node version of this test. > > Thanks, > Mark > > [8]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > [8]PETSC ERROR: GPU error > [8]PETSC ERROR: cuda error 715 (cudaErrorIllegalInstruction) : an illegal > instruction was encountered > [8]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting. > [8]PETSC ERROR: Petsc Development GIT revision: v3.15.4-943-g83e1f11c26 > GIT Date: 2021-09-25 18:33:40 -0400 > [8]PETSC ERROR: ../ex13-cu on a arch-summit-opt-gnu-cuda named f10n17 by > Unknown Sun Sep 26 13:29:03 2021 > [8]PETSC ERROR: Configure options --with-fc=mpifort --with-cc=mpicc > --with-cxx=mpiCC --CFLAGS="-fPIC -g -DLANDAU_DIM=2 -DLANDAU_MAX_SPECIES=10 > -DLANDAU_MAX_Q=4" --CXXFLAGS="-fPIC -g -DLANDAU_DIM=2 > -DLANDAU_MAX_SPECIES=10 -DLANDAU_MAX_Q=4" --CUDAFLAGS="-g -Xcompiler > -DLANDAU_DIM=2 -DLANDAU_MAX_SPECIES=10 -DLANDAU_MAX_Q=4" --FCFLAGS="-fPIC > -g" --COPTFLAGS=-O3 --CXXOPTFLAGS=-O3 --with-ssl=0 --with-batch=0 > --with-mpiexec="jsrun -g1" --with-cuda=1 --with-cudac=nvcc > --with-cuda-arch=70 --download-p4est=1 --download-zlib > --with-blaslapack-lib="-L/sw/summit/spack-envs/base/opt/linux-rhel8-ppc64le/gcc-9.1.0/netlib-lapack-3.9.1-e6vxode53ghsjrop2dfwlq77s3dvkr7t/lib64 > -lblas -llapack" --with-x=0 --with-64-bit-indices=0 --with-debugging=0 > PETSC_ARCH=arch-summit-opt-gnu-cuda > [8]PETSC ERROR: #1 PetscSFLinkSyncStream_CUDA() at > /gpfs/alpine/csc314/scratch/adams/petsc/src/vec/is/sf/impls/basic/cuda/ > sfcuda.cu:872 > [8]PETSC ERROR: #2 PetscSFLinkSyncStreamBeforeCallMPI() at > /gpfs/alpine/csc314/scratch/adams/petsc/include/../src/vec/is/sf/impls/basic/sfpack.h:344 > [8]PETSC ERROR: #3 PetscSFLinkStartRequests_MPI() at > /gpfs/alpine/csc314/scratch/adams/petsc/src/vec/is/sf/impls/basic/sfmpi.c:40 > [8]PETSC ERROR: #4 PetscSFLinkStartCommunication() at > /gpfs/alpine/csc314/scratch/adams/petsc/include/../src/vec/is/sf/impls/basic/sfpack.h:267 > [8]PETSC ERROR: #5 PetscSFBcastBegin_Basic() at > /gpfs/alpine/csc314/scratch/adams/petsc/src/vec/is/sf/impls/basic/sfbasic.c:191 > [8]PETSC ERROR: #6 PetscSFBcastWithMemTypeBegin() at > /gpfs/alpine/csc314/scratch/adams/petsc/src/vec/is/sf/interface/sf.c:1493 > [8]PETSC ERROR: #7 DMGlobalToLocalBegin() at > /gpfs/alpine/csc314/scratch/adams/petsc/src/dm/interface/dm.c:2613 > [8]PETSC ERROR: #8 SNESComputeJacobian_DMLocal() at > /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/utils/dmlocalsnes.c:119 > [8]PETSC ERROR: #9 SNESComputeJacobian() at > /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/interface/snes.c:2824 > [8]PETSC ERROR: #10 SNESSolve_KSPONLY() at > /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/impls/ksponly/ksponly.c:43 > [8]PETSC ERROR: #11 SNESSolve() at > /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/interface/snes.c:4769 > [8]PETSC ERROR: #12 main() at ex13.c:169 >

1 0

cuda-memcheck finds an error on Summit
by Mark Adams 26 Sep '21

26 Sep '21

FYI, I am getting this with cuda-memcheck on Summit with CUDA 11.0.3: jsrun -n 48 -a 6 -c 6 -g 1 -r 6 --smpiargs -gpu cuda-memcheck ../ex13-cu -dm_plex_box_faces 4,6,12 -petscpartitioner_simple_node_grid 2,2,2 -dm_plex_box_upper 2,3,6 -petscpartitioner_simple_process_grid 2,3,6 -dm_refine 3 -dm_mat_type aijcusparse -dm_vec_type cuda -dm_view This job runs with Kokkos and it runs with these 8 nodes with -dm_refine 2 instead of 3. And it runs with the 1 node version of this test. Thanks, Mark [8]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [8]PETSC ERROR: GPU error [8]PETSC ERROR: cuda error 715 (cudaErrorIllegalInstruction) : an illegal instruction was encountered [8]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting. [8]PETSC ERROR: Petsc Development GIT revision: v3.15.4-943-g83e1f11c26 GIT Date: 2021-09-25 18:33:40 -0400 [8]PETSC ERROR: ../ex13-cu on a arch-summit-opt-gnu-cuda named f10n17 by Unknown Sun Sep 26 13:29:03 2021 [8]PETSC ERROR: Configure options --with-fc=mpifort --with-cc=mpicc --with-cxx=mpiCC --CFLAGS="-fPIC -g -DLANDAU_DIM=2 -DLANDAU_MAX_SPECIES=10 -DLANDAU_MAX_Q=4" --CXXFLAGS="-fPIC -g -DLANDAU_DIM=2 -DLANDAU_MAX_SPECIES=10 -DLANDAU_MAX_Q=4" --CUDAFLAGS="-g -Xcompiler -DLANDAU_DIM=2 -DLANDAU_MAX_SPECIES=10 -DLANDAU_MAX_Q=4" --FCFLAGS="-fPIC -g" --COPTFLAGS=-O3 --CXXOPTFLAGS=-O3 --with-ssl=0 --with-batch=0 --with-mpiexec="jsrun -g1" --with-cuda=1 --with-cudac=nvcc --with-cuda-arch=70 --download-p4est=1 --download-zlib --with-blaslapack-lib="-L/sw/summit/spack-envs/base/opt/linux-rhel8-ppc64le/gcc-9.1.0/netlib-lapack-3.9.1-e6vxode53ghsjrop2dfwlq77s3dvkr7t/lib64 -lblas -llapack" --with-x=0 --with-64-bit-indices=0 --with-debugging=0 PETSC_ARCH=arch-summit-opt-gnu-cuda [8]PETSC ERROR: #1 PetscSFLinkSyncStream_CUDA() at /gpfs/alpine/csc314/scratch/adams/petsc/src/vec/is/sf/impls/basic/cuda/ sfcuda.cu:872 [8]PETSC ERROR: #2 PetscSFLinkSyncStreamBeforeCallMPI() at /gpfs/alpine/csc314/scratch/adams/petsc/include/../src/vec/is/sf/impls/basic/sfpack.h:344 [8]PETSC ERROR: #3 PetscSFLinkStartRequests_MPI() at /gpfs/alpine/csc314/scratch/adams/petsc/src/vec/is/sf/impls/basic/sfmpi.c:40 [8]PETSC ERROR: #4 PetscSFLinkStartCommunication() at /gpfs/alpine/csc314/scratch/adams/petsc/include/../src/vec/is/sf/impls/basic/sfpack.h:267 [8]PETSC ERROR: #5 PetscSFBcastBegin_Basic() at /gpfs/alpine/csc314/scratch/adams/petsc/src/vec/is/sf/impls/basic/sfbasic.c:191 [8]PETSC ERROR: #6 PetscSFBcastWithMemTypeBegin() at /gpfs/alpine/csc314/scratch/adams/petsc/src/vec/is/sf/interface/sf.c:1493 [8]PETSC ERROR: #7 DMGlobalToLocalBegin() at /gpfs/alpine/csc314/scratch/adams/petsc/src/dm/interface/dm.c:2613 [8]PETSC ERROR: #8 SNESComputeJacobian_DMLocal() at /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/utils/dmlocalsnes.c:119 [8]PETSC ERROR: #9 SNESComputeJacobian() at /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/interface/snes.c:2824 [8]PETSC ERROR: #10 SNESSolve_KSPONLY() at /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/impls/ksponly/ksponly.c:43 [8]PETSC ERROR: #11 SNESSolve() at /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/interface/snes.c:4769 [8]PETSC ERROR: #12 main() at ex13.c:169

1 0

Re: [petsc-dev] cuSparse vector performance issue
by Mark Adams 25 Sep '21

25 Sep '21

On Sat, Sep 25, 2021 at 8:12 PM Junchao Zhang <junchao.zhang(a)gmail.com> wrote: > > > > On Sat, Sep 25, 2021 at 4:45 PM Mark Adams <mfadams(a)lbl.gov> wrote: > >> I am testing my Landau code, which is MPI serial, but with many >> independent MPI processes driving each GPU, in an MPI parallel harness code >> (Landau ex2). >> >> Vector operations with Kokkos Kernels and cuSparse are about the same (KK >> is faster) and a bit expensive with one process / GPU. About the same as my >> Jacobian construction, which is expensive but optimized on the GPU. (I am >> using arkimex adaptive TS. I am guessing that it does a lot of vector ops, >> because there are a lot.) >> >> With 14 or 15 processes, all doing the same MPI serial problem, cuSparse >> is about 2.5x more expensive than KK. KK does degrad by about 15% from the >> one processor case. So KK is doing fine, but something bad is >> happening with cuSparse. >> > AIJKOKKOS and AIJCUSPARSE have different algorithms? I don't know. To > know exactly, the best approach is to consult with Peng@nvidia to profile > the code. > Yea, I could ask Peng if he has any thoughts. I am also now having a problem with snes/tests/ex13 scaling study (for my ECP report). The cuSparse version of GAMG is hanging on an 8 node job with a refinement of 3. It works on one node with a refinement of 4 and on 8 nodes with a refinement of 2. I recently moved from CUDA-10 to CUDA-11 on summit because MPS seems to be working with CUDA-11 whereas it was not a while ago. I think I will try going back to CUDA-10 and see if I see anything change. Thanks, Mark > > >> >> Anyone have any thoughts on this? >> >> Thanks, >> Mark >> >>

1 0

Re: [petsc-dev] cuSparse vector performance issue
by Junchao Zhang 25 Sep '21

25 Sep '21

On Sat, Sep 25, 2021 at 4:45 PM Mark Adams <mfadams(a)lbl.gov> wrote: > I am testing my Landau code, which is MPI serial, but with many > independent MPI processes driving each GPU, in an MPI parallel harness code > (Landau ex2). > > Vector operations with Kokkos Kernels and cuSparse are about the same (KK > is faster) and a bit expensive with one process / GPU. About the same as my > Jacobian construction, which is expensive but optimized on the GPU. (I am > using arkimex adaptive TS. I am guessing that it does a lot of vector ops, > because there are a lot.) > > With 14 or 15 processes, all doing the same MPI serial problem, cuSparse > is about 2.5x more expensive than KK. KK does degrad by about 15% from the > one processor case. So KK is doing fine, but something bad is > happening with cuSparse. > AIJKOKKOS and AIJCUSPARSE have different algorithms? I don't know. To know exactly, the best approach is to consult with Peng@nvidia to profile the code. > > Anyone have any thoughts on this? > > Thanks, > Mark > >

1 0

cuSparse vector performance issue
by Mark Adams 25 Sep '21

25 Sep '21

I am testing my Landau code, which is MPI serial, but with many independent MPI processes driving each GPU, in an MPI parallel harness code (Landau ex2). Vector operations with Kokkos Kernels and cuSparse are about the same (KK is faster) and a bit expensive with one process / GPU. About the same as my Jacobian construction, which is expensive but optimized on the GPU. (I am using arkimex adaptive TS. I am guessing that it does a lot of vector ops, because there are a lot.) With 14 or 15 processes, all doing the same MPI serial problem, cuSparse is about 2.5x more expensive than KK. KK does degrad by about 15% from the one processor case. So KK is doing fine, but something bad is happening with cuSparse. Anyone have any thoughts on this? Thanks, Mark

1 0

Re: [petsc-dev] make test check-test-errors returning $? = 0
by Pierre Jolivet 23 Sep '21

23 Sep '21

> On 23 Sep 2021, at 2:14 PM, Matthew Knepley <knepley(a)gmail.com> wrote: > > On Thu, Sep 23, 2021 at 5:07 AM Pierre Jolivet <pierre(a)joliv.et <mailto:[email protected]>> wrote: > Hello, > When a tests fails because the timeout limit is reached, check-test-errors still returns a 0 exit code. > Is this intended? Any way to bypass this and return something different than 0, as it could return false negative in CI tests? > > This is the intended behavior. We ignore timeouts, but report them. Indeed, I see the failed test in the summary at the end of make test. That’s no big deal, GitHub workers are slow (or maybe I should stay away from ${DATAFILESPATH}/matrices/arco6), I’ll increase my timeout value for now. Thanks, Pierre > I would not have a problem with a flag to convert these to errors, but I would leave this default. > > Thanks, > > Matt > > Thanks, > Pierre > > $ make -f gmakefile test s='ksp_ksp_tutorials-ex2_1' EXTRA_OPTIONS="-m 200 -n 200" check-test-errors 2>&1 > /dev/null; echo $? > make: *** [check-test-errors] Error 1 > 2 > $ make -f gmakefile test s='ksp_ksp_tutorials-ex2_1' EXTRA_OPTIONS="-m 200 -n 200" check-test-errors TIMEOUT=1 2>&1 > /dev/null; echo $? > 0 > > > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>

1 0