December 2021 - petsc-dev - mailman3.cels.anl.gov

Re: [petsc-dev] spock
by Mark Adams 10 Dec '21

10 Dec '21

It seems to be hanging on the 2 processor test. I'll try running jobs manually. On Fri, Dec 10, 2021 at 9:34 AM Satish Balay <balay(a)mcs.anl.gov> wrote: > Merged now. And the following now works [for me]. > > 1025 git fetch -p > 1026 git checkout origin/main > 1027 ./config/examples/arch-olcf-spock.py && make > 1028 MPIR_CVAR_GPU_EAGER_DEVICE_MEM=0 MPICH_GPU_SUPPORT_ENABLED=1 > MPICH_SMP_SINGLE_COPY_MODE=CMA make check > > Satish > > On Fri, 10 Dec 2021, Satish Balay via petsc-dev wrote: > > > Works for me [per instructions in balay/update-spock, > config/examples/arch-olcf-spock.py] with main - without these additional > options > > > > I'll go ahead and merge in balay/update-spock > > > > Satish > > > > ----- > > > > 1009 git fetch -p > > 1015 module load emacs > > 1016 module load rocm/4.3.0 > > 1018 git reset --hard > > 1019 git checkout origin/main > > 1020 git merge origin/balay/update-spock > > 1021 ./config/examples/arch-olcf-spock.py && make > > > > > > > > [balay(a)login2.spock petsc]$ MPIR_CVAR_GPU_EAGER_DEVICE_MEM=0 > MPICH_GPU_SUPPORT_ENABLED=1 MPICH_SMP_SINGLE_COPY_MODE=CMA make check > > Running check examples to verify correct installation > > Using PETSC_DIR=/autofs/nccs-svm1_home1/balay/petsc and > PETSC_ARCH=arch-olcf-spock > > C/C++ example src/snes/tutorials/ex19 run successfully with 1 MPI process > > C/C++ example src/snes/tutorials/ex19 run successfully with 2 MPI > processes > > C/C++ example src/snes/tutorials/ex3k run successfully with > kokkos-kernels > > *******************Error detected during compile or > link!******************* > > See http://www.mcs.anl.gov/petsc/documentation/faq.html > > /ccs/home/balay/petsc/src/snes/tutorials ex5f > > ********************************************************* > > ftn -fPIC -fPIC -I/autofs/nccs-svm1_home1/balay/petsc/include > -I/autofs/nccs-svm1_home1/balay/petsc/arch-olcf-spock/include > -I/opt/rocm-4.3.0/include ex5f.F90 > -Wl,-rpath,/autofs/nccs-svm1_home1/balay/petsc/arch-olcf-spock/lib > -L/autofs/nccs-svm1_home1/balay/petsc/arch-olcf-spock/lib > -Wl,-rpath,/autofs/nccs-svm1_home1/balay/petsc/arch-olcf-spock/lib > -L/autofs/nccs-svm1_home1/balay/petsc/arch-olcf-spock/lib > -Wl,-rpath,/opt/rocm-4.3.0/lib -L/opt/rocm-4.3.0/lib > -Wl,-rpath,/opt/cray/pe/mpich/8.1.10/gtl/lib > -L/opt/cray/pe/mpich/8.1.10/gtl/lib > -Wl,-rpath,/opt/cray/pe/gcc/8.1.0/snos/lib64 > -L/opt/cray/pe/gcc/8.1.0/snos/lib64 -Wl,-rpath,/opt/cray/pe/libsci/ > 21.08.1.2/CRAY/9.0/x86_64/lib -L/opt/cray/pe/libsci/ > 21.08.1.2/CRAY/9.0/x86_64/lib > -Wl,-rpath,/opt/cray/pe/mpich/8.1.10/ofi/cray/10.0/lib > -L/opt/cray/pe/mpich/8.1.10/ofi/cray/10.0/lib > -Wl,-rpath,/opt/cray/pe/dsmml/0.2.2/dsmml/lib > -L/opt/cray/pe/dsmml/0.2.2/dsmml/lib -Wl,-rpath,/opt/cray/pe/pmi/6.0.14/lib > -L/opt/cray/pe/pmi/6 > > .0.14/li > > b -Wl,-rpath,/opt/cray/pe/cce/12.0.3/cce/x86_64/lib > -L/opt/cray/pe/cce/12.0.3/cce/x86_64/lib > -Wl,-rpath,/opt/cray/xpmem/2.2.40-2.1_2.44__g3cf3325.shasta/lib64 > -L/opt/cray/xpmem/2.2.40-2.1_2.44__g3cf3325.shasta/lib64 > -Wl,-rpath,/opt/cray/pe/cce/12.0.3/cce-clang/x86_64/lib/clang/12.0.0/lib/linux > -L/opt/cray/pe/cce/12.0.3/cce-clang/x86_64/lib/clang/12.0.0/lib/linux > -Wl,-rpath,/opt/cray/pe/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0 > -L/opt/cray/pe/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0 > -Wl,-rpath,/opt/cray/pe/cce/12.0.3/binutils/x86_64/x86_64-unknown-linux-gnu/lib > -L/opt/cray/pe/cce/12.0.3/binutils/x86_64/x86_64-unknown-linux-gnu/lib > -lpetsc -lmagma -lkokkoskernels -lkokkoscontainers -lkokkoscore -lhipsparse > -lhipblas -lrocsparse -lrocsolver -lrocblas -lrocrand -lamdhip64 -lstdc++ > -ldl -lmpi_gtl_hsa -lmpifort_cray -lmpi_cray -ldsmml -lpmi -lpmi2 -lxpmem > -lpgas-shmem -lquadmath -lmodules -lfi -lcraymath -lf -lu -lcsup -lgfortran > -lpthread -lgcc_eh -lm -lclang_rt.craypg > > o-x86_64 > > -lclang_rt.builtins-x86_64 -lquadmath -lstdc++ -ldl -lmpi_gtl_hsa -o > ex5f/opt/cray/pe/cce/12.0.3/binutils/x86_64/x86_64-pc-linux-gnu/bin/ld: > warning: alignment 128 of symbol > `$host_init$$runtime_init_for_iso_c_binding$iso_c_binding_' in > /opt/cray/pe/cce/12.0.3/cce/x86_64/lib/libmodules.so is smaller than 256 in > /tmp/pe_202599/ex5f_1.o > > /opt/cray/pe/cce/12.0.3/binutils/x86_64/x86_64-pc-linux-gnu/bin/ld: > warning: alignment 64 of symbol `$data_init$iso_c_binding_' in > /opt/cray/pe/cce/12.0.3/cce/x86_64/lib/libmodules.so is smaller than 256 in > /tmp/pe_202599/ex5f_1.o > > Fortran example src/snes/tutorials/ex5f run successfully with 1 MPI > process > > Completed test examples > > [balay(a)login2.spock petsc]$ > > > > > > On Fri, 10 Dec 2021, Mark Adams wrote: > > > > > FWIW, here is my current status. > > > > > > 08:08 main= spock:/gpfs/alpine/csc314/scratch/adams/petsc$ make > > > PETSC_DIR=/gpfs/alpine/csc314/scratch/adams/petsc > > > PETSC_ARCH=arch-olcf-spock check > > > Running check examples to verify correct installation > > > Using PETSC_DIR=/gpfs/alpine/csc314/scratch/adams/petsc and > > > PETSC_ARCH=arch-olcf-spock > > > Possible error running C/C++ src/snes/tutorials/ex19 with 1 MPI process > > > See http://www.mcs.anl.gov/petsc/documentation/faq.html > > > lid velocity = 0.0016, prandtl # = 1., grashof # = 1. > > > 0 KSP Residual norm 0.0406612 > > > 1 KSP Residual norm 0.036923 > > > 2 KSP Residual norm 0.0191849 > > > 3 KSP Residual norm 0.00201589 > > > 4 KSP Residual norm 0.000376045 > > > 5 KSP Residual norm 4.2974e-05 > > > 6 KSP Residual norm 5.96585e-06 > > > 7 KSP Residual norm 4.5398e-07 > > > 8 KSP Residual norm 6.30474e-08 > > > 9 KSP Residual norm 5.55518e-09 > > > 10 KSP Residual norm 6.180e-10 > > > 11 KSP Residual norm 6.211e-11 > > > Linear solve converged due to CONVERGED_RTOL iterations 11 > > > 0 KSP Residual norm 3.32845e-06 > > > 1 KSP Residual norm 9.0003e-07 > > > 2 KSP Residual norm 1.32594e-07 > > > 3 KSP Residual norm 1.49857e-08 > > > 4 KSP Residual norm 1.31887e-09 > > > 5 KSP Residual norm 2.105e-10 > > > 6 KSP Residual norm 2.827e-11 > > > 7 KSP Residual norm < 1.e-11 > > > 8 KSP Residual norm < 1.e-11 > > > 9 KSP Residual norm < 1.e-11 > > > 10 KSP Residual norm < 1.e-11 > > > Linear solve converged due to CONVERGED_RTOL iterations 10 > > > Number of SNES iterations = 2 > > > Possible error running C/C++ src/snes/tutorials/ex19 with 2 MPI > processes > > > See http://www.mcs.anl.gov/petsc/documentation/faq.html > > > lid velocity = 0.0016, prandtl # = 1., grashof # = 1. > > > 0 KSP Residual norm 0.0406612 > > > 1 KSP Residual norm 0.0281101 > > > 2 KSP Residual norm 0.00773873 > > > 3 KSP Residual norm 0.00165731 > > > 4 KSP Residual norm 0.000395614 > > > 5 KSP Residual norm 8.67655e-05 > > > 6 KSP Residual norm 1.69495e-05 > > > 7 KSP Residual norm 3.70051e-06 > > > 8 KSP Residual norm 5.97067e-07 > > > 9 KSP Residual norm 1.02242e-07 > > > 10 KSP Residual norm 1.75727e-08 > > > 11 KSP Residual norm 3.84826e-09 > > > 12 KSP Residual norm 6.414e-10 > > > 13 KSP Residual norm 1.380e-10 > > > Linear solve converged due to CONVERGED_RTOL iterations 13 > > > 0 KSP Residual norm 3.32846e-06 > > > 1 KSP Residual norm 8.99139e-07 > > > 2 KSP Residual norm 1.72893e-07 > > > 3 KSP Residual norm 3.733e-08 > > > 4 KSP Residual norm 6.67427e-09 > > > 5 KSP Residual norm 1.22785e-09 > > > 6 KSP Residual norm 2.551e-10 > > > 7 KSP Residual norm 5.458e-11 > > > 8 KSP Residual norm 1.050e-11 > > > 9 KSP Residual norm < 1.e-11 > > > 10 KSP Residual norm < 1.e-11 > > > 11 KSP Residual norm < 1.e-11 > > > 12 KSP Residual norm < 1.e-11 > > > Linear solve converged due to CONVERGED_RTOL iterations 12 > > > Number of SNES iterations = 2 > > > 3,5c3,14 > > > < 1 SNES Function norm 4.12227e-06 > > > < 2 SNES Function norm 6.098e-11 > > > < Number of SNES iterations = 2 > > > --- > > > > 0 KSP Residual norm 0.0406612 > > > > 1 KSP Residual norm 0.21263 > > > > 2 KSP Residual norm 1.09192 > > > > 3 KSP Residual norm 6.9087 > > > > 4 KSP Residual norm 23.4292 > > > > 5 KSP Residual norm 57.7558 > > > > 6 KSP Residual norm 118.076 > > > > 7 KSP Residual norm 213.527 > > > > 8 KSP Residual norm 354.101 > > > > 9 KSP Residual norm 550.58 > > > > Linear solve did not converge due to DIVERGED_DTOL iterations 9 > > > > Number of SNES iterations = 0 > > > /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials > > > Possible problem with ex19 running with hypre, diffs above > > > ========================================= > > > gmake[3]: [makefile:115: runex3k_kokkos] Error 134 (ignored) > > > 21,25c21,26 > > > < 1 SNES Function norm 2.952582418265e-01 > > > < 2 SNES Function norm 4.502293658739e-04 > > > < 3 SNES Function norm 1.389665806646e-09 > > > < Number of SNES iterations = 3 > > > < Norm of error 1.49752e-10 Iterations 3 > > > --- > > > > Memory access fault by GPU node-4 (Agent handle: 0xb08c90) on address > > > 0xe17000. Reason: Page not present or supervisor privilege. > > > > Memory access fault by GPU node-5 (Agent handle: 0xb0d3c0) on address > > > 0xe11000. Reason: Page not present or supervisor privilege. > > > > srun: error: spock25: task 0: Aborted > > > > srun: launch/slurm: _step_signal: Terminating StepId=304034.3 > > > > slurmstepd: error: *** STEP 304034.3 ON spock25 CANCELLED AT > > > 2021-12-10T08:08:40 *** > > > > srun: error: spock25: task 1: Aborted (core dumped) > > > /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials > > > Possible problem with ex3k running with kokkos-kernels, diffs above > > > ========================================= > > > *******************Error detected during compile or > link!******************* > > > See http://www.mcs.anl.gov/petsc/documentation/faq.html > > > /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials ex5f > > > ********************************************************* > > > ftn -fPIC -fPIC -I/gpfs/alpine/csc314/scratch/adams/petsc/include > > > -I/gpfs/alpine/csc314/scratch/adams/petsc/arch-olcf-spock/include > > > > -I/gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray/include > > > -I/opt/rocm-4.3.0/include ex5f.F90 > > > -Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-olcf-spock/lib > > > -L/gpfs/alpine/csc314/scratch/adams/petsc/arch-olcf-spock/lib > > > > -Wl,-rpath,/gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray/lib > > > -L/gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray/lib > > > -Wl,-rpath,/opt/rocm-4.3.0/lib -L/opt/rocm-4.3.0/lib > > > -Wl,-rpath,/opt/cray/pe/mpich/8.1.10/gtl/lib > > > -L/opt/cray/pe/mpich/8.1.10/gtl/lib > > > -Wl,-rpath,/opt/cray/pe/gcc/8.1.0/snos/lib64 > > > -L/opt/cray/pe/gcc/8.1.0/snos/lib64 -Wl,-rpath,/opt/cray/pe/libsci/ > > > 21.08.1.2/CRAY/9.0/x86_64/lib -L/opt/cray/pe/libsci/ > > > 21.08.1.2/CRAY/9.0/x86_64/lib > > > -Wl,-rpath,/opt/cray/pe/mpich/8.1.10/ofi/cray/10.0/lib > > > -L/opt/cray/pe/mpich/8.1.10/ofi/cray/10.0/lib > > > -Wl,-rpath,/opt/cray/pe/dsmml/0.2.2/dsmml/lib > > > -L/opt/cray/pe/dsmml/0.2.2/dsmml/lib > -Wl,-rpath,/opt/cray/pe/pmi/6.0.14/lib > > > -L/opt/cray/pe/pmi/6.0.14/lib > > > -Wl,-rpath,/opt/cray/pe/cce/12.0.3/cce/x86_64/lib > > > -L/opt/cray/pe/cce/12.0.3/cce/x86_64/lib > > > -Wl,-rpath,/opt/cray/xpmem/2.2.40-2.1_2.44__g3cf3325.shasta/lib64 > > > -L/opt/cray/xpmem/2.2.40-2.1_2.44__g3cf3325.shasta/lib64 > > > > -Wl,-rpath,/opt/cray/pe/cce/12.0.3/cce-clang/x86_64/lib/clang/12.0.0/lib/linux > > > -L/opt/cray/pe/cce/12.0.3/cce-clang/x86_64/lib/clang/12.0.0/lib/linux > > > -Wl,-rpath,/opt/cray/pe/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0 > > > -L/opt/cray/pe/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0 > > > > -Wl,-rpath,/opt/cray/pe/cce/12.0.3/binutils/x86_64/x86_64-unknown-linux-gnu/lib > > > -L/opt/cray/pe/cce/12.0.3/binutils/x86_64/x86_64-unknown-linux-gnu/lib > > > -lpetsc -lHYPRE -lkokkoskernels -lkokkoscontainers -lkokkoscore > -lhipsparse > > > -lhipblas -lrocsparse -lrocsolver -lrocblas -lrocrand -lamdhip64 > -lstdc++ > > > -ldl -lmpi_gtl_hsa -lmpifort_cray -lmpi_cray -ldsmml -lpmi -lpmi2 > -lxpmem > > > -lpgas-shmem -lquadmath -lmodules -lfi -lcraymath -lf -lu -lcsup > -lgfortran > > > -lpthread -lgcc_eh -lm -lclang_rt.craypgo-x86_64 > -lclang_rt.builtins-x86_64 > > > -lquadmath -lstdc++ -ldl -lmpi_gtl_hsa -o ex5f > > > /opt/cray/pe/cce/12.0.3/binutils/x86_64/x86_64-pc-linux-gnu/bin/ld: > > > warning: alignment 128 of symbol > > > `$host_init$$runtime_init_for_iso_c_binding$iso_c_binding_' in > > > /opt/cray/pe/cce/12.0.3/cce/x86_64/lib/libmodules.so is smaller than > 256 in > > > /tmp/pe_46424/ex5f_1.o > > > /opt/cray/pe/cce/12.0.3/binutils/x86_64/x86_64-pc-linux-gnu/bin/ld: > > > warning: alignment 64 of symbol `$data_init$iso_c_binding_' in > > > /opt/cray/pe/cce/12.0.3/cce/x86_64/lib/libmodules.so is smaller than > 256 in > > > /tmp/pe_46424/ex5f_1.o > > > Possible error running Fortran example src/snes/tutorials/ex5f with 1 > MPI > > > process > > > See http://www.mcs.anl.gov/petsc/documentation/faq.html > > > 0 KSP Residual norm < 1.e-11 > > > Linear solve converged due to CONVERGED_ATOL iterations 0 > > > > > > On Fri, Dec 10, 2021 at 8:07 AM Mark Adams <mfadams(a)lbl.gov> wrote: > > > > > > > I am trying to get Spock working (again) and am having problems. > > > > > > > > * make check seems to fail but it is hard to see what is going on. > Maybe > > > > we should start here, but let me continue. > > > > > > > > * GAMG seems to work on the CPU > > > > > > > > * I have this for configuring with Kokkos. I am guessing these > versions > > > > are out of data. What is current practice: > > > > '--with-kokkos-hip-arch=VEGA908', > > > > '--download-kokkos-commit=3.4.01', > > > > '--download-kokkos-kernels-commit=3.4.01', > > > > > > > > * Should I hold off (and tell my eager user to do same)? > > > > > > > > Thanks, > > > > Mark > > > > > > > > > > >

1 0

Re: [petsc-dev] Hypre error on updated Perlmutter
by Mark Adams 10 Dec '21

10 Dec '21

On Fri, Dec 10, 2021 at 10:39 AM Paul Lin <paullin(a)lbl.gov> wrote: > Hi Mark, > > Regarding the error: > "PETSC ERROR: cuda error 46 (cudaErrorDevicesUnavailable) : all > CUDA-capable devices are busy or unavailable" > > how are you requesting a perlmutter compute node? > I can run CUDA jobs so I think it is a problem with the (hypre) build. > > thanks > -paul > > > > On Fri, Dec 10, 2021 at 6:54 AM Mark Adams <mfadams(a)lbl.gov> wrote: > >> And more Perlmutter weirdness. >> >> If I configure with the above CRAY_ACCEL_TARGET=nvidia80 I get this >> (configure.log) error. (some CUDA aware MPI related errors) >> >> But if I configure with CRAY_ACCEL_TARGET="" it gets into Kokkos and I >> get this configure2.log with: >> >> #error -- unsupported pgc++ configuration! Only pgc++ 18, 19 and 20 are >> supported! >> >> I have not seen this before. >> >> As far as the first problem, If I load the cudatoolkit, which they say >> you can do *or* set CRAY_ACCEL_TARGET=nvidia80 , the problems go away or >> maybe fails before it gets to the first error, but it fails. >> I get the configure3 error that has these old warnings, but I'm not sure >> why it failed exactly. >> >> This was sort of working yesterday. I did rebase today, but even when >> working this has been fragile. >> >> Any suggestions? >> Thanks, >> Mark >> >> >> On Thu, Dec 9, 2021 at 1:59 PM Mark Adams <mfadams(a)lbl.gov> wrote: >> >>> Well I found, accidentally, that turning CUDA aware MPI on with export >>> CRAY_ACCEL_TARGET=nvidia80 >>> seems to have fixed this. >>> Not sure what is going on. >>> >>> On Thu, Dec 9, 2021 at 11:21 AM Mark Adams <mfadams(a)lbl.gov> wrote: >>> >>>> I am getting this error. I have built this w/o hypre and the test are >>>> fine, including the CUDA tests. >>>> Any ideas? >>>> >>>> I notice that the tests use -dm_mat_type aijcusparse with hypre. >>>> >>>> Thanks, >>>> Mark >>>> >>>> 08:13 nid003929 adams/fix_mat_ex5k= perlmutter:~/petsc$ make >>>> PETSC_DIR=/global/homes/m/madams/petsc >>>> PETSC_ARCH=arch-perlmutter-opt-nvidia-cuda check >>>> Running check examples to verify correct installation >>>> Using PETSC_DIR=/global/homes/m/madams/petsc and >>>> PETSC_ARCH=arch-perlmutter-opt-nvidia-cuda >>>> C/C++ example src/snes/tutorials/ex19 run successfully with 1 MPI >>>> process >>>> C/C++ example src/snes/tutorials/ex19 run successfully with 2 MPI >>>> processes >>>> 1,5c1,70 >>>> < lid velocity = 0.0016, prandtl # = 1., grashof # = 1. >>>> < 0 SNES Function norm 0.0406612 >>>> < 1 SNES Function norm 4.12227e-06 >>>> < 2 SNES Function norm 6.098e-11 >>>> < Number of SNES iterations = 2 >>>> --- >>>> > [0]PETSC ERROR: --------------------- Error Message >>>> -------------------------------------------------------------- >>>> > [0]PETSC ERROR: GPU error >>>> > [0]PETSC ERROR: cuda error 46 (cudaErrorDevicesUnavailable) : all >>>> CUDA-capable devices are busy or unavailable >>>> > [0]PETSC ERROR: See https://petsc.org/release/faq/ for trouble >>>> shooting. >>>> > [0]PETSC ERROR: Petsc Development GIT revision: >>>> v3.16.1-442-gebb4a459f5 GIT Date: 2021-12-08 08:59:23 -0500 >>>> > [0]PETSC ERROR: /global/u2/m/madams/petsc/src/snes/tutorials/./ex19 >>>> on a named nid003929 by madams Thu Dec 9 08:13:49 2021 >>>> > [0]PETSC ERROR: Configure options --CFLAGS=" -g -DLANDAU_DIM=2 >>>> -DLANDAU_MAX_SPECIES=10 -DLANDAU_MAX_Q=4 -mp=gpu" --CXXFLAGS=" -g >>>> -DLANDAU_DIM=2 -DLANDAU_MAX_SPECIES=10 -DLANDAU_MAX_Q=4 -mp=gpu" --FFLAGS=" >>>> -g -mp=gpu" --with-cc=cc --with-cxx=CC --with-fc=ftn >>>> --with-cudac=/opt/nvidia/hpc_sdk/Linux_x86_64/21.9/compilers/bin/nvcc >>>> --with-debugging=0 --download-hypre=1 --with-cuda=1 --with-cuda-arch=80 >>>> --with-mpiexec=srun --with-batch=0 --download-p4est=1 --with-zlib=1 >>>> --with-make-np=8 >>>> --prefix=/global/cfs/projectdirs/m3904/petsc/current/perlmutter-opt-nvidia21.9 >>>> PETSC_ARCH=arch-perlmutter-opt-nvidia-cuda >>>> > [0]PETSC ERROR: #1 initialize() at >>>> /global/u2/m/madams/petsc/src/sys/objects/device/impls/cupm/cupmdevice.cxx:70 >>>> > [0]PETSC ERROR: #2 getDevice() at >>>> /global/u2/m/madams/petsc/src/sys/objects/device/impls/cupm/cupmdevice.cxx:360 >>>> > [0]PETSC ERROR: #3 PetscDeviceCreate() at >>>> /global/u2/m/madams/petsc/src/sys/objects/device/interface/device.cxx:102 >>>> > [0]PETSC ERROR: #4 PetscDeviceInitializeDefaultDevice_Internal() at >>>> /global/u2/m/madams/petsc/src/sys/objects/device/interface/device.cxx:266 >>>> > [1]PETSC ERROR: --------------------- Error Message >>>> -------------------------------------------------------------- >>>> > [1]PETSC ERROR: GPU error >>>> > [1]PETSC ERROR: cuda error 46 (cudaErrorDevicesUnavailable) : all >>>> CUDA-capable devices are busy or unavailable >>>> > [1]PETSC ERROR: See https://petsc.org/release/faq/ for trouble >>>> shooting. >>>> > [1]PETSC ERROR: Petsc Development GIT revision: >>>> v3.16.1-442-gebb4a459f5 GIT Date: 2021-12-08 08:59:23 -0500 >>>> > [1]PETSC ERROR: /global/u2/m/madams/petsc/src/snes/tutorials/./ex19 >>>> on a named nid003929 by madams Thu Dec 9 08:13:49 2021 >>>> > [1]PETSC ERROR: Configure options --CFLAGS=" -g -DLANDAU_DIM=2 >>>> -DLANDAU_MAX_SPECIES=10 -DLANDAU_MAX_Q=4 -mp=gpu" --CXXFLAGS=" -g >>>> -DLANDAU_DIM=2 -DLANDAU_MAX_SPECIES=10 -DLANDAU_MAX_Q=4 -mp=gpu" --FFLAGS=" >>>> -g -mp=gpu" --with-cc=cc --with-cxx=CC --with-fc=ftn >>>> --with-cudac=/opt/nvidia/hpc_sdk/Linux_x86_64/21.9/compilers/bin/nvcc >>>> --with-debugging=0 --download-hypre=1 --with-cuda=1 --with-cuda-arch=80 >>>> --with-mpiexec=srun --with-batch=0 --download-p4est=1 --with-zlib=1 >>>> --with-make-np=8 >>>> --prefix=/global/cfs/projectdirs/m3904/petsc/current/perlmutter-opt-nvidia21.9 >>>> PETSC_ARCH=arch-perlmutter-opt-nvidia-cuda >>>> > [1]PETSC ERROR: #1 initialize() at >>>> /global/u2/m/madams/petsc/src/sys/objects/device/impls/cupm/cupmdevice.cxx:70 >>>> > [1]PETSC ERROR: #2 getDevice() at >>>> /global/u2/m/madams/petsc/src/sys/objects/device/impls/cupm/cupmdevice.cxx:360 >>>> > [1]PETSC ERROR: #3 PetscDeviceCreate() at >>>> /global/u2/m/madams/petsc/src/sys/objects/device/interface/device.cxx:102 >>>> > [1]PETSC ERROR: #4 PetscDeviceInitializeDefaultDevice_Internal() at >>>> /global/u2/m/madams/petsc/src/sys/objects/device/interface/device.cxx:266 >>>> > [1]PETSC ERROR: #5 PetscDeviceInitialize() at >>>> /global/u2/m/madams/petsc/src/sys/objects/device/interface/device.cxx:227 >>>> > [1]PETSC ERROR: #6 PCCreate_HYPRE() at >>>> /global/u2/m/madams/petsc/src/ksp/pc/impls/hypre/hypre.c:2224 >>>> > [1]PETSC ERROR: #7 PCSetType() at >>>> /global/u2/m/madams/petsc/src/ksp/pc/interface/pcset.c:84 >>>> > [1]PETSC ERROR: #8 PCSetFromOptions() at >>>> /global/u2/m/madams/petsc/src/ksp/pc/interface/pcset.c:154 >>>> > [1]PETSC ERROR: #9 KSPSetFromOptions() at >>>> /global/u2/m/madams/petsc/src/ksp/ksp/interface/itcl.c:356 >>>> > [1]PETSC ERROR: #10 SNESSetFromOptions() at >>>> /global/u2/m/madams/petsc/src/snes/interface/snes.c:1113 >>>> > [1]PETSC ERROR: #11 main() at ex19.c:150 >>>> > [1]PETSC ERROR: PETSc Option Table entries: >>>> > [1]PETSC ERROR: -da_refine 3 >>>> > [1]PETSC ERROR: -dm_mat_type aijcusparse >>>> > [1]PETSC ERROR: -dm_vec_type cuda >>>> > [0]PETSC ERROR: #5 PetscDeviceInitialize() at >>>> /global/u2/m/madams/petsc/src/sys/objects/device/interface/device.cxx:227 >>>> > [0]PETSC ERROR: #6 PCCreate_HYPRE() at >>>> /global/u2/m/madams/petsc/src/ksp/pc/impls/hypre/hypre.c:2224 >>>> > [0]PETSC ERROR: #7 PCSetType() at >>>> /global/u2/m/madams/petsc/src/ksp/pc/interface/pcset.c:84 >>>> > [0]PETSC ERROR: #8 PCSetFromOptions() at >>>> /global/u2/m/madams/petsc/src/ksp/pc/interface/pcset.c:154 >>>> > [0]PETSC ERROR: #9 KSPSetFromOptions() at >>>> /global/u2/m/madams/petsc/src/ksp/ksp/interface/itcl.c:356 >>>> > [0]PETSC ERROR: #10 SNESSetFromOptions() at >>>> /global/u2/m/madams/petsc/src/snes/interface/snes.c:1113 >>>> > [0]PETSC ERROR: #11 main() at ex19.c:150 >>>> > [0]PETSC ERROR: PETSc Option Table entries: >>>> > [0]PETSC ERROR: -da_refine 3 >>>> > [0]PETSC ERROR: -dm_mat_type aijcusparse >>>> > [0]PETSC ERROR: -dm_vec_type cuda >>>> > [0]PETSC ERROR: -ksp_norm_type unpreconditioned >>>> > [0]PETSC ERROR: -nox >>>> > [0]PETSC ERROR: -nox_warning >>>> > [0]PETSC ERROR: -pc_type hypre >>>> > [0]PETSC ERROR: -snes_monitor_short >>>> > [0]PETSC ERROR: -use_gpu_aware_mpi 0 >>>> > [0]PETSC ERROR: ----------------End of Error Message -------send >>>> entire error message to petsc-maint(a)mcs.anl.gov---------- >>>> > [1]PETSC ERROR: -ksp_norm_type unpreconditioned >>>> > [1]PETSC ERROR: -nox >>>> > [1]PETSC ERROR: -nox_warning >>>> > [1]PETSC ERROR: -pc_type hypre >>>> > [1]PETSC ERROR: -snes_monitor_short >>>> > [1]PETSC ERROR: -use_gpu_aware_mpi 0 >>>> > [1]PETSC ERROR: ----------------End of Error Message -------send >>>> entire error message to petsc-maint(a)mcs.anl.gov---------- >>>> > MPICH Notice [Rank 0] [job id 832277.2] [Thu Dec 9 08:13:50 2021] >>>> [nid003929] - Abort(97) (rank 0 in comm 0): application called >>>> MPI_Abort(MPI_COMM_WORLD, 97) - process 0 >>>> > >>>> > aborting job: >>>> > application called MPI_Abort(MPI_COMM_WORLD, 97) - process 0 >>>> > MPICH Notice [Rank 1] [job id 832277.2] [Thu Dec 9 08:13:50 2021] >>>> [nid003929] - Abort(97) (rank 1 in comm 0): application called >>>> MPI_Abort(MPI_COMM_WORLD, 97) - process 1 >>>> > >>>> > aborting job: >>>> > application called MPI_Abort(MPI_COMM_WORLD, 97) - process 1 >>>> > srun: error: nid003929: task 1: Exited with exit code 255 >>>> > srun: launch/slurm: _step_signal: Terminating StepId=832277.2 >>>> > slurmstepd: error: *** STEP 832277.2 ON nid003929 CANCELLED AT >>>> 2021-12-09T16:13:50 *** >>>> > srun: error: nid003929: task 0: Exited with exit code 255 >>>> /global/homes/m/madams/petsc/src/snes/tutorials >>>> Possible problem with ex19 running with hypre, diffs above >>>> ========================================= >>>> C/C++ example src/snes/tutorials/ex19 run successfully with cuda >>>> Fortran example src/snes/tutorials/ex5f run successfully with 1 MPI >>>> process >>>> >>>

1 0

Re: [petsc-dev] Hypre error on updated Perlmutter
by Paul Lin 10 Dec '21

10 Dec '21

Hi Mark, Regarding the error: "PETSC ERROR: cuda error 46 (cudaErrorDevicesUnavailable) : all CUDA-capable devices are busy or unavailable" how are you requesting a perlmutter compute node? thanks -paul On Fri, Dec 10, 2021 at 6:54 AM Mark Adams <mfadams(a)lbl.gov> wrote: > And more Perlmutter weirdness. > > If I configure with the above CRAY_ACCEL_TARGET=nvidia80 I get this > (configure.log) error. (some CUDA aware MPI related errors) > > But if I configure with CRAY_ACCEL_TARGET="" it gets into Kokkos and I get > this configure2.log with: > > #error -- unsupported pgc++ configuration! Only pgc++ 18, 19 and 20 are > supported! > > I have not seen this before. > > As far as the first problem, If I load the cudatoolkit, which they say you > can do *or* set CRAY_ACCEL_TARGET=nvidia80 , the problems go away or maybe > fails before it gets to the first error, but it fails. > I get the configure3 error that has these old warnings, but I'm not sure > why it failed exactly. > > This was sort of working yesterday. I did rebase today, but even when > working this has been fragile. > > Any suggestions? > Thanks, > Mark > > > On Thu, Dec 9, 2021 at 1:59 PM Mark Adams <mfadams(a)lbl.gov> wrote: > >> Well I found, accidentally, that turning CUDA aware MPI on with export >> CRAY_ACCEL_TARGET=nvidia80 >> seems to have fixed this. >> Not sure what is going on. >> >> On Thu, Dec 9, 2021 at 11:21 AM Mark Adams <mfadams(a)lbl.gov> wrote: >> >>> I am getting this error. I have built this w/o hypre and the test are >>> fine, including the CUDA tests. >>> Any ideas? >>> >>> I notice that the tests use -dm_mat_type aijcusparse with hypre. >>> >>> Thanks, >>> Mark >>> >>> 08:13 nid003929 adams/fix_mat_ex5k= perlmutter:~/petsc$ make >>> PETSC_DIR=/global/homes/m/madams/petsc >>> PETSC_ARCH=arch-perlmutter-opt-nvidia-cuda check >>> Running check examples to verify correct installation >>> Using PETSC_DIR=/global/homes/m/madams/petsc and >>> PETSC_ARCH=arch-perlmutter-opt-nvidia-cuda >>> C/C++ example src/snes/tutorials/ex19 run successfully with 1 MPI process >>> C/C++ example src/snes/tutorials/ex19 run successfully with 2 MPI >>> processes >>> 1,5c1,70 >>> < lid velocity = 0.0016, prandtl # = 1., grashof # = 1. >>> < 0 SNES Function norm 0.0406612 >>> < 1 SNES Function norm 4.12227e-06 >>> < 2 SNES Function norm 6.098e-11 >>> < Number of SNES iterations = 2 >>> --- >>> > [0]PETSC ERROR: --------------------- Error Message >>> -------------------------------------------------------------- >>> > [0]PETSC ERROR: GPU error >>> > [0]PETSC ERROR: cuda error 46 (cudaErrorDevicesUnavailable) : all >>> CUDA-capable devices are busy or unavailable >>> > [0]PETSC ERROR: See https://petsc.org/release/faq/ for trouble >>> shooting. >>> > [0]PETSC ERROR: Petsc Development GIT revision: >>> v3.16.1-442-gebb4a459f5 GIT Date: 2021-12-08 08:59:23 -0500 >>> > [0]PETSC ERROR: /global/u2/m/madams/petsc/src/snes/tutorials/./ex19 on >>> a named nid003929 by madams Thu Dec 9 08:13:49 2021 >>> > [0]PETSC ERROR: Configure options --CFLAGS=" -g -DLANDAU_DIM=2 >>> -DLANDAU_MAX_SPECIES=10 -DLANDAU_MAX_Q=4 -mp=gpu" --CXXFLAGS=" -g >>> -DLANDAU_DIM=2 -DLANDAU_MAX_SPECIES=10 -DLANDAU_MAX_Q=4 -mp=gpu" --FFLAGS=" >>> -g -mp=gpu" --with-cc=cc --with-cxx=CC --with-fc=ftn >>> --with-cudac=/opt/nvidia/hpc_sdk/Linux_x86_64/21.9/compilers/bin/nvcc >>> --with-debugging=0 --download-hypre=1 --with-cuda=1 --with-cuda-arch=80 >>> --with-mpiexec=srun --with-batch=0 --download-p4est=1 --with-zlib=1 >>> --with-make-np=8 >>> --prefix=/global/cfs/projectdirs/m3904/petsc/current/perlmutter-opt-nvidia21.9 >>> PETSC_ARCH=arch-perlmutter-opt-nvidia-cuda >>> > [0]PETSC ERROR: #1 initialize() at >>> /global/u2/m/madams/petsc/src/sys/objects/device/impls/cupm/cupmdevice.cxx:70 >>> > [0]PETSC ERROR: #2 getDevice() at >>> /global/u2/m/madams/petsc/src/sys/objects/device/impls/cupm/cupmdevice.cxx:360 >>> > [0]PETSC ERROR: #3 PetscDeviceCreate() at >>> /global/u2/m/madams/petsc/src/sys/objects/device/interface/device.cxx:102 >>> > [0]PETSC ERROR: #4 PetscDeviceInitializeDefaultDevice_Internal() at >>> /global/u2/m/madams/petsc/src/sys/objects/device/interface/device.cxx:266 >>> > [1]PETSC ERROR: --------------------- Error Message >>> -------------------------------------------------------------- >>> > [1]PETSC ERROR: GPU error >>> > [1]PETSC ERROR: cuda error 46 (cudaErrorDevicesUnavailable) : all >>> CUDA-capable devices are busy or unavailable >>> > [1]PETSC ERROR: See https://petsc.org/release/faq/ for trouble >>> shooting. >>> > [1]PETSC ERROR: Petsc Development GIT revision: >>> v3.16.1-442-gebb4a459f5 GIT Date: 2021-12-08 08:59:23 -0500 >>> > [1]PETSC ERROR: /global/u2/m/madams/petsc/src/snes/tutorials/./ex19 on >>> a named nid003929 by madams Thu Dec 9 08:13:49 2021 >>> > [1]PETSC ERROR: Configure options --CFLAGS=" -g -DLANDAU_DIM=2 >>> -DLANDAU_MAX_SPECIES=10 -DLANDAU_MAX_Q=4 -mp=gpu" --CXXFLAGS=" -g >>> -DLANDAU_DIM=2 -DLANDAU_MAX_SPECIES=10 -DLANDAU_MAX_Q=4 -mp=gpu" --FFLAGS=" >>> -g -mp=gpu" --with-cc=cc --with-cxx=CC --with-fc=ftn >>> --with-cudac=/opt/nvidia/hpc_sdk/Linux_x86_64/21.9/compilers/bin/nvcc >>> --with-debugging=0 --download-hypre=1 --with-cuda=1 --with-cuda-arch=80 >>> --with-mpiexec=srun --with-batch=0 --download-p4est=1 --with-zlib=1 >>> --with-make-np=8 >>> --prefix=/global/cfs/projectdirs/m3904/petsc/current/perlmutter-opt-nvidia21.9 >>> PETSC_ARCH=arch-perlmutter-opt-nvidia-cuda >>> > [1]PETSC ERROR: #1 initialize() at >>> /global/u2/m/madams/petsc/src/sys/objects/device/impls/cupm/cupmdevice.cxx:70 >>> > [1]PETSC ERROR: #2 getDevice() at >>> /global/u2/m/madams/petsc/src/sys/objects/device/impls/cupm/cupmdevice.cxx:360 >>> > [1]PETSC ERROR: #3 PetscDeviceCreate() at >>> /global/u2/m/madams/petsc/src/sys/objects/device/interface/device.cxx:102 >>> > [1]PETSC ERROR: #4 PetscDeviceInitializeDefaultDevice_Internal() at >>> /global/u2/m/madams/petsc/src/sys/objects/device/interface/device.cxx:266 >>> > [1]PETSC ERROR: #5 PetscDeviceInitialize() at >>> /global/u2/m/madams/petsc/src/sys/objects/device/interface/device.cxx:227 >>> > [1]PETSC ERROR: #6 PCCreate_HYPRE() at >>> /global/u2/m/madams/petsc/src/ksp/pc/impls/hypre/hypre.c:2224 >>> > [1]PETSC ERROR: #7 PCSetType() at >>> /global/u2/m/madams/petsc/src/ksp/pc/interface/pcset.c:84 >>> > [1]PETSC ERROR: #8 PCSetFromOptions() at >>> /global/u2/m/madams/petsc/src/ksp/pc/interface/pcset.c:154 >>> > [1]PETSC ERROR: #9 KSPSetFromOptions() at >>> /global/u2/m/madams/petsc/src/ksp/ksp/interface/itcl.c:356 >>> > [1]PETSC ERROR: #10 SNESSetFromOptions() at >>> /global/u2/m/madams/petsc/src/snes/interface/snes.c:1113 >>> > [1]PETSC ERROR: #11 main() at ex19.c:150 >>> > [1]PETSC ERROR: PETSc Option Table entries: >>> > [1]PETSC ERROR: -da_refine 3 >>> > [1]PETSC ERROR: -dm_mat_type aijcusparse >>> > [1]PETSC ERROR: -dm_vec_type cuda >>> > [0]PETSC ERROR: #5 PetscDeviceInitialize() at >>> /global/u2/m/madams/petsc/src/sys/objects/device/interface/device.cxx:227 >>> > [0]PETSC ERROR: #6 PCCreate_HYPRE() at >>> /global/u2/m/madams/petsc/src/ksp/pc/impls/hypre/hypre.c:2224 >>> > [0]PETSC ERROR: #7 PCSetType() at >>> /global/u2/m/madams/petsc/src/ksp/pc/interface/pcset.c:84 >>> > [0]PETSC ERROR: #8 PCSetFromOptions() at >>> /global/u2/m/madams/petsc/src/ksp/pc/interface/pcset.c:154 >>> > [0]PETSC ERROR: #9 KSPSetFromOptions() at >>> /global/u2/m/madams/petsc/src/ksp/ksp/interface/itcl.c:356 >>> > [0]PETSC ERROR: #10 SNESSetFromOptions() at >>> /global/u2/m/madams/petsc/src/snes/interface/snes.c:1113 >>> > [0]PETSC ERROR: #11 main() at ex19.c:150 >>> > [0]PETSC ERROR: PETSc Option Table entries: >>> > [0]PETSC ERROR: -da_refine 3 >>> > [0]PETSC ERROR: -dm_mat_type aijcusparse >>> > [0]PETSC ERROR: -dm_vec_type cuda >>> > [0]PETSC ERROR: -ksp_norm_type unpreconditioned >>> > [0]PETSC ERROR: -nox >>> > [0]PETSC ERROR: -nox_warning >>> > [0]PETSC ERROR: -pc_type hypre >>> > [0]PETSC ERROR: -snes_monitor_short >>> > [0]PETSC ERROR: -use_gpu_aware_mpi 0 >>> > [0]PETSC ERROR: ----------------End of Error Message -------send >>> entire error message to petsc-maint(a)mcs.anl.gov---------- >>> > [1]PETSC ERROR: -ksp_norm_type unpreconditioned >>> > [1]PETSC ERROR: -nox >>> > [1]PETSC ERROR: -nox_warning >>> > [1]PETSC ERROR: -pc_type hypre >>> > [1]PETSC ERROR: -snes_monitor_short >>> > [1]PETSC ERROR: -use_gpu_aware_mpi 0 >>> > [1]PETSC ERROR: ----------------End of Error Message -------send >>> entire error message to petsc-maint(a)mcs.anl.gov---------- >>> > MPICH Notice [Rank 0] [job id 832277.2] [Thu Dec 9 08:13:50 2021] >>> [nid003929] - Abort(97) (rank 0 in comm 0): application called >>> MPI_Abort(MPI_COMM_WORLD, 97) - process 0 >>> > >>> > aborting job: >>> > application called MPI_Abort(MPI_COMM_WORLD, 97) - process 0 >>> > MPICH Notice [Rank 1] [job id 832277.2] [Thu Dec 9 08:13:50 2021] >>> [nid003929] - Abort(97) (rank 1 in comm 0): application called >>> MPI_Abort(MPI_COMM_WORLD, 97) - process 1 >>> > >>> > aborting job: >>> > application called MPI_Abort(MPI_COMM_WORLD, 97) - process 1 >>> > srun: error: nid003929: task 1: Exited with exit code 255 >>> > srun: launch/slurm: _step_signal: Terminating StepId=832277.2 >>> > slurmstepd: error: *** STEP 832277.2 ON nid003929 CANCELLED AT >>> 2021-12-09T16:13:50 *** >>> > srun: error: nid003929: task 0: Exited with exit code 255 >>> /global/homes/m/madams/petsc/src/snes/tutorials >>> Possible problem with ex19 running with hypre, diffs above >>> ========================================= >>> C/C++ example src/snes/tutorials/ex19 run successfully with cuda >>> Fortran example src/snes/tutorials/ex5f run successfully with 1 MPI >>> process >>> >>

1 0

Re: [petsc-dev] Hypre error on updated Perlmutter
by Mark Adams 10 Dec '21

10 Dec '21

And more Perlmutter weirdness. If I configure with the above CRAY_ACCEL_TARGET=nvidia80 I get this (configure.log) error. (some CUDA aware MPI related errors) But if I configure with CRAY_ACCEL_TARGET="" it gets into Kokkos and I get this configure2.log with: #error -- unsupported pgc++ configuration! Only pgc++ 18, 19 and 20 are supported! I have not seen this before. As far as the first problem, If I load the cudatoolkit, which they say you can do *or* set CRAY_ACCEL_TARGET=nvidia80 , the problems go away or maybe fails before it gets to the first error, but it fails. I get the configure3 error that has these old warnings, but I'm not sure why it failed exactly. This was sort of working yesterday. I did rebase today, but even when working this has been fragile. Any suggestions? Thanks, Mark On Thu, Dec 9, 2021 at 1:59 PM Mark Adams <mfadams(a)lbl.gov> wrote: > Well I found, accidentally, that turning CUDA aware MPI on with export > CRAY_ACCEL_TARGET=nvidia80 > seems to have fixed this. > Not sure what is going on. > > On Thu, Dec 9, 2021 at 11:21 AM Mark Adams <mfadams(a)lbl.gov> wrote: > >> I am getting this error. I have built this w/o hypre and the test are >> fine, including the CUDA tests. >> Any ideas? >> >> I notice that the tests use -dm_mat_type aijcusparse with hypre. >> >> Thanks, >> Mark >> >> 08:13 nid003929 adams/fix_mat_ex5k= perlmutter:~/petsc$ make >> PETSC_DIR=/global/homes/m/madams/petsc >> PETSC_ARCH=arch-perlmutter-opt-nvidia-cuda check >> Running check examples to verify correct installation >> Using PETSC_DIR=/global/homes/m/madams/petsc and >> PETSC_ARCH=arch-perlmutter-opt-nvidia-cuda >> C/C++ example src/snes/tutorials/ex19 run successfully with 1 MPI process >> C/C++ example src/snes/tutorials/ex19 run successfully with 2 MPI >> processes >> 1,5c1,70 >> < lid velocity = 0.0016, prandtl # = 1., grashof # = 1. >> < 0 SNES Function norm 0.0406612 >> < 1 SNES Function norm 4.12227e-06 >> < 2 SNES Function norm 6.098e-11 >> < Number of SNES iterations = 2 >> --- >> > [0]PETSC ERROR: --------------------- Error Message >> -------------------------------------------------------------- >> > [0]PETSC ERROR: GPU error >> > [0]PETSC ERROR: cuda error 46 (cudaErrorDevicesUnavailable) : all >> CUDA-capable devices are busy or unavailable >> > [0]PETSC ERROR: See https://petsc.org/release/faq/ for trouble >> shooting. >> > [0]PETSC ERROR: Petsc Development GIT revision: v3.16.1-442-gebb4a459f5 >> GIT Date: 2021-12-08 08:59:23 -0500 >> > [0]PETSC ERROR: /global/u2/m/madams/petsc/src/snes/tutorials/./ex19 on >> a named nid003929 by madams Thu Dec 9 08:13:49 2021 >> > [0]PETSC ERROR: Configure options --CFLAGS=" -g -DLANDAU_DIM=2 >> -DLANDAU_MAX_SPECIES=10 -DLANDAU_MAX_Q=4 -mp=gpu" --CXXFLAGS=" -g >> -DLANDAU_DIM=2 -DLANDAU_MAX_SPECIES=10 -DLANDAU_MAX_Q=4 -mp=gpu" --FFLAGS=" >> -g -mp=gpu" --with-cc=cc --with-cxx=CC --with-fc=ftn >> --with-cudac=/opt/nvidia/hpc_sdk/Linux_x86_64/21.9/compilers/bin/nvcc >> --with-debugging=0 --download-hypre=1 --with-cuda=1 --with-cuda-arch=80 >> --with-mpiexec=srun --with-batch=0 --download-p4est=1 --with-zlib=1 >> --with-make-np=8 >> --prefix=/global/cfs/projectdirs/m3904/petsc/current/perlmutter-opt-nvidia21.9 >> PETSC_ARCH=arch-perlmutter-opt-nvidia-cuda >> > [0]PETSC ERROR: #1 initialize() at >> /global/u2/m/madams/petsc/src/sys/objects/device/impls/cupm/cupmdevice.cxx:70 >> > [0]PETSC ERROR: #2 getDevice() at >> /global/u2/m/madams/petsc/src/sys/objects/device/impls/cupm/cupmdevice.cxx:360 >> > [0]PETSC ERROR: #3 PetscDeviceCreate() at >> /global/u2/m/madams/petsc/src/sys/objects/device/interface/device.cxx:102 >> > [0]PETSC ERROR: #4 PetscDeviceInitializeDefaultDevice_Internal() at >> /global/u2/m/madams/petsc/src/sys/objects/device/interface/device.cxx:266 >> > [1]PETSC ERROR: --------------------- Error Message >> -------------------------------------------------------------- >> > [1]PETSC ERROR: GPU error >> > [1]PETSC ERROR: cuda error 46 (cudaErrorDevicesUnavailable) : all >> CUDA-capable devices are busy or unavailable >> > [1]PETSC ERROR: See https://petsc.org/release/faq/ for trouble >> shooting. >> > [1]PETSC ERROR: Petsc Development GIT revision: v3.16.1-442-gebb4a459f5 >> GIT Date: 2021-12-08 08:59:23 -0500 >> > [1]PETSC ERROR: /global/u2/m/madams/petsc/src/snes/tutorials/./ex19 on >> a named nid003929 by madams Thu Dec 9 08:13:49 2021 >> > [1]PETSC ERROR: Configure options --CFLAGS=" -g -DLANDAU_DIM=2 >> -DLANDAU_MAX_SPECIES=10 -DLANDAU_MAX_Q=4 -mp=gpu" --CXXFLAGS=" -g >> -DLANDAU_DIM=2 -DLANDAU_MAX_SPECIES=10 -DLANDAU_MAX_Q=4 -mp=gpu" --FFLAGS=" >> -g -mp=gpu" --with-cc=cc --with-cxx=CC --with-fc=ftn >> --with-cudac=/opt/nvidia/hpc_sdk/Linux_x86_64/21.9/compilers/bin/nvcc >> --with-debugging=0 --download-hypre=1 --with-cuda=1 --with-cuda-arch=80 >> --with-mpiexec=srun --with-batch=0 --download-p4est=1 --with-zlib=1 >> --with-make-np=8 >> --prefix=/global/cfs/projectdirs/m3904/petsc/current/perlmutter-opt-nvidia21.9 >> PETSC_ARCH=arch-perlmutter-opt-nvidia-cuda >> > [1]PETSC ERROR: #1 initialize() at >> /global/u2/m/madams/petsc/src/sys/objects/device/impls/cupm/cupmdevice.cxx:70 >> > [1]PETSC ERROR: #2 getDevice() at >> /global/u2/m/madams/petsc/src/sys/objects/device/impls/cupm/cupmdevice.cxx:360 >> > [1]PETSC ERROR: #3 PetscDeviceCreate() at >> /global/u2/m/madams/petsc/src/sys/objects/device/interface/device.cxx:102 >> > [1]PETSC ERROR: #4 PetscDeviceInitializeDefaultDevice_Internal() at >> /global/u2/m/madams/petsc/src/sys/objects/device/interface/device.cxx:266 >> > [1]PETSC ERROR: #5 PetscDeviceInitialize() at >> /global/u2/m/madams/petsc/src/sys/objects/device/interface/device.cxx:227 >> > [1]PETSC ERROR: #6 PCCreate_HYPRE() at >> /global/u2/m/madams/petsc/src/ksp/pc/impls/hypre/hypre.c:2224 >> > [1]PETSC ERROR: #7 PCSetType() at >> /global/u2/m/madams/petsc/src/ksp/pc/interface/pcset.c:84 >> > [1]PETSC ERROR: #8 PCSetFromOptions() at >> /global/u2/m/madams/petsc/src/ksp/pc/interface/pcset.c:154 >> > [1]PETSC ERROR: #9 KSPSetFromOptions() at >> /global/u2/m/madams/petsc/src/ksp/ksp/interface/itcl.c:356 >> > [1]PETSC ERROR: #10 SNESSetFromOptions() at >> /global/u2/m/madams/petsc/src/snes/interface/snes.c:1113 >> > [1]PETSC ERROR: #11 main() at ex19.c:150 >> > [1]PETSC ERROR: PETSc Option Table entries: >> > [1]PETSC ERROR: -da_refine 3 >> > [1]PETSC ERROR: -dm_mat_type aijcusparse >> > [1]PETSC ERROR: -dm_vec_type cuda >> > [0]PETSC ERROR: #5 PetscDeviceInitialize() at >> /global/u2/m/madams/petsc/src/sys/objects/device/interface/device.cxx:227 >> > [0]PETSC ERROR: #6 PCCreate_HYPRE() at >> /global/u2/m/madams/petsc/src/ksp/pc/impls/hypre/hypre.c:2224 >> > [0]PETSC ERROR: #7 PCSetType() at >> /global/u2/m/madams/petsc/src/ksp/pc/interface/pcset.c:84 >> > [0]PETSC ERROR: #8 PCSetFromOptions() at >> /global/u2/m/madams/petsc/src/ksp/pc/interface/pcset.c:154 >> > [0]PETSC ERROR: #9 KSPSetFromOptions() at >> /global/u2/m/madams/petsc/src/ksp/ksp/interface/itcl.c:356 >> > [0]PETSC ERROR: #10 SNESSetFromOptions() at >> /global/u2/m/madams/petsc/src/snes/interface/snes.c:1113 >> > [0]PETSC ERROR: #11 main() at ex19.c:150 >> > [0]PETSC ERROR: PETSc Option Table entries: >> > [0]PETSC ERROR: -da_refine 3 >> > [0]PETSC ERROR: -dm_mat_type aijcusparse >> > [0]PETSC ERROR: -dm_vec_type cuda >> > [0]PETSC ERROR: -ksp_norm_type unpreconditioned >> > [0]PETSC ERROR: -nox >> > [0]PETSC ERROR: -nox_warning >> > [0]PETSC ERROR: -pc_type hypre >> > [0]PETSC ERROR: -snes_monitor_short >> > [0]PETSC ERROR: -use_gpu_aware_mpi 0 >> > [0]PETSC ERROR: ----------------End of Error Message -------send entire >> error message to petsc-maint(a)mcs.anl.gov---------- >> > [1]PETSC ERROR: -ksp_norm_type unpreconditioned >> > [1]PETSC ERROR: -nox >> > [1]PETSC ERROR: -nox_warning >> > [1]PETSC ERROR: -pc_type hypre >> > [1]PETSC ERROR: -snes_monitor_short >> > [1]PETSC ERROR: -use_gpu_aware_mpi 0 >> > [1]PETSC ERROR: ----------------End of Error Message -------send entire >> error message to petsc-maint(a)mcs.anl.gov---------- >> > MPICH Notice [Rank 0] [job id 832277.2] [Thu Dec 9 08:13:50 2021] >> [nid003929] - Abort(97) (rank 0 in comm 0): application called >> MPI_Abort(MPI_COMM_WORLD, 97) - process 0 >> > >> > aborting job: >> > application called MPI_Abort(MPI_COMM_WORLD, 97) - process 0 >> > MPICH Notice [Rank 1] [job id 832277.2] [Thu Dec 9 08:13:50 2021] >> [nid003929] - Abort(97) (rank 1 in comm 0): application called >> MPI_Abort(MPI_COMM_WORLD, 97) - process 1 >> > >> > aborting job: >> > application called MPI_Abort(MPI_COMM_WORLD, 97) - process 1 >> > srun: error: nid003929: task 1: Exited with exit code 255 >> > srun: launch/slurm: _step_signal: Terminating StepId=832277.2 >> > slurmstepd: error: *** STEP 832277.2 ON nid003929 CANCELLED AT >> 2021-12-09T16:13:50 *** >> > srun: error: nid003929: task 0: Exited with exit code 255 >> /global/homes/m/madams/petsc/src/snes/tutorials >> Possible problem with ex19 running with hypre, diffs above >> ========================================= >> C/C++ example src/snes/tutorials/ex19 run successfully with cuda >> Fortran example src/snes/tutorials/ex5f run successfully with 1 MPI >> process >> >

1 0

Re: [petsc-dev] spock
by Satish Balay 10 Dec '21

10 Dec '21

Merged now. And the following now works [for me]. 1025 git fetch -p 1026 git checkout origin/main 1027 ./config/examples/arch-olcf-spock.py && make 1028 MPIR_CVAR_GPU_EAGER_DEVICE_MEM=0 MPICH_GPU_SUPPORT_ENABLED=1 MPICH_SMP_SINGLE_COPY_MODE=CMA make check Satish On Fri, 10 Dec 2021, Satish Balay via petsc-dev wrote: > Works for me [per instructions in balay/update-spock, config/examples/arch-olcf-spock.py] with main - without these additional options > > I'll go ahead and merge in balay/update-spock > > Satish > > ----- > > 1009 git fetch -p > 1015 module load emacs > 1016 module load rocm/4.3.0 > 1018 git reset --hard > 1019 git checkout origin/main > 1020 git merge origin/balay/update-spock > 1021 ./config/examples/arch-olcf-spock.py && make > > > > [balay(a)login2.spock petsc]$ MPIR_CVAR_GPU_EAGER_DEVICE_MEM=0 MPICH_GPU_SUPPORT_ENABLED=1 MPICH_SMP_SINGLE_COPY_MODE=CMA make check > Running check examples to verify correct installation > Using PETSC_DIR=/autofs/nccs-svm1_home1/balay/petsc and PETSC_ARCH=arch-olcf-spock > C/C++ example src/snes/tutorials/ex19 run successfully with 1 MPI process > C/C++ example src/snes/tutorials/ex19 run successfully with 2 MPI processes > C/C++ example src/snes/tutorials/ex3k run successfully with kokkos-kernels > *******************Error detected during compile or link!******************* > See http://www.mcs.anl.gov/petsc/documentation/faq.html > /ccs/home/balay/petsc/src/snes/tutorials ex5f > ********************************************************* > ftn -fPIC -fPIC -I/autofs/nccs-svm1_home1/balay/petsc/include -I/autofs/nccs-svm1_home1/balay/petsc/arch-olcf-spock/include -I/opt/rocm-4.3.0/include ex5f.F90 -Wl,-rpath,/autofs/nccs-svm1_home1/balay/petsc/arch-olcf-spock/lib -L/autofs/nccs-svm1_home1/balay/petsc/arch-olcf-spock/lib -Wl,-rpath,/autofs/nccs-svm1_home1/balay/petsc/arch-olcf-spock/lib -L/autofs/nccs-svm1_home1/balay/petsc/arch-olcf-spock/lib -Wl,-rpath,/opt/rocm-4.3.0/lib -L/opt/rocm-4.3.0/lib -Wl,-rpath,/opt/cray/pe/mpich/8.1.10/gtl/lib -L/opt/cray/pe/mpich/8.1.10/gtl/lib -Wl,-rpath,/opt/cray/pe/gcc/8.1.0/snos/lib64 -L/opt/cray/pe/gcc/8.1.0/snos/lib64 -Wl,-rpath,/opt/cray/pe/libsci/21.08.1.2/CRAY/9.0/x86_64/lib -L/opt/cray/pe/libsci/21.08.1.2/CRAY/9.0/x86_64/lib -Wl,-rpath,/opt/cray/pe/mpich/8.1.10/ofi/cray/10.0/lib -L/opt/cray/pe/mpich/8.1.10/ofi/cray/10.0/lib -Wl,-rpath,/opt/cray/pe/dsmml/0.2.2/dsmml/lib -L/opt/cray/pe/dsmml/0.2.2/dsmml/lib -Wl,-rpath,/opt/cray/pe/pmi/6.0.14/lib -L/opt/cray/pe/pmi /6 > .0.14/li > b -Wl,-rpath,/opt/cray/pe/cce/12.0.3/cce/x86_64/lib -L/opt/cray/pe/cce/12.0.3/cce/x86_64/lib -Wl,-rpath,/opt/cray/xpmem/2.2.40-2.1_2.44__g3cf3325.shasta/lib64 -L/opt/cray/xpmem/2.2.40-2.1_2.44__g3cf3325.shasta/lib64 -Wl,-rpath,/opt/cray/pe/cce/12.0.3/cce-clang/x86_64/lib/clang/12.0.0/lib/linux -L/opt/cray/pe/cce/12.0.3/cce-clang/x86_64/lib/clang/12.0.0/lib/linux -Wl,-rpath,/opt/cray/pe/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0 -L/opt/cray/pe/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0 -Wl,-rpath,/opt/cray/pe/cce/12.0.3/binutils/x86_64/x86_64-unknown-linux-gnu/lib -L/opt/cray/pe/cce/12.0.3/binutils/x86_64/x86_64-unknown-linux-gnu/lib -lpetsc -lmagma -lkokkoskernels -lkokkoscontainers -lkokkoscore -lhipsparse -lhipblas -lrocsparse -lrocsolver -lrocblas -lrocrand -lamdhip64 -lstdc++ -ldl -lmpi_gtl_hsa -lmpifort_cray -lmpi_cray -ldsmml -lpmi -lpmi2 -lxpmem -lpgas-shmem -lquadmath -lmodules -lfi -lcraymath -lf -lu -lcsup -lgfortran -lpthread -lgcc_eh -lm -lclang_rt.cray pg > o-x86_64 > -lclang_rt.builtins-x86_64 -lquadmath -lstdc++ -ldl -lmpi_gtl_hsa -o ex5f/opt/cray/pe/cce/12.0.3/binutils/x86_64/x86_64-pc-linux-gnu/bin/ld: warning: alignment 128 of symbol `$host_init$$runtime_init_for_iso_c_binding$iso_c_binding_' in /opt/cray/pe/cce/12.0.3/cce/x86_64/lib/libmodules.so is smaller than 256 in /tmp/pe_202599/ex5f_1.o > /opt/cray/pe/cce/12.0.3/binutils/x86_64/x86_64-pc-linux-gnu/bin/ld: warning: alignment 64 of symbol `$data_init$iso_c_binding_' in /opt/cray/pe/cce/12.0.3/cce/x86_64/lib/libmodules.so is smaller than 256 in /tmp/pe_202599/ex5f_1.o > Fortran example src/snes/tutorials/ex5f run successfully with 1 MPI process > Completed test examples > [balay(a)login2.spock petsc]$ > > > On Fri, 10 Dec 2021, Mark Adams wrote: > > > FWIW, here is my current status. > > > > 08:08 main= spock:/gpfs/alpine/csc314/scratch/adams/petsc$ make > > PETSC_DIR=/gpfs/alpine/csc314/scratch/adams/petsc > > PETSC_ARCH=arch-olcf-spock check > > Running check examples to verify correct installation > > Using PETSC_DIR=/gpfs/alpine/csc314/scratch/adams/petsc and > > PETSC_ARCH=arch-olcf-spock > > Possible error running C/C++ src/snes/tutorials/ex19 with 1 MPI process > > See http://www.mcs.anl.gov/petsc/documentation/faq.html > > lid velocity = 0.0016, prandtl # = 1., grashof # = 1. > > 0 KSP Residual norm 0.0406612 > > 1 KSP Residual norm 0.036923 > > 2 KSP Residual norm 0.0191849 > > 3 KSP Residual norm 0.00201589 > > 4 KSP Residual norm 0.000376045 > > 5 KSP Residual norm 4.2974e-05 > > 6 KSP Residual norm 5.96585e-06 > > 7 KSP Residual norm 4.5398e-07 > > 8 KSP Residual norm 6.30474e-08 > > 9 KSP Residual norm 5.55518e-09 > > 10 KSP Residual norm 6.180e-10 > > 11 KSP Residual norm 6.211e-11 > > Linear solve converged due to CONVERGED_RTOL iterations 11 > > 0 KSP Residual norm 3.32845e-06 > > 1 KSP Residual norm 9.0003e-07 > > 2 KSP Residual norm 1.32594e-07 > > 3 KSP Residual norm 1.49857e-08 > > 4 KSP Residual norm 1.31887e-09 > > 5 KSP Residual norm 2.105e-10 > > 6 KSP Residual norm 2.827e-11 > > 7 KSP Residual norm < 1.e-11 > > 8 KSP Residual norm < 1.e-11 > > 9 KSP Residual norm < 1.e-11 > > 10 KSP Residual norm < 1.e-11 > > Linear solve converged due to CONVERGED_RTOL iterations 10 > > Number of SNES iterations = 2 > > Possible error running C/C++ src/snes/tutorials/ex19 with 2 MPI processes > > See http://www.mcs.anl.gov/petsc/documentation/faq.html > > lid velocity = 0.0016, prandtl # = 1., grashof # = 1. > > 0 KSP Residual norm 0.0406612 > > 1 KSP Residual norm 0.0281101 > > 2 KSP Residual norm 0.00773873 > > 3 KSP Residual norm 0.00165731 > > 4 KSP Residual norm 0.000395614 > > 5 KSP Residual norm 8.67655e-05 > > 6 KSP Residual norm 1.69495e-05 > > 7 KSP Residual norm 3.70051e-06 > > 8 KSP Residual norm 5.97067e-07 > > 9 KSP Residual norm 1.02242e-07 > > 10 KSP Residual norm 1.75727e-08 > > 11 KSP Residual norm 3.84826e-09 > > 12 KSP Residual norm 6.414e-10 > > 13 KSP Residual norm 1.380e-10 > > Linear solve converged due to CONVERGED_RTOL iterations 13 > > 0 KSP Residual norm 3.32846e-06 > > 1 KSP Residual norm 8.99139e-07 > > 2 KSP Residual norm 1.72893e-07 > > 3 KSP Residual norm 3.733e-08 > > 4 KSP Residual norm 6.67427e-09 > > 5 KSP Residual norm 1.22785e-09 > > 6 KSP Residual norm 2.551e-10 > > 7 KSP Residual norm 5.458e-11 > > 8 KSP Residual norm 1.050e-11 > > 9 KSP Residual norm < 1.e-11 > > 10 KSP Residual norm < 1.e-11 > > 11 KSP Residual norm < 1.e-11 > > 12 KSP Residual norm < 1.e-11 > > Linear solve converged due to CONVERGED_RTOL iterations 12 > > Number of SNES iterations = 2 > > 3,5c3,14 > > < 1 SNES Function norm 4.12227e-06 > > < 2 SNES Function norm 6.098e-11 > > < Number of SNES iterations = 2 > > --- > > > 0 KSP Residual norm 0.0406612 > > > 1 KSP Residual norm 0.21263 > > > 2 KSP Residual norm 1.09192 > > > 3 KSP Residual norm 6.9087 > > > 4 KSP Residual norm 23.4292 > > > 5 KSP Residual norm 57.7558 > > > 6 KSP Residual norm 118.076 > > > 7 KSP Residual norm 213.527 > > > 8 KSP Residual norm 354.101 > > > 9 KSP Residual norm 550.58 > > > Linear solve did not converge due to DIVERGED_DTOL iterations 9 > > > Number of SNES iterations = 0 > > /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials > > Possible problem with ex19 running with hypre, diffs above > > ========================================= > > gmake[3]: [makefile:115: runex3k_kokkos] Error 134 (ignored) > > 21,25c21,26 > > < 1 SNES Function norm 2.952582418265e-01 > > < 2 SNES Function norm 4.502293658739e-04 > > < 3 SNES Function norm 1.389665806646e-09 > > < Number of SNES iterations = 3 > > < Norm of error 1.49752e-10 Iterations 3 > > --- > > > Memory access fault by GPU node-4 (Agent handle: 0xb08c90) on address > > 0xe17000. Reason: Page not present or supervisor privilege. > > > Memory access fault by GPU node-5 (Agent handle: 0xb0d3c0) on address > > 0xe11000. Reason: Page not present or supervisor privilege. > > > srun: error: spock25: task 0: Aborted > > > srun: launch/slurm: _step_signal: Terminating StepId=304034.3 > > > slurmstepd: error: *** STEP 304034.3 ON spock25 CANCELLED AT > > 2021-12-10T08:08:40 *** > > > srun: error: spock25: task 1: Aborted (core dumped) > > /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials > > Possible problem with ex3k running with kokkos-kernels, diffs above > > ========================================= > > *******************Error detected during compile or link!******************* > > See http://www.mcs.anl.gov/petsc/documentation/faq.html > > /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials ex5f > > ********************************************************* > > ftn -fPIC -fPIC -I/gpfs/alpine/csc314/scratch/adams/petsc/include > > -I/gpfs/alpine/csc314/scratch/adams/petsc/arch-olcf-spock/include > > -I/gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray/include > > -I/opt/rocm-4.3.0/include ex5f.F90 > > -Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-olcf-spock/lib > > -L/gpfs/alpine/csc314/scratch/adams/petsc/arch-olcf-spock/lib > > -Wl,-rpath,/gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray/lib > > -L/gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray/lib > > -Wl,-rpath,/opt/rocm-4.3.0/lib -L/opt/rocm-4.3.0/lib > > -Wl,-rpath,/opt/cray/pe/mpich/8.1.10/gtl/lib > > -L/opt/cray/pe/mpich/8.1.10/gtl/lib > > -Wl,-rpath,/opt/cray/pe/gcc/8.1.0/snos/lib64 > > -L/opt/cray/pe/gcc/8.1.0/snos/lib64 -Wl,-rpath,/opt/cray/pe/libsci/ > > 21.08.1.2/CRAY/9.0/x86_64/lib -L/opt/cray/pe/libsci/ > > 21.08.1.2/CRAY/9.0/x86_64/lib > > -Wl,-rpath,/opt/cray/pe/mpich/8.1.10/ofi/cray/10.0/lib > > -L/opt/cray/pe/mpich/8.1.10/ofi/cray/10.0/lib > > -Wl,-rpath,/opt/cray/pe/dsmml/0.2.2/dsmml/lib > > -L/opt/cray/pe/dsmml/0.2.2/dsmml/lib -Wl,-rpath,/opt/cray/pe/pmi/6.0.14/lib > > -L/opt/cray/pe/pmi/6.0.14/lib > > -Wl,-rpath,/opt/cray/pe/cce/12.0.3/cce/x86_64/lib > > -L/opt/cray/pe/cce/12.0.3/cce/x86_64/lib > > -Wl,-rpath,/opt/cray/xpmem/2.2.40-2.1_2.44__g3cf3325.shasta/lib64 > > -L/opt/cray/xpmem/2.2.40-2.1_2.44__g3cf3325.shasta/lib64 > > -Wl,-rpath,/opt/cray/pe/cce/12.0.3/cce-clang/x86_64/lib/clang/12.0.0/lib/linux > > -L/opt/cray/pe/cce/12.0.3/cce-clang/x86_64/lib/clang/12.0.0/lib/linux > > -Wl,-rpath,/opt/cray/pe/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0 > > -L/opt/cray/pe/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0 > > -Wl,-rpath,/opt/cray/pe/cce/12.0.3/binutils/x86_64/x86_64-unknown-linux-gnu/lib > > -L/opt/cray/pe/cce/12.0.3/binutils/x86_64/x86_64-unknown-linux-gnu/lib > > -lpetsc -lHYPRE -lkokkoskernels -lkokkoscontainers -lkokkoscore -lhipsparse > > -lhipblas -lrocsparse -lrocsolver -lrocblas -lrocrand -lamdhip64 -lstdc++ > > -ldl -lmpi_gtl_hsa -lmpifort_cray -lmpi_cray -ldsmml -lpmi -lpmi2 -lxpmem > > -lpgas-shmem -lquadmath -lmodules -lfi -lcraymath -lf -lu -lcsup -lgfortran > > -lpthread -lgcc_eh -lm -lclang_rt.craypgo-x86_64 -lclang_rt.builtins-x86_64 > > -lquadmath -lstdc++ -ldl -lmpi_gtl_hsa -o ex5f > > /opt/cray/pe/cce/12.0.3/binutils/x86_64/x86_64-pc-linux-gnu/bin/ld: > > warning: alignment 128 of symbol > > `$host_init$$runtime_init_for_iso_c_binding$iso_c_binding_' in > > /opt/cray/pe/cce/12.0.3/cce/x86_64/lib/libmodules.so is smaller than 256 in > > /tmp/pe_46424/ex5f_1.o > > /opt/cray/pe/cce/12.0.3/binutils/x86_64/x86_64-pc-linux-gnu/bin/ld: > > warning: alignment 64 of symbol `$data_init$iso_c_binding_' in > > /opt/cray/pe/cce/12.0.3/cce/x86_64/lib/libmodules.so is smaller than 256 in > > /tmp/pe_46424/ex5f_1.o > > Possible error running Fortran example src/snes/tutorials/ex5f with 1 MPI > > process > > See http://www.mcs.anl.gov/petsc/documentation/faq.html > > 0 KSP Residual norm < 1.e-11 > > Linear solve converged due to CONVERGED_ATOL iterations 0 > > > > On Fri, Dec 10, 2021 at 8:07 AM Mark Adams <mfadams(a)lbl.gov> wrote: > > > > > I am trying to get Spock working (again) and am having problems. > > > > > > * make check seems to fail but it is hard to see what is going on. Maybe > > > we should start here, but let me continue. > > > > > > * GAMG seems to work on the CPU > > > > > > * I have this for configuring with Kokkos. I am guessing these versions > > > are out of data. What is current practice: > > > '--with-kokkos-hip-arch=VEGA908', > > > '--download-kokkos-commit=3.4.01', > > > '--download-kokkos-kernels-commit=3.4.01', > > > > > > * Should I hold off (and tell my eager user to do same)? > > > > > > Thanks, > > > Mark > > > > > >

1 0

Re: [petsc-dev] spock
by Satish Balay 10 Dec '21

10 Dec '21

Works for me [per instructions in balay/update-spock, config/examples/arch-olcf-spock.py] with main - without these additional options I'll go ahead and merge in balay/update-spock Satish ----- 1009 git fetch -p 1015 module load emacs 1016 module load rocm/4.3.0 1018 git reset --hard 1019 git checkout origin/main 1020 git merge origin/balay/update-spock 1021 ./config/examples/arch-olcf-spock.py && make [balay(a)login2.spock petsc]$ MPIR_CVAR_GPU_EAGER_DEVICE_MEM=0 MPICH_GPU_SUPPORT_ENABLED=1 MPICH_SMP_SINGLE_COPY_MODE=CMA make check Running check examples to verify correct installation Using PETSC_DIR=/autofs/nccs-svm1_home1/balay/petsc and PETSC_ARCH=arch-olcf-spock C/C++ example src/snes/tutorials/ex19 run successfully with 1 MPI process C/C++ example src/snes/tutorials/ex19 run successfully with 2 MPI processes C/C++ example src/snes/tutorials/ex3k run successfully with kokkos-kernels *******************Error detected during compile or link!******************* See http://www.mcs.anl.gov/petsc/documentation/faq.html /ccs/home/balay/petsc/src/snes/tutorials ex5f ********************************************************* ftn -fPIC -fPIC -I/autofs/nccs-svm1_home1/balay/petsc/include -I/autofs/nccs-svm1_home1/balay/petsc/arch-olcf-spock/include -I/opt/rocm-4.3.0/include ex5f.F90 -Wl,-rpath,/autofs/nccs-svm1_home1/balay/petsc/arch-olcf-spock/lib -L/autofs/nccs-svm1_home1/balay/petsc/arch-olcf-spock/lib -Wl,-rpath,/autofs/nccs-svm1_home1/balay/petsc/arch-olcf-spock/lib -L/autofs/nccs-svm1_home1/balay/petsc/arch-olcf-spock/lib -Wl,-rpath,/opt/rocm-4.3.0/lib -L/opt/rocm-4.3.0/lib -Wl,-rpath,/opt/cray/pe/mpich/8.1.10/gtl/lib -L/opt/cray/pe/mpich/8.1.10/gtl/lib -Wl,-rpath,/opt/cray/pe/gcc/8.1.0/snos/lib64 -L/opt/cray/pe/gcc/8.1.0/snos/lib64 -Wl,-rpath,/opt/cray/pe/libsci/21.08.1.2/CRAY/9.0/x86_64/lib -L/opt/cray/pe/libsci/21.08.1.2/CRAY/9.0/x86_64/lib -Wl,-rpath,/opt/cray/pe/mpich/8.1.10/ofi/cray/10.0/lib -L/opt/cray/pe/mpich/8.1.10/ofi/cray/10.0/lib -Wl,-rpath,/opt/cray/pe/dsmml/0.2.2/dsmml/lib -L/opt/cray/pe/dsmml/0.2.2/dsmml/lib -Wl,-rpath,/opt/cray/pe/pmi/6.0.14/lib -L/opt/cray/pe/pmi/6 .0.14/li b -Wl,-rpath,/opt/cray/pe/cce/12.0.3/cce/x86_64/lib -L/opt/cray/pe/cce/12.0.3/cce/x86_64/lib -Wl,-rpath,/opt/cray/xpmem/2.2.40-2.1_2.44__g3cf3325.shasta/lib64 -L/opt/cray/xpmem/2.2.40-2.1_2.44__g3cf3325.shasta/lib64 -Wl,-rpath,/opt/cray/pe/cce/12.0.3/cce-clang/x86_64/lib/clang/12.0.0/lib/linux -L/opt/cray/pe/cce/12.0.3/cce-clang/x86_64/lib/clang/12.0.0/lib/linux -Wl,-rpath,/opt/cray/pe/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0 -L/opt/cray/pe/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0 -Wl,-rpath,/opt/cray/pe/cce/12.0.3/binutils/x86_64/x86_64-unknown-linux-gnu/lib -L/opt/cray/pe/cce/12.0.3/binutils/x86_64/x86_64-unknown-linux-gnu/lib -lpetsc -lmagma -lkokkoskernels -lkokkoscontainers -lkokkoscore -lhipsparse -lhipblas -lrocsparse -lrocsolver -lrocblas -lrocrand -lamdhip64 -lstdc++ -ldl -lmpi_gtl_hsa -lmpifort_cray -lmpi_cray -ldsmml -lpmi -lpmi2 -lxpmem -lpgas-shmem -lquadmath -lmodules -lfi -lcraymath -lf -lu -lcsup -lgfortran -lpthread -lgcc_eh -lm -lclang_rt.craypg o-x86_64 -lclang_rt.builtins-x86_64 -lquadmath -lstdc++ -ldl -lmpi_gtl_hsa -o ex5f/opt/cray/pe/cce/12.0.3/binutils/x86_64/x86_64-pc-linux-gnu/bin/ld: warning: alignment 128 of symbol `$host_init$$runtime_init_for_iso_c_binding$iso_c_binding_' in /opt/cray/pe/cce/12.0.3/cce/x86_64/lib/libmodules.so is smaller than 256 in /tmp/pe_202599/ex5f_1.o /opt/cray/pe/cce/12.0.3/binutils/x86_64/x86_64-pc-linux-gnu/bin/ld: warning: alignment 64 of symbol `$data_init$iso_c_binding_' in /opt/cray/pe/cce/12.0.3/cce/x86_64/lib/libmodules.so is smaller than 256 in /tmp/pe_202599/ex5f_1.o Fortran example src/snes/tutorials/ex5f run successfully with 1 MPI process Completed test examples [balay(a)login2.spock petsc]$ On Fri, 10 Dec 2021, Mark Adams wrote: > FWIW, here is my current status. > > 08:08 main= spock:/gpfs/alpine/csc314/scratch/adams/petsc$ make > PETSC_DIR=/gpfs/alpine/csc314/scratch/adams/petsc > PETSC_ARCH=arch-olcf-spock check > Running check examples to verify correct installation > Using PETSC_DIR=/gpfs/alpine/csc314/scratch/adams/petsc and > PETSC_ARCH=arch-olcf-spock > Possible error running C/C++ src/snes/tutorials/ex19 with 1 MPI process > See http://www.mcs.anl.gov/petsc/documentation/faq.html > lid velocity = 0.0016, prandtl # = 1., grashof # = 1. > 0 KSP Residual norm 0.0406612 > 1 KSP Residual norm 0.036923 > 2 KSP Residual norm 0.0191849 > 3 KSP Residual norm 0.00201589 > 4 KSP Residual norm 0.000376045 > 5 KSP Residual norm 4.2974e-05 > 6 KSP Residual norm 5.96585e-06 > 7 KSP Residual norm 4.5398e-07 > 8 KSP Residual norm 6.30474e-08 > 9 KSP Residual norm 5.55518e-09 > 10 KSP Residual norm 6.180e-10 > 11 KSP Residual norm 6.211e-11 > Linear solve converged due to CONVERGED_RTOL iterations 11 > 0 KSP Residual norm 3.32845e-06 > 1 KSP Residual norm 9.0003e-07 > 2 KSP Residual norm 1.32594e-07 > 3 KSP Residual norm 1.49857e-08 > 4 KSP Residual norm 1.31887e-09 > 5 KSP Residual norm 2.105e-10 > 6 KSP Residual norm 2.827e-11 > 7 KSP Residual norm < 1.e-11 > 8 KSP Residual norm < 1.e-11 > 9 KSP Residual norm < 1.e-11 > 10 KSP Residual norm < 1.e-11 > Linear solve converged due to CONVERGED_RTOL iterations 10 > Number of SNES iterations = 2 > Possible error running C/C++ src/snes/tutorials/ex19 with 2 MPI processes > See http://www.mcs.anl.gov/petsc/documentation/faq.html > lid velocity = 0.0016, prandtl # = 1., grashof # = 1. > 0 KSP Residual norm 0.0406612 > 1 KSP Residual norm 0.0281101 > 2 KSP Residual norm 0.00773873 > 3 KSP Residual norm 0.00165731 > 4 KSP Residual norm 0.000395614 > 5 KSP Residual norm 8.67655e-05 > 6 KSP Residual norm 1.69495e-05 > 7 KSP Residual norm 3.70051e-06 > 8 KSP Residual norm 5.97067e-07 > 9 KSP Residual norm 1.02242e-07 > 10 KSP Residual norm 1.75727e-08 > 11 KSP Residual norm 3.84826e-09 > 12 KSP Residual norm 6.414e-10 > 13 KSP Residual norm 1.380e-10 > Linear solve converged due to CONVERGED_RTOL iterations 13 > 0 KSP Residual norm 3.32846e-06 > 1 KSP Residual norm 8.99139e-07 > 2 KSP Residual norm 1.72893e-07 > 3 KSP Residual norm 3.733e-08 > 4 KSP Residual norm 6.67427e-09 > 5 KSP Residual norm 1.22785e-09 > 6 KSP Residual norm 2.551e-10 > 7 KSP Residual norm 5.458e-11 > 8 KSP Residual norm 1.050e-11 > 9 KSP Residual norm < 1.e-11 > 10 KSP Residual norm < 1.e-11 > 11 KSP Residual norm < 1.e-11 > 12 KSP Residual norm < 1.e-11 > Linear solve converged due to CONVERGED_RTOL iterations 12 > Number of SNES iterations = 2 > 3,5c3,14 > < 1 SNES Function norm 4.12227e-06 > < 2 SNES Function norm 6.098e-11 > < Number of SNES iterations = 2 > --- > > 0 KSP Residual norm 0.0406612 > > 1 KSP Residual norm 0.21263 > > 2 KSP Residual norm 1.09192 > > 3 KSP Residual norm 6.9087 > > 4 KSP Residual norm 23.4292 > > 5 KSP Residual norm 57.7558 > > 6 KSP Residual norm 118.076 > > 7 KSP Residual norm 213.527 > > 8 KSP Residual norm 354.101 > > 9 KSP Residual norm 550.58 > > Linear solve did not converge due to DIVERGED_DTOL iterations 9 > > Number of SNES iterations = 0 > /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials > Possible problem with ex19 running with hypre, diffs above > ========================================= > gmake[3]: [makefile:115: runex3k_kokkos] Error 134 (ignored) > 21,25c21,26 > < 1 SNES Function norm 2.952582418265e-01 > < 2 SNES Function norm 4.502293658739e-04 > < 3 SNES Function norm 1.389665806646e-09 > < Number of SNES iterations = 3 > < Norm of error 1.49752e-10 Iterations 3 > --- > > Memory access fault by GPU node-4 (Agent handle: 0xb08c90) on address > 0xe17000. Reason: Page not present or supervisor privilege. > > Memory access fault by GPU node-5 (Agent handle: 0xb0d3c0) on address > 0xe11000. Reason: Page not present or supervisor privilege. > > srun: error: spock25: task 0: Aborted > > srun: launch/slurm: _step_signal: Terminating StepId=304034.3 > > slurmstepd: error: *** STEP 304034.3 ON spock25 CANCELLED AT > 2021-12-10T08:08:40 *** > > srun: error: spock25: task 1: Aborted (core dumped) > /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials > Possible problem with ex3k running with kokkos-kernels, diffs above > ========================================= > *******************Error detected during compile or link!******************* > See http://www.mcs.anl.gov/petsc/documentation/faq.html > /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials ex5f > ********************************************************* > ftn -fPIC -fPIC -I/gpfs/alpine/csc314/scratch/adams/petsc/include > -I/gpfs/alpine/csc314/scratch/adams/petsc/arch-olcf-spock/include > -I/gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray/include > -I/opt/rocm-4.3.0/include ex5f.F90 > -Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-olcf-spock/lib > -L/gpfs/alpine/csc314/scratch/adams/petsc/arch-olcf-spock/lib > -Wl,-rpath,/gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray/lib > -L/gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray/lib > -Wl,-rpath,/opt/rocm-4.3.0/lib -L/opt/rocm-4.3.0/lib > -Wl,-rpath,/opt/cray/pe/mpich/8.1.10/gtl/lib > -L/opt/cray/pe/mpich/8.1.10/gtl/lib > -Wl,-rpath,/opt/cray/pe/gcc/8.1.0/snos/lib64 > -L/opt/cray/pe/gcc/8.1.0/snos/lib64 -Wl,-rpath,/opt/cray/pe/libsci/ > 21.08.1.2/CRAY/9.0/x86_64/lib -L/opt/cray/pe/libsci/ > 21.08.1.2/CRAY/9.0/x86_64/lib > -Wl,-rpath,/opt/cray/pe/mpich/8.1.10/ofi/cray/10.0/lib > -L/opt/cray/pe/mpich/8.1.10/ofi/cray/10.0/lib > -Wl,-rpath,/opt/cray/pe/dsmml/0.2.2/dsmml/lib > -L/opt/cray/pe/dsmml/0.2.2/dsmml/lib -Wl,-rpath,/opt/cray/pe/pmi/6.0.14/lib > -L/opt/cray/pe/pmi/6.0.14/lib > -Wl,-rpath,/opt/cray/pe/cce/12.0.3/cce/x86_64/lib > -L/opt/cray/pe/cce/12.0.3/cce/x86_64/lib > -Wl,-rpath,/opt/cray/xpmem/2.2.40-2.1_2.44__g3cf3325.shasta/lib64 > -L/opt/cray/xpmem/2.2.40-2.1_2.44__g3cf3325.shasta/lib64 > -Wl,-rpath,/opt/cray/pe/cce/12.0.3/cce-clang/x86_64/lib/clang/12.0.0/lib/linux > -L/opt/cray/pe/cce/12.0.3/cce-clang/x86_64/lib/clang/12.0.0/lib/linux > -Wl,-rpath,/opt/cray/pe/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0 > -L/opt/cray/pe/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0 > -Wl,-rpath,/opt/cray/pe/cce/12.0.3/binutils/x86_64/x86_64-unknown-linux-gnu/lib > -L/opt/cray/pe/cce/12.0.3/binutils/x86_64/x86_64-unknown-linux-gnu/lib > -lpetsc -lHYPRE -lkokkoskernels -lkokkoscontainers -lkokkoscore -lhipsparse > -lhipblas -lrocsparse -lrocsolver -lrocblas -lrocrand -lamdhip64 -lstdc++ > -ldl -lmpi_gtl_hsa -lmpifort_cray -lmpi_cray -ldsmml -lpmi -lpmi2 -lxpmem > -lpgas-shmem -lquadmath -lmodules -lfi -lcraymath -lf -lu -lcsup -lgfortran > -lpthread -lgcc_eh -lm -lclang_rt.craypgo-x86_64 -lclang_rt.builtins-x86_64 > -lquadmath -lstdc++ -ldl -lmpi_gtl_hsa -o ex5f > /opt/cray/pe/cce/12.0.3/binutils/x86_64/x86_64-pc-linux-gnu/bin/ld: > warning: alignment 128 of symbol > `$host_init$$runtime_init_for_iso_c_binding$iso_c_binding_' in > /opt/cray/pe/cce/12.0.3/cce/x86_64/lib/libmodules.so is smaller than 256 in > /tmp/pe_46424/ex5f_1.o > /opt/cray/pe/cce/12.0.3/binutils/x86_64/x86_64-pc-linux-gnu/bin/ld: > warning: alignment 64 of symbol `$data_init$iso_c_binding_' in > /opt/cray/pe/cce/12.0.3/cce/x86_64/lib/libmodules.so is smaller than 256 in > /tmp/pe_46424/ex5f_1.o > Possible error running Fortran example src/snes/tutorials/ex5f with 1 MPI > process > See http://www.mcs.anl.gov/petsc/documentation/faq.html > 0 KSP Residual norm < 1.e-11 > Linear solve converged due to CONVERGED_ATOL iterations 0 > > On Fri, Dec 10, 2021 at 8:07 AM Mark Adams <mfadams(a)lbl.gov> wrote: > > > I am trying to get Spock working (again) and am having problems. > > > > * make check seems to fail but it is hard to see what is going on. Maybe > > we should start here, but let me continue. > > > > * GAMG seems to work on the CPU > > > > * I have this for configuring with Kokkos. I am guessing these versions > > are out of data. What is current practice: > > '--with-kokkos-hip-arch=VEGA908', > > '--download-kokkos-commit=3.4.01', > > '--download-kokkos-kernels-commit=3.4.01', > > > > * Should I hold off (and tell my eager user to do same)? > > > > Thanks, > > Mark > > >

1 0

Re: [petsc-dev] spock
by Mark Adams 10 Dec '21

10 Dec '21

FWIW, here is my current status. 08:08 main= spock:/gpfs/alpine/csc314/scratch/adams/petsc$ make PETSC_DIR=/gpfs/alpine/csc314/scratch/adams/petsc PETSC_ARCH=arch-olcf-spock check Running check examples to verify correct installation Using PETSC_DIR=/gpfs/alpine/csc314/scratch/adams/petsc and PETSC_ARCH=arch-olcf-spock Possible error running C/C++ src/snes/tutorials/ex19 with 1 MPI process See http://www.mcs.anl.gov/petsc/documentation/faq.html lid velocity = 0.0016, prandtl # = 1., grashof # = 1. 0 KSP Residual norm 0.0406612 1 KSP Residual norm 0.036923 2 KSP Residual norm 0.0191849 3 KSP Residual norm 0.00201589 4 KSP Residual norm 0.000376045 5 KSP Residual norm 4.2974e-05 6 KSP Residual norm 5.96585e-06 7 KSP Residual norm 4.5398e-07 8 KSP Residual norm 6.30474e-08 9 KSP Residual norm 5.55518e-09 10 KSP Residual norm 6.180e-10 11 KSP Residual norm 6.211e-11 Linear solve converged due to CONVERGED_RTOL iterations 11 0 KSP Residual norm 3.32845e-06 1 KSP Residual norm 9.0003e-07 2 KSP Residual norm 1.32594e-07 3 KSP Residual norm 1.49857e-08 4 KSP Residual norm 1.31887e-09 5 KSP Residual norm 2.105e-10 6 KSP Residual norm 2.827e-11 7 KSP Residual norm < 1.e-11 8 KSP Residual norm < 1.e-11 9 KSP Residual norm < 1.e-11 10 KSP Residual norm < 1.e-11 Linear solve converged due to CONVERGED_RTOL iterations 10 Number of SNES iterations = 2 Possible error running C/C++ src/snes/tutorials/ex19 with 2 MPI processes See http://www.mcs.anl.gov/petsc/documentation/faq.html lid velocity = 0.0016, prandtl # = 1., grashof # = 1. 0 KSP Residual norm 0.0406612 1 KSP Residual norm 0.0281101 2 KSP Residual norm 0.00773873 3 KSP Residual norm 0.00165731 4 KSP Residual norm 0.000395614 5 KSP Residual norm 8.67655e-05 6 KSP Residual norm 1.69495e-05 7 KSP Residual norm 3.70051e-06 8 KSP Residual norm 5.97067e-07 9 KSP Residual norm 1.02242e-07 10 KSP Residual norm 1.75727e-08 11 KSP Residual norm 3.84826e-09 12 KSP Residual norm 6.414e-10 13 KSP Residual norm 1.380e-10 Linear solve converged due to CONVERGED_RTOL iterations 13 0 KSP Residual norm 3.32846e-06 1 KSP Residual norm 8.99139e-07 2 KSP Residual norm 1.72893e-07 3 KSP Residual norm 3.733e-08 4 KSP Residual norm 6.67427e-09 5 KSP Residual norm 1.22785e-09 6 KSP Residual norm 2.551e-10 7 KSP Residual norm 5.458e-11 8 KSP Residual norm 1.050e-11 9 KSP Residual norm < 1.e-11 10 KSP Residual norm < 1.e-11 11 KSP Residual norm < 1.e-11 12 KSP Residual norm < 1.e-11 Linear solve converged due to CONVERGED_RTOL iterations 12 Number of SNES iterations = 2 3,5c3,14 < 1 SNES Function norm 4.12227e-06 < 2 SNES Function norm 6.098e-11 < Number of SNES iterations = 2 --- > 0 KSP Residual norm 0.0406612 > 1 KSP Residual norm 0.21263 > 2 KSP Residual norm 1.09192 > 3 KSP Residual norm 6.9087 > 4 KSP Residual norm 23.4292 > 5 KSP Residual norm 57.7558 > 6 KSP Residual norm 118.076 > 7 KSP Residual norm 213.527 > 8 KSP Residual norm 354.101 > 9 KSP Residual norm 550.58 > Linear solve did not converge due to DIVERGED_DTOL iterations 9 > Number of SNES iterations = 0 /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials Possible problem with ex19 running with hypre, diffs above ========================================= gmake[3]: [makefile:115: runex3k_kokkos] Error 134 (ignored) 21,25c21,26 < 1 SNES Function norm 2.952582418265e-01 < 2 SNES Function norm 4.502293658739e-04 < 3 SNES Function norm 1.389665806646e-09 < Number of SNES iterations = 3 < Norm of error 1.49752e-10 Iterations 3 --- > Memory access fault by GPU node-4 (Agent handle: 0xb08c90) on address 0xe17000. Reason: Page not present or supervisor privilege. > Memory access fault by GPU node-5 (Agent handle: 0xb0d3c0) on address 0xe11000. Reason: Page not present or supervisor privilege. > srun: error: spock25: task 0: Aborted > srun: launch/slurm: _step_signal: Terminating StepId=304034.3 > slurmstepd: error: *** STEP 304034.3 ON spock25 CANCELLED AT 2021-12-10T08:08:40 *** > srun: error: spock25: task 1: Aborted (core dumped) /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials Possible problem with ex3k running with kokkos-kernels, diffs above ========================================= *******************Error detected during compile or link!******************* See http://www.mcs.anl.gov/petsc/documentation/faq.html /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials ex5f ********************************************************* ftn -fPIC -fPIC -I/gpfs/alpine/csc314/scratch/adams/petsc/include -I/gpfs/alpine/csc314/scratch/adams/petsc/arch-olcf-spock/include -I/gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray/include -I/opt/rocm-4.3.0/include ex5f.F90 -Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-olcf-spock/lib -L/gpfs/alpine/csc314/scratch/adams/petsc/arch-olcf-spock/lib -Wl,-rpath,/gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray/lib -L/gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray/lib -Wl,-rpath,/opt/rocm-4.3.0/lib -L/opt/rocm-4.3.0/lib -Wl,-rpath,/opt/cray/pe/mpich/8.1.10/gtl/lib -L/opt/cray/pe/mpich/8.1.10/gtl/lib -Wl,-rpath,/opt/cray/pe/gcc/8.1.0/snos/lib64 -L/opt/cray/pe/gcc/8.1.0/snos/lib64 -Wl,-rpath,/opt/cray/pe/libsci/ 21.08.1.2/CRAY/9.0/x86_64/lib -L/opt/cray/pe/libsci/ 21.08.1.2/CRAY/9.0/x86_64/lib -Wl,-rpath,/opt/cray/pe/mpich/8.1.10/ofi/cray/10.0/lib -L/opt/cray/pe/mpich/8.1.10/ofi/cray/10.0/lib -Wl,-rpath,/opt/cray/pe/dsmml/0.2.2/dsmml/lib -L/opt/cray/pe/dsmml/0.2.2/dsmml/lib -Wl,-rpath,/opt/cray/pe/pmi/6.0.14/lib -L/opt/cray/pe/pmi/6.0.14/lib -Wl,-rpath,/opt/cray/pe/cce/12.0.3/cce/x86_64/lib -L/opt/cray/pe/cce/12.0.3/cce/x86_64/lib -Wl,-rpath,/opt/cray/xpmem/2.2.40-2.1_2.44__g3cf3325.shasta/lib64 -L/opt/cray/xpmem/2.2.40-2.1_2.44__g3cf3325.shasta/lib64 -Wl,-rpath,/opt/cray/pe/cce/12.0.3/cce-clang/x86_64/lib/clang/12.0.0/lib/linux -L/opt/cray/pe/cce/12.0.3/cce-clang/x86_64/lib/clang/12.0.0/lib/linux -Wl,-rpath,/opt/cray/pe/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0 -L/opt/cray/pe/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0 -Wl,-rpath,/opt/cray/pe/cce/12.0.3/binutils/x86_64/x86_64-unknown-linux-gnu/lib -L/opt/cray/pe/cce/12.0.3/binutils/x86_64/x86_64-unknown-linux-gnu/lib -lpetsc -lHYPRE -lkokkoskernels -lkokkoscontainers -lkokkoscore -lhipsparse -lhipblas -lrocsparse -lrocsolver -lrocblas -lrocrand -lamdhip64 -lstdc++ -ldl -lmpi_gtl_hsa -lmpifort_cray -lmpi_cray -ldsmml -lpmi -lpmi2 -lxpmem -lpgas-shmem -lquadmath -lmodules -lfi -lcraymath -lf -lu -lcsup -lgfortran -lpthread -lgcc_eh -lm -lclang_rt.craypgo-x86_64 -lclang_rt.builtins-x86_64 -lquadmath -lstdc++ -ldl -lmpi_gtl_hsa -o ex5f /opt/cray/pe/cce/12.0.3/binutils/x86_64/x86_64-pc-linux-gnu/bin/ld: warning: alignment 128 of symbol `$host_init$$runtime_init_for_iso_c_binding$iso_c_binding_' in /opt/cray/pe/cce/12.0.3/cce/x86_64/lib/libmodules.so is smaller than 256 in /tmp/pe_46424/ex5f_1.o /opt/cray/pe/cce/12.0.3/binutils/x86_64/x86_64-pc-linux-gnu/bin/ld: warning: alignment 64 of symbol `$data_init$iso_c_binding_' in /opt/cray/pe/cce/12.0.3/cce/x86_64/lib/libmodules.so is smaller than 256 in /tmp/pe_46424/ex5f_1.o Possible error running Fortran example src/snes/tutorials/ex5f with 1 MPI process See http://www.mcs.anl.gov/petsc/documentation/faq.html 0 KSP Residual norm < 1.e-11 Linear solve converged due to CONVERGED_ATOL iterations 0 On Fri, Dec 10, 2021 at 8:07 AM Mark Adams <mfadams(a)lbl.gov> wrote: > I am trying to get Spock working (again) and am having problems. > > * make check seems to fail but it is hard to see what is going on. Maybe > we should start here, but let me continue. > > * GAMG seems to work on the CPU > > * I have this for configuring with Kokkos. I am guessing these versions > are out of data. What is current practice: > '--with-kokkos-hip-arch=VEGA908', > '--download-kokkos-commit=3.4.01', > '--download-kokkos-kernels-commit=3.4.01', > > * Should I hold off (and tell my eager user to do same)? > > Thanks, > Mark >

1 0

spock
by Mark Adams 10 Dec '21

10 Dec '21

I am trying to get Spock working (again) and am having problems. * make check seems to fail but it is hard to see what is going on. Maybe we should start here, but let me continue. * GAMG seems to work on the CPU * I have this for configuring with Kokkos. I am guessing these versions are out of data. What is current practice: '--with-kokkos-hip-arch=VEGA908', '--download-kokkos-commit=3.4.01', '--download-kokkos-kernels-commit=3.4.01', * Should I hold off (and tell my eager user to do same)? Thanks, Mark

1 0

Re: [petsc-dev] Hypre error on updated Perlmutter
by Mark Adams 09 Dec '21

09 Dec '21

Well I found, accidentally, that turning CUDA aware MPI on with export CRAY_ACCEL_TARGET=nvidia80 seems to have fixed this. Not sure what is going on. On Thu, Dec 9, 2021 at 11:21 AM Mark Adams <mfadams(a)lbl.gov> wrote: > I am getting this error. I have built this w/o hypre and the test are > fine, including the CUDA tests. > Any ideas? > > I notice that the tests use -dm_mat_type aijcusparse with hypre. > > Thanks, > Mark > > 08:13 nid003929 adams/fix_mat_ex5k= perlmutter:~/petsc$ make > PETSC_DIR=/global/homes/m/madams/petsc > PETSC_ARCH=arch-perlmutter-opt-nvidia-cuda check > Running check examples to verify correct installation > Using PETSC_DIR=/global/homes/m/madams/petsc and > PETSC_ARCH=arch-perlmutter-opt-nvidia-cuda > C/C++ example src/snes/tutorials/ex19 run successfully with 1 MPI process > C/C++ example src/snes/tutorials/ex19 run successfully with 2 MPI processes > 1,5c1,70 > < lid velocity = 0.0016, prandtl # = 1., grashof # = 1. > < 0 SNES Function norm 0.0406612 > < 1 SNES Function norm 4.12227e-06 > < 2 SNES Function norm 6.098e-11 > < Number of SNES iterations = 2 > --- > > [0]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > > [0]PETSC ERROR: GPU error > > [0]PETSC ERROR: cuda error 46 (cudaErrorDevicesUnavailable) : all > CUDA-capable devices are busy or unavailable > > [0]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting. > > [0]PETSC ERROR: Petsc Development GIT revision: v3.16.1-442-gebb4a459f5 > GIT Date: 2021-12-08 08:59:23 -0500 > > [0]PETSC ERROR: /global/u2/m/madams/petsc/src/snes/tutorials/./ex19 on a > named nid003929 by madams Thu Dec 9 08:13:49 2021 > > [0]PETSC ERROR: Configure options --CFLAGS=" -g -DLANDAU_DIM=2 > -DLANDAU_MAX_SPECIES=10 -DLANDAU_MAX_Q=4 -mp=gpu" --CXXFLAGS=" -g > -DLANDAU_DIM=2 -DLANDAU_MAX_SPECIES=10 -DLANDAU_MAX_Q=4 -mp=gpu" --FFLAGS=" > -g -mp=gpu" --with-cc=cc --with-cxx=CC --with-fc=ftn > --with-cudac=/opt/nvidia/hpc_sdk/Linux_x86_64/21.9/compilers/bin/nvcc > --with-debugging=0 --download-hypre=1 --with-cuda=1 --with-cuda-arch=80 > --with-mpiexec=srun --with-batch=0 --download-p4est=1 --with-zlib=1 > --with-make-np=8 > --prefix=/global/cfs/projectdirs/m3904/petsc/current/perlmutter-opt-nvidia21.9 > PETSC_ARCH=arch-perlmutter-opt-nvidia-cuda > > [0]PETSC ERROR: #1 initialize() at > /global/u2/m/madams/petsc/src/sys/objects/device/impls/cupm/cupmdevice.cxx:70 > > [0]PETSC ERROR: #2 getDevice() at > /global/u2/m/madams/petsc/src/sys/objects/device/impls/cupm/cupmdevice.cxx:360 > > [0]PETSC ERROR: #3 PetscDeviceCreate() at > /global/u2/m/madams/petsc/src/sys/objects/device/interface/device.cxx:102 > > [0]PETSC ERROR: #4 PetscDeviceInitializeDefaultDevice_Internal() at > /global/u2/m/madams/petsc/src/sys/objects/device/interface/device.cxx:266 > > [1]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > > [1]PETSC ERROR: GPU error > > [1]PETSC ERROR: cuda error 46 (cudaErrorDevicesUnavailable) : all > CUDA-capable devices are busy or unavailable > > [1]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting. > > [1]PETSC ERROR: Petsc Development GIT revision: v3.16.1-442-gebb4a459f5 > GIT Date: 2021-12-08 08:59:23 -0500 > > [1]PETSC ERROR: /global/u2/m/madams/petsc/src/snes/tutorials/./ex19 on a > named nid003929 by madams Thu Dec 9 08:13:49 2021 > > [1]PETSC ERROR: Configure options --CFLAGS=" -g -DLANDAU_DIM=2 > -DLANDAU_MAX_SPECIES=10 -DLANDAU_MAX_Q=4 -mp=gpu" --CXXFLAGS=" -g > -DLANDAU_DIM=2 -DLANDAU_MAX_SPECIES=10 -DLANDAU_MAX_Q=4 -mp=gpu" --FFLAGS=" > -g -mp=gpu" --with-cc=cc --with-cxx=CC --with-fc=ftn > --with-cudac=/opt/nvidia/hpc_sdk/Linux_x86_64/21.9/compilers/bin/nvcc > --with-debugging=0 --download-hypre=1 --with-cuda=1 --with-cuda-arch=80 > --with-mpiexec=srun --with-batch=0 --download-p4est=1 --with-zlib=1 > --with-make-np=8 > --prefix=/global/cfs/projectdirs/m3904/petsc/current/perlmutter-opt-nvidia21.9 > PETSC_ARCH=arch-perlmutter-opt-nvidia-cuda > > [1]PETSC ERROR: #1 initialize() at > /global/u2/m/madams/petsc/src/sys/objects/device/impls/cupm/cupmdevice.cxx:70 > > [1]PETSC ERROR: #2 getDevice() at > /global/u2/m/madams/petsc/src/sys/objects/device/impls/cupm/cupmdevice.cxx:360 > > [1]PETSC ERROR: #3 PetscDeviceCreate() at > /global/u2/m/madams/petsc/src/sys/objects/device/interface/device.cxx:102 > > [1]PETSC ERROR: #4 PetscDeviceInitializeDefaultDevice_Internal() at > /global/u2/m/madams/petsc/src/sys/objects/device/interface/device.cxx:266 > > [1]PETSC ERROR: #5 PetscDeviceInitialize() at > /global/u2/m/madams/petsc/src/sys/objects/device/interface/device.cxx:227 > > [1]PETSC ERROR: #6 PCCreate_HYPRE() at > /global/u2/m/madams/petsc/src/ksp/pc/impls/hypre/hypre.c:2224 > > [1]PETSC ERROR: #7 PCSetType() at > /global/u2/m/madams/petsc/src/ksp/pc/interface/pcset.c:84 > > [1]PETSC ERROR: #8 PCSetFromOptions() at > /global/u2/m/madams/petsc/src/ksp/pc/interface/pcset.c:154 > > [1]PETSC ERROR: #9 KSPSetFromOptions() at > /global/u2/m/madams/petsc/src/ksp/ksp/interface/itcl.c:356 > > [1]PETSC ERROR: #10 SNESSetFromOptions() at > /global/u2/m/madams/petsc/src/snes/interface/snes.c:1113 > > [1]PETSC ERROR: #11 main() at ex19.c:150 > > [1]PETSC ERROR: PETSc Option Table entries: > > [1]PETSC ERROR: -da_refine 3 > > [1]PETSC ERROR: -dm_mat_type aijcusparse > > [1]PETSC ERROR: -dm_vec_type cuda > > [0]PETSC ERROR: #5 PetscDeviceInitialize() at > /global/u2/m/madams/petsc/src/sys/objects/device/interface/device.cxx:227 > > [0]PETSC ERROR: #6 PCCreate_HYPRE() at > /global/u2/m/madams/petsc/src/ksp/pc/impls/hypre/hypre.c:2224 > > [0]PETSC ERROR: #7 PCSetType() at > /global/u2/m/madams/petsc/src/ksp/pc/interface/pcset.c:84 > > [0]PETSC ERROR: #8 PCSetFromOptions() at > /global/u2/m/madams/petsc/src/ksp/pc/interface/pcset.c:154 > > [0]PETSC ERROR: #9 KSPSetFromOptions() at > /global/u2/m/madams/petsc/src/ksp/ksp/interface/itcl.c:356 > > [0]PETSC ERROR: #10 SNESSetFromOptions() at > /global/u2/m/madams/petsc/src/snes/interface/snes.c:1113 > > [0]PETSC ERROR: #11 main() at ex19.c:150 > > [0]PETSC ERROR: PETSc Option Table entries: > > [0]PETSC ERROR: -da_refine 3 > > [0]PETSC ERROR: -dm_mat_type aijcusparse > > [0]PETSC ERROR: -dm_vec_type cuda > > [0]PETSC ERROR: -ksp_norm_type unpreconditioned > > [0]PETSC ERROR: -nox > > [0]PETSC ERROR: -nox_warning > > [0]PETSC ERROR: -pc_type hypre > > [0]PETSC ERROR: -snes_monitor_short > > [0]PETSC ERROR: -use_gpu_aware_mpi 0 > > [0]PETSC ERROR: ----------------End of Error Message -------send entire > error message to petsc-maint(a)mcs.anl.gov---------- > > [1]PETSC ERROR: -ksp_norm_type unpreconditioned > > [1]PETSC ERROR: -nox > > [1]PETSC ERROR: -nox_warning > > [1]PETSC ERROR: -pc_type hypre > > [1]PETSC ERROR: -snes_monitor_short > > [1]PETSC ERROR: -use_gpu_aware_mpi 0 > > [1]PETSC ERROR: ----------------End of Error Message -------send entire > error message to petsc-maint(a)mcs.anl.gov---------- > > MPICH Notice [Rank 0] [job id 832277.2] [Thu Dec 9 08:13:50 2021] > [nid003929] - Abort(97) (rank 0 in comm 0): application called > MPI_Abort(MPI_COMM_WORLD, 97) - process 0 > > > > aborting job: > > application called MPI_Abort(MPI_COMM_WORLD, 97) - process 0 > > MPICH Notice [Rank 1] [job id 832277.2] [Thu Dec 9 08:13:50 2021] > [nid003929] - Abort(97) (rank 1 in comm 0): application called > MPI_Abort(MPI_COMM_WORLD, 97) - process 1 > > > > aborting job: > > application called MPI_Abort(MPI_COMM_WORLD, 97) - process 1 > > srun: error: nid003929: task 1: Exited with exit code 255 > > srun: launch/slurm: _step_signal: Terminating StepId=832277.2 > > slurmstepd: error: *** STEP 832277.2 ON nid003929 CANCELLED AT > 2021-12-09T16:13:50 *** > > srun: error: nid003929: task 0: Exited with exit code 255 > /global/homes/m/madams/petsc/src/snes/tutorials > Possible problem with ex19 running with hypre, diffs above > ========================================= > C/C++ example src/snes/tutorials/ex19 run successfully with cuda > Fortran example src/snes/tutorials/ex5f run successfully with 1 MPI process >

1 0

Hypre error on updated Perlmutter
by Mark Adams 09 Dec '21

09 Dec '21

I am getting this error. I have built this w/o hypre and the test are fine, including the CUDA tests. Any ideas? I notice that the tests use -dm_mat_type aijcusparse with hypre. Thanks, Mark 08:13 nid003929 adams/fix_mat_ex5k= perlmutter:~/petsc$ make PETSC_DIR=/global/homes/m/madams/petsc PETSC_ARCH=arch-perlmutter-opt-nvidia-cuda check Running check examples to verify correct installation Using PETSC_DIR=/global/homes/m/madams/petsc and PETSC_ARCH=arch-perlmutter-opt-nvidia-cuda C/C++ example src/snes/tutorials/ex19 run successfully with 1 MPI process C/C++ example src/snes/tutorials/ex19 run successfully with 2 MPI processes 1,5c1,70 < lid velocity = 0.0016, prandtl # = 1., grashof # = 1. < 0 SNES Function norm 0.0406612 < 1 SNES Function norm 4.12227e-06 < 2 SNES Function norm 6.098e-11 < Number of SNES iterations = 2 --- > [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > [0]PETSC ERROR: GPU error > [0]PETSC ERROR: cuda error 46 (cudaErrorDevicesUnavailable) : all CUDA-capable devices are busy or unavailable > [0]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting. > [0]PETSC ERROR: Petsc Development GIT revision: v3.16.1-442-gebb4a459f5 GIT Date: 2021-12-08 08:59:23 -0500 > [0]PETSC ERROR: /global/u2/m/madams/petsc/src/snes/tutorials/./ex19 on a named nid003929 by madams Thu Dec 9 08:13:49 2021 > [0]PETSC ERROR: Configure options --CFLAGS=" -g -DLANDAU_DIM=2 -DLANDAU_MAX_SPECIES=10 -DLANDAU_MAX_Q=4 -mp=gpu" --CXXFLAGS=" -g -DLANDAU_DIM=2 -DLANDAU_MAX_SPECIES=10 -DLANDAU_MAX_Q=4 -mp=gpu" --FFLAGS=" -g -mp=gpu" --with-cc=cc --with-cxx=CC --with-fc=ftn --with-cudac=/opt/nvidia/hpc_sdk/Linux_x86_64/21.9/compilers/bin/nvcc --with-debugging=0 --download-hypre=1 --with-cuda=1 --with-cuda-arch=80 --with-mpiexec=srun --with-batch=0 --download-p4est=1 --with-zlib=1 --with-make-np=8 --prefix=/global/cfs/projectdirs/m3904/petsc/current/perlmutter-opt-nvidia21.9 PETSC_ARCH=arch-perlmutter-opt-nvidia-cuda > [0]PETSC ERROR: #1 initialize() at /global/u2/m/madams/petsc/src/sys/objects/device/impls/cupm/cupmdevice.cxx:70 > [0]PETSC ERROR: #2 getDevice() at /global/u2/m/madams/petsc/src/sys/objects/device/impls/cupm/cupmdevice.cxx:360 > [0]PETSC ERROR: #3 PetscDeviceCreate() at /global/u2/m/madams/petsc/src/sys/objects/device/interface/device.cxx:102 > [0]PETSC ERROR: #4 PetscDeviceInitializeDefaultDevice_Internal() at /global/u2/m/madams/petsc/src/sys/objects/device/interface/device.cxx:266 > [1]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > [1]PETSC ERROR: GPU error > [1]PETSC ERROR: cuda error 46 (cudaErrorDevicesUnavailable) : all CUDA-capable devices are busy or unavailable > [1]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting. > [1]PETSC ERROR: Petsc Development GIT revision: v3.16.1-442-gebb4a459f5 GIT Date: 2021-12-08 08:59:23 -0500 > [1]PETSC ERROR: /global/u2/m/madams/petsc/src/snes/tutorials/./ex19 on a named nid003929 by madams Thu Dec 9 08:13:49 2021 > [1]PETSC ERROR: Configure options --CFLAGS=" -g -DLANDAU_DIM=2 -DLANDAU_MAX_SPECIES=10 -DLANDAU_MAX_Q=4 -mp=gpu" --CXXFLAGS=" -g -DLANDAU_DIM=2 -DLANDAU_MAX_SPECIES=10 -DLANDAU_MAX_Q=4 -mp=gpu" --FFLAGS=" -g -mp=gpu" --with-cc=cc --with-cxx=CC --with-fc=ftn --with-cudac=/opt/nvidia/hpc_sdk/Linux_x86_64/21.9/compilers/bin/nvcc --with-debugging=0 --download-hypre=1 --with-cuda=1 --with-cuda-arch=80 --with-mpiexec=srun --with-batch=0 --download-p4est=1 --with-zlib=1 --with-make-np=8 --prefix=/global/cfs/projectdirs/m3904/petsc/current/perlmutter-opt-nvidia21.9 PETSC_ARCH=arch-perlmutter-opt-nvidia-cuda > [1]PETSC ERROR: #1 initialize() at /global/u2/m/madams/petsc/src/sys/objects/device/impls/cupm/cupmdevice.cxx:70 > [1]PETSC ERROR: #2 getDevice() at /global/u2/m/madams/petsc/src/sys/objects/device/impls/cupm/cupmdevice.cxx:360 > [1]PETSC ERROR: #3 PetscDeviceCreate() at /global/u2/m/madams/petsc/src/sys/objects/device/interface/device.cxx:102 > [1]PETSC ERROR: #4 PetscDeviceInitializeDefaultDevice_Internal() at /global/u2/m/madams/petsc/src/sys/objects/device/interface/device.cxx:266 > [1]PETSC ERROR: #5 PetscDeviceInitialize() at /global/u2/m/madams/petsc/src/sys/objects/device/interface/device.cxx:227 > [1]PETSC ERROR: #6 PCCreate_HYPRE() at /global/u2/m/madams/petsc/src/ksp/pc/impls/hypre/hypre.c:2224 > [1]PETSC ERROR: #7 PCSetType() at /global/u2/m/madams/petsc/src/ksp/pc/interface/pcset.c:84 > [1]PETSC ERROR: #8 PCSetFromOptions() at /global/u2/m/madams/petsc/src/ksp/pc/interface/pcset.c:154 > [1]PETSC ERROR: #9 KSPSetFromOptions() at /global/u2/m/madams/petsc/src/ksp/ksp/interface/itcl.c:356 > [1]PETSC ERROR: #10 SNESSetFromOptions() at /global/u2/m/madams/petsc/src/snes/interface/snes.c:1113 > [1]PETSC ERROR: #11 main() at ex19.c:150 > [1]PETSC ERROR: PETSc Option Table entries: > [1]PETSC ERROR: -da_refine 3 > [1]PETSC ERROR: -dm_mat_type aijcusparse > [1]PETSC ERROR: -dm_vec_type cuda > [0]PETSC ERROR: #5 PetscDeviceInitialize() at /global/u2/m/madams/petsc/src/sys/objects/device/interface/device.cxx:227 > [0]PETSC ERROR: #6 PCCreate_HYPRE() at /global/u2/m/madams/petsc/src/ksp/pc/impls/hypre/hypre.c:2224 > [0]PETSC ERROR: #7 PCSetType() at /global/u2/m/madams/petsc/src/ksp/pc/interface/pcset.c:84 > [0]PETSC ERROR: #8 PCSetFromOptions() at /global/u2/m/madams/petsc/src/ksp/pc/interface/pcset.c:154 > [0]PETSC ERROR: #9 KSPSetFromOptions() at /global/u2/m/madams/petsc/src/ksp/ksp/interface/itcl.c:356 > [0]PETSC ERROR: #10 SNESSetFromOptions() at /global/u2/m/madams/petsc/src/snes/interface/snes.c:1113 > [0]PETSC ERROR: #11 main() at ex19.c:150 > [0]PETSC ERROR: PETSc Option Table entries: > [0]PETSC ERROR: -da_refine 3 > [0]PETSC ERROR: -dm_mat_type aijcusparse > [0]PETSC ERROR: -dm_vec_type cuda > [0]PETSC ERROR: -ksp_norm_type unpreconditioned > [0]PETSC ERROR: -nox > [0]PETSC ERROR: -nox_warning > [0]PETSC ERROR: -pc_type hypre > [0]PETSC ERROR: -snes_monitor_short > [0]PETSC ERROR: -use_gpu_aware_mpi 0 > [0]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint(a)mcs.anl.gov---------- > [1]PETSC ERROR: -ksp_norm_type unpreconditioned > [1]PETSC ERROR: -nox > [1]PETSC ERROR: -nox_warning > [1]PETSC ERROR: -pc_type hypre > [1]PETSC ERROR: -snes_monitor_short > [1]PETSC ERROR: -use_gpu_aware_mpi 0 > [1]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint(a)mcs.anl.gov---------- > MPICH Notice [Rank 0] [job id 832277.2] [Thu Dec 9 08:13:50 2021] [nid003929] - Abort(97) (rank 0 in comm 0): application called MPI_Abort(MPI_COMM_WORLD, 97) - process 0 > > aborting job: > application called MPI_Abort(MPI_COMM_WORLD, 97) - process 0 > MPICH Notice [Rank 1] [job id 832277.2] [Thu Dec 9 08:13:50 2021] [nid003929] - Abort(97) (rank 1 in comm 0): application called MPI_Abort(MPI_COMM_WORLD, 97) - process 1 > > aborting job: > application called MPI_Abort(MPI_COMM_WORLD, 97) - process 1 > srun: error: nid003929: task 1: Exited with exit code 255 > srun: launch/slurm: _step_signal: Terminating StepId=832277.2 > slurmstepd: error: *** STEP 832277.2 ON nid003929 CANCELLED AT 2021-12-09T16:13:50 *** > srun: error: nid003929: task 0: Exited with exit code 255 /global/homes/m/madams/petsc/src/snes/tutorials Possible problem with ex19 running with hypre, diffs above ========================================= C/C++ example src/snes/tutorials/ex19 run successfully with cuda Fortran example src/snes/tutorials/ex5f run successfully with 1 MPI process

1 0