I can build a non-kokkos PETSc here (Cori/GPU) but I get this error with Kokkos. Any suggestions? Thanks, Mark ============================================================================================= ============================================================================================= Trying to download git://https://bitbucket.org/petsc/pkg-sowing.git for SOWING ============================================================================================= ============================================================================================= Trying to download git://https://bitbucket.org/petsc/pkg-sowing.git for SOWING ============================================================================================= ============================================================================================= Trying to download https://bitbucket.org/petsc/pkg-sowing/get/v1.1.26-p1.tar.gz for SOWING ============================================================================================= ******************************************************************************* UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log for details): ------------------------------------------------------------------------------- Error during download/extract/detection of SOWING: Unable to clone sowing Could not execute "['git clone https://bitbucket.org/petsc/pkg-sowing.git /global/u2/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/externalpackages/git.sowing']": Cloning into '/global/u2/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/externalpackages/git.sowing'... fatal: unable to access 'https://bitbucket.org/petsc/pkg-sowing.git/': The requested URL returned error: 503 Unable to download package SOWING from: git:// https://bitbucket.org/petsc/pkg-sowing.git * If URL specified manually - perhaps there is a typo? * If your network is disconnected - please reconnect and rerun ./configure * Or perhaps you have a firewall blocking the download * You can run with --with-packages-download-dir=/adirectory and ./configure will instruct you what packages to download manually * or you can download the above URL manually, to /yourselectedlocation and use the configure option: --download-sowing=/yourselectedlocation Unable to clone sowing Could not execute "['git clone https://bitbucket.org/petsc/pkg-sowing.git /global/u2/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/externalpackages/git.sowing']": Cloning into '/global/u2/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/externalpackages/git.sowing'... fatal: unable to access 'https://bitbucket.org/petsc/pkg-sowing.git/': The requested URL returned error: 503 Unable to download package SOWING from: git:// https://bitbucket.org/petsc/pkg-sowing.git * If URL specified manually - perhaps there is a typo? * If your network is disconnected - please reconnect and rerun ./configure * Or perhaps you have a firewall blocking the download * You can run with --with-packages-download-dir=/adirectory and ./configure will instruct you what packages to download manually * or you can download the above URL manually, to /yourselectedlocation and use the configure option: --download-sowing=/yourselectedlocation file could not be opened successfully Downloaded package SOWING from: https://bitbucket.org/petsc/pkg-sowing/get/v1.1.26-p1.tar.gz is not a tarball. [or installed python cannot process compressed files] * If you are behind a firewall - please fix your proxy and rerun ./configure For example at LANL you may need to set the environmental variable http_proxy (or HTTP_PROXY?) to http://proxyout.lanl.gov * You can run with --with-packages-download-dir=/adirectory and ./configure will instruct you what packages to download manually * or you can download the above URL manually, to /yourselectedlocation/v1.1.26-p1.tar.gz and use the configure option: --download-sowing=/yourselectedlocation/v1.1.26-p1.tar.gz *******************************************************************************
Bitbucket is off-line. Do you need fortran stubs at this moment? If not use --with-fortran-interface=0
On May 15, 2021, at 2:51 PM, Mark Adams <[email protected]> wrote:
I can build a non-kokkos PETSc here (Cori/GPU) but I get this error with Kokkos. Any suggestions? Thanks, Mark
============================================================================================= ============================================================================================= Trying to download git://https://bitbucket.org/petsc/pkg-sowing.git <https://bitbucket.org/petsc/pkg-sowing.git> for SOWING ============================================================================================= ============================================================================================= Trying to download git://https://bitbucket.org/petsc/pkg-sowing.git <https://bitbucket.org/petsc/pkg-sowing.git> for SOWING ============================================================================================= ============================================================================================= Trying to download https://bitbucket.org/petsc/pkg-sowing/get/v1.1.26-p1.tar.gz <https://bitbucket.org/petsc/pkg-sowing/get/v1.1.26-p1.tar.gz> for SOWING ============================================================================================= ******************************************************************************* UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log for details): ------------------------------------------------------------------------------- Error during download/extract/detection of SOWING: Unable to clone sowing Could not execute "['git clone https://bitbucket.org/petsc/pkg-sowing.git <https://bitbucket.org/petsc/pkg-sowing.git> /global/u2/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/externalpackages/git.sowing']": Cloning into '/global/u2/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/externalpackages/git.sowing'... fatal: unable to access 'https://bitbucket.org/petsc/pkg-sowing.git/ <https://bitbucket.org/petsc/pkg-sowing.git/>': The requested URL returned error: 503 Unable to download package SOWING from: git://https://bitbucket.org/petsc/pkg-sowing.git <https://bitbucket.org/petsc/pkg-sowing.git> * If URL specified manually - perhaps there is a typo? * If your network is disconnected - please reconnect and rerun ./configure * Or perhaps you have a firewall blocking the download * You can run with --with-packages-download-dir=/adirectory and ./configure will instruct you what packages to download manually * or you can download the above URL manually, to /yourselectedlocation and use the configure option: --download-sowing=/yourselectedlocation Unable to clone sowing Could not execute "['git clone https://bitbucket.org/petsc/pkg-sowing.git <https://bitbucket.org/petsc/pkg-sowing.git> /global/u2/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/externalpackages/git.sowing']": Cloning into '/global/u2/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/externalpackages/git.sowing'... fatal: unable to access 'https://bitbucket.org/petsc/pkg-sowing.git/ <https://bitbucket.org/petsc/pkg-sowing.git/>': The requested URL returned error: 503 Unable to download package SOWING from: git://https://bitbucket.org/petsc/pkg-sowing.git <https://bitbucket.org/petsc/pkg-sowing.git> * If URL specified manually - perhaps there is a typo? * If your network is disconnected - please reconnect and rerun ./configure * Or perhaps you have a firewall blocking the download * You can run with --with-packages-download-dir=/adirectory and ./configure will instruct you what packages to download manually * or you can download the above URL manually, to /yourselectedlocation and use the configure option: --download-sowing=/yourselectedlocation file could not be opened successfully Downloaded package SOWING from: https://bitbucket.org/petsc/pkg-sowing/get/v1.1.26-p1.tar.gz <https://bitbucket.org/petsc/pkg-sowing/get/v1.1.26-p1.tar.gz> is not a tarball. [or installed python cannot process compressed files] * If you are behind a firewall - please fix your proxy and rerun ./configure For example at LANL you may need to set the environmental variable http_proxy (or HTTP_PROXY?) to http://proxyout.lanl.gov <http://proxyout.lanl.gov/> * You can run with --with-packages-download-dir=/adirectory and ./configure will instruct you what packages to download manually * or you can download the above URL manually, to /yourselectedlocation/v1.1.26-p1.tar.gz and use the configure option: --download-sowing=/yourselectedlocation/v1.1.26-p1.tar.gz ******************************************************************************* <configure.log>
Thanks, Now I get an error in make all. THis was working a few weeks ago. Make.log was empty but here is the output. .... CC arch-cori-gpu-opt-kokkos-gcc/obj/sys/error/ftn-custom/zerrf.o CC arch-cori-gpu-opt-kokkos-gcc/obj/sys/f90-src/f90_cwrap.o CC arch-cori-gpu-opt-kokkos-gcc/obj/vec/is/utils/hdf5io.o PATH=/global/common/sw/cray/sles15/x86_64/zlib/1.2.11/gcc/8.2.0/pep2pal/bin:/global/common/sw/cray/cnl7/haswell/cmake/3.18.2/bin:/opt/cray/rca/2.2.20-7.0.1.1_4.62__g8e3fb5b.ari/bin:/opt/cray/alps/6.6.58-7.0.1.1_6.20__g437d88db.ari/sbin:/opt/cray/alps/default/bin:/opt/cray/job/2.2.4-7.0.1.1_3.48__g36b56f4.ari/bin:/opt/cray/pe/craype/2.6.2/bin:/usr/common/software/sles15_cgpu/openmpi/4.0.3/gcc/bin:/usr/common/software/sles15_cgpu/hpcsdk/20.11/Linux_x86_64/20.11/compilers/bin:/usr/common/software/sles15_cgpu/hpcsdk/20.11/Linux_x86_64/20.11/cuda/bin:/usr/common/software/sles15_cgpu/cuda/11.1.1/bin:/usr/common/software/sles15_cgpu/gcc/8.3.0/bin:/opt/esslurm/bin:/opt/cray/pe/modules/ 3.2.11.4/bin:/usr/common/software/bin:/usr/common/nsg/bin:/opt/ovis/bin:/opt/ovis/sbin:/usr/local/bin:/usr/bin:/bin:/usr/lib/mit/bin:/usr/lib/mit/sbin:/opt/cray/pe/bin:`dirname /usr/common/software/sles15_cgpu/cuda/11.1.1/bin/nvcc` NVCC_WRAPPER_DEFAULT_COMPILER=/usr/common/software/sles15_cgpu/openmpi/4.0.3/gcc/bin/mpicxx /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/bin/nvcc_wrapper --expt-extended-lambda -c -arch=sm_70 -Xcompiler -rdynamic -DLANDAU_DIM=2 -DLANDAU_MAX_SPECIES=10 -DPETSC_HAVE_CUDA_ATOMIC -DLANDAU_MAX_Q=4 -Xcompiler -fPIC -O3 -gencode arch=compute_80,code=sm_80 -I/usr/common/software/sles15_cgpu/openmpi/4.0.3/gcc/include -Wno-deprecated-gpu-targets -I/global/homes/m/madams/petsc/include -I/global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/include -I/usr/common/software/sles15_cgpu/openmpi/4.0.3/gcc/include -I/usr/common/software/sles15_cgpu/cuda/11.1.1/include /global/u2/m/madams/petsc/src/sys/objects/kokkos/kinit.kokkos.cxx -o arch-cori-gpu-opt-kokkos-gcc/obj/sys/objects/kokkos/kinit.o g++: error: arch=compute_80: No such file or directory g++: error: code=sm_80: No such file or directory gmake[3]: *** [gmakefile:188: arch-cori-gpu-opt-kokkos-gcc/obj/sys/objects/kokkos/kinit.o] Error 1 gmake[3]: *** Waiting for unfinished jobs.... CC arch-cori-gpu-opt-kokkos-gcc/obj/vec/is/utils/isio.o CC arch-cori-gpu-opt-kokkos-gcc/obj/vec/is/utils/ftn-auto/pmapf.o CC arch-cori-gpu-opt-kokkos-gcc/obj/vec/is/utils/pmap.o CC arch-cori-gpu-opt-kokkos-gcc/obj/vec/is/utils/psort.o CC arch-cori-gpu-opt-kokkos-gcc/obj/vec/is/utils/isltog.o FC arch-cori-gpu-opt-kokkos-gcc/obj/vec/f90-mod/petscvecmod.o CUDAC arch-cori-gpu-opt-kokkos-gcc/obj/sys/memory/cuda/mcudahost.o CUDAC.dep arch-cori-gpu-opt-kokkos-gcc/obj/sys/memory/cuda/mcudahost.o gmake[2]: *** [/global/homes/m/madams/petsc/lib/petsc/conf/rules:50: libs] Error 2 **************************ERROR************************************* Error during compile, check arch-cori-gpu-opt-kokkos-gcc/lib/petsc/conf/make.log Send it and arch-cori-gpu-opt-kokkos-gcc/lib/petsc/conf/configure.log to [email protected] ******************************************************************** gmake[1]: *** [makefile:40: all] Error 1 make: *** [GNUmakefile:9: all] Error 2 On Sat, May 15, 2021 at 5:30 PM Barry Smith <[email protected]> wrote:
Bitbucket is off-line. Do you need fortran stubs at this moment? If not use --with-fortran-interface=0
On May 15, 2021, at 2:51 PM, Mark Adams <[email protected]> wrote:
I can build a non-kokkos PETSc here (Cori/GPU) but I get this error with Kokkos. Any suggestions? Thanks, Mark
=============================================================================================
=============================================================================================
Trying to download git://https://bitbucket.org/petsc/pkg-sowing.git for SOWING
=============================================================================================
=============================================================================================
Trying to download git://https://bitbucket.org/petsc/pkg-sowing.git for SOWING
=============================================================================================
=============================================================================================
Trying to download https://bitbucket.org/petsc/pkg-sowing/get/v1.1.26-p1.tar.gz for SOWING
=============================================================================================
******************************************************************************* UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log for details):
------------------------------------------------------------------------------- Error during download/extract/detection of SOWING: Unable to clone sowing Could not execute "['git clone https://bitbucket.org/petsc/pkg-sowing.git /global/u2/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/externalpackages/git.sowing']": Cloning into '/global/u2/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/externalpackages/git.sowing'... fatal: unable to access 'https://bitbucket.org/petsc/pkg-sowing.git/': The requested URL returned error: 503 Unable to download package SOWING from: git:// https://bitbucket.org/petsc/pkg-sowing.git * If URL specified manually - perhaps there is a typo? * If your network is disconnected - please reconnect and rerun ./configure * Or perhaps you have a firewall blocking the download * You can run with --with-packages-download-dir=/adirectory and ./configure will instruct you what packages to download manually * or you can download the above URL manually, to /yourselectedlocation and use the configure option: --download-sowing=/yourselectedlocation Unable to clone sowing Could not execute "['git clone https://bitbucket.org/petsc/pkg-sowing.git /global/u2/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/externalpackages/git.sowing']": Cloning into '/global/u2/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/externalpackages/git.sowing'... fatal: unable to access 'https://bitbucket.org/petsc/pkg-sowing.git/': The requested URL returned error: 503 Unable to download package SOWING from: git:// https://bitbucket.org/petsc/pkg-sowing.git * If URL specified manually - perhaps there is a typo? * If your network is disconnected - please reconnect and rerun ./configure * Or perhaps you have a firewall blocking the download * You can run with --with-packages-download-dir=/adirectory and ./configure will instruct you what packages to download manually * or you can download the above URL manually, to /yourselectedlocation and use the configure option: --download-sowing=/yourselectedlocation file could not be opened successfully Downloaded package SOWING from: https://bitbucket.org/petsc/pkg-sowing/get/v1.1.26-p1.tar.gz is not a tarball. [or installed python cannot process compressed files] * If you are behind a firewall - please fix your proxy and rerun ./configure For example at LANL you may need to set the environmental variable http_proxy (or HTTP_PROXY?) to http://proxyout.lanl.gov * You can run with --with-packages-download-dir=/adirectory and ./configure will instruct you what packages to download manually * or you can download the above URL manually, to /yourselectedlocation/v1.1.26-p1.tar.gz and use the configure option: --download-sowing=/yourselectedlocation/v1.1.26-p1.tar.gz
******************************************************************************* <configure.log>
Add --show to the failing nvcc_wrapper command and see how nvcc_wrapper parses the options PATH=/global/common/sw/cray/sles15/x86_64/zlib/1.2.11/gcc/8.2.0/pep2pal/bin:/global/common/sw/cray/cnl7/haswell/cmake/3.18.2/bin:/opt/cray/rca/2.2.20-7.0.1.1_4.62__g8e3fb5b.ari/bin:/opt/cray/alps/6.6.58-7.0.1.1_6.20__g437d88db.ari/sbin:/opt/cray/alps/default/bin:/opt/cray/job/2.2.4-7.0.1.1_3.48__g36b56f4.ari/bin:/opt/cray/pe/craype/2.6.2/bin:/usr/common/software/sles15_cgpu/openmpi/4.0.3/gcc/bin:/usr/common/software/sles15_cgpu/hpcsdk/20.11/Linux_x86_64/20.11/compilers/bin:/usr/common/software/sles15_cgpu/hpcsdk/20.11/Linux_x86_64/20.11/cuda/bin:/usr/common/software/sles15_cgpu/cuda/11.1.1/bin:/usr/common/software/sles15_cgpu/gcc/8.3.0/bin:/opt/esslurm/bin:/opt/cray/pe/modules/ 3.2.11.4/bin:/usr/common/software/bin:/usr/common/nsg/bin:/opt/ovis/bin:/opt/ovis/sbin:/usr/local/bin:/usr/bin:/bin:/usr/lib/mit/bin:/usr/lib/mit/sbin:/opt/cray/pe/bin:`dirname <http://3.2.11.4/bin:/usr/common/software/bin:/usr/common/nsg/bin:/opt/ovis/bin:/opt/ovis/sbin:/usr/local/bin:/usr/bin:/bin:/usr/lib/mit/bin:/usr/lib/mit/sbin:/opt/cray/pe/bin:dirname> /usr/common/software/sles15_cgpu/cuda/11.1.1/bin/nvcc` NVCC_WRAPPER_DEFAULT_COMPILER=/usr/common/software/sles15_cgpu/openmpi/4.0.3/gcc/bin/mpicxx /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/bin/nvcc_wrapper --expt-extended-lambda -c -arch=sm_70 -Xcompiler -rdynamic -DLANDAU_DIM=2 -DLANDAU_MAX_SPECIES=10 -DPETSC_HAVE_CUDA_ATOMIC -DLANDAU_MAX_Q=4 -Xcompiler -fPIC -O3 -gencode arch=compute_80,code=sm_80 -I/usr/common/software/sles15_cgpu/openmpi/4.0.3/gcc/include -Wno-deprecated-gpu-targets -I/global/homes/m/madams/petsc/include -I/global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/include -I/usr/common/software/sles15_cgpu/openmpi/4.0.3/gcc/include -I/usr/common/software/sles15_cgpu/cuda/11.1.1/include /global/u2/m/madams/petsc/src/sys/objects/kokkos/kinit.kokkos.cxx -o arch-cori-gpu-opt-kokkos-gcc/obj/sys/objects/kokkos/kinit.o --show --Junchao Zhang On Sat, May 15, 2021 at 7:14 PM Mark Adams <[email protected]> wrote:
Thanks, Now I get an error in make all. THis was working a few weeks ago. Make.log was empty but here is the output. .... CC arch-cori-gpu-opt-kokkos-gcc/obj/sys/error/ftn-custom/zerrf.o CC arch-cori-gpu-opt-kokkos-gcc/obj/sys/f90-src/f90_cwrap.o CC arch-cori-gpu-opt-kokkos-gcc/obj/vec/is/utils/hdf5io.o
PATH=/global/common/sw/cray/sles15/x86_64/zlib/1.2.11/gcc/8.2.0/pep2pal/bin:/global/common/sw/cray/cnl7/haswell/cmake/3.18.2/bin:/opt/cray/rca/2.2.20-7.0.1.1_4.62__g8e3fb5b.ari/bin:/opt/cray/alps/6.6.58-7.0.1.1_6.20__g437d88db.ari/sbin:/opt/cray/alps/default/bin:/opt/cray/job/2.2.4-7.0.1.1_3.48__g36b56f4.ari/bin:/opt/cray/pe/craype/2.6.2/bin:/usr/common/software/sles15_cgpu/openmpi/4.0.3/gcc/bin:/usr/common/software/sles15_cgpu/hpcsdk/20.11/Linux_x86_64/20.11/compilers/bin:/usr/common/software/sles15_cgpu/hpcsdk/20.11/Linux_x86_64/20.11/cuda/bin:/usr/common/software/sles15_cgpu/cuda/11.1.1/bin:/usr/common/software/sles15_cgpu/gcc/8.3.0/bin:/opt/esslurm/bin:/opt/cray/pe/modules/ 3.2.11.4/bin:/usr/common/software/bin:/usr/common/nsg/bin:/opt/ovis/bin:/opt/ovis/sbin:/usr/local/bin:/usr/bin:/bin:/usr/lib/mit/bin:/usr/lib/mit/sbin:/opt/cray/pe/bin:`dirname <http://3.2.11.4/bin:/usr/common/software/bin:/usr/common/nsg/bin:/opt/ovis/bin:/opt/ovis/sbin:/usr/local/bin:/usr/bin:/bin:/usr/lib/mit/bin:/usr/lib/mit/sbin:/opt/cray/pe/bin:dirname> /usr/common/software/sles15_cgpu/cuda/11.1.1/bin/nvcc` NVCC_WRAPPER_DEFAULT_COMPILER=/usr/common/software/sles15_cgpu/openmpi/4.0.3/gcc/bin/mpicxx /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/bin/nvcc_wrapper --expt-extended-lambda -c -arch=sm_70 -Xcompiler -rdynamic -DLANDAU_DIM=2 -DLANDAU_MAX_SPECIES=10 -DPETSC_HAVE_CUDA_ATOMIC -DLANDAU_MAX_Q=4 -Xcompiler -fPIC -O3 -gencode arch=compute_80,code=sm_80 -I/usr/common/software/sles15_cgpu/openmpi/4.0.3/gcc/include -Wno-deprecated-gpu-targets -I/global/homes/m/madams/petsc/include -I/global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/include -I/usr/common/software/sles15_cgpu/openmpi/4.0.3/gcc/include -I/usr/common/software/sles15_cgpu/cuda/11.1.1/include /global/u2/m/madams/petsc/src/sys/objects/kokkos/kinit.kokkos.cxx -o arch-cori-gpu-opt-kokkos-gcc/obj/sys/objects/kokkos/kinit.o g++: error: arch=compute_80: No such file or directory g++: error: code=sm_80: No such file or directory gmake[3]: *** [gmakefile:188: arch-cori-gpu-opt-kokkos-gcc/obj/sys/objects/kokkos/kinit.o] Error 1 gmake[3]: *** Waiting for unfinished jobs.... CC arch-cori-gpu-opt-kokkos-gcc/obj/vec/is/utils/isio.o CC arch-cori-gpu-opt-kokkos-gcc/obj/vec/is/utils/ftn-auto/pmapf.o CC arch-cori-gpu-opt-kokkos-gcc/obj/vec/is/utils/pmap.o CC arch-cori-gpu-opt-kokkos-gcc/obj/vec/is/utils/psort.o CC arch-cori-gpu-opt-kokkos-gcc/obj/vec/is/utils/isltog.o FC arch-cori-gpu-opt-kokkos-gcc/obj/vec/f90-mod/petscvecmod.o CUDAC arch-cori-gpu-opt-kokkos-gcc/obj/sys/memory/cuda/mcudahost.o CUDAC.dep arch-cori-gpu-opt-kokkos-gcc/obj/sys/memory/cuda/mcudahost.o gmake[2]: *** [/global/homes/m/madams/petsc/lib/petsc/conf/rules:50: libs] Error 2 **************************ERROR************************************* Error during compile, check arch-cori-gpu-opt-kokkos-gcc/lib/petsc/conf/make.log Send it and arch-cori-gpu-opt-kokkos-gcc/lib/petsc/conf/configure.log to [email protected] ******************************************************************** gmake[1]: *** [makefile:40: all] Error 1 make: *** [GNUmakefile:9: all] Error 2
On Sat, May 15, 2021 at 5:30 PM Barry Smith <[email protected]> wrote:
Bitbucket is off-line. Do you need fortran stubs at this moment? If not use --with-fortran-interface=0
On May 15, 2021, at 2:51 PM, Mark Adams <[email protected]> wrote:
I can build a non-kokkos PETSc here (Cori/GPU) but I get this error with Kokkos. Any suggestions? Thanks, Mark
=============================================================================================
=============================================================================================
Trying to download git://https://bitbucket.org/petsc/pkg-sowing.git for SOWING
=============================================================================================
=============================================================================================
Trying to download git://https://bitbucket.org/petsc/pkg-sowing.git for SOWING
=============================================================================================
=============================================================================================
Trying to download https://bitbucket.org/petsc/pkg-sowing/get/v1.1.26-p1.tar.gz for SOWING
=============================================================================================
******************************************************************************* UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log for details):
------------------------------------------------------------------------------- Error during download/extract/detection of SOWING: Unable to clone sowing Could not execute "['git clone https://bitbucket.org/petsc/pkg-sowing.git /global/u2/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/externalpackages/git.sowing']": Cloning into '/global/u2/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/externalpackages/git.sowing'... fatal: unable to access 'https://bitbucket.org/petsc/pkg-sowing.git/': The requested URL returned error: 503 Unable to download package SOWING from: git:// https://bitbucket.org/petsc/pkg-sowing.git * If URL specified manually - perhaps there is a typo? * If your network is disconnected - please reconnect and rerun ./configure * Or perhaps you have a firewall blocking the download * You can run with --with-packages-download-dir=/adirectory and ./configure will instruct you what packages to download manually * or you can download the above URL manually, to /yourselectedlocation and use the configure option: --download-sowing=/yourselectedlocation Unable to clone sowing Could not execute "['git clone https://bitbucket.org/petsc/pkg-sowing.git /global/u2/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/externalpackages/git.sowing']": Cloning into '/global/u2/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/externalpackages/git.sowing'... fatal: unable to access 'https://bitbucket.org/petsc/pkg-sowing.git/': The requested URL returned error: 503 Unable to download package SOWING from: git:// https://bitbucket.org/petsc/pkg-sowing.git * If URL specified manually - perhaps there is a typo? * If your network is disconnected - please reconnect and rerun ./configure * Or perhaps you have a firewall blocking the download * You can run with --with-packages-download-dir=/adirectory and ./configure will instruct you what packages to download manually * or you can download the above URL manually, to /yourselectedlocation and use the configure option: --download-sowing=/yourselectedlocation file could not be opened successfully Downloaded package SOWING from: https://bitbucket.org/petsc/pkg-sowing/get/v1.1.26-p1.tar.gz is not a tarball. [or installed python cannot process compressed files] * If you are behind a firewall - please fix your proxy and rerun ./configure For example at LANL you may need to set the environmental variable http_proxy (or HTTP_PROXY?) to http://proxyout.lanl.gov * You can run with --with-packages-download-dir=/adirectory and ./configure will instruct you what packages to download manually * or you can download the above URL manually, to /yourselectedlocation/v1.1.26-p1.tar.gz and use the configure option: --download-sowing=/yourselectedlocation/v1.1.26-p1.tar.gz
******************************************************************************* <configure.log>
Looks like nvcc_wrapper gets confused when both -arch and -gencode are used. But handles either of them separately correctly. Mark, PETSc configure now handles the -arch and --with-kokkos-cuda-arch= business automatically so you do not, and generally shouldn't pass -arch sm_70 in --CUDAFLAGS anymore or use the --with-kokkos-cuda-arch option anymore. If you need to set the -arch to something earlier than the system supports (rare) you can now use -with-cuda-gencodearch to set that instead of using CUDAFLAGS. Barry $ NVCC_WRAPPER_DEFAULT_COMPILER=mpicxx /home/bsmith/petsc/arch-main/bin/nvcc_wrapper --show --expt-extended-lambda -c -Xcompiler -fPIC -g -gencode arch=compute_70,code=sm_70 -I/home/bsmith/soft/gnu-mpich/include -Wno-deprecated-gpu-targets -I/home/bsmith/petsc/include -I/home/bsmith/petsc/arch-main/include -I/nfs/apps/spacks/2021-02-09/opt/spack/linux-centos7-x86_64/gcc-7.3.0/cuda-11.2.0-ikruu5mo6dtt3avvmwsejouqhxu4btdm/include /home/bsmith/petsc/src/ts/utils/dmplexlandau/kokkos/landau.kokkos.cxx -o arch-main/obj/ts/utils/dmplexlandau/kokkos/landau.o nvcc --expt-extended-lambda -gencode arch=compute_70,code=sm_70 -Wno-deprecated-gpu-targets -ccbin mpicxx -g -I/home/bsmith/soft/gnu-mpich/include -I/home/bsmith/petsc/include -I/home/bsmith/petsc/arch-main/include -I/nfs/apps/spacks/2021-02-09/opt/spack/linux-centos7-x86_64/gcc-7.3.0/cuda-11.2.0-ikruu5mo6dtt3avvmwsejouqhxu4btdm/include -Xcompiler -fPIC -x cu /home/bsmith/petsc/src/ts/utils/dmplexlandau/kokkos/landau.kokkos.cxx -c -o arch-main/obj/ts/utils/dmplexlandau/kokkos/landau.o ~/petsc (main=) arch-main $ NVCC_WRAPPER_DEFAULT_COMPILER=mpicxx /home/bsmith/petsc/arch-main/bin/nvcc_wrapper --show --expt-extended-lambda -c -arch=sm_70 -Xcompiler -rdynamic -DLANDAU_DIM=2 -DLANDAU_MAX_SPECIES=10 -DPETSC_HAVE_CUDA_ATOMIC -DLANDAU_MAX_Q=4 -Xcompiler -fPIC -O3 -gencode arch=compute_80,code=sm_80 -I/usr/common/software/sles15_cgpu/openmpi/4.0.3/gcc/include -Wno-deprecated-gpu-targets nvcc --expt-extended-lambda -arch=sm_70 -Wno-deprecated-gpu-targets -ccbin mpicxx -DLANDAU_DIM=2 -DLANDAU_MAX_SPECIES=10 -DPETSC_HAVE_CUDA_ATOMIC -DLANDAU_MAX_Q=4 -O3 -I/usr/common/software/sles15_cgpu/openmpi/4.0.3/gcc/include -Xcompiler -rdynamic,-fPIC,arch=compute_80,code=sm_80 -c ~/petsc (main=) arch-main $ NVCC_WRAPPER_DEFAULT_COMPILER=mpicxx /home/bsmith/petsc/arch-main/bin/nvcc_wrapper --show --expt-extended-lambda -c -Xcompiler -fPIC -g -arch=sm_70 -gencode arch=compute_70,code=sm_70 -I/home/bsmith/soft/gnu-mpich/include -Wno-deprecated-gpu-targets -I/home/bsmith/petsc/include -I/home/bsmith/petsc/arch-main/include -I/nfs/apps/spacks/2021-02-09/opt/spack/linux-centos7-x86_64/gcc-7.3.0/cuda-11.2.0-ikruu5mo6dtt3avvmwsejouqhxu4btdm/include /home/bsmith/petsc/src/ts/utils/dmplexlandau/kokkos/landau.kokkos.cxx -o arch-main/obj/ts/utils/dmplexlandau/kokkos/landau.o nvcc --expt-extended-lambda -arch=sm_70 -Wno-deprecated-gpu-targets -ccbin mpicxx -g -I/home/bsmith/soft/gnu-mpich/include -I/home/bsmith/petsc/include -I/home/bsmith/petsc/arch-main/include -I/nfs/apps/spacks/2021-02-09/opt/spack/linux-centos7-x86_64/gcc-7.3.0/cuda-11.2.0-ikruu5mo6dtt3avvmwsejouqhxu4btdm/include -Xcompiler -fPIC,arch=compute_70,code=sm_70 -x cu /home/bsmith/petsc/src/ts/utils/dmplexlandau/kokkos/landau.kokkos.cxx -c -o arch-main/obj/ts/utils/dmplexlandau/kokkos/landau.o
On May 15, 2021, at 8:44 PM, Junchao Zhang <[email protected]> wrote:
Add --show to the failing nvcc_wrapper command and see how nvcc_wrapper parses the options
PATH=/global/common/sw/cray/sles15/x86_64/zlib/1.2.11/gcc/8.2.0/pep2pal/bin:/global/common/sw/cray/cnl7/haswell/cmake/3.18.2/bin:/opt/cray/rca/2.2.20-7.0.1.1_4.62__g8e3fb5b.ari/bin:/opt/cray/alps/6.6.58-7.0.1.1_6.20__g437d88db.ari/sbin:/opt/cray/alps/default/bin:/opt/cray/job/2.2.4-7.0.1.1_3.48__g36b56f4.ari/bin:/opt/cray/pe/craype/2.6.2/bin:/usr/common/software/sles15_cgpu/openmpi/4.0.3/gcc/bin:/usr/common/software/sles15_cgpu/hpcsdk/20.11/Linux_x86_64/20.11/compilers/bin:/usr/common/software/sles15_cgpu/hpcsdk/20.11/Linux_x86_64/20.11/cuda/bin:/usr/common/software/sles15_cgpu/cuda/11.1.1/bin:/usr/common/software/sles15_cgpu/gcc/8.3.0/bin:/opt/esslurm/bin:/opt/cray/pe/modules/3.2.11.4/bin:/usr/common/software/bin:/usr/common/nsg/bin:/opt/ovis/bin:/opt/ovis/sbin:/usr/local/bin:/usr/bin:/bin:/usr/lib/mit/bin:/usr/lib/mit/sbin:/opt/cray/pe/bin:`dirname <http://3.2.11.4/bin:/usr/common/software/bin:/usr/common/nsg/bin:/opt/ovis/bin:/opt/ovis/sbin:/usr/local/bin:/usr/bin:/bin:/usr/lib/mit/bin:/usr/lib/mit/sbin:/opt/cray/pe/bin:dirname> /usr/common/software/sles15_cgpu/cuda/11.1.1/bin/nvcc` NVCC_WRAPPER_DEFAULT_COMPILER=/usr/common/software/sles15_cgpu/openmpi/4.0.3/gcc/bin/mpicxx /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/bin/nvcc_wrapper --expt-extended-lambda -c -arch=sm_70 -Xcompiler -rdynamic -DLANDAU_DIM=2 -DLANDAU_MAX_SPECIES=10 -DPETSC_HAVE_CUDA_ATOMIC -DLANDAU_MAX_Q=4 -Xcompiler -fPIC -O3 -gencode arch=compute_80,code=sm_80 -I/usr/common/software/sles15_cgpu/openmpi/4.0.3/gcc/include -Wno-deprecated-gpu-targets -I/global/homes/m/madams/petsc/include -I/global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/include -I/usr/common/software/sles15_cgpu/openmpi/4.0.3/gcc/include -I/usr/common/software/sles15_cgpu/cuda/11.1.1/include /global/u2/m/madams/petsc/src/sys/objects/kokkos/kinit.kokkos.cxx -o arch-cori-gpu-opt-kokkos-gcc/obj/sys/objects/kokkos/kinit.o --show
--Junchao Zhang
On Sat, May 15, 2021 at 7:14 PM Mark Adams <[email protected] <mailto:[email protected]>> wrote: Thanks, Now I get an error in make all. THis was working a few weeks ago. Make.log was empty but here is the output. .... CC arch-cori-gpu-opt-kokkos-gcc/obj/sys/error/ftn-custom/zerrf.o CC arch-cori-gpu-opt-kokkos-gcc/obj/sys/f90-src/f90_cwrap.o CC arch-cori-gpu-opt-kokkos-gcc/obj/vec/is/utils/hdf5io.o PATH=/global/common/sw/cray/sles15/x86_64/zlib/1.2.11/gcc/8.2.0/pep2pal/bin:/global/common/sw/cray/cnl7/haswell/cmake/3.18.2/bin:/opt/cray/rca/2.2.20-7.0.1.1_4.62__g8e3fb5b.ari/bin:/opt/cray/alps/6.6.58-7.0.1.1_6.20__g437d88db.ari/sbin:/opt/cray/alps/default/bin:/opt/cray/job/2.2.4-7.0.1.1_3.48__g36b56f4.ari/bin:/opt/cray/pe/craype/2.6.2/bin:/usr/common/software/sles15_cgpu/openmpi/4.0.3/gcc/bin:/usr/common/software/sles15_cgpu/hpcsdk/20.11/Linux_x86_64/20.11/compilers/bin:/usr/common/software/sles15_cgpu/hpcsdk/20.11/Linux_x86_64/20.11/cuda/bin:/usr/common/software/sles15_cgpu/cuda/11.1.1/bin:/usr/common/software/sles15_cgpu/gcc/8.3.0/bin:/opt/esslurm/bin:/opt/cray/pe/modules/3.2.11.4/bin:/usr/common/software/bin:/usr/common/nsg/bin:/opt/ovis/bin:/opt/ovis/sbin:/usr/local/bin:/usr/bin:/bin:/usr/lib/mit/bin:/usr/lib/mit/sbin:/opt/cray/pe/bin:`dirname <http://3.2.11.4/bin:/usr/common/software/bin:/usr/common/nsg/bin:/opt/ovis/bin:/opt/ovis/sbin:/usr/local/bin:/usr/bin:/bin:/usr/lib/mit/bin:/usr/lib/mit/sbin:/opt/cray/pe/bin:dirname> /usr/common/software/sles15_cgpu/cuda/11.1.1/bin/nvcc` NVCC_WRAPPER_DEFAULT_COMPILER=/usr/common/software/sles15_cgpu/openmpi/4.0.3/gcc/bin/mpicxx /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/bin/nvcc_wrapper --expt-extended-lambda -c -arch=sm_70 -Xcompiler -rdynamic -DLANDAU_DIM=2 -DLANDAU_MAX_SPECIES=10 -DPETSC_HAVE_CUDA_ATOMIC -DLANDAU_MAX_Q=4 -Xcompiler -fPIC -O3 -gencode arch=compute_80,code=sm_80 -I/usr/common/software/sles15_cgpu/openmpi/4.0.3/gcc/include -Wno-deprecated-gpu-targets -I/global/homes/m/madams/petsc/include -I/global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/include -I/usr/common/software/sles15_cgpu/openmpi/4.0.3/gcc/include -I/usr/common/software/sles15_cgpu/cuda/11.1.1/include /global/u2/m/madams/petsc/src/sys/objects/kokkos/kinit.kokkos.cxx -o arch-cori-gpu-opt-kokkos-gcc/obj/sys/objects/kokkos/kinit.o g++: error: arch=compute_80: No such file or directory g++: error: code=sm_80: No such file or directory gmake[3]: *** [gmakefile:188: arch-cori-gpu-opt-kokkos-gcc/obj/sys/objects/kokkos/kinit.o] Error 1 gmake[3]: *** Waiting for unfinished jobs.... CC arch-cori-gpu-opt-kokkos-gcc/obj/vec/is/utils/isio.o CC arch-cori-gpu-opt-kokkos-gcc/obj/vec/is/utils/ftn-auto/pmapf.o CC arch-cori-gpu-opt-kokkos-gcc/obj/vec/is/utils/pmap.o CC arch-cori-gpu-opt-kokkos-gcc/obj/vec/is/utils/psort.o CC arch-cori-gpu-opt-kokkos-gcc/obj/vec/is/utils/isltog.o FC arch-cori-gpu-opt-kokkos-gcc/obj/vec/f90-mod/petscvecmod.o CUDAC arch-cori-gpu-opt-kokkos-gcc/obj/sys/memory/cuda/mcudahost.o CUDAC.dep arch-cori-gpu-opt-kokkos-gcc/obj/sys/memory/cuda/mcudahost.o gmake[2]: *** [/global/homes/m/madams/petsc/lib/petsc/conf/rules:50: libs] Error 2 **************************ERROR************************************* Error during compile, check arch-cori-gpu-opt-kokkos-gcc/lib/petsc/conf/make.log Send it and arch-cori-gpu-opt-kokkos-gcc/lib/petsc/conf/configure.log to [email protected] <mailto:[email protected]> ******************************************************************** gmake[1]: *** [makefile:40: all] Error 1 make: *** [GNUmakefile:9: all] Error 2
On Sat, May 15, 2021 at 5:30 PM Barry Smith <[email protected] <mailto:[email protected]>> wrote:
Bitbucket is off-line. Do you need fortran stubs at this moment? If not use --with-fortran-interface=0
On May 15, 2021, at 2:51 PM, Mark Adams <[email protected] <mailto:[email protected]>> wrote:
I can build a non-kokkos PETSc here (Cori/GPU) but I get this error with Kokkos. Any suggestions? Thanks, Mark
============================================================================================= ============================================================================================= Trying to download git://https://bitbucket.org/petsc/pkg-sowing.git <https://bitbucket.org/petsc/pkg-sowing.git> for SOWING ============================================================================================= ============================================================================================= Trying to download git://https://bitbucket.org/petsc/pkg-sowing.git <https://bitbucket.org/petsc/pkg-sowing.git> for SOWING ============================================================================================= ============================================================================================= Trying to download https://bitbucket.org/petsc/pkg-sowing/get/v1.1.26-p1.tar.gz <https://bitbucket.org/petsc/pkg-sowing/get/v1.1.26-p1.tar.gz> for SOWING ============================================================================================= ******************************************************************************* UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log for details): ------------------------------------------------------------------------------- Error during download/extract/detection of SOWING: Unable to clone sowing Could not execute "['git clone https://bitbucket.org/petsc/pkg-sowing.git <https://bitbucket.org/petsc/pkg-sowing.git> /global/u2/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/externalpackages/git.sowing']": Cloning into '/global/u2/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/externalpackages/git.sowing'... fatal: unable to access 'https://bitbucket.org/petsc/pkg-sowing.git/ <https://bitbucket.org/petsc/pkg-sowing.git/>': The requested URL returned error: 503 Unable to download package SOWING from: git://https://bitbucket.org/petsc/pkg-sowing.git <https://bitbucket.org/petsc/pkg-sowing.git> * If URL specified manually - perhaps there is a typo? * If your network is disconnected - please reconnect and rerun ./configure * Or perhaps you have a firewall blocking the download * You can run with --with-packages-download-dir=/adirectory and ./configure will instruct you what packages to download manually * or you can download the above URL manually, to /yourselectedlocation and use the configure option: --download-sowing=/yourselectedlocation Unable to clone sowing Could not execute "['git clone https://bitbucket.org/petsc/pkg-sowing.git <https://bitbucket.org/petsc/pkg-sowing.git> /global/u2/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/externalpackages/git.sowing']": Cloning into '/global/u2/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/externalpackages/git.sowing'... fatal: unable to access 'https://bitbucket.org/petsc/pkg-sowing.git/ <https://bitbucket.org/petsc/pkg-sowing.git/>': The requested URL returned error: 503 Unable to download package SOWING from: git://https://bitbucket.org/petsc/pkg-sowing.git <https://bitbucket.org/petsc/pkg-sowing.git> * If URL specified manually - perhaps there is a typo? * If your network is disconnected - please reconnect and rerun ./configure * Or perhaps you have a firewall blocking the download * You can run with --with-packages-download-dir=/adirectory and ./configure will instruct you what packages to download manually * or you can download the above URL manually, to /yourselectedlocation and use the configure option: --download-sowing=/yourselectedlocation file could not be opened successfully Downloaded package SOWING from: https://bitbucket.org/petsc/pkg-sowing/get/v1.1.26-p1.tar.gz <https://bitbucket.org/petsc/pkg-sowing/get/v1.1.26-p1.tar.gz> is not a tarball. [or installed python cannot process compressed files] * If you are behind a firewall - please fix your proxy and rerun ./configure For example at LANL you may need to set the environmental variable http_proxy (or HTTP_PROXY?) to http://proxyout.lanl.gov <http://proxyout.lanl.gov/> * You can run with --with-packages-download-dir=/adirectory and ./configure will instruct you what packages to download manually * or you can download the above URL manually, to /yourselectedlocation/v1.1.26-p1.tar.gz and use the configure option: --download-sowing=/yourselectedlocation/v1.1.26-p1.tar.gz ******************************************************************************* <configure.log>
progress: 04:08 cgpu11 adams/cusparse-cpu-solve *= ~/petsc$ make PETSC_DIR=/global/homes/m/madams/petsc PETSC_ARCH=arch-cori-gpu-opt-kokkos-gcc check Running check examples to verify correct installation Using PETSC_DIR=/global/homes/m/madams/petsc and PETSC_ARCH=arch-cori-gpu-opt-kokkos-gcc C/C++ example src/snes/tutorials/ex19 run successfully with 1 MPI process Possible error running C/C++ src/snes/tutorials/ex19 with 2 MPI processes See http://www.mcs.anl.gov/petsc/documentation/faq.html srun: error: Unable to create step for job 1921749: More processors requested than permitted 2,15c2,32 < 0 SNES Function norm 2.391552133017e-01 < 0 KSP Residual norm 2.325621076120e-01 < 1 KSP Residual norm 1.654206318674e-02 < 2 KSP Residual norm 7.202836119880e-04 < 3 KSP Residual norm 1.796861424199e-05 < 4 KSP Residual norm 2.461332992052e-07 < 1 SNES Function norm 6.826585648929e-05 < 0 KSP Residual norm 2.347339172985e-05 < 1 KSP Residual norm 8.356798075993e-07 < 2 KSP Residual norm 1.844045309619e-08 < 3 KSP Residual norm 5.336386977405e-10 < 4 KSP Residual norm 2.662608472862e-11 < 2 SNES Function norm 6.549682264799e-11 < Number of SNES iterations = 2 ---
Kokkos::Cuda::initialize ERROR: likely mismatch of architecture [cgpu11:39927] *** Process received signal *** [cgpu11:39927] Signal: Aborted (6) [cgpu11:39927] Signal code: (-6) [cgpu11:39927] [ 0] /lib64/libpthread.so.0(+0x12360)[0x2aab12731360] [cgpu11:39927] [ 1] /lib64/libc.so.6(gsignal+0x110)[0x2aab12973160] [cgpu11:39927] [ 2] /lib64/libc.so.6(abort+0x151)[0x2aab12974741] [cgpu11:39927] [ 3] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libkokkoscore.so.3.4(+0x408e5)[0x2aaabbfb88e5] [cgpu11:39927] [ 4] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libkokkoscore.so.3.4(_ZN6Kokkos4Impl12CudaInternal10initializeEiP11CUstream_st+0xb5c)[0x2aaabbfc762c] [cgpu11:39927] [ 5] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libkokkoscore.so.3.4(_ZN6Kokkos4Impl16ExecSpaceManager17initialize_spacesERKNS_13InitArgumentsE+0x24)[0x2aaabbfa1224] [cgpu11:39927] [ 6] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libkokkoscore.so.3.4(_ZN6Kokkos4Impl70_GLOBAL__N__46_tmpxft_00011209_00000000_7_Kokkos_Core_cpp1_ii_889c95a619initialize_internalERKNS_13InitArgumentsE+0x11)[0x2aaabbfa49d1] [cgpu11:39927] [ 7] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libpetsc.so.3.015(PetscKokkosInitializeCheck+0x1f9)[0x2aaaaafad809] [cgpu11:39927] [ 8] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libpetsc.so.3.015(+0x68541b)[0x2aaaab35841b] [cgpu11:39927] [ 9] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libpetsc.so.3.015(+0x36a224)[0x2aaaab03d224] [cgpu11:39927] [10] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libpetsc.so.3.015(+0x35f596)[0x2aaaab032596] [cgpu11:39927] [11] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libpetsc.so.3.015(+0x2f9d94)[0x2aaaaafccd94] [cgpu11:39927] [12] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libpetsc.so.3.015(PetscSFBcastWithMemTypeBegin+0x11f)[0x2aaaab04bc3f] [cgpu11:39927] [13] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libpetsc.so.3.015(+0x37f53a)[0x2aaaab05253a] [cgpu11:39927] [14] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libpetsc.so.3.015(VecScatterBegin+0x70)[0x2aaaab057b30] [cgpu11:39927] [15] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libpetsc.so.3.015(DMGlobalToLocalBegin_DA+0x1f)[0x2aaaabc27a5f] [cgpu11:39927] [16] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libpetsc.so.3.015(DMGlobalToLocalBegin+0x1dd)[0x2aaaabe61b3d] [cgpu11:39927] [17] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libpetsc.so.3.015(+0x144567a)[0x2aaaac11867a] [cgpu11:39927] [18] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libpetsc.so.3.015(SNESComputeFunction+0xf5)[0x2aaaac138ef5] [cgpu11:39927] [19] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libpetsc.so.3.015(+0x14ad0de)[0x2aaaac1800de] [cgpu11:39927] [20] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libpetsc.so.3.015(SNESSolve+0x821)[0x2aaaac146ed1] [cgpu11:39927] [21] /global/u2/m/madams/petsc/src/snes/tutorials/./ex19[0x401a69] [cgpu11:39927] [22] /lib64/libc.so.6(__libc_start_main+0xea)[0x2aab1295df8a] [cgpu11:39927] [23] /global/u2/m/madams/petsc/src/snes/tutorials/./ex19[0x4021ba] [cgpu11:39927] *** End of error message *** srun: error: cgpu11: task 0: Aborted srun: Terminating job step 1921749.6 /global/homes/m/madams/petsc/src/snes/tutorials Possible problem with ex19 running with cuda, diffs above ========================================= gmake[3]: [makefile:102: runex3k_kokkos] Error 1 (ignored)
On Sun, May 16, 2021 at 12:31 AM Barry Smith <[email protected]> wrote:
Looks like nvcc_wrapper gets confused when both -arch and -gencode are used. But handles either of them separately correctly.
Mark,
PETSc configure now handles the -arch and --with-kokkos-cuda-arch= business automatically so you do not, and generally shouldn't pass -arch sm_70 in --CUDAFLAGS anymore or use the --with-kokkos-cuda-arch option anymore.
If you need to set the -arch to something earlier than the system supports (rare) you can now use -with-cuda-gencodearch to set that instead of using CUDAFLAGS.
Barry
$ NVCC_WRAPPER_DEFAULT_COMPILER=mpicxx /home/bsmith/petsc/arch-main/bin/nvcc_wrapper --show --expt-extended-lambda -c -Xcompiler -fPIC -g -gencode arch=compute_70,code=sm_70 -I/home/bsmith/soft/gnu-mpich/include -Wno-deprecated-gpu-targets -I/home/bsmith/petsc/include -I/home/bsmith/petsc/arch-main/include -I/nfs/apps/spacks/2021-02-09/opt/spack/linux-centos7-x86_64/gcc-7.3.0/cuda-11.2.0-ikruu5mo6dtt3avvmwsejouqhxu4btdm/include /home/bsmith/petsc/src/ts/utils/dmplexlandau/kokkos/landau.kokkos.cxx -o arch-main/obj/ts/utils/dmplexlandau/kokkos/landau.o nvcc --expt-extended-lambda -gencode arch=compute_70,code=sm_70 -Wno-deprecated-gpu-targets -ccbin mpicxx -g -I/home/bsmith/soft/gnu-mpich/include -I/home/bsmith/petsc/include -I/home/bsmith/petsc/arch-main/include -I/nfs/apps/spacks/2021-02-09/opt/spack/linux-centos7-x86_64/gcc-7.3.0/cuda-11.2.0-ikruu5mo6dtt3avvmwsejouqhxu4btdm/include -Xcompiler -fPIC -x cu /home/bsmith/petsc/src/ts/utils/dmplexlandau/kokkos/landau.kokkos.cxx -c -o arch-main/obj/ts/utils/dmplexlandau/kokkos/landau.o ~/petsc* (main=)* arch-main
$ NVCC_WRAPPER_DEFAULT_COMPILER=mpicxx /home/bsmith/petsc/arch-main/bin/nvcc_wrapper --show --expt-extended-lambda -c -arch=sm_70 -Xcompiler -rdynamic -DLANDAU_DIM=2 -DLANDAU_MAX_SPECIES=10 -DPETSC_HAVE_CUDA_ATOMIC -DLANDAU_MAX_Q=4 -Xcompiler -fPIC -O3 -gencode arch=compute_80,code=sm_80 -I/usr/common/software/sles15_cgpu/openmpi/4.0.3/gcc/include -Wno-deprecated-gpu-targets nvcc --expt-extended-lambda -arch=sm_70 -Wno-deprecated-gpu-targets -ccbin mpicxx -DLANDAU_DIM=2 -DLANDAU_MAX_SPECIES=10 -DPETSC_HAVE_CUDA_ATOMIC -DLANDAU_MAX_Q=4 -O3 -I/usr/common/software/sles15_cgpu/openmpi/4.0.3/gcc/include -Xcompiler -rdynamic,-fPIC,arch=compute_80,code=sm_80 -c ~/petsc* (main=)* arch-main
$ NVCC_WRAPPER_DEFAULT_COMPILER=mpicxx /home/bsmith/petsc/arch-main/bin/nvcc_wrapper --show --expt-extended-lambda -c -Xcompiler -fPIC -g -arch=sm_70 -gencode arch=compute_70,code=sm_70 -I/home/bsmith/soft/gnu-mpich/include -Wno-deprecated-gpu-targets -I/home/bsmith/petsc/include -I/home/bsmith/petsc/arch-main/include -I/nfs/apps/spacks/2021-02-09/opt/spack/linux-centos7-x86_64/gcc-7.3.0/cuda-11.2.0-ikruu5mo6dtt3avvmwsejouqhxu4btdm/include /home/bsmith/petsc/src/ts/utils/dmplexlandau/kokkos/landau.kokkos.cxx -o arch-main/obj/ts/utils/dmplexlandau/kokkos/landau.o nvcc --expt-extended-lambda -arch=sm_70 -Wno-deprecated-gpu-targets -ccbin mpicxx -g -I/home/bsmith/soft/gnu-mpich/include -I/home/bsmith/petsc/include -I/home/bsmith/petsc/arch-main/include -I/nfs/apps/spacks/2021-02-09/opt/spack/linux-centos7-x86_64/gcc-7.3.0/cuda-11.2.0-ikruu5mo6dtt3avvmwsejouqhxu4btdm/include -Xcompiler -fPIC,arch=compute_70,code=sm_70 -x cu /home/bsmith/petsc/src/ts/utils/dmplexlandau/kokkos/landau.kokkos.cxx -c -o arch-main/obj/ts/utils/dmplexlandau/kokkos/landau.o
On May 15, 2021, at 8:44 PM, Junchao Zhang <[email protected]> wrote:
Add --show to the failing nvcc_wrapper command and see how nvcc_wrapper parses the options
PATH=/global/common/sw/cray/sles15/x86_64/zlib/1.2.11/gcc/8.2.0/pep2pal/bin:/global/common/sw/cray/cnl7/haswell/cmake/3.18.2/bin:/opt/cray/rca/2.2.20-7.0.1.1_4.62__g8e3fb5b.ari/bin:/opt/cray/alps/6.6.58-7.0.1.1_6.20__g437d88db.ari/sbin:/opt/cray/alps/default/bin:/opt/cray/job/2.2.4-7.0.1.1_3.48__g36b56f4.ari/bin:/opt/cray/pe/craype/2.6.2/bin:/usr/common/software/sles15_cgpu/openmpi/4.0.3/gcc/bin:/usr/common/software/sles15_cgpu/hpcsdk/20.11/Linux_x86_64/20.11/compilers/bin:/usr/common/software/sles15_cgpu/hpcsdk/20.11/Linux_x86_64/20.11/cuda/bin:/usr/common/software/sles15_cgpu/cuda/11.1.1/bin:/usr/common/software/sles15_cgpu/gcc/8.3.0/bin:/opt/esslurm/bin:/opt/cray/pe/modules/ 3.2.11.4/bin:/usr/common/software/bin:/usr/common/nsg/bin:/opt/ovis/bin:/opt/ovis/sbin:/usr/local/bin:/usr/bin:/bin:/usr/lib/mit/bin:/usr/lib/mit/sbin:/opt/cray/pe/bin:`dirname <http://3.2.11.4/bin:/usr/common/software/bin:/usr/common/nsg/bin:/opt/ovis/bin:/opt/ovis/sbin:/usr/local/bin:/usr/bin:/bin:/usr/lib/mit/bin:/usr/lib/mit/sbin:/opt/cray/pe/bin:dirname> /usr/common/software/sles15_cgpu/cuda/11.1.1/bin/nvcc` NVCC_WRAPPER_DEFAULT_COMPILER=/usr/common/software/sles15_cgpu/openmpi/4.0.3/gcc/bin/mpicxx /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/bin/nvcc_wrapper --expt-extended-lambda -c -arch=sm_70 -Xcompiler -rdynamic -DLANDAU_DIM=2 -DLANDAU_MAX_SPECIES=10 -DPETSC_HAVE_CUDA_ATOMIC -DLANDAU_MAX_Q=4 -Xcompiler -fPIC -O3 -gencode arch=compute_80,code=sm_80 -I/usr/common/software/sles15_cgpu/openmpi/4.0.3/gcc/include -Wno-deprecated-gpu-targets -I/global/homes/m/madams/petsc/include -I/global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/include -I/usr/common/software/sles15_cgpu/openmpi/4.0.3/gcc/include -I/usr/common/software/sles15_cgpu/cuda/11.1.1/include /global/u2/m/madams/petsc/src/sys/objects/kokkos/kinit.kokkos.cxx -o arch-cori-gpu-opt-kokkos-gcc/obj/sys/objects/kokkos/kinit.o --show
--Junchao Zhang
On Sat, May 15, 2021 at 7:14 PM Mark Adams <[email protected]> wrote:
Thanks, Now I get an error in make all. THis was working a few weeks ago. Make.log was empty but here is the output. .... CC arch-cori-gpu-opt-kokkos-gcc/obj/sys/error/ftn-custom/zerrf.o CC arch-cori-gpu-opt-kokkos-gcc/obj/sys/f90-src/f90_cwrap.o CC arch-cori-gpu-opt-kokkos-gcc/obj/vec/is/utils/hdf5io.o
PATH=/global/common/sw/cray/sles15/x86_64/zlib/1.2.11/gcc/8.2.0/pep2pal/bin:/global/common/sw/cray/cnl7/haswell/cmake/3.18.2/bin:/opt/cray/rca/2.2.20-7.0.1.1_4.62__g8e3fb5b.ari/bin:/opt/cray/alps/6.6.58-7.0.1.1_6.20__g437d88db.ari/sbin:/opt/cray/alps/default/bin:/opt/cray/job/2.2.4-7.0.1.1_3.48__g36b56f4.ari/bin:/opt/cray/pe/craype/2.6.2/bin:/usr/common/software/sles15_cgpu/openmpi/4.0.3/gcc/bin:/usr/common/software/sles15_cgpu/hpcsdk/20.11/Linux_x86_64/20.11/compilers/bin:/usr/common/software/sles15_cgpu/hpcsdk/20.11/Linux_x86_64/20.11/cuda/bin:/usr/common/software/sles15_cgpu/cuda/11.1.1/bin:/usr/common/software/sles15_cgpu/gcc/8.3.0/bin:/opt/esslurm/bin:/opt/cray/pe/modules/ 3.2.11.4/bin:/usr/common/software/bin:/usr/common/nsg/bin:/opt/ovis/bin:/opt/ovis/sbin:/usr/local/bin:/usr/bin:/bin:/usr/lib/mit/bin:/usr/lib/mit/sbin:/opt/cray/pe/bin:`dirname <http://3.2.11.4/bin:/usr/common/software/bin:/usr/common/nsg/bin:/opt/ovis/bin:/opt/ovis/sbin:/usr/local/bin:/usr/bin:/bin:/usr/lib/mit/bin:/usr/lib/mit/sbin:/opt/cray/pe/bin:dirname> /usr/common/software/sles15_cgpu/cuda/11.1.1/bin/nvcc` NVCC_WRAPPER_DEFAULT_COMPILER=/usr/common/software/sles15_cgpu/openmpi/4.0.3/gcc/bin/mpicxx /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/bin/nvcc_wrapper --expt-extended-lambda -c -arch=sm_70 -Xcompiler -rdynamic -DLANDAU_DIM=2 -DLANDAU_MAX_SPECIES=10 -DPETSC_HAVE_CUDA_ATOMIC -DLANDAU_MAX_Q=4 -Xcompiler -fPIC -O3 -gencode arch=compute_80,code=sm_80 -I/usr/common/software/sles15_cgpu/openmpi/4.0.3/gcc/include -Wno-deprecated-gpu-targets -I/global/homes/m/madams/petsc/include -I/global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/include -I/usr/common/software/sles15_cgpu/openmpi/4.0.3/gcc/include -I/usr/common/software/sles15_cgpu/cuda/11.1.1/include /global/u2/m/madams/petsc/src/sys/objects/kokkos/kinit.kokkos.cxx -o arch-cori-gpu-opt-kokkos-gcc/obj/sys/objects/kokkos/kinit.o g++: error: arch=compute_80: No such file or directory g++: error: code=sm_80: No such file or directory gmake[3]: *** [gmakefile:188: arch-cori-gpu-opt-kokkos-gcc/obj/sys/objects/kokkos/kinit.o] Error 1 gmake[3]: *** Waiting for unfinished jobs.... CC arch-cori-gpu-opt-kokkos-gcc/obj/vec/is/utils/isio.o CC arch-cori-gpu-opt-kokkos-gcc/obj/vec/is/utils/ftn-auto/pmapf.o CC arch-cori-gpu-opt-kokkos-gcc/obj/vec/is/utils/pmap.o CC arch-cori-gpu-opt-kokkos-gcc/obj/vec/is/utils/psort.o CC arch-cori-gpu-opt-kokkos-gcc/obj/vec/is/utils/isltog.o FC arch-cori-gpu-opt-kokkos-gcc/obj/vec/f90-mod/petscvecmod.o CUDAC arch-cori-gpu-opt-kokkos-gcc/obj/sys/memory/cuda/mcudahost.o CUDAC.dep arch-cori-gpu-opt-kokkos-gcc/obj/sys/memory/cuda/mcudahost.o gmake[2]: *** [/global/homes/m/madams/petsc/lib/petsc/conf/rules:50: libs] Error 2 **************************ERROR************************************* Error during compile, check arch-cori-gpu-opt-kokkos-gcc/lib/petsc/conf/make.log Send it and arch-cori-gpu-opt-kokkos-gcc/lib/petsc/conf/configure.log to [email protected] ******************************************************************** gmake[1]: *** [makefile:40: all] Error 1 make: *** [GNUmakefile:9: all] Error 2
On Sat, May 15, 2021 at 5:30 PM Barry Smith <[email protected]> wrote:
Bitbucket is off-line. Do you need fortran stubs at this moment? If not use --with-fortran-interface=0
On May 15, 2021, at 2:51 PM, Mark Adams <[email protected]> wrote:
I can build a non-kokkos PETSc here (Cori/GPU) but I get this error with Kokkos. Any suggestions? Thanks, Mark
=============================================================================================
=============================================================================================
Trying to download git://https://bitbucket.org/petsc/pkg-sowing.git for SOWING
=============================================================================================
=============================================================================================
Trying to download git://https://bitbucket.org/petsc/pkg-sowing.git for SOWING
=============================================================================================
=============================================================================================
Trying to download https://bitbucket.org/petsc/pkg-sowing/get/v1.1.26-p1.tar.gz for SOWING
=============================================================================================
******************************************************************************* UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log for details):
------------------------------------------------------------------------------- Error during download/extract/detection of SOWING: Unable to clone sowing Could not execute "['git clone https://bitbucket.org/petsc/pkg-sowing.git /global/u2/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/externalpackages/git.sowing']": Cloning into '/global/u2/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/externalpackages/git.sowing'... fatal: unable to access 'https://bitbucket.org/petsc/pkg-sowing.git/': The requested URL returned error: 503 Unable to download package SOWING from: git:// https://bitbucket.org/petsc/pkg-sowing.git * If URL specified manually - perhaps there is a typo? * If your network is disconnected - please reconnect and rerun ./configure * Or perhaps you have a firewall blocking the download * You can run with --with-packages-download-dir=/adirectory and ./configure will instruct you what packages to download manually * or you can download the above URL manually, to /yourselectedlocation and use the configure option: --download-sowing=/yourselectedlocation Unable to clone sowing Could not execute "['git clone https://bitbucket.org/petsc/pkg-sowing.git /global/u2/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/externalpackages/git.sowing']": Cloning into '/global/u2/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/externalpackages/git.sowing'... fatal: unable to access 'https://bitbucket.org/petsc/pkg-sowing.git/': The requested URL returned error: 503 Unable to download package SOWING from: git:// https://bitbucket.org/petsc/pkg-sowing.git * If URL specified manually - perhaps there is a typo? * If your network is disconnected - please reconnect and rerun ./configure * Or perhaps you have a firewall blocking the download * You can run with --with-packages-download-dir=/adirectory and ./configure will instruct you what packages to download manually * or you can download the above URL manually, to /yourselectedlocation and use the configure option: --download-sowing=/yourselectedlocation file could not be opened successfully Downloaded package SOWING from: https://bitbucket.org/petsc/pkg-sowing/get/v1.1.26-p1.tar.gz is not a tarball. [or installed python cannot process compressed files] * If you are behind a firewall - please fix your proxy and rerun ./configure For example at LANL you may need to set the environmental variable http_proxy (or HTTP_PROXY?) to http://proxyout.lanl.gov * You can run with --with-packages-download-dir=/adirectory and ./configure will instruct you what packages to download manually * or you can download the above URL manually, to /yourselectedlocation/v1.1.26-p1.tar.gz and use the configure option: --download-sowing=/yourselectedlocation/v1.1.26-p1.tar.gz
******************************************************************************* <configure.log>
Remove --with-kokkos-cuda-arch=VOLTA70 from your configure Barry: It seems we should either disable this option, or rename it to --with-cuda-arch and get it accepted by both kokkos.py and cuda.py --Junchao Zhang On Sun, May 16, 2021 at 6:15 AM Mark Adams <[email protected]> wrote:
progress:
04:08 cgpu11 adams/cusparse-cpu-solve *= ~/petsc$ make PETSC_DIR=/global/homes/m/madams/petsc PETSC_ARCH=arch-cori-gpu-opt-kokkos-gcc check Running check examples to verify correct installation Using PETSC_DIR=/global/homes/m/madams/petsc and PETSC_ARCH=arch-cori-gpu-opt-kokkos-gcc C/C++ example src/snes/tutorials/ex19 run successfully with 1 MPI process Possible error running C/C++ src/snes/tutorials/ex19 with 2 MPI processes See http://www.mcs.anl.gov/petsc/documentation/faq.html srun: error: Unable to create step for job 1921749: More processors requested than permitted 2,15c2,32 < 0 SNES Function norm 2.391552133017e-01 < 0 KSP Residual norm 2.325621076120e-01 < 1 KSP Residual norm 1.654206318674e-02 < 2 KSP Residual norm 7.202836119880e-04 < 3 KSP Residual norm 1.796861424199e-05 < 4 KSP Residual norm 2.461332992052e-07 < 1 SNES Function norm 6.826585648929e-05 < 0 KSP Residual norm 2.347339172985e-05 < 1 KSP Residual norm 8.356798075993e-07 < 2 KSP Residual norm 1.844045309619e-08 < 3 KSP Residual norm 5.336386977405e-10 < 4 KSP Residual norm 2.662608472862e-11 < 2 SNES Function norm 6.549682264799e-11 < Number of SNES iterations = 2 ---
Kokkos::Cuda::initialize ERROR: likely mismatch of architecture [cgpu11:39927] *** Process received signal *** [cgpu11:39927] Signal: Aborted (6) [cgpu11:39927] Signal code: (-6) [cgpu11:39927] [ 0] /lib64/libpthread.so.0(+0x12360)[0x2aab12731360] [cgpu11:39927] [ 1] /lib64/libc.so.6(gsignal+0x110)[0x2aab12973160] [cgpu11:39927] [ 2] /lib64/libc.so.6(abort+0x151)[0x2aab12974741] [cgpu11:39927] [ 3] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libkokkoscore.so.3.4(+0x408e5)[0x2aaabbfb88e5] [cgpu11:39927] [ 4] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libkokkoscore.so.3.4(_ZN6Kokkos4Impl12CudaInternal10initializeEiP11CUstream_st+0xb5c)[0x2aaabbfc762c] [cgpu11:39927] [ 5] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libkokkoscore.so.3.4(_ZN6Kokkos4Impl16ExecSpaceManager17initialize_spacesERKNS_13InitArgumentsE+0x24)[0x2aaabbfa1224] [cgpu11:39927] [ 6] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libkokkoscore.so.3.4(_ZN6Kokkos4Impl70_GLOBAL__N__46_tmpxft_00011209_00000000_7_Kokkos_Core_cpp1_ii_889c95a619initialize_internalERKNS_13InitArgumentsE+0x11)[0x2aaabbfa49d1] [cgpu11:39927] [ 7] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libpetsc.so.3.015(PetscKokkosInitializeCheck+0x1f9)[0x2aaaaafad809] [cgpu11:39927] [ 8] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libpetsc.so.3.015(+0x68541b)[0x2aaaab35841b] [cgpu11:39927] [ 9] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libpetsc.so.3.015(+0x36a224)[0x2aaaab03d224] [cgpu11:39927] [10] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libpetsc.so.3.015(+0x35f596)[0x2aaaab032596] [cgpu11:39927] [11] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libpetsc.so.3.015(+0x2f9d94)[0x2aaaaafccd94] [cgpu11:39927] [12] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libpetsc.so.3.015(PetscSFBcastWithMemTypeBegin+0x11f)[0x2aaaab04bc3f] [cgpu11:39927] [13] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libpetsc.so.3.015(+0x37f53a)[0x2aaaab05253a] [cgpu11:39927] [14] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libpetsc.so.3.015(VecScatterBegin+0x70)[0x2aaaab057b30] [cgpu11:39927] [15] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libpetsc.so.3.015(DMGlobalToLocalBegin_DA+0x1f)[0x2aaaabc27a5f] [cgpu11:39927] [16] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libpetsc.so.3.015(DMGlobalToLocalBegin+0x1dd)[0x2aaaabe61b3d] [cgpu11:39927] [17] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libpetsc.so.3.015(+0x144567a)[0x2aaaac11867a] [cgpu11:39927] [18] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libpetsc.so.3.015(SNESComputeFunction+0xf5)[0x2aaaac138ef5] [cgpu11:39927] [19] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libpetsc.so.3.015(+0x14ad0de)[0x2aaaac1800de] [cgpu11:39927] [20] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libpetsc.so.3.015(SNESSolve+0x821)[0x2aaaac146ed1] [cgpu11:39927] [21] /global/u2/m/madams/petsc/src/snes/tutorials/./ex19[0x401a69] [cgpu11:39927] [22] /lib64/libc.so.6(__libc_start_main+0xea)[0x2aab1295df8a] [cgpu11:39927] [23] /global/u2/m/madams/petsc/src/snes/tutorials/./ex19[0x4021ba] [cgpu11:39927] *** End of error message *** srun: error: cgpu11: task 0: Aborted srun: Terminating job step 1921749.6 /global/homes/m/madams/petsc/src/snes/tutorials Possible problem with ex19 running with cuda, diffs above ========================================= gmake[3]: [makefile:102: runex3k_kokkos] Error 1 (ignored)
On Sun, May 16, 2021 at 12:31 AM Barry Smith <[email protected]> wrote:
Looks like nvcc_wrapper gets confused when both -arch and -gencode are used. But handles either of them separately correctly.
Mark,
PETSc configure now handles the -arch and --with-kokkos-cuda-arch= business automatically so you do not, and generally shouldn't pass -arch sm_70 in --CUDAFLAGS anymore or use the --with-kokkos-cuda-arch option anymore.
If you need to set the -arch to something earlier than the system supports (rare) you can now use -with-cuda-gencodearch to set that instead of using CUDAFLAGS.
Barry
$ NVCC_WRAPPER_DEFAULT_COMPILER=mpicxx /home/bsmith/petsc/arch-main/bin/nvcc_wrapper --show --expt-extended-lambda -c -Xcompiler -fPIC -g -gencode arch=compute_70,code=sm_70 -I/home/bsmith/soft/gnu-mpich/include -Wno-deprecated-gpu-targets -I/home/bsmith/petsc/include -I/home/bsmith/petsc/arch-main/include -I/nfs/apps/spacks/2021-02-09/opt/spack/linux-centos7-x86_64/gcc-7.3.0/cuda-11.2.0-ikruu5mo6dtt3avvmwsejouqhxu4btdm/include /home/bsmith/petsc/src/ts/utils/dmplexlandau/kokkos/landau.kokkos.cxx -o arch-main/obj/ts/utils/dmplexlandau/kokkos/landau.o nvcc --expt-extended-lambda -gencode arch=compute_70,code=sm_70 -Wno-deprecated-gpu-targets -ccbin mpicxx -g -I/home/bsmith/soft/gnu-mpich/include -I/home/bsmith/petsc/include -I/home/bsmith/petsc/arch-main/include -I/nfs/apps/spacks/2021-02-09/opt/spack/linux-centos7-x86_64/gcc-7.3.0/cuda-11.2.0-ikruu5mo6dtt3avvmwsejouqhxu4btdm/include -Xcompiler -fPIC -x cu /home/bsmith/petsc/src/ts/utils/dmplexlandau/kokkos/landau.kokkos.cxx -c -o arch-main/obj/ts/utils/dmplexlandau/kokkos/landau.o ~/petsc* (main=)* arch-main
$ NVCC_WRAPPER_DEFAULT_COMPILER=mpicxx /home/bsmith/petsc/arch-main/bin/nvcc_wrapper --show --expt-extended-lambda -c -arch=sm_70 -Xcompiler -rdynamic -DLANDAU_DIM=2 -DLANDAU_MAX_SPECIES=10 -DPETSC_HAVE_CUDA_ATOMIC -DLANDAU_MAX_Q=4 -Xcompiler -fPIC -O3 -gencode arch=compute_80,code=sm_80 -I/usr/common/software/sles15_cgpu/openmpi/4.0.3/gcc/include -Wno-deprecated-gpu-targets nvcc --expt-extended-lambda -arch=sm_70 -Wno-deprecated-gpu-targets -ccbin mpicxx -DLANDAU_DIM=2 -DLANDAU_MAX_SPECIES=10 -DPETSC_HAVE_CUDA_ATOMIC -DLANDAU_MAX_Q=4 -O3 -I/usr/common/software/sles15_cgpu/openmpi/4.0.3/gcc/include -Xcompiler -rdynamic,-fPIC,arch=compute_80,code=sm_80 -c ~/petsc* (main=)* arch-main
$ NVCC_WRAPPER_DEFAULT_COMPILER=mpicxx /home/bsmith/petsc/arch-main/bin/nvcc_wrapper --show --expt-extended-lambda -c -Xcompiler -fPIC -g -arch=sm_70 -gencode arch=compute_70,code=sm_70 -I/home/bsmith/soft/gnu-mpich/include -Wno-deprecated-gpu-targets -I/home/bsmith/petsc/include -I/home/bsmith/petsc/arch-main/include -I/nfs/apps/spacks/2021-02-09/opt/spack/linux-centos7-x86_64/gcc-7.3.0/cuda-11.2.0-ikruu5mo6dtt3avvmwsejouqhxu4btdm/include /home/bsmith/petsc/src/ts/utils/dmplexlandau/kokkos/landau.kokkos.cxx -o arch-main/obj/ts/utils/dmplexlandau/kokkos/landau.o nvcc --expt-extended-lambda -arch=sm_70 -Wno-deprecated-gpu-targets -ccbin mpicxx -g -I/home/bsmith/soft/gnu-mpich/include -I/home/bsmith/petsc/include -I/home/bsmith/petsc/arch-main/include -I/nfs/apps/spacks/2021-02-09/opt/spack/linux-centos7-x86_64/gcc-7.3.0/cuda-11.2.0-ikruu5mo6dtt3avvmwsejouqhxu4btdm/include -Xcompiler -fPIC,arch=compute_70,code=sm_70 -x cu /home/bsmith/petsc/src/ts/utils/dmplexlandau/kokkos/landau.kokkos.cxx -c -o arch-main/obj/ts/utils/dmplexlandau/kokkos/landau.o
On May 15, 2021, at 8:44 PM, Junchao Zhang <[email protected]> wrote:
Add --show to the failing nvcc_wrapper command and see how nvcc_wrapper parses the options
PATH=/global/common/sw/cray/sles15/x86_64/zlib/1.2.11/gcc/8.2.0/pep2pal/bin:/global/common/sw/cray/cnl7/haswell/cmake/3.18.2/bin:/opt/cray/rca/2.2.20-7.0.1.1_4.62__g8e3fb5b.ari/bin:/opt/cray/alps/6.6.58-7.0.1.1_6.20__g437d88db.ari/sbin:/opt/cray/alps/default/bin:/opt/cray/job/2.2.4-7.0.1.1_3.48__g36b56f4.ari/bin:/opt/cray/pe/craype/2.6.2/bin:/usr/common/software/sles15_cgpu/openmpi/4.0.3/gcc/bin:/usr/common/software/sles15_cgpu/hpcsdk/20.11/Linux_x86_64/20.11/compilers/bin:/usr/common/software/sles15_cgpu/hpcsdk/20.11/Linux_x86_64/20.11/cuda/bin:/usr/common/software/sles15_cgpu/cuda/11.1.1/bin:/usr/common/software/sles15_cgpu/gcc/8.3.0/bin:/opt/esslurm/bin:/opt/cray/pe/modules/ 3.2.11.4/bin:/usr/common/software/bin:/usr/common/nsg/bin:/opt/ovis/bin:/opt/ovis/sbin:/usr/local/bin:/usr/bin:/bin:/usr/lib/mit/bin:/usr/lib/mit/sbin:/opt/cray/pe/bin:`dirname <http://3.2.11.4/bin:/usr/common/software/bin:/usr/common/nsg/bin:/opt/ovis/bin:/opt/ovis/sbin:/usr/local/bin:/usr/bin:/bin:/usr/lib/mit/bin:/usr/lib/mit/sbin:/opt/cray/pe/bin:dirname> /usr/common/software/sles15_cgpu/cuda/11.1.1/bin/nvcc` NVCC_WRAPPER_DEFAULT_COMPILER=/usr/common/software/sles15_cgpu/openmpi/4.0.3/gcc/bin/mpicxx /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/bin/nvcc_wrapper --expt-extended-lambda -c -arch=sm_70 -Xcompiler -rdynamic -DLANDAU_DIM=2 -DLANDAU_MAX_SPECIES=10 -DPETSC_HAVE_CUDA_ATOMIC -DLANDAU_MAX_Q=4 -Xcompiler -fPIC -O3 -gencode arch=compute_80,code=sm_80 -I/usr/common/software/sles15_cgpu/openmpi/4.0.3/gcc/include -Wno-deprecated-gpu-targets -I/global/homes/m/madams/petsc/include -I/global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/include -I/usr/common/software/sles15_cgpu/openmpi/4.0.3/gcc/include -I/usr/common/software/sles15_cgpu/cuda/11.1.1/include /global/u2/m/madams/petsc/src/sys/objects/kokkos/kinit.kokkos.cxx -o arch-cori-gpu-opt-kokkos-gcc/obj/sys/objects/kokkos/kinit.o --show
--Junchao Zhang
On Sat, May 15, 2021 at 7:14 PM Mark Adams <[email protected]> wrote:
Thanks, Now I get an error in make all. THis was working a few weeks ago. Make.log was empty but here is the output. .... CC arch-cori-gpu-opt-kokkos-gcc/obj/sys/error/ftn-custom/zerrf.o CC arch-cori-gpu-opt-kokkos-gcc/obj/sys/f90-src/f90_cwrap.o CC arch-cori-gpu-opt-kokkos-gcc/obj/vec/is/utils/hdf5io.o
PATH=/global/common/sw/cray/sles15/x86_64/zlib/1.2.11/gcc/8.2.0/pep2pal/bin:/global/common/sw/cray/cnl7/haswell/cmake/3.18.2/bin:/opt/cray/rca/2.2.20-7.0.1.1_4.62__g8e3fb5b.ari/bin:/opt/cray/alps/6.6.58-7.0.1.1_6.20__g437d88db.ari/sbin:/opt/cray/alps/default/bin:/opt/cray/job/2.2.4-7.0.1.1_3.48__g36b56f4.ari/bin:/opt/cray/pe/craype/2.6.2/bin:/usr/common/software/sles15_cgpu/openmpi/4.0.3/gcc/bin:/usr/common/software/sles15_cgpu/hpcsdk/20.11/Linux_x86_64/20.11/compilers/bin:/usr/common/software/sles15_cgpu/hpcsdk/20.11/Linux_x86_64/20.11/cuda/bin:/usr/common/software/sles15_cgpu/cuda/11.1.1/bin:/usr/common/software/sles15_cgpu/gcc/8.3.0/bin:/opt/esslurm/bin:/opt/cray/pe/modules/ 3.2.11.4/bin:/usr/common/software/bin:/usr/common/nsg/bin:/opt/ovis/bin:/opt/ovis/sbin:/usr/local/bin:/usr/bin:/bin:/usr/lib/mit/bin:/usr/lib/mit/sbin:/opt/cray/pe/bin:`dirname <http://3.2.11.4/bin:/usr/common/software/bin:/usr/common/nsg/bin:/opt/ovis/bin:/opt/ovis/sbin:/usr/local/bin:/usr/bin:/bin:/usr/lib/mit/bin:/usr/lib/mit/sbin:/opt/cray/pe/bin:dirname> /usr/common/software/sles15_cgpu/cuda/11.1.1/bin/nvcc` NVCC_WRAPPER_DEFAULT_COMPILER=/usr/common/software/sles15_cgpu/openmpi/4.0.3/gcc/bin/mpicxx /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/bin/nvcc_wrapper --expt-extended-lambda -c -arch=sm_70 -Xcompiler -rdynamic -DLANDAU_DIM=2 -DLANDAU_MAX_SPECIES=10 -DPETSC_HAVE_CUDA_ATOMIC -DLANDAU_MAX_Q=4 -Xcompiler -fPIC -O3 -gencode arch=compute_80,code=sm_80 -I/usr/common/software/sles15_cgpu/openmpi/4.0.3/gcc/include -Wno-deprecated-gpu-targets -I/global/homes/m/madams/petsc/include -I/global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/include -I/usr/common/software/sles15_cgpu/openmpi/4.0.3/gcc/include -I/usr/common/software/sles15_cgpu/cuda/11.1.1/include /global/u2/m/madams/petsc/src/sys/objects/kokkos/kinit.kokkos.cxx -o arch-cori-gpu-opt-kokkos-gcc/obj/sys/objects/kokkos/kinit.o g++: error: arch=compute_80: No such file or directory g++: error: code=sm_80: No such file or directory gmake[3]: *** [gmakefile:188: arch-cori-gpu-opt-kokkos-gcc/obj/sys/objects/kokkos/kinit.o] Error 1 gmake[3]: *** Waiting for unfinished jobs.... CC arch-cori-gpu-opt-kokkos-gcc/obj/vec/is/utils/isio.o CC arch-cori-gpu-opt-kokkos-gcc/obj/vec/is/utils/ftn-auto/pmapf.o CC arch-cori-gpu-opt-kokkos-gcc/obj/vec/is/utils/pmap.o CC arch-cori-gpu-opt-kokkos-gcc/obj/vec/is/utils/psort.o CC arch-cori-gpu-opt-kokkos-gcc/obj/vec/is/utils/isltog.o FC arch-cori-gpu-opt-kokkos-gcc/obj/vec/f90-mod/petscvecmod.o CUDAC arch-cori-gpu-opt-kokkos-gcc/obj/sys/memory/cuda/mcudahost.o CUDAC.dep arch-cori-gpu-opt-kokkos-gcc/obj/sys/memory/cuda/mcudahost.o gmake[2]: *** [/global/homes/m/madams/petsc/lib/petsc/conf/rules:50: libs] Error 2 **************************ERROR************************************* Error during compile, check arch-cori-gpu-opt-kokkos-gcc/lib/petsc/conf/make.log Send it and arch-cori-gpu-opt-kokkos-gcc/lib/petsc/conf/configure.log to [email protected] ******************************************************************** gmake[1]: *** [makefile:40: all] Error 1 make: *** [GNUmakefile:9: all] Error 2
On Sat, May 15, 2021 at 5:30 PM Barry Smith <[email protected]> wrote:
Bitbucket is off-line. Do you need fortran stubs at this moment? If not use --with-fortran-interface=0
On May 15, 2021, at 2:51 PM, Mark Adams <[email protected]> wrote:
I can build a non-kokkos PETSc here (Cori/GPU) but I get this error with Kokkos. Any suggestions? Thanks, Mark
=============================================================================================
=============================================================================================
Trying to download git://https://bitbucket.org/petsc/pkg-sowing.git for SOWING
=============================================================================================
=============================================================================================
Trying to download git://https://bitbucket.org/petsc/pkg-sowing.git for SOWING
=============================================================================================
=============================================================================================
Trying to download https://bitbucket.org/petsc/pkg-sowing/get/v1.1.26-p1.tar.gz for SOWING
=============================================================================================
******************************************************************************* UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log for details):
------------------------------------------------------------------------------- Error during download/extract/detection of SOWING: Unable to clone sowing Could not execute "['git clone https://bitbucket.org/petsc/pkg-sowing.git /global/u2/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/externalpackages/git.sowing']": Cloning into '/global/u2/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/externalpackages/git.sowing'... fatal: unable to access 'https://bitbucket.org/petsc/pkg-sowing.git/': The requested URL returned error: 503 Unable to download package SOWING from: git:// https://bitbucket.org/petsc/pkg-sowing.git * If URL specified manually - perhaps there is a typo? * If your network is disconnected - please reconnect and rerun ./configure * Or perhaps you have a firewall blocking the download * You can run with --with-packages-download-dir=/adirectory and ./configure will instruct you what packages to download manually * or you can download the above URL manually, to /yourselectedlocation and use the configure option: --download-sowing=/yourselectedlocation Unable to clone sowing Could not execute "['git clone https://bitbucket.org/petsc/pkg-sowing.git /global/u2/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/externalpackages/git.sowing']": Cloning into '/global/u2/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/externalpackages/git.sowing'... fatal: unable to access 'https://bitbucket.org/petsc/pkg-sowing.git/': The requested URL returned error: 503 Unable to download package SOWING from: git:// https://bitbucket.org/petsc/pkg-sowing.git * If URL specified manually - perhaps there is a typo? * If your network is disconnected - please reconnect and rerun ./configure * Or perhaps you have a firewall blocking the download * You can run with --with-packages-download-dir=/adirectory and ./configure will instruct you what packages to download manually * or you can download the above URL manually, to /yourselectedlocation and use the configure option: --download-sowing=/yourselectedlocation file could not be opened successfully Downloaded package SOWING from: https://bitbucket.org/petsc/pkg-sowing/get/v1.1.26-p1.tar.gz is not a tarball. [or installed python cannot process compressed files] * If you are behind a firewall - please fix your proxy and rerun ./configure For example at LANL you may need to set the environmental variable http_proxy (or HTTP_PROXY?) to http://proxyout.lanl.gov * You can run with --with-packages-download-dir=/adirectory and ./configure will instruct you what packages to download manually * or you can download the above URL manually, to /yourselectedlocation/v1.1.26-p1.tar.gz and use the configure option: --download-sowing=/yourselectedlocation/v1.1.26-p1.tar.gz
******************************************************************************* <configure.log>
Whoops, between scp'ing and soft links I gave you the wrong log file. I removed this already. On Sun, May 16, 2021 at 8:37 AM Junchao Zhang <[email protected]> wrote:
Remove --with-kokkos-cuda-arch=VOLTA70 from your configure
Barry: It seems we should either disable this option, or rename it to --with-cuda-arch and get it accepted by both kokkos.py and cuda.py
--Junchao Zhang
On Sun, May 16, 2021 at 6:15 AM Mark Adams <[email protected]> wrote:
progress:
04:08 cgpu11 adams/cusparse-cpu-solve *= ~/petsc$ make PETSC_DIR=/global/homes/m/madams/petsc PETSC_ARCH=arch-cori-gpu-opt-kokkos-gcc check Running check examples to verify correct installation Using PETSC_DIR=/global/homes/m/madams/petsc and PETSC_ARCH=arch-cori-gpu-opt-kokkos-gcc C/C++ example src/snes/tutorials/ex19 run successfully with 1 MPI process Possible error running C/C++ src/snes/tutorials/ex19 with 2 MPI processes See http://www.mcs.anl.gov/petsc/documentation/faq.html srun: error: Unable to create step for job 1921749: More processors requested than permitted 2,15c2,32 < 0 SNES Function norm 2.391552133017e-01 < 0 KSP Residual norm 2.325621076120e-01 < 1 KSP Residual norm 1.654206318674e-02 < 2 KSP Residual norm 7.202836119880e-04 < 3 KSP Residual norm 1.796861424199e-05 < 4 KSP Residual norm 2.461332992052e-07 < 1 SNES Function norm 6.826585648929e-05 < 0 KSP Residual norm 2.347339172985e-05 < 1 KSP Residual norm 8.356798075993e-07 < 2 KSP Residual norm 1.844045309619e-08 < 3 KSP Residual norm 5.336386977405e-10 < 4 KSP Residual norm 2.662608472862e-11 < 2 SNES Function norm 6.549682264799e-11 < Number of SNES iterations = 2 ---
Kokkos::Cuda::initialize ERROR: likely mismatch of architecture [cgpu11:39927] *** Process received signal *** [cgpu11:39927] Signal: Aborted (6) [cgpu11:39927] Signal code: (-6) [cgpu11:39927] [ 0] /lib64/libpthread.so.0(+0x12360)[0x2aab12731360] [cgpu11:39927] [ 1] /lib64/libc.so.6(gsignal+0x110)[0x2aab12973160] [cgpu11:39927] [ 2] /lib64/libc.so.6(abort+0x151)[0x2aab12974741] [cgpu11:39927] [ 3] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libkokkoscore.so.3.4(+0x408e5)[0x2aaabbfb88e5] [cgpu11:39927] [ 4] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libkokkoscore.so.3.4(_ZN6Kokkos4Impl12CudaInternal10initializeEiP11CUstream_st+0xb5c)[0x2aaabbfc762c] [cgpu11:39927] [ 5] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libkokkoscore.so.3.4(_ZN6Kokkos4Impl16ExecSpaceManager17initialize_spacesERKNS_13InitArgumentsE+0x24)[0x2aaabbfa1224] [cgpu11:39927] [ 6] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libkokkoscore.so.3.4(_ZN6Kokkos4Impl70_GLOBAL__N__46_tmpxft_00011209_00000000_7_Kokkos_Core_cpp1_ii_889c95a619initialize_internalERKNS_13InitArgumentsE+0x11)[0x2aaabbfa49d1] [cgpu11:39927] [ 7] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libpetsc.so.3.015(PetscKokkosInitializeCheck+0x1f9)[0x2aaaaafad809] [cgpu11:39927] [ 8] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libpetsc.so.3.015(+0x68541b)[0x2aaaab35841b] [cgpu11:39927] [ 9] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libpetsc.so.3.015(+0x36a224)[0x2aaaab03d224] [cgpu11:39927] [10] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libpetsc.so.3.015(+0x35f596)[0x2aaaab032596] [cgpu11:39927] [11] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libpetsc.so.3.015(+0x2f9d94)[0x2aaaaafccd94] [cgpu11:39927] [12] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libpetsc.so.3.015(PetscSFBcastWithMemTypeBegin+0x11f)[0x2aaaab04bc3f] [cgpu11:39927] [13] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libpetsc.so.3.015(+0x37f53a)[0x2aaaab05253a] [cgpu11:39927] [14] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libpetsc.so.3.015(VecScatterBegin+0x70)[0x2aaaab057b30] [cgpu11:39927] [15] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libpetsc.so.3.015(DMGlobalToLocalBegin_DA+0x1f)[0x2aaaabc27a5f] [cgpu11:39927] [16] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libpetsc.so.3.015(DMGlobalToLocalBegin+0x1dd)[0x2aaaabe61b3d] [cgpu11:39927] [17] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libpetsc.so.3.015(+0x144567a)[0x2aaaac11867a] [cgpu11:39927] [18] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libpetsc.so.3.015(SNESComputeFunction+0xf5)[0x2aaaac138ef5] [cgpu11:39927] [19] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libpetsc.so.3.015(+0x14ad0de)[0x2aaaac1800de] [cgpu11:39927] [20] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libpetsc.so.3.015(SNESSolve+0x821)[0x2aaaac146ed1] [cgpu11:39927] [21] /global/u2/m/madams/petsc/src/snes/tutorials/./ex19[0x401a69] [cgpu11:39927] [22] /lib64/libc.so.6(__libc_start_main+0xea)[0x2aab1295df8a] [cgpu11:39927] [23] /global/u2/m/madams/petsc/src/snes/tutorials/./ex19[0x4021ba] [cgpu11:39927] *** End of error message *** srun: error: cgpu11: task 0: Aborted srun: Terminating job step 1921749.6 /global/homes/m/madams/petsc/src/snes/tutorials Possible problem with ex19 running with cuda, diffs above ========================================= gmake[3]: [makefile:102: runex3k_kokkos] Error 1 (ignored)
On Sun, May 16, 2021 at 12:31 AM Barry Smith <[email protected]> wrote:
Looks like nvcc_wrapper gets confused when both -arch and -gencode are used. But handles either of them separately correctly.
Mark,
PETSc configure now handles the -arch and --with-kokkos-cuda-arch= business automatically so you do not, and generally shouldn't pass -arch sm_70 in --CUDAFLAGS anymore or use the --with-kokkos-cuda-arch option anymore.
If you need to set the -arch to something earlier than the system supports (rare) you can now use -with-cuda-gencodearch to set that instead of using CUDAFLAGS.
Barry
$ NVCC_WRAPPER_DEFAULT_COMPILER=mpicxx /home/bsmith/petsc/arch-main/bin/nvcc_wrapper --show --expt-extended-lambda -c -Xcompiler -fPIC -g -gencode arch=compute_70,code=sm_70 -I/home/bsmith/soft/gnu-mpich/include -Wno-deprecated-gpu-targets -I/home/bsmith/petsc/include -I/home/bsmith/petsc/arch-main/include -I/nfs/apps/spacks/2021-02-09/opt/spack/linux-centos7-x86_64/gcc-7.3.0/cuda-11.2.0-ikruu5mo6dtt3avvmwsejouqhxu4btdm/include /home/bsmith/petsc/src/ts/utils/dmplexlandau/kokkos/landau.kokkos.cxx -o arch-main/obj/ts/utils/dmplexlandau/kokkos/landau.o nvcc --expt-extended-lambda -gencode arch=compute_70,code=sm_70 -Wno-deprecated-gpu-targets -ccbin mpicxx -g -I/home/bsmith/soft/gnu-mpich/include -I/home/bsmith/petsc/include -I/home/bsmith/petsc/arch-main/include -I/nfs/apps/spacks/2021-02-09/opt/spack/linux-centos7-x86_64/gcc-7.3.0/cuda-11.2.0-ikruu5mo6dtt3avvmwsejouqhxu4btdm/include -Xcompiler -fPIC -x cu /home/bsmith/petsc/src/ts/utils/dmplexlandau/kokkos/landau.kokkos.cxx -c -o arch-main/obj/ts/utils/dmplexlandau/kokkos/landau.o ~/petsc* (main=)* arch-main
$ NVCC_WRAPPER_DEFAULT_COMPILER=mpicxx /home/bsmith/petsc/arch-main/bin/nvcc_wrapper --show --expt-extended-lambda -c -arch=sm_70 -Xcompiler -rdynamic -DLANDAU_DIM=2 -DLANDAU_MAX_SPECIES=10 -DPETSC_HAVE_CUDA_ATOMIC -DLANDAU_MAX_Q=4 -Xcompiler -fPIC -O3 -gencode arch=compute_80,code=sm_80 -I/usr/common/software/sles15_cgpu/openmpi/4.0.3/gcc/include -Wno-deprecated-gpu-targets nvcc --expt-extended-lambda -arch=sm_70 -Wno-deprecated-gpu-targets -ccbin mpicxx -DLANDAU_DIM=2 -DLANDAU_MAX_SPECIES=10 -DPETSC_HAVE_CUDA_ATOMIC -DLANDAU_MAX_Q=4 -O3 -I/usr/common/software/sles15_cgpu/openmpi/4.0.3/gcc/include -Xcompiler -rdynamic,-fPIC,arch=compute_80,code=sm_80 -c ~/petsc* (main=)* arch-main
$ NVCC_WRAPPER_DEFAULT_COMPILER=mpicxx /home/bsmith/petsc/arch-main/bin/nvcc_wrapper --show --expt-extended-lambda -c -Xcompiler -fPIC -g -arch=sm_70 -gencode arch=compute_70,code=sm_70 -I/home/bsmith/soft/gnu-mpich/include -Wno-deprecated-gpu-targets -I/home/bsmith/petsc/include -I/home/bsmith/petsc/arch-main/include -I/nfs/apps/spacks/2021-02-09/opt/spack/linux-centos7-x86_64/gcc-7.3.0/cuda-11.2.0-ikruu5mo6dtt3avvmwsejouqhxu4btdm/include /home/bsmith/petsc/src/ts/utils/dmplexlandau/kokkos/landau.kokkos.cxx -o arch-main/obj/ts/utils/dmplexlandau/kokkos/landau.o nvcc --expt-extended-lambda -arch=sm_70 -Wno-deprecated-gpu-targets -ccbin mpicxx -g -I/home/bsmith/soft/gnu-mpich/include -I/home/bsmith/petsc/include -I/home/bsmith/petsc/arch-main/include -I/nfs/apps/spacks/2021-02-09/opt/spack/linux-centos7-x86_64/gcc-7.3.0/cuda-11.2.0-ikruu5mo6dtt3avvmwsejouqhxu4btdm/include -Xcompiler -fPIC,arch=compute_70,code=sm_70 -x cu /home/bsmith/petsc/src/ts/utils/dmplexlandau/kokkos/landau.kokkos.cxx -c -o arch-main/obj/ts/utils/dmplexlandau/kokkos/landau.o
On May 15, 2021, at 8:44 PM, Junchao Zhang <[email protected]> wrote:
Add --show to the failing nvcc_wrapper command and see how nvcc_wrapper parses the options
PATH=/global/common/sw/cray/sles15/x86_64/zlib/1.2.11/gcc/8.2.0/pep2pal/bin:/global/common/sw/cray/cnl7/haswell/cmake/3.18.2/bin:/opt/cray/rca/2.2.20-7.0.1.1_4.62__g8e3fb5b.ari/bin:/opt/cray/alps/6.6.58-7.0.1.1_6.20__g437d88db.ari/sbin:/opt/cray/alps/default/bin:/opt/cray/job/2.2.4-7.0.1.1_3.48__g36b56f4.ari/bin:/opt/cray/pe/craype/2.6.2/bin:/usr/common/software/sles15_cgpu/openmpi/4.0.3/gcc/bin:/usr/common/software/sles15_cgpu/hpcsdk/20.11/Linux_x86_64/20.11/compilers/bin:/usr/common/software/sles15_cgpu/hpcsdk/20.11/Linux_x86_64/20.11/cuda/bin:/usr/common/software/sles15_cgpu/cuda/11.1.1/bin:/usr/common/software/sles15_cgpu/gcc/8.3.0/bin:/opt/esslurm/bin:/opt/cray/pe/modules/ 3.2.11.4/bin:/usr/common/software/bin:/usr/common/nsg/bin:/opt/ovis/bin:/opt/ovis/sbin:/usr/local/bin:/usr/bin:/bin:/usr/lib/mit/bin:/usr/lib/mit/sbin:/opt/cray/pe/bin:`dirname <http://3.2.11.4/bin:/usr/common/software/bin:/usr/common/nsg/bin:/opt/ovis/bin:/opt/ovis/sbin:/usr/local/bin:/usr/bin:/bin:/usr/lib/mit/bin:/usr/lib/mit/sbin:/opt/cray/pe/bin:dirname> /usr/common/software/sles15_cgpu/cuda/11.1.1/bin/nvcc` NVCC_WRAPPER_DEFAULT_COMPILER=/usr/common/software/sles15_cgpu/openmpi/4.0.3/gcc/bin/mpicxx /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/bin/nvcc_wrapper --expt-extended-lambda -c -arch=sm_70 -Xcompiler -rdynamic -DLANDAU_DIM=2 -DLANDAU_MAX_SPECIES=10 -DPETSC_HAVE_CUDA_ATOMIC -DLANDAU_MAX_Q=4 -Xcompiler -fPIC -O3 -gencode arch=compute_80,code=sm_80 -I/usr/common/software/sles15_cgpu/openmpi/4.0.3/gcc/include -Wno-deprecated-gpu-targets -I/global/homes/m/madams/petsc/include -I/global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/include -I/usr/common/software/sles15_cgpu/openmpi/4.0.3/gcc/include -I/usr/common/software/sles15_cgpu/cuda/11.1.1/include /global/u2/m/madams/petsc/src/sys/objects/kokkos/kinit.kokkos.cxx -o arch-cori-gpu-opt-kokkos-gcc/obj/sys/objects/kokkos/kinit.o --show
--Junchao Zhang
On Sat, May 15, 2021 at 7:14 PM Mark Adams <[email protected]> wrote:
Thanks, Now I get an error in make all. THis was working a few weeks ago. Make.log was empty but here is the output. .... CC arch-cori-gpu-opt-kokkos-gcc/obj/sys/error/ftn-custom/zerrf.o CC arch-cori-gpu-opt-kokkos-gcc/obj/sys/f90-src/f90_cwrap.o CC arch-cori-gpu-opt-kokkos-gcc/obj/vec/is/utils/hdf5io.o
PATH=/global/common/sw/cray/sles15/x86_64/zlib/1.2.11/gcc/8.2.0/pep2pal/bin:/global/common/sw/cray/cnl7/haswell/cmake/3.18.2/bin:/opt/cray/rca/2.2.20-7.0.1.1_4.62__g8e3fb5b.ari/bin:/opt/cray/alps/6.6.58-7.0.1.1_6.20__g437d88db.ari/sbin:/opt/cray/alps/default/bin:/opt/cray/job/2.2.4-7.0.1.1_3.48__g36b56f4.ari/bin:/opt/cray/pe/craype/2.6.2/bin:/usr/common/software/sles15_cgpu/openmpi/4.0.3/gcc/bin:/usr/common/software/sles15_cgpu/hpcsdk/20.11/Linux_x86_64/20.11/compilers/bin:/usr/common/software/sles15_cgpu/hpcsdk/20.11/Linux_x86_64/20.11/cuda/bin:/usr/common/software/sles15_cgpu/cuda/11.1.1/bin:/usr/common/software/sles15_cgpu/gcc/8.3.0/bin:/opt/esslurm/bin:/opt/cray/pe/modules/ 3.2.11.4/bin:/usr/common/software/bin:/usr/common/nsg/bin:/opt/ovis/bin:/opt/ovis/sbin:/usr/local/bin:/usr/bin:/bin:/usr/lib/mit/bin:/usr/lib/mit/sbin:/opt/cray/pe/bin:`dirname <http://3.2.11.4/bin:/usr/common/software/bin:/usr/common/nsg/bin:/opt/ovis/bin:/opt/ovis/sbin:/usr/local/bin:/usr/bin:/bin:/usr/lib/mit/bin:/usr/lib/mit/sbin:/opt/cray/pe/bin:dirname> /usr/common/software/sles15_cgpu/cuda/11.1.1/bin/nvcc` NVCC_WRAPPER_DEFAULT_COMPILER=/usr/common/software/sles15_cgpu/openmpi/4.0.3/gcc/bin/mpicxx /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/bin/nvcc_wrapper --expt-extended-lambda -c -arch=sm_70 -Xcompiler -rdynamic -DLANDAU_DIM=2 -DLANDAU_MAX_SPECIES=10 -DPETSC_HAVE_CUDA_ATOMIC -DLANDAU_MAX_Q=4 -Xcompiler -fPIC -O3 -gencode arch=compute_80,code=sm_80 -I/usr/common/software/sles15_cgpu/openmpi/4.0.3/gcc/include -Wno-deprecated-gpu-targets -I/global/homes/m/madams/petsc/include -I/global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/include -I/usr/common/software/sles15_cgpu/openmpi/4.0.3/gcc/include -I/usr/common/software/sles15_cgpu/cuda/11.1.1/include /global/u2/m/madams/petsc/src/sys/objects/kokkos/kinit.kokkos.cxx -o arch-cori-gpu-opt-kokkos-gcc/obj/sys/objects/kokkos/kinit.o g++: error: arch=compute_80: No such file or directory g++: error: code=sm_80: No such file or directory gmake[3]: *** [gmakefile:188: arch-cori-gpu-opt-kokkos-gcc/obj/sys/objects/kokkos/kinit.o] Error 1 gmake[3]: *** Waiting for unfinished jobs.... CC arch-cori-gpu-opt-kokkos-gcc/obj/vec/is/utils/isio.o CC arch-cori-gpu-opt-kokkos-gcc/obj/vec/is/utils/ftn-auto/pmapf.o CC arch-cori-gpu-opt-kokkos-gcc/obj/vec/is/utils/pmap.o CC arch-cori-gpu-opt-kokkos-gcc/obj/vec/is/utils/psort.o CC arch-cori-gpu-opt-kokkos-gcc/obj/vec/is/utils/isltog.o FC arch-cori-gpu-opt-kokkos-gcc/obj/vec/f90-mod/petscvecmod.o CUDAC arch-cori-gpu-opt-kokkos-gcc/obj/sys/memory/cuda/mcudahost.o CUDAC.dep arch-cori-gpu-opt-kokkos-gcc/obj/sys/memory/cuda/mcudahost.o gmake[2]: *** [/global/homes/m/madams/petsc/lib/petsc/conf/rules:50: libs] Error 2 **************************ERROR************************************* Error during compile, check arch-cori-gpu-opt-kokkos-gcc/lib/petsc/conf/make.log Send it and arch-cori-gpu-opt-kokkos-gcc/lib/petsc/conf/configure.log to [email protected] ******************************************************************** gmake[1]: *** [makefile:40: all] Error 1 make: *** [GNUmakefile:9: all] Error 2
On Sat, May 15, 2021 at 5:30 PM Barry Smith <[email protected]> wrote:
Bitbucket is off-line. Do you need fortran stubs at this moment? If not use --with-fortran-interface=0
On May 15, 2021, at 2:51 PM, Mark Adams <[email protected]> wrote:
I can build a non-kokkos PETSc here (Cori/GPU) but I get this error with Kokkos. Any suggestions? Thanks, Mark
=============================================================================================
=============================================================================================
Trying to download git://https://bitbucket.org/petsc/pkg-sowing.git for SOWING
=============================================================================================
=============================================================================================
Trying to download git://https://bitbucket.org/petsc/pkg-sowing.git for SOWING
=============================================================================================
=============================================================================================
Trying to download https://bitbucket.org/petsc/pkg-sowing/get/v1.1.26-p1.tar.gz for SOWING
=============================================================================================
******************************************************************************* UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log for details):
------------------------------------------------------------------------------- Error during download/extract/detection of SOWING: Unable to clone sowing Could not execute "['git clone https://bitbucket.org/petsc/pkg-sowing.git /global/u2/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/externalpackages/git.sowing']": Cloning into '/global/u2/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/externalpackages/git.sowing'... fatal: unable to access 'https://bitbucket.org/petsc/pkg-sowing.git/': The requested URL returned error: 503 Unable to download package SOWING from: git:// https://bitbucket.org/petsc/pkg-sowing.git * If URL specified manually - perhaps there is a typo? * If your network is disconnected - please reconnect and rerun ./configure * Or perhaps you have a firewall blocking the download * You can run with --with-packages-download-dir=/adirectory and ./configure will instruct you what packages to download manually * or you can download the above URL manually, to /yourselectedlocation and use the configure option: --download-sowing=/yourselectedlocation Unable to clone sowing Could not execute "['git clone https://bitbucket.org/petsc/pkg-sowing.git /global/u2/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/externalpackages/git.sowing']": Cloning into '/global/u2/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/externalpackages/git.sowing'... fatal: unable to access 'https://bitbucket.org/petsc/pkg-sowing.git/': The requested URL returned error: 503 Unable to download package SOWING from: git:// https://bitbucket.org/petsc/pkg-sowing.git * If URL specified manually - perhaps there is a typo? * If your network is disconnected - please reconnect and rerun ./configure * Or perhaps you have a firewall blocking the download * You can run with --with-packages-download-dir=/adirectory and ./configure will instruct you what packages to download manually * or you can download the above URL manually, to /yourselectedlocation and use the configure option: --download-sowing=/yourselectedlocation file could not be opened successfully Downloaded package SOWING from: https://bitbucket.org/petsc/pkg-sowing/get/v1.1.26-p1.tar.gz is not a tarball. [or installed python cannot process compressed files] * If you are behind a firewall - please fix your proxy and rerun ./configure For example at LANL you may need to set the environmental variable http_proxy (or HTTP_PROXY?) to http://proxyout.lanl.gov * You can run with --with-packages-download-dir=/adirectory and ./configure will instruct you what packages to download manually * or you can download the above URL manually, to /yourselectedlocation/v1.1.26-p1.tar.gz and use the configure option: --download-sowing=/yourselectedlocation/v1.1.26-p1.tar.gz
******************************************************************************* <configure.log>
Check the meaning of 'srun: error: Unable to create step for job 1921749: More processors requested than permitted' --Junchao Zhang On Sun, May 16, 2021 at 7:58 AM Mark Adams <[email protected]> wrote:
Whoops, between scp'ing and soft links I gave you the wrong log file. I removed this already.
On Sun, May 16, 2021 at 8:37 AM Junchao Zhang <[email protected]> wrote:
Remove --with-kokkos-cuda-arch=VOLTA70 from your configure
Barry: It seems we should either disable this option, or rename it to --with-cuda-arch and get it accepted by both kokkos.py and cuda.py
--Junchao Zhang
On Sun, May 16, 2021 at 6:15 AM Mark Adams <[email protected]> wrote:
progress:
04:08 cgpu11 adams/cusparse-cpu-solve *= ~/petsc$ make PETSC_DIR=/global/homes/m/madams/petsc PETSC_ARCH=arch-cori-gpu-opt-kokkos-gcc check Running check examples to verify correct installation Using PETSC_DIR=/global/homes/m/madams/petsc and PETSC_ARCH=arch-cori-gpu-opt-kokkos-gcc C/C++ example src/snes/tutorials/ex19 run successfully with 1 MPI process Possible error running C/C++ src/snes/tutorials/ex19 with 2 MPI processes See http://www.mcs.anl.gov/petsc/documentation/faq.html srun: error: Unable to create step for job 1921749: More processors requested than permitted 2,15c2,32 < 0 SNES Function norm 2.391552133017e-01 < 0 KSP Residual norm 2.325621076120e-01 < 1 KSP Residual norm 1.654206318674e-02 < 2 KSP Residual norm 7.202836119880e-04 < 3 KSP Residual norm 1.796861424199e-05 < 4 KSP Residual norm 2.461332992052e-07 < 1 SNES Function norm 6.826585648929e-05 < 0 KSP Residual norm 2.347339172985e-05 < 1 KSP Residual norm 8.356798075993e-07 < 2 KSP Residual norm 1.844045309619e-08 < 3 KSP Residual norm 5.336386977405e-10 < 4 KSP Residual norm 2.662608472862e-11 < 2 SNES Function norm 6.549682264799e-11 < Number of SNES iterations = 2 ---
Kokkos::Cuda::initialize ERROR: likely mismatch of architecture [cgpu11:39927] *** Process received signal *** [cgpu11:39927] Signal: Aborted (6) [cgpu11:39927] Signal code: (-6) [cgpu11:39927] [ 0] /lib64/libpthread.so.0(+0x12360)[0x2aab12731360] [cgpu11:39927] [ 1] /lib64/libc.so.6(gsignal+0x110)[0x2aab12973160] [cgpu11:39927] [ 2] /lib64/libc.so.6(abort+0x151)[0x2aab12974741] [cgpu11:39927] [ 3] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libkokkoscore.so.3.4(+0x408e5)[0x2aaabbfb88e5] [cgpu11:39927] [ 4] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libkokkoscore.so.3.4(_ZN6Kokkos4Impl12CudaInternal10initializeEiP11CUstream_st+0xb5c)[0x2aaabbfc762c] [cgpu11:39927] [ 5] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libkokkoscore.so.3.4(_ZN6Kokkos4Impl16ExecSpaceManager17initialize_spacesERKNS_13InitArgumentsE+0x24)[0x2aaabbfa1224] [cgpu11:39927] [ 6] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libkokkoscore.so.3.4(_ZN6Kokkos4Impl70_GLOBAL__N__46_tmpxft_00011209_00000000_7_Kokkos_Core_cpp1_ii_889c95a619initialize_internalERKNS_13InitArgumentsE+0x11)[0x2aaabbfa49d1] [cgpu11:39927] [ 7] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libpetsc.so.3.015(PetscKokkosInitializeCheck+0x1f9)[0x2aaaaafad809] [cgpu11:39927] [ 8] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libpetsc.so.3.015(+0x68541b)[0x2aaaab35841b] [cgpu11:39927] [ 9] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libpetsc.so.3.015(+0x36a224)[0x2aaaab03d224] [cgpu11:39927] [10] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libpetsc.so.3.015(+0x35f596)[0x2aaaab032596] [cgpu11:39927] [11] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libpetsc.so.3.015(+0x2f9d94)[0x2aaaaafccd94] [cgpu11:39927] [12] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libpetsc.so.3.015(PetscSFBcastWithMemTypeBegin+0x11f)[0x2aaaab04bc3f] [cgpu11:39927] [13] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libpetsc.so.3.015(+0x37f53a)[0x2aaaab05253a] [cgpu11:39927] [14] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libpetsc.so.3.015(VecScatterBegin+0x70)[0x2aaaab057b30] [cgpu11:39927] [15] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libpetsc.so.3.015(DMGlobalToLocalBegin_DA+0x1f)[0x2aaaabc27a5f] [cgpu11:39927] [16] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libpetsc.so.3.015(DMGlobalToLocalBegin+0x1dd)[0x2aaaabe61b3d] [cgpu11:39927] [17] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libpetsc.so.3.015(+0x144567a)[0x2aaaac11867a] [cgpu11:39927] [18] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libpetsc.so.3.015(SNESComputeFunction+0xf5)[0x2aaaac138ef5] [cgpu11:39927] [19] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libpetsc.so.3.015(+0x14ad0de)[0x2aaaac1800de] [cgpu11:39927] [20] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libpetsc.so.3.015(SNESSolve+0x821)[0x2aaaac146ed1] [cgpu11:39927] [21] /global/u2/m/madams/petsc/src/snes/tutorials/./ex19[0x401a69] [cgpu11:39927] [22] /lib64/libc.so.6(__libc_start_main+0xea)[0x2aab1295df8a] [cgpu11:39927] [23] /global/u2/m/madams/petsc/src/snes/tutorials/./ex19[0x4021ba] [cgpu11:39927] *** End of error message *** srun: error: cgpu11: task 0: Aborted srun: Terminating job step 1921749.6 /global/homes/m/madams/petsc/src/snes/tutorials Possible problem with ex19 running with cuda, diffs above ========================================= gmake[3]: [makefile:102: runex3k_kokkos] Error 1 (ignored)
On Sun, May 16, 2021 at 12:31 AM Barry Smith <[email protected]> wrote:
Looks like nvcc_wrapper gets confused when both -arch and -gencode are used. But handles either of them separately correctly.
Mark,
PETSc configure now handles the -arch and --with-kokkos-cuda-arch= business automatically so you do not, and generally shouldn't pass -arch sm_70 in --CUDAFLAGS anymore or use the --with-kokkos-cuda-arch option anymore.
If you need to set the -arch to something earlier than the system supports (rare) you can now use -with-cuda-gencodearch to set that instead of using CUDAFLAGS.
Barry
$ NVCC_WRAPPER_DEFAULT_COMPILER=mpicxx /home/bsmith/petsc/arch-main/bin/nvcc_wrapper --show --expt-extended-lambda -c -Xcompiler -fPIC -g -gencode arch=compute_70,code=sm_70 -I/home/bsmith/soft/gnu-mpich/include -Wno-deprecated-gpu-targets -I/home/bsmith/petsc/include -I/home/bsmith/petsc/arch-main/include -I/nfs/apps/spacks/2021-02-09/opt/spack/linux-centos7-x86_64/gcc-7.3.0/cuda-11.2.0-ikruu5mo6dtt3avvmwsejouqhxu4btdm/include /home/bsmith/petsc/src/ts/utils/dmplexlandau/kokkos/landau.kokkos.cxx -o arch-main/obj/ts/utils/dmplexlandau/kokkos/landau.o nvcc --expt-extended-lambda -gencode arch=compute_70,code=sm_70 -Wno-deprecated-gpu-targets -ccbin mpicxx -g -I/home/bsmith/soft/gnu-mpich/include -I/home/bsmith/petsc/include -I/home/bsmith/petsc/arch-main/include -I/nfs/apps/spacks/2021-02-09/opt/spack/linux-centos7-x86_64/gcc-7.3.0/cuda-11.2.0-ikruu5mo6dtt3avvmwsejouqhxu4btdm/include -Xcompiler -fPIC -x cu /home/bsmith/petsc/src/ts/utils/dmplexlandau/kokkos/landau.kokkos.cxx -c -o arch-main/obj/ts/utils/dmplexlandau/kokkos/landau.o ~/petsc* (main=)* arch-main
$ NVCC_WRAPPER_DEFAULT_COMPILER=mpicxx /home/bsmith/petsc/arch-main/bin/nvcc_wrapper --show --expt-extended-lambda -c -arch=sm_70 -Xcompiler -rdynamic -DLANDAU_DIM=2 -DLANDAU_MAX_SPECIES=10 -DPETSC_HAVE_CUDA_ATOMIC -DLANDAU_MAX_Q=4 -Xcompiler -fPIC -O3 -gencode arch=compute_80,code=sm_80 -I/usr/common/software/sles15_cgpu/openmpi/4.0.3/gcc/include -Wno-deprecated-gpu-targets nvcc --expt-extended-lambda -arch=sm_70 -Wno-deprecated-gpu-targets -ccbin mpicxx -DLANDAU_DIM=2 -DLANDAU_MAX_SPECIES=10 -DPETSC_HAVE_CUDA_ATOMIC -DLANDAU_MAX_Q=4 -O3 -I/usr/common/software/sles15_cgpu/openmpi/4.0.3/gcc/include -Xcompiler -rdynamic,-fPIC,arch=compute_80,code=sm_80 -c ~/petsc* (main=)* arch-main
$ NVCC_WRAPPER_DEFAULT_COMPILER=mpicxx /home/bsmith/petsc/arch-main/bin/nvcc_wrapper --show --expt-extended-lambda -c -Xcompiler -fPIC -g -arch=sm_70 -gencode arch=compute_70,code=sm_70 -I/home/bsmith/soft/gnu-mpich/include -Wno-deprecated-gpu-targets -I/home/bsmith/petsc/include -I/home/bsmith/petsc/arch-main/include -I/nfs/apps/spacks/2021-02-09/opt/spack/linux-centos7-x86_64/gcc-7.3.0/cuda-11.2.0-ikruu5mo6dtt3avvmwsejouqhxu4btdm/include /home/bsmith/petsc/src/ts/utils/dmplexlandau/kokkos/landau.kokkos.cxx -o arch-main/obj/ts/utils/dmplexlandau/kokkos/landau.o nvcc --expt-extended-lambda -arch=sm_70 -Wno-deprecated-gpu-targets -ccbin mpicxx -g -I/home/bsmith/soft/gnu-mpich/include -I/home/bsmith/petsc/include -I/home/bsmith/petsc/arch-main/include -I/nfs/apps/spacks/2021-02-09/opt/spack/linux-centos7-x86_64/gcc-7.3.0/cuda-11.2.0-ikruu5mo6dtt3avvmwsejouqhxu4btdm/include -Xcompiler -fPIC,arch=compute_70,code=sm_70 -x cu /home/bsmith/petsc/src/ts/utils/dmplexlandau/kokkos/landau.kokkos.cxx -c -o arch-main/obj/ts/utils/dmplexlandau/kokkos/landau.o
On May 15, 2021, at 8:44 PM, Junchao Zhang <[email protected]> wrote:
Add --show to the failing nvcc_wrapper command and see how nvcc_wrapper parses the options
PATH=/global/common/sw/cray/sles15/x86_64/zlib/1.2.11/gcc/8.2.0/pep2pal/bin:/global/common/sw/cray/cnl7/haswell/cmake/3.18.2/bin:/opt/cray/rca/2.2.20-7.0.1.1_4.62__g8e3fb5b.ari/bin:/opt/cray/alps/6.6.58-7.0.1.1_6.20__g437d88db.ari/sbin:/opt/cray/alps/default/bin:/opt/cray/job/2.2.4-7.0.1.1_3.48__g36b56f4.ari/bin:/opt/cray/pe/craype/2.6.2/bin:/usr/common/software/sles15_cgpu/openmpi/4.0.3/gcc/bin:/usr/common/software/sles15_cgpu/hpcsdk/20.11/Linux_x86_64/20.11/compilers/bin:/usr/common/software/sles15_cgpu/hpcsdk/20.11/Linux_x86_64/20.11/cuda/bin:/usr/common/software/sles15_cgpu/cuda/11.1.1/bin:/usr/common/software/sles15_cgpu/gcc/8.3.0/bin:/opt/esslurm/bin:/opt/cray/pe/modules/ 3.2.11.4/bin:/usr/common/software/bin:/usr/common/nsg/bin:/opt/ovis/bin:/opt/ovis/sbin:/usr/local/bin:/usr/bin:/bin:/usr/lib/mit/bin:/usr/lib/mit/sbin:/opt/cray/pe/bin:`dirname <http://3.2.11.4/bin:/usr/common/software/bin:/usr/common/nsg/bin:/opt/ovis/bin:/opt/ovis/sbin:/usr/local/bin:/usr/bin:/bin:/usr/lib/mit/bin:/usr/lib/mit/sbin:/opt/cray/pe/bin:dirname> /usr/common/software/sles15_cgpu/cuda/11.1.1/bin/nvcc` NVCC_WRAPPER_DEFAULT_COMPILER=/usr/common/software/sles15_cgpu/openmpi/4.0.3/gcc/bin/mpicxx /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/bin/nvcc_wrapper --expt-extended-lambda -c -arch=sm_70 -Xcompiler -rdynamic -DLANDAU_DIM=2 -DLANDAU_MAX_SPECIES=10 -DPETSC_HAVE_CUDA_ATOMIC -DLANDAU_MAX_Q=4 -Xcompiler -fPIC -O3 -gencode arch=compute_80,code=sm_80 -I/usr/common/software/sles15_cgpu/openmpi/4.0.3/gcc/include -Wno-deprecated-gpu-targets -I/global/homes/m/madams/petsc/include -I/global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/include -I/usr/common/software/sles15_cgpu/openmpi/4.0.3/gcc/include -I/usr/common/software/sles15_cgpu/cuda/11.1.1/include /global/u2/m/madams/petsc/src/sys/objects/kokkos/kinit.kokkos.cxx -o arch-cori-gpu-opt-kokkos-gcc/obj/sys/objects/kokkos/kinit.o --show
--Junchao Zhang
On Sat, May 15, 2021 at 7:14 PM Mark Adams <[email protected]> wrote:
Thanks, Now I get an error in make all. THis was working a few weeks ago. Make.log was empty but here is the output. .... CC arch-cori-gpu-opt-kokkos-gcc/obj/sys/error/ftn-custom/zerrf.o CC arch-cori-gpu-opt-kokkos-gcc/obj/sys/f90-src/f90_cwrap.o CC arch-cori-gpu-opt-kokkos-gcc/obj/vec/is/utils/hdf5io.o
PATH=/global/common/sw/cray/sles15/x86_64/zlib/1.2.11/gcc/8.2.0/pep2pal/bin:/global/common/sw/cray/cnl7/haswell/cmake/3.18.2/bin:/opt/cray/rca/2.2.20-7.0.1.1_4.62__g8e3fb5b.ari/bin:/opt/cray/alps/6.6.58-7.0.1.1_6.20__g437d88db.ari/sbin:/opt/cray/alps/default/bin:/opt/cray/job/2.2.4-7.0.1.1_3.48__g36b56f4.ari/bin:/opt/cray/pe/craype/2.6.2/bin:/usr/common/software/sles15_cgpu/openmpi/4.0.3/gcc/bin:/usr/common/software/sles15_cgpu/hpcsdk/20.11/Linux_x86_64/20.11/compilers/bin:/usr/common/software/sles15_cgpu/hpcsdk/20.11/Linux_x86_64/20.11/cuda/bin:/usr/common/software/sles15_cgpu/cuda/11.1.1/bin:/usr/common/software/sles15_cgpu/gcc/8.3.0/bin:/opt/esslurm/bin:/opt/cray/pe/modules/ 3.2.11.4/bin:/usr/common/software/bin:/usr/common/nsg/bin:/opt/ovis/bin:/opt/ovis/sbin:/usr/local/bin:/usr/bin:/bin:/usr/lib/mit/bin:/usr/lib/mit/sbin:/opt/cray/pe/bin:`dirname <http://3.2.11.4/bin:/usr/common/software/bin:/usr/common/nsg/bin:/opt/ovis/bin:/opt/ovis/sbin:/usr/local/bin:/usr/bin:/bin:/usr/lib/mit/bin:/usr/lib/mit/sbin:/opt/cray/pe/bin:dirname> /usr/common/software/sles15_cgpu/cuda/11.1.1/bin/nvcc` NVCC_WRAPPER_DEFAULT_COMPILER=/usr/common/software/sles15_cgpu/openmpi/4.0.3/gcc/bin/mpicxx /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/bin/nvcc_wrapper --expt-extended-lambda -c -arch=sm_70 -Xcompiler -rdynamic -DLANDAU_DIM=2 -DLANDAU_MAX_SPECIES=10 -DPETSC_HAVE_CUDA_ATOMIC -DLANDAU_MAX_Q=4 -Xcompiler -fPIC -O3 -gencode arch=compute_80,code=sm_80 -I/usr/common/software/sles15_cgpu/openmpi/4.0.3/gcc/include -Wno-deprecated-gpu-targets -I/global/homes/m/madams/petsc/include -I/global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/include -I/usr/common/software/sles15_cgpu/openmpi/4.0.3/gcc/include -I/usr/common/software/sles15_cgpu/cuda/11.1.1/include /global/u2/m/madams/petsc/src/sys/objects/kokkos/kinit.kokkos.cxx -o arch-cori-gpu-opt-kokkos-gcc/obj/sys/objects/kokkos/kinit.o g++: error: arch=compute_80: No such file or directory g++: error: code=sm_80: No such file or directory gmake[3]: *** [gmakefile:188: arch-cori-gpu-opt-kokkos-gcc/obj/sys/objects/kokkos/kinit.o] Error 1 gmake[3]: *** Waiting for unfinished jobs.... CC arch-cori-gpu-opt-kokkos-gcc/obj/vec/is/utils/isio.o CC arch-cori-gpu-opt-kokkos-gcc/obj/vec/is/utils/ftn-auto/pmapf.o CC arch-cori-gpu-opt-kokkos-gcc/obj/vec/is/utils/pmap.o CC arch-cori-gpu-opt-kokkos-gcc/obj/vec/is/utils/psort.o CC arch-cori-gpu-opt-kokkos-gcc/obj/vec/is/utils/isltog.o FC arch-cori-gpu-opt-kokkos-gcc/obj/vec/f90-mod/petscvecmod.o CUDAC arch-cori-gpu-opt-kokkos-gcc/obj/sys/memory/cuda/mcudahost.o CUDAC.dep arch-cori-gpu-opt-kokkos-gcc/obj/sys/memory/cuda/mcudahost.o gmake[2]: *** [/global/homes/m/madams/petsc/lib/petsc/conf/rules:50: libs] Error 2 **************************ERROR************************************* Error during compile, check arch-cori-gpu-opt-kokkos-gcc/lib/petsc/conf/make.log Send it and arch-cori-gpu-opt-kokkos-gcc/lib/petsc/conf/configure.log to [email protected] ******************************************************************** gmake[1]: *** [makefile:40: all] Error 1 make: *** [GNUmakefile:9: all] Error 2
On Sat, May 15, 2021 at 5:30 PM Barry Smith <[email protected]> wrote:
Bitbucket is off-line. Do you need fortran stubs at this moment? If not use --with-fortran-interface=0
On May 15, 2021, at 2:51 PM, Mark Adams <[email protected]> wrote:
I can build a non-kokkos PETSc here (Cori/GPU) but I get this error with Kokkos. Any suggestions? Thanks, Mark
=============================================================================================
=============================================================================================
Trying to download git://https://bitbucket.org/petsc/pkg-sowing.git for SOWING
=============================================================================================
=============================================================================================
Trying to download git://https://bitbucket.org/petsc/pkg-sowing.git for SOWING
=============================================================================================
=============================================================================================
Trying to download https://bitbucket.org/petsc/pkg-sowing/get/v1.1.26-p1.tar.gz for SOWING
=============================================================================================
******************************************************************************* UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log for details):
------------------------------------------------------------------------------- Error during download/extract/detection of SOWING: Unable to clone sowing Could not execute "['git clone https://bitbucket.org/petsc/pkg-sowing.git /global/u2/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/externalpackages/git.sowing']": Cloning into '/global/u2/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/externalpackages/git.sowing'... fatal: unable to access 'https://bitbucket.org/petsc/pkg-sowing.git/': The requested URL returned error: 503 Unable to download package SOWING from: git:// https://bitbucket.org/petsc/pkg-sowing.git * If URL specified manually - perhaps there is a typo? * If your network is disconnected - please reconnect and rerun ./configure * Or perhaps you have a firewall blocking the download * You can run with --with-packages-download-dir=/adirectory and ./configure will instruct you what packages to download manually * or you can download the above URL manually, to /yourselectedlocation and use the configure option: --download-sowing=/yourselectedlocation Unable to clone sowing Could not execute "['git clone https://bitbucket.org/petsc/pkg-sowing.git /global/u2/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/externalpackages/git.sowing']": Cloning into '/global/u2/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/externalpackages/git.sowing'... fatal: unable to access 'https://bitbucket.org/petsc/pkg-sowing.git/': The requested URL returned error: 503 Unable to download package SOWING from: git:// https://bitbucket.org/petsc/pkg-sowing.git * If URL specified manually - perhaps there is a typo? * If your network is disconnected - please reconnect and rerun ./configure * Or perhaps you have a firewall blocking the download * You can run with --with-packages-download-dir=/adirectory and ./configure will instruct you what packages to download manually * or you can download the above URL manually, to /yourselectedlocation and use the configure option: --download-sowing=/yourselectedlocation file could not be opened successfully Downloaded package SOWING from: https://bitbucket.org/petsc/pkg-sowing/get/v1.1.26-p1.tar.gz is not a tarball. [or installed python cannot process compressed files] * If you are behind a firewall - please fix your proxy and rerun ./configure For example at LANL you may need to set the environmental variable http_proxy (or HTTP_PROXY?) to http://proxyout.lanl.gov * You can run with --with-packages-download-dir=/adirectory and ./configure will instruct you what packages to download manually * or you can download the above URL manually, to /yourselectedlocation/v1.1.26-p1.tar.gz and use the configure option: --download-sowing=/yourselectedlocation/v1.1.26-p1.tar.gz
******************************************************************************* <configure.log>
On May 16, 2021, at 7:37 AM, Junchao Zhang <[email protected]> wrote:
Remove --with-kokkos-cuda-arch=VOLTA70 from your configure
Yes, I think it should be removed.
Barry: It seems we should either disable this option, or rename it to --with-cuda-arch and get it accepted by both kokkos.py and cuda.py
--Junchao Zhang
On Sun, May 16, 2021 at 6:15 AM Mark Adams <[email protected] <mailto:[email protected]>> wrote: progress:
04:08 cgpu11 adams/cusparse-cpu-solve *= ~/petsc$ make PETSC_DIR=/global/homes/m/madams/petsc PETSC_ARCH=arch-cori-gpu-opt-kokkos-gcc check Running check examples to verify correct installation Using PETSC_DIR=/global/homes/m/madams/petsc and PETSC_ARCH=arch-cori-gpu-opt-kokkos-gcc C/C++ example src/snes/tutorials/ex19 run successfully with 1 MPI process Possible error running C/C++ src/snes/tutorials/ex19 with 2 MPI processes See http://www.mcs.anl.gov/petsc/documentation/faq.html <http://www.mcs.anl.gov/petsc/documentation/faq.html> srun: error: Unable to create step for job 1921749: More processors requested than permitted 2,15c2,32 < 0 SNES Function norm 2.391552133017e-01 < 0 KSP Residual norm 2.325621076120e-01 < 1 KSP Residual norm 1.654206318674e-02 < 2 KSP Residual norm 7.202836119880e-04 < 3 KSP Residual norm 1.796861424199e-05 < 4 KSP Residual norm 2.461332992052e-07 < 1 SNES Function norm 6.826585648929e-05 < 0 KSP Residual norm 2.347339172985e-05 < 1 KSP Residual norm 8.356798075993e-07 < 2 KSP Residual norm 1.844045309619e-08 < 3 KSP Residual norm 5.336386977405e-10 < 4 KSP Residual norm 2.662608472862e-11 < 2 SNES Function norm 6.549682264799e-11 < Number of SNES iterations = 2 ---
Kokkos::Cuda::initialize ERROR: likely mismatch of architecture [cgpu11:39927] *** Process received signal *** [cgpu11:39927] Signal: Aborted (6) [cgpu11:39927] Signal code: (-6) [cgpu11:39927] [ 0] /lib64/libpthread.so.0(+0x12360)[0x2aab12731360] [cgpu11:39927] [ 1] /lib64/libc.so.6(gsignal+0x110)[0x2aab12973160] [cgpu11:39927] [ 2] /lib64/libc.so.6(abort+0x151)[0x2aab12974741] [cgpu11:39927] [ 3] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libkokkoscore.so.3.4(+0x408e5)[0x2aaabbfb88e5] [cgpu11:39927] [ 4] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libkokkoscore.so.3.4(_ZN6Kokkos4Impl12CudaInternal10initializeEiP11CUstream_st+0xb5c)[0x2aaabbfc762c] [cgpu11:39927] [ 5] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libkokkoscore.so.3.4(_ZN6Kokkos4Impl16ExecSpaceManager17initialize_spacesERKNS_13InitArgumentsE+0x24)[0x2aaabbfa1224] [cgpu11:39927] [ 6] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libkokkoscore.so.3.4(_ZN6Kokkos4Impl70_GLOBAL__N__46_tmpxft_00011209_00000000_7_Kokkos_Core_cpp1_ii_889c95a619initialize_internalERKNS_13InitArgumentsE+0x11)[0x2aaabbfa49d1] [cgpu11:39927] [ 7] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libpetsc.so.3.015(PetscKokkosInitializeCheck+0x1f9)[0x2aaaaafad809] [cgpu11:39927] [ 8] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libpetsc.so.3.015(+0x68541b)[0x2aaaab35841b] [cgpu11:39927] [ 9] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libpetsc.so.3.015(+0x36a224)[0x2aaaab03d224] [cgpu11:39927] [10] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libpetsc.so.3.015(+0x35f596)[0x2aaaab032596] [cgpu11:39927] [11] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libpetsc.so.3.015(+0x2f9d94)[0x2aaaaafccd94] [cgpu11:39927] [12] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libpetsc.so.3.015(PetscSFBcastWithMemTypeBegin+0x11f)[0x2aaaab04bc3f] [cgpu11:39927] [13] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libpetsc.so.3.015(+0x37f53a)[0x2aaaab05253a] [cgpu11:39927] [14] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libpetsc.so.3.015(VecScatterBegin+0x70)[0x2aaaab057b30] [cgpu11:39927] [15] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libpetsc.so.3.015(DMGlobalToLocalBegin_DA+0x1f)[0x2aaaabc27a5f] [cgpu11:39927] [16] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libpetsc.so.3.015(DMGlobalToLocalBegin+0x1dd)[0x2aaaabe61b3d] [cgpu11:39927] [17] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libpetsc.so.3.015(+0x144567a)[0x2aaaac11867a] [cgpu11:39927] [18] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libpetsc.so.3.015(SNESComputeFunction+0xf5)[0x2aaaac138ef5] [cgpu11:39927] [19] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libpetsc.so.3.015(+0x14ad0de)[0x2aaaac1800de] [cgpu11:39927] [20] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libpetsc.so.3.015(SNESSolve+0x821)[0x2aaaac146ed1] [cgpu11:39927] [21] /global/u2/m/madams/petsc/src/snes/tutorials/./ex19[0x401a69] [cgpu11:39927] [22] /lib64/libc.so.6(__libc_start_main+0xea)[0x2aab1295df8a] [cgpu11:39927] [23] /global/u2/m/madams/petsc/src/snes/tutorials/./ex19[0x4021ba] [cgpu11:39927] *** End of error message *** srun: error: cgpu11: task 0: Aborted srun: Terminating job step 1921749.6 /global/homes/m/madams/petsc/src/snes/tutorials Possible problem with ex19 running with cuda, diffs above ========================================= gmake[3]: [makefile:102: runex3k_kokkos] Error 1 (ignored)
On Sun, May 16, 2021 at 12:31 AM Barry Smith <[email protected] <mailto:[email protected]>> wrote:
Looks like nvcc_wrapper gets confused when both -arch and -gencode are used. But handles either of them separately correctly.
Mark,
PETSc configure now handles the -arch and --with-kokkos-cuda-arch= business automatically so you do not, and generally shouldn't pass -arch sm_70 in --CUDAFLAGS anymore or use the --with-kokkos-cuda-arch option anymore.
If you need to set the -arch to something earlier than the system supports (rare) you can now use -with-cuda-gencodearch to set that instead of using CUDAFLAGS.
Barry
$ NVCC_WRAPPER_DEFAULT_COMPILER=mpicxx /home/bsmith/petsc/arch-main/bin/nvcc_wrapper --show --expt-extended-lambda -c -Xcompiler -fPIC -g -gencode arch=compute_70,code=sm_70 -I/home/bsmith/soft/gnu-mpich/include -Wno-deprecated-gpu-targets -I/home/bsmith/petsc/include -I/home/bsmith/petsc/arch-main/include -I/nfs/apps/spacks/2021-02-09/opt/spack/linux-centos7-x86_64/gcc-7.3.0/cuda-11.2.0-ikruu5mo6dtt3avvmwsejouqhxu4btdm/include /home/bsmith/petsc/src/ts/utils/dmplexlandau/kokkos/landau.kokkos.cxx -o arch-main/obj/ts/utils/dmplexlandau/kokkos/landau.o nvcc --expt-extended-lambda -gencode arch=compute_70,code=sm_70 -Wno-deprecated-gpu-targets -ccbin mpicxx -g -I/home/bsmith/soft/gnu-mpich/include -I/home/bsmith/petsc/include -I/home/bsmith/petsc/arch-main/include -I/nfs/apps/spacks/2021-02-09/opt/spack/linux-centos7-x86_64/gcc-7.3.0/cuda-11.2.0-ikruu5mo6dtt3avvmwsejouqhxu4btdm/include -Xcompiler -fPIC -x cu /home/bsmith/petsc/src/ts/utils/dmplexlandau/kokkos/landau.kokkos.cxx -c -o arch-main/obj/ts/utils/dmplexlandau/kokkos/landau.o ~/petsc (main=) arch-main
$ NVCC_WRAPPER_DEFAULT_COMPILER=mpicxx /home/bsmith/petsc/arch-main/bin/nvcc_wrapper --show --expt-extended-lambda -c -arch=sm_70 -Xcompiler -rdynamic -DLANDAU_DIM=2 -DLANDAU_MAX_SPECIES=10 -DPETSC_HAVE_CUDA_ATOMIC -DLANDAU_MAX_Q=4 -Xcompiler -fPIC -O3 -gencode arch=compute_80,code=sm_80 -I/usr/common/software/sles15_cgpu/openmpi/4.0.3/gcc/include -Wno-deprecated-gpu-targets nvcc --expt-extended-lambda -arch=sm_70 -Wno-deprecated-gpu-targets -ccbin mpicxx -DLANDAU_DIM=2 -DLANDAU_MAX_SPECIES=10 -DPETSC_HAVE_CUDA_ATOMIC -DLANDAU_MAX_Q=4 -O3 -I/usr/common/software/sles15_cgpu/openmpi/4.0.3/gcc/include -Xcompiler -rdynamic,-fPIC,arch=compute_80,code=sm_80 -c ~/petsc (main=) arch-main
$ NVCC_WRAPPER_DEFAULT_COMPILER=mpicxx /home/bsmith/petsc/arch-main/bin/nvcc_wrapper --show --expt-extended-lambda -c -Xcompiler -fPIC -g -arch=sm_70 -gencode arch=compute_70,code=sm_70 -I/home/bsmith/soft/gnu-mpich/include -Wno-deprecated-gpu-targets -I/home/bsmith/petsc/include -I/home/bsmith/petsc/arch-main/include -I/nfs/apps/spacks/2021-02-09/opt/spack/linux-centos7-x86_64/gcc-7.3.0/cuda-11.2.0-ikruu5mo6dtt3avvmwsejouqhxu4btdm/include /home/bsmith/petsc/src/ts/utils/dmplexlandau/kokkos/landau.kokkos.cxx -o arch-main/obj/ts/utils/dmplexlandau/kokkos/landau.o nvcc --expt-extended-lambda -arch=sm_70 -Wno-deprecated-gpu-targets -ccbin mpicxx -g -I/home/bsmith/soft/gnu-mpich/include -I/home/bsmith/petsc/include -I/home/bsmith/petsc/arch-main/include -I/nfs/apps/spacks/2021-02-09/opt/spack/linux-centos7-x86_64/gcc-7.3.0/cuda-11.2.0-ikruu5mo6dtt3avvmwsejouqhxu4btdm/include -Xcompiler -fPIC,arch=compute_70,code=sm_70 -x cu /home/bsmith/petsc/src/ts/utils/dmplexlandau/kokkos/landau.kokkos.cxx -c -o arch-main/obj/ts/utils/dmplexlandau/kokkos/landau.o
On May 15, 2021, at 8:44 PM, Junchao Zhang <[email protected] <mailto:[email protected]>> wrote:
Add --show to the failing nvcc_wrapper command and see how nvcc_wrapper parses the options
PATH=/global/common/sw/cray/sles15/x86_64/zlib/1.2.11/gcc/8.2.0/pep2pal/bin:/global/common/sw/cray/cnl7/haswell/cmake/3.18.2/bin:/opt/cray/rca/2.2.20-7.0.1.1_4.62__g8e3fb5b.ari/bin:/opt/cray/alps/6.6.58-7.0.1.1_6.20__g437d88db.ari/sbin:/opt/cray/alps/default/bin:/opt/cray/job/2.2.4-7.0.1.1_3.48__g36b56f4.ari/bin:/opt/cray/pe/craype/2.6.2/bin:/usr/common/software/sles15_cgpu/openmpi/4.0.3/gcc/bin:/usr/common/software/sles15_cgpu/hpcsdk/20.11/Linux_x86_64/20.11/compilers/bin:/usr/common/software/sles15_cgpu/hpcsdk/20.11/Linux_x86_64/20.11/cuda/bin:/usr/common/software/sles15_cgpu/cuda/11.1.1/bin:/usr/common/software/sles15_cgpu/gcc/8.3.0/bin:/opt/esslurm/bin:/opt/cray/pe/modules/3.2.11.4/bin:/usr/common/software/bin:/usr/common/nsg/bin:/opt/ovis/bin:/opt/ovis/sbin:/usr/local/bin:/usr/bin:/bin:/usr/lib/mit/bin:/usr/lib/mit/sbin:/opt/cray/pe/bin:`dirname <http://3.2.11.4/bin:/usr/common/software/bin:/usr/common/nsg/bin:/opt/ovis/bin:/opt/ovis/sbin:/usr/local/bin:/usr/bin:/bin:/usr/lib/mit/bin:/usr/lib/mit/sbin:/opt/cray/pe/bin:dirname> /usr/common/software/sles15_cgpu/cuda/11.1.1/bin/nvcc` NVCC_WRAPPER_DEFAULT_COMPILER=/usr/common/software/sles15_cgpu/openmpi/4.0.3/gcc/bin/mpicxx /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/bin/nvcc_wrapper --expt-extended-lambda -c -arch=sm_70 -Xcompiler -rdynamic -DLANDAU_DIM=2 -DLANDAU_MAX_SPECIES=10 -DPETSC_HAVE_CUDA_ATOMIC -DLANDAU_MAX_Q=4 -Xcompiler -fPIC -O3 -gencode arch=compute_80,code=sm_80 -I/usr/common/software/sles15_cgpu/openmpi/4.0.3/gcc/include -Wno-deprecated-gpu-targets -I/global/homes/m/madams/petsc/include -I/global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/include -I/usr/common/software/sles15_cgpu/openmpi/4.0.3/gcc/include -I/usr/common/software/sles15_cgpu/cuda/11.1.1/include /global/u2/m/madams/petsc/src/sys/objects/kokkos/kinit.kokkos.cxx -o arch-cori-gpu-opt-kokkos-gcc/obj/sys/objects/kokkos/kinit.o --show
--Junchao Zhang
On Sat, May 15, 2021 at 7:14 PM Mark Adams <[email protected] <mailto:[email protected]>> wrote: Thanks, Now I get an error in make all. THis was working a few weeks ago. Make.log was empty but here is the output. .... CC arch-cori-gpu-opt-kokkos-gcc/obj/sys/error/ftn-custom/zerrf.o CC arch-cori-gpu-opt-kokkos-gcc/obj/sys/f90-src/f90_cwrap.o CC arch-cori-gpu-opt-kokkos-gcc/obj/vec/is/utils/hdf5io.o PATH=/global/common/sw/cray/sles15/x86_64/zlib/1.2.11/gcc/8.2.0/pep2pal/bin:/global/common/sw/cray/cnl7/haswell/cmake/3.18.2/bin:/opt/cray/rca/2.2.20-7.0.1.1_4.62__g8e3fb5b.ari/bin:/opt/cray/alps/6.6.58-7.0.1.1_6.20__g437d88db.ari/sbin:/opt/cray/alps/default/bin:/opt/cray/job/2.2.4-7.0.1.1_3.48__g36b56f4.ari/bin:/opt/cray/pe/craype/2.6.2/bin:/usr/common/software/sles15_cgpu/openmpi/4.0.3/gcc/bin:/usr/common/software/sles15_cgpu/hpcsdk/20.11/Linux_x86_64/20.11/compilers/bin:/usr/common/software/sles15_cgpu/hpcsdk/20.11/Linux_x86_64/20.11/cuda/bin:/usr/common/software/sles15_cgpu/cuda/11.1.1/bin:/usr/common/software/sles15_cgpu/gcc/8.3.0/bin:/opt/esslurm/bin:/opt/cray/pe/modules/3.2.11.4/bin:/usr/common/software/bin:/usr/common/nsg/bin:/opt/ovis/bin:/opt/ovis/sbin:/usr/local/bin:/usr/bin:/bin:/usr/lib/mit/bin:/usr/lib/mit/sbin:/opt/cray/pe/bin:`dirname <http://3.2.11.4/bin:/usr/common/software/bin:/usr/common/nsg/bin:/opt/ovis/bin:/opt/ovis/sbin:/usr/local/bin:/usr/bin:/bin:/usr/lib/mit/bin:/usr/lib/mit/sbin:/opt/cray/pe/bin:dirname> /usr/common/software/sles15_cgpu/cuda/11.1.1/bin/nvcc` NVCC_WRAPPER_DEFAULT_COMPILER=/usr/common/software/sles15_cgpu/openmpi/4.0.3/gcc/bin/mpicxx /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/bin/nvcc_wrapper --expt-extended-lambda -c -arch=sm_70 -Xcompiler -rdynamic -DLANDAU_DIM=2 -DLANDAU_MAX_SPECIES=10 -DPETSC_HAVE_CUDA_ATOMIC -DLANDAU_MAX_Q=4 -Xcompiler -fPIC -O3 -gencode arch=compute_80,code=sm_80 -I/usr/common/software/sles15_cgpu/openmpi/4.0.3/gcc/include -Wno-deprecated-gpu-targets -I/global/homes/m/madams/petsc/include -I/global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/include -I/usr/common/software/sles15_cgpu/openmpi/4.0.3/gcc/include -I/usr/common/software/sles15_cgpu/cuda/11.1.1/include /global/u2/m/madams/petsc/src/sys/objects/kokkos/kinit.kokkos.cxx -o arch-cori-gpu-opt-kokkos-gcc/obj/sys/objects/kokkos/kinit.o g++: error: arch=compute_80: No such file or directory g++: error: code=sm_80: No such file or directory gmake[3]: *** [gmakefile:188: arch-cori-gpu-opt-kokkos-gcc/obj/sys/objects/kokkos/kinit.o] Error 1 gmake[3]: *** Waiting for unfinished jobs.... CC arch-cori-gpu-opt-kokkos-gcc/obj/vec/is/utils/isio.o CC arch-cori-gpu-opt-kokkos-gcc/obj/vec/is/utils/ftn-auto/pmapf.o CC arch-cori-gpu-opt-kokkos-gcc/obj/vec/is/utils/pmap.o CC arch-cori-gpu-opt-kokkos-gcc/obj/vec/is/utils/psort.o CC arch-cori-gpu-opt-kokkos-gcc/obj/vec/is/utils/isltog.o FC arch-cori-gpu-opt-kokkos-gcc/obj/vec/f90-mod/petscvecmod.o CUDAC arch-cori-gpu-opt-kokkos-gcc/obj/sys/memory/cuda/mcudahost.o CUDAC.dep arch-cori-gpu-opt-kokkos-gcc/obj/sys/memory/cuda/mcudahost.o gmake[2]: *** [/global/homes/m/madams/petsc/lib/petsc/conf/rules:50: libs] Error 2 **************************ERROR************************************* Error during compile, check arch-cori-gpu-opt-kokkos-gcc/lib/petsc/conf/make.log Send it and arch-cori-gpu-opt-kokkos-gcc/lib/petsc/conf/configure.log to [email protected] <mailto:[email protected]> ******************************************************************** gmake[1]: *** [makefile:40: all] Error 1 make: *** [GNUmakefile:9: all] Error 2
On Sat, May 15, 2021 at 5:30 PM Barry Smith <[email protected] <mailto:[email protected]>> wrote:
Bitbucket is off-line. Do you need fortran stubs at this moment? If not use --with-fortran-interface=0
On May 15, 2021, at 2:51 PM, Mark Adams <[email protected] <mailto:[email protected]>> wrote:
I can build a non-kokkos PETSc here (Cori/GPU) but I get this error with Kokkos. Any suggestions? Thanks, Mark
============================================================================================= ============================================================================================= Trying to download git://https://bitbucket.org/petsc/pkg-sowing.git <https://bitbucket.org/petsc/pkg-sowing.git> for SOWING ============================================================================================= ============================================================================================= Trying to download git://https://bitbucket.org/petsc/pkg-sowing.git <https://bitbucket.org/petsc/pkg-sowing.git> for SOWING ============================================================================================= ============================================================================================= Trying to download https://bitbucket.org/petsc/pkg-sowing/get/v1.1.26-p1.tar.gz <https://bitbucket.org/petsc/pkg-sowing/get/v1.1.26-p1.tar.gz> for SOWING ============================================================================================= ******************************************************************************* UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log for details): ------------------------------------------------------------------------------- Error during download/extract/detection of SOWING: Unable to clone sowing Could not execute "['git clone https://bitbucket.org/petsc/pkg-sowing.git <https://bitbucket.org/petsc/pkg-sowing.git> /global/u2/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/externalpackages/git.sowing']": Cloning into '/global/u2/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/externalpackages/git.sowing'... fatal: unable to access 'https://bitbucket.org/petsc/pkg-sowing.git/ <https://bitbucket.org/petsc/pkg-sowing.git/>': The requested URL returned error: 503 Unable to download package SOWING from: git://https://bitbucket.org/petsc/pkg-sowing.git <https://bitbucket.org/petsc/pkg-sowing.git> * If URL specified manually - perhaps there is a typo? * If your network is disconnected - please reconnect and rerun ./configure * Or perhaps you have a firewall blocking the download * You can run with --with-packages-download-dir=/adirectory and ./configure will instruct you what packages to download manually * or you can download the above URL manually, to /yourselectedlocation and use the configure option: --download-sowing=/yourselectedlocation Unable to clone sowing Could not execute "['git clone https://bitbucket.org/petsc/pkg-sowing.git <https://bitbucket.org/petsc/pkg-sowing.git> /global/u2/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/externalpackages/git.sowing']": Cloning into '/global/u2/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/externalpackages/git.sowing'... fatal: unable to access 'https://bitbucket.org/petsc/pkg-sowing.git/ <https://bitbucket.org/petsc/pkg-sowing.git/>': The requested URL returned error: 503 Unable to download package SOWING from: git://https://bitbucket.org/petsc/pkg-sowing.git <https://bitbucket.org/petsc/pkg-sowing.git> * If URL specified manually - perhaps there is a typo? * If your network is disconnected - please reconnect and rerun ./configure * Or perhaps you have a firewall blocking the download * You can run with --with-packages-download-dir=/adirectory and ./configure will instruct you what packages to download manually * or you can download the above URL manually, to /yourselectedlocation and use the configure option: --download-sowing=/yourselectedlocation file could not be opened successfully Downloaded package SOWING from: https://bitbucket.org/petsc/pkg-sowing/get/v1.1.26-p1.tar.gz <https://bitbucket.org/petsc/pkg-sowing/get/v1.1.26-p1.tar.gz> is not a tarball. [or installed python cannot process compressed files] * If you are behind a firewall - please fix your proxy and rerun ./configure For example at LANL you may need to set the environmental variable http_proxy (or HTTP_PROXY?) to http://proxyout.lanl.gov <http://proxyout.lanl.gov/> * You can run with --with-packages-download-dir=/adirectory and ./configure will instruct you what packages to download manually * or you can download the above URL manually, to /yourselectedlocation/v1.1.26-p1.tar.gz and use the configure option: --download-sowing=/yourselectedlocation/v1.1.26-p1.tar.gz ******************************************************************************* <configure.log>
Mark, The kokkos-kernels did not get rebuilt so they will likely have a different geodearch. I think you need to rm -rf arch-cori-gpu-opt-kokkos-gcc but was --with-mpi-dir=/usr/common/software/sles15_cgpu/openmpi/4.0.3/gcc built with CUDA aware MPI and a different gencodarch than is supported by the hardware? If so, I guess you need to build PETSc using -with-cuda-gencodearch=70 if that is what the MPI was built with. Barry
On May 16, 2021, at 6:13 AM, Mark Adams <[email protected]> wrote:
progress:
04:08 cgpu11 adams/cusparse-cpu-solve *= ~/petsc$ make PETSC_DIR=/global/homes/m/madams/petsc PETSC_ARCH=arch-cori-gpu-opt-kokkos-gcc check Running check examples to verify correct installation Using PETSC_DIR=/global/homes/m/madams/petsc and PETSC_ARCH=arch-cori-gpu-opt-kokkos-gcc C/C++ example src/snes/tutorials/ex19 run successfully with 1 MPI process Possible error running C/C++ src/snes/tutorials/ex19 with 2 MPI processes See http://www.mcs.anl.gov/petsc/documentation/faq.html <http://www.mcs.anl.gov/petsc/documentation/faq.html> srun: error: Unable to create step for job 1921749: More processors requested than permitted 2,15c2,32 < 0 SNES Function norm 2.391552133017e-01 < 0 KSP Residual norm 2.325621076120e-01 < 1 KSP Residual norm 1.654206318674e-02 < 2 KSP Residual norm 7.202836119880e-04 < 3 KSP Residual norm 1.796861424199e-05 < 4 KSP Residual norm 2.461332992052e-07 < 1 SNES Function norm 6.826585648929e-05 < 0 KSP Residual norm 2.347339172985e-05 < 1 KSP Residual norm 8.356798075993e-07 < 2 KSP Residual norm 1.844045309619e-08 < 3 KSP Residual norm 5.336386977405e-10 < 4 KSP Residual norm 2.662608472862e-11 < 2 SNES Function norm 6.549682264799e-11 < Number of SNES iterations = 2 ---
Kokkos::Cuda::initialize ERROR: likely mismatch of architecture [cgpu11:39927] *** Process received signal *** [cgpu11:39927] Signal: Aborted (6) [cgpu11:39927] Signal code: (-6) [cgpu11:39927] [ 0] /lib64/libpthread.so.0(+0x12360)[0x2aab12731360] [cgpu11:39927] [ 1] /lib64/libc.so.6(gsignal+0x110)[0x2aab12973160] [cgpu11:39927] [ 2] /lib64/libc.so.6(abort+0x151)[0x2aab12974741] [cgpu11:39927] [ 3] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libkokkoscore.so.3.4(+0x408e5)[0x2aaabbfb88e5] [cgpu11:39927] [ 4] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libkokkoscore.so.3.4(_ZN6Kokkos4Impl12CudaInternal10initializeEiP11CUstream_st+0xb5c)[0x2aaabbfc762c] [cgpu11:39927] [ 5] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libkokkoscore.so.3.4(_ZN6Kokkos4Impl16ExecSpaceManager17initialize_spacesERKNS_13InitArgumentsE+0x24)[0x2aaabbfa1224] [cgpu11:39927] [ 6] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libkokkoscore.so.3.4(_ZN6Kokkos4Impl70_GLOBAL__N__46_tmpxft_00011209_00000000_7_Kokkos_Core_cpp1_ii_889c95a619initialize_internalERKNS_13InitArgumentsE+0x11)[0x2aaabbfa49d1] [cgpu11:39927] [ 7] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libpetsc.so.3.015(PetscKokkosInitializeCheck+0x1f9)[0x2aaaaafad809] [cgpu11:39927] [ 8] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libpetsc.so.3.015(+0x68541b)[0x2aaaab35841b] [cgpu11:39927] [ 9] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libpetsc.so.3.015(+0x36a224)[0x2aaaab03d224] [cgpu11:39927] [10] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libpetsc.so.3.015(+0x35f596)[0x2aaaab032596] [cgpu11:39927] [11] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libpetsc.so.3.015(+0x2f9d94)[0x2aaaaafccd94] [cgpu11:39927] [12] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libpetsc.so.3.015(PetscSFBcastWithMemTypeBegin+0x11f)[0x2aaaab04bc3f] [cgpu11:39927] [13] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libpetsc.so.3.015(+0x37f53a)[0x2aaaab05253a] [cgpu11:39927] [14] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libpetsc.so.3.015(VecScatterBegin+0x70)[0x2aaaab057b30] [cgpu11:39927] [15] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libpetsc.so.3.015(DMGlobalToLocalBegin_DA+0x1f)[0x2aaaabc27a5f] [cgpu11:39927] [16] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libpetsc.so.3.015(DMGlobalToLocalBegin+0x1dd)[0x2aaaabe61b3d] [cgpu11:39927] [17] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libpetsc.so.3.015(+0x144567a)[0x2aaaac11867a] [cgpu11:39927] [18] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libpetsc.so.3.015(SNESComputeFunction+0xf5)[0x2aaaac138ef5] [cgpu11:39927] [19] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libpetsc.so.3.015(+0x14ad0de)[0x2aaaac1800de] [cgpu11:39927] [20] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libpetsc.so.3.015(SNESSolve+0x821)[0x2aaaac146ed1] [cgpu11:39927] [21] /global/u2/m/madams/petsc/src/snes/tutorials/./ex19[0x401a69] [cgpu11:39927] [22] /lib64/libc.so.6(__libc_start_main+0xea)[0x2aab1295df8a] [cgpu11:39927] [23] /global/u2/m/madams/petsc/src/snes/tutorials/./ex19[0x4021ba] [cgpu11:39927] *** End of error message *** srun: error: cgpu11: task 0: Aborted srun: Terminating job step 1921749.6 /global/homes/m/madams/petsc/src/snes/tutorials Possible problem with ex19 running with cuda, diffs above ========================================= gmake[3]: [makefile:102: runex3k_kokkos] Error 1 (ignored)
On Sun, May 16, 2021 at 12:31 AM Barry Smith <[email protected] <mailto:[email protected]>> wrote:
Looks like nvcc_wrapper gets confused when both -arch and -gencode are used. But handles either of them separately correctly.
Mark,
PETSc configure now handles the -arch and --with-kokkos-cuda-arch= business automatically so you do not, and generally shouldn't pass -arch sm_70 in --CUDAFLAGS anymore or use the --with-kokkos-cuda-arch option anymore.
If you need to set the -arch to something earlier than the system supports (rare) you can now use -with-cuda-gencodearch to set that instead of using CUDAFLAGS.
Barry
$ NVCC_WRAPPER_DEFAULT_COMPILER=mpicxx /home/bsmith/petsc/arch-main/bin/nvcc_wrapper --show --expt-extended-lambda -c -Xcompiler -fPIC -g -gencode arch=compute_70,code=sm_70 -I/home/bsmith/soft/gnu-mpich/include -Wno-deprecated-gpu-targets -I/home/bsmith/petsc/include -I/home/bsmith/petsc/arch-main/include -I/nfs/apps/spacks/2021-02-09/opt/spack/linux-centos7-x86_64/gcc-7.3.0/cuda-11.2.0-ikruu5mo6dtt3avvmwsejouqhxu4btdm/include /home/bsmith/petsc/src/ts/utils/dmplexlandau/kokkos/landau.kokkos.cxx -o arch-main/obj/ts/utils/dmplexlandau/kokkos/landau.o nvcc --expt-extended-lambda -gencode arch=compute_70,code=sm_70 -Wno-deprecated-gpu-targets -ccbin mpicxx -g -I/home/bsmith/soft/gnu-mpich/include -I/home/bsmith/petsc/include -I/home/bsmith/petsc/arch-main/include -I/nfs/apps/spacks/2021-02-09/opt/spack/linux-centos7-x86_64/gcc-7.3.0/cuda-11.2.0-ikruu5mo6dtt3avvmwsejouqhxu4btdm/include -Xcompiler -fPIC -x cu /home/bsmith/petsc/src/ts/utils/dmplexlandau/kokkos/landau.kokkos.cxx -c -o arch-main/obj/ts/utils/dmplexlandau/kokkos/landau.o ~/petsc (main=) arch-main
$ NVCC_WRAPPER_DEFAULT_COMPILER=mpicxx /home/bsmith/petsc/arch-main/bin/nvcc_wrapper --show --expt-extended-lambda -c -arch=sm_70 -Xcompiler -rdynamic -DLANDAU_DIM=2 -DLANDAU_MAX_SPECIES=10 -DPETSC_HAVE_CUDA_ATOMIC -DLANDAU_MAX_Q=4 -Xcompiler -fPIC -O3 -gencode arch=compute_80,code=sm_80 -I/usr/common/software/sles15_cgpu/openmpi/4.0.3/gcc/include -Wno-deprecated-gpu-targets nvcc --expt-extended-lambda -arch=sm_70 -Wno-deprecated-gpu-targets -ccbin mpicxx -DLANDAU_DIM=2 -DLANDAU_MAX_SPECIES=10 -DPETSC_HAVE_CUDA_ATOMIC -DLANDAU_MAX_Q=4 -O3 -I/usr/common/software/sles15_cgpu/openmpi/4.0.3/gcc/include -Xcompiler -rdynamic,-fPIC,arch=compute_80,code=sm_80 -c ~/petsc (main=) arch-main
$ NVCC_WRAPPER_DEFAULT_COMPILER=mpicxx /home/bsmith/petsc/arch-main/bin/nvcc_wrapper --show --expt-extended-lambda -c -Xcompiler -fPIC -g -arch=sm_70 -gencode arch=compute_70,code=sm_70 -I/home/bsmith/soft/gnu-mpich/include -Wno-deprecated-gpu-targets -I/home/bsmith/petsc/include -I/home/bsmith/petsc/arch-main/include -I/nfs/apps/spacks/2021-02-09/opt/spack/linux-centos7-x86_64/gcc-7.3.0/cuda-11.2.0-ikruu5mo6dtt3avvmwsejouqhxu4btdm/include /home/bsmith/petsc/src/ts/utils/dmplexlandau/kokkos/landau.kokkos.cxx -o arch-main/obj/ts/utils/dmplexlandau/kokkos/landau.o nvcc --expt-extended-lambda -arch=sm_70 -Wno-deprecated-gpu-targets -ccbin mpicxx -g -I/home/bsmith/soft/gnu-mpich/include -I/home/bsmith/petsc/include -I/home/bsmith/petsc/arch-main/include -I/nfs/apps/spacks/2021-02-09/opt/spack/linux-centos7-x86_64/gcc-7.3.0/cuda-11.2.0-ikruu5mo6dtt3avvmwsejouqhxu4btdm/include -Xcompiler -fPIC,arch=compute_70,code=sm_70 -x cu /home/bsmith/petsc/src/ts/utils/dmplexlandau/kokkos/landau.kokkos.cxx -c -o arch-main/obj/ts/utils/dmplexlandau/kokkos/landau.o
On May 15, 2021, at 8:44 PM, Junchao Zhang <[email protected] <mailto:[email protected]>> wrote:
Add --show to the failing nvcc_wrapper command and see how nvcc_wrapper parses the options
PATH=/global/common/sw/cray/sles15/x86_64/zlib/1.2.11/gcc/8.2.0/pep2pal/bin:/global/common/sw/cray/cnl7/haswell/cmake/3.18.2/bin:/opt/cray/rca/2.2.20-7.0.1.1_4.62__g8e3fb5b.ari/bin:/opt/cray/alps/6.6.58-7.0.1.1_6.20__g437d88db.ari/sbin:/opt/cray/alps/default/bin:/opt/cray/job/2.2.4-7.0.1.1_3.48__g36b56f4.ari/bin:/opt/cray/pe/craype/2.6.2/bin:/usr/common/software/sles15_cgpu/openmpi/4.0.3/gcc/bin:/usr/common/software/sles15_cgpu/hpcsdk/20.11/Linux_x86_64/20.11/compilers/bin:/usr/common/software/sles15_cgpu/hpcsdk/20.11/Linux_x86_64/20.11/cuda/bin:/usr/common/software/sles15_cgpu/cuda/11.1.1/bin:/usr/common/software/sles15_cgpu/gcc/8.3.0/bin:/opt/esslurm/bin:/opt/cray/pe/modules/3.2.11.4/bin:/usr/common/software/bin:/usr/common/nsg/bin:/opt/ovis/bin:/opt/ovis/sbin:/usr/local/bin:/usr/bin:/bin:/usr/lib/mit/bin:/usr/lib/mit/sbin:/opt/cray/pe/bin:`dirname <http://3.2.11.4/bin:/usr/common/software/bin:/usr/common/nsg/bin:/opt/ovis/bin:/opt/ovis/sbin:/usr/local/bin:/usr/bin:/bin:/usr/lib/mit/bin:/usr/lib/mit/sbin:/opt/cray/pe/bin:dirname> /usr/common/software/sles15_cgpu/cuda/11.1.1/bin/nvcc` NVCC_WRAPPER_DEFAULT_COMPILER=/usr/common/software/sles15_cgpu/openmpi/4.0.3/gcc/bin/mpicxx /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/bin/nvcc_wrapper --expt-extended-lambda -c -arch=sm_70 -Xcompiler -rdynamic -DLANDAU_DIM=2 -DLANDAU_MAX_SPECIES=10 -DPETSC_HAVE_CUDA_ATOMIC -DLANDAU_MAX_Q=4 -Xcompiler -fPIC -O3 -gencode arch=compute_80,code=sm_80 -I/usr/common/software/sles15_cgpu/openmpi/4.0.3/gcc/include -Wno-deprecated-gpu-targets -I/global/homes/m/madams/petsc/include -I/global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/include -I/usr/common/software/sles15_cgpu/openmpi/4.0.3/gcc/include -I/usr/common/software/sles15_cgpu/cuda/11.1.1/include /global/u2/m/madams/petsc/src/sys/objects/kokkos/kinit.kokkos.cxx -o arch-cori-gpu-opt-kokkos-gcc/obj/sys/objects/kokkos/kinit.o --show
--Junchao Zhang
On Sat, May 15, 2021 at 7:14 PM Mark Adams <[email protected] <mailto:[email protected]>> wrote: Thanks, Now I get an error in make all. THis was working a few weeks ago. Make.log was empty but here is the output. .... CC arch-cori-gpu-opt-kokkos-gcc/obj/sys/error/ftn-custom/zerrf.o CC arch-cori-gpu-opt-kokkos-gcc/obj/sys/f90-src/f90_cwrap.o CC arch-cori-gpu-opt-kokkos-gcc/obj/vec/is/utils/hdf5io.o PATH=/global/common/sw/cray/sles15/x86_64/zlib/1.2.11/gcc/8.2.0/pep2pal/bin:/global/common/sw/cray/cnl7/haswell/cmake/3.18.2/bin:/opt/cray/rca/2.2.20-7.0.1.1_4.62__g8e3fb5b.ari/bin:/opt/cray/alps/6.6.58-7.0.1.1_6.20__g437d88db.ari/sbin:/opt/cray/alps/default/bin:/opt/cray/job/2.2.4-7.0.1.1_3.48__g36b56f4.ari/bin:/opt/cray/pe/craype/2.6.2/bin:/usr/common/software/sles15_cgpu/openmpi/4.0.3/gcc/bin:/usr/common/software/sles15_cgpu/hpcsdk/20.11/Linux_x86_64/20.11/compilers/bin:/usr/common/software/sles15_cgpu/hpcsdk/20.11/Linux_x86_64/20.11/cuda/bin:/usr/common/software/sles15_cgpu/cuda/11.1.1/bin:/usr/common/software/sles15_cgpu/gcc/8.3.0/bin:/opt/esslurm/bin:/opt/cray/pe/modules/3.2.11.4/bin:/usr/common/software/bin:/usr/common/nsg/bin:/opt/ovis/bin:/opt/ovis/sbin:/usr/local/bin:/usr/bin:/bin:/usr/lib/mit/bin:/usr/lib/mit/sbin:/opt/cray/pe/bin:`dirname <http://3.2.11.4/bin:/usr/common/software/bin:/usr/common/nsg/bin:/opt/ovis/bin:/opt/ovis/sbin:/usr/local/bin:/usr/bin:/bin:/usr/lib/mit/bin:/usr/lib/mit/sbin:/opt/cray/pe/bin:dirname> /usr/common/software/sles15_cgpu/cuda/11.1.1/bin/nvcc` NVCC_WRAPPER_DEFAULT_COMPILER=/usr/common/software/sles15_cgpu/openmpi/4.0.3/gcc/bin/mpicxx /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/bin/nvcc_wrapper --expt-extended-lambda -c -arch=sm_70 -Xcompiler -rdynamic -DLANDAU_DIM=2 -DLANDAU_MAX_SPECIES=10 -DPETSC_HAVE_CUDA_ATOMIC -DLANDAU_MAX_Q=4 -Xcompiler -fPIC -O3 -gencode arch=compute_80,code=sm_80 -I/usr/common/software/sles15_cgpu/openmpi/4.0.3/gcc/include -Wno-deprecated-gpu-targets -I/global/homes/m/madams/petsc/include -I/global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/include -I/usr/common/software/sles15_cgpu/openmpi/4.0.3/gcc/include -I/usr/common/software/sles15_cgpu/cuda/11.1.1/include /global/u2/m/madams/petsc/src/sys/objects/kokkos/kinit.kokkos.cxx -o arch-cori-gpu-opt-kokkos-gcc/obj/sys/objects/kokkos/kinit.o g++: error: arch=compute_80: No such file or directory g++: error: code=sm_80: No such file or directory gmake[3]: *** [gmakefile:188: arch-cori-gpu-opt-kokkos-gcc/obj/sys/objects/kokkos/kinit.o] Error 1 gmake[3]: *** Waiting for unfinished jobs.... CC arch-cori-gpu-opt-kokkos-gcc/obj/vec/is/utils/isio.o CC arch-cori-gpu-opt-kokkos-gcc/obj/vec/is/utils/ftn-auto/pmapf.o CC arch-cori-gpu-opt-kokkos-gcc/obj/vec/is/utils/pmap.o CC arch-cori-gpu-opt-kokkos-gcc/obj/vec/is/utils/psort.o CC arch-cori-gpu-opt-kokkos-gcc/obj/vec/is/utils/isltog.o FC arch-cori-gpu-opt-kokkos-gcc/obj/vec/f90-mod/petscvecmod.o CUDAC arch-cori-gpu-opt-kokkos-gcc/obj/sys/memory/cuda/mcudahost.o CUDAC.dep arch-cori-gpu-opt-kokkos-gcc/obj/sys/memory/cuda/mcudahost.o gmake[2]: *** [/global/homes/m/madams/petsc/lib/petsc/conf/rules:50: libs] Error 2 **************************ERROR************************************* Error during compile, check arch-cori-gpu-opt-kokkos-gcc/lib/petsc/conf/make.log Send it and arch-cori-gpu-opt-kokkos-gcc/lib/petsc/conf/configure.log to [email protected] <mailto:[email protected]> ******************************************************************** gmake[1]: *** [makefile:40: all] Error 1 make: *** [GNUmakefile:9: all] Error 2
On Sat, May 15, 2021 at 5:30 PM Barry Smith <[email protected] <mailto:[email protected]>> wrote:
Bitbucket is off-line. Do you need fortran stubs at this moment? If not use --with-fortran-interface=0
On May 15, 2021, at 2:51 PM, Mark Adams <[email protected] <mailto:[email protected]>> wrote:
I can build a non-kokkos PETSc here (Cori/GPU) but I get this error with Kokkos. Any suggestions? Thanks, Mark
============================================================================================= ============================================================================================= Trying to download git://https://bitbucket.org/petsc/pkg-sowing.git <https://bitbucket.org/petsc/pkg-sowing.git> for SOWING ============================================================================================= ============================================================================================= Trying to download git://https://bitbucket.org/petsc/pkg-sowing.git <https://bitbucket.org/petsc/pkg-sowing.git> for SOWING ============================================================================================= ============================================================================================= Trying to download https://bitbucket.org/petsc/pkg-sowing/get/v1.1.26-p1.tar.gz <https://bitbucket.org/petsc/pkg-sowing/get/v1.1.26-p1.tar.gz> for SOWING ============================================================================================= ******************************************************************************* UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log for details): ------------------------------------------------------------------------------- Error during download/extract/detection of SOWING: Unable to clone sowing Could not execute "['git clone https://bitbucket.org/petsc/pkg-sowing.git <https://bitbucket.org/petsc/pkg-sowing.git> /global/u2/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/externalpackages/git.sowing']": Cloning into '/global/u2/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/externalpackages/git.sowing'... fatal: unable to access 'https://bitbucket.org/petsc/pkg-sowing.git/ <https://bitbucket.org/petsc/pkg-sowing.git/>': The requested URL returned error: 503 Unable to download package SOWING from: git://https://bitbucket.org/petsc/pkg-sowing.git <https://bitbucket.org/petsc/pkg-sowing.git> * If URL specified manually - perhaps there is a typo? * If your network is disconnected - please reconnect and rerun ./configure * Or perhaps you have a firewall blocking the download * You can run with --with-packages-download-dir=/adirectory and ./configure will instruct you what packages to download manually * or you can download the above URL manually, to /yourselectedlocation and use the configure option: --download-sowing=/yourselectedlocation Unable to clone sowing Could not execute "['git clone https://bitbucket.org/petsc/pkg-sowing.git <https://bitbucket.org/petsc/pkg-sowing.git> /global/u2/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/externalpackages/git.sowing']": Cloning into '/global/u2/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/externalpackages/git.sowing'... fatal: unable to access 'https://bitbucket.org/petsc/pkg-sowing.git/ <https://bitbucket.org/petsc/pkg-sowing.git/>': The requested URL returned error: 503 Unable to download package SOWING from: git://https://bitbucket.org/petsc/pkg-sowing.git <https://bitbucket.org/petsc/pkg-sowing.git> * If URL specified manually - perhaps there is a typo? * If your network is disconnected - please reconnect and rerun ./configure * Or perhaps you have a firewall blocking the download * You can run with --with-packages-download-dir=/adirectory and ./configure will instruct you what packages to download manually * or you can download the above URL manually, to /yourselectedlocation and use the configure option: --download-sowing=/yourselectedlocation file could not be opened successfully Downloaded package SOWING from: https://bitbucket.org/petsc/pkg-sowing/get/v1.1.26-p1.tar.gz <https://bitbucket.org/petsc/pkg-sowing/get/v1.1.26-p1.tar.gz> is not a tarball. [or installed python cannot process compressed files] * If you are behind a firewall - please fix your proxy and rerun ./configure For example at LANL you may need to set the environmental variable http_proxy (or HTTP_PROXY?) to http://proxyout.lanl.gov <http://proxyout.lanl.gov/> * You can run with --with-packages-download-dir=/adirectory and ./configure will instruct you what packages to download manually * or you can download the above URL manually, to /yourselectedlocation/v1.1.26-p1.tar.gz and use the configure option: --download-sowing=/yourselectedlocation/v1.1.26-p1.tar.gz ******************************************************************************* <configure.log>
<make.log><configure.log>
On Sun, May 16, 2021 at 9:58 AM Barry Smith <[email protected]> wrote:
Mark,
The kokkos-kernels did not get rebuilt so they will likely have a different geodearch.
I think you need to rm -rf arch-cori-gpu-opt-kokkos-gcc
Yes, of course, but did not help. I still get this error.
but was --with-mpi-dir=/usr/common/software/sles15_cgpu/openmpi/4.0.3/gcc built with CUDA aware MPI and a different gencodarch than is supported by the hardware?
I don't know. I use MPI very very little (and not timed) so I don't care about it. This is just what I got to compile and work.
If so, I guess you need to build PETSc using -with-cuda-gencodearch=70 if that is what the MPI was built with.
OK, that worked, thanks,
Barry
On May 16, 2021, at 6:13 AM, Mark Adams <[email protected]> wrote:
progress:
04:08 cgpu11 adams/cusparse-cpu-solve *= ~/petsc$ make PETSC_DIR=/global/homes/m/madams/petsc PETSC_ARCH=arch-cori-gpu-opt-kokkos-gcc check Running check examples to verify correct installation Using PETSC_DIR=/global/homes/m/madams/petsc and PETSC_ARCH=arch-cori-gpu-opt-kokkos-gcc C/C++ example src/snes/tutorials/ex19 run successfully with 1 MPI process Possible error running C/C++ src/snes/tutorials/ex19 with 2 MPI processes See http://www.mcs.anl.gov/petsc/documentation/faq.html srun: error: Unable to create step for job 1921749: More processors requested than permitted 2,15c2,32 < 0 SNES Function norm 2.391552133017e-01 < 0 KSP Residual norm 2.325621076120e-01 < 1 KSP Residual norm 1.654206318674e-02 < 2 KSP Residual norm 7.202836119880e-04 < 3 KSP Residual norm 1.796861424199e-05 < 4 KSP Residual norm 2.461332992052e-07 < 1 SNES Function norm 6.826585648929e-05 < 0 KSP Residual norm 2.347339172985e-05 < 1 KSP Residual norm 8.356798075993e-07 < 2 KSP Residual norm 1.844045309619e-08 < 3 KSP Residual norm 5.336386977405e-10 < 4 KSP Residual norm 2.662608472862e-11 < 2 SNES Function norm 6.549682264799e-11 < Number of SNES iterations = 2 ---
Kokkos::Cuda::initialize ERROR: likely mismatch of architecture [cgpu11:39927] *** Process received signal *** [cgpu11:39927] Signal: Aborted (6) [cgpu11:39927] Signal code: (-6) [cgpu11:39927] [ 0] /lib64/libpthread.so.0(+0x12360)[0x2aab12731360] [cgpu11:39927] [ 1] /lib64/libc.so.6(gsignal+0x110)[0x2aab12973160] [cgpu11:39927] [ 2] /lib64/libc.so.6(abort+0x151)[0x2aab12974741] [cgpu11:39927] [ 3] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libkokkoscore.so.3.4(+0x408e5)[0x2aaabbfb88e5] [cgpu11:39927] [ 4] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libkokkoscore.so.3.4(_ZN6Kokkos4Impl12CudaInternal10initializeEiP11CUstream_st+0xb5c)[0x2aaabbfc762c] [cgpu11:39927] [ 5] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libkokkoscore.so.3.4(_ZN6Kokkos4Impl16ExecSpaceManager17initialize_spacesERKNS_13InitArgumentsE+0x24)[0x2aaabbfa1224] [cgpu11:39927] [ 6] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libkokkoscore.so.3.4(_ZN6Kokkos4Impl70_GLOBAL__N__46_tmpxft_00011209_00000000_7_Kokkos_Core_cpp1_ii_889c95a619initialize_internalERKNS_13InitArgumentsE+0x11)[0x2aaabbfa49d1] [cgpu11:39927] [ 7] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libpetsc.so.3.015(PetscKokkosInitializeCheck+0x1f9)[0x2aaaaafad809] [cgpu11:39927] [ 8] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libpetsc.so.3.015(+0x68541b)[0x2aaaab35841b] [cgpu11:39927] [ 9] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libpetsc.so.3.015(+0x36a224)[0x2aaaab03d224] [cgpu11:39927] [10] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libpetsc.so.3.015(+0x35f596)[0x2aaaab032596] [cgpu11:39927] [11] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libpetsc.so.3.015(+0x2f9d94)[0x2aaaaafccd94] [cgpu11:39927] [12] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libpetsc.so.3.015(PetscSFBcastWithMemTypeBegin+0x11f)[0x2aaaab04bc3f] [cgpu11:39927] [13] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libpetsc.so.3.015(+0x37f53a)[0x2aaaab05253a] [cgpu11:39927] [14] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libpetsc.so.3.015(VecScatterBegin+0x70)[0x2aaaab057b30] [cgpu11:39927] [15] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libpetsc.so.3.015(DMGlobalToLocalBegin_DA+0x1f)[0x2aaaabc27a5f] [cgpu11:39927] [16] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libpetsc.so.3.015(DMGlobalToLocalBegin+0x1dd)[0x2aaaabe61b3d] [cgpu11:39927] [17] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libpetsc.so.3.015(+0x144567a)[0x2aaaac11867a] [cgpu11:39927] [18] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libpetsc.so.3.015(SNESComputeFunction+0xf5)[0x2aaaac138ef5] [cgpu11:39927] [19] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libpetsc.so.3.015(+0x14ad0de)[0x2aaaac1800de] [cgpu11:39927] [20] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libpetsc.so.3.015(SNESSolve+0x821)[0x2aaaac146ed1] [cgpu11:39927] [21] /global/u2/m/madams/petsc/src/snes/tutorials/./ex19[0x401a69] [cgpu11:39927] [22] /lib64/libc.so.6(__libc_start_main+0xea)[0x2aab1295df8a] [cgpu11:39927] [23] /global/u2/m/madams/petsc/src/snes/tutorials/./ex19[0x4021ba] [cgpu11:39927] *** End of error message *** srun: error: cgpu11: task 0: Aborted srun: Terminating job step 1921749.6 /global/homes/m/madams/petsc/src/snes/tutorials Possible problem with ex19 running with cuda, diffs above ========================================= gmake[3]: [makefile:102: runex3k_kokkos] Error 1 (ignored)
On Sun, May 16, 2021 at 12:31 AM Barry Smith <[email protected]> wrote:
Looks like nvcc_wrapper gets confused when both -arch and -gencode are used. But handles either of them separately correctly.
Mark,
PETSc configure now handles the -arch and --with-kokkos-cuda-arch= business automatically so you do not, and generally shouldn't pass -arch sm_70 in --CUDAFLAGS anymore or use the --with-kokkos-cuda-arch option anymore.
If you need to set the -arch to something earlier than the system supports (rare) you can now use -with-cuda-gencodearch to set that instead of using CUDAFLAGS.
Barry
$ NVCC_WRAPPER_DEFAULT_COMPILER=mpicxx /home/bsmith/petsc/arch-main/bin/nvcc_wrapper --show --expt-extended-lambda -c -Xcompiler -fPIC -g -gencode arch=compute_70,code=sm_70 -I/home/bsmith/soft/gnu-mpich/include -Wno-deprecated-gpu-targets -I/home/bsmith/petsc/include -I/home/bsmith/petsc/arch-main/include -I/nfs/apps/spacks/2021-02-09/opt/spack/linux-centos7-x86_64/gcc-7.3.0/cuda-11.2.0-ikruu5mo6dtt3avvmwsejouqhxu4btdm/include /home/bsmith/petsc/src/ts/utils/dmplexlandau/kokkos/landau.kokkos.cxx -o arch-main/obj/ts/utils/dmplexlandau/kokkos/landau.o nvcc --expt-extended-lambda -gencode arch=compute_70,code=sm_70 -Wno-deprecated-gpu-targets -ccbin mpicxx -g -I/home/bsmith/soft/gnu-mpich/include -I/home/bsmith/petsc/include -I/home/bsmith/petsc/arch-main/include -I/nfs/apps/spacks/2021-02-09/opt/spack/linux-centos7-x86_64/gcc-7.3.0/cuda-11.2.0-ikruu5mo6dtt3avvmwsejouqhxu4btdm/include -Xcompiler -fPIC -x cu /home/bsmith/petsc/src/ts/utils/dmplexlandau/kokkos/landau.kokkos.cxx -c -o arch-main/obj/ts/utils/dmplexlandau/kokkos/landau.o ~/petsc* (main=)* arch-main
$ NVCC_WRAPPER_DEFAULT_COMPILER=mpicxx /home/bsmith/petsc/arch-main/bin/nvcc_wrapper --show --expt-extended-lambda -c -arch=sm_70 -Xcompiler -rdynamic -DLANDAU_DIM=2 -DLANDAU_MAX_SPECIES=10 -DPETSC_HAVE_CUDA_ATOMIC -DLANDAU_MAX_Q=4 -Xcompiler -fPIC -O3 -gencode arch=compute_80,code=sm_80 -I/usr/common/software/sles15_cgpu/openmpi/4.0.3/gcc/include -Wno-deprecated-gpu-targets nvcc --expt-extended-lambda -arch=sm_70 -Wno-deprecated-gpu-targets -ccbin mpicxx -DLANDAU_DIM=2 -DLANDAU_MAX_SPECIES=10 -DPETSC_HAVE_CUDA_ATOMIC -DLANDAU_MAX_Q=4 -O3 -I/usr/common/software/sles15_cgpu/openmpi/4.0.3/gcc/include -Xcompiler -rdynamic,-fPIC,arch=compute_80,code=sm_80 -c ~/petsc* (main=)* arch-main
$ NVCC_WRAPPER_DEFAULT_COMPILER=mpicxx /home/bsmith/petsc/arch-main/bin/nvcc_wrapper --show --expt-extended-lambda -c -Xcompiler -fPIC -g -arch=sm_70 -gencode arch=compute_70,code=sm_70 -I/home/bsmith/soft/gnu-mpich/include -Wno-deprecated-gpu-targets -I/home/bsmith/petsc/include -I/home/bsmith/petsc/arch-main/include -I/nfs/apps/spacks/2021-02-09/opt/spack/linux-centos7-x86_64/gcc-7.3.0/cuda-11.2.0-ikruu5mo6dtt3avvmwsejouqhxu4btdm/include /home/bsmith/petsc/src/ts/utils/dmplexlandau/kokkos/landau.kokkos.cxx -o arch-main/obj/ts/utils/dmplexlandau/kokkos/landau.o nvcc --expt-extended-lambda -arch=sm_70 -Wno-deprecated-gpu-targets -ccbin mpicxx -g -I/home/bsmith/soft/gnu-mpich/include -I/home/bsmith/petsc/include -I/home/bsmith/petsc/arch-main/include -I/nfs/apps/spacks/2021-02-09/opt/spack/linux-centos7-x86_64/gcc-7.3.0/cuda-11.2.0-ikruu5mo6dtt3avvmwsejouqhxu4btdm/include -Xcompiler -fPIC,arch=compute_70,code=sm_70 -x cu /home/bsmith/petsc/src/ts/utils/dmplexlandau/kokkos/landau.kokkos.cxx -c -o arch-main/obj/ts/utils/dmplexlandau/kokkos/landau.o
On May 15, 2021, at 8:44 PM, Junchao Zhang <[email protected]> wrote:
Add --show to the failing nvcc_wrapper command and see how nvcc_wrapper parses the options
PATH=/global/common/sw/cray/sles15/x86_64/zlib/1.2.11/gcc/8.2.0/pep2pal/bin:/global/common/sw/cray/cnl7/haswell/cmake/3.18.2/bin:/opt/cray/rca/2.2.20-7.0.1.1_4.62__g8e3fb5b.ari/bin:/opt/cray/alps/6.6.58-7.0.1.1_6.20__g437d88db.ari/sbin:/opt/cray/alps/default/bin:/opt/cray/job/2.2.4-7.0.1.1_3.48__g36b56f4.ari/bin:/opt/cray/pe/craype/2.6.2/bin:/usr/common/software/sles15_cgpu/openmpi/4.0.3/gcc/bin:/usr/common/software/sles15_cgpu/hpcsdk/20.11/Linux_x86_64/20.11/compilers/bin:/usr/common/software/sles15_cgpu/hpcsdk/20.11/Linux_x86_64/20.11/cuda/bin:/usr/common/software/sles15_cgpu/cuda/11.1.1/bin:/usr/common/software/sles15_cgpu/gcc/8.3.0/bin:/opt/esslurm/bin:/opt/cray/pe/modules/ 3.2.11.4/bin:/usr/common/software/bin:/usr/common/nsg/bin:/opt/ovis/bin:/opt/ovis/sbin:/usr/local/bin:/usr/bin:/bin:/usr/lib/mit/bin:/usr/lib/mit/sbin:/opt/cray/pe/bin:`dirname <http://3.2.11.4/bin:/usr/common/software/bin:/usr/common/nsg/bin:/opt/ovis/bin:/opt/ovis/sbin:/usr/local/bin:/usr/bin:/bin:/usr/lib/mit/bin:/usr/lib/mit/sbin:/opt/cray/pe/bin:dirname> /usr/common/software/sles15_cgpu/cuda/11.1.1/bin/nvcc` NVCC_WRAPPER_DEFAULT_COMPILER=/usr/common/software/sles15_cgpu/openmpi/4.0.3/gcc/bin/mpicxx /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/bin/nvcc_wrapper --expt-extended-lambda -c -arch=sm_70 -Xcompiler -rdynamic -DLANDAU_DIM=2 -DLANDAU_MAX_SPECIES=10 -DPETSC_HAVE_CUDA_ATOMIC -DLANDAU_MAX_Q=4 -Xcompiler -fPIC -O3 -gencode arch=compute_80,code=sm_80 -I/usr/common/software/sles15_cgpu/openmpi/4.0.3/gcc/include -Wno-deprecated-gpu-targets -I/global/homes/m/madams/petsc/include -I/global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/include -I/usr/common/software/sles15_cgpu/openmpi/4.0.3/gcc/include -I/usr/common/software/sles15_cgpu/cuda/11.1.1/include /global/u2/m/madams/petsc/src/sys/objects/kokkos/kinit.kokkos.cxx -o arch-cori-gpu-opt-kokkos-gcc/obj/sys/objects/kokkos/kinit.o --show
--Junchao Zhang
On Sat, May 15, 2021 at 7:14 PM Mark Adams <[email protected]> wrote:
Thanks, Now I get an error in make all. THis was working a few weeks ago. Make.log was empty but here is the output. .... CC arch-cori-gpu-opt-kokkos-gcc/obj/sys/error/ftn-custom/zerrf.o CC arch-cori-gpu-opt-kokkos-gcc/obj/sys/f90-src/f90_cwrap.o CC arch-cori-gpu-opt-kokkos-gcc/obj/vec/is/utils/hdf5io.o
PATH=/global/common/sw/cray/sles15/x86_64/zlib/1.2.11/gcc/8.2.0/pep2pal/bin:/global/common/sw/cray/cnl7/haswell/cmake/3.18.2/bin:/opt/cray/rca/2.2.20-7.0.1.1_4.62__g8e3fb5b.ari/bin:/opt/cray/alps/6.6.58-7.0.1.1_6.20__g437d88db.ari/sbin:/opt/cray/alps/default/bin:/opt/cray/job/2.2.4-7.0.1.1_3.48__g36b56f4.ari/bin:/opt/cray/pe/craype/2.6.2/bin:/usr/common/software/sles15_cgpu/openmpi/4.0.3/gcc/bin:/usr/common/software/sles15_cgpu/hpcsdk/20.11/Linux_x86_64/20.11/compilers/bin:/usr/common/software/sles15_cgpu/hpcsdk/20.11/Linux_x86_64/20.11/cuda/bin:/usr/common/software/sles15_cgpu/cuda/11.1.1/bin:/usr/common/software/sles15_cgpu/gcc/8.3.0/bin:/opt/esslurm/bin:/opt/cray/pe/modules/ 3.2.11.4/bin:/usr/common/software/bin:/usr/common/nsg/bin:/opt/ovis/bin:/opt/ovis/sbin:/usr/local/bin:/usr/bin:/bin:/usr/lib/mit/bin:/usr/lib/mit/sbin:/opt/cray/pe/bin:`dirname <http://3.2.11.4/bin:/usr/common/software/bin:/usr/common/nsg/bin:/opt/ovis/bin:/opt/ovis/sbin:/usr/local/bin:/usr/bin:/bin:/usr/lib/mit/bin:/usr/lib/mit/sbin:/opt/cray/pe/bin:dirname> /usr/common/software/sles15_cgpu/cuda/11.1.1/bin/nvcc` NVCC_WRAPPER_DEFAULT_COMPILER=/usr/common/software/sles15_cgpu/openmpi/4.0.3/gcc/bin/mpicxx /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/bin/nvcc_wrapper --expt-extended-lambda -c -arch=sm_70 -Xcompiler -rdynamic -DLANDAU_DIM=2 -DLANDAU_MAX_SPECIES=10 -DPETSC_HAVE_CUDA_ATOMIC -DLANDAU_MAX_Q=4 -Xcompiler -fPIC -O3 -gencode arch=compute_80,code=sm_80 -I/usr/common/software/sles15_cgpu/openmpi/4.0.3/gcc/include -Wno-deprecated-gpu-targets -I/global/homes/m/madams/petsc/include -I/global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/include -I/usr/common/software/sles15_cgpu/openmpi/4.0.3/gcc/include -I/usr/common/software/sles15_cgpu/cuda/11.1.1/include /global/u2/m/madams/petsc/src/sys/objects/kokkos/kinit.kokkos.cxx -o arch-cori-gpu-opt-kokkos-gcc/obj/sys/objects/kokkos/kinit.o g++: error: arch=compute_80: No such file or directory g++: error: code=sm_80: No such file or directory gmake[3]: *** [gmakefile:188: arch-cori-gpu-opt-kokkos-gcc/obj/sys/objects/kokkos/kinit.o] Error 1 gmake[3]: *** Waiting for unfinished jobs.... CC arch-cori-gpu-opt-kokkos-gcc/obj/vec/is/utils/isio.o CC arch-cori-gpu-opt-kokkos-gcc/obj/vec/is/utils/ftn-auto/pmapf.o CC arch-cori-gpu-opt-kokkos-gcc/obj/vec/is/utils/pmap.o CC arch-cori-gpu-opt-kokkos-gcc/obj/vec/is/utils/psort.o CC arch-cori-gpu-opt-kokkos-gcc/obj/vec/is/utils/isltog.o FC arch-cori-gpu-opt-kokkos-gcc/obj/vec/f90-mod/petscvecmod.o CUDAC arch-cori-gpu-opt-kokkos-gcc/obj/sys/memory/cuda/mcudahost.o CUDAC.dep arch-cori-gpu-opt-kokkos-gcc/obj/sys/memory/cuda/mcudahost.o gmake[2]: *** [/global/homes/m/madams/petsc/lib/petsc/conf/rules:50: libs] Error 2 **************************ERROR************************************* Error during compile, check arch-cori-gpu-opt-kokkos-gcc/lib/petsc/conf/make.log Send it and arch-cori-gpu-opt-kokkos-gcc/lib/petsc/conf/configure.log to [email protected] ******************************************************************** gmake[1]: *** [makefile:40: all] Error 1 make: *** [GNUmakefile:9: all] Error 2
On Sat, May 15, 2021 at 5:30 PM Barry Smith <[email protected]> wrote:
Bitbucket is off-line. Do you need fortran stubs at this moment? If not use --with-fortran-interface=0
On May 15, 2021, at 2:51 PM, Mark Adams <[email protected]> wrote:
I can build a non-kokkos PETSc here (Cori/GPU) but I get this error with Kokkos. Any suggestions? Thanks, Mark
=============================================================================================
=============================================================================================
Trying to download git://https://bitbucket.org/petsc/pkg-sowing.git for SOWING
=============================================================================================
=============================================================================================
Trying to download git://https://bitbucket.org/petsc/pkg-sowing.git for SOWING
=============================================================================================
=============================================================================================
Trying to download https://bitbucket.org/petsc/pkg-sowing/get/v1.1.26-p1.tar.gz for SOWING
=============================================================================================
******************************************************************************* UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log for details):
------------------------------------------------------------------------------- Error during download/extract/detection of SOWING: Unable to clone sowing Could not execute "['git clone https://bitbucket.org/petsc/pkg-sowing.git /global/u2/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/externalpackages/git.sowing']": Cloning into '/global/u2/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/externalpackages/git.sowing'... fatal: unable to access 'https://bitbucket.org/petsc/pkg-sowing.git/': The requested URL returned error: 503 Unable to download package SOWING from: git:// https://bitbucket.org/petsc/pkg-sowing.git * If URL specified manually - perhaps there is a typo? * If your network is disconnected - please reconnect and rerun ./configure * Or perhaps you have a firewall blocking the download * You can run with --with-packages-download-dir=/adirectory and ./configure will instruct you what packages to download manually * or you can download the above URL manually, to /yourselectedlocation and use the configure option: --download-sowing=/yourselectedlocation Unable to clone sowing Could not execute "['git clone https://bitbucket.org/petsc/pkg-sowing.git /global/u2/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/externalpackages/git.sowing']": Cloning into '/global/u2/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/externalpackages/git.sowing'... fatal: unable to access 'https://bitbucket.org/petsc/pkg-sowing.git/': The requested URL returned error: 503 Unable to download package SOWING from: git:// https://bitbucket.org/petsc/pkg-sowing.git * If URL specified manually - perhaps there is a typo? * If your network is disconnected - please reconnect and rerun ./configure * Or perhaps you have a firewall blocking the download * You can run with --with-packages-download-dir=/adirectory and ./configure will instruct you what packages to download manually * or you can download the above URL manually, to /yourselectedlocation and use the configure option: --download-sowing=/yourselectedlocation file could not be opened successfully Downloaded package SOWING from: https://bitbucket.org/petsc/pkg-sowing/get/v1.1.26-p1.tar.gz is not a tarball. [or installed python cannot process compressed files] * If you are behind a firewall - please fix your proxy and rerun ./configure For example at LANL you may need to set the environmental variable http_proxy (or HTTP_PROXY?) to http://proxyout.lanl.gov * You can run with --with-packages-download-dir=/adirectory and ./configure will instruct you what packages to download manually * or you can download the above URL manually, to /yourselectedlocation/v1.1.26-p1.tar.gz and use the configure option: --download-sowing=/yourselectedlocation/v1.1.26-p1.tar.gz
******************************************************************************* <configure.log>
<make.log><configure.log>
I now get this error. A blas error from VecAXPBYPCZ ... Any ideas? terminate called after throwing an instance of 'std::runtime_error' what(): cudaFuncGetAttributes(&attr_tmp, base_t::get_kernel_func()) error( cudaErrorInvalidDeviceFunction): invalid device function /global/u2/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/include/Cuda/Kokkos_Cuda_KernelLaunch.hpp:654 Traceback functionality not available [cgpu16:55192] *** Process received signal *** [cgpu16:55192] Signal: Aborted (6) [cgpu16:55192] Signal code: (-6) [cgpu16:55192] [ 0] /lib64/libpthread.so.0(+0x12360)[0x2aab12445360] [cgpu16:55192] [ 1] /lib64/libc.so.6(gsignal+0x110)[0x2aab12687160] [cgpu16:55192] [ 2] /lib64/libc.so.6(abort+0x151)[0x2aab12688741] [cgpu16:55192] [ 3] /usr/common/software/sles15_cgpu/gcc/8.3.0/lib64/libstdc++.so.6(+0x93e83)[0x2aab10cb0e83] [cgpu16:55192] [ 4] /usr/common/software/sles15_cgpu/gcc/8.3.0/lib64/libstdc++.so.6(+0x99de6)[0x2aab10cb6de6] [cgpu16:55192] [ 5] /usr/common/software/sles15_cgpu/gcc/8.3.0/lib64/libstdc++.so.6(+0x99e21)[0x2aab10cb6e21] [cgpu16:55192] [ 6] /usr/common/software/sles15_cgpu/gcc/8.3.0/lib64/libstdc++.so.6(+0x9a053)[0x2aab10cb7053] [cgpu16:55192] [ 7] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libkokkoscore.so.3.4(+0x26a7f)[0x2aaabbcb3a7f] [cgpu16:55192] [ 8] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libkokkoscore.so.3.4(_ZN6Kokkos4Impl25cuda_internal_error_throwE9cudaErrorPKcS3_i+0x29d)[0x2aaabbcdab9d] [cgpu16:55192] [ 9] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libkokkoskernels.so(_ZN10KokkosBlas4Impl16V_Update_GenericIN6Kokkos4ViewIPKdJNS2_10LayoutLeftENS2_6DeviceINS2_4CudaENS2_9CudaSpaceEEENS2_12MemoryTraitsILj1EEEEEESD_NS3_IPdJS6_SA_SC_EEEiEEvRKNT_20non_const_value_typeERKSG_RKNT0_20non_const_value_typeERKSM_RKNT1_20non_const_value_typeERKSS_iii+0x3357)[0x2aaaae7108a7] [cgpu16:55192] [10] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libkokkoskernels.so(_ZN10KokkosBlas4Impl6UpdateIN6Kokkos4ViewIPKdJNS2_10LayoutLeftENS2_6DeviceINS2_4CudaENS2_9CudaSpaceEEENS2_12MemoryTraitsILj1EEEEEESD_NS3_IPdJS6_SA_SC_EEELi1ELb0ELb1EE6updateERS4_RKSD_SH_SJ_SH_RKSF_+0xc1)[0x2aaaae7171a1] [cgpu16:55192] [11] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libpetsc.so.3.015(_ZN10KokkosBlas6updateIN6Kokkos4ViewIPKdJNS1_9CudaSpaceEEEES6_NS2_IPdJS5_EEEEEvRKNT_20non_const_value_typeERKS9_RKNT0_20non_const_value_typeERKSF_RKNT1_20non_const_value_typeERKSL_+0x271)[0x2aaaab76d781] [cgpu16:55192] [12] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libpetsc.so.3.015(+0xa9333b)[0x2aaaab76633b] [cgpu16:55192] [13] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libpetsc.so.3.015(VecAXPBYPCZ+0x261)[0x2aaaab0b03c1] [cgpu16:55192] [14] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libpetsc.so.3.015(+0x155144e)[0x2aaaac22444e] [cgpu16:55192] [15] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libpetsc.so.3.015(SNESTSFormFunction+0xa)[0x2aaaac1c9c1a] [cgpu16:55192] [16] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libpetsc.so.3.015(SNESComputeFunction+0xf5)[0x2aaaac138675] [cgpu16:55192] [17] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libpetsc.so.3.015(+0x14ac85e)[0x2aaaac17f85e] [cgpu16:55192] [18] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libpetsc.so.3.015(SNESSolve+0x821)[0x2aaaac146651] [cgpu16:55192] [19] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libpetsc.so.3.015(+0x155526c)[0x2aaaac22826c] [cgpu16:55192] [20] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libpetsc.so.3.015(TSStep+0x1f5)[0x2aaaac1d6a05] [cgpu16:55192] [21] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libpetsc.so.3.015(TSSolve+0x6a5)[0x2aaaac1dc455] [cgpu16:55192] [22] ../ex2-kok[0x4033eb] [cgpu16:55192] [23] /lib64/libc.so.6(__libc_start_main+0xea)[0x2aab12671f8a] [cgpu16:55192] [24] ../ex2-kok[0x404aaa] [cgpu16:55192] *** End of error message *** /global/homes/m/madams/mps-wrapper.sh: line 30: 55192 Aborted "$@" 0 stopping nvidia-cuda-mps-control on cgpu16
Could still be a gencode arch issue. Is it possible that Kokkos was built with the 80 arch and when you reran configure with 70 it did not rebuild Kokkos because it didn't know it needed to? Sorry, but this may require another rm -rf arch* and running ./configure again. https://docs.nvidia.com/cuda/cuda-runtime-api/group__CUDART__TYPES.html#grou... <https://docs.nvidia.com/cuda/cuda-runtime-api/group__CUDART__TYPES.html#group__CUDART__TYPES_1gg3f51e3575c2178246db0a94a430e0038b6af535e7e53d3f21e2437e8977b8c2e> cudaErrorInvalidDeviceFunction = 98 The requested device function does not exist or is not compiled for the proper device architecture.
On May 16, 2021, at 9:09 PM, Mark Adams <[email protected]> wrote:
I now get this error. A blas error from VecAXPBYPCZ ... Any ideas?
terminate called after throwing an instance of 'std::runtime_error' what(): cudaFuncGetAttributes(&attr_tmp, base_t::get_kernel_func()) error( cudaErrorInvalidDeviceFunction): invalid device function /global/u2/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/include/Cuda/Kokkos_Cuda_KernelLaunch.hpp:654 Traceback functionality not available
[cgpu16:55192] *** Process received signal *** [cgpu16:55192] Signal: Aborted (6) [cgpu16:55192] Signal code: (-6) [cgpu16:55192] [ 0] /lib64/libpthread.so.0(+0x12360)[0x2aab12445360] [cgpu16:55192] [ 1] /lib64/libc.so.6(gsignal+0x110)[0x2aab12687160] [cgpu16:55192] [ 2] /lib64/libc.so.6(abort+0x151)[0x2aab12688741] [cgpu16:55192] [ 3] /usr/common/software/sles15_cgpu/gcc/8.3.0/lib64/libstdc++.so.6(+0x93e83)[0x2aab10cb0e83] [cgpu16:55192] [ 4] /usr/common/software/sles15_cgpu/gcc/8.3.0/lib64/libstdc++.so.6(+0x99de6)[0x2aab10cb6de6] [cgpu16:55192] [ 5] /usr/common/software/sles15_cgpu/gcc/8.3.0/lib64/libstdc++.so.6(+0x99e21)[0x2aab10cb6e21] [cgpu16:55192] [ 6] /usr/common/software/sles15_cgpu/gcc/8.3.0/lib64/libstdc++.so.6(+0x9a053)[0x2aab10cb7053] [cgpu16:55192] [ 7] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libkokkoscore.so.3.4(+0x26a7f)[0x2aaabbcb3a7f] [cgpu16:55192] [ 8] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libkokkoscore.so.3.4(_ZN6Kokkos4Impl25cuda_internal_error_throwE9cudaErrorPKcS3_i+0x29d)[0x2aaabbcdab9d] [cgpu16:55192] [ 9] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libkokkoskernels.so(_ZN10KokkosBlas4Impl16V_Update_GenericIN6Kokkos4ViewIPKdJNS2_10LayoutLeftENS2_6DeviceINS2_4CudaENS2_9CudaSpaceEEENS2_12MemoryTraitsILj1EEEEEESD_NS3_IPdJS6_SA_SC_EEEiEEvRKNT_20non_const_value_typeERKSG_RKNT0_20non_const_value_typeERKSM_RKNT1_20non_const_value_typeERKSS_iii+0x3357)[0x2aaaae7108a7] [cgpu16:55192] [10] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libkokkoskernels.so(_ZN10KokkosBlas4Impl6UpdateIN6Kokkos4ViewIPKdJNS2_10LayoutLeftENS2_6DeviceINS2_4CudaENS2_9CudaSpaceEEENS2_12MemoryTraitsILj1EEEEEESD_NS3_IPdJS6_SA_SC_EEELi1ELb0ELb1EE6updateERS4_RKSD_SH_SJ_SH_RKSF_+0xc1)[0x2aaaae7171a1] [cgpu16:55192] [11] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libpetsc.so.3.015(_ZN10KokkosBlas6updateIN6Kokkos4ViewIPKdJNS1_9CudaSpaceEEEES6_NS2_IPdJS5_EEEEEvRKNT_20non_const_value_typeERKS9_RKNT0_20non_const_value_typeERKSF_RKNT1_20non_const_value_typeERKSL_+0x271)[0x2aaaab76d781] [cgpu16:55192] [12] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libpetsc.so.3.015(+0xa9333b)[0x2aaaab76633b] [cgpu16:55192] [13] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libpetsc.so.3.015(VecAXPBYPCZ+0x261)[0x2aaaab0b03c1] [cgpu16:55192] [14] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libpetsc.so.3.015(+0x155144e)[0x2aaaac22444e] [cgpu16:55192] [15] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libpetsc.so.3.015(SNESTSFormFunction+0xa)[0x2aaaac1c9c1a] [cgpu16:55192] [16] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libpetsc.so.3.015(SNESComputeFunction+0xf5)[0x2aaaac138675] [cgpu16:55192] [17] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libpetsc.so.3.015(+0x14ac85e)[0x2aaaac17f85e] [cgpu16:55192] [18] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libpetsc.so.3.015(SNESSolve+0x821)[0x2aaaac146651] [cgpu16:55192] [19] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libpetsc.so.3.015(+0x155526c)[0x2aaaac22826c] [cgpu16:55192] [20] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libpetsc.so.3.015(TSStep+0x1f5)[0x2aaaac1d6a05] [cgpu16:55192] [21] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libpetsc.so.3.015(TSSolve+0x6a5)[0x2aaaac1dc455] [cgpu16:55192] [22] ../ex2-kok[0x4033eb] [cgpu16:55192] [23] /lib64/libc.so.6(__libc_start_main+0xea)[0x2aab12671f8a] [cgpu16:55192] [24] ../ex2-kok[0x404aaa] [cgpu16:55192] *** End of error message *** /global/homes/m/madams/mps-wrapper.sh: line 30: 55192 Aborted "$@" 0 stopping nvidia-cuda-mps-control on cgpu16
I thought I did a clean make but I made a clean one now and it seems to be working now. Also, I am trying to fix this error message that I get on Cori with 'make check'. I set mpiexec='srun -G 2 -c 20' and get an interactive shell with these parameters, but I get error messages on Kokkos: Possible error running C/C++ src/snes/tutorials/ex19 with 2 MPI processes See http://www.mcs.anl.gov/petsc/documentation/faq.html *srun: error: Unable to create step for job 1923618: More processors requested than permitted*C/C++ example src/snes/tutorials/ex19 run successfully with cuda gmake[3]: [makefile:102: runex3k_kokkos] Error 1 (ignored) 1,25c1 < atol=1e-50, rtol=1e-08, stol=1e-08, maxit=50, maxf=10000 < Vec Object: Exact Solution 2 MPI processes < type: mpikokkos < Process [0] < 0. < 0.015625 < 0.125 < Process [1] < 0.421875 < 1. < Vec Object: Forcing function 2 MPI processes < type: mpikokkos < Process [0] < 1e-72 < 1.50024 < 3.01563 < Process [1] < 4.67798 < 7. < 0 SNES Function norm 5.414682427127e+00 < 1 SNES Function norm 2.952582418265e-01 < 2 SNES Function norm 4.502293658739e-04 < 3 SNES Function norm 1.389665806646e-09 < Number of SNES iterations = 3 < Norm of error 1.49752e-10 Iterations 3 --- *> srun: error: Unable to create step for job 1923618: More processors requested than permitted*/global/homes/m/madams/petsc/src/snes/tutorials Possible problem with ex3k running with kokkos-kernels, diffs above ========================================= Fortran example src/snes/tutorials/ex5f run successfully with 1 MPI process Completed test examples On Sun, May 16, 2021 at 11:14 PM Barry Smith <[email protected]> wrote:
Could still be a gencode arch issue. Is it possible that Kokkos was built with the 80 arch and when you reran configure with 70 it did not rebuild Kokkos because it didn't know it needed to?
Sorry, but this may require another rm -rf arch* and running ./configure again.
https://docs.nvidia.com/cuda/cuda-runtime-api/group__CUDART__TYPES.html#grou...
cudaErrorInvalidDeviceFunction = 98The requested device function does not exist or is not compiled for the proper device architecture.
On May 16, 2021, at 9:09 PM, Mark Adams <[email protected]> wrote:
I now get this error. A blas error from VecAXPBYPCZ ... Any ideas?
terminate called after throwing an instance of 'std::runtime_error' what(): cudaFuncGetAttributes(&attr_tmp, base_t::get_kernel_func()) error( cudaErrorInvalidDeviceFunction): invalid device function /global/u2/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/include/Cuda/Kokkos_Cuda_KernelLaunch.hpp:654 Traceback functionality not available
[cgpu16:55192] *** Process received signal *** [cgpu16:55192] Signal: Aborted (6) [cgpu16:55192] Signal code: (-6) [cgpu16:55192] [ 0] /lib64/libpthread.so.0(+0x12360)[0x2aab12445360] [cgpu16:55192] [ 1] /lib64/libc.so.6(gsignal+0x110)[0x2aab12687160] [cgpu16:55192] [ 2] /lib64/libc.so.6(abort+0x151)[0x2aab12688741] [cgpu16:55192] [ 3] /usr/common/software/sles15_cgpu/gcc/8.3.0/lib64/libstdc++.so.6(+0x93e83)[0x2aab10cb0e83] [cgpu16:55192] [ 4] /usr/common/software/sles15_cgpu/gcc/8.3.0/lib64/libstdc++.so.6(+0x99de6)[0x2aab10cb6de6] [cgpu16:55192] [ 5] /usr/common/software/sles15_cgpu/gcc/8.3.0/lib64/libstdc++.so.6(+0x99e21)[0x2aab10cb6e21] [cgpu16:55192] [ 6] /usr/common/software/sles15_cgpu/gcc/8.3.0/lib64/libstdc++.so.6(+0x9a053)[0x2aab10cb7053] [cgpu16:55192] [ 7] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libkokkoscore.so.3.4(+0x26a7f)[0x2aaabbcb3a7f] [cgpu16:55192] [ 8] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libkokkoscore.so.3.4(_ZN6Kokkos4Impl25cuda_internal_error_throwE9cudaErrorPKcS3_i+0x29d)[0x2aaabbcdab9d] [cgpu16:55192] [ 9] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libkokkoskernels.so(_ZN10KokkosBlas4Impl16V_Update_GenericIN6Kokkos4ViewIPKdJNS2_10LayoutLeftENS2_6DeviceINS2_4CudaENS2_9CudaSpaceEEENS2_12MemoryTraitsILj1EEEEEESD_NS3_IPdJS6_SA_SC_EEEiEEvRKNT_20non_const_value_typeERKSG_RKNT0_20non_const_value_typeERKSM_RKNT1_20non_const_value_typeERKSS_iii+0x3357)[0x2aaaae7108a7] [cgpu16:55192] [10] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libkokkoskernels.so(_ZN10KokkosBlas4Impl6UpdateIN6Kokkos4ViewIPKdJNS2_10LayoutLeftENS2_6DeviceINS2_4CudaENS2_9CudaSpaceEEENS2_12MemoryTraitsILj1EEEEEESD_NS3_IPdJS6_SA_SC_EEELi1ELb0ELb1EE6updateERS4_RKSD_SH_SJ_SH_RKSF_+0xc1)[0x2aaaae7171a1] [cgpu16:55192] [11] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libpetsc.so.3.015(_ZN10KokkosBlas6updateIN6Kokkos4ViewIPKdJNS1_9CudaSpaceEEEES6_NS2_IPdJS5_EEEEEvRKNT_20non_const_value_typeERKS9_RKNT0_20non_const_value_typeERKSF_RKNT1_20non_const_value_typeERKSL_+0x271)[0x2aaaab76d781] [cgpu16:55192] [12] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libpetsc.so.3.015(+0xa9333b)[0x2aaaab76633b] [cgpu16:55192] [13] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libpetsc.so.3.015(VecAXPBYPCZ+0x261)[0x2aaaab0b03c1] [cgpu16:55192] [14] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libpetsc.so.3.015(+0x155144e)[0x2aaaac22444e] [cgpu16:55192] [15] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libpetsc.so.3.015(SNESTSFormFunction+0xa)[0x2aaaac1c9c1a] [cgpu16:55192] [16] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libpetsc.so.3.015(SNESComputeFunction+0xf5)[0x2aaaac138675] [cgpu16:55192] [17] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libpetsc.so.3.015(+0x14ac85e)[0x2aaaac17f85e] [cgpu16:55192] [18] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libpetsc.so.3.015(SNESSolve+0x821)[0x2aaaac146651] [cgpu16:55192] [19] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libpetsc.so.3.015(+0x155526c)[0x2aaaac22826c] [cgpu16:55192] [20] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libpetsc.so.3.015(TSStep+0x1f5)[0x2aaaac1d6a05] [cgpu16:55192] [21] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libpetsc.so.3.015(TSSolve+0x6a5)[0x2aaaac1dc455] [cgpu16:55192] [22] ../ex2-kok[0x4033eb] [cgpu16:55192] [23] /lib64/libc.so.6(__libc_start_main+0xea)[0x2aab12671f8a] [cgpu16:55192] [24] ../ex2-kok[0x404aaa] [cgpu16:55192] *** End of error message *** /global/homes/m/madams/mps-wrapper.sh: line 30: 55192 Aborted "$@" 0 stopping nvidia-cuda-mps-control on cgpu16
Damn, I am getting this problem on Summit and did a clean configure. I removed the Kokkos arch=70 line and added '--with-cudac-gencodearch=70', Any ideas? < Number of SNES iterations = 2 ---
Kokkos::Cuda::initialize ERROR: likely mismatch of architecture [h50n11:35759] *** Process received signal *** [h50n11:35759] Signal: Aborted (6) [h50n11:35759] Signal code: (-6) [h50n11:35759] [ 0] [0x2000000504d8] [h50n11:35759] [ 1] /lib64/libc.so.6(abort+0x2b4)[0x200032322094] [h50n11:35759] [ 2] /gpfs/alpine/csc314/scratch/adams/petsc/arch-summit-opt-gnu-kokkos-notpl-cuda10/lib/libkokkoscore.so.3.4(_ZN6Kokkos4Impl10host_abortEPKc+0x58)[0x20000f944558] [h50n11:35759] [ 3] /gpfs/alpine/csc314/scratch/adams/petsc/arch-summit-opt-gnu-kokkos-notpl-cuda10/lib/libkokkoscore.so.3.4(_ZN6Kokkos4Impl12CudaInternal10initializeEiP11CUstream_st+0xe60)[0x20000f95c210] [h50n11:35759] [ 4] /gpfs/alpine/csc314/scratch/adams/petsc/arch-summit-opt-gnu-kokkos-notpl-cuda10/lib/libkokkoscore.so.3.4(_ZN6Kokkos4Cuda15impl_initializeENS0_12SelectDeviceEm+0x30)[0x20000f95c2b0] [h50n11:35759] [ 5] /gpfs/alpine/csc314/scratch/adams/petsc/arch-summit-opt-gnu-kokkos-notpl-cuda10/lib/libkokkoscore.so.3.4(_ZN6Kokkos4Impl20CudaSpaceInitializer10initializeERKNS_13InitArgumentsE+0x34)[0x20000f95c314] [h50n11:35759] [ 6] /gpfs/alpine/csc314/scratch/adams/petsc/arch-summit-opt-gnu-kokkos-notpl-cuda10/lib/libkokkoscore.so.3.4(_ZN6Kokkos4Impl16ExecSpaceManager17initialize_spacesERKNS_13InitArgumentsE+0x60)[0x20000f926aa0] [h50n11:35759] [ 7] /gpfs/alpine/csc314/scratch/adams/petsc/arch-summit-opt-gnu-kokkos-notpl-cuda10/lib/libkokkoscore.so.3.4(_ZN6Kokkos4Impl70_GLOBAL__N__46_tmpxft_0001ef6b_00000000_6_Kokkos_Core_cpp1_ii_889c95a619initialize_backendsERKNS_13InitArgumentsE+0x2c)[0x20000f926dac] [h50n11:35759] [ 8] /gpfs/alpine/csc314/scratch/adams/petsc/arch-summit-opt-gnu-kokkos-notpl-cuda10/lib/libkokkoscore.so.3.4(_ZN6Kokkos4Impl70_GLOBAL__N__46_tmpxft_0001ef6b_00000000_6_Kokkos_Core_cpp1_ii_889c95a619initialize_internalERKNS_13InitArgumentsE+0x2c)[0x20000f92b73c] [h50n11:35759] [ 9] /gpfs/alpine/csc314/scratch/adams/petsc/arch-summit-opt-gnu-kokkos-notpl-cuda10/lib/libkokkoscore.so.3.4(_ZN6Kokkos10initializeENS_13InitArgumentsE+0x2c)[0x20000f92d08c] [h50n11:35759] [10] /gpfs/alpine/csc314/scratch/adams/petsc/arch-summit-opt-gnu-kokkos-notpl-cuda10/lib/libpetsc.so.3.015(PetscKokkosInitializeCheck+0x1f4)[0x200000343424] [h50n11:35759] [11] /gpfs/alpine/csc314/scratch/adams/petsc/arch-summit-opt-gnu-kokkos-notpl-cuda10/lib/libpetsc.so.3.015(+0x68b3dc)[0x20000077b3dc] [h50n11:35759] [12] /gpfs/alpine/csc314/scratch/adams/petsc/arch-summit-opt-gnu-kokkos-notpl-cuda10/lib/libpetsc.so.3.015(+0x3119b4)[0x2000004019b4] [h50n11:35759] [13] /gpfs/alpine/csc314/scratch/adams/petsc/arch-summit-opt-gnu-kokkos-notpl-cuda10/lib/libpetsc.so.3.015(+0x306790)[0x2000003f6790] [h50n11:35759] [14] /gpfs/alpine/csc314/scratch/adams/petsc/arch-summit-opt-gnu-kokkos-notpl-cuda10/lib/libpetsc.so.3.015(+0x274d24)[0x200000364d24] [h50n11:35759] [15] /gpfs/alpine/csc314/scratch/adams/petsc/arch-summit-opt-gnu-kokkos-notpl-cuda10/lib/libpetsc.so.3.015(PetscSFBcastWithMemTypeBegin+0xd4)[0x200000412504] [h50n11:35759] [16] /gpfs/alpine/csc314/scratch/adams/petsc/arch-summit-opt-gnu-kokkos-notpl-cuda10/lib/libpetsc.so.3.015(+0x329f9c)[0x200000419f9c] [h50n11:35759] [17] /gpfs/alpine/csc314/scratch/adams/petsc/arch-summit-opt-gnu-kokkos-notpl-cuda10/lib/libpetsc.so.3.015(VecScatterBegin+0x9c)[0x20000041fa8c] [h50n11:35759] [18] /gpfs/alpine/csc314/scratch/adams/petsc/arch-summit-opt-gnu-kokkos-notpl-cuda10/lib/libpetsc.so.3.015(DMGlobalToLocalBegin_DA+0x30)[0x2000010ef560] [h50n11:35759] [19] /gpfs/alpine/csc314/scratch/adams/petsc/arch-summit-opt-gnu-kokkos-notpl-cuda10/lib/libpetsc.so.3.015(DMGlobalToLocalBegin+0x290)[0x2000013314b0] [h50n11:35759] [20] /gpfs/alpine/csc314/scratch/adams/petsc/arch-summit-opt-gnu-kokkos-notpl-cuda10/lib/libpetsc.so.3.015(+0x1513c10)[0x200001603c10] [h50n11:35759] [21] /gpfs/alpine/csc314/scratch/adams/petsc/arch-summit-opt-gnu-kokkos-notpl-cuda10/lib/libpetsc.so.3.015(SNESComputeFunction+0x164)[0x200001625584] [h50n11:35759] [22] /gpfs/alpine/csc314/scratch/adams/petsc/arch-summit-opt-gnu-kokkos-notpl-cuda10/lib/libpetsc.so.3.015(+0x15830f4)[0x2000016730f4] [h50n11:35759] [23] /gpfs/alpine/csc314/scratch/adams/petsc/arch-summit-opt-gnu-kokkos-notpl-cuda10/lib/libpetsc.so.3.015(SNESSolve+0x814)[0x200001634c44] [h50n11:35759] [24] ./ex19[0x10001a70] [h50n11:35759] [25] /lib64/libc.so.6(+0x25200)[0x200032305200] [h50n11:35759] [26] /lib64/libc.so.6(__libc_start_main+0xc4)[0x2000323053f4] [h50n11:35759] *** End of error message *** ERROR: One or more process (first noticed rank 0) terminated with signal 6 (core dumped) /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials
On Mon, May 17, 2021 at 8:24 AM Mark Adams <[email protected]> wrote:
I thought I did a clean make but I made a clean one now and it seems to be working now.
Also, I am trying to fix this error message that I get on Cori with 'make check'. I set mpiexec='srun -G 2 -c 20' and get an interactive shell with these parameters, but I get error messages on Kokkos:
Possible error running C/C++ src/snes/tutorials/ex19 with 2 MPI processes See http://www.mcs.anl.gov/petsc/documentation/faq.html
*srun: error: Unable to create step for job 1923618: More processors requested than permitted*C/C++ example src/snes/tutorials/ex19 run successfully with cuda gmake[3]: [makefile:102: runex3k_kokkos] Error 1 (ignored) 1,25c1 < atol=1e-50, rtol=1e-08, stol=1e-08, maxit=50, maxf=10000 < Vec Object: Exact Solution 2 MPI processes < type: mpikokkos < Process [0] < 0. < 0.015625 < 0.125 < Process [1] < 0.421875 < 1. < Vec Object: Forcing function 2 MPI processes < type: mpikokkos < Process [0] < 1e-72 < 1.50024 < 3.01563 < Process [1] < 4.67798 < 7. < 0 SNES Function norm 5.414682427127e+00 < 1 SNES Function norm 2.952582418265e-01 < 2 SNES Function norm 4.502293658739e-04 < 3 SNES Function norm 1.389665806646e-09 < Number of SNES iterations = 3 < Norm of error 1.49752e-10 Iterations 3 ---
*> srun: error: Unable to create step for job 1923618: More processors requested than permitted*/global/homes/m/madams/petsc/src/snes/tutorials Possible problem with ex3k running with kokkos-kernels, diffs above ========================================= Fortran example src/snes/tutorials/ex5f run successfully with 1 MPI process Completed test examples
On Sun, May 16, 2021 at 11:14 PM Barry Smith <[email protected]> wrote:
Could still be a gencode arch issue. Is it possible that Kokkos was built with the 80 arch and when you reran configure with 70 it did not rebuild Kokkos because it didn't know it needed to?
Sorry, but this may require another rm -rf arch* and running ./configure again.
https://docs.nvidia.com/cuda/cuda-runtime-api/group__CUDART__TYPES.html#grou...
cudaErrorInvalidDeviceFunction = 98The requested device function does not exist or is not compiled for the proper device architecture.
On May 16, 2021, at 9:09 PM, Mark Adams <[email protected]> wrote:
I now get this error. A blas error from VecAXPBYPCZ ... Any ideas?
terminate called after throwing an instance of 'std::runtime_error' what(): cudaFuncGetAttributes(&attr_tmp, base_t::get_kernel_func()) error( cudaErrorInvalidDeviceFunction): invalid device function /global/u2/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/include/Cuda/Kokkos_Cuda_KernelLaunch.hpp:654 Traceback functionality not available
[cgpu16:55192] *** Process received signal *** [cgpu16:55192] Signal: Aborted (6) [cgpu16:55192] Signal code: (-6) [cgpu16:55192] [ 0] /lib64/libpthread.so.0(+0x12360)[0x2aab12445360] [cgpu16:55192] [ 1] /lib64/libc.so.6(gsignal+0x110)[0x2aab12687160] [cgpu16:55192] [ 2] /lib64/libc.so.6(abort+0x151)[0x2aab12688741] [cgpu16:55192] [ 3] /usr/common/software/sles15_cgpu/gcc/8.3.0/lib64/libstdc++.so.6(+0x93e83)[0x2aab10cb0e83] [cgpu16:55192] [ 4] /usr/common/software/sles15_cgpu/gcc/8.3.0/lib64/libstdc++.so.6(+0x99de6)[0x2aab10cb6de6] [cgpu16:55192] [ 5] /usr/common/software/sles15_cgpu/gcc/8.3.0/lib64/libstdc++.so.6(+0x99e21)[0x2aab10cb6e21] [cgpu16:55192] [ 6] /usr/common/software/sles15_cgpu/gcc/8.3.0/lib64/libstdc++.so.6(+0x9a053)[0x2aab10cb7053] [cgpu16:55192] [ 7] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libkokkoscore.so.3.4(+0x26a7f)[0x2aaabbcb3a7f] [cgpu16:55192] [ 8] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libkokkoscore.so.3.4(_ZN6Kokkos4Impl25cuda_internal_error_throwE9cudaErrorPKcS3_i+0x29d)[0x2aaabbcdab9d] [cgpu16:55192] [ 9] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libkokkoskernels.so(_ZN10KokkosBlas4Impl16V_Update_GenericIN6Kokkos4ViewIPKdJNS2_10LayoutLeftENS2_6DeviceINS2_4CudaENS2_9CudaSpaceEEENS2_12MemoryTraitsILj1EEEEEESD_NS3_IPdJS6_SA_SC_EEEiEEvRKNT_20non_const_value_typeERKSG_RKNT0_20non_const_value_typeERKSM_RKNT1_20non_const_value_typeERKSS_iii+0x3357)[0x2aaaae7108a7] [cgpu16:55192] [10] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libkokkoskernels.so(_ZN10KokkosBlas4Impl6UpdateIN6Kokkos4ViewIPKdJNS2_10LayoutLeftENS2_6DeviceINS2_4CudaENS2_9CudaSpaceEEENS2_12MemoryTraitsILj1EEEEEESD_NS3_IPdJS6_SA_SC_EEELi1ELb0ELb1EE6updateERS4_RKSD_SH_SJ_SH_RKSF_+0xc1)[0x2aaaae7171a1] [cgpu16:55192] [11] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libpetsc.so.3.015(_ZN10KokkosBlas6updateIN6Kokkos4ViewIPKdJNS1_9CudaSpaceEEEES6_NS2_IPdJS5_EEEEEvRKNT_20non_const_value_typeERKS9_RKNT0_20non_const_value_typeERKSF_RKNT1_20non_const_value_typeERKSL_+0x271)[0x2aaaab76d781] [cgpu16:55192] [12] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libpetsc.so.3.015(+0xa9333b)[0x2aaaab76633b] [cgpu16:55192] [13] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libpetsc.so.3.015(VecAXPBYPCZ+0x261)[0x2aaaab0b03c1] [cgpu16:55192] [14] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libpetsc.so.3.015(+0x155144e)[0x2aaaac22444e] [cgpu16:55192] [15] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libpetsc.so.3.015(SNESTSFormFunction+0xa)[0x2aaaac1c9c1a] [cgpu16:55192] [16] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libpetsc.so.3.015(SNESComputeFunction+0xf5)[0x2aaaac138675] [cgpu16:55192] [17] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libpetsc.so.3.015(+0x14ac85e)[0x2aaaac17f85e] [cgpu16:55192] [18] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libpetsc.so.3.015(SNESSolve+0x821)[0x2aaaac146651] [cgpu16:55192] [19] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libpetsc.so.3.015(+0x155526c)[0x2aaaac22826c] [cgpu16:55192] [20] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libpetsc.so.3.015(TSStep+0x1f5)[0x2aaaac1d6a05] [cgpu16:55192] [21] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libpetsc.so.3.015(TSSolve+0x6a5)[0x2aaaac1dc455] [cgpu16:55192] [22] ../ex2-kok[0x4033eb] [cgpu16:55192] [23] /lib64/libc.so.6(__libc_start_main+0xea)[0x2aab12671f8a] [cgpu16:55192] [24] ../ex2-kok[0x404aaa] [cgpu16:55192] *** End of error message *** /global/homes/m/madams/mps-wrapper.sh: line 30: 55192 Aborted "$@" 0 stopping nvidia-cuda-mps-control on cgpu16
* '--with-cuda-gencodearch=70',* --Junchao Zhang On Tue, May 18, 2021 at 6:29 AM Mark Adams <[email protected]> wrote:
Damn, I am getting this problem on Summit and did a clean configure. I removed the Kokkos arch=70 line and added '--with-cudac-gencodearch=70',
Any ideas?
< Number of SNES iterations = 2 ---
Kokkos::Cuda::initialize ERROR: likely mismatch of architecture [h50n11:35759] *** Process received signal *** [h50n11:35759] Signal: Aborted (6) [h50n11:35759] Signal code: (-6) [h50n11:35759] [ 0] [0x2000000504d8] [h50n11:35759] [ 1] /lib64/libc.so.6(abort+0x2b4)[0x200032322094] [h50n11:35759] [ 2] /gpfs/alpine/csc314/scratch/adams/petsc/arch-summit-opt-gnu-kokkos-notpl-cuda10/lib/libkokkoscore.so.3.4(_ZN6Kokkos4Impl10host_abortEPKc+0x58)[0x20000f944558] [h50n11:35759] [ 3] /gpfs/alpine/csc314/scratch/adams/petsc/arch-summit-opt-gnu-kokkos-notpl-cuda10/lib/libkokkoscore.so.3.4(_ZN6Kokkos4Impl12CudaInternal10initializeEiP11CUstream_st+0xe60)[0x20000f95c210] [h50n11:35759] [ 4] /gpfs/alpine/csc314/scratch/adams/petsc/arch-summit-opt-gnu-kokkos-notpl-cuda10/lib/libkokkoscore.so.3.4(_ZN6Kokkos4Cuda15impl_initializeENS0_12SelectDeviceEm+0x30)[0x20000f95c2b0] [h50n11:35759] [ 5] /gpfs/alpine/csc314/scratch/adams/petsc/arch-summit-opt-gnu-kokkos-notpl-cuda10/lib/libkokkoscore.so.3.4(_ZN6Kokkos4Impl20CudaSpaceInitializer10initializeERKNS_13InitArgumentsE+0x34)[0x20000f95c314] [h50n11:35759] [ 6] /gpfs/alpine/csc314/scratch/adams/petsc/arch-summit-opt-gnu-kokkos-notpl-cuda10/lib/libkokkoscore.so.3.4(_ZN6Kokkos4Impl16ExecSpaceManager17initialize_spacesERKNS_13InitArgumentsE+0x60)[0x20000f926aa0] [h50n11:35759] [ 7] /gpfs/alpine/csc314/scratch/adams/petsc/arch-summit-opt-gnu-kokkos-notpl-cuda10/lib/libkokkoscore.so.3.4(_ZN6Kokkos4Impl70_GLOBAL__N__46_tmpxft_0001ef6b_00000000_6_Kokkos_Core_cpp1_ii_889c95a619initialize_backendsERKNS_13InitArgumentsE+0x2c)[0x20000f926dac] [h50n11:35759] [ 8] /gpfs/alpine/csc314/scratch/adams/petsc/arch-summit-opt-gnu-kokkos-notpl-cuda10/lib/libkokkoscore.so.3.4(_ZN6Kokkos4Impl70_GLOBAL__N__46_tmpxft_0001ef6b_00000000_6_Kokkos_Core_cpp1_ii_889c95a619initialize_internalERKNS_13InitArgumentsE+0x2c)[0x20000f92b73c] [h50n11:35759] [ 9] /gpfs/alpine/csc314/scratch/adams/petsc/arch-summit-opt-gnu-kokkos-notpl-cuda10/lib/libkokkoscore.so.3.4(_ZN6Kokkos10initializeENS_13InitArgumentsE+0x2c)[0x20000f92d08c] [h50n11:35759] [10] /gpfs/alpine/csc314/scratch/adams/petsc/arch-summit-opt-gnu-kokkos-notpl-cuda10/lib/libpetsc.so.3.015(PetscKokkosInitializeCheck+0x1f4)[0x200000343424] [h50n11:35759] [11] /gpfs/alpine/csc314/scratch/adams/petsc/arch-summit-opt-gnu-kokkos-notpl-cuda10/lib/libpetsc.so.3.015(+0x68b3dc)[0x20000077b3dc] [h50n11:35759] [12] /gpfs/alpine/csc314/scratch/adams/petsc/arch-summit-opt-gnu-kokkos-notpl-cuda10/lib/libpetsc.so.3.015(+0x3119b4)[0x2000004019b4] [h50n11:35759] [13] /gpfs/alpine/csc314/scratch/adams/petsc/arch-summit-opt-gnu-kokkos-notpl-cuda10/lib/libpetsc.so.3.015(+0x306790)[0x2000003f6790] [h50n11:35759] [14] /gpfs/alpine/csc314/scratch/adams/petsc/arch-summit-opt-gnu-kokkos-notpl-cuda10/lib/libpetsc.so.3.015(+0x274d24)[0x200000364d24] [h50n11:35759] [15] /gpfs/alpine/csc314/scratch/adams/petsc/arch-summit-opt-gnu-kokkos-notpl-cuda10/lib/libpetsc.so.3.015(PetscSFBcastWithMemTypeBegin+0xd4)[0x200000412504] [h50n11:35759] [16] /gpfs/alpine/csc314/scratch/adams/petsc/arch-summit-opt-gnu-kokkos-notpl-cuda10/lib/libpetsc.so.3.015(+0x329f9c)[0x200000419f9c] [h50n11:35759] [17] /gpfs/alpine/csc314/scratch/adams/petsc/arch-summit-opt-gnu-kokkos-notpl-cuda10/lib/libpetsc.so.3.015(VecScatterBegin+0x9c)[0x20000041fa8c] [h50n11:35759] [18] /gpfs/alpine/csc314/scratch/adams/petsc/arch-summit-opt-gnu-kokkos-notpl-cuda10/lib/libpetsc.so.3.015(DMGlobalToLocalBegin_DA+0x30)[0x2000010ef560] [h50n11:35759] [19] /gpfs/alpine/csc314/scratch/adams/petsc/arch-summit-opt-gnu-kokkos-notpl-cuda10/lib/libpetsc.so.3.015(DMGlobalToLocalBegin+0x290)[0x2000013314b0] [h50n11:35759] [20] /gpfs/alpine/csc314/scratch/adams/petsc/arch-summit-opt-gnu-kokkos-notpl-cuda10/lib/libpetsc.so.3.015(+0x1513c10)[0x200001603c10] [h50n11:35759] [21] /gpfs/alpine/csc314/scratch/adams/petsc/arch-summit-opt-gnu-kokkos-notpl-cuda10/lib/libpetsc.so.3.015(SNESComputeFunction+0x164)[0x200001625584] [h50n11:35759] [22] /gpfs/alpine/csc314/scratch/adams/petsc/arch-summit-opt-gnu-kokkos-notpl-cuda10/lib/libpetsc.so.3.015(+0x15830f4)[0x2000016730f4] [h50n11:35759] [23] /gpfs/alpine/csc314/scratch/adams/petsc/arch-summit-opt-gnu-kokkos-notpl-cuda10/lib/libpetsc.so.3.015(SNESSolve+0x814)[0x200001634c44] [h50n11:35759] [24] ./ex19[0x10001a70] [h50n11:35759] [25] /lib64/libc.so.6(+0x25200)[0x200032305200] [h50n11:35759] [26] /lib64/libc.so.6(__libc_start_main+0xc4)[0x2000323053f4] [h50n11:35759] *** End of error message *** ERROR: One or more process (first noticed rank 0) terminated with signal 6 (core dumped) /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials
On Mon, May 17, 2021 at 8:24 AM Mark Adams <[email protected]> wrote:
I thought I did a clean make but I made a clean one now and it seems to be working now.
Also, I am trying to fix this error message that I get on Cori with 'make check'. I set mpiexec='srun -G 2 -c 20' and get an interactive shell with these parameters, but I get error messages on Kokkos:
Possible error running C/C++ src/snes/tutorials/ex19 with 2 MPI processes See http://www.mcs.anl.gov/petsc/documentation/faq.html
*srun: error: Unable to create step for job 1923618: More processors requested than permitted*C/C++ example src/snes/tutorials/ex19 run successfully with cuda gmake[3]: [makefile:102: runex3k_kokkos] Error 1 (ignored) 1,25c1 < atol=1e-50, rtol=1e-08, stol=1e-08, maxit=50, maxf=10000 < Vec Object: Exact Solution 2 MPI processes < type: mpikokkos < Process [0] < 0. < 0.015625 < 0.125 < Process [1] < 0.421875 < 1. < Vec Object: Forcing function 2 MPI processes < type: mpikokkos < Process [0] < 1e-72 < 1.50024 < 3.01563 < Process [1] < 4.67798 < 7. < 0 SNES Function norm 5.414682427127e+00 < 1 SNES Function norm 2.952582418265e-01 < 2 SNES Function norm 4.502293658739e-04 < 3 SNES Function norm 1.389665806646e-09 < Number of SNES iterations = 3 < Norm of error 1.49752e-10 Iterations 3 ---
*> srun: error: Unable to create step for job 1923618: More processors requested than permitted*/global/homes/m/madams/petsc/src/snes/tutorials Possible problem with ex3k running with kokkos-kernels, diffs above ========================================= Fortran example src/snes/tutorials/ex5f run successfully with 1 MPI process Completed test examples
On Sun, May 16, 2021 at 11:14 PM Barry Smith <[email protected]> wrote:
Could still be a gencode arch issue. Is it possible that Kokkos was built with the 80 arch and when you reran configure with 70 it did not rebuild Kokkos because it didn't know it needed to?
Sorry, but this may require another rm -rf arch* and running ./configure again.
https://docs.nvidia.com/cuda/cuda-runtime-api/group__CUDART__TYPES.html#grou...
cudaErrorInvalidDeviceFunction = 98The requested device function does not exist or is not compiled for the proper device architecture.
On May 16, 2021, at 9:09 PM, Mark Adams <[email protected]> wrote:
I now get this error. A blas error from VecAXPBYPCZ ... Any ideas?
terminate called after throwing an instance of 'std::runtime_error' what(): cudaFuncGetAttributes(&attr_tmp, base_t::get_kernel_func()) error( cudaErrorInvalidDeviceFunction): invalid device function /global/u2/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/include/Cuda/Kokkos_Cuda_KernelLaunch.hpp:654 Traceback functionality not available
[cgpu16:55192] *** Process received signal *** [cgpu16:55192] Signal: Aborted (6) [cgpu16:55192] Signal code: (-6) [cgpu16:55192] [ 0] /lib64/libpthread.so.0(+0x12360)[0x2aab12445360] [cgpu16:55192] [ 1] /lib64/libc.so.6(gsignal+0x110)[0x2aab12687160] [cgpu16:55192] [ 2] /lib64/libc.so.6(abort+0x151)[0x2aab12688741] [cgpu16:55192] [ 3] /usr/common/software/sles15_cgpu/gcc/8.3.0/lib64/libstdc++.so.6(+0x93e83)[0x2aab10cb0e83] [cgpu16:55192] [ 4] /usr/common/software/sles15_cgpu/gcc/8.3.0/lib64/libstdc++.so.6(+0x99de6)[0x2aab10cb6de6] [cgpu16:55192] [ 5] /usr/common/software/sles15_cgpu/gcc/8.3.0/lib64/libstdc++.so.6(+0x99e21)[0x2aab10cb6e21] [cgpu16:55192] [ 6] /usr/common/software/sles15_cgpu/gcc/8.3.0/lib64/libstdc++.so.6(+0x9a053)[0x2aab10cb7053] [cgpu16:55192] [ 7] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libkokkoscore.so.3.4(+0x26a7f)[0x2aaabbcb3a7f] [cgpu16:55192] [ 8] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libkokkoscore.so.3.4(_ZN6Kokkos4Impl25cuda_internal_error_throwE9cudaErrorPKcS3_i+0x29d)[0x2aaabbcdab9d] [cgpu16:55192] [ 9] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libkokkoskernels.so(_ZN10KokkosBlas4Impl16V_Update_GenericIN6Kokkos4ViewIPKdJNS2_10LayoutLeftENS2_6DeviceINS2_4CudaENS2_9CudaSpaceEEENS2_12MemoryTraitsILj1EEEEEESD_NS3_IPdJS6_SA_SC_EEEiEEvRKNT_20non_const_value_typeERKSG_RKNT0_20non_const_value_typeERKSM_RKNT1_20non_const_value_typeERKSS_iii+0x3357)[0x2aaaae7108a7] [cgpu16:55192] [10] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libkokkoskernels.so(_ZN10KokkosBlas4Impl6UpdateIN6Kokkos4ViewIPKdJNS2_10LayoutLeftENS2_6DeviceINS2_4CudaENS2_9CudaSpaceEEENS2_12MemoryTraitsILj1EEEEEESD_NS3_IPdJS6_SA_SC_EEELi1ELb0ELb1EE6updateERS4_RKSD_SH_SJ_SH_RKSF_+0xc1)[0x2aaaae7171a1] [cgpu16:55192] [11] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libpetsc.so.3.015(_ZN10KokkosBlas6updateIN6Kokkos4ViewIPKdJNS1_9CudaSpaceEEEES6_NS2_IPdJS5_EEEEEvRKNT_20non_const_value_typeERKS9_RKNT0_20non_const_value_typeERKSF_RKNT1_20non_const_value_typeERKSL_+0x271)[0x2aaaab76d781] [cgpu16:55192] [12] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libpetsc.so.3.015(+0xa9333b)[0x2aaaab76633b] [cgpu16:55192] [13] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libpetsc.so.3.015(VecAXPBYPCZ+0x261)[0x2aaaab0b03c1] [cgpu16:55192] [14] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libpetsc.so.3.015(+0x155144e)[0x2aaaac22444e] [cgpu16:55192] [15] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libpetsc.so.3.015(SNESTSFormFunction+0xa)[0x2aaaac1c9c1a] [cgpu16:55192] [16] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libpetsc.so.3.015(SNESComputeFunction+0xf5)[0x2aaaac138675] [cgpu16:55192] [17] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libpetsc.so.3.015(+0x14ac85e)[0x2aaaac17f85e] [cgpu16:55192] [18] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libpetsc.so.3.015(SNESSolve+0x821)[0x2aaaac146651] [cgpu16:55192] [19] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libpetsc.so.3.015(+0x155526c)[0x2aaaac22826c] [cgpu16:55192] [20] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libpetsc.so.3.015(TSStep+0x1f5)[0x2aaaac1d6a05] [cgpu16:55192] [21] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libpetsc.so.3.015(TSSolve+0x6a5)[0x2aaaac1dc455] [cgpu16:55192] [22] ../ex2-kok[0x4033eb] [cgpu16:55192] [23] /lib64/libc.so.6(__libc_start_main+0xea)[0x2aab12671f8a] [cgpu16:55192] [24] ../ex2-kok[0x404aaa] [cgpu16:55192] *** End of error message *** /global/homes/m/madams/mps-wrapper.sh: line 30: 55192 Aborted "$@" 0 stopping nvidia-cuda-mps-control on cgpu16
configure prints the information about CUDA at the end of the run, you can check that information to see which was actually used. I have a new MR where PETSc records the gencodearch it was built with and then when your program starts up CUDA it verifies that the hardware supports the gencodearch it was built with. Hopefully this will alleviate difficulties in the future. Of course this won't help when using libraries that use CUDA built externally from PETSc. Barry
On May 18, 2021, at 10:30 AM, Junchao Zhang <[email protected]> wrote:
'--with-cuda-gencodearch=70',
--Junchao Zhang
On Tue, May 18, 2021 at 6:29 AM Mark Adams <[email protected] <mailto:[email protected]>> wrote: Damn, I am getting this problem on Summit and did a clean configure. I removed the Kokkos arch=70 line and added '--with-cudac-gencodearch=70',
Any ideas?
< Number of SNES iterations = 2 ---
Kokkos::Cuda::initialize ERROR: likely mismatch of architecture [h50n11:35759] *** Process received signal *** [h50n11:35759] Signal: Aborted (6) [h50n11:35759] Signal code: (-6) [h50n11:35759] [ 0] [0x2000000504d8] [h50n11:35759] [ 1] /lib64/libc.so.6(abort+0x2b4)[0x200032322094] [h50n11:35759] [ 2] /gpfs/alpine/csc314/scratch/adams/petsc/arch-summit-opt-gnu-kokkos-notpl-cuda10/lib/libkokkoscore.so.3.4(_ZN6Kokkos4Impl10host_abortEPKc+0x58)[0x20000f944558] [h50n11:35759] [ 3] /gpfs/alpine/csc314/scratch/adams/petsc/arch-summit-opt-gnu-kokkos-notpl-cuda10/lib/libkokkoscore.so.3.4(_ZN6Kokkos4Impl12CudaInternal10initializeEiP11CUstream_st+0xe60)[0x20000f95c210] [h50n11:35759] [ 4] /gpfs/alpine/csc314/scratch/adams/petsc/arch-summit-opt-gnu-kokkos-notpl-cuda10/lib/libkokkoscore.so.3.4(_ZN6Kokkos4Cuda15impl_initializeENS0_12SelectDeviceEm+0x30)[0x20000f95c2b0] [h50n11:35759] [ 5] /gpfs/alpine/csc314/scratch/adams/petsc/arch-summit-opt-gnu-kokkos-notpl-cuda10/lib/libkokkoscore.so.3.4(_ZN6Kokkos4Impl20CudaSpaceInitializer10initializeERKNS_13InitArgumentsE+0x34)[0x20000f95c314] [h50n11:35759] [ 6] /gpfs/alpine/csc314/scratch/adams/petsc/arch-summit-opt-gnu-kokkos-notpl-cuda10/lib/libkokkoscore.so.3.4(_ZN6Kokkos4Impl16ExecSpaceManager17initialize_spacesERKNS_13InitArgumentsE+0x60)[0x20000f926aa0] [h50n11:35759] [ 7] /gpfs/alpine/csc314/scratch/adams/petsc/arch-summit-opt-gnu-kokkos-notpl-cuda10/lib/libkokkoscore.so.3.4(_ZN6Kokkos4Impl70_GLOBAL__N__46_tmpxft_0001ef6b_00000000_6_Kokkos_Core_cpp1_ii_889c95a619initialize_backendsERKNS_13InitArgumentsE+0x2c)[0x20000f926dac] [h50n11:35759] [ 8] /gpfs/alpine/csc314/scratch/adams/petsc/arch-summit-opt-gnu-kokkos-notpl-cuda10/lib/libkokkoscore.so.3.4(_ZN6Kokkos4Impl70_GLOBAL__N__46_tmpxft_0001ef6b_00000000_6_Kokkos_Core_cpp1_ii_889c95a619initialize_internalERKNS_13InitArgumentsE+0x2c)[0x20000f92b73c] [h50n11:35759] [ 9] /gpfs/alpine/csc314/scratch/adams/petsc/arch-summit-opt-gnu-kokkos-notpl-cuda10/lib/libkokkoscore.so.3.4(_ZN6Kokkos10initializeENS_13InitArgumentsE+0x2c)[0x20000f92d08c] [h50n11:35759] [10] /gpfs/alpine/csc314/scratch/adams/petsc/arch-summit-opt-gnu-kokkos-notpl-cuda10/lib/libpetsc.so.3.015(PetscKokkosInitializeCheck+0x1f4)[0x200000343424] [h50n11:35759] [11] /gpfs/alpine/csc314/scratch/adams/petsc/arch-summit-opt-gnu-kokkos-notpl-cuda10/lib/libpetsc.so.3.015(+0x68b3dc)[0x20000077b3dc] [h50n11:35759] [12] /gpfs/alpine/csc314/scratch/adams/petsc/arch-summit-opt-gnu-kokkos-notpl-cuda10/lib/libpetsc.so.3.015(+0x3119b4)[0x2000004019b4] [h50n11:35759] [13] /gpfs/alpine/csc314/scratch/adams/petsc/arch-summit-opt-gnu-kokkos-notpl-cuda10/lib/libpetsc.so.3.015(+0x306790)[0x2000003f6790] [h50n11:35759] [14] /gpfs/alpine/csc314/scratch/adams/petsc/arch-summit-opt-gnu-kokkos-notpl-cuda10/lib/libpetsc.so.3.015(+0x274d24)[0x200000364d24] [h50n11:35759] [15] /gpfs/alpine/csc314/scratch/adams/petsc/arch-summit-opt-gnu-kokkos-notpl-cuda10/lib/libpetsc.so.3.015(PetscSFBcastWithMemTypeBegin+0xd4)[0x200000412504] [h50n11:35759] [16] /gpfs/alpine/csc314/scratch/adams/petsc/arch-summit-opt-gnu-kokkos-notpl-cuda10/lib/libpetsc.so.3.015(+0x329f9c)[0x200000419f9c] [h50n11:35759] [17] /gpfs/alpine/csc314/scratch/adams/petsc/arch-summit-opt-gnu-kokkos-notpl-cuda10/lib/libpetsc.so.3.015(VecScatterBegin+0x9c)[0x20000041fa8c] [h50n11:35759] [18] /gpfs/alpine/csc314/scratch/adams/petsc/arch-summit-opt-gnu-kokkos-notpl-cuda10/lib/libpetsc.so.3.015(DMGlobalToLocalBegin_DA+0x30)[0x2000010ef560] [h50n11:35759] [19] /gpfs/alpine/csc314/scratch/adams/petsc/arch-summit-opt-gnu-kokkos-notpl-cuda10/lib/libpetsc.so.3.015(DMGlobalToLocalBegin+0x290)[0x2000013314b0] [h50n11:35759] [20] /gpfs/alpine/csc314/scratch/adams/petsc/arch-summit-opt-gnu-kokkos-notpl-cuda10/lib/libpetsc.so.3.015(+0x1513c10)[0x200001603c10] [h50n11:35759] [21] /gpfs/alpine/csc314/scratch/adams/petsc/arch-summit-opt-gnu-kokkos-notpl-cuda10/lib/libpetsc.so.3.015(SNESComputeFunction+0x164)[0x200001625584] [h50n11:35759] [22] /gpfs/alpine/csc314/scratch/adams/petsc/arch-summit-opt-gnu-kokkos-notpl-cuda10/lib/libpetsc.so.3.015(+0x15830f4)[0x2000016730f4] [h50n11:35759] [23] /gpfs/alpine/csc314/scratch/adams/petsc/arch-summit-opt-gnu-kokkos-notpl-cuda10/lib/libpetsc.so.3.015(SNESSolve+0x814)[0x200001634c44] [h50n11:35759] [24] ./ex19[0x10001a70] [h50n11:35759] [25] /lib64/libc.so.6(+0x25200)[0x200032305200] [h50n11:35759] [26] /lib64/libc.so.6(__libc_start_main+0xc4)[0x2000323053f4] [h50n11:35759] *** End of error message *** ERROR: One or more process (first noticed rank 0) terminated with signal 6 (core dumped) /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials
On Mon, May 17, 2021 at 8:24 AM Mark Adams <[email protected] <mailto:[email protected]>> wrote: I thought I did a clean make but I made a clean one now and it seems to be working now.
Also, I am trying to fix this error message that I get on Cori with 'make check'. I set mpiexec='srun -G 2 -c 20' and get an interactive shell with these parameters, but I get error messages on Kokkos:
Possible error running C/C++ src/snes/tutorials/ex19 with 2 MPI processes See http://www.mcs.anl.gov/petsc/documentation/faq.html <http://www.mcs.anl.gov/petsc/documentation/faq.html> srun: error: Unable to create step for job 1923618: More processors requested than permitted C/C++ example src/snes/tutorials/ex19 run successfully with cuda gmake[3]: [makefile:102: runex3k_kokkos] Error 1 (ignored) 1,25c1 < atol=1e-50, rtol=1e-08, stol=1e-08, maxit=50, maxf=10000 < Vec Object: Exact Solution 2 MPI processes < type: mpikokkos < Process [0] < 0. < 0.015625 < 0.125 < Process [1] < 0.421875 < 1. < Vec Object: Forcing function 2 MPI processes < type: mpikokkos < Process [0] < 1e-72 < 1.50024 < 3.01563 < Process [1] < 4.67798 < 7. < 0 SNES Function norm 5.414682427127e+00 < 1 SNES Function norm 2.952582418265e-01 < 2 SNES Function norm 4.502293658739e-04 < 3 SNES Function norm 1.389665806646e-09 < Number of SNES iterations = 3 < Norm of error 1.49752e-10 Iterations 3 ---
srun: error: Unable to create step for job 1923618: More processors requested than permitted /global/homes/m/madams/petsc/src/snes/tutorials Possible problem with ex3k running with kokkos-kernels, diffs above ========================================= Fortran example src/snes/tutorials/ex5f run successfully with 1 MPI process Completed test examples
On Sun, May 16, 2021 at 11:14 PM Barry Smith <[email protected] <mailto:[email protected]>> wrote:
Could still be a gencode arch issue. Is it possible that Kokkos was built with the 80 arch and when you reran configure with 70 it did not rebuild Kokkos because it didn't know it needed to?
Sorry, but this may require another rm -rf arch* and running ./configure again.
https://docs.nvidia.com/cuda/cuda-runtime-api/group__CUDART__TYPES.html#grou... <https://docs.nvidia.com/cuda/cuda-runtime-api/group__CUDART__TYPES.html#group__CUDART__TYPES_1gg3f51e3575c2178246db0a94a430e0038b6af535e7e53d3f21e2437e8977b8c2e>
cudaErrorInvalidDeviceFunction = 98 The requested device function does not exist or is not compiled for the proper device architecture.
On May 16, 2021, at 9:09 PM, Mark Adams <[email protected] <mailto:[email protected]>> wrote:
I now get this error. A blas error from VecAXPBYPCZ ... Any ideas?
terminate called after throwing an instance of 'std::runtime_error' what(): cudaFuncGetAttributes(&attr_tmp, base_t::get_kernel_func()) error( cudaErrorInvalidDeviceFunction): invalid device function /global/u2/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/include/Cuda/Kokkos_Cuda_KernelLaunch.hpp:654 Traceback functionality not available
[cgpu16:55192] *** Process received signal *** [cgpu16:55192] Signal: Aborted (6) [cgpu16:55192] Signal code: (-6) [cgpu16:55192] [ 0] /lib64/libpthread.so.0(+0x12360)[0x2aab12445360] [cgpu16:55192] [ 1] /lib64/libc.so.6(gsignal+0x110)[0x2aab12687160] [cgpu16:55192] [ 2] /lib64/libc.so.6(abort+0x151)[0x2aab12688741] [cgpu16:55192] [ 3] /usr/common/software/sles15_cgpu/gcc/8.3.0/lib64/libstdc++.so.6(+0x93e83)[0x2aab10cb0e83] [cgpu16:55192] [ 4] /usr/common/software/sles15_cgpu/gcc/8.3.0/lib64/libstdc++.so.6(+0x99de6)[0x2aab10cb6de6] [cgpu16:55192] [ 5] /usr/common/software/sles15_cgpu/gcc/8.3.0/lib64/libstdc++.so.6(+0x99e21)[0x2aab10cb6e21] [cgpu16:55192] [ 6] /usr/common/software/sles15_cgpu/gcc/8.3.0/lib64/libstdc++.so.6(+0x9a053)[0x2aab10cb7053] [cgpu16:55192] [ 7] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libkokkoscore.so.3.4(+0x26a7f)[0x2aaabbcb3a7f] [cgpu16:55192] [ 8] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libkokkoscore.so.3.4(_ZN6Kokkos4Impl25cuda_internal_error_throwE9cudaErrorPKcS3_i+0x29d)[0x2aaabbcdab9d] [cgpu16:55192] [ 9] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libkokkoskernels.so(_ZN10KokkosBlas4Impl16V_Update_GenericIN6Kokkos4ViewIPKdJNS2_10LayoutLeftENS2_6DeviceINS2_4CudaENS2_9CudaSpaceEEENS2_12MemoryTraitsILj1EEEEEESD_NS3_IPdJS6_SA_SC_EEEiEEvRKNT_20non_const_value_typeERKSG_RKNT0_20non_const_value_typeERKSM_RKNT1_20non_const_value_typeERKSS_iii+0x3357)[0x2aaaae7108a7] [cgpu16:55192] [10] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libkokkoskernels.so(_ZN10KokkosBlas4Impl6UpdateIN6Kokkos4ViewIPKdJNS2_10LayoutLeftENS2_6DeviceINS2_4CudaENS2_9CudaSpaceEEENS2_12MemoryTraitsILj1EEEEEESD_NS3_IPdJS6_SA_SC_EEELi1ELb0ELb1EE6updateERS4_RKSD_SH_SJ_SH_RKSF_+0xc1)[0x2aaaae7171a1] [cgpu16:55192] [11] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libpetsc.so.3.015(_ZN10KokkosBlas6updateIN6Kokkos4ViewIPKdJNS1_9CudaSpaceEEEES6_NS2_IPdJS5_EEEEEvRKNT_20non_const_value_typeERKS9_RKNT0_20non_const_value_typeERKSF_RKNT1_20non_const_value_typeERKSL_+0x271)[0x2aaaab76d781] [cgpu16:55192] [12] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libpetsc.so.3.015(+0xa9333b)[0x2aaaab76633b] [cgpu16:55192] [13] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libpetsc.so.3.015(VecAXPBYPCZ+0x261)[0x2aaaab0b03c1] [cgpu16:55192] [14] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libpetsc.so.3.015(+0x155144e)[0x2aaaac22444e] [cgpu16:55192] [15] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libpetsc.so.3.015(SNESTSFormFunction+0xa)[0x2aaaac1c9c1a] [cgpu16:55192] [16] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libpetsc.so.3.015(SNESComputeFunction+0xf5)[0x2aaaac138675] [cgpu16:55192] [17] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libpetsc.so.3.015(+0x14ac85e)[0x2aaaac17f85e] [cgpu16:55192] [18] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libpetsc.so.3.015(SNESSolve+0x821)[0x2aaaac146651] [cgpu16:55192] [19] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libpetsc.so.3.015(+0x155526c)[0x2aaaac22826c] [cgpu16:55192] [20] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libpetsc.so.3.015(TSStep+0x1f5)[0x2aaaac1d6a05] [cgpu16:55192] [21] /global/homes/m/madams/petsc/arch-cori-gpu-opt-kokkos-gcc/lib/libpetsc.so.3.015(TSSolve+0x6a5)[0x2aaaac1dc455] [cgpu16:55192] [22] ../ex2-kok[0x4033eb] [cgpu16:55192] [23] /lib64/libc.so.6(__libc_start_main+0xea)[0x2aab12671f8a] [cgpu16:55192] [24] ../ex2-kok[0x404aaa] [cgpu16:55192] *** End of error message *** /global/homes/m/madams/mps-wrapper.sh: line 30: 55192 Aborted "$@" 0 stopping nvidia-cuda-mps-control on cgpu16
participants (3)
-
Barry Smith -
Junchao Zhang -
Mark Adams