July 2021 - petsc-dev - mailman3.cels.anl.gov

Re: [petsc-dev] Kokkos make error on Spock
by Mark Adams 17 Jul '21

17 Jul '21

Also, the application is a C++ code with PETSc Fortran methods. Thanks, Mark On Sat, Jul 17, 2021 at 11:17 AM Mark Adams <mfadams(a)lbl.gov> wrote: > And I can run a fortran test, with warnings, but C tests fail: > > 11:15 jczhang/fix-cray-mpicxx-includes/main= > /gpfs/alpine/csc314/scratch/adams/petsc$ make > PETSC_DIR=/gpfs/alpine/csc314/scratch/adams/petsc > PETSC_ARCH=arch-spock-opt-cray-kokkos -f gmakefile test > search='ts_utils_dmplexlandau_tutorials-ex1f90_0' > Using MAKEFLAGS: -- search=ts_utils_dmplexlandau_tutorials-ex1f90_0 > PETSC_ARCH=arch-spock-opt-cray-kokkos > PETSC_DIR=/gpfs/alpine/csc314/scratch/adams/petsc > FC > arch-spock-opt-cray-kokkos/tests/ts/utils/dmplexlandau/tutorials/ex1f90.o > FLINKER > arch-spock-opt-cray-kokkos/tests/ts/utils/dmplexlandau/tutorials/ex1f90 > /opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/bin/ld: > warning: alignment 128 of symbol > `$host_init$$runtime_init_for_iso_c_binding$iso_c_binding_' in > /opt/cray/pe/cce/11.0.4/cce/x86_64/lib/libmodules.so is smaller than 256 in > arch-spock-opt-cray-kokkos/tests/ts/utils/dmplexlandau/tutorials/ex1f90.o > /opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/bin/ld: > warning: alignment 64 of symbol `$data_init$iso_c_binding_' in > /opt/cray/pe/cce/11.0.4/cce/x86_64/lib/libmodules.so is smaller than 256 in > arch-spock-opt-cray-kokkos/tests/ts/utils/dmplexlandau/tutorials/ex1f90.o > TEST > arch-spock-opt-cray-kokkos/tests/counts/ts_utils_dmplexlandau_tutorials-ex1f90_0.counts > ok ts_utils_dmplexlandau_tutorials-ex1f90_0 > ok diff-ts_utils_dmplexlandau_tutorials-ex1f90_0 > > > On Sat, Jul 17, 2021 at 10:53 AM Mark Adams <mfadams(a)lbl.gov> wrote: > >> HUmm, I can not reproduce this. >> >> 10:49 jczhang/fix-cray-mpicxx-includes/main= >> /gpfs/alpine/csc314/scratch/adams/petsc$ make >> PETSC_DIR=/gpfs/alpine/csc314/scratch/adams/petsc >> PETSC_ARCH=arch-spock-dbg-kokkos check >> Running check examples to verify correct installation >> Using PETSC_DIR=/gpfs/alpine/csc314/scratch/adams/petsc and >> PETSC_ARCH=arch-spock-dbg-kokkos >> gmake[3]: >> [/gpfs/alpine/csc314/scratch/adams/petsc/lib/petsc/conf/rules:301: >> ex19.PETSc] Error 2 (ignored) >> *******************Error detected during compile or >> link!******************* >> See http://www.mcs.anl.gov/petsc/documentation/faq.html >> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials ex19 >> >> ********************************************************************************* >> cc -L/sw/spock/spack-envs/views/rocm-4.1.0/lib -lamdhip64 -lhsa-runtime64 >> -fPIC -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas >> -fstack-protector -Qunused-arguments -fvisibility=hidden -g -O0 -fPIC >> -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas >> -fstack-protector -Qunused-arguments -fvisibility=hidden -g -O0 >> -I/sw/spock/spack-envs/views/rocm-4.1.0/include >> -I/gpfs/alpine/csc314/scratch/adams/petsc/include >> -I/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/include >> -I/sw/spock/spack-envs/views/rocm-4.1.0/include >> -I/sw/spock/spack-envs/views/rocm-4.1.0/include ex19.c >> -Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib >> -L/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib >> -Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib >> -L/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib >> -Wl,-rpath,/sw/spock/spack-envs/views/rocm-4.1.0/lib >> -L/sw/spock/spack-envs/views/rocm-4.1.0/lib >> -Wl,-rpath,/opt/gcc/8.1.0/snos/lib64 -L/opt/gcc/8.1.0/snos/lib64 >> -Wl,-rpath,/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib >> -L/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib >> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib >> -L/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib >> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/gtl/lib >> -L/opt/cray/pe/mpich/8.1.4/gtl/lib -Wl,-rpath,/opt/cray/pe/pmi/6.0.10/lib >> -L/opt/cray/pe/pmi/6.0.10/lib -Wl,-rpath,/opt/cray/pe/dsmml/0.1.4/dsmml/lib >> -L/opt/cray/pe/dsmml/0.1.4/dsmml/lib >> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce/x86_64/lib >> -L/opt/cray/pe/cce/11.0.4/cce/x86_64/lib >> -Wl,-rpath,/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64 >> -L/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64 >> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux >> -L/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux >> -Wl,-rpath,/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0 >> -L/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0 >> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib >> -L/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib >> -lpetsc -lkokkoskernels -lkokkoscontainers -lkokkoscore -lhipsparse >> -lhipblas -lrocsparse -lrocsolver -lrocblas -lamdhip64 -lhsa-runtime64 >> -lstdc++ -ldl -lpmi -lsci_cray_mpi -lsci_cray -lmpifort_cray -lmpi_cray >> -lmpi_gtl_hsa -lxpmem -ldsmml -lpgas-shmem -lquadmath -lcrayacc_amdgpu >> -lopenacc -lmodules -lfi -lcraymath -lf -lu -lcsup -lgfortran -lpthread >> -lgcc_eh -lm -lclang_rt.craypgo-x86_64 -lclang_rt.builtins-x86_64 >> -lquadmath -lstdc++ -ldl -o ex19 >> ld.lld: error: >> /gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib/libpetsc.so: >> undefined reference to .omp_offloading.img_start.cray_amdgcn-amd-amdhsa >> [--no-allow-shlib-undefined] >> ld.lld: error: >> /gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib/libpetsc.so: >> undefined reference to .omp_offloading.img_size.cray_amdgcn-amd-amdhsa >> [--no-allow-shlib-undefined] >> ld.lld: error: >> /gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib/libpetsc.so: >> undefined reference to .omp_offloading.img_cache.cray_amdgcn-amd-amdhsa >> [--no-allow-shlib-undefined] >> clang-11: error: linker command failed with exit code 1 (use -v to see >> invocation) >> gmake[4]: *** [<builtin>: ex19] Error 1 >> gmake[3]: >> [/gpfs/alpine/csc314/scratch/adams/petsc/lib/petsc/conf/rules:301: >> ex3k.PETSc] Error 2 (ignored) >> *******************Error detected during compile or >> link!******************* >> See http://www.mcs.anl.gov/petsc/documentation/faq.html >> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials ex3k >> >> ********************************************************************************* >> hipcc -fPIC -Wall -Wwrite-strings -Wno-strict-aliasing >> -Wno-unknown-pragmas -fvisibility=hidden -g -O0 -std=c++14 >> -I/gpfs/alpine/csc314/scratch/adams/petsc/include >> -I/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/include >> -I/sw/spock/spack-envs/views/rocm-4.1.0/include >> -I/sw/spock/spack-envs/views/rocm-4.1.0/include -I/opt/cray/pe/libsci/ >> 21.04.1.1/CRAY/9.0/x86_64/include >> -I/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/include >> -I/opt/cray/pe/pmi/6.0.10/include -I/opt/cray/pe/dsmml/0.1.4/dsmml//include >> -I/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/include >> -L/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib >> -L/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib >> -L/opt/cray/pe/mpich/8.1.4/gtl/lib -L/opt/cray/pe/pmi/6.0.10/lib >> -L/opt/cray/pe/dsmml/0.1.4/dsmml//lib >> -L/opt/cray/pe/cce/11.0.4/cce/x86_64/lib/pkgconfig/../ >> -L/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64 >> -Wl,--as-needed,-lsci_cray_mpi,--no-as-needed >> -Wl,--as-needed,-lsci_cray,--no-as-needed >> -Wl,--as-needed,-lmpi_cray,--no-as-needed >> -Wl,--as-needed,-ldsmml,--no-as-needed >> -Wl,--as-needed,-lpgas-shmem,--no-as-needed -lpmi -ldl -lmpi_gtl_hsa >> -lxpmem -lquadmath -lmodules -lfi -lcraymath -lf -lu -lcsup >> ex3k.kokkos.cxx >> -Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib >> -L/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib >> -Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib >> -L/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib >> -Wl,-rpath,/sw/spock/spack-envs/views/rocm-4.1.0/lib >> -L/sw/spock/spack-envs/views/rocm-4.1.0/lib >> -Wl,-rpath,/opt/gcc/8.1.0/snos/lib64 -L/opt/gcc/8.1.0/snos/lib64 >> -Wl,-rpath,/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib >> -L/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib >> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib >> -L/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib >> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/gtl/lib >> -L/opt/cray/pe/mpich/8.1.4/gtl/lib -Wl,-rpath,/opt/cray/pe/pmi/6.0.10/lib >> -L/opt/cray/pe/pmi/6.0.10/lib -Wl,-rpath,/opt/cray/pe/dsmml/0.1.4/dsmml/lib >> -L/opt/cray/pe/dsmml/0.1.4/dsmml/lib >> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce/x86_64/lib >> -L/opt/cray/pe/cce/11.0.4/cce/x86_64/lib >> -Wl,-rpath,/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64 >> -L/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64 >> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux >> -L/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux >> -Wl,-rpath,/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0 >> -L/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0 >> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib >> -L/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib >> -lpetsc -lkokkoskernels -lkokkoscontainers -lkokkoscore -lhipsparse >> -lhipblas -lrocsparse -lrocsolver -lrocblas -lamdhip64 -lhsa-runtime64 >> -lstdc++ -ldl -lpmi -lsci_cray_mpi -lsci_cray -lmpifort_cray -lmpi_cray >> -lmpi_gtl_hsa -lxpmem -ldsmml -lpgas-shmem -lquadmath -lcrayacc_amdgpu >> -lopenacc -lmodules -lfi -lcraymath -lf -lu -lcsup -lgfortran -lpthread >> -lgcc_eh -lm -lclang_rt.craypgo-x86_64 -lclang_rt.builtins-x86_64 >> -lquadmath -lstdc++ -ldl -o ex3k >> /usr/lib64/gcc/x86_64-suse-linux/7/../../../../x86_64-suse-linux/bin/ld: >> /gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib/libpetsc.so: >> undefined reference to `.omp_offloading.img_size.cray_amdgcn-amd-amdhsa' >> /usr/lib64/gcc/x86_64-suse-linux/7/../../../../x86_64-suse-linux/bin/ld: >> /gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib/libpetsc.so: >> undefined reference to `.omp_offloading.img_start.cray_amdgcn-amd-amdhsa' >> /usr/lib64/gcc/x86_64-suse-linux/7/../../../../x86_64-suse-linux/bin/ld: >> /gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib/libpetsc.so: >> undefined reference to `.omp_offloading.img_cache.cray_amdgcn-amd-amdhsa' >> clang-12: error: linker command failed with exit code 1 (use -v to see >> invocation) >> gmake[4]: *** >> [/gpfs/alpine/csc314/scratch/adams/petsc/lib/petsc/conf/test:26: ex3k] >> Error 1 >> gmake[3]: [makefile:102: runex3k_kokkos] Error 2 (ignored) >> 1,25c1,4 >> < atol=1e-50, rtol=1e-08, stol=1e-08, maxit=50, maxf=10000 >> < Vec Object: Exact Solution 2 MPI processes >> < type: mpikokkos >> < Process [0] >> < 0. >> < 0.015625 >> < 0.125 >> < Process [1] >> < 0.421875 >> < 1. >> < Vec Object: Forcing function 2 MPI processes >> < type: mpikokkos >> < Process [0] >> < 1e-72 >> < 1.50024 >> < 3.01563 >> < Process [1] >> < 4.67798 >> < 7. >> < 0 SNES Function norm 5.414682427127e+00 >> < 1 SNES Function norm 2.952582418265e-01 >> < 2 SNES Function norm 4.502293658739e-04 >> < 3 SNES Function norm 1.389665806646e-09 >> < Number of SNES iterations = 3 >> < Norm of error 1.49752e-10 Iterations 3 >> --- >> > slurmstepd: error: execve(): >> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials/./ex3k: No such >> file or directory >> > slurmstepd: error: execve(): >> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials/./ex3k: No such >> file or directory >> > srun: error: spock01: tasks 0-1: Exited with exit code 2 >> > srun: launch/slurm: _step_signal: Terminating StepId=272979.0 >> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials >> Possible problem with ex3k running with kokkos-kernels, diffs above >> ========================================= >> *******************Error detected during compile or >> link!******************* >> See http://www.mcs.anl.gov/petsc/documentation/faq.html >> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials ex5f >> ********************************************************* >> ftn -L/sw/spock/spack-envs/views/rocm-4.1.0/lib -lamdhip64 >> -lhsa-runtime64 -fPIC -g -O0 -fPIC -g -O0 >> -I/gpfs/alpine/csc314/scratch/adams/petsc/include >> -I/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/include >> -I/sw/spock/spack-envs/views/rocm-4.1.0/include ex5f.F90 >> -Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib >> -L/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib >> -Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib >> -L/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib >> -Wl,-rpath,/sw/spock/spack-envs/views/rocm-4.1.0/lib >> -L/sw/spock/spack-envs/views/rocm-4.1.0/lib >> -Wl,-rpath,/opt/gcc/8.1.0/snos/lib64 -L/opt/gcc/8.1.0/snos/lib64 >> -Wl,-rpath,/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib >> -L/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib >> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib >> -L/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib >> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/gtl/lib >> -L/opt/cray/pe/mpich/8.1.4/gtl/lib -Wl,-rpath,/opt/cray/pe/pmi/6.0.10/lib >> -L/opt/cray/pe/pmi/6.0.10/lib -Wl,-rpath,/opt/cray/pe/dsmml/0.1.4/dsmml/lib >> -L/opt/cray/pe/dsmml/0.1.4/dsmml/lib >> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce/x86_64/lib >> -L/opt/cray/pe/cce/11.0.4/cce/x86_64/lib >> -Wl,-rpath,/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64 >> -L/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64 >> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux >> -L/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux >> -Wl,-rpath,/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0 >> -L/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0 >> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib >> -L/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib >> -lpetsc -lkokkoskernels -lkokkoscontainers -lkokkoscore -lhipsparse >> -lhipblas -lrocsparse -lrocsolver -lrocblas -lamdhip64 -lhsa-runtime64 >> -lstdc++ -ldl -lpmi -lsci_cray_mpi -lsci_cray -lmpifort_cray -lmpi_cray >> -lmpi_gtl_hsa -lxpmem -ldsmml -lpgas-shmem -lquadmath -lcrayacc_amdgpu >> -lopenacc -lmodules -lfi -lcraymath -lf -lu -lcsup -lgfortran -lpthread >> -lgcc_eh -lm -lclang_rt.craypgo-x86_64 -lclang_rt.builtins-x86_64 >> -lquadmath -lstdc++ -ldl -o ex5f >> /opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/bin/ld: >> warning: alignment 128 of symbol >> `$host_init$$runtime_init_for_iso_c_binding$iso_c_binding_' in >> /opt/cray/pe/cce/11.0.4/cce/x86_64/lib/libmodules.so is smaller than 256 in >> /tmp/pe_30429/ex5f_1.o >> /opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/bin/ld: >> warning: alignment 64 of symbol `$data_init$iso_c_binding_' in >> /opt/cray/pe/cce/11.0.4/cce/x86_64/lib/libmodules.so is smaller than 256 in >> /tmp/pe_30429/ex5f_1.o >> Fortran example src/snes/tutorials/ex5f run successfully with 1 MPI >> process >> Completed test examples >> 10:49 jczhang/fix-cray-mpicxx-includes/main= >> /gpfs/alpine/csc314/scratch/adams/petsc$ ll >> >> On Fri, Jul 16, 2021 at 11:28 PM Junchao Zhang <junchao.zhang(a)gmail.com> >> wrote: >> >>> Mark, I configured with Fortran binding enabled with main + MR !4175 >>> <https://gitlab.com/petsc/petsc/-/merge_requests/4175> >>> >>> '--with-mpiexec=srun', >>> '--with-shared-libraries=1', >>> '--with-cc=cc', >>> '--with-cxx=CC', >>> '--with-fc=ftn', >>> '--with-fortran-bindings', >>> '--with-hip', >>> '--with-hipc=hipcc', >>> '--with-debugging', >>> '--CPPFLAGS=-I${ROCM_PATH}/include', >>> '--CXXPPFLAGS=-I${ROCM_PATH}/include', >>> '--CC_LINKER_FLAGS=-L${ROCM_PATH}/lib -lamdhip64 -lhsa-runtime64', >>> '--CXX_LINKER_FLAGS=-L${ROCM_PATH}/lib -lamdhip64 -lhsa-runtime64', >>> '--FC_LINKER_FLAGS=-L${ROCM_PATH}/lib -lamdhip64 -lhsa-runtime64', >>> '--COPTFLAGS=-g -O0', >>> '--CXXOPTFLAGS=-g -O0', >>> '--FOPTFLAGS=-g -O0', >>> '--download-kokkos', >>> '--download-kokkos-kernels', >>> '--download-kokkos-commit=3.4.01', >>> '--download-kokkos-kernels-commit=3.4.01', >>> '--with-kokkos-hip-arch=VEGA908', >>> >>> and 'make check' ran smoothly on a compute node >>> >>> $ make check >>> Running check examples to verify correct installation >>> Using PETSC_DIR=/ccs/home/jczhang/petsc and >>> PETSC_ARCH=arch-spock-cray-kokkos-dbg >>> C/C++ example src/snes/tutorials/ex19 run successfully with 1 MPI process >>> C/C++ example src/snes/tutorials/ex19 run successfully with 2 MPI >>> processes >>> C/C++ example src/snes/tutorials/ex3k run successfully with >>> kokkos-kernels >>> Fortran example src/snes/tutorials/ex5f run successfully with 1 MPI >>> process >>> >>> Completed test examples >>> >>> >>> --Junchao Zhang >>> >>> >>> On Fri, Jul 16, 2021 at 6:04 PM Mark Adams <mfadams(a)lbl.gov> wrote: >>> >>>> And I find that this error, on non-Kokkos C tests, is fixed by turning >>>> the fortran bindings off: >>>> >>>> ld.lld: error: >>>> /gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/lib/libpetsc.so: >>>> undefined reference to .omp_offloading.img_start.cray_amdgcn-amd-amdhsa >>>> [--no-allow-shlib-undefined] >>>> ld.lld: error: >>>> /gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/lib/libpetsc.so: >>>> undefined reference to .omp_offloading.img_size.cray_amdgcn-amd-amdhsa >>>> [--no-allow-shlib-undefined] >>>> ld.lld: error: >>>> /gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/lib/libpetsc.so: >>>> undefined reference to .omp_offloading.img_cache.cray_amdgcn-amd-amdhsa >>>> [--no-allow-shlib-undefined] >>>> >>>> On Fri, Jul 16, 2021 at 3:53 PM Mark Adams <mfadams(a)lbl.gov> wrote: >>>> >>>>> Not complex. THis has some overlap with my problem w/o Kokkos. >>>>> >>>>> On Fri, Jul 16, 2021 at 12:54 PM Junchao Zhang < >>>>> junchao.zhang(a)gmail.com> wrote: >>>>> >>>>>> Do you use complex? post your configure.log. >>>>>> >>>>>> --Junchao Zhang >>>>>> >>>>>> >>>>>> On Fri, Jul 16, 2021 at 9:47 AM Mark Adams <mfadams(a)lbl.gov> wrote: >>>>>> >>>>>>> The simple Kokkos example is failing for me on Spock. >>>>>>> Any ideas? >>>>>>> Thanks, >>>>>>> >>>>>>> 10:44 main *= >>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials$ make >>>>>>> PETSC_DIR=/gpfs/alpine/csc314/scratch/adams/petsc >>>>>>> PETSC_ARCH=arch-spock-opt-cray-kokkos ex3k >>>>>>> MPICH_CXX="hipcc" OMPI_CXX="hipcc" CC >>>>>>> -L/sw/spock/spack-envs/views/rocm-4.1.0/lib -lhsa-runtime64 >>>>>>> -L/sw/spock/spack-envs/views/rocm-4.1.0/lib -lamdhip64 -lhsa-runtime64 >>>>>>> -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas >>>>>>> -fstack-protector -fvisibility=hidden -g -O2 -fPIC -Wall -Wwrite-strings >>>>>>> -Wno-strict-aliasing -Wno-unknown-pragmas -fvisibility=hidden -g -O3 >>>>>>> -std=c++14 -I/gpfs/alpine/csc314/scratch/adams/petsc/include >>>>>>> -I/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-opt-cray-kokkos/include >>>>>>> -I/gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include >>>>>>> -I/sw/spock/spack-envs/views/rocm-4.1.0/include >>>>>>> -I/sw/spock/spack-envs/views/rocm-4.1.0/include ex3k.kokkos.cxx >>>>>>> -Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-opt-cray-kokkos/lib >>>>>>> -L/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-opt-cray-kokkos/lib >>>>>>> -Wl,-rpath,/gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/lib >>>>>>> -L/gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/lib >>>>>>> -Wl,-rpath,/sw/spock/spack-envs/views/rocm-4.1.0/lib >>>>>>> -L/sw/spock/spack-envs/views/rocm-4.1.0/lib >>>>>>> -Wl,-rpath,/opt/gcc/8.1.0/snos/lib64 -L/opt/gcc/8.1.0/snos/lib64 >>>>>>> -Wl,-rpath,/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib >>>>>>> -L/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib >>>>>>> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib >>>>>>> -L/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib >>>>>>> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/gtl/lib >>>>>>> -L/opt/cray/pe/mpich/8.1.4/gtl/lib -Wl,-rpath,/opt/cray/pe/pmi/6.0.10/lib >>>>>>> -L/opt/cray/pe/pmi/6.0.10/lib -Wl,-rpath,/opt/cray/pe/dsmml/0.1.4/dsmml/lib >>>>>>> -L/opt/cray/pe/dsmml/0.1.4/dsmml/lib >>>>>>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce/x86_64/lib >>>>>>> -L/opt/cray/pe/cce/11.0.4/cce/x86_64/lib >>>>>>> -Wl,-rpath,/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64 >>>>>>> -L/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64 >>>>>>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux >>>>>>> -L/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux >>>>>>> -Wl,-rpath,/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0 >>>>>>> -L/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0 >>>>>>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib >>>>>>> -L/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib >>>>>>> -lpetsc -lkokkoskernels -lkokkoscontainers -lkokkoscore -lhipsparse >>>>>>> -lhipblas -lrocsparse -lrocsolver -lrocblas -lamdhip64 -lhsa-runtime64 >>>>>>> -lstdc++ -ldl -lpmi -lsci_cray_mpi -lsci_cray -lmpifort_cray -lmpi_cray >>>>>>> -lmpi_gtl_hsa -lxpmem -ldsmml -lpgas-shmem -lquadmath -lcrayacc_amdgpu >>>>>>> -lopenacc -lmodules -lfi -lcraymath -lf -lu -lcsup -lgfortran -lpthread >>>>>>> -lgcc_eh -lm -lclang_rt.craypgo-x86_64 -lclang_rt.builtins-x86_64 >>>>>>> -lquadmath -lstdc++ -ldl -o ex3k >>>>>>> In file included from ex3k.kokkos.cxx:3: >>>>>>> In file included from >>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscdmda_kokkos.hpp:4: >>>>>>> In file included from >>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec_kokkos.hpp:14: >>>>>>> In file included from >>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec.h:9: >>>>>>> In file included from >>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsys.h:42: >>>>>>> In file included from >>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsystypes.h:255: >>>>>>> In file included from >>>>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Complex.hpp:47: >>>>>>> In file included from >>>>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Atomic.hpp:212: >>>>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/impl/Kokkos_Atomic_Exchange.hpp:259:11: >>>>>>> error: no member named 'lock_address_host_space' in namespace >>>>>>> 'Kokkos::Impl::Kokkos::Impl'; did you mean simply 'lock_address_host_space'? >>>>>>> while (!Impl::lock_address_host_space((void*)dest)) >>>>>>> ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~ >>>>>>> lock_address_host_space >>>>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_HostSpace.hpp:83:6: >>>>>>> note: 'lock_address_host_space' declared here >>>>>>> bool lock_address_host_space(void* ptr); >>>>>>> ^ >>>>>>> In file included from ex3k.kokkos.cxx:3: >>>>>>> In file included from >>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscdmda_kokkos.hpp:4: >>>>>>> In file included from >>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec_kokkos.hpp:14: >>>>>>> In file included from >>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec.h:9: >>>>>>> In file included from >>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsys.h:42: >>>>>>> In file included from >>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsystypes.h:255: >>>>>>> In file included from >>>>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Complex.hpp:47: >>>>>>> In file included from >>>>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Atomic.hpp:212: >>>>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/impl/Kokkos_Atomic_Exchange.hpp:261:3: >>>>>>> error: no member named 'memory_fence' in namespace 'Kokkos::Impl::Kokkos'; >>>>>>> did you mean simply 'memory_fence'? >>>>>>> Kokkos::memory_fence(); >>>>>>> ^~~~~~~~~~~~~~~~~~~~ >>>>>>> memory_fence >>>>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/impl/Kokkos_Memory_Fence.hpp:53:6: >>>>>>> note: 'memory_fence' declared here >>>>>>> void memory_fence() { >>>>>>> ^ >>>>>>> In file included from ex3k.kokkos.cxx:3: >>>>>>> In file included from >>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscdmda_kokkos.hpp:4: >>>>>>> In file included from >>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec_kokkos.hpp:14: >>>>>>> In file included from >>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec.h:9: >>>>>>> In file included from >>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsys.h:42: >>>>>>> In file included from >>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsystypes.h:255: >>>>>>> In file included from >>>>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Complex.hpp:47: >>>>>>> In file included from >>>>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Atomic.hpp:212: >>>>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/impl/Kokkos_Atomic_Exchange.hpp:277:3: >>>>>>> error: no member named 'memory_fence' in namespace 'Kokkos::Impl::Kokkos'; >>>>>>> did you mean simply 'memory_fence'? >>>>>>> Kokkos::memory_fence(); >>>>>>> ^~~~~~~~~~~~~~~~~~~~ >>>>>>> memory_fence >>>>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/impl/Kokkos_Memory_Fence.hpp:53:6: >>>>>>> note: 'memory_fence' declared here >>>>>>> void memory_fence() { >>>>>>> ^ >>>>>>> >>>>>>

1 0

Re: [petsc-dev] Kokkos make error on Spock
by Mark Adams 17 Jul '21

17 Jul '21

And I can run a fortran test, with warnings, but C tests fail: 11:15 jczhang/fix-cray-mpicxx-includes/main= /gpfs/alpine/csc314/scratch/adams/petsc$ make PETSC_DIR=/gpfs/alpine/csc314/scratch/adams/petsc PETSC_ARCH=arch-spock-opt-cray-kokkos -f gmakefile test search='ts_utils_dmplexlandau_tutorials-ex1f90_0' Using MAKEFLAGS: -- search=ts_utils_dmplexlandau_tutorials-ex1f90_0 PETSC_ARCH=arch-spock-opt-cray-kokkos PETSC_DIR=/gpfs/alpine/csc314/scratch/adams/petsc FC arch-spock-opt-cray-kokkos/tests/ts/utils/dmplexlandau/tutorials/ex1f90.o FLINKER arch-spock-opt-cray-kokkos/tests/ts/utils/dmplexlandau/tutorials/ex1f90 /opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/bin/ld: warning: alignment 128 of symbol `$host_init$$runtime_init_for_iso_c_binding$iso_c_binding_' in /opt/cray/pe/cce/11.0.4/cce/x86_64/lib/libmodules.so is smaller than 256 in arch-spock-opt-cray-kokkos/tests/ts/utils/dmplexlandau/tutorials/ex1f90.o /opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/bin/ld: warning: alignment 64 of symbol `$data_init$iso_c_binding_' in /opt/cray/pe/cce/11.0.4/cce/x86_64/lib/libmodules.so is smaller than 256 in arch-spock-opt-cray-kokkos/tests/ts/utils/dmplexlandau/tutorials/ex1f90.o TEST arch-spock-opt-cray-kokkos/tests/counts/ts_utils_dmplexlandau_tutorials-ex1f90_0.counts ok ts_utils_dmplexlandau_tutorials-ex1f90_0 ok diff-ts_utils_dmplexlandau_tutorials-ex1f90_0 On Sat, Jul 17, 2021 at 10:53 AM Mark Adams <mfadams(a)lbl.gov> wrote: > HUmm, I can not reproduce this. > > 10:49 jczhang/fix-cray-mpicxx-includes/main= > /gpfs/alpine/csc314/scratch/adams/petsc$ make > PETSC_DIR=/gpfs/alpine/csc314/scratch/adams/petsc > PETSC_ARCH=arch-spock-dbg-kokkos check > Running check examples to verify correct installation > Using PETSC_DIR=/gpfs/alpine/csc314/scratch/adams/petsc and > PETSC_ARCH=arch-spock-dbg-kokkos > gmake[3]: > [/gpfs/alpine/csc314/scratch/adams/petsc/lib/petsc/conf/rules:301: > ex19.PETSc] Error 2 (ignored) > *******************Error detected during compile or > link!******************* > See http://www.mcs.anl.gov/petsc/documentation/faq.html > /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials ex19 > > ********************************************************************************* > cc -L/sw/spock/spack-envs/views/rocm-4.1.0/lib -lamdhip64 -lhsa-runtime64 > -fPIC -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas > -fstack-protector -Qunused-arguments -fvisibility=hidden -g -O0 -fPIC > -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas > -fstack-protector -Qunused-arguments -fvisibility=hidden -g -O0 > -I/sw/spock/spack-envs/views/rocm-4.1.0/include > -I/gpfs/alpine/csc314/scratch/adams/petsc/include > -I/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/include > -I/sw/spock/spack-envs/views/rocm-4.1.0/include > -I/sw/spock/spack-envs/views/rocm-4.1.0/include ex19.c > -Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib > -L/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib > -Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib > -L/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib > -Wl,-rpath,/sw/spock/spack-envs/views/rocm-4.1.0/lib > -L/sw/spock/spack-envs/views/rocm-4.1.0/lib > -Wl,-rpath,/opt/gcc/8.1.0/snos/lib64 -L/opt/gcc/8.1.0/snos/lib64 > -Wl,-rpath,/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib > -L/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib > -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib > -L/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib > -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/gtl/lib > -L/opt/cray/pe/mpich/8.1.4/gtl/lib -Wl,-rpath,/opt/cray/pe/pmi/6.0.10/lib > -L/opt/cray/pe/pmi/6.0.10/lib -Wl,-rpath,/opt/cray/pe/dsmml/0.1.4/dsmml/lib > -L/opt/cray/pe/dsmml/0.1.4/dsmml/lib > -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce/x86_64/lib > -L/opt/cray/pe/cce/11.0.4/cce/x86_64/lib > -Wl,-rpath,/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64 > -L/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64 > -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux > -L/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux > -Wl,-rpath,/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0 > -L/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0 > -Wl,-rpath,/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib > -L/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib > -lpetsc -lkokkoskernels -lkokkoscontainers -lkokkoscore -lhipsparse > -lhipblas -lrocsparse -lrocsolver -lrocblas -lamdhip64 -lhsa-runtime64 > -lstdc++ -ldl -lpmi -lsci_cray_mpi -lsci_cray -lmpifort_cray -lmpi_cray > -lmpi_gtl_hsa -lxpmem -ldsmml -lpgas-shmem -lquadmath -lcrayacc_amdgpu > -lopenacc -lmodules -lfi -lcraymath -lf -lu -lcsup -lgfortran -lpthread > -lgcc_eh -lm -lclang_rt.craypgo-x86_64 -lclang_rt.builtins-x86_64 > -lquadmath -lstdc++ -ldl -o ex19 > ld.lld: error: > /gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib/libpetsc.so: > undefined reference to .omp_offloading.img_start.cray_amdgcn-amd-amdhsa > [--no-allow-shlib-undefined] > ld.lld: error: > /gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib/libpetsc.so: > undefined reference to .omp_offloading.img_size.cray_amdgcn-amd-amdhsa > [--no-allow-shlib-undefined] > ld.lld: error: > /gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib/libpetsc.so: > undefined reference to .omp_offloading.img_cache.cray_amdgcn-amd-amdhsa > [--no-allow-shlib-undefined] > clang-11: error: linker command failed with exit code 1 (use -v to see > invocation) > gmake[4]: *** [<builtin>: ex19] Error 1 > gmake[3]: > [/gpfs/alpine/csc314/scratch/adams/petsc/lib/petsc/conf/rules:301: > ex3k.PETSc] Error 2 (ignored) > *******************Error detected during compile or > link!******************* > See http://www.mcs.anl.gov/petsc/documentation/faq.html > /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials ex3k > > ********************************************************************************* > hipcc -fPIC -Wall -Wwrite-strings -Wno-strict-aliasing > -Wno-unknown-pragmas -fvisibility=hidden -g -O0 -std=c++14 > -I/gpfs/alpine/csc314/scratch/adams/petsc/include > -I/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/include > -I/sw/spock/spack-envs/views/rocm-4.1.0/include > -I/sw/spock/spack-envs/views/rocm-4.1.0/include -I/opt/cray/pe/libsci/ > 21.04.1.1/CRAY/9.0/x86_64/include > -I/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/include > -I/opt/cray/pe/pmi/6.0.10/include -I/opt/cray/pe/dsmml/0.1.4/dsmml//include > -I/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/include > -L/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib > -L/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib > -L/opt/cray/pe/mpich/8.1.4/gtl/lib -L/opt/cray/pe/pmi/6.0.10/lib > -L/opt/cray/pe/dsmml/0.1.4/dsmml//lib > -L/opt/cray/pe/cce/11.0.4/cce/x86_64/lib/pkgconfig/../ > -L/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64 > -Wl,--as-needed,-lsci_cray_mpi,--no-as-needed > -Wl,--as-needed,-lsci_cray,--no-as-needed > -Wl,--as-needed,-lmpi_cray,--no-as-needed > -Wl,--as-needed,-ldsmml,--no-as-needed > -Wl,--as-needed,-lpgas-shmem,--no-as-needed -lpmi -ldl -lmpi_gtl_hsa > -lxpmem -lquadmath -lmodules -lfi -lcraymath -lf -lu -lcsup > ex3k.kokkos.cxx > -Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib > -L/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib > -Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib > -L/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib > -Wl,-rpath,/sw/spock/spack-envs/views/rocm-4.1.0/lib > -L/sw/spock/spack-envs/views/rocm-4.1.0/lib > -Wl,-rpath,/opt/gcc/8.1.0/snos/lib64 -L/opt/gcc/8.1.0/snos/lib64 > -Wl,-rpath,/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib > -L/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib > -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib > -L/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib > -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/gtl/lib > -L/opt/cray/pe/mpich/8.1.4/gtl/lib -Wl,-rpath,/opt/cray/pe/pmi/6.0.10/lib > -L/opt/cray/pe/pmi/6.0.10/lib -Wl,-rpath,/opt/cray/pe/dsmml/0.1.4/dsmml/lib > -L/opt/cray/pe/dsmml/0.1.4/dsmml/lib > -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce/x86_64/lib > -L/opt/cray/pe/cce/11.0.4/cce/x86_64/lib > -Wl,-rpath,/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64 > -L/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64 > -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux > -L/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux > -Wl,-rpath,/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0 > -L/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0 > -Wl,-rpath,/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib > -L/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib > -lpetsc -lkokkoskernels -lkokkoscontainers -lkokkoscore -lhipsparse > -lhipblas -lrocsparse -lrocsolver -lrocblas -lamdhip64 -lhsa-runtime64 > -lstdc++ -ldl -lpmi -lsci_cray_mpi -lsci_cray -lmpifort_cray -lmpi_cray > -lmpi_gtl_hsa -lxpmem -ldsmml -lpgas-shmem -lquadmath -lcrayacc_amdgpu > -lopenacc -lmodules -lfi -lcraymath -lf -lu -lcsup -lgfortran -lpthread > -lgcc_eh -lm -lclang_rt.craypgo-x86_64 -lclang_rt.builtins-x86_64 > -lquadmath -lstdc++ -ldl -o ex3k > /usr/lib64/gcc/x86_64-suse-linux/7/../../../../x86_64-suse-linux/bin/ld: > /gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib/libpetsc.so: > undefined reference to `.omp_offloading.img_size.cray_amdgcn-amd-amdhsa' > /usr/lib64/gcc/x86_64-suse-linux/7/../../../../x86_64-suse-linux/bin/ld: > /gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib/libpetsc.so: > undefined reference to `.omp_offloading.img_start.cray_amdgcn-amd-amdhsa' > /usr/lib64/gcc/x86_64-suse-linux/7/../../../../x86_64-suse-linux/bin/ld: > /gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib/libpetsc.so: > undefined reference to `.omp_offloading.img_cache.cray_amdgcn-amd-amdhsa' > clang-12: error: linker command failed with exit code 1 (use -v to see > invocation) > gmake[4]: *** > [/gpfs/alpine/csc314/scratch/adams/petsc/lib/petsc/conf/test:26: ex3k] > Error 1 > gmake[3]: [makefile:102: runex3k_kokkos] Error 2 (ignored) > 1,25c1,4 > < atol=1e-50, rtol=1e-08, stol=1e-08, maxit=50, maxf=10000 > < Vec Object: Exact Solution 2 MPI processes > < type: mpikokkos > < Process [0] > < 0. > < 0.015625 > < 0.125 > < Process [1] > < 0.421875 > < 1. > < Vec Object: Forcing function 2 MPI processes > < type: mpikokkos > < Process [0] > < 1e-72 > < 1.50024 > < 3.01563 > < Process [1] > < 4.67798 > < 7. > < 0 SNES Function norm 5.414682427127e+00 > < 1 SNES Function norm 2.952582418265e-01 > < 2 SNES Function norm 4.502293658739e-04 > < 3 SNES Function norm 1.389665806646e-09 > < Number of SNES iterations = 3 > < Norm of error 1.49752e-10 Iterations 3 > --- > > slurmstepd: error: execve(): > /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials/./ex3k: No such > file or directory > > slurmstepd: error: execve(): > /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials/./ex3k: No such > file or directory > > srun: error: spock01: tasks 0-1: Exited with exit code 2 > > srun: launch/slurm: _step_signal: Terminating StepId=272979.0 > /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials > Possible problem with ex3k running with kokkos-kernels, diffs above > ========================================= > *******************Error detected during compile or > link!******************* > See http://www.mcs.anl.gov/petsc/documentation/faq.html > /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials ex5f > ********************************************************* > ftn -L/sw/spock/spack-envs/views/rocm-4.1.0/lib -lamdhip64 -lhsa-runtime64 > -fPIC -g -O0 -fPIC -g -O0 > -I/gpfs/alpine/csc314/scratch/adams/petsc/include > -I/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/include > -I/sw/spock/spack-envs/views/rocm-4.1.0/include ex5f.F90 > -Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib > -L/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib > -Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib > -L/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib > -Wl,-rpath,/sw/spock/spack-envs/views/rocm-4.1.0/lib > -L/sw/spock/spack-envs/views/rocm-4.1.0/lib > -Wl,-rpath,/opt/gcc/8.1.0/snos/lib64 -L/opt/gcc/8.1.0/snos/lib64 > -Wl,-rpath,/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib > -L/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib > -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib > -L/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib > -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/gtl/lib > -L/opt/cray/pe/mpich/8.1.4/gtl/lib -Wl,-rpath,/opt/cray/pe/pmi/6.0.10/lib > -L/opt/cray/pe/pmi/6.0.10/lib -Wl,-rpath,/opt/cray/pe/dsmml/0.1.4/dsmml/lib > -L/opt/cray/pe/dsmml/0.1.4/dsmml/lib > -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce/x86_64/lib > -L/opt/cray/pe/cce/11.0.4/cce/x86_64/lib > -Wl,-rpath,/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64 > -L/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64 > -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux > -L/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux > -Wl,-rpath,/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0 > -L/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0 > -Wl,-rpath,/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib > -L/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib > -lpetsc -lkokkoskernels -lkokkoscontainers -lkokkoscore -lhipsparse > -lhipblas -lrocsparse -lrocsolver -lrocblas -lamdhip64 -lhsa-runtime64 > -lstdc++ -ldl -lpmi -lsci_cray_mpi -lsci_cray -lmpifort_cray -lmpi_cray > -lmpi_gtl_hsa -lxpmem -ldsmml -lpgas-shmem -lquadmath -lcrayacc_amdgpu > -lopenacc -lmodules -lfi -lcraymath -lf -lu -lcsup -lgfortran -lpthread > -lgcc_eh -lm -lclang_rt.craypgo-x86_64 -lclang_rt.builtins-x86_64 > -lquadmath -lstdc++ -ldl -o ex5f > /opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/bin/ld: > warning: alignment 128 of symbol > `$host_init$$runtime_init_for_iso_c_binding$iso_c_binding_' in > /opt/cray/pe/cce/11.0.4/cce/x86_64/lib/libmodules.so is smaller than 256 in > /tmp/pe_30429/ex5f_1.o > /opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/bin/ld: > warning: alignment 64 of symbol `$data_init$iso_c_binding_' in > /opt/cray/pe/cce/11.0.4/cce/x86_64/lib/libmodules.so is smaller than 256 in > /tmp/pe_30429/ex5f_1.o > Fortran example src/snes/tutorials/ex5f run successfully with 1 MPI process > Completed test examples > 10:49 jczhang/fix-cray-mpicxx-includes/main= > /gpfs/alpine/csc314/scratch/adams/petsc$ ll > > On Fri, Jul 16, 2021 at 11:28 PM Junchao Zhang <junchao.zhang(a)gmail.com> > wrote: > >> Mark, I configured with Fortran binding enabled with main + MR !4175 >> <https://gitlab.com/petsc/petsc/-/merge_requests/4175> >> >> '--with-mpiexec=srun', >> '--with-shared-libraries=1', >> '--with-cc=cc', >> '--with-cxx=CC', >> '--with-fc=ftn', >> '--with-fortran-bindings', >> '--with-hip', >> '--with-hipc=hipcc', >> '--with-debugging', >> '--CPPFLAGS=-I${ROCM_PATH}/include', >> '--CXXPPFLAGS=-I${ROCM_PATH}/include', >> '--CC_LINKER_FLAGS=-L${ROCM_PATH}/lib -lamdhip64 -lhsa-runtime64', >> '--CXX_LINKER_FLAGS=-L${ROCM_PATH}/lib -lamdhip64 -lhsa-runtime64', >> '--FC_LINKER_FLAGS=-L${ROCM_PATH}/lib -lamdhip64 -lhsa-runtime64', >> '--COPTFLAGS=-g -O0', >> '--CXXOPTFLAGS=-g -O0', >> '--FOPTFLAGS=-g -O0', >> '--download-kokkos', >> '--download-kokkos-kernels', >> '--download-kokkos-commit=3.4.01', >> '--download-kokkos-kernels-commit=3.4.01', >> '--with-kokkos-hip-arch=VEGA908', >> >> and 'make check' ran smoothly on a compute node >> >> $ make check >> Running check examples to verify correct installation >> Using PETSC_DIR=/ccs/home/jczhang/petsc and >> PETSC_ARCH=arch-spock-cray-kokkos-dbg >> C/C++ example src/snes/tutorials/ex19 run successfully with 1 MPI process >> C/C++ example src/snes/tutorials/ex19 run successfully with 2 MPI >> processes >> C/C++ example src/snes/tutorials/ex3k run successfully with kokkos-kernels >> Fortran example src/snes/tutorials/ex5f run successfully with 1 MPI >> process >> >> Completed test examples >> >> >> --Junchao Zhang >> >> >> On Fri, Jul 16, 2021 at 6:04 PM Mark Adams <mfadams(a)lbl.gov> wrote: >> >>> And I find that this error, on non-Kokkos C tests, is fixed by turning >>> the fortran bindings off: >>> >>> ld.lld: error: >>> /gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/lib/libpetsc.so: >>> undefined reference to .omp_offloading.img_start.cray_amdgcn-amd-amdhsa >>> [--no-allow-shlib-undefined] >>> ld.lld: error: >>> /gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/lib/libpetsc.so: >>> undefined reference to .omp_offloading.img_size.cray_amdgcn-amd-amdhsa >>> [--no-allow-shlib-undefined] >>> ld.lld: error: >>> /gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/lib/libpetsc.so: >>> undefined reference to .omp_offloading.img_cache.cray_amdgcn-amd-amdhsa >>> [--no-allow-shlib-undefined] >>> >>> On Fri, Jul 16, 2021 at 3:53 PM Mark Adams <mfadams(a)lbl.gov> wrote: >>> >>>> Not complex. THis has some overlap with my problem w/o Kokkos. >>>> >>>> On Fri, Jul 16, 2021 at 12:54 PM Junchao Zhang <junchao.zhang(a)gmail.com> >>>> wrote: >>>> >>>>> Do you use complex? post your configure.log. >>>>> >>>>> --Junchao Zhang >>>>> >>>>> >>>>> On Fri, Jul 16, 2021 at 9:47 AM Mark Adams <mfadams(a)lbl.gov> wrote: >>>>> >>>>>> The simple Kokkos example is failing for me on Spock. >>>>>> Any ideas? >>>>>> Thanks, >>>>>> >>>>>> 10:44 main *= >>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials$ make >>>>>> PETSC_DIR=/gpfs/alpine/csc314/scratch/adams/petsc >>>>>> PETSC_ARCH=arch-spock-opt-cray-kokkos ex3k >>>>>> MPICH_CXX="hipcc" OMPI_CXX="hipcc" CC >>>>>> -L/sw/spock/spack-envs/views/rocm-4.1.0/lib -lhsa-runtime64 >>>>>> -L/sw/spock/spack-envs/views/rocm-4.1.0/lib -lamdhip64 -lhsa-runtime64 >>>>>> -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas >>>>>> -fstack-protector -fvisibility=hidden -g -O2 -fPIC -Wall -Wwrite-strings >>>>>> -Wno-strict-aliasing -Wno-unknown-pragmas -fvisibility=hidden -g -O3 >>>>>> -std=c++14 -I/gpfs/alpine/csc314/scratch/adams/petsc/include >>>>>> -I/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-opt-cray-kokkos/include >>>>>> -I/gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include >>>>>> -I/sw/spock/spack-envs/views/rocm-4.1.0/include >>>>>> -I/sw/spock/spack-envs/views/rocm-4.1.0/include ex3k.kokkos.cxx >>>>>> -Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-opt-cray-kokkos/lib >>>>>> -L/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-opt-cray-kokkos/lib >>>>>> -Wl,-rpath,/gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/lib >>>>>> -L/gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/lib >>>>>> -Wl,-rpath,/sw/spock/spack-envs/views/rocm-4.1.0/lib >>>>>> -L/sw/spock/spack-envs/views/rocm-4.1.0/lib >>>>>> -Wl,-rpath,/opt/gcc/8.1.0/snos/lib64 -L/opt/gcc/8.1.0/snos/lib64 >>>>>> -Wl,-rpath,/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib >>>>>> -L/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib >>>>>> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib >>>>>> -L/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib >>>>>> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/gtl/lib >>>>>> -L/opt/cray/pe/mpich/8.1.4/gtl/lib -Wl,-rpath,/opt/cray/pe/pmi/6.0.10/lib >>>>>> -L/opt/cray/pe/pmi/6.0.10/lib -Wl,-rpath,/opt/cray/pe/dsmml/0.1.4/dsmml/lib >>>>>> -L/opt/cray/pe/dsmml/0.1.4/dsmml/lib >>>>>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce/x86_64/lib >>>>>> -L/opt/cray/pe/cce/11.0.4/cce/x86_64/lib >>>>>> -Wl,-rpath,/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64 >>>>>> -L/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64 >>>>>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux >>>>>> -L/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux >>>>>> -Wl,-rpath,/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0 >>>>>> -L/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0 >>>>>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib >>>>>> -L/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib >>>>>> -lpetsc -lkokkoskernels -lkokkoscontainers -lkokkoscore -lhipsparse >>>>>> -lhipblas -lrocsparse -lrocsolver -lrocblas -lamdhip64 -lhsa-runtime64 >>>>>> -lstdc++ -ldl -lpmi -lsci_cray_mpi -lsci_cray -lmpifort_cray -lmpi_cray >>>>>> -lmpi_gtl_hsa -lxpmem -ldsmml -lpgas-shmem -lquadmath -lcrayacc_amdgpu >>>>>> -lopenacc -lmodules -lfi -lcraymath -lf -lu -lcsup -lgfortran -lpthread >>>>>> -lgcc_eh -lm -lclang_rt.craypgo-x86_64 -lclang_rt.builtins-x86_64 >>>>>> -lquadmath -lstdc++ -ldl -o ex3k >>>>>> In file included from ex3k.kokkos.cxx:3: >>>>>> In file included from >>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscdmda_kokkos.hpp:4: >>>>>> In file included from >>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec_kokkos.hpp:14: >>>>>> In file included from >>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec.h:9: >>>>>> In file included from >>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsys.h:42: >>>>>> In file included from >>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsystypes.h:255: >>>>>> In file included from >>>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Complex.hpp:47: >>>>>> In file included from >>>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Atomic.hpp:212: >>>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/impl/Kokkos_Atomic_Exchange.hpp:259:11: >>>>>> error: no member named 'lock_address_host_space' in namespace >>>>>> 'Kokkos::Impl::Kokkos::Impl'; did you mean simply 'lock_address_host_space'? >>>>>> while (!Impl::lock_address_host_space((void*)dest)) >>>>>> ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~ >>>>>> lock_address_host_space >>>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_HostSpace.hpp:83:6: >>>>>> note: 'lock_address_host_space' declared here >>>>>> bool lock_address_host_space(void* ptr); >>>>>> ^ >>>>>> In file included from ex3k.kokkos.cxx:3: >>>>>> In file included from >>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscdmda_kokkos.hpp:4: >>>>>> In file included from >>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec_kokkos.hpp:14: >>>>>> In file included from >>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec.h:9: >>>>>> In file included from >>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsys.h:42: >>>>>> In file included from >>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsystypes.h:255: >>>>>> In file included from >>>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Complex.hpp:47: >>>>>> In file included from >>>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Atomic.hpp:212: >>>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/impl/Kokkos_Atomic_Exchange.hpp:261:3: >>>>>> error: no member named 'memory_fence' in namespace 'Kokkos::Impl::Kokkos'; >>>>>> did you mean simply 'memory_fence'? >>>>>> Kokkos::memory_fence(); >>>>>> ^~~~~~~~~~~~~~~~~~~~ >>>>>> memory_fence >>>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/impl/Kokkos_Memory_Fence.hpp:53:6: >>>>>> note: 'memory_fence' declared here >>>>>> void memory_fence() { >>>>>> ^ >>>>>> In file included from ex3k.kokkos.cxx:3: >>>>>> In file included from >>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscdmda_kokkos.hpp:4: >>>>>> In file included from >>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec_kokkos.hpp:14: >>>>>> In file included from >>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec.h:9: >>>>>> In file included from >>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsys.h:42: >>>>>> In file included from >>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsystypes.h:255: >>>>>> In file included from >>>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Complex.hpp:47: >>>>>> In file included from >>>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Atomic.hpp:212: >>>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/impl/Kokkos_Atomic_Exchange.hpp:277:3: >>>>>> error: no member named 'memory_fence' in namespace 'Kokkos::Impl::Kokkos'; >>>>>> did you mean simply 'memory_fence'? >>>>>> Kokkos::memory_fence(); >>>>>> ^~~~~~~~~~~~~~~~~~~~ >>>>>> memory_fence >>>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/impl/Kokkos_Memory_Fence.hpp:53:6: >>>>>> note: 'memory_fence' declared here >>>>>> void memory_fence() { >>>>>> ^ >>>>>> >>>>>

1 0

Re: [petsc-dev] Kokkos make error on Spock
by Mark Adams 17 Jul '21

17 Jul '21

HUmm, I can not reproduce this. 10:49 jczhang/fix-cray-mpicxx-includes/main= /gpfs/alpine/csc314/scratch/adams/petsc$ make PETSC_DIR=/gpfs/alpine/csc314/scratch/adams/petsc PETSC_ARCH=arch-spock-dbg-kokkos check Running check examples to verify correct installation Using PETSC_DIR=/gpfs/alpine/csc314/scratch/adams/petsc and PETSC_ARCH=arch-spock-dbg-kokkos gmake[3]: [/gpfs/alpine/csc314/scratch/adams/petsc/lib/petsc/conf/rules:301: ex19.PETSc] Error 2 (ignored) *******************Error detected during compile or link!******************* See http://www.mcs.anl.gov/petsc/documentation/faq.html /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials ex19 ********************************************************************************* cc -L/sw/spock/spack-envs/views/rocm-4.1.0/lib -lamdhip64 -lhsa-runtime64 -fPIC -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -fstack-protector -Qunused-arguments -fvisibility=hidden -g -O0 -fPIC -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -fstack-protector -Qunused-arguments -fvisibility=hidden -g -O0 -I/sw/spock/spack-envs/views/rocm-4.1.0/include -I/gpfs/alpine/csc314/scratch/adams/petsc/include -I/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/include -I/sw/spock/spack-envs/views/rocm-4.1.0/include -I/sw/spock/spack-envs/views/rocm-4.1.0/include ex19.c -Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib -L/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib -Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib -L/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib -Wl,-rpath,/sw/spock/spack-envs/views/rocm-4.1.0/lib -L/sw/spock/spack-envs/views/rocm-4.1.0/lib -Wl,-rpath,/opt/gcc/8.1.0/snos/lib64 -L/opt/gcc/8.1.0/snos/lib64 -Wl,-rpath,/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib -L/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib -L/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/gtl/lib -L/opt/cray/pe/mpich/8.1.4/gtl/lib -Wl,-rpath,/opt/cray/pe/pmi/6.0.10/lib -L/opt/cray/pe/pmi/6.0.10/lib -Wl,-rpath,/opt/cray/pe/dsmml/0.1.4/dsmml/lib -L/opt/cray/pe/dsmml/0.1.4/dsmml/lib -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce/x86_64/lib -L/opt/cray/pe/cce/11.0.4/cce/x86_64/lib -Wl,-rpath,/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64 -L/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64 -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux -L/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux -Wl,-rpath,/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0 -L/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0 -Wl,-rpath,/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib -L/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib -lpetsc -lkokkoskernels -lkokkoscontainers -lkokkoscore -lhipsparse -lhipblas -lrocsparse -lrocsolver -lrocblas -lamdhip64 -lhsa-runtime64 -lstdc++ -ldl -lpmi -lsci_cray_mpi -lsci_cray -lmpifort_cray -lmpi_cray -lmpi_gtl_hsa -lxpmem -ldsmml -lpgas-shmem -lquadmath -lcrayacc_amdgpu -lopenacc -lmodules -lfi -lcraymath -lf -lu -lcsup -lgfortran -lpthread -lgcc_eh -lm -lclang_rt.craypgo-x86_64 -lclang_rt.builtins-x86_64 -lquadmath -lstdc++ -ldl -o ex19 ld.lld: error: /gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib/libpetsc.so: undefined reference to .omp_offloading.img_start.cray_amdgcn-amd-amdhsa [--no-allow-shlib-undefined] ld.lld: error: /gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib/libpetsc.so: undefined reference to .omp_offloading.img_size.cray_amdgcn-amd-amdhsa [--no-allow-shlib-undefined] ld.lld: error: /gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib/libpetsc.so: undefined reference to .omp_offloading.img_cache.cray_amdgcn-amd-amdhsa [--no-allow-shlib-undefined] clang-11: error: linker command failed with exit code 1 (use -v to see invocation) gmake[4]: *** [<builtin>: ex19] Error 1 gmake[3]: [/gpfs/alpine/csc314/scratch/adams/petsc/lib/petsc/conf/rules:301: ex3k.PETSc] Error 2 (ignored) *******************Error detected during compile or link!******************* See http://www.mcs.anl.gov/petsc/documentation/faq.html /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials ex3k ********************************************************************************* hipcc -fPIC -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -fvisibility=hidden -g -O0 -std=c++14 -I/gpfs/alpine/csc314/scratch/adams/petsc/include -I/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/include -I/sw/spock/spack-envs/views/rocm-4.1.0/include -I/sw/spock/spack-envs/views/rocm-4.1.0/include -I/opt/cray/pe/libsci/ 21.04.1.1/CRAY/9.0/x86_64/include -I/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/include -I/opt/cray/pe/pmi/6.0.10/include -I/opt/cray/pe/dsmml/0.1.4/dsmml//include -I/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/include -L/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib -L/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib -L/opt/cray/pe/mpich/8.1.4/gtl/lib -L/opt/cray/pe/pmi/6.0.10/lib -L/opt/cray/pe/dsmml/0.1.4/dsmml//lib -L/opt/cray/pe/cce/11.0.4/cce/x86_64/lib/pkgconfig/../ -L/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64 -Wl,--as-needed,-lsci_cray_mpi,--no-as-needed -Wl,--as-needed,-lsci_cray,--no-as-needed -Wl,--as-needed,-lmpi_cray,--no-as-needed -Wl,--as-needed,-ldsmml,--no-as-needed -Wl,--as-needed,-lpgas-shmem,--no-as-needed -lpmi -ldl -lmpi_gtl_hsa -lxpmem -lquadmath -lmodules -lfi -lcraymath -lf -lu -lcsup ex3k.kokkos.cxx -Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib -L/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib -Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib -L/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib -Wl,-rpath,/sw/spock/spack-envs/views/rocm-4.1.0/lib -L/sw/spock/spack-envs/views/rocm-4.1.0/lib -Wl,-rpath,/opt/gcc/8.1.0/snos/lib64 -L/opt/gcc/8.1.0/snos/lib64 -Wl,-rpath,/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib -L/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib -L/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/gtl/lib -L/opt/cray/pe/mpich/8.1.4/gtl/lib -Wl,-rpath,/opt/cray/pe/pmi/6.0.10/lib -L/opt/cray/pe/pmi/6.0.10/lib -Wl,-rpath,/opt/cray/pe/dsmml/0.1.4/dsmml/lib -L/opt/cray/pe/dsmml/0.1.4/dsmml/lib -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce/x86_64/lib -L/opt/cray/pe/cce/11.0.4/cce/x86_64/lib -Wl,-rpath,/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64 -L/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64 -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux -L/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux -Wl,-rpath,/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0 -L/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0 -Wl,-rpath,/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib -L/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib -lpetsc -lkokkoskernels -lkokkoscontainers -lkokkoscore -lhipsparse -lhipblas -lrocsparse -lrocsolver -lrocblas -lamdhip64 -lhsa-runtime64 -lstdc++ -ldl -lpmi -lsci_cray_mpi -lsci_cray -lmpifort_cray -lmpi_cray -lmpi_gtl_hsa -lxpmem -ldsmml -lpgas-shmem -lquadmath -lcrayacc_amdgpu -lopenacc -lmodules -lfi -lcraymath -lf -lu -lcsup -lgfortran -lpthread -lgcc_eh -lm -lclang_rt.craypgo-x86_64 -lclang_rt.builtins-x86_64 -lquadmath -lstdc++ -ldl -o ex3k /usr/lib64/gcc/x86_64-suse-linux/7/../../../../x86_64-suse-linux/bin/ld: /gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib/libpetsc.so: undefined reference to `.omp_offloading.img_size.cray_amdgcn-amd-amdhsa' /usr/lib64/gcc/x86_64-suse-linux/7/../../../../x86_64-suse-linux/bin/ld: /gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib/libpetsc.so: undefined reference to `.omp_offloading.img_start.cray_amdgcn-amd-amdhsa' /usr/lib64/gcc/x86_64-suse-linux/7/../../../../x86_64-suse-linux/bin/ld: /gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib/libpetsc.so: undefined reference to `.omp_offloading.img_cache.cray_amdgcn-amd-amdhsa' clang-12: error: linker command failed with exit code 1 (use -v to see invocation) gmake[4]: *** [/gpfs/alpine/csc314/scratch/adams/petsc/lib/petsc/conf/test:26: ex3k] Error 1 gmake[3]: [makefile:102: runex3k_kokkos] Error 2 (ignored) 1,25c1,4 < atol=1e-50, rtol=1e-08, stol=1e-08, maxit=50, maxf=10000 < Vec Object: Exact Solution 2 MPI processes < type: mpikokkos < Process [0] < 0. < 0.015625 < 0.125 < Process [1] < 0.421875 < 1. < Vec Object: Forcing function 2 MPI processes < type: mpikokkos < Process [0] < 1e-72 < 1.50024 < 3.01563 < Process [1] < 4.67798 < 7. < 0 SNES Function norm 5.414682427127e+00 < 1 SNES Function norm 2.952582418265e-01 < 2 SNES Function norm 4.502293658739e-04 < 3 SNES Function norm 1.389665806646e-09 < Number of SNES iterations = 3 < Norm of error 1.49752e-10 Iterations 3 --- > slurmstepd: error: execve(): /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials/./ex3k: No such file or directory > slurmstepd: error: execve(): /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials/./ex3k: No such file or directory > srun: error: spock01: tasks 0-1: Exited with exit code 2 > srun: launch/slurm: _step_signal: Terminating StepId=272979.0 /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials Possible problem with ex3k running with kokkos-kernels, diffs above ========================================= *******************Error detected during compile or link!******************* See http://www.mcs.anl.gov/petsc/documentation/faq.html /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials ex5f ********************************************************* ftn -L/sw/spock/spack-envs/views/rocm-4.1.0/lib -lamdhip64 -lhsa-runtime64 -fPIC -g -O0 -fPIC -g -O0 -I/gpfs/alpine/csc314/scratch/adams/petsc/include -I/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/include -I/sw/spock/spack-envs/views/rocm-4.1.0/include ex5f.F90 -Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib -L/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib -Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib -L/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib -Wl,-rpath,/sw/spock/spack-envs/views/rocm-4.1.0/lib -L/sw/spock/spack-envs/views/rocm-4.1.0/lib -Wl,-rpath,/opt/gcc/8.1.0/snos/lib64 -L/opt/gcc/8.1.0/snos/lib64 -Wl,-rpath,/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib -L/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib -L/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/gtl/lib -L/opt/cray/pe/mpich/8.1.4/gtl/lib -Wl,-rpath,/opt/cray/pe/pmi/6.0.10/lib -L/opt/cray/pe/pmi/6.0.10/lib -Wl,-rpath,/opt/cray/pe/dsmml/0.1.4/dsmml/lib -L/opt/cray/pe/dsmml/0.1.4/dsmml/lib -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce/x86_64/lib -L/opt/cray/pe/cce/11.0.4/cce/x86_64/lib -Wl,-rpath,/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64 -L/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64 -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux -L/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux -Wl,-rpath,/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0 -L/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0 -Wl,-rpath,/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib -L/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib -lpetsc -lkokkoskernels -lkokkoscontainers -lkokkoscore -lhipsparse -lhipblas -lrocsparse -lrocsolver -lrocblas -lamdhip64 -lhsa-runtime64 -lstdc++ -ldl -lpmi -lsci_cray_mpi -lsci_cray -lmpifort_cray -lmpi_cray -lmpi_gtl_hsa -lxpmem -ldsmml -lpgas-shmem -lquadmath -lcrayacc_amdgpu -lopenacc -lmodules -lfi -lcraymath -lf -lu -lcsup -lgfortran -lpthread -lgcc_eh -lm -lclang_rt.craypgo-x86_64 -lclang_rt.builtins-x86_64 -lquadmath -lstdc++ -ldl -o ex5f /opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/bin/ld: warning: alignment 128 of symbol `$host_init$$runtime_init_for_iso_c_binding$iso_c_binding_' in /opt/cray/pe/cce/11.0.4/cce/x86_64/lib/libmodules.so is smaller than 256 in /tmp/pe_30429/ex5f_1.o /opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/bin/ld: warning: alignment 64 of symbol `$data_init$iso_c_binding_' in /opt/cray/pe/cce/11.0.4/cce/x86_64/lib/libmodules.so is smaller than 256 in /tmp/pe_30429/ex5f_1.o Fortran example src/snes/tutorials/ex5f run successfully with 1 MPI process Completed test examples 10:49 jczhang/fix-cray-mpicxx-includes/main= /gpfs/alpine/csc314/scratch/adams/petsc$ ll On Fri, Jul 16, 2021 at 11:28 PM Junchao Zhang <junchao.zhang(a)gmail.com> wrote: > Mark, I configured with Fortran binding enabled with main + MR !4175 > <https://gitlab.com/petsc/petsc/-/merge_requests/4175> > > '--with-mpiexec=srun', > '--with-shared-libraries=1', > '--with-cc=cc', > '--with-cxx=CC', > '--with-fc=ftn', > '--with-fortran-bindings', > '--with-hip', > '--with-hipc=hipcc', > '--with-debugging', > '--CPPFLAGS=-I${ROCM_PATH}/include', > '--CXXPPFLAGS=-I${ROCM_PATH}/include', > '--CC_LINKER_FLAGS=-L${ROCM_PATH}/lib -lamdhip64 -lhsa-runtime64', > '--CXX_LINKER_FLAGS=-L${ROCM_PATH}/lib -lamdhip64 -lhsa-runtime64', > '--FC_LINKER_FLAGS=-L${ROCM_PATH}/lib -lamdhip64 -lhsa-runtime64', > '--COPTFLAGS=-g -O0', > '--CXXOPTFLAGS=-g -O0', > '--FOPTFLAGS=-g -O0', > '--download-kokkos', > '--download-kokkos-kernels', > '--download-kokkos-commit=3.4.01', > '--download-kokkos-kernels-commit=3.4.01', > '--with-kokkos-hip-arch=VEGA908', > > and 'make check' ran smoothly on a compute node > > $ make check > Running check examples to verify correct installation > Using PETSC_DIR=/ccs/home/jczhang/petsc and > PETSC_ARCH=arch-spock-cray-kokkos-dbg > C/C++ example src/snes/tutorials/ex19 run successfully with 1 MPI process > C/C++ example src/snes/tutorials/ex19 run successfully with 2 MPI processes > C/C++ example src/snes/tutorials/ex3k run successfully with kokkos-kernels > Fortran example src/snes/tutorials/ex5f run successfully with 1 MPI process > > Completed test examples > > > --Junchao Zhang > > > On Fri, Jul 16, 2021 at 6:04 PM Mark Adams <mfadams(a)lbl.gov> wrote: > >> And I find that this error, on non-Kokkos C tests, is fixed by turning >> the fortran bindings off: >> >> ld.lld: error: >> /gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/lib/libpetsc.so: >> undefined reference to .omp_offloading.img_start.cray_amdgcn-amd-amdhsa >> [--no-allow-shlib-undefined] >> ld.lld: error: >> /gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/lib/libpetsc.so: >> undefined reference to .omp_offloading.img_size.cray_amdgcn-amd-amdhsa >> [--no-allow-shlib-undefined] >> ld.lld: error: >> /gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/lib/libpetsc.so: >> undefined reference to .omp_offloading.img_cache.cray_amdgcn-amd-amdhsa >> [--no-allow-shlib-undefined] >> >> On Fri, Jul 16, 2021 at 3:53 PM Mark Adams <mfadams(a)lbl.gov> wrote: >> >>> Not complex. THis has some overlap with my problem w/o Kokkos. >>> >>> On Fri, Jul 16, 2021 at 12:54 PM Junchao Zhang <junchao.zhang(a)gmail.com> >>> wrote: >>> >>>> Do you use complex? post your configure.log. >>>> >>>> --Junchao Zhang >>>> >>>> >>>> On Fri, Jul 16, 2021 at 9:47 AM Mark Adams <mfadams(a)lbl.gov> wrote: >>>> >>>>> The simple Kokkos example is failing for me on Spock. >>>>> Any ideas? >>>>> Thanks, >>>>> >>>>> 10:44 main *= >>>>> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials$ make >>>>> PETSC_DIR=/gpfs/alpine/csc314/scratch/adams/petsc >>>>> PETSC_ARCH=arch-spock-opt-cray-kokkos ex3k >>>>> MPICH_CXX="hipcc" OMPI_CXX="hipcc" CC >>>>> -L/sw/spock/spack-envs/views/rocm-4.1.0/lib -lhsa-runtime64 >>>>> -L/sw/spock/spack-envs/views/rocm-4.1.0/lib -lamdhip64 -lhsa-runtime64 >>>>> -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas >>>>> -fstack-protector -fvisibility=hidden -g -O2 -fPIC -Wall -Wwrite-strings >>>>> -Wno-strict-aliasing -Wno-unknown-pragmas -fvisibility=hidden -g -O3 >>>>> -std=c++14 -I/gpfs/alpine/csc314/scratch/adams/petsc/include >>>>> -I/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-opt-cray-kokkos/include >>>>> -I/gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include >>>>> -I/sw/spock/spack-envs/views/rocm-4.1.0/include >>>>> -I/sw/spock/spack-envs/views/rocm-4.1.0/include ex3k.kokkos.cxx >>>>> -Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-opt-cray-kokkos/lib >>>>> -L/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-opt-cray-kokkos/lib >>>>> -Wl,-rpath,/gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/lib >>>>> -L/gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/lib >>>>> -Wl,-rpath,/sw/spock/spack-envs/views/rocm-4.1.0/lib >>>>> -L/sw/spock/spack-envs/views/rocm-4.1.0/lib >>>>> -Wl,-rpath,/opt/gcc/8.1.0/snos/lib64 -L/opt/gcc/8.1.0/snos/lib64 >>>>> -Wl,-rpath,/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib >>>>> -L/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib >>>>> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib >>>>> -L/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib >>>>> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/gtl/lib >>>>> -L/opt/cray/pe/mpich/8.1.4/gtl/lib -Wl,-rpath,/opt/cray/pe/pmi/6.0.10/lib >>>>> -L/opt/cray/pe/pmi/6.0.10/lib -Wl,-rpath,/opt/cray/pe/dsmml/0.1.4/dsmml/lib >>>>> -L/opt/cray/pe/dsmml/0.1.4/dsmml/lib >>>>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce/x86_64/lib >>>>> -L/opt/cray/pe/cce/11.0.4/cce/x86_64/lib >>>>> -Wl,-rpath,/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64 >>>>> -L/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64 >>>>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux >>>>> -L/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux >>>>> -Wl,-rpath,/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0 >>>>> -L/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0 >>>>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib >>>>> -L/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib >>>>> -lpetsc -lkokkoskernels -lkokkoscontainers -lkokkoscore -lhipsparse >>>>> -lhipblas -lrocsparse -lrocsolver -lrocblas -lamdhip64 -lhsa-runtime64 >>>>> -lstdc++ -ldl -lpmi -lsci_cray_mpi -lsci_cray -lmpifort_cray -lmpi_cray >>>>> -lmpi_gtl_hsa -lxpmem -ldsmml -lpgas-shmem -lquadmath -lcrayacc_amdgpu >>>>> -lopenacc -lmodules -lfi -lcraymath -lf -lu -lcsup -lgfortran -lpthread >>>>> -lgcc_eh -lm -lclang_rt.craypgo-x86_64 -lclang_rt.builtins-x86_64 >>>>> -lquadmath -lstdc++ -ldl -o ex3k >>>>> In file included from ex3k.kokkos.cxx:3: >>>>> In file included from >>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscdmda_kokkos.hpp:4: >>>>> In file included from >>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec_kokkos.hpp:14: >>>>> In file included from >>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec.h:9: >>>>> In file included from >>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsys.h:42: >>>>> In file included from >>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsystypes.h:255: >>>>> In file included from >>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Complex.hpp:47: >>>>> In file included from >>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Atomic.hpp:212: >>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/impl/Kokkos_Atomic_Exchange.hpp:259:11: >>>>> error: no member named 'lock_address_host_space' in namespace >>>>> 'Kokkos::Impl::Kokkos::Impl'; did you mean simply 'lock_address_host_space'? >>>>> while (!Impl::lock_address_host_space((void*)dest)) >>>>> ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~ >>>>> lock_address_host_space >>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_HostSpace.hpp:83:6: >>>>> note: 'lock_address_host_space' declared here >>>>> bool lock_address_host_space(void* ptr); >>>>> ^ >>>>> In file included from ex3k.kokkos.cxx:3: >>>>> In file included from >>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscdmda_kokkos.hpp:4: >>>>> In file included from >>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec_kokkos.hpp:14: >>>>> In file included from >>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec.h:9: >>>>> In file included from >>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsys.h:42: >>>>> In file included from >>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsystypes.h:255: >>>>> In file included from >>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Complex.hpp:47: >>>>> In file included from >>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Atomic.hpp:212: >>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/impl/Kokkos_Atomic_Exchange.hpp:261:3: >>>>> error: no member named 'memory_fence' in namespace 'Kokkos::Impl::Kokkos'; >>>>> did you mean simply 'memory_fence'? >>>>> Kokkos::memory_fence(); >>>>> ^~~~~~~~~~~~~~~~~~~~ >>>>> memory_fence >>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/impl/Kokkos_Memory_Fence.hpp:53:6: >>>>> note: 'memory_fence' declared here >>>>> void memory_fence() { >>>>> ^ >>>>> In file included from ex3k.kokkos.cxx:3: >>>>> In file included from >>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscdmda_kokkos.hpp:4: >>>>> In file included from >>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec_kokkos.hpp:14: >>>>> In file included from >>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec.h:9: >>>>> In file included from >>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsys.h:42: >>>>> In file included from >>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsystypes.h:255: >>>>> In file included from >>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Complex.hpp:47: >>>>> In file included from >>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Atomic.hpp:212: >>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/impl/Kokkos_Atomic_Exchange.hpp:277:3: >>>>> error: no member named 'memory_fence' in namespace 'Kokkos::Impl::Kokkos'; >>>>> did you mean simply 'memory_fence'? >>>>> Kokkos::memory_fence(); >>>>> ^~~~~~~~~~~~~~~~~~~~ >>>>> memory_fence >>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/impl/Kokkos_Memory_Fence.hpp:53:6: >>>>> note: 'memory_fence' declared here >>>>> void memory_fence() { >>>>> ^ >>>>> >>>>

1 0

Re: [petsc-dev] Kokkos make error on Spock
by Junchao Zhang 16 Jul '21

16 Jul '21

Mark, I configured with Fortran binding enabled with main + MR !4175 <https://gitlab.com/petsc/petsc/-/merge_requests/4175> '--with-mpiexec=srun', '--with-shared-libraries=1', '--with-cc=cc', '--with-cxx=CC', '--with-fc=ftn', '--with-fortran-bindings', '--with-hip', '--with-hipc=hipcc', '--with-debugging', '--CPPFLAGS=-I${ROCM_PATH}/include', '--CXXPPFLAGS=-I${ROCM_PATH}/include', '--CC_LINKER_FLAGS=-L${ROCM_PATH}/lib -lamdhip64 -lhsa-runtime64', '--CXX_LINKER_FLAGS=-L${ROCM_PATH}/lib -lamdhip64 -lhsa-runtime64', '--FC_LINKER_FLAGS=-L${ROCM_PATH}/lib -lamdhip64 -lhsa-runtime64', '--COPTFLAGS=-g -O0', '--CXXOPTFLAGS=-g -O0', '--FOPTFLAGS=-g -O0', '--download-kokkos', '--download-kokkos-kernels', '--download-kokkos-commit=3.4.01', '--download-kokkos-kernels-commit=3.4.01', '--with-kokkos-hip-arch=VEGA908', and 'make check' ran smoothly on a compute node $ make check Running check examples to verify correct installation Using PETSC_DIR=/ccs/home/jczhang/petsc and PETSC_ARCH=arch-spock-cray-kokkos-dbg C/C++ example src/snes/tutorials/ex19 run successfully with 1 MPI process C/C++ example src/snes/tutorials/ex19 run successfully with 2 MPI processes C/C++ example src/snes/tutorials/ex3k run successfully with kokkos-kernels Fortran example src/snes/tutorials/ex5f run successfully with 1 MPI process Completed test examples --Junchao Zhang On Fri, Jul 16, 2021 at 6:04 PM Mark Adams <mfadams(a)lbl.gov> wrote: > And I find that this error, on non-Kokkos C tests, is fixed by turning the > fortran bindings off: > > ld.lld: error: > /gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/lib/libpetsc.so: > undefined reference to .omp_offloading.img_start.cray_amdgcn-amd-amdhsa > [--no-allow-shlib-undefined] > ld.lld: error: > /gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/lib/libpetsc.so: > undefined reference to .omp_offloading.img_size.cray_amdgcn-amd-amdhsa > [--no-allow-shlib-undefined] > ld.lld: error: > /gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/lib/libpetsc.so: > undefined reference to .omp_offloading.img_cache.cray_amdgcn-amd-amdhsa > [--no-allow-shlib-undefined] > > On Fri, Jul 16, 2021 at 3:53 PM Mark Adams <mfadams(a)lbl.gov> wrote: > >> Not complex. THis has some overlap with my problem w/o Kokkos. >> >> On Fri, Jul 16, 2021 at 12:54 PM Junchao Zhang <junchao.zhang(a)gmail.com> >> wrote: >> >>> Do you use complex? post your configure.log. >>> >>> --Junchao Zhang >>> >>> >>> On Fri, Jul 16, 2021 at 9:47 AM Mark Adams <mfadams(a)lbl.gov> wrote: >>> >>>> The simple Kokkos example is failing for me on Spock. >>>> Any ideas? >>>> Thanks, >>>> >>>> 10:44 main *= >>>> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials$ make >>>> PETSC_DIR=/gpfs/alpine/csc314/scratch/adams/petsc >>>> PETSC_ARCH=arch-spock-opt-cray-kokkos ex3k >>>> MPICH_CXX="hipcc" OMPI_CXX="hipcc" CC >>>> -L/sw/spock/spack-envs/views/rocm-4.1.0/lib -lhsa-runtime64 >>>> -L/sw/spock/spack-envs/views/rocm-4.1.0/lib -lamdhip64 -lhsa-runtime64 >>>> -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas >>>> -fstack-protector -fvisibility=hidden -g -O2 -fPIC -Wall -Wwrite-strings >>>> -Wno-strict-aliasing -Wno-unknown-pragmas -fvisibility=hidden -g -O3 >>>> -std=c++14 -I/gpfs/alpine/csc314/scratch/adams/petsc/include >>>> -I/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-opt-cray-kokkos/include >>>> -I/gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include >>>> -I/sw/spock/spack-envs/views/rocm-4.1.0/include >>>> -I/sw/spock/spack-envs/views/rocm-4.1.0/include ex3k.kokkos.cxx >>>> -Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-opt-cray-kokkos/lib >>>> -L/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-opt-cray-kokkos/lib >>>> -Wl,-rpath,/gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/lib >>>> -L/gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/lib >>>> -Wl,-rpath,/sw/spock/spack-envs/views/rocm-4.1.0/lib >>>> -L/sw/spock/spack-envs/views/rocm-4.1.0/lib >>>> -Wl,-rpath,/opt/gcc/8.1.0/snos/lib64 -L/opt/gcc/8.1.0/snos/lib64 >>>> -Wl,-rpath,/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib >>>> -L/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib >>>> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib >>>> -L/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib >>>> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/gtl/lib >>>> -L/opt/cray/pe/mpich/8.1.4/gtl/lib -Wl,-rpath,/opt/cray/pe/pmi/6.0.10/lib >>>> -L/opt/cray/pe/pmi/6.0.10/lib -Wl,-rpath,/opt/cray/pe/dsmml/0.1.4/dsmml/lib >>>> -L/opt/cray/pe/dsmml/0.1.4/dsmml/lib >>>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce/x86_64/lib >>>> -L/opt/cray/pe/cce/11.0.4/cce/x86_64/lib >>>> -Wl,-rpath,/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64 >>>> -L/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64 >>>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux >>>> -L/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux >>>> -Wl,-rpath,/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0 >>>> -L/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0 >>>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib >>>> -L/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib >>>> -lpetsc -lkokkoskernels -lkokkoscontainers -lkokkoscore -lhipsparse >>>> -lhipblas -lrocsparse -lrocsolver -lrocblas -lamdhip64 -lhsa-runtime64 >>>> -lstdc++ -ldl -lpmi -lsci_cray_mpi -lsci_cray -lmpifort_cray -lmpi_cray >>>> -lmpi_gtl_hsa -lxpmem -ldsmml -lpgas-shmem -lquadmath -lcrayacc_amdgpu >>>> -lopenacc -lmodules -lfi -lcraymath -lf -lu -lcsup -lgfortran -lpthread >>>> -lgcc_eh -lm -lclang_rt.craypgo-x86_64 -lclang_rt.builtins-x86_64 >>>> -lquadmath -lstdc++ -ldl -o ex3k >>>> In file included from ex3k.kokkos.cxx:3: >>>> In file included from >>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscdmda_kokkos.hpp:4: >>>> In file included from >>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec_kokkos.hpp:14: >>>> In file included from >>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec.h:9: >>>> In file included from >>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsys.h:42: >>>> In file included from >>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsystypes.h:255: >>>> In file included from >>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Complex.hpp:47: >>>> In file included from >>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Atomic.hpp:212: >>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/impl/Kokkos_Atomic_Exchange.hpp:259:11: >>>> error: no member named 'lock_address_host_space' in namespace >>>> 'Kokkos::Impl::Kokkos::Impl'; did you mean simply 'lock_address_host_space'? >>>> while (!Impl::lock_address_host_space((void*)dest)) >>>> ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~ >>>> lock_address_host_space >>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_HostSpace.hpp:83:6: >>>> note: 'lock_address_host_space' declared here >>>> bool lock_address_host_space(void* ptr); >>>> ^ >>>> In file included from ex3k.kokkos.cxx:3: >>>> In file included from >>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscdmda_kokkos.hpp:4: >>>> In file included from >>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec_kokkos.hpp:14: >>>> In file included from >>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec.h:9: >>>> In file included from >>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsys.h:42: >>>> In file included from >>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsystypes.h:255: >>>> In file included from >>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Complex.hpp:47: >>>> In file included from >>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Atomic.hpp:212: >>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/impl/Kokkos_Atomic_Exchange.hpp:261:3: >>>> error: no member named 'memory_fence' in namespace 'Kokkos::Impl::Kokkos'; >>>> did you mean simply 'memory_fence'? >>>> Kokkos::memory_fence(); >>>> ^~~~~~~~~~~~~~~~~~~~ >>>> memory_fence >>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/impl/Kokkos_Memory_Fence.hpp:53:6: >>>> note: 'memory_fence' declared here >>>> void memory_fence() { >>>> ^ >>>> In file included from ex3k.kokkos.cxx:3: >>>> In file included from >>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscdmda_kokkos.hpp:4: >>>> In file included from >>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec_kokkos.hpp:14: >>>> In file included from >>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec.h:9: >>>> In file included from >>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsys.h:42: >>>> In file included from >>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsystypes.h:255: >>>> In file included from >>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Complex.hpp:47: >>>> In file included from >>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Atomic.hpp:212: >>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/impl/Kokkos_Atomic_Exchange.hpp:277:3: >>>> error: no member named 'memory_fence' in namespace 'Kokkos::Impl::Kokkos'; >>>> did you mean simply 'memory_fence'? >>>> Kokkos::memory_fence(); >>>> ^~~~~~~~~~~~~~~~~~~~ >>>> memory_fence >>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/impl/Kokkos_Memory_Fence.hpp:53:6: >>>> note: 'memory_fence' declared here >>>> void memory_fence() { >>>> ^ >>>> >>>

1 0

Re: [petsc-dev] Kokkos make error on Spock
by Mark Adams 16 Jul '21

16 Jul '21

And I find that this error, on non-Kokkos C tests, is fixed by turning the fortran bindings off: ld.lld: error: /gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/lib/libpetsc.so: undefined reference to .omp_offloading.img_start.cray_amdgcn-amd-amdhsa [--no-allow-shlib-undefined] ld.lld: error: /gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/lib/libpetsc.so: undefined reference to .omp_offloading.img_size.cray_amdgcn-amd-amdhsa [--no-allow-shlib-undefined] ld.lld: error: /gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/lib/libpetsc.so: undefined reference to .omp_offloading.img_cache.cray_amdgcn-amd-amdhsa [--no-allow-shlib-undefined] On Fri, Jul 16, 2021 at 3:53 PM Mark Adams <mfadams(a)lbl.gov> wrote: > Not complex. THis has some overlap with my problem w/o Kokkos. > > On Fri, Jul 16, 2021 at 12:54 PM Junchao Zhang <junchao.zhang(a)gmail.com> > wrote: > >> Do you use complex? post your configure.log. >> >> --Junchao Zhang >> >> >> On Fri, Jul 16, 2021 at 9:47 AM Mark Adams <mfadams(a)lbl.gov> wrote: >> >>> The simple Kokkos example is failing for me on Spock. >>> Any ideas? >>> Thanks, >>> >>> 10:44 main *= >>> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials$ make >>> PETSC_DIR=/gpfs/alpine/csc314/scratch/adams/petsc >>> PETSC_ARCH=arch-spock-opt-cray-kokkos ex3k >>> MPICH_CXX="hipcc" OMPI_CXX="hipcc" CC >>> -L/sw/spock/spack-envs/views/rocm-4.1.0/lib -lhsa-runtime64 >>> -L/sw/spock/spack-envs/views/rocm-4.1.0/lib -lamdhip64 -lhsa-runtime64 >>> -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas >>> -fstack-protector -fvisibility=hidden -g -O2 -fPIC -Wall -Wwrite-strings >>> -Wno-strict-aliasing -Wno-unknown-pragmas -fvisibility=hidden -g -O3 >>> -std=c++14 -I/gpfs/alpine/csc314/scratch/adams/petsc/include >>> -I/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-opt-cray-kokkos/include >>> -I/gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include >>> -I/sw/spock/spack-envs/views/rocm-4.1.0/include >>> -I/sw/spock/spack-envs/views/rocm-4.1.0/include ex3k.kokkos.cxx >>> -Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-opt-cray-kokkos/lib >>> -L/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-opt-cray-kokkos/lib >>> -Wl,-rpath,/gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/lib >>> -L/gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/lib >>> -Wl,-rpath,/sw/spock/spack-envs/views/rocm-4.1.0/lib >>> -L/sw/spock/spack-envs/views/rocm-4.1.0/lib >>> -Wl,-rpath,/opt/gcc/8.1.0/snos/lib64 -L/opt/gcc/8.1.0/snos/lib64 >>> -Wl,-rpath,/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib >>> -L/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib >>> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib >>> -L/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib >>> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/gtl/lib >>> -L/opt/cray/pe/mpich/8.1.4/gtl/lib -Wl,-rpath,/opt/cray/pe/pmi/6.0.10/lib >>> -L/opt/cray/pe/pmi/6.0.10/lib -Wl,-rpath,/opt/cray/pe/dsmml/0.1.4/dsmml/lib >>> -L/opt/cray/pe/dsmml/0.1.4/dsmml/lib >>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce/x86_64/lib >>> -L/opt/cray/pe/cce/11.0.4/cce/x86_64/lib >>> -Wl,-rpath,/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64 >>> -L/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64 >>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux >>> -L/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux >>> -Wl,-rpath,/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0 >>> -L/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0 >>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib >>> -L/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib >>> -lpetsc -lkokkoskernels -lkokkoscontainers -lkokkoscore -lhipsparse >>> -lhipblas -lrocsparse -lrocsolver -lrocblas -lamdhip64 -lhsa-runtime64 >>> -lstdc++ -ldl -lpmi -lsci_cray_mpi -lsci_cray -lmpifort_cray -lmpi_cray >>> -lmpi_gtl_hsa -lxpmem -ldsmml -lpgas-shmem -lquadmath -lcrayacc_amdgpu >>> -lopenacc -lmodules -lfi -lcraymath -lf -lu -lcsup -lgfortran -lpthread >>> -lgcc_eh -lm -lclang_rt.craypgo-x86_64 -lclang_rt.builtins-x86_64 >>> -lquadmath -lstdc++ -ldl -o ex3k >>> In file included from ex3k.kokkos.cxx:3: >>> In file included from >>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscdmda_kokkos.hpp:4: >>> In file included from >>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec_kokkos.hpp:14: >>> In file included from >>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec.h:9: >>> In file included from >>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsys.h:42: >>> In file included from >>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsystypes.h:255: >>> In file included from >>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Complex.hpp:47: >>> In file included from >>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Atomic.hpp:212: >>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/impl/Kokkos_Atomic_Exchange.hpp:259:11: >>> error: no member named 'lock_address_host_space' in namespace >>> 'Kokkos::Impl::Kokkos::Impl'; did you mean simply 'lock_address_host_space'? >>> while (!Impl::lock_address_host_space((void*)dest)) >>> ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~ >>> lock_address_host_space >>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_HostSpace.hpp:83:6: >>> note: 'lock_address_host_space' declared here >>> bool lock_address_host_space(void* ptr); >>> ^ >>> In file included from ex3k.kokkos.cxx:3: >>> In file included from >>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscdmda_kokkos.hpp:4: >>> In file included from >>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec_kokkos.hpp:14: >>> In file included from >>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec.h:9: >>> In file included from >>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsys.h:42: >>> In file included from >>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsystypes.h:255: >>> In file included from >>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Complex.hpp:47: >>> In file included from >>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Atomic.hpp:212: >>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/impl/Kokkos_Atomic_Exchange.hpp:261:3: >>> error: no member named 'memory_fence' in namespace 'Kokkos::Impl::Kokkos'; >>> did you mean simply 'memory_fence'? >>> Kokkos::memory_fence(); >>> ^~~~~~~~~~~~~~~~~~~~ >>> memory_fence >>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/impl/Kokkos_Memory_Fence.hpp:53:6: >>> note: 'memory_fence' declared here >>> void memory_fence() { >>> ^ >>> In file included from ex3k.kokkos.cxx:3: >>> In file included from >>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscdmda_kokkos.hpp:4: >>> In file included from >>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec_kokkos.hpp:14: >>> In file included from >>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec.h:9: >>> In file included from >>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsys.h:42: >>> In file included from >>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsystypes.h:255: >>> In file included from >>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Complex.hpp:47: >>> In file included from >>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Atomic.hpp:212: >>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/impl/Kokkos_Atomic_Exchange.hpp:277:3: >>> error: no member named 'memory_fence' in namespace 'Kokkos::Impl::Kokkos'; >>> did you mean simply 'memory_fence'? >>> Kokkos::memory_fence(); >>> ^~~~~~~~~~~~~~~~~~~~ >>> memory_fence >>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/impl/Kokkos_Memory_Fence.hpp:53:6: >>> note: 'memory_fence' declared here >>> void memory_fence() { >>> ^ >>> >>

1 0

Re: [petsc-dev] Kokkos make error on Spock
by Mark Adams 16 Jul '21

16 Jul '21

Ha, Landau is working with the Kokkos backend and it is running the the GPU! My Kokkos build was clean, but let me try it again. On Fri, Jul 16, 2021 at 5:19 PM Junchao Zhang <junchao.zhang(a)gmail.com> wrote: > I don't understand this when linking ex19 > ld.lld: error: > /gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-opt-cray-kokkos/lib/libpetsc.so: > undefined reference to .omp_offloading.img_start.cray_amdgcn-amd-amdhsa > [--no-allow-shlib-undefined] > Do you have a stale libpetsc.so? > > For ex3k, it is a makefile problem (I am fixing). If you do not directly > build an executable from *.kokkos.cxx, then you can avoid this problem. > For example, snes/tests/ex13 works with kokkos options on Spock. > > --Junchao Zhang > > > On Fri, Jul 16, 2021 at 2:53 PM Mark Adams <mfadams(a)lbl.gov> wrote: > >> Not complex. THis has some overlap with my problem w/o Kokkos. >> >> On Fri, Jul 16, 2021 at 12:54 PM Junchao Zhang <junchao.zhang(a)gmail.com> >> wrote: >> >>> Do you use complex? post your configure.log. >>> >>> --Junchao Zhang >>> >>> >>> On Fri, Jul 16, 2021 at 9:47 AM Mark Adams <mfadams(a)lbl.gov> wrote: >>> >>>> The simple Kokkos example is failing for me on Spock. >>>> Any ideas? >>>> Thanks, >>>> >>>> 10:44 main *= >>>> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials$ make >>>> PETSC_DIR=/gpfs/alpine/csc314/scratch/adams/petsc >>>> PETSC_ARCH=arch-spock-opt-cray-kokkos ex3k >>>> MPICH_CXX="hipcc" OMPI_CXX="hipcc" CC >>>> -L/sw/spock/spack-envs/views/rocm-4.1.0/lib -lhsa-runtime64 >>>> -L/sw/spock/spack-envs/views/rocm-4.1.0/lib -lamdhip64 -lhsa-runtime64 >>>> -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas >>>> -fstack-protector -fvisibility=hidden -g -O2 -fPIC -Wall -Wwrite-strings >>>> -Wno-strict-aliasing -Wno-unknown-pragmas -fvisibility=hidden -g -O3 >>>> -std=c++14 -I/gpfs/alpine/csc314/scratch/adams/petsc/include >>>> -I/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-opt-cray-kokkos/include >>>> -I/gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include >>>> -I/sw/spock/spack-envs/views/rocm-4.1.0/include >>>> -I/sw/spock/spack-envs/views/rocm-4.1.0/include ex3k.kokkos.cxx >>>> -Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-opt-cray-kokkos/lib >>>> -L/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-opt-cray-kokkos/lib >>>> -Wl,-rpath,/gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/lib >>>> -L/gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/lib >>>> -Wl,-rpath,/sw/spock/spack-envs/views/rocm-4.1.0/lib >>>> -L/sw/spock/spack-envs/views/rocm-4.1.0/lib >>>> -Wl,-rpath,/opt/gcc/8.1.0/snos/lib64 -L/opt/gcc/8.1.0/snos/lib64 >>>> -Wl,-rpath,/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib >>>> -L/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib >>>> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib >>>> -L/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib >>>> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/gtl/lib >>>> -L/opt/cray/pe/mpich/8.1.4/gtl/lib -Wl,-rpath,/opt/cray/pe/pmi/6.0.10/lib >>>> -L/opt/cray/pe/pmi/6.0.10/lib -Wl,-rpath,/opt/cray/pe/dsmml/0.1.4/dsmml/lib >>>> -L/opt/cray/pe/dsmml/0.1.4/dsmml/lib >>>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce/x86_64/lib >>>> -L/opt/cray/pe/cce/11.0.4/cce/x86_64/lib >>>> -Wl,-rpath,/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64 >>>> -L/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64 >>>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux >>>> -L/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux >>>> -Wl,-rpath,/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0 >>>> -L/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0 >>>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib >>>> -L/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib >>>> -lpetsc -lkokkoskernels -lkokkoscontainers -lkokkoscore -lhipsparse >>>> -lhipblas -lrocsparse -lrocsolver -lrocblas -lamdhip64 -lhsa-runtime64 >>>> -lstdc++ -ldl -lpmi -lsci_cray_mpi -lsci_cray -lmpifort_cray -lmpi_cray >>>> -lmpi_gtl_hsa -lxpmem -ldsmml -lpgas-shmem -lquadmath -lcrayacc_amdgpu >>>> -lopenacc -lmodules -lfi -lcraymath -lf -lu -lcsup -lgfortran -lpthread >>>> -lgcc_eh -lm -lclang_rt.craypgo-x86_64 -lclang_rt.builtins-x86_64 >>>> -lquadmath -lstdc++ -ldl -o ex3k >>>> In file included from ex3k.kokkos.cxx:3: >>>> In file included from >>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscdmda_kokkos.hpp:4: >>>> In file included from >>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec_kokkos.hpp:14: >>>> In file included from >>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec.h:9: >>>> In file included from >>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsys.h:42: >>>> In file included from >>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsystypes.h:255: >>>> In file included from >>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Complex.hpp:47: >>>> In file included from >>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Atomic.hpp:212: >>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/impl/Kokkos_Atomic_Exchange.hpp:259:11: >>>> error: no member named 'lock_address_host_space' in namespace >>>> 'Kokkos::Impl::Kokkos::Impl'; did you mean simply 'lock_address_host_space'? >>>> while (!Impl::lock_address_host_space((void*)dest)) >>>> ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~ >>>> lock_address_host_space >>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_HostSpace.hpp:83:6: >>>> note: 'lock_address_host_space' declared here >>>> bool lock_address_host_space(void* ptr); >>>> ^ >>>> In file included from ex3k.kokkos.cxx:3: >>>> In file included from >>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscdmda_kokkos.hpp:4: >>>> In file included from >>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec_kokkos.hpp:14: >>>> In file included from >>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec.h:9: >>>> In file included from >>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsys.h:42: >>>> In file included from >>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsystypes.h:255: >>>> In file included from >>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Complex.hpp:47: >>>> In file included from >>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Atomic.hpp:212: >>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/impl/Kokkos_Atomic_Exchange.hpp:261:3: >>>> error: no member named 'memory_fence' in namespace 'Kokkos::Impl::Kokkos'; >>>> did you mean simply 'memory_fence'? >>>> Kokkos::memory_fence(); >>>> ^~~~~~~~~~~~~~~~~~~~ >>>> memory_fence >>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/impl/Kokkos_Memory_Fence.hpp:53:6: >>>> note: 'memory_fence' declared here >>>> void memory_fence() { >>>> ^ >>>> In file included from ex3k.kokkos.cxx:3: >>>> In file included from >>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscdmda_kokkos.hpp:4: >>>> In file included from >>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec_kokkos.hpp:14: >>>> In file included from >>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec.h:9: >>>> In file included from >>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsys.h:42: >>>> In file included from >>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsystypes.h:255: >>>> In file included from >>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Complex.hpp:47: >>>> In file included from >>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Atomic.hpp:212: >>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/impl/Kokkos_Atomic_Exchange.hpp:277:3: >>>> error: no member named 'memory_fence' in namespace 'Kokkos::Impl::Kokkos'; >>>> did you mean simply 'memory_fence'? >>>> Kokkos::memory_fence(); >>>> ^~~~~~~~~~~~~~~~~~~~ >>>> memory_fence >>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/impl/Kokkos_Memory_Fence.hpp:53:6: >>>> note: 'memory_fence' declared here >>>> void memory_fence() { >>>> ^ >>>> >>>

1 0

Re: [petsc-dev] Kokkos make error on Spock
by Junchao Zhang 16 Jul '21

16 Jul '21

I don't understand this when linking ex19 ld.lld: error: /gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-opt-cray-kokkos/lib/libpetsc.so: undefined reference to .omp_offloading.img_start.cray_amdgcn-amd-amdhsa [--no-allow-shlib-undefined] Do you have a stale libpetsc.so? For ex3k, it is a makefile problem (I am fixing). If you do not directly build an executable from *.kokkos.cxx, then you can avoid this problem. For example, snes/tests/ex13 works with kokkos options on Spock. --Junchao Zhang On Fri, Jul 16, 2021 at 2:53 PM Mark Adams <mfadams(a)lbl.gov> wrote: > Not complex. THis has some overlap with my problem w/o Kokkos. > > On Fri, Jul 16, 2021 at 12:54 PM Junchao Zhang <junchao.zhang(a)gmail.com> > wrote: > >> Do you use complex? post your configure.log. >> >> --Junchao Zhang >> >> >> On Fri, Jul 16, 2021 at 9:47 AM Mark Adams <mfadams(a)lbl.gov> wrote: >> >>> The simple Kokkos example is failing for me on Spock. >>> Any ideas? >>> Thanks, >>> >>> 10:44 main *= >>> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials$ make >>> PETSC_DIR=/gpfs/alpine/csc314/scratch/adams/petsc >>> PETSC_ARCH=arch-spock-opt-cray-kokkos ex3k >>> MPICH_CXX="hipcc" OMPI_CXX="hipcc" CC >>> -L/sw/spock/spack-envs/views/rocm-4.1.0/lib -lhsa-runtime64 >>> -L/sw/spock/spack-envs/views/rocm-4.1.0/lib -lamdhip64 -lhsa-runtime64 >>> -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas >>> -fstack-protector -fvisibility=hidden -g -O2 -fPIC -Wall -Wwrite-strings >>> -Wno-strict-aliasing -Wno-unknown-pragmas -fvisibility=hidden -g -O3 >>> -std=c++14 -I/gpfs/alpine/csc314/scratch/adams/petsc/include >>> -I/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-opt-cray-kokkos/include >>> -I/gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include >>> -I/sw/spock/spack-envs/views/rocm-4.1.0/include >>> -I/sw/spock/spack-envs/views/rocm-4.1.0/include ex3k.kokkos.cxx >>> -Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-opt-cray-kokkos/lib >>> -L/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-opt-cray-kokkos/lib >>> -Wl,-rpath,/gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/lib >>> -L/gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/lib >>> -Wl,-rpath,/sw/spock/spack-envs/views/rocm-4.1.0/lib >>> -L/sw/spock/spack-envs/views/rocm-4.1.0/lib >>> -Wl,-rpath,/opt/gcc/8.1.0/snos/lib64 -L/opt/gcc/8.1.0/snos/lib64 >>> -Wl,-rpath,/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib >>> -L/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib >>> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib >>> -L/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib >>> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/gtl/lib >>> -L/opt/cray/pe/mpich/8.1.4/gtl/lib -Wl,-rpath,/opt/cray/pe/pmi/6.0.10/lib >>> -L/opt/cray/pe/pmi/6.0.10/lib -Wl,-rpath,/opt/cray/pe/dsmml/0.1.4/dsmml/lib >>> -L/opt/cray/pe/dsmml/0.1.4/dsmml/lib >>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce/x86_64/lib >>> -L/opt/cray/pe/cce/11.0.4/cce/x86_64/lib >>> -Wl,-rpath,/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64 >>> -L/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64 >>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux >>> -L/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux >>> -Wl,-rpath,/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0 >>> -L/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0 >>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib >>> -L/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib >>> -lpetsc -lkokkoskernels -lkokkoscontainers -lkokkoscore -lhipsparse >>> -lhipblas -lrocsparse -lrocsolver -lrocblas -lamdhip64 -lhsa-runtime64 >>> -lstdc++ -ldl -lpmi -lsci_cray_mpi -lsci_cray -lmpifort_cray -lmpi_cray >>> -lmpi_gtl_hsa -lxpmem -ldsmml -lpgas-shmem -lquadmath -lcrayacc_amdgpu >>> -lopenacc -lmodules -lfi -lcraymath -lf -lu -lcsup -lgfortran -lpthread >>> -lgcc_eh -lm -lclang_rt.craypgo-x86_64 -lclang_rt.builtins-x86_64 >>> -lquadmath -lstdc++ -ldl -o ex3k >>> In file included from ex3k.kokkos.cxx:3: >>> In file included from >>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscdmda_kokkos.hpp:4: >>> In file included from >>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec_kokkos.hpp:14: >>> In file included from >>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec.h:9: >>> In file included from >>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsys.h:42: >>> In file included from >>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsystypes.h:255: >>> In file included from >>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Complex.hpp:47: >>> In file included from >>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Atomic.hpp:212: >>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/impl/Kokkos_Atomic_Exchange.hpp:259:11: >>> error: no member named 'lock_address_host_space' in namespace >>> 'Kokkos::Impl::Kokkos::Impl'; did you mean simply 'lock_address_host_space'? >>> while (!Impl::lock_address_host_space((void*)dest)) >>> ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~ >>> lock_address_host_space >>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_HostSpace.hpp:83:6: >>> note: 'lock_address_host_space' declared here >>> bool lock_address_host_space(void* ptr); >>> ^ >>> In file included from ex3k.kokkos.cxx:3: >>> In file included from >>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscdmda_kokkos.hpp:4: >>> In file included from >>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec_kokkos.hpp:14: >>> In file included from >>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec.h:9: >>> In file included from >>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsys.h:42: >>> In file included from >>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsystypes.h:255: >>> In file included from >>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Complex.hpp:47: >>> In file included from >>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Atomic.hpp:212: >>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/impl/Kokkos_Atomic_Exchange.hpp:261:3: >>> error: no member named 'memory_fence' in namespace 'Kokkos::Impl::Kokkos'; >>> did you mean simply 'memory_fence'? >>> Kokkos::memory_fence(); >>> ^~~~~~~~~~~~~~~~~~~~ >>> memory_fence >>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/impl/Kokkos_Memory_Fence.hpp:53:6: >>> note: 'memory_fence' declared here >>> void memory_fence() { >>> ^ >>> In file included from ex3k.kokkos.cxx:3: >>> In file included from >>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscdmda_kokkos.hpp:4: >>> In file included from >>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec_kokkos.hpp:14: >>> In file included from >>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec.h:9: >>> In file included from >>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsys.h:42: >>> In file included from >>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsystypes.h:255: >>> In file included from >>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Complex.hpp:47: >>> In file included from >>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Atomic.hpp:212: >>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/impl/Kokkos_Atomic_Exchange.hpp:277:3: >>> error: no member named 'memory_fence' in namespace 'Kokkos::Impl::Kokkos'; >>> did you mean simply 'memory_fence'? >>> Kokkos::memory_fence(); >>> ^~~~~~~~~~~~~~~~~~~~ >>> memory_fence >>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/impl/Kokkos_Memory_Fence.hpp:53:6: >>> note: 'memory_fence' declared here >>> void memory_fence() { >>> ^ >>> >>

1 0

Re: [petsc-dev] Kokkos make error on Spock
by Mark Adams 16 Jul '21

16 Jul '21

Not complex. THis has some overlap with my problem w/o Kokkos. On Fri, Jul 16, 2021 at 12:54 PM Junchao Zhang <junchao.zhang(a)gmail.com> wrote: > Do you use complex? post your configure.log. > > --Junchao Zhang > > > On Fri, Jul 16, 2021 at 9:47 AM Mark Adams <mfadams(a)lbl.gov> wrote: > >> The simple Kokkos example is failing for me on Spock. >> Any ideas? >> Thanks, >> >> 10:44 main *= /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials$ >> make PETSC_DIR=/gpfs/alpine/csc314/scratch/adams/petsc >> PETSC_ARCH=arch-spock-opt-cray-kokkos ex3k >> MPICH_CXX="hipcc" OMPI_CXX="hipcc" CC >> -L/sw/spock/spack-envs/views/rocm-4.1.0/lib -lhsa-runtime64 >> -L/sw/spock/spack-envs/views/rocm-4.1.0/lib -lamdhip64 -lhsa-runtime64 >> -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas >> -fstack-protector -fvisibility=hidden -g -O2 -fPIC -Wall -Wwrite-strings >> -Wno-strict-aliasing -Wno-unknown-pragmas -fvisibility=hidden -g -O3 >> -std=c++14 -I/gpfs/alpine/csc314/scratch/adams/petsc/include >> -I/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-opt-cray-kokkos/include >> -I/gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include >> -I/sw/spock/spack-envs/views/rocm-4.1.0/include >> -I/sw/spock/spack-envs/views/rocm-4.1.0/include ex3k.kokkos.cxx >> -Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-opt-cray-kokkos/lib >> -L/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-opt-cray-kokkos/lib >> -Wl,-rpath,/gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/lib >> -L/gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/lib >> -Wl,-rpath,/sw/spock/spack-envs/views/rocm-4.1.0/lib >> -L/sw/spock/spack-envs/views/rocm-4.1.0/lib >> -Wl,-rpath,/opt/gcc/8.1.0/snos/lib64 -L/opt/gcc/8.1.0/snos/lib64 >> -Wl,-rpath,/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib >> -L/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib >> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib >> -L/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib >> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/gtl/lib >> -L/opt/cray/pe/mpich/8.1.4/gtl/lib -Wl,-rpath,/opt/cray/pe/pmi/6.0.10/lib >> -L/opt/cray/pe/pmi/6.0.10/lib -Wl,-rpath,/opt/cray/pe/dsmml/0.1.4/dsmml/lib >> -L/opt/cray/pe/dsmml/0.1.4/dsmml/lib >> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce/x86_64/lib >> -L/opt/cray/pe/cce/11.0.4/cce/x86_64/lib >> -Wl,-rpath,/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64 >> -L/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64 >> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux >> -L/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux >> -Wl,-rpath,/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0 >> -L/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0 >> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib >> -L/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib >> -lpetsc -lkokkoskernels -lkokkoscontainers -lkokkoscore -lhipsparse >> -lhipblas -lrocsparse -lrocsolver -lrocblas -lamdhip64 -lhsa-runtime64 >> -lstdc++ -ldl -lpmi -lsci_cray_mpi -lsci_cray -lmpifort_cray -lmpi_cray >> -lmpi_gtl_hsa -lxpmem -ldsmml -lpgas-shmem -lquadmath -lcrayacc_amdgpu >> -lopenacc -lmodules -lfi -lcraymath -lf -lu -lcsup -lgfortran -lpthread >> -lgcc_eh -lm -lclang_rt.craypgo-x86_64 -lclang_rt.builtins-x86_64 >> -lquadmath -lstdc++ -ldl -o ex3k >> In file included from ex3k.kokkos.cxx:3: >> In file included from >> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscdmda_kokkos.hpp:4: >> In file included from >> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec_kokkos.hpp:14: >> In file included from >> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec.h:9: >> In file included from >> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsys.h:42: >> In file included from >> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsystypes.h:255: >> In file included from >> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Complex.hpp:47: >> In file included from >> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Atomic.hpp:212: >> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/impl/Kokkos_Atomic_Exchange.hpp:259:11: >> error: no member named 'lock_address_host_space' in namespace >> 'Kokkos::Impl::Kokkos::Impl'; did you mean simply 'lock_address_host_space'? >> while (!Impl::lock_address_host_space((void*)dest)) >> ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~ >> lock_address_host_space >> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_HostSpace.hpp:83:6: >> note: 'lock_address_host_space' declared here >> bool lock_address_host_space(void* ptr); >> ^ >> In file included from ex3k.kokkos.cxx:3: >> In file included from >> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscdmda_kokkos.hpp:4: >> In file included from >> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec_kokkos.hpp:14: >> In file included from >> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec.h:9: >> In file included from >> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsys.h:42: >> In file included from >> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsystypes.h:255: >> In file included from >> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Complex.hpp:47: >> In file included from >> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Atomic.hpp:212: >> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/impl/Kokkos_Atomic_Exchange.hpp:261:3: >> error: no member named 'memory_fence' in namespace 'Kokkos::Impl::Kokkos'; >> did you mean simply 'memory_fence'? >> Kokkos::memory_fence(); >> ^~~~~~~~~~~~~~~~~~~~ >> memory_fence >> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/impl/Kokkos_Memory_Fence.hpp:53:6: >> note: 'memory_fence' declared here >> void memory_fence() { >> ^ >> In file included from ex3k.kokkos.cxx:3: >> In file included from >> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscdmda_kokkos.hpp:4: >> In file included from >> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec_kokkos.hpp:14: >> In file included from >> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec.h:9: >> In file included from >> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsys.h:42: >> In file included from >> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsystypes.h:255: >> In file included from >> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Complex.hpp:47: >> In file included from >> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Atomic.hpp:212: >> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/impl/Kokkos_Atomic_Exchange.hpp:277:3: >> error: no member named 'memory_fence' in namespace 'Kokkos::Impl::Kokkos'; >> did you mean simply 'memory_fence'? >> Kokkos::memory_fence(); >> ^~~~~~~~~~~~~~~~~~~~ >> memory_fence >> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/impl/Kokkos_Memory_Fence.hpp:53:6: >> note: 'memory_fence' declared here >> void memory_fence() { >> ^ >> >

1 0

Re: [petsc-dev] Kokkos make error on Spock
by Junchao Zhang 16 Jul '21

16 Jul '21

Do you use complex? post your configure.log. --Junchao Zhang On Fri, Jul 16, 2021 at 9:47 AM Mark Adams <mfadams(a)lbl.gov> wrote: > The simple Kokkos example is failing for me on Spock. > Any ideas? > Thanks, > > 10:44 main *= /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials$ > make PETSC_DIR=/gpfs/alpine/csc314/scratch/adams/petsc > PETSC_ARCH=arch-spock-opt-cray-kokkos ex3k > MPICH_CXX="hipcc" OMPI_CXX="hipcc" CC > -L/sw/spock/spack-envs/views/rocm-4.1.0/lib -lhsa-runtime64 > -L/sw/spock/spack-envs/views/rocm-4.1.0/lib -lamdhip64 -lhsa-runtime64 > -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas > -fstack-protector -fvisibility=hidden -g -O2 -fPIC -Wall -Wwrite-strings > -Wno-strict-aliasing -Wno-unknown-pragmas -fvisibility=hidden -g -O3 > -std=c++14 -I/gpfs/alpine/csc314/scratch/adams/petsc/include > -I/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-opt-cray-kokkos/include > -I/gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include > -I/sw/spock/spack-envs/views/rocm-4.1.0/include > -I/sw/spock/spack-envs/views/rocm-4.1.0/include ex3k.kokkos.cxx > -Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-opt-cray-kokkos/lib > -L/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-opt-cray-kokkos/lib > -Wl,-rpath,/gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/lib > -L/gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/lib > -Wl,-rpath,/sw/spock/spack-envs/views/rocm-4.1.0/lib > -L/sw/spock/spack-envs/views/rocm-4.1.0/lib > -Wl,-rpath,/opt/gcc/8.1.0/snos/lib64 -L/opt/gcc/8.1.0/snos/lib64 > -Wl,-rpath,/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib > -L/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib > -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib > -L/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib > -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/gtl/lib > -L/opt/cray/pe/mpich/8.1.4/gtl/lib -Wl,-rpath,/opt/cray/pe/pmi/6.0.10/lib > -L/opt/cray/pe/pmi/6.0.10/lib -Wl,-rpath,/opt/cray/pe/dsmml/0.1.4/dsmml/lib > -L/opt/cray/pe/dsmml/0.1.4/dsmml/lib > -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce/x86_64/lib > -L/opt/cray/pe/cce/11.0.4/cce/x86_64/lib > -Wl,-rpath,/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64 > -L/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64 > -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux > -L/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux > -Wl,-rpath,/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0 > -L/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0 > -Wl,-rpath,/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib > -L/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib > -lpetsc -lkokkoskernels -lkokkoscontainers -lkokkoscore -lhipsparse > -lhipblas -lrocsparse -lrocsolver -lrocblas -lamdhip64 -lhsa-runtime64 > -lstdc++ -ldl -lpmi -lsci_cray_mpi -lsci_cray -lmpifort_cray -lmpi_cray > -lmpi_gtl_hsa -lxpmem -ldsmml -lpgas-shmem -lquadmath -lcrayacc_amdgpu > -lopenacc -lmodules -lfi -lcraymath -lf -lu -lcsup -lgfortran -lpthread > -lgcc_eh -lm -lclang_rt.craypgo-x86_64 -lclang_rt.builtins-x86_64 > -lquadmath -lstdc++ -ldl -o ex3k > In file included from ex3k.kokkos.cxx:3: > In file included from > /gpfs/alpine/csc314/scratch/adams/petsc/include/petscdmda_kokkos.hpp:4: > In file included from > /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec_kokkos.hpp:14: > In file included from > /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec.h:9: > In file included from > /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsys.h:42: > In file included from > /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsystypes.h:255: > In file included from > /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Complex.hpp:47: > In file included from > /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Atomic.hpp:212: > /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/impl/Kokkos_Atomic_Exchange.hpp:259:11: > error: no member named 'lock_address_host_space' in namespace > 'Kokkos::Impl::Kokkos::Impl'; did you mean simply 'lock_address_host_space'? > while (!Impl::lock_address_host_space((void*)dest)) > ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~ > lock_address_host_space > /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_HostSpace.hpp:83:6: > note: 'lock_address_host_space' declared here > bool lock_address_host_space(void* ptr); > ^ > In file included from ex3k.kokkos.cxx:3: > In file included from > /gpfs/alpine/csc314/scratch/adams/petsc/include/petscdmda_kokkos.hpp:4: > In file included from > /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec_kokkos.hpp:14: > In file included from > /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec.h:9: > In file included from > /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsys.h:42: > In file included from > /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsystypes.h:255: > In file included from > /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Complex.hpp:47: > In file included from > /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Atomic.hpp:212: > /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/impl/Kokkos_Atomic_Exchange.hpp:261:3: > error: no member named 'memory_fence' in namespace 'Kokkos::Impl::Kokkos'; > did you mean simply 'memory_fence'? > Kokkos::memory_fence(); > ^~~~~~~~~~~~~~~~~~~~ > memory_fence > /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/impl/Kokkos_Memory_Fence.hpp:53:6: > note: 'memory_fence' declared here > void memory_fence() { > ^ > In file included from ex3k.kokkos.cxx:3: > In file included from > /gpfs/alpine/csc314/scratch/adams/petsc/include/petscdmda_kokkos.hpp:4: > In file included from > /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec_kokkos.hpp:14: > In file included from > /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec.h:9: > In file included from > /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsys.h:42: > In file included from > /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsystypes.h:255: > In file included from > /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Complex.hpp:47: > In file included from > /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Atomic.hpp:212: > /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/impl/Kokkos_Atomic_Exchange.hpp:277:3: > error: no member named 'memory_fence' in namespace 'Kokkos::Impl::Kokkos'; > did you mean simply 'memory_fence'? > Kokkos::memory_fence(); > ^~~~~~~~~~~~~~~~~~~~ > memory_fence > /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/impl/Kokkos_Memory_Fence.hpp:53:6: > note: 'memory_fence' declared here > void memory_fence() { > ^ >

1 0

Kokkos make error on Spock
by Mark Adams 16 Jul '21

16 Jul '21

The simple Kokkos example is failing for me on Spock. Any ideas? Thanks, 10:44 main *= /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials$ make PETSC_DIR=/gpfs/alpine/csc314/scratch/adams/petsc PETSC_ARCH=arch-spock-opt-cray-kokkos ex3k MPICH_CXX="hipcc" OMPI_CXX="hipcc" CC -L/sw/spock/spack-envs/views/rocm-4.1.0/lib -lhsa-runtime64 -L/sw/spock/spack-envs/views/rocm-4.1.0/lib -lamdhip64 -lhsa-runtime64 -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -fstack-protector -fvisibility=hidden -g -O2 -fPIC -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -fvisibility=hidden -g -O3 -std=c++14 -I/gpfs/alpine/csc314/scratch/adams/petsc/include -I/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-opt-cray-kokkos/include -I/gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include -I/sw/spock/spack-envs/views/rocm-4.1.0/include -I/sw/spock/spack-envs/views/rocm-4.1.0/include ex3k.kokkos.cxx -Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-opt-cray-kokkos/lib -L/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-opt-cray-kokkos/lib -Wl,-rpath,/gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/lib -L/gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/lib -Wl,-rpath,/sw/spock/spack-envs/views/rocm-4.1.0/lib -L/sw/spock/spack-envs/views/rocm-4.1.0/lib -Wl,-rpath,/opt/gcc/8.1.0/snos/lib64 -L/opt/gcc/8.1.0/snos/lib64 -Wl,-rpath,/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib -L/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib -L/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/gtl/lib -L/opt/cray/pe/mpich/8.1.4/gtl/lib -Wl,-rpath,/opt/cray/pe/pmi/6.0.10/lib -L/opt/cray/pe/pmi/6.0.10/lib -Wl,-rpath,/opt/cray/pe/dsmml/0.1.4/dsmml/lib -L/opt/cray/pe/dsmml/0.1.4/dsmml/lib -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce/x86_64/lib -L/opt/cray/pe/cce/11.0.4/cce/x86_64/lib -Wl,-rpath,/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64 -L/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64 -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux -L/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux -Wl,-rpath,/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0 -L/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0 -Wl,-rpath,/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib -L/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib -lpetsc -lkokkoskernels -lkokkoscontainers -lkokkoscore -lhipsparse -lhipblas -lrocsparse -lrocsolver -lrocblas -lamdhip64 -lhsa-runtime64 -lstdc++ -ldl -lpmi -lsci_cray_mpi -lsci_cray -lmpifort_cray -lmpi_cray -lmpi_gtl_hsa -lxpmem -ldsmml -lpgas-shmem -lquadmath -lcrayacc_amdgpu -lopenacc -lmodules -lfi -lcraymath -lf -lu -lcsup -lgfortran -lpthread -lgcc_eh -lm -lclang_rt.craypgo-x86_64 -lclang_rt.builtins-x86_64 -lquadmath -lstdc++ -ldl -o ex3k In file included from ex3k.kokkos.cxx:3: In file included from /gpfs/alpine/csc314/scratch/adams/petsc/include/petscdmda_kokkos.hpp:4: In file included from /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec_kokkos.hpp:14: In file included from /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec.h:9: In file included from /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsys.h:42: In file included from /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsystypes.h:255: In file included from /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Complex.hpp:47: In file included from /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Atomic.hpp:212: /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/impl/Kokkos_Atomic_Exchange.hpp:259:11: error: no member named 'lock_address_host_space' in namespace 'Kokkos::Impl::Kokkos::Impl'; did you mean simply 'lock_address_host_space'? while (!Impl::lock_address_host_space((void*)dest)) ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~ lock_address_host_space /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_HostSpace.hpp:83:6: note: 'lock_address_host_space' declared here bool lock_address_host_space(void* ptr); ^ In file included from ex3k.kokkos.cxx:3: In file included from /gpfs/alpine/csc314/scratch/adams/petsc/include/petscdmda_kokkos.hpp:4: In file included from /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec_kokkos.hpp:14: In file included from /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec.h:9: In file included from /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsys.h:42: In file included from /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsystypes.h:255: In file included from /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Complex.hpp:47: In file included from /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Atomic.hpp:212: /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/impl/Kokkos_Atomic_Exchange.hpp:261:3: error: no member named 'memory_fence' in namespace 'Kokkos::Impl::Kokkos'; did you mean simply 'memory_fence'? Kokkos::memory_fence(); ^~~~~~~~~~~~~~~~~~~~ memory_fence /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/impl/Kokkos_Memory_Fence.hpp:53:6: note: 'memory_fence' declared here void memory_fence() { ^ In file included from ex3k.kokkos.cxx:3: In file included from /gpfs/alpine/csc314/scratch/adams/petsc/include/petscdmda_kokkos.hpp:4: In file included from /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec_kokkos.hpp:14: In file included from /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec.h:9: In file included from /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsys.h:42: In file included from /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsystypes.h:255: In file included from /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Complex.hpp:47: In file included from /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Atomic.hpp:212: /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/impl/Kokkos_Atomic_Exchange.hpp:277:3: error: no member named 'memory_fence' in namespace 'Kokkos::Impl::Kokkos'; did you mean simply 'memory_fence'? Kokkos::memory_fence(); ^~~~~~~~~~~~~~~~~~~~ memory_fence /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/impl/Kokkos_Memory_Fence.hpp:53:6: note: 'memory_fence' declared here void memory_fence() { ^

1 0