Ha, Landau is working with the Kokkos backend and it is running the the GPU! My Kokkos build was clean, but let me try it again. On Fri, Jul 16, 2021 at 5:19 PM Junchao Zhang <[email protected]> wrote:

I don't understand this when linking ex19 ld.lld: error: /gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-opt-cray-kokkos/lib/libpetsc.so: undefined reference to .omp_offloading.img_start.cray_amdgcn-amd-amdhsa [--no-allow-shlib-undefined] Do you have a stale libpetsc.so?

For ex3k, it is a makefile problem (I am fixing). If you do not directly build an executable from *.kokkos.cxx, then you can avoid this problem. For example, snes/tests/ex13 works with kokkos options on Spock.

--Junchao Zhang

On Fri, Jul 16, 2021 at 2:53 PM Mark Adams <[email protected]> wrote:

...

Not complex. THis has some overlap with my problem w/o Kokkos.

On Fri, Jul 16, 2021 at 12:54 PM Junchao Zhang <[email protected]> wrote:

...

Do you use complex? post your configure.log.

--Junchao Zhang

On Fri, Jul 16, 2021 at 9:47 AM Mark Adams <[email protected]> wrote:

...

The simple Kokkos example is failing for me on Spock. Any ideas? Thanks,

10:44 main *= /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials$ make PETSC_DIR=/gpfs/alpine/csc314/scratch/adams/petsc PETSC_ARCH=arch-spock-opt-cray-kokkos ex3k MPICH_CXX="hipcc" OMPI_CXX="hipcc" CC -L/sw/spock/spack-envs/views/rocm-4.1.0/lib -lhsa-runtime64 -L/sw/spock/spack-envs/views/rocm-4.1.0/lib -lamdhip64 -lhsa-runtime64 -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -fstack-protector -fvisibility=hidden -g -O2 -fPIC -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -fvisibility=hidden -g -O3 -std=c++14 -I/gpfs/alpine/csc314/scratch/adams/petsc/include -I/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-opt-cray-kokkos/include -I/gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include -I/sw/spock/spack-envs/views/rocm-4.1.0/include -I/sw/spock/spack-envs/views/rocm-4.1.0/include ex3k.kokkos.cxx -Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-opt-cray-kokkos/lib -L/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-opt-cray-kokkos/lib -Wl,-rpath,/gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/lib -L/gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/lib -Wl,-rpath,/sw/spock/spack-envs/views/rocm-4.1.0/lib -L/sw/spock/spack-envs/views/rocm-4.1.0/lib -Wl,-rpath,/opt/gcc/8.1.0/snos/lib64 -L/opt/gcc/8.1.0/snos/lib64 -Wl,-rpath,/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib -L/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib -L/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/gtl/lib -L/opt/cray/pe/mpich/8.1.4/gtl/lib -Wl,-rpath,/opt/cray/pe/pmi/6.0.10/lib -L/opt/cray/pe/pmi/6.0.10/lib -Wl,-rpath,/opt/cray/pe/dsmml/0.1.4/dsmml/lib -L/opt/cray/pe/dsmml/0.1.4/dsmml/lib -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce/x86_64/lib -L/opt/cray/pe/cce/11.0.4/cce/x86_64/lib -Wl,-rpath,/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64 -L/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64 -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux -L/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux -Wl,-rpath,/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0 -L/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0 -Wl,-rpath,/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib -L/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib -lpetsc -lkokkoskernels -lkokkoscontainers -lkokkoscore -lhipsparse -lhipblas -lrocsparse -lrocsolver -lrocblas -lamdhip64 -lhsa-runtime64 -lstdc++ -ldl -lpmi -lsci_cray_mpi -lsci_cray -lmpifort_cray -lmpi_cray -lmpi_gtl_hsa -lxpmem -ldsmml -lpgas-shmem -lquadmath -lcrayacc_amdgpu -lopenacc -lmodules -lfi -lcraymath -lf -lu -lcsup -lgfortran -lpthread -lgcc_eh -lm -lclang_rt.craypgo-x86_64 -lclang_rt.builtins-x86_64 -lquadmath -lstdc++ -ldl -o ex3k In file included from ex3k.kokkos.cxx:3: In file included from /gpfs/alpine/csc314/scratch/adams/petsc/include/petscdmda_kokkos.hpp:4: In file included from /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec_kokkos.hpp:14: In file included from /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec.h:9: In file included from /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsys.h:42: In file included from /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsystypes.h:255: In file included from /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Complex.hpp:47: In file included from /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Atomic.hpp:212: /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/impl/Kokkos_Atomic_Exchange.hpp:259:11: error: no member named 'lock_address_host_space' in namespace 'Kokkos::Impl::Kokkos::Impl'; did you mean simply 'lock_address_host_space'? while (!Impl::lock_address_host_space((void*)dest)) ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~ lock_address_host_space /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_HostSpace.hpp:83:6: note: 'lock_address_host_space' declared here bool lock_address_host_space(void* ptr); ^ In file included from ex3k.kokkos.cxx:3: In file included from /gpfs/alpine/csc314/scratch/adams/petsc/include/petscdmda_kokkos.hpp:4: In file included from /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec_kokkos.hpp:14: In file included from /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec.h:9: In file included from /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsys.h:42: In file included from /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsystypes.h:255: In file included from /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Complex.hpp:47: In file included from /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Atomic.hpp:212: /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/impl/Kokkos_Atomic_Exchange.hpp:261:3: error: no member named 'memory_fence' in namespace 'Kokkos::Impl::Kokkos'; did you mean simply 'memory_fence'? Kokkos::memory_fence(); ^~~~~~~~~~~~~~~~~~~~ memory_fence /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/impl/Kokkos_Memory_Fence.hpp:53:6: note: 'memory_fence' declared here void memory_fence() { ^ In file included from ex3k.kokkos.cxx:3: In file included from /gpfs/alpine/csc314/scratch/adams/petsc/include/petscdmda_kokkos.hpp:4: In file included from /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec_kokkos.hpp:14: In file included from /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec.h:9: In file included from /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsys.h:42: In file included from /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsystypes.h:255: In file included from /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Complex.hpp:47: In file included from /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Atomic.hpp:212: /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/impl/Kokkos_Atomic_Exchange.hpp:277:3: error: no member named 'memory_fence' in namespace 'Kokkos::Impl::Kokkos'; did you mean simply 'memory_fence'? Kokkos::memory_fence(); ^~~~~~~~~~~~~~~~~~~~ memory_fence /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/impl/Kokkos_Memory_Fence.hpp:53:6: note: 'memory_fence' declared here void memory_fence() { ^