Re: [petsc-dev] Kokkos make error on Spock
OK, it looks like this flag caused the problem and does not seem to be necessary: '--LDFLAGS=-L'+os.environ['ROCM_PATH']+'/lib -lhsa-runtime64', On Tue, Jul 20, 2021 at 9:00 AM Mark Adams <[email protected]> wrote:
On Mon, Jul 19, 2021 at 6:41 PM Scott Kruger <[email protected]> wrote:
Mark,
On tulip, things with fortran went wonky when a `-fast` flag snuck into the flags (snuck in being me copying a previous configure file and not questioning the flags). The reason is that for clang, `-fast` implies link time optimization (lto) for C/C++ code, but flang doesn't support lto so things got weird. I suspect that gfortran does not either (but perhaps the real question is why not use flang?)
It does work with GNU. I am doing this for an application and I want to avoid dictating the program env.
I'm (re)trying with -O0 just to check.
I look at configure and I don't see anything in your flags that would trigger lto, but as Junchao says, it might be picking something up from Kokkos, so perhaps this is the issue.
The configure files that I sent did not have Kokkos. They were stripped down.
I have asked ORNL, but don't expect help. The person (Matt) that is helping me for other things is very helpful so perhaps he can come up with some ideas. Otherwise I will tell the app that they need to use GNU for now.
Thanks, Mark
Scott
On 2021-07-19 07:19, Mark Adams did write:
Thanks, but this happens w/o Kokkos. I've stripped this down and attached good/bad logs without/with Fortran bindings. Hope this helps, Thanks again, Mark
On Sun, Jul 18, 2021 at 12:00 PM Stefano Zampini < [email protected]> wrote:
This is probably kokkos pulling in the dependencies where compiling kokkos source within PETSc.
Il Dom 18 Lug 2021, 16:29 Mark Adams <[email protected]> ha scritto:
Whoops, this error was just from not telling gfortran to allow long lines.
Anway, I did find that the when fortran bindings are enabled this OMP thing gets into the library. No idea how. I'll askm ORNL.
10:26 jczhang/fix-cray-mpicxx-includes/main= /gpfs/alpine/csc314/scratch/adams/petsc2$ nm
/gpfs/alpine/csc314/scratch/adams/petsc2/arch-spock-opt-cray-kokkos/lib/libpetsc.so
|g offload U .omp_offloading.img_cache.cray_amdgcn-amd-amdhsa U .omp_offloading.img_size.cray_amdgcn-amd-amdhsa U .omp_offloading.img_start.cray_amdgcn-amd-amdhsa 0000000001d457b0 T vecgetoffloadmask_
On Sun, Jul 18, 2021 at 8:43 AM Mark Adams <[email protected]> wrote:
Ah, your test was not on Spock ... I have something working but this is strange.
I switched to the GNU ProgEnv. and it passed the Fortran test in 'check', but this failed:
*08:26 2 jczhang/fix-cray-mpicxx-includes/main= /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials$ make
PETSC_DIR=/gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-dbg-gnu
PETSC_ARCH="" ex40f90ftn -L/sw/spock/spack-envs/views/rocm-4.1.0/lib -lamdhip64 -lhsa-runtime64 -fPIC -g -O0 -fPIC -g -O0
-I/gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-dbg-gnu/include
-I/sw/spock/spack-envs/views/rocm-4.1.0/include ex40f90.F90
-Wl,-rpath,/gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-dbg-gnu/lib
-L/gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-dbg-gnu/lib
-Wl,-rpath,/sw/spock/spack-envs/views/rocm-4.1.0/lib -L/sw/spock/spack-envs/views/rocm-4.1.0/lib -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/ofi/gnu/9.1/lib -L/opt/cray/pe/mpich/8.1.4/ofi/gnu/9.1/lib -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/gtl/lib -L/opt/cray/pe/mpich/8.1.4/gtl/lib -Wl,-rpath,/opt/cray/pe/libsci/21.04.1.1/GNU/9.1/x86_64/lib <http://21.04.1.1/GNU/9.1/x86_64/lib> -L/opt/cray/pe/libsci/21.04.1.1/GNU/9.1/x86_64/lib <http://21.04.1.1/GNU/9.1/x86_64/lib> -Wl,-rpath,/opt/cray/pe/pmi/6.0.10/lib -L/opt/cray/pe/pmi/6.0.10/lib -Wl,-rpath,/opt/cray/pe/dsmml/0.1.4/dsmml/lib -L/opt/cray/pe/dsmml/0.1.4/dsmml/lib -Wl,-rpath,/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64 -L/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64 -Wl,-rpath,/opt/gcc/10.2.0/snos/lib/gcc/x86_64-suse-linux/10.2.0 -L/opt/gcc/10.2.0/snos/lib/gcc/x86_64-suse-linux/10.2.0 -Wl,-rpath,/opt/gcc/10.2.0/snos/lib64 -L/opt/gcc/10.2.0/snos/lib64 -Wl,-rpath,/sw/spock/spack-envs/views/rocm-4.1.0/llvm/lib -L/sw/spock/spack-envs/views/rocm-4.1.0/llvm/lib -Wl,-rpath,/opt/gcc/10.2.0/snos/lib -L/opt/gcc/10.2.0/snos/lib -lpetsc -lhipsparse -lhipblas -lrocsparse -lrocsolver -lrocblas -lamdhip64 -lhsa-runtime64 -lstdc++ -ldl -lpmi -lmpi_gtl_hsa -lxpmem -lgfortran -lm -lsci_gnu_82_mpi -lsci_gnu_82 -lmpifort_gnu_91 -lmpi_gnu_91 -ldsmml -lgfortran -lquadmath -lpthread -lm -lgcc_s -lquadmath -lstdc++ -ldl -o ex40f90ex40f90.F90:35:132: 35 | call
DMDACreate2d(PETSC_COMM_WORLD,DM_BOUNDARY_NONE,DM_BOUNDARY_NONE,DMDA_STENCIL_BOX,ten,ten,PETSC_DECIDE,PETSC_DECIDE,two,one,
& |
1Error:
Line truncated at (1) [-Werror=line-truncation]ex40f90.F90:35:132:* * ....*
But ex5f (the Fortran check test) does work manually also (sanity check).
Oddly I seem to have the cray compilers:
08:34 jczhang/fix-cray-mpicxx-includes/main= /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials$ which ftn /opt/cray/pe/craype/2.7.6/bin/ftn
yet it is GNU:
08:34 jczhang/fix-cray-mpicxx-includes/main= /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials$ ftn --version GNU Fortran (GCC) 10.2.0 20200723 (Cray Inc.) Copyright (C) 2020 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
The documentation says ftn should be here:
08:34 jczhang/fix-cray-mpicxx-includes/main= /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials$ $GCC_PATH/bin/gfortran --version GNU Fortran (GCC) 10.2.0 20200723 (Cray Inc.) Copyright (C) 2020 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
Same, but the path is not the same:
08:39 jczhang/fix-cray-mpicxx-includes/main= /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials$ echo $GCC_PATH /opt/gcc/10.2.0
Anyway, thanks for your help, Mark
On Sat, Jul 17, 2021 at 11:26 PM Junchao Zhang < [email protected]> wrote:
Mark, I can reproduce this error with PrgEnv-cray, i.e., using the Cray compiler (clang-11). Previously I used PrgEnv-gnu, which did not have this error. Probably it is a problem of Spock. But I am not sure.
--Junchao Zhang
On Sat, Jul 17, 2021 at 10:17 AM Mark Adams <[email protected]> wrote:
And I can run a fortran test, with warnings, but C tests fail:
11:15 jczhang/fix-cray-mpicxx-includes/main= /gpfs/alpine/csc314/scratch/adams/petsc$ make PETSC_DIR=/gpfs/alpine/csc314/scratch/adams/petsc PETSC_ARCH=arch-spock-opt-cray-kokkos -f gmakefile test search='ts_utils_dmplexlandau_tutorials-ex1f90_0' Using MAKEFLAGS: -- search=ts_utils_dmplexlandau_tutorials-ex1f90_0 PETSC_ARCH=arch-spock-opt-cray-kokkos PETSC_DIR=/gpfs/alpine/csc314/scratch/adams/petsc FC
arch-spock-opt-cray-kokkos/tests/ts/utils/dmplexlandau/tutorials/ex1f90.o
FLINKER
arch-spock-opt-cray-kokkos/tests/ts/utils/dmplexlandau/tutorials/ex1f90
/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/bin/ld:
warning: alignment 128 of symbol `$host_init$$runtime_init_for_iso_c_binding$iso_c_binding_' in /opt/cray/pe/cce/11.0.4/cce/x86_64/lib/libmodules.so is smaller than 256 in
arch-spock-opt-cray-kokkos/tests/ts/utils/dmplexlandau/tutorials/ex1f90.o
/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/bin/ld:
warning: alignment 64 of symbol `$data_init$iso_c_binding_' in /opt/cray/pe/cce/11.0.4/cce/x86_64/lib/libmodules.so is smaller than 256 in
arch-spock-opt-cray-kokkos/tests/ts/utils/dmplexlandau/tutorials/ex1f90.o
TEST
arch-spock-opt-cray-kokkos/tests/counts/ts_utils_dmplexlandau_tutorials-ex1f90_0.counts
ok ts_utils_dmplexlandau_tutorials-ex1f90_0 ok diff-ts_utils_dmplexlandau_tutorials-ex1f90_0
On Sat, Jul 17, 2021 at 10:53 AM Mark Adams <[email protected]> wrote:
HUmm, I can not reproduce this.
10:49 jczhang/fix-cray-mpicxx-includes/main= /gpfs/alpine/csc314/scratch/adams/petsc$ make PETSC_DIR=/gpfs/alpine/csc314/scratch/adams/petsc PETSC_ARCH=arch-spock-dbg-kokkos check Running check examples to verify correct installation Using PETSC_DIR=/gpfs/alpine/csc314/scratch/adams/petsc and PETSC_ARCH=arch-spock-dbg-kokkos gmake[3]:
[/gpfs/alpine/csc314/scratch/adams/petsc/lib/petsc/conf/rules:301:
ex19.PETSc] Error 2 (ignored) *******************Error detected during compile or link!******************* See http://www.mcs.anl.gov/petsc/documentation/faq.html /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials ex19
cc -L/sw/spock/spack-envs/views/rocm-4.1.0/lib -lamdhip64 -lhsa-runtime64 -fPIC -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -fstack-protector -Qunused-arguments -fvisibility=hidden -g -O0 -fPIC -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -fstack-protector -Qunused-arguments -fvisibility=hidden -g -O0 -I/sw/spock/spack-envs/views/rocm-4.1.0/include -I/gpfs/alpine/csc314/scratch/adams/petsc/include
-I/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/include
-I/sw/spock/spack-envs/views/rocm-4.1.0/include -I/sw/spock/spack-envs/views/rocm-4.1.0/include ex19.c
-Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib
-L/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib
-Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib
-L/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib
-Wl,-rpath,/sw/spock/spack-envs/views/rocm-4.1.0/lib -L/sw/spock/spack-envs/views/rocm-4.1.0/lib -Wl,-rpath,/opt/gcc/8.1.0/snos/lib64 -L/opt/gcc/8.1.0/snos/lib64 -Wl,-rpath,/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib -L/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib -L/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/gtl/lib -L/opt/cray/pe/mpich/8.1.4/gtl/lib -Wl,-rpath,/opt/cray/pe/pmi/6.0.10/lib -L/opt/cray/pe/pmi/6.0.10/lib -Wl,-rpath,/opt/cray/pe/dsmml/0.1.4/dsmml/lib -L/opt/cray/pe/dsmml/0.1.4/dsmml/lib -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce/x86_64/lib -L/opt/cray/pe/cce/11.0.4/cce/x86_64/lib -Wl,-rpath,/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64 -L/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64
-Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux
-L/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux
-Wl,-rpath,/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0 -L/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0
-Wl,-rpath,/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib
-L/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib
-lpetsc -lkokkoskernels -lkokkoscontainers -lkokkoscore -lhipsparse -lhipblas -lrocsparse -lrocsolver -lrocblas -lamdhip64 -lhsa-runtime64 -lstdc++ -ldl -lpmi -lsci_cray_mpi -lsci_cray -lmpifort_cray -lmpi_cray -lmpi_gtl_hsa -lxpmem -ldsmml -lpgas-shmem -lquadmath -lcrayacc_amdgpu -lopenacc -lmodules -lfi -lcraymath -lf -lu -lcsup -lgfortran -lpthread -lgcc_eh -lm -lclang_rt.craypgo-x86_64 -lclang_rt.builtins-x86_64 -lquadmath -lstdc++ -ldl -o ex19 ld.lld: error:
/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib/libpetsc.so:
undefined reference to .omp_offloading.img_start.cray_amdgcn-amd-amdhsa [--no-allow-shlib-undefined] ld.lld: error:
/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib/libpetsc.so:
undefined reference to .omp_offloading.img_size.cray_amdgcn-amd-amdhsa [--no-allow-shlib-undefined] ld.lld: error:
/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib/libpetsc.so:
undefined reference to .omp_offloading.img_cache.cray_amdgcn-amd-amdhsa [--no-allow-shlib-undefined] clang-11: error: linker command failed with exit code 1 (use -v to see invocation) gmake[4]: *** [<builtin>: ex19] Error 1 gmake[3]:
[/gpfs/alpine/csc314/scratch/adams/petsc/lib/petsc/conf/rules:301:
ex3k.PETSc] Error 2 (ignored) *******************Error detected during compile or link!******************* See http://www.mcs.anl.gov/petsc/documentation/faq.html /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials ex3k
hipcc -fPIC -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -fvisibility=hidden -g -O0 -std=c++14 -I/gpfs/alpine/csc314/scratch/adams/petsc/include
-I/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/include
-I/sw/spock/spack-envs/views/rocm-4.1.0/include -I/sw/spock/spack-envs/views/rocm-4.1.0/include -I/opt/cray/pe/libsci/ 21.04.1.1/CRAY/9.0/x86_64/include -I/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/include -I/opt/cray/pe/pmi/6.0.10/include -I/opt/cray/pe/dsmml/0.1.4/dsmml//include -I/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/include -L/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib -L/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib -L/opt/cray/pe/mpich/8.1.4/gtl/lib -L/opt/cray/pe/pmi/6.0.10/lib -L/opt/cray/pe/dsmml/0.1.4/dsmml//lib -L/opt/cray/pe/cce/11.0.4/cce/x86_64/lib/pkgconfig/../ -L/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64 -Wl,--as-needed,-lsci_cray_mpi,--no-as-needed -Wl,--as-needed,-lsci_cray,--no-as-needed -Wl,--as-needed,-lmpi_cray,--no-as-needed -Wl,--as-needed,-ldsmml,--no-as-needed -Wl,--as-needed,-lpgas-shmem,--no-as-needed -lpmi -ldl -lmpi_gtl_hsa -lxpmem -lquadmath -lmodules -lfi -lcraymath -lf -lu -lcsup ex3k.kokkos.cxx
-Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib
-L/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib
-Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib
-L/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib
-Wl,-rpath,/sw/spock/spack-envs/views/rocm-4.1.0/lib -L/sw/spock/spack-envs/views/rocm-4.1.0/lib -Wl,-rpath,/opt/gcc/8.1.0/snos/lib64 -L/opt/gcc/8.1.0/snos/lib64 -Wl,-rpath,/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib -L/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib -L/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/gtl/lib -L/opt/cray/pe/mpich/8.1.4/gtl/lib -Wl,-rpath,/opt/cray/pe/pmi/6.0.10/lib -L/opt/cray/pe/pmi/6.0.10/lib -Wl,-rpath,/opt/cray/pe/dsmml/0.1.4/dsmml/lib -L/opt/cray/pe/dsmml/0.1.4/dsmml/lib -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce/x86_64/lib -L/opt/cray/pe/cce/11.0.4/cce/x86_64/lib -Wl,-rpath,/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64 -L/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64
-Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux
-L/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux
-Wl,-rpath,/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0 -L/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0
-Wl,-rpath,/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib
-L/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib
-lpetsc -lkokkoskernels -lkokkoscontainers -lkokkoscore -lhipsparse -lhipblas -lrocsparse -lrocsolver -lrocblas -lamdhip64 -lhsa-runtime64 -lstdc++ -ldl -lpmi -lsci_cray_mpi -lsci_cray -lmpifort_cray -lmpi_cray -lmpi_gtl_hsa -lxpmem -ldsmml -lpgas-shmem -lquadmath -lcrayacc_amdgpu -lopenacc -lmodules -lfi -lcraymath -lf -lu -lcsup -lgfortran -lpthread -lgcc_eh -lm -lclang_rt.craypgo-x86_64 -lclang_rt.builtins-x86_64 -lquadmath -lstdc++ -ldl -o ex3k
/usr/lib64/gcc/x86_64-suse-linux/7/../../../../x86_64-suse-linux/bin/ld:
/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib/libpetsc.so:
undefined reference to `.omp_offloading.img_size.cray_amdgcn-amd-amdhsa'
/usr/lib64/gcc/x86_64-suse-linux/7/../../../../x86_64-suse-linux/bin/ld:
/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib/libpetsc.so:
undefined reference to `.omp_offloading.img_start.cray_amdgcn-amd-amdhsa'
/usr/lib64/gcc/x86_64-suse-linux/7/../../../../x86_64-suse-linux/bin/ld:
/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib/libpetsc.so:
undefined reference to `.omp_offloading.img_cache.cray_amdgcn-amd-amdhsa' clang-12: error: linker command failed with exit code 1 (use -v to see invocation) gmake[4]: *** [/gpfs/alpine/csc314/scratch/adams/petsc/lib/petsc/conf/test:26: ex3k] Error 1 gmake[3]: [makefile:102: runex3k_kokkos] Error 2 (ignored) 1,25c1,4 < atol=1e-50, rtol=1e-08, stol=1e-08, maxit=50, maxf=10000 < Vec Object: Exact Solution 2 MPI processes < type: mpikokkos < Process [0] < 0. < 0.015625 < 0.125 < Process [1] < 0.421875 < 1. < Vec Object: Forcing function 2 MPI processes < type: mpikokkos < Process [0] < 1e-72 < 1.50024 < 3.01563 < Process [1] < 4.67798 < 7. < 0 SNES Function norm 5.414682427127e+00 < 1 SNES Function norm 2.952582418265e-01 < 2 SNES Function norm 4.502293658739e-04 < 3 SNES Function norm 1.389665806646e-09 < Number of SNES iterations = 3 < Norm of error 1.49752e-10 Iterations 3 ---
slurmstepd: error: execve():
/gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials/./ex3k: No such
file or directory
slurmstepd: error: execve():
/gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials/./ex3k: No such
file or directory
srun: error: spock01: tasks 0-1: Exited with exit code 2 srun: launch/slurm: _step_signal: Terminating StepId=272979.0 /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials Possible problem with ex3k running with kokkos-kernels, diffs above ========================================= *******************Error detected during compile or link!******************* See http://www.mcs.anl.gov/petsc/documentation/faq.html /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials ex5f
ftn -L/sw/spock/spack-envs/views/rocm-4.1.0/lib -lamdhip64 -lhsa-runtime64 -fPIC -g -O0 -fPIC -g -O0 -I/gpfs/alpine/csc314/scratch/adams/petsc/include
-I/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/include
-I/sw/spock/spack-envs/views/rocm-4.1.0/include ex5f.F90
-Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib
-L/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib
-Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib
-L/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib
-Wl,-rpath,/sw/spock/spack-envs/views/rocm-4.1.0/lib -L/sw/spock/spack-envs/views/rocm-4.1.0/lib -Wl,-rpath,/opt/gcc/8.1.0/snos/lib64 -L/opt/gcc/8.1.0/snos/lib64 -Wl,-rpath,/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib -L/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib -L/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/gtl/lib -L/opt/cray/pe/mpich/8.1.4/gtl/lib -Wl,-rpath,/opt/cray/pe/pmi/6.0.10/lib -L/opt/cray/pe/pmi/6.0.10/lib -Wl,-rpath,/opt/cray/pe/dsmml/0.1.4/dsmml/lib -L/opt/cray/pe/dsmml/0.1.4/dsmml/lib -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce/x86_64/lib -L/opt/cray/pe/cce/11.0.4/cce/x86_64/lib -Wl,-rpath,/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64 -L/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64
-Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux
-L/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux
-Wl,-rpath,/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0 -L/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0
-Wl,-rpath,/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib
-L/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib
-lpetsc -lkokkoskernels -lkokkoscontainers -lkokkoscore -lhipsparse -lhipblas -lrocsparse -lrocsolver -lrocblas -lamdhip64 -lhsa-runtime64 -lstdc++ -ldl -lpmi -lsci_cray_mpi -lsci_cray -lmpifort_cray -lmpi_cray -lmpi_gtl_hsa -lxpmem -ldsmml -lpgas-shmem -lquadmath -lcrayacc_amdgpu -lopenacc -lmodules -lfi -lcraymath -lf -lu -lcsup -lgfortran -lpthread -lgcc_eh -lm -lclang_rt.craypgo-x86_64 -lclang_rt.builtins-x86_64 -lquadmath -lstdc++ -ldl -o ex5f
/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/bin/ld:
warning: alignment 128 of symbol `$host_init$$runtime_init_for_iso_c_binding$iso_c_binding_' in /opt/cray/pe/cce/11.0.4/cce/x86_64/lib/libmodules.so is smaller than 256 in /tmp/pe_30429/ex5f_1.o
/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/bin/ld:
warning: alignment 64 of symbol `$data_init$iso_c_binding_' in /opt/cray/pe/cce/11.0.4/cce/x86_64/lib/libmodules.so is smaller than 256 in /tmp/pe_30429/ex5f_1.o Fortran example src/snes/tutorials/ex5f run successfully with 1 MPI process Completed test examples 10:49 jczhang/fix-cray-mpicxx-includes/main= /gpfs/alpine/csc314/scratch/adams/petsc$ ll
On Fri, Jul 16, 2021 at 11:28 PM Junchao Zhang < [email protected]> wrote:
Mark, I configured with Fortran binding enabled with main + MR !4175 <https://gitlab.com/petsc/petsc/-/merge_requests/4175>
'--with-mpiexec=srun', '--with-shared-libraries=1', '--with-cc=cc', '--with-cxx=CC', '--with-fc=ftn', '--with-fortran-bindings', '--with-hip', '--with-hipc=hipcc', '--with-debugging', '--CPPFLAGS=-I${ROCM_PATH}/include', '--CXXPPFLAGS=-I${ROCM_PATH}/include', '--CC_LINKER_FLAGS=-L${ROCM_PATH}/lib -lamdhip64 -lhsa-runtime64', '--CXX_LINKER_FLAGS=-L${ROCM_PATH}/lib -lamdhip64 -lhsa-runtime64', '--FC_LINKER_FLAGS=-L${ROCM_PATH}/lib -lamdhip64 -lhsa-runtime64', '--COPTFLAGS=-g -O0', '--CXXOPTFLAGS=-g -O0', '--FOPTFLAGS=-g -O0', '--download-kokkos', '--download-kokkos-kernels', '--download-kokkos-commit=3.4.01', '--download-kokkos-kernels-commit=3.4.01', '--with-kokkos-hip-arch=VEGA908',
and 'make check' ran smoothly on a compute node
$ make check Running check examples to verify correct installation Using PETSC_DIR=/ccs/home/jczhang/petsc and PETSC_ARCH=arch-spock-cray-kokkos-dbg C/C++ example src/snes/tutorials/ex19 run successfully with 1 MPI process C/C++ example src/snes/tutorials/ex19 run successfully with 2 MPI processes C/C++ example src/snes/tutorials/ex3k run successfully with kokkos-kernels Fortran example src/snes/tutorials/ex5f run successfully with 1 MPI process
Completed test examples
--Junchao Zhang
On Fri, Jul 16, 2021 at 6:04 PM Mark Adams <[email protected]> wrote:
And I find that this error, on non-Kokkos C tests, is fixed by turning the fortran bindings off:
ld.lld: error:
/gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/lib/libpetsc.so:
undefined reference to .omp_offloading.img_start.cray_amdgcn-amd-amdhsa [--no-allow-shlib-undefined] ld.lld: error:
/gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/lib/libpetsc.so:
undefined reference to .omp_offloading.img_size.cray_amdgcn-amd-amdhsa [--no-allow-shlib-undefined] ld.lld: error:
/gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/lib/libpetsc.so:
undefined reference to .omp_offloading.img_cache.cray_amdgcn-amd-amdhsa [--no-allow-shlib-undefined]
On Fri, Jul 16, 2021 at 3:53 PM Mark Adams <[email protected]> wrote:
Not complex. THis has some overlap with my problem w/o Kokkos.
On Fri, Jul 16, 2021 at 12:54 PM Junchao Zhang < [email protected]> wrote:
Do you use complex? post your configure.log.
--Junchao Zhang
On Fri, Jul 16, 2021 at 9:47 AM Mark Adams <[email protected]> wrote:
The simple Kokkos example is failing for me on Spock. Any ideas? Thanks,
10:44 main *= /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials$ make PETSC_DIR=/gpfs/alpine/csc314/scratch/adams/petsc PETSC_ARCH=arch-spock-opt-cray-kokkos ex3k MPICH_CXX="hipcc" OMPI_CXX="hipcc" CC -L/sw/spock/spack-envs/views/rocm-4.1.0/lib -lhsa-runtime64 -L/sw/spock/spack-envs/views/rocm-4.1.0/lib -lamdhip64 -lhsa-runtime64 -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -fstack-protector -fvisibility=hidden -g -O2 -fPIC -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -fvisibility=hidden -g -O3 -std=c++14 -I/gpfs/alpine/csc314/scratch/adams/petsc/include
-I/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-opt-cray-kokkos/include
-I/gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include
-I/sw/spock/spack-envs/views/rocm-4.1.0/include -I/sw/spock/spack-envs/views/rocm-4.1.0/include ex3k.kokkos.cxx
-Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-opt-cray-kokkos/lib
-L/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-opt-cray-kokkos/lib
-Wl,-rpath,/gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/lib
-L/gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/lib
-Wl,-rpath,/sw/spock/spack-envs/views/rocm-4.1.0/lib -L/sw/spock/spack-envs/views/rocm-4.1.0/lib -Wl,-rpath,/opt/gcc/8.1.0/snos/lib64 -L/opt/gcc/8.1.0/snos/lib64 -Wl,-rpath,/opt/cray/pe/libsci/ 21.04.1.1/CRAY/9.0/x86_64/lib -L/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib -L/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/gtl/lib -L/opt/cray/pe/mpich/8.1.4/gtl/lib -Wl,-rpath,/opt/cray/pe/pmi/6.0.10/lib -L/opt/cray/pe/pmi/6.0.10/lib -Wl,-rpath,/opt/cray/pe/dsmml/0.1.4/dsmml/lib -L/opt/cray/pe/dsmml/0.1.4/dsmml/lib -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce/x86_64/lib -L/opt/cray/pe/cce/11.0.4/cce/x86_64/lib
-Wl,-rpath,/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64
-L/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64
-Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux
-L/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux
-Wl,-rpath,/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0
-L/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0
-Wl,-rpath,/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib
-L/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib
-lpetsc -lkokkoskernels -lkokkoscontainers -lkokkoscore -lhipsparse -lhipblas -lrocsparse -lrocsolver -lrocblas -lamdhip64 -lhsa-runtime64 -lstdc++ -ldl -lpmi -lsci_cray_mpi -lsci_cray -lmpifort_cray -lmpi_cray -lmpi_gtl_hsa -lxpmem -ldsmml -lpgas-shmem -lquadmath -lcrayacc_amdgpu -lopenacc -lmodules -lfi -lcraymath -lf -lu -lcsup -lgfortran -lpthread -lgcc_eh -lm -lclang_rt.craypgo-x86_64 -lclang_rt.builtins-x86_64 -lquadmath -lstdc++ -ldl -o ex3k In file included from ex3k.kokkos.cxx:3: In file included from
/gpfs/alpine/csc314/scratch/adams/petsc/include/petscdmda_kokkos.hpp:4:
In file included from
/gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec_kokkos.hpp:14:
In file included from
/gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec.h:9:
In file included from
/gpfs/alpine/csc314/scratch/adams/petsc/include/petscsys.h:42:
In file included from
/gpfs/alpine/csc314/scratch/adams/petsc/include/petscsystypes.h:255:
In file included from
/gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Complex.hpp:47:
In file included from
/gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Atomic.hpp:212:
/gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/impl/Kokkos_Atomic_Exchange.hpp:259:11:
error: no member named 'lock_address_host_space' in namespace 'Kokkos::Impl::Kokkos::Impl'; did you mean simply 'lock_address_host_space'? while (!Impl::lock_address_host_space((void*)dest)) ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~ lock_address_host_space
/gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_HostSpace.hpp:83:6:
note: 'lock_address_host_space' declared here bool lock_address_host_space(void* ptr); ^ In file included from ex3k.kokkos.cxx:3: In file included from
/gpfs/alpine/csc314/scratch/adams/petsc/include/petscdmda_kokkos.hpp:4:
In file included from
/gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec_kokkos.hpp:14:
In file included from
/gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec.h:9:
In file included from
/gpfs/alpine/csc314/scratch/adams/petsc/include/petscsys.h:42:
In file included from
/gpfs/alpine/csc314/scratch/adams/petsc/include/petscsystypes.h:255:
In file included from
/gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Complex.hpp:47:
In file included from
/gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Atomic.hpp:212:
/gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/impl/Kokkos_Atomic_Exchange.hpp:261:3:
error: no member named 'memory_fence' in namespace 'Kokkos::Impl::Kokkos'; did you mean simply 'memory_fence'? Kokkos::memory_fence(); ^~~~~~~~~~~~~~~~~~~~ memory_fence
/gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/impl/Kokkos_Memory_Fence.hpp:53:6:
note: 'memory_fence' declared here void memory_fence() { ^ In file included from ex3k.kokkos.cxx:3: In file included from
/gpfs/alpine/csc314/scratch/adams/petsc/include/petscdmda_kokkos.hpp:4:
In file included from
/gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec_kokkos.hpp:14:
In file included from
/gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec.h:9:
In file included from
/gpfs/alpine/csc314/scratch/adams/petsc/include/petscsys.h:42:
In file included from
/gpfs/alpine/csc314/scratch/adams/petsc/include/petscsystypes.h:255:
In file included from
/gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Complex.hpp:47:
In file included from
/gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Atomic.hpp:212:
/gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/impl/Kokkos_Atomic_Exchange.hpp:277:3:
error: no member named 'memory_fence' in namespace 'Kokkos::Impl::Kokkos'; did you mean simply 'memory_fence'? Kokkos::memory_fence(); ^~~~~~~~~~~~~~~~~~~~ memory_fence
/gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/impl/Kokkos_Memory_Fence.hpp:53:6:
note: 'memory_fence' declared here void memory_fence() { ^
-- Scott Kruger Tech-X Corporation [email protected] 5621 Arapahoe Ave, Suite A Phone: (720) 466-3196 Boulder, CO 80303 Fax: (303) 448-7756
participants (1)
-
Mark Adams