petsc-dev

Download

[email protected]

July 2021

9 participants
31 discussions

Re: [petsc-dev] Kokkos make error on Spock
by Scott Kruger 20 Jul '21

20 Jul '21

Cray puts the runtime somewhere and then uses their library to put it in. I should have mentioned this earlier, but we have config/examples/arch-olcf-frontier-hip-beta.py for trying to keep up with Cray's default hip, but I think it's different than what Junchao does for Kokkos which is why I didn't mention it earlier. Scott On 2021-07-20 11:26, Mark Adams did write: > OK, it looks like this flag caused the problem and does not seem to be > necessary: > > '--LDFLAGS=-L'+os.environ['ROCM_PATH']+'/lib -lhsa-runtime64', > > On Tue, Jul 20, 2021 at 9:00 AM Mark Adams <mfadams(a)lbl.gov> wrote: > > > > > > > On Mon, Jul 19, 2021 at 6:41 PM Scott Kruger <kruger(a)txcorp.com> wrote: > > > >> > >> Mark, > >> > >> On tulip, things with fortran went wonky when a `-fast` flag snuck into > >> the flags (snuck in being me copying a previous configure file and not > >> questioning the flags). The reason is that for clang, `-fast` implies > >> link time optimization (lto) for C/C++ code, but flang doesn't support > >> lto so things got weird. I suspect that gfortran does not either (but > >> perhaps the real question is why not use flang?) > >> > > > > It does work with GNU. I am doing this for an application and I want to > > avoid dictating the program env. > > > > I'm (re)trying with -O0 just to check. > > > > > >> > >> I look at configure and I don't see anything in your flags that would > >> trigger lto, but as Junchao says, it might be picking something up from > >> Kokkos, so perhaps this is the issue. > >> > >> > > The configure files that I sent did not have Kokkos. They were stripped > > down. > > > > I have asked ORNL, but don't expect help. > > The person (Matt) that is helping me for other things is very helpful so > > perhaps he can come up with some ideas. > > Otherwise I will tell the app that they need to use GNU for now. > > > > Thanks, > > Mark > > > > > > > >> Scott > >> > >> > >> On 2021-07-19 07:19, Mark Adams did write: > >> > Thanks, but this happens w/o Kokkos. > >> > I've stripped this down and attached good/bad logs without/with Fortran > >> > bindings. > >> > Hope this helps, > >> > Thanks again, > >> > Mark > >> > > >> > On Sun, Jul 18, 2021 at 12:00 PM Stefano Zampini < > >> stefano.zampini(a)gmail.com> > >> > wrote: > >> > > >> > > This is probably kokkos pulling in the dependencies where compiling > >> kokkos > >> > > source within PETSc. > >> > > > >> > > Il Dom 18 Lug 2021, 16:29 Mark Adams <mfadams(a)lbl.gov> ha scritto: > >> > > > >> > >> Whoops, this error was just from not telling gfortran to allow long > >> lines. > >> > >> > >> > >> Anway, I did find that the when fortran bindings are enabled this OMP > >> > >> thing gets into the library. No idea how. I'll askm ORNL. > >> > >> > >> > >> 10:26 jczhang/fix-cray-mpicxx-includes/main= > >> > >> /gpfs/alpine/csc314/scratch/adams/petsc2$ nm > >> > >> > >> /gpfs/alpine/csc314/scratch/adams/petsc2/arch-spock-opt-cray-kokkos/lib/libpetsc.so > >> > >> |g offload > >> > >> U .omp_offloading.img_cache.cray_amdgcn-amd-amdhsa > >> > >> U .omp_offloading.img_size.cray_amdgcn-amd-amdhsa > >> > >> U .omp_offloading.img_start.cray_amdgcn-amd-amdhsa > >> > >> 0000000001d457b0 T vecgetoffloadmask_ > >> > >> > >> > >> > >> > >> On Sun, Jul 18, 2021 at 8:43 AM Mark Adams <mfadams(a)lbl.gov> wrote: > >> > >> > >> > >>> Ah, your test was not on Spock ... I have something working but > >> this is > >> > >>> strange. > >> > >>> > >> > >>> I switched to the GNU ProgEnv. and it passed the Fortran test in > >> > >>> 'check', but this failed: > >> > >>> > >> > >>> > >> > >>> > >> > >>> > >> > >>> > >> > >>> > >> > >>> > >> > >>> > >> > >>> > >> > >>> *08:26 2 jczhang/fix-cray-mpicxx-includes/main= > >> > >>> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials$ make > >> > >>> > >> PETSC_DIR=/gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-dbg-gnu > >> > >>> PETSC_ARCH="" ex40f90ftn -L/sw/spock/spack-envs/views/rocm-4.1.0/lib > >> > >>> -lamdhip64 -lhsa-runtime64 -fPIC -g -O0 -fPIC -g -O0 > >> > >>> > >> -I/gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-dbg-gnu/include > >> > >>> -I/sw/spock/spack-envs/views/rocm-4.1.0/include ex40f90.F90 > >> > >>> > >> -Wl,-rpath,/gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-dbg-gnu/lib > >> > >>> > >> -L/gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-dbg-gnu/lib > >> > >>> -Wl,-rpath,/sw/spock/spack-envs/views/rocm-4.1.0/lib > >> > >>> -L/sw/spock/spack-envs/views/rocm-4.1.0/lib > >> > >>> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/ofi/gnu/9.1/lib > >> > >>> -L/opt/cray/pe/mpich/8.1.4/ofi/gnu/9.1/lib > >> > >>> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/gtl/lib > >> > >>> -L/opt/cray/pe/mpich/8.1.4/gtl/lib > >> > >>> -Wl,-rpath,/opt/cray/pe/libsci/21.04.1.1/GNU/9.1/x86_64/lib > >> > >>> <http://21.04.1.1/GNU/9.1/x86_64/lib> > >> > >>> -L/opt/cray/pe/libsci/21.04.1.1/GNU/9.1/x86_64/lib > >> > >>> <http://21.04.1.1/GNU/9.1/x86_64/lib> > >> > >>> -Wl,-rpath,/opt/cray/pe/pmi/6.0.10/lib -L/opt/cray/pe/pmi/6.0.10/lib > >> > >>> -Wl,-rpath,/opt/cray/pe/dsmml/0.1.4/dsmml/lib > >> > >>> -L/opt/cray/pe/dsmml/0.1.4/dsmml/lib > >> > >>> -Wl,-rpath,/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64 > >> > >>> -L/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64 > >> > >>> -Wl,-rpath,/opt/gcc/10.2.0/snos/lib/gcc/x86_64-suse-linux/10.2.0 > >> > >>> -L/opt/gcc/10.2.0/snos/lib/gcc/x86_64-suse-linux/10.2.0 > >> > >>> -Wl,-rpath,/opt/gcc/10.2.0/snos/lib64 -L/opt/gcc/10.2.0/snos/lib64 > >> > >>> -Wl,-rpath,/sw/spock/spack-envs/views/rocm-4.1.0/llvm/lib > >> > >>> -L/sw/spock/spack-envs/views/rocm-4.1.0/llvm/lib > >> > >>> -Wl,-rpath,/opt/gcc/10.2.0/snos/lib -L/opt/gcc/10.2.0/snos/lib > >> -lpetsc > >> > >>> -lhipsparse -lhipblas -lrocsparse -lrocsolver -lrocblas -lamdhip64 > >> > >>> -lhsa-runtime64 -lstdc++ -ldl -lpmi -lmpi_gtl_hsa -lxpmem > >> -lgfortran -lm > >> > >>> -lsci_gnu_82_mpi -lsci_gnu_82 -lmpifort_gnu_91 -lmpi_gnu_91 -ldsmml > >> > >>> -lgfortran -lquadmath -lpthread -lm -lgcc_s -lquadmath -lstdc++ > >> -ldl -o > >> > >>> ex40f90ex40f90.F90:35:132: 35 | call > >> > >>> > >> DMDACreate2d(PETSC_COMM_WORLD,DM_BOUNDARY_NONE,DM_BOUNDARY_NONE,DMDA_STENCIL_BOX,ten,ten,PETSC_DECIDE,PETSC_DECIDE,two,one, > >> > >>> & | > >> > >>> > >> 1Error: > >> > >>> Line truncated at (1) [-Werror=line-truncation]ex40f90.F90:35:132:* > >> > >>> * ....* > >> > >>> > >> > >>> But ex5f (the Fortran check test) does work manually also (sanity > >> check). > >> > >>> > >> > >>> Oddly I seem to have the cray compilers: > >> > >>> > >> > >>> 08:34 jczhang/fix-cray-mpicxx-includes/main= > >> > >>> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials$ which > >> ftn > >> > >>> /opt/cray/pe/craype/2.7.6/bin/ftn > >> > >>> > >> > >>> yet it is GNU: > >> > >>> > >> > >>> 08:34 jczhang/fix-cray-mpicxx-includes/main= > >> > >>> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials$ ftn > >> --version > >> > >>> GNU Fortran (GCC) 10.2.0 20200723 (Cray Inc.) > >> > >>> Copyright (C) 2020 Free Software Foundation, Inc. > >> > >>> This is free software; see the source for copying conditions. > >> There is > >> > >>> NO > >> > >>> warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR > >> > >>> PURPOSE. > >> > >>> > >> > >>> The documentation says ftn should be here: > >> > >>> > >> > >>> 08:34 jczhang/fix-cray-mpicxx-includes/main= > >> > >>> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials$ > >> > >>> $GCC_PATH/bin/gfortran --version > >> > >>> GNU Fortran (GCC) 10.2.0 20200723 (Cray Inc.) > >> > >>> Copyright (C) 2020 Free Software Foundation, Inc. > >> > >>> This is free software; see the source for copying conditions. > >> There is > >> > >>> NO > >> > >>> warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR > >> > >>> PURPOSE. > >> > >>> > >> > >>> Same, but the path is not the same: > >> > >>> > >> > >>> 08:39 jczhang/fix-cray-mpicxx-includes/main= > >> > >>> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials$ echo > >> $GCC_PATH > >> > >>> /opt/gcc/10.2.0 > >> > >>> > >> > >>> Anyway, thanks for your help, > >> > >>> Mark > >> > >>> > >> > >>> On Sat, Jul 17, 2021 at 11:26 PM Junchao Zhang < > >> junchao.zhang(a)gmail.com> > >> > >>> wrote: > >> > >>> > >> > >>>> Mark, I can reproduce this error with PrgEnv-cray, i.e., using the > >> > >>>> Cray compiler (clang-11). Previously I used PrgEnv-gnu, which did > >> not have > >> > >>>> this error. > >> > >>>> Probably it is a problem of Spock. But I am not sure. > >> > >>>> > >> > >>>> --Junchao Zhang > >> > >>>> > >> > >>>> > >> > >>>> On Sat, Jul 17, 2021 at 10:17 AM Mark Adams <mfadams(a)lbl.gov> > >> wrote: > >> > >>>> > >> > >>>>> And I can run a fortran test, with warnings, but C tests fail: > >> > >>>>> > >> > >>>>> 11:15 jczhang/fix-cray-mpicxx-includes/main= > >> > >>>>> /gpfs/alpine/csc314/scratch/adams/petsc$ make > >> > >>>>> PETSC_DIR=/gpfs/alpine/csc314/scratch/adams/petsc > >> > >>>>> PETSC_ARCH=arch-spock-opt-cray-kokkos -f gmakefile test > >> > >>>>> search='ts_utils_dmplexlandau_tutorials-ex1f90_0' > >> > >>>>> Using MAKEFLAGS: -- > >> search=ts_utils_dmplexlandau_tutorials-ex1f90_0 > >> > >>>>> PETSC_ARCH=arch-spock-opt-cray-kokkos > >> > >>>>> PETSC_DIR=/gpfs/alpine/csc314/scratch/adams/petsc > >> > >>>>> FC > >> > >>>>> > >> arch-spock-opt-cray-kokkos/tests/ts/utils/dmplexlandau/tutorials/ex1f90.o > >> > >>>>> FLINKER > >> > >>>>> > >> arch-spock-opt-cray-kokkos/tests/ts/utils/dmplexlandau/tutorials/ex1f90 > >> > >>>>> > >> /opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/bin/ld: > >> > >>>>> warning: alignment 128 of symbol > >> > >>>>> `$host_init$$runtime_init_for_iso_c_binding$iso_c_binding_' in > >> > >>>>> /opt/cray/pe/cce/11.0.4/cce/x86_64/lib/libmodules.so is smaller > >> than 256 in > >> > >>>>> > >> arch-spock-opt-cray-kokkos/tests/ts/utils/dmplexlandau/tutorials/ex1f90.o > >> > >>>>> > >> /opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/bin/ld: > >> > >>>>> warning: alignment 64 of symbol `$data_init$iso_c_binding_' in > >> > >>>>> /opt/cray/pe/cce/11.0.4/cce/x86_64/lib/libmodules.so is smaller > >> than 256 in > >> > >>>>> > >> arch-spock-opt-cray-kokkos/tests/ts/utils/dmplexlandau/tutorials/ex1f90.o > >> > >>>>> TEST > >> > >>>>> > >> arch-spock-opt-cray-kokkos/tests/counts/ts_utils_dmplexlandau_tutorials-ex1f90_0.counts > >> > >>>>> ok ts_utils_dmplexlandau_tutorials-ex1f90_0 > >> > >>>>> ok diff-ts_utils_dmplexlandau_tutorials-ex1f90_0 > >> > >>>>> > >> > >>>>> > >> > >>>>> On Sat, Jul 17, 2021 at 10:53 AM Mark Adams <mfadams(a)lbl.gov> > >> wrote: > >> > >>>>> > >> > >>>>>> HUmm, I can not reproduce this. > >> > >>>>>> > >> > >>>>>> 10:49 jczhang/fix-cray-mpicxx-includes/main= > >> > >>>>>> /gpfs/alpine/csc314/scratch/adams/petsc$ make > >> > >>>>>> PETSC_DIR=/gpfs/alpine/csc314/scratch/adams/petsc > >> > >>>>>> PETSC_ARCH=arch-spock-dbg-kokkos check > >> > >>>>>> Running check examples to verify correct installation > >> > >>>>>> Using PETSC_DIR=/gpfs/alpine/csc314/scratch/adams/petsc and > >> > >>>>>> PETSC_ARCH=arch-spock-dbg-kokkos > >> > >>>>>> gmake[3]: > >> > >>>>>> > >> [/gpfs/alpine/csc314/scratch/adams/petsc/lib/petsc/conf/rules:301: > >> > >>>>>> ex19.PETSc] Error 2 (ignored) > >> > >>>>>> *******************Error detected during compile or > >> > >>>>>> link!******************* > >> > >>>>>> See http://www.mcs.anl.gov/petsc/documentation/faq.html > >> > >>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials ex19 > >> > >>>>>> > >> > >>>>>> > >> ********************************************************************************* > >> > >>>>>> cc -L/sw/spock/spack-envs/views/rocm-4.1.0/lib -lamdhip64 > >> > >>>>>> -lhsa-runtime64 -fPIC -Wall -Wwrite-strings -Wno-strict-aliasing > >> > >>>>>> -Wno-unknown-pragmas -fstack-protector -Qunused-arguments > >> > >>>>>> -fvisibility=hidden -g -O0 -fPIC -Wall -Wwrite-strings > >> > >>>>>> -Wno-strict-aliasing -Wno-unknown-pragmas -fstack-protector > >> > >>>>>> -Qunused-arguments -fvisibility=hidden -g -O0 > >> > >>>>>> -I/sw/spock/spack-envs/views/rocm-4.1.0/include > >> > >>>>>> -I/gpfs/alpine/csc314/scratch/adams/petsc/include > >> > >>>>>> > >> -I/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/include > >> > >>>>>> -I/sw/spock/spack-envs/views/rocm-4.1.0/include > >> > >>>>>> -I/sw/spock/spack-envs/views/rocm-4.1.0/include ex19.c > >> > >>>>>> > >> -Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib > >> > >>>>>> > >> -L/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib > >> > >>>>>> > >> -Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib > >> > >>>>>> > >> -L/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib > >> > >>>>>> -Wl,-rpath,/sw/spock/spack-envs/views/rocm-4.1.0/lib > >> > >>>>>> -L/sw/spock/spack-envs/views/rocm-4.1.0/lib > >> > >>>>>> -Wl,-rpath,/opt/gcc/8.1.0/snos/lib64 -L/opt/gcc/8.1.0/snos/lib64 > >> > >>>>>> -Wl,-rpath,/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib > >> > >>>>>> -L/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib > >> > >>>>>> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib > >> > >>>>>> -L/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib > >> > >>>>>> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/gtl/lib > >> > >>>>>> -L/opt/cray/pe/mpich/8.1.4/gtl/lib > >> -Wl,-rpath,/opt/cray/pe/pmi/6.0.10/lib > >> > >>>>>> -L/opt/cray/pe/pmi/6.0.10/lib > >> -Wl,-rpath,/opt/cray/pe/dsmml/0.1.4/dsmml/lib > >> > >>>>>> -L/opt/cray/pe/dsmml/0.1.4/dsmml/lib > >> > >>>>>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce/x86_64/lib > >> > >>>>>> -L/opt/cray/pe/cce/11.0.4/cce/x86_64/lib > >> > >>>>>> -Wl,-rpath,/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64 > >> > >>>>>> -L/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64 > >> > >>>>>> > >> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux > >> > >>>>>> > >> -L/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux > >> > >>>>>> -Wl,-rpath,/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0 > >> > >>>>>> -L/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0 > >> > >>>>>> > >> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib > >> > >>>>>> > >> -L/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib > >> > >>>>>> -lpetsc -lkokkoskernels -lkokkoscontainers -lkokkoscore > >> -lhipsparse > >> > >>>>>> -lhipblas -lrocsparse -lrocsolver -lrocblas -lamdhip64 > >> -lhsa-runtime64 > >> > >>>>>> -lstdc++ -ldl -lpmi -lsci_cray_mpi -lsci_cray -lmpifort_cray > >> -lmpi_cray > >> > >>>>>> -lmpi_gtl_hsa -lxpmem -ldsmml -lpgas-shmem -lquadmath > >> -lcrayacc_amdgpu > >> > >>>>>> -lopenacc -lmodules -lfi -lcraymath -lf -lu -lcsup -lgfortran > >> -lpthread > >> > >>>>>> -lgcc_eh -lm -lclang_rt.craypgo-x86_64 -lclang_rt.builtins-x86_64 > >> > >>>>>> -lquadmath -lstdc++ -ldl -o ex19 > >> > >>>>>> ld.lld: error: > >> > >>>>>> > >> /gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib/libpetsc.so: > >> > >>>>>> undefined reference to > >> .omp_offloading.img_start.cray_amdgcn-amd-amdhsa > >> > >>>>>> [--no-allow-shlib-undefined] > >> > >>>>>> ld.lld: error: > >> > >>>>>> > >> /gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib/libpetsc.so: > >> > >>>>>> undefined reference to > >> .omp_offloading.img_size.cray_amdgcn-amd-amdhsa > >> > >>>>>> [--no-allow-shlib-undefined] > >> > >>>>>> ld.lld: error: > >> > >>>>>> > >> /gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib/libpetsc.so: > >> > >>>>>> undefined reference to > >> .omp_offloading.img_cache.cray_amdgcn-amd-amdhsa > >> > >>>>>> [--no-allow-shlib-undefined] > >> > >>>>>> clang-11: error: linker command failed with exit code 1 (use -v > >> to > >> > >>>>>> see invocation) > >> > >>>>>> gmake[4]: *** [<builtin>: ex19] Error 1 > >> > >>>>>> gmake[3]: > >> > >>>>>> > >> [/gpfs/alpine/csc314/scratch/adams/petsc/lib/petsc/conf/rules:301: > >> > >>>>>> ex3k.PETSc] Error 2 (ignored) > >> > >>>>>> *******************Error detected during compile or > >> > >>>>>> link!******************* > >> > >>>>>> See http://www.mcs.anl.gov/petsc/documentation/faq.html > >> > >>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials ex3k > >> > >>>>>> > >> > >>>>>> > >> ********************************************************************************* > >> > >>>>>> hipcc -fPIC -Wall -Wwrite-strings -Wno-strict-aliasing > >> > >>>>>> -Wno-unknown-pragmas -fvisibility=hidden -g -O0 -std=c++14 > >> > >>>>>> -I/gpfs/alpine/csc314/scratch/adams/petsc/include > >> > >>>>>> > >> -I/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/include > >> > >>>>>> -I/sw/spock/spack-envs/views/rocm-4.1.0/include > >> > >>>>>> -I/sw/spock/spack-envs/views/rocm-4.1.0/include > >> -I/opt/cray/pe/libsci/ > >> > >>>>>> 21.04.1.1/CRAY/9.0/x86_64/include > >> > >>>>>> -I/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/include > >> > >>>>>> -I/opt/cray/pe/pmi/6.0.10/include > >> -I/opt/cray/pe/dsmml/0.1.4/dsmml//include > >> > >>>>>> -I/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/include > >> > >>>>>> -L/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib > >> > >>>>>> -L/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib > >> > >>>>>> -L/opt/cray/pe/mpich/8.1.4/gtl/lib -L/opt/cray/pe/pmi/6.0.10/lib > >> > >>>>>> -L/opt/cray/pe/dsmml/0.1.4/dsmml//lib > >> > >>>>>> -L/opt/cray/pe/cce/11.0.4/cce/x86_64/lib/pkgconfig/../ > >> > >>>>>> -L/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64 > >> > >>>>>> -Wl,--as-needed,-lsci_cray_mpi,--no-as-needed > >> > >>>>>> -Wl,--as-needed,-lsci_cray,--no-as-needed > >> > >>>>>> -Wl,--as-needed,-lmpi_cray,--no-as-needed > >> > >>>>>> -Wl,--as-needed,-ldsmml,--no-as-needed > >> > >>>>>> -Wl,--as-needed,-lpgas-shmem,--no-as-needed -lpmi -ldl > >> -lmpi_gtl_hsa > >> > >>>>>> -lxpmem -lquadmath -lmodules -lfi -lcraymath -lf -lu -lcsup > >> > >>>>>> ex3k.kokkos.cxx > >> > >>>>>> > >> -Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib > >> > >>>>>> > >> -L/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib > >> > >>>>>> > >> -Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib > >> > >>>>>> > >> -L/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib > >> > >>>>>> -Wl,-rpath,/sw/spock/spack-envs/views/rocm-4.1.0/lib > >> > >>>>>> -L/sw/spock/spack-envs/views/rocm-4.1.0/lib > >> > >>>>>> -Wl,-rpath,/opt/gcc/8.1.0/snos/lib64 -L/opt/gcc/8.1.0/snos/lib64 > >> > >>>>>> -Wl,-rpath,/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib > >> > >>>>>> -L/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib > >> > >>>>>> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib > >> > >>>>>> -L/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib > >> > >>>>>> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/gtl/lib > >> > >>>>>> -L/opt/cray/pe/mpich/8.1.4/gtl/lib > >> -Wl,-rpath,/opt/cray/pe/pmi/6.0.10/lib > >> > >>>>>> -L/opt/cray/pe/pmi/6.0.10/lib > >> -Wl,-rpath,/opt/cray/pe/dsmml/0.1.4/dsmml/lib > >> > >>>>>> -L/opt/cray/pe/dsmml/0.1.4/dsmml/lib > >> > >>>>>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce/x86_64/lib > >> > >>>>>> -L/opt/cray/pe/cce/11.0.4/cce/x86_64/lib > >> > >>>>>> -Wl,-rpath,/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64 > >> > >>>>>> -L/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64 > >> > >>>>>> > >> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux > >> > >>>>>> > >> -L/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux > >> > >>>>>> -Wl,-rpath,/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0 > >> > >>>>>> -L/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0 > >> > >>>>>> > >> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib > >> > >>>>>> > >> -L/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib > >> > >>>>>> -lpetsc -lkokkoskernels -lkokkoscontainers -lkokkoscore > >> -lhipsparse > >> > >>>>>> -lhipblas -lrocsparse -lrocsolver -lrocblas -lamdhip64 > >> -lhsa-runtime64 > >> > >>>>>> -lstdc++ -ldl -lpmi -lsci_cray_mpi -lsci_cray -lmpifort_cray > >> -lmpi_cray > >> > >>>>>> -lmpi_gtl_hsa -lxpmem -ldsmml -lpgas-shmem -lquadmath > >> -lcrayacc_amdgpu > >> > >>>>>> -lopenacc -lmodules -lfi -lcraymath -lf -lu -lcsup -lgfortran > >> -lpthread > >> > >>>>>> -lgcc_eh -lm -lclang_rt.craypgo-x86_64 -lclang_rt.builtins-x86_64 > >> > >>>>>> -lquadmath -lstdc++ -ldl -o ex3k > >> > >>>>>> > >> /usr/lib64/gcc/x86_64-suse-linux/7/../../../../x86_64-suse-linux/bin/ld: > >> > >>>>>> > >> /gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib/libpetsc.so: > >> > >>>>>> undefined reference to > >> `.omp_offloading.img_size.cray_amdgcn-amd-amdhsa' > >> > >>>>>> > >> /usr/lib64/gcc/x86_64-suse-linux/7/../../../../x86_64-suse-linux/bin/ld: > >> > >>>>>> > >> /gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib/libpetsc.so: > >> > >>>>>> undefined reference to > >> `.omp_offloading.img_start.cray_amdgcn-amd-amdhsa' > >> > >>>>>> > >> /usr/lib64/gcc/x86_64-suse-linux/7/../../../../x86_64-suse-linux/bin/ld: > >> > >>>>>> > >> /gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib/libpetsc.so: > >> > >>>>>> undefined reference to > >> `.omp_offloading.img_cache.cray_amdgcn-amd-amdhsa' > >> > >>>>>> clang-12: error: linker command failed with exit code 1 (use -v > >> to > >> > >>>>>> see invocation) > >> > >>>>>> gmake[4]: *** > >> > >>>>>> [/gpfs/alpine/csc314/scratch/adams/petsc/lib/petsc/conf/test:26: > >> ex3k] > >> > >>>>>> Error 1 > >> > >>>>>> gmake[3]: [makefile:102: runex3k_kokkos] Error 2 (ignored) > >> > >>>>>> 1,25c1,4 > >> > >>>>>> < atol=1e-50, rtol=1e-08, stol=1e-08, maxit=50, maxf=10000 > >> > >>>>>> < Vec Object: Exact Solution 2 MPI processes > >> > >>>>>> < type: mpikokkos > >> > >>>>>> < Process [0] > >> > >>>>>> < 0. > >> > >>>>>> < 0.015625 > >> > >>>>>> < 0.125 > >> > >>>>>> < Process [1] > >> > >>>>>> < 0.421875 > >> > >>>>>> < 1. > >> > >>>>>> < Vec Object: Forcing function 2 MPI processes > >> > >>>>>> < type: mpikokkos > >> > >>>>>> < Process [0] > >> > >>>>>> < 1e-72 > >> > >>>>>> < 1.50024 > >> > >>>>>> < 3.01563 > >> > >>>>>> < Process [1] > >> > >>>>>> < 4.67798 > >> > >>>>>> < 7. > >> > >>>>>> < 0 SNES Function norm 5.414682427127e+00 > >> > >>>>>> < 1 SNES Function norm 2.952582418265e-01 > >> > >>>>>> < 2 SNES Function norm 4.502293658739e-04 > >> > >>>>>> < 3 SNES Function norm 1.389665806646e-09 > >> > >>>>>> < Number of SNES iterations = 3 > >> > >>>>>> < Norm of error 1.49752e-10 Iterations 3 > >> > >>>>>> --- > >> > >>>>>> > slurmstepd: error: execve(): > >> > >>>>>> > >> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials/./ex3k: No such > >> > >>>>>> file or directory > >> > >>>>>> > slurmstepd: error: execve(): > >> > >>>>>> > >> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials/./ex3k: No such > >> > >>>>>> file or directory > >> > >>>>>> > srun: error: spock01: tasks 0-1: Exited with exit code 2 > >> > >>>>>> > srun: launch/slurm: _step_signal: Terminating StepId=272979.0 > >> > >>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials > >> > >>>>>> Possible problem with ex3k running with kokkos-kernels, diffs > >> above > >> > >>>>>> ========================================= > >> > >>>>>> *******************Error detected during compile or > >> > >>>>>> link!******************* > >> > >>>>>> See http://www.mcs.anl.gov/petsc/documentation/faq.html > >> > >>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials ex5f > >> > >>>>>> ********************************************************* > >> > >>>>>> ftn -L/sw/spock/spack-envs/views/rocm-4.1.0/lib -lamdhip64 > >> > >>>>>> -lhsa-runtime64 -fPIC -g -O0 -fPIC -g -O0 > >> > >>>>>> -I/gpfs/alpine/csc314/scratch/adams/petsc/include > >> > >>>>>> > >> -I/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/include > >> > >>>>>> -I/sw/spock/spack-envs/views/rocm-4.1.0/include ex5f.F90 > >> > >>>>>> > >> -Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib > >> > >>>>>> > >> -L/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib > >> > >>>>>> > >> -Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib > >> > >>>>>> > >> -L/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib > >> > >>>>>> -Wl,-rpath,/sw/spock/spack-envs/views/rocm-4.1.0/lib > >> > >>>>>> -L/sw/spock/spack-envs/views/rocm-4.1.0/lib > >> > >>>>>> -Wl,-rpath,/opt/gcc/8.1.0/snos/lib64 -L/opt/gcc/8.1.0/snos/lib64 > >> > >>>>>> -Wl,-rpath,/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib > >> > >>>>>> -L/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib > >> > >>>>>> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib > >> > >>>>>> -L/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib > >> > >>>>>> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/gtl/lib > >> > >>>>>> -L/opt/cray/pe/mpich/8.1.4/gtl/lib > >> -Wl,-rpath,/opt/cray/pe/pmi/6.0.10/lib > >> > >>>>>> -L/opt/cray/pe/pmi/6.0.10/lib > >> -Wl,-rpath,/opt/cray/pe/dsmml/0.1.4/dsmml/lib > >> > >>>>>> -L/opt/cray/pe/dsmml/0.1.4/dsmml/lib > >> > >>>>>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce/x86_64/lib > >> > >>>>>> -L/opt/cray/pe/cce/11.0.4/cce/x86_64/lib > >> > >>>>>> -Wl,-rpath,/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64 > >> > >>>>>> -L/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64 > >> > >>>>>> > >> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux > >> > >>>>>> > >> -L/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux > >> > >>>>>> -Wl,-rpath,/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0 > >> > >>>>>> -L/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0 > >> > >>>>>> > >> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib > >> > >>>>>> > >> -L/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib > >> > >>>>>> -lpetsc -lkokkoskernels -lkokkoscontainers -lkokkoscore > >> -lhipsparse > >> > >>>>>> -lhipblas -lrocsparse -lrocsolver -lrocblas -lamdhip64 > >> -lhsa-runtime64 > >> > >>>>>> -lstdc++ -ldl -lpmi -lsci_cray_mpi -lsci_cray -lmpifort_cray > >> -lmpi_cray > >> > >>>>>> -lmpi_gtl_hsa -lxpmem -ldsmml -lpgas-shmem -lquadmath > >> -lcrayacc_amdgpu > >> > >>>>>> -lopenacc -lmodules -lfi -lcraymath -lf -lu -lcsup -lgfortran > >> -lpthread > >> > >>>>>> -lgcc_eh -lm -lclang_rt.craypgo-x86_64 -lclang_rt.builtins-x86_64 > >> > >>>>>> -lquadmath -lstdc++ -ldl -o ex5f > >> > >>>>>> > >> /opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/bin/ld: > >> > >>>>>> warning: alignment 128 of symbol > >> > >>>>>> `$host_init$$runtime_init_for_iso_c_binding$iso_c_binding_' in > >> > >>>>>> /opt/cray/pe/cce/11.0.4/cce/x86_64/lib/libmodules.so is smaller > >> than 256 in > >> > >>>>>> /tmp/pe_30429/ex5f_1.o > >> > >>>>>> > >> /opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/bin/ld: > >> > >>>>>> warning: alignment 64 of symbol `$data_init$iso_c_binding_' in > >> > >>>>>> /opt/cray/pe/cce/11.0.4/cce/x86_64/lib/libmodules.so is smaller > >> than 256 in > >> > >>>>>> /tmp/pe_30429/ex5f_1.o > >> > >>>>>> Fortran example src/snes/tutorials/ex5f run successfully with 1 > >> MPI > >> > >>>>>> process > >> > >>>>>> Completed test examples > >> > >>>>>> 10:49 jczhang/fix-cray-mpicxx-includes/main= > >> > >>>>>> /gpfs/alpine/csc314/scratch/adams/petsc$ ll > >> > >>>>>> > >> > >>>>>> On Fri, Jul 16, 2021 at 11:28 PM Junchao Zhang < > >> > >>>>>> junchao.zhang(a)gmail.com> wrote: > >> > >>>>>> > >> > >>>>>>> Mark, I configured with Fortran binding enabled with main + MR > >> !4175 > >> > >>>>>>> <https://gitlab.com/petsc/petsc/-/merge_requests/4175> > >> > >>>>>>> > >> > >>>>>>> '--with-mpiexec=srun', > >> > >>>>>>> '--with-shared-libraries=1', > >> > >>>>>>> '--with-cc=cc', > >> > >>>>>>> '--with-cxx=CC', > >> > >>>>>>> '--with-fc=ftn', > >> > >>>>>>> '--with-fortran-bindings', > >> > >>>>>>> '--with-hip', > >> > >>>>>>> '--with-hipc=hipcc', > >> > >>>>>>> '--with-debugging', > >> > >>>>>>> '--CPPFLAGS=-I${ROCM_PATH}/include', > >> > >>>>>>> '--CXXPPFLAGS=-I${ROCM_PATH}/include', > >> > >>>>>>> '--CC_LINKER_FLAGS=-L${ROCM_PATH}/lib -lamdhip64 > >> -lhsa-runtime64', > >> > >>>>>>> '--CXX_LINKER_FLAGS=-L${ROCM_PATH}/lib -lamdhip64 > >> -lhsa-runtime64', > >> > >>>>>>> '--FC_LINKER_FLAGS=-L${ROCM_PATH}/lib -lamdhip64 > >> -lhsa-runtime64', > >> > >>>>>>> '--COPTFLAGS=-g -O0', > >> > >>>>>>> '--CXXOPTFLAGS=-g -O0', > >> > >>>>>>> '--FOPTFLAGS=-g -O0', > >> > >>>>>>> '--download-kokkos', > >> > >>>>>>> '--download-kokkos-kernels', > >> > >>>>>>> '--download-kokkos-commit=3.4.01', > >> > >>>>>>> '--download-kokkos-kernels-commit=3.4.01', > >> > >>>>>>> '--with-kokkos-hip-arch=VEGA908', > >> > >>>>>>> > >> > >>>>>>> and 'make check' ran smoothly on a compute node > >> > >>>>>>> > >> > >>>>>>> $ make check > >> > >>>>>>> Running check examples to verify correct installation > >> > >>>>>>> Using PETSC_DIR=/ccs/home/jczhang/petsc and > >> > >>>>>>> PETSC_ARCH=arch-spock-cray-kokkos-dbg > >> > >>>>>>> C/C++ example src/snes/tutorials/ex19 run successfully with 1 > >> MPI > >> > >>>>>>> process > >> > >>>>>>> C/C++ example src/snes/tutorials/ex19 run successfully with 2 > >> MPI > >> > >>>>>>> processes > >> > >>>>>>> C/C++ example src/snes/tutorials/ex3k run successfully with > >> > >>>>>>> kokkos-kernels > >> > >>>>>>> Fortran example src/snes/tutorials/ex5f run successfully with 1 > >> MPI > >> > >>>>>>> process > >> > >>>>>>> > >> > >>>>>>> Completed test examples > >> > >>>>>>> > >> > >>>>>>> > >> > >>>>>>> --Junchao Zhang > >> > >>>>>>> > >> > >>>>>>> > >> > >>>>>>> On Fri, Jul 16, 2021 at 6:04 PM Mark Adams <mfadams(a)lbl.gov> > >> wrote: > >> > >>>>>>> > >> > >>>>>>>> And I find that this error, on non-Kokkos C tests, is fixed by > >> > >>>>>>>> turning the fortran bindings off: > >> > >>>>>>>> > >> > >>>>>>>> ld.lld: error: > >> > >>>>>>>> > >> /gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/lib/libpetsc.so: > >> > >>>>>>>> undefined reference to > >> .omp_offloading.img_start.cray_amdgcn-amd-amdhsa > >> > >>>>>>>> [--no-allow-shlib-undefined] > >> > >>>>>>>> ld.lld: error: > >> > >>>>>>>> > >> /gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/lib/libpetsc.so: > >> > >>>>>>>> undefined reference to > >> .omp_offloading.img_size.cray_amdgcn-amd-amdhsa > >> > >>>>>>>> [--no-allow-shlib-undefined] > >> > >>>>>>>> ld.lld: error: > >> > >>>>>>>> > >> /gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/lib/libpetsc.so: > >> > >>>>>>>> undefined reference to > >> .omp_offloading.img_cache.cray_amdgcn-amd-amdhsa > >> > >>>>>>>> [--no-allow-shlib-undefined] > >> > >>>>>>>> > >> > >>>>>>>> On Fri, Jul 16, 2021 at 3:53 PM Mark Adams <mfadams(a)lbl.gov> > >> wrote: > >> > >>>>>>>> > >> > >>>>>>>>> Not complex. THis has some overlap with my problem w/o Kokkos. > >> > >>>>>>>>> > >> > >>>>>>>>> On Fri, Jul 16, 2021 at 12:54 PM Junchao Zhang < > >> > >>>>>>>>> junchao.zhang(a)gmail.com> wrote: > >> > >>>>>>>>> > >> > >>>>>>>>>> Do you use complex? post your configure.log. > >> > >>>>>>>>>> > >> > >>>>>>>>>> --Junchao Zhang > >> > >>>>>>>>>> > >> > >>>>>>>>>> > >> > >>>>>>>>>> On Fri, Jul 16, 2021 at 9:47 AM Mark Adams <mfadams(a)lbl.gov> > >> > >>>>>>>>>> wrote: > >> > >>>>>>>>>> > >> > >>>>>>>>>>> The simple Kokkos example is failing for me on Spock. > >> > >>>>>>>>>>> Any ideas? > >> > >>>>>>>>>>> Thanks, > >> > >>>>>>>>>>> > >> > >>>>>>>>>>> 10:44 main *= > >> > >>>>>>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials$ > >> make > >> > >>>>>>>>>>> PETSC_DIR=/gpfs/alpine/csc314/scratch/adams/petsc > >> > >>>>>>>>>>> PETSC_ARCH=arch-spock-opt-cray-kokkos ex3k > >> > >>>>>>>>>>> MPICH_CXX="hipcc" OMPI_CXX="hipcc" CC > >> > >>>>>>>>>>> -L/sw/spock/spack-envs/views/rocm-4.1.0/lib -lhsa-runtime64 > >> > >>>>>>>>>>> -L/sw/spock/spack-envs/views/rocm-4.1.0/lib -lamdhip64 > >> -lhsa-runtime64 > >> > >>>>>>>>>>> -Wall -Wwrite-strings -Wno-strict-aliasing > >> -Wno-unknown-pragmas > >> > >>>>>>>>>>> -fstack-protector -fvisibility=hidden -g -O2 -fPIC -Wall > >> -Wwrite-strings > >> > >>>>>>>>>>> -Wno-strict-aliasing -Wno-unknown-pragmas > >> -fvisibility=hidden -g -O3 > >> > >>>>>>>>>>> -std=c++14 > >> -I/gpfs/alpine/csc314/scratch/adams/petsc/include > >> > >>>>>>>>>>> > >> -I/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-opt-cray-kokkos/include > >> > >>>>>>>>>>> > >> -I/gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include > >> > >>>>>>>>>>> -I/sw/spock/spack-envs/views/rocm-4.1.0/include > >> > >>>>>>>>>>> -I/sw/spock/spack-envs/views/rocm-4.1.0/include > >> ex3k.kokkos.cxx > >> > >>>>>>>>>>> > >> -Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-opt-cray-kokkos/lib > >> > >>>>>>>>>>> > >> -L/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-opt-cray-kokkos/lib > >> > >>>>>>>>>>> > >> -Wl,-rpath,/gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/lib > >> > >>>>>>>>>>> > >> -L/gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/lib > >> > >>>>>>>>>>> -Wl,-rpath,/sw/spock/spack-envs/views/rocm-4.1.0/lib > >> > >>>>>>>>>>> -L/sw/spock/spack-envs/views/rocm-4.1.0/lib > >> > >>>>>>>>>>> -Wl,-rpath,/opt/gcc/8.1.0/snos/lib64 > >> -L/opt/gcc/8.1.0/snos/lib64 > >> > >>>>>>>>>>> -Wl,-rpath,/opt/cray/pe/libsci/ > >> 21.04.1.1/CRAY/9.0/x86_64/lib > >> > >>>>>>>>>>> -L/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib > >> > >>>>>>>>>>> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib > >> > >>>>>>>>>>> -L/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib > >> > >>>>>>>>>>> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/gtl/lib > >> > >>>>>>>>>>> -L/opt/cray/pe/mpich/8.1.4/gtl/lib > >> -Wl,-rpath,/opt/cray/pe/pmi/6.0.10/lib > >> > >>>>>>>>>>> -L/opt/cray/pe/pmi/6.0.10/lib > >> -Wl,-rpath,/opt/cray/pe/dsmml/0.1.4/dsmml/lib > >> > >>>>>>>>>>> -L/opt/cray/pe/dsmml/0.1.4/dsmml/lib > >> > >>>>>>>>>>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce/x86_64/lib > >> > >>>>>>>>>>> -L/opt/cray/pe/cce/11.0.4/cce/x86_64/lib > >> > >>>>>>>>>>> > >> -Wl,-rpath,/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64 > >> > >>>>>>>>>>> -L/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64 > >> > >>>>>>>>>>> > >> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux > >> > >>>>>>>>>>> > >> -L/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux > >> > >>>>>>>>>>> > >> -Wl,-rpath,/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0 > >> > >>>>>>>>>>> -L/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0 > >> > >>>>>>>>>>> > >> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib > >> > >>>>>>>>>>> > >> -L/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib > >> > >>>>>>>>>>> -lpetsc -lkokkoskernels -lkokkoscontainers -lkokkoscore > >> -lhipsparse > >> > >>>>>>>>>>> -lhipblas -lrocsparse -lrocsolver -lrocblas -lamdhip64 > >> -lhsa-runtime64 > >> > >>>>>>>>>>> -lstdc++ -ldl -lpmi -lsci_cray_mpi -lsci_cray > >> -lmpifort_cray -lmpi_cray > >> > >>>>>>>>>>> -lmpi_gtl_hsa -lxpmem -ldsmml -lpgas-shmem -lquadmath > >> -lcrayacc_amdgpu > >> > >>>>>>>>>>> -lopenacc -lmodules -lfi -lcraymath -lf -lu -lcsup > >> -lgfortran -lpthread > >> > >>>>>>>>>>> -lgcc_eh -lm -lclang_rt.craypgo-x86_64 > >> -lclang_rt.builtins-x86_64 > >> > >>>>>>>>>>> -lquadmath -lstdc++ -ldl -o ex3k > >> > >>>>>>>>>>> In file included from ex3k.kokkos.cxx:3: > >> > >>>>>>>>>>> In file included from > >> > >>>>>>>>>>> > >> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscdmda_kokkos.hpp:4: > >> > >>>>>>>>>>> In file included from > >> > >>>>>>>>>>> > >> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec_kokkos.hpp:14: > >> > >>>>>>>>>>> In file included from > >> > >>>>>>>>>>> > >> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec.h:9: > >> > >>>>>>>>>>> In file included from > >> > >>>>>>>>>>> > >> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsys.h:42: > >> > >>>>>>>>>>> In file included from > >> > >>>>>>>>>>> > >> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsystypes.h:255: > >> > >>>>>>>>>>> In file included from > >> > >>>>>>>>>>> > >> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Complex.hpp:47: > >> > >>>>>>>>>>> In file included from > >> > >>>>>>>>>>> > >> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Atomic.hpp:212: > >> > >>>>>>>>>>> > >> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/impl/Kokkos_Atomic_Exchange.hpp:259:11: > >> > >>>>>>>>>>> error: no member named 'lock_address_host_space' in > >> namespace > >> > >>>>>>>>>>> 'Kokkos::Impl::Kokkos::Impl'; did you mean simply > >> 'lock_address_host_space'? > >> > >>>>>>>>>>> while (!Impl::lock_address_host_space((void*)dest)) > >> > >>>>>>>>>>> ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~ > >> > >>>>>>>>>>> lock_address_host_space > >> > >>>>>>>>>>> > >> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_HostSpace.hpp:83:6: > >> > >>>>>>>>>>> note: 'lock_address_host_space' declared here > >> > >>>>>>>>>>> bool lock_address_host_space(void* ptr); > >> > >>>>>>>>>>> ^ > >> > >>>>>>>>>>> In file included from ex3k.kokkos.cxx:3: > >> > >>>>>>>>>>> In file included from > >> > >>>>>>>>>>> > >> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscdmda_kokkos.hpp:4: > >> > >>>>>>>>>>> In file included from > >> > >>>>>>>>>>> > >> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec_kokkos.hpp:14: > >> > >>>>>>>>>>> In file included from > >> > >>>>>>>>>>> > >> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec.h:9: > >> > >>>>>>>>>>> In file included from > >> > >>>>>>>>>>> > >> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsys.h:42: > >> > >>>>>>>>>>> In file included from > >> > >>>>>>>>>>> > >> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsystypes.h:255: > >> > >>>>>>>>>>> In file included from > >> > >>>>>>>>>>> > >> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Complex.hpp:47: > >> > >>>>>>>>>>> In file included from > >> > >>>>>>>>>>> > >> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Atomic.hpp:212: > >> > >>>>>>>>>>> > >> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/impl/Kokkos_Atomic_Exchange.hpp:261:3: > >> > >>>>>>>>>>> error: no member named 'memory_fence' in namespace > >> 'Kokkos::Impl::Kokkos'; > >> > >>>>>>>>>>> did you mean simply 'memory_fence'? > >> > >>>>>>>>>>> Kokkos::memory_fence(); > >> > >>>>>>>>>>> ^~~~~~~~~~~~~~~~~~~~ > >> > >>>>>>>>>>> memory_fence > >> > >>>>>>>>>>> > >> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/impl/Kokkos_Memory_Fence.hpp:53:6: > >> > >>>>>>>>>>> note: 'memory_fence' declared here > >> > >>>>>>>>>>> void memory_fence() { > >> > >>>>>>>>>>> ^ > >> > >>>>>>>>>>> In file included from ex3k.kokkos.cxx:3: > >> > >>>>>>>>>>> In file included from > >> > >>>>>>>>>>> > >> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscdmda_kokkos.hpp:4: > >> > >>>>>>>>>>> In file included from > >> > >>>>>>>>>>> > >> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec_kokkos.hpp:14: > >> > >>>>>>>>>>> In file included from > >> > >>>>>>>>>>> > >> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec.h:9: > >> > >>>>>>>>>>> In file included from > >> > >>>>>>>>>>> > >> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsys.h:42: > >> > >>>>>>>>>>> In file included from > >> > >>>>>>>>>>> > >> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsystypes.h:255: > >> > >>>>>>>>>>> In file included from > >> > >>>>>>>>>>> > >> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Complex.hpp:47: > >> > >>>>>>>>>>> In file included from > >> > >>>>>>>>>>> > >> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Atomic.hpp:212: > >> > >>>>>>>>>>> > >> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/impl/Kokkos_Atomic_Exchange.hpp:277:3: > >> > >>>>>>>>>>> error: no member named 'memory_fence' in namespace > >> 'Kokkos::Impl::Kokkos'; > >> > >>>>>>>>>>> did you mean simply 'memory_fence'? > >> > >>>>>>>>>>> Kokkos::memory_fence(); > >> > >>>>>>>>>>> ^~~~~~~~~~~~~~~~~~~~ > >> > >>>>>>>>>>> memory_fence > >> > >>>>>>>>>>> > >> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/impl/Kokkos_Memory_Fence.hpp:53:6: > >> > >>>>>>>>>>> note: 'memory_fence' declared here > >> > >>>>>>>>>>> void memory_fence() { > >> > >>>>>>>>>>> ^ > >> > >>>>>>>>>>> > >> > >>>>>>>>>> > >> > >> > >> > >> > >> -- > >> Scott Kruger > >> Tech-X Corporation kruger(a)txcorp.com > >> 5621 Arapahoe Ave, Suite A Phone: (720) 466-3196 > >> Boulder, CO 80303 Fax: (303) 448-7756 > >> > > -- Scott Kruger Tech-X Corporation kruger(a)txcorp.com 5621 Arapahoe Ave, Suite A Phone: (720) 466-3196 Boulder, CO 80303 Fax: (303) 448-7756

1 0

Re: [petsc-dev] Notice: CI outage July 18-20th
by Satish Balay 20 Jul '21

20 Jul '21

To update: CI should be functional now. Satish On Wed, 14 Jul 2021, Satish Balay via petsc-dev wrote: > A heads-up, > > PETSc CI will be down during this outage - and it can take some time (after this outage) to bring back all the CI machines. > > Satish > > ---------- Forwarded message ---------- > Date: Mon, 12 Jul 2021 22:11:41 +0000 > From: "Murphy-Olson, Daniel E." <d(a)anl.gov> > To: "systems-announce(a)cels.anl.gov" <systems-announce(a)cels.anl.gov> > Subject: [Systems Announce] Computing outage, July 18-20th > > Due to the construction in the data center related to the Aurora project, the power to building 240 (including the data center) will be interrupted for most of the day on Monday, July 19. > > > For CELS Computing, this will affect the various computing clusters in the room (you should get a separate communication on those), as well as all file servers for the MCS Legacy computing environment and the CELS General Compute Environment (GCE) will be offline beginning at 5PM on Sunday, July 18. As such, all workstations and login nodes will be down. Account management for both (accounts.mcs.anl.gov, accounts.cels.anl.gov) will also be offline. > > > Affected services will be down until Tuesday morning. We will send updates as we determine when services can be restored and/or when they become available again. > > > BIO divisional users, your Y drive and Z drive will be down during this period. > > > Web services hosted and provided by CELS Systems (Wordpress, Confluence, Jira, Gitlab, svn, repocafe), mailing lists, and divisional websites will remain up and online, though some aspects that rely on components in building 240 may be unresponsive. This does not affect general email services, nor any other services provided by BIS. > > > If power is restored sooner than the expected window, we will bring things back online and send announcements accordingly. > > > We know this is not convenient. Thanks for your understanding. > > > -Dan >

1 0

Re: [petsc-dev] Kokkos make error on Spock
by Mark Adams 20 Jul '21

20 Jul '21

OK, it looks like this flag caused the problem and does not seem to be necessary: '--LDFLAGS=-L'+os.environ['ROCM_PATH']+'/lib -lhsa-runtime64', On Tue, Jul 20, 2021 at 9:00 AM Mark Adams <mfadams(a)lbl.gov> wrote: > > > On Mon, Jul 19, 2021 at 6:41 PM Scott Kruger <kruger(a)txcorp.com> wrote: > >> >> Mark, >> >> On tulip, things with fortran went wonky when a `-fast` flag snuck into >> the flags (snuck in being me copying a previous configure file and not >> questioning the flags). The reason is that for clang, `-fast` implies >> link time optimization (lto) for C/C++ code, but flang doesn't support >> lto so things got weird. I suspect that gfortran does not either (but >> perhaps the real question is why not use flang?) >> > > It does work with GNU. I am doing this for an application and I want to > avoid dictating the program env. > > I'm (re)trying with -O0 just to check. > > >> >> I look at configure and I don't see anything in your flags that would >> trigger lto, but as Junchao says, it might be picking something up from >> Kokkos, so perhaps this is the issue. >> >> > The configure files that I sent did not have Kokkos. They were stripped > down. > > I have asked ORNL, but don't expect help. > The person (Matt) that is helping me for other things is very helpful so > perhaps he can come up with some ideas. > Otherwise I will tell the app that they need to use GNU for now. > > Thanks, > Mark > > > >> Scott >> >> >> On 2021-07-19 07:19, Mark Adams did write: >> > Thanks, but this happens w/o Kokkos. >> > I've stripped this down and attached good/bad logs without/with Fortran >> > bindings. >> > Hope this helps, >> > Thanks again, >> > Mark >> > >> > On Sun, Jul 18, 2021 at 12:00 PM Stefano Zampini < >> stefano.zampini(a)gmail.com> >> > wrote: >> > >> > > This is probably kokkos pulling in the dependencies where compiling >> kokkos >> > > source within PETSc. >> > > >> > > Il Dom 18 Lug 2021, 16:29 Mark Adams <mfadams(a)lbl.gov> ha scritto: >> > > >> > >> Whoops, this error was just from not telling gfortran to allow long >> lines. >> > >> >> > >> Anway, I did find that the when fortran bindings are enabled this OMP >> > >> thing gets into the library. No idea how. I'll askm ORNL. >> > >> >> > >> 10:26 jczhang/fix-cray-mpicxx-includes/main= >> > >> /gpfs/alpine/csc314/scratch/adams/petsc2$ nm >> > >> >> /gpfs/alpine/csc314/scratch/adams/petsc2/arch-spock-opt-cray-kokkos/lib/libpetsc.so >> > >> |g offload >> > >> U .omp_offloading.img_cache.cray_amdgcn-amd-amdhsa >> > >> U .omp_offloading.img_size.cray_amdgcn-amd-amdhsa >> > >> U .omp_offloading.img_start.cray_amdgcn-amd-amdhsa >> > >> 0000000001d457b0 T vecgetoffloadmask_ >> > >> >> > >> >> > >> On Sun, Jul 18, 2021 at 8:43 AM Mark Adams <mfadams(a)lbl.gov> wrote: >> > >> >> > >>> Ah, your test was not on Spock ... I have something working but >> this is >> > >>> strange. >> > >>> >> > >>> I switched to the GNU ProgEnv. and it passed the Fortran test in >> > >>> 'check', but this failed: >> > >>> >> > >>> >> > >>> >> > >>> >> > >>> >> > >>> >> > >>> >> > >>> >> > >>> >> > >>> *08:26 2 jczhang/fix-cray-mpicxx-includes/main= >> > >>> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials$ make >> > >>> >> PETSC_DIR=/gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-dbg-gnu >> > >>> PETSC_ARCH="" ex40f90ftn -L/sw/spock/spack-envs/views/rocm-4.1.0/lib >> > >>> -lamdhip64 -lhsa-runtime64 -fPIC -g -O0 -fPIC -g -O0 >> > >>> >> -I/gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-dbg-gnu/include >> > >>> -I/sw/spock/spack-envs/views/rocm-4.1.0/include ex40f90.F90 >> > >>> >> -Wl,-rpath,/gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-dbg-gnu/lib >> > >>> >> -L/gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-dbg-gnu/lib >> > >>> -Wl,-rpath,/sw/spock/spack-envs/views/rocm-4.1.0/lib >> > >>> -L/sw/spock/spack-envs/views/rocm-4.1.0/lib >> > >>> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/ofi/gnu/9.1/lib >> > >>> -L/opt/cray/pe/mpich/8.1.4/ofi/gnu/9.1/lib >> > >>> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/gtl/lib >> > >>> -L/opt/cray/pe/mpich/8.1.4/gtl/lib >> > >>> -Wl,-rpath,/opt/cray/pe/libsci/21.04.1.1/GNU/9.1/x86_64/lib >> > >>> <http://21.04.1.1/GNU/9.1/x86_64/lib> >> > >>> -L/opt/cray/pe/libsci/21.04.1.1/GNU/9.1/x86_64/lib >> > >>> <http://21.04.1.1/GNU/9.1/x86_64/lib> >> > >>> -Wl,-rpath,/opt/cray/pe/pmi/6.0.10/lib -L/opt/cray/pe/pmi/6.0.10/lib >> > >>> -Wl,-rpath,/opt/cray/pe/dsmml/0.1.4/dsmml/lib >> > >>> -L/opt/cray/pe/dsmml/0.1.4/dsmml/lib >> > >>> -Wl,-rpath,/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64 >> > >>> -L/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64 >> > >>> -Wl,-rpath,/opt/gcc/10.2.0/snos/lib/gcc/x86_64-suse-linux/10.2.0 >> > >>> -L/opt/gcc/10.2.0/snos/lib/gcc/x86_64-suse-linux/10.2.0 >> > >>> -Wl,-rpath,/opt/gcc/10.2.0/snos/lib64 -L/opt/gcc/10.2.0/snos/lib64 >> > >>> -Wl,-rpath,/sw/spock/spack-envs/views/rocm-4.1.0/llvm/lib >> > >>> -L/sw/spock/spack-envs/views/rocm-4.1.0/llvm/lib >> > >>> -Wl,-rpath,/opt/gcc/10.2.0/snos/lib -L/opt/gcc/10.2.0/snos/lib >> -lpetsc >> > >>> -lhipsparse -lhipblas -lrocsparse -lrocsolver -lrocblas -lamdhip64 >> > >>> -lhsa-runtime64 -lstdc++ -ldl -lpmi -lmpi_gtl_hsa -lxpmem >> -lgfortran -lm >> > >>> -lsci_gnu_82_mpi -lsci_gnu_82 -lmpifort_gnu_91 -lmpi_gnu_91 -ldsmml >> > >>> -lgfortran -lquadmath -lpthread -lm -lgcc_s -lquadmath -lstdc++ >> -ldl -o >> > >>> ex40f90ex40f90.F90:35:132: 35 | call >> > >>> >> DMDACreate2d(PETSC_COMM_WORLD,DM_BOUNDARY_NONE,DM_BOUNDARY_NONE,DMDA_STENCIL_BOX,ten,ten,PETSC_DECIDE,PETSC_DECIDE,two,one, >> > >>> & | >> > >>> >> 1Error: >> > >>> Line truncated at (1) [-Werror=line-truncation]ex40f90.F90:35:132:* >> > >>> * ....* >> > >>> >> > >>> But ex5f (the Fortran check test) does work manually also (sanity >> check). >> > >>> >> > >>> Oddly I seem to have the cray compilers: >> > >>> >> > >>> 08:34 jczhang/fix-cray-mpicxx-includes/main= >> > >>> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials$ which >> ftn >> > >>> /opt/cray/pe/craype/2.7.6/bin/ftn >> > >>> >> > >>> yet it is GNU: >> > >>> >> > >>> 08:34 jczhang/fix-cray-mpicxx-includes/main= >> > >>> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials$ ftn >> --version >> > >>> GNU Fortran (GCC) 10.2.0 20200723 (Cray Inc.) >> > >>> Copyright (C) 2020 Free Software Foundation, Inc. >> > >>> This is free software; see the source for copying conditions. >> There is >> > >>> NO >> > >>> warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR >> > >>> PURPOSE. >> > >>> >> > >>> The documentation says ftn should be here: >> > >>> >> > >>> 08:34 jczhang/fix-cray-mpicxx-includes/main= >> > >>> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials$ >> > >>> $GCC_PATH/bin/gfortran --version >> > >>> GNU Fortran (GCC) 10.2.0 20200723 (Cray Inc.) >> > >>> Copyright (C) 2020 Free Software Foundation, Inc. >> > >>> This is free software; see the source for copying conditions. >> There is >> > >>> NO >> > >>> warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR >> > >>> PURPOSE. >> > >>> >> > >>> Same, but the path is not the same: >> > >>> >> > >>> 08:39 jczhang/fix-cray-mpicxx-includes/main= >> > >>> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials$ echo >> $GCC_PATH >> > >>> /opt/gcc/10.2.0 >> > >>> >> > >>> Anyway, thanks for your help, >> > >>> Mark >> > >>> >> > >>> On Sat, Jul 17, 2021 at 11:26 PM Junchao Zhang < >> junchao.zhang(a)gmail.com> >> > >>> wrote: >> > >>> >> > >>>> Mark, I can reproduce this error with PrgEnv-cray, i.e., using the >> > >>>> Cray compiler (clang-11). Previously I used PrgEnv-gnu, which did >> not have >> > >>>> this error. >> > >>>> Probably it is a problem of Spock. But I am not sure. >> > >>>> >> > >>>> --Junchao Zhang >> > >>>> >> > >>>> >> > >>>> On Sat, Jul 17, 2021 at 10:17 AM Mark Adams <mfadams(a)lbl.gov> >> wrote: >> > >>>> >> > >>>>> And I can run a fortran test, with warnings, but C tests fail: >> > >>>>> >> > >>>>> 11:15 jczhang/fix-cray-mpicxx-includes/main= >> > >>>>> /gpfs/alpine/csc314/scratch/adams/petsc$ make >> > >>>>> PETSC_DIR=/gpfs/alpine/csc314/scratch/adams/petsc >> > >>>>> PETSC_ARCH=arch-spock-opt-cray-kokkos -f gmakefile test >> > >>>>> search='ts_utils_dmplexlandau_tutorials-ex1f90_0' >> > >>>>> Using MAKEFLAGS: -- >> search=ts_utils_dmplexlandau_tutorials-ex1f90_0 >> > >>>>> PETSC_ARCH=arch-spock-opt-cray-kokkos >> > >>>>> PETSC_DIR=/gpfs/alpine/csc314/scratch/adams/petsc >> > >>>>> FC >> > >>>>> >> arch-spock-opt-cray-kokkos/tests/ts/utils/dmplexlandau/tutorials/ex1f90.o >> > >>>>> FLINKER >> > >>>>> >> arch-spock-opt-cray-kokkos/tests/ts/utils/dmplexlandau/tutorials/ex1f90 >> > >>>>> >> /opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/bin/ld: >> > >>>>> warning: alignment 128 of symbol >> > >>>>> `$host_init$$runtime_init_for_iso_c_binding$iso_c_binding_' in >> > >>>>> /opt/cray/pe/cce/11.0.4/cce/x86_64/lib/libmodules.so is smaller >> than 256 in >> > >>>>> >> arch-spock-opt-cray-kokkos/tests/ts/utils/dmplexlandau/tutorials/ex1f90.o >> > >>>>> >> /opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/bin/ld: >> > >>>>> warning: alignment 64 of symbol `$data_init$iso_c_binding_' in >> > >>>>> /opt/cray/pe/cce/11.0.4/cce/x86_64/lib/libmodules.so is smaller >> than 256 in >> > >>>>> >> arch-spock-opt-cray-kokkos/tests/ts/utils/dmplexlandau/tutorials/ex1f90.o >> > >>>>> TEST >> > >>>>> >> arch-spock-opt-cray-kokkos/tests/counts/ts_utils_dmplexlandau_tutorials-ex1f90_0.counts >> > >>>>> ok ts_utils_dmplexlandau_tutorials-ex1f90_0 >> > >>>>> ok diff-ts_utils_dmplexlandau_tutorials-ex1f90_0 >> > >>>>> >> > >>>>> >> > >>>>> On Sat, Jul 17, 2021 at 10:53 AM Mark Adams <mfadams(a)lbl.gov> >> wrote: >> > >>>>> >> > >>>>>> HUmm, I can not reproduce this. >> > >>>>>> >> > >>>>>> 10:49 jczhang/fix-cray-mpicxx-includes/main= >> > >>>>>> /gpfs/alpine/csc314/scratch/adams/petsc$ make >> > >>>>>> PETSC_DIR=/gpfs/alpine/csc314/scratch/adams/petsc >> > >>>>>> PETSC_ARCH=arch-spock-dbg-kokkos check >> > >>>>>> Running check examples to verify correct installation >> > >>>>>> Using PETSC_DIR=/gpfs/alpine/csc314/scratch/adams/petsc and >> > >>>>>> PETSC_ARCH=arch-spock-dbg-kokkos >> > >>>>>> gmake[3]: >> > >>>>>> >> [/gpfs/alpine/csc314/scratch/adams/petsc/lib/petsc/conf/rules:301: >> > >>>>>> ex19.PETSc] Error 2 (ignored) >> > >>>>>> *******************Error detected during compile or >> > >>>>>> link!******************* >> > >>>>>> See http://www.mcs.anl.gov/petsc/documentation/faq.html >> > >>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials ex19 >> > >>>>>> >> > >>>>>> >> ********************************************************************************* >> > >>>>>> cc -L/sw/spock/spack-envs/views/rocm-4.1.0/lib -lamdhip64 >> > >>>>>> -lhsa-runtime64 -fPIC -Wall -Wwrite-strings -Wno-strict-aliasing >> > >>>>>> -Wno-unknown-pragmas -fstack-protector -Qunused-arguments >> > >>>>>> -fvisibility=hidden -g -O0 -fPIC -Wall -Wwrite-strings >> > >>>>>> -Wno-strict-aliasing -Wno-unknown-pragmas -fstack-protector >> > >>>>>> -Qunused-arguments -fvisibility=hidden -g -O0 >> > >>>>>> -I/sw/spock/spack-envs/views/rocm-4.1.0/include >> > >>>>>> -I/gpfs/alpine/csc314/scratch/adams/petsc/include >> > >>>>>> >> -I/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/include >> > >>>>>> -I/sw/spock/spack-envs/views/rocm-4.1.0/include >> > >>>>>> -I/sw/spock/spack-envs/views/rocm-4.1.0/include ex19.c >> > >>>>>> >> -Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib >> > >>>>>> >> -L/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib >> > >>>>>> >> -Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib >> > >>>>>> >> -L/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib >> > >>>>>> -Wl,-rpath,/sw/spock/spack-envs/views/rocm-4.1.0/lib >> > >>>>>> -L/sw/spock/spack-envs/views/rocm-4.1.0/lib >> > >>>>>> -Wl,-rpath,/opt/gcc/8.1.0/snos/lib64 -L/opt/gcc/8.1.0/snos/lib64 >> > >>>>>> -Wl,-rpath,/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib >> > >>>>>> -L/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib >> > >>>>>> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib >> > >>>>>> -L/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib >> > >>>>>> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/gtl/lib >> > >>>>>> -L/opt/cray/pe/mpich/8.1.4/gtl/lib >> -Wl,-rpath,/opt/cray/pe/pmi/6.0.10/lib >> > >>>>>> -L/opt/cray/pe/pmi/6.0.10/lib >> -Wl,-rpath,/opt/cray/pe/dsmml/0.1.4/dsmml/lib >> > >>>>>> -L/opt/cray/pe/dsmml/0.1.4/dsmml/lib >> > >>>>>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce/x86_64/lib >> > >>>>>> -L/opt/cray/pe/cce/11.0.4/cce/x86_64/lib >> > >>>>>> -Wl,-rpath,/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64 >> > >>>>>> -L/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64 >> > >>>>>> >> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux >> > >>>>>> >> -L/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux >> > >>>>>> -Wl,-rpath,/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0 >> > >>>>>> -L/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0 >> > >>>>>> >> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib >> > >>>>>> >> -L/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib >> > >>>>>> -lpetsc -lkokkoskernels -lkokkoscontainers -lkokkoscore >> -lhipsparse >> > >>>>>> -lhipblas -lrocsparse -lrocsolver -lrocblas -lamdhip64 >> -lhsa-runtime64 >> > >>>>>> -lstdc++ -ldl -lpmi -lsci_cray_mpi -lsci_cray -lmpifort_cray >> -lmpi_cray >> > >>>>>> -lmpi_gtl_hsa -lxpmem -ldsmml -lpgas-shmem -lquadmath >> -lcrayacc_amdgpu >> > >>>>>> -lopenacc -lmodules -lfi -lcraymath -lf -lu -lcsup -lgfortran >> -lpthread >> > >>>>>> -lgcc_eh -lm -lclang_rt.craypgo-x86_64 -lclang_rt.builtins-x86_64 >> > >>>>>> -lquadmath -lstdc++ -ldl -o ex19 >> > >>>>>> ld.lld: error: >> > >>>>>> >> /gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib/libpetsc.so: >> > >>>>>> undefined reference to >> .omp_offloading.img_start.cray_amdgcn-amd-amdhsa >> > >>>>>> [--no-allow-shlib-undefined] >> > >>>>>> ld.lld: error: >> > >>>>>> >> /gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib/libpetsc.so: >> > >>>>>> undefined reference to >> .omp_offloading.img_size.cray_amdgcn-amd-amdhsa >> > >>>>>> [--no-allow-shlib-undefined] >> > >>>>>> ld.lld: error: >> > >>>>>> >> /gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib/libpetsc.so: >> > >>>>>> undefined reference to >> .omp_offloading.img_cache.cray_amdgcn-amd-amdhsa >> > >>>>>> [--no-allow-shlib-undefined] >> > >>>>>> clang-11: error: linker command failed with exit code 1 (use -v >> to >> > >>>>>> see invocation) >> > >>>>>> gmake[4]: *** [<builtin>: ex19] Error 1 >> > >>>>>> gmake[3]: >> > >>>>>> >> [/gpfs/alpine/csc314/scratch/adams/petsc/lib/petsc/conf/rules:301: >> > >>>>>> ex3k.PETSc] Error 2 (ignored) >> > >>>>>> *******************Error detected during compile or >> > >>>>>> link!******************* >> > >>>>>> See http://www.mcs.anl.gov/petsc/documentation/faq.html >> > >>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials ex3k >> > >>>>>> >> > >>>>>> >> ********************************************************************************* >> > >>>>>> hipcc -fPIC -Wall -Wwrite-strings -Wno-strict-aliasing >> > >>>>>> -Wno-unknown-pragmas -fvisibility=hidden -g -O0 -std=c++14 >> > >>>>>> -I/gpfs/alpine/csc314/scratch/adams/petsc/include >> > >>>>>> >> -I/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/include >> > >>>>>> -I/sw/spock/spack-envs/views/rocm-4.1.0/include >> > >>>>>> -I/sw/spock/spack-envs/views/rocm-4.1.0/include >> -I/opt/cray/pe/libsci/ >> > >>>>>> 21.04.1.1/CRAY/9.0/x86_64/include >> > >>>>>> -I/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/include >> > >>>>>> -I/opt/cray/pe/pmi/6.0.10/include >> -I/opt/cray/pe/dsmml/0.1.4/dsmml//include >> > >>>>>> -I/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/include >> > >>>>>> -L/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib >> > >>>>>> -L/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib >> > >>>>>> -L/opt/cray/pe/mpich/8.1.4/gtl/lib -L/opt/cray/pe/pmi/6.0.10/lib >> > >>>>>> -L/opt/cray/pe/dsmml/0.1.4/dsmml//lib >> > >>>>>> -L/opt/cray/pe/cce/11.0.4/cce/x86_64/lib/pkgconfig/../ >> > >>>>>> -L/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64 >> > >>>>>> -Wl,--as-needed,-lsci_cray_mpi,--no-as-needed >> > >>>>>> -Wl,--as-needed,-lsci_cray,--no-as-needed >> > >>>>>> -Wl,--as-needed,-lmpi_cray,--no-as-needed >> > >>>>>> -Wl,--as-needed,-ldsmml,--no-as-needed >> > >>>>>> -Wl,--as-needed,-lpgas-shmem,--no-as-needed -lpmi -ldl >> -lmpi_gtl_hsa >> > >>>>>> -lxpmem -lquadmath -lmodules -lfi -lcraymath -lf -lu -lcsup >> > >>>>>> ex3k.kokkos.cxx >> > >>>>>> >> -Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib >> > >>>>>> >> -L/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib >> > >>>>>> >> -Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib >> > >>>>>> >> -L/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib >> > >>>>>> -Wl,-rpath,/sw/spock/spack-envs/views/rocm-4.1.0/lib >> > >>>>>> -L/sw/spock/spack-envs/views/rocm-4.1.0/lib >> > >>>>>> -Wl,-rpath,/opt/gcc/8.1.0/snos/lib64 -L/opt/gcc/8.1.0/snos/lib64 >> > >>>>>> -Wl,-rpath,/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib >> > >>>>>> -L/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib >> > >>>>>> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib >> > >>>>>> -L/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib >> > >>>>>> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/gtl/lib >> > >>>>>> -L/opt/cray/pe/mpich/8.1.4/gtl/lib >> -Wl,-rpath,/opt/cray/pe/pmi/6.0.10/lib >> > >>>>>> -L/opt/cray/pe/pmi/6.0.10/lib >> -Wl,-rpath,/opt/cray/pe/dsmml/0.1.4/dsmml/lib >> > >>>>>> -L/opt/cray/pe/dsmml/0.1.4/dsmml/lib >> > >>>>>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce/x86_64/lib >> > >>>>>> -L/opt/cray/pe/cce/11.0.4/cce/x86_64/lib >> > >>>>>> -Wl,-rpath,/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64 >> > >>>>>> -L/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64 >> > >>>>>> >> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux >> > >>>>>> >> -L/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux >> > >>>>>> -Wl,-rpath,/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0 >> > >>>>>> -L/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0 >> > >>>>>> >> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib >> > >>>>>> >> -L/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib >> > >>>>>> -lpetsc -lkokkoskernels -lkokkoscontainers -lkokkoscore >> -lhipsparse >> > >>>>>> -lhipblas -lrocsparse -lrocsolver -lrocblas -lamdhip64 >> -lhsa-runtime64 >> > >>>>>> -lstdc++ -ldl -lpmi -lsci_cray_mpi -lsci_cray -lmpifort_cray >> -lmpi_cray >> > >>>>>> -lmpi_gtl_hsa -lxpmem -ldsmml -lpgas-shmem -lquadmath >> -lcrayacc_amdgpu >> > >>>>>> -lopenacc -lmodules -lfi -lcraymath -lf -lu -lcsup -lgfortran >> -lpthread >> > >>>>>> -lgcc_eh -lm -lclang_rt.craypgo-x86_64 -lclang_rt.builtins-x86_64 >> > >>>>>> -lquadmath -lstdc++ -ldl -o ex3k >> > >>>>>> >> /usr/lib64/gcc/x86_64-suse-linux/7/../../../../x86_64-suse-linux/bin/ld: >> > >>>>>> >> /gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib/libpetsc.so: >> > >>>>>> undefined reference to >> `.omp_offloading.img_size.cray_amdgcn-amd-amdhsa' >> > >>>>>> >> /usr/lib64/gcc/x86_64-suse-linux/7/../../../../x86_64-suse-linux/bin/ld: >> > >>>>>> >> /gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib/libpetsc.so: >> > >>>>>> undefined reference to >> `.omp_offloading.img_start.cray_amdgcn-amd-amdhsa' >> > >>>>>> >> /usr/lib64/gcc/x86_64-suse-linux/7/../../../../x86_64-suse-linux/bin/ld: >> > >>>>>> >> /gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib/libpetsc.so: >> > >>>>>> undefined reference to >> `.omp_offloading.img_cache.cray_amdgcn-amd-amdhsa' >> > >>>>>> clang-12: error: linker command failed with exit code 1 (use -v >> to >> > >>>>>> see invocation) >> > >>>>>> gmake[4]: *** >> > >>>>>> [/gpfs/alpine/csc314/scratch/adams/petsc/lib/petsc/conf/test:26: >> ex3k] >> > >>>>>> Error 1 >> > >>>>>> gmake[3]: [makefile:102: runex3k_kokkos] Error 2 (ignored) >> > >>>>>> 1,25c1,4 >> > >>>>>> < atol=1e-50, rtol=1e-08, stol=1e-08, maxit=50, maxf=10000 >> > >>>>>> < Vec Object: Exact Solution 2 MPI processes >> > >>>>>> < type: mpikokkos >> > >>>>>> < Process [0] >> > >>>>>> < 0. >> > >>>>>> < 0.015625 >> > >>>>>> < 0.125 >> > >>>>>> < Process [1] >> > >>>>>> < 0.421875 >> > >>>>>> < 1. >> > >>>>>> < Vec Object: Forcing function 2 MPI processes >> > >>>>>> < type: mpikokkos >> > >>>>>> < Process [0] >> > >>>>>> < 1e-72 >> > >>>>>> < 1.50024 >> > >>>>>> < 3.01563 >> > >>>>>> < Process [1] >> > >>>>>> < 4.67798 >> > >>>>>> < 7. >> > >>>>>> < 0 SNES Function norm 5.414682427127e+00 >> > >>>>>> < 1 SNES Function norm 2.952582418265e-01 >> > >>>>>> < 2 SNES Function norm 4.502293658739e-04 >> > >>>>>> < 3 SNES Function norm 1.389665806646e-09 >> > >>>>>> < Number of SNES iterations = 3 >> > >>>>>> < Norm of error 1.49752e-10 Iterations 3 >> > >>>>>> --- >> > >>>>>> > slurmstepd: error: execve(): >> > >>>>>> >> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials/./ex3k: No such >> > >>>>>> file or directory >> > >>>>>> > slurmstepd: error: execve(): >> > >>>>>> >> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials/./ex3k: No such >> > >>>>>> file or directory >> > >>>>>> > srun: error: spock01: tasks 0-1: Exited with exit code 2 >> > >>>>>> > srun: launch/slurm: _step_signal: Terminating StepId=272979.0 >> > >>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials >> > >>>>>> Possible problem with ex3k running with kokkos-kernels, diffs >> above >> > >>>>>> ========================================= >> > >>>>>> *******************Error detected during compile or >> > >>>>>> link!******************* >> > >>>>>> See http://www.mcs.anl.gov/petsc/documentation/faq.html >> > >>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials ex5f >> > >>>>>> ********************************************************* >> > >>>>>> ftn -L/sw/spock/spack-envs/views/rocm-4.1.0/lib -lamdhip64 >> > >>>>>> -lhsa-runtime64 -fPIC -g -O0 -fPIC -g -O0 >> > >>>>>> -I/gpfs/alpine/csc314/scratch/adams/petsc/include >> > >>>>>> >> -I/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/include >> > >>>>>> -I/sw/spock/spack-envs/views/rocm-4.1.0/include ex5f.F90 >> > >>>>>> >> -Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib >> > >>>>>> >> -L/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib >> > >>>>>> >> -Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib >> > >>>>>> >> -L/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib >> > >>>>>> -Wl,-rpath,/sw/spock/spack-envs/views/rocm-4.1.0/lib >> > >>>>>> -L/sw/spock/spack-envs/views/rocm-4.1.0/lib >> > >>>>>> -Wl,-rpath,/opt/gcc/8.1.0/snos/lib64 -L/opt/gcc/8.1.0/snos/lib64 >> > >>>>>> -Wl,-rpath,/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib >> > >>>>>> -L/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib >> > >>>>>> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib >> > >>>>>> -L/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib >> > >>>>>> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/gtl/lib >> > >>>>>> -L/opt/cray/pe/mpich/8.1.4/gtl/lib >> -Wl,-rpath,/opt/cray/pe/pmi/6.0.10/lib >> > >>>>>> -L/opt/cray/pe/pmi/6.0.10/lib >> -Wl,-rpath,/opt/cray/pe/dsmml/0.1.4/dsmml/lib >> > >>>>>> -L/opt/cray/pe/dsmml/0.1.4/dsmml/lib >> > >>>>>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce/x86_64/lib >> > >>>>>> -L/opt/cray/pe/cce/11.0.4/cce/x86_64/lib >> > >>>>>> -Wl,-rpath,/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64 >> > >>>>>> -L/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64 >> > >>>>>> >> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux >> > >>>>>> >> -L/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux >> > >>>>>> -Wl,-rpath,/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0 >> > >>>>>> -L/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0 >> > >>>>>> >> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib >> > >>>>>> >> -L/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib >> > >>>>>> -lpetsc -lkokkoskernels -lkokkoscontainers -lkokkoscore >> -lhipsparse >> > >>>>>> -lhipblas -lrocsparse -lrocsolver -lrocblas -lamdhip64 >> -lhsa-runtime64 >> > >>>>>> -lstdc++ -ldl -lpmi -lsci_cray_mpi -lsci_cray -lmpifort_cray >> -lmpi_cray >> > >>>>>> -lmpi_gtl_hsa -lxpmem -ldsmml -lpgas-shmem -lquadmath >> -lcrayacc_amdgpu >> > >>>>>> -lopenacc -lmodules -lfi -lcraymath -lf -lu -lcsup -lgfortran >> -lpthread >> > >>>>>> -lgcc_eh -lm -lclang_rt.craypgo-x86_64 -lclang_rt.builtins-x86_64 >> > >>>>>> -lquadmath -lstdc++ -ldl -o ex5f >> > >>>>>> >> /opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/bin/ld: >> > >>>>>> warning: alignment 128 of symbol >> > >>>>>> `$host_init$$runtime_init_for_iso_c_binding$iso_c_binding_' in >> > >>>>>> /opt/cray/pe/cce/11.0.4/cce/x86_64/lib/libmodules.so is smaller >> than 256 in >> > >>>>>> /tmp/pe_30429/ex5f_1.o >> > >>>>>> >> /opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/bin/ld: >> > >>>>>> warning: alignment 64 of symbol `$data_init$iso_c_binding_' in >> > >>>>>> /opt/cray/pe/cce/11.0.4/cce/x86_64/lib/libmodules.so is smaller >> than 256 in >> > >>>>>> /tmp/pe_30429/ex5f_1.o >> > >>>>>> Fortran example src/snes/tutorials/ex5f run successfully with 1 >> MPI >> > >>>>>> process >> > >>>>>> Completed test examples >> > >>>>>> 10:49 jczhang/fix-cray-mpicxx-includes/main= >> > >>>>>> /gpfs/alpine/csc314/scratch/adams/petsc$ ll >> > >>>>>> >> > >>>>>> On Fri, Jul 16, 2021 at 11:28 PM Junchao Zhang < >> > >>>>>> junchao.zhang(a)gmail.com> wrote: >> > >>>>>> >> > >>>>>>> Mark, I configured with Fortran binding enabled with main + MR >> !4175 >> > >>>>>>> <https://gitlab.com/petsc/petsc/-/merge_requests/4175> >> > >>>>>>> >> > >>>>>>> '--with-mpiexec=srun', >> > >>>>>>> '--with-shared-libraries=1', >> > >>>>>>> '--with-cc=cc', >> > >>>>>>> '--with-cxx=CC', >> > >>>>>>> '--with-fc=ftn', >> > >>>>>>> '--with-fortran-bindings', >> > >>>>>>> '--with-hip', >> > >>>>>>> '--with-hipc=hipcc', >> > >>>>>>> '--with-debugging', >> > >>>>>>> '--CPPFLAGS=-I${ROCM_PATH}/include', >> > >>>>>>> '--CXXPPFLAGS=-I${ROCM_PATH}/include', >> > >>>>>>> '--CC_LINKER_FLAGS=-L${ROCM_PATH}/lib -lamdhip64 >> -lhsa-runtime64', >> > >>>>>>> '--CXX_LINKER_FLAGS=-L${ROCM_PATH}/lib -lamdhip64 >> -lhsa-runtime64', >> > >>>>>>> '--FC_LINKER_FLAGS=-L${ROCM_PATH}/lib -lamdhip64 >> -lhsa-runtime64', >> > >>>>>>> '--COPTFLAGS=-g -O0', >> > >>>>>>> '--CXXOPTFLAGS=-g -O0', >> > >>>>>>> '--FOPTFLAGS=-g -O0', >> > >>>>>>> '--download-kokkos', >> > >>>>>>> '--download-kokkos-kernels', >> > >>>>>>> '--download-kokkos-commit=3.4.01', >> > >>>>>>> '--download-kokkos-kernels-commit=3.4.01', >> > >>>>>>> '--with-kokkos-hip-arch=VEGA908', >> > >>>>>>> >> > >>>>>>> and 'make check' ran smoothly on a compute node >> > >>>>>>> >> > >>>>>>> $ make check >> > >>>>>>> Running check examples to verify correct installation >> > >>>>>>> Using PETSC_DIR=/ccs/home/jczhang/petsc and >> > >>>>>>> PETSC_ARCH=arch-spock-cray-kokkos-dbg >> > >>>>>>> C/C++ example src/snes/tutorials/ex19 run successfully with 1 >> MPI >> > >>>>>>> process >> > >>>>>>> C/C++ example src/snes/tutorials/ex19 run successfully with 2 >> MPI >> > >>>>>>> processes >> > >>>>>>> C/C++ example src/snes/tutorials/ex3k run successfully with >> > >>>>>>> kokkos-kernels >> > >>>>>>> Fortran example src/snes/tutorials/ex5f run successfully with 1 >> MPI >> > >>>>>>> process >> > >>>>>>> >> > >>>>>>> Completed test examples >> > >>>>>>> >> > >>>>>>> >> > >>>>>>> --Junchao Zhang >> > >>>>>>> >> > >>>>>>> >> > >>>>>>> On Fri, Jul 16, 2021 at 6:04 PM Mark Adams <mfadams(a)lbl.gov> >> wrote: >> > >>>>>>> >> > >>>>>>>> And I find that this error, on non-Kokkos C tests, is fixed by >> > >>>>>>>> turning the fortran bindings off: >> > >>>>>>>> >> > >>>>>>>> ld.lld: error: >> > >>>>>>>> >> /gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/lib/libpetsc.so: >> > >>>>>>>> undefined reference to >> .omp_offloading.img_start.cray_amdgcn-amd-amdhsa >> > >>>>>>>> [--no-allow-shlib-undefined] >> > >>>>>>>> ld.lld: error: >> > >>>>>>>> >> /gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/lib/libpetsc.so: >> > >>>>>>>> undefined reference to >> .omp_offloading.img_size.cray_amdgcn-amd-amdhsa >> > >>>>>>>> [--no-allow-shlib-undefined] >> > >>>>>>>> ld.lld: error: >> > >>>>>>>> >> /gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/lib/libpetsc.so: >> > >>>>>>>> undefined reference to >> .omp_offloading.img_cache.cray_amdgcn-amd-amdhsa >> > >>>>>>>> [--no-allow-shlib-undefined] >> > >>>>>>>> >> > >>>>>>>> On Fri, Jul 16, 2021 at 3:53 PM Mark Adams <mfadams(a)lbl.gov> >> wrote: >> > >>>>>>>> >> > >>>>>>>>> Not complex. THis has some overlap with my problem w/o Kokkos. >> > >>>>>>>>> >> > >>>>>>>>> On Fri, Jul 16, 2021 at 12:54 PM Junchao Zhang < >> > >>>>>>>>> junchao.zhang(a)gmail.com> wrote: >> > >>>>>>>>> >> > >>>>>>>>>> Do you use complex? post your configure.log. >> > >>>>>>>>>> >> > >>>>>>>>>> --Junchao Zhang >> > >>>>>>>>>> >> > >>>>>>>>>> >> > >>>>>>>>>> On Fri, Jul 16, 2021 at 9:47 AM Mark Adams <mfadams(a)lbl.gov> >> > >>>>>>>>>> wrote: >> > >>>>>>>>>> >> > >>>>>>>>>>> The simple Kokkos example is failing for me on Spock. >> > >>>>>>>>>>> Any ideas? >> > >>>>>>>>>>> Thanks, >> > >>>>>>>>>>> >> > >>>>>>>>>>> 10:44 main *= >> > >>>>>>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials$ >> make >> > >>>>>>>>>>> PETSC_DIR=/gpfs/alpine/csc314/scratch/adams/petsc >> > >>>>>>>>>>> PETSC_ARCH=arch-spock-opt-cray-kokkos ex3k >> > >>>>>>>>>>> MPICH_CXX="hipcc" OMPI_CXX="hipcc" CC >> > >>>>>>>>>>> -L/sw/spock/spack-envs/views/rocm-4.1.0/lib -lhsa-runtime64 >> > >>>>>>>>>>> -L/sw/spock/spack-envs/views/rocm-4.1.0/lib -lamdhip64 >> -lhsa-runtime64 >> > >>>>>>>>>>> -Wall -Wwrite-strings -Wno-strict-aliasing >> -Wno-unknown-pragmas >> > >>>>>>>>>>> -fstack-protector -fvisibility=hidden -g -O2 -fPIC -Wall >> -Wwrite-strings >> > >>>>>>>>>>> -Wno-strict-aliasing -Wno-unknown-pragmas >> -fvisibility=hidden -g -O3 >> > >>>>>>>>>>> -std=c++14 >> -I/gpfs/alpine/csc314/scratch/adams/petsc/include >> > >>>>>>>>>>> >> -I/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-opt-cray-kokkos/include >> > >>>>>>>>>>> >> -I/gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include >> > >>>>>>>>>>> -I/sw/spock/spack-envs/views/rocm-4.1.0/include >> > >>>>>>>>>>> -I/sw/spock/spack-envs/views/rocm-4.1.0/include >> ex3k.kokkos.cxx >> > >>>>>>>>>>> >> -Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-opt-cray-kokkos/lib >> > >>>>>>>>>>> >> -L/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-opt-cray-kokkos/lib >> > >>>>>>>>>>> >> -Wl,-rpath,/gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/lib >> > >>>>>>>>>>> >> -L/gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/lib >> > >>>>>>>>>>> -Wl,-rpath,/sw/spock/spack-envs/views/rocm-4.1.0/lib >> > >>>>>>>>>>> -L/sw/spock/spack-envs/views/rocm-4.1.0/lib >> > >>>>>>>>>>> -Wl,-rpath,/opt/gcc/8.1.0/snos/lib64 >> -L/opt/gcc/8.1.0/snos/lib64 >> > >>>>>>>>>>> -Wl,-rpath,/opt/cray/pe/libsci/ >> 21.04.1.1/CRAY/9.0/x86_64/lib >> > >>>>>>>>>>> -L/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib >> > >>>>>>>>>>> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib >> > >>>>>>>>>>> -L/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib >> > >>>>>>>>>>> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/gtl/lib >> > >>>>>>>>>>> -L/opt/cray/pe/mpich/8.1.4/gtl/lib >> -Wl,-rpath,/opt/cray/pe/pmi/6.0.10/lib >> > >>>>>>>>>>> -L/opt/cray/pe/pmi/6.0.10/lib >> -Wl,-rpath,/opt/cray/pe/dsmml/0.1.4/dsmml/lib >> > >>>>>>>>>>> -L/opt/cray/pe/dsmml/0.1.4/dsmml/lib >> > >>>>>>>>>>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce/x86_64/lib >> > >>>>>>>>>>> -L/opt/cray/pe/cce/11.0.4/cce/x86_64/lib >> > >>>>>>>>>>> >> -Wl,-rpath,/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64 >> > >>>>>>>>>>> -L/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64 >> > >>>>>>>>>>> >> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux >> > >>>>>>>>>>> >> -L/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux >> > >>>>>>>>>>> >> -Wl,-rpath,/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0 >> > >>>>>>>>>>> -L/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0 >> > >>>>>>>>>>> >> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib >> > >>>>>>>>>>> >> -L/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib >> > >>>>>>>>>>> -lpetsc -lkokkoskernels -lkokkoscontainers -lkokkoscore >> -lhipsparse >> > >>>>>>>>>>> -lhipblas -lrocsparse -lrocsolver -lrocblas -lamdhip64 >> -lhsa-runtime64 >> > >>>>>>>>>>> -lstdc++ -ldl -lpmi -lsci_cray_mpi -lsci_cray >> -lmpifort_cray -lmpi_cray >> > >>>>>>>>>>> -lmpi_gtl_hsa -lxpmem -ldsmml -lpgas-shmem -lquadmath >> -lcrayacc_amdgpu >> > >>>>>>>>>>> -lopenacc -lmodules -lfi -lcraymath -lf -lu -lcsup >> -lgfortran -lpthread >> > >>>>>>>>>>> -lgcc_eh -lm -lclang_rt.craypgo-x86_64 >> -lclang_rt.builtins-x86_64 >> > >>>>>>>>>>> -lquadmath -lstdc++ -ldl -o ex3k >> > >>>>>>>>>>> In file included from ex3k.kokkos.cxx:3: >> > >>>>>>>>>>> In file included from >> > >>>>>>>>>>> >> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscdmda_kokkos.hpp:4: >> > >>>>>>>>>>> In file included from >> > >>>>>>>>>>> >> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec_kokkos.hpp:14: >> > >>>>>>>>>>> In file included from >> > >>>>>>>>>>> >> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec.h:9: >> > >>>>>>>>>>> In file included from >> > >>>>>>>>>>> >> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsys.h:42: >> > >>>>>>>>>>> In file included from >> > >>>>>>>>>>> >> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsystypes.h:255: >> > >>>>>>>>>>> In file included from >> > >>>>>>>>>>> >> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Complex.hpp:47: >> > >>>>>>>>>>> In file included from >> > >>>>>>>>>>> >> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Atomic.hpp:212: >> > >>>>>>>>>>> >> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/impl/Kokkos_Atomic_Exchange.hpp:259:11: >> > >>>>>>>>>>> error: no member named 'lock_address_host_space' in >> namespace >> > >>>>>>>>>>> 'Kokkos::Impl::Kokkos::Impl'; did you mean simply >> 'lock_address_host_space'? >> > >>>>>>>>>>> while (!Impl::lock_address_host_space((void*)dest)) >> > >>>>>>>>>>> ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~ >> > >>>>>>>>>>> lock_address_host_space >> > >>>>>>>>>>> >> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_HostSpace.hpp:83:6: >> > >>>>>>>>>>> note: 'lock_address_host_space' declared here >> > >>>>>>>>>>> bool lock_address_host_space(void* ptr); >> > >>>>>>>>>>> ^ >> > >>>>>>>>>>> In file included from ex3k.kokkos.cxx:3: >> > >>>>>>>>>>> In file included from >> > >>>>>>>>>>> >> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscdmda_kokkos.hpp:4: >> > >>>>>>>>>>> In file included from >> > >>>>>>>>>>> >> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec_kokkos.hpp:14: >> > >>>>>>>>>>> In file included from >> > >>>>>>>>>>> >> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec.h:9: >> > >>>>>>>>>>> In file included from >> > >>>>>>>>>>> >> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsys.h:42: >> > >>>>>>>>>>> In file included from >> > >>>>>>>>>>> >> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsystypes.h:255: >> > >>>>>>>>>>> In file included from >> > >>>>>>>>>>> >> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Complex.hpp:47: >> > >>>>>>>>>>> In file included from >> > >>>>>>>>>>> >> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Atomic.hpp:212: >> > >>>>>>>>>>> >> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/impl/Kokkos_Atomic_Exchange.hpp:261:3: >> > >>>>>>>>>>> error: no member named 'memory_fence' in namespace >> 'Kokkos::Impl::Kokkos'; >> > >>>>>>>>>>> did you mean simply 'memory_fence'? >> > >>>>>>>>>>> Kokkos::memory_fence(); >> > >>>>>>>>>>> ^~~~~~~~~~~~~~~~~~~~ >> > >>>>>>>>>>> memory_fence >> > >>>>>>>>>>> >> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/impl/Kokkos_Memory_Fence.hpp:53:6: >> > >>>>>>>>>>> note: 'memory_fence' declared here >> > >>>>>>>>>>> void memory_fence() { >> > >>>>>>>>>>> ^ >> > >>>>>>>>>>> In file included from ex3k.kokkos.cxx:3: >> > >>>>>>>>>>> In file included from >> > >>>>>>>>>>> >> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscdmda_kokkos.hpp:4: >> > >>>>>>>>>>> In file included from >> > >>>>>>>>>>> >> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec_kokkos.hpp:14: >> > >>>>>>>>>>> In file included from >> > >>>>>>>>>>> >> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec.h:9: >> > >>>>>>>>>>> In file included from >> > >>>>>>>>>>> >> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsys.h:42: >> > >>>>>>>>>>> In file included from >> > >>>>>>>>>>> >> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsystypes.h:255: >> > >>>>>>>>>>> In file included from >> > >>>>>>>>>>> >> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Complex.hpp:47: >> > >>>>>>>>>>> In file included from >> > >>>>>>>>>>> >> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Atomic.hpp:212: >> > >>>>>>>>>>> >> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/impl/Kokkos_Atomic_Exchange.hpp:277:3: >> > >>>>>>>>>>> error: no member named 'memory_fence' in namespace >> 'Kokkos::Impl::Kokkos'; >> > >>>>>>>>>>> did you mean simply 'memory_fence'? >> > >>>>>>>>>>> Kokkos::memory_fence(); >> > >>>>>>>>>>> ^~~~~~~~~~~~~~~~~~~~ >> > >>>>>>>>>>> memory_fence >> > >>>>>>>>>>> >> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/impl/Kokkos_Memory_Fence.hpp:53:6: >> > >>>>>>>>>>> note: 'memory_fence' declared here >> > >>>>>>>>>>> void memory_fence() { >> > >>>>>>>>>>> ^ >> > >>>>>>>>>>> >> > >>>>>>>>>> >> >> >> >> >> -- >> Scott Kruger >> Tech-X Corporation kruger(a)txcorp.com >> 5621 Arapahoe Ave, Suite A Phone: (720) 466-3196 >> Boulder, CO 80303 Fax: (303) 448-7756 >> >

1 0

Re: [petsc-dev] Kokkos make error on Spock
by Mark Adams 20 Jul '21

20 Jul '21

On Mon, Jul 19, 2021 at 6:41 PM Scott Kruger <kruger(a)txcorp.com> wrote: > > Mark, > > On tulip, things with fortran went wonky when a `-fast` flag snuck into > the flags (snuck in being me copying a previous configure file and not > questioning the flags). The reason is that for clang, `-fast` implies > link time optimization (lto) for C/C++ code, but flang doesn't support > lto so things got weird. I suspect that gfortran does not either (but > perhaps the real question is why not use flang?) > It does work with GNU. I am doing this for an application and I want to avoid dictating the program env. I'm (re)trying with -O0 just to check. > > I look at configure and I don't see anything in your flags that would > trigger lto, but as Junchao says, it might be picking something up from > Kokkos, so perhaps this is the issue. > > The configure files that I sent did not have Kokkos. They were stripped down. I have asked ORNL, but don't expect help. The person (Matt) that is helping me for other things is very helpful so perhaps he can come up with some ideas. Otherwise I will tell the app that they need to use GNU for now. Thanks, Mark > Scott > > > On 2021-07-19 07:19, Mark Adams did write: > > Thanks, but this happens w/o Kokkos. > > I've stripped this down and attached good/bad logs without/with Fortran > > bindings. > > Hope this helps, > > Thanks again, > > Mark > > > > On Sun, Jul 18, 2021 at 12:00 PM Stefano Zampini < > stefano.zampini(a)gmail.com> > > wrote: > > > > > This is probably kokkos pulling in the dependencies where compiling > kokkos > > > source within PETSc. > > > > > > Il Dom 18 Lug 2021, 16:29 Mark Adams <mfadams(a)lbl.gov> ha scritto: > > > > > >> Whoops, this error was just from not telling gfortran to allow long > lines. > > >> > > >> Anway, I did find that the when fortran bindings are enabled this OMP > > >> thing gets into the library. No idea how. I'll askm ORNL. > > >> > > >> 10:26 jczhang/fix-cray-mpicxx-includes/main= > > >> /gpfs/alpine/csc314/scratch/adams/petsc2$ nm > > >> > /gpfs/alpine/csc314/scratch/adams/petsc2/arch-spock-opt-cray-kokkos/lib/libpetsc.so > > >> |g offload > > >> U .omp_offloading.img_cache.cray_amdgcn-amd-amdhsa > > >> U .omp_offloading.img_size.cray_amdgcn-amd-amdhsa > > >> U .omp_offloading.img_start.cray_amdgcn-amd-amdhsa > > >> 0000000001d457b0 T vecgetoffloadmask_ > > >> > > >> > > >> On Sun, Jul 18, 2021 at 8:43 AM Mark Adams <mfadams(a)lbl.gov> wrote: > > >> > > >>> Ah, your test was not on Spock ... I have something working but this > is > > >>> strange. > > >>> > > >>> I switched to the GNU ProgEnv. and it passed the Fortran test in > > >>> 'check', but this failed: > > >>> > > >>> > > >>> > > >>> > > >>> > > >>> > > >>> > > >>> > > >>> > > >>> *08:26 2 jczhang/fix-cray-mpicxx-includes/main= > > >>> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials$ make > > >>> > PETSC_DIR=/gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-dbg-gnu > > >>> PETSC_ARCH="" ex40f90ftn -L/sw/spock/spack-envs/views/rocm-4.1.0/lib > > >>> -lamdhip64 -lhsa-runtime64 -fPIC -g -O0 -fPIC -g -O0 > > >>> > -I/gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-dbg-gnu/include > > >>> -I/sw/spock/spack-envs/views/rocm-4.1.0/include ex40f90.F90 > > >>> > -Wl,-rpath,/gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-dbg-gnu/lib > > >>> > -L/gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-dbg-gnu/lib > > >>> -Wl,-rpath,/sw/spock/spack-envs/views/rocm-4.1.0/lib > > >>> -L/sw/spock/spack-envs/views/rocm-4.1.0/lib > > >>> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/ofi/gnu/9.1/lib > > >>> -L/opt/cray/pe/mpich/8.1.4/ofi/gnu/9.1/lib > > >>> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/gtl/lib > > >>> -L/opt/cray/pe/mpich/8.1.4/gtl/lib > > >>> -Wl,-rpath,/opt/cray/pe/libsci/21.04.1.1/GNU/9.1/x86_64/lib > > >>> <http://21.04.1.1/GNU/9.1/x86_64/lib> > > >>> -L/opt/cray/pe/libsci/21.04.1.1/GNU/9.1/x86_64/lib > > >>> <http://21.04.1.1/GNU/9.1/x86_64/lib> > > >>> -Wl,-rpath,/opt/cray/pe/pmi/6.0.10/lib -L/opt/cray/pe/pmi/6.0.10/lib > > >>> -Wl,-rpath,/opt/cray/pe/dsmml/0.1.4/dsmml/lib > > >>> -L/opt/cray/pe/dsmml/0.1.4/dsmml/lib > > >>> -Wl,-rpath,/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64 > > >>> -L/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64 > > >>> -Wl,-rpath,/opt/gcc/10.2.0/snos/lib/gcc/x86_64-suse-linux/10.2.0 > > >>> -L/opt/gcc/10.2.0/snos/lib/gcc/x86_64-suse-linux/10.2.0 > > >>> -Wl,-rpath,/opt/gcc/10.2.0/snos/lib64 -L/opt/gcc/10.2.0/snos/lib64 > > >>> -Wl,-rpath,/sw/spock/spack-envs/views/rocm-4.1.0/llvm/lib > > >>> -L/sw/spock/spack-envs/views/rocm-4.1.0/llvm/lib > > >>> -Wl,-rpath,/opt/gcc/10.2.0/snos/lib -L/opt/gcc/10.2.0/snos/lib > -lpetsc > > >>> -lhipsparse -lhipblas -lrocsparse -lrocsolver -lrocblas -lamdhip64 > > >>> -lhsa-runtime64 -lstdc++ -ldl -lpmi -lmpi_gtl_hsa -lxpmem -lgfortran > -lm > > >>> -lsci_gnu_82_mpi -lsci_gnu_82 -lmpifort_gnu_91 -lmpi_gnu_91 -ldsmml > > >>> -lgfortran -lquadmath -lpthread -lm -lgcc_s -lquadmath -lstdc++ -ldl > -o > > >>> ex40f90ex40f90.F90:35:132: 35 | call > > >>> > DMDACreate2d(PETSC_COMM_WORLD,DM_BOUNDARY_NONE,DM_BOUNDARY_NONE,DMDA_STENCIL_BOX,ten,ten,PETSC_DECIDE,PETSC_DECIDE,two,one, > > >>> & | > > >>> > 1Error: > > >>> Line truncated at (1) [-Werror=line-truncation]ex40f90.F90:35:132:* > > >>> * ....* > > >>> > > >>> But ex5f (the Fortran check test) does work manually also (sanity > check). > > >>> > > >>> Oddly I seem to have the cray compilers: > > >>> > > >>> 08:34 jczhang/fix-cray-mpicxx-includes/main= > > >>> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials$ which ftn > > >>> /opt/cray/pe/craype/2.7.6/bin/ftn > > >>> > > >>> yet it is GNU: > > >>> > > >>> 08:34 jczhang/fix-cray-mpicxx-includes/main= > > >>> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials$ ftn > --version > > >>> GNU Fortran (GCC) 10.2.0 20200723 (Cray Inc.) > > >>> Copyright (C) 2020 Free Software Foundation, Inc. > > >>> This is free software; see the source for copying conditions. There > is > > >>> NO > > >>> warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR > > >>> PURPOSE. > > >>> > > >>> The documentation says ftn should be here: > > >>> > > >>> 08:34 jczhang/fix-cray-mpicxx-includes/main= > > >>> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials$ > > >>> $GCC_PATH/bin/gfortran --version > > >>> GNU Fortran (GCC) 10.2.0 20200723 (Cray Inc.) > > >>> Copyright (C) 2020 Free Software Foundation, Inc. > > >>> This is free software; see the source for copying conditions. There > is > > >>> NO > > >>> warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR > > >>> PURPOSE. > > >>> > > >>> Same, but the path is not the same: > > >>> > > >>> 08:39 jczhang/fix-cray-mpicxx-includes/main= > > >>> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials$ echo > $GCC_PATH > > >>> /opt/gcc/10.2.0 > > >>> > > >>> Anyway, thanks for your help, > > >>> Mark > > >>> > > >>> On Sat, Jul 17, 2021 at 11:26 PM Junchao Zhang < > junchao.zhang(a)gmail.com> > > >>> wrote: > > >>> > > >>>> Mark, I can reproduce this error with PrgEnv-cray, i.e., using the > > >>>> Cray compiler (clang-11). Previously I used PrgEnv-gnu, which did > not have > > >>>> this error. > > >>>> Probably it is a problem of Spock. But I am not sure. > > >>>> > > >>>> --Junchao Zhang > > >>>> > > >>>> > > >>>> On Sat, Jul 17, 2021 at 10:17 AM Mark Adams <mfadams(a)lbl.gov> > wrote: > > >>>> > > >>>>> And I can run a fortran test, with warnings, but C tests fail: > > >>>>> > > >>>>> 11:15 jczhang/fix-cray-mpicxx-includes/main= > > >>>>> /gpfs/alpine/csc314/scratch/adams/petsc$ make > > >>>>> PETSC_DIR=/gpfs/alpine/csc314/scratch/adams/petsc > > >>>>> PETSC_ARCH=arch-spock-opt-cray-kokkos -f gmakefile test > > >>>>> search='ts_utils_dmplexlandau_tutorials-ex1f90_0' > > >>>>> Using MAKEFLAGS: -- search=ts_utils_dmplexlandau_tutorials-ex1f90_0 > > >>>>> PETSC_ARCH=arch-spock-opt-cray-kokkos > > >>>>> PETSC_DIR=/gpfs/alpine/csc314/scratch/adams/petsc > > >>>>> FC > > >>>>> > arch-spock-opt-cray-kokkos/tests/ts/utils/dmplexlandau/tutorials/ex1f90.o > > >>>>> FLINKER > > >>>>> > arch-spock-opt-cray-kokkos/tests/ts/utils/dmplexlandau/tutorials/ex1f90 > > >>>>> /opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/bin/ld: > > >>>>> warning: alignment 128 of symbol > > >>>>> `$host_init$$runtime_init_for_iso_c_binding$iso_c_binding_' in > > >>>>> /opt/cray/pe/cce/11.0.4/cce/x86_64/lib/libmodules.so is smaller > than 256 in > > >>>>> > arch-spock-opt-cray-kokkos/tests/ts/utils/dmplexlandau/tutorials/ex1f90.o > > >>>>> /opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/bin/ld: > > >>>>> warning: alignment 64 of symbol `$data_init$iso_c_binding_' in > > >>>>> /opt/cray/pe/cce/11.0.4/cce/x86_64/lib/libmodules.so is smaller > than 256 in > > >>>>> > arch-spock-opt-cray-kokkos/tests/ts/utils/dmplexlandau/tutorials/ex1f90.o > > >>>>> TEST > > >>>>> > arch-spock-opt-cray-kokkos/tests/counts/ts_utils_dmplexlandau_tutorials-ex1f90_0.counts > > >>>>> ok ts_utils_dmplexlandau_tutorials-ex1f90_0 > > >>>>> ok diff-ts_utils_dmplexlandau_tutorials-ex1f90_0 > > >>>>> > > >>>>> > > >>>>> On Sat, Jul 17, 2021 at 10:53 AM Mark Adams <mfadams(a)lbl.gov> > wrote: > > >>>>> > > >>>>>> HUmm, I can not reproduce this. > > >>>>>> > > >>>>>> 10:49 jczhang/fix-cray-mpicxx-includes/main= > > >>>>>> /gpfs/alpine/csc314/scratch/adams/petsc$ make > > >>>>>> PETSC_DIR=/gpfs/alpine/csc314/scratch/adams/petsc > > >>>>>> PETSC_ARCH=arch-spock-dbg-kokkos check > > >>>>>> Running check examples to verify correct installation > > >>>>>> Using PETSC_DIR=/gpfs/alpine/csc314/scratch/adams/petsc and > > >>>>>> PETSC_ARCH=arch-spock-dbg-kokkos > > >>>>>> gmake[3]: > > >>>>>> [/gpfs/alpine/csc314/scratch/adams/petsc/lib/petsc/conf/rules:301: > > >>>>>> ex19.PETSc] Error 2 (ignored) > > >>>>>> *******************Error detected during compile or > > >>>>>> link!******************* > > >>>>>> See http://www.mcs.anl.gov/petsc/documentation/faq.html > > >>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials ex19 > > >>>>>> > > >>>>>> > ********************************************************************************* > > >>>>>> cc -L/sw/spock/spack-envs/views/rocm-4.1.0/lib -lamdhip64 > > >>>>>> -lhsa-runtime64 -fPIC -Wall -Wwrite-strings -Wno-strict-aliasing > > >>>>>> -Wno-unknown-pragmas -fstack-protector -Qunused-arguments > > >>>>>> -fvisibility=hidden -g -O0 -fPIC -Wall -Wwrite-strings > > >>>>>> -Wno-strict-aliasing -Wno-unknown-pragmas -fstack-protector > > >>>>>> -Qunused-arguments -fvisibility=hidden -g -O0 > > >>>>>> -I/sw/spock/spack-envs/views/rocm-4.1.0/include > > >>>>>> -I/gpfs/alpine/csc314/scratch/adams/petsc/include > > >>>>>> > -I/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/include > > >>>>>> -I/sw/spock/spack-envs/views/rocm-4.1.0/include > > >>>>>> -I/sw/spock/spack-envs/views/rocm-4.1.0/include ex19.c > > >>>>>> > -Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib > > >>>>>> > -L/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib > > >>>>>> > -Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib > > >>>>>> > -L/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib > > >>>>>> -Wl,-rpath,/sw/spock/spack-envs/views/rocm-4.1.0/lib > > >>>>>> -L/sw/spock/spack-envs/views/rocm-4.1.0/lib > > >>>>>> -Wl,-rpath,/opt/gcc/8.1.0/snos/lib64 -L/opt/gcc/8.1.0/snos/lib64 > > >>>>>> -Wl,-rpath,/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib > > >>>>>> -L/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib > > >>>>>> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib > > >>>>>> -L/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib > > >>>>>> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/gtl/lib > > >>>>>> -L/opt/cray/pe/mpich/8.1.4/gtl/lib > -Wl,-rpath,/opt/cray/pe/pmi/6.0.10/lib > > >>>>>> -L/opt/cray/pe/pmi/6.0.10/lib > -Wl,-rpath,/opt/cray/pe/dsmml/0.1.4/dsmml/lib > > >>>>>> -L/opt/cray/pe/dsmml/0.1.4/dsmml/lib > > >>>>>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce/x86_64/lib > > >>>>>> -L/opt/cray/pe/cce/11.0.4/cce/x86_64/lib > > >>>>>> -Wl,-rpath,/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64 > > >>>>>> -L/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64 > > >>>>>> > -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux > > >>>>>> > -L/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux > > >>>>>> -Wl,-rpath,/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0 > > >>>>>> -L/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0 > > >>>>>> > -Wl,-rpath,/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib > > >>>>>> > -L/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib > > >>>>>> -lpetsc -lkokkoskernels -lkokkoscontainers -lkokkoscore > -lhipsparse > > >>>>>> -lhipblas -lrocsparse -lrocsolver -lrocblas -lamdhip64 > -lhsa-runtime64 > > >>>>>> -lstdc++ -ldl -lpmi -lsci_cray_mpi -lsci_cray -lmpifort_cray > -lmpi_cray > > >>>>>> -lmpi_gtl_hsa -lxpmem -ldsmml -lpgas-shmem -lquadmath > -lcrayacc_amdgpu > > >>>>>> -lopenacc -lmodules -lfi -lcraymath -lf -lu -lcsup -lgfortran > -lpthread > > >>>>>> -lgcc_eh -lm -lclang_rt.craypgo-x86_64 -lclang_rt.builtins-x86_64 > > >>>>>> -lquadmath -lstdc++ -ldl -o ex19 > > >>>>>> ld.lld: error: > > >>>>>> > /gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib/libpetsc.so: > > >>>>>> undefined reference to > .omp_offloading.img_start.cray_amdgcn-amd-amdhsa > > >>>>>> [--no-allow-shlib-undefined] > > >>>>>> ld.lld: error: > > >>>>>> > /gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib/libpetsc.so: > > >>>>>> undefined reference to > .omp_offloading.img_size.cray_amdgcn-amd-amdhsa > > >>>>>> [--no-allow-shlib-undefined] > > >>>>>> ld.lld: error: > > >>>>>> > /gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib/libpetsc.so: > > >>>>>> undefined reference to > .omp_offloading.img_cache.cray_amdgcn-amd-amdhsa > > >>>>>> [--no-allow-shlib-undefined] > > >>>>>> clang-11: error: linker command failed with exit code 1 (use -v to > > >>>>>> see invocation) > > >>>>>> gmake[4]: *** [<builtin>: ex19] Error 1 > > >>>>>> gmake[3]: > > >>>>>> [/gpfs/alpine/csc314/scratch/adams/petsc/lib/petsc/conf/rules:301: > > >>>>>> ex3k.PETSc] Error 2 (ignored) > > >>>>>> *******************Error detected during compile or > > >>>>>> link!******************* > > >>>>>> See http://www.mcs.anl.gov/petsc/documentation/faq.html > > >>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials ex3k > > >>>>>> > > >>>>>> > ********************************************************************************* > > >>>>>> hipcc -fPIC -Wall -Wwrite-strings -Wno-strict-aliasing > > >>>>>> -Wno-unknown-pragmas -fvisibility=hidden -g -O0 -std=c++14 > > >>>>>> -I/gpfs/alpine/csc314/scratch/adams/petsc/include > > >>>>>> > -I/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/include > > >>>>>> -I/sw/spock/spack-envs/views/rocm-4.1.0/include > > >>>>>> -I/sw/spock/spack-envs/views/rocm-4.1.0/include > -I/opt/cray/pe/libsci/ > > >>>>>> 21.04.1.1/CRAY/9.0/x86_64/include > > >>>>>> -I/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/include > > >>>>>> -I/opt/cray/pe/pmi/6.0.10/include > -I/opt/cray/pe/dsmml/0.1.4/dsmml//include > > >>>>>> -I/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/include > > >>>>>> -L/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib > > >>>>>> -L/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib > > >>>>>> -L/opt/cray/pe/mpich/8.1.4/gtl/lib -L/opt/cray/pe/pmi/6.0.10/lib > > >>>>>> -L/opt/cray/pe/dsmml/0.1.4/dsmml//lib > > >>>>>> -L/opt/cray/pe/cce/11.0.4/cce/x86_64/lib/pkgconfig/../ > > >>>>>> -L/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64 > > >>>>>> -Wl,--as-needed,-lsci_cray_mpi,--no-as-needed > > >>>>>> -Wl,--as-needed,-lsci_cray,--no-as-needed > > >>>>>> -Wl,--as-needed,-lmpi_cray,--no-as-needed > > >>>>>> -Wl,--as-needed,-ldsmml,--no-as-needed > > >>>>>> -Wl,--as-needed,-lpgas-shmem,--no-as-needed -lpmi -ldl > -lmpi_gtl_hsa > > >>>>>> -lxpmem -lquadmath -lmodules -lfi -lcraymath -lf -lu -lcsup > > >>>>>> ex3k.kokkos.cxx > > >>>>>> > -Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib > > >>>>>> > -L/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib > > >>>>>> > -Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib > > >>>>>> > -L/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib > > >>>>>> -Wl,-rpath,/sw/spock/spack-envs/views/rocm-4.1.0/lib > > >>>>>> -L/sw/spock/spack-envs/views/rocm-4.1.0/lib > > >>>>>> -Wl,-rpath,/opt/gcc/8.1.0/snos/lib64 -L/opt/gcc/8.1.0/snos/lib64 > > >>>>>> -Wl,-rpath,/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib > > >>>>>> -L/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib > > >>>>>> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib > > >>>>>> -L/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib > > >>>>>> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/gtl/lib > > >>>>>> -L/opt/cray/pe/mpich/8.1.4/gtl/lib > -Wl,-rpath,/opt/cray/pe/pmi/6.0.10/lib > > >>>>>> -L/opt/cray/pe/pmi/6.0.10/lib > -Wl,-rpath,/opt/cray/pe/dsmml/0.1.4/dsmml/lib > > >>>>>> -L/opt/cray/pe/dsmml/0.1.4/dsmml/lib > > >>>>>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce/x86_64/lib > > >>>>>> -L/opt/cray/pe/cce/11.0.4/cce/x86_64/lib > > >>>>>> -Wl,-rpath,/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64 > > >>>>>> -L/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64 > > >>>>>> > -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux > > >>>>>> > -L/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux > > >>>>>> -Wl,-rpath,/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0 > > >>>>>> -L/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0 > > >>>>>> > -Wl,-rpath,/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib > > >>>>>> > -L/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib > > >>>>>> -lpetsc -lkokkoskernels -lkokkoscontainers -lkokkoscore > -lhipsparse > > >>>>>> -lhipblas -lrocsparse -lrocsolver -lrocblas -lamdhip64 > -lhsa-runtime64 > > >>>>>> -lstdc++ -ldl -lpmi -lsci_cray_mpi -lsci_cray -lmpifort_cray > -lmpi_cray > > >>>>>> -lmpi_gtl_hsa -lxpmem -ldsmml -lpgas-shmem -lquadmath > -lcrayacc_amdgpu > > >>>>>> -lopenacc -lmodules -lfi -lcraymath -lf -lu -lcsup -lgfortran > -lpthread > > >>>>>> -lgcc_eh -lm -lclang_rt.craypgo-x86_64 -lclang_rt.builtins-x86_64 > > >>>>>> -lquadmath -lstdc++ -ldl -o ex3k > > >>>>>> > /usr/lib64/gcc/x86_64-suse-linux/7/../../../../x86_64-suse-linux/bin/ld: > > >>>>>> > /gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib/libpetsc.so: > > >>>>>> undefined reference to > `.omp_offloading.img_size.cray_amdgcn-amd-amdhsa' > > >>>>>> > /usr/lib64/gcc/x86_64-suse-linux/7/../../../../x86_64-suse-linux/bin/ld: > > >>>>>> > /gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib/libpetsc.so: > > >>>>>> undefined reference to > `.omp_offloading.img_start.cray_amdgcn-amd-amdhsa' > > >>>>>> > /usr/lib64/gcc/x86_64-suse-linux/7/../../../../x86_64-suse-linux/bin/ld: > > >>>>>> > /gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib/libpetsc.so: > > >>>>>> undefined reference to > `.omp_offloading.img_cache.cray_amdgcn-amd-amdhsa' > > >>>>>> clang-12: error: linker command failed with exit code 1 (use -v to > > >>>>>> see invocation) > > >>>>>> gmake[4]: *** > > >>>>>> [/gpfs/alpine/csc314/scratch/adams/petsc/lib/petsc/conf/test:26: > ex3k] > > >>>>>> Error 1 > > >>>>>> gmake[3]: [makefile:102: runex3k_kokkos] Error 2 (ignored) > > >>>>>> 1,25c1,4 > > >>>>>> < atol=1e-50, rtol=1e-08, stol=1e-08, maxit=50, maxf=10000 > > >>>>>> < Vec Object: Exact Solution 2 MPI processes > > >>>>>> < type: mpikokkos > > >>>>>> < Process [0] > > >>>>>> < 0. > > >>>>>> < 0.015625 > > >>>>>> < 0.125 > > >>>>>> < Process [1] > > >>>>>> < 0.421875 > > >>>>>> < 1. > > >>>>>> < Vec Object: Forcing function 2 MPI processes > > >>>>>> < type: mpikokkos > > >>>>>> < Process [0] > > >>>>>> < 1e-72 > > >>>>>> < 1.50024 > > >>>>>> < 3.01563 > > >>>>>> < Process [1] > > >>>>>> < 4.67798 > > >>>>>> < 7. > > >>>>>> < 0 SNES Function norm 5.414682427127e+00 > > >>>>>> < 1 SNES Function norm 2.952582418265e-01 > > >>>>>> < 2 SNES Function norm 4.502293658739e-04 > > >>>>>> < 3 SNES Function norm 1.389665806646e-09 > > >>>>>> < Number of SNES iterations = 3 > > >>>>>> < Norm of error 1.49752e-10 Iterations 3 > > >>>>>> --- > > >>>>>> > slurmstepd: error: execve(): > > >>>>>> > /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials/./ex3k: No such > > >>>>>> file or directory > > >>>>>> > slurmstepd: error: execve(): > > >>>>>> > /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials/./ex3k: No such > > >>>>>> file or directory > > >>>>>> > srun: error: spock01: tasks 0-1: Exited with exit code 2 > > >>>>>> > srun: launch/slurm: _step_signal: Terminating StepId=272979.0 > > >>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials > > >>>>>> Possible problem with ex3k running with kokkos-kernels, diffs > above > > >>>>>> ========================================= > > >>>>>> *******************Error detected during compile or > > >>>>>> link!******************* > > >>>>>> See http://www.mcs.anl.gov/petsc/documentation/faq.html > > >>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials ex5f > > >>>>>> ********************************************************* > > >>>>>> ftn -L/sw/spock/spack-envs/views/rocm-4.1.0/lib -lamdhip64 > > >>>>>> -lhsa-runtime64 -fPIC -g -O0 -fPIC -g -O0 > > >>>>>> -I/gpfs/alpine/csc314/scratch/adams/petsc/include > > >>>>>> > -I/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/include > > >>>>>> -I/sw/spock/spack-envs/views/rocm-4.1.0/include ex5f.F90 > > >>>>>> > -Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib > > >>>>>> > -L/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib > > >>>>>> > -Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib > > >>>>>> > -L/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib > > >>>>>> -Wl,-rpath,/sw/spock/spack-envs/views/rocm-4.1.0/lib > > >>>>>> -L/sw/spock/spack-envs/views/rocm-4.1.0/lib > > >>>>>> -Wl,-rpath,/opt/gcc/8.1.0/snos/lib64 -L/opt/gcc/8.1.0/snos/lib64 > > >>>>>> -Wl,-rpath,/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib > > >>>>>> -L/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib > > >>>>>> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib > > >>>>>> -L/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib > > >>>>>> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/gtl/lib > > >>>>>> -L/opt/cray/pe/mpich/8.1.4/gtl/lib > -Wl,-rpath,/opt/cray/pe/pmi/6.0.10/lib > > >>>>>> -L/opt/cray/pe/pmi/6.0.10/lib > -Wl,-rpath,/opt/cray/pe/dsmml/0.1.4/dsmml/lib > > >>>>>> -L/opt/cray/pe/dsmml/0.1.4/dsmml/lib > > >>>>>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce/x86_64/lib > > >>>>>> -L/opt/cray/pe/cce/11.0.4/cce/x86_64/lib > > >>>>>> -Wl,-rpath,/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64 > > >>>>>> -L/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64 > > >>>>>> > -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux > > >>>>>> > -L/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux > > >>>>>> -Wl,-rpath,/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0 > > >>>>>> -L/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0 > > >>>>>> > -Wl,-rpath,/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib > > >>>>>> > -L/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib > > >>>>>> -lpetsc -lkokkoskernels -lkokkoscontainers -lkokkoscore > -lhipsparse > > >>>>>> -lhipblas -lrocsparse -lrocsolver -lrocblas -lamdhip64 > -lhsa-runtime64 > > >>>>>> -lstdc++ -ldl -lpmi -lsci_cray_mpi -lsci_cray -lmpifort_cray > -lmpi_cray > > >>>>>> -lmpi_gtl_hsa -lxpmem -ldsmml -lpgas-shmem -lquadmath > -lcrayacc_amdgpu > > >>>>>> -lopenacc -lmodules -lfi -lcraymath -lf -lu -lcsup -lgfortran > -lpthread > > >>>>>> -lgcc_eh -lm -lclang_rt.craypgo-x86_64 -lclang_rt.builtins-x86_64 > > >>>>>> -lquadmath -lstdc++ -ldl -o ex5f > > >>>>>> > /opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/bin/ld: > > >>>>>> warning: alignment 128 of symbol > > >>>>>> `$host_init$$runtime_init_for_iso_c_binding$iso_c_binding_' in > > >>>>>> /opt/cray/pe/cce/11.0.4/cce/x86_64/lib/libmodules.so is smaller > than 256 in > > >>>>>> /tmp/pe_30429/ex5f_1.o > > >>>>>> > /opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/bin/ld: > > >>>>>> warning: alignment 64 of symbol `$data_init$iso_c_binding_' in > > >>>>>> /opt/cray/pe/cce/11.0.4/cce/x86_64/lib/libmodules.so is smaller > than 256 in > > >>>>>> /tmp/pe_30429/ex5f_1.o > > >>>>>> Fortran example src/snes/tutorials/ex5f run successfully with 1 > MPI > > >>>>>> process > > >>>>>> Completed test examples > > >>>>>> 10:49 jczhang/fix-cray-mpicxx-includes/main= > > >>>>>> /gpfs/alpine/csc314/scratch/adams/petsc$ ll > > >>>>>> > > >>>>>> On Fri, Jul 16, 2021 at 11:28 PM Junchao Zhang < > > >>>>>> junchao.zhang(a)gmail.com> wrote: > > >>>>>> > > >>>>>>> Mark, I configured with Fortran binding enabled with main + MR > !4175 > > >>>>>>> <https://gitlab.com/petsc/petsc/-/merge_requests/4175> > > >>>>>>> > > >>>>>>> '--with-mpiexec=srun', > > >>>>>>> '--with-shared-libraries=1', > > >>>>>>> '--with-cc=cc', > > >>>>>>> '--with-cxx=CC', > > >>>>>>> '--with-fc=ftn', > > >>>>>>> '--with-fortran-bindings', > > >>>>>>> '--with-hip', > > >>>>>>> '--with-hipc=hipcc', > > >>>>>>> '--with-debugging', > > >>>>>>> '--CPPFLAGS=-I${ROCM_PATH}/include', > > >>>>>>> '--CXXPPFLAGS=-I${ROCM_PATH}/include', > > >>>>>>> '--CC_LINKER_FLAGS=-L${ROCM_PATH}/lib -lamdhip64 > -lhsa-runtime64', > > >>>>>>> '--CXX_LINKER_FLAGS=-L${ROCM_PATH}/lib -lamdhip64 > -lhsa-runtime64', > > >>>>>>> '--FC_LINKER_FLAGS=-L${ROCM_PATH}/lib -lamdhip64 > -lhsa-runtime64', > > >>>>>>> '--COPTFLAGS=-g -O0', > > >>>>>>> '--CXXOPTFLAGS=-g -O0', > > >>>>>>> '--FOPTFLAGS=-g -O0', > > >>>>>>> '--download-kokkos', > > >>>>>>> '--download-kokkos-kernels', > > >>>>>>> '--download-kokkos-commit=3.4.01', > > >>>>>>> '--download-kokkos-kernels-commit=3.4.01', > > >>>>>>> '--with-kokkos-hip-arch=VEGA908', > > >>>>>>> > > >>>>>>> and 'make check' ran smoothly on a compute node > > >>>>>>> > > >>>>>>> $ make check > > >>>>>>> Running check examples to verify correct installation > > >>>>>>> Using PETSC_DIR=/ccs/home/jczhang/petsc and > > >>>>>>> PETSC_ARCH=arch-spock-cray-kokkos-dbg > > >>>>>>> C/C++ example src/snes/tutorials/ex19 run successfully with 1 MPI > > >>>>>>> process > > >>>>>>> C/C++ example src/snes/tutorials/ex19 run successfully with 2 MPI > > >>>>>>> processes > > >>>>>>> C/C++ example src/snes/tutorials/ex3k run successfully with > > >>>>>>> kokkos-kernels > > >>>>>>> Fortran example src/snes/tutorials/ex5f run successfully with 1 > MPI > > >>>>>>> process > > >>>>>>> > > >>>>>>> Completed test examples > > >>>>>>> > > >>>>>>> > > >>>>>>> --Junchao Zhang > > >>>>>>> > > >>>>>>> > > >>>>>>> On Fri, Jul 16, 2021 at 6:04 PM Mark Adams <mfadams(a)lbl.gov> > wrote: > > >>>>>>> > > >>>>>>>> And I find that this error, on non-Kokkos C tests, is fixed by > > >>>>>>>> turning the fortran bindings off: > > >>>>>>>> > > >>>>>>>> ld.lld: error: > > >>>>>>>> > /gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/lib/libpetsc.so: > > >>>>>>>> undefined reference to > .omp_offloading.img_start.cray_amdgcn-amd-amdhsa > > >>>>>>>> [--no-allow-shlib-undefined] > > >>>>>>>> ld.lld: error: > > >>>>>>>> > /gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/lib/libpetsc.so: > > >>>>>>>> undefined reference to > .omp_offloading.img_size.cray_amdgcn-amd-amdhsa > > >>>>>>>> [--no-allow-shlib-undefined] > > >>>>>>>> ld.lld: error: > > >>>>>>>> > /gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/lib/libpetsc.so: > > >>>>>>>> undefined reference to > .omp_offloading.img_cache.cray_amdgcn-amd-amdhsa > > >>>>>>>> [--no-allow-shlib-undefined] > > >>>>>>>> > > >>>>>>>> On Fri, Jul 16, 2021 at 3:53 PM Mark Adams <mfadams(a)lbl.gov> > wrote: > > >>>>>>>> > > >>>>>>>>> Not complex. THis has some overlap with my problem w/o Kokkos. > > >>>>>>>>> > > >>>>>>>>> On Fri, Jul 16, 2021 at 12:54 PM Junchao Zhang < > > >>>>>>>>> junchao.zhang(a)gmail.com> wrote: > > >>>>>>>>> > > >>>>>>>>>> Do you use complex? post your configure.log. > > >>>>>>>>>> > > >>>>>>>>>> --Junchao Zhang > > >>>>>>>>>> > > >>>>>>>>>> > > >>>>>>>>>> On Fri, Jul 16, 2021 at 9:47 AM Mark Adams <mfadams(a)lbl.gov> > > >>>>>>>>>> wrote: > > >>>>>>>>>> > > >>>>>>>>>>> The simple Kokkos example is failing for me on Spock. > > >>>>>>>>>>> Any ideas? > > >>>>>>>>>>> Thanks, > > >>>>>>>>>>> > > >>>>>>>>>>> 10:44 main *= > > >>>>>>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials$ > make > > >>>>>>>>>>> PETSC_DIR=/gpfs/alpine/csc314/scratch/adams/petsc > > >>>>>>>>>>> PETSC_ARCH=arch-spock-opt-cray-kokkos ex3k > > >>>>>>>>>>> MPICH_CXX="hipcc" OMPI_CXX="hipcc" CC > > >>>>>>>>>>> -L/sw/spock/spack-envs/views/rocm-4.1.0/lib -lhsa-runtime64 > > >>>>>>>>>>> -L/sw/spock/spack-envs/views/rocm-4.1.0/lib -lamdhip64 > -lhsa-runtime64 > > >>>>>>>>>>> -Wall -Wwrite-strings -Wno-strict-aliasing > -Wno-unknown-pragmas > > >>>>>>>>>>> -fstack-protector -fvisibility=hidden -g -O2 -fPIC -Wall > -Wwrite-strings > > >>>>>>>>>>> -Wno-strict-aliasing -Wno-unknown-pragmas > -fvisibility=hidden -g -O3 > > >>>>>>>>>>> -std=c++14 -I/gpfs/alpine/csc314/scratch/adams/petsc/include > > >>>>>>>>>>> > -I/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-opt-cray-kokkos/include > > >>>>>>>>>>> > -I/gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include > > >>>>>>>>>>> -I/sw/spock/spack-envs/views/rocm-4.1.0/include > > >>>>>>>>>>> -I/sw/spock/spack-envs/views/rocm-4.1.0/include > ex3k.kokkos.cxx > > >>>>>>>>>>> > -Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-opt-cray-kokkos/lib > > >>>>>>>>>>> > -L/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-opt-cray-kokkos/lib > > >>>>>>>>>>> > -Wl,-rpath,/gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/lib > > >>>>>>>>>>> > -L/gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/lib > > >>>>>>>>>>> -Wl,-rpath,/sw/spock/spack-envs/views/rocm-4.1.0/lib > > >>>>>>>>>>> -L/sw/spock/spack-envs/views/rocm-4.1.0/lib > > >>>>>>>>>>> -Wl,-rpath,/opt/gcc/8.1.0/snos/lib64 > -L/opt/gcc/8.1.0/snos/lib64 > > >>>>>>>>>>> -Wl,-rpath,/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib > > >>>>>>>>>>> -L/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib > > >>>>>>>>>>> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib > > >>>>>>>>>>> -L/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib > > >>>>>>>>>>> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/gtl/lib > > >>>>>>>>>>> -L/opt/cray/pe/mpich/8.1.4/gtl/lib > -Wl,-rpath,/opt/cray/pe/pmi/6.0.10/lib > > >>>>>>>>>>> -L/opt/cray/pe/pmi/6.0.10/lib > -Wl,-rpath,/opt/cray/pe/dsmml/0.1.4/dsmml/lib > > >>>>>>>>>>> -L/opt/cray/pe/dsmml/0.1.4/dsmml/lib > > >>>>>>>>>>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce/x86_64/lib > > >>>>>>>>>>> -L/opt/cray/pe/cce/11.0.4/cce/x86_64/lib > > >>>>>>>>>>> > -Wl,-rpath,/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64 > > >>>>>>>>>>> -L/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64 > > >>>>>>>>>>> > -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux > > >>>>>>>>>>> > -L/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux > > >>>>>>>>>>> > -Wl,-rpath,/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0 > > >>>>>>>>>>> -L/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0 > > >>>>>>>>>>> > -Wl,-rpath,/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib > > >>>>>>>>>>> > -L/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib > > >>>>>>>>>>> -lpetsc -lkokkoskernels -lkokkoscontainers -lkokkoscore > -lhipsparse > > >>>>>>>>>>> -lhipblas -lrocsparse -lrocsolver -lrocblas -lamdhip64 > -lhsa-runtime64 > > >>>>>>>>>>> -lstdc++ -ldl -lpmi -lsci_cray_mpi -lsci_cray -lmpifort_cray > -lmpi_cray > > >>>>>>>>>>> -lmpi_gtl_hsa -lxpmem -ldsmml -lpgas-shmem -lquadmath > -lcrayacc_amdgpu > > >>>>>>>>>>> -lopenacc -lmodules -lfi -lcraymath -lf -lu -lcsup > -lgfortran -lpthread > > >>>>>>>>>>> -lgcc_eh -lm -lclang_rt.craypgo-x86_64 > -lclang_rt.builtins-x86_64 > > >>>>>>>>>>> -lquadmath -lstdc++ -ldl -o ex3k > > >>>>>>>>>>> In file included from ex3k.kokkos.cxx:3: > > >>>>>>>>>>> In file included from > > >>>>>>>>>>> > /gpfs/alpine/csc314/scratch/adams/petsc/include/petscdmda_kokkos.hpp:4: > > >>>>>>>>>>> In file included from > > >>>>>>>>>>> > /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec_kokkos.hpp:14: > > >>>>>>>>>>> In file included from > > >>>>>>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec.h:9: > > >>>>>>>>>>> In file included from > > >>>>>>>>>>> > /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsys.h:42: > > >>>>>>>>>>> In file included from > > >>>>>>>>>>> > /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsystypes.h:255: > > >>>>>>>>>>> In file included from > > >>>>>>>>>>> > /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Complex.hpp:47: > > >>>>>>>>>>> In file included from > > >>>>>>>>>>> > /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Atomic.hpp:212: > > >>>>>>>>>>> > /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/impl/Kokkos_Atomic_Exchange.hpp:259:11: > > >>>>>>>>>>> error: no member named 'lock_address_host_space' in namespace > > >>>>>>>>>>> 'Kokkos::Impl::Kokkos::Impl'; did you mean simply > 'lock_address_host_space'? > > >>>>>>>>>>> while (!Impl::lock_address_host_space((void*)dest)) > > >>>>>>>>>>> ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~ > > >>>>>>>>>>> lock_address_host_space > > >>>>>>>>>>> > /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_HostSpace.hpp:83:6: > > >>>>>>>>>>> note: 'lock_address_host_space' declared here > > >>>>>>>>>>> bool lock_address_host_space(void* ptr); > > >>>>>>>>>>> ^ > > >>>>>>>>>>> In file included from ex3k.kokkos.cxx:3: > > >>>>>>>>>>> In file included from > > >>>>>>>>>>> > /gpfs/alpine/csc314/scratch/adams/petsc/include/petscdmda_kokkos.hpp:4: > > >>>>>>>>>>> In file included from > > >>>>>>>>>>> > /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec_kokkos.hpp:14: > > >>>>>>>>>>> In file included from > > >>>>>>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec.h:9: > > >>>>>>>>>>> In file included from > > >>>>>>>>>>> > /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsys.h:42: > > >>>>>>>>>>> In file included from > > >>>>>>>>>>> > /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsystypes.h:255: > > >>>>>>>>>>> In file included from > > >>>>>>>>>>> > /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Complex.hpp:47: > > >>>>>>>>>>> In file included from > > >>>>>>>>>>> > /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Atomic.hpp:212: > > >>>>>>>>>>> > /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/impl/Kokkos_Atomic_Exchange.hpp:261:3: > > >>>>>>>>>>> error: no member named 'memory_fence' in namespace > 'Kokkos::Impl::Kokkos'; > > >>>>>>>>>>> did you mean simply 'memory_fence'? > > >>>>>>>>>>> Kokkos::memory_fence(); > > >>>>>>>>>>> ^~~~~~~~~~~~~~~~~~~~ > > >>>>>>>>>>> memory_fence > > >>>>>>>>>>> > /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/impl/Kokkos_Memory_Fence.hpp:53:6: > > >>>>>>>>>>> note: 'memory_fence' declared here > > >>>>>>>>>>> void memory_fence() { > > >>>>>>>>>>> ^ > > >>>>>>>>>>> In file included from ex3k.kokkos.cxx:3: > > >>>>>>>>>>> In file included from > > >>>>>>>>>>> > /gpfs/alpine/csc314/scratch/adams/petsc/include/petscdmda_kokkos.hpp:4: > > >>>>>>>>>>> In file included from > > >>>>>>>>>>> > /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec_kokkos.hpp:14: > > >>>>>>>>>>> In file included from > > >>>>>>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec.h:9: > > >>>>>>>>>>> In file included from > > >>>>>>>>>>> > /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsys.h:42: > > >>>>>>>>>>> In file included from > > >>>>>>>>>>> > /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsystypes.h:255: > > >>>>>>>>>>> In file included from > > >>>>>>>>>>> > /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Complex.hpp:47: > > >>>>>>>>>>> In file included from > > >>>>>>>>>>> > /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Atomic.hpp:212: > > >>>>>>>>>>> > /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/impl/Kokkos_Atomic_Exchange.hpp:277:3: > > >>>>>>>>>>> error: no member named 'memory_fence' in namespace > 'Kokkos::Impl::Kokkos'; > > >>>>>>>>>>> did you mean simply 'memory_fence'? > > >>>>>>>>>>> Kokkos::memory_fence(); > > >>>>>>>>>>> ^~~~~~~~~~~~~~~~~~~~ > > >>>>>>>>>>> memory_fence > > >>>>>>>>>>> > /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/impl/Kokkos_Memory_Fence.hpp:53:6: > > >>>>>>>>>>> note: 'memory_fence' declared here > > >>>>>>>>>>> void memory_fence() { > > >>>>>>>>>>> ^ > > >>>>>>>>>>> > > >>>>>>>>>> > > > > > -- > Scott Kruger > Tech-X Corporation kruger(a)txcorp.com > 5621 Arapahoe Ave, Suite A Phone: (720) 466-3196 > Boulder, CO 80303 Fax: (303) 448-7756 >

1 0

Re: [petsc-dev] Kokkos make error on Spock
by Scott Kruger 19 Jul '21

19 Jul '21

Mark, On tulip, things with fortran went wonky when a `-fast` flag snuck into the flags (snuck in being me copying a previous configure file and not questioning the flags). The reason is that for clang, `-fast` implies link time optimization (lto) for C/C++ code, but flang doesn't support lto so things got weird. I suspect that gfortran does not either (but perhaps the real question is why not use flang?) I look at configure and I don't see anything in your flags that would trigger lto, but as Junchao says, it might be picking something up from Kokkos, so perhaps this is the issue. Scott On 2021-07-19 07:19, Mark Adams did write: > Thanks, but this happens w/o Kokkos. > I've stripped this down and attached good/bad logs without/with Fortran > bindings. > Hope this helps, > Thanks again, > Mark > > On Sun, Jul 18, 2021 at 12:00 PM Stefano Zampini <stefano.zampini(a)gmail.com> > wrote: > > > This is probably kokkos pulling in the dependencies where compiling kokkos > > source within PETSc. > > > > Il Dom 18 Lug 2021, 16:29 Mark Adams <mfadams(a)lbl.gov> ha scritto: > > > >> Whoops, this error was just from not telling gfortran to allow long lines. > >> > >> Anway, I did find that the when fortran bindings are enabled this OMP > >> thing gets into the library. No idea how. I'll askm ORNL. > >> > >> 10:26 jczhang/fix-cray-mpicxx-includes/main= > >> /gpfs/alpine/csc314/scratch/adams/petsc2$ nm > >> /gpfs/alpine/csc314/scratch/adams/petsc2/arch-spock-opt-cray-kokkos/lib/libpetsc.so > >> |g offload > >> U .omp_offloading.img_cache.cray_amdgcn-amd-amdhsa > >> U .omp_offloading.img_size.cray_amdgcn-amd-amdhsa > >> U .omp_offloading.img_start.cray_amdgcn-amd-amdhsa > >> 0000000001d457b0 T vecgetoffloadmask_ > >> > >> > >> On Sun, Jul 18, 2021 at 8:43 AM Mark Adams <mfadams(a)lbl.gov> wrote: > >> > >>> Ah, your test was not on Spock ... I have something working but this is > >>> strange. > >>> > >>> I switched to the GNU ProgEnv. and it passed the Fortran test in > >>> 'check', but this failed: > >>> > >>> > >>> > >>> > >>> > >>> > >>> > >>> > >>> > >>> *08:26 2 jczhang/fix-cray-mpicxx-includes/main= > >>> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials$ make > >>> PETSC_DIR=/gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-dbg-gnu > >>> PETSC_ARCH="" ex40f90ftn -L/sw/spock/spack-envs/views/rocm-4.1.0/lib > >>> -lamdhip64 -lhsa-runtime64 -fPIC -g -O0 -fPIC -g -O0 > >>> -I/gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-dbg-gnu/include > >>> -I/sw/spock/spack-envs/views/rocm-4.1.0/include ex40f90.F90 > >>> -Wl,-rpath,/gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-dbg-gnu/lib > >>> -L/gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-dbg-gnu/lib > >>> -Wl,-rpath,/sw/spock/spack-envs/views/rocm-4.1.0/lib > >>> -L/sw/spock/spack-envs/views/rocm-4.1.0/lib > >>> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/ofi/gnu/9.1/lib > >>> -L/opt/cray/pe/mpich/8.1.4/ofi/gnu/9.1/lib > >>> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/gtl/lib > >>> -L/opt/cray/pe/mpich/8.1.4/gtl/lib > >>> -Wl,-rpath,/opt/cray/pe/libsci/21.04.1.1/GNU/9.1/x86_64/lib > >>> <http://21.04.1.1/GNU/9.1/x86_64/lib> > >>> -L/opt/cray/pe/libsci/21.04.1.1/GNU/9.1/x86_64/lib > >>> <http://21.04.1.1/GNU/9.1/x86_64/lib> > >>> -Wl,-rpath,/opt/cray/pe/pmi/6.0.10/lib -L/opt/cray/pe/pmi/6.0.10/lib > >>> -Wl,-rpath,/opt/cray/pe/dsmml/0.1.4/dsmml/lib > >>> -L/opt/cray/pe/dsmml/0.1.4/dsmml/lib > >>> -Wl,-rpath,/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64 > >>> -L/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64 > >>> -Wl,-rpath,/opt/gcc/10.2.0/snos/lib/gcc/x86_64-suse-linux/10.2.0 > >>> -L/opt/gcc/10.2.0/snos/lib/gcc/x86_64-suse-linux/10.2.0 > >>> -Wl,-rpath,/opt/gcc/10.2.0/snos/lib64 -L/opt/gcc/10.2.0/snos/lib64 > >>> -Wl,-rpath,/sw/spock/spack-envs/views/rocm-4.1.0/llvm/lib > >>> -L/sw/spock/spack-envs/views/rocm-4.1.0/llvm/lib > >>> -Wl,-rpath,/opt/gcc/10.2.0/snos/lib -L/opt/gcc/10.2.0/snos/lib -lpetsc > >>> -lhipsparse -lhipblas -lrocsparse -lrocsolver -lrocblas -lamdhip64 > >>> -lhsa-runtime64 -lstdc++ -ldl -lpmi -lmpi_gtl_hsa -lxpmem -lgfortran -lm > >>> -lsci_gnu_82_mpi -lsci_gnu_82 -lmpifort_gnu_91 -lmpi_gnu_91 -ldsmml > >>> -lgfortran -lquadmath -lpthread -lm -lgcc_s -lquadmath -lstdc++ -ldl -o > >>> ex40f90ex40f90.F90:35:132: 35 | call > >>> DMDACreate2d(PETSC_COMM_WORLD,DM_BOUNDARY_NONE,DM_BOUNDARY_NONE,DMDA_STENCIL_BOX,ten,ten,PETSC_DECIDE,PETSC_DECIDE,two,one, > >>> & | > >>> 1Error: > >>> Line truncated at (1) [-Werror=line-truncation]ex40f90.F90:35:132:* > >>> * ....* > >>> > >>> But ex5f (the Fortran check test) does work manually also (sanity check). > >>> > >>> Oddly I seem to have the cray compilers: > >>> > >>> 08:34 jczhang/fix-cray-mpicxx-includes/main= > >>> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials$ which ftn > >>> /opt/cray/pe/craype/2.7.6/bin/ftn > >>> > >>> yet it is GNU: > >>> > >>> 08:34 jczhang/fix-cray-mpicxx-includes/main= > >>> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials$ ftn --version > >>> GNU Fortran (GCC) 10.2.0 20200723 (Cray Inc.) > >>> Copyright (C) 2020 Free Software Foundation, Inc. > >>> This is free software; see the source for copying conditions. There is > >>> NO > >>> warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR > >>> PURPOSE. > >>> > >>> The documentation says ftn should be here: > >>> > >>> 08:34 jczhang/fix-cray-mpicxx-includes/main= > >>> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials$ > >>> $GCC_PATH/bin/gfortran --version > >>> GNU Fortran (GCC) 10.2.0 20200723 (Cray Inc.) > >>> Copyright (C) 2020 Free Software Foundation, Inc. > >>> This is free software; see the source for copying conditions. There is > >>> NO > >>> warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR > >>> PURPOSE. > >>> > >>> Same, but the path is not the same: > >>> > >>> 08:39 jczhang/fix-cray-mpicxx-includes/main= > >>> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials$ echo $GCC_PATH > >>> /opt/gcc/10.2.0 > >>> > >>> Anyway, thanks for your help, > >>> Mark > >>> > >>> On Sat, Jul 17, 2021 at 11:26 PM Junchao Zhang <junchao.zhang(a)gmail.com> > >>> wrote: > >>> > >>>> Mark, I can reproduce this error with PrgEnv-cray, i.e., using the > >>>> Cray compiler (clang-11). Previously I used PrgEnv-gnu, which did not have > >>>> this error. > >>>> Probably it is a problem of Spock. But I am not sure. > >>>> > >>>> --Junchao Zhang > >>>> > >>>> > >>>> On Sat, Jul 17, 2021 at 10:17 AM Mark Adams <mfadams(a)lbl.gov> wrote: > >>>> > >>>>> And I can run a fortran test, with warnings, but C tests fail: > >>>>> > >>>>> 11:15 jczhang/fix-cray-mpicxx-includes/main= > >>>>> /gpfs/alpine/csc314/scratch/adams/petsc$ make > >>>>> PETSC_DIR=/gpfs/alpine/csc314/scratch/adams/petsc > >>>>> PETSC_ARCH=arch-spock-opt-cray-kokkos -f gmakefile test > >>>>> search='ts_utils_dmplexlandau_tutorials-ex1f90_0' > >>>>> Using MAKEFLAGS: -- search=ts_utils_dmplexlandau_tutorials-ex1f90_0 > >>>>> PETSC_ARCH=arch-spock-opt-cray-kokkos > >>>>> PETSC_DIR=/gpfs/alpine/csc314/scratch/adams/petsc > >>>>> FC > >>>>> arch-spock-opt-cray-kokkos/tests/ts/utils/dmplexlandau/tutorials/ex1f90.o > >>>>> FLINKER > >>>>> arch-spock-opt-cray-kokkos/tests/ts/utils/dmplexlandau/tutorials/ex1f90 > >>>>> /opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/bin/ld: > >>>>> warning: alignment 128 of symbol > >>>>> `$host_init$$runtime_init_for_iso_c_binding$iso_c_binding_' in > >>>>> /opt/cray/pe/cce/11.0.4/cce/x86_64/lib/libmodules.so is smaller than 256 in > >>>>> arch-spock-opt-cray-kokkos/tests/ts/utils/dmplexlandau/tutorials/ex1f90.o > >>>>> /opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/bin/ld: > >>>>> warning: alignment 64 of symbol `$data_init$iso_c_binding_' in > >>>>> /opt/cray/pe/cce/11.0.4/cce/x86_64/lib/libmodules.so is smaller than 256 in > >>>>> arch-spock-opt-cray-kokkos/tests/ts/utils/dmplexlandau/tutorials/ex1f90.o > >>>>> TEST > >>>>> arch-spock-opt-cray-kokkos/tests/counts/ts_utils_dmplexlandau_tutorials-ex1f90_0.counts > >>>>> ok ts_utils_dmplexlandau_tutorials-ex1f90_0 > >>>>> ok diff-ts_utils_dmplexlandau_tutorials-ex1f90_0 > >>>>> > >>>>> > >>>>> On Sat, Jul 17, 2021 at 10:53 AM Mark Adams <mfadams(a)lbl.gov> wrote: > >>>>> > >>>>>> HUmm, I can not reproduce this. > >>>>>> > >>>>>> 10:49 jczhang/fix-cray-mpicxx-includes/main= > >>>>>> /gpfs/alpine/csc314/scratch/adams/petsc$ make > >>>>>> PETSC_DIR=/gpfs/alpine/csc314/scratch/adams/petsc > >>>>>> PETSC_ARCH=arch-spock-dbg-kokkos check > >>>>>> Running check examples to verify correct installation > >>>>>> Using PETSC_DIR=/gpfs/alpine/csc314/scratch/adams/petsc and > >>>>>> PETSC_ARCH=arch-spock-dbg-kokkos > >>>>>> gmake[3]: > >>>>>> [/gpfs/alpine/csc314/scratch/adams/petsc/lib/petsc/conf/rules:301: > >>>>>> ex19.PETSc] Error 2 (ignored) > >>>>>> *******************Error detected during compile or > >>>>>> link!******************* > >>>>>> See http://www.mcs.anl.gov/petsc/documentation/faq.html > >>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials ex19 > >>>>>> > >>>>>> ********************************************************************************* > >>>>>> cc -L/sw/spock/spack-envs/views/rocm-4.1.0/lib -lamdhip64 > >>>>>> -lhsa-runtime64 -fPIC -Wall -Wwrite-strings -Wno-strict-aliasing > >>>>>> -Wno-unknown-pragmas -fstack-protector -Qunused-arguments > >>>>>> -fvisibility=hidden -g -O0 -fPIC -Wall -Wwrite-strings > >>>>>> -Wno-strict-aliasing -Wno-unknown-pragmas -fstack-protector > >>>>>> -Qunused-arguments -fvisibility=hidden -g -O0 > >>>>>> -I/sw/spock/spack-envs/views/rocm-4.1.0/include > >>>>>> -I/gpfs/alpine/csc314/scratch/adams/petsc/include > >>>>>> -I/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/include > >>>>>> -I/sw/spock/spack-envs/views/rocm-4.1.0/include > >>>>>> -I/sw/spock/spack-envs/views/rocm-4.1.0/include ex19.c > >>>>>> -Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib > >>>>>> -L/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib > >>>>>> -Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib > >>>>>> -L/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib > >>>>>> -Wl,-rpath,/sw/spock/spack-envs/views/rocm-4.1.0/lib > >>>>>> -L/sw/spock/spack-envs/views/rocm-4.1.0/lib > >>>>>> -Wl,-rpath,/opt/gcc/8.1.0/snos/lib64 -L/opt/gcc/8.1.0/snos/lib64 > >>>>>> -Wl,-rpath,/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib > >>>>>> -L/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib > >>>>>> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib > >>>>>> -L/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib > >>>>>> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/gtl/lib > >>>>>> -L/opt/cray/pe/mpich/8.1.4/gtl/lib -Wl,-rpath,/opt/cray/pe/pmi/6.0.10/lib > >>>>>> -L/opt/cray/pe/pmi/6.0.10/lib -Wl,-rpath,/opt/cray/pe/dsmml/0.1.4/dsmml/lib > >>>>>> -L/opt/cray/pe/dsmml/0.1.4/dsmml/lib > >>>>>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce/x86_64/lib > >>>>>> -L/opt/cray/pe/cce/11.0.4/cce/x86_64/lib > >>>>>> -Wl,-rpath,/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64 > >>>>>> -L/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64 > >>>>>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux > >>>>>> -L/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux > >>>>>> -Wl,-rpath,/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0 > >>>>>> -L/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0 > >>>>>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib > >>>>>> -L/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib > >>>>>> -lpetsc -lkokkoskernels -lkokkoscontainers -lkokkoscore -lhipsparse > >>>>>> -lhipblas -lrocsparse -lrocsolver -lrocblas -lamdhip64 -lhsa-runtime64 > >>>>>> -lstdc++ -ldl -lpmi -lsci_cray_mpi -lsci_cray -lmpifort_cray -lmpi_cray > >>>>>> -lmpi_gtl_hsa -lxpmem -ldsmml -lpgas-shmem -lquadmath -lcrayacc_amdgpu > >>>>>> -lopenacc -lmodules -lfi -lcraymath -lf -lu -lcsup -lgfortran -lpthread > >>>>>> -lgcc_eh -lm -lclang_rt.craypgo-x86_64 -lclang_rt.builtins-x86_64 > >>>>>> -lquadmath -lstdc++ -ldl -o ex19 > >>>>>> ld.lld: error: > >>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib/libpetsc.so: > >>>>>> undefined reference to .omp_offloading.img_start.cray_amdgcn-amd-amdhsa > >>>>>> [--no-allow-shlib-undefined] > >>>>>> ld.lld: error: > >>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib/libpetsc.so: > >>>>>> undefined reference to .omp_offloading.img_size.cray_amdgcn-amd-amdhsa > >>>>>> [--no-allow-shlib-undefined] > >>>>>> ld.lld: error: > >>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib/libpetsc.so: > >>>>>> undefined reference to .omp_offloading.img_cache.cray_amdgcn-amd-amdhsa > >>>>>> [--no-allow-shlib-undefined] > >>>>>> clang-11: error: linker command failed with exit code 1 (use -v to > >>>>>> see invocation) > >>>>>> gmake[4]: *** [<builtin>: ex19] Error 1 > >>>>>> gmake[3]: > >>>>>> [/gpfs/alpine/csc314/scratch/adams/petsc/lib/petsc/conf/rules:301: > >>>>>> ex3k.PETSc] Error 2 (ignored) > >>>>>> *******************Error detected during compile or > >>>>>> link!******************* > >>>>>> See http://www.mcs.anl.gov/petsc/documentation/faq.html > >>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials ex3k > >>>>>> > >>>>>> ********************************************************************************* > >>>>>> hipcc -fPIC -Wall -Wwrite-strings -Wno-strict-aliasing > >>>>>> -Wno-unknown-pragmas -fvisibility=hidden -g -O0 -std=c++14 > >>>>>> -I/gpfs/alpine/csc314/scratch/adams/petsc/include > >>>>>> -I/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/include > >>>>>> -I/sw/spock/spack-envs/views/rocm-4.1.0/include > >>>>>> -I/sw/spock/spack-envs/views/rocm-4.1.0/include -I/opt/cray/pe/libsci/ > >>>>>> 21.04.1.1/CRAY/9.0/x86_64/include > >>>>>> -I/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/include > >>>>>> -I/opt/cray/pe/pmi/6.0.10/include -I/opt/cray/pe/dsmml/0.1.4/dsmml//include > >>>>>> -I/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/include > >>>>>> -L/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib > >>>>>> -L/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib > >>>>>> -L/opt/cray/pe/mpich/8.1.4/gtl/lib -L/opt/cray/pe/pmi/6.0.10/lib > >>>>>> -L/opt/cray/pe/dsmml/0.1.4/dsmml//lib > >>>>>> -L/opt/cray/pe/cce/11.0.4/cce/x86_64/lib/pkgconfig/../ > >>>>>> -L/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64 > >>>>>> -Wl,--as-needed,-lsci_cray_mpi,--no-as-needed > >>>>>> -Wl,--as-needed,-lsci_cray,--no-as-needed > >>>>>> -Wl,--as-needed,-lmpi_cray,--no-as-needed > >>>>>> -Wl,--as-needed,-ldsmml,--no-as-needed > >>>>>> -Wl,--as-needed,-lpgas-shmem,--no-as-needed -lpmi -ldl -lmpi_gtl_hsa > >>>>>> -lxpmem -lquadmath -lmodules -lfi -lcraymath -lf -lu -lcsup > >>>>>> ex3k.kokkos.cxx > >>>>>> -Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib > >>>>>> -L/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib > >>>>>> -Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib > >>>>>> -L/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib > >>>>>> -Wl,-rpath,/sw/spock/spack-envs/views/rocm-4.1.0/lib > >>>>>> -L/sw/spock/spack-envs/views/rocm-4.1.0/lib > >>>>>> -Wl,-rpath,/opt/gcc/8.1.0/snos/lib64 -L/opt/gcc/8.1.0/snos/lib64 > >>>>>> -Wl,-rpath,/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib > >>>>>> -L/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib > >>>>>> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib > >>>>>> -L/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib > >>>>>> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/gtl/lib > >>>>>> -L/opt/cray/pe/mpich/8.1.4/gtl/lib -Wl,-rpath,/opt/cray/pe/pmi/6.0.10/lib > >>>>>> -L/opt/cray/pe/pmi/6.0.10/lib -Wl,-rpath,/opt/cray/pe/dsmml/0.1.4/dsmml/lib > >>>>>> -L/opt/cray/pe/dsmml/0.1.4/dsmml/lib > >>>>>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce/x86_64/lib > >>>>>> -L/opt/cray/pe/cce/11.0.4/cce/x86_64/lib > >>>>>> -Wl,-rpath,/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64 > >>>>>> -L/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64 > >>>>>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux > >>>>>> -L/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux > >>>>>> -Wl,-rpath,/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0 > >>>>>> -L/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0 > >>>>>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib > >>>>>> -L/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib > >>>>>> -lpetsc -lkokkoskernels -lkokkoscontainers -lkokkoscore -lhipsparse > >>>>>> -lhipblas -lrocsparse -lrocsolver -lrocblas -lamdhip64 -lhsa-runtime64 > >>>>>> -lstdc++ -ldl -lpmi -lsci_cray_mpi -lsci_cray -lmpifort_cray -lmpi_cray > >>>>>> -lmpi_gtl_hsa -lxpmem -ldsmml -lpgas-shmem -lquadmath -lcrayacc_amdgpu > >>>>>> -lopenacc -lmodules -lfi -lcraymath -lf -lu -lcsup -lgfortran -lpthread > >>>>>> -lgcc_eh -lm -lclang_rt.craypgo-x86_64 -lclang_rt.builtins-x86_64 > >>>>>> -lquadmath -lstdc++ -ldl -o ex3k > >>>>>> /usr/lib64/gcc/x86_64-suse-linux/7/../../../../x86_64-suse-linux/bin/ld: > >>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib/libpetsc.so: > >>>>>> undefined reference to `.omp_offloading.img_size.cray_amdgcn-amd-amdhsa' > >>>>>> /usr/lib64/gcc/x86_64-suse-linux/7/../../../../x86_64-suse-linux/bin/ld: > >>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib/libpetsc.so: > >>>>>> undefined reference to `.omp_offloading.img_start.cray_amdgcn-amd-amdhsa' > >>>>>> /usr/lib64/gcc/x86_64-suse-linux/7/../../../../x86_64-suse-linux/bin/ld: > >>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib/libpetsc.so: > >>>>>> undefined reference to `.omp_offloading.img_cache.cray_amdgcn-amd-amdhsa' > >>>>>> clang-12: error: linker command failed with exit code 1 (use -v to > >>>>>> see invocation) > >>>>>> gmake[4]: *** > >>>>>> [/gpfs/alpine/csc314/scratch/adams/petsc/lib/petsc/conf/test:26: ex3k] > >>>>>> Error 1 > >>>>>> gmake[3]: [makefile:102: runex3k_kokkos] Error 2 (ignored) > >>>>>> 1,25c1,4 > >>>>>> < atol=1e-50, rtol=1e-08, stol=1e-08, maxit=50, maxf=10000 > >>>>>> < Vec Object: Exact Solution 2 MPI processes > >>>>>> < type: mpikokkos > >>>>>> < Process [0] > >>>>>> < 0. > >>>>>> < 0.015625 > >>>>>> < 0.125 > >>>>>> < Process [1] > >>>>>> < 0.421875 > >>>>>> < 1. > >>>>>> < Vec Object: Forcing function 2 MPI processes > >>>>>> < type: mpikokkos > >>>>>> < Process [0] > >>>>>> < 1e-72 > >>>>>> < 1.50024 > >>>>>> < 3.01563 > >>>>>> < Process [1] > >>>>>> < 4.67798 > >>>>>> < 7. > >>>>>> < 0 SNES Function norm 5.414682427127e+00 > >>>>>> < 1 SNES Function norm 2.952582418265e-01 > >>>>>> < 2 SNES Function norm 4.502293658739e-04 > >>>>>> < 3 SNES Function norm 1.389665806646e-09 > >>>>>> < Number of SNES iterations = 3 > >>>>>> < Norm of error 1.49752e-10 Iterations 3 > >>>>>> --- > >>>>>> > slurmstepd: error: execve(): > >>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials/./ex3k: No such > >>>>>> file or directory > >>>>>> > slurmstepd: error: execve(): > >>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials/./ex3k: No such > >>>>>> file or directory > >>>>>> > srun: error: spock01: tasks 0-1: Exited with exit code 2 > >>>>>> > srun: launch/slurm: _step_signal: Terminating StepId=272979.0 > >>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials > >>>>>> Possible problem with ex3k running with kokkos-kernels, diffs above > >>>>>> ========================================= > >>>>>> *******************Error detected during compile or > >>>>>> link!******************* > >>>>>> See http://www.mcs.anl.gov/petsc/documentation/faq.html > >>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials ex5f > >>>>>> ********************************************************* > >>>>>> ftn -L/sw/spock/spack-envs/views/rocm-4.1.0/lib -lamdhip64 > >>>>>> -lhsa-runtime64 -fPIC -g -O0 -fPIC -g -O0 > >>>>>> -I/gpfs/alpine/csc314/scratch/adams/petsc/include > >>>>>> -I/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/include > >>>>>> -I/sw/spock/spack-envs/views/rocm-4.1.0/include ex5f.F90 > >>>>>> -Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib > >>>>>> -L/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib > >>>>>> -Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib > >>>>>> -L/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib > >>>>>> -Wl,-rpath,/sw/spock/spack-envs/views/rocm-4.1.0/lib > >>>>>> -L/sw/spock/spack-envs/views/rocm-4.1.0/lib > >>>>>> -Wl,-rpath,/opt/gcc/8.1.0/snos/lib64 -L/opt/gcc/8.1.0/snos/lib64 > >>>>>> -Wl,-rpath,/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib > >>>>>> -L/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib > >>>>>> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib > >>>>>> -L/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib > >>>>>> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/gtl/lib > >>>>>> -L/opt/cray/pe/mpich/8.1.4/gtl/lib -Wl,-rpath,/opt/cray/pe/pmi/6.0.10/lib > >>>>>> -L/opt/cray/pe/pmi/6.0.10/lib -Wl,-rpath,/opt/cray/pe/dsmml/0.1.4/dsmml/lib > >>>>>> -L/opt/cray/pe/dsmml/0.1.4/dsmml/lib > >>>>>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce/x86_64/lib > >>>>>> -L/opt/cray/pe/cce/11.0.4/cce/x86_64/lib > >>>>>> -Wl,-rpath,/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64 > >>>>>> -L/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64 > >>>>>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux > >>>>>> -L/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux > >>>>>> -Wl,-rpath,/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0 > >>>>>> -L/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0 > >>>>>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib > >>>>>> -L/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib > >>>>>> -lpetsc -lkokkoskernels -lkokkoscontainers -lkokkoscore -lhipsparse > >>>>>> -lhipblas -lrocsparse -lrocsolver -lrocblas -lamdhip64 -lhsa-runtime64 > >>>>>> -lstdc++ -ldl -lpmi -lsci_cray_mpi -lsci_cray -lmpifort_cray -lmpi_cray > >>>>>> -lmpi_gtl_hsa -lxpmem -ldsmml -lpgas-shmem -lquadmath -lcrayacc_amdgpu > >>>>>> -lopenacc -lmodules -lfi -lcraymath -lf -lu -lcsup -lgfortran -lpthread > >>>>>> -lgcc_eh -lm -lclang_rt.craypgo-x86_64 -lclang_rt.builtins-x86_64 > >>>>>> -lquadmath -lstdc++ -ldl -o ex5f > >>>>>> /opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/bin/ld: > >>>>>> warning: alignment 128 of symbol > >>>>>> `$host_init$$runtime_init_for_iso_c_binding$iso_c_binding_' in > >>>>>> /opt/cray/pe/cce/11.0.4/cce/x86_64/lib/libmodules.so is smaller than 256 in > >>>>>> /tmp/pe_30429/ex5f_1.o > >>>>>> /opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/bin/ld: > >>>>>> warning: alignment 64 of symbol `$data_init$iso_c_binding_' in > >>>>>> /opt/cray/pe/cce/11.0.4/cce/x86_64/lib/libmodules.so is smaller than 256 in > >>>>>> /tmp/pe_30429/ex5f_1.o > >>>>>> Fortran example src/snes/tutorials/ex5f run successfully with 1 MPI > >>>>>> process > >>>>>> Completed test examples > >>>>>> 10:49 jczhang/fix-cray-mpicxx-includes/main= > >>>>>> /gpfs/alpine/csc314/scratch/adams/petsc$ ll > >>>>>> > >>>>>> On Fri, Jul 16, 2021 at 11:28 PM Junchao Zhang < > >>>>>> junchao.zhang(a)gmail.com> wrote: > >>>>>> > >>>>>>> Mark, I configured with Fortran binding enabled with main + MR !4175 > >>>>>>> <https://gitlab.com/petsc/petsc/-/merge_requests/4175> > >>>>>>> > >>>>>>> '--with-mpiexec=srun', > >>>>>>> '--with-shared-libraries=1', > >>>>>>> '--with-cc=cc', > >>>>>>> '--with-cxx=CC', > >>>>>>> '--with-fc=ftn', > >>>>>>> '--with-fortran-bindings', > >>>>>>> '--with-hip', > >>>>>>> '--with-hipc=hipcc', > >>>>>>> '--with-debugging', > >>>>>>> '--CPPFLAGS=-I${ROCM_PATH}/include', > >>>>>>> '--CXXPPFLAGS=-I${ROCM_PATH}/include', > >>>>>>> '--CC_LINKER_FLAGS=-L${ROCM_PATH}/lib -lamdhip64 -lhsa-runtime64', > >>>>>>> '--CXX_LINKER_FLAGS=-L${ROCM_PATH}/lib -lamdhip64 -lhsa-runtime64', > >>>>>>> '--FC_LINKER_FLAGS=-L${ROCM_PATH}/lib -lamdhip64 -lhsa-runtime64', > >>>>>>> '--COPTFLAGS=-g -O0', > >>>>>>> '--CXXOPTFLAGS=-g -O0', > >>>>>>> '--FOPTFLAGS=-g -O0', > >>>>>>> '--download-kokkos', > >>>>>>> '--download-kokkos-kernels', > >>>>>>> '--download-kokkos-commit=3.4.01', > >>>>>>> '--download-kokkos-kernels-commit=3.4.01', > >>>>>>> '--with-kokkos-hip-arch=VEGA908', > >>>>>>> > >>>>>>> and 'make check' ran smoothly on a compute node > >>>>>>> > >>>>>>> $ make check > >>>>>>> Running check examples to verify correct installation > >>>>>>> Using PETSC_DIR=/ccs/home/jczhang/petsc and > >>>>>>> PETSC_ARCH=arch-spock-cray-kokkos-dbg > >>>>>>> C/C++ example src/snes/tutorials/ex19 run successfully with 1 MPI > >>>>>>> process > >>>>>>> C/C++ example src/snes/tutorials/ex19 run successfully with 2 MPI > >>>>>>> processes > >>>>>>> C/C++ example src/snes/tutorials/ex3k run successfully with > >>>>>>> kokkos-kernels > >>>>>>> Fortran example src/snes/tutorials/ex5f run successfully with 1 MPI > >>>>>>> process > >>>>>>> > >>>>>>> Completed test examples > >>>>>>> > >>>>>>> > >>>>>>> --Junchao Zhang > >>>>>>> > >>>>>>> > >>>>>>> On Fri, Jul 16, 2021 at 6:04 PM Mark Adams <mfadams(a)lbl.gov> wrote: > >>>>>>> > >>>>>>>> And I find that this error, on non-Kokkos C tests, is fixed by > >>>>>>>> turning the fortran bindings off: > >>>>>>>> > >>>>>>>> ld.lld: error: > >>>>>>>> /gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/lib/libpetsc.so: > >>>>>>>> undefined reference to .omp_offloading.img_start.cray_amdgcn-amd-amdhsa > >>>>>>>> [--no-allow-shlib-undefined] > >>>>>>>> ld.lld: error: > >>>>>>>> /gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/lib/libpetsc.so: > >>>>>>>> undefined reference to .omp_offloading.img_size.cray_amdgcn-amd-amdhsa > >>>>>>>> [--no-allow-shlib-undefined] > >>>>>>>> ld.lld: error: > >>>>>>>> /gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/lib/libpetsc.so: > >>>>>>>> undefined reference to .omp_offloading.img_cache.cray_amdgcn-amd-amdhsa > >>>>>>>> [--no-allow-shlib-undefined] > >>>>>>>> > >>>>>>>> On Fri, Jul 16, 2021 at 3:53 PM Mark Adams <mfadams(a)lbl.gov> wrote: > >>>>>>>> > >>>>>>>>> Not complex. THis has some overlap with my problem w/o Kokkos. > >>>>>>>>> > >>>>>>>>> On Fri, Jul 16, 2021 at 12:54 PM Junchao Zhang < > >>>>>>>>> junchao.zhang(a)gmail.com> wrote: > >>>>>>>>> > >>>>>>>>>> Do you use complex? post your configure.log. > >>>>>>>>>> > >>>>>>>>>> --Junchao Zhang > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> On Fri, Jul 16, 2021 at 9:47 AM Mark Adams <mfadams(a)lbl.gov> > >>>>>>>>>> wrote: > >>>>>>>>>> > >>>>>>>>>>> The simple Kokkos example is failing for me on Spock. > >>>>>>>>>>> Any ideas? > >>>>>>>>>>> Thanks, > >>>>>>>>>>> > >>>>>>>>>>> 10:44 main *= > >>>>>>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials$ make > >>>>>>>>>>> PETSC_DIR=/gpfs/alpine/csc314/scratch/adams/petsc > >>>>>>>>>>> PETSC_ARCH=arch-spock-opt-cray-kokkos ex3k > >>>>>>>>>>> MPICH_CXX="hipcc" OMPI_CXX="hipcc" CC > >>>>>>>>>>> -L/sw/spock/spack-envs/views/rocm-4.1.0/lib -lhsa-runtime64 > >>>>>>>>>>> -L/sw/spock/spack-envs/views/rocm-4.1.0/lib -lamdhip64 -lhsa-runtime64 > >>>>>>>>>>> -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas > >>>>>>>>>>> -fstack-protector -fvisibility=hidden -g -O2 -fPIC -Wall -Wwrite-strings > >>>>>>>>>>> -Wno-strict-aliasing -Wno-unknown-pragmas -fvisibility=hidden -g -O3 > >>>>>>>>>>> -std=c++14 -I/gpfs/alpine/csc314/scratch/adams/petsc/include > >>>>>>>>>>> -I/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-opt-cray-kokkos/include > >>>>>>>>>>> -I/gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include > >>>>>>>>>>> -I/sw/spock/spack-envs/views/rocm-4.1.0/include > >>>>>>>>>>> -I/sw/spock/spack-envs/views/rocm-4.1.0/include ex3k.kokkos.cxx > >>>>>>>>>>> -Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-opt-cray-kokkos/lib > >>>>>>>>>>> -L/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-opt-cray-kokkos/lib > >>>>>>>>>>> -Wl,-rpath,/gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/lib > >>>>>>>>>>> -L/gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/lib > >>>>>>>>>>> -Wl,-rpath,/sw/spock/spack-envs/views/rocm-4.1.0/lib > >>>>>>>>>>> -L/sw/spock/spack-envs/views/rocm-4.1.0/lib > >>>>>>>>>>> -Wl,-rpath,/opt/gcc/8.1.0/snos/lib64 -L/opt/gcc/8.1.0/snos/lib64 > >>>>>>>>>>> -Wl,-rpath,/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib > >>>>>>>>>>> -L/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib > >>>>>>>>>>> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib > >>>>>>>>>>> -L/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib > >>>>>>>>>>> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/gtl/lib > >>>>>>>>>>> -L/opt/cray/pe/mpich/8.1.4/gtl/lib -Wl,-rpath,/opt/cray/pe/pmi/6.0.10/lib > >>>>>>>>>>> -L/opt/cray/pe/pmi/6.0.10/lib -Wl,-rpath,/opt/cray/pe/dsmml/0.1.4/dsmml/lib > >>>>>>>>>>> -L/opt/cray/pe/dsmml/0.1.4/dsmml/lib > >>>>>>>>>>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce/x86_64/lib > >>>>>>>>>>> -L/opt/cray/pe/cce/11.0.4/cce/x86_64/lib > >>>>>>>>>>> -Wl,-rpath,/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64 > >>>>>>>>>>> -L/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64 > >>>>>>>>>>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux > >>>>>>>>>>> -L/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux > >>>>>>>>>>> -Wl,-rpath,/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0 > >>>>>>>>>>> -L/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0 > >>>>>>>>>>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib > >>>>>>>>>>> -L/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib > >>>>>>>>>>> -lpetsc -lkokkoskernels -lkokkoscontainers -lkokkoscore -lhipsparse > >>>>>>>>>>> -lhipblas -lrocsparse -lrocsolver -lrocblas -lamdhip64 -lhsa-runtime64 > >>>>>>>>>>> -lstdc++ -ldl -lpmi -lsci_cray_mpi -lsci_cray -lmpifort_cray -lmpi_cray > >>>>>>>>>>> -lmpi_gtl_hsa -lxpmem -ldsmml -lpgas-shmem -lquadmath -lcrayacc_amdgpu > >>>>>>>>>>> -lopenacc -lmodules -lfi -lcraymath -lf -lu -lcsup -lgfortran -lpthread > >>>>>>>>>>> -lgcc_eh -lm -lclang_rt.craypgo-x86_64 -lclang_rt.builtins-x86_64 > >>>>>>>>>>> -lquadmath -lstdc++ -ldl -o ex3k > >>>>>>>>>>> In file included from ex3k.kokkos.cxx:3: > >>>>>>>>>>> In file included from > >>>>>>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscdmda_kokkos.hpp:4: > >>>>>>>>>>> In file included from > >>>>>>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec_kokkos.hpp:14: > >>>>>>>>>>> In file included from > >>>>>>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec.h:9: > >>>>>>>>>>> In file included from > >>>>>>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsys.h:42: > >>>>>>>>>>> In file included from > >>>>>>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsystypes.h:255: > >>>>>>>>>>> In file included from > >>>>>>>>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Complex.hpp:47: > >>>>>>>>>>> In file included from > >>>>>>>>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Atomic.hpp:212: > >>>>>>>>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/impl/Kokkos_Atomic_Exchange.hpp:259:11: > >>>>>>>>>>> error: no member named 'lock_address_host_space' in namespace > >>>>>>>>>>> 'Kokkos::Impl::Kokkos::Impl'; did you mean simply 'lock_address_host_space'? > >>>>>>>>>>> while (!Impl::lock_address_host_space((void*)dest)) > >>>>>>>>>>> ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~ > >>>>>>>>>>> lock_address_host_space > >>>>>>>>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_HostSpace.hpp:83:6: > >>>>>>>>>>> note: 'lock_address_host_space' declared here > >>>>>>>>>>> bool lock_address_host_space(void* ptr); > >>>>>>>>>>> ^ > >>>>>>>>>>> In file included from ex3k.kokkos.cxx:3: > >>>>>>>>>>> In file included from > >>>>>>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscdmda_kokkos.hpp:4: > >>>>>>>>>>> In file included from > >>>>>>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec_kokkos.hpp:14: > >>>>>>>>>>> In file included from > >>>>>>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec.h:9: > >>>>>>>>>>> In file included from > >>>>>>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsys.h:42: > >>>>>>>>>>> In file included from > >>>>>>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsystypes.h:255: > >>>>>>>>>>> In file included from > >>>>>>>>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Complex.hpp:47: > >>>>>>>>>>> In file included from > >>>>>>>>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Atomic.hpp:212: > >>>>>>>>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/impl/Kokkos_Atomic_Exchange.hpp:261:3: > >>>>>>>>>>> error: no member named 'memory_fence' in namespace 'Kokkos::Impl::Kokkos'; > >>>>>>>>>>> did you mean simply 'memory_fence'? > >>>>>>>>>>> Kokkos::memory_fence(); > >>>>>>>>>>> ^~~~~~~~~~~~~~~~~~~~ > >>>>>>>>>>> memory_fence > >>>>>>>>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/impl/Kokkos_Memory_Fence.hpp:53:6: > >>>>>>>>>>> note: 'memory_fence' declared here > >>>>>>>>>>> void memory_fence() { > >>>>>>>>>>> ^ > >>>>>>>>>>> In file included from ex3k.kokkos.cxx:3: > >>>>>>>>>>> In file included from > >>>>>>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscdmda_kokkos.hpp:4: > >>>>>>>>>>> In file included from > >>>>>>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec_kokkos.hpp:14: > >>>>>>>>>>> In file included from > >>>>>>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec.h:9: > >>>>>>>>>>> In file included from > >>>>>>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsys.h:42: > >>>>>>>>>>> In file included from > >>>>>>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsystypes.h:255: > >>>>>>>>>>> In file included from > >>>>>>>>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Complex.hpp:47: > >>>>>>>>>>> In file included from > >>>>>>>>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Atomic.hpp:212: > >>>>>>>>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/impl/Kokkos_Atomic_Exchange.hpp:277:3: > >>>>>>>>>>> error: no member named 'memory_fence' in namespace 'Kokkos::Impl::Kokkos'; > >>>>>>>>>>> did you mean simply 'memory_fence'? > >>>>>>>>>>> Kokkos::memory_fence(); > >>>>>>>>>>> ^~~~~~~~~~~~~~~~~~~~ > >>>>>>>>>>> memory_fence > >>>>>>>>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/impl/Kokkos_Memory_Fence.hpp:53:6: > >>>>>>>>>>> note: 'memory_fence' declared here > >>>>>>>>>>> void memory_fence() { > >>>>>>>>>>> ^ > >>>>>>>>>>> > >>>>>>>>>> -- Scott Kruger Tech-X Corporation kruger(a)txcorp.com 5621 Arapahoe Ave, Suite A Phone: (720) 466-3196 Boulder, CO 80303 Fax: (303) 448-7756

1 0

Re: [petsc-dev] Kokkos make error on Spock
by Mark Adams 19 Jul '21

19 Jul '21

Thanks, but this happens w/o Kokkos. I've stripped this down and attached good/bad logs without/with Fortran bindings. Hope this helps, Thanks again, Mark On Sun, Jul 18, 2021 at 12:00 PM Stefano Zampini <stefano.zampini(a)gmail.com> wrote: > This is probably kokkos pulling in the dependencies where compiling kokkos > source within PETSc. > > Il Dom 18 Lug 2021, 16:29 Mark Adams <mfadams(a)lbl.gov> ha scritto: > >> Whoops, this error was just from not telling gfortran to allow long lines. >> >> Anway, I did find that the when fortran bindings are enabled this OMP >> thing gets into the library. No idea how. I'll askm ORNL. >> >> 10:26 jczhang/fix-cray-mpicxx-includes/main= >> /gpfs/alpine/csc314/scratch/adams/petsc2$ nm >> /gpfs/alpine/csc314/scratch/adams/petsc2/arch-spock-opt-cray-kokkos/lib/libpetsc.so >> |g offload >> U .omp_offloading.img_cache.cray_amdgcn-amd-amdhsa >> U .omp_offloading.img_size.cray_amdgcn-amd-amdhsa >> U .omp_offloading.img_start.cray_amdgcn-amd-amdhsa >> 0000000001d457b0 T vecgetoffloadmask_ >> >> >> On Sun, Jul 18, 2021 at 8:43 AM Mark Adams <mfadams(a)lbl.gov> wrote: >> >>> Ah, your test was not on Spock ... I have something working but this is >>> strange. >>> >>> I switched to the GNU ProgEnv. and it passed the Fortran test in >>> 'check', but this failed: >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> *08:26 2 jczhang/fix-cray-mpicxx-includes/main= >>> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials$ make >>> PETSC_DIR=/gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-dbg-gnu >>> PETSC_ARCH="" ex40f90ftn -L/sw/spock/spack-envs/views/rocm-4.1.0/lib >>> -lamdhip64 -lhsa-runtime64 -fPIC -g -O0 -fPIC -g -O0 >>> -I/gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-dbg-gnu/include >>> -I/sw/spock/spack-envs/views/rocm-4.1.0/include ex40f90.F90 >>> -Wl,-rpath,/gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-dbg-gnu/lib >>> -L/gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-dbg-gnu/lib >>> -Wl,-rpath,/sw/spock/spack-envs/views/rocm-4.1.0/lib >>> -L/sw/spock/spack-envs/views/rocm-4.1.0/lib >>> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/ofi/gnu/9.1/lib >>> -L/opt/cray/pe/mpich/8.1.4/ofi/gnu/9.1/lib >>> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/gtl/lib >>> -L/opt/cray/pe/mpich/8.1.4/gtl/lib >>> -Wl,-rpath,/opt/cray/pe/libsci/21.04.1.1/GNU/9.1/x86_64/lib >>> <http://21.04.1.1/GNU/9.1/x86_64/lib> >>> -L/opt/cray/pe/libsci/21.04.1.1/GNU/9.1/x86_64/lib >>> <http://21.04.1.1/GNU/9.1/x86_64/lib> >>> -Wl,-rpath,/opt/cray/pe/pmi/6.0.10/lib -L/opt/cray/pe/pmi/6.0.10/lib >>> -Wl,-rpath,/opt/cray/pe/dsmml/0.1.4/dsmml/lib >>> -L/opt/cray/pe/dsmml/0.1.4/dsmml/lib >>> -Wl,-rpath,/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64 >>> -L/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64 >>> -Wl,-rpath,/opt/gcc/10.2.0/snos/lib/gcc/x86_64-suse-linux/10.2.0 >>> -L/opt/gcc/10.2.0/snos/lib/gcc/x86_64-suse-linux/10.2.0 >>> -Wl,-rpath,/opt/gcc/10.2.0/snos/lib64 -L/opt/gcc/10.2.0/snos/lib64 >>> -Wl,-rpath,/sw/spock/spack-envs/views/rocm-4.1.0/llvm/lib >>> -L/sw/spock/spack-envs/views/rocm-4.1.0/llvm/lib >>> -Wl,-rpath,/opt/gcc/10.2.0/snos/lib -L/opt/gcc/10.2.0/snos/lib -lpetsc >>> -lhipsparse -lhipblas -lrocsparse -lrocsolver -lrocblas -lamdhip64 >>> -lhsa-runtime64 -lstdc++ -ldl -lpmi -lmpi_gtl_hsa -lxpmem -lgfortran -lm >>> -lsci_gnu_82_mpi -lsci_gnu_82 -lmpifort_gnu_91 -lmpi_gnu_91 -ldsmml >>> -lgfortran -lquadmath -lpthread -lm -lgcc_s -lquadmath -lstdc++ -ldl -o >>> ex40f90ex40f90.F90:35:132: 35 | call >>> DMDACreate2d(PETSC_COMM_WORLD,DM_BOUNDARY_NONE,DM_BOUNDARY_NONE,DMDA_STENCIL_BOX,ten,ten,PETSC_DECIDE,PETSC_DECIDE,two,one, >>> & | >>> 1Error: >>> Line truncated at (1) [-Werror=line-truncation]ex40f90.F90:35:132:* >>> * ....* >>> >>> But ex5f (the Fortran check test) does work manually also (sanity check). >>> >>> Oddly I seem to have the cray compilers: >>> >>> 08:34 jczhang/fix-cray-mpicxx-includes/main= >>> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials$ which ftn >>> /opt/cray/pe/craype/2.7.6/bin/ftn >>> >>> yet it is GNU: >>> >>> 08:34 jczhang/fix-cray-mpicxx-includes/main= >>> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials$ ftn --version >>> GNU Fortran (GCC) 10.2.0 20200723 (Cray Inc.) >>> Copyright (C) 2020 Free Software Foundation, Inc. >>> This is free software; see the source for copying conditions. There is >>> NO >>> warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR >>> PURPOSE. >>> >>> The documentation says ftn should be here: >>> >>> 08:34 jczhang/fix-cray-mpicxx-includes/main= >>> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials$ >>> $GCC_PATH/bin/gfortran --version >>> GNU Fortran (GCC) 10.2.0 20200723 (Cray Inc.) >>> Copyright (C) 2020 Free Software Foundation, Inc. >>> This is free software; see the source for copying conditions. There is >>> NO >>> warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR >>> PURPOSE. >>> >>> Same, but the path is not the same: >>> >>> 08:39 jczhang/fix-cray-mpicxx-includes/main= >>> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials$ echo $GCC_PATH >>> /opt/gcc/10.2.0 >>> >>> Anyway, thanks for your help, >>> Mark >>> >>> On Sat, Jul 17, 2021 at 11:26 PM Junchao Zhang <junchao.zhang(a)gmail.com> >>> wrote: >>> >>>> Mark, I can reproduce this error with PrgEnv-cray, i.e., using the >>>> Cray compiler (clang-11). Previously I used PrgEnv-gnu, which did not have >>>> this error. >>>> Probably it is a problem of Spock. But I am not sure. >>>> >>>> --Junchao Zhang >>>> >>>> >>>> On Sat, Jul 17, 2021 at 10:17 AM Mark Adams <mfadams(a)lbl.gov> wrote: >>>> >>>>> And I can run a fortran test, with warnings, but C tests fail: >>>>> >>>>> 11:15 jczhang/fix-cray-mpicxx-includes/main= >>>>> /gpfs/alpine/csc314/scratch/adams/petsc$ make >>>>> PETSC_DIR=/gpfs/alpine/csc314/scratch/adams/petsc >>>>> PETSC_ARCH=arch-spock-opt-cray-kokkos -f gmakefile test >>>>> search='ts_utils_dmplexlandau_tutorials-ex1f90_0' >>>>> Using MAKEFLAGS: -- search=ts_utils_dmplexlandau_tutorials-ex1f90_0 >>>>> PETSC_ARCH=arch-spock-opt-cray-kokkos >>>>> PETSC_DIR=/gpfs/alpine/csc314/scratch/adams/petsc >>>>> FC >>>>> arch-spock-opt-cray-kokkos/tests/ts/utils/dmplexlandau/tutorials/ex1f90.o >>>>> FLINKER >>>>> arch-spock-opt-cray-kokkos/tests/ts/utils/dmplexlandau/tutorials/ex1f90 >>>>> /opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/bin/ld: >>>>> warning: alignment 128 of symbol >>>>> `$host_init$$runtime_init_for_iso_c_binding$iso_c_binding_' in >>>>> /opt/cray/pe/cce/11.0.4/cce/x86_64/lib/libmodules.so is smaller than 256 in >>>>> arch-spock-opt-cray-kokkos/tests/ts/utils/dmplexlandau/tutorials/ex1f90.o >>>>> /opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/bin/ld: >>>>> warning: alignment 64 of symbol `$data_init$iso_c_binding_' in >>>>> /opt/cray/pe/cce/11.0.4/cce/x86_64/lib/libmodules.so is smaller than 256 in >>>>> arch-spock-opt-cray-kokkos/tests/ts/utils/dmplexlandau/tutorials/ex1f90.o >>>>> TEST >>>>> arch-spock-opt-cray-kokkos/tests/counts/ts_utils_dmplexlandau_tutorials-ex1f90_0.counts >>>>> ok ts_utils_dmplexlandau_tutorials-ex1f90_0 >>>>> ok diff-ts_utils_dmplexlandau_tutorials-ex1f90_0 >>>>> >>>>> >>>>> On Sat, Jul 17, 2021 at 10:53 AM Mark Adams <mfadams(a)lbl.gov> wrote: >>>>> >>>>>> HUmm, I can not reproduce this. >>>>>> >>>>>> 10:49 jczhang/fix-cray-mpicxx-includes/main= >>>>>> /gpfs/alpine/csc314/scratch/adams/petsc$ make >>>>>> PETSC_DIR=/gpfs/alpine/csc314/scratch/adams/petsc >>>>>> PETSC_ARCH=arch-spock-dbg-kokkos check >>>>>> Running check examples to verify correct installation >>>>>> Using PETSC_DIR=/gpfs/alpine/csc314/scratch/adams/petsc and >>>>>> PETSC_ARCH=arch-spock-dbg-kokkos >>>>>> gmake[3]: >>>>>> [/gpfs/alpine/csc314/scratch/adams/petsc/lib/petsc/conf/rules:301: >>>>>> ex19.PETSc] Error 2 (ignored) >>>>>> *******************Error detected during compile or >>>>>> link!******************* >>>>>> See http://www.mcs.anl.gov/petsc/documentation/faq.html >>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials ex19 >>>>>> >>>>>> ********************************************************************************* >>>>>> cc -L/sw/spock/spack-envs/views/rocm-4.1.0/lib -lamdhip64 >>>>>> -lhsa-runtime64 -fPIC -Wall -Wwrite-strings -Wno-strict-aliasing >>>>>> -Wno-unknown-pragmas -fstack-protector -Qunused-arguments >>>>>> -fvisibility=hidden -g -O0 -fPIC -Wall -Wwrite-strings >>>>>> -Wno-strict-aliasing -Wno-unknown-pragmas -fstack-protector >>>>>> -Qunused-arguments -fvisibility=hidden -g -O0 >>>>>> -I/sw/spock/spack-envs/views/rocm-4.1.0/include >>>>>> -I/gpfs/alpine/csc314/scratch/adams/petsc/include >>>>>> -I/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/include >>>>>> -I/sw/spock/spack-envs/views/rocm-4.1.0/include >>>>>> -I/sw/spock/spack-envs/views/rocm-4.1.0/include ex19.c >>>>>> -Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib >>>>>> -L/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib >>>>>> -Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib >>>>>> -L/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib >>>>>> -Wl,-rpath,/sw/spock/spack-envs/views/rocm-4.1.0/lib >>>>>> -L/sw/spock/spack-envs/views/rocm-4.1.0/lib >>>>>> -Wl,-rpath,/opt/gcc/8.1.0/snos/lib64 -L/opt/gcc/8.1.0/snos/lib64 >>>>>> -Wl,-rpath,/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib >>>>>> -L/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib >>>>>> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib >>>>>> -L/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib >>>>>> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/gtl/lib >>>>>> -L/opt/cray/pe/mpich/8.1.4/gtl/lib -Wl,-rpath,/opt/cray/pe/pmi/6.0.10/lib >>>>>> -L/opt/cray/pe/pmi/6.0.10/lib -Wl,-rpath,/opt/cray/pe/dsmml/0.1.4/dsmml/lib >>>>>> -L/opt/cray/pe/dsmml/0.1.4/dsmml/lib >>>>>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce/x86_64/lib >>>>>> -L/opt/cray/pe/cce/11.0.4/cce/x86_64/lib >>>>>> -Wl,-rpath,/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64 >>>>>> -L/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64 >>>>>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux >>>>>> -L/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux >>>>>> -Wl,-rpath,/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0 >>>>>> -L/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0 >>>>>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib >>>>>> -L/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib >>>>>> -lpetsc -lkokkoskernels -lkokkoscontainers -lkokkoscore -lhipsparse >>>>>> -lhipblas -lrocsparse -lrocsolver -lrocblas -lamdhip64 -lhsa-runtime64 >>>>>> -lstdc++ -ldl -lpmi -lsci_cray_mpi -lsci_cray -lmpifort_cray -lmpi_cray >>>>>> -lmpi_gtl_hsa -lxpmem -ldsmml -lpgas-shmem -lquadmath -lcrayacc_amdgpu >>>>>> -lopenacc -lmodules -lfi -lcraymath -lf -lu -lcsup -lgfortran -lpthread >>>>>> -lgcc_eh -lm -lclang_rt.craypgo-x86_64 -lclang_rt.builtins-x86_64 >>>>>> -lquadmath -lstdc++ -ldl -o ex19 >>>>>> ld.lld: error: >>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib/libpetsc.so: >>>>>> undefined reference to .omp_offloading.img_start.cray_amdgcn-amd-amdhsa >>>>>> [--no-allow-shlib-undefined] >>>>>> ld.lld: error: >>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib/libpetsc.so: >>>>>> undefined reference to .omp_offloading.img_size.cray_amdgcn-amd-amdhsa >>>>>> [--no-allow-shlib-undefined] >>>>>> ld.lld: error: >>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib/libpetsc.so: >>>>>> undefined reference to .omp_offloading.img_cache.cray_amdgcn-amd-amdhsa >>>>>> [--no-allow-shlib-undefined] >>>>>> clang-11: error: linker command failed with exit code 1 (use -v to >>>>>> see invocation) >>>>>> gmake[4]: *** [<builtin>: ex19] Error 1 >>>>>> gmake[3]: >>>>>> [/gpfs/alpine/csc314/scratch/adams/petsc/lib/petsc/conf/rules:301: >>>>>> ex3k.PETSc] Error 2 (ignored) >>>>>> *******************Error detected during compile or >>>>>> link!******************* >>>>>> See http://www.mcs.anl.gov/petsc/documentation/faq.html >>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials ex3k >>>>>> >>>>>> ********************************************************************************* >>>>>> hipcc -fPIC -Wall -Wwrite-strings -Wno-strict-aliasing >>>>>> -Wno-unknown-pragmas -fvisibility=hidden -g -O0 -std=c++14 >>>>>> -I/gpfs/alpine/csc314/scratch/adams/petsc/include >>>>>> -I/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/include >>>>>> -I/sw/spock/spack-envs/views/rocm-4.1.0/include >>>>>> -I/sw/spock/spack-envs/views/rocm-4.1.0/include -I/opt/cray/pe/libsci/ >>>>>> 21.04.1.1/CRAY/9.0/x86_64/include >>>>>> -I/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/include >>>>>> -I/opt/cray/pe/pmi/6.0.10/include -I/opt/cray/pe/dsmml/0.1.4/dsmml//include >>>>>> -I/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/include >>>>>> -L/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib >>>>>> -L/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib >>>>>> -L/opt/cray/pe/mpich/8.1.4/gtl/lib -L/opt/cray/pe/pmi/6.0.10/lib >>>>>> -L/opt/cray/pe/dsmml/0.1.4/dsmml//lib >>>>>> -L/opt/cray/pe/cce/11.0.4/cce/x86_64/lib/pkgconfig/../ >>>>>> -L/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64 >>>>>> -Wl,--as-needed,-lsci_cray_mpi,--no-as-needed >>>>>> -Wl,--as-needed,-lsci_cray,--no-as-needed >>>>>> -Wl,--as-needed,-lmpi_cray,--no-as-needed >>>>>> -Wl,--as-needed,-ldsmml,--no-as-needed >>>>>> -Wl,--as-needed,-lpgas-shmem,--no-as-needed -lpmi -ldl -lmpi_gtl_hsa >>>>>> -lxpmem -lquadmath -lmodules -lfi -lcraymath -lf -lu -lcsup >>>>>> ex3k.kokkos.cxx >>>>>> -Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib >>>>>> -L/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib >>>>>> -Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib >>>>>> -L/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib >>>>>> -Wl,-rpath,/sw/spock/spack-envs/views/rocm-4.1.0/lib >>>>>> -L/sw/spock/spack-envs/views/rocm-4.1.0/lib >>>>>> -Wl,-rpath,/opt/gcc/8.1.0/snos/lib64 -L/opt/gcc/8.1.0/snos/lib64 >>>>>> -Wl,-rpath,/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib >>>>>> -L/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib >>>>>> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib >>>>>> -L/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib >>>>>> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/gtl/lib >>>>>> -L/opt/cray/pe/mpich/8.1.4/gtl/lib -Wl,-rpath,/opt/cray/pe/pmi/6.0.10/lib >>>>>> -L/opt/cray/pe/pmi/6.0.10/lib -Wl,-rpath,/opt/cray/pe/dsmml/0.1.4/dsmml/lib >>>>>> -L/opt/cray/pe/dsmml/0.1.4/dsmml/lib >>>>>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce/x86_64/lib >>>>>> -L/opt/cray/pe/cce/11.0.4/cce/x86_64/lib >>>>>> -Wl,-rpath,/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64 >>>>>> -L/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64 >>>>>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux >>>>>> -L/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux >>>>>> -Wl,-rpath,/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0 >>>>>> -L/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0 >>>>>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib >>>>>> -L/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib >>>>>> -lpetsc -lkokkoskernels -lkokkoscontainers -lkokkoscore -lhipsparse >>>>>> -lhipblas -lrocsparse -lrocsolver -lrocblas -lamdhip64 -lhsa-runtime64 >>>>>> -lstdc++ -ldl -lpmi -lsci_cray_mpi -lsci_cray -lmpifort_cray -lmpi_cray >>>>>> -lmpi_gtl_hsa -lxpmem -ldsmml -lpgas-shmem -lquadmath -lcrayacc_amdgpu >>>>>> -lopenacc -lmodules -lfi -lcraymath -lf -lu -lcsup -lgfortran -lpthread >>>>>> -lgcc_eh -lm -lclang_rt.craypgo-x86_64 -lclang_rt.builtins-x86_64 >>>>>> -lquadmath -lstdc++ -ldl -o ex3k >>>>>> /usr/lib64/gcc/x86_64-suse-linux/7/../../../../x86_64-suse-linux/bin/ld: >>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib/libpetsc.so: >>>>>> undefined reference to `.omp_offloading.img_size.cray_amdgcn-amd-amdhsa' >>>>>> /usr/lib64/gcc/x86_64-suse-linux/7/../../../../x86_64-suse-linux/bin/ld: >>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib/libpetsc.so: >>>>>> undefined reference to `.omp_offloading.img_start.cray_amdgcn-amd-amdhsa' >>>>>> /usr/lib64/gcc/x86_64-suse-linux/7/../../../../x86_64-suse-linux/bin/ld: >>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib/libpetsc.so: >>>>>> undefined reference to `.omp_offloading.img_cache.cray_amdgcn-amd-amdhsa' >>>>>> clang-12: error: linker command failed with exit code 1 (use -v to >>>>>> see invocation) >>>>>> gmake[4]: *** >>>>>> [/gpfs/alpine/csc314/scratch/adams/petsc/lib/petsc/conf/test:26: ex3k] >>>>>> Error 1 >>>>>> gmake[3]: [makefile:102: runex3k_kokkos] Error 2 (ignored) >>>>>> 1,25c1,4 >>>>>> < atol=1e-50, rtol=1e-08, stol=1e-08, maxit=50, maxf=10000 >>>>>> < Vec Object: Exact Solution 2 MPI processes >>>>>> < type: mpikokkos >>>>>> < Process [0] >>>>>> < 0. >>>>>> < 0.015625 >>>>>> < 0.125 >>>>>> < Process [1] >>>>>> < 0.421875 >>>>>> < 1. >>>>>> < Vec Object: Forcing function 2 MPI processes >>>>>> < type: mpikokkos >>>>>> < Process [0] >>>>>> < 1e-72 >>>>>> < 1.50024 >>>>>> < 3.01563 >>>>>> < Process [1] >>>>>> < 4.67798 >>>>>> < 7. >>>>>> < 0 SNES Function norm 5.414682427127e+00 >>>>>> < 1 SNES Function norm 2.952582418265e-01 >>>>>> < 2 SNES Function norm 4.502293658739e-04 >>>>>> < 3 SNES Function norm 1.389665806646e-09 >>>>>> < Number of SNES iterations = 3 >>>>>> < Norm of error 1.49752e-10 Iterations 3 >>>>>> --- >>>>>> > slurmstepd: error: execve(): >>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials/./ex3k: No such >>>>>> file or directory >>>>>> > slurmstepd: error: execve(): >>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials/./ex3k: No such >>>>>> file or directory >>>>>> > srun: error: spock01: tasks 0-1: Exited with exit code 2 >>>>>> > srun: launch/slurm: _step_signal: Terminating StepId=272979.0 >>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials >>>>>> Possible problem with ex3k running with kokkos-kernels, diffs above >>>>>> ========================================= >>>>>> *******************Error detected during compile or >>>>>> link!******************* >>>>>> See http://www.mcs.anl.gov/petsc/documentation/faq.html >>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials ex5f >>>>>> ********************************************************* >>>>>> ftn -L/sw/spock/spack-envs/views/rocm-4.1.0/lib -lamdhip64 >>>>>> -lhsa-runtime64 -fPIC -g -O0 -fPIC -g -O0 >>>>>> -I/gpfs/alpine/csc314/scratch/adams/petsc/include >>>>>> -I/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/include >>>>>> -I/sw/spock/spack-envs/views/rocm-4.1.0/include ex5f.F90 >>>>>> -Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib >>>>>> -L/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib >>>>>> -Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib >>>>>> -L/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib >>>>>> -Wl,-rpath,/sw/spock/spack-envs/views/rocm-4.1.0/lib >>>>>> -L/sw/spock/spack-envs/views/rocm-4.1.0/lib >>>>>> -Wl,-rpath,/opt/gcc/8.1.0/snos/lib64 -L/opt/gcc/8.1.0/snos/lib64 >>>>>> -Wl,-rpath,/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib >>>>>> -L/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib >>>>>> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib >>>>>> -L/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib >>>>>> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/gtl/lib >>>>>> -L/opt/cray/pe/mpich/8.1.4/gtl/lib -Wl,-rpath,/opt/cray/pe/pmi/6.0.10/lib >>>>>> -L/opt/cray/pe/pmi/6.0.10/lib -Wl,-rpath,/opt/cray/pe/dsmml/0.1.4/dsmml/lib >>>>>> -L/opt/cray/pe/dsmml/0.1.4/dsmml/lib >>>>>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce/x86_64/lib >>>>>> -L/opt/cray/pe/cce/11.0.4/cce/x86_64/lib >>>>>> -Wl,-rpath,/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64 >>>>>> -L/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64 >>>>>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux >>>>>> -L/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux >>>>>> -Wl,-rpath,/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0 >>>>>> -L/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0 >>>>>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib >>>>>> -L/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib >>>>>> -lpetsc -lkokkoskernels -lkokkoscontainers -lkokkoscore -lhipsparse >>>>>> -lhipblas -lrocsparse -lrocsolver -lrocblas -lamdhip64 -lhsa-runtime64 >>>>>> -lstdc++ -ldl -lpmi -lsci_cray_mpi -lsci_cray -lmpifort_cray -lmpi_cray >>>>>> -lmpi_gtl_hsa -lxpmem -ldsmml -lpgas-shmem -lquadmath -lcrayacc_amdgpu >>>>>> -lopenacc -lmodules -lfi -lcraymath -lf -lu -lcsup -lgfortran -lpthread >>>>>> -lgcc_eh -lm -lclang_rt.craypgo-x86_64 -lclang_rt.builtins-x86_64 >>>>>> -lquadmath -lstdc++ -ldl -o ex5f >>>>>> /opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/bin/ld: >>>>>> warning: alignment 128 of symbol >>>>>> `$host_init$$runtime_init_for_iso_c_binding$iso_c_binding_' in >>>>>> /opt/cray/pe/cce/11.0.4/cce/x86_64/lib/libmodules.so is smaller than 256 in >>>>>> /tmp/pe_30429/ex5f_1.o >>>>>> /opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/bin/ld: >>>>>> warning: alignment 64 of symbol `$data_init$iso_c_binding_' in >>>>>> /opt/cray/pe/cce/11.0.4/cce/x86_64/lib/libmodules.so is smaller than 256 in >>>>>> /tmp/pe_30429/ex5f_1.o >>>>>> Fortran example src/snes/tutorials/ex5f run successfully with 1 MPI >>>>>> process >>>>>> Completed test examples >>>>>> 10:49 jczhang/fix-cray-mpicxx-includes/main= >>>>>> /gpfs/alpine/csc314/scratch/adams/petsc$ ll >>>>>> >>>>>> On Fri, Jul 16, 2021 at 11:28 PM Junchao Zhang < >>>>>> junchao.zhang(a)gmail.com> wrote: >>>>>> >>>>>>> Mark, I configured with Fortran binding enabled with main + MR !4175 >>>>>>> <https://gitlab.com/petsc/petsc/-/merge_requests/4175> >>>>>>> >>>>>>> '--with-mpiexec=srun', >>>>>>> '--with-shared-libraries=1', >>>>>>> '--with-cc=cc', >>>>>>> '--with-cxx=CC', >>>>>>> '--with-fc=ftn', >>>>>>> '--with-fortran-bindings', >>>>>>> '--with-hip', >>>>>>> '--with-hipc=hipcc', >>>>>>> '--with-debugging', >>>>>>> '--CPPFLAGS=-I${ROCM_PATH}/include', >>>>>>> '--CXXPPFLAGS=-I${ROCM_PATH}/include', >>>>>>> '--CC_LINKER_FLAGS=-L${ROCM_PATH}/lib -lamdhip64 -lhsa-runtime64', >>>>>>> '--CXX_LINKER_FLAGS=-L${ROCM_PATH}/lib -lamdhip64 -lhsa-runtime64', >>>>>>> '--FC_LINKER_FLAGS=-L${ROCM_PATH}/lib -lamdhip64 -lhsa-runtime64', >>>>>>> '--COPTFLAGS=-g -O0', >>>>>>> '--CXXOPTFLAGS=-g -O0', >>>>>>> '--FOPTFLAGS=-g -O0', >>>>>>> '--download-kokkos', >>>>>>> '--download-kokkos-kernels', >>>>>>> '--download-kokkos-commit=3.4.01', >>>>>>> '--download-kokkos-kernels-commit=3.4.01', >>>>>>> '--with-kokkos-hip-arch=VEGA908', >>>>>>> >>>>>>> and 'make check' ran smoothly on a compute node >>>>>>> >>>>>>> $ make check >>>>>>> Running check examples to verify correct installation >>>>>>> Using PETSC_DIR=/ccs/home/jczhang/petsc and >>>>>>> PETSC_ARCH=arch-spock-cray-kokkos-dbg >>>>>>> C/C++ example src/snes/tutorials/ex19 run successfully with 1 MPI >>>>>>> process >>>>>>> C/C++ example src/snes/tutorials/ex19 run successfully with 2 MPI >>>>>>> processes >>>>>>> C/C++ example src/snes/tutorials/ex3k run successfully with >>>>>>> kokkos-kernels >>>>>>> Fortran example src/snes/tutorials/ex5f run successfully with 1 MPI >>>>>>> process >>>>>>> >>>>>>> Completed test examples >>>>>>> >>>>>>> >>>>>>> --Junchao Zhang >>>>>>> >>>>>>> >>>>>>> On Fri, Jul 16, 2021 at 6:04 PM Mark Adams <mfadams(a)lbl.gov> wrote: >>>>>>> >>>>>>>> And I find that this error, on non-Kokkos C tests, is fixed by >>>>>>>> turning the fortran bindings off: >>>>>>>> >>>>>>>> ld.lld: error: >>>>>>>> /gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/lib/libpetsc.so: >>>>>>>> undefined reference to .omp_offloading.img_start.cray_amdgcn-amd-amdhsa >>>>>>>> [--no-allow-shlib-undefined] >>>>>>>> ld.lld: error: >>>>>>>> /gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/lib/libpetsc.so: >>>>>>>> undefined reference to .omp_offloading.img_size.cray_amdgcn-amd-amdhsa >>>>>>>> [--no-allow-shlib-undefined] >>>>>>>> ld.lld: error: >>>>>>>> /gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/lib/libpetsc.so: >>>>>>>> undefined reference to .omp_offloading.img_cache.cray_amdgcn-amd-amdhsa >>>>>>>> [--no-allow-shlib-undefined] >>>>>>>> >>>>>>>> On Fri, Jul 16, 2021 at 3:53 PM Mark Adams <mfadams(a)lbl.gov> wrote: >>>>>>>> >>>>>>>>> Not complex. THis has some overlap with my problem w/o Kokkos. >>>>>>>>> >>>>>>>>> On Fri, Jul 16, 2021 at 12:54 PM Junchao Zhang < >>>>>>>>> junchao.zhang(a)gmail.com> wrote: >>>>>>>>> >>>>>>>>>> Do you use complex? post your configure.log. >>>>>>>>>> >>>>>>>>>> --Junchao Zhang >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Fri, Jul 16, 2021 at 9:47 AM Mark Adams <mfadams(a)lbl.gov> >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>>> The simple Kokkos example is failing for me on Spock. >>>>>>>>>>> Any ideas? >>>>>>>>>>> Thanks, >>>>>>>>>>> >>>>>>>>>>> 10:44 main *= >>>>>>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials$ make >>>>>>>>>>> PETSC_DIR=/gpfs/alpine/csc314/scratch/adams/petsc >>>>>>>>>>> PETSC_ARCH=arch-spock-opt-cray-kokkos ex3k >>>>>>>>>>> MPICH_CXX="hipcc" OMPI_CXX="hipcc" CC >>>>>>>>>>> -L/sw/spock/spack-envs/views/rocm-4.1.0/lib -lhsa-runtime64 >>>>>>>>>>> -L/sw/spock/spack-envs/views/rocm-4.1.0/lib -lamdhip64 -lhsa-runtime64 >>>>>>>>>>> -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas >>>>>>>>>>> -fstack-protector -fvisibility=hidden -g -O2 -fPIC -Wall -Wwrite-strings >>>>>>>>>>> -Wno-strict-aliasing -Wno-unknown-pragmas -fvisibility=hidden -g -O3 >>>>>>>>>>> -std=c++14 -I/gpfs/alpine/csc314/scratch/adams/petsc/include >>>>>>>>>>> -I/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-opt-cray-kokkos/include >>>>>>>>>>> -I/gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include >>>>>>>>>>> -I/sw/spock/spack-envs/views/rocm-4.1.0/include >>>>>>>>>>> -I/sw/spock/spack-envs/views/rocm-4.1.0/include ex3k.kokkos.cxx >>>>>>>>>>> -Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-opt-cray-kokkos/lib >>>>>>>>>>> -L/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-opt-cray-kokkos/lib >>>>>>>>>>> -Wl,-rpath,/gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/lib >>>>>>>>>>> -L/gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/lib >>>>>>>>>>> -Wl,-rpath,/sw/spock/spack-envs/views/rocm-4.1.0/lib >>>>>>>>>>> -L/sw/spock/spack-envs/views/rocm-4.1.0/lib >>>>>>>>>>> -Wl,-rpath,/opt/gcc/8.1.0/snos/lib64 -L/opt/gcc/8.1.0/snos/lib64 >>>>>>>>>>> -Wl,-rpath,/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib >>>>>>>>>>> -L/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib >>>>>>>>>>> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib >>>>>>>>>>> -L/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib >>>>>>>>>>> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/gtl/lib >>>>>>>>>>> -L/opt/cray/pe/mpich/8.1.4/gtl/lib -Wl,-rpath,/opt/cray/pe/pmi/6.0.10/lib >>>>>>>>>>> -L/opt/cray/pe/pmi/6.0.10/lib -Wl,-rpath,/opt/cray/pe/dsmml/0.1.4/dsmml/lib >>>>>>>>>>> -L/opt/cray/pe/dsmml/0.1.4/dsmml/lib >>>>>>>>>>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce/x86_64/lib >>>>>>>>>>> -L/opt/cray/pe/cce/11.0.4/cce/x86_64/lib >>>>>>>>>>> -Wl,-rpath,/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64 >>>>>>>>>>> -L/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64 >>>>>>>>>>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux >>>>>>>>>>> -L/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux >>>>>>>>>>> -Wl,-rpath,/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0 >>>>>>>>>>> -L/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0 >>>>>>>>>>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib >>>>>>>>>>> -L/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib >>>>>>>>>>> -lpetsc -lkokkoskernels -lkokkoscontainers -lkokkoscore -lhipsparse >>>>>>>>>>> -lhipblas -lrocsparse -lrocsolver -lrocblas -lamdhip64 -lhsa-runtime64 >>>>>>>>>>> -lstdc++ -ldl -lpmi -lsci_cray_mpi -lsci_cray -lmpifort_cray -lmpi_cray >>>>>>>>>>> -lmpi_gtl_hsa -lxpmem -ldsmml -lpgas-shmem -lquadmath -lcrayacc_amdgpu >>>>>>>>>>> -lopenacc -lmodules -lfi -lcraymath -lf -lu -lcsup -lgfortran -lpthread >>>>>>>>>>> -lgcc_eh -lm -lclang_rt.craypgo-x86_64 -lclang_rt.builtins-x86_64 >>>>>>>>>>> -lquadmath -lstdc++ -ldl -o ex3k >>>>>>>>>>> In file included from ex3k.kokkos.cxx:3: >>>>>>>>>>> In file included from >>>>>>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscdmda_kokkos.hpp:4: >>>>>>>>>>> In file included from >>>>>>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec_kokkos.hpp:14: >>>>>>>>>>> In file included from >>>>>>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec.h:9: >>>>>>>>>>> In file included from >>>>>>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsys.h:42: >>>>>>>>>>> In file included from >>>>>>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsystypes.h:255: >>>>>>>>>>> In file included from >>>>>>>>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Complex.hpp:47: >>>>>>>>>>> In file included from >>>>>>>>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Atomic.hpp:212: >>>>>>>>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/impl/Kokkos_Atomic_Exchange.hpp:259:11: >>>>>>>>>>> error: no member named 'lock_address_host_space' in namespace >>>>>>>>>>> 'Kokkos::Impl::Kokkos::Impl'; did you mean simply 'lock_address_host_space'? >>>>>>>>>>> while (!Impl::lock_address_host_space((void*)dest)) >>>>>>>>>>> ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~ >>>>>>>>>>> lock_address_host_space >>>>>>>>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_HostSpace.hpp:83:6: >>>>>>>>>>> note: 'lock_address_host_space' declared here >>>>>>>>>>> bool lock_address_host_space(void* ptr); >>>>>>>>>>> ^ >>>>>>>>>>> In file included from ex3k.kokkos.cxx:3: >>>>>>>>>>> In file included from >>>>>>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscdmda_kokkos.hpp:4: >>>>>>>>>>> In file included from >>>>>>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec_kokkos.hpp:14: >>>>>>>>>>> In file included from >>>>>>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec.h:9: >>>>>>>>>>> In file included from >>>>>>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsys.h:42: >>>>>>>>>>> In file included from >>>>>>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsystypes.h:255: >>>>>>>>>>> In file included from >>>>>>>>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Complex.hpp:47: >>>>>>>>>>> In file included from >>>>>>>>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Atomic.hpp:212: >>>>>>>>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/impl/Kokkos_Atomic_Exchange.hpp:261:3: >>>>>>>>>>> error: no member named 'memory_fence' in namespace 'Kokkos::Impl::Kokkos'; >>>>>>>>>>> did you mean simply 'memory_fence'? >>>>>>>>>>> Kokkos::memory_fence(); >>>>>>>>>>> ^~~~~~~~~~~~~~~~~~~~ >>>>>>>>>>> memory_fence >>>>>>>>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/impl/Kokkos_Memory_Fence.hpp:53:6: >>>>>>>>>>> note: 'memory_fence' declared here >>>>>>>>>>> void memory_fence() { >>>>>>>>>>> ^ >>>>>>>>>>> In file included from ex3k.kokkos.cxx:3: >>>>>>>>>>> In file included from >>>>>>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscdmda_kokkos.hpp:4: >>>>>>>>>>> In file included from >>>>>>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec_kokkos.hpp:14: >>>>>>>>>>> In file included from >>>>>>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec.h:9: >>>>>>>>>>> In file included from >>>>>>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsys.h:42: >>>>>>>>>>> In file included from >>>>>>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsystypes.h:255: >>>>>>>>>>> In file included from >>>>>>>>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Complex.hpp:47: >>>>>>>>>>> In file included from >>>>>>>>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Atomic.hpp:212: >>>>>>>>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/impl/Kokkos_Atomic_Exchange.hpp:277:3: >>>>>>>>>>> error: no member named 'memory_fence' in namespace 'Kokkos::Impl::Kokkos'; >>>>>>>>>>> did you mean simply 'memory_fence'? >>>>>>>>>>> Kokkos::memory_fence(); >>>>>>>>>>> ^~~~~~~~~~~~~~~~~~~~ >>>>>>>>>>> memory_fence >>>>>>>>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/impl/Kokkos_Memory_Fence.hpp:53:6: >>>>>>>>>>> note: 'memory_fence' declared here >>>>>>>>>>> void memory_fence() { >>>>>>>>>>> ^ >>>>>>>>>>> >>>>>>>>>>

1 0

Re: [petsc-dev] Kokkos make error on Spock
by Stefano Zampini 18 Jul '21

18 Jul '21

This is probably kokkos pulling in the dependencies where compiling kokkos source within PETSc. Il Dom 18 Lug 2021, 16:29 Mark Adams <mfadams(a)lbl.gov> ha scritto: > Whoops, this error was just from not telling gfortran to allow long lines. > > Anway, I did find that the when fortran bindings are enabled this OMP > thing gets into the library. No idea how. I'll askm ORNL. > > 10:26 jczhang/fix-cray-mpicxx-includes/main= > /gpfs/alpine/csc314/scratch/adams/petsc2$ nm > /gpfs/alpine/csc314/scratch/adams/petsc2/arch-spock-opt-cray-kokkos/lib/libpetsc.so > |g offload > U .omp_offloading.img_cache.cray_amdgcn-amd-amdhsa > U .omp_offloading.img_size.cray_amdgcn-amd-amdhsa > U .omp_offloading.img_start.cray_amdgcn-amd-amdhsa > 0000000001d457b0 T vecgetoffloadmask_ > > > On Sun, Jul 18, 2021 at 8:43 AM Mark Adams <mfadams(a)lbl.gov> wrote: > >> Ah, your test was not on Spock ... I have something working but this is >> strange. >> >> I switched to the GNU ProgEnv. and it passed the Fortran test in 'check', >> but this failed: >> >> >> >> >> >> >> >> >> >> *08:26 2 jczhang/fix-cray-mpicxx-includes/main= >> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials$ make >> PETSC_DIR=/gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-dbg-gnu >> PETSC_ARCH="" ex40f90ftn -L/sw/spock/spack-envs/views/rocm-4.1.0/lib >> -lamdhip64 -lhsa-runtime64 -fPIC -g -O0 -fPIC -g -O0 >> -I/gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-dbg-gnu/include >> -I/sw/spock/spack-envs/views/rocm-4.1.0/include ex40f90.F90 >> -Wl,-rpath,/gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-dbg-gnu/lib >> -L/gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-dbg-gnu/lib >> -Wl,-rpath,/sw/spock/spack-envs/views/rocm-4.1.0/lib >> -L/sw/spock/spack-envs/views/rocm-4.1.0/lib >> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/ofi/gnu/9.1/lib >> -L/opt/cray/pe/mpich/8.1.4/ofi/gnu/9.1/lib >> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/gtl/lib >> -L/opt/cray/pe/mpich/8.1.4/gtl/lib >> -Wl,-rpath,/opt/cray/pe/libsci/21.04.1.1/GNU/9.1/x86_64/lib >> <http://21.04.1.1/GNU/9.1/x86_64/lib> >> -L/opt/cray/pe/libsci/21.04.1.1/GNU/9.1/x86_64/lib >> <http://21.04.1.1/GNU/9.1/x86_64/lib> >> -Wl,-rpath,/opt/cray/pe/pmi/6.0.10/lib -L/opt/cray/pe/pmi/6.0.10/lib >> -Wl,-rpath,/opt/cray/pe/dsmml/0.1.4/dsmml/lib >> -L/opt/cray/pe/dsmml/0.1.4/dsmml/lib >> -Wl,-rpath,/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64 >> -L/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64 >> -Wl,-rpath,/opt/gcc/10.2.0/snos/lib/gcc/x86_64-suse-linux/10.2.0 >> -L/opt/gcc/10.2.0/snos/lib/gcc/x86_64-suse-linux/10.2.0 >> -Wl,-rpath,/opt/gcc/10.2.0/snos/lib64 -L/opt/gcc/10.2.0/snos/lib64 >> -Wl,-rpath,/sw/spock/spack-envs/views/rocm-4.1.0/llvm/lib >> -L/sw/spock/spack-envs/views/rocm-4.1.0/llvm/lib >> -Wl,-rpath,/opt/gcc/10.2.0/snos/lib -L/opt/gcc/10.2.0/snos/lib -lpetsc >> -lhipsparse -lhipblas -lrocsparse -lrocsolver -lrocblas -lamdhip64 >> -lhsa-runtime64 -lstdc++ -ldl -lpmi -lmpi_gtl_hsa -lxpmem -lgfortran -lm >> -lsci_gnu_82_mpi -lsci_gnu_82 -lmpifort_gnu_91 -lmpi_gnu_91 -ldsmml >> -lgfortran -lquadmath -lpthread -lm -lgcc_s -lquadmath -lstdc++ -ldl -o >> ex40f90ex40f90.F90:35:132: 35 | call >> DMDACreate2d(PETSC_COMM_WORLD,DM_BOUNDARY_NONE,DM_BOUNDARY_NONE,DMDA_STENCIL_BOX,ten,ten,PETSC_DECIDE,PETSC_DECIDE,two,one, >> & | >> 1Error: >> Line truncated at (1) [-Werror=line-truncation]ex40f90.F90:35:132:* >> * ....* >> >> But ex5f (the Fortran check test) does work manually also (sanity check). >> >> Oddly I seem to have the cray compilers: >> >> 08:34 jczhang/fix-cray-mpicxx-includes/main= >> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials$ which ftn >> /opt/cray/pe/craype/2.7.6/bin/ftn >> >> yet it is GNU: >> >> 08:34 jczhang/fix-cray-mpicxx-includes/main= >> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials$ ftn --version >> GNU Fortran (GCC) 10.2.0 20200723 (Cray Inc.) >> Copyright (C) 2020 Free Software Foundation, Inc. >> This is free software; see the source for copying conditions. There is NO >> warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR >> PURPOSE. >> >> The documentation says ftn should be here: >> >> 08:34 jczhang/fix-cray-mpicxx-includes/main= >> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials$ >> $GCC_PATH/bin/gfortran --version >> GNU Fortran (GCC) 10.2.0 20200723 (Cray Inc.) >> Copyright (C) 2020 Free Software Foundation, Inc. >> This is free software; see the source for copying conditions. There is NO >> warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR >> PURPOSE. >> >> Same, but the path is not the same: >> >> 08:39 jczhang/fix-cray-mpicxx-includes/main= >> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials$ echo $GCC_PATH >> /opt/gcc/10.2.0 >> >> Anyway, thanks for your help, >> Mark >> >> On Sat, Jul 17, 2021 at 11:26 PM Junchao Zhang <junchao.zhang(a)gmail.com> >> wrote: >> >>> Mark, I can reproduce this error with PrgEnv-cray, i.e., using the Cray >>> compiler (clang-11). Previously I used PrgEnv-gnu, which did not have this >>> error. >>> Probably it is a problem of Spock. But I am not sure. >>> >>> --Junchao Zhang >>> >>> >>> On Sat, Jul 17, 2021 at 10:17 AM Mark Adams <mfadams(a)lbl.gov> wrote: >>> >>>> And I can run a fortran test, with warnings, but C tests fail: >>>> >>>> 11:15 jczhang/fix-cray-mpicxx-includes/main= >>>> /gpfs/alpine/csc314/scratch/adams/petsc$ make >>>> PETSC_DIR=/gpfs/alpine/csc314/scratch/adams/petsc >>>> PETSC_ARCH=arch-spock-opt-cray-kokkos -f gmakefile test >>>> search='ts_utils_dmplexlandau_tutorials-ex1f90_0' >>>> Using MAKEFLAGS: -- search=ts_utils_dmplexlandau_tutorials-ex1f90_0 >>>> PETSC_ARCH=arch-spock-opt-cray-kokkos >>>> PETSC_DIR=/gpfs/alpine/csc314/scratch/adams/petsc >>>> FC >>>> arch-spock-opt-cray-kokkos/tests/ts/utils/dmplexlandau/tutorials/ex1f90.o >>>> FLINKER >>>> arch-spock-opt-cray-kokkos/tests/ts/utils/dmplexlandau/tutorials/ex1f90 >>>> /opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/bin/ld: >>>> warning: alignment 128 of symbol >>>> `$host_init$$runtime_init_for_iso_c_binding$iso_c_binding_' in >>>> /opt/cray/pe/cce/11.0.4/cce/x86_64/lib/libmodules.so is smaller than 256 in >>>> arch-spock-opt-cray-kokkos/tests/ts/utils/dmplexlandau/tutorials/ex1f90.o >>>> /opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/bin/ld: >>>> warning: alignment 64 of symbol `$data_init$iso_c_binding_' in >>>> /opt/cray/pe/cce/11.0.4/cce/x86_64/lib/libmodules.so is smaller than 256 in >>>> arch-spock-opt-cray-kokkos/tests/ts/utils/dmplexlandau/tutorials/ex1f90.o >>>> TEST >>>> arch-spock-opt-cray-kokkos/tests/counts/ts_utils_dmplexlandau_tutorials-ex1f90_0.counts >>>> ok ts_utils_dmplexlandau_tutorials-ex1f90_0 >>>> ok diff-ts_utils_dmplexlandau_tutorials-ex1f90_0 >>>> >>>> >>>> On Sat, Jul 17, 2021 at 10:53 AM Mark Adams <mfadams(a)lbl.gov> wrote: >>>> >>>>> HUmm, I can not reproduce this. >>>>> >>>>> 10:49 jczhang/fix-cray-mpicxx-includes/main= >>>>> /gpfs/alpine/csc314/scratch/adams/petsc$ make >>>>> PETSC_DIR=/gpfs/alpine/csc314/scratch/adams/petsc >>>>> PETSC_ARCH=arch-spock-dbg-kokkos check >>>>> Running check examples to verify correct installation >>>>> Using PETSC_DIR=/gpfs/alpine/csc314/scratch/adams/petsc and >>>>> PETSC_ARCH=arch-spock-dbg-kokkos >>>>> gmake[3]: >>>>> [/gpfs/alpine/csc314/scratch/adams/petsc/lib/petsc/conf/rules:301: >>>>> ex19.PETSc] Error 2 (ignored) >>>>> *******************Error detected during compile or >>>>> link!******************* >>>>> See http://www.mcs.anl.gov/petsc/documentation/faq.html >>>>> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials ex19 >>>>> >>>>> ********************************************************************************* >>>>> cc -L/sw/spock/spack-envs/views/rocm-4.1.0/lib -lamdhip64 >>>>> -lhsa-runtime64 -fPIC -Wall -Wwrite-strings -Wno-strict-aliasing >>>>> -Wno-unknown-pragmas -fstack-protector -Qunused-arguments >>>>> -fvisibility=hidden -g -O0 -fPIC -Wall -Wwrite-strings >>>>> -Wno-strict-aliasing -Wno-unknown-pragmas -fstack-protector >>>>> -Qunused-arguments -fvisibility=hidden -g -O0 >>>>> -I/sw/spock/spack-envs/views/rocm-4.1.0/include >>>>> -I/gpfs/alpine/csc314/scratch/adams/petsc/include >>>>> -I/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/include >>>>> -I/sw/spock/spack-envs/views/rocm-4.1.0/include >>>>> -I/sw/spock/spack-envs/views/rocm-4.1.0/include ex19.c >>>>> -Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib >>>>> -L/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib >>>>> -Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib >>>>> -L/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib >>>>> -Wl,-rpath,/sw/spock/spack-envs/views/rocm-4.1.0/lib >>>>> -L/sw/spock/spack-envs/views/rocm-4.1.0/lib >>>>> -Wl,-rpath,/opt/gcc/8.1.0/snos/lib64 -L/opt/gcc/8.1.0/snos/lib64 >>>>> -Wl,-rpath,/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib >>>>> -L/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib >>>>> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib >>>>> -L/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib >>>>> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/gtl/lib >>>>> -L/opt/cray/pe/mpich/8.1.4/gtl/lib -Wl,-rpath,/opt/cray/pe/pmi/6.0.10/lib >>>>> -L/opt/cray/pe/pmi/6.0.10/lib -Wl,-rpath,/opt/cray/pe/dsmml/0.1.4/dsmml/lib >>>>> -L/opt/cray/pe/dsmml/0.1.4/dsmml/lib >>>>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce/x86_64/lib >>>>> -L/opt/cray/pe/cce/11.0.4/cce/x86_64/lib >>>>> -Wl,-rpath,/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64 >>>>> -L/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64 >>>>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux >>>>> -L/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux >>>>> -Wl,-rpath,/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0 >>>>> -L/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0 >>>>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib >>>>> -L/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib >>>>> -lpetsc -lkokkoskernels -lkokkoscontainers -lkokkoscore -lhipsparse >>>>> -lhipblas -lrocsparse -lrocsolver -lrocblas -lamdhip64 -lhsa-runtime64 >>>>> -lstdc++ -ldl -lpmi -lsci_cray_mpi -lsci_cray -lmpifort_cray -lmpi_cray >>>>> -lmpi_gtl_hsa -lxpmem -ldsmml -lpgas-shmem -lquadmath -lcrayacc_amdgpu >>>>> -lopenacc -lmodules -lfi -lcraymath -lf -lu -lcsup -lgfortran -lpthread >>>>> -lgcc_eh -lm -lclang_rt.craypgo-x86_64 -lclang_rt.builtins-x86_64 >>>>> -lquadmath -lstdc++ -ldl -o ex19 >>>>> ld.lld: error: >>>>> /gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib/libpetsc.so: >>>>> undefined reference to .omp_offloading.img_start.cray_amdgcn-amd-amdhsa >>>>> [--no-allow-shlib-undefined] >>>>> ld.lld: error: >>>>> /gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib/libpetsc.so: >>>>> undefined reference to .omp_offloading.img_size.cray_amdgcn-amd-amdhsa >>>>> [--no-allow-shlib-undefined] >>>>> ld.lld: error: >>>>> /gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib/libpetsc.so: >>>>> undefined reference to .omp_offloading.img_cache.cray_amdgcn-amd-amdhsa >>>>> [--no-allow-shlib-undefined] >>>>> clang-11: error: linker command failed with exit code 1 (use -v to see >>>>> invocation) >>>>> gmake[4]: *** [<builtin>: ex19] Error 1 >>>>> gmake[3]: >>>>> [/gpfs/alpine/csc314/scratch/adams/petsc/lib/petsc/conf/rules:301: >>>>> ex3k.PETSc] Error 2 (ignored) >>>>> *******************Error detected during compile or >>>>> link!******************* >>>>> See http://www.mcs.anl.gov/petsc/documentation/faq.html >>>>> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials ex3k >>>>> >>>>> ********************************************************************************* >>>>> hipcc -fPIC -Wall -Wwrite-strings -Wno-strict-aliasing >>>>> -Wno-unknown-pragmas -fvisibility=hidden -g -O0 -std=c++14 >>>>> -I/gpfs/alpine/csc314/scratch/adams/petsc/include >>>>> -I/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/include >>>>> -I/sw/spock/spack-envs/views/rocm-4.1.0/include >>>>> -I/sw/spock/spack-envs/views/rocm-4.1.0/include -I/opt/cray/pe/libsci/ >>>>> 21.04.1.1/CRAY/9.0/x86_64/include >>>>> -I/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/include >>>>> -I/opt/cray/pe/pmi/6.0.10/include -I/opt/cray/pe/dsmml/0.1.4/dsmml//include >>>>> -I/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/include >>>>> -L/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib >>>>> -L/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib >>>>> -L/opt/cray/pe/mpich/8.1.4/gtl/lib -L/opt/cray/pe/pmi/6.0.10/lib >>>>> -L/opt/cray/pe/dsmml/0.1.4/dsmml//lib >>>>> -L/opt/cray/pe/cce/11.0.4/cce/x86_64/lib/pkgconfig/../ >>>>> -L/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64 >>>>> -Wl,--as-needed,-lsci_cray_mpi,--no-as-needed >>>>> -Wl,--as-needed,-lsci_cray,--no-as-needed >>>>> -Wl,--as-needed,-lmpi_cray,--no-as-needed >>>>> -Wl,--as-needed,-ldsmml,--no-as-needed >>>>> -Wl,--as-needed,-lpgas-shmem,--no-as-needed -lpmi -ldl -lmpi_gtl_hsa >>>>> -lxpmem -lquadmath -lmodules -lfi -lcraymath -lf -lu -lcsup >>>>> ex3k.kokkos.cxx >>>>> -Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib >>>>> -L/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib >>>>> -Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib >>>>> -L/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib >>>>> -Wl,-rpath,/sw/spock/spack-envs/views/rocm-4.1.0/lib >>>>> -L/sw/spock/spack-envs/views/rocm-4.1.0/lib >>>>> -Wl,-rpath,/opt/gcc/8.1.0/snos/lib64 -L/opt/gcc/8.1.0/snos/lib64 >>>>> -Wl,-rpath,/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib >>>>> -L/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib >>>>> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib >>>>> -L/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib >>>>> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/gtl/lib >>>>> -L/opt/cray/pe/mpich/8.1.4/gtl/lib -Wl,-rpath,/opt/cray/pe/pmi/6.0.10/lib >>>>> -L/opt/cray/pe/pmi/6.0.10/lib -Wl,-rpath,/opt/cray/pe/dsmml/0.1.4/dsmml/lib >>>>> -L/opt/cray/pe/dsmml/0.1.4/dsmml/lib >>>>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce/x86_64/lib >>>>> -L/opt/cray/pe/cce/11.0.4/cce/x86_64/lib >>>>> -Wl,-rpath,/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64 >>>>> -L/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64 >>>>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux >>>>> -L/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux >>>>> -Wl,-rpath,/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0 >>>>> -L/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0 >>>>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib >>>>> -L/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib >>>>> -lpetsc -lkokkoskernels -lkokkoscontainers -lkokkoscore -lhipsparse >>>>> -lhipblas -lrocsparse -lrocsolver -lrocblas -lamdhip64 -lhsa-runtime64 >>>>> -lstdc++ -ldl -lpmi -lsci_cray_mpi -lsci_cray -lmpifort_cray -lmpi_cray >>>>> -lmpi_gtl_hsa -lxpmem -ldsmml -lpgas-shmem -lquadmath -lcrayacc_amdgpu >>>>> -lopenacc -lmodules -lfi -lcraymath -lf -lu -lcsup -lgfortran -lpthread >>>>> -lgcc_eh -lm -lclang_rt.craypgo-x86_64 -lclang_rt.builtins-x86_64 >>>>> -lquadmath -lstdc++ -ldl -o ex3k >>>>> /usr/lib64/gcc/x86_64-suse-linux/7/../../../../x86_64-suse-linux/bin/ld: >>>>> /gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib/libpetsc.so: >>>>> undefined reference to `.omp_offloading.img_size.cray_amdgcn-amd-amdhsa' >>>>> /usr/lib64/gcc/x86_64-suse-linux/7/../../../../x86_64-suse-linux/bin/ld: >>>>> /gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib/libpetsc.so: >>>>> undefined reference to `.omp_offloading.img_start.cray_amdgcn-amd-amdhsa' >>>>> /usr/lib64/gcc/x86_64-suse-linux/7/../../../../x86_64-suse-linux/bin/ld: >>>>> /gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib/libpetsc.so: >>>>> undefined reference to `.omp_offloading.img_cache.cray_amdgcn-amd-amdhsa' >>>>> clang-12: error: linker command failed with exit code 1 (use -v to see >>>>> invocation) >>>>> gmake[4]: *** >>>>> [/gpfs/alpine/csc314/scratch/adams/petsc/lib/petsc/conf/test:26: ex3k] >>>>> Error 1 >>>>> gmake[3]: [makefile:102: runex3k_kokkos] Error 2 (ignored) >>>>> 1,25c1,4 >>>>> < atol=1e-50, rtol=1e-08, stol=1e-08, maxit=50, maxf=10000 >>>>> < Vec Object: Exact Solution 2 MPI processes >>>>> < type: mpikokkos >>>>> < Process [0] >>>>> < 0. >>>>> < 0.015625 >>>>> < 0.125 >>>>> < Process [1] >>>>> < 0.421875 >>>>> < 1. >>>>> < Vec Object: Forcing function 2 MPI processes >>>>> < type: mpikokkos >>>>> < Process [0] >>>>> < 1e-72 >>>>> < 1.50024 >>>>> < 3.01563 >>>>> < Process [1] >>>>> < 4.67798 >>>>> < 7. >>>>> < 0 SNES Function norm 5.414682427127e+00 >>>>> < 1 SNES Function norm 2.952582418265e-01 >>>>> < 2 SNES Function norm 4.502293658739e-04 >>>>> < 3 SNES Function norm 1.389665806646e-09 >>>>> < Number of SNES iterations = 3 >>>>> < Norm of error 1.49752e-10 Iterations 3 >>>>> --- >>>>> > slurmstepd: error: execve(): >>>>> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials/./ex3k: No such >>>>> file or directory >>>>> > slurmstepd: error: execve(): >>>>> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials/./ex3k: No such >>>>> file or directory >>>>> > srun: error: spock01: tasks 0-1: Exited with exit code 2 >>>>> > srun: launch/slurm: _step_signal: Terminating StepId=272979.0 >>>>> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials >>>>> Possible problem with ex3k running with kokkos-kernels, diffs above >>>>> ========================================= >>>>> *******************Error detected during compile or >>>>> link!******************* >>>>> See http://www.mcs.anl.gov/petsc/documentation/faq.html >>>>> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials ex5f >>>>> ********************************************************* >>>>> ftn -L/sw/spock/spack-envs/views/rocm-4.1.0/lib -lamdhip64 >>>>> -lhsa-runtime64 -fPIC -g -O0 -fPIC -g -O0 >>>>> -I/gpfs/alpine/csc314/scratch/adams/petsc/include >>>>> -I/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/include >>>>> -I/sw/spock/spack-envs/views/rocm-4.1.0/include ex5f.F90 >>>>> -Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib >>>>> -L/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib >>>>> -Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib >>>>> -L/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib >>>>> -Wl,-rpath,/sw/spock/spack-envs/views/rocm-4.1.0/lib >>>>> -L/sw/spock/spack-envs/views/rocm-4.1.0/lib >>>>> -Wl,-rpath,/opt/gcc/8.1.0/snos/lib64 -L/opt/gcc/8.1.0/snos/lib64 >>>>> -Wl,-rpath,/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib >>>>> -L/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib >>>>> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib >>>>> -L/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib >>>>> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/gtl/lib >>>>> -L/opt/cray/pe/mpich/8.1.4/gtl/lib -Wl,-rpath,/opt/cray/pe/pmi/6.0.10/lib >>>>> -L/opt/cray/pe/pmi/6.0.10/lib -Wl,-rpath,/opt/cray/pe/dsmml/0.1.4/dsmml/lib >>>>> -L/opt/cray/pe/dsmml/0.1.4/dsmml/lib >>>>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce/x86_64/lib >>>>> -L/opt/cray/pe/cce/11.0.4/cce/x86_64/lib >>>>> -Wl,-rpath,/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64 >>>>> -L/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64 >>>>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux >>>>> -L/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux >>>>> -Wl,-rpath,/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0 >>>>> -L/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0 >>>>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib >>>>> -L/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib >>>>> -lpetsc -lkokkoskernels -lkokkoscontainers -lkokkoscore -lhipsparse >>>>> -lhipblas -lrocsparse -lrocsolver -lrocblas -lamdhip64 -lhsa-runtime64 >>>>> -lstdc++ -ldl -lpmi -lsci_cray_mpi -lsci_cray -lmpifort_cray -lmpi_cray >>>>> -lmpi_gtl_hsa -lxpmem -ldsmml -lpgas-shmem -lquadmath -lcrayacc_amdgpu >>>>> -lopenacc -lmodules -lfi -lcraymath -lf -lu -lcsup -lgfortran -lpthread >>>>> -lgcc_eh -lm -lclang_rt.craypgo-x86_64 -lclang_rt.builtins-x86_64 >>>>> -lquadmath -lstdc++ -ldl -o ex5f >>>>> /opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/bin/ld: >>>>> warning: alignment 128 of symbol >>>>> `$host_init$$runtime_init_for_iso_c_binding$iso_c_binding_' in >>>>> /opt/cray/pe/cce/11.0.4/cce/x86_64/lib/libmodules.so is smaller than 256 in >>>>> /tmp/pe_30429/ex5f_1.o >>>>> /opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/bin/ld: >>>>> warning: alignment 64 of symbol `$data_init$iso_c_binding_' in >>>>> /opt/cray/pe/cce/11.0.4/cce/x86_64/lib/libmodules.so is smaller than 256 in >>>>> /tmp/pe_30429/ex5f_1.o >>>>> Fortran example src/snes/tutorials/ex5f run successfully with 1 MPI >>>>> process >>>>> Completed test examples >>>>> 10:49 jczhang/fix-cray-mpicxx-includes/main= >>>>> /gpfs/alpine/csc314/scratch/adams/petsc$ ll >>>>> >>>>> On Fri, Jul 16, 2021 at 11:28 PM Junchao Zhang < >>>>> junchao.zhang(a)gmail.com> wrote: >>>>> >>>>>> Mark, I configured with Fortran binding enabled with main + MR !4175 >>>>>> <https://gitlab.com/petsc/petsc/-/merge_requests/4175> >>>>>> >>>>>> '--with-mpiexec=srun', >>>>>> '--with-shared-libraries=1', >>>>>> '--with-cc=cc', >>>>>> '--with-cxx=CC', >>>>>> '--with-fc=ftn', >>>>>> '--with-fortran-bindings', >>>>>> '--with-hip', >>>>>> '--with-hipc=hipcc', >>>>>> '--with-debugging', >>>>>> '--CPPFLAGS=-I${ROCM_PATH}/include', >>>>>> '--CXXPPFLAGS=-I${ROCM_PATH}/include', >>>>>> '--CC_LINKER_FLAGS=-L${ROCM_PATH}/lib -lamdhip64 -lhsa-runtime64', >>>>>> '--CXX_LINKER_FLAGS=-L${ROCM_PATH}/lib -lamdhip64 -lhsa-runtime64', >>>>>> '--FC_LINKER_FLAGS=-L${ROCM_PATH}/lib -lamdhip64 -lhsa-runtime64', >>>>>> '--COPTFLAGS=-g -O0', >>>>>> '--CXXOPTFLAGS=-g -O0', >>>>>> '--FOPTFLAGS=-g -O0', >>>>>> '--download-kokkos', >>>>>> '--download-kokkos-kernels', >>>>>> '--download-kokkos-commit=3.4.01', >>>>>> '--download-kokkos-kernels-commit=3.4.01', >>>>>> '--with-kokkos-hip-arch=VEGA908', >>>>>> >>>>>> and 'make check' ran smoothly on a compute node >>>>>> >>>>>> $ make check >>>>>> Running check examples to verify correct installation >>>>>> Using PETSC_DIR=/ccs/home/jczhang/petsc and >>>>>> PETSC_ARCH=arch-spock-cray-kokkos-dbg >>>>>> C/C++ example src/snes/tutorials/ex19 run successfully with 1 MPI >>>>>> process >>>>>> C/C++ example src/snes/tutorials/ex19 run successfully with 2 MPI >>>>>> processes >>>>>> C/C++ example src/snes/tutorials/ex3k run successfully with >>>>>> kokkos-kernels >>>>>> Fortran example src/snes/tutorials/ex5f run successfully with 1 MPI >>>>>> process >>>>>> >>>>>> Completed test examples >>>>>> >>>>>> >>>>>> --Junchao Zhang >>>>>> >>>>>> >>>>>> On Fri, Jul 16, 2021 at 6:04 PM Mark Adams <mfadams(a)lbl.gov> wrote: >>>>>> >>>>>>> And I find that this error, on non-Kokkos C tests, is fixed by >>>>>>> turning the fortran bindings off: >>>>>>> >>>>>>> ld.lld: error: >>>>>>> /gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/lib/libpetsc.so: >>>>>>> undefined reference to .omp_offloading.img_start.cray_amdgcn-amd-amdhsa >>>>>>> [--no-allow-shlib-undefined] >>>>>>> ld.lld: error: >>>>>>> /gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/lib/libpetsc.so: >>>>>>> undefined reference to .omp_offloading.img_size.cray_amdgcn-amd-amdhsa >>>>>>> [--no-allow-shlib-undefined] >>>>>>> ld.lld: error: >>>>>>> /gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/lib/libpetsc.so: >>>>>>> undefined reference to .omp_offloading.img_cache.cray_amdgcn-amd-amdhsa >>>>>>> [--no-allow-shlib-undefined] >>>>>>> >>>>>>> On Fri, Jul 16, 2021 at 3:53 PM Mark Adams <mfadams(a)lbl.gov> wrote: >>>>>>> >>>>>>>> Not complex. THis has some overlap with my problem w/o Kokkos. >>>>>>>> >>>>>>>> On Fri, Jul 16, 2021 at 12:54 PM Junchao Zhang < >>>>>>>> junchao.zhang(a)gmail.com> wrote: >>>>>>>> >>>>>>>>> Do you use complex? post your configure.log. >>>>>>>>> >>>>>>>>> --Junchao Zhang >>>>>>>>> >>>>>>>>> >>>>>>>>> On Fri, Jul 16, 2021 at 9:47 AM Mark Adams <mfadams(a)lbl.gov> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> The simple Kokkos example is failing for me on Spock. >>>>>>>>>> Any ideas? >>>>>>>>>> Thanks, >>>>>>>>>> >>>>>>>>>> 10:44 main *= >>>>>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials$ make >>>>>>>>>> PETSC_DIR=/gpfs/alpine/csc314/scratch/adams/petsc >>>>>>>>>> PETSC_ARCH=arch-spock-opt-cray-kokkos ex3k >>>>>>>>>> MPICH_CXX="hipcc" OMPI_CXX="hipcc" CC >>>>>>>>>> -L/sw/spock/spack-envs/views/rocm-4.1.0/lib -lhsa-runtime64 >>>>>>>>>> -L/sw/spock/spack-envs/views/rocm-4.1.0/lib -lamdhip64 -lhsa-runtime64 >>>>>>>>>> -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas >>>>>>>>>> -fstack-protector -fvisibility=hidden -g -O2 -fPIC -Wall -Wwrite-strings >>>>>>>>>> -Wno-strict-aliasing -Wno-unknown-pragmas -fvisibility=hidden -g -O3 >>>>>>>>>> -std=c++14 -I/gpfs/alpine/csc314/scratch/adams/petsc/include >>>>>>>>>> -I/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-opt-cray-kokkos/include >>>>>>>>>> -I/gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include >>>>>>>>>> -I/sw/spock/spack-envs/views/rocm-4.1.0/include >>>>>>>>>> -I/sw/spock/spack-envs/views/rocm-4.1.0/include ex3k.kokkos.cxx >>>>>>>>>> -Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-opt-cray-kokkos/lib >>>>>>>>>> -L/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-opt-cray-kokkos/lib >>>>>>>>>> -Wl,-rpath,/gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/lib >>>>>>>>>> -L/gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/lib >>>>>>>>>> -Wl,-rpath,/sw/spock/spack-envs/views/rocm-4.1.0/lib >>>>>>>>>> -L/sw/spock/spack-envs/views/rocm-4.1.0/lib >>>>>>>>>> -Wl,-rpath,/opt/gcc/8.1.0/snos/lib64 -L/opt/gcc/8.1.0/snos/lib64 >>>>>>>>>> -Wl,-rpath,/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib >>>>>>>>>> -L/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib >>>>>>>>>> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib >>>>>>>>>> -L/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib >>>>>>>>>> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/gtl/lib >>>>>>>>>> -L/opt/cray/pe/mpich/8.1.4/gtl/lib -Wl,-rpath,/opt/cray/pe/pmi/6.0.10/lib >>>>>>>>>> -L/opt/cray/pe/pmi/6.0.10/lib -Wl,-rpath,/opt/cray/pe/dsmml/0.1.4/dsmml/lib >>>>>>>>>> -L/opt/cray/pe/dsmml/0.1.4/dsmml/lib >>>>>>>>>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce/x86_64/lib >>>>>>>>>> -L/opt/cray/pe/cce/11.0.4/cce/x86_64/lib >>>>>>>>>> -Wl,-rpath,/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64 >>>>>>>>>> -L/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64 >>>>>>>>>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux >>>>>>>>>> -L/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux >>>>>>>>>> -Wl,-rpath,/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0 >>>>>>>>>> -L/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0 >>>>>>>>>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib >>>>>>>>>> -L/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib >>>>>>>>>> -lpetsc -lkokkoskernels -lkokkoscontainers -lkokkoscore -lhipsparse >>>>>>>>>> -lhipblas -lrocsparse -lrocsolver -lrocblas -lamdhip64 -lhsa-runtime64 >>>>>>>>>> -lstdc++ -ldl -lpmi -lsci_cray_mpi -lsci_cray -lmpifort_cray -lmpi_cray >>>>>>>>>> -lmpi_gtl_hsa -lxpmem -ldsmml -lpgas-shmem -lquadmath -lcrayacc_amdgpu >>>>>>>>>> -lopenacc -lmodules -lfi -lcraymath -lf -lu -lcsup -lgfortran -lpthread >>>>>>>>>> -lgcc_eh -lm -lclang_rt.craypgo-x86_64 -lclang_rt.builtins-x86_64 >>>>>>>>>> -lquadmath -lstdc++ -ldl -o ex3k >>>>>>>>>> In file included from ex3k.kokkos.cxx:3: >>>>>>>>>> In file included from >>>>>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscdmda_kokkos.hpp:4: >>>>>>>>>> In file included from >>>>>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec_kokkos.hpp:14: >>>>>>>>>> In file included from >>>>>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec.h:9: >>>>>>>>>> In file included from >>>>>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsys.h:42: >>>>>>>>>> In file included from >>>>>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsystypes.h:255: >>>>>>>>>> In file included from >>>>>>>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Complex.hpp:47: >>>>>>>>>> In file included from >>>>>>>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Atomic.hpp:212: >>>>>>>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/impl/Kokkos_Atomic_Exchange.hpp:259:11: >>>>>>>>>> error: no member named 'lock_address_host_space' in namespace >>>>>>>>>> 'Kokkos::Impl::Kokkos::Impl'; did you mean simply 'lock_address_host_space'? >>>>>>>>>> while (!Impl::lock_address_host_space((void*)dest)) >>>>>>>>>> ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~ >>>>>>>>>> lock_address_host_space >>>>>>>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_HostSpace.hpp:83:6: >>>>>>>>>> note: 'lock_address_host_space' declared here >>>>>>>>>> bool lock_address_host_space(void* ptr); >>>>>>>>>> ^ >>>>>>>>>> In file included from ex3k.kokkos.cxx:3: >>>>>>>>>> In file included from >>>>>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscdmda_kokkos.hpp:4: >>>>>>>>>> In file included from >>>>>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec_kokkos.hpp:14: >>>>>>>>>> In file included from >>>>>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec.h:9: >>>>>>>>>> In file included from >>>>>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsys.h:42: >>>>>>>>>> In file included from >>>>>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsystypes.h:255: >>>>>>>>>> In file included from >>>>>>>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Complex.hpp:47: >>>>>>>>>> In file included from >>>>>>>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Atomic.hpp:212: >>>>>>>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/impl/Kokkos_Atomic_Exchange.hpp:261:3: >>>>>>>>>> error: no member named 'memory_fence' in namespace 'Kokkos::Impl::Kokkos'; >>>>>>>>>> did you mean simply 'memory_fence'? >>>>>>>>>> Kokkos::memory_fence(); >>>>>>>>>> ^~~~~~~~~~~~~~~~~~~~ >>>>>>>>>> memory_fence >>>>>>>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/impl/Kokkos_Memory_Fence.hpp:53:6: >>>>>>>>>> note: 'memory_fence' declared here >>>>>>>>>> void memory_fence() { >>>>>>>>>> ^ >>>>>>>>>> In file included from ex3k.kokkos.cxx:3: >>>>>>>>>> In file included from >>>>>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscdmda_kokkos.hpp:4: >>>>>>>>>> In file included from >>>>>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec_kokkos.hpp:14: >>>>>>>>>> In file included from >>>>>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec.h:9: >>>>>>>>>> In file included from >>>>>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsys.h:42: >>>>>>>>>> In file included from >>>>>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsystypes.h:255: >>>>>>>>>> In file included from >>>>>>>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Complex.hpp:47: >>>>>>>>>> In file included from >>>>>>>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Atomic.hpp:212: >>>>>>>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/impl/Kokkos_Atomic_Exchange.hpp:277:3: >>>>>>>>>> error: no member named 'memory_fence' in namespace 'Kokkos::Impl::Kokkos'; >>>>>>>>>> did you mean simply 'memory_fence'? >>>>>>>>>> Kokkos::memory_fence(); >>>>>>>>>> ^~~~~~~~~~~~~~~~~~~~ >>>>>>>>>> memory_fence >>>>>>>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/impl/Kokkos_Memory_Fence.hpp:53:6: >>>>>>>>>> note: 'memory_fence' declared here >>>>>>>>>> void memory_fence() { >>>>>>>>>> ^ >>>>>>>>>> >>>>>>>>>

1 0

Re: [petsc-dev] Kokkos make error on Spock
by Mark Adams 18 Jul '21

18 Jul '21

Whoops, this error was just from not telling gfortran to allow long lines. Anway, I did find that the when fortran bindings are enabled this OMP thing gets into the library. No idea how. I'll askm ORNL. 10:26 jczhang/fix-cray-mpicxx-includes/main= /gpfs/alpine/csc314/scratch/adams/petsc2$ nm /gpfs/alpine/csc314/scratch/adams/petsc2/arch-spock-opt-cray-kokkos/lib/libpetsc.so |g offload U .omp_offloading.img_cache.cray_amdgcn-amd-amdhsa U .omp_offloading.img_size.cray_amdgcn-amd-amdhsa U .omp_offloading.img_start.cray_amdgcn-amd-amdhsa 0000000001d457b0 T vecgetoffloadmask_ On Sun, Jul 18, 2021 at 8:43 AM Mark Adams <mfadams(a)lbl.gov> wrote: > Ah, your test was not on Spock ... I have something working but this is > strange. > > I switched to the GNU ProgEnv. and it passed the Fortran test in 'check', > but this failed: > > > > > > > > > > *08:26 2 jczhang/fix-cray-mpicxx-includes/main= > /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials$ make > PETSC_DIR=/gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-dbg-gnu > PETSC_ARCH="" ex40f90ftn -L/sw/spock/spack-envs/views/rocm-4.1.0/lib > -lamdhip64 -lhsa-runtime64 -fPIC -g -O0 -fPIC -g -O0 > -I/gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-dbg-gnu/include > -I/sw/spock/spack-envs/views/rocm-4.1.0/include ex40f90.F90 > -Wl,-rpath,/gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-dbg-gnu/lib > -L/gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-dbg-gnu/lib > -Wl,-rpath,/sw/spock/spack-envs/views/rocm-4.1.0/lib > -L/sw/spock/spack-envs/views/rocm-4.1.0/lib > -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/ofi/gnu/9.1/lib > -L/opt/cray/pe/mpich/8.1.4/ofi/gnu/9.1/lib > -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/gtl/lib > -L/opt/cray/pe/mpich/8.1.4/gtl/lib > -Wl,-rpath,/opt/cray/pe/libsci/21.04.1.1/GNU/9.1/x86_64/lib > <http://21.04.1.1/GNU/9.1/x86_64/lib> > -L/opt/cray/pe/libsci/21.04.1.1/GNU/9.1/x86_64/lib > <http://21.04.1.1/GNU/9.1/x86_64/lib> > -Wl,-rpath,/opt/cray/pe/pmi/6.0.10/lib -L/opt/cray/pe/pmi/6.0.10/lib > -Wl,-rpath,/opt/cray/pe/dsmml/0.1.4/dsmml/lib > -L/opt/cray/pe/dsmml/0.1.4/dsmml/lib > -Wl,-rpath,/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64 > -L/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64 > -Wl,-rpath,/opt/gcc/10.2.0/snos/lib/gcc/x86_64-suse-linux/10.2.0 > -L/opt/gcc/10.2.0/snos/lib/gcc/x86_64-suse-linux/10.2.0 > -Wl,-rpath,/opt/gcc/10.2.0/snos/lib64 -L/opt/gcc/10.2.0/snos/lib64 > -Wl,-rpath,/sw/spock/spack-envs/views/rocm-4.1.0/llvm/lib > -L/sw/spock/spack-envs/views/rocm-4.1.0/llvm/lib > -Wl,-rpath,/opt/gcc/10.2.0/snos/lib -L/opt/gcc/10.2.0/snos/lib -lpetsc > -lhipsparse -lhipblas -lrocsparse -lrocsolver -lrocblas -lamdhip64 > -lhsa-runtime64 -lstdc++ -ldl -lpmi -lmpi_gtl_hsa -lxpmem -lgfortran -lm > -lsci_gnu_82_mpi -lsci_gnu_82 -lmpifort_gnu_91 -lmpi_gnu_91 -ldsmml > -lgfortran -lquadmath -lpthread -lm -lgcc_s -lquadmath -lstdc++ -ldl -o > ex40f90ex40f90.F90:35:132: 35 | call > DMDACreate2d(PETSC_COMM_WORLD,DM_BOUNDARY_NONE,DM_BOUNDARY_NONE,DMDA_STENCIL_BOX,ten,ten,PETSC_DECIDE,PETSC_DECIDE,two,one, > & | > 1Error: > Line truncated at (1) [-Werror=line-truncation]ex40f90.F90:35:132:* > * ....* > > But ex5f (the Fortran check test) does work manually also (sanity check). > > Oddly I seem to have the cray compilers: > > 08:34 jczhang/fix-cray-mpicxx-includes/main= > /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials$ which ftn > /opt/cray/pe/craype/2.7.6/bin/ftn > > yet it is GNU: > > 08:34 jczhang/fix-cray-mpicxx-includes/main= > /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials$ ftn --version > GNU Fortran (GCC) 10.2.0 20200723 (Cray Inc.) > Copyright (C) 2020 Free Software Foundation, Inc. > This is free software; see the source for copying conditions. There is NO > warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. > > The documentation says ftn should be here: > > 08:34 jczhang/fix-cray-mpicxx-includes/main= > /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials$ > $GCC_PATH/bin/gfortran --version > GNU Fortran (GCC) 10.2.0 20200723 (Cray Inc.) > Copyright (C) 2020 Free Software Foundation, Inc. > This is free software; see the source for copying conditions. There is NO > warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. > > Same, but the path is not the same: > > 08:39 jczhang/fix-cray-mpicxx-includes/main= > /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials$ echo $GCC_PATH > /opt/gcc/10.2.0 > > Anyway, thanks for your help, > Mark > > On Sat, Jul 17, 2021 at 11:26 PM Junchao Zhang <junchao.zhang(a)gmail.com> > wrote: > >> Mark, I can reproduce this error with PrgEnv-cray, i.e., using the Cray >> compiler (clang-11). Previously I used PrgEnv-gnu, which did not have this >> error. >> Probably it is a problem of Spock. But I am not sure. >> >> --Junchao Zhang >> >> >> On Sat, Jul 17, 2021 at 10:17 AM Mark Adams <mfadams(a)lbl.gov> wrote: >> >>> And I can run a fortran test, with warnings, but C tests fail: >>> >>> 11:15 jczhang/fix-cray-mpicxx-includes/main= >>> /gpfs/alpine/csc314/scratch/adams/petsc$ make >>> PETSC_DIR=/gpfs/alpine/csc314/scratch/adams/petsc >>> PETSC_ARCH=arch-spock-opt-cray-kokkos -f gmakefile test >>> search='ts_utils_dmplexlandau_tutorials-ex1f90_0' >>> Using MAKEFLAGS: -- search=ts_utils_dmplexlandau_tutorials-ex1f90_0 >>> PETSC_ARCH=arch-spock-opt-cray-kokkos >>> PETSC_DIR=/gpfs/alpine/csc314/scratch/adams/petsc >>> FC >>> arch-spock-opt-cray-kokkos/tests/ts/utils/dmplexlandau/tutorials/ex1f90.o >>> FLINKER >>> arch-spock-opt-cray-kokkos/tests/ts/utils/dmplexlandau/tutorials/ex1f90 >>> /opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/bin/ld: >>> warning: alignment 128 of symbol >>> `$host_init$$runtime_init_for_iso_c_binding$iso_c_binding_' in >>> /opt/cray/pe/cce/11.0.4/cce/x86_64/lib/libmodules.so is smaller than 256 in >>> arch-spock-opt-cray-kokkos/tests/ts/utils/dmplexlandau/tutorials/ex1f90.o >>> /opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/bin/ld: >>> warning: alignment 64 of symbol `$data_init$iso_c_binding_' in >>> /opt/cray/pe/cce/11.0.4/cce/x86_64/lib/libmodules.so is smaller than 256 in >>> arch-spock-opt-cray-kokkos/tests/ts/utils/dmplexlandau/tutorials/ex1f90.o >>> TEST >>> arch-spock-opt-cray-kokkos/tests/counts/ts_utils_dmplexlandau_tutorials-ex1f90_0.counts >>> ok ts_utils_dmplexlandau_tutorials-ex1f90_0 >>> ok diff-ts_utils_dmplexlandau_tutorials-ex1f90_0 >>> >>> >>> On Sat, Jul 17, 2021 at 10:53 AM Mark Adams <mfadams(a)lbl.gov> wrote: >>> >>>> HUmm, I can not reproduce this. >>>> >>>> 10:49 jczhang/fix-cray-mpicxx-includes/main= >>>> /gpfs/alpine/csc314/scratch/adams/petsc$ make >>>> PETSC_DIR=/gpfs/alpine/csc314/scratch/adams/petsc >>>> PETSC_ARCH=arch-spock-dbg-kokkos check >>>> Running check examples to verify correct installation >>>> Using PETSC_DIR=/gpfs/alpine/csc314/scratch/adams/petsc and >>>> PETSC_ARCH=arch-spock-dbg-kokkos >>>> gmake[3]: >>>> [/gpfs/alpine/csc314/scratch/adams/petsc/lib/petsc/conf/rules:301: >>>> ex19.PETSc] Error 2 (ignored) >>>> *******************Error detected during compile or >>>> link!******************* >>>> See http://www.mcs.anl.gov/petsc/documentation/faq.html >>>> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials ex19 >>>> >>>> ********************************************************************************* >>>> cc -L/sw/spock/spack-envs/views/rocm-4.1.0/lib -lamdhip64 >>>> -lhsa-runtime64 -fPIC -Wall -Wwrite-strings -Wno-strict-aliasing >>>> -Wno-unknown-pragmas -fstack-protector -Qunused-arguments >>>> -fvisibility=hidden -g -O0 -fPIC -Wall -Wwrite-strings >>>> -Wno-strict-aliasing -Wno-unknown-pragmas -fstack-protector >>>> -Qunused-arguments -fvisibility=hidden -g -O0 >>>> -I/sw/spock/spack-envs/views/rocm-4.1.0/include >>>> -I/gpfs/alpine/csc314/scratch/adams/petsc/include >>>> -I/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/include >>>> -I/sw/spock/spack-envs/views/rocm-4.1.0/include >>>> -I/sw/spock/spack-envs/views/rocm-4.1.0/include ex19.c >>>> -Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib >>>> -L/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib >>>> -Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib >>>> -L/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib >>>> -Wl,-rpath,/sw/spock/spack-envs/views/rocm-4.1.0/lib >>>> -L/sw/spock/spack-envs/views/rocm-4.1.0/lib >>>> -Wl,-rpath,/opt/gcc/8.1.0/snos/lib64 -L/opt/gcc/8.1.0/snos/lib64 >>>> -Wl,-rpath,/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib >>>> -L/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib >>>> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib >>>> -L/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib >>>> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/gtl/lib >>>> -L/opt/cray/pe/mpich/8.1.4/gtl/lib -Wl,-rpath,/opt/cray/pe/pmi/6.0.10/lib >>>> -L/opt/cray/pe/pmi/6.0.10/lib -Wl,-rpath,/opt/cray/pe/dsmml/0.1.4/dsmml/lib >>>> -L/opt/cray/pe/dsmml/0.1.4/dsmml/lib >>>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce/x86_64/lib >>>> -L/opt/cray/pe/cce/11.0.4/cce/x86_64/lib >>>> -Wl,-rpath,/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64 >>>> -L/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64 >>>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux >>>> -L/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux >>>> -Wl,-rpath,/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0 >>>> -L/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0 >>>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib >>>> -L/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib >>>> -lpetsc -lkokkoskernels -lkokkoscontainers -lkokkoscore -lhipsparse >>>> -lhipblas -lrocsparse -lrocsolver -lrocblas -lamdhip64 -lhsa-runtime64 >>>> -lstdc++ -ldl -lpmi -lsci_cray_mpi -lsci_cray -lmpifort_cray -lmpi_cray >>>> -lmpi_gtl_hsa -lxpmem -ldsmml -lpgas-shmem -lquadmath -lcrayacc_amdgpu >>>> -lopenacc -lmodules -lfi -lcraymath -lf -lu -lcsup -lgfortran -lpthread >>>> -lgcc_eh -lm -lclang_rt.craypgo-x86_64 -lclang_rt.builtins-x86_64 >>>> -lquadmath -lstdc++ -ldl -o ex19 >>>> ld.lld: error: >>>> /gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib/libpetsc.so: >>>> undefined reference to .omp_offloading.img_start.cray_amdgcn-amd-amdhsa >>>> [--no-allow-shlib-undefined] >>>> ld.lld: error: >>>> /gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib/libpetsc.so: >>>> undefined reference to .omp_offloading.img_size.cray_amdgcn-amd-amdhsa >>>> [--no-allow-shlib-undefined] >>>> ld.lld: error: >>>> /gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib/libpetsc.so: >>>> undefined reference to .omp_offloading.img_cache.cray_amdgcn-amd-amdhsa >>>> [--no-allow-shlib-undefined] >>>> clang-11: error: linker command failed with exit code 1 (use -v to see >>>> invocation) >>>> gmake[4]: *** [<builtin>: ex19] Error 1 >>>> gmake[3]: >>>> [/gpfs/alpine/csc314/scratch/adams/petsc/lib/petsc/conf/rules:301: >>>> ex3k.PETSc] Error 2 (ignored) >>>> *******************Error detected during compile or >>>> link!******************* >>>> See http://www.mcs.anl.gov/petsc/documentation/faq.html >>>> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials ex3k >>>> >>>> ********************************************************************************* >>>> hipcc -fPIC -Wall -Wwrite-strings -Wno-strict-aliasing >>>> -Wno-unknown-pragmas -fvisibility=hidden -g -O0 -std=c++14 >>>> -I/gpfs/alpine/csc314/scratch/adams/petsc/include >>>> -I/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/include >>>> -I/sw/spock/spack-envs/views/rocm-4.1.0/include >>>> -I/sw/spock/spack-envs/views/rocm-4.1.0/include -I/opt/cray/pe/libsci/ >>>> 21.04.1.1/CRAY/9.0/x86_64/include >>>> -I/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/include >>>> -I/opt/cray/pe/pmi/6.0.10/include -I/opt/cray/pe/dsmml/0.1.4/dsmml//include >>>> -I/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/include >>>> -L/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib >>>> -L/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib >>>> -L/opt/cray/pe/mpich/8.1.4/gtl/lib -L/opt/cray/pe/pmi/6.0.10/lib >>>> -L/opt/cray/pe/dsmml/0.1.4/dsmml//lib >>>> -L/opt/cray/pe/cce/11.0.4/cce/x86_64/lib/pkgconfig/../ >>>> -L/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64 >>>> -Wl,--as-needed,-lsci_cray_mpi,--no-as-needed >>>> -Wl,--as-needed,-lsci_cray,--no-as-needed >>>> -Wl,--as-needed,-lmpi_cray,--no-as-needed >>>> -Wl,--as-needed,-ldsmml,--no-as-needed >>>> -Wl,--as-needed,-lpgas-shmem,--no-as-needed -lpmi -ldl -lmpi_gtl_hsa >>>> -lxpmem -lquadmath -lmodules -lfi -lcraymath -lf -lu -lcsup >>>> ex3k.kokkos.cxx >>>> -Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib >>>> -L/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib >>>> -Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib >>>> -L/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib >>>> -Wl,-rpath,/sw/spock/spack-envs/views/rocm-4.1.0/lib >>>> -L/sw/spock/spack-envs/views/rocm-4.1.0/lib >>>> -Wl,-rpath,/opt/gcc/8.1.0/snos/lib64 -L/opt/gcc/8.1.0/snos/lib64 >>>> -Wl,-rpath,/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib >>>> -L/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib >>>> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib >>>> -L/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib >>>> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/gtl/lib >>>> -L/opt/cray/pe/mpich/8.1.4/gtl/lib -Wl,-rpath,/opt/cray/pe/pmi/6.0.10/lib >>>> -L/opt/cray/pe/pmi/6.0.10/lib -Wl,-rpath,/opt/cray/pe/dsmml/0.1.4/dsmml/lib >>>> -L/opt/cray/pe/dsmml/0.1.4/dsmml/lib >>>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce/x86_64/lib >>>> -L/opt/cray/pe/cce/11.0.4/cce/x86_64/lib >>>> -Wl,-rpath,/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64 >>>> -L/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64 >>>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux >>>> -L/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux >>>> -Wl,-rpath,/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0 >>>> -L/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0 >>>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib >>>> -L/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib >>>> -lpetsc -lkokkoskernels -lkokkoscontainers -lkokkoscore -lhipsparse >>>> -lhipblas -lrocsparse -lrocsolver -lrocblas -lamdhip64 -lhsa-runtime64 >>>> -lstdc++ -ldl -lpmi -lsci_cray_mpi -lsci_cray -lmpifort_cray -lmpi_cray >>>> -lmpi_gtl_hsa -lxpmem -ldsmml -lpgas-shmem -lquadmath -lcrayacc_amdgpu >>>> -lopenacc -lmodules -lfi -lcraymath -lf -lu -lcsup -lgfortran -lpthread >>>> -lgcc_eh -lm -lclang_rt.craypgo-x86_64 -lclang_rt.builtins-x86_64 >>>> -lquadmath -lstdc++ -ldl -o ex3k >>>> /usr/lib64/gcc/x86_64-suse-linux/7/../../../../x86_64-suse-linux/bin/ld: >>>> /gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib/libpetsc.so: >>>> undefined reference to `.omp_offloading.img_size.cray_amdgcn-amd-amdhsa' >>>> /usr/lib64/gcc/x86_64-suse-linux/7/../../../../x86_64-suse-linux/bin/ld: >>>> /gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib/libpetsc.so: >>>> undefined reference to `.omp_offloading.img_start.cray_amdgcn-amd-amdhsa' >>>> /usr/lib64/gcc/x86_64-suse-linux/7/../../../../x86_64-suse-linux/bin/ld: >>>> /gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib/libpetsc.so: >>>> undefined reference to `.omp_offloading.img_cache.cray_amdgcn-amd-amdhsa' >>>> clang-12: error: linker command failed with exit code 1 (use -v to see >>>> invocation) >>>> gmake[4]: *** >>>> [/gpfs/alpine/csc314/scratch/adams/petsc/lib/petsc/conf/test:26: ex3k] >>>> Error 1 >>>> gmake[3]: [makefile:102: runex3k_kokkos] Error 2 (ignored) >>>> 1,25c1,4 >>>> < atol=1e-50, rtol=1e-08, stol=1e-08, maxit=50, maxf=10000 >>>> < Vec Object: Exact Solution 2 MPI processes >>>> < type: mpikokkos >>>> < Process [0] >>>> < 0. >>>> < 0.015625 >>>> < 0.125 >>>> < Process [1] >>>> < 0.421875 >>>> < 1. >>>> < Vec Object: Forcing function 2 MPI processes >>>> < type: mpikokkos >>>> < Process [0] >>>> < 1e-72 >>>> < 1.50024 >>>> < 3.01563 >>>> < Process [1] >>>> < 4.67798 >>>> < 7. >>>> < 0 SNES Function norm 5.414682427127e+00 >>>> < 1 SNES Function norm 2.952582418265e-01 >>>> < 2 SNES Function norm 4.502293658739e-04 >>>> < 3 SNES Function norm 1.389665806646e-09 >>>> < Number of SNES iterations = 3 >>>> < Norm of error 1.49752e-10 Iterations 3 >>>> --- >>>> > slurmstepd: error: execve(): >>>> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials/./ex3k: No such >>>> file or directory >>>> > slurmstepd: error: execve(): >>>> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials/./ex3k: No such >>>> file or directory >>>> > srun: error: spock01: tasks 0-1: Exited with exit code 2 >>>> > srun: launch/slurm: _step_signal: Terminating StepId=272979.0 >>>> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials >>>> Possible problem with ex3k running with kokkos-kernels, diffs above >>>> ========================================= >>>> *******************Error detected during compile or >>>> link!******************* >>>> See http://www.mcs.anl.gov/petsc/documentation/faq.html >>>> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials ex5f >>>> ********************************************************* >>>> ftn -L/sw/spock/spack-envs/views/rocm-4.1.0/lib -lamdhip64 >>>> -lhsa-runtime64 -fPIC -g -O0 -fPIC -g -O0 >>>> -I/gpfs/alpine/csc314/scratch/adams/petsc/include >>>> -I/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/include >>>> -I/sw/spock/spack-envs/views/rocm-4.1.0/include ex5f.F90 >>>> -Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib >>>> -L/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib >>>> -Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib >>>> -L/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib >>>> -Wl,-rpath,/sw/spock/spack-envs/views/rocm-4.1.0/lib >>>> -L/sw/spock/spack-envs/views/rocm-4.1.0/lib >>>> -Wl,-rpath,/opt/gcc/8.1.0/snos/lib64 -L/opt/gcc/8.1.0/snos/lib64 >>>> -Wl,-rpath,/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib >>>> -L/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib >>>> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib >>>> -L/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib >>>> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/gtl/lib >>>> -L/opt/cray/pe/mpich/8.1.4/gtl/lib -Wl,-rpath,/opt/cray/pe/pmi/6.0.10/lib >>>> -L/opt/cray/pe/pmi/6.0.10/lib -Wl,-rpath,/opt/cray/pe/dsmml/0.1.4/dsmml/lib >>>> -L/opt/cray/pe/dsmml/0.1.4/dsmml/lib >>>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce/x86_64/lib >>>> -L/opt/cray/pe/cce/11.0.4/cce/x86_64/lib >>>> -Wl,-rpath,/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64 >>>> -L/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64 >>>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux >>>> -L/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux >>>> -Wl,-rpath,/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0 >>>> -L/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0 >>>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib >>>> -L/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib >>>> -lpetsc -lkokkoskernels -lkokkoscontainers -lkokkoscore -lhipsparse >>>> -lhipblas -lrocsparse -lrocsolver -lrocblas -lamdhip64 -lhsa-runtime64 >>>> -lstdc++ -ldl -lpmi -lsci_cray_mpi -lsci_cray -lmpifort_cray -lmpi_cray >>>> -lmpi_gtl_hsa -lxpmem -ldsmml -lpgas-shmem -lquadmath -lcrayacc_amdgpu >>>> -lopenacc -lmodules -lfi -lcraymath -lf -lu -lcsup -lgfortran -lpthread >>>> -lgcc_eh -lm -lclang_rt.craypgo-x86_64 -lclang_rt.builtins-x86_64 >>>> -lquadmath -lstdc++ -ldl -o ex5f >>>> /opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/bin/ld: >>>> warning: alignment 128 of symbol >>>> `$host_init$$runtime_init_for_iso_c_binding$iso_c_binding_' in >>>> /opt/cray/pe/cce/11.0.4/cce/x86_64/lib/libmodules.so is smaller than 256 in >>>> /tmp/pe_30429/ex5f_1.o >>>> /opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/bin/ld: >>>> warning: alignment 64 of symbol `$data_init$iso_c_binding_' in >>>> /opt/cray/pe/cce/11.0.4/cce/x86_64/lib/libmodules.so is smaller than 256 in >>>> /tmp/pe_30429/ex5f_1.o >>>> Fortran example src/snes/tutorials/ex5f run successfully with 1 MPI >>>> process >>>> Completed test examples >>>> 10:49 jczhang/fix-cray-mpicxx-includes/main= >>>> /gpfs/alpine/csc314/scratch/adams/petsc$ ll >>>> >>>> On Fri, Jul 16, 2021 at 11:28 PM Junchao Zhang <junchao.zhang(a)gmail.com> >>>> wrote: >>>> >>>>> Mark, I configured with Fortran binding enabled with main + MR !4175 >>>>> <https://gitlab.com/petsc/petsc/-/merge_requests/4175> >>>>> >>>>> '--with-mpiexec=srun', >>>>> '--with-shared-libraries=1', >>>>> '--with-cc=cc', >>>>> '--with-cxx=CC', >>>>> '--with-fc=ftn', >>>>> '--with-fortran-bindings', >>>>> '--with-hip', >>>>> '--with-hipc=hipcc', >>>>> '--with-debugging', >>>>> '--CPPFLAGS=-I${ROCM_PATH}/include', >>>>> '--CXXPPFLAGS=-I${ROCM_PATH}/include', >>>>> '--CC_LINKER_FLAGS=-L${ROCM_PATH}/lib -lamdhip64 -lhsa-runtime64', >>>>> '--CXX_LINKER_FLAGS=-L${ROCM_PATH}/lib -lamdhip64 -lhsa-runtime64', >>>>> '--FC_LINKER_FLAGS=-L${ROCM_PATH}/lib -lamdhip64 -lhsa-runtime64', >>>>> '--COPTFLAGS=-g -O0', >>>>> '--CXXOPTFLAGS=-g -O0', >>>>> '--FOPTFLAGS=-g -O0', >>>>> '--download-kokkos', >>>>> '--download-kokkos-kernels', >>>>> '--download-kokkos-commit=3.4.01', >>>>> '--download-kokkos-kernels-commit=3.4.01', >>>>> '--with-kokkos-hip-arch=VEGA908', >>>>> >>>>> and 'make check' ran smoothly on a compute node >>>>> >>>>> $ make check >>>>> Running check examples to verify correct installation >>>>> Using PETSC_DIR=/ccs/home/jczhang/petsc and >>>>> PETSC_ARCH=arch-spock-cray-kokkos-dbg >>>>> C/C++ example src/snes/tutorials/ex19 run successfully with 1 MPI >>>>> process >>>>> C/C++ example src/snes/tutorials/ex19 run successfully with 2 MPI >>>>> processes >>>>> C/C++ example src/snes/tutorials/ex3k run successfully with >>>>> kokkos-kernels >>>>> Fortran example src/snes/tutorials/ex5f run successfully with 1 MPI >>>>> process >>>>> >>>>> Completed test examples >>>>> >>>>> >>>>> --Junchao Zhang >>>>> >>>>> >>>>> On Fri, Jul 16, 2021 at 6:04 PM Mark Adams <mfadams(a)lbl.gov> wrote: >>>>> >>>>>> And I find that this error, on non-Kokkos C tests, is fixed by >>>>>> turning the fortran bindings off: >>>>>> >>>>>> ld.lld: error: >>>>>> /gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/lib/libpetsc.so: >>>>>> undefined reference to .omp_offloading.img_start.cray_amdgcn-amd-amdhsa >>>>>> [--no-allow-shlib-undefined] >>>>>> ld.lld: error: >>>>>> /gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/lib/libpetsc.so: >>>>>> undefined reference to .omp_offloading.img_size.cray_amdgcn-amd-amdhsa >>>>>> [--no-allow-shlib-undefined] >>>>>> ld.lld: error: >>>>>> /gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/lib/libpetsc.so: >>>>>> undefined reference to .omp_offloading.img_cache.cray_amdgcn-amd-amdhsa >>>>>> [--no-allow-shlib-undefined] >>>>>> >>>>>> On Fri, Jul 16, 2021 at 3:53 PM Mark Adams <mfadams(a)lbl.gov> wrote: >>>>>> >>>>>>> Not complex. THis has some overlap with my problem w/o Kokkos. >>>>>>> >>>>>>> On Fri, Jul 16, 2021 at 12:54 PM Junchao Zhang < >>>>>>> junchao.zhang(a)gmail.com> wrote: >>>>>>> >>>>>>>> Do you use complex? post your configure.log. >>>>>>>> >>>>>>>> --Junchao Zhang >>>>>>>> >>>>>>>> >>>>>>>> On Fri, Jul 16, 2021 at 9:47 AM Mark Adams <mfadams(a)lbl.gov> wrote: >>>>>>>> >>>>>>>>> The simple Kokkos example is failing for me on Spock. >>>>>>>>> Any ideas? >>>>>>>>> Thanks, >>>>>>>>> >>>>>>>>> 10:44 main *= >>>>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials$ make >>>>>>>>> PETSC_DIR=/gpfs/alpine/csc314/scratch/adams/petsc >>>>>>>>> PETSC_ARCH=arch-spock-opt-cray-kokkos ex3k >>>>>>>>> MPICH_CXX="hipcc" OMPI_CXX="hipcc" CC >>>>>>>>> -L/sw/spock/spack-envs/views/rocm-4.1.0/lib -lhsa-runtime64 >>>>>>>>> -L/sw/spock/spack-envs/views/rocm-4.1.0/lib -lamdhip64 -lhsa-runtime64 >>>>>>>>> -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas >>>>>>>>> -fstack-protector -fvisibility=hidden -g -O2 -fPIC -Wall -Wwrite-strings >>>>>>>>> -Wno-strict-aliasing -Wno-unknown-pragmas -fvisibility=hidden -g -O3 >>>>>>>>> -std=c++14 -I/gpfs/alpine/csc314/scratch/adams/petsc/include >>>>>>>>> -I/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-opt-cray-kokkos/include >>>>>>>>> -I/gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include >>>>>>>>> -I/sw/spock/spack-envs/views/rocm-4.1.0/include >>>>>>>>> -I/sw/spock/spack-envs/views/rocm-4.1.0/include ex3k.kokkos.cxx >>>>>>>>> -Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-opt-cray-kokkos/lib >>>>>>>>> -L/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-opt-cray-kokkos/lib >>>>>>>>> -Wl,-rpath,/gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/lib >>>>>>>>> -L/gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/lib >>>>>>>>> -Wl,-rpath,/sw/spock/spack-envs/views/rocm-4.1.0/lib >>>>>>>>> -L/sw/spock/spack-envs/views/rocm-4.1.0/lib >>>>>>>>> -Wl,-rpath,/opt/gcc/8.1.0/snos/lib64 -L/opt/gcc/8.1.0/snos/lib64 >>>>>>>>> -Wl,-rpath,/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib >>>>>>>>> -L/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib >>>>>>>>> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib >>>>>>>>> -L/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib >>>>>>>>> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/gtl/lib >>>>>>>>> -L/opt/cray/pe/mpich/8.1.4/gtl/lib -Wl,-rpath,/opt/cray/pe/pmi/6.0.10/lib >>>>>>>>> -L/opt/cray/pe/pmi/6.0.10/lib -Wl,-rpath,/opt/cray/pe/dsmml/0.1.4/dsmml/lib >>>>>>>>> -L/opt/cray/pe/dsmml/0.1.4/dsmml/lib >>>>>>>>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce/x86_64/lib >>>>>>>>> -L/opt/cray/pe/cce/11.0.4/cce/x86_64/lib >>>>>>>>> -Wl,-rpath,/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64 >>>>>>>>> -L/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64 >>>>>>>>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux >>>>>>>>> -L/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux >>>>>>>>> -Wl,-rpath,/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0 >>>>>>>>> -L/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0 >>>>>>>>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib >>>>>>>>> -L/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib >>>>>>>>> -lpetsc -lkokkoskernels -lkokkoscontainers -lkokkoscore -lhipsparse >>>>>>>>> -lhipblas -lrocsparse -lrocsolver -lrocblas -lamdhip64 -lhsa-runtime64 >>>>>>>>> -lstdc++ -ldl -lpmi -lsci_cray_mpi -lsci_cray -lmpifort_cray -lmpi_cray >>>>>>>>> -lmpi_gtl_hsa -lxpmem -ldsmml -lpgas-shmem -lquadmath -lcrayacc_amdgpu >>>>>>>>> -lopenacc -lmodules -lfi -lcraymath -lf -lu -lcsup -lgfortran -lpthread >>>>>>>>> -lgcc_eh -lm -lclang_rt.craypgo-x86_64 -lclang_rt.builtins-x86_64 >>>>>>>>> -lquadmath -lstdc++ -ldl -o ex3k >>>>>>>>> In file included from ex3k.kokkos.cxx:3: >>>>>>>>> In file included from >>>>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscdmda_kokkos.hpp:4: >>>>>>>>> In file included from >>>>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec_kokkos.hpp:14: >>>>>>>>> In file included from >>>>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec.h:9: >>>>>>>>> In file included from >>>>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsys.h:42: >>>>>>>>> In file included from >>>>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsystypes.h:255: >>>>>>>>> In file included from >>>>>>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Complex.hpp:47: >>>>>>>>> In file included from >>>>>>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Atomic.hpp:212: >>>>>>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/impl/Kokkos_Atomic_Exchange.hpp:259:11: >>>>>>>>> error: no member named 'lock_address_host_space' in namespace >>>>>>>>> 'Kokkos::Impl::Kokkos::Impl'; did you mean simply 'lock_address_host_space'? >>>>>>>>> while (!Impl::lock_address_host_space((void*)dest)) >>>>>>>>> ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~ >>>>>>>>> lock_address_host_space >>>>>>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_HostSpace.hpp:83:6: >>>>>>>>> note: 'lock_address_host_space' declared here >>>>>>>>> bool lock_address_host_space(void* ptr); >>>>>>>>> ^ >>>>>>>>> In file included from ex3k.kokkos.cxx:3: >>>>>>>>> In file included from >>>>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscdmda_kokkos.hpp:4: >>>>>>>>> In file included from >>>>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec_kokkos.hpp:14: >>>>>>>>> In file included from >>>>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec.h:9: >>>>>>>>> In file included from >>>>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsys.h:42: >>>>>>>>> In file included from >>>>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsystypes.h:255: >>>>>>>>> In file included from >>>>>>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Complex.hpp:47: >>>>>>>>> In file included from >>>>>>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Atomic.hpp:212: >>>>>>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/impl/Kokkos_Atomic_Exchange.hpp:261:3: >>>>>>>>> error: no member named 'memory_fence' in namespace 'Kokkos::Impl::Kokkos'; >>>>>>>>> did you mean simply 'memory_fence'? >>>>>>>>> Kokkos::memory_fence(); >>>>>>>>> ^~~~~~~~~~~~~~~~~~~~ >>>>>>>>> memory_fence >>>>>>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/impl/Kokkos_Memory_Fence.hpp:53:6: >>>>>>>>> note: 'memory_fence' declared here >>>>>>>>> void memory_fence() { >>>>>>>>> ^ >>>>>>>>> In file included from ex3k.kokkos.cxx:3: >>>>>>>>> In file included from >>>>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscdmda_kokkos.hpp:4: >>>>>>>>> In file included from >>>>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec_kokkos.hpp:14: >>>>>>>>> In file included from >>>>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec.h:9: >>>>>>>>> In file included from >>>>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsys.h:42: >>>>>>>>> In file included from >>>>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsystypes.h:255: >>>>>>>>> In file included from >>>>>>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Complex.hpp:47: >>>>>>>>> In file included from >>>>>>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Atomic.hpp:212: >>>>>>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/impl/Kokkos_Atomic_Exchange.hpp:277:3: >>>>>>>>> error: no member named 'memory_fence' in namespace 'Kokkos::Impl::Kokkos'; >>>>>>>>> did you mean simply 'memory_fence'? >>>>>>>>> Kokkos::memory_fence(); >>>>>>>>> ^~~~~~~~~~~~~~~~~~~~ >>>>>>>>> memory_fence >>>>>>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/impl/Kokkos_Memory_Fence.hpp:53:6: >>>>>>>>> note: 'memory_fence' declared here >>>>>>>>> void memory_fence() { >>>>>>>>> ^ >>>>>>>>> >>>>>>>>

1 0

Re: [petsc-dev] Kokkos make error on Spock
by Mark Adams 18 Jul '21

18 Jul '21

Ah, your test was not on Spock ... I have something working but this is strange. I switched to the GNU ProgEnv. and it passed the Fortran test in 'check', but this failed: *08:26 2 jczhang/fix-cray-mpicxx-includes/main= /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials$ make PETSC_DIR=/gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-dbg-gnu PETSC_ARCH="" ex40f90ftn -L/sw/spock/spack-envs/views/rocm-4.1.0/lib -lamdhip64 -lhsa-runtime64 -fPIC -g -O0 -fPIC -g -O0 -I/gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-dbg-gnu/include -I/sw/spock/spack-envs/views/rocm-4.1.0/include ex40f90.F90 -Wl,-rpath,/gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-dbg-gnu/lib -L/gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-dbg-gnu/lib -Wl,-rpath,/sw/spock/spack-envs/views/rocm-4.1.0/lib -L/sw/spock/spack-envs/views/rocm-4.1.0/lib -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/ofi/gnu/9.1/lib -L/opt/cray/pe/mpich/8.1.4/ofi/gnu/9.1/lib -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/gtl/lib -L/opt/cray/pe/mpich/8.1.4/gtl/lib -Wl,-rpath,/opt/cray/pe/libsci/21.04.1.1/GNU/9.1/x86_64/lib <http://21.04.1.1/GNU/9.1/x86_64/lib> -L/opt/cray/pe/libsci/21.04.1.1/GNU/9.1/x86_64/lib <http://21.04.1.1/GNU/9.1/x86_64/lib> -Wl,-rpath,/opt/cray/pe/pmi/6.0.10/lib -L/opt/cray/pe/pmi/6.0.10/lib -Wl,-rpath,/opt/cray/pe/dsmml/0.1.4/dsmml/lib -L/opt/cray/pe/dsmml/0.1.4/dsmml/lib -Wl,-rpath,/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64 -L/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64 -Wl,-rpath,/opt/gcc/10.2.0/snos/lib/gcc/x86_64-suse-linux/10.2.0 -L/opt/gcc/10.2.0/snos/lib/gcc/x86_64-suse-linux/10.2.0 -Wl,-rpath,/opt/gcc/10.2.0/snos/lib64 -L/opt/gcc/10.2.0/snos/lib64 -Wl,-rpath,/sw/spock/spack-envs/views/rocm-4.1.0/llvm/lib -L/sw/spock/spack-envs/views/rocm-4.1.0/llvm/lib -Wl,-rpath,/opt/gcc/10.2.0/snos/lib -L/opt/gcc/10.2.0/snos/lib -lpetsc -lhipsparse -lhipblas -lrocsparse -lrocsolver -lrocblas -lamdhip64 -lhsa-runtime64 -lstdc++ -ldl -lpmi -lmpi_gtl_hsa -lxpmem -lgfortran -lm -lsci_gnu_82_mpi -lsci_gnu_82 -lmpifort_gnu_91 -lmpi_gnu_91 -ldsmml -lgfortran -lquadmath -lpthread -lm -lgcc_s -lquadmath -lstdc++ -ldl -o ex40f90ex40f90.F90:35:132: 35 | call DMDACreate2d(PETSC_COMM_WORLD,DM_BOUNDARY_NONE,DM_BOUNDARY_NONE,DMDA_STENCIL_BOX,ten,ten,PETSC_DECIDE,PETSC_DECIDE,two,one, & | 1Error: Line truncated at (1) [-Werror=line-truncation]ex40f90.F90:35:132:* * ....* But ex5f (the Fortran check test) does work manually also (sanity check). Oddly I seem to have the cray compilers: 08:34 jczhang/fix-cray-mpicxx-includes/main= /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials$ which ftn /opt/cray/pe/craype/2.7.6/bin/ftn yet it is GNU: 08:34 jczhang/fix-cray-mpicxx-includes/main= /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials$ ftn --version GNU Fortran (GCC) 10.2.0 20200723 (Cray Inc.) Copyright (C) 2020 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. The documentation says ftn should be here: 08:34 jczhang/fix-cray-mpicxx-includes/main= /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials$ $GCC_PATH/bin/gfortran --version GNU Fortran (GCC) 10.2.0 20200723 (Cray Inc.) Copyright (C) 2020 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. Same, but the path is not the same: 08:39 jczhang/fix-cray-mpicxx-includes/main= /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials$ echo $GCC_PATH /opt/gcc/10.2.0 Anyway, thanks for your help, Mark On Sat, Jul 17, 2021 at 11:26 PM Junchao Zhang <junchao.zhang(a)gmail.com> wrote: > Mark, I can reproduce this error with PrgEnv-cray, i.e., using the Cray > compiler (clang-11). Previously I used PrgEnv-gnu, which did not have this > error. > Probably it is a problem of Spock. But I am not sure. > > --Junchao Zhang > > > On Sat, Jul 17, 2021 at 10:17 AM Mark Adams <mfadams(a)lbl.gov> wrote: > >> And I can run a fortran test, with warnings, but C tests fail: >> >> 11:15 jczhang/fix-cray-mpicxx-includes/main= >> /gpfs/alpine/csc314/scratch/adams/petsc$ make >> PETSC_DIR=/gpfs/alpine/csc314/scratch/adams/petsc >> PETSC_ARCH=arch-spock-opt-cray-kokkos -f gmakefile test >> search='ts_utils_dmplexlandau_tutorials-ex1f90_0' >> Using MAKEFLAGS: -- search=ts_utils_dmplexlandau_tutorials-ex1f90_0 >> PETSC_ARCH=arch-spock-opt-cray-kokkos >> PETSC_DIR=/gpfs/alpine/csc314/scratch/adams/petsc >> FC >> arch-spock-opt-cray-kokkos/tests/ts/utils/dmplexlandau/tutorials/ex1f90.o >> FLINKER >> arch-spock-opt-cray-kokkos/tests/ts/utils/dmplexlandau/tutorials/ex1f90 >> /opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/bin/ld: >> warning: alignment 128 of symbol >> `$host_init$$runtime_init_for_iso_c_binding$iso_c_binding_' in >> /opt/cray/pe/cce/11.0.4/cce/x86_64/lib/libmodules.so is smaller than 256 in >> arch-spock-opt-cray-kokkos/tests/ts/utils/dmplexlandau/tutorials/ex1f90.o >> /opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/bin/ld: >> warning: alignment 64 of symbol `$data_init$iso_c_binding_' in >> /opt/cray/pe/cce/11.0.4/cce/x86_64/lib/libmodules.so is smaller than 256 in >> arch-spock-opt-cray-kokkos/tests/ts/utils/dmplexlandau/tutorials/ex1f90.o >> TEST >> arch-spock-opt-cray-kokkos/tests/counts/ts_utils_dmplexlandau_tutorials-ex1f90_0.counts >> ok ts_utils_dmplexlandau_tutorials-ex1f90_0 >> ok diff-ts_utils_dmplexlandau_tutorials-ex1f90_0 >> >> >> On Sat, Jul 17, 2021 at 10:53 AM Mark Adams <mfadams(a)lbl.gov> wrote: >> >>> HUmm, I can not reproduce this. >>> >>> 10:49 jczhang/fix-cray-mpicxx-includes/main= >>> /gpfs/alpine/csc314/scratch/adams/petsc$ make >>> PETSC_DIR=/gpfs/alpine/csc314/scratch/adams/petsc >>> PETSC_ARCH=arch-spock-dbg-kokkos check >>> Running check examples to verify correct installation >>> Using PETSC_DIR=/gpfs/alpine/csc314/scratch/adams/petsc and >>> PETSC_ARCH=arch-spock-dbg-kokkos >>> gmake[3]: >>> [/gpfs/alpine/csc314/scratch/adams/petsc/lib/petsc/conf/rules:301: >>> ex19.PETSc] Error 2 (ignored) >>> *******************Error detected during compile or >>> link!******************* >>> See http://www.mcs.anl.gov/petsc/documentation/faq.html >>> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials ex19 >>> >>> ********************************************************************************* >>> cc -L/sw/spock/spack-envs/views/rocm-4.1.0/lib -lamdhip64 >>> -lhsa-runtime64 -fPIC -Wall -Wwrite-strings -Wno-strict-aliasing >>> -Wno-unknown-pragmas -fstack-protector -Qunused-arguments >>> -fvisibility=hidden -g -O0 -fPIC -Wall -Wwrite-strings >>> -Wno-strict-aliasing -Wno-unknown-pragmas -fstack-protector >>> -Qunused-arguments -fvisibility=hidden -g -O0 >>> -I/sw/spock/spack-envs/views/rocm-4.1.0/include >>> -I/gpfs/alpine/csc314/scratch/adams/petsc/include >>> -I/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/include >>> -I/sw/spock/spack-envs/views/rocm-4.1.0/include >>> -I/sw/spock/spack-envs/views/rocm-4.1.0/include ex19.c >>> -Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib >>> -L/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib >>> -Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib >>> -L/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib >>> -Wl,-rpath,/sw/spock/spack-envs/views/rocm-4.1.0/lib >>> -L/sw/spock/spack-envs/views/rocm-4.1.0/lib >>> -Wl,-rpath,/opt/gcc/8.1.0/snos/lib64 -L/opt/gcc/8.1.0/snos/lib64 >>> -Wl,-rpath,/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib >>> -L/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib >>> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib >>> -L/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib >>> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/gtl/lib >>> -L/opt/cray/pe/mpich/8.1.4/gtl/lib -Wl,-rpath,/opt/cray/pe/pmi/6.0.10/lib >>> -L/opt/cray/pe/pmi/6.0.10/lib -Wl,-rpath,/opt/cray/pe/dsmml/0.1.4/dsmml/lib >>> -L/opt/cray/pe/dsmml/0.1.4/dsmml/lib >>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce/x86_64/lib >>> -L/opt/cray/pe/cce/11.0.4/cce/x86_64/lib >>> -Wl,-rpath,/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64 >>> -L/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64 >>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux >>> -L/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux >>> -Wl,-rpath,/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0 >>> -L/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0 >>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib >>> -L/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib >>> -lpetsc -lkokkoskernels -lkokkoscontainers -lkokkoscore -lhipsparse >>> -lhipblas -lrocsparse -lrocsolver -lrocblas -lamdhip64 -lhsa-runtime64 >>> -lstdc++ -ldl -lpmi -lsci_cray_mpi -lsci_cray -lmpifort_cray -lmpi_cray >>> -lmpi_gtl_hsa -lxpmem -ldsmml -lpgas-shmem -lquadmath -lcrayacc_amdgpu >>> -lopenacc -lmodules -lfi -lcraymath -lf -lu -lcsup -lgfortran -lpthread >>> -lgcc_eh -lm -lclang_rt.craypgo-x86_64 -lclang_rt.builtins-x86_64 >>> -lquadmath -lstdc++ -ldl -o ex19 >>> ld.lld: error: >>> /gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib/libpetsc.so: >>> undefined reference to .omp_offloading.img_start.cray_amdgcn-amd-amdhsa >>> [--no-allow-shlib-undefined] >>> ld.lld: error: >>> /gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib/libpetsc.so: >>> undefined reference to .omp_offloading.img_size.cray_amdgcn-amd-amdhsa >>> [--no-allow-shlib-undefined] >>> ld.lld: error: >>> /gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib/libpetsc.so: >>> undefined reference to .omp_offloading.img_cache.cray_amdgcn-amd-amdhsa >>> [--no-allow-shlib-undefined] >>> clang-11: error: linker command failed with exit code 1 (use -v to see >>> invocation) >>> gmake[4]: *** [<builtin>: ex19] Error 1 >>> gmake[3]: >>> [/gpfs/alpine/csc314/scratch/adams/petsc/lib/petsc/conf/rules:301: >>> ex3k.PETSc] Error 2 (ignored) >>> *******************Error detected during compile or >>> link!******************* >>> See http://www.mcs.anl.gov/petsc/documentation/faq.html >>> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials ex3k >>> >>> ********************************************************************************* >>> hipcc -fPIC -Wall -Wwrite-strings -Wno-strict-aliasing >>> -Wno-unknown-pragmas -fvisibility=hidden -g -O0 -std=c++14 >>> -I/gpfs/alpine/csc314/scratch/adams/petsc/include >>> -I/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/include >>> -I/sw/spock/spack-envs/views/rocm-4.1.0/include >>> -I/sw/spock/spack-envs/views/rocm-4.1.0/include -I/opt/cray/pe/libsci/ >>> 21.04.1.1/CRAY/9.0/x86_64/include >>> -I/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/include >>> -I/opt/cray/pe/pmi/6.0.10/include -I/opt/cray/pe/dsmml/0.1.4/dsmml//include >>> -I/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/include >>> -L/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib >>> -L/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib >>> -L/opt/cray/pe/mpich/8.1.4/gtl/lib -L/opt/cray/pe/pmi/6.0.10/lib >>> -L/opt/cray/pe/dsmml/0.1.4/dsmml//lib >>> -L/opt/cray/pe/cce/11.0.4/cce/x86_64/lib/pkgconfig/../ >>> -L/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64 >>> -Wl,--as-needed,-lsci_cray_mpi,--no-as-needed >>> -Wl,--as-needed,-lsci_cray,--no-as-needed >>> -Wl,--as-needed,-lmpi_cray,--no-as-needed >>> -Wl,--as-needed,-ldsmml,--no-as-needed >>> -Wl,--as-needed,-lpgas-shmem,--no-as-needed -lpmi -ldl -lmpi_gtl_hsa >>> -lxpmem -lquadmath -lmodules -lfi -lcraymath -lf -lu -lcsup >>> ex3k.kokkos.cxx >>> -Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib >>> -L/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib >>> -Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib >>> -L/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib >>> -Wl,-rpath,/sw/spock/spack-envs/views/rocm-4.1.0/lib >>> -L/sw/spock/spack-envs/views/rocm-4.1.0/lib >>> -Wl,-rpath,/opt/gcc/8.1.0/snos/lib64 -L/opt/gcc/8.1.0/snos/lib64 >>> -Wl,-rpath,/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib >>> -L/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib >>> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib >>> -L/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib >>> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/gtl/lib >>> -L/opt/cray/pe/mpich/8.1.4/gtl/lib -Wl,-rpath,/opt/cray/pe/pmi/6.0.10/lib >>> -L/opt/cray/pe/pmi/6.0.10/lib -Wl,-rpath,/opt/cray/pe/dsmml/0.1.4/dsmml/lib >>> -L/opt/cray/pe/dsmml/0.1.4/dsmml/lib >>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce/x86_64/lib >>> -L/opt/cray/pe/cce/11.0.4/cce/x86_64/lib >>> -Wl,-rpath,/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64 >>> -L/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64 >>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux >>> -L/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux >>> -Wl,-rpath,/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0 >>> -L/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0 >>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib >>> -L/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib >>> -lpetsc -lkokkoskernels -lkokkoscontainers -lkokkoscore -lhipsparse >>> -lhipblas -lrocsparse -lrocsolver -lrocblas -lamdhip64 -lhsa-runtime64 >>> -lstdc++ -ldl -lpmi -lsci_cray_mpi -lsci_cray -lmpifort_cray -lmpi_cray >>> -lmpi_gtl_hsa -lxpmem -ldsmml -lpgas-shmem -lquadmath -lcrayacc_amdgpu >>> -lopenacc -lmodules -lfi -lcraymath -lf -lu -lcsup -lgfortran -lpthread >>> -lgcc_eh -lm -lclang_rt.craypgo-x86_64 -lclang_rt.builtins-x86_64 >>> -lquadmath -lstdc++ -ldl -o ex3k >>> /usr/lib64/gcc/x86_64-suse-linux/7/../../../../x86_64-suse-linux/bin/ld: >>> /gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib/libpetsc.so: >>> undefined reference to `.omp_offloading.img_size.cray_amdgcn-amd-amdhsa' >>> /usr/lib64/gcc/x86_64-suse-linux/7/../../../../x86_64-suse-linux/bin/ld: >>> /gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib/libpetsc.so: >>> undefined reference to `.omp_offloading.img_start.cray_amdgcn-amd-amdhsa' >>> /usr/lib64/gcc/x86_64-suse-linux/7/../../../../x86_64-suse-linux/bin/ld: >>> /gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib/libpetsc.so: >>> undefined reference to `.omp_offloading.img_cache.cray_amdgcn-amd-amdhsa' >>> clang-12: error: linker command failed with exit code 1 (use -v to see >>> invocation) >>> gmake[4]: *** >>> [/gpfs/alpine/csc314/scratch/adams/petsc/lib/petsc/conf/test:26: ex3k] >>> Error 1 >>> gmake[3]: [makefile:102: runex3k_kokkos] Error 2 (ignored) >>> 1,25c1,4 >>> < atol=1e-50, rtol=1e-08, stol=1e-08, maxit=50, maxf=10000 >>> < Vec Object: Exact Solution 2 MPI processes >>> < type: mpikokkos >>> < Process [0] >>> < 0. >>> < 0.015625 >>> < 0.125 >>> < Process [1] >>> < 0.421875 >>> < 1. >>> < Vec Object: Forcing function 2 MPI processes >>> < type: mpikokkos >>> < Process [0] >>> < 1e-72 >>> < 1.50024 >>> < 3.01563 >>> < Process [1] >>> < 4.67798 >>> < 7. >>> < 0 SNES Function norm 5.414682427127e+00 >>> < 1 SNES Function norm 2.952582418265e-01 >>> < 2 SNES Function norm 4.502293658739e-04 >>> < 3 SNES Function norm 1.389665806646e-09 >>> < Number of SNES iterations = 3 >>> < Norm of error 1.49752e-10 Iterations 3 >>> --- >>> > slurmstepd: error: execve(): >>> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials/./ex3k: No such >>> file or directory >>> > slurmstepd: error: execve(): >>> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials/./ex3k: No such >>> file or directory >>> > srun: error: spock01: tasks 0-1: Exited with exit code 2 >>> > srun: launch/slurm: _step_signal: Terminating StepId=272979.0 >>> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials >>> Possible problem with ex3k running with kokkos-kernels, diffs above >>> ========================================= >>> *******************Error detected during compile or >>> link!******************* >>> See http://www.mcs.anl.gov/petsc/documentation/faq.html >>> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials ex5f >>> ********************************************************* >>> ftn -L/sw/spock/spack-envs/views/rocm-4.1.0/lib -lamdhip64 >>> -lhsa-runtime64 -fPIC -g -O0 -fPIC -g -O0 >>> -I/gpfs/alpine/csc314/scratch/adams/petsc/include >>> -I/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/include >>> -I/sw/spock/spack-envs/views/rocm-4.1.0/include ex5f.F90 >>> -Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib >>> -L/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib >>> -Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib >>> -L/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib >>> -Wl,-rpath,/sw/spock/spack-envs/views/rocm-4.1.0/lib >>> -L/sw/spock/spack-envs/views/rocm-4.1.0/lib >>> -Wl,-rpath,/opt/gcc/8.1.0/snos/lib64 -L/opt/gcc/8.1.0/snos/lib64 >>> -Wl,-rpath,/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib >>> -L/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib >>> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib >>> -L/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib >>> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/gtl/lib >>> -L/opt/cray/pe/mpich/8.1.4/gtl/lib -Wl,-rpath,/opt/cray/pe/pmi/6.0.10/lib >>> -L/opt/cray/pe/pmi/6.0.10/lib -Wl,-rpath,/opt/cray/pe/dsmml/0.1.4/dsmml/lib >>> -L/opt/cray/pe/dsmml/0.1.4/dsmml/lib >>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce/x86_64/lib >>> -L/opt/cray/pe/cce/11.0.4/cce/x86_64/lib >>> -Wl,-rpath,/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64 >>> -L/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64 >>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux >>> -L/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux >>> -Wl,-rpath,/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0 >>> -L/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0 >>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib >>> -L/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib >>> -lpetsc -lkokkoskernels -lkokkoscontainers -lkokkoscore -lhipsparse >>> -lhipblas -lrocsparse -lrocsolver -lrocblas -lamdhip64 -lhsa-runtime64 >>> -lstdc++ -ldl -lpmi -lsci_cray_mpi -lsci_cray -lmpifort_cray -lmpi_cray >>> -lmpi_gtl_hsa -lxpmem -ldsmml -lpgas-shmem -lquadmath -lcrayacc_amdgpu >>> -lopenacc -lmodules -lfi -lcraymath -lf -lu -lcsup -lgfortran -lpthread >>> -lgcc_eh -lm -lclang_rt.craypgo-x86_64 -lclang_rt.builtins-x86_64 >>> -lquadmath -lstdc++ -ldl -o ex5f >>> /opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/bin/ld: >>> warning: alignment 128 of symbol >>> `$host_init$$runtime_init_for_iso_c_binding$iso_c_binding_' in >>> /opt/cray/pe/cce/11.0.4/cce/x86_64/lib/libmodules.so is smaller than 256 in >>> /tmp/pe_30429/ex5f_1.o >>> /opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/bin/ld: >>> warning: alignment 64 of symbol `$data_init$iso_c_binding_' in >>> /opt/cray/pe/cce/11.0.4/cce/x86_64/lib/libmodules.so is smaller than 256 in >>> /tmp/pe_30429/ex5f_1.o >>> Fortran example src/snes/tutorials/ex5f run successfully with 1 MPI >>> process >>> Completed test examples >>> 10:49 jczhang/fix-cray-mpicxx-includes/main= >>> /gpfs/alpine/csc314/scratch/adams/petsc$ ll >>> >>> On Fri, Jul 16, 2021 at 11:28 PM Junchao Zhang <junchao.zhang(a)gmail.com> >>> wrote: >>> >>>> Mark, I configured with Fortran binding enabled with main + MR !4175 >>>> <https://gitlab.com/petsc/petsc/-/merge_requests/4175> >>>> >>>> '--with-mpiexec=srun', >>>> '--with-shared-libraries=1', >>>> '--with-cc=cc', >>>> '--with-cxx=CC', >>>> '--with-fc=ftn', >>>> '--with-fortran-bindings', >>>> '--with-hip', >>>> '--with-hipc=hipcc', >>>> '--with-debugging', >>>> '--CPPFLAGS=-I${ROCM_PATH}/include', >>>> '--CXXPPFLAGS=-I${ROCM_PATH}/include', >>>> '--CC_LINKER_FLAGS=-L${ROCM_PATH}/lib -lamdhip64 -lhsa-runtime64', >>>> '--CXX_LINKER_FLAGS=-L${ROCM_PATH}/lib -lamdhip64 -lhsa-runtime64', >>>> '--FC_LINKER_FLAGS=-L${ROCM_PATH}/lib -lamdhip64 -lhsa-runtime64', >>>> '--COPTFLAGS=-g -O0', >>>> '--CXXOPTFLAGS=-g -O0', >>>> '--FOPTFLAGS=-g -O0', >>>> '--download-kokkos', >>>> '--download-kokkos-kernels', >>>> '--download-kokkos-commit=3.4.01', >>>> '--download-kokkos-kernels-commit=3.4.01', >>>> '--with-kokkos-hip-arch=VEGA908', >>>> >>>> and 'make check' ran smoothly on a compute node >>>> >>>> $ make check >>>> Running check examples to verify correct installation >>>> Using PETSC_DIR=/ccs/home/jczhang/petsc and >>>> PETSC_ARCH=arch-spock-cray-kokkos-dbg >>>> C/C++ example src/snes/tutorials/ex19 run successfully with 1 MPI >>>> process >>>> C/C++ example src/snes/tutorials/ex19 run successfully with 2 MPI >>>> processes >>>> C/C++ example src/snes/tutorials/ex3k run successfully with >>>> kokkos-kernels >>>> Fortran example src/snes/tutorials/ex5f run successfully with 1 MPI >>>> process >>>> >>>> Completed test examples >>>> >>>> >>>> --Junchao Zhang >>>> >>>> >>>> On Fri, Jul 16, 2021 at 6:04 PM Mark Adams <mfadams(a)lbl.gov> wrote: >>>> >>>>> And I find that this error, on non-Kokkos C tests, is fixed by turning >>>>> the fortran bindings off: >>>>> >>>>> ld.lld: error: >>>>> /gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/lib/libpetsc.so: >>>>> undefined reference to .omp_offloading.img_start.cray_amdgcn-amd-amdhsa >>>>> [--no-allow-shlib-undefined] >>>>> ld.lld: error: >>>>> /gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/lib/libpetsc.so: >>>>> undefined reference to .omp_offloading.img_size.cray_amdgcn-amd-amdhsa >>>>> [--no-allow-shlib-undefined] >>>>> ld.lld: error: >>>>> /gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/lib/libpetsc.so: >>>>> undefined reference to .omp_offloading.img_cache.cray_amdgcn-amd-amdhsa >>>>> [--no-allow-shlib-undefined] >>>>> >>>>> On Fri, Jul 16, 2021 at 3:53 PM Mark Adams <mfadams(a)lbl.gov> wrote: >>>>> >>>>>> Not complex. THis has some overlap with my problem w/o Kokkos. >>>>>> >>>>>> On Fri, Jul 16, 2021 at 12:54 PM Junchao Zhang < >>>>>> junchao.zhang(a)gmail.com> wrote: >>>>>> >>>>>>> Do you use complex? post your configure.log. >>>>>>> >>>>>>> --Junchao Zhang >>>>>>> >>>>>>> >>>>>>> On Fri, Jul 16, 2021 at 9:47 AM Mark Adams <mfadams(a)lbl.gov> wrote: >>>>>>> >>>>>>>> The simple Kokkos example is failing for me on Spock. >>>>>>>> Any ideas? >>>>>>>> Thanks, >>>>>>>> >>>>>>>> 10:44 main *= >>>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials$ make >>>>>>>> PETSC_DIR=/gpfs/alpine/csc314/scratch/adams/petsc >>>>>>>> PETSC_ARCH=arch-spock-opt-cray-kokkos ex3k >>>>>>>> MPICH_CXX="hipcc" OMPI_CXX="hipcc" CC >>>>>>>> -L/sw/spock/spack-envs/views/rocm-4.1.0/lib -lhsa-runtime64 >>>>>>>> -L/sw/spock/spack-envs/views/rocm-4.1.0/lib -lamdhip64 -lhsa-runtime64 >>>>>>>> -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas >>>>>>>> -fstack-protector -fvisibility=hidden -g -O2 -fPIC -Wall -Wwrite-strings >>>>>>>> -Wno-strict-aliasing -Wno-unknown-pragmas -fvisibility=hidden -g -O3 >>>>>>>> -std=c++14 -I/gpfs/alpine/csc314/scratch/adams/petsc/include >>>>>>>> -I/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-opt-cray-kokkos/include >>>>>>>> -I/gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include >>>>>>>> -I/sw/spock/spack-envs/views/rocm-4.1.0/include >>>>>>>> -I/sw/spock/spack-envs/views/rocm-4.1.0/include ex3k.kokkos.cxx >>>>>>>> -Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-opt-cray-kokkos/lib >>>>>>>> -L/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-opt-cray-kokkos/lib >>>>>>>> -Wl,-rpath,/gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/lib >>>>>>>> -L/gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/lib >>>>>>>> -Wl,-rpath,/sw/spock/spack-envs/views/rocm-4.1.0/lib >>>>>>>> -L/sw/spock/spack-envs/views/rocm-4.1.0/lib >>>>>>>> -Wl,-rpath,/opt/gcc/8.1.0/snos/lib64 -L/opt/gcc/8.1.0/snos/lib64 >>>>>>>> -Wl,-rpath,/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib >>>>>>>> -L/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib >>>>>>>> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib >>>>>>>> -L/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib >>>>>>>> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/gtl/lib >>>>>>>> -L/opt/cray/pe/mpich/8.1.4/gtl/lib -Wl,-rpath,/opt/cray/pe/pmi/6.0.10/lib >>>>>>>> -L/opt/cray/pe/pmi/6.0.10/lib -Wl,-rpath,/opt/cray/pe/dsmml/0.1.4/dsmml/lib >>>>>>>> -L/opt/cray/pe/dsmml/0.1.4/dsmml/lib >>>>>>>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce/x86_64/lib >>>>>>>> -L/opt/cray/pe/cce/11.0.4/cce/x86_64/lib >>>>>>>> -Wl,-rpath,/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64 >>>>>>>> -L/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64 >>>>>>>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux >>>>>>>> -L/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux >>>>>>>> -Wl,-rpath,/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0 >>>>>>>> -L/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0 >>>>>>>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib >>>>>>>> -L/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib >>>>>>>> -lpetsc -lkokkoskernels -lkokkoscontainers -lkokkoscore -lhipsparse >>>>>>>> -lhipblas -lrocsparse -lrocsolver -lrocblas -lamdhip64 -lhsa-runtime64 >>>>>>>> -lstdc++ -ldl -lpmi -lsci_cray_mpi -lsci_cray -lmpifort_cray -lmpi_cray >>>>>>>> -lmpi_gtl_hsa -lxpmem -ldsmml -lpgas-shmem -lquadmath -lcrayacc_amdgpu >>>>>>>> -lopenacc -lmodules -lfi -lcraymath -lf -lu -lcsup -lgfortran -lpthread >>>>>>>> -lgcc_eh -lm -lclang_rt.craypgo-x86_64 -lclang_rt.builtins-x86_64 >>>>>>>> -lquadmath -lstdc++ -ldl -o ex3k >>>>>>>> In file included from ex3k.kokkos.cxx:3: >>>>>>>> In file included from >>>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscdmda_kokkos.hpp:4: >>>>>>>> In file included from >>>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec_kokkos.hpp:14: >>>>>>>> In file included from >>>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec.h:9: >>>>>>>> In file included from >>>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsys.h:42: >>>>>>>> In file included from >>>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsystypes.h:255: >>>>>>>> In file included from >>>>>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Complex.hpp:47: >>>>>>>> In file included from >>>>>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Atomic.hpp:212: >>>>>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/impl/Kokkos_Atomic_Exchange.hpp:259:11: >>>>>>>> error: no member named 'lock_address_host_space' in namespace >>>>>>>> 'Kokkos::Impl::Kokkos::Impl'; did you mean simply 'lock_address_host_space'? >>>>>>>> while (!Impl::lock_address_host_space((void*)dest)) >>>>>>>> ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~ >>>>>>>> lock_address_host_space >>>>>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_HostSpace.hpp:83:6: >>>>>>>> note: 'lock_address_host_space' declared here >>>>>>>> bool lock_address_host_space(void* ptr); >>>>>>>> ^ >>>>>>>> In file included from ex3k.kokkos.cxx:3: >>>>>>>> In file included from >>>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscdmda_kokkos.hpp:4: >>>>>>>> In file included from >>>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec_kokkos.hpp:14: >>>>>>>> In file included from >>>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec.h:9: >>>>>>>> In file included from >>>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsys.h:42: >>>>>>>> In file included from >>>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsystypes.h:255: >>>>>>>> In file included from >>>>>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Complex.hpp:47: >>>>>>>> In file included from >>>>>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Atomic.hpp:212: >>>>>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/impl/Kokkos_Atomic_Exchange.hpp:261:3: >>>>>>>> error: no member named 'memory_fence' in namespace 'Kokkos::Impl::Kokkos'; >>>>>>>> did you mean simply 'memory_fence'? >>>>>>>> Kokkos::memory_fence(); >>>>>>>> ^~~~~~~~~~~~~~~~~~~~ >>>>>>>> memory_fence >>>>>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/impl/Kokkos_Memory_Fence.hpp:53:6: >>>>>>>> note: 'memory_fence' declared here >>>>>>>> void memory_fence() { >>>>>>>> ^ >>>>>>>> In file included from ex3k.kokkos.cxx:3: >>>>>>>> In file included from >>>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscdmda_kokkos.hpp:4: >>>>>>>> In file included from >>>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec_kokkos.hpp:14: >>>>>>>> In file included from >>>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec.h:9: >>>>>>>> In file included from >>>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsys.h:42: >>>>>>>> In file included from >>>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsystypes.h:255: >>>>>>>> In file included from >>>>>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Complex.hpp:47: >>>>>>>> In file included from >>>>>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Atomic.hpp:212: >>>>>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/impl/Kokkos_Atomic_Exchange.hpp:277:3: >>>>>>>> error: no member named 'memory_fence' in namespace 'Kokkos::Impl::Kokkos'; >>>>>>>> did you mean simply 'memory_fence'? >>>>>>>> Kokkos::memory_fence(); >>>>>>>> ^~~~~~~~~~~~~~~~~~~~ >>>>>>>> memory_fence >>>>>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/impl/Kokkos_Memory_Fence.hpp:53:6: >>>>>>>> note: 'memory_fence' declared here >>>>>>>> void memory_fence() { >>>>>>>> ^ >>>>>>>> >>>>>>>

1 0

Re: [petsc-dev] Kokkos make error on Spock
by Junchao Zhang 17 Jul '21

17 Jul '21

Mark, I can reproduce this error with PrgEnv-cray, i.e., using the Cray compiler (clang-11). Previously I used PrgEnv-gnu, which did not have this error. Probably it is a problem of Spock. But I am not sure. --Junchao Zhang On Sat, Jul 17, 2021 at 10:17 AM Mark Adams <mfadams(a)lbl.gov> wrote: > And I can run a fortran test, with warnings, but C tests fail: > > 11:15 jczhang/fix-cray-mpicxx-includes/main= > /gpfs/alpine/csc314/scratch/adams/petsc$ make > PETSC_DIR=/gpfs/alpine/csc314/scratch/adams/petsc > PETSC_ARCH=arch-spock-opt-cray-kokkos -f gmakefile test > search='ts_utils_dmplexlandau_tutorials-ex1f90_0' > Using MAKEFLAGS: -- search=ts_utils_dmplexlandau_tutorials-ex1f90_0 > PETSC_ARCH=arch-spock-opt-cray-kokkos > PETSC_DIR=/gpfs/alpine/csc314/scratch/adams/petsc > FC > arch-spock-opt-cray-kokkos/tests/ts/utils/dmplexlandau/tutorials/ex1f90.o > FLINKER > arch-spock-opt-cray-kokkos/tests/ts/utils/dmplexlandau/tutorials/ex1f90 > /opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/bin/ld: > warning: alignment 128 of symbol > `$host_init$$runtime_init_for_iso_c_binding$iso_c_binding_' in > /opt/cray/pe/cce/11.0.4/cce/x86_64/lib/libmodules.so is smaller than 256 in > arch-spock-opt-cray-kokkos/tests/ts/utils/dmplexlandau/tutorials/ex1f90.o > /opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/bin/ld: > warning: alignment 64 of symbol `$data_init$iso_c_binding_' in > /opt/cray/pe/cce/11.0.4/cce/x86_64/lib/libmodules.so is smaller than 256 in > arch-spock-opt-cray-kokkos/tests/ts/utils/dmplexlandau/tutorials/ex1f90.o > TEST > arch-spock-opt-cray-kokkos/tests/counts/ts_utils_dmplexlandau_tutorials-ex1f90_0.counts > ok ts_utils_dmplexlandau_tutorials-ex1f90_0 > ok diff-ts_utils_dmplexlandau_tutorials-ex1f90_0 > > > On Sat, Jul 17, 2021 at 10:53 AM Mark Adams <mfadams(a)lbl.gov> wrote: > >> HUmm, I can not reproduce this. >> >> 10:49 jczhang/fix-cray-mpicxx-includes/main= >> /gpfs/alpine/csc314/scratch/adams/petsc$ make >> PETSC_DIR=/gpfs/alpine/csc314/scratch/adams/petsc >> PETSC_ARCH=arch-spock-dbg-kokkos check >> Running check examples to verify correct installation >> Using PETSC_DIR=/gpfs/alpine/csc314/scratch/adams/petsc and >> PETSC_ARCH=arch-spock-dbg-kokkos >> gmake[3]: >> [/gpfs/alpine/csc314/scratch/adams/petsc/lib/petsc/conf/rules:301: >> ex19.PETSc] Error 2 (ignored) >> *******************Error detected during compile or >> link!******************* >> See http://www.mcs.anl.gov/petsc/documentation/faq.html >> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials ex19 >> >> ********************************************************************************* >> cc -L/sw/spock/spack-envs/views/rocm-4.1.0/lib -lamdhip64 -lhsa-runtime64 >> -fPIC -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas >> -fstack-protector -Qunused-arguments -fvisibility=hidden -g -O0 -fPIC >> -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas >> -fstack-protector -Qunused-arguments -fvisibility=hidden -g -O0 >> -I/sw/spock/spack-envs/views/rocm-4.1.0/include >> -I/gpfs/alpine/csc314/scratch/adams/petsc/include >> -I/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/include >> -I/sw/spock/spack-envs/views/rocm-4.1.0/include >> -I/sw/spock/spack-envs/views/rocm-4.1.0/include ex19.c >> -Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib >> -L/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib >> -Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib >> -L/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib >> -Wl,-rpath,/sw/spock/spack-envs/views/rocm-4.1.0/lib >> -L/sw/spock/spack-envs/views/rocm-4.1.0/lib >> -Wl,-rpath,/opt/gcc/8.1.0/snos/lib64 -L/opt/gcc/8.1.0/snos/lib64 >> -Wl,-rpath,/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib >> -L/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib >> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib >> -L/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib >> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/gtl/lib >> -L/opt/cray/pe/mpich/8.1.4/gtl/lib -Wl,-rpath,/opt/cray/pe/pmi/6.0.10/lib >> -L/opt/cray/pe/pmi/6.0.10/lib -Wl,-rpath,/opt/cray/pe/dsmml/0.1.4/dsmml/lib >> -L/opt/cray/pe/dsmml/0.1.4/dsmml/lib >> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce/x86_64/lib >> -L/opt/cray/pe/cce/11.0.4/cce/x86_64/lib >> -Wl,-rpath,/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64 >> -L/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64 >> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux >> -L/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux >> -Wl,-rpath,/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0 >> -L/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0 >> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib >> -L/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib >> -lpetsc -lkokkoskernels -lkokkoscontainers -lkokkoscore -lhipsparse >> -lhipblas -lrocsparse -lrocsolver -lrocblas -lamdhip64 -lhsa-runtime64 >> -lstdc++ -ldl -lpmi -lsci_cray_mpi -lsci_cray -lmpifort_cray -lmpi_cray >> -lmpi_gtl_hsa -lxpmem -ldsmml -lpgas-shmem -lquadmath -lcrayacc_amdgpu >> -lopenacc -lmodules -lfi -lcraymath -lf -lu -lcsup -lgfortran -lpthread >> -lgcc_eh -lm -lclang_rt.craypgo-x86_64 -lclang_rt.builtins-x86_64 >> -lquadmath -lstdc++ -ldl -o ex19 >> ld.lld: error: >> /gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib/libpetsc.so: >> undefined reference to .omp_offloading.img_start.cray_amdgcn-amd-amdhsa >> [--no-allow-shlib-undefined] >> ld.lld: error: >> /gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib/libpetsc.so: >> undefined reference to .omp_offloading.img_size.cray_amdgcn-amd-amdhsa >> [--no-allow-shlib-undefined] >> ld.lld: error: >> /gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib/libpetsc.so: >> undefined reference to .omp_offloading.img_cache.cray_amdgcn-amd-amdhsa >> [--no-allow-shlib-undefined] >> clang-11: error: linker command failed with exit code 1 (use -v to see >> invocation) >> gmake[4]: *** [<builtin>: ex19] Error 1 >> gmake[3]: >> [/gpfs/alpine/csc314/scratch/adams/petsc/lib/petsc/conf/rules:301: >> ex3k.PETSc] Error 2 (ignored) >> *******************Error detected during compile or >> link!******************* >> See http://www.mcs.anl.gov/petsc/documentation/faq.html >> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials ex3k >> >> ********************************************************************************* >> hipcc -fPIC -Wall -Wwrite-strings -Wno-strict-aliasing >> -Wno-unknown-pragmas -fvisibility=hidden -g -O0 -std=c++14 >> -I/gpfs/alpine/csc314/scratch/adams/petsc/include >> -I/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/include >> -I/sw/spock/spack-envs/views/rocm-4.1.0/include >> -I/sw/spock/spack-envs/views/rocm-4.1.0/include -I/opt/cray/pe/libsci/ >> 21.04.1.1/CRAY/9.0/x86_64/include >> -I/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/include >> -I/opt/cray/pe/pmi/6.0.10/include -I/opt/cray/pe/dsmml/0.1.4/dsmml//include >> -I/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/include >> -L/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib >> -L/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib >> -L/opt/cray/pe/mpich/8.1.4/gtl/lib -L/opt/cray/pe/pmi/6.0.10/lib >> -L/opt/cray/pe/dsmml/0.1.4/dsmml//lib >> -L/opt/cray/pe/cce/11.0.4/cce/x86_64/lib/pkgconfig/../ >> -L/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64 >> -Wl,--as-needed,-lsci_cray_mpi,--no-as-needed >> -Wl,--as-needed,-lsci_cray,--no-as-needed >> -Wl,--as-needed,-lmpi_cray,--no-as-needed >> -Wl,--as-needed,-ldsmml,--no-as-needed >> -Wl,--as-needed,-lpgas-shmem,--no-as-needed -lpmi -ldl -lmpi_gtl_hsa >> -lxpmem -lquadmath -lmodules -lfi -lcraymath -lf -lu -lcsup >> ex3k.kokkos.cxx >> -Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib >> -L/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib >> -Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib >> -L/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib >> -Wl,-rpath,/sw/spock/spack-envs/views/rocm-4.1.0/lib >> -L/sw/spock/spack-envs/views/rocm-4.1.0/lib >> -Wl,-rpath,/opt/gcc/8.1.0/snos/lib64 -L/opt/gcc/8.1.0/snos/lib64 >> -Wl,-rpath,/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib >> -L/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib >> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib >> -L/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib >> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/gtl/lib >> -L/opt/cray/pe/mpich/8.1.4/gtl/lib -Wl,-rpath,/opt/cray/pe/pmi/6.0.10/lib >> -L/opt/cray/pe/pmi/6.0.10/lib -Wl,-rpath,/opt/cray/pe/dsmml/0.1.4/dsmml/lib >> -L/opt/cray/pe/dsmml/0.1.4/dsmml/lib >> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce/x86_64/lib >> -L/opt/cray/pe/cce/11.0.4/cce/x86_64/lib >> -Wl,-rpath,/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64 >> -L/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64 >> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux >> -L/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux >> -Wl,-rpath,/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0 >> -L/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0 >> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib >> -L/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib >> -lpetsc -lkokkoskernels -lkokkoscontainers -lkokkoscore -lhipsparse >> -lhipblas -lrocsparse -lrocsolver -lrocblas -lamdhip64 -lhsa-runtime64 >> -lstdc++ -ldl -lpmi -lsci_cray_mpi -lsci_cray -lmpifort_cray -lmpi_cray >> -lmpi_gtl_hsa -lxpmem -ldsmml -lpgas-shmem -lquadmath -lcrayacc_amdgpu >> -lopenacc -lmodules -lfi -lcraymath -lf -lu -lcsup -lgfortran -lpthread >> -lgcc_eh -lm -lclang_rt.craypgo-x86_64 -lclang_rt.builtins-x86_64 >> -lquadmath -lstdc++ -ldl -o ex3k >> /usr/lib64/gcc/x86_64-suse-linux/7/../../../../x86_64-suse-linux/bin/ld: >> /gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib/libpetsc.so: >> undefined reference to `.omp_offloading.img_size.cray_amdgcn-amd-amdhsa' >> /usr/lib64/gcc/x86_64-suse-linux/7/../../../../x86_64-suse-linux/bin/ld: >> /gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib/libpetsc.so: >> undefined reference to `.omp_offloading.img_start.cray_amdgcn-amd-amdhsa' >> /usr/lib64/gcc/x86_64-suse-linux/7/../../../../x86_64-suse-linux/bin/ld: >> /gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib/libpetsc.so: >> undefined reference to `.omp_offloading.img_cache.cray_amdgcn-amd-amdhsa' >> clang-12: error: linker command failed with exit code 1 (use -v to see >> invocation) >> gmake[4]: *** >> [/gpfs/alpine/csc314/scratch/adams/petsc/lib/petsc/conf/test:26: ex3k] >> Error 1 >> gmake[3]: [makefile:102: runex3k_kokkos] Error 2 (ignored) >> 1,25c1,4 >> < atol=1e-50, rtol=1e-08, stol=1e-08, maxit=50, maxf=10000 >> < Vec Object: Exact Solution 2 MPI processes >> < type: mpikokkos >> < Process [0] >> < 0. >> < 0.015625 >> < 0.125 >> < Process [1] >> < 0.421875 >> < 1. >> < Vec Object: Forcing function 2 MPI processes >> < type: mpikokkos >> < Process [0] >> < 1e-72 >> < 1.50024 >> < 3.01563 >> < Process [1] >> < 4.67798 >> < 7. >> < 0 SNES Function norm 5.414682427127e+00 >> < 1 SNES Function norm 2.952582418265e-01 >> < 2 SNES Function norm 4.502293658739e-04 >> < 3 SNES Function norm 1.389665806646e-09 >> < Number of SNES iterations = 3 >> < Norm of error 1.49752e-10 Iterations 3 >> --- >> > slurmstepd: error: execve(): >> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials/./ex3k: No such >> file or directory >> > slurmstepd: error: execve(): >> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials/./ex3k: No such >> file or directory >> > srun: error: spock01: tasks 0-1: Exited with exit code 2 >> > srun: launch/slurm: _step_signal: Terminating StepId=272979.0 >> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials >> Possible problem with ex3k running with kokkos-kernels, diffs above >> ========================================= >> *******************Error detected during compile or >> link!******************* >> See http://www.mcs.anl.gov/petsc/documentation/faq.html >> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials ex5f >> ********************************************************* >> ftn -L/sw/spock/spack-envs/views/rocm-4.1.0/lib -lamdhip64 >> -lhsa-runtime64 -fPIC -g -O0 -fPIC -g -O0 >> -I/gpfs/alpine/csc314/scratch/adams/petsc/include >> -I/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/include >> -I/sw/spock/spack-envs/views/rocm-4.1.0/include ex5f.F90 >> -Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib >> -L/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib >> -Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib >> -L/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib >> -Wl,-rpath,/sw/spock/spack-envs/views/rocm-4.1.0/lib >> -L/sw/spock/spack-envs/views/rocm-4.1.0/lib >> -Wl,-rpath,/opt/gcc/8.1.0/snos/lib64 -L/opt/gcc/8.1.0/snos/lib64 >> -Wl,-rpath,/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib >> -L/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib >> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib >> -L/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib >> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/gtl/lib >> -L/opt/cray/pe/mpich/8.1.4/gtl/lib -Wl,-rpath,/opt/cray/pe/pmi/6.0.10/lib >> -L/opt/cray/pe/pmi/6.0.10/lib -Wl,-rpath,/opt/cray/pe/dsmml/0.1.4/dsmml/lib >> -L/opt/cray/pe/dsmml/0.1.4/dsmml/lib >> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce/x86_64/lib >> -L/opt/cray/pe/cce/11.0.4/cce/x86_64/lib >> -Wl,-rpath,/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64 >> -L/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64 >> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux >> -L/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux >> -Wl,-rpath,/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0 >> -L/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0 >> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib >> -L/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib >> -lpetsc -lkokkoskernels -lkokkoscontainers -lkokkoscore -lhipsparse >> -lhipblas -lrocsparse -lrocsolver -lrocblas -lamdhip64 -lhsa-runtime64 >> -lstdc++ -ldl -lpmi -lsci_cray_mpi -lsci_cray -lmpifort_cray -lmpi_cray >> -lmpi_gtl_hsa -lxpmem -ldsmml -lpgas-shmem -lquadmath -lcrayacc_amdgpu >> -lopenacc -lmodules -lfi -lcraymath -lf -lu -lcsup -lgfortran -lpthread >> -lgcc_eh -lm -lclang_rt.craypgo-x86_64 -lclang_rt.builtins-x86_64 >> -lquadmath -lstdc++ -ldl -o ex5f >> /opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/bin/ld: >> warning: alignment 128 of symbol >> `$host_init$$runtime_init_for_iso_c_binding$iso_c_binding_' in >> /opt/cray/pe/cce/11.0.4/cce/x86_64/lib/libmodules.so is smaller than 256 in >> /tmp/pe_30429/ex5f_1.o >> /opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/bin/ld: >> warning: alignment 64 of symbol `$data_init$iso_c_binding_' in >> /opt/cray/pe/cce/11.0.4/cce/x86_64/lib/libmodules.so is smaller than 256 in >> /tmp/pe_30429/ex5f_1.o >> Fortran example src/snes/tutorials/ex5f run successfully with 1 MPI >> process >> Completed test examples >> 10:49 jczhang/fix-cray-mpicxx-includes/main= >> /gpfs/alpine/csc314/scratch/adams/petsc$ ll >> >> On Fri, Jul 16, 2021 at 11:28 PM Junchao Zhang <junchao.zhang(a)gmail.com> >> wrote: >> >>> Mark, I configured with Fortran binding enabled with main + MR !4175 >>> <https://gitlab.com/petsc/petsc/-/merge_requests/4175> >>> >>> '--with-mpiexec=srun', >>> '--with-shared-libraries=1', >>> '--with-cc=cc', >>> '--with-cxx=CC', >>> '--with-fc=ftn', >>> '--with-fortran-bindings', >>> '--with-hip', >>> '--with-hipc=hipcc', >>> '--with-debugging', >>> '--CPPFLAGS=-I${ROCM_PATH}/include', >>> '--CXXPPFLAGS=-I${ROCM_PATH}/include', >>> '--CC_LINKER_FLAGS=-L${ROCM_PATH}/lib -lamdhip64 -lhsa-runtime64', >>> '--CXX_LINKER_FLAGS=-L${ROCM_PATH}/lib -lamdhip64 -lhsa-runtime64', >>> '--FC_LINKER_FLAGS=-L${ROCM_PATH}/lib -lamdhip64 -lhsa-runtime64', >>> '--COPTFLAGS=-g -O0', >>> '--CXXOPTFLAGS=-g -O0', >>> '--FOPTFLAGS=-g -O0', >>> '--download-kokkos', >>> '--download-kokkos-kernels', >>> '--download-kokkos-commit=3.4.01', >>> '--download-kokkos-kernels-commit=3.4.01', >>> '--with-kokkos-hip-arch=VEGA908', >>> >>> and 'make check' ran smoothly on a compute node >>> >>> $ make check >>> Running check examples to verify correct installation >>> Using PETSC_DIR=/ccs/home/jczhang/petsc and >>> PETSC_ARCH=arch-spock-cray-kokkos-dbg >>> C/C++ example src/snes/tutorials/ex19 run successfully with 1 MPI process >>> C/C++ example src/snes/tutorials/ex19 run successfully with 2 MPI >>> processes >>> C/C++ example src/snes/tutorials/ex3k run successfully with >>> kokkos-kernels >>> Fortran example src/snes/tutorials/ex5f run successfully with 1 MPI >>> process >>> >>> Completed test examples >>> >>> >>> --Junchao Zhang >>> >>> >>> On Fri, Jul 16, 2021 at 6:04 PM Mark Adams <mfadams(a)lbl.gov> wrote: >>> >>>> And I find that this error, on non-Kokkos C tests, is fixed by turning >>>> the fortran bindings off: >>>> >>>> ld.lld: error: >>>> /gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/lib/libpetsc.so: >>>> undefined reference to .omp_offloading.img_start.cray_amdgcn-amd-amdhsa >>>> [--no-allow-shlib-undefined] >>>> ld.lld: error: >>>> /gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/lib/libpetsc.so: >>>> undefined reference to .omp_offloading.img_size.cray_amdgcn-amd-amdhsa >>>> [--no-allow-shlib-undefined] >>>> ld.lld: error: >>>> /gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/lib/libpetsc.so: >>>> undefined reference to .omp_offloading.img_cache.cray_amdgcn-amd-amdhsa >>>> [--no-allow-shlib-undefined] >>>> >>>> On Fri, Jul 16, 2021 at 3:53 PM Mark Adams <mfadams(a)lbl.gov> wrote: >>>> >>>>> Not complex. THis has some overlap with my problem w/o Kokkos. >>>>> >>>>> On Fri, Jul 16, 2021 at 12:54 PM Junchao Zhang < >>>>> junchao.zhang(a)gmail.com> wrote: >>>>> >>>>>> Do you use complex? post your configure.log. >>>>>> >>>>>> --Junchao Zhang >>>>>> >>>>>> >>>>>> On Fri, Jul 16, 2021 at 9:47 AM Mark Adams <mfadams(a)lbl.gov> wrote: >>>>>> >>>>>>> The simple Kokkos example is failing for me on Spock. >>>>>>> Any ideas? >>>>>>> Thanks, >>>>>>> >>>>>>> 10:44 main *= >>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials$ make >>>>>>> PETSC_DIR=/gpfs/alpine/csc314/scratch/adams/petsc >>>>>>> PETSC_ARCH=arch-spock-opt-cray-kokkos ex3k >>>>>>> MPICH_CXX="hipcc" OMPI_CXX="hipcc" CC >>>>>>> -L/sw/spock/spack-envs/views/rocm-4.1.0/lib -lhsa-runtime64 >>>>>>> -L/sw/spock/spack-envs/views/rocm-4.1.0/lib -lamdhip64 -lhsa-runtime64 >>>>>>> -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas >>>>>>> -fstack-protector -fvisibility=hidden -g -O2 -fPIC -Wall -Wwrite-strings >>>>>>> -Wno-strict-aliasing -Wno-unknown-pragmas -fvisibility=hidden -g -O3 >>>>>>> -std=c++14 -I/gpfs/alpine/csc314/scratch/adams/petsc/include >>>>>>> -I/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-opt-cray-kokkos/include >>>>>>> -I/gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include >>>>>>> -I/sw/spock/spack-envs/views/rocm-4.1.0/include >>>>>>> -I/sw/spock/spack-envs/views/rocm-4.1.0/include ex3k.kokkos.cxx >>>>>>> -Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-opt-cray-kokkos/lib >>>>>>> -L/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-opt-cray-kokkos/lib >>>>>>> -Wl,-rpath,/gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/lib >>>>>>> -L/gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/lib >>>>>>> -Wl,-rpath,/sw/spock/spack-envs/views/rocm-4.1.0/lib >>>>>>> -L/sw/spock/spack-envs/views/rocm-4.1.0/lib >>>>>>> -Wl,-rpath,/opt/gcc/8.1.0/snos/lib64 -L/opt/gcc/8.1.0/snos/lib64 >>>>>>> -Wl,-rpath,/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib >>>>>>> -L/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib >>>>>>> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib >>>>>>> -L/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib >>>>>>> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/gtl/lib >>>>>>> -L/opt/cray/pe/mpich/8.1.4/gtl/lib -Wl,-rpath,/opt/cray/pe/pmi/6.0.10/lib >>>>>>> -L/opt/cray/pe/pmi/6.0.10/lib -Wl,-rpath,/opt/cray/pe/dsmml/0.1.4/dsmml/lib >>>>>>> -L/opt/cray/pe/dsmml/0.1.4/dsmml/lib >>>>>>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce/x86_64/lib >>>>>>> -L/opt/cray/pe/cce/11.0.4/cce/x86_64/lib >>>>>>> -Wl,-rpath,/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64 >>>>>>> -L/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64 >>>>>>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux >>>>>>> -L/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux >>>>>>> -Wl,-rpath,/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0 >>>>>>> -L/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0 >>>>>>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib >>>>>>> -L/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib >>>>>>> -lpetsc -lkokkoskernels -lkokkoscontainers -lkokkoscore -lhipsparse >>>>>>> -lhipblas -lrocsparse -lrocsolver -lrocblas -lamdhip64 -lhsa-runtime64 >>>>>>> -lstdc++ -ldl -lpmi -lsci_cray_mpi -lsci_cray -lmpifort_cray -lmpi_cray >>>>>>> -lmpi_gtl_hsa -lxpmem -ldsmml -lpgas-shmem -lquadmath -lcrayacc_amdgpu >>>>>>> -lopenacc -lmodules -lfi -lcraymath -lf -lu -lcsup -lgfortran -lpthread >>>>>>> -lgcc_eh -lm -lclang_rt.craypgo-x86_64 -lclang_rt.builtins-x86_64 >>>>>>> -lquadmath -lstdc++ -ldl -o ex3k >>>>>>> In file included from ex3k.kokkos.cxx:3: >>>>>>> In file included from >>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscdmda_kokkos.hpp:4: >>>>>>> In file included from >>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec_kokkos.hpp:14: >>>>>>> In file included from >>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec.h:9: >>>>>>> In file included from >>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsys.h:42: >>>>>>> In file included from >>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsystypes.h:255: >>>>>>> In file included from >>>>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Complex.hpp:47: >>>>>>> In file included from >>>>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Atomic.hpp:212: >>>>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/impl/Kokkos_Atomic_Exchange.hpp:259:11: >>>>>>> error: no member named 'lock_address_host_space' in namespace >>>>>>> 'Kokkos::Impl::Kokkos::Impl'; did you mean simply 'lock_address_host_space'? >>>>>>> while (!Impl::lock_address_host_space((void*)dest)) >>>>>>> ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~ >>>>>>> lock_address_host_space >>>>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_HostSpace.hpp:83:6: >>>>>>> note: 'lock_address_host_space' declared here >>>>>>> bool lock_address_host_space(void* ptr); >>>>>>> ^ >>>>>>> In file included from ex3k.kokkos.cxx:3: >>>>>>> In file included from >>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscdmda_kokkos.hpp:4: >>>>>>> In file included from >>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec_kokkos.hpp:14: >>>>>>> In file included from >>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec.h:9: >>>>>>> In file included from >>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsys.h:42: >>>>>>> In file included from >>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsystypes.h:255: >>>>>>> In file included from >>>>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Complex.hpp:47: >>>>>>> In file included from >>>>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Atomic.hpp:212: >>>>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/impl/Kokkos_Atomic_Exchange.hpp:261:3: >>>>>>> error: no member named 'memory_fence' in namespace 'Kokkos::Impl::Kokkos'; >>>>>>> did you mean simply 'memory_fence'? >>>>>>> Kokkos::memory_fence(); >>>>>>> ^~~~~~~~~~~~~~~~~~~~ >>>>>>> memory_fence >>>>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/impl/Kokkos_Memory_Fence.hpp:53:6: >>>>>>> note: 'memory_fence' declared here >>>>>>> void memory_fence() { >>>>>>> ^ >>>>>>> In file included from ex3k.kokkos.cxx:3: >>>>>>> In file included from >>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscdmda_kokkos.hpp:4: >>>>>>> In file included from >>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec_kokkos.hpp:14: >>>>>>> In file included from >>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec.h:9: >>>>>>> In file included from >>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsys.h:42: >>>>>>> In file included from >>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsystypes.h:255: >>>>>>> In file included from >>>>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Complex.hpp:47: >>>>>>> In file included from >>>>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Atomic.hpp:212: >>>>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/impl/Kokkos_Atomic_Exchange.hpp:277:3: >>>>>>> error: no member named 'memory_fence' in namespace 'Kokkos::Impl::Kokkos'; >>>>>>> did you mean simply 'memory_fence'? >>>>>>> Kokkos::memory_fence(); >>>>>>> ^~~~~~~~~~~~~~~~~~~~ >>>>>>> memory_fence >>>>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/impl/Kokkos_Memory_Fence.hpp:53:6: >>>>>>> note: 'memory_fence' declared here >>>>>>> void memory_fence() { >>>>>>> ^ >>>>>>> >>>>>>

1 0