Mark,
On tulip, things with fortran went wonky when a `-fast` flag snuck into
the flags (snuck in being me copying a previous configure file and not
questioning the flags). The reason is that for clang, `-fast` implies
link time optimization (lto) for C/C++ code, but flang doesn't support
lto so things got weird. I suspect that gfortran does not either (but
perhaps the real question is why not use flang?)
It does work with GNU. I am doing this for an application and I want to avoid dictating the program env.
I'm (re)trying with -O0 just to check.
I look at configure and I don't see anything in your flags that would
trigger lto, but as Junchao says, it might be picking something up from
Kokkos, so perhaps this is the issue.
The configure files that I sent did not have Kokkos. They were stripped down.
I have asked ORNL, but don't expect help.
The person (Matt) that is helping me for other things is very helpful so perhaps he can come up with some ideas.
Otherwise I will tell the app that they need to use GNU for now.
Thanks,
Mark
Scott
On 2021-07-19 07:19, Mark Adams did write:
> Thanks, but this happens w/o Kokkos.
> I've stripped this down and attached good/bad logs without/with Fortran
> bindings.
> Hope this helps,
> Thanks again,
> Mark
>
> On Sun, Jul 18, 2021 at 12:00 PM Stefano Zampini <stefano.zampini@gmail.com>
> wrote:
>
> > This is probably kokkos pulling in the dependencies where compiling kokkos
> > source within PETSc.
> >
> > Il Dom 18 Lug 2021, 16:29 Mark Adams <mfadams@lbl.gov> ha scritto:
> >
> >> Whoops, this error was just from not telling gfortran to allow long lines.
> >>
> >> Anway, I did find that the when fortran bindings are enabled this OMP
> >> thing gets into the library. No idea how. I'll askm ORNL.
> >>
> >> 10:26 jczhang/fix-cray-mpicxx-includes/main=
> >> /gpfs/alpine/csc314/scratch/adams/petsc2$ nm
> >> /gpfs/alpine/csc314/scratch/adams/petsc2/arch-spock-opt-cray-kokkos/lib/libpetsc.so
> >> |g offload
> >> U .omp_offloading.img_cache.cray_amdgcn-amd-amdhsa
> >> U .omp_offloading.img_size.cray_amdgcn-amd-amdhsa
> >> U .omp_offloading.img_start.cray_amdgcn-amd-amdhsa
> >> 0000000001d457b0 T vecgetoffloadmask_
> >>
> >>
> >> On Sun, Jul 18, 2021 at 8:43 AM Mark Adams <mfadams@lbl.gov> wrote:
> >>
> >>> Ah, your test was not on Spock ... I have something working but this is
> >>> strange.
> >>>
> >>> I switched to the GNU ProgEnv. and it passed the Fortran test in
> >>> 'check', but this failed:
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>> *08:26 2 jczhang/fix-cray-mpicxx-includes/main=
> >>> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials$ make
> >>> PETSC_DIR=/gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-dbg-gnu
> >>> PETSC_ARCH="" ex40f90ftn -L/sw/spock/spack-envs/views/rocm-4.1.0/lib
> >>> -lamdhip64 -lhsa-runtime64 -fPIC -g -O0 -fPIC -g -O0
> >>> -I/gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-dbg-gnu/include
> >>> -I/sw/spock/spack-envs/views/rocm-4.1.0/include ex40f90.F90
> >>> -Wl,-rpath,/gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-dbg-gnu/lib
> >>> -L/gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-dbg-gnu/lib
> >>> -Wl,-rpath,/sw/spock/spack-envs/views/rocm-4.1.0/lib
> >>> -L/sw/spock/spack-envs/views/rocm-4.1.0/lib
> >>> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/ofi/gnu/9.1/lib
> >>> -L/opt/cray/pe/mpich/8.1.4/ofi/gnu/9.1/lib
> >>> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/gtl/lib
> >>> -L/opt/cray/pe/mpich/8.1.4/gtl/lib
> >>> -Wl,-rpath,/opt/cray/pe/libsci/21.04.1.1/GNU/9.1/x86_64/lib
> >>> <http://21.04.1.1/GNU/9.1/x86_64/lib>
> >>> -L/opt/cray/pe/libsci/21.04.1.1/GNU/9.1/x86_64/lib
> >>> <http://21.04.1.1/GNU/9.1/x86_64/lib>
> >>> -Wl,-rpath,/opt/cray/pe/pmi/6.0.10/lib -L/opt/cray/pe/pmi/6.0.10/lib
> >>> -Wl,-rpath,/opt/cray/pe/dsmml/0.1.4/dsmml/lib
> >>> -L/opt/cray/pe/dsmml/0.1.4/dsmml/lib
> >>> -Wl,-rpath,/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64
> >>> -L/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64
> >>> -Wl,-rpath,/opt/gcc/10.2.0/snos/lib/gcc/x86_64-suse-linux/10.2.0
> >>> -L/opt/gcc/10.2.0/snos/lib/gcc/x86_64-suse-linux/10.2.0
> >>> -Wl,-rpath,/opt/gcc/10.2.0/snos/lib64 -L/opt/gcc/10.2.0/snos/lib64
> >>> -Wl,-rpath,/sw/spock/spack-envs/views/rocm-4.1.0/llvm/lib
> >>> -L/sw/spock/spack-envs/views/rocm-4.1.0/llvm/lib
> >>> -Wl,-rpath,/opt/gcc/10.2.0/snos/lib -L/opt/gcc/10.2.0/snos/lib -lpetsc
> >>> -lhipsparse -lhipblas -lrocsparse -lrocsolver -lrocblas -lamdhip64
> >>> -lhsa-runtime64 -lstdc++ -ldl -lpmi -lmpi_gtl_hsa -lxpmem -lgfortran -lm
> >>> -lsci_gnu_82_mpi -lsci_gnu_82 -lmpifort_gnu_91 -lmpi_gnu_91 -ldsmml
> >>> -lgfortran -lquadmath -lpthread -lm -lgcc_s -lquadmath -lstdc++ -ldl -o
> >>> ex40f90ex40f90.F90:35:132: 35 | call
> >>> DMDACreate2d(PETSC_COMM_WORLD,DM_BOUNDARY_NONE,DM_BOUNDARY_NONE,DMDA_STENCIL_BOX,ten,ten,PETSC_DECIDE,PETSC_DECIDE,two,one,
> >>> & |
> >>> 1Error:
> >>> Line truncated at (1) [-Werror=line-truncation]ex40f90.F90:35:132:*
> >>> * ....*
> >>>
> >>> But ex5f (the Fortran check test) does work manually also (sanity check).
> >>>
> >>> Oddly I seem to have the cray compilers:
> >>>
> >>> 08:34 jczhang/fix-cray-mpicxx-includes/main=
> >>> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials$ which ftn
> >>> /opt/cray/pe/craype/2.7.6/bin/ftn
> >>>
> >>> yet it is GNU:
> >>>
> >>> 08:34 jczhang/fix-cray-mpicxx-includes/main=
> >>> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials$ ftn --version
> >>> GNU Fortran (GCC) 10.2.0 20200723 (Cray Inc.)
> >>> Copyright (C) 2020 Free Software Foundation, Inc.
> >>> This is free software; see the source for copying conditions. There is
> >>> NO
> >>> warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR
> >>> PURPOSE.
> >>>
> >>> The documentation says ftn should be here:
> >>>
> >>> 08:34 jczhang/fix-cray-mpicxx-includes/main=
> >>> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials$
> >>> $GCC_PATH/bin/gfortran --version
> >>> GNU Fortran (GCC) 10.2.0 20200723 (Cray Inc.)
> >>> Copyright (C) 2020 Free Software Foundation, Inc.
> >>> This is free software; see the source for copying conditions. There is
> >>> NO
> >>> warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR
> >>> PURPOSE.
> >>>
> >>> Same, but the path is not the same:
> >>>
> >>> 08:39 jczhang/fix-cray-mpicxx-includes/main=
> >>> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials$ echo $GCC_PATH
> >>> /opt/gcc/10.2.0
> >>>
> >>> Anyway, thanks for your help,
> >>> Mark
> >>>
> >>> On Sat, Jul 17, 2021 at 11:26 PM Junchao Zhang <junchao.zhang@gmail.com>
> >>> wrote:
> >>>
> >>>> Mark, I can reproduce this error with PrgEnv-cray, i.e., using the
> >>>> Cray compiler (clang-11). Previously I used PrgEnv-gnu, which did not have
> >>>> this error.
> >>>> Probably it is a problem of Spock. But I am not sure.
> >>>>
> >>>> --Junchao Zhang
> >>>>
> >>>>
> >>>> On Sat, Jul 17, 2021 at 10:17 AM Mark Adams <mfadams@lbl.gov> wrote:
> >>>>
> >>>>> And I can run a fortran test, with warnings, but C tests fail:
> >>>>>
> >>>>> 11:15 jczhang/fix-cray-mpicxx-includes/main=
> >>>>> /gpfs/alpine/csc314/scratch/adams/petsc$ make
> >>>>> PETSC_DIR=/gpfs/alpine/csc314/scratch/adams/petsc
> >>>>> PETSC_ARCH=arch-spock-opt-cray-kokkos -f gmakefile test
> >>>>> search='ts_utils_dmplexlandau_tutorials-ex1f90_0'
> >>>>> Using MAKEFLAGS: -- search=ts_utils_dmplexlandau_tutorials-ex1f90_0
> >>>>> PETSC_ARCH=arch-spock-opt-cray-kokkos
> >>>>> PETSC_DIR=/gpfs/alpine/csc314/scratch/adams/petsc
> >>>>> FC
> >>>>> arch-spock-opt-cray-kokkos/tests/ts/utils/dmplexlandau/tutorials/ex1f90.o
> >>>>> FLINKER
> >>>>> arch-spock-opt-cray-kokkos/tests/ts/utils/dmplexlandau/tutorials/ex1f90
> >>>>> /opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/bin/ld:
> >>>>> warning: alignment 128 of symbol
> >>>>> `$host_init$$runtime_init_for_iso_c_binding$iso_c_binding_' in
> >>>>> /opt/cray/pe/cce/11.0.4/cce/x86_64/lib/libmodules.so is smaller than 256 in
> >>>>> arch-spock-opt-cray-kokkos/tests/ts/utils/dmplexlandau/tutorials/ex1f90.o
> >>>>> /opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/bin/ld:
> >>>>> warning: alignment 64 of symbol `$data_init$iso_c_binding_' in
> >>>>> /opt/cray/pe/cce/11.0.4/cce/x86_64/lib/libmodules.so is smaller than 256 in
> >>>>> arch-spock-opt-cray-kokkos/tests/ts/utils/dmplexlandau/tutorials/ex1f90.o
> >>>>> TEST
> >>>>> arch-spock-opt-cray-kokkos/tests/counts/ts_utils_dmplexlandau_tutorials-ex1f90_0.counts
> >>>>> ok ts_utils_dmplexlandau_tutorials-ex1f90_0
> >>>>> ok diff-ts_utils_dmplexlandau_tutorials-ex1f90_0
> >>>>>
> >>>>>
> >>>>> On Sat, Jul 17, 2021 at 10:53 AM Mark Adams <mfadams@lbl.gov> wrote:
> >>>>>
> >>>>>> HUmm, I can not reproduce this.
> >>>>>>
> >>>>>> 10:49 jczhang/fix-cray-mpicxx-includes/main=
> >>>>>> /gpfs/alpine/csc314/scratch/adams/petsc$ make
> >>>>>> PETSC_DIR=/gpfs/alpine/csc314/scratch/adams/petsc
> >>>>>> PETSC_ARCH=arch-spock-dbg-kokkos check
> >>>>>> Running check examples to verify correct installation
> >>>>>> Using PETSC_DIR=/gpfs/alpine/csc314/scratch/adams/petsc and
> >>>>>> PETSC_ARCH=arch-spock-dbg-kokkos
> >>>>>> gmake[3]:
> >>>>>> [/gpfs/alpine/csc314/scratch/adams/petsc/lib/petsc/conf/rules:301:
> >>>>>> ex19.PETSc] Error 2 (ignored)
> >>>>>> *******************Error detected during compile or
> >>>>>> link!*******************
> >>>>>> See http://www.mcs.anl.gov/petsc/documentation/faq.html
> >>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials ex19
> >>>>>>
> >>>>>> *********************************************************************************
> >>>>>> cc -L/sw/spock/spack-envs/views/rocm-4.1.0/lib -lamdhip64
> >>>>>> -lhsa-runtime64 -fPIC -Wall -Wwrite-strings -Wno-strict-aliasing
> >>>>>> -Wno-unknown-pragmas -fstack-protector -Qunused-arguments
> >>>>>> -fvisibility=hidden -g -O0 -fPIC -Wall -Wwrite-strings
> >>>>>> -Wno-strict-aliasing -Wno-unknown-pragmas -fstack-protector
> >>>>>> -Qunused-arguments -fvisibility=hidden -g -O0
> >>>>>> -I/sw/spock/spack-envs/views/rocm-4.1.0/include
> >>>>>> -I/gpfs/alpine/csc314/scratch/adams/petsc/include
> >>>>>> -I/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/include
> >>>>>> -I/sw/spock/spack-envs/views/rocm-4.1.0/include
> >>>>>> -I/sw/spock/spack-envs/views/rocm-4.1.0/include ex19.c
> >>>>>> -Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib
> >>>>>> -L/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib
> >>>>>> -Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib
> >>>>>> -L/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib
> >>>>>> -Wl,-rpath,/sw/spock/spack-envs/views/rocm-4.1.0/lib
> >>>>>> -L/sw/spock/spack-envs/views/rocm-4.1.0/lib
> >>>>>> -Wl,-rpath,/opt/gcc/8.1.0/snos/lib64 -L/opt/gcc/8.1.0/snos/lib64
> >>>>>> -Wl,-rpath,/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib
> >>>>>> -L/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib
> >>>>>> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib
> >>>>>> -L/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib
> >>>>>> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/gtl/lib
> >>>>>> -L/opt/cray/pe/mpich/8.1.4/gtl/lib -Wl,-rpath,/opt/cray/pe/pmi/6.0.10/lib
> >>>>>> -L/opt/cray/pe/pmi/6.0.10/lib -Wl,-rpath,/opt/cray/pe/dsmml/0.1.4/dsmml/lib
> >>>>>> -L/opt/cray/pe/dsmml/0.1.4/dsmml/lib
> >>>>>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce/x86_64/lib
> >>>>>> -L/opt/cray/pe/cce/11.0.4/cce/x86_64/lib
> >>>>>> -Wl,-rpath,/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64
> >>>>>> -L/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64
> >>>>>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux
> >>>>>> -L/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux
> >>>>>> -Wl,-rpath,/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0
> >>>>>> -L/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0
> >>>>>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib
> >>>>>> -L/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib
> >>>>>> -lpetsc -lkokkoskernels -lkokkoscontainers -lkokkoscore -lhipsparse
> >>>>>> -lhipblas -lrocsparse -lrocsolver -lrocblas -lamdhip64 -lhsa-runtime64
> >>>>>> -lstdc++ -ldl -lpmi -lsci_cray_mpi -lsci_cray -lmpifort_cray -lmpi_cray
> >>>>>> -lmpi_gtl_hsa -lxpmem -ldsmml -lpgas-shmem -lquadmath -lcrayacc_amdgpu
> >>>>>> -lopenacc -lmodules -lfi -lcraymath -lf -lu -lcsup -lgfortran -lpthread
> >>>>>> -lgcc_eh -lm -lclang_rt.craypgo-x86_64 -lclang_rt.builtins-x86_64
> >>>>>> -lquadmath -lstdc++ -ldl -o ex19
> >>>>>> ld.lld: error:
> >>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib/libpetsc.so:
> >>>>>> undefined reference to .omp_offloading.img_start.cray_amdgcn-amd-amdhsa
> >>>>>> [--no-allow-shlib-undefined]
> >>>>>> ld.lld: error:
> >>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib/libpetsc.so:
> >>>>>> undefined reference to .omp_offloading.img_size.cray_amdgcn-amd-amdhsa
> >>>>>> [--no-allow-shlib-undefined]
> >>>>>> ld.lld: error:
> >>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib/libpetsc.so:
> >>>>>> undefined reference to .omp_offloading.img_cache.cray_amdgcn-amd-amdhsa
> >>>>>> [--no-allow-shlib-undefined]
> >>>>>> clang-11: error: linker command failed with exit code 1 (use -v to
> >>>>>> see invocation)
> >>>>>> gmake[4]: *** [<builtin>: ex19] Error 1
> >>>>>> gmake[3]:
> >>>>>> [/gpfs/alpine/csc314/scratch/adams/petsc/lib/petsc/conf/rules:301:
> >>>>>> ex3k.PETSc] Error 2 (ignored)
> >>>>>> *******************Error detected during compile or
> >>>>>> link!*******************
> >>>>>> See http://www.mcs.anl.gov/petsc/documentation/faq.html
> >>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials ex3k
> >>>>>>
> >>>>>> *********************************************************************************
> >>>>>> hipcc -fPIC -Wall -Wwrite-strings -Wno-strict-aliasing
> >>>>>> -Wno-unknown-pragmas -fvisibility=hidden -g -O0 -std=c++14
> >>>>>> -I/gpfs/alpine/csc314/scratch/adams/petsc/include
> >>>>>> -I/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/include
> >>>>>> -I/sw/spock/spack-envs/views/rocm-4.1.0/include
> >>>>>> -I/sw/spock/spack-envs/views/rocm-4.1.0/include -I/opt/cray/pe/libsci/
> >>>>>> 21.04.1.1/CRAY/9.0/x86_64/include
> >>>>>> -I/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/include
> >>>>>> -I/opt/cray/pe/pmi/6.0.10/include -I/opt/cray/pe/dsmml/0.1.4/dsmml//include
> >>>>>> -I/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/include
> >>>>>> -L/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib
> >>>>>> -L/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib
> >>>>>> -L/opt/cray/pe/mpich/8.1.4/gtl/lib -L/opt/cray/pe/pmi/6.0.10/lib
> >>>>>> -L/opt/cray/pe/dsmml/0.1.4/dsmml//lib
> >>>>>> -L/opt/cray/pe/cce/11.0.4/cce/x86_64/lib/pkgconfig/../
> >>>>>> -L/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64
> >>>>>> -Wl,--as-needed,-lsci_cray_mpi,--no-as-needed
> >>>>>> -Wl,--as-needed,-lsci_cray,--no-as-needed
> >>>>>> -Wl,--as-needed,-lmpi_cray,--no-as-needed
> >>>>>> -Wl,--as-needed,-ldsmml,--no-as-needed
> >>>>>> -Wl,--as-needed,-lpgas-shmem,--no-as-needed -lpmi -ldl -lmpi_gtl_hsa
> >>>>>> -lxpmem -lquadmath -lmodules -lfi -lcraymath -lf -lu -lcsup
> >>>>>> ex3k.kokkos.cxx
> >>>>>> -Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib
> >>>>>> -L/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib
> >>>>>> -Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib
> >>>>>> -L/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib
> >>>>>> -Wl,-rpath,/sw/spock/spack-envs/views/rocm-4.1.0/lib
> >>>>>> -L/sw/spock/spack-envs/views/rocm-4.1.0/lib
> >>>>>> -Wl,-rpath,/opt/gcc/8.1.0/snos/lib64 -L/opt/gcc/8.1.0/snos/lib64
> >>>>>> -Wl,-rpath,/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib
> >>>>>> -L/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib
> >>>>>> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib
> >>>>>> -L/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib
> >>>>>> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/gtl/lib
> >>>>>> -L/opt/cray/pe/mpich/8.1.4/gtl/lib -Wl,-rpath,/opt/cray/pe/pmi/6.0.10/lib
> >>>>>> -L/opt/cray/pe/pmi/6.0.10/lib -Wl,-rpath,/opt/cray/pe/dsmml/0.1.4/dsmml/lib
> >>>>>> -L/opt/cray/pe/dsmml/0.1.4/dsmml/lib
> >>>>>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce/x86_64/lib
> >>>>>> -L/opt/cray/pe/cce/11.0.4/cce/x86_64/lib
> >>>>>> -Wl,-rpath,/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64
> >>>>>> -L/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64
> >>>>>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux
> >>>>>> -L/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux
> >>>>>> -Wl,-rpath,/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0
> >>>>>> -L/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0
> >>>>>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib
> >>>>>> -L/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib
> >>>>>> -lpetsc -lkokkoskernels -lkokkoscontainers -lkokkoscore -lhipsparse
> >>>>>> -lhipblas -lrocsparse -lrocsolver -lrocblas -lamdhip64 -lhsa-runtime64
> >>>>>> -lstdc++ -ldl -lpmi -lsci_cray_mpi -lsci_cray -lmpifort_cray -lmpi_cray
> >>>>>> -lmpi_gtl_hsa -lxpmem -ldsmml -lpgas-shmem -lquadmath -lcrayacc_amdgpu
> >>>>>> -lopenacc -lmodules -lfi -lcraymath -lf -lu -lcsup -lgfortran -lpthread
> >>>>>> -lgcc_eh -lm -lclang_rt.craypgo-x86_64 -lclang_rt.builtins-x86_64
> >>>>>> -lquadmath -lstdc++ -ldl -o ex3k
> >>>>>> /usr/lib64/gcc/x86_64-suse-linux/7/../../../../x86_64-suse-linux/bin/ld:
> >>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib/libpetsc.so:
> >>>>>> undefined reference to `.omp_offloading.img_size.cray_amdgcn-amd-amdhsa'
> >>>>>> /usr/lib64/gcc/x86_64-suse-linux/7/../../../../x86_64-suse-linux/bin/ld:
> >>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib/libpetsc.so:
> >>>>>> undefined reference to `.omp_offloading.img_start.cray_amdgcn-amd-amdhsa'
> >>>>>> /usr/lib64/gcc/x86_64-suse-linux/7/../../../../x86_64-suse-linux/bin/ld:
> >>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib/libpetsc.so:
> >>>>>> undefined reference to `.omp_offloading.img_cache.cray_amdgcn-amd-amdhsa'
> >>>>>> clang-12: error: linker command failed with exit code 1 (use -v to
> >>>>>> see invocation)
> >>>>>> gmake[4]: ***
> >>>>>> [/gpfs/alpine/csc314/scratch/adams/petsc/lib/petsc/conf/test:26: ex3k]
> >>>>>> Error 1
> >>>>>> gmake[3]: [makefile:102: runex3k_kokkos] Error 2 (ignored)
> >>>>>> 1,25c1,4
> >>>>>> < atol=1e-50, rtol=1e-08, stol=1e-08, maxit=50, maxf=10000
> >>>>>> < Vec Object: Exact Solution 2 MPI processes
> >>>>>> < type: mpikokkos
> >>>>>> < Process [0]
> >>>>>> < 0.
> >>>>>> < 0.015625
> >>>>>> < 0.125
> >>>>>> < Process [1]
> >>>>>> < 0.421875
> >>>>>> < 1.
> >>>>>> < Vec Object: Forcing function 2 MPI processes
> >>>>>> < type: mpikokkos
> >>>>>> < Process [0]
> >>>>>> < 1e-72
> >>>>>> < 1.50024
> >>>>>> < 3.01563
> >>>>>> < Process [1]
> >>>>>> < 4.67798
> >>>>>> < 7.
> >>>>>> < 0 SNES Function norm 5.414682427127e+00
> >>>>>> < 1 SNES Function norm 2.952582418265e-01
> >>>>>> < 2 SNES Function norm 4.502293658739e-04
> >>>>>> < 3 SNES Function norm 1.389665806646e-09
> >>>>>> < Number of SNES iterations = 3
> >>>>>> < Norm of error 1.49752e-10 Iterations 3
> >>>>>> ---
> >>>>>> > slurmstepd: error: execve():
> >>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials/./ex3k: No such
> >>>>>> file or directory
> >>>>>> > slurmstepd: error: execve():
> >>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials/./ex3k: No such
> >>>>>> file or directory
> >>>>>> > srun: error: spock01: tasks 0-1: Exited with exit code 2
> >>>>>> > srun: launch/slurm: _step_signal: Terminating StepId=272979.0
> >>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials
> >>>>>> Possible problem with ex3k running with kokkos-kernels, diffs above
> >>>>>> =========================================
> >>>>>> *******************Error detected during compile or
> >>>>>> link!*******************
> >>>>>> See http://www.mcs.anl.gov/petsc/documentation/faq.html
> >>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials ex5f
> >>>>>> *********************************************************
> >>>>>> ftn -L/sw/spock/spack-envs/views/rocm-4.1.0/lib -lamdhip64
> >>>>>> -lhsa-runtime64 -fPIC -g -O0 -fPIC -g -O0
> >>>>>> -I/gpfs/alpine/csc314/scratch/adams/petsc/include
> >>>>>> -I/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/include
> >>>>>> -I/sw/spock/spack-envs/views/rocm-4.1.0/include ex5f.F90
> >>>>>> -Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib
> >>>>>> -L/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib
> >>>>>> -Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib
> >>>>>> -L/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib
> >>>>>> -Wl,-rpath,/sw/spock/spack-envs/views/rocm-4.1.0/lib
> >>>>>> -L/sw/spock/spack-envs/views/rocm-4.1.0/lib
> >>>>>> -Wl,-rpath,/opt/gcc/8.1.0/snos/lib64 -L/opt/gcc/8.1.0/snos/lib64
> >>>>>> -Wl,-rpath,/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib
> >>>>>> -L/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib
> >>>>>> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib
> >>>>>> -L/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib
> >>>>>> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/gtl/lib
> >>>>>> -L/opt/cray/pe/mpich/8.1.4/gtl/lib -Wl,-rpath,/opt/cray/pe/pmi/6.0.10/lib
> >>>>>> -L/opt/cray/pe/pmi/6.0.10/lib -Wl,-rpath,/opt/cray/pe/dsmml/0.1.4/dsmml/lib
> >>>>>> -L/opt/cray/pe/dsmml/0.1.4/dsmml/lib
> >>>>>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce/x86_64/lib
> >>>>>> -L/opt/cray/pe/cce/11.0.4/cce/x86_64/lib
> >>>>>> -Wl,-rpath,/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64
> >>>>>> -L/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64
> >>>>>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux
> >>>>>> -L/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux
> >>>>>> -Wl,-rpath,/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0
> >>>>>> -L/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0
> >>>>>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib
> >>>>>> -L/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib
> >>>>>> -lpetsc -lkokkoskernels -lkokkoscontainers -lkokkoscore -lhipsparse
> >>>>>> -lhipblas -lrocsparse -lrocsolver -lrocblas -lamdhip64 -lhsa-runtime64
> >>>>>> -lstdc++ -ldl -lpmi -lsci_cray_mpi -lsci_cray -lmpifort_cray -lmpi_cray
> >>>>>> -lmpi_gtl_hsa -lxpmem -ldsmml -lpgas-shmem -lquadmath -lcrayacc_amdgpu
> >>>>>> -lopenacc -lmodules -lfi -lcraymath -lf -lu -lcsup -lgfortran -lpthread
> >>>>>> -lgcc_eh -lm -lclang_rt.craypgo-x86_64 -lclang_rt.builtins-x86_64
> >>>>>> -lquadmath -lstdc++ -ldl -o ex5f
> >>>>>> /opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/bin/ld:
> >>>>>> warning: alignment 128 of symbol
> >>>>>> `$host_init$$runtime_init_for_iso_c_binding$iso_c_binding_' in
> >>>>>> /opt/cray/pe/cce/11.0.4/cce/x86_64/lib/libmodules.so is smaller than 256 in
> >>>>>> /tmp/pe_30429/ex5f_1.o
> >>>>>> /opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/bin/ld:
> >>>>>> warning: alignment 64 of symbol `$data_init$iso_c_binding_' in
> >>>>>> /opt/cray/pe/cce/11.0.4/cce/x86_64/lib/libmodules.so is smaller than 256 in
> >>>>>> /tmp/pe_30429/ex5f_1.o
> >>>>>> Fortran example src/snes/tutorials/ex5f run successfully with 1 MPI
> >>>>>> process
> >>>>>> Completed test examples
> >>>>>> 10:49 jczhang/fix-cray-mpicxx-includes/main=
> >>>>>> /gpfs/alpine/csc314/scratch/adams/petsc$ ll
> >>>>>>
> >>>>>> On Fri, Jul 16, 2021 at 11:28 PM Junchao Zhang <
> >>>>>> junchao.zhang@gmail.com> wrote:
> >>>>>>
> >>>>>>> Mark, I configured with Fortran binding enabled with main + MR !4175
> >>>>>>> <https://gitlab.com/petsc/petsc/-/merge_requests/4175>
> >>>>>>>
> >>>>>>> '--with-mpiexec=srun',
> >>>>>>> '--with-shared-libraries=1',
> >>>>>>> '--with-cc=cc',
> >>>>>>> '--with-cxx=CC',
> >>>>>>> '--with-fc=ftn',
> >>>>>>> '--with-fortran-bindings',
> >>>>>>> '--with-hip',
> >>>>>>> '--with-hipc=hipcc',
> >>>>>>> '--with-debugging',
> >>>>>>> '--CPPFLAGS=-I${ROCM_PATH}/include',
> >>>>>>> '--CXXPPFLAGS=-I${ROCM_PATH}/include',
> >>>>>>> '--CC_LINKER_FLAGS=-L${ROCM_PATH}/lib -lamdhip64 -lhsa-runtime64',
> >>>>>>> '--CXX_LINKER_FLAGS=-L${ROCM_PATH}/lib -lamdhip64 -lhsa-runtime64',
> >>>>>>> '--FC_LINKER_FLAGS=-L${ROCM_PATH}/lib -lamdhip64 -lhsa-runtime64',
> >>>>>>> '--COPTFLAGS=-g -O0',
> >>>>>>> '--CXXOPTFLAGS=-g -O0',
> >>>>>>> '--FOPTFLAGS=-g -O0',
> >>>>>>> '--download-kokkos',
> >>>>>>> '--download-kokkos-kernels',
> >>>>>>> '--download-kokkos-commit=3.4.01',
> >>>>>>> '--download-kokkos-kernels-commit=3.4.01',
> >>>>>>> '--with-kokkos-hip-arch=VEGA908',
> >>>>>>>
> >>>>>>> and 'make check' ran smoothly on a compute node
> >>>>>>>
> >>>>>>> $ make check
> >>>>>>> Running check examples to verify correct installation
> >>>>>>> Using PETSC_DIR=/ccs/home/jczhang/petsc and
> >>>>>>> PETSC_ARCH=arch-spock-cray-kokkos-dbg
> >>>>>>> C/C++ example src/snes/tutorials/ex19 run successfully with 1 MPI
> >>>>>>> process
> >>>>>>> C/C++ example src/snes/tutorials/ex19 run successfully with 2 MPI
> >>>>>>> processes
> >>>>>>> C/C++ example src/snes/tutorials/ex3k run successfully with
> >>>>>>> kokkos-kernels
> >>>>>>> Fortran example src/snes/tutorials/ex5f run successfully with 1 MPI
> >>>>>>> process
> >>>>>>>
> >>>>>>> Completed test examples
> >>>>>>>
> >>>>>>>
> >>>>>>> --Junchao Zhang
> >>>>>>>
> >>>>>>>
> >>>>>>> On Fri, Jul 16, 2021 at 6:04 PM Mark Adams <mfadams@lbl.gov> wrote:
> >>>>>>>
> >>>>>>>> And I find that this error, on non-Kokkos C tests, is fixed by
> >>>>>>>> turning the fortran bindings off:
> >>>>>>>>
> >>>>>>>> ld.lld: error:
> >>>>>>>> /gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/lib/libpetsc.so:
> >>>>>>>> undefined reference to .omp_offloading.img_start.cray_amdgcn-amd-amdhsa
> >>>>>>>> [--no-allow-shlib-undefined]
> >>>>>>>> ld.lld: error:
> >>>>>>>> /gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/lib/libpetsc.so:
> >>>>>>>> undefined reference to .omp_offloading.img_size.cray_amdgcn-amd-amdhsa
> >>>>>>>> [--no-allow-shlib-undefined]
> >>>>>>>> ld.lld: error:
> >>>>>>>> /gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/lib/libpetsc.so:
> >>>>>>>> undefined reference to .omp_offloading.img_cache.cray_amdgcn-amd-amdhsa
> >>>>>>>> [--no-allow-shlib-undefined]
> >>>>>>>>
> >>>>>>>> On Fri, Jul 16, 2021 at 3:53 PM Mark Adams <mfadams@lbl.gov> wrote:
> >>>>>>>>
> >>>>>>>>> Not complex. THis has some overlap with my problem w/o Kokkos.
> >>>>>>>>>
> >>>>>>>>> On Fri, Jul 16, 2021 at 12:54 PM Junchao Zhang <
> >>>>>>>>> junchao.zhang@gmail.com> wrote:
> >>>>>>>>>
> >>>>>>>>>> Do you use complex? post your configure.log.
> >>>>>>>>>>
> >>>>>>>>>> --Junchao Zhang
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> On Fri, Jul 16, 2021 at 9:47 AM Mark Adams <mfadams@lbl.gov>
> >>>>>>>>>> wrote:
> >>>>>>>>>>
> >>>>>>>>>>> The simple Kokkos example is failing for me on Spock.
> >>>>>>>>>>> Any ideas?
> >>>>>>>>>>> Thanks,
> >>>>>>>>>>>
> >>>>>>>>>>> 10:44 main *=
> >>>>>>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials$ make
> >>>>>>>>>>> PETSC_DIR=/gpfs/alpine/csc314/scratch/adams/petsc
> >>>>>>>>>>> PETSC_ARCH=arch-spock-opt-cray-kokkos ex3k
> >>>>>>>>>>> MPICH_CXX="hipcc" OMPI_CXX="hipcc" CC
> >>>>>>>>>>> -L/sw/spock/spack-envs/views/rocm-4.1.0/lib -lhsa-runtime64
> >>>>>>>>>>> -L/sw/spock/spack-envs/views/rocm-4.1.0/lib -lamdhip64 -lhsa-runtime64
> >>>>>>>>>>> -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas
> >>>>>>>>>>> -fstack-protector -fvisibility=hidden -g -O2 -fPIC -Wall -Wwrite-strings
> >>>>>>>>>>> -Wno-strict-aliasing -Wno-unknown-pragmas -fvisibility=hidden -g -O3
> >>>>>>>>>>> -std=c++14 -I/gpfs/alpine/csc314/scratch/adams/petsc/include
> >>>>>>>>>>> -I/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-opt-cray-kokkos/include
> >>>>>>>>>>> -I/gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include
> >>>>>>>>>>> -I/sw/spock/spack-envs/views/rocm-4.1.0/include
> >>>>>>>>>>> -I/sw/spock/spack-envs/views/rocm-4.1.0/include ex3k.kokkos.cxx
> >>>>>>>>>>> -Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-opt-cray-kokkos/lib
> >>>>>>>>>>> -L/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-opt-cray-kokkos/lib
> >>>>>>>>>>> -Wl,-rpath,/gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/lib
> >>>>>>>>>>> -L/gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/lib
> >>>>>>>>>>> -Wl,-rpath,/sw/spock/spack-envs/views/rocm-4.1.0/lib
> >>>>>>>>>>> -L/sw/spock/spack-envs/views/rocm-4.1.0/lib
> >>>>>>>>>>> -Wl,-rpath,/opt/gcc/8.1.0/snos/lib64 -L/opt/gcc/8.1.0/snos/lib64
> >>>>>>>>>>> -Wl,-rpath,/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib
> >>>>>>>>>>> -L/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib
> >>>>>>>>>>> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib
> >>>>>>>>>>> -L/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib
> >>>>>>>>>>> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/gtl/lib
> >>>>>>>>>>> -L/opt/cray/pe/mpich/8.1.4/gtl/lib -Wl,-rpath,/opt/cray/pe/pmi/6.0.10/lib
> >>>>>>>>>>> -L/opt/cray/pe/pmi/6.0.10/lib -Wl,-rpath,/opt/cray/pe/dsmml/0.1.4/dsmml/lib
> >>>>>>>>>>> -L/opt/cray/pe/dsmml/0.1.4/dsmml/lib
> >>>>>>>>>>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce/x86_64/lib
> >>>>>>>>>>> -L/opt/cray/pe/cce/11.0.4/cce/x86_64/lib
> >>>>>>>>>>> -Wl,-rpath,/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64
> >>>>>>>>>>> -L/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64
> >>>>>>>>>>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux
> >>>>>>>>>>> -L/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux
> >>>>>>>>>>> -Wl,-rpath,/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0
> >>>>>>>>>>> -L/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0
> >>>>>>>>>>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib
> >>>>>>>>>>> -L/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib
> >>>>>>>>>>> -lpetsc -lkokkoskernels -lkokkoscontainers -lkokkoscore -lhipsparse
> >>>>>>>>>>> -lhipblas -lrocsparse -lrocsolver -lrocblas -lamdhip64 -lhsa-runtime64
> >>>>>>>>>>> -lstdc++ -ldl -lpmi -lsci_cray_mpi -lsci_cray -lmpifort_cray -lmpi_cray
> >>>>>>>>>>> -lmpi_gtl_hsa -lxpmem -ldsmml -lpgas-shmem -lquadmath -lcrayacc_amdgpu
> >>>>>>>>>>> -lopenacc -lmodules -lfi -lcraymath -lf -lu -lcsup -lgfortran -lpthread
> >>>>>>>>>>> -lgcc_eh -lm -lclang_rt.craypgo-x86_64 -lclang_rt.builtins-x86_64
> >>>>>>>>>>> -lquadmath -lstdc++ -ldl -o ex3k
> >>>>>>>>>>> In file included from ex3k.kokkos.cxx:3:
> >>>>>>>>>>> In file included from
> >>>>>>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscdmda_kokkos.hpp:4:
> >>>>>>>>>>> In file included from
> >>>>>>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec_kokkos.hpp:14:
> >>>>>>>>>>> In file included from
> >>>>>>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec.h:9:
> >>>>>>>>>>> In file included from
> >>>>>>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsys.h:42:
> >>>>>>>>>>> In file included from
> >>>>>>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsystypes.h:255:
> >>>>>>>>>>> In file included from
> >>>>>>>>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Complex.hpp:47:
> >>>>>>>>>>> In file included from
> >>>>>>>>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Atomic.hpp:212:
> >>>>>>>>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/impl/Kokkos_Atomic_Exchange.hpp:259:11:
> >>>>>>>>>>> error: no member named 'lock_address_host_space' in namespace
> >>>>>>>>>>> 'Kokkos::Impl::Kokkos::Impl'; did you mean simply 'lock_address_host_space'?
> >>>>>>>>>>> while (!Impl::lock_address_host_space((void*)dest))
> >>>>>>>>>>> ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> >>>>>>>>>>> lock_address_host_space
> >>>>>>>>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_HostSpace.hpp:83:6:
> >>>>>>>>>>> note: 'lock_address_host_space' declared here
> >>>>>>>>>>> bool lock_address_host_space(void* ptr);
> >>>>>>>>>>> ^
> >>>>>>>>>>> In file included from ex3k.kokkos.cxx:3:
> >>>>>>>>>>> In file included from
> >>>>>>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscdmda_kokkos.hpp:4:
> >>>>>>>>>>> In file included from
> >>>>>>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec_kokkos.hpp:14:
> >>>>>>>>>>> In file included from
> >>>>>>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec.h:9:
> >>>>>>>>>>> In file included from
> >>>>>>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsys.h:42:
> >>>>>>>>>>> In file included from
> >>>>>>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsystypes.h:255:
> >>>>>>>>>>> In file included from
> >>>>>>>>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Complex.hpp:47:
> >>>>>>>>>>> In file included from
> >>>>>>>>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Atomic.hpp:212:
> >>>>>>>>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/impl/Kokkos_Atomic_Exchange.hpp:261:3:
> >>>>>>>>>>> error: no member named 'memory_fence' in namespace 'Kokkos::Impl::Kokkos';
> >>>>>>>>>>> did you mean simply 'memory_fence'?
> >>>>>>>>>>> Kokkos::memory_fence();
> >>>>>>>>>>> ^~~~~~~~~~~~~~~~~~~~
> >>>>>>>>>>> memory_fence
> >>>>>>>>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/impl/Kokkos_Memory_Fence.hpp:53:6:
> >>>>>>>>>>> note: 'memory_fence' declared here
> >>>>>>>>>>> void memory_fence() {
> >>>>>>>>>>> ^
> >>>>>>>>>>> In file included from ex3k.kokkos.cxx:3:
> >>>>>>>>>>> In file included from
> >>>>>>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscdmda_kokkos.hpp:4:
> >>>>>>>>>>> In file included from
> >>>>>>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec_kokkos.hpp:14:
> >>>>>>>>>>> In file included from
> >>>>>>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec.h:9:
> >>>>>>>>>>> In file included from
> >>>>>>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsys.h:42:
> >>>>>>>>>>> In file included from
> >>>>>>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsystypes.h:255:
> >>>>>>>>>>> In file included from
> >>>>>>>>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Complex.hpp:47:
> >>>>>>>>>>> In file included from
> >>>>>>>>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Atomic.hpp:212:
> >>>>>>>>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/impl/Kokkos_Atomic_Exchange.hpp:277:3:
> >>>>>>>>>>> error: no member named 'memory_fence' in namespace 'Kokkos::Impl::Kokkos';
> >>>>>>>>>>> did you mean simply 'memory_fence'?
> >>>>>>>>>>> Kokkos::memory_fence();
> >>>>>>>>>>> ^~~~~~~~~~~~~~~~~~~~
> >>>>>>>>>>> memory_fence
> >>>>>>>>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/impl/Kokkos_Memory_Fence.hpp:53:6:
> >>>>>>>>>>> note: 'memory_fence' declared here
> >>>>>>>>>>> void memory_fence() {
> >>>>>>>>>>> ^
> >>>>>>>>>>>
> >>>>>>>>>>
--
Scott Kruger
Tech-X Corporation kruger@txcorp.com
5621 Arapahoe Ave, Suite A Phone: (720) 466-3196
Boulder, CO 80303 Fax: (303) 448-7756