April 2021 - petsc-dev - mailman3.cels.anl.gov

Re: [petsc-dev] p4est w/o MPI
by Matthew Knepley 16 Apr '21

16 Apr '21

You would have to 1) Modify p4est to use mpiuni.h 2) Fix MPIUNI until p4est works with it 3) Fixup the p4est modification to be done in configure Matt On Fri, Apr 16, 2021 at 10:23 AM Mark Adams <mfadams(a)lbl.gov> wrote: > I don't have MPI (Fugaku w/ gcc) and p4est seems to need it. Is there a > work around? > Thanks, > Mark > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>

1 0

p4est w/o MPI
by Mark Adams 16 Apr '21

16 Apr '21

I don't have MPI (Fugaku w/ gcc) and p4est seems to need it. Is there a work around? Thanks, Mark

1 0

Re: [petsc-dev] Error on Fugaku
by Mark Adams 16 Apr '21

16 Apr '21

Kokkos-Serial is working on Landau ex2, but I get this error with OMP. gmake[2]: *** [src/CMakeFiles/kokkoskernels.dir/build.make:1040: src/CMakeFiles/kokkoskernels.dir/impl/generated_specializations_cpp/spmv/Sparse_spmv_mv_struct_eti_DOUBLE_ORDINAL_INT_OFFSET_INT_LAYOUTLEFT_EXECSPACE_SERIAL_MEMSPACE_HOSTSPACE.cpp.o] Error 9 Good old Error 9. The Kokkos people (Trott) have a Fujitsu box that they seem willing to dive in. I will run this by them. On Thu, Apr 15, 2021 at 3:11 PM Satish Balay <balay(a)mcs.anl.gov> wrote: > BTW: > > >>>> > login6$ mpifccpx -fopenmp -v sizeof.c > fccpx: Fujitsu C/C++ Compiler 4.5.0 tcsds-1.2.31 > simulating gcc version 6.1 > /opt/FJSVxtclanga/tcsds-1.2.31/bin/../lib/ccpcompx --preinclude > /opt/FJSVxtclanga/tcsds-1.2.31/bin/../lib/fcc.pre --gcc -Dunix -Dlinux > -D__FUJITSU -D__FCC_major__=4 -D__FCC_minor__=5 -D__FCC_patchlevel__=0 > -D__FCC_version__="4.5.0" -D__aarch64__ -D__unix -D_OPENMP=201107 > -D__fcc_version__=0x800 -D__fcc_version=800 -D__USER_LABEL_PREFIX__= > -D__ARM_ARCH=8 -D__ARM_FEATURE_SVE -D__FP_FAST_FMA -D__ELF__ -D__unix__ > -D__linux__ -D__linux -Asystem(unix) -D__LIBC_6B -D__LP64__ -D_LP64 --K=omp > -I/opt/FJSVxtclanga/.common/MELI024/include/mpi/fujitsu -D_REENTRANT > --K=noocl -D_REENTRANT -D__MT__ --zmode=64 > --sys_include=/opt/FJSVxtclanga/tcsds-1.2.31/bin/../include > --sys_include=/opt/FJSVxos/devkit/aarch64/rfs/usr/include sizeof.c -- > -ncmdname=fccpx -zcfc=target_sve -Kopenmp -V -Kthreadsafe -Klargepage -otmp > /home/ra010009/a04201/asmHUx9OT.s > ccpcompx: Fujitsu C/C++ Compiler 4.5.0 (Mar 4 2021 13:29:06) > /opt/FJSVxtclanga/tcsds-1.2.31/bin/../bin/checklogpx -v -zlangC MPI:V4.5.0 > -fopenmp -v /home/ra010009/a04201/asmHUx9OT.s > /opt/FJSVxos/devkit/aarch64/bin/aarch64-linux-gnu-as -o > /home/ra010009/a04201/fccmPXe0C.o -march=armv8.3-a+sve -v > /home/ra010009/a04201/asmHUx9OT.s > GNU assembler version 2.30 (aarch64-linux-gnu) using BFD version version > 2.30-49.el8 > /opt/FJSVxos/devkit/aarch64/bin/aarch64-linux-gnu-ld --build-id > --sysroot=/opt/FJSVxos/devkit/aarch64/rfs --eh-frame-hdr -m aarch64linux > -dynamic-linker /lib/ld-linux-aarch64.so.1 > /opt/FJSVxos/devkit/aarch64/rfs/usr/lib64/crt1.o > /opt/FJSVxos/devkit/aarch64/rfs/usr/lib64/crti.o > /opt/FJSVxos/devkit/aarch64/lib/gcc/aarch64-linux-gnu/8/crtbegin.o > -lfjprofmpi -L/opt/FJSVxos/devkit/aarch64/lib/gcc/aarch64-linux-gnu/8 -v > /home/ra010009/a04201/fccmPXe0C.o -L/opt/FJSVxtclanga/.common/MELI024/lib64 > -lmpi -lfjstring_internal > -L/opt/FJSVxos/devkit/aarch64/lib/gcc/aarch64-linux-gnu/8/../../../../aarch64-linux-gnu/lib/../lib64 > /opt/FJSVxtclanga/tcsds-1.2.31/bin/../lib64/fjcrt0.o > /opt/FJSVxtclanga/tcsds-1.2.31/bin/../lib64/fjlang08.o > /opt/FJSVxtclanga/tcsds-1.2.31/bin/../lib64/fjomp.o > /opt/FJSVxtclanga/tcsds-1.2.31/bin/../lib64/fjhpctag.o > -L/opt/FJSVxtclanga/tcsds-1.2.31/bin/../lib64 -lfjomphk -lfjomp -lfj90i > -lfj90fmt_sve -lfj90f -lfjsrcinfo -lfjcrt -lfjompcrt -lfjprofcore > -lfjprofomp -lmpg -lm -l > atomic -lpthread -L/opt/FJSVxos/devkit/aarch64/rfs/usr/lib64 > -L/opt/FJSVxos/devkit/aarch64/rfs/opt/FJSVxos/mmm/lib64 > -L/opt/FJSVxtclanga/tcsds-1.2.31/bin/../lib64/nofjobj > -T/opt/FJSVxos/devkit/aarch64/rfs/opt/FJSVxos/mmm/util/bss-2mb.lds -lm -lrt > -lpthread -lelf -lz -ldl -lgcc_s -lgcc -lc -lgcc_s -lgcc -lc > /opt/FJSVxos/devkit/aarch64/lib/gcc/aarch64-linux-gnu/8/crtend.o > /opt/FJSVxos/devkit/aarch64/rfs/usr/lib64/crtn.o > GNU ld version 2.30-49.el8 > <<<< > > i.e -fopenmp is internally mapped to -Kopenmp > > Satish > > On Wed, 14 Apr 2021, Mark Adams wrote: > > > The compiler: > > > > FCC (FCC) 4.5.0 20210304 > > > > *simulating gcc version 6.1*Copyright FUJITSU LIMITED 2019-2021 > > > > and > > -Kopenmp specifies to enable the Specification of OpenMP > Application > > Program Interface. The -Knoopenmp option invalidates the -Kopenmp > > option. -Knoopenmp is set by default. > > > > When the -Kopenmp option is specified, -mt is assumed. > > > > > > *The -Kopenmp option is needed if an object program compiled with it > > exists in the command line as an input file.* > > > > I am going to try w/o Kopenmp. > > > > On Wed, Apr 14, 2021 at 8:15 PM Satish Balay <balay(a)mcs.anl.gov> wrote: > > > > > I don't understand kokkos build errors. > > > > > > For one - kokkos-kernels are taking long to build. And on the back end > - > > > I'm getting into time limit [and terminated] > > > > > > > Kokkos take a very long time in configure on Summit. > > > > Thanks, > > > > > > > > > > Trying a build on the front-end - its running - for a long time.. > > > > > > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ > > > COMMAND > > > 4016901 > > > a04201 20 0 11.0g 10.7g 13876 R 99.3 11.6 66:22.35 ccpcompx > > > > > > > > > The above compile process has been running for more than an hour. Ok - > its > > > finally done. > > > > > > When I enable openmp - kokkos cmake completes - but I get errors at > build > > > time [BTW: I don't understand difference between -fopenmp and > -Kopenmp] > > > > > > login6$ > > > > ../petsc.save/arch-linux2-c-opt/lib/petsc/conf/reconfigure-arch-linux2-c-opt.py > > > --with-p4est-dir=$HOME/p4est-install > --with-zlib-dir=$HOME/p4est-install > > > -download-kokkos --download-kokkos-kernelsx > > > --download-kokkos-commit=origin/develop > > > --download-kokkos-kernels-commit=origin/develop --download-cmake= > > > > https://github.com/Kitware/CMake/releases/download/v3.20.1/cmake-3.20.1.tar… > > > '--download-kokkos-cmake-arguments=-DBUILD_TESTING=OFF > > > -DKokkos_ENABLE_LIBDL=OFF -DKokkos_ENABLE_AGGRESSIVE_VECTORIZATION=ON > > > -DKokkos_ENABLE_OPENMP=ON' --ignoreLinkOutput=1 --with-openmp=1 > > > > > > > > > > /a04201/petsc.x/arch-linux2-c-opt/externalpackages/git.kokkos/petsc-build/core/src > > > > -I/vol0004/ra010009/a04201/petsc.x/arch-linux2-c-opt/externalpackages/git.kokkos/core/src > > > > -I/vol0004/ra010009/a04201/petsc.x/arch-linux2-c-opt/externalpackages/git.kokkos/petsc-build > > > -O -fopenmp -fPIC -fopenmp -O -fopenmp -fPIC -fopenmp -fopenmp > -std=c++14 > > > -o CMakeFiles/kokkoscore.dir/OpenMP/Kokkos_OpenMP_Task.cpp.o -c > > > > /vol0004/ra010009/a04201/petsc.x/arch-linux2-c-opt/externalpackages/git.kokkos/core/src/OpenMP/Kokkos_OpenMP_Task.cpp > > > gmake[2]: Leaving directory > > > > '/vol0004/ra010009/a04201/petsc.x/arch-linux2-c-opt/externalpackages/git.kokkos/petsc-build' > > > gmake[1]: Leaving directory > > > > '/vol0004/ra010009/a04201/petsc.x/arch-linux2-c-opt/externalpackages/git.kokkos/petsc-build'/home/ra010009/a04201/asmqkagl0.s: > > > Assembler messages: > > > /home/ra010009/a04201/asmqkagl0.s:20333: Error: symbol `.LEHB41' is > > > already defined > > > /home/ra010009/a04201/asmqkagl0.s:21334: Error: symbol `.LEHB41' is > > > already defined > > > gmake[2]: *** [core/src/CMakeFiles/kokkoscore.dir/build.make:312: > > > core/src/CMakeFiles/kokkoscore.dir/OpenMP/Kokkos_OpenMP_Exec.cpp.o] > Error 1 > > > > > > Note: I have a successful configure run with > p4est+kokkos+kokkos-kernels, > > > without-openmp > > > [for this - I had to build p4est on the back-end, then switch over to > > > front-end - and build the reset of the packages] > > > > > > Now I'm stuck at compiling petsc sources - > > > src/vec/is/sf/impls/basic/kokkos/sfkok.kokkos.cxx > > > > > > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ > > > COMMAND > > > 56491 > > > a04201 20 0 4944176 4.5g 11216 R 98.4 4.9 12:45.52 ccpcompx > > > > > > > > > > > > Satish > > > > > > On Wed, 14 Apr 2021, Mark Adams wrote: > > > > > > > PETSc seems to be skipping the compiler that it should use: > > > > > > > > -- Check for working CXX compiler: > > > > /opt/FJSVxtclanga/tcsds-1.2.31/bin/mpiFCC - skipped > > > > > > > > I see -fopenmp was added. This might be wrong. I use -Kopenmp. > > > > > > > > --help says: > > > > > > > > -fopenmp > > > > > > > > The -fopenmp option specifies to enable Specification of > OpenMP > > > > Appli- > > > > cation Program Interface. > > > > > > > > When the -fopenmp option is specified, -mt is set. > > > > > > > > The -fopenmp option is needed if an object program compiled > with > > > the > > > > -fopenmp option exists in the command line as input files. > > > > > > > > Any ideas? > > > > > > > > > > > > On Wed, Apr 14, 2021 at 5:17 PM Satish Balay <balay(a)mcs.anl.gov> > wrote: > > > > > > > > > The following is working for me on the compute node [its completed > > > kokkos > > > > > (core) build - and is now building kokkos-kernels > > > > > > > > > > ~/petsc.save/arch-arm/lib/petsc/conf/reconfigure-arch-arm.py > > > > > --download-p4est --download-zlib --download-kokkos > > > > > --download-kokkos-kernels --download-kokkos-commit=origin/develop > > > > > --download-cmake= > > > > > > > > > https://github.com/Kitware/CMake/releases/download/v3.20.1/cmake-3.20.1.tar… > > > > > > > > > > -DKokkos_ENABLE_OPENMP=ON give me some issue - so I didn't try any > of > > > the > > > > > additional options.. > > > > > > > > > > Satish > > > > > > > > > > On Wed, 14 Apr 2021, Mark Adams wrote: > > > > > > > > > > > Satish, I get this error. > > > > > > > > > > > > I wonder if this syntax is wrong: > > > > > > > > > > > > '--download-kokkos-cmake-arguments=-DCMAKE_BUILD_TYPE=Release > > > > > > -DBUILD_TESTING=OFF -DKokkos_ENABLE_LIBDL=OFF > > > -DKokkos_ENABLE_OPENMP=ON > > > > > > -DKokkos_ENABLE_SERIAL=ON > > > -DKokkos_ENABLE_AGGRESSIVE_VECTORIZATION=ON', > > > > > > > > > > > > > > > > > > > > > > > > > > > ============================================================================================= > > > > > > Configuring KOKKOS with cmake; > this may > > > > > take > > > > > > several minutes > > > > > > > > > > > > > > > > > > > > > ============================================================================================= > > > > > > > > > > > > Executing: /usr/bin/cmake .. > > > > > > -DCMAKE_INSTALL_PREFIX=/home/ra010009/a04199/petsc/arch-arm > > > > > > > > > > > > > > > -DCMAKE_INSTALL_NAME_DIR:STRING="/home/ra010009/a04199/petsc/arch-arm/lib" > > > > > > -DCMAKE_INSTALL_LIBDIR:STRING="lib" -DCMAKE_VERBOSE_MAKEFILE=1 > > > > > > -DCMAKE_BUILD_TYPE=Release -DCMAKE_C_COMPILER="mpifcc" > > > > > > -DMPI_C_COMPILER="mpifcc" -DCMAKE_AR=/usr/bin/ar > > > > > > -DCMAKE_RANLIB=/usr/bin/ranlib -DCMAKE_C_FLAGS:STRING="-fPIC > > > > > -Kfast,openmp > > > > > > -fopenmp" -DCMAKE_C_FLAGS_DEBUG:STRING="-fPIC -Kfast,openmp > -fopenmp" > > > > > > -DCMAKE_C_FLAGS_RELEASE:STRING="-fPIC -Kfast,openmp -fopenmp" > > > > > > -DCMAKE_CXX_COMPILER="mpiFCC" -DMPI_CXX_COMPILER="mpiFCC" > > > > > > -DCMAKE_CXX_FLAGS:STRING="-Kfast,openmp -fopenmp -fPIC -fopenmp" > > > > > > -DCMAKE_CXX_FLAGS_DEBUG:STRING="-Kfast,openmp -fopenmp -fPIC > > > -fopenmp" > > > > > > -DCMAKE_CXX_FLAGS_RELEASE:STRING="-Kfast,openmp -fopenmp -fPIC > > > -fopenmp" > > > > > > -DCMAKE_Fortran_COMPILER="mpifrt" -DMPI_Fortran_COMPILER="mpifrt" > > > > > > -DCMAKE_Fortran_FLAGS:STRING="-fPIC -O -fopenmp" > > > > > > -DCMAKE_Fortran_FLAGS_DEBUG:STRING="-fPIC -O -fopenmp" > > > > > > -DCMAKE_Fortran_FLAGS_RELEASE:STRING="-fPIC -O -fopenmp" > > > > > > -DCMAKE_EXE_LINKER_FLAGS:STRING=" -fopenmp" > > > -DBUILD_SHARED_LIBS:BOOL=ON > > > > > > -DUSE_XSDK_DEFAULTS=YES -DXSDK_ENABLE_DEBUG=NO > > > > > > -DCMAKE_INSTALL_RPATH_USE_LINK_PATH:BOOL=ON > > > > > > -DCMAKE_BUILD_WITH_INSTALL_RPATH:BOOL=ON -DKokkos_ENABLE_MPI=ON > > > > > > -DKokkos_ENABLE_SERIAL=ON -DKokkos_ENABLE_OPENMP=ON > > > > > > -DCMAKE_CXX_STANDARD="14" -DCMAKE_BUILD_TYPE=Release > > > -DBUILD_TESTING=OFF > > > > > > -DKokkos_ENABLE_LIBDL=OFF -DKokkos_ENABLE_OPENMP=ON > > > > > > -DKokkos_ENABLE_SERIAL=ON > -DKokkos_ENABLE_AGGRESSIVE_VECTORIZATION=ON > > > > > > stdout: > > > > > > -- Setting default Kokkos CXX standard to 14 > > > > > > -- The CXX compiler identification is Fujitsu > > > > > > -- Check for working CXX compiler: > > > > > /opt/FJSVxtclanga/tcsds-1.2.31/bin/mpiFCC > > > > > > -- Check for working CXX compiler: > > > > > > /opt/FJSVxtclanga/tcsds-1.2.31/bin/mpiFCC -- works > > > > > > -- Detecting CXX compiler ABI info > > > > > > -- Detecting CXX compiler ABI info - done > > > > > > -- The project name is: Kokkos > > > > > > -- Configuring incomplete, errors occurred! > > > > > > See also > > > > > > > > > > > > > > > "/vol0004/ra010009/a04199/petsc/arch-arm/externalpackages/git.kokkos/petsc-build/CMakeFiles/CMakeOutput.log". > > > > > > Error configuring KOKKOS with cmake Could not > > > execute > > > > > > "['/usr/bin/cmake .. > > > > > > -DCMAKE_INSTALL_PREFIX=/home/ra010009/a04199/petsc/arch-arm > > > > > > > > > > > > > > > -DCMAKE_INSTALL_NAME_DIR:STRING="/home/ra010009/a04199/petsc/arch-arm/lib" > > > > > > -DCMAKE_INSTALL_LIBDIR:STRING="lib" -DCMAKE_VERBOSE_MAKEFILE=1 > > > > > > -DCMAKE_BUILD_TYPE=Release -DCMAKE_C_COMPILER="mpifcc" > > > > > > -DMPI_C_COMPILER="mpifcc" -DCMAKE_AR=/usr/bin/ar > > > > > > -DCMAKE_RANLIB=/usr/bin/ranlib -DCMAKE_C_FLAGS:STRING="-fPIC > > > > > -Kfast,openmp > > > > > > -fopenmp" -DCMAKE_C_FLAGS_DEBUG:STRING="-fPIC -Kfast,openmp > -fopenmp" > > > > > > -DCMAKE_C_FLAGS_RELEASE:STRING="-fPIC -Kfast,openmp -fopenmp" > > > > > > -DCMAKE_CXX_COMPILER="mpiFCC" -DMPI_CXX_COMPILER="mpiFCC" > > > > > > -DCMAKE_CXX_FLAGS:STRING="-Kfast,openmp -fopenmp -fPIC -fopenmp" > > > > > > -DCMAKE_CXX_FLAGS_DEBUG:STRING="-Kfast,openmp -fopenmp -fPIC > > > -fopenmp" > > > > > > -DCMAKE_CXX_FLAGS_RELEASE:STRING="-Kfast,openmp -fopenmp -fPIC > > > -fopenmp" > > > > > > -DCMAKE_Fortran_COMPILER="mpifrt" -DMPI_Fortran_COMPILER="mpifrt" > > > > > > -DCMAKE_Fortran_FLAGS:STRING="-fPIC -O -fopenmp" > > > > > > -DCMAKE_Fortran_FLAGS_DEBUG:STRING="-fPIC -O -fopenmp" > > > > > > -DCMAKE_Fortran_FLAGS_RELEASE:STRING="-fPIC -O -fopenmp" > > > > > > -DCMAKE_EXE_LINKER_FLAGS:STRING=" -fopenmp" > > > -DBUILD_SHARED_LIBS:BOOL=ON > > > > > > -DUSE_XSDK_DEFAULTS=YES -DXSDK_ENABLE_DEBUG=NO > > > > > > -DCMAKE_INSTALL_RPATH_USE_LINK_PATH:BOOL=ON > > > > > > -DCMAKE_BUILD_WITH_INSTALL_RPATH:BOOL=ON -DKokkos_ENABLE_MPI=ON > > > > > > -DKokkos_ENABLE_SERIAL=ON -DKokkos_ENABLE_OPENMP=ON > > > > > > -DCMAKE_CXX_STANDARD="14" -DCMAKE_BUILD_TYPE=Release > > > -DBUILD_TESTING=OFF > > > > > > -DKokkos_ENABLE_LIBDL=OFF -DKokkos_ENABLE_OPENMP=ON > > > > > > -DKokkos_ENABLE_SERIAL=ON > > > -DKokkos_ENABLE_AGGRESSIVE_VECTORIZATION=ON']": > > > > > > -- Setting default Kokkos CXX standard to 14 > > > > > > -- The CXX compiler identification is Fujitsu > > > > > > -- Check for working CXX compiler: > > > > > /opt/FJSVxtclanga/tcsds-1.2.31/bin/mpiFCC > > > > > > -- Check for working CXX compiler: > > > > > > /opt/FJSVxtclanga/tcsds-1.2.31/bin/mpiFCC -- works > > > > > > -- Detecting CXX compiler ABI info > > > > > > -- Detecting CXX compiler ABI info - done > > > > > > -- The project name is: Kokkos > > > > > > -- Configuring incomplete, errors occurred! > > > > > > See also > > > > > > > > > > > > > > > "/vol0004/ra010009/a04199/petsc/arch-arm/externalpackages/git.kokkos/petsc-build/CMakeFiles/CMakeOutput.log".CMake > > > > > > Error at cmake/kokkos_compiler_id.cmake:129 (STRING): > > > > > > > > > > > > * STRING sub-command REPLACE requires at least four > arguments.*Call > > > > > Stack > > > > > > (most recent call first): > > > > > > cmake/kokkos_tribits.cmake:174 (INCLUDE) > > > > > > CMakeLists.txt:166 (KOKKOS_SETUP_BUILD_ENVIRONMENT) > > > > > > > > > > > > On Wed, Apr 14, 2021 at 3:34 PM Satish Balay <balay(a)mcs.anl.gov> > > > wrote: > > > > > > > > > > > > > Additional kokkos cmake arguments can be passed in via > > > > > > > --download-kokkos-cmake-arguments=string option. > > > > > > > > > > > > > > Satish > > > > > > > > > > > > > > On Wed, 14 Apr 2021, Mark Adams wrote: > > > > > > > > > > > > > > > Satish, > > > > > > > > > > > > > > > > For the fujitsu compiler OMP is -Kopenmp. > > > > > > > > > > > > > > > > Sarat (cc'ed) tells me that he built Kokkos with: > > > > > > > > > > > > > > > > cmake -DCMAKE_BUILD_TYPE=Release \ > > > > > > > > -DCMAKE_INSTALL_PREFIX=${KOKKOS_SRC_DIR}/install \ > > > > > > > > -DBUILD_TESTING=OFF \ > > > > > > > > -DKokkos_ENABLE_LIBDL=OFF \ > > > > > > > > -DKokkos_ENABLE_OPENMP=ON \ > > > > > > > > -DKokkos_ENABLE_SERIAL=ON \ > > > > > > > > -DKokkos_ENABLE_AGGRESSIVE_VECTORIZATION=ON \ > > > > > > > > .. > > > > > > > > > > > > > > > > How might I make this happen in PETSc? > > > > > > > > > > > > > > > > Thanks, > > > > > > > > Mark > > > > > > > > > > > > > > > > > > > > > > > > On Wed, Apr 14, 2021 at 2:44 PM Satish Balay < > balay(a)mcs.anl.gov> > > > > > wrote: > > > > > > > > > > > > > > > > > On Wed, 14 Apr 2021, Mark Adams wrote: > > > > > > > > > > > > > > > > > > > I have this building now. > > > > > > > > > > Do you know anything about OpenMP? > > > > > > > > > > I can add --with-openmp > > > > > > > > > > That should get Kokkos to be made with OpenMP. > > > > > > > > > > Should PETSc deal with the compilers correctly? > > > > > > > > > > > > > > > > > > Well it tries the following compiler options for openmp. > > > > > > > > > > > > > > > > > > oflags = ["-fopenmp", # Gnu > > > > > > > > > "-qsmp=omp",# IBM XL C/C++ > > > > > > > > > "-h omp", # Cray. Must come after XL > because XL > > > > > > > interprets > > > > > > > > > this option as meaning "-soname omp" > > > > > > > > > "-mp", # Portland Group > > > > > > > > > "-Qopenmp", # Intel windows > > > > > > > > > "-openmp", # Intel > > > > > > > > > "-xopenmp", # Sun > > > > > > > > > "+Oopenmp", # HP > > > > > > > > > "/openmp" # Microsoft Visual Studio > > > > > > > > > ] > > > > > > > > > > > > > > > > > > I don't know what the flag for fugaku compiler is. > > > > > > > > > > > > > > > > > > Satish > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > Mark > > > > > > > > > > > > > > > > > > > > On Wed, Apr 14, 2021 at 1:45 PM Mark Adams < > mfadams(a)lbl.gov> > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > If you feel inspired you could try Kokkos :|| > > > > > > > > > > > I am in a parking lot waiting for my daughter but can > try > > > this > > > > > > > when I > > > > > > > > > get > > > > > > > > > > > home, > > > > > > > > > > > Thanks again, > > > > > > > > > > > Mark > > > > > > > > > > > > > > > > > > > > > > On Wed, Apr 14, 2021 at 1:33 PM Satish Balay < > > > > > balay(a)mcs.anl.gov> > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > >> I think I allocated a single node - and did the build > on > > > it. > > > > > > > > > > >> > > > > > > > > > > >> Now I'm getting an error - don't know what changed.. > > > > > > > > > > >> > > > > > > > > > > >> login6$ pjsub --interact -L "node=1" -L > > > > > "rscunit=rscunit_ft01" -L > > > > > > > > > > >> "rscgrp=eap-int" -L "elapse=1:00:00" --sparam > > > "wait-time=600" > > > > > > > > > > >> [ERR.] PJM 0059 pjsub rscgrp=eap-int is disabled. > > > > > > > > > > >> > > > > > > > > > > >> Ok - the following worked.. > > > > > > > > > > >> > > > > > > > > > > >> login6$ pjsub --interact -L "node=1" -L > > > > > "rscunit=rscunit_ft01" -L > > > > > > > > > > >> "elapse=1:00:00" --sparam "wait-time=600" > > > > > > > > > > >> [INFO] PJM 0000 pjsub Job 6301572 submitted. > > > > > > > > > > >> [INFO] PJM 0081 .connected. > > > > > > > > > > >> [INFO] PJM 0082 pjsub Interactive job 6301572 started. > > > > > > > > > > >> [a04201@j31-3110s petsc]$ > > > > > > > > > > >> > > > > > > > > > > >> Ok - trying this build now. > > > > > > > > > > >> > > > > > > > > > > >> [a04201@j31-3110s petsc]$ cat > > > > > > > > > > >> > > > ~/petsc.save/arch-arm/lib/petsc/conf/reconfigure-arch-arm.py > > > > > > > > > > >> #!/usr/bin/python3 > > > > > > > > > > >> if __name__ == '__main__': > > > > > > > > > > >> import sys > > > > > > > > > > >> import os > > > > > > > > > > >> sys.path.insert(0, os.path.abspath('config')) > > > > > > > > > > >> import configure > > > > > > > > > > >> configure_options = [ > > > > > > > > > > >> '--with-blaslapack-lib=-lfjlapack', > > > > > > > > > > >> '--with-debugging=0', > > > > > > > > > > >> 'CC=mpifcc', > > > > > > > > > > >> 'CXX=mpiFCC', > > > > > > > > > > >> 'FC=mpifrt', > > > > > > > > > > >> 'PETSC_ARCH=arch-arm', > > > > > > > > > > >> ] > > > > > > > > > > >> configure.petsc_configure(configure_options) > > > > > > > > > > >> [a04201@j31-3110s petsc]$ > > > > > > > > > > >> > > > ~/petsc.save/arch-arm/lib/petsc/conf/reconfigure-arch-arm.py > > > > > > > > > > >> --download-p4est --download-zlib > > > > > > > > > > >> <snip> > > > > > > > > > > >> p4est: > > > > > > > > > > >> Includes: > > > -I/vol0004/ra010009/a04201/petsc/arch-arm/include > > > > > > > > > > >> Library: > > > > > -Wl,-rpath,/vol0004/ra010009/a04201/petsc/arch-arm/lib > > > > > > > > > > >> -L/vol0004/ra010009/a04201/petsc/arch-arm/lib -lp4est > -lsc > > > > > > > > > > >> > > > > > > > > > > >> Ok - this worked for me. > > > > > > > > > > >> > > > > > > > > > > >> Satish > > > > > > > > > > >> > > > > > > > > > > >> On Wed, 14 Apr 2021, Mark Adams wrote: > > > > > > > > > > >> > > > > > > > > > > >> > Do you recall what nodes you use to build on a > "compute" > > > > > node, > > > > > > > to > > > > > > > > > avoid > > > > > > > > > > >> > cross compilation? > > > > > > > > > > >> > > > > > > > > > > > >> > On Wed, Apr 14, 2021 at 12:08 PM Satish Balay < > > > > > > > balay(a)mcs.anl.gov> > > > > > > > > > > >> wrote: > > > > > > > > > > >> > > > > > > > > > > > >> > > looks like p4est cannot be cross-compiled. > > > > > > > > > > >> > > > > > > > > > > > > >> > > Satish > > > > > > > > > > >> > > > > > > > > > > > > >> > > On Wed, 14 Apr 2021, Mark Adams wrote: > > > > > > > > > > >> > > > > > > > > > > > > >> > > > I get this error with p4est on Fugaku. > > > > > > > > > > >> > > > It is a Fortran error. Odd. > > > > > > > > > > >> > > > Mark > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >

1 0

Re: [petsc-dev] Fortran macOS Linkage Warning
by Pierre Jolivet 16 Apr '21

16 Apr '21

Barry, That issue + this commit: $ git bisect bad 36c166f2bbb44c2987d8fb7385fbe9e2d1cfb0c8 is the first bad commit commit 36c166f2bbb44c2987d8fb7385fbe9e2d1cfb0c8 Author: Barry Smith <bsmith(a)mcs.anl.gov> Date: Sat Apr 10 13:40:10 2021 -0500 makes it actually impossible to ./configure on macOS. Do we really expect all macOS users to know about the variable MACOSX_DEPLOYMENT_TARGET? Thanks, Pierre $ ./configure =============================================================================== Configuring PETSc to compile on your system =============================================================================== TESTING: checkFortranLibraries from config.compilers(config/BuildSystem/config/compilers.py:921) ******************************************************************************* UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log for details): ------------------------------------------------------------------------------- Fortran libraries cannot be used with C as linker ******************************************************************************* Executing: mpicc -o /var/folders/_9/0qn1m4z92wzbhz9434vx39fc0000gn/T/petsc-bhav6feq/config.setCompilers/conftest -Wl,-bind_at_load -Wl,-multiply_defined,suppress -Wl,-multiply_defined -Wl,suppress -Wl,-commons,use_dylibs -Wl,-search_paths_first -Wl,-no_compact_unwind -fPIC -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -fstack-protector -fno-stack-check -Qunused-arguments -fvisibility=hidden -g3 /var/folders/_9/0qn1m4z92wzbhz9434vx39fc0000gn/T/petsc-bhav6feq/config.setCompilers/conftest.o -lc++ -ldl -lc++ -ldl -Wl,-rpath,/usr/local/Cellar/mpich/3.4.1_1/lib -L/usr/local/Cellar/mpich/3.4.1_1/lib -lmpifort -lmpi -lpmpi -lgfortran -Wl,-rpath,/usr/local/Cellar/gcc/10.2.0_4/lib/gcc/10/gcc/x86_64-apple-darwin20/10.2.0 -L/usr/local/Cellar/gcc/10.2.0_4/lib/gcc/10/gcc/x86_64-apple-darwin20/10.2.0 -Wl,-rpath,/usr/local/Cellar/gcc/10.2.0_4/lib/gcc/10 -L/usr/local/Cellar/gcc/10.2.0_4/lib/gcc/10 -lgfortran -lquadmath -lm Possible ERROR while running linker: stderr: ld: warning: dylib (/usr/local/Cellar/mpich/3.4.1_1/lib/libmpifort.dylib) was built for newer macOS version (11.2) than being linked (11.0) ld: warning: dylib (/usr/local/Cellar/gcc/10.2.0_4/lib/gcc/10/libgfortran.dylib) was built for newer macOS version (11.2) than being linked (11.0) ld: warning: dylib (/usr/local/Cellar/gcc/10.2.0_4/lib/gcc/10/libquadmath.dylib) was built for newer macOS version (11.2) than being linked (11.0) ld: warning: dylib (/usr/local/Cellar/gcc/10.2.0_4/lib/gcc/10/libquadmath.0.dylib) was built for newer macOS version (11.2) than being linked (11.0) compilers: Cannot compile/link C with mpicc. > On 12 Mar 2021, at 6:16 PM, Barry Smith <bsmith(a)petsc.dev> wrote: > > > You should be able to edit lib/petsc/conf/rules and add it to the filter for .cu.PETSc .kokkos.cxx.PETSc .c.PETSc .cxx.PETSc: and .F.PETSc .F90.PETSc: > > I think it is fine to hide this warning; it is impossible for me to understand anyway :-(. > > Barry > > > > >> On Mar 12, 2021, at 2:44 AM, Pierre Jolivet <pierre(a)joliv.et <mailto:[email protected]>> wrote: >> >> Sorry to dig up this old thread, but this is triggering false positives while make checking for people who don’t want to toy around with MACOSX_DEPLOYMENT_TARGET. >> Could/should we filter this somehow, or have BuildSystem tinker with the environment variable itself? >> >> Thanks, >> Pierre >> >> $ make check >> Running check examples to verify correct installation >> Using PETSC_DIR=/Users/jolivet/repo/petsc and PETSC_ARCH=arch-darwin-c-debug-real >> *******************Error detected during compile or link!******************* >> See http://www.mcs.anl.gov/petsc/documentation/faq.html <http://www.mcs.anl.gov/petsc/documentation/faq.html> >> /Users/jolivet/repo/petsc/src/snes/tutorials ex19 >> ********************************************************************************* >> mpicc -Wl,-bind_at_load -Wl,-multiply_defined,suppress -Wl,-multiply_defined -Wl,suppress -Wl,-commons,use_dylibs -Wl,-search_paths_first -Wl,-no_compact_unwind -Wall -g -Wall -g -I/Users/jolivet/repo/petsc/include -I/Users/jolivet/repo/petsc/arch-darwin-c-debug-real/include -I/opt/intel/mkl/include ex19.c -Wl,-rpath,/Users/jolivet/repo/petsc/arch-darwin-c-debug-real/lib -L/Users/jolivet/repo/petsc/arch-darwin-c-debug-real/lib -Wl,-rpath,/Users/jolivet/repo/petsc/arch-darwin-c-debug-real/lib -L/Users/jolivet/repo/petsc/arch-darwin-c-debug-real/lib -Wl,-rpath,/opt/intel/mkl/lib -L/opt/intel/mkl/lib -Wl,-rpath,/opt/intel/lib -L/opt/intel/lib -Wl,-rpath,/usr/local/Cellar/mpich/3.4.1_1/lib -L/usr/local/Cellar/mpich/3.4.1_1/lib -Wl,-rpath,/usr/local/Cellar/gcc/10.2.0_4/lib/gcc/10/gcc/x86_64-apple-darwin20/10.2.0 -L/usr/local/Cellar/gcc/10.2.0_4/lib/gcc/10/gcc/x86_64-apple-darwin20/10.2.0 -Wl,-rpath,/usr/local/Cellar/gcc/10.2.0_4/lib/gcc/10 -L/usr/local/Cellar/gcc/10.2.0_4/lib/gcc/10 -lpetsc -lHYPRE -lcmumps -ldmumps -lsmumps -lzmumps -lmumps_common -lpord -lmkl_scalapack_lp64 -lmkl_blacs_mpich_lp64 -lEl -lElSuiteSparse -lpmrrr -lp4est -lsc -liomp5 -lmkl_intel_lp64 -lmkl_intel_thread -lmkl_core -lparmmg -lmmg -lmmg3d -lptesmumps -lptscotchparmetis -lptscotch -lptscotcherr -lesmumps -lscotch -lscotcherr -lparmetis -lmetis -ltriangle -ltet -lz -lX11 -lc++ -ldl -lmpifort -lmpi -lpmpi -lgfortran -lquadmath -lm -lc++ -ldl -o ex19 >> ld: warning: dylib (/usr/local/Cellar/mpich/3.4.1_1/lib/libmpifort.dylib) was built for newer macOS version (11.1) than being linked (11.0) >> ld: warning: dylib (/usr/local/Cellar/gcc/10.2.0_4/lib/gcc/10/libgfortran.dylib) was built for newer macOS version (11.2) than being linked (11.0) >> ld: warning: dylib (/usr/local/Cellar/gcc/10.2.0_4/lib/gcc/10/libquadmath.dylib) was built for newer macOS version (11.2) than being linked (11.0) >> ld: warning: dylib (/usr/local/Cellar/gcc/10.2.0_4/lib/gcc/10/libquadmath.0.dylib) was built for newer macOS version (11.2) than being linked (11.0) >> ld: warning: object file (/Users/jolivet/repo/petsc/arch-darwin-c-debug-real/lib/libcmumps.a(cana_aux.o)) was built for newer macOS version (11.2) than being linked (11.0) >> ld: warning: object file (/Users/jolivet/repo/petsc/arch-darwin-c-debug-real/lib/libdmumps.a(dana_aux.o)) was built for newer macOS version (11.2) than being linked (11.0) >> ld: warning: object file (/Users/jolivet/repo/petsc/arch-darwin-c-debug-real/lib/libsmumps.a(sana_aux.o)) was built for newer macOS version (11.2) than being linked (11.0) >> ld: warning: object file (/Users/jolivet/repo/petsc/arch-darwin-c-debug-real/lib/libzmumps.a(zana_aux.o)) was built for newer macOS version (11.2) than being linked (11.0) >> ld: warning: object file (/Users/jolivet/repo/petsc/arch-darwin-c-debug-real/lib/libmumps_common.a(front_data_mgt_m.o)) was built for newer macOS version (11.2) than being linked (11.0) >> C/C++ example src/snes/tutorials/ex19 run successfully with 1 MPI process >> >> >>> On 15 Feb 2021, at 6:08 PM, Jacob Faibussowitsch <jacob.fai(a)gmail.com <mailto:[email protected]>> wrote: >>> >>> Yes, this also seems to work on my end. >>> >>> Best regards, >>> >>> Jacob Faibussowitsch >>> (Jacob Fai - booss - oh - vitch) >>> Cell: (312) 694-3391 >>> >>>> On Feb 15, 2021, at 12:05, Satish Balay <balay(a)mcs.anl.gov <mailto:[email protected]>> wrote: >>>> >>>> Does setting MACOSX_DEPLOYMENT_TARGET also work? >>>> >>>> Satish >>>> >>>> On Mon, 15 Feb 2021, Jacob Faibussowitsch wrote: >>>> >>>>> I think this problem ultimately stems from the fact that gcc/g++ and gfortran are not from the same people on macOS. It seems like clang is configured to build for 11.0 target, whilst gcc defaults to newest available target. And indeed if I change my compile command there is no warning: >>>>> >>>>> mkdir -p build && gfortran -c -o ./build/conftest.o conftest.F90 && gcc-10 -o ./build/libconftest.dylib ./build/conftest.o -dynamiclib -undefined dynamic_lookup && rm -rf ./build >>>>> >>>>> Note I now call gcc-10 as the linker which is the __actual__ gcc (not clang in a trench coat). >>>>> >>>>> Best regards, >>>>> >>>>> Jacob Faibussowitsch >>>>> (Jacob Fai - booss - oh - vitch) >>>>> Cell: (312) 694-3391 >>>>> >>>>>> On Feb 15, 2021, at 11:46, Jacob Faibussowitsch <jacob.fai(a)gmail.com <mailto:[email protected]>> wrote: >>>>>> >>>>>> Ok so if I pass -mmacosx-version-min=11.2 to the linker the warning disappears... >>>>>> >>>>>> Best regards, >>>>>> >>>>>> Jacob Faibussowitsch >>>>>> (Jacob Fai - booss - oh - vitch) >>>>>> Cell: (312) 694-3391 >>>>>> >>>>>>> On Feb 15, 2021, at 11:41, Jacob Faibussowitsch <jacob.fai(a)gmail.com <mailto:[email protected]> <mailto:[email protected] <mailto:[email protected]>>> wrote: >>>>>>> >>>>>>> I can reproduce the warning compiling just that simple code you listed. This is the compile command: >>>>>>> >>>>>>> mkdir -p build && gfortran -c -o ./build/conftest.o conftest.F90 && gcc -o ./build/libconftest.dylib ./build/conftest.o -dynamiclib -undefined dynamic_lookup && rm -rf ./build >>>>>>> >>>>>>> And error: >>>>>>> ld: warning: object file (./build/conftest.o) was built for newer macOS version (11.2) than being linked (11.0) >>>>>>> >>>>>>> Note I recently uninstalled and reinstalled command line tools too, which does not seem to have fixed it. >>>>>>> >>>>>>> Best regards, >>>>>>> >>>>>>> Jacob Faibussowitsch >>>>>>> (Jacob Fai - booss - oh - vitch) >>>>>>> Cell: (312) 694-3391 >>>>>>> >>>>>>>> On Feb 15, 2021, at 11:30, Satish Balay <balay(a)mcs.anl.gov <mailto:[email protected]> <mailto:[email protected] <mailto:[email protected]>>> wrote: >>>>>>>> >>>>>>>> Here is the compile command and the warning: >>>>>>>> >>>>>>>>>>>>>> >>>>>>>> ===== Checking linker >>>>>>>> Executing: gfortran -c -o /var/folders/9w/7dlszmmn6q1gd5yf3r9n79bc0000gn/T/petsc-dk9aiv7x/config.setCompilers/conftest.o -I/var/folders/9w/7dlszmmn6q1gd5yf3r9n79bc0000gn/T/petsc-dk9aiv7x/config.setCompilers /var/folders/9w/7dlszmmn6q1gd5yf3r9n79bc0000gn/T/petsc-dk9aiv7x/config.setCompilers/conftest.F90 >>>>>>>> Successful compile: >>>>>>>> Source: >>>>>>>> function foo(a) >>>>>>>> real:: a,x,bar >>>>>>>> common /xx/ x >>>>>>>> x=a >>>>>>>> foo = bar(x) >>>>>>>> end >>>>>>>> Executing: gcc -o /var/folders/9w/7dlszmmn6q1gd5yf3r9n79bc0000gn/T/petsc-dk9aiv7x/config.setCompilers/libconftest.dylib -dynamiclib -single_module -undefined dynamic_lookup -multiply_defined suppress /var/folders/9w/7dlszmmn6q1gd5yf3r9n79bc0000gn/T/petsc-dk9aiv7x/config.setCompilers/conftest.o >>>>>>>> Possible ERROR while running linker: >>>>>>>> stderr: >>>>>>>> ld: warning: object file (/var/folders/9w/7dlszmmn6q1gd5yf3r9n79bc0000gn/T/petsc-dk9aiv7x/config.setCompilers/conftest.o) was built for newer macOS version (11.2) than being linked (11.0) >>>>>>>> Accepted FC PIC code without compiler flag >>>>>>>> ============================================================================================= >>>>>>>> <<<<<< >>>>>>>> >>>>>>>> Its likely due to the way brew built gfortran. You can try compiling a simple test code and see if you can reproduce this warning. >>>>>>>> >>>>>>>> There is a related issue at: >>>>>>>> >>>>>>>> https://github.com/spack/spack/issues/21644 <https://github.com/spack/spack/issues/21644> <https://github.com/spack/spack/issues/21644 <https://github.com/spack/spack/issues/21644>> >>>>>>>> >>>>>>>> Satish >>>>>>>> >>>>>>>> >>>>>>>> On Sat, 13 Feb 2021, Jacob Faibussowitsch wrote: >>>>>>>> >>>>>>>>> Hello All, >>>>>>>>> >>>>>>>>> I’ve been getting these warnings for weeks, at first I figured it was due to gfortran brew package not being updated for Big Sur, but since it has persisted through a few gfortran brew updates I’m thinking it may be petsc related. This is on clean build on master. >>>>>>>>> >>>>>>>>> ld: warning: object file (master-arch-darwin-c-debug/obj/sys/classes/bag/f2003-src/fsrc/bagenum.o) was built for newer macOS version (11.2) than being linked (11.0) >>>>>>>>> ld: warning: object file (master-arch-darwin-c-debug/obj/sys/f90-src/fsrc/f90_fwrap.o) was built for newer macOS version (11.2) than being linked (11.0) >>>>>>>>> ld: warning: object file (master-arch-darwin-c-debug/obj/sys/objects/f2003-src/fsrc/optionenum.o) was built for newer macOS version (11.2) than being linked (11.0) >>>>>>>>> ld: warning: object file (master-arch-darwin-c-debug/obj/sys/fsrc/somefort.o) was built for newer macOS version (11.2) than being linked (11.0) >>>>>>>>> ld: warning: object file (master-arch-darwin-c-debug/obj/sys/f90-mod/petscsysmod.o) was built for newer macOS version (11.2) than being linked (11.0) >>>>>>>>> >>>>>>>>> Best regards, >>>>>>>>> >>>>>>>>> Jacob Faibussowitsch >>>>>>>>> (Jacob Fai - booss - oh - vitch) >>>>>>>>> Cell: (312) 694-3391 >>> >> >

1 0

Re: [petsc-dev] Error on Fugaku
by Satish Balay 15 Apr '21

15 Apr '21

BTW: >>>> login6$ mpifccpx -fopenmp -v sizeof.c fccpx: Fujitsu C/C++ Compiler 4.5.0 tcsds-1.2.31 simulating gcc version 6.1 /opt/FJSVxtclanga/tcsds-1.2.31/bin/../lib/ccpcompx --preinclude /opt/FJSVxtclanga/tcsds-1.2.31/bin/../lib/fcc.pre --gcc -Dunix -Dlinux -D__FUJITSU -D__FCC_major__=4 -D__FCC_minor__=5 -D__FCC_patchlevel__=0 -D__FCC_version__="4.5.0" -D__aarch64__ -D__unix -D_OPENMP=201107 -D__fcc_version__=0x800 -D__fcc_version=800 -D__USER_LABEL_PREFIX__= -D__ARM_ARCH=8 -D__ARM_FEATURE_SVE -D__FP_FAST_FMA -D__ELF__ -D__unix__ -D__linux__ -D__linux -Asystem(unix) -D__LIBC_6B -D__LP64__ -D_LP64 --K=omp -I/opt/FJSVxtclanga/.common/MELI024/include/mpi/fujitsu -D_REENTRANT --K=noocl -D_REENTRANT -D__MT__ --zmode=64 --sys_include=/opt/FJSVxtclanga/tcsds-1.2.31/bin/../include --sys_include=/opt/FJSVxos/devkit/aarch64/rfs/usr/include sizeof.c -- -ncmdname=fccpx -zcfc=target_sve -Kopenmp -V -Kthreadsafe -Klargepage -otmp /home/ra010009/a04201/asmHUx9OT.s ccpcompx: Fujitsu C/C++ Compiler 4.5.0 (Mar 4 2021 13:29:06) /opt/FJSVxtclanga/tcsds-1.2.31/bin/../bin/checklogpx -v -zlangC MPI:V4.5.0 -fopenmp -v /home/ra010009/a04201/asmHUx9OT.s /opt/FJSVxos/devkit/aarch64/bin/aarch64-linux-gnu-as -o /home/ra010009/a04201/fccmPXe0C.o -march=armv8.3-a+sve -v /home/ra010009/a04201/asmHUx9OT.s GNU assembler version 2.30 (aarch64-linux-gnu) using BFD version version 2.30-49.el8 /opt/FJSVxos/devkit/aarch64/bin/aarch64-linux-gnu-ld --build-id --sysroot=/opt/FJSVxos/devkit/aarch64/rfs --eh-frame-hdr -m aarch64linux -dynamic-linker /lib/ld-linux-aarch64.so.1 /opt/FJSVxos/devkit/aarch64/rfs/usr/lib64/crt1.o /opt/FJSVxos/devkit/aarch64/rfs/usr/lib64/crti.o /opt/FJSVxos/devkit/aarch64/lib/gcc/aarch64-linux-gnu/8/crtbegin.o -lfjprofmpi -L/opt/FJSVxos/devkit/aarch64/lib/gcc/aarch64-linux-gnu/8 -v /home/ra010009/a04201/fccmPXe0C.o -L/opt/FJSVxtclanga/.common/MELI024/lib64 -lmpi -lfjstring_internal -L/opt/FJSVxos/devkit/aarch64/lib/gcc/aarch64-linux-gnu/8/../../../../aarch64-linux-gnu/lib/../lib64 /opt/FJSVxtclanga/tcsds-1.2.31/bin/../lib64/fjcrt0.o /opt/FJSVxtclanga/tcsds-1.2.31/bin/../lib64/fjlang08.o /opt/FJSVxtclanga/tcsds-1.2.31/bin/../lib64/fjomp.o /opt/FJSVxtclanga/tcsds-1.2.31/bin/../lib64/fjhpctag.o -L/opt/FJSVxtclanga/tcsds-1.2.31/bin/../lib64 -lfjomphk -lfjomp -lfj90i -lfj90fmt_sve -lfj90f -lfjsrcinfo -lfjcrt -lfjompcrt -lfjprofcore -lfjprofomp -lmpg -lm -latomic -lpthread -L/opt/FJSVxos/devkit/aarch64/rfs/usr/lib64 -L/opt/FJSVxos/devkit/aarch64/rfs/opt/FJSVxos/mmm/lib64 -L/opt/FJSVxtclanga/tcsds-1.2.31/bin/../lib64/nofjobj -T/opt/FJSVxos/devkit/aarch64/rfs/opt/FJSVxos/mmm/util/bss-2mb.lds -lm -lrt -lpthread -lelf -lz -ldl -lgcc_s -lgcc -lc -lgcc_s -lgcc -lc /opt/FJSVxos/devkit/aarch64/lib/gcc/aarch64-linux-gnu/8/crtend.o /opt/FJSVxos/devkit/aarch64/rfs/usr/lib64/crtn.o GNU ld version 2.30-49.el8 <<<< i.e -fopenmp is internally mapped to -Kopenmp Satish On Wed, 14 Apr 2021, Mark Adams wrote: > The compiler: > > FCC (FCC) 4.5.0 20210304 > > *simulating gcc version 6.1*Copyright FUJITSU LIMITED 2019-2021 > > and > -Kopenmp specifies to enable the Specification of OpenMP Application > Program Interface. The -Knoopenmp option invalidates the -Kopenmp > option. -Knoopenmp is set by default. > > When the -Kopenmp option is specified, -mt is assumed. > > > *The -Kopenmp option is needed if an object program compiled with it > exists in the command line as an input file.* > > I am going to try w/o Kopenmp. > > On Wed, Apr 14, 2021 at 8:15 PM Satish Balay <balay(a)mcs.anl.gov> wrote: > > > I don't understand kokkos build errors. > > > > For one - kokkos-kernels are taking long to build. And on the back end - > > I'm getting into time limit [and terminated] > > > > Kokkos take a very long time in configure on Summit. > > Thanks, > > > > > > Trying a build on the front-end - its running - for a long time.. > > > > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ > > COMMAND > > 4016901 > > a04201 20 0 11.0g 10.7g 13876 R 99.3 11.6 66:22.35 ccpcompx > > > > > > The above compile process has been running for more than an hour. Ok - its > > finally done. > > > > When I enable openmp - kokkos cmake completes - but I get errors at build > > time [BTW: I don't understand difference between -fopenmp and -Kopenmp] > > > > login6$ > > ../petsc.save/arch-linux2-c-opt/lib/petsc/conf/reconfigure-arch-linux2-c-opt.py > > --with-p4est-dir=$HOME/p4est-install --with-zlib-dir=$HOME/p4est-install > > -download-kokkos --download-kokkos-kernelsx > > --download-kokkos-commit=origin/develop > > --download-kokkos-kernels-commit=origin/develop --download-cmake= > > https://github.com/Kitware/CMake/releases/download/v3.20.1/cmake-3.20.1.tar… > > '--download-kokkos-cmake-arguments=-DBUILD_TESTING=OFF > > -DKokkos_ENABLE_LIBDL=OFF -DKokkos_ENABLE_AGGRESSIVE_VECTORIZATION=ON > > -DKokkos_ENABLE_OPENMP=ON' --ignoreLinkOutput=1 --with-openmp=1 > > > > > > /a04201/petsc.x/arch-linux2-c-opt/externalpackages/git.kokkos/petsc-build/core/src > > -I/vol0004/ra010009/a04201/petsc.x/arch-linux2-c-opt/externalpackages/git.kokkos/core/src > > -I/vol0004/ra010009/a04201/petsc.x/arch-linux2-c-opt/externalpackages/git.kokkos/petsc-build > > -O -fopenmp -fPIC -fopenmp -O -fopenmp -fPIC -fopenmp -fopenmp -std=c++14 > > -o CMakeFiles/kokkoscore.dir/OpenMP/Kokkos_OpenMP_Task.cpp.o -c > > /vol0004/ra010009/a04201/petsc.x/arch-linux2-c-opt/externalpackages/git.kokkos/core/src/OpenMP/Kokkos_OpenMP_Task.cpp > > gmake[2]: Leaving directory > > '/vol0004/ra010009/a04201/petsc.x/arch-linux2-c-opt/externalpackages/git.kokkos/petsc-build' > > gmake[1]: Leaving directory > > '/vol0004/ra010009/a04201/petsc.x/arch-linux2-c-opt/externalpackages/git.kokkos/petsc-build'/home/ra010009/a04201/asmqkagl0.s: > > Assembler messages: > > /home/ra010009/a04201/asmqkagl0.s:20333: Error: symbol `.LEHB41' is > > already defined > > /home/ra010009/a04201/asmqkagl0.s:21334: Error: symbol `.LEHB41' is > > already defined > > gmake[2]: *** [core/src/CMakeFiles/kokkoscore.dir/build.make:312: > > core/src/CMakeFiles/kokkoscore.dir/OpenMP/Kokkos_OpenMP_Exec.cpp.o] Error 1 > > > > Note: I have a successful configure run with p4est+kokkos+kokkos-kernels, > > without-openmp > > [for this - I had to build p4est on the back-end, then switch over to > > front-end - and build the reset of the packages] > > > > Now I'm stuck at compiling petsc sources - > > src/vec/is/sf/impls/basic/kokkos/sfkok.kokkos.cxx > > > > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ > > COMMAND > > 56491 > > a04201 20 0 4944176 4.5g 11216 R 98.4 4.9 12:45.52 ccpcompx > > > > > > > > Satish > > > > On Wed, 14 Apr 2021, Mark Adams wrote: > > > > > PETSc seems to be skipping the compiler that it should use: > > > > > > -- Check for working CXX compiler: > > > /opt/FJSVxtclanga/tcsds-1.2.31/bin/mpiFCC - skipped > > > > > > I see -fopenmp was added. This might be wrong. I use -Kopenmp. > > > > > > --help says: > > > > > > -fopenmp > > > > > > The -fopenmp option specifies to enable Specification of OpenMP > > > Appli- > > > cation Program Interface. > > > > > > When the -fopenmp option is specified, -mt is set. > > > > > > The -fopenmp option is needed if an object program compiled with > > the > > > -fopenmp option exists in the command line as input files. > > > > > > Any ideas? > > > > > > > > > On Wed, Apr 14, 2021 at 5:17 PM Satish Balay <balay(a)mcs.anl.gov> wrote: > > > > > > > The following is working for me on the compute node [its completed > > kokkos > > > > (core) build - and is now building kokkos-kernels > > > > > > > > ~/petsc.save/arch-arm/lib/petsc/conf/reconfigure-arch-arm.py > > > > --download-p4est --download-zlib --download-kokkos > > > > --download-kokkos-kernels --download-kokkos-commit=origin/develop > > > > --download-cmake= > > > > > > https://github.com/Kitware/CMake/releases/download/v3.20.1/cmake-3.20.1.tar… > > > > > > > > -DKokkos_ENABLE_OPENMP=ON give me some issue - so I didn't try any of > > the > > > > additional options.. > > > > > > > > Satish > > > > > > > > On Wed, 14 Apr 2021, Mark Adams wrote: > > > > > > > > > Satish, I get this error. > > > > > > > > > > I wonder if this syntax is wrong: > > > > > > > > > > '--download-kokkos-cmake-arguments=-DCMAKE_BUILD_TYPE=Release > > > > > -DBUILD_TESTING=OFF -DKokkos_ENABLE_LIBDL=OFF > > -DKokkos_ENABLE_OPENMP=ON > > > > > -DKokkos_ENABLE_SERIAL=ON > > -DKokkos_ENABLE_AGGRESSIVE_VECTORIZATION=ON', > > > > > > > > > > > > > > > > > > > > > ============================================================================================= > > > > > Configuring KOKKOS with cmake; this may > > > > take > > > > > several minutes > > > > > > > > > > > > > > > > ============================================================================================= > > > > > > > > > > Executing: /usr/bin/cmake .. > > > > > -DCMAKE_INSTALL_PREFIX=/home/ra010009/a04199/petsc/arch-arm > > > > > > > > > > > -DCMAKE_INSTALL_NAME_DIR:STRING="/home/ra010009/a04199/petsc/arch-arm/lib" > > > > > -DCMAKE_INSTALL_LIBDIR:STRING="lib" -DCMAKE_VERBOSE_MAKEFILE=1 > > > > > -DCMAKE_BUILD_TYPE=Release -DCMAKE_C_COMPILER="mpifcc" > > > > > -DMPI_C_COMPILER="mpifcc" -DCMAKE_AR=/usr/bin/ar > > > > > -DCMAKE_RANLIB=/usr/bin/ranlib -DCMAKE_C_FLAGS:STRING="-fPIC > > > > -Kfast,openmp > > > > > -fopenmp" -DCMAKE_C_FLAGS_DEBUG:STRING="-fPIC -Kfast,openmp -fopenmp" > > > > > -DCMAKE_C_FLAGS_RELEASE:STRING="-fPIC -Kfast,openmp -fopenmp" > > > > > -DCMAKE_CXX_COMPILER="mpiFCC" -DMPI_CXX_COMPILER="mpiFCC" > > > > > -DCMAKE_CXX_FLAGS:STRING="-Kfast,openmp -fopenmp -fPIC -fopenmp" > > > > > -DCMAKE_CXX_FLAGS_DEBUG:STRING="-Kfast,openmp -fopenmp -fPIC > > -fopenmp" > > > > > -DCMAKE_CXX_FLAGS_RELEASE:STRING="-Kfast,openmp -fopenmp -fPIC > > -fopenmp" > > > > > -DCMAKE_Fortran_COMPILER="mpifrt" -DMPI_Fortran_COMPILER="mpifrt" > > > > > -DCMAKE_Fortran_FLAGS:STRING="-fPIC -O -fopenmp" > > > > > -DCMAKE_Fortran_FLAGS_DEBUG:STRING="-fPIC -O -fopenmp" > > > > > -DCMAKE_Fortran_FLAGS_RELEASE:STRING="-fPIC -O -fopenmp" > > > > > -DCMAKE_EXE_LINKER_FLAGS:STRING=" -fopenmp" > > -DBUILD_SHARED_LIBS:BOOL=ON > > > > > -DUSE_XSDK_DEFAULTS=YES -DXSDK_ENABLE_DEBUG=NO > > > > > -DCMAKE_INSTALL_RPATH_USE_LINK_PATH:BOOL=ON > > > > > -DCMAKE_BUILD_WITH_INSTALL_RPATH:BOOL=ON -DKokkos_ENABLE_MPI=ON > > > > > -DKokkos_ENABLE_SERIAL=ON -DKokkos_ENABLE_OPENMP=ON > > > > > -DCMAKE_CXX_STANDARD="14" -DCMAKE_BUILD_TYPE=Release > > -DBUILD_TESTING=OFF > > > > > -DKokkos_ENABLE_LIBDL=OFF -DKokkos_ENABLE_OPENMP=ON > > > > > -DKokkos_ENABLE_SERIAL=ON -DKokkos_ENABLE_AGGRESSIVE_VECTORIZATION=ON > > > > > stdout: > > > > > -- Setting default Kokkos CXX standard to 14 > > > > > -- The CXX compiler identification is Fujitsu > > > > > -- Check for working CXX compiler: > > > > /opt/FJSVxtclanga/tcsds-1.2.31/bin/mpiFCC > > > > > -- Check for working CXX compiler: > > > > > /opt/FJSVxtclanga/tcsds-1.2.31/bin/mpiFCC -- works > > > > > -- Detecting CXX compiler ABI info > > > > > -- Detecting CXX compiler ABI info - done > > > > > -- The project name is: Kokkos > > > > > -- Configuring incomplete, errors occurred! > > > > > See also > > > > > > > > > > > "/vol0004/ra010009/a04199/petsc/arch-arm/externalpackages/git.kokkos/petsc-build/CMakeFiles/CMakeOutput.log". > > > > > Error configuring KOKKOS with cmake Could not > > execute > > > > > "['/usr/bin/cmake .. > > > > > -DCMAKE_INSTALL_PREFIX=/home/ra010009/a04199/petsc/arch-arm > > > > > > > > > > > -DCMAKE_INSTALL_NAME_DIR:STRING="/home/ra010009/a04199/petsc/arch-arm/lib" > > > > > -DCMAKE_INSTALL_LIBDIR:STRING="lib" -DCMAKE_VERBOSE_MAKEFILE=1 > > > > > -DCMAKE_BUILD_TYPE=Release -DCMAKE_C_COMPILER="mpifcc" > > > > > -DMPI_C_COMPILER="mpifcc" -DCMAKE_AR=/usr/bin/ar > > > > > -DCMAKE_RANLIB=/usr/bin/ranlib -DCMAKE_C_FLAGS:STRING="-fPIC > > > > -Kfast,openmp > > > > > -fopenmp" -DCMAKE_C_FLAGS_DEBUG:STRING="-fPIC -Kfast,openmp -fopenmp" > > > > > -DCMAKE_C_FLAGS_RELEASE:STRING="-fPIC -Kfast,openmp -fopenmp" > > > > > -DCMAKE_CXX_COMPILER="mpiFCC" -DMPI_CXX_COMPILER="mpiFCC" > > > > > -DCMAKE_CXX_FLAGS:STRING="-Kfast,openmp -fopenmp -fPIC -fopenmp" > > > > > -DCMAKE_CXX_FLAGS_DEBUG:STRING="-Kfast,openmp -fopenmp -fPIC > > -fopenmp" > > > > > -DCMAKE_CXX_FLAGS_RELEASE:STRING="-Kfast,openmp -fopenmp -fPIC > > -fopenmp" > > > > > -DCMAKE_Fortran_COMPILER="mpifrt" -DMPI_Fortran_COMPILER="mpifrt" > > > > > -DCMAKE_Fortran_FLAGS:STRING="-fPIC -O -fopenmp" > > > > > -DCMAKE_Fortran_FLAGS_DEBUG:STRING="-fPIC -O -fopenmp" > > > > > -DCMAKE_Fortran_FLAGS_RELEASE:STRING="-fPIC -O -fopenmp" > > > > > -DCMAKE_EXE_LINKER_FLAGS:STRING=" -fopenmp" > > -DBUILD_SHARED_LIBS:BOOL=ON > > > > > -DUSE_XSDK_DEFAULTS=YES -DXSDK_ENABLE_DEBUG=NO > > > > > -DCMAKE_INSTALL_RPATH_USE_LINK_PATH:BOOL=ON > > > > > -DCMAKE_BUILD_WITH_INSTALL_RPATH:BOOL=ON -DKokkos_ENABLE_MPI=ON > > > > > -DKokkos_ENABLE_SERIAL=ON -DKokkos_ENABLE_OPENMP=ON > > > > > -DCMAKE_CXX_STANDARD="14" -DCMAKE_BUILD_TYPE=Release > > -DBUILD_TESTING=OFF > > > > > -DKokkos_ENABLE_LIBDL=OFF -DKokkos_ENABLE_OPENMP=ON > > > > > -DKokkos_ENABLE_SERIAL=ON > > -DKokkos_ENABLE_AGGRESSIVE_VECTORIZATION=ON']": > > > > > -- Setting default Kokkos CXX standard to 14 > > > > > -- The CXX compiler identification is Fujitsu > > > > > -- Check for working CXX compiler: > > > > /opt/FJSVxtclanga/tcsds-1.2.31/bin/mpiFCC > > > > > -- Check for working CXX compiler: > > > > > /opt/FJSVxtclanga/tcsds-1.2.31/bin/mpiFCC -- works > > > > > -- Detecting CXX compiler ABI info > > > > > -- Detecting CXX compiler ABI info - done > > > > > -- The project name is: Kokkos > > > > > -- Configuring incomplete, errors occurred! > > > > > See also > > > > > > > > > > > "/vol0004/ra010009/a04199/petsc/arch-arm/externalpackages/git.kokkos/petsc-build/CMakeFiles/CMakeOutput.log".CMake > > > > > Error at cmake/kokkos_compiler_id.cmake:129 (STRING): > > > > > > > > > > * STRING sub-command REPLACE requires at least four arguments.*Call > > > > Stack > > > > > (most recent call first): > > > > > cmake/kokkos_tribits.cmake:174 (INCLUDE) > > > > > CMakeLists.txt:166 (KOKKOS_SETUP_BUILD_ENVIRONMENT) > > > > > > > > > > On Wed, Apr 14, 2021 at 3:34 PM Satish Balay <balay(a)mcs.anl.gov> > > wrote: > > > > > > > > > > > Additional kokkos cmake arguments can be passed in via > > > > > > --download-kokkos-cmake-arguments=string option. > > > > > > > > > > > > Satish > > > > > > > > > > > > On Wed, 14 Apr 2021, Mark Adams wrote: > > > > > > > > > > > > > Satish, > > > > > > > > > > > > > > For the fujitsu compiler OMP is -Kopenmp. > > > > > > > > > > > > > > Sarat (cc'ed) tells me that he built Kokkos with: > > > > > > > > > > > > > > cmake -DCMAKE_BUILD_TYPE=Release \ > > > > > > > -DCMAKE_INSTALL_PREFIX=${KOKKOS_SRC_DIR}/install \ > > > > > > > -DBUILD_TESTING=OFF \ > > > > > > > -DKokkos_ENABLE_LIBDL=OFF \ > > > > > > > -DKokkos_ENABLE_OPENMP=ON \ > > > > > > > -DKokkos_ENABLE_SERIAL=ON \ > > > > > > > -DKokkos_ENABLE_AGGRESSIVE_VECTORIZATION=ON \ > > > > > > > .. > > > > > > > > > > > > > > How might I make this happen in PETSc? > > > > > > > > > > > > > > Thanks, > > > > > > > Mark > > > > > > > > > > > > > > > > > > > > > On Wed, Apr 14, 2021 at 2:44 PM Satish Balay <balay(a)mcs.anl.gov> > > > > wrote: > > > > > > > > > > > > > > > On Wed, 14 Apr 2021, Mark Adams wrote: > > > > > > > > > > > > > > > > > I have this building now. > > > > > > > > > Do you know anything about OpenMP? > > > > > > > > > I can add --with-openmp > > > > > > > > > That should get Kokkos to be made with OpenMP. > > > > > > > > > Should PETSc deal with the compilers correctly? > > > > > > > > > > > > > > > > Well it tries the following compiler options for openmp. > > > > > > > > > > > > > > > > oflags = ["-fopenmp", # Gnu > > > > > > > > "-qsmp=omp",# IBM XL C/C++ > > > > > > > > "-h omp", # Cray. Must come after XL because XL > > > > > > interprets > > > > > > > > this option as meaning "-soname omp" > > > > > > > > "-mp", # Portland Group > > > > > > > > "-Qopenmp", # Intel windows > > > > > > > > "-openmp", # Intel > > > > > > > > "-xopenmp", # Sun > > > > > > > > "+Oopenmp", # HP > > > > > > > > "/openmp" # Microsoft Visual Studio > > > > > > > > ] > > > > > > > > > > > > > > > > I don't know what the flag for fugaku compiler is. > > > > > > > > > > > > > > > > Satish > > > > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > Mark > > > > > > > > > > > > > > > > > > On Wed, Apr 14, 2021 at 1:45 PM Mark Adams <mfadams(a)lbl.gov> > > > > wrote: > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > If you feel inspired you could try Kokkos :|| > > > > > > > > > > I am in a parking lot waiting for my daughter but can try > > this > > > > > > when I > > > > > > > > get > > > > > > > > > > home, > > > > > > > > > > Thanks again, > > > > > > > > > > Mark > > > > > > > > > > > > > > > > > > > > On Wed, Apr 14, 2021 at 1:33 PM Satish Balay < > > > > balay(a)mcs.anl.gov> > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > >> I think I allocated a single node - and did the build on > > it. > > > > > > > > > >> > > > > > > > > > >> Now I'm getting an error - don't know what changed.. > > > > > > > > > >> > > > > > > > > > >> login6$ pjsub --interact -L "node=1" -L > > > > "rscunit=rscunit_ft01" -L > > > > > > > > > >> "rscgrp=eap-int" -L "elapse=1:00:00" --sparam > > "wait-time=600" > > > > > > > > > >> [ERR.] PJM 0059 pjsub rscgrp=eap-int is disabled. > > > > > > > > > >> > > > > > > > > > >> Ok - the following worked.. > > > > > > > > > >> > > > > > > > > > >> login6$ pjsub --interact -L "node=1" -L > > > > "rscunit=rscunit_ft01" -L > > > > > > > > > >> "elapse=1:00:00" --sparam "wait-time=600" > > > > > > > > > >> [INFO] PJM 0000 pjsub Job 6301572 submitted. > > > > > > > > > >> [INFO] PJM 0081 .connected. > > > > > > > > > >> [INFO] PJM 0082 pjsub Interactive job 6301572 started. > > > > > > > > > >> [a04201@j31-3110s petsc]$ > > > > > > > > > >> > > > > > > > > > >> Ok - trying this build now. > > > > > > > > > >> > > > > > > > > > >> [a04201@j31-3110s petsc]$ cat > > > > > > > > > >> > > ~/petsc.save/arch-arm/lib/petsc/conf/reconfigure-arch-arm.py > > > > > > > > > >> #!/usr/bin/python3 > > > > > > > > > >> if __name__ == '__main__': > > > > > > > > > >> import sys > > > > > > > > > >> import os > > > > > > > > > >> sys.path.insert(0, os.path.abspath('config')) > > > > > > > > > >> import configure > > > > > > > > > >> configure_options = [ > > > > > > > > > >> '--with-blaslapack-lib=-lfjlapack', > > > > > > > > > >> '--with-debugging=0', > > > > > > > > > >> 'CC=mpifcc', > > > > > > > > > >> 'CXX=mpiFCC', > > > > > > > > > >> 'FC=mpifrt', > > > > > > > > > >> 'PETSC_ARCH=arch-arm', > > > > > > > > > >> ] > > > > > > > > > >> configure.petsc_configure(configure_options) > > > > > > > > > >> [a04201@j31-3110s petsc]$ > > > > > > > > > >> > > ~/petsc.save/arch-arm/lib/petsc/conf/reconfigure-arch-arm.py > > > > > > > > > >> --download-p4est --download-zlib > > > > > > > > > >> <snip> > > > > > > > > > >> p4est: > > > > > > > > > >> Includes: > > -I/vol0004/ra010009/a04201/petsc/arch-arm/include > > > > > > > > > >> Library: > > > > -Wl,-rpath,/vol0004/ra010009/a04201/petsc/arch-arm/lib > > > > > > > > > >> -L/vol0004/ra010009/a04201/petsc/arch-arm/lib -lp4est -lsc > > > > > > > > > >> > > > > > > > > > >> Ok - this worked for me. > > > > > > > > > >> > > > > > > > > > >> Satish > > > > > > > > > >> > > > > > > > > > >> On Wed, 14 Apr 2021, Mark Adams wrote: > > > > > > > > > >> > > > > > > > > > >> > Do you recall what nodes you use to build on a "compute" > > > > node, > > > > > > to > > > > > > > > avoid > > > > > > > > > >> > cross compilation? > > > > > > > > > >> > > > > > > > > > > >> > On Wed, Apr 14, 2021 at 12:08 PM Satish Balay < > > > > > > balay(a)mcs.anl.gov> > > > > > > > > > >> wrote: > > > > > > > > > >> > > > > > > > > > > >> > > looks like p4est cannot be cross-compiled. > > > > > > > > > >> > > > > > > > > > > > >> > > Satish > > > > > > > > > >> > > > > > > > > > > > >> > > On Wed, 14 Apr 2021, Mark Adams wrote: > > > > > > > > > >> > > > > > > > > > > > >> > > > I get this error with p4est on Fugaku. > > > > > > > > > >> > > > It is a Fortran error. Odd. > > > > > > > > > >> > > > Mark > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >

1 0

Re: [petsc-dev] Error on Fugaku
by Satish Balay 14 Apr '21

14 Apr '21

On Wed, 14 Apr 2021, Satish Balay via petsc-dev wrote: > Now I'm stuck at compiling petsc sources - src/vec/is/sf/impls/basic/kokkos/sfkok.kokkos.cxx > > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 56491 a04201 20 0 4944176 4.5g 11216 R 98.4 4.9 12:45.52 ccpcompx This compile is still ongoing.. PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 56491 a04201 20 0 26.5g 26.2g 12512 R 99.7 28.4 122:41.51 ccpcompx Satish

1 0

Re: [petsc-dev] Error on Fugaku
by Mark Adams 14 Apr '21

14 Apr '21

The compiler: FCC (FCC) 4.5.0 20210304 *simulating gcc version 6.1*Copyright FUJITSU LIMITED 2019-2021 and -Kopenmp specifies to enable the Specification of OpenMP Application Program Interface. The -Knoopenmp option invalidates the -Kopenmp option. -Knoopenmp is set by default. When the -Kopenmp option is specified, -mt is assumed. *The -Kopenmp option is needed if an object program compiled with it exists in the command line as an input file.* I am going to try w/o Kopenmp. On Wed, Apr 14, 2021 at 8:15 PM Satish Balay <balay(a)mcs.anl.gov> wrote: > I don't understand kokkos build errors. > > For one - kokkos-kernels are taking long to build. And on the back end - > I'm getting into time limit [and terminated] > Kokkos take a very long time in configure on Summit. Thanks, > > Trying a build on the front-end - its running - for a long time.. > > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ > COMMAND > 4016901 > a04201 20 0 11.0g 10.7g 13876 R 99.3 11.6 66:22.35 ccpcompx > > > The above compile process has been running for more than an hour. Ok - its > finally done. > > When I enable openmp - kokkos cmake completes - but I get errors at build > time [BTW: I don't understand difference between -fopenmp and -Kopenmp] > > login6$ > ../petsc.save/arch-linux2-c-opt/lib/petsc/conf/reconfigure-arch-linux2-c-opt.py > --with-p4est-dir=$HOME/p4est-install --with-zlib-dir=$HOME/p4est-install > -download-kokkos --download-kokkos-kernelsx > --download-kokkos-commit=origin/develop > --download-kokkos-kernels-commit=origin/develop --download-cmake= > https://github.com/Kitware/CMake/releases/download/v3.20.1/cmake-3.20.1.tar… > '--download-kokkos-cmake-arguments=-DBUILD_TESTING=OFF > -DKokkos_ENABLE_LIBDL=OFF -DKokkos_ENABLE_AGGRESSIVE_VECTORIZATION=ON > -DKokkos_ENABLE_OPENMP=ON' --ignoreLinkOutput=1 --with-openmp=1 > > > /a04201/petsc.x/arch-linux2-c-opt/externalpackages/git.kokkos/petsc-build/core/src > -I/vol0004/ra010009/a04201/petsc.x/arch-linux2-c-opt/externalpackages/git.kokkos/core/src > -I/vol0004/ra010009/a04201/petsc.x/arch-linux2-c-opt/externalpackages/git.kokkos/petsc-build > -O -fopenmp -fPIC -fopenmp -O -fopenmp -fPIC -fopenmp -fopenmp -std=c++14 > -o CMakeFiles/kokkoscore.dir/OpenMP/Kokkos_OpenMP_Task.cpp.o -c > /vol0004/ra010009/a04201/petsc.x/arch-linux2-c-opt/externalpackages/git.kokkos/core/src/OpenMP/Kokkos_OpenMP_Task.cpp > gmake[2]: Leaving directory > '/vol0004/ra010009/a04201/petsc.x/arch-linux2-c-opt/externalpackages/git.kokkos/petsc-build' > gmake[1]: Leaving directory > '/vol0004/ra010009/a04201/petsc.x/arch-linux2-c-opt/externalpackages/git.kokkos/petsc-build'/home/ra010009/a04201/asmqkagl0.s: > Assembler messages: > /home/ra010009/a04201/asmqkagl0.s:20333: Error: symbol `.LEHB41' is > already defined > /home/ra010009/a04201/asmqkagl0.s:21334: Error: symbol `.LEHB41' is > already defined > gmake[2]: *** [core/src/CMakeFiles/kokkoscore.dir/build.make:312: > core/src/CMakeFiles/kokkoscore.dir/OpenMP/Kokkos_OpenMP_Exec.cpp.o] Error 1 > > Note: I have a successful configure run with p4est+kokkos+kokkos-kernels, > without-openmp > [for this - I had to build p4est on the back-end, then switch over to > front-end - and build the reset of the packages] > > Now I'm stuck at compiling petsc sources - > src/vec/is/sf/impls/basic/kokkos/sfkok.kokkos.cxx > > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ > COMMAND > 56491 > a04201 20 0 4944176 4.5g 11216 R 98.4 4.9 12:45.52 ccpcompx > > > > Satish > > On Wed, 14 Apr 2021, Mark Adams wrote: > > > PETSc seems to be skipping the compiler that it should use: > > > > -- Check for working CXX compiler: > > /opt/FJSVxtclanga/tcsds-1.2.31/bin/mpiFCC - skipped > > > > I see -fopenmp was added. This might be wrong. I use -Kopenmp. > > > > --help says: > > > > -fopenmp > > > > The -fopenmp option specifies to enable Specification of OpenMP > > Appli- > > cation Program Interface. > > > > When the -fopenmp option is specified, -mt is set. > > > > The -fopenmp option is needed if an object program compiled with > the > > -fopenmp option exists in the command line as input files. > > > > Any ideas? > > > > > > On Wed, Apr 14, 2021 at 5:17 PM Satish Balay <balay(a)mcs.anl.gov> wrote: > > > > > The following is working for me on the compute node [its completed > kokkos > > > (core) build - and is now building kokkos-kernels > > > > > > ~/petsc.save/arch-arm/lib/petsc/conf/reconfigure-arch-arm.py > > > --download-p4est --download-zlib --download-kokkos > > > --download-kokkos-kernels --download-kokkos-commit=origin/develop > > > --download-cmake= > > > > https://github.com/Kitware/CMake/releases/download/v3.20.1/cmake-3.20.1.tar… > > > > > > -DKokkos_ENABLE_OPENMP=ON give me some issue - so I didn't try any of > the > > > additional options.. > > > > > > Satish > > > > > > On Wed, 14 Apr 2021, Mark Adams wrote: > > > > > > > Satish, I get this error. > > > > > > > > I wonder if this syntax is wrong: > > > > > > > > '--download-kokkos-cmake-arguments=-DCMAKE_BUILD_TYPE=Release > > > > -DBUILD_TESTING=OFF -DKokkos_ENABLE_LIBDL=OFF > -DKokkos_ENABLE_OPENMP=ON > > > > -DKokkos_ENABLE_SERIAL=ON > -DKokkos_ENABLE_AGGRESSIVE_VECTORIZATION=ON', > > > > > > > > > > > > > > > > ============================================================================================= > > > > Configuring KOKKOS with cmake; this may > > > take > > > > several minutes > > > > > > > > > > > > ============================================================================================= > > > > > > > > Executing: /usr/bin/cmake .. > > > > -DCMAKE_INSTALL_PREFIX=/home/ra010009/a04199/petsc/arch-arm > > > > > > > > -DCMAKE_INSTALL_NAME_DIR:STRING="/home/ra010009/a04199/petsc/arch-arm/lib" > > > > -DCMAKE_INSTALL_LIBDIR:STRING="lib" -DCMAKE_VERBOSE_MAKEFILE=1 > > > > -DCMAKE_BUILD_TYPE=Release -DCMAKE_C_COMPILER="mpifcc" > > > > -DMPI_C_COMPILER="mpifcc" -DCMAKE_AR=/usr/bin/ar > > > > -DCMAKE_RANLIB=/usr/bin/ranlib -DCMAKE_C_FLAGS:STRING="-fPIC > > > -Kfast,openmp > > > > -fopenmp" -DCMAKE_C_FLAGS_DEBUG:STRING="-fPIC -Kfast,openmp -fopenmp" > > > > -DCMAKE_C_FLAGS_RELEASE:STRING="-fPIC -Kfast,openmp -fopenmp" > > > > -DCMAKE_CXX_COMPILER="mpiFCC" -DMPI_CXX_COMPILER="mpiFCC" > > > > -DCMAKE_CXX_FLAGS:STRING="-Kfast,openmp -fopenmp -fPIC -fopenmp" > > > > -DCMAKE_CXX_FLAGS_DEBUG:STRING="-Kfast,openmp -fopenmp -fPIC > -fopenmp" > > > > -DCMAKE_CXX_FLAGS_RELEASE:STRING="-Kfast,openmp -fopenmp -fPIC > -fopenmp" > > > > -DCMAKE_Fortran_COMPILER="mpifrt" -DMPI_Fortran_COMPILER="mpifrt" > > > > -DCMAKE_Fortran_FLAGS:STRING="-fPIC -O -fopenmp" > > > > -DCMAKE_Fortran_FLAGS_DEBUG:STRING="-fPIC -O -fopenmp" > > > > -DCMAKE_Fortran_FLAGS_RELEASE:STRING="-fPIC -O -fopenmp" > > > > -DCMAKE_EXE_LINKER_FLAGS:STRING=" -fopenmp" > -DBUILD_SHARED_LIBS:BOOL=ON > > > > -DUSE_XSDK_DEFAULTS=YES -DXSDK_ENABLE_DEBUG=NO > > > > -DCMAKE_INSTALL_RPATH_USE_LINK_PATH:BOOL=ON > > > > -DCMAKE_BUILD_WITH_INSTALL_RPATH:BOOL=ON -DKokkos_ENABLE_MPI=ON > > > > -DKokkos_ENABLE_SERIAL=ON -DKokkos_ENABLE_OPENMP=ON > > > > -DCMAKE_CXX_STANDARD="14" -DCMAKE_BUILD_TYPE=Release > -DBUILD_TESTING=OFF > > > > -DKokkos_ENABLE_LIBDL=OFF -DKokkos_ENABLE_OPENMP=ON > > > > -DKokkos_ENABLE_SERIAL=ON -DKokkos_ENABLE_AGGRESSIVE_VECTORIZATION=ON > > > > stdout: > > > > -- Setting default Kokkos CXX standard to 14 > > > > -- The CXX compiler identification is Fujitsu > > > > -- Check for working CXX compiler: > > > /opt/FJSVxtclanga/tcsds-1.2.31/bin/mpiFCC > > > > -- Check for working CXX compiler: > > > > /opt/FJSVxtclanga/tcsds-1.2.31/bin/mpiFCC -- works > > > > -- Detecting CXX compiler ABI info > > > > -- Detecting CXX compiler ABI info - done > > > > -- The project name is: Kokkos > > > > -- Configuring incomplete, errors occurred! > > > > See also > > > > > > > > "/vol0004/ra010009/a04199/petsc/arch-arm/externalpackages/git.kokkos/petsc-build/CMakeFiles/CMakeOutput.log". > > > > Error configuring KOKKOS with cmake Could not > execute > > > > "['/usr/bin/cmake .. > > > > -DCMAKE_INSTALL_PREFIX=/home/ra010009/a04199/petsc/arch-arm > > > > > > > > -DCMAKE_INSTALL_NAME_DIR:STRING="/home/ra010009/a04199/petsc/arch-arm/lib" > > > > -DCMAKE_INSTALL_LIBDIR:STRING="lib" -DCMAKE_VERBOSE_MAKEFILE=1 > > > > -DCMAKE_BUILD_TYPE=Release -DCMAKE_C_COMPILER="mpifcc" > > > > -DMPI_C_COMPILER="mpifcc" -DCMAKE_AR=/usr/bin/ar > > > > -DCMAKE_RANLIB=/usr/bin/ranlib -DCMAKE_C_FLAGS:STRING="-fPIC > > > -Kfast,openmp > > > > -fopenmp" -DCMAKE_C_FLAGS_DEBUG:STRING="-fPIC -Kfast,openmp -fopenmp" > > > > -DCMAKE_C_FLAGS_RELEASE:STRING="-fPIC -Kfast,openmp -fopenmp" > > > > -DCMAKE_CXX_COMPILER="mpiFCC" -DMPI_CXX_COMPILER="mpiFCC" > > > > -DCMAKE_CXX_FLAGS:STRING="-Kfast,openmp -fopenmp -fPIC -fopenmp" > > > > -DCMAKE_CXX_FLAGS_DEBUG:STRING="-Kfast,openmp -fopenmp -fPIC > -fopenmp" > > > > -DCMAKE_CXX_FLAGS_RELEASE:STRING="-Kfast,openmp -fopenmp -fPIC > -fopenmp" > > > > -DCMAKE_Fortran_COMPILER="mpifrt" -DMPI_Fortran_COMPILER="mpifrt" > > > > -DCMAKE_Fortran_FLAGS:STRING="-fPIC -O -fopenmp" > > > > -DCMAKE_Fortran_FLAGS_DEBUG:STRING="-fPIC -O -fopenmp" > > > > -DCMAKE_Fortran_FLAGS_RELEASE:STRING="-fPIC -O -fopenmp" > > > > -DCMAKE_EXE_LINKER_FLAGS:STRING=" -fopenmp" > -DBUILD_SHARED_LIBS:BOOL=ON > > > > -DUSE_XSDK_DEFAULTS=YES -DXSDK_ENABLE_DEBUG=NO > > > > -DCMAKE_INSTALL_RPATH_USE_LINK_PATH:BOOL=ON > > > > -DCMAKE_BUILD_WITH_INSTALL_RPATH:BOOL=ON -DKokkos_ENABLE_MPI=ON > > > > -DKokkos_ENABLE_SERIAL=ON -DKokkos_ENABLE_OPENMP=ON > > > > -DCMAKE_CXX_STANDARD="14" -DCMAKE_BUILD_TYPE=Release > -DBUILD_TESTING=OFF > > > > -DKokkos_ENABLE_LIBDL=OFF -DKokkos_ENABLE_OPENMP=ON > > > > -DKokkos_ENABLE_SERIAL=ON > -DKokkos_ENABLE_AGGRESSIVE_VECTORIZATION=ON']": > > > > -- Setting default Kokkos CXX standard to 14 > > > > -- The CXX compiler identification is Fujitsu > > > > -- Check for working CXX compiler: > > > /opt/FJSVxtclanga/tcsds-1.2.31/bin/mpiFCC > > > > -- Check for working CXX compiler: > > > > /opt/FJSVxtclanga/tcsds-1.2.31/bin/mpiFCC -- works > > > > -- Detecting CXX compiler ABI info > > > > -- Detecting CXX compiler ABI info - done > > > > -- The project name is: Kokkos > > > > -- Configuring incomplete, errors occurred! > > > > See also > > > > > > > > "/vol0004/ra010009/a04199/petsc/arch-arm/externalpackages/git.kokkos/petsc-build/CMakeFiles/CMakeOutput.log".CMake > > > > Error at cmake/kokkos_compiler_id.cmake:129 (STRING): > > > > > > > > * STRING sub-command REPLACE requires at least four arguments.*Call > > > Stack > > > > (most recent call first): > > > > cmake/kokkos_tribits.cmake:174 (INCLUDE) > > > > CMakeLists.txt:166 (KOKKOS_SETUP_BUILD_ENVIRONMENT) > > > > > > > > On Wed, Apr 14, 2021 at 3:34 PM Satish Balay <balay(a)mcs.anl.gov> > wrote: > > > > > > > > > Additional kokkos cmake arguments can be passed in via > > > > > --download-kokkos-cmake-arguments=string option. > > > > > > > > > > Satish > > > > > > > > > > On Wed, 14 Apr 2021, Mark Adams wrote: > > > > > > > > > > > Satish, > > > > > > > > > > > > For the fujitsu compiler OMP is -Kopenmp. > > > > > > > > > > > > Sarat (cc'ed) tells me that he built Kokkos with: > > > > > > > > > > > > cmake -DCMAKE_BUILD_TYPE=Release \ > > > > > > -DCMAKE_INSTALL_PREFIX=${KOKKOS_SRC_DIR}/install \ > > > > > > -DBUILD_TESTING=OFF \ > > > > > > -DKokkos_ENABLE_LIBDL=OFF \ > > > > > > -DKokkos_ENABLE_OPENMP=ON \ > > > > > > -DKokkos_ENABLE_SERIAL=ON \ > > > > > > -DKokkos_ENABLE_AGGRESSIVE_VECTORIZATION=ON \ > > > > > > .. > > > > > > > > > > > > How might I make this happen in PETSc? > > > > > > > > > > > > Thanks, > > > > > > Mark > > > > > > > > > > > > > > > > > > On Wed, Apr 14, 2021 at 2:44 PM Satish Balay <balay(a)mcs.anl.gov> > > > wrote: > > > > > > > > > > > > > On Wed, 14 Apr 2021, Mark Adams wrote: > > > > > > > > > > > > > > > I have this building now. > > > > > > > > Do you know anything about OpenMP? > > > > > > > > I can add --with-openmp > > > > > > > > That should get Kokkos to be made with OpenMP. > > > > > > > > Should PETSc deal with the compilers correctly? > > > > > > > > > > > > > > Well it tries the following compiler options for openmp. > > > > > > > > > > > > > > oflags = ["-fopenmp", # Gnu > > > > > > > "-qsmp=omp",# IBM XL C/C++ > > > > > > > "-h omp", # Cray. Must come after XL because XL > > > > > interprets > > > > > > > this option as meaning "-soname omp" > > > > > > > "-mp", # Portland Group > > > > > > > "-Qopenmp", # Intel windows > > > > > > > "-openmp", # Intel > > > > > > > "-xopenmp", # Sun > > > > > > > "+Oopenmp", # HP > > > > > > > "/openmp" # Microsoft Visual Studio > > > > > > > ] > > > > > > > > > > > > > > I don't know what the flag for fugaku compiler is. > > > > > > > > > > > > > > Satish > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > Mark > > > > > > > > > > > > > > > > On Wed, Apr 14, 2021 at 1:45 PM Mark Adams <mfadams(a)lbl.gov> > > > wrote: > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > If you feel inspired you could try Kokkos :|| > > > > > > > > > I am in a parking lot waiting for my daughter but can try > this > > > > > when I > > > > > > > get > > > > > > > > > home, > > > > > > > > > Thanks again, > > > > > > > > > Mark > > > > > > > > > > > > > > > > > > On Wed, Apr 14, 2021 at 1:33 PM Satish Balay < > > > balay(a)mcs.anl.gov> > > > > > > > wrote: > > > > > > > > > > > > > > > > > >> I think I allocated a single node - and did the build on > it. > > > > > > > > >> > > > > > > > > >> Now I'm getting an error - don't know what changed.. > > > > > > > > >> > > > > > > > > >> login6$ pjsub --interact -L "node=1" -L > > > "rscunit=rscunit_ft01" -L > > > > > > > > >> "rscgrp=eap-int" -L "elapse=1:00:00" --sparam > "wait-time=600" > > > > > > > > >> [ERR.] PJM 0059 pjsub rscgrp=eap-int is disabled. > > > > > > > > >> > > > > > > > > >> Ok - the following worked.. > > > > > > > > >> > > > > > > > > >> login6$ pjsub --interact -L "node=1" -L > > > "rscunit=rscunit_ft01" -L > > > > > > > > >> "elapse=1:00:00" --sparam "wait-time=600" > > > > > > > > >> [INFO] PJM 0000 pjsub Job 6301572 submitted. > > > > > > > > >> [INFO] PJM 0081 .connected. > > > > > > > > >> [INFO] PJM 0082 pjsub Interactive job 6301572 started. > > > > > > > > >> [a04201@j31-3110s petsc]$ > > > > > > > > >> > > > > > > > > >> Ok - trying this build now. > > > > > > > > >> > > > > > > > > >> [a04201@j31-3110s petsc]$ cat > > > > > > > > >> > ~/petsc.save/arch-arm/lib/petsc/conf/reconfigure-arch-arm.py > > > > > > > > >> #!/usr/bin/python3 > > > > > > > > >> if __name__ == '__main__': > > > > > > > > >> import sys > > > > > > > > >> import os > > > > > > > > >> sys.path.insert(0, os.path.abspath('config')) > > > > > > > > >> import configure > > > > > > > > >> configure_options = [ > > > > > > > > >> '--with-blaslapack-lib=-lfjlapack', > > > > > > > > >> '--with-debugging=0', > > > > > > > > >> 'CC=mpifcc', > > > > > > > > >> 'CXX=mpiFCC', > > > > > > > > >> 'FC=mpifrt', > > > > > > > > >> 'PETSC_ARCH=arch-arm', > > > > > > > > >> ] > > > > > > > > >> configure.petsc_configure(configure_options) > > > > > > > > >> [a04201@j31-3110s petsc]$ > > > > > > > > >> > ~/petsc.save/arch-arm/lib/petsc/conf/reconfigure-arch-arm.py > > > > > > > > >> --download-p4est --download-zlib > > > > > > > > >> <snip> > > > > > > > > >> p4est: > > > > > > > > >> Includes: > -I/vol0004/ra010009/a04201/petsc/arch-arm/include > > > > > > > > >> Library: > > > -Wl,-rpath,/vol0004/ra010009/a04201/petsc/arch-arm/lib > > > > > > > > >> -L/vol0004/ra010009/a04201/petsc/arch-arm/lib -lp4est -lsc > > > > > > > > >> > > > > > > > > >> Ok - this worked for me. > > > > > > > > >> > > > > > > > > >> Satish > > > > > > > > >> > > > > > > > > >> On Wed, 14 Apr 2021, Mark Adams wrote: > > > > > > > > >> > > > > > > > > >> > Do you recall what nodes you use to build on a "compute" > > > node, > > > > > to > > > > > > > avoid > > > > > > > > >> > cross compilation? > > > > > > > > >> > > > > > > > > > >> > On Wed, Apr 14, 2021 at 12:08 PM Satish Balay < > > > > > balay(a)mcs.anl.gov> > > > > > > > > >> wrote: > > > > > > > > >> > > > > > > > > > >> > > looks like p4est cannot be cross-compiled. > > > > > > > > >> > > > > > > > > > > >> > > Satish > > > > > > > > >> > > > > > > > > > > >> > > On Wed, 14 Apr 2021, Mark Adams wrote: > > > > > > > > >> > > > > > > > > > > >> > > > I get this error with p4est on Fugaku. > > > > > > > > >> > > > It is a Fortran error. Odd. > > > > > > > > >> > > > Mark > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >

1 0

Re: [petsc-dev] Error on Fugaku
by Satish Balay 14 Apr '21

14 Apr '21

I don't understand kokkos build errors. For one - kokkos-kernels are taking long to build. And on the back end - I'm getting into time limit [and terminated] Trying a build on the front-end - its running - for a long time.. PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 4016901 a04201 20 0 11.0g 10.7g 13876 R 99.3 11.6 66:22.35 ccpcompx The above compile process has been running for more than an hour. Ok - its finally done. When I enable openmp - kokkos cmake completes - but I get errors at build time [BTW: I don't understand difference between -fopenmp and -Kopenmp] login6$ ../petsc.save/arch-linux2-c-opt/lib/petsc/conf/reconfigure-arch-linux2-c-opt.py --with-p4est-dir=$HOME/p4est-install --with-zlib-dir=$HOME/p4est-install -download-kokkos --download-kokkos-kernelsx --download-kokkos-commit=origin/develop --download-kokkos-kernels-commit=origin/develop --download-cmake=https://github.com/Kitware/CMake/releases/download/v3.20.1/cmake-3.20.1.tar.gz '--download-kokkos-cmake-arguments=-DBUILD_TESTING=OFF -DKokkos_ENABLE_LIBDL=OFF -DKokkos_ENABLE_AGGRESSIVE_VECTORIZATION=ON -DKokkos_ENABLE_OPENMP=ON' --ignoreLinkOutput=1 --with-openmp=1 /a04201/petsc.x/arch-linux2-c-opt/externalpackages/git.kokkos/petsc-build/core/src -I/vol0004/ra010009/a04201/petsc.x/arch-linux2-c-opt/externalpackages/git.kokkos/core/src -I/vol0004/ra010009/a04201/petsc.x/arch-linux2-c-opt/externalpackages/git.kokkos/petsc-build -O -fopenmp -fPIC -fopenmp -O -fopenmp -fPIC -fopenmp -fopenmp -std=c++14 -o CMakeFiles/kokkoscore.dir/OpenMP/Kokkos_OpenMP_Task.cpp.o -c /vol0004/ra010009/a04201/petsc.x/arch-linux2-c-opt/externalpackages/git.kokkos/core/src/OpenMP/Kokkos_OpenMP_Task.cpp gmake[2]: Leaving directory '/vol0004/ra010009/a04201/petsc.x/arch-linux2-c-opt/externalpackages/git.kokkos/petsc-build' gmake[1]: Leaving directory '/vol0004/ra010009/a04201/petsc.x/arch-linux2-c-opt/externalpackages/git.kokkos/petsc-build'/home/ra010009/a04201/asmqkagl0.s: Assembler messages: /home/ra010009/a04201/asmqkagl0.s:20333: Error: symbol `.LEHB41' is already defined /home/ra010009/a04201/asmqkagl0.s:21334: Error: symbol `.LEHB41' is already defined gmake[2]: *** [core/src/CMakeFiles/kokkoscore.dir/build.make:312: core/src/CMakeFiles/kokkoscore.dir/OpenMP/Kokkos_OpenMP_Exec.cpp.o] Error 1 Note: I have a successful configure run with p4est+kokkos+kokkos-kernels, without-openmp [for this - I had to build p4est on the back-end, then switch over to front-end - and build the reset of the packages] Now I'm stuck at compiling petsc sources - src/vec/is/sf/impls/basic/kokkos/sfkok.kokkos.cxx PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 56491 a04201 20 0 4944176 4.5g 11216 R 98.4 4.9 12:45.52 ccpcompx Satish On Wed, 14 Apr 2021, Mark Adams wrote: > PETSc seems to be skipping the compiler that it should use: > > -- Check for working CXX compiler: > /opt/FJSVxtclanga/tcsds-1.2.31/bin/mpiFCC - skipped > > I see -fopenmp was added. This might be wrong. I use -Kopenmp. > > --help says: > > -fopenmp > > The -fopenmp option specifies to enable Specification of OpenMP > Appli- > cation Program Interface. > > When the -fopenmp option is specified, -mt is set. > > The -fopenmp option is needed if an object program compiled with the > -fopenmp option exists in the command line as input files. > > Any ideas? > > > On Wed, Apr 14, 2021 at 5:17 PM Satish Balay <balay(a)mcs.anl.gov> wrote: > > > The following is working for me on the compute node [its completed kokkos > > (core) build - and is now building kokkos-kernels > > > > ~/petsc.save/arch-arm/lib/petsc/conf/reconfigure-arch-arm.py > > --download-p4est --download-zlib --download-kokkos > > --download-kokkos-kernels --download-kokkos-commit=origin/develop > > --download-cmake= > > https://github.com/Kitware/CMake/releases/download/v3.20.1/cmake-3.20.1.tar… > > > > -DKokkos_ENABLE_OPENMP=ON give me some issue - so I didn't try any of the > > additional options.. > > > > Satish > > > > On Wed, 14 Apr 2021, Mark Adams wrote: > > > > > Satish, I get this error. > > > > > > I wonder if this syntax is wrong: > > > > > > '--download-kokkos-cmake-arguments=-DCMAKE_BUILD_TYPE=Release > > > -DBUILD_TESTING=OFF -DKokkos_ENABLE_LIBDL=OFF -DKokkos_ENABLE_OPENMP=ON > > > -DKokkos_ENABLE_SERIAL=ON -DKokkos_ENABLE_AGGRESSIVE_VECTORIZATION=ON', > > > > > > > > > > > ============================================================================================= > > > Configuring KOKKOS with cmake; this may > > take > > > several minutes > > > > > > > > ============================================================================================= > > > > > > Executing: /usr/bin/cmake .. > > > -DCMAKE_INSTALL_PREFIX=/home/ra010009/a04199/petsc/arch-arm > > > > > -DCMAKE_INSTALL_NAME_DIR:STRING="/home/ra010009/a04199/petsc/arch-arm/lib" > > > -DCMAKE_INSTALL_LIBDIR:STRING="lib" -DCMAKE_VERBOSE_MAKEFILE=1 > > > -DCMAKE_BUILD_TYPE=Release -DCMAKE_C_COMPILER="mpifcc" > > > -DMPI_C_COMPILER="mpifcc" -DCMAKE_AR=/usr/bin/ar > > > -DCMAKE_RANLIB=/usr/bin/ranlib -DCMAKE_C_FLAGS:STRING="-fPIC > > -Kfast,openmp > > > -fopenmp" -DCMAKE_C_FLAGS_DEBUG:STRING="-fPIC -Kfast,openmp -fopenmp" > > > -DCMAKE_C_FLAGS_RELEASE:STRING="-fPIC -Kfast,openmp -fopenmp" > > > -DCMAKE_CXX_COMPILER="mpiFCC" -DMPI_CXX_COMPILER="mpiFCC" > > > -DCMAKE_CXX_FLAGS:STRING="-Kfast,openmp -fopenmp -fPIC -fopenmp" > > > -DCMAKE_CXX_FLAGS_DEBUG:STRING="-Kfast,openmp -fopenmp -fPIC -fopenmp" > > > -DCMAKE_CXX_FLAGS_RELEASE:STRING="-Kfast,openmp -fopenmp -fPIC -fopenmp" > > > -DCMAKE_Fortran_COMPILER="mpifrt" -DMPI_Fortran_COMPILER="mpifrt" > > > -DCMAKE_Fortran_FLAGS:STRING="-fPIC -O -fopenmp" > > > -DCMAKE_Fortran_FLAGS_DEBUG:STRING="-fPIC -O -fopenmp" > > > -DCMAKE_Fortran_FLAGS_RELEASE:STRING="-fPIC -O -fopenmp" > > > -DCMAKE_EXE_LINKER_FLAGS:STRING=" -fopenmp" -DBUILD_SHARED_LIBS:BOOL=ON > > > -DUSE_XSDK_DEFAULTS=YES -DXSDK_ENABLE_DEBUG=NO > > > -DCMAKE_INSTALL_RPATH_USE_LINK_PATH:BOOL=ON > > > -DCMAKE_BUILD_WITH_INSTALL_RPATH:BOOL=ON -DKokkos_ENABLE_MPI=ON > > > -DKokkos_ENABLE_SERIAL=ON -DKokkos_ENABLE_OPENMP=ON > > > -DCMAKE_CXX_STANDARD="14" -DCMAKE_BUILD_TYPE=Release -DBUILD_TESTING=OFF > > > -DKokkos_ENABLE_LIBDL=OFF -DKokkos_ENABLE_OPENMP=ON > > > -DKokkos_ENABLE_SERIAL=ON -DKokkos_ENABLE_AGGRESSIVE_VECTORIZATION=ON > > > stdout: > > > -- Setting default Kokkos CXX standard to 14 > > > -- The CXX compiler identification is Fujitsu > > > -- Check for working CXX compiler: > > /opt/FJSVxtclanga/tcsds-1.2.31/bin/mpiFCC > > > -- Check for working CXX compiler: > > > /opt/FJSVxtclanga/tcsds-1.2.31/bin/mpiFCC -- works > > > -- Detecting CXX compiler ABI info > > > -- Detecting CXX compiler ABI info - done > > > -- The project name is: Kokkos > > > -- Configuring incomplete, errors occurred! > > > See also > > > > > "/vol0004/ra010009/a04199/petsc/arch-arm/externalpackages/git.kokkos/petsc-build/CMakeFiles/CMakeOutput.log". > > > Error configuring KOKKOS with cmake Could not execute > > > "['/usr/bin/cmake .. > > > -DCMAKE_INSTALL_PREFIX=/home/ra010009/a04199/petsc/arch-arm > > > > > -DCMAKE_INSTALL_NAME_DIR:STRING="/home/ra010009/a04199/petsc/arch-arm/lib" > > > -DCMAKE_INSTALL_LIBDIR:STRING="lib" -DCMAKE_VERBOSE_MAKEFILE=1 > > > -DCMAKE_BUILD_TYPE=Release -DCMAKE_C_COMPILER="mpifcc" > > > -DMPI_C_COMPILER="mpifcc" -DCMAKE_AR=/usr/bin/ar > > > -DCMAKE_RANLIB=/usr/bin/ranlib -DCMAKE_C_FLAGS:STRING="-fPIC > > -Kfast,openmp > > > -fopenmp" -DCMAKE_C_FLAGS_DEBUG:STRING="-fPIC -Kfast,openmp -fopenmp" > > > -DCMAKE_C_FLAGS_RELEASE:STRING="-fPIC -Kfast,openmp -fopenmp" > > > -DCMAKE_CXX_COMPILER="mpiFCC" -DMPI_CXX_COMPILER="mpiFCC" > > > -DCMAKE_CXX_FLAGS:STRING="-Kfast,openmp -fopenmp -fPIC -fopenmp" > > > -DCMAKE_CXX_FLAGS_DEBUG:STRING="-Kfast,openmp -fopenmp -fPIC -fopenmp" > > > -DCMAKE_CXX_FLAGS_RELEASE:STRING="-Kfast,openmp -fopenmp -fPIC -fopenmp" > > > -DCMAKE_Fortran_COMPILER="mpifrt" -DMPI_Fortran_COMPILER="mpifrt" > > > -DCMAKE_Fortran_FLAGS:STRING="-fPIC -O -fopenmp" > > > -DCMAKE_Fortran_FLAGS_DEBUG:STRING="-fPIC -O -fopenmp" > > > -DCMAKE_Fortran_FLAGS_RELEASE:STRING="-fPIC -O -fopenmp" > > > -DCMAKE_EXE_LINKER_FLAGS:STRING=" -fopenmp" -DBUILD_SHARED_LIBS:BOOL=ON > > > -DUSE_XSDK_DEFAULTS=YES -DXSDK_ENABLE_DEBUG=NO > > > -DCMAKE_INSTALL_RPATH_USE_LINK_PATH:BOOL=ON > > > -DCMAKE_BUILD_WITH_INSTALL_RPATH:BOOL=ON -DKokkos_ENABLE_MPI=ON > > > -DKokkos_ENABLE_SERIAL=ON -DKokkos_ENABLE_OPENMP=ON > > > -DCMAKE_CXX_STANDARD="14" -DCMAKE_BUILD_TYPE=Release -DBUILD_TESTING=OFF > > > -DKokkos_ENABLE_LIBDL=OFF -DKokkos_ENABLE_OPENMP=ON > > > -DKokkos_ENABLE_SERIAL=ON -DKokkos_ENABLE_AGGRESSIVE_VECTORIZATION=ON']": > > > -- Setting default Kokkos CXX standard to 14 > > > -- The CXX compiler identification is Fujitsu > > > -- Check for working CXX compiler: > > /opt/FJSVxtclanga/tcsds-1.2.31/bin/mpiFCC > > > -- Check for working CXX compiler: > > > /opt/FJSVxtclanga/tcsds-1.2.31/bin/mpiFCC -- works > > > -- Detecting CXX compiler ABI info > > > -- Detecting CXX compiler ABI info - done > > > -- The project name is: Kokkos > > > -- Configuring incomplete, errors occurred! > > > See also > > > > > "/vol0004/ra010009/a04199/petsc/arch-arm/externalpackages/git.kokkos/petsc-build/CMakeFiles/CMakeOutput.log".CMake > > > Error at cmake/kokkos_compiler_id.cmake:129 (STRING): > > > > > > * STRING sub-command REPLACE requires at least four arguments.*Call > > Stack > > > (most recent call first): > > > cmake/kokkos_tribits.cmake:174 (INCLUDE) > > > CMakeLists.txt:166 (KOKKOS_SETUP_BUILD_ENVIRONMENT) > > > > > > On Wed, Apr 14, 2021 at 3:34 PM Satish Balay <balay(a)mcs.anl.gov> wrote: > > > > > > > Additional kokkos cmake arguments can be passed in via > > > > --download-kokkos-cmake-arguments=string option. > > > > > > > > Satish > > > > > > > > On Wed, 14 Apr 2021, Mark Adams wrote: > > > > > > > > > Satish, > > > > > > > > > > For the fujitsu compiler OMP is -Kopenmp. > > > > > > > > > > Sarat (cc'ed) tells me that he built Kokkos with: > > > > > > > > > > cmake -DCMAKE_BUILD_TYPE=Release \ > > > > > -DCMAKE_INSTALL_PREFIX=${KOKKOS_SRC_DIR}/install \ > > > > > -DBUILD_TESTING=OFF \ > > > > > -DKokkos_ENABLE_LIBDL=OFF \ > > > > > -DKokkos_ENABLE_OPENMP=ON \ > > > > > -DKokkos_ENABLE_SERIAL=ON \ > > > > > -DKokkos_ENABLE_AGGRESSIVE_VECTORIZATION=ON \ > > > > > .. > > > > > > > > > > How might I make this happen in PETSc? > > > > > > > > > > Thanks, > > > > > Mark > > > > > > > > > > > > > > > On Wed, Apr 14, 2021 at 2:44 PM Satish Balay <balay(a)mcs.anl.gov> > > wrote: > > > > > > > > > > > On Wed, 14 Apr 2021, Mark Adams wrote: > > > > > > > > > > > > > I have this building now. > > > > > > > Do you know anything about OpenMP? > > > > > > > I can add --with-openmp > > > > > > > That should get Kokkos to be made with OpenMP. > > > > > > > Should PETSc deal with the compilers correctly? > > > > > > > > > > > > Well it tries the following compiler options for openmp. > > > > > > > > > > > > oflags = ["-fopenmp", # Gnu > > > > > > "-qsmp=omp",# IBM XL C/C++ > > > > > > "-h omp", # Cray. Must come after XL because XL > > > > interprets > > > > > > this option as meaning "-soname omp" > > > > > > "-mp", # Portland Group > > > > > > "-Qopenmp", # Intel windows > > > > > > "-openmp", # Intel > > > > > > "-xopenmp", # Sun > > > > > > "+Oopenmp", # HP > > > > > > "/openmp" # Microsoft Visual Studio > > > > > > ] > > > > > > > > > > > > I don't know what the flag for fugaku compiler is. > > > > > > > > > > > > Satish > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > Mark > > > > > > > > > > > > > > On Wed, Apr 14, 2021 at 1:45 PM Mark Adams <mfadams(a)lbl.gov> > > wrote: > > > > > > > > > > > > > > > Thanks, > > > > > > > > If you feel inspired you could try Kokkos :|| > > > > > > > > I am in a parking lot waiting for my daughter but can try this > > > > when I > > > > > > get > > > > > > > > home, > > > > > > > > Thanks again, > > > > > > > > Mark > > > > > > > > > > > > > > > > On Wed, Apr 14, 2021 at 1:33 PM Satish Balay < > > balay(a)mcs.anl.gov> > > > > > > wrote: > > > > > > > > > > > > > > > >> I think I allocated a single node - and did the build on it. > > > > > > > >> > > > > > > > >> Now I'm getting an error - don't know what changed.. > > > > > > > >> > > > > > > > >> login6$ pjsub --interact -L "node=1" -L > > "rscunit=rscunit_ft01" -L > > > > > > > >> "rscgrp=eap-int" -L "elapse=1:00:00" --sparam "wait-time=600" > > > > > > > >> [ERR.] PJM 0059 pjsub rscgrp=eap-int is disabled. > > > > > > > >> > > > > > > > >> Ok - the following worked.. > > > > > > > >> > > > > > > > >> login6$ pjsub --interact -L "node=1" -L > > "rscunit=rscunit_ft01" -L > > > > > > > >> "elapse=1:00:00" --sparam "wait-time=600" > > > > > > > >> [INFO] PJM 0000 pjsub Job 6301572 submitted. > > > > > > > >> [INFO] PJM 0081 .connected. > > > > > > > >> [INFO] PJM 0082 pjsub Interactive job 6301572 started. > > > > > > > >> [a04201@j31-3110s petsc]$ > > > > > > > >> > > > > > > > >> Ok - trying this build now. > > > > > > > >> > > > > > > > >> [a04201@j31-3110s petsc]$ cat > > > > > > > >> ~/petsc.save/arch-arm/lib/petsc/conf/reconfigure-arch-arm.py > > > > > > > >> #!/usr/bin/python3 > > > > > > > >> if __name__ == '__main__': > > > > > > > >> import sys > > > > > > > >> import os > > > > > > > >> sys.path.insert(0, os.path.abspath('config')) > > > > > > > >> import configure > > > > > > > >> configure_options = [ > > > > > > > >> '--with-blaslapack-lib=-lfjlapack', > > > > > > > >> '--with-debugging=0', > > > > > > > >> 'CC=mpifcc', > > > > > > > >> 'CXX=mpiFCC', > > > > > > > >> 'FC=mpifrt', > > > > > > > >> 'PETSC_ARCH=arch-arm', > > > > > > > >> ] > > > > > > > >> configure.petsc_configure(configure_options) > > > > > > > >> [a04201@j31-3110s petsc]$ > > > > > > > >> ~/petsc.save/arch-arm/lib/petsc/conf/reconfigure-arch-arm.py > > > > > > > >> --download-p4est --download-zlib > > > > > > > >> <snip> > > > > > > > >> p4est: > > > > > > > >> Includes: -I/vol0004/ra010009/a04201/petsc/arch-arm/include > > > > > > > >> Library: > > -Wl,-rpath,/vol0004/ra010009/a04201/petsc/arch-arm/lib > > > > > > > >> -L/vol0004/ra010009/a04201/petsc/arch-arm/lib -lp4est -lsc > > > > > > > >> > > > > > > > >> Ok - this worked for me. > > > > > > > >> > > > > > > > >> Satish > > > > > > > >> > > > > > > > >> On Wed, 14 Apr 2021, Mark Adams wrote: > > > > > > > >> > > > > > > > >> > Do you recall what nodes you use to build on a "compute" > > node, > > > > to > > > > > > avoid > > > > > > > >> > cross compilation? > > > > > > > >> > > > > > > > > >> > On Wed, Apr 14, 2021 at 12:08 PM Satish Balay < > > > > balay(a)mcs.anl.gov> > > > > > > > >> wrote: > > > > > > > >> > > > > > > > > >> > > looks like p4est cannot be cross-compiled. > > > > > > > >> > > > > > > > > > >> > > Satish > > > > > > > >> > > > > > > > > > >> > > On Wed, 14 Apr 2021, Mark Adams wrote: > > > > > > > >> > > > > > > > > > >> > > > I get this error with p4est on Fugaku. > > > > > > > >> > > > It is a Fortran error. Odd. > > > > > > > >> > > > Mark > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >

1 0

Re: [petsc-dev] Error on Fugaku
by Mark Adams 14 Apr '21

14 Apr '21

PETSc seems to be skipping the compiler that it should use: -- Check for working CXX compiler: /opt/FJSVxtclanga/tcsds-1.2.31/bin/mpiFCC - skipped I see -fopenmp was added. This might be wrong. I use -Kopenmp. --help says: -fopenmp The -fopenmp option specifies to enable Specification of OpenMP Appli- cation Program Interface. When the -fopenmp option is specified, -mt is set. The -fopenmp option is needed if an object program compiled with the -fopenmp option exists in the command line as input files. Any ideas? On Wed, Apr 14, 2021 at 5:17 PM Satish Balay <balay(a)mcs.anl.gov> wrote: > The following is working for me on the compute node [its completed kokkos > (core) build - and is now building kokkos-kernels > > ~/petsc.save/arch-arm/lib/petsc/conf/reconfigure-arch-arm.py > --download-p4est --download-zlib --download-kokkos > --download-kokkos-kernels --download-kokkos-commit=origin/develop > --download-cmake= > https://github.com/Kitware/CMake/releases/download/v3.20.1/cmake-3.20.1.tar… > > -DKokkos_ENABLE_OPENMP=ON give me some issue - so I didn't try any of the > additional options.. > > Satish > > On Wed, 14 Apr 2021, Mark Adams wrote: > > > Satish, I get this error. > > > > I wonder if this syntax is wrong: > > > > '--download-kokkos-cmake-arguments=-DCMAKE_BUILD_TYPE=Release > > -DBUILD_TESTING=OFF -DKokkos_ENABLE_LIBDL=OFF -DKokkos_ENABLE_OPENMP=ON > > -DKokkos_ENABLE_SERIAL=ON -DKokkos_ENABLE_AGGRESSIVE_VECTORIZATION=ON', > > > > > > > ============================================================================================= > > Configuring KOKKOS with cmake; this may > take > > several minutes > > > > > ============================================================================================= > > > > Executing: /usr/bin/cmake .. > > -DCMAKE_INSTALL_PREFIX=/home/ra010009/a04199/petsc/arch-arm > > > -DCMAKE_INSTALL_NAME_DIR:STRING="/home/ra010009/a04199/petsc/arch-arm/lib" > > -DCMAKE_INSTALL_LIBDIR:STRING="lib" -DCMAKE_VERBOSE_MAKEFILE=1 > > -DCMAKE_BUILD_TYPE=Release -DCMAKE_C_COMPILER="mpifcc" > > -DMPI_C_COMPILER="mpifcc" -DCMAKE_AR=/usr/bin/ar > > -DCMAKE_RANLIB=/usr/bin/ranlib -DCMAKE_C_FLAGS:STRING="-fPIC > -Kfast,openmp > > -fopenmp" -DCMAKE_C_FLAGS_DEBUG:STRING="-fPIC -Kfast,openmp -fopenmp" > > -DCMAKE_C_FLAGS_RELEASE:STRING="-fPIC -Kfast,openmp -fopenmp" > > -DCMAKE_CXX_COMPILER="mpiFCC" -DMPI_CXX_COMPILER="mpiFCC" > > -DCMAKE_CXX_FLAGS:STRING="-Kfast,openmp -fopenmp -fPIC -fopenmp" > > -DCMAKE_CXX_FLAGS_DEBUG:STRING="-Kfast,openmp -fopenmp -fPIC -fopenmp" > > -DCMAKE_CXX_FLAGS_RELEASE:STRING="-Kfast,openmp -fopenmp -fPIC -fopenmp" > > -DCMAKE_Fortran_COMPILER="mpifrt" -DMPI_Fortran_COMPILER="mpifrt" > > -DCMAKE_Fortran_FLAGS:STRING="-fPIC -O -fopenmp" > > -DCMAKE_Fortran_FLAGS_DEBUG:STRING="-fPIC -O -fopenmp" > > -DCMAKE_Fortran_FLAGS_RELEASE:STRING="-fPIC -O -fopenmp" > > -DCMAKE_EXE_LINKER_FLAGS:STRING=" -fopenmp" -DBUILD_SHARED_LIBS:BOOL=ON > > -DUSE_XSDK_DEFAULTS=YES -DXSDK_ENABLE_DEBUG=NO > > -DCMAKE_INSTALL_RPATH_USE_LINK_PATH:BOOL=ON > > -DCMAKE_BUILD_WITH_INSTALL_RPATH:BOOL=ON -DKokkos_ENABLE_MPI=ON > > -DKokkos_ENABLE_SERIAL=ON -DKokkos_ENABLE_OPENMP=ON > > -DCMAKE_CXX_STANDARD="14" -DCMAKE_BUILD_TYPE=Release -DBUILD_TESTING=OFF > > -DKokkos_ENABLE_LIBDL=OFF -DKokkos_ENABLE_OPENMP=ON > > -DKokkos_ENABLE_SERIAL=ON -DKokkos_ENABLE_AGGRESSIVE_VECTORIZATION=ON > > stdout: > > -- Setting default Kokkos CXX standard to 14 > > -- The CXX compiler identification is Fujitsu > > -- Check for working CXX compiler: > /opt/FJSVxtclanga/tcsds-1.2.31/bin/mpiFCC > > -- Check for working CXX compiler: > > /opt/FJSVxtclanga/tcsds-1.2.31/bin/mpiFCC -- works > > -- Detecting CXX compiler ABI info > > -- Detecting CXX compiler ABI info - done > > -- The project name is: Kokkos > > -- Configuring incomplete, errors occurred! > > See also > > > "/vol0004/ra010009/a04199/petsc/arch-arm/externalpackages/git.kokkos/petsc-build/CMakeFiles/CMakeOutput.log". > > Error configuring KOKKOS with cmake Could not execute > > "['/usr/bin/cmake .. > > -DCMAKE_INSTALL_PREFIX=/home/ra010009/a04199/petsc/arch-arm > > > -DCMAKE_INSTALL_NAME_DIR:STRING="/home/ra010009/a04199/petsc/arch-arm/lib" > > -DCMAKE_INSTALL_LIBDIR:STRING="lib" -DCMAKE_VERBOSE_MAKEFILE=1 > > -DCMAKE_BUILD_TYPE=Release -DCMAKE_C_COMPILER="mpifcc" > > -DMPI_C_COMPILER="mpifcc" -DCMAKE_AR=/usr/bin/ar > > -DCMAKE_RANLIB=/usr/bin/ranlib -DCMAKE_C_FLAGS:STRING="-fPIC > -Kfast,openmp > > -fopenmp" -DCMAKE_C_FLAGS_DEBUG:STRING="-fPIC -Kfast,openmp -fopenmp" > > -DCMAKE_C_FLAGS_RELEASE:STRING="-fPIC -Kfast,openmp -fopenmp" > > -DCMAKE_CXX_COMPILER="mpiFCC" -DMPI_CXX_COMPILER="mpiFCC" > > -DCMAKE_CXX_FLAGS:STRING="-Kfast,openmp -fopenmp -fPIC -fopenmp" > > -DCMAKE_CXX_FLAGS_DEBUG:STRING="-Kfast,openmp -fopenmp -fPIC -fopenmp" > > -DCMAKE_CXX_FLAGS_RELEASE:STRING="-Kfast,openmp -fopenmp -fPIC -fopenmp" > > -DCMAKE_Fortran_COMPILER="mpifrt" -DMPI_Fortran_COMPILER="mpifrt" > > -DCMAKE_Fortran_FLAGS:STRING="-fPIC -O -fopenmp" > > -DCMAKE_Fortran_FLAGS_DEBUG:STRING="-fPIC -O -fopenmp" > > -DCMAKE_Fortran_FLAGS_RELEASE:STRING="-fPIC -O -fopenmp" > > -DCMAKE_EXE_LINKER_FLAGS:STRING=" -fopenmp" -DBUILD_SHARED_LIBS:BOOL=ON > > -DUSE_XSDK_DEFAULTS=YES -DXSDK_ENABLE_DEBUG=NO > > -DCMAKE_INSTALL_RPATH_USE_LINK_PATH:BOOL=ON > > -DCMAKE_BUILD_WITH_INSTALL_RPATH:BOOL=ON -DKokkos_ENABLE_MPI=ON > > -DKokkos_ENABLE_SERIAL=ON -DKokkos_ENABLE_OPENMP=ON > > -DCMAKE_CXX_STANDARD="14" -DCMAKE_BUILD_TYPE=Release -DBUILD_TESTING=OFF > > -DKokkos_ENABLE_LIBDL=OFF -DKokkos_ENABLE_OPENMP=ON > > -DKokkos_ENABLE_SERIAL=ON -DKokkos_ENABLE_AGGRESSIVE_VECTORIZATION=ON']": > > -- Setting default Kokkos CXX standard to 14 > > -- The CXX compiler identification is Fujitsu > > -- Check for working CXX compiler: > /opt/FJSVxtclanga/tcsds-1.2.31/bin/mpiFCC > > -- Check for working CXX compiler: > > /opt/FJSVxtclanga/tcsds-1.2.31/bin/mpiFCC -- works > > -- Detecting CXX compiler ABI info > > -- Detecting CXX compiler ABI info - done > > -- The project name is: Kokkos > > -- Configuring incomplete, errors occurred! > > See also > > > "/vol0004/ra010009/a04199/petsc/arch-arm/externalpackages/git.kokkos/petsc-build/CMakeFiles/CMakeOutput.log".CMake > > Error at cmake/kokkos_compiler_id.cmake:129 (STRING): > > > > * STRING sub-command REPLACE requires at least four arguments.*Call > Stack > > (most recent call first): > > cmake/kokkos_tribits.cmake:174 (INCLUDE) > > CMakeLists.txt:166 (KOKKOS_SETUP_BUILD_ENVIRONMENT) > > > > On Wed, Apr 14, 2021 at 3:34 PM Satish Balay <balay(a)mcs.anl.gov> wrote: > > > > > Additional kokkos cmake arguments can be passed in via > > > --download-kokkos-cmake-arguments=string option. > > > > > > Satish > > > > > > On Wed, 14 Apr 2021, Mark Adams wrote: > > > > > > > Satish, > > > > > > > > For the fujitsu compiler OMP is -Kopenmp. > > > > > > > > Sarat (cc'ed) tells me that he built Kokkos with: > > > > > > > > cmake -DCMAKE_BUILD_TYPE=Release \ > > > > -DCMAKE_INSTALL_PREFIX=${KOKKOS_SRC_DIR}/install \ > > > > -DBUILD_TESTING=OFF \ > > > > -DKokkos_ENABLE_LIBDL=OFF \ > > > > -DKokkos_ENABLE_OPENMP=ON \ > > > > -DKokkos_ENABLE_SERIAL=ON \ > > > > -DKokkos_ENABLE_AGGRESSIVE_VECTORIZATION=ON \ > > > > .. > > > > > > > > How might I make this happen in PETSc? > > > > > > > > Thanks, > > > > Mark > > > > > > > > > > > > On Wed, Apr 14, 2021 at 2:44 PM Satish Balay <balay(a)mcs.anl.gov> > wrote: > > > > > > > > > On Wed, 14 Apr 2021, Mark Adams wrote: > > > > > > > > > > > I have this building now. > > > > > > Do you know anything about OpenMP? > > > > > > I can add --with-openmp > > > > > > That should get Kokkos to be made with OpenMP. > > > > > > Should PETSc deal with the compilers correctly? > > > > > > > > > > Well it tries the following compiler options for openmp. > > > > > > > > > > oflags = ["-fopenmp", # Gnu > > > > > "-qsmp=omp",# IBM XL C/C++ > > > > > "-h omp", # Cray. Must come after XL because XL > > > interprets > > > > > this option as meaning "-soname omp" > > > > > "-mp", # Portland Group > > > > > "-Qopenmp", # Intel windows > > > > > "-openmp", # Intel > > > > > "-xopenmp", # Sun > > > > > "+Oopenmp", # HP > > > > > "/openmp" # Microsoft Visual Studio > > > > > ] > > > > > > > > > > I don't know what the flag for fugaku compiler is. > > > > > > > > > > Satish > > > > > > > > > > > > > > > > Thanks, > > > > > > Mark > > > > > > > > > > > > On Wed, Apr 14, 2021 at 1:45 PM Mark Adams <mfadams(a)lbl.gov> > wrote: > > > > > > > > > > > > > Thanks, > > > > > > > If you feel inspired you could try Kokkos :|| > > > > > > > I am in a parking lot waiting for my daughter but can try this > > > when I > > > > > get > > > > > > > home, > > > > > > > Thanks again, > > > > > > > Mark > > > > > > > > > > > > > > On Wed, Apr 14, 2021 at 1:33 PM Satish Balay < > balay(a)mcs.anl.gov> > > > > > wrote: > > > > > > > > > > > > > >> I think I allocated a single node - and did the build on it. > > > > > > >> > > > > > > >> Now I'm getting an error - don't know what changed.. > > > > > > >> > > > > > > >> login6$ pjsub --interact -L "node=1" -L > "rscunit=rscunit_ft01" -L > > > > > > >> "rscgrp=eap-int" -L "elapse=1:00:00" --sparam "wait-time=600" > > > > > > >> [ERR.] PJM 0059 pjsub rscgrp=eap-int is disabled. > > > > > > >> > > > > > > >> Ok - the following worked.. > > > > > > >> > > > > > > >> login6$ pjsub --interact -L "node=1" -L > "rscunit=rscunit_ft01" -L > > > > > > >> "elapse=1:00:00" --sparam "wait-time=600" > > > > > > >> [INFO] PJM 0000 pjsub Job 6301572 submitted. > > > > > > >> [INFO] PJM 0081 .connected. > > > > > > >> [INFO] PJM 0082 pjsub Interactive job 6301572 started. > > > > > > >> [a04201@j31-3110s petsc]$ > > > > > > >> > > > > > > >> Ok - trying this build now. > > > > > > >> > > > > > > >> [a04201@j31-3110s petsc]$ cat > > > > > > >> ~/petsc.save/arch-arm/lib/petsc/conf/reconfigure-arch-arm.py > > > > > > >> #!/usr/bin/python3 > > > > > > >> if __name__ == '__main__': > > > > > > >> import sys > > > > > > >> import os > > > > > > >> sys.path.insert(0, os.path.abspath('config')) > > > > > > >> import configure > > > > > > >> configure_options = [ > > > > > > >> '--with-blaslapack-lib=-lfjlapack', > > > > > > >> '--with-debugging=0', > > > > > > >> 'CC=mpifcc', > > > > > > >> 'CXX=mpiFCC', > > > > > > >> 'FC=mpifrt', > > > > > > >> 'PETSC_ARCH=arch-arm', > > > > > > >> ] > > > > > > >> configure.petsc_configure(configure_options) > > > > > > >> [a04201@j31-3110s petsc]$ > > > > > > >> ~/petsc.save/arch-arm/lib/petsc/conf/reconfigure-arch-arm.py > > > > > > >> --download-p4est --download-zlib > > > > > > >> <snip> > > > > > > >> p4est: > > > > > > >> Includes: -I/vol0004/ra010009/a04201/petsc/arch-arm/include > > > > > > >> Library: > -Wl,-rpath,/vol0004/ra010009/a04201/petsc/arch-arm/lib > > > > > > >> -L/vol0004/ra010009/a04201/petsc/arch-arm/lib -lp4est -lsc > > > > > > >> > > > > > > >> Ok - this worked for me. > > > > > > >> > > > > > > >> Satish > > > > > > >> > > > > > > >> On Wed, 14 Apr 2021, Mark Adams wrote: > > > > > > >> > > > > > > >> > Do you recall what nodes you use to build on a "compute" > node, > > > to > > > > > avoid > > > > > > >> > cross compilation? > > > > > > >> > > > > > > > >> > On Wed, Apr 14, 2021 at 12:08 PM Satish Balay < > > > balay(a)mcs.anl.gov> > > > > > > >> wrote: > > > > > > >> > > > > > > > >> > > looks like p4est cannot be cross-compiled. > > > > > > >> > > > > > > > > >> > > Satish > > > > > > >> > > > > > > > > >> > > On Wed, 14 Apr 2021, Mark Adams wrote: > > > > > > >> > > > > > > > > >> > > > I get this error with p4est on Fugaku. > > > > > > >> > > > It is a Fortran error. Odd. > > > > > > >> > > > Mark > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >

1 0

Re: [petsc-dev] Error on Fugaku
by Satish Balay 14 Apr '21

14 Apr '21

The following is working for me on the compute node [its completed kokkos (core) build - and is now building kokkos-kernels ~/petsc.save/arch-arm/lib/petsc/conf/reconfigure-arch-arm.py --download-p4est --download-zlib --download-kokkos --download-kokkos-kernels --download-kokkos-commit=origin/develop --download-cmake=https://github.com/Kitware/CMake/releases/download/v3.20.1/cmake-3.20.1.tar.gz -DKokkos_ENABLE_OPENMP=ON give me some issue - so I didn't try any of the additional options.. Satish On Wed, 14 Apr 2021, Mark Adams wrote: > Satish, I get this error. > > I wonder if this syntax is wrong: > > '--download-kokkos-cmake-arguments=-DCMAKE_BUILD_TYPE=Release > -DBUILD_TESTING=OFF -DKokkos_ENABLE_LIBDL=OFF -DKokkos_ENABLE_OPENMP=ON > -DKokkos_ENABLE_SERIAL=ON -DKokkos_ENABLE_AGGRESSIVE_VECTORIZATION=ON', > > > ============================================================================================= > Configuring KOKKOS with cmake; this may take > several minutes > > ============================================================================================= > > Executing: /usr/bin/cmake .. > -DCMAKE_INSTALL_PREFIX=/home/ra010009/a04199/petsc/arch-arm > -DCMAKE_INSTALL_NAME_DIR:STRING="/home/ra010009/a04199/petsc/arch-arm/lib" > -DCMAKE_INSTALL_LIBDIR:STRING="lib" -DCMAKE_VERBOSE_MAKEFILE=1 > -DCMAKE_BUILD_TYPE=Release -DCMAKE_C_COMPILER="mpifcc" > -DMPI_C_COMPILER="mpifcc" -DCMAKE_AR=/usr/bin/ar > -DCMAKE_RANLIB=/usr/bin/ranlib -DCMAKE_C_FLAGS:STRING="-fPIC -Kfast,openmp > -fopenmp" -DCMAKE_C_FLAGS_DEBUG:STRING="-fPIC -Kfast,openmp -fopenmp" > -DCMAKE_C_FLAGS_RELEASE:STRING="-fPIC -Kfast,openmp -fopenmp" > -DCMAKE_CXX_COMPILER="mpiFCC" -DMPI_CXX_COMPILER="mpiFCC" > -DCMAKE_CXX_FLAGS:STRING="-Kfast,openmp -fopenmp -fPIC -fopenmp" > -DCMAKE_CXX_FLAGS_DEBUG:STRING="-Kfast,openmp -fopenmp -fPIC -fopenmp" > -DCMAKE_CXX_FLAGS_RELEASE:STRING="-Kfast,openmp -fopenmp -fPIC -fopenmp" > -DCMAKE_Fortran_COMPILER="mpifrt" -DMPI_Fortran_COMPILER="mpifrt" > -DCMAKE_Fortran_FLAGS:STRING="-fPIC -O -fopenmp" > -DCMAKE_Fortran_FLAGS_DEBUG:STRING="-fPIC -O -fopenmp" > -DCMAKE_Fortran_FLAGS_RELEASE:STRING="-fPIC -O -fopenmp" > -DCMAKE_EXE_LINKER_FLAGS:STRING=" -fopenmp" -DBUILD_SHARED_LIBS:BOOL=ON > -DUSE_XSDK_DEFAULTS=YES -DXSDK_ENABLE_DEBUG=NO > -DCMAKE_INSTALL_RPATH_USE_LINK_PATH:BOOL=ON > -DCMAKE_BUILD_WITH_INSTALL_RPATH:BOOL=ON -DKokkos_ENABLE_MPI=ON > -DKokkos_ENABLE_SERIAL=ON -DKokkos_ENABLE_OPENMP=ON > -DCMAKE_CXX_STANDARD="14" -DCMAKE_BUILD_TYPE=Release -DBUILD_TESTING=OFF > -DKokkos_ENABLE_LIBDL=OFF -DKokkos_ENABLE_OPENMP=ON > -DKokkos_ENABLE_SERIAL=ON -DKokkos_ENABLE_AGGRESSIVE_VECTORIZATION=ON > stdout: > -- Setting default Kokkos CXX standard to 14 > -- The CXX compiler identification is Fujitsu > -- Check for working CXX compiler: /opt/FJSVxtclanga/tcsds-1.2.31/bin/mpiFCC > -- Check for working CXX compiler: > /opt/FJSVxtclanga/tcsds-1.2.31/bin/mpiFCC -- works > -- Detecting CXX compiler ABI info > -- Detecting CXX compiler ABI info - done > -- The project name is: Kokkos > -- Configuring incomplete, errors occurred! > See also > "/vol0004/ra010009/a04199/petsc/arch-arm/externalpackages/git.kokkos/petsc-build/CMakeFiles/CMakeOutput.log". > Error configuring KOKKOS with cmake Could not execute > "['/usr/bin/cmake .. > -DCMAKE_INSTALL_PREFIX=/home/ra010009/a04199/petsc/arch-arm > -DCMAKE_INSTALL_NAME_DIR:STRING="/home/ra010009/a04199/petsc/arch-arm/lib" > -DCMAKE_INSTALL_LIBDIR:STRING="lib" -DCMAKE_VERBOSE_MAKEFILE=1 > -DCMAKE_BUILD_TYPE=Release -DCMAKE_C_COMPILER="mpifcc" > -DMPI_C_COMPILER="mpifcc" -DCMAKE_AR=/usr/bin/ar > -DCMAKE_RANLIB=/usr/bin/ranlib -DCMAKE_C_FLAGS:STRING="-fPIC -Kfast,openmp > -fopenmp" -DCMAKE_C_FLAGS_DEBUG:STRING="-fPIC -Kfast,openmp -fopenmp" > -DCMAKE_C_FLAGS_RELEASE:STRING="-fPIC -Kfast,openmp -fopenmp" > -DCMAKE_CXX_COMPILER="mpiFCC" -DMPI_CXX_COMPILER="mpiFCC" > -DCMAKE_CXX_FLAGS:STRING="-Kfast,openmp -fopenmp -fPIC -fopenmp" > -DCMAKE_CXX_FLAGS_DEBUG:STRING="-Kfast,openmp -fopenmp -fPIC -fopenmp" > -DCMAKE_CXX_FLAGS_RELEASE:STRING="-Kfast,openmp -fopenmp -fPIC -fopenmp" > -DCMAKE_Fortran_COMPILER="mpifrt" -DMPI_Fortran_COMPILER="mpifrt" > -DCMAKE_Fortran_FLAGS:STRING="-fPIC -O -fopenmp" > -DCMAKE_Fortran_FLAGS_DEBUG:STRING="-fPIC -O -fopenmp" > -DCMAKE_Fortran_FLAGS_RELEASE:STRING="-fPIC -O -fopenmp" > -DCMAKE_EXE_LINKER_FLAGS:STRING=" -fopenmp" -DBUILD_SHARED_LIBS:BOOL=ON > -DUSE_XSDK_DEFAULTS=YES -DXSDK_ENABLE_DEBUG=NO > -DCMAKE_INSTALL_RPATH_USE_LINK_PATH:BOOL=ON > -DCMAKE_BUILD_WITH_INSTALL_RPATH:BOOL=ON -DKokkos_ENABLE_MPI=ON > -DKokkos_ENABLE_SERIAL=ON -DKokkos_ENABLE_OPENMP=ON > -DCMAKE_CXX_STANDARD="14" -DCMAKE_BUILD_TYPE=Release -DBUILD_TESTING=OFF > -DKokkos_ENABLE_LIBDL=OFF -DKokkos_ENABLE_OPENMP=ON > -DKokkos_ENABLE_SERIAL=ON -DKokkos_ENABLE_AGGRESSIVE_VECTORIZATION=ON']": > -- Setting default Kokkos CXX standard to 14 > -- The CXX compiler identification is Fujitsu > -- Check for working CXX compiler: /opt/FJSVxtclanga/tcsds-1.2.31/bin/mpiFCC > -- Check for working CXX compiler: > /opt/FJSVxtclanga/tcsds-1.2.31/bin/mpiFCC -- works > -- Detecting CXX compiler ABI info > -- Detecting CXX compiler ABI info - done > -- The project name is: Kokkos > -- Configuring incomplete, errors occurred! > See also > "/vol0004/ra010009/a04199/petsc/arch-arm/externalpackages/git.kokkos/petsc-build/CMakeFiles/CMakeOutput.log".CMake > Error at cmake/kokkos_compiler_id.cmake:129 (STRING): > > * STRING sub-command REPLACE requires at least four arguments.*Call Stack > (most recent call first): > cmake/kokkos_tribits.cmake:174 (INCLUDE) > CMakeLists.txt:166 (KOKKOS_SETUP_BUILD_ENVIRONMENT) > > On Wed, Apr 14, 2021 at 3:34 PM Satish Balay <balay(a)mcs.anl.gov> wrote: > > > Additional kokkos cmake arguments can be passed in via > > --download-kokkos-cmake-arguments=string option. > > > > Satish > > > > On Wed, 14 Apr 2021, Mark Adams wrote: > > > > > Satish, > > > > > > For the fujitsu compiler OMP is -Kopenmp. > > > > > > Sarat (cc'ed) tells me that he built Kokkos with: > > > > > > cmake -DCMAKE_BUILD_TYPE=Release \ > > > -DCMAKE_INSTALL_PREFIX=${KOKKOS_SRC_DIR}/install \ > > > -DBUILD_TESTING=OFF \ > > > -DKokkos_ENABLE_LIBDL=OFF \ > > > -DKokkos_ENABLE_OPENMP=ON \ > > > -DKokkos_ENABLE_SERIAL=ON \ > > > -DKokkos_ENABLE_AGGRESSIVE_VECTORIZATION=ON \ > > > .. > > > > > > How might I make this happen in PETSc? > > > > > > Thanks, > > > Mark > > > > > > > > > On Wed, Apr 14, 2021 at 2:44 PM Satish Balay <balay(a)mcs.anl.gov> wrote: > > > > > > > On Wed, 14 Apr 2021, Mark Adams wrote: > > > > > > > > > I have this building now. > > > > > Do you know anything about OpenMP? > > > > > I can add --with-openmp > > > > > That should get Kokkos to be made with OpenMP. > > > > > Should PETSc deal with the compilers correctly? > > > > > > > > Well it tries the following compiler options for openmp. > > > > > > > > oflags = ["-fopenmp", # Gnu > > > > "-qsmp=omp",# IBM XL C/C++ > > > > "-h omp", # Cray. Must come after XL because XL > > interprets > > > > this option as meaning "-soname omp" > > > > "-mp", # Portland Group > > > > "-Qopenmp", # Intel windows > > > > "-openmp", # Intel > > > > "-xopenmp", # Sun > > > > "+Oopenmp", # HP > > > > "/openmp" # Microsoft Visual Studio > > > > ] > > > > > > > > I don't know what the flag for fugaku compiler is. > > > > > > > > Satish > > > > > > > > > > > > > Thanks, > > > > > Mark > > > > > > > > > > On Wed, Apr 14, 2021 at 1:45 PM Mark Adams <mfadams(a)lbl.gov> wrote: > > > > > > > > > > > Thanks, > > > > > > If you feel inspired you could try Kokkos :|| > > > > > > I am in a parking lot waiting for my daughter but can try this > > when I > > > > get > > > > > > home, > > > > > > Thanks again, > > > > > > Mark > > > > > > > > > > > > On Wed, Apr 14, 2021 at 1:33 PM Satish Balay <balay(a)mcs.anl.gov> > > > > wrote: > > > > > > > > > > > >> I think I allocated a single node - and did the build on it. > > > > > >> > > > > > >> Now I'm getting an error - don't know what changed.. > > > > > >> > > > > > >> login6$ pjsub --interact -L "node=1" -L "rscunit=rscunit_ft01" -L > > > > > >> "rscgrp=eap-int" -L "elapse=1:00:00" --sparam "wait-time=600" > > > > > >> [ERR.] PJM 0059 pjsub rscgrp=eap-int is disabled. > > > > > >> > > > > > >> Ok - the following worked.. > > > > > >> > > > > > >> login6$ pjsub --interact -L "node=1" -L "rscunit=rscunit_ft01" -L > > > > > >> "elapse=1:00:00" --sparam "wait-time=600" > > > > > >> [INFO] PJM 0000 pjsub Job 6301572 submitted. > > > > > >> [INFO] PJM 0081 .connected. > > > > > >> [INFO] PJM 0082 pjsub Interactive job 6301572 started. > > > > > >> [a04201@j31-3110s petsc]$ > > > > > >> > > > > > >> Ok - trying this build now. > > > > > >> > > > > > >> [a04201@j31-3110s petsc]$ cat > > > > > >> ~/petsc.save/arch-arm/lib/petsc/conf/reconfigure-arch-arm.py > > > > > >> #!/usr/bin/python3 > > > > > >> if __name__ == '__main__': > > > > > >> import sys > > > > > >> import os > > > > > >> sys.path.insert(0, os.path.abspath('config')) > > > > > >> import configure > > > > > >> configure_options = [ > > > > > >> '--with-blaslapack-lib=-lfjlapack', > > > > > >> '--with-debugging=0', > > > > > >> 'CC=mpifcc', > > > > > >> 'CXX=mpiFCC', > > > > > >> 'FC=mpifrt', > > > > > >> 'PETSC_ARCH=arch-arm', > > > > > >> ] > > > > > >> configure.petsc_configure(configure_options) > > > > > >> [a04201@j31-3110s petsc]$ > > > > > >> ~/petsc.save/arch-arm/lib/petsc/conf/reconfigure-arch-arm.py > > > > > >> --download-p4est --download-zlib > > > > > >> <snip> > > > > > >> p4est: > > > > > >> Includes: -I/vol0004/ra010009/a04201/petsc/arch-arm/include > > > > > >> Library: -Wl,-rpath,/vol0004/ra010009/a04201/petsc/arch-arm/lib > > > > > >> -L/vol0004/ra010009/a04201/petsc/arch-arm/lib -lp4est -lsc > > > > > >> > > > > > >> Ok - this worked for me. > > > > > >> > > > > > >> Satish > > > > > >> > > > > > >> On Wed, 14 Apr 2021, Mark Adams wrote: > > > > > >> > > > > > >> > Do you recall what nodes you use to build on a "compute" node, > > to > > > > avoid > > > > > >> > cross compilation? > > > > > >> > > > > > > >> > On Wed, Apr 14, 2021 at 12:08 PM Satish Balay < > > balay(a)mcs.anl.gov> > > > > > >> wrote: > > > > > >> > > > > > > >> > > looks like p4est cannot be cross-compiled. > > > > > >> > > > > > > > >> > > Satish > > > > > >> > > > > > > > >> > > On Wed, 14 Apr 2021, Mark Adams wrote: > > > > > >> > > > > > > > >> > > > I get this error with p4est on Fugaku. > > > > > >> > > > It is a Fortran error. Odd. > > > > > >> > > > Mark > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > > > > > > > > > > > > > > > > > >

1 0