June 2021 - petsc-users - mailman3.cels.anl.gov

ex5k.kokkos compile error
by Mark Adams 03 Jun '21

03 Jun '21

I am getting this error: 09:22 adams/landau-mass-opt= /gpfs/alpine/csc314/scratch/adams/petsc/src/mat/tutorials$ make PETSC_DIR=/gpfs/alpine/csc314/scratch/adams/petsc PETSC_ARCH=arch-summit-opt-gnu-kokkos-notpl-cuda10 ex5k.kokkos mpicxx -fPIC -g -DLANDAU_DIM=2 -DLANDAU_MAX_SPECIES=10 -DPETSC_HAVE_CUDA_ATOMIC -DLANDAU_MAX_Q=4 -O0 -fPIC -g -DLANDAU_DIM=2 -DLANDAU_MAX_SPECIES=10 -DPETSC_HAVE_CUDA_ATOMIC -DLANDAU_MAX_Q=4 -O0 -fPIC -I/gpfs/alpine/csc314/scratch/adams/petsc/include -I/gpfs/alpine/csc314/scratch/adams/petsc/arch-summit-opt-gnu-kokkos-notpl-cuda10/include -I/sw/summit/cuda/10.1.243/include ex5k.kokkos.cxx -Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-summit-opt-gnu-kokkos-notpl-cuda10/lib -L/gpfs/alpine/csc314/scratch/adams/petsc/arch-summit-opt-gnu-kokkos-notpl-cuda10/lib -Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-summit-opt-gnu-kokkos-notpl-cuda10/lib -L/gpfs/alpine/csc314/scratch/adams/petsc/arch-summit-opt-gnu-kokkos-notpl-cuda10/lib -L/autofs/nccs-svm1_sw/summit/.swci/1-compute/opt/spack/20180914/linux-rhel7-ppc64le/gcc-6.4.0/netlib-lapack-3.8.0-wcabdyqhdi5rooxbkqa6x5d7hxyxwdkm/lib64 -Wl,-rpath,/sw/summit/cuda/10.1.243/lib64 -L/sw/summit/cuda/10.1.243/lib64 -lpetsc -lkokkoskernels -lkokkoscontainers -lkokkoscore -lp4est -lsc -lblas -llapack -ltriangle -lm -lz -lcudart -lcufft -lcublas -lcusparse -lcusolver -lcurand -lstdc++ -ldl -o ex5k.kokkos In file included from /gpfs/alpine/csc314/scratch/adams/petsc/arch-summit-opt-gnu-kokkos-notpl-cuda10/include/KokkosCore_Config_SetupBackend.hpp:47:0, from /gpfs/alpine/csc314/scratch/adams/petsc/arch-summit-opt-gnu-kokkos-notpl-cuda10/include/Kokkos_Macros.hpp:109, from /gpfs/alpine/csc314/scratch/adams/petsc/arch-summit-opt-gnu-kokkos-notpl-cuda10/include/Kokkos_Core_fwd.hpp:52, from /gpfs/alpine/csc314/scratch/adams/petsc/arch-summit-opt-gnu-kokkos-notpl-cuda10/include/Kokkos_Core.hpp:51, from ex5k.kokkos.cxx:10: /gpfs/alpine/csc314/scratch/adams/petsc/arch-summit-opt-gnu-kokkos-notpl-cuda10/include/setup/Kokkos_Setup_Cuda.hpp:54:2: error: #error "KOKKOS_ENABLE_CUDA defined but the compiler is not defining the __CUDACC__ macro as expected" #error \ ^~~~~ In file included from /sw/summit/cuda/10.1.243/include/crt/common_functions.h:267:0, from /sw/summit/cuda/10.1.243/include/cuda_runtime.h:115, from /gpfs/alpine/csc314/scratch/adams/petsc/arch-summit-opt-gnu-kokkos-notpl-cuda10/include/setup/Kokkos_Setup_Cuda.hpp:68, from /gpfs/alpine/csc314/scratch/adams/petsc/arch-summit-opt-gnu-kokkos-notpl-cuda10/include/KokkosCore_Config_SetupBackend.hpp:47, from /gpfs/alpine/csc314/scratch/adams/petsc/arch-summit-opt-gnu-kokkos-notpl-cuda10/include/Kokkos_Macros.hpp:109, from /gpfs/alpine/csc314/scratch/adams/petsc/arch-summit-opt-gnu-kokkos-notpl-cuda10/include/Kokkos_Core_fwd.hpp:52, from /gpfs/alpine/csc314/scratch/adams/petsc/arch-summit-opt-gnu-kokkos-notpl-cuda10/include/Kokkos_Core.hpp:51, from ex5k.kokkos.cxx:10: /sw/summit/cuda/10.1.243/include/crt/math_functions.h:8944:72: error: 'int std::isnan(double)' conflicts with a previous declaration __DEVICE_FUNCTIONS_DECL__ __cudart_builtin__ int isnan(double x) throw(); ^ Here is my config file: #!/usr/bin/env python # -lnvToolsExt if __name__ == '__main__': import sys import os sys.path.insert(0, os.path.abspath('config')) import configure configure_options = [ '--with-fc=0', '--CFLAGS=-fPIC -g -DLANDAU_DIM=2 -DLANDAU_MAX_SPECIES=10 -DPETSC_HAVE_CUDA_ATOMIC -DLANDAU_MAX_Q=4', '--CXXFLAGS=-fPIC -g -DLANDAU_DIM=2 -DLANDAU_MAX_SPECIES=10 -DPETSC_HAVE_CUDA_ATOMIC -DLANDAU_MAX_Q=4', '--COPTFLAGS=-O0', '--CXXOPTFLAGS=-O0', '--CUDAOPTFLAGS=-O0', # 'LIBS= -L'+os.environ['CUDAPATH']+'/lib64 -lnvToolsExt', '--CUDAFLAGS=-Xcompiler -rdynamic -lineinfo -DLANDAU_DIM=2 -DLANDAU_MAX_SPECIES=10 -DPETSC_HAVE_CUDA_ATOMIC -DLANDAU_MAX_Q=4', '--with-ssl=0', '--with-batch=0', '--with-cxx=mpicxx', '--with-mpiexec=jsrun -g1 ', '--with-openmp=0', # '--with-threadsafety=1', # '--with-log=0', '--with-cuda=1', '--with-cuda-gencodearch=70', '--with-cudac=nvcc', '--download-p4est=1', '--download-zlib', # '--download-hdf5=1', # '--download-metis', # '--download-superlu_dist', # '--download-superlu_dist-commit=HEAD', # '--download-hypre-configure-arguments=HYPRE_CUDA_SM=70', #'--with-hwloc=0', # '--download-parmetis', #'--download-hypre', '--download-triangle', #'--download-amgx', #'--download-fblaslapack', '--with-blaslapack-lib=-L' + os.environ['OLCF_NETLIB_LAPACK_ROOT'] + '/lib64 -lblas -llapack', #'--download-openblas', '--with-cc=mpicc', #'--with-fc=mpif90', '--with-shared-libraries=1', # '--known-mpi-shared-libraries=1', '--with-x=0', '--with-64-bit-indices=0', '--with-debugging=1', '--download-kokkos', '--with-ctable=0', '--with-make-np=8', '--download-kokkos-kernels', # '--with-kokkos-cuda-arch=VOLTA70', '--with-kokkos-kernels-tpl=0', '--download-kokkos-cmake-arguments=-DKokkos_ENABLE_AGGRESSIVE_VECTORIZATION=ON -DKokkos_ARCH_POWER9=ON', # '--download-kokkos-commit=origin/develop', # '--download-kokkos-kernels-commit=origin/develop', 'PETSC_ARCH=arch-summit-opt-gnu-kokkos-notpl-cuda10', ] configure.petsc_configure(configure_options) ~ ~ ~ ~ ~ ~ ~

1 0

Rather different matrix product results on multiple processes
by Peder Jørgensgaard Olesen 03 Jun '21

03 Jun '21

Hello, When computing a matrix product of the type R = D.DT using MatMatTransposeMult() I find I get rather different results depending on the number of processes. In one example using a data set that is small compared to the application I get Frobenius norms |R| = 1.047e3 on a single process, 1.0363e3 on a single HPC node (40 cores), and 9.7307e2 on two nodes. I have ascertained that the single process result is indeed the correct one (i.e., eigenvectors of R form a proper basis for the columns of D), so naturally I'd love to be able to reproduce this result across different parallel setups. How might I achieve this? I'm attaching MWE code and the data set used for the example. Thanks in advance! Best Regards Peder Jørgensgaard Olesen PhD Student, Turbulence Research Lab Dept. of Mechanical Engineering Technical University of Denmark Niels Koppels Allé Bygning 403, Rum 105 DK-2800 Kgs. Lyngby

5 7

SLEPc: non-real singular vectors from SVD of real matrix
by Peder Jørgensgaard Olesen 02 Jun '21

02 Jun '21

Hello I've noticed that doing a singular value decomposition of a real matrix appears to result in non-real singular vectors. This should not be the case - singular vectors of a real matrix must be real-valued. In the example attached I read a matrix from a binary file (also attached), perform the SVD, and write singular vectors to an HDF5 file which I subsequently inspect using h5dump, revealing non-zero imaginary parts of vector elements as highlighted below: [pjool@svol mwes]$ h5dump -d U0_sample_svd -c "5,2" sample_svecs.h5 HDF5 "sample_svecs.h5" { DATASET "U0_sample_svd" { DATATYPE H5T_IEEE_F64LE DATASPACE SIMPLE { ( 882, 2 ) / ( 882, 2 ) } SUBSET { START ( 0, 0 ); STRIDE ( 1, 1 ); COUNT ( 5, 2 ); BLOCK ( 1, 1 ); DATA { (0,0): 0.0226108, 0.0299595, (1,0): 0.035414, 0.0469237, (2,0): 0.0276317, 0.0366122, (3,0): 0.0145344, 0.0192581, (4,0): 0.0110376, 0.0146249 } } ATTRIBUTE "complex" { DATATYPE H5T_STD_I32LE DATASPACE SCALAR DATA { (0): 1 } } } } I also extract the imaginary part of the input matrix and print its norm to ensure that the matrix is indeed real. I'm running v3.14, but I don't believe that alone should cause the issue, since it what I'm trying to do appears like a rather common and basic task. What might be the reason behind this behavior, and what can be done to resolve it? Med venlig hilsen / Best Regards Peder Jørgensgaard Olesen PhD Student, Turbulence Research Lab Dept. of Mechanical Engineering Technical University of Denmark Niels Koppels Allé Bygning 403, Rum 105 DK-2800 Kgs. Lyngby

3 4