September 2021 - petsc-users - mailman3.cels.anl.gov

New error on Summit
by Mark Adams 23 Sep '21

23 Sep '21

This was working before but now I get this (strange) error: Run Build Command(s):/usr/bin/gmake -f Makefile cmTC_429f1/fast && /usr/bin/gmake -f CMakeFiles/cmTC_429f1.dir/build.make CMakeFiles/cmTC_429f1.dir/build gmake[1]: Entering directory '/gpfs/alpine/csc314/scratch/adams/petsc/arch-summit-opt-gnu-kokkos-cuda/externalpackages/git.kokkos/petsc-build/CMakeFiles/CMakeTmp' Building CXX object CMakeFiles/cmTC_429f1.dir/testCXXCompiler.cxx.o /gpfs/alpine/csc314/scratch/adams/petsc/arch-summit-opt-gnu-kokkos-cuda/externalpackages/git.kokkos/bin/nvcc_wrapper -fPIC -g -DLANDAU_DIM=2 -DLANDAU_MAX_SPECIES=10 -DLANDAU_MAX_Q=4 -O -+ -qPIC -std=gnu++14 -o CMakeFiles/cmTC_429f1.dir/testCXXCompiler.cxx.o -c /gpfs/alpine/csc314/scratch/adams/petsc/arch-summit-opt-gnu-kokkos-cuda/externalpackages/git.kokkos/petsc-build/CMakeFiles/CMakeTmp/testCXXCompiler.cxx nvcc_wrapper has been given GNU extension standard flag -std=gnu++14 - reverting flag to -std=c++14 nvcc warning : The 'compute_35', 'compute_37', 'compute_50', 'sm_35', 'sm_37' and 'sm_50' architectures are deprecated, and may be removed in a future release (Use -Wno-deprecated-gpu-targets to suppress warning). g++: error: unrecognized command line option �~@~X-qPIC�~@~Y; did you mean �~@~X-fPIC�~@~Y? gmake[1]: *** [CMakeFiles/cmTC_429f1.dir/build.make:78: CMakeFiles/cmTC_429f1.dir/testCXXCompiler.cxx.o] Error 1 gmake[1]: Leaving directory '/gpfs/alpine/csc314/scratch/adams/petsc/arch-summit-opt-gnu-kokkos-cuda/externalpackages/git.kokkos/petsc-build/CMakeFiles/CMakeTmp' gmake: *** [Makefile:127: cmTC_429f1/fast] Error 2

2 2

Is this a bug? MatMultAdd_SeqBAIJ_11
by Daniel Stone 21 Sep '21

21 Sep '21

Hello, If we look at lines 2330-2331 in file baij2.c, it looks like there are some mistakes in assigning the `sum..` variables to the z array, causing the function MatMultAdd_SeqBAIJ_11() to not produce the correct answer. I don't have a good example program to demonstrate this yet - it's currently causing problems in a dev branch of pflotan_ogs that can produce blocksize 11 matrices. When in parallel, a standard matrix-vector multiplication calls MatMultAdd for the off-proc contributions, and the result is wrong when this is redirected to MatMultAdd_SeqBAIJ_11. Seems to be the root cause of several solvers failing such as fgmres. Can anyone confirm that these two lines seem incorrect? Thanks, Daniel

4 4

Uses of VecGetArrayF90() and VecGetArrayReadF90() in Recent versions of GNU Fortran.
by 李昊霖 20 Sep '21

20 Sep '21

Dear all, I used PETSc in my full Fortran codes and it worked well when my codes were compiled by GNU Fortran (GCC) 4.8.4. But for some reasons, I had to update the GNU Fortran (GCC) to version 10.0.1. Then I reinstalled the MPICH and PETSc with the newer complier and compiled my codes successfully. However, I got the following error massage: Index '1' of dimension 1 of array 'xx' above upper bound of 0. where xx is the Fortran pointer obtained by calling VecGetArrayF90(vec,xx,ierr). The vector was built successfully, but it seemed that the Fortran pointer xx was not built. I got the same error massage when using VecGetArrayReadF90(). So, are the VecGetArrayF90() and VecGetArrayReadF90() not compatible with the recent versions of GNU Fortran? Or is there any other way to access the vectors? For solving a linear problem Ab=x, I use VecGetArrayF90() to get the Fortran pointer to update b and use VecGetArrayReadF90() to get the values of x. After some tests, I found that VecGetArrayF90() could be replaced by VecSetValues(), but VecGetValues() used to replace VecGetArrayReadF90() could not be run with multiple threads. I look forward to your reply and thank you for any suggestions. Best regards, Haolin Li

2 1

Re: [petsc-users] DMView and DMLoad
by Lawrence Mitchell 20 Sep '21

20 Sep '21

Dear Sergio, (Added petsc-users back to cc), > On 20 Sep 2021, at 14:08, sergio.bengoechea(a)ovgu.de wrote: > > Dear Lawrence, > > thanks for the HDF5 saving and loading example. > > In the documentation you sent (https://petsc.org/main/docs/manual/dmplex/#saving-and-loading-data-with-hdf5) the link to a more comprehensive example (DMPlex Tutorial ex12) is not working. Hmm, I'm not in charge of how the documentation gets built. @Patrick, if I look here: https://petsc.org/main/src/dm/impls/plex/tutorials/, I don't see ex12.c (even though it is here https://gitlab.com/petsc/petsc/-/tree/main/src/dm/impls/plex/tutorials) The relevant link in the docs goes to https://petsc.org/main/docs/src/dm/impls/plex/tutorials/ex12.c.html (which is 404), I suppose it should be main/src/... (not main/docs/src...) > I am afraid I would need to see the whole example to make our case work. If I could have access the completed source code of that example would be of a great help. You can find the relevant example in the PETSc source tree https://gitlab.com/petsc/petsc/-/blob/main/src/dm/impls/plex/tutorials/ex12… Lawrence

3 2

How to get specific ordering
by Carlos Velazquez 20 Sep '21

20 Sep '21

Hi there. I have a question whether I can get the DAG values in DMPlex in a different order. I am working on a code that uses the same node ordering to form the elements found in the mesh file. Example: If I use the file "doublet-tet.msh" that is inside the DMPlex mesh files folder, this file tells us that the elements are formed as follows: 1 - 2 4 3 1 2 - 2 3 4 5 Element 1 by nodes 2, 4, 3, 1 Element 2 by nodes 2, 3, 4, 5 So what I'm doing to get this through DMPlex is getting the transitive closure with DMPlexGetTransitiveClosure and getting the points from level 0, which is the node level, but the ordering is different. Example: With DMPlexGetTransitiveClosure I obtain that the elements are formed as follows: 0 - 5 3 4 2 1 - 4 3 5 6 Element 1 by nodes 5, 3, 4, 2 Element 2 by nodes 4, 3, 5, 6 But comparing this ordering with the previous one in the coordinate matrix I can see that the order is not equivalent. I would like to know if there is a way to modify the ordering of the graph to obtain the same ordering that is in the mesh file for the nodes that make up the elements or even if there is some way to configure it for a specific desired order.

2 1

Spock link error
by Mark Adams 19 Sep '21

19 Sep '21

I am getting to see this error. It seems to be suggesting that I turn --no-allow-shlib-undefined off. Any ideas? Thanks, Mark 09:09 main= /gpfs/alpine/csc314/scratch/adams/petsc$ make PETSC_DIR=/gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-opt-cray-new PETSC_ARCH="" check Running check examples to verify correct installation Using PETSC_DIR=/gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-opt-cray-new and PETSC_ARCH= gmake[3]: [/gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-opt-cray-new/lib/petsc/conf/rules:301: ex19.PETSc] Error 2 (ignored) *******************Error detected during compile or link!******************* See http://www.mcs.anl.gov/petsc/documentation/faq.html /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials ex19 ********************************************************************************* cc -fPIC -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -fstack-protector -Qunused-arguments -fvisibility=hidden -g -O2 -fPIC -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -fstack-protector -Qunused-arguments -fvisibility=hidden -g -O2 -I/gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-opt-cray-new/include -I/opt/rocm-4.2.0/include ex19.c -Wl,-rpath,/gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-opt-cray-new/lib -L/gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-opt-cray-new/lib -Wl,-rpath,/gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-opt-cray-new/lib -L/gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-opt-cray-new/lib -Wl,-rpath,/opt/rocm-4.2.0/lib -L/opt/rocm-4.2.0/lib -Wl,-rpath,/opt/cray/pe/gcc/8.1.0/snos/lib64 -L/opt/cray/pe/gcc/8.1.0/snos/lib64 -Wl,-rpath,/opt/cray/pe/libsci/ 21.06.1.1/CRAY/9.0/x86_64/lib -L/opt/cray/pe/libsci/ 21.06.1.1/CRAY/9.0/x86_64/lib -Wl,-rpath,/opt/cray/pe/mpich/8.1.7/ofi/cray/10.0/lib -L/opt/cray/pe/mpich/8.1.7/ofi/cray/10.0/lib -Wl,-rpath,/opt/cray/pe/mpich/default/gtl/lib -L/opt/cray/pe/mpich/default/gtl/lib -Wl,-rpath,/opt/cray/pe/dsmml/0.1.5/dsmml/lib -L/opt/cray/pe/dsmml/0.1.5/dsmml/lib -Wl,-rpath,/opt/cray/pe/pmi/6.0.12/lib -L/opt/cray/pe/pmi/6.0.12/lib -Wl,-rpath,/opt/cray/pe/cce/12.0.1/cce/x86_64/lib -L/opt/cray/pe/cce/12.0.1/cce/x86_64/lib -Wl,-rpath,/opt/cray/xpmem/2.2.40-2.1_2.44__g3cf3325.shasta/lib64 -L/opt/cray/xpmem/2.2.40-2.1_2.44__g3cf3325.shasta/lib64 -Wl,-rpath,/opt/cray/pe/cce/12.0.1/cce-clang/x86_64/lib/clang/12.0.0/lib/linux -L/opt/cray/pe/cce/12.0.1/cce-clang/x86_64/lib/clang/12.0.0/lib/linux -Wl,-rpath,/opt/cray/pe/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0 -L/opt/cray/pe/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0 -Wl,-rpath,/opt/cray/pe/cce/12.0.1/binutils/x86_64/x86_64-unknown-linux-gnu/lib -L/opt/cray/pe/cce/12.0.1/binutils/x86_64/x86_64-unknown-linux-gnu/lib -Wl,-rpath,/opt/cray/pe/cce/12.0.1/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib -L/opt/cray/pe/cce/12.0.1/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib -lpetsc -lparmetis -lmetis -lhipsparse -lhipblas -lrocsparse -lrocsolver -lrocblas -lrocrand -lamdhip64 -lstdc++ -ldl -lmpifort_cray -lmpi_cray -lmpi_gtl_hsa -ldsmml -lpmi -lxpmem -lpgas-shmem -lquadmath -lcrayacc_amdgpu -lopenacc -lmodules -lfi -lcraymath -lf -lu -lcsup -lgfortran -lpthread -lgcc_eh -lm -lclang_rt.craypgo-x86_64 -lclang_rt.builtins-x86_64 -lquadmath -lstdc++ -ldl -o ex19 *ld.lld: error: /gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-opt-cray-new/lib/libpetsc.so: undefined reference to .omp_offloading.img_start.cray_amdgcn-amd-amdhsa [--no-allow-shlib-undefined]ld.lld: error: /gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-opt-cray-new/lib/libpetsc.so: undefined reference to .omp_offloading.img_size.cray_amdgcn-amd-amdhsa [--no-allow-shlib-undefined]ld.lld: error: /gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-opt-cray-new/lib/libpetsc.so: undefined reference to .omp_offloading.img_cache.cray_amdgcn-amd-amdhsa [--no-allow-shlib-undefined]* clang-12: error: linker command failed with exit code 1 (use -v to see invocation) gmake[4]: *** [<builtin>: ex19] Error 1 *******************Error detected during compile or link!******************* See http://www.mcs.anl.gov/petsc/documentation/faq.html /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials ex5f ********************************************************* ftn -fPIC -g -O2 -fPIC -g -O2 -I/gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-opt-cray-new/include -I/opt/rocm-4.2.0/include ex5f.F90 -Wl,-rpath,/gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-opt-cray-new/lib -L/gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-opt-cray-new/lib -Wl,-rpath,/gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-opt-cray-new/lib -L/gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-opt-cray-new/lib -Wl,-rpath,/opt/rocm-4.2.0/lib -L/opt/rocm-4.2.0/lib -Wl,-rpath,/opt/cray/pe/gcc/8.1.0/snos/lib64 -L/opt/cray/pe/gcc/8.1.0/snos/lib64 -Wl,-rpath,/opt/cray/pe/libsci/ 21.06.1.1/CRAY/9.0/x86_64/lib -L/opt/cray/pe/libsci/ 21.06.1.1/CRAY/9.0/x86_64/lib -Wl,-rpath,/opt/cray/pe/mpich/8.1.7/ofi/cray/10.0/lib -L/opt/cray/pe/mpich/8.1.7/ofi/cray/10.0/lib -Wl,-rpath,/opt/cray/pe/mpich/default/gtl/lib -L/opt/cray/pe/mpich/default/gtl/lib -Wl,-rpath,/opt/cray/pe/dsmml/0.1.5/dsmml/lib -L/opt/cray/pe/dsmml/0.1.5/dsmml/lib -Wl,-rpath,/opt/cray/pe/pmi/6.0.12/lib -L/opt/cray/pe/pmi/6.0.12/lib -Wl,-rpath,/opt/cray/pe/cce/12.0.1/cce/x86_64/lib -L/opt/cray/pe/cce/12.0.1/cce/x86_64/lib -Wl,-rpath,/opt/cray/xpmem/2.2.40-2.1_2.44__g3cf3325.shasta/lib64 -L/opt/cray/xpmem/2.2.40-2.1_2.44__g3cf3325.shasta/lib64 -Wl,-rpath,/opt/cray/pe/cce/12.0.1/cce-clang/x86_64/lib/clang/12.0.0/lib/linux -L/opt/cray/pe/cce/12.0.1/cce-clang/x86_64/lib/clang/12.0.0/lib/linux -Wl,-rpath,/opt/cray/pe/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0 -L/opt/cray/pe/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0 -Wl,-rpath,/opt/cray/pe/cce/12.0.1/binutils/x86_64/x86_64-unknown-linux-gnu/lib -L/opt/cray/pe/cce/12.0.1/binutils/x86_64/x86_64-unknown-linux-gnu/lib -Wl,-rpath,/opt/cray/pe/cce/12.0.1/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib -L/opt/cray/pe/cce/12.0.1/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib -lpetsc -lparmetis -lmetis -lhipsparse -lhipblas -lrocsparse -lrocsolver -lrocblas -lrocrand -lamdhip64 -lstdc++ -ldl -lmpifort_cray -lmpi_cray -lmpi_gtl_hsa -ldsmml -lpmi -lxpmem -lpgas-shmem -lquadmath -lcrayacc_amdgpu -lopenacc -lmodules -lfi -lcraymath -lf -lu -lcsup -lgfortran -lpthread -lgcc_eh -lm -lclang_rt.craypgo-x86_64 -lclang_rt.builtins-x86_64 -lquadmath -lstdc++ -ldl -o ex5f /opt/cray/pe/cce/12.0.1/binutils/x86_64/x86_64-pc-linux-gnu/bin/ld: warning: alignment 128 of symbol `$host_init$$runtime_init_for_iso_c_binding$iso_c_binding_' in /opt/cray/pe/cce/12.0.1/cce/x86_64/lib/libmodules.so is smaller than 256 in /tmp/pe_39617/ex5f_1.o /opt/cray/pe/cce/12.0.1/binutils/x86_64/x86_64-pc-linux-gnu/bin/ld: warning: alignment 64 of symbol `$data_init$iso_c_binding_' in /opt/cray/pe/cce/12.0.1/cce/x86_64/lib/libmodules.so is smaller than 256 in /tmp/pe_39617/ex5f_1.o Fortran example src/snes/tutorials/ex5f run successfully with 1 MPI process Completed test examples 09:12 main= /gpfs/alpine/csc314/scratch/adams/petsc$ module list Currently Loaded Modules: 1) craype-x86-rome 4) perftools-base/21.05.0 7) cray-pmi-lib/6.0.12 10) cray-dsmml/0.1.5 13) PrgEnv-cray/8.1.0 16) rocm/4.2.0 19) autoconf/2.69 2) libfabric/1.11.0.4.75 5) xpmem/2.2.40-2.1_2.44__g3cf3325.shasta 8) cce/12.0.1 11) cray-mpich/8.1.7 14) DefApps/default 17) emacs/27.2 20) automake/1.16.3 3) craype-network-ofi 6) cray-pmi/6.0.12 9) craype/2.7.8 12) cray-libsci/21.06.1.1 15) craype-accel-amd-gfx908 18) zlib/1.2.11 21) libtool/2.4.6

2 2

Solving two successive linear systems
by Samuel Estes 17 Sep '21

17 Sep '21

Hi, I have two related questions about the best way to use the KSP solver: First, I have an adaptive FEM code which solves the same linear system at each iteration until the grid is refined at which point, obviously, the size of the linear system changes. Currently, I just call: KSPSetOperators(ksp,A,A); KSPSetFromOptions(); KSPSolve(ksp,b,x); In a separate part of the code, I re-create the matrix and vectors and call KSPReset(ksp); whenever the grid is refined. Is this an optimal way to do things? In particular, does KSPSetFromOptions need to be called before each solve or can I just call it once somewhere else and then be done with it. Does it need to be called after each call to KSPReset? There is a section in the PETSc Manual about solving successive linear systems but it is rather terse so I'm just trying to get a sense of how to optimally code this. Second, one model in this code actually successively solves two linear systems of different sizes (one system is n x n and the other is 3*n x 3*n). I solve this by creating two matrices, two right hand sides, and two solution vectors for each system. I currently just use one KSP object which I reset after each use since the linear system changes size each time. Would it be more efficient to simply allocate a second ksp solver object so that I don't have to call KSPReset every time? I'm not sure how much memory a ksp object requires or how much computation I would save by using a second solver. Any ideas here? This part of the code is also adaptive. Thanks in advance for the help. I hope my questions are clear. If not, I'm happy to clarify. Sam

2 2

Redirecting petsc output
by Fackler, Philip 15 Sep '21

15 Sep '21

Is there a way to customize how petsc writes information? Instead of writing to stdout (for example: 0 TS dt 0.1 time 0.), what if we want to log that message to a file other output from Xolotl? I'm assuming there are multiple ways of getting this result. What's common practice with petsc folks? Thanks, Philip Fackler Research Software Engineer, Application Engineering Group Advanced Computing Systems Research Section Computer Science and Mathematics Division Oak Ridge National Laboratory

2 3

MatZeroRows and full assembly
by Anthony Paul Haas 13 Sep '21

13 Sep '21

Hello, Is it allowed after a MatZeroRows to insert more values in the row that was just zeroed with MatSetValues and then perform another full assembly of the matrix? Thanks, Anthony

3 3

Configure slowness
by Hamid 13 Sep '21

13 Sep '21

3 2