April 2021 - petsc-users - mailman3.cels.anl.gov

How to add a source term for PETSCFV ?
by Thibault Bridel-Bertomeu 14 Apr '21

14 Apr '21

Dear all, I have been studying ex11.c from ts/tutorials to understand how to solve an hyperbolic system of equations using PETSCFV. I first worked on the Euler equations for inviscid fluids and based on what ex11.c presents, I was able to add the right PETSc instructions in an already existing in-house code with different gas models to solve the problems in parallel (MPI) and with the AMR capabilities offered by P4EST. Now my goal is to move to Navier-Stokes equations. Theoretically the system is not completely hyperbolic and can be seen as one with an hyperbolic part (identical to the Euler equations) and a parabolic part coming from the RHS diffusion terms. I have been looking into the manual and also the sources of PETSc around the DM, DMPlex, DS and FV classes but I could not find anything that speaks to me as "adding a RHS to an hyperbolic system of equations" or "adding a source term to an hyperbolic system of equations". What's more, that source term depends on the derivatives of the context variables ... I wanted to know if anyone maybe had a suggestion regarding this issue ? Thank you very much in advance, Thibault Bridel-Bertomeu — Eng, MSc, PhD Research Engineer CEA/CESTA 33114 LE BARP Tel.: (+33)557046924 Mob.: (+33)611025322 Mail: thibault.bridelbertomeu(a)gmail.com

4 7

GAMG preconditioning
by Milan Pelletier 12 Apr '21

12 Apr '21

Dear all, I am currently trying to use PETSc with CG solver and GAMG preconditioner. I have started with the following set of parameters: -ksp_type cg -pc_type gamg -pc_gamg_agg_nsmooths 1 -pc_gamg_threshold 0.02 -mg_levels_ksp_type chebyshev -mg_levels_pc_type sor -mg_levels_ksp_max_it 2 Unfortunately, the preconditioning seems to run extremely slowly. I tried to play around with the numbers, to check if I could notice some difference, but could not observe significant changes. As a comparison, the KSPSetup call with GAMG PC takes more than 10 times longer than completing the whole computation (preconditioning + ~400 KSP iterations to convergence) of the similar case using the following parameters : -ksp_type cg -pc_type ilu -pc_factor_levels 0 The matrix size for my case is ~1,850,000*1,850,000 elements, with ~38,000,000 non-zero terms (i.e. ~20 per row). For both ILU and AMG cases I use matseqaij/vecseq storage (as a first step I work with only 1 MPI process). Is there something wrong in the parameter set I have been using? I understand that the preconditioning overhead with AMG is higher than with ILU, but I would also expect CG/GAMG to be competitive against CG/ILU, especially considering the relatively big problem size. For information, I am using the PETSc version built from commit 6840fe907c1a3d26068082d180636158471d79a2 (release branch from April 7, 2021). Any clue or idea would be greatly appreciated! Thanks for your help, Best regards, Milan Pelletier

3 2

Undefined reference in PETSc 3.13+ with old MPI version
by Danyang Su 12 Apr '21

12 Apr '21

Dear PETSc developers and users, I am trying to install the latest PETSc version on an ancient cluster. The OpenMPI version is 1.6.5 and Compiler is Intel 14.0, which are the newest on that cluster. I have no problem to install PETSc up to version 3.12.5. However, if I try to use PETSc 3.13+, there are three undefined reference errors in MPI_Win_allocate, MPI_Win_attach and MPI_Win_create_dynamic. I know these three functions are available from OpenMPI 2.0+. Because the cluster is not in technical support anymore, there is no way I can install new OpenMPI version or do some update. Is it possible to disable these three functions in PETSc 3.13+ version? The errors occur in ‘make check’ step: /home/dsu/soft/petsc/petsc-3.13.0/linux-intel-openmpi/lib/libpetsc.so: undefined reference to `MPI_Win_allocate' /home/dsu/soft/petsc/petsc-3.13.0/linux-intel-openmpi/lib/libpetsc.so: undefined reference to `MPI_Win_attach' /home/dsu/soft/petsc/petsc-3.13.0/linux-intel-openmpi/lib/libpetsc.so: undefined reference to `MPI_Win_create_dynamic' The configuration used is shown below: ./configure --with-cc=mpicc --with-cxx=mpicxx --with-fc=mpif90 --download-mumps --download-scalapack --download-parmetis --download-metis --download-fblaslapack --download-hypre --download-superlu --download-hdf5=yes --with-debugging=0 COPTFLAGS="-O3 -march=native -mtune=native" CXXOPTFLAGS="-O3 -march=native -mtune=native" FOPTFLAGS="-O3 -march=native -mtune=nativels" Thanks, Danyang

4 14

BoomerAMG Hypre options
by sthavishtha bhopalam 12 Apr '21

12 Apr '21

Hello PETSc users I am trying to experiment with Hypre's BoomerAMG preconditioner which continually yields the error message "Linear solve did not converge due to DIVERGED_DTOL iterations 1". I would appreciate if someone can suggest some ways I could get BoomerAMG to yield converged results - the attached output shows a snippet of the error message. *Command Line options I used for BoomerAMG* : *-pc_type hypre -pc_hypre_type boomeramg -pchypreboomeramgtol 1.0e-3 -pchypreboomeramgstrongthreshold 0.25 -ksp_type richardson -pc_hypre_boomeramg_max_iter 6 -snes_rtol 1.0e-3 -ksp_rtol 1.0e-3 -ksp_view -snes_view -ksp_monitor -snes_monitor -ksp_max_it 100 -ksp_converged_reason -snes_converged_reason* I also tried using *-ksp_type gmres*, different values of *-pc_hypre_boomeramg_max_iter*, *-pchypreboomeramgstrongthreshold*, *-ksp_initial_guess_nonzero* but all yielded the same error message. However, the direct solver converges as required - the attached output shows a snippet of the norms from the SNES and KSP. *Command Line options I used for the direct solver* : *-ksp_type gmres -pc_type lu -pc_factor_shift_type nonzero -pc_factor_mat_solver_type mumps -snes_converged_reason -ksp_converged_reason -ksp_rtol 1e-3 -snes_rtol 1e-3 -ksp_monitor -snes_monitor* There is no particular reason for using the AMG here, but I just wanted to familiarize with it's options to see which of them need to be particularly tuned to yield converged and correct results. Thanks ------------------------------------------- Regards Sthavishtha

3 6

Newbie question: Strange failure when calling PetscIntView from slepc application
by dazza simplythebest 10 Apr '21

10 Apr '21

Dear All, I am getting a puzzling 'Segmentation Violation' error when I try to write out an integer array using PetscIntView in a Fortran code. I have written the small code below which reproduces the problem. All this code does is create a PetscInt array, initialise this array, then try to write it out to screen. Interestingly PetscIntView does seem to correctly write out all the values to the screen (which agree with a direct write), but then fails before it can return to the main program (see output pasted in below). I think I must be doing something quite silly, but just can't quite see what it is! Any suggestions will be very welcome. Many thanks, Dan Code: MODULE ALL_STAB_ROUTINES IMPLICIT NONE CONTAINS SUBROUTINE WRITE_ROWS_TO_PETSC_MATRIX( ISIZE, JALOC) #include <slepc/finclude/slepceps.h> use slepceps IMPLICIT NONE PetscInt, INTENT (IN) :: ISIZE PetscInt, INTENT(INOUT), DIMENSION(0:ISIZE-1) :: JALOC PetscErrorCode :: ierr write(*,*)'check 02: ',shape(jaloc),lbound(jaloc),ubound(jaloc) write(*,*)jaloc write(*,*)'now for PetscIntView ...' call PetscIntView(ISIZE,JALOC, PETSC_VIEWER_STDOUT_WORLD) CHKERRA(ierr) END SUBROUTINE WRITE_ROWS_TO_PETSC_MATRIX END MODULE ALL_STAB_ROUTINES program stabbo USE MPI #include <slepc/finclude/slepceps.h> use slepceps USE ALL_STAB_ROUTINES IMPLICIT NONE PetscInt, ALLOCATABLE, DIMENSION(:) :: JALOC PetscInt, PARAMETER :: ISIZE = 10 PetscInt, parameter :: FOUR=4 PetscErrorCode :: ierr_pets call SlepcInitialize(PETSC_NULL_CHARACTER,ierr_pets) ALLOCATE(JALOC(0:ISIZE-1)) JALOC = FOUR write(*,*)'check 01: ',shape(jaloc),lbound(jaloc),ubound(jaloc) CALL WRITE_ROWS_TO_PETSC_MATRIX(ISIZE, JALOC) CALL SlepcFinalize(ierr_pets) END PROGRAM STABBO Output: dan@super01 /data/work/rotplane/omega_to_zero/stability/test/tmp10/tmp3 $ mpiexec.hydra -n 1 ./trashy.exe check 01: 10 0 9 check 02: 10 0 9 4 4 4 4 4 4 4 4 4 4 now for PetscIntView ... 0: 4 4 4 4 4 4 4 4 4 4 [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [0]PETSC ERROR: or see https://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [0]PETSC ERROR: likely location of problem given in stack below [0]PETSC ERROR: --------------------- Stack Frames ------------------------------------ [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, [0]PETSC ERROR: INSTEAD the line number of the start of the function [0]PETSC ERROR: is given. [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: Signal received [0]PETSC ERROR: See https://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [0]PETSC ERROR: Petsc Release Version 3.14.5, Mar 03, 2021 [0]PETSC ERROR: ./trashy.exe on a named super01 by darren Fri Apr 9 16:28:25 2021 [0]PETSC ERROR: Configure options --package-prefix-hash=/home/darren/petsc-hash-pkgs --with-cc=mpiicc --with-cxx=mpiicpc --with-fc=mpiifort --with-mpiexec=mpiexec.hydra COPTFLAGS="-g -O" FOPTFLAGS="-g -O" CXXOPTFLAGS="-g -O" --with-64-bit-indices=1 --with-scalar-type=complex --with-precision=double --with-debugging=1 --with-blaslapack-dir=/opt/intel/compilers_and_libraries_2020.0.166/linux/mkl --with-mkl_pardiso-dir=/opt/intel/compilers_and_libraries_2020.0.166/linux/mkl --with-mkl_cpardiso-dir=/opt/intel/compilers_and_libraries_2020.0.166/linux/mkl --download-mumps --download-scalapack --download-cmake PETSC_ARCH=arch-ci-linux-intel-mkl-cmplx-ilp64-dbg-ftn-with-external [0]PETSC ERROR: #1 User provided function() line 0 in unknown file [0]PETSC ERROR: Checking the memory for corruption. Abort(50176059) on node 0 (rank 0 in comm 0): application called MPI_Abort(MPI_COMM_WORLD, 50176059) - process 0

4 8

PETSc log
by Majid Rasouli 10 Apr '21

10 Apr '21

Hi all, I want to solve a linear system using gamg and time the setup and solve phases. I have tried PetscLogDefaultBegin(). It prints many timings including KSPSetUp and KSPSolve. Is it accurate to only report these two timings for the setup and solve phases? Thanks, Majid.

2 1

Running SNES in parallel
by Suwun Suwunnarat 08 Apr '21

08 Apr '21

To whom it may concern, I would like to modify the code in SNES's ex2.c to run in parallel. https://www.mcs.anl.gov/petsc/petsc-3.6.4/src/snes/examples/tutorials/ex2.c… I modified my ex2.c, as attached here. Even though ex3.c is the parallel version, I would like my implementation to "not contain" DMDA. (Despite version 3.6.4, it works for version 3.15.0, which I use in my debugging purpose.) The "crucial" parts of ex2.c that I modified are (apart from deletion of comments, exact solution verification, etc): - (line 63-64) matrix preallocation - use MatGetOwnershipRange on a PETSc matrix jac (in FormJacobian) - use VecGetOwnershipRange on every PETSc vector in the program I try to run the program with mpiexec. For n = 1 it works fine for -pc_type lu -snes_type newtonls. However, when I increase n to be n = 2, the code does not iterate further. (It does not go into the FormJacobian for after the first iteration.) I try several different pc_type and snes_type, but still do not get the result as obtained with n = 1. I attached the terminal printout also. Since the manual (3.15.0) said that (p81, section 2.4, 1st paragraph) "Also, the SNES interface is identical for the uniprocess and parallel cases; the only difference in the parallel version is that each process typically forms only its local contribution to various matrices and vectors" Therefore, I think that by modified only the vectors and matrices part, I should be able to get parallelism. Regards, Suwun PS1: I compile using mpicc and use mpiexec found in the directory ${PETSC_DIR}/arch-mumps-opt/bin/ to run the code. PS2: I attached the install.sh (I install with MUMPS) that I use to install PETSc. Note that I also turn on --with-debuggin=yes PS3: my OS is Ubuntu 20.04.2. Intel CPU. My environment should have everything necessary (gfortran, gcc, g++, valgrind, cmake).

2 1

make check error?
by Bhamidipati, Vikram 06 Apr '21

06 Apr '21

>From configure.log and also my make check command I receive error message due to my Lapack libraries depending on older version of gfortran compared to petsc libraries. I am copy+pasting make check output: -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Running check examples to verify correct installation Using PETSC_DIR=/home/vbhamidipati/petsc/petsc and PETSC_ARCH=arch-linux-c-debug *******************Error detected during compile or link!******************* See http://www.mcs.anl.gov/petsc/documentation/faq.html /home/vbhamidipati/petsc/petsc/src/snes/tutorials ex19 ********************************************************************************* mpicc -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -fstack-protector -fvisibility=hidden -g3 -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -fstack-protector -fvisibility=hidden -g3 -I/home/vbhamidipati/petsc/petsc/include -I/home/vbhamidipati/petsc/petsc/arch-linux-c-debug/include ex19.c -Wl,-rpath,/home/vbhamidipati/petsc/petsc/arch-linux-c-debug/lib -L/home/vbhamidipati/petsc/petsc/arch-linux-c-debug/lib -Wl,-rpath,/usr/lib64/mpich-3.2/lib -L/usr/lib64/mpich-3.2/lib -Wl,-rpath,/opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8 -L/opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8 -Wl,-rpath,/opt/rh/devtoolset-8/root/usr/lib64 -L/opt/rh/devtoolset-8/root/usr/lib64 -Wl,-rpath,/opt/rh/devtoolset-8/root/usr/lib -L/opt/rh/devtoolset-8/root/usr/lib -lpetsc -llapack -lblas -lpthread -lm -lX11 -lstdc++ -ldl -lmpifort -lmpi -lgfortran -lm -lgfortran -lm -lgcc_s -lquadmath -lstdc++ -ldl -o ex19 /opt/rh/devtoolset-8/root/usr/libexec/gcc/x86_64-redhat-linux/8/ld: warning: libgfortran.so.3, needed by /lib/../lib64/liblapack.so, may conflict with libgfortran.so.5 /opt/rh/devtoolset-8/root/usr/libexec/gcc/x86_64-redhat-linux/8/ld: warning: libgfortran.so.3, needed by /lib/../lib64/liblapack.so, may conflict with libgfortran.so.5 C/C++ example src/snes/tutorials/ex19 run successfully with 1 MPI process C/C++ example src/snes/tutorials/ex19 run successfully with 2 MPI processes *******************Error detected during compile or link!******************* See http://www.mcs.anl.gov/petsc/documentation/faq.html /home/vbhamidipati/petsc/petsc/src/snes/tutorials ex5f ********************************************************* mpif90 -Wall -ffree-line-length-0 -Wno-unused-dummy-argument -g -Wall -ffree-line-length-0 -Wno-unused-dummy-argument -g -I/home/vbhamidipati/petsc/petsc/include -I/home/vbhamidipati/petsc/petsc/arch-linux-c-debug/include ex5f.F90 -Wl,-rpath,/home/vbhamidipati/petsc/petsc/arch-linux-c-debug/lib -L/home/vbhamidipati/petsc/petsc/arch-linux-c-debug/lib -Wl,-rpath,/usr/lib64/mpich-3.2/lib -L/usr/lib64/mpich-3.2/lib -Wl,-rpath,/opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8 -L/opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8 -Wl,-rpath,/opt/rh/devtoolset-8/root/usr/lib64 -L/opt/rh/devtoolset-8/root/usr/lib64 -Wl,-rpath,/opt/rh/devtoolset-8/root/usr/lib -L/opt/rh/devtoolset-8/root/usr/lib -lpetsc -llapack -lblas -lpthread -lm -lX11 -lstdc++ -ldl -lmpifort -lmpi -lgfortran -lm -lgfortran -lm -lgcc_s -lquadmath -lstdc++ -ldl -o ex5f /opt/rh/devtoolset-8/root/usr/libexec/gcc/x86_64-redhat-linux/8/ld: warning: libgfortran.so.3, needed by /lib/../lib64/liblapack.so, may conflict with libgfortran.so.5 /opt/rh/devtoolset-8/root/usr/libexec/gcc/x86_64-redhat-linux/8/ld: warning: libgfortran.so.3, needed by /lib/../lib64/liblapack.so, may conflict with libgfortran.so.5 /opt/rh/devtoolset-8/root/usr/libexec/gcc/x86_64-redhat-linux/8/ld: warning: libgfortran.so.3, needed by /lib/../lib64/liblapack.so, may conflict with libgfortran.so.5 /opt/rh/devtoolset-8/root/usr/libexec/gcc/x86_64-redhat-linux/8/ld: warning: libgfortran.so.3, needed by /lib/../lib64/liblapack.so, may conflict with libgfortran.so.5 Fortran example src/snes/tutorials/ex5f run successfully with 1 MPI process Completed test examples -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Is this OK? Or should I replace Lapack libraries and rebuild petsc? Thanks, Vikram --------------------------------------------------- Vikram Bhamidipati Senior Research Engineer Computational Material Integrity Section Materials Engineering Department Mechanical Engineering Division Ph: (210) 522-2576

2 1

correction to previous email
by Zane Charles Jakobs 05 Apr '21

05 Apr '21

Hi PETSc devs, In an email I sent a few minutes ago, I claimed that the TS was writing files of size about 1MB per timestep. That is incorrect; it is actually writing files with a few kB (about 10 kB) per timestep. My questions in the previous email (why are these so big, and can I change that somehow?) still apply though, since this is still much larger than these files "should" be based on the size of the state (128 bits) and the time variable (64 bits). My apologies for the error, and thank you for any help you can provide. -Zane Jakobs

3 3

TSSetSaveTrajectory writes massive files
by Zane Charles Jakobs 05 Apr '21

05 Apr '21

Hi PETSc devs, I'm using TSAdjoint and TAO to solve an optimization problem involving a very simple two-variable ODE, but I've noticed that at each time step, thanks to TSSetSaveTrajectory(), the TS saves an approximately 1MB file. This is turning into a problem for me if I want to use a small timestep (while simple, the ODE I'm solving is very stiff), since the timestepping is slowed down a lot by writing the record file and it also uses a *ton* of disk space. I can't think of a good reason for the TS trajectory files to be so big (after all, my state is two doubles! That's only 128 bits, plus 64 for the time!), which leaves me with two questions: 1. Is there a good reason for the files to be so big? 2. Is there a way to change that? Thank you! -Zane Jakobs

1 0