October 2024 - petsc-users - mailman3.cels.anl.gov

local/global DMPlex Vec output
by Matteo Semplice 30 Jan '25

30 Jan '25

Dear petsc-users, I am having issues with output of parallel data attached to a DMPlex (or maybe more fundamental ones about DMPlex...). So I currently 1. create a DMPlex (DMPlexCreateGmshFromFile or DMPlexCreateBoxMesh) 2. partition it 3. and create a section for my data layout with DMPlexCreateSection(ctx.dmMesh, NULL, numComp, numDof, numBC, NULL, NULL, NULL, NULL, &sUavg) 4. DMSetLocalSection(ctx.dmMesh, sUavg) 5. create solLoc and solGlob vectors with DMCreateGlobalVector and DMCreateLocalVector 6. solve .... 7. VecView(ctx.solGlob, vtkViewer) on a .vtu file but when I load data in ParaView I get more cells than expected and it is as if the cells in the halo are put twice in output. (I could create a MWE if the above is not clear) I guess that the culprit is point (4), but if I replace it with DMSetGlobalSection then I cannot create the local vector at point (5). How should I handle this properly? In my code I need to create both local and global vectors, to perform at least GlobalToLocal and to save the global data. (On a side note, I tried also HDF5 but then it complains about the DM not having a DS...; really, any working solution that allows data to be explored with Paraview is fine) Thanks for any advice! Matteo Semplice

3 7

Ghost particles for DMSWARM (or similar)
by MIGUEL MOLINOS PEREZ 19 Nov '24

19 Nov '24

3 24

Spring 2024 PETSc meeting and release announcement 3.20
by Barry Smith 09 Nov '24

09 Nov '24

The May 23-24, 2024 PETSc annual meeting in Cologne Germany website is now available: https://cds.uni-koeln.de/en/workshops/petsc-2024/home Please register and start making your travel plans. We are pleased to announce the release of PETSc version 3.20.0 at https://petsc.org/release/download/ A list of the major changes and updates can be found at https://petsc.org/release/changes/320/ The release tarball location is changed from https://ftp.mcs.anl.gov/pub/petsc to https://web.cels.anl.gov/projects/petsc/download The final update to petsc-3.19 i.e petsc-3.19.6 is also available We recommend upgrading to PETSc 3.20.0 soon. As always, please report problems to petsc-maint(a)mcs.anl.gov and ask questions at petsc-users(a)mcs.anl.gov This release includes contributions from David Ham Iglesia Dolci Jack Betteridge Koki Sagiyama Marcin Rogowski Nathan Collier Aidan Hamilton Albert Cowie Barry Smith Blaise Bourdin Connor Ward David Andrs David Wells Duncan Campbell Hansol Suh Hong Zhang Jacob Faibussowitsch James Wright Jed Brown jeremy theler Joseph Pusztay Jose Roman Junchao Zhang Lisandro Dalcin Mark Adams Matthew Knepley Mr. Hong Zhang Pablo Brubeck Pierre Jolivet Pieter Ghysels Richard Tran Mills Satish Balay Sebastian Grimberg Stefano Zampini Suyash Tandon Toby Isaac YANG Zongze Zach Atkins and bug reports/proposed improvements received from Carl-Johan Thore Chris Douglas Daniel@OLCF Duan Junming Edoardo alinovi Philip Fackler Frederic Vi Getnet Betrie Aron Graham Hongrui Yu James Wright Jin Chen Jonas Lundgren Karsten Lettmann Boris Kaus LEONARDO MUTTI Maciej Lewkowicz maitri ksh Milan Pelletier Mr. Hong Zhang Ngoc Mai Monica Huynh Peng Wang Sébastien Gilles Daniel Taller Vibhas Mishra Wei Liu Li Xiaoye As always, thanks for your support, Barry

1 2

Expected weak scaling behaviour for AMG libraries?
by Khurana, Parv 08 Nov '24

08 Nov '24

Hello PETSc Community, I am trying to understand the scaling behaviour of AMG methods in PETSc (Hypre for now) and how many DOFs/Rank are needed for a performant AMG solve. I’m currently conducting weak scaling tests using src/snes/tutorials/ex12.c in 3D, applying Dirichlet BCs with FEM at P=1. The tests keep DOFs per processor constant while increasing the mesh size and processor count, specifically: * 20000 and 80000 DOF/RANK configurations. * Running SNES twice, using GMRES with a tolerance of 1e-5 and preconditioning with Hypre-BoomerAMG. Unfortunately, parallel efficiency degrades noticeably with increased processor counts. Are there any insights or rules of thumb for using AMG more effectively? I have been looking at this issue for a while now and would love to engage in a further discussion. Please find below the weak scaling results and the options I use to run the tests. [cid:b3265a6e-baed-4fc1-a58d-0eebbb7de5a4] #Run type -run_type full -petscpartitioner_type simple #Mesh settings -dm_plex_dim 3 -dm_plex_simplex 1 -dm_refine 5 #Varied this -dm_plex_box_faces 6,6,6 #BCs and FEM space -bc_type dirichlet -petscspace_degree 1 #Solver settings -snes_max_it 2 -ksp_type gmres -ksp_rtol 1.0e-5 #Same settings as what we use for LOR -pc_type hypre -pc_hypre_type boomeramg -pc_hypre_boomeramg_coarsen_type hmis -pc_hypre_boomeramg_relax_type_all symmetric-sor/jacobi -pc_hypre_boomeramg_strong_threshold 0.7 -pc_hypre_boomeramg_interp_type ext+i -pc_hypre_boomeramg_P_max 2 -pc_hypre_boomeramg_truncfactor 0.3 Best, Parv

3 4

Rigid body nullspace for Stokes operator
by Amneet Bhalla 05 Nov '24

05 Nov '24

Hi Folks, I am trying to solve the momentum equation in a projection preconditioner using GAMG or Hypre solver. The equation looks like for velocity variable *v* looks like: [image: Screenshot 2024-10-28 at 4.15.17 PM.png] Here, μ is spatially varying dynamic viscosity and λ is spatially varying bulk viscosity. I understand that I need to specify rigid body nullspace modes to the multigrid solver in order to accelerate its convergence. Looking into this routine MatNullSpaceCreateRigidBody() ( https://urldefense.us/v3/__https://petsc.org/release/manualpages/Mat/MatNul… ), I see that I need to provide the coordinates of each node. I am using staggered grid discretization. Do I need to provide coordinates of staggered grid locations? Thanks, -- --Amneet

4 20

Example of SNES using matrix free jacobian
by Daniel Pino Munoz 04 Nov '24

04 Nov '24

Dear all, I have a linear problem that I am currently solving with a KSP matrix-free. I would like to move on to a non linear problem, so figure I could start by solving the same linear problem using SNES. So I am setting the problem as follows: SNESCreate(PETSC_COMM_WORLD, &snes); MatCreateShell(PETSC_COMM_WORLD, n_dofs, n_dofs, PETSC_DETERMINE, PETSC_DETERMINE, &ctx, &J); MatCreateShell(PETSC_COMM_WORLD, n_dofs, n_dofs, PETSC_DETERMINE, PETSC_DETERMINE, &ctx, &B); MatShellSetOperation(J, MATOP_MULT, (void (*)(void))(Multiplication)); MatCreateVecs(J, &x_sol, &b); VecDuplicate(x_sol, &r); SNESSetFromOptions(snes); SNESSetFunction(snes, r, &(computeResidual), &ctx); SNESSetUseMatrixFree(snes, PETSC_FALSE, PETSC_TRUE); SNESGetLineSearch(snes, &linesearch); SNESGetKSP(snes, &ksp); KSPSetOperators(ksp, J, J); KSPSetInitialGuessNonzero(ksp, PETSC_TRUE); I tested it with a small problem (compiled in debug) and it works. When I compiled it in Release, it crashes with a segfault. I tried running the Debug version through valgrind, but even for a small problem, it is too slow. So I was wondering if you guys could see any rocky mistake on the lines I used above? Otherwise, is there any example that uses a SNES combined with a matrix free KSP operator? Thank you, Daniel

2 8

Problem running geoclaw example with petsc 3.22
by Praveen C 31 Oct '24

31 Oct '24

Dear all I am not able to run the boussinesq example from geoclaw using petsc(a)3.22.0 https://urldefense.us/v3/__https://github.com/clawpack/geoclaw/tree/3303883… It runs with petsc(a)3.21.6 The error I get is given below. After printing this, the code does not progress. I use the following petsc options # set min numbers of matrix rows per MPI rank (default is 10000) -mpi_linear_solve_minimum_count_per_rank 5000 # Krylov linear solver: -mpi_linear_solver_server -mpi_linear_solver_server_view -ksp_type gmres -ksp_max_it 200 -ksp_reuse_preconditioner -ksp_rtol 1.e-9 # preconditioner: -pc_type gamg I installed petsc and other dependencies for clawpack using miniforge. Thanks pc ==> Use Bouss. in water deeper than 1.0000000000000000 Using a PETSc solver Using Bouss equations from the start rnode allocated... node allocated... listOfGrids allocated... Storage allocated... bndList allocated... Gridding level 1 at t = 0.000000E+00: 4 grids with 10000 cells Setting initial dt to 2.9999999999999999E-002 max threads set to 6 Done reading data, starting computation ... Total zeta at initial time: 39269.907650665169 GEOCLAW: Frame 0 output files done at time t = 0.000000D+00 [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: Petsc has generated inconsistent data [0]PETSC ERROR: Unable to locate PCMPI allocated shared address 0x55e6d750ae20 [0]PETSC ERROR: WARNING! There are unused option(s) set! Could be the program crashed before usage or a spelling mistake, etc! [0]PETSC ERROR: Option left: name:-ksp_max_it value: 200 source: file [0]PETSC ERROR: Option left: name:-ksp_reuse_preconditioner (no value) source: file [0]PETSC ERROR: Option left: name:-ksp_rtol value: 1.e-9 source: file [0]PETSC ERROR: Option left: name:-ksp_type value: gmres source: file [0]PETSC ERROR: Option left: name:-mpi_linear_solve_minimum_count_per_rank value: 5000 source: file [0]PETSC ERROR: Option left: name:-mpi_linear_solver_server_view (no value) source: file [0]PETSC ERROR: Option left: name:-pc_type value: gamg source: file [0]PETSC ERROR: See https://urldefense.us/v3/__https://petsc.org/release/faq/__;!!G_uCfscf7eWS!… for trouble shooting. [0]PETSC ERROR: Petsc Release Version 3.22.0, Sep 28, 2024 [0]PETSC ERROR: /home/praveen/bouss/radial_flat/xgeoclaw with 6 MPI process(es) and PETSC_ARCH on euler by praveen Thu Oct 17 21:49:54 2024 [0]PETSC ERROR: Configure options: AR=${PREFIX}/bin/x86_64-conda-linux-gnu-ar CC=mpicc CXX=mpicxx FC=mpifort CFLAGS="-march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /opt/miniforge/envs/claw/include " CPPFLAGS="-DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -isystem /opt/miniforge/envs/claw/include" CXXFLAGS="-fvisibility-inlines-hidden -fmessage-length=0 -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /opt/miniforge/envs/claw/include " FFLAGS="-march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /opt/miniforge/envs/claw/include -Wl,--no-as-needed" LDFLAGS="-pthread -fopenmp -Wl,-O2 -Wl,--sort-common -Wl,--as-needed -Wl,-z,relro -Wl,-z,now -Wl,--disable-new-dtags -Wl,--gc-sections -Wl,--allow-shlib-undefined -Wl,-rpath,/opt/miniforge/envs/claw/lib -Wl,-rpath-link,/opt/miniforge/envs/claw/lib -L/opt/miniforge/envs/claw/lib -Wl,-rpath-link,/opt/miniforge/envs/claw/lib" LIBS="-Wl,-rpath,/opt/miniforge/envs/claw/lib -lmpi_mpifh -lgfortran" --COPTFLAGS=-O3 --CXXOPTFLAGS=-O3 --FOPTFLAGS=-O3 --with-clib-autodetect=0 --with-cxxlib-autodetect=0 --with-fortranlib-autodetect=0 --with-debugging=0 --with-blas-lib=libblas.so --with-lapack-lib=liblapack.so --with-yaml=1 --with-hdf5=1 --with-fftw=1 --with-hwloc=0 --with-hypre=1 --with-metis=1 --with-mpi=1 --with-mumps=1 --with-parmetis=1 --with-pthread=1 --with-ptscotch=1 --with-shared-libraries --with-ssl=0 --with-scalapack=1 --with-superlu=1 --with-superlu_dist=1 --with-superlu_dist-include=/opt/miniforge/envs/claw/include/superlu-dist --with-superlu_dist-lib=-lsuperlu_dist --with-suitesparse=1 --with-suitesparse-dir=/opt/miniforge/envs/claw --with-x=0 --with-scalar-type=real --with-cuda=0 --prefix=/opt/miniforge/envs/claw [0]PETSC ERROR: #1 PetscShmgetMapAddresses() at /home/conda/feedstock_root/build_artifacts/petsc_1728030599661/work/src/sys/utils/server.c:114 [0]PETSC ERROR: #2 PCMPISetMat() at /home/conda/feedstock_root/build_artifacts/petsc_1728030599661/work/src/ksp/pc/impls/mpi/pcmpi.c:269 [0]PETSC ERROR: #3 PCSetUp_MPI() at /home/conda/feedstock_root/build_artifacts/petsc_1728030599661/work/src/ksp/pc/impls/mpi/pcmpi.c:853 [0]PETSC ERROR: #4 PCSetUp() at /home/conda/feedstock_root/build_artifacts/petsc_1728030599661/work/src/ksp/pc/interface/precon.c:1071 [0]PETSC ERROR: #5 KSPSetUp() at /home/conda/feedstock_root/build_artifacts/petsc_1728030599661/work/src/ksp/ksp/interface/itfunc.c:415 [0]PETSC ERROR: #6 KSPSolve_Private() at /home/conda/feedstock_root/build_artifacts/petsc_1728030599661/work/src/ksp/ksp/interface/itfunc.c:826 [0]PETSC ERROR: #7 KSPSolve() at /home/conda/feedstock_root/build_artifacts/petsc_1728030599661/work/src/ksp/ksp/interface/itfunc.c:1075

2 18

Check the precision for real numbers
by Runjian Wu 29 Oct '24

29 Oct '24

Hi all, I have a question about how to return the precision for real number in Fortran code? Thanks, Runjian

2 1

Requesting help with SNES non-linear example
by Vaishak Prasad 27 Oct '24

27 Oct '24

Dear All, Greetings from India. I am new to petsc4py. I was trying out the example on slides 18, and 19 at https://urldefense.us/v3/__https://www.bu.edu/pasi/files/2011/01/Lisandro-D… However, I get a zero vector for the solution `x`. I am using petsc4py 3.21.3 on Linux. Any help would be appreciated. With regards -- Vaishak Prasad Post Doctoral Fellow Astrophysical Relativity Group International Center for Theoretical Sciences Tata Institute of Fundamental Research Survey No. 151, Shivakote, Hesaraghatta Hobli, Bengaluru, Karnataka, India, Earth, Solar system, Milky Way Galaxy, The Local Group

2 5

Block sparse matrix
by Diego Magela Lemos 24 Oct '24

24 Oct '24

Is there something similar to SciPy sparse block_array ( https://urldefense.us/v3/__https://docs.scipy.org/doc/scipy/reference/gener… ) or bmat ( https://urldefense.us/v3/__https://docs.scipy.org/doc/scipy/reference/gener… ) in PETSc?

3 3