May 2024 - petsc-users - mailman3.cels.anl.gov

Diagnosing Convergence Issue in Fieldsplit Problem
by Colton Bryant 23 May '24

23 May '24

Hello, I am solving the Stokes equations on a MAC grid discretized by finite differences using a DMSTAG object. I have tested the solver quite extensively on manufactured problems and it seems to work well. As I am still just trying to get things working and not yet worried about speed I am using the following solver options: -pc_type fieldsplit -pc_fieldsplit_detect_saddle_point -fieldsplit_0_pc_type lu -fieldsplit_1_ksp_rtol 1.e-8 However I am now using this solver as an inner step of a larger code and have run into issues. The code repeatedly solves the Stokes equations with varying right hand sides coming from changing problem geometry (the solver is a part of an overset grid scheme coupled to a level set method evolving in time). After a couple timesteps I observe the following output when running with -fieldsplit_1_ksp_converged_reason -fieldsplit_1_ksp_monitor_true_residual: Residual norms for back_fieldsplit_1_ solve. 0 KSP preconditioned resid norm 2.826514299465e-02 true resid norm 2.826514299465e-02 ||r(i)||/||b|| 1.000000000000e+00 1 KSP preconditioned resid norm 7.286621865915e-03 true resid norm 7.286621865915e-03 ||r(i)||/||b|| 2.577953300039e-01 2 KSP preconditioned resid norm 1.500598474492e-03 true resid norm 1.500598474492e-03 ||r(i)||/||b|| 5.309007192273e-02 3 KSP preconditioned resid norm 3.796396924978e-04 true resid norm 3.796396924978e-04 ||r(i)||/||b|| 1.343137349666e-02 4 KSP preconditioned resid norm 8.091057439816e-05 true resid norm 8.091057439816e-05 ||r(i)||/||b|| 2.862556697960e-03 5 KSP preconditioned resid norm 3.689113122359e-05 true resid norm 3.689113122359e-05 ||r(i)||/||b|| 1.305181128239e-03 6 KSP preconditioned resid norm 2.116450533352e-05 true resid norm 2.116450533352e-05 ||r(i)||/||b|| 7.487846545662e-04 7 KSP preconditioned resid norm 3.968234031201e-06 true resid norm 3.968234031200e-06 ||r(i)||/||b|| 1.403932055801e-04 8 KSP preconditioned resid norm 6.666949419511e-07 true resid norm 6.666949419506e-07 ||r(i)||/||b|| 2.358717739644e-05 9 KSP preconditioned resid norm 1.941522884928e-07 true resid norm 1.941522884931e-07 ||r(i)||/||b|| 6.868965372998e-06 10 KSP preconditioned resid norm 6.729545258682e-08 true resid norm 6.729545258626e-08 ||r(i)||/||b|| 2.380863687793e-06 11 KSP preconditioned resid norm 3.009070131709e-08 true resid norm 3.009070131735e-08 ||r(i)||/||b|| 1.064586912687e-06 12 KSP preconditioned resid norm 7.849353009588e-09 true resid norm 7.849353009903e-09 ||r(i)||/||b|| 2.777043445840e-07 13 KSP preconditioned resid norm 2.306283345754e-09 true resid norm 2.306283346677e-09 ||r(i)||/||b|| 8.159461097060e-08 14 KSP preconditioned resid norm 9.336302495083e-10 true resid norm 9.336302502503e-10 ||r(i)||/||b|| 3.303115255517e-08 15 KSP preconditioned resid norm 6.537456143401e-10 true resid norm 6.537456141617e-10 ||r(i)||/||b|| 2.312903968982e-08 16 KSP preconditioned resid norm 6.389159552788e-10 true resid norm 6.389159550304e-10 ||r(i)||/||b|| 2.260437724130e-08 17 KSP preconditioned resid norm 6.380905134246e-10 true resid norm 6.380905136023e-10 ||r(i)||/||b|| 2.257517372981e-08 18 KSP preconditioned resid norm 6.380440605992e-10 true resid norm 6.380440604688e-10 ||r(i)||/||b|| 2.257353025207e-08 19 KSP preconditioned resid norm 6.380427156582e-10 true resid norm 6.380427157894e-10 ||r(i)||/||b|| 2.257348267830e-08 20 KSP preconditioned resid norm 6.380426714897e-10 true resid norm 6.380426714004e-10 ||r(i)||/||b|| 2.257348110785e-08 21 KSP preconditioned resid norm 6.380426656970e-10 true resid norm 6.380426658839e-10 ||r(i)||/||b|| 2.257348091268e-08 22 KSP preconditioned resid norm 6.380426650538e-10 true resid norm 6.380426650287e-10 ||r(i)||/||b|| 2.257348088242e-08 23 KSP preconditioned resid norm 6.380426649918e-10 true resid norm 6.380426645888e-10 ||r(i)||/||b|| 2.257348086686e-08 24 KSP preconditioned resid norm 6.380426649803e-10 true resid norm 6.380426644294e-10 ||r(i)||/||b|| 2.257348086122e-08 25 KSP preconditioned resid norm 6.380426649796e-10 true resid norm 6.380426649774e-10 ||r(i)||/||b|| 2.257348088061e-08 26 KSP preconditioned resid norm 6.380426649795e-10 true resid norm 6.380426653788e-10 ||r(i)||/||b|| 2.257348089481e-08 27 KSP preconditioned resid norm 6.380426649795e-10 true resid norm 6.380426646744e-10 ||r(i)||/||b|| 2.257348086989e-08 28 KSP preconditioned resid norm 6.380426649795e-10 true resid norm 6.380426650818e-10 ||r(i)||/||b|| 2.257348088430e-08 29 KSP preconditioned resid norm 6.380426649795e-10 true resid norm 6.380426649518e-10 ||r(i)||/||b|| 2.257348087970e-08 30 KSP preconditioned resid norm 6.380426652142e-10 true resid norm 6.380426652142e-10 ||r(i)||/||b|| 2.257348088898e-08 31 KSP preconditioned resid norm 6.380426652141e-10 true resid norm 6.380426646799e-10 ||r(i)||/||b|| 2.257348087008e-08 32 KSP preconditioned resid norm 6.380426652141e-10 true resid norm 6.380426648077e-10 ||r(i)||/||b|| 2.257348087460e-08 33 KSP preconditioned resid norm 6.380426652141e-10 true resid norm 6.380426649048e-10 ||r(i)||/||b|| 2.257348087804e-08 34 KSP preconditioned resid norm 6.380426652141e-10 true resid norm 6.380426648142e-10 ||r(i)||/||b|| 2.257348087483e-08 35 KSP preconditioned resid norm 6.380426652141e-10 true resid norm 6.380426651079e-10 ||r(i)||/||b|| 2.257348088522e-08 36 KSP preconditioned resid norm 6.380426652141e-10 true resid norm 6.380426650433e-10 ||r(i)||/||b|| 2.257348088294e-08 37 KSP preconditioned resid norm 6.380426652141e-10 true resid norm 6.380426649765e-10 ||r(i)||/||b|| 2.257348088057e-08 38 KSP preconditioned resid norm 6.380426652141e-10 true resid norm 6.380426650364e-10 ||r(i)||/||b|| 2.257348088269e-08 39 KSP preconditioned resid norm 6.380426652141e-10 true resid norm 6.380426650051e-10 ||r(i)||/||b|| 2.257348088159e-08 40 KSP preconditioned resid norm 6.380426652141e-10 true resid norm 6.380426651154e-10 ||r(i)||/||b|| 2.257348088549e-08 41 KSP preconditioned resid norm 6.380426652141e-10 true resid norm 6.380426650246e-10 ||r(i)||/||b|| 2.257348088227e-08 42 KSP preconditioned resid norm 6.380426652141e-10 true resid norm 6.380426650702e-10 ||r(i)||/||b|| 2.257348088389e-08 43 KSP preconditioned resid norm 6.380426652141e-10 true resid norm 6.380426651686e-10 ||r(i)||/||b|| 2.257348088737e-08 44 KSP preconditioned resid norm 6.380426652141e-10 true resid norm 6.380426650870e-10 ||r(i)||/||b|| 2.257348088448e-08 45 KSP preconditioned resid norm 6.380426652141e-10 true resid norm 6.380426651208e-10 ||r(i)||/||b|| 2.257348088568e-08 46 KSP preconditioned resid norm 6.380426652141e-10 true resid norm 6.380426651441e-10 ||r(i)||/||b|| 2.257348088650e-08 47 KSP preconditioned resid norm 6.380426652141e-10 true resid norm 6.380426650955e-10 ||r(i)||/||b|| 2.257348088478e-08 48 KSP preconditioned resid norm 6.380426652141e-10 true resid norm 6.380426650877e-10 ||r(i)||/||b|| 2.257348088451e-08 49 KSP preconditioned resid norm 6.380426652141e-10 true resid norm 6.380426651240e-10 ||r(i)||/||b|| 2.257348088579e-08 50 KSP preconditioned resid norm 6.380426652141e-10 true resid norm 6.380426650534e-10 ||r(i)||/||b|| 2.257348088329e-08 51 KSP preconditioned resid norm 6.380426652141e-10 true resid norm 6.380426648615e-10 ||r(i)||/||b|| 2.257348087651e-08 52 KSP preconditioned resid norm 6.380426652141e-10 true resid norm 6.380426649523e-10 ||r(i)||/||b|| 2.257348087972e-08 53 KSP preconditioned resid norm 6.380426652140e-10 true resid norm 6.380426652601e-10 ||r(i)||/||b|| 2.257348089061e-08 54 KSP preconditioned resid norm 6.380426652125e-10 true resid norm 6.380427512852e-10 ||r(i)||/||b|| 2.257348393411e-08 55 KSP preconditioned resid norm 6.380426651849e-10 true resid norm 6.380603444402e-10 ||r(i)||/||b|| 2.257410636701e-08 56 KSP preconditioned resid norm 6.380426646751e-10 true resid norm 6.439925413105e-10 ||r(i)||/||b|| 2.278398313542e-08 57 KSP preconditioned resid norm 6.380426514019e-10 true resid norm 2.674218007058e-09 ||r(i)||/||b|| 9.461186902765e-08 58 KSP preconditioned resid norm 6.380425077384e-10 true resid norm 2.406759314486e-08 ||r(i)||/||b|| 8.514937691775e-07 59 KSP preconditioned resid norm 6.380406171326e-10 true resid norm 3.100137288622e-07 ||r(i)||/||b|| 1.096805803957e-05 Linear back_fieldsplit_1_ solve did not converge due to DIVERGED_BREAKDOWN iterations 60 Any advice on steps I could take to elucidate the issue would be greatly appreciated. Thanks so much for any help in advance! Best, Colton Bryant

2 5

How to specify different MPI communication patterns.
by Randall Mackie 21 May '24

21 May '24

Dear PETSc team, A few years ago we were having some issue with MPI communications with large numbers of processes and subcomms, see this thread here: https://urldefense.us/v3/__https://lists.mcs.anl.gov/mailman/htdig/petsc-us… We are once again encountering strange issues when running our code on a new cluster and after a month of various tests we have not found a solution, but we think it has something to do with network traffic and high MPI communications, similar perhaps to the thread from 3 years ago. Is it still possible to change the communication pattern with the option -build_twosided_allreduce (and is that the right syntax?). Are there other runtime options that we can try to change the MPI communication type for all underlying communications? Thank you, Randy M.

2 1

Help with Integrating PETSc into Fortran Groundwater Flow Simulation Code
by Shatanawi, Sawsan Muhammad 18 May '24

18 May '24

Hello everyone, I hope this email finds you well. My Name is Sawsan Shatanawi, and I am currently working on developing a Fortran code for simulating groundwater flow in a 3D system. The code involves solving a nonlinear system, and I have created the matrix to be solved using the PCG solver and Picard iteration. However, when I tried to assign it as a PETSc matrix I started getting a lot of error messages. I am kindly asking if someone can help me, I would be happy to share my code with him/her. Please find the attached file contains a list of errors I have gotten Thank you in advance for your time and assistance. Best regards, Sawsan

5 29

Using CUDA Graphs with MatComputeOperator
by Sreeram R Venkat 15 May '24

15 May '24

I have a MatShell object that I want to convert to a MATDENSECUDA. Normally, I use MatComputeOperator for this. However, I would now also like to use a CUDA Graph so that all the calls to MatMult are captured. I can wrap a code like for (int i = 0; i < N; i++) MatMult(A, x,y); in a CUDA Graph, and it runs fine. If I try to wrap MatComputeOperator in a graph, I get runtime errors like cuda error 906 (cudaErrorStreamCaptureImplicit) : operation would make the legacy stream depend on a capturing blocking stream I tried modifying the MatConvert_Shell routine to only put the graph around the main for loop, but that still gives the same errors. Is there a way to use CUDA Graphs here (either through a modified MatConvert_Shell or otherwise)? Thanks, Sreeram

2 2

DMPlex periodic face coordinates
by Matteo Semplice 15 May '24

15 May '24

Dear petsc-users, I am playing with DMPlexGetCellCoordinates and observing that it returns correct periodic coordinates for cells, but not for faces. More precisely, adding PetscCall(DMPlexGetHeightStratum(dm, 1, &fStart, &fEnd)); for (f = fStart; f < fEnd; ++f) { const PetscScalar *array; PetscScalar *x = NULL; PetscInt ndof; PetscBool isDG; PetscCall(DMPlexGetCellCoordinates(dm, f, &isDG, &ndof, &array, &x)); PetscCheck(ndof % cdim == 0, PETSC_COMM_SELF, PETSC_ERR_ARG_INCOMP, "ndof not divisible by cdim"); PetscCall(PetscPrintf(PETSC_COMM_SELF, "Face #%" PetscInt_FMT " coordinates\n", f - fStart)); for (PetscInt i = 0; i < ndof; i += cdim) PetscCall(PetscScalarView(cdim, &x[i], PETSC_VIEWER_STDOUT_SELF)); PetscCall(DMPlexRestoreCellCoordinates(dm, f, &isDG, &ndof, &array, &x)); } to src/dm/impls/plex/tutorials/ex8.c, I get $ ./ex8 -dm_plex_dim 2 -petscspace_degree 1 -dm_plex_simplex 0 -dm_plex_box_faces 3,2 -dm_plex_box_bd periodic,none -dm_view -view_coord DM Object: box 1 MPI process type: plex box in 2 dimensions: Number of 0-cells per rank: 9 Number of 1-cells per rank: 15 Number of 2-cells per rank: 6 Periodic mesh (PERIODIC, NONE) coordinates localized [...] Element #2 coordinates 0: 6.6667e-01 0.0000e+00 0: 1.0000e+00 0.0000e+00 <<<----- correct 0: 1.0000e+00 5.0000e-01 0: 6.6667e-01 5.0000e-01 [...] Face #0 coordinates 0: 0.0000e+00 0.0000e+00 0: 3.3333e-01 0.0000e+00 Face #1 coordinates 0: 3.3333e-01 0.0000e+00 0: 6.6667e-01 0.0000e+00 Face #2 coordinates 0: 6.6667e-01 0.0000e+00 0: 0.0000e+00 0.0000e+00 <<<---- should be (0.66,0.00) and (1.00,0.00) Is there a way to recover correct periodic coordinates in the case of periodic DMPLex? Thanks in advance Matteo

3 3

duplicated libs
by Runjian Wu 15 May '24

15 May '24

Hi all, After I compiled PETSc, I found some duplicated libs in the variable PETSC_EXTERNAL_LIB_BASIC, e.g., -lm, -lgfortran -lstdc++. I am curious how it happened and how to remove the duplicates? Thanks, Runjian Wu

3 11

GMSH entities
by Adrian Croucher 14 May '24

14 May '24

2 5

Re: [petsc-users] VecGetArrayF90 vs. VecGetArrayReadF90
by Barry Smith 14 May '24

14 May '24

It errors in C because the argument is labeled with const, but there does not seem to be a way in Fortran to indicate an array is read only. > On May 13, 2024, at 10:21 PM, Runjian Wu <wurunjian(a)gmail.com> wrote: > > I only know the "intent(in)" attribute for the dummy arguments. > > In the counter function VecGetArrayRead(..), if I write values, will the compiler report an error? > > Runjian > > On Mon, May 13, 2024 at 9:36 PM Barry Smith <bsmith(a)petsc.dev <mailto:[email protected]>> wrote: >> >> I couldn't find a way in Fortran to declare an array as read-only. Is there such support? >> >> Barry >> >> >>> On May 13, 2024, at 7:28 AM, Runjian Wu <wurunjian(a)gmail.com <mailto:[email protected]>> wrote: >>> >>> This Message Is From an External Sender >>> This message came from outside your organization. >>> Hi all, >>> >>> I have a question about VecGetArrayReadF90(...). If I use VecGetArrayReadF90(...), I can still write entries into the array like VecGetArrayF90(...). Is it possible to report an error during compile process? >>> >>> Thanks, >>> >>> Runjian Wu >>

2 1

Using Compute-Sanitizer with PETSc
by Sreeram R Venkat 13 May '24

13 May '24

I am trying to check my program for GPU memory leaks with the compute-sanitizer tool. If I run my application with: mpiexec -n 1 compute-sanitizer --tool memcheck --leak-check full ./a.out args I get the message: Error: No attachable process found. compute-sanitizer timed-out. Adding --target-processes all does not help. Is there anything else I should be doing? Thanks, Sreeram

2 1

Re: [petsc-users] [SLEPc] Best method to compute all eigenvalues of a MatShell
by Jose E. Roman 13 May '24

13 May '24

2 2