- petsc-users - mailman3.cels.anl.gov

Re: [petsc-users]DG coordinates for edges/faces in periodic meshes
by Matthew Knepley 07 Aug '26

07 Aug '26

On Fri, Jul 31, 2026 at 6:28 AM Matteo Semplice via petsc-users < petsc-users(a)lists.mcs.anl.gov> wrote: > Dear PETSc, > > I do not understand the behaviour of DMPlexGetCellCoordinates on a > Plex generated to a periodic MSH mesh. > > I have in the code > > DM dmMesh; > PetscCall( DMCreate(MPI_COMM_WORLD, &dmMesh) ); > PetscCall( DMSetType(dmMesh, DMPLEX) ); > PetscCall( DMSetFromOptions(dmMesh) ); > > and pass the option > > -dm_plex_filename periodiccube.msh > > to load a periodic mesh of tetrahedra for the unit cube. > > When calling DMPlexGetCellCoordinates on 3d elements (cells), I get DG > coordinates (independently on passing or not -dm_localize) > > But when calling DMPlexGetCellCoordinates on 1d or 2d elements (edges > and faces), I DO NOT get DG coordinates (independently on passing or not > -dm_localize). > > I can work around this, but is there a way to have DG coordinates also > for intermediate elements in the Plex? I see an option in > DMCreateBoxMesh, but no options for loaded meshes. > Sorry it took me so long to respond. The intention is for -dm_localize_height to work. Does that work for you? Thanks, Matt > Thanks > > Matteo > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uC… <https://urldefense.us/v3/__http://www.cse.buffalo.edu/*knepley/__;fg!!G_uCf… >

1 0

leave
by Amit Itagi 06 Aug '26

06 Aug '26

1 0

[petsc-users]Re: plotting petsc streams
by Junchao Zhang 06 Aug '26

06 Aug '26

Hi Chris, It looks good to me, except the thin8MPIscaling case, where I think the baseline should be one process instead of 8. But that is user's problem (scaling log should always start with 1). Could you make an MR? Thanks, --Junchao Zhang On Mon, Aug 3, 2026 at 8:59 AM Klaij, Christiaan via petsc-users < petsc-users(a)lists.mcs.anl.gov> wrote: > Hi Matt and Jed, > > Here's a new version, to be compared with 3.25.3, together with > three examples. The original use case with the entire range > 1..128, and two other cases: a subset starting at 1 and a > subset starting at 8. > > Hope you find it useful, > Chris > > ________________________________________ > > dr. ir. Christiaan Klaij | senior researcher > Research & Development | CFD Development > T +31 317 49 33 44 <+31%20317%2049%2033%2044> | https://urldefense.us/v3/__http://www.marin.nl__;!!G_uCfscf7eWS!Yk5VkqcMaek… > <https://urldefense.us/v3/__https://www.marin.nl/__;!!G_uCfscf7eWS!YnDGIWz6e…> > [image: Facebook] > <https://urldefense.us/v3/__https://www.facebook.com/marin.wageningen__;!!G_…> > [image: LinkedIn] > <https://urldefense.us/v3/__https://www.linkedin.com/company/marin__;!!G_uCf…> > [image: YouTube] > <https://urldefense.us/v3/__https://www.youtube.com/marinmultimedia__;!!G_uC…> > > From: Klaij, Christiaan <C.Klaij(a)marin.nl> > Sent: Monday, August 3, 2026 11:21 AM > To: Matthew Knepley > Cc: petsc-users > Subject: Re: [petsc-users]plotting petsc streams > > OK, I will make an attempt to improve the script. > > Chris > > ________________________________________ > From: Matthew Knepley <knepley(a)gmail.com> > Sent: Friday, July 31, 2026 2:15 PM > To: Klaij, Christiaan > Cc: petsc-users > Subject: Re: [petsc-users]plotting petsc streams > > Yes, that script only reads in column 2. It works because 'make streams' > always uses the range 1-N for procs. An upgrade would be welcome. > > Thanks, > > Matt > > On Fri, Jul 31, 2026 at 8:01 AM Klaij, Christiaan via petsc-users < > petsc-users(a)lists.mcs.anl.gov<mailto:[email protected]>> > wrote: > With petsc 3.25.3 I made the following scaling.log using > benchmarks/streams on an AMD node: > > $ cat scaling.log > 1 28371.4 Rate (MB/s) 1.0 > 8 240264.1 Rate (MB/s) 8.5 > 16 327898.5 Rate (MB/s) 11.6 > 24 323839.0 Rate (MB/s) 11.4 > 32 326434.0 Rate (MB/s) 11.5 > 40 321278.1 Rate (MB/s) 11.3 > 48 323708.0 Rate (MB/s) 11.4 > 56 318307.4 Rate (MB/s) 11.2 > 64 319949.7 Rate (MB/s) 11.3 > 96 306451.5 Rate (MB/s) 10.8 > 128 313116.4 Rate (MB/s) 11.0 > > Next, benchmarks/streams/process.py gives me the attached > plot. But this plot doesn't make sense to me, for instance it reads > as if I get a 8x speedup going from 1 to 2 procs. It seems the > script is ignoring the first column. Clearly, I'm not using it as > intended? > [cid:ii_19fb8189e5dfa6e50331] > > dr. ir. Christiaan Klaij | senior researcher > Research & Development | CFD Development > T +31 317 49 33 44<tel:+31%20317%2049%2033%2044> | https://urldefense.us/v3/__http://www.marin.nl__;!!G_uCfscf7eWS!Yk5VkqcMaek… > <https://urldefense.us/v3/__http://www.marin.nl__;!!G_uCfscf7eWS!YnDGIWz6ei0…> > < > https://urldefense.us/v3/__https://www.marin.nl/__;!!G_uCfscf7eWS!YvlClmSlF… > > > [Facebook]< > https://urldefense.us/v3/__https://www.facebook.com/marin.wageningen__;!!G_… > > > [LinkedIn]< > https://urldefense.us/v3/__https://www.linkedin.com/company/marin__;!!G_uCf… > > > [YouTube]< > https://urldefense.us/v3/__https://www.youtube.com/marinmultimedia__;!!G_uC… > > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uC… > <https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uC…> > <https://urldefense.us/v3/__http://www.cse.buffalo.edu/*knepley/__;fg!!G_uCf… > <https://urldefense.us/v3/__http://www.cse.buffalo.edu/*knepley/__;fg!!G_uCf…> > > >

2 2

Moving data between two Vecs managed using different DMDAs
by Constantine Khrulev 23 Jul '26

23 Jul '26

Hello, I would like to move data between two Vecs managed by DMDAs. The DMDAs involved may use the same MPI Communicator or possibly different (possibly disjoint) MPI communicators. Crucially, Vecs may have different parallel decompositions but the same global size, so interpolation is not needed. In other words, I would like to set up data exchange to couple components of our model (PISM). All components use DMDA but may use different communicators, domain decomposition, etc, but the same global grid dimensions. Would it be reasonable to use PETSc's tools for this? And if it is, do you have any advice for me? I think I can accomplish this using DMDACreatePatchIS and VecScatter, or maybe using PetscSF. Thank you! -- Constantine

3 2

`VecCreateGhostWithArray` ownership layout vs application indices after migrate
by Miguel Molinos 17 Jul '26

17 Jul '26

Dear all, I am using "VecCreateGhostWithArray" to wrap particle fields (positions, densities, etc.) stored in a “DMSwarm" structure. On each rank the local array is packed as [owned particles | ghost particles]. My understanding from the implementation in pbvec.c is: 1. “PetscSplitOwnership" + "VecSetSizes(n, N)" assigns each rank a contiguous ownership range [rstart, rstart+n). 2. Local owned entry "I" therefore corresponds to global index "rstart + I". 3. The ghosts[] array in “ VecCreateGhostWithArray" must contain those PETSc global vector indices, because “VecGhostUpdate" scatters from the global vector into the local ghost padding. Is that correct? If so, after rebalancing particles with “DMSwarmMigrate", the application “AO" numbering attached to each particle is no longer the same as the contiguous ownership of a newly created ghosted Vec. In that case, should one always renumber the owned particle ids to "rstart + I" (and let ghosts inherit those ids) before calling “VecCreateGhostWithArray"? Thanks, Miguel Miguel Molinos Escuela de Ingenieros de Caminos, Canales y Puertos Dep. Mecánica Medios Continuos y Teoría de Estructuras Prof. Aranguren, 3 - Ciudad Universitaria. Planta 9. 28040, Madrid

2 2

compilation of fortran program that uses petsc hangs
by Ruyter de Wildt, Martijn de 08 Jul '26

08 Jul '26

Good morning, When I try to compile a fortran program that uses petsc, the make command hangs and I can only stop it by killing the make process. I could not find any information about this problem and hope that you can shed some light on it. I can reproduce the problem with one of the examples that is included in the PETSc installation. To do this, I used the following steps: * export PETSC_DIR=[PETSC_INSTALL_DIR] * export PETSC_ARCH= * mkdir -p test_petsc/build * cp [PETSC_INSTALL_DIR]/share/petsc/examples/src/sys/tutorials/ex16f.F90 test_petsc * cp [PETSC_INSTALL_DIR]/share/petsc/CMakeLists.txt test_petsc * in CMakeLists.txt, I replaced all ‘ex1’ with ’ex16f’ and all ‘.c’ with ‘F.90’. * cd test_petsc/build * cmake .. * make VERBOSE=1 The last command hangs after showing “Scanning dependencies of target test”. Environment: * HPC with Rocky Linux 9.7 * PETSc version 3.25.3, compiled from source using cmake * mpi implementation: Intel 2021.16 * compiler: fortran Intel 2025.2.0 ifx * CMake version 3.31.6 Other remarks: * the problem did not occur when we still used an older version of PETSc, version 3.22.4 (without any changes in environment and/or build process) * I can build the c version of the same file, ex16.c, without any problems and delay * the problem seems to occur only when a .h file that depends on petscys.h is included in the fortran file (so also petscmath.h, petscerror.h, petscviever.h, etc.) * it seems that the process gets stuck in an endless loop while scanning dependencies * the actual fortran preprocessor: ifx -I[PETSC_INSTALL_DIR]/include -E ../ex16f.F90 > test.i does not hang and completes immediately * the problem does not occur for PETSC 3.22.4 but it does occur for versions 3.24.6 and later * with cmake 3.26.5 and 3.29.1 the problem also occurs Kind regards, Martijn de Ruyter de Wildt dr. Martijn de Ruyter de Wildt | senior software engineer Research & Development | CFD Development T +31 317 49 35 60 | M +31 639 27 86 06 | M.d.Ruyterd-Wildt(a)marin.nl | https://urldefense.us/v3/__http://www.marin.nl__;!!G_uCfscf7eWS!Y6BnHJQKe2D…

2 3

orientation of DMPLEX for tetrehedral meshes
by Matteo Semplice 03 Jul '26

03 Jul '26

Dear PETSc, I am a bit puzzled by the 3d tet mesh representations as DMPlex. I boiled it down to a mesh of a single tet. More specifically, * if a I create the mesh as const PetscInt dim=3; const PetscInt numCells=1; // The number of cells, only on process 0 const PetscInt numVertices=4; // The number of vertices owned by this process, or PETSC_DECIDE, only on process 0 const PetscInt numCorners=4; // The number of vertices for each cell, only on process 0 const PetscBool interpolate=PETSC_TRUE; // should intermediate mesh entities (faces, edges) be created? // An array of numCells×numCornersnumCells×numCorners numbers, the vertices for each cell, only on process 0 const PetscInt cells[] = {0,1,2,3 }; const PetscInt spaceDim=3; // The spatial dimension used for coordinates // An array of numVertices×spaceDimnumVertices×spaceDim numbers, the coordinates of each vertex, only on process 0 const PetscReal vertexCoords[] = { 0,0,0, 1,0,0, 0,1,0, 0,0,1 }; PetscCall( DMPlexCreateFromCellListPetsc(MPI_COMM_WORLD, dim, numCells, numVertices, numCorners, interpolate, cells, spaceDim, vertexCoords, dmMesh) ); I get the representation that I would expect: DM Object: Mesh 1 MPI process type: plex Mesh in 3 dimensions: Number of 0-cells per rank: 4 Number of 1-cells per rank: 6 Number of 2-cells per rank: 4 Number of 3-cells per rank: 1 Labels: celltype: 4 strata with value/size (0 (4), 1 (6), 3 (4), 6 (1)) depth: 4 strata with value/size (0 (4), 1 (6), 2 (4), 3 (1)) Mesh with 1 cells with DMPlexGetCellCoordinates giving me the coordinates CELL 0 with vertices at ( 0.000, 0, 0.000 ) ( 1.000, 0, 0.000 ) ( 0.000, 1, 0.000 ) ( 0.000, 0, 1.000 ) * on the other hand, if I load the MSH file $MeshFormat 2.2 0 8 $EndMeshFormat $Nodes 4 1 0.0 0.0 0.0 2 1.0 0.0 0.0 3 0.0 1.0 0.0 4 0.0 0.0 1.0 $EndNodes $Elements 1 1 4 2 0 1 1 2 3 4 $EndElements with DMPlexCreateGmshFromFile, the first and second vertices are swapped and I get DM Object: Mesh 1 MPI process type: plex Mesh in 3 dimensions: Number of 0-cells per rank: 4 Number of 1-cells per rank: 6 Number of 2-cells per rank: 4 Number of 3-cells per rank: 1 Labels: celltype: 4 strata with value/size (0 (4), 1 (6), 3 (4), 6 (1)) depth: 4 strata with value/size (0 (4), 1 (6), 2 (4), 3 (1)) Cell Sets: 1 strata with value/size (0 (1)) Mesh with 1 cells with DMPlexGetCellCoordinates giving me CELL 0 with vertices at ( 1.000, 0, 0.000 ) ( 0.000, 0, 0.000 ) ( 0.000, 1, 0.000 ) ( 0.000, 0, 1.000 ) I understand that this behaviour is somehow consistent with the formulas in DMPlexComputeTetrahedronGeometry_Internal, where you compute the Jacobians swappnig the order of vertices (and with 0.5 factors that I do not understand at all... it looks as if your reference element was not the unit simplex) if (J) { for (d = 0; d < dim; d++) { /* I orient with outward face normals */ J[d * dim + 0] = 0.5 * (PetscRealPart(coords[2 * dim + d]) - PetscRealPart(coords[0 * dim + d])); J[d * dim + 1] = 0.5 * (PetscRealPart(coords[1 * dim + d]) - PetscRealPart(coords[0 * dim + d])); J[d * dim + 2] = 0.5 * (PetscRealPart(coords[3 * dim + d]) - PetscRealPart(coords[0 * dim + d])); } } but still I am worried by this fact that different ways of creating a mesh gives a DMPlex with different orientations. Can you explain the rationale behind this or refer me to some documentation? Best Matteo -- Prof. Matteo Semplice Università degli Studi dell’Insubria Dipartimento di Scienza e Alta Tecnologia – DiSAT Professore Associato Via Valleggio, 11 – 22100 Como (CO) – Italia tel.: +39 031 2386316

3 5

PETSC_VERSION_GT doesn't behave as I would have expected
by Eike Mueller 01 Jul '26

01 Jul '26

Dear PETSc developers, I’m using a code which checks #if PETSC_VERSION_GT(3, 24, 5) // do something #endif For my PETSc installation I have: #define PETSC_VERSION_RELEASE 0 #define PETSC_VERSION_MAJOR 3 #define PETSC_VERSION_MINOR 24 #define PETSC_VERSION_SUBMINOR 4 Looking at petscversion.h, I believe that PETSC_VERSION_GT(3, 24, 5) will always evaluate to true since PETSC_VERSION_RELEASE==0. This confused me, since there is no mention of PETSC_VERSION_RELEASE in the documentation at https://urldefense.us/v3/__https://petsc.org/release/manual/versionchecking… Is this intended behaviour? Is there a good way of checking the version irrespective of RELEASE? I can of course always copy and modify the definition of PETSC_VERSION_LT and remove the PETSC_VERSION_RELEASE == 1 bit, but that’s probably not the best solution. Thanks a lot, Eike

3 5

using intel mkl for optimized blas/lapack on amd nodes
by Klaij, Christiaan 29 Jun '26

29 Jun '26

I came across this issue: https://urldefense.us/v3/__https://danieldk.eu/software/misc/intel-mkl-on-a… We do configure petsc with the intel mkl for optimized blas/lapack using --with-blas-lapack-dir=${MKL_DIR}. How much would this issue affect the performance of petsc on amd nodes? Do you have the proposed workaround in place? Chris dr. ir. Christiaan Klaij | senior researcher Research & Development | CFD Development T +31 317 49 33 44 | https://urldefense.us/v3/__http://www.marin.nl__;!!G_uCfscf7eWS!fsSSG7b4cHz…

5 10

Problems with PetscFECreateLagrange(PETSC_COMM_WORLD, 3, 1, PETSC_TRUE, 2, 3, or 4, PETSC_DETERMINE, &fe_u)
by Rafel Amer Ramon 27 Jun '26

27 Jun '26

3 3