October 2023 - petsc-users - mailman3.cels.anl.gov

Spring 2024 PETSc meeting and release announcement 3.20
by Barry Smith 09 Nov '24

09 Nov '24

The May 23-24, 2024 PETSc annual meeting in Cologne Germany website is now available: https://cds.uni-koeln.de/en/workshops/petsc-2024/home Please register and start making your travel plans. We are pleased to announce the release of PETSc version 3.20.0 at https://petsc.org/release/download/ A list of the major changes and updates can be found at https://petsc.org/release/changes/320/ The release tarball location is changed from https://ftp.mcs.anl.gov/pub/petsc to https://web.cels.anl.gov/projects/petsc/download The final update to petsc-3.19 i.e petsc-3.19.6 is also available We recommend upgrading to PETSc 3.20.0 soon. As always, please report problems to petsc-maint(a)mcs.anl.gov and ask questions at petsc-users(a)mcs.anl.gov This release includes contributions from David Ham Iglesia Dolci Jack Betteridge Koki Sagiyama Marcin Rogowski Nathan Collier Aidan Hamilton Albert Cowie Barry Smith Blaise Bourdin Connor Ward David Andrs David Wells Duncan Campbell Hansol Suh Hong Zhang Jacob Faibussowitsch James Wright Jed Brown jeremy theler Joseph Pusztay Jose Roman Junchao Zhang Lisandro Dalcin Mark Adams Matthew Knepley Mr. Hong Zhang Pablo Brubeck Pierre Jolivet Pieter Ghysels Richard Tran Mills Satish Balay Sebastian Grimberg Stefano Zampini Suyash Tandon Toby Isaac YANG Zongze Zach Atkins and bug reports/proposed improvements received from Carl-Johan Thore Chris Douglas Daniel@OLCF Duan Junming Edoardo alinovi Philip Fackler Frederic Vi Getnet Betrie Aron Graham Hongrui Yu James Wright Jin Chen Jonas Lundgren Karsten Lettmann Boris Kaus LEONARDO MUTTI Maciej Lewkowicz maitri ksh Milan Pelletier Mr. Hong Zhang Ngoc Mai Monica Huynh Peng Wang Sébastien Gilles Daniel Taller Vibhas Mishra Wei Liu Li Xiaoye As always, thanks for your support, Barry

1 2

How to combine different element types into a single DMPlex?
by Eric Chamberland 01 Aug '24

01 Aug '24

Hi, while playing with DMPlexBuildFromCellListParallel, I noticed we have to specify "numCorners" which is a fixed value, then gives a fixed number of nodes for a series of elements. How can I then add, for example, triangles and quadrangles into a DMPlex? Thanks, Eric -- Eric Chamberland, ing., M. Ing Professionnel de recherche GIREF/Université Laval (418) 656-2131 poste 41 22 42

4 13

HDF5 corruption
by Adrian Croucher 13 May '24

13 May '24

hi, One of the users of my PETSc-based code has reported that HDF5 output files can be corrupted and unusable if e.g. the run is killed. I've just done a bit of reading about this and it appears to be a known issue with HDF5. Some people suggest flushing the HDF5 file periodically to help prevent data loss. I had a look at PetscViewerFlush() but it doesn't seem to be implemented for the HDF5 viewer- is that correct? I am currently calling PetscViewerHDF5Open() once at the start of the run and closing it at the end. Some other people suggest doing this before and after each write instead. Is there likely to be a significant performance penalty in doing that? I gather HDF5 journaling has been promised for a while to get around this problem but as far as I can see it hasn't materialised yet... Regards, Adrian -- Dr Adrian Croucher Senior Research Fellow Department of Engineering Science University of Auckland, New Zealand email: a.croucher(a)auckland.ac.nz tel: +64 (0)9 923 4611

4 17

Using PETSc GPU backend
by Ng, Cho-Kuen 13 Apr '24

13 Apr '24

I installed PETSc on Perlmutter using "spack install petsc+cuda+zoltan" and used it by "spack load petsc/fwge6pf". Then I compiled the application code (purely CPU code) linking to the petsc package, hoping that I can get performance improvement using the petsc GPU backend. However, the timing was the same using the same number of MPI tasks with and without GPU accelerators. Have I missed something in the process, for example, setting up PETSc options at runtime to use the GPU backend? Thanks, Cho

5 27

Fortran Interface NULL object / Casting
by Nicholas Arnold-Medabalimi 23 Feb '24

23 Feb '24

Hi Petsc Users I am trying to use DMAddField in a Fortran code. I had some questions on casting/passing NULL. I follow how to pass NULL for standard types (INT, CHAR, etc). Is there a method/best practice for passing NULL for Petsc type arguments? (In this case DMAddLabel I'd want to pass NULL to a DMLabel Parameter not an intrinsic type) For the 2nd question, In some cases in C we would cast a more specific object to Petsc Object DMAddField(dm, NULL, (PetscObject)fvm); (where fvm is a PetscFVM type) Is there a method or practice to do the same in Fortran. Thanks Nicholas -- Nicholas Arnold-Medabalimi Ph.D. Candidate Computational Aeroscience Lab University of Michigan

3 2

DMSwarm on multiple processors
by Joauma Marichal 20 Dec '23

20 Dec '23

Hello, I am using the DMSwarm library in some Eulerian-Lagrangian approach to have vapor bubbles in water. I have obtained nice results recently and wanted to perform bigger simulations. Unfortunately, when I increase the number of processors used to run the simulation, I get the following error: free(): invalid size [cns136:590327] *** Process received signal *** [cns136:590327] Signal: Aborted (6) [cns136:590327] Signal code: (-6) [cns136:590327] [ 0] /lib64/libc.so.6(+0x4eb20)[0x7f56cd4c9b20] [cns136:590327] [ 1] /lib64/libc.so.6(gsignal+0x10f)[0x7f56cd4c9a9f] [cns136:590327] [ 2] /lib64/libc.so.6(abort+0x127)[0x7f56cd49ce05] [cns136:590327] [ 3] /lib64/libc.so.6(+0x91037)[0x7f56cd50c037] [cns136:590327] [ 4] /lib64/libc.so.6(+0x9819c)[0x7f56cd51319c] [cns136:590327] [ 5] /lib64/libc.so.6(+0x99aac)[0x7f56cd514aac] [cns136:590327] [ 6] /gpfs/home/acad/ucl-tfl/marichaj/marha/lib_petsc/lib/libpetsc.so.3.019(PetscSFSetUpRanks+0x4c4)[0x7f56cea71e64] [cns136:590327] [ 7] /gpfs/home/acad/ucl-tfl/marichaj/marha/lib_petsc/lib/libpetsc.so.3.019(+0x841642)[0x7f56cea83642] [cns136:590327] [ 8] /gpfs/home/acad/ucl-tfl/marichaj/marha/lib_petsc/lib/libpetsc.so.3.019(PetscSFSetUp+0x9e)[0x7f56cea7043e] [cns136:590327] [ 9] /gpfs/home/acad/ucl-tfl/marichaj/marha/lib_petsc/lib/libpetsc.so.3.019(VecScatterCreate+0x164e)[0x7f56cea7bbde] [cns136:590327] [10] /gpfs/home/acad/ucl-tfl/marichaj/marha/lib_petsc/lib/libpetsc.so.3.019(DMSetUp_DA_3D+0x3e38)[0x7f56cee84dd8] [cns136:590327] [11] /gpfs/home/acad/ucl-tfl/marichaj/marha/lib_petsc/lib/libpetsc.so.3.019(DMSetUp_DA+0xd8)[0x7f56cee9b448] [cns136:590327] [12] /gpfs/home/acad/ucl-tfl/marichaj/marha/lib_petsc/lib/libpetsc.so.3.019(DMSetUp+0x20)[0x7f56cededa20] [cns136:590327] [13] ./cobpor[0x4418dc] [cns136:590327] [14] ./cobpor[0x408b63] [cns136:590327] [15] /lib64/libc.so.6(__libc_start_main+0xf3)[0x7f56cd4b5cf3] [cns136:590327] [16] ./cobpor[0x40bdee] [cns136:590327] *** End of error message *** -------------------------------------------------------------------------- Primary job terminated normally, but 1 process returned a non-zero exit code. Per user-direction, the job has been aborted. -------------------------------------------------------------------------- -------------------------------------------------------------------------- mpiexec noticed that process rank 84 with PID 590327 on node cns136 exited on signal 6 (Aborted). -------------------------------------------------------------------------- When I reduce the number of processors the error disappears and when I run my code without the vapor bubbles it also works. The problem seems to take place at this moment: DMCreate(PETSC_COMM_WORLD,swarm); DMSetType(*swarm,DMSWARM); DMSetDimension(*swarm,3); DMSwarmSetType(*swarm,DMSWARM_PIC); DMSwarmSetCellDM(*swarm,*dmcell); Thanks a lot for your help. Best regards, Joauma

3 12

PETSc 3.14 to PETSc 3.20: Different (slower) convergence for classical AMG (sequential and especially in parallel)
by LEDAC Pierre 14 Dec '23

14 Dec '23

Hello all, I am struggling to find the same convergence in iterations when using classical algebric multigrid in my code with PETSc 3.20 compared to PETSc 3.14. I am using in order to solve a Poisson system: -ksp_type cg -pc_type gamg -pc_gamg_type classical I read the different releases notes between 3.15 and 3.20: https://petsc.org/release/changes/317 https://petsc.org/main/manualpages/PC/PCGAMGSetThreshold/ And have a look at the archive mailing list (especially this one: https://www.mail-archive.com/[email protected]/msg46688.html) so I added some other options to try to have the same behaviour than PETSc 3.14: -ksp_type cg -pc_type gamg -pc_gamg_type classical -mg_levels_pc_type sor -pc_gamg_threshold 0. It improves the convergence but there still a different convergence though (26 vs 18 iterations). On another of my test case, the number of levels is different (e.g. 6 vs 4) also, and here it is the same, but with a different coarsening according to the output from the -ksp_view option The main point is that the convergence dramatically degrades in parallel on a third test case, so I can't upgrade to PETSc 3.20 for now unhappily. I send you the partial report (petsc_314_vs_petsc_320.ksp_view) with -ksp_view (left PETSc 3.14, right PETSc 3.20) and the configure/command line options used (in petsc_XXX_petsc.TU files). Could my issue related to the following 3.18 change ? I have not tried the first one. * Remove PCGAMGSetSymGraph() and -pc_gamg_sym_graph. The user should now indicate symmetry and structural symmetry using MatSetOption<https://petsc.org/release/manualpages/Mat/MatSetOption/>() and GAMG will symmetrize the graph if a symmetric options is not set * Change -pc_gamg_reuse_interpolation default from false to true. Any advice would be greatly appreciated, Pierre LEDAC Commissariat à l’énergie atomique et aux énergies alternatives Centre de SACLAY DES/ISAS/DM2S/SGLS/LCAN Bâtiment 451 – point courrier n°43 F-91191 Gif-sur-Yvette +33 1 69 08 04 03 +33 6 83 42 05 79

2 3

Bug Report TaoALMM class
by Stephan Köhler 10 Dec '23

10 Dec '23

Dear PETSc/Tao team, I think there might be a bug in the Tao ALMM class: In the function TaoALMMComputeAugLagAndGradient_Private(), see, eg. https://petsc.org/release/src/tao/constrained/impls/almm/almm.c.html#TAOALMM line 648 the gradient seems to be wrong. The given function and gradient computation is Lc = F + Ye^TCe + Yi^T(Ci - S) + 0.5*mu*[Ce^TCe + (Ci - S)^T(Ci - S)], dLc/dX = dF/dX + Ye^TAe + Yi^TAi + 0.5*mu*[Ce^TAe + (Ci - S)^TAi], but I think the gradient should be (without 0.5) dLc/dX = dF/dX + Ye^TAe + Yi^TAi + mu*[Ce^TAe + (Ci - S)^TAi]. Kind regards, Stephan Köhler -- Stephan Köhler TU Bergakademie Freiberg Institut für numerische Mathematik und Optimierung Akademiestraße 6 09599 Freiberg Gebäudeteil Mittelbau, Zimmer 2.07 Telefon: +49 (0)3731 39-3173 (Büro)

2 2

Scattering a vector to/from a subset of processors
by Sreeram R Venkat 06 Dec '23

06 Dec '23

Suppose I am running on 12 processors, and I have a vector "v" of size 36 partitioned over the first 4. v still uses the PETSC_COMM_WORLD, so it has a layout of (9, 9, 9, 9, 0, 0, ..., 0). Now, I would like to repartition it over all 12 processors, so that the layout becomes (3, 3, 3, ..., 3). I've been trying to use VecScatter to do this, but I'm not sure what IndexSets to use for the sender and receiver. The result I am trying to achieve is this: Assume the vector is v = <0, 1, 2, ..., 35> Start Finish Proc | Entries Proc | Entries 0 | 0,...,8 0 | 0, 1, 2 1 | 9,...,17 1 | 3, 4, 5 2 | 18,...,26 2 | 6, 7, 8 3 | 27,...,35 3 | 9, 10, 11 4 | None 4 | 12, 13, 14 5 | None 5 | 15, 16, 17 6 | None 6 | 18, 19, 20 7 | None 7 | 21, 22, 23 8 | None 8 | 24, 25, 26 9 | None 9 | 27, 28, 29 10 | None 10 | 30, 31, 32 11 | None 11 | 33, 34, 35 Appreciate any help you can provide on this. Thanks, Sreeram

2 8

Unexpected performance losses switching to COO interface
by Fackler, Philip 30 Nov '23

30 Nov '23

We finally have xolotl ported to use the new COO interface and the aijkokkos implementation for Mat (and kokkos for Vec). Comparing this port to our previous version (using MatSetValuesStencil and the default Mat and Vec implementations), we expected to see an improvement in performance for both the "serial" and "cuda" builds (here I'm referring to the kokkos configuration). Attached are two plots that show timings for three different cases. All of these were run on Ascent (the Summit-like training system) with 6 MPI tasks (on a single node). The CUDA cases were given one GPU per task (and used CUDA-aware MPI). The labels on the blue bars indicate speedup. In all cases we used "-fieldsplit_0_pc_type jacobi" to keep the comparison as consistent as possible. The performance of RHSJacobian (where the bulk of computation happens in xolotl) behaved basically as expected (better than expected in the serial build). NE_3 case in CUDA was the only one that performed worse, but not surprisingly, since its workload for the GPUs is much smaller. We've still got more optimization to do on this. The real surprise was how much worse the overall solve times were. This seems to be due simply to switching to the kokkos-based implementation. I'm wondering if there are any changes we can make in configuration or runtime arguments to help with PETSc's performance here. Any help looking into this would be appreciated. The tarballs linked here<https://drive.google.com/file/d/19X_L3SVkGBM9YUzXnRR_kVWFG0JFwqZ3/view?usp=…> and here<https://drive.google.com/file/d/15yDBN7-YlO1g6RJNPYNImzr611i1Ffhv/view?usp=…> are profiling databases which, once extracted, can be viewed with hpcviewer. I don't know how helpful that will be, but hopefully it can give you some direction. Thanks for your help, Philip Fackler Research Software Engineer, Application Engineering Group Advanced Computing Systems Research Section Computer Science and Mathematics Division Oak Ridge National Laboratory

5 19