November 2023 - petsc-users - mailman3.cels.anl.gov

Spring 2024 PETSc meeting and release announcement 3.20
by Barry Smith 09 Nov '24

09 Nov '24

The May 23-24, 2024 PETSc annual meeting in Cologne Germany website is now available: https://cds.uni-koeln.de/en/workshops/petsc-2024/home Please register and start making your travel plans. We are pleased to announce the release of PETSc version 3.20.0 at https://petsc.org/release/download/ A list of the major changes and updates can be found at https://petsc.org/release/changes/320/ The release tarball location is changed from https://ftp.mcs.anl.gov/pub/petsc to https://web.cels.anl.gov/projects/petsc/download The final update to petsc-3.19 i.e petsc-3.19.6 is also available We recommend upgrading to PETSc 3.20.0 soon. As always, please report problems to petsc-maint(a)mcs.anl.gov and ask questions at petsc-users(a)mcs.anl.gov This release includes contributions from David Ham Iglesia Dolci Jack Betteridge Koki Sagiyama Marcin Rogowski Nathan Collier Aidan Hamilton Albert Cowie Barry Smith Blaise Bourdin Connor Ward David Andrs David Wells Duncan Campbell Hansol Suh Hong Zhang Jacob Faibussowitsch James Wright Jed Brown jeremy theler Joseph Pusztay Jose Roman Junchao Zhang Lisandro Dalcin Mark Adams Matthew Knepley Mr. Hong Zhang Pablo Brubeck Pierre Jolivet Pieter Ghysels Richard Tran Mills Satish Balay Sebastian Grimberg Stefano Zampini Suyash Tandon Toby Isaac YANG Zongze Zach Atkins and bug reports/proposed improvements received from Carl-Johan Thore Chris Douglas Daniel@OLCF Duan Junming Edoardo alinovi Philip Fackler Frederic Vi Getnet Betrie Aron Graham Hongrui Yu James Wright Jin Chen Jonas Lundgren Karsten Lettmann Boris Kaus LEONARDO MUTTI Maciej Lewkowicz maitri ksh Milan Pelletier Mr. Hong Zhang Ngoc Mai Monica Huynh Peng Wang Sébastien Gilles Daniel Taller Vibhas Mishra Wei Liu Li Xiaoye As always, thanks for your support, Barry

1 2

How to combine different element types into a single DMPlex?
by Eric Chamberland 01 Aug '24

01 Aug '24

Hi, while playing with DMPlexBuildFromCellListParallel, I noticed we have to specify "numCorners" which is a fixed value, then gives a fixed number of nodes for a series of elements. How can I then add, for example, triangles and quadrangles into a DMPlex? Thanks, Eric -- Eric Chamberland, ing., M. Ing Professionnel de recherche GIREF/Université Laval (418) 656-2131 poste 41 22 42

4 13

HDF5 corruption
by Adrian Croucher 13 May '24

13 May '24

hi, One of the users of my PETSc-based code has reported that HDF5 output files can be corrupted and unusable if e.g. the run is killed. I've just done a bit of reading about this and it appears to be a known issue with HDF5. Some people suggest flushing the HDF5 file periodically to help prevent data loss. I had a look at PetscViewerFlush() but it doesn't seem to be implemented for the HDF5 viewer- is that correct? I am currently calling PetscViewerHDF5Open() once at the start of the run and closing it at the end. Some other people suggest doing this before and after each write instead. Is there likely to be a significant performance penalty in doing that? I gather HDF5 journaling has been promised for a while to get around this problem but as far as I can see it hasn't materialised yet... Regards, Adrian -- Dr Adrian Croucher Senior Research Fellow Department of Engineering Science University of Auckland, New Zealand email: a.croucher(a)auckland.ac.nz tel: +64 (0)9 923 4611

4 17

Using PETSc GPU backend
by Ng, Cho-Kuen 13 Apr '24

13 Apr '24

I installed PETSc on Perlmutter using "spack install petsc+cuda+zoltan" and used it by "spack load petsc/fwge6pf". Then I compiled the application code (purely CPU code) linking to the petsc package, hoping that I can get performance improvement using the petsc GPU backend. However, the timing was the same using the same number of MPI tasks with and without GPU accelerators. Have I missed something in the process, for example, setting up PETSc options at runtime to use the GPU backend? Thanks, Cho

5 27

Fortran Interface NULL object / Casting
by Nicholas Arnold-Medabalimi 23 Feb '24

23 Feb '24

Hi Petsc Users I am trying to use DMAddField in a Fortran code. I had some questions on casting/passing NULL. I follow how to pass NULL for standard types (INT, CHAR, etc). Is there a method/best practice for passing NULL for Petsc type arguments? (In this case DMAddLabel I'd want to pass NULL to a DMLabel Parameter not an intrinsic type) For the 2nd question, In some cases in C we would cast a more specific object to Petsc Object DMAddField(dm, NULL, (PetscObject)fvm); (where fvm is a PetscFVM type) Is there a method or practice to do the same in Fortran. Thanks Nicholas -- Nicholas Arnold-Medabalimi Ph.D. Candidate Computational Aeroscience Lab University of Michigan

3 2

DMSwarm on multiple processors
by Joauma Marichal 20 Dec '23

20 Dec '23

Hello, I am using the DMSwarm library in some Eulerian-Lagrangian approach to have vapor bubbles in water. I have obtained nice results recently and wanted to perform bigger simulations. Unfortunately, when I increase the number of processors used to run the simulation, I get the following error: free(): invalid size [cns136:590327] *** Process received signal *** [cns136:590327] Signal: Aborted (6) [cns136:590327] Signal code: (-6) [cns136:590327] [ 0] /lib64/libc.so.6(+0x4eb20)[0x7f56cd4c9b20] [cns136:590327] [ 1] /lib64/libc.so.6(gsignal+0x10f)[0x7f56cd4c9a9f] [cns136:590327] [ 2] /lib64/libc.so.6(abort+0x127)[0x7f56cd49ce05] [cns136:590327] [ 3] /lib64/libc.so.6(+0x91037)[0x7f56cd50c037] [cns136:590327] [ 4] /lib64/libc.so.6(+0x9819c)[0x7f56cd51319c] [cns136:590327] [ 5] /lib64/libc.so.6(+0x99aac)[0x7f56cd514aac] [cns136:590327] [ 6] /gpfs/home/acad/ucl-tfl/marichaj/marha/lib_petsc/lib/libpetsc.so.3.019(PetscSFSetUpRanks+0x4c4)[0x7f56cea71e64] [cns136:590327] [ 7] /gpfs/home/acad/ucl-tfl/marichaj/marha/lib_petsc/lib/libpetsc.so.3.019(+0x841642)[0x7f56cea83642] [cns136:590327] [ 8] /gpfs/home/acad/ucl-tfl/marichaj/marha/lib_petsc/lib/libpetsc.so.3.019(PetscSFSetUp+0x9e)[0x7f56cea7043e] [cns136:590327] [ 9] /gpfs/home/acad/ucl-tfl/marichaj/marha/lib_petsc/lib/libpetsc.so.3.019(VecScatterCreate+0x164e)[0x7f56cea7bbde] [cns136:590327] [10] /gpfs/home/acad/ucl-tfl/marichaj/marha/lib_petsc/lib/libpetsc.so.3.019(DMSetUp_DA_3D+0x3e38)[0x7f56cee84dd8] [cns136:590327] [11] /gpfs/home/acad/ucl-tfl/marichaj/marha/lib_petsc/lib/libpetsc.so.3.019(DMSetUp_DA+0xd8)[0x7f56cee9b448] [cns136:590327] [12] /gpfs/home/acad/ucl-tfl/marichaj/marha/lib_petsc/lib/libpetsc.so.3.019(DMSetUp+0x20)[0x7f56cededa20] [cns136:590327] [13] ./cobpor[0x4418dc] [cns136:590327] [14] ./cobpor[0x408b63] [cns136:590327] [15] /lib64/libc.so.6(__libc_start_main+0xf3)[0x7f56cd4b5cf3] [cns136:590327] [16] ./cobpor[0x40bdee] [cns136:590327] *** End of error message *** -------------------------------------------------------------------------- Primary job terminated normally, but 1 process returned a non-zero exit code. Per user-direction, the job has been aborted. -------------------------------------------------------------------------- -------------------------------------------------------------------------- mpiexec noticed that process rank 84 with PID 590327 on node cns136 exited on signal 6 (Aborted). -------------------------------------------------------------------------- When I reduce the number of processors the error disappears and when I run my code without the vapor bubbles it also works. The problem seems to take place at this moment: DMCreate(PETSC_COMM_WORLD,swarm); DMSetType(*swarm,DMSWARM); DMSetDimension(*swarm,3); DMSwarmSetType(*swarm,DMSWARM_PIC); DMSwarmSetCellDM(*swarm,*dmcell); Thanks a lot for your help. Best regards, Joauma

3 12

PETSc 3.14 to PETSc 3.20: Different (slower) convergence for classical AMG (sequential and especially in parallel)
by LEDAC Pierre 14 Dec '23

14 Dec '23

Hello all, I am struggling to find the same convergence in iterations when using classical algebric multigrid in my code with PETSc 3.20 compared to PETSc 3.14. I am using in order to solve a Poisson system: -ksp_type cg -pc_type gamg -pc_gamg_type classical I read the different releases notes between 3.15 and 3.20: https://petsc.org/release/changes/317 https://petsc.org/main/manualpages/PC/PCGAMGSetThreshold/ And have a look at the archive mailing list (especially this one: https://www.mail-archive.com/[email protected]/msg46688.html) so I added some other options to try to have the same behaviour than PETSc 3.14: -ksp_type cg -pc_type gamg -pc_gamg_type classical -mg_levels_pc_type sor -pc_gamg_threshold 0. It improves the convergence but there still a different convergence though (26 vs 18 iterations). On another of my test case, the number of levels is different (e.g. 6 vs 4) also, and here it is the same, but with a different coarsening according to the output from the -ksp_view option The main point is that the convergence dramatically degrades in parallel on a third test case, so I can't upgrade to PETSc 3.20 for now unhappily. I send you the partial report (petsc_314_vs_petsc_320.ksp_view) with -ksp_view (left PETSc 3.14, right PETSc 3.20) and the configure/command line options used (in petsc_XXX_petsc.TU files). Could my issue related to the following 3.18 change ? I have not tried the first one. * Remove PCGAMGSetSymGraph() and -pc_gamg_sym_graph. The user should now indicate symmetry and structural symmetry using MatSetOption<https://petsc.org/release/manualpages/Mat/MatSetOption/>() and GAMG will symmetrize the graph if a symmetric options is not set * Change -pc_gamg_reuse_interpolation default from false to true. Any advice would be greatly appreciated, Pierre LEDAC Commissariat à l’énergie atomique et aux énergies alternatives Centre de SACLAY DES/ISAS/DM2S/SGLS/LCAN Bâtiment 451 – point courrier n°43 F-91191 Gif-sur-Yvette +33 1 69 08 04 03 +33 6 83 42 05 79

2 3

Re: [petsc-users] Domain decomposition in PETSc for Molecular Dynamics
by MIGUEL MOLINOS PEREZ 13 Dec '23

13 Dec '23

Thank you Mark! I will have a look to it. Best, Miguel On 4 Nov 2023, at 13:54, Matthew Knepley <knepley(a)gmail.com> wrote: On Sat, Nov 4, 2023 at 8:40 AM Mark Adams <mfadams(a)lbl.gov<mailto:[email protected]>> wrote: Hi MIGUEL, This might be a good place to start: https://petsc.org/main/manual/vec/ Feel free to ask more specific questions, but the docs are a good place to start. Thanks, Mark On Fri, Nov 3, 2023 at 5:19 AM MIGUEL MOLINOS PEREZ <mmolinos(a)us.es<mailto:[email protected]>> wrote: Dear all, I am currently working on the development of a in-house molecular dynamics code using PETSc and C++. So far the code works great, however it is a little bit slow since I am not exploiting MPI for PETSc vectors. I was wondering if there is a way to perform the domain decomposition efficiently using some PETSc functionality. Any feedback is highly appreciated. It sounds like you mean "is there a way to specify a communication construct that can send my particle information automatically". We use PetscSF for that. You can see how this works with the DMSwarm class, which represents a particle discretization. You can either use that, or if it does not work for you, do the same things with your class. Thanks, Matt Best regards, Miguel -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/<http://www.cse.buffalo.edu/~knepley/>

2 5

Bug Report TaoALMM class
by Stephan Köhler 10 Dec '23

10 Dec '23

Dear PETSc/Tao team, I think there might be a bug in the Tao ALMM class: In the function TaoALMMComputeAugLagAndGradient_Private(), see, eg. https://petsc.org/release/src/tao/constrained/impls/almm/almm.c.html#TAOALMM line 648 the gradient seems to be wrong. The given function and gradient computation is Lc = F + Ye^TCe + Yi^T(Ci - S) + 0.5*mu*[Ce^TCe + (Ci - S)^T(Ci - S)], dLc/dX = dF/dX + Ye^TAe + Yi^TAi + 0.5*mu*[Ce^TAe + (Ci - S)^TAi], but I think the gradient should be (without 0.5) dLc/dX = dF/dX + Ye^TAe + Yi^TAi + mu*[Ce^TAe + (Ci - S)^TAi]. Kind regards, Stephan Köhler -- Stephan Köhler TU Bergakademie Freiberg Institut für numerische Mathematik und Optimierung Akademiestraße 6 09599 Freiberg Gebäudeteil Mittelbau, Zimmer 2.07 Telefon: +49 (0)3731 39-3173 (Büro)

2 2

Scattering a vector to/from a subset of processors
by Sreeram R Venkat 06 Dec '23

06 Dec '23

Suppose I am running on 12 processors, and I have a vector "v" of size 36 partitioned over the first 4. v still uses the PETSC_COMM_WORLD, so it has a layout of (9, 9, 9, 9, 0, 0, ..., 0). Now, I would like to repartition it over all 12 processors, so that the layout becomes (3, 3, 3, ..., 3). I've been trying to use VecScatter to do this, but I'm not sure what IndexSets to use for the sender and receiver. The result I am trying to achieve is this: Assume the vector is v = <0, 1, 2, ..., 35> Start Finish Proc | Entries Proc | Entries 0 | 0,...,8 0 | 0, 1, 2 1 | 9,...,17 1 | 3, 4, 5 2 | 18,...,26 2 | 6, 7, 8 3 | 27,...,35 3 | 9, 10, 11 4 | None 4 | 12, 13, 14 5 | None 5 | 15, 16, 17 6 | None 6 | 18, 19, 20 7 | None 7 | 21, 22, 23 8 | None 8 | 24, 25, 26 9 | None 9 | 27, 28, 29 10 | None 10 | 30, 31, 32 11 | None 11 | 33, 34, 35 Appreciate any help you can provide on this. Thanks, Sreeram

2 8