October 2022 - petsc-users - mailman3.cels.anl.gov

Slepc, shell matrix, parallel, halo exchange
by feng wang 13 Oct '22

13 Oct '22

Hello, I am using Slepc with a shell matrix. The sequential version seems working and now I am trying to make it run in parallel. The partition of the domain is done, I am not sure how to do the halo exchange in the shell matrix in Slepc. I have a parallel version of matrix-free GMRES in my code with Petsc. I was using VecCreateGhostBlock to create vector with ghost cells, and then used VecGhostUpdateBegin/End for the halo exchange in the shell matrix, would this be the same for Slepc? Thanks, Feng

3 14

DMLocalToLocal with DMPlex in Fortran
by Mike Michell 12 Oct '22

12 Oct '22

Hi, As a follow-up to this email thread, https://www.mail-archive.com/[email protected]/msg44070.html Are DMLocalToLocalBegin() and DMLocalToLocalEnd() really available for DMPlex with Fortran on the latest version of PETSc (3.17.99 from GitLab)? Matt commented that the Fortran bindings were updated so that those functions must be available in the latest version of PETSc, however, it seems still they are not working from my test with DMPlex in Fortran. Can anyone provide some comments? Probably I am missing some mandatory header file? Currently, I have headers; #include "petsc/finclude/petscvec.h" #include "petsc/finclude/petscdmplex.h" #include "petsc/finclude/petscdmlabel.h" #include "petsc/finclude/petscdm.h" Thanks, Mike

2 9

How to get total subsnes iterations
by Takahashi, Tadanaga 12 Oct '22

12 Oct '22

Hi. I am using the snes nasm for the global solver and snes newtonls for the local subdomain solver. I am trying to get the total number of Newton iterations for just one subdomain. I've tried: SNESNASMGetSNES(snes,0,&subsnes); SNESSolve(snes,NULL,u_initial); SNESGetNumberFunctionEvals(subsnes,&Newt_its); but this just gets me the number of Newton iterations just on the final nasm iteration. If I understand correctly, the information in the subsnes is repeatedly destroyed while the SNESSolve is running. Is there any way to extract the total subsnes iterations after the SNESSolve? If not, how would I extract the information?

2 1

VecScatter
by Jorti, Zakariae 11 Oct '22

11 Oct '22

Hello, I have a code that handles a PETSc Vec on many procs and I would like to use VecScatterCreateToZero to have all elements of this vector on a single proc, call VecGetArrayRead on this proc to get the corresponding array to carry out some diagnostics. Unfortunately, what I noticed is that the ordering of the initial Vec is not preserved after the VecScatterCreateToZero call. Is there a way to have the same ordering for both Vecs? You will find below a minimal example that demonstrates the issue. Many thanks, Zakariae ------------------------------------------------------------- static char help[] = "Demonstrates ordering change after VecScatterCreateToZero call.\n\n"; #include <petscpf.h> #include <petscdm.h> #include <petscdmda.h> int main(int argc,char **argv) { Vec xy; DM da; PetscErrorCode ierr; PetscInt m = 11, n = 11, dof = 2; PetscMPIInt rank; DMDACoor2d **coors; ierr = PetscInitialize(&argc,&argv,(char*)0,help);if (ierr) return ierr; ierr = DMDACreate2d(PETSC_COMM_WORLD, DM_BOUNDARY_NONE, DM_BOUNDARY_NONE,DMDA_STENCIL_BOX,m,n,PETSC_DECIDE,PETSC_DECIDE,dof,1,0,0,&da);CHKERRQ(ierr); ierr = DMSetFromOptions(da);CHKERRQ(ierr); ierr = DMSetUp(da);CHKERRQ(ierr); ierr = DMDASetUniformCoordinates(da,0.0,1.0,0.0,1.0,0.0,1.0);CHKERRQ(ierr); ierr = DMGetCoordinates(da,&xy);CHKERRQ(ierr); PetscInt i, j, ixs, ixm, iys, iym; MPI_Comm_rank(PETSC_COMM_WORLD, &rank); DMDAGetCorners(da, &ixs, &iys, 0, &ixm, &iym, 0); DMDAVecGetArray(da, xy, &coors); for (j = iys; j < iys + iym; j++) { for (i = ixs; i < ixs + ixm; i++) { PetscPrintf(PETSC_COMM_SELF, "rank=%d, %d, %d, (%g, %g)\n",rank, i, j,coors[j][i].x,coors[j][i].y); } } DMDAVecRestoreArray(da, xy, &coors); VecScatter scat; Vec Xseq; const PetscScalar *array; /* create scater to zero */ VecScatterCreateToZero(xy, &scat, &Xseq); VecScatterBegin(scat, xy, Xseq, INSERT_VALUES, SCATTER_FORWARD); VecScatterEnd(scat, xy, Xseq, INSERT_VALUES, SCATTER_FORWARD); if (rank == 0) { PetscInt sizeX; VecGetSize(Xseq, &sizeX); PetscPrintf(PETSC_COMM_SELF,"The size of Xseq is %d, and the grid size is %d\n",sizeX,11*11); VecGetArrayRead(Xseq, &array); for (j = 0; j < 11; j++) { for (i = 0; i < 11; i++) { PetscPrintf(PETSC_COMM_SELF, "%d,%d, (%g,%g)\n", i, j, (double)array[2*(j*11+i)], (double)array[1+2*(j*11+i)]); } } VecRestoreArrayRead(Xseq, &array); } /* Free work space. All PETSc objects should be destroyed when they are no longer needed. */ ierr = DMDestroy(&da);CHKERRQ(ierr); ierr = PetscFinalize(); return ierr; }

2 1

make all check error
by Sijie Zhang 11 Oct '22

11 Oct '22

Hi, When I try to install petsc on my PC and run the make all check commend it has the following error. Can you help me to troubleshoot that? Thanks. Sijie ++++++++++++++++++++++++++++++++++++++++++++++++ Running check examples to verify correct installation Using PETSC_DIR=/home/zhangsijie1995/Documents/Packages/petsc-3.18.0 and PETSC_ARCH=arch-linux-c-debug Possible error running C/C++ src/snes/tutorials/ex19 with 1 MPI process See https://petsc.org/release/faq/ hwloc/linux: Ignoring PCI device with non-16bit domain. Pass --enable-32bits-pci-domain to configure to support such devices (warning: it would break the library ABI, don't enable unless really needed). lid velocity = 0.0016, prandtl # = 1., grashof # = 1. Number of SNES iterations = 2 Possible error running C/C++ src/snes/tutorials/ex19 with 2 MPI processes See https://petsc.org/release/faq/ hwloc/linux: Ignoring PCI device with non-16bit domain. Pass --enable-32bits-pci-domain to configure to support such devices (warning: it would break the library ABI, don't enable unless really needed). hwloc/linux: Ignoring PCI device with non-16bit domain. Pass --enable-32bits-pci-domain to configure to support such devices (warning: it would break the library ABI, don't enable unless really needed). lid velocity = 0.0016, prandtl # = 1., grashof # = 1. Number of SNES iterations = 2 Possible error running Fortran example src/snes/tutorials/ex5f with 1 MPI process See https://petsc.org/release/faq/ hwloc/linux: Ignoring PCI device with non-16bit domain. Pass --enable-32bits-pci-domain to configure to support such devices (warning: it would break the library ABI, don't enable unless really needed). Number of SNES iterations = 3 Completed test examples Error while running make check gmake[1]: *** [makefile:149: check] Error 1 make: *** [GNUmakefile:17: check] Error 2 ++++++++++++++++++++++++++++++++++++++++++++++++

4 4

MSPIN
by Alexander Lindsay 10 Oct '22

10 Oct '22

I know that PETSc has native support for ASPIN. Has anyone tried MSPIN? I wouldn't be surprised if someone has implemented it in user code. Wondering what the barriers would be to creating an option like `-snes_type mspin` ?

2 1

suppress CUDA warning & choose MCA parameter for mpirun during make PETSC_ARCH=arch-linux-c-debug check
by Rob Kudyba 10 Oct '22

10 Oct '22

We are on RHEL 8, using modules that we can load/unload various version of packages/libraries, and I have OpenMPI 4.1.1 with CUDA aware loaded along with GDAL 3.3.0, GCC 10.2.0, and cmake 3.22.1 make PETSC_DIR=/path/to/petsc PETSC_ARCH=arch-linux-c-debug check fails with the below errors, Running check examples to verify correct installation Using PETSC_DIR=/path/to/petsc and PETSC_ARCH=arch-linux-c-debug Possible error running C/C++ src/snes/tutorials/ex19 with 1 MPI process See https://petsc.org/release/faq/ -------------------------------------------------------------------------- The library attempted to open the following supporting CUDA libraries, but each of them failed. CUDA-aware support is disabled. libcuda.so.1: cannot open shared object file: No such file or directory libcuda.dylib: cannot open shared object file: No such file or directory /usr/lib64/libcuda.so.1: cannot open shared object file: No such file or directory /usr/lib64/libcuda.dylib: cannot open shared object file: No such file or directory If you are not interested in CUDA-aware support, then run with --mca opal_warn_on_missing_libcuda 0 to suppress this message. If you are interested in CUDA-aware support, then try setting LD_LIBRARY_PATH to the location of libcuda.so.1 to get passed this issue. -------------------------------------------------------------------------- -------------------------------------------------------------------------- WARNING: There was an error initializing an OpenFabrics device. Local host: g117 Local device: mlx5_0 -------------------------------------------------------------------------- lid velocity = 0.0016, prandtl # = 1., grashof # = 1. Number of SNES iterations = 2 Possible error running C/C++ src/snes/tutorials/ex19 with 2 MPI processes See https://petsc.org/release/faq/ The library attempted to open the following supporting CUDA libraries, but each of them failed. CUDA-aware support is disabled. libcuda.so.1: cannot open shared object file: No such file or directory libcuda.dylib: cannot open shared object file: No such file or directory /usr/lib64/libcuda.so.1: cannot open shared object file: No such file or directory /usr/lib64/libcuda.dylib: cannot open shared object file: No such file or directory If you are not interested in CUDA-aware support, then run with --mca opal_warn_on_missing_libcuda 0 to suppress this message. If you are interested in CUDA-aware support, then try setting LD_LIBRARY_PATH to the locationof libcuda.so.1 to get passed this issue. WARNING: There was an error initializing an OpenFabrics device. Local host: xxx Local device: mlx5_0 lid velocity = 0.0016, prandtl # = 1., grashof # = 1. Number of SNES iterations = 2 [g117:4162783] 1 more process has sent help message help-mpi-common-cuda.txt / dlopen failed [g117:4162783] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages [g117:4162783] 1 more process has sent help message help-mpi-btl-openib.txt / error in device init Completed test examples Error while running make check gmake[1]: *** [makefile:149: check] Error 1 make: *** [GNUmakefile:17: check] Error 2 Where is $MPI_RUN set? I'd like to be able to pass options such as --mca orte_base_help_aggregate 0 --mca opal_warn_on_missing_libcuda 0 -mca pml ucx --mca btl '^openib' which will help me troubleshoot and hide unneeded warnings. Thanks, Rob

5 16

PETSc 2023 User Meeting and New Release 3.18
by Barry Smith 09 Oct '22

09 Oct '22

We are pleased to announce the - next PETSc users meeting, June 5-7, 2023, in Chicago on the campus of IIT (mark your calendars, further information will be forthcoming soon) and the - release of PETSc version 3.18.0 at https://petsc.org/release/install/download/ <https://petsc.org/release/install/download/>. This release includes + far more extensive solvers on GPUs, including an efficient MatSetValuesCOO() for setting matrix values directly on the GPU + improved interfaces to Kokkos, so user GPU code can now be written in Kokkos, CUDA, or HIP. + vastly improved documentation at petsc.org with a more comprehensive user manual and better search capabilities (due to Patrick Sanan) We recommend upgrading to PETSc 3.18.0 soon. As always, please report problems to petsc-maint(a)mcs.anl.gov <mailto:[email protected]> and ask questions at petsc-users(a)mcs.anl.gov <mailto:[email protected]> A list of the major changes and updates can be found at https://petsc.org/release/docs/changes/318 <https://petsc.org/release/docs/changes/318> The final update to petsc-3.17 i.e petsc-3.17.5 is also available This release includes contributions from AdelekeBankole Aidan Hamilton Albert Cowie Alexis Marboeuf Barry Smith Blaise Bourdin Dave May David Andrs David Wells Fande Kong ftrigaux Getnet Betrie Hong Zhang Jacob Faibussowitsch James Wright JDBetteridge Jed Brown Jeremy L Thompson Joe Wallwork Jose Roman Junchao Zhang Justin Chang Kaushik Kulkarni Kerry Key Koki Sagiyama Lawrence Mitchell Lisandro Dalcin Mark Adams Martin Diehl Matthew Knepley Matthew Woehlk Min RK Mr. Hong Zhang Patrick Farrell Patrick Sanan Pierre Jolivet Richard Tran Mills Romain Beucher Satish Balay Scott Kruger Stefano Zampini suyashtn Toby Isaac Todd Munson Umberto Zerbinati Vaclav Hapla Zongze Yang and bug reports/proposed improvements received from Brad Aagaard Abylay Zhumekenov Aidan Hamilton Alfredo J Duarte Gomez Bro H Collins, Eric Benjamin Dudson Ed Bueler Erhan Turan Eric Chamberland Fackler, Philip flw(a)rzg.mpg.de <mailto:[email protected]> Glenn Hammond Henrik Büsing Jacob Simon Merson Jesper Lund-Carlson Jin Chen John Snyder Jose E. Roman Kaustubh Khedkar Lisandro Dalcin Lucas Banting Matthew Knepley Nicolas Berlie Nicolas Tardieu Robert Nourgaliev Olivier Jamond Patrick Sanan Pierre Jolivet Rafel Amer Ramon Randall J LeVeque Richard F. Katz Sanjay Govindjee san.temporal(a)gmail.com <mailto:[email protected]> Sidarth Narayanan Stefano Zampini TAY wee-beng Victor Eijkhout Victoria Hamtiaux Xiaoye S. Li Xu Hui Yang Liu Ye Changqing Zakariae Jorti

1 0

Regarding the status of MatSolve on GPUs
by Sajid Ali 06 Oct '22

06 Oct '22

Hi PETSc-developers, Does PETSc currently provide (either native or third party support) for MatSolve that can be performed entirely on a GPU given a factored matrix? i.e. a direct solver that would store the factors L and U on the device and use the GPU to solve the linear system. It does not matter if the GPU is not used for the factorization as we intend to solve the same linear system for 100s of iterations and thus try to prevent GPU->CPU transfers for the MatSolve phase. Currently, I've built PETSc@main (commit 9c433d, 10/03) with superlu-dist@develop, both of which are configured with CUDA. With this, I'm seeing that each call to PCApply/MatSolve involves one GPU->CPU transfer. Is it possible to avoid this? Thank You, Sajid Ali (he/him) | Research Associate Scientific Computing Division Fermi National Accelerator Laboratory s-sajid-ali.github.io

2 1

PetscLogView produces nan's instead of timing data when using GPUs
by Sajid Ali 06 Oct '22

06 Oct '22

Hi PETSc-developers, I'm having trouble with getting performance logs from an application that uses PETSc. There are no issues when I run it on a CPU, but every time a GPU is used there is no timing data and almost all times are replaced by times that are just `nan` (on two different clusters). I am attaching the log files for both cases with this email. Could someone explain what is happening here ? In case it helps, here are the routines used to initialize/finalize the application that also handle initializing/finalizing PETSc and printing the PETSc performance logs to PETSC_VIEWER_STDOUT_WORLD : https://github.com/fnalacceleratormodeling/synergia2/blob/devel3/src/synerg… Thank You, Sajid Ali (he/him) | Research Associate Scientific Computing Division Fermi National Accelerator Laboratory s-sajid-ali.github.io

2 2