July 2021 - petsc-users - mailman3.cels.anl.gov

download zlib error
by Mark Adams 07 Jul '21

07 Jul '21

I am getting some sort of error in build zlib on Spock at ORNL. Other libraries are downloaded and I am sure the network is fine. Any ideas? Thanks, Mark

4 22

PetSc and CrayPat: MPI assertion errors
by Vijay S Kumar 06 Jul '21

06 Jul '21

Hello all, By way of background, we have a PetSc-based solver that we run on our in-house Cray system. We are carrying out performance analysis using profilers in the CrayPat suite that provide more fine-grained performance-related information than the PetSc log_view summary. When instrumented using CrayPat perftools, it turns out that the MPI initialization (MPI_Init) internally invoked by PetscInitialize is not picked up by the profiler. That is, simply specifying the following: ierr = PetscInitialize(&argc,&argv,(char*)0,NULL);if (ierr) return ierr; results in the following runtime error: CrayPat/X: Version 7.1.1 Revision 7c0ddd79b 08/19/19 16:58:46 Attempting to use an MPI routine before initializing MPICH To circumvent this, we had to explicitly call MPI_Init prior to PetscInitialize: MPI_Init(&argc,&argv); ierr = PetscInitialize(&argc,&argv,(char*)0,NULL);if (ierr) return ierr; However, the side-effect of this above workaround seems to be several downstream runtime (assertion) errors with VecAssemblyBegin/End and MatAssemblyBeing/End statements: CrayPat/X: Version 7.1.1 Revision 7c0ddd79b 08/19/19 16:58:46 main.x: ../rtsum.c:5662: __pat_trsup_trace_waitsome_rtsum: Assertion `recv_count != MPI_UNDEFINED' failed. [email protected]:769 VecAssemblyEnd@0x2aaab951b3ba VecAssemblyEnd_MPI_BTS@0x2aaab950b179 MPI_Waitsome@0x43a238 __pat_trsup_trace_waitsome_rtsum@0x5f1a17 __GI___assert_fail@0x2aaabc61e7d1 __assert_fail_base@0x2aaabc61e759 __GI_abort@0x2aaabc627740 __GI_raise@0x2aaabc626160 Interestingly, we do not see such errors when there is no explicit MPI_Init, and no instrumentation for performance. Looking for someone to help throw more light on why PetSc Mat/Vec AssemblyEnd statements lead to such MPI-level assertion errors in cases where MPI_Init is explicitly called. (Or alternatively, is there a way to call PetscInitialize in a manner that ensures that the MPI initialization is picked up by the profilers in question?) We would highly appreciate any help/pointers, Thanks! Vijay

4 3

Kokkos and HIP
by Mark Adams 06 Jul '21

06 Jul '21

I am getting a make error on Spock at ORNL with Kokkos and HIP. This was building last week and this was a clean build. Thanks, Mark

3 3

HDF5 DM and VecView with MPI :: Crash
by Thibault Bridel-Bertomeu 06 Jul '21

06 Jul '21

Dear all, I keep having this error on one of the supercomputers I have access to : [1]PETSC ERROR: The EXACT line numbers in the error traceback are not available. [1]PETSC ERROR: instead the line number of the start of the function is given. [1]PETSC ERROR: #1 H5Dcreate2() at /ccc/work/cont001/ocre/bridelbert/05-PETSC/src/vec/vec/impls/mpi/pdvec.c:690 [1]PETSC ERROR: #2 VecView_MPI_HDF5() at /ccc/work/cont001/ocre/bridelbert/05-PETSC/src/vec/vec/impls/mpi/pdvec.c:594 [1]PETSC ERROR: #3 VecView_MPI() at /ccc/work/cont001/ocre/bridelbert/05-PETSC/src/vec/vec/impls/mpi/pdvec.c:787 [1]PETSC ERROR: #4 VecView() at /ccc/work/cont001/ocre/bridelbert/05-PETSC/src/vec/vec/interface/vector.c:576 [1]PETSC ERROR: #5 DMPlexCoordinatesView_HDF5_Internal() at /ccc/work/cont001/ocre/bridelbert/05-PETSC/src/dm/impls/plex/plexhdf5.c:560 [1]PETSC ERROR: #6 DMPlexView_HDF5_Internal() at /ccc/work/cont001/ocre/bridelbert/05-PETSC/src/dm/impls/plex/plexhdf5.c:802 [1]PETSC ERROR: #7 DMView_Plex() at /ccc/work/cont001/ocre/bridelbert/05-PETSC/src/dm/impls/plex/plex.c:1366 [1]PETSC ERROR: #8 DMView() at /ccc/work/cont001/ocre/bridelbert/05-PETSC/src/dm/interface/dm.c:954 [1]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- The configure options are as follow : [1]PETSC ERROR: Configure options --with-clean=1 --prefix=/ccc/work/cont001/ocre/bridelbert/05-PETSC/build_uns3D_inti --with-make-np=8 --with-windows-graphics=0 --with-debugging=1 --download-mpich-shared=0 --with-x=0 --with-pthread=0 --with-valgrind=0 --PETSC_ARCH=INTI_UNS3D --with-fc=/ccc/products/openmpi-4.0.3/gcc--8.3.0/default/bin/mpifort --with-cc=/ccc/products/openmpi-4.0.3/gcc--8.3.0/default/bin/mpicc --with-cxx=/ccc/products/openmpi-4.0.3/gcc--8.3.0/default/bin/mpicxx --with-openmp=0 --download-sowing=/ccc/work/cont001/ocre/bridelbert/v1.1.26-p2.tar.gz --download-metis=/ccc/work/cont001/ocre/bridelbert/git.metis.tar.gz --download-parmetis=/ccc/work/cont001/ocre/bridelbert/git.parmetis.tar.gz --download-fblaslapack=/ccc/work/cont001/ocre/bridelbert/git.fblaslapack.tar.gz --with-cmake-dir=/ccc/products/cmake-3.13.3/system/default --download-hdf5=/ccc/work/cont001/ocre/bridelbert/hdf5-1.12.0.tar.bz2 --download-netcdf=/ccc/work/cont001/ocre/bridelbert/netcdf-4.5.0.tar.gz --download-pnetcdf=/ccc/work/cont001/ocre/bridelbert/pnetcdf-1.12.1.tar.gz --download-exodusii=/ccc/work/cont001/ocre/bridelbert/v2021-01-20.tar.gz --download-zlib=/ccc/work/cont001/ocre/bridelbert/zlib-1.2.11.tar.gz The piece of code that is responsible is that one : call PetscViewerHDF5Open(PETSC_COMM_WORLD, "debug_initmesh.h5", FILE_MODE_WRITE, hdf5Viewer, ierr); CHKERRA(ierr) call PetscViewerPushFormat(hdf5Viewer, PETSC_VIEWER_HDF5_XDMF, ierr); CHKERRA(ierr) call DMView(dm, hdf5Viewer, ierr); CHKERRA(ierr) call PetscViewerPopFormat(hdf5Viewer, ierr); CHKERRA(ierr) call PetscViewerDestroy(hdf5Viewer, ierr); CHKERRA(ierr) I tried with gcc, intel compiler, openmpi 2.x.x or openmpi 4.x.x ... same problems ... can anyone please advise ? It's starting to make me quite crazy ... x( Thank you !!! Thibault

2 2

Scatter parallel Vec to sequential Vec on non-zeroth process
by Peder Jørgensgaard Olesen 04 Jul '21

04 Jul '21

Hello I have a parallel vector, and I'd like to copy its data to a sequential vector located on a given process that is not necessarily the 0th process, so using VecScatterCreateToZero wouldn't do here. How might one approach this? Med venlig hilsen / Best Regards Peder Jørgensgaard Olesen PhD Student, Turbulence Research Lab Dept. of Mechanical Engineering Technical University of Denmark Koppels Allé Bygning 403, Rum 105 DK-2800 Kgs. Lyngby

4 15

PETSc with Julia Binary Builder
by Kozdon, Jeremy (CIV) 02 Jul '21

02 Jul '21

I have been talking with Boris Kaus and Patrick Sanan about trying to revive the Julia PETSc interface wrappers. One of the first things to get going is to use Julia's binary builder [1] to wrap more scalar, real, and int type builds of the PETSc library; the current distribution is just Real, double, Int32. I've been working on a PR for this [2] but have been running into some build issues on some architectures [3]. I doubt that anyone here is an expert with Julia's binary builder system, but I was wondering if anyone who is better with the PETSc build system can see anything obvious from the configure.log [4] that might help me sort out what's going on. This exact script worked on 2020-08-20 [5] to build the libraries, se something has obviously changed with either the Julia build system and/or one (or more!) of the dependency binaries. For those that don't know, Julia's binary builder system essentially allows users to download binaries directly from the web for any system that the Julia Programing language distributes binaries for. So a (desktop) user can get MPI, PETSc, etc. without the headache of having to build anything from scratch; obviously on clusters you would still want to use system MPIs and what not. ---- [1] https://github.com/JuliaPackaging/BinaryBuilder.jl [2] https://github.com/JuliaPackaging/Yggdrasil/pull/3249 [3] https://github.com/JuliaPackaging/Yggdrasil/pull/3249#issuecomment-872698681 [4] https://gist.github.com/jkozdon/c161fb15f2df23c3fbc0a5a095887ef8#file-confi… [5] https://github.com/JuliaBinaryWrappers/PETSc_jll.jl/releases/tag/PETSc-v3.1…

6 11

MatNest with Shell blocks for multipysics
by Matteo Semplice 01 Jul '21

01 Jul '21

Hi. We are designing a PETSc application that will employ a SNES solver on a multiphysics problem whose jacobian will have a 2x2 block form, say A=[A00,A01;A10,A11]. We already have code for the top left block A_00 (a MatShell and a related Shell preconditioner) that we wish to reuse. We could implement the other blocks as Shells or assembled matrices. We'd like also to compare our method with existing ones, so we'd like to be quite flexible in the choice of KSP and PC within the SNES. (To this end, implementing an assembled version of the A00 and the other blocks would be easy) I am assuming that, in order to have one or more shell blocks, the full jacobian should be a nested matrix, and I am wondering what is the best way to design the code. We are going to use DMDA's to manipulate Vecs for both variable sets, so the DMComposite approach of https://www.mcs.anl.gov/petsc/petsc-current/src/snes/tutorials/ex28.c.html is intriguing, but I have read in the comments that it has issues with MatNest type. My next guess would be to create the four submatrices ahead and then insert them in a MatNest, like in the Stokes example of https://www.mcs.anl.gov/petsc/petsc-current/src/snes/tutorials/ex70.c.html. However, in order to have shell blocks I guess it is almost mandatory to have the matrix partitioned among cpus as the Vecs are and I don't understand how Vecs end up being partitioned in ex70. We could - create a DMComposite and create the Vecs with it - get the local sizes of the Vecs and subVecs for the two variable groups - create the matrix as in ex70, using the shell type where/when needed, but instead of MatSetSizes(matblock, NULL, NULL, globalRows, globalCols) call MatSetSizes(matblock, localRows, localCols, NULL, NULL) using the local sizes of the subvectors. Does this sound a viable approach? Or do you have some different suggestions? Thanks Matteo

4 6

Visualization and I/O
by Alfredo J Duarte Gomez 30 Jun '21

30 Jun '21

Good evening, My group and I are in the process of developing an application using PETSc, and we are currently planning and figuring out the best objects to do this. I currently have some questions about visualization and I/O. Some of the most important features that we want in the process of developing this code is the ability for easy checkpoint/restarts and visualization. As we are working with a time dependent PDE, I have stumbled upon the "ts" and "tstrajectory" objects that seem to have some very helpful features. However, I am having some trouble extracting information out of the the "tstrajectory" object in an HDF5 format. I initially thought something like this would work, where tj was obtained using TSGetTrajectory(ts,&tj) after a successful solve of the ts object: PetscViewer viewer; PetscViewerHDF5Open(PETSC_COMM_WORLD,"output.h5",FILE_MODE_WRITE,&viewer);CHKERRQ(ierr); TSTrajectoryView(tj,viewer);CHKERRQ(ierr); PetscViewerDestroy(&viewer);CHKERRQ(ierr); However, the output file comes out empty. Do I have to extract the solutions one by one? How would I do that? Does the HDF5 viewer only work with vector objects or can it be generalized to other objects? I am trying to maintain the highest degree of abstraction possible (i.e. multiple 2d ode solutions in one hdf5 file). This seems favorable to me because it would be easy to use the HDF5 file for postprocessing/visualization and restarts, but let me know if you think this is a bad idea and it would be better to just use the HDF5 for output/postprocess and use native PETSc binary files for restarts. Additionally, as I move towards large visualizations, I would like to use Paraview. I saw that the best way of doing this is to use the petsc_gen_xdmf.py script on an HDF5 solution file. Is this still the best way for PETSc visualization with Paraview? Thank you, -- Alfredo Duarte Graduate Research Assistant The University of Texas at Austin

2 1

Boundary ghost cell types in PETSc 3.15
by Adrian Croucher 30 Jun '21

30 Jun '21

hi I've been testing building my code with PETSc 3.15 and have hit a weird problem. If I create a DMPlex and add boundary ghost cells using DMPlexConstructGhostCells(), I expect the boundary ghost cells to have cell type DM_POLYTOPE_FV_GHOST, if I check the cell types using DMPlexGetCellTypeLabel() and DMLabelGetValue(). That is true if I build using PETSc 3.14. But with 3.15 it appears the boundary ghost cells are ending up with cell type DM_POLYTOPE_INTERIOR_GHOST. These types are defined in include/petscdmtypes.h and currently DM_POLYTOPE_FV_GHOST = 11 and DM_POLYTOPE_INTERIOR_GHOST = 12. I've attached a minimal example program in Fortran. It creates a box mesh and adds boundary ghost cells, then counts how many have type DM_POLYTOPE_FV_GHOST. With PETSc 3.14 it is 24, the same as the number of ghost cells. With PETSc 3.15 it is zero, and the boundary ghost cells all have type 12. Any clues? Regards, Adrian -- Dr Adrian Croucher Senior Research Fellow Department of Engineering Science University of Auckland, New Zealand email: a.croucher(a)auckland.ac.nz tel: +64 (0)9 923 4611

2 3