January 2022 - petsc-users - mailman3.cels.anl.gov

How to combine different element types into a single DMPlex?
by Eric Chamberland 01 Aug '24

01 Aug '24

Hi, while playing with DMPlexBuildFromCellListParallel, I noticed we have to specify "numCorners" which is a fixed value, then gives a fixed number of nodes for a series of elements. How can I then add, for example, triangles and quadrangles into a DMPlex? Thanks, Eric -- Eric Chamberland, ing., M. Ing Professionnel de recherche GIREF/Université Laval (418) 656-2131 poste 41 22 42

4 13

HDF5 corruption
by Adrian Croucher 13 May '24

13 May '24

hi, One of the users of my PETSc-based code has reported that HDF5 output files can be corrupted and unusable if e.g. the run is killed. I've just done a bit of reading about this and it appears to be a known issue with HDF5. Some people suggest flushing the HDF5 file periodically to help prevent data loss. I had a look at PetscViewerFlush() but it doesn't seem to be implemented for the HDF5 viewer- is that correct? I am currently calling PetscViewerHDF5Open() once at the start of the run and closing it at the end. Some other people suggest doing this before and after each write instead. Is there likely to be a significant performance penalty in doing that? I gather HDF5 journaling has been promised for a while to get around this problem but as far as I can see it hasn't materialised yet... Regards, Adrian -- Dr Adrian Croucher Senior Research Fellow Department of Engineering Science University of Auckland, New Zealand email: a.croucher(a)auckland.ac.nz tel: +64 (0)9 923 4611

4 17

Nodes coordinates in distributed dmplex
by Alejandro D Otero 16 Jul '22

16 Jul '22

Hi, sorry if this is an obvious question, but I cannot figure out how to recover finite element nodes coordinates once I have distributed a mesh stored as a dmplex. I am using petsc4py as interface to petsc rutines. I first created a dmplex using: dm.createFromCellList() In a sequential run I got the coordinates with: Coords = dm.getCoordinates() which gave a sequential vector with the coordinates of the mesh nodes. When I distribute the mesh with: dm.distribute() each mpi process has it own dm but the indexing of the vector resulting from getCoordinates() or getCoordinatesLocal() seems not consistent with the local numbering of the cells and nodes. Which is the correct way of doing this in PETSc philosophy? Thanks in advance, Alejandro

4 9

DMView and DMLoad
by Berend van Wachem 09 Mar '22

09 Mar '22

Dear PETSc-team, We are trying to save and load distributed DMPlex and its associated physical fields (created with DMCreateGlobalVector) (Uvelocity, VVelocity, ...) in HDF5_XDMF format. To achieve this, we do the following: 1) save in the same xdmf.h5 file: DMView( DM , H5_XDMF_Viewer ); VecView( UVelocity, H5_XDMF_Viewer ); 2) load the dm: DMPlexCreateFromfile(PETSC_COMM_WORLD, Filename, PETSC_TRUE, DM); 3) load the physical field: VecLoad( UVelocity, H5_XDMF_Viewer ); There are no errors in the execution, but the loaded DM is distributed differently to the original one, which results in the incorrect placement of the values of the physical fields (UVelocity etc.) in the domain. This approach is used to restart the simulation with the last saved DM. Is there something we are missing, or there exists alternative routes to this goal? Can we somehow get the IS of the redistribution, so we can re-distribute the vector data as well? Many thanks, best regards, Berend.

5 13

HDF5 timestepping in PETSc 3.16
by Adrian Croucher 28 Feb '22

28 Feb '22

hi I am just testing out PETSc 3.16 and making the necessary changes to my code. Amongst other things I now have to add a PetscViewerHDF5PushTimestepping() call before starting to output time-dependent results to HDF5 using a PetscViewer. I now also have to add this call before reading in sets of previously computed time-dependent results (for restarting a simulation from the results of a previous run). The problem with this is that if I try to read in the results of any previous run, computed with an earlier version of PETSc (< 3.16), an error is raised because the time-dependent datasets in the file do not have the 'timestepping' attribute. Is there something else I need to do to make this work? - Adrian -- Dr Adrian Croucher Senior Research Fellow Department of Engineering Science University of Auckland, New Zealand email: a.croucher(a)auckland.ac.nz tel: +64 (0)9 923 4611

3 5

NVIDIA HPC SDK and complex data type
by Jonathan D. Halverson 24 Feb '22

24 Feb '22

Hello, We are unable to build PETSc using the NVIDIA HPC SDK and --with-scalar-type=complex. Below is our procedure: $ module load nvhpc/21.11 $ module load openmpi/nvhpc-21.11/4.1.2/64 $ git clone -b release https://gitlab.com/petsc/petsc.git petsc; cd petsc $ ./configure --with-debugging=1 --with-scalar-type=complex PETSC_ARCH=openmpi-power $ make PETSC_DIR=/home/$USER/software/petsc PETSC_ARCH=openmpi-power all $ make PETSC_DIR=/home/$USER/software/petsc PETSC_ARCH=openmpi-power check "make check" fails with a segmentation fault when running ex19. The fortran test ex5f passes. The procedure above fails on x86_64 and POWER both running RHEL8. It also fails using nvhpc 20.7. The procedure above works for "real" instead of "complex". A "hello world" MPI code using a complex data type works with our nvhpc modules. The procedure above works successfully when GCC and an Open MPI library built using GCC is used. The only trouble is the combination of PETSc with nvhpc and complex. Any known issues? The build log for the procedure above is here: https://tigress-web.princeton.edu/~jdh4/petsc_nvhpc_complex_17dec2021.log Jon

4 7

PCSetCoordinates does not set coordinates of sub PC (fieldsplit) objects
by Nicolás Barnafi 21 Feb '22

21 Feb '22

Dear community, I am working on a block preconditioner, where one of the blocks uses HYPRE's AMS. As it requires the coordinates of the dofs, I have done so to the PC object. I expected the coordinates to be inherited down to the subblocks, is this not the case? (it seems so as I couldn't find a specialized FIELDSPLIT SetCoordinates function). If this feature is missing, please give me some hints on where to add the missing function, I would gladly do it. If not, please let me know why it was dismissed, in order to do things the hard way [as in hard-coded ;)]. Kind regards, Nicolas

3 7

MatPreallocatorPreallocate segfault with PETSC 3.16
by Marius Buerkle 03 Feb '22

03 Feb '22

Hi, I try to use MatPreallocatorPreallocate to allocate a MATMPIAIJ matrix A . I define the MATPREALLOCATOR preM with MatSetValues and then call MatPreallocatorPreallocate to get A. This works on the first call to MatPreallocatorPreallocate, but if I call MatPreallocatorPreallocate again with the same preM to get another matrix B then I get a segfault, although the program continues to run (see below). It worked with PETSC 3.15 but with 3.16 I stopped working. When I check mat_info_nz_allocated and mat_info_nz_used for the allocated matrix it looks correct for the first call, but on the second call mat_info_nz_used is 0. I also attached a minimal example. [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: Null argument, when expecting valid pointer [1]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: Null Pointer: Parameter # 1 [0]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting. [0]PETSC ERROR: [1]PETSC ERROR: Null argument, when expecting valid pointer [1]PETSC ERROR: Petsc Development GIT revision: v3.16.3-686-g5e81a90 GIT Date: 2022-01-23 05:13:26 +0000 [0]PETSC ERROR: ./prem_test on a named cd001 by cdfmat_marius Mon Jan 24 18:21:17 2022 [0]PETSC ERROR: Null Pointer: Parameter # 1 [1]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting. [1]PETSC ERROR: Configure options --prefix=/home/cdfmat_marius/prog/petsc/petsc_main_dbg --with-scalar-type=complex --with-fortran-kernels=1 --with-64-bit-indices=0 --CC=mpiicc --COPTFLAGS="-g -traceback" --CXX=mpiicpc --CXXOPTFLAGS="-g -traceback" --FC=mpiifort --FOPTFLAGS="-g -traceback" --with-mpi=1 --with-x=0 --with-cuda=0 --download-parmetis=/home/cdfmat_marius/prog/petsc/git/petsc_main/externalpackages/git.parmetis.tar.gz --download-parmetis-commit=HEAD --download-metis=/home/cdfmat_marius/prog/petsc/git/petsc_main/externalpackages/git.metis.tar.gz --download-metis-commit=HEAD --download-slepc=/home/cdfmat_marius/prog/petsc/git/petsc_main/externalpackages/git.slepc_main.tar.gz --download-slepc-commit=HEAD --download-superlu_dist=/home/cdfmat_marius/prog/petsc/git/petsc_main/externalpackages/git.superlu_dist.tar.gz --download-superlu_dist-commit=HEAD --download-mumps=/home/cdfmat_marius/prog/petsc/git/petsc_main/externalpackages/git.mumps.tar.gz --download-mumps-commit=HEAD --download-hypre=/home/cdfmat_marius/prog/petsc/git/petsc_main/externalpackages/git.hypre.tar.gz --download-hypre-commit=HEAD --download-hwloc=/home/cdfmat_marius/prog/petsc/git/petsc_main/externalpackages/hwloc-2.5.0.tar.gz --download-sowing=/home/cdfmat_marius/prog/petsc/git/petsc_main/externalpackages/git.sowing.tar.gz --download-elemental=/home/cdfmat_marius/prog/petsc/git/petsc_main/externalpackages/git.elemental.tar.gz --download-elemental-commit=HEAD --download-make=/home/cdfmat_marius/prog/petsc/git/petsc_main/externalpackages/make-4.2.1-6.fc28.tar.gz --download-ptscotch=/home/cdfmat_marius/prog/petsc/git/petsc_main/externalpackages/git.ptscotch.tar.gz --download-ptscotch-commit=HEAD --with-openmp=0 --with-pthread=0 --with-cxx-dialect=c++11 --with-debugging=1 --with-cuda=0 --with-cudac=0 --with-valgrind=0 --with-blaslapack-lib="-mkl=sequential -lmkl_scalapack_lp64 -lmkl_blacs_intelmpi_lp64 -lpthread -lm -ldl" --with-scalapack-lib="-mkl=sequential -lmkl_scalapack_lp64 -lmkl_blacs_intelmpi_lp64 -lpthread -lm -ldl" --with-mkl_pardiso-dir=/home/appli/intel/compilers_and_libraries_2020.4.304/linux/mkl --with-mkl_cpardiso-dir=/home/appli/intel/compilers_and_libraries_2020.4.304/linux/mkl --with-mkl_sparse-dir=/home/appli/intel/compilers_and_libraries_2020.4.304/linux/mkl --with-mkl_sparse_optimize-dir=/home/appli/intel/compilers_and_libraries_2020.4.304/linux/mkl [0]PETSC ERROR: Petsc Development GIT revision: v3.16.3-686-g5e81a90 GIT Date: 2022-01-23 05:13:26 +0000 [1]PETSC ERROR: ./prem_test on a named cd001 by cdfmat_marius Mon Jan 24 18:21:17 2022 [1]PETSC ERROR: #1 PetscHSetIJGetSize() at /home/cdfmat_marius/prog/petsc/git/petsc_main/include/petsc/private/hashsetij.h:13 [0]PETSC ERROR: #2 MatPreallocatorPreallocate_Preallocator() at /home/cdfmat_marius/prog/petsc/git/petsc_main/src/mat/impls/preallocator/matpreallocator.c:165 [0]PETSC ERROR: #3 MatPreallocatorPreallocate() at /home/cdfmat_marius/prog/petsc/git/petsc_main/src/mat/impls/preallocator/matpreallocator.c:234 Configure options --prefix=/home/cdfmat_marius/prog/petsc/petsc_main_dbg --with-scalar-type=complex --with-fortran-kernels=1 --with-64-bit-indices=0 --CC=mpiicc --COPTFLAGS="-g -traceback" --CXX=mpiicpc --CXXOPTFLAGS="-g -traceback" --FC=mpiifort --FOPTFLAGS="-g -traceback" --with-mpi=1 --with-x=0 --with-cuda=0 --download-parmetis=/home/cdfmat_marius/prog/petsc/git/petsc_main/externalpackages/git.parmetis.tar.gz --download-parmetis-commit=HEAD --download-metis=/home/cdfmat_marius/prog/petsc/git/petsc_main/externalpackages/git.metis.tar.gz --download-metis-commit=HEAD --download-slepc=/home/cdfmat_marius/prog/petsc/git/petsc_main/externalpackages/git.slepc_main.tar.gz --download-slepc-commit=HEAD --download-superlu_dist=/home/cdfmat_marius/prog/petsc/git/petsc_main/externalpackages/git.superlu_dist.tar.gz --download-superlu_dist-commit=HEAD --download-mumps=/home/cdfmat_marius/prog/petsc/git/petsc_main/externalpackages/git.mumps.tar.gz --download-mumps-commit=HEAD --download-hypre=/home/cdfmat_marius/prog/petsc/git/petsc_main/externalpackages/git.hypre.tar.gz --download-hypre-commit=HEAD --download-hwloc=/home/cdfmat_marius/prog/petsc/git/petsc_main/externalpackages/hwloc-2.5.0.tar.gz --download-sowing=/home/cdfmat_marius/prog/petsc/git/petsc_main/externalpackages/git.sowing.tar.gz --download-elemental=/home/cdfmat_marius/prog/petsc/git/petsc_main/externalpackages/git.elemental.tar.gz --download-elemental-commit=HEAD --download-make=/home/cdfmat_marius/prog/petsc/git/petsc_main/externalpackages/make-4.2.1-6.fc28.tar.gz --download-ptscotch=/home/cdfmat_marius/prog/petsc/git/petsc_main/externalpackages/git.ptscotch.tar.gz --download-ptscotch-commit=HEAD --with-openmp=0 --with-pthread=0 --with-cxx-dialect=c++11 --with-debugging=1 --with-cuda=0 --with-cudac=0 --with-valgrind=0 --with-blaslapack-lib="-mkl=sequential -lmkl_scalapack_lp64 -lmkl_blacs_intelmpi_lp64 -lpthread -lm -ldl" --with-scalapack-lib="-mkl=sequential -lmkl_scalapack_lp64 -lmkl_blacs_intelmpi_lp64 -lpthread -lm -ldl" --with-mkl_pardiso-dir=/home/appli/intel/compilers_and_libraries_2020.4.304/linux/mkl --with-mkl_cpardiso-dir=/home/appli/intel/compilers_and_libraries_2020.4.304/linux/mkl --with-mkl_sparse-dir=/home/appli/intel/compilers_and_libraries_2020.4.304/linux/mkl --with-mkl_sparse_optimize-dir=/home/appli/intel/compilers_and_libraries_2020.4.304/linux/mkl [1]PETSC ERROR: #1 PetscHSetIJGetSize() at /home/cdfmat_marius/prog/petsc/git/petsc_main/include/petsc/private/hashsetij.h:13 [1]PETSC ERROR: #2 MatPreallocatorPreallocate_Preallocator() at /home/cdfmat_marius/prog/petsc/git/petsc_main/src/mat/impls/preallocator/matpreallocator.c:165 [1]PETSC ERROR: #3 MatPreallocatorPreallocate() at /home/cdfmat_marius/prog/petsc/git/petsc_main/src/mat/impls/preallocator/matpreallocator.c:234 Best and Thanks, Marius

4 12

Re: [petsc-users] Cannot eagerly initialize cuda, as doing so results in cuda error 35 (cudaErrorInsufficientDriver) : CUDA driver version is insufficient for CUDA runtime version
by Fande Kong 31 Jan '22

31 Jan '22

On Tue, Jan 25, 2022 at 9:04 AM Jacob Faibussowitsch <jacob.fai(a)gmail.com> wrote: > Configure should not have an impact here I think. The reason I had you run > `cudaGetDeviceCount()` is because this is the CUDA call (and in fact the > only CUDA call) in the initialization sequence that returns the error code. > There should be no prior CUDA calls. Maybe this is a problem with > oversubscribing GPU’s? In the runs that crash, how many ranks are using any > given GPU at once? Maybe MPS is required. > I used one MPI rank. Fande > > Best regards, > > Jacob Faibussowitsch > (Jacob Fai - booss - oh - vitch) > > On Jan 21, 2022, at 12:01, Fande Kong <fdkong.jd(a)gmail.com> wrote: > > Thanks Jacob, > > On Thu, Jan 20, 2022 at 6:25 PM Jacob Faibussowitsch <jacob.fai(a)gmail.com> > wrote: > >> Segfault is caused by the following check at >> src/sys/objects/device/impls/cupm/cupmdevice.cxx:349 being a >> PetscUnlikelyDebug() rather than just PetscUnlikely(): >> >> ``` >> if (PetscUnlikelyDebug(_defaultDevice < 0)) { // _defaultDevice is in >> fact < 0 here and uncaught >> ``` >> >> To clarify: >> >> “lazy” initialization is not that lazy after all, it still does some 50% >> of the initialization that “eager” initialization does. It stops short >> initializing the CUDA runtime, checking CUDA aware MPI, gathering device >> data, and initializing cublas and friends. Lazy also importantly swallows >> any errors that crop up during initialization, storing the resulting error >> code for later (specifically _defaultDevice = -init_error_value;). >> >> So whether you initialize lazily or eagerly makes no difference here, as >> _defaultDevice will always contain -35. >> >> The bigger question is why cudaGetDeviceCount() is returning >> cudaErrorInsufficientDriver. Can you compile and run >> >> ``` >> #include <cuda_runtime.h> >> >> int main() >> { >> int ndev; >> return cudaGetDeviceCount(&ndev): >> } >> ``` >> >> Then show the value of "echo $?”? >> > > Modify your code a little to get more information. > > #include <cuda_runtime.h> > #include <cstdio> > > int main() > { > int ndev; > int error = cudaGetDeviceCount(&ndev); > printf("ndev %d \n", ndev); > printf("error %d \n", error); > return 0; > } > > Results: > > $ ./a.out > ndev 4 > error 0 > > > I have not read the PETSc cuda initialization code yet. If I need to guess > at what was happening. I will naively think that PETSc did not get correct > GPU information in the configuration because the compiler node does not > have GPUs, and there was no way to get any GPU device information. > > During the runtime on GPU nodes, PETSc might have incorrect information > grabbed during configuration and had this kind of false error message. > > Thanks, > > Fande > > > >> >> Best regards, >> >> Jacob Faibussowitsch >> (Jacob Fai - booss - oh - vitch) >> >> On Jan 20, 2022, at 17:47, Matthew Knepley <knepley(a)gmail.com> wrote: >> >> On Thu, Jan 20, 2022 at 6:44 PM Fande Kong <fdkong.jd(a)gmail.com> wrote: >> >>> Thanks, Jed >>> >>> On Thu, Jan 20, 2022 at 4:34 PM Jed Brown <jed(a)jedbrown.org> wrote: >>> >>>> You can't create CUDA or Kokkos Vecs if you're running on a node >>>> without a GPU. >>> >>> >>> I am running the code on compute nodes that do have GPUs. >>> >> >> If you are actually running on GPUs, why would you need lazy >> initialization? It would not break with GPUs present. >> >> Matt >> >> >>> With PETSc-3.16.1, I got good speedup by running GAMG on GPUs. That >>> might be a bug of PETSc-main. >>> >>> Thanks, >>> >>> Fande >>> >>> >>> >>> KSPSetUp 13 1.0 6.4400e-01 1.0 2.02e+09 1.0 0.0e+00 0.0e+00 >>> 0.0e+00 0 5 0 0 0 0 5 0 0 0 3140 64630 15 1.05e+02 5 >>> 3.49e+01 100 >>> KSPSolve 1 1.0 1.0109e+00 1.0 3.49e+10 1.0 0.0e+00 0.0e+00 >>> 0.0e+00 0 87 0 0 0 0 87 0 0 0 34522 69556 4 4.35e-03 1 >>> 2.38e-03 100 >>> KSPGMRESOrthog 142 1.0 1.2674e-01 1.0 1.06e+10 1.0 0.0e+00 0.0e+00 >>> 0.0e+00 0 27 0 0 0 0 27 0 0 0 83755 87801 0 0.00e+00 0 >>> 0.00e+00 100 >>> SNESSolve 1 1.0 4.4402e+01 1.0 4.00e+10 1.0 0.0e+00 0.0e+00 >>> 0.0e+00 21100 0 0 0 21100 0 0 0 901 51365 57 1.10e+03 52 >>> 8.78e+02 100 >>> SNESSetUp 1 1.0 3.9101e-05 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 >>> 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 0 0 0.00e+00 0 >>> 0.00e+00 0 >>> SNESFunctionEval 2 1.0 1.7097e+01 1.0 1.60e+07 1.0 0.0e+00 0.0e+00 >>> 0.0e+00 8 0 0 0 0 8 0 0 0 0 1 0 0 0.00e+00 6 >>> 1.92e+02 0 >>> SNESJacobianEval 1 1.0 1.6213e+01 1.0 2.80e+07 1.0 0.0e+00 0.0e+00 >>> 0.0e+00 8 0 0 0 0 8 0 0 0 0 2 0 0 0.00e+00 1 >>> 3.20e+01 0 >>> SNESLineSearch 1 1.0 8.5582e+00 1.0 1.24e+08 1.0 0.0e+00 0.0e+00 >>> 0.0e+00 4 0 0 0 0 4 0 0 0 0 14 64153 1 3.20e+01 3 >>> 9.61e+01 94 >>> PCGAMGGraph_AGG 5 1.0 3.0509e+00 1.0 8.19e+07 1.0 0.0e+00 0.0e+00 >>> 0.0e+00 1 0 0 0 0 1 0 0 0 0 27 0 5 3.49e+01 9 >>> 7.43e+01 0 >>> PCGAMGCoarse_AGG 5 1.0 3.8711e+00 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 >>> 0.0e+00 2 0 0 0 0 2 0 0 0 0 0 0 0 0.00e+00 0 >>> 0.00e+00 0 >>> PCGAMGProl_AGG 5 1.0 7.0748e-01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 >>> 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 0 0 0.00e+00 0 >>> 0.00e+00 0 >>> PCGAMGPOpt_AGG 5 1.0 1.2904e+00 1.0 2.14e+09 1.0 0.0e+00 0.0e+00 >>> 0.0e+00 1 5 0 0 0 1 5 0 0 0 1661 29807 26 7.15e+02 20 >>> 2.90e+02 99 >>> GAMG: createProl 5 1.0 8.9489e+00 1.0 2.22e+09 1.0 0.0e+00 0.0e+00 >>> 0.0e+00 4 6 0 0 0 4 6 0 0 0 249 29666 31 7.50e+02 29 >>> 3.64e+02 96 >>> Graph 10 1.0 3.0478e+00 1.0 8.19e+07 1.0 0.0e+00 >>> 0.0e+00 0.0e+00 1 0 0 0 0 1 0 0 0 0 27 0 5 >>> 3.49e+01 9 7.43e+01 0 >>> MIS/Agg 5 1.0 4.1290e-01 1.0 0.00e+00 0.0 0.0e+00 >>> 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 0 0 >>> 0.00e+00 0 0.00e+00 0 >>> SA: col data 5 1.0 1.9127e-02 1.0 0.00e+00 0.0 0.0e+00 >>> 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 0 0 >>> 0.00e+00 0 0.00e+00 0 >>> SA: frmProl0 5 1.0 6.2662e-01 1.0 0.00e+00 0.0 0.0e+00 >>> 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 0 0 >>> 0.00e+00 0 0.00e+00 0 >>> SA: smooth 5 1.0 4.9595e-01 1.0 1.21e+08 1.0 0.0e+00 >>> 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 244 2709 15 >>> 1.97e+02 15 2.55e+02 90 >>> GAMG: partLevel 5 1.0 4.7330e-01 1.0 6.98e+08 1.0 0.0e+00 0.0e+00 >>> 0.0e+00 0 2 0 0 0 0 2 0 0 0 1475 4120 5 1.78e+02 10 >>> 2.55e+02 100 >>> PCGAMG Squ l00 1 1.0 2.6027e+00 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 >>> 0.0e+00 1 0 0 0 0 1 0 0 0 0 0 0 0 0.00e+00 0 >>> 0.00e+00 0 >>> PCGAMG Gal l00 1 1.0 3.8406e-01 1.0 5.48e+08 1.0 0.0e+00 0.0e+00 >>> 0.0e+00 0 1 0 0 0 0 1 0 0 0 1426 4270 1 1.48e+02 2 >>> 2.11e+02 100 >>> PCGAMG Opt l00 1 1.0 2.4932e-01 1.0 7.20e+07 1.0 0.0e+00 0.0e+00 >>> 0.0e+00 0 0 0 0 0 0 0 0 0 0 289 2653 1 6.41e+01 1 >>> 1.13e+02 100 >>> PCGAMG Gal l01 1 1.0 6.6279e-02 1.0 1.09e+08 1.0 0.0e+00 0.0e+00 >>> 0.0e+00 0 0 0 0 0 0 0 0 0 0 1645 3851 1 2.40e+01 2 >>> 3.64e+01 100 >>> PCGAMG Opt l01 1 1.0 2.9544e-02 1.0 7.15e+06 1.0 0.0e+00 0.0e+00 >>> 0.0e+00 0 0 0 0 0 0 0 0 0 0 242 1671 1 4.84e+00 1 >>> 1.23e+01 100 >>> PCGAMG Gal l02 1 1.0 1.8874e-02 1.0 3.72e+07 1.0 0.0e+00 0.0e+00 >>> 0.0e+00 0 0 0 0 0 0 0 0 0 0 1974 3636 1 5.04e+00 2 >>> 6.58e+00 100 >>> PCGAMG Opt l02 1 1.0 7.4353e-03 1.0 2.40e+06 1.0 0.0e+00 0.0e+00 >>> 0.0e+00 0 0 0 0 0 0 0 0 0 0 323 1457 1 7.71e-01 1 >>> 2.30e+00 100 >>> PCGAMG Gal l03 1 1.0 2.8479e-03 1.0 4.10e+06 1.0 0.0e+00 0.0e+00 >>> 0.0e+00 0 0 0 0 0 0 0 0 0 0 1440 2266 1 4.44e-01 2 >>> 5.51e-01 100 >>> PCGAMG Opt l03 1 1.0 8.2684e-04 1.0 2.80e+05 1.0 0.0e+00 0.0e+00 >>> 0.0e+00 0 0 0 0 0 0 0 0 0 0 339 1667 1 6.72e-02 1 >>> 2.03e-01 100 >>> PCGAMG Gal l04 1 1.0 1.2238e-03 1.0 2.09e+05 1.0 0.0e+00 0.0e+00 >>> 0.0e+00 0 0 0 0 0 0 0 0 0 0 170 244 1 2.05e-02 2 >>> 2.53e-02 100 >>> PCGAMG Opt l04 1 1.0 4.1008e-04 1.0 1.77e+04 1.0 0.0e+00 0.0e+00 >>> 0.0e+00 0 0 0 0 0 0 0 0 0 0 43 165 1 4.49e-03 1 >>> 1.19e-02 100 >>> PCSetUp 2 1.0 9.9632e+00 1.0 4.95e+09 1.0 0.0e+00 0.0e+00 >>> 0.0e+00 5 12 0 0 0 5 12 0 0 0 496 17826 55 1.03e+03 45 >>> 6.54e+02 98 >>> PCSetUpOnBlocks 44 1.0 9.9087e-04 1.0 2.88e+03 1.0 >>> >>> >>> >>> >>> >>> >>>> The point of lazy initialization is to make it possible to run a solve >>>> that doesn't use a GPU in PETSC_ARCH that supports GPUs, regardless of >>>> whether a GPU is actually present. >>>> >>>> Fande Kong <fdkong.jd(a)gmail.com> writes: >>>> >>>> > I spoke too soon. It seems that we have trouble creating cuda/kokkos >>>> vecs >>>> > now. Got Segmentation fault. >>>> > >>>> > Thanks, >>>> > >>>> > Fande >>>> > >>>> > Program received signal SIGSEGV, Segmentation fault. >>>> > 0x00002aaab5558b11 in >>>> > >>>> Petsc::CUPMDevice<(Petsc::CUPMDeviceType)0>::CUPMDeviceInternal::initialize >>>> > (this=0x1) at >>>> > >>>> /home/kongf/workhome/sawtooth/moosegpu/petsc/src/sys/objects/device/impls/cupm/cupmdevice.cxx:54 >>>> > 54 PetscErrorCode CUPMDevice<T>::CUPMDeviceInternal::initialize() >>>> noexcept >>>> > Missing separate debuginfos, use: debuginfo-install >>>> > bzip2-libs-1.0.6-13.el7.x86_64 elfutils-libelf-0.176-5.el7.x86_64 >>>> > elfutils-libs-0.176-5.el7.x86_64 glibc-2.17-325.el7_9.x86_64 >>>> > libX11-1.6.7-4.el7_9.x86_64 libXau-1.0.8-2.1.el7.x86_64 >>>> > libattr-2.4.46-13.el7.x86_64 libcap-2.22-11.el7.x86_64 >>>> > libibmad-5.4.0.MLNX20190423.1d917ae-0.1.49224.x86_64 >>>> > libibumad-43.1.1.MLNX20200211.078947f-0.1.49224.x86_64 >>>> > libibverbs-41mlnx1-OFED.4.9.0.0.7.49224.x86_64 >>>> > libmlx4-41mlnx1-OFED.4.7.3.0.3.49224.x86_64 >>>> > libmlx5-41mlnx1-OFED.4.9.0.1.2.49224.x86_64 libnl3-3.2.28-4.el7.x86_64 >>>> > librdmacm-41mlnx1-OFED.4.7.3.0.6.49224.x86_64 >>>> > librxe-41mlnx1-OFED.4.4.2.4.6.49224.x86_64 libxcb-1.13-1.el7.x86_64 >>>> > libxml2-2.9.1-6.el7_9.6.x86_64 numactl-libs-2.0.12-5.el7.x86_64 >>>> > systemd-libs-219-78.el7_9.3.x86_64 xz-libs-5.2.2-1.el7.x86_64 >>>> > zlib-1.2.7-19.el7_9.x86_64 >>>> > (gdb) bt >>>> > #0 0x00002aaab5558b11 in >>>> > >>>> Petsc::CUPMDevice<(Petsc::CUPMDeviceType)0>::CUPMDeviceInternal::initialize >>>> > (this=0x1) at >>>> > >>>> /home/kongf/workhome/sawtooth/moosegpu/petsc/src/sys/objects/device/impls/cupm/cupmdevice.cxx:54 >>>> > #1 0x00002aaab5558db7 in >>>> > Petsc::CUPMDevice<(Petsc::CUPMDeviceType)0>::getDevice >>>> > (this=this@entry=0x2aaab7f37b70 >>>> > <CUDADevice>, device=0x115da00, id=-35, id@entry=-1) at >>>> > >>>> /home/kongf/workhome/sawtooth/moosegpu/petsc/src/sys/objects/device/impls/cupm/cupmdevice.cxx:344 >>>> > #2 0x00002aaab55577de in PetscDeviceCreate (type=type@entry >>>> =PETSC_DEVICE_CUDA, >>>> > devid=devid@entry=-1, device=device@entry=0x2aaab7f37b48 >>>> > <defaultDevices+8>) at >>>> > >>>> /home/kongf/workhome/sawtooth/moosegpu/petsc/src/sys/objects/device/interface/device.cxx:107 >>>> > #3 0x00002aaab5557b3a in PetscDeviceInitializeDefaultDevice_Internal >>>> > (type=type@entry=PETSC_DEVICE_CUDA, >>>> defaultDeviceId=defaultDeviceId@entry=-1) >>>> > at >>>> > >>>> /home/kongf/workhome/sawtooth/moosegpu/petsc/src/sys/objects/device/interface/device.cxx:273 >>>> > #4 0x00002aaab5557bf6 in PetscDeviceInitialize >>>> > (type=type@entry=PETSC_DEVICE_CUDA) >>>> > at >>>> > >>>> /home/kongf/workhome/sawtooth/moosegpu/petsc/src/sys/objects/device/interface/device.cxx:234 >>>> > #5 0x00002aaab5661fcd in VecCreate_SeqCUDA (V=0x115d150) at >>>> > >>>> /home/kongf/workhome/sawtooth/moosegpu/petsc/src/vec/vec/impls/seq/seqcuda/veccuda.c:244 >>>> > #6 0x00002aaab5649b40 in VecSetType (vec=vec@entry=0x115d150, >>>> > method=method@entry=0x2aaab70b45b8 "seqcuda") at >>>> > >>>> /home/kongf/workhome/sawtooth/moosegpu/petsc/src/vec/vec/interface/vecreg.c:93 >>>> > #7 0x00002aaab579c33f in VecCreate_CUDA (v=0x115d150) at >>>> > >>>> /home/kongf/workhome/sawtooth/moosegpu/petsc/src/vec/vec/impls/mpi/mpicuda/ >>>> > mpicuda.cu:214 >>>> > #8 0x00002aaab5649b40 in VecSetType (vec=vec@entry=0x115d150, >>>> > method=method@entry=0x7fffffff9260 "cuda") at >>>> > >>>> /home/kongf/workhome/sawtooth/moosegpu/petsc/src/vec/vec/interface/vecreg.c:93 >>>> > #9 0x00002aaab5648bf1 in VecSetTypeFromOptions_Private >>>> (vec=0x115d150, >>>> > PetscOptionsObject=0x7fffffff9210) at >>>> > >>>> /home/kongf/workhome/sawtooth/moosegpu/petsc/src/vec/vec/interface/vector.c:1263 >>>> > #10 VecSetFromOptions (vec=0x115d150) at >>>> > >>>> /home/kongf/workhome/sawtooth/moosegpu/petsc/src/vec/vec/interface/vector.c:1297 >>>> > #11 0x00002aaab02ef227 in libMesh::PetscVector<double>::init >>>> > (this=0x11cd1a0, n=441, n_local=441, fast=false, >>>> ptype=libMesh::PARALLEL) >>>> > at >>>> > >>>> /home/kongf/workhome/sawtooth/moosegpu/scripts/../libmesh/installed/include/libmesh/petsc_vector.h:693 >>>> > >>>> > On Thu, Jan 20, 2022 at 1:09 PM Fande Kong <fdkong.jd(a)gmail.com> >>>> wrote: >>>> > >>>> >> Thanks, Jed, >>>> >> >>>> >> This worked! >>>> >> >>>> >> Fande >>>> >> >>>> >> On Wed, Jan 19, 2022 at 11:03 PM Jed Brown <jed(a)jedbrown.org> wrote: >>>> >> >>>> >>> Fande Kong <fdkong.jd(a)gmail.com> writes: >>>> >>> >>>> >>> > On Wed, Jan 19, 2022 at 11:39 AM Jacob Faibussowitsch < >>>> >>> jacob.fai(a)gmail.com> >>>> >>> > wrote: >>>> >>> > >>>> >>> >> Are you running on login nodes or compute nodes (I can’t seem to >>>> tell >>>> >>> from >>>> >>> >> the configure.log)? >>>> >>> >> >>>> >>> > >>>> >>> > I was compiling codes on login nodes, and running codes on compute >>>> >>> nodes. >>>> >>> > Login nodes do not have GPUs, but compute nodes do have GPUs. >>>> >>> > >>>> >>> > Just to be clear, the same thing (code, machine) with PETSc-3.16.1 >>>> >>> worked >>>> >>> > perfectly. I have this trouble with PETSc-main. >>>> >>> >>>> >>> I assume you can >>>> >>> >>>> >>> export PETSC_OPTIONS='-device_enable lazy' >>>> >>> >>>> >>> and it'll work. >>>> >>> >>>> >>> I think this should be the default. The main complaint is that >>>> timing the >>>> >>> first GPU-using event isn't accurate if it includes initialization, >>>> but I >>>> >>> think this is mostly hypothetical because you can't trust any >>>> timing that >>>> >>> doesn't preload in some form and the first GPU-using event will >>>> almost >>>> >>> always be something uninteresting so I think it will rarely lead to >>>> >>> confusion. Meanwhile, eager initialization is viscerally disruptive >>>> for >>>> >>> lots of people. >>>> >>> >>>> >> >>>> >>> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> >> https://www.cse.buffalo.edu/~knepley/ >> <http://www.cse.buffalo.edu/~knepley/> >> >> >

3 15

Question about PCFieldSplit
by Jorti, Zakariae 31 Jan '22

31 Jan '22

Hello, I have a DMStag grid with one dof on each edge and face center. I want to use a PCFieldSplit preconditioner on a Jacobian matrix that I assume is already split but I am not sure how to determine the fields. In the DMStag examples (ex2.c and ex3.c), the function PCFieldSplitSetDetectSaddlePoint is used to determine those fields based on zero diagonal entries. In my case, I have a Jacobian matrix that does not have zero diagonal entries. Can I use that PCFieldSplitSetDetectSaddlePoint in this case? If not, how should I do? Should I do like this example (https://www.mcs.anl.gov/petsc/petsc-master/src/ksp/ksp/tutorials/ex43.c.html) const PetscInt Bfields[1] = {0},Efields[1] = {1}; KSPGetPC(ksp,&pc); PCFieldSplitSetBlockSize(pc,2); PCFieldSplitSetFields(pc,"B",1,Bfields,Bfields); PCFieldSplitSetFields(pc,"E",1,Efields,Efields); where my B unknowns are defined on face centers and E unknowns are defined on edge centers? One last thing, I do not know which field comes first. Is it the one defined for face dofs or edge dofs. Thank you. Best regards, Zakariae

4 14