November 2023 - petsc-users - mailman3.cels.anl.gov

Better solver and preconditioner to use multiple GPU
by Ramoni Z. Sedano Azevedo 09 Nov '23

09 Nov '23

Hey! I am using PETSC in Fortran code and we apply the MPI process to parallelize the code. At the moment, the options that have been used are -ksp_monitor_true_residual -ksp_type bcgs -pc_type bjacobi -sub_pc_type ilu -sub_pc_factor_levels 3 -sub_pc_factor_fill 6 Now, we want to use multiple GPUs and I would like to know if there is a better solver and preconditioner pair to apply in this case. Yours sincerely, Ramoni Z. S . Azevedo

4 4

PETSC breaks when using HYPRE preconditioner
by Pantelis Moschopoulos 09 Nov '23

09 Nov '23

Hello everyone, I am trying to use Petsc coupled with Hypre BoomerAMG as preconditioner in our in-house code to simulate the transient motion of complex fluid with finite elements. The problem is that after a random number of iterations, an error arises when the Hypre is called. The error that I get in the terminal is the following: [16]PETSC ERROR: ------------------------------------------------------------------------ [16]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [16]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [16]PETSC ERROR: or see https://petsc.org/release/faq/#valgrind and https://petsc.org/release/faq/ [16]PETSC ERROR: --------------------- Stack Frames ------------------------------------ [16]PETSC ERROR: The line numbers in the error traceback are not always exact. [16]PETSC ERROR: #1 Hypre solve [16]PETSC ERROR: #2 PCApply_HYPRE() at /home/pmosx/Libraries/petsc/src/ksp/pc/impls/hypre/hypre.c:451 [16]PETSC ERROR: #3 PCApply() at /home/pmosx/Libraries/petsc/src/ksp/pc/interface/precon.c:486 [16]PETSC ERROR: #4 PCApplyBAorAB() at /home/pmosx/Libraries/petsc/src/ksp/pc/interface/precon.c:756 [16]PETSC ERROR: #5 KSP_PCApplyBAorAB() at /home/pmosx/Libraries/petsc/include/petsc/private/kspimpl.h:443 [16]PETSC ERROR: #6 KSPGMRESCycle() at /home/pmosx/Libraries/petsc/src/ksp/ksp/impls/gmres/gmres.c:146 [16]PETSC ERROR: #7 KSPSolve_GMRES() at /home/pmosx/Libraries/petsc/src/ksp/ksp/impls/gmres/gmres.c:227 [16]PETSC ERROR: #8 KSPSolve_Private() at /home/pmosx/Libraries/petsc/src/ksp/ksp/interface/itfunc.c:910 [16]PETSC ERROR: #9 KSPSolve() at /home/pmosx/Libraries/petsc/src/ksp/ksp/interface/itfunc.c:1082 On the same time, I use valgrind and when the program stops, it reports the following: ==1261647== Invalid read of size 8 ==1261647== at 0x4841C74: _intel_fast_memcpy (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so) ==1261647== by 0x16231F73: hypre_GaussElimSolve (in /home/pmosx/Libraries/PETSC_INS_DIR_INTELDebug/lib/libHYPRE-2.29.0.so) ==1261647== by 0x1622DB4F: hypre_BoomerAMGCycle (in /home/pmosx/Libraries/PETSC_INS_DIR_INTELDebug/lib/libHYPRE-2.29.0.so) ==1261647== by 0x1620002E: hypre_BoomerAMGSolve (in /home/pmosx/Libraries/PETSC_INS_DIR_INTELDebug/lib/libHYPRE-2.29.0.so) ==1261647== by 0x12B6F8F8: PCApply_HYPRE (in /home/pmosx/Libraries/PETSC_INS_DIR_INTELDebug/lib/libpetsc.so.3.20.1) ==1261647== by 0x12C38785: PCApply (in /home/pmosx/Libraries/PETSC_INS_DIR_INTELDebug/lib/libpetsc.so.3.20.1) ==1261647== by 0x12C36A39: PCApplyBAorAB (in /home/pmosx/Libraries/PETSC_INS_DIR_INTELDebug/lib/libpetsc.so.3.20.1) ==1261647== by 0x126299E1: KSPGMRESCycle (in /home/pmosx/Libraries/PETSC_INS_DIR_INTELDebug/lib/libpetsc.so.3.20.1) ==1261647== by 0x12628051: KSPSolve_GMRES (in /home/pmosx/Libraries/PETSC_INS_DIR_INTELDebug/lib/libpetsc.so.3.20.1) ==1261647== by 0x127A532E: KSPSolve_Private (in /home/pmosx/Libraries/PETSC_INS_DIR_INTELDebug/lib/libpetsc.so.3.20.1) ==1261647== by 0x127A3C8A: KSPSolve (in /home/pmosx/Libraries/PETSC_INS_DIR_INTELDebug/lib/libpetsc.so.3.20.1) ==1261647== by 0x12C50AF1: kspsolve_ (in /home/pmosx/Libraries/PETSC_INS_DIR_INTELDebug/lib/libpetsc.so.3.20.1) ==1261647== Address 0x0 is not stack'd, malloc'd or (recently) free'd ==1261647== This is indeed a very peculiar error. I cannot understand why it happens. In our solution procedure, we split the equations and we solve them segregated. I create two different ksp (1_ksp and 2_ksp) using KSPSetOptionsPrefix. Might this choice create a confusion and results in this error? Any help is much appreciated. Pantelis

2 2

Storing Values using a Triplet for using later
by Brandon Denton 09 Nov '23

09 Nov '23

Good Afternoon, Is there a structure within PETSc that allows storage of a value using a triple similar to PetscHMapIJSet with the key using a struct{PetscScalar i, j, k;}? I'm trying to access mesh information (the shape function coefficients I will calculate prior to their use) who's values I want to store in the auxiliary array available in the Residual Functions of PETSc's FEM infrastructure. After some trial and error work, I've come to the realization that the coordinates (x[]) available in the auxiliary functions is the centroid of the cell/element currently being evaluated. This triplet is unique for each cell/element for a valid mesh so I think it's reasonable to use this triplet as a key for looking up stored values unique to each cell/element. My plan is to attached the map to the Application Context, also available to Auxiliary Functions, to enable these calculations. Does such a map infrastructure exist within PETSc? If so, could you point me to a reference for it? If not, does anyone have any suggestions on how to solve this problem? Thank you in advance for your time. Brandon Denton

4 4

DMPlex and Gmsh
by Sharan Roongta 09 Nov '23

09 Nov '23

Dear Petsc team, I want to load a .msh file generated using Gmsh software into the DMPlex object. There are several things I would want to clarify, but I would like to start with "Physical tags". If I have defined "Physical Points", "Physical Surface", and "Physical Volume" in my .geo file, I get the physical tags in the ".msh" file. When I load this mesh in DMPlex, and view the DM: call DMView(globalMesh, PETSC_VIEWER_STDOUT_WORLD,err_PETSc) CHKERRQ(err_PETSc) This is the output I get: DM Object: n/a 1 MPI process type: plex n/a in 3 dimensions: Number of 0-cells per rank: 14 Number of 1-cells per rank: 49 Number of 2-cells per rank: 60 Number of 3-cells per rank: 24 Labels: celltype: 4 strata with value/size (0 (14), 6 (24), 3 (60), 1 (49)) depth: 4 strata with value/size (0 (14), 1 (49), 2 (60), 3 (24)) Cell Sets: 1 strata with value/size (8 (24)) Face Sets: 6 strata with value/size (2 (4), 3 (4), 4 (4), 5 (4), 6 (4), 7 (4)) I was expecting to get the "Node Sets" or "Vertex Sets" also. Is my assumption wrong? If yes, then how can one figure out the boundary nodes and their tags where I want to apply certain boundary conditions? Currently we apply boundary conditions on faces, therefore "Face Sets" was enough. But now we want to apply displacements on certain boundary nodes. I have also attached the .geo and .msh file (hope you can open it) The Petsc version I am using is 3.18.6. Thanks and Regards, Sharan Roongta ----------------------------------------------- Max-Planck-Institut für Eisenforschung GmbH Max-Planck-Straße 1 D-40237 Düsseldorf Handelsregister B 2533 Amtsgericht Düsseldorf Geschäftsführung Prof. Dr. Gerhard Dehm Prof. Dr. Jörg Neugebauer Prof. Dr. Dierk Raabe Dr. Kai de Weldige Ust.-Id.-Nr.: DE 11 93 58 514 Steuernummer: 105 5891 1000 ------------------------------------------------- Please consider that invitations and e-mails of our institute are only valid if they end with ...(a)mpie.de. If you are not sure of the validity please contact rco(a)mpie.de Bitte beachten Sie, dass Einladungen zu Veranstaltungen und E-Mails aus unserem Haus nur mit der Endung ...(a)mpie.de gültig sind. In Zweifelsfällen wenden Sie sich bitte an rco(a)mpie.de ------------------------------------------------- Stay up to date and follow us on LinkedIn, Twitter and YouTube. Max-Planck-Institut für Eisenforschung GmbH Max-Planck-Straße 1 D-40237 Düsseldorf Handelsregister B 2533 Amtsgericht Düsseldorf Geschäftsführung Prof. Dr. Gerhard Dehm Prof. Dr. Jörg Neugebauer Prof. Dr. Dierk Raabe Dr. Kai de Weldige Ust.-Id.-Nr.: DE 11 93 58 514 Steuernummer: 105 5891 1000 Please consider that invitations and e-mails of our institute are only valid if they end with …(a)mpie.de. If you are not sure of the validity please contact rco(a)mpie.de Bitte beachten Sie, dass Einladungen zu Veranstaltungen und E-Mails aus unserem Haus nur mit der Endung …(a)mpie.de gültig sind. In Zweifelsfällen wenden Sie sich bitte an rco(a)mpie.de -------------------------------------------------

3 3

error in configuring PETSc
by 张胜 08 Nov '23

08 Nov '23

Dear PETSc developer, I use the following commands to configure petsc, but errors occur: ./configure --with-cc=gcc-11 --with-cxx=g++-11 --with-fc=gfortran-11 --download-fftw --download-openmpi --download-fblaslapack --free-line-length-0 -g -fallow-argument-mismatch --enable-shared --enable-parallel --enable-fortran --with-zlibs=yes --with-szlib=no --with-cxx-dialect=C++11 --with-c2html=0 --with-x=0 --download-hdf5-fortran-bindings=1 --download-hdf5 I tried many times but cannot fix it. So I ask help for you. Thanks in advance. Best regards, Sheng Zhang Ph.D School of Materials Science and Engineering Shanghai Jiao Tong University 800 Dongchuan Road Shanghai, 200240 China

3 3

Error in configuring PETSc with Cygwin on Windows by using Intel MPI
by 程奔 08 Nov '23

08 Nov '23

Hello, Recently I try to install PETSc with Cygwin since I'd like to use PETSc with Visual Studio on Windows10 plateform.For the sake of clarity, I firstly list the softwares/packages used below: 1. PETSc: version 3.19.2 2. VS: version 2022 3. Intel MPI: download Intel oneAPI Base Toolkit and HPC Toolkit 4. Cygwin: see the picture attatched (see picture cygwin) And the compiler option in configuration is: ./configure --prefix=/cygdrive/g/mypetsc/petsc2023 --with-cc='win32fe cl' --with-fc='win32fe ifort' --with-cxx='win32fe cl' --with-shared-libraries=0 --with-mpi-include=/cygdrive/g/Intel/oneAPI/mpi/2021.10.0/include --with-mpi-lib=/cygdrive/g/Intel/oneAPI/mpi/2021.10.0/libTherefore, I write this e-mail to look for your help. /release/impi.lib --with-mpiexec=/cygdrive/g/Intel/oneAPI/mpi/2021.10.0/bin/mpiexec -localonly but there return an error: ********************************************************************************************* OSError while running ./configure --------------------------------------------------------------------------------------------- Cannot run executables created with FC. If this machine uses a batch system to submit jobs you will need to configure using ./configure with the additional option --with-batch. Otherwise there is problem with the compilers. Can you compile and run code with your compiler '/cygdrive/g/mypetsc/petsc-3.19.2/lib/petsc/bin/win32fe/win32fe ifort'? See https://petsc.org/release/faq/#error-libimf ********************************************************************************************* Then I try to open configure.log in petsc, but there turnout an error that I can't open it.(see picture 1) And then I right click on properties and click safety,it just turnout "The permissions on test directory are incorrectly ordered.which may cause some entries to be ineffective." (see picture 2) And it also likely seen “NULL SID” as the top entry in permission lists.(see picture 3) Then i follow this blog(https://blog.dhampir.no/content/forcing-cygwin-to-create-sane-permissions-on-windows) to edit /etc/fstab in Cygwin, and add “noacl” to the mount options for /cygdrive. But it's not working. So I can't sent configure.log to you guys, it seems cygwin that installed in my computer happened to some problem. Mayebe the error happened in the configure on petsc just because of this reason. So I wrrit this email to report my problem and ask for your help. Looking forward your reply! sinserely, Cheng.

2 2

Fortran, Hypre, PETSc, 64bit-integers
by Bastian Löhrer 08 Nov '23

08 Nov '23

1 0

Error using Metis with PETSc installed with MUMPS
by Victoria Rolandi 07 Nov '23

07 Nov '23

Hi, I'm solving a large sparse linear system in parallel and I am using PETSc with MUMPS. I am trying to test different options, like the ordering of the matrix. Everything works if I use the *-mat_mumps_icntl_7 2 *or *-mat_mumps_icntl_7 0 *options (with the first one, AMF, performing better than AMD), however when I test METIS *-mat_mumps_icntl_7** 5 *I get an error (reported at the end of the email). I have configured PETSc with the following options: --with-cc=mpiicc --with-cxx=mpiicpc --with-fc=mpiifort --with-scalar-type=complex --with-debugging=0 --with-precision=single --download-mumps --download-scalapack --download-parmetis --download-metis and the installation didn't give any problems. Could you help me understand why metis is not working? Thank you in advance, Victoria Error: ****** ANALYSIS STEP ******** ** Maximum transversal (ICNTL(6)) not allowed because matrix is distributed Processing a graph of size: 699150 with 69238690 edges Ordering based on METIS 510522 37081376 [100] [10486 699150] Error! Unknown CType: -1

4 13

Performance problem about output mesh in vtk format
by Gong Yujie 05 Nov '23

05 Nov '23

Dear PETSc developers, I'm trying to output a result data in vtk format and find that it is quite slow. Then I try to check this issue by a simple test code: PetscCall(PetscInitialize(&argc,&argv,(char*)0,NULL)); DM dm,dmParallel,dmAux; PetscBool interpolate=PETSC_TRUE; PetscCall(DMPlexCreateExodusFromFile(PETSC_COMM_WORLD,"artery_plaque.exo",interpolate,&dm)); PetscCall(DMViewFromOptions(dm,NULL,"-dm_view")); PetscCall(PetscFinalize()); and run with ./dm_test -dm_view vtk:./ksp_data/abc.vtk -log_view It took about 600s to output the mesh. I'm not sure if there is something wrong in my code or my configuration of PETSc. Could you please give me some advice on this? Best Regards, Yujie P.S. The result for log_view **************************************************************************************************************************************************************** *** WIDEN YOUR WINDOW TO 160 CHARACTERS. Use 'enscript -r -fCourier9' to print this document *** **************************************************************************************************************************************************************** ------------------------------------------------------------------ PETSc Performance Summary: ------------------------------------------------------------------ ./dm_test on a arch-linux-c-opt named DESKTOP-0H8HCOD with 1 processor, by qingfeng Mon Nov 6 10:43:31 2023 Using Petsc Release Version 3.19.5, unknown Max Max/Min Avg Total Time (sec): 6.286e+02 1.000 6.286e+02 Objects: 1.400e+02 1.000 1.400e+02 Flops: 0.000e+00 0.000 0.000e+00 0.000e+00 Flops/sec: 0.000e+00 0.000 0.000e+00 0.000e+00 MPI Msg Count: 0.000e+00 0.000 0.000e+00 0.000e+00 MPI Msg Len (bytes): 0.000e+00 0.000 0.000e+00 0.000e+00 MPI Reductions: 0.000e+00 0.000 Flop counting convention: 1 flop = 1 real number operation of type (multiply/divide/add/subtract) e.g., VecAXPY() for real vectors of length N --> 2N flops and VecAXPY() for complex vectors of length N --> 8N flops Summary of Stages: ----- Time ------ ----- Flop ------ --- Messages --- -- Message Lengths -- -- Reductions -- Avg %Total Avg %Total Count %Total Avg %Total Count %Total 0: Main Stage: 6.2859e+02 100.0% 0.0000e+00 0.0% 0.000e+00 0.0% 0.000e+00 0.0% 0.000e+00 0.0% ------------------------------------------------------------------------------------------------------------------------ See the 'Profiling' chapter of the users' manual for details on interpreting output. Phase summary info: Count: number of times phase was executed Time and Flop: Max - maximum over all processors Ratio - ratio of maximum to minimum over all processors Mess: number of messages sent AvgLen: average message length (bytes) Reduct: number of global reductions Global: entire computation Stage: stages of a computation. Set stages with PetscLogStagePush() and PetscLogStagePop(). %T - percent time in this phase %F - percent flop in this phase %M - percent messages in this phase %L - percent message lengths in this phase %R - percent reductions in this phase Total Mflop/s: 10e-6 * (sum of flop over all processors)/(max time over all processors) ------------------------------------------------------------------------------------------------------------------------ Event Count Time (sec) Flop --- Global --- --- Stage ---- Total Max Ratio Max Ratio Max Ratio Mess AvgLen Reduct %T %F %M %L %R %T %F %M %L %R Mflop/s ------------------------------------------------------------------------------------------------------------------------ --- Event Stage 0: Main Stage DMPlexInterp 1 1.0 3.1186e+00 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 DMPlexStratify 3 1.0 4.2802e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 DMPlexSymmetrize 3 1.0 1.0806e-01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 ------------------------------------------------------------------------------------------------------------------------ Object Type Creations Destructions. Reports information only for process 0. --- Event Stage 0: Main Stage Container 2 1 Distributed Mesh 5 3 DM Label 20 8 Index Set 64 52 Section 17 12 Star Forest Graph 10 7 Discrete System 7 5 Weak Form 7 5 GraphPartitioner 3 2 Matrix 2 1 Vector 1 0 Viewer 2 1 ======================================================================================================================== Average time to get PetscTime(): 1.8e-08 #PETSc Option Table entries: -dm_view vtk:./ksp_data/abc.vtk # (source: command line) -log_view # (source: command line) #End of PETSc Option Table entries Compiled without FORTRAN kernels Compiled with full precision matrices (default) sizeof(short) 2 sizeof(int) 4 sizeof(long) 8 sizeof(void*) 8 sizeof(PetscScalar) 8 sizeof(PetscInt) 4 Configure options: --with-debugging=0 --with-strict-petscerrorcode --download-openmpi --download-metis --download-exodusii --download-parmetis --download-netcdf --download-pnetcdf --download-hdf5 --download-zlib --download-superlu --download-superlu_dist --download-triangle --download-cmake --download-fblaslapack --download-slepc ----------------------------------------- Libraries compiled on 2023-09-15 02:34:25 on DESKTOP-0H8HCOD Machine characteristics: Linux-5.10.16.3-microsoft-standard-WSL2-x86_64-with-glibc2.29 Using PETSc directory: /home/qingfeng/petsc/optpetsc3-19-5/petsc Using PETSc arch: arch-linux-c-opt ----------------------------------------- Using C compiler: /home/qingfeng/petsc/optpetsc3-19-5/petsc/arch-linux-c-opt/bin/mpicc -fPIC -Wall -Wwrite-strings -Wno-unknown-pragmas -Wno-lto-type-mismatch -fstack-protector -fvisibility=hidden -g -O Using Fortran compiler: /home/qingfeng/petsc/optpetsc3-19-5/petsc/arch-linux-c-opt/bin/mpif90 -fPIC -Wall -ffree-line-length-none -ffree-line-length-0 -Wno-lto-type-mismatch -Wno-unused-dummy-argument -g -O ----------------------------------------- Using include paths: -I/home/qingfeng/petsc/optpetsc3-19-5/petsc/include -I/home/qingfeng/petsc/optpetsc3-19-5/petsc/arch-linux-c-opt/include ----------------------------------------- Using C linker: /home/qingfeng/petsc/optpetsc3-19-5/petsc/arch-linux-c-opt/bin/mpicc Using Fortran linker: /home/qingfeng/petsc/optpetsc3-19-5/petsc/arch-linux-c-opt/bin/mpif90 Using libraries: -Wl,-rpath,/home/qingfeng/petsc/optpetsc3-19-5/petsc/arch-linux-c-opt/lib -L/home/qingfeng/petsc/optpetsc3-19-5/petsc/arch-linux-c-opt/lib -lpetsc -Wl,-rpath,/home/qingfeng/petsc/optpetsc3-19-5/petsc/arch-linux-c-opt/lib -L/home/qingfeng/petsc/optpetsc3-19-5/petsc/arch-linux-c-opt/lib -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/9 -L/usr/lib/gcc/x86_64-linux-gnu/9 -lsuperlu -lsuperlu_dist -lflapack -lfblas -lexoIIv2for32 -lexodus -lnetcdf -lpnetcdf -lhdf5_hl -lhdf5 -lparmetis -lmetis -ltriangle -lm -lz -ldl -lmpi_usempif08 -lmpi_usempi_ignore_tkr -lmpi_mpifh -lmpi -lgfortran -lm -lgfortran -lm -lgcc_s -lquadmath -lpthread -lstdc++ -ldl ----------------------------------------- The mesh information: DM Object: Created by ICEMCFD - EXODUS II Interface 1 MPI process type: plex Created by ICEMCFD - EXODUS II Interface in 3 dimensions: Number of 0-cells per rank: 134549 Number of 1-cells per rank: 841756 Number of 2-cells per rank: 1366008 Number of 3-cells per rank: 658801 Labels: celltype: 4 strata with value/size (0 (134549), 6 (658801), 3 (1366008), 1 (841756)) depth: 4 strata with value/size (0 (134549), 1 (841756), 2 (1366008), 3 (658801)) Cell Sets: 2 strata with value/size (1 (604426), 2 (54375)) Vertex Sets: 5 strata with value/size (3 (481), 4 (27248), 5 (20560), 6 (653), 7 (2370)) Face Sets: 5 strata with value/size (8 (740), 9 (54206), 10 (40857), 11 (999), 12 (4534)) SMALLER: 1 strata with value/size (8 (740)) OUTER: 1 strata with value/size (9 (54206)) INNER: 1 strata with value/size (10 (40857)) BIGGER: 1 strata with value/size (11 (999))

2 1

Domain decomposition in PETSc for Molecular Dynamics
by MIGUEL MOLINOS PEREZ 04 Nov '23

04 Nov '23

Dear all, I am currently working on the development of a in-house molecular dynamics code using PETSc and C++. So far the code works great, however it is a little bit slow since I am not exploiting MPI for PETSc vectors. I was wondering if there is a way to perform the domain decomposition efficiently using some PETSc functionality. Any feedback is highly appreciated. Best regards, Miguel

3 2