October 2023 - petsc-users - mailman3.cels.anl.gov

Performance of Conda Binary vs Self Compiled Version
by Jorge Nin 21 Oct '23

21 Oct '23

Hi, I was playing around with a self compiled version and, and a the Conda binary of Petsc on the same problem, on my M1 Mac. Interestingly I found that the Conda binary solves the problem 2-3 times slower vs the self compiled version. (For context I’m using the petsc4py python interface) I’ve attached two log views to show the comparison. I was mostly curious about the possible cause for this. I was also curious how I could use my own compiled version of PETSc in my Conda install? Best, Jorge

4 5

use MATSEQAIJMKL in 64-bit indices
by Di Miao 20 Oct '23

20 Oct '23

Hi, I found that when compiled with '--with-64-bit-indices=1' option, the following three definitions in petscconf.h will be removed: #define PETSC_HAVE_MKL_SPARSE 1 #define PETSC_HAVE_MKL_SPARSE_OPTIMIZE 1 #define PETSC_HAVE_MKL_SPARSE_SP2M_FEATURE 1 I believe mkl can also use 64-bit indices (libmkl_intel_ilp64). I tried to add ' --with-mkl_sparse=1 --with-mkl_sparse_optimize=1' into configuration but does not succeed. Would I know if it is possible to use MATSEQAIJMKL matrix type in 64-bit mode? Regards, Di

2 2

Coordinate format internal reordering
by Enrico 20 Oct '23

20 Oct '23

Hello, I'm trying to use Petsc to solve a linear system in an application. I'm using the coordinate format to define the matrix and the vector (it should work better on GPU but at the moment every test is on CPU). After the call to VecSetValuesCOO, I've noticed that the vector is storing the data in a different way from my application. For example with two processes in the application process 0 owns cells 2, 3, 4 process 1 owns cells 0, 1, 5 But in the vector data structure of Petsc process 0 owns cells 0, 1, 2 process 1 owns cells 3, 4, 5 This is in principle not a big issue, but after solving the linear system I get the solution vector x and I want to get the values in the correct processes. Is there a way to get vector values from other processes or to get a mapping so that I can do it myself? Cheers, Enrico Degregori

2 15

Error when installing PETSc
by Gong Yujie 18 Oct '23

18 Oct '23

Dear PETSc developers, I got an error message when installing PETSc with a clang compiler. Could you please help me find the problem? The configure.log is attached. Best Regards, Yujie Here is the detail of the error: ============================================================================================= Configuring PETSc to compile on your system ============================================================================================= ============================================================================================= ***** WARNING: Using default optimization C flags -g -O3 You might consider manually setting optimal optimization flags for your system with COPTFLAGS="optimization flags" see config/examples/arch-*-opt.py for examples ============================================================================================= ============================================================================================= ***** WARNING: Using default Cxx optimization flags -g -O3 You might consider manually setting optimal optimization flags for your system with CXXOPTFLAGS="optimization flags" see config/examples/arch-*-opt.py for examples ============================================================================================= ============================================================================================= ***** WARNING: Using default FORTRAN optimization flags -O You might consider manually setting optimal optimization flags for your system with FOPTFLAGS="optimization flags" see config/examples/arch-*-opt.py for examples ============================================================================================= ============================================================================================= Trying to download https://download.open-mpi.org/release/open-mpi/v4.1/openmpi-4.1.0.tar.gz for ============================================================================================= ============================================================================================= Running configure on OPENMPI; this may take several minutes ============================================================================================= ============================================================================================= Running make on OPENMPI; this may take several minutes ============================================================================================= ******************************************************************************* UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log for details): ------------------------------------------------------------------------------- Error running make; make install on OPENMPI *******************************************************************************

3 2

Re: [petsc-users] About recent changes in GAMG
by Mark Adams 18 Oct '23

18 Oct '23

Hi Jeremy, I hope you don't mind putting this on the list (w/o data), but this is documentation and you are the second user that found regressions. Sorry for the churn. There is a lot here so we can iterate, but here is a pass at your questions. *** Using MIS-2 instead of square graph was motivated by setup cost/performance but on GPUs with some recent fixes in Kokkos (in a branch) square graph seems OK. My experience was that square graph is better in terms of quality and we have a power user, like you all, that found this also. So I switched the default back to square graph. Interesting that you found that MIS-2 (new method) could be faster, but it might be because the two methods coarsen at different rates and that can make a big difference. (the way to test would be to adjust parameters to get similar coarsen rates, but I digress) It's hard to understand the differences between these two methods in terms of aggregate quality so we need to just experiment and have options. *** As far as your thermal problem. There was a complaint that the eigen estimates for chebyshev smoother were not recomputed for nonlinear problems and I added an option to do that and turned it on by default: Use '-pc_gamg_recompute_esteig false' to get back to the original. (I should have turned it off by default) Now, if your problem is symmetric and you use CG to compute the eigen estimates there should be no difference. If you use CG to compute the eigen estimates in GAMG (and have GAMG give them to cheby, the default) that when you recompute the eigen estimates the cheby eigen estimator is used and that will use gmres by default unless you set the SPD property in your matrix. So if you set '-pc_gamg_esteig_ksp_type cg' you want to also set '-mg_levels_esteig_ksp_type cg' (verify with -ksp_view and -options_left) CG is a much better estimator for SPD. And I found that the cheby eigen estimator uses an LAPACK *eigen* method to compute the eigen bounds and GAMG uses a *singular value* method. The two give very different results on the lid driven cavity test (ex19). eigen is lower, which is safer but not optimal if it is too low. I have a branch to have cheby use the singular value method, but I don't plan on merging it (enough churn and I don't understand these differences). *** '-pc_gamg_low_memory_threshold_filter false' recovers the old filtering method. This is the default now because there is a bug in the (new) low memory filter. This bug is very rare and catastrophic. We are working on it and will turn it on by default when it's fixed. This does not affect the semantics of the solver, just work and memory complexity. *** As far as tet4 vs tet10, I would guess that tet4 wants more aggressive coarsening. The default is to do aggressive on one (1) level. You might want more levels for tet4. And the new MIS-k coarsening can use any k (default is 2) wth '-mat_coarsen_misk_distance k' (eg, k=3) I have not added hooks to have a more complex schedule to specify the method on each level. Thanks, Mark On Tue, Oct 17, 2023 at 9:33 PM Jeremy Theler (External) < jeremy.theler-ext(a)ansys.com> wrote: > Hey Mark > > Regarding the changes in the coarsening algorithm in 3.20 with respect to > 3.19 in general we see that for some problems the MIS strategy gives and > overall performance which is slightly better and for some others it is > slightly worse than the "baseline" from 3.19. > We also saw that current main has switched back to the old square > coarsening algorithm by default, which again, in some cases is better and > in others is worse than 3.19 without any extra command-line option. > > Now what seems weird to us is that we have a test case which is a heat > conduction problem with radiation boundary conditions (so it is non linear) > using tet10 and we see > > 1. that in parallel v3.20 is way worse than v3.19, although the memory > usage is similar > 2. that petsc main (with no extra flags, just the defaults) recover > the 3.19 performance but memory usage is significantly larger > > > I tried using the -pc_gamg_low_memory_threshold_filter flag and the > results were the same. > > Find attached the log and snes views of 3.19, 3.20 and main using 4 MPI > ranks. > Is there any explanation about these two points we are seeing? > Another weird finding is that if we use tet4 instead of tet10, v3.20 is > only 10% slower than the other two and main does not need more memory than > the other two. > > BTW, I have dozens of other log view outputs comparing 3.19, 3.20 and main > should you be interested. > > Let me know if it is better to move this discussion into the PETSc mailing > list. > > Regards, > jeremy theler > > >

1 0

SLEPc/NEP for shell matrice T(lambda) and T'(lambda)
by Kenneth C Hall 17 Oct '23

17 Oct '23

Hi all, I have a very large eigenvalue problem of the form T(\lambda).x = 0. The eigenvalues appear in a complicated way, and I must use a matrix-free approach to compute the products T.x and T’.x. I am trying to implement in SLEPc/NEP. To get started, I have defined a much smaller and simpler system of the form A.x - \lambda x = 0 where A is a 10x10 matrix. This is of course a simple standard eigenvalue problem, but I am using it as a surrogate to understand how to use NEP. I have set the problem up using shell matrices (as that is my ultimate goal). The full code is attached, but here is a smaller snippet of code: !.... Create matrix-free operators for A and B PetscCall(MatCreateShell(PETSC_COMM_SELF,n,n,PETSC_DETERMINE,PETSC_DETERMINE, PETSC_NULL_INTEGER, A, ierr)) PetscCall(MatCreateShell(PETSC_COMM_SELF,n,n,PETSC_DETERMINE,PETSC_DETERMINE, PETSC_NULL_INTEGER, B, ierr)) PetscCall(MatShellSetOperation(A, MATOP_MULT, MatMult_A, ierr)) PetscCall(MatShellSetOperation(B, MATOP_MULT, MatMult_B, ierr)) !.... Create nonlinear eigensolver PetscCall(NEPCreate(PETSC_COMM_SELF, nep, ierr)) !.... Set the problem type PetscCall(NEPSetProblemType(nep, NEP_GENERAL, ierr)) ! !.... set the solver type PetscCall(NEPSetType(nep, NEPNLEIGS, ierr)) ! !.... Set functions and Jacobians for NEP PetscCall(NEPSetFunction(nep, A, A, MyNEPFunction, PETSC_NULL_INTEGER, ierr)) PetscCall(NEPSetJacobian(nep, B, MyNEPJacobian, PETSC_NULL_INTEGER, ierr)) The code runs, calls MyNEPFunction and MatMult_A multiple times, sweeping over the prescribed RG range, but crashes before it ever calls MyNEPJacobian or MatMult_B. The NEP viewer and error messages are attached. Any help on getting this problem properly set up would be greatly appreciated. Kenneth Hall ATTACHMENTS: test_nep.f90 code_output

2 8

Using Sundials from PETSc
by Vanella, Marcos (Fed) 16 Oct '23

16 Oct '23

Hi, we were wondering if it would be possible to call the latest version of Sundials from PETSc? We are interested in doing chemistry using GPUs and already have interfaces to PETSc from our code. Thanks, Marcos

3 5

Parallel DMPlex
by erdemguer 16 Oct '23

16 Oct '23

Hi, I am currently using DMPlex in my code. It runs serially at the moment, but I'm interested in adding parallel options. Here is my workflow: Create a DMPlex mesh from GMSH. Reorder it with DMPlexPermute. Create necessary pre-processing arrays related to the mesh/problem. Create field(s) with multi-dofs. Create residual vectors. Define a function to calculate the residual for each cell and, use SNES. As you can see, I'm not using FV or FE structures (most examples do). Now, I'm trying to implement this in parallel using a similar approach. However, I'm struggling to understand how to create corresponding vectors and how to obtain index sets for each processor. Is there a tutorial or paper that covers this topic? Thank you. Guer. Sent with [Proton Mail](https://proton.me/) secure email.

2 14

FEM Implementation of NS with SUPG Stabilization
by Brandon Denton 11 Oct '23

11 Oct '23

Good Evening, I am looking to implement a form of Navier-Stokes with SUPG Stabilization and shock capturing using PETSc's FEM infrastructure. In this implementation, I need access to the cell's shape function gradients and natural coordinate gradients for calculations within the point-wise residual calculations. How do I get these quantities at the quadrature points? The signatures for fo and f1 don't seem to contain this information. Thank you in advance for your time. Brandon

3 12

Configuration of PETSc with Intel OneAPI and Intel MPI fails
by Richter, Roland 11 Oct '23

11 Oct '23

Hei, I'm currently trying to install PETSc on a server (Ubuntu 22.04) with Intel MPI and Intel OneAPI. To combine both, I have to use f. ex. "mpiicc -cc=icx" as C-compiler, as described by https://stackoverflow.com/a/76362396. Therefore, I adapted the configure-line as follow: ./configure --prefix=/media/storage/local_opt/petsc --with-scalar-type=complex --with-cc="mpiicc -cc=icx" --with-cxx="mpiicpc -cxx=icpx" --CPPFLAGS="-fPIC -march=native -mavx2" --CXXFLAGS="-fPIC -march=native -mavx2" --with-fc="mpiifort -fc=ifx" --with-pic=true --with-mpi=true --with-blaslapack-dir=/opt/intel/oneapi/mkl/latest/lib/intel64/ --with-openmp=true --download-hdf5=yes --download-netcdf=yes --download-chaco=no --download-metis=yes --download-slepc=yes --download-suitesparse=yes --download-eigen=yes --download-parmetis=yes --download-ptscotch=yes --download-mumps=yes --download-scalapack=yes --download-superlu=yes --download-superlu_dist=yes --with-mkl_pardiso=1 --with-boost=1 --with-boost-dir=/media/storage/local_opt/boost --download-opencascade=yes --with-fftw=1 --with-fftw-dir=/media/storage/local_opt/fftw3 --download-kokkos=yes --with-mkl_sparse=1 --with-mkl_cpardiso=1 --with-mkl_sparse_optimize=1 --download-muparser=no --download-p4est=yes --download-sowing=yes --download-viennalcl=yes --with-zlib --force=1 --with-clean=1 --with-cuda=1 The configuration, however, fails with The CMAKE_C_COMPILER: mpiicc -cc=icx is not a full path and was not found in the PATH for all additional modules which use a cmake-based configuration approach (such as OPENCASCADE). How could I solve that problem? Thank you! Regards, Roland Richter

4 5