December 2016 - petsc-users - mailman3.cels.anl.gov

Nodes coordinates in distributed dmplex
by Alejandro D Otero 16 Jul '22

16 Jul '22

Hi, sorry if this is an obvious question, but I cannot figure out how to recover finite element nodes coordinates once I have distributed a mesh stored as a dmplex. I am using petsc4py as interface to petsc rutines. I first created a dmplex using: dm.createFromCellList() In a sequential run I got the coordinates with: Coords = dm.getCoordinates() which gave a sequential vector with the coordinates of the mesh nodes. When I distribute the mesh with: dm.distribute() each mpi process has it own dm but the indexing of the vector resulting from getCoordinates() or getCoordinatesLocal() seems not consistent with the local numbering of the cells and nodes. Which is the correct way of doing this in PETSc philosophy? Thanks in advance, Alejandro

4 9

MatGetDiagonal
by Nachiket Gokhale 09 Dec '19

09 Dec '19

Any chance of making this work in serial? http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatGetDiago… Not a show stopper, I am trying to get the diagonal of some small projected, dense matrices (which come from large sparse matrices). I am running in serial because 1) Since my projected matrices are small, and 2) PETSc does not do certain matrix multiplications involving a dense matrix in parallel, -Nachiket

4 5

monitoring the convergence of fieldsplit 0 and 1
by Klaij, Christiaan 15 May '18

15 May '18

A question when using PCFieldSplit: for each linear iteration of the system, how many iterations for fielsplit 0 and 1? One way to find out is to run with -ksp_monitor, -fieldsplit_0_ksp_monitor and -fieldsplit_0_ksp_monitor. This gives the complete convergence history. Another way, suggested by Matt, is to use -ksp_monitor, -fieldsplit_0_ksp_converged_reason and -fieldsplit_1_ksp_converged_reason. This gives only the totals for fieldsplit 0 and 1 (but without saying for which one). Both ways require to somehow process the output, which is a bit inconvenient. Could KSPGetResidualHistory perhaps return (some) information on the subsystems' convergence for processing inside the code? Chris dr. ir. Christiaan Klaij CFD Researcher Research & Development E mailto:[email protected] T +31 317 49 33 44 MARIN 2, Haagsteeg, P.O. Box 28, 6700 AA Wageningen, The Netherlands T +31 317 49 39 11, F +31 317 49 32 45, I www.marin.nl

4 19

Question about memory usage in Multigrid preconditioner
by frank 22 Jan '18

22 Jan '18

Hi, I am using the CG ksp solver and Multigrid preconditioner to solve a linear system in parallel. I chose to use the 'Telescope' as the preconditioner on the coarse mesh for its good performance. The petsc options file is attached. The domain is a 3d box. It works well when the grid is 1536*128*384 and the process mesh is 96*8*24. When I double the size of grid and keep the same process mesh and petsc options, I get an "out of memory" error from the super-cluster I am using. Each process has access to at least 8G memory, which should be more than enough for my application. I am sure that all the other parts of my code( except the linear solver ) do not use much memory. So I doubt if there is something wrong with the linear solver. The error occurs before the linear system is completely solved so I don't have the info from ksp view. I am not able to re-produce the error with a smaller problem either. In addition, I tried to use the block jacobi as the preconditioner with the same grid and same decomposition. The linear solver runs extremely slow but there is no memory error. How can I diagnose what exactly cause the error? Thank you so much. Frank

7 56

For Fortran users of PETSc development version
by Barry Smith 17 May '17

17 May '17

For Fortran users of the PETSc development (git master branch) version I have updated and simplified the Fortran usage of PETSc in the past few weeks. I will put the branch barry/fortran-update into the master branch on Monday. The usage changes are A) for each Fortran function (and main) use the following subroutine mysubroutine(.....) #include <petsc/finclude/petscxxx.h> use petscxxx implicit none For example if you are using SNES in your code you would have #include <petsc/finclude/petscsnes.h> use petscsnes implicit none B) Instead of PETSC_NULL_OBJECT you must pass PETSC_NULL_XXX (for example PETSC_NULL_VEC) using the specific object type XXX that the function call is expecting. C) Objects can be declared either as XXX a or type(iXXX) a, for example Mat a or type(iMat) a. (Note that previously for those who used types it was type(Mat) but that can no longer be used. Notes: 1) There are no longer any .h90 files that may be included 2) Like C the include files are now nested so you no longer need to include for example #include <petsc/finclude/petscsys.h> #include <petsc/finclude/petscvec.h> #include <petsc/finclude/petscmat.h> #include <petsc/finclude/petscpc.h> #include <petsc/finclude/petscksp.h> you can just include #include <petsc/finclude/petscksp.h> 3) there is now type checking of most function calls. This will help eliminate bugs due to incorrect calling sequences. Note that Fortran distinguishes between a argument that is a scalar (zero dimensional array), a one dimensional array and a two dimensional array (etc). So you may get compile warnings because you are passing in an array when PETSc expects a scalar or vis-versa. If you get these simply fix your declaration of the variable to match what is expected. In some routines like MatSetValues() and friends you can pass either scalars, one dimensional arrays or two dimensional arrays, if you get errors here please send mail to petsc-maint(a)mcs.anl.gov and include enough of your code so we can see the dimensions of all your variables so we can fix the problems. 4) You can continue to use either fixed (.F extension) or free format (.F90 extension) for your source 5) All the examples in PETSc have been updated so consult them for clarifications. Please report any problems to petsc-maint(a)mcs.anl.gov Thanks Barry

2 3

How to use class function in SNESSetFunction
by behzad baghapour 20 Mar '17

20 Mar '17

Dear developers, In my code, the function "residual" used in SNESSetFunction() for calculating Nonlinear function is a member of a class called "solver". When compiling with Petsc, I received the error: error: argument of type ‘PetscErrorCode (solver::)(_p_SNES*, _p_Vec*, _p_Vec*, void*)’ does not match ‘PetscErrorCode (*)(_p_SNES*, _p_Vec*, _p_Vec*, void*)’ How can I access that class member function is SNESSetFunction()? I need to keep my code structure as before. Thanks, BB.

3 4

Nested Fieldsplit for custom index sets
by Safin, Artur 23 Jan '17

23 Jan '17

Hello all, I would like to work out how to get nested fieldsplit to work correctly. I have a submatrix (labeled fieldsplit_P) that I would like to block precondition with sub-blocks A & B. To do this, I access the PC object within fieldsplit_P, and pass index sets corresponding to these sub-blocks (P_A_IS, P_B_IS) that tell how the matrix should be split. This is what I have: -------------------------------------------------------------------------------- KSP *ksp_all, ksp_P; PCFieldSplitGetSubKSP(pc, &i, &ksp_all); ksp_P = ksp_all[0]; PC pc_P; KSPGetPC(ksp_P, &pc_P); // This should extract the preconditioner for fieldsplit P PCFieldSplitSetIS(pc_P, "A", P_A_IS); PCFieldSplitSetIS(pc_P, "B", P_B_IS); -------------------------------------------------------------------------------- And these are the run-time arguments: -------------------------------------------------------------------------------- -pc_type fieldsplit -pc_fieldsplit_type multiplicative -fieldsplit_P_ksp_type gmres -fieldsplit_P_pc_type fieldsplit -fieldsplit_P_pc_fieldsplit_type multiplicative -fieldsplit_P_fieldsplit_A_ksp_type gmres -fieldsplit_P_fieldsplit_B_pc_type lu -fieldsplit_P_fieldsplit_B_ksp_type preonly -------------------------------------------------------------------------------- But that does not work: -------------------------------------------------------------------------------- [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: No support for this operation for this object type [0]PETSC ERROR: Cannot locate function PCFieldSplitSetIS_C in object [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [0]PETSC ERROR: Petsc Release Version 3.7.2, Jun, 05, 2016 [0]PETSC ERROR: ./main_2D on a x86_64 named artur-ubuntu by artur Tue Jul 26 18:55:29 2016 [0]PETSC ERROR: Configure options --with-scalar-type=complex --with-mpi=1 --with-clanguage=c++ --with-cc=mpicc --with-fc=gfortran --with-cxx=mpic++ --with-fc=mpif90 --download-mumps --download-scalapack [0]PETSC ERROR: #1 PCFieldSplitSetIS() line 1756 in /home/artur/Rorsrach/Packages/petsc-3.7.2/src/ksp/pc/impls/fieldsplit/fieldsplit.c [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: No support for this operation for this object type [0]PETSC ERROR: Cannot locate function PCFieldSplitSetIS_C in object [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [0]PETSC ERROR: Petsc Release Version 3.7.2, Jun, 05, 2016 [0]PETSC ERROR: ./main_2D on a x86_64 named artur-ubuntu by artur Tue Jul 26 18:55:29 2016 [0]PETSC ERROR: Configure options --with-scalar-type=complex --with-mpi=1 --with-clanguage=c++ --with-cc=mpicc --with-fc=gfortran --with-cxx=mpic++ --with-fc=mpif90 --download-mumps --download-scalapack [0]PETSC ERROR: #2 PCFieldSplitSetIS() line 1756 in /home/artur/Rorsrach/Packages/petsc-3.7.2/src/ksp/pc/impls/fieldsplit/fieldsplit.c -------------------------------------------------------------------------------- It seems that the preconditioner object for fieldsplit_P does not know that it is also of fieldsplit type. Does anyone have any idea of how I can specify the proper fieldsplit? Best, Artur

4 9

Error with SuperLU_DIST (mkl related?)
by Eric Chamberland 01 Jan '17

01 Jan '17

Hi, I am just starting to debug a bug encountered with and only with SuperLU_Dist combined with MKL on a 2 processes validation test. (the same test works fine with MUMPS on 2 processes). I just noticed that the SuperLU_Dist version installed by PETSc configure script is 5.1.0 and the latest SuperLU_DIST is 5.1.3. Before going further, I just want to ask: Is there any specific reason to stick to 5.1.0? Here is some more information: On process 2 I have this printed in stdout: Intel MKL ERROR: Parameter 6 was incorrect on entry to DTRSM . and in stderr: Test.ProblemeEFGen.opt: malloc.c:2369: sysmalloc: Assertion `(old_top == (((mbinptr) (((char *) &((av)->bins[((1) - 1) * 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) || ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 *(sizeof(size_t))) - 1)) & ~((2 *(sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long) old_end & pagemask) == 0)' failed. [saruman:15771] *** Process received signal *** This is the 7th call to KSPSolve in the same execution. Here is the last KSPView: KSP Object:(o_slin) 2 MPI processes type: preonly maximum iterations=10000, initial guess is zero tolerances: relative=1e-05, absolute=1e-50, divergence=10000. left preconditioning using NONE norm type for convergence test PC Object:(o_slin) 2 MPI processes type: lu LU: out-of-place factorization tolerance for zero pivot 2.22045e-14 matrix ordering: natural factor fill ratio given 0., needed 0. Factored matrix follows: Mat Object: 2 MPI processes type: mpiaij rows=382, cols=382 package used to perform factorization: superlu_dist total: nonzeros=0, allocated nonzeros=0 total number of mallocs used during MatSetValues calls =0 SuperLU_DIST run parameters: Process grid nprow 2 x npcol 1 Equilibrate matrix TRUE Matrix input mode 1 Replace tiny pivots FALSE Use iterative refinement FALSE Processors in row 2 col partition 1 Row permutation LargeDiag Column permutation METIS_AT_PLUS_A Parallel symbolic factorization FALSE Repeated factorization SamePattern linear system matrix = precond matrix: Mat Object: (o_slin) 2 MPI processes type: mpiaij rows=382, cols=382 total: nonzeros=4458, allocated nonzeros=4458 total number of mallocs used during MatSetValues calls =0 using I-node (on process 0) routines: found 109 nodes, limit used is 5 I know this information is not enough to help debug, but I would like to know if PETSc guys will upgrade to 5.1.3 before trying to debug anything. Thanks, Eric

4 15

Translation and Interpretation Solution !!
by Pooja Singh 30 Dec '16

30 Dec '16

Hello Sir/Ma'am, Hope you are doing best, Need Document Translation, Interpretation or Localization? We provide Translation Services for all type of documents in languages like Russian, Polish, Chinese, German, Swedish, Spanish, Ukrainian, Italian, Arabic, Japanese, Thai, Dutch, French, Vietnamese, Swedish, Korean and regional Languages. Specializations: Medical Translation, Marketing Material Translation, Academic Translation, Book Translation, Financial Translation, Technical Translation, Legal Translation, E-learning course translation, Website and Software Localization. Major clientele: HP, Samsung Engineering, Fluor, ABB Ltd, TOYO, Sulzer, Emerson, TATA, NIIT, Petrofac, BHEL, Siemens Ltd, Larsen & Toubro, Posco, Schneider Electric. If you have translation or interpretation requirements then please share with us in detail. I will be happy to connect with you. Best Regards, Pooja Singh Team-Tr & In Note-If this is not relevant you then revert with "No" in subject line.

1 0

Interaction between multiple DMDA grid
by Somdeb Bandopadhyay 28 Dec '16

28 Dec '16

Hi all, I am trying to modify a single block incompressible solver for flow over open cavity. One way I could do it, is to set part of the domain as flagged and apply velocity field=0 all over it. But it does not seem to be appropriate way (I am wasting alot of grid, just to have a small cavity). So I was wondering, is there any example to interact between two DMDA grids? I am using petsc to solve the poisson equation so if I can just create two set of grid (say DA_grid1 and DA_grid2) and have a interface between them, my problem would be alot more easier. Thanks a lot in advance. Somdeb

2 2