May 2017 - petsc-users - mailman3.cels.anl.gov

Nodes coordinates in distributed dmplex
by Alejandro D Otero 16 Jul '22

16 Jul '22

Hi, sorry if this is an obvious question, but I cannot figure out how to recover finite element nodes coordinates once I have distributed a mesh stored as a dmplex. I am using petsc4py as interface to petsc rutines. I first created a dmplex using: dm.createFromCellList() In a sequential run I got the coordinates with: Coords = dm.getCoordinates() which gave a sequential vector with the coordinates of the mesh nodes. When I distribute the mesh with: dm.distribute() each mpi process has it own dm but the indexing of the vector resulting from getCoordinates() or getCoordinatesLocal() seems not consistent with the local numbering of the cells and nodes. Which is the correct way of doing this in PETSc philosophy? Thanks in advance, Alejandro

4 9

MatGetDiagonal
by Nachiket Gokhale 09 Dec '19

09 Dec '19

Any chance of making this work in serial? http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatGetDiago… Not a show stopper, I am trying to get the diagonal of some small projected, dense matrices (which come from large sparse matrices). I am running in serial because 1) Since my projected matrices are small, and 2) PETSc does not do certain matrix multiplications involving a dense matrix in parallel, -Nachiket

4 5

GAMG scaling
by Mark Adams 24 Dec '18

24 Dec '18

(Hong), what is the current state of optimizing RAP for scaling? Nate, is driving 3D elasticity problems at scaling with GAMG and we are working out performance problems. They are hitting problems at ~1.5B dof problems on a basic Cray (XC30 I think). Thanks, Mark

7 28

monitoring the convergence of fieldsplit 0 and 1
by Klaij, Christiaan 15 May '18

15 May '18

A question when using PCFieldSplit: for each linear iteration of the system, how many iterations for fielsplit 0 and 1? One way to find out is to run with -ksp_monitor, -fieldsplit_0_ksp_monitor and -fieldsplit_0_ksp_monitor. This gives the complete convergence history. Another way, suggested by Matt, is to use -ksp_monitor, -fieldsplit_0_ksp_converged_reason and -fieldsplit_1_ksp_converged_reason. This gives only the totals for fieldsplit 0 and 1 (but without saying for which one). Both ways require to somehow process the output, which is a bit inconvenient. Could KSPGetResidualHistory perhaps return (some) information on the subsystems' convergence for processing inside the code? Chris dr. ir. Christiaan Klaij CFD Researcher Research & Development E mailto:[email protected] T +31 317 49 33 44 MARIN 2, Haagsteeg, P.O. Box 28, 6700 AA Wageningen, The Netherlands T +31 317 49 39 11, F +31 317 49 32 45, I www.marin.nl

4 19

Question about memory usage in Multigrid preconditioner
by frank 22 Jan '18

22 Jan '18

Hi, I am using the CG ksp solver and Multigrid preconditioner to solve a linear system in parallel. I chose to use the 'Telescope' as the preconditioner on the coarse mesh for its good performance. The petsc options file is attached. The domain is a 3d box. It works well when the grid is 1536*128*384 and the process mesh is 96*8*24. When I double the size of grid and keep the same process mesh and petsc options, I get an "out of memory" error from the super-cluster I am using. Each process has access to at least 8G memory, which should be more than enough for my application. I am sure that all the other parts of my code( except the linear solver ) do not use much memory. So I doubt if there is something wrong with the linear solver. The error occurs before the linear system is completely solved so I don't have the info from ksp view. I am not able to re-produce the error with a smaller problem either. In addition, I tried to use the block jacobi as the preconditioner with the same grid and same decomposition. The linear solver runs extremely slow but there is no memory error. How can I diagnose what exactly cause the error? Thank you so much. Frank

7 56

DMPlex distribution with FVM adjacency
by Lawrence Mitchell 04 Aug '17

04 Aug '17

Dear petsc-users, I am trying to distribute a triangle mesh with a cell halo defined by FVM adjacency (i.e. if I have a facet in the initial (0-overlap) distribution, I want the cell on the other side of it). Reading the documentation, I think I do: DMPlexSetAdjacencyUseCone(PETSC_TRUE) DMPlexSetAdjacencyUseClosure(PETSC_FALSE) and then DMPlexDistribute(..., ovelap=1) If I do this for a simple mesh and then try and do anything on it, I run into all sorts of problems because I have a plex where I have some facets, but not even one cell in the support of the facet. Is this to be expected? For example the following petsc4py code breaks when run on 3 processes: $ mpiexec -n 3 python bork.py [1] DMPlexGetOrdering() line 133 in /data/lmitche1/src/deps/petsc/src/dm/impls/plex/plexreorder.c [1] DMPlexCreateOrderingClosure_Static() line 41 in /data/lmitche1/src/deps/petsc/src/dm/impls/plex/plexreorder.c [1] Petsc has generated inconsistent data [1] Number of depth 2 faces 34 does not match permuted nubmer 29 : error code 77 [2] DMPlexGetOrdering() line 133 in /data/lmitche1/src/deps/petsc/src/dm/impls/plex/plexreorder.c [2] DMPlexCreateOrderingClosure_Static() line 41 in /data/lmitche1/src/deps/petsc/src/dm/impls/plex/plexreorder.c [2] Petsc has generated inconsistent data [2] Number of depth 2 faces 33 does not match permuted nubmer 28 : error code 77 [0] DMPlexGetOrdering() line 133 in /data/lmitche1/src/deps/petsc/src/dm/impls/plex/plexreorder.c [0] DMPlexCreateOrderingClosure_Static() line 41 in /data/lmitche1/src/deps/petsc/src/dm/impls/plex/plexreorder.c [0] Petsc has generated inconsistent data [0] Number of depth 2 faces 33 does not match permuted nubmer 31 $ cat > bork.py<<\EOF from petsc4py import PETSc import numpy as np Lx = Ly = 1 nx = ny = 4 xcoords = np.linspace(0.0, Lx, nx + 1, dtype=PETSc.RealType) ycoords = np.linspace(0.0, Ly, ny + 1, dtype=PETSc.RealType) coords = np.asarray(np.meshgrid(xcoords, ycoords)).swapaxes(0, 2).reshape(-1, 2) # cell vertices i, j = np.meshgrid(np.arange(nx, dtype=PETSc.IntType), np.arange(ny, dtype=PETSc.IntType)) cells = [i*(ny+1) + j, i*(ny+1) + j+1, (i+1)*(ny+1) + j+1, (i+1)*(ny+1) + j] cells = np.asarray(cells, dtype=PETSc.IntType).swapaxes(0, 2).reshape(-1, 4) idx = [0, 1, 3, 1, 2, 3] cells = cells[:, idx].reshape(-1, 3) comm = PETSc.COMM_WORLD if comm.rank == 0: dm = PETSc.DMPlex().createFromCellList(2, cells, coords, comm=comm) else: dm = PETSc.DMPlex().createFromCellList(2, np.zeros((0, 4), dtype=PETSc.IntType), np.zeros((0, 2), dtype=PETSc.RealType), comm=comm) dm.setAdjacencyUseClosure(False) dm.setAdjacencyUseCone(True) dm.distribute(overlap=1) dm.getOrdering(PETSc.Mat.OrderingType.RCM) dm.view() EOF Am I doing something wrong? Is this not expected to work? Cheers, Lawrence

4 22

specifying vertex coordinates using DMPlexCreateFromCellListParallel
by Hassan Raiesi 29 Jun '17

29 Jun '17

Hello, I'm trying to use DMPlexCreateFromCellListParallel to create a DM from an already partitioned mesh, It requires an array of numVertices*spaceDim numbers, but how should one order the coordinates of the vertices? we only pass the global vertex numbers using 'const int cells[]' to define the cell-connectivity, so passing the vertex coordinates in local ordering wouldn't make sense? If it needs to be in global ordering, should I sort the global index of the node numbers owned by each rank (as they wont be continuous). Thank you Hassan Raiesi, Bombardier Aerospace www.bombardier.com

3 14

PetscFECreateDefault in Fortran
by Justin Pogacnik 02 Jun '17

02 Jun '17

Hello, I'm developing a finite element code in fortran 90. I recently updated my PETSc and am now getting the following error during compile/linking on an existing application: Undefined symbols for architecture x86_64: "_petscfecreatedefault_", referenced from: _MAIN__ in fe_test.o ld: symbol(s) not found for architecture x86_64 collect2: error: ld returned 1 exit status make: *** [dist/fe_test] Error 1 I'm running Mac OS X Yosemite (10.10.5). I've created a "minimum working example" (attached) that re-creates the problem. It's basically just dm/impls/plex/examples/tutorials/ex3f90, but tries to create a PetscFE object. Everything goes fine and the DM looks like what is expected if PetscFECreateDefault is commented out. Any idea what am I missing? Many thanks! Justin

2 9

PETSc User Meeting 2017, June 14-16 in Boulder, Colorado
by Jed Brown 01 Jun '17

01 Jun '17

We'd like to invite you to join us at the 2017 PETSc User Meeting held at the University of Colorado Boulder on June 14-16, 2017. http://www.mcs.anl.gov/petsc/meetings/2017/ The first day consists of tutorials on various aspects and features of PETSc. The second and third days will be devoted to exchange, discussions, and a refinement of strategies for the future with our users. We encourage you to present work illustrating your own use of PETSc, for example in applications or in libraries built on top of PETSc. Registration for the PETSc User Meeting 2017 is free for students and $75 for non-students. We can host a maximum of 150 participants, so register soon (and by May 15). http://www.eventzilla.net/web/e/petsc-user-meeting-2017-2138890185 We are also offering low-cost lodging on campus. A lodging registration site will be available soon and announced here and on the website. Thanks to the generosity of Intel, we will be able to offer a limited number of student travel grants. We are also soliciting additional sponsors -- please contact us if you are interested. We are looking forward to seeing you in Boulder! Please contact us at petsc2017(a)mcs.anl.gov if you have any questions or comments.

3 5

slepc on 1D row distributed matrix
by Kannan, Ramakrishnan 31 May '17

31 May '17

Hello, I have got a sparse 1D row distributed matrix in which every MPI process owns an m/p x n of the global matrix mxn. I am running NHEP with krylovschur on it. It is throwing me some wrong error. For your reference, I have attached the modified ex5.c in which I SetSizes on the matrix to emulate the 1D row distribution and the log file with the error. In the unmodified ex5.c, for m=5, N=15, the local_m and the local_n is 3x3. How is the global 15x15 matrix distributed locally as 3x3 matrices? When I print the global matrix, it doesn’t appear to be diagonal as well. If slepc doesn’t support sparse 1D row distributed matrix, how do I need to redistribute it such that I can run NHEP on this. -- Regards, Ramki

3 3