November 2019 - petsc-users - mailman3.cels.anl.gov

Nodes coordinates in distributed dmplex
by Alejandro D Otero 16 Jul '22

16 Jul '22

Hi, sorry if this is an obvious question, but I cannot figure out how to recover finite element nodes coordinates once I have distributed a mesh stored as a dmplex. I am using petsc4py as interface to petsc rutines. I first created a dmplex using: dm.createFromCellList() In a sequential run I got the coordinates with: Coords = dm.getCoordinates() which gave a sequential vector with the coordinates of the mesh nodes. When I distribute the mesh with: dm.distribute() each mpi process has it own dm but the indexing of the vector resulting from getCoordinates() or getCoordinatesLocal() seems not consistent with the local numbering of the cells and nodes. Which is the correct way of doing this in PETSc philosophy? Thanks in advance, Alejandro

4 9

Problem coupling Petsc into OpenFOAM
by Vu Do Quoc 26 Jun '20

26 Jun '20

Hi all, I am trying to insert Petsc to OpenFOAM opensource software. I have been successfully compiling Petsc with an available solver in OpenFOAM by linking it with the shared library libpetsc.so. However, when I call the solver to run a test case, I got an error saying that: "libpetsc.so cannot be found", even though the library still exists in the $PETSC_ARCH/lib folder. I have been struggling for weeks but still, have not been able to figure it out. Therefore I would be very grateful for any suggestion to solve this problem. Thanks in advance for your time, Best regards, Vu Do -- *Vu Q. Do*------------------------------------------------ *Student of Aeronautical Engineering* Programme de Formation d'Ingénieurs d'Excellence au Vietnam *- PFIEV* School of Transportation Engineering Hanoi University of Science and Technology 01 Dai Co Viet Avenue, Hanoi, Vietnam E-mail: vu.doquochust(a)gmail.com

7 11

Converting complex PDE to real for KNL performance ?
by Sajid Ali 15 Apr '20

15 Apr '20

Hi, I'm able to solve the following equation using complex numbers (with ts_type cn and pc_type gamg) : u_t = A*u'' + F_t*u; (where A = -1j/(2k) amd u'' refers to u_xx+u_yy implemented with the familiar 5-point stencil) Now, I want to solve the same problem using real numbers. The equivalent equations are: u_t_real = 1/(2k) * u''_imag + F_real*u_real - F_imag*u_imag u_t_imag = -1/(2k) * u''_real + F_imag*u_real - F_real*u_imag Thus, if we now take our new u vector to have twice the length of the problem we're solving, keeping the first half as real and the second half as imaginary, we'd get a matrix that had matrices computing the laplacian via the 5-point stencil in the top-right and bottom-left corners and a diagonal [F_real+F_imag, F_real-F_imag] term. I tried doing this and the gamg preconditioner complains about an unsymmetric matrix. If i use the default preconditioner, I get DIVERGED_NONLINEAR_SOLVE. Is there a way to better organize the matrix ? PS: I'm trying to do this using only real numbers because I realized that the optimized avx-512 kernels for KNL are not implemented for complex numbers. Would that be implemented soon ? Thank You, Sajid Ali Applied Physics Northwestern University

7 15

Memory optimization
by Perceval Desforges 10 Dec '19

10 Dec '19

Hello all, I am trying to obtain all the eigenvalues in a certain interval for a fairly large matrix (1000000 * 1000000). I therefore use the spectrum slicing method detailed in section 3.4.5 of the manual. The calculations are run on a processor with 20 cores and 96 Go of RAM. The options I use are : -bv_type vecs -eps_krylovschur_detect_zeros 1 -mat_mumps_icntl_13 1 -mat_mumps_icntl_24 1 -mat_mumps_cntl_3 1e-12 However the program quickly crashes with this error: slurmstepd: error: Step 2115.0 exceeded virtual memory limit (312121084 > 107317760), being killed I've tried reducing the amount of memory used by slepc with the -mat_mumps_icntl_14 option by setting it at -70 for example but then I get this error: [1]PETSC ERROR: Error in external library [1]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: INFOG(1)=-9, INFO(2)=82733614 which is an error due to setting the mumps icntl option so low from what I've gathered. Is there any other way I can reduce memory usage? Thanks, Regards, Perceval, P.S. I sent the same email a few minutes ago but I think I made a mistake in the address, I'm sorry if I've sent it twice.

4 13

MatGetDiagonal
by Nachiket Gokhale 09 Dec '19

09 Dec '19

Any chance of making this work in serial? http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatGetDiago… Not a show stopper, I am trying to get the diagonal of some small projected, dense matrices (which come from large sparse matrices). I am running in serial because 1) Since my projected matrices are small, and 2) PETSc does not do certain matrix multiplications involving a dense matrix in parallel, -Nachiket

4 5

Weird behaviour of PCGAMG in coupled poroelasticity
by Felipe Giacomelli 02 Dec '19

02 Dec '19

Hello, I'm trying to solve Biot's poroelasticity (Cryer's sphere problem) through a fully coupled scheme. Thus, the solution of a single linear system yields both displacement and pressure fields, |K L | | u | = |b_u|. |Q (A + H) | | p | = |b_p| The linear system is asymmetric, given that the discrete equations were obtained through the Element based Finite Volume Method (EbFVM). An unstructured tetrahedral grid is utilised, it has about 10000 nodal points (not coarse, nor too refined). Therefore, GMRES and GAMG are employed to solve it. Furthermore, the program was parallelised through a Domain Decomposition Method. Thus, each processor works in its subdomain only. So far, so good. For a given set of poroelastic properties (which are constant throughout time and space), the speedup increases as more processors are utilised: coupling intensity: 7.51e-01 proc solve time [s] 1 314.23 2 171.65 3 143.21 4 149.26 (> 143.21, but ok) However, after making the problem MORE coupled (different poroelastic properties), a strange behavior is observed: coupling intensity: 2.29e+01 proc solve time [s] 1 28909.35 2 192.39 3 181.29 4 14463.63 Recalling that GMRES and GAMG are used, KSP takes about 4300 iterations to converge when 1 processor is employed. On the other hand, for 2 processors, KSP takes around 30 iterations to reach convergence. Hence, explaining the difference between the solution times. Increasing the coupling even MORE, everything goes as expected: coupling intensity: 4.63e+01 proc solve time [s] 1 229.26 2 146.04 3 121.49 4 107.80 Because of this, I ask: * What may be the source of this behavior? Can it be predicted? * How can I remedy this situation? At last, are there better solver-pc choices for coupled poroelasticity? Thank you, Felipe

3 3

Floating point exception
by baikadi pranay 01 Dec '19

01 Dec '19

Hello PETSc users, I am currently trying to build a 1-D Schrodinger solver. I have built my hamiltonian matrix (of size 121 x 121) and i'm trying to find the eigenvalues. I have the following lines of code for the solver: *call EPSCreate(PETSC_COMM_WORLD,eps,ierr)* *call EPSSetOperators(eps,ham,S,ierr)call EPSSetProblemType(eps,EPS_GHEP,ierr)* *call EPSSetFromOptions(eps,ierr)call EPSSetDimensions(eps,10,PETSC_DEFAULT_INTEGER,PETSC_DEFAULT_INTEGER,ierr)call EPSSolve(eps,ierr)call EPSDestroy(eps,ierr)* At the EPSSolve line, i get the following error: *[0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------[0]PETSC ERROR: Floating point exception[0]PETSC ERROR: Vec entry at local location 0 is not-a-number or infinite at end of function: Parameter number 3[0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html <http://www.mcs.anl.gov/petsc/documentation/faq.html> for trouble shooting.[0]PETSC ERROR: Petsc Release Version 3.11.1, Apr, 12, 2019* I am using the options *-st_pc_factor_shift_type NONZERO -st_pc_factor_shift_amount 1* ( else I end up getting the "zero pivot in LU factorization" error ). I outputted my matrix to matlab and confirmed that the null space is empty and the matrix is not singular. I am not sure why I'm getting this error. Could you provide me a hint as to how to solve this problem. Sincerely, Pranay.

3 5

Outputting matrix for viewing in matlab
by baikadi pranay 30 Nov '19

30 Nov '19

Hello PETSc users, I have a sparse matrix built and I want to output the matrix for viewing in matlab. However i'm having difficulty outputting the matrix. I am writing my program in Fortran90 and I've included the following lines to output the matrix. *call PetscViewerBinaryOpen(PETSC_COMM_SELF,'matrix',FILE_MODE_WRITE,view,ierr) call PetscViewerBinaryGetDescriptor(view,fd,ierr) call PetscBinaryWrite(fd,ham,1,PETSC_SCALAR,PETSC_FALSE,ierr)* These lines do create a matrix but matlab says its not a binary file. Could you please provide me some inputs on where I'm going wrong and how to proceed with this problem. I can provide any further information that you might need to help me solve this problem. Thank you. Sincerely, Pranay.

3 4

Domain decomposition using DMPLEX
by Swarnava Ghosh 29 Nov '19

29 Nov '19

Dear PETSc users and developers, I am working with dmplex to distribute a 3D unstructured mesh made of tetrahedrons in a cuboidal domain. I had a few queries: 1) Is there any way of ensuring load balancing based on the number of vertices per MPI process. 2) As the global domain is cuboidal, is the resulting domain decomposition also cuboidal on every MPI process? If not, is there a way to ensure this? For example in DMDA, the default domain decomposition for a cuboidal domain is cuboidal. Sincerely, SG

4 9

Petsc Matrix modifications
by Brandon Denton 26 Nov '19

26 Nov '19

Good Morning, Is it possible to expand a matrix in petsc? I current created and loaded a matrix (6 x 5) which holds information required later in my program. I would like to store additional information in the matrix by expanding its size, let's say make it at 10 x 5 matrix. How is this accomplished in petsc. When I try to use MatSetSize() or MatSetValue() my code throws errors. What is the process for accomplishing this? Thank you in advance for your time. Brandon

2 1