May 2021 - petsc-users - mailman3.cels.anl.gov

Nodes coordinates in distributed dmplex
by Alejandro D Otero 16 Jul '22

16 Jul '22

Hi, sorry if this is an obvious question, but I cannot figure out how to recover finite element nodes coordinates once I have distributed a mesh stored as a dmplex. I am using petsc4py as interface to petsc rutines. I first created a dmplex using: dm.createFromCellList() In a sequential run I got the coordinates with: Coords = dm.getCoordinates() which gave a sequential vector with the coordinates of the mesh nodes. When I distribute the mesh with: dm.distribute() each mpi process has it own dm but the indexing of the vector resulting from getCoordinates() or getCoordinatesLocal() seems not consistent with the local numbering of the cells and nodes. Which is the correct way of doing this in PETSc philosophy? Thanks in advance, Alejandro

4 9

Question about PCFieldSplit
by Jorti, Zakariae 31 Jan '22

31 Jan '22

Hello, I have a DMStag grid with one dof on each edge and face center. I want to use a PCFieldSplit preconditioner on a Jacobian matrix that I assume is already split but I am not sure how to determine the fields. In the DMStag examples (ex2.c and ex3.c), the function PCFieldSplitSetDetectSaddlePoint is used to determine those fields based on zero diagonal entries. In my case, I have a Jacobian matrix that does not have zero diagonal entries. Can I use that PCFieldSplitSetDetectSaddlePoint in this case? If not, how should I do? Should I do like this example (https://www.mcs.anl.gov/petsc/petsc-master/src/ksp/ksp/tutorials/ex43.c.html) const PetscInt Bfields[1] = {0},Efields[1] = {1}; KSPGetPC(ksp,&pc); PCFieldSplitSetBlockSize(pc,2); PCFieldSplitSetFields(pc,"B",1,Bfields,Bfields); PCFieldSplitSetFields(pc,"E",1,Efields,Efields); where my B unknowns are defined on face centers and E unknowns are defined on edge centers? One last thing, I do not know which field comes first. Is it the one defined for face dofs or edge dofs. Thank you. Best regards, Zakariae

4 14

PETSC installation on Cray
by Enrico 13 Sep '21

13 Sep '21

Hi, I'm having some problems installing PETSC with Cray compiler. I use this configuration: ./configure --with-cc=cc --with-cxx=CC --with-fc=0 --with-debugging=1 --with-shared-libraries=1 COPTFLAGS=-O0 CXXOPTFLAGS=-O0 and when I do make all I get the following error because of cmathcalls.h: CC-1043 craycc: ERROR File = /usr/include/bits/cmathcalls.h, Line = 55 _Complex can only be used with floating-point types. __MATHCALL (cacos, (_Mdouble_complex_ __z)); ^ Am I doing something wrong? Regards, Enrico Degregori

4 5

Rather different matrix product results on multiple processes
by Peder Jørgensgaard Olesen 03 Jun '21

03 Jun '21

Hello, When computing a matrix product of the type R = D.DT using MatMatTransposeMult() I find I get rather different results depending on the number of processes. In one example using a data set that is small compared to the application I get Frobenius norms |R| = 1.047e3 on a single process, 1.0363e3 on a single HPC node (40 cores), and 9.7307e2 on two nodes. I have ascertained that the single process result is indeed the correct one (i.e., eigenvectors of R form a proper basis for the columns of D), so naturally I'd love to be able to reproduce this result across different parallel setups. How might I achieve this? I'm attaching MWE code and the data set used for the example. Thanks in advance! Best Regards Peder Jørgensgaard Olesen PhD Student, Turbulence Research Lab Dept. of Mechanical Engineering Technical University of Denmark Niels Koppels Allé Bygning 403, Rum 105 DK-2800 Kgs. Lyngby

5 7

SLEPc: non-real singular vectors from SVD of real matrix
by Peder Jørgensgaard Olesen 02 Jun '21

02 Jun '21

Hello I've noticed that doing a singular value decomposition of a real matrix appears to result in non-real singular vectors. This should not be the case - singular vectors of a real matrix must be real-valued. In the example attached I read a matrix from a binary file (also attached), perform the SVD, and write singular vectors to an HDF5 file which I subsequently inspect using h5dump, revealing non-zero imaginary parts of vector elements as highlighted below: [pjool@svol mwes]$ h5dump -d U0_sample_svd -c "5,2" sample_svecs.h5 HDF5 "sample_svecs.h5" { DATASET "U0_sample_svd" { DATATYPE H5T_IEEE_F64LE DATASPACE SIMPLE { ( 882, 2 ) / ( 882, 2 ) } SUBSET { START ( 0, 0 ); STRIDE ( 1, 1 ); COUNT ( 5, 2 ); BLOCK ( 1, 1 ); DATA { (0,0): 0.0226108, 0.0299595, (1,0): 0.035414, 0.0469237, (2,0): 0.0276317, 0.0366122, (3,0): 0.0145344, 0.0192581, (4,0): 0.0110376, 0.0146249 } } ATTRIBUTE "complex" { DATATYPE H5T_STD_I32LE DATASPACE SCALAR DATA { (0): 1 } } } } I also extract the imaginary part of the input matrix and print its norm to ensure that the matrix is indeed real. I'm running v3.14, but I don't believe that alone should cause the issue, since it what I'm trying to do appears like a rather common and basic task. What might be the reason behind this behavior, and what can be done to resolve it? Med venlig hilsen / Best Regards Peder Jørgensgaard Olesen PhD Student, Turbulence Research Lab Dept. of Mechanical Engineering Technical University of Denmark Niels Koppels Allé Bygning 403, Rum 105 DK-2800 Kgs. Lyngby

3 4

Performing a coordinate system rotation for the stiffness matrix
by Michael Wick 31 May '21

31 May '21

Hi PETSc team: I am considering implementing a skew roller boundary condition for my elasticity problem. The method is based on this journal paper: http://inside.mines.edu/~vgriffit/pubs/All_J_Pubs/18.pdf Or you may find the method in the attached Bathe's slides, pages 9 -10. Roughly speaking, a (very) sparse matrix T will be created which takes the shape [ I, O; O, R], where R is a 3x3 rotation matrix. And the original linear problem K U = F will be modified into (T^t K T) (T^t U) = T^t F. In doing so, one can enforce a roller boundary condition on a slanted surface. I think it can be an easy option if I can generate the T matrix and do two matrix multiplications to get T^t K T. I noticed that there is a MatPtAP function. Yet, after reading a previous discussion, it seems that this function is not designed for this purposes ( https://lists.mcs.anl.gov/pipermail/petsc-users/2018-June/035477.html) I assume I can only call MatMatMult & MatTransposeMatMult to do this job, correct? Is there any existingly PETSc function to do T^t K T in one call? Thanks, Mike

6 5

Collect Trajectories components
by Francesco Brarda 31 May '21

31 May '21

Hi! I solved an ODE system with TS. Now I would like to save one of the trajectories in specific times. To do so, I used TSTrajectoryGetVecs. The values of the variable I am interested in is on one processor. I want to collect these values in a parallel vector, but I had the error: [0]PETSC ERROR: Invalid argument [0]PETSC ERROR: Real value must be same on all processes, argument # 2 [0]PETSC ERROR: See https://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [0]PETSC ERROR: Petsc Release Version 3.14.4, unknown [0]PETSC ERROR: ./petsc_sir on a arch-debug named srvulx13 by fbrarda Wed May 26 12:00:42 2021 [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --with-openblas-dir=/opt/packages/openblas/0.2.13-gcc --download-mpich PETSC_ARCH=arch-debug [0]PETSC ERROR: #1 TSHistoryGetLocFromTime() line 134 in /home/fbrarda/petsc/src/ts/interface/tshistory.c [0]PETSC ERROR: #2 TSTrajectoryReconstruct_Private() line 55 in /home/fbrarda/petsc/src/ts/trajectory/utils/reconstruct.c [0]PETSC ERROR: #3 TSTrajectoryGetVecs() line 239 in /home/fbrarda/petsc/src/ts/trajectory/interface/traj.c Is there any specific routine I can use to overcome this issue? Should I use VecScatter? I hope I made myself clear. Best, Francesco

3 9

CUDA MatSetValues test
by Mark Adams 30 May '21

30 May '21

Is there a test with MatSetValues and CUDA?

3 21

strange segv
by Mark Adams 29 May '21

29 May '21

I am running on Summit with Kokkos-CUDA and I am getting a segv that looks like some sort of a compile/link mismatch. I also have a user with a C++ code that is getting strange segvs when calling MatSetValues with CUDA (I know MatSetValues is not a cupsarse method, but that is the report that I have). I have no idea if these are related but they both involve C -- C++ calls ... I started with a clean build (attached) and I ran in DDT. DDT stopped at the call in plexland.c to the KokkosLanau operator. I stepped into this function and then took this screenshot of the stack, with the Kokkos call and PETSc signal handler. Make check does not seem to be running Kokkos tests: 15:02 adams/landau-mass-opt *= /gpfs/alpine/csc314/scratch/adams/petsc$ make PETSC_DIR=/gpfs/alpine/csc314/scratch/adams/petsc PETSC_ARCH=arch-summit-opt-gnu-kokkos-notpl-cuda10 check Running check examples to verify correct installation Using PETSC_DIR=/gpfs/alpine/csc314/scratch/adams/petsc and PETSC_ARCH=arch-summit-opt-gnu-kokkos-notpl-cuda10 C/C++ example src/snes/tutorials/ex19 run successfully with 1 MPI process C/C++ example src/snes/tutorials/ex19 run successfully with 2 MPI processes C/C++ example src/snes/tutorials/ex19 run successfully with cuda Completed test examples Also, I ran this AM with another branch that had not been rebased with main as recently as this branch (adams/landau-mass-opt). Any ideas?

2 2

building with PGI and Kokkos
by Mark Adams 29 May '21

29 May '21

I am trying to build with Kokkos and PGI on Summit and I can't seem to get Kokkos to accept that the C++ compiler supports C++14. Any ideas? Mark

3 3