January 2024 - petsc-users - mailman3.cels.anl.gov

KSP has an extra iteration when use shell matrix
by Yi Hu 06 Feb '24

06 Feb '24

Dear PETSc team, I am still trying to sort out my previous thread https://lists.mcs.anl.gov/pipermail/petsc-users/2024-January/050079.html using a minimal working example. However, I encountered another problem. Basically I combined the basic usage of SNES solver and shell matrix and tried to make it work. The jacobian of my snes is replaced by a customized MATOP_MULT. The minimal example code can be viewed here https://github.com/hyharry/small_petsc_test/blob/master/test_shell_jac/ex1f… When running with -ksp_monitor -ksp_converged_reason, it shows an extra mymult step, and hence ruin my previous converged KSP result. Implement a customized converged call-back also does not help. I am wondering how to skip this extra ksp iteration. Could anyone help me on this? Thanks for your help. Best wishes, Yi ------------------------------------------------- Stay up to date and follow us on LinkedIn, Twitter and YouTube. Max-Planck-Institut für Eisenforschung GmbH Max-Planck-Straße 1 D-40237 Düsseldorf Handelsregister B 2533 Amtsgericht Düsseldorf Geschäftsführung Prof. Dr. Gerhard Dehm Prof. Dr. Jörg Neugebauer Prof. Dr. Dierk Raabe Dr. Kai de Weldige Ust.-Id.-Nr.: DE 11 93 58 514 Steuernummer: 105 5891 1000 Please consider that invitations and e-mails of our institute are only valid if they end with …(a)mpie.de. If you are not sure of the validity please contact rco(a)mpie.de Bitte beachten Sie, dass Einladungen zu Veranstaltungen und E-Mails aus unserem Haus nur mit der Endung …(a)mpie.de gültig sind. In Zweifelsfällen wenden Sie sich bitte an rco(a)mpie.de -------------------------------------------------

2 9

errors with hypre with MPI and multiple GPUs on a node
by Yesypenko, Anna 04 Feb '24

04 Feb '24

Dear Petsc devs, I'm encountering an error running hypre on a single node with multiple GPUs. The issue is in the setup phase. I'm trying to troubleshoot, but don't know where to start. Are the system routines PetScCUDAInitialize and PetScCUDAInitializeCheck available in python? How do I verify that GPUs are assigned properly to each MPI process? In this case, I have 3 tasks and 3 GPUs. The code works with pc-type hypre on a single GPU. Any suggestions are appreciated! Below is the error trace: `` TACC: Starting up job 1490124 TACC: Setting up parallel environment for MVAPICH2+mpispawn. TACC: Starting parallel tasks... [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [0]PETSC ERROR: or see https://petsc.org/release/faq/#valgrind and https://petsc.org/release/faq/ [0]PETSC ERROR: or try https://docs.nvidia.com/cuda/cuda-memcheck/index.html on NVIDIA CUDA systems to find memory corruption errors [0]PETSC ERROR: --------------------- Stack Frames ------------------------------------ [0]PETSC ERROR: The line numbers in the error traceback are not always exact. [0]PETSC ERROR: #1 hypre_ParCSRMatrixMigrate() [0]PETSC ERROR: #2 MatBindToCPU_HYPRE() at /work/06368/annayesy/ls6/petsc/src/mat/impls/hypre/mhypre.c:1394 [0]PETSC ERROR: #3 MatAssemblyEnd_HYPRE() at /work/06368/annayesy/ls6/petsc/src/mat/impls/hypre/mhypre.c:1471 [0]PETSC ERROR: #4 MatAssemblyEnd() at /work/06368/annayesy/ls6/petsc/src/mat/interface/matrix.c:5773 [0]PETSC ERROR: #5 MatConvert_AIJ_HYPRE() at /work/06368/annayesy/ls6/petsc/src/mat/impls/hypre/mhypre.c:660 [0]PETSC ERROR: #6 MatConvert() at /work/06368/annayesy/ls6/petsc/src/mat/interface/matrix.c:4421 [0]PETSC ERROR: #7 PCSetUp_HYPRE() at /work/06368/annayesy/ls6/petsc/src/ksp/pc/impls/hypre/hypre.c:245 [0]PETSC ERROR: #8 PCSetUp() at /work/06368/annayesy/ls6/petsc/src/ksp/pc/interface/precon.c:1080 [0]PETSC ERROR: #9 KSPSetUp() at /work/06368/annayesy/ls6/petsc/src/ksp/ksp/interface/itfunc.c:415 [0]PETSC ERROR: #10 KSPSolve_Private() at /work/06368/annayesy/ls6/petsc/src/ksp/ksp/interface/itfunc.c:833 [0]PETSC ERROR: #11 KSPSolve() at /work/06368/annayesy/ls6/petsc/src/ksp/ksp/interface/itfunc.c:1080 application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0 `` Below is a minimum working example: `` import numpy,petsc4py,sys,time petsc4py.init(sys.argv) from petsc4py import PETSc from time import time n = int(5e5); comm = PETSc.COMM_WORLD pA = PETSc.Mat(comm=comm) pA.create(comm=comm) pA.setSizes((n,n)) pA.setType(PETSc.Mat.Type.AIJ) pA.setPreallocationNNZ(3) rstart,rend=pA.getOwnershipRange() print("\t Processor %d of %d gets indices %d:%d"%(comm.Get_rank(),comm.Get_size(),rstart,rend)) if (rstart == 0): pA.setValue(0,0,2); pA.setValue(0,1,-1) if (rend == n): pA.setValue(n-1,n-2,-1); pA.setValue(n-1,n-1,2) for index in range(rstart,rend): if (rstart > 0): pA.setValue(index,index-1,-1) pA.setValue(index,index,2) if (rend < n): pA.setValue(index,index+1,-1) pA.assemble() pA = pA.convert(mat_type='aijcusparse') px,pb = pA.createVecs() pb.set(1.0); px.set(1.0) ksp = PETSc.KSP().create() ksp.setOperators(pA) ksp.setConvergenceHistory() ksp.setType('cg') ksp.getPC().setType('hypre') ksp.setTolerances(rtol=1e-10) ksp.solve(pb, px) # error is generated here `` Best, Anna

3 11

Preconditioning of Liouvillian Superoperator
by Niclas Götting 01 Feb '24

01 Feb '24

Hi all, I've been trying for the last couple of days to solve a linear system using iterative methods. The system size itself scales exponentially (64^N) with the number of components, so I receive sizes of * (64, 64) for one component * (4096, 4096) for two components * (262144, 262144) for three components I can solve the first two cases with direct solvers and don't run into any problems; however, the last case is the first nontrivial and it's too large for a direct solution, which is why I believe that I need an iterative solver. As I know the solution for the first two cases, I tried to reproduce them using GMRES and failed on the second, because GMRES didn't converge and seems to have been going in the wrong direction (the vector to which it "tries" to converge is a totally different one than the correct solution). I went as far as -ksp_max_it 1000000, which takes orders of magnitude longer than the LU solution and I'd intuitively think that GMRES should not take *that* much longer than LU. Here is the information I have about this (4096, 4096) system: * not symmetric (which is why I went for GMRES) * not singular (SVD: condition number 1.427743623238e+06, 0 of 4096 singular values are (nearly) zero) * solving without preconditioning does not converge (DIVERGED_ITS) * solving with iLU and natural ordering fails due to zeros on the diagonal * solving with iLU and RCM ordering does not converge (DIVERGED_ITS) After some searching I also found [this](http://arxiv.org/abs/1504.06768) paper, which mentions the use of ILUTP, which I believe in PETSc should be used via hypre, which, however, threw a SEGV for me, and I'm not sure if it's worth debugging at this point in time, because I might be missing something entirely different. Does anybody have an idea how this system could be solved in finite time, such that the method also scales to the three component problem? Thank you all very much in advance! Best regards Niclas

4 6

PETSc init question
by Alain O' Miniussi 31 Jan '24

31 Jan '24

Hi, It is indicated in: https://petsc.org/release/manualpages/Sys/PetscInitialize/ that the init function will call MPI_Init. What if MPI_Init was already called (as it is the case in my application) and what about MPI_Init_thread. Wouldn't it be more convenient to have a Petsc init function taking a already initialized communicator as argument ? Also, that initialization seems to imply that it is not possible to have multiple instance of PETSc on different communicators. Is that the case ? Thanks ---- Alain Miniussi DSI, Pôles Calcul et Genie Log. Observatoire de la Côte d'Azur Tél. : +33609650665

3 2

Parallel vector layout for TAO optimization with separable state/design structure
by Guenther, Stefanie 30 Jan '24

30 Jan '24

Hi Petsc team, I have a question regarding parallel layout of a Petsc vector to be used in TAO optimizers for cases where the optimization variables split into ‘design’ and ‘state’ variables (e.g. such as in PDE-constrained optimization as in tao_lcl). In our case, the state variable naturally parallelizes evenly amongst multiple processors and this distribution is fixed. The ‘design’ vector however does not, it is very short compared to the state vector and it is required on all state-processors when evaluating the objective function and gradient. My question would be how the TAO optimization vector x = [design,state] should be created in such a way that the ‘state’ part is distributed as needed in our solver, while the design part is not. My only idea so far was to copy the design variables to all processors and augment / interleave the optimization vector as x = [state_proc1,design, state_proc2, design, … ] . When creating this vector in parallel on PETSC_COMM_WORLD, each processor would then own the same number of variables ( [state_proc<i>, design] ), as long as the numbers match up, and I would only need to be careful when gathering the gradient wrt the design parts from all processors. This seems cumbersome however, and I would be worried whether the optimization problem is harder to solve this way. Is there any other way to achieve this splitting, that I am missing here? Note that the distribution of the state itself is given and can not be changed, and that the state vs design vectors have very different (and independent) dimensions. Thanks for your help and thoughts! Best, Stefanie

3 2

pc_redistribute issue
by Mark Adams 29 Jan '24

29 Jan '24

I am not getting ksp_rtol 1e-12 into pc_redistribute correctly? * Linear redistribute_ solve converged due to CONVERGED_RTOL iterations 1 0 KSP Residual norm 2.182384017537e+02 1 KSP Residual norm 1.889764161573e-04 * Number of iterations = 1 N = 47628 Residual norm 8.65917e-07 #PETSc Option Table entries: -f S.bin # (source: command line) -ksp_monitor # (source: command line) -ksp_type preonly # (source: command line) -mat_block_size 36 # (source: command line) -mat_view ascii::ascii_info # (source: command line) -options_left # (source: command line) -pc_type redistribute # (source: command line) -redistribute_ksp_converged_reason # (source: command line) *-redistribute_ksp_rtol 1e-12 # (source: command line)*-redistribute_ksp_type gmres # (source: command line) -redistribute_pc_type bjacobi # (source: command line) -redistribute_sub_pc_factor_mat_solver_type mumps # (source: command line) -redistribute_sub_pc_type lu # (source: command line) #End of PETSc Option Table entries *There are no unused options.*

3 6

Bug in VecNorm, 3.20.3
by michael＠paraffinalia.co.uk 26 Jan '24

26 Jan '24

Hello, I have used the GMRES solver in PETSc successfully up to now, but on installing the most recent release, 3.20.3, the solver fails by exiting early. Output from the code is: lt-nbi-solve-laplace: starting PETSc solver [23.0537] 0 KSP Residual norm < 1.e-11 Linear solve converged due to CONVERGED_ATOL iterations 0 lt-nbi-solve-laplace: 0 iterations [23.0542] (22.9678) and tracing execution shows the norm returned by VecNorm to be 0. If I modify the function by commenting out line 217 of src/vec/vec/interface/rvector.c /* if (flg) PetscFunctionReturn(PETSC_SUCCESS); */ the code executes correctly: lt-nbi-solve-laplace: starting PETSc solver [22.9392] 0 KSP Residual norm 1.10836 1 KSP Residual norm 0.0778301 2 KSP Residual norm 0.0125121 3 KSP Residual norm 0.00165836 4 KSP Residual norm 0.000164066 5 KSP Residual norm 2.12824e-05 6 KSP Residual norm 4.50696e-06 7 KSP Residual norm 5.85082e-07 Linear solve converged due to CONVERGED_RTOL iterations 7 My compile options are: PETSC_ARCH=linux-gnu-real ./configure --with-mpi=0 --with-scalar-type=real --with-threadsafety --with-debugging=0 --with-log=0 --with-openmp uname -a returns: 5.15.80 #1 SMP PREEMPT Sun Nov 27 13:28:05 CST 2022 x86_64 Intel(R) Core(TM) i5-6200U CPU @ 2.30GHz GenuineIntel GNU/Linux

5 7

Trying to understand -log_view when using HIP kernels (ex34)
by Dave May 26 Jan '24

26 Jan '24

Hi all, I am trying to understand the logging information associated with the %flops-performed-on-the-gpu reported by -log_view when running src/ksp/ksp/tutorials/ex34 with the following options -da_grid_x 192 -da_grid_y 192 -da_grid_z 192 -dm_mat_type seqaijhipsparse -dm_vec_type seqhip -ksp_max_it 10 -ksp_monitor -ksp_type richardson -ksp_view -log_view -mg_coarse_ksp_max_it 2 -mg_coarse_ksp_type richardson -mg_coarse_pc_type none -mg_levels_ksp_type richardson -mg_levels_pc_type none -options_left -pc_mg_levels 3 -pc_mg_log -pc_type mg This config is not intended to actually solve the problem, rather it is a stripped down set of options designed to understand what parts of the smoothers are being executed on the GPU. With respect to the log file attached, my first set of questions related to the data reported under "Event Stage 2: MG Apply". [1] Why is the log littered with nan's? * I don't understand how and why "GPU Mflop/s" should be reported as nan when a value is given for "GPU %F" (see MatMult for example). * For events executed on the GPU, I assume the column "Time (sec)" relates to "CPU execute time", this would explain why we see a nan in "Time (sec)" for MatMult. If my assumption is correct, how should I interpret the column "Flop (Max)" which is showing 1.92e+09? I would assume of "Time (sec)" relates to the CPU then "Flop (Max)" should also relate to CPU and GPU flops would be logged in "GPU Mflop/s" [2] More curious is that within "Event Stage 2: MG Apply" KSPSolve, MGSmooth Level 0, MGSmooth Level 1, MGSmooth Level 2 all report "GPU %F" as 93. I believe this value should be 100 as the smoother (and coarse grid solver) are configured as richardson(2)+none and thus should run entirely on the GPU. Furthermore, when one inspects all events listed under "Event Stage 2: MG Apply" those events which do flops correctly report "GPU %F" as 100. And the events showing "GPU %F" = 0 such as MatHIPSPARSCopyTo, VecCopy, VecSet, PCApply, DCtxSync don't do any flops (on the CPU or GPU) - which is also correct (although non GPU events should show nan??) Hence I am wondering what is the explanation for the missing 7% from "GPU %F" for KSPSolve and MGSmooth {0,1,2}?? Does anyone understand this -log_view, or can explain to me how to interpret it? It could simply be that: a) something is messed up with -pc_mg_log b) something is messed up with the PETSc build c) I am putting too much faith in -log_view and should profile the code differently. Either way I'd really like to understand what is going on. Cheers, Dave

4 9

User meeting in Cologne: Visit to KU Leuven
by Martin Diehl 26 Jan '24

26 Jan '24

Dear PETSc team, I was wondering if any of you would be interested to visit the NUMA research group at KU Leuven (https://wms.cs.kuleuven.be/groups/NUMA) for a after or before the user meeting in Cologne. We have/had several PETSc users and in general there seems to be a big overlap between scientific interests. with best regards Martin -- KU Leuven Department of Computer Science Department of Materials Engineering Celestijnenlaan 200a 3001 Leuven, Belgium

1 0

Question about a parallel implementation of PCFIELDSPLIT
by Pantelis Moschopoulos 24 Jan '24

24 Jan '24

Dear all, When I am using PCFIELDSPLIT and pc type "schur" in serial mode everything works fine. When I turn now to parallel, I observe that the number of ranks that I can use must divide the number of N without any remainder, where N is the number of unknowns. Otherwise, an error of the following form emerges: "Local columns of A10 3473 do not equal local rows of A00 3471". Can I do something to overcome this? Thanks, Pantelis

3 11