Hi all, We are using PETSc 3.20 in our code and running succesfully several solvers on Nvidia GPU with OpenMPI library which are not GPU aware (so I need to add the flag -use_gpu_aware_mpi 0). But now, when using OpenMPI GPU Aware library (OpenMPI 4.0.5 ou 4.1.5 from NVHPC), some parallel calculations failed with KSP_DIVERGED_ITS or KSP_DIVERGED_DTOL with several configurations. It may run wells on a small test case with (matrix is symmetric): -ksp_type cg -pc_type gamg -pc_gamg_type classical But suddenly with a number of devices for instance bigger than 4 or 8, it may fail. If I switch to another solver (BiCGstab), it may converge: -ksp_type bcgs -pc_type gamg -pc_gamg_type classical The more sensitive cases where it diverges are the following: -ksp_type cg -pc_type hypre -pc_hypre_type boomeramg -ksp_type cg -pc_type gamg -pc_gamg_type classical And the bcgs turnaroud doesn't work each time... It seems to work without problem with aggregation (at least 128 GPUs on my simulation): -ksp_type cg -pc_type gamg -pc_gamg_type agg So I guess there is a weird thing happening in my code during the solve in PETSc with MPI GPU Aware, as all the previous configurations works with non GPU aware MPI. Here is the -ksp_view log during one fail with the first configuration: KSP Object: () 8 MPI processes type: cg maximum iterations=10000, nonzero initial guess tolerances: relative=0., absolute=0.0001, divergence=10000. left preconditioning using UNPRECONDITIONED norm type for convergence test PC Object: () 8 MPI processes type: hypre HYPRE BoomerAMG preconditioning Cycle type V Maximum number of levels 25 Maximum number of iterations PER hypre call 1 Convergence tolerance PER hypre call 0. Threshold for strong coupling 0.7 Interpolation truncation factor 0. Interpolation: max elements per row 0 Number of levels of aggressive coarsening 0 Number of paths for aggressive coarsening 1 Maximum row sums 0.9 Sweeps down 1 Sweeps up 1 Sweeps on coarse 1 Relax down l1scaled-Jacobi Relax up l1scaled-Jacobi Relax on coarse Gaussian-elimination Relax weight (all) 1. Outer relax weight (all) 1. Maximum size of coarsest grid 9 Minimum size of coarsest grid 1 Not using CF-relaxation Not using more complex smoothers. Measure type local Coarsen type PMIS Interpolation type ext+i SpGEMM type cusparse linear system matrix = precond matrix: Mat Object: () 8 MPI processes type: mpiaijcusparse rows=64000, cols=64000 total: nonzeros=311040, allocated nonzeros=311040 total number of mallocs used during MatSetValues calls=0 not using I-node (on process 0) routines I didn't succeed for the moment creating a reproducer with ex.c examples... Did you see this kind of behaviour before? Should I update my PETSc version ? Thanks for any advice, Pierre LEDAC Commissariat à l’énergie atomique et aux énergies alternatives Centre de SACLAY DES/ISAS/DM2S/SGLS/LCAN Bâtiment 451 – point courrier n°43 F-91191 Gif-sur-Yvette +33 1 69 08 04 03 +33 6 83 42 05 79