On Sat, Jan 9, 2021 at 5:14 PM Barry Smith <[email protected]> wrote:

If it is non-overlapping do you mean block Jacobi with multiple blocks per MPI rank? (one block per rank is trivial and should work now).

Yes, only one MPI rank.

If you mean block Jacobi with multiple blocks per MPI rank you should start with the PCApply_BJacob_Multiblock(). It monkey's with pointers into the vector and then calls KSPSolve() for each block. So you just need a non-blocking KSPSolve(). What KSP and what PC do you want to use per block?

SuperLU

If you want to use LU then I think you proceed largely as I said a few days ago. All the routines in MatSolve_SeqAIJCUSparse can be made non-blocking as discussed with each one using its own stream (supplied with a hack or with approaches from Junchao and Jacob in the future possibly; but a hack is all that is needed for this trivial case.)

I'm not sure what goes into this hack. I am using fieldsplit and I now see that I don't specify asm, just lu, so I guess PCApply_FieldSplit is the target KSP Object: 1 MPI processes type: preonly maximum iterations=10000, initial guess is zero tolerances: relative=1e-05, absolute=1e-50, divergence=10000. left preconditioning using NONE norm type for convergence test PC Object: 1 MPI processes type: fieldsplit FieldSplit with ADDITIVE composition: total splits = 2 Solver info for each split is in the following KSP objects: Split number 0 Defined by IS KSP Object: (fieldsplit_e_) 1 MPI processes type: preonly maximum iterations=10000, initial guess is zero tolerances: relative=1e-05, absolute=1e-50, divergence=10000. left preconditioning using NONE norm type for convergence test PC Object: (fieldsplit_e_) 1 MPI processes type: lu out-of-place factorization tolerance for zero pivot 2.22045e-14 matrix ordering: nd factor fill ratio given 5., needed 1.30805 Factored matrix follows: Mat Object: 1 MPI processes type: seqaij rows=448, cols=448 package used to perform factorization: petsc total: nonzeros=14038, allocated nonzeros=14038 using I-node routines: found 175 nodes, limit used is 5 linear system matrix = precond matrix: Mat Object: (fieldsplit_e_) 1 MPI processes type: seqaij rows=448, cols=448 total: nonzeros=10732, allocated nonzeros=10732 total number of mallocs used during MatSetValues calls=0 using I-node routines: found 197 nodes, limit used is 5 Split number 1 Defined by IS KSP Object: (fieldsplit_i1_) 1 MPI processes ....