On Sat, Sep 25, 2021 at 8:12 PM Junchao Zhang <[email protected]> wrote:

On Sat, Sep 25, 2021 at 4:45 PM Mark Adams <[email protected]> wrote:

...

I am testing my Landau code, which is MPI serial, but with many independent MPI processes driving each GPU, in an MPI parallel harness code (Landau ex2).

Vector operations with Kokkos Kernels and cuSparse are about the same (KK is faster) and a bit expensive with one process / GPU. About the same as my Jacobian construction, which is expensive but optimized on the GPU. (I am using arkimex adaptive TS. I am guessing that it does a lot of vector ops, because there are a lot.)

With 14 or 15 processes, all doing the same MPI serial problem, cuSparse is about 2.5x more expensive than KK. KK does degrad by about 15% from the one processor case. So KK is doing fine, but something bad is happening with cuSparse.

AIJKOKKOS and AIJCUSPARSE have different algorithms? I don't know. To know exactly, the best approach is to consult with Peng@nvidia to profile the code.

Yea, I could ask Peng if he has any thoughts. I am also now having a problem with snes/tests/ex13 scaling study (for my ECP report). The cuSparse version of GAMG is hanging on an 8 node job with a refinement of 3. It works on one node with a refinement of 4 and on 8 nodes with a refinement of 2. I recently moved from CUDA-10 to CUDA-11 on summit because MPS seems to be working with CUDA-11 whereas it was not a while ago. I think I will try going back to CUDA-10 and see if I see anything change. Thanks, Mark

...

Anyone have any thoughts on this?

Thanks, Mark