January 2021 - petsc-dev - mailman3.cels.anl.gov

Re: [petsc-dev] model for parallel ASM
by Mark Adams 10 Jan '21

10 Jan '21

On Sat, Jan 9, 2021 at 7:37 PM Jacob Faibussowitsch <jacob.fai(a)gmail.com> wrote: > It is a single object that holds a pointer to every stream implementation > and toggleable type so it can be universally passed around. Currently has a > cudaStream and a hipStream but this is easily extendable to any other > stream implementation. > Do you have any thoughts on how this would work with Kokkos? Would you want to feed Kokkos your Cuda/Hip, etc, stream or add a Kokkos backend to your object? Junchao might be the person to ask. I would guess Kokkos View (vector) objects carry a stream because they block on a "deep_copy", that moves data to/from the GPU, and it is blocking. Thanks, Mark > Best regards, > > Jacob Faibussowitsch > (Jacob Fai - booss - oh - vitch) > Cell: +1 (312) 694-3391 > > On Jan 9, 2021, at 18:19, Mark Adams <mfadams(a)lbl.gov> wrote: > > > Is this stream object going to have Cuda, Kokkos, etc., implementations? > > On Sat, Jan 9, 2021 at 4:09 PM Jacob Faibussowitsch <jacob.fai(a)gmail.com> > wrote: > >> I’m currently working on an implementation of a general PetscStream >> object. Currently it only supports Vector ops and has a proof of concept >> KSPCG, but should be extensible to other objects when finished. Junchao is >> also indirectly working on pipeline support in his NVSHMEM MR. Take a look >> at either MR, it would be very useful to get your input, as tailoring >> either of these approaches for pipelined algorithms is key. >> >> Best regards, >> >> Jacob Faibussowitsch >> (Jacob Fai - booss - oh - vitch) >> Cell: (312) 694-3391 >> >> On Jan 9, 2021, at 15:01, Mark Adams <mfadams(a)lbl.gov> wrote: >> >> I would like to put a non-overlapping ASM solve on the GPU. It's not >> clear that we have a model for this. >> >> PCApply_ASM currently pipelines the scater with the subdomain solves. I >> think we would want to change this and do a 1) scatter begin loop, 2) >> scatter end and non-blocking solve loop, 3) solve-wait and scatter >> begging loop and 4) scatter end loop. >> >> I'm not sure how to go about doing this. >> * Should we make a new PCApply_ASM_PARALLEL or dump this pipelining >> algorithm and rewrite PCApply_ASM? >> * Add a solver-wait method to KSP? >> >> Thoughts? >> >> Mark >> >> >>

1 0

Re: [petsc-dev] model for parallel ASM
by Barry Smith 09 Jan '21

09 Jan '21

Ok, fieldsplit, pretty much the same as what I said for block Jacobi. static PetscErrorCode PCApply_FieldSplit(PC pc,Vec x,Vec y) { PC_FieldSplit *jac = (PC_FieldSplit*)pc->data; PetscErrorCode ierr; PC_FieldSplitLink ilink = jac->head; PetscInt cnt,bs; PetscFunctionBegin; if (jac->type == PC_COMPOSITE_ADDITIVE) { if (jac->defaultsplit) { ierr = VecGetBlockSize(x,&bs);CHKERRQ(ierr); if (jac->bs > 0 && bs != jac->bs) SETERRQ2(PetscObjectComm((PetscObject)pc),PETSC_ERR_ARG_WRONGSTATE,"Blocksize of x vector %D does not match fieldsplit blocksize %D",bs,jac->bs); ierr = VecGetBlockSize(y,&bs);CHKERRQ(ierr); if (jac->bs > 0 && bs != jac->bs) SETERRQ2(PetscObjectComm((PetscObject)pc),PETSC_ERR_ARG_WRONGSTATE,"Blocksize of y vector %D does not match fieldsplit blocksize %D",bs,jac->bs); ierr = VecStrideGatherAll(x,jac->x,INSERT_VALUES);CHKERRQ(ierr); while (ilink) { ierr = PetscLogEventBegin(ilink->event,ilink->ksp,ilink->x,ilink->y,NULL);CHKERRQ(ierr); ierr = KSPSolve(ilink->ksp,ilink->x,ilink->y);CHKERRQ(ierr); you need these KSPSolve() to not block which if you are using SuperLU_DIST means each call to PetscStackCall("SuperLU_DIST:PStatInit",PStatInit(&stat)); /* Initialize the statistics variables. */ #if defined(PETSC_USE_COMPLEX) PetscStackCall("SuperLU_DIST:pzgssvx",pzgssvx(&lu->options,&lu->A_sup,&lu->ScalePermstruct,(doublecomplex*)bptr,m,1,&lu->grid,&lu->LUstruct,&lu->SOLVEstruct,berr,&stat,&info)); #else PetscStackCall("SuperLU_DIST:pdgssvx",pdgssvx(&lu->options,&lu->A_sup,&lu->ScalePermstruct,bptr,m,1,&lu->grid,&lu->LUstruct,&lu->SOLVEstruct,berr,&stat,&info)); #endif must be on a different stream and cannot block. Which means essentially the PStatInit and pdgssvx have to run completely on the GPU or have some complex CPU babysitting that allows them to be interlaced on the GPU while doing their CPU stuff. I doubt either thing is true and you are best off using the cuSparse Solvers which do run completely on the GPU in any stream you give them. The factorization is even more of a nightmare, I think Sherry uses the GPU just to do blocks for her sequential factorization so I doubt very much it can handle multiple simultaneous factorizations. ierr = KSPCheckSolve(ilink->ksp,pc,ilink->y);CHKERRQ(ierr); ierr = PetscLogEventEnd(ilink->event,ilink->ksp,ilink->x,ilink->y,NULL);CHKERRQ(ierr); ilink = ilink->next; } > On Jan 9, 2021, at 6:06 PM, Mark Adams <mfadams(a)lbl.gov> wrote: > > > > On Sat, Jan 9, 2021 at 5:14 PM Barry Smith <bsmith(a)petsc.dev <mailto:[email protected]>> wrote: > > If it is non-overlapping do you mean block Jacobi with multiple blocks per MPI rank? (one block per rank is trivial and should work now). > > Yes, only one MPI rank. > > > If you mean block Jacobi with multiple blocks per MPI rank you should start with the PCApply_BJacob_Multiblock(). It monkey's with pointers into the vector and then calls KSPSolve() for each block. So you just need a non-blocking KSPSolve(). What KSP and what PC do you want to use per block? > > SuperLU > > If you want to use LU then I think you proceed largely as I said a few days ago. All the routines in MatSolve_SeqAIJCUSparse can be made non-blocking as discussed with each one using its own stream (supplied with a hack or with approaches from Junchao and Jacob in the future possibly; but a hack is all that is needed for this trivial case.) > > I'm not sure what goes into this hack. > > I am using fieldsplit and I now see that I don't specify asm, just lu, so I guess PCApply_FieldSplit is the target > > KSP Object: 1 MPI processes > type: preonly > maximum iterations=10000, initial guess is zero > tolerances: relative=1e-05, absolute=1e-50, divergence=10000. > left preconditioning > using NONE norm type for convergence test > PC Object: 1 MPI processes > type: fieldsplit > FieldSplit with ADDITIVE composition: total splits = 2 > Solver info for each split is in the following KSP objects: > Split number 0 Defined by IS > KSP Object: (fieldsplit_e_) 1 MPI processes > type: preonly > maximum iterations=10000, initial guess is zero > tolerances: relative=1e-05, absolute=1e-50, divergence=10000. > left preconditioning > using NONE norm type for convergence test > PC Object: (fieldsplit_e_) 1 MPI processes > type: lu > out-of-place factorization > tolerance for zero pivot 2.22045e-14 > matrix ordering: nd > factor fill ratio given 5., needed 1.30805 > Factored matrix follows: > Mat Object: 1 MPI processes > type: seqaij > rows=448, cols=448 > package used to perform factorization: petsc > total: nonzeros=14038, allocated nonzeros=14038 > using I-node routines: found 175 nodes, limit used is 5 > linear system matrix = precond matrix: > Mat Object: (fieldsplit_e_) 1 MPI processes > type: seqaij > rows=448, cols=448 > total: nonzeros=10732, allocated nonzeros=10732 > total number of mallocs used during MatSetValues calls=0 > using I-node routines: found 197 nodes, limit used is 5 > Split number 1 Defined by IS > KSP Object: (fieldsplit_i1_) 1 MPI processes > .... >

1 0

Re: [petsc-dev] looking at all threads in MR after they are resolved
by Barry Smith 09 Jan '21

09 Jan '21

> On Jan 9, 2021, at 8:16 PM, Satish Balay <balay(a)mcs.anl.gov> wrote: > > On Sat, 9 Jan 2021, Barry Smith wrote: > >> >> For unresolved threads you can press the arrow in the upper right to find the next unresolved thread. >> >> But how do you march through all the resolved threads? I want to check a MR where I made some threads and now they are "resolved" but I want to check how they were resolved. Any way to do this? The is a little sideways <> labeled collapse all threads, expand all threads but it does not seem to do much of anything. >> > > down arrow (V) 'expand all threads' works for me. But what does it do? Do you then need to scroll down through the 50 pages looking for thread comments with your eyeballs that happen randomly very few screen fulls? Barry > > Satish

1 0

Re: [petsc-dev] looking at all threads in MR after they are resolved
by Satish Balay 09 Jan '21

09 Jan '21

On Sat, 9 Jan 2021, Barry Smith wrote: > > For unresolved threads you can press the arrow in the upper right to find the next unresolved thread. > > But how do you march through all the resolved threads? I want to check a MR where I made some threads and now they are "resolved" but I want to check how they were resolved. Any way to do this? The is a little sideways <> labeled collapse all threads, expand all threads but it does not seem to do much of anything. > down arrow (V) 'expand all threads' works for me. Satish

1 0

Re: [petsc-dev] model for parallel ASM
by Jacob Faibussowitsch 09 Jan '21

09 Jan '21

It is a single object that holds a pointer to every stream implementation and toggleable type so it can be universally passed around. Currently has a cudaStream and a hipStream but this is easily extendable to any other stream implementation. Best regards, Jacob Faibussowitsch (Jacob Fai - booss - oh - vitch) Cell: +1 (312) 694-3391 > On Jan 9, 2021, at 18:19, Mark Adams <mfadams(a)lbl.gov> wrote: > > > Is this stream object going to have Cuda, Kokkos, etc., implementations? > >> On Sat, Jan 9, 2021 at 4:09 PM Jacob Faibussowitsch <jacob.fai(a)gmail.com> wrote: >> I’m currently working on an implementation of a general PetscStream object. Currently it only supports Vector ops and has a proof of concept KSPCG, but should be extensible to other objects when finished. Junchao is also indirectly working on pipeline support in his NVSHMEM MR. Take a look at either MR, it would be very useful to get your input, as tailoring either of these approaches for pipelined algorithms is key. >> >> Best regards, >> >> Jacob Faibussowitsch >> (Jacob Fai - booss - oh - vitch) >> Cell: (312) 694-3391 >> >>> On Jan 9, 2021, at 15:01, Mark Adams <mfadams(a)lbl.gov> wrote: >>> >>> I would like to put a non-overlapping ASM solve on the GPU. It's not clear that we have a model for this. >>> >>> PCApply_ASM currently pipelines the scater with the subdomain solves. I think we would want to change this and do a 1) scatter begin loop, 2) scatter end and non-blocking solve loop, 3) solve-wait and scatter begging loop and 4) scatter end loop. >>> >>> I'm not sure how to go about doing this. >>> * Should we make a new PCApply_ASM_PARALLEL or dump this pipelining algorithm and rewrite PCApply_ASM? >>> * Add a solver-wait method to KSP? >>> >>> Thoughts? >>> >>> Mark >>

1 0

Re: [petsc-dev] model for parallel ASM
by Mark Adams 09 Jan '21

09 Jan '21

Is this stream object going to have Cuda, Kokkos, etc., implementations? On Sat, Jan 9, 2021 at 4:09 PM Jacob Faibussowitsch <jacob.fai(a)gmail.com> wrote: > I’m currently working on an implementation of a general PetscStream > object. Currently it only supports Vector ops and has a proof of concept > KSPCG, but should be extensible to other objects when finished. Junchao is > also indirectly working on pipeline support in his NVSHMEM MR. Take a look > at either MR, it would be very useful to get your input, as tailoring > either of these approaches for pipelined algorithms is key. > > Best regards, > > Jacob Faibussowitsch > (Jacob Fai - booss - oh - vitch) > Cell: (312) 694-3391 > > On Jan 9, 2021, at 15:01, Mark Adams <mfadams(a)lbl.gov> wrote: > > I would like to put a non-overlapping ASM solve on the GPU. It's not clear > that we have a model for this. > > PCApply_ASM currently pipelines the scater with the subdomain solves. I > think we would want to change this and do a 1) scatter begin loop, 2) > scatter end and non-blocking solve loop, 3) solve-wait and scatter > begging loop and 4) scatter end loop. > > I'm not sure how to go about doing this. > * Should we make a new PCApply_ASM_PARALLEL or dump this pipelining > algorithm and rewrite PCApply_ASM? > * Add a solver-wait method to KSP? > > Thoughts? > > Mark > > >

1 0

Re: [petsc-dev] model for parallel ASM
by Mark Adams 09 Jan '21

09 Jan '21

On Sat, Jan 9, 2021 at 5:14 PM Barry Smith <bsmith(a)petsc.dev> wrote: > > If it is non-overlapping do you mean block Jacobi with multiple blocks > per MPI rank? (one block per rank is trivial and should work now). > Yes, only one MPI rank. > > If you mean block Jacobi with multiple blocks per MPI rank you should > start with the PCApply_BJacob_Multiblock(). It monkey's with pointers into > the vector and then calls KSPSolve() for each block. So you just need a > non-blocking KSPSolve(). What KSP and what PC do you want to use per block? SuperLU > If you want to use LU then I think you proceed largely as I said a few > days ago. All the routines in MatSolve_SeqAIJCUSparse can be made > non-blocking as discussed with each one using its own stream (supplied with > a hack or with approaches from Junchao and Jacob in the future possibly; > but a hack is all that is needed for this trivial case.) > I'm not sure what goes into this hack. I am using fieldsplit and I now see that I don't specify asm, just lu, so I guess PCApply_FieldSplit is the target KSP Object: 1 MPI processes type: preonly maximum iterations=10000, initial guess is zero tolerances: relative=1e-05, absolute=1e-50, divergence=10000. left preconditioning using NONE norm type for convergence test PC Object: 1 MPI processes type: fieldsplit FieldSplit with ADDITIVE composition: total splits = 2 Solver info for each split is in the following KSP objects: Split number 0 Defined by IS KSP Object: (fieldsplit_e_) 1 MPI processes type: preonly maximum iterations=10000, initial guess is zero tolerances: relative=1e-05, absolute=1e-50, divergence=10000. left preconditioning using NONE norm type for convergence test PC Object: (fieldsplit_e_) 1 MPI processes type: lu out-of-place factorization tolerance for zero pivot 2.22045e-14 matrix ordering: nd factor fill ratio given 5., needed 1.30805 Factored matrix follows: Mat Object: 1 MPI processes type: seqaij rows=448, cols=448 package used to perform factorization: petsc total: nonzeros=14038, allocated nonzeros=14038 using I-node routines: found 175 nodes, limit used is 5 linear system matrix = precond matrix: Mat Object: (fieldsplit_e_) 1 MPI processes type: seqaij rows=448, cols=448 total: nonzeros=10732, allocated nonzeros=10732 total number of mallocs used during MatSetValues calls=0 using I-node routines: found 197 nodes, limit used is 5 Split number 1 Defined by IS KSP Object: (fieldsplit_i1_) 1 MPI processes ....

1 0

Re: [petsc-dev] looking at all threads in MR after they are resolved
by Jacob Faibussowitsch 09 Jan '21

09 Jan '21

AFAIK there is no direct way to do it, but this is what I do. Right at the top, under the MR blurb and reviewer status (opposite the thumbs up, thumbs down, emoji buttons) you should see a drop-down with the options: - Show all activity - Show comments only - Show history only Switch that to show comments only (I keep this as my default). Now you should only see peoples comments, go ahead and collapse all threads so that only the top level comment is visible. If that thread was resolved, then right under the name of the person making the comment, you should see: "Resolved by <person who resolved> <https://gitlab.com/Jfaibussowitsch><some time> ago” And just manually scroll through it. Best regards, Jacob Faibussowitsch (Jacob Fai - booss - oh - vitch) Cell: (312) 694-3391 > On Jan 9, 2021, at 16:22, Barry Smith <bsmith(a)petsc.dev> wrote: > > > For unresolved threads you can press the arrow in the upper right to find the next unresolved thread. > > But how do you march through all the resolved threads? I want to check a MR where I made some threads and now they are "resolved" but I want to check how they were resolved. Any way to do this? The is a little sideways <> labeled collapse all threads, expand all threads but it does not seem to do much of anything. > > Barry >

1 0

looking at all threads in MR after they are resolved
by Barry Smith 09 Jan '21

09 Jan '21

For unresolved threads you can press the arrow in the upper right to find the next unresolved thread. But how do you march through all the resolved threads? I want to check a MR where I made some threads and now they are "resolved" but I want to check how they were resolved. Any way to do this? The is a little sideways <> labeled collapse all threads, expand all threads but it does not seem to do much of anything. Barry

1 0

Re: [petsc-dev] model for parallel ASM
by Barry Smith 09 Jan '21

09 Jan '21

If it is non-overlapping do you mean block Jacobi with multiple blocks per MPI rank? (one block per rank is trivial and should work now). If you mean block Jacobi with multiple blocks per MPI rank you should start with the PCApply_BJacob_Multiblock(). It monkey's with pointers into the vector and then calls KSPSolve() for each block. So you just need a non-blocking KSPSolve(). What KSP and what PC do you want to use per block? If you want to use LU then I think you proceed largely as I said a few days ago. All the routines in MatSolve_SeqAIJCUSparse can be made non-blocking as discussed with each one using its own stream (supplied with a hack or with approaches from Junchao and Jacob in the future possibly; but a hack is all that is needed for this trivial case.) Barry > On Jan 9, 2021, at 3:01 PM, Mark Adams <mfadams(a)lbl.gov> wrote: > > I would like to put a non-overlapping ASM solve on the GPU. It's not clear that we have a model for this. > > PCApply_ASM currently pipelines the scater with the subdomain solves. I think we would want to change this and do a 1) scatter begin loop, 2) scatter end and non-blocking solve loop, 3) solve-wait and scatter begging loop and 4) scatter end loop. > > I'm not sure how to go about doing this. > * Should we make a new PCApply_ASM_PARALLEL or dump this pipelining algorithm and rewrite PCApply_ASM? > * Add a solver-wait method to KSP? > > Thoughts? > > Mark

1 0