January 2021 - petsc-dev - mailman3.cels.anl.gov

Re: [petsc-dev] OpenMP and web page
by Barry Smith 15 Jan '21

15 Jan '21

I agree. We should just make sure it "disappears" with the transition to the Spinx docs. Barry > On Jan 15, 2021, at 10:32 AM, jacob.fai(a)gmail.com wrote: > > Alternatively, I simply don’t include it in the MR to port the website to sphinx. > > Best regards, > > Jacob Faibussowitsch > (Jacob Fai - booss - oh - vitch) > Cell: (312) 694-3391 > > From: petsc-dev <petsc-dev-bounces(a)mcs.anl.gov> On Behalf Of Mark Adams > Sent: Friday, January 15, 2021 10:20 > To: For users of the development version of PETSc <petsc-dev(a)mcs.anl.gov> > Subject: [petsc-dev] OpenMP and web page > > I am experimenting with launching asynchronous GPU solves in PCFieldsplit/additive using OpenMP and I found this web page, which I think is obsolete and should be removed: https://www.mcs.anl.gov/petsc/miscellaneous/petscthreads.html <https://www.mcs.anl.gov/petsc/miscellaneous/petscthreads.html> > > Mark

1 0

Fwd: [ECP-ST-Math-Lib] [EXTERNAL] opportunities for HI/Facilities interactions
by Barry Smith 15 Jan '21

15 Jan '21

It would make sense to try to utilize this help if people have specific questions/problems with specific upcoming ECP systems. Not CUDA :-(. If so it might make sense to collect them and then pass them through Richard/Todd as a single response to Sherry. Barry > Begin forwarded message: > > From: "Xiaoye S. Li via ECP-ST-Math-Lib" <ecp-st-math-lib(a)elist.ornl.gov> > Subject: [ECP-ST-Math-Lib] [EXTERNAL] opportunities for HI/Facilities interactions > Date: January 14, 2021 at 11:01:48 PM CST > To: ecp-st-math-lib(a)elist.ornl.gov > Reply-To: "Xiaoye S. Li" <xsli(a)lbl.gov> > Message-Id: <CAFvbobV8JdeZFnQRqfXuKW4BPQwKzb2CzxwrQhv_JVpDvzsGLA(a)mail.gmail.com> > > Hi All, > > We would like to find out whether each L4 project in math libraries has needs for more support from HI/Facilities to assist in porting and optimizing codes on the early access systems. AD teams have many facilities/vendor staff members to help with each code; these are designated performance engineering experts, called AppInt staff. > > If you see potential benefits engaging with AppInt staff, please let me know the following: > > 1. what specific needs would you like additional AppInt support for porting and optimization? > > 2. what skills from AppInt staff are required in order to better assist your team? > > 3. what important capabilities and expertise that AppInt staff could benefit from knowing your project, so that they can better help AD teams? > > Please respond at your earliest convenience. > Thanks, > Sherry > _______________________________________________ > ECP-ST-Math-Lib mailing list > ECP-ST-Math-Lib(a)elist.ornl.gov > https://elist.ornl.gov/mailman/listinfo/ecp-st-math-lib > To unsubscribe, send a blank email to ecp-st-math-lib-unsubscribe(a)elist.ornl.gov

1 0

Re: [petsc-dev] OpenMP and web page
by jacob.fai＠gmail.com 15 Jan '21

15 Jan '21

Alternatively, I simply don’t include it in the MR to port the website to sphinx. Best regards, Jacob Faibussowitsch (Jacob Fai - booss - oh - vitch) Cell: (312) 694-3391 From: petsc-dev <petsc-dev-bounces(a)mcs.anl.gov> On Behalf Of Mark Adams Sent: Friday, January 15, 2021 10:20 To: For users of the development version of PETSc <petsc-dev(a)mcs.anl.gov> Subject: [petsc-dev] OpenMP and web page I am experimenting with launching asynchronous GPU solves in PCFieldsplit/additive using OpenMP and I found this web page, which I think is obsolete and should be removed: https://www.mcs.anl.gov/petsc/miscellaneous/petscthreads.html Mark

1 0

OpenMP and web page
by Mark Adams 15 Jan '21

15 Jan '21

I am experimenting with launching asynchronous GPU solves in PCFieldsplit/additive using OpenMP and I found this web page, which I think is obsolete and should be removed: https://www.mcs.anl.gov/petsc/miscellaneous/petscthreads.html Mark

1 0

Re: [petsc-dev] About parallel of ILU
by Ed Bueler 12 Jan '21

12 Jan '21

At a practical level you may be looking for parallel ways to use PETSc's native (but serial) ILU. The simplest parallel extension is processor-block Jacobi which applies ILU on the processor-owned degrees of freedom but with no overlap: -pc_type bjacobi -sub_pc_type ilu The overlapping version is with ASM: -pc_type asm -sub_pc_type ilu where -pc_asm_overlap controls the overlap distance. Finally one can use the serial ILU as a smoother in geometric multigrid, for example in combination with overlapping ASM: -pc_type mg -mg_levels_pc_type asm -mg_levels_sub_pc_type ilu Apologies if this was all assumed to be known in your original question. Indeed, application of parallel direct solvers in PETSc often uses hyper/mumps/superlu, though I'm not familiar with ILU in these packages. Ed Dear Professor, > I'm writing about this mail about the ILU algorithm in PETSc. I have > noticed that the ILU in PETSs cannot be paralleled, and the Hypre providing > the parallel implement of ILU. > My questions is : why the PETSc does not provide the parallel implement of > ILU directly, instead of linking to the Hypre to obtain the Parallel ILU > algorithm? > PS: the attachment is the Parallel ILU algorithm in Hypre. > I'm looking forward to hear you! > Best, > Gang Chen > Sichuan Univertsity, > School of Mathematics, > Chengdu, China -- Ed Bueler Dept of Mathematics and Statistics University of Alaska Fairbanks Fairbanks, AK 99775-6660 306C Chapman

1 0

Re: [petsc-dev] About parallel of ILU
by Jed Brown 12 Jan '21

12 Jan '21

"Chen Gang" <569615491(a)qq.com> writes: > Dear Professor, > > > I'm writing about this mail about the ILU algorithm in PETSc. I have noticed that the ILU in PETSs cannot be paralleled, and the Hypre providing the parallel implement of ILU. > > > My questions is : why the PETSc does not provide the parallel implement of ILU directly, instead of linking to the Hypre to obtain the Parallel ILU algorithm? We have limited time and try to focus on things we think will provide the most value. If the implementation in Hypre works for you, by all means, go ahead and use it. ILU isn't a great parallel algorithm -- it isn't very strong for large problems (a math issue) except when using fancy (usually expensive) multilevel ILU methods that overlap with some algebraic multigrid methods, and it doesn't parallelize very well (also a math issue, unless you use reorderings that undermine any advantage relative to additive Schwarz). If you want to contribute a parallel ILU, we'd be happy to review a merge request.

1 0

About parallel of ILU
by Chen Gang 11 Jan '21

11 Jan '21

Dear Professor, I'm writing about this mail about the ILU algorithm in PETSc. I have noticed that the ILU in PETSs cannot be paralleled, and the Hypre providing the parallel implement of ILU. My questions is : why the PETSc does not provide the parallel implement of ILU directly, instead of linking to the Hypre to obtain the Parallel ILU algorithm? PS: the attachment is the Parallel ILU algorithm in Hypre. I'm looking forward to hear you! Best, Gang Chen Sichuan Univertsity, School of Mathematics, Chengdu, China

1 0

Re: [petsc-dev] model for parallel ASM
by Mark Adams 11 Jan '21

11 Jan '21

I've managed to read through your MR posts today. As I said there getting parallel fieldsplit/additive solves is what I'm after. Maybe start with Jacobi. And I am MPI serial so my scatter gathers are local in PCApply_FieldSplit. FYI, Christan Trott responded with this wrt the future Kokkos interface for streams. crtrott <https://app.slack.com/team/U5C9R5T53>[image: :slack_call:] 3 hours ago <https://kokkosteam.slack.com/archives/C5BGU5NDQ/p1610383431196100?thread_ts…> So basicall So basically we are still fixing some kinks in the whole thing (you can imagine there are a million funky corner cases if you suddenly start throwing around even more asynchronous stuff, in particular if you potentially do that from multiple host threads, and if its not even clear if a stream object is permanently associated with one thread or passed back and forth) crtrott <https://app.slack.com/team/U5C9R5T53>[image: :slack_call:] 3 hours ago <https://kokkosteam.slack.com/archives/C5BGU5NDQ/p1610383488196300?thread_ts…> but in the future you will be able to do something like: auto instances = Kokkos::partition_exec_space(Kokkos::DefaultExecutionSpace(),4); crtrott <https://app.slack.com/team/U5C9R5T53>[image: :slack_call:] 3 hours ago <https://kokkosteam.slack.com/archives/C5BGU5NDQ/p1610383504196500?thread_ts…> and get four semantically independent execution spaces back crtrott <https://app.slack.com/team/U5C9R5T53>[image: :slack_call:] 3 hours ago <https://kokkosteam.slack.com/archives/C5BGU5NDQ/p1610383541196700?thread_ts…> with some backends they will still serialize (like CPU OpenMP) since you just get back the default instance 4 times crtrott <https://app.slack.com/team/U5C9R5T53>[image: :slack_call:] 3 hours ago <https://kokkosteam.slack.com/archives/C5BGU5NDQ/p1610383549196900?thread_ts…> on stuff like CUDA you get independent streams crtrott <https://app.slack.com/team/U5C9R5T53>[image: :slack_call:] 3 hours ago <https://kokkosteam.slack.com/archives/C5BGU5NDQ/p1610383571197100?thread_ts…> and on other thread based backends we may partition the threadpool for real On Mon, Jan 11, 2021 at 11:52 AM Jacob Faibussowitsch <jacob.fai(a)gmail.com> wrote: > Hmm I suppose this means Kokkos should accept a stream like we expect it > to? According to this somewhat recent merged PR: > https://github.com/kokkos/kokkos/pull/1919 you can now make a > "Kokkos::Cuda” object, and pass it as arg1 to range policies as an > execution space. Here’s what I found on it (the cuda specific one is > useless): > > https://github.com/kokkos/kokkos/wiki/ExecutionSpaceConcept > https://github.com/kokkos/kokkos/wiki/Kokkos%3A%3AExecutionSpaceConcept > <https://github.com/kokkos/kokkos/wiki/Kokkos::ExecutionSpaceConcept> > https://github.com/kokkos/kokkos/wiki/Kokkos%3A%3ACuda > <https://github.com/kokkos/kokkos/wiki/Kokkos::Cuda> <—— cuda specific > > Best regards, > > Jacob Faibussowitsch > (Jacob Fai - booss - oh - vitch) > Cell: (312) 694-3391 > > On Jan 11, 2021, at 10:35, Mark Adams <mfadams(a)lbl.gov> wrote: > > Jacob, I'm not sure I understand this response. I could not find you on > the Kokkos slack channel. > > Me: And My colleague in PETSc, Jacob Faibussowitsch, has talked to you > about Kokkos taking a Cuda, Hip, etc., stream. This is something that would > make it easier to deal with asynchronous GPU solvers in PETSc. We just > wanted to check on this. > > Trott: Kokkos itself can do it for practically every operation > > Maybe you want to talk with him at some point, but we can worry about > getting Cuda to work for now. > > On Sun, Jan 10, 2021 at 2:28 PM Jacob Faibussowitsch <jacob.fai(a)gmail.com> > wrote: > >> I would like as much as possible to pass the cuda and hip streams to >> Kokkos, since I can directly handle much of the annoyance with wrangling >> multiple streams and stream objects externally. Last I checked on this >> Kokkos was moving towards allowing association of streams to functions, but >> admittedly this was a while back. >> >> Best regards, >> >> Jacob Faibussowitsch >> (Jacob Fai - booss - oh - vitch) >> Cell: (312) 694-3391 >> >> On Jan 10, 2021, at 13:10, Mark Adams <mfadams(a)lbl.gov> wrote: >> >> >> >> On Sat, Jan 9, 2021 at 7:37 PM Jacob Faibussowitsch <jacob.fai(a)gmail.com> >> wrote: >> >>> It is a single object that holds a pointer to every stream >>> implementation and toggleable type so it can be universally passed around. >>> Currently has a cudaStream and a hipStream but this is easily extendable to >>> any other stream implementation. >>> >> >> Do you have any thoughts on how this would work with Kokkos? >> >> Would you want to feed Kokkos your Cuda/Hip, etc, stream or add a Kokkos >> backend to your object? >> >> Junchao might be the person to ask. I would guess Kokkos View (vector) >> objects carry a stream because they block on a "deep_copy", that moves data >> to/from the GPU, and it is blocking. >> >> Thanks, >> Mark >> >> >>> Best regards, >>> >>> Jacob Faibussowitsch >>> (Jacob Fai - booss - oh - vitch) >>> Cell: +1 (312) 694-3391 >>> >>> On Jan 9, 2021, at 18:19, Mark Adams <mfadams(a)lbl.gov> wrote: >>> >>> >>> Is this stream object going to have Cuda, Kokkos, etc., implementations? >>> >>> On Sat, Jan 9, 2021 at 4:09 PM Jacob Faibussowitsch <jacob.fai(a)gmail.com> >>> wrote: >>> >>>> I’m currently working on an implementation of a general PetscStream >>>> object. Currently it only supports Vector ops and has a proof of concept >>>> KSPCG, but should be extensible to other objects when finished. Junchao is >>>> also indirectly working on pipeline support in his NVSHMEM MR. Take a look >>>> at either MR, it would be very useful to get your input, as tailoring >>>> either of these approaches for pipelined algorithms is key. >>>> >>>> Best regards, >>>> >>>> Jacob Faibussowitsch >>>> (Jacob Fai - booss - oh - vitch) >>>> Cell: (312) 694-3391 >>>> >>>> On Jan 9, 2021, at 15:01, Mark Adams <mfadams(a)lbl.gov> wrote: >>>> >>>> I would like to put a non-overlapping ASM solve on the GPU. It's not >>>> clear that we have a model for this. >>>> >>>> PCApply_ASM currently pipelines the scater with the subdomain solves. I >>>> think we would want to change this and do a 1) scatter begin loop, 2) >>>> scatter end and non-blocking solve loop, 3) solve-wait and scatter >>>> begging loop and 4) scatter end loop. >>>> >>>> I'm not sure how to go about doing this. >>>> * Should we make a new PCApply_ASM_PARALLEL or dump this pipelining >>>> algorithm and rewrite PCApply_ASM? >>>> * Add a solver-wait method to KSP? >>>> >>>> Thoughts? >>>> >>>> Mark >>>> >>>> >>>> >> >

1 0

Re: [petsc-dev] looking at all threads in MR after they are resolved
by Satish Balay 11 Jan '21

11 Jan '21

On Sat, 9 Jan 2021, Barry Smith wrote: > > > > On Jan 9, 2021, at 8:16 PM, Satish Balay <balay(a)mcs.anl.gov> wrote: > > > > On Sat, 9 Jan 2021, Barry Smith wrote: > > > >> > >> For unresolved threads you can press the arrow in the upper right to find the next unresolved thread. > >> > >> But how do you march through all the resolved threads? I want to check a MR where I made some threads and now they are "resolved" but I want to check how they were resolved. Any way to do this? The is a little sideways <> labeled collapse all threads, expand all threads but it does not seem to do much of anything. > >> > > > > down arrow (V) 'expand all threads' works for me. > > But what does it do? Do you then need to scroll down through the 50 pages looking for thread comments with your eyeballs that happen randomly very few screen fulls? Somehow you want some metadata from the threads [on how was it resolved] - without looking at the expanded thread? For one - there is no such metadata [on why the thread is marked resolved] - Mabye everyone who resolves the thread would have to add an extra message [which will result in extra e-mail] - on why they are resolving it?. And then - one would do a search for this 'resolve-reason' text? Satish

1 0

Re: [petsc-dev] model for parallel ASM
by Jacob Faibussowitsch 11 Jan '21

11 Jan '21

Hmm I suppose this means Kokkos should accept a stream like we expect it to? According to this somewhat recent merged PR: https://github.com/kokkos/kokkos/pull/1919 <https://github.com/kokkos/kokkos/pull/1919> you can now make a "Kokkos::Cuda” object, and pass it as arg1 to range policies as an execution space. Here’s what I found on it (the cuda specific one is useless): https://github.com/kokkos/kokkos/wiki/ExecutionSpaceConcept <https://github.com/kokkos/kokkos/wiki/ExecutionSpaceConcept> https://github.com/kokkos/kokkos/wiki/Kokkos%3A%3AExecutionSpaceConcept <https://github.com/kokkos/kokkos/wiki/Kokkos::ExecutionSpaceConcept> https://github.com/kokkos/kokkos/wiki/Kokkos%3A%3ACuda <https://github.com/kokkos/kokkos/wiki/Kokkos::Cuda> <—— cuda specific Best regards, Jacob Faibussowitsch (Jacob Fai - booss - oh - vitch) Cell: (312) 694-3391 > On Jan 11, 2021, at 10:35, Mark Adams <mfadams(a)lbl.gov> wrote: > > Jacob, I'm not sure I understand this response. I could not find you on the Kokkos slack channel. > > Me: And My colleague in PETSc, Jacob Faibussowitsch, has talked to you about Kokkos taking a Cuda, Hip, etc., stream. This is something that would make it easier to deal with asynchronous GPU solvers in PETSc. We just wanted to check on this. > > Trott: Kokkos itself can do it for practically every operation > > Maybe you want to talk with him at some point, but we can worry about getting Cuda to work for now. > > On Sun, Jan 10, 2021 at 2:28 PM Jacob Faibussowitsch <jacob.fai(a)gmail.com <mailto:[email protected]>> wrote: > I would like as much as possible to pass the cuda and hip streams to Kokkos, since I can directly handle much of the annoyance with wrangling multiple streams and stream objects externally. Last I checked on this Kokkos was moving towards allowing association of streams to functions, but admittedly this was a while back. > > Best regards, > > Jacob Faibussowitsch > (Jacob Fai - booss - oh - vitch) > Cell: (312) 694-3391 > >> On Jan 10, 2021, at 13:10, Mark Adams <mfadams(a)lbl.gov <mailto:[email protected]>> wrote: >> >> >> >> On Sat, Jan 9, 2021 at 7:37 PM Jacob Faibussowitsch <jacob.fai(a)gmail.com <mailto:[email protected]>> wrote: >> It is a single object that holds a pointer to every stream implementation and toggleable type so it can be universally passed around. Currently has a cudaStream and a hipStream but this is easily extendable to any other stream implementation. >> >> Do you have any thoughts on how this would work with Kokkos? >> >> Would you want to feed Kokkos your Cuda/Hip, etc, stream or add a Kokkos backend to your object? >> >> Junchao might be the person to ask. I would guess Kokkos View (vector) objects carry a stream because they block on a "deep_copy", that moves data to/from the GPU, and it is blocking. >> >> Thanks, >> Mark >> >> >> Best regards, >> >> Jacob Faibussowitsch >> (Jacob Fai - booss - oh - vitch) >> Cell: +1 (312) 694-3391 >> >>> On Jan 9, 2021, at 18:19, Mark Adams <mfadams(a)lbl.gov <mailto:[email protected]>> wrote: >>> >>> >>> Is this stream object going to have Cuda, Kokkos, etc., implementations? >>> >>> On Sat, Jan 9, 2021 at 4:09 PM Jacob Faibussowitsch <jacob.fai(a)gmail.com <mailto:[email protected]>> wrote: >>> I’m currently working on an implementation of a general PetscStream object. Currently it only supports Vector ops and has a proof of concept KSPCG, but should be extensible to other objects when finished. Junchao is also indirectly working on pipeline support in his NVSHMEM MR. Take a look at either MR, it would be very useful to get your input, as tailoring either of these approaches for pipelined algorithms is key. >>> >>> Best regards, >>> >>> Jacob Faibussowitsch >>> (Jacob Fai - booss - oh - vitch) >>> Cell: (312) 694-3391 >>> >>>> On Jan 9, 2021, at 15:01, Mark Adams <mfadams(a)lbl.gov <mailto:[email protected]>> wrote: >>>> >>>> I would like to put a non-overlapping ASM solve on the GPU. It's not clear that we have a model for this. >>>> >>>> PCApply_ASM currently pipelines the scater with the subdomain solves. I think we would want to change this and do a 1) scatter begin loop, 2) scatter end and non-blocking solve loop, 3) solve-wait and scatter begging loop and 4) scatter end loop. >>>> >>>> I'm not sure how to go about doing this. >>>> * Should we make a new PCApply_ASM_PARALLEL or dump this pipelining algorithm and rewrite PCApply_ASM? >>>> * Add a solver-wait method to KSP? >>>> >>>> Thoughts? >>>> >>>> Mark >>> >

1 0