November 2021 - petsc-dev - mailman3.cels.anl.gov

Re: [petsc-dev] HIP / hypre
by Justin Chang 10 Nov '21

10 Nov '21

Jacob, What’s the plan going forward with this unified cuda/hip branch? Is this related to what some of us have been hearing about PETSc eventually going with a unified wrapper over both the CUDA and HIP API? We’re interested in making the HIP port more mature, including having support for the HIP part of HYPRE, but we’re unsure what direction you guys want to go with the GPU route. Thanks, Justin On Wed, Nov 10, 2021 at 12:19 PM Justin Chang <jychang48(a)gmail.com> wrote: > Cc’ing Paul since I misspelled his email address initially > > On Wed, Nov 10, 2021 at 12:17 PM Jacob Faibussowitsch <jacob.fai(a)gmail.com> > wrote: > >> I’m in the process of implementing asynchronous GPU support for petsc. A >> side effect of this is that I unify the cuda/hip interface such that >> anywhere we have cuda-like code we will automatically also get the hip >> variant. >> >> The scaffolding is in include/petsc/private/cupminterface.hpp, but for >> concrete examples see the jacobf/2021-10-21/veccupm-async branch for the >> WIP port of VecSeq in src/vec/vec/impls/seq/seqcupm/veccupm.hpp. >> >> Best regards, >> >> Jacob Faibussowitsch >> (Jacob Fai - booss - oh - vitch) >> >> On Nov 10, 2021, at 11:50, Justin Chang <jychang48(a)gmail.com> wrote: >> >> Paul Bauman was also involved with the HIP port of HYPRE. Several of us >> at AMD are interested in getting HIP support for PETSc in general, and >> having HYPRE support would greatly help >> >> On Wed, Nov 10, 2021 at 11:47 AM Stefano Zampini < >> stefano.zampini(a)gmail.com> wrote: >> >>> I did the work last summer. It's already available in 3.16 >>> >>> Il Mer 10 Nov 2021, 20:44 Mark Adams <mfadams(a)lbl.gov> ha scritto: >>> >>>> Hypre has released HIP support and Ulrike says: >>>> >>>> I just want to let you know that hypre can now be used through PETSc >>>> with GPUs (both Nvidia and AMD). >>>> >>>> I am guessing we have some work to do to make this happen. >>>> >>>> What should I do? >>>> >>> >>

1 0

Re: [petsc-dev] HIP / hypre
by Justin Chang 10 Nov '21

10 Nov '21

Cc’ing Paul since I misspelled his email address initially On Wed, Nov 10, 2021 at 12:17 PM Jacob Faibussowitsch <jacob.fai(a)gmail.com> wrote: > I’m in the process of implementing asynchronous GPU support for petsc. A > side effect of this is that I unify the cuda/hip interface such that > anywhere we have cuda-like code we will automatically also get the hip > variant. > > The scaffolding is in include/petsc/private/cupminterface.hpp, but for > concrete examples see the jacobf/2021-10-21/veccupm-async branch for the > WIP port of VecSeq in src/vec/vec/impls/seq/seqcupm/veccupm.hpp. > > Best regards, > > Jacob Faibussowitsch > (Jacob Fai - booss - oh - vitch) > > On Nov 10, 2021, at 11:50, Justin Chang <jychang48(a)gmail.com> wrote: > > Paul Bauman was also involved with the HIP port of HYPRE. Several of us at > AMD are interested in getting HIP support for PETSc in general, and having > HYPRE support would greatly help > > On Wed, Nov 10, 2021 at 11:47 AM Stefano Zampini < > stefano.zampini(a)gmail.com> wrote: > >> I did the work last summer. It's already available in 3.16 >> >> Il Mer 10 Nov 2021, 20:44 Mark Adams <mfadams(a)lbl.gov> ha scritto: >> >>> Hypre has released HIP support and Ulrike says: >>> >>> I just want to let you know that hypre can now be used through PETSc >>> with GPUs (both Nvidia and AMD). >>> >>> I am guessing we have some work to do to make this happen. >>> >>> What should I do? >>> >> >

1 0

Re: [petsc-dev] HIP / hypre
by Jacob Faibussowitsch 10 Nov '21

10 Nov '21

I’m in the process of implementing asynchronous GPU support for petsc. A side effect of this is that I unify the cuda/hip interface such that anywhere we have cuda-like code we will automatically also get the hip variant. The scaffolding is in include/petsc/private/cupminterface.hpp, but for concrete examples see the jacobf/2021-10-21/veccupm-async branch for the WIP port of VecSeq in src/vec/vec/impls/seq/seqcupm/veccupm.hpp. Best regards, Jacob Faibussowitsch (Jacob Fai - booss - oh - vitch) > On Nov 10, 2021, at 11:50, Justin Chang <jychang48(a)gmail.com> wrote: > > Paul Bauman was also involved with the HIP port of HYPRE. Several of us at AMD are interested in getting HIP support for PETSc in general, and having HYPRE support would greatly help > > On Wed, Nov 10, 2021 at 11:47 AM Stefano Zampini <stefano.zampini(a)gmail.com <mailto:[email protected]>> wrote: > I did the work last summer. It's already available in 3.16 > > Il Mer 10 Nov 2021, 20:44 Mark Adams <mfadams(a)lbl.gov <mailto:[email protected]>> ha scritto: > Hypre has released HIP support and Ulrike says: > > I just want to let you know that hypre can now be used through PETSc with GPUs (both Nvidia and AMD). > > I am guessing we have some work to do to make this happen. > > What should I do?

1 0

Re: [petsc-dev] HIP / hypre
by Mark Adams 10 Nov '21

10 Nov '21

For HIP? I know you did CUDA. I think HIP support is pretty new. On Wed, Nov 10, 2021 at 12:47 PM Stefano Zampini <stefano.zampini(a)gmail.com> wrote: > I did the work last summer. It's already available in 3.16 > > Il Mer 10 Nov 2021, 20:44 Mark Adams <mfadams(a)lbl.gov> ha scritto: > >> Hypre has released HIP support and Ulrike says: >> >> I just want to let you know that hypre can now be used through PETSc with >> GPUs (both Nvidia and AMD). >> >> I am guessing we have some work to do to make this happen. >> >> What should I do? >> >

1 0

Re: [petsc-dev] HIP / hypre
by Justin Chang 10 Nov '21

10 Nov '21

Paul Bauman was also involved with the HIP port of HYPRE. Several of us at AMD are interested in getting HIP support for PETSc in general, and having HYPRE support would greatly help On Wed, Nov 10, 2021 at 11:47 AM Stefano Zampini <stefano.zampini(a)gmail.com> wrote: > I did the work last summer. It's already available in 3.16 > > Il Mer 10 Nov 2021, 20:44 Mark Adams <mfadams(a)lbl.gov> ha scritto: > >> Hypre has released HIP support and Ulrike says: >> >> I just want to let you know that hypre can now be used through PETSc with >> GPUs (both Nvidia and AMD). >> >> I am guessing we have some work to do to make this happen. >> >> What should I do? >> >

1 0

Re: [petsc-dev] HIP / hypre
by Stefano Zampini 10 Nov '21

10 Nov '21

I did the work last summer. It's already available in 3.16 Il Mer 10 Nov 2021, 20:44 Mark Adams <mfadams(a)lbl.gov> ha scritto: > Hypre has released HIP support and Ulrike says: > > I just want to let you know that hypre can now be used through PETSc with > GPUs (both Nvidia and AMD). > > I am guessing we have some work to do to make this happen. > > What should I do? >

1 0

HIP / hypre
by Mark Adams 10 Nov '21

10 Nov '21

Hypre has released HIP support and Ulrike says: I just want to let you know that hypre can now be used through PETSc with GPUs (both Nvidia and AMD). I am guessing we have some work to do to make this happen. What should I do?

1 0

Re: [petsc-dev] DMPLEX with VecGhost
by Barry Smith 09 Nov '21

09 Nov '21

Jed, Matt, Junchao, Lawrence, I need some expert advice with this https://gitlab.com/petsc/petsc/-/merge_requests/4542 <https://gitlab.com/petsc/petsc/-/merge_requests/4542>. I was able to translate the cython to PETSc C for moving the ghost points to the end in the local section. But I don't know * the best way to generate the ghosts list needed by VecCreateGhost()? or should I not use VecCreateGhost() * but somehow instead use the existing DMGetSectionSF() to be able to PetscSF between the owned part and the ghosted part of a hybrid vector (that is, don't use VecCreateGhost() but create an equivalent somehow using the sf that already exists?) Thanks Barry > On Oct 28, 2021, at 6:37 PM, Jed Brown <jed(a)jedbrown.org> wrote: > > Yeah, I think that's about true. Should the interface for this be to set a flag before creating fields (and perhaps that flag can become true by default at some point)? > > Barry Smith <bsmith(a)petsc.dev> writes: > >> Seems like it should be possible to simply "rewrite" the cython code into PETSc C. >> >> Barry >> >> >>> On Oct 28, 2021, at 12:36 PM, Matthew Knepley <knepley(a)gmail.com> wrote: >>> >>> I agree it is. >>> >>> Matt >>> >>> On Thu, Oct 28, 2021 at 11:53 AM Barry Smith <bsmith(a)petsc.dev <mailto:[email protected]>> wrote: >>> >>> Thanks, sounds like something DMPLEX should be automatically optionally doing directly. >>> >>>> On Oct 28, 2021, at 10:58 AM, Lawrence Mitchell <wence(a)gmx.li <mailto:[email protected]>> wrote: >>>> >>>> Hi Barry et al., >>>> >>>>> On 28 Oct 2021, at 15:37, Barry Smith <bsmith(a)petsc.dev <mailto:[email protected]>> wrote: >>>>> >>>>> >>>>> >>>>>> On Oct 28, 2021, at 10:31 AM, Matthew Knepley <knepley(a)gmail.com <mailto:[email protected]>> wrote: >>>>>> >>>>>> On Thu, Oct 28, 2021 at 9:37 AM Barry Smith <bsmith(a)petsc.dev <mailto:[email protected]>> wrote: >>>>>> >>>>>> Matt, >>>>>> >>>>>> How difficult would it be to rework DMPLEX to allow the use of VecGhost? We have performance problems with GPUs with simple DMNETWORK models because the code spends more time uselessly copying the local part of the vector to another vector in global to local and local to global; more than 1/2 the time of the total simulation. >>>>>> >>>>>> Firedrake already does this because they "vec ghost" their vectors by default. Here is what you need: >>>>>> >>>>>> When you create the PetscSection, by default it orders the unknowns according to the default point numbering. This >>>>>> is what causes the ghost unknowns to be mixed in with the local unknowns. However, PetscSection allows you to set >>>>>> a point permutation >>>>>> >>>>>> https://petsc.org/main/docs/manualpages/PetscSection/PetscSectionSetPermuta… <https://petsc.org/main/docs/manualpages/PetscSection/PetscSectionSetPermuta…> >>>>>> >>>>>> This determines the order of dogs by iterating through points in this permutation, and you can put all shared points at the end. >>>>> >>>>> How do I know what are shared points to put at the end? Couldn't DMPLEX do this automatically with an option? Where is the Firedrake code that does this with DMPLEX so I can see it? >>>> >>>> The DM's "point sf" indicates shared points. To label them, do something like: >>>> >>>> DMCreateLabel(dm, "ghost_points"); >>>> DMGetPointSF(dm, &pointsf); >>>> PetscSFGetGraph(pointsf, NULL, &nleaves, &ilocal, NULL); >>>> for (PetscInt p = 0; p < nleaves; p++) { >>>> DMSetLabelValue(dm, "ghost_points", p, 1); >>>> } >>>> >>>> Then you do something like (this is more pseudo-codey): >>>> >>>> PetscInt *permutation; >>>> DMPlexGetChart(dm, &pStart, &pEnd); >>>> PetscMalloc1(pEnd - pStart, &permutation); >>>> PetscInt offsets[2] = {0, pEnd - pStart - nleaves}; >>>> DMGetLabel(dm, &ghosts); >>>> DMLabelCreateIndex(ghosts, pStart, pEnd); >>>> for (PetscInt p = pStart, p < pEnd; p++) { >>>> DMLabelHasPoint(ghosts, p, &has); >>>> if (has) { >>>> // this point is ghost point >>>> permutation[offsets[1]++] = p; >>>> } else { >>>> permutation[offsets[0]++] = p; >>>> } >>>> } >>>> DMLabelDestroyIndex(ghosts, pStart, pEnd); >>>> ISCreateGeneral(..., permutation, &isperm); >>>> >>>> // Now whenever your do PetscSectionCreate, do >>>> PetscSectionSetPermutation(..., isperm); >>>> >>>> And now ghost point dofs will appear after local ones. >>>> >>>> Note that this is probably more complicated for multi-field setups, depending whether you are point major or field major. >>>> >>>> You can see what we actually do (if you like reading Cython) here: https://github.com/firedrakeproject/firedrake/blob/master/firedrake/cython/… <https://github.com/firedrakeproject/firedrake/blob/master/firedrake/cython/…> >>>> >>>> It's more complicated because here we are doing some additional things: >>>> >>>> 1. We compute an RCM-order traversal for cells with DMPlexGetOrdering; >>>> 2. Rather than ordering all plex points in the permutation, we walk the cells in the RCM order and then greedily number points in the transitive closure (so that in a section cell and vertex dofs from the same cell will be "close" to each other in the final Vec). >>>> >>>> Thanks, >>>> >>>> Lawrence >>> >>> >>> >>> -- >>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. >>> -- Norbert Wiener >>> >>> https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>

1 0

Re: [petsc-dev] [DocTip!] #2: Aiming for self-updating docs
by Barry Smith 08 Nov '21

08 Nov '21

If we were all as smart as Knuth the community could have followed through with his idea and iteratively developed an extremely powerful set of tools, sadly we are not. > On Nov 8, 2021, at 2:23 PM, Mark Adams <mfadams(a)lbl.gov> wrote: > > > For those who love literate programming and fortran, PPPL developed this > in the 80's: > https://w3.pppl.gov/~krommes/fweb.html <https://w3.pppl.gov/~krommes/fweb.html> > and I believe it is still maintained. > > I've been working with PPPL people 15+ years and have not seen a code that uses it. > However I did once get caught by a crazy, brilliant but crazy, Russian at PPPL that yammered on to me about a Latex programming language, that sounded insane but that may have been the delivery, that may very well have been this project. > I am sure it is dead.

1 0

Re: [petsc-dev] I am getting this error ...
by Mark Adams 08 Nov '21

08 Nov '21

Yea, Junchao and I got this from NERSc within hours of each other. But, I was given NULL and I told them so. I got this response today. (*emphasis added*) Mark Yeah I am aware where that information came from in the NUG slack channel, ideally you just want to set it to NULL, you can use host for other purposes (like disabling offload on A100s in PrgEnv-Cray) but I wouldn't do it in your case for PrgEnv-nvidia -- *please tell them to do the same* (it will work, but it's not recommended for this case). Thanks, Ronnie

1 0