October 2021 - petsc-dev - mailman3.cels.anl.gov

Re: [petsc-dev] Anyone run STREAMS on an Apple M1 system?
by Pierre Jolivet 29 Oct '21

29 Oct '21

To address your original question in case you didn’t get an answer in private, Barry, see infra what I get on a Macmini9,1. I can send you an update when I get my hands on the newer MBP if you don’t have new batch of results by then. Thanks, Pierre /Users/jolivet/petsc/arch-darwin-c-opt/bin/mpicc -o MPIVersion.o -c -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -fstack-protector -fno-stack-check -Qunused-arguments -fvisibility=hidden -Ofast -I/Users/jolivet/petsc/include -I/Users/jolivet/petsc/arch-darwin-c-opt/include `pwd`/MPIVersion.c Running streams with '/Users/jolivet/petsc/arch-darwin-c-opt/bin/mpiexec ' using 'NPMAX=8' 1 58465.2862 Rate (MB/s) 2 57883.6350 Rate (MB/s) 0.990051 3 57538.6275 Rate (MB/s) 0.98415 4 57343.4226 Rate (MB/s) 0.980811 5 53210.3840 Rate (MB/s) 0.910119 6 50956.9355 Rate (MB/s) 0.871576 7 51621.7521 Rate (MB/s) 0.882947 8 48377.2931 Rate (MB/s) 0.827453 ------------------------------------------------ Traceback (most recent call last): File "process.py", line 89, in <module> process(sys.argv[1],len(sys.argv)-2) File "process.py", line 33, in process speedups[i] = triads[i]/triads[0] TypeError: 'dict_values' object is not subscriptable make[2]: [mpistream] Error 1 (ignored) Traceback (most recent call last): File "process.py", line 89, in <module> process(sys.argv[1],len(sys.argv)-2) File "process.py", line 33, in process speedups[i] = triads[i]/triads[0] TypeError: 'dict_values' object is not subscriptable make[2]: [mpistreams] Error 1 (ignored) > On 25 Oct 2021, at 10:52 PM, Jed Brown <jed(a)jedbrown.org> wrote: > > This shows 240 GB/s using 10 cores (8 performance + 2 efficiency) and 224 GB/s with 8 cores (as you'd most likely run HPC apps). Good, but far from the theoretical 400 GB/s headline. > > https://www.anandtech.com/show/17024/apple-m1-max-performance-review/2 > > Barry Smith <bsmith(a)petsc.dev> writes: > >> Thanks, presumably we'll see the new Mac's there in a few days. BTW: the old streams benchmark page should point to this site; google is worthless. >> >> I get 24 on my Intel MacBook Pro and it also saturates with 1 core. >> >> Barry >> >> >>> On Oct 18, 2021, at 4:48 PM, Jed Brown <jed(a)jedbrown.org> wrote: >>> >>> I don't have one, but this suggests it gets about 40 GB/s and can be saturated by a single core. I believe it uses two channels of LPDDR4X-4266, which has a theoretical peak of 68 GB/s. >>> >>> https://browser.geekbench.com/v3/cpu/8931693 >>> >>> The press release claims up to 400 GB/s on the Max using DDR5. I assume that's calculated based on 8 channels of LPDDR5-6400, which seems like a surprisingly big step and I'm skeptical of what will actually be realized. >>> >>> Barry Smith <bsmith(a)petsc.dev> writes: >>> >>>> Can anyone who owns an Apple M1 system run the MPI streams benchmark? Make sure the -O3 (or something) optimization flags are turned on. >>>> >>>> Thanks >>>> >>>> Barry

1 0

Re: [petsc-dev] DMPLEX with VecGhost
by Jed Brown 28 Oct '21

28 Oct '21

Yeah, I think that's about true. Should the interface for this be to set a flag before creating fields (and perhaps that flag can become true by default at some point)? Barry Smith <bsmith(a)petsc.dev> writes: > Seems like it should be possible to simply "rewrite" the cython code into PETSc C. > > Barry > > >> On Oct 28, 2021, at 12:36 PM, Matthew Knepley <knepley(a)gmail.com> wrote: >> >> I agree it is. >> >> Matt >> >> On Thu, Oct 28, 2021 at 11:53 AM Barry Smith <bsmith(a)petsc.dev <mailto:[email protected]>> wrote: >> >> Thanks, sounds like something DMPLEX should be automatically optionally doing directly. >> >> > On Oct 28, 2021, at 10:58 AM, Lawrence Mitchell <wence(a)gmx.li <mailto:[email protected]>> wrote: >> > >> > Hi Barry et al., >> > >> >> On 28 Oct 2021, at 15:37, Barry Smith <bsmith(a)petsc.dev <mailto:[email protected]>> wrote: >> >> >> >> >> >> >> >>> On Oct 28, 2021, at 10:31 AM, Matthew Knepley <knepley(a)gmail.com <mailto:[email protected]>> wrote: >> >>> >> >>> On Thu, Oct 28, 2021 at 9:37 AM Barry Smith <bsmith(a)petsc.dev <mailto:[email protected]>> wrote: >> >>> >> >>> Matt, >> >>> >> >>> How difficult would it be to rework DMPLEX to allow the use of VecGhost? We have performance problems with GPUs with simple DMNETWORK models because the code spends more time uselessly copying the local part of the vector to another vector in global to local and local to global; more than 1/2 the time of the total simulation. >> >>> >> >>> Firedrake already does this because they "vec ghost" their vectors by default. Here is what you need: >> >>> >> >>> When you create the PetscSection, by default it orders the unknowns according to the default point numbering. This >> >>> is what causes the ghost unknowns to be mixed in with the local unknowns. However, PetscSection allows you to set >> >>> a point permutation >> >>> >> >>> https://petsc.org/main/docs/manualpages/PetscSection/PetscSectionSetPermuta… <https://petsc.org/main/docs/manualpages/PetscSection/PetscSectionSetPermuta…> >> >>> >> >>> This determines the order of dogs by iterating through points in this permutation, and you can put all shared points at the end. >> >> >> >> How do I know what are shared points to put at the end? Couldn't DMPLEX do this automatically with an option? Where is the Firedrake code that does this with DMPLEX so I can see it? >> > >> > The DM's "point sf" indicates shared points. To label them, do something like: >> > >> > DMCreateLabel(dm, "ghost_points"); >> > DMGetPointSF(dm, &pointsf); >> > PetscSFGetGraph(pointsf, NULL, &nleaves, &ilocal, NULL); >> > for (PetscInt p = 0; p < nleaves; p++) { >> > DMSetLabelValue(dm, "ghost_points", p, 1); >> > } >> > >> > Then you do something like (this is more pseudo-codey): >> > >> > PetscInt *permutation; >> > DMPlexGetChart(dm, &pStart, &pEnd); >> > PetscMalloc1(pEnd - pStart, &permutation); >> > PetscInt offsets[2] = {0, pEnd - pStart - nleaves}; >> > DMGetLabel(dm, &ghosts); >> > DMLabelCreateIndex(ghosts, pStart, pEnd); >> > for (PetscInt p = pStart, p < pEnd; p++) { >> > DMLabelHasPoint(ghosts, p, &has); >> > if (has) { >> > // this point is ghost point >> > permutation[offsets[1]++] = p; >> > } else { >> > permutation[offsets[0]++] = p; >> > } >> > } >> > DMLabelDestroyIndex(ghosts, pStart, pEnd); >> > ISCreateGeneral(..., permutation, &isperm); >> > >> > // Now whenever your do PetscSectionCreate, do >> > PetscSectionSetPermutation(..., isperm); >> > >> > And now ghost point dofs will appear after local ones. >> > >> > Note that this is probably more complicated for multi-field setups, depending whether you are point major or field major. >> > >> > You can see what we actually do (if you like reading Cython) here: https://github.com/firedrakeproject/firedrake/blob/master/firedrake/cython/… <https://github.com/firedrakeproject/firedrake/blob/master/firedrake/cython/…> >> > >> > It's more complicated because here we are doing some additional things: >> > >> > 1. We compute an RCM-order traversal for cells with DMPlexGetOrdering; >> > 2. Rather than ordering all plex points in the permutation, we walk the cells in the RCM order and then greedily number points in the transitive closure (so that in a section cell and vertex dofs from the same cell will be "close" to each other in the final Vec). >> > >> > Thanks, >> > >> > Lawrence >> >> >> >> -- >> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. >> -- Norbert Wiener >> >> https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>

1 0

Re: [petsc-dev] DMPLEX with VecGhost
by Barry Smith 28 Oct '21

28 Oct '21

Seems like it should be possible to simply "rewrite" the cython code into PETSc C. Barry > On Oct 28, 2021, at 12:36 PM, Matthew Knepley <knepley(a)gmail.com> wrote: > > I agree it is. > > Matt > > On Thu, Oct 28, 2021 at 11:53 AM Barry Smith <bsmith(a)petsc.dev <mailto:[email protected]>> wrote: > > Thanks, sounds like something DMPLEX should be automatically optionally doing directly. > > > On Oct 28, 2021, at 10:58 AM, Lawrence Mitchell <wence(a)gmx.li <mailto:[email protected]>> wrote: > > > > Hi Barry et al., > > > >> On 28 Oct 2021, at 15:37, Barry Smith <bsmith(a)petsc.dev <mailto:[email protected]>> wrote: > >> > >> > >> > >>> On Oct 28, 2021, at 10:31 AM, Matthew Knepley <knepley(a)gmail.com <mailto:[email protected]>> wrote: > >>> > >>> On Thu, Oct 28, 2021 at 9:37 AM Barry Smith <bsmith(a)petsc.dev <mailto:[email protected]>> wrote: > >>> > >>> Matt, > >>> > >>> How difficult would it be to rework DMPLEX to allow the use of VecGhost? We have performance problems with GPUs with simple DMNETWORK models because the code spends more time uselessly copying the local part of the vector to another vector in global to local and local to global; more than 1/2 the time of the total simulation. > >>> > >>> Firedrake already does this because they "vec ghost" their vectors by default. Here is what you need: > >>> > >>> When you create the PetscSection, by default it orders the unknowns according to the default point numbering. This > >>> is what causes the ghost unknowns to be mixed in with the local unknowns. However, PetscSection allows you to set > >>> a point permutation > >>> > >>> https://petsc.org/main/docs/manualpages/PetscSection/PetscSectionSetPermuta… <https://petsc.org/main/docs/manualpages/PetscSection/PetscSectionSetPermuta…> > >>> > >>> This determines the order of dogs by iterating through points in this permutation, and you can put all shared points at the end. > >> > >> How do I know what are shared points to put at the end? Couldn't DMPLEX do this automatically with an option? Where is the Firedrake code that does this with DMPLEX so I can see it? > > > > The DM's "point sf" indicates shared points. To label them, do something like: > > > > DMCreateLabel(dm, "ghost_points"); > > DMGetPointSF(dm, &pointsf); > > PetscSFGetGraph(pointsf, NULL, &nleaves, &ilocal, NULL); > > for (PetscInt p = 0; p < nleaves; p++) { > > DMSetLabelValue(dm, "ghost_points", p, 1); > > } > > > > Then you do something like (this is more pseudo-codey): > > > > PetscInt *permutation; > > DMPlexGetChart(dm, &pStart, &pEnd); > > PetscMalloc1(pEnd - pStart, &permutation); > > PetscInt offsets[2] = {0, pEnd - pStart - nleaves}; > > DMGetLabel(dm, &ghosts); > > DMLabelCreateIndex(ghosts, pStart, pEnd); > > for (PetscInt p = pStart, p < pEnd; p++) { > > DMLabelHasPoint(ghosts, p, &has); > > if (has) { > > // this point is ghost point > > permutation[offsets[1]++] = p; > > } else { > > permutation[offsets[0]++] = p; > > } > > } > > DMLabelDestroyIndex(ghosts, pStart, pEnd); > > ISCreateGeneral(..., permutation, &isperm); > > > > // Now whenever your do PetscSectionCreate, do > > PetscSectionSetPermutation(..., isperm); > > > > And now ghost point dofs will appear after local ones. > > > > Note that this is probably more complicated for multi-field setups, depending whether you are point major or field major. > > > > You can see what we actually do (if you like reading Cython) here: https://github.com/firedrakeproject/firedrake/blob/master/firedrake/cython/… <https://github.com/firedrakeproject/firedrake/blob/master/firedrake/cython/…> > > > > It's more complicated because here we are doing some additional things: > > > > 1. We compute an RCM-order traversal for cells with DMPlexGetOrdering; > > 2. Rather than ordering all plex points in the permutation, we walk the cells in the RCM order and then greedily number points in the transitive closure (so that in a section cell and vertex dofs from the same cell will be "close" to each other in the final Vec). > > > > Thanks, > > > > Lawrence > > > > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>

1 0

Re: [petsc-dev] DMPLEX with VecGhost
by Matthew Knepley 28 Oct '21

28 Oct '21

I agree it is. Matt On Thu, Oct 28, 2021 at 11:53 AM Barry Smith <bsmith(a)petsc.dev> wrote: > > Thanks, sounds like something DMPLEX should be automatically optionally > doing directly. > > > On Oct 28, 2021, at 10:58 AM, Lawrence Mitchell <wence(a)gmx.li> wrote: > > > > Hi Barry et al., > > > >> On 28 Oct 2021, at 15:37, Barry Smith <bsmith(a)petsc.dev> wrote: > >> > >> > >> > >>> On Oct 28, 2021, at 10:31 AM, Matthew Knepley <knepley(a)gmail.com> > wrote: > >>> > >>> On Thu, Oct 28, 2021 at 9:37 AM Barry Smith <bsmith(a)petsc.dev> wrote: > >>> > >>> Matt, > >>> > >>> How difficult would it be to rework DMPLEX to allow the use of > VecGhost? We have performance problems with GPUs with simple DMNETWORK > models because the code spends more time uselessly copying the local part > of the vector to another vector in global to local and local to global; > more than 1/2 the time of the total simulation. > >>> > >>> Firedrake already does this because they "vec ghost" their vectors by > default. Here is what you need: > >>> > >>> When you create the PetscSection, by default it orders the unknowns > according to the default point numbering. This > >>> is what causes the ghost unknowns to be mixed in with the local > unknowns. However, PetscSection allows you to set > >>> a point permutation > >>> > >>> > https://petsc.org/main/docs/manualpages/PetscSection/PetscSectionSetPermuta… > >>> > >>> This determines the order of dogs by iterating through points in this > permutation, and you can put all shared points at the end. > >> > >> How do I know what are shared points to put at the end? Couldn't > DMPLEX do this automatically with an option? Where is the Firedrake code > that does this with DMPLEX so I can see it? > > > > The DM's "point sf" indicates shared points. To label them, do something > like: > > > > DMCreateLabel(dm, "ghost_points"); > > DMGetPointSF(dm, &pointsf); > > PetscSFGetGraph(pointsf, NULL, &nleaves, &ilocal, NULL); > > for (PetscInt p = 0; p < nleaves; p++) { > > DMSetLabelValue(dm, "ghost_points", p, 1); > > } > > > > Then you do something like (this is more pseudo-codey): > > > > PetscInt *permutation; > > DMPlexGetChart(dm, &pStart, &pEnd); > > PetscMalloc1(pEnd - pStart, &permutation); > > PetscInt offsets[2] = {0, pEnd - pStart - nleaves}; > > DMGetLabel(dm, &ghosts); > > DMLabelCreateIndex(ghosts, pStart, pEnd); > > for (PetscInt p = pStart, p < pEnd; p++) { > > DMLabelHasPoint(ghosts, p, &has); > > if (has) { > > // this point is ghost point > > permutation[offsets[1]++] = p; > > } else { > > permutation[offsets[0]++] = p; > > } > > } > > DMLabelDestroyIndex(ghosts, pStart, pEnd); > > ISCreateGeneral(..., permutation, &isperm); > > > > // Now whenever your do PetscSectionCreate, do > > PetscSectionSetPermutation(..., isperm); > > > > And now ghost point dofs will appear after local ones. > > > > Note that this is probably more complicated for multi-field setups, > depending whether you are point major or field major. > > > > You can see what we actually do (if you like reading Cython) here: > https://github.com/firedrakeproject/firedrake/blob/master/firedrake/cython/… > > > > It's more complicated because here we are doing some additional things: > > > > 1. We compute an RCM-order traversal for cells with DMPlexGetOrdering; > > 2. Rather than ordering all plex points in the permutation, we walk the > cells in the RCM order and then greedily number points in the transitive > closure (so that in a section cell and vertex dofs from the same cell will > be "close" to each other in the final Vec). > > > > Thanks, > > > > Lawrence > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>

1 0

Re: [petsc-dev] DMPLEX with VecGhost
by Barry Smith 28 Oct '21

28 Oct '21

Thanks, sounds like something DMPLEX should be automatically optionally doing directly. > On Oct 28, 2021, at 10:58 AM, Lawrence Mitchell <wence(a)gmx.li> wrote: > > Hi Barry et al., > >> On 28 Oct 2021, at 15:37, Barry Smith <bsmith(a)petsc.dev> wrote: >> >> >> >>> On Oct 28, 2021, at 10:31 AM, Matthew Knepley <knepley(a)gmail.com> wrote: >>> >>> On Thu, Oct 28, 2021 at 9:37 AM Barry Smith <bsmith(a)petsc.dev> wrote: >>> >>> Matt, >>> >>> How difficult would it be to rework DMPLEX to allow the use of VecGhost? We have performance problems with GPUs with simple DMNETWORK models because the code spends more time uselessly copying the local part of the vector to another vector in global to local and local to global; more than 1/2 the time of the total simulation. >>> >>> Firedrake already does this because they "vec ghost" their vectors by default. Here is what you need: >>> >>> When you create the PetscSection, by default it orders the unknowns according to the default point numbering. This >>> is what causes the ghost unknowns to be mixed in with the local unknowns. However, PetscSection allows you to set >>> a point permutation >>> >>> https://petsc.org/main/docs/manualpages/PetscSection/PetscSectionSetPermuta… >>> >>> This determines the order of dogs by iterating through points in this permutation, and you can put all shared points at the end. >> >> How do I know what are shared points to put at the end? Couldn't DMPLEX do this automatically with an option? Where is the Firedrake code that does this with DMPLEX so I can see it? > > The DM's "point sf" indicates shared points. To label them, do something like: > > DMCreateLabel(dm, "ghost_points"); > DMGetPointSF(dm, &pointsf); > PetscSFGetGraph(pointsf, NULL, &nleaves, &ilocal, NULL); > for (PetscInt p = 0; p < nleaves; p++) { > DMSetLabelValue(dm, "ghost_points", p, 1); > } > > Then you do something like (this is more pseudo-codey): > > PetscInt *permutation; > DMPlexGetChart(dm, &pStart, &pEnd); > PetscMalloc1(pEnd - pStart, &permutation); > PetscInt offsets[2] = {0, pEnd - pStart - nleaves}; > DMGetLabel(dm, &ghosts); > DMLabelCreateIndex(ghosts, pStart, pEnd); > for (PetscInt p = pStart, p < pEnd; p++) { > DMLabelHasPoint(ghosts, p, &has); > if (has) { > // this point is ghost point > permutation[offsets[1]++] = p; > } else { > permutation[offsets[0]++] = p; > } > } > DMLabelDestroyIndex(ghosts, pStart, pEnd); > ISCreateGeneral(..., permutation, &isperm); > > // Now whenever your do PetscSectionCreate, do > PetscSectionSetPermutation(..., isperm); > > And now ghost point dofs will appear after local ones. > > Note that this is probably more complicated for multi-field setups, depending whether you are point major or field major. > > You can see what we actually do (if you like reading Cython) here: https://github.com/firedrakeproject/firedrake/blob/master/firedrake/cython/… > > It's more complicated because here we are doing some additional things: > > 1. We compute an RCM-order traversal for cells with DMPlexGetOrdering; > 2. Rather than ordering all plex points in the permutation, we walk the cells in the RCM order and then greedily number points in the transitive closure (so that in a section cell and vertex dofs from the same cell will be "close" to each other in the final Vec). > > Thanks, > > Lawrence

1 0

Re: [petsc-dev] DMPLEX with VecGhost
by Lawrence Mitchell 28 Oct '21

28 Oct '21

Hi Barry et al., > On 28 Oct 2021, at 15:37, Barry Smith <bsmith(a)petsc.dev> wrote: > > > >> On Oct 28, 2021, at 10:31 AM, Matthew Knepley <knepley(a)gmail.com> wrote: >> >> On Thu, Oct 28, 2021 at 9:37 AM Barry Smith <bsmith(a)petsc.dev> wrote: >> >> Matt, >> >> How difficult would it be to rework DMPLEX to allow the use of VecGhost? We have performance problems with GPUs with simple DMNETWORK models because the code spends more time uselessly copying the local part of the vector to another vector in global to local and local to global; more than 1/2 the time of the total simulation. >> >> Firedrake already does this because they "vec ghost" their vectors by default. Here is what you need: >> >> When you create the PetscSection, by default it orders the unknowns according to the default point numbering. This >> is what causes the ghost unknowns to be mixed in with the local unknowns. However, PetscSection allows you to set >> a point permutation >> >> https://petsc.org/main/docs/manualpages/PetscSection/PetscSectionSetPermuta… >> >> This determines the order of dogs by iterating through points in this permutation, and you can put all shared points at the end. > > How do I know what are shared points to put at the end? Couldn't DMPLEX do this automatically with an option? Where is the Firedrake code that does this with DMPLEX so I can see it? The DM's "point sf" indicates shared points. To label them, do something like: DMCreateLabel(dm, "ghost_points"); DMGetPointSF(dm, &pointsf); PetscSFGetGraph(pointsf, NULL, &nleaves, &ilocal, NULL); for (PetscInt p = 0; p < nleaves; p++) { DMSetLabelValue(dm, "ghost_points", p, 1); } Then you do something like (this is more pseudo-codey): PetscInt *permutation; DMPlexGetChart(dm, &pStart, &pEnd); PetscMalloc1(pEnd - pStart, &permutation); PetscInt offsets[2] = {0, pEnd - pStart - nleaves}; DMGetLabel(dm, &ghosts); DMLabelCreateIndex(ghosts, pStart, pEnd); for (PetscInt p = pStart, p < pEnd; p++) { DMLabelHasPoint(ghosts, p, &has); if (has) { // this point is ghost point permutation[offsets[1]++] = p; } else { permutation[offsets[0]++] = p; } } DMLabelDestroyIndex(ghosts, pStart, pEnd); ISCreateGeneral(..., permutation, &isperm); // Now whenever your do PetscSectionCreate, do PetscSectionSetPermutation(..., isperm); And now ghost point dofs will appear after local ones. Note that this is probably more complicated for multi-field setups, depending whether you are point major or field major. You can see what we actually do (if you like reading Cython) here: https://github.com/firedrakeproject/firedrake/blob/master/firedrake/cython/… It's more complicated because here we are doing some additional things: 1. We compute an RCM-order traversal for cells with DMPlexGetOrdering; 2. Rather than ordering all plex points in the permutation, we walk the cells in the RCM order and then greedily number points in the transitive closure (so that in a section cell and vertex dofs from the same cell will be "close" to each other in the final Vec). Thanks, Lawrence

1 0

Re: [petsc-dev] DMPLEX with VecGhost
by Matthew Knepley 28 Oct '21

28 Oct '21

On Thu, Oct 28, 2021 at 10:37 AM Barry Smith <bsmith(a)petsc.dev> wrote: > > > On Oct 28, 2021, at 10:31 AM, Matthew Knepley <knepley(a)gmail.com> wrote: > > On Thu, Oct 28, 2021 at 9:37 AM Barry Smith <bsmith(a)petsc.dev> wrote: > >> >> Matt, >> >> How difficult would it be to rework DMPLEX to allow the use of >> VecGhost? We have performance problems with GPUs with simple DMNETWORK >> models because the code spends more time uselessly copying the local part >> of the vector to another vector in global to local and local to global; >> more than 1/2 the time of the total simulation. >> > > Firedrake already does this because they "vec ghost" their vectors by > default. Here is what you need: > > When you create the PetscSection, by default it orders the unknowns > according to the default point numbering. This > is what causes the ghost unknowns to be mixed in with the > local unknowns. However, PetscSection allows you to set > a point permutation > > > https://petsc.org/main/docs/manualpages/PetscSection/PetscSectionSetPermuta… > > This determines the order of dogs by iterating through points in this > permutation, and you can put all shared points at the end. > > > How do I know what are shared points to put at the end? Couldn't DMPLEX > do this automatically with an option? Where is the Firedrake code that does > this with DMPLEX so I can see it? > Yes, it could. I was thinking this as I wrote you, and also thinking how tired I am :) Here is how to do it. Suppose you only care about ghost points (Firedrake wants three layers: totalling internal, roots, and ghosts), then there is a simple test DMGetLocalSection(dm, &s) PetscSectionGetChart(s, &pStart, &pEnd); DMGetPointSF(dm, &sf); PetscSFGetGraph(sf, NULL, &nleaves, &leaves, NULL) for (p = pStart; p < pEnd; ++p) { PetscFindInt(p, nleaves, leaves, &loc); if (loc >= 0) {I am a ghost point} } If you also care about roots (shared points that you own), you also need PetscSFComputeDegreeBegin(sf, &rootdegree); PetscSFComputeDegreeEnd(sf, &rootdegree); and then also check rootdegree[p] > 0 If you get it working, I will put it inside Plex and make an option to turn it on. Thanks, Matt > Does this make sense? > > Thanks, > > Matt > > >> Barry >> > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > <http://www.cse.buffalo.edu/~knepley/> > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>

1 0

Re: [petsc-dev] DMPLEX with VecGhost
by Barry Smith 28 Oct '21

28 Oct '21

> On Oct 28, 2021, at 10:31 AM, Matthew Knepley <knepley(a)gmail.com> wrote: > > On Thu, Oct 28, 2021 at 9:37 AM Barry Smith <bsmith(a)petsc.dev <mailto:[email protected]>> wrote: > > Matt, > > How difficult would it be to rework DMPLEX to allow the use of VecGhost? We have performance problems with GPUs with simple DMNETWORK models because the code spends more time uselessly copying the local part of the vector to another vector in global to local and local to global; more than 1/2 the time of the total simulation. > > Firedrake already does this because they "vec ghost" their vectors by default. Here is what you need: > > When you create the PetscSection, by default it orders the unknowns according to the default point numbering. This > is what causes the ghost unknowns to be mixed in with the local unknowns. However, PetscSection allows you to set > a point permutation > > https://petsc.org/main/docs/manualpages/PetscSection/PetscSectionSetPermuta… <https://petsc.org/main/docs/manualpages/PetscSection/PetscSectionSetPermuta…> > > This determines the order of dogs by iterating through points in this permutation, and you can put all shared points at the end. How do I know what are shared points to put at the end? Couldn't DMPLEX do this automatically with an option? Where is the Firedrake code that does this with DMPLEX so I can see it? > > Does this make sense? > > Thanks, > > Matt > > Barry > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>

1 0

Re: [petsc-dev] DMPLEX with VecGhost
by Matthew Knepley 28 Oct '21

28 Oct '21

On Thu, Oct 28, 2021 at 9:37 AM Barry Smith <bsmith(a)petsc.dev> wrote: > > Matt, > > How difficult would it be to rework DMPLEX to allow the use of > VecGhost? We have performance problems with GPUs with simple DMNETWORK > models because the code spends more time uselessly copying the local part > of the vector to another vector in global to local and local to global; > more than 1/2 the time of the total simulation. > Firedrake already does this because they "vec ghost" their vectors by default. Here is what you need: When you create the PetscSection, by default it orders the unknowns according to the default point numbering. This is what causes the ghost unknowns to be mixed in with the local unknowns. However, PetscSection allows you to set a point permutation https://petsc.org/main/docs/manualpages/PetscSection/PetscSectionSetPermuta… This determines the order of dogs by iterating through points in this permutation, and you can put all shared points at the end. Does this make sense? Thanks, Matt > Barry > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>

1 0

DMPLEX with VecGhost
by Barry Smith 28 Oct '21

28 Oct '21

Matt, How difficult would it be to rework DMPLEX to allow the use of VecGhost? We have performance problems with GPUs with simple DMNETWORK models because the code spends more time uselessly copying the local part of the vector to another vector in global to local and local to global; more than 1/2 the time of the total simulation. Barry

1 0