February 2013 - petsc-dev - mailman3.cels.anl.gov

Re: [petsc-dev] Need mechanism to set LocalToGlobal mapping in a DMShell
by Jed Brown 21 Feb '13

21 Feb '13

On Thu, Feb 21, 2013 at 1:23 PM, Jed Brown <jedbrown(a)mcs.anl.gov> wrote: > On Thu, Feb 21, 2013 at 1:10 PM, Matthew Knepley <knepley(a)gmail.com>wrote: > >> Yes, he needs the local to global scatters. Rich, I think the easiest way >> to do this now may be >> to define your data layout as a PetscSection (these are all local) using >> DMSetDefaultSection(), >> and record who owns which point using a PetscSF with DMSetPointSF(). >> >> Using this information, the DM will automatically construct a scatter >> between local and global, >> and the CreateLocal/GlobalVector() functions will also work right. This >> is what its is designed for. >> >> Does this make sense? >> > > > Rich, this will work fine if you don't mind having PETSc do the > communication. It will be done using PetscSFBcastBegin/End, which should > have similar performance to VecScatterBegin/End. (I should do a performance > test of that.) If you want to use your own internal communication routines, > then we need to add the DMShell functions to get your functions called. > It occurs to me that the description above might be pretty confusing. The idea is that we start with your local space and lump it together into (possibly variable-size) chunks, usually one per topological point in the mesh. Matt would describe this chunking with a PetscSection where the chunks are indexed by "point" (an integer). The next component is that we have to describe how the points fit together, so we make a PetscSF where the "roots" are the global points, referred to by (rank, offset) pairs or global indices, and the "leaves" are the local space. You use PetscSFSetGraph() to create this, which Matt calls the "Point SF". If you tell the DM about these two things by using DMSetDefaultSection() and DMSetPointSF(), then DM will create the SF to communicate the Vec data (using simple composition). You could also just use DMSetDefaultSF() directly, in which case you don't ever need to touch a PetscSection. The alternative to use VecScatter is also fine, but sometimes it is easier to use PetscSF. The Point SF, in particular, is very useful for handling integer data in the setup phase. Its API is richer than VecScatter, in addition to supporting more data types.

1 0

Re: [petsc-dev] Need mechanism to set LocalToGlobal mapping in a DMShell
by Barry Smith 21 Feb '13

21 Feb '13

On Feb 21, 2013, at 1:39 PM, Jed Brown <jedbrown(a)mcs.anl.gov> wrote: > > On Thu, Feb 21, 2013 at 1:31 PM, Barry Smith <bsmith(a)mcs.anl.gov> wrote: > I don't understand. There is a DMGlobalToLocalBegin/End() which includes > > } else { > ierr = (*dm->ops->globaltolocalbegin)(dm,g,mode == INSERT_ALL_VALUES ? INSERT_VALUES : (mode == ADD_ALL_VALUES ? ADD_VALUES : mode),l);CHKERRQ(ierr); > } > > For Shell, Richard should be able to provide his own begin/end functions. Which in his case he is already calling the VecScatterBegin/End() directly in his code so he should just be able to keep the scatter context he is already creating and use it in shell. No reason to mess with sections or SF, he already has the scatter and just needs to get it into the DMShell and shell should have it as a matter of course. Just because everyone was to lazy to add DMSetGlobalToLocalBegin/End() doesn't mean it should exist. Rich can add it himself trivially and then use (even from Fortran). > > > Yes, if he has VecScatters already that he wants to reuse, this is a fine option. I proposed this interface earlier in this thread. > > DMShellSetGlobalToLocal(DM dm,PetscErrorCode (*begin)(DM,Vec,InsertMode, > Vec),PetscErrorCode (*end)(DM,Vec,InsertMode,Vec)); Ok, Richard should add this and then end of conversation. Barry >

1 0

Re: [petsc-dev] Need mechanism to set LocalToGlobal mapping in a DMShell
by Jed Brown 21 Feb '13

21 Feb '13

On Thu, Feb 21, 2013 at 1:31 PM, Barry Smith <bsmith(a)mcs.anl.gov> wrote: > I don't understand. There is a DMGlobalToLocalBegin/End() which includes > > } else { > ierr = (*dm->ops->globaltolocalbegin)(dm,g,mode == INSERT_ALL_VALUES ? > INSERT_VALUES : (mode == ADD_ALL_VALUES ? ADD_VALUES : > mode),l);CHKERRQ(ierr); > } > > For Shell, Richard should be able to provide his own begin/end functions. > Which in his case he is already calling the VecScatterBegin/End() directly > in his code so he should just be able to keep the scatter context he is > already creating and use it in shell. No reason to mess with sections or > SF, he already has the scatter and just needs to get it into the DMShell > and shell should have it as a matter of course. Just because everyone was > to lazy to add DMSetGlobalToLocalBegin/End() doesn't mean it should exist. > Rich can add it himself trivially and then use (even from Fortran). > Yes, if he has VecScatters already that he wants to reuse, this is a fine option. I proposed this interface earlier in this thread. DMShellSetGlobalToLocal(DM dm,PetscErrorCode (*begin)(DM,Vec,InsertMode, Vec),PetscErrorCode (*end)(DM,Vec,InsertMode,Vec));

1 0

Re: [petsc-dev] Need mechanism to set LocalToGlobal mapping in a DMShell
by Barry Smith 21 Feb '13

21 Feb '13

On Feb 21, 2013, at 1:23 PM, Jed Brown <jedbrown(a)mcs.anl.gov> wrote: > > On Thu, Feb 21, 2013 at 1:10 PM, Matthew Knepley <knepley(a)gmail.com> wrote: > Yes, he needs the local to global scatters. Rich, I think the easiest way to do this now may be > to define your data layout as a PetscSection (these are all local) using DMSetDefaultSection(), > and record who owns which point using a PetscSF with DMSetPointSF(). > > Using this information, the DM will automatically construct a scatter between local and global, > and the CreateLocal/GlobalVector() functions will also work right. This is what its is designed for. > > Does this make sense? > > > Rich, this will work fine if you don't mind having PETSc do the communication. It will be done using PetscSFBcastBegin/End, which should have similar performance to VecScatterBegin/End. (I should do a performance test of that.) If you want to use your own internal communication routines, then we need to add the DMShell functions to get your functions called. I don't understand. There is a DMGlobalToLocalBegin/End() which includes } else { ierr = (*dm->ops->globaltolocalbegin)(dm,g,mode == INSERT_ALL_VALUES ? INSERT_VALUES : (mode == ADD_ALL_VALUES ? ADD_VALUES : mode),l);CHKERRQ(ierr); } For Shell, Richard should be able to provide his own begin/end functions. Which in his case he is already calling the VecScatterBegin/End() directly in his code so he should just be able to keep the scatter context he is already creating and use it in shell. No reason to mess with sections or SF, he already has the scatter and just needs to get it into the DMShell and shell should have it as a matter of course. Just because everyone was to lazy to add DMSetGlobalToLocalBegin/End() doesn't mean it should exist. Rich can add it himself trivially and then use (even from Fortran). Barry

1 0

Re: [petsc-dev] Need mechanism to set LocalToGlobal mapping in a DMShell
by Jed Brown 21 Feb '13

21 Feb '13

On Thu, Feb 21, 2013 at 1:10 PM, Matthew Knepley <knepley(a)gmail.com> wrote: > Yes, he needs the local to global scatters. Rich, I think the easiest way > to do this now may be > to define your data layout as a PetscSection (these are all local) using > DMSetDefaultSection(), > and record who owns which point using a PetscSF with DMSetPointSF(). > > Using this information, the DM will automatically construct a scatter > between local and global, > and the CreateLocal/GlobalVector() functions will also work right. This is > what its is designed for. > > Does this make sense? > Rich, this will work fine if you don't mind having PETSc do the communication. It will be done using PetscSFBcastBegin/End, which should have similar performance to VecScatterBegin/End. (I should do a performance test of that.) If you want to use your own internal communication routines, then we need to add the DMShell functions to get your functions called.

1 0

Re: [petsc-dev] Need mechanism to set LocalToGlobal mapping in a DMShell
by Matthew Knepley 21 Feb '13

21 Feb '13

On Thu, Feb 21, 2013 at 2:07 PM, Jed Brown <jedbrown(a)mcs.anl.gov> wrote: > > On Thu, Feb 21, 2013 at 1:05 PM, Peter Brune <prbrune(a)gmail.com> wrote: > >> >> The Set is the issue at hand; >> > > I think he needs scatters, not ISLocalToGlobalMappings, as mentioned in my > other email. > Yes, he needs the local to global scatters. Rich, I think the easiest way to do this now may be to define your data layout as a PetscSection (these are all local) using DMSetDefaultSection(), and record who owns which point using a PetscSF with DMSetPointSF(). Using this information, the DM will automatically construct a scatter between local and global, and the CreateLocal/GlobalVector() functions will also work right. This is what its is designed for. Does this make sense? Matt > > >> and it probably belongs in DMShell as a Set/SetCreate pair like >> everything else in DMShell. >> > > Yes > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener

1 0

Re: [petsc-dev] Need mechanism to set LocalToGlobal mapping in a DMShell
by Jed Brown 21 Feb '13

21 Feb '13

On Thu, Feb 21, 2013 at 1:05 PM, Peter Brune <prbrune(a)gmail.com> wrote: > > The Set is the issue at hand; > I think he needs scatters, not ISLocalToGlobalMappings, as mentioned in my other email. > and it probably belongs in DMShell as a Set/SetCreate pair like everything > else in DMShell. > Yes

1 0

Re: [petsc-dev] Need mechanism to set LocalToGlobal mapping in a DMShell
by Peter Brune 21 Feb '13

21 Feb '13

On Thu, Feb 21, 2013 at 12:58 PM, Dmitry Karpeev <karpeev(a)mcs.anl.gov>wrote: > > > On Thu, Feb 21, 2013 at 12:54 PM, Richard Tran Mills <rtm(a)eecs.utk.edu>wrote: > >> Hi Jed, >> >> The reason is that in the PFLOTRAN code we currently have some code like >> this: >> >> select case(discretization%itype) >> case(STRUCTURED_GRID,STRUCTURED_GRID_MIMETIC) >> call >> DMLocalToGlobalBegin(dm_ptr%sgdm,local_vec,INSERT_VALUES,global_vec,ierr) >> call >> DMLocalToGlobalEnd(dm_ptr%sgdm,local_vec,INSERT_VALUES,global_vec,ierr) >> case(UNSTRUCTURED_GRID) >> call VecScatterBegin(dm_ptr%ugdm%scatter_ltog,local_vec,global_vec, >> & >> INSERT_VALUES,SCATTER_FORWARD,ierr) >> call VecScatterEnd(dm_ptr%ugdm%scatter_ltog,local_vec,global_vec, & >> INSERT_VALUES,SCATTER_FORWARD,ierr) >> end select >> >> where in the STRUCTURED_GRID case we have a DMDA. In the >> UNSTRUCTURED_GRID case we have no DM, and I want to pack things into a >> DMShell so that have a DM and the various DM calls work (and then I can >> eliminate these 'select case' statements). >> >> (Eventually I'd like to be using more of the PETSc infrastructure and not >> even calling DMLocalToGlobalBegin(), etc., in the PFLOTRAN code, but that >> can wait a bit.) >> > I would vote for DMGetLocalToGlobalMapping() etc. > That has the flexibility to be used with DM-impl-oblivious codes. > The Set is the issue at hand; and it probably belongs in DMShell as a Set/SetCreate pair like everything else in DMShell. > Dmitry. > >> >> >> --Richard >> >> >> >> On 2/21/13 1:42 PM, Jed Brown wrote: >> >> It would be natural to have a matching interface, but why do you need it? >> You're in charge of making the matrices and vectors so you can give them >> local-to-global mappings. The only solver that calls >> DMGetLocalToGlobalMapping is PCEXOTIC, which only works with DMDA anyway. >> >> >> On Thu, Feb 21, 2013 at 12:35 PM, Richard Tran Mills <rtm(a)eecs.utk.edu>wrote: >> >>> Hi Folks, >>> >>> I want to set a LocalToGlobal mapping in a DMShell object. Should we >>> add a DMShell routine to do this, or just add a DMSetLocalToGlobalMapping() >>> routine? >>> >>> Thanks, >>> Richard >>> >>> -- >>> Richard Tran Mills, Ph.D. >>> Computational Earth Scientist | Joint Assistant Professor >>> Hydrogeochemical Dynamics Team | EECS and Earth & Planetary Sciences >>> Oak Ridge National Laboratory | University of Tennessee, Knoxville >>> E-mail: rmills(a)ornl.gov V: 865-241-3198 http://climate.ornl.gov/~rmills >>> >>> >> >> >> -- >> Richard Tran Mills, Ph.D. >> Computational Earth Scientist | Joint Assistant Professor >> Hydrogeochemical Dynamics Team | EECS and Earth & Planetary Sciences >> Oak Ridge National Laboratory | University of Tennessee, Knoxville >> E-mail: rmills(a)ornl.gov V: 865-241-3198 http://climate.ornl.gov/~rmills >> >> > >

1 0

Re: [petsc-dev] Need mechanism to set LocalToGlobal mapping in a DMShell
by Jed Brown 21 Feb '13

21 Feb '13

On Thu, Feb 21, 2013 at 12:58 PM, Dmitry Karpeev <karpeev(a)mcs.anl.gov>wrote: > I would vote for DMGetLocalToGlobalMapping() etc. > That has the flexibility to be used with DM-impl-oblivious codes. > http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/DM/DMGetLocalTo… It looks like Rich wants to run his scatters using DMGlobalToLocalBegin/End, and doesn't really care about about the ISLocalToGlobalMapping.

1 0

Re: [petsc-dev] Need mechanism to set LocalToGlobal mapping in a DMShell
by Jed Brown 21 Feb '13

21 Feb '13

On Thu, Feb 21, 2013 at 12:48 PM, Peter Brune <prbrune(a)gmail.com> wrote: > Jed, I disagree; It's quite natural to want custom bindings for these > things if you want some new mesh type to have local-to-global and > global-to-local operations over that mesh. Having it abstracted out to the > matrix or vector is constraining. > > They should totally be in there and should be implementation-level in > DMShell. > There is a member dm->ltogmap that these access, not a member of DM_Shell. I'm saying that almost nobody uses those at present. Yes, they should be implemented for symmetry, but it is unlikely to be necessary for Rich to have some capability _now_. I foresee more solvers using it in the future, and it's cleaner to get it from the DM than from the Vecs, so I agree that it should be set. The scatter DMGlobalToLocalBegin/End is a different matter. It is currently never called by the solvers directly, but it may be convenient for Rich's code to call the same function for DMShell and others. This should perhaps be called DMShellSetGlobalToLocal(DM dm,PetscErrorCode (*begin)(DM,Vec,InsertMode,Vec),PetscErrorCode (*end)(DM,Vec,InsertMode,Vec));

1 0