February 2013 - petsc-dev - mailman3.cels.anl.gov

Re: [petsc-dev] [petsc-users] MATNEST with shell matrices
by Boyce Griffith 21 Feb '13

21 Feb '13

On Feb 21, 2013, at 8:44 PM, Jed Brown <jedbrown(a)mcs.anl.gov> wrote: > On Thu, Feb 21, 2013 at 7:28 PM, Boyce Griffith <griffith(a)cims.nyu.edu> wrote: > > I would very happily use an "undefined" vector size. (I will also happily change the bogus local sizes from 0 to 1, and the global sizes accordingly. :-) > > The PETSc solvers that we use in the code should be oblivious to the size of the vector. I've actually never been all that happy with size "0", but it was in some inherited code, and at least so far, it has never caused any problems over the 10+ years it has been in use. (If ever did start causing problems, I certainly would not come complaining to petsc-users or petsc-dev. It isn't like I do not realize that "0" is a potentially problematic value here.) > > I actually have always hoped that setting size to "0" would help smoke out any places where something actually did use the vector size. > > Okay, let's move the design discussion to petsc-dev. > > I want to understand the extent to your adaptivity and how your Vecs "behave". In which places do you (mathematically) change the dimension of a vector? These applications are mainly transient solvers in which regridding happens "between" time steps. > If I start with Vecs X and Y and I write > > VecCopy(X,Y); > YourSolver(X); /* solve F(X) = 0 with adaptivity */ > > can I then compare the result like this? > > VecAXPY(Y,-1.,X); Absolutely. > Do vector-space properties of Y (e.g., norm) change when I call YourSolver(X). (Some codes keep a global view of all vectors, and when they adapt, they regrid all vectors. But regridding doesn't preserve things like norms, so you end up with tangled semantics.) At present all solvers operate only on fixed grid configurations. I do not expect this to change any time soon. -- Boyce

1 0

Re: [petsc-dev] [petsc-users] MATNEST with shell matrices
by Barry Smith 21 Feb '13

21 Feb '13

The way I view changing sizes inside a (non)linear solver is that one does an approximate (non)linear solver, decides to adapt the mesh, then does another approximate (non)linear solver etc. So even though formally they may not be stopping the SNESSolve() and restarting one can think of it that way. Of course inside a Krylov linear solver this is nonsense because the inner produces cannot be done between old and new but with plain old linear multigrid it is ok. Barry On Feb 21, 2013, at 7:44 PM, Jed Brown <jedbrown(a)mcs.anl.gov> wrote: > On Thu, Feb 21, 2013 at 7:28 PM, Boyce Griffith <griffith(a)cims.nyu.edu> wrote: > > I would very happily use an "undefined" vector size. (I will also happily change the bogus local sizes from 0 to 1, and the global sizes accordingly. :-) > > The PETSc solvers that we use in the code should be oblivious to the size of the vector. I've actually never been all that happy with size "0", but it was in some inherited code, and at least so far, it has never caused any problems over the 10+ years it has been in use. (If ever did start causing problems, I certainly would not come complaining to petsc-users or petsc-dev. It isn't like I do not realize that "0" is a potentially problematic value here.) > > I actually have always hoped that setting size to "0" would help smoke out any places where something actually did use the vector size. > > Okay, let's move the design discussion to petsc-dev. > > I want to understand the extent to your adaptivity and how your Vecs "behave". In which places do you (mathematically) change the dimension of a vector? > > If I start with Vecs X and Y and I write > > VecCopy(X,Y); > YourSolver(X); /* solve F(X) = 0 with adaptivity */ > > can I then compare the result like this? > > VecAXPY(Y,-1.,X); > > Do vector-space properties of Y (e.g., norm) change when I call YourSolver(X). (Some codes keep a global view of all vectors, and when they adapt, they regrid all vectors. But regridding doesn't preserve things like norms, so you end up with tangled semantics.)

1 0

Re: [petsc-dev] [petsc-users] MATNEST with shell matrices
by Jed Brown 21 Feb '13

21 Feb '13

On Thu, Feb 21, 2013 at 7:28 PM, Boyce Griffith <griffith(a)cims.nyu.edu>wrote: > > I would very happily use an "undefined" vector size. (I will also happily > change the bogus local sizes from 0 to 1, and the global sizes accordingly. > :-) > > The PETSc solvers that we use in the code should be oblivious to the size > of the vector. I've actually never been all that happy with size "0", but > it was in some inherited code, and at least so far, it has never caused any > problems over the 10+ years it has been in use. (If ever did start causing > problems, I certainly would not come complaining to petsc-users or > petsc-dev. It isn't like I do not realize that "0" is a potentially > problematic value here.) > > I actually have always hoped that setting size to "0" would help smoke out > any places where something actually did use the vector size. > Okay, let's move the design discussion to petsc-dev. I want to understand the extent to your adaptivity and how your Vecs "behave". In which places do you (mathematically) change the dimension of a vector? If I start with Vecs X and Y and I write VecCopy(X,Y); YourSolver(X); /* solve F(X) = 0 with adaptivity */ can I then compare the result like this? VecAXPY(Y,-1.,X); Do vector-space properties of Y (e.g., norm) change when I call YourSolver(X). (Some codes keep a global view of all vectors, and when they adapt, they regrid all vectors. But regridding doesn't preserve things like norms, so you end up with tangled semantics.)

1 0

Re: [petsc-dev] Alternatives to cholmod
by Dave Nystrom 21 Feb '13

21 Feb '13

Jed Brown writes: > On Thu, Feb 21, 2013 at 4:47 PM, Dave Nystrom <dnystrom1(a)comcast.net> wrote: > > > I've been using cholmod from SuiteSparse for over a year now to solve one > > of > > my linear systems that is difficult to solve with an iterative method. Are > > there other sparse direct solvers, such as clique, that I should try > > sometime? My linear system is SPD and I am solving it on a single node. > > My > > code has not yet been parallelized using MPI - it is only a 2D code. > > > > Clique is a distributed memory package, though of course it can be used in > serial. You can also try MUMPS. But Cholmod should be a good package for > this scenario. Are you having problems with it or just looking for other > (likely slower) packages to spend time installing? Cholmod is working fine for me. I have tried other packages in the past such as MUMPS and SuperLU and found them to be slower. Actually a lot slower - like 2x to 4x slower. I was just wondering if there were newer packages that would be worth trying i.e. that might have even better performance than cholmod. Sounds like not.

1 0

Re: [petsc-dev] Alternatives to cholmod
by Jed Brown 21 Feb '13

21 Feb '13

On Thu, Feb 21, 2013 at 4:47 PM, Dave Nystrom <dnystrom1(a)comcast.net> wrote: > I've been using cholmod from SuiteSparse for over a year now to solve one > of > my linear systems that is difficult to solve with an iterative method. Are > there other sparse direct solvers, such as clique, that I should try > sometime? My linear system is SPD and I am solving it on a single node. > My > code has not yet been parallelized using MPI - it is only a 2D code. > Clique is a distributed memory package, though of course it can be used in serial. You can also try MUMPS. But Cholmod should be a good package for this scenario. Are you having problems with it or just looking for other (likely slower) packages to spend time installing?

1 0

Alternatives to cholmod
by Dave Nystrom 21 Feb '13

21 Feb '13

I've been using cholmod from SuiteSparse for over a year now to solve one of my linear systems that is difficult to solve with an iterative method. Are there other sparse direct solvers, such as clique, that I should try sometime? My linear system is SPD and I am solving it on a single node. My code has not yet been parallelized using MPI - it is only a 2D code. Thanks, Dave

1 0

Re: [petsc-dev] Need mechanism to set LocalToGlobal mapping in a DMShell
by Barry Smith 21 Feb '13

21 Feb '13

On Feb 21, 2013, at 3:31 PM, Richard Tran Mills <rtm(a)eecs.utk.edu> wrote: > I will add DMShellSetGlobalToLocal (and DMShellSetLocalToGlobal). > > One other thing: We also need to be do LocalToLocal operations. For the structured grid case, we call DMDALocalToLocalBegin/End, and for the unstructured grid case we construct our own VecScatter context essentially just how it is done in daltol.c for DMDA. Is there a good reason that there is no DMLocalToLocalBegin/End? I don't think so. Just laziness in not moving them over when DM was created from DA. You can add them. Barry > > --Richard > > On 2/21/13 2:42 PM, Barry Smith wrote: >> On Feb 21, 2013, at 1:39 PM, Jed Brown <jedbrown(a)mcs.anl.gov> wrote: >> >>> On Thu, Feb 21, 2013 at 1:31 PM, Barry Smith <bsmith(a)mcs.anl.gov> wrote: >>> I don't understand. There is a DMGlobalToLocalBegin/End() which includes >>> >>> } else { >>> ierr = (*dm->ops->globaltolocalbegin)(dm,g,mode == INSERT_ALL_VALUES ? INSERT_VALUES : (mode == ADD_ALL_VALUES ? ADD_VALUES : mode),l);CHKERRQ(ierr); >>> } >>> >>> For Shell, Richard should be able to provide his own begin/end functions. Which in his case he is already calling the VecScatterBegin/End() directly in his code so he should just be able to keep the scatter context he is already creating and use it in shell. No reason to mess with sections or SF, he already has the scatter and just needs to get it into the DMShell and shell should have it as a matter of course. Just because everyone was to lazy to add DMSetGlobalToLocalBegin/End() doesn't mean it should exist. Rich can add it himself trivially and then use (even from Fortran). >>> >>> >>> Yes, if he has VecScatters already that he wants to reuse, this is a fine option. I proposed this interface earlier in this thread. >>> >>> DMShellSetGlobalToLocal(DM dm,PetscErrorCode (*begin)(DM,Vec,InsertMode, >>> Vec),PetscErrorCode (*end)(DM,Vec,InsertMode,Vec)); >> Ok, Richard should add this and then end of conversation. >> >> Barry >> > > > -- > Richard Tran Mills, Ph.D. > Computational Earth Scientist | Joint Assistant Professor > Hydrogeochemical Dynamics Team | EECS and Earth & Planetary Sciences > Oak Ridge National Laboratory | University of Tennessee, Knoxville > E-mail: rmills(a)ornl.gov V: 865-241-3198 http://climate.ornl.gov/~rmills >

1 0

Re: [petsc-dev] Need mechanism to set LocalToGlobal mapping in a DMShell
by Richard Tran Mills 21 Feb '13

21 Feb '13

I will add DMShellSetGlobalToLocal (and DMShellSetLocalToGlobal). One other thing: We also need to be do LocalToLocal operations. For the structured grid case, we call DMDALocalToLocalBegin/End, and for the unstructured grid case we construct our own VecScatter context essentially just how it is done in daltol.c for DMDA. Is there a good reason that there is no DMLocalToLocalBegin/End? --Richard On 2/21/13 2:42 PM, Barry Smith wrote: > On Feb 21, 2013, at 1:39 PM, Jed Brown <jedbrown(a)mcs.anl.gov> wrote: > >> On Thu, Feb 21, 2013 at 1:31 PM, Barry Smith <bsmith(a)mcs.anl.gov> wrote: >> I don't understand. There is a DMGlobalToLocalBegin/End() which includes >> >> } else { >> ierr = (*dm->ops->globaltolocalbegin)(dm,g,mode == INSERT_ALL_VALUES ? INSERT_VALUES : (mode == ADD_ALL_VALUES ? ADD_VALUES : mode),l);CHKERRQ(ierr); >> } >> >> For Shell, Richard should be able to provide his own begin/end functions. Which in his case he is already calling the VecScatterBegin/End() directly in his code so he should just be able to keep the scatter context he is already creating and use it in shell. No reason to mess with sections or SF, he already has the scatter and just needs to get it into the DMShell and shell should have it as a matter of course. Just because everyone was to lazy to add DMSetGlobalToLocalBegin/End() doesn't mean it should exist. Rich can add it himself trivially and then use (even from Fortran). >> >> >> Yes, if he has VecScatters already that he wants to reuse, this is a fine option. I proposed this interface earlier in this thread. >> >> DMShellSetGlobalToLocal(DM dm,PetscErrorCode (*begin)(DM,Vec,InsertMode, >> Vec),PetscErrorCode (*end)(DM,Vec,InsertMode,Vec)); > Ok, Richard should add this and then end of conversation. > > Barry > -- Richard Tran Mills, Ph.D. Computational Earth Scientist | Joint Assistant Professor Hydrogeochemical Dynamics Team | EECS and Earth & Planetary Sciences Oak Ridge National Laboratory | University of Tennessee, Knoxville E-mail: rmills(a)ornl.gov V: 865-241-3198 http://climate.ornl.gov/~rmills

1 0

Re: [petsc-dev] Need mechanism to set LocalToGlobal mapping in a DMShell
by Jed Brown 21 Feb '13

21 Feb '13

On Thu, Feb 21, 2013 at 2:02 PM, Barry Smith <bsmith(a)mcs.anl.gov> wrote: > The issue is not allowing or even often using SF. The issue is that > DMGlobalToLocal() already has the flexibility to use a globaltolocal > function pointer. And I object to hiding that or pretending it doesn't > exist and saying one HAS to use SF especially if one already has code that > does exactly what is needed some other way. > Sure, if he has the code, then by all means, he should have DM call his code. But if that code was either not already written or if he could get better reuse internally by using PetscSF, it may be a fine choice.

1 0

Re: [petsc-dev] Need mechanism to set LocalToGlobal mapping in a DMShell
by Barry Smith 21 Feb '13

21 Feb '13

Jed, The issue is not allowing or even often using SF. The issue is that DMGlobalToLocal() already has the flexibility to use a globaltolocal function pointer. And I object to hiding that or pretending it doesn't exist and saying one HAS to use SF especially if one already has code that does exactly what is needed some other way. Barry On Feb 21, 2013, at 1:45 PM, Jed Brown <jedbrown(a)mcs.anl.gov> wrote: > > On Thu, Feb 21, 2013 at 1:23 PM, Jed Brown <jedbrown(a)mcs.anl.gov> wrote: > On Thu, Feb 21, 2013 at 1:10 PM, Matthew Knepley <knepley(a)gmail.com> wrote: > Yes, he needs the local to global scatters. Rich, I think the easiest way to do this now may be > to define your data layout as a PetscSection (these are all local) using DMSetDefaultSection(), > and record who owns which point using a PetscSF with DMSetPointSF(). > > Using this information, the DM will automatically construct a scatter between local and global, > and the CreateLocal/GlobalVector() functions will also work right. This is what its is designed for. > > Does this make sense? > > > Rich, this will work fine if you don't mind having PETSc do the communication. It will be done using PetscSFBcastBegin/End, which should have similar performance to VecScatterBegin/End. (I should do a performance test of that.) If you want to use your own internal communication routines, then we need to add the DMShell functions to get your functions called. > > It occurs to me that the description above might be pretty confusing. The idea is that we start with your local space and lump it together into (possibly variable-size) chunks, usually one per topological point in the mesh. Matt would describe this chunking with a PetscSection where the chunks are indexed by "point" (an integer). > > The next component is that we have to describe how the points fit together, so we make a PetscSF where the "roots" are the global points, referred to by (rank, offset) pairs or global indices, and the "leaves" are the local space. You use PetscSFSetGraph() to create this, which Matt calls the "Point SF". > > If you tell the DM about these two things by using DMSetDefaultSection() and DMSetPointSF(), then DM will create the SF to communicate the Vec data (using simple composition). You could also just use DMSetDefaultSF() directly, in which case you don't ever need to touch a PetscSection. > > > The alternative to use VecScatter is also fine, but sometimes it is easier to use PetscSF. The Point SF, in particular, is very useful for handling integer data in the setup phase. Its API is richer than VecScatter, in addition to supporting more data types.

1 0