February 2012 - petsc-dev - mailman3.cels.anl.gov

Re: [petsc-dev] FAS with TS
by Jed Brown 07 Feb '12

07 Feb '12

On Wed, Feb 8, 2012 at 04:19, Peter Brune <prbrune(a)gmail.com> wrote: > > > On Tue, Feb 7, 2012 at 6:40 PM, Jed Brown <jedbrown(a)mcs.anl.gov> wrote: > >> Suppose I want to solve a finite time step problem using FAS. The time >> step will be defined something like >> >> static PetscErrorCode SNESTSFormFunction_ARKIMEX(SNES snes,Vec X,Vec F,TS >> ts) >> { >> TS_ARKIMEX *ark = (TS_ARKIMEX*)ts->data; >> PetscErrorCode ierr; >> >> PetscFunctionBegin; >> ierr = >> VecAXPBYPCZ(ark->Ydot,-ark->shift,ark->shift,0,ark->Z,X);CHKERRQ(ierr); /* >> Ydot = shift*(X-Z) */ >> ierr = >> TSComputeIFunction(ts,ark->stage_time,X,ark->Ydot,F,ark->imex);CHKERRQ(ierr); >> PetscFunctionReturn(0); >> } >> >> >> where ark->Z was determined from previous stages and ark->Ydot is a work >> vector. So FAS creates some hierarchy, but it isn't very forthcoming about >> showing the hierarchy to the user. But I need a way to >> >> 1. Restrict ark->Z to the coarse level and obtain the work vector >> ark->Ydot somewhere that I can reuse it. The user might also have to >> restrict "auxiliary" variables to coarse grids. It's conceivable that they >> would want to do this in a different way from the restriction/prolongation >> for the solution variables. For example, if you were using upwinded R and P >> for a hyperbolic problem, but you needed to restrict coefficients (like >> bathymetry). Or they might want to sample differently out of some file. So >> I think we need an auxiliary transfer callback that is at least called >> every top-level SNESSolve() and optionally also called in every transfer >> (because it could be homogenizing coefficient structure that is a nonlinear >> function of state). >> >> > The restriction (injection, etc) might be one step too hard at this > point. We have to define the problem at each level in FAS and grid > sequencing so we might as well make it easy. Is it possible to have the > callback for the SNES point to things that have to be prolonged or > restricted? This could be called both before FAS and before grid > sequencing to get the coefficients up to the next level. We could provide > some sort of default that goes through a number of user-provided vectors on > a level given vectors on another level, and inject them. A user with > user-created multilevel coefficients could provide their own similar > callbacks. > So coefficients usually won't live on the same DM as the solution. This case with TS is easier in that regard, but it can't be the sole driver of design. There needs to be a general function to perform level setup. /* called by SNESSetUp_FAS() before restricting a nonlinear solution to a coarser level (but usually only used the first time unless this is doing solution-dependent homogenization) */ typedef PetscErrorCode (*SNESFASRestrictHook)(SNES fine,Vec Xfine,SNES coarse); PetscErrorCode SNESFASSetRestrictHook(SNES fine,SNESFASRestrictHook hook,void *ctx); /* called in SNESSolve() each time a new solution is prolongated */ typedef PetscErrorCode (*SNESGridSequenceInterpolateHook)(SNES coarse,Vec Xcoarse,SNES fine); PetscErrorCode SNESGridSequenceSetInterpolateHook(SNES coarse,SNESGridSequenceInterpolateHook hook,void *ctx); I think we also need to add state to DMCreateInterpolation() so that we can make it nonlinear when appropriate. > >> 2. Put the DM from this SNES into the TS that calls the user's IFunction. >> The easiest way is to have TSPushDM(TS,DM) that just jams it in there, >> caching the old DM (so the user's call to TSGetDM() works correctly), and >> TSPopDM(TS). This is ugly and is scary if the user does something weird >> like TSGetSolution() (returning the state at the beginning of the step, not >> the current value at the stage). This is something that doesn't make >> semantic sense except maybe if they have some weird diagnostics, but >> TSGetIJacobian() might be used for caching, and I'm scared of the semantics >> it would involve to support this sort of caching. >> >> > This option seems like it would be scary complex. I don't like it. > I think it's likely simpler than the others, at least as a first implementation. I'm not wild about it either, but I'm not sure the alternatives are better. > > >> The alternative is to make different TSs for each level and somehow >> locate the correct TS (maybe cache it in the SNES hierarchy). I think this >> could be quite confusing to restrict all the members to the coarse grid. >> But having a multi-level TS object may eventually be sensible because >> multilevel time integration schemes exist. Spectral deferred correction is >> one example where coarse grids (in space and time) can be used to >> accelerate convergence to a high-order solution. But these somewhat >> naturally expose additional parallelism in time (usually more >> communication, but also more concurrency), so it likely needs a more >> complete solution that includes ways to instantiate entire models on >> sub-communicators (we'll also want an interface of this type for UQ). >> >> > I'm not sure this would be harmonious with FAS given the coarse problem is > redefined with a defect from the fine problem (making a FAS cycle on a > actual fine solution do nothing) at each stage. As I understand it, we'd > need a "real" time derivative rather than a derivative of the defect > problem. Therefore, we would be stuck solving at each level and using each > level for FAS independently. > Multilevel time integration would involve having a fully functional TS on each level, each of which may or may not use FAS. A conventional time integration scheme solved by FAS need these things with type TS to pass to the user, that implement TSGetDM() and perhaps some similar functionality. They also need the affine vectors to define the residual. Maybe it would be better to make a little container that was basically just a struct, but masqueraded as a TS. > However, having a multilevel time integration scheme outside of FAS > accessible to FAS to accelerate solution might be an interesting project; > are there references? Would this look like grid sequencing + FAS with the > grid sequence solutions moved forward in time? > Hmm, I haven't seen anything really like that, but note that the stability threshold for explicit methods is larger on coarse levels.

1 0

Re: [petsc-dev] petsc-dev on bitbucket
by Dmitry Karpeev 07 Feb '12

07 Feb '12

Rich Martineau volunteers to knock down any barriers (including locked doors) between PETSc and the wider world. Apparently, he's still not past his prime. Dmitry. On Tue, Feb 7, 2012 at 10:23 PM, Barry Smith <bsmith(a)mcs.anl.gov> wrote: > > On Feb 7, 2012, at 9:09 PM, Sean Farley wrote: > > > I'm sure Jed (or Matt in his prime) could have run over to IIT and > restarted the machine in less time than this :-) > > > > Sure, and like everybody else they would have had to wait outside until > they had keys :-) > > Those guys are very resourceful; I cannot image a simple locked door > would be an issue for them. > > Barry > > Besides who the heck set up the machine so it cannot be started > remotely? Should have used an Apple machine :-) > > >

1 0

Re: [petsc-dev] petsc-dev on bitbucket
by Barry Smith 07 Feb '12

07 Feb '12

On Feb 7, 2012, at 9:09 PM, Sean Farley wrote: > I'm sure Jed (or Matt in his prime) could have run over to IIT and restarted the machine in less time than this :-) > > Sure, and like everybody else they would have had to wait outside until they had keys :-) Those guys are very resourceful; I cannot image a simple locked door would be an issue for them. Barry Besides who the heck set up the machine so it cannot be started remotely? Should have used an Apple machine :-)

1 0

Re: [petsc-dev] petsc-dev on bitbucket
by Hong Zhang 07 Feb '12

07 Feb '12

The lab opens almost all the time - graduate students stay in the lab 7x24 ... Hong On Tue, Feb 7, 2012 at 9:09 PM, Sean Farley <sean(a)mcs.anl.gov> wrote: > I'm sure Jed (or Matt in his prime) could have run over to IIT and >> restarted the machine in less time than this :-) > > > Sure, and like everybody else they would have had to wait outside until > they had keys :-) >

1 0

Re: [petsc-dev] petsc-dev on bitbucket
by Sean Farley 07 Feb '12

07 Feb '12

> > I'm sure Jed (or Matt in his prime) could have run over to IIT and > restarted the machine in less time than this :-) Sure, and like everybody else they would have had to wait outside until they had keys :-)

1 0

Re: [petsc-dev] petsc-dev on bitbucket
by Barry Smith 07 Feb '12

07 Feb '12

On Feb 7, 2012, at 8:14 PM, Sean Farley wrote: > "Yoo, i've switched petsc-dev over to bitbucket since petsc.cs.itt.edu is down pull/push from https://bitbucket.org/petsc/petsc-dev > > Sean" - Barry impersonating me I'm sure Jed (or Matt in his prime) could have run over to IIT and restarted the machine in less time than this :-) Thanks Barry > > Since petsc.cs is down, I don't have everyone's public SSH key. For now, if you need push access, just sign up for a free bitbucket account (can also use OpenID) and reply to this email with your bitbucket username. Barry, Matt, Jed, and Peter have write access already. Hong (hzhang), Dmitry (karpeev), and Lois (curfman) should have write access as long as the username in the parenthesis is your correct bitbucket name. > > Also, BuildSystem is here: > > https://bitbucket.org/petsc/buildsystem > > where the same people as above have write access. > > P.S. - To change your current default push url in mercurial, just edit $PETSC_DIR/.hg/hgrc > > [paths] > default = https://bitbucket.org/petsc/petsc-dev > > or if you uploaded your public SSH key to bitbucket already > > [paths] > default = ssh://[email protected]/petsc/petsc-dev > > More info here: > http://confluence.atlassian.com/display/BITBUCKET/Using+the+SSH+protocol+wi… > > P.P.S - While we're at it, here is a nice shortcut in mercurial. Place this in your ~/.hgrc > > [extensions] > schemes = > > [schemes] > petsc = ssh://[email protected]/petsc > > You can now clone like so: `hg clone petsc://petsc-dev` or `hg clone petsc://buildsystem`

1 0

Re: [petsc-dev] petsc-dev on bitbucket
by Matthew Knepley 07 Feb '12

07 Feb '12

On Tue, Feb 7, 2012 at 8:14 PM, Sean Farley <sean(a)mcs.anl.gov> wrote: > "Yoo, i've switched petsc-dev over to bitbucket since petsc.cs.itt.edu is > down pull/push from https://bitbucket.org/petsc/petsc-dev > > Sean" - Barry impersonating me > I have pushed to bring it up to date with the latest master. Matt > Since petsc.cs is down, I don't have everyone's public SSH key. For now, > if you need push access, just sign up for a free bitbucket account (can > also use OpenID) and reply to this email with your bitbucket username. > Barry, Matt, Jed, and Peter have write access already. Hong (hzhang), > Dmitry (karpeev), and Lois (curfman) should have write access as long as > the username in the parenthesis is your correct bitbucket name. > > Also, BuildSystem is here: > > https://bitbucket.org/petsc/buildsystem > > where the same people as above have write access. > > P.S. - To change your current default push url in mercurial, just edit > $PETSC_DIR/.hg/hgrc > > [paths] > default = https://bitbucket.org/petsc/petsc-dev > > or if you uploaded your public SSH key to bitbucket already > > [paths] > default = ssh://[email protected]/petsc/petsc-dev > > More info here: > > http://confluence.atlassian.com/display/BITBUCKET/Using+the+SSH+protocol+wi… > > P.P.S - While we're at it, here is a nice shortcut in mercurial. Place > this in your ~/.hgrc > > [extensions] > schemes = > > [schemes] > petsc = ssh://[email protected]/petsc > > You can now clone like so: `hg clone petsc://petsc-dev` or `hg clone > petsc://buildsystem` > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener

1 0

petsc-dev on bitbucket
by Sean Farley 07 Feb '12

07 Feb '12

"Yoo, i've switched petsc-dev over to bitbucket since petsc.cs.itt.edu is down pull/push from https://bitbucket.org/petsc/petsc-dev Sean" - Barry impersonating me Since petsc.cs is down, I don't have everyone's public SSH key. For now, if you need push access, just sign up for a free bitbucket account (can also use OpenID) and reply to this email with your bitbucket username. Barry, Matt, Jed, and Peter have write access already. Hong (hzhang), Dmitry (karpeev), and Lois (curfman) should have write access as long as the username in the parenthesis is your correct bitbucket name. Also, BuildSystem is here: https://bitbucket.org/petsc/buildsystem where the same people as above have write access. P.S. - To change your current default push url in mercurial, just edit $PETSC_DIR/.hg/hgrc [paths] default = https://bitbucket.org/petsc/petsc-dev or if you uploaded your public SSH key to bitbucket already [paths] default = ssh://[email protected]/petsc/petsc-dev More info here: http://confluence.atlassian.com/display/BITBUCKET/Using+the+SSH+protocol+wi… P.P.S - While we're at it, here is a nice shortcut in mercurial. Place this in your ~/.hgrc [extensions] schemes = [schemes] petsc = ssh://[email protected]/petsc You can now clone like so: `hg clone petsc://petsc-dev` or `hg clone petsc://buildsystem`

1 0

Re: [petsc-dev] FAS with TS
by Peter Brune 07 Feb '12

07 Feb '12

On Tue, Feb 7, 2012 at 6:40 PM, Jed Brown <jedbrown(a)mcs.anl.gov> wrote: > Suppose I want to solve a finite time step problem using FAS. The time > step will be defined something like > > static PetscErrorCode SNESTSFormFunction_ARKIMEX(SNES snes,Vec X,Vec F,TS > ts) > { > TS_ARKIMEX *ark = (TS_ARKIMEX*)ts->data; > PetscErrorCode ierr; > > PetscFunctionBegin; > ierr = > VecAXPBYPCZ(ark->Ydot,-ark->shift,ark->shift,0,ark->Z,X);CHKERRQ(ierr); /* > Ydot = shift*(X-Z) */ > ierr = > TSComputeIFunction(ts,ark->stage_time,X,ark->Ydot,F,ark->imex);CHKERRQ(ierr); > PetscFunctionReturn(0); > } > > > where ark->Z was determined from previous stages and ark->Ydot is a work > vector. So FAS creates some hierarchy, but it isn't very forthcoming about > showing the hierarchy to the user. But I need a way to > > 1. Restrict ark->Z to the coarse level and obtain the work vector > ark->Ydot somewhere that I can reuse it. The user might also have to > restrict "auxiliary" variables to coarse grids. It's conceivable that they > would want to do this in a different way from the restriction/prolongation > for the solution variables. For example, if you were using upwinded R and P > for a hyperbolic problem, but you needed to restrict coefficients (like > bathymetry). Or they might want to sample differently out of some file. So > I think we need an auxiliary transfer callback that is at least called > every top-level SNESSolve() and optionally also called in every transfer > (because it could be homogenizing coefficient structure that is a nonlinear > function of state). > > The restriction (injection, etc) might be one step too hard at this point. We have to define the problem at each level in FAS and grid sequencing so we might as well make it easy. Is it possible to have the callback for the SNES point to things that have to be prolonged or restricted? This could be called both before FAS and before grid sequencing to get the coefficients up to the next level. We could provide some sort of default that goes through a number of user-provided vectors on a level given vectors on another level, and inject them. A user with user-created multilevel coefficients could provide their own similar callbacks. > 2. Put the DM from this SNES into the TS that calls the user's IFunction. > The easiest way is to have TSPushDM(TS,DM) that just jams it in there, > caching the old DM (so the user's call to TSGetDM() works correctly), and > TSPopDM(TS). This is ugly and is scary if the user does something weird > like TSGetSolution() (returning the state at the beginning of the step, not > the current value at the stage). This is something that doesn't make > semantic sense except maybe if they have some weird diagnostics, but > TSGetIJacobian() might be used for caching, and I'm scared of the semantics > it would involve to support this sort of caching. > > This option seems like it would be scary complex. I don't like it. > The alternative is to make different TSs for each level and somehow locate > the correct TS (maybe cache it in the SNES hierarchy). I think this could > be quite confusing to restrict all the members to the coarse grid. But > having a multi-level TS object may eventually be sensible because > multilevel time integration schemes exist. Spectral deferred correction is > one example where coarse grids (in space and time) can be used to > accelerate convergence to a high-order solution. But these somewhat > naturally expose additional parallelism in time (usually more > communication, but also more concurrency), so it likely needs a more > complete solution that includes ways to instantiate entire models on > sub-communicators (we'll also want an interface of this type for UQ). > > I'm not sure this would be harmonious with FAS given the coarse problem is redefined with a defect from the fine problem (making a FAS cycle on a actual fine solution do nothing) at each stage. As I understand it, we'd need a "real" time derivative rather than a derivative of the defect problem. Therefore, we would be stuck solving at each level and using each level for FAS independently. However, having a multilevel time integration scheme outside of FAS accessible to FAS to accelerate solution might be an interesting project; are there references? Would this look like grid sequencing + FAS with the grid sequence solutions moved forward in time? - Peter > Preferences or suggestions? >

1 0

FAS with TS
by Jed Brown 07 Feb '12

07 Feb '12

Suppose I want to solve a finite time step problem using FAS. The time step will be defined something like static PetscErrorCode SNESTSFormFunction_ARKIMEX(SNES snes,Vec X,Vec F,TS ts) { TS_ARKIMEX *ark = (TS_ARKIMEX*)ts->data; PetscErrorCode ierr; PetscFunctionBegin; ierr = VecAXPBYPCZ(ark->Ydot,-ark->shift,ark->shift,0,ark->Z,X);CHKERRQ(ierr); /* Ydot = shift*(X-Z) */ ierr = TSComputeIFunction(ts,ark->stage_time,X,ark->Ydot,F,ark->imex);CHKERRQ(ierr); PetscFunctionReturn(0); } where ark->Z was determined from previous stages and ark->Ydot is a work vector. So FAS creates some hierarchy, but it isn't very forthcoming about showing the hierarchy to the user. But I need a way to 1. Restrict ark->Z to the coarse level and obtain the work vector ark->Ydot somewhere that I can reuse it. The user might also have to restrict "auxiliary" variables to coarse grids. It's conceivable that they would want to do this in a different way from the restriction/prolongation for the solution variables. For example, if you were using upwinded R and P for a hyperbolic problem, but you needed to restrict coefficients (like bathymetry). Or they might want to sample differently out of some file. So I think we need an auxiliary transfer callback that is at least called every top-level SNESSolve() and optionally also called in every transfer (because it could be homogenizing coefficient structure that is a nonlinear function of state). 2. Put the DM from this SNES into the TS that calls the user's IFunction. The easiest way is to have TSPushDM(TS,DM) that just jams it in there, caching the old DM (so the user's call to TSGetDM() works correctly), and TSPopDM(TS). This is ugly and is scary if the user does something weird like TSGetSolution() (returning the state at the beginning of the step, not the current value at the stage). This is something that doesn't make semantic sense except maybe if they have some weird diagnostics, but TSGetIJacobian() might be used for caching, and I'm scared of the semantics it would involve to support this sort of caching. The alternative is to make different TSs for each level and somehow locate the correct TS (maybe cache it in the SNES hierarchy). I think this could be quite confusing to restrict all the members to the coarse grid. But having a multi-level TS object may eventually be sensible because multilevel time integration schemes exist. Spectral deferred correction is one example where coarse grids (in space and time) can be used to accelerate convergence to a high-order solution. But these somewhat naturally expose additional parallelism in time (usually more communication, but also more concurrency), so it likely needs a more complete solution that includes ways to instantiate entire models on sub-communicators (we'll also want an interface of this type for UQ). Preferences or suggestions?

1 0