November 2011 - petsc-dev - mailman3.cels.anl.gov

Re: [petsc-dev] [petsc-users] Any suggestion for this kinds of matrix?
by Mark F. Adams 11 Nov '11

11 Nov '11

Ok, I'll listend to scaling arguments. I'm a firm believer in a solver being invariant to scaling/units. First, "my" matrices (actually XGC1 matrices from one of my projects and I don't even know exactly what they do) put 1.0 on the diagonal for BCs and god knows how the thing is scaled (1e13 apparently). But this does not matter because the RHS and initial guess are 0.0 so the BCs have been completely removed from the algebra, even if they are still in the data structures. As I recall the preconditioned residual was confusing because since these are Laplacian matrices with a scale of 1e13, as you found, the residual dropped like 10 orders of magnitude in the first iteration, which was pretty confusing. So I don't see these XGC1 problems as being arguments for preconditioned residual, in fact they argue against it, right? But if they played games like zeroing rows and keeping columns and putting non-homogenous Dirichlet BCs on the RHS, or penalty BCs then I could see this being a problem. Anyway, I respect that you have experience with tons of crazy crap that users do and set the defaults accordingly. Mark On Nov 10, 2011, at 9:35 PM, Jed Brown wrote: > Mark, YOU obtained a matrix with poorly scaled boundary conditions (by a cool factor of 1e13). Some people use penalties which is worse. The preconditioned norm fixes the scaling. We used -ksp_diagonal_scale to actually modify the matrix entries, but that is a bit fragile/restrictive. > > I think we would have just as much confusion on petsc-maint/users (from the residual dropping abruptly on the first iteration) if we made unpreconditioned the default. > > On Nov 10, 2011 7:50 PM, "Barry Smith" <bsmith(a)mcs.anl.gov> wrote: > > On Nov 10, 2011, at 5:12 PM, Mark F. Adams wrote: > > > > >> > > > > Would this problem also be fixed with -ksp_norm_type unpreconditioned ??? > > > > In that case you could make this the default for debug and not debug, which is cleaner (I've never figured out why you even offer preconditioned norm as an option, much less the default) > > > > Someday I'll have to write a paper showing why :-) > >

1 0

Re: [petsc-dev] ugly shit
by Satish Balay 11 Nov '11

11 Nov '11

On Fri, 11 Nov 2011, Kai Germaschewski wrote: > One was NASA's CCMC (community coordinated modeling center) running > mvapich, the other, IIRC, was a local cluster with mpich-gm (Myrinet). I guess sysadmins on some of the older machines that are chugging along are reluctant to upgrade. [If its wokred fine so far why bother changing? And change usually means tons of user e-mail - extra effor on a machine that can potentially be replaced soon..] However I believe - in these cases - the user can install a newer MPI with same functionality [mvapich or mpich2-gm] - and be able to use such clusters. Should we rely on this approach? Its not clear to me if there are older non-cluster machines still in use - with vendor MPI thats not easy for the user to circumvent with his/her own newer MPI install.. Satish

1 0

Re: [petsc-dev] ugly shit
by Kai Germaschewski 11 Nov '11

11 Nov '11

On Thu, Nov 10, 2011 at 10:56 PM, Jed Brown <jedbrown(a)mcs.anl.gov> wrote: > On Thu, Nov 10, 2011 at 21:14, Kai Germaschewski < > kai.germaschewski(a)unh.edu> wrote: > >> Just a (kinda arbitrary) data point: I introduced MPI_IN_PLACE in my code >> a year or so ago (not realizing that it requires MPI-2.0 at the time) and >> within a year, I hit two cases of machines that were still running MPI 1.x >> -- and the code doesn't get to run in all that many places in the first >> place. >> > > What sort of configurations were these? > One was NASA's CCMC (community coordinated modeling center) running mvapich, the other, IIRC, was a local cluster with mpich-gm (Myrinet). I agree, this is mostly cosmetic, but there are more MPI-2 features that > would also be useful. There are places where having MPI 1-sided would > greatly simplify, for example. > If there are other useful features of MPI2 you want to use, that seems like a much better reason. --Kai -- Kai Germaschewski Assistant Professor, Dept of Physics / Space Science Center University of New Hampshire, Durham, NH 03824 office: Morse Hall 245E phone: +1-603-862-2912 fax: +1-603-862-2771

1 0

Re: [petsc-dev] TS monitoring at fixed times?
by Jed Brown 11 Nov '11

11 Nov '11

On Sun, Oct 23, 2011 at 15:20, Barry Smith <bsmith(a)mcs.anl.gov> wrote: > The simplest could be simply -ts_monitor_dt .1 and then there could be a > version that takes an array of times. I think we want to have each monitor run with a different set of times. This could make an API like: TSMonitorSet(TS ts,PetscInt ntimes,PetscReal *times,PetscErrorCode (*monitor)(TS,PetscInt,PetscReal,Vec,void*),void *mctx,PetscErrorCode (*mdestroy)(void**)); where ntimes=0 implies every step. But there are lots of other ways the user might want to specify: 1. every N steps, whatever the size 2. at regular intervals, but simulation will do many millions of steps, so don't want to name all the times up-front. 3. user-defined, possibly based on the solution (e.g. write the state at the beginning of the step when an eigenvalue of Jacobian crosses imaginary axis during the step). What's wrong with the monitor making all these decisions? I would prefer that, but it means that every kind of monitor needs its own implementation of the interpolated time interval feature.

1 0

Re: [petsc-dev] ugly shit
by Jed Brown 10 Nov '11

10 Nov '11

On Thu, Nov 10, 2011 at 21:14, Kai Germaschewski <kai.germaschewski(a)unh.edu>wrote: > Just a (kinda arbitrary) data point: I introduced MPI_IN_PLACE in my code > a year or so ago (not realizing that it requires MPI-2.0 at the time) and > within a year, I hit two cases of machines that were still running MPI 1.x > -- and the code doesn't get to run in all that many places in the first > place. > What sort of configurations were these? > > Having said that, I think it's generally a good idea to not keep backwards > compatibility cruft around forever. OTOH, requiring MPI2 just for > MPI_IN_PLACE isn't a real convincing case, since it's so easy to do the > same with MPI1. > I agree, this is mostly cosmetic, but there are more MPI-2 features that would also be useful. There are places where having MPI 1-sided would greatly simplify, for example.

1 0

Re: [petsc-dev] ugly shit
by Kai Germaschewski 10 Nov '11

10 Nov '11

On Thu, Nov 10, 2011 at 8:43 PM, Jed Brown <jedbrown(a)mcs.anl.gov> wrote: How do we find out who needs pre-MPI-2.0? Just a (kinda arbitrary) data point: I introduced MPI_IN_PLACE in my code a year or so ago (not realizing that it requires MPI-2.0 at the time) and within a year, I hit two cases of machines that were still running MPI 1.x -- and the code doesn't get to run in all that many places in the first place. Having said that, I think it's generally a good idea to not keep backwards compatibility cruft around forever. OTOH, requiring MPI2 just for MPI_IN_PLACE isn't a real convincing case, since it's so easy to do the same with MPI1. --Kai -- Kai Germaschewski Assistant Professor, Dept of Physics / Space Science Center University of New Hampshire, Durham, NH 03824 office: Morse Hall 245E phone: +1-603-862-2912 fax: +1-603-862-2771

1 0

Re: [petsc-dev] XXXGetContext() in Fortran
by Barry Smith 10 Nov '11

10 Nov '11

On Nov 10, 2011, at 4:46 PM, Jed Brown wrote: > On Wed, Nov 9, 2011 at 12:52, Barry Smith <bsmith(a)mcs.anl.gov> wrote: > So what both Satish's and Ethan's email are saying is so long as you tell the Fortran function the interface that shows the type being returned you can "trick" the Fortran into thinking that data type is just coming from another Fortran routine. > > I think this should resolve your issues and everything should be straightforward (no need for transfer function). Please let us know if this is problematic. > > Thanks everyone. This works, and in hindsight, I should have looked at snes/examples/tutorials/ex5f90.F because it does this with SNESGetApplicationContext. > > The unfortunate thing is that it appears that the user needs to write a custom interface for each such function in order to get the compiler to pass the pointer in a useful way. Is there a way for us to put some interface definitions into the library? An (ugly) alternative would be to provide some CPP macros that the user can use to get interface definitions without needing to hand-type it (they would pass in their user-defined type). I think it is fine that the user writes the interface themselves and I think trying to have ways to "help" them won't help them. So long as we have examples to show how to do this and they are linked (automatically) from the manual pages I think that is the right approach. > > How can we manage multiple contexts with different types in the same file? It looks like perhaps I can put the interface definitions in different modules, which I can import at subroutine granularity. If I want to get the context out of two MatShells inside the same function, I would have to write helper functions so that they can import different interface definitions via different modules? There is a truly perverse, but relatively easy way to handle this. Fortran allows having several interfaces that map a single function name to several actual function names. This is done, for example, in include/finclude/petscdmda.h90. So basically we would need to provide several C functions for each operation. For example SNESGetApplicationContext1(), SNESGetApplicationContext2() that have identical source code. Then the user writes one interface that refers to both of these with different derived types as the arguments. Barry Note we would also provide the usual SNESGetApplicationContext() C stub for use in the normal case of only one needed.

1 0

Re: [petsc-dev] [petsc-users] Any suggestion for this kinds of matrix?
by Jed Brown 10 Nov '11

10 Nov '11

Mark, YOU obtained a matrix with poorly scaled boundary conditions (by a cool factor of 1e13). Some people use penalties which is worse. The preconditioned norm fixes the scaling. We used -ksp_diagonal_scale to actually modify the matrix entries, but that is a bit fragile/restrictive. I think we would have just as much confusion on petsc-maint/users (from the residual dropping abruptly on the first iteration) if we made unpreconditioned the default. On Nov 10, 2011 7:50 PM, "Barry Smith" <bsmith(a)mcs.anl.gov> wrote: > > On Nov 10, 2011, at 5:12 PM, Mark F. Adams wrote: > > > > >> > > > > Would this problem also be fixed with -ksp_norm_type unpreconditioned ??? > > > > In that case you could make this the default for debug and not debug, > which is cleaner (I've never figured out why you even offer preconditioned > norm as an option, much less the default) > > > > Someday I'll have to write a paper showing why :-) > > >

1 0

Re: [petsc-dev] [petsc-users] Any suggestion for this kinds of matrix?
by Barry Smith 10 Nov '11

10 Nov '11

On Nov 10, 2011, at 5:12 PM, Mark F. Adams wrote: > >> > > Would this problem also be fixed with -ksp_norm_type unpreconditioned ??? > > In that case you could make this the default for debug and not debug, which is cleaner (I've never figured out why you even offer preconditioned norm as an option, much less the default) > Someday I'll have to write a paper showing why :-)

1 0

Re: [petsc-dev] ugly shit
by Jed Brown 10 Nov '11

10 Nov '11

True. Easy, but tedious. But so are #ifdefs. How do we find out who needs pre-MPI-2.0? On Nov 10, 2011 7:38 PM, "Barry Smith" <bsmith(a)mcs.anl.gov> wrote: > > On Nov 10, 2011, at 4:10 PM, Jed Brown wrote: > > > On Thu, Nov 10, 2011 at 16:05, Barry Smith <bsmith(a)mcs.anl.gov> wrote: > > You could have ./configure check for MPI_IN_PLACE and generate an > error if it doesn't exist. But do we want to say PETSc requires MPI 2.0 > (or at least parts of it)???? Since we already have support for not having > MPI_IN_PLACE I'd only like to remove that extra support if we knew that all > (sane) users had access to MPI_IN_PLACE. Do we know this? > > > > If we found an implementation that doesn't have it, we could define the > MPI_IN_PLACE value and have our own wrapper that did the right thing > (allocated the work array). > > I don't see this being simple. We would need to implement many MPI > functions that take MPI_IN_PLACE. > > > > > > > > > > In general, can we rely on any MPI-2.0 features that have been > implemented in mpiuni and remove the macro checking? > > > > I don't have a clue what parts of MPI-2.0 that MPIUni does or does > not support :-(. I'm afraid we'll need to find out one operation at a > time. It definitely doesn't have the one-sided stuff, can/should that be > added? I don't see a big upside in putting the one-sided stuff into MPI uni. > > > > MPI one-sided is normally used to talk to remote processes, much like > MPI_Send, which we currently define to produce an error if called. I think > it would make sense to do the same thing with the one-sided stuff. > >

1 0