May 2012 - petsc-dev - mailman3.cels.anl.gov

Re: [petsc-dev] [petsc-users] petsc-users Digest, Vol 38, Issue 41
by Barry Smith 06 May '12

06 May '12

You present a case where the one level method should be a direct solve. I don't think this disproves my point that there is NO correct general abstract or practical answer to what it should be. Perhaps this should be a property of PCMG, if in the one level case it is a direct solve or an iterative. Then when you create the PCMG for grid sequencing you call something like PCMGSetOneLevelDefaultDirectSolver() or some similar ugly name. Barry On May 6, 2012, at 5:31 PM, Jed Brown wrote: > On Sat, Feb 18, 2012 at 11:59 AM, Barry Smith <bsmith(a)mcs.anl.gov> wrote: > > -pc_type mg with one level just applies a normal smoother. I've sometimes thought it should do a coarse-level solve instead, but I haven't messed with it. Barry, why doesn't it do a direct solve? > > > 1) Because MG is an accelerator of the basic smoother, MG is not a deccelerator of a direct solver. That is the action of adding a coarser level is suppose to improve the convergence of the solver. > > 2) Because if you used a direct solver and the user switched from one to two levels they would be dismayed at the worsening of the convergence. If the user ran a large problem on one level it would run out of memory. > > 3) I don't think there is really a "correct" abstract or practical answer to which it should be (hence my two snide answers above) I am happy with the current default > > So with -snes_grid_sequence, the first solve should do something reasonable. If we give -pc_type mg, the first solve in the sequence (-mg_levels_0_, not -mg_coarse_) is doing a crappy iterative solve. It's obnoxious to have to specify what to do for both of these separately, even though the problem is of the same size in both cases, so it's usually desirable for the same method to be used in both cases.

1 0

Re: [petsc-dev] Attaching a near null space to an IS
by Barry Smith 06 May '12

06 May '12

I think it belongs in the DM (or the matrix if it was put in the matrix). When you select a subpart of the DM that new DM should contain the near null space for the subpart. The DM associated with the field defining IS is where all properties should go, not the IS itself. Of course, we really have not yet solidified the getting of the new DM for the field yet, maybe after the release. Barry On May 6, 2012, at 6:30 PM, Matthew Knepley wrote: > On Sun, May 6, 2012 at 7:15 PM, Jed Brown <jedbrown(a)mcs.anl.gov> wrote: > Matt introduced this concept, he says the IS is a better place to attach things. > > http://petsc.cs.iit.edu/petsc/petsc-dev/rev/2ad289ac99e0 > > I don't understand why the IS is better (because it's mostly immutable?). I'm worried that putting it there is going to be fragile because the near null space is not a property of an IS at all. > > Its not a property of your matrix either, or you would not need me to tell you. Its a property of the operator. The operator > is defined by the DM. The near null space is actually a property of a suboperator, defined by the DM using a field (we are > not allowing arbitrary divisions). The representation of a field in PETSc is an IS (http://www.mcs.anl.gov/petsc/petsc-dev/docs/manualpages/DM/DMCreateFieldIS.…) so it makes sense to attach field information to the IS. Moreover, it > makes a hell of a lat more sense to attach an auxiliary operator (like L_p) to this IS than to a matrix. > > Furthermore, this scheme is completely workable in a nested context. The user can specify the IS, or pull out the DM IS > and play with it, without a bunch of cumbersome copies hanging around that we do not want and can't destroy. That is what > would happen with persistent submatrices. Lastly, I am running this for PyLith and it works great. > > Matt > > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener

1 0

Re: [petsc-dev] Attaching a near null space to an IS
by Matthew Knepley 06 May '12

06 May '12

On Sun, May 6, 2012 at 7:15 PM, Jed Brown <jedbrown(a)mcs.anl.gov> wrote: > Matt introduced this concept, he says the IS is a better place to attach > things. > > http://petsc.cs.iit.edu/petsc/petsc-dev/rev/2ad289ac99e0 > > I don't understand why the IS is better (because it's mostly immutable?). > I'm worried that putting it there is going to be fragile because the near > null space is not a property of an IS at all. > Its not a property of your matrix either, or you would not need me to tell you. Its a property of the operator. The operator is defined by the DM. The near null space is actually a property of a suboperator, defined by the DM using a field (we are not allowing arbitrary divisions). The representation of a field in PETSc is an IS ( http://www.mcs.anl.gov/petsc/petsc-dev/docs/manualpages/DM/DMCreateFieldIS.…) so it makes sense to attach field information to the IS. Moreover, it makes a hell of a lat more sense to attach an auxiliary operator (like L_p) to this IS than to a matrix. Furthermore, this scheme is completely workable in a nested context. The user can specify the IS, or pull out the DM IS and play with it, without a bunch of cumbersome copies hanging around that we do not want and can't destroy. That is what would happen with persistent submatrices. Lastly, I am running this for PyLith and it works great. Matt -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener

1 0

Re: [petsc-dev] DM RefineLevel and CoarsenLevel
by Matthew Knepley 06 May '12

06 May '12

On Sun, May 6, 2012 at 5:57 PM, Jed Brown <jedbrown(a)mcs.anl.gov> wrote: > Now the next round: > > For semi-coarsening, we used to have stuff like -da_refine_hierarchy_x > 1,1,3 -da_refine_hierarchy_y 2,2,1 -da_refine_hierarchy_z 2,2,1. Two > changes make this harder now: > > 1. You essentially got rid of DMRefineHierarchy (it's not called any > more), so each call to DMRefine and DMCoarsen have to figure out where they > are. > This is a huge mistake. The only way unstructured stuff works is through this interface. That is why I added it Matt > 2. Since the coarse DMs are not reused by PCMG, but instead created again > using DMCoarsen, we have to figure out how to reverse the refinement > process so that the same coarse grids get reconstructed again. > > I added a DMRefineHook so that we can port data the other way and I > modified DMCoarsen_DA and DMRefine_DA to not call DMDACreate{1,2,3}d > because it eagerly calls DMSetFromOptions before we can set the > refinement/coarsen level. Unless someone stops me, I'm also going to add > coarsen_{x,y,z} fields to DM_DA because the refinement ratio may have > nothing to do with the coarsening ratio. > > I have no idea how to expose semi-coarsening through a C API other than to > hold the refinement/coarsening path arrays in each DM_DA so that > refinement/coarsening steps can be retraced. > > On Sun, May 6, 2012 at 1:57 PM, Barry Smith <bsmith(a)mcs.anl.gov> wrote: > >> >> Fine >> >> On May 6, 2012, at 2:44 PM, Jed Brown wrote: >> >> > Should the refinement level be copied over by DMCoarsen (and the >> coarsen level be copied by DMRefine)? >> > >> > It's useful for diagnostics to be able to define a universal level. If >> I use PCMG and -snes_grid_sequence, there is effectively a sequence like >> > >> > DMCreate(,&dm0); // r=0,c=0 >> > DMRefine(dm0,&dmf); // r=1,c=0 >> > DMCoarsen(dmf,&dmc); // r=0,c=1 >> > >> > >> > I would like a way to identify dmc as being on the "same level" as dm0. >> >> > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener

1 0

Attaching a near null space to an IS
by Jed Brown 06 May '12

06 May '12

Matt introduced this concept, he says the IS is a better place to attach things. http://petsc.cs.iit.edu/petsc/petsc-dev/rev/2ad289ac99e0 I don't understand why the IS is better (because it's mostly immutable?). I'm worried that putting it there is going to be fragile because the near null space is not a property of an IS at all.

1 0

Re: [petsc-dev] buildsystem: cusp location
by Jed Brown 06 May '12

06 May '12

On Mon, Apr 16, 2012 at 10:16 AM, Sean Farley <sean(a)mcs.anl.gov> wrote: > Given our --with-package-dir model of checking all kinds of possibilities >> I don't see why that extra include would be required part? Isn't it just >> one possibility? >> >> What is the argument about, just have cusp.py check more possibilities. > > > What's the status on this? I haven't seen a changeset that fixes this yet. > I think this is all that was needed. http://petsc.cs.iit.edu/petsc/BuildSystem/rev/1cca006554d6

1 0

Re: [petsc-dev] [petsc-users] petsc-users Digest, Vol 38, Issue 41
by Jed Brown 06 May '12

06 May '12

On Sat, Feb 18, 2012 at 11:59 AM, Barry Smith <bsmith(a)mcs.anl.gov> wrote: > > -pc_type mg with one level just applies a normal smoother. I've > sometimes thought it should do a coarse-level solve instead, but I > haven't messed with it. Barry, why doesn't it do a direct solve? > > > 1) Because MG is an accelerator of the basic smoother, MG is not a > deccelerator of a direct solver. That is the action of adding a coarser > level is suppose to improve the convergence of the solver. > > 2) Because if you used a direct solver and the user switched from one to > two levels they would be dismayed at the worsening of the convergence. If > the user ran a large problem on one level it would run out of memory. > > 3) I don't think there is really a "correct" abstract or practical answer > to which it should be (hence my two snide answers above) I am happy with > the current default So with -snes_grid_sequence, the first solve should do something reasonable. If we give -pc_type mg, the first solve in the sequence (-mg_levels_0_, not -mg_coarse_) is doing a crappy iterative solve. It's obnoxious to have to specify what to do for both of these separately, even though the problem is of the same size in both cases, so it's usually desirable for the same method to be used in both cases.

1 0

Re: [petsc-dev] DM RefineLevel and CoarsenLevel
by Jed Brown 06 May '12

06 May '12

Now the next round: For semi-coarsening, we used to have stuff like -da_refine_hierarchy_x 1,1,3 -da_refine_hierarchy_y 2,2,1 -da_refine_hierarchy_z 2,2,1. Two changes make this harder now: 1. You essentially got rid of DMRefineHierarchy (it's not called any more), so each call to DMRefine and DMCoarsen have to figure out where they are. 2. Since the coarse DMs are not reused by PCMG, but instead created again using DMCoarsen, we have to figure out how to reverse the refinement process so that the same coarse grids get reconstructed again. I added a DMRefineHook so that we can port data the other way and I modified DMCoarsen_DA and DMRefine_DA to not call DMDACreate{1,2,3}d because it eagerly calls DMSetFromOptions before we can set the refinement/coarsen level. Unless someone stops me, I'm also going to add coarsen_{x,y,z} fields to DM_DA because the refinement ratio may have nothing to do with the coarsening ratio. I have no idea how to expose semi-coarsening through a C API other than to hold the refinement/coarsening path arrays in each DM_DA so that refinement/coarsening steps can be retraced. On Sun, May 6, 2012 at 1:57 PM, Barry Smith <bsmith(a)mcs.anl.gov> wrote: > > Fine > > On May 6, 2012, at 2:44 PM, Jed Brown wrote: > > > Should the refinement level be copied over by DMCoarsen (and the coarsen > level be copied by DMRefine)? > > > > It's useful for diagnostics to be able to define a universal level. If I > use PCMG and -snes_grid_sequence, there is effectively a sequence like > > > > DMCreate(,&dm0); // r=0,c=0 > > DMRefine(dm0,&dmf); // r=1,c=0 > > DMCoarsen(dmf,&dmc); // r=0,c=1 > > > > > > I would like a way to identify dmc as being on the "same level" as dm0. > >

1 0

Re: [petsc-dev] Error during KSPDestroy
by Barry Smith 06 May '12

06 May '12

Alexander, I cannot reproduce this on my mac with 3 different blas/lapack. Could you please run the case below but with --download-f-blas-lapack (you forgot the -f last time)? Send us the valgrind results. This will tell use the exact line number in dlasq3() that is triggering the bad read. Barry On May 6, 2012, at 9:16 AM, Alexander Grayver wrote: > On 06.05.2012 15:34, Matthew Knepley wrote: >> On Sun, May 6, 2012 at 9:24 AM, Alexander Grayver <agrayver(a)gfz-potsdam.de> wrote: >> Hm, valgrind gives a lot of output like that (see full log in previous message): >> >> Can you run this with --download-f-blas-lapack? This sounds much more like an MKL bug. > > I did: > --download-scalapack --download-blacs --download-blas-lapack --with-precision=double --with-scalar-type=complex > > The error is still there. I checked "ldd solveTest", mkl is not used for sure. This is not an MKL problem I guess: > > ==13600== Invalid read of size 8 > ==13600== at 0x58636AF: dlasq3_ (in /usr/local/lib/liblapack.so.3.2.2) > ==13600== by 0x5862C84: dlasq2_ (in /usr/local/lib/liblapack.so.3.2.2) > ==13600== by 0x5861F2C: dlasq1_ (in /usr/local/lib/liblapack.so.3.2.2) > ==13600== by 0x571A479: zbdsqr_ (in /usr/local/lib/liblapack.so.3.2.2) > ==13600== by 0x57466A7: zgesvd_ (in /usr/local/lib/liblapack.so.3.2.2) > ==13600== by 0x694687: KSPComputeExtremeSingularValues_GMRES (gmreig.c:46) > ==13600== by 0x69C62A: KSPComputeExtremeSingularValues (itfunc.c:47) > ==13600== by 0x44E02C: main (solveTest.c:62) > ==13600== Address 0x10826b90 is 16 bytes before a block of size 832 alloc'd > ==13600== at 0x4C25D66: memalign (vg_replace_malloc.c:694) > ==13600== by 0x4B5ACB: PetscMallocAlign (mal.c:30) > ==13600== by 0x686181: KSPSetUp_GMRES (gmres.c:73) > ==13600== by 0x69E6FE: KSPSetUp (itfunc.c:239) > ==13600== by 0x6A090C: KSPSolve (itfunc.c:402) > ==13600== by 0x44E009: main (solveTest.c:61) > > The weird thing is that the it gives correct result, so zgesvd works fine. > > And also running this program with 10 iterations in valgrind doesn't produce error. The low above is with 100 iterations. > Without valgrind the error is always there. > > -- > Regards, > Alexander >

1 0

Re: [petsc-dev] DM RefineLevel and CoarsenLevel
by Barry Smith 06 May '12

06 May '12

Fine On May 6, 2012, at 2:44 PM, Jed Brown wrote: > Should the refinement level be copied over by DMCoarsen (and the coarsen level be copied by DMRefine)? > > It's useful for diagnostics to be able to define a universal level. If I use PCMG and -snes_grid_sequence, there is effectively a sequence like > > DMCreate(,&dm0); // r=0,c=0 > DMRefine(dm0,&dmf); // r=1,c=0 > DMCoarsen(dmf,&dmc); // r=0,c=1 > > > I would like a way to identify dmc as being on the "same level" as dm0.

1 0