April 2011 - petsc-dev - mailman3.cels.anl.gov

Re: [petsc-dev] [petsc-users] FETI-DP
by Jed Brown 20 Apr '11

20 Apr '11

On Wed, Apr 20, 2011 at 17:15, Thomas Witkowski < thomas.witkowski(a)tu-dresden.de> wrote: > They really can't because sometimes the Lagrange multiplier itself will be >> stored on a different process. Also, this representation has lots of empty >> rows where as storing B, every row has entries (usually 2 entries). >> > Why could it happen to define a Lagrange multiplier, that "connects" to > subdomains, i.e. two processes, on a third process? Usually you would not choose this, but the Lagrange multiplier connects two subdomains and is owned by only one process. The other needs some communication to get its influence.

1 0

Re: [petsc-dev] [petsc-users] FETI-DP
by Thomas Witkowski 20 Apr '11

20 Apr '11

Jed Brown wrote: > By the way, you should definitely read Klawonn and Rheinbach's > "Inexact FETI-DP methods" > (http://onlinelibrary.wiley.com/doi/10.1002/nme.1758/pdf , ask me if > you don't have access). I think it has the most clear description of > what is going on, and offers the most flexibility in terms of which > spaces to iterate in. Thanks for the hint, I will do so. Thomas

1 0

Re: [petsc-dev] [petsc-users] FETI-DP
by Thomas Witkowski 20 Apr '11

20 Apr '11

Jed Brown wrote: > On Wed, Apr 20, 2011 at 17:05, Thomas Witkowski > <thomas.witkowski(a)tu-dresden.de > <mailto:[email protected]>> wrote: > > Following the mathematical representation wouldn't make it more > sense to store B but B^T, because the local matrices B_i^T are > real local and would have no off-diagonal elements? > > > They really can't because sometimes the Lagrange multiplier itself > will be stored on a different process. Also, this representation has > lots of empty rows where as storing B, every row has entries (usually > 2 entries). Why could it happen to define a Lagrange multiplier, that "connects" to subdomains, i.e. two processes, on a third process? The last argument is clear to me!

1 0

Re: [petsc-dev] [petsc-users] FETI-DP
by Jed Brown 20 Apr '11

20 Apr '11

By the way, you should definitely read Klawonn and Rheinbach's "Inexact FETI-DP methods" (http://onlinelibrary.wiley.com/doi/10.1002/nme.1758/pdf , ask me if you don't have access). I think it has the most clear description of what is going on, and offers the most flexibility in terms of which spaces to iterate in.

1 0

Re: [petsc-dev] [petsc-maint #70913] petsc-dev compilation error
by Barry Smith 20 Apr '11

20 Apr '11

I think Jed pointed out a problem with the current approach. We don't really want to have MANSECs for each DMXXXX but we do want different include files. I forgot what Jed and I concluded but it could be that some makefiles need SUBMANSEC that appends onto the MANSEC. By the way it is DMDA not dadm, the base class always comes first. Barry On Apr 20, 2011, at 9:39 AM, Matthew Knepley wrote: > On Wed, Apr 20, 2011 at 9:10 AM, Satish Balay <petsc-maint(a)mcs.anl.gov>wrote: > >> On Wed, 20 Apr 2011, Matthew Knepley wrote: >> >>> On Wed, Apr 20, 2011 at 8:16 AM, Satish Balay <petsc-maint(a)mcs.anl.gov >>> wrote: >>> >>>> On Wed, 20 Apr 2011, Matthew Knepley wrote: >>>> >>>>> On Wed, Apr 20, 2011 at 5:05 AM, Sylvain Barbot < >>>> sylbar.vainbot(a)gmail.com>wrote: >>>>> >>>>>> Hi, >>>>>> >>>>>> I run into errors trying to compile petsc-dev. >>>>>> Should i download another version of it, or i'm doing something >> wrong >>>>>> in the config? >>>>>> >>>>> >>>>> Satish, this is a --with-fortran-interfaces problem. What needs to be >>>>> included in the new DM >>>>> headers in order to generate these? >>>> >>>> The sources should have a makefile with the correct 'MANSEC' value. >>>> >>> >>> 1) What generates the interfaces? >> >> bfort via bin/maint/generatefortranstubs.py >>> >>> 2) I do not know what correct might be >>> >> >> If some of the sources should be tagged as 'DADM' instedad of 'DM' - >> then MANSEC=DADM should be listed in the corresponding makefiles for >> the sources. >> >> Or else we'll have to remove all dadm.h related includes [and merge >> with dm.h] >> > > Okay, I will try to fix it. Can you check the docs to make sure my fixes > do the right thing for documentation, etc? > > Matt > > >> Satish >> >>> Matt >>> >>> >>>> Satish >>>> >>>>> >>>>> Thanks, >>>>> >>>>> Matt >>>>> >>>>> >>>>>> Thanks for your help, >>>>>> Sylvain Barbot >>>>>> >>>>>> >>>>> >>>>> >>>>> >>>> >>>> >>>> >>> >>> >>> >> >> >> > > > -- > What most experimenters take for granted before they begin their experiments > is infinitely more interesting than any results to which their experiments > lead. > -- Norbert Wiener > > On Wed, Apr 20, 2011 at 9:10 AM, Satish Balay <petsc-maint(a)mcs.anl.gov> wrote: > On Wed, 20 Apr 2011, Matthew Knepley wrote: > > > On Wed, Apr 20, 2011 at 8:16 AM, Satish Balay <petsc-maint(a)mcs.anl.gov>wrote: > > > > > On Wed, 20 Apr 2011, Matthew Knepley wrote: > > > > > > > On Wed, Apr 20, 2011 at 5:05 AM, Sylvain Barbot < > > > sylbar.vainbot(a)gmail.com>wrote: > > > > > > > > > Hi, > > > > > > > > > > I run into errors trying to compile petsc-dev. > > > > > Should i download another version of it, or i'm doing something wrong > > > > > in the config? > > > > > > > > > > > > > Satish, this is a --with-fortran-interfaces problem. What needs to be > > > > included in the new DM > > > > headers in order to generate these? > > > > > > The sources should have a makefile with the correct 'MANSEC' value. > > > > > > > 1) What generates the interfaces? > > bfort via bin/maint/generatefortranstubs.py > > > > 2) I do not know what correct might be > > > > If some of the sources should be tagged as 'DADM' instedad of 'DM' - > then MANSEC=DADM should be listed in the corresponding makefiles for > the sources. > > Or else we'll have to remove all dadm.h related includes [and merge > with dm.h] > > Okay, I will try to fix it. Can you check the docs to make sure my fixes > do the right thing for documentation, etc? > > Matt > > Satish > > > Matt > > > > > > > Satish > > > > > > > > > > > Thanks, > > > > > > > > Matt > > > > > > > > > > > > > Thanks for your help, > > > > > Sylvain Barbot > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener

1 0

Re: [petsc-dev] [petsc-users] FETI-DP
by Jed Brown 20 Apr '11

20 Apr '11

On Wed, Apr 20, 2011 at 17:05, Thomas Witkowski < thomas.witkowski(a)tu-dresden.de> wrote: > Following the mathematical representation wouldn't make it more sense to > store B but B^T, because the local matrices B_i^T are real local and would > have no off-diagonal elements? > They really can't because sometimes the Lagrange multiplier itself will be stored on a different process. Also, this representation has lots of empty rows where as storing B, every row has entries (usually 2 entries).

1 0

Re: [petsc-dev] [petsc-users] FETI-DP
by Thomas Witkowski 20 Apr '11

20 Apr '11

Jed Brown wrote: > > I'm confuse because in the work of Klawoon/Rheinbach, it is > claimed that the following operator can be solved in a pure local way: > > F = \sum_{i=1}^{N} B^i inv(K_BB^i) trans(B^i) > > > Did they use "F" for this thing? Usually F is the FETI-DP operator > which involves a Schur complement of the entire partially assembled > operator in the dual space. In any case, this thing is not purely > local since the jump operators B^i need neighboring values so it has > the same communication as a MatMult. "F" is here the usual FETI-DP operator. I just omitted the coarse space part for more readability. > > > With B^i the jump operators and K_BB^i the discretization of the > sub domains with the primal nodes. > > > I think you mean "with the primal nodes removed". Yes! > > > From the notation it follows that EACH local solve takes the whole > vector of Lagrange multipliers. But this is not applicable for a > good parallel implementation. Any hint on this topic would be > helpful for me to understand this problem. > > > I can't tell from their papers how B is stored. It would be natural to > simply store B as a normal assembled matrix with a standard row > partition of the Lagrange multipliers. Then you would apply the > subdomain solve operator using > > MatMultTranspose(B,XLambdaGlobal,XGlobal); > for (i=0; i<nlocalsub; i++) { > Vec XSubdomain,YSubdomain; > VecGetSubVector(XGlobal,sublocal[i],&XSubdomain); // no copy if > subdomains are contiguous > VecGetSubVector(YGlobal,sublocal[i],&YSubdomain); // also no copy > KSPSolve(kspK_BB[i],XSubdomain,YSubdomain); // purely local solve, > often KSPPREONLY and PCLU > VecRestoreSubVector(XGlobal,sublocal[i],&XSubdomain); > VecRestoreSubVector(YGlobal,sublocal[i],&YSubdomain); > } > MatMult(B,YGlobal,YLambdaGlobal); > > All the communication is in the MatMultTranspose and MatMult. The > "Global" vectors here are global with respect to K_BB (interior and > interface dofs, primal dofs removed). I don't think there is need to > ever store K_BB as a parallel matrix, it would be a separate matrix > per subdomain (in the general case, subdomains could be parallel on > subcommunicators). > > This code should handle nlocalsub subdomains owned by the local > communicator, typically PETSC_COMM_SELF. The index sets (IS) in > sublocal represent the global (space of K_BB) dofs, usually these are > contiguous sets so they can be represented very cheaply. Following the mathematical representation wouldn't make it more sense to store B but B^T, because the local matrices B_i^T are real local and would have no off-diagonal elements? Also the very first multiplication MatMult(B,XLambdaGlobal,XGlobal) could be done in a local way. But I'm not sure what will than happen with the very last matrix multiplication, MatMultTranspose(B,YGlobal,YLambdaGlobal). Would it potentially result in an all-to-all communication?

1 0

Re: [petsc-dev] ML changing the number of levels
by Jed Brown 20 Apr '11

20 Apr '11

On Wed, Apr 20, 2011 at 16:10, Leo van Kampenhout <lvankampenhout(a)gmail.com>wrote: > Okay, I think the default setting for PCMLMaxNLevelsis 10, so i'm not > nearly there. It uses same number of levels (4) for a 64^3 grid and a 200^3 > grid. Isn't that strange? Not really, it's coarsening rapidly like it's supposed to.

1 0

Re: [petsc-dev] ML changing the number of levels
by Leo van Kampenhout 20 Apr '11

20 Apr '11

Okay, I think the default setting for PCMLMaxNLevelsis 10, so i'm not nearly there. It uses same number of levels (4) for a 64^3 grid and a 200^3 grid. Isn't that strange? Thanks, 2011/4/20 Jed Brown <jed(a)59a2.org> > On Wed, Apr 20, 2011 at 15:35, Leo van Kampenhout < > lvankampenhout(a)gmail.com> wrote: > >> I was wondering if it is already possible to set the number of levels with >> the ML package. I tried -pc_mg_levels and PCMGSetLevels, both do not work. >> > > -pc_ml_maxnlevels (there is not currently a PCMLSetMaxNLevels()) > > >> >> I have an 8*10^6 Poisson matrix, and so far ML has given me the best >> results. However, I think with increasing the gridsize, i should also >> increase the number of multigrid levels. What are your thoughts? >> > > ML automatically decides the number of levels, subject to the constraint > above. >

1 0

Re: [petsc-dev] ML changing the number of levels
by Jed Brown 20 Apr '11

20 Apr '11

On Wed, Apr 20, 2011 at 15:35, Leo van Kampenhout <lvankampenhout(a)gmail.com>wrote: > I was wondering if it is already possible to set the number of levels with > the ML package. I tried -pc_mg_levels and PCMGSetLevels, both do not work. > -pc_ml_maxnlevels (there is not currently a PCMLSetMaxNLevels()) > > I have an 8*10^6 Poisson matrix, and so far ML has given me the best > results. However, I think with increasing the gridsize, i should also > increase the number of multigrid levels. What are your thoughts? > ML automatically decides the number of levels, subject to the constraint above.

1 0