March 2011 - petsc-dev - mailman3.cels.anl.gov

Re: [petsc-dev] Using PCFieldSplitSetIS
by Jed Brown 16 Mar '11

16 Mar '11

On Wed, Mar 16, 2011 at 07:37, Thomas Witkowski < thomas.witkowski(a)tu-dresden.de> wrote: > Thanks for explanations! It works fine in my code. But I have two questions > about it, maybe you can help me with them: > - To the first, is the LU factorization on block A_00 done only once? > Yes, one way to confirm this is to run with -log_summary and check MatLUFactor{Sym,Num}. > - I have run the code with -fieldsplit_1_ksp_monitor to get some more > information about the internal solves. I expected to get information about > one iterative solver (for solving the Schur complement system), but I got > three, all of them need around 20 iterations for my example. Could you > explain to me what is actually solved there? > I'm going to guess that you still have an outer KSP that (in the global norm, rather than the partitioned norm used inside of splits) has a tighter tolerance, therefore it takes a few outer iterations. If you use loose inner tolerances then the preconditioner becomes nonlinear and you'll need to use FGMRES for the outer. When in doubt, run with -ksp_view and show us the results if you don't understand. Additionally, monitoring inner solves separately can be useful, e.g. -fieldsplit_0_ksp_converged_reason -fieldsplit_1_ksp_monitor -ksp_monitor_true_residual.

1 0

Re: [petsc-dev] Using PCFieldSplitSetIS
by Thomas Witkowski 16 Mar '11

16 Mar '11

Jed Brown wrote: > On Mon, Mar 14, 2011 at 12:32, Thomas Witkowski > <thomas.witkowski(a)tu-dresden.de > <mailto:[email protected]>> wrote: > > Should I define blocks or splits for the subdomains and the > interior nodes? And what is the best way to force PETSc to make > some LU factorization on each subdomain and to store it (it is > needed to create the reduced Schur system, to define the action of > the Schur complement operator and to solve the subdomain unknows > in the last step) and to use it later? > > > Okay, define two splits. The first consists of all the interior nodes, > the second has all the interface nodes. Now use -pc_fieldsplit_type > schur -fieldsplit_0_ksp_type preonly -fieldsplit_0_pc_type bjacobi > -fieldsplit_0_sub_pc_type lu. Remember to look at -ksp_view and -help > for options. You have a choice of preconditioning the Schur > complement, by default it just uses the interface matrix itself (which > is usually nearly diagonal). Thanks for explanations! It works fine in my code. But I have two questions about it, maybe you can help me with them: - To the first, is the LU factorization on block A_00 done only once? - I have run the code with -fieldsplit_1_ksp_monitor to get some more information about the internal solves. I expected to get information about one iterative solver (for solving the Schur complement system), but I got three, all of them need around 20 iterations for my example. Could you explain to me what is actually solved there? When I have a look in the manual about the implementation of -pc_fieldsplit_type schur, there are three ksp objects. But two of them make a solve with A_00, which should take only one iteration. Thomas > > That is, performing block Jacobi with direct subdomain solves on the > (parallel) interior matrix will be the same as a direct solve with > this matrix because all the subdomains are actually uncoupled. > > My point about exposing less concurrency had to do with always needing > to solve problems with the parallel interior-node matrix which could > actually be stored separately since the systems are not truly coupled. > This is most relevant with multiple subdomains per process or if you > are forming an explicit Schur complement (to build a course level > operator, such as with FETI-DP/BDDC).

1 0

Re: [petsc-dev] Lagging the numerical factorization for one of the fields of fieldsplit preconditioner
by Shri 15 Mar '11

15 Mar '11

> > The system i'm trying to solve is of the form > > 0 = A*y + g(x) > > 0 = f(x,y) > > which has the jacobian > > > > | A dg_dx | > > | df_dy df_dx | > > > > I'm using the schur preconditioner which needs factorization of A > > and (df_dx - df_dy*inv(A)*dg_dx). > > df_dx - df_dy*inv(A)*dg_dx is never formed (and hence never factored) > it is dense and way to expensive to ever form. Now approx > inverse(df_dx) may be used as a preconditioner for the Schur > complement and this means df_dx would be factored. > > Please send me the output from running with -snes_view so I can see > exactly what Schur complement preconditioner you are forming. See the attached file. > > In affect I think you are correct we want A to not be factored each > time because A is a constant matrix that does not change. Is this > correct? Yes, A is constant. > > Do you form the big-assed matrix > > | A dg_dx | > > | df_dy df_dx | > > explicitly as one big matrix? Yes. > And each time-step you changes values in > the dg_dx, df_dy and df_dx parts with MatSetValues? Yes,that's correct. > > > One fix may be to make PCSetUp() smarter and know the matrix has not > changed and hence it doesn't have to be refactored (instead of only > relying on the flag passed to KSPSetOperators()). > > Barry > > > > > Since,A is constant > > it needs to be factorized only once while (df_dx - > > df_dy*inv(A)*dg_dx) is not constant and needs to be refactorized. > > > >> This will be > >> tricky because we don't have a mechanism to pass this information > >> down > >> into PCFIELDSPLIT. Tell me exactly what you want to do that it > >> cannot > >> currently do and I'll what we can do about it. > > > > Currently,the factorization is done for 'both or none'.I wanted the > > factorization to be done for 'either,both,or none'. > > I was thinking about having an option say > > -fieldsplit_xxx_lag_preconditioner <niterations> which > > will delay the refactorization for field xxx by niterations and if > > niterations > > is set to -1 then it will not update the factorization. > > > > Thanks, > > Shri > > > >> > >> Barry > >> > >> > >> > >> On Mar 14, 2011, at 4:41 PM, Shri wrote: > >> > >>> > >>> | A B | |dx| |fx| > >>> Jacobian = = > >>> | C D | |dy| |fy| > >>> > >>> While solving this linear system with a fieldsplit precondioner,is > >>> there a way to lag the numerical factorization of the A matrix. > >>> The > >>> A matrix in my application is constant and is about 3-4 times > >>> bigger > >>> than the D matrix. The log summary shows that about 90% of the > >>> time > >>> is spent in the numerical factorization phase. So i would like to > >>> do > >>> the numerical factorization for A matrix just once and not have it > >>> factorized for all the snes iterations. > >>> Speaking on the same note, should snes lag_preconditioner > >>> functionality be moved to ksp or maybe a > >>> -fieldsplit_xxx_lag_preconditioner option? > >>> > >>> Thanks, > >>> Shri

1 0

Re: [petsc-dev] BuildSystem concerns
by Satish Balay 15 Mar '11

15 Mar '11

On Tue, 15 Mar 2011, Barry Smith wrote: > > Stuff like this in BlasLapack.py > > def getSharedFlag(self,cflags): > for flag in ['-PIC', '-fPIC', '-KPIC', '-qpic']: > if cflags.find(flag) >=0: return flag > return '' > > def getPrecisionFlag(self,cflags): > for flag in ['-m32', '-m64', '-xarch=v9','-q64']: > if cflags.find(flag) >=0: return flag > return '' > > def getWindowsNonOptFlags(self,cflags): > for flag in ['-MT','-MTd','-MD','-threads']: > if cflags.find(flag) >=0: return flag > return '' > > > are of major concern. So someone adds a new "precision" flag in BuildSystem they have to know to go into BlasLapack.py and edit it? Now there is another package that needs this stuff: tetgen. Both these packages need 'NOOPT'. [if we eliminate all options - then there are compile breakages] The above code developed adhoc - whenever there was breakage - we hacked in a fix. But I'm not sure if there is a proper fix. If we assume all optflags are in COPTFLAGS/FOPTFLAGS - then I guess BuidSystem could do this automatically instead of using: self.setCompilers.getCompilerFlags() use a new function provided by buidsystem: self.setCompilers.getCompilerNOOPTFlags() This one does the comparision between the current flags - and stripts out the relavent C/F/CXX OPTFLAGS set by user [or the defaults set by buildsystem] If optflags creep in via some other configure option [thats users problem]? Satish

1 0

Re: [petsc-dev] BuildSystem concerns
by Matthew Knepley 15 Mar '11

15 Mar '11

On Tue, Mar 15, 2011 at 7:06 PM, Barry Smith <bsmith(a)mcs.anl.gov> wrote: > > Stuff like this in BlasLapack.py > > def getSharedFlag(self,cflags): > for flag in ['-PIC', '-fPIC', '-KPIC', '-qpic']: > if cflags.find(flag) >=0: return flag > return '' > > def getPrecisionFlag(self,cflags): > for flag in ['-m32', '-m64', '-xarch=v9','-q64']: > if cflags.find(flag) >=0: return flag > return '' > > def getWindowsNonOptFlags(self,cflags): > for flag in ['-MT','-MTd','-MD','-threads']: > if cflags.find(flag) >=0: return flag > return '' > > > are of major concern. So someone adds a new "precision" flag in BuildSystem > they have to know to go into BlasLapack.py and edit it? Is the intent to deal with crappy flags from users here? If not, we can use the information about precision and shared libraries from modules rather than from flags. Do not know about Windows. Matt > > Barry > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener

1 0

Re: [petsc-dev] BuildSystem concerns
by Sean Farley 15 Mar '11

15 Mar '11

> > Stuff like this in BlasLapack.py > > def getSharedFlag(self,cflags): > for flag in ['-PIC', '-fPIC', '-KPIC', '-qpic']: > if cflags.find(flag) >=0: return flag > return '' > > def getPrecisionFlag(self,cflags): > for flag in ['-m32', '-m64', '-xarch=v9','-q64']: > if cflags.find(flag) >=0: return flag > return '' > > def getWindowsNonOptFlags(self,cflags): > for flag in ['-MT','-MTd','-MD','-threads']: > if cflags.find(flag) >=0: return flag > return '' > > > are of major concern. So someone adds a new "precision" flag in BuildSystem > they have to know to go into BlasLapack.py and edit it? This is absurd, we should be generating BlasLapack.py! -- What most researchers put in their email signature is infinitely more interesting than the rest of their message. -- Norbert Wiener

1 0

BuildSystem concerns
by Barry Smith 15 Mar '11

15 Mar '11

Stuff like this in BlasLapack.py def getSharedFlag(self,cflags): for flag in ['-PIC', '-fPIC', '-KPIC', '-qpic']: if cflags.find(flag) >=0: return flag return '' def getPrecisionFlag(self,cflags): for flag in ['-m32', '-m64', '-xarch=v9','-q64']: if cflags.find(flag) >=0: return flag return '' def getWindowsNonOptFlags(self,cflags): for flag in ['-MT','-MTd','-MD','-threads']: if cflags.find(flag) >=0: return flag return '' are of major concern. So someone adds a new "precision" flag in BuildSystem they have to know to go into BlasLapack.py and edit it? Barry

1 0

Re: [petsc-dev] Lagging the numerical factorization for one of the fields of fieldsplit preconditioner
by Jed Brown 15 Mar '11

15 Mar '11

On Wed, Mar 16, 2011 at 00:39, Barry Smith <bsmith(a)mcs.anl.gov> wrote: > When you "retain ownership" you could also do things like zero the > setvalues() pointer etc to prevent the user from changing it? You could. Alternatively, there could be MatSetMutable(Mat,PetscBool) that would generate an error on mutation. Then if a user was sure there were no copies, they could change the flag and mutate away. If they were not sure, they could make a copy.

1 0

Re: [petsc-dev] Lagging the numerical factorization for one of the fields of fieldsplit preconditioner
by Barry Smith 15 Mar '11

15 Mar '11

On Mar 15, 2011, at 6:33 PM, Jed Brown wrote: > On Wed, Mar 16, 2011 at 00:29, Barry Smith <bsmith(a)mcs.anl.gov> wrote: > Of course I know what A is. Because I am only doing this tracking after a MatGetSubmatrix() has been called which provide the information about what A is. > > Yes, that's much better and another reason for MatGetSubMatrix() to retain ownership of the submatrices. When you "retain ownership" you could also do things like zero the setvalues() pointer etc to prevent the user from changing it? Barry

1 0

Re: [petsc-dev] Lagging the numerical factorization for one of the fields of fieldsplit preconditioner
by Jed Brown 15 Mar '11

15 Mar '11

On Wed, Mar 16, 2011 at 00:29, Barry Smith <bsmith(a)mcs.anl.gov> wrote: > Of course I know what A is. Because I am only doing this tracking after a > MatGetSubmatrix() has been called which provide the information about what A > is. Yes, that's much better and another reason for MatGetSubMatrix() to retain ownership of the submatrices.

1 0