August 2007 - petsc-dev - mailman3.cels.anl.gov

Re: changes to SNES
by Lisandro Dalcin 31 Aug '07

31 Aug '07

Barry, Matt Do you know/remember why SNES has both 1- "vec_sol", and "vec_sol_always" 2 - "vec_func", and "vec_func_always" Is this extrictly needed? Do you remember some corner case needing this? On 8/31/07, Barry Smith <bsmith(a)mcs.anl.gov> wrote: > > Lisandro, > > The changes sound fine for petsc-dev, improving the > reference counting. But this is too large a change for a patch. > Patches are supposed to be minimulistic changes that fix particular > problems that are impacting end users. > > Barry > > On Fri, 31 Aug 2007, Lisandro Dalcin wrote: > > > There are some things broken in SNES, and others that could be > > improved. In particular, I am going to make sure SNES take ownership > > and correctly manage reference counting of all the user provided > > vectors, that is: > > > > - function vector (passed in SNESSetFunction) > > - solution vector (passed in SNESSolve/SNESSetSolution) > > - afine vector (passed in SNESSolve/SNESSetRhs) > > > > All this is similar to what I did in the past for KSP > > > > I almost sure all those changes are completelly backward compatible. > > For example, the user was in charge of calling VecDestroy() with the > > provided function vector, but after calling SNESSolve. Now, users are > > able to call it after SNESSetFunction is called, so they do not need > > to mantain a reference to the function vector. The same applies to > > solution and afine vectors. > > > > So I want to ask you: can I push this in release-2.3.3? > > > > > > > > > > -- Lisandro Dalcï¿½n --------------- Centro Internacional de Mï¿½todos Computacionales en Ingenierï¿½a (CIMEC) Instituto de Desarrollo Tecnolï¿½gico para la Industria Quï¿½mica (INTEC) Consejo Nacional de Investigaciones Cientï¿½ficas y Tï¿½cnicas (CONICET) PTLC - Gï¿½emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594

1 0

Re: changes to SNES
by Barry Smith 31 Aug '07

31 Aug '07

Lisandro, The changes sound fine for petsc-dev, improving the reference counting. But this is too large a change for a patch. Patches are supposed to be minimulistic changes that fix particular problems that are impacting end users. Barry On Fri, 31 Aug 2007, Lisandro Dalcin wrote: > There are some things broken in SNES, and others that could be > improved. In particular, I am going to make sure SNES take ownership > and correctly manage reference counting of all the user provided > vectors, that is: > > - function vector (passed in SNESSetFunction) > - solution vector (passed in SNESSolve/SNESSetSolution) > - afine vector (passed in SNESSolve/SNESSetRhs) > > All this is similar to what I did in the past for KSP > > I almost sure all those changes are completelly backward compatible. > For example, the user was in charge of calling VecDestroy() with the > provided function vector, but after calling SNESSolve. Now, users are > able to call it after SNESSetFunction is called, so they do not need > to mantain a reference to the function vector. The same applies to > solution and afine vectors. > > So I want to ask you: can I push this in release-2.3.3? > > > >

1 0

changes to SNES
by Lisandro Dalcin 31 Aug '07

31 Aug '07

There are some things broken in SNES, and others that could be improved. In particular, I am going to make sure SNES take ownership and correctly manage reference counting of all the user provided vectors, that is: - function vector (passed in SNESSetFunction) - solution vector (passed in SNESSolve/SNESSetSolution) - afine vector (passed in SNESSolve/SNESSetRhs) All this is similar to what I did in the past for KSP I almost sure all those changes are completelly backward compatible. For example, the user was in charge of calling VecDestroy() with the provided function vector, but after calling SNESSolve. Now, users are able to call it after SNESSetFunction is called, so they do not need to mantain a reference to the function vector. The same applies to solution and afine vectors. So I want to ask you: can I push this in release-2.3.3? -- Lisandro Dalcï¿½n --------------- Centro Internacional de Mï¿½todos Computacionales en Ingenierï¿½a (CIMEC) Instituto de Desarrollo Tecnolï¿½gico para la Industria Quï¿½mica (INTEC) Consejo Nacional de Investigaciones Cientï¿½ficas y Tï¿½cnicas (CONICET) PTLC - Gï¿½emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594

1 0

Re: GMRES/FGMRES and KSP_NORM_NO
by Barry Smith 29 Aug '07

29 Aug '07

Good argument. Make the change requiring setting both. Barry On Wed, 29 Aug 2007, Lisandro Dalcin wrote: > On 8/29/07, Barry Smith <bsmith(a)mcs.anl.gov> wrote: > > On Wed, 29 Aug 2007, Lisandro Dalcin wrote: > > > Now, other issue, regarding this adition. Currently, passing > > > '-ksp_norm_type NO' ALSO set KSPSkipConverged. Should I remove that? > > > > No, I think it should remain the same. > > OK, but then there is something that is a bit confusing IMHO, > > 1- if you use command line your automatically get KSP_NORM_NO + KSPSkipConverged > > 1- if you call KSPSetNormType(ksp, KSP_NORM_NO), you do not get > SkipConverged. Perhaps KSPSetNormType() should be in charge of setting > SkipConverged()? Or is this too much second-guessing the user? > > >

1 0

Re: GMRES/FGMRES and KSP_NORM_NO
by Lisandro Dalcin 29 Aug '07

29 Aug '07

On 8/29/07, Barry Smith <bsmith(a)mcs.anl.gov> wrote: > On Wed, 29 Aug 2007, Lisandro Dalcin wrote: > > Now, other issue, regarding this adition. Currently, passing > > '-ksp_norm_type NO' ALSO set KSPSkipConverged. Should I remove that? > > No, I think it should remain the same. OK, but then there is something that is a bit confusing IMHO, 1- if you use command line your automatically get KSP_NORM_NO + KSPSkipConverged 1- if you call KSPSetNormType(ksp, KSP_NORM_NO), you do not get SkipConverged. Perhaps KSPSetNormType() should be in charge of setting SkipConverged()? Or is this too much second-guessing the user? -- Lisandro Dalcï¿½n --------------- Centro Internacional de Mï¿½todos Computacionales en Ingenierï¿½a (CIMEC) Instituto de Desarrollo Tecnolï¿½gico para la Industria Quï¿½mica (INTEC) Consejo Nacional de Investigaciones Cientï¿½ficas y Tï¿½cnicas (CONICET) PTLC - Gï¿½emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594

1 0

Re: GMRES/FGMRES and KSP_NORM_NO
by Barry Smith 29 Aug '07

29 Aug '07

On Wed, 29 Aug 2007, Lisandro Dalcin wrote: > On 8/29/07, Barry Smith <bsmith(a)mcs.anl.gov> wrote: > > You could perhaps an additional option > > -ksp_convergence_test <skip,default> > > if you like that allows turning it off from the command line. > > You are completely right, I'll add this on petsc-dev. > > Now, other issue, regarding this adition. Currently, passing > '-ksp_norm_type NO' ALSO set KSPSkipConverged. Should I remove that? No, I think it should remain the same. Barry > The 'correct way' for both features would be > > $ XXX -ksp_norm_type NO -ksp_convergence_test skip > > Sounds this fine? > > > > > > On Wed, 29 Aug 2007, Lisandro Dalcin wrote: > > > > > GMRES and FGMRES generate an error if the user sets KSP_NORM_NO? Is > > > this really needed? > > > > > > Curently using option '-ksp_norm_type NO' is handy, as it implies > > > KSPSkipConverged being set. > > > > > > > > > > > > >

1 0

Re: GMRES/FGMRES and KSP_NORM_NO
by Lisandro Dalcin 29 Aug '07

29 Aug '07

On 8/29/07, Barry Smith <bsmith(a)mcs.anl.gov> wrote: > You could perhaps an additional option > -ksp_convergence_test <skip,default> > if you like that allows turning it off from the command line. You are completely right, I'll add this on petsc-dev. Now, other issue, regarding this adition. Currently, passing '-ksp_norm_type NO' ALSO set KSPSkipConverged. Should I remove that? The 'correct way' for both features would be $ XXX -ksp_norm_type NO -ksp_convergence_test skip Sounds this fine? > > On Wed, 29 Aug 2007, Lisandro Dalcin wrote: > > > GMRES and FGMRES generate an error if the user sets KSP_NORM_NO? Is > > this really needed? > > > > Curently using option '-ksp_norm_type NO' is handy, as it implies > > KSPSkipConverged being set. > > > > > > -- Lisandro Dalcï¿½n --------------- Centro Internacional de Mï¿½todos Computacionales en Ingenierï¿½a (CIMEC) Instituto de Desarrollo Tecnolï¿½gico para la Industria Quï¿½mica (INTEC) Consejo Nacional de Investigaciones Cientï¿½ficas y Tï¿½cnicas (CONICET) PTLC - Gï¿½emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594

1 0

Re: GMRES/FGMRES and KSP_NORM_NO
by Barry Smith 29 Aug '07

29 Aug '07

Lisandro, No. NORM_NO is suppose to mean that the norm computations are turned OFF. In the gmres algorithms the norm calculation cannot be turned off, therefore setting this flag should generate an error as it currently does. Barry You could perhaps an additional option -ksp_convergence_test <skip,default> if you like that allows turning it off from the command line. On Wed, 29 Aug 2007, Lisandro Dalcin wrote: > GMRES and FGMRES generate an error if the user sets KSP_NORM_NO? Is > this really needed? > > Curently using option '-ksp_norm_type NO' is handy, as it implies > KSPSkipConverged being set. > >

1 0

RE: Cray and PETSc
by Stephen R Ball 29 Aug '07

29 Aug '07

Hi Adrian You know me already, as I have already accepted to be a tester for your Cray optimised version of PETSc. I just thought that I would chip in here and let you know what external packages we currently use with PETSc. These are Hypre, Spooles and ML. Regards Stephen R. Ball Parallel Technology Support HPC DPD RAS Rm: G17 Bldg: E1.1 AWE(A) Aldermaston Reading Berkshire ENGLAND RG7 4PR Tel: +44 (0)118 982 4528 e-mail: stephen.r.ball(a)awe.co.uk -----Original Message----- From: owner-petsc-dev(a)mcs.anl.gov [mailto:[email protected]] On Behalf Of Adrian Tate Sent: 23 August 2007 19:22 To: Tom Cortese Cc: Barry Smith; petsc-developers(a)mcs.anl.gov; petsc-dev(a)mcs.anl.gov; Keita Teranishi Subject: EXTERNAL: RE: Cray and PETSc Tom Thanks for your comments - I'm sure your experiences with PETSc on Crays will be helpful to us. Perhaps Keita and I could communicate directly with you on these issues and to find out more about the requirements of your users? It is our ultimate goal to provide everything that Cray customers need with PETSc, but I can't comment on how feasible that is without knowing more about the packages that are widely used. For our initial release we were thinking of providing HYPRE and ParMETIS, we also provide SuperLU already within Cray LibSci. MUMPS is certainly something we should consider. This is a perfect opportunity to ask PETSc developers that use Cray hardware to suggest packages that they'd like to see provided by Cray. Adrian -----Original Message----- From: Tom Cortese [mailto:[email protected]] Sent: Thursday, August 23, 2007 10:53 AM To: Adrian Tate Cc: Barry Smith; petsc-developers(a)mcs.anl.gov; petsc-dev(a)mcs.anl.gov; Keita Teranishi Subject: RE: Cray and PETSc Hello all, First of all, I am NOT a PETSc developer, but as a member of the DoD HPC Modernization Program I DO install PETSc on a wide variety of hardware platforms (including XT3, XD1, and even X1 [that was rough; had to cross-compile on one machine and transfer files]), and have had numerous email exchanges working out all of the little quirks involved in installing PETSc on these machines, so the PETSc folks have been kind enough to let me watch this development email list. I hope I'm not over-stepping my bounds by asking this, but I have two comments: 1) Making PETSc available on the Cray machines sounds like a GREAT idea 2) Several of our users also like to have packages such as HYPRE and MUMPS, etc... installed along with PETSc. Would such external packages also be installed by Cray, or would users requiring a combination of, say, PETSc and MUMPS still have to make their own installation? Thanks, -Tom Cortese PET Computational Environnment Onsite On Thu, 23 Aug 2007, Adrian Tate wrote: > > Hi Barry > > Thanks for your reply. > > We will expect to provide the new full releases of PETSc as soon as we possibly can. Obviously there is something of a lag because Cray need to test, package and release the library, but we'll try to do so with minimal delay. Obviously any heads-up you can give us in advance will help reduce the lag. With respect to patch versions, it is going to be very difficult for Cray to release every new version, and by looking at the logs it seems that many of the patches would not be relevant to a Cray specific build. We will endeavor to monitor all patch releases and will go with an unscheduled new release of PETSc when we see a relevant patch. One thing that we can do to minimize confusion is to informally provide the most currently patched version to any customer who is experiencing a problem, although we might not go with a full release of the version in question. > > Regarding Cray modifications, we (Keita Teranishi or myself) will check in periodically with you to show you our improvements and to see which, if any need to be channeled back into the PETSc build. > > I hope this support structure works OK from your perspective and I look forward to working with the PETSc team. > > Thanks! > Adrian > > -----Original Message----- > From: Barry Smith [mailto:[email protected]] > Sent: Monday, August 20, 2007 11:12 AM > To: Adrian Tate > Cc: petsc-developers(a)mcs.anl.gov; petsc-dev(a)mcs.anl.gov > Subject: Re: Cray and PETSc > > > Adrian, > > Thank you for in the inquiry. > > > On Thu, 16 Aug 2007, Adrian Tate wrote: > >> Hello Barry >> >> I'm not sure we met in person - I am the lead of the libraries group >> at Cray. We decided some time ago that our iterative solver strategy >> would be to leverage PETSc, and to hopefully provide some Cray >> specific tunings to improve performance of the KSP solvers (largely >> through our custom sparse BLAS and some parallel tuning for our >> interconnect). I believe that John Lewis has been in communication >> with you with respect to the tuning of Sparse BLAS and their >> integration with the PETSc build. >> >> We are however, considering packaging and supplying PETSc along with >> the scientific library that we provide (libSci). This allows for >> better integration of our internal kernels and also it means that we >> are no longer requiring users (including benchmakers and internal >> users) to build their own PETSc. I see from your online page that >> doing so is acceptable as long as we use a copyright switch during >> configure. By applying this switch, do we make our PETSc library >> unsupported from your perspective? > > The GNU copyright code is tiny; it would not effect the usability > or support of PETSc > >> We do not expect to be able to >> provide anywhere near the degree of support that your team provide, >> and I was hoping to supply a pre-built library whose users could still >> seek assistance through your normal support channels - is this >> realistic? >> > Yes. > > The two isses that concern us with pre-packaged versions of PETSc are > 1) keeping up to date on our releases. We generally make two releases > a year and much prefer that users use the most recently release. > If they are using an older release it means we are less able to help > them. > 2) keeping up to date on our patches. We may make several bug patches > to each release. Users with a pre-packaged version have trouble keeping > up with the source code patches we provide. > >> >> >> >> >> Also, I would be interested to know your degree of interest in the >> Cray-specific modifications that we make - would you prefer those to >> be channeled back into the PETSc library? > > If they involved directly changing PETSc code, we much prefer to get > it channeled back into the master copy of the source code. Makes it > much easier to debug user code. If it is auxiliary code, like a faster > ddot() etc then it is more appropriate to not try to include it. > >> Any other comments you have >> on the way that Cray can contribute to the PETSc project, I would be >> very glad to hear. > > PETSc has a variety of "tuning factors" that could theoretically be > set optimally for a particular machine, these range from simple things > like compiler options, to a choice between C and Fortran versions of > the same code (what we call them Fortran kernels), to different loop > unrollings (in the inline.h file), even something like PetscMemzero() > which has five possible forms. Now currently we do not tune these > or even have a test harness for selecting a good tuning. One thing > you could/would like to do, is determine good choices for these > options on your machines. Just as a simple example, on some Linux systems > the basic libraries are just compiled with the GNU compiler, hence the > system memset() is not particularly effective. A version compiled of PETSc > compiled using a "Fortran memset" may be much faster, or Intel provides > its own _intel_fast_memset() which is better. I've seen a few percent > increase in performance of entire nonlinear PDE solver applications > just from using the non-default memset(). [This was specifically on > an Itanium system, but is likely on other configurations also]. > > > Barry > > >> >> >> >> Regards, >> >> >> Adrian Tate >> >> --- >> >> Technical Lead >> >> Math Software >> Cray Inc. >> >> (206) 349 5868 >> >> > > -- _______________________________________________________________________________ The information in this email and in any attachment(s) is commercial in confidence. If you are not the named addressee(s) or if you receive this email in error then any distribution, copying or use of this communication or the information in it is strictly prohibited. Please notify us immediately by email at admin.internet(at)awe.co.uk, and then delete this message from your computer. While attachments are virus checked, AWE plc does not accept any liability in respect of any virus which is not detected. AWE Plc Registered in England and Wales Registration No 02763902 AWE, Aldermaston, Reading, RG7 4PR

1 0

Re: KSPSkipConverged
by Todd Munson 29 Aug '07

29 Aug '07

Lisandro, All of these problems have been previously addressed in the development version, along with some minor changes to the code. I have not patched the release (nor do I know how to do this). The iteration counter now uses the number of matrix-vector products. Note that some of the same problems were in the nash and stcg code. Todd. On Tue, 28 Aug 2007, Lisandro Dalcin wrote: > On 8/28/07, Matthew Knepley <knepley(a)gmail.com> wrote: > > I will ask Todd about gltr since it might be supposed to > > do something > > funny. You really do not want to look at it. > > Now I understand way you said that!! Well, let me say that actually > GLTR is almost OK, the only two things to remark are: > > 1- KSPMonitor is called allways with it=0; perhaps this is OK, but > perhaps no. You get always output like '0 KSP Residual norm' > > 2 - At some point, it converges and returns without incrementing the > 'ksp->its' counter. That's the reason I though there could be a > problem. However, as the implementation is a bit contrived and > 'ksp->its' is used everywere, > > I think I'll leave GLTR for review by the author, please FWD this mail > to him... > > Regards, > > > > > > Matt > > > > > Below the result (petsc4py is a nice tool for test/debug, isn't it?) > > > > > > > > > tfqmr - CONVERGED_ITS - iters: 5, ||r|| = 4.881889e-03, > > > x0,N-1=0.999996,9.978373 > > > minres - CONVERGED_ITS - iters: 5, ||r|| = 5.356222e-02, > > > x0,N-1=1.000009,9.701171 > > > fgmres - DIVERGED_ITS - iters: 5, ||r|| = 5.356222e-02, > > > x0,N-1=1.000009,9.701171 > > > stcg - CONVERGED_ITS - iters: 5, ||r|| = 5.629995e-02, > > > x0,N-1=1.000052,9.771470 > > > qcg - CONVERGED_ITS - iters: 5, ||r|| = 5.629995e-02, > > > x0,N-1=-1.000052,-9.771470 > > > cg - CONVERGED_ITS - iters: 5, ||r|| = 5.629995e-02, > > > x0,N-1=1.000052,9.771470 > > > lgmres - DIVERGED_ITS - iters: 5, ||r|| = 5.356222e-02, > > > x0,N-1=1.000009,9.701171 > > > cgne - CONVERGED_ITS - iters: 5, ||r|| = 7.192229e-02, > > > x0,N-1=1.000000,7.116166 > > > chebychev - DIVERGED_ITS - iters: 5, ||r|| = 2.591834e+00, > > > x0,N-1=0.636559,0.708271 > > > cgs - CONVERGED_ITS - iters: 5, ||r|| = 1.457830e-03, > > > x0,N-1=1.000000,9.994777 > > > bicg - CONVERGED_ITS - iters: 5, ||r|| = 5.629995e-02, > > > x0,N-1=1.000052,9.771470 > > > lsqr - CONVERGED_ITS - iters: 5, ||r|| = 4.612376e-01, > > > x0,N-1=1.000000,7.116166 > > > gltr - CONVERGED_ITS - iters: 4, ||r|| = 5.629995e-02, > > > x0,N-1=1.000052,9.771470 > > > tcqmr - DIVERGED_ITS - iters: 5, ||r|| = 0.000000e+00, > > > x0,N-1=1.000009,9.701171 > > > bcgs - CONVERGED_ITS - iters: 5, ||r|| = 2.566301e-03, > > > x0,N-1=0.999703,9.982615 > > > cr - CONVERGED_ITS - iters: 5, ||r|| = 5.356222e-02, > > > x0,N-1=1.000009,9.701171 > > > symmlq - CONVERGED_ITS - iters: 5, ||r|| = 5.629995e-02, > > > x0,N-1=1.000052,9.771470 > > > bcgsl - CONVERGED_ITS - iters: 5, ||r|| = 1.232695e-03, > > > x0,N-1=0.999988,9.999315 > > > lcd - DIVERGED_ITS - iters: 5, ||r|| = 5.629995e-02, > > > x0,N-1=1.000052,9.771470 > > > preonly - CONVERGED_ITS - iters: 1, ||r|| = 0.000000e+00, > > > x0,N-1=1.000000,1.000000 > > > gmres - DIVERGED_ITS - iters: 5, ||r|| = 5.356222e-02, > > > x0,N-1=1.000009,9.701171 > > > richardson - DIVERGED_ITS - iters: 5, ||r|| = 1.215430e+00, > > > x0,N-1=1.000000,4.095100 > > > > > > > > > > > > > > > > > > > > On 8/28/07, Lisandro Dalcin <dalcinl(a)gmail.com> wrote: > > > > > Does it make sense to change KSPSkipConverged to return > > > > > KSP_CONVERGED_ITS if iternum==maxit ? > > > > > > > > > > KSP_DIVERGED_ITS means convergence failure, but IMHO, KSPSkipConverged > > > > > should not imply convergence failure (this has implications in SNES). > > > > > > > > > > > > > > > -- > > > > > Lisandro Dalcï¿½n > > > > > --------------- > > > > > Centro Internacional de Mï¿½todos Computacionales en Ingenierï¿½a (CIMEC) > > > > > Instituto de Desarrollo Tecnolï¿½gico para la Industria Quï¿½mica (INTEC) > > > > > Consejo Nacional de Investigaciones Cientï¿½ficas y Tï¿½cnicas (CONICET) > > > > > PTLC - Gï¿½emes 3450, (3000) Santa Fe, Argentina > > > > > Tel/Fax: +54-(0)342-451.1594 > > > > > > > > > > > > > > > > > > > > > > -- > > > > What most experimenters take for granted before they begin their > > > > experiments is infinitely more interesting than any results to which > > > > their experiments lead. > > > > -- Norbert Wiener > > > > > > > > > > > > > > > > > -- > > > Lisandro Dalcï¿½n > > > --------------- > > > Centro Internacional de Mï¿½todos Computacionales en Ingenierï¿½a (CIMEC) > > > Instituto de Desarrollo Tecnolï¿½gico para la Industria Quï¿½mica (INTEC) > > > Consejo Nacional de Investigaciones Cientï¿½ficas y Tï¿½cnicas (CONICET) > > > PTLC - Gï¿½emes 3450, (3000) Santa Fe, Argentina > > > Tel/Fax: +54-(0)342-451.1594 > > > > > > > > > > > > -- > > What most experimenters take for granted before they begin their > > experiments is infinitely more interesting than any results to which > > their experiments lead. > > -- Norbert Wiener > > > > > > > -- > Lisandro Dalcï¿½n > --------------- > Centro Internacional de Mï¿½todos Computacionales en Ingenierï¿½a (CIMEC) > Instituto de Desarrollo Tecnolï¿½gico para la Industria Quï¿½mica (INTEC) > Consejo Nacional de Investigaciones Cientï¿½ficas y Tï¿½cnicas (CONICET) > PTLC - Gï¿½emes 3450, (3000) Santa Fe, Argentina > Tel/Fax: +54-(0)342-451.1594 > >

1 0