February 2013 - petsc-dev - mailman3.cels.anl.gov

Re: [petsc-dev] [petsc-maint #154101] clarification of FP error thrown in VECNORM in 3.3 PETSc
by Barry Smith 22 Feb '13

22 Feb '13

Steve, I have removed the extra checks on norm, dot and mdot in petsc-3.3 in the next patch petsc-3.3 should behave like previous versions of PETSc. Barry On Feb 22, 2013, at 6:35 PM, "Cea, Stephen M" <stephen.m.cea(a)intel.com> wrote: > Thanks for the quick responses! > > For the short time I am pretty sure that we can just treat the error from KSP_Solve like non convergence. Long term what you propose sounds good. > > Thanks again, > steve > > -----Original Message----- > From: Barry Smith [mailto:[email protected]] > Sent: Friday, February 22, 2013 4:20 PM > To: petsc-maint(a)mcs.anl.gov; Jed Brown > Cc: For users of the development version of PETSc; Cea, Stephen M > Subject: Re: [petsc-dev] [petsc-maint #154101] clarification of FP error thrown in VECNORM in 3.3 PETSc > > > On Feb 22, 2013, at 6:02 PM, Jed Brown <jedbrown(a)mcs.anl.gov> wrote: > >> On Fri, Feb 22, 2013 at 5:57 PM, Barry Smith <bsmith(a)mcs.anl.gov> wrote: >> >>> We will figure out a way to manage this. I am thinking a global >>> flag you can set indicating you don't want these checks performed; >>> then you will get back the previous behavior. >> >> >> >> Would it be enough to have a flag per-object? > > Ok, I did a search through the code to see where we are checking it. > > 1) VecNorm, VecDot, VecMDot -- and generate an error > > 2) Many of the Krylov solvers and nonlinear solvers check the results of (most - should be all) norms and dots and either > > a) generate a not converged KSP or SNES converged reason (good) > > b) generate an error (bad, most do this). > > Note that currently the checks in the solvers are redundant because an error was already generated in the VecNorm, Dot or MDot so NaN never > gets to the iterative solver > > 3) Few other irrelevant places. > > The simplest thing would be to simply remove the checks in 1). Then, eventually, update all the solvers to NOT generate an error but instead return with an appropriate diverged reason. The drawback is that other code using dots and norms will always chug along without generating an error if users don't check the results themselves. > > I vote we just make a patch in 3.3 to remove the checks in norm, dot, mdot and Jed adds another project to the bitbucket list saying upgrade the checks in the solvers to return with appropriate diverged reason instead of generating an error. Then the whole flag business goes away. > > > > Barry > > >> >> I don't like a global flag because it would have to be set >> collectively (this isn't nice if you subproblems on subcomms and not >> everyone wants to tolerate NaNs.) Note that to tolerate NaNs and bail >> out cleanly, a reduction is required in places where it would otherwise not be. >> >

1 0

Re: [petsc-dev] [petsc-maint #154101] clarification of FP error thrown in VECNORM in 3.3 PETSc
by Barry Smith 22 Feb '13

22 Feb '13

On Feb 22, 2013, at 8:40 PM, Jed Brown <jedbrown(a)mcs.anl.gov> wrote: > > On Fri, Feb 22, 2013 at 6:19 PM, Barry Smith <bsmith(a)mcs.anl.gov> wrote: > > Ok, I did a search through the code to see where we are checking it. > > 1) VecNorm, VecDot, VecMDot -- and generate an error > > Yeah, I never liked this. > > > 2) Many of the Krylov solvers and nonlinear solvers check the results of (most - should be all) norms and dots and either > > a) generate a not converged KSP or SNES converged reason (good) > > b) generate an error (bad, most do this). > > We want the solvers to have an option to raise an error because it helps users track down where they made a mistake (especially if they're running in a shitty environment where -fp_trap and gdb doesn't work nicely). I'm not sure which should be the default. Currently KSPSetErrorIfNotConverged() will trigger an error at the end of the KSPSolve() routine if not converged (including NaN). If this is not close enough to the problem for you we could use the same flag to stop the code with a SETERRQ() as soon as the norm, dot, mdot problem is detected? Or we could add yet another flag? Barry > > > Note that currently the checks in the solvers are redundant because an error was already generated in the VecNorm, Dot or MDot so NaN never > gets to the iterative solver > > 3) Few other irrelevant places. > > The simplest thing would be to simply remove the checks in 1). Then, eventually, update all the solvers to NOT generate an error but instead return with > an appropriate diverged reason. The drawback is that other code using dots and norms will always chug along without generating an error if users don't > check the results themselves. > > I think this is an event that the caller should deal with since only they have enough context to return cleanly. Yes, it's more code, but it's the right thing. > > I vote we just make a patch in 3.3 to remove the checks in norm, dot, mdot and Jed adds another project to the bitbucket list saying upgrade the checks in the solvers to return with appropriate diverged reason instead of generating an error. Then the whole flag business goes away. > > That's fine with me.

1 0

Re: [petsc-dev] [petsc-maint #154101] clarification of FP error thrown in VECNORM in 3.3 PETSc
by Jed Brown 22 Feb '13

22 Feb '13

On Fri, Feb 22, 2013 at 6:19 PM, Barry Smith <bsmith(a)mcs.anl.gov> wrote: > > Ok, I did a search through the code to see where we are checking it. > > 1) VecNorm, VecDot, VecMDot -- and generate an error > Yeah, I never liked this. > > 2) Many of the Krylov solvers and nonlinear solvers check the results > of (most - should be all) norms and dots and either > > a) generate a not converged KSP or SNES converged reason (good) > > b) generate an error (bad, most do this). > We want the solvers to have an option to raise an error because it helps users track down where they made a mistake (especially if they're running in a shitty environment where -fp_trap and gdb doesn't work nicely). I'm not sure which should be the default. > > Note that currently the checks in the solvers are redundant because > an error was already generated in the VecNorm, Dot or MDot so NaN never > gets to the iterative solver > > 3) Few other irrelevant places. > > The simplest thing would be to simply remove the checks in 1). Then, > eventually, update all the solvers to NOT generate an error but instead > return with > an appropriate diverged reason. The drawback is that other code using dots > and norms will always chug along without generating an error if users don't > check the results themselves. > I think this is an event that the caller should deal with since only they have enough context to return cleanly. Yes, it's more code, but it's the right thing. > I vote we just make a patch in 3.3 to remove the checks in norm, dot, > mdot and Jed adds another project to the bitbucket list saying upgrade the > checks in the solvers to return with appropriate diverged reason instead of > generating an error. Then the whole flag business goes away. That's fine with me.

1 0

Re: [petsc-dev] [petsc-maint #154101] clarification of FP error thrown in VECNORM in 3.3 PETSc
by Cea, Stephen M 22 Feb '13

22 Feb '13

Thanks for the quick responses! For the short time I am pretty sure that we can just treat the error from KSP_Solve like non convergence. Long term what you propose sounds good. Thanks again, steve -----Original Message----- From: Barry Smith [mailto:[email protected]] Sent: Friday, February 22, 2013 4:20 PM To: petsc-maint(a)mcs.anl.gov; Jed Brown Cc: For users of the development version of PETSc; Cea, Stephen M Subject: Re: [petsc-dev] [petsc-maint #154101] clarification of FP error thrown in VECNORM in 3.3 PETSc On Feb 22, 2013, at 6:02 PM, Jed Brown <jedbrown(a)mcs.anl.gov> wrote: > On Fri, Feb 22, 2013 at 5:57 PM, Barry Smith <bsmith(a)mcs.anl.gov> wrote: > >> We will figure out a way to manage this. I am thinking a global >> flag you can set indicating you don't want these checks performed; >> then you will get back the previous behavior. > > > > Would it be enough to have a flag per-object? Ok, I did a search through the code to see where we are checking it. 1) VecNorm, VecDot, VecMDot -- and generate an error 2) Many of the Krylov solvers and nonlinear solvers check the results of (most - should be all) norms and dots and either a) generate a not converged KSP or SNES converged reason (good) b) generate an error (bad, most do this). Note that currently the checks in the solvers are redundant because an error was already generated in the VecNorm, Dot or MDot so NaN never gets to the iterative solver 3) Few other irrelevant places. The simplest thing would be to simply remove the checks in 1). Then, eventually, update all the solvers to NOT generate an error but instead return with an appropriate diverged reason. The drawback is that other code using dots and norms will always chug along without generating an error if users don't check the results themselves. I vote we just make a patch in 3.3 to remove the checks in norm, dot, mdot and Jed adds another project to the bitbucket list saying upgrade the checks in the solvers to return with appropriate diverged reason instead of generating an error. Then the whole flag business goes away. Barry > > I don't like a global flag because it would have to be set > collectively (this isn't nice if you subproblems on subcomms and not > everyone wants to tolerate NaNs.) Note that to tolerate NaNs and bail > out cleanly, a reduction is required in places where it would otherwise not be. >

1 0

Re: [petsc-dev] [petsc-maint #154101] clarification of FP error thrown in VECNORM in 3.3 PETSc
by Barry Smith 22 Feb '13

22 Feb '13

On Feb 22, 2013, at 6:02 PM, Jed Brown <jedbrown(a)mcs.anl.gov> wrote: > On Fri, Feb 22, 2013 at 5:57 PM, Barry Smith <bsmith(a)mcs.anl.gov> wrote: > >> We will figure out a way to manage this. I am thinking a global flag >> you can set indicating you don't want these checks performed; then you will >> get back the previous behavior. > > > > Would it be enough to have a flag per-object? Ok, I did a search through the code to see where we are checking it. 1) VecNorm, VecDot, VecMDot -- and generate an error 2) Many of the Krylov solvers and nonlinear solvers check the results of (most - should be all) norms and dots and either a) generate a not converged KSP or SNES converged reason (good) b) generate an error (bad, most do this). Note that currently the checks in the solvers are redundant because an error was already generated in the VecNorm, Dot or MDot so NaN never gets to the iterative solver 3) Few other irrelevant places. The simplest thing would be to simply remove the checks in 1). Then, eventually, update all the solvers to NOT generate an error but instead return with an appropriate diverged reason. The drawback is that other code using dots and norms will always chug along without generating an error if users don't check the results themselves. I vote we just make a patch in 3.3 to remove the checks in norm, dot, mdot and Jed adds another project to the bitbucket list saying upgrade the checks in the solvers to return with appropriate diverged reason instead of generating an error. Then the whole flag business goes away. Barry > > I don't like a global flag because it would have to be set collectively > (this isn't nice if you subproblems on subcomms and not everyone wants to > tolerate NaNs.) Note that to tolerate NaNs and bail out cleanly, a > reduction is required in places where it would otherwise not be. >

1 0

Re: [petsc-dev] [petsc-maint #154101] clarification of FP error thrown in VECNORM in 3.3 PETSc
by Jed Brown 22 Feb '13

22 Feb '13

On Fri, Feb 22, 2013 at 5:57 PM, Barry Smith <bsmith(a)mcs.anl.gov> wrote: > We will figure out a way to manage this. I am thinking a global flag > you can set indicating you don't want these checks performed; then you will > get back the previous behavior. Would it be enough to have a flag per-object? I don't like a global flag because it would have to be set collectively (this isn't nice if you subproblems on subcomms and not everyone wants to tolerate NaNs.) Note that to tolerate NaNs and bail out cleanly, a reduction is required in places where it would otherwise not be.

1 0

Re: [petsc-dev] [petsc-maint #154101] clarification of FP error thrown in VECNORM in 3.3 PETSc
by Barry Smith 22 Feb '13

22 Feb '13

Steve, Thanks for your report and sorry about the difficulties. We made the change thinking the sooner we report a FP error the better for the user. But you are right that sometimes it makes sense for the application to handle it directly. We will figure out a way to manage this. I am thinking a global flag you can set indicating you don't want these checks performed; then you will get back the previous behavior. Barry On Feb 22, 2013, at 5:07 PM, "Cea, Stephen M" <stephen.m.cea(a)intel.com> wrote: > Hi, > > We are upgrading from 3.0 to 3.3 version of PETSc. It is looking good because it is considerably faster. We have one issue that the vecnorm has been changed and it sets an error if finds a nan / inf in the vector. See message at the end. We have always exited the program when we get a PETSc error but for this case it is not a good idea since we used to find this ourselves and then cut the timestep and resolve. We might be able to just treat the error from KSPSolve ( iError = KSPSolve(slp.sles, b, x); ) as non convergence and cut the time step and resolve. But I am wondering if we can tell from the return which error it is. I tried looking in the manual but couldn't find a list of error codes etc. > > Thanks, > steve > > > > > > [0]PETSC ERROR: --------------------- Error Message ------------------------------------ > [0]PETSC ERROR: Floating point exception! > [0]PETSC ERROR: Infinite or not-a-number generated in norm! > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: Petsc Release Version 3.3.0, Patch 3, Wed Aug 29 11:26:24 CDT 2012 > [0]PETSC ERROR: See docs/changes/index.html for recent updates. > [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. > [0]PETSC ERROR: See docs/index.html for manual pages. > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: /nfs/pdx/disks/tcad_ptm_pdmg_work_03/usr/adlilak/FLPSCHECKINS/FLPS_NZ_PETSC/floops/src/floops.linux64 on a with-CPP- named dlxc0894.jf.intel.com by adlilak Thu Feb 21 23:43:15 2013 > [0]PETSC ERROR: Libraries linked from /p/dt/sdework9/mrduda/INN/Build2/petsc-3.3-p3/with-CPP-nomumps-icc13/lib > [0]PETSC ERROR: Configure run at Fri Oct 19 09:42:58 2012 > [0]PETSC ERROR: Configure options --with-debugging=0 --with-mpi-compilers=0 --with-mpi-shared=0 --with-mpi=1 --with-mpi-include=/p/dt/sde/tools/em64t_SLES10/MPI/4.0.3.004/include64 --with-mpi-lib="[/p/dt/sde/tools/em64t_SLES10/MPI/4.0.3.004/lib64/libmpi.a,/p/dt/sde/tools/em64t_SLES10/MPI/4.0.3.004/lib64/libmpiif.a]" --with-clanguage=C++ --with-vendor-compilers=intel --with-cc=icc --COPTFLAGS="-O2 -g -sox" --with-cxx=icpc --CXXOPTFLAGS="-O2 -g -sox" --with-fc=ifort --FOPTFLAGS="-O2 -g -sox" --LDFLAGS=-Wl,-rpath,$GCCDIR/lib64 --with-blas-lapack-lib="[/p/dt/sdework12/mrduda/KNC/c_compiler/install/mkl/lib/intel64/libmkl_rt.so]" --with-scalapack=1 --with-scalapack-include=/p/dt/sdework12/mrduda/KNC/c_compiler/install/mkl/include --with-scalapack-lib=/p/dt/sdework12/mrduda/KNC/c_compiler/install/mkl/lib/intel64/libmkl_scalapack_lp64.a --with-blacs=1 --with-blacs-include=/p/dt/sdework12/mrduda/KNC/c_compiler/install/mkl/include --with-blacs-lib=/p/dt/sdework12/mrduda/KNC/c_compiler/ > install/mkl/lib/intel64/libmkl_blacs_intelmpi_lp64.a > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: VecNorm() line 169 in /p/dt/sdework9/mrduda/INN/Build2/petsc-3.3-p3/src/vec/vec/interface/rvector.c > [0]PETSC ERROR: KSPSolve_BCGS() line 78 in /p/dt/sdework9/mrduda/INN/Build2/petsc-3.3-p3/src/ksp/ksp/impls/bcgs/bcgs.c > [0]PETSC ERROR: KSPSolve() line 446 in /p/dt/sdework9/mrduda/INN/Build2/petsc-3.3-p3/src/ksp/ksp/interface/itfunc.c > >

1 0

Re: [petsc-dev] [petsc-users] MATNEST with shell matrices
by Jed Brown 22 Feb '13

22 Feb '13

On Fri, Feb 22, 2013 at 4:43 PM, Boyce Griffith <griffith(a)cims.nyu.edu>wrote: > Sorry for the delay. The thing that "owns" the data that the Vec > represents also manages regridding and interpolation between grids. So if > you had something that did time stepping, the old Vec would get the rug > pulled out from under it if YourSolver(X) regridded things. (It doesn't > have to be this way --- you could have different AMR grids floating around > --- but this is how all of the transient solvers currently work in the > code.) Okay, so in light of this, I don't want to allow dynamically sized Vecs in PETSc. I know it's complicated to have multiple AMR grids and try to difference between them while preserving linear algebraic invariants so unless there is a real code that works that way, it's just going to encourage misuse. Instead, I'd like you to make the top-level interfaces (beneath which regridding could occur) return new Vecs. (If you don't regrid, you can return the old Vec.) You can also keep a reference count on your old AMR grid so that if the user holds any Vecs beyond the regridding and tries to use them for anything that cannot be done after the "rug was pulled out from under it", you'll be able to fail with a useful error message.

1 0

Re: [petsc-dev] [petsc-users] MATNEST with shell matrices
by Boyce Griffith 22 Feb '13

22 Feb '13

On 2/21/13 8:57 PM, Jed Brown wrote: > > On Thu, Feb 21, 2013 at 7:53 PM, Boyce Griffith <griffith(a)cims.nyu.edu > <mailto:[email protected]>> wrote: > > > These applications are mainly transient solvers in which regridding > happens "between" time steps. > > > > Okay, what if YourSolver(X) takes time steps? The old Vec remembers what > space it used to be part of? Sorry for the delay. The thing that "owns" the data that the Vec represents also manages regridding and interpolation between grids. So if you had something that did time stepping, the old Vec would get the rug pulled out from under it if YourSolver(X) regridded things. (It doesn't have to be this way --- you could have different AMR grids floating around --- but this is how all of the transient solvers currently work in the code.) -- Boyce

1 0

Re: [petsc-dev] [petsc-users] MATNEST with shell matrices
by Jed Brown 21 Feb '13

21 Feb '13

On Thu, Feb 21, 2013 at 7:53 PM, Boyce Griffith <griffith(a)cims.nyu.edu>wrote: > > These applications are mainly transient solvers in which regridding > happens "between" time steps. > Okay, what if YourSolver(X) takes time steps? The old Vec remembers what space it used to be part of?

1 0