February 2017 - petsc-dev - mailman3.cels.anl.gov

Re: [petsc-dev] every test example runs in a new directory with new test harness
by Barry Smith 05 Feb '17

05 Feb '17

Easier just to have the same example reload the file generated and run with that. > On Feb 5, 2017, at 9:52 PM, Satish Balay <balay(a)mcs.anl.gov> wrote: > > Probably not [yet]. one potential issue is 'nsize' is listed separately [to mpiexec].. > > Satish > > On Sun, 5 Feb 2017, Barry Smith wrote: > >> >> Does the test harness support this? If so it sounds like an ok solution >> >> >>> On Feb 5, 2017, at 9:41 PM, Satish Balay <balay(a)mcs.anl.gov> wrote: >>> >>> Perhaps the following could be converted into a single run script with >>> 2 mpiexec commands. >>> >>>>>>>>>>>> >>> test: >>> suffix: restart_0 >>> requires: hdf5 >>> args: -run_type test -refinement_limit 0.0 -bc_type dirichlet -interpolate 1 -petscspace_order 1 -dm_view hdf5:sol.h5 -vec_view hdf5:sol.h5::append >>> args: -run_type test -refinement_limit 0.0 -bc_type dirichlet -interpolate 1 -petscspace_order 1 -f sol.h5 -restart >>> <<<<<<< >>> >>> Satish >>> >>> On Sun, 5 Feb 2017, Barry Smith wrote: >>> >>>> >>>> test: >>>> suffix: restart_0 >>>> requires: hdf5 >>>> args: -run_type test -refinement_limit 0.0 -bc_type dirichlet -interpolate 1 -petscspace_order 1 -dm_view hdf5:sol.h5 -vec_view hdf5:sol.h5::append >>>> >>>> test: >>>> suffix: restart_1 >>>> requires: hdf5 >>>> args: -run_type test -refinement_limit 0.0 -bc_type dirichlet -interpolate 1 -petscspace_order 1 -f sol.h5 -restart >>>> >>>> See a problem? >>>> >>>> Should the same run of the example view the files and then load them back in? versus trying to read in a data file from another run that may not even have been created before and even if it was, the file was definitely created in a different directory? >>>> >>>> I think that if you want to do this kind of test the same example must first write the file and then read it back in. Cannot assume any relationship between different runs of tests with parallel scheduling. >>>> >>>> Barry >>>> >>>> >>> >> >> >

1 0

Re: [petsc-dev] every test example runs in a new directory with new test harness
by Satish Balay 05 Feb '17

05 Feb '17

Probably not [yet]. one potential issue is 'nsize' is listed separately [to mpiexec].. Satish On Sun, 5 Feb 2017, Barry Smith wrote: > > Does the test harness support this? If so it sounds like an ok solution > > > > On Feb 5, 2017, at 9:41 PM, Satish Balay <balay(a)mcs.anl.gov> wrote: > > > > Perhaps the following could be converted into a single run script with > > 2 mpiexec commands. > > > >>>>>>>>>> > > test: > > suffix: restart_0 > > requires: hdf5 > > args: -run_type test -refinement_limit 0.0 -bc_type dirichlet -interpolate 1 -petscspace_order 1 -dm_view hdf5:sol.h5 -vec_view hdf5:sol.h5::append > > args: -run_type test -refinement_limit 0.0 -bc_type dirichlet -interpolate 1 -petscspace_order 1 -f sol.h5 -restart > > <<<<<<< > > > > Satish > > > > On Sun, 5 Feb 2017, Barry Smith wrote: > > > >> > >> test: > >> suffix: restart_0 > >> requires: hdf5 > >> args: -run_type test -refinement_limit 0.0 -bc_type dirichlet -interpolate 1 -petscspace_order 1 -dm_view hdf5:sol.h5 -vec_view hdf5:sol.h5::append > >> > >> test: > >> suffix: restart_1 > >> requires: hdf5 > >> args: -run_type test -refinement_limit 0.0 -bc_type dirichlet -interpolate 1 -petscspace_order 1 -f sol.h5 -restart > >> > >> See a problem? > >> > >> Should the same run of the example view the files and then load them back in? versus trying to read in a data file from another run that may not even have been created before and even if it was, the file was definitely created in a different directory? > >> > >> I think that if you want to do this kind of test the same example must first write the file and then read it back in. Cannot assume any relationship between different runs of tests with parallel scheduling. > >> > >> Barry > >> > >> > > > >

1 0

Re: [petsc-dev] every test example runs in a new directory with new test harness
by Barry Smith 05 Feb '17

05 Feb '17

Does the test harness support this? If so it sounds like an ok solution > On Feb 5, 2017, at 9:41 PM, Satish Balay <balay(a)mcs.anl.gov> wrote: > > Perhaps the following could be converted into a single run script with > 2 mpiexec commands. > >>>>>>>>>> > test: > suffix: restart_0 > requires: hdf5 > args: -run_type test -refinement_limit 0.0 -bc_type dirichlet -interpolate 1 -petscspace_order 1 -dm_view hdf5:sol.h5 -vec_view hdf5:sol.h5::append > args: -run_type test -refinement_limit 0.0 -bc_type dirichlet -interpolate 1 -petscspace_order 1 -f sol.h5 -restart > <<<<<<< > > Satish > > On Sun, 5 Feb 2017, Barry Smith wrote: > >> >> test: >> suffix: restart_0 >> requires: hdf5 >> args: -run_type test -refinement_limit 0.0 -bc_type dirichlet -interpolate 1 -petscspace_order 1 -dm_view hdf5:sol.h5 -vec_view hdf5:sol.h5::append >> >> test: >> suffix: restart_1 >> requires: hdf5 >> args: -run_type test -refinement_limit 0.0 -bc_type dirichlet -interpolate 1 -petscspace_order 1 -f sol.h5 -restart >> >> See a problem? >> >> Should the same run of the example view the files and then load them back in? versus trying to read in a data file from another run that may not even have been created before and even if it was, the file was definitely created in a different directory? >> >> I think that if you want to do this kind of test the same example must first write the file and then read it back in. Cannot assume any relationship between different runs of tests with parallel scheduling. >> >> Barry >> >> >

1 0

Re: [petsc-dev] every test example runs in a new directory with new test harness
by Satish Balay 05 Feb '17

05 Feb '17

Perhaps the following could be converted into a single run script with 2 mpiexec commands. >>>>>>>>> test: suffix: restart_0 requires: hdf5 args: -run_type test -refinement_limit 0.0 -bc_type dirichlet -interpolate 1 -petscspace_order 1 -dm_view hdf5:sol.h5 -vec_view hdf5:sol.h5::append args: -run_type test -refinement_limit 0.0 -bc_type dirichlet -interpolate 1 -petscspace_order 1 -f sol.h5 -restart <<<<<<< Satish On Sun, 5 Feb 2017, Barry Smith wrote: > > test: > suffix: restart_0 > requires: hdf5 > args: -run_type test -refinement_limit 0.0 -bc_type dirichlet -interpolate 1 -petscspace_order 1 -dm_view hdf5:sol.h5 -vec_view hdf5:sol.h5::append > > test: > suffix: restart_1 > requires: hdf5 > args: -run_type test -refinement_limit 0.0 -bc_type dirichlet -interpolate 1 -petscspace_order 1 -f sol.h5 -restart > > See a problem? > > Should the same run of the example view the files and then load them back in? versus trying to read in a data file from another run that may not even have been created before and even if it was, the file was definitely created in a different directory? > > I think that if you want to do this kind of test the same example must first write the file and then read it back in. Cannot assume any relationship between different runs of tests with parallel scheduling. > > Barry > >

1 0

partitioners that use bullshit random numbers
by Barry Smith 05 Feb '17

05 Feb '17

Is there any way we can substitute a deterministic random number generators for the bullshit partitioning packages that insist on using "system" non-portable random number generators? We should have tests that use metis/parmetis and chaco or whatever but we shouldn't have to deal with their bad decisions. Matt, what horrible partitioning is being used here? Barry not ok diff-snes_tutorials-ex12_tri_bddc # 2,12c2,11 # < 0 KSP Residual norm 11.5911 # < 1 KSP Residual norm 0.419162 # < 2 KSP Residual norm 0.0466633 # < 3 KSP Residual norm 0.00282513 # < 4 KSP Residual norm 0.000140711 # < 5 KSP Residual norm 2.35332e-05 # < 6 KSP Residual norm 8.28552e-07 # < 7 KSP Residual norm 1.43643e-08 # < 8 KSP Residual norm 2.827e-10 # < Linear solve converged due to CONVERGED_RTOL iterations 8 # < 1 SNES Function norm 7.836e-10 # --- # > 0 KSP Residual norm 11.6648 # > 1 KSP Residual norm 0.443991 # > 2 KSP Residual norm 0.0393777 # > 3 KSP Residual norm 0.00187772 # > 4 KSP Residual norm 0.000140435 # > 5 KSP Residual norm 1.08843e-05 # > 6 KSP Residual norm 2.87537e-07 # > 7 KSP Residual norm 5.20043e-09 # > Linear solve converged due to CONVERGED_RTOL iterations 7 # > 1 SNES Function norm 1.59735e-08 # 18c17 # < total number of linear solver iterations=8 # --- # > total number of linear solver iterations=7 # 76,77c75,76 # < BDDC: Global dofs sizes: all 225 interface 43 coarse 7 # < BDDC: Coarsening ratios: all/coarse 32 interface/coarse 6 # --- # > BDDC: Global dofs sizes: all 225 interface 57 coarse 9 # > BDDC: Coarsening ratios: all/coarse 25 interface/coarse 6 # 81,84c80,83 # < BDDC: Interior dofs : 21 49 36 # < BDDC: Interface dofs : 15 22 17 # < BDDC: Primal dofs : 3 4 3 # < BDDC: Local dofs : 36 64 54 # --- # > BDDC: Interior dofs : 21 42 33 # > BDDC: Interface dofs : 15 36 23 # > BDDC: Primal dofs : 3 6 4 # > BDDC: Local dofs : 36 78 57 # 99,100c98,99 # < rows=7, cols=7 # < total: nonzeros=35, allocated nonzeros=43 # --- # > rows=9, cols=9 # > total: nonzeros=61, allocated nonzeros=68 # 115c114 # < factor fill ratio given 5., needed 1.17143 # --- # > factor fill ratio given 5., needed 1.13115 # 119c118 # < rows=7, cols=7 # --- # > rows=9, cols=9 # 121c120 # < total: nonzeros=41, allocated nonzeros=41 # --- # > total: nonzeros=69, allocated nonzeros=69 # 127,128c126,127 # < rows=7, cols=7 # < total: nonzeros=35, allocated nonzeros=35 # --- # > rows=9, cols=9 # > total: nonzeros=61, allocated nonzeros=61 # 130c129 # < not using I-node routines # --- # > using I-node routines: found 7 nodes, limit used is 5 # 135c134 # < total: nonzeros=1585, allocated nonzeros=1764 # --- # > total: nonzeros=1625, allocated nonzeros=1855

1 0

every test example runs in a new directory with new test harness
by Barry Smith 05 Feb '17

05 Feb '17

test: suffix: restart_0 requires: hdf5 args: -run_type test -refinement_limit 0.0 -bc_type dirichlet -interpolate 1 -petscspace_order 1 -dm_view hdf5:sol.h5 -vec_view hdf5:sol.h5::append test: suffix: restart_1 requires: hdf5 args: -run_type test -refinement_limit 0.0 -bc_type dirichlet -interpolate 1 -petscspace_order 1 -f sol.h5 -restart See a problem? Should the same run of the example view the files and then load them back in? versus trying to read in a data file from another run that may not even have been created before and even if it was, the file was definitely created in a different directory? I think that if you want to do this kind of test the same example must first write the file and then read it back in. Cannot assume any relationship between different runs of tests with parallel scheduling. Barry

1 0

confused about DMFOREST
by Barry Smith 03 Feb '17

03 Feb '17

I am confused about the organization of DMFOREST, and think it could be refactored to be clearer. It seems DMFOREST is like an abstract based class in that it can't actually do anything itself to manage a forest. Meanwhile DMP4EST and DMP8EST provide implementations of DMFOREST for 2 and 3 dimensions. There is this weird shit DMForestRegisterType(DMType); and DMIsForest(DM,PetscBool*); to try to manage the fact that though DMP4EST and DMP8EST are conceptually subclasses of DMFOREST they are actually implemented as subclasses of DM. Question: Will p4est and p8est be the only implementations of DMFOREST or is there a thought there may be other implementations of DMFOREST in the future (be realistic)? Note that p4est and p8est are really a single implementation for different dimensions. If the answer is: they will be the only implementations of DMFOREST then I think the code should be refactored so DMP4EST and DMP8EST as well as DMForestRegisterType(DMType); and DMIsForest() don't exist and then setup for DMFOREST directly sets the function pointers depending on the dimension to p4est or p8est functions. Much clearer conceptually. If the answer is: there will/should be more implementations for DMFOREST then I think the code should be refactored so that there exists a DMFORESTType, DMFORESTRegister(), DMFORESTP4EST, DMFORESTP8EST etc so that these implementations are conceptually and in the code a subclass (subtype) of DMFOREST. This would be clearer for users than the strange hybrid DMP4EST is a subclass of DMFOREST but isn't really that is currently implemented. In both cases the actually code changes that need to be made are relatively small. Am I totally missing something in my understanding? Is there a reason for this half-plant/half-animal implementation? Barry

1 0

Re: [petsc-dev] [petsc-maint] Petsc KSP options that provide the most 'robust' solve possible
by Hong 03 Feb '17

03 Feb '17

On Fri, Feb 3, 2017 at 11:44 AM, Matthew Knepley <knepley(a)gmail.com> wrote: > On Fri, Feb 3, 2017 at 11:41 AM, Barry Smith <bsmith(a)mcs.anl.gov> wrote: > >> >> > On Feb 3, 2017, at 12:59 AM, Mark McClure <mark.w.mccl(a)gmail.com> >> wrote: >> > >> > Hi, Jed and Matt. >> > >> > To close the loop, it turns out that the condition number issue had a >> simple explanation. The BC equation that I had added to the system had very >> different units from the other equations. Multiplying that row by a >> constant (effectively, modifying the units of the equation) improved the >> condition number of the matrix by many orders of magnitude. Big picture, >> this did not have an apparent effect on any of the numerical performance - >> the linear solver still had nonconvergence in the same places (when I used >> a large number of processes) and when I use a smaller number of processors >> so that the linear solve always converges, the overall numerical scheme is >> unaffected by whether or not I scale the extra BC equation. I hadn't >> realized how important it is to make sure the equations are scaled >> consistently/nondimensionalized when using the convergence number to >> evaluate whether a system is singular. Interesting experience. >> > >> > Thanks again for the help. I'll use the direct solver you suggested as >> a backup and send the user a warning if they try to use too many processors >> on a small problem. >> > >> > An aside, at first, when I saw that my overall numerical scheme was >> failing, I didn't realize the problem was that the linear solver was not >> converging. I spent a fair amount time debugging until I found the issue >> and learned how to use KSPConvergedReason. Because if KSP doesn't converge, >> it merely returns incorrect numbers. Without knowing to run >> KSPConvergedReason, an inexperienced user (like me) many not know about the >> nonconvergence in the linear solve and could spend a lot of time checking >> for other issues. It might be worth changing the default behavior for >> nonconvergence to be that it returns an nan so that the user gets a clear >> signal that the values coming out of KSP cannot be used. >> >> Because the linear solvers in PETSc are usually used by the nonlinear >> solvers and ODE integrators that have recovery methods for failure in a >> linear solve we moved away from the "crash and burn" on failed linear >> solver approach; since that makes the recovery more difficult. >> >> We could consider the following; if the KSP is not created by a SNES >> or TS it defaults to "crash and burn" on failed linear solve this would >> help newbies who are only solving linear systems and expecting a "crash and >> burn" on failure. >> > > I am for that. > I like this feature. Hong > > Matt > > >> What do people think? >> >> Barry >> >> > >> > Regards, >> > Mark >> > >> > >> > >> > On Thu, Feb 2, 2017 at 7:48 AM, Jed Brown <jed(a)jedbrown.org> wrote: >> > Mark McClure <mark.w.mccl(a)gmail.com> writes: >> > >> > > I think you are right that I have an issue with how the BC is >> implemented. >> > > It is a pipe flow simulation that is solving mass and momentum balance >> > > simultaneously (corresponding unknowns are pressure and flow rate). I >> am >> > > applying a constant mass flow rate boundary condition. Upon further >> > > consideration, it may be that I am not properly providing a boundary >> > > condition for the momentum balance equation at the inlet. If I did, >> the >> > > inlet pressure could be readily calculated standalone, >> > >> > Momentum inflow is common, but if you also have momentum outflow (i.e., >> > all Dirichlet conditions for momentum) then there is a null space of >> > constant pressure -- pressure is only determined up to a constant. >> > See the user manual section on solving singular equations. >> > >> > > outside the system of equations, and the problematic equation would be >> > > removed. >> > >> >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener >

1 0

Re: [petsc-dev] [petsc-maint] Petsc KSP options that provide the most 'robust' solve possible
by Matthew Knepley 03 Feb '17

03 Feb '17

On Fri, Feb 3, 2017 at 11:41 AM, Barry Smith <bsmith(a)mcs.anl.gov> wrote: > > > On Feb 3, 2017, at 12:59 AM, Mark McClure <mark.w.mccl(a)gmail.com> wrote: > > > > Hi, Jed and Matt. > > > > To close the loop, it turns out that the condition number issue had a > simple explanation. The BC equation that I had added to the system had very > different units from the other equations. Multiplying that row by a > constant (effectively, modifying the units of the equation) improved the > condition number of the matrix by many orders of magnitude. Big picture, > this did not have an apparent effect on any of the numerical performance - > the linear solver still had nonconvergence in the same places (when I used > a large number of processes) and when I use a smaller number of processors > so that the linear solve always converges, the overall numerical scheme is > unaffected by whether or not I scale the extra BC equation. I hadn't > realized how important it is to make sure the equations are scaled > consistently/nondimensionalized when using the convergence number to > evaluate whether a system is singular. Interesting experience. > > > > Thanks again for the help. I'll use the direct solver you suggested as a > backup and send the user a warning if they try to use too many processors > on a small problem. > > > > An aside, at first, when I saw that my overall numerical scheme was > failing, I didn't realize the problem was that the linear solver was not > converging. I spent a fair amount time debugging until I found the issue > and learned how to use KSPConvergedReason. Because if KSP doesn't converge, > it merely returns incorrect numbers. Without knowing to run > KSPConvergedReason, an inexperienced user (like me) many not know about the > nonconvergence in the linear solve and could spend a lot of time checking > for other issues. It might be worth changing the default behavior for > nonconvergence to be that it returns an nan so that the user gets a clear > signal that the values coming out of KSP cannot be used. > > Because the linear solvers in PETSc are usually used by the nonlinear > solvers and ODE integrators that have recovery methods for failure in a > linear solve we moved away from the "crash and burn" on failed linear > solver approach; since that makes the recovery more difficult. > > We could consider the following; if the KSP is not created by a SNES > or TS it defaults to "crash and burn" on failed linear solve this would > help newbies who are only solving linear systems and expecting a "crash and > burn" on failure. > I am for that. Matt > What do people think? > > Barry > > > > > Regards, > > Mark > > > > > > > > On Thu, Feb 2, 2017 at 7:48 AM, Jed Brown <jed(a)jedbrown.org> wrote: > > Mark McClure <mark.w.mccl(a)gmail.com> writes: > > > > > I think you are right that I have an issue with how the BC is > implemented. > > > It is a pipe flow simulation that is solving mass and momentum balance > > > simultaneously (corresponding unknowns are pressure and flow rate). I > am > > > applying a constant mass flow rate boundary condition. Upon further > > > consideration, it may be that I am not properly providing a boundary > > > condition for the momentum balance equation at the inlet. If I did, the > > > inlet pressure could be readily calculated standalone, > > > > Momentum inflow is common, but if you also have momentum outflow (i.e., > > all Dirichlet conditions for momentum) then there is a null space of > > constant pressure -- pressure is only determined up to a constant. > > See the user manual section on solving singular equations. > > > > > outside the system of equations, and the problematic equation would be > > > removed. > > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener

1 0

Re: [petsc-dev] [petsc-maint] Petsc KSP options that provide the most 'robust' solve possible
by Barry Smith 03 Feb '17

03 Feb '17

> On Feb 3, 2017, at 12:59 AM, Mark McClure <mark.w.mccl(a)gmail.com> wrote: > > Hi, Jed and Matt. > > To close the loop, it turns out that the condition number issue had a simple explanation. The BC equation that I had added to the system had very different units from the other equations. Multiplying that row by a constant (effectively, modifying the units of the equation) improved the condition number of the matrix by many orders of magnitude. Big picture, this did not have an apparent effect on any of the numerical performance - the linear solver still had nonconvergence in the same places (when I used a large number of processes) and when I use a smaller number of processors so that the linear solve always converges, the overall numerical scheme is unaffected by whether or not I scale the extra BC equation. I hadn't realized how important it is to make sure the equations are scaled consistently/nondimensionalized when using the convergence number to evaluate whether a system is singular. Interesting experience. > > Thanks again for the help. I'll use the direct solver you suggested as a backup and send the user a warning if they try to use too many processors on a small problem. > > An aside, at first, when I saw that my overall numerical scheme was failing, I didn't realize the problem was that the linear solver was not converging. I spent a fair amount time debugging until I found the issue and learned how to use KSPConvergedReason. Because if KSP doesn't converge, it merely returns incorrect numbers. Without knowing to run KSPConvergedReason, an inexperienced user (like me) many not know about the nonconvergence in the linear solve and could spend a lot of time checking for other issues. It might be worth changing the default behavior for nonconvergence to be that it returns an nan so that the user gets a clear signal that the values coming out of KSP cannot be used. Because the linear solvers in PETSc are usually used by the nonlinear solvers and ODE integrators that have recovery methods for failure in a linear solve we moved away from the "crash and burn" on failed linear solver approach; since that makes the recovery more difficult. We could consider the following; if the KSP is not created by a SNES or TS it defaults to "crash and burn" on failed linear solve this would help newbies who are only solving linear systems and expecting a "crash and burn" on failure. What do people think? Barry > > Regards, > Mark > > > > On Thu, Feb 2, 2017 at 7:48 AM, Jed Brown <jed(a)jedbrown.org> wrote: > Mark McClure <mark.w.mccl(a)gmail.com> writes: > > > I think you are right that I have an issue with how the BC is implemented. > > It is a pipe flow simulation that is solving mass and momentum balance > > simultaneously (corresponding unknowns are pressure and flow rate). I am > > applying a constant mass flow rate boundary condition. Upon further > > consideration, it may be that I am not properly providing a boundary > > condition for the momentum balance equation at the inlet. If I did, the > > inlet pressure could be readily calculated standalone, > > Momentum inflow is common, but if you also have momentum outflow (i.e., > all Dirichlet conditions for momentum) then there is a null space of > constant pressure -- pressure is only determined up to a constant. > See the user manual section on solving singular equations. > > > outside the system of equations, and the problematic equation would be > > removed. >

1 0