July 2018 - petsc-dev - mailman3.cels.anl.gov

Re: [petsc-dev] Bad scaling of GAMG in FieldSplit
by Jed Brown 27 Jul '18

27 Jul '18

Pierre Jolivet <pierre.jolivet(a)enseeiht.fr> writes: >> On 27 Jul 2018, at 5:48 PM, Jed Brown <jed(a)jedbrown.org> wrote: >> >> Pierre Jolivet <pierre.jolivet(a)enseeiht.fr> writes: >> >>>> On 27 Jul 2018, at 5:33 PM, Jed Brown <jed(a)jedbrown.org> wrote: >>>> >>>> Pierre Jolivet <pierre.jolivet(a)enseeiht.fr> writes: >>>> >>>>>> On 27 Jul 2018, at 5:12 PM, Jed Brown <jed(a)jedbrown.org> wrote: >>>>>> >>>>>> Pierre Jolivet <pierre.jolivet(a)enseeiht.fr> writes: >>>>>> >>>>>>> Everything is fine with GAMG I think, please find the (trimmed) -eps_view attached. The problem is that, correct me if I’m wrong, there is no easy way to redistribute data efficiently from within PETSc when using fieldsplit with unbalanced number of unknowns per field. For the other three fields, the solvers are still behaving somehow properly. Now if I’d like to optimize this some more, I’d probably need to switch from a fieldsplit to a MatNest, with submatrices from different communicators, so that I don’t have all processes handling the pressure space. But this is apparently not allowed. >>>>>> >>>>>> What if pressure is still on a global communicator, but all the degrees >>>>>> of freedom are in a subset? Then MatMult and the like have nothing to >>>>>> send or receive on the processes without any dofs. Since there are no >>>>>> reductions in PCApply (there are in setup), it should complete >>>>>> immediately for all the processes that don't have any dofs, right? >>>>> >>>>> My PC is PCKSP, with GMRES underneath, so there are reductions on the global communicator in PCApply. >>>> >>>> Why PCKSP in the pressure solve? >>> >>> I need to solve (on the pressure space) Ax = b with A^-1 = B^-1+ C^-1. >>> I do an additive PCCOMPOSITE, with two PCKSP. B is a mass matrix (PCJACOBI, no problem), C is a shifted Laplacian (PCGAMG, problem…). >> >> You need C^{-1} to be applied accurately, not just a V-cycle? > > Right, rtol of 10^-3 for both solves, otherwise the EPS needs more iterations to converge. You don't enforce the accuracy in the outer Krylov method? >> We could add support for subcommunicators when dof partitions are >> possible, but it make profiling more difficult to interpret and may have >> unexpected consequences if not done in a tightly controlled way. >> >> >> Barry, should we change PCCompositeAddPC to use a KSP (or otherwise) so >> that KSPSolve is called like all other composed solvers? > > (I’m not sure my usage of PCComposite is the most common so I’m perfectly fine with wrapping everything around PCKSPs) My point is that all other methods of solver composition (ASM, Fieldsplit, Galerkin, MG) call KSPSolve, possibly configured by default as PREONLY, which is just a bit simpler to work with.

1 0

Re: [petsc-dev] Bad scaling of GAMG in FieldSplit
by Pierre Jolivet 27 Jul '18

27 Jul '18

> On 27 Jul 2018, at 5:48 PM, Jed Brown <jed(a)jedbrown.org> wrote: > > Pierre Jolivet <pierre.jolivet(a)enseeiht.fr> writes: > >>> On 27 Jul 2018, at 5:33 PM, Jed Brown <jed(a)jedbrown.org> wrote: >>> >>> Pierre Jolivet <pierre.jolivet(a)enseeiht.fr> writes: >>> >>>>> On 27 Jul 2018, at 5:12 PM, Jed Brown <jed(a)jedbrown.org> wrote: >>>>> >>>>> Pierre Jolivet <pierre.jolivet(a)enseeiht.fr> writes: >>>>> >>>>>> Everything is fine with GAMG I think, please find the (trimmed) -eps_view attached. The problem is that, correct me if I’m wrong, there is no easy way to redistribute data efficiently from within PETSc when using fieldsplit with unbalanced number of unknowns per field. For the other three fields, the solvers are still behaving somehow properly. Now if I’d like to optimize this some more, I’d probably need to switch from a fieldsplit to a MatNest, with submatrices from different communicators, so that I don’t have all processes handling the pressure space. But this is apparently not allowed. >>>>> >>>>> What if pressure is still on a global communicator, but all the degrees >>>>> of freedom are in a subset? Then MatMult and the like have nothing to >>>>> send or receive on the processes without any dofs. Since there are no >>>>> reductions in PCApply (there are in setup), it should complete >>>>> immediately for all the processes that don't have any dofs, right? >>>> >>>> My PC is PCKSP, with GMRES underneath, so there are reductions on the global communicator in PCApply. >>> >>> Why PCKSP in the pressure solve? >> >> I need to solve (on the pressure space) Ax = b with A^-1 = B^-1+ C^-1. >> I do an additive PCCOMPOSITE, with two PCKSP. B is a mass matrix (PCJACOBI, no problem), C is a shifted Laplacian (PCGAMG, problem…). > > You need C^{-1} to be applied accurately, not just a V-cycle? Right, rtol of 10^-3 for both solves, otherwise the EPS needs more iterations to converge. > We could add support for subcommunicators when dof partitions are > possible, but it make profiling more difficult to interpret and may have > unexpected consequences if not done in a tightly controlled way. > > > Barry, should we change PCCompositeAddPC to use a KSP (or otherwise) so > that KSPSolve is called like all other composed solvers? (I’m not sure my usage of PCComposite is the most common so I’m perfectly fine with wrapping everything around PCKSPs)

1 0

Re: [petsc-dev] Bad scaling of GAMG in FieldSplit
by Jed Brown 27 Jul '18

27 Jul '18

Pierre Jolivet <pierre.jolivet(a)enseeiht.fr> writes: >> On 27 Jul 2018, at 5:33 PM, Jed Brown <jed(a)jedbrown.org> wrote: >> >> Pierre Jolivet <pierre.jolivet(a)enseeiht.fr> writes: >> >>>> On 27 Jul 2018, at 5:12 PM, Jed Brown <jed(a)jedbrown.org> wrote: >>>> >>>> Pierre Jolivet <pierre.jolivet(a)enseeiht.fr> writes: >>>> >>>>> Everything is fine with GAMG I think, please find the (trimmed) -eps_view attached. The problem is that, correct me if I’m wrong, there is no easy way to redistribute data efficiently from within PETSc when using fieldsplit with unbalanced number of unknowns per field. For the other three fields, the solvers are still behaving somehow properly. Now if I’d like to optimize this some more, I’d probably need to switch from a fieldsplit to a MatNest, with submatrices from different communicators, so that I don’t have all processes handling the pressure space. But this is apparently not allowed. >>>> >>>> What if pressure is still on a global communicator, but all the degrees >>>> of freedom are in a subset? Then MatMult and the like have nothing to >>>> send or receive on the processes without any dofs. Since there are no >>>> reductions in PCApply (there are in setup), it should complete >>>> immediately for all the processes that don't have any dofs, right? >>> >>> My PC is PCKSP, with GMRES underneath, so there are reductions on the global communicator in PCApply. >> >> Why PCKSP in the pressure solve? > > I need to solve (on the pressure space) Ax = b with A^-1 = B^-1+ C^-1. > I do an additive PCCOMPOSITE, with two PCKSP. B is a mass matrix (PCJACOBI, no problem), C is a shifted Laplacian (PCGAMG, problem…). You need C^{-1} to be applied accurately, not just a V-cycle? We could add support for subcommunicators when dof partitions are possible, but it make profiling more difficult to interpret and may have unexpected consequences if not done in a tightly controlled way. Barry, should we change PCCompositeAddPC to use a KSP (or otherwise) so that KSPSolve is called like all other composed solvers?

1 0

Re: [petsc-dev] Bad scaling of GAMG in FieldSplit
by Pierre Jolivet 27 Jul '18

27 Jul '18

> On 27 Jul 2018, at 5:33 PM, Jed Brown <jed(a)jedbrown.org> wrote: > > Pierre Jolivet <pierre.jolivet(a)enseeiht.fr> writes: > >>> On 27 Jul 2018, at 5:12 PM, Jed Brown <jed(a)jedbrown.org> wrote: >>> >>> Pierre Jolivet <pierre.jolivet(a)enseeiht.fr> writes: >>> >>>> Everything is fine with GAMG I think, please find the (trimmed) -eps_view attached. The problem is that, correct me if I’m wrong, there is no easy way to redistribute data efficiently from within PETSc when using fieldsplit with unbalanced number of unknowns per field. For the other three fields, the solvers are still behaving somehow properly. Now if I’d like to optimize this some more, I’d probably need to switch from a fieldsplit to a MatNest, with submatrices from different communicators, so that I don’t have all processes handling the pressure space. But this is apparently not allowed. >>> >>> What if pressure is still on a global communicator, but all the degrees >>> of freedom are in a subset? Then MatMult and the like have nothing to >>> send or receive on the processes without any dofs. Since there are no >>> reductions in PCApply (there are in setup), it should complete >>> immediately for all the processes that don't have any dofs, right? >> >> My PC is PCKSP, with GMRES underneath, so there are reductions on the global communicator in PCApply. > > Why PCKSP in the pressure solve? I need to solve (on the pressure space) Ax = b with A^-1 = B^-1+ C^-1. I do an additive PCCOMPOSITE, with two PCKSP. B is a mass matrix (PCJACOBI, no problem), C is a shifted Laplacian (PCGAMG, problem…).

1 0

Re: [petsc-dev] Bad scaling of GAMG in FieldSplit
by Jed Brown 27 Jul '18

27 Jul '18

Pierre Jolivet <pierre.jolivet(a)enseeiht.fr> writes: >> On 27 Jul 2018, at 5:12 PM, Jed Brown <jed(a)jedbrown.org> wrote: >> >> Pierre Jolivet <pierre.jolivet(a)enseeiht.fr> writes: >> >>> Everything is fine with GAMG I think, please find the (trimmed) -eps_view attached. The problem is that, correct me if I’m wrong, there is no easy way to redistribute data efficiently from within PETSc when using fieldsplit with unbalanced number of unknowns per field. For the other three fields, the solvers are still behaving somehow properly. Now if I’d like to optimize this some more, I’d probably need to switch from a fieldsplit to a MatNest, with submatrices from different communicators, so that I don’t have all processes handling the pressure space. But this is apparently not allowed. >> >> What if pressure is still on a global communicator, but all the degrees >> of freedom are in a subset? Then MatMult and the like have nothing to >> send or receive on the processes without any dofs. Since there are no >> reductions in PCApply (there are in setup), it should complete >> immediately for all the processes that don't have any dofs, right? > > My PC is PCKSP, with GMRES underneath, so there are reductions on the global communicator in PCApply. Why PCKSP in the pressure solve?

1 0

Re: [petsc-dev] Bad scaling of GAMG in FieldSplit
by Pierre Jolivet 27 Jul '18

27 Jul '18

> On 27 Jul 2018, at 5:12 PM, Jed Brown <jed(a)jedbrown.org> wrote: > > Pierre Jolivet <pierre.jolivet(a)enseeiht.fr> writes: > >> Everything is fine with GAMG I think, please find the (trimmed) -eps_view attached. The problem is that, correct me if I’m wrong, there is no easy way to redistribute data efficiently from within PETSc when using fieldsplit with unbalanced number of unknowns per field. For the other three fields, the solvers are still behaving somehow properly. Now if I’d like to optimize this some more, I’d probably need to switch from a fieldsplit to a MatNest, with submatrices from different communicators, so that I don’t have all processes handling the pressure space. But this is apparently not allowed. > > What if pressure is still on a global communicator, but all the degrees > of freedom are in a subset? Then MatMult and the like have nothing to > send or receive on the processes without any dofs. Since there are no > reductions in PCApply (there are in setup), it should complete > immediately for all the processes that don't have any dofs, right? My PC is PCKSP, with GMRES underneath, so there are reductions on the global communicator in PCApply.

1 0

Re: [petsc-dev] Bad scaling of GAMG in FieldSplit
by Mark Adams 27 Jul '18

27 Jul '18

> > Everything is fine with GAMG I think, please find the (trimmed) -eps_view > attached. > This looks fine. The eigen estimates are pretty low, but I don't know what they really are. > The problem is that, correct me if I’m wrong, there is no easy way to > redistribute data efficiently from within PETSc when using fieldsplit with > unbalanced number of unknowns per field. > Note, as you scale up the coarse grids become more important re complexity. So the process reduction and potential repartitioning will become more noticable. At extreme scale you can spend a majority of the time in the coarse grids. > For the other three fields, the solvers are still behaving somehow > properly. Now if I’d like to optimize this some more, I’d probably need to > switch from a fieldsplit to a MatNest, with submatrices from different > communicators, so that I don’t have all processes handling the pressure > space. But this is apparently not allowed. > > Thanks, > Pierre > > > > >> >> 2) have the sub_0_ and the sub_1_ work on two different nonoverlapping >>> communicators of size PETSC_COMM_WORLD/2, do the solve concurrently, and >>> then sum the solutions (only worth doing because of -pc_composite_type >>> additive). I have no idea if this easily doable with PETSc command line >>> arguments >>> > >>> > 1) is the more flexible approach, as you have better control over the >>> system sizes after 'telescoping’. >>> >>> Right, but the advantage of 2) is that I wouldn't have one half or more >>> of processes idling and I could overlap the solves of both subpc in the >>> PCCOMPOSITE. >>> >>> I’m attaching the -log_view for both runs (I trimmed some options). >>> >>> Thanks for your help, >>> Pierre >>> >>> >>> > Best regards, >>> > Karli >> >> >> >

1 0

Re: [petsc-dev] Bad scaling of GAMG in FieldSplit
by Mark Adams 27 Jul '18

27 Jul '18

On Fri, Jul 27, 2018 at 11:12 AM Jed Brown <jed(a)jedbrown.org> wrote: > Pierre Jolivet <pierre.jolivet(a)enseeiht.fr> writes: > > > Everything is fine with GAMG I think, please find the (trimmed) > -eps_view attached. The problem is that, correct me if I’m wrong, there is > no easy way to redistribute data efficiently from within PETSc when using > fieldsplit with unbalanced number of unknowns per field. For the other > three fields, the solvers are still behaving somehow properly. Now if I’d > like to optimize this some more, I’d probably need to switch from a > fieldsplit to a MatNest, with submatrices from different communicators, so > that I don’t have all processes handling the pressure space. But this is > apparently not allowed. > > What if pressure is still on a global communicator, but all the degrees > of freedom are in a subset? Then MatMult and the like have nothing to > send or receive on the processes without any dofs. Since there are no > reductions in PCApply (there are in setup), it should complete > immediately for all the processes that don't have any dofs, right? > Yes, idle processes should just fall through.

1 0

Re: [petsc-dev] Bad scaling of GAMG in FieldSplit
by Mark Adams 27 Jul '18

27 Jul '18

> > > What does Telescope do on the fine grids? Is this a structured grid and > Telescope does geometric MG? > > > No, it’s a MatMPIAIJ. Telescope just concatenates local matrices together, > makes some process idle (throughout the complete setup + solve), while the > others do the work. > So you are reducing processors on the fine grid. You are way into strong scaling. GAMG has a parameter for how many equations/proc to aim for in process reduction. So the 2nd grid will be reduced if you like. It can also repartition these grids because concatenation is not likely to be very good.

1 0

Re: [petsc-dev] Bad scaling of GAMG in FieldSplit
by Jed Brown 27 Jul '18

27 Jul '18

Pierre Jolivet <pierre.jolivet(a)enseeiht.fr> writes: > Everything is fine with GAMG I think, please find the (trimmed) -eps_view attached. The problem is that, correct me if I’m wrong, there is no easy way to redistribute data efficiently from within PETSc when using fieldsplit with unbalanced number of unknowns per field. For the other three fields, the solvers are still behaving somehow properly. Now if I’d like to optimize this some more, I’d probably need to switch from a fieldsplit to a MatNest, with submatrices from different communicators, so that I don’t have all processes handling the pressure space. But this is apparently not allowed. What if pressure is still on a global communicator, but all the degrees of freedom are in a subset? Then MatMult and the like have nothing to send or receive on the processes without any dofs. Since there are no reductions in PCApply (there are in setup), it should complete immediately for all the processes that don't have any dofs, right?

1 0