June 2016 - petsc-dev - mailman3.cels.anl.gov

Re: [petsc-dev] WTF
by Jeff Hammond 30 Jun '16

30 Jun '16

> The default settings often don’t work well for 3D problems. Are 2D (1D?) problems really the common case for PDE solvers? Aren't interesting problems 3D? Shouldn't the defaults be set to optimize for 3D? Jeff -- Jeff Hammond jeff.science(a)gmail.com http://jeffhammond.github.io/

1 0

Re: [petsc-dev] WTF
by Yang, Ulrike Meier 30 Jun '16

30 Jun '16

Hi Barry, It would not be possible to detect whether this is 3D, but we could add a function like hypre_BoomerAMGSet3D or so which would change the default settings or we could change the default settings (I have been hesitant to do so, since the current settings still seem to work better for 2D, but if there are more 3D users out there by now and the 3D settings are not significantly worse for 2D it might make sense to do so). Ulrike -----Original Message----- From: Barry Smith [mailto:[email protected]] Sent: Thursday, June 30, 2016 9:59 AM To: Yang, Ulrike Meier <yang11(a)llnl.gov> Cc: Justin Chang <jychang48(a)gmail.com>; Jeff Hammond <jeff.science(a)gmail.com>; PETSc <petsc-dev(a)mcs.anl.gov>; Falgout Rob (rfalgout(a)llnl.gov) <rfalgout(a)llnl.gov> Subject: Re: [petsc-dev] WTF > On Jun 30, 2016, at 9:55 AM, Yang, Ulrike Meier <yang11(a)llnl.gov> wrote: > > Do you know which settings were used for BoomerAMG? > The default settings often don’t work well for 3D problems. Ulrike, Would it be possible for BoomerAMG to detect the dimension, or use dimension information provided by the user to adjust the default settings automatically without the user needing to adjust them? The problem is that too many people can run their code with the default, get bad performance and not realize that it could be (much) better so they either just live with the poorer than necessary performance, or worse, publish results with the poor performance. Barry > Ulrike > > From: Justin Chang [mailto:[email protected]] > Sent: Thursday, June 30, 2016 12:40 AM > To: Barry Smith <bsmith(a)mcs.anl.gov> > Cc: Jeff Hammond <jeff.science(a)gmail.com>; Yang, Ulrike Meier <yang11(a)llnl.gov>; PETSc <petsc-dev(a)mcs.anl.gov>; Falgout Rob (rfalgout(a)llnl.gov) <rfalgout(a)llnl.gov> > Subject: Re: [petsc-dev] WTF > > That guy's results actually make sense to me. > > I also get poor strong-scaling for the FEM version of the poisson equation (via firedrake) using HYPRE's boomerAMG. The studies were done on Intel E5-2670 machines and had proper OpenMPI bindings.No HYPRE configure options were set via command line so I just used whatever the default setting were. > > If he used ML, GAMG, or even ILU he would likely get much better scaling as I have. > > Attached is a speedup plot of a much smaller problem I did (225k dofs), but you can still see a similar progression on how HYPRE deteriorates. > > Compared to the other preconditioners, I noticed that HYPRE has a much lower flop-to-byte ratio which suggests to me that based on the current solver configurations, HYPRE is likely going to be more memory-bandwidth and suffer from lack of memory usage as more cores are used. > > Not sure how to properly configure any of these multigrid preconditioners, but figured I'd offer my two cents. > > Justin > > On Thu, Jun 30, 2016 at 5:18 AM, Barry Smith <bsmith(a)mcs.anl.gov> wrote: > > > On Jun 29, 2016, at 10:06 PM, Jeff Hammond <jeff.science(a)gmail.com> wrote: > > > > > > > > On Wednesday, June 29, 2016, Barry Smith <bsmith(a)mcs.anl.gov> wrote: > > > > Who are these people and why to they have this webpage? > > > > > > Pop up 2-3 directories and you'll see this is a grad student who appears to be trying to learn applied math. Is this really your enemy? Don't you guys have some DOE bigwigs to bash? > > > > Almost for sure they are doing no process binding and no proper assignment of processes to memory domains. > > > > > > MVAPICH2 sets affinity by default. Details not given but "infiniband enabled" means it might have been used. I don't know what OpenMPI does by default but affinity alone doesn't explain this. > > By affinity you mean that the process just remains on the same core right? You could be right I think the main affect is a bad assignment of processes to cores/memory domains. > > > > > In addition they are likely filling up all the cores on the first node before adding processes to the second core etc. > > > > > > That's how I would show scaling. Are you suggesting using all the nodes and doing breadth first placement? > > I would fill up one process per memory domain moving across the nodes; then go back and start a second process on each memory domain. etc You can also just go across nodes as you suggest and then across memory domains > > If you fill up the entire node of cores and then go to the next node you get this affect that the performance goes way down as you fill up the last of the cores (because no more memory bandwidth is available) and then performance goes up again as you jump to the next node and suddenly have a big chunk of additional bandwidth. You also have weird load balancing problem because the first 16 processes are going slow because they share some bandwidth while the 17 runs much faster since it can hog more bandwidth. > > > > > Jeff > > > > If the studies had been done properly there should be very little fail off on the strong scaling in going from 1 to 2 to 4 processes and even beyond. Similarly the huge fail off in going from 4 to 8 to 16 would not occur for weak scaling. > > > > Barry > > > > > > > On Jun 29, 2016, at 7:47 PM, Matthew Knepley <knepley(a)gmail.com> wrote: > > > > > > > > > > > > http://guest.ams.sunysb.edu/~zgao/work/airfoil/scaling.html > > > > > > Can we rerun this on something at ANL since I think this cannot be true. > > > > > > Matt > > > > > > -- > > > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > > > -- Norbert Wiener > > > > > > > > -- > > Jeff Hammond > > jeff.science(a)gmail.com > > http://jeffhammond.github.io/ >

1 0

Re: [petsc-dev] WTF
by Barry Smith 30 Jun '16

30 Jun '16

> On Jun 30, 2016, at 9:55 AM, Yang, Ulrike Meier <yang11(a)llnl.gov> wrote: > > Do you know which settings were used for BoomerAMG? > The default settings often don’t work well for 3D problems. Ulrike, Would it be possible for BoomerAMG to detect the dimension, or use dimension information provided by the user to adjust the default settings automatically without the user needing to adjust them? The problem is that too many people can run their code with the default, get bad performance and not realize that it could be (much) better so they either just live with the poorer than necessary performance, or worse, publish results with the poor performance. Barry > Ulrike > > From: Justin Chang [mailto:[email protected]] > Sent: Thursday, June 30, 2016 12:40 AM > To: Barry Smith <bsmith(a)mcs.anl.gov> > Cc: Jeff Hammond <jeff.science(a)gmail.com>; Yang, Ulrike Meier <yang11(a)llnl.gov>; PETSc <petsc-dev(a)mcs.anl.gov>; Falgout Rob (rfalgout(a)llnl.gov) <rfalgout(a)llnl.gov> > Subject: Re: [petsc-dev] WTF > > That guy's results actually make sense to me. > > I also get poor strong-scaling for the FEM version of the poisson equation (via firedrake) using HYPRE's boomerAMG. The studies were done on Intel E5-2670 machines and had proper OpenMPI bindings.No HYPRE configure options were set via command line so I just used whatever the default setting were. > > If he used ML, GAMG, or even ILU he would likely get much better scaling as I have. > > Attached is a speedup plot of a much smaller problem I did (225k dofs), but you can still see a similar progression on how HYPRE deteriorates. > > Compared to the other preconditioners, I noticed that HYPRE has a much lower flop-to-byte ratio which suggests to me that based on the current solver configurations, HYPRE is likely going to be more memory-bandwidth and suffer from lack of memory usage as more cores are used. > > Not sure how to properly configure any of these multigrid preconditioners, but figured I'd offer my two cents. > > Justin > > On Thu, Jun 30, 2016 at 5:18 AM, Barry Smith <bsmith(a)mcs.anl.gov> wrote: > > > On Jun 29, 2016, at 10:06 PM, Jeff Hammond <jeff.science(a)gmail.com> wrote: > > > > > > > > On Wednesday, June 29, 2016, Barry Smith <bsmith(a)mcs.anl.gov> wrote: > > > > Who are these people and why to they have this webpage? > > > > > > Pop up 2-3 directories and you'll see this is a grad student who appears to be trying to learn applied math. Is this really your enemy? Don't you guys have some DOE bigwigs to bash? > > > > Almost for sure they are doing no process binding and no proper assignment of processes to memory domains. > > > > > > MVAPICH2 sets affinity by default. Details not given but "infiniband enabled" means it might have been used. I don't know what OpenMPI does by default but affinity alone doesn't explain this. > > By affinity you mean that the process just remains on the same core right? You could be right I think the main affect is a bad assignment of processes to cores/memory domains. > > > > > In addition they are likely filling up all the cores on the first node before adding processes to the second core etc. > > > > > > That's how I would show scaling. Are you suggesting using all the nodes and doing breadth first placement? > > I would fill up one process per memory domain moving across the nodes; then go back and start a second process on each memory domain. etc You can also just go across nodes as you suggest and then across memory domains > > If you fill up the entire node of cores and then go to the next node you get this affect that the performance goes way down as you fill up the last of the cores (because no more memory bandwidth is available) and then performance goes up again as you jump to the next node and suddenly have a big chunk of additional bandwidth. You also have weird load balancing problem because the first 16 processes are going slow because they share some bandwidth while the 17 runs much faster since it can hog more bandwidth. > > > > > Jeff > > > > If the studies had been done properly there should be very little fail off on the strong scaling in going from 1 to 2 to 4 processes and even beyond. Similarly the huge fail off in going from 4 to 8 to 16 would not occur for weak scaling. > > > > Barry > > > > > > > On Jun 29, 2016, at 7:47 PM, Matthew Knepley <knepley(a)gmail.com> wrote: > > > > > > > > > > > > http://guest.ams.sunysb.edu/~zgao/work/airfoil/scaling.html > > > > > > Can we rerun this on something at ANL since I think this cannot be true. > > > > > > Matt > > > > > > -- > > > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > > > -- Norbert Wiener > > > > > > > > -- > > Jeff Hammond > > jeff.science(a)gmail.com > > http://jeffhammond.github.io/ >

1 0

Re: [petsc-dev] WTF
by Matthew Knepley 30 Jun '16

30 Jun '16

On Thu, Jun 30, 2016 at 4:55 PM, Yang, Ulrike Meier <yang11(a)llnl.gov> wrote: > Do you know which settings were used for BoomerAMG? > > The default settings often don’t work well for 3D problems. > > Ulrike, the default setting were not coarsening aggressively enough. The FireDrake guys used these which seem to work well: -ksp_type cg -pc_type hypre -pc_hypre_type boomeramg -pc_hypre_boomeramg_strong_threshold 0.75 -pc_hypre_boomeramg_agg_nl 2 -ksp_rtol 1e-6 I sent that group these options to try. Thanks, Matt > Ulrike > > > > *From:* Justin Chang [mailto:[email protected]] > *Sent:* Thursday, June 30, 2016 12:40 AM > *To:* Barry Smith <bsmith(a)mcs.anl.gov> > *Cc:* Jeff Hammond <jeff.science(a)gmail.com>; Yang, Ulrike Meier < > yang11(a)llnl.gov>; PETSc <petsc-dev(a)mcs.anl.gov>; Falgout Rob ( > rfalgout(a)llnl.gov) <rfalgout(a)llnl.gov> > *Subject:* Re: [petsc-dev] WTF > > > > That guy's results actually make sense to me. > > > > I also get poor strong-scaling for the FEM version of the poisson equation > (via firedrake) using HYPRE's boomerAMG. The studies were done on Intel > E5-2670 machines and had proper OpenMPI bindings.No HYPRE configure options > were set via command line so I just used whatever the default setting were. > > > > If he used ML, GAMG, or even ILU he would likely get much better scaling > as I have. > > > > Attached is a speedup plot of a much smaller problem I did (225k dofs), > but you can still see a similar progression on how HYPRE deteriorates. > > > > Compared to the other preconditioners, I noticed that HYPRE has a much > lower flop-to-byte ratio which suggests to me that based on the current > solver configurations, HYPRE is likely going to be more memory-bandwidth > and suffer from lack of memory usage as more cores are used. > > > > Not sure how to properly configure any of these multigrid preconditioners, > but figured I'd offer my two cents. > > > > Justin > > > > On Thu, Jun 30, 2016 at 5:18 AM, Barry Smith <bsmith(a)mcs.anl.gov> wrote: > > > > On Jun 29, 2016, at 10:06 PM, Jeff Hammond <jeff.science(a)gmail.com> > wrote: > > > > > > > > On Wednesday, June 29, 2016, Barry Smith <bsmith(a)mcs.anl.gov> wrote: > > > > Who are these people and why to they have this webpage? > > > > > > Pop up 2-3 directories and you'll see this is a grad student who appears > to be trying to learn applied math. Is this really your enemy? Don't you > guys have some DOE bigwigs to bash? > > > > Almost for sure they are doing no process binding and no proper > assignment of processes to memory domains. > > > > > > MVAPICH2 sets affinity by default. Details not given but "infiniband > enabled" means it might have been used. I don't know what OpenMPI does by > default but affinity alone doesn't explain this. > > By affinity you mean that the process just remains on the same core > right? You could be right I think the main affect is a bad assignment of > processes to cores/memory domains. > > > > > In addition they are likely filling up all the cores on the first node > before adding processes to the second core etc. > > > > > > That's how I would show scaling. Are you suggesting using all the nodes > and doing breadth first placement? > > I would fill up one process per memory domain moving across the nodes; > then go back and start a second process on each memory domain. etc You can > also just go across nodes as you suggest and then across memory domains > > If you fill up the entire node of cores and then go to the next node > you get this affect that the performance goes way down as you fill up the > last of the cores (because no more memory bandwidth is available) and then > performance goes up again as you jump to the next node and suddenly have a > big chunk of additional bandwidth. You also have weird load balancing > problem because the first 16 processes are going slow because they share > some bandwidth while the 17 runs much faster since it can hog more > bandwidth. > > > > > > Jeff > > > > If the studies had been done properly there should be very little fail > off on the strong scaling in going from 1 to 2 to 4 processes and even > beyond. Similarly the huge fail off in going from 4 to 8 to 16 would not > occur for weak scaling. > > > > Barry > > > > > > > On Jun 29, 2016, at 7:47 PM, Matthew Knepley <knepley(a)gmail.com> > wrote: > > > > > > > > > > > > http://guest.ams.sunysb.edu/~zgao/work/airfoil/scaling.html > > > > > > Can we rerun this on something at ANL since I think this cannot be > true. > > > > > > Matt > > > > > > -- > > > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > > > -- Norbert Wiener > > > > > > > > -- > > Jeff Hammond > > jeff.science(a)gmail.com > > http://jeffhammond.github.io/ > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener

1 0

Re: [petsc-dev] WTF
by Yang, Ulrike Meier 30 Jun '16

30 Jun '16

Do you know which settings were used for BoomerAMG? The default settings often don’t work well for 3D problems. Ulrike From: Justin Chang [mailto:[email protected]] Sent: Thursday, June 30, 2016 12:40 AM To: Barry Smith <bsmith(a)mcs.anl.gov> Cc: Jeff Hammond <jeff.science(a)gmail.com>; Yang, Ulrike Meier <yang11(a)llnl.gov>; PETSc <petsc-dev(a)mcs.anl.gov>; Falgout Rob (rfalgout(a)llnl.gov) <rfalgout(a)llnl.gov> Subject: Re: [petsc-dev] WTF That guy's results actually make sense to me. I also get poor strong-scaling for the FEM version of the poisson equation (via firedrake) using HYPRE's boomerAMG. The studies were done on Intel E5-2670 machines and had proper OpenMPI bindings.No HYPRE configure options were set via command line so I just used whatever the default setting were. If he used ML, GAMG, or even ILU he would likely get much better scaling as I have. Attached is a speedup plot of a much smaller problem I did (225k dofs), but you can still see a similar progression on how HYPRE deteriorates. Compared to the other preconditioners, I noticed that HYPRE has a much lower flop-to-byte ratio which suggests to me that based on the current solver configurations, HYPRE is likely going to be more memory-bandwidth and suffer from lack of memory usage as more cores are used. Not sure how to properly configure any of these multigrid preconditioners, but figured I'd offer my two cents. Justin On Thu, Jun 30, 2016 at 5:18 AM, Barry Smith <bsmith(a)mcs.anl.gov<mailto:[email protected]>> wrote: > On Jun 29, 2016, at 10:06 PM, Jeff Hammond <jeff.science(a)gmail.com<mailto:[email protected]>> wrote: > > > > On Wednesday, June 29, 2016, Barry Smith <bsmith(a)mcs.anl.gov<mailto:[email protected]>> wrote: > > Who are these people and why to they have this webpage? > > > Pop up 2-3 directories and you'll see this is a grad student who appears to be trying to learn applied math. Is this really your enemy? Don't you guys have some DOE bigwigs to bash? > > Almost for sure they are doing no process binding and no proper assignment of processes to memory domains. > > > MVAPICH2 sets affinity by default. Details not given but "infiniband enabled" means it might have been used. I don't know what OpenMPI does by default but affinity alone doesn't explain this. By affinity you mean that the process just remains on the same core right? You could be right I think the main affect is a bad assignment of processes to cores/memory domains. > > In addition they are likely filling up all the cores on the first node before adding processes to the second core etc. > > > That's how I would show scaling. Are you suggesting using all the nodes and doing breadth first placement? I would fill up one process per memory domain moving across the nodes; then go back and start a second process on each memory domain. etc You can also just go across nodes as you suggest and then across memory domains If you fill up the entire node of cores and then go to the next node you get this affect that the performance goes way down as you fill up the last of the cores (because no more memory bandwidth is available) and then performance goes up again as you jump to the next node and suddenly have a big chunk of additional bandwidth. You also have weird load balancing problem because the first 16 processes are going slow because they share some bandwidth while the 17 runs much faster since it can hog more bandwidth. > > Jeff > > If the studies had been done properly there should be very little fail off on the strong scaling in going from 1 to 2 to 4 processes and even beyond. Similarly the huge fail off in going from 4 to 8 to 16 would not occur for weak scaling. > > Barry > > > > On Jun 29, 2016, at 7:47 PM, Matthew Knepley <knepley(a)gmail.com<mailto:[email protected]>> wrote: > > > > > > > > http://guest.ams.sunysb.edu/~zgao/work/airfoil/scaling.html > > > > Can we rerun this on something at ANL since I think this cannot be true. > > > > Matt > > > > -- > > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > > -- Norbert Wiener > > > > -- > Jeff Hammond > jeff.science(a)gmail.com<mailto:[email protected]> > http://jeffhammond.github.io/

1 0

Re: [petsc-dev] WTF
by Jeff Hammond 30 Jun '16

30 Jun '16

On Wed, Jun 29, 2016 at 8:18 PM, Barry Smith <bsmith(a)mcs.anl.gov> wrote: > > > On Jun 29, 2016, at 10:06 PM, Jeff Hammond <jeff.science(a)gmail.com> > wrote: > > > > > > > > On Wednesday, June 29, 2016, Barry Smith <bsmith(a)mcs.anl.gov> wrote: > > > > Who are these people and why to they have this webpage? > > > > > > Pop up 2-3 directories and you'll see this is a grad student who appears > to be trying to learn applied math. Is this really your enemy? Don't you > guys have some DOE bigwigs to bash? > > > > Almost for sure they are doing no process binding and no proper > assignment of processes to memory domains. > > > > > > MVAPICH2 sets affinity by default. Details not given but "infiniband > enabled" means it might have been used. I don't know what OpenMPI does by > default but affinity alone doesn't explain this. > > By affinity you mean that the process just remains on the same core > right? You could be right I think the main affect is a bad assignment of > processes to cores/memory domains. > > Yes, affinity to cores. I checked and: - Open-MPI does no binding by default ( https://www.open-mpi.org/faq/?category=tuning#using-paffinity-v1.4) - MVAPICH2 sets affinity by default except when MPI_THREAD_MULTIPLE is used ( http://mvapich.cse.ohio-state.edu/static/media/mvapich/mvapich2-2.0-usergui… ). - I am not certain what Intel MPI does in every case, but at least on Xeon Phi it defaults to compact placement ( https://software.intel.com/en-us/articles/mpi-and-process-pinning-on-xeon-p…) which is almost certainly wrong for bandwidth-limited apps (where scatter makes more sense). > > > > In addition they are likely filling up all the cores on the first node > before adding processes to the second core etc. > > > > > > That's how I would show scaling. Are you suggesting using all the nodes > and doing breadth first placement? > > I would fill up one process per memory domain moving across the nodes; > then go back and start a second process on each memory domain. etc You can > also just go across nodes as you suggest and then across memory domains > > That's reasonable. I just don't bother showing scaling except in the unit of charge, which in most cases is nodes (exception: Blue Gene). There is no way to decompose node resources in a completely reliable way, so one should always use the full node as effectively as possible for every node count. The other exception is the cloud, there hypervisors are presumably doing a halfway decent job of dividing up resources (and adding enough overhead that performance is irrelevant anyways :-) ) and one can plot scaling in the number of (virtual) cores. > If you fill up the entire node of cores and then go to the next node > you get this affect that the performance goes way down as you fill up the > last of the cores (because no more memory bandwidth is available) and then > performance goes up again as you jump to the next node and suddenly have a > big chunk of additional bandwidth. You also have weird load balancing > problem because the first 16 processes are going slow because they share > some bandwidth while the 17 runs much faster since it can hog more > bandwidth. > > Indeed, 17 on 2 should be distributed as 9 and 8, not 16 and 1, although using nproc%nnode!=0 is silly. I thought you meant scaling up to 20 with 1 ppn on 20 nodes, then going to 40 with 2 ppn, etc. Jeff > > > > Jeff > > > > If the studies had been done properly there should be very little fail > off on the strong scaling in going from 1 to 2 to 4 processes and even > beyond. Similarly the huge fail off in going from 4 to 8 to 16 would not > occur for weak scaling. > > > > Barry > > > > > > > On Jun 29, 2016, at 7:47 PM, Matthew Knepley <knepley(a)gmail.com> > wrote: > > > > > > > > > > > > http://guest.ams.sunysb.edu/~zgao/work/airfoil/scaling.html > > > > > > Can we rerun this on something at ANL since I think this cannot be > true. > > > > > > Matt > > > > > > -- > > > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > > > -- Norbert Wiener > > > > > > > > -- > > Jeff Hammond > > jeff.science(a)gmail.com > > http://jeffhammond.github.io/ > > -- Jeff Hammond jeff.science(a)gmail.com http://jeffhammond.github.io/

1 0

Re: [petsc-dev] WTF
by Andrew McRae 30 Jun '16

30 Jun '16

Justin, Note that the Firedrake authors did their own Poisson scaling in the firedrake paper (arXiv <https://arxiv.org/abs/1501.01809>), and they needed to set custom BoomerAMG options to get good performance. See pages 6 and 16-18. If I recall correctly, with the out-of-the-box options, far too many coarsening stages were done. This was a 3D Poisson equation, and the weak scaling results use the space P3, so your results may vary. Andrew On 30 June 2016 at 08:40, Justin Chang <jychang48(a)gmail.com> wrote: > That guy's results actually make sense to me. > > I also get poor strong-scaling for the FEM version of the poisson equation > (via firedrake) using HYPRE's boomerAMG. The studies were done on Intel > E5-2670 machines and had proper OpenMPI bindings.No HYPRE configure options > were set via command line so I just used whatever the default setting were. > > If he used ML, GAMG, or even ILU he would likely get much better scaling > as I have. > > Attached is a speedup plot of a much smaller problem I did (225k dofs), > but you can still see a similar progression on how HYPRE deteriorates. > > Compared to the other preconditioners, I noticed that HYPRE has a much > lower flop-to-byte ratio which suggests to me that based on the current > solver configurations, HYPRE is likely going to be more memory-bandwidth > and suffer from lack of memory usage as more cores are used. > > Not sure how to properly configure any of these multigrid preconditioners, > but figured I'd offer my two cents. > > Justin > > On Thu, Jun 30, 2016 at 5:18 AM, Barry Smith <bsmith(a)mcs.anl.gov> wrote: > >> >> > On Jun 29, 2016, at 10:06 PM, Jeff Hammond <jeff.science(a)gmail.com> >> wrote: >> > >> > >> > >> > On Wednesday, June 29, 2016, Barry Smith <bsmith(a)mcs.anl.gov> wrote: >> > >> > Who are these people and why to they have this webpage? >> > >> > >> > Pop up 2-3 directories and you'll see this is a grad student who >> appears to be trying to learn applied math. Is this really your enemy? >> Don't you guys have some DOE bigwigs to bash? >> > >> > Almost for sure they are doing no process binding and no proper >> assignment of processes to memory domains. >> > >> > >> > MVAPICH2 sets affinity by default. Details not given but "infiniband >> enabled" means it might have been used. I don't know what OpenMPI does by >> default but affinity alone doesn't explain this. >> >> By affinity you mean that the process just remains on the same core >> right? You could be right I think the main affect is a bad assignment of >> processes to cores/memory domains. >> >> > >> > In addition they are likely filling up all the cores on the first node >> before adding processes to the second core etc. >> > >> > >> > That's how I would show scaling. Are you suggesting using all the nodes >> and doing breadth first placement? >> >> I would fill up one process per memory domain moving across the nodes; >> then go back and start a second process on each memory domain. etc You can >> also just go across nodes as you suggest and then across memory domains >> >> If you fill up the entire node of cores and then go to the next node >> you get this affect that the performance goes way down as you fill up the >> last of the cores (because no more memory bandwidth is available) and then >> performance goes up again as you jump to the next node and suddenly have a >> big chunk of additional bandwidth. You also have weird load balancing >> problem because the first 16 processes are going slow because they share >> some bandwidth while the 17 runs much faster since it can hog more >> bandwidth. >> >> > >> > Jeff >> > >> > If the studies had been done properly there should be very little fail >> off on the strong scaling in going from 1 to 2 to 4 processes and even >> beyond. Similarly the huge fail off in going from 4 to 8 to 16 would not >> occur for weak scaling. >> > >> > Barry >> > >> > >> > > On Jun 29, 2016, at 7:47 PM, Matthew Knepley <knepley(a)gmail.com> >> wrote: >> > > >> > > >> > > >> > > http://guest.ams.sunysb.edu/~zgao/work/airfoil/scaling.html >> > > >> > > Can we rerun this on something at ANL since I think this cannot be >> true. >> > > >> > > Matt >> > > >> > > -- >> > > What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> > > -- Norbert Wiener >> > >> > >> > >> > -- >> > Jeff Hammond >> > jeff.science(a)gmail.com >> > http://jeffhammond.github.io/ >> >> >

1 0

Re: [petsc-dev] WTF
by Justin Chang 30 Jun '16

30 Jun '16

That guy's results actually make sense to me. I also get poor strong-scaling for the FEM version of the poisson equation (via firedrake) using HYPRE's boomerAMG. The studies were done on Intel E5-2670 machines and had proper OpenMPI bindings.No HYPRE configure options were set via command line so I just used whatever the default setting were. If he used ML, GAMG, or even ILU he would likely get much better scaling as I have. Attached is a speedup plot of a much smaller problem I did (225k dofs), but you can still see a similar progression on how HYPRE deteriorates. Compared to the other preconditioners, I noticed that HYPRE has a much lower flop-to-byte ratio which suggests to me that based on the current solver configurations, HYPRE is likely going to be more memory-bandwidth and suffer from lack of memory usage as more cores are used. Not sure how to properly configure any of these multigrid preconditioners, but figured I'd offer my two cents. Justin On Thu, Jun 30, 2016 at 5:18 AM, Barry Smith <bsmith(a)mcs.anl.gov> wrote: > > > On Jun 29, 2016, at 10:06 PM, Jeff Hammond <jeff.science(a)gmail.com> > wrote: > > > > > > > > On Wednesday, June 29, 2016, Barry Smith <bsmith(a)mcs.anl.gov> wrote: > > > > Who are these people and why to they have this webpage? > > > > > > Pop up 2-3 directories and you'll see this is a grad student who appears > to be trying to learn applied math. Is this really your enemy? Don't you > guys have some DOE bigwigs to bash? > > > > Almost for sure they are doing no process binding and no proper > assignment of processes to memory domains. > > > > > > MVAPICH2 sets affinity by default. Details not given but "infiniband > enabled" means it might have been used. I don't know what OpenMPI does by > default but affinity alone doesn't explain this. > > By affinity you mean that the process just remains on the same core > right? You could be right I think the main affect is a bad assignment of > processes to cores/memory domains. > > > > > In addition they are likely filling up all the cores on the first node > before adding processes to the second core etc. > > > > > > That's how I would show scaling. Are you suggesting using all the nodes > and doing breadth first placement? > > I would fill up one process per memory domain moving across the nodes; > then go back and start a second process on each memory domain. etc You can > also just go across nodes as you suggest and then across memory domains > > If you fill up the entire node of cores and then go to the next node > you get this affect that the performance goes way down as you fill up the > last of the cores (because no more memory bandwidth is available) and then > performance goes up again as you jump to the next node and suddenly have a > big chunk of additional bandwidth. You also have weird load balancing > problem because the first 16 processes are going slow because they share > some bandwidth while the 17 runs much faster since it can hog more > bandwidth. > > > > > Jeff > > > > If the studies had been done properly there should be very little fail > off on the strong scaling in going from 1 to 2 to 4 processes and even > beyond. Similarly the huge fail off in going from 4 to 8 to 16 would not > occur for weak scaling. > > > > Barry > > > > > > > On Jun 29, 2016, at 7:47 PM, Matthew Knepley <knepley(a)gmail.com> > wrote: > > > > > > > > > > > > http://guest.ams.sunysb.edu/~zgao/work/airfoil/scaling.html > > > > > > Can we rerun this on something at ANL since I think this cannot be > true. > > > > > > Matt > > > > > > -- > > > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > > > -- Norbert Wiener > > > > > > > > -- > > Jeff Hammond > > jeff.science(a)gmail.com > > http://jeffhammond.github.io/ > >

1 0

Re: [petsc-dev] WTF
by Barry Smith 29 Jun '16

29 Jun '16

> On Jun 29, 2016, at 10:06 PM, Jeff Hammond <jeff.science(a)gmail.com> wrote: > > > > On Wednesday, June 29, 2016, Barry Smith <bsmith(a)mcs.anl.gov> wrote: > > Who are these people and why to they have this webpage? > > > Pop up 2-3 directories and you'll see this is a grad student who appears to be trying to learn applied math. Is this really your enemy? Don't you guys have some DOE bigwigs to bash? > > Almost for sure they are doing no process binding and no proper assignment of processes to memory domains. > > > MVAPICH2 sets affinity by default. Details not given but "infiniband enabled" means it might have been used. I don't know what OpenMPI does by default but affinity alone doesn't explain this. By affinity you mean that the process just remains on the same core right? You could be right I think the main affect is a bad assignment of processes to cores/memory domains. > > In addition they are likely filling up all the cores on the first node before adding processes to the second core etc. > > > That's how I would show scaling. Are you suggesting using all the nodes and doing breadth first placement? I would fill up one process per memory domain moving across the nodes; then go back and start a second process on each memory domain. etc You can also just go across nodes as you suggest and then across memory domains If you fill up the entire node of cores and then go to the next node you get this affect that the performance goes way down as you fill up the last of the cores (because no more memory bandwidth is available) and then performance goes up again as you jump to the next node and suddenly have a big chunk of additional bandwidth. You also have weird load balancing problem because the first 16 processes are going slow because they share some bandwidth while the 17 runs much faster since it can hog more bandwidth. > > Jeff > > If the studies had been done properly there should be very little fail off on the strong scaling in going from 1 to 2 to 4 processes and even beyond. Similarly the huge fail off in going from 4 to 8 to 16 would not occur for weak scaling. > > Barry > > > > On Jun 29, 2016, at 7:47 PM, Matthew Knepley <knepley(a)gmail.com> wrote: > > > > > > > > http://guest.ams.sunysb.edu/~zgao/work/airfoil/scaling.html > > > > Can we rerun this on something at ANL since I think this cannot be true. > > > > Matt > > > > -- > > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > > -- Norbert Wiener > > > > -- > Jeff Hammond > jeff.science(a)gmail.com > http://jeffhammond.github.io/

1 0

Re: [petsc-dev] WTF
by Jeff Hammond 29 Jun '16

29 Jun '16

On Wednesday, June 29, 2016, Barry Smith <bsmith(a)mcs.anl.gov> wrote: > > Who are these people and why to they have this webpage? > > Pop up 2-3 directories and you'll see this is a grad student who appears to be trying to learn applied math. Is this really your enemy? Don't you guys have some DOE bigwigs to bash? > Almost for sure they are doing no process binding and no proper > assignment of processes to memory domains. > MVAPICH2 sets affinity by default. Details not given but "infiniband enabled" means it might have been used. I don't know what OpenMPI does by default but affinity alone doesn't explain this. > In addition they are likely filling up all the cores on the first node > before adding processes to the second core etc. > That's how I would show scaling. Are you suggesting using all the nodes and doing breadth first placement? Jeff > If the studies had been done properly there should be very little fail off > on the strong scaling in going from 1 to 2 to 4 processes and even beyond. > Similarly the huge fail off in going from 4 to 8 to 16 would not occur for > weak scaling. > > Barry > > > > On Jun 29, 2016, at 7:47 PM, Matthew Knepley <knepley(a)gmail.com > <javascript:;>> wrote: > > > > > > > > http://guest.ams.sunysb.edu/~zgao/work/airfoil/scaling.html > > > > Can we rerun this on something at ANL since I think this cannot be true. > > > > Matt > > > > -- > > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > > -- Norbert Wiener > > -- Jeff Hammond jeff.science(a)gmail.com http://jeffhammond.github.io/

1 0