July 2016 - petsc-dev - mailman3.cels.anl.gov

Re: [petsc-dev] PETSc blame digest (next) 2016-07-02
by Dave May 02 Jul '16

02 Jul '16

These printfs fixes have been pushed into knepley/feature-plex-particle-example Could someone please do the merge into next? On Saturday, 2 July 2016, PETSc checkBuilds <petsc-checkbuilds(a)mcs.anl.gov> wrote: > > > Dear PETSc developer, > > This email contains listings of contributions attributed to you by > `git blame` that caused compiler errors or warnings in PETSc automated > testing. Follow the links to see the full log files. Please attempt to fix > the issues promptly or let us know at petsc-dev(a)mcs.anl.gov <javascript:;> > if you are unable > to resolve the issues. > > Thanks, > The PETSc development team > > ---- > > warnings attributed to commit > https://bitbucket.org/petsc/petsc/commits/b16650c > DMSwarm collect: > > src/dm/impls/swarm/swarm_migrate.c:481 > [ > http://ftp.mcs.anl.gov/pub/petsc/nightlylogs/archive/2016/07/02/build_next_… > ] > > /sandbox/petsc/petsc.clone-3/src/dm/impls/swarm/swarm_migrate.c:481:91: > warning: format '%e' expects argument of type 'double', but argument 4 has > type 'PetscReal {aka __float128}' [-Wformat=] > [ > http://ftp.mcs.anl.gov/pub/petsc/nightlylogs/archive/2016/07/02/build_next_… > ] > > /sandbox/petsc/petsc.clone-3/src/dm/impls/swarm/swarm_migrate.c:481:12: > warning: format '%e' expects argument of type 'double', but argument 4 has > type 'PetscReal' [-Wformat=] > [ > http://ftp.mcs.anl.gov/pub/petsc/nightlylogs/archive/2016/07/02/build_next_… > ] > > /sandbox/petsc/petsc.clone-3/src/dm/impls/swarm/swarm_migrate.c:481:12: > warning: format '%e' expects argument of type 'double', but argument 5 has > type 'PetscReal' [-Wformat=] > [ > http://ftp.mcs.anl.gov/pub/petsc/nightlylogs/archive/2016/07/02/build_next_… > ] > > /sandbox/petsc/petsc.clone-3/src/dm/impls/swarm/swarm_migrate.c:481:91: > warning: format '%e' expects argument of type 'double', but argument 7 has > type 'PetscReal {aka __float128}' [-Wformat=] > [ > http://ftp.mcs.anl.gov/pub/petsc/nightlylogs/archive/2016/07/02/build_next_… > ] > > /sandbox/petsc/petsc.clone-3/src/dm/impls/swarm/swarm_migrate.c:481:91: > warning: format '%e' expects argument of type 'double', but argument 5 has > type 'PetscReal {aka __float128}' [-Wformat=] > [ > http://ftp.mcs.anl.gov/pub/petsc/nightlylogs/archive/2016/07/02/build_next_… > ] > > /sandbox/petsc/petsc.clone-3/src/dm/impls/swarm/swarm_migrate.c:481:12: > warning: format '%e' expects argument of type 'double', but argument 7 has > type 'PetscReal' [-Wformat=] > [ > http://ftp.mcs.anl.gov/pub/petsc/nightlylogs/archive/2016/07/02/build_next_… > ] > > /sandbox/petsc/petsc.clone-3/src/dm/impls/swarm/swarm_migrate.c:481:12: > warning: format '%e' expects argument of type 'double', but argument 6 has > type 'PetscReal' [-Wformat=] > [ > http://ftp.mcs.anl.gov/pub/petsc/nightlylogs/archive/2016/07/02/build_next_… > ] > > /sandbox/petsc/petsc.clone-3/src/dm/impls/swarm/swarm_migrate.c:481:91: > warning: format '%e' expects argument of type 'double', but argument 6 has > type 'PetscReal {aka __float128}' [-Wformat=] > > ---- > To opt-out from receiving these messages - send a request to > petsc-dev(a)mcs.anl.gov <javascript:;>. >

1 0

Re: [petsc-dev] WTF
by Mark Adams 02 Jul '16

02 Jul '16

Justin, this is interesting data. It would be nice to take the normalization out because GAMG could just be running very slow on one processor, which makes scaling look better. hypre's algorithm is pretty good on scalar Poisson, low order discretization (and more). Hypre and GAMG should be with say 50% of each other one one processor and hypre should be faster, if both are optimized well (which might not be practical here, but just my 2c). Taking the normalization out of the plot is annoying because then you have to have multiple "perfect" lines. Another way to do this is do what we (Jed) have been calling "dynamic range" instead of "strong scaling" where you decrease the problem size instead of increasing the amount of parallelism. So all data uses the same number of processors. This is just a different way to look at "strong scaling", in the general sense, and is neither better or worse. But, and advantage of this is that you can plot "equations solved"/second vs Solve time. Now perfect scaling is a flat line so there is no need to draw (multiple) diagonal lines for perfect scaling. The height of the lines (dof/sec) is raw performance. It takes a second to get your head wrapped around these dynamic range plots but I like them. I've attached a talk that goes through these plots and explains how to think about them, etc. (see the plot slides in the middle of the slides) On Thu, Jun 30, 2016 at 9:40 AM, Justin Chang <jychang48(a)gmail.com> wrote: > That guy's results actually make sense to me. > > I also get poor strong-scaling for the FEM version of the poisson equation > (via firedrake) using HYPRE's boomerAMG. The studies were done on Intel > E5-2670 machines and had proper OpenMPI bindings.No HYPRE configure options > were set via command line so I just used whatever the default setting were. > > If he used ML, GAMG, or even ILU he would likely get much better scaling > as I have. > > Attached is a speedup plot of a much smaller problem I did (225k dofs), > but you can still see a similar progression on how HYPRE deteriorates. > > Compared to the other preconditioners, I noticed that HYPRE has a much > lower flop-to-byte ratio which suggests to me that based on the current > solver configurations, HYPRE is likely going to be more memory-bandwidth > and suffer from lack of memory usage as more cores are used. > > Not sure how to properly configure any of these multigrid preconditioners, > but figured I'd offer my two cents. > > Justin > > On Thu, Jun 30, 2016 at 5:18 AM, Barry Smith <bsmith(a)mcs.anl.gov> wrote: > >> >> > On Jun 29, 2016, at 10:06 PM, Jeff Hammond <jeff.science(a)gmail.com> >> wrote: >> > >> > >> > >> > On Wednesday, June 29, 2016, Barry Smith <bsmith(a)mcs.anl.gov> wrote: >> > >> > Who are these people and why to they have this webpage? >> > >> > >> > Pop up 2-3 directories and you'll see this is a grad student who >> appears to be trying to learn applied math. Is this really your enemy? >> Don't you guys have some DOE bigwigs to bash? >> > >> > Almost for sure they are doing no process binding and no proper >> assignment of processes to memory domains. >> > >> > >> > MVAPICH2 sets affinity by default. Details not given but "infiniband >> enabled" means it might have been used. I don't know what OpenMPI does by >> default but affinity alone doesn't explain this. >> >> By affinity you mean that the process just remains on the same core >> right? You could be right I think the main affect is a bad assignment of >> processes to cores/memory domains. >> >> > >> > In addition they are likely filling up all the cores on the first node >> before adding processes to the second core etc. >> > >> > >> > That's how I would show scaling. Are you suggesting using all the nodes >> and doing breadth first placement? >> >> I would fill up one process per memory domain moving across the nodes; >> then go back and start a second process on each memory domain. etc You can >> also just go across nodes as you suggest and then across memory domains >> >> If you fill up the entire node of cores and then go to the next node >> you get this affect that the performance goes way down as you fill up the >> last of the cores (because no more memory bandwidth is available) and then >> performance goes up again as you jump to the next node and suddenly have a >> big chunk of additional bandwidth. You also have weird load balancing >> problem because the first 16 processes are going slow because they share >> some bandwidth while the 17 runs much faster since it can hog more >> bandwidth. >> >> > >> > Jeff >> > >> > If the studies had been done properly there should be very little fail >> off on the strong scaling in going from 1 to 2 to 4 processes and even >> beyond. Similarly the huge fail off in going from 4 to 8 to 16 would not >> occur for weak scaling. >> > >> > Barry >> > >> > >> > > On Jun 29, 2016, at 7:47 PM, Matthew Knepley <knepley(a)gmail.com> >> wrote: >> > > >> > > >> > > >> > > http://guest.ams.sunysb.edu/~zgao/work/airfoil/scaling.html >> > > >> > > Can we rerun this on something at ANL since I think this cannot be >> true. >> > > >> > > Matt >> > > >> > > -- >> > > What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> > > -- Norbert Wiener >> > >> > >> > >> > -- >> > Jeff Hammond >> > jeff.science(a)gmail.com >> > http://jeffhammond.github.io/ >> >> >

1 0

Re: [petsc-dev] WTF
by Mark Adams 02 Jul '16

02 Jul '16

On Thu, Jun 30, 2016 at 8:48 PM, Jeff Hammond <jeff.science(a)gmail.com> wrote: > > The default settings often don’t work well for 3D problems. > > Are 2D (1D?) problems really the common case for PDE solvers? Aren't > interesting problems 3D? Shouldn't the defaults be set to optimize for 3D? > Lots of people do 2D in production (it might just be part of a larger 3D algorithm like the XGC code). Also 2D is useful for debugging, regression, incremental development. I would prefer that hypre's defaults be for 3D and I like Barry's suggestion of changing the default with the dimension. We can get a shot at the dimension, without any API changes, by just inferring from the block size. Better than nothing. But I do find it annoying to search around for my 3D hypre parameters when I'm (customers) doing a 3D problems. I am sure lots of cycles are wasted by PETSc users using hypre in 3D with the defaults. > > Jeff > > -- > Jeff Hammond > jeff.science(a)gmail.com > http://jeffhammond.github.io/ >

1 0

Re: [petsc-dev] Soliciting suggestions for linear solver work under SciDAC 4 Institutes
by Patrick Sanan 02 Jul '16

02 Jul '16

Maybe a general class of opportunities for PETSc could be wrapped up under "good abstractions for memory space awareness". That is, all these topics have been discussed recently: 1. Re Jeff's excellent suggestion about OpenMP, it would be nice if more were done to promote the alternative, MPI-based shared memory capability. It's quite clear that this is a good way to go, but this doesn't mean that people (who often equate shared memory parallelism with threads) will use it, so the more that can be done to make the internals of the library use this approach and the more that the classes (DM in particular) can make it easy to do things "the right way", the better. 2. As we saw from Richard's talk on KNL, that device will feature (when used a certain way) one NUMA domain which can provide much greater memory bandwidth. As in the discussions here before on the topic, it's a real challenge to figure out how to to make something like this usable via PETSc, and an even greater one to pick defaults in a way that won't sometimes produce bewildering performance results. Nevertheless, there's a good chance this is the kind of hardware that people will be running PETSc on, and given that this is something which attacks memory bandwidth bottlenecks, something that could speed up a lot of PETSc code. 3. People will likely also be running on machines with coprocessors/accelerators/etc for a while, and there needs to be plumbing to deal with the fact that each device may provide an extra memory space which needs to be managed properly in the flat MPI environment. This is of course related to point 1, as good use of MPI-3 shared memory seems like a reasonable way forward. 4. Related topics re MPI-4/endpoints and how PETSc really could be used properly from an existing multi-threaded environment. Karl has talked about the challenges of doing this the right way a lot recently. With all of these, introducing some clever abstractions to allow these sorts of things to be as transparent/automatic/encapsulated as possible might be very valuable and well-used additions to the library. On Sat, Jul 2, 2016 at 1:13 AM, Jeff Hammond <jeff.science(a)gmail.com> wrote: > Obviously, holistic support for OpenMP is critical to the future of PETSc > :-D > > On a more serious note, Matt and I have discussed the use of PETSc for > sparse multidimensional array computations for dimensions greater than 2, > also known as tensor computations. The associated paper describing previous > work with dense arrays is > http://www.eecs.berkeley.edu/Pubs/TechRpts/2012/EECS-2012-210.html. There > was even an unsuccessful SciDAC application proposal that described how > PETSc could be used for that domain when sparsity is important. To start, > all we'd need is sparse matrix x sparse matrix multiplication, which I hear > the multigrid folks also need. Sparse times dense is also important. > Sparse tensor factorization would also help, but I get that there are enough > open math questions there that it might be impractical to try to implement > something in PETSc in the near future. > > Maybe I am just biased because I spend all of my time reading > www.nextplatform.com, but I hear machine learning is becoming an important > HPC workload. While the most hyped efforts related to running inaccurate - > the technical term is half-precision - dense matrix multiplication as fast > as possible, I suspect that more elegant approaches will prevail. > Presumably there is something that PETSc can do to enable machine learning > algorithms. As most of the existing approaches use silly programming models > based on MapReduce, it can't be too hard for PETSc to do better. > > Jeff > > On Fri, Jul 1, 2016 at 2:32 PM, Barry Smith <bsmith(a)mcs.anl.gov> wrote: >> >> >> The DOE SciDAC institutes have supported PETSc linear solver >> research/code development for the past fifteen years. >> >> This email is to solicit ideas for linear solver research/code >> development work for the next round of SciDAC institutes (which will be a 4 >> year period) in PETSc. Please send me any ideas, no matter how crazy, on >> things you feel are missing, broken, or incomplete in PETSc with regard to >> linear solvers that we should propose to work on. In particular, issues >> coming from particular classes of applications would be good. Generic "multi >> physics" coupling types of things are too general (and old :-)) while work >> for extreme large scale is also out since that is covered under another call >> (ECP). But particular types of optimizations etc for existing or new codes >> could be in, just not for the very large scale. >> >> Rough ideas and pointers to publications are all useful. There is an >> extremely short fuse so the sooner the better, >> >> Thanks >> >> Barry >> >> >> > > > > -- > Jeff Hammond > jeff.science(a)gmail.com > http://jeffhammond.github.io/

1 0

Re: [petsc-dev] Soliciting suggestions for linear solver work under SciDAC 4 Institutes
by Jeff Hammond 01 Jul '16

01 Jul '16

Obviously, holistic support for OpenMP is critical to the future of PETSc :-D On a more serious note, Matt and I have discussed the use of PETSc for sparse multidimensional array computations for dimensions greater than 2, also known as tensor computations. The associated paper describing previous work with dense arrays is http://www.eecs.berkeley.edu/Pubs/TechRpts/2012/EECS-2012-210.html. There was even an unsuccessful SciDAC application proposal that described how PETSc could be used for that domain when sparsity is important. To start, all we'd need is sparse matrix x sparse matrix multiplication, which I hear the multigrid folks also need. Sparse times dense is also important. Sparse tensor factorization would also help, but I get that there are enough open math questions there that it might be impractical to try to implement something in PETSc in the near future. Maybe I am just biased because I spend all of my time reading www.nextplatform.com, but I hear machine learning is becoming an important HPC workload. While the most hyped efforts related to running inaccurate - the technical term is half-precision - dense matrix multiplication as fast as possible, I suspect that more elegant approaches will prevail. Presumably there is something that PETSc can do to enable machine learning algorithms. As most of the existing approaches use silly programming models based on MapReduce, it can't be too hard for PETSc to do better. Jeff On Fri, Jul 1, 2016 at 2:32 PM, Barry Smith <bsmith(a)mcs.anl.gov> wrote: > > The DOE SciDAC institutes have supported PETSc linear solver > research/code development for the past fifteen years. > > This email is to solicit ideas for linear solver research/code > development work for the next round of SciDAC institutes (which will be a 4 > year period) in PETSc. Please send me any ideas, no matter how crazy, on > things you feel are missing, broken, or incomplete in PETSc with regard to > linear solvers that we should propose to work on. In particular, issues > coming from particular classes of applications would be good. Generic > "multi physics" coupling types of things are too general (and old :-)) > while work for extreme large scale is also out since that is covered under > another call (ECP). But particular types of optimizations etc for existing > or new codes could be in, just not for the very large scale. > > Rough ideas and pointers to publications are all useful. There is an > extremely short fuse so the sooner the better, > > Thanks > > Barry > > > > -- Jeff Hammond jeff.science(a)gmail.com http://jeffhammond.github.io/

1 0

Soliciting suggestions for linear solver work under SciDAC 4 Institutes
by Barry Smith 01 Jul '16

01 Jul '16

The DOE SciDAC institutes have supported PETSc linear solver research/code development for the past fifteen years. This email is to solicit ideas for linear solver research/code development work for the next round of SciDAC institutes (which will be a 4 year period) in PETSc. Please send me any ideas, no matter how crazy, on things you feel are missing, broken, or incomplete in PETSc with regard to linear solvers that we should propose to work on. In particular, issues coming from particular classes of applications would be good. Generic "multi physics" coupling types of things are too general (and old :-)) while work for extreme large scale is also out since that is covered under another call (ECP). But particular types of optimizations etc for existing or new codes could be in, just not for the very large scale. Rough ideas and pointers to publications are all useful. There is an extremely short fuse so the sooner the better, Thanks Barry

1 0

Re: [petsc-dev] pets error Segmentation Violation
by Barry Smith 01 Jul '16

01 Jul '16

No idea. You need to do what it says and run with valgrind or in a debugger. From the crash message it looks like it is crashing in your main program. Barry > On Jul 1, 2016, at 1:26 PM, 张江 <zhangjiang.dudu(a)gmail.com> wrote: > > Hi, > > I am trying to read a large data (11.7GB) with libmesh (integrated with PETSc) and use it for my application. The program runs well when using just one process. But in parallel (mpirun -n 4), some errors came out: > > [0]PETSC ERROR: ------------------------------------------------------------------------ > [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range > [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger > [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind > [0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors > [0]PETSC ERROR: likely location of problem given in stack below > [0]PETSC ERROR: --------------------- Stack Frames ------------------------------------ > [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, > [0]PETSC ERROR: INSTEAD the line number of the start of the function > [0]PETSC ERROR: is given. > [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > [0]PETSC ERROR: Signal received > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. > [0]PETSC ERROR: Petsc Release Version 3.7.2, Jun, 05, 2016 > [0]PETSC ERROR: ./ptracer on a arch-linux2-c-debug named compute001 by jiangzhang Fri Jul 1 10:07:07 2016 > [0]PETSC ERROR: Configure options --prefix=/nfs/proj-tpeterka/jiang/opt/petsc-3.7.2 --download-fblaslapack --with-mpi-dir=/nfs/proj-tpeterka/jiang/libraries/mpich-3.2 > [0]PETSC ERROR: #1 User provided function() line 0 in unknown file > application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0 > > Anybody know the possible causes? >

1 0

pets error Segmentation Violation
by 张江 01 Jul '16

01 Jul '16

Hi, I am trying to read a large data (11.7GB) with libmesh (integrated with PETSc) and use it for my application. The program runs well when using just one process. But in parallel (mpirun -n 4), some errors came out: [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [0]PETSC ERROR: likely location of problem given in stack below [0]PETSC ERROR: --------------------- Stack Frames ------------------------------------ [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, [0]PETSC ERROR: INSTEAD the line number of the start of the function [0]PETSC ERROR: is given. [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: Signal received [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [0]PETSC ERROR: Petsc Release Version 3.7.2, Jun, 05, 2016 [0]PETSC ERROR: ./ptracer on a arch-linux2-c-debug named compute001 by jiangzhang Fri Jul 1 10:07:07 2016 [0]PETSC ERROR: Configure options --prefix=/nfs/proj-tpeterka/jiang/opt/petsc-3.7.2 --download-fblaslapack --with-mpi-dir=/nfs/proj-tpeterka/jiang/libraries/mpich-3.2 [0]PETSC ERROR: #1 User provided function() line 0 in unknown file application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0 Anybody know the possible causes?

1 0

Re: [petsc-dev] WTF
by Yang, Ulrike Meier 30 Jun '16

30 Jun '16

That might be the way to go ☺ Ulrike From: Jeff Hammond [mailto:[email protected]] Sent: Thursday, June 30, 2016 11:49 AM To: Yang, Ulrike Meier <yang11(a)llnl.gov> Cc: Barry Smith <bsmith(a)mcs.anl.gov>; Justin Chang <jychang48(a)gmail.com>; PETSc <petsc-dev(a)mcs.anl.gov>; Falgout Rob (rfalgout(a)llnl.gov) <rfalgout(a)llnl.gov> Subject: Re: [petsc-dev] WTF > The default settings often don’t work well for 3D problems. Are 2D (1D?) problems really the common case for PDE solvers? Aren't interesting problems 3D? Shouldn't the defaults be set to optimize for 3D? Jeff -- Jeff Hammond jeff.science(a)gmail.com<mailto:[email protected]> http://jeffhammond.github.io/

1 0