April 2013 - petsc-dev - mailman3.cels.anl.gov

Re: [petsc-dev] examples/benchmarks for weak and strong scaling exercise
by Matthew Knepley 12 Apr '13

12 Apr '13

On Fri, Apr 12, 2013 at 9:18 AM, Chris Kees <cekees(a)gmail.com> wrote: > I updated the results for the Bratu problem on our SGI. It has 8 cores > per node (two 4-core processors per node), and I ran from 1 to 256 > cores. The log_summary output is attached for both studies. Question: > Strong scaling: This looks fine. You get the classic memory bandwidth starvation after 2 cores on the same node (although your scaling does not completely bottom out), and among nodes the scaling is great. Weak scaling: I have to go through the logs, but obviously something is wrong. I am betting it is the failure to increase the GMG levels with increasing problem size. > is there anything about the memory usage of that problem that doesn't > scale? The memory usage looks steady at < 1GB per core based on > log_summary. I ask because last night I tried to do one more level of > refinement for weak scaling on 1024 cores and it crashed. I ran the > same job on 512 cores this morning, and it ran fine so I'm hoping the > issue was a temporary system problem. > No, the memory usage is scalable. Thanks, Matt > Notes: > > There is a shift in the strong scaling curve as it fills up the first > node (i.e. from 1 to 16 cores), then it looks perfect. The shift > seems reasonable due to the sharing of the cache by 4 cores. > > The weak scaling shows slight growth in the wall clock from 6.3 > seconds to 17 seconds. I'm going to run that again with a larger > coarse grid in order to increase the runtime to several minutes. > > Graphs: https://proteus.usace.army.mil/home/pub/17/ > > On Thu, Apr 11, 2013 at 12:46 PM, Jed Brown <jedbrown(a)mcs.anl.gov> wrote: > > Chris Kees <cekees(a)gmail.com> writes: > > > >> Thanks a lot. I did a little example with the Bratu problem and posted > it here: > >> > >> https://proteus.usace.army.mil/home/pub/17/ > >> > >> I used boomeramg instead of geometric multigrid because I was getting > >> an error with the options above: > >> > >> %mpiexec -np 4 ./ex5 -mx 129 -my 129 -Nx 2 -Ny 2 -pc_type mg > -pc_mg_levels 2 > >> [0]PETSC ERROR: --------------------- Error Message > >> ------------------------------------ > >> [0]PETSC ERROR: Argument out of range! > >> [0]PETSC ERROR: New nonzero at (66,1) caused a malloc! > >> [0]PETSC ERROR: > >> ------------------------------------------------------------------------ > > > > That test hard-codes evil things (presumably for testing purposes, > > though maybe the functionality has been subsumed). Please use > > src/snes/examples/tutorials/ex5.c instead. > > > > mpiexec -n 4 ./ex5 -da_grid_x 65 -da_grid_y 65 -pc_type mg -log_summary > -da_refine 1 > > > > Increase '-da_refine 1' to get higher resolution. (This will increase > > the number of MG levels used by PCMG.) > > > > Switch '-da_refine 1' to '-snes_grid_sequence 1' if you want FMG, but > > note that it's trickier to profile because proportionately more time is > > spent in coarse levels (although the total solve time is lower). > > > >> > >> I like the ice paper and will try to get the contractor started on > >> reproducing those results. > >> > >> -Chris > >> > >> On Wed, Apr 10, 2013 at 1:13 PM, Nystrom, William D <wdn(a)lanl.gov> > wrote: > >>> Sorry. I overlooked that the URL was using git protocol. My bad. > >>> > >>> Dave > >>> > >>> ________________________________________ > >>> From: Jed Brown [five9a2(a)gmail.com] on behalf of Jed Brown [ > jedbrown(a)mcs.anl.gov] > >>> Sent: Wednesday, April 10, 2013 12:10 PM > >>> To: Nystrom, William D; For users of the development version of PETSc; > Chris Kees > >>> Subject: Re: [petsc-dev] examples/benchmarks for weak and strong > scaling exercise > >>> > >>> "Nystrom, William D" <wdn(a)lanl.gov> writes: > >>> > >>>> Jed, > >>>> > >>>> I tried cloning your tme-ice git repo as follows and it failed: > >>>> > >>>> % git clone --recursive git://github.com/jedbrown/tme-ice.git tme_ice > >>>> Cloning into 'tme_ice'... > >>>> fatal: unable to connect to github.com: > >>>> github.com[0: 204.232.175.90]: errno=Connection timed out > >>>> > >>>> I'm doing this from an xterm that allows me to clone petsc just fine. > >>> > >>> You're using https or ssh to clone PETSc, but the git:// to clone > >>> tme-ice. The LANL network is blocking that port, so just use the https > >>> or ssh protocol. > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener

1 0

Re: [petsc-dev] John McCalpin talk: Explicit Hardware and Software Semantics for Data Movement in Parallel Computing
by Victor Eijkhout 12 Apr '13

12 Apr '13

https://meeting.austin.utexas.edu/p46an19xdfk/ Really good stuff. Victor.

1 0

Re: [petsc-dev] Has anyone seen this? mpich flat_namespace results in segfaults
by Brad Aagaard 12 Apr '13

12 Apr '13

The debugger shows the segfault ends up at misaligned_stack_error_entering_dyld_stub_binder() Here is the discussion of this issue on the libmesh-devel mailing list: http://www.mail-archive.com/[email protected]/msg04078.ht… The only time I get this segfaulting in building/testing PyLith is within netcdf routines. This same error occurs in the nctest for both 4.1.3 and 4.2.1. Building netcdf with the same gcc compilers works fine, but when I build netcdf with mpicc, I get this segfaults in the nctest. When I remove the flat_namespace link flag from mpicc, mpicxx, etc, these segfaults are gone. This started happening when I moved from a laptop running OSX 10.6 to a laptop running 10.8. Brad On 4/12/13 11:08 AM, Matthew Knepley wrote: > From Brad: > > * mpich linking flags are broken/fragile for Lion and Mountain Lion > > mpich wants to add -flat_namespace to the linking flags. I think this > is based on --enable-shared-libs=osx-gcc. The flat_namespace doesn't > work in many cases on Lion and Mountain Lion-- it caused dynamic library > loading to fail for netcdf, resulting in odd segmentation faults. It > took me a long time to figure this one out. I finally found an email > chain in the libmesh mailing list discussing this problem. I am not > quite sure how PETSc handles building mpich on Lion and Mountain Lion. > It may be extremely fragile in that other things built with the mpich > compilers may segfault when loading dynamic libraries. > > I fixed this by simply removing the flat_namespace flag from the > mpicc, mpicxx, mpif90, mpif77 scripts after installing mpich but before > using it. > > Matt > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which > their experiments lead. > -- Norbert Wiener

1 0

Has anyone seen this?
by Matthew Knepley 12 Apr '13

12 Apr '13

>From Brad: * mpich linking flags are broken/fragile for Lion and Mountain Lion mpich wants to add -flat_namespace to the linking flags. I think this is based on --enable-shared-libs=osx-gcc. The flat_namespace doesn't work in many cases on Lion and Mountain Lion-- it caused dynamic library loading to fail for netcdf, resulting in odd segmentation faults. It took me a long time to figure this one out. I finally found an email chain in the libmesh mailing list discussing this problem. I am not quite sure how PETSc handles building mpich on Lion and Mountain Lion. It may be extremely fragile in that other things built with the mpich compilers may segfault when loading dynamic libraries. I fixed this by simply removing the flat_namespace flag from the mpicc, mpicxx, mpif90, mpif77 scripts after installing mpich but before using it. Matt -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener

1 0

Re: [petsc-dev] examples/benchmarks for weak and strong scaling exercise
by Chris Kees 12 Apr '13

12 Apr '13

I updated the results for the Bratu problem on our SGI. It has 8 cores per node (two 4-core processors per node), and I ran from 1 to 256 cores. The log_summary output is attached for both studies. Question: is there anything about the memory usage of that problem that doesn't scale? The memory usage looks steady at < 1GB per core based on log_summary. I ask because last night I tried to do one more level of refinement for weak scaling on 1024 cores and it crashed. I ran the same job on 512 cores this morning, and it ran fine so I'm hoping the issue was a temporary system problem. Notes: There is a shift in the strong scaling curve as it fills up the first node (i.e. from 1 to 16 cores), then it looks perfect. The shift seems reasonable due to the sharing of the cache by 4 cores. The weak scaling shows slight growth in the wall clock from 6.3 seconds to 17 seconds. I'm going to run that again with a larger coarse grid in order to increase the runtime to several minutes. Graphs: https://proteus.usace.army.mil/home/pub/17/ On Thu, Apr 11, 2013 at 12:46 PM, Jed Brown <jedbrown(a)mcs.anl.gov> wrote: > Chris Kees <cekees(a)gmail.com> writes: > >> Thanks a lot. I did a little example with the Bratu problem and posted it here: >> >> https://proteus.usace.army.mil/home/pub/17/ >> >> I used boomeramg instead of geometric multigrid because I was getting >> an error with the options above: >> >> %mpiexec -np 4 ./ex5 -mx 129 -my 129 -Nx 2 -Ny 2 -pc_type mg -pc_mg_levels 2 >> [0]PETSC ERROR: --------------------- Error Message >> ------------------------------------ >> [0]PETSC ERROR: Argument out of range! >> [0]PETSC ERROR: New nonzero at (66,1) caused a malloc! >> [0]PETSC ERROR: >> ------------------------------------------------------------------------ > > That test hard-codes evil things (presumably for testing purposes, > though maybe the functionality has been subsumed). Please use > src/snes/examples/tutorials/ex5.c instead. > > mpiexec -n 4 ./ex5 -da_grid_x 65 -da_grid_y 65 -pc_type mg -log_summary -da_refine 1 > > Increase '-da_refine 1' to get higher resolution. (This will increase > the number of MG levels used by PCMG.) > > Switch '-da_refine 1' to '-snes_grid_sequence 1' if you want FMG, but > note that it's trickier to profile because proportionately more time is > spent in coarse levels (although the total solve time is lower). > >> >> I like the ice paper and will try to get the contractor started on >> reproducing those results. >> >> -Chris >> >> On Wed, Apr 10, 2013 at 1:13 PM, Nystrom, William D <wdn(a)lanl.gov> wrote: >>> Sorry. I overlooked that the URL was using git protocol. My bad. >>> >>> Dave >>> >>> ________________________________________ >>> From: Jed Brown [five9a2(a)gmail.com] on behalf of Jed Brown [jedbrown(a)mcs.anl.gov] >>> Sent: Wednesday, April 10, 2013 12:10 PM >>> To: Nystrom, William D; For users of the development version of PETSc; Chris Kees >>> Subject: Re: [petsc-dev] examples/benchmarks for weak and strong scaling exercise >>> >>> "Nystrom, William D" <wdn(a)lanl.gov> writes: >>> >>>> Jed, >>>> >>>> I tried cloning your tme-ice git repo as follows and it failed: >>>> >>>> % git clone --recursive git://github.com/jedbrown/tme-ice.git tme_ice >>>> Cloning into 'tme_ice'... >>>> fatal: unable to connect to github.com: >>>> github.com[0: 204.232.175.90]: errno=Connection timed out >>>> >>>> I'm doing this from an xterm that allows me to clone petsc just fine. >>> >>> You're using https or ssh to clone PETSc, but the git:// to clone >>> tme-ice. The LANL network is blocking that port, so just use the https >>> or ssh protocol.

1 0

Re: [petsc-dev] Fwd: [petsc-users] cannot find 'libz.a' when configuring
by Barry Smith 11 Apr '13

11 Apr '13

If hdf5 provided a package file we could use that to get the needed info https://github.com/mokus0/bindings-hdf5/blob/master/doc/hdf5.pc > > Okay, I would like to distinguish between good behavior and HDF5. If we > can ask whether the package enables it, we are fine. If there is a simple > test we can do, we are fine. What can we do for HDF5? > > Matt > > > Satish > > > > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener

1 0

Re: [petsc-dev] John McCalpin talk: Explicit Hardware and Software Semantics for Data Movement in Parallel Computing
by Matthew Knepley 11 Apr '13

11 Apr '13

On Thu, Apr 11, 2013 at 9:13 PM, Victor Eijkhout <eijkhout(a)tacc.utexas.edu>wrote: > > On Apr 11, 2013, at 7:28 PM, Matthew Knepley <knepley(a)gmail.com> wrote: > > Are you taping this so the unwashed in Chicago (barely) can watch it? > > > You did see this? > I am actually in witness protection in Providence and can't make that time, so taped would be great. Thanks, Matt > > Please join me in an Adobe Connect Meeting.**** >> ** ** >> Meeting Name: TACC Seminar Series**** >> Summary:**** >> Invited By: Allison Warner (awarner(a)tacc.utexas.edu)**** >> When: 04/12/2013 1:00 PM - 2:00 PM**** >> Time Zone: (GMT-06:00) Central Time (US and Canada)**** >> ** ** >> Conference Number(s): 1-877-820-7831**** >> Participant Code: 124574#**** >> ** ** >> To join the meeting:**** >> https://meeting.austin.utexas.edu/taccseminar/ >> >> > But yes, will be taped. > > Victor. > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener

1 0

Re: [petsc-dev] John McCalpin talk: Explicit Hardware and Software Semantics for Data Movement in Parallel Computing
by Victor Eijkhout 11 Apr '13

11 Apr '13

On Apr 11, 2013, at 7:28 PM, Matthew Knepley <knepley(a)gmail.com<mailto:[email protected]>> wrote: Are you taping this so the unwashed in Chicago (barely) can watch it? You did see this? Please join me in an Adobe Connect Meeting. Meeting Name: TACC Seminar Series Summary: Invited By: Allison Warner (awarner(a)tacc.utexas.edu<mailto:[email protected]>) When: 04/12/2013 1:00 PM - 2:00 PM Time Zone: (GMT-06:00) Central Time (US and Canada) Conference Number(s): 1-877-820-7831<tel:1-877-820-7831> Participant Code: 124574# To join the meeting: https://meeting.austin.utexas.edu/taccseminar/ But yes, will be taped. Victor.

1 0

Re: [petsc-dev] John McCalpin talk: Explicit Hardware and Software Semantics for Data Movement in Parallel Computing
by Victor Eijkhout 11 Apr '13

11 Apr '13

On Apr 11, 2013, at 7:28 PM, Matthew Knepley <knepley(a)gmail.com<mailto:[email protected]>> wrote: Are you taping this so the unwashed in Chicago (barely) can watch it? Ok, will do. Victor.

1 0

Re: [petsc-dev] John McCalpin talk: Explicit Hardware and Software Semantics for Data Movement in Parallel Computing
by Matthew Knepley 11 Apr '13

11 Apr '13

Are you taping this so the unwashed in Chicago (barely) can watch it? THanks, Matt On Thu, Apr 11, 2013 at 12:27 PM, Victor Eijkhout <eijkhout(a)tacc.utexas.edu>wrote: > > > Begin forwarded message: > > *From: *Allison Warner <allison(a)tacc.utexas.edu> > *Subject: ****Reminder: TACC Seminar tomorrow -4/12/13* > *Date: *April 11, 2013 11:51:07 AM CDT > *To: *TACC Team <team(a)tacc.utexas.edu> > > Join us tomorrow for part two of the TACC seminar: Explicit Hardware > and Software Semantics for Data Movement in Parallel Computing at 1pm in > ROC 1.900- the new seminar room. > April 12, 2013#<https://team.tacc.utexas.edu/staff-info/-/wikid/xJ3E/Main/TACC+Tech+Talks#s…> > > *John McCalpin* > > > *Explicit Architectural Support for Data Movement and Interprocess > Communication* > > *Abstract:* > Despite their substantial complexity, current computer architectures are, > in several important ways, essentially unchanged from the single-processor, > flat-memory, modified Harvard architectures first developed in the late > 1960's. Current technology, on the other hand, has completely different > balances than it did 40+ years ago, so that massive investments in > processor design are now required to hide the fundamental (and growing) > mismatch between what the architectures make visible and what the > technology makes expensive. > In the context of current technology, the most important factors in > application performance are often related to control of data motion > "vertically" through the memory/cache hierarchy and to efficiently > performing communication/synchronization operations "horizontally" between > cooperating parallel processes. These data transfers are expected to > dominate the power budget as well as the performance equation in future > systems. > Current architectures do not provide the features to manage or optimize > these operations. Several examples of performance, power, and/or cost > benefits of providing architectural control over data motion and > communication/synchronization will be presented and discussed. > > > *Bio:* > John McCalpin is a Research Scientist in the HPC Group of the Texas > Advanced Computing Center. McCalpin began his career in HPC working in > large-scale ocean circulation modeling, then spent 12 years in industry, > holding leadership positions in the system architecture and processor > performance teams at SGI, IBM, and AMD. McCalpin has been at TACC since > 1999, where he focuses on performance analysis, performance tools, and > architecture evaluation. > > Please join me in an Adobe Connect Meeting.**** > ** ** > Meeting Name: TACC Seminar Series**** > Summary:**** > Invited By: Allison Warner (awarner(a)tacc.utexas.edu)**** > When: 04/12/2013 1:00 PM - 2:00 PM**** > Time Zone: (GMT-06:00) Central Time (US and Canada)**** > ** ** > Conference Number(s): 1-877-820-7831**** > Participant Code: 124574#**** > ** ** > To join the meeting:**** > https://meeting.austin.utexas.edu/taccseminar/ > > > ____________________________________ > Allison Warner > Executive Assistant > Texas Advanced Computing Center > The University of Texas at Austin > (512) 475-9451 > allison(a)tacc.utexas.edu > > > > ** > > Connect to TACC: > > [image: Connect with TACC on Facebook]<http://www.facebook.com/tacc.utexas> > [image: Connect with TACC on Twitter] <http://twitter.com/TACC> [image: > Connect with TACC on Googleplus]<https://plus.google.com/117998677621567541132> > [image: Connect with TACC on Rss]<http://www.tacc.utexas.edu/news/subscribe> > [image: Connect with TACC on Linkedin]<http://www.linkedin.com/company/229459> > [image: Connect with TACC on Youtube]<http://www.youtube.com/user/TACCutexas> > [image: Connect with TACC on Calendar]<http://www.tacc.utexas.edu/calendar/> > [image: Connect with TACC on Connect] <http://www.tacc.utexas.edu/> > > * > * > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener

1 0