On Fri, Apr 12, 2013 at 9:18 AM, Chris Kees <[email protected]> wrote:

I updated the results for the Bratu problem on our SGI. It has 8 cores per node (two 4-core processors per node), and I ran from 1 to 256 cores. The log_summary output is attached for both studies. Question:

Strong scaling: This looks fine. You get the classic memory bandwidth starvation after 2 cores on the same node (although your scaling does not completely bottom out), and among nodes the scaling is great. Weak scaling: I have to go through the logs, but obviously something is wrong. I am betting it is the failure to increase the GMG levels with increasing problem size.

is there anything about the memory usage of that problem that doesn't scale? The memory usage looks steady at < 1GB per core based on log_summary. I ask because last night I tried to do one more level of refinement for weak scaling on 1024 cores and it crashed. I ran the same job on 512 cores this morning, and it ran fine so I'm hoping the issue was a temporary system problem.

No, the memory usage is scalable. Thanks, Matt

Notes:

There is a shift in the strong scaling curve as it fills up the first node (i.e. from 1 to 16 cores), then it looks perfect. The shift seems reasonable due to the sharing of the cache by 4 cores.

The weak scaling shows slight growth in the wall clock from 6.3 seconds to 17 seconds. I'm going to run that again with a larger coarse grid in order to increase the runtime to several minutes.

Graphs: https://proteus.usace.army.mil/home/pub/17/

On Thu, Apr 11, 2013 at 12:46 PM, Jed Brown <[email protected]> wrote:

...

Chris Kees <[email protected]> writes:

...

Thanks a lot. I did a little example with the Bratu problem and posted it here:

https://proteus.usace.army.mil/home/pub/17/

I used boomeramg instead of geometric multigrid because I was getting an error with the options above:

%mpiexec -np 4 ./ex5 -mx 129 -my 129 -Nx 2 -Ny 2 -pc_type mg -pc_mg_levels 2 [0]PETSC ERROR: --------------------- Error Message ------------------------------------ [0]PETSC ERROR: Argument out of range! [0]PETSC ERROR: New nonzero at (66,1) caused a malloc! [0]PETSC ERROR: ------------------------------------------------------------------------

That test hard-codes evil things (presumably for testing purposes, though maybe the functionality has been subsumed). Please use src/snes/examples/tutorials/ex5.c instead.

mpiexec -n 4 ./ex5 -da_grid_x 65 -da_grid_y 65 -pc_type mg -log_summary -da_refine 1

Increase '-da_refine 1' to get higher resolution. (This will increase the number of MG levels used by PCMG.)

Switch '-da_refine 1' to '-snes_grid_sequence 1' if you want FMG, but note that it's trickier to profile because proportionately more time is spent in coarse levels (although the total solve time is lower).

...

I like the ice paper and will try to get the contractor started on reproducing those results.

-Chris

On Wed, Apr 10, 2013 at 1:13 PM, Nystrom, William D <[email protected]>

wrote:

...

...

Sorry. I overlooked that the URL was using git protocol. My bad.

Dave

________________________________________ From: Jed Brown [[email protected]] on behalf of Jed Brown [ [email protected]] Sent: Wednesday, April 10, 2013 12:10 PM To: Nystrom, William D; For users of the development version of PETSc; Chris Kees Subject: Re: [petsc-dev] examples/benchmarks for weak and strong scaling exercise

"Nystrom, William D" <[email protected]> writes:

...

Jed,

I tried cloning your tme-ice git repo as follows and it failed:

% git clone --recursive git://github.com/jedbrown/tme-ice.git tme_ice Cloning into 'tme_ice'... fatal: unable to connect to github.com: github.com[0: 204.232.175.90]: errno=Connection timed out

I'm doing this from an xterm that allows me to clone petsc just fine.

You're using https or ssh to clone PETSc, but the git:// to clone tme-ice. The LANL network is blocking that port, so just use the https or ssh protocol.

-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener