April 2013 - petsc-dev - mailman3.cels.anl.gov

Re: [petsc-dev] examples/benchmarks for weak and strong scaling exercise
by Jed Brown 10 Apr '13

10 Apr '13

"Nystrom, William D" <wdn(a)lanl.gov> writes: > Jed, > > I tried cloning your tme-ice git repo as follows and it failed: > > % git clone --recursive git://github.com/jedbrown/tme-ice.git tme_ice > Cloning into 'tme_ice'... > fatal: unable to connect to github.com: > github.com[0: 204.232.175.90]: errno=Connection timed out > > I'm doing this from an xterm that allows me to clone petsc just fine. You're using https or ssh to clone PETSc, but the git:// to clone tme-ice. The LANL network is blocking that port, so just use the https or ssh protocol.

1 0

Re: [petsc-dev] examples/benchmarks for weak and strong scaling exercise
by Nystrom, William D 10 Apr '13

10 Apr '13

Jed, I tried cloning your tme-ice git repo as follows and it failed: % git clone --recursive git://github.com/jedbrown/tme-ice.git tme_ice Cloning into 'tme_ice'... fatal: unable to connect to github.com: github.com[0: 204.232.175.90]: errno=Connection timed out I'm doing this from an xterm that allows me to clone petsc just fine. Any idea what the problem might be? Dave ________________________________________ From: petsc-dev-bounces(a)mcs.anl.gov [petsc-dev-bounces(a)mcs.anl.gov] on behalf of Jed Brown [jedbrown(a)mcs.anl.gov] Sent: Wednesday, April 10, 2013 9:22 AM To: Chris Kees; petsc-dev(a)mcs.anl.gov Subject: Re: [petsc-dev] examples/benchmarks for weak and strong scaling exercise Chris Kees <cekees(a)gmail.com> writes: > Hi guys, > > Could somebody point me to some examples you guys routinely use for > weak and strong scaling studies (maybe even with scripts, option > files, or prior results on recent hardware)? I'm thinking of 3D > Poisson with finite differences and geometric multigrid or something > like that. One option would be to use src/snes/examples/tutorials/ex48.c and use the configurations from http://dx.doi.org/10.1137/110834512 (http://59A2.org/files/hstat.pdf) which you can find in the paper repository: https://github.com/jedbrown/tme-ice Look in shaheen/b/. Those runs were using DMMG so the command line will have to be modified slightly, but it should be straightforward and you can compare to the runex48_* targets in src/snes/examples/tutorials/makefile. > We've been trying to work toward scaling studies of the field split > and Schur complement preconditioners for our multiphase flow solvers, > but I'm realizing that we need to do more thorough testing of the > petsc installation itself and make sure we're using timing/profiling > "best practices" and such. > > We are using petsc-dev on the hardware below. I promise to quit using > petsc-dev as soon as the next release comes out:) We're actually happy to have people using petsc-dev. One motivation for our new workflow is that we can now provide a pretty stable 'master' so that we can interact with users on new features without the latency of a release cycle and without frequent breakage.

1 0

Re: [petsc-dev] examples/benchmarks for weak and strong scaling exercise
by Jed Brown 10 Apr '13

10 Apr '13

Chris Kees <cekees(a)gmail.com> writes: > Hi guys, > > Could somebody point me to some examples you guys routinely use for > weak and strong scaling studies (maybe even with scripts, option > files, or prior results on recent hardware)? I'm thinking of 3D > Poisson with finite differences and geometric multigrid or something > like that. One option would be to use src/snes/examples/tutorials/ex48.c and use the configurations from http://dx.doi.org/10.1137/110834512 (http://59A2.org/files/hstat.pdf) which you can find in the paper repository: https://github.com/jedbrown/tme-ice Look in shaheen/b/. Those runs were using DMMG so the command line will have to be modified slightly, but it should be straightforward and you can compare to the runex48_* targets in src/snes/examples/tutorials/makefile. > We've been trying to work toward scaling studies of the field split > and Schur complement preconditioners for our multiphase flow solvers, > but I'm realizing that we need to do more thorough testing of the > petsc installation itself and make sure we're using timing/profiling > "best practices" and such. > > We are using petsc-dev on the hardware below. I promise to quit using > petsc-dev as soon as the next release comes out:) We're actually happy to have people using petsc-dev. One motivation for our new workflow is that we can now provide a pretty stable 'master' so that we can interact with users on new features without the latency of a release cycle and without frequent breakage.

1 0

Re: [petsc-dev] examples/benchmarks for weak and strong scaling exercise
by Matthew Knepley 10 Apr '13

10 Apr '13

On Wed, Apr 10, 2013 at 10:04 AM, Chris Kees <cekees(a)gmail.com> wrote: > Hi guys, > > Could somebody point me to some examples you guys routinely use for > weak and strong scaling studies (maybe even with scripts, option > files, or prior results on recent hardware)? I'm thinking of 3D > Poisson with finite differences and geometric multigrid or something > like that. > I am trying to write this stuff down, but there is not much written right now. Let's start at the beginning. SNES ex5 is the simplest example that can be used for this I think. It is 2D Poisson (actually Bratu). Its really easy to get weak scaling by adjusting the grid size using -da_grid_x <m> -da_grid_y <n> You can turn on MG using -pc_type mg -pc_mg_levels <n> although the slightly non-intuitive thing is that then the grid size you input is for the coarse grid. > We've been trying to work toward scaling studies of the field split > and Schur complement preconditioners for our multiphase flow solvers, > but I'm realizing that we need to do more thorough testing of the > petsc installation itself and make sure we're using timing/profiling > "best practices" and such. > > We are using petsc-dev on the hardware below. I promise to quit using > petsc-dev as soon as the next release comes out:) Several versions of > PETSc are also installed by the system maintainers, but my sense is > that there is very little testing done on any of the installations. > I think using petsc-dev is the "right thing", and it is now much much more stable (the 'master' branch on BitBucket). Thanks, Matt > http://www.erdc.hpc.mil/hardware/index.html > > Chris > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener

1 0

examples/benchmarks for weak and strong scaling exercise
by Chris Kees 10 Apr '13

10 Apr '13

Hi guys, Could somebody point me to some examples you guys routinely use for weak and strong scaling studies (maybe even with scripts, option files, or prior results on recent hardware)? I'm thinking of 3D Poisson with finite differences and geometric multigrid or something like that. We've been trying to work toward scaling studies of the field split and Schur complement preconditioners for our multiphase flow solvers, but I'm realizing that we need to do more thorough testing of the petsc installation itself and make sure we're using timing/profiling "best practices" and such. We are using petsc-dev on the hardware below. I promise to quit using petsc-dev as soon as the next release comes out:) Several versions of PETSc are also installed by the system maintainers, but my sense is that there is very little testing done on any of the installations. http://www.erdc.hpc.mil/hardware/index.html Chris

1 0

Re: [petsc-dev] VecMDot_SeqCUSP improved
by Karl Rupp 07 Apr '13

07 Apr '13

Hi Paul, you're very welcome. I'm glad it works out fine for your case as well. Best regards, Karli On 04/07/2013 11:17 PM, Paul Mullowney wrote: > VecMDot is performing great for my examples. About 2X faster than the > trivial implementation that I originally suggested when I reported the > problem. > Thanks Karl. > -Paul >> El 26/03/2013, a las 20:15, Karl Rupp escribiï¿½: >> >>> Hi Jose, >>> >>> here's the benchmark data obtained on my local machine running an >>> NVIDIA GTX 285 for vectors of size 100k: >>> >>> # Master >>> >>> ./ex43 -n 100000 -k 200 -mdot -log_summary >>> VecMDot 5.6363e+01 >>> >>> ./ex43 -n 100000 -k 200 -mdot -log_summary -vec_type cusp >>> VecMDot 2.1936e+01 >>> >>> ./ex43 -n 100000 -k 200 -log_summary >>> VecDot 5.1124e+01 >>> >>> ./ex43 -n 100000 -k 200 -log_summary -vec_type cusp >>> VecDot 4.0968e+01 >>> >>> >>> # Next >>> >>> ./ex43 -n 100000 -k 200 -mdot -log_summary >>> VecMDot 5.6417e+01 >>> >>> ./ex43 -n 100000 -k 200 -mdot -log_summary -vec_type cusp >>> VecMDot 1.0281e+01 >>> >>> ./ex43 -n 100000 -k 200 -log_summary >>> VecDot 5.0886e+01 >>> >>> ./ex43 -n 100000 -k 200 -log_summary -vec_type cusp >>> VecDot 4.1905e+01 >>> >>> >>> This makes 10sec in next vs. 20 sec. on master for the >>> CUDA-accelerated mdot-case. The factor of two is actually as >>> expected, because in the 'old' kernel the data movement is twice as >>> what it is in the custom kernel version. The factor of four with >>> respect to VecDot is not entirely clear to me, I'd rather expect a >>> factor close to 2. Presumably the more frequent host<-> device >>> transfers add extra overhead. >>> >>> Best regards, >>> Karli >> Here are my numbers for this size. They are similar to yours (a bit >> worse, though). Also, I tried with ViennaCL which gave very poor >> performance (is this normal?). >> >> # Master >> >> ./ex43 -n 100000 -k 200 -mdot -log_summary >> VecMDot 4.0681e+01 >> >> ./ex43 -n 100000 -k 200 -mdot -log_summary -vec_type cusp >> VecMDot 2.4489e+01 >> >> ./ex43 -n 100000 -k 200 -log_summary >> VecDot 5.9457e+01 >> >> ./ex43 -n 100000 -k 200 -log_summary -vec_type cusp >> VecDot 5.0021e+01 >> >> # Next >> >> ./ex43 -n 100000 -k 200 -mdot -log_summary >> VecMDot 4.4252e+01 >> >> ./ex43 -n 100000 -k 200 -mdot -log_summary -vec_type cusp >> VecMDot 1.2176e+01 >> >> ./ex43 -n 100000 -k 200 -log_summary >> VecDot 5.9847e+01 >> >> ./ex43 -n 100000 -k 200 -log_summary -vec_type cusp >> VecDot 5.0080e+01 >> >> # ViennaCL >> >> ./ex43 -n 100000 -k 200 -mdot -log_summary -vec_type viennacl >> VecMDot 9.4478e+01 >> >> ./ex43 -n 100000 -k 200 -log_summary -vec_type viennacl >> VecMDot 1.2311e+02 >> >> >> I tried a full SLEPc computation, with a matrix of order 256,000 and >> making VecMDot operate on 40 vectors. The gain from 'master' to 'next' >> is 91 seconds to 53 seconds. So, yes it is good improvement. Thanks. >> However, I still see only a modest speedup (about 4) with respect to >> CPU (since we do some optimizations for the CPU). Also, performance >> depends a lot on the different matrix dimensions. I have to figure out >> how to optimize it more for the GPU as well. >> >> Jose >> >

1 0

Re: [petsc-dev] VecMDot_SeqCUSP improved
by Paul Mullowney 07 Apr '13

07 Apr '13

VecMDot is performing great for my examples. About 2X faster than the trivial implementation that I originally suggested when I reported the problem. Thanks Karl. -Paul > El 26/03/2013, a las 20:15, Karl Rupp escribiï¿½: > >> Hi Jose, >> >> here's the benchmark data obtained on my local machine running an NVIDIA GTX 285 for vectors of size 100k: >> >> # Master >> >> ./ex43 -n 100000 -k 200 -mdot -log_summary >> VecMDot 5.6363e+01 >> >> ./ex43 -n 100000 -k 200 -mdot -log_summary -vec_type cusp >> VecMDot 2.1936e+01 >> >> ./ex43 -n 100000 -k 200 -log_summary >> VecDot 5.1124e+01 >> >> ./ex43 -n 100000 -k 200 -log_summary -vec_type cusp >> VecDot 4.0968e+01 >> >> >> # Next >> >> ./ex43 -n 100000 -k 200 -mdot -log_summary >> VecMDot 5.6417e+01 >> >> ./ex43 -n 100000 -k 200 -mdot -log_summary -vec_type cusp >> VecMDot 1.0281e+01 >> >> ./ex43 -n 100000 -k 200 -log_summary >> VecDot 5.0886e+01 >> >> ./ex43 -n 100000 -k 200 -log_summary -vec_type cusp >> VecDot 4.1905e+01 >> >> >> This makes 10sec in next vs. 20 sec. on master for the CUDA-accelerated mdot-case. The factor of two is actually as expected, because in the 'old' kernel the data movement is twice as what it is in the custom kernel version. The factor of four with respect to VecDot is not entirely clear to me, I'd rather expect a factor close to 2. Presumably the more frequent host<-> device transfers add extra overhead. >> >> Best regards, >> Karli > Here are my numbers for this size. They are similar to yours (a bit worse, though). Also, I tried with ViennaCL which gave very poor performance (is this normal?). > > # Master > > ./ex43 -n 100000 -k 200 -mdot -log_summary > VecMDot 4.0681e+01 > > ./ex43 -n 100000 -k 200 -mdot -log_summary -vec_type cusp > VecMDot 2.4489e+01 > > ./ex43 -n 100000 -k 200 -log_summary > VecDot 5.9457e+01 > > ./ex43 -n 100000 -k 200 -log_summary -vec_type cusp > VecDot 5.0021e+01 > > # Next > > ./ex43 -n 100000 -k 200 -mdot -log_summary > VecMDot 4.4252e+01 > > ./ex43 -n 100000 -k 200 -mdot -log_summary -vec_type cusp > VecMDot 1.2176e+01 > > ./ex43 -n 100000 -k 200 -log_summary > VecDot 5.9847e+01 > > ./ex43 -n 100000 -k 200 -log_summary -vec_type cusp > VecDot 5.0080e+01 > > # ViennaCL > > ./ex43 -n 100000 -k 200 -mdot -log_summary -vec_type viennacl > VecMDot 9.4478e+01 > > ./ex43 -n 100000 -k 200 -log_summary -vec_type viennacl > VecMDot 1.2311e+02 > > > I tried a full SLEPc computation, with a matrix of order 256,000 and making VecMDot operate on 40 vectors. The gain from 'master' to 'next' is 91 seconds to 53 seconds. So, yes it is good improvement. Thanks. However, I still see only a modest speedup (about 4) with respect to CPU (since we do some optimizations for the CPU). Also, performance depends a lot on the different matrix dimensions. I have to figure out how to optimize it more for the GPU as well. > > Jose >

1 0

Re: [petsc-dev] macro wrappers around functions to caste suck
by Jed Brown 07 Apr '13

07 Apr '13

Barry Smith <bsmith(a)mcs.anl.gov> writes: > Presumably the cast for function pointers IS needed for certain > C++ compilers and cannot (or should not?) be done manually by > users so we would do the trick for those? Yeah, that cast to (PetscVoidFunction*) is clumsy and we really want to interpret the function as untyped. Some compilers will warn despite the cast if a user tries to pass something that is not a function pointer, thus I think the hidden cast is definitely the right thing there. The PetscObject argument is far from opaque (function pointer or non) data so I think a cast there is more dubious.

1 0

Re: [petsc-dev] macro wrappers around functions to caste suck
by Barry Smith 07 Apr '13

07 Apr '13

On Apr 7, 2013, at 5:02 PM, Jed Brown <jedbrown(a)mcs.anl.gov> wrote: > Barry Smith <bsmith(a)mcs.anl.gov> writes: > >> Seems inconsistent to only handle the cast automatically for >> some. We should do it all or nothing. That is be C++ish and let the >> user always pass a subclass where the class is expected or be C ish >> and always require the cast. > > Unfortunately, there is no way to allow implicit conversion for > subclasses without implicitly converting everything. True, people could pass any crazy thing in there. The only hope is that the runtime checking of PetscObjects would get any non-PetscObjects passed in. > The explicit cast > is visually jarring enough to make it obvious at the call site that that > argument must be a PetscObject. I'm pretty indifferent about changing > it. I am fine with leaving the code as is at this time. Presumably the cast for function pointers IS needed for certain C++ compilers and cannot (or should not?) be done manually by users so we would do the trick for those? Barry

1 0

Re: [petsc-dev] macro wrappers around functions to caste suck
by Jed Brown 07 Apr '13

07 Apr '13

Barry Smith <bsmith(a)mcs.anl.gov> writes: > Seems inconsistent to only handle the cast automatically for > some. We should do it all or nothing. That is be C++ish and let the > user always pass a subclass where the class is expected or be C ish > and always require the cast. Unfortunately, there is no way to allow implicit conversion for subclasses without implicitly converting everything. The explicit cast is visually jarring enough to make it obvious at the call site that that argument must be a PetscObject. I'm pretty indifferent about changing it.

1 0