February 2012 - petsc-dev - mailman3.cels.anl.gov

Re: [petsc-dev] Merge sets
by Jed Brown 18 Feb '12

18 Feb '12

On Sat, Feb 18, 2012 at 16:18, Matthew Knepley <knepley(a)gmail.com> wrote: > I have lost the BB battle, however can we at least start using > > hg pull --rebase > > so that we avoid this > > https://bitbucket.org/petsc/petsc-dev/changeset/ad9064ecab66 > This is a visualization issue. $ hg diff -r 63b68cc62c3d..ad9064ecab66 diff --git a/src/ksp/pc/impls/mg/mg.c b/src/ksp/pc/impls/mg/mg.c --- a/src/ksp/pc/impls/mg/mg.c +++ b/src/ksp/pc/impls/mg/mg.c @@ -1026,17 +1026,22 @@ . -pc_mg_type <additive,multiplicative,full,cascade> - multiplicative is the default . -pc_mg_log - log information about time spent on each level of the solver . -pc_mg_monitor - print information on the multigrid convergence -. -pc_mg_galerkin - use Galerkin process to compute coarser operators -- -pc_mg_dump_matlab - dumps the matrices for each level and the restriction/interpolation matrices +. -pc_mg_galerkin - use Galerkin process to compute coarser operators, i.e. Acoarse = R A R' +. -pc_mg_multiplicative_cycles - number of cycles to use as the preconditioner (defaults to 1) +. -pc_mg_dump_matlab - dumps the matrices for each level and the restriction/interpolation matrices to the Socket viewer for reading from MATLAB. +- -pc_mg_dump_binary - dumps the matrices for each level and the restriction/interpolation matrices + to the binary output file called binaryoutput Notes: By default this uses GMRES on the fine grid smoother so this should be used with KSPFGMRES or the smoother changed to not use GMRES + When run with a single level the smoother options are used on that level NOT the coarse grid solver options + Level: intermediate Concepts: multigrid/multilevel -.seealso: PCCreate(), PCSetType(), PCType (for list of available types), PC, PCMGType, PCEXOTIC +.seealso: PCCreate(), PCSetType(), PCType (for list of available types), PC, PCMGType, PCEXOTIC, PCGAMG, PCML, PCHYPRE PCMGSetLevels(), PCMGGetLevels(), PCMGSetType(), PCMGSetCycleType(), PCMGSetNumberSmoothDown(), PCMGSetNumberSmoothUp(), PCMGGetCoarseSolve(), PCMGSetResidual(), PCMGSetInterpolation(), PCMGSetRestriction(), PCMGGetSmoother(), PCMGGetSmootherUp(), PCMGGetSmootherDown(),

1 0

Merge sets
by Matthew Knepley 18 Feb '12

18 Feb '12

I have lost the BB battle, however can we at least start using hg pull --rebase so that we avoid this https://bitbucket.org/petsc/petsc-dev/changeset/ad9064ecab66 Matt -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener

1 0

Re: [petsc-dev] DMGetMatrix --> DMGetMatrices?
by Barry Smith 18 Feb '12

18 Feb '12

On Feb 17, 2012, at 10:12 PM, Dmitry Karpeev wrote: > The need for computation reuse (between J and J_pre, for example) might require a change in the SNES callback model. > Right now each call -- FormFunction, FormJacobian -- is stateless. It takes a Vec X and computes the corresponding quantities -- residual, J, J_pre -- solely from X. Any callback that wanted to reuse any other callback's computation would have to be able to make sure the reuse is safe: am I being called with the same X as this previous callback that already precomputed a bunch of stuff for me? This is only possible with a costly, collective (and imprecise) vector comparison, or a fragile state tracker within X. If this is unacceptable, the callbacks must carry (and share) a state, and, thus, become objects. > > That's probably undesirable within the basic SNES callback model, but may be natural within the DM-based callback model: a SNES sets its DM's Vec X and calls DMFunction, DMJacobian without any argument -- those calls pull out X and compute whatever they need, including some common data for others (e.g., PC DM) to reuse. A new call to DMSetVec() will reset these common data structures. This way the DM can also be viewed as a "factory". > > This would probably make my interaction with libMesh a bit easier, but isn't strictly necessary. Tell us what is making your interactions with libMesh less easy. That is, tell us your problem, not your proposed solution to the problem, what specifically are you trying to do with libMesh and why is it less easy then you would like. Barry > Still, it would be good to know any thoughts on this. > > Dmitry. > > On Sun, Feb 12, 2012 at 10:32 AM, Dmitry Karpeev <karpeev(a)mcs.anl.gov> wrote: > > > On Sat, Feb 11, 2012 at 8:36 AM, Jed Brown <jedbrown(a)mcs.anl.gov> wrote: > On Sat, Feb 11, 2012 at 08:32, Dmitry Karpeev <karpeev(a)mcs.anl.gov> wrote: > This sounds fine to me, but what's the API for doing this? You want to do KSPSetDM() and PCSetDM(), > so that the user has to explicitly pull out the PC to set its DM? > > The PC *always* owns both matrices anyway. > > I think there is a rather natural API for handling DM grouping, code reuse and dealing with the issue of single matrix per DM: > DM should support DMGetKSPDM(), DMGetPCDM() -- similar to DMDAGetCoordinateDA() -- each of which will compute the correct matrix. These calls can return the original DM (or PETSC_NULL) by default. SNES will pull out the KSP DM out to set it on the KSP, KSP will do the same for the PC, etc. DM already has Vec x, so all of the DMs can share x and compute their respective matrices at the same x and reuse whatever computation is reusable. That's up to the implementation, though. > > Dmitry. > >

1 0

Re: [petsc-dev] DMGetMatrix --> DMGetMatrices?
by Dmitry Karpeev 18 Feb '12

18 Feb '12

The *API* guarantees that? Once we have more complicated reuse (e.g., between J and J_pre) it might be trickier. MG setup already stashes the Vec in the top-level DM to reuse it when constructing the different levels. Dmitry. On Sat, Feb 18, 2012 at 5:21 AM, Jed Brown <jedbrown(a)mcs.anl.gov> wrote: > The current API guarantees the FormFunction is called before FormJacobian. > Do you have a use case where that is not satisfied? > On Feb 17, 2012 11:13 PM, "Dmitry Karpeev" <karpeev(a)mcs.anl.gov> wrote: > >> The need for computation reuse (between J and J_pre, for example) might >> require a change in the SNES callback model. >> Right now each call -- FormFunction, FormJacobian -- is stateless. It >> takes a Vec X and computes the corresponding quantities -- residual, J, >> J_pre -- solely from X. Any callback that wanted to reuse any other >> callback's computation would have to be able to make sure the reuse is >> safe: am I being called with the same X as this previous callback that >> already precomputed a bunch of stuff for me? This is only possible with a >> costly, collective (and imprecise) vector comparison, or a fragile state >> tracker within X. If this is unacceptable, the callbacks must carry (and >> share) a state, and, thus, become objects. >> >> That's probably undesirable within the basic SNES callback model, but may >> be natural within the DM-based callback model: a SNES sets its DM's Vec X >> and calls DMFunction, DMJacobian without any argument -- those calls pull >> out X and compute whatever they need, including some common data for others >> (e.g., PC DM) to reuse. A new call to DMSetVec() will reset these common >> data structures. This way the DM can also be viewed as a "factory". >> >> This would probably make my interaction with libMesh a bit easier, but >> isn't strictly necessary. Still, it would be good to know any thoughts on >> this. >> >> Dmitry. >> >> On Sun, Feb 12, 2012 at 10:32 AM, Dmitry Karpeev <karpeev(a)mcs.anl.gov>wrote: >> >>> >>> >>> On Sat, Feb 11, 2012 at 8:36 AM, Jed Brown <jedbrown(a)mcs.anl.gov> wrote: >>> >>>> On Sat, Feb 11, 2012 at 08:32, Dmitry Karpeev <karpeev(a)mcs.anl.gov>wrote: >>>> >>>>> This sounds fine to me, but what's the API for doing this? You want to >>>>> do KSPSetDM() and PCSetDM(), >>>>> so that the user has to explicitly pull out the PC to set its DM? >>>>> >>>> >>>> The PC *always* owns both matrices anyway. >>>> >>> >>> I think there is a rather natural API for handling DM grouping, code >>> reuse and dealing with the issue of single matrix per DM: >>> DM should support DMGetKSPDM(), DMGetPCDM() -- similar to >>> DMDAGetCoordinateDA() -- each of which will compute the correct matrix. >>> These calls can return the original DM (or PETSC_NULL) by default. SNES >>> will pull out the KSP DM out to set it on the KSP, KSP will do the same for >>> the PC, etc. DM already has Vec x, so all of the DMs can share x and >>> compute their respective matrices at the same x and reuse whatever >>> computation is reusable. That's up to the implementation, though. >>> >>> Dmitry. >>> >>> >>

1 0

Re: [petsc-dev] DMGetMatrix --> DMGetMatrices?
by Jed Brown 18 Feb '12

18 Feb '12

The current API guarantees the FormFunction is called before FormJacobian. Do you have a use case where that is not satisfied? On Feb 17, 2012 11:13 PM, "Dmitry Karpeev" <karpeev(a)mcs.anl.gov> wrote: > The need for computation reuse (between J and J_pre, for example) might > require a change in the SNES callback model. > Right now each call -- FormFunction, FormJacobian -- is stateless. It > takes a Vec X and computes the corresponding quantities -- residual, J, > J_pre -- solely from X. Any callback that wanted to reuse any other > callback's computation would have to be able to make sure the reuse is > safe: am I being called with the same X as this previous callback that > already precomputed a bunch of stuff for me? This is only possible with a > costly, collective (and imprecise) vector comparison, or a fragile state > tracker within X. If this is unacceptable, the callbacks must carry (and > share) a state, and, thus, become objects. > > That's probably undesirable within the basic SNES callback model, but may > be natural within the DM-based callback model: a SNES sets its DM's Vec X > and calls DMFunction, DMJacobian without any argument -- those calls pull > out X and compute whatever they need, including some common data for others > (e.g., PC DM) to reuse. A new call to DMSetVec() will reset these common > data structures. This way the DM can also be viewed as a "factory". > > This would probably make my interaction with libMesh a bit easier, but > isn't strictly necessary. Still, it would be good to know any thoughts on > this. > > Dmitry. > > On Sun, Feb 12, 2012 at 10:32 AM, Dmitry Karpeev <karpeev(a)mcs.anl.gov>wrote: > >> >> >> On Sat, Feb 11, 2012 at 8:36 AM, Jed Brown <jedbrown(a)mcs.anl.gov> wrote: >> >>> On Sat, Feb 11, 2012 at 08:32, Dmitry Karpeev <karpeev(a)mcs.anl.gov>wrote: >>> >>>> This sounds fine to me, but what's the API for doing this? You want to >>>> do KSPSetDM() and PCSetDM(), >>>> so that the user has to explicitly pull out the PC to set its DM? >>>> >>> >>> The PC *always* owns both matrices anyway. >>> >> >> I think there is a rather natural API for handling DM grouping, code >> reuse and dealing with the issue of single matrix per DM: >> DM should support DMGetKSPDM(), DMGetPCDM() -- similar to >> DMDAGetCoordinateDA() -- each of which will compute the correct matrix. >> These calls can return the original DM (or PETSC_NULL) by default. SNES >> will pull out the KSP DM out to set it on the KSP, KSP will do the same for >> the PC, etc. DM already has Vec x, so all of the DMs can share x and >> compute their respective matrices at the same x and reuse whatever >> computation is reusable. That's up to the implementation, though. >> >> Dmitry. >> >> >

1 0

Re: [petsc-dev] DMGetMatrix --> DMGetMatrices?
by Dmitry Karpeev 17 Feb '12

17 Feb '12

The need for computation reuse (between J and J_pre, for example) might require a change in the SNES callback model. Right now each call -- FormFunction, FormJacobian -- is stateless. It takes a Vec X and computes the corresponding quantities -- residual, J, J_pre -- solely from X. Any callback that wanted to reuse any other callback's computation would have to be able to make sure the reuse is safe: am I being called with the same X as this previous callback that already precomputed a bunch of stuff for me? This is only possible with a costly, collective (and imprecise) vector comparison, or a fragile state tracker within X. If this is unacceptable, the callbacks must carry (and share) a state, and, thus, become objects. That's probably undesirable within the basic SNES callback model, but may be natural within the DM-based callback model: a SNES sets its DM's Vec X and calls DMFunction, DMJacobian without any argument -- those calls pull out X and compute whatever they need, including some common data for others (e.g., PC DM) to reuse. A new call to DMSetVec() will reset these common data structures. This way the DM can also be viewed as a "factory". This would probably make my interaction with libMesh a bit easier, but isn't strictly necessary. Still, it would be good to know any thoughts on this. Dmitry. On Sun, Feb 12, 2012 at 10:32 AM, Dmitry Karpeev <karpeev(a)mcs.anl.gov>wrote: > > > On Sat, Feb 11, 2012 at 8:36 AM, Jed Brown <jedbrown(a)mcs.anl.gov> wrote: > >> On Sat, Feb 11, 2012 at 08:32, Dmitry Karpeev <karpeev(a)mcs.anl.gov>wrote: >> >>> This sounds fine to me, but what's the API for doing this? You want to >>> do KSPSetDM() and PCSetDM(), >>> so that the user has to explicitly pull out the PC to set its DM? >>> >> >> The PC *always* owns both matrices anyway. >> > > I think there is a rather natural API for handling DM grouping, code reuse > and dealing with the issue of single matrix per DM: > DM should support DMGetKSPDM(), DMGetPCDM() -- similar to > DMDAGetCoordinateDA() -- each of which will compute the correct matrix. > These calls can return the original DM (or PETSC_NULL) by default. SNES > will pull out the KSP DM out to set it on the KSP, KSP will do the same for > the PC, etc. DM already has Vec x, so all of the DMs can share x and > compute their respective matrices at the same x and reuse whatever > computation is reusable. That's up to the implementation, though. > > Dmitry. > >

1 0

Re: [petsc-dev] [petsc-maint #106126] petsc-dev not giving parmetis mpi information JED READ THIS
by Jed Brown 17 Feb '12

17 Feb '12

On Thu, Feb 16, 2012 at 23:15, Sean Farley <sean.michael.farley(a)gmail.com>wrote: > Yes, this is what I was working on but my time got blown away by other > things. Using the parmetis makefile is ridiculous. > Agreed, I have no idea why upstream decided to do it that way.

1 0

Re: [petsc-dev] [petsc-maint #106126] petsc-dev not giving parmetis mpi information JED READ THIS
by Sean Farley 16 Feb '12

16 Feb '12

Yes, this is what I was working on but my time got blown away by other things. Using the parmetis makefile is ridiculous. On Thursday, February 16, 2012, Barry Smith wrote: > > PETSc-folks > > Actually why does parmetis.py even use that stupid gnumake file, we > could change parmetis.py to issue the cmake builder command directly and > then the make on the cmake generated makefile also directly and totally > bypass that absurd makefile. Then we can have better control over what gets > passed to the cmake builder and can pass down easily the needed include > directories for MPI > > Barry > > On Feb 16, 2012, at 10:00 PM, Barry Smith wrote: > > > > > John, > > > > So I looked in the parmetis Makefile and it looks like it has no > direct mechanism to pass in MPI include file directories etc. I do not know > much about cmake the only way I could see getting that information in is via > > > > ifneq ($(cflags), not-set) > > CONFIG_FLAGS += -DCMAKE_C_FLAGS:STRING="$(cflags)" > > endif > > > > but I don't know if this is a legitimate way in cmake to pass in include > directories to cmake. JED????? > > > > If the machine has MPI compiler wrappers (like most systems do) and you > used that rather than passing icc and the MPI information to PETSc's > ./configure that would also work. > > > > > > Barry > > > > > > On Feb 16, 2012, at 11:58 AM, John Fettig wrote: > > > >> Hello PETSc gurus, > >> > >> I'm trying to install petsc-dev using --download-parmetis=1, and it > isn't > >> passing any info that I give it about MPI to the parmetis configure. > >> Looking at config/PETSc/packages/parmetis.py, it doesn't look like > there is > >> any attempt made to pass along this info. Am I doing something wrong? > >> (configure.log attached) > >> > >> Thanks, > >> John > >> > >> <configure.log.gz> > > > >

1 0

Re: [petsc-dev] [petsc-maint #106126] petsc-dev not giving parmetis mpi information JED READ THIS
by Barry Smith 16 Feb '12

16 Feb '12

PETSc-folks Actually why does parmetis.py even use that stupid gnumake file, we could change parmetis.py to issue the cmake builder command directly and then the make on the cmake generated makefile also directly and totally bypass that absurd makefile. Then we can have better control over what gets passed to the cmake builder and can pass down easily the needed include directories for MPI Barry On Feb 16, 2012, at 10:00 PM, Barry Smith wrote: > > John, > > So I looked in the parmetis Makefile and it looks like it has no direct mechanism to pass in MPI include file directories etc. I do not know much about cmake the only way I could see getting that information in is via > > ifneq ($(cflags), not-set) > CONFIG_FLAGS += -DCMAKE_C_FLAGS:STRING="$(cflags)" > endif > > but I don't know if this is a legitimate way in cmake to pass in include directories to cmake. JED????? > > If the machine has MPI compiler wrappers (like most systems do) and you used that rather than passing icc and the MPI information to PETSc's ./configure that would also work. > > > Barry > > > On Feb 16, 2012, at 11:58 AM, John Fettig wrote: > >> Hello PETSc gurus, >> >> I'm trying to install petsc-dev using --download-parmetis=1, and it isn't >> passing any info that I give it about MPI to the parmetis configure. >> Looking at config/PETSc/packages/parmetis.py, it doesn't look like there is >> any attempt made to pass along this info. Am I doing something wrong? >> (configure.log attached) >> >> Thanks, >> John >> >> <configure.log.gz> >

1 0

OpenMP/Vec
by Gerard Gorman 16 Feb '12

16 Feb '12

Hi I have been running benchmarks on the OpenMP branch of petsc-dev on an Intel Westmere (Intel(R) Xeon(R) CPU X5670 @ 2.93GHz). You can see all graphs + test code + code to generate results in the tar ball linked below and I am just going to give a quick summary here. http://amcg.ese.ic.ac.uk/~ggorman/omp_vec_benchmarks.tar.gz There are 3 sets of results: gcc/ : GCC 4.6 intel/ : Intel 12.0 with MKL intel-pinning/ : as above put applying hard affinity. Files matching mpi_*.pdf show the MPI speedup and parallel efficiency for a range of vector sizes. Similarly for omp_*.pdf with respect to OpenMP. The remaining files directly compare scaling of MPI Vs OpenMP for the various tests for the largest vector size. I think the results are very encouraging and there are many interesting little details in there. I am just going to summarise a few here that I think are particularly important. 1. In most cases the threaded code performs as well as, and in many cases better then the mpi code. 2. For GCC I did not use a threaded blas. For Intel I used -lmkl_intel_thread. However, it appears dnrm2 is not threaded. It seems to be a common feature among other threaded blas libraries that Level 1 is not completely threaded (e.g. cray). Unfortunately most of this is experience/anecdotal information. I do not know of any proper survey. We have the option here of either rolling our own or ignoring the issue until profiling shows it is a problem...and eventually someone else will release a fully threaded blas. 3. Comparing intel/ and intel-pinning/ is particularly interesting. "First touch" has been applied to all memory in VecCreate so that memory should be paged correctly for NUMA. But first touch does not gain you much if threads migrate, so for the intel-pinning/ results I set the env KMP_AFFINITY=scatter to get hard affinity. You can clearly from the results that this improves parallel efficiency by a few percentage points in many cases. It also really smooths out efficiency dips as you run on different number of threads. Full blown benchmarks would not make a lot of sense until we get the Mat classes threaded in a similar fashion. However, at this point I would like feedback on the direction this is taking and if we can start getting code committed. Cheers Gerard

1 0