April 2018 - petsc-dev - mailman3.cels.anl.gov

Re: [petsc-dev] [issue1595] Issues of limited number of MPI communicators when having many instances of hypre boomerAMG with Moose
by Yang, Ulrike Meier 03 Apr '18

03 Apr '18

The sub-communicator on the coarsest grid is generally not of size 1. You can avoid using this version with the sub communicator by setting the smoother/solver on the coarse grid to a different one using HYPRE_BoomerAMGSetCycleRelaxtype( <solver>, relax_type, 3); If you set relax_type to 19, it will use the old version for Gaussian elimination. You could also set it to a smoother, e.g. 6, which would be 1 sweep of SGS. Ulrike -----Original Message----- From: Smith, Barry F. <bsmith(a)mcs.anl.gov> Sent: Tuesday, April 3, 2018 12:56 PM To: hypre-support <hypre-support(a)llnl.gov> Cc: fande.kong(a)inl.gov; friedmud(a)gmail.com; Li, Ruipeng <li50(a)llnl.gov>; Osei-Kuffuor, Daniel <oseikuffuor1(a)llnl.gov>; petsc-dev(a)mcs.anl.gov; Schroder, Jacob B. <schroder2(a)llnl.gov>; tzanio(a)llnl.gov; umyang(a)llnl.gov; Wang, Lu <wang84(a)llnl.gov> Subject: Re: [issue1595] Issues of limited number of MPI communicators when having many instances of hypre boomerAMG with Moose > On Apr 3, 2018, at 1:46 PM, Rob Falgout hypre Tracker <hypre-support(a)llnl.gov> wrote: > > > Rob Falgout <rfalgout(a)llnl.gov> added the comment: > > Hi Barry, > > It looks like the only time we call MPI_Comm_create is to build a communicator for the coarsest grid solve using Gaussian elimination. There are probably alternatives that do not require creating a sub-communicator. When the sub communicator is of size 1 you can use MPI_COMM_SELF instead of creating a new communicator each time. > Ulrike or someone else more familiar with the code should comment. > > I don't see a need to do a Comm_dup() before calling hypre. If I have 1000 hypre solvers on the same communicator how do I know that hypre won't send messages between the different solvers and hence gets messed up? In other words how do you handle tags to prevent conflicts between different matrices? What communicators do you actually do communication on and where do you get them? From the hypre matrix? > > Hope this helps. > > -Rob > > ---------- > status: unread -> chatting > > ____________________________________________ > hypre Issue Tracker <hypre-support(a)llnl.gov> > <http://cascb1.llnl.gov/hypre/issue1595> > ____________________________________________

1 0

Re: [petsc-dev] [issue1595] Issues of limited number of MPI communicators when having many instances of hypre boomerAMG with Moose
by Smith, Barry F. 03 Apr '18

03 Apr '18

> On Apr 3, 2018, at 1:46 PM, Rob Falgout hypre Tracker <hypre-support(a)llnl.gov> wrote: > > > Rob Falgout <rfalgout(a)llnl.gov> added the comment: > > Hi Barry, > > It looks like the only time we call MPI_Comm_create is to build a communicator for the coarsest grid solve using Gaussian elimination. There are probably alternatives that do not require creating a sub-communicator. When the sub communicator is of size 1 you can use MPI_COMM_SELF instead of creating a new communicator each time. > Ulrike or someone else more familiar with the code should comment. > > I don't see a need to do a Comm_dup() before calling hypre. If I have 1000 hypre solvers on the same communicator how do I know that hypre won't send messages between the different solvers and hence gets messed up? In other words how do you handle tags to prevent conflicts between different matrices? What communicators do you actually do communication on and where do you get them? From the hypre matrix? > > Hope this helps. > > -Rob > > ---------- > status: unread -> chatting > > ____________________________________________ > hypre Issue Tracker <hypre-support(a)llnl.gov> > <http://cascb1.llnl.gov/hypre/issue1595> > ____________________________________________

1 0

[issue1595] Issues of limited number of MPI communicators when having many instances of hypre boomerAMG with Moose
by Rob Falgout hypre Tracker 03 Apr '18

03 Apr '18

Rob Falgout <rfalgout(a)llnl.gov> added the comment: Hi Barry, It looks like the only time we call MPI_Comm_create is to build a communicator for the coarsest grid solve using Gaussian elimination. There are probably alternatives that do not require creating a sub-communicator. Ulrike or someone else more familiar with the code should comment. I don't see a need to do a Comm_dup() before calling hypre. Hope this helps. -Rob ---------- status: unread -> chatting ____________________________________________ hypre Issue Tracker <hypre-support(a)llnl.gov> <http://cascb1.llnl.gov/hypre/issue1595> ____________________________________________

1 0

Issues of limited number of MPI communicators when having many instances of hypre boomerAMG with Moose
by Smith, Barry F. 03 Apr '18

03 Apr '18

hypre developers, Some of the MPI implementations only support around 2000 MPI communicators, this can cause difficulties for MOOSE users who have many instances of hypre BoomerAMG at the same time. Thus I have some questions for you, For each HYPRE matrix we do a MPI_Comm_dup() to make sure that hypre has a unique communicator that won't conflict with PETSc communicators or other hypre communicators associated with other matrices. Do we need to do this? Is hypre smart enough that we could pass the same communicator with all the matrices (assuming the matrices live on the same communicator)? It appears that each BoomerAMG instance calls MPI_Comm_create() down in the guts (even on one process where the new communicator is the same size as the original communicator). Is this necessary? Could you detect when the sub-communicator is the same size as the original communicator and then skip the MPI_Comm_create()? Due to the limitations above MOOSE users are limited to about 1,000 instances of hypre BoomerAMG at the same time Any thoughts on how this limitation could be increased? Thanks Barry

1 0

Re: [petsc-dev] Is ./configure --help broken?
by Satish Balay 02 Apr '18

02 Apr '18

On Mon, 2 Apr 2018, Satish Balay wrote: > On Fri, 16 Mar 2018, Satish Balay wrote: > > > On Fri, 16 Mar 2018, Matthew Knepley wrote: > > > > > On Fri, Mar 16, 2018 at 9:26 AM, Satish Balay <balay(a)mcs.anl.gov> wrote: > > > > > > > On Fri, 16 Mar 2018, Matthew Knepley wrote: > > > > > > > > > This works for me. Do you want to put it in your branch, or should I > > > > check > > > > > it in? > > > > > > > > > > Matt > > > > > > > > > > diff --git a/config/BuildSystem/RDict.py b/config/BuildSystem/RDict.py > > > > > index 2dc745f..efc8b81 100755 > > > > > --- a/config/BuildSystem/RDict.py > > > > > +++ b/config/BuildSystem/RDict.py > > > > > @@ -218,7 +218,7 @@ Arg class, which wraps the usual value.''' > > > > > print > > > > > '----------------------------------------------------------- > > > > ------------' > > > > > pass > > > > > dict.__setitem__(self, key, value) > > > > > - self.save() > > > > > + self.save(force = 0) > > > > > else: > > > > > return self.send(key, value) > > > > > return > > > > > > > > How about reducing the number of times save is called - i.e try > > > > calling it only once at the end. Would that work? > > > > > > > > diff --git a/config/BuildSystem/RDict.py b/config/BuildSystem/RDict.py > > > > index 2dc745f1aa..98a2a16a29 100755 > > > > --- a/config/BuildSystem/RDict.py > > > > +++ b/config/BuildSystem/RDict.py > > > > @@ -195,7 +195,7 @@ Arg class, which wraps the usual value.''' > > > > else: > > > > self.writeLogLine('__getitem__: Setting local type for '+key) > > > > dict.__setitem__(self, key, nargs.Arg(key)) > > > > - self.save() > > > > + #self.save() > > > > self.writeLogLine('__getitem__: Setting local value for '+key) > > > > return dict.__getitem__(self, key).getValue() > > > > > > > > @@ -218,7 +218,7 @@ Arg class, which wraps the usual value.''' > > > > print '----------------------------- > > > > ------------------------------------------' > > > > pass > > > > dict.__setitem__(self, key, value) > > > > - self.save() > > > > + #self.save() > > > > else: > > > > return self.send(key, value) > > > > return > > > > @@ -232,14 +232,14 @@ Arg class, which wraps the usual value.''' > > > > dict.__setitem__(self, key, nargs.Arg(key)) > > > > dict.__getitem__(self, key).setValue(value) > > > > self.writeLogLine('__setitem__: Set value for '+key+' to > > > > '+str(dict.__getitem__(self, key))) > > > > - self.save() > > > > + #self.save() > > > > return > > > > > > > > def __delitem__(self, key): > > > > '''Checks for the key locally, and if not found consults the parent. > > > > Deletes the Arg completely.''' > > > > if dict.has_key(self, key): > > > > dict.__delitem__(self, key) > > > > - self.save() > > > > + #self.save() > > > > elif not self.parent is None: > > > > self.send(key) > > > > return > > > > @@ -248,7 +248,7 @@ Arg class, which wraps the usual value.''' > > > > '''Clears both the local and parent dictionaries''' > > > > if dict.__len__(self): > > > > dict.clear(self) > > > > - self.save() > > > > + #self.save() > > > > if not self.parent is None: > > > > self.send() > > > > return > > > > > > > > My simple 'configure --prefix && make && make install' test appears to > > > > work with the above change - but don't know if the random crashes in > > > > nightlybuilds will come back or not.. > > > > > > > > > I do not want to do that until its rewritten. There is already a save call > > > at the end which was not working. > > > > I suspect the issue is with the skip timmer with save(force=0) sometimes interfering with the safe(force=1) call at the end. > > > > Hence my attempt to eliminate save(force=0) calls.. > > We haven't yet resolved this issue. I'd like to get a fix in which won't bring back the random Rdict.db errors.. I added my patch to balay/fix-slow-help and merged to next for testing.. Satish

1 0

Re: [petsc-dev] upcoming release and testing
by Karl Rupp 02 Apr '18

02 Apr '18

Hi Satish, CUDA and ViennaCL can be enabled for the first time in this release. Best regards, Karli On 04/02/2018 08:15 PM, Satish Balay wrote: > Karl, > > Are we disabling CUDA usage for this release aswell? > > Thanks, > Satish >

1 0

Re: [petsc-dev] futurize
by Jed Brown 02 Apr '18

02 Apr '18

Lisandro Dalcin <dalcinl(a)gmail.com> writes: > On 2 April 2018 at 23:08, Jed Brown <jed(a)jedbrown.org> wrote: >> Would it be too disruptive to run futurize "stage 1" before the 3.9 >> release? > > Would this introduce any dependency outside plain Python 2 stdlib? No. These are the only new imports. +from functools import reduce +from __future__ import absolute_import +from __future__ import print_function https://bitbucket.org/petsc/petsc/pull-requests/920/python-futurize-stage-1… "Stage 2" (not done here) would add a dependency on "future" (or some manual work to remove it, mostly with strings). > Sorry, I'm not an expert in these futurize/modernize/six/whatever > stuff, I got used to write Py 2/3 compatible code from scratch without > these tools. > >> The source would remain Python-2 only, but would be >> "modernized" so that the next stage (supporting Python-3) would be less >> disruptive. The goal here would be to make it easier to port >> configuration changes from 'maint' to 'master' after the release, under >> the assumption that we'll want to support Python-3 before a 3.10 >> release. > > Well, while I'm a bit worried about dependencies, I think your goal > makes total sense and I would support it. > > > -- > Lisandro Dalcin > ============ > Research Scientist > Computer, Electrical and Mathematical Sciences & Engineering (CEMSE) > Extreme Computing Research Center (ECRC) > King Abdullah University of Science and Technology (KAUST) > http://ecrc.kaust.edu.sa/ > > 4700 King Abdullah University of Science and Technology > al-Khawarizmi Bldg (Bldg 1), Office # 0109 > Thuwal 23955-6900, Kingdom of Saudi Arabia > http://www.kaust.edu.sa > > Office Phone: +966 12 808-0459

1 0

Re: [petsc-dev] Is ./configure --help broken?
by Satish Balay 02 Apr '18

02 Apr '18

On Fri, 16 Mar 2018, Satish Balay wrote: > On Fri, 16 Mar 2018, Matthew Knepley wrote: > > > On Fri, Mar 16, 2018 at 9:26 AM, Satish Balay <balay(a)mcs.anl.gov> wrote: > > > > > On Fri, 16 Mar 2018, Matthew Knepley wrote: > > > > > > > This works for me. Do you want to put it in your branch, or should I > > > check > > > > it in? > > > > > > > > Matt > > > > > > > > diff --git a/config/BuildSystem/RDict.py b/config/BuildSystem/RDict.py > > > > index 2dc745f..efc8b81 100755 > > > > --- a/config/BuildSystem/RDict.py > > > > +++ b/config/BuildSystem/RDict.py > > > > @@ -218,7 +218,7 @@ Arg class, which wraps the usual value.''' > > > > print > > > > '----------------------------------------------------------- > > > ------------' > > > > pass > > > > dict.__setitem__(self, key, value) > > > > - self.save() > > > > + self.save(force = 0) > > > > else: > > > > return self.send(key, value) > > > > return > > > > > > How about reducing the number of times save is called - i.e try > > > calling it only once at the end. Would that work? > > > > > > diff --git a/config/BuildSystem/RDict.py b/config/BuildSystem/RDict.py > > > index 2dc745f1aa..98a2a16a29 100755 > > > --- a/config/BuildSystem/RDict.py > > > +++ b/config/BuildSystem/RDict.py > > > @@ -195,7 +195,7 @@ Arg class, which wraps the usual value.''' > > > else: > > > self.writeLogLine('__getitem__: Setting local type for '+key) > > > dict.__setitem__(self, key, nargs.Arg(key)) > > > - self.save() > > > + #self.save() > > > self.writeLogLine('__getitem__: Setting local value for '+key) > > > return dict.__getitem__(self, key).getValue() > > > > > > @@ -218,7 +218,7 @@ Arg class, which wraps the usual value.''' > > > print '----------------------------- > > > ------------------------------------------' > > > pass > > > dict.__setitem__(self, key, value) > > > - self.save() > > > + #self.save() > > > else: > > > return self.send(key, value) > > > return > > > @@ -232,14 +232,14 @@ Arg class, which wraps the usual value.''' > > > dict.__setitem__(self, key, nargs.Arg(key)) > > > dict.__getitem__(self, key).setValue(value) > > > self.writeLogLine('__setitem__: Set value for '+key+' to > > > '+str(dict.__getitem__(self, key))) > > > - self.save() > > > + #self.save() > > > return > > > > > > def __delitem__(self, key): > > > '''Checks for the key locally, and if not found consults the parent. > > > Deletes the Arg completely.''' > > > if dict.has_key(self, key): > > > dict.__delitem__(self, key) > > > - self.save() > > > + #self.save() > > > elif not self.parent is None: > > > self.send(key) > > > return > > > @@ -248,7 +248,7 @@ Arg class, which wraps the usual value.''' > > > '''Clears both the local and parent dictionaries''' > > > if dict.__len__(self): > > > dict.clear(self) > > > - self.save() > > > + #self.save() > > > if not self.parent is None: > > > self.send() > > > return > > > > > > My simple 'configure --prefix && make && make install' test appears to > > > work with the above change - but don't know if the random crashes in > > > nightlybuilds will come back or not.. > > > > > > I do not want to do that until its rewritten. There is already a save call > > at the end which was not working. > > I suspect the issue is with the skip timmer with save(force=0) sometimes interfering with the safe(force=1) call at the end. > > Hence my attempt to eliminate save(force=0) calls.. We haven't yet resolved this issue. I'd like to get a fix in which won't bring back the random Rdict.db errors.. Thanks, Satish

1 0

Re: [petsc-dev] futurize
by Lisandro Dalcin 02 Apr '18

02 Apr '18

On 2 April 2018 at 23:08, Jed Brown <jed(a)jedbrown.org> wrote: > Would it be too disruptive to run futurize "stage 1" before the 3.9 > release? Would this introduce any dependency outside plain Python 2 stdlib? Sorry, I'm not an expert in these futurize/modernize/six/whatever stuff, I got used to write Py 2/3 compatible code from scratch without these tools. > The source would remain Python-2 only, but would be > "modernized" so that the next stage (supporting Python-3) would be less > disruptive. The goal here would be to make it easier to port > configuration changes from 'maint' to 'master' after the release, under > the assumption that we'll want to support Python-3 before a 3.10 > release. Well, while I'm a bit worried about dependencies, I think your goal makes total sense and I would support it. -- Lisandro Dalcin ============ Research Scientist Computer, Electrical and Mathematical Sciences & Engineering (CEMSE) Extreme Computing Research Center (ECRC) King Abdullah University of Science and Technology (KAUST) http://ecrc.kaust.edu.sa/ 4700 King Abdullah University of Science and Technology al-Khawarizmi Bldg (Bldg 1), Office # 0109 Thuwal 23955-6900, Kingdom of Saudi Arabia http://www.kaust.edu.sa Office Phone: +966 12 808-0459

1 0

Re: [petsc-dev] upcoming release and testing
by Satish Balay 02 Apr '18

02 Apr '18

On Mon, 2 Apr 2018, Satish Balay wrote: > On Mon, 2 Apr 2018, Satish Balay wrote: > > > Thanks! updated knepley/fix-ex62-tests and merged to next-tmp > > next-tmp has plenty of errors.. Forgot to include the dashboard url http://ftp.mcs.anl.gov/pub/petsc/nightlylogs/archive/2018/04/02/next-tmp.ht… Satish > > It currently has: > origin/knepley/fix-ex62-tests > origin/knepley/fix-dm-gmg > origin/knepley/fix-cylinder-mesh > origin/dalcinl/fix-scatter-destroy > origin/dalcinl/fix-vecscatter-initpkg > origin/balay/update-openmpi-3.0.1 > > So I don't [yet] know which of these branches are causing grief - and > which ones are ok - so I can't merge any of them to master.. > > Satish >

1 0