April 2018 - petsc-dev - mailman3.cels.anl.gov

Re: [petsc-dev] Nullspace with right preconditioning
by Matthew Knepley 04 Apr '18

04 Apr '18

On Wed, Apr 4, 2018 at 3:23 PM, Smith, Barry F. <bsmith(a)mcs.anl.gov> wrote: > > It may not be broken; it may just be misunderstood or incorrect approach. > > You'll need to explain why and where exactly you think it is broken. I am solving Stokes. I will push the example for you to run tonight. Basically, I put in a constant for the pressure as the nullspace. If you run with gmres/left it gives back a pressure with integral 0. If you run with gmres/right, it gives back a pressure with a big integral. Matt > > Barry > > > > > > On Apr 4, 2018, at 1:13 PM, Matthew Knepley <knepley(a)gmail.com> wrote: > > > > This is broken. Did we know this was broken? > > > > Thanks, > > > > Matt > > > > -- > > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > > -- Norbert Wiener > > > > https://www.cse.buffalo.edu/~knepley/ > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ <http://www.caam.rice.edu/~mk51/>

1 0

Re: [petsc-dev] Nullspace with right preconditioning
by Smith, Barry F. 04 Apr '18

04 Apr '18

It may not be broken; it may just be misunderstood or incorrect approach. You'll need to explain why and where exactly you think it is broken. Barry > On Apr 4, 2018, at 1:13 PM, Matthew Knepley <knepley(a)gmail.com> wrote: > > This is broken. Did we know this was broken? > > Thanks, > > Matt > > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/

1 0

Nullspace with right preconditioning
by Matthew Knepley 04 Apr '18

04 Apr '18

This is broken. Did we know this was broken? Thanks, Matt -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ <http://www.caam.rice.edu/~mk51/>

1 0

[issue1595] Issues of limited number of MPI communicators when having many instances of hypre boomerAMG with Moose
by Rob Falgout hypre Tracker 04 Apr '18

04 Apr '18

Rob Falgout <rfalgout(a)llnl.gov> added the comment: FYI, I meant to say "corresponding user recv requests" in 4 below. -Rob ____________________________________________ hypre Issue Tracker <hypre-support(a)llnl.gov> <http://cascb1.llnl.gov/hypre/issue1595> ____________________________________________

1 0

Re: [petsc-dev] upcoming release and testing [v2]
by Satish Balay 04 Apr '18

04 Apr '18

FYI - I have the release related strings in https://bitbucket.org/petsc/petsc/branch/balay/release-3.9 This change should be similar to what was done for 3.8 release https://bitbucket.org/petsc/petsc/commits/0e50f9e530a7b78427514d3e384f6941d… For now - I'm using tomorrow's date [but will update it when we are ready to do the release] This branch will be rebased as needed. Satish

1 0

[issue1595] Issues of limited number of MPI communicators when having many instances of hypre boomerAMG with Moose
by Rob Falgout hypre Tracker 04 Apr '18

04 Apr '18

Rob Falgout <rfalgout(a)llnl.gov> added the comment: Hi All, Some comments and questions: 1. The Comm_create is used to create a subcommunicator that involves only the currently active MPI tasks so that the Allgather() will happen only over that subset. I don't think we can create this once, attach it to a parent communicator, and use it again in a different solve based on a different AMG setup (because the active tasks will likely be different). If the AMG setup is the same, the same communicator will be used. When the AMG Destroy() is called, that communicator is freed. 2. Ulrike has already provided a way to get around the Comm_create() by changing one of the AMG parameters. This is something that PETSc has control over and can do. 3. The idea of dup'ing a communicator, attaching it as an attribute to a parent communicator, and using it as hypre's internal communicator makes sense to me. I think it would require quite a bit of code changes to implement. My next comment is important to consider as well. 4. Regarding pending point-to-point messages, you are right that this could create problems without creating deadlock. I had not thought about this scenario. However, as long as all of the corresponding user send requests have also been issued (in a non-blocking manner), is there still really a problem here? MPI is supposed to guarantee message order, so any receives that hypre posts from the same task with the same tag should not interfere with an already posted (but not finalized) receive that the user has posted, right? -Rob ____________________________________________ hypre Issue Tracker <hypre-support(a)llnl.gov> <http://cascb1.llnl.gov/hypre/issue1595> ____________________________________________

1 0

Add PetscFree1() ?
by Lisandro Dalcin 04 Apr '18

04 Apr '18

Jed, just for the sake of consistency, don't we need PetscFree1()? Also, PetscFree() is a macro, and evaluates twice the first arg, I just had to fix a bug in my code that was doing while (ctx->size > 0) PetscFree(ctx->arrayOfPointers[--ctx->size]); PetscFree(ctx->arrayOfPointers) I guess we should just rewrite PetscFree() to be safer, but too close to release right now to play these games... -- Lisandro Dalcin ============ Research Scientist Computer, Electrical and Mathematical Sciences & Engineering (CEMSE) Extreme Computing Research Center (ECRC) King Abdullah University of Science and Technology (KAUST) http://ecrc.kaust.edu.sa/ 4700 King Abdullah University of Science and Technology al-Khawarizmi Bldg (Bldg 1), Office # 0109 Thuwal 23955-6900, Kingdom of Saudi Arabia http://www.kaust.edu.sa Office Phone: +966 12 808-0459

1 0

Re: [petsc-dev] FAS indentation
by Satish Balay 04 Apr '18

04 Apr '18

On Wed, 4 Apr 2018, Dener, Alp wrote: > (I didn’t wanna push directly to next without approval). Just a note: we would never commit/push directly to next. All fixes should go to the corresponding feature branch - this way they will get to master when ready [and next is never directly merged to master] One reason for a direct commit/push to next could be to fix a bad merge in next. Satish

1 0

upcoming release and testing [v2]
by Satish Balay 04 Apr '18

04 Apr '18

[starting a new thread] As of now master builds are clean! http://ftp.mcs.anl.gov/pub/petsc/nightlylogs/archive/2018/04/03/master.html next is improved but still has issues that need fixing. http://ftp.mcs.anl.gov/pub/petsc/nightlylogs/archive/2018/04/04/next.html most-likely are are related to the following branches [and some dependencies/conflicts with changes in master] origin/knepley/fix-dm-gmg origin/knepley/fix-ex62-tests origin/knepley/fix-fe-bd-integral origin/knepley/fix-fe-vector-spaces origin/knepley/fix-snes-ex69 origin/tisaac/feature-dmfield I have the following branches in queue for master [via next-tmp - this build will start now] origin/dalcinl/fix-scatter-destroy origin/dalcinl/fix-vecscatter-initpkg origin/knepley/fix-cylinder-mesh origin/rmills/feature-aijmkl-add-matptap-symmetric origin/barry/removed-unneeded-mpi-comm-dup-boomeramg origin/balay/fix-9bus-ex9busdmnetwork-chkstk-on-windows origin/jed/futurize-stage-1 1. If there are any other branches/PRs that should go into the release - let me know 2. reminder - please check and update src/docs/website/documentation/changes/dev.html [or send me diffs] Thanks! Satish

1 0

Re: [petsc-dev] [issue1595] Issues of limited number of MPI communicators when having many instances of hypre boomerAMG with Moose
by Jed Brown 04 Apr '18

04 Apr '18

Yes, this is a real issue for MOOSE which sometimes has thousands of active single-field solvers. PETSc can limit the number of fine-level communicators by retaining the dup'd communicator so the same communicator can be passed to hypre for each solver, but cannot control the MPI_Comm_create for a parallel coarse level. Hypre could do that internally by attaching the coarse communicator as an attribute (on the relevant ranks) of the larger communicator. The separate tag space is important because point-to-point messaging can be pending when hypre is called -- that does not lead to deadlock, but it is important that hypre not post sends or receives with those tags lest messages be delivered incorrectly. I feel like your response below is a false economy. Nobody would fault hypre for dup'ing once. But with the current interface, it is laborious to impossible (in case of parallel coarse solve) to create a thousand hypre solvers without having a thousand communicators. Assuming you are not convinced, we will handle this in PETSc the same way PETSc does for itself, but (a) we still can't control the communicator for a parallel coarse solve and (b) this issue may crop up again if some other user attempts to do this sort of solve without using PETSc. Rob Falgout hypre Tracker <hypre-support(a)llnl.gov> writes: > Rob Falgout <rfalgout(a)llnl.gov> added the comment: > > Is somebody actually having a problem with communicator conflicts right now? > > I thought the reason for this thread was to reduce the number of communicators because of limits in MPI implementations. Somebody has to reduce the Comm_create() and Comm_dup() calls. We responded with one way to reduce the create() calls in BoomerAMG, but now you are asking us to put them back in by calling dup()? I'm confused about what we are trying to achieve here now. > > The reason I suggested that the user be responsible for calling dup() is twofold: 1) I don't think it is common for users to run hypre in parallel with other user code where both are using the same communicator (I'm not sure how this could even work without deadlocking since hypre calls are collective); 2) Making libraries lower down on the call stack be responsible for calling dup() seems less scalable than the other way around and more likely to increase the number of communicators used. > > Anyway, I'm still confused about what we are trying to achieve so maybe somebody can try to summarize again? > > -Rob > > ____________________________________________ > hypre Issue Tracker <hypre-support(a)llnl.gov> > <http://cascb1.llnl.gov/hypre/issue1595> > ____________________________________________

1 0