Firstly, thank you, everyone, for taking time to talk about this issue!  We definitely appreciate it.

Jed has this right - we would ideally not want to setup and teardown the preconditioner for each solve every iteration.

For VERY large solves (10k+ simultaneous solves) using this method we have no choice - and have done exactly what has been suggested: setup-solve-teardown for each one in serial.  But that's an extreme case where we can't fit all of the matrices in memory at the same time.  For the "normal" runs (10-1000 solves) that's not the case.

Derek

On Tue, Apr 3, 2018 at 4:37 PM Jed Brown <jed@jedbrown.org> wrote:
This is likely in the context of a preconditioner for a composite solve,
so the many BoomerAMG solvers will persist.  They all get set up and
then applied in sequence to the many scalar systems at each iteration of
an outer method.

If hypre doesn't want to dup its communicators internally, then PETSc
will dup a communicator for hypre, attach it to the outer communicator,
and look for it each time a new hypre object is created.  It's easy for
PETSc (we've done this for PETSc since the early days), but a headache
for other libraries.

"Li, Ruipeng" <li50@llnl.gov> writes:

> Hi, Fande,
>
> "For instance, if an application has multiple fields and each field will be solved by a HYPRE solver,  then we end up with  multiple hypre solves. The multiple hypre solves are not run simultaneously, instead, it will be run one-by-one. If we assign the same communicator to the multiple hypre solves, any potential tag conflict?"
>
> In this scenario, is just the solve phase of BoomerAMG one after another, or also the setup of AMG? Since BoomerAMG takes the communicator from the A matrix, if the entire hypre solve (setup + solve) is serial, there should be no conflicts.
>
> -Ruipeng
>
> . -- .- .. .-.. / ..-. .-. --- -- / .-. ..- .. .--. . -. --. / .-.. ..
>  Ruipeng Li
>  Center for Applied Scientific Computing
>  Lawrence Livermore National Laboratory
>  P.O. Box 808, L-561
>  Livermore, CA 94551
>  phone - (925) 422-6037,  email - li50@llnl.gov
>
>
> ________________________________________
> From: Kong, Fande <fande.kong@inl.gov>
> Sent: Tuesday, April 3, 2018 1:34 PM
> To: hypre-support
> Cc: Barry Smith; Derek Gaston; Li, Ruipeng; Osei-Kuffuor, Daniel; For users of the development version of PETSc; Schroder, Jacob B.; tzanio@llnl.gov; umyang@llnl.gov; Wang, Lu
> Subject: Re: [issue1595] Issues of limited number of MPI communicators when having many instances of hypre boomerAMG with Moose
>
>
>
> On Tue, Apr 3, 2018 at 2:12 PM, Rob Falgout hypre Tracker <hypre-support@llnl.gov<mailto:hypre-support@llnl.gov>> wrote:
>
> Rob Falgout <rfalgout@llnl.gov<mailto:rfalgout@llnl.gov>> added the comment:
>
> Hi Barry,
>
> Can you explain the scenario where multiple hypre solves would be working in parallel with the same communicator?  Thanks!
>
> Hi Rob,
>
> For instance, if an application has multiple fields and each field will be solved by a HYPRE solver,  then we end up with  multiple hypre solves. The multiple hypre solves are not run simultaneously, instead, it will be run one-by-one. If we assign the same communicator to the multiple hypre solves, any potential tag conflict?
>
> Fande,
>
>
>
> -Rob
>
> ____________________________________________
> hypre Issue Tracker <hypre-support@llnl.gov<mailto:hypre-support@llnl.gov>>
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__cascb1.llnl.gov_hypre_issue1595&d=DwIFaQ&c=54IZrppPQZKX9mLzcGdPfFD1hxrcB__aEkJFOKJFd00&r=DUUt3SRGI0_JgtNaS3udV68GRkgV4ts7XKfj2opmiCY&m=3KsQjTxI8eRWrSIJtB3o9NlBc4bEuuP4HxiT-BlMx3U&s=FjvfvXD21bX8LA2TAD-J6MWCJyMNm0aameRp_9nfLYs&e=>
> ____________________________________________