On Fri, May 21, 2021 at 9:34 AM Pierre Bernigaud <[email protected]> wrote:

Greetings,

I am currently working on a CFD solver using PETSc. I have a non linear system which is solved using 2D_DMDA/SNES, and submitted to boundary conditions that are treated implicitly and updated in the FormFunction. The calculation of one of these boundary conditions requires the resolution of an other non linear system.

I am hence using a nested 1D_DMDA/SNES system within the FormFunction of my main SNES solver to solve for this boundary condition. This is working fine, but doing a scalability study we found out that this causes the code to show sub-par acceleration properties.

Have you ever encountered this kind of nested SNES application, and are there some critical points to be aware of in order to avoid a loss of performances? For instance, the sub 1D_DMDA/SNES objects are created and destroyed at each update of the boundary, hence at each call to FormFunction, which results in an important number of object manipulation. Could this be a problem?

Yes. You should keep this subsolver around for as long as the outer solver lives.

Furthermore, the use of a sub 1D_DMDA/SNES allows to use multiple processors to solve for the boundary condition, composed of say N cells. When running the code with M > N processors, everything is working great, but I am curious about the state of the (M-N) processors which aren’t working on boundary condition problem. Do they just stay idle?

Yes, they are idle. You could try to fold the boundary condition into the same system as the bulk, but this is not a clear win since you will still have load imbalance. What percentage of time does the boundary solve take? Thanks, Matt

Thank you for your help. Respectfully, Pierre Bernigaud

-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>