We (Alex and Fande) have had a discussion again on the concerned issue. An alternative way to resolve our problem could be an extra optional flag (MAT_NO_SHRINK_MEMORY) to tell PETSc not to shrink memory at certain stages.

The typical usage would be that we turn this flag on at the beginning of our FormJacobian, and then turn this flag off at the very end of our FormJacobian right before we call MatAssembly(final).  We may argue that you should call MatAssembly(final) only once, and then you will be fine. "Call MatAssembly(final) only once" only works in an ideal world. In the real world, we have to manipulate the matrix in a very complicated way to formulate physics.  

As always, we are willing to contribute but want to make sure our idea makes sense for you guys before spending time on that.

Thanks,

Fande

On Thu, Jul 15, 2021 at 11:03 AM Alexander Lindsay <alexlindsay239@gmail.com> wrote:
As per https://gitlab.com/petsc/petsc/-/issues/852 we have some fairly complex functions in MOOSE that sometimes require the matrix to be "assembled" at an intermediate stage of our Jacobian evaluation in order to perform multi-body Constraints. In an ideal world I would like to assemble with FLUSH because as per my post today in petsc-users FINAL may shrink my allocation before I'm ready for it to.

We cannot currently use FLUSH at intermediate stages, however, because there is a late call to MatZeroRows which we leverage before applying our last set of objects (which are Dirichlet boundary conditions). MatZeroRows checks the `assembled` flag which only gets set during FINAL assembly and errors out if `assembled` is not true. From my understanding, MatZeroRows should be a safe operation even if the matrix has just been FLUSHed. MatZeroRows is just one example; I would guess there are other operations that could potentially be done with a FLUSHed matrix as opposed to a fully assembled one. Consequently, I'm wondering if it makes sense to add something like a `flushed` data member to _p_Mat that would allow for some less strict checking in operations like MatZeroRows.

Alex