Mark, in a nonsymmetric (but structurally symmetric) bs=2 problem from Lulu, we are tripping this assertion. else{ /* I'm stealing this local, owned by a ghost */ assert(sgid==-1); What does this mean? Can't we at least always use SETERRQ for errors? Also from that section of code, I notice you are using PETSC_ERR_LIB for internal PETSc logic/corruption errors. They look like they should be PETSC_ERR_PLIB.