Yes I think that makes sense. How do you access the associated PetscSF for the DMComplex? Is it possible to define multiple PetscSF's for a given DMComplex, each associated with different PetscSections? My current understanding of using DMComplex is (for the serial case): 1) Create a DMComplex object, set its cones/supports, etc. 2) Create PetscSections that describe the layout of data across the DMComplex 3) Set a PetscSection as the default section 4) Use DMCreateLocalVector to create the Vec objects needed for the computation Is this correct? In the parallel case, what else needs to be done? -Chris On Fri, Aug 17, 2012 at 1:24 PM, Matthew Knepley <[email protected]> wrote:

On Fri, Aug 17, 2012 at 1:21 PM, Chris Eldred <[email protected]> wrote:

...

I was wondering how ghost values were handled when creating a distributed (global) vector from a specific PetscSection over a DMComplex object- ie what defines the stencils used to figure out which values need to be ghosted? What about local vs. global indexes?

This answer will have to be long because the question is mixing levels in the interface.

A PetscSection is only an offset structure, so it has no designated fields for overlap information. What I do for DM is to create 2 sections: one describing the layout of the local vector, and one describing the global vector and ghosts. The global section can be created automatically given the local section and a PetscSF (successor to VecScatter).

The PetscSF describes the parallel overlap, and also has scatters and reductions. This is created by the DMComplex instead of a scatter since it can handle arbitrary types, and has a better query interface. DMComplex creates a (non-overlapping) partition of some stratum of points, usually cells for FEM. Then the closure(p) U star(p) for each point is added to that partition, and the dofs on any shared point are ghosted.

This is all now done using the vanilla DM interface, so DMGlobalToLocal() and LocalToGlobal() and cousins work as normal.

Does that make sense?

Matt

...

-Chris

-- Chris Eldred DOE Computational Science Graduate Fellow Graduate Student, Atmospheric Science, Colorado State University B.S. Applied Computational Physics, Carnegie Mellon University, 2009 [email protected]

-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener

-- Chris Eldred DOE Computational Science Graduate Fellow Graduate Student, Atmospheric Science, Colorado State University B.S. Applied Computational Physics, Carnegie Mellon University, 2009 [email protected]