Thanks- that helps a lot. If I need stencils that are larger than closure(p) U star(p) (for a higher-order finite difference method, for example), I assume that I need to create my own PetscSF's that describe which points need to be ghosted? Is there some documentation or example code that explains the theory behind star forests? On Sat, Aug 18, 2012 at 3:49 PM, Matthew Knepley <[email protected]> wrote:

On Sat, Aug 18, 2012 at 4:11 PM, Chris Eldred <[email protected]> wrote:

...

Yes I think that makes sense. How do you access the associated PetscSF for the DMComplex? Is it possible to define multiple PetscSF's for a given DMComplex, each associated with different PetscSections?

1) DMGetDefaultSF() for the one for the default Section

2) Yes, but you need to create them yourself using DMCreateDefaultSF()

...

My current understanding of using DMComplex is (for the serial case):

1) Create a DMComplex object, set its cones/supports, etc.

A good guide is to look at complexcreate.c, where I create a few default meshes.

...

2) Create PetscSections that describe the layout of data across the DMComplex 3) Set a PetscSection as the default section 4) Use DMCreateLocalVector to create the Vec objects needed for the computation

Is this correct?

Yes

...

In the parallel case, what else needs to be done?

In parallel,

a) I call DMComplexDistribute(). The alternative is to create your mesh directly in parallel, but this does not work with any mesh generator I know, and you would have to create the PetscSF for points that this function makes automatically.

b) Now the section you create is local

c) You can use the local section and pointSF to create a global section, which knows about overlap

d) You can use the local and global section to make the dofSF

c) and d) are done automatically for the default section (data layout).

Matt

...

-Chris

On Fri, Aug 17, 2012 at 1:24 PM, Matthew Knepley <[email protected]> wrote:

...

On Fri, Aug 17, 2012 at 1:21 PM, Chris Eldred <[email protected]> wrote:

...

I was wondering how ghost values were handled when creating a distributed (global) vector from a specific PetscSection over a DMComplex object- ie what defines the stencils used to figure out which values need to be ghosted? What about local vs. global indexes?

This answer will have to be long because the question is mixing levels in the interface.

A PetscSection is only an offset structure, so it has no designated fields for overlap information. What I do for DM is to create 2 sections: one describing the layout of the local vector, and one describing the global vector and ghosts. The global section can be created automatically given the local section and a PetscSF (successor to VecScatter).

The PetscSF describes the parallel overlap, and also has scatters and reductions. This is created by the DMComplex instead of a scatter since it can handle arbitrary types, and has a better query interface. DMComplex creates a (non-overlapping) partition of some stratum of points, usually cells for FEM. Then the closure(p) U star(p) for each point is added to that partition, and the dofs on any shared point are ghosted.

This is all now done using the vanilla DM interface, so DMGlobalToLocal() and LocalToGlobal() and cousins work as normal.

Does that make sense?

Matt

...

-Chris

-- Chris Eldred DOE Computational Science Graduate Fellow Graduate Student, Atmospheric Science, Colorado State University B.S. Applied Computational Physics, Carnegie Mellon University, 2009 [email protected]

-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener

-- Chris Eldred DOE Computational Science Graduate Fellow Graduate Student, Atmospheric Science, Colorado State University B.S. Applied Computational Physics, Carnegie Mellon University, 2009 [email protected]

-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener

-- Chris Eldred DOE Computational Science Graduate Fellow Graduate Student, Atmospheric Science, Colorado State University B.S. Applied Computational Physics, Carnegie Mellon University, 2009 [email protected]