Thanks Mark, I agree with all that. Slight comments below. Mark Miller wrote:

Hi Tim,

This is very clear to me and I agree with it. But, since some of the discussion after Lori left the conversation is missing, I'd like to take a stab at adding that here simply in order to take a snapshot of the history of this discussion...

We basically went full circle in our discussion. The conversation started with the notion that part neighbors are based upon the sharing of mesh entities. An objection was raised because we can often think of circumstances in which two parts need to exchange data across the 'boundary' between them even when they DO NOT necessarily share mesh entities. Rercusive Coordinate Bisection and Finite Volume computations were raised as two examples. In the former, RCB, I believe we also made arguments that there does in fact exist a 'shared mesh entity' analog in that there are mesh edges that are 'cut' by the RCB algorithm and it is those edges that wind up being 'shared' between parts [I did not consider this yesterday but what if such edges are never actually instantiated in iMesh? Would this make the argument theoretically correct but practically speaking not implementable?]

I wondered about this and ended up removing a clause about shared entities being explicitly represented for this reason. For example, an application may request 2d neighbors for a part. In this case these entities should be instantiated in some way (at least enough to assign a handle for them). 1d neighbors wouldn't need to be instantiated in this case, even though technically they'd still exist.

We observed that there are likely to be many situations in which processors (or the parts they own) need to communicate that have nothing to do with underlying mesh topology. So the question arose, do we expect our Part/Partition model and implementation to support a more general notion of 'neighbor' or one that is specific to mesh topology. Ultimately, we argeed on the latter.

We also argued that a notion of neighbor based solely on mesh topology is a 'sufficient bootstrap' to enable applications requiring a more general notion of 'neighbor' to affect their communication patterns. [As an aside, I don't think we considered this argument in great detail and I wonder if it is in fact true in all cases that more general notions of neighbor can always be derived from the topologically specific one being proposed here. I'm inclined NOT to believe that. At the same time, I think it is also arguable that such situations are too 'application specific' to be worried about explicitly supporting in the Part/Parition model we are developing.]

This is sort of a circular argument, but I'll make it anyway... more general notions of neighbors that involve topology will be derivable from topology, and those that aren't are likely not derivable from mesh-intrinsic information, and are therefore application-specific.

We also agreed that our intention in developing the Part/Partition model on top of iMesh is to support the topologically common operations most applications will need and not necessarily ALL possible communication patterns applications might use.

One question we need to re-visit is what are the implications (storage and/or performance), if neighbor relationships other than those determined by topology, are not supported?

Since these relationships should be storable under our current data model, I think applications can decide whether to use memory or cpu time to recover them. This does beg the question though about whether we can tag parts (you all probably know my preference on that). - tim

Mark

Tim Tautges wrote:

...

A question arose in yesterday's telecon on what constitutes a part neighbor. After some discussion, we came up with the following definition and rationale.

Def: for a given part, a part neighbor of dimension d is any other part which shares mesh entities of dimension d with that part (i.e. the entity is represented on both parts).

Rationale: - it is desirable to tie the definition of a part neighbor with information intrinsic to the mesh, i.e. one could derive information about part neighbors using only information about the mesh entities and the partition of entities over processors

- in general, applications will have other criteria determining which other processors a given processor communicates with; that information can be stored in iMesh using sets and tags already, and applications will presumably be able to pass those things to communication functions in the itaps parallel interface (thus driving the application-specific communication patterns)

Question: are ghosted/copied entities between parts considered shared? I would assert that ghosting/copying should be between processors, not parts, so this question is not relevant with respect to part neighbors.

- tim

-- ================================================================ "You will keep in perfect peace him whose mind is steadfast, because he trusts in you." Isaiah 26:3

Tim Tautges Argonne National Laboratory ([email protected]) (telecommuting from UW-Madison) phone: (608) 263-8485 1500 Engineering Dr. fax: (608) 263-4499 Madison, WI 53706

-- Mark C. Miller, Lawrence Livermore National Laboratory email: mailto:[email protected] (M/T/W) (925)-423-5901 (!!LLNL BUSINESS ONLY!!) (Th/F) (530)-753-8511 (!!LLNL BUSINESS ONLY!!)

-- ================================================================ "You will keep in perfect peace him whose mind is steadfast, because he trusts in you." Isaiah 26:3 Tim Tautges Argonne National Laboratory ([email protected]) (telecommuting from UW-Madison) phone: (608) 263-8485 1500 Engineering Dr. fax: (608) 263-4499 Madison, WI 53706

Mark, There is one thing Lori specially mentioned in terms of an application, that the (lower-order) entities at inter-part boundaries exist only on one part ("uniquely owned") and so higher-order mesh entities on all but one will not be closed (so a mesh face on inter-part boundary bounding two mesh regions on two different parts will exist on one part). In this way I think there might be a more conceptual way to describe part neighbors then by specially using term "sharing of entities". And as we discussed it cannot be any arbitrary description but has to be related to underlying mesh being partitioned. Please let me know your comments. Thanks, Onkar

Hi Tim,

This is very clear to me and I agree with it. But, since some of the discussion after Lori left the conversation is missing, I'd like to take a stab at adding that here simply in order to take a snapshot of the history of this discussion...

We basically went full circle in our discussion. The conversation started with the notion that part neighbors are based upon the sharing of mesh entities. An objection was raised because we can often think of circumstances in which two parts need to exchange data across the 'boundary' between them even when they DO NOT necessarily share mesh entities. Rercusive Coordinate Bisection and Finite Volume computations were raised as two examples. In the former, RCB, I believe we also made arguments that there does in fact exist a 'shared mesh entity' analog in that there are mesh edges that are 'cut' by the RCB algorithm and it is those edges that wind up being 'shared' between parts [I did not consider this yesterday but what if such edges are never actually instantiated in iMesh? Would this make the argument theoretically correct but practically speaking not implementable?]

We observed that there are likely to be many situations in which processors (or the parts they own) need to communicate that have nothing to do with underlying mesh topology. So the question arose, do we expect our Part/Partition model and implementation to support a more general notion of 'neighbor' or one that is specific to mesh topology. Ultimately, we argeed on the latter.

We also argued that a notion of neighbor based solely on mesh topology is a 'sufficient bootstrap' to enable applications requiring a more general notion of 'neighbor' to affect their communication patterns. [As an aside, I don't think we considered this argument in great detail and I wonder if it is in fact true in all cases that more general notions of neighbor can always be derived from the topologically specific one being proposed here. I'm inclined NOT to believe that. At the same time, I think it is also arguable that such situations are too 'application specific' to be worried about explicitly supporting in the Part/Parition model we are developing.]

We also agreed that our intention in developing the Part/Partition model on top of iMesh is to support the topologically common operations most applications will need and not necessarily ALL possible communication patterns applications might use.

One question we need to re-visit is what are the implications (storage and/or performance), if neighbor relationships other than those determined by topology, are not supported?

Mark

Tim Tautges wrote:

...

A question arose in yesterday's telecon on what constitutes a part neighbor. After some discussion, we came up with the following definition and rationale.

Def: for a given part, a part neighbor of dimension d is any other part which shares mesh entities of dimension d with that part (i.e. the entity is represented on both parts).

Rationale: - it is desirable to tie the definition of a part neighbor with information intrinsic to the mesh, i.e. one could derive information about part neighbors using only information about the mesh entities and the partition of entities over processors

- in general, applications will have other criteria determining which other processors a given processor communicates with; that information can be stored in iMesh using sets and tags already, and applications will presumably be able to pass those things to communication functions in the itaps parallel interface (thus driving the application-specific communication patterns)

Question: are ghosted/copied entities between parts considered shared? I would assert that ghosting/copying should be between processors, not parts, so this question is not relevant with respect to part neighbors.

- tim

-- ================================================================ "You will keep in perfect peace him whose mind is steadfast, because he trusts in you." Isaiah 26:3

Tim Tautges Argonne National Laboratory ([email protected]) (telecommuting from UW-Madison) phone: (608) 263-8485 1500 Engineering Dr. fax: (608) 263-4499 Madison, WI 53706

-- Mark C. Miller, Lawrence Livermore National Laboratory email: mailto:[email protected] (M/T/W) (925)-423-5901 (!!LLNL BUSINESS ONLY!!) (Th/F) (530)-753-8511 (!!LLNL BUSINESS ONLY!!)

I am not sure I fully understand this. I understand having a unique owner. However, it this is saying a face on an inter-part boundary only exists on one part and not the other, then it is not possible for the iMesh representation on the other side to be complete. All mesh entities of order 1 and greater must be able to understand what bounds them. (Maybe I misunderstand what is being said. Onkar Sahni wrote:

Mark,

There is one thing Lori specially mentioned in terms of an application, that the (lower-order) entities at inter-part boundaries exist only on one part ("uniquely owned") and so higher-order mesh entities on all but one will not be closed (so a mesh face on inter-part boundary bounding two mesh regions on two different parts will exist on one part).

In this way I think there might be a more conceptual way to describe part neighbors then by specially using term "sharing of entities". And as we discussed it cannot be any arbitrary description but has to be related to underlying mesh being partitioned.

Please let me know your comments.

Thanks, Onkar

...

Hi Tim,

This is very clear to me and I agree with it. But, since some of the discussion after Lori left the conversation is missing, I'd like to take a stab at adding that here simply in order to take a snapshot of the history of this discussion...

We basically went full circle in our discussion. The conversation started with the notion that part neighbors are based upon the sharing of mesh entities. An objection was raised because we can often think of circumstances in which two parts need to exchange data across the 'boundary' between them even when they DO NOT necessarily share mesh entities. Rercusive Coordinate Bisection and Finite Volume computations were raised as two examples. In the former, RCB, I believe we also made arguments that there does in fact exist a 'shared mesh entity' analog in that there are mesh edges that are 'cut' by the RCB algorithm and it is those edges that wind up being 'shared' between parts [I did not consider this yesterday but what if such edges are never actually instantiated in iMesh? Would this make the argument theoretically correct but practically speaking not implementable?]

We observed that there are likely to be many situations in which processors (or the parts they own) need to communicate that have nothing to do with underlying mesh topology. So the question arose, do we expect our Part/Partition model and implementation to support a more general notion of 'neighbor' or one that is specific to mesh topology. Ultimately, we argeed on the latter.

We also argued that a notion of neighbor based solely on mesh topology is a 'sufficient bootstrap' to enable applications requiring a more general notion of 'neighbor' to affect their communication patterns. [As an aside, I don't think we considered this argument in great detail and I wonder if it is in fact true in all cases that more general notions of neighbor can always be derived from the topologically specific one being proposed here. I'm inclined NOT to believe that. At the same time, I think it is also arguable that such situations are too 'application specific' to be worried about explicitly supporting in the Part/Parition model we are developing.]

We also agreed that our intention in developing the Part/Partition model on top of iMesh is to support the topologically common operations most applications will need and not necessarily ALL possible communication patterns applications might use.

One question we need to re-visit is what are the implications (storage and/or performance), if neighbor relationships other than those determined by topology, are not supported?

Mark

Tim Tautges wrote:

...

A question arose in yesterday's telecon on what constitutes a part neighbor. After some discussion, we came up with the following definition and rationale.

Def: for a given part, a part neighbor of dimension d is any other part which shares mesh entities of dimension d with that part (i.e. the entity is represented on both parts).

Rationale: - it is desirable to tie the definition of a part neighbor with information intrinsic to the mesh, i.e. one could derive information about part neighbors using only information about the mesh entities and the partition of entities over processors

- in general, applications will have other criteria determining which other processors a given processor communicates with; that information can be stored in iMesh using sets and tags already, and applications will presumably be able to pass those things to communication functions in the itaps parallel interface (thus driving the application-specific communication patterns)

Question: are ghosted/copied entities between parts considered shared? I would assert that ghosting/copying should be between processors, not parts, so this question is not relevant with respect to part neighbors.

- tim

-- ================================================================ "You will keep in perfect peace him whose mind is steadfast, because he trusts in you." Isaiah 26:3

Tim Tautges Argonne National Laboratory ([email protected]) (telecommuting from UW-Madison) phone: (608) 263-8485 1500 Engineering Dr. fax: (608) 263-4499 Madison, WI 53706 -- Mark C. Miller, Lawrence Livermore National Laboratory email: mailto:[email protected] (M/T/W) (925)-423-5901 (!!LLNL BUSINESS ONLY!!) (Th/F) (530)-753-8511 (!!LLNL BUSINESS ONLY!!)

I think neighbors 'of' dimension d are neighbors that share mesh entities of at most dimension d. For example, neighbors 'of' dimension 1 are neighbors that share only edges and vertices.

Then using 'd' twice in def. seems confusing. And in example, I think neighbors 'of' dimension 2 (i.e., face, say, for finite-volume) will be ones that have faces on inter-part boundaries (not edges and not vertices). - Onkar