Hi, Tim. Actually, the situation is worse than you assumed. We did not ratify the proposal made after COB Thursday; we ratified a proposal made 40 minutes into the conference call. :) Of course, we ratified it contingent on getting feedback from you. That is why I sent the notes from yesterday's meeting. Did you get them? I'm sorry I wasn't clear in expressing that intent. We agreed that iMesh functions taking a mesh instance and an entity set handle as an argument would now also take a partition instance and part handle. These extra arguments could be NULL, providing the current iMesh serial capability. Or they could be valid handles for partitions and parts, in which case the implementation would do the appropriate set operations to return values in the entity set that are also in the part. Jason did a good job of explaining the drawbacks of the COB Thursday proposal. We also picked a date/time for the next phone conf: Jan 7 at 1:30 PST. Does that work for you? If not, we will reschedule. We want to have a parallel interface that is presentable for the next bootcamp (Mar 11) so your attendance at the phone confs is important. Karen On 12/18/07 9:39 AM, "Tim Tautges" <[email protected]> wrote:
As usual, I picked the wrong telecon to miss. There should be some rule about not allowing major interface change proposals coming out on Thursday after COB (in the midwest) and essentially ratifying those changes Monday morning. But anyway...
Carl Ollivier-Gooch wrote:
Hello, all.
During our last telecon, I had what I thought was an idea that would unify the parallel query for sets stuff. At the time, the idea was vaguely formed, and I hadn't thought through all of the consequences. I may still not have, but I've at least got it to the point of being ready to drag it out into the light of day.
Essentially, the issue is that we want to have much the same set of queries for mesh instances, parts, and partitions, but we haven't worked out how to handle this without exploding the number of functions.
In the serial interface, we have lots of queries that take both a mesh instance and an entity set argument. In the now more-or-less deprecated SIDL paradigm, the mesh instance in an object (in the C++ sense), and we unified global and entity set calls by creating a root set that is, essentially, shorthand for "everything in the mesh instance".
I think the above is a very useful thing to keep in mind. The two things that change when you go from there to parallel is that a) mesh instances together form a collection that in some way describes a global mesh, and b) some sets can be thought of as spanning processors, but only implicitly (that is, we don't assume the handles are the same).
In parallel, the mesh instance resides at the process level in a partition / process / part heirarchy (black / red, green in the attached picture; yes, it's a very obvious illustration, but perhaps useful for reference) . We also want to be able to query partitions and parts (although for partitions we only want to query for numbers of things, not lists of things). Also, we'd like to able to query sets other than the root set (like that lovely blue set that spans a bunch of parts...).
I think it's the other way around: the part/partition resides in the mesh instance. In parallel, there's an implicit relation between a partition on one processor and on another processor, such that both/all processors understand that their local partition information is a piece of a collection over all processors. This model reduces to the trivial case for a single processor, both for partition(s) in mesh instances and parts in partitions.
I propose that we think about both parts and mesh instances as coverings of the global mesh (they are), and that we also think about parts, mesh instances, and the partition as containing -all- data about some (topologically and/or geometrically compact subset) or all of the global mesh (as opposed to entity sets, which are deliberately more selective). Yes, I know they're all collections of entities, but stick with me while I justify why I think this semantic distinction is worthwhile. Also yes, we haven't decided yet (have we?) whether a part will officially contain everything or just the entities that were partitioned; I don't -think- that any of what I'm about to propose is adversely affected, either way.
I don't think the partition can be a covering of the whole mesh on a given processor. Given a collection of 2d and 3d elements, you need to distinguish between the 2d elements that are just adjacent to objects in the partition and the 2d elements which are in fact objects in the partition.
I did not see anything following which justifies the semantic distinction you're talking about.
Finally, it's worth noting that, while a partition contains many parts distributed over many processes, each process is associated with a unique partition (MPI communicator or equivalent) and each part is associated both with a unique process and a unique partition. So specifying both part handle and partition handle is actually redundant.
Each part is associated with a unique partition and process, but each process is not associated with a unique partition. There may be multiple partitions in use at any given time, and a given process may be responsible for one or more parts in each partition. Take parallel contact detection for instance. One partition is over volume elements, the other is over faces on the skin of those elements. Processors participate in calculations for both partitions at different times.
Specifying both a part and partition handle may not be strictly redundant, and in terms of implementation I don't think it's useful to to require that the partition be derivable from the part handle. I do think that in most cases, an application will know the partition that's being dealt with. For the few cases where a part is a member of multiple partitions (if we want to allow that, and I don't see a major reason why not to), there should be a function to get the partition(s) that a part is a member of.
Given that backgroun, I propose that we overload our current query functions so that any place where a mesh instance is currently usable, a part handle or partition handle is also usable, and continue to use either a bona fide entity set or the (global placeholder) root set as the second argument. In this scenario, we would have, for instance:
/ partition handle getNumOfType| mesh instance , root set handle , type, result, err ) \ part handle entity set handle
The other iMesh function that is obviously of this type is getNumOfTopo.
A somewhat larger collection of functions would be able to take both mesh instances and part handles as the first argument, but not partition handles: getAllVtxCoords, getVtxCoordIndex, getEntities.
My guess is that iterator functions will fall into the second category.
Many of the set queries (number of sets, number of children, identities of sets and children) are a bit more ambiguous here, to my surprise: essentially, if you ask for the number of sets contained in a given set for a particular part, you presumably mean the number of contained sets that intersect that part. While I don't have a problem with that definition, I'm not sure it's an entirely straightforward one, either conceptually or in implementation. But I haven't thought about it in a great deal of detail, either... One way to finesse this would be to create sets at the part level rather than the mesh instance level; then all is easy again.
So again, I go back to asking: what are the core needs that prevent us from using sets as both parts and partitions? The entity set mechanism was designed with this specific usage in mind.
I know some haven't fully implemented sets, but is that going to be more difficult than the degree of interface changes being discussed here?
The things that are missing in sets for use as partitions and parts are exactly the extensions we'll already need in sets for going to parallel, no matter what we choose for partition and part representation. Specifically, we'll need some notion of a correspondence between sets on different processors, to handle boundary conditions on mesh spanning processors.
I'll make a diagram equivalent to Carl's today, maybe that will make things a bit clearer.
- tim
Now, there are a couple of beneath-the-hood requirements for implementations here:
1. All part handles must be unique, even in the presence of multiple partitions. Pointer-type handles will easily satisfy this; integer-type handles may need to reserve some bits for partition ID and some for part ID.
2. In most (all?) implementations, finding the result for calls with (part handle, entity set handle...) will require an implementation to do some sort of intersection internally. This will prove especially challenging for iterator in the presence of mesh modification and/or migration.
I recognize also that this paradigm shift potentially ambiguates the use of calls like addEntToSet for adding entities to a part; we can either overload those functions (probably renamed to addEntToCollection) or create new addEntToPart functions (I think there are only going to be four of these: add/rmv single/array).
Okay, so there's my proposal. I'll now stand back and let people poke holes in the idea; that may not prove to be too difficult.
Carl
------------------------------------------------------------------------