Sorry for the radio silence. Some random thoughts: There seems to be dual meanings to Partition/Part/entities in a Part: one meaning is the collection of things sent to a partitioner like Zoltan and grouped into Parts. The other implies ownership of each entity, including other-dimensional entities adjacent to the entities sent to Zoltan (I say other here instead of lower to cover vertex-based partitions as well as region-based partitions). I think Jason's argument is important here: the easy things should be easy, and more complicated things possible. I think this is related to Seegyoung's point about having a single API. I think what this boils down to is an assertion that we should think primarily in terms of one iMesh Instance per process. For those applications wanting to use multiple instances, that implies communication between the instances at least conceptually, if not in the implementation also. What this boils down to even further is that Part and iMesh instance are synonymous. The issue of an entity assigned to exactly one Part morphs to one of representation vs. ownership: an entity can be represented in more than one instance (e.g. faces on part boundaries, in an element-based partition), but can be owned by only one of those instances. One of the reasons we initially thought in terms of multiple parts per process was to handle over-partitionings, where a process could be assigned multiple parts, e.g. for better load balancing, and flexibility in what's stored with a mesh or how it's stored. I think that concept is an artifact of how a partitioning is stored with the mesh (or how the mesh is stored in files to reflect a partitioning). From an application's point of view, in most cases it's still going to want to access the mesh from a single instance on a given process, no matter how that mesh was loaded initially. That's also how the mesh looks from most applications' points of view. The use case of multiple parts, with shared interface entities duplicated, is mostly handled by having separate entities in the database, with special relations between them implying that they're the same. One of the complications that arises from having one instance per process (and one part per instance) is how do you repartition, or partition in serial. I think that concept is handled by the notion of initializing a given parallel mesh either a) from a collection of collections, or a partition of parts, stored with and loaded with the mesh, or b) communication from one instance to another, using parallel migration of entities. Of course, a) is easily handled using sets, if you have them; if you don't, it's just as easily handled using multiple files. It also naturally handles establishing a parallel mesh based on one of several "partitionings" (collection of collections) stored with a mesh, e.g. using the geometric volumes, material types, or true partitions (generated by a partitioner). It's almost sounding to me like I'm agreeing mostly with Mark B's view of things, with the added concept of loading a parallel mesh based on some collection structure. The next question becomes whether we can standardize the method for specifying that collectionn structure from applications. I can describe how MOAB does that now, that works for the three types of partitions mentioned above, if anybody's interested. - tim Devine, Karen D wrote:
Notes from 4/12/10 phone conference attended by
RPI: Mark S., S. Yung (spelling?), Ting Uwisc: Tim T, Jason UBC: Carl SNL: Vitus, Karen LLNL: Mark M. Simmetrix: Mark B., Saurabh Tendulkar
Agreed: We will meet again on 4/19 at 8:30am PST.
Action Items:
1. Before our next meeting, write your view of the first big question below and email it to the list. Include pros, cons, how apps will use it, how it will affect implementations, etc. Mail it to [email protected].
2. Read other peoples' write-ups. Definitely reply for clarification of details, but let's save the discussion for the phone call where it is easier to follow.
My notes from the meeting:
Agreed: From the iMeshP perspective, an application will typically have, for a given mesh, one iMesh_instance per process, containing all entities stored on that process. The iMesh_instance may have a partition associated with it, with zero, one or multiple parts per process. The underlying implementation may implement this design in any way desired.
If I understand correctly, Simmetrix intends to implement multiple parts per process by having, internally, multiple "meshes," each representing a part. The iMesh functions would return results from the union of these meshes; the iMeshP part-specific functions would return results from individual meshes. (Does that sound correct, Mark B. and Saurabh?)
The first big question: Can an entity exist in an iMesh_instance that has a partition without being owned by any part? Discussion: There are several points in iMeshP operations where an entity may not (yet) have a part assignment. For example, entities are created with iMesh_createEnt; they can then be added to a part with iMesh_addEntToSet (overloaded with a part handle). But in-between the two function calls, the entity exists without being in a part. Some think this state is acceptable, with the understanding that parallel operations are not possible on entities that are not in parts. But this state causes problems for some implementations, which assume an entity is always in some part (with a default part assignment set during entity creation). If we require that an entity always be assigned to some part, we'll likely need additional interface functions, such as iMeshP_createEnt that creates an entity and assigns it to a part. Even if we don't have the requirement, we could consider adding such functions as a convenience.
The second big question: How should we enable multiple partitions of a single mesh in iMeshP?
The third question: Should iMeshP_destroyPart should return an error if the part to be destroyed is not empty? The answer to this question depends, to some extent, on our answer to the first big question above. Cons: Requires users to explicitly remove entities from parts before deleting the parts. This operation requires three function calls: iMeshP_getEntities, iMesh_rmvEntFromSet (overloaded with part handle), iMeshP_destroyPart. Pros: Prevents users from shooting themselves in the foot. Users would possibly call iMeshP_getEntities and iMesh_rmvEntFromSet anyway to do migration to new part.
-- ================================================================ "You will keep in perfect peace him whose mind is steadfast, because he trusts in you." Isaiah 26:3 Tim Tautges Argonne National Laboratory ([email protected]) (telecommuting from UW-Madison) phone: (608) 263-8485 1500 Engineering Dr. fax: (608) 263-4499 Madison, WI 53706