I think it's worthy to explain the distributed mesh design of the FMDB before we have further iMeshP discussion. The FMDB has 3-level hierarchy, "DB instance - mesh instance - part handle", where all entities are assigned to a part handle when it's created. - one global FMDB instance per machine - one mesh instance per process(or) : global FMDB instance can access N mesh instances on N process(or) machine - multiple parts per mesh instance : mesh instance can access all parts contained in it through part handle - each part has an entity container : when an entity is created, the part handle is required so the entity is contained in the part. : mesh instance can access all mesh entities through part handle - multiple parts are supported also in a serial mesh : when a mesh file is loaded in a serial machine, one part is created on a mesh instance if no specific request for multiple parts is made. We do not support "partition" at this moment. If we will allow multiple partitions per process, I would rather prefer the term "part-subpart" to "partition-part". Assuming one mesh instance per process (the FMDB design), if we think the definition of "partition" as a group of mesh instances which meet specific criteria (e.g. mesh instances on neighboring processors for expediting the communication), we will need more in depth investigation for what it really should be. The FMDB design described above works seamlessly both in parallel and serial environment. IOW, we view a serial mesh as a special case of parallel mesh, which means a serial mesh is a distributed mesh running on one process. So conceptually and technically, we do not have any difference between serial and parallel programs. By all means, the FMDB provides a single set of API which works both in parallel and serial, which is quite opposite to what we are doing for iMesh and iMeshP. I understand our history for providing two different mesh API for serial and parallel but not so sure if the end user will be ok with having two different set of API's with understanding of our excuse(?) of "history". Seegyoung
Notes from 4/12/10 phone conference attended by
RPI: Mark S., S. Yung (spelling?), Ting Uwisc: Tim T, Jason UBC: Carl SNL: Vitus, Karen LLNL: Mark M. Simmetrix: Mark B., Saurabh Tendulkar
Agreed: We will meet again on 4/19 at 8:30am PST.
Action Items:
1. Before our next meeting, write your view of the first big question below and email it to the list. Include pros, cons, how apps will use it, how it will affect implementations, etc. Mail it to [email protected].
2. Read other peoples' write-ups. Definitely reply for clarification of details, but let's save the discussion for the phone call where it is easier to follow.
My notes from the meeting:
Agreed: From the iMeshP perspective, an application will typically have, for a given mesh, one iMesh_instance per process, containing all entities stored on that process. The iMesh_instance may have a partition associated with it, with zero, one or multiple parts per process. The underlying implementation may implement this design in any way desired.
If I understand correctly, Simmetrix intends to implement multiple parts per process by having, internally, multiple "meshes," each representing a part. The iMesh functions would return results from the union of these meshes; the iMeshP part-specific functions would return results from individual meshes. (Does that sound correct, Mark B. and Saurabh?)
The first big question: Can an entity exist in an iMesh_instance that has a partition without being owned by any part? Discussion: There are several points in iMeshP operations where an entity may not (yet) have a part assignment. For example, entities are created with iMesh_createEnt; they can then be added to a part with iMesh_addEntToSet (overloaded with a part handle). But in-between the two function calls, the entity exists without being in a part. Some think this state is acceptable, with the understanding that parallel operations are not possible on entities that are not in parts. But this state causes problems for some implementations, which assume an entity is always in some part (with a default part assignment set during entity creation). If we require that an entity always be assigned to some part, we'll likely need additional interface functions, such as iMeshP_createEnt that creates an entity and assigns it to a part. Even if we don't have the requirement, we could consider adding such functions as a convenience.
The second big question: How should we enable multiple partitions of a single mesh in iMeshP?
The third question: Should iMeshP_destroyPart should return an error if the part to be destroyed is not empty? The answer to this question depends, to some extent, on our answer to the first big question above. Cons: Requires users to explicitly remove entities from parts before deleting the parts. This operation requires three function calls: iMeshP_getEntities, iMesh_rmvEntFromSet (overloaded with part handle), iMeshP_destroyPart. Pros: Prevents users from shooting themselves in the foot. Users would possibly call iMeshP_getEntities and iMesh_rmvEntFromSet anyway to do migration to new part.