Notes from 4/12/10 phone conference attended by RPI: Mark S., S. Yung (spelling?), Ting Uwisc: Tim T, Jason UBC: Carl SNL: Vitus, Karen LLNL: Mark M. Simmetrix: Mark B., Saurabh Tendulkar Agreed: We will meet again on 4/19 at 8:30am PST. Action Items: 1. Before our next meeting, write your view of the first big question below and email it to the list. Include pros, cons, how apps will use it, how it will affect implementations, etc. Mail it to [email protected]. 2. Read other peoples' write-ups. Definitely reply for clarification of details, but let's save the discussion for the phone call where it is easier to follow. My notes from the meeting: Agreed: From the iMeshP perspective, an application will typically have, for a given mesh, one iMesh_instance per process, containing all entities stored on that process. The iMesh_instance may have a partition associated with it, with zero, one or multiple parts per process. The underlying implementation may implement this design in any way desired. If I understand correctly, Simmetrix intends to implement multiple parts per process by having, internally, multiple "meshes," each representing a part. The iMesh functions would return results from the union of these meshes; the iMeshP part-specific functions would return results from individual meshes. (Does that sound correct, Mark B. and Saurabh?) The first big question: Can an entity exist in an iMesh_instance that has a partition without being owned by any part? Discussion: There are several points in iMeshP operations where an entity may not (yet) have a part assignment. For example, entities are created with iMesh_createEnt; they can then be added to a part with iMesh_addEntToSet (overloaded with a part handle). But in-between the two function calls, the entity exists without being in a part. Some think this state is acceptable, with the understanding that parallel operations are not possible on entities that are not in parts. But this state causes problems for some implementations, which assume an entity is always in some part (with a default part assignment set during entity creation). If we require that an entity always be assigned to some part, we'll likely need additional interface functions, such as iMeshP_createEnt that creates an entity and assigns it to a part. Even if we don't have the requirement, we could consider adding such functions as a convenience. The second big question: How should we enable multiple partitions of a single mesh in iMeshP? The third question: Should iMeshP_destroyPart should return an error if the part to be destroyed is not empty? The answer to this question depends, to some extent, on our answer to the first big question above. Cons: Requires users to explicitly remove entities from parts before deleting the parts. This operation requires three function calls: iMeshP_getEntities, iMesh_rmvEntFromSet (overloaded with part handle), iMeshP_destroyPart. Pros: Prevents users from shooting themselves in the foot. Users would possibly call iMeshP_getEntities and iMesh_rmvEntFromSet anyway to do migration to new part.
Karen, I was also there but did not tell you because people were talking. But I have taken note of everything. Xiaolin On Mon, 2010-04-12 at 12:46 -0600, Devine, Karen D wrote:
Notes from 4/12/10 phone conference attended by
RPI: Mark S., S. Yung (spelling?), Ting Uwisc: Tim T, Jason UBC: Carl SNL: Vitus, Karen LLNL: Mark M. Simmetrix: Mark B., Saurabh Tendulkar
Agreed: We will meet again on 4/19 at 8:30am PST.
Action Items:
1. Before our next meeting, write your view of the first big question below and email it to the list. Include pros, cons, how apps will use it, how it will affect implementations, etc. Mail it to [email protected].
2. Read other peoples' write-ups. Definitely reply for clarification of details, but let's save the discussion for the phone call where it is easier to follow.
My notes from the meeting:
Agreed: From the iMeshP perspective, an application will typically have, for a given mesh, one iMesh_instance per process, containing all entities stored on that process. The iMesh_instance may have a partition associated with it, with zero, one or multiple parts per process. The underlying implementation may implement this design in any way desired.
If I understand correctly, Simmetrix intends to implement multiple parts per process by having, internally, multiple "meshes," each representing a part. The iMesh functions would return results from the union of these meshes; the iMeshP part-specific functions would return results from individual meshes. (Does that sound correct, Mark B. and Saurabh?)
The first big question: Can an entity exist in an iMesh_instance that has a partition without being owned by any part? Discussion: There are several points in iMeshP operations where an entity may not (yet) have a part assignment. For example, entities are created with iMesh_createEnt; they can then be added to a part with iMesh_addEntToSet (overloaded with a part handle). But in-between the two function calls, the entity exists without being in a part. Some think this state is acceptable, with the understanding that parallel operations are not possible on entities that are not in parts. But this state causes problems for some implementations, which assume an entity is always in some part (with a default part assignment set during entity creation). If we require that an entity always be assigned to some part, we'll likely need additional interface functions, such as iMeshP_createEnt that creates an entity and assigns it to a part. Even if we don't have the requirement, we could consider adding such functions as a convenience.
The second big question: How should we enable multiple partitions of a single mesh in iMeshP?
The third question: Should iMeshP_destroyPart should return an error if the part to be destroyed is not empty? The answer to this question depends, to some extent, on our answer to the first big question above. Cons: Requires users to explicitly remove entities from parts before deleting the parts. This operation requires three function calls: iMeshP_getEntities, iMesh_rmvEntFromSet (overloaded with part handle), iMeshP_destroyPart. Pros: Prevents users from shooting themselves in the foot. Users would possibly call iMeshP_getEntities and iMesh_rmvEntFromSet anyway to do migration to new part.
Karen, Summary looks good. Thanks. The RPI people were Seegyoung Seol and Ting Xie. Mark Devine, Karen D wrote:
Notes from 4/12/10 phone conference attended by
RPI: Mark S., S. Yung (spelling?), Ting Uwisc: Tim T, Jason UBC: Carl SNL: Vitus, Karen LLNL: Mark M. Simmetrix: Mark B., Saurabh Tendulkar
Agreed: We will meet again on 4/19 at 8:30am PST.
Action Items:
1. Before our next meeting, write your view of the first big question below and email it to the list. Include pros, cons, how apps will use it, how it will affect implementations, etc. Mail it to [email protected].
2. Read other peoples' write-ups. Definitely reply for clarification of details, but let's save the discussion for the phone call where it is easier to follow.
My notes from the meeting:
Agreed: From the iMeshP perspective, an application will typically have, for a given mesh, one iMesh_instance per process, containing all entities stored on that process. The iMesh_instance may have a partition associated with it, with zero, one or multiple parts per process. The underlying implementation may implement this design in any way desired.
If I understand correctly, Simmetrix intends to implement multiple parts per process by having, internally, multiple "meshes," each representing a part. The iMesh functions would return results from the union of these meshes; the iMeshP part-specific functions would return results from individual meshes. (Does that sound correct, Mark B. and Saurabh?)
The first big question: Can an entity exist in an iMesh_instance that has a partition without being owned by any part? Discussion: There are several points in iMeshP operations where an entity may not (yet) have a part assignment. For example, entities are created with iMesh_createEnt; they can then be added to a part with iMesh_addEntToSet (overloaded with a part handle). But in-between the two function calls, the entity exists without being in a part. Some think this state is acceptable, with the understanding that parallel operations are not possible on entities that are not in parts. But this state causes problems for some implementations, which assume an entity is always in some part (with a default part assignment set during entity creation). If we require that an entity always be assigned to some part, we'll likely need additional interface functions, such as iMeshP_createEnt that creates an entity and assigns it to a part. Even if we don't have the requirement, we could consider adding such functions as a convenience.
The second big question: How should we enable multiple partitions of a single mesh in iMeshP?
The third question: Should iMeshP_destroyPart should return an error if the part to be destroyed is not empty? The answer to this question depends, to some extent, on our answer to the first big question above. Cons: Requires users to explicitly remove entities from parts before deleting the parts. This operation requires three function calls: iMeshP_getEntities, iMesh_rmvEntFromSet (overloaded with part handle), iMeshP_destroyPart. Pros: Prevents users from shooting themselves in the foot. Users would possibly call iMeshP_getEntities and iMesh_rmvEntFromSet anyway to do migration to new part.
A couple comments on your notes below. mark On Apr 12, 2010, at 2:46 PM, Devine, Karen D wrote:
Notes from 4/12/10 phone conference attended by
RPI: Mark S., S. Yung (spelling?), Ting Uwisc: Tim T, Jason UBC: Carl SNL: Vitus, Karen LLNL: Mark M. Simmetrix: Mark B., Saurabh Tendulkar
Agreed: We will meet again on 4/19 at 8:30am PST.
Action Items:
1. Before our next meeting, write your view of the first big question below and email it to the list. Include pros, cons, how apps will use it, how it will affect implementations, etc. Mail it to [email protected].
2. Read other peoples' write-ups. Definitely reply for clarification of details, but let's save the discussion for the phone call where it is easier to follow.
My notes from the meeting:
Agreed: From the iMeshP perspective, an application will typically have, for a given mesh, one iMesh_instance per process, containing all entities stored on that process. The iMesh_instance may have a partition associated with it, with zero, one or multiple parts per process. The underlying implementation may implement this design in any way desired.
The question we were asking related to this was whether that was the intent of the original iMeshP design. I certainly confirm we agreed that was the original intent. I'll reserve agreement on whether that's actually the best choice.
If I understand correctly, Simmetrix intends to implement multiple parts per process by having, internally, multiple "meshes," each representing a part. The iMesh functions would return results from the union of these meshes; the iMeshP part-specific functions would return results from individual meshes. (Does that sound correct, Mark B. and Saurabh?)
Yes, that is likely how we would deal with that. As discussed it still does leave open issues related to your questions below.
The first big question: Can an entity exist in an iMesh_instance that has a partition without being owned by any part? Discussion: There are several points in iMeshP operations where an entity may not (yet) have a part assignment. For example, entities are created with iMesh_createEnt; they can then be added to a part with iMesh_addEntToSet (overloaded with a part handle). But in-between the two function calls, the entity exists without being in a part. Some think this state is acceptable, with the understanding that parallel operations are not possible on entities that are not in parts. But this state causes problems for some implementations, which assume an entity is always in some part (with a default part assignment set during entity creation). If we require that an entity always be assigned to some part, we'll likely need additional interface functions, such as iMeshP_createEnt that creates an entity and assigns it to a part. Even if we don't have the requirement, we could consider adding such functions as a convenience.
The second big question: How should we enable multiple partitions of a single mesh in iMeshP?
The third question: Should iMeshP_destroyPart should return an error if the part to be destroyed is not empty? The answer to this question depends, to some extent, on our answer to the first big question above. Cons: Requires users to explicitly remove entities from parts before deleting the parts. This operation requires three function calls: iMeshP_getEntities, iMesh_rmvEntFromSet (overloaded with part handle), iMeshP_destroyPart. Pros: Prevents users from shooting themselves in the foot. Users would possibly call iMeshP_getEntities and iMesh_rmvEntFromSet anyway to do migration to new part.
On Apr 12, 2010, at 2:46 PM, Devine, Karen D wrote:
The first big question: Can an entity exist in an iMesh_instance that has a partition without being owned by any part?
First, a disclaimer: somewhere in this I'm going to write partition instead of part (since what you call "parts" we call "partitions", what you call a "partition" we call a "partitioning"). I will try to avoid that. It seems to me that a, not unreasonable, implementation of a partitioned mesh is one where each part is equivalent to a mesh (when I say "mesh" in here I mean "mesh instance" in the iMesh sense). This is the way that our partitioned mesh works and, I believe, pretty much how RPI's works. So we have 2 out of (how many?) iMeshP implementations that are designed this way. From here on I'm going to assume that this is a reasonable implementation, so my conclusions are only valid if you agree with that. The pros of this implementation: - Since there is already a way of knowing what entities are in a mesh, there is zero overhead, for the single partition case, of knowing what entities are in a part. I think this is an important consideration since the vast majority of applications won't use multiple partitions. - There is no need for functions to loop over the contents of a part, as there are already those for a mesh - I would think that Tim would like this approach for this reason - smaller API :) - Code that works in serial on a mesh can be used for anything that can be done on just the interior of a part (or on the entire part if it doesn't modify the part boundary) - Since operations to create mesh entities are done on a mesh, there isn't ever any question on what part an entity belongs to. The cons of this implementation: - It's not obvious how to deal with multiple partitions (answer to this in second question below) - entities on the part boundary must be duplicated even if they are on the same process - (please feel free to add to this list since I can't think of any others) Some conceptual differences of this implementation: - rather than taking entities out of a part and putting them in another, you change their part ownership by telling them (or something doing the repartitioning), "I'm going to want you to be in this other part later". Then after you've done that for all the entities (in a consistent way), you repartition the mesh. The nice thing about this is that there is never a point in time (from the user perspective) that the mesh and the parts are inconsistent in any way. For this kind of implementation, allowing a situation where a mesh entity is not in a part is equivalent to it not being in a mesh. While one could certainly have implementations where that is allowed, in our implementation - which I think is consistent with the intents of iMesh - the mesh owns the mesh entities (in the OO sense - if the mesh is deleted it takes all of the mesh entities with it). As was mentioned in the paragraph I deleted below, when we have multiple parts per process, we have multiple meshes per process one per part. We can certainly make those look like a single iMesh_instance through the interface. In reality we could also have an additional hidden mesh that was the home for any mesh entities that aren't on a partition. The downsides to that are a) it's relatively expensive to move entities from one mesh to another for us (you have to copy it and delete the old one - which really gets messy when something off-process might be referring to it), b) there are rules that don't allow an entity to attach to lower order entities in a different mesh (for very good reasons). This isn't an answer to "the first big question" but is a explanation of why it's a problem for some implementations.
The second big question: How should we enable multiple partitions of a single mesh in iMeshP?
My understanding from what's been discussed on this mailing list is that it's expected that different partitions don't have any correlation to each other (in other words, it's the rare exception that two different partitions will result in entities being on the same process in both). If you don't agree with that, then, again, my conclusions based on that are invalid. If partitions are not correlated then any entity in more than one partition must effectively be copied for each additional partition they are in. The only way for an entity to exist on more than one process is for there to be more than one copy of it (if someone has cases where there are multiple partitions but they don't actually need to know any information about the entities in one - other than that they are in it - that would be a counter example to this). Given this then a second (or nth) partition is really just a partition of a copy of a mesh (or a subset of entities in the mesh). So why try really hard to hide that fact? I've heard Mark S. say that there is a need for another mechanism for dividing up a mesh (I'm purposely avoiding the word partitioning here) on these currently mythical computers that will exist in 10 years. It's going to be scary if there are meshes, partitions and parts, and, umm, divisions and divs ...
The third question: Should iMeshP_destroyPart should return an error if the part to be destroyed is not empty? The answer to this question depends, to some extent, on our answer to the first big question above. Cons: Requires users to explicitly remove entities from parts before deleting the parts. This operation requires three function calls: iMeshP_getEntities, iMesh_rmvEntFromSet (overloaded with part handle), iMeshP_destroyPart. Pros: Prevents users from shooting themselves in the foot. Users would possibly call iMeshP_getEntities and iMesh_rmvEntFromSet anyway to do migration to new part.
This becomes a moot point in my world... mark
The things Mark Beall indicates below sound about right with respect to the RPI iMeshP without going into the implementation details of a "mesh instance" since others will have to comment on implementation details. Since there are a limited number of iMeshP implementations, I do think that considering the experience of those implementations in refining the iMeshP design does make some sense. In terms of the machines of the future, they are not that far down the line. We all hope to be working on the Argonne Blue Gene/Q early in 2012. That machine will have 16 cores per node and I expect that to get good performance out of that machine that we will do something on the order of defining parts on the node level and doing something different across the cores of the node. (This is instead of having the parts at the core level.) Mark Mark Beall wrote:
On Apr 12, 2010, at 2:46 PM, Devine, Karen D wrote:
The first big question: Can an entity exist in an iMesh_instance that has a partition without being owned by any part?
First, a disclaimer: somewhere in this I'm going to write partition instead of part (since what you call "parts" we call "partitions", what you call a "partition" we call a "partitioning"). I will try to avoid that.
It seems to me that a, not unreasonable, implementation of a partitioned mesh is one where each part is equivalent to a mesh (when I say "mesh" in here I mean "mesh instance" in the iMesh sense). This is the way that our partitioned mesh works and, I believe, pretty much how RPI's works. So we have 2 out of (how many?) iMeshP implementations that are designed this way. From here on I'm going to assume that this is a reasonable implementation, so my conclusions are only valid if you agree with that.
The pros of this implementation: - Since there is already a way of knowing what entities are in a mesh, there is zero overhead, for the single partition case, of knowing what entities are in a part. I think this is an important consideration since the vast majority of applications won't use multiple partitions. - There is no need for functions to loop over the contents of a part, as there are already those for a mesh - I would think that Tim would like this approach for this reason - smaller API :) - Code that works in serial on a mesh can be used for anything that can be done on just the interior of a part (or on the entire part if it doesn't modify the part boundary) - Since operations to create mesh entities are done on a mesh, there isn't ever any question on what part an entity belongs to.
The cons of this implementation: - It's not obvious how to deal with multiple partitions (answer to this in second question below) - entities on the part boundary must be duplicated even if they are on the same process - (please feel free to add to this list since I can't think of any others)
Some conceptual differences of this implementation: - rather than taking entities out of a part and putting them in another, you change their part ownership by telling them (or something doing the repartitioning), "I'm going to want you to be in this other part later". Then after you've done that for all the entities (in a consistent way), you repartition the mesh. The nice thing about this is that there is never a point in time (from the user perspective) that the mesh and the parts are inconsistent in any way.
For this kind of implementation, allowing a situation where a mesh entity is not in a part is equivalent to it not being in a mesh. While one could certainly have implementations where that is allowed, in our implementation - which I think is consistent with the intents of iMesh - the mesh owns the mesh entities (in the OO sense - if the mesh is deleted it takes all of the mesh entities with it).
As was mentioned in the paragraph I deleted below, when we have multiple parts per process, we have multiple meshes per process one per part. We can certainly make those look like a single iMesh_instance through the interface. In reality we could also have an additional hidden mesh that was the home for any mesh entities that aren't on a partition. The downsides to that are a) it's relatively expensive to move entities from one mesh to another for us (you have to copy it and delete the old one - which really gets messy when something off-process might be referring to it), b) there are rules that don't allow an entity to attach to lower order entities in a different mesh (for very good reasons).
This isn't an answer to "the first big question" but is a explanation of why it's a problem for some implementations.
The second big question: How should we enable multiple partitions of a single mesh in iMeshP?
My understanding from what's been discussed on this mailing list is that it's expected that different partitions don't have any correlation to each other (in other words, it's the rare exception that two different partitions will result in entities being on the same process in both). If you don't agree with that, then, again, my conclusions based on that are invalid.
If partitions are not correlated then any entity in more than one partition must effectively be copied for each additional partition they are in. The only way for an entity to exist on more than one process is for there to be more than one copy of it (if someone has cases where there are multiple partitions but they don't actually need to know any information about the entities in one - other than that they are in it - that would be a counter example to this).
Given this then a second (or nth) partition is really just a partition of a copy of a mesh (or a subset of entities in the mesh). So why try really hard to hide that fact?
I've heard Mark S. say that there is a need for another mechanism for dividing up a mesh (I'm purposely avoiding the word partitioning here) on these currently mythical computers that will exist in 10 years. It's going to be scary if there are meshes, partitions and parts, and, umm, divisions and divs ...
The third question: Should iMeshP_destroyPart should return an error if the part to be destroyed is not empty? The answer to this question depends, to some extent, on our answer to the first big question above. Cons: Requires users to explicitly remove entities from parts before deleting the parts. This operation requires three function calls: iMeshP_getEntities, iMesh_rmvEntFromSet (overloaded with part handle), iMeshP_destroyPart. Pros: Prevents users from shooting themselves in the foot. Users would possibly call iMeshP_getEntities and iMesh_rmvEntFromSet anyway to do migration to new part.
This becomes a moot point in my world...
mark
Mark, I think that having one iMesh instance per Part would be problematic for "contact"-type problems. If I understand the nature of the problem correctly, there are two partitioning of the mesh: one that distributes volume elements and one that distributes 2D elements on interfaces. If there were multiple (for simplicity lets say two) iMesh instances for a single process where one contains the volume elements and one contains the 2D elements then there would be no way to obtain information about the relation between the 2D elements and the volume elements. - jason
Jason, It is a strong function of how one would support the mesh in parallel. I do know how I would do partitioning and dynamic re-partitioning (to account for contact evolution) for efficient parallel simulation using iMeshP's FMDB implementation and the issue of where the mesh instance is defined creates no problem at all. Mark Jason Kraftcheck wrote:
Mark,
I think that having one iMesh instance per Part would be problematic for "contact"-type problems. If I understand the nature of the problem correctly, there are two partitioning of the mesh: one that distributes volume elements and one that distributes 2D elements on interfaces. If there were multiple (for simplicity lets say two) iMesh instances for a single process where one contains the volume elements and one contains the 2D elements then there would be no way to obtain information about the relation between the 2D elements and the volume elements.
- jason
Jason, Assuming that the two partitions are not 100% correlated (the interface elements are not on the same processors as their associated volume elements), I think you have the same problem with the current design. The interface elements will be copies on different processors in the second partition. If someone has a way to avoid this, feel free to point it out since it's the basis of my entire argument here. If the two partitions are 100% correlated, then it's the same as if you were doing it in serial and in that case I don't see that you would want to use a partition to represent the interface elements, you'd just use a mesh entity set. mark On Apr 13, 2010, at 10:24 AM, Jason Kraftcheck wrote:
Mark,
I think that having one iMesh instance per Part would be problematic for "contact"-type problems. If I understand the nature of the problem correctly, there are two partitioning of the mesh: one that distributes volume elements and one that distributes 2D elements on interfaces. If there were multiple (for simplicity lets say two) iMesh instances for a single process where one contains the volume elements and one contains the 2D elements then there would be no way to obtain information about the relation between the 2D elements and the volume elements.
- jason
I have a more radical proposal for addressing the issues from the last iMeshP telecon. Re-consider the idea of an "active" partition. That is, there is one partition that corresponds to how elements are currently distributed. The application can define other partitions with the intent of either transitioning to that partition as the active one directly or writing the mesh to disk with that defined as the partition that will be active when the mesh is read back in. Let's take this idea one step further and define the active partition to always have one part per process*. If there were more parts than processes in the input data, then there may exist an inactive partition corresponding to what was defined in the input. As far as I could follow the discussion, there were two proposed mechanisms for re-partitioning the mesh in parallel. The first was to move entities between parts in the active partition and the second was to define a new partitioning completely before transitioning to that. I mentioned the latter above. As for the former, if there were fewer parts than processes when the mesh is loaded, then some processes would end up with empty Parts in the active partition. Existing functions could be used to move entities between Parts. So far, we've assumed one iMesh instance per process. Mark B. would like one iMesh instance per Part. Having one active Part per process results in both of those being true for the active partition. As for the meaning of iMeshP_destroyPart: the 'active' partition is implicitly defined by the number of processes. It would be an error to ask to create or destroy a part in the active partition because that would imply adding or removing processes, which is beyond the scope of iMeshP. For modification of in-active partitions, deleting the part doesn't imply anything about the ownership of the entity so it shouldn't be a problem to delete non-empty parts. But beyond all of the above, the biggest advantage of this change is the simplification of the data model for simple use cases. I worry that allowing for multiple parts per process and such other complexities that are unnecessary for simple use cases (e.g. a simple heat-transfer simulation) might impede adoption of the API. It is important that the API support re-partitioning and other advanced use cases, but it should also be simple and easy to understand and use for the many simpler use cases. And having multiple iMesh instances per process is definitely moving in the wrong direction on the simplicity scale. It is perhaps a little late in the game for API changes, but this change would facility some simplification of the API also. Consider some existing code for which someone wants to implement the iMeshP API so as to integrate some other simulation code. Our current API seems overly complicated to implement for someone who as no interest in parallel repartitioning. We could divide the current API into two separate subsets: one for communicating data and one for querying and modifying inactive partitions. All functions in the former could work with rank rather than a part ID or part handle. The only overlap between the two sub-APIs would be the function to transition to a different active partition. Functions for adding and removing entities from Parts could go with the partition modification subset. For redistribution within the active partition, a simple function to change ownership would be sufficient and need not even do any communication. Presumably any such function would need to be called on at least the current and new owning processes (and presumably also any processes which have a ghost of the entity.) If the function accepts an entity handle then the entity must already be ghosted on the new owning process. The existing ghosting code can be used to communicate entities and the change of ownership can then be done with no communication at all. - jason * Perhaps 'process' isn't the correct granularity for the future really big machines that Mark S. mentioned. I use that term here on the assumption that there will be a single multi-threaded process on a multi-core node in that case. If that assumption is incorrect, substitute 'node' or whatever is appropriate in place of 'process'.
I think it's worthy to explain the distributed mesh design of the FMDB before we have further iMeshP discussion. The FMDB has 3-level hierarchy, "DB instance - mesh instance - part handle", where all entities are assigned to a part handle when it's created. - one global FMDB instance per machine - one mesh instance per process(or) : global FMDB instance can access N mesh instances on N process(or) machine - multiple parts per mesh instance : mesh instance can access all parts contained in it through part handle - each part has an entity container : when an entity is created, the part handle is required so the entity is contained in the part. : mesh instance can access all mesh entities through part handle - multiple parts are supported also in a serial mesh : when a mesh file is loaded in a serial machine, one part is created on a mesh instance if no specific request for multiple parts is made. We do not support "partition" at this moment. If we will allow multiple partitions per process, I would rather prefer the term "part-subpart" to "partition-part". Assuming one mesh instance per process (the FMDB design), if we think the definition of "partition" as a group of mesh instances which meet specific criteria (e.g. mesh instances on neighboring processors for expediting the communication), we will need more in depth investigation for what it really should be. The FMDB design described above works seamlessly both in parallel and serial environment. IOW, we view a serial mesh as a special case of parallel mesh, which means a serial mesh is a distributed mesh running on one process. So conceptually and technically, we do not have any difference between serial and parallel programs. By all means, the FMDB provides a single set of API which works both in parallel and serial, which is quite opposite to what we are doing for iMesh and iMeshP. I understand our history for providing two different mesh API for serial and parallel but not so sure if the end user will be ok with having two different set of API's with understanding of our excuse(?) of "history". Seegyoung
Notes from 4/12/10 phone conference attended by
RPI: Mark S., S. Yung (spelling?), Ting Uwisc: Tim T, Jason UBC: Carl SNL: Vitus, Karen LLNL: Mark M. Simmetrix: Mark B., Saurabh Tendulkar
Agreed: We will meet again on 4/19 at 8:30am PST.
Action Items:
1. Before our next meeting, write your view of the first big question below and email it to the list. Include pros, cons, how apps will use it, how it will affect implementations, etc. Mail it to [email protected].
2. Read other peoples' write-ups. Definitely reply for clarification of details, but let's save the discussion for the phone call where it is easier to follow.
My notes from the meeting:
Agreed: From the iMeshP perspective, an application will typically have, for a given mesh, one iMesh_instance per process, containing all entities stored on that process. The iMesh_instance may have a partition associated with it, with zero, one or multiple parts per process. The underlying implementation may implement this design in any way desired.
If I understand correctly, Simmetrix intends to implement multiple parts per process by having, internally, multiple "meshes," each representing a part. The iMesh functions would return results from the union of these meshes; the iMeshP part-specific functions would return results from individual meshes. (Does that sound correct, Mark B. and Saurabh?)
The first big question: Can an entity exist in an iMesh_instance that has a partition without being owned by any part? Discussion: There are several points in iMeshP operations where an entity may not (yet) have a part assignment. For example, entities are created with iMesh_createEnt; they can then be added to a part with iMesh_addEntToSet (overloaded with a part handle). But in-between the two function calls, the entity exists without being in a part. Some think this state is acceptable, with the understanding that parallel operations are not possible on entities that are not in parts. But this state causes problems for some implementations, which assume an entity is always in some part (with a default part assignment set during entity creation). If we require that an entity always be assigned to some part, we'll likely need additional interface functions, such as iMeshP_createEnt that creates an entity and assigns it to a part. Even if we don't have the requirement, we could consider adding such functions as a convenience.
The second big question: How should we enable multiple partitions of a single mesh in iMeshP?
The third question: Should iMeshP_destroyPart should return an error if the part to be destroyed is not empty? The answer to this question depends, to some extent, on our answer to the first big question above. Cons: Requires users to explicitly remove entities from parts before deleting the parts. This operation requires three function calls: iMeshP_getEntities, iMesh_rmvEntFromSet (overloaded with part handle), iMeshP_destroyPart. Pros: Prevents users from shooting themselves in the foot. Users would possibly call iMeshP_getEntities and iMesh_rmvEntFromSet anyway to do migration to new part.
Sorry for the radio silence. Some random thoughts: There seems to be dual meanings to Partition/Part/entities in a Part: one meaning is the collection of things sent to a partitioner like Zoltan and grouped into Parts. The other implies ownership of each entity, including other-dimensional entities adjacent to the entities sent to Zoltan (I say other here instead of lower to cover vertex-based partitions as well as region-based partitions). I think Jason's argument is important here: the easy things should be easy, and more complicated things possible. I think this is related to Seegyoung's point about having a single API. I think what this boils down to is an assertion that we should think primarily in terms of one iMesh Instance per process. For those applications wanting to use multiple instances, that implies communication between the instances at least conceptually, if not in the implementation also. What this boils down to even further is that Part and iMesh instance are synonymous. The issue of an entity assigned to exactly one Part morphs to one of representation vs. ownership: an entity can be represented in more than one instance (e.g. faces on part boundaries, in an element-based partition), but can be owned by only one of those instances. One of the reasons we initially thought in terms of multiple parts per process was to handle over-partitionings, where a process could be assigned multiple parts, e.g. for better load balancing, and flexibility in what's stored with a mesh or how it's stored. I think that concept is an artifact of how a partitioning is stored with the mesh (or how the mesh is stored in files to reflect a partitioning). From an application's point of view, in most cases it's still going to want to access the mesh from a single instance on a given process, no matter how that mesh was loaded initially. That's also how the mesh looks from most applications' points of view. The use case of multiple parts, with shared interface entities duplicated, is mostly handled by having separate entities in the database, with special relations between them implying that they're the same. One of the complications that arises from having one instance per process (and one part per instance) is how do you repartition, or partition in serial. I think that concept is handled by the notion of initializing a given parallel mesh either a) from a collection of collections, or a partition of parts, stored with and loaded with the mesh, or b) communication from one instance to another, using parallel migration of entities. Of course, a) is easily handled using sets, if you have them; if you don't, it's just as easily handled using multiple files. It also naturally handles establishing a parallel mesh based on one of several "partitionings" (collection of collections) stored with a mesh, e.g. using the geometric volumes, material types, or true partitions (generated by a partitioner). It's almost sounding to me like I'm agreeing mostly with Mark B's view of things, with the added concept of loading a parallel mesh based on some collection structure. The next question becomes whether we can standardize the method for specifying that collectionn structure from applications. I can describe how MOAB does that now, that works for the three types of partitions mentioned above, if anybody's interested. - tim Devine, Karen D wrote:
Notes from 4/12/10 phone conference attended by
RPI: Mark S., S. Yung (spelling?), Ting Uwisc: Tim T, Jason UBC: Carl SNL: Vitus, Karen LLNL: Mark M. Simmetrix: Mark B., Saurabh Tendulkar
Agreed: We will meet again on 4/19 at 8:30am PST.
Action Items:
1. Before our next meeting, write your view of the first big question below and email it to the list. Include pros, cons, how apps will use it, how it will affect implementations, etc. Mail it to [email protected].
2. Read other peoples' write-ups. Definitely reply for clarification of details, but let's save the discussion for the phone call where it is easier to follow.
My notes from the meeting:
Agreed: From the iMeshP perspective, an application will typically have, for a given mesh, one iMesh_instance per process, containing all entities stored on that process. The iMesh_instance may have a partition associated with it, with zero, one or multiple parts per process. The underlying implementation may implement this design in any way desired.
If I understand correctly, Simmetrix intends to implement multiple parts per process by having, internally, multiple "meshes," each representing a part. The iMesh functions would return results from the union of these meshes; the iMeshP part-specific functions would return results from individual meshes. (Does that sound correct, Mark B. and Saurabh?)
The first big question: Can an entity exist in an iMesh_instance that has a partition without being owned by any part? Discussion: There are several points in iMeshP operations where an entity may not (yet) have a part assignment. For example, entities are created with iMesh_createEnt; they can then be added to a part with iMesh_addEntToSet (overloaded with a part handle). But in-between the two function calls, the entity exists without being in a part. Some think this state is acceptable, with the understanding that parallel operations are not possible on entities that are not in parts. But this state causes problems for some implementations, which assume an entity is always in some part (with a default part assignment set during entity creation). If we require that an entity always be assigned to some part, we'll likely need additional interface functions, such as iMeshP_createEnt that creates an entity and assigns it to a part. Even if we don't have the requirement, we could consider adding such functions as a convenience.
The second big question: How should we enable multiple partitions of a single mesh in iMeshP?
The third question: Should iMeshP_destroyPart should return an error if the part to be destroyed is not empty? The answer to this question depends, to some extent, on our answer to the first big question above. Cons: Requires users to explicitly remove entities from parts before deleting the parts. This operation requires three function calls: iMeshP_getEntities, iMesh_rmvEntFromSet (overloaded with part handle), iMeshP_destroyPart. Pros: Prevents users from shooting themselves in the foot. Users would possibly call iMeshP_getEntities and iMesh_rmvEntFromSet anyway to do migration to new part.
-- ================================================================ "You will keep in perfect peace him whose mind is steadfast, because he trusts in you." Isaiah 26:3 Tim Tautges Argonne National Laboratory ([email protected]) (telecommuting from UW-Madison) phone: (608) 263-8485 1500 Engineering Dr. fax: (608) 263-4499 Madison, WI 53706
participants (7)
-
Devine, Karen D -
Jason Kraftcheck -
Mark Beall -
Mark Shephard -
seol＠scorec.rpi.edu -
Tim Tautges -
Xiaolin Li