Assignments for parallel interface, due before 2/5 phone conf
In the last phone conf, someone suggested that we put together a draft of the syntax for the capabilities on which we agreed. So I added syntax from Carl's, Onkar's and Tim's past emails to our Combined document. For readability, I turned the file into a C header file; the discussions from the Combined document are now comments in the C file. The file is attached. For now, I'd rather not discuss: - values of "prefix_". - my use of "//" for comments being non-standard for some C compilers. - my inconsistent use of "const" (unless I used it for a return argument -- oops!) - whether or not this file would actually compile. Here are your assignments: ALL: Please give your opinions of the questions below. Some are simple questions resulting from the fact that I'm not used to ITAPS code conventions. Mark M. and Lori: - Review all functions at a high level to identify gaps and duplications in capability. Suggest functions that should be added or removed. - Review all functions at a high level to determine whether our syntax is generally consistent with ITAPS' conventions and generally consistent within the document. You shouldn't catch every detail; just let us know whether someone who is comfortable with other ITAPS interfaces would find this interface comfortable as well. Mark M and Onkar: - Review all functions at a high level and indicate whether their use of communication will be definite, likely, or nonexistent. If they incur communication, will the communication be global (e.g., MPI_Allreduce) or point-to-point (e.g., MPI_Send/MPI_Recv)? We'll eventually modify the function names to indicate whether or not they incur communication. Onkar: - Review in detail the Entity Functionality section. This section has the entity copying capability, including the add/remove copies that you discussed in the last phone conf. Vitus: - Propose the communication functions needed to support prefix_sendEntArrToParts and prefix_receiveEntArrToParts. Carl: - Should functions prefix_getEntSets and prefix_getNumEntSets be added to Carl's sixteen functions? They appear to be analogous to prefix_getEntities and prefix_getNumOfType. - Review in detail the Part Functionality section. Tim: - Review in detail the Partition Functionality section. ------------------------ Questions for everyone: - When we specify a part handle, must we always also specify a partition handle? That is, should a part be uniquely identified by the pair (partition_handle, part_handle), just as entities are uniquely identified by (mesh_instance, entity_handle)? "No" is the most convenient answer to this question; if parts are uniquely identified by only the part handle, we can do the entity-set argument overloading that we discussed at bootcamp. We can also remove the partition_handle argument from many of the functions. I had previously argued for use of the tuple, but many things would be easier if we didn't need it. Can all the implementations generate part handles that are unique within an iMesh_Instance, rather than within a partition? - With iterators over parts or part boundaries, is it necessary to have separate interface functions for getNext, reset, and end, or will the standard iMesh functions for getNext, reset, and end suffice? That is, once an iterator is initiated, do we need to continue passing the part handle and, perhaps, neighbor-part id, to the iterator, or can the iterator "remember" these just as it remembers entity type and topology? If we can re-use the iMesh getNext, reset and end, we can remove six functions from Carl's sixteen functions, bringing the count back to his original estimate of ten. - Which functions should operate on single parts and/or entities, and which should have array versions (function_name and function_nameArr)? I included both wherever we had a comment "Need single-part and array-part versions" in the combined document, but I won't be offended if we remove some of the functions. - Since global IDs for entities are not used anywhere in the interface, I propose their generation and use should be a service on top of the interface rather than part of the interface. What say you?
Karen, One quick input on:
- "When we specify a part handle, must we always also specify a partition handle? That is, should a part be uniquely identified by the pair (partition_handle, part_handle), just as entities are uniquely identified by (mesh_instance, entity_handle)? "No" is the most convenient answer to this question; if parts are uniquely identified by only the part handle, we can do the entity-set argument overloading that we discussed at bootcamp. We can also remove the partition_handle argument from many of the functions. I had previously argued for use of the tuple, but many things would be easier if we didn't need it. Can all the implementations generate part handles that are unique within an iMesh_Instance, rather than within a partition?"
What follows does not directly indicates an answer to your question. However, today one of our research meetings where we are working on getting mesh adapt to 23K processors on the Blue Gene, we found that having a global numbering for the equivalent of part boundary entities might be a problem for scaling on Blue Gene like machines. Without trying to explaining the specific problem we came across (it would take a while) because we currently have global part ID's, the one potential solution to avoid needing to wasting memory proportional to at least the number of processors (an issue when we will have 1M or more of them) was to assign an owner to each part interface in which case the order of memory usage reduces from the total number of parts boundaries on each processor something like the number of part boundaries times the number of parts sharing those boundaries. We find that for properly partitioned meshes, both of these numbers tend to go no higher that a fined number independent of the number of processors. Thus, I am starting to worry about anything that has memory requirements proportional to the number processors in the system being required. Mark Devine, Karen D wrote:
In the last phone conf, someone suggested that we put together a draft of the syntax for the capabilities on which we agreed. So I added syntax from Carl's, Onkar's and Tim's past emails to our Combined document. For readability, I turned the file into a C header file; the discussions from the Combined document are now comments in the C file. The file is attached.
For now, I'd rather not discuss: - values of "prefix_". - my use of "//" for comments being non-standard for some C compilers. - my inconsistent use of "const" (unless I used it for a return argument -- oops!) - whether or not this file would actually compile.
Here are your assignments:
ALL: Please give your opinions of the questions below. Some are simple questions resulting from the fact that I'm not used to ITAPS code conventions.
Mark M. and Lori: - Review all functions at a high level to identify gaps and duplications in capability. Suggest functions that should be added or removed. - Review all functions at a high level to determine whether our syntax is generally consistent with ITAPS' conventions and generally consistent within the document. You shouldn't catch every detail; just let us know whether someone who is comfortable with other ITAPS interfaces would find this interface comfortable as well.
Mark M and Onkar: - Review all functions at a high level and indicate whether their use of communication will be definite, likely, or nonexistent. If they incur communication, will the communication be global (e.g., MPI_Allreduce) or point-to-point (e.g., MPI_Send/MPI_Recv)? We'll eventually modify the function names to indicate whether or not they incur communication.
Onkar: - Review in detail the Entity Functionality section. This section has the entity copying capability, including the add/remove copies that you discussed in the last phone conf.
Vitus: - Propose the communication functions needed to support prefix_sendEntArrToParts and prefix_receiveEntArrToParts.
Carl: - Should functions prefix_getEntSets and prefix_getNumEntSets be added to Carl's sixteen functions�? They appear to be analogous to prefix_getEntities and prefix_getNumOfType. - Review in detail the Part Functionality section.
Tim: - Review in detail the Partition Functionality section.
------------------------ Questions for everyone:
- When we specify a part handle, must we always also specify a partition handle? That is, should a part be uniquely identified by the pair (partition_handle, part_handle), just as entities are uniquely identified by (mesh_instance, entity_handle)? "No" is the most convenient answer to this question; if parts are uniquely identified by only the part handle, we can do the entity-set argument overloading that we discussed at bootcamp. We can also remove the partition_handle argument from many of the functions. I had previously argued for use of the tuple, but many things would be easier if we didn't need it. Can all the implementations generate part handles that are unique within an iMesh_Instance, rather than within a partition?
- With iterators over parts or part boundaries, is it necessary to have separate interface functions for getNext, reset, and end, or will the standard iMesh functions for getNext, reset, and end suffice? That is, once an iterator is initiated, do we need to continue passing the part handle and, perhaps, neighbor-part id, to the iterator, or can the iterator "remember" these just as it remembers entity type and topology? If we can re-use the iMesh getNext, reset and end, we can remove six functions from Carl's sixteen functions�, bringing the count back to his original estimate of ten.
- Which functions should operate on single parts and/or entities, and which should have array versions (function_name and function_nameArr)? I included both wherever we had a comment "Need single-part and array-part versions" in the combined document, but I won't be offended if we remove some of the functions.
- Since global IDs for entities are not used anywhere in the interface, I propose their generation and use should be a service on top of the interface rather than part of the interface. What say you?
Devine, Karen D wrote:
In the last phone conf, someone suggested that we put together a draft of the syntax for the capabilities on which we agreed. So I added syntax from Carl's, Onkar's and Tim's past emails to our Combined document. For readability, I turned the file into a C header file; the discussions from the Combined document are now comments in the C file. The file is attached.
I actually got this nearly done a week ago, but didn't get it sent until now...
Carl: - Should functions prefix_getEntSets and prefix_getNumEntSets be added to Carl's sixteen functions�? They appear to be analogous to prefix_getEntities and prefix_getNumOfType.
This is probably a good idea, for those who both (a) choose to program on a part-by-part basis rather than process-by-process and (b) choose to use sets for various things. I've got no objection.
- Review in detail the Part Functionality section.
createPart: How do we envision the adjudication of global part ID's? Does part creation imply a request to some oracle for a part ID? Does each process have its own unique space for part ID's? Using n bits for process rank, m bits for local ID would create part ID's that were easy to create and interpret... with cleverness, we could work things so that n+m <= 31 (a billion part ID's...) so that this all looks like an int, and we don't have to mess with the sign bit. destroyPart: Setting handles to NULL on destruction assumes implicitly that no implementation will ever use 0 as a valid part handle. While it might be a good idea to invalidate the handle, the invalid value may have to be implementation dependent. getPartIdsFromPartHandle: The part handle presumably must be local here; this should be added to the comment. getNumPartNbors/getPartNbors: communicating or pre-computed? If the latter, how/when are these updated? And again, how are we defining which parts are neighbors? Part and part bdry iterators: I don't have (at least at this point) a firm opinion about whether we should have separate functions to increment, reset, and destroy these iterators. It's pretty easy to argue that, semantically, the iterator should know how to do those things for itself, and that it isn't too hard for an implementation to handle that internal state. -------------------------- One entity-related comment: getEntityOwner: I disagree with Karen. I think it -does- make sense to be able to determine the owner of an arbitrary entity. We have agreed that exactly one part will have ownership of an entity in the right-to-modify sense, and IMO this is what this call should return. We may also need functionality to identify the entities that were used in -determining- the partition, but I see that as a different issue.
------------------------ Questions for everyone:
- When we specify a part handle, must we always also specify a partition handle? That is, should a part be uniquely identified by the pair (partition_handle, part_handle), just as entities are uniquely identified by (mesh_instance, entity_handle)? "No" is the most convenient answer to this question; if parts are uniquely identified by only the part handle, we can do the entity-set argument overloading that we discussed at bootcamp. We can also remove the partition_handle argument from many of the functions. I had previously argued for use of the tuple, but many things would be easier if we didn't need it. Can all the implementations generate part handles that are unique within an iMesh_Instance, rather than within a partition?
My concern here is that this is a potentially difficult constraint for implementations that want to use integer part handles. Somehow, they have to coordinate all partitions active in an instance to produce unique integer ID's. Also, part handles will have to be unique from set handles as well. (Again, not a problem with pointers, but potentially a problem for integer handles.) I'm not saying that I expect to implement handles as ints, but it's not an unreasonable choice, and we want to be careful not to exclude it without good reason.
- With iterators over parts or part boundaries, is it necessary to have separate interface functions for getNext, reset, and end, or will the standard iMesh functions for getNext, reset, and end suffice? That is, once an iterator is initiated, do we need to continue passing the part handle and, perhaps, neighbor-part id, to the iterator, or can the iterator "remember" these just as it remembers entity type and topology? If we can re-use the iMesh getNext, reset and end, we can remove six functions from Carl's sixteen functions�, bringing the count back to his original estimate of ten.
See above.
- Since global IDs for entities are not used anywhere in the interface, I propose their generation and use should be a service on top of the interface rather than part of the interface. What say you?
No objection. Carl -- ------------------------------------------------------------------------ Dr. Carl Ollivier-Gooch, P.Eng. Voice: +1-604-822-1854 Associate Professor Fax: +1-604-822-2403 Department of Mechanical Engineering email: [email protected] University of British Columbia http://www.mech.ubc.ca/~cfog Vancouver, BC V6T 1Z4 http://tetra.mech.ubc.ca/ANSLab/ ------------------------------------------------------------------------
Carl wrote:
createPart: How do we envision the adjudication of global part ID's? Does part creation imply a request to some oracle for a part ID? Does each process have its own unique space for part ID's? Using n bits for process rank, m bits for local ID would create part ID's that were easy to create and interpret... with cleverness, we could work things so that n+m <= 31 (a billion part ID's...) so that this all looks like an int, and we don't have to mess with the sign bit.
I agree that we need to reserve constant part IDs on each process/rank ('m' bits). I think we may also want to allow for more than 31 bits for part IDs and thus, use iPartID_type (typedef to appropriate type like 32-bit-int and it may be decided by application during compile time). In the same context, we may want to revisit routines like prefix_getNumPartIds. Do we really need them for each rank?
getPartIdsFromPartHandle: The part handle presumably must be local here; this should be added to the comment.
I think part-handles are always local on-process ones.
getNumPartNbors/getPartNbors: communicating or pre-computed? If the latter, how/when are these updated? And again, how are we defining which parts are neighbors?
I think the reason to support this is to have this pre-commuted. These are maintained and updated during mesh migration and parallel sub-division (send/recieveEntArr and add/rmvCopy).
getEntityOwner: I disagree with Karen. I think it -does- make sense to be able to determine the owner of an arbitrary entity. We have agreed that exactly one part will have ownership of an entity in the right-to-modify sense, and IMO this is what this call should return. We may also need functionality to identify the entities that were used in -determining- the partition, but I see that as a different issue.
I thought this is for any arbitrary entity. Actually I may have added to confusion by referring 'entity functionality' as 'functionality for inter-part boundary entities' but I agree that it allows for non-inter-part boundary entities too like prefix_getEntityParStatus for internal mesh faces or regions. - Onkar
I have attached part of my assignment which discusses communication requirement of functions at high level. Other part of assignment on entity functionality will follow (hopefully before end of today). I will also try to send general overall comments and concerns for ITAPS parallel interface.
------------------------ Questions for everyone:
- When we specify a part handle, must we always also specify a partition handle? That is, should a part be uniquely identified by the pair (partition_handle, part_handle), just as entities are uniquely identified by (mesh_instance, entity_handle)? "No" is the most convenient answer to this question; if parts are uniquely identified by only the part handle, we can do the entity-set argument overloading that we discussed at bootcamp. We can also remove the partition_handle argument from many of the functions. I had previously argued for use of the tuple, but many things would be easier if we didn't need it. Can all the implementations generate part handles that are unique within an iMesh_Instance, rather than within a partition?
As of now, we (RPI) are considering a parent/root/primary partition for any single mesh-instance (i.e., parent-partition<->mesh-instance is one-to-one), and hence, part IDs are unique in an mesh instance (and parent-partition).
- With iterators over parts or part boundaries, is it necessary to have separate interface functions for getNext, reset, and end, or will the standard iMesh functions for getNext, reset, and end suffice? That is, once an iterator is initiated, do we need to continue passing the part handle and, perhaps, neighbor-part id, to the iterator, or can the iterator "remember" these just as it remembers entity type and topology? If we can re-use the iMesh getNext, reset and end, we can remove six functions from Carl's sixteen functions�, bringing the count back to his original estimate of ten.
I think it sounds fine and we can support this.
- Which functions should operate on single parts and/or entities, and which should have array versions (function_name and function_nameArr)? I included both wherever we had a comment "Need single-part and array-part versions" in the combined document, but I won't be offended if we remove some of the functions.
Not sure as of now.
- Since global IDs for entities are not used anywhere in the interface, I propose their generation and use should be a service on top of the interface rather than part of the interface. What say you?
Sounds fine. Thanks, Onkar
In the last phone conf, someone suggested that we put together a draft of the syntax for the capabilities on which we agreed. So I added syntax from Carl's, Onkar's and Tim's past emails to our Combined document. For readability, I turned the file into a C header file; the discussions from the Combined document are now comments in the C file. The file is attached.
For now, I'd rather not discuss: - values of "prefix_". - my use of "//" for comments being non-standard for some C compilers. - my inconsistent use of "const" (unless I used it for a return argument -- oops!) - whether or not this file would actually compile.
Here are your assignments:
ALL: Please give your opinions of the questions below. Some are simple questions resulting from the fact that I'm not used to ITAPS code conventions.
Mark M. and Lori: - Review all functions at a high level to identify gaps and duplications in capability. Suggest functions that should be added or removed. - Review all functions at a high level to determine whether our syntax is generally consistent with ITAPS' conventions and generally consistent within the document. You shouldn't catch every detail; just let us know whether someone who is comfortable with other ITAPS interfaces would find this interface comfortable as well.
Mark M and Onkar: - Review all functions at a high level and indicate whether their use of communication will be definite, likely, or nonexistent. If they incur communication, will the communication be global (e.g., MPI_Allreduce) or point-to-point (e.g., MPI_Send/MPI_Recv)? We'll eventually modify the function names to indicate whether or not they incur communication.
Onkar: - Review in detail the Entity Functionality section. This section has the entity copying capability, including the add/remove copies that you discussed in the last phone conf.
Vitus: - Propose the communication functions needed to support prefix_sendEntArrToParts and prefix_receiveEntArrToParts.
Carl: - Should functions prefix_getEntSets and prefix_getNumEntSets be added to Carl's sixteen functions�? They appear to be analogous to prefix_getEntities and prefix_getNumOfType. - Review in detail the Part Functionality section.
Tim: - Review in detail the Partition Functionality section.
------------------------ Questions for everyone:
- When we specify a part handle, must we always also specify a partition handle? That is, should a part be uniquely identified by the pair (partition_handle, part_handle), just as entities are uniquely identified by (mesh_instance, entity_handle)? "No" is the most convenient answer to this question; if parts are uniquely identified by only the part handle, we can do the entity-set argument overloading that we discussed at bootcamp. We can also remove the partition_handle argument from many of the functions. I had previously argued for use of the tuple, but many things would be easier if we didn't need it. Can all the implementations generate part handles that are unique within an iMesh_Instance, rather than within a partition?
- With iterators over parts or part boundaries, is it necessary to have separate interface functions for getNext, reset, and end, or will the standard iMesh functions for getNext, reset, and end suffice? That is, once an iterator is initiated, do we need to continue passing the part handle and, perhaps, neighbor-part id, to the iterator, or can the iterator "remember" these just as it remembers entity type and topology? If we can re-use the iMesh getNext, reset and end, we can remove six functions from Carl's sixteen functions�, bringing the count back to his original estimate of ten.
- Which functions should operate on single parts and/or entities, and which should have array versions (function_name and function_nameArr)? I included both wherever we had a comment "Need single-part and array-part versions" in the combined document, but I won't be offended if we remove some of the functions.
- Since global IDs for entities are not used anywhere in the interface, I propose their generation and use should be a service on top of the interface rather than part of the interface. What say you?
I have attached other part of assignment on entity functionality. As I mentioned earlier, a function to get residence parts of an entity on inter-part boundary will be helpful. See example in the end of attachment. Thanks, Onkar
I have attached part of my assignment which discusses communication requirement of functions at high level.
Other part of assignment on entity functionality will follow (hopefully before end of today).
I will also try to send general overall comments and concerns for ITAPS parallel interface.
------------------------ Questions for everyone:
- When we specify a part handle, must we always also specify a partition handle? That is, should a part be uniquely identified by the pair (partition_handle, part_handle), just as entities are uniquely identified by (mesh_instance, entity_handle)? "No" is the most convenient answer to this question; if parts are uniquely identified by only the part handle, we can do the entity-set argument overloading that we discussed at bootcamp. We can also remove the partition_handle argument from many of the functions. I had previously argued for use of the tuple, but many things would be easier if we didn't need it. Can all the implementations generate part handles that are unique within an iMesh_Instance, rather than within a partition?
As of now, we (RPI) are considering a parent/root/primary partition for any single mesh-instance (i.e., parent-partition<->mesh-instance is one-to-one), and hence, part IDs are unique in an mesh instance (and parent-partition).
- With iterators over parts or part boundaries, is it necessary to have separate interface functions for getNext, reset, and end, or will the standard iMesh functions for getNext, reset, and end suffice? That is, once an iterator is initiated, do we need to continue passing the part handle and, perhaps, neighbor-part id, to the iterator, or can the iterator "remember" these just as it remembers entity type and topology? If we can re-use the iMesh getNext, reset and end, we can remove six functions from Carl's sixteen functions�, bringing the count back to his original estimate of ten.
I think it sounds fine and we can support this.
- Which functions should operate on single parts and/or entities, and which should have array versions (function_name and function_nameArr)? I included both wherever we had a comment "Need single-part and array-part versions" in the combined document, but I won't be offended if we remove some of the functions.
Not sure as of now.
- Since global IDs for entities are not used anywhere in the interface, I propose their generation and use should be a service on top of the interface rather than part of the interface. What say you?
Sounds fine.
Thanks, Onkar
In the last phone conf, someone suggested that we put together a draft of the syntax for the capabilities on which we agreed. So I added syntax from Carl's, Onkar's and Tim's past emails to our Combined document. For readability, I turned the file into a C header file; the discussions from the Combined document are now comments in the C file. The file is attached.
For now, I'd rather not discuss: - values of "prefix_". - my use of "//" for comments being non-standard for some C compilers. - my inconsistent use of "const" (unless I used it for a return argument -- oops!) - whether or not this file would actually compile.
Here are your assignments:
ALL: Please give your opinions of the questions below. Some are simple questions resulting from the fact that I'm not used to ITAPS code conventions.
Mark M. and Lori: - Review all functions at a high level to identify gaps and duplications in capability. Suggest functions that should be added or removed. - Review all functions at a high level to determine whether our syntax is generally consistent with ITAPS' conventions and generally consistent within the document. You shouldn't catch every detail; just let us know whether someone who is comfortable with other ITAPS interfaces would find this interface comfortable as well.
Mark M and Onkar: - Review all functions at a high level and indicate whether their use of communication will be definite, likely, or nonexistent. If they incur communication, will the communication be global (e.g., MPI_Allreduce) or point-to-point (e.g., MPI_Send/MPI_Recv)? We'll eventually modify the function names to indicate whether or not they incur communication.
Onkar: - Review in detail the Entity Functionality section. This section has the entity copying capability, including the add/remove copies that you discussed in the last phone conf.
Vitus: - Propose the communication functions needed to support prefix_sendEntArrToParts and prefix_receiveEntArrToParts.
Carl: - Should functions prefix_getEntSets and prefix_getNumEntSets be added to Carl's sixteen functions�? They appear to be analogous to prefix_getEntities and prefix_getNumOfType. - Review in detail the Part Functionality section.
Tim: - Review in detail the Partition Functionality section.
------------------------ Questions for everyone:
- When we specify a part handle, must we always also specify a partition handle? That is, should a part be uniquely identified by the pair (partition_handle, part_handle), just as entities are uniquely identified by (mesh_instance, entity_handle)? "No" is the most convenient answer to this question; if parts are uniquely identified by only the part handle, we can do the entity-set argument overloading that we discussed at bootcamp. We can also remove the partition_handle argument from many of the functions. I had previously argued for use of the tuple, but many things would be easier if we didn't need it. Can all the implementations generate part handles that are unique within an iMesh_Instance, rather than within a partition?
- With iterators over parts or part boundaries, is it necessary to have separate interface functions for getNext, reset, and end, or will the standard iMesh functions for getNext, reset, and end suffice? That is, once an iterator is initiated, do we need to continue passing the part handle and, perhaps, neighbor-part id, to the iterator, or can the iterator "remember" these just as it remembers entity type and topology? If we can re-use the iMesh getNext, reset and end, we can remove six functions from Carl's sixteen functions�, bringing the count back to his original estimate of ten.
- Which functions should operate on single parts and/or entities, and which should have array versions (function_name and function_nameArr)? I included both wherever we had a comment "Need single-part and array-part versions" in the combined document, but I won't be offended if we remove some of the functions.
- Since global IDs for entities are not used anywhere in the interface, I propose their generation and use should be a service on top of the interface rather than part of the interface. What say you?
Onkar: One question from our EntityLevelComments.txt:
1) prefix_addCopyEnt will require remote_entity_handle along with remote_part_id (entity_handle is local on-process handle). prefix_removeCopyEnt may have either remote_part_id or remote_entity_handle (or both).
How would a processor provide a remote entity handle for an entity it is copying into a remote part? Say entity handles are addresses. Then before the copy is added, how can a processor know its remote address? Karen On 2/4/08 1:58 PM, "Onkar Sahni" <[email protected]> wrote:
I have attached other part of assignment on entity functionality.
As I mentioned earlier, a function to get residence parts of an entity on inter-part boundary will be helpful. See example in the end of attachment.
Thanks, Onkar
I have attached part of my assignment which discusses communication requirement of functions at high level.
Other part of assignment on entity functionality will follow (hopefully before end of today).
I will also try to send general overall comments and concerns for ITAPS parallel interface.
------------------------ Questions for everyone:
- When we specify a part handle, must we always also specify a partition handle? That is, should a part be uniquely identified by the pair (partition_handle, part_handle), just as entities are uniquely identified by (mesh_instance, entity_handle)? "No" is the most convenient answer to this question; if parts are uniquely identified by only the part handle, we can do the entity-set argument overloading that we discussed at bootcamp. We can also remove the partition_handle argument from many of the functions. I had previously argued for use of the tuple, but many things would be easier if we didn't need it. Can all the implementations generate part handles that are unique within an iMesh_Instance, rather than within a partition?
As of now, we (RPI) are considering a parent/root/primary partition for any single mesh-instance (i.e., parent-partition<->mesh-instance is one-to-one), and hence, part IDs are unique in an mesh instance (and parent-partition).
- With iterators over parts or part boundaries, is it necessary to have separate interface functions for getNext, reset, and end, or will the standard iMesh functions for getNext, reset, and end suffice? That is, once an iterator is initiated, do we need to continue passing the part handle and, perhaps, neighbor-part id, to the iterator, or can the iterator "remember" these just as it remembers entity type and topology? If we can re-use the iMesh getNext, reset and end, we can remove six functions from Carl's sixteen functions�, bringing the count back to his original estimate of ten.
I think it sounds fine and we can support this.
- Which functions should operate on single parts and/or entities, and which should have array versions (function_name and function_nameArr)? I included both wherever we had a comment "Need single-part and array-part versions" in the combined document, but I won't be offended if we remove some of the functions.
Not sure as of now.
- Since global IDs for entities are not used anywhere in the interface, I propose their generation and use should be a service on top of the interface rather than part of the interface. What say you?
Sounds fine.
Thanks, Onkar
In the last phone conf, someone suggested that we put together a draft of the syntax for the capabilities on which we agreed. So I added syntax from Carl's, Onkar's and Tim's past emails to our Combined document. For readability, I turned the file into a C header file; the discussions from the Combined document are now comments in the C file. The file is attached.
For now, I'd rather not discuss: - values of "prefix_". - my use of "//" for comments being non-standard for some C compilers. - my inconsistent use of "const" (unless I used it for a return argument -- oops!) - whether or not this file would actually compile.
Here are your assignments:
ALL: Please give your opinions of the questions below. Some are simple questions resulting from the fact that I'm not used to ITAPS code conventions.
Mark M. and Lori: - Review all functions at a high level to identify gaps and duplications in capability. Suggest functions that should be added or removed. - Review all functions at a high level to determine whether our syntax is generally consistent with ITAPS' conventions and generally consistent within the document. You shouldn't catch every detail; just let us know whether someone who is comfortable with other ITAPS interfaces would find this interface comfortable as well.
Mark M and Onkar: - Review all functions at a high level and indicate whether their use of communication will be definite, likely, or nonexistent. If they incur communication, will the communication be global (e.g., MPI_Allreduce) or point-to-point (e.g., MPI_Send/MPI_Recv)? We'll eventually modify the function names to indicate whether or not they incur communication.
Onkar: - Review in detail the Entity Functionality section. This section has the entity copying capability, including the add/remove copies that you discussed in the last phone conf.
Vitus: - Propose the communication functions needed to support prefix_sendEntArrToParts and prefix_receiveEntArrToParts.
Carl: - Should functions prefix_getEntSets and prefix_getNumEntSets be added to Carl's sixteen functions�? They appear to be analogous to prefix_getEntities and prefix_getNumOfType. - Review in detail the Part Functionality section.
Tim: - Review in detail the Partition Functionality section.
------------------------ Questions for everyone:
- When we specify a part handle, must we always also specify a partition handle? That is, should a part be uniquely identified by the pair (partition_handle, part_handle), just as entities are uniquely identified by (mesh_instance, entity_handle)? "No" is the most convenient answer to this question; if parts are uniquely identified by only the part handle, we can do the entity-set argument overloading that we discussed at bootcamp. We can also remove the partition_handle argument from many of the functions. I had previously argued for use of the tuple, but many things would be easier if we didn't need it. Can all the implementations generate part handles that are unique within an iMesh_Instance, rather than within a partition?
- With iterators over parts or part boundaries, is it necessary to have separate interface functions for getNext, reset, and end, or will the standard iMesh functions for getNext, reset, and end suffice? That is, once an iterator is initiated, do we need to continue passing the part handle and, perhaps, neighbor-part id, to the iterator, or can the iterator "remember" these just as it remembers entity type and topology? If we can re-use the iMesh getNext, reset and end, we can remove six functions from Carl's sixteen functions�, bringing the count back to his original estimate of ten.
- Which functions should operate on single parts and/or entities, and which should have array versions (function_name and function_nameArr)? I included both wherever we had a comment "Need single-part and array-part versions" in the combined document, but I won't be offended if we remove some of the functions.
- Since global IDs for entities are not used anywhere in the interface, I propose their generation and use should be a service on top of the interface rather than part of the interface. What say you?
Onkar:
One question from our EntityLevelComments.txt:
1) prefix_addCopyEnt will require remote_entity_handle along with remote_part_id (entity_handle is local on-process handle). prefix_removeCopyEnt may have either remote_part_id or remote_entity_handle (or both).
How would a processor provide a remote entity handle for an entity it is copying into a remote part? Say entity handles are addresses. Then before the copy is added, how can a processor know its remote address?
I don't know what you mean by "copying into a remote part". Let me clarify in general, prefix_addCopyEnt is on-process call, where entity_handle is on-process handle and entity exists on more than one part. See Fig. 2 under Example 2 in attached examples document (same as what I sent before), M^1_0 and M^1_1 are edges on P0 and P1, respectively (where M^i_j follows latex syntax and denotes mesh entity with dimension i (<=3) and local-on-part-entity-index j) and goes under parallel split/subdivision. After split, M^1_3 and M^1_4 are created on P0 and M^1_5 and M^1_6 are created on P1. M^1_3 is image of M^1_5 on P0 and vice-versa (and same for M^1_4 and M^1_6), so to link this information in terms of remote copies we need prefix_addCopyEnt(). For example, for prefix_addCopyEnt on P0, entity_handle=M^1_3 and remote_entity_handle=M^1_5 and remote_part_id=P1 (of course M^1_5 and P1 are communicated to P0 before prefix_addCopyEnt). The arrows in Fig. 2 denotes messages. How does one know M^1_3 and M^1_5 needs to be linked is through remote copy information for M^1_0 and M^1_1, parent edges (so M^1_0 on P0 knows M^1_1, P1 as remote copy information and M^1_1 on P1 knows M^1_0, P0 as remote copy information). I hope I do not have typos. Note that remote copy information is always maintained as migration and/or parallel subdivision takes place (starting with serial mesh, on one part, remote copy information evolves as mesh is migrated/partitioned and while starting with parallel mesh it is generated in loading of mesh). Let me know if this is not clear. - Onkar
Karen
On 2/4/08 1:58 PM, "Onkar Sahni" <[email protected]> wrote:
I have attached other part of assignment on entity functionality.
As I mentioned earlier, a function to get residence parts of an entity on inter-part boundary will be helpful. See example in the end of attachment.
Thanks, Onkar
I have attached part of my assignment which discusses communication requirement of functions at high level.
Other part of assignment on entity functionality will follow (hopefully before end of today).
I will also try to send general overall comments and concerns for ITAPS parallel interface.
------------------------ Questions for everyone:
- When we specify a part handle, must we always also specify a partition handle? That is, should a part be uniquely identified by the pair (partition_handle, part_handle), just as entities are uniquely identified by (mesh_instance, entity_handle)? "No" is the most convenient answer to this question; if parts are uniquely identified by only the part handle, we can do the entity-set argument overloading that we discussed at bootcamp. We can also remove the partition_handle argument from many of the functions. I had previously argued for use of the tuple, but many things would be easier if we didn't need it. Can all the implementations generate part handles that are unique within an iMesh_Instance, rather than within a partition?
As of now, we (RPI) are considering a parent/root/primary partition for any single mesh-instance (i.e., parent-partition<->mesh-instance is one-to-one), and hence, part IDs are unique in an mesh instance (and parent-partition).
- With iterators over parts or part boundaries, is it necessary to have separate interface functions for getNext, reset, and end, or will the standard iMesh functions for getNext, reset, and end suffice? That is, once an iterator is initiated, do we need to continue passing the part handle and, perhaps, neighbor-part id, to the iterator, or can the iterator "remember" these just as it remembers entity type and topology? If we can re-use the iMesh getNext, reset and end, we can remove six functions from Carl's sixteen functions�, bringing the count back to his original estimate of ten.
I think it sounds fine and we can support this.
- Which functions should operate on single parts and/or entities, and which should have array versions (function_name and function_nameArr)? I included both wherever we had a comment "Need single-part and array-part versions" in the combined document, but I won't be offended if we remove some of the functions.
Not sure as of now.
- Since global IDs for entities are not used anywhere in the interface, I propose their generation and use should be a service on top of the interface rather than part of the interface. What say you?
Sounds fine.
Thanks, Onkar
In the last phone conf, someone suggested that we put together a draft of the syntax for the capabilities on which we agreed. So I added syntax from Carl's, Onkar's and Tim's past emails to our Combined document. For readability, I turned the file into a C header file; the discussions from the Combined document are now comments in the C file. The file is attached.
For now, I'd rather not discuss: - values of "prefix_". - my use of "//" for comments being non-standard for some C compilers. - my inconsistent use of "const" (unless I used it for a return argument -- oops!) - whether or not this file would actually compile.
Here are your assignments:
ALL: Please give your opinions of the questions below. Some are simple questions resulting from the fact that I'm not used to ITAPS code conventions.
Mark M. and Lori: - Review all functions at a high level to identify gaps and duplications in capability. Suggest functions that should be added or removed. - Review all functions at a high level to determine whether our syntax is generally consistent with ITAPS' conventions and generally consistent within the document. You shouldn't catch every detail; just let us know whether someone who is comfortable with other ITAPS interfaces would find this interface comfortable as well.
Mark M and Onkar: - Review all functions at a high level and indicate whether their use of communication will be definite, likely, or nonexistent. If they incur communication, will the communication be global (e.g., MPI_Allreduce) or point-to-point (e.g., MPI_Send/MPI_Recv)? We'll eventually modify the function names to indicate whether or not they incur communication.
Onkar: - Review in detail the Entity Functionality section. This section has the entity copying capability, including the add/remove copies that you discussed in the last phone conf.
Vitus: - Propose the communication functions needed to support prefix_sendEntArrToParts and prefix_receiveEntArrToParts.
Carl: - Should functions prefix_getEntSets and prefix_getNumEntSets be added to Carl's sixteen functions�? They appear to be analogous to prefix_getEntities and prefix_getNumOfType. - Review in detail the Part Functionality section.
Tim: - Review in detail the Partition Functionality section.
------------------------ Questions for everyone:
- When we specify a part handle, must we always also specify a partition handle? That is, should a part be uniquely identified by the pair (partition_handle, part_handle), just as entities are uniquely identified by (mesh_instance, entity_handle)? "No" is the most convenient answer to this question; if parts are uniquely identified by only the part handle, we can do the entity-set argument overloading that we discussed at bootcamp. We can also remove the partition_handle argument from many of the functions. I had previously argued for use of the tuple, but many things would be easier if we didn't need it. Can all the implementations generate part handles that are unique within an iMesh_Instance, rather than within a partition?
- With iterators over parts or part boundaries, is it necessary to have separate interface functions for getNext, reset, and end, or will the standard iMesh functions for getNext, reset, and end suffice? That is, once an iterator is initiated, do we need to continue passing the part handle and, perhaps, neighbor-part id, to the iterator, or can the iterator "remember" these just as it remembers entity type and topology? If we can re-use the iMesh getNext, reset and end, we can remove six functions from Carl's sixteen functions�, bringing the count back to his original estimate of ten.
- Which functions should operate on single parts and/or entities, and which should have array versions (function_name and function_nameArr)? I included both wherever we had a comment "Need single-part and array-part versions" in the combined document, but I won't be offended if we remove some of the functions.
- Since global IDs for entities are not used anywhere in the interface, I propose their generation and use should be a service on top of the interface rather than part of the interface. What say you?
Sorry for many messages. I have now attached three things: 1) Our assignment on overall communication requirement (one change from before prefix_getNumPartsPar - not sure.), see CommInterface.h. 2) Our assignment on entity level functionality, see EntityLevelComments.txt. 3) Overall comments and concerns (overlaps my and/or other peoples previous inputs), see OverallComments.txt. Thanks, Onkar
I have attached other part of assignment on entity functionality.
As I mentioned earlier, a function to get residence parts of an entity on inter-part boundary will be helpful. See example in the end of attachment.
Thanks, Onkar
I have attached part of my assignment which discusses communication requirement of functions at high level.
Other part of assignment on entity functionality will follow (hopefully before end of today).
I will also try to send general overall comments and concerns for ITAPS parallel interface.
------------------------ Questions for everyone:
- When we specify a part handle, must we always also specify a partition handle? That is, should a part be uniquely identified by the pair (partition_handle, part_handle), just as entities are uniquely identified by (mesh_instance, entity_handle)? "No" is the most convenient answer to this question; if parts are uniquely identified by only the part handle, we can do the entity-set argument overloading that we discussed at bootcamp. We can also remove the partition_handle argument from many of the functions. I had previously argued for use of the tuple, but many things would be easier if we didn't need it. Can all the implementations generate part handles that are unique within an iMesh_Instance, rather than within a partition?
As of now, we (RPI) are considering a parent/root/primary partition for any single mesh-instance (i.e., parent-partition<->mesh-instance is one-to-one), and hence, part IDs are unique in an mesh instance (and parent-partition).
- With iterators over parts or part boundaries, is it necessary to have separate interface functions for getNext, reset, and end, or will the standard iMesh functions for getNext, reset, and end suffice? That is, once an iterator is initiated, do we need to continue passing the part handle and, perhaps, neighbor-part id, to the iterator, or can the iterator "remember" these just as it remembers entity type and topology? If we can re-use the iMesh getNext, reset and end, we can remove six functions from Carl's sixteen functions�, bringing the count back to his original estimate of ten.
I think it sounds fine and we can support this.
- Which functions should operate on single parts and/or entities, and which should have array versions (function_name and function_nameArr)? I included both wherever we had a comment "Need single-part and array-part versions" in the combined document, but I won't be offended if we remove some of the functions.
Not sure as of now.
- Since global IDs for entities are not used anywhere in the interface, I propose their generation and use should be a service on top of the interface rather than part of the interface. What say you?
Sounds fine.
Thanks, Onkar
In the last phone conf, someone suggested that we put together a draft of the syntax for the capabilities on which we agreed. So I added syntax from Carl's, Onkar's and Tim's past emails to our Combined document. For readability, I turned the file into a C header file; the discussions from the Combined document are now comments in the C file. The file is attached.
For now, I'd rather not discuss: - values of "prefix_". - my use of "//" for comments being non-standard for some C compilers. - my inconsistent use of "const" (unless I used it for a return argument -- oops!) - whether or not this file would actually compile.
Here are your assignments:
ALL: Please give your opinions of the questions below. Some are simple questions resulting from the fact that I'm not used to ITAPS code conventions.
Mark M. and Lori: - Review all functions at a high level to identify gaps and duplications in capability. Suggest functions that should be added or removed. - Review all functions at a high level to determine whether our syntax is generally consistent with ITAPS' conventions and generally consistent within the document. You shouldn't catch every detail; just let us know whether someone who is comfortable with other ITAPS interfaces would find this interface comfortable as well.
Mark M and Onkar: - Review all functions at a high level and indicate whether their use of communication will be definite, likely, or nonexistent. If they incur communication, will the communication be global (e.g., MPI_Allreduce) or point-to-point (e.g., MPI_Send/MPI_Recv)? We'll eventually modify the function names to indicate whether or not they incur communication.
Onkar: - Review in detail the Entity Functionality section. This section has the entity copying capability, including the add/remove copies that you discussed in the last phone conf.
Vitus: - Propose the communication functions needed to support prefix_sendEntArrToParts and prefix_receiveEntArrToParts.
Carl: - Should functions prefix_getEntSets and prefix_getNumEntSets be added to Carl's sixteen functions�? They appear to be analogous to prefix_getEntities and prefix_getNumOfType. - Review in detail the Part Functionality section.
Tim: - Review in detail the Partition Functionality section.
------------------------ Questions for everyone:
- When we specify a part handle, must we always also specify a partition handle? That is, should a part be uniquely identified by the pair (partition_handle, part_handle), just as entities are uniquely identified by (mesh_instance, entity_handle)? "No" is the most convenient answer to this question; if parts are uniquely identified by only the part handle, we can do the entity-set argument overloading that we discussed at bootcamp. We can also remove the partition_handle argument from many of the functions. I had previously argued for use of the tuple, but many things would be easier if we didn't need it. Can all the implementations generate part handles that are unique within an iMesh_Instance, rather than within a partition?
- With iterators over parts or part boundaries, is it necessary to have separate interface functions for getNext, reset, and end, or will the standard iMesh functions for getNext, reset, and end suffice? That is, once an iterator is initiated, do we need to continue passing the part handle and, perhaps, neighbor-part id, to the iterator, or can the iterator "remember" these just as it remembers entity type and topology? If we can re-use the iMesh getNext, reset and end, we can remove six functions from Carl's sixteen functions�, bringing the count back to his original estimate of ten.
- Which functions should operate on single parts and/or entities, and which should have array versions (function_name and function_nameArr)? I included both wherever we had a comment "Need single-part and array-part versions" in the combined document, but I won't be offended if we remove some of the functions.
- Since global IDs for entities are not used anywhere in the interface, I propose their generation and use should be a service on top of the interface rather than part of the interface. What say you?
Here are my comments, some of them covered by others already. - tim -- ================================================================ "You will keep in perfect peace him whose mind is steadfast, because he trusts in you." Isaiah 26:3 Tim Tautges Argonne National Laboratory ([email protected]) (telecommuting from UW-Madison) phone: (608) 263-8485 1500 Engineering Dr. fax: (608) 263-4499 Madison, WI 53706 - "Local" operations are operations performed with respect to a part. TJT: not quite, local are first local to an instance/processor, then in very specific cases local to a part. Partition instances know which mesh instance's entities they partition. TJT: don't think so, analogous to an entity set not knowing which instance it's in Mesh instances know which partitions they are in. TJT: other way around, mesh instances know which partitions they contain These functions will be reinterpreted so that they can accept either a mesh instance and EntitySet handle, or a partition handle and a part handle. See Carl's Oct 15 email for more detail. TJT: nope, mesh instance is always required; entity set handle can be either that or a part handle void prefix_createPartitionMPI: . does this function (and destroy) imply communication if called in parallel? . shouldn't the communicator be optional, to allow creating partitions in serial? // Given a partition handle, return the mesh instance that is // included in the partition. // KDD: DO WE NEED THIS CAPABILITY? IT IS HARD TO SPECIFY // KDD: IF iMesh_Instance MUST ALWAYS BE FIRST ARG OF iMesh FUNCTIONS. :) TJT: agreed, we don't need it, analogous to sets getRank, getRankArr: might these imply communication? getNumOfTypePar, getNumOfTopoPar: does it make sense to have an entity set handle parameter for this function? What does that mean in parallel? Do entity sets given on each processor have to be ones shared in parallel? prefix_getNumEntSetsPar: we don't add entity sets to partitions, so how can we get them for a partition? Is this function really for getting the entity sets over all procs? Need to distinguish partition from mesh across all processes in the communicator Global IDs: I agree it should be a service. Parts/part ids: why have functions which take/return parts as handles and ids both? Why not just deal with handles, and provide ids as separate functionality? getNumOfType/Topo, getEntSets: shouldn't this functionality be available from EntSet functions, like add/rmv to/from part? getAllVtxCoords, getVtxCoordIndex, getEntities, getAdjEntities, others: similar comment as for getNumEntSetsPar above getEntityOwner, others: would suggest using getEntityPart, etc., since Owner is ambiguous isEntOwner, isEntParStatus: isEntOwner should tell strictly whether entity is in specified part (also, why have this be separate function from getEntityOwner?); isEntParStatus tells whether entity is a copy/ghost and other info getNumCopiesEnt: num copies on processes or parts? addCopyEnt, removeCopyEnt: do these functions communicate? - For functions that replace a set handle with a part handle, return all appropriate entities in a part, whether they are owned or are copies. The application can test for ownership if needed. TJT: is a copied entity put in a part? I don't think it should be... Need some functionality which exchanges tag data for entities on interface and ghost entities (e.g. parallel smoothing application)
On Tue, 2008-01-22 at 11:48 -0700, Devine, Karen D wrote:
Vitus: - Propose the communication functions needed to support prefix_sendEntArrToParts and prefix_receiveEntArrToParts.
Karen, Here are the minor syntax corrections we discussed for these two functions. void prefix_sendEntArrToParts(iMesh_Instance instance, /*in*/ const prefix_PartitionHandle partition_handle, /*in*/ const iBase_EntityHandle *entity_handles, /*in*/ const int entity_handles_size, /*in*/ const int *target_part_ids, /*in*/ int command_code, // e.g., MIGRATE,COPY int *err); void prefix_receiveEntArrToParts(iMesh_Instance instance, /*in*/ const prefix_PartitionHandle partition_handle, /*inout*/ iBase_EntityHandle **entity_handles, /*inout*/ int *entity_handles_size, /*out*/ int *entity_handles_allocated, /*inout*/ int *target_part_ids, /*out*/ int *target_part_ids_allocated, /*inout*/ int command_code, // e.g., MIGRATE,COPY int *err); The prefix_sendEntArrToParts() should be a collective call. With the entity_handles and target_part_ids in all the calls, the implementation can determine the pairwise send/receives with all to all communication. This can be held by the implementation until the prefix_receiveEntArrToParts() call or passed back to the application in an additional argument. If the additional argument is used, it should be passed to prefix_receiveEntArrToParts(). Vitus
Vitus: Thanks for the corrections and sorry for the confusion. I think these calls need some sort of "message tag" (analogous to MPI's message tag) that allows sends and receives to be interwoven. Also, we should consider whether we need a post/send/wait model as Onkar described. Karen On 2/4/08 1:30 PM, "Vitus Leung" <[email protected]> wrote:
On Tue, 2008-01-22 at 11:48 -0700, Devine, Karen D wrote:
Vitus: - Propose the communication functions needed to support prefix_sendEntArrToParts and prefix_receiveEntArrToParts.
Karen,
Here are the minor syntax corrections we discussed for these two functions.
void prefix_sendEntArrToParts(iMesh_Instance instance, /*in*/ const prefix_PartitionHandle partition_handle, /*in*/ const iBase_EntityHandle *entity_handles, /*in*/ const int entity_handles_size, /*in*/ const int *target_part_ids, /*in*/ int command_code, // e.g., MIGRATE,COPY int *err);
void prefix_receiveEntArrToParts(iMesh_Instance instance, /*in*/ const prefix_PartitionHandle partition_handle, /*inout*/ iBase_EntityHandle **entity_handles, /*inout*/ int *entity_handles_size, /*out*/ int *entity_handles_allocated, /*inout*/ int *target_part_ids, /*out*/ int *target_part_ids_allocated, /*inout*/ int command_code, // e.g., MIGRATE,COPY int *err);
The prefix_sendEntArrToParts() should be a collective call. With the entity_handles and target_part_ids in all the calls, the implementation can determine the pairwise send/receives with all to all communication. This can be held by the implementation until the prefix_receiveEntArrToParts() call or passed back to the application in an additional argument. If the additional argument is used, it should be passed to prefix_receiveEntArrToParts().
Vitus
On Tue, 2008-02-05 at 10:45 -0700, Devine, Karen D wrote:
Vitus: Thanks for the corrections and sorry for the confusion. I think these calls need some sort of "message tag" (analogous to MPI's message tag) that allows sends and receives to be interwoven. Also, we should consider whether we need a post/send/wait model as Onkar described.
Karen
Actually, no confusion. Good suggestion o the "message tag" that would allow sends and receives to be interwoven and better support the post/send/wait model that Onkar suggested. Also, good suggestions from Onkar on ghosts and Par in name. However, I still think that the two calls are sufficient. void prefix_sendEntArrToPartsPar(iMesh_Instance instance, /*in*/ const prefix_PartitionHandle partition_handle, /*in*/ const iBase_EntityHandle *entity_handles, /*in*/ const int entity_handles_size, /*in*/ const int *target_part_ids, /*in*/ int command_code, // e.g., MIGRATE,COPY /*in*/ int update_ghost, // e.g., YES,NO /*in*/ int message_tag, int *err); // The implementation has everything it needs to determine sends and // receives. It can send and post non-blocking receives. Upon return, // the application can do other stuff for latency hiding. void prefix_receiveEntArrToPartsPar(iMesh_Instance instance, /*in*/ const prefix_PartitionHandle partition_handle, /*inout*/ iBase_EntityHandle **entity_handles, /*inout*/ int *entity_handles_size, /*out*/ int *entity_handles_allocated, /*inout*/ int *target_part_ids, /*out*/ int *target_part_ids_allocated, /*inout*/ int command_code, // e.g., MIGRATE,COPY /*in*/ int update_ghost, // e.g., YES,NO /*in*/ int message_tag, int *err); // The implementation can post blocking receives or wait for non- // blocking receives previously posted. Targets are correct, because // there may be multiple parts on a processor and the application does // not need to know sources. The message_tag ties // a send and receive pair together. Vitus
Actually, no confusion. Good suggestion o the "message tag" that would allow sends and receives to be interwoven and better support the post/send/wait model that Onkar suggested. Also, good suggestions from Onkar on ghosts and Par in name. However, I still think that the two calls are sufficient.
I do not know if two calls will be sufficient for non-blocking send and recv., I still think we need equivalent of finishSendRecv (like MPI_Waitall). And don't we need a call for setup of communications upfront, if we do then we can pass message tag, update_ghost, command_code in there and store them in itapsSendRecvTag and reuse this tag in other calls (send, recv etc.). For 'target' in recv., entity_handles can easily provide this information and I am not sure that source will not be useful or needed for all applications. In any case, syntax and name (where to use 'Par') might change on which calls (or model) we decide/agree on. Thanks, Onkar
void prefix_sendEntArrToPartsPar(iMesh_Instance instance, /*in*/ const prefix_PartitionHandle partition_handle, /*in*/ const iBase_EntityHandle *entity_handles, /*in*/ const int entity_handles_size, /*in*/ const int *target_part_ids, /*in*/ int command_code, // e.g., MIGRATE,COPY /*in*/ int update_ghost, // e.g., YES,NO /*in*/ int message_tag, int *err);
// The implementation has everything it needs to determine sends and // receives. It can send and post non-blocking receives. Upon return, // the application can do other stuff for latency hiding.
void prefix_receiveEntArrToPartsPar(iMesh_Instance instance, /*in*/ const prefix_PartitionHandle partition_handle, /*inout*/ iBase_EntityHandle **entity_handles, /*inout*/ int *entity_handles_size, /*out*/ int *entity_handles_allocated, /*inout*/ int *target_part_ids, /*out*/ int *target_part_ids_allocated, /*inout*/ int command_code, // e.g., MIGRATE,COPY /*in*/ int update_ghost, // e.g., YES,NO /*in*/ int message_tag, int *err);
// The implementation can post blocking receives or wait for non- // blocking receives previously posted. Targets are correct, because // there may be multiple parts on a processor and the application does // not need to know sources. The message_tag ties // a send and receive pair together.
Vitus
Hi Karen and all, I reviewed the document and have the following additional big picture questions for everyone (I agree we need to discuss the 4 at the bottom of this email as they occurred to me too when reading through the spec). I have a number of smaller issues as well, but will focus only on the larger items here. I apologize if this duplicates other folks' input - I haven't had time to review all the email sent out today. - Our notion of partition neighbors appears to be defined by 'sharing' entities and I'm wondering about the case in which all entities are uniquely owned by a partition so nothing is 'shared'. This is how I think about partitions, and I think about partition neighbors as being defined by adjacency information in the mesh. - I found the difference between partition handles and partition IDs, and when you would use which one, to be very confusing and non intuitive. Do we really truly need both, or can we just go with part IDs? - the prefix_getNumParts and prefix_getParts seems to duplicate the functionality just above it in the file if you just add a myrank, so I suggest eliminating these two functions. Speaking of the functions just above (getNumPartIds and getNumPartIdsArr) - not really sure why you bring "Ids" into the function name other than to avoid conflict with the function below which I suggest eliminating) - Global IDs. I'm fine with having this be a service on top of the interface, but we need the functionality in Mesquite, so I would propose it be separate from the load balancing service. - In many instances there's a "getNumX" and a "getX" - I think we should look at whether or not we always need both functions or if we can just use getX, which returns the num as a byproduct. I realize there are many cases where you do need to separate the calls, but perhaps not all. - prefix_sendEntArrToParts and the corresponding receive function both appear to take an entityset handle - I would have expected they take an entityhandle... is this a typo, or am I missing something? - I like the idea of getting rid of the extra iterator functions (other than the init function) if we can - isEntOwner appears to return "boundary", I think we should limit it to "owner" or "copy" --- "boundary" fits more naturally with, and is already included in, the next function isEntParStatus (which should probably be 'getEntParStatus') - I struggle with the copyEntOnPart functions... I don't fully understand what this adds to the interface and I think it could lead to headaches when thinking about consistency of information across processors. - Seems as though the addCopyEnt and removeCopyEnt could be combined with the other add functions (that are like entityset functions) with an appropriate enumerator (e.g. OWNED, COPY, etc).. Lori Devine, Karen D wrote:
In the last phone conf, someone suggested that we put together a draft of the syntax for the capabilities on which we agreed. So I added syntax from Carl's, Onkar's and Tim's past emails to our Combined document. For readability, I turned the file into a C header file; the discussions from the Combined document are now comments in the C file. The file is attached.
For now, I'd rather not discuss: - values of "prefix_". - my use of "//" for comments being non-standard for some C compilers. - my inconsistent use of "const" (unless I used it for a return argument -- oops!) - whether or not this file would actually compile.
Here are your assignments:
ALL: Please give your opinions of the questions below. Some are simple questions resulting from the fact that I'm not used to ITAPS code conventions.
Mark M. and Lori: - Review all functions at a high level to identify gaps and duplications in capability. Suggest functions that should be added or removed. - Review all functions at a high level to determine whether our syntax is generally consistent with ITAPS' conventions and generally consistent within the document. You shouldn't catch every detail; just let us know whether someone who is comfortable with other ITAPS interfaces would find this interface comfortable as well.
Mark M and Onkar: - Review all functions at a high level and indicate whether their use of communication will be definite, likely, or nonexistent. If they incur communication, will the communication be global (e.g., MPI_Allreduce) or point-to-point (e.g., MPI_Send/MPI_Recv)? We'll eventually modify the function names to indicate whether or not they incur communication.
Onkar: - Review in detail the Entity Functionality section. This section has the entity copying capability, including the add/remove copies that you discussed in the last phone conf.
Vitus: - Propose the communication functions needed to support prefix_sendEntArrToParts and prefix_receiveEntArrToParts.
Carl: - Should functions prefix_getEntSets and prefix_getNumEntSets be added to Carl's sixteen functions�? They appear to be analogous to prefix_getEntities and prefix_getNumOfType. - Review in detail the Part Functionality section.
Tim: - Review in detail the Partition Functionality section.
------------------------ Questions for everyone:
- When we specify a part handle, must we always also specify a partition handle? That is, should a part be uniquely identified by the pair (partition_handle, part_handle), just as entities are uniquely identified by (mesh_instance, entity_handle)? "No" is the most convenient answer to this question; if parts are uniquely identified by only the part handle, we can do the entity-set argument overloading that we discussed at bootcamp. We can also remove the partition_handle argument from many of the functions. I had previously argued for use of the tuple, but many things would be easier if we didn't need it. Can all the implementations generate part handles that are unique within an iMesh_Instance, rather than within a partition?
- With iterators over parts or part boundaries, is it necessary to have separate interface functions for getNext, reset, and end, or will the standard iMesh functions for getNext, reset, and end suffice? That is, once an iterator is initiated, do we need to continue passing the part handle and, perhaps, neighbor-part id, to the iterator, or can the iterator "remember" these just as it remembers entity type and topology? If we can re-use the iMesh getNext, reset and end, we can remove six functions from Carl's sixteen functions�, bringing the count back to his original estimate of ten.
- Which functions should operate on single parts and/or entities, and which should have array versions (function_name and function_nameArr)? I included both wherever we had a comment "Need single-part and array-part versions" in the combined document, but I won't be offended if we remove some of the functions.
- Since global IDs for entities are not used anywhere in the interface, I propose their generation and use should be a service on top of the interface rather than part of the interface. What say you?
Lori, I have tried to answer some of your specific queries, see below. I am still trying to work on my overall comments and concerns (which too will now have many overlapping inputs).
- I found the difference between partition handles and partition IDs, and when you would use which one, to be very confusing and non intuitive. Do we really truly need both, or can we just go with part IDs?
I think a comment will be useful to clarify (as Karen, Carl... agree) that part IDs are mainly needed to refer to remote parts like if we want to migrate entities from a part to another part then another part is described in terms of 'part ID' (i.e., destination_part_id). We need part-handles, that can be pointers and are local to a process, at least in serial iMesh functions (in place of entity-set-handle); and also for other things.
- the prefix_getNumParts and prefix_getParts seems to duplicate the functionality just above it in the file if you just add a myrank, so I suggest eliminating these two functions. Speaking of the functions just above (getNumPartIds and getNumPartIdsArr) - not really sure why you bring "Ids" into the function name other than to avoid conflict with the function below which I suggest eliminating)
Ones with rank can provide info. for any rank (off-process) and moreover, prefix_getParts provide local on-process part-handles (that can be further used in serial iMesh functions). I do not remember why we need getNumPartIds and getnumPartIdsArr. If possible we should avoid them.
- prefix_sendEntArrToParts and the corresponding receive function both appear to take an entityset handle - I would have expected they take an entityhandle... is this a typo, or am I missing something?
Yes, infact we do not require part handle(s) (Vitus too proposed these syntax corrections). And in receive I think 'target' should refer as 'source'.
- I struggle with the copyEntOnPart functions... I don't fully understand what this adds to the interface and I think it could lead to headaches when thinking about consistency of information across processors.
Remote copy information is very critical in all of RPI's applications. These functionalities provide link for shared entities. getCopyEntOnPart is same as getCopiesEnt but only for a specific part. If names are confusing then I think some re-naming might be required like changing getCopyEntOnPart to getEntCopyOnAPart. I think providing appropriate documentation and code-examples may avoid difficulties in thinking about them.
- Seems as though the addCopyEnt and removeCopyEnt could be combined with the other add functions (that are like entityset functions) with an appropriate enumerator (e.g. OWNED, COPY, etc)..
No this is not the case. I want to also clarify that addCopyEnt can be called many times for a particular entity on a specific part (in case where we want to add multiple remote copies for a particular entity on a specific part, we may also want to provide add/rmvCopies). Please let me know if something is not clear. - Onkar
Lori
Devine, Karen D wrote:
In the last phone conf, someone suggested that we put together a draft of the syntax for the capabilities on which we agreed. So I added syntax from Carl's, Onkar's and Tim's past emails to our Combined document. For readability, I turned the file into a C header file; the discussions from the Combined document are now comments in the C file. The file is attached.
For now, I'd rather not discuss: - values of "prefix_". - my use of "//" for comments being non-standard for some C compilers. - my inconsistent use of "const" (unless I used it for a return argument -- oops!) - whether or not this file would actually compile.
Here are your assignments:
ALL: Please give your opinions of the questions below. Some are simple questions resulting from the fact that I'm not used to ITAPS code conventions.
Mark M. and Lori: - Review all functions at a high level to identify gaps and duplications in capability. Suggest functions that should be added or removed. - Review all functions at a high level to determine whether our syntax is generally consistent with ITAPS' conventions and generally consistent within the document. You shouldn't catch every detail; just let us know whether someone who is comfortable with other ITAPS interfaces would find this interface comfortable as well.
Mark M and Onkar: - Review all functions at a high level and indicate whether their use of communication will be definite, likely, or nonexistent. If they incur communication, will the communication be global (e.g., MPI_Allreduce) or point-to-point (e.g., MPI_Send/MPI_Recv)? We'll eventually modify the function names to indicate whether or not they incur communication.
Onkar: - Review in detail the Entity Functionality section. This section has the entity copying capability, including the add/remove copies that you discussed in the last phone conf.
Vitus: - Propose the communication functions needed to support prefix_sendEntArrToParts and prefix_receiveEntArrToParts.
Carl: - Should functions prefix_getEntSets and prefix_getNumEntSets be added to Carl's sixteen functions�? They appear to be analogous to prefix_getEntities and prefix_getNumOfType. - Review in detail the Part Functionality section.
Tim: - Review in detail the Partition Functionality section.
------------------------ Questions for everyone:
- When we specify a part handle, must we always also specify a partition handle? That is, should a part be uniquely identified by the pair (partition_handle, part_handle), just as entities are uniquely identified by (mesh_instance, entity_handle)? "No" is the most convenient answer to this question; if parts are uniquely identified by only the part handle, we can do the entity-set argument overloading that we discussed at bootcamp. We can also remove the partition_handle argument from many of the functions. I had previously argued for use of the tuple, but many things would be easier if we didn't need it. Can all the implementations generate part handles that are unique within an iMesh_Instance, rather than within a partition?
- With iterators over parts or part boundaries, is it necessary to have separate interface functions for getNext, reset, and end, or will the standard iMesh functions for getNext, reset, and end suffice? That is, once an iterator is initiated, do we need to continue passing the part handle and, perhaps, neighbor-part id, to the iterator, or can the iterator "remember" these just as it remembers entity type and topology? If we can re-use the iMesh getNext, reset and end, we can remove six functions from Carl's sixteen functions�, bringing the count back to his original estimate of ten.
- Which functions should operate on single parts and/or entities, and which should have array versions (function_name and function_nameArr)? I included both wherever we had a comment "Need single-part and array-part versions" in the combined document, but I won't be offended if we remove some of the functions.
- Since global IDs for entities are not used anywhere in the interface, I propose their generation and use should be a service on top of the interface rather than part of the interface. What say you?
participants (7)
-
Carl Ollivier-Gooch -
Devine, Karen D -
Lori A. Diachin -
Mark Shephard -
Onkar Sahni -
Tim Tautges -
Vitus Leung