Attached are the revised DraftInterface.h and presentation from Bootcamp.
Below are comments and questions that arose during our discussion. (These
comments are also listed in DraftInterface.h.)
- Changed getPartRank to getRankOfPart. (Carl)
- Made sure iMeshP_getNumOfTypeAll and iMeshP_getNumOfTopoAll were
documented as collective operations. (Carl)
- Changed suffix "Par" to "All". (Lori)
- Added iMeshP_testPart() to test status of part handle, returning LOCAL,
REMOTE, or INVALID. (Mark M, Lori).
- Changed iMeshP_addCopyOf and iMeshP_rmvCopyOf back to iMeshP_addGhostOf
and iMeshP_rmvGhostOf. If we wanted to use these functions for adding
boundary copies, we'd have to include a list of already existing remote
copies in the arguments, as well as communicate with parts already owning
copies to let them know a ghost copy has been made. Actually, this raises
an interesting question: does a boundary copy need to know about all ghost
copies of it?
- Changed getEntParStatus to getEntStatus. (Lori)
- Changed sendEntArrToPartsPar to exchEntArrToPartsAll. (Lori,Tim)
TO DO:
Parts and Processes:
- Martin argued for consecutive unique Part IDs in addition to or instead
of Part handles. He will send use cases. If we decide to add them back to
the interface, we could compute them in iMeshP_syncPartitionAll rather than
in iMeshP_createPart. That is, an application couldn't access them until
after iMeshP_syncPartitionAll.
- Are part handles globally unique? They probably need to be globally
unique in order for them to be useful as remote part handles. Also, does
the process rank need to be encoded in the part handle in order to map from
parts to processes for communication?
- If in iMeshP_syncPartitionAll, we computed a mapping from part handles to
integers 0,..., k-1, we could store only ranges of integers to achieve the
part-to-process and process-to-parts mappings; this would require O(P)
storage per process for P processes.
- Alternatively, the mapping of all parts to processes can be stored in
O(k) total memory, distributed across processors (e.g., a distributed data
directory) but interrogating the directory requires communication.
- iMeshP_getPartsOnRank created discussion and needs to be resolved.
IMeshP_getPartsOnRank would likely require either O(k) storage per process
for k parts or communication. For other points, please see Mark M's 3/12/08
email.
CreateEnt:
- Carl asked if we should have a version of createEnt that accepts a part
handle. Should this function be used only for creating owned entities?
How do you envision creating part boundary entities when a parallel mesh is
initially loaded?
Ghost entities:
- We currently have a mechanism only for pushing ghosts onto other parts.
Will we want a mechanism for pulling them, too? (E.g., a part says, "I want
ghosts for this entity.")
PartNbor functions:
- Did we agree to remove the entity type from these functions? That is, do
we want to return the part handles for all parts that have any copies? The
entity type was intended to allow us to get the part handles for all parts
that have copies of a given type (perhaps ALL_TYPES).
Functions handling both Parts and Entity Sets:
- Tim said these function names (e.g., iMeshP_getNumOfType,
iMeshP_getAllVtxCoord) are too close to existing iMesh function names, even
though the argument lists would be different. He agreed to email
suggestions for better names.
Copies:
- Functions getNumCopies, getCopies, getCopyParts, and getCopyOnPart have
different behavior for ghost and part-boundary copies. Ghosts will return
only itself and its owner in getCopies; part-boundary entities will return
copies on other parts as well.
- Tim envisions applications (e.g., FETI methods) updating tag data in
their ghosts that they would like to accumulate back to the owner. Xiaolin
said that he writes in his ghosts, but doesn't send those values back to the
owner. Currently, we have the ability only to send tag data from owners to
ghosts. Tim will look at this issue and propose a solution.
Communication:
- Although we should think in terms of parts, communication really occurs
with respect to processes. We need to make sure all our communication
routines make sense with respect to both processes and parts, and perhaps,
revise their descriptions. Also, if we send to parts, the implementation
really does need the mapping of parts to processes.
Entity Owner/Status Queries
- Should we combine functions getEntOwnerPart and getEntStatus into one
function? Or should we combine functions getOwnerCopy and getEntOwner into
one function? Or should we remove getOwnerCopy and make applications call
getOwnerPart followed by getCopyOnPart?
