Carl led a discussion on non-collective communication needed. (Thanks,
Carl.)
Premises:
- The interface should provide functions to handle commonly needed
communication of mesh entities (so the application doesn't have to do it).
- Use non-blocking communication whenever possible.
- All entities on part boundaries know all remote handles and part handles
of all copies.
- All ghost entities know the remote handle and part handle of the entity
owner.
- Update of ghost information will be done at end of individual migrations
(rather than kept consistent through mesh modification).
- Some sort of entity locking may be needed, at least within
implementations.
- Tag and set info for entities would be included in the migration.
Non-collective communication needed:
1. Requests to migrate remote entities and (optionally) their higher-order
adjacencies. Note that the iMesh implementation must be smart enough to
change ownership of lower-order entities that become orphaned due to
migration.
2. Push entity data from owner to other parts for migrating entities and
updating remote copies.
3. Allow parts to poll for requests and check status of their own
outstanding requests.
4. A "closing" blocking (perhaps collective) call that ensures all parts
are done and all requests are handled. This function may be where the
ghosts are updated for mesh consistency.
5. Capability to replace an entity with new entities, both locally and
remotely. This capability would include requests to perform an operation on
an entity copy (e.g., split that edge), as well as communication to update
remote entity handles for the new entities created.
We discussed a use-case -- edge splitting -- for need 5 above:
- Part A decides to split edge eA; it creates edges eA1 and eA2 and vertex
vA.
- Part A sends requests to all parts with copies of edge eA (say Part B),
telling them to split the edge, too. (This request might have the same form
as need 1, with a "SPLIT" command.)
- All processes poll for requests.
- Part B, having a copy eB of edge eA, receive Part A's request, and splits
edge eB, creating edges eB1 and eB2 and vertices vB.
- Part A sends message to Part B saying "edge eA is now edges eA1 and eA2
with vertex vA."
- Part B sends message to Part A saying "edge eB is now edges eB1 and eB2
with vertex vB."
- Parts A and B receive these update messages and locally update their
remote entity handles for the new edges and vertices.
- Of course, if entity eA is copied on more than one part, all parts with
copies participate in the split and communication.
- In this example, the interface would need functions for sending requests;
polling for and receiving requests; sending the entity-update data; and
receiving and processing the entity-update data.
Mark Miller reminded us that we should allow implementations to bundle
multiple messages together before sending.
Onkar said the addCopyEnt and rmvCopyEnt functions may be made obsolete by
the non-collective communication functions, so I may comment them out of the
draft for now.
