- itaps-parallel - mailman3.cels.anl.gov

Tag communication operations
by Tim Tautges 27 Feb '08

27 Feb '08

Hi all, Regarding tag communication operations, the goal is to provide in the interface functions that most parallel applications are likely to need. One functionality that comes to mind right away is exchanging tag information on shared and/or ghosted entities. We could get fancy about it, getting more into what kinds of operations would be provided (min/max, average, etc.), but at this point I think it would be better to keep it simple. So, I'd propose two functions: sendTagsShared: communicate specified tag to all processors sharing entities owned by this processor sendTagsGhosted: same, but for ghosted entities The particular syntax for these functions can come later, I just want to get some comments on the general idea. If applications wanted some sort of higher-level behavior, like the operations mentioned above, additional tags could be used to implement those fairly easily. - tim -- ================================================================ "You will keep in perfect peace him whose mind is steadfast, because he trusts in you." Isaiah 26:3 Tim Tautges Argonne National Laboratory (tautges(a)mcs.anl.gov) (telecommuting from UW-Madison) phone: (608) 263-8485 1500 Engineering Dr. fax: (608) 263-4499 Madison, WI 53706

3 3

Will miss phone call...
by Tim Tautges 21 Feb '08

21 Feb '08

Sorry folks, had to attend another mtg down at ANL. My schedule's pretty open the next few weeks, so schedule a mtg any day (except maybe Friday afternoon). - tim -- ================================================================ "You will keep in perfect peace him whose mind is steadfast, because he trusts in you." Isaiah 26:3 Tim Tautges Argonne National Laboratory (tautges(a)mcs.anl.gov) (telecommuting from UW-Madison) phone: (608) 263-8485 1500 Engineering Dr. fax: (608) 263-4499 Madison, WI 53706

2 1

Revised DraftInterface.h
by Devine, Karen D 14 Feb '08

14 Feb '08

Attached is a revised version of DraftInterface.h, along with the CVS diff from the previous version you saw. In going through everyone's reviews, I saw that confusion remains regarding copies and their management. In this week's phone conf, we'll discuss the communication functions (sendEntArrToPartsPar/recvEntArrFromPartsPar) as well as any function with Copy/Copies and Owned in its name. There are many other parts of the interface marked "TODO". Let's discuss them over email to reach consensus on as many as possible before using phone time for them. I have not received the new part-neighbor definition yet, so the old one is still in the document. Finally, I tried to address all comments in the reviews, but if there is a point that I missed or didn't address to your satisfaction, please let me know. Thanks for your contributions! Karen

5 10

Re: Question for Harold
by Tim Tautges 12 Feb '08

12 Feb '08

[bounced because of non-member submission, so I'm re-sending] Karin: After looking at things and thinking about it I am fine with (mesh instance, part handle) for referencing. Creating a whole new set of interface functions seems like a lot of work for interface developers and the applications. So if it can be avoided I certainly am for that. Harold... -----Original Message----- From: Devine, Karen D [mailto:[email protected]] Sent: Monday, February 11, 2008 10:08 AM To: Trease, Harold E Cc: itaps-parallel(a)mcs.anl.gov Subject: Question for Harold Hi, Harold. The ITAPS Parallel Interface subcommittee has a question for you. We have defined partitions to be assignments of entities to subsets, and parts to be those subsets. We assume that a mesh instance may have several partitions, but a partition contains entities from only one mesh instance. We also assume a processor may have many parts, but each part is contained completely within only one processor. We identify partitions with a (mesh instance, partition handle) pair. That is, functions that operate on partitions would take a mesh instance and a partition handle as input arguments. Similarly, within a processor, we would like to identify parts with a (mesh instance, part handle) pair so that we can overload many of the iMesh EntitySet functions with part handles. That is, we would like to allow, say, iMesh_getNumOfType to accept either an entity set handle OR a part handle, and return the number of entities of a given type in the entity set or the part, respectively. There was some concern, however, that (1) some implementations may not be able to distinguish between part and entity set handles, and (2) some implementations might not be able to uniquely identify parts given only the mesh instance and part handle (e.g., they might also need the partition handle). The alternative to overloading the EntitySet argument is to define many new interface functions that take the mesh instance, partition handle, and part handle as input and duplicate the EntitySet functionality on parts. Within the subcommittee, we agreed that we could accept the (mesh instance, part handle) pair only if your implementations could support it. So, you wield the power now. :) Would the (mesh instance, part handle) pair be sufficient for uniquely identify parts on a processor? Thanks. Karen -- ================================================================ "You will keep in perfect peace him whose mind is steadfast, because he trusts in you." Isaiah 26:3 Tim Tautges Argonne National Laboratory (tautges(a)mcs.anl.gov) (telecommuting from UW-Madison) phone: (608) 263-8485 1500 Engineering Dr. fax: (608) 263-4499 Madison, WI 53706

1 0

Question for Harold
by Devine, Karen D 11 Feb '08

11 Feb '08

Hi, Harold. The ITAPS Parallel Interface subcommittee has a question for you. We have defined partitions to be assignments of entities to subsets, and parts to be those subsets. We assume that a mesh instance may have several partitions, but a partition contains entities from only one mesh instance. We also assume a processor may have many parts, but each part is contained completely within only one processor. We identify partitions with a (mesh instance, partition handle) pair. That is, functions that operate on partitions would take a mesh instance and a partition handle as input arguments. Similarly, within a processor, we would like to identify parts with a (mesh instance, part handle) pair so that we can overload many of the iMesh EntitySet functions with part handles. That is, we would like to allow, say, iMesh_getNumOfType to accept either an entity set handle OR a part handle, and return the number of entities of a given type in the entity set or the part, respectively. There was some concern, however, that (1) some implementations may not be able to distinguish between part and entity set handles, and (2) some implementations might not be able to uniquely identify parts given only the mesh instance and part handle (e.g., they might also need the partition handle). The alternative to overloading the EntitySet argument is to define many new interface functions that take the mesh instance, partition handle, and part handle as input and duplicate the EntitySet functionality on parts. Within the subcommittee, we agreed that we could accept the (mesh instance, part handle) pair only if your implementations could support it. So, you wield the power now. :) Would the (mesh instance, part handle) pair be sufficient for uniquely identify parts on a processor? Thanks. Karen

1 0

abstraction for 'neighbors of a part'
by Onkar Sahni 09 Feb '08

09 Feb '08

Carl, Mark M., Mark S. (RPI folks), Tim (also others if you are interested): Tim, I just wanted to provide some ideas before tomorrow's ITAPS parallel-interface phone conf. such that we can discuss them at that time and hence, I am writing this email with some of my ideas and proposal. There are two parts to this email: (a) points to consider in abstraction of definition for 'neighbors of a part' and (b) "an" abstraction for 'neighbors of a part'. Read below. ------------------------------------------------------------ Points to consider in definition for 'neighbors of a part': 1) Provide abstraction in definition for 'neighbors of a part', which is based on underlying mesh being partitioned (and is not completely arbitrary), and that can fit applications need. Basically, abstract ways of describing 'neighbors of a part' that application can understand, request and use (at least as building blocks to possibly achieve their more complex needs). 2) In this process, possibly proceed with most basic one (inclusive of common apps.) or may be two, or even handful, but allow for future possibilities of incorporating other reasonable ones. I do not think at this stage in ITAPS parallel-interface we want to get all possible abstractions. 3) If we decide on more than one, then we can consider the possibility of providing functionality to dynamically switch between them (i.e., on the fly, at run-time). (Non-manifold geometries can be of interest/importance and may have extra valid/reasonable requirements which will introduce complexities and should be considered). ------------------------------------------------------------ ------------------------------------------------------------ Proposal for "a" most basic definition of 'neighbors of a part' (that will satisfy (building block) needs of common apps. (which I can think of)): Provided entity-type(s), two parts are considered neighbors if each of them at least "possess" one (first-order) adjacent "contributor" to any entity of specified-type(s), where, "possess" implies internal and also owner (for example, non-ghost mesh regions in a part) and "contributor" are mesh entities that do not bound any higher-order entity (for example, in a case of cube with an extra hanging flap/face (like tin-sheet in a brick wall), the mesh faces on flap (tin-sheet) and mesh regions within cube (brick-wall) are contributor entities). At this point, this may look crude and complex (also lousy) and using mathematical form may clear a lot things. In any case, to make this more abstract we can further decide what are parameters various applications want to have control over (like specific entity-type or entity-types, first-order or second-order adjacencies, "contributor" or "highest-dimension" or "specified-type" adjacencies etc.). Moreover, implementations can tell which abstractions they support and in each abstraction what are constraints over input parameters (some implementations may say they don't support non-manifold cases). ------------------------------------------------------------ Feel free to polish/improve on above points. Also please let me know your feedback. Thanks, Onkar

2 2

Neighbor definition
by Tim Tautges 06 Feb '08

06 Feb '08

A question arose in yesterday's telecon on what constitutes a part neighbor. After some discussion, we came up with the following definition and rationale. Def: for a given part, a part neighbor of dimension d is any other part which shares mesh entities of dimension d with that part (i.e. the entity is represented on both parts). Rationale: - it is desirable to tie the definition of a part neighbor with information intrinsic to the mesh, i.e. one could derive information about part neighbors using only information about the mesh entities and the partition of entities over processors - in general, applications will have other criteria determining which other processors a given processor communicates with; that information can be stored in iMesh using sets and tags already, and applications will presumably be able to pass those things to communication functions in the itaps parallel interface (thus driving the application-specific communication patterns) Question: are ghosted/copied entities between parts considered shared? I would assert that ghosting/copying should be between processors, not parts, so this question is not relevant with respect to part neighbors. - tim -- ================================================================ "You will keep in perfect peace him whose mind is steadfast, because he trusts in you." Isaiah 26:3 Tim Tautges Argonne National Laboratory (tautges(a)mcs.anl.gov) (telecommuting from UW-Madison) phone: (608) 263-8485 1500 Engineering Dr. fax: (608) 263-4499 Madison, WI 53706

5 11

Assignments for parallel interface, due before 2/5 phone conf
by Devine, Karen D 05 Feb '08

05 Feb '08

In the last phone conf, someone suggested that we put together a draft of the syntax for the capabilities on which we agreed. So I added syntax from Carl's, Onkar's and Tim's past emails to our Combined document. For readability, I turned the file into a C header file; the discussions from the Combined document are now comments in the C file. The file is attached. For now, I'd rather not discuss: - values of "prefix_". - my use of "//" for comments being non-standard for some C compilers. - my inconsistent use of "const" (unless I used it for a return argument -- oops!) - whether or not this file would actually compile. Here are your assignments: ALL: Please give your opinions of the questions below. Some are simple questions resulting from the fact that I'm not used to ITAPS code conventions. Mark M. and Lori: - Review all functions at a high level to identify gaps and duplications in capability. Suggest functions that should be added or removed. - Review all functions at a high level to determine whether our syntax is generally consistent with ITAPS' conventions and generally consistent within the document. You shouldn't catch every detail; just let us know whether someone who is comfortable with other ITAPS interfaces would find this interface comfortable as well. Mark M and Onkar: - Review all functions at a high level and indicate whether their use of communication will be definite, likely, or nonexistent. If they incur communication, will the communication be global (e.g., MPI_Allreduce) or point-to-point (e.g., MPI_Send/MPI_Recv)? We'll eventually modify the function names to indicate whether or not they incur communication. Onkar: - Review in detail the Entity Functionality section. This section has the entity copying capability, including the add/remove copies that you discussed in the last phone conf. Vitus: - Propose the communication functions needed to support prefix_sendEntArrToParts and prefix_receiveEntArrToParts. Carl: - Should functions prefix_getEntSets and prefix_getNumEntSets be added to Carl's sixteen functions? They appear to be analogous to prefix_getEntities and prefix_getNumOfType. - Review in detail the Part Functionality section. Tim: - Review in detail the Partition Functionality section. ------------------------ Questions for everyone: - When we specify a part handle, must we always also specify a partition handle? That is, should a part be uniquely identified by the pair (partition_handle, part_handle), just as entities are uniquely identified by (mesh_instance, entity_handle)? "No" is the most convenient answer to this question; if parts are uniquely identified by only the part handle, we can do the entity-set argument overloading that we discussed at bootcamp. We can also remove the partition_handle argument from many of the functions. I had previously argued for use of the tuple, but many things would be easier if we didn't need it. Can all the implementations generate part handles that are unique within an iMesh_Instance, rather than within a partition? - With iterators over parts or part boundaries, is it necessary to have separate interface functions for getNext, reset, and end, or will the standard iMesh functions for getNext, reset, and end suffice? That is, once an iterator is initiated, do we need to continue passing the part handle and, perhaps, neighbor-part id, to the iterator, or can the iterator "remember" these just as it remembers entity type and topology? If we can re-use the iMesh getNext, reset and end, we can remove six functions from Carl's sixteen functions, bringing the count back to his original estimate of ten. - Which functions should operate on single parts and/or entities, and which should have array versions (function_name and function_nameArr)? I included both wherever we had a comment "Need single-part and array-part versions" in the combined document, but I won't be offended if we remove some of the functions. - Since global IDs for entities are not used anywhere in the interface, I propose their generation and use should be a service on top of the interface rather than part of the interface. What say you?

7 15

Re: Assignments for parallel interface
by Devine, Karen D 04 Feb '08

04 Feb '08

Hi, Carl. Thanks for your comments. Please see my follow-ups below. If I don't provide a follow-up, I either don't know the best answer or I generally agree and will update the document appropriately. (You'll have to guess which case is true. :) Karen On 2/4/08 12:42 PM, "Carl Ollivier-Gooch" <cfog(a)mech.ubc.ca> wrote: > Devine, Karen D wrote: >> In the last phone conf, someone suggested that we put together a draft of >> the syntax for the capabilities on which we agreed. So I added syntax from >> Carl's, Onkar's and Tim's past emails to our Combined document. For >> readability, I turned the file into a C header file; the discussions from >> the Combined document are now comments in the C file. The file is attached. > > I actually got this nearly done a week ago, but didn't get it sent until > now... > >> Carl: >> - Should functions prefix_getEntSets and prefix_getNumEntSets be added to >> Carl's sixteen functions? They appear to be analogous to >> prefix_getEntities and prefix_getNumOfType. > > This is probably a good idea, for those who both (a) choose to program > on a part-by-part basis rather than process-by-process and (b) choose to > use sets for various things. I've got no objection. > >> - Review in detail the Part Functionality section. > > createPart: How do we envision the adjudication of global part ID's? > Does part creation imply a request to some oracle for a part ID? Does > each process have its own unique space for part ID's? Using n bits for > process rank, m bits for local ID would create part ID's that were easy > to create and interpret... with cleverness, we could work things so > that n+m <= 31 (a billion part ID's...) so that this all looks like an > int, and we don't have to mess with the sign bit. I think part IDs should always travel with partition handles, since part IDs often refer to off-process parts. So they have to be unique only with respect to a partition. Then your scheme would allow a billion part IDs per partition. Very cool!!! I hadn't thought about how to assign the part IDs, though; I like to push everything into the implementation. :) > > destroyPart: Setting handles to NULL on destruction assumes implicitly > that no implementation will ever use 0 as a valid part handle. While it > might be a good idea to invalidate the handle, the invalid value may > have to be implementation dependent. > > getPartIdsFromPartHandle: The part handle presumably must be local here; > this should be added to the comment. Throughout the interface, I assumed part handles were valid only locally. I'll make that clearer in the comments. > > getNumPartNbors/getPartNbors: communicating or pre-computed? If the > latter, how/when are these updated? And again, how are we defining > which parts are neighbors? My comments for getNumPartNbors/getPartNbors failed to mention that entity type is an input. From our phone conversations, we decided Part A "neighbors" Part B if A and B share entities of a given dimension; functions computing part neighbors will accept an entity type (possibly iBase_ALL_TYPES). I'll update the comments. I don't know the answer on the communicating vs. pre-computed question. Whatever we decide, we'll have to document it. This question seems related to getting adjacency info at part boundaries. > > Part and part bdry iterators: I don't have (at least at this point) a > firm opinion about whether we should have separate functions to > increment, reset, and destroy these iterators. It's pretty easy to > argue that, semantically, the iterator should know how to do those > things for itself, and that it isn't too hard for an implementation to > handle that internal state. > > -------------------------- > > One entity-related comment: > > getEntityOwner: I disagree with Karen. I think it -does- make sense to > be able to determine the owner of an arbitrary entity. We have agreed > that exactly one part will have ownership of an entity in the > right-to-modify sense, and IMO this is what this call should return. We > may also need functionality to identify the entities that were used in > -determining- the partition, but I see that as a different issue. You are correct that every entity needs an "owner" with right-to-modify. We had said that part assignments computed with respect to a set of entities would induce part assignments to lower-dimensional entities in an implementation-dependent fashion. That is, if a partition is computed with respect to regions, queries about ownership of faces and vertices are valid. Is it more correct to say that a partition should be able to induce part assignments on all entities adjacent (in any manner) to entities involved in computing the partition? I'm OK with that, but we should state it clearly. An error would be returned, then, only if an entity that isn't adjacent to any of the partitioned entities is requested. For example, say entity set A contains a mesh and entity set B contains another mesh completed separate from set A. If we partition set A, and then call getEntityOwner with partition A and an entity in set B, getEntityOwner returns an error. Does that seem correct? > >> ------------------------ >> Questions for everyone: >> >> - When we specify a part handle, must we always also specify a partition >> handle? That is, should a part be uniquely identified by the pair >> (partition_handle, part_handle), just as entities are uniquely identified by >> (mesh_instance, entity_handle)? "No" is the most convenient answer to this >> question; if parts are uniquely identified by only the part handle, we can >> do the entity-set argument overloading that we discussed at bootcamp. We >> can also remove the partition_handle argument from many of the functions. I >> had previously argued for use of the tuple, but many things would be easier >> if we didn't need it. Can all the implementations generate part handles >> that are unique within an iMesh_Instance, rather than within a partition? > > My concern here is that this is a potentially difficult constraint for > implementations that want to use integer part handles. Somehow, they > have to coordinate all partitions active in an instance to produce > unique integer ID's. Also, part handles will have to be unique from set > handles as well. (Again, not a problem with pointers, but potentially a > problem for integer handles.) I'm not saying that I expect to implement > handles as ints, but it's not an unreasonable choice, and we want to be > careful not to exclude it without good reason. Yes, I go both ways on this issue. It is more object-oriented and extensible to use the tuples. But it's just so darned convenient to allow the part handles to be interchangeable with set handles. > >> - With iterators over parts or part boundaries, is it necessary to have >> separate interface functions for getNext, reset, and end, or will the >> standard iMesh functions for getNext, reset, and end suffice? That is, once >> an iterator is initiated, do we need to continue passing the part handle >> and, perhaps, neighbor-part id, to the iterator, or can the iterator >> "remember" these just as it remembers entity type and topology? If we can >> re-use the iMesh getNext, reset and end, we can remove six functions from >> Carl's sixteen functions, bringing the count back to his original estimate >> of ten. > > See above. > >> - Since global IDs for entities are not used anywhere in the interface, I >> propose their generation and use should be a service on top of the interface >> rather than part of the interface. What say you? > > No objection. > > Carl > > -- > ------------------------------------------------------------------------ > Dr. Carl Ollivier-Gooch, P.Eng. Voice: +1-604-822-1854 > Associate Professor Fax: +1-604-822-2403 > Department of Mechanical Engineering email: cfog(a)mech.ubc.ca > University of British Columbia http://www.mech.ubc.ca/~cfog > Vancouver, BC V6T 1Z4 http://tetra.mech.ubc.ca/ANSLab/ > ------------------------------------------------------------------------ > > >

1 0

Phone conference reminder
by Devine, Karen D 04 Feb '08

04 Feb '08

Reminder: The parallel interface subcommittee (Tim, Jason, Mark M., Lori, Carl, Karen, Vitus, Mark S., Onkar, and other appropriate RPI people) will meet on Tuesday, 2/5, at 12:30 PST. Please be ready to discuss your assignments regarding the draft interface. Email any comments that you would like people to consider before the meeting. Karen

1 0