- itaps-parallel - mailman3.cels.anl.gov

Comments on parallel stuff so far.
by Carl Ollivier-Gooch 30 Oct '07

30 Oct '07

Hello, all. For a while there, things were moving too fast for me to keep up. :-) Now that I've read through all the email, I've got some comments, which I'm sure comes as a shock to all of you. :-) First, some comments on the long thread about (essentially) using existing set functionality versus not: Part of the problem may be that we currently have no clearly-defined top-level partition. As a result, we don't have a way to create a new part (and keep track of it). We also don't have a way to segregate two separate partitionings. Some of Tim's comments (for example, about the use of a tag convention to identify which set handles are parts, and presumably to use the value of that tag to identify which partition they belong to) may stem from the need to carry multiple partitions at once, for which there's currently no proposal about functionality. (Question as an aside for Tim: Are you thinking about different partitionings of the same mesh, or different meshes on the same domain with different partitionings? If the former, are the partitionings nested, or not necessarily?) We should at least consider having a "partition" object that holds all the top-level data, including handles for all the parts, part-processor mapping, maybe an MPI communicator for the partition, etc. Then we can create/destroy parts (including registering them with the partition; obviously creating the partition with some number of parts is a great piece of common shorthand), assign them to processors, report on the mapping of parts to processors, etc, which is functionality that may be (as I read it) only partially proposed at this point. I recognize that some of these functions may turn out to be trivial, but conceptually they're still worth considering because it's functionality that pretty clearly needs to be there. Obviously some of this functionality is already proposed in various pieces that are currently under discussion. This level also seems to be the logical place for global calls, like those Tim proposes, as well as some of those in the RPI document (like getTotNumOfParts, getNumOfPartsArr, etc). An advantage to this is that we can then use a partition handle of some sort in calls like getGlobalPartOnProc (one of numerous examples) that require the ability to distinguish between multiple partitions when multiple partitions exist. Also, we need to be able to identify which partition a part belongs to. This can be a function or a tag convention; I'm thinking of it just as functionality at this point; whether it eventually is a function or a conventional tag or something else is a question for a bit later. Overall, what I'm proposing looks more or less like the following: Partition level: Data: . IDs of all parts (every part belongs to a partition, like every serial set belongs to a root set) . mapping of parts to processors . the existence of a list of processors implies an MPI communicator, presumably; whether/how to expose this communicator to apps? New Functionality: . getTotNumOfParts . getNumOfPartsArr . global<->local part ID mapping and part<->processor mapping, supplementing or superceding what's currently proposed (getPartIDArrOnProc, isPartOnProc, getGlobalPartOnProc, getLocalPartOnProc) . addPartOnProc, rmvProcOnPart . createPart, destroyPart . getNumEntsPar, getNumEntSetsPar, getNumEntsProcPar, getNumEntSetsProcPar . getTagsPar, getTagsProcPar, get by tag . global reduce on tags . get/setGlobalIDSize Processor level: Data: . IDs of all parts on that processor . iMesh instance New Functionality: . getNumOfPartsOnProc . getPartsOnProc (all part handles) . part iterator (could be replaced by mapping local ID to part handle) . local ID <-> (local) part handle . iMesh handle . what partition do I belong to? Part level: Data: . owned entities . copies of non-owned interpart bdry entities New Functionality: . getNghbPartIds (if this returns global part IDs (proc rank + local ID), then get getNghbProcRanks is redundant) . part bdry iterators (both single and array versions) . part bdry size info . getOwnerOfEnt, getCopiesOfEnt, getCopyOfEnt, getNumOfCopiesOfEnt . entity categorization within part . addCopyOfEnt, rmvCopyOfEnt . disambiguate ownership of shared entity . what partition do I belong to? . get/set/createGlobalID[Arr] . getEntHandle[Arr] from global ID ====================================================================== On the question of how to add entities to parts, I think we need to examine the use cases for adding entities to parts. I can think of several scenarios: 1. A new entity has been created by a modify operation and needs to be associated with a part. 2. An entity is being migrated from one part to another (small one-to-one migration due to change of ownership; data is almost certainly moving from one part to another). 3. A copy is being created (depending on the implementation, some data may have to copied from one part to another). 4. A new partition has been created and parts must be populated (massive all-to-all migration; vast amounts of data flying everywhere). That list probably isn't exhaustive, but it'll do for now. Of those four, I see one, possibly two, that -don't- involve parallel data transfer. Also all (potentially) will affect not only ownership of the primary entities partitioned, but also the part boundaries. At the very least, we need to be very clear that we're doing things that involve parallel communication here. One possible way to handle this: o We could handle scenario 1 cleanly with set syntax, with the -specific- caveat clearly documented that this function is for use only for local entities. This is enforceable by requiring the entity args to be handles (necessarily local to the processor, though not the part) and mandating an error return when trying a back-door migration between parts on a single processor. This is certainly possible, but is it common enough and unambiguous enough to justify doing it, especially in the face of a need for a different mechanism for other scenarios? o As I just hinted at, I don't think the current set calls are appropriate for Scenarios 2-4 (and other migration scenarios), because the syntax then hides that there's parallel communication going on (as well as the issue of needing to be able to grab a communicator, possibly). Among other viable approaches, the migration scenarios could be handled by an explicit sendEntToPart/receiveEnts call pair, which I envision using syntax that is qualitatively like this: iPart_sendEntArrToParts(source_part_handle, ent_handle_array, target_part_ID_array, MIGRATE); iPart_sendEntArrToParts(source_part_handle, ent_handle_array, target_part_ID_array, COPY); iPart_receiveEnts; Yes, I can envision potentially sending ghost data using this same approach. And yes, the actual data packaged up and sent from part to part is going to need to be implementation dependent (and possibly application specifiable...). And yes again, sendEntArrToParts(...MIGRATE) should delete local data; when is a good question... No, I don't think single-entity versions of these are likely to be a good idea for scalability reasons. The last call (receiveEnts) would block waiting for data; a broadcast up front is probably needed to make sure every processor knows what messages it has to wait for. Once all the data is received, then the receiveEnts call can also do whatever implementation-dependent things are needed to make sure the interior/bdry/ghost info has been properly updated internally. I like the pair of calls not only because it gives a natural place to update things like part bdry info, but also because it allows at least some latency hiding, which an immediate, one-call migrate functionality doesn't. ================================================================ And now that I've no doubt said something that everyone will agree with and something that everyone will disagree with, I'll move on to some specific comments. :-) RPI (Parts, etc) I had some comments about iProcParts, iPart, iPartMesh that are now covered by the discussion above. part iterator: Natural analog to entities, BUT do we really need this, given that the array of part handles for a given processor is likely to be small enough that we won't mind just grabbing the whole array and iterating outside the interface? getPartMesh: Given that we can call all applicable iMesh functions with a part handle, do you see a role for this other than mesh modification? [Comments on part<->processor mapping subsumed above.] isEntOnPartBdry, isEntOwner, isEntGhost: I think we should combine these into a single call with an enumerated "return" value. Simpler now, and allows for future expansion in the list of properties we care about, if needed. Also, need a way to set/negotiate ownership of shared entities on part boundaries. Tim's stuff: getEntByTypeAndTagPar, getEntSetByTagPar: We don't have this functionality in serial. Maybe we should, but we currently don't. getTag*Operate: Again, we haven't got this in serial. Does the existence of such operations imply that we expect to implement fields as tags? (Because that wasn't what I was assuming about field implementations at all, personally...) Note that I'm not opposed to this sort of global reduction operation, I just wonder whether it'll see use outside of field-like situations. If not, then it should be in parallel fields, not parallel mesh, and usage for fields-implemented-as-tags should be handled there. Global IDs: One question here. As I understand it, the intention that getEntHandle[Arr] will return a local handle (owned, copied, or ghosted) for the global ID if possible, and otherwise give an error. Is that correct? I now return you to your regularly scheduled afternoon. Carl -- ------------------------------------------------------------------------ Dr. Carl Ollivier-Gooch, P.Eng. Voice: +1-604-822-1854 Associate Professor Fax: +1-604-822-2403 Department of Mechanical Engineering email: cfog(a)mech.ubc.ca University of British Columbia http://www.mech.ubc.ca/~cfog Vancouver, BC V6T 1Z4 http://tetra.mech.ubc.ca/ANSLab/ ------------------------------------------------------------------------

3 3

Re: Parallel Interface philosophical comments
by Onkar Sahni 26 Oct '07

26 Oct '07

>> There are multiple things which I want to make clear: >> >> (a) Part is not necessarily an entity-set (some implementations may >> choose >> it to be). Moreover, operations supported on entity-sets do not have to >> work on parts for implementations which do not consider parts as >> entity-sets. > > My understanding from the last boot camp was that we did need to support > current iMesh entity set functions on part handles. I could see in some > cases a particular implementation not supporting a few functions, but on > the other hand, many of the iPart functions in your proposal have direct > analogs in the iMesh entity set functions. I agree that all of mesh-specific functions, like iMesh_getEntities, iMesh_getAllVtxCoords, iMesh_initEntIter etc., will be made compatible/enabled to work with part handles (that will fall under Req. 14 of req. doc. with CVS rev. 1.14). For clarification, by operations on entity sets I meant set-specific operations/functions, like iMesh_addEntToSet, iMesh_rmvEntFromSet, iMesh_subtract, iMesh_intersect, iMesh_unite etc. My understanding, from the last boot camp, is that these functions will not be enabled to accept part-handles. Infact, just wandering why we call, for example, iMesh_subtract and not iMesh_subtractSets? Please let me know the functions (in our proposal) you think have direct-analogs and we will try to look into them (for the ones you mentioned in one of your previous responses, i.e., iProcParts_getPartIdArrOnProc and iProcParts iterator, I do not really see any direct-analogs). >> >> (b) Allowing part-handles to be used along with root-set-handle, >> entity-set-handle etc. is only required for existing iMesh functions >> (not >> for new ones, see next point). Infact, I see a potential problem here, >> consider a specific case where user wants to get entities in an specific >> entity-set for only a specific part; how will we support it? >> > > I'm not sure I understand this statement, and the comment down by your > signature below. Are you saying you're ok supporting current iMesh > functions on part handles, but that maybe future ones might be > unsupported? I meant functions that fall under Req. 14 of req. doc. with CVS rev. 1.14 (assuming those are mesh-specific functions) will be enabled to accept root-set-, entity-set- and part-handles. I am not sure what you mean by future ones, if you are thinking about future releases/versions of our code then specific functions enabled to take part-handles now will be supported later, say, functions iMesh_getEntities, iMesh_getAllVtxCoords, iMesh_initEntIter are ones (say where we agree to allow part-handles) then we will support them in future releases/versions of our code. >> (c) The new interface functions, we need to define for parts (to support >> parallel requirements/functionalities), will take part-handles as input >> and not any set-handles (if some implementation wants to use entity-sets >> for parts they can have part-handle and entity-set-handle as one and the >> same). > > I agree that it will have to be an error if an application passes in a > set handle representing a set and not a part to one of the functions > expecting a part handle. It'll have to be up to the implementation to > decide how to do that (using casting, looking for the tag designating > part-type sets, etc.) I guess I am not sure what you mean by set-handle not representing a part (or part-type sets). In our implementation there will be no set-handle representing a part (or part-type sets) rather there will be part-handle. I think there is a strong distinction between these two things, i.e., set/set-handle and part/part-handle (as a part need not necessarily be a set in an implementation so why call part-type set or set-handle representing a part). - Onkar > > - tim > >> >>> iRel is for relating data between different interfaces. As I >>> understand >>> it, we are building parallel functionality as part of iMesh. >>> Therefore, >>> we shouldn't need to use iRel for this. >>> >>>> There are a number of groups that use classification as the primary >>>> mode >>>> of interaction for doing things similar to how the RPI group does. >>>> Just >>>> as we would not want you to have redo all your parallel work to do >>>> things strictly from a classification perspective, we do not expect >>>> that >>>> the ITAPS parallel interface will require RPI to re-do all its >>>> parallel >>>> adaptive work (years worth of work dating back to the mid-90's when we >>>> were the first to do it) that is being used for SciDAC applications to >>>> fit with a specific one way of doing things. >>>> >>>> By the way, some feel the classification is a simple and powerful way >>>> to >>>> do things and mesh sets are actually substantial extra overhead to a >>>> well defined model of using classification. In my experience, I would >>>> love to have both classification and reverse classification at all >>>> times, since depending on the question being asked, one is a faster >>>> way >>>> to answer it. However, due to the extra data and headache of keeping >>>> things correctly set, most I know tend to focus on one and use the >>>> other >>>> temporary manner when its advantageous. >>>> >>> I'm not advocating that we do things only with reverse classification, >>> I >>> agree we need to be able to go both ways. I'm just arguing that if we >>> do pass part handles through the interface, then we operate on those >>> part handles using existing functions, rather than make new ones that >>> do >>> virtually the same thing. >> >> For existing functions we can support compatibility with part-handles >> but >> main concern is for new ones (which doesn't do existing/same things). >> >> - Onkar >> >> >>> - tim >>> >>>> Mark >>>> >>>> Tim Tautges wrote: >>>>> >>>>> Onkar Sahni wrote: >>>>>> Hi all, >>>>>> >>>>>> Here are some of my questions and comments on ITAPS parallel >>>>>> interface: >>>>>> >>>>>> 1) Is there a difference between "instance" and "handle" in ITAPS >>>>>> terminology (or are there any definition for these)? (I may have >>>>>> missed it >>>>>> if this was resolved/defined in previous ITAPS interfaces like >>>>>> iMesh). >>>>>> >>>>> We use the generic term "handle" as an identifier for entities, >>>>> entity >>>>> sets, the interface itself, and tags. In our C headers, these are >>>>> typedef'd to void*. There are also iterators, which we haven't >>>>> really >>>>> called handles, but could be referred to that way. >>>>> >>>>>> 2) For situations where we want to support multi-parts per >>>>>> processor/process: >>>>>> (a) How do we want to distinguish between processor/process level >>>>>> functionalities against part level ones (keeping user- and >>>>>> implementation-convenience and efficiency in mind)? Ones which would >>>>>> fall >>>>>> under processor level are, like list of parts on processor, MPI_Rank >>>>>> etc., >>>>>> whereas ones which would fall under part level are, like neighboring >>>>>> parts >>>>>> (not procs.) etc. Further, there could be ones where users wants to >>>>>> get/iterate mesh entities (and get adjacencies) at a processor >>>>>> and/or >>>>>> part >>>>>> level. >>>>> Processor-level queries are on the iMesh instance, part-level queries >>>>> are on the entity set representing the part. Our iterator functions >>>>> operate on both (with processor-level queries simply using the root >>>>> set). >>>>> >>>>>> (b) Assuming a part-handle is supported, how can user/application >>>>>> get >>>>>> access to it (say on a processor where part resides)? Basically will >>>>>> there >>>>>> be a way to directly access the part-handle (like part iterator at >>>>>> iProcParts level) or user has to access it through tags/tag names, >>>>>> in >>>>>> which case user has to figure out what is the tag/tag name for a >>>>>> particular part (and who defines these tags/tag names). >>>>> I advocate having both: a conventional tag given to parts, so that >>>>> you >>>>> can find sets that are parts by looking for sets with that tag; and >>>>> also having a convenience function (as part of the parallel >>>>> extensions >>>>> to iMesh) which returns all the parts in the given instance (i.e. on >>>>> the processor). Note an important capability here for this function >>>>> is to pass in an optional tag name/handle indicating the partition >>>>> you're asking about. That allows multiple partitions to co-exist, >>>>> which is important for some applications. >>>>> >>>>>> 3) How general we want to be in interface for different data >>>>>> models/implementations (as in future we may want to incorporate/fit >>>>>> other >>>>>> implementations), especially between serial and parallel data. Two >>>>>> possibilities for data models I can think of are: (a) one (serial) >>>>>> data >>>>>> model is enhanced to store parallel information and (b) serial one >>>>>> is >>>>>> supplemented by a separate parallel one to support parallel >>>>>> functionalities (in possibility (b) all parallel-specific >>>>>> information >>>>>> is >>>>>> stored in a separate data to serial-specific data). We could design >>>>>> interface that would fit both possibilities, this is the reason why >>>>>> we >>>>>> included iPartMesh (section 3.4) level functions which are specific >>>>>> to >>>>>> parallel functionalities in mesh (and passing handle to serial data >>>>>> in >>>>>> these interface will not work for second possibility (b)). I do not >>>>>> know >>>>>> how much relevant is this. >>>>> I think it's very important to put the parallel data into our >>>>> existing >>>>> data models, or extend the data model if that can't be done (which I >>>>> doubt, since I've already embedded this information in our current >>>>> data model). This is important because it allows us to use current >>>>> tools to examine parallel data. Note also that one may sometimes be >>>>> examining parallel data in a serial environment (e.g. running a >>>>> partitioner in serial), so we shouldn't restrict accessing parallel >>>>> data to a parallel environment. >>>>> >>>>> - tim >>>>> >>>>>> Let me know if you have questions. >>>>>> >>>>>> Vitus, due to above questions I did not respond to your comments. I >>>>>> do not >>>>>> know if above questions and comments make things clear. If you still >>>>>> have >>>>>> anything specific please let us know. >>>>>> >>>>>> Karen, Is there a way to create an email list for this subcommittee >>>>>> as if >>>>>> we want to add (and remove) people (and it will keep record of all >>>>>> the >>>>>> messages). I have added Ting in this response as she is also working >>>>>> on >>>>>> this stuff from RPI side. >>>>>> >>>>>> Thanks, >>>>>> Onkar >>>>>> >>>>>>> Hi all, >>>>>>> I'd like to re-iterate what I think should be the general >>>>>>> behavior of >>>>>>> functions in the parallel interface. IMO, we should take advantage >>>>>>> of >>>>>>> the simple yet powerful data model we already have for iMesh. That >>>>>>> means using mesh sets for things which are obviously sets, and >>>>>>> using >>>>>>> the >>>>>>> functions which already exist for accessing data in those sets. I >>>>>>> think >>>>>>> it does make sense in certain cases to make convenience functions >>>>>>> in >>>>>>> the >>>>>>> parallel interface for accessing e.g. all the sets which represent >>>>>>> parts >>>>>>> of a partition, on a current processor or globally, but I think we >>>>>>> should minimize these. >>>>>>> Based on the general behavior described above, I think many of >>>>>>> the >>>>>>> functions proposed for global ids and parts/partitions already >>>>>>> exist >>>>>>> in >>>>>>> iMesh. For example, iPart_getGlobalID, iPart_createGlobalID, >>>>>>> iProcParts_getPartIdArrOnProc, iProcParts iterator functions, etc. >>>>>>> >>>>>>> - tim >>>>>>> >>>>>>> -- >>>>>>> ================================================================ >>>>>>> "You will keep in perfect peace him whose mind is >>>>>>> steadfast, because he trusts in you." Isaiah 26:3 >>>>>>> >>>>>>> Tim Tautges Argonne National Laboratory >>>>>>> (tautges(a)mcs.anl.gov) (telecommuting from UW-Madison) >>>>>>> phone: (608) 263-8485 1500 Engineering Dr. >>>>>>> fax: (608) 263-4499 Madison, WI 53706 >>>>>>> >>>>>>> >>>>>> >>>>>> >>>> >>> -- >>> ================================================================ >>> "You will keep in perfect peace him whose mind is >>> steadfast, because he trusts in you." Isaiah 26:3 >>> >>> Tim Tautges Argonne National Laboratory >>> (tautges(a)mcs.anl.gov) (telecommuting from UW-Madison) >>> phone: (608) 263-8485 1500 Engineering Dr. >>> fax: (608) 263-4499 Madison, WI 53706 >>> >>> >> >> >> > > -- > ================================================================ > "You will keep in perfect peace him whose mind is > steadfast, because he trusts in you." Isaiah 26:3 > > Tim Tautges Argonne National Laboratory > (tautges(a)mcs.anl.gov) (telecommuting from UW-Madison) > phone: (608) 263-8485 1500 Engineering Dr. > fax: (608) 263-4499 Madison, WI 53706 > >

3 6

Web archives up
by Tim Tautges 26 Oct '07

26 Oct '07

Hi all, The web archives of this list are up, at http://www-unix.mcs.anl.gov/web-mail-archive/lists/itaps-parallel/maillist.…. I'm at ANL right now, so haven't verified that it's accessible from outside, but it should be. - tim -- ================================================================ "You will keep in perfect peace him whose mind is steadfast, because he trusts in you." Isaiah 26:3 Tim Tautges Argonne National Laboratory (tautges(a)mcs.anl.gov) (telecommuting from UW-Madison) phone: (608) 263-8485 1500 Engineering Dr. fax: (608) 263-4499 Madison, WI 53706

2 2

Phone conference
by Devine, Karen D. 25 Oct '07

25 Oct '07

Everyone is available for our phone conference on Wednesday 10/31 at 9am Pacific, so please add the conference to your calendar. I'll send an agenda soon, with assignments to lead parts of the discussion. I hope this isn't the scariest thing you do on Halloween. :) Thanks. Karen

1 0

[Fwd: Re: Subcommittee for Parallel Interface Definition: Global IDs]
by Tim Tautges 24 Oct '07

24 Oct '07

[Forwarding for archival] -------- Original Message -------- Subject: Re: Subcommittee for Parallel Interface Definition: Global IDs Date: Fri, 19 Oct 2007 11:41:11 -0600 From: Vitus Leung <vjleung(a)sandia.gov> To: Onkar Sahni <osahni(a)scorec.rpi.edu> CC: Devine, Karen D. <kddevin(a)sandia.gov>, Mark Shephard <shephard(a)scorec.rpi.edu>, Kenneth Jansen <kjansen(a)scorec.rpi.edu>, Tim Tautges <tautges(a)mcs.anl.gov>, Carl Ollivier-Gooch <cfog(a)mech.ubc.ca>, Lori A. Diachin <diachin2(a)llnl.gov>, Jason Kraftcheck <kraftche(a)cae.wisc.edu>, Knupp, Patrick <pknupp(a)sandia.gov> References: <C338DFBB.B5CD%kddevin(a)sandia.gov> <49549.128.113.131.145.1192482310.squirrel(a)www.scorec.rpi.edu> Onkar, Overall, the functions look good. However, I wonder if moving some of the functionality from the iProcParts level into the iPart level and removing the iProcParts level would simplify things. Similarly, I wonder if the iPartMesh level should simply be the iMesh level. At a lower level, I found the use of both zero-based and one-based indexing in Subsection 2.1 to be a little confusing. Also, Subsection 3.2.1 is titled "Return Number of Parts on Processor" when the first two of the three functions actually return number of parts on all processors. For iMesh compatibility, the functions in Subsection 3.4 should take the mesh instance as well as the set handle, and second-order adjacencies should be provided above the iMesh level. Best regards, Vitus On Mon, 2007-10-15 at 17:05 -0400, Onkar Sahni wrote: > Hello all, > > I have attached interface functions related to partition model. The > document first provides basic terminology and then provides interface > functions in three levels: > > 1) Processor level (for multi-parts per processor). > 2) Part level (part Id, processor rank etc.). > 3) Part mesh level (entity level part information). > > - Onkar > > > > > Honored Parallel Interface Subcommittee members: > > > > Please note that Step 0 of the Parallel Interface draft is due today. > > > > Step 0: draft what functions are needed in each > > category; due Oct 15. > > Step 1: agree on what functions are needed in each > > category; due Oct. 18. > > Step 2: formalize the syntax for those functions; > > due Oct 25. > > Step 3: review and revise output from Steps 1 and 2, > > and write first draft; due Nov. 16. > > > > Global Queries: Argonne (Tim) > > Global IDs: Sandia (Vitus, Karen) > > Partition Model: RPI (Mark, Ken, Onkar) > > Partitioned Mesh: UBC (Carl) > > > > So we should circulate our draft of functions today and have a lively > > discussion for the rest of the week. > > > > Toward that end, the Global IDs function list is attached (GlobalIDs.txt). > > Please respond to all with your comments. > > > > The original email of tasks (draftzero_assignments.txt) is also attached > > in > > case you already deleted the email of assignments. Let me know if you > > need > > more info. > > > > Karen > > > > > > -- ================================================================ "You will keep in perfect peace him whose mind is steadfast, because he trusts in you." Isaiah 26:3 Tim Tautges Argonne National Laboratory (tautges(a)mcs.anl.gov) (telecommuting from UW-Madison) phone: (608) 263-8485 1500 Engineering Dr. fax: (608) 263-4499 Madison, WI 53706

1 0

[Fwd: Re: Subcommittee for Parallel Interface Definition: Global IDs]
by Tim Tautges 24 Oct '07

24 Oct '07

[Forwarding for archival] -------- Original Message -------- Subject: Re: Subcommittee for Parallel Interface Definition: Global IDs Date: Wed, 17 Oct 2007 11:11:11 -0600 From: Devine, Karen D. <kddevin(a)sandia.gov> To: Carl Ollivier-Gooch <cfog(a)mech.ubc.ca> CC: Onkar Sahni <osahni(a)scorec.rpi.edu>, Mark Shephard <shephard(a)scorec.rpi.edu>, Kenneth Jansen <kjansen(a)scorec.rpi.edu>, Tim Tautges <tautges(a)mcs.anl.gov>, Vitus Leung <vjleung(a)sandia.gov>, Lori A. Diachin <diachin2(a)llnl.gov>, Jason Kraftcheck <kraftche(a)cae.wisc.edu>, Knupp, Patrick <pknupp(a)sandia.gov> Carl: Thank you for your thorough review of extensions needed to support parts within the current interface. To get the discussions going (and remind people to review other's contributions), I'll offer the following questions/comments. First a general question: Is determining whether a handle is an entity set handle or a part handle left to the implementation? Or do we need to specify a convention? > 1. The use of the root set in the serial interface can be viewed as a > convention to allow us to query the whole database (in parallel, all > data in an instance) and an entity set using the same function. Let me make sure I understand. Do you mean that calling, say, iMesh_getNumOfType with the root set returns the global (across all processors) number of entities of a certain type? If so, I'm OK with this -- it fits with the definition of a root set -- but I wondered whether it caused other people heartburn. Lori, for example, said she used one root set per processor that provided local-to-the-processor info. Under this suggestion, local-to-the-processor info would be provided by an iPart, not the root set, right? Thanks again. Karen -- ================================================================ "You will keep in perfect peace him whose mind is steadfast, because he trusts in you." Isaiah 26:3 Tim Tautges Argonne National Laboratory (tautges(a)mcs.anl.gov) (telecommuting from UW-Madison) phone: (608) 263-8485 1500 Engineering Dr. fax: (608) 263-4499 Madison, WI 53706

1 0

[Fwd: Re: Subcommittee for Parallel Interface Definition: Global IDs]
by Tim Tautges 24 Oct '07

24 Oct '07

[Forwarding for archival] -------- Original Message -------- Subject: Re: Subcommittee for Parallel Interface Definition: Global IDs Date: Mon, 15 Oct 2007 17:05:10 -0400 (EDT) From: Onkar Sahni <osahni(a)scorec.rpi.edu> To: Devine, Karen D. <kddevin(a)sandia.gov> CC: Mark Shephard <shephard(a)scorec.rpi.edu>, Kenneth Jansen <kjansen(a)scorec.rpi.edu>, Tim Tautges <tautges(a)mcs.anl.gov>, Vitus Leung <vjleung(a)sandia.gov>, Karen Devine <kddevin(a)sandia.gov>, Carl Ollivier-Gooch <cfog(a)mech.ubc.ca>, Lori A. Diachin <diachin2(a)llnl.gov>, Jason Kraftcheck <kraftche(a)cae.wisc.edu>, Knupp, Patrick <pknupp(a)sandia.gov> References: <C338DFBB.B5CD%kddevin(a)sandia.gov> Hello all, I have attached interface functions related to partition model. The document first provides basic terminology and then provides interface functions in three levels: 1) Processor level (for multi-parts per processor). 2) Part level (part Id, processor rank etc.). 3) Part mesh level (entity level part information). - Onkar > > Honored Parallel Interface Subcommittee members: > > Please note that Step 0 of the Parallel Interface draft is due today. > > Step 0: draft what functions are needed in each > category; due Oct 15. > Step 1: agree on what functions are needed in each > category; due Oct. 18. > Step 2: formalize the syntax for those functions; > due Oct 25. > Step 3: review and revise output from Steps 1 and 2, > and write first draft; due Nov. 16. > > Global Queries: Argonne (Tim) > Global IDs: Sandia (Vitus, Karen) > Partition Model: RPI (Mark, Ken, Onkar) > Partitioned Mesh: UBC (Carl) > > So we should circulate our draft of functions today and have a lively > discussion for the rest of the week. > > Toward that end, the Global IDs function list is attached (GlobalIDs.txt). > Please respond to all with your comments. > > The original email of tasks (draftzero_assignments.txt) is also attached > in > case you already deleted the email of assignments. Let me know if you > need > more info. > > Karen > > > -- ================================================================ "You will keep in perfect peace him whose mind is steadfast, because he trusts in you." Isaiah 26:3 Tim Tautges Argonne National Laboratory (tautges(a)mcs.anl.gov) (telecommuting from UW-Madison) phone: (608) 263-8485 1500 Engineering Dr. fax: (608) 263-4499 Madison, WI 53706

1 0

[Fwd: Re: Subcommittee for Parallel Interface Definition: Global IDs]
by Tim Tautges 24 Oct '07

24 Oct '07

[Forwarding for archival] -------- Original Message -------- Subject: Re: Subcommittee for Parallel Interface Definition: Global IDs Date: Mon, 15 Oct 2007 13:38:25 -0700 From: Carl Ollivier-Gooch <cfog(a)mech.ubc.ca> To: Devine, Karen D. <kddevin(a)sandia.gov> CC: Onkar Sahni <osahni(a)scorec.rpi.edu>, Mark Shephard <shephard(a)scorec.rpi.edu>, Kenneth Jansen <kjansen(a)scorec.rpi.edu>, Tim Tautges <tautges(a)mcs.anl.gov>, Vitus Leung <vjleung(a)sandia.gov>, "Lori A. Diachin" <diachin2(a)llnl.gov>, Jason Kraftcheck <kraftche(a)cae.wisc.edu>, "Knupp, Patrick" <pknupp(a)sandia.gov> References: <C338DFBB.B5CD%kddevin(a)sandia.gov> Hello, all. Here's my current version of what we need to do about the serial interface to deal with partitioned meshes. Most of it is actually pretty easy; the notable exception is adjacencies, where data requested may or may not be available, and we need to provide appropriate feedback to applications. Carl -- ------------------------------------------------------------------------ Dr. Carl Ollivier-Gooch, P.Eng. Voice: +1-604-822-1854 Associate Professor Fax: +1-604-822-2403 Department of Mechanical Engineering email: cfog(a)mech.ubc.ca University of British Columbia http://www.mech.ubc.ca/~cfog Vancouver, BC V6T 1Z4 http://tetra.mech.ubc.ca/ANSLab/ ------------------------------------------------------------------------ -- ================================================================ "You will keep in perfect peace him whose mind is steadfast, because he trusts in you." Isaiah 26:3 Tim Tautges Argonne National Laboratory (tautges(a)mcs.anl.gov) (telecommuting from UW-Madison) phone: (608) 263-8485 1500 Engineering Dr. fax: (608) 263-4499 Madison, WI 53706 Partitioned Meshes Or How to Use Parts in the Current Serial Interface After looking carefully over the current serial interface, I've come to two conclusions: 1. The use of the root set in the serial interface can be viewed as a convention to allow us to query the whole database (in parallel, all data in an instance) and an entity set using the same function. 2. For nearly all functions, we can easily extend that convention to allow part handles a co-equal status with the root set and ordinary entity sets. There are a few exceptions, in my opinion, and also some cases where the semantics of the current interface need extension / reinterpretation. I'm only going to address those cases here. 1. iMesh_getAllVtxCoords, iMesh_getAdjEntities: the in_entity_set argument, when called with a part, indicates whether the -part- owns the vertex / other entity. 2. iMesh_getEntities: As it stands, this should only return owned entities, IMO. We may want to add a flag to allow return of ghosted entities (or, with more nuance, interior, boundary, ghosted, etc, in combinations). In any event, anything that's returned should be returned with a -local- handle. 3. iMesh_getAdjEntities, and other adjacency calls: What to do about adjacencies that aren't in the part? In some cases, that data may be ghosted, and it should be returned if it is. If the data isn't ghosted, then in my view the safest choice is to return a (implementation dependent) null handle and a flag saying that data wasn't returnable (i.e., in_set, not_in_set, not_in_set_and_I_have_no_local_handle). 4. iMesh_getAllVtxCoords: In isolation, this one is pretty safe, because we can interpret this as returning only vertices owned by the part. But currently iMesh_getVtxCoordIndex is supposed to return (a) a list of vertex coordinates for all verts bounding the entities returned by some adjacency call and (b) whether those verts are "in the entity set". The latter is actually easier than the former, since the entities and/or their verts may be non-local and non-ghosted. Our treatment here should be consistent with iMesh_getAdjEntities. In any event, getVtxCoordIndex should be returning local indices. (IMO, part local, not instance local, when called with a part handle.) 5. Set functions. getNumEntSets, genEntSets, addEntSet, rmvEntSet, isEntContained, and isEntSetContained are all perfectly reasonable calls when the entity set that is doing the containing is a part. Most other functions --- including add/rmvEnt and parent-child stuff --- are I think semantic abuse. I'm on the fence about set Boolean ops, just as I was with the root set. 6. Mesh modification. These should, arguably, act on the part, not on the instance, in parallel. After all, if you have multiple parts in an instance, you need to assign new entities to a part at creation time, or you no longer have a partition. IMO, it's more natural to do this all in one go rather than create-then-assign, although implementations can certainly do that internally if they wish. 7. Tags. We can tag sets (including the root set), and we need to be able to tag parts. So we should use the existing interface, and just pass a part handle instead of a set handle. (We could also add another family of functions specifically to tag parts, but I don't think that's necessary. As an aside, we probably are going to need to start making a distinction between groupings of entities that are entity sets and groupings of entities that are parts or instance root-sets. I propose that when we mean all of these things (sets, parts, root sets), we use "collections", and when we mean specifically entity sets, we continue to use "sets". That distinctions may help clean up the semantics of the interface a bit (as well as the documentation). Yes, this means adding two more named opaque types (part handle and collection handle) instead of one, but I think it's a good trade-off for clarity.

1 0

[Fwd: Subcommittee for Parallel Interface Definition: Global IDs]
by Tim Tautges 24 Oct '07

24 Oct '07

[Forwarding for archival] -------- Original Message -------- Subject: Subcommittee for Parallel Interface Definition: Global IDs Date: Mon, 15 Oct 2007 09:13:28 -0600 From: Devine, Karen D. <kddevin(a)sandia.gov> To: Onkar Sahni <osahni(a)scorec.rpi.edu>, Mark Shephard <shephard(a)scorec.rpi.edu>, Kenneth Jansen <kjansen(a)scorec.rpi.edu>, Tim Tautges <tautges(a)mcs.anl.gov>, Vitus Leung <vjleung(a)sandia.gov>, Karen Devine <kddevin(a)sandia.gov>, Carl Ollivier-Gooch <cfog(a)mech.ubc.ca>, Lori A. Diachin <diachin2(a)llnl.gov>, Jason Kraftcheck <kraftche(a)cae.wisc.edu>, Knupp, Patrick <pknupp(a)sandia.gov> Honored Parallel Interface Subcommittee members: Please note that Step 0 of the Parallel Interface draft is due today. Step 0: draft what functions are needed in each category; due Oct 15. Step 1: agree on what functions are needed in each category; due Oct. 18. Step 2: formalize the syntax for those functions; due Oct 25. Step 3: review and revise output from Steps 1 and 2, and write first draft; due Nov. 16. Global Queries: Argonne (Tim) Global IDs: Sandia (Vitus, Karen) Partition Model: RPI (Mark, Ken, Onkar) Partitioned Mesh: UBC (Carl) So we should circulate our draft of functions today and have a lively discussion for the rest of the week. Toward that end, the Global IDs function list is attached (GlobalIDs.txt). Please respond to all with your comments. The original email of tasks (draftzero_assignments.txt) is also attached in case you already deleted the email of assignments. Let me know if you need more info. Karen -- ================================================================ "You will keep in perfect peace him whose mind is steadfast, because he trusts in you." Isaiah 26:3 Tim Tautges Argonne National Laboratory (tautges(a)mcs.anl.gov) (telecommuting from UW-Madison) phone: (608) 263-8485 1500 Engineering Dr. fax: (608) 263-4499 Madison, WI 53706

1 0

[Fwd: Subcommittee assignments for Parallel Interface Definition]
by Tim Tautges 24 Oct '07

24 Oct '07

[Forwarding for archival] -------- Original Message -------- Subject: Subcommittee assignments for Parallel Interface Definition Date: Wed, 03 Oct 2007 14:37:11 -0600 From: Devine, Karen D. <kddevin(a)sandia.gov> To: Tim Tautges <tautges(a)mcs.anl.gov>, Mark Shephard <shephard(a)scorec.rpi.edu>, Kenneth Jansen <kjansen(a)scorec.rpi.edu>, Lori A. Diachin <diachin2(a)llnl.gov>, Vitus Leung <vjleung(a)sandia.gov>, Carl Ollivier-Gooch <cfog(a)mech.ubc.ca>, Karen Devine <kddevin(a)sandia.gov>, Onkar Sahni <osahni(a)scorec.rpi.edu> Honored selectees: As promised, here are the assignments for draft zero of the parallel interface. Our task due by November 16 is to build the interface for Sections 3.2 (Basic Parallelism) and 3.3 (Partitions) of the requirements document. There are four categories of interface needed. - Global queries (including MPI Communicators) - Global IDs - Partition Model - Partitioned Mesh Here's the plan: Step 0: draft what functions are needed in each category; due Oct 15. Step 1: agree on what functions are needed in each category; due Oct. 18. Step 2: formalize the syntax for those functions; due Oct 25. Step 3: review and revise output from Steps 1 and 2, and write first draft; due Nov. 16. Below is an outline of what should be included in each category. The requirement numbers are those used in CVS Version 1.14 of the requirements document, distributed October 2. Global Queries: - Req-5: MPI Communicators We did not agree at the bootcamp whether MPI communicators should be set once or passed to each function. It seems natural that an MPI communicator be associated with a mesh; that is, a mesh would know which processors contained any part of the mesh. Different communicators could be passed to services like load balancing, but in general, a query like "give me the global number of element in this mesh" would use the mesh's communicator. So should we have one function that associates a communicator with a mesh? Or should both a communicator and a mesh be associated with a partition? Or are there cases where separate communicators would be needed for each function? Or...? Also here, we should probably decide what a root set really is in parallel. Should there be one root set per processor, with functions invoked on the root set returning only info on the processor? If so, should there be an additional layer above the root set representing the global mesh? Just for my understanding, is a root set equivalent to a mesh instance? If, say, a multiphysics application has two meshes, does it have two root sets? - Req-6: Global queries Which queries should we support? Should they all be associated with a mesh? - This category also enforces requirements Req-2, Req-3, and Req-4. Global IDs: - Req-7: Global IDs We agreed at bootcamp that global IDs would be represented as an array of integers. We need to define the data types and functions on global IDs, as well as their immutability. Partition Model - We decided to have a separate iPart handle that would be passed to ITAPS functions. We'll also need some functions that return information about a partition. These functions are required by Req-10 to Req-13. Partitioned Mesh - We decided that iPart handles could be passed to many ITAPS functions in place of Entity handles. We need to now examine whether that decision was correct. So for this section, we need to discuss how easy substitution of iPart handles will be and what additional capability will be needed. This section deals with Req-14 to Req-18. OK, now the exciting part -- the part you've all been waiting for -- your assignments! We'll divide and conquer, having each group take care of one category as follows. Global Queries: Argonne (Tim) Global IDs: Sandia (Vitus, Karen) Partition Model: RPI (Mark, Ken, Onkar) Partitioned Mesh: UBC (Carl) By Oct 15, please complete Step 0 and send the output to this distribution list for review and discussion. Thanks. Karen -- ================================================================ "You will keep in perfect peace him whose mind is steadfast, because he trusts in you." Isaiah 26:3 Tim Tautges Argonne National Laboratory (tautges(a)mcs.anl.gov) (telecommuting from UW-Madison) phone: (608) 263-8485 1500 Engineering Dr. fax: (608) 263-4499 Madison, WI 53706

1 0