May 2013 - discuss - mailman3.cels.anl.gov

Re: [mpich-discuss] [PATCH v2] test: add attrdeleteget, MPI_Attr_get called from delete_fn
by Jeff Squyres (jsquyres) 21 May '13

21 May '13

FWIW: I don't know if attributes/callbacks are the purview of hybrid, but then again, I don't know exactly who's purview they fall under... On May 20, 2013, at 6:54 PM, Fab Tillier <ftillier(a)microsoft.com> wrote: > Pavan Balaji wrote on Mon, 20 May 2013 at 15:45:35 > >> >> On 05/20/2013 11:16 AM US Central Time, Fab Tillier wrote: >>> In MPI-3, section 8.7.1 (page 363) states that delete callbacks are >>> invoked on MPI_COMM_SELF before anything else happens, and that >>> MPI_FINALIZED will return false from any of these delete callbacks. >>> There's no mention about attribute callbacks on other built-in >>> communicators, and no mention of behavior for built-in types. Likely >>> they should be invoked too for consistency's sake. >> >> I'd think not. For built-in communicators, it makes sense since they >> cannot be freed by the user. For other communicators, it's the user's >> responsibility to free them. So the callbacks should be triggered at >> that time. > > I meant for all built-in objects (communicators and types), not just MPI_COMM_SELF. I agree that any user-allocated objects are the users's responsibility (just say no to automatic garbage collection). > >>> What's interesting is that this section states that there is an >>> ordering enforced (reverse from the order in which they are set), >>> while MPI_COMM_FREE (page 248) states that the callbacks are invoked >>> in arbitrary order. >>> >>> We should probably make that consistent, and to preserve backward >>> compatibility, the "arbitrary order" may have to become stricter for >>> implementations to match the finalize behavior. >> >> Gross. Yes, I agree it should be consistent. > > I added some comments about the above to the ticket (https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/374) as well as raised an issue about the keyval parameter value to the callbacks when the keyval has been freed by the user. > > Anyway, I think we're all converging on our opinions, which is always a good thing. > > Cheers, > -Fab -- Jeff Squyres jsquyres(a)cisco.com For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/

1 0

Re: [mpich-discuss] [PATCH v2] test: add attrdeleteget, MPI_Attr_get called from delete_fn
by Fab Tillier 20 May '13

20 May '13

Pavan Balaji wrote on Mon, 20 May 2013 at 15:45:35 > > On 05/20/2013 11:16 AM US Central Time, Fab Tillier wrote: >> In MPI-3, section 8.7.1 (page 363) states that delete callbacks are >> invoked on MPI_COMM_SELF before anything else happens, and that >> MPI_FINALIZED will return false from any of these delete callbacks. >> There's no mention about attribute callbacks on other built-in >> communicators, and no mention of behavior for built-in types. Likely >> they should be invoked too for consistency's sake. > > I'd think not. For built-in communicators, it makes sense since they > cannot be freed by the user. For other communicators, it's the user's > responsibility to free them. So the callbacks should be triggered at > that time. I meant for all built-in objects (communicators and types), not just MPI_COMM_SELF. I agree that any user-allocated objects are the users's responsibility (just say no to automatic garbage collection). >> What's interesting is that this section states that there is an >> ordering enforced (reverse from the order in which they are set), >> while MPI_COMM_FREE (page 248) states that the callbacks are invoked >> in arbitrary order. >> >> We should probably make that consistent, and to preserve backward >> compatibility, the "arbitrary order" may have to become stricter for >> implementations to match the finalize behavior. > > Gross. Yes, I agree it should be consistent. I added some comments about the above to the ticket (https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/374) as well as raised an issue about the keyval parameter value to the callbacks when the keyval has been freed by the user. Anyway, I think we're all converging on our opinions, which is always a good thing. Cheers, -Fab

1 0

Re: [mpich-discuss] [PATCH v2] test: add attrdeleteget, MPI_Attr_get called from delete_fn
by Pavan Balaji 20 May '13

20 May '13

On 05/20/2013 11:16 AM US Central Time, Fab Tillier wrote: > In MPI-3, section 8.7.1 (page 363) states that delete callbacks are > invoked on MPI_COMM_SELF before anything else happens, and that > MPI_FINALIZED will return false from any of these delete callbacks. > There's no mention about attribute callbacks on other built-in > communicators, and no mention of behavior for built-in types. Likely > they should be invoked too for consistency's sake. I'd think not. For built-in communicators, it makes sense since they cannot be freed by the user. For other communicators, it's the user's responsibility to free them. So the callbacks should be triggered at that time. > What's interesting is that this section states that there is an > ordering enforced (reverse from the order in which they are set), > while MPI_COMM_FREE (page 248) states that the callbacks are invoked > in arbitrary order. > > We should probably make that consistent, and to preserve backward > compatibility, the "arbitrary order" may have to become stricter for > implementations to match the finalize behavior. Gross. Yes, I agree it should be consistent. -- Pavan Balaji http://www.mcs.anl.gov/~balaji

1 0

Re: [mpich-discuss] [PATCH v2] test: add attrdeleteget, MPI_Attr_get called from delete_fn
by Fab Tillier 20 May '13

20 May '13

Pavan Balaji wrote on Mon, 20 May 2013 at 09:03:04 > > On 05/20/2013 10:49 AM US Central Time, Fab Tillier wrote: >> Pavan Balaji wrote on Mon, 20 May 2013 at 06:57:42 >> >>>> And what about the non-deterministic effects that I called out in >>>> the email back in 2011 -- are those allowed, and if so, do we >>>> need to say that they're non-deterministic? ...and so on.- >>> >>> Right, we did talk about the non-determinism on this email thread >>> before you joined in, but we concluded that there's only one >>> sensible way. Of course, that should be clarified in the standard. >>> But it's not a major change in implementations, I think. >> >> One thing we didn't discuss yet is whether the callbacks are called >> during MPI_Finalize, for attributes assigned to built-in >> communicators and/or types, and what operations are allowed therein. >> This makes it that much more interesting to do non-blocking I/O from >> the callback, since technically the application has called >> MPI_Finalize. However, if we go with the "as if MPI_*_DELETE_ATTR was >> called before the call, then anything should be valid, and we should >> have an advice to users regarding Finalize. > > Are we supposed to call those callbacks at Finalize? MPI is not > required to clean up the comm resources, is it? In MPI-3, section 8.7.1 (page 363) states that delete callbacks are invoked on MPI_COMM_SELF before anything else happens, and that MPI_FINALIZED will return false from any of these delete callbacks. There's no mention about attribute callbacks on other built-in communicators, and no mention of behavior for built-in types. Likely they should be invoked too for consistency's sake. What's interesting is that this section states that there is an ordering enforced (reverse from the order in which they are set), while MPI_COMM_FREE (page 248) states that the callbacks are invoked in arbitrary order. We should probably make that consistent, and to preserve backward compatibility, the "arbitrary order" may have to become stricter for implementations to match the finalize behavior. Cheers, -Fab

1 0

Re: [mpich-discuss] [PATCH v2] test: add attrdeleteget, MPI_Attr_get called from delete_fn
by Pavan Balaji 20 May '13

20 May '13

On 05/20/2013 10:49 AM US Central Time, Fab Tillier wrote: > Pavan Balaji wrote on Mon, 20 May 2013 at 06:57:42 > >>> And what about the non-deterministic effects that I called out in >>> the email back in 2011 -- are those allowed, and if so, do we >>> need to say that they're non-deterministic? ...and so on.- >> >> Right, we did talk about the non-determinism on this email thread >> before you joined in, but we concluded that there's only one >> sensible way. Of course, that should be clarified in the standard. >> But it's not a major change in implementations, I think. > > One thing we didn't discuss yet is whether the callbacks are called > during MPI_Finalize, for attributes assigned to built-in > communicators and/or types, and what operations are allowed therein. > This makes it that much more interesting to do non-blocking I/O from > the callback, since technically the application has called > MPI_Finalize. However, if we go with the "as if MPI_*_DELETE_ATTR was > called before the call, then anything should be valid, and we should > have an advice to users regarding Finalize. Are we supposed to call those callbacks at Finalize? MPI is not required to clean up the comm resources, is it? -- Pavan -- Pavan Balaji http://www.mcs.anl.gov/~balaji

1 0

Re: [mpich-discuss] [PATCH v2] test: add attrdeleteget, MPI_Attr_get called from delete_fn
by Fab Tillier 20 May '13

20 May '13

Pavan Balaji wrote on Mon, 20 May 2013 at 06:57:42 >> And what about the non-deterministic effects that I called out in the >> email back in 2011 -- are those allowed, and if so, do we need to say >> that they're non-deterministic? ...and so on.- > > Right, we did talk about the non-determinism on this email thread before > you joined in, but we concluded that there's only one sensible way. Of > course, that should be clarified in the standard. But it's not a major > change in implementations, I think. One thing we didn't discuss yet is whether the callbacks are called during MPI_Finalize, for attributes assigned to built-in communicators and/or types, and what operations are allowed therein. This makes it that much more interesting to do non-blocking I/O from the callback, since technically the application has called MPI_Finalize. However, if we go with the "as if MPI_*_DELETE_ATTR was called before the call, then anything should be valid, and we should have an advice to users regarding Finalize. Boy, did this got messy quickly... -Fab

1 0

Re: [mpich-discuss] [PATCH v2] test: add attrdeleteget, MPI_Attr_get called from delete_fn
by Pavan Balaji 20 May '13

20 May '13

On 05/20/2013 08:55 AM US Central Time, Jeff Squyres (jsquyres) wrote: > On May 20, 2013, at 9:52 AM, Pavan Balaji <balaji(a)mcs.anl.gov> wrote: > >> As nasty as it is, apart from implementation difficulty, what's the >> problem with it in the standard (apart from wording clarification)? > > The standard doesn't say what you're allowed (or not allowed) to do at all. Yes, I'm looking at this as wording clarification. Right now, since the standard doesn't disallow it, applications use it as if it's allowed. > For example, can you call MPI_SEND from within an attribute call back? Or MPI_COMM_SPAWN? Yes to both. > And what about the non-deterministic effects that I called out in the > email back in 2011 -- are those allowed, and if so, do we need to say > that they're non-deterministic? ...and so on.- Right, we did talk about the non-determinism on this email thread before you joined in, but we concluded that there's only one sensible way. Of course, that should be clarified in the standard. But it's not a major change in implementations, I think. -- Pavan - Pavan Balaji http://www.mcs.anl.gov/~balaji

1 0

Re: [mpich-discuss] [PATCH v2] test: add attrdeleteget, MPI_Attr_get called from delete_fn
by Jeff Squyres (jsquyres) 20 May '13

20 May '13

On May 20, 2013, at 9:52 AM, Pavan Balaji <balaji(a)mcs.anl.gov> wrote: > As nasty as it is, apart from implementation difficulty, what's the > problem with it in the standard (apart from wording clarification)? The standard doesn't say what you're allowed (or not allowed) to do at all. For example, can you call MPI_SEND from within an attribute call back? Or MPI_COMM_SPAWN? And what about the non-deterministic effects that I called out in the email back in 2011 -- are those allowed, and if so, do we need to say that they're non-deterministic? ...and so on. -- Jeff Squyres jsquyres(a)cisco.com For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/

1 0

Re: [mpich-discuss] [PATCH v2] test: add attrdeleteget, MPI_Attr_get called from delete_fn
by Pavan Balaji 20 May '13

20 May '13

On 05/20/2013 08:39 AM US Central Time, Jeff Squyres (jsquyres) wrote: > 1. What exactly is allowed to be done in attribute callbacks (or any > callbacks, actually) -- to include whether you can GET_ATTR, etc. I might have missed this at the Forum. This issue came up within MPICH some time back. Right now, since the standard is so restrictive, we had to make a bunch of our functions reentrant-safe to allow for this case (though I'm sure we missed a few). As nasty as it is, apart from implementation difficulty, what's the problem with it in the standard (apart from wording clarification)? -- Pavan -- Pavan Balaji http://www.mcs.anl.gov/~balaji

1 0

Re: [mpich-discuss] [PATCH v2] test: add attrdeleteget, MPI_Attr_get called from delete_fn
by Jeff Squyres (jsquyres) 20 May '13

20 May '13

On May 18, 2013, at 4:41 PM, Fab Tillier <ftillier(a)microsoft.com> wrote: > I agree the standard should clarify (I'll take a look at the ticket you created), and I'd like to see a clarification of the behavior with respect to the attribute delete callbacks. Specifically, the standard does not state explicitly whether the attributes are destroyed within the call to MPI_COMM_FREE, or only once all references on the communicator are released. Sorry for jumping in late... I actually brought this exact issue up with the Forum a few years ago. There's a few larger issues: 1. What exactly is allowed to be done in attribute callbacks (or any callbacks, actually) -- to include whether you can GET_ATTR, etc. 2. Is the Fortran attribute extra state parameter passed by value or by reference? Sadly, it looks like I only made a ticket on #2: https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/159 But for #1, there was some web discussion about this, but the issue never really went anywhere. Here's the first mail about it, from back in 2011: http://lists.mpi-forum.org/mpi-forum/2011/04/1035.php -- Jeff Squyres jsquyres(a)cisco.com For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/

1 0