On 05/18/2013 03:41 PM US Central Time, Fab Tillier wrote:
I wonder what other implementations would do if you initiated an MPI_Bcast or similar on such a communicator from the delete callback (say you wanted to notify everyone the attribute was being deleted, or whatnot). I'm reluctant to special case certain functions as working on a freed communicator, but perhaps the structure of our implementation could change to not invalidate the communicator.
This should be valid, IMO. See below.
Yes, the test relies on that. The problem is MPI_Attr_get returning an error when called from delete_fn, presumably due to eager invalidation of the communicator.
The issue here is that the communicator is reference counted. It is deleted when the reference count reaches zero (at which point it is no longer valid, as communicator lookup checks that the reference count is non-zero). Having the reference count bounce through zero (to initiate destruction), then back to 1 so that the attribute delete callbacks can access it, then back to zero, is really messy, especially if the delete callbacks were to initiate I/O operations.
In MPICH, here's what we do: 1. Run the user callback. If it returned an error, return that error to the user. 2. Decrement the communicator ref-count. 3. If the ref-count has reached zero, free the attributes and then free the communicator. If the user did a MPI_BCAST, there's no problem since after the callback, the communicator ref-count does not include the BCAST. On the other hand, if the user did an MPI_IBCAST, then the ref-count will not touch zero, so the communicator is not actually deleted internally. The attribute might not actually get deleted if the ref-count didn't reach zero, but you can't access them anyway, since the comm handle if not valid. They'll eventually get freed when the ref-count reaches zero. -- Pavan -- Pavan Balaji http://www.mcs.anl.gov/~balaji