The nesting functionality makes use of thread-private storage, which involves potentially a routine call (for the thread-private storage). I agree that it shouldn't cost much. But it is something that's needed only if MPI routines call MPI routines in the implementation; if instead MPI routines only called internal routines, the entire nesting block could be eliminated. Bill On Jan 31, 2008, at 2:49 PM, Pavan Balaji wrote:
Bill,
The nesting functionality is just regular increment (not even atomic increment). So, do we expect it to have too much performance impact? I tried this on breadboard and the performance difference was within the variation due to noise. I'm trying it on BG/P now.
-- Pavan
On 01/31/2008 01:17 PM, William Gropp wrote:
No, something simpler. Just ignore the nest count code entirely; we only use it for error handling and perhaps for choosing whether to get the lock in nested MPI calls, of which these examples have none (or maybe one, if there's a Barrier somewhere). That will tell us how much the get thread specific etc costs. Its probably not much, though it might be a non-trivial part of the overall latency. Bill On Jan 31, 2008, at 1:03 PM, Pavan Balaji wrote:
Bill,
We know that holding the global lock serializes the MPI code. The only other questions are (1) how much does the nest count cost, (2) how much would fine grain locking/atomic updates cost (e.g., for the communicator reference count update in an uncompleted Irecv), and (3) how narrow could a global lock be for the necessary updates instead of fine-grain? We can answer (1) by temporarily changing the definition of the nest count macros and rerunning these tests.
Do you mean acquire the global lock in the nest count increment?
-- Pavan
-- Pavan Balaji http://www.mcs.anl.gov/~balaji
William Gropp Paul and Cynthia Saylor Professor of Computer Science University of Illinois Urbana-Champaign
-- Pavan Balaji http://www.mcs.anl.gov/~balaji
William Gropp Paul and Cynthia Saylor Professor of Computer Science University of Illinois Urbana-Champaign