I agree with Darius's suggestions - an easy test using proc_null is to remove the global mutex from the cs_enter/exit to see if that's the dominant source of the cost. (I'm also not exactly sure how the -r changes the test).
I tried removing the global mutex last night, but the test segfaults. I'll need to dig in more to figure out how exactly I can trick the "no-lock" case. Btw, "-r" is the round-robin allocation of processes, i.e., each core gets a different process. The default run (without -r) is for all processes on one core.
With the global mutex in place, if the threads that don't have the
lock are busy waiting on it, that traffic might look like a L2 miss.
I think pthread lock waits are blocking for events, not polling. I'll check.
The fine grain locks should eliminate that contention. As Darius
notes, it would be interesting to know what other sources of L2
misses there might be (e.g., reading from the shared MPIR_Process
structure).
From my tests with blocking send, it looks like that's the only source. The non-blocking isend model is a lot more complicated to find the cache misses, but given that the number of cache misses is high for the blocking version as well, the lock seems to be a major, if not the only, contributor to the cache misses.
That's not to say that I'll not try out the non-blocking version, but that'll take some more time. Are we having the telecon with IBM this Tuesday?
-- Pavan
--
Pavan Balaji