Darius, I ran a bunch of tests last night. I didn't send out the details in the previous email, as they were too many cases. Here are the details for the ones you asked for:
Are the majority of the cache misses due to the global lock (i.e., MPIU_THREAD_SINGLE_CS_ENTER/_EXIT)? Try commenting that macro out then run the test with blocking communication and proc_null (to avoid the need for a lock).
Note that the blocking send call also calls the lock. The lock is released only when the call actually blocks for an event. So there are some cache misses even in this case, though lesser than in the non-blocking version. I also tried getting rid of the lock and running (an incorrect version of) the test. But that seg faults.
Another thing to check is to instrument MPICH2 to read only the L2 cache misses around the lock/unlock operations, then read the total L2 cache misses for the MPI calls. I'm just interested in seeing how bad the rest of the code is wrt L2 cache misses.
Yup, tried that too. Well, in the blocking case I got MPI_Send to return immediately after the lock/unlock and there was not much change in the cache misses. So, it looks like the lock/unlock is the major contributor to the cache misses. For the MPI_Isend case, that's more difficult to figure out. -- Pavan -- Pavan Balaji http://www.mcs.anl.gov/~balaji