We know that one source of the performance drop in this example is that there is a common lock. Further, instrumentation is tricky, because in some cases, the cost of the lock operations themselves should be relatively small (if the lock doesn't wait) and timing short durations accurately can be tricky. An alternative, at least as a first step, is to disable to thread-safe version of the operation and measure the difference (this has its own sources of errors, of course). For example, it might be possible to turn off the handling of the nesting count - that would tell us whether we need to switch to a strategy that doesn't use a thread-specific nest count.

One of the reasons for using the single global lock was that the hope was that most of the overhead (for real messages, not zero-byte messages) was in the actual communications, and the one big lock wouldn't be the dominant term. Doing the test with one lock per VC helps test that hypothesis, and may give real applications more of a performance boost than eliminating the global lock. For that test, we should send reasonable-sized messages.

On Jan 30, 2008, at 1:17 PM, Pavan Balaji wrote:

I think we should worry about the locking overhead alone and not dilute the benchmark by adding network communication in it. So, each process sending to itself would be better, IMO.

Btw, there was a suggestion earlier that we should try to place one lock per VC to see if the performance improves. However, I think we are jumping too far ahead with that. Even with a global lock, I would expect that the performance would stay constant (note that ideal behavior is linear increase in messaging rate with number of threads). However, it's dropping currently. I feel that the priority order should be:

1. Instrument the code for overheads with the global lock that is causing the performance to drop.

2. Instrument the hardware -- we never ran SMP mode BG/P with 4 processes (not threads) doing the messaging rate test separately.

3. Port MPICH2/BG to trunk so that we can try out new enhancements on both breadboard and BG/P.

Thoughts?

-- Pavan

On 01/30/2008 01:07 PM, Darius Buntinas wrote:

I just want to check what the benchmark should do.
My understanding from the conference calls and Sameer's slides is that they're interested in the case where multiple threads of a process are sending/receiving to/from different processes. This allows the process to use multiple DMAs concurrently.
Right now, we've been testing the case where one process is sending/receiving with itself. So moving the locks around VCs may decrease the size of the CS, but will still serialize.
So what's the test we want to run?
I imagine something like P processes each with P-1 threads where each thread is communicating with a different process. But then we need a machine with P(P-1) cores.
On the 8-core nodes we have, we can run 3 processes with 2 threads each. But I think we need more threads to really stress things.
It seems that what we can see from the one or two processes is the just the effect of the size of the CS.
Comments?
-d

--
Pavan Balaji
http://www.mcs.anl.gov/~balaji

William Gropp

Paul and Cynthia Saylor Professor of Computer Science

University of Illinois Urbana-Champaign