I think we should worry about the locking overhead alone and not dilute the benchmark by adding network communication in it. So, each process sending to itself would be better, IMO. Btw, there was a suggestion earlier that we should try to place one lock per VC to see if the performance improves. However, I think we are jumping too far ahead with that. Even with a global lock, I would expect that the performance would stay constant (note that ideal behavior is linear increase in messaging rate with number of threads). However, it's dropping currently. I feel that the priority order should be: 1. Instrument the code for overheads with the global lock that is causing the performance to drop. 2. Instrument the hardware -- we never ran SMP mode BG/P with 4 processes (not threads) doing the messaging rate test separately. 3. Port MPICH2/BG to trunk so that we can try out new enhancements on both breadboard and BG/P. Thoughts? -- Pavan On 01/30/2008 01:07 PM, Darius Buntinas wrote:
I just want to check what the benchmark should do.
My understanding from the conference calls and Sameer's slides is that they're interested in the case where multiple threads of a process are sending/receiving to/from different processes. This allows the process to use multiple DMAs concurrently.
Right now, we've been testing the case where one process is sending/receiving with itself. So moving the locks around VCs may decrease the size of the CS, but will still serialize.
So what's the test we want to run?
I imagine something like P processes each with P-1 threads where each thread is communicating with a different process. But then we need a machine with P(P-1) cores.
On the 8-core nodes we have, we can run 3 processes with 2 threads each. But I think we need more threads to really stress things.
It seems that what we can see from the one or two processes is the just the effect of the size of the CS.
Comments?
-d
-- Pavan Balaji http://www.mcs.anl.gov/~balaji