I think we should worry about the locking overhead alone and not dilute the benchmark by adding network communication in it. So, each process sending to itself would be better, IMO.
I think we want to know how much is the locking overhead in the presence of network communication because that is what happens in reality.
Btw, there was a suggestion earlier that we should try to place one lock per VC to see if the performance improves. However, I think we are jumping too far ahead with that. Even with a global lock, I would expect that the performance would stay constant (note that ideal behavior is linear increase in messaging rate with number of threads). However, it's dropping currently.
Why would the performance stay constant with the global lock?
1. Instrument the code for overheads with the global lock that is causing the performance to drop.
2. Instrument the hardware -- we never ran SMP mode BG/P with 4 processes (not threads) doing the messaging rate test separately.
3. Port MPICH2/BG to trunk so that we can try out new enhancements on both breadboard and BG/P.
I think Sameer would like us to hack IBM's MPI implementation directly. That would save us the porting time as well (for now). Rajeev