I removed the send and receive calls from the timing loop and replaced them with statements that increment and decrement a volatile variable. The volatile variable is in an array, so each thread accesses its own variable. The array elements are also padded so each is in its own cache line. So there should be no contention between threads on the variable.
It looks like the test program is affecting the results. :-)
I removed the locks and nesting code (i.e., defined CS_ENTER/EXIT to nothing). I then changed the test to use MPI_Send and MPI_Recv (instead of isend and irecv), so I wouldn't have to use a lock when allocating a request. I then ran it on intel-loaner1 (dual quadcore) sending and receiving to MPI_PROC_NULL. Below are the results (first with locking intact, then with the locking removed).
There's an improvement in message rate, but we still see message rate decrease as we add additional threads. What's interesting, is that even the single threaded case (that doesn't call MPI_Init_thread) shows improvement with locks removed. I can't explain that, since it's not using locks (or at least shouldn't). I'll look into this further.
-d
CONTROL CASE WITH LOCKING INTACT:
# MPI Message Rate Test (10000 iterations 1 process(es) using MPI_PROC_NULL)
# nthreads Size Latency (us) Message Rate (MMPS)
1 0 0.15 6.774
2 0 0.57 1.755
3 0 1.29 0.773
4 0 2.07 0.483
5 0 2.73 0.366
6 0 3.12 0.321
7 0 3.45 0.290
8 0 3.69 0.271
LOCKING REMOVED:
% ./threads-2 -t 8 -l 10000 -pn
# MPI Message Rate Test (10000 iterations 1 process(es) using MPI_PROC_NULL)
# nthreads Size Latency (us) Message Rate (MMPS)
1 0 0.09 11.709
2 0 0.11 9.128
3 0 0.20 5.032
4 0 0.26 3.779
5 0 0.32 3.158
6 0 0.38 2.635
7 0 0.45 2.243
8 0 0.50 1.989
On 01/31/2008 02:55 PM, William Gropp wrote:
The nesting functionality makes use of thread-private storage, which involves potentially a routine call (for the thread-private storage). I agree that it shouldn't cost much. But it is something that's needed only if MPI routines call MPI routines in the implementation; if instead MPI routines only called internal routines, the entire nesting block could be eliminated.
Bill
On Jan 31, 2008, at 2:49 PM, Pavan Balaji wrote:
Bill,
The nesting functionality is just regular increment (not even atomic increment). So, do we expect it to have too much performance impact? I tried this on breadboard and the performance difference was within the variation due to noise. I'm trying it on BG/P now.
-- Pavan
On 01/31/2008 01:17 PM, William Gropp wrote:
No, something simpler. Just ignore the nest count code entirely; we only use it for error handling and perhaps for choosing whether to get the lock in nested MPI calls, of which these examples have none (or maybe one, if there's a Barrier somewhere). That will tell us how much the get thread specific etc costs. Its probably not much, though it might be a non-trivial part of the overall latency.
Bill
On Jan 31, 2008, at 1:03 PM, Pavan Balaji wrote:
Bill,
We know that holding the global lock serializes the MPI code. The only other questions are (1) how much does the nest count cost, (2) how much would fine grain locking/atomic updates cost (e.g., for the communicator reference count update in an uncompleted Irecv), and (3) how narrow could a global lock be for the necessary updates instead of fine-grain?
We can answer (1) by temporarily changing the definition of the nest count macros and rerunning these tests.
Do you mean acquire the global lock in the nest count increment?
-- Pavan
--
Pavan Balaji
William Gropp
Paul and Cynthia Saylor Professor of Computer Science
University of Illinois Urbana-Champaign
--
Pavan Balaji
William Gropp
Paul and Cynthia Saylor Professor of Computer Science
University of Illinois Urbana-Champaign