Darius, What exactly is the pthread_barrier problem? Can you point it out in the code? -- Pavan On 02/01/2008 11:08 AM, Darius Buntinas wrote:
OK I fixed a bug in the test program that calculated the message rate. I then kept the number of threads created constant and kept the extra ones idle (waiting in a pthread barrier). This should normalize differences in scheduling, barrier times, etc. Now when I run the dummy test (no MPI calls) I get expected results:
% ./threads-2 -t 8 -l 10000000 # MPI Message Rate Test (10000000 iterations 1 process(es)) # nthreads Size Latency (us) Message Rate (MMPS) 1 0 0.00 452.149 2 0 0.00 444.716 3 0 0.00 442.400 4 0 0.00 444.068 5 0 0.00 441.870 6 0 0.00 442.154 7 0 0.00 442.156 8 0 0.00 442.810
Running on a single core shows "message rate" decreasing as the number of active threads increase, as expected.
% taskset -c 1 ./threads-2 -t 8 -l 10000000 # MPI Message Rate Test (10000000 iterations 1 process(es)) # nthreads Size Latency (us) Message Rate (MMPS) 1 0 0.00 454.413 2 0 0.00 226.710 3 0 0.01 150.964 4 0 0.01 113.207 5 0 0.01 90.535 6 0 0.01 75.443 7 0 0.02 64.649 8 0 0.02 56.562
I'll run some non-dummy tests and see what happens.
-d
On 01/31/2008 06:21 PM, Darius Buntinas wrote:
I removed the send and receive calls from the timing loop and replaced them with statements that increment and decrement a volatile variable. The volatile variable is in an array, so each thread accesses its own variable. The array elements are also padded so each is in its own cache line. So there should be no contention between threads on the variable.
It looks like the test program is affecting the results. :-)
# MPI Message Rate Test (10000000 iterations 1 process(es)) # nthreads Size Latency (us) Message Rate (MMPS) 1 0 0.00 306.547 2 0 0.00 483.213 3 0 0.01 184.908 4 0 0.01 114.825 5 0 0.01 78.911 6 0 0.02 59.759 7 0 0.02 47.333 8 0 0.03 38.067
-d
On 01/31/2008 06:02 PM, Darius Buntinas wrote:
I removed the locks and nesting code (i.e., defined CS_ENTER/EXIT to nothing). I then changed the test to use MPI_Send and MPI_Recv (instead of isend and irecv), so I wouldn't have to use a lock when allocating a request. I then ran it on intel-loaner1 (dual quadcore) sending and receiving to MPI_PROC_NULL. Below are the results (first with locking intact, then with the locking removed).
There's an improvement in message rate, but we still see message rate decrease as we add additional threads. What's interesting, is that even the single threaded case (that doesn't call MPI_Init_thread) shows improvement with locks removed. I can't explain that, since it's not using locks (or at least shouldn't). I'll look into this further.
-d
CONTROL CASE WITH LOCKING INTACT: # MPI Message Rate Test (10000 iterations 1 process(es) using MPI_PROC_NULL) # nthreads Size Latency (us) Message Rate (MMPS) 1 0 0.15 6.774 2 0 0.57 1.755 3 0 1.29 0.773 4 0 2.07 0.483 5 0 2.73 0.366 6 0 3.12 0.321 7 0 3.45 0.290 8 0 3.69 0.271
LOCKING REMOVED: % ./threads-2 -t 8 -l 10000 -pn # MPI Message Rate Test (10000 iterations 1 process(es) using MPI_PROC_NULL) # nthreads Size Latency (us) Message Rate (MMPS) 1 0 0.09 11.709 2 0 0.11 9.128 3 0 0.20 5.032 4 0 0.26 3.779 5 0 0.32 3.158 6 0 0.38 2.635 7 0 0.45 2.243 8 0 0.50 1.989
On 01/31/2008 02:55 PM, William Gropp wrote:
The nesting functionality makes use of thread-private storage, which involves potentially a routine call (for the thread-private storage). I agree that it shouldn't cost much. But it is something that's needed only if MPI routines call MPI routines in the implementation; if instead MPI routines only called internal routines, the entire nesting block could be eliminated.
Bill
On Jan 31, 2008, at 2:49 PM, Pavan Balaji wrote:
Bill,
The nesting functionality is just regular increment (not even atomic increment). So, do we expect it to have too much performance impact? I tried this on breadboard and the performance difference was within the variation due to noise. I'm trying it on BG/P now.
-- Pavan
On 01/31/2008 01:17 PM, William Gropp wrote:
No, something simpler. Just ignore the nest count code entirely; we only use it for error handling and perhaps for choosing whether to get the lock in nested MPI calls, of which these examples have none (or maybe one, if there's a Barrier somewhere). That will tell us how much the get thread specific etc costs. Its probably not much, though it might be a non-trivial part of the overall latency. Bill On Jan 31, 2008, at 1:03 PM, Pavan Balaji wrote:
Bill,
We know that holding the global lock serializes the MPI code. The only other questions are (1) how much does the nest count cost, (2) how much would fine grain locking/atomic updates cost (e.g., for the communicator reference count update in an uncompleted Irecv), and (3) how narrow could a global lock be for the necessary updates instead of fine-grain? We can answer (1) by temporarily changing the definition of the nest count macros and rerunning these tests.
Do you mean acquire the global lock in the nest count increment?
-- Pavan
-- Pavan Balaji http://www.mcs.anl.gov/~balaji
William Gropp Paul and Cynthia Saylor Professor of Computer Science University of Illinois Urbana-Champaign
-- Pavan Balaji http://www.mcs.anl.gov/~balaji
William Gropp Paul and Cynthia Saylor Professor of Computer Science University of Illinois Urbana-Champaign
-- Pavan Balaji http://www.mcs.anl.gov/~balaji