These are very interesting. Is there a chance that the large latency on different cores is because of some occasional scheduling jitter - if one thread is occasionally not running when the others start, there will be one huge latency which will get averaged out in the test. That's less likely if they're all on one core. To test the other issue wrt shared data, can you put together a program that (a) reads from some shared data (like the MPIR_Process structure) and (b) updates some shared data (just a counter, e.g.,), and run a similar test? Other than access the MPIR_Process and PerThread structures, there shouldn't be much shared access in the MPI_PROC_NULL case (if this has the nesting count stuff enabled, that's another test - using pthread_get_specific (or whatever it is called) in the two cases. Bill On Feb 1, 2008, at 12:47 AM, Pavan Balaji wrote:
Ok, we can't do this kind of scheduling on the BG/P (can't have more than one process on a processor). So, I don't have those numbers.
Given this new insight on the impact of process-to-core mappings, we'll need to take step back on what exactly our approach should be. It looks like the problem is not the locks alone. It is probably shared variables that are thrashing in cache. Does this mean we have to rethink how we allocate shared buffers and other objects shared between the different threads?
-- Pavan
On 01/31/2008 09:00 PM, Pavan Balaji wrote:
Holy googly moogly! The OS schedules each process on a different core. I forced it to schedule everything on one core and the performance suddenly shot up. Here are the numbers: /* Round robin scheduling of processes to cores */ #bb09# /home/balaji> mpiexec -n 1 ./threads -t 4 -l 100000 -r # MPI Message Rate Test with 4 Threads # nthreads Size Latency (us) Message Rate (MMPS) 1 0 0.13 7.899 2 0 1.12 1.780 3 0 3.14 0.955 4 0 4.59 0.871 /* Scheduling all processes on one core */ #bb09# /home/balaji> mpiexec -n 1 ./threads -t 4 -l 100000 # MPI Message Rate Test with 4 Threads # nthreads Size Latency (us) Message Rate (MMPS) 1 0 0.13 7.855 2 0 0.26 7.819 3 0 0.38 7.826 4 0 0.51 7.768 -- Pavan On 01/31/2008 07:17 PM, William Gropp wrote:
At what point does the OS decide to schedule these threads on different cores? In the classic multithreaded GUI application, the best strategy is to schedule all threads on the same core. Bill
On Jan 31, 2008, at 6:58 PM, Darius Buntinas wrote:
Sure, it's on bblogin ~buntinas/ng-mpich/mrate/test/ threads-2.c . It's based on your threads.c that I grabbed from your directory around 4:00.
I moved the pthread_barriers outside the timing loop and ran it again. The "message rate" improved, but still decreased as the number of threads increased.
-d
On 01/31/2008 06:24 PM, Pavan Balaji wrote:
Very weird. Can you send me your test program? I'd like to poke into it a bit. -- Pavan On 01/31/2008 06:21 PM, Darius Buntinas wrote:
I removed the send and receive calls from the timing loop and replaced them with statements that increment and decrement a volatile variable. The volatile variable is in an array, so each thread accesses its own variable. The array elements are also padded so each is in its own cache line. So there should be no contention between threads on the variable.
It looks like the test program is affecting the results. :-)
# MPI Message Rate Test (10000000 iterations 1 process(es)) # nthreads Size Latency (us) Message Rate (MMPS) 1 0 0.00 306.547 2 0 0.00 483.213 3 0 0.01 184.908 4 0 0.01 114.825 5 0 0.01 78.911 6 0 0.02 59.759 7 0 0.02 47.333 8 0 0.03 38.067
-d
On 01/31/2008 06:02 PM, Darius Buntinas wrote:
I removed the locks and nesting code (i.e., defined CS_ENTER/ EXIT to nothing). I then changed the test to use MPI_Send and MPI_Recv (instead of isend and irecv), so I wouldn't have to use a lock when allocating a request. I then ran it on intel-loaner1 (dual quadcore) sending and receiving to MPI_PROC_NULL. Below are the results (first with locking intact, then with the locking removed).
There's an improvement in message rate, but we still see message rate decrease as we add additional threads. What's interesting, is that even the single threaded case (that doesn't call MPI_Init_thread) shows improvement with locks removed. I can't explain that, since it's not using locks (or at least shouldn't). I'll look into this further.
-d
CONTROL CASE WITH LOCKING INTACT: # MPI Message Rate Test (10000 iterations 1 process(es) using MPI_PROC_NULL) # nthreads Size Latency (us) Message Rate (MMPS) 1 0 0.15 6.774 2 0 0.57 1.755 3 0 1.29 0.773 4 0 2.07 0.483 5 0 2.73 0.366 6 0 3.12 0.321 7 0 3.45 0.290 8 0 3.69 0.271
LOCKING REMOVED: % ./threads-2 -t 8 -l 10000 -pn # MPI Message Rate Test (10000 iterations 1 process(es) using MPI_PROC_NULL) # nthreads Size Latency (us) Message Rate (MMPS) 1 0 0.09 11.709 2 0 0.11 9.128 3 0 0.20 5.032 4 0 0.26 3.779 5 0 0.32 3.158 6 0 0.38 2.635 7 0 0.45 2.243 8 0 0.50 1.989
On 01/31/2008 02:55 PM, William Gropp wrote:
The nesting functionality makes use of thread-private storage, which involves potentially a routine call (for the thread-private storage). I agree that it shouldn't cost much. But it is something that's needed only if MPI routines call MPI routines in the implementation; if instead MPI routines only called internal routines, the entire nesting block could be eliminated.
Bill
On Jan 31, 2008, at 2:49 PM, Pavan Balaji wrote:
Bill,
The nesting functionality is just regular increment (not even atomic increment). So, do we expect it to have too much performance impact? I tried this on breadboard and the performance difference was within the variation due to noise. I'm trying it on BG/P now.
-- Pavan
On 01/31/2008 01:17 PM, William Gropp wrote:
No, something simpler. Just ignore the nest count code entirely; we only use it for error handling and perhaps for choosing whether to get the lock in nested MPI calls, of which these examples have none (or maybe one, if there's a Barrier somewhere). That will tell us how much the get thread specific etc costs. Its probably not much, though it might be a non-trivial part of the overall latency. Bill On Jan 31, 2008, at 1:03 PM, Pavan Balaji wrote:
Bill,
We know that holding the global lock serializes the MPI code. The only other questions are (1) how much does the nest count cost, (2) how much would fine grain locking/atomic updates cost (e.g., for the communicator reference count update in an uncompleted Irecv), and (3) how narrow could a global lock be for the necessary updates instead of fine-grain? We can answer (1) by temporarily changing the definition of the nest count macros and rerunning these tests.
Do you mean acquire the global lock in the nest count increment?
-- Pavan
-- Pavan Balaji http://www.mcs.anl.gov/~balaji
William Gropp Paul and Cynthia Saylor Professor of Computer Science University of Illinois Urbana-Champaign
-- Pavan Balaji http://www.mcs.anl.gov/~balaji
William Gropp Paul and Cynthia Saylor Professor of Computer Science University of Illinois Urbana-Champaign
William Gropp Paul and Cynthia Saylor Professor of Computer Science University of Illinois Urbana-Champaign
-- Pavan Balaji http://www.mcs.anl.gov/~balaji
William Gropp Paul and Cynthia Saylor Professor of Computer Science University of Illinois Urbana-Champaign