There's always some other processes running (demons etc.), so you do need to be careful about what the OS is doing. The code was designed to minimize the updates of shared structures, so it is odd that the threads on separate cores are showing such bad performance. On the perthread case, I don't know if any special effort was made to ensure that the perthread data structures were on separate cache lines. The accesses to the MPIR_Process structure will also depend on the cache policy on loads - we're hoping that read access is shared, but if it isn't, we'd see something like this. Bill On Feb 1, 2008, at 10:32 AM, Pavan Balaji wrote:
Is there a chance that the large latency on different cores is because of some occasional scheduling jitter - if one thread is occasionally not running when the others start, there will be one huge latency which will get averaged out in the test. That's less likely if they're all on one core.
It's unlikely that there's any scheduling jitter. There are exactly the same number of processes as cores in the round-robin case (worse performance case), so there should not be any contention. Also, if there was some jitter, it would show up as noise. The drop in performance with increasing threads is *very* consistent.
There could be scheduling jitter in the single core case since all processes are scheduled on the same core. But this is the case with the higher performance. Also, since the lock anyway serializes the code, it should not matter that the processes are sharing the same core. I agree that this is not the ideal behavior. But it'll let us narrow down the non-lock-related issues.
To test the other issue wrt shared data, can you put together a program that (a) reads from some shared data (like the MPIR_Process structure) and (b) updates some shared data (just a counter, e.g.,), and run a similar test?
Ok, I can try this.
Other than access the MPIR_Process and PerThread structures, there shouldn't be much shared access in the MPI_PROC_NULL case (if this has the nesting count stuff enabled, that's another test - using pthread_get_specific (or whatever it is called) in the two cases.
Is there a way to find out what data is thrashing across the different cores without analyzing the entire source code? The PerThread stuff should *not* be shared, right? Each thread allocates this individually and access it individually. Also, I think MPIR_Process will only be written to at the start of the program and only read after that. So, that should not cause any cache thrashing either. I think PAPI is my friend here. Let me try that.
Btw, the nesting count stuff was not disabled in the tests I ran. But that's a thread private field too.
-- Pavan
-- Pavan Balaji http://www.mcs.anl.gov/~balaji
William Gropp Paul and Cynthia Saylor Professor of Computer Science University of Illinois Urbana-Champaign