There's always some other processes running (demons etc.), so you do need to be careful about what the OS is doing.
Yeah, but they are all sleeping or awake very rarely. On a 1-node benchmark, I don't know if that would add too much overhead.
The code was designed to minimize the updates of shared structures, so it is odd that the threads on separate cores are showing such bad performance.
Yeah, I was looking through the code and couldn't find any obvious problematic spots. This will require more digging into the code.
On the perthread case, I don't know if any special effort was made to ensure that the perthread data structures were on separate cache lines. The accesses to the MPIR_Process structure will also depend on the cache policy on loads - we're hoping that read access is shared, but if it isn't, we'd see something like this.
I know that AMD allows for shared read. So on breadboard at least this should not be a problem. -- Pavan -- Pavan Balaji http://www.mcs.anl.gov/~balaji