On Thu, Sep 29, 2011 at 07:44, Matthew Knepley <[email protected]> wrote:
The way I read these numbers is that there is bandwidth for about 3 cores on this machine, and non-negligible synchronization penalty:
1 proc 2 proc 4 proc 8 proc VecAXPBYCZ 496 857 1064 1070 VecDot 451 724 1089 736 MatMult 434 638 701 703
Matt, thanks for pulling out this summary. The synchronization for the dot product is clearly expensive here. I'm surprised it's so significant on such a small problem, but it is a common problem for scalability. I think that MPI is putting one process per socket when you go from 1 to 2 processes. This gives you pretty good speedup despite the fact that memory traffic for both sockets is routed through the "Blackford" chipset. Intel fixed this in later generations by throwing out the notion of uniform memory access by having independent memory banks for each socket (which AMD had at the time these chips came out). If not odd hardware issues such as having enough memory streams and imbalances, one process per socket can saturate the memory bandwidth, so the speed-up you get from 2 procs (1 per socket) to 4 procs (2 per socket) is minimal. On this architecture, you typically see the STREAM bandwidth go down slightly as you add more than 2 procs per socket, so it's no surprise that it doesn't help here. Note that your naive 1D partition is getting worse as you add processes. The MatMult should scale out somewhat better if you use a 2D decomposition, as is done by any of the examples using a DA (DMDA in 3.2) for grid management.
The bandwidth tops out between 2 and 4 cores (The 5345 should have 10.6 GB/s but you should runs streams as Barry says to see what is achievable). There is obviously a penalty for VecDot against VecAXPYCZ, which is the sync penalty which also seems to affect MatMult. Maybe Jed can explain that.