On Sat, Feb 23, 2013 at 8:21 PM, Jeff Hammond <[email protected]> wrote:
I personally think that MKL, meaning the BLAS, is an amazing programming model.
Are you trolling me?
However, my point here was merely to say that Intel has an integrated software stack that manages thread resources intelligently relative to N independent implementations of MPI, BLAS/LAPACK, and OpenMP/TBB, for example. I was under the impression that Intel presents stupid oversubscription, which a good thing for users but essentially impossible to implement portably.
It's relatively easy to be integrated when you're not interoperable.
I think the only way for async progress to work well is to have fine-grain locking since of MPI, as is done in PAMID. Any implementation that resorts to fat-locking is probably better off without async progress unless the application is doing something really silly (like never calling MPI on a rank that is the target of MPI RMA).
There's more than 100k cycles between times that I enter the MPI stack. There are only two threads ever that contend for locks (my funneled thread and MPICH's async-progress thread). I'm not convinced that you need super fine-grained locks to make progress during that time period. FWIW, my experience has been that standard Nemesis does a _much_ better job of making asynchronous progress than Cray's implementation. If we could look at the code for Cray's implementation, we might be able to get a better idea of why.