On Sat, Feb 23, 2013 at 8:21 PM, Jeff Hammond <jhammond@alcf.anl.gov> wrote:

I personally think that MKL, meaning the BLAS, is an amazing
programming model.

Are you trolling me?

However, my point here was merely to say that
Intel has an integrated software stack that manages thread resources
intelligently relative to N independent implementations of MPI,
BLAS/LAPACK, and OpenMP/TBB, for example. I was under the impression
that Intel presents stupid oversubscription, which a good thing for
users but essentially impossible to implement portably.

It's relatively easy to be integrated when you're not interoperable.

I think the only way for async progress to work well is to have
fine-grain locking since of MPI, as is done in PAMID. Any
implementation that resorts to fat-locking is probably better off
without async progress unless the application is doing something
really silly (like never calling MPI on a rank that is the target of
MPI RMA).

There's more than 100k cycles between times that I enter the MPI stack. There are only two threads ever that contend for locks (my funneled thread and MPICH's async-progress thread). I'm not convinced that you need super fine-grained locks to make progress during that time period. FWIW, my experience has been that standard Nemesis does a _much_ better job of making asynchronous progress than Cray's implementation. If we could look at the code for Cray's implementation, we might be able to get a better idea of why.