Unfortunately the MPI application I'm trying to run would require too much effort to convert to hybrid OpenMP/MPI to be viable option at this point.

Does your code actually scale to 240 MPI ranks per card? What is the performance relative to peak or a fully-populated MPI-only run on e.g SNB? You're going to have to convert to MPI+Threads eventually so you might as well start now. It is completely unreasonable to expect that MPI-only is reasonable usage on a 60-core chip with a ring interconnect. I am so convinced of the impossibility that MPI_Allreduce scales to

100 MPI procs in shared memory that I will learn Ada as punishment if I'm wrong.

If you don't like OpenMP, you could consider Pthreads or TBB instead, but something tells me that if you're unwilling to use OpenMP, it's because you have an F77 code with common blocks that cannot be made thread-safe for all the beer in Australia. Jeff -- Jeff Hammond [email protected]