On Thu, Jul 23, 2020 at 9:35 PM Satish Balay <[email protected]> wrote:

On Thu, 23 Jul 2020, Jeff Hammond wrote:

...

Open-MPI refuses to let users over subscribe without an extra flag to mpirun.

Yes - and when using this flag - it lets the run through - but there is still performance degradation in oversubscribe mode.

...

I think Intel MPI has an option for blocking poll that supports oversubscription “nicely”.

What option is this? Is it compile time option or something for mpiexec?

https://software.intel.com/content/www/us/en/develop/articles/tuning-the-int... Apply wait mode to oversubscribed jobs This option is particularly relevant for oversubscribed MPI jobs. The goal is to enable the wait mode of the progress engine in order to wait for messages without polling the fabric(s). This can save CPU cycles but decreases the message-response rate (latency), so it should be used with caution. To enable wait mode simply use: I_MPI_WAIT_MODE=1 Jeff

Satish

...

MPICH might have a “no local” option that disables shared memory, in which case nemesis over libfabric with the sockets or TCP provider _might_ do the right thing. But you should ask MPICH people for details.

Jeff

On Thu, Jul 23, 2020 at 12:40 PM Jed Brown <[email protected]> wrote:

...

I think we should default to ch3:nemesis when --download-mpich, and only do ch3:sock when requested (which we would do in CI).

Satish Balay via petsc-dev <[email protected]> writes:

...

Primarily because ch3:sock performance does not degrade in oversubscribe mode - which is developer friendly - i.e on your laptop.

And folks doing optimized runs should use a properly tuned MPI for their setup anyway.

In this case --download-mpich-device=ch3:nemesis is likely appropriate if using --download-mpich [and not using a separate/optimized MPI]

Having defaults that satisfy all use cases is not practical.

Satish

On Wed, 22 Jul 2020, Matthew Knepley wrote:

...

We default to ch3:sock. Scott MacLachlan just had a long thread on the Firedrake list where it ended up that reconfiguring using ch3:nemesis had a 2x performance boost on his 16-core proc, and noticeable effect on the 4 core speedup.

Why do we default to sock?

Thanks,

Matt

-- Jeff Hammond [email protected] http://jeffhammond.github.io/