Hey Toni, I’d be surprised that the performance drops that much, but you can try —with-device=ch4:ofi:psm2 to convert at least some of the branches to be compile-time instead of runtime. After that, I don’t remember enough about OFI 1.5. There might have been some changes in MPICH over the last year or two that makes that version not perform as well… Thanks, Wes

On Jun 15, 2021, at 4:46 AM, Antonio Peña via discuss <[email protected]> wrote:

Hi folks,

I'm setting up an MPICH over libfabric over psm2 for MareNostrum (Omni-Path), to try out some ideas.

I've compiled libfabric 1.5 (last one that compiles in this machine) over opa-psm2-11.2.185, and mpich-3.4.2 + mpich-4.0a1 in both ch3 and ch4 (yes 4 MPICH variants). There's only psm2 support in libfabric, so no danger of falling back to other providers. ldd confirms my libfabric is linked.

./fi_info provider: psm2 fabric: psm2 domain: psm2 version: 1.5 type: FI_EP_RDM protocol: FI_PROTO_PSMX2

I'm comparing 2-node pt2pt performance against impi/2017.4 using osu microbenchmarks.

While both fi_pingong and impi give me a max. BW of ~10 MB/s, all mpich versions stick at ~3 MB/s.

Is this expected? I mean, is there so much secret sauce in impi? Or, am likely doing something wrong?

I'm doing fairly plain configures, nothing fancy, e.g.: ./configure --prefix=... --with-device=ch4:ofi --with-libfabric=...

I'd appreciate some guidance - my MPICH tweaking is a little rusted :)

Best, Toni _______________________________________________ discuss mailing list [email protected] To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss