- discuss - mailman3.cels.anl.gov

Re: [mpich-discuss] How does MPICH dtermine available NICs?
by Kevin Buckley 10 Mar '26

10 Mar '26

1 0

Re: [mpich-discuss] How does MPICH dtermine available NICs?
by Raymond, Michael 10 Mar '26

10 Mar '26

The NIC symmetry test is specific to HPE Cray MPICH. As a workaround, try setting MPICH_OFI_NUM_NICS=1. — Michael Raymond HPE HPC programming environment From: Kevin Buckley via discuss <discuss(a)mpich.org> Date: Tuesday, March 10, 2026 at 00:47 To: discuss(a)mpich.org <discuss(a)mpich.org> Cc: Kevin Buckley <kevin.buckley.pawsey.org.au(a)gmail.com> Subject: Re: [mpich-discuss] How does MPICH dtermine available NICs? This Message Is From an External Sender This message came from outside your organization. On 2026/03/09 23:24, Zhou, Hui wrote: > Could you try the upstream MPICH? I followed you suggestion to try the upstream MPICH. Trying to configure 3.4a2, the version "inside" the cray-mpich, modulo any vendor updates made to that base release, saw ... configure: error: The Fortran compiler gfortran will not compile files that\ call the same routine with arguments of different types. $ $ which gfortran /opt/cray/pe/gcc-native/14/bin/gfortran $ Wasn't sure if that meant that the MPICH code is too old to make correct compiler checks, or that the compiler is too new to respond them - so I shelved that line of attack. Was able to configure and build 5.0.0, and have used that to compile and run the same noddy "connectivity_c.c" example code that I had been seeing the issue with. It all seems to work without any warnings (as does compiling/ running within an OpenMPI 5.0.3 environment), in that a 2-node job spanning a node with just one NIC, and the dual-NIC node with one NIC disabled, completes, as does a 2-node job spanning the node with just one NIC, and a dual-NIC node with neither NIC disabled. Would you though, expect the vanilla MPICH 5.0.0 to have warned me about the NIC_SYMMETRY inconsistencies? Your comment in the original reply, > The upstream MPICH should be fine with different number of > NICs on different nodes. By default, the process picks a > nic that is closest to process's CPU affinity; or if you > have multiple processes on the same node, each process > will try pick a different nic in a round-robin fashion. > Usually a disabled nic won't be selected during init, would suggest that "later" (than cray-mpich's 3.4a2 starting point, plus whatver enhancements Cray and/or HPE have since added) MPICH-s would not bother to give the warning, as those versions would "just do the right thing" (tm)? If so, think it's sounding as though the consideration of the disabled-at-boot-time NIC as somehow being "available" for MPI traffic, is tied into something that the vendor is doing: somewhere within their overall MPI+Slingshot stack. Thanks again for fielding this one, Kevin M. Buckley -- Supercomputing Systems Administrator Pawsey Supercomputing Centre PERTH Australia

1 0

Re: [mpich-discuss] How does MPICH dtermine available NICs?
by Kevin Buckley 10 Mar '26

10 Mar '26

1 0

Re: [mpich-discuss] How does MPICH dtermine available NICs?
by Kevin Buckley 09 Mar '26

09 Mar '26

1 0

Re: [mpich-discuss] How does MPICH dtermine available NICs?
by Zhou, Hui 09 Mar '26

09 Mar '26

Could you try the upstream MPICH? - https://urldefense.us/v3/__https://www.mpich.org/downloads/__;!!G_uCfscf7eW… The behavior between Cray MPICH and upstream MPICH may have diverged. The upstream MPICH should be fine with different number of NICs on different nodes. By default, the process picks a nic that is closest to process's CPU affinity; or if you have multiple processes on the same node, each process will try pick a different nic in a round-robin fashion. Usually a disabled nic won't be selected during init, but if you see it behave otherwise (weird), I encourage you to create an issue at https://urldefense.us/v3/__https://github.com/pmodels/mpich/issues__;!!G_uC… . Cheers, Hui ________________________________ From: Kevin Buckley via discuss <discuss(a)mpich.org> Sent: Monday, March 9, 2026 12:22 AM To: discuss(a)mpich.org <discuss(a)mpich.org> Cc: Kevin Buckley <kevin.buckley.pawsey.org.au(a)gmail.com> Subject: [mpich-discuss] How does MPICH dtermine available NICs? I do hope this is the right list on which to ask this, but it all seems a bit weird to me, so I thought I'd "turn pro". TL;DR: it gets really weird at the bottom. I am trying to work out how MPICH determines the number of NICs that it "thinks" ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd I do hope this is the right list on which to ask this, but it all seems a bit weird to me, so I thought I'd "turn pro". TL;DR: it gets really weird at the bottom. I am trying to work out how MPICH determines the number of NICs that it "thinks" a node has. Here's what I am seeing, as a result of my nodes having an "Inconsistent number of NICs across the job" PE 0: == Node nid001764 has 1 NIC(s) available PE 0: == Node nid002792 has 2 NIC(s) available PE 0: (where the job is a noddy test job that runs two processes per node, across two nodes) however, as far as I am concerned, the second NIC on that nid002792 has been disabled at Boot-time, by replacing STARTMODE='auto' with STARTMODE='off' in the /etc/sysconfig/network/ifcfg-hsn1 file. As far as wicked (which is really using an ifcfg-compat mode, and not any new wicked-goodness) is concerned, the second interface is "not up", and hasn't been configured, hence: nid002792:~ # wicked ifstatus hsn0 hsn0 up link: #5, state up, mtu 9000 type: ethernet, hwaddr 02:00:00:00:60:73 config: compat:suse:/etc/sysconfig/network/ifcfg-hsn0 leases: ipv4 static granted addr: ipv4 10.253.133.14/17 [static] route: ipv4 172.18.0.0/16 via 10.253.255.254 proto boot nid002792:~ # wicked ifstatus hsn1 hsn1 device-unconfigured link: #6, state up, mtu 1500 type: ethernet, hwaddr 02:00:00:00:60:33 nid002792:~ # and what's more, there is no route using that second interface, which there would have been, had I not ferkled the ifcfg script: nid002792:~ # ip route default via 10.253.128.3 dev hsn0 10.168.28.0/22 dev bond0 proto kernel scope link src 10.168.28.23 10.253.128.0/17 dev hsn0 proto kernel scope link src 10.253.133.14 172.18.0.0/16 via 10.253.255.254 dev hsn0 172.23.0.0/16 via 10.168.31.254 dev bond0 nid002792:~ # THE REALLY WEIRD BIT If I don't disable the second NIC at boot time, but then, once the node has booted, explicitly "ifdown" it, manually, with wicked's wrapped version of an ifdown: wicked --systemd ifdown hsn1 then MPICH jobs lauching on the node DON'T SEE THE SECOND NIC? For reference. here are the two "old-school" network config files nid002792:~ # cat /etc/sysconfig/network/ifcfg-hsn0 STARTMODE='auto' BOOTPROTO='static' IPADDR='10.253.133.14' NETMASK='255.255.128.0' MTU='9000' LINK_REQUIRED='yes' POST_UP_SCRIPT="systemd:[email protected]" nid002792:~ # cat /etc/sysconfig/network/ifcfg-hsn1 STARTMODE='off' BOOTPROTO='static' IPADDR='10.253.133.11' NETMASK='255.255.128.0' MTU='9000' LINK_REQUIRED='yes' POST_UP_SCRIPT="systemd:[email protected]" nid002792:~ # Now, "I would have thought" (tm) that MPICH, on "seeing" an interface that hadn't been START-ed, would not have considered it as a NIC, for NIC_SYMMETRY puposes? I am (more than) aware that I can prevent the messages about NIC_SYMMETRY inconsistencies, but that's not the issue here; the issue here is that MPICH seems to think a NIC that hasn't been START-ed is worthy of consideration. FWIW, it's cray-mpich/8.1.32 so MPICH 3.4a2, under the hood, where the hood belongs to an HPE/Cray EX, running SLES 15 SP6. The info from the cray-mpich/8.1.32 module says - Cray MPICH offers support for multiple NICs per node. Starting with version 8.0.8, by default Cray MPICH will use all available NICs on a node. but maybe their definiiton of "available" differs from the one that I have become accustomed to over the years, to wit: if it's not START-ed; it's not available? Interested to hear any thoughts on this? Kevin M. Buckley -- Supercomputing Systems Administrator Pawsey Supercomputing Centre PERTH Australia

1 0

How does MPICH dtermine available NICs?
by Kevin Buckley 09 Mar '26

09 Mar '26

1 0

Re: [mpich-discuss] PMI 2 deprecation in MPICH 5.0.0
by Zhou, Hui 12 Feb '26

12 Feb '26

Felip, Thanks for the question. The original thesis on PMI 2 was too broad, yet not broad enough to meet the current need. For example, a major feature in PMI 2 is multi-thread performance. But in reality, PMI is mainly used during MPI_Init or other non-thread-performance sensitive areas that the complexity added to the design doesn't really justify its usage. In practice, almost all vendors of PMI 2 are simply providing a wrapper that does not take advantage of any PMI 2 features that can't be obtained with the PMI 1 interface. Meanwhile, PMI 2 kills the one feature of PMI 1 -- simplicity. In fact, I believe almost all vendors, regardless what there default interface is, can effortlessly support PMI 1. And because of PMI 1's simplicity, PMI 1 can more easily extend and adapt to new requirement. For example, we have matched the launching scalability using PMI 1 in hydra (MPICH's launcher). And just added one extension allowed us to support MPI sessions dynamic launching. Compared to PMIx, PMIx is overly complex and broad, and it was designed way beyond the scope of MPI. As a result, it is much cumbersome to fork or independently implement PMIx than with PMI 1. Some of the PMIx features, including more advanced KVS scope and PMI events are enticing, but we believe it makes more sense to extend from the simple base of PMI 1 then to inherit the entity of PMIx and control it back. Our current MPICH team is quite lean, but we still think we can keep advancing MPI if we effectively focus. Focusing on PMI 1 is one of such decisions. At this point, PMI 2 is still supported but won't be further developed. Hope this helps. -- Hui ________________________________ From: Felip Moll via discuss <discuss(a)mpich.org> Sent: Thursday, February 12, 2026 11:41 AM To: discuss(a)mpich.org <discuss(a)mpich.org> Cc: Felip Moll <fmoll(a)nvidia.com> Subject: [mpich-discuss] PMI 2 deprecation in MPICH 5.0.0 Hello, MPICH 5. 0. 0 has been released, and the release notes announces that PMI 2 is now deprecated, and recommends switching to PMI 1. It also says PMI 1 upgraded to PMI 1. 2. Looking at some commits I also see that new features will be added ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Hello, MPICH 5.0.0 has been released, and the release notes announces that PMI 2 is now deprecated, and recommends switching to PMI 1. It also says PMI 1 upgraded to PMI 1.2. Looking at some commits I also see that new features will be added to PMI 1. I would like to understand the technical reasons behind deprecating PMI 2, as theorically it had a better startup latency, improved KVS, and so on compared to PMI 1 [1] I understand that PMIx should be more appropriate than PMI 2 for large scale apps and this would probably be a reason to reduce efforts on supporting other PMIs, but why keep PMI1 and not PMI2? Where does PMI 2 stand at that point? Thank you!. [1] PMI: A Scalable Parallel Process-Management Interface for Extreme-Scale Systems<https://www.mcs.anl.gov/papers/P1760.pdf > Felip Moll Slurm Sr Engineer NVIDIA<https://urldefense.com/v3/__http:/www.nvidia.com/__;!!LpKI!m1it1uHwfgYmHS7q… > [nvidia.com]<https://urldefense.com/v3/__http:/www.nvidia.com/__;!!LpKI!m1it1uHwfgYmHS7q… >

1 0

PMI 2 deprecation in MPICH 5.0.0
by Felip Moll 12 Feb '26

12 Feb '26

Hello, MPICH 5.0.0 has been released, and the release notes announces that PMI 2 is now deprecated, and recommends switching to PMI 1. It also says PMI 1 upgraded to PMI 1.2. Looking at some commits I also see that new features will be added to PMI 1. I would like to understand the technical reasons behind deprecating PMI 2, as theorically it had a better startup latency, improved KVS, and so on compared to PMI 1 [1] I understand that PMIx should be more appropriate than PMI 2 for large scale apps and this would probably be a reason to reduce efforts on supporting other PMIs, but why keep PMI1 and not PMI2? Where does PMI 2 stand at that point? Thank you!. [1] PMI: A Scalable Parallel Process-Management Interface for Extreme-Scale Systems<https://www.mcs.anl.gov/papers/P1760.pdf > Felip Moll Slurm Sr Engineer NVIDIA<https://urldefense.com/v3/__http:/www.nvidia.com/__;!!LpKI!m1it1uHwfgYmHS7q… > [nvidia.com]<https://urldefense.com/v3/__http:/www.nvidia.com/__;!!LpKI!m1it1uHwfgYmHS7q… >

1 0

REMINDER: Join us Thursday, February 12th for an MPICH BoF as part of CASS Community BoF Days
by Raffenetti, Ken 12 Feb '26

12 Feb '26

Reminder that the MPICH BoF starts in about 30 minutes! There is still time to register and attend. Updated schedule below. Date and Time: Thursday, February 12th, 10:00am – 11:30am Central Time Registration: https://urldefense.us/v3/__https://www.zoomgov.com/meeting/register/EQumnsm… Agenda: * ANL Update – Hui Zhou * Partner Updates * Kim McMahon, Hewlett Packard Enterprise * Nat Shineman, Ohio State University * Simon Pickartz, ParTec AG * Gengbin Zheng, Intel * User Presentations * Nathan Nichols, ANL/ALCF * Junchao Zhang, ANL/PETSc * Q&A Cheers, The MPICH team

1 0

Join us Thursday, February 12th for an MPICH BoF as part of CASS Community BoF Days
by Raffenetti, Ken 09 Feb '26

09 Feb '26

All, The MPICH team will host a Birds-of-a-Feather session this Thursday, February 12th at 10am Central time as part of the CASS Community BoF Days https://urldefense.us/v3/__https://cass.community/news/2026-02-10-cass-bof-… . We hope to see folks there to discuss upcoming MPICH release plans and to hear presentations from MPICH partners and users. Date and Time: Thursday, February 12th, 10:00am – 11:30am Central Time Registration: https://urldefense.us/v3/__https://www.zoomgov.com/meeting/register/EQumnsm… Agenda: * ANL Update – Hui Zhou * Partner Updates * Kim McMahon, Hewlett Packard Enterprise * Nat Shineman, Ohio State University * Simon Pickartz, ParTec AG * Maria Garzaran, Intel * User Presentations * Thomas Applencourt, ANL/ALCF * Junchao Zhang, ANL/PETSc * Q&A Cheers, The MPICH team

1 0