March 2026 - discuss - mailman3.cels.anl.gov

MPI ABI in mpich 5.0*
by Alastair McKinstry 31 Mar '26

31 Mar '26

1 0

Re: [mpich-discuss] How does MPICH dtermine available NICs?
by Raymond, Michael 11 Mar '26

11 Mar '26

The NIC symmetry test warns users about nodes that unexpectedly have down NICs, so they don’t wonder later why they got strange performance. In your case, when other code in MPICH decides that one of the nodes has two NICs, it causes the symmetry test to fail. A fix to that other code should cause the symmetry test to pass. — Michael Raymond HPE HPC programming environment From: Kevin Buckley via discuss <discuss(a)mpich.org> Date: Tuesday, March 10, 2026 at 20:33 To: discuss(a)mpich.org <discuss(a)mpich.org> Cc: Kevin Buckley <kevin.buckley.pawsey.org.au(a)gmail.com> Subject: Re: [mpich-discuss] How does MPICH dtermine available NICs? This Message Is From an External Sender This message came from outside your organization. On 2026/03/10 23:34, Raymond, Michael wrote: > The NIC symmetry test is specific to HPE Cray MPICH. Cheers for that, Raymond: always good to know the provenance of things. Presumably, it's not needed in "upcoming" HPE Cray MPICH-s, where later, underlying MPICH-s do the "pick a nic" thing? Kevin M. Buckley -- Supercomputing Systems Administrator Pawsey Supercomputing Centre PERTH Australia

1 0

Re: [mpich-discuss] How does MPICH dtermine available NICs?
by Kevin Buckley 11 Mar '26

11 Mar '26

1 0

Re: [mpich-discuss] How does MPICH dtermine available NICs?
by Kevin Buckley 10 Mar '26

10 Mar '26

1 0

Re: [mpich-discuss] How does MPICH dtermine available NICs?
by Raymond, Michael 10 Mar '26

10 Mar '26

The NIC symmetry test is specific to HPE Cray MPICH. As a workaround, try setting MPICH_OFI_NUM_NICS=1. — Michael Raymond HPE HPC programming environment From: Kevin Buckley via discuss <discuss(a)mpich.org> Date: Tuesday, March 10, 2026 at 00:47 To: discuss(a)mpich.org <discuss(a)mpich.org> Cc: Kevin Buckley <kevin.buckley.pawsey.org.au(a)gmail.com> Subject: Re: [mpich-discuss] How does MPICH dtermine available NICs? This Message Is From an External Sender This message came from outside your organization. On 2026/03/09 23:24, Zhou, Hui wrote: > Could you try the upstream MPICH? I followed you suggestion to try the upstream MPICH. Trying to configure 3.4a2, the version "inside" the cray-mpich, modulo any vendor updates made to that base release, saw ... configure: error: The Fortran compiler gfortran will not compile files that\ call the same routine with arguments of different types. $ $ which gfortran /opt/cray/pe/gcc-native/14/bin/gfortran $ Wasn't sure if that meant that the MPICH code is too old to make correct compiler checks, or that the compiler is too new to respond them - so I shelved that line of attack. Was able to configure and build 5.0.0, and have used that to compile and run the same noddy "connectivity_c.c" example code that I had been seeing the issue with. It all seems to work without any warnings (as does compiling/ running within an OpenMPI 5.0.3 environment), in that a 2-node job spanning a node with just one NIC, and the dual-NIC node with one NIC disabled, completes, as does a 2-node job spanning the node with just one NIC, and a dual-NIC node with neither NIC disabled. Would you though, expect the vanilla MPICH 5.0.0 to have warned me about the NIC_SYMMETRY inconsistencies? Your comment in the original reply, > The upstream MPICH should be fine with different number of > NICs on different nodes. By default, the process picks a > nic that is closest to process's CPU affinity; or if you > have multiple processes on the same node, each process > will try pick a different nic in a round-robin fashion. > Usually a disabled nic won't be selected during init, would suggest that "later" (than cray-mpich's 3.4a2 starting point, plus whatver enhancements Cray and/or HPE have since added) MPICH-s would not bother to give the warning, as those versions would "just do the right thing" (tm)? If so, think it's sounding as though the consideration of the disabled-at-boot-time NIC as somehow being "available" for MPI traffic, is tied into something that the vendor is doing: somewhere within their overall MPI+Slingshot stack. Thanks again for fielding this one, Kevin M. Buckley -- Supercomputing Systems Administrator Pawsey Supercomputing Centre PERTH Australia

1 0

Re: [mpich-discuss] How does MPICH dtermine available NICs?
by Kevin Buckley 10 Mar '26

10 Mar '26

1 0

Re: [mpich-discuss] How does MPICH dtermine available NICs?
by Kevin Buckley 09 Mar '26

09 Mar '26

1 0

Re: [mpich-discuss] How does MPICH dtermine available NICs?
by Zhou, Hui 09 Mar '26

09 Mar '26

Could you try the upstream MPICH? - https://urldefense.us/v3/__https://www.mpich.org/downloads/__;!!G_uCfscf7eW… The behavior between Cray MPICH and upstream MPICH may have diverged. The upstream MPICH should be fine with different number of NICs on different nodes. By default, the process picks a nic that is closest to process's CPU affinity; or if you have multiple processes on the same node, each process will try pick a different nic in a round-robin fashion. Usually a disabled nic won't be selected during init, but if you see it behave otherwise (weird), I encourage you to create an issue at https://urldefense.us/v3/__https://github.com/pmodels/mpich/issues__;!!G_uC… . Cheers, Hui ________________________________ From: Kevin Buckley via discuss <discuss(a)mpich.org> Sent: Monday, March 9, 2026 12:22 AM To: discuss(a)mpich.org <discuss(a)mpich.org> Cc: Kevin Buckley <kevin.buckley.pawsey.org.au(a)gmail.com> Subject: [mpich-discuss] How does MPICH dtermine available NICs? I do hope this is the right list on which to ask this, but it all seems a bit weird to me, so I thought I'd "turn pro". TL;DR: it gets really weird at the bottom. I am trying to work out how MPICH determines the number of NICs that it "thinks" ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd I do hope this is the right list on which to ask this, but it all seems a bit weird to me, so I thought I'd "turn pro". TL;DR: it gets really weird at the bottom. I am trying to work out how MPICH determines the number of NICs that it "thinks" a node has. Here's what I am seeing, as a result of my nodes having an "Inconsistent number of NICs across the job" PE 0: == Node nid001764 has 1 NIC(s) available PE 0: == Node nid002792 has 2 NIC(s) available PE 0: (where the job is a noddy test job that runs two processes per node, across two nodes) however, as far as I am concerned, the second NIC on that nid002792 has been disabled at Boot-time, by replacing STARTMODE='auto' with STARTMODE='off' in the /etc/sysconfig/network/ifcfg-hsn1 file. As far as wicked (which is really using an ifcfg-compat mode, and not any new wicked-goodness) is concerned, the second interface is "not up", and hasn't been configured, hence: nid002792:~ # wicked ifstatus hsn0 hsn0 up link: #5, state up, mtu 9000 type: ethernet, hwaddr 02:00:00:00:60:73 config: compat:suse:/etc/sysconfig/network/ifcfg-hsn0 leases: ipv4 static granted addr: ipv4 10.253.133.14/17 [static] route: ipv4 172.18.0.0/16 via 10.253.255.254 proto boot nid002792:~ # wicked ifstatus hsn1 hsn1 device-unconfigured link: #6, state up, mtu 1500 type: ethernet, hwaddr 02:00:00:00:60:33 nid002792:~ # and what's more, there is no route using that second interface, which there would have been, had I not ferkled the ifcfg script: nid002792:~ # ip route default via 10.253.128.3 dev hsn0 10.168.28.0/22 dev bond0 proto kernel scope link src 10.168.28.23 10.253.128.0/17 dev hsn0 proto kernel scope link src 10.253.133.14 172.18.0.0/16 via 10.253.255.254 dev hsn0 172.23.0.0/16 via 10.168.31.254 dev bond0 nid002792:~ # THE REALLY WEIRD BIT If I don't disable the second NIC at boot time, but then, once the node has booted, explicitly "ifdown" it, manually, with wicked's wrapped version of an ifdown: wicked --systemd ifdown hsn1 then MPICH jobs lauching on the node DON'T SEE THE SECOND NIC? For reference. here are the two "old-school" network config files nid002792:~ # cat /etc/sysconfig/network/ifcfg-hsn0 STARTMODE='auto' BOOTPROTO='static' IPADDR='10.253.133.14' NETMASK='255.255.128.0' MTU='9000' LINK_REQUIRED='yes' POST_UP_SCRIPT="systemd:[email protected]" nid002792:~ # cat /etc/sysconfig/network/ifcfg-hsn1 STARTMODE='off' BOOTPROTO='static' IPADDR='10.253.133.11' NETMASK='255.255.128.0' MTU='9000' LINK_REQUIRED='yes' POST_UP_SCRIPT="systemd:[email protected]" nid002792:~ # Now, "I would have thought" (tm) that MPICH, on "seeing" an interface that hadn't been START-ed, would not have considered it as a NIC, for NIC_SYMMETRY puposes? I am (more than) aware that I can prevent the messages about NIC_SYMMETRY inconsistencies, but that's not the issue here; the issue here is that MPICH seems to think a NIC that hasn't been START-ed is worthy of consideration. FWIW, it's cray-mpich/8.1.32 so MPICH 3.4a2, under the hood, where the hood belongs to an HPE/Cray EX, running SLES 15 SP6. The info from the cray-mpich/8.1.32 module says - Cray MPICH offers support for multiple NICs per node. Starting with version 8.0.8, by default Cray MPICH will use all available NICs on a node. but maybe their definiiton of "available" differs from the one that I have become accustomed to over the years, to wit: if it's not START-ed; it's not available? Interested to hear any thoughts on this? Kevin M. Buckley -- Supercomputing Systems Administrator Pawsey Supercomputing Centre PERTH Australia

1 0

How does MPICH dtermine available NICs?
by Kevin Buckley 09 Mar '26

09 Mar '26

1 0