The NIC symmetry test is specific to HPE Cray MPICH. As a workaround, try setting MPICH_OFI_NUM_NICS=1. — Michael Raymond HPE HPC programming environment From: Kevin Buckley via discuss <[email protected]> Date: Tuesday, March 10, 2026 at 00:47 To: [email protected] <[email protected]> Cc: Kevin Buckley <[email protected]> Subject: Re: [mpich-discuss] How does MPICH dtermine available NICs? This Message Is From an External Sender This message came from outside your organization. On 2026/03/09 23:24, Zhou, Hui wrote:

Could you try the upstream MPICH?

I followed you suggestion to try the upstream MPICH. Trying to configure 3.4a2, the version "inside" the cray-mpich, modulo any vendor updates made to that base release, saw ... configure: error: The Fortran compiler gfortran will not compile files that\ call the same routine with arguments of different types. $ $ which gfortran /opt/cray/pe/gcc-native/14/bin/gfortran $ Wasn't sure if that meant that the MPICH code is too old to make correct compiler checks, or that the compiler is too new to respond them - so I shelved that line of attack. Was able to configure and build 5.0.0, and have used that to compile and run the same noddy "connectivity_c.c" example code that I had been seeing the issue with. It all seems to work without any warnings (as does compiling/ running within an OpenMPI 5.0.3 environment), in that a 2-node job spanning a node with just one NIC, and the dual-NIC node with one NIC disabled, completes, as does a 2-node job spanning the node with just one NIC, and a dual-NIC node with neither NIC disabled. Would you though, expect the vanilla MPICH 5.0.0 to have warned me about the NIC_SYMMETRY inconsistencies? Your comment in the original reply,

The upstream MPICH should be fine with different number of NICs on different nodes. By default, the process picks a nic that is closest to process's CPU affinity; or if you have multiple processes on the same node, each process will try pick a different nic in a round-robin fashion. Usually a disabled nic won't be selected during init,

would suggest that "later" (than cray-mpich's 3.4a2 starting point, plus whatver enhancements Cray and/or HPE have since added) MPICH-s would not bother to give the warning, as those versions would "just do the right thing" (tm)? If so, think it's sounding as though the consideration of the disabled-at-boot-time NIC as somehow being "available" for MPI traffic, is tied into something that the vendor is doing: somewhere within their overall MPI+Slingshot stack. Thanks again for fielding this one, Kevin M. Buckley -- Supercomputing Systems Administrator Pawsey Supercomputing Centre PERTH Australia