On 2026/03/11 10:43, McMahon, Kim wrote:
> 
> Cray MPICH detects "available" NICS on a node by calling the libfabric fi_getinto() call.
> ...
> I suspect your method for disabling the NIC via "STARTMODE='off'" does not fail these checks.

Indeed, Kim: indeed.

But anyroad,

I believe that I have just found a way to REALLY DISABLE the 2nd
NIC, as the node provisions, having tried quite a few that didn't
see it disabled, as far as Cray MPICH'a detection was concerned.


The "trick" seems to be to leave the STARTMODE, in

   /etc/sysconfig/network/ifcfg-hsn1

as it was before I thought to override its default pre-boot
creation, via,

   /etc/opt/sgi/conf.d/15-network-setup

so we now have

   # Kevin says let it come up
   STARTMODE='auto'

but then replace the ifcfg's POST_UP_SCRIPT argument, which
would normally be

   POST_UP_SCRIPT="systemd:cm-slingshot-ama@.service"

with

   POST_UP_SCRIPT="wicked:post-up/disable-hsn1-script"

where that script just runs an

   /usr/sbin/wicked --systemd ifdown <interface>


FWIW, I had even tried something a bit more "old school"
than using the wicked-newness, first, vis:

   POST_UP_SCRIPT="compat:suse:disable-hsn1-script"

with the same script payload, but that hadn't worked.
(Too old school perhaps?)


For completeness, here's what that last approach achieves:

Earlier attempts were leaving the interface thus:

nid002792:~ # /usr/sbin/wicked ifstatus hsn1
hsn1            device-unconfigured
       link:     #6, state up, mtu 9000
       type:     ethernet, hwaddr 02:00:00:00:60:33
nid002792:~ #

whereas, using that "new-fangled" wicked script, sees

nid002792:~ # /usr/sbin/wicked ifstatus hsn1
hsn1            device-unconfigured
       link:     #6, state down, mtu 9000
       type:     ethernet, hwaddr 02:00:00:00:60:33
nid002792:~ #

Pay special attention to the "state down".


As we only need to do this for a few nodes within the
EX here (bit of a much longer story!), I believe that
this approach will do what we need.

Rest assured though, that if it doesn't then, as I was
down to my last idea, we will probably raise it with
HPE/Cray, given that the underlying issue does seem to
have its provenance there, and not in upstream MPICH.

Thanks for the insight above,

Kevin M. Buckley
--
Supercomputing Systems Administrator
Pawsey Supercomputing Centre
PERTH
Australia