Hi Kurt, This looks to be an issue allocating resources on infiniband device on the node. MPI_Init should not require any special system settings. Are you able to run infiniband diagnostics without any MPI library? Ibstatus should tell you if the IB card is online and what state it is in. From there, you could try running an ib_send_bw test across 2 nodes and verify that traffic is flowing. Ken From: "Mccall, Kurt E. (MSFC-EV41) via discuss" <[email protected]> Reply-To: "[email protected]" <[email protected]> Date: Wednesday, October 18, 2023 at 12:36 PM To: "[email protected]" <[email protected]> Cc: "Mccall, Kurt E. (MSFC-EV41)" <[email protected]> Subject: [mpich-discuss] Configuring MPICH 4.1.2 without increasing the locked memory limit Hi, I have configured MPICH 4.1.2 with both –with-device=ch4:ofi and –with-device=ch4:ucx. My application fails in both cases when it can’t allocate enough memory. For –with-device=ch4:ofi : Unable to create send CQ of size 5080 on mlx5_0: Cannot allocate memory n001.cluster.pssclabs.com:rank0.NeedlesMpiMM: Unable to initialize verbs NIC /sys/class/infiniband/mlx5_0 (unit 0:0) n001.cluster.pssclabs.com:rank0: PSM3 can't open nic unit: 0 (err=23) Abort(606197135): Fatal error in internal_Init: Other MPI error, error stack: internal_Init(66)........: MPI_Init(argc=0x7ffc1cbd334c, argv=0x7ffc1cbd3340) failed MPII_Init_thread(234)....: MPID_Init(513)...........: MPIDI_OFI_init_local(604): create_vni_context(982)..: OFI endpoint open failed (ofi_init.c:982:create_vni_context:Cannot allocate memory) Configuring using –with-device=ch4:ucx, there was a very similar error involving /sys/class/infiniband/mlx5_0 that explicitly stating that the locked memory limit (ulimit -l) needs to be set to “unlimited”. Are there any other ch4 device configuration options that don’t require unlimited locked memory? Thanks, Kurt