The ucx netmod provides better support on infiniband. Could you try configure it with --with-device=ch4:ucx​? -- Hui ________________________________ From: Mccall, Kurt E. (MSFC-EV41) via discuss <[email protected]> Sent: Monday, July 8, 2024 3:00 PM To: [email protected] <[email protected]> Cc: Mccall, Kurt E. (MSFC-EV41) <[email protected]> Subject: [mpich-discuss] MPICH fails to allocate memory at beginning of job I configured MPICH 4. 1. 2 as follows. Any clue as to what would fix the error below? $ cd mpich-4. 1. 2-build $ ./mpich-4. 1. 2/configure --prefix=/opt/mpich --with-device=ch4: ofi --with-libfabric-embedded --with-slurm -enable-debuginfo --enable-g=debug ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd I configured MPICH 4.1.2 as follows. Any clue as to what would fix the error below? $ cd mpich-4.1.2-build $ ./mpich-4.1.2/configure --prefix=/opt/mpich --with-device=ch4:ofi --with-libfabric-embedded --with-slurm -enable-debuginfo --enable-g=debug 2>&1 | tee c.txt When I run a job, the error message on each node is: n007.cluster.pssclabs.com:rank6.HaystackMpiMM: Failed to modify UD QP to INIT on mlx5_0: Operation not permitted Abort(337761679): Fatal error in internal_Init: Other MPI error, error stack: internal_Init(66)........: MPI_Init(argc=0x7ffd93a663ec, argv=0x7ffd93a663e0) failed MPII_Init_thread(234)....: MPID_Init(513)...........: MPIDI_OFI_init_local(604): create_vni_context(982)..: OFI endpoint open failed (ofi_init.c:982:create_vni_context:Cannot allocate memory) The run command is: mpiexec -launcher ssh \ -print-all-exitcodes \ -wdir ${work_dir} \ -np ${num_proc} \ -ppn 1 \ my_program … Thanks, Kurt