October 2023 - discuss - mailman3.cels.anl.gov

Re: [mpich-discuss] Can't allocate comm memory for ch4, UCX
by Raffenetti, Ken 24 Oct '23

24 Oct '23

Hi Kurt, A little searching suggests that the 64kb limit you see in the log is due to the Slurm daemons starting before the increased limit is set. https://www.mail-archive.com/[email protected]/msg00960.html It seems you can remedy this by adding lines to the Slurm startup script to increase the locked memory limit before starting the daemons. Can you give it a try? Ken From: "Mccall, Kurt E. (MSFC-EV41) via discuss" <discuss(a)mpich.org> Reply-To: "discuss(a)mpich.org" <discuss(a)mpich.org> Date: Saturday, October 21, 2023 at 1:59 PM To: "discuss(a)mpich.org" <discuss(a)mpich.org> Cc: "Mccall, Kurt E. (MSFC-EV41)" <kurt.e.mccall(a)nasa.gov> Subject: [mpich-discuss] Can't allocate comm memory for ch4, UCX Any clue what is causing this problem? My job is failing on all compute nodes with this message: ib_iface.c:1060 UCX ERROR ibv_create_cq(cqe=4096) failed: Cannot allocate memory : Please set max locked memory (ulimit -l) to 'unlimited' (current: 64 kbytes) It doesn’t make sense because on all nodes except for the head node, the memory locking limit is “unlimited”: $ ssh n004 $ ulimit -Hl -Sl max locked memory (kbytes, -l) unlimited max locked memory (kbytes, -l) unlimited On the head node, the limit is still set at 64 kb, but I thought that it wouldn’t matter since the processes run on the compute nodes. I configured MPICH with ch4 and UCX 1.15.0: $ ../mpich-4.1.2/configure --prefix=/opt/mpich --with-device=ch4:ucx --with-ucx=/home/kmccall/ucx-1.15.0/install --with-slurm -enable-debuginfo --enable-g=debug 2>&1 | tee c.txt and my launch command is unremarkable: mpiexec \ -print-all-exitcodes \ -wdir ${work_dir} \ -np ${num_proc} \ -ppn 1 \ application… The cluster is running Red Hat Enterprise Linux release 8.6, slurm 20.11.8, and the compiler version is g++ 8.5.0. Just for reference, here is the compete error message sequenct from slurm.out SLURM_JOB_NODELIST = n[004-008] [1697913222.014184] [n004:39165:0] ib_iface.c:1060 UCX ERROR ibv_create_cq(cqe=4096) failed: Cannot allocate memory : Please set max locked memory (ulimit -l) to 'unlimited' (current: 64 kbytes) [1697913222.014288] [n006:38767:0] ib_iface.c:1060 UCX ERROR ibv_create_cq(cqe=4096) failed: Cannot allocate memory : Please set max locked memory (ulimit -l) to 'unlimited' (current: 64 kbytes) [1697913222.014374] [n005:39166:0] ib_iface.c:1060 UCX ERROR ibv_create_cq(cqe=4096) failed: Cannot allocate memory : Please set max locked memory (ulimit -l) to 'unlimited' (current: 64 kbytes) [1697913222.014851] [n008:38524:0] ib_iface.c:1060 UCX ERROR ibv_create_cq(cqe=4096) failed: Cannot allocate memory : Please set max locked memory (ulimit -l) to 'unlimited' (current: 64 kbytes) [1697913222.014807] [n007:38753:0] ib_iface.c:1060 UCX ERROR ibv_create_cq(cqe=4096) failed: Cannot allocate memory : Please set max locked memory (ulimit -l) to 'unlimited' (current: 64 kbytes) Abort(607769871) on node 0 (rank 0 in comm 0): Fatal error in internal_Init: Other MPI error, error stack: internal_Init(66).............: MPI_Init(argc=0x7ffe9e2b475c, argv=0x7ffe9e2b4750) failed MPII_Init_thread(257).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(770).........: MPIR_Comm_commit_internal(558): MPID_Comm_commit_pre_hook(158): MPIDI_UCX_init_world(252).....: init_worker(38)...............: ucx function returned with failed status(ucx_init.c 38 init_worker Input/output error) Abort(339334415) on node 2 (rank 2 in comm 0): Fatal error in internal_Init: Other MPI error, error stack: internal_Init(66).............: MPI_Init(argc=0x7ffd5fd79e5c, argv=0x7ffd5fd79e50) failed MPII_Init_thread(257).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(770).........: MPIR_Comm_commit_internal(558): MPID_Comm_commit_pre_hook(158): MPIDI_UCX_init_world(252).....: init_worker(38)...............: ucx function returned with failed status(ucx_init.c 38 init_worker Input/output error)

1 0

Re: [mpich-discuss] Configuring MPICH 4.1.2 without increasing the locked memory limit
by Raffenetti, Ken 24 Oct '23

24 Oct '23

Hi Kurt, This looks to be an issue allocating resources on infiniband device on the node. MPI_Init should not require any special system settings. Are you able to run infiniband diagnostics without any MPI library? Ibstatus should tell you if the IB card is online and what state it is in. From there, you could try running an ib_send_bw test across 2 nodes and verify that traffic is flowing. Ken From: "Mccall, Kurt E. (MSFC-EV41) via discuss" <discuss(a)mpich.org> Reply-To: "discuss(a)mpich.org" <discuss(a)mpich.org> Date: Wednesday, October 18, 2023 at 12:36 PM To: "discuss(a)mpich.org" <discuss(a)mpich.org> Cc: "Mccall, Kurt E. (MSFC-EV41)" <kurt.e.mccall(a)nasa.gov> Subject: [mpich-discuss] Configuring MPICH 4.1.2 without increasing the locked memory limit Hi, I have configured MPICH 4.1.2 with both –with-device=ch4:ofi and –with-device=ch4:ucx. My application fails in both cases when it can’t allocate enough memory. For –with-device=ch4:ofi : Unable to create send CQ of size 5080 on mlx5_0: Cannot allocate memory n001.cluster.pssclabs.com:rank0.NeedlesMpiMM: Unable to initialize verbs NIC /sys/class/infiniband/mlx5_0 (unit 0:0) n001.cluster.pssclabs.com:rank0: PSM3 can't open nic unit: 0 (err=23) Abort(606197135): Fatal error in internal_Init: Other MPI error, error stack: internal_Init(66)........: MPI_Init(argc=0x7ffc1cbd334c, argv=0x7ffc1cbd3340) failed MPII_Init_thread(234)....: MPID_Init(513)...........: MPIDI_OFI_init_local(604): create_vni_context(982)..: OFI endpoint open failed (ofi_init.c:982:create_vni_context:Cannot allocate memory) Configuring using –with-device=ch4:ucx, there was a very similar error involving /sys/class/infiniband/mlx5_0 that explicitly stating that the locked memory limit (ulimit -l) needs to be set to “unlimited”. Are there any other ch4 device configuration options that don’t require unlimited locked memory? Thanks, Kurt

1 0

Can't allocate comm memory for ch4, UCX
by Mccall, Kurt E. (MSFC-EV41) 21 Oct '23

21 Oct '23

Any clue what is causing this problem? My job is failing on all compute nodes with this message: ib_iface.c:1060 UCX ERROR ibv_create_cq(cqe=4096) failed: Cannot allocate memory : Please set max locked memory (ulimit -l) to 'unlimited' (current: 64 kbytes) It doesn't make sense because on all nodes except for the head node, the memory locking limit is "unlimited": $ ssh n004 $ ulimit -Hl -Sl max locked memory (kbytes, -l) unlimited max locked memory (kbytes, -l) unlimited On the head node, the limit is still set at 64 kb, but I thought that it wouldn't matter since the processes run on the compute nodes. I configured MPICH with ch4 and UCX 1.15.0: $ ../mpich-4.1.2/configure --prefix=/opt/mpich --with-device=ch4:ucx --with-ucx=/home/kmccall/ucx-1.15.0/install --with-slurm -enable-debuginfo --enable-g=debug 2>&1 | tee c.txt and my launch command is unremarkable: mpiexec \ -print-all-exitcodes \ -wdir ${work_dir} \ -np ${num_proc} \ -ppn 1 \ application... The cluster is running Red Hat Enterprise Linux release 8.6, slurm 20.11.8, and the compiler version is g++ 8.5.0. Just for reference, here is the compete error message sequenct from slurm.out SLURM_JOB_NODELIST = n[004-008] [1697913222.014184] [n004:39165:0] ib_iface.c:1060 UCX ERROR ibv_create_cq(cqe=4096) failed: Cannot allocate memory : Please set max locked memory (ulimit -l) to 'unlimited' (current: 64 kbytes) [1697913222.014288] [n006:38767:0] ib_iface.c:1060 UCX ERROR ibv_create_cq(cqe=4096) failed: Cannot allocate memory : Please set max locked memory (ulimit -l) to 'unlimited' (current: 64 kbytes) [1697913222.014374] [n005:39166:0] ib_iface.c:1060 UCX ERROR ibv_create_cq(cqe=4096) failed: Cannot allocate memory : Please set max locked memory (ulimit -l) to 'unlimited' (current: 64 kbytes) [1697913222.014851] [n008:38524:0] ib_iface.c:1060 UCX ERROR ibv_create_cq(cqe=4096) failed: Cannot allocate memory : Please set max locked memory (ulimit -l) to 'unlimited' (current: 64 kbytes) [1697913222.014807] [n007:38753:0] ib_iface.c:1060 UCX ERROR ibv_create_cq(cqe=4096) failed: Cannot allocate memory : Please set max locked memory (ulimit -l) to 'unlimited' (current: 64 kbytes) Abort(607769871) on node 0 (rank 0 in comm 0): Fatal error in internal_Init: Other MPI error, error stack: internal_Init(66).............: MPI_Init(argc=0x7ffe9e2b475c, argv=0x7ffe9e2b4750) failed MPII_Init_thread(257).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(770).........: MPIR_Comm_commit_internal(558): MPID_Comm_commit_pre_hook(158): MPIDI_UCX_init_world(252).....: init_worker(38)...............: ucx function returned with failed status(ucx_init.c 38 init_worker Input/output error) Abort(339334415) on node 2 (rank 2 in comm 0): Fatal error in internal_Init: Other MPI error, error stack: internal_Init(66).............: MPI_Init(argc=0x7ffd5fd79e5c, argv=0x7ffd5fd79e50) failed MPII_Init_thread(257).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(770).........: MPIR_Comm_commit_internal(558): MPID_Comm_commit_pre_hook(158): MPIDI_UCX_init_world(252).....: init_worker(38)...............: ucx function returned with failed status(ucx_init.c 38 init_worker Input/output error)

1 0

Configuring MPICH 4.1.2 without increasing the locked memory limit
by Mccall, Kurt E. (MSFC-EV41) 18 Oct '23

18 Oct '23

Hi, I have configured MPICH 4.1.2 with both -with-device=ch4:ofi and -with-device=ch4:ucx. My application fails in both cases when it can't allocate enough memory. For -with-device=ch4:ofi : Unable to create send CQ of size 5080 on mlx5_0: Cannot allocate memory n001.cluster.pssclabs.com:rank0.NeedlesMpiMM: Unable to initialize verbs NIC /sys/class/infiniband/mlx5_0 (unit 0:0) n001.cluster.pssclabs.com:rank0: PSM3 can't open nic unit: 0 (err=23) Abort(606197135): Fatal error in internal_Init: Other MPI error, error stack: internal_Init(66)........: MPI_Init(argc=0x7ffc1cbd334c, argv=0x7ffc1cbd3340) failed MPII_Init_thread(234)....: MPID_Init(513)...........: MPIDI_OFI_init_local(604): create_vni_context(982)..: OFI endpoint open failed (ofi_init.c:982:create_vni_context:Cannot allocate memory) Configuring using -with-device=ch4:ucx, there was a very similar error involving /sys/class/infiniband/mlx5_0 that explicitly stating that the locked memory limit (ulimit -l) needs to be set to "unlimited". Are there any other ch4 device configuration options that don't require unlimited locked memory? Thanks, Kurt

1 0

Re: [mpich-discuss] MPI_Comm_free assertion failure
by Zhou, Hui 10 Oct '23

10 Oct '23

`../mpich-4.1.2/src/mpid/ch3/src/ch3u_request.c at line 480: ` that line is ``` MPIDI_Datatype_get_info(rreq->dev.user_count, rreq->dev.datatype, dt_contig, userbuf_sz, dt_ptr, dt_true_lb); ``` That is a macro and the assertion is inside the macro expansion. Usually, the assertion indicates a wrong datatype handle. Do you think you can dump the datatype as hex value? -- Hui ________________________________ From: Mccall, Kurt E. (MSFC-EV41) via discuss <discuss(a)mpich.org> Sent: Tuesday, October 10, 2023 11:05 AM To: discuss(a)mpich.org <discuss(a)mpich.org> Cc: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov> Subject: [mpich-discuss] MPI_Comm_free assertion failure Before calling MPI_Finalize, I am calling MPI_Comm_free at both ends (processes) of an intercommunicator created by MPI_Intercomm_create and am crashing with the following error in one of the processes: Assertion failed in file ../mpich-4.1.2/src/mpid/ch3/src/ch3u_request.c at line 480: /opt/mpich/lib/libmpi.so.12(+0x2d3bf3) [0x7f6d10e1fbf3] /opt/mpich/lib/libmpi.so.12(+0x225124) [0x7f6d10d71124] /opt/mpich/lib/libmpi.so.12(+0x33016) [0x7f6d10b7f016] /opt/mpich/lib/libmpi.so.12(+0x29eada) [0x7f6d10deaada] /opt/mpich/lib/libmpi.so.12(+0x1dfb7c) [0x7f6d10d2bb7c] /opt/mpich/lib/libmpi.so.12(+0x1d78cd) [0x7f6d10d238cd] /opt/mpich/lib/libmpi.so.12(PMPI_Comm_free+0xe7) [0x7f6d10bfe6b7] /home/kmccall/Needles2/NeedlesMpiMM(_ZN7needles12IntercomList20handleFailedIntercomERSt17_Rb_tree_iteratorISt4pairIKiPNS_16IntercomListElemEEEiPKc+0xae) [0x4c348a] /home/kmccall/Needles2/NeedlesMpiMM(_ZN7needles12IntercomList19disconnectIntercomsERilb+0xcf) [0x4c36fb] /home/kmccall/Needles2/NeedlesMpiMM(_ZN7needles16NeedlesMpiMaster8finalizeEv+0x25f) [0x496e25] /home/kmccall/Needles2/NeedlesMpiMM(_ZN7needles16NeedlesMpiMaster4loopEv+0x74e) [0x49355a] /home/kmccall/Needles2/NeedlesMpiMM(main+0x4af) [0x48d1e5] /lib64/libc.so.6(__libc_start_main+0xf3) [0x7f6d0f186ca3] /home/kmccall/Needles2/NeedlesMpiMM(_start+0x2e) [0x48cc7e] internal ABORT - process 0 I looked in ch3u_request.c and there seems to be no assertion at line 480. If I forgo the MPI_Comm_free in the first process, the same error then occurs in the second process. Does this give you any clue as to what I am doing wrong? Thanks, Kurt

1 0

MPI_Comm_free assertion failure
by Mccall, Kurt E. (MSFC-EV41) 10 Oct '23

10 Oct '23

Before calling MPI_Finalize, I am calling MPI_Comm_free at both ends (processes) of an intercommunicator created by MPI_Intercomm_create and am crashing with the following error in one of the processes: Assertion failed in file ../mpich-4.1.2/src/mpid/ch3/src/ch3u_request.c at line 480: /opt/mpich/lib/libmpi.so.12(+0x2d3bf3) [0x7f6d10e1fbf3] /opt/mpich/lib/libmpi.so.12(+0x225124) [0x7f6d10d71124] /opt/mpich/lib/libmpi.so.12(+0x33016) [0x7f6d10b7f016] /opt/mpich/lib/libmpi.so.12(+0x29eada) [0x7f6d10deaada] /opt/mpich/lib/libmpi.so.12(+0x1dfb7c) [0x7f6d10d2bb7c] /opt/mpich/lib/libmpi.so.12(+0x1d78cd) [0x7f6d10d238cd] /opt/mpich/lib/libmpi.so.12(PMPI_Comm_free+0xe7) [0x7f6d10bfe6b7] /home/kmccall/Needles2/NeedlesMpiMM(_ZN7needles12IntercomList20handleFailedIntercomERSt17_Rb_tree_iteratorISt4pairIKiPNS_16IntercomListElemEEEiPKc+0xae) [0x4c348a] /home/kmccall/Needles2/NeedlesMpiMM(_ZN7needles12IntercomList19disconnectIntercomsERilb+0xcf) [0x4c36fb] /home/kmccall/Needles2/NeedlesMpiMM(_ZN7needles16NeedlesMpiMaster8finalizeEv+0x25f) [0x496e25] /home/kmccall/Needles2/NeedlesMpiMM(_ZN7needles16NeedlesMpiMaster4loopEv+0x74e) [0x49355a] /home/kmccall/Needles2/NeedlesMpiMM(main+0x4af) [0x48d1e5] /lib64/libc.so.6(__libc_start_main+0xf3) [0x7f6d0f186ca3] /home/kmccall/Needles2/NeedlesMpiMM(_start+0x2e) [0x48cc7e] internal ABORT - process 0 I looked in ch3u_request.c and there seems to be no assertion at line 480. If I forgo the MPI_Comm_free in the first process, the same error then occurs in the second process. Does this give you any clue as to what I am doing wrong? Thanks, Kurt

1 0