- discuss - mailman3.cels.anl.gov

Re: [mpich-discuss] MPI user-defined datatype size
by George Bosilca 04 Dec '23

04 Dec '23

MPI does not know about the padding because it does not really create a struct in the same sense as the compiler. As a result, you need to resize the datatype to account for the padding. Check the examples 5.16 and 5.17 in the MPI standard to see how to use it. For your particular example, replacing the last 2 lines of code with: MPI_Type_create_struct(struct_len, block_len, displacements, types, &tmp_ddt); MPI_Type_create_resized(tmp_ddt, 0, sizeof(), &mpi_data_type); MPI_Type_connit(mpi_data_type); should make the MPI struct match the compiler struct. While you can manipulate the pointers directly and compute the displacements by hand, you should not use a size_t for that. MPI_Aint is a signed type, ssize_t or some form of ptrdiff_t. Instead you should use the MPI suggested way via MPI_Get_address (to obtain the address of an object as an MPI_Aint), and then MPI_Aint_diff to compute relative displacements. George. On Sun, Dec 3, 2023 at 3:55 PM Mccall, Kurt E. (MSFC-EV41) via discuss < discuss(a)mpich.org> wrote: > Hi, > > > > I’m running MPICH 4.1.2. > > > > Should a MPI user-defined datatype created by MPI_Type_create_struct have > the same size as the C++ object on which it is based? Comparing > sizeof(cpp_obj) and the output > > From MPI_Type_size(mpi_data_type_, &size1), the C++ object is four bytes > larger than the MPI datatype. I don’t know if this is because I have to > do something special to take > > padding into account when I create the MPI user datatype. > > > > Here is my C++ message class: > > > > struct MsgTupleSeq > > { > > static const int N_INDICES_MAX_ = 4; // culprit? > > > > int n_vars_; > > int n_ndxs_; > > short start_[N_INDICES_MAX_]; > > short end_[N_INDICES_MAX_]; > > short current_[N_INDICES_MAX_]; > > long long n_tuples_; > > int unique_id_; > > int subordinate_id_; > > int opcode_; > > char error_msg_[512]; > > } > > > > That is 564 bytes for the non-static members, but such an object is 568 > bytes long with the 4 bytes of padding that the compiler inserts. > > > > I define the MPI user datatype with the code below, using the MsgTupleSeq > object to compute the offsets. MPI_Type_size returns 564 bytes for the > user datatype, > > so it seems like it isn’t seeing the padding. > > > > MsgTupleSeq obj; > > MPI_Datatype mpi_data_type; > > int struct_len = 10, i = 0; > > > > int block_len[struct_len]; > > MPI_Datatype types[struct_len]; > > MPI_Aint displacements[struct_len]; > > > > // the integer "n_vars_" member > > block_len[i] = 1; > > types[i] = MPI_INT; > > displacements[i] = (size_t) &obj.n_vars_ - (size_t) &obj; > > > > // the integer "n_ndxs_" member > > ++i; > > block_len[i] = 1; > > types[i] = MPI_INT; > > displacements[i] = (size_t) &obj.n_ndxs_ - (size_t) &obj; > > > > // the short integer "start_" array member > > ++i; > > block_len[i] = N_INDICES_MAX_; > > types[i] = MPI_SHORT; > > displacements[i] = (size_t) &obj.start_ - (size_t) &obj; > > > > // the short integer "end_" array member > > ++i; > > block_len[i] = N_INDICES_MAX_; > > types[i] = MPI_SHORT; > > displacements[i] = (size_t) &obj.end_ - (size_t) &obj; > > > > // the short integer "current_" array member > > ++i; > > block_len[i] = N_INDICES_MAX_; > > types[i] = MPI_SHORT; > > displacements[i] = (size_t) &obj.current_ - (size_t) &obj; > > > > // the long long integer "n_tuples_" member > > ++i; > > block_len[i] = 1; > > types[i] = MPI_LONG_LONG; > > displacements[i] = (size_t) &obj.n_tuples_ - (size_t) &obj; > > > > // the integer "unique_id_" member > > ++i; > > block_len[i] = 1; > > types[i] = MPI_INT; > > displacements[i] = (size_t) &obj.unique_id_ - (size_t) &obj; > > > > // the integer "subordinate_id_" member > > ++i; > > block_len[i] = 1; > > types[i] = MPI_INT; > > displacements[i] = (size_t) &obj.subordinate_id_ - (size_t) &obj; > > > > // the integer "opcode_" member > > ++i; > > block_len[i] = 1; > > types[i] = MPI_INT; > > displacements[i] = (size_t) &obj.opcode_ - (size_t) &obj; > > > > // the char array "error_msg_" member > > ++i; > > block_len[i] = 512; > > types[i] = MPI_CHAR; > > displacements[i] = (size_t) &obj.error_msg_[0] - (size_t) &obj; > > > > MPI_Type_create_struct(struct_len, block_len, displacements, types, > &mpi_data_type); > > MPI_Type_commit(&mpi_data_type); > > > > > > > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss >

1 0

MPI user-defined datatype size
by Mccall, Kurt E. (MSFC-EV41) 03 Dec '23

03 Dec '23

Hi, I'm running MPICH 4.1.2. Should a MPI user-defined datatype created by MPI_Type_create_struct have the same size as the C++ object on which it is based? Comparing sizeof(cpp_obj) and the output >From MPI_Type_size(mpi_data_type_, &size1), the C++ object is four bytes larger than the MPI datatype. I don't know if this is because I have to do something special to take padding into account when I create the MPI user datatype. Here is my C++ message class: struct MsgTupleSeq { static const int N_INDICES_MAX_ = 4; // culprit? int n_vars_; int n_ndxs_; short start_[N_INDICES_MAX_]; short end_[N_INDICES_MAX_]; short current_[N_INDICES_MAX_]; long long n_tuples_; int unique_id_; int subordinate_id_; int opcode_; char error_msg_[512]; } That is 564 bytes for the non-static members, but such an object is 568 bytes long with the 4 bytes of padding that the compiler inserts. I define the MPI user datatype with the code below, using the MsgTupleSeq object to compute the offsets. MPI_Type_size returns 564 bytes for the user datatype, so it seems like it isn't seeing the padding. MsgTupleSeq obj; MPI_Datatype mpi_data_type; int struct_len = 10, i = 0; int block_len[struct_len]; MPI_Datatype types[struct_len]; MPI_Aint displacements[struct_len]; // the integer "n_vars_" member block_len[i] = 1; types[i] = MPI_INT; displacements[i] = (size_t) &obj.n_vars_ - (size_t) &obj; // the integer "n_ndxs_" member ++i; block_len[i] = 1; types[i] = MPI_INT; displacements[i] = (size_t) &obj.n_ndxs_ - (size_t) &obj; // the short integer "start_" array member ++i; block_len[i] = N_INDICES_MAX_; types[i] = MPI_SHORT; displacements[i] = (size_t) &obj.start_ - (size_t) &obj; // the short integer "end_" array member ++i; block_len[i] = N_INDICES_MAX_; types[i] = MPI_SHORT; displacements[i] = (size_t) &obj.end_ - (size_t) &obj; // the short integer "current_" array member ++i; block_len[i] = N_INDICES_MAX_; types[i] = MPI_SHORT; displacements[i] = (size_t) &obj.current_ - (size_t) &obj; // the long long integer "n_tuples_" member ++i; block_len[i] = 1; types[i] = MPI_LONG_LONG; displacements[i] = (size_t) &obj.n_tuples_ - (size_t) &obj; // the integer "unique_id_" member ++i; block_len[i] = 1; types[i] = MPI_INT; displacements[i] = (size_t) &obj.unique_id_ - (size_t) &obj; // the integer "subordinate_id_" member ++i; block_len[i] = 1; types[i] = MPI_INT; displacements[i] = (size_t) &obj.subordinate_id_ - (size_t) &obj; // the integer "opcode_" member ++i; block_len[i] = 1; types[i] = MPI_INT; displacements[i] = (size_t) &obj.opcode_ - (size_t) &obj; // the char array "error_msg_" member ++i; block_len[i] = 512; types[i] = MPI_CHAR; displacements[i] = (size_t) &obj.error_msg_[0] - (size_t) &obj; MPI_Type_create_struct(struct_len, block_len, displacements, types, &mpi_data_type); MPI_Type_commit(&mpi_data_type);

1 0

Slides from the SC23 MPICH BoF now available
by Raffenetti, Ken 01 Dec '23

01 Dec '23

Hi all, Thank again to all the speakers and everyone who attended the MPICH BoF. It was a lively event :). We have posted the slides here for anyone to access: https://github.com/pmodels/mpich/wiki/2023-Supercomputing-BoF Cheers, Ken

1 0

Join us for the MPICH BoF at SC23! Wednesday, 11/15 @ 5:15pm MST
by Raffenetti, Ken 10 Nov '23

10 Nov '23

All, The MPICH team will host an SC23 Birds-of-a-Feather session Wednesday, November 15th at 5:15pm MST. We hope to see you there to discuss the latest developments in MPICH, and to hear presentations from MPICH partners and users. Agenda: * ANL Update – Yanfei Guo * Partner Presentations * Maria Garzaran (Intel) * Hari Subramoni (MVAPICH2) * Heidi Poxon (Amazon Web Services) * Simon Pickartz (ParTec AG) * Jim Dinan (NVIDIA) * Michael Raymond (HPE) * User Presentations * Paul Fischer (UIUC, Nek5000/NekRS) * Junchao Zhang (ANL, PETSc) * Q&A More details at https://github.com/pmodels/mpich/wiki/2023-Supercomputing-BoF. Thanks, The MPICH team

1 0

Re: [mpich-discuss] Can't allocate comm memory for ch4, UCX
by Raffenetti, Ken 24 Oct '23

24 Oct '23

Hi Kurt, A little searching suggests that the 64kb limit you see in the log is due to the Slurm daemons starting before the increased limit is set. https://www.mail-archive.com/[email protected]/msg00960.html It seems you can remedy this by adding lines to the Slurm startup script to increase the locked memory limit before starting the daemons. Can you give it a try? Ken From: "Mccall, Kurt E. (MSFC-EV41) via discuss" <discuss(a)mpich.org> Reply-To: "discuss(a)mpich.org" <discuss(a)mpich.org> Date: Saturday, October 21, 2023 at 1:59 PM To: "discuss(a)mpich.org" <discuss(a)mpich.org> Cc: "Mccall, Kurt E. (MSFC-EV41)" <kurt.e.mccall(a)nasa.gov> Subject: [mpich-discuss] Can't allocate comm memory for ch4, UCX Any clue what is causing this problem? My job is failing on all compute nodes with this message: ib_iface.c:1060 UCX ERROR ibv_create_cq(cqe=4096) failed: Cannot allocate memory : Please set max locked memory (ulimit -l) to 'unlimited' (current: 64 kbytes) It doesn’t make sense because on all nodes except for the head node, the memory locking limit is “unlimited”: $ ssh n004 $ ulimit -Hl -Sl max locked memory (kbytes, -l) unlimited max locked memory (kbytes, -l) unlimited On the head node, the limit is still set at 64 kb, but I thought that it wouldn’t matter since the processes run on the compute nodes. I configured MPICH with ch4 and UCX 1.15.0: $ ../mpich-4.1.2/configure --prefix=/opt/mpich --with-device=ch4:ucx --with-ucx=/home/kmccall/ucx-1.15.0/install --with-slurm -enable-debuginfo --enable-g=debug 2>&1 | tee c.txt and my launch command is unremarkable: mpiexec \ -print-all-exitcodes \ -wdir ${work_dir} \ -np ${num_proc} \ -ppn 1 \ application… The cluster is running Red Hat Enterprise Linux release 8.6, slurm 20.11.8, and the compiler version is g++ 8.5.0. Just for reference, here is the compete error message sequenct from slurm.out SLURM_JOB_NODELIST = n[004-008] [1697913222.014184] [n004:39165:0] ib_iface.c:1060 UCX ERROR ibv_create_cq(cqe=4096) failed: Cannot allocate memory : Please set max locked memory (ulimit -l) to 'unlimited' (current: 64 kbytes) [1697913222.014288] [n006:38767:0] ib_iface.c:1060 UCX ERROR ibv_create_cq(cqe=4096) failed: Cannot allocate memory : Please set max locked memory (ulimit -l) to 'unlimited' (current: 64 kbytes) [1697913222.014374] [n005:39166:0] ib_iface.c:1060 UCX ERROR ibv_create_cq(cqe=4096) failed: Cannot allocate memory : Please set max locked memory (ulimit -l) to 'unlimited' (current: 64 kbytes) [1697913222.014851] [n008:38524:0] ib_iface.c:1060 UCX ERROR ibv_create_cq(cqe=4096) failed: Cannot allocate memory : Please set max locked memory (ulimit -l) to 'unlimited' (current: 64 kbytes) [1697913222.014807] [n007:38753:0] ib_iface.c:1060 UCX ERROR ibv_create_cq(cqe=4096) failed: Cannot allocate memory : Please set max locked memory (ulimit -l) to 'unlimited' (current: 64 kbytes) Abort(607769871) on node 0 (rank 0 in comm 0): Fatal error in internal_Init: Other MPI error, error stack: internal_Init(66).............: MPI_Init(argc=0x7ffe9e2b475c, argv=0x7ffe9e2b4750) failed MPII_Init_thread(257).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(770).........: MPIR_Comm_commit_internal(558): MPID_Comm_commit_pre_hook(158): MPIDI_UCX_init_world(252).....: init_worker(38)...............: ucx function returned with failed status(ucx_init.c 38 init_worker Input/output error) Abort(339334415) on node 2 (rank 2 in comm 0): Fatal error in internal_Init: Other MPI error, error stack: internal_Init(66).............: MPI_Init(argc=0x7ffd5fd79e5c, argv=0x7ffd5fd79e50) failed MPII_Init_thread(257).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(770).........: MPIR_Comm_commit_internal(558): MPID_Comm_commit_pre_hook(158): MPIDI_UCX_init_world(252).....: init_worker(38)...............: ucx function returned with failed status(ucx_init.c 38 init_worker Input/output error)

1 0

Re: [mpich-discuss] Configuring MPICH 4.1.2 without increasing the locked memory limit
by Raffenetti, Ken 24 Oct '23

24 Oct '23

Hi Kurt, This looks to be an issue allocating resources on infiniband device on the node. MPI_Init should not require any special system settings. Are you able to run infiniband diagnostics without any MPI library? Ibstatus should tell you if the IB card is online and what state it is in. From there, you could try running an ib_send_bw test across 2 nodes and verify that traffic is flowing. Ken From: "Mccall, Kurt E. (MSFC-EV41) via discuss" <discuss(a)mpich.org> Reply-To: "discuss(a)mpich.org" <discuss(a)mpich.org> Date: Wednesday, October 18, 2023 at 12:36 PM To: "discuss(a)mpich.org" <discuss(a)mpich.org> Cc: "Mccall, Kurt E. (MSFC-EV41)" <kurt.e.mccall(a)nasa.gov> Subject: [mpich-discuss] Configuring MPICH 4.1.2 without increasing the locked memory limit Hi, I have configured MPICH 4.1.2 with both –with-device=ch4:ofi and –with-device=ch4:ucx. My application fails in both cases when it can’t allocate enough memory. For –with-device=ch4:ofi : Unable to create send CQ of size 5080 on mlx5_0: Cannot allocate memory n001.cluster.pssclabs.com:rank0.NeedlesMpiMM: Unable to initialize verbs NIC /sys/class/infiniband/mlx5_0 (unit 0:0) n001.cluster.pssclabs.com:rank0: PSM3 can't open nic unit: 0 (err=23) Abort(606197135): Fatal error in internal_Init: Other MPI error, error stack: internal_Init(66)........: MPI_Init(argc=0x7ffc1cbd334c, argv=0x7ffc1cbd3340) failed MPII_Init_thread(234)....: MPID_Init(513)...........: MPIDI_OFI_init_local(604): create_vni_context(982)..: OFI endpoint open failed (ofi_init.c:982:create_vni_context:Cannot allocate memory) Configuring using –with-device=ch4:ucx, there was a very similar error involving /sys/class/infiniband/mlx5_0 that explicitly stating that the locked memory limit (ulimit -l) needs to be set to “unlimited”. Are there any other ch4 device configuration options that don’t require unlimited locked memory? Thanks, Kurt

1 0

Can't allocate comm memory for ch4, UCX
by Mccall, Kurt E. (MSFC-EV41) 21 Oct '23

21 Oct '23

Any clue what is causing this problem? My job is failing on all compute nodes with this message: ib_iface.c:1060 UCX ERROR ibv_create_cq(cqe=4096) failed: Cannot allocate memory : Please set max locked memory (ulimit -l) to 'unlimited' (current: 64 kbytes) It doesn't make sense because on all nodes except for the head node, the memory locking limit is "unlimited": $ ssh n004 $ ulimit -Hl -Sl max locked memory (kbytes, -l) unlimited max locked memory (kbytes, -l) unlimited On the head node, the limit is still set at 64 kb, but I thought that it wouldn't matter since the processes run on the compute nodes. I configured MPICH with ch4 and UCX 1.15.0: $ ../mpich-4.1.2/configure --prefix=/opt/mpich --with-device=ch4:ucx --with-ucx=/home/kmccall/ucx-1.15.0/install --with-slurm -enable-debuginfo --enable-g=debug 2>&1 | tee c.txt and my launch command is unremarkable: mpiexec \ -print-all-exitcodes \ -wdir ${work_dir} \ -np ${num_proc} \ -ppn 1 \ application... The cluster is running Red Hat Enterprise Linux release 8.6, slurm 20.11.8, and the compiler version is g++ 8.5.0. Just for reference, here is the compete error message sequenct from slurm.out SLURM_JOB_NODELIST = n[004-008] [1697913222.014184] [n004:39165:0] ib_iface.c:1060 UCX ERROR ibv_create_cq(cqe=4096) failed: Cannot allocate memory : Please set max locked memory (ulimit -l) to 'unlimited' (current: 64 kbytes) [1697913222.014288] [n006:38767:0] ib_iface.c:1060 UCX ERROR ibv_create_cq(cqe=4096) failed: Cannot allocate memory : Please set max locked memory (ulimit -l) to 'unlimited' (current: 64 kbytes) [1697913222.014374] [n005:39166:0] ib_iface.c:1060 UCX ERROR ibv_create_cq(cqe=4096) failed: Cannot allocate memory : Please set max locked memory (ulimit -l) to 'unlimited' (current: 64 kbytes) [1697913222.014851] [n008:38524:0] ib_iface.c:1060 UCX ERROR ibv_create_cq(cqe=4096) failed: Cannot allocate memory : Please set max locked memory (ulimit -l) to 'unlimited' (current: 64 kbytes) [1697913222.014807] [n007:38753:0] ib_iface.c:1060 UCX ERROR ibv_create_cq(cqe=4096) failed: Cannot allocate memory : Please set max locked memory (ulimit -l) to 'unlimited' (current: 64 kbytes) Abort(607769871) on node 0 (rank 0 in comm 0): Fatal error in internal_Init: Other MPI error, error stack: internal_Init(66).............: MPI_Init(argc=0x7ffe9e2b475c, argv=0x7ffe9e2b4750) failed MPII_Init_thread(257).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(770).........: MPIR_Comm_commit_internal(558): MPID_Comm_commit_pre_hook(158): MPIDI_UCX_init_world(252).....: init_worker(38)...............: ucx function returned with failed status(ucx_init.c 38 init_worker Input/output error) Abort(339334415) on node 2 (rank 2 in comm 0): Fatal error in internal_Init: Other MPI error, error stack: internal_Init(66).............: MPI_Init(argc=0x7ffd5fd79e5c, argv=0x7ffd5fd79e50) failed MPII_Init_thread(257).........: MPIR_init_comm_world(34)......: MPIR_Comm_commit(770).........: MPIR_Comm_commit_internal(558): MPID_Comm_commit_pre_hook(158): MPIDI_UCX_init_world(252).....: init_worker(38)...............: ucx function returned with failed status(ucx_init.c 38 init_worker Input/output error)

1 0

Configuring MPICH 4.1.2 without increasing the locked memory limit
by Mccall, Kurt E. (MSFC-EV41) 18 Oct '23

18 Oct '23

Hi, I have configured MPICH 4.1.2 with both -with-device=ch4:ofi and -with-device=ch4:ucx. My application fails in both cases when it can't allocate enough memory. For -with-device=ch4:ofi : Unable to create send CQ of size 5080 on mlx5_0: Cannot allocate memory n001.cluster.pssclabs.com:rank0.NeedlesMpiMM: Unable to initialize verbs NIC /sys/class/infiniband/mlx5_0 (unit 0:0) n001.cluster.pssclabs.com:rank0: PSM3 can't open nic unit: 0 (err=23) Abort(606197135): Fatal error in internal_Init: Other MPI error, error stack: internal_Init(66)........: MPI_Init(argc=0x7ffc1cbd334c, argv=0x7ffc1cbd3340) failed MPII_Init_thread(234)....: MPID_Init(513)...........: MPIDI_OFI_init_local(604): create_vni_context(982)..: OFI endpoint open failed (ofi_init.c:982:create_vni_context:Cannot allocate memory) Configuring using -with-device=ch4:ucx, there was a very similar error involving /sys/class/infiniband/mlx5_0 that explicitly stating that the locked memory limit (ulimit -l) needs to be set to "unlimited". Are there any other ch4 device configuration options that don't require unlimited locked memory? Thanks, Kurt

1 0

Re: [mpich-discuss] MPI_Comm_free assertion failure
by Zhou, Hui 10 Oct '23

10 Oct '23

`../mpich-4.1.2/src/mpid/ch3/src/ch3u_request.c at line 480: ` that line is ``` MPIDI_Datatype_get_info(rreq->dev.user_count, rreq->dev.datatype, dt_contig, userbuf_sz, dt_ptr, dt_true_lb); ``` That is a macro and the assertion is inside the macro expansion. Usually, the assertion indicates a wrong datatype handle. Do you think you can dump the datatype as hex value? -- Hui ________________________________ From: Mccall, Kurt E. (MSFC-EV41) via discuss <discuss(a)mpich.org> Sent: Tuesday, October 10, 2023 11:05 AM To: discuss(a)mpich.org <discuss(a)mpich.org> Cc: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov> Subject: [mpich-discuss] MPI_Comm_free assertion failure Before calling MPI_Finalize, I am calling MPI_Comm_free at both ends (processes) of an intercommunicator created by MPI_Intercomm_create and am crashing with the following error in one of the processes: Assertion failed in file ../mpich-4.1.2/src/mpid/ch3/src/ch3u_request.c at line 480: /opt/mpich/lib/libmpi.so.12(+0x2d3bf3) [0x7f6d10e1fbf3] /opt/mpich/lib/libmpi.so.12(+0x225124) [0x7f6d10d71124] /opt/mpich/lib/libmpi.so.12(+0x33016) [0x7f6d10b7f016] /opt/mpich/lib/libmpi.so.12(+0x29eada) [0x7f6d10deaada] /opt/mpich/lib/libmpi.so.12(+0x1dfb7c) [0x7f6d10d2bb7c] /opt/mpich/lib/libmpi.so.12(+0x1d78cd) [0x7f6d10d238cd] /opt/mpich/lib/libmpi.so.12(PMPI_Comm_free+0xe7) [0x7f6d10bfe6b7] /home/kmccall/Needles2/NeedlesMpiMM(_ZN7needles12IntercomList20handleFailedIntercomERSt17_Rb_tree_iteratorISt4pairIKiPNS_16IntercomListElemEEEiPKc+0xae) [0x4c348a] /home/kmccall/Needles2/NeedlesMpiMM(_ZN7needles12IntercomList19disconnectIntercomsERilb+0xcf) [0x4c36fb] /home/kmccall/Needles2/NeedlesMpiMM(_ZN7needles16NeedlesMpiMaster8finalizeEv+0x25f) [0x496e25] /home/kmccall/Needles2/NeedlesMpiMM(_ZN7needles16NeedlesMpiMaster4loopEv+0x74e) [0x49355a] /home/kmccall/Needles2/NeedlesMpiMM(main+0x4af) [0x48d1e5] /lib64/libc.so.6(__libc_start_main+0xf3) [0x7f6d0f186ca3] /home/kmccall/Needles2/NeedlesMpiMM(_start+0x2e) [0x48cc7e] internal ABORT - process 0 I looked in ch3u_request.c and there seems to be no assertion at line 480. If I forgo the MPI_Comm_free in the first process, the same error then occurs in the second process. Does this give you any clue as to what I am doing wrong? Thanks, Kurt

1 0

MPI_Comm_free assertion failure
by Mccall, Kurt E. (MSFC-EV41) 10 Oct '23

10 Oct '23

Before calling MPI_Finalize, I am calling MPI_Comm_free at both ends (processes) of an intercommunicator created by MPI_Intercomm_create and am crashing with the following error in one of the processes: Assertion failed in file ../mpich-4.1.2/src/mpid/ch3/src/ch3u_request.c at line 480: /opt/mpich/lib/libmpi.so.12(+0x2d3bf3) [0x7f6d10e1fbf3] /opt/mpich/lib/libmpi.so.12(+0x225124) [0x7f6d10d71124] /opt/mpich/lib/libmpi.so.12(+0x33016) [0x7f6d10b7f016] /opt/mpich/lib/libmpi.so.12(+0x29eada) [0x7f6d10deaada] /opt/mpich/lib/libmpi.so.12(+0x1dfb7c) [0x7f6d10d2bb7c] /opt/mpich/lib/libmpi.so.12(+0x1d78cd) [0x7f6d10d238cd] /opt/mpich/lib/libmpi.so.12(PMPI_Comm_free+0xe7) [0x7f6d10bfe6b7] /home/kmccall/Needles2/NeedlesMpiMM(_ZN7needles12IntercomList20handleFailedIntercomERSt17_Rb_tree_iteratorISt4pairIKiPNS_16IntercomListElemEEEiPKc+0xae) [0x4c348a] /home/kmccall/Needles2/NeedlesMpiMM(_ZN7needles12IntercomList19disconnectIntercomsERilb+0xcf) [0x4c36fb] /home/kmccall/Needles2/NeedlesMpiMM(_ZN7needles16NeedlesMpiMaster8finalizeEv+0x25f) [0x496e25] /home/kmccall/Needles2/NeedlesMpiMM(_ZN7needles16NeedlesMpiMaster4loopEv+0x74e) [0x49355a] /home/kmccall/Needles2/NeedlesMpiMM(main+0x4af) [0x48d1e5] /lib64/libc.so.6(__libc_start_main+0xf3) [0x7f6d0f186ca3] /home/kmccall/Needles2/NeedlesMpiMM(_start+0x2e) [0x48cc7e] internal ABORT - process 0 I looked in ch3u_request.c and there seems to be no assertion at line 480. If I forgo the MPI_Comm_free in the first process, the same error then occurs in the second process. Does this give you any clue as to what I am doing wrong? Thanks, Kurt

1 0