July 2024 - discuss - mailman3.cels.anl.gov

mpich-4.2.2 and ABI compatibility
by Edric Ellis 31 Jul '24

31 Jul '24

In trying to upgrade our build of mpich to 4.2.2 from 4.1.2, I notice that the libraries are not completely binary compatible - at least, not in the way that our system presumes will be the case. We currently support users bringing their own MPI, providing it is ABI-compatible with our "primary" MPI, which is whatever current release of mpich we're using. In the past, this has meant that in practice, because nothing we do needs the very latest functionality provided by mpich, even very old mpich-derivative MPIs would work. With mpich-4.2.2, I see from the CHANGES file that many of the functions that were implemented as macros (e.g. MPI_Comm_f2c) are now "real" functions - to support the requirements of the MPI-4.1 standard. I notice further that the implementation of those functions is basically the same as the old macro. For now, I'm planning to work around this by effectively using the old macro definitions for cases where a user-supplied MPI library doesn't provide the necessary entry points. Does that seem reasonable? In fact, I note this PR https://urldefense.us/v3/__https://github.com/pmodels/mpich/pull/6711__;!!G… in which Ken Raffenetti says: "Just thinking from the ABI perspective, I don't think there is any backwards compatibility concern. Apps compiled when these were macros will continue to work with newer MPICH libraries." So, I infer from that comment that the approach I'm contemplating should be fine. I imagine the other ABI-compatible libraries listed here https://urldefense.us/v3/__https://www.mpich.org/abi/__;!!G_uCfscf7eWS!Z3Ge… will eventually adopt this change too - I don't suppose there is any known timeline when this might happen? Cheers, Edric.

1 0

Re: [mpich-discuss] OFI poll failed error if using more than one cluster node
by Stephen Wong 29 Jul '24

29 Jul '24

For the build configured with --with-device=ch4:ofi, I got ==== Various sizes and limits ==== sizeof(MPIDI_per_vci_t): 192 Required minimum FI_VERSION: 0, current version: 10014 provider: tcp, score = 4, pref = 0, FI_SOCKADDR_IN [16] 192.168.1.5 provider: tcp, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::1ff:fe23:4567:890a provider: tcp, score = 4, pref = 0, FI_SOCKADDR_IN [16] 127.0.0.1 provider: tcp, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] ::1 provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 192.168.1.5 provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::1ff:fe23:4567:890a provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 127.0.0.1 provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] ::1 provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 192.168.1.5 provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::1ff:fe23:4567:890a provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 127.0.0.1 provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] ::1 provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 192.168.1.5 provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::1ff:fe23:4567:890a provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 127.0.0.1 provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] ::1 provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 192.168.1.5 provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::1ff:fe23:4567:890a provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 127.0.0.1 provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] ::1 provider: udp;ofi_rxd, score = 5, pref = -2, FI_SOCKADDR_IN [16] 192.168.1.5 provider: udp;ofi_rxd, score = 5, pref = -2, FI_SOCKADDR_IN6 [28] fe80::1ff:fe23:4567:890a provider: udp;ofi_rxd, score = 5, pref = -2, FI_SOCKADDR_IN [16] 127.0.0.1 provider: udp;ofi_rxd, score = 5, pref = -2, FI_SOCKADDR_IN6 [28] ::1 provider: shm, score = 4, pref = -2, FI_ADDR_STR [14] - fi_shm://4595 provider: shm, score = 4, pref = -2, FI_ADDR_STR [14] - fi_shm://4595 provider: udp, score = 0, pref = 0, FI_SOCKADDR_IN [16] 192.168.1.5 provider: udp, score = 0, pref = 0, FI_SOCKADDR_IN6 [28] fe80::1ff:fe23:4567:890a provider: udp, score = 0, pref = 0, FI_SOCKADDR_IN [16] 127.0.0.1 provider: udp, score = 0, pref = 0, FI_SOCKADDR_IN6 [28] ::1 provider: tcp, score = 0, pref = 0, FI_SOCKADDR_IN [16] 192.168.1.5 provider: tcp, score = 0, pref = 0, FI_SOCKADDR_IN6 [28] fe80::1ff:fe23:4567:890a provider: tcp, score = 0, pref = 0, FI_SOCKADDR_IN [16] 127.0.0.1 provider: tcp, score = 0, pref = 0, FI_SOCKADDR_IN6 [28] ::1 provider: tcp, score = 4, pref = 0, FI_SOCKADDR_IN [16] 192.168.1.5 provider: tcp, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::1ff:fe23:4567:890a provider: tcp, score = 4, pref = 0, FI_SOCKADDR_IN [16] 127.0.0.1 provider: tcp, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] ::1 provider: sockets, score = 3, pref = 0, FI_SOCKADDR_IN [16] 192.168.1.5 provider: sockets, score = 3, pref = 0, FI_SOCKADDR_IN6 [28] fe80::1ff:fe23:4567:890a provider: sockets, score = 3, pref = 0, FI_SOCKADDR_IN [16] 127.0.0.1 provider: sockets, score = 3, pref = 0, FI_SOCKADDR_IN6 [28] ::1 provider: sockets, score = 5, pref = 0, FI_SOCKADDR_IN [16] 192.168.1.5 provider: sockets, score = 5, pref = 0, FI_SOCKADDR_IN6 [28] fe80::1ff:fe23:4567:890a provider: sockets, score = 5, pref = 0, FI_SOCKADDR_IN [16] 127.0.0.1 provider: sockets, score = 5, pref = 0, FI_SOCKADDR_IN6 [28] ::1 provider: sockets, score = 5, pref = 0, FI_SOCKADDR_IN [16] 192.168.1.5 provider: sockets, score = 5, pref = 0, FI_SOCKADDR_IN6 [28] fe80::1ff:fe23:4567:890a provider: sockets, score = 5, pref = 0, FI_SOCKADDR_IN [16] 127.0.0.1 provider: sockets, score = 5, pref = 0, FI_SOCKADDR_IN6 [28] ::1 provider: sm2, score = 3, pref = 0, FI_ADDR_STR [13] - fi_sm2://4595 provider: sm2, score = 3, pref = 0, FI_ADDR_STR [13] - fi_sm2://4595 Required minimum FI_VERSION: 10005, current version: 10014 ==== Capability set configuration ==== libfabric provider: sockets - 192.168.1.0/24 MPIDI_OFI_ENABLE_DATA: 1 MPIDI_OFI_ENABLE_AV_TABLE: 1 MPIDI_OFI_ENABLE_SCALABLE_ENDPOINTS: 1 MPIDI_OFI_ENABLE_SHARED_CONTEXTS: 0 MPIDI_OFI_ENABLE_MR_VIRT_ADDRESS: 0 MPIDI_OFI_ENABLE_MR_ALLOCATED: 0 MPIDI_OFI_ENABLE_MR_REGISTER_NULL: 1 MPIDI_OFI_ENABLE_MR_PROV_KEY: 0 MPIDI_OFI_ENABLE_TAGGED: 1 MPIDI_OFI_ENABLE_AM: 1 MPIDI_OFI_ENABLE_RMA: 1 MPIDI_OFI_ENABLE_ATOMICS: 1 MPIDI_OFI_FETCH_ATOMIC_IOVECS: 1 MPIDI_OFI_ENABLE_DATA_AUTO_PROGRESS: 0 MPIDI_OFI_ENABLE_CONTROL_AUTO_PROGRESS: 0 MPIDI_OFI_ENABLE_PT2PT_NOPACK: 1 MPIDI_OFI_ENABLE_TRIGGERED: 0 MPIDI_OFI_ENABLE_HMEM: 0 MPIDI_OFI_NUM_AM_BUFFERS: 8 MPIDI_OFI_NUM_OPTIMIZED_MEMORY_REGIONS: 0 MPIDI_OFI_CONTEXT_BITS: 20 MPIDI_OFI_SOURCE_BITS: 0 MPIDI_OFI_TAG_BITS: 31 MPIDI_OFI_VNI_USE_DOMAIN: 1 MAXIMUM SUPPORTED RANKS: 4294967296 MAXIMUM TAG: 2147483648 ==== Provider global thresholds ==== max_buffered_send: 255 max_buffered_write: 255 max_msg_size: 9223372036854775807 max_order_raw: -1 max_order_war: -1 max_order_waw: -1 tx_iov_limit: 8 rx_iov_limit: 8 rma_iov_limit: 8 max_mr_key_size: 8 ==== Various sizes and limits ==== MPIDI_OFI_AM_MSG_HEADER_SIZE: 24 MPIDI_OFI_MAX_AM_HDR_SIZE: 255 sizeof(MPIDI_OFI_am_request_header_t): 416 sizeof(MPIDI_OFI_per_vci_t): 52480 MPIDI_OFI_AM_HDR_POOL_CELL_SIZE: 1024 MPIDI_OFI_DEFAULT_SHORT_SEND_SIZE: 16384 ==== OFI dynamic settings ==== num_vcis: 1 num_nics: 1 ====================================== *mpiexec -host host1,host2 -n 2 cpi* Abort(883025295) on node 1: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(48306).............: MPI_Init(argc=0x7ffcb8c7aedc, argv=0x7ffcb8c7aed0) failed MPII_Init_thread(265)............: MPIR_init_comm_world(34).........: MPIR_Comm_commit(823)............: MPID_Comm_commit_post_hook(222)..: MPIDI_world_post_init(660).......: MPIDI_OFI_init_vcis(842).........: check_num_nics(891)..............: MPIR_Allreduce_allcomm_auto(4726): MPIC_Sendrecv(306)...............: MPIC_Wait(91)....................: MPIR_Wait(780)...................: MPIR_Wait_state(737).............: MPIDI_progress_test(134).........: MPIDI_OFI_handle_cq_error(791)...: OFI poll failed (ofi_events.c:793:MPIDI_OFI_handle_cq_error:Input/output error) Abort(883025295) on node 0: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(48306).............: MPI_Init(argc=0x7fff18beee1c, argv=0x7fff18beee10) failed MPII_Init_thread(265)............: MPIR_init_comm_world(34).........: MPIR_Comm_commit(823)............: MPID_Comm_commit_post_hook(222)..: MPIDI_world_post_init(660).......: MPIDI_OFI_init_vcis(842).........: check_num_nics(891)..............: MPIR_Allreduce_allcomm_auto(4726): MPIC_Sendrecv(306)...............: MPIC_Wait(91)....................: MPIR_Wait(780)...................: MPIR_Wait_state(737).............: MPIDI_progress_test(134).........: MPIDI_OFI_handle_cq_error(791)...: OFI poll failed (ofi_events.c:793:MPIDI_OFI_handle_cq_error:Input/output error) On Fri, 26 Jul 2024 at 19:04, Zhou, Hui <zhouh(a)anl.gov> wrote: > Could you run cpi with MPIR_CVAR_DEBUG_SUMMARY=1 and post the output? > > Hui Zhou > ------------------------------ > *From:* Stephen Wong via discuss <discuss(a)mpich.org> > *Sent:* Friday, July 26, 2024 4:46 AM > *To:* discuss(a)mpich.org <discuss(a)mpich.org> > *Cc:* Stephen Wong <stephen.photond(a)gmail.com> > *Subject:* [mpich-discuss] OFI poll failed error if using more than one > cluster node > > Hi, (I sent this previously without a subject line. ) I am using MPICH 4. > 2. 2 on Ubuntu 24. 04 testing with the small program cpi that calculates > the value of pi using MPI. I can start on host1 to run cpi on either host1 > or host2 alone and I can > ZjQcmQRYFpfptBannerStart > This Message Is From an External Sender > This message came from outside your organization. > > ZjQcmQRYFpfptBannerEnd > Hi, > > (I sent this previously without a subject line.) > > I am using MPICH 4.2.2 on Ubuntu 24.04 testing with the small program cpi > that calculates the value of pi using MPI. I can start on host1 to run cpi > on either host1 or host2 alone and I can start on host2 to run cpi on > either host2 or host1 alone. The problem occurs only if I try to use both > host1 and host2 together. > > This is done using, for example, the command > *mpiexec -host host1,host2 -n 2 cpi* > then it ends with the error > > Abort(77718927) on node 1: Fatal error in internal_Init: Other MPI error, > error stack: > internal_Init(48306).............: MPI_Init(argc=0x7ffdb68e7fec, > argv=0x7ffdb68e7fe0) failed > MPII_Init_thread(265)............: > MPIR_init_comm_world(34).........: > MPIR_Comm_commit(823)............: > MPID_Comm_commit_post_hook(222)..: > MPIDI_world_post_init(660).......: > MPIDI_OFI_init_vcis(842).........: > check_num_nics(891)..............: > MPIR_Allreduce_allcomm_auto(4726): > MPIC_Sendrecv(306)...............: > MPIC_Wait(91)....................: > MPIR_Wait(780)...................: > MPIR_Wait_state(737).............: > MPIDI_progress_test(134).........: > MPIDI_OFI_handle_cq_error(791)...: OFI poll failed > (ofi_events.c:793:MPIDI_OFI_handle_cq_error:Input/output error) > Abort(77718927) on node 0: Fatal error in internal_Init: Other MPI error, > error stack: > internal_Init(48306).............: MPI_Init(argc=0x7ffcb5b28adc, > argv=0x7ffcb5b28ad0) failed > MPII_Init_thread(265)............: > MPIR_init_comm_world(34).........: > MPIR_Comm_commit(823)............: > MPID_Comm_commit_post_hook(222)..: > MPIDI_world_post_init(660).......: > MPIDI_OFI_init_vcis(842).........: > check_num_nics(891)..............: > MPIR_Allreduce_allcomm_auto(4726): > MPIC_Sendrecv(306)...............: > MPIC_Wait(91)....................: > MPIR_Wait(780)...................: > MPIR_Wait_state(737).............: > MPIDI_progress_test(134).........: > MPIDI_OFI_handle_cq_error(791)...: OFI poll failed > (ofi_events.c:793:MPIDI_OFI_handle_cq_error:Input/output error) > I searched through the archive of this mailing list and there is only one > thread that has this OFI poll failed error. > In the thread, it mentioned this has something to do with the device > configuration of ch4:ofi ? > I checked my configure log and it has > device : ch4:ofi (embedded libfabric) > in the configuration when I built the MPI. So I am wondering if I should > switch this option to something else? If this will fix it. I am not too > sure what other option I could substitute for ch4:ofi. > > ***************************************************** > > Next I tried running configure for a build with the --enable-device = > ch3:nemesis option. > Now again I can run the cpi on any of host1 or host2 alone. If I run it > on host1 and host2 together, it just crashed with a core dump. > > Using the --enable-device = ch3:sock configure option resulted in more or > less the same problem but now it just quits silently when running on host1 > and host2 together. > > Any ideas? > Thanks! > Stephen. > >

1 0

Re: [mpich-discuss] OFI poll failed error if using more than one cluster node
by Zhou, Hui 26 Jul '24

26 Jul '24

Could you run cpi with MPIR_CVAR_DEBUG_SUMMARY=1 and post the output? Hui Zhou ________________________________ From: Stephen Wong via discuss <discuss(a)mpich.org> Sent: Friday, July 26, 2024 4:46 AM To: discuss(a)mpich.org <discuss(a)mpich.org> Cc: Stephen Wong <stephen.photond(a)gmail.com> Subject: [mpich-discuss] OFI poll failed error if using more than one cluster node Hi, (I sent this previously without a subject line. ) I am using MPICH 4. 2. 2 on Ubuntu 24. 04 testing with the small program cpi that calculates the value of pi using MPI. I can start on host1 to run cpi on either host1 or host2 alone and I can ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Hi, (I sent this previously without a subject line.) I am using MPICH 4.2.2 on Ubuntu 24.04 testing with the small program cpi that calculates the value of pi using MPI. I can start on host1 to run cpi on either host1 or host2 alone and I can start on host2 to run cpi on either host2 or host1 alone. The problem occurs only if I try to use both host1 and host2 together. This is done using, for example, the command mpiexec -host host1,host2 -n 2 cpi then it ends with the error Abort(77718927) on node 1: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(48306).............: MPI_Init(argc=0x7ffdb68e7fec, argv=0x7ffdb68e7fe0) failed MPII_Init_thread(265)............: MPIR_init_comm_world(34).........: MPIR_Comm_commit(823)............: MPID_Comm_commit_post_hook(222)..: MPIDI_world_post_init(660).......: MPIDI_OFI_init_vcis(842).........: check_num_nics(891)..............: MPIR_Allreduce_allcomm_auto(4726): MPIC_Sendrecv(306)...............: MPIC_Wait(91)....................: MPIR_Wait(780)...................: MPIR_Wait_state(737).............: MPIDI_progress_test(134).........: MPIDI_OFI_handle_cq_error(791)...: OFI poll failed (ofi_events.c:793:MPIDI_OFI_handle_cq_error:Input/output error) Abort(77718927) on node 0: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(48306).............: MPI_Init(argc=0x7ffcb5b28adc, argv=0x7ffcb5b28ad0) failed MPII_Init_thread(265)............: MPIR_init_comm_world(34).........: MPIR_Comm_commit(823)............: MPID_Comm_commit_post_hook(222)..: MPIDI_world_post_init(660).......: MPIDI_OFI_init_vcis(842).........: check_num_nics(891)..............: MPIR_Allreduce_allcomm_auto(4726): MPIC_Sendrecv(306)...............: MPIC_Wait(91)....................: MPIR_Wait(780)...................: MPIR_Wait_state(737).............: MPIDI_progress_test(134).........: MPIDI_OFI_handle_cq_error(791)...: OFI poll failed (ofi_events.c:793:MPIDI_OFI_handle_cq_error:Input/output error) I searched through the archive of this mailing list and there is only one thread that has this OFI poll failed error. In the thread, it mentioned this has something to do with the device configuration of ch4:ofi ? I checked my configure log and it has device : ch4:ofi (embedded libfabric) in the configuration when I built the MPI. So I am wondering if I should switch this option to something else? If this will fix it. I am not too sure what other option I could substitute for ch4:ofi. ***************************************************** Next I tried running configure for a build with the --enable-device = ch3:nemesis option. Now again I can run the cpi on any of host1 or host2 alone. If I run it on host1 and host2 together, it just crashed with a core dump. Using the --enable-device = ch3:sock configure option resulted in more or less the same problem but now it just quits silently when running on host1 and host2 together. Any ideas? Thanks! Stephen.

1 0

OFI poll failed error if using more than one cluster node
by Stephen Wong 26 Jul '24

26 Jul '24

Hi, (I sent this previously without a subject line.) I am using MPICH 4.2.2 on Ubuntu 24.04 testing with the small program cpi that calculates the value of pi using MPI. I can start on host1 to run cpi on either host1 or host2 alone and I can start on host2 to run cpi on either host2 or host1 alone. The problem occurs only if I try to use both host1 and host2 together. This is done using, for example, the command *mpiexec -host host1,host2 -n 2 cpi* then it ends with the error Abort(77718927) on node 1: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(48306).............: MPI_Init(argc=0x7ffdb68e7fec, argv=0x7ffdb68e7fe0) failed MPII_Init_thread(265)............: MPIR_init_comm_world(34).........: MPIR_Comm_commit(823)............: MPID_Comm_commit_post_hook(222)..: MPIDI_world_post_init(660).......: MPIDI_OFI_init_vcis(842).........: check_num_nics(891)..............: MPIR_Allreduce_allcomm_auto(4726): MPIC_Sendrecv(306)...............: MPIC_Wait(91)....................: MPIR_Wait(780)...................: MPIR_Wait_state(737).............: MPIDI_progress_test(134).........: MPIDI_OFI_handle_cq_error(791)...: OFI poll failed (ofi_events.c:793:MPIDI_OFI_handle_cq_error:Input/output error) Abort(77718927) on node 0: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(48306).............: MPI_Init(argc=0x7ffcb5b28adc, argv=0x7ffcb5b28ad0) failed MPII_Init_thread(265)............: MPIR_init_comm_world(34).........: MPIR_Comm_commit(823)............: MPID_Comm_commit_post_hook(222)..: MPIDI_world_post_init(660).......: MPIDI_OFI_init_vcis(842).........: check_num_nics(891)..............: MPIR_Allreduce_allcomm_auto(4726): MPIC_Sendrecv(306)...............: MPIC_Wait(91)....................: MPIR_Wait(780)...................: MPIR_Wait_state(737).............: MPIDI_progress_test(134).........: MPIDI_OFI_handle_cq_error(791)...: OFI poll failed (ofi_events.c:793:MPIDI_OFI_handle_cq_error:Input/output error) I searched through the archive of this mailing list and there is only one thread that has this OFI poll failed error. In the thread, it mentioned this has something to do with the device configuration of ch4:ofi ? I checked my configure log and it has device : ch4:ofi (embedded libfabric) in the configuration when I built the MPI. So I am wondering if I should switch this option to something else? If this will fix it. I am not too sure what other option I could substitute for ch4:ofi. ***************************************************** Next I tried running configure for a build with the --enable-device = ch3:nemesis option. Now again I can run the cpi on any of host1 or host2 alone. If I run it on host1 and host2 together, it just crashed with a core dump. Using the --enable-device = ch3:sock configure option resulted in more or less the same problem but now it just quits silently when running on host1 and host2 together. Any ideas? Thanks! Stephen.

1 0

(no subject)
by Stephen Wong 26 Jul '24

26 Jul '24

Hi, I am using MPICH 4.2.2 on Ubuntu 24.04 testing with the small program cpi that calculates the value of pi using MPI. I can start on host1 to run cpi on either host1 or host2 alone and I can start on host2 to run cpi on either host2 or host1 alone. The problem occurs only if I try to use both host1 and host2 together. This is done using, for example, the command *mpiexec -host host1,host2 -n 2 cpi* then it ends with the error Abort(77718927) on node 1: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(48306).............: MPI_Init(argc=0x7ffdb68e7fec, argv=0x7ffdb68e7fe0) failed MPII_Init_thread(265)............: MPIR_init_comm_world(34).........: MPIR_Comm_commit(823)............: MPID_Comm_commit_post_hook(222)..: MPIDI_world_post_init(660).......: MPIDI_OFI_init_vcis(842).........: check_num_nics(891)..............: MPIR_Allreduce_allcomm_auto(4726): MPIC_Sendrecv(306)...............: MPIC_Wait(91)....................: MPIR_Wait(780)...................: MPIR_Wait_state(737).............: MPIDI_progress_test(134).........: MPIDI_OFI_handle_cq_error(791)...: OFI poll failed (ofi_events.c:793:MPIDI_OFI_handle_cq_error:Input/output error) Abort(77718927) on node 0: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(48306).............: MPI_Init(argc=0x7ffcb5b28adc, argv=0x7ffcb5b28ad0) failed MPII_Init_thread(265)............: MPIR_init_comm_world(34).........: MPIR_Comm_commit(823)............: MPID_Comm_commit_post_hook(222)..: MPIDI_world_post_init(660).......: MPIDI_OFI_init_vcis(842).........: check_num_nics(891)..............: MPIR_Allreduce_allcomm_auto(4726): MPIC_Sendrecv(306)...............: MPIC_Wait(91)....................: MPIR_Wait(780)...................: MPIR_Wait_state(737).............: MPIDI_progress_test(134).........: MPIDI_OFI_handle_cq_error(791)...: OFI poll failed (ofi_events.c:793:MPIDI_OFI_handle_cq_error:Input/output error) I searched through the archive of this mailing list and there is only one thread that has this OFI poll failed error. In the thread, it mentioned this has something to do with the device configuration of ch4:ofi ? I checked my configure log and it has device : ch4:ofi (embedded libfabric) in the configuration when I built the MPI. So I am wondering if I should switch this option to something else? If this will fix it. I am not too sure what other option I could substitute for ch4:ofi. ***************************************************** Next I tried running configure for a build with the --enable-device = ch3:nemesis option. Now again I can run the cpi on any of host1 or host2 alone. If I run it on host1 and host2 together, it just crashed with a core dump. Using the --enable-device = ch3:sock configure option resulted in more or less the same problem but now it just quits silently when running on host1 and host2 together. Any ideas? Thanks! Stephen.

1 0

Re: [mpich-discuss] yaksa: leaked handle pool objects
by Zhou, Hui 23 Jul '24

23 Jul '24

They are leak check warnings. They indicate not all datatype handles are freed (via MPI_Type_free). We understand not all users care about these handle leaks, thus this warning will be disabled by default in the next release. -- Hui Zhou From: Mccall, Kurt E. (MSFC-EV41) via discuss <discuss(a)mpich.org> Date: Tuesday, July 23, 2024 at 11:08 AM To: discuss(a)mpich.org <discuss(a)mpich.org> Cc: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov> Subject: [mpich-discuss] yaksa: leaked handle pool objects In my MPICH 4. 1. 2 application, I am getting a few of these warnings, when MPI_Finalize is called. [WARNING] yaksa: 2 leaked handle pool objects Is there any way to tell where/why they are occurring? Thanks, Kurt . ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd In my MPICH 4.1.2 application, I am getting a few of these warnings, when MPI_Finalize is called. [WARNING] yaksa: 2 leaked handle pool objects Is there any way to tell where/why they are occurring? Thanks, Kurt .

1 0

yaksa: leaked handle pool objects
by Mccall, Kurt E. (MSFC-EV41) 23 Jul '24

23 Jul '24

In my MPICH 4.1.2 application, I am getting a few of these warnings, when MPI_Finalize is called. [WARNING] yaksa: 2 leaked handle pool objects Is there any way to tell where/why they are occurring? Thanks, Kurt .

1 0

Re: [mpich-discuss] Seeking possible causes for an assertion error in socksm.c at line 600: hdr.pkt_type == MPIDI_NEM_TCP_SOCKSM_PKT_ID_INFO || ...
by kumar.tarun＠siemens.com 17 Jul '24

17 Jul '24

Thanks. I had further discussion on this on github: https://urldefense.us/v3/__https://github.com/pmodels/mpich/discussions/705… I’m reposting my last response there as I need some answers. We got the confirmation from the customer that there was a security application which was somehow killing the application as the application(and it's executables) were not whitelisted. I have couple of questions: 1. Is the latest stable release of mpi4.2.2 stable against such instances? Is single machine run still require TCP ports or is it done using shared memory? 2. Is there anything that we can do while configuring mpich 3.2.1(the current version we are using) to make the application more robust against such instances? As I said it's a single machine run, so I'm still not sure why TCP ports are involved. Thanks Tarun From: Raffenetti, Ken <raffenet(a)anl.gov> Sent: Sunday, July 7, 2024 5:24 AM To: discuss(a)mpich.org Cc: Kumar, Tarun (DI SW ICS DVT RD QSCE) <kumar.tarun(a)siemens.com> Subject: Re: [mpich-discuss] Seeking possible causes for an assertion error in socksm.c at line 600: hdr.pkt_type == MPIDI_NEM_TCP_SOCKSM_PKT_ID_INFO || ... Are you able to update the MPICH version? MPICH 3.2.1 was released in 2017 and is no longer actively supported/maintained. Ken From: "kumar.tarun--- via discuss" <discuss(a)mpich.org<mailto:[email protected]>> Reply-To: "discuss(a)mpich.org<mailto:[email protected]>" <discuss(a)mpich.org<mailto:[email protected]>> Date: Saturday, July 6, 2024 at 3:49 PM To: "discuss(a)mpich.org<mailto:[email protected]>" <discuss(a)mpich.org<mailto:[email protected]>> Cc: "kumar.tarun(a)siemens.com<mailto:[email protected]>" <kumar.tarun(a)siemens.com<mailto:[email protected]>> Subject: [mpich-discuss] Seeking possible causes for an assertion error in socksm.c at line 600: hdr.pkt_type == MPIDI_NEM_TCP_SOCKSM_PKT_ID_INFO || ... Hi, We are hitting following assertion: Assertion failed in file .. . nemesis/netmod/tcp/socksm. c at line 600: hdr. pkt_type == MPIDI_NEM_TCP_SOCKSM_PKT_ID_INFO || .. . I looked at the assert and it looks like this in file …/nemesis/netmod/tcp/socksm. c ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Hi, We are hitting following assertion: Assertion failed in file ...nemesis/netmod/tcp/socksm.c at line 600: hdr.pkt_type == MPIDI_NEM_TCP_SOCKSM_PKT_ID_INFO || ... I looked at the assert and it looks like this in file …/nemesis/netmod/tcp/socksm.c MPIU_Assert(hdr.pkt_type == MPIDI_NEM_TCP_SOCKSM_PKT_ID_INFO || hdr.pkt_type == MPIDI_NEM_TCP_SOCKSM_PKT_TMPVC_INFO); We have tried multiple cores/partitions from 2 to 8 and the behaviour is same. Also a process is aborted and a message appears to suggest that. Mostly it's process 0 which is aborted but I have seen other processes as well reporting the crash. We are using mpich-3.2.1. I'm trying to understand possible causes for this error? I have explored the forum and no possible causes, like machine going out of memory etc are applicable here. Please suggest. Are there any debug/log/trace options I can use with mpiexec to further root cause? Regards Tarun

1 0

Re: [mpich-discuss] MPICH fails to allocate memory at beginning of job
by Zhou, Hui 08 Jul '24

08 Jul '24

The ucx netmod provides better support on infiniband. Could you try configure it with --with-device=ch4:ucx? -- Hui ________________________________ From: Mccall, Kurt E. (MSFC-EV41) via discuss <discuss(a)mpich.org> Sent: Monday, July 8, 2024 3:00 PM To: discuss(a)mpich.org <discuss(a)mpich.org> Cc: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov> Subject: [mpich-discuss] MPICH fails to allocate memory at beginning of job I configured MPICH 4. 1. 2 as follows. Any clue as to what would fix the error below? $ cd mpich-4. 1. 2-build $ ./mpich-4. 1. 2/configure --prefix=/opt/mpich --with-device=ch4: ofi --with-libfabric-embedded --with-slurm -enable-debuginfo --enable-g=debug ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd I configured MPICH 4.1.2 as follows. Any clue as to what would fix the error below? $ cd mpich-4.1.2-build $ ./mpich-4.1.2/configure --prefix=/opt/mpich --with-device=ch4:ofi --with-libfabric-embedded --with-slurm -enable-debuginfo --enable-g=debug 2>&1 | tee c.txt When I run a job, the error message on each node is: n007.cluster.pssclabs.com:rank6.HaystackMpiMM: Failed to modify UD QP to INIT on mlx5_0: Operation not permitted Abort(337761679): Fatal error in internal_Init: Other MPI error, error stack: internal_Init(66)........: MPI_Init(argc=0x7ffd93a663ec, argv=0x7ffd93a663e0) failed MPII_Init_thread(234)....: MPID_Init(513)...........: MPIDI_OFI_init_local(604): create_vni_context(982)..: OFI endpoint open failed (ofi_init.c:982:create_vni_context:Cannot allocate memory) The run command is: mpiexec -launcher ssh \ -print-all-exitcodes \ -wdir ${work_dir} \ -np ${num_proc} \ -ppn 1 \ my_program … Thanks, Kurt

1 0

MPICH fails to allocate memory at beginning of job
by Mccall, Kurt E. (MSFC-EV41) 08 Jul '24

08 Jul '24

I configured MPICH 4.1.2 as follows. Any clue as to what would fix the error below? $ cd mpich-4.1.2-build $ ./mpich-4.1.2/configure --prefix=/opt/mpich --with-device=ch4:ofi --with-libfabric-embedded --with-slurm -enable-debuginfo --enable-g=debug 2>&1 | tee c.txt When I run a job, the error message on each node is: n007.cluster.pssclabs.com:rank6.HaystackMpiMM: Failed to modify UD QP to INIT on mlx5_0: Operation not permitted Abort(337761679): Fatal error in internal_Init: Other MPI error, error stack: internal_Init(66)........: MPI_Init(argc=0x7ffd93a663ec, argv=0x7ffd93a663e0) failed MPII_Init_thread(234)....: MPID_Init(513)...........: MPIDI_OFI_init_local(604): create_vni_context(982)..: OFI endpoint open failed (ofi_init.c:982:create_vni_context:Cannot allocate memory) The run command is: mpiexec -launcher ssh \ -print-all-exitcodes \ -wdir ${work_dir} \ -np ${num_proc} \ -ppn 1 \ my_program ... Thanks, Kurt

1 0