Hi Iker, Does it work with `FI_PROVIDER="verbs;ofi_rxm"? Hui ________________________________ From: Iker Martín Álvarez via discuss <[email protected]> Sent: Monday, October 28, 2024 12:34 PM To: [email protected] <[email protected]> Cc: Iker Martín Álvarez <[email protected]> Subject: [mpich-discuss] Occasional hang with MPI_Intercomm_merge and OFI+provider verbs Hi, Lately I have been dealing with an unexpected problem when using MPI_Comm_spawn + MPI_Intercomm_merge, where on some occasions my application hangs when two conditions are met. Specifically, the hang occurs when using the resulting Intracommunicator ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Hi, Lately I have been dealing with an unexpected problem when using MPI_Comm_spawn + MPI_Intercomm_merge, where on some occasions my application hangs when two conditions are met. Specifically, the hang occurs when using the resulting Intracommunicator of MPI_Intercomm_merge in collective operations as MPI_Bcast. The conditions are - There is an oversubscription state. The number of processes is greater than the available number of physical cores. - Using CH4:ofi with FI_PROVIDER="verbs:ofi_rxd". I tested a minimal code with MPICH 4.2.0 and MPICH 4.2.3 configured as: ./configure --prefix=... --with-device=ch4:ofi --disable-psm3 The minimal code<https://urldefense.us/v3/__https://lorca.act.uji.es/gitlab/martini/mpich_ofi_rxd_intracomm_hang/-/blob/main/BaseCode.c__;!!G_uCfscf7eWS!ZoB5c9APxvNk40SirehC83dWaIUE_w3yOQ2EEoJ4MaJlUaREOTMYR85Dd4SCEE0Exrr4U6ZETBg1sg$> to reproduce the problem is the following: ========================== #include <stdio.h> #include <stdlib.h> #include <mpi.h> int main(int argc, char* argv[]) { int rank, numP, numO; int rootBcast, order; double test = 0; int solution = 0; MPI_Init(&argc, &argv); MPI_Comm_rank(MPI_COMM_WORLD, &rank); MPI_Comm_size(MPI_COMM_WORLD, &numP); MPI_Comm intercomm, intracomm; MPI_Comm_get_parent(&intercomm); if(intercomm == MPI_COMM_NULL) { numO = atoi(argv[1]); MPI_Comm_spawn(argv[0], MPI_ARGV_NULL, numO, MPI_INFO_NULL, 0, MPI_COMM_WORLD, &intercomm, MPI_ERRCODES_IGNORE); order = 0; } else { order = 1; } MPI_Intercomm_merge(intercomm, order, &intracomm); printf("TEST 1 P%02d/%d\n", rank, numP); MPI_Bcast(&test, 1, MPI_DOUBLE, 0, intracomm); // Hangs here if(solution) { MPI_Barrier(intercomm); } printf("TEST 2 P%02d/%d\n", rank, numP); MPI_Finalize(); return 0; } ========================== The code only hangs at the MPI_Bcast operation for some of the spawned processes. All my executions have been with a single node of 20 cores, with 10 initial processes and spawning 20 at the MPI_Comm_spawn function. If I change the variable "solution" to 1, I rarely get a hang of the application, but it still happens on some occasions.

From my perspective, the code seems to follow the standard. Is this the case? I have been able to run the code with other providers for OFI, but I am confused as to why it does not work in this case.

Thank you for your time. Best regards, Iker