Hi, Lately I have been dealing with an unexpected problem when using MPI_Comm_spawn + MPI_Intercomm_merge, where on some occasions my application hangs when two conditions are met. Specifically, the hang occurs when using the resulting Intracommunicator of MPI_Intercomm_merge in collective operations as MPI_Bcast. The conditions are - There is an oversubscription state. The number of processes is greater than the available number of physical cores. - Using CH4:ofi with FI_PROVIDER="verbs:ofi_rxd". I tested a minimal code with MPICH 4.2.0 and MPICH 4.2.3 configured as: ./configure --prefix=... --with-device=ch4:ofi --disable-psm3 The minimal code <https://urldefense.us/v3/__https://lorca.act.uji.es/gitlab/martini/mpich_ofi... > to reproduce the problem is the following: ========================== #include <stdio.h>#include <stdlib.h>#include <mpi.h>int main(int argc, char* argv[]) { int rank, numP, numO; int rootBcast, order; double test = 0; int solution = 0; MPI_Init(&argc, &argv); MPI_Comm_rank(MPI_COMM_WORLD, &rank); MPI_Comm_size(MPI_COMM_WORLD, &numP); MPI_Comm intercomm, intracomm; MPI_Comm_get_parent(&intercomm); if(intercomm == MPI_COMM_NULL) { numO = atoi(argv[1]); MPI_Comm_spawn(argv[0], MPI_ARGV_NULL, numO, MPI_INFO_NULL, 0, MPI_COMM_WORLD, &intercomm, MPI_ERRCODES_IGNORE); order = 0; } else { order = 1; } MPI_Intercomm_merge(intercomm, order, &intracomm); printf("TEST 1 P%02d/%d\n", rank, numP); MPI_Bcast(&test, 1, MPI_DOUBLE, 0, intracomm); // Hangs here if(solution) { MPI_Barrier(intercomm); } printf("TEST 2 P%02d/%d\n", rank, numP); MPI_Finalize(); return 0;} ========================== The code only hangs at the MPI_Bcast operation for some of the spawned processes. All my executions have been with a single node of 20 cores, with 10 initial processes and spawning 20 at the MPI_Comm_spawn function. If I change the variable "solution" to 1, I rarely get a hang of the application, but it still happens on some occasions.
From my perspective, the code seems to follow the standard. Is this the case? I have been able to run the code with other providers for OFI, but I am confused as to why it does not work in this case.
Thank you for your time. Best regards, Iker