Hi, Lately I have been dealing with an unexpected problem when using MPI_Comm_spawn + MPI_Intercomm_merge, where on some occasions my application hangs when two conditions are met. Specifically, the hang occurs when using the resulting Intracommunicator
Hi,
Lately I have been dealing with an unexpected problem when using MPI_Comm_spawn + MPI_Intercomm_merge, where on some occasions my application hangs when two conditions are met.
Specifically, the hang occurs when using the resulting Intracommunicator of MPI_Intercomm_merge in collective operations as MPI_Bcast. The conditions are
- There is an oversubscription state. The number of processes is greater than the available number of physical cores.
- Using CH4:ofi with FI_PROVIDER="verbs:ofi_rxd".
I tested a minimal code with MPICH 4.2.0 and MPICH 4.2.3 configured as:
./configure --prefix=... --with-device=ch4:ofi --disable-psm3
==========================
#include <stdio.h>
#include <stdlib.h>
#include <mpi.h>
int main(int argc, char* argv[]) {
int rank, numP, numO;
int rootBcast, order;
double test = 0;
int solution = 0;
MPI_Init(&argc, &argv);
MPI_Comm_rank(MPI_COMM_WORLD, &rank);
MPI_Comm_size(MPI_COMM_WORLD, &numP);
MPI_Comm intercomm, intracomm;
MPI_Comm_get_parent(&intercomm);
if(intercomm == MPI_COMM_NULL) {
numO = atoi(argv[1]);
MPI_Comm_spawn(argv[0], MPI_ARGV_NULL, numO, MPI_INFO_NULL, 0, MPI_COMM_WORLD, &intercomm, MPI_ERRCODES_IGNORE);
order = 0;
} else { order = 1; }
MPI_Intercomm_merge(intercomm, order, &intracomm);
printf("TEST 1 P%02d/%d\n", rank, numP);
MPI_Bcast(&test, 1, MPI_DOUBLE, 0, intracomm); // Hangs here
if(solution) { MPI_Barrier(intercomm); }
printf("TEST 2 P%02d/%d\n", rank, numP);
MPI_Finalize();
return 0;
}
==========================
The code only hangs at the MPI_Bcast operation for some of the spawned processes. All my executions have been with a single node of 20 cores, with 10 initial processes and spawning 20 at the MPI_Comm_spawn function. If I change the variable "solution" to 1,
I rarely get a hang of the application, but it still happens on some occasions.
>From my perspective, the code seems to follow the standard. Is this the case? I have been able to run the code with other providers for OFI, but I am confused as to why it does not work in this case.
Thank you for your time.