Hang after MPI_Intercomm_merge with OFI+provider verbs
Hello, Lately, I have encountered an issue with the usage of MPI_Comm_spawn + MPI_Intercomm_merge, where in some occasions my application hangs indefinitely. I was hoping that this forum could advise me on a solution. For my tests, I run the code with 2 processes and give as an argument the amount of processes to spawn in MPI_Comm_spawn, where I choose 20. In my experimentation, 3 to 4 out of 100 executions hang indefinitely at the function MPI_Barrier. Nevertheless, if I remove both MPI_Intercomm_merge functions, the code always finalizes. Also, prior to executing the code I set the environment variable FI_PROVIDER=verbs. However, if I change the provider to "tcp", or "udp", the code doesn't hang, but the performance is lower for other providers different to "verbs". The MPICH version used is 4.2.1 with the following configure line: ./configure --prefix=... --with-device=ch4:ofi --disable-psm3 In addition, I have attached a file <https://urldefense.us/v3/__https://lorca.act.uji.es/gitlab/martini/mpich_ofi... > with the output of executing mpirun with "MPIR_CVAR_DEBUG_SUMMARY=1" and unsetting FI_PROVIDER. The minimal code <https://urldefense.us/v3/__https://lorca.act.uji.es/gitlab/martini/mpich_ofi... > to reproduce the problem is the following: =============== #include <stdio.h> #include <stdlib.h> #include <mpi.h> int main(int argc, char* argv[]) { MPI_Init(&argc, &argv); MPI_Comm spawn, new_comm; MPI_Comm_get_parent(&spawn); if(spawn == MPI_COMM_NULL) { int num_c = atoi(argv[1]); MPI_Comm_spawn(argv[0], MPI_ARGV_NULL, num_c, MPI_INFO_NULL, 0, MPI_COMM_WORLD, &spawn, MPI_ERRCODES_IGNORE); MPI_Intercomm_merge(spawn, 0, &new_comm); } else { MPI_Intercomm_merge(spawn, 1, &new_comm); MPI_Barrier(MPI_COMM_WORLD); } MPI_Finalize(); } =============== Thank you for your help. Best regards, Iker
participants (1)
-
Iker Martín Álvarez