I haven’t run the code, but shouldn’t the workers pass 0 (rank of master) as the root instead of MPI_PROC_NULL? Rajeev From: "Mccall, Kurt E. (MSFC-EV41) via discuss" <[email protected]> Reply-To: "[email protected]" <[email protected]> Date: Sunday, April 21, 2024 at 4:53 PM To: "[email protected]" <[email protected]> Cc: "Mccall, Kurt E. (MSFC-EV41)" <[email protected]> Subject: [mpich-discuss] MPI_Reduce on an inter-communicator hangs I am calling MPI_Reduce on a set of inter-communicators created by MPI_Comm_spawn, each with one process in the local group (the single manager) and two processes in the remote group (the workers). The inter- communicators are visited one at ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd I am calling MPI_Reduce on a set of inter-communicators created by MPI_Comm_spawn, each with one process in the local group (the single manager) and two processes in the remote group (the workers). The inter- communicators are visited one at a time in the manager. All workers enter and exit MPI_Reduce without blocking, but the manager enters the first MPI_Reduce for the first inter-communicator and never returns. What am I doing wrong? I am using MPICH 4.1.2. Here is my manager code: #define N_PROC 4 #define N_IN_GROUP 2 int main(int argc, char *argv[]) { int rank, world_size, error_codes[N_PROC]; MPI_Comm intercoms[N_PROC]; char hostname[64]; MPI_Init(&argc, &argv); MPI_Comm_rank(MPI_COMM_WORLD, &rank); MPI_Comm_size(MPI_COMM_WORLD, &world_size); gethostname(hostname, sizeof(hostname)); char *p = strstr(hostname, "."); *p = '\0'; MPI_Info info; MPI_Info_create(&info); MPI_Info_set(info, "host", hostname); MPI_Info_set(info, "bind_to", "core"); for (int i = 0; i < N_PROC; ++i) { MPI_Comm_spawn("test_reduce_work", argv, N_IN_GROUP, info, 0, MPI_COMM_SELF, &intercoms[i], &error_codes[i]); } sleep(10); unsigned array[100]{0}; for (int i = 0; i < N_PROC; ++i) { cout << "MANAGER: starting reduction " << i << "\n"; MPI_Reduce(NULL, array, 100, MPI_UNSIGNED, MPI_SUM, MPI_ROOT, intercoms[i]); cout << "MANAGER: finished reduction " << i << "\n"; // we never reach this point } for (int i = 0; i < 100; ++i) cout << array[i] << " "; cout << endl; MPI_Finalize(); } And here is my worker code: int main(int argc, char *argv[]) { int rank, world_size; MPI_Comm manager_intercom; MPI_Init(&argc, &argv); MPI_Comm_rank(MPI_COMM_WORLD, &rank); MPI_Comm_size(MPI_COMM_WORLD, &world_size); MPI_Comm_get_parent(&manager_intercom); unsigned array[100]{1}; cout << "WORKER: starting reduction\n"; MPI_Reduce(array, NULL, 100, MPI_UNSIGNED, MPI_SUM, MPI_PROC_NULL, manager_intercom); cout << "WORKER: finishing reduction\n"; sleep(10); MPI_Finalize(); } Finally, here it the invocation: $ mpiexec -launcher ssh -print-all-exitcodes -wdir /home/kmccall/test_dir -np 1 -ppn 1 test_reduce_man Thanks, Kurt