April 2024 - discuss - mailman3.cels.anl.gov

Re: [mpich-discuss] Differences between ch3:nemesis and ch4:ofi:tcp with MPI_Barrier before completion of MPI_Isend
by Raffenetti, Ken 29 Apr '24

29 Apr '24

I was able to get the code from the list archive. I will update after I dig in. Thanks, Ken From: Edric Ellis <eellis(a)mathworks.com> Date: Friday, April 26, 2024 at 5:11 AM To: "Raffenetti, Ken" <raffenet(a)anl.gov>, "discuss(a)mpich.org" <discuss(a)mpich.org> Subject: RE: [mpich-discuss] Differences between ch3:nemesis and ch4:ofi:tcp with MPI_Barrier before completion of MPI_Isend Hm, I attempted to attach my full code, but maybe it didn’t get through correctly. Weirdly though when I look at the web view of this discussion group, *only* the code shows up (initially): https: //lists. mpich. org/pipermail/discuss/2024-April/006578. html ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Hm, I attempted to attach my full code, but maybe it didn’t get through correctly. Weirdly though when I look at the web view of this discussion group, *only* the code shows up (initially): https://urldefense.us/v3/__https://lists.mpich.org/pipermail/discuss/2024-A… <https://urldefense.us/v3/__https:/lists.mpich.org/pipermail/discuss/2024-Ap…> Hopefully that is visible? (I think the problem shows up only when using 2 hosts) (On looking again at that code, there’s a stray logging message about MPI_Comm_free – initially I was using separate communicators for the ISend and the Barrier – but that seems to make no difference). Cheers, Edric. From: Raffenetti, Ken <raffenet(a)anl.gov> Sent: Thursday, April 25, 2024 7:00 PM To: discuss(a)mpich.org Cc: Edric Ellis <eellis(a)mathworks.com> Subject: Re: [mpich-discuss] Differences between ch3:nemesis and ch4:ofi:tcp with MPI_Barrier before completion of MPI_Isend Hi Edric, I don’t see anything wrong in your pseudo code. I believe it is a correct pattern. I ran some experiments myself on some local machines and could not cause a hang, so if you come up with a reproducer please send it along. Ken From: Edric Ellis via discuss <discuss(a)mpich.org<mailto:[email protected]>> Reply-To: "discuss(a)mpich.org<mailto:[email protected]>" <discuss(a)mpich.org<mailto:[email protected]>> Date: Wednesday, April 24, 2024 at 3:05 AM To: "discuss(a)mpich.org<mailto:[email protected]>" <discuss(a)mpich.org<mailto:[email protected]>> Cc: Edric Ellis <eellis(a)mathworks.com<mailto:[email protected]>> Subject: [mpich-discuss] Differences between ch3:nemesis and ch4:ofi:tcp with MPI_Barrier before completion of MPI_Isend I'm trying to understand if a change in behaviour I'm seeing is expected or not. My code initiates an MPI_Isend on rank==1, and before waiting for completion of that send, all ranks perform an MPI_Barrier. This works fine on ch3: nemesis. It ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd I'm trying to understand if a change in behaviour I'm seeing is expected or not. My code initiates an MPI_Isend on rank==1, and before waiting for completion of that send, all ranks perform an MPI_Barrier. This works fine on ch3:nemesis. It works fine on ch4:ofi when the message is small (presumably using the "eager" protocol). When using ch4:ofi (the embedded tcp provider) and the message is large (presumably switching to "rendezvous" protocol), rank 0 never leaves the MPI_Barrier call. (I think the SHM piece of ch4 does not show the problem) Should this work? I cannot find anything in the MPI standard that says it should not, but perhaps I'm not looking in the right place. I'm using mpich-4.1.2 in both cases, either in "--with-device=ch3:nemesis" mode or "--with-libfabric=embedded --with-device=ch4:ofi:tcp". Here's a sketch of the problematic section of code, I'll attempt to attach a full reproduction (but I'm not sure if that works?) // setup code... if (rank == 1) { MPI_Isend(data, count, MPI_INT, 0, TAG, comm, &req); } else { std::this_thread::sleep_for(std::chrono::seconds(1)); } MPI_Barrier(comm); MPI_Barrier(comm); // cleanup code, receive the message etc... Cheers, Edric.

1 0

Re: [mpich-discuss] Zero-copy non-contiguous types
by Zhou, Hui 29 Apr '24

29 Apr '24

Yes, intra-node ipc including non-contig datatypes is supported via xpmem. -- Hui Zhou ________________________________ From: Pascal Boeschoten via discuss <discuss(a)mpich.org> Sent: Monday, April 29, 2024 2:37 AM To: discuss(a)mpich.org <discuss(a)mpich.org> Cc: Pascal Boeschoten <pascal.boeschoten(a)hapteon.com> Subject: [mpich-discuss] Zero-copy non-contiguous types Hello, Does MPICH support zero-copy intra-node communication for non-contiguous datatypes? Thanks, Pascal Boeschoten ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Hello, Does MPICH support zero-copy intra-node communication for non-contiguous datatypes? Thanks, Pascal Boeschoten

1 0

Zero-copy non-contiguous types
by Pascal Boeschoten 29 Apr '24

29 Apr '24

Hello, Does MPICH support zero-copy intra-node communication for non-contiguous datatypes? Thanks, Pascal Boeschoten

1 0

Re: [mpich-discuss] Differences between ch3:nemesis and ch4:ofi:tcp with MPI_Barrier before completion of MPI_Isend
by Edric Ellis 26 Apr '24

26 Apr '24

Hm, I attempted to attach my full code, but maybe it didn’t get through correctly. Weirdly though when I look at the web view of this discussion group, *only* the code shows up (initially): https://urldefense.us/v3/__https://lists.mpich.org/pipermail/discuss/2024-A… Hopefully that is visible? (I think the problem shows up only when using 2 hosts) (On looking again at that code, there’s a stray logging message about MPI_Comm_free – initially I was using separate communicators for the ISend and the Barrier – but that seems to make no difference). Cheers, Edric. From: Raffenetti, Ken <raffenet(a)anl.gov> Sent: Thursday, April 25, 2024 7:00 PM To: discuss(a)mpich.org Cc: Edric Ellis <eellis(a)mathworks.com> Subject: Re: [mpich-discuss] Differences between ch3:nemesis and ch4:ofi:tcp with MPI_Barrier before completion of MPI_Isend Hi Edric, I don’t see anything wrong in your pseudo code. I believe it is a correct pattern. I ran some experiments myself on some local machines and could not cause a hang, so if you come up with a reproducer please send it along. Ken From: Edric Ellis via discuss <discuss(a)mpich.org<mailto:[email protected]>> Reply-To: "discuss(a)mpich.org<mailto:[email protected]>" <discuss(a)mpich.org<mailto:[email protected]>> Date: Wednesday, April 24, 2024 at 3:05 AM To: "discuss(a)mpich.org<mailto:[email protected]>" <discuss(a)mpich.org<mailto:[email protected]>> Cc: Edric Ellis <eellis(a)mathworks.com<mailto:[email protected]>> Subject: [mpich-discuss] Differences between ch3:nemesis and ch4:ofi:tcp with MPI_Barrier before completion of MPI_Isend I'm trying to understand if a change in behaviour I'm seeing is expected or not. My code initiates an MPI_Isend on rank==1, and before waiting for completion of that send, all ranks perform an MPI_Barrier. This works fine on ch3: nemesis. It ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd I'm trying to understand if a change in behaviour I'm seeing is expected or not. My code initiates an MPI_Isend on rank==1, and before waiting for completion of that send, all ranks perform an MPI_Barrier. This works fine on ch3:nemesis. It works fine on ch4:ofi when the message is small (presumably using the "eager" protocol). When using ch4:ofi (the embedded tcp provider) and the message is large (presumably switching to "rendezvous" protocol), rank 0 never leaves the MPI_Barrier call. (I think the SHM piece of ch4 does not show the problem) Should this work? I cannot find anything in the MPI standard that says it should not, but perhaps I'm not looking in the right place. I'm using mpich-4.1.2 in both cases, either in "--with-device=ch3:nemesis" mode or "--with-libfabric=embedded --with-device=ch4:ofi:tcp". Here's a sketch of the problematic section of code, I'll attempt to attach a full reproduction (but I'm not sure if that works?) // setup code... if (rank == 1) { MPI_Isend(data, count, MPI_INT, 0, TAG, comm, &req); } else { std::this_thread::sleep_for(std::chrono::seconds(1)); } MPI_Barrier(comm); MPI_Barrier(comm); // cleanup code, receive the message etc... Cheers, Edric.

1 0

Re: [mpich-discuss] Differences between ch3:nemesis and ch4:ofi:tcp with MPI_Barrier before completion of MPI_Isend
by Raffenetti, Ken 25 Apr '24

25 Apr '24

Hi Edric, I don’t see anything wrong in your pseudo code. I believe it is a correct pattern. I ran some experiments myself on some local machines and could not cause a hang, so if you come up with a reproducer please send it along. Ken From: Edric Ellis via discuss <discuss(a)mpich.org> Reply-To: "discuss(a)mpich.org" <discuss(a)mpich.org> Date: Wednesday, April 24, 2024 at 3:05 AM To: "discuss(a)mpich.org" <discuss(a)mpich.org> Cc: Edric Ellis <eellis(a)mathworks.com> Subject: [mpich-discuss] Differences between ch3:nemesis and ch4:ofi:tcp with MPI_Barrier before completion of MPI_Isend I'm trying to understand if a change in behaviour I'm seeing is expected or not. My code initiates an MPI_Isend on rank==1, and before waiting for completion of that send, all ranks perform an MPI_Barrier. This works fine on ch3: nemesis. It ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd I'm trying to understand if a change in behaviour I'm seeing is expected or not. My code initiates an MPI_Isend on rank==1, and before waiting for completion of that send, all ranks perform an MPI_Barrier. This works fine on ch3:nemesis. It works fine on ch4:ofi when the message is small (presumably using the "eager" protocol). When using ch4:ofi (the embedded tcp provider) and the message is large (presumably switching to "rendezvous" protocol), rank 0 never leaves the MPI_Barrier call. (I think the SHM piece of ch4 does not show the problem) Should this work? I cannot find anything in the MPI standard that says it should not, but perhaps I'm not looking in the right place. I'm using mpich-4.1.2 in both cases, either in "--with-device=ch3:nemesis" mode or "--with-libfabric=embedded --with-device=ch4:ofi:tcp". Here's a sketch of the problematic section of code, I'll attempt to attach a full reproduction (but I'm not sure if that works?) // setup code... if (rank == 1) { MPI_Isend(data, count, MPI_INT, 0, TAG, comm, &req); } else { std::this_thread::sleep_for(std::chrono::seconds(1)); } MPI_Barrier(comm); MPI_Barrier(comm); // cleanup code, receive the message etc... Cheers, Edric.

1 0

Differences between ch3:nemesis and ch4:ofi:tcp with MPI_Barrier before completion of MPI_Isend
by Edric Ellis 24 Apr '24

24 Apr '24

1 0

Re: [mpich-discuss] Environment variable forwarding using Hydra and "-launcher ssh"
by Raffenetti, Ken 22 Apr '24

22 Apr '24

Hi Edric, It looks like we may have unintentionally inverted the logic on that function return value. I’ll submit a PR to fix. Thanks for bringing it to our attention. Ken From: Edric Ellis via discuss <discuss(a)mpich.org> Reply-To: "discuss(a)mpich.org" <discuss(a)mpich.org> Date: Monday, April 22, 2024 at 9:44 AM To: "discuss(a)mpich.org" <discuss(a)mpich.org> Cc: Edric Ellis <eellis(a)mathworks.com> Subject: [mpich-discuss] Environment variable forwarding using Hydra and "-launcher ssh" We’re in the process of moving from mpich-3. x to mpich-4. 1. 2. We’ve run into some odd behaviour on SLURM related to environment variable forwarding by mpiexec. It looks like mpiexec now propagates only SLURM_* environment variables, ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd We’re in the process of moving from mpich-3.x to mpich-4.1.2. We’ve run into some odd behaviour on SLURM related to environment variable forwarding by mpiexec. It looks like mpiexec now propagates only SLURM_* environment variables, instead of filtering them out (or intending to). Consider something like this: $ mpiexec -launcher slurm printenv HOME SLURM_JOBID Using mpich-3.x, the HOME variable gets forward. Using mpich-4.1.2, it does not. I believe that mpich-3.x intends to filter out SLURM_JOBID, but the value still seems to be present, maybe srun forwards that. It’s the fact that HOME doesn’t get through using mpich-4.1.2 that is causing us problems. Running mpich- Here’s what I think is the relevant change for SLURM: https://urldefense.us/v3/__https://github.com/pmodels/mpich/commit/95ba4ddc… <https://urldefense.us/v3/__https:/github.com/pmodels/mpich/commit/95ba4ddc7…> . Am I reading that correctly? The doc here https://urldefense.us/v3/__https://github.com/pmodels/mpich/blob/main/doc/w… <https://urldefense.us/v3/__https:/github.com/pmodels/mpich/blob/main/doc/wi…> states that SLURM_ things should be filtered out, but that doesn’t appear to be happening? For reference, here’s what mpich-4.1.2 “mpiexec -verbose -launcher slurm” prints: mpiexec options: ---------------- Base path: /path/to/mpich-4.1.2 Launcher: slurm Debug level: 1 Enable X: -1 Global environment: ------------------- SLURM_JOBID=102437 SLURM_JOB_USER=eellis SLURM_JOB_QOS=normal SLURM_JOB_NUM_NODES=2 SLURM_TASKS_PER_NODE=1(x2) SLURM_TOPOLOGY_ADDR_PATTERN=node … many more SLURM_* And here’s what mpich-3.x prints: mpiexec options: ---------------- Base path: /path/to/mpich-3.x Launcher: slurm Debug level: 1 Enable X: -1 Global environment: ------------------- ALTERNATE_EDITOR=emacs MAIL=/var/mail/eellis USER=eellis SLURM_JOB_USER=eellis l=/local/eellis XDG_SESSION_TYPE=unspecified SLURM_JOB_QOS=normal … no SLURM_JOBID Cheers, Edric.

1 0

Environment variable forwarding using Hydra and "-launcher ssh"
by Edric Ellis 22 Apr '24

22 Apr '24

We're in the process of moving from mpich-3.x to mpich-4.1.2. We've run into some odd behaviour on SLURM related to environment variable forwarding by mpiexec. It looks like mpiexec now propagates only SLURM_* environment variables, instead of filtering them out (or intending to). Consider something like this: $ mpiexec -launcher slurm printenv HOME SLURM_JOBID Using mpich-3.x, the HOME variable gets forward. Using mpich-4.1.2, it does not. I believe that mpich-3.x intends to filter out SLURM_JOBID, but the value still seems to be present, maybe srun forwards that. It's the fact that HOME doesn't get through using mpich-4.1.2 that is causing us problems. Running mpich- Here's what I think is the relevant change for SLURM: https://urldefense.us/v3/__https://github.com/pmodels/mpich/commit/95ba4ddc… . Am I reading that correctly? The doc here https://urldefense.us/v3/__https://github.com/pmodels/mpich/blob/main/doc/w… states that SLURM_ things should be filtered out, but that doesn't appear to be happening? For reference, here's what mpich-4.1.2 "mpiexec -verbose -launcher slurm" prints: mpiexec options: ---------------- Base path: /path/to/mpich-4.1.2 Launcher: slurm Debug level: 1 Enable X: -1 Global environment: ------------------- SLURM_JOBID=102437 SLURM_JOB_USER=eellis SLURM_JOB_QOS=normal SLURM_JOB_NUM_NODES=2 SLURM_TASKS_PER_NODE=1(x2) SLURM_TOPOLOGY_ADDR_PATTERN=node ... many more SLURM_* And here's what mpich-3.x prints: mpiexec options: ---------------- Base path: /path/to/mpich-3.x Launcher: slurm Debug level: 1 Enable X: -1 Global environment: ------------------- ALTERNATE_EDITOR=emacs MAIL=/var/mail/eellis USER=eellis SLURM_JOB_USER=eellis l=/local/eellis XDG_SESSION_TYPE=unspecified SLURM_JOB_QOS=normal ... no SLURM_JOBID Cheers, Edric.

1 0

Re: [mpich-discuss] MPI_Reduce on an inter-communicator hangs
by Thakur, Rajeev 21 Apr '24

21 Apr '24

I haven’t run the code, but shouldn’t the workers pass 0 (rank of master) as the root instead of MPI_PROC_NULL? Rajeev From: "Mccall, Kurt E. (MSFC-EV41) via discuss" <discuss(a)mpich.org> Reply-To: "discuss(a)mpich.org" <discuss(a)mpich.org> Date: Sunday, April 21, 2024 at 4:53 PM To: "discuss(a)mpich.org" <discuss(a)mpich.org> Cc: "Mccall, Kurt E. (MSFC-EV41)" <kurt.e.mccall(a)nasa.gov> Subject: [mpich-discuss] MPI_Reduce on an inter-communicator hangs I am calling MPI_Reduce on a set of inter-communicators created by MPI_Comm_spawn, each with one process in the local group (the single manager) and two processes in the remote group (the workers). The inter- communicators are visited one at ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd I am calling MPI_Reduce on a set of inter-communicators created by MPI_Comm_spawn, each with one process in the local group (the single manager) and two processes in the remote group (the workers). The inter- communicators are visited one at a time in the manager. All workers enter and exit MPI_Reduce without blocking, but the manager enters the first MPI_Reduce for the first inter-communicator and never returns. What am I doing wrong? I am using MPICH 4.1.2. Here is my manager code: #define N_PROC 4 #define N_IN_GROUP 2 int main(int argc, char *argv[]) { int rank, world_size, error_codes[N_PROC]; MPI_Comm intercoms[N_PROC]; char hostname[64]; MPI_Init(&argc, &argv); MPI_Comm_rank(MPI_COMM_WORLD, &rank); MPI_Comm_size(MPI_COMM_WORLD, &world_size); gethostname(hostname, sizeof(hostname)); char *p = strstr(hostname, "."); *p = '\0'; MPI_Info info; MPI_Info_create(&info); MPI_Info_set(info, "host", hostname); MPI_Info_set(info, "bind_to", "core"); for (int i = 0; i < N_PROC; ++i) { MPI_Comm_spawn("test_reduce_work", argv, N_IN_GROUP, info, 0, MPI_COMM_SELF, &intercoms[i], &error_codes[i]); } sleep(10); unsigned array[100]{0}; for (int i = 0; i < N_PROC; ++i) { cout << "MANAGER: starting reduction " << i << "\n"; MPI_Reduce(NULL, array, 100, MPI_UNSIGNED, MPI_SUM, MPI_ROOT, intercoms[i]); cout << "MANAGER: finished reduction " << i << "\n"; // we never reach this point } for (int i = 0; i < 100; ++i) cout << array[i] << " "; cout << endl; MPI_Finalize(); } And here is my worker code: int main(int argc, char *argv[]) { int rank, world_size; MPI_Comm manager_intercom; MPI_Init(&argc, &argv); MPI_Comm_rank(MPI_COMM_WORLD, &rank); MPI_Comm_size(MPI_COMM_WORLD, &world_size); MPI_Comm_get_parent(&manager_intercom); unsigned array[100]{1}; cout << "WORKER: starting reduction\n"; MPI_Reduce(array, NULL, 100, MPI_UNSIGNED, MPI_SUM, MPI_PROC_NULL, manager_intercom); cout << "WORKER: finishing reduction\n"; sleep(10); MPI_Finalize(); } Finally, here it the invocation: $ mpiexec -launcher ssh -print-all-exitcodes -wdir /home/kmccall/test_dir -np 1 -ppn 1 test_reduce_man Thanks, Kurt

1 0

Re: [mpich-discuss] MPI_Reduce on an inter-communicator hangs
by Jenke, Joachim 21 Apr '24

21 Apr '24

Hi Kurt, The workers need to pass root=0. Mpi_proc_null indicates that the local process is not involved in the communication (like in p2p). This would be the appropriate argument for non-root arguments in the manager group, i.e., if you start with more than one process. > all other processes in the same group as the root use MPI_PROC_NULL https://urldefense.us/v3/__https://www.mpi-forum.org/docs/mpi-2.2/mpi22-rep… It also explains your observation of falling through the call without waiting. Best Joachim ________________________________ From: Mccall, Kurt E. (MSFC-EV41) via discuss <discuss(a)mpich.org> Sent: Sunday, April 21, 2024 11:52:48 PM To: discuss(a)mpich.org <discuss(a)mpich.org> Cc: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov> Subject: [mpich-discuss] MPI_Reduce on an inter-communicator hangs I am calling MPI_Reduce on a set of inter-communicators created by MPI_Comm_spawn, each with one process in the local group (the single manager) and two processes in the remote group (the workers). The inter- communicators are visited one at ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd I am calling MPI_Reduce on a set of inter-communicators created by MPI_Comm_spawn, each with one process in the local group (the single manager) and two processes in the remote group (the workers). The inter- communicators are visited one at a time in the manager. All workers enter and exit MPI_Reduce without blocking, but the manager enters the first MPI_Reduce for the first inter-communicator and never returns. What am I doing wrong? I am using MPICH 4.1.2. Here is my manager code: #define N_PROC 4 #define N_IN_GROUP 2 int main(int argc, char *argv[]) { int rank, world_size, error_codes[N_PROC]; MPI_Comm intercoms[N_PROC]; char hostname[64]; MPI_Init(&argc, &argv); MPI_Comm_rank(MPI_COMM_WORLD, &rank); MPI_Comm_size(MPI_COMM_WORLD, &world_size); gethostname(hostname, sizeof(hostname)); char *p = strstr(hostname, "."); *p = '\0'; MPI_Info info; MPI_Info_create(&info); MPI_Info_set(info, "host", hostname); MPI_Info_set(info, "bind_to", "core"); for (int i = 0; i < N_PROC; ++i) { MPI_Comm_spawn("test_reduce_work", argv, N_IN_GROUP, info, 0, MPI_COMM_SELF, &intercoms[i], &error_codes[i]); } sleep(10); unsigned array[100]{0}; for (int i = 0; i < N_PROC; ++i) { cout << "MANAGER: starting reduction " << i << "\n"; MPI_Reduce(NULL, array, 100, MPI_UNSIGNED, MPI_SUM, MPI_ROOT, intercoms[i]); cout << "MANAGER: finished reduction " << i << "\n"; // we never reach this point } for (int i = 0; i < 100; ++i) cout << array[i] << " "; cout << endl; MPI_Finalize(); } And here is my worker code: int main(int argc, char *argv[]) { int rank, world_size; MPI_Comm manager_intercom; MPI_Init(&argc, &argv); MPI_Comm_rank(MPI_COMM_WORLD, &rank); MPI_Comm_size(MPI_COMM_WORLD, &world_size); MPI_Comm_get_parent(&manager_intercom); unsigned array[100]{1}; cout << "WORKER: starting reduction\n"; MPI_Reduce(array, NULL, 100, MPI_UNSIGNED, MPI_SUM, MPI_PROC_NULL, manager_intercom); cout << "WORKER: finishing reduction\n"; sleep(10); MPI_Finalize(); } Finally, here it the invocation: $ mpiexec -launcher ssh -print-all-exitcodes -wdir /home/kmccall/test_dir -np 1 -ppn 1 test_reduce_man Thanks, Kurt

1 0