- discuss - mailman3.cels.anl.gov

MPI debugging using PTP
by Dr. Yehuda Singer 09 Jun '24

09 Jun '24

Dear Sir/Mrs. I use the Eclipse scientific package. . I develop automatic parallelization. One mode is that we use multiple processors connected in a network. We want that all the cores of each processors will work efficiently on The application. The first stage is to take a 40 cores machine and run the application in groups of 8 cores using MPI. The only way to do it is using the PTP. In any case the Eclipses scientific activates the PTP. There were the following problems in activating the parallel debugger: 1. The script for MPICH2 to start debugging was full with logical errors. I revised it and now it works. I can send it to you. 2. The SDM package which is written in Chas not been maintained. I organized it as an Eclipse C/C++ project and changed definitions which do not follow the C standard. C98 Enclosed please find: 1. The MPI code in C++ 2. Screen snap-shot. Best regards, Yehuda Dr. Yehuda Singer Parallel Solutions Ltd. https://urldefense.us/v3/__http://www.llp.co.il__;!!G_uCfscf7eWS!biR3k0xs8t… <https://urldefense.us/v3/__http://www.llp.co.il/__;!!G_uCfscf7eWS!biR3k0xs8… > Office: 972-74-7012159 Mobile: 972-52-2306-311

1 0

Re: [mpich-discuss] Zero-copy non-contiguous types
by Zhou, Hui 01 May '24

01 May '24

As long as the configure find the xpmem – use --with-xpmem= to set its installation path – it should configure and use xpmem for IPC by default. There is environment variable MPIR_CVAR_CH4_XPMEM_ENABLE which is set to 1 by default. And there is MPIR_CVAR_CH4_IPC_XPMEM_P2P_THRESHOLD which is set to 16384 by default. -- Hui ________________________________ From: Pascal Boeschoten via discuss <discuss(a)mpich.org> Sent: Wednesday, May 1, 2024 12:58 PM Cc: Pascal Boeschoten <pascal.boeschoten(a)hapteon.com>; discuss(a)mpich.org <discuss(a)mpich.org> Subject: Re: [mpich-discuss] Zero-copy non-contiguous types Hello, Thanks. Besides compiling MPICH with xpmem support, are there any particular configuration options or environment variables needed for this to happen? Kind regards, Pascal Boeschoten On Mon, 29 Apr 2024 at 15: 54, Zhou, Hui <zhouh@ anl. gov> ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Hello, Thanks. Besides compiling MPICH with xpmem support, are there any particular configuration options or environment variables needed for this to happen? Kind regards, Pascal Boeschoten On Mon, 29 Apr 2024 at 15:54, Zhou, Hui <zhouh(a)anl.gov<mailto:[email protected]>> wrote: Yes, intra-node ipc including non-contig datatypes is supported via xpmem. -- Hui Zhou ________________________________ From: Pascal Boeschoten via discuss <discuss(a)mpich.org<mailto:[email protected]>> Sent: Monday, April 29, 2024 2:37 AM To: discuss(a)mpich.org<mailto:[email protected]> <discuss(a)mpich.org<mailto:[email protected]>> Cc: Pascal Boeschoten <pascal.boeschoten(a)hapteon.com<mailto:[email protected]>> Subject: [mpich-discuss] Zero-copy non-contiguous types Hello, Does MPICH support zero-copy intra-node communication for non-contiguous datatypes? Thanks, Pascal Boeschoten ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Hello, Does MPICH support zero-copy intra-node communication for non-contiguous datatypes? Thanks, Pascal Boeschoten

1 0

Re: [mpich-discuss] Zero-copy non-contiguous types
by Pascal Boeschoten 01 May '24

01 May '24

Hello, Thanks. Besides compiling MPICH with xpmem support, are there any particular configuration options or environment variables needed for this to happen? Kind regards, Pascal Boeschoten On Mon, 29 Apr 2024 at 15:54, Zhou, Hui <zhouh(a)anl.gov> wrote: > Yes, intra-node ipc including non-contig datatypes is supported via xpmem. > > -- > Hui Zhou > ------------------------------ > *From:* Pascal Boeschoten via discuss <discuss(a)mpich.org> > *Sent:* Monday, April 29, 2024 2:37 AM > *To:* discuss(a)mpich.org <discuss(a)mpich.org> > *Cc:* Pascal Boeschoten <pascal.boeschoten(a)hapteon.com> > *Subject:* [mpich-discuss] Zero-copy non-contiguous types > > Hello, Does MPICH support zero-copy intra-node communication for > non-contiguous datatypes? Thanks, Pascal Boeschoten ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ > ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ > ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ > ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ > ZjQcmQRYFpfptBannerStart > This Message Is From an External Sender > This message came from outside your organization. > > ZjQcmQRYFpfptBannerEnd > Hello, > > Does MPICH support zero-copy intra-node communication for non-contiguous > datatypes? > > Thanks, > Pascal Boeschoten >

1 0

Re: [mpich-discuss] Differences between ch3:nemesis and ch4:ofi:tcp with MPI_Barrier before completion of MPI_Isend
by Raffenetti, Ken 29 Apr '24

29 Apr '24

I was able to get the code from the list archive. I will update after I dig in. Thanks, Ken From: Edric Ellis <eellis(a)mathworks.com> Date: Friday, April 26, 2024 at 5:11 AM To: "Raffenetti, Ken" <raffenet(a)anl.gov>, "discuss(a)mpich.org" <discuss(a)mpich.org> Subject: RE: [mpich-discuss] Differences between ch3:nemesis and ch4:ofi:tcp with MPI_Barrier before completion of MPI_Isend Hm, I attempted to attach my full code, but maybe it didn’t get through correctly. Weirdly though when I look at the web view of this discussion group, *only* the code shows up (initially): https: //lists. mpich. org/pipermail/discuss/2024-April/006578. html ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Hm, I attempted to attach my full code, but maybe it didn’t get through correctly. Weirdly though when I look at the web view of this discussion group, *only* the code shows up (initially): https://urldefense.us/v3/__https://lists.mpich.org/pipermail/discuss/2024-A… <https://urldefense.us/v3/__https:/lists.mpich.org/pipermail/discuss/2024-Ap…> Hopefully that is visible? (I think the problem shows up only when using 2 hosts) (On looking again at that code, there’s a stray logging message about MPI_Comm_free – initially I was using separate communicators for the ISend and the Barrier – but that seems to make no difference). Cheers, Edric. From: Raffenetti, Ken <raffenet(a)anl.gov> Sent: Thursday, April 25, 2024 7:00 PM To: discuss(a)mpich.org Cc: Edric Ellis <eellis(a)mathworks.com> Subject: Re: [mpich-discuss] Differences between ch3:nemesis and ch4:ofi:tcp with MPI_Barrier before completion of MPI_Isend Hi Edric, I don’t see anything wrong in your pseudo code. I believe it is a correct pattern. I ran some experiments myself on some local machines and could not cause a hang, so if you come up with a reproducer please send it along. Ken From: Edric Ellis via discuss <discuss(a)mpich.org<mailto:[email protected]>> Reply-To: "discuss(a)mpich.org<mailto:[email protected]>" <discuss(a)mpich.org<mailto:[email protected]>> Date: Wednesday, April 24, 2024 at 3:05 AM To: "discuss(a)mpich.org<mailto:[email protected]>" <discuss(a)mpich.org<mailto:[email protected]>> Cc: Edric Ellis <eellis(a)mathworks.com<mailto:[email protected]>> Subject: [mpich-discuss] Differences between ch3:nemesis and ch4:ofi:tcp with MPI_Barrier before completion of MPI_Isend I'm trying to understand if a change in behaviour I'm seeing is expected or not. My code initiates an MPI_Isend on rank==1, and before waiting for completion of that send, all ranks perform an MPI_Barrier. This works fine on ch3: nemesis. It ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd I'm trying to understand if a change in behaviour I'm seeing is expected or not. My code initiates an MPI_Isend on rank==1, and before waiting for completion of that send, all ranks perform an MPI_Barrier. This works fine on ch3:nemesis. It works fine on ch4:ofi when the message is small (presumably using the "eager" protocol). When using ch4:ofi (the embedded tcp provider) and the message is large (presumably switching to "rendezvous" protocol), rank 0 never leaves the MPI_Barrier call. (I think the SHM piece of ch4 does not show the problem) Should this work? I cannot find anything in the MPI standard that says it should not, but perhaps I'm not looking in the right place. I'm using mpich-4.1.2 in both cases, either in "--with-device=ch3:nemesis" mode or "--with-libfabric=embedded --with-device=ch4:ofi:tcp". Here's a sketch of the problematic section of code, I'll attempt to attach a full reproduction (but I'm not sure if that works?) // setup code... if (rank == 1) { MPI_Isend(data, count, MPI_INT, 0, TAG, comm, &req); } else { std::this_thread::sleep_for(std::chrono::seconds(1)); } MPI_Barrier(comm); MPI_Barrier(comm); // cleanup code, receive the message etc... Cheers, Edric.

1 0

Re: [mpich-discuss] Zero-copy non-contiguous types
by Zhou, Hui 29 Apr '24

29 Apr '24

Yes, intra-node ipc including non-contig datatypes is supported via xpmem. -- Hui Zhou ________________________________ From: Pascal Boeschoten via discuss <discuss(a)mpich.org> Sent: Monday, April 29, 2024 2:37 AM To: discuss(a)mpich.org <discuss(a)mpich.org> Cc: Pascal Boeschoten <pascal.boeschoten(a)hapteon.com> Subject: [mpich-discuss] Zero-copy non-contiguous types Hello, Does MPICH support zero-copy intra-node communication for non-contiguous datatypes? Thanks, Pascal Boeschoten ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Hello, Does MPICH support zero-copy intra-node communication for non-contiguous datatypes? Thanks, Pascal Boeschoten

1 0

Zero-copy non-contiguous types
by Pascal Boeschoten 29 Apr '24

29 Apr '24

Hello, Does MPICH support zero-copy intra-node communication for non-contiguous datatypes? Thanks, Pascal Boeschoten

1 0

Re: [mpich-discuss] Differences between ch3:nemesis and ch4:ofi:tcp with MPI_Barrier before completion of MPI_Isend
by Edric Ellis 26 Apr '24

26 Apr '24

Hm, I attempted to attach my full code, but maybe it didn’t get through correctly. Weirdly though when I look at the web view of this discussion group, *only* the code shows up (initially): https://urldefense.us/v3/__https://lists.mpich.org/pipermail/discuss/2024-A… Hopefully that is visible? (I think the problem shows up only when using 2 hosts) (On looking again at that code, there’s a stray logging message about MPI_Comm_free – initially I was using separate communicators for the ISend and the Barrier – but that seems to make no difference). Cheers, Edric. From: Raffenetti, Ken <raffenet(a)anl.gov> Sent: Thursday, April 25, 2024 7:00 PM To: discuss(a)mpich.org Cc: Edric Ellis <eellis(a)mathworks.com> Subject: Re: [mpich-discuss] Differences between ch3:nemesis and ch4:ofi:tcp with MPI_Barrier before completion of MPI_Isend Hi Edric, I don’t see anything wrong in your pseudo code. I believe it is a correct pattern. I ran some experiments myself on some local machines and could not cause a hang, so if you come up with a reproducer please send it along. Ken From: Edric Ellis via discuss <discuss(a)mpich.org<mailto:[email protected]>> Reply-To: "discuss(a)mpich.org<mailto:[email protected]>" <discuss(a)mpich.org<mailto:[email protected]>> Date: Wednesday, April 24, 2024 at 3:05 AM To: "discuss(a)mpich.org<mailto:[email protected]>" <discuss(a)mpich.org<mailto:[email protected]>> Cc: Edric Ellis <eellis(a)mathworks.com<mailto:[email protected]>> Subject: [mpich-discuss] Differences between ch3:nemesis and ch4:ofi:tcp with MPI_Barrier before completion of MPI_Isend I'm trying to understand if a change in behaviour I'm seeing is expected or not. My code initiates an MPI_Isend on rank==1, and before waiting for completion of that send, all ranks perform an MPI_Barrier. This works fine on ch3: nemesis. It ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd I'm trying to understand if a change in behaviour I'm seeing is expected or not. My code initiates an MPI_Isend on rank==1, and before waiting for completion of that send, all ranks perform an MPI_Barrier. This works fine on ch3:nemesis. It works fine on ch4:ofi when the message is small (presumably using the "eager" protocol). When using ch4:ofi (the embedded tcp provider) and the message is large (presumably switching to "rendezvous" protocol), rank 0 never leaves the MPI_Barrier call. (I think the SHM piece of ch4 does not show the problem) Should this work? I cannot find anything in the MPI standard that says it should not, but perhaps I'm not looking in the right place. I'm using mpich-4.1.2 in both cases, either in "--with-device=ch3:nemesis" mode or "--with-libfabric=embedded --with-device=ch4:ofi:tcp". Here's a sketch of the problematic section of code, I'll attempt to attach a full reproduction (but I'm not sure if that works?) // setup code... if (rank == 1) { MPI_Isend(data, count, MPI_INT, 0, TAG, comm, &req); } else { std::this_thread::sleep_for(std::chrono::seconds(1)); } MPI_Barrier(comm); MPI_Barrier(comm); // cleanup code, receive the message etc... Cheers, Edric.

1 0

Re: [mpich-discuss] Differences between ch3:nemesis and ch4:ofi:tcp with MPI_Barrier before completion of MPI_Isend
by Raffenetti, Ken 25 Apr '24

25 Apr '24

Hi Edric, I don’t see anything wrong in your pseudo code. I believe it is a correct pattern. I ran some experiments myself on some local machines and could not cause a hang, so if you come up with a reproducer please send it along. Ken From: Edric Ellis via discuss <discuss(a)mpich.org> Reply-To: "discuss(a)mpich.org" <discuss(a)mpich.org> Date: Wednesday, April 24, 2024 at 3:05 AM To: "discuss(a)mpich.org" <discuss(a)mpich.org> Cc: Edric Ellis <eellis(a)mathworks.com> Subject: [mpich-discuss] Differences between ch3:nemesis and ch4:ofi:tcp with MPI_Barrier before completion of MPI_Isend I'm trying to understand if a change in behaviour I'm seeing is expected or not. My code initiates an MPI_Isend on rank==1, and before waiting for completion of that send, all ranks perform an MPI_Barrier. This works fine on ch3: nemesis. It ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd I'm trying to understand if a change in behaviour I'm seeing is expected or not. My code initiates an MPI_Isend on rank==1, and before waiting for completion of that send, all ranks perform an MPI_Barrier. This works fine on ch3:nemesis. It works fine on ch4:ofi when the message is small (presumably using the "eager" protocol). When using ch4:ofi (the embedded tcp provider) and the message is large (presumably switching to "rendezvous" protocol), rank 0 never leaves the MPI_Barrier call. (I think the SHM piece of ch4 does not show the problem) Should this work? I cannot find anything in the MPI standard that says it should not, but perhaps I'm not looking in the right place. I'm using mpich-4.1.2 in both cases, either in "--with-device=ch3:nemesis" mode or "--with-libfabric=embedded --with-device=ch4:ofi:tcp". Here's a sketch of the problematic section of code, I'll attempt to attach a full reproduction (but I'm not sure if that works?) // setup code... if (rank == 1) { MPI_Isend(data, count, MPI_INT, 0, TAG, comm, &req); } else { std::this_thread::sleep_for(std::chrono::seconds(1)); } MPI_Barrier(comm); MPI_Barrier(comm); // cleanup code, receive the message etc... Cheers, Edric.

1 0

Differences between ch3:nemesis and ch4:ofi:tcp with MPI_Barrier before completion of MPI_Isend
by Edric Ellis 24 Apr '24

24 Apr '24

1 0

Re: [mpich-discuss] Environment variable forwarding using Hydra and "-launcher ssh"
by Raffenetti, Ken 22 Apr '24

22 Apr '24

Hi Edric, It looks like we may have unintentionally inverted the logic on that function return value. I’ll submit a PR to fix. Thanks for bringing it to our attention. Ken From: Edric Ellis via discuss <discuss(a)mpich.org> Reply-To: "discuss(a)mpich.org" <discuss(a)mpich.org> Date: Monday, April 22, 2024 at 9:44 AM To: "discuss(a)mpich.org" <discuss(a)mpich.org> Cc: Edric Ellis <eellis(a)mathworks.com> Subject: [mpich-discuss] Environment variable forwarding using Hydra and "-launcher ssh" We’re in the process of moving from mpich-3. x to mpich-4. 1. 2. We’ve run into some odd behaviour on SLURM related to environment variable forwarding by mpiexec. It looks like mpiexec now propagates only SLURM_* environment variables, ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd We’re in the process of moving from mpich-3.x to mpich-4.1.2. We’ve run into some odd behaviour on SLURM related to environment variable forwarding by mpiexec. It looks like mpiexec now propagates only SLURM_* environment variables, instead of filtering them out (or intending to). Consider something like this: $ mpiexec -launcher slurm printenv HOME SLURM_JOBID Using mpich-3.x, the HOME variable gets forward. Using mpich-4.1.2, it does not. I believe that mpich-3.x intends to filter out SLURM_JOBID, but the value still seems to be present, maybe srun forwards that. It’s the fact that HOME doesn’t get through using mpich-4.1.2 that is causing us problems. Running mpich- Here’s what I think is the relevant change for SLURM: https://urldefense.us/v3/__https://github.com/pmodels/mpich/commit/95ba4ddc… <https://urldefense.us/v3/__https:/github.com/pmodels/mpich/commit/95ba4ddc7…> . Am I reading that correctly? The doc here https://urldefense.us/v3/__https://github.com/pmodels/mpich/blob/main/doc/w… <https://urldefense.us/v3/__https:/github.com/pmodels/mpich/blob/main/doc/wi…> states that SLURM_ things should be filtered out, but that doesn’t appear to be happening? For reference, here’s what mpich-4.1.2 “mpiexec -verbose -launcher slurm” prints: mpiexec options: ---------------- Base path: /path/to/mpich-4.1.2 Launcher: slurm Debug level: 1 Enable X: -1 Global environment: ------------------- SLURM_JOBID=102437 SLURM_JOB_USER=eellis SLURM_JOB_QOS=normal SLURM_JOB_NUM_NODES=2 SLURM_TASKS_PER_NODE=1(x2) SLURM_TOPOLOGY_ADDR_PATTERN=node … many more SLURM_* And here’s what mpich-3.x prints: mpiexec options: ---------------- Base path: /path/to/mpich-3.x Launcher: slurm Debug level: 1 Enable X: -1 Global environment: ------------------- ALTERNATE_EDITOR=emacs MAIL=/var/mail/eellis USER=eellis SLURM_JOB_USER=eellis l=/local/eellis XDG_SESSION_TYPE=unspecified SLURM_JOB_QOS=normal … no SLURM_JOBID Cheers, Edric.

1 0