July 2025 - discuss - mailman3.cels.anl.gov

Re: [mpich-discuss] MPICH 5.0.1 performance on HPE SS11 plus more - a slurm problem
by Howard Pritchard 30 Jul '25

30 Jul '25

Hi Rajeev, Here are the results for 4.3.x branch: hpp@nid001293:/usr/projects/artab/users/hpp/osu-micro-benchmarks-5.8-mpich/mpi/pt2pt>srun --mpi=pmix -n 2 ./osu_latency # OSU MPI Latency Test v5.8 # Size Latency (us) 0 1.92 1 1.98 2 1.98 4 1.98 8 1.98 16 1.98 32 1.99 64 1.99 128 2.47 256 2.59 512 2.65 1024 2.76 2048 2.95 4096 3.00 8192 5.96 16384 6.64 32768 7.44 65536 8.75 131072 11.52 262144 17.08 524288 27.96 1048576 49.38 2097152 92.96 4194304 179.74 These are more like i would expect for SS11/OFI CXI provider. Howard Am Mi., 30. Juli 2025 um 12:48 Uhr schrieb Thakur, Rajeev <thakur(a)anl.gov>: > Hi Howard, > > What was the latency with the 4.3.x branch? > > > > Rajeev > > > > > > *From: *Howard Pritchard via discuss <discuss(a)mpich.org> > *Reply-To: *"discuss(a)mpich.org" <discuss(a)mpich.org> > *Date: *Wednesday, July 30, 2025 at 1:43 PM > *To: *"Zhou, Hui" <zhouh(a)anl.gov> > *Cc: *Howard Pritchard <hppritcha(a)gmail.com>, "discuss(a)mpich.org" < > discuss(a)mpich.org> > *Subject: *Re: [mpich-discuss] MPICH 5.0.1 performance on HPE SS11 plus > more - a slurm problem > > > > Hi Hui That didn’t help. I am not surprised though as our cluster is an > NVIDIA free zone. What did help is to switch to the mpich 4. 3. x branch > and latency results are nominal and the slurm problem went away too. So we > will stick with that branch. > > ZjQcmQRYFpfptBannerStart > > *This Message Is From an External Sender * > > This message came from outside your organization. > > ZjQcmQRYFpfptBannerEnd > > Hi Hui > > > > That didn’t help. I am not surprised though as our cluster is an NVIDIA > free zone. What did help is to switch to the mpich 4.3.x branch and > latency results are nominal and the slurm problem went away too. So we > will stick with that branch. > > > > Howard > > > > On Mon, Jul 28, 2025 at 4:15 PM Zhou, Hui <zhouh(a)anl.gov> wrote: > > Hi Howard, > > > > I wonder whether it is due to the overhead of querying pointer > attributes. Could you try disable GPU support with `MPIR_CVAR_ENABLE_GPU=0` > and see if the latency improves? > > > > Hui > ------------------------------ > > *From:* Howard Pritchard via discuss <discuss(a)mpich.org> > *Sent:* Monday, July 28, 2025 9:41 AM > *To:* discuss(a)mpich.org <discuss(a)mpich.org> > *Cc:* Howard Pritchard <hppritcha(a)gmail.com> > *Subject:* [mpich-discuss] MPICH 5.0.1 performance on HPE SS11 plus more > - a slurm problem > > > > Hi Folks, We are seeing a strange performance issue on our HPE SS11 system > when testing osu_latency inter-node with MPICH. First the info: system > using libfabric 1. 22. 0 slurm - 24. 11. 5 Here's my mpichversion output: > MPICH Version: 5. 0. 0a1 > > ZjQcmQRYFpfptBannerStart > > *This Message Is From an External Sender * > > This message came from outside your organization. > > > > ZjQcmQRYFpfptBannerEnd > > Hi Folks, > > > > We are seeing a strange performance issue on our HPE SS11 system when > testing osu_latency inter-node with MPICH. > > > > First the info: > > system using libfabric 1.22.0 > > slurm - 24.11.5 > > > > Here's my mpichversion output: > > > > MPICH Version: 5.0.0a1 > > MPICH Release date: unreleased development copy > > MPICH ABI: 0:0:0 > > MPICH Device: ch4:ofi > > MPICH configure: --prefix=/XXXX/mpich_again/install --enable-g=no > --enable-error-checking=no --with-device=ch4:ofi --enable-threads=multiple > --with-ch4-shmmods=posix,xpmem --enable-thread-cs=per-vci > --with-libfabric=/opt/cray/libfabric/1.22.0 > --with-xpmem=/opt/cray/xpmem/default --with-pmix=/opt/pmix/gcc4x/5.0.8 > --enable-fast=O3 > > MPICH CC: gcc -O3 > > MPICH CXX: g++ -O3 > > MPICH F77: gfortran -O3 > > MPICH FC: gfortran -O3 > > MPICH features: threadcomm > > > > And here's the OSU latency results: > > > slurmstepd: error: mpi/pmix_v4: pmixp_coll_belong_chk: nid001439 [1]: > pmixp_coll.c:280: No process controlled by this slurmstepd is involved in > this collective. > > slurmstepd: error: mpi/pmix_v4: _process_server_request: nid001439 [1]: > pmixp_server.c:923: Unable to pmixp_state_coll_get() > > slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_check: nid001438 [0]: > pmixp_coll_ring.c:614: 0x15005c005dc0: unexpected contrib from nid001439:1, > expected is 0 > > slurmstepd: error: mpi/pmix_v4: _process_server_request: nid001438 [0]: > pmixp_server.c:937: 0x15005c005dc0: unexpected contrib from nid001439:1, > coll->seq=0, seq=0 > > slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_reset_if_to: nid001438 > [0]: pmixp_coll_ring.c:738: 0x1500580532f0: collective timeout seq=0 > > slurmstepd: error: mpi/pmix_v4: pmixp_coll_log: nid001438 [0]: > pmixp_coll.c:286: Dumping collective state > > slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: > pmixp_coll_ring.c:756: 0x1500580532f0: COLL_FENCE_RING state seq=0 > > slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: > pmixp_coll_ring.c:758: my peerid: 0:nid001438 > > slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: > pmixp_coll_ring.c:765: neighbor id: next 1:nid001439, prev 1:nid001439 > > slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: > pmixp_coll_ring.c:775: Context ptr=0x150058053368, #0, in-use=0 > > slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: > pmixp_coll_ring.c:775: Context ptr=0x1500580533a0, #1, in-use=0 > > slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: > pmixp_coll_ring.c:775: Context ptr=0x1500580533d8, #2, in-use=1 > > slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: > pmixp_coll_ring.c:786: seq=0 contribs: loc=1/prev=0/fwd=1 > > slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: > pmixp_coll_ring.c:788: neighbor contribs [2]: > > slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: > pmixp_coll_ring.c:821: done contrib: - > > slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: > pmixp_coll_ring.c:823: wait contrib: nid001439 > > slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: > pmixp_coll_ring.c:825: status=PMIXP_COLL_RING_PROGRESS > > slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: > pmixp_coll_ring.c:829: buf (offset/size): 36/16384 > > slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_reset_if_to: nid001439 > [1]: pmixp_coll_ring.c:738: 0x151d0c053400: collective timeout seq=0 > > slurmstepd: error: mpi/pmix_v4: pmixp_coll_log: nid001439 [1]: > pmixp_coll.c:286: Dumping collective state > > slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: > pmixp_coll_ring.c:756: 0x151d0c053400: COLL_FENCE_RING state seq=0 > > slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: > pmixp_coll_ring.c:758: my peerid: 1:nid001439 > > slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: > pmixp_coll_ring.c:765: neighbor id: next 0:nid001438, prev 0:nid001438 > > slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: > pmixp_coll_ring.c:775: Context ptr=0x151d0c053478, #0, in-use=0 > > slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: > pmixp_coll_ring.c:775: Context ptr=0x151d0c0534b0, #1, in-use=0 > > slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: > pmixp_coll_ring.c:775: Context ptr=0x151d0c0534e8, #2, in-use=1 > > slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: > pmixp_coll_ring.c:786: seq=0 contribs: loc=1/prev=0/fwd=1 > > slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: > pmixp_coll_ring.c:788: neighbor contribs [2]: > > slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: > pmixp_coll_ring.c:821: done contrib: - > > slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: > pmixp_coll_ring.c:823: wait contrib: nid001438 > > slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: > pmixp_coll_ring.c:825: status=PMIXP_COLL_RING_PROGRESS > > slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: > pmixp_coll_ring.c:829: buf (offset/size): 36/16384 > > # OSU MPI Latency Test v5.8 > > # Size Latency (us) > > 0 1.66 > > 1 9.29 > > 2 9.57 > > 4 9.69 > > 8 9.76 > > 16 9.77 > > 32 9.76 > > 64 9.77 > > 128 10.32 > > 256 7.54 > > 512 7.45 > > 1024 7.38 > > 2048 7.37 > > 4096 7.45 > > 8192 9.21 > > 16384 9.70 > > 32768 10.63 > > 65536 13.15 > > 131072 16.96 > > 262144 23.84 > > 524288 36.16 > > 1048576 60.36 > > 2097152 108.43 > > 4194304 228.31 > > > > Note the slurm behavior is - I launch the job. Go get coffee, do some > duo-lingo, read some emails, then after about 10 minutes the osu latency > runs. > > > > I did not get the slurm problems using an older mpich 4.3.1 but did get > the same performance issue. 9 usecs doesn't seem right for an 8-byte > pingpong over libfabric S11. I was expecting more like 1.6 or so. > > > > I am confident the slurm issue is unrelated to the latency issue. > > Thanks for any suggestions on how to address either issue however. > > > > > >

1 0

Re: [mpich-discuss] MPICH 5.0.1 performance on HPE SS11 plus more - a slurm problem
by Thakur, Rajeev 30 Jul '25

30 Jul '25

Hi Howard, What was the latency with the 4.3.x branch? Rajeev From: Howard Pritchard via discuss <discuss(a)mpich.org> Reply-To: "discuss(a)mpich.org" <discuss(a)mpich.org> Date: Wednesday, July 30, 2025 at 1:43 PM To: "Zhou, Hui" <zhouh(a)anl.gov> Cc: Howard Pritchard <hppritcha(a)gmail.com>, "discuss(a)mpich.org" <discuss(a)mpich.org> Subject: Re: [mpich-discuss] MPICH 5.0.1 performance on HPE SS11 plus more - a slurm problem Hi Hui That didn’t help. I am not surprised though as our cluster is an NVIDIA free zone. What did help is to switch to the mpich 4. 3. x branch and latency results are nominal and the slurm problem went away too. So we will stick with that branch. ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Hi Hui That didn’t help. I am not surprised though as our cluster is an NVIDIA free zone. What did help is to switch to the mpich 4.3.x branch and latency results are nominal and the slurm problem went away too. So we will stick with that branch. Howard On Mon, Jul 28, 2025 at 4:15 PM Zhou, Hui <zhouh(a)anl.gov<mailto:[email protected]>> wrote: Hi Howard, I wonder whether it is due to the overhead of querying pointer attributes. Could you try disable GPU support with `MPIR_CVAR_ENABLE_GPU=0` and see if the latency improves? Hui ________________________________ From: Howard Pritchard via discuss <discuss(a)mpich.org<mailto:[email protected]>> Sent: Monday, July 28, 2025 9:41 AM To: discuss(a)mpich.org<mailto:[email protected]> <discuss(a)mpich.org<mailto:[email protected]>> Cc: Howard Pritchard <hppritcha(a)gmail.com<mailto:[email protected]>> Subject: [mpich-discuss] MPICH 5.0.1 performance on HPE SS11 plus more - a slurm problem Hi Folks, We are seeing a strange performance issue on our HPE SS11 system when testing osu_latency inter-node with MPICH. First the info: system using libfabric 1. 22. 0 slurm - 24. 11. 5 Here's my mpichversion output: MPICH Version: 5. 0. 0a1 ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Hi Folks, We are seeing a strange performance issue on our HPE SS11 system when testing osu_latency inter-node with MPICH. First the info: system using libfabric 1.22.0 slurm - 24.11.5 Here's my mpichversion output: MPICH Version: 5.0.0a1 MPICH Release date: unreleased development copy MPICH ABI: 0:0:0 MPICH Device: ch4:ofi MPICH configure: --prefix=/XXXX/mpich_again/install --enable-g=no --enable-error-checking=no --with-device=ch4:ofi --enable-threads=multiple --with-ch4-shmmods=posix,xpmem --enable-thread-cs=per-vci --with-libfabric=/opt/cray/libfabric/1.22.0 --with-xpmem=/opt/cray/xpmem/default --with-pmix=/opt/pmix/gcc4x/5.0.8 --enable-fast=O3 MPICH CC: gcc -O3 MPICH CXX: g++ -O3 MPICH F77: gfortran -O3 MPICH FC: gfortran -O3 MPICH features: threadcomm And here's the OSU latency results: slurmstepd: error: mpi/pmix_v4: pmixp_coll_belong_chk: nid001439 [1]: pmixp_coll.c:280: No process controlled by this slurmstepd is involved in this collective. slurmstepd: error: mpi/pmix_v4: _process_server_request: nid001439 [1]: pmixp_server.c:923: Unable to pmixp_state_coll_get() slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_check: nid001438 [0]: pmixp_coll_ring.c:614: 0x15005c005dc0: unexpected contrib from nid001439:1, expected is 0 slurmstepd: error: mpi/pmix_v4: _process_server_request: nid001438 [0]: pmixp_server.c:937: 0x15005c005dc0: unexpected contrib from nid001439:1, coll->seq=0, seq=0 slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_reset_if_to: nid001438 [0]: pmixp_coll_ring.c:738: 0x1500580532f0: collective timeout seq=0 slurmstepd: error: mpi/pmix_v4: pmixp_coll_log: nid001438 [0]: pmixp_coll.c:286: Dumping collective state slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: pmixp_coll_ring.c:756: 0x1500580532f0: COLL_FENCE_RING state seq=0 slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: pmixp_coll_ring.c:758: my peerid: 0:nid001438 slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: pmixp_coll_ring.c:765: neighbor id: next 1:nid001439, prev 1:nid001439 slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: pmixp_coll_ring.c:775: Context ptr=0x150058053368, #0, in-use=0 slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: pmixp_coll_ring.c:775: Context ptr=0x1500580533a0, #1, in-use=0 slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: pmixp_coll_ring.c:775: Context ptr=0x1500580533d8, #2, in-use=1 slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: pmixp_coll_ring.c:786: seq=0 contribs: loc=1/prev=0/fwd=1 slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: pmixp_coll_ring.c:788: neighbor contribs [2]: slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: pmixp_coll_ring.c:821: done contrib: - slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: pmixp_coll_ring.c:823: wait contrib: nid001439 slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: pmixp_coll_ring.c:825: status=PMIXP_COLL_RING_PROGRESS slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: pmixp_coll_ring.c:829: buf (offset/size): 36/16384 slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_reset_if_to: nid001439 [1]: pmixp_coll_ring.c:738: 0x151d0c053400: collective timeout seq=0 slurmstepd: error: mpi/pmix_v4: pmixp_coll_log: nid001439 [1]: pmixp_coll.c:286: Dumping collective state slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: pmixp_coll_ring.c:756: 0x151d0c053400: COLL_FENCE_RING state seq=0 slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: pmixp_coll_ring.c:758: my peerid: 1:nid001439 slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: pmixp_coll_ring.c:765: neighbor id: next 0:nid001438, prev 0:nid001438 slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: pmixp_coll_ring.c:775: Context ptr=0x151d0c053478, #0, in-use=0 slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: pmixp_coll_ring.c:775: Context ptr=0x151d0c0534b0, #1, in-use=0 slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: pmixp_coll_ring.c:775: Context ptr=0x151d0c0534e8, #2, in-use=1 slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: pmixp_coll_ring.c:786: seq=0 contribs: loc=1/prev=0/fwd=1 slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: pmixp_coll_ring.c:788: neighbor contribs [2]: slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: pmixp_coll_ring.c:821: done contrib: - slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: pmixp_coll_ring.c:823: wait contrib: nid001438 slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: pmixp_coll_ring.c:825: status=PMIXP_COLL_RING_PROGRESS slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: pmixp_coll_ring.c:829: buf (offset/size): 36/16384 # OSU MPI Latency Test v5.8 # Size Latency (us) 0 1.66 1 9.29 2 9.57 4 9.69 8 9.76 16 9.77 32 9.76 64 9.77 128 10.32 256 7.54 512 7.45 1024 7.38 2048 7.37 4096 7.45 8192 9.21 16384 9.70 32768 10.63 65536 13.15 131072 16.96 262144 23.84 524288 36.16 1048576 60.36 2097152 108.43 4194304 228.31 Note the slurm behavior is - I launch the job. Go get coffee, do some duo-lingo, read some emails, then after about 10 minutes the osu latency runs. I did not get the slurm problems using an older mpich 4.3.1 but did get the same performance issue. 9 usecs doesn't seem right for an 8-byte pingpong over libfabric S11. I was expecting more like 1.6 or so. I am confident the slurm issue is unrelated to the latency issue. Thanks for any suggestions on how to address either issue however.

1 0

Re: [mpich-discuss] MPICH 5.0.1 performance on HPE SS11 plus more - a slurm problem
by Howard Pritchard 30 Jul '25

30 Jul '25

Hi Hui That didn’t help. I am not surprised though as our cluster is an NVIDIA free zone. What did help is to switch to the mpich 4.3.x branch and latency results are nominal and the slurm problem went away too. So we will stick with that branch. Howard On Mon, Jul 28, 2025 at 4:15 PM Zhou, Hui <zhouh(a)anl.gov> wrote: > Hi Howard, > > I wonder whether it is due to the overhead of querying pointer > attributes. Could you try disable GPU support with `MPIR_CVAR_ENABLE_GPU=0` > and see if the latency improves? > > Hui > ------------------------------ > *From:* Howard Pritchard via discuss <discuss(a)mpich.org> > *Sent:* Monday, July 28, 2025 9:41 AM > *To:* discuss(a)mpich.org <discuss(a)mpich.org> > *Cc:* Howard Pritchard <hppritcha(a)gmail.com> > *Subject:* [mpich-discuss] MPICH 5.0.1 performance on HPE SS11 plus more > - a slurm problem > > Hi Folks, We are seeing a strange performance issue on our HPE SS11 system > when testing osu_latency inter-node with MPICH. First the info: system > using libfabric 1. 22. 0 slurm - 24. 11. 5 Here's my mpichversion output: > MPICH Version: 5. 0. 0a1 > ZjQcmQRYFpfptBannerStart > This Message Is From an External Sender > This message came from outside your organization. > > ZjQcmQRYFpfptBannerEnd > Hi Folks, > > We are seeing a strange performance issue on our HPE SS11 system when > testing osu_latency inter-node with MPICH. > > First the info: > system using libfabric 1.22.0 > slurm - 24.11.5 > > Here's my mpichversion output: > > MPICH Version: 5.0.0a1 > > MPICH Release date: unreleased development copy > > MPICH ABI: 0:0:0 > > MPICH Device: ch4:ofi > > MPICH configure: --prefix=/XXXX/mpich_again/install --enable-g=no > --enable-error-checking=no --with-device=ch4:ofi --enable-threads=multiple > --with-ch4-shmmods=posix,xpmem --enable-thread-cs=per-vci > --with-libfabric=/opt/cray/libfabric/1.22.0 > --with-xpmem=/opt/cray/xpmem/default --with-pmix=/opt/pmix/gcc4x/5.0.8 > --enable-fast=O3 > > MPICH CC: gcc -O3 > > MPICH CXX: g++ -O3 > > MPICH F77: gfortran -O3 > > MPICH FC: gfortran -O3 > > MPICH features: threadcomm > > > > And here's the OSU latency results: > > slurmstepd: error: mpi/pmix_v4: pmixp_coll_belong_chk: nid001439 [1]: > pmixp_coll.c:280: No process controlled by this slurmstepd is involved in > this collective. > > slurmstepd: error: mpi/pmix_v4: _process_server_request: nid001439 [1]: > pmixp_server.c:923: Unable to pmixp_state_coll_get() > > slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_check: nid001438 [0]: > pmixp_coll_ring.c:614: 0x15005c005dc0: unexpected contrib from nid001439:1, > expected is 0 > > slurmstepd: error: mpi/pmix_v4: _process_server_request: nid001438 [0]: > pmixp_server.c:937: 0x15005c005dc0: unexpected contrib from nid001439:1, > coll->seq=0, seq=0 > > slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_reset_if_to: nid001438 > [0]: pmixp_coll_ring.c:738: 0x1500580532f0: collective timeout seq=0 > > slurmstepd: error: mpi/pmix_v4: pmixp_coll_log: nid001438 [0]: > pmixp_coll.c:286: Dumping collective state > > slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: > pmixp_coll_ring.c:756: 0x1500580532f0: COLL_FENCE_RING state seq=0 > > slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: > pmixp_coll_ring.c:758: my peerid: 0:nid001438 > > slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: > pmixp_coll_ring.c:765: neighbor id: next 1:nid001439, prev 1:nid001439 > > slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: > pmixp_coll_ring.c:775: Context ptr=0x150058053368, #0, in-use=0 > > slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: > pmixp_coll_ring.c:775: Context ptr=0x1500580533a0, #1, in-use=0 > > slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: > pmixp_coll_ring.c:775: Context ptr=0x1500580533d8, #2, in-use=1 > > slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: > pmixp_coll_ring.c:786: seq=0 contribs: loc=1/prev=0/fwd=1 > > slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: > pmixp_coll_ring.c:788: neighbor contribs [2]: > > slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: > pmixp_coll_ring.c:821: done contrib: - > > slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: > pmixp_coll_ring.c:823: wait contrib: nid001439 > > slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: > pmixp_coll_ring.c:825: status=PMIXP_COLL_RING_PROGRESS > > slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: > pmixp_coll_ring.c:829: buf (offset/size): 36/16384 > > slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_reset_if_to: nid001439 > [1]: pmixp_coll_ring.c:738: 0x151d0c053400: collective timeout seq=0 > > slurmstepd: error: mpi/pmix_v4: pmixp_coll_log: nid001439 [1]: > pmixp_coll.c:286: Dumping collective state > > slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: > pmixp_coll_ring.c:756: 0x151d0c053400: COLL_FENCE_RING state seq=0 > > slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: > pmixp_coll_ring.c:758: my peerid: 1:nid001439 > > slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: > pmixp_coll_ring.c:765: neighbor id: next 0:nid001438, prev 0:nid001438 > > slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: > pmixp_coll_ring.c:775: Context ptr=0x151d0c053478, #0, in-use=0 > > slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: > pmixp_coll_ring.c:775: Context ptr=0x151d0c0534b0, #1, in-use=0 > > slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: > pmixp_coll_ring.c:775: Context ptr=0x151d0c0534e8, #2, in-use=1 > > slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: > pmixp_coll_ring.c:786: seq=0 contribs: loc=1/prev=0/fwd=1 > > slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: > pmixp_coll_ring.c:788: neighbor contribs [2]: > > slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: > pmixp_coll_ring.c:821: done contrib: - > > slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: > pmixp_coll_ring.c:823: wait contrib: nid001438 > > slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: > pmixp_coll_ring.c:825: status=PMIXP_COLL_RING_PROGRESS > > slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: > pmixp_coll_ring.c:829: buf (offset/size): 36/16384 > > # OSU MPI Latency Test v5.8 > > # Size Latency (us) > > 0 1.66 > > 1 9.29 > > 2 9.57 > > 4 9.69 > > 8 9.76 > > 16 9.77 > > 32 9.76 > > 64 9.77 > > 128 10.32 > > 256 7.54 > > 512 7.45 > > 1024 7.38 > > 2048 7.37 > > 4096 7.45 > > 8192 9.21 > > 16384 9.70 > > 32768 10.63 > > 65536 13.15 > > 131072 16.96 > > 262144 23.84 > > 524288 36.16 > > 1048576 60.36 > > 2097152 108.43 > > 4194304 228.31 > > > Note the slurm behavior is - I launch the job. Go get coffee, do some > duo-lingo, read some emails, then after about 10 minutes the osu latency > runs. > > > I did not get the slurm problems using an older mpich 4.3.1 but did get > the same performance issue. 9 usecs doesn't seem right for an 8-byte > pingpong over libfabric S11. I was expecting more like 1.6 or so. > > > I am confident the slurm issue is unrelated to the latency issue. > > Thanks for any suggestions on how to address either issue however. > > > >

1 0

Re: [mpich-discuss] MPICH 5.0.1 performance on HPE SS11 plus more - a slurm problem
by Zhou, Hui 28 Jul '25

28 Jul '25

Hi Howard, I wonder whether it is due to the overhead of querying pointer attributes. Could you try disable GPU support with `MPIR_CVAR_ENABLE_GPU=0` and see if the latency improves? Hui ________________________________ From: Howard Pritchard via discuss <discuss(a)mpich.org> Sent: Monday, July 28, 2025 9:41 AM To: discuss(a)mpich.org <discuss(a)mpich.org> Cc: Howard Pritchard <hppritcha(a)gmail.com> Subject: [mpich-discuss] MPICH 5.0.1 performance on HPE SS11 plus more - a slurm problem Hi Folks, We are seeing a strange performance issue on our HPE SS11 system when testing osu_latency inter-node with MPICH. First the info: system using libfabric 1. 22. 0 slurm - 24. 11. 5 Here's my mpichversion output: MPICH Version: 5. 0. 0a1 ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Hi Folks, We are seeing a strange performance issue on our HPE SS11 system when testing osu_latency inter-node with MPICH. First the info: system using libfabric 1.22.0 slurm - 24.11.5 Here's my mpichversion output: MPICH Version: 5.0.0a1 MPICH Release date: unreleased development copy MPICH ABI: 0:0:0 MPICH Device: ch4:ofi MPICH configure: --prefix=/XXXX/mpich_again/install --enable-g=no --enable-error-checking=no --with-device=ch4:ofi --enable-threads=multiple --with-ch4-shmmods=posix,xpmem --enable-thread-cs=per-vci --with-libfabric=/opt/cray/libfabric/1.22.0 --with-xpmem=/opt/cray/xpmem/default --with-pmix=/opt/pmix/gcc4x/5.0.8 --enable-fast=O3 MPICH CC: gcc -O3 MPICH CXX: g++ -O3 MPICH F77: gfortran -O3 MPICH FC: gfortran -O3 MPICH features: threadcomm And here's the OSU latency results: slurmstepd: error: mpi/pmix_v4: pmixp_coll_belong_chk: nid001439 [1]: pmixp_coll.c:280: No process controlled by this slurmstepd is involved in this collective. slurmstepd: error: mpi/pmix_v4: _process_server_request: nid001439 [1]: pmixp_server.c:923: Unable to pmixp_state_coll_get() slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_check: nid001438 [0]: pmixp_coll_ring.c:614: 0x15005c005dc0: unexpected contrib from nid001439:1, expected is 0 slurmstepd: error: mpi/pmix_v4: _process_server_request: nid001438 [0]: pmixp_server.c:937: 0x15005c005dc0: unexpected contrib from nid001439:1, coll->seq=0, seq=0 slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_reset_if_to: nid001438 [0]: pmixp_coll_ring.c:738: 0x1500580532f0: collective timeout seq=0 slurmstepd: error: mpi/pmix_v4: pmixp_coll_log: nid001438 [0]: pmixp_coll.c:286: Dumping collective state slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: pmixp_coll_ring.c:756: 0x1500580532f0: COLL_FENCE_RING state seq=0 slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: pmixp_coll_ring.c:758: my peerid: 0:nid001438 slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: pmixp_coll_ring.c:765: neighbor id: next 1:nid001439, prev 1:nid001439 slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: pmixp_coll_ring.c:775: Context ptr=0x150058053368, #0, in-use=0 slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: pmixp_coll_ring.c:775: Context ptr=0x1500580533a0, #1, in-use=0 slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: pmixp_coll_ring.c:775: Context ptr=0x1500580533d8, #2, in-use=1 slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: pmixp_coll_ring.c:786: seq=0 contribs: loc=1/prev=0/fwd=1 slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: pmixp_coll_ring.c:788: neighbor contribs [2]: slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: pmixp_coll_ring.c:821: done contrib: - slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: pmixp_coll_ring.c:823: wait contrib: nid001439 slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: pmixp_coll_ring.c:825: status=PMIXP_COLL_RING_PROGRESS slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: pmixp_coll_ring.c:829: buf (offset/size): 36/16384 slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_reset_if_to: nid001439 [1]: pmixp_coll_ring.c:738: 0x151d0c053400: collective timeout seq=0 slurmstepd: error: mpi/pmix_v4: pmixp_coll_log: nid001439 [1]: pmixp_coll.c:286: Dumping collective state slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: pmixp_coll_ring.c:756: 0x151d0c053400: COLL_FENCE_RING state seq=0 slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: pmixp_coll_ring.c:758: my peerid: 1:nid001439 slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: pmixp_coll_ring.c:765: neighbor id: next 0:nid001438, prev 0:nid001438 slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: pmixp_coll_ring.c:775: Context ptr=0x151d0c053478, #0, in-use=0 slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: pmixp_coll_ring.c:775: Context ptr=0x151d0c0534b0, #1, in-use=0 slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: pmixp_coll_ring.c:775: Context ptr=0x151d0c0534e8, #2, in-use=1 slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: pmixp_coll_ring.c:786: seq=0 contribs: loc=1/prev=0/fwd=1 slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: pmixp_coll_ring.c:788: neighbor contribs [2]: slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: pmixp_coll_ring.c:821: done contrib: - slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: pmixp_coll_ring.c:823: wait contrib: nid001438 slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: pmixp_coll_ring.c:825: status=PMIXP_COLL_RING_PROGRESS slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: pmixp_coll_ring.c:829: buf (offset/size): 36/16384 # OSU MPI Latency Test v5.8 # Size Latency (us) 0 1.66 1 9.29 2 9.57 4 9.69 8 9.76 16 9.77 32 9.76 64 9.77 128 10.32 256 7.54 512 7.45 1024 7.38 2048 7.37 4096 7.45 8192 9.21 16384 9.70 32768 10.63 65536 13.15 131072 16.96 262144 23.84 524288 36.16 1048576 60.36 2097152 108.43 4194304 228.31 Note the slurm behavior is - I launch the job. Go get coffee, do some duo-lingo, read some emails, then after about 10 minutes the osu latency runs. I did not get the slurm problems using an older mpich 4.3.1 but did get the same performance issue. 9 usecs doesn't seem right for an 8-byte pingpong over libfabric S11. I was expecting more like 1.6 or so. I am confident the slurm issue is unrelated to the latency issue. Thanks for any suggestions on how to address either issue however.

1 0

MPICH 5.0.1 performance on HPE SS11 plus more - a slurm problem
by Howard Pritchard 28 Jul '25

28 Jul '25

Hi Folks, We are seeing a strange performance issue on our HPE SS11 system when testing osu_latency inter-node with MPICH. First the info: system using libfabric 1.22.0 slurm - 24.11.5 Here's my mpichversion output: MPICH Version: 5.0.0a1 MPICH Release date: unreleased development copy MPICH ABI: 0:0:0 MPICH Device: ch4:ofi MPICH configure: --prefix=/XXXX/mpich_again/install --enable-g=no --enable-error-checking=no --with-device=ch4:ofi --enable-threads=multiple --with-ch4-shmmods=posix,xpmem --enable-thread-cs=per-vci --with-libfabric=/opt/cray/libfabric/1.22.0 --with-xpmem=/opt/cray/xpmem/default --with-pmix=/opt/pmix/gcc4x/5.0.8 --enable-fast=O3 MPICH CC: gcc -O3 MPICH CXX: g++ -O3 MPICH F77: gfortran -O3 MPICH FC: gfortran -O3 MPICH features: threadcomm And here's the OSU latency results: slurmstepd: error: mpi/pmix_v4: pmixp_coll_belong_chk: nid001439 [1]: pmixp_coll.c:280: No process controlled by this slurmstepd is involved in this collective. slurmstepd: error: mpi/pmix_v4: _process_server_request: nid001439 [1]: pmixp_server.c:923: Unable to pmixp_state_coll_get() slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_check: nid001438 [0]: pmixp_coll_ring.c:614: 0x15005c005dc0: unexpected contrib from nid001439:1, expected is 0 slurmstepd: error: mpi/pmix_v4: _process_server_request: nid001438 [0]: pmixp_server.c:937: 0x15005c005dc0: unexpected contrib from nid001439:1, coll->seq=0, seq=0 slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_reset_if_to: nid001438 [0]: pmixp_coll_ring.c:738: 0x1500580532f0: collective timeout seq=0 slurmstepd: error: mpi/pmix_v4: pmixp_coll_log: nid001438 [0]: pmixp_coll.c:286: Dumping collective state slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: pmixp_coll_ring.c:756: 0x1500580532f0: COLL_FENCE_RING state seq=0 slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: pmixp_coll_ring.c:758: my peerid: 0:nid001438 slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: pmixp_coll_ring.c:765: neighbor id: next 1:nid001439, prev 1:nid001439 slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: pmixp_coll_ring.c:775: Context ptr=0x150058053368, #0, in-use=0 slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: pmixp_coll_ring.c:775: Context ptr=0x1500580533a0, #1, in-use=0 slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: pmixp_coll_ring.c:775: Context ptr=0x1500580533d8, #2, in-use=1 slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: pmixp_coll_ring.c:786: seq=0 contribs: loc=1/prev=0/fwd=1 slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: pmixp_coll_ring.c:788: neighbor contribs [2]: slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: pmixp_coll_ring.c:821: done contrib: - slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: pmixp_coll_ring.c:823: wait contrib: nid001439 slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: pmixp_coll_ring.c:825: status=PMIXP_COLL_RING_PROGRESS slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: pmixp_coll_ring.c:829: buf (offset/size): 36/16384 slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_reset_if_to: nid001439 [1]: pmixp_coll_ring.c:738: 0x151d0c053400: collective timeout seq=0 slurmstepd: error: mpi/pmix_v4: pmixp_coll_log: nid001439 [1]: pmixp_coll.c:286: Dumping collective state slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: pmixp_coll_ring.c:756: 0x151d0c053400: COLL_FENCE_RING state seq=0 slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: pmixp_coll_ring.c:758: my peerid: 1:nid001439 slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: pmixp_coll_ring.c:765: neighbor id: next 0:nid001438, prev 0:nid001438 slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: pmixp_coll_ring.c:775: Context ptr=0x151d0c053478, #0, in-use=0 slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: pmixp_coll_ring.c:775: Context ptr=0x151d0c0534b0, #1, in-use=0 slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: pmixp_coll_ring.c:775: Context ptr=0x151d0c0534e8, #2, in-use=1 slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: pmixp_coll_ring.c:786: seq=0 contribs: loc=1/prev=0/fwd=1 slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: pmixp_coll_ring.c:788: neighbor contribs [2]: slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: pmixp_coll_ring.c:821: done contrib: - slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: pmixp_coll_ring.c:823: wait contrib: nid001438 slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: pmixp_coll_ring.c:825: status=PMIXP_COLL_RING_PROGRESS slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: pmixp_coll_ring.c:829: buf (offset/size): 36/16384 # OSU MPI Latency Test v5.8 # Size Latency (us) 0 1.66 1 9.29 2 9.57 4 9.69 8 9.76 16 9.77 32 9.76 64 9.77 128 10.32 256 7.54 512 7.45 1024 7.38 2048 7.37 4096 7.45 8192 9.21 16384 9.70 32768 10.63 65536 13.15 131072 16.96 262144 23.84 524288 36.16 1048576 60.36 2097152 108.43 4194304 228.31 Note the slurm behavior is - I launch the job. Go get coffee, do some duo-lingo, read some emails, then after about 10 minutes the osu latency runs. I did not get the slurm problems using an older mpich 4.3.1 but did get the same performance issue. 9 usecs doesn't seem right for an 8-byte pingpong over libfabric S11. I was expecting more like 1.6 or so. I am confident the slurm issue is unrelated to the latency issue. Thanks for any suggestions on how to address either issue however.

1 0

Re: [mpich-discuss] Configuration failure for 4.3.1.
by Zhou, Hui 22 Jul '25

22 Jul '25

Hi Kurt, The error is complaining the file `src/tools/perf/rocm/Makefile` is in use while config.status is trying to recreate the file. I am not sure what is causing it. Did you try again and does the same error occur? Hui ________________________________ From: Mccall, Kurt E. (MSFC-EV41) via discuss <discuss(a)mpich.org> Sent: Monday, July 21, 2025 4:19 PM To: Zhou, Hui via discuss <discuss(a)mpich.org> Cc: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov> Subject: [mpich-discuss] Configuration failure for 4.3.1. I’m having some trouble configuring 4. 3. 1. The following command finishes successfully: $ .. /mpich-4. 3. 1/configure --prefix=/opt/mpich --with-device=ch4: ofi --with-slurm -enable-debuginfo --enable-g=debug 2>&1 | tee c. txt But this ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd I’m having some trouble configuring 4.3.1. The following command finishes successfully: $ ../mpich-4.3.1/configure --prefix=/opt/mpich --with-device=ch4:ofi --with-slurm -enable-debuginfo --enable-g=debug 2>&1 | tee c.txt But this one fails with the error below: $ ../mpich-4.3.1/configure --prefix=/opt/mpich --with-device=ch4:ucx --with-ucx=embedded --with-slurm -enable-debuginfo --enable-g=debug 2>&1 | tee c.txt config.status: creating src/tools/perf/rocm/Makefile mv: cannot move './confYHXukm/out' to 'src/tools/perf/rocm/Makefile': Device or resource busy config.status: error: could not create src/tools/perf/rocm/Makefile Are my arguments incorrect? I can send c.txt in a separate message. Redhat 4.18.0, gcc 8.5.0. Thanks, Kurt

1 0

Configuration failure for 4.3.1.
by Mccall, Kurt E. (MSFC-EV41) 21 Jul '25

21 Jul '25

I'm having some trouble configuring 4.3.1. The following command finishes successfully: $ ../mpich-4.3.1/configure --prefix=/opt/mpich --with-device=ch4:ofi --with-slurm -enable-debuginfo --enable-g=debug 2>&1 | tee c.txt But this one fails with the error below: $ ../mpich-4.3.1/configure --prefix=/opt/mpich --with-device=ch4:ucx --with-ucx=embedded --with-slurm -enable-debuginfo --enable-g=debug 2>&1 | tee c.txt config.status: creating src/tools/perf/rocm/Makefile mv: cannot move './confYHXukm/out' to 'src/tools/perf/rocm/Makefile': Device or resource busy config.status: error: could not create src/tools/perf/rocm/Makefile Are my arguments incorrect? I can send c.txt in a separate message. Redhat 4.18.0, gcc 8.5.0. Thanks, Kurt

1 0

MPIX Streams Usage
by Ryan Deng 05 Jul '25

05 Jul '25

Hi, I had some questions about how MPIX_Streams work under the hood. I had posted a couple of times on the github issues page before about it but thought this might be a more appropriate place for it. Right now, my code involves many MPIX_Stream_isends/MPIX_Stream_irecvs as processes asynchronously communicate with one another, and I have a separate thread in the background constantly advancing progress on streams. It's my understanding that all operations on a stream must occur serially. I assume this means that things operations like sending and receiving, calling MPIX_Stream_progress/MPI_Wait cannot occur at the same time without locks? Are there any other operations on streams that cannot occur concurrently? In addition, I had also triggered the assertion here<https://urldefense.us/v3/__https://github.com/pmodels/mpich/blob/837914f6e8… >. [https://urldefense.us/v3/__https://opengraph.githubassets.com/a68c91e94d3a9… ]<https://urldefense.us/v3/__https://github.com/pmodels/mpich/blob/837914f6e8… > mpich/src/mpid/ch4/shm/posix/posix_progress.h at 837914f6e810bbbd5e7ad391bbad9f546ffe3dab · pmodels/mpich<https://urldefense.us/v3/__https://github.com/pmodels/mpich/blob/837914f6e8… > Official MPICH Repository. Contribute to pmodels/mpich development by creating an account on GitHub. github.com Does this mean that I cannot send multiple messages at the same time if it's between two different streams? On the github issue here<https://urldefense.us/v3/__https://github.com/pmodels/mpich/issues/7492__;!… >, Hui Zhou mentioned that it is due to messaging protocol consistency. I'm not quite sure what he means by it and what that means in terms of what I can and cannot do in my application involving MPIX_Streams. Thanks, Ryan

1 0