- discuss - mailman3.cels.anl.gov

Re: [mpich-discuss] MPICH: SHM bandwidth very low on IPC test
by Zhou, Hui 14 Aug '25

14 Aug '25

Hi Sam, I think you are not getting the 10 GB/s because the buffer you used in your test code is freshly malloc'ed. The actual memory still needs to paged in as you access it, which adds to the messaging overhead. Try repeat the measurement a few times, you should see higher bandwidth number in the later rounds. The intranode data movement is limited by CPU execution. Since there is only a single thread moving the data, it is typically limited at around 10GB/sec. In order to reach higher bandwidth, you need use more processes or threads. The latter is tricky since you need tell MPI about the thread context. With MPICH, use different communicators in different threads and enable `MPIR_CVAR_CH4_NUM_VCIS=8` (adjust the number to the number of your threads). GPU to GPU bandwidth is higher because it is offloaded to the GPU copy engine, which can max the PCIe bandwidth. -- Hui ________________________________ From: Sam Austin via discuss <discuss(a)mpich.org> Sent: Thursday, August 14, 2025 12:16 AM To: Jenke, Joachim <jenke(a)itc.rwth-aachen.de> Cc: Sam Austin <sam.austin.p(a)gmail.com>; discuss(a)mpich.org <discuss(a)mpich.org> Subject: Re: [mpich-discuss] MPICH: SHM bandwidth very low on IPC test Hi Joachim, Thanks for this suggestion! I used stream to test the single-core memory bandwidth. I am running on a Xeon E5-2699A v4, which has 55MB last level cache. So, I ran with 30 million elements per the instructions. It appears that I am ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Hi Joachim, Thanks for this suggestion! I used stream to test the single-core memory bandwidth. I am running on a Xeon E5-2699A v4, which has 55MB last level cache. So, I ran with 30 million elements per the instructions. It appears that I am seeing about 10 GB/s if I'm reading that right? If so, I am still not sure why I am only seeing ~3.5 GB/s on shared memory performance with MPICH. ------------------------------------------------------------- STREAM version $Revision: 5.10 $ ------------------------------------------------------------- This system uses 8 bytes per array element. ------------------------------------------------------------- Array size = 30000000 (elements), Offset = 0 (elements) Memory per array = 228.9 MiB (= 0.2 GiB). Total memory required = 686.6 MiB (= 0.7 GiB). Each kernel will be executed 10 times. The *best* time for each kernel (excluding the first iteration) will be used to compute the reported bandwidth. ------------------------------------------------------------- Your clock granularity/precision appears to be 1 microseconds. Each test below will take on the order of 30276 microseconds. (= 30276 clock ticks) Increase the size of the arrays if this shows that you are not getting at least 20 clock ticks per test. ------------------------------------------------------------- WARNING -- The above is only a rough guideline. For best results, please be sure you know the precision of your system timer. ------------------------------------------------------------- Function Best Rate MB/s Avg time Min time Max time Copy: 10038.1 0.048342 0.047818 0.050004 Scale: 10342.2 0.048738 0.046412 0.056605 Add: 10580.3 0.068542 0.068051 0.069805 Triad: 10703.0 0.067615 0.067271 0.068143 ------------------------------------------------------------- Solution Validates: avg error less than 1.000000e-13 on all three arrays ------------------------------------------------------------- Thanks, Sam On Wed, Aug 13, 2025 at 5:26 PM Jenke, Joachim <jenke(a)itc.rwth-aachen.de<mailto:[email protected]>> wrote: Hi Sam, Can you try out stream to understand the single-core memory bandwidth of the system? https://urldefense.us/v3/__https://www.cs.virginia.edu/stream/ref.html__;!!… <https://urldefense.us/v3/__https://www.cs.virginia.edu/stream/ref.html__;!!…> Copy bandwidth for large junks (exceeding cache sizes) should provide you an upper bound for shm communication bandwidth. Best Joachim Am 13.08.2025 22:04 schrieb Sam Austin via discuss <discuss(a)mpich.org<mailto:[email protected]>>: Hi all, I am working to configure MPICH and run a few examples on my standalone server (single node). Here are the system specs: Server: Dell PowerEdge C4130 CPUs: 2x Xeon E5-2699A v4 GPUs: 4x Tesla V100s connected with NVLink, tied to motherboard ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Hi all, I am working to configure MPICH and run a few examples on my standalone server (single node). Here are the system specs: Server: Dell PowerEdge C4130 CPUs: 2x Xeon E5-2699A v4 GPUs: 4x Tesla V100s connected with NVLink, tied to motherboard with PCIe gen 3 OS: Ubuntu 24.04 LTS I intend to use this system to develop multi-process programs for eventual execution in a large, distributed HPC environment. I ran a few tests with and without CUDA support; here is my mpichversion output: MPICH Version: 4.3.1 MPICH Release date: Fri Jun 20 09:24:41 AM CDT 2025 MPICH ABI: 17:1:5 MPICH Device: ch4:ofi MPICH configure: --prefix=/opt/mpich/4.2.1-cpu --without-cuda MPICH CC: gcc -O2 MPICH CXX: g++ -O2 MPICH F77: gfortran -O2 MPICH FC: gfortran -O2 MPICH features: threadcomm The first example that I ran was a bandwidth test for CPU-CPU and GPU-GPU communication. This simple program sends small packets back and forth between processes to test the bandwidth over the various intra-node networks. The GPU-GPU bandwidth test showed that the GPU interconnect was saturating at ~45 GB/s, which is nominal for the NVLink interconnect topology present on the node (this was run with a CUDA-aware build of MPICH). The problem appears during the CPU-CPU IPC test. In theory, this test is pretty vanilla, as it is communicating between processes using shared memory, and does not involve traversing any of the intra-node networks (PCIe or NVLink). My understanding is that the bandwidth observed on the CPU-CPU IPC test should be quite high, at least higher than 10 GB/s. However, the intra-node IPC bandwidth appears to be very low, around 3.5 GB/s, when running this test. I tried the following fixes in an attempt to force MPICH to use shared memory, but to no avail: Passing the option to explicitly specify `nemesis` during the build configuration: "--with-device=ch3:nemesis --with-cuda" Passing the option to explicitly specify shared memory with ch4 to the configuration: "--with-ch4-shmmods=posix --with-cuda" Rebuilding MPICH without GPU support: "--without-cuda" Switching to Open MPI and running the same test These results, especially the last one in which I saw the same issues when running with Open MPI, makes me think it might be an issue with my system configuration. The question is: why is the IPC bandwidth so low despite supposedly using the SHM protocol? I'm wondering if anyone has encountered this issue before or might be able to lend some advice here. Any help would be greatly appreciated! Some interesting observations from the output below: when I run with "mpiexec -np 2 -genv FI_PROVIDER=shm ...", the log file reports "Opened fabric: shm". However, when I run without "-genv FI_PROVIDER=shm", the log file reports "Opened fabric: 10.133.0.0/21<https://urldefense.us/v3/__http://10.133.0.0/21__;!!G_uCfscf7eWS!ZaUD7Nw-pS…>", which I believe means that MPICH is falling back on the TCP socket protocol. In this case, my key point of confusion is that the observed bandwidth is essentially the same between the SHM and TCP protocols. Perhaps my test script isn't set up properly? Thanks, Sam The following is attached below: Bandwidth test program Run script for the program Output of the script on my machine ---------------------------------------------------------------------------------------------------------------- In case the attachment doesn't go through, here are the contents of my test program, "shmem_check.cpp": // shmem_check.cpp // // This is a minimal benchmark to test the raw bandwidth of MPI communication // between two processes on the same node, using only host (CPU) memory. // It completely removes CUDA to isolate the performance of the MPI library's // on-node communication mechanism (e.g., shared memory vs. TCP loopback). // // Compile/run: // /opt/mpich/4.2.1-cpu/bin/mpicxx -std=c++17 -I/opt/mpich/4.2.1-cpu/include shmem_check.cpp -o shmem_check // /opt/mpich/4.2.1-cpu/bin/mpiexec -np 2 ./shmem_check #include <iostream> #include <vector> #include <numeric> #include <mpi.h> int main(int argc, char* argv[]) { MPI_Init(&argc, &argv); int rank, size; MPI_Comm_rank(MPI_COMM_WORLD, &rank); MPI_Comm_size(MPI_COMM_WORLD, &size); if (size != 2) { if (rank == 0) { std::cerr << "Error: This program must be run with exactly 2 MPI processes." << std::endl; } MPI_Finalize(); return 1; } const int num_samples = 100; const long long packet_size = 1LL << 28; // 256 MB // Allocate standard host memory. 'new' is sufficient. char* buffer = new char[packet_size]; if (rank == 0) { std::cout << "--- Starting Host-to-Host MPI Bandwidth Test ---" << std::endl; std::cout << "Packet Size: " << (packet_size / (1024*1024)) << " MB" << std::endl; } std::vector<double> timings; for (int i = 0; i < num_samples; ++i) { MPI_Barrier(MPI_COMM_WORLD); double start_time = MPI_Wtime(); if (rank == 0) { MPI_Send(buffer, packet_size, MPI_CHAR, 1, 0, MPI_COMM_WORLD); MPI_Recv(buffer, 1, MPI_CHAR, 1, 1, MPI_COMM_WORLD, MPI_STATUS_IGNORE); // Wait for confirmation } else { // rank == 1 MPI_Recv(buffer, packet_size, MPI_CHAR, 0, 0, MPI_COMM_WORLD, MPI_STATUS_IGNORE); MPI_Send(buffer, 1, MPI_CHAR, 0, 1, MPI_COMM_WORLD); // Send confirmation } double end_time = MPI_Wtime(); if (i >= 10) { // Discard warmup runs timings.push_back(end_time - start_time); } } if (rank == 0) { double total_time = std::accumulate(timings.begin(), timings.end(), 0.0); double avg_time = total_time / timings.size(); double bandwidth = (static_cast<double>(packet_size) / (1024.0 * 1024.0 * 1024.0)) / avg_time; std::cout << "------------------------------------------------" << std::endl; std::cout << "Average Host-to-Host Bandwidth: " << bandwidth << " GB/s" << std::endl; std::cout << "------------------------------------------------" << std::endl; } // Clean up host memory delete[] buffer; MPI_Finalize(); return 0; } ---------------------------------------------------------------------------------------------------------------- Here is the script to run the test with verbose compilation and the `shm` layer forced and unforced: #!/usr/bin/zsh source ~/.zshrc # Compile /opt/mpich/4.2.1-cpu/bin/mpicxx -std=c++17 -I/opt/mpich/4.2.1-cpu/include shmem_check.cpp -o shmem_check # Run with shm forced /opt/mpich/4.2.1-cpu/bin/mpiexec -np 2 -genv FI_PROVIDER=shm -genv FI_LOG_LEVEL=debug ./shmem_check 2> output_shm.txt # Run without shm forced /opt/mpich/4.2.1-cpu/bin/mpiexec -np 2 -genv FI_LOG_LEVEL=debug ./shmem_check 2> output_no_shm.txt echo "Output of script with SHM forced: " grep -i "opened fabric" output_shm.txt echo "Output of script with SHM not forced: " grep -i "opened fabric" output_no_shm.txt ---------------------------------------------------------------------------------------------------------------- Here is the output : --- Starting Host-to-Host MPI Bandwidth Test --- Packet Size: 256 MB ------------------------------------------------ Average Host-to-Host Bandwidth: 3.35709 GB/s ------------------------------------------------ --- Starting Host-to-Host MPI Bandwidth Test --- Packet Size: 256 MB ------------------------------------------------ Average Host-to-Host Bandwidth: 3.54924 GB/s ------------------------------------------------ Output of script with SHM forced: libfabric:3174297:1755114546::core:core:fi_fabric_():1503<info> Opened fabric: shm libfabric:3174298:1755114546::core:core:fi_fabric_():1503<info> Opened fabric: shm Output of script with SHM not forced: libfabric:3174351:1755114554::core:core:fi_fabric_():1503<info> Opened fabric: 10.133.0.0/21<https://urldefense.us/v3/__http://10.133.0.0/21__;!!G_uCfscf7eWS!ZaUD7Nw-pS…> libfabric:3174350:1755114554::core:core:fi_fabric_():1503<info> Opened fabric: 10.133.0.0/21<https://urldefense.us/v3/__http://10.133.0.0/21__;!!G_uCfscf7eWS!ZaUD7Nw-pS…>

1 0

Re: [mpich-discuss] MPICH: SHM bandwidth very low on IPC test
by Sam Austin 14 Aug '25

14 Aug '25

Hi Joachim, Thanks for this suggestion! I used stream to test the single-core memory bandwidth. I am running on a Xeon E5-2699A v4, which has 55MB last level cache. So, I ran with 30 million elements per the instructions. It appears that I am seeing about 10 GB/s if I'm reading that right? If so, I am still not sure why I am only seeing ~3.5 GB/s on shared memory performance with MPICH. ------------------------------------------------------------- STREAM version $Revision: 5.10 $ ------------------------------------------------------------- This system uses 8 bytes per array element. ------------------------------------------------------------- Array size = 30000000 (elements), Offset = 0 (elements) Memory per array = 228.9 MiB (= 0.2 GiB). Total memory required = 686.6 MiB (= 0.7 GiB). Each kernel will be executed 10 times. The *best* time for each kernel (excluding the first iteration) will be used to compute the reported bandwidth. ------------------------------------------------------------- Your clock granularity/precision appears to be 1 microseconds. Each test below will take on the order of 30276 microseconds. (= 30276 clock ticks) Increase the size of the arrays if this shows that you are not getting at least 20 clock ticks per test. ------------------------------------------------------------- WARNING -- The above is only a rough guideline. For best results, please be sure you know the precision of your system timer. ------------------------------------------------------------- Function Best Rate MB/s Avg time Min time Max time Copy: 10038.1 0.048342 0.047818 0.050004 Scale: 10342.2 0.048738 0.046412 0.056605 Add: 10580.3 0.068542 0.068051 0.069805 Triad: 10703.0 0.067615 0.067271 0.068143 ------------------------------------------------------------- Solution Validates: avg error less than 1.000000e-13 on all three arrays ------------------------------------------------------------- Thanks, Sam On Wed, Aug 13, 2025 at 5:26 PM Jenke, Joachim <jenke(a)itc.rwth-aachen.de> wrote: > Hi Sam, > > Can you try out stream to understand the single-core memory bandwidth of > the system? > > https://urldefense.us/v3/__https://www.cs.virginia.edu/stream/ref.html__;!!… > > Copy bandwidth for large junks (exceeding cache sizes) should provide you > an upper bound for shm communication bandwidth. > > Best > Joachim > > Am 13.08.2025 22:04 schrieb Sam Austin via discuss <discuss(a)mpich.org>: > Hi all, I am working to configure MPICH and run a few examples on my > standalone server (single node). Here are the system specs: Server: Dell > PowerEdge C4130 CPUs: 2x Xeon E5-2699A v4 GPUs: 4x Tesla V100s connected > with NVLink, tied to motherboard > ZjQcmQRYFpfptBannerStart > This Message Is From an External Sender > This message came from outside your organization. > > ZjQcmQRYFpfptBannerEnd > Hi all, > > I am working to configure MPICH and run a few examples on my standalone > server (single node). Here are the system specs: > Server: Dell PowerEdge C4130 > CPUs: 2x Xeon E5-2699A v4 > GPUs: 4x Tesla V100s connected with NVLink, tied to motherboard with PCIe > gen 3 > OS: Ubuntu 24.04 LTS > I intend to use this system to develop multi-process programs for eventual > execution in a large, distributed HPC environment. I ran a few tests with > and without CUDA support; here is my mpichversion output: > > MPICH Version: 4.3.1 > MPICH Release date: Fri Jun 20 09:24:41 AM CDT 2025 > MPICH ABI: 17:1:5 > MPICH Device: ch4:ofi > MPICH configure: --prefix=/opt/mpich/4.2.1-cpu --without-cuda > MPICH CC: gcc -O2 > MPICH CXX: g++ -O2 > MPICH F77: gfortran -O2 > MPICH FC: gfortran -O2 > MPICH features: threadcomm > > The first example that I ran was a bandwidth test for CPU-CPU and GPU-GPU > communication. This simple program sends small packets back and forth > between processes to test the bandwidth over the various intra-node > networks. > > The GPU-GPU bandwidth test showed that the GPU interconnect was saturating > at ~45 GB/s, which is nominal for the NVLink interconnect topology present > on the node (this was run with a CUDA-aware build of MPICH). The problem > appears during the CPU-CPU IPC test. In theory, this test is pretty > vanilla, as it is communicating between processes using shared memory, and > does not involve traversing any of the intra-node networks (PCIe or > NVLink). My understanding is that the bandwidth observed on the CPU-CPU IPC > test should be quite high, at least higher than 10 GB/s. > > However, the intra-node IPC bandwidth appears to be very low, around 3.5 > GB/s, when running this test. I tried the following fixes in an attempt to > force MPICH to use shared memory, but to no avail: > Passing the option to explicitly specify `nemesis` during the build > configuration: "--with-device=ch3:nemesis --with-cuda" > Passing the option to explicitly specify shared memory with ch4 to the > configuration: "--with-ch4-shmmods=posix --with-cuda" > Rebuilding MPICH without GPU support: "--without-cuda" > Switching to Open MPI and running the same test > These results, especially the last one in which I saw the same issues when > running with Open MPI, makes me think it might be an issue with my system > configuration. The question is: why is the IPC bandwidth so low despite > supposedly using the SHM protocol? I'm wondering if anyone has encountered > this issue before or might be able to lend some advice here. Any help would > be greatly appreciated! > > Some interesting observations from the output below: when I run with > "mpiexec -np 2 -genv FI_PROVIDER=shm ...", the log file reports "Opened > fabric: shm". However, when I run without "-genv FI_PROVIDER=shm", the log > file reports "Opened fabric: 10.133.0.0/21 > <https://urldefense.us/v3/__http://10.133.0.0/21__;!!G_uCfscf7eWS!ZaUD7Nw-pS…>", > which I believe means that MPICH is falling back on the TCP socket > protocol. In this case, my key point of confusion is that the observed > bandwidth is essentially the same between the SHM and TCP protocols. > Perhaps my test script isn't set up properly? > > Thanks, > Sam > > The following is attached below: > Bandwidth test program > Run script for the program > Output of the script on my machine > > ---------------------------------------------------------------------------------------------------------------- > In case the attachment doesn't go through, here are the contents of my > test program, "shmem_check.cpp": > > // shmem_check.cpp > // > // This is a minimal benchmark to test the raw bandwidth of MPI > communication > // between two processes on the same node, using only host (CPU) memory. > // It completely removes CUDA to isolate the performance of the MPI > library's > // on-node communication mechanism (e.g., shared memory vs. TCP loopback). > // > // Compile/run: > // /opt/mpich/4.2.1-cpu/bin/mpicxx -std=c++17 > -I/opt/mpich/4.2.1-cpu/include shmem_check.cpp -o shmem_check > // /opt/mpich/4.2.1-cpu/bin/mpiexec -np 2 ./shmem_check > > #include <iostream> > #include <vector> > #include <numeric> > #include <mpi.h> > > int main(int argc, char* argv[]) { > MPI_Init(&argc, &argv); > > int rank, size; > MPI_Comm_rank(MPI_COMM_WORLD, &rank); > MPI_Comm_size(MPI_COMM_WORLD, &size); > > if (size != 2) { > if (rank == 0) { > std::cerr << "Error: This program must be run with exactly 2 > MPI processes." << std::endl; > } > MPI_Finalize(); > return 1; > } > > const int num_samples = 100; > const long long packet_size = 1LL << 28; // 256 MB > > // Allocate standard host memory. 'new' is sufficient. > char* buffer = new char[packet_size]; > > if (rank == 0) { > std::cout << "--- Starting Host-to-Host MPI Bandwidth Test ---" << > std::endl; > std::cout << "Packet Size: " << (packet_size / (1024*1024)) << " > MB" << std::endl; > } > > std::vector<double> timings; > for (int i = 0; i < num_samples; ++i) { > MPI_Barrier(MPI_COMM_WORLD); > double start_time = MPI_Wtime(); > > if (rank == 0) { > MPI_Send(buffer, packet_size, MPI_CHAR, 1, 0, MPI_COMM_WORLD); > MPI_Recv(buffer, 1, MPI_CHAR, 1, 1, MPI_COMM_WORLD, > MPI_STATUS_IGNORE); // Wait for confirmation > } else { // rank == 1 > MPI_Recv(buffer, packet_size, MPI_CHAR, 0, 0, MPI_COMM_WORLD, > MPI_STATUS_IGNORE); > MPI_Send(buffer, 1, MPI_CHAR, 0, 1, MPI_COMM_WORLD); // Send > confirmation > } > > double end_time = MPI_Wtime(); > if (i >= 10) { // Discard warmup runs > timings.push_back(end_time - start_time); > } > } > > if (rank == 0) { > double total_time = std::accumulate(timings.begin(), > timings.end(), 0.0); > double avg_time = total_time / timings.size(); > double bandwidth = (static_cast<double>(packet_size) / (1024.0 * > 1024.0 * 1024.0)) / avg_time; > > std::cout << "------------------------------------------------" << > std::endl; > std::cout << "Average Host-to-Host Bandwidth: " << bandwidth << " > GB/s" << std::endl; > std::cout << "------------------------------------------------" << > std::endl; > } > > // Clean up host memory > delete[] buffer; > > MPI_Finalize(); > return 0; > } > > > ---------------------------------------------------------------------------------------------------------------- > Here is the script to run the test with verbose compilation and the `shm` > layer forced and unforced: > > #!/usr/bin/zsh > source ~/.zshrc > > # Compile > /opt/mpich/4.2.1-cpu/bin/mpicxx -std=c++17 -I/opt/mpich/4.2.1-cpu/include > shmem_check.cpp -o shmem_check > > # Run with shm forced > /opt/mpich/4.2.1-cpu/bin/mpiexec -np 2 -genv FI_PROVIDER=shm -genv > FI_LOG_LEVEL=debug ./shmem_check 2> output_shm.txt > > # Run without shm forced > /opt/mpich/4.2.1-cpu/bin/mpiexec -np 2 -genv FI_LOG_LEVEL=debug > ./shmem_check 2> output_no_shm.txt > > echo "Output of script with SHM forced: " > grep -i "opened fabric" output_shm.txt > > echo "Output of script with SHM not forced: " > grep -i "opened fabric" output_no_shm.txt > > > ---------------------------------------------------------------------------------------------------------------- > Here is the output : > > --- Starting Host-to-Host MPI Bandwidth Test --- > Packet Size: 256 MB > ------------------------------------------------ > Average Host-to-Host Bandwidth: 3.35709 GB/s > ------------------------------------------------ > --- Starting Host-to-Host MPI Bandwidth Test --- > Packet Size: 256 MB > ------------------------------------------------ > Average Host-to-Host Bandwidth: 3.54924 GB/s > ------------------------------------------------ > Output of script with SHM forced: > libfabric:3174297:1755114546::core:core:fi_fabric_():1503<info> Opened > fabric: shm > libfabric:3174298:1755114546::core:core:fi_fabric_():1503<info> Opened > fabric: shm > Output of script with SHM not forced: > libfabric:3174351:1755114554::core:core:fi_fabric_():1503<info> Opened > fabric: 10.133.0.0/21 > <https://urldefense.us/v3/__http://10.133.0.0/21__;!!G_uCfscf7eWS!ZaUD7Nw-pS…> > libfabric:3174350:1755114554::core:core:fi_fabric_():1503<info> Opened > fabric: 10.133.0.0/21 > <https://urldefense.us/v3/__http://10.133.0.0/21__;!!G_uCfscf7eWS!ZaUD7Nw-pS…> >

1 0

Re: [mpich-discuss] MPICH: SHM bandwidth very low on IPC test
by Jenke, Joachim 13 Aug '25

13 Aug '25

Hi Sam, Can you try out stream to understand the single-core memory bandwidth of the system? https://urldefense.us/v3/__https://www.cs.virginia.edu/stream/ref.html__;!!… Copy bandwidth for large junks (exceeding cache sizes) should provide you an upper bound for shm communication bandwidth. Best Joachim Am 13.08.2025 22:04 schrieb Sam Austin via discuss <discuss(a)mpich.org>: Hi all, I am working to configure MPICH and run a few examples on my standalone server (single node). Here are the system specs: Server: Dell PowerEdge C4130 CPUs: 2x Xeon E5-2699A v4 GPUs: 4x Tesla V100s connected with NVLink, tied to motherboard ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Hi all, I am working to configure MPICH and run a few examples on my standalone server (single node). Here are the system specs: Server: Dell PowerEdge C4130 CPUs: 2x Xeon E5-2699A v4 GPUs: 4x Tesla V100s connected with NVLink, tied to motherboard with PCIe gen 3 OS: Ubuntu 24.04 LTS I intend to use this system to develop multi-process programs for eventual execution in a large, distributed HPC environment. I ran a few tests with and without CUDA support; here is my mpichversion output: MPICH Version: 4.3.1 MPICH Release date: Fri Jun 20 09:24:41 AM CDT 2025 MPICH ABI: 17:1:5 MPICH Device: ch4:ofi MPICH configure: --prefix=/opt/mpich/4.2.1-cpu --without-cuda MPICH CC: gcc -O2 MPICH CXX: g++ -O2 MPICH F77: gfortran -O2 MPICH FC: gfortran -O2 MPICH features: threadcomm The first example that I ran was a bandwidth test for CPU-CPU and GPU-GPU communication. This simple program sends small packets back and forth between processes to test the bandwidth over the various intra-node networks. The GPU-GPU bandwidth test showed that the GPU interconnect was saturating at ~45 GB/s, which is nominal for the NVLink interconnect topology present on the node (this was run with a CUDA-aware build of MPICH). The problem appears during the CPU-CPU IPC test. In theory, this test is pretty vanilla, as it is communicating between processes using shared memory, and does not involve traversing any of the intra-node networks (PCIe or NVLink). My understanding is that the bandwidth observed on the CPU-CPU IPC test should be quite high, at least higher than 10 GB/s. However, the intra-node IPC bandwidth appears to be very low, around 3.5 GB/s, when running this test. I tried the following fixes in an attempt to force MPICH to use shared memory, but to no avail: Passing the option to explicitly specify `nemesis` during the build configuration: "--with-device=ch3:nemesis --with-cuda" Passing the option to explicitly specify shared memory with ch4 to the configuration: "--with-ch4-shmmods=posix --with-cuda" Rebuilding MPICH without GPU support: "--without-cuda" Switching to Open MPI and running the same test These results, especially the last one in which I saw the same issues when running with Open MPI, makes me think it might be an issue with my system configuration. The question is: why is the IPC bandwidth so low despite supposedly using the SHM protocol? I'm wondering if anyone has encountered this issue before or might be able to lend some advice here. Any help would be greatly appreciated! Some interesting observations from the output below: when I run with "mpiexec -np 2 -genv FI_PROVIDER=shm ...", the log file reports "Opened fabric: shm". However, when I run without "-genv FI_PROVIDER=shm", the log file reports "Opened fabric: 10.133.0.0/21<https://urldefense.us/v3/__http://10.133.0.0/21__;!!G_uCfscf7eWS!ZaUD7Nw-pS…>", which I believe means that MPICH is falling back on the TCP socket protocol. In this case, my key point of confusion is that the observed bandwidth is essentially the same between the SHM and TCP protocols. Perhaps my test script isn't set up properly? Thanks, Sam The following is attached below: Bandwidth test program Run script for the program Output of the script on my machine ---------------------------------------------------------------------------------------------------------------- In case the attachment doesn't go through, here are the contents of my test program, "shmem_check.cpp": // shmem_check.cpp // // This is a minimal benchmark to test the raw bandwidth of MPI communication // between two processes on the same node, using only host (CPU) memory. // It completely removes CUDA to isolate the performance of the MPI library's // on-node communication mechanism (e.g., shared memory vs. TCP loopback). // // Compile/run: // /opt/mpich/4.2.1-cpu/bin/mpicxx -std=c++17 -I/opt/mpich/4.2.1-cpu/include shmem_check.cpp -o shmem_check // /opt/mpich/4.2.1-cpu/bin/mpiexec -np 2 ./shmem_check #include <iostream> #include <vector> #include <numeric> #include <mpi.h> int main(int argc, char* argv[]) { MPI_Init(&argc, &argv); int rank, size; MPI_Comm_rank(MPI_COMM_WORLD, &rank); MPI_Comm_size(MPI_COMM_WORLD, &size); if (size != 2) { if (rank == 0) { std::cerr << "Error: This program must be run with exactly 2 MPI processes." << std::endl; } MPI_Finalize(); return 1; } const int num_samples = 100; const long long packet_size = 1LL << 28; // 256 MB // Allocate standard host memory. 'new' is sufficient. char* buffer = new char[packet_size]; if (rank == 0) { std::cout << "--- Starting Host-to-Host MPI Bandwidth Test ---" << std::endl; std::cout << "Packet Size: " << (packet_size / (1024*1024)) << " MB" << std::endl; } std::vector<double> timings; for (int i = 0; i < num_samples; ++i) { MPI_Barrier(MPI_COMM_WORLD); double start_time = MPI_Wtime(); if (rank == 0) { MPI_Send(buffer, packet_size, MPI_CHAR, 1, 0, MPI_COMM_WORLD); MPI_Recv(buffer, 1, MPI_CHAR, 1, 1, MPI_COMM_WORLD, MPI_STATUS_IGNORE); // Wait for confirmation } else { // rank == 1 MPI_Recv(buffer, packet_size, MPI_CHAR, 0, 0, MPI_COMM_WORLD, MPI_STATUS_IGNORE); MPI_Send(buffer, 1, MPI_CHAR, 0, 1, MPI_COMM_WORLD); // Send confirmation } double end_time = MPI_Wtime(); if (i >= 10) { // Discard warmup runs timings.push_back(end_time - start_time); } } if (rank == 0) { double total_time = std::accumulate(timings.begin(), timings.end(), 0.0); double avg_time = total_time / timings.size(); double bandwidth = (static_cast<double>(packet_size) / (1024.0 * 1024.0 * 1024.0)) / avg_time; std::cout << "------------------------------------------------" << std::endl; std::cout << "Average Host-to-Host Bandwidth: " << bandwidth << " GB/s" << std::endl; std::cout << "------------------------------------------------" << std::endl; } // Clean up host memory delete[] buffer; MPI_Finalize(); return 0; } ---------------------------------------------------------------------------------------------------------------- Here is the script to run the test with verbose compilation and the `shm` layer forced and unforced: #!/usr/bin/zsh source ~/.zshrc # Compile /opt/mpich/4.2.1-cpu/bin/mpicxx -std=c++17 -I/opt/mpich/4.2.1-cpu/include shmem_check.cpp -o shmem_check # Run with shm forced /opt/mpich/4.2.1-cpu/bin/mpiexec -np 2 -genv FI_PROVIDER=shm -genv FI_LOG_LEVEL=debug ./shmem_check 2> output_shm.txt # Run without shm forced /opt/mpich/4.2.1-cpu/bin/mpiexec -np 2 -genv FI_LOG_LEVEL=debug ./shmem_check 2> output_no_shm.txt echo "Output of script with SHM forced: " grep -i "opened fabric" output_shm.txt echo "Output of script with SHM not forced: " grep -i "opened fabric" output_no_shm.txt ---------------------------------------------------------------------------------------------------------------- Here is the output : --- Starting Host-to-Host MPI Bandwidth Test --- Packet Size: 256 MB ------------------------------------------------ Average Host-to-Host Bandwidth: 3.35709 GB/s ------------------------------------------------ --- Starting Host-to-Host MPI Bandwidth Test --- Packet Size: 256 MB ------------------------------------------------ Average Host-to-Host Bandwidth: 3.54924 GB/s ------------------------------------------------ Output of script with SHM forced: libfabric:3174297:1755114546::core:core:fi_fabric_():1503<info> Opened fabric: shm libfabric:3174298:1755114546::core:core:fi_fabric_():1503<info> Opened fabric: shm Output of script with SHM not forced: libfabric:3174351:1755114554::core:core:fi_fabric_():1503<info> Opened fabric: 10.133.0.0/21<https://urldefense.us/v3/__http://10.133.0.0/21__;!!G_uCfscf7eWS!ZaUD7Nw-pS…> libfabric:3174350:1755114554::core:core:fi_fabric_():1503<info> Opened fabric: 10.133.0.0/21<https://urldefense.us/v3/__http://10.133.0.0/21__;!!G_uCfscf7eWS!ZaUD7Nw-pS…>

1 0

MPICH: SHM bandwidth very low on IPC test
by Sam Austin 13 Aug '25

13 Aug '25

Hi all, I am working to configure MPICH and run a few examples on my standalone server (single node). Here are the system specs: Server: Dell PowerEdge C4130 CPUs: 2x Xeon E5-2699A v4 GPUs: 4x Tesla V100s connected with NVLink, tied to motherboard with PCIe gen 3 OS: Ubuntu 24.04 LTS I intend to use this system to develop multi-process programs for eventual execution in a large, distributed HPC environment. I ran a few tests with and without CUDA support; here is my mpichversion output: MPICH Version: 4.3.1 MPICH Release date: Fri Jun 20 09:24:41 AM CDT 2025 MPICH ABI: 17:1:5 MPICH Device: ch4:ofi MPICH configure: --prefix=/opt/mpich/4.2.1-cpu --without-cuda MPICH CC: gcc -O2 MPICH CXX: g++ -O2 MPICH F77: gfortran -O2 MPICH FC: gfortran -O2 MPICH features: threadcomm The first example that I ran was a bandwidth test for CPU-CPU and GPU-GPU communication. This simple program sends small packets back and forth between processes to test the bandwidth over the various intra-node networks. The GPU-GPU bandwidth test showed that the GPU interconnect was saturating at ~45 GB/s, which is nominal for the NVLink interconnect topology present on the node (this was run with a CUDA-aware build of MPICH). The problem appears during the CPU-CPU IPC test. In theory, this test is pretty vanilla, as it is communicating between processes using shared memory, and does not involve traversing any of the intra-node networks (PCIe or NVLink). My understanding is that the bandwidth observed on the CPU-CPU IPC test should be quite high, at least higher than 10 GB/s. However, the intra-node IPC bandwidth appears to be very low, around 3.5 GB/s, when running this test. I tried the following fixes in an attempt to force MPICH to use shared memory, but to no avail: Passing the option to explicitly specify `nemesis` during the build configuration: "--with-device=ch3:nemesis --with-cuda" Passing the option to explicitly specify shared memory with ch4 to the configuration: "--with-ch4-shmmods=posix --with-cuda" Rebuilding MPICH without GPU support: "--without-cuda" Switching to Open MPI and running the same test These results, especially the last one in which I saw the same issues when running with Open MPI, makes me think it might be an issue with my system configuration. The question is: why is the IPC bandwidth so low despite supposedly using the SHM protocol? I'm wondering if anyone has encountered this issue before or might be able to lend some advice here. Any help would be greatly appreciated! Some interesting observations from the output below: when I run with "mpiexec -np 2 -genv FI_PROVIDER=shm ...", the log file reports "Opened fabric: shm". However, when I run without "-genv FI_PROVIDER=shm", the log file reports "Opened fabric: 10.133.0.0/21", which I believe means that MPICH is falling back on the TCP socket protocol. In this case, my key point of confusion is that the observed bandwidth is essentially the same between the SHM and TCP protocols. Perhaps my test script isn't set up properly? Thanks, Sam The following is attached below: Bandwidth test program Run script for the program Output of the script on my machine ---------------------------------------------------------------------------------------------------------------- In case the attachment doesn't go through, here are the contents of my test program, "shmem_check.cpp": // shmem_check.cpp // // This is a minimal benchmark to test the raw bandwidth of MPI communication // between two processes on the same node, using only host (CPU) memory. // It completely removes CUDA to isolate the performance of the MPI library's // on-node communication mechanism (e.g., shared memory vs. TCP loopback). // // Compile/run: // /opt/mpich/4.2.1-cpu/bin/mpicxx -std=c++17 -I/opt/mpich/4.2.1-cpu/include shmem_check.cpp -o shmem_check // /opt/mpich/4.2.1-cpu/bin/mpiexec -np 2 ./shmem_check #include <iostream> #include <vector> #include <numeric> #include <mpi.h> int main(int argc, char* argv[]) { MPI_Init(&argc, &argv); int rank, size; MPI_Comm_rank(MPI_COMM_WORLD, &rank); MPI_Comm_size(MPI_COMM_WORLD, &size); if (size != 2) { if (rank == 0) { std::cerr << "Error: This program must be run with exactly 2 MPI processes." << std::endl; } MPI_Finalize(); return 1; } const int num_samples = 100; const long long packet_size = 1LL << 28; // 256 MB // Allocate standard host memory. 'new' is sufficient. char* buffer = new char[packet_size]; if (rank == 0) { std::cout << "--- Starting Host-to-Host MPI Bandwidth Test ---" << std::endl; std::cout << "Packet Size: " << (packet_size / (1024*1024)) << " MB" << std::endl; } std::vector<double> timings; for (int i = 0; i < num_samples; ++i) { MPI_Barrier(MPI_COMM_WORLD); double start_time = MPI_Wtime(); if (rank == 0) { MPI_Send(buffer, packet_size, MPI_CHAR, 1, 0, MPI_COMM_WORLD); MPI_Recv(buffer, 1, MPI_CHAR, 1, 1, MPI_COMM_WORLD, MPI_STATUS_IGNORE); // Wait for confirmation } else { // rank == 1 MPI_Recv(buffer, packet_size, MPI_CHAR, 0, 0, MPI_COMM_WORLD, MPI_STATUS_IGNORE); MPI_Send(buffer, 1, MPI_CHAR, 0, 1, MPI_COMM_WORLD); // Send confirmation } double end_time = MPI_Wtime(); if (i >= 10) { // Discard warmup runs timings.push_back(end_time - start_time); } } if (rank == 0) { double total_time = std::accumulate(timings.begin(), timings.end(), 0.0); double avg_time = total_time / timings.size(); double bandwidth = (static_cast<double>(packet_size) / (1024.0 * 1024.0 * 1024.0)) / avg_time; std::cout << "------------------------------------------------" << std::endl; std::cout << "Average Host-to-Host Bandwidth: " << bandwidth << " GB/s" << std::endl; std::cout << "------------------------------------------------" << std::endl; } // Clean up host memory delete[] buffer; MPI_Finalize(); return 0; } ---------------------------------------------------------------------------------------------------------------- Here is the script to run the test with verbose compilation and the `shm` layer forced and unforced: #!/usr/bin/zsh source ~/.zshrc # Compile /opt/mpich/4.2.1-cpu/bin/mpicxx -std=c++17 -I/opt/mpich/4.2.1-cpu/include shmem_check.cpp -o shmem_check # Run with shm forced /opt/mpich/4.2.1-cpu/bin/mpiexec -np 2 -genv FI_PROVIDER=shm -genv FI_LOG_LEVEL=debug ./shmem_check 2> output_shm.txt # Run without shm forced /opt/mpich/4.2.1-cpu/bin/mpiexec -np 2 -genv FI_LOG_LEVEL=debug ./shmem_check 2> output_no_shm.txt echo "Output of script with SHM forced: " grep -i "opened fabric" output_shm.txt echo "Output of script with SHM not forced: " grep -i "opened fabric" output_no_shm.txt ---------------------------------------------------------------------------------------------------------------- Here is the output : --- Starting Host-to-Host MPI Bandwidth Test --- Packet Size: 256 MB ------------------------------------------------ Average Host-to-Host Bandwidth: 3.35709 GB/s ------------------------------------------------ --- Starting Host-to-Host MPI Bandwidth Test --- Packet Size: 256 MB ------------------------------------------------ Average Host-to-Host Bandwidth: 3.54924 GB/s ------------------------------------------------ Output of script with SHM forced: libfabric:3174297:1755114546::core:core:fi_fabric_():1503<info> Opened fabric: shm libfabric:3174298:1755114546::core:core:fi_fabric_():1503<info> Opened fabric: shm Output of script with SHM not forced: libfabric:3174351:1755114554::core:core:fi_fabric_():1503<info> Opened fabric: 10.133.0.0/21 libfabric:3174350:1755114554::core:core:fi_fabric_():1503<info> Opened fabric: 10.133.0.0/21

1 0

Re: [mpich-discuss] MPICH 5.0.1 performance on HPE SS11 plus more - a slurm problem
by Zhou, Hui 13 Aug '25

13 Aug '25

Hi Howard, Could you run with `MPIR_CVAR_DEBUG_SUMMARY=1`? It should print some debug messages. I want to confirm it is running the `cxi` provider. Hui ________________________________ From: Howard Pritchard <hppritcha(a)gmail.com> Sent: Wednesday, July 30, 2025 4:37 PM To: Thakur, Rajeev <thakur(a)anl.gov> Cc: discuss(a)mpich.org <discuss(a)mpich.org>; Zhou, Hui <zhouh(a)anl.gov> Subject: Re: [mpich-discuss] MPICH 5.0.1 performance on HPE SS11 plus more - a slurm problem You don't often get email from hppritcha(a)gmail.com. Learn why this is important<https://urldefense.us/v3/__https://aka.ms/LearnAboutSenderIdentification__;… > Hi Rajeev, Here are the results for 4. 3. x branch: hpp@ nid001293: /usr/projects/artab/users/hpp/osu-micro-benchmarks-5. 8-mpich/mpi/pt2pt>srun --mpi=pmix -n 2 ./osu_latency # OSU MPI Latency Test v5. 8 # Size Latency (us) 0 1. 92 1 1. 98 2 1. 98 ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Hi Rajeev, Here are the results for 4.3.x branch: hpp@nid001293:/usr/projects/artab/users/hpp/osu-micro-benchmarks-5.8-mpich/mpi/pt2pt>srun --mpi=pmix -n 2 ./osu_latency # OSU MPI Latency Test v5.8 # Size Latency (us) 0 1.92 1 1.98 2 1.98 4 1.98 8 1.98 16 1.98 32 1.99 64 1.99 128 2.47 256 2.59 512 2.65 1024 2.76 2048 2.95 4096 3.00 8192 5.96 16384 6.64 32768 7.44 65536 8.75 131072 11.52 262144 17.08 524288 27.96 1048576 49.38 2097152 92.96 4194304 179.74 These are more like i would expect for SS11/OFI CXI provider. Howard Am Mi., 30. Juli 2025 um 12:48 Uhr schrieb Thakur, Rajeev <thakur(a)anl.gov<mailto:[email protected]>>: Hi Howard, What was the latency with the 4.3.x branch? Rajeev From: Howard Pritchard via discuss <discuss(a)mpich.org<mailto:[email protected]>> Reply-To: "discuss(a)mpich.org<mailto:[email protected]>" <discuss(a)mpich.org<mailto:[email protected]>> Date: Wednesday, July 30, 2025 at 1:43 PM To: "Zhou, Hui" <zhouh(a)anl.gov<mailto:[email protected]>> Cc: Howard Pritchard <hppritcha(a)gmail.com<mailto:[email protected]>>, "discuss(a)mpich.org<mailto:[email protected]>" <discuss(a)mpich.org<mailto:[email protected]>> Subject: Re: [mpich-discuss] MPICH 5.0.1 performance on HPE SS11 plus more - a slurm problem Hi Hui That didn’t help. I am not surprised though as our cluster is an NVIDIA free zone. What did help is to switch to the mpich 4. 3. x branch and latency results are nominal and the slurm problem went away too. So we will stick with that branch. ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Hi Hui That didn’t help. I am not surprised though as our cluster is an NVIDIA free zone. What did help is to switch to the mpich 4.3.x branch and latency results are nominal and the slurm problem went away too. So we will stick with that branch. Howard On Mon, Jul 28, 2025 at 4:15 PM Zhou, Hui <zhouh(a)anl.gov<mailto:[email protected]>> wrote: Hi Howard, I wonder whether it is due to the overhead of querying pointer attributes. Could you try disable GPU support with `MPIR_CVAR_ENABLE_GPU=0` and see if the latency improves? Hui ________________________________ From: Howard Pritchard via discuss <discuss(a)mpich.org<mailto:[email protected]>> Sent: Monday, July 28, 2025 9:41 AM To: discuss(a)mpich.org<mailto:[email protected]> <discuss(a)mpich.org<mailto:[email protected]>> Cc: Howard Pritchard <hppritcha(a)gmail.com<mailto:[email protected]>> Subject: [mpich-discuss] MPICH 5.0.1 performance on HPE SS11 plus more - a slurm problem Hi Folks, We are seeing a strange performance issue on our HPE SS11 system when testing osu_latency inter-node with MPICH. First the info: system using libfabric 1. 22. 0 slurm - 24. 11. 5 Here's my mpichversion output: MPICH Version: 5. 0. 0a1 ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Hi Folks, We are seeing a strange performance issue on our HPE SS11 system when testing osu_latency inter-node with MPICH. First the info: system using libfabric 1.22.0 slurm - 24.11.5 Here's my mpichversion output: MPICH Version: 5.0.0a1 MPICH Release date: unreleased development copy MPICH ABI: 0:0:0 MPICH Device: ch4:ofi MPICH configure: --prefix=/XXXX/mpich_again/install --enable-g=no --enable-error-checking=no --with-device=ch4:ofi --enable-threads=multiple --with-ch4-shmmods=posix,xpmem --enable-thread-cs=per-vci --with-libfabric=/opt/cray/libfabric/1.22.0 --with-xpmem=/opt/cray/xpmem/default --with-pmix=/opt/pmix/gcc4x/5.0.8 --enable-fast=O3 MPICH CC: gcc -O3 MPICH CXX: g++ -O3 MPICH F77: gfortran -O3 MPICH FC: gfortran -O3 MPICH features: threadcomm And here's the OSU latency results: slurmstepd: error: mpi/pmix_v4: pmixp_coll_belong_chk: nid001439 [1]: pmixp_coll.c:280: No process controlled by this slurmstepd is involved in this collective. slurmstepd: error: mpi/pmix_v4: _process_server_request: nid001439 [1]: pmixp_server.c:923: Unable to pmixp_state_coll_get() slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_check: nid001438 [0]: pmixp_coll_ring.c:614: 0x15005c005dc0: unexpected contrib from nid001439:1, expected is 0 slurmstepd: error: mpi/pmix_v4: _process_server_request: nid001438 [0]: pmixp_server.c:937: 0x15005c005dc0: unexpected contrib from nid001439:1, coll->seq=0, seq=0 slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_reset_if_to: nid001438 [0]: pmixp_coll_ring.c:738: 0x1500580532f0: collective timeout seq=0 slurmstepd: error: mpi/pmix_v4: pmixp_coll_log: nid001438 [0]: pmixp_coll.c:286: Dumping collective state slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: pmixp_coll_ring.c:756: 0x1500580532f0: COLL_FENCE_RING state seq=0 slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: pmixp_coll_ring.c:758: my peerid: 0:nid001438 slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: pmixp_coll_ring.c:765: neighbor id: next 1:nid001439, prev 1:nid001439 slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: pmixp_coll_ring.c:775: Context ptr=0x150058053368, #0, in-use=0 slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: pmixp_coll_ring.c:775: Context ptr=0x1500580533a0, #1, in-use=0 slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: pmixp_coll_ring.c:775: Context ptr=0x1500580533d8, #2, in-use=1 slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: pmixp_coll_ring.c:786: seq=0 contribs: loc=1/prev=0/fwd=1 slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: pmixp_coll_ring.c:788: neighbor contribs [2]: slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: pmixp_coll_ring.c:821: done contrib: - slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: pmixp_coll_ring.c:823: wait contrib: nid001439 slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: pmixp_coll_ring.c:825: status=PMIXP_COLL_RING_PROGRESS slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: pmixp_coll_ring.c:829: buf (offset/size): 36/16384 slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_reset_if_to: nid001439 [1]: pmixp_coll_ring.c:738: 0x151d0c053400: collective timeout seq=0 slurmstepd: error: mpi/pmix_v4: pmixp_coll_log: nid001439 [1]: pmixp_coll.c:286: Dumping collective state slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: pmixp_coll_ring.c:756: 0x151d0c053400: COLL_FENCE_RING state seq=0 slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: pmixp_coll_ring.c:758: my peerid: 1:nid001439 slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: pmixp_coll_ring.c:765: neighbor id: next 0:nid001438, prev 0:nid001438 slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: pmixp_coll_ring.c:775: Context ptr=0x151d0c053478, #0, in-use=0 slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: pmixp_coll_ring.c:775: Context ptr=0x151d0c0534b0, #1, in-use=0 slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: pmixp_coll_ring.c:775: Context ptr=0x151d0c0534e8, #2, in-use=1 slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: pmixp_coll_ring.c:786: seq=0 contribs: loc=1/prev=0/fwd=1 slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: pmixp_coll_ring.c:788: neighbor contribs [2]: slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: pmixp_coll_ring.c:821: done contrib: - slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: pmixp_coll_ring.c:823: wait contrib: nid001438 slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: pmixp_coll_ring.c:825: status=PMIXP_COLL_RING_PROGRESS slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: pmixp_coll_ring.c:829: buf (offset/size): 36/16384 # OSU MPI Latency Test v5.8 # Size Latency (us) 0 1.66 1 9.29 2 9.57 4 9.69 8 9.76 16 9.77 32 9.76 64 9.77 128 10.32 256 7.54 512 7.45 1024 7.38 2048 7.37 4096 7.45 8192 9.21 16384 9.70 32768 10.63 65536 13.15 131072 16.96 262144 23.84 524288 36.16 1048576 60.36 2097152 108.43 4194304 228.31 Note the slurm behavior is - I launch the job. Go get coffee, do some duo-lingo, read some emails, then after about 10 minutes the osu latency runs. I did not get the slurm problems using an older mpich 4.3.1 but did get the same performance issue. 9 usecs doesn't seem right for an 8-byte pingpong over libfabric S11. I was expecting more like 1.6 or so. I am confident the slurm issue is unrelated to the latency issue. Thanks for any suggestions on how to address either issue however.

1 0

Re: [mpich-discuss] MPICH 5.0.1 performance on HPE SS11 plus more - a slurm problem
by Howard Pritchard 30 Jul '25

30 Jul '25

Hi Rajeev, Here are the results for 4.3.x branch: hpp@nid001293:/usr/projects/artab/users/hpp/osu-micro-benchmarks-5.8-mpich/mpi/pt2pt>srun --mpi=pmix -n 2 ./osu_latency # OSU MPI Latency Test v5.8 # Size Latency (us) 0 1.92 1 1.98 2 1.98 4 1.98 8 1.98 16 1.98 32 1.99 64 1.99 128 2.47 256 2.59 512 2.65 1024 2.76 2048 2.95 4096 3.00 8192 5.96 16384 6.64 32768 7.44 65536 8.75 131072 11.52 262144 17.08 524288 27.96 1048576 49.38 2097152 92.96 4194304 179.74 These are more like i would expect for SS11/OFI CXI provider. Howard Am Mi., 30. Juli 2025 um 12:48 Uhr schrieb Thakur, Rajeev <thakur(a)anl.gov>: > Hi Howard, > > What was the latency with the 4.3.x branch? > > > > Rajeev > > > > > > *From: *Howard Pritchard via discuss <discuss(a)mpich.org> > *Reply-To: *"discuss(a)mpich.org" <discuss(a)mpich.org> > *Date: *Wednesday, July 30, 2025 at 1:43 PM > *To: *"Zhou, Hui" <zhouh(a)anl.gov> > *Cc: *Howard Pritchard <hppritcha(a)gmail.com>, "discuss(a)mpich.org" < > discuss(a)mpich.org> > *Subject: *Re: [mpich-discuss] MPICH 5.0.1 performance on HPE SS11 plus > more - a slurm problem > > > > Hi Hui That didn’t help. I am not surprised though as our cluster is an > NVIDIA free zone. What did help is to switch to the mpich 4. 3. x branch > and latency results are nominal and the slurm problem went away too. So we > will stick with that branch. > > ZjQcmQRYFpfptBannerStart > > *This Message Is From an External Sender * > > This message came from outside your organization. > > ZjQcmQRYFpfptBannerEnd > > Hi Hui > > > > That didn’t help. I am not surprised though as our cluster is an NVIDIA > free zone. What did help is to switch to the mpich 4.3.x branch and > latency results are nominal and the slurm problem went away too. So we > will stick with that branch. > > > > Howard > > > > On Mon, Jul 28, 2025 at 4:15 PM Zhou, Hui <zhouh(a)anl.gov> wrote: > > Hi Howard, > > > > I wonder whether it is due to the overhead of querying pointer > attributes. Could you try disable GPU support with `MPIR_CVAR_ENABLE_GPU=0` > and see if the latency improves? > > > > Hui > ------------------------------ > > *From:* Howard Pritchard via discuss <discuss(a)mpich.org> > *Sent:* Monday, July 28, 2025 9:41 AM > *To:* discuss(a)mpich.org <discuss(a)mpich.org> > *Cc:* Howard Pritchard <hppritcha(a)gmail.com> > *Subject:* [mpich-discuss] MPICH 5.0.1 performance on HPE SS11 plus more > - a slurm problem > > > > Hi Folks, We are seeing a strange performance issue on our HPE SS11 system > when testing osu_latency inter-node with MPICH. First the info: system > using libfabric 1. 22. 0 slurm - 24. 11. 5 Here's my mpichversion output: > MPICH Version: 5. 0. 0a1 > > ZjQcmQRYFpfptBannerStart > > *This Message Is From an External Sender * > > This message came from outside your organization. > > > > ZjQcmQRYFpfptBannerEnd > > Hi Folks, > > > > We are seeing a strange performance issue on our HPE SS11 system when > testing osu_latency inter-node with MPICH. > > > > First the info: > > system using libfabric 1.22.0 > > slurm - 24.11.5 > > > > Here's my mpichversion output: > > > > MPICH Version: 5.0.0a1 > > MPICH Release date: unreleased development copy > > MPICH ABI: 0:0:0 > > MPICH Device: ch4:ofi > > MPICH configure: --prefix=/XXXX/mpich_again/install --enable-g=no > --enable-error-checking=no --with-device=ch4:ofi --enable-threads=multiple > --with-ch4-shmmods=posix,xpmem --enable-thread-cs=per-vci > --with-libfabric=/opt/cray/libfabric/1.22.0 > --with-xpmem=/opt/cray/xpmem/default --with-pmix=/opt/pmix/gcc4x/5.0.8 > --enable-fast=O3 > > MPICH CC: gcc -O3 > > MPICH CXX: g++ -O3 > > MPICH F77: gfortran -O3 > > MPICH FC: gfortran -O3 > > MPICH features: threadcomm > > > > And here's the OSU latency results: > > > slurmstepd: error: mpi/pmix_v4: pmixp_coll_belong_chk: nid001439 [1]: > pmixp_coll.c:280: No process controlled by this slurmstepd is involved in > this collective. > > slurmstepd: error: mpi/pmix_v4: _process_server_request: nid001439 [1]: > pmixp_server.c:923: Unable to pmixp_state_coll_get() > > slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_check: nid001438 [0]: > pmixp_coll_ring.c:614: 0x15005c005dc0: unexpected contrib from nid001439:1, > expected is 0 > > slurmstepd: error: mpi/pmix_v4: _process_server_request: nid001438 [0]: > pmixp_server.c:937: 0x15005c005dc0: unexpected contrib from nid001439:1, > coll->seq=0, seq=0 > > slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_reset_if_to: nid001438 > [0]: pmixp_coll_ring.c:738: 0x1500580532f0: collective timeout seq=0 > > slurmstepd: error: mpi/pmix_v4: pmixp_coll_log: nid001438 [0]: > pmixp_coll.c:286: Dumping collective state > > slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: > pmixp_coll_ring.c:756: 0x1500580532f0: COLL_FENCE_RING state seq=0 > > slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: > pmixp_coll_ring.c:758: my peerid: 0:nid001438 > > slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: > pmixp_coll_ring.c:765: neighbor id: next 1:nid001439, prev 1:nid001439 > > slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: > pmixp_coll_ring.c:775: Context ptr=0x150058053368, #0, in-use=0 > > slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: > pmixp_coll_ring.c:775: Context ptr=0x1500580533a0, #1, in-use=0 > > slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: > pmixp_coll_ring.c:775: Context ptr=0x1500580533d8, #2, in-use=1 > > slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: > pmixp_coll_ring.c:786: seq=0 contribs: loc=1/prev=0/fwd=1 > > slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: > pmixp_coll_ring.c:788: neighbor contribs [2]: > > slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: > pmixp_coll_ring.c:821: done contrib: - > > slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: > pmixp_coll_ring.c:823: wait contrib: nid001439 > > slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: > pmixp_coll_ring.c:825: status=PMIXP_COLL_RING_PROGRESS > > slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: > pmixp_coll_ring.c:829: buf (offset/size): 36/16384 > > slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_reset_if_to: nid001439 > [1]: pmixp_coll_ring.c:738: 0x151d0c053400: collective timeout seq=0 > > slurmstepd: error: mpi/pmix_v4: pmixp_coll_log: nid001439 [1]: > pmixp_coll.c:286: Dumping collective state > > slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: > pmixp_coll_ring.c:756: 0x151d0c053400: COLL_FENCE_RING state seq=0 > > slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: > pmixp_coll_ring.c:758: my peerid: 1:nid001439 > > slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: > pmixp_coll_ring.c:765: neighbor id: next 0:nid001438, prev 0:nid001438 > > slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: > pmixp_coll_ring.c:775: Context ptr=0x151d0c053478, #0, in-use=0 > > slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: > pmixp_coll_ring.c:775: Context ptr=0x151d0c0534b0, #1, in-use=0 > > slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: > pmixp_coll_ring.c:775: Context ptr=0x151d0c0534e8, #2, in-use=1 > > slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: > pmixp_coll_ring.c:786: seq=0 contribs: loc=1/prev=0/fwd=1 > > slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: > pmixp_coll_ring.c:788: neighbor contribs [2]: > > slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: > pmixp_coll_ring.c:821: done contrib: - > > slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: > pmixp_coll_ring.c:823: wait contrib: nid001438 > > slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: > pmixp_coll_ring.c:825: status=PMIXP_COLL_RING_PROGRESS > > slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: > pmixp_coll_ring.c:829: buf (offset/size): 36/16384 > > # OSU MPI Latency Test v5.8 > > # Size Latency (us) > > 0 1.66 > > 1 9.29 > > 2 9.57 > > 4 9.69 > > 8 9.76 > > 16 9.77 > > 32 9.76 > > 64 9.77 > > 128 10.32 > > 256 7.54 > > 512 7.45 > > 1024 7.38 > > 2048 7.37 > > 4096 7.45 > > 8192 9.21 > > 16384 9.70 > > 32768 10.63 > > 65536 13.15 > > 131072 16.96 > > 262144 23.84 > > 524288 36.16 > > 1048576 60.36 > > 2097152 108.43 > > 4194304 228.31 > > > > Note the slurm behavior is - I launch the job. Go get coffee, do some > duo-lingo, read some emails, then after about 10 minutes the osu latency > runs. > > > > I did not get the slurm problems using an older mpich 4.3.1 but did get > the same performance issue. 9 usecs doesn't seem right for an 8-byte > pingpong over libfabric S11. I was expecting more like 1.6 or so. > > > > I am confident the slurm issue is unrelated to the latency issue. > > Thanks for any suggestions on how to address either issue however. > > > > > >

1 0

Re: [mpich-discuss] MPICH 5.0.1 performance on HPE SS11 plus more - a slurm problem
by Thakur, Rajeev 30 Jul '25

30 Jul '25

Hi Howard, What was the latency with the 4.3.x branch? Rajeev From: Howard Pritchard via discuss <discuss(a)mpich.org> Reply-To: "discuss(a)mpich.org" <discuss(a)mpich.org> Date: Wednesday, July 30, 2025 at 1:43 PM To: "Zhou, Hui" <zhouh(a)anl.gov> Cc: Howard Pritchard <hppritcha(a)gmail.com>, "discuss(a)mpich.org" <discuss(a)mpich.org> Subject: Re: [mpich-discuss] MPICH 5.0.1 performance on HPE SS11 plus more - a slurm problem Hi Hui That didn’t help. I am not surprised though as our cluster is an NVIDIA free zone. What did help is to switch to the mpich 4. 3. x branch and latency results are nominal and the slurm problem went away too. So we will stick with that branch. ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Hi Hui That didn’t help. I am not surprised though as our cluster is an NVIDIA free zone. What did help is to switch to the mpich 4.3.x branch and latency results are nominal and the slurm problem went away too. So we will stick with that branch. Howard On Mon, Jul 28, 2025 at 4:15 PM Zhou, Hui <zhouh(a)anl.gov<mailto:[email protected]>> wrote: Hi Howard, I wonder whether it is due to the overhead of querying pointer attributes. Could you try disable GPU support with `MPIR_CVAR_ENABLE_GPU=0` and see if the latency improves? Hui ________________________________ From: Howard Pritchard via discuss <discuss(a)mpich.org<mailto:[email protected]>> Sent: Monday, July 28, 2025 9:41 AM To: discuss(a)mpich.org<mailto:[email protected]> <discuss(a)mpich.org<mailto:[email protected]>> Cc: Howard Pritchard <hppritcha(a)gmail.com<mailto:[email protected]>> Subject: [mpich-discuss] MPICH 5.0.1 performance on HPE SS11 plus more - a slurm problem Hi Folks, We are seeing a strange performance issue on our HPE SS11 system when testing osu_latency inter-node with MPICH. First the info: system using libfabric 1. 22. 0 slurm - 24. 11. 5 Here's my mpichversion output: MPICH Version: 5. 0. 0a1 ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Hi Folks, We are seeing a strange performance issue on our HPE SS11 system when testing osu_latency inter-node with MPICH. First the info: system using libfabric 1.22.0 slurm - 24.11.5 Here's my mpichversion output: MPICH Version: 5.0.0a1 MPICH Release date: unreleased development copy MPICH ABI: 0:0:0 MPICH Device: ch4:ofi MPICH configure: --prefix=/XXXX/mpich_again/install --enable-g=no --enable-error-checking=no --with-device=ch4:ofi --enable-threads=multiple --with-ch4-shmmods=posix,xpmem --enable-thread-cs=per-vci --with-libfabric=/opt/cray/libfabric/1.22.0 --with-xpmem=/opt/cray/xpmem/default --with-pmix=/opt/pmix/gcc4x/5.0.8 --enable-fast=O3 MPICH CC: gcc -O3 MPICH CXX: g++ -O3 MPICH F77: gfortran -O3 MPICH FC: gfortran -O3 MPICH features: threadcomm And here's the OSU latency results: slurmstepd: error: mpi/pmix_v4: pmixp_coll_belong_chk: nid001439 [1]: pmixp_coll.c:280: No process controlled by this slurmstepd is involved in this collective. slurmstepd: error: mpi/pmix_v4: _process_server_request: nid001439 [1]: pmixp_server.c:923: Unable to pmixp_state_coll_get() slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_check: nid001438 [0]: pmixp_coll_ring.c:614: 0x15005c005dc0: unexpected contrib from nid001439:1, expected is 0 slurmstepd: error: mpi/pmix_v4: _process_server_request: nid001438 [0]: pmixp_server.c:937: 0x15005c005dc0: unexpected contrib from nid001439:1, coll->seq=0, seq=0 slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_reset_if_to: nid001438 [0]: pmixp_coll_ring.c:738: 0x1500580532f0: collective timeout seq=0 slurmstepd: error: mpi/pmix_v4: pmixp_coll_log: nid001438 [0]: pmixp_coll.c:286: Dumping collective state slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: pmixp_coll_ring.c:756: 0x1500580532f0: COLL_FENCE_RING state seq=0 slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: pmixp_coll_ring.c:758: my peerid: 0:nid001438 slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: pmixp_coll_ring.c:765: neighbor id: next 1:nid001439, prev 1:nid001439 slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: pmixp_coll_ring.c:775: Context ptr=0x150058053368, #0, in-use=0 slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: pmixp_coll_ring.c:775: Context ptr=0x1500580533a0, #1, in-use=0 slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: pmixp_coll_ring.c:775: Context ptr=0x1500580533d8, #2, in-use=1 slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: pmixp_coll_ring.c:786: seq=0 contribs: loc=1/prev=0/fwd=1 slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: pmixp_coll_ring.c:788: neighbor contribs [2]: slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: pmixp_coll_ring.c:821: done contrib: - slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: pmixp_coll_ring.c:823: wait contrib: nid001439 slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: pmixp_coll_ring.c:825: status=PMIXP_COLL_RING_PROGRESS slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: pmixp_coll_ring.c:829: buf (offset/size): 36/16384 slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_reset_if_to: nid001439 [1]: pmixp_coll_ring.c:738: 0x151d0c053400: collective timeout seq=0 slurmstepd: error: mpi/pmix_v4: pmixp_coll_log: nid001439 [1]: pmixp_coll.c:286: Dumping collective state slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: pmixp_coll_ring.c:756: 0x151d0c053400: COLL_FENCE_RING state seq=0 slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: pmixp_coll_ring.c:758: my peerid: 1:nid001439 slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: pmixp_coll_ring.c:765: neighbor id: next 0:nid001438, prev 0:nid001438 slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: pmixp_coll_ring.c:775: Context ptr=0x151d0c053478, #0, in-use=0 slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: pmixp_coll_ring.c:775: Context ptr=0x151d0c0534b0, #1, in-use=0 slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: pmixp_coll_ring.c:775: Context ptr=0x151d0c0534e8, #2, in-use=1 slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: pmixp_coll_ring.c:786: seq=0 contribs: loc=1/prev=0/fwd=1 slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: pmixp_coll_ring.c:788: neighbor contribs [2]: slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: pmixp_coll_ring.c:821: done contrib: - slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: pmixp_coll_ring.c:823: wait contrib: nid001438 slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: pmixp_coll_ring.c:825: status=PMIXP_COLL_RING_PROGRESS slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: pmixp_coll_ring.c:829: buf (offset/size): 36/16384 # OSU MPI Latency Test v5.8 # Size Latency (us) 0 1.66 1 9.29 2 9.57 4 9.69 8 9.76 16 9.77 32 9.76 64 9.77 128 10.32 256 7.54 512 7.45 1024 7.38 2048 7.37 4096 7.45 8192 9.21 16384 9.70 32768 10.63 65536 13.15 131072 16.96 262144 23.84 524288 36.16 1048576 60.36 2097152 108.43 4194304 228.31 Note the slurm behavior is - I launch the job. Go get coffee, do some duo-lingo, read some emails, then after about 10 minutes the osu latency runs. I did not get the slurm problems using an older mpich 4.3.1 but did get the same performance issue. 9 usecs doesn't seem right for an 8-byte pingpong over libfabric S11. I was expecting more like 1.6 or so. I am confident the slurm issue is unrelated to the latency issue. Thanks for any suggestions on how to address either issue however.

1 0

Re: [mpich-discuss] MPICH 5.0.1 performance on HPE SS11 plus more - a slurm problem
by Howard Pritchard 30 Jul '25

30 Jul '25

Hi Hui That didn’t help. I am not surprised though as our cluster is an NVIDIA free zone. What did help is to switch to the mpich 4.3.x branch and latency results are nominal and the slurm problem went away too. So we will stick with that branch. Howard On Mon, Jul 28, 2025 at 4:15 PM Zhou, Hui <zhouh(a)anl.gov> wrote: > Hi Howard, > > I wonder whether it is due to the overhead of querying pointer > attributes. Could you try disable GPU support with `MPIR_CVAR_ENABLE_GPU=0` > and see if the latency improves? > > Hui > ------------------------------ > *From:* Howard Pritchard via discuss <discuss(a)mpich.org> > *Sent:* Monday, July 28, 2025 9:41 AM > *To:* discuss(a)mpich.org <discuss(a)mpich.org> > *Cc:* Howard Pritchard <hppritcha(a)gmail.com> > *Subject:* [mpich-discuss] MPICH 5.0.1 performance on HPE SS11 plus more > - a slurm problem > > Hi Folks, We are seeing a strange performance issue on our HPE SS11 system > when testing osu_latency inter-node with MPICH. First the info: system > using libfabric 1. 22. 0 slurm - 24. 11. 5 Here's my mpichversion output: > MPICH Version: 5. 0. 0a1 > ZjQcmQRYFpfptBannerStart > This Message Is From an External Sender > This message came from outside your organization. > > ZjQcmQRYFpfptBannerEnd > Hi Folks, > > We are seeing a strange performance issue on our HPE SS11 system when > testing osu_latency inter-node with MPICH. > > First the info: > system using libfabric 1.22.0 > slurm - 24.11.5 > > Here's my mpichversion output: > > MPICH Version: 5.0.0a1 > > MPICH Release date: unreleased development copy > > MPICH ABI: 0:0:0 > > MPICH Device: ch4:ofi > > MPICH configure: --prefix=/XXXX/mpich_again/install --enable-g=no > --enable-error-checking=no --with-device=ch4:ofi --enable-threads=multiple > --with-ch4-shmmods=posix,xpmem --enable-thread-cs=per-vci > --with-libfabric=/opt/cray/libfabric/1.22.0 > --with-xpmem=/opt/cray/xpmem/default --with-pmix=/opt/pmix/gcc4x/5.0.8 > --enable-fast=O3 > > MPICH CC: gcc -O3 > > MPICH CXX: g++ -O3 > > MPICH F77: gfortran -O3 > > MPICH FC: gfortran -O3 > > MPICH features: threadcomm > > > > And here's the OSU latency results: > > slurmstepd: error: mpi/pmix_v4: pmixp_coll_belong_chk: nid001439 [1]: > pmixp_coll.c:280: No process controlled by this slurmstepd is involved in > this collective. > > slurmstepd: error: mpi/pmix_v4: _process_server_request: nid001439 [1]: > pmixp_server.c:923: Unable to pmixp_state_coll_get() > > slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_check: nid001438 [0]: > pmixp_coll_ring.c:614: 0x15005c005dc0: unexpected contrib from nid001439:1, > expected is 0 > > slurmstepd: error: mpi/pmix_v4: _process_server_request: nid001438 [0]: > pmixp_server.c:937: 0x15005c005dc0: unexpected contrib from nid001439:1, > coll->seq=0, seq=0 > > slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_reset_if_to: nid001438 > [0]: pmixp_coll_ring.c:738: 0x1500580532f0: collective timeout seq=0 > > slurmstepd: error: mpi/pmix_v4: pmixp_coll_log: nid001438 [0]: > pmixp_coll.c:286: Dumping collective state > > slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: > pmixp_coll_ring.c:756: 0x1500580532f0: COLL_FENCE_RING state seq=0 > > slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: > pmixp_coll_ring.c:758: my peerid: 0:nid001438 > > slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: > pmixp_coll_ring.c:765: neighbor id: next 1:nid001439, prev 1:nid001439 > > slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: > pmixp_coll_ring.c:775: Context ptr=0x150058053368, #0, in-use=0 > > slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: > pmixp_coll_ring.c:775: Context ptr=0x1500580533a0, #1, in-use=0 > > slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: > pmixp_coll_ring.c:775: Context ptr=0x1500580533d8, #2, in-use=1 > > slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: > pmixp_coll_ring.c:786: seq=0 contribs: loc=1/prev=0/fwd=1 > > slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: > pmixp_coll_ring.c:788: neighbor contribs [2]: > > slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: > pmixp_coll_ring.c:821: done contrib: - > > slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: > pmixp_coll_ring.c:823: wait contrib: nid001439 > > slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: > pmixp_coll_ring.c:825: status=PMIXP_COLL_RING_PROGRESS > > slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: > pmixp_coll_ring.c:829: buf (offset/size): 36/16384 > > slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_reset_if_to: nid001439 > [1]: pmixp_coll_ring.c:738: 0x151d0c053400: collective timeout seq=0 > > slurmstepd: error: mpi/pmix_v4: pmixp_coll_log: nid001439 [1]: > pmixp_coll.c:286: Dumping collective state > > slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: > pmixp_coll_ring.c:756: 0x151d0c053400: COLL_FENCE_RING state seq=0 > > slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: > pmixp_coll_ring.c:758: my peerid: 1:nid001439 > > slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: > pmixp_coll_ring.c:765: neighbor id: next 0:nid001438, prev 0:nid001438 > > slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: > pmixp_coll_ring.c:775: Context ptr=0x151d0c053478, #0, in-use=0 > > slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: > pmixp_coll_ring.c:775: Context ptr=0x151d0c0534b0, #1, in-use=0 > > slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: > pmixp_coll_ring.c:775: Context ptr=0x151d0c0534e8, #2, in-use=1 > > slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: > pmixp_coll_ring.c:786: seq=0 contribs: loc=1/prev=0/fwd=1 > > slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: > pmixp_coll_ring.c:788: neighbor contribs [2]: > > slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: > pmixp_coll_ring.c:821: done contrib: - > > slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: > pmixp_coll_ring.c:823: wait contrib: nid001438 > > slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: > pmixp_coll_ring.c:825: status=PMIXP_COLL_RING_PROGRESS > > slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: > pmixp_coll_ring.c:829: buf (offset/size): 36/16384 > > # OSU MPI Latency Test v5.8 > > # Size Latency (us) > > 0 1.66 > > 1 9.29 > > 2 9.57 > > 4 9.69 > > 8 9.76 > > 16 9.77 > > 32 9.76 > > 64 9.77 > > 128 10.32 > > 256 7.54 > > 512 7.45 > > 1024 7.38 > > 2048 7.37 > > 4096 7.45 > > 8192 9.21 > > 16384 9.70 > > 32768 10.63 > > 65536 13.15 > > 131072 16.96 > > 262144 23.84 > > 524288 36.16 > > 1048576 60.36 > > 2097152 108.43 > > 4194304 228.31 > > > Note the slurm behavior is - I launch the job. Go get coffee, do some > duo-lingo, read some emails, then after about 10 minutes the osu latency > runs. > > > I did not get the slurm problems using an older mpich 4.3.1 but did get > the same performance issue. 9 usecs doesn't seem right for an 8-byte > pingpong over libfabric S11. I was expecting more like 1.6 or so. > > > I am confident the slurm issue is unrelated to the latency issue. > > Thanks for any suggestions on how to address either issue however. > > > >

1 0

Re: [mpich-discuss] MPICH 5.0.1 performance on HPE SS11 plus more - a slurm problem
by Zhou, Hui 28 Jul '25

28 Jul '25

Hi Howard, I wonder whether it is due to the overhead of querying pointer attributes. Could you try disable GPU support with `MPIR_CVAR_ENABLE_GPU=0` and see if the latency improves? Hui ________________________________ From: Howard Pritchard via discuss <discuss(a)mpich.org> Sent: Monday, July 28, 2025 9:41 AM To: discuss(a)mpich.org <discuss(a)mpich.org> Cc: Howard Pritchard <hppritcha(a)gmail.com> Subject: [mpich-discuss] MPICH 5.0.1 performance on HPE SS11 plus more - a slurm problem Hi Folks, We are seeing a strange performance issue on our HPE SS11 system when testing osu_latency inter-node with MPICH. First the info: system using libfabric 1. 22. 0 slurm - 24. 11. 5 Here's my mpichversion output: MPICH Version: 5. 0. 0a1 ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Hi Folks, We are seeing a strange performance issue on our HPE SS11 system when testing osu_latency inter-node with MPICH. First the info: system using libfabric 1.22.0 slurm - 24.11.5 Here's my mpichversion output: MPICH Version: 5.0.0a1 MPICH Release date: unreleased development copy MPICH ABI: 0:0:0 MPICH Device: ch4:ofi MPICH configure: --prefix=/XXXX/mpich_again/install --enable-g=no --enable-error-checking=no --with-device=ch4:ofi --enable-threads=multiple --with-ch4-shmmods=posix,xpmem --enable-thread-cs=per-vci --with-libfabric=/opt/cray/libfabric/1.22.0 --with-xpmem=/opt/cray/xpmem/default --with-pmix=/opt/pmix/gcc4x/5.0.8 --enable-fast=O3 MPICH CC: gcc -O3 MPICH CXX: g++ -O3 MPICH F77: gfortran -O3 MPICH FC: gfortran -O3 MPICH features: threadcomm And here's the OSU latency results: slurmstepd: error: mpi/pmix_v4: pmixp_coll_belong_chk: nid001439 [1]: pmixp_coll.c:280: No process controlled by this slurmstepd is involved in this collective. slurmstepd: error: mpi/pmix_v4: _process_server_request: nid001439 [1]: pmixp_server.c:923: Unable to pmixp_state_coll_get() slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_check: nid001438 [0]: pmixp_coll_ring.c:614: 0x15005c005dc0: unexpected contrib from nid001439:1, expected is 0 slurmstepd: error: mpi/pmix_v4: _process_server_request: nid001438 [0]: pmixp_server.c:937: 0x15005c005dc0: unexpected contrib from nid001439:1, coll->seq=0, seq=0 slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_reset_if_to: nid001438 [0]: pmixp_coll_ring.c:738: 0x1500580532f0: collective timeout seq=0 slurmstepd: error: mpi/pmix_v4: pmixp_coll_log: nid001438 [0]: pmixp_coll.c:286: Dumping collective state slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: pmixp_coll_ring.c:756: 0x1500580532f0: COLL_FENCE_RING state seq=0 slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: pmixp_coll_ring.c:758: my peerid: 0:nid001438 slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: pmixp_coll_ring.c:765: neighbor id: next 1:nid001439, prev 1:nid001439 slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: pmixp_coll_ring.c:775: Context ptr=0x150058053368, #0, in-use=0 slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: pmixp_coll_ring.c:775: Context ptr=0x1500580533a0, #1, in-use=0 slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: pmixp_coll_ring.c:775: Context ptr=0x1500580533d8, #2, in-use=1 slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: pmixp_coll_ring.c:786: seq=0 contribs: loc=1/prev=0/fwd=1 slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: pmixp_coll_ring.c:788: neighbor contribs [2]: slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: pmixp_coll_ring.c:821: done contrib: - slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: pmixp_coll_ring.c:823: wait contrib: nid001439 slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: pmixp_coll_ring.c:825: status=PMIXP_COLL_RING_PROGRESS slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: pmixp_coll_ring.c:829: buf (offset/size): 36/16384 slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_reset_if_to: nid001439 [1]: pmixp_coll_ring.c:738: 0x151d0c053400: collective timeout seq=0 slurmstepd: error: mpi/pmix_v4: pmixp_coll_log: nid001439 [1]: pmixp_coll.c:286: Dumping collective state slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: pmixp_coll_ring.c:756: 0x151d0c053400: COLL_FENCE_RING state seq=0 slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: pmixp_coll_ring.c:758: my peerid: 1:nid001439 slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: pmixp_coll_ring.c:765: neighbor id: next 0:nid001438, prev 0:nid001438 slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: pmixp_coll_ring.c:775: Context ptr=0x151d0c053478, #0, in-use=0 slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: pmixp_coll_ring.c:775: Context ptr=0x151d0c0534b0, #1, in-use=0 slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: pmixp_coll_ring.c:775: Context ptr=0x151d0c0534e8, #2, in-use=1 slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: pmixp_coll_ring.c:786: seq=0 contribs: loc=1/prev=0/fwd=1 slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: pmixp_coll_ring.c:788: neighbor contribs [2]: slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: pmixp_coll_ring.c:821: done contrib: - slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: pmixp_coll_ring.c:823: wait contrib: nid001438 slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: pmixp_coll_ring.c:825: status=PMIXP_COLL_RING_PROGRESS slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: pmixp_coll_ring.c:829: buf (offset/size): 36/16384 # OSU MPI Latency Test v5.8 # Size Latency (us) 0 1.66 1 9.29 2 9.57 4 9.69 8 9.76 16 9.77 32 9.76 64 9.77 128 10.32 256 7.54 512 7.45 1024 7.38 2048 7.37 4096 7.45 8192 9.21 16384 9.70 32768 10.63 65536 13.15 131072 16.96 262144 23.84 524288 36.16 1048576 60.36 2097152 108.43 4194304 228.31 Note the slurm behavior is - I launch the job. Go get coffee, do some duo-lingo, read some emails, then after about 10 minutes the osu latency runs. I did not get the slurm problems using an older mpich 4.3.1 but did get the same performance issue. 9 usecs doesn't seem right for an 8-byte pingpong over libfabric S11. I was expecting more like 1.6 or so. I am confident the slurm issue is unrelated to the latency issue. Thanks for any suggestions on how to address either issue however.

1 0

MPICH 5.0.1 performance on HPE SS11 plus more - a slurm problem
by Howard Pritchard 28 Jul '25

28 Jul '25

Hi Folks, We are seeing a strange performance issue on our HPE SS11 system when testing osu_latency inter-node with MPICH. First the info: system using libfabric 1.22.0 slurm - 24.11.5 Here's my mpichversion output: MPICH Version: 5.0.0a1 MPICH Release date: unreleased development copy MPICH ABI: 0:0:0 MPICH Device: ch4:ofi MPICH configure: --prefix=/XXXX/mpich_again/install --enable-g=no --enable-error-checking=no --with-device=ch4:ofi --enable-threads=multiple --with-ch4-shmmods=posix,xpmem --enable-thread-cs=per-vci --with-libfabric=/opt/cray/libfabric/1.22.0 --with-xpmem=/opt/cray/xpmem/default --with-pmix=/opt/pmix/gcc4x/5.0.8 --enable-fast=O3 MPICH CC: gcc -O3 MPICH CXX: g++ -O3 MPICH F77: gfortran -O3 MPICH FC: gfortran -O3 MPICH features: threadcomm And here's the OSU latency results: slurmstepd: error: mpi/pmix_v4: pmixp_coll_belong_chk: nid001439 [1]: pmixp_coll.c:280: No process controlled by this slurmstepd is involved in this collective. slurmstepd: error: mpi/pmix_v4: _process_server_request: nid001439 [1]: pmixp_server.c:923: Unable to pmixp_state_coll_get() slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_check: nid001438 [0]: pmixp_coll_ring.c:614: 0x15005c005dc0: unexpected contrib from nid001439:1, expected is 0 slurmstepd: error: mpi/pmix_v4: _process_server_request: nid001438 [0]: pmixp_server.c:937: 0x15005c005dc0: unexpected contrib from nid001439:1, coll->seq=0, seq=0 slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_reset_if_to: nid001438 [0]: pmixp_coll_ring.c:738: 0x1500580532f0: collective timeout seq=0 slurmstepd: error: mpi/pmix_v4: pmixp_coll_log: nid001438 [0]: pmixp_coll.c:286: Dumping collective state slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: pmixp_coll_ring.c:756: 0x1500580532f0: COLL_FENCE_RING state seq=0 slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: pmixp_coll_ring.c:758: my peerid: 0:nid001438 slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: pmixp_coll_ring.c:765: neighbor id: next 1:nid001439, prev 1:nid001439 slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: pmixp_coll_ring.c:775: Context ptr=0x150058053368, #0, in-use=0 slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: pmixp_coll_ring.c:775: Context ptr=0x1500580533a0, #1, in-use=0 slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: pmixp_coll_ring.c:775: Context ptr=0x1500580533d8, #2, in-use=1 slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: pmixp_coll_ring.c:786: seq=0 contribs: loc=1/prev=0/fwd=1 slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: pmixp_coll_ring.c:788: neighbor contribs [2]: slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: pmixp_coll_ring.c:821: done contrib: - slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: pmixp_coll_ring.c:823: wait contrib: nid001439 slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: pmixp_coll_ring.c:825: status=PMIXP_COLL_RING_PROGRESS slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: pmixp_coll_ring.c:829: buf (offset/size): 36/16384 slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_reset_if_to: nid001439 [1]: pmixp_coll_ring.c:738: 0x151d0c053400: collective timeout seq=0 slurmstepd: error: mpi/pmix_v4: pmixp_coll_log: nid001439 [1]: pmixp_coll.c:286: Dumping collective state slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: pmixp_coll_ring.c:756: 0x151d0c053400: COLL_FENCE_RING state seq=0 slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: pmixp_coll_ring.c:758: my peerid: 1:nid001439 slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: pmixp_coll_ring.c:765: neighbor id: next 0:nid001438, prev 0:nid001438 slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: pmixp_coll_ring.c:775: Context ptr=0x151d0c053478, #0, in-use=0 slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: pmixp_coll_ring.c:775: Context ptr=0x151d0c0534b0, #1, in-use=0 slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: pmixp_coll_ring.c:775: Context ptr=0x151d0c0534e8, #2, in-use=1 slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: pmixp_coll_ring.c:786: seq=0 contribs: loc=1/prev=0/fwd=1 slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: pmixp_coll_ring.c:788: neighbor contribs [2]: slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: pmixp_coll_ring.c:821: done contrib: - slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: pmixp_coll_ring.c:823: wait contrib: nid001438 slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: pmixp_coll_ring.c:825: status=PMIXP_COLL_RING_PROGRESS slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: pmixp_coll_ring.c:829: buf (offset/size): 36/16384 # OSU MPI Latency Test v5.8 # Size Latency (us) 0 1.66 1 9.29 2 9.57 4 9.69 8 9.76 16 9.77 32 9.76 64 9.77 128 10.32 256 7.54 512 7.45 1024 7.38 2048 7.37 4096 7.45 8192 9.21 16384 9.70 32768 10.63 65536 13.15 131072 16.96 262144 23.84 524288 36.16 1048576 60.36 2097152 108.43 4194304 228.31 Note the slurm behavior is - I launch the job. Go get coffee, do some duo-lingo, read some emails, then after about 10 minutes the osu latency runs. I did not get the slurm problems using an older mpich 4.3.1 but did get the same performance issue. 9 usecs doesn't seem right for an 8-byte pingpong over libfabric S11. I was expecting more like 1.6 or so. I am confident the slurm issue is unrelated to the latency issue. Thanks for any suggestions on how to address either issue however.

1 0