discuss
Threads by month
- ----- 2026 -----
- July
- June
- May
- April
- March
- February
- January
- ----- 2025 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2024 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2023 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2022 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2021 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2020 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2019 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2018 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2017 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2016 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2015 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2014 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2013 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2012 -----
- December
- November
June 2024
- 7 participants
- 48 discussions
Hi Victor,
https://urldefense.us/v3/__https://github.com/pmodels/mpich/pull/7033__;!!G… should address this issue. It contains patches that were submitted/accepted in upstream libfabric. I have already backported to the 4.2.x branch, and a soon to be released 4.2.2 will have the fix.
Thanks,
Ken
From: Victor Eijkhout via discuss <discuss(a)mpich.org>
Reply-To: "discuss(a)mpich.org" <discuss(a)mpich.org>
Date: Friday, June 28, 2024 at 2:59 PM
To: "discuss(a)mpich.org" <discuss(a)mpich.org>
Cc: Victor Eijkhout <eijkhout(a)tacc.utexas.edu>
Subject: [mpich-discuss] Failure to compile with gcc/14
Gcc/14 installed on Mac OS 14. 5 through brew, compiling the latest MPICH 4. 2. 1 release. src/hmem. c: 216: 45: error: passing argument 3 of 'ofi_iov_bytes_to_copy' from incompatible pointer type [-Wincompatible-pointer-types] 216 | &hmem_iov_offset,
ZjQcmQRYFpfptBannerStart
This Message Is From an External Sender
This message came from outside your organization.
ZjQcmQRYFpfptBannerEnd
Gcc/14 installed on Mac OS 14.5 through brew, compiling the latest MPICH 4.2.1 release.
src/hmem.c:216:45: error: passing argument 3 of 'ofi_iov_bytes_to_copy' from incompatible pointer type [-Wincompatible-pointer-types]
216 | &hmem_iov_offset, &hmem_buf);
| ^~~~~~~~~~~~~~~~
| |
| uint64_t * {aka long long unsigned int *}
In file included from src/hmem.c:41:
./include/ofi_iov.h:87:47: note: expected 'size_t *' {aka 'long unsigned int *'} but argument is of type 'uint64_t *' {aka 'long long unsigned int *'}
87 | size_t *size, size_t *offset, char **copy_buf)
| ~~~~~~~~^~~~~~
src/hmem.c: In function 'ofi_copy_hmem_iov_buf':
src/hmem.c:252:45: error: passing argument 3 of 'ofi_iov_bytes_to_copy' from incompatible pointer type [-Wincompatible-pointer-types]
252 | &hmem_iov_offset, &hmem_buf);
| ^~~~~~~~~~~~~~~~
| |
| uint64_t * {aka long long unsigned int *}
./include/ofi_iov.h:87:47: note: expected 'size_t *' {aka 'long unsigned int *'} but argument is of type 'uint64_t *' {aka 'long long unsigned int *'}
87 | size_t *size, size_t *offset, char **copy_buf)
| ~~~~~~~~^~~~~~
src/hmem.c: In function 'ofi_copy_mr_iov':
src/hmem.c:282:61: error: passing argument 3 of 'ofi_iov_bytes_to_copy' from incompatible pointer type [-Wincompatible-pointer-types]
282 | len = ofi_iov_bytes_to_copy(&iov[i], &size, &offset, &hmem_buf);
| ^~~~~~~
| |
| uint64_t * {aka long long unsigned int *}
./include/ofi_iov.h:87:47: note: expected 'size_t *' {aka 'long unsigned int *'} but argument is of type 'uint64_t *' {aka 'long long unsigned int *'}
87 | size_t *size, size_t *offset, char **copy_buf)
| ~~~~~~~~^~~~~~
_______________________________________________
discuss mailing list discuss(a)mpich.org
To manage subscription options or unsubscribe:
https://urldefense.us/v3/__https://lists.mpich.org/mailman/listinfo/discuss…
1
0
1
0
Apparently I was having some issues with respect to the libraries that were getting configured when I built MPICH versus the libraries that were available on the compute nodes. I’ve got that sorted out.
I configure with --with-device=ch4:ofi and got rid of the --without-ucx flag. I’m still getting a crash on MPI_Wait. If I don’t set FI_PROVIDER, then I get the error
Bad : modifier in $ (-).
Abort(944950159) on node 1: Fatal error in internal_Wait: Other MPI error, error stack:
internal_Wait(68205)..........: MPI_Wait(request=0x5e88690, status=0x7ffc63581a40) failed
MPIR_Wait(780)................:
MPIR_Wait_state(737)..........:
MPIDI_progress_test(134)......:
MPIDI_OFI_handle_cq_error(791): OFI poll failed (ofi_events.c:793:MPIDI_OFI_handle_cq_error:Input/output error)
If I set FI_PROVIDER=verbs, I get
Abort(673331087) on node 5 (rank 5 in comm 496): Fatal error in internal_Recv: Other MPI error, error stack:
internal_Recv(61700)..........: MPI_Recv(buf=0x2b4e0fa00000, count=4194304, MPI_CHAR, 0, 27624, comm=0x84000001, status=0x7ffdb2e9bdd0) failed
MPIR_Wait(780)................:
MPIR_Wait_state(737)..........:
MPIDI_progress_test(134)......:
MPIDI_OFI_handle_cq_error(791): OFI poll failed (ofi_events.c:793:MPIDI_OFI_handle_cq_error:Bad address)
This error appears on ranks 2 and 5 of a 6 processor job on 2 nodes, but the remaining nodes still appear to be failing on an MPI_Wait call.
From: Zhou, Hui <zhouh(a)anl.gov>
Date: Friday, June 21, 2024 at 11:03 AM
To: Palmer, Bruce J <Bruce.Palmer(a)pnnl.gov>, discuss(a)mpich.org <discuss(a)mpich.org>
Subject: Re: Fail on MPI_Wait
>../configure --prefix=/people/d3g293/mpich/mpich-4.2.1/build_decptn/install \
--with-device=ch4:ofi:sockets --with-libfabric=embedded \
--without-ucx CC=gcc CXX=g++
You are statically using "sockets" provider. Try --with-device=ch4:ofi
Hui
________________________________
From: Palmer, Bruce J <Bruce.Palmer(a)pnnl.gov>
Sent: Friday, June 21, 2024 12:27 PM
To: Zhou, Hui <zhouh(a)anl.gov>; discuss(a)mpich.org <discuss(a)mpich.org>
Subject: RE: Fail on MPI_Wait
Hui, When I set FI_PROVIDER=tcp, the code crashes in MPI_Init. Specifically, this code will fail on one process: #include "mpi. h" int main(int argc, char **argv) { MPI_Init(&argc, &argv); MPI_Finalize(); } I’m running on a
ZjQcmQRYFpfptBannerStart
This Message Is From an External Sender
This message came from outside your organization.
ZjQcmQRYFpfptBannerEnd
Hui,
When I set FI_PROVIDER=tcp, the code crashes in MPI_Init. Specifically, this code will fail on one process:
#include "mpi.h"
int main(int argc, char **argv) {
MPI_Init(&argc, &argv);
MPI_Finalize();
}
I’m running on a system with the following modules
[d3g293@deception02 testing]$ module list
Currently Loaded Modulefiles:
1) gcc/11.2.0 3) python/3.7.0 5) mkl/2019u4
2) cmake/3.21.4 4) git/2.42.0(default) 6) cuda/11.8
and a home-built version of mpich-4.2.1 configured with
../configure --prefix=/people/d3g293/mpich/mpich-4.2.1/build_decptn/install \
--with-device=ch4:ofi:sockets --with-libfabric=embedded \
--without-ucx CC=gcc CXX=g++
I thought it might have something to do with using a build configuration in my application build that is set up to include Cuda, but it also fails in MPI_Init with a non-Cuda configuraton if I set the FI_PROVIDER variable.
Bruce
From: Zhou, Hui <zhouh(a)anl.gov>
Sent: Friday, June 14, 2024 9:27 AM
To: Palmer, Bruce J <Bruce.Palmer(a)pnnl.gov>; discuss(a)mpich.org
Subject: Re: Fail on MPI_Wait
Never mind. It is v4.2.1.
________________________________
From: Zhou, Hui <zhouh(a)anl.gov<mailto:[email protected]>>
Sent: Friday, June 14, 2024 11:26 AM
To: Palmer, Bruce J <Bruce.Palmer(a)pnnl.gov<mailto:[email protected]>>; discuss(a)mpich.org<mailto:[email protected]> <discuss(a)mpich.org<mailto:[email protected]>>
Subject: Re: Fail on MPI_Wait
Bruce,
What is the mpich version, BTW?
--
Hui
________________________________
From: Zhou, Hui <zhouh(a)anl.gov<mailto:[email protected]>>
Sent: Friday, June 14, 2024 10:55 AM
To: Palmer, Bruce J <Bruce.Palmer(a)pnnl.gov<mailto:[email protected]>>; discuss(a)mpich.org<mailto:[email protected]> <discuss(a)mpich.org<mailto:[email protected]>>
Subject: Re: Fail on MPI_Wait
Bruce,
You are using the sockets provider. Could try set FI_PROVIDER=tcp to see if it makes a difference?
Meanwhile, if you can get a small reproducer – with the sockets provider or any provider, I'll try to debug it. It is difficult to guess the true source of the issue without a reproducer.
--
Hui
________________________________
From: Palmer, Bruce J <Bruce.Palmer(a)pnnl.gov<mailto:[email protected]>>
Sent: Friday, June 14, 2024 10:47 AM
To: Zhou, Hui <zhouh(a)anl.gov<mailto:[email protected]>>; discuss(a)mpich.org<mailto:[email protected]> <discuss(a)mpich.org<mailto:[email protected]>>
Subject: Re: Fail on MPI_Wait
The output to standard out from running on 2 nodes and one process per node is attached. From: Zhou, Hui <zhouh@ anl. gov> Date: Tuesday, June 11, 2024 at 5: 49 PM To: discuss@ mpich. org <discuss@ mpich. org> Cc: Palmer, Bruce J <Bruce. Palmer@ pnnl. gov>
ZjQcmQRYFpfptBannerStart
This Message Is From an External Sender
This message came from outside your organization.
ZjQcmQRYFpfptBannerEnd
The output to standard out from running on 2 nodes and one process per node is attached.
From: Zhou, Hui <zhouh(a)anl.gov<mailto:[email protected]>>
Date: Tuesday, June 11, 2024 at 5:49 PM
To: discuss(a)mpich.org<mailto:[email protected]> <discuss(a)mpich.org<mailto:[email protected]>>
Cc: Palmer, Bruce J <Bruce.Palmer(a)pnnl.gov<mailto:[email protected]>>
Subject: Re: Fail on MPI_Wait
>MPIDI_OFI_handle_cq_error(793): OFI poll failed (ofi_events.c:793:MPIDI_OFI_handle_cq_error:Input/output error)
This is an error coming from the libfabric provider. First we need find out which provider are you using. Try set environment variable MPIR_CVAR_DEBUG_SUMMARY=1 and run a simple MPI_INIT+MPI_Finalize test code. Could post its console output?
--
Hui
________________________________
From: Palmer, Bruce J via discuss <discuss(a)mpich.org<mailto:[email protected]>>
Sent: Tuesday, June 11, 2024 3:17 PM
To: discuss(a)mpich.org<mailto:[email protected]> <discuss(a)mpich.org<mailto:[email protected]>>
Cc: Palmer, Bruce J <Bruce.Palmer(a)pnnl.gov<mailto:[email protected]>>
Subject: [mpich-discuss] Fail on MPI_Wait
Hi, I’m trying to debug a GPU-aware runtime for the Global Arrays library. We had a version of this working a while ago, but it has mysteriously started failing and we are trying to track down why. Currently, we are getting failures in MPI_Wait
ZjQcmQRYFpfptBannerStart
This Message Is From an External Sender
This message came from outside your organization.
ZjQcmQRYFpfptBannerEnd
Hi,
I’m trying to debug a GPU-aware runtime for the Global Arrays library. We had a version of this working a while ago, but it has mysteriously started failing and we are trying to track down why. Currently, we are getting failures in MPI_Wait and were wondering if anyone could provide some information on what exactly seems to be failing inside the wait call. The error we are getting is
Abort(206752655) on node 0: Fatal error in internal_Wait: Other MPI error, error stack:
internal_Wait(68205)..........: MPI_Wait(request=0x500847a0, status=0x7ffff9331800) failed
MPIR_Wait(780)................:
MPIR_Wait_state(737)..........:
MPIDI_progress_test(134)......:
MPIDI_OFI_handle_cq_error(793): OFI poll failed (ofi_events.c:793:MPIDI_OFI_handle_cq_error:Input/output error)
I’ve verified that the handle corresponding to 0x500847a0 is getting set earlier in the code in an MPI_Isend call and that no MPI_Wait or MPI_Test is called on the handle before it crashes with the above error message. I’m using MPICH 4.2.1 using gcc/8.3.0. The MPICH library was configured with
../configure --prefix=/people/d3g293/mpich/mpich-4.2.1/build_newell/install \
--with-device=ch4:ofi:sockets --with-libfabric=embedded \
--without-ucx --enable-threads=multiple --with-slurm \
CC=gcc CXX=g+
I’ve tried building with UCX and gotten the same results.
Are these errors indicative of corruption of the request handle or problems with some internal MPI data structures or something else? Any information you can provide would be appreciated.
Thanks,
Bruce
1
0
Hello,
Lately, I have encountered an issue with the usage of MPI_Comm_spawn +
MPI_Intercomm_merge, where in some occasions my application hangs
indefinitely. I was hoping that this forum could advise me on a solution.
For my tests, I run the code with 2 processes and give as an argument the
amount of processes to spawn in MPI_Comm_spawn, where I choose 20. In my
experimentation, 3 to 4 out of 100 executions hang indefinitely at the
function MPI_Barrier. Nevertheless, if I remove both MPI_Intercomm_merge
functions, the code always finalizes.
Also, prior to executing the code I set the environment variable
FI_PROVIDER=verbs.
However, if I change the provider to "tcp", or "udp", the code
doesn't hang, but the performance is lower for other providers different to
"verbs".
The MPICH version used is 4.2.1 with the following configure line:
./configure --prefix=... --with-device=ch4:ofi --disable-psm3
In addition, I have attached a file
<https://urldefense.us/v3/__https://lorca.act.uji.es/gitlab/martini/mpich_of… >
with
the output of executing mpirun with "MPIR_CVAR_DEBUG_SUMMARY=1" and
unsetting FI_PROVIDER.
The minimal code
<https://urldefense.us/v3/__https://lorca.act.uji.es/gitlab/martini/mpich_of… >
to reproduce the problem is the following:
===============
#include <stdio.h>
#include <stdlib.h>
#include <mpi.h>
int main(int argc, char* argv[]) {
MPI_Init(&argc, &argv);
MPI_Comm spawn, new_comm;
MPI_Comm_get_parent(&spawn);
if(spawn == MPI_COMM_NULL) {
int num_c = atoi(argv[1]);
MPI_Comm_spawn(argv[0], MPI_ARGV_NULL, num_c, MPI_INFO_NULL, 0,
MPI_COMM_WORLD, &spawn, MPI_ERRCODES_IGNORE);
MPI_Intercomm_merge(spawn, 0, &new_comm);
} else {
MPI_Intercomm_merge(spawn, 1, &new_comm);
MPI_Barrier(MPI_COMM_WORLD);
}
MPI_Finalize();
}
===============
Thank you for your help.
Best regards,
Iker
1
0
>../configure --prefix=/people/d3g293/mpich/mpich-4.2.1/build_decptn/install \
--with-device=ch4:ofi:sockets --with-libfabric=embedded \
--without-ucx CC=gcc CXX=g++
You are statically using "sockets" provider. Try --with-device=ch4:ofi
Hui
________________________________
From: Palmer, Bruce J <Bruce.Palmer(a)pnnl.gov>
Sent: Friday, June 21, 2024 12:27 PM
To: Zhou, Hui <zhouh(a)anl.gov>; discuss(a)mpich.org <discuss(a)mpich.org>
Subject: RE: Fail on MPI_Wait
Hui, When I set FI_PROVIDER=tcp, the code crashes in MPI_Init. Specifically, this code will fail on one process: #include "mpi. h" int main(int argc, char **argv) { MPI_Init(&argc, &argv); MPI_Finalize(); } I’m running on a
ZjQcmQRYFpfptBannerStart
This Message Is From an External Sender
This message came from outside your organization.
ZjQcmQRYFpfptBannerEnd
Hui,
When I set FI_PROVIDER=tcp, the code crashes in MPI_Init. Specifically, this code will fail on one process:
#include "mpi.h"
int main(int argc, char **argv) {
MPI_Init(&argc, &argv);
MPI_Finalize();
}
I’m running on a system with the following modules
[d3g293@deception02 testing]$ module list
Currently Loaded Modulefiles:
1) gcc/11.2.0 3) python/3.7.0 5) mkl/2019u4
2) cmake/3.21.4 4) git/2.42.0(default) 6) cuda/11.8
and a home-built version of mpich-4.2.1 configured with
../configure --prefix=/people/d3g293/mpich/mpich-4.2.1/build_decptn/install \
--with-device=ch4:ofi:sockets --with-libfabric=embedded \
--without-ucx CC=gcc CXX=g++
I thought it might have something to do with using a build configuration in my application build that is set up to include Cuda, but it also fails in MPI_Init with a non-Cuda configuraton if I set the FI_PROVIDER variable.
Bruce
From: Zhou, Hui <zhouh(a)anl.gov>
Sent: Friday, June 14, 2024 9:27 AM
To: Palmer, Bruce J <Bruce.Palmer(a)pnnl.gov>; discuss(a)mpich.org
Subject: Re: Fail on MPI_Wait
Never mind. It is v4.2.1.
________________________________
From: Zhou, Hui <zhouh(a)anl.gov<mailto:[email protected]>>
Sent: Friday, June 14, 2024 11:26 AM
To: Palmer, Bruce J <Bruce.Palmer(a)pnnl.gov<mailto:[email protected]>>; discuss(a)mpich.org<mailto:[email protected]> <discuss(a)mpich.org<mailto:[email protected]>>
Subject: Re: Fail on MPI_Wait
Bruce,
What is the mpich version, BTW?
--
Hui
________________________________
From: Zhou, Hui <zhouh(a)anl.gov<mailto:[email protected]>>
Sent: Friday, June 14, 2024 10:55 AM
To: Palmer, Bruce J <Bruce.Palmer(a)pnnl.gov<mailto:[email protected]>>; discuss(a)mpich.org<mailto:[email protected]> <discuss(a)mpich.org<mailto:[email protected]>>
Subject: Re: Fail on MPI_Wait
Bruce,
You are using the sockets provider. Could try set FI_PROVIDER=tcp to see if it makes a difference?
Meanwhile, if you can get a small reproducer – with the sockets provider or any provider, I'll try to debug it. It is difficult to guess the true source of the issue without a reproducer.
--
Hui
________________________________
From: Palmer, Bruce J <Bruce.Palmer(a)pnnl.gov<mailto:[email protected]>>
Sent: Friday, June 14, 2024 10:47 AM
To: Zhou, Hui <zhouh(a)anl.gov<mailto:[email protected]>>; discuss(a)mpich.org<mailto:[email protected]> <discuss(a)mpich.org<mailto:[email protected]>>
Subject: Re: Fail on MPI_Wait
The output to standard out from running on 2 nodes and one process per node is attached. From: Zhou, Hui <zhouh@ anl. gov> Date: Tuesday, June 11, 2024 at 5: 49 PM To: discuss@ mpich. org <discuss@ mpich. org> Cc: Palmer, Bruce J <Bruce. Palmer@ pnnl. gov>
ZjQcmQRYFpfptBannerStart
This Message Is From an External Sender
This message came from outside your organization.
ZjQcmQRYFpfptBannerEnd
The output to standard out from running on 2 nodes and one process per node is attached.
From: Zhou, Hui <zhouh(a)anl.gov<mailto:[email protected]>>
Date: Tuesday, June 11, 2024 at 5:49 PM
To: discuss(a)mpich.org<mailto:[email protected]> <discuss(a)mpich.org<mailto:[email protected]>>
Cc: Palmer, Bruce J <Bruce.Palmer(a)pnnl.gov<mailto:[email protected]>>
Subject: Re: Fail on MPI_Wait
>MPIDI_OFI_handle_cq_error(793): OFI poll failed (ofi_events.c:793:MPIDI_OFI_handle_cq_error:Input/output error)
This is an error coming from the libfabric provider. First we need find out which provider are you using. Try set environment variable MPIR_CVAR_DEBUG_SUMMARY=1 and run a simple MPI_INIT+MPI_Finalize test code. Could post its console output?
--
Hui
________________________________
From: Palmer, Bruce J via discuss <discuss(a)mpich.org<mailto:[email protected]>>
Sent: Tuesday, June 11, 2024 3:17 PM
To: discuss(a)mpich.org<mailto:[email protected]> <discuss(a)mpich.org<mailto:[email protected]>>
Cc: Palmer, Bruce J <Bruce.Palmer(a)pnnl.gov<mailto:[email protected]>>
Subject: [mpich-discuss] Fail on MPI_Wait
Hi, I’m trying to debug a GPU-aware runtime for the Global Arrays library. We had a version of this working a while ago, but it has mysteriously started failing and we are trying to track down why. Currently, we are getting failures in MPI_Wait
ZjQcmQRYFpfptBannerStart
This Message Is From an External Sender
This message came from outside your organization.
ZjQcmQRYFpfptBannerEnd
Hi,
I’m trying to debug a GPU-aware runtime for the Global Arrays library. We had a version of this working a while ago, but it has mysteriously started failing and we are trying to track down why. Currently, we are getting failures in MPI_Wait and were wondering if anyone could provide some information on what exactly seems to be failing inside the wait call. The error we are getting is
Abort(206752655) on node 0: Fatal error in internal_Wait: Other MPI error, error stack:
internal_Wait(68205)..........: MPI_Wait(request=0x500847a0, status=0x7ffff9331800) failed
MPIR_Wait(780)................:
MPIR_Wait_state(737)..........:
MPIDI_progress_test(134)......:
MPIDI_OFI_handle_cq_error(793): OFI poll failed (ofi_events.c:793:MPIDI_OFI_handle_cq_error:Input/output error)
I’ve verified that the handle corresponding to 0x500847a0 is getting set earlier in the code in an MPI_Isend call and that no MPI_Wait or MPI_Test is called on the handle before it crashes with the above error message. I’m using MPICH 4.2.1 using gcc/8.3.0. The MPICH library was configured with
../configure --prefix=/people/d3g293/mpich/mpich-4.2.1/build_newell/install \
--with-device=ch4:ofi:sockets --with-libfabric=embedded \
--without-ucx --enable-threads=multiple --with-slurm \
CC=gcc CXX=g+
I’ve tried building with UCX and gotten the same results.
Are these errors indicative of corruption of the request handle or problems with some internal MPI data structures or something else? Any information you can provide would be appreciated.
Thanks,
Bruce
1
0
Hui,
When I set FI_PROVIDER=tcp, the code crashes in MPI_Init. Specifically, this code will fail on one process:
#include "mpi.h"
int main(int argc, char **argv) {
MPI_Init(&argc, &argv);
MPI_Finalize();
}
I’m running on a system with the following modules
[d3g293@deception02 testing]$ module list
Currently Loaded Modulefiles:
1) gcc/11.2.0 3) python/3.7.0 5) mkl/2019u4
2) cmake/3.21.4 4) git/2.42.0(default) 6) cuda/11.8
and a home-built version of mpich-4.2.1 configured with
../configure --prefix=/people/d3g293/mpich/mpich-4.2.1/build_decptn/install \
--with-device=ch4:ofi:sockets --with-libfabric=embedded \
--without-ucx CC=gcc CXX=g++
I thought it might have something to do with using a build configuration in my application build that is set up to include Cuda, but it also fails in MPI_Init with a non-Cuda configuraton if I set the FI_PROVIDER variable.
Bruce
From: Zhou, Hui <zhouh(a)anl.gov>
Sent: Friday, June 14, 2024 9:27 AM
To: Palmer, Bruce J <Bruce.Palmer(a)pnnl.gov>; discuss(a)mpich.org
Subject: Re: Fail on MPI_Wait
Never mind. It is v4.2.1.
________________________________
From: Zhou, Hui <zhouh(a)anl.gov<mailto:[email protected]>>
Sent: Friday, June 14, 2024 11:26 AM
To: Palmer, Bruce J <Bruce.Palmer(a)pnnl.gov<mailto:[email protected]>>; discuss(a)mpich.org<mailto:[email protected]> <discuss(a)mpich.org<mailto:[email protected]>>
Subject: Re: Fail on MPI_Wait
Bruce,
What is the mpich version, BTW?
--
Hui
________________________________
From: Zhou, Hui <zhouh(a)anl.gov<mailto:[email protected]>>
Sent: Friday, June 14, 2024 10:55 AM
To: Palmer, Bruce J <Bruce.Palmer(a)pnnl.gov<mailto:[email protected]>>; discuss(a)mpich.org<mailto:[email protected]> <discuss(a)mpich.org<mailto:[email protected]>>
Subject: Re: Fail on MPI_Wait
Bruce,
You are using the sockets provider. Could try set FI_PROVIDER=tcp to see if it makes a difference?
Meanwhile, if you can get a small reproducer – with the sockets provider or any provider, I'll try to debug it. It is difficult to guess the true source of the issue without a reproducer.
--
Hui
________________________________
From: Palmer, Bruce J <Bruce.Palmer(a)pnnl.gov<mailto:[email protected]>>
Sent: Friday, June 14, 2024 10:47 AM
To: Zhou, Hui <zhouh(a)anl.gov<mailto:[email protected]>>; discuss(a)mpich.org<mailto:[email protected]> <discuss(a)mpich.org<mailto:[email protected]>>
Subject: Re: Fail on MPI_Wait
The output to standard out from running on 2 nodes and one process per node is attached. From: Zhou, Hui <zhouh@ anl. gov> Date: Tuesday, June 11, 2024 at 5: 49 PM To: discuss@ mpich. org <discuss@ mpich. org> Cc: Palmer, Bruce J <Bruce. Palmer@ pnnl. gov>
ZjQcmQRYFpfptBannerStart
This Message Is From an External Sender
This message came from outside your organization.
ZjQcmQRYFpfptBannerEnd
The output to standard out from running on 2 nodes and one process per node is attached.
From: Zhou, Hui <zhouh(a)anl.gov<mailto:[email protected]>>
Date: Tuesday, June 11, 2024 at 5:49 PM
To: discuss(a)mpich.org<mailto:[email protected]> <discuss(a)mpich.org<mailto:[email protected]>>
Cc: Palmer, Bruce J <Bruce.Palmer(a)pnnl.gov<mailto:[email protected]>>
Subject: Re: Fail on MPI_Wait
>MPIDI_OFI_handle_cq_error(793): OFI poll failed (ofi_events.c:793:MPIDI_OFI_handle_cq_error:Input/output error)
This is an error coming from the libfabric provider. First we need find out which provider are you using. Try set environment variable MPIR_CVAR_DEBUG_SUMMARY=1 and run a simple MPI_INIT+MPI_Finalize test code. Could post its console output?
--
Hui
________________________________
From: Palmer, Bruce J via discuss <discuss(a)mpich.org<mailto:[email protected]>>
Sent: Tuesday, June 11, 2024 3:17 PM
To: discuss(a)mpich.org<mailto:[email protected]> <discuss(a)mpich.org<mailto:[email protected]>>
Cc: Palmer, Bruce J <Bruce.Palmer(a)pnnl.gov<mailto:[email protected]>>
Subject: [mpich-discuss] Fail on MPI_Wait
Hi, I’m trying to debug a GPU-aware runtime for the Global Arrays library. We had a version of this working a while ago, but it has mysteriously started failing and we are trying to track down why. Currently, we are getting failures in MPI_Wait
ZjQcmQRYFpfptBannerStart
This Message Is From an External Sender
This message came from outside your organization.
ZjQcmQRYFpfptBannerEnd
Hi,
I’m trying to debug a GPU-aware runtime for the Global Arrays library. We had a version of this working a while ago, but it has mysteriously started failing and we are trying to track down why. Currently, we are getting failures in MPI_Wait and were wondering if anyone could provide some information on what exactly seems to be failing inside the wait call. The error we are getting is
Abort(206752655) on node 0: Fatal error in internal_Wait: Other MPI error, error stack:
internal_Wait(68205)..........: MPI_Wait(request=0x500847a0, status=0x7ffff9331800) failed
MPIR_Wait(780)................:
MPIR_Wait_state(737)..........:
MPIDI_progress_test(134)......:
MPIDI_OFI_handle_cq_error(793): OFI poll failed (ofi_events.c:793:MPIDI_OFI_handle_cq_error:Input/output error)
I’ve verified that the handle corresponding to 0x500847a0 is getting set earlier in the code in an MPI_Isend call and that no MPI_Wait or MPI_Test is called on the handle before it crashes with the above error message. I’m using MPICH 4.2.1 using gcc/8.3.0. The MPICH library was configured with
../configure --prefix=/people/d3g293/mpich/mpich-4.2.1/build_newell/install \
--with-device=ch4:ofi:sockets --with-libfabric=embedded \
--without-ucx --enable-threads=multiple --with-slurm \
CC=gcc CXX=g+
I’ve tried building with UCX and gotten the same results.
Are these errors indicative of corruption of the request handle or problems with some internal MPI data structures or something else? Any information you can provide would be appreciated.
Thanks,
Bruce
1
0
Unfortunately, still couldn’t make it work.
Any more debugging tips?
It does work if I just use 1 host
FI_PROVIDER=verbs MPIR_CVAR_DEBUG_SUMMARY=1 /opt/mpich/bin/mpirun -n 5 -hosts 192.168.2.200 ./a.out
However, when I see what is on the wire, it is TCP messages. I am looking for RoCE messages on it.
root@ampere-altra-2-1:/mpich/examples# FI_PROVIDER=verbs MPIR_CVAR_DEBUG_SUMMARY=1 /opt/mpich/bin/mpirun -n 5 -hosts 192.168.2.200,192.168.2.100 ./a.out
==== Various sizes and limits ====
sizeof(MPIDI_per_vci_t): 192
Required minimum FI_VERSION: 0, current version: 10015
provider: verbs, score = 0, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200
provider: verbs, score = 0, pref = 0, FI_SOCKADDR_IN6 [28] fe80::526b:4bff:fefc:1342
provider: verbs, score = 0, pref = 0, FI_SOCKADDR_IB [48]
provider: verbs, score = 0, pref = 0, FI_SOCKADDR_IB [48]
provider: verbs, score = 0, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200
provider: verbs, score = 0, pref = 0, FI_SOCKADDR_IN6 [28] fe80::526b:4bff:fefc:1342
provider: verbs, score = 0, pref = 0, FI_SOCKADDR_IB [48]
provider: verbs, score = 0, pref = 0, FI_SOCKADDR_IB [48]
provider: verbs, score = 0, pref = 0, FI_FORMAT_UNSPEC [32]
provider: verbs, score = 0, pref = 0, FI_FORMAT_UNSPEC [32]
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::526b:4bff:fefc:1342
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48]
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48]
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::526b:4bff:fefc:1342
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48]
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48]
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::526b:4bff:fefc:1342
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48]
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48]
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::526b:4bff:fefc:1342
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48]
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48]
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::526b:4bff:fefc:1342
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48]
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48]
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::526b:4bff:fefc:1342
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48]
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48]
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::526b:4bff:fefc:1342
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48]
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48]
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::526b:4bff:fefc:1342
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48]
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48]
provider: verbs;ofi_rxd, score = 5, pref = -2, FI_FORMAT_UNSPEC [32]
provider: verbs;ofi_rxd, score = 5, pref = -2, FI_FORMAT_UNSPEC [32]
Required minimum FI_VERSION: 10006, current version: 10015
==== Capability set configuration ====
libfabric provider: verbs;ofi_rxm - IB-0xfe80000000000000
MPIDI_OFI_ENABLE_DATA: 0
MPIDI_OFI_ENABLE_AV_TABLE: 1
MPIDI_OFI_ENABLE_SCALABLE_ENDPOINTS: 0
MPIDI_OFI_ENABLE_SHARED_CONTEXTS: 0
MPIDI_OFI_ENABLE_MR_VIRT_ADDRESS: 1
MPIDI_OFI_ENABLE_MR_ALLOCATED: 1
MPIDI_OFI_ENABLE_MR_REGISTER_NULL: 1
MPIDI_OFI_ENABLE_MR_PROV_KEY: 1
MPIDI_OFI_ENABLE_TAGGED: 1
MPIDI_OFI_ENABLE_AM: 1
MPIDI_OFI_ENABLE_RMA: 1
MPIDI_OFI_ENABLE_ATOMICS: 0
MPIDI_OFI_FETCH_ATOMIC_IOVECS: 1
MPIDI_OFI_ENABLE_DATA_AUTO_PROGRESS: 0
MPIDI_OFI_ENABLE_CONTROL_AUTO_PROGRESS: 0
MPIDI_OFI_ENABLE_PT2PT_NOPACK: 1
MPIDI_OFI_ENABLE_TRIGGERED: 0
MPIDI_OFI_ENABLE_HMEM: 0
MPIDI_OFI_NUM_AM_BUFFERS: 8
MPIDI_OFI_NUM_OPTIMIZED_MEMORY_REGIONS: 0
MPIDI_OFI_CONTEXT_BITS: 16
MPIDI_OFI_SOURCE_BITS: 23
MPIDI_OFI_TAG_BITS: 20
MPIDI_OFI_VNI_USE_DOMAIN: 1
MAXIMUM SUPPORTED RANKS: 8388608
MAXIMUM TAG: 1048576
==== Provider global thresholds ====
max_buffered_send: 192
max_buffered_write: 192
max_msg_size: 1073741824
max_order_raw: 1073741824
max_order_war: 0
max_order_waw: 1073741824
tx_iov_limit: 4
rx_iov_limit: 4
rma_iov_limit: 1
max_mr_key_size: 4
==== Various sizes and limits ====
MPIDI_OFI_AM_MSG_HEADER_SIZE: 24
MPIDI_OFI_MAX_AM_HDR_SIZE: 255
sizeof(MPIDI_OFI_am_request_header_t): 416
sizeof(MPIDI_OFI_per_vci_t): 52480
MPIDI_OFI_AM_HDR_POOL_CELL_SIZE: 1024
MPIDI_OFI_DEFAULT_SHORT_SEND_SIZE: 16384
Assertion failed in file src/mpid/ch4/netmod/ofi/init_addrxchg.c at line 151: mapped_table[i] != FI_ADDR_NOTAVAIL
Assertion failed in file src/mpid/ch4/netmod/ofi/init_addrxchg.c at line 151: mapped_table[i] != FI_ADDR_NOTAVAIL
Assertion failed in file src/mpid/ch4/netmod/ofi/init_addrxchg.c at line 151: mapped_table[i] != FI_ADDR_NOTAVAIL
Assertion failed in file src/mpid/ch4/netmod/ofi/init_addrxchg.c at line 151: mapped_table[i] != FI_ADDR_NOTAVAIL
Assertion failed in file src/mpid/ch4/netmod/ofi/init_addrxchg.c at line 151: mapped_table[i] != FI_ADDR_NOTAVAIL
/opt/mpich/lib/libmpi.so.0(+0x58c1b5) [0x7fdf0919f1b5]
/opt/mpich/lib/libmpi.so.0(+0x491858) [0x7fdf090a4858]
/opt/mpich/lib/libmpi.so.0(+0x55443c) [0x7fdf0916743c]
/opt/mpich/lib/libmpi.so.0(+0x53409d) [0x7fdf0914709d]
/opt/mpich/lib/libmpi.so.0(+0x4dc78f) [0x7fdf090ef78f]
/opt/mpich/lib/libmpi.so.0(+0x4df10a) [0x7fdf090f210a]
/opt/mpich/lib/libmpi.so.0(+0x3deab6) [0x7fdf08ff1ab6]
/opt/mpich/lib/libmpi.so.0(+0x3e0732) [0x7fdf08ff3732]
/opt/mpich/lib/libmpi.so.0(+0x3dd075) [0x7fdf08ff0075]
/opt/mpich/lib/libmpi.so.0(+0x418215) [0x7fdf0902b215]
/opt/mpich/lib/libmpi.so.0(+0x4188fa) [0x7fdf0902b8fa]
/opt/mpich/lib/libmpi.so.0(+0x58c1b5) [0x7f3a84bfb1b5]
/opt/mpich/lib/libmpi.so.0(+0x491858) [0x7f3a84b00858]
/opt/mpich/lib/libmpi.so.0(+0x55443c) [0x7f3a84bc343c]
/opt/mpich/lib/libmpi.so.0(+0x53409d) [0x7f3a84ba309d]
/opt/mpich/lib/libmpi.so.0(+0x4dc78f) [0x7f3a84b4b78f]
/opt/mpich/lib/libmpi.so.0(+0x4df10a) [0x7f3a84b4e10a]
/opt/mpich/lib/libmpi.so.0(+0x3deab6) [0x7f3a84a4dab6]
/opt/mpich/lib/libmpi.so.0(+0x3e0732) [0x7f3a84a4f732]
/opt/mpich/lib/libmpi.so.0(+0x3dd075) [0x7f3a84a4c075]
/opt/mpich/lib/libmpi.so.0(+0x418215) [0x7f3a84a87215]
/opt/mpich/lib/libmpi.so.0(+0x4188fa) [0x7f3a84a878fa]
/opt/mpich/lib/libmpi.so.0(MPI_Init+0x34) [0x7f3a84746594]
./a.out(+0x121a) [0x5606031f021a]
/lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7f3a8446bd90]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80) [0x7f3a8446be40]
/opt/mpich/lib/libmpi.so.0(MPI_Init+0x34) [0x7fdf08cea594]
./a.out(+0x121a) [0x55accc98e21a]
/lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7fdf08a0fd90]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80) [0x7fdf08a0fe40]
./a.out(+0x1125) [0x55accc98e125]
Abort(1) on node 2: Internal error
/opt/mpich/lib/libmpi.so.0(+0x58c1b5) [0x7fa1131a11b5]
/opt/mpich/lib/libmpi.so.0(+0x491858) [0x7fa1130a6858]
/opt/mpich/lib/libmpi.so.0(+0x55443c) [0x7fa11316943c]
/opt/mpich/lib/libmpi.so.0(+0x53409d) [0x7fa11314909d]
/opt/mpich/lib/libmpi.so.0(+0x4dc78f) [0x7fa1130f178f]
/opt/mpich/lib/libmpi.so.0(+0x4df10a) [0x7fa1130f410a]
/opt/mpich/lib/libmpi.so.0(+0x3deab6) [0x7fa112ff3ab6]
/opt/mpich/lib/libmpi.so.0(+0x3e0732) [0x7fa112ff5732]
/opt/mpich/lib/libmpi.so.0(+0x3dd075) [0x7fa112ff2075]
/opt/mpich/lib/libmpi.so.0(+0x418215) [0x7fa11302d215]
/opt/mpich/lib/libmpi.so.0(+0x4188fa) [0x7fa11302d8fa]
/opt/mpich/lib/libmpi.so.0(MPI_Init+0x34) [0x7fa112cec594]
./a.out(+0x121a) [0x55a0b562221a]
/lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7fa112a11d90]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80) [0x7fa112a11e40]
./a.out(+0x1125) [0x55a0b5622125]
Abort(1) on node 4: Internal error
./a.out(+0x1125) [0x5606031f0125]
Abort(1) on node 0: Internal error
/opt/mpich/lib/libmpi.so.0(+0x59d26c) [0xffffb4dbd26c]
/opt/mpich/lib/libmpi.so.0(+0x4a6b58) [0xffffb4cc6b58]
/opt/mpich/lib/libmpi.so.0(+0x5678b0) [0xffffb4d878b0]
/opt/mpich/lib/libmpi.so.0(+0x548c54) [0xffffb4d68c54]
/opt/mpich/lib/libmpi.so.0(+0x4f710c) [0xffffb4d1710c]
/opt/mpich/lib/libmpi.so.0(+0x4f9890) [0xffffb4d19890]
/opt/mpich/lib/libmpi.so.0(+0x59d26c) [0xffffa9e9d26c]
/opt/mpich/lib/libmpi.so.0(+0x4a6b58) [0xffffa9da6b58]
/opt/mpich/lib/libmpi.so.0(+0x5678b0) [0xffffa9e678b0]
/opt/mpich/lib/libmpi.so.0(+0x548c54) [0xffffa9e48c54]
/opt/mpich/lib/libmpi.so.0(+0x4f710c) [0xffffa9df710c]
/opt/mpich/lib/libmpi.so.0(+0x4f9890) [0xffffa9df9890]
/opt/mpich/lib/libmpi.so.0(+0x3ffd2c) [0xffffa9cffd2c]
/opt/mpich/lib/libmpi.so.0(+0x4017ec) [0xffffa9d017ec]
/opt/mpich/lib/libmpi.so.0(+0x3fe384) [0xffffa9cfe384]
/opt/mpich/lib/libmpi.so.0(+0x436a64) [0xffffa9d36a64]
/opt/mpich/lib/libmpi.so.0(+0x43700c) [0xffffa9d3700c]
/opt/mpich/lib/libmpi.so.0(MPI_Init+0x44) [0xffffa9a2eeb4]
./a.out(+0x9c4) [0xaaaabd4709c4]
/lib/aarch64-linux-gnu/libc.so.6(+0x273fc) [0xffffa97773fc]
/lib/aarch64-linux-gnu/libc.so.6(__libc_start_main+0x98) [0xffffa97774cc]
./a.out(+0x8b0) [0xaaaabd4708b0]
/opt/mpich/lib/libmpi.so.0(+0x3ffd2c) [0xffffb4c1fd2c]
/opt/mpich/lib/libmpi.so.0(+0x4017ec) [0xffffb4c217ec]
/opt/mpich/lib/libmpi.so.0(+0x3fe384) [0xffffb4c1e384]
/opt/mpich/lib/libmpi.so.0(+0x436a64) [0xffffb4c56a64]
/opt/mpich/lib/libmpi.so.0(+0x43700c) [0xffffb4c5700c]
/opt/mpich/lib/libmpi.so.0(MPI_Init+0x44) [0xffffb494eeb4]
./a.out(+0x9c4) [0xaaaab8e909c4]
/lib/aarch64-linux-gnu/libc.so.6(+0x273fc) [0xffffb46973fc]
/lib/aarch64-linux-gnu/libc.so.6(__libc_start_main+0x98) [0xffffb46974cc]
./a.out(+0x8b0) [0xaaaab8e908b0]
Abort(1) on node 1: Internal error
Abort(1) on node 3: Internal error
a.out: prov/util/src/util_mem_monitor.c:172: ofi_monitor_cleanup: Assertion `dlist_empty(&monitor->list)' failed.
a.out: prov/util/src/util_mem_monitor.c:172: ofi_monitor_cleanup: Assertion `dlist_empty(&monitor->list)' failed.
===================================================================================
= BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
= PID 105 RUNNING AT 192.168.2.100
= EXIT CODE: 134
= CLEANING UP REMAINING PROCESSES
= YOU CAN IGNORE THE BELOW CLEANUP MESSAGES
===================================================================================
YOUR APPLICATION TERMINATED WITH THE EXIT STRING: Aborted (signal 6)
This typically refers to a problem with your application.
Please see the FAQ page for debugging suggestions
From: Niyaz Murshed via discuss <discuss(a)mpich.org>
Date: Tuesday, June 18, 2024 at 11:13 AM
To: Zhou, Hui <zhouh(a)anl.gov>, discuss(a)mpich.org <discuss(a)mpich.org>
Cc: Niyaz Murshed <Niyaz.Murshed(a)arm.com>, nd <nd(a)arm.com>
Subject: Re: [mpich-discuss] Mpich over RDMA sample
Okay, let me keep digging. If verbs;ofi_rxm is good, then my understanding is that I should see the RoCE messages over the wire, correct? i. e once I can get the sample application working. Currently, we still see the error as below. I will
ZjQcmQRYFpfptBannerStart
This Message Is From an External Sender
This message came from outside your organization.
ZjQcmQRYFpfptBannerEnd
Okay, let me keep digging.
If verbs;ofi_rxm is good, then my understanding is that I should see the RoCE messages over the wire, correct?
i.e once I can get the sample application working.
Currently, we still see the error as below. I will try to use your patch for interface selection to see what happens.
==== Capability set configuration ====
libfabric provider: verbs;ofi_rxm - IB-0xfe80000000000000
Assertion failed in file src/mpid/ch4/netmod/ofi/init_addrxchg.c at line 151: mapped_table[i] != FI_ADDR_NOTAVAIL
Assertion failed in file src/mpid/ch4/netmod/ofi/init_addrxchg.c at line 151: mapped_table[i] != FI_ADDR_NOTAVAIL
Assertion failed in file src/mpid/ch4/netmod/ofi/init_addrxchg.c at line 151: mapped_table[i] != FI_ADDR_NOTAVAIL
Assertion failed in file src/mpid/ch4/netmod/ofi/init_addrxchg.c at line 151: mapped_table[i] != FI_ADDR_NOTAVAIL
Assertion failed in file src/mpid/ch4/netmod/ofi/init_addrxchg.c at line 151: mapped_table[i] != FI_ADDR_NOTAVAIL
From: Zhou, Hui <zhouh(a)anl.gov>
Date: Tuesday, June 18, 2024 at 11:05 AM
To: Niyaz Murshed <Niyaz.Murshed(a)arm.com>, discuss(a)mpich.org <discuss(a)mpich.org>
Cc: nd <nd(a)arm.com>
Subject: Re: Mpich over RDMA sample
Niyaz,
FYI -https://urldefense.us/v3/__https://ofiwg.github.io/libfabric/v1.21.0/man/fi_provider.7.html__;!!G_uCfscf7eWS!cDbck3MzxE4MNvuedpeu36W-QEWamMuYg13g3KM2baHb-NZlWpKNLupIgqwWVrph458DsJyPiWyBSS3K0ZE$ <https://urldefense.us/v3/__https:/ofiwg.github.io/libfabric/v1.21.0/man/fi_…>
Hui
________________________________
From: Niyaz Murshed <Niyaz.Murshed(a)arm.com>
Sent: Tuesday, June 18, 2024 11:00 AM
To: Zhou, Hui <zhouh(a)anl.gov>; discuss(a)mpich.org <discuss(a)mpich.org>
Cc: nd <nd(a)arm.com>
Subject: Re: Mpich over RDMA sample
Hi Hui, When you say message semantic, do you mean endpoint type message ? Mpich is running over libfabric. When I run sample application on libfabric, it does have message endpoints over RMA. From: Zhou, Hui <zhouh@ anl. gov> Date: Tuesday,
ZjQcmQRYFpfptBannerStart
This Message Is From an External Sender
This message came from outside your organization.
ZjQcmQRYFpfptBannerEnd
Hi Hui,
When you say message semantic, do you mean endpoint type message ?
Mpich is running over libfabric.
When I run sample application on libfabric, it does have message endpoints over RMA.
From: Zhou, Hui <zhouh(a)anl.gov>
Date: Tuesday, June 18, 2024 at 9:44 AM
To: Niyaz Murshed <Niyaz.Murshed(a)arm.com>, discuss(a)mpich.org <discuss(a)mpich.org>
Cc: nd <nd(a)arm.com>
Subject: Re: Mpich over RDMA sample
The "verbs" provider alone does not provide the message semantics we needed for MPI. "ofi_rxm" is the utility provider that provides the message semantics. "verbs;ofi_rxm" is the provider combination you want.
Hui
________________________________
From: Niyaz Murshed <Niyaz.Murshed(a)arm.com>
Sent: Tuesday, June 18, 2024 9:14 AM
To: Zhou, Hui <zhouh(a)anl.gov>; discuss(a)mpich.org <discuss(a)mpich.org>
Cc: nd <nd(a)arm.com>
Subject: Re: Mpich over RDMA sample
Hi Hui, I have set the env variable to FI_PROVIDER=verbs , after thich I get the below. libfabric provider: verbs;ofi_rxm - IB-0xfe80000000000000 It does not chose just “verbs” , it selects verbs;ofi_rxf I am wanting to select the below: provider:
ZjQcmQRYFpfptBannerStart
This Message Is From an External Sender
This message came from outside your organization.
ZjQcmQRYFpfptBannerEnd
Hi Hui,
I have set the env variable to FI_PROVIDER=verbs , after thich I get the below.
libfabric provider: verbs;ofi_rxm - IB-0xfe80000000000000
It does not chose just “verbs” , it selects verbs;ofi_rxf
I am wanting to select the below:
provider: verbs, score = 0, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200
I would really like to thank you for such fast responses.
From: Zhou, Hui <zhouh(a)anl.gov>
Date: Tuesday, June 18, 2024 at 8:58 AM
To: Niyaz Murshed <Niyaz.Murshed(a)arm.com>, discuss(a)mpich.org <discuss(a)mpich.org>
Cc: nd <nd(a)arm.com>
Subject: Re: Mpich over RDMA sample
Niyaz,
Set environment variable FI_PROVIDER=verbs to select the verbs provider.
Hui
________________________________
From: Niyaz Murshed <Niyaz.Murshed(a)arm.com>
Sent: Monday, June 17, 2024 8:57 PM
To: Zhou, Hui <zhouh(a)anl.gov>; discuss(a)mpich.org <discuss(a)mpich.org>
Cc: nd <nd(a)arm.com>
Subject: Re: Mpich over RDMA sample
Hi Hui, I would still need some assistance in this. I am not able to make mpich chose verbs provider ☹ With the log : provider: verbs, score = 0, pref = 0, FI_SOCKADDR_IN [16] 192. 168. 2. 200 Is my assumption to think the above log represent
ZjQcmQRYFpfptBannerStart
This Message Is From an External Sender
This message came from outside your organization.
ZjQcmQRYFpfptBannerEnd
Hi Hui,
I would still need some assistance in this. I am not able to make mpich chose verbs provider ☹
With the log :
provider: verbs, score = 0, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200
Is my assumption to think the above log represent that verbs provider knows about a path to 192.168.2.200 ?
How does the application hello-world decide which provider to use?
Running on server: 192.168.2.100
root@ampere-altra-2-1# FI_PROVIDER=verbs MPIR_CVAR_DEBUG_SUMMARY=1 mpirun -n 5 -hosts 192.168.2.200,192.168.2.100 /mpich/examples/a.out
==== Various sizes and limits ====
sizeof(MPIDI_per_vci_t): 192
Required minimum FI_VERSION: 0, current version: 10015
provider: verbs, score = 0, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200
provider: verbs, score = 0, pref = 0, FI_SOCKADDR_IN6 [28] fe80::526b:4bff:fefc:1342
provider: verbs, score = 0, pref = 0, FI_SOCKADDR_IB [48]
provider: verbs, score = 0, pref = 0, FI_SOCKADDR_IB [48]
provider: verbs, score = 0, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200
provider: verbs, score = 0, pref = 0, FI_SOCKADDR_IN6 [28] fe80::526b:4bff:fefc:1342
provider: verbs, score = 0, pref = 0, FI_SOCKADDR_IB [48]
provider: verbs, score = 0, pref = 0, FI_SOCKADDR_IB [48]
provider: verbs, score = 0, pref = 0, FI_FORMAT_UNSPEC [32]
provider: verbs, score = 0, pref = 0, FI_FORMAT_UNSPEC [32]
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::526b:4bff:fefc:1342
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48]
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48]
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::526b:4bff:fefc:1342
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48]
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48]
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::526b:4bff:fefc:1342
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48]
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48]
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::526b:4bff:fefc:1342
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48]
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48]
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::526b:4bff:fefc:1342
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48]
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48]
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::526b:4bff:fefc:1342
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48]
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48]
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::526b:4bff:fefc:1342
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48]
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48]
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::526b:4bff:fefc:1342
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48]
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48]
provider: verbs;ofi_rxd, score = 5, pref = -2, FI_FORMAT_UNSPEC [32]
provider: verbs;ofi_rxd, score = 5, pref = -2, FI_FORMAT_UNSPEC [32]
Required minimum FI_VERSION: 10006, current version: 10015
==== Capability set configuration ====
libfabric provider: verbs;ofi_rxm - IB-0xfe80000000000000
MPIDI_OFI_ENABLE_DATA: 0
MPIDI_OFI_ENABLE_AV_TABLE: 1
MPIDI_OFI_ENABLE_SCALABLE_ENDPOINTS: 0
MPIDI_OFI_ENABLE_SHARED_CONTEXTS: 0
MPIDI_OFI_ENABLE_MR_VIRT_ADDRESS: 1
MPIDI_OFI_ENABLE_MR_ALLOCATED: 1
MPIDI_OFI_ENABLE_MR_REGISTER_NULL: 1
MPIDI_OFI_ENABLE_MR_PROV_KEY: 1
MPIDI_OFI_ENABLE_TAGGED: 1
MPIDI_OFI_ENABLE_AM: 1
MPIDI_OFI_ENABLE_RMA: 1
MPIDI_OFI_ENABLE_ATOMICS: 0
MPIDI_OFI_FETCH_ATOMIC_IOVECS: 1
MPIDI_OFI_ENABLE_DATA_AUTO_PROGRESS: 0
MPIDI_OFI_ENABLE_CONTROL_AUTO_PROGRESS: 0
MPIDI_OFI_ENABLE_PT2PT_NOPACK: 1
MPIDI_OFI_ENABLE_TRIGGERED: 0
MPIDI_OFI_ENABLE_HMEM: 0
MPIDI_OFI_NUM_AM_BUFFERS: 8
MPIDI_OFI_NUM_OPTIMIZED_MEMORY_REGIONS: 0
MPIDI_OFI_CONTEXT_BITS: 16
MPIDI_OFI_SOURCE_BITS: 23
MPIDI_OFI_TAG_BITS: 20
MPIDI_OFI_VNI_USE_DOMAIN: 1
MAXIMUM SUPPORTED RANKS: 8388608
MAXIMUM TAG: 1048576
==== Provider global thresholds ====
max_buffered_send: 192
max_buffered_write: 192
max_msg_size: 1073741824
max_order_raw: 1073741824
max_order_war: 0
max_order_waw: 1073741824
tx_iov_limit: 4
rx_iov_limit: 4
rma_iov_limit: 1
max_mr_key_size: 4
==== Various sizes and limits ====
MPIDI_OFI_AM_MSG_HEADER_SIZE: 24
MPIDI_OFI_MAX_AM_HDR_SIZE: 255
sizeof(MPIDI_OFI_am_request_header_t): 416
sizeof(MPIDI_OFI_per_vci_t): 52480
MPIDI_OFI_AM_HDR_POOL_CELL_SIZE: 1024
MPIDI_OFI_DEFAULT_SHORT_SEND_SIZE: 16384
Assertion failed in file src/mpid/ch4/netmod/ofi/init_addrxchg.c at line 151: mapped_table[i] != FI_ADDR_NOTAVAIL
Assertion failed in file src/mpid/ch4/netmod/ofi/init_addrxchg.c at line 151: mapped_table[i] != FI_ADDR_NOTAVAIL
Assertion failed in file src/mpid/ch4/netmod/ofi/init_addrxchg.c at line 151: mapped_table[i] != FI_ADDR_NOTAVAIL
Assertion failed in file src/mpid/ch4/netmod/ofi/init_addrxchg.c at line 151: mapped_table[i] != FI_ADDR_NOTAVAIL
Assertion failed in file src/mpid/ch4/netmod/ofi/init_addrxchg.c at line 151: mapped_table[i] != FI_ADDR_NOTAVAIL
If I do not use the FI_PROVIDER=verbs, it will work over sockets.
root@ampere-altra-2-1: # MPIR_CVAR_DEBUG_SUMMARY=1 mpirun -n 5 -hosts 192.168.2.200,192.168.2.100 /mpich/examples/a.out
==== Various sizes and limits ====
sizeof(MPIDI_per_vci_t): 192
Required minimum FI_VERSION: 0, current version: 10015
provider: verbs, score = 0, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200
provider: verbs, score = 0, pref = 0, FI_SOCKADDR_IN6 [28] fe80::526b:4bff:fefc:1342
provider: verbs, score = 0, pref = 0, FI_SOCKADDR_IB [48]
provider: verbs, score = 0, pref = 0, FI_SOCKADDR_IB [48]
provider: verbs, score = 0, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200
provider: verbs, score = 0, pref = 0, FI_SOCKADDR_IN6 [28] fe80::526b:4bff:fefc:1342
provider: verbs, score = 0, pref = 0, FI_SOCKADDR_IB [48]
provider: verbs, score = 0, pref = 0, FI_SOCKADDR_IB [48]
provider: verbs, score = 0, pref = 0, FI_FORMAT_UNSPEC [32]
provider: verbs, score = 0, pref = 0, FI_FORMAT_UNSPEC [32]
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::526b:4bff:fefc:1342
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48]
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48]
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::526b:4bff:fefc:1342
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48]
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48]
provider: tcp, score = 4, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200
provider: tcp, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::526b:4bff:fefc:1342
provider: tcp, score = 4, pref = 0, FI_SOCKADDR_IN [16] 10.118.91.159
provider: tcp, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::be97:e1ff:fe9d:7caa
provider: tcp, score = 4, pref = 0, FI_SOCKADDR_IN [16] 127.0.0.1
provider: tcp, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] ::1
provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200
provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::526b:4bff:fefc:1342
provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 10.118.91.159
provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::be97:e1ff:fe9d:7caa
provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 127.0.0.1
provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] ::1
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::526b:4bff:fefc:1342
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48]
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48]
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::526b:4bff:fefc:1342
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48]
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48]
provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200
provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::526b:4bff:fefc:1342
provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 10.118.91.159
provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::be97:e1ff:fe9d:7caa
provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 127.0.0.1
provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] ::1
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::526b:4bff:fefc:1342
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48]
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48]
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::526b:4bff:fefc:1342
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48]
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48]
provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200
provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::526b:4bff:fefc:1342
provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 10.118.91.159
provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::be97:e1ff:fe9d:7caa
provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 127.0.0.1
provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] ::1
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::526b:4bff:fefc:1342
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48]
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48]
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::526b:4bff:fefc:1342
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48]
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48]
provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200
provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::526b:4bff:fefc:1342
provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 10.118.91.159
provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::be97:e1ff:fe9d:7caa
provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 127.0.0.1
provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] ::1
provider: verbs;ofi_rxd, score = 5, pref = -2, FI_FORMAT_UNSPEC [32]
provider: verbs;ofi_rxd, score = 5, pref = -2, FI_FORMAT_UNSPEC [32]
provider: udp;ofi_rxd, score = 5, pref = -2, FI_SOCKADDR_IN [16] 192.168.2.200
provider: udp;ofi_rxd, score = 5, pref = -2, FI_SOCKADDR_IN6 [28] fe80::526b:4bff:fefc:1342
provider: udp;ofi_rxd, score = 5, pref = -2, FI_SOCKADDR_IN [16] 10.118.91.159
provider: udp;ofi_rxd, score = 5, pref = -2, FI_SOCKADDR_IN6 [28] fe80::be97:e1ff:fe9d:7caa
provider: udp;ofi_rxd, score = 5, pref = -2, FI_SOCKADDR_IN [16] 127.0.0.1
provider: udp;ofi_rxd, score = 5, pref = -2, FI_SOCKADDR_IN6 [28] ::1
provider: shm, score = 4, pref = -2, FI_ADDR_STR [14] - fi_shm://1177
provider: shm, score = 4, pref = -2, FI_ADDR_STR [14] - fi_shm://1177
provider: udp, score = 0, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200
provider: udp, score = 0, pref = 0, FI_SOCKADDR_IN6 [28] fe80::526b:4bff:fefc:1342
provider: udp, score = 0, pref = 0, FI_SOCKADDR_IN [16] 10.118.91.159
provider: udp, score = 0, pref = 0, FI_SOCKADDR_IN6 [28] fe80::be97:e1ff:fe9d:7caa
provider: udp, score = 0, pref = 0, FI_SOCKADDR_IN [16] 127.0.0.1
provider: udp, score = 0, pref = 0, FI_SOCKADDR_IN6 [28] ::1
provider: tcp, score = 0, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200
provider: tcp, score = 0, pref = 0, FI_SOCKADDR_IN6 [28] fe80::526b:4bff:fefc:1342
provider: tcp, score = 0, pref = 0, FI_SOCKADDR_IN [16] 10.118.91.159
provider: tcp, score = 0, pref = 0, FI_SOCKADDR_IN6 [28] fe80::be97:e1ff:fe9d:7caa
provider: tcp, score = 0, pref = 0, FI_SOCKADDR_IN [16] 127.0.0.1
provider: tcp, score = 0, pref = 0, FI_SOCKADDR_IN6 [28] ::1
provider: tcp, score = 4, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200
provider: tcp, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::526b:4bff:fefc:1342
provider: tcp, score = 4, pref = 0, FI_SOCKADDR_IN [16] 10.118.91.159
provider: tcp, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::be97:e1ff:fe9d:7caa
provider: tcp, score = 4, pref = 0, FI_SOCKADDR_IN [16] 127.0.0.1
provider: tcp, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] ::1
provider: sockets, score = 3, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200
provider: sockets, score = 3, pref = 0, FI_SOCKADDR_IN6 [28] fe80::526b:4bff:fefc:1342
provider: sockets, score = 3, pref = 0, FI_SOCKADDR_IN [16] 10.118.91.159
provider: sockets, score = 3, pref = 0, FI_SOCKADDR_IN6 [28] fe80::be97:e1ff:fe9d:7caa
provider: sockets, score = 3, pref = 0, FI_SOCKADDR_IN [16] 127.0.0.1
provider: sockets, score = 3, pref = 0, FI_SOCKADDR_IN6 [28] ::1
provider: sockets, score = 5, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200
provider: sockets, score = 5, pref = 0, FI_SOCKADDR_IN6 [28] fe80::526b:4bff:fefc:1342
provider: sockets, score = 5, pref = 0, FI_SOCKADDR_IN [16] 10.118.91.159
provider: sockets, score = 5, pref = 0, FI_SOCKADDR_IN6 [28] fe80::be97:e1ff:fe9d:7caa
provider: sockets, score = 5, pref = 0, FI_SOCKADDR_IN [16] 127.0.0.1
provider: sockets, score = 5, pref = 0, FI_SOCKADDR_IN6 [28] ::1
provider: sockets, score = 5, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200
provider: sockets, score = 5, pref = 0, FI_SOCKADDR_IN6 [28] fe80::526b:4bff:fefc:1342
provider: sockets, score = 5, pref = 0, FI_SOCKADDR_IN [16] 10.118.91.159
provider: sockets, score = 5, pref = 0, FI_SOCKADDR_IN6 [28] fe80::be97:e1ff:fe9d:7caa
provider: sockets, score = 5, pref = 0, FI_SOCKADDR_IN [16] 127.0.0.1
provider: sockets, score = 5, pref = 0, FI_SOCKADDR_IN6 [28] ::1
provider: sm2, score = 3, pref = 0, FI_ADDR_STR [13] - fi_sm2://1177
provider: sm2, score = 3, pref = 0, FI_ADDR_STR [13] - fi_sm2://1177
Required minimum FI_VERSION: 10005, current version: 10015
==== Capability set configuration ====
libfabric provider: sockets - 192.168.2.0/24
MPIDI_OFI_ENABLE_DATA: 1
MPIDI_OFI_ENABLE_AV_TABLE: 1
MPIDI_OFI_ENABLE_SCALABLE_ENDPOINTS: 1
MPIDI_OFI_ENABLE_SHARED_CONTEXTS: 0
MPIDI_OFI_ENABLE_MR_VIRT_ADDRESS: 0
MPIDI_OFI_ENABLE_MR_ALLOCATED: 0
MPIDI_OFI_ENABLE_MR_REGISTER_NULL: 1
MPIDI_OFI_ENABLE_MR_PROV_KEY: 0
MPIDI_OFI_ENABLE_TAGGED: 1
MPIDI_OFI_ENABLE_AM: 1
MPIDI_OFI_ENABLE_RMA: 1
MPIDI_OFI_ENABLE_ATOMICS: 1
MPIDI_OFI_FETCH_ATOMIC_IOVECS: 1
MPIDI_OFI_ENABLE_DATA_AUTO_PROGRESS: 0
MPIDI_OFI_ENABLE_CONTROL_AUTO_PROGRESS: 0
MPIDI_OFI_ENABLE_PT2PT_NOPACK: 1
MPIDI_OFI_ENABLE_TRIGGERED: 0
MPIDI_OFI_ENABLE_HMEM: 0
MPIDI_OFI_NUM_AM_BUFFERS: 8
MPIDI_OFI_NUM_OPTIMIZED_MEMORY_REGIONS: 0
MPIDI_OFI_CONTEXT_BITS: 20
MPIDI_OFI_SOURCE_BITS: 0
MPIDI_OFI_TAG_BITS: 31
MPIDI_OFI_VNI_USE_DOMAIN: 1
MAXIMUM SUPPORTED RANKS: 4294967296
MAXIMUM TAG: 2147483648
==== Provider global thresholds ====
max_buffered_send: 255
max_buffered_write: 255
max_msg_size: 9223372036854775807
max_order_raw: -1
max_order_war: -1
max_order_waw: -1
tx_iov_limit: 8
rx_iov_limit: 8
rma_iov_limit: 8
max_mr_key_size: 8
==== Various sizes and limits ====
MPIDI_OFI_AM_MSG_HEADER_SIZE: 24
MPIDI_OFI_MAX_AM_HDR_SIZE: 255
sizeof(MPIDI_OFI_am_request_header_t): 416
sizeof(MPIDI_OFI_per_vci_t): 52480
MPIDI_OFI_AM_HDR_POOL_CELL_SIZE: 1024
MPIDI_OFI_DEFAULT_SHORT_SEND_SIZE: 16384
==== OFI dynamic settings ====
num_vcis: 1
num_nics: 1
======================================
error checking : enabled
QMPI : disabled
debugger support : disabled
thread level : MPI_THREAD_SINGLE
thread CS : per-vci
threadcomm : enabled
==== data structure summary ====
sizeof(MPIR_Comm): 1808
sizeof(MPIR_Request): 512
sizeof(MPIR_Datatype): 280
================================
Hello world from process 0 of 5
Hello world from process 2 of 5
Hello world from process 1 of 5
Hello world from process 4 of 5
Hello world from process 3 of 5
From: Zhou, Hui <zhouh(a)anl.gov>
Date: Wednesday, June 12, 2024 at 7:22 PM
To: Niyaz Murshed <Niyaz.Murshed(a)arm.com>, discuss(a)mpich.org <discuss(a)mpich.org>
Subject: Re: Mpich over RDMA sample
It’s possible that the native path for RMA on verbs are turned off due to verbs does not meet MPI’s full atomic ordering semantics. We’ll check and follow up.
--
Hui Zhou
From: Niyaz Murshed <Niyaz.Murshed(a)arm.com>
Date: Wednesday, June 12, 2024 at 3:59 PM
To: Zhou, Hui <zhouh(a)anl.gov>, discuss(a)mpich.org <discuss(a)mpich.org>
Cc: nd <nd(a)arm.com>
Subject: Re: Mpich over RDMA sample
I think image is not gone : https: //ibb. co/RDGnJfy From: Niyaz Murshed <Niyaz. Murshed@ arm. com> Date: Wednesday, June 12, 2024 at 3: 56 PM To: Zhou, Hui <zhouh@ anl. gov>, discuss@ mpich. org <discuss@ mpich. org> Cc: nd <nd@ arm. com>
ZjQcmQRYFpfptBannerStart
This Message Is From an External Sender
This message came from outside your organization.
ZjQcmQRYFpfptBannerEnd
I think image is not gone : https://urldefense.us/v3/__https://ibb.co/RDGnJfy__;!!G_uCfscf7eWS!cDbck3Mz… <https://urldefense.us/v3/__https:/ibb.co/RDGnJfy__;!!G_uCfscf7eWS!YH6L3OHfP…>
From: Niyaz Murshed <Niyaz.Murshed(a)arm.com>
Date: Wednesday, June 12, 2024 at 3:56 PM
To: Zhou, Hui <zhouh(a)anl.gov>, discuss(a)mpich.org <discuss(a)mpich.org>
Cc: nd <nd(a)arm.com>
Subject: Re: Mpich over RDMA sample
Thank you Hui for the reply.
I was expecting to see RoCE messages in the packet capture.
When I use verbs on libfabric sample tests, I see the RoCE messages as below :
[cid:[email protected]]
From: Zhou, Hui <zhouh(a)anl.gov>
Date: Wednesday, June 12, 2024 at 3:41 PM
To: Niyaz Murshed <Niyaz.Murshed(a)arm.com>, discuss(a)mpich.org <discuss(a)mpich.org>
Cc: nd <nd(a)arm.com>
Subject: Re: Mpich over RDMA sample
Niyaz,
ofi_rxm:verbs is the correct provider. The ofi_rxm provides the verbs provider additional messaging semantics that are needed to be used in MPI.
--
Hui Zhou
From: Niyaz Murshed <Niyaz.Murshed(a)arm.com>
Date: Wednesday, June 12, 2024 at 2:54 PM
To: Zhou, Hui <zhouh(a)anl.gov>, discuss(a)mpich.org <discuss(a)mpich.org>
Cc: nd <nd(a)arm.com>
Subject: Re: Mpich over RDMA sample
Thank you for the reply. I did that; however, it selects ofi_rxm: verbs libfabric: 412098: 1718221892: : ofi_rxm: av: ofi_av_insert_addr(): 313<info> fi_addr: 1 libfabric: 412099: 1718221892: : ofi_rxm: av: ofi_av_insert_addr(): 313<info> fi_addr:
ZjQcmQRYFpfptBannerStart
This Message Is From an External Sender
This message came from outside your organization.
ZjQcmQRYFpfptBannerEnd
Thank you for the reply.
I did that; however, it selects ofi_rxm:verbs
libfabric:412098:1718221892::ofi_rxm:av:ofi_av_insert_addr():313<info> fi_addr: 1
libfabric:412099:1718221892::ofi_rxm:av:ofi_av_insert_addr():313<info> fi_addr: 1
options:
backend: cpu
iters: 16
warmup_iters: 16
cache: 1
min_elem_count: 1
max_elem_count: 1
elem_counts: [1]
validate: last
window_size: 64
#------------------------------------------------------------
# Benchmarking: Bandwidth
# #processes: 2
#------------------------------------------------------------
#bytes #repetitions Mbytes/sec
4 16 0.02
# All done
libfabric:412098:1718221892::ofi_rxm:ep_ctrl:rxm_stop_listen():864<info> stopping CM thread
libfabric:412099:1718221892::ofi_rxm:ep_ctrl:rxm_stop_listen():864<info> stopping CM thread
As per https://urldefense.us/v3/__https://www.intel.com/content/www/us/en/docs/mpi… <https://urldefense.us/v3/__https:/www.intel.com/content/www/us/en/docs/mpi-…> , we need to add FI_PROVIDER=^ofi_rxm , but if do that, it moves back to sockets provider.
Is here a way to combine ^ofi_rxm and verbs
From: Zhou, Hui <zhouh(a)anl.gov>
Date: Wednesday, June 12, 2024 at 2:41 PM
To: Niyaz Murshed <Niyaz.Murshed(a)arm.com>, discuss(a)mpich.org <discuss(a)mpich.org>
Cc: nd <nd(a)arm.com>
Subject: Re: Mpich over RDMA sample
Niyaz,
All you need to do is to set an environment variable `FI_PROVIDER=verbs`.
--
Hui Zhou
From: Niyaz Murshed <Niyaz.Murshed(a)arm.com>
Date: Wednesday, June 12, 2024 at 1:23 PM
To: Zhou, Hui <zhouh(a)anl.gov>, discuss(a)mpich.org <discuss(a)mpich.org>
Cc: nd <nd(a)arm.com>
Subject: Re: Mpich over RDMA sample
When testing with Libfabric, verbs provider is selected. I did have to use “-e msg -d mlx5_1” so that it selects verbs. I was checking if there is anything like that for mpich sample tests. Else might need to do some hack in the code to force
ZjQcmQRYFpfptBannerStart
This Message Is From an External Sender
This message came from outside your organization.
ZjQcmQRYFpfptBannerEnd
When testing with Libfabric, verbs provider is selected. I did have to use “-e msg -d mlx5_1” so that it selects verbs.
I was checking if there is anything like that for mpich sample tests. Else might need to do some hack in the code to force the selection of verbs.
From: Zhou, Hui <zhouh(a)anl.gov>
Date: Wednesday, June 12, 2024 at 1:15 PM
To: discuss(a)mpich.org <discuss(a)mpich.org>
Cc: Niyaz Murshed <Niyaz.Murshed(a)arm.com>, nd <nd(a)arm.com>
Subject: Re: Mpich over RDMA sample
Libfabric support multiple providers. Sounds like it was selecting the sockets or tcp provider rather than a provider that support RoCE. I am not exactly sure whether the verbs provider will do that. If you can confirm the provider using libfabric tests, then you can try forcing MPICH to use that provider by setting the FI_PROVIDER environment variable.
--
Hui Zhou
From: Niyaz Murshed via discuss <discuss(a)mpich.org>
Date: Wednesday, June 12, 2024 at 9:03 AM
To: discuss(a)mpich.org <discuss(a)mpich.org>
Cc: Niyaz Murshed <Niyaz.Murshed(a)arm.com>, nd <nd(a)arm.com>
Subject: [mpich-discuss] Mpich over RDMA sample
Hello, I am trying to learn about MPICH and its performance over RDMA. I am using libfabric and installed mpich using the below configure. ./configure --prefix=/opt/mpich/ --with-ofi=/opt/libfabric/ When I run any applications between 2 directly
ZjQcmQRYFpfptBannerStart
This Message Is From an External Sender
This message came from outside your organization.
ZjQcmQRYFpfptBannerEnd
Hello,
I am trying to learn about MPICH and its performance over RDMA.
I am using libfabric and installed mpich using the below configure.
./configure --prefix=/opt/mpich/ --with-ofi=/opt/libfabric/
When I run any applications between 2 directly connected servers having Mellanox NICs, I see that communication is happening over tcp and not over RoCE.
Is there any way to test commination over RoCE ?
For eg. I was able to test it for libfabric using the below sample that comes along with libfabric to test RMA.
Is there something similar for MPICH ? or use the current sample to use RoCE by some parameter?
Server :
fi_rma_bw -s 192.168.1.100 -e msg -d mlx5_1 -S 1024 -I 1
Client :
fi_rma_bw -s 192.168.1.200 -e msg -d mlx5_3 192.168.1.100 -S 1024 -I 1
Regards,
Niyaz
1
0
Okay, let me keep digging.
If verbs;ofi_rxm is good, then my understanding is that I should see the RoCE messages over the wire, correct?
i.e once I can get the sample application working.
Currently, we still see the error as below. I will try to use your patch for interface selection to see what happens.
==== Capability set configuration ====
libfabric provider: verbs;ofi_rxm - IB-0xfe80000000000000
Assertion failed in file src/mpid/ch4/netmod/ofi/init_addrxchg.c at line 151: mapped_table[i] != FI_ADDR_NOTAVAIL
Assertion failed in file src/mpid/ch4/netmod/ofi/init_addrxchg.c at line 151: mapped_table[i] != FI_ADDR_NOTAVAIL
Assertion failed in file src/mpid/ch4/netmod/ofi/init_addrxchg.c at line 151: mapped_table[i] != FI_ADDR_NOTAVAIL
Assertion failed in file src/mpid/ch4/netmod/ofi/init_addrxchg.c at line 151: mapped_table[i] != FI_ADDR_NOTAVAIL
Assertion failed in file src/mpid/ch4/netmod/ofi/init_addrxchg.c at line 151: mapped_table[i] != FI_ADDR_NOTAVAIL
From: Zhou, Hui <zhouh(a)anl.gov>
Date: Tuesday, June 18, 2024 at 11:05 AM
To: Niyaz Murshed <Niyaz.Murshed(a)arm.com>, discuss(a)mpich.org <discuss(a)mpich.org>
Cc: nd <nd(a)arm.com>
Subject: Re: Mpich over RDMA sample
Niyaz,
FYI -https://urldefense.us/v3/__https://ofiwg.github.io/libfabric/v1.21.0/man/fi_provider.7.html__;!!G_uCfscf7eWS!cn5cXL4QAe2E8ZgxXqVCAl7c3CheHJj61w7oq7Xjax_lCc5i0vxNUMPTygWv434dybYxB1kttyfXaQHq74c$
Hui
________________________________
From: Niyaz Murshed <Niyaz.Murshed(a)arm.com>
Sent: Tuesday, June 18, 2024 11:00 AM
To: Zhou, Hui <zhouh(a)anl.gov>; discuss(a)mpich.org <discuss(a)mpich.org>
Cc: nd <nd(a)arm.com>
Subject: Re: Mpich over RDMA sample
Hi Hui, When you say message semantic, do you mean endpoint type message ? Mpich is running over libfabric. When I run sample application on libfabric, it does have message endpoints over RMA. From: Zhou, Hui <zhouh@ anl. gov> Date: Tuesday,
ZjQcmQRYFpfptBannerStart
This Message Is From an External Sender
This message came from outside your organization.
ZjQcmQRYFpfptBannerEnd
Hi Hui,
When you say message semantic, do you mean endpoint type message ?
Mpich is running over libfabric.
When I run sample application on libfabric, it does have message endpoints over RMA.
From: Zhou, Hui <zhouh(a)anl.gov>
Date: Tuesday, June 18, 2024 at 9:44 AM
To: Niyaz Murshed <Niyaz.Murshed(a)arm.com>, discuss(a)mpich.org <discuss(a)mpich.org>
Cc: nd <nd(a)arm.com>
Subject: Re: Mpich over RDMA sample
The "verbs" provider alone does not provide the message semantics we needed for MPI. "ofi_rxm" is the utility provider that provides the message semantics. "verbs;ofi_rxm" is the provider combination you want.
Hui
________________________________
From: Niyaz Murshed <Niyaz.Murshed(a)arm.com>
Sent: Tuesday, June 18, 2024 9:14 AM
To: Zhou, Hui <zhouh(a)anl.gov>; discuss(a)mpich.org <discuss(a)mpich.org>
Cc: nd <nd(a)arm.com>
Subject: Re: Mpich over RDMA sample
Hi Hui, I have set the env variable to FI_PROVIDER=verbs , after thich I get the below. libfabric provider: verbs;ofi_rxm - IB-0xfe80000000000000 It does not chose just “verbs” , it selects verbs;ofi_rxf I am wanting to select the below: provider:
ZjQcmQRYFpfptBannerStart
This Message Is From an External Sender
This message came from outside your organization.
ZjQcmQRYFpfptBannerEnd
Hi Hui,
I have set the env variable to FI_PROVIDER=verbs , after thich I get the below.
libfabric provider: verbs;ofi_rxm - IB-0xfe80000000000000
It does not chose just “verbs” , it selects verbs;ofi_rxf
I am wanting to select the below:
provider: verbs, score = 0, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200
I would really like to thank you for such fast responses.
From: Zhou, Hui <zhouh(a)anl.gov>
Date: Tuesday, June 18, 2024 at 8:58 AM
To: Niyaz Murshed <Niyaz.Murshed(a)arm.com>, discuss(a)mpich.org <discuss(a)mpich.org>
Cc: nd <nd(a)arm.com>
Subject: Re: Mpich over RDMA sample
Niyaz,
Set environment variable FI_PROVIDER=verbs to select the verbs provider.
Hui
________________________________
From: Niyaz Murshed <Niyaz.Murshed(a)arm.com>
Sent: Monday, June 17, 2024 8:57 PM
To: Zhou, Hui <zhouh(a)anl.gov>; discuss(a)mpich.org <discuss(a)mpich.org>
Cc: nd <nd(a)arm.com>
Subject: Re: Mpich over RDMA sample
Hi Hui, I would still need some assistance in this. I am not able to make mpich chose verbs provider ☹ With the log : provider: verbs, score = 0, pref = 0, FI_SOCKADDR_IN [16] 192. 168. 2. 200 Is my assumption to think the above log represent
ZjQcmQRYFpfptBannerStart
This Message Is From an External Sender
This message came from outside your organization.
ZjQcmQRYFpfptBannerEnd
Hi Hui,
I would still need some assistance in this. I am not able to make mpich chose verbs provider ☹
With the log :
provider: verbs, score = 0, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200
Is my assumption to think the above log represent that verbs provider knows about a path to 192.168.2.200 ?
How does the application hello-world decide which provider to use?
Running on server: 192.168.2.100
root@ampere-altra-2-1# FI_PROVIDER=verbs MPIR_CVAR_DEBUG_SUMMARY=1 mpirun -n 5 -hosts 192.168.2.200,192.168.2.100 /mpich/examples/a.out
==== Various sizes and limits ====
sizeof(MPIDI_per_vci_t): 192
Required minimum FI_VERSION: 0, current version: 10015
provider: verbs, score = 0, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200
provider: verbs, score = 0, pref = 0, FI_SOCKADDR_IN6 [28] fe80::526b:4bff:fefc:1342
provider: verbs, score = 0, pref = 0, FI_SOCKADDR_IB [48]
provider: verbs, score = 0, pref = 0, FI_SOCKADDR_IB [48]
provider: verbs, score = 0, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200
provider: verbs, score = 0, pref = 0, FI_SOCKADDR_IN6 [28] fe80::526b:4bff:fefc:1342
provider: verbs, score = 0, pref = 0, FI_SOCKADDR_IB [48]
provider: verbs, score = 0, pref = 0, FI_SOCKADDR_IB [48]
provider: verbs, score = 0, pref = 0, FI_FORMAT_UNSPEC [32]
provider: verbs, score = 0, pref = 0, FI_FORMAT_UNSPEC [32]
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::526b:4bff:fefc:1342
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48]
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48]
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::526b:4bff:fefc:1342
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48]
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48]
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::526b:4bff:fefc:1342
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48]
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48]
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::526b:4bff:fefc:1342
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48]
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48]
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::526b:4bff:fefc:1342
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48]
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48]
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::526b:4bff:fefc:1342
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48]
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48]
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::526b:4bff:fefc:1342
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48]
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48]
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::526b:4bff:fefc:1342
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48]
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48]
provider: verbs;ofi_rxd, score = 5, pref = -2, FI_FORMAT_UNSPEC [32]
provider: verbs;ofi_rxd, score = 5, pref = -2, FI_FORMAT_UNSPEC [32]
Required minimum FI_VERSION: 10006, current version: 10015
==== Capability set configuration ====
libfabric provider: verbs;ofi_rxm - IB-0xfe80000000000000
MPIDI_OFI_ENABLE_DATA: 0
MPIDI_OFI_ENABLE_AV_TABLE: 1
MPIDI_OFI_ENABLE_SCALABLE_ENDPOINTS: 0
MPIDI_OFI_ENABLE_SHARED_CONTEXTS: 0
MPIDI_OFI_ENABLE_MR_VIRT_ADDRESS: 1
MPIDI_OFI_ENABLE_MR_ALLOCATED: 1
MPIDI_OFI_ENABLE_MR_REGISTER_NULL: 1
MPIDI_OFI_ENABLE_MR_PROV_KEY: 1
MPIDI_OFI_ENABLE_TAGGED: 1
MPIDI_OFI_ENABLE_AM: 1
MPIDI_OFI_ENABLE_RMA: 1
MPIDI_OFI_ENABLE_ATOMICS: 0
MPIDI_OFI_FETCH_ATOMIC_IOVECS: 1
MPIDI_OFI_ENABLE_DATA_AUTO_PROGRESS: 0
MPIDI_OFI_ENABLE_CONTROL_AUTO_PROGRESS: 0
MPIDI_OFI_ENABLE_PT2PT_NOPACK: 1
MPIDI_OFI_ENABLE_TRIGGERED: 0
MPIDI_OFI_ENABLE_HMEM: 0
MPIDI_OFI_NUM_AM_BUFFERS: 8
MPIDI_OFI_NUM_OPTIMIZED_MEMORY_REGIONS: 0
MPIDI_OFI_CONTEXT_BITS: 16
MPIDI_OFI_SOURCE_BITS: 23
MPIDI_OFI_TAG_BITS: 20
MPIDI_OFI_VNI_USE_DOMAIN: 1
MAXIMUM SUPPORTED RANKS: 8388608
MAXIMUM TAG: 1048576
==== Provider global thresholds ====
max_buffered_send: 192
max_buffered_write: 192
max_msg_size: 1073741824
max_order_raw: 1073741824
max_order_war: 0
max_order_waw: 1073741824
tx_iov_limit: 4
rx_iov_limit: 4
rma_iov_limit: 1
max_mr_key_size: 4
==== Various sizes and limits ====
MPIDI_OFI_AM_MSG_HEADER_SIZE: 24
MPIDI_OFI_MAX_AM_HDR_SIZE: 255
sizeof(MPIDI_OFI_am_request_header_t): 416
sizeof(MPIDI_OFI_per_vci_t): 52480
MPIDI_OFI_AM_HDR_POOL_CELL_SIZE: 1024
MPIDI_OFI_DEFAULT_SHORT_SEND_SIZE: 16384
Assertion failed in file src/mpid/ch4/netmod/ofi/init_addrxchg.c at line 151: mapped_table[i] != FI_ADDR_NOTAVAIL
Assertion failed in file src/mpid/ch4/netmod/ofi/init_addrxchg.c at line 151: mapped_table[i] != FI_ADDR_NOTAVAIL
Assertion failed in file src/mpid/ch4/netmod/ofi/init_addrxchg.c at line 151: mapped_table[i] != FI_ADDR_NOTAVAIL
Assertion failed in file src/mpid/ch4/netmod/ofi/init_addrxchg.c at line 151: mapped_table[i] != FI_ADDR_NOTAVAIL
Assertion failed in file src/mpid/ch4/netmod/ofi/init_addrxchg.c at line 151: mapped_table[i] != FI_ADDR_NOTAVAIL
If I do not use the FI_PROVIDER=verbs, it will work over sockets.
root@ampere-altra-2-1: # MPIR_CVAR_DEBUG_SUMMARY=1 mpirun -n 5 -hosts 192.168.2.200,192.168.2.100 /mpich/examples/a.out
==== Various sizes and limits ====
sizeof(MPIDI_per_vci_t): 192
Required minimum FI_VERSION: 0, current version: 10015
provider: verbs, score = 0, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200
provider: verbs, score = 0, pref = 0, FI_SOCKADDR_IN6 [28] fe80::526b:4bff:fefc:1342
provider: verbs, score = 0, pref = 0, FI_SOCKADDR_IB [48]
provider: verbs, score = 0, pref = 0, FI_SOCKADDR_IB [48]
provider: verbs, score = 0, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200
provider: verbs, score = 0, pref = 0, FI_SOCKADDR_IN6 [28] fe80::526b:4bff:fefc:1342
provider: verbs, score = 0, pref = 0, FI_SOCKADDR_IB [48]
provider: verbs, score = 0, pref = 0, FI_SOCKADDR_IB [48]
provider: verbs, score = 0, pref = 0, FI_FORMAT_UNSPEC [32]
provider: verbs, score = 0, pref = 0, FI_FORMAT_UNSPEC [32]
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::526b:4bff:fefc:1342
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48]
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48]
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::526b:4bff:fefc:1342
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48]
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48]
provider: tcp, score = 4, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200
provider: tcp, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::526b:4bff:fefc:1342
provider: tcp, score = 4, pref = 0, FI_SOCKADDR_IN [16] 10.118.91.159
provider: tcp, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::be97:e1ff:fe9d:7caa
provider: tcp, score = 4, pref = 0, FI_SOCKADDR_IN [16] 127.0.0.1
provider: tcp, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] ::1
provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200
provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::526b:4bff:fefc:1342
provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 10.118.91.159
provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::be97:e1ff:fe9d:7caa
provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 127.0.0.1
provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] ::1
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::526b:4bff:fefc:1342
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48]
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48]
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::526b:4bff:fefc:1342
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48]
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48]
provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200
provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::526b:4bff:fefc:1342
provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 10.118.91.159
provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::be97:e1ff:fe9d:7caa
provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 127.0.0.1
provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] ::1
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::526b:4bff:fefc:1342
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48]
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48]
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::526b:4bff:fefc:1342
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48]
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48]
provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200
provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::526b:4bff:fefc:1342
provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 10.118.91.159
provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::be97:e1ff:fe9d:7caa
provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 127.0.0.1
provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] ::1
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::526b:4bff:fefc:1342
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48]
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48]
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::526b:4bff:fefc:1342
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48]
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48]
provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200
provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::526b:4bff:fefc:1342
provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 10.118.91.159
provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::be97:e1ff:fe9d:7caa
provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 127.0.0.1
provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] ::1
provider: verbs;ofi_rxd, score = 5, pref = -2, FI_FORMAT_UNSPEC [32]
provider: verbs;ofi_rxd, score = 5, pref = -2, FI_FORMAT_UNSPEC [32]
provider: udp;ofi_rxd, score = 5, pref = -2, FI_SOCKADDR_IN [16] 192.168.2.200
provider: udp;ofi_rxd, score = 5, pref = -2, FI_SOCKADDR_IN6 [28] fe80::526b:4bff:fefc:1342
provider: udp;ofi_rxd, score = 5, pref = -2, FI_SOCKADDR_IN [16] 10.118.91.159
provider: udp;ofi_rxd, score = 5, pref = -2, FI_SOCKADDR_IN6 [28] fe80::be97:e1ff:fe9d:7caa
provider: udp;ofi_rxd, score = 5, pref = -2, FI_SOCKADDR_IN [16] 127.0.0.1
provider: udp;ofi_rxd, score = 5, pref = -2, FI_SOCKADDR_IN6 [28] ::1
provider: shm, score = 4, pref = -2, FI_ADDR_STR [14] - fi_shm://1177
provider: shm, score = 4, pref = -2, FI_ADDR_STR [14] - fi_shm://1177
provider: udp, score = 0, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200
provider: udp, score = 0, pref = 0, FI_SOCKADDR_IN6 [28] fe80::526b:4bff:fefc:1342
provider: udp, score = 0, pref = 0, FI_SOCKADDR_IN [16] 10.118.91.159
provider: udp, score = 0, pref = 0, FI_SOCKADDR_IN6 [28] fe80::be97:e1ff:fe9d:7caa
provider: udp, score = 0, pref = 0, FI_SOCKADDR_IN [16] 127.0.0.1
provider: udp, score = 0, pref = 0, FI_SOCKADDR_IN6 [28] ::1
provider: tcp, score = 0, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200
provider: tcp, score = 0, pref = 0, FI_SOCKADDR_IN6 [28] fe80::526b:4bff:fefc:1342
provider: tcp, score = 0, pref = 0, FI_SOCKADDR_IN [16] 10.118.91.159
provider: tcp, score = 0, pref = 0, FI_SOCKADDR_IN6 [28] fe80::be97:e1ff:fe9d:7caa
provider: tcp, score = 0, pref = 0, FI_SOCKADDR_IN [16] 127.0.0.1
provider: tcp, score = 0, pref = 0, FI_SOCKADDR_IN6 [28] ::1
provider: tcp, score = 4, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200
provider: tcp, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::526b:4bff:fefc:1342
provider: tcp, score = 4, pref = 0, FI_SOCKADDR_IN [16] 10.118.91.159
provider: tcp, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::be97:e1ff:fe9d:7caa
provider: tcp, score = 4, pref = 0, FI_SOCKADDR_IN [16] 127.0.0.1
provider: tcp, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] ::1
provider: sockets, score = 3, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200
provider: sockets, score = 3, pref = 0, FI_SOCKADDR_IN6 [28] fe80::526b:4bff:fefc:1342
provider: sockets, score = 3, pref = 0, FI_SOCKADDR_IN [16] 10.118.91.159
provider: sockets, score = 3, pref = 0, FI_SOCKADDR_IN6 [28] fe80::be97:e1ff:fe9d:7caa
provider: sockets, score = 3, pref = 0, FI_SOCKADDR_IN [16] 127.0.0.1
provider: sockets, score = 3, pref = 0, FI_SOCKADDR_IN6 [28] ::1
provider: sockets, score = 5, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200
provider: sockets, score = 5, pref = 0, FI_SOCKADDR_IN6 [28] fe80::526b:4bff:fefc:1342
provider: sockets, score = 5, pref = 0, FI_SOCKADDR_IN [16] 10.118.91.159
provider: sockets, score = 5, pref = 0, FI_SOCKADDR_IN6 [28] fe80::be97:e1ff:fe9d:7caa
provider: sockets, score = 5, pref = 0, FI_SOCKADDR_IN [16] 127.0.0.1
provider: sockets, score = 5, pref = 0, FI_SOCKADDR_IN6 [28] ::1
provider: sockets, score = 5, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200
provider: sockets, score = 5, pref = 0, FI_SOCKADDR_IN6 [28] fe80::526b:4bff:fefc:1342
provider: sockets, score = 5, pref = 0, FI_SOCKADDR_IN [16] 10.118.91.159
provider: sockets, score = 5, pref = 0, FI_SOCKADDR_IN6 [28] fe80::be97:e1ff:fe9d:7caa
provider: sockets, score = 5, pref = 0, FI_SOCKADDR_IN [16] 127.0.0.1
provider: sockets, score = 5, pref = 0, FI_SOCKADDR_IN6 [28] ::1
provider: sm2, score = 3, pref = 0, FI_ADDR_STR [13] - fi_sm2://1177
provider: sm2, score = 3, pref = 0, FI_ADDR_STR [13] - fi_sm2://1177
Required minimum FI_VERSION: 10005, current version: 10015
==== Capability set configuration ====
libfabric provider: sockets - 192.168.2.0/24
MPIDI_OFI_ENABLE_DATA: 1
MPIDI_OFI_ENABLE_AV_TABLE: 1
MPIDI_OFI_ENABLE_SCALABLE_ENDPOINTS: 1
MPIDI_OFI_ENABLE_SHARED_CONTEXTS: 0
MPIDI_OFI_ENABLE_MR_VIRT_ADDRESS: 0
MPIDI_OFI_ENABLE_MR_ALLOCATED: 0
MPIDI_OFI_ENABLE_MR_REGISTER_NULL: 1
MPIDI_OFI_ENABLE_MR_PROV_KEY: 0
MPIDI_OFI_ENABLE_TAGGED: 1
MPIDI_OFI_ENABLE_AM: 1
MPIDI_OFI_ENABLE_RMA: 1
MPIDI_OFI_ENABLE_ATOMICS: 1
MPIDI_OFI_FETCH_ATOMIC_IOVECS: 1
MPIDI_OFI_ENABLE_DATA_AUTO_PROGRESS: 0
MPIDI_OFI_ENABLE_CONTROL_AUTO_PROGRESS: 0
MPIDI_OFI_ENABLE_PT2PT_NOPACK: 1
MPIDI_OFI_ENABLE_TRIGGERED: 0
MPIDI_OFI_ENABLE_HMEM: 0
MPIDI_OFI_NUM_AM_BUFFERS: 8
MPIDI_OFI_NUM_OPTIMIZED_MEMORY_REGIONS: 0
MPIDI_OFI_CONTEXT_BITS: 20
MPIDI_OFI_SOURCE_BITS: 0
MPIDI_OFI_TAG_BITS: 31
MPIDI_OFI_VNI_USE_DOMAIN: 1
MAXIMUM SUPPORTED RANKS: 4294967296
MAXIMUM TAG: 2147483648
==== Provider global thresholds ====
max_buffered_send: 255
max_buffered_write: 255
max_msg_size: 9223372036854775807
max_order_raw: -1
max_order_war: -1
max_order_waw: -1
tx_iov_limit: 8
rx_iov_limit: 8
rma_iov_limit: 8
max_mr_key_size: 8
==== Various sizes and limits ====
MPIDI_OFI_AM_MSG_HEADER_SIZE: 24
MPIDI_OFI_MAX_AM_HDR_SIZE: 255
sizeof(MPIDI_OFI_am_request_header_t): 416
sizeof(MPIDI_OFI_per_vci_t): 52480
MPIDI_OFI_AM_HDR_POOL_CELL_SIZE: 1024
MPIDI_OFI_DEFAULT_SHORT_SEND_SIZE: 16384
==== OFI dynamic settings ====
num_vcis: 1
num_nics: 1
======================================
error checking : enabled
QMPI : disabled
debugger support : disabled
thread level : MPI_THREAD_SINGLE
thread CS : per-vci
threadcomm : enabled
==== data structure summary ====
sizeof(MPIR_Comm): 1808
sizeof(MPIR_Request): 512
sizeof(MPIR_Datatype): 280
================================
Hello world from process 0 of 5
Hello world from process 2 of 5
Hello world from process 1 of 5
Hello world from process 4 of 5
Hello world from process 3 of 5
From: Zhou, Hui <zhouh(a)anl.gov>
Date: Wednesday, June 12, 2024 at 7:22 PM
To: Niyaz Murshed <Niyaz.Murshed(a)arm.com>, discuss(a)mpich.org <discuss(a)mpich.org>
Subject: Re: Mpich over RDMA sample
It’s possible that the native path for RMA on verbs are turned off due to verbs does not meet MPI’s full atomic ordering semantics. We’ll check and follow up.
--
Hui Zhou
From: Niyaz Murshed <Niyaz.Murshed(a)arm.com>
Date: Wednesday, June 12, 2024 at 3:59 PM
To: Zhou, Hui <zhouh(a)anl.gov>, discuss(a)mpich.org <discuss(a)mpich.org>
Cc: nd <nd(a)arm.com>
Subject: Re: Mpich over RDMA sample
I think image is not gone : https: //ibb. co/RDGnJfy From: Niyaz Murshed <Niyaz. Murshed@ arm. com> Date: Wednesday, June 12, 2024 at 3: 56 PM To: Zhou, Hui <zhouh@ anl. gov>, discuss@ mpich. org <discuss@ mpich. org> Cc: nd <nd@ arm. com>
ZjQcmQRYFpfptBannerStart
This Message Is From an External Sender
This message came from outside your organization.
ZjQcmQRYFpfptBannerEnd
I think image is not gone : https://urldefense.us/v3/__https://ibb.co/RDGnJfy__;!!G_uCfscf7eWS!cn5cXL4Q… <https://urldefense.us/v3/__https:/ibb.co/RDGnJfy__;!!G_uCfscf7eWS!YH6L3OHfP…>
From: Niyaz Murshed <Niyaz.Murshed(a)arm.com>
Date: Wednesday, June 12, 2024 at 3:56 PM
To: Zhou, Hui <zhouh(a)anl.gov>, discuss(a)mpich.org <discuss(a)mpich.org>
Cc: nd <nd(a)arm.com>
Subject: Re: Mpich over RDMA sample
Thank you Hui for the reply.
I was expecting to see RoCE messages in the packet capture.
When I use verbs on libfabric sample tests, I see the RoCE messages as below :
[cid:[email protected]]
From: Zhou, Hui <zhouh(a)anl.gov>
Date: Wednesday, June 12, 2024 at 3:41 PM
To: Niyaz Murshed <Niyaz.Murshed(a)arm.com>, discuss(a)mpich.org <discuss(a)mpich.org>
Cc: nd <nd(a)arm.com>
Subject: Re: Mpich over RDMA sample
Niyaz,
ofi_rxm:verbs is the correct provider. The ofi_rxm provides the verbs provider additional messaging semantics that are needed to be used in MPI.
--
Hui Zhou
From: Niyaz Murshed <Niyaz.Murshed(a)arm.com>
Date: Wednesday, June 12, 2024 at 2:54 PM
To: Zhou, Hui <zhouh(a)anl.gov>, discuss(a)mpich.org <discuss(a)mpich.org>
Cc: nd <nd(a)arm.com>
Subject: Re: Mpich over RDMA sample
Thank you for the reply. I did that; however, it selects ofi_rxm: verbs libfabric: 412098: 1718221892: : ofi_rxm: av: ofi_av_insert_addr(): 313<info> fi_addr: 1 libfabric: 412099: 1718221892: : ofi_rxm: av: ofi_av_insert_addr(): 313<info> fi_addr:
ZjQcmQRYFpfptBannerStart
This Message Is From an External Sender
This message came from outside your organization.
ZjQcmQRYFpfptBannerEnd
Thank you for the reply.
I did that; however, it selects ofi_rxm:verbs
libfabric:412098:1718221892::ofi_rxm:av:ofi_av_insert_addr():313<info> fi_addr: 1
libfabric:412099:1718221892::ofi_rxm:av:ofi_av_insert_addr():313<info> fi_addr: 1
options:
backend: cpu
iters: 16
warmup_iters: 16
cache: 1
min_elem_count: 1
max_elem_count: 1
elem_counts: [1]
validate: last
window_size: 64
#------------------------------------------------------------
# Benchmarking: Bandwidth
# #processes: 2
#------------------------------------------------------------
#bytes #repetitions Mbytes/sec
4 16 0.02
# All done
libfabric:412098:1718221892::ofi_rxm:ep_ctrl:rxm_stop_listen():864<info> stopping CM thread
libfabric:412099:1718221892::ofi_rxm:ep_ctrl:rxm_stop_listen():864<info> stopping CM thread
As per https://urldefense.us/v3/__https://www.intel.com/content/www/us/en/docs/mpi… <https://urldefense.us/v3/__https:/www.intel.com/content/www/us/en/docs/mpi-…> , we need to add FI_PROVIDER=^ofi_rxm , but if do that, it moves back to sockets provider.
Is here a way to combine ^ofi_rxm and verbs
From: Zhou, Hui <zhouh(a)anl.gov>
Date: Wednesday, June 12, 2024 at 2:41 PM
To: Niyaz Murshed <Niyaz.Murshed(a)arm.com>, discuss(a)mpich.org <discuss(a)mpich.org>
Cc: nd <nd(a)arm.com>
Subject: Re: Mpich over RDMA sample
Niyaz,
All you need to do is to set an environment variable `FI_PROVIDER=verbs`.
--
Hui Zhou
From: Niyaz Murshed <Niyaz.Murshed(a)arm.com>
Date: Wednesday, June 12, 2024 at 1:23 PM
To: Zhou, Hui <zhouh(a)anl.gov>, discuss(a)mpich.org <discuss(a)mpich.org>
Cc: nd <nd(a)arm.com>
Subject: Re: Mpich over RDMA sample
When testing with Libfabric, verbs provider is selected. I did have to use “-e msg -d mlx5_1” so that it selects verbs. I was checking if there is anything like that for mpich sample tests. Else might need to do some hack in the code to force
ZjQcmQRYFpfptBannerStart
This Message Is From an External Sender
This message came from outside your organization.
ZjQcmQRYFpfptBannerEnd
When testing with Libfabric, verbs provider is selected. I did have to use “-e msg -d mlx5_1” so that it selects verbs.
I was checking if there is anything like that for mpich sample tests. Else might need to do some hack in the code to force the selection of verbs.
From: Zhou, Hui <zhouh(a)anl.gov>
Date: Wednesday, June 12, 2024 at 1:15 PM
To: discuss(a)mpich.org <discuss(a)mpich.org>
Cc: Niyaz Murshed <Niyaz.Murshed(a)arm.com>, nd <nd(a)arm.com>
Subject: Re: Mpich over RDMA sample
Libfabric support multiple providers. Sounds like it was selecting the sockets or tcp provider rather than a provider that support RoCE. I am not exactly sure whether the verbs provider will do that. If you can confirm the provider using libfabric tests, then you can try forcing MPICH to use that provider by setting the FI_PROVIDER environment variable.
--
Hui Zhou
From: Niyaz Murshed via discuss <discuss(a)mpich.org>
Date: Wednesday, June 12, 2024 at 9:03 AM
To: discuss(a)mpich.org <discuss(a)mpich.org>
Cc: Niyaz Murshed <Niyaz.Murshed(a)arm.com>, nd <nd(a)arm.com>
Subject: [mpich-discuss] Mpich over RDMA sample
Hello, I am trying to learn about MPICH and its performance over RDMA. I am using libfabric and installed mpich using the below configure. ./configure --prefix=/opt/mpich/ --with-ofi=/opt/libfabric/ When I run any applications between 2 directly
ZjQcmQRYFpfptBannerStart
This Message Is From an External Sender
This message came from outside your organization.
ZjQcmQRYFpfptBannerEnd
Hello,
I am trying to learn about MPICH and its performance over RDMA.
I am using libfabric and installed mpich using the below configure.
./configure --prefix=/opt/mpich/ --with-ofi=/opt/libfabric/
When I run any applications between 2 directly connected servers having Mellanox NICs, I see that communication is happening over tcp and not over RoCE.
Is there any way to test commination over RoCE ?
For eg. I was able to test it for libfabric using the below sample that comes along with libfabric to test RMA.
Is there something similar for MPICH ? or use the current sample to use RoCE by some parameter?
Server :
fi_rma_bw -s 192.168.1.100 -e msg -d mlx5_1 -S 1024 -I 1
Client :
fi_rma_bw -s 192.168.1.200 -e msg -d mlx5_3 192.168.1.100 -S 1024 -I 1
Regards,
Niyaz
1
0
Niyaz,
FYI -https://urldefense.us/v3/__https://ofiwg.github.io/libfabric/v1.21.0/man/fi_provider.7.html__;!!G_uCfscf7eWS!Z-ju_mFvcfItasVHydCwS8PBDxrKUNt0Qhpc_A-fBZVtuiGT0SEErcNkNECkd0UPEwT9GtRQR0Rj$
Hui
________________________________
From: Niyaz Murshed <Niyaz.Murshed(a)arm.com>
Sent: Tuesday, June 18, 2024 11:00 AM
To: Zhou, Hui <zhouh(a)anl.gov>; discuss(a)mpich.org <discuss(a)mpich.org>
Cc: nd <nd(a)arm.com>
Subject: Re: Mpich over RDMA sample
Hi Hui, When you say message semantic, do you mean endpoint type message ? Mpich is running over libfabric. When I run sample application on libfabric, it does have message endpoints over RMA. From: Zhou, Hui <zhouh@ anl. gov> Date: Tuesday,
ZjQcmQRYFpfptBannerStart
This Message Is From an External Sender
This message came from outside your organization.
ZjQcmQRYFpfptBannerEnd
Hi Hui,
When you say message semantic, do you mean endpoint type message ?
Mpich is running over libfabric.
When I run sample application on libfabric, it does have message endpoints over RMA.
From: Zhou, Hui <zhouh(a)anl.gov>
Date: Tuesday, June 18, 2024 at 9:44 AM
To: Niyaz Murshed <Niyaz.Murshed(a)arm.com>, discuss(a)mpich.org <discuss(a)mpich.org>
Cc: nd <nd(a)arm.com>
Subject: Re: Mpich over RDMA sample
The "verbs" provider alone does not provide the message semantics we needed for MPI. "ofi_rxm" is the utility provider that provides the message semantics. "verbs;ofi_rxm" is the provider combination you want.
Hui
________________________________
From: Niyaz Murshed <Niyaz.Murshed(a)arm.com>
Sent: Tuesday, June 18, 2024 9:14 AM
To: Zhou, Hui <zhouh(a)anl.gov>; discuss(a)mpich.org <discuss(a)mpich.org>
Cc: nd <nd(a)arm.com>
Subject: Re: Mpich over RDMA sample
Hi Hui, I have set the env variable to FI_PROVIDER=verbs , after thich I get the below. libfabric provider: verbs;ofi_rxm - IB-0xfe80000000000000 It does not chose just “verbs” , it selects verbs;ofi_rxf I am wanting to select the below: provider:
ZjQcmQRYFpfptBannerStart
This Message Is From an External Sender
This message came from outside your organization.
ZjQcmQRYFpfptBannerEnd
Hi Hui,
I have set the env variable to FI_PROVIDER=verbs , after thich I get the below.
libfabric provider: verbs;ofi_rxm - IB-0xfe80000000000000
It does not chose just “verbs” , it selects verbs;ofi_rxf
I am wanting to select the below:
provider: verbs, score = 0, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200
I would really like to thank you for such fast responses.
From: Zhou, Hui <zhouh(a)anl.gov>
Date: Tuesday, June 18, 2024 at 8:58 AM
To: Niyaz Murshed <Niyaz.Murshed(a)arm.com>, discuss(a)mpich.org <discuss(a)mpich.org>
Cc: nd <nd(a)arm.com>
Subject: Re: Mpich over RDMA sample
Niyaz,
Set environment variable FI_PROVIDER=verbs to select the verbs provider.
Hui
________________________________
From: Niyaz Murshed <Niyaz.Murshed(a)arm.com>
Sent: Monday, June 17, 2024 8:57 PM
To: Zhou, Hui <zhouh(a)anl.gov>; discuss(a)mpich.org <discuss(a)mpich.org>
Cc: nd <nd(a)arm.com>
Subject: Re: Mpich over RDMA sample
Hi Hui, I would still need some assistance in this. I am not able to make mpich chose verbs provider ☹ With the log : provider: verbs, score = 0, pref = 0, FI_SOCKADDR_IN [16] 192. 168. 2. 200 Is my assumption to think the above log represent
ZjQcmQRYFpfptBannerStart
This Message Is From an External Sender
This message came from outside your organization.
ZjQcmQRYFpfptBannerEnd
Hi Hui,
I would still need some assistance in this. I am not able to make mpich chose verbs provider ☹
With the log :
provider: verbs, score = 0, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200
Is my assumption to think the above log represent that verbs provider knows about a path to 192.168.2.200 ?
How does the application hello-world decide which provider to use?
Running on server: 192.168.2.100
root@ampere-altra-2-1# FI_PROVIDER=verbs MPIR_CVAR_DEBUG_SUMMARY=1 mpirun -n 5 -hosts 192.168.2.200,192.168.2.100 /mpich/examples/a.out
==== Various sizes and limits ====
sizeof(MPIDI_per_vci_t): 192
Required minimum FI_VERSION: 0, current version: 10015
provider: verbs, score = 0, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200
provider: verbs, score = 0, pref = 0, FI_SOCKADDR_IN6 [28] fe80::526b:4bff:fefc:1342
provider: verbs, score = 0, pref = 0, FI_SOCKADDR_IB [48]
provider: verbs, score = 0, pref = 0, FI_SOCKADDR_IB [48]
provider: verbs, score = 0, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200
provider: verbs, score = 0, pref = 0, FI_SOCKADDR_IN6 [28] fe80::526b:4bff:fefc:1342
provider: verbs, score = 0, pref = 0, FI_SOCKADDR_IB [48]
provider: verbs, score = 0, pref = 0, FI_SOCKADDR_IB [48]
provider: verbs, score = 0, pref = 0, FI_FORMAT_UNSPEC [32]
provider: verbs, score = 0, pref = 0, FI_FORMAT_UNSPEC [32]
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::526b:4bff:fefc:1342
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48]
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48]
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::526b:4bff:fefc:1342
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48]
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48]
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::526b:4bff:fefc:1342
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48]
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48]
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::526b:4bff:fefc:1342
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48]
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48]
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::526b:4bff:fefc:1342
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48]
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48]
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::526b:4bff:fefc:1342
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48]
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48]
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::526b:4bff:fefc:1342
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48]
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48]
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::526b:4bff:fefc:1342
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48]
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48]
provider: verbs;ofi_rxd, score = 5, pref = -2, FI_FORMAT_UNSPEC [32]
provider: verbs;ofi_rxd, score = 5, pref = -2, FI_FORMAT_UNSPEC [32]
Required minimum FI_VERSION: 10006, current version: 10015
==== Capability set configuration ====
libfabric provider: verbs;ofi_rxm - IB-0xfe80000000000000
MPIDI_OFI_ENABLE_DATA: 0
MPIDI_OFI_ENABLE_AV_TABLE: 1
MPIDI_OFI_ENABLE_SCALABLE_ENDPOINTS: 0
MPIDI_OFI_ENABLE_SHARED_CONTEXTS: 0
MPIDI_OFI_ENABLE_MR_VIRT_ADDRESS: 1
MPIDI_OFI_ENABLE_MR_ALLOCATED: 1
MPIDI_OFI_ENABLE_MR_REGISTER_NULL: 1
MPIDI_OFI_ENABLE_MR_PROV_KEY: 1
MPIDI_OFI_ENABLE_TAGGED: 1
MPIDI_OFI_ENABLE_AM: 1
MPIDI_OFI_ENABLE_RMA: 1
MPIDI_OFI_ENABLE_ATOMICS: 0
MPIDI_OFI_FETCH_ATOMIC_IOVECS: 1
MPIDI_OFI_ENABLE_DATA_AUTO_PROGRESS: 0
MPIDI_OFI_ENABLE_CONTROL_AUTO_PROGRESS: 0
MPIDI_OFI_ENABLE_PT2PT_NOPACK: 1
MPIDI_OFI_ENABLE_TRIGGERED: 0
MPIDI_OFI_ENABLE_HMEM: 0
MPIDI_OFI_NUM_AM_BUFFERS: 8
MPIDI_OFI_NUM_OPTIMIZED_MEMORY_REGIONS: 0
MPIDI_OFI_CONTEXT_BITS: 16
MPIDI_OFI_SOURCE_BITS: 23
MPIDI_OFI_TAG_BITS: 20
MPIDI_OFI_VNI_USE_DOMAIN: 1
MAXIMUM SUPPORTED RANKS: 8388608
MAXIMUM TAG: 1048576
==== Provider global thresholds ====
max_buffered_send: 192
max_buffered_write: 192
max_msg_size: 1073741824
max_order_raw: 1073741824
max_order_war: 0
max_order_waw: 1073741824
tx_iov_limit: 4
rx_iov_limit: 4
rma_iov_limit: 1
max_mr_key_size: 4
==== Various sizes and limits ====
MPIDI_OFI_AM_MSG_HEADER_SIZE: 24
MPIDI_OFI_MAX_AM_HDR_SIZE: 255
sizeof(MPIDI_OFI_am_request_header_t): 416
sizeof(MPIDI_OFI_per_vci_t): 52480
MPIDI_OFI_AM_HDR_POOL_CELL_SIZE: 1024
MPIDI_OFI_DEFAULT_SHORT_SEND_SIZE: 16384
Assertion failed in file src/mpid/ch4/netmod/ofi/init_addrxchg.c at line 151: mapped_table[i] != FI_ADDR_NOTAVAIL
Assertion failed in file src/mpid/ch4/netmod/ofi/init_addrxchg.c at line 151: mapped_table[i] != FI_ADDR_NOTAVAIL
Assertion failed in file src/mpid/ch4/netmod/ofi/init_addrxchg.c at line 151: mapped_table[i] != FI_ADDR_NOTAVAIL
Assertion failed in file src/mpid/ch4/netmod/ofi/init_addrxchg.c at line 151: mapped_table[i] != FI_ADDR_NOTAVAIL
Assertion failed in file src/mpid/ch4/netmod/ofi/init_addrxchg.c at line 151: mapped_table[i] != FI_ADDR_NOTAVAIL
If I do not use the FI_PROVIDER=verbs, it will work over sockets.
root@ampere-altra-2-1: # MPIR_CVAR_DEBUG_SUMMARY=1 mpirun -n 5 -hosts 192.168.2.200,192.168.2.100 /mpich/examples/a.out
==== Various sizes and limits ====
sizeof(MPIDI_per_vci_t): 192
Required minimum FI_VERSION: 0, current version: 10015
provider: verbs, score = 0, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200
provider: verbs, score = 0, pref = 0, FI_SOCKADDR_IN6 [28] fe80::526b:4bff:fefc:1342
provider: verbs, score = 0, pref = 0, FI_SOCKADDR_IB [48]
provider: verbs, score = 0, pref = 0, FI_SOCKADDR_IB [48]
provider: verbs, score = 0, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200
provider: verbs, score = 0, pref = 0, FI_SOCKADDR_IN6 [28] fe80::526b:4bff:fefc:1342
provider: verbs, score = 0, pref = 0, FI_SOCKADDR_IB [48]
provider: verbs, score = 0, pref = 0, FI_SOCKADDR_IB [48]
provider: verbs, score = 0, pref = 0, FI_FORMAT_UNSPEC [32]
provider: verbs, score = 0, pref = 0, FI_FORMAT_UNSPEC [32]
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::526b:4bff:fefc:1342
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48]
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48]
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::526b:4bff:fefc:1342
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48]
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48]
provider: tcp, score = 4, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200
provider: tcp, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::526b:4bff:fefc:1342
provider: tcp, score = 4, pref = 0, FI_SOCKADDR_IN [16] 10.118.91.159
provider: tcp, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::be97:e1ff:fe9d:7caa
provider: tcp, score = 4, pref = 0, FI_SOCKADDR_IN [16] 127.0.0.1
provider: tcp, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] ::1
provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200
provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::526b:4bff:fefc:1342
provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 10.118.91.159
provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::be97:e1ff:fe9d:7caa
provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 127.0.0.1
provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] ::1
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::526b:4bff:fefc:1342
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48]
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48]
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::526b:4bff:fefc:1342
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48]
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48]
provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200
provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::526b:4bff:fefc:1342
provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 10.118.91.159
provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::be97:e1ff:fe9d:7caa
provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 127.0.0.1
provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] ::1
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::526b:4bff:fefc:1342
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48]
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48]
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::526b:4bff:fefc:1342
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48]
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48]
provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200
provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::526b:4bff:fefc:1342
provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 10.118.91.159
provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::be97:e1ff:fe9d:7caa
provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 127.0.0.1
provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] ::1
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::526b:4bff:fefc:1342
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48]
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48]
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::526b:4bff:fefc:1342
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48]
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48]
provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200
provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::526b:4bff:fefc:1342
provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 10.118.91.159
provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::be97:e1ff:fe9d:7caa
provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 127.0.0.1
provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] ::1
provider: verbs;ofi_rxd, score = 5, pref = -2, FI_FORMAT_UNSPEC [32]
provider: verbs;ofi_rxd, score = 5, pref = -2, FI_FORMAT_UNSPEC [32]
provider: udp;ofi_rxd, score = 5, pref = -2, FI_SOCKADDR_IN [16] 192.168.2.200
provider: udp;ofi_rxd, score = 5, pref = -2, FI_SOCKADDR_IN6 [28] fe80::526b:4bff:fefc:1342
provider: udp;ofi_rxd, score = 5, pref = -2, FI_SOCKADDR_IN [16] 10.118.91.159
provider: udp;ofi_rxd, score = 5, pref = -2, FI_SOCKADDR_IN6 [28] fe80::be97:e1ff:fe9d:7caa
provider: udp;ofi_rxd, score = 5, pref = -2, FI_SOCKADDR_IN [16] 127.0.0.1
provider: udp;ofi_rxd, score = 5, pref = -2, FI_SOCKADDR_IN6 [28] ::1
provider: shm, score = 4, pref = -2, FI_ADDR_STR [14] - fi_shm://1177
provider: shm, score = 4, pref = -2, FI_ADDR_STR [14] - fi_shm://1177
provider: udp, score = 0, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200
provider: udp, score = 0, pref = 0, FI_SOCKADDR_IN6 [28] fe80::526b:4bff:fefc:1342
provider: udp, score = 0, pref = 0, FI_SOCKADDR_IN [16] 10.118.91.159
provider: udp, score = 0, pref = 0, FI_SOCKADDR_IN6 [28] fe80::be97:e1ff:fe9d:7caa
provider: udp, score = 0, pref = 0, FI_SOCKADDR_IN [16] 127.0.0.1
provider: udp, score = 0, pref = 0, FI_SOCKADDR_IN6 [28] ::1
provider: tcp, score = 0, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200
provider: tcp, score = 0, pref = 0, FI_SOCKADDR_IN6 [28] fe80::526b:4bff:fefc:1342
provider: tcp, score = 0, pref = 0, FI_SOCKADDR_IN [16] 10.118.91.159
provider: tcp, score = 0, pref = 0, FI_SOCKADDR_IN6 [28] fe80::be97:e1ff:fe9d:7caa
provider: tcp, score = 0, pref = 0, FI_SOCKADDR_IN [16] 127.0.0.1
provider: tcp, score = 0, pref = 0, FI_SOCKADDR_IN6 [28] ::1
provider: tcp, score = 4, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200
provider: tcp, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::526b:4bff:fefc:1342
provider: tcp, score = 4, pref = 0, FI_SOCKADDR_IN [16] 10.118.91.159
provider: tcp, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::be97:e1ff:fe9d:7caa
provider: tcp, score = 4, pref = 0, FI_SOCKADDR_IN [16] 127.0.0.1
provider: tcp, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] ::1
provider: sockets, score = 3, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200
provider: sockets, score = 3, pref = 0, FI_SOCKADDR_IN6 [28] fe80::526b:4bff:fefc:1342
provider: sockets, score = 3, pref = 0, FI_SOCKADDR_IN [16] 10.118.91.159
provider: sockets, score = 3, pref = 0, FI_SOCKADDR_IN6 [28] fe80::be97:e1ff:fe9d:7caa
provider: sockets, score = 3, pref = 0, FI_SOCKADDR_IN [16] 127.0.0.1
provider: sockets, score = 3, pref = 0, FI_SOCKADDR_IN6 [28] ::1
provider: sockets, score = 5, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200
provider: sockets, score = 5, pref = 0, FI_SOCKADDR_IN6 [28] fe80::526b:4bff:fefc:1342
provider: sockets, score = 5, pref = 0, FI_SOCKADDR_IN [16] 10.118.91.159
provider: sockets, score = 5, pref = 0, FI_SOCKADDR_IN6 [28] fe80::be97:e1ff:fe9d:7caa
provider: sockets, score = 5, pref = 0, FI_SOCKADDR_IN [16] 127.0.0.1
provider: sockets, score = 5, pref = 0, FI_SOCKADDR_IN6 [28] ::1
provider: sockets, score = 5, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200
provider: sockets, score = 5, pref = 0, FI_SOCKADDR_IN6 [28] fe80::526b:4bff:fefc:1342
provider: sockets, score = 5, pref = 0, FI_SOCKADDR_IN [16] 10.118.91.159
provider: sockets, score = 5, pref = 0, FI_SOCKADDR_IN6 [28] fe80::be97:e1ff:fe9d:7caa
provider: sockets, score = 5, pref = 0, FI_SOCKADDR_IN [16] 127.0.0.1
provider: sockets, score = 5, pref = 0, FI_SOCKADDR_IN6 [28] ::1
provider: sm2, score = 3, pref = 0, FI_ADDR_STR [13] - fi_sm2://1177
provider: sm2, score = 3, pref = 0, FI_ADDR_STR [13] - fi_sm2://1177
Required minimum FI_VERSION: 10005, current version: 10015
==== Capability set configuration ====
libfabric provider: sockets - 192.168.2.0/24
MPIDI_OFI_ENABLE_DATA: 1
MPIDI_OFI_ENABLE_AV_TABLE: 1
MPIDI_OFI_ENABLE_SCALABLE_ENDPOINTS: 1
MPIDI_OFI_ENABLE_SHARED_CONTEXTS: 0
MPIDI_OFI_ENABLE_MR_VIRT_ADDRESS: 0
MPIDI_OFI_ENABLE_MR_ALLOCATED: 0
MPIDI_OFI_ENABLE_MR_REGISTER_NULL: 1
MPIDI_OFI_ENABLE_MR_PROV_KEY: 0
MPIDI_OFI_ENABLE_TAGGED: 1
MPIDI_OFI_ENABLE_AM: 1
MPIDI_OFI_ENABLE_RMA: 1
MPIDI_OFI_ENABLE_ATOMICS: 1
MPIDI_OFI_FETCH_ATOMIC_IOVECS: 1
MPIDI_OFI_ENABLE_DATA_AUTO_PROGRESS: 0
MPIDI_OFI_ENABLE_CONTROL_AUTO_PROGRESS: 0
MPIDI_OFI_ENABLE_PT2PT_NOPACK: 1
MPIDI_OFI_ENABLE_TRIGGERED: 0
MPIDI_OFI_ENABLE_HMEM: 0
MPIDI_OFI_NUM_AM_BUFFERS: 8
MPIDI_OFI_NUM_OPTIMIZED_MEMORY_REGIONS: 0
MPIDI_OFI_CONTEXT_BITS: 20
MPIDI_OFI_SOURCE_BITS: 0
MPIDI_OFI_TAG_BITS: 31
MPIDI_OFI_VNI_USE_DOMAIN: 1
MAXIMUM SUPPORTED RANKS: 4294967296
MAXIMUM TAG: 2147483648
==== Provider global thresholds ====
max_buffered_send: 255
max_buffered_write: 255
max_msg_size: 9223372036854775807
max_order_raw: -1
max_order_war: -1
max_order_waw: -1
tx_iov_limit: 8
rx_iov_limit: 8
rma_iov_limit: 8
max_mr_key_size: 8
==== Various sizes and limits ====
MPIDI_OFI_AM_MSG_HEADER_SIZE: 24
MPIDI_OFI_MAX_AM_HDR_SIZE: 255
sizeof(MPIDI_OFI_am_request_header_t): 416
sizeof(MPIDI_OFI_per_vci_t): 52480
MPIDI_OFI_AM_HDR_POOL_CELL_SIZE: 1024
MPIDI_OFI_DEFAULT_SHORT_SEND_SIZE: 16384
==== OFI dynamic settings ====
num_vcis: 1
num_nics: 1
======================================
error checking : enabled
QMPI : disabled
debugger support : disabled
thread level : MPI_THREAD_SINGLE
thread CS : per-vci
threadcomm : enabled
==== data structure summary ====
sizeof(MPIR_Comm): 1808
sizeof(MPIR_Request): 512
sizeof(MPIR_Datatype): 280
================================
Hello world from process 0 of 5
Hello world from process 2 of 5
Hello world from process 1 of 5
Hello world from process 4 of 5
Hello world from process 3 of 5
From: Zhou, Hui <zhouh(a)anl.gov>
Date: Wednesday, June 12, 2024 at 7:22 PM
To: Niyaz Murshed <Niyaz.Murshed(a)arm.com>, discuss(a)mpich.org <discuss(a)mpich.org>
Subject: Re: Mpich over RDMA sample
It’s possible that the native path for RMA on verbs are turned off due to verbs does not meet MPI’s full atomic ordering semantics. We’ll check and follow up.
--
Hui Zhou
From: Niyaz Murshed <Niyaz.Murshed(a)arm.com>
Date: Wednesday, June 12, 2024 at 3:59 PM
To: Zhou, Hui <zhouh(a)anl.gov>, discuss(a)mpich.org <discuss(a)mpich.org>
Cc: nd <nd(a)arm.com>
Subject: Re: Mpich over RDMA sample
I think image is not gone : https: //ibb. co/RDGnJfy From: Niyaz Murshed <Niyaz. Murshed@ arm. com> Date: Wednesday, June 12, 2024 at 3: 56 PM To: Zhou, Hui <zhouh@ anl. gov>, discuss@ mpich. org <discuss@ mpich. org> Cc: nd <nd@ arm. com>
ZjQcmQRYFpfptBannerStart
This Message Is From an External Sender
This message came from outside your organization.
ZjQcmQRYFpfptBannerEnd
I think image is not gone : https://urldefense.us/v3/__https://ibb.co/RDGnJfy__;!!G_uCfscf7eWS!Z-ju_mFv… <https://urldefense.us/v3/__https:/ibb.co/RDGnJfy__;!!G_uCfscf7eWS!YH6L3OHfP…>
From: Niyaz Murshed <Niyaz.Murshed(a)arm.com>
Date: Wednesday, June 12, 2024 at 3:56 PM
To: Zhou, Hui <zhouh(a)anl.gov>, discuss(a)mpich.org <discuss(a)mpich.org>
Cc: nd <nd(a)arm.com>
Subject: Re: Mpich over RDMA sample
Thank you Hui for the reply.
I was expecting to see RoCE messages in the packet capture.
When I use verbs on libfabric sample tests, I see the RoCE messages as below :
[cid:[email protected]]
From: Zhou, Hui <zhouh(a)anl.gov>
Date: Wednesday, June 12, 2024 at 3:41 PM
To: Niyaz Murshed <Niyaz.Murshed(a)arm.com>, discuss(a)mpich.org <discuss(a)mpich.org>
Cc: nd <nd(a)arm.com>
Subject: Re: Mpich over RDMA sample
Niyaz,
ofi_rxm:verbs is the correct provider. The ofi_rxm provides the verbs provider additional messaging semantics that are needed to be used in MPI.
--
Hui Zhou
From: Niyaz Murshed <Niyaz.Murshed(a)arm.com>
Date: Wednesday, June 12, 2024 at 2:54 PM
To: Zhou, Hui <zhouh(a)anl.gov>, discuss(a)mpich.org <discuss(a)mpich.org>
Cc: nd <nd(a)arm.com>
Subject: Re: Mpich over RDMA sample
Thank you for the reply. I did that; however, it selects ofi_rxm: verbs libfabric: 412098: 1718221892: : ofi_rxm: av: ofi_av_insert_addr(): 313<info> fi_addr: 1 libfabric: 412099: 1718221892: : ofi_rxm: av: ofi_av_insert_addr(): 313<info> fi_addr:
ZjQcmQRYFpfptBannerStart
This Message Is From an External Sender
This message came from outside your organization.
ZjQcmQRYFpfptBannerEnd
Thank you for the reply.
I did that; however, it selects ofi_rxm:verbs
libfabric:412098:1718221892::ofi_rxm:av:ofi_av_insert_addr():313<info> fi_addr: 1
libfabric:412099:1718221892::ofi_rxm:av:ofi_av_insert_addr():313<info> fi_addr: 1
options:
backend: cpu
iters: 16
warmup_iters: 16
cache: 1
min_elem_count: 1
max_elem_count: 1
elem_counts: [1]
validate: last
window_size: 64
#------------------------------------------------------------
# Benchmarking: Bandwidth
# #processes: 2
#------------------------------------------------------------
#bytes #repetitions Mbytes/sec
4 16 0.02
# All done
libfabric:412098:1718221892::ofi_rxm:ep_ctrl:rxm_stop_listen():864<info> stopping CM thread
libfabric:412099:1718221892::ofi_rxm:ep_ctrl:rxm_stop_listen():864<info> stopping CM thread
As per https://urldefense.us/v3/__https://www.intel.com/content/www/us/en/docs/mpi… <https://urldefense.us/v3/__https:/www.intel.com/content/www/us/en/docs/mpi-…> , we need to add FI_PROVIDER=^ofi_rxm , but if do that, it moves back to sockets provider.
Is here a way to combine ^ofi_rxm and verbs
From: Zhou, Hui <zhouh(a)anl.gov>
Date: Wednesday, June 12, 2024 at 2:41 PM
To: Niyaz Murshed <Niyaz.Murshed(a)arm.com>, discuss(a)mpich.org <discuss(a)mpich.org>
Cc: nd <nd(a)arm.com>
Subject: Re: Mpich over RDMA sample
Niyaz,
All you need to do is to set an environment variable `FI_PROVIDER=verbs`.
--
Hui Zhou
From: Niyaz Murshed <Niyaz.Murshed(a)arm.com>
Date: Wednesday, June 12, 2024 at 1:23 PM
To: Zhou, Hui <zhouh(a)anl.gov>, discuss(a)mpich.org <discuss(a)mpich.org>
Cc: nd <nd(a)arm.com>
Subject: Re: Mpich over RDMA sample
When testing with Libfabric, verbs provider is selected. I did have to use “-e msg -d mlx5_1” so that it selects verbs. I was checking if there is anything like that for mpich sample tests. Else might need to do some hack in the code to force
ZjQcmQRYFpfptBannerStart
This Message Is From an External Sender
This message came from outside your organization.
ZjQcmQRYFpfptBannerEnd
When testing with Libfabric, verbs provider is selected. I did have to use “-e msg -d mlx5_1” so that it selects verbs.
I was checking if there is anything like that for mpich sample tests. Else might need to do some hack in the code to force the selection of verbs.
From: Zhou, Hui <zhouh(a)anl.gov>
Date: Wednesday, June 12, 2024 at 1:15 PM
To: discuss(a)mpich.org <discuss(a)mpich.org>
Cc: Niyaz Murshed <Niyaz.Murshed(a)arm.com>, nd <nd(a)arm.com>
Subject: Re: Mpich over RDMA sample
Libfabric support multiple providers. Sounds like it was selecting the sockets or tcp provider rather than a provider that support RoCE. I am not exactly sure whether the verbs provider will do that. If you can confirm the provider using libfabric tests, then you can try forcing MPICH to use that provider by setting the FI_PROVIDER environment variable.
--
Hui Zhou
From: Niyaz Murshed via discuss <discuss(a)mpich.org>
Date: Wednesday, June 12, 2024 at 9:03 AM
To: discuss(a)mpich.org <discuss(a)mpich.org>
Cc: Niyaz Murshed <Niyaz.Murshed(a)arm.com>, nd <nd(a)arm.com>
Subject: [mpich-discuss] Mpich over RDMA sample
Hello, I am trying to learn about MPICH and its performance over RDMA. I am using libfabric and installed mpich using the below configure. ./configure --prefix=/opt/mpich/ --with-ofi=/opt/libfabric/ When I run any applications between 2 directly
ZjQcmQRYFpfptBannerStart
This Message Is From an External Sender
This message came from outside your organization.
ZjQcmQRYFpfptBannerEnd
Hello,
I am trying to learn about MPICH and its performance over RDMA.
I am using libfabric and installed mpich using the below configure.
./configure --prefix=/opt/mpich/ --with-ofi=/opt/libfabric/
When I run any applications between 2 directly connected servers having Mellanox NICs, I see that communication is happening over tcp and not over RoCE.
Is there any way to test commination over RoCE ?
For eg. I was able to test it for libfabric using the below sample that comes along with libfabric to test RMA.
Is there something similar for MPICH ? or use the current sample to use RoCE by some parameter?
Server :
fi_rma_bw -s 192.168.1.100 -e msg -d mlx5_1 -S 1024 -I 1
Client :
fi_rma_bw -s 192.168.1.200 -e msg -d mlx5_3 192.168.1.100 -S 1024 -I 1
Regards,
Niyaz
1
0
Hi Hui,
When you say message semantic, do you mean endpoint type message ?
Mpich is running over libfabric.
When I run sample application on libfabric, it does have message endpoints over RMA.
From: Zhou, Hui <zhouh(a)anl.gov>
Date: Tuesday, June 18, 2024 at 9:44 AM
To: Niyaz Murshed <Niyaz.Murshed(a)arm.com>, discuss(a)mpich.org <discuss(a)mpich.org>
Cc: nd <nd(a)arm.com>
Subject: Re: Mpich over RDMA sample
The "verbs" provider alone does not provide the message semantics we needed for MPI. "ofi_rxm" is the utility provider that provides the message semantics. "verbs;ofi_rxm" is the provider combination you want.
Hui
________________________________
From: Niyaz Murshed <Niyaz.Murshed(a)arm.com>
Sent: Tuesday, June 18, 2024 9:14 AM
To: Zhou, Hui <zhouh(a)anl.gov>; discuss(a)mpich.org <discuss(a)mpich.org>
Cc: nd <nd(a)arm.com>
Subject: Re: Mpich over RDMA sample
Hi Hui, I have set the env variable to FI_PROVIDER=verbs , after thich I get the below. libfabric provider: verbs;ofi_rxm - IB-0xfe80000000000000 It does not chose just “verbs” , it selects verbs;ofi_rxf I am wanting to select the below: provider:
ZjQcmQRYFpfptBannerStart
This Message Is From an External Sender
This message came from outside your organization.
ZjQcmQRYFpfptBannerEnd
Hi Hui,
I have set the env variable to FI_PROVIDER=verbs , after thich I get the below.
libfabric provider: verbs;ofi_rxm - IB-0xfe80000000000000
It does not chose just “verbs” , it selects verbs;ofi_rxf
I am wanting to select the below:
provider: verbs, score = 0, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200
I would really like to thank you for such fast responses.
From: Zhou, Hui <zhouh(a)anl.gov>
Date: Tuesday, June 18, 2024 at 8:58 AM
To: Niyaz Murshed <Niyaz.Murshed(a)arm.com>, discuss(a)mpich.org <discuss(a)mpich.org>
Cc: nd <nd(a)arm.com>
Subject: Re: Mpich over RDMA sample
Niyaz,
Set environment variable FI_PROVIDER=verbs to select the verbs provider.
Hui
________________________________
From: Niyaz Murshed <Niyaz.Murshed(a)arm.com>
Sent: Monday, June 17, 2024 8:57 PM
To: Zhou, Hui <zhouh(a)anl.gov>; discuss(a)mpich.org <discuss(a)mpich.org>
Cc: nd <nd(a)arm.com>
Subject: Re: Mpich over RDMA sample
Hi Hui, I would still need some assistance in this. I am not able to make mpich chose verbs provider ☹ With the log : provider: verbs, score = 0, pref = 0, FI_SOCKADDR_IN [16] 192. 168. 2. 200 Is my assumption to think the above log represent
ZjQcmQRYFpfptBannerStart
This Message Is From an External Sender
This message came from outside your organization.
ZjQcmQRYFpfptBannerEnd
Hi Hui,
I would still need some assistance in this. I am not able to make mpich chose verbs provider ☹
With the log :
provider: verbs, score = 0, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200
Is my assumption to think the above log represent that verbs provider knows about a path to 192.168.2.200 ?
How does the application hello-world decide which provider to use?
Running on server: 192.168.2.100
root@ampere-altra-2-1# FI_PROVIDER=verbs MPIR_CVAR_DEBUG_SUMMARY=1 mpirun -n 5 -hosts 192.168.2.200,192.168.2.100 /mpich/examples/a.out
==== Various sizes and limits ====
sizeof(MPIDI_per_vci_t): 192
Required minimum FI_VERSION: 0, current version: 10015
provider: verbs, score = 0, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200
provider: verbs, score = 0, pref = 0, FI_SOCKADDR_IN6 [28] fe80::526b:4bff:fefc:1342
provider: verbs, score = 0, pref = 0, FI_SOCKADDR_IB [48]
provider: verbs, score = 0, pref = 0, FI_SOCKADDR_IB [48]
provider: verbs, score = 0, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200
provider: verbs, score = 0, pref = 0, FI_SOCKADDR_IN6 [28] fe80::526b:4bff:fefc:1342
provider: verbs, score = 0, pref = 0, FI_SOCKADDR_IB [48]
provider: verbs, score = 0, pref = 0, FI_SOCKADDR_IB [48]
provider: verbs, score = 0, pref = 0, FI_FORMAT_UNSPEC [32]
provider: verbs, score = 0, pref = 0, FI_FORMAT_UNSPEC [32]
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::526b:4bff:fefc:1342
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48]
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48]
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::526b:4bff:fefc:1342
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48]
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48]
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::526b:4bff:fefc:1342
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48]
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48]
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::526b:4bff:fefc:1342
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48]
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48]
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::526b:4bff:fefc:1342
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48]
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48]
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::526b:4bff:fefc:1342
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48]
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48]
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::526b:4bff:fefc:1342
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48]
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48]
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::526b:4bff:fefc:1342
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48]
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48]
provider: verbs;ofi_rxd, score = 5, pref = -2, FI_FORMAT_UNSPEC [32]
provider: verbs;ofi_rxd, score = 5, pref = -2, FI_FORMAT_UNSPEC [32]
Required minimum FI_VERSION: 10006, current version: 10015
==== Capability set configuration ====
libfabric provider: verbs;ofi_rxm - IB-0xfe80000000000000
MPIDI_OFI_ENABLE_DATA: 0
MPIDI_OFI_ENABLE_AV_TABLE: 1
MPIDI_OFI_ENABLE_SCALABLE_ENDPOINTS: 0
MPIDI_OFI_ENABLE_SHARED_CONTEXTS: 0
MPIDI_OFI_ENABLE_MR_VIRT_ADDRESS: 1
MPIDI_OFI_ENABLE_MR_ALLOCATED: 1
MPIDI_OFI_ENABLE_MR_REGISTER_NULL: 1
MPIDI_OFI_ENABLE_MR_PROV_KEY: 1
MPIDI_OFI_ENABLE_TAGGED: 1
MPIDI_OFI_ENABLE_AM: 1
MPIDI_OFI_ENABLE_RMA: 1
MPIDI_OFI_ENABLE_ATOMICS: 0
MPIDI_OFI_FETCH_ATOMIC_IOVECS: 1
MPIDI_OFI_ENABLE_DATA_AUTO_PROGRESS: 0
MPIDI_OFI_ENABLE_CONTROL_AUTO_PROGRESS: 0
MPIDI_OFI_ENABLE_PT2PT_NOPACK: 1
MPIDI_OFI_ENABLE_TRIGGERED: 0
MPIDI_OFI_ENABLE_HMEM: 0
MPIDI_OFI_NUM_AM_BUFFERS: 8
MPIDI_OFI_NUM_OPTIMIZED_MEMORY_REGIONS: 0
MPIDI_OFI_CONTEXT_BITS: 16
MPIDI_OFI_SOURCE_BITS: 23
MPIDI_OFI_TAG_BITS: 20
MPIDI_OFI_VNI_USE_DOMAIN: 1
MAXIMUM SUPPORTED RANKS: 8388608
MAXIMUM TAG: 1048576
==== Provider global thresholds ====
max_buffered_send: 192
max_buffered_write: 192
max_msg_size: 1073741824
max_order_raw: 1073741824
max_order_war: 0
max_order_waw: 1073741824
tx_iov_limit: 4
rx_iov_limit: 4
rma_iov_limit: 1
max_mr_key_size: 4
==== Various sizes and limits ====
MPIDI_OFI_AM_MSG_HEADER_SIZE: 24
MPIDI_OFI_MAX_AM_HDR_SIZE: 255
sizeof(MPIDI_OFI_am_request_header_t): 416
sizeof(MPIDI_OFI_per_vci_t): 52480
MPIDI_OFI_AM_HDR_POOL_CELL_SIZE: 1024
MPIDI_OFI_DEFAULT_SHORT_SEND_SIZE: 16384
Assertion failed in file src/mpid/ch4/netmod/ofi/init_addrxchg.c at line 151: mapped_table[i] != FI_ADDR_NOTAVAIL
Assertion failed in file src/mpid/ch4/netmod/ofi/init_addrxchg.c at line 151: mapped_table[i] != FI_ADDR_NOTAVAIL
Assertion failed in file src/mpid/ch4/netmod/ofi/init_addrxchg.c at line 151: mapped_table[i] != FI_ADDR_NOTAVAIL
Assertion failed in file src/mpid/ch4/netmod/ofi/init_addrxchg.c at line 151: mapped_table[i] != FI_ADDR_NOTAVAIL
Assertion failed in file src/mpid/ch4/netmod/ofi/init_addrxchg.c at line 151: mapped_table[i] != FI_ADDR_NOTAVAIL
If I do not use the FI_PROVIDER=verbs, it will work over sockets.
root@ampere-altra-2-1: # MPIR_CVAR_DEBUG_SUMMARY=1 mpirun -n 5 -hosts 192.168.2.200,192.168.2.100 /mpich/examples/a.out
==== Various sizes and limits ====
sizeof(MPIDI_per_vci_t): 192
Required minimum FI_VERSION: 0, current version: 10015
provider: verbs, score = 0, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200
provider: verbs, score = 0, pref = 0, FI_SOCKADDR_IN6 [28] fe80::526b:4bff:fefc:1342
provider: verbs, score = 0, pref = 0, FI_SOCKADDR_IB [48]
provider: verbs, score = 0, pref = 0, FI_SOCKADDR_IB [48]
provider: verbs, score = 0, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200
provider: verbs, score = 0, pref = 0, FI_SOCKADDR_IN6 [28] fe80::526b:4bff:fefc:1342
provider: verbs, score = 0, pref = 0, FI_SOCKADDR_IB [48]
provider: verbs, score = 0, pref = 0, FI_SOCKADDR_IB [48]
provider: verbs, score = 0, pref = 0, FI_FORMAT_UNSPEC [32]
provider: verbs, score = 0, pref = 0, FI_FORMAT_UNSPEC [32]
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::526b:4bff:fefc:1342
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48]
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48]
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::526b:4bff:fefc:1342
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48]
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48]
provider: tcp, score = 4, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200
provider: tcp, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::526b:4bff:fefc:1342
provider: tcp, score = 4, pref = 0, FI_SOCKADDR_IN [16] 10.118.91.159
provider: tcp, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::be97:e1ff:fe9d:7caa
provider: tcp, score = 4, pref = 0, FI_SOCKADDR_IN [16] 127.0.0.1
provider: tcp, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] ::1
provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200
provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::526b:4bff:fefc:1342
provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 10.118.91.159
provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::be97:e1ff:fe9d:7caa
provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 127.0.0.1
provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] ::1
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::526b:4bff:fefc:1342
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48]
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48]
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::526b:4bff:fefc:1342
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48]
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48]
provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200
provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::526b:4bff:fefc:1342
provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 10.118.91.159
provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::be97:e1ff:fe9d:7caa
provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 127.0.0.1
provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] ::1
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::526b:4bff:fefc:1342
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48]
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48]
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::526b:4bff:fefc:1342
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48]
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48]
provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200
provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::526b:4bff:fefc:1342
provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 10.118.91.159
provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::be97:e1ff:fe9d:7caa
provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 127.0.0.1
provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] ::1
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::526b:4bff:fefc:1342
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48]
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48]
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::526b:4bff:fefc:1342
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48]
provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48]
provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200
provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::526b:4bff:fefc:1342
provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 10.118.91.159
provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::be97:e1ff:fe9d:7caa
provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 127.0.0.1
provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] ::1
provider: verbs;ofi_rxd, score = 5, pref = -2, FI_FORMAT_UNSPEC [32]
provider: verbs;ofi_rxd, score = 5, pref = -2, FI_FORMAT_UNSPEC [32]
provider: udp;ofi_rxd, score = 5, pref = -2, FI_SOCKADDR_IN [16] 192.168.2.200
provider: udp;ofi_rxd, score = 5, pref = -2, FI_SOCKADDR_IN6 [28] fe80::526b:4bff:fefc:1342
provider: udp;ofi_rxd, score = 5, pref = -2, FI_SOCKADDR_IN [16] 10.118.91.159
provider: udp;ofi_rxd, score = 5, pref = -2, FI_SOCKADDR_IN6 [28] fe80::be97:e1ff:fe9d:7caa
provider: udp;ofi_rxd, score = 5, pref = -2, FI_SOCKADDR_IN [16] 127.0.0.1
provider: udp;ofi_rxd, score = 5, pref = -2, FI_SOCKADDR_IN6 [28] ::1
provider: shm, score = 4, pref = -2, FI_ADDR_STR [14] - fi_shm://1177
provider: shm, score = 4, pref = -2, FI_ADDR_STR [14] - fi_shm://1177
provider: udp, score = 0, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200
provider: udp, score = 0, pref = 0, FI_SOCKADDR_IN6 [28] fe80::526b:4bff:fefc:1342
provider: udp, score = 0, pref = 0, FI_SOCKADDR_IN [16] 10.118.91.159
provider: udp, score = 0, pref = 0, FI_SOCKADDR_IN6 [28] fe80::be97:e1ff:fe9d:7caa
provider: udp, score = 0, pref = 0, FI_SOCKADDR_IN [16] 127.0.0.1
provider: udp, score = 0, pref = 0, FI_SOCKADDR_IN6 [28] ::1
provider: tcp, score = 0, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200
provider: tcp, score = 0, pref = 0, FI_SOCKADDR_IN6 [28] fe80::526b:4bff:fefc:1342
provider: tcp, score = 0, pref = 0, FI_SOCKADDR_IN [16] 10.118.91.159
provider: tcp, score = 0, pref = 0, FI_SOCKADDR_IN6 [28] fe80::be97:e1ff:fe9d:7caa
provider: tcp, score = 0, pref = 0, FI_SOCKADDR_IN [16] 127.0.0.1
provider: tcp, score = 0, pref = 0, FI_SOCKADDR_IN6 [28] ::1
provider: tcp, score = 4, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200
provider: tcp, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::526b:4bff:fefc:1342
provider: tcp, score = 4, pref = 0, FI_SOCKADDR_IN [16] 10.118.91.159
provider: tcp, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::be97:e1ff:fe9d:7caa
provider: tcp, score = 4, pref = 0, FI_SOCKADDR_IN [16] 127.0.0.1
provider: tcp, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] ::1
provider: sockets, score = 3, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200
provider: sockets, score = 3, pref = 0, FI_SOCKADDR_IN6 [28] fe80::526b:4bff:fefc:1342
provider: sockets, score = 3, pref = 0, FI_SOCKADDR_IN [16] 10.118.91.159
provider: sockets, score = 3, pref = 0, FI_SOCKADDR_IN6 [28] fe80::be97:e1ff:fe9d:7caa
provider: sockets, score = 3, pref = 0, FI_SOCKADDR_IN [16] 127.0.0.1
provider: sockets, score = 3, pref = 0, FI_SOCKADDR_IN6 [28] ::1
provider: sockets, score = 5, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200
provider: sockets, score = 5, pref = 0, FI_SOCKADDR_IN6 [28] fe80::526b:4bff:fefc:1342
provider: sockets, score = 5, pref = 0, FI_SOCKADDR_IN [16] 10.118.91.159
provider: sockets, score = 5, pref = 0, FI_SOCKADDR_IN6 [28] fe80::be97:e1ff:fe9d:7caa
provider: sockets, score = 5, pref = 0, FI_SOCKADDR_IN [16] 127.0.0.1
provider: sockets, score = 5, pref = 0, FI_SOCKADDR_IN6 [28] ::1
provider: sockets, score = 5, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200
provider: sockets, score = 5, pref = 0, FI_SOCKADDR_IN6 [28] fe80::526b:4bff:fefc:1342
provider: sockets, score = 5, pref = 0, FI_SOCKADDR_IN [16] 10.118.91.159
provider: sockets, score = 5, pref = 0, FI_SOCKADDR_IN6 [28] fe80::be97:e1ff:fe9d:7caa
provider: sockets, score = 5, pref = 0, FI_SOCKADDR_IN [16] 127.0.0.1
provider: sockets, score = 5, pref = 0, FI_SOCKADDR_IN6 [28] ::1
provider: sm2, score = 3, pref = 0, FI_ADDR_STR [13] - fi_sm2://1177
provider: sm2, score = 3, pref = 0, FI_ADDR_STR [13] - fi_sm2://1177
Required minimum FI_VERSION: 10005, current version: 10015
==== Capability set configuration ====
libfabric provider: sockets - 192.168.2.0/24
MPIDI_OFI_ENABLE_DATA: 1
MPIDI_OFI_ENABLE_AV_TABLE: 1
MPIDI_OFI_ENABLE_SCALABLE_ENDPOINTS: 1
MPIDI_OFI_ENABLE_SHARED_CONTEXTS: 0
MPIDI_OFI_ENABLE_MR_VIRT_ADDRESS: 0
MPIDI_OFI_ENABLE_MR_ALLOCATED: 0
MPIDI_OFI_ENABLE_MR_REGISTER_NULL: 1
MPIDI_OFI_ENABLE_MR_PROV_KEY: 0
MPIDI_OFI_ENABLE_TAGGED: 1
MPIDI_OFI_ENABLE_AM: 1
MPIDI_OFI_ENABLE_RMA: 1
MPIDI_OFI_ENABLE_ATOMICS: 1
MPIDI_OFI_FETCH_ATOMIC_IOVECS: 1
MPIDI_OFI_ENABLE_DATA_AUTO_PROGRESS: 0
MPIDI_OFI_ENABLE_CONTROL_AUTO_PROGRESS: 0
MPIDI_OFI_ENABLE_PT2PT_NOPACK: 1
MPIDI_OFI_ENABLE_TRIGGERED: 0
MPIDI_OFI_ENABLE_HMEM: 0
MPIDI_OFI_NUM_AM_BUFFERS: 8
MPIDI_OFI_NUM_OPTIMIZED_MEMORY_REGIONS: 0
MPIDI_OFI_CONTEXT_BITS: 20
MPIDI_OFI_SOURCE_BITS: 0
MPIDI_OFI_TAG_BITS: 31
MPIDI_OFI_VNI_USE_DOMAIN: 1
MAXIMUM SUPPORTED RANKS: 4294967296
MAXIMUM TAG: 2147483648
==== Provider global thresholds ====
max_buffered_send: 255
max_buffered_write: 255
max_msg_size: 9223372036854775807
max_order_raw: -1
max_order_war: -1
max_order_waw: -1
tx_iov_limit: 8
rx_iov_limit: 8
rma_iov_limit: 8
max_mr_key_size: 8
==== Various sizes and limits ====
MPIDI_OFI_AM_MSG_HEADER_SIZE: 24
MPIDI_OFI_MAX_AM_HDR_SIZE: 255
sizeof(MPIDI_OFI_am_request_header_t): 416
sizeof(MPIDI_OFI_per_vci_t): 52480
MPIDI_OFI_AM_HDR_POOL_CELL_SIZE: 1024
MPIDI_OFI_DEFAULT_SHORT_SEND_SIZE: 16384
==== OFI dynamic settings ====
num_vcis: 1
num_nics: 1
======================================
error checking : enabled
QMPI : disabled
debugger support : disabled
thread level : MPI_THREAD_SINGLE
thread CS : per-vci
threadcomm : enabled
==== data structure summary ====
sizeof(MPIR_Comm): 1808
sizeof(MPIR_Request): 512
sizeof(MPIR_Datatype): 280
================================
Hello world from process 0 of 5
Hello world from process 2 of 5
Hello world from process 1 of 5
Hello world from process 4 of 5
Hello world from process 3 of 5
From: Zhou, Hui <zhouh(a)anl.gov>
Date: Wednesday, June 12, 2024 at 7:22 PM
To: Niyaz Murshed <Niyaz.Murshed(a)arm.com>, discuss(a)mpich.org <discuss(a)mpich.org>
Subject: Re: Mpich over RDMA sample
It’s possible that the native path for RMA on verbs are turned off due to verbs does not meet MPI’s full atomic ordering semantics. We’ll check and follow up.
--
Hui Zhou
From: Niyaz Murshed <Niyaz.Murshed(a)arm.com>
Date: Wednesday, June 12, 2024 at 3:59 PM
To: Zhou, Hui <zhouh(a)anl.gov>, discuss(a)mpich.org <discuss(a)mpich.org>
Cc: nd <nd(a)arm.com>
Subject: Re: Mpich over RDMA sample
I think image is not gone : https: //ibb. co/RDGnJfy From: Niyaz Murshed <Niyaz. Murshed@ arm. com> Date: Wednesday, June 12, 2024 at 3: 56 PM To: Zhou, Hui <zhouh@ anl. gov>, discuss@ mpich. org <discuss@ mpich. org> Cc: nd <nd@ arm. com>
ZjQcmQRYFpfptBannerStart
This Message Is From an External Sender
This message came from outside your organization.
ZjQcmQRYFpfptBannerEnd
I think image is not gone : https://urldefense.us/v3/__https://ibb.co/RDGnJfy__;!!G_uCfscf7eWS!YTDZ_B-S… <https://urldefense.us/v3/__https:/ibb.co/RDGnJfy__;!!G_uCfscf7eWS!YH6L3OHfP…>
From: Niyaz Murshed <Niyaz.Murshed(a)arm.com>
Date: Wednesday, June 12, 2024 at 3:56 PM
To: Zhou, Hui <zhouh(a)anl.gov>, discuss(a)mpich.org <discuss(a)mpich.org>
Cc: nd <nd(a)arm.com>
Subject: Re: Mpich over RDMA sample
Thank you Hui for the reply.
I was expecting to see RoCE messages in the packet capture.
When I use verbs on libfabric sample tests, I see the RoCE messages as below :
[cid:[email protected]]
From: Zhou, Hui <zhouh(a)anl.gov>
Date: Wednesday, June 12, 2024 at 3:41 PM
To: Niyaz Murshed <Niyaz.Murshed(a)arm.com>, discuss(a)mpich.org <discuss(a)mpich.org>
Cc: nd <nd(a)arm.com>
Subject: Re: Mpich over RDMA sample
Niyaz,
ofi_rxm:verbs is the correct provider. The ofi_rxm provides the verbs provider additional messaging semantics that are needed to be used in MPI.
--
Hui Zhou
From: Niyaz Murshed <Niyaz.Murshed(a)arm.com>
Date: Wednesday, June 12, 2024 at 2:54 PM
To: Zhou, Hui <zhouh(a)anl.gov>, discuss(a)mpich.org <discuss(a)mpich.org>
Cc: nd <nd(a)arm.com>
Subject: Re: Mpich over RDMA sample
Thank you for the reply. I did that; however, it selects ofi_rxm: verbs libfabric: 412098: 1718221892: : ofi_rxm: av: ofi_av_insert_addr(): 313<info> fi_addr: 1 libfabric: 412099: 1718221892: : ofi_rxm: av: ofi_av_insert_addr(): 313<info> fi_addr:
ZjQcmQRYFpfptBannerStart
This Message Is From an External Sender
This message came from outside your organization.
ZjQcmQRYFpfptBannerEnd
Thank you for the reply.
I did that; however, it selects ofi_rxm:verbs
libfabric:412098:1718221892::ofi_rxm:av:ofi_av_insert_addr():313<info> fi_addr: 1
libfabric:412099:1718221892::ofi_rxm:av:ofi_av_insert_addr():313<info> fi_addr: 1
options:
backend: cpu
iters: 16
warmup_iters: 16
cache: 1
min_elem_count: 1
max_elem_count: 1
elem_counts: [1]
validate: last
window_size: 64
#------------------------------------------------------------
# Benchmarking: Bandwidth
# #processes: 2
#------------------------------------------------------------
#bytes #repetitions Mbytes/sec
4 16 0.02
# All done
libfabric:412098:1718221892::ofi_rxm:ep_ctrl:rxm_stop_listen():864<info> stopping CM thread
libfabric:412099:1718221892::ofi_rxm:ep_ctrl:rxm_stop_listen():864<info> stopping CM thread
As per https://urldefense.us/v3/__https://www.intel.com/content/www/us/en/docs/mpi… <https://urldefense.us/v3/__https:/www.intel.com/content/www/us/en/docs/mpi-…> , we need to add FI_PROVIDER=^ofi_rxm , but if do that, it moves back to sockets provider.
Is here a way to combine ^ofi_rxm and verbs
From: Zhou, Hui <zhouh(a)anl.gov>
Date: Wednesday, June 12, 2024 at 2:41 PM
To: Niyaz Murshed <Niyaz.Murshed(a)arm.com>, discuss(a)mpich.org <discuss(a)mpich.org>
Cc: nd <nd(a)arm.com>
Subject: Re: Mpich over RDMA sample
Niyaz,
All you need to do is to set an environment variable `FI_PROVIDER=verbs`.
--
Hui Zhou
From: Niyaz Murshed <Niyaz.Murshed(a)arm.com>
Date: Wednesday, June 12, 2024 at 1:23 PM
To: Zhou, Hui <zhouh(a)anl.gov>, discuss(a)mpich.org <discuss(a)mpich.org>
Cc: nd <nd(a)arm.com>
Subject: Re: Mpich over RDMA sample
When testing with Libfabric, verbs provider is selected. I did have to use “-e msg -d mlx5_1” so that it selects verbs. I was checking if there is anything like that for mpich sample tests. Else might need to do some hack in the code to force
ZjQcmQRYFpfptBannerStart
This Message Is From an External Sender
This message came from outside your organization.
ZjQcmQRYFpfptBannerEnd
When testing with Libfabric, verbs provider is selected. I did have to use “-e msg -d mlx5_1” so that it selects verbs.
I was checking if there is anything like that for mpich sample tests. Else might need to do some hack in the code to force the selection of verbs.
From: Zhou, Hui <zhouh(a)anl.gov>
Date: Wednesday, June 12, 2024 at 1:15 PM
To: discuss(a)mpich.org <discuss(a)mpich.org>
Cc: Niyaz Murshed <Niyaz.Murshed(a)arm.com>, nd <nd(a)arm.com>
Subject: Re: Mpich over RDMA sample
Libfabric support multiple providers. Sounds like it was selecting the sockets or tcp provider rather than a provider that support RoCE. I am not exactly sure whether the verbs provider will do that. If you can confirm the provider using libfabric tests, then you can try forcing MPICH to use that provider by setting the FI_PROVIDER environment variable.
--
Hui Zhou
From: Niyaz Murshed via discuss <discuss(a)mpich.org>
Date: Wednesday, June 12, 2024 at 9:03 AM
To: discuss(a)mpich.org <discuss(a)mpich.org>
Cc: Niyaz Murshed <Niyaz.Murshed(a)arm.com>, nd <nd(a)arm.com>
Subject: [mpich-discuss] Mpich over RDMA sample
Hello, I am trying to learn about MPICH and its performance over RDMA. I am using libfabric and installed mpich using the below configure. ./configure --prefix=/opt/mpich/ --with-ofi=/opt/libfabric/ When I run any applications between 2 directly
ZjQcmQRYFpfptBannerStart
This Message Is From an External Sender
This message came from outside your organization.
ZjQcmQRYFpfptBannerEnd
Hello,
I am trying to learn about MPICH and its performance over RDMA.
I am using libfabric and installed mpich using the below configure.
./configure --prefix=/opt/mpich/ --with-ofi=/opt/libfabric/
When I run any applications between 2 directly connected servers having Mellanox NICs, I see that communication is happening over tcp and not over RoCE.
Is there any way to test commination over RoCE ?
For eg. I was able to test it for libfabric using the below sample that comes along with libfabric to test RMA.
Is there something similar for MPICH ? or use the current sample to use RoCE by some parameter?
Server :
fi_rma_bw -s 192.168.1.100 -e msg -d mlx5_1 -S 1024 -I 1
Client :
fi_rma_bw -s 192.168.1.200 -e msg -d mlx5_3 192.168.1.100 -S 1024 -I 1
Regards,
Niyaz
1
0