- discuss - mailman3.cels.anl.gov

Re: [mpich-discuss] Custom rank for processes
by Niyaz Murshed 01 Jul '24

01 Jul '24

What is the rankmap used for ? ________________________________ From: Jenke, Joachim <jenke(a)itc.rwth-aachen.de> Sent: Monday, July 1, 2024 2:21:53 PM To: discuss(a)mpich.org <discuss(a)mpich.org> Cc: Niyaz Murshed <Niyaz.Murshed(a)arm.com>; nd <nd(a)arm.com> Subject: Re: Custom rank for processes MPI defines rank numbers to be consecutive from 0 to size-1. Your suggested numbering will not work. -Joachim ________________________________ From: Niyaz Murshed via discuss <discuss(a)mpich.org> Sent: Monday, July 1, 2024 9:09:41 PM To: discuss(a)mpich.org <discuss(a)mpich.org> Cc: Niyaz Murshed <Niyaz.Murshed(a)arm.com>; nd <nd(a)arm.com> Subject: [mpich-discuss] Custom rank for processes Hello, Is it possible to provide custom ranks to processes ? I see a param called “-rankmap” .. Does it provide this option? Is it possible to do something like below : Node1 => 2 processes => rank 2,3 Node2 => 2 processes => rank ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Hello, Is it possible to provide custom ranks to processes ? I see a param called “-rankmap” .. Does it provide this option? Is it possible to do something like below : Node1 => 2 processes => rank 2,3 Node2 => 2 processes => rank 20,30 Thanks.

1 0

Re: [mpich-discuss] Custom rank for processes
by Jenke, Joachim 01 Jul '24

01 Jul '24

MPI defines rank numbers to be consecutive from 0 to size-1. Your suggested numbering will not work. -Joachim ________________________________ From: Niyaz Murshed via discuss <discuss(a)mpich.org> Sent: Monday, July 1, 2024 9:09:41 PM To: discuss(a)mpich.org <discuss(a)mpich.org> Cc: Niyaz Murshed <Niyaz.Murshed(a)arm.com>; nd <nd(a)arm.com> Subject: [mpich-discuss] Custom rank for processes Hello, Is it possible to provide custom ranks to processes ? I see a param called “-rankmap” .. Does it provide this option? Is it possible to do something like below : Node1 => 2 processes => rank 2,3 Node2 => 2 processes => rank ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Hello, Is it possible to provide custom ranks to processes ? I see a param called “-rankmap” .. Does it provide this option? Is it possible to do something like below : Node1 => 2 processes => rank 2,3 Node2 => 2 processes => rank 20,30 Thanks.

1 0

Custom rank for processes
by Niyaz Murshed 01 Jul '24

01 Jul '24

Hello, Is it possible to provide custom ranks to processes ? I see a param called “-rankmap” .. Does it provide this option? Is it possible to do something like below : Node1 => 2 processes => rank 2,3 Node2 => 2 processes => rank 20,30 Thanks.

1 0

Re: [mpich-discuss] Failure to compile with gcc/14
by Raffenetti, Ken 28 Jun '24

28 Jun '24

Hi Victor, https://urldefense.us/v3/__https://github.com/pmodels/mpich/pull/7033__;!!G… should address this issue. It contains patches that were submitted/accepted in upstream libfabric. I have already backported to the 4.2.x branch, and a soon to be released 4.2.2 will have the fix. Thanks, Ken From: Victor Eijkhout via discuss <discuss(a)mpich.org> Reply-To: "discuss(a)mpich.org" <discuss(a)mpich.org> Date: Friday, June 28, 2024 at 2:59 PM To: "discuss(a)mpich.org" <discuss(a)mpich.org> Cc: Victor Eijkhout <eijkhout(a)tacc.utexas.edu> Subject: [mpich-discuss] Failure to compile with gcc/14 Gcc/14 installed on Mac OS 14. 5 through brew, compiling the latest MPICH 4. 2. 1 release. src/hmem. c: 216: 45: error: passing argument 3 of 'ofi_iov_bytes_to_copy' from incompatible pointer type [-Wincompatible-pointer-types] 216 | &hmem_iov_offset, ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Gcc/14 installed on Mac OS 14.5 through brew, compiling the latest MPICH 4.2.1 release. src/hmem.c:216:45: error: passing argument 3 of 'ofi_iov_bytes_to_copy' from incompatible pointer type [-Wincompatible-pointer-types] 216 | &hmem_iov_offset, &hmem_buf); | ^~~~~~~~~~~~~~~~ | | | uint64_t * {aka long long unsigned int *} In file included from src/hmem.c:41: ./include/ofi_iov.h:87:47: note: expected 'size_t *' {aka 'long unsigned int *'} but argument is of type 'uint64_t *' {aka 'long long unsigned int *'} 87 | size_t *size, size_t *offset, char **copy_buf) | ~~~~~~~~^~~~~~ src/hmem.c: In function 'ofi_copy_hmem_iov_buf': src/hmem.c:252:45: error: passing argument 3 of 'ofi_iov_bytes_to_copy' from incompatible pointer type [-Wincompatible-pointer-types] 252 | &hmem_iov_offset, &hmem_buf); | ^~~~~~~~~~~~~~~~ | | | uint64_t * {aka long long unsigned int *} ./include/ofi_iov.h:87:47: note: expected 'size_t *' {aka 'long unsigned int *'} but argument is of type 'uint64_t *' {aka 'long long unsigned int *'} 87 | size_t *size, size_t *offset, char **copy_buf) | ~~~~~~~~^~~~~~ src/hmem.c: In function 'ofi_copy_mr_iov': src/hmem.c:282:61: error: passing argument 3 of 'ofi_iov_bytes_to_copy' from incompatible pointer type [-Wincompatible-pointer-types] 282 | len = ofi_iov_bytes_to_copy(&iov[i], &size, &offset, &hmem_buf); | ^~~~~~~ | | | uint64_t * {aka long long unsigned int *} ./include/ofi_iov.h:87:47: note: expected 'size_t *' {aka 'long unsigned int *'} but argument is of type 'uint64_t *' {aka 'long long unsigned int *'} 87 | size_t *size, size_t *offset, char **copy_buf) | ~~~~~~~~^~~~~~ _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://urldefense.us/v3/__https://lists.mpich.org/mailman/listinfo/discuss…

1 0

Failure to compile with gcc/14
by Victor Eijkhout 28 Jun '24

28 Jun '24

1 0

Re: [mpich-discuss] Fail on MPI_Wait
by Palmer, Bruce J 26 Jun '24

26 Jun '24

Apparently I was having some issues with respect to the libraries that were getting configured when I built MPICH versus the libraries that were available on the compute nodes. I’ve got that sorted out. I configure with --with-device=ch4:ofi and got rid of the --without-ucx flag. I’m still getting a crash on MPI_Wait. If I don’t set FI_PROVIDER, then I get the error Bad : modifier in $ (-). Abort(944950159) on node 1: Fatal error in internal_Wait: Other MPI error, error stack: internal_Wait(68205)..........: MPI_Wait(request=0x5e88690, status=0x7ffc63581a40) failed MPIR_Wait(780)................: MPIR_Wait_state(737)..........: MPIDI_progress_test(134)......: MPIDI_OFI_handle_cq_error(791): OFI poll failed (ofi_events.c:793:MPIDI_OFI_handle_cq_error:Input/output error) If I set FI_PROVIDER=verbs, I get Abort(673331087) on node 5 (rank 5 in comm 496): Fatal error in internal_Recv: Other MPI error, error stack: internal_Recv(61700)..........: MPI_Recv(buf=0x2b4e0fa00000, count=4194304, MPI_CHAR, 0, 27624, comm=0x84000001, status=0x7ffdb2e9bdd0) failed MPIR_Wait(780)................: MPIR_Wait_state(737)..........: MPIDI_progress_test(134)......: MPIDI_OFI_handle_cq_error(791): OFI poll failed (ofi_events.c:793:MPIDI_OFI_handle_cq_error:Bad address) This error appears on ranks 2 and 5 of a 6 processor job on 2 nodes, but the remaining nodes still appear to be failing on an MPI_Wait call. From: Zhou, Hui <zhouh(a)anl.gov> Date: Friday, June 21, 2024 at 11:03 AM To: Palmer, Bruce J <Bruce.Palmer(a)pnnl.gov>, discuss(a)mpich.org <discuss(a)mpich.org> Subject: Re: Fail on MPI_Wait >../configure --prefix=/people/d3g293/mpich/mpich-4.2.1/build_decptn/install \ --with-device=ch4:ofi:sockets --with-libfabric=embedded \ --without-ucx CC=gcc CXX=g++ You are statically using "sockets" provider. Try --with-device=ch4:ofi Hui ________________________________ From: Palmer, Bruce J <Bruce.Palmer(a)pnnl.gov> Sent: Friday, June 21, 2024 12:27 PM To: Zhou, Hui <zhouh(a)anl.gov>; discuss(a)mpich.org <discuss(a)mpich.org> Subject: RE: Fail on MPI_Wait Hui, When I set FI_PROVIDER=tcp, the code crashes in MPI_Init. Specifically, this code will fail on one process: #include "mpi. h" int main(int argc, char **argv) { MPI_Init(&argc, &argv); MPI_Finalize(); } I’m running on a ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Hui, When I set FI_PROVIDER=tcp, the code crashes in MPI_Init. Specifically, this code will fail on one process: #include "mpi.h" int main(int argc, char **argv) { MPI_Init(&argc, &argv); MPI_Finalize(); } I’m running on a system with the following modules [d3g293@deception02 testing]$ module list Currently Loaded Modulefiles: 1) gcc/11.2.0 3) python/3.7.0 5) mkl/2019u4 2) cmake/3.21.4 4) git/2.42.0(default) 6) cuda/11.8 and a home-built version of mpich-4.2.1 configured with ../configure --prefix=/people/d3g293/mpich/mpich-4.2.1/build_decptn/install \ --with-device=ch4:ofi:sockets --with-libfabric=embedded \ --without-ucx CC=gcc CXX=g++ I thought it might have something to do with using a build configuration in my application build that is set up to include Cuda, but it also fails in MPI_Init with a non-Cuda configuraton if I set the FI_PROVIDER variable. Bruce From: Zhou, Hui <zhouh(a)anl.gov> Sent: Friday, June 14, 2024 9:27 AM To: Palmer, Bruce J <Bruce.Palmer(a)pnnl.gov>; discuss(a)mpich.org Subject: Re: Fail on MPI_Wait Never mind. It is v4.2.1. ________________________________ From: Zhou, Hui <zhouh(a)anl.gov<mailto:[email protected]>> Sent: Friday, June 14, 2024 11:26 AM To: Palmer, Bruce J <Bruce.Palmer(a)pnnl.gov<mailto:[email protected]>>; discuss(a)mpich.org<mailto:[email protected]> <discuss(a)mpich.org<mailto:[email protected]>> Subject: Re: Fail on MPI_Wait Bruce, What is the mpich version, BTW? -- Hui ________________________________ From: Zhou, Hui <zhouh(a)anl.gov<mailto:[email protected]>> Sent: Friday, June 14, 2024 10:55 AM To: Palmer, Bruce J <Bruce.Palmer(a)pnnl.gov<mailto:[email protected]>>; discuss(a)mpich.org<mailto:[email protected]> <discuss(a)mpich.org<mailto:[email protected]>> Subject: Re: Fail on MPI_Wait Bruce, You are using the sockets provider. Could try set FI_PROVIDER=tcp to see if it makes a difference? Meanwhile, if you can get a small reproducer – with the sockets provider or any provider, I'll try to debug it. It is difficult to guess the true source of the issue without a reproducer. -- Hui ________________________________ From: Palmer, Bruce J <Bruce.Palmer(a)pnnl.gov<mailto:[email protected]>> Sent: Friday, June 14, 2024 10:47 AM To: Zhou, Hui <zhouh(a)anl.gov<mailto:[email protected]>>; discuss(a)mpich.org<mailto:[email protected]> <discuss(a)mpich.org<mailto:[email protected]>> Subject: Re: Fail on MPI_Wait The output to standard out from running on 2 nodes and one process per node is attached. From: Zhou, Hui <zhouh@ anl. gov> Date: Tuesday, June 11, 2024 at 5: 49 PM To: discuss@ mpich. org <discuss@ mpich. org> Cc: Palmer, Bruce J <Bruce. Palmer@ pnnl. gov> ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd The output to standard out from running on 2 nodes and one process per node is attached. From: Zhou, Hui <zhouh(a)anl.gov<mailto:[email protected]>> Date: Tuesday, June 11, 2024 at 5:49 PM To: discuss(a)mpich.org<mailto:[email protected]> <discuss(a)mpich.org<mailto:[email protected]>> Cc: Palmer, Bruce J <Bruce.Palmer(a)pnnl.gov<mailto:[email protected]>> Subject: Re: Fail on MPI_Wait >MPIDI_OFI_handle_cq_error(793): OFI poll failed (ofi_events.c:793:MPIDI_OFI_handle_cq_error:Input/output error) This is an error coming from the libfabric provider. First we need find out which provider are you using. Try set environment variable MPIR_CVAR_DEBUG_SUMMARY=1 and run a simple MPI_INIT+MPI_Finalize test code. Could post its console output? -- Hui ________________________________ From: Palmer, Bruce J via discuss <discuss(a)mpich.org<mailto:[email protected]>> Sent: Tuesday, June 11, 2024 3:17 PM To: discuss(a)mpich.org<mailto:[email protected]> <discuss(a)mpich.org<mailto:[email protected]>> Cc: Palmer, Bruce J <Bruce.Palmer(a)pnnl.gov<mailto:[email protected]>> Subject: [mpich-discuss] Fail on MPI_Wait Hi, I’m trying to debug a GPU-aware runtime for the Global Arrays library. We had a version of this working a while ago, but it has mysteriously started failing and we are trying to track down why. Currently, we are getting failures in MPI_Wait ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Hi, I’m trying to debug a GPU-aware runtime for the Global Arrays library. We had a version of this working a while ago, but it has mysteriously started failing and we are trying to track down why. Currently, we are getting failures in MPI_Wait and were wondering if anyone could provide some information on what exactly seems to be failing inside the wait call. The error we are getting is Abort(206752655) on node 0: Fatal error in internal_Wait: Other MPI error, error stack: internal_Wait(68205)..........: MPI_Wait(request=0x500847a0, status=0x7ffff9331800) failed MPIR_Wait(780)................: MPIR_Wait_state(737)..........: MPIDI_progress_test(134)......: MPIDI_OFI_handle_cq_error(793): OFI poll failed (ofi_events.c:793:MPIDI_OFI_handle_cq_error:Input/output error) I’ve verified that the handle corresponding to 0x500847a0 is getting set earlier in the code in an MPI_Isend call and that no MPI_Wait or MPI_Test is called on the handle before it crashes with the above error message. I’m using MPICH 4.2.1 using gcc/8.3.0. The MPICH library was configured with ../configure --prefix=/people/d3g293/mpich/mpich-4.2.1/build_newell/install \ --with-device=ch4:ofi:sockets --with-libfabric=embedded \ --without-ucx --enable-threads=multiple --with-slurm \ CC=gcc CXX=g+ I’ve tried building with UCX and gotten the same results. Are these errors indicative of corruption of the request handle or problems with some internal MPI data structures or something else? Any information you can provide would be appreciated. Thanks, Bruce

1 0

Hang after MPI_Intercomm_merge with OFI+provider verbs
by Iker Martín Álvarez 25 Jun '24

25 Jun '24

Hello, Lately, I have encountered an issue with the usage of MPI_Comm_spawn + MPI_Intercomm_merge, where in some occasions my application hangs indefinitely. I was hoping that this forum could advise me on a solution. For my tests, I run the code with 2 processes and give as an argument the amount of processes to spawn in MPI_Comm_spawn, where I choose 20. In my experimentation, 3 to 4 out of 100 executions hang indefinitely at the function MPI_Barrier. Nevertheless, if I remove both MPI_Intercomm_merge functions, the code always finalizes. Also, prior to executing the code I set the environment variable FI_PROVIDER=verbs. However, if I change the provider to "tcp", or "udp", the code doesn't hang, but the performance is lower for other providers different to "verbs". The MPICH version used is 4.2.1 with the following configure line: ./configure --prefix=... --with-device=ch4:ofi --disable-psm3 In addition, I have attached a file <https://urldefense.us/v3/__https://lorca.act.uji.es/gitlab/martini/mpich_of… > with the output of executing mpirun with "MPIR_CVAR_DEBUG_SUMMARY=1" and unsetting FI_PROVIDER. The minimal code <https://urldefense.us/v3/__https://lorca.act.uji.es/gitlab/martini/mpich_of… > to reproduce the problem is the following: =============== #include <stdio.h> #include <stdlib.h> #include <mpi.h> int main(int argc, char* argv[]) { MPI_Init(&argc, &argv); MPI_Comm spawn, new_comm; MPI_Comm_get_parent(&spawn); if(spawn == MPI_COMM_NULL) { int num_c = atoi(argv[1]); MPI_Comm_spawn(argv[0], MPI_ARGV_NULL, num_c, MPI_INFO_NULL, 0, MPI_COMM_WORLD, &spawn, MPI_ERRCODES_IGNORE); MPI_Intercomm_merge(spawn, 0, &new_comm); } else { MPI_Intercomm_merge(spawn, 1, &new_comm); MPI_Barrier(MPI_COMM_WORLD); } MPI_Finalize(); } =============== Thank you for your help. Best regards, Iker

1 0

Re: [mpich-discuss] Fail on MPI_Wait
by Zhou, Hui 21 Jun '24

21 Jun '24

>../configure --prefix=/people/d3g293/mpich/mpich-4.2.1/build_decptn/install \ --with-device=ch4:ofi:sockets --with-libfabric=embedded \ --without-ucx CC=gcc CXX=g++ You are statically using "sockets" provider. Try --with-device=ch4:ofi Hui ________________________________ From: Palmer, Bruce J <Bruce.Palmer(a)pnnl.gov> Sent: Friday, June 21, 2024 12:27 PM To: Zhou, Hui <zhouh(a)anl.gov>; discuss(a)mpich.org <discuss(a)mpich.org> Subject: RE: Fail on MPI_Wait Hui, When I set FI_PROVIDER=tcp, the code crashes in MPI_Init. Specifically, this code will fail on one process: #include "mpi. h" int main(int argc, char **argv) { MPI_Init(&argc, &argv); MPI_Finalize(); } I’m running on a ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Hui, When I set FI_PROVIDER=tcp, the code crashes in MPI_Init. Specifically, this code will fail on one process: #include "mpi.h" int main(int argc, char **argv) { MPI_Init(&argc, &argv); MPI_Finalize(); } I’m running on a system with the following modules [d3g293@deception02 testing]$ module list Currently Loaded Modulefiles: 1) gcc/11.2.0 3) python/3.7.0 5) mkl/2019u4 2) cmake/3.21.4 4) git/2.42.0(default) 6) cuda/11.8 and a home-built version of mpich-4.2.1 configured with ../configure --prefix=/people/d3g293/mpich/mpich-4.2.1/build_decptn/install \ --with-device=ch4:ofi:sockets --with-libfabric=embedded \ --without-ucx CC=gcc CXX=g++ I thought it might have something to do with using a build configuration in my application build that is set up to include Cuda, but it also fails in MPI_Init with a non-Cuda configuraton if I set the FI_PROVIDER variable. Bruce From: Zhou, Hui <zhouh(a)anl.gov> Sent: Friday, June 14, 2024 9:27 AM To: Palmer, Bruce J <Bruce.Palmer(a)pnnl.gov>; discuss(a)mpich.org Subject: Re: Fail on MPI_Wait Never mind. It is v4.2.1. ________________________________ From: Zhou, Hui <zhouh(a)anl.gov<mailto:[email protected]>> Sent: Friday, June 14, 2024 11:26 AM To: Palmer, Bruce J <Bruce.Palmer(a)pnnl.gov<mailto:[email protected]>>; discuss(a)mpich.org<mailto:[email protected]> <discuss(a)mpich.org<mailto:[email protected]>> Subject: Re: Fail on MPI_Wait Bruce, What is the mpich version, BTW? -- Hui ________________________________ From: Zhou, Hui <zhouh(a)anl.gov<mailto:[email protected]>> Sent: Friday, June 14, 2024 10:55 AM To: Palmer, Bruce J <Bruce.Palmer(a)pnnl.gov<mailto:[email protected]>>; discuss(a)mpich.org<mailto:[email protected]> <discuss(a)mpich.org<mailto:[email protected]>> Subject: Re: Fail on MPI_Wait Bruce, You are using the sockets provider. Could try set FI_PROVIDER=tcp to see if it makes a difference? Meanwhile, if you can get a small reproducer – with the sockets provider or any provider, I'll try to debug it. It is difficult to guess the true source of the issue without a reproducer. -- Hui ________________________________ From: Palmer, Bruce J <Bruce.Palmer(a)pnnl.gov<mailto:[email protected]>> Sent: Friday, June 14, 2024 10:47 AM To: Zhou, Hui <zhouh(a)anl.gov<mailto:[email protected]>>; discuss(a)mpich.org<mailto:[email protected]> <discuss(a)mpich.org<mailto:[email protected]>> Subject: Re: Fail on MPI_Wait The output to standard out from running on 2 nodes and one process per node is attached. From: Zhou, Hui <zhouh@ anl. gov> Date: Tuesday, June 11, 2024 at 5: 49 PM To: discuss@ mpich. org <discuss@ mpich. org> Cc: Palmer, Bruce J <Bruce. Palmer@ pnnl. gov> ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd The output to standard out from running on 2 nodes and one process per node is attached. From: Zhou, Hui <zhouh(a)anl.gov<mailto:[email protected]>> Date: Tuesday, June 11, 2024 at 5:49 PM To: discuss(a)mpich.org<mailto:[email protected]> <discuss(a)mpich.org<mailto:[email protected]>> Cc: Palmer, Bruce J <Bruce.Palmer(a)pnnl.gov<mailto:[email protected]>> Subject: Re: Fail on MPI_Wait >MPIDI_OFI_handle_cq_error(793): OFI poll failed (ofi_events.c:793:MPIDI_OFI_handle_cq_error:Input/output error) This is an error coming from the libfabric provider. First we need find out which provider are you using. Try set environment variable MPIR_CVAR_DEBUG_SUMMARY=1 and run a simple MPI_INIT+MPI_Finalize test code. Could post its console output? -- Hui ________________________________ From: Palmer, Bruce J via discuss <discuss(a)mpich.org<mailto:[email protected]>> Sent: Tuesday, June 11, 2024 3:17 PM To: discuss(a)mpich.org<mailto:[email protected]> <discuss(a)mpich.org<mailto:[email protected]>> Cc: Palmer, Bruce J <Bruce.Palmer(a)pnnl.gov<mailto:[email protected]>> Subject: [mpich-discuss] Fail on MPI_Wait Hi, I’m trying to debug a GPU-aware runtime for the Global Arrays library. We had a version of this working a while ago, but it has mysteriously started failing and we are trying to track down why. Currently, we are getting failures in MPI_Wait ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Hi, I’m trying to debug a GPU-aware runtime for the Global Arrays library. We had a version of this working a while ago, but it has mysteriously started failing and we are trying to track down why. Currently, we are getting failures in MPI_Wait and were wondering if anyone could provide some information on what exactly seems to be failing inside the wait call. The error we are getting is Abort(206752655) on node 0: Fatal error in internal_Wait: Other MPI error, error stack: internal_Wait(68205)..........: MPI_Wait(request=0x500847a0, status=0x7ffff9331800) failed MPIR_Wait(780)................: MPIR_Wait_state(737)..........: MPIDI_progress_test(134)......: MPIDI_OFI_handle_cq_error(793): OFI poll failed (ofi_events.c:793:MPIDI_OFI_handle_cq_error:Input/output error) I’ve verified that the handle corresponding to 0x500847a0 is getting set earlier in the code in an MPI_Isend call and that no MPI_Wait or MPI_Test is called on the handle before it crashes with the above error message. I’m using MPICH 4.2.1 using gcc/8.3.0. The MPICH library was configured with ../configure --prefix=/people/d3g293/mpich/mpich-4.2.1/build_newell/install \ --with-device=ch4:ofi:sockets --with-libfabric=embedded \ --without-ucx --enable-threads=multiple --with-slurm \ CC=gcc CXX=g+ I’ve tried building with UCX and gotten the same results. Are these errors indicative of corruption of the request handle or problems with some internal MPI data structures or something else? Any information you can provide would be appreciated. Thanks, Bruce

1 0

Re: [mpich-discuss] Fail on MPI_Wait
by Palmer, Bruce J 21 Jun '24

21 Jun '24

Hui, When I set FI_PROVIDER=tcp, the code crashes in MPI_Init. Specifically, this code will fail on one process: #include "mpi.h" int main(int argc, char **argv) { MPI_Init(&argc, &argv); MPI_Finalize(); } I’m running on a system with the following modules [d3g293@deception02 testing]$ module list Currently Loaded Modulefiles: 1) gcc/11.2.0 3) python/3.7.0 5) mkl/2019u4 2) cmake/3.21.4 4) git/2.42.0(default) 6) cuda/11.8 and a home-built version of mpich-4.2.1 configured with ../configure --prefix=/people/d3g293/mpich/mpich-4.2.1/build_decptn/install \ --with-device=ch4:ofi:sockets --with-libfabric=embedded \ --without-ucx CC=gcc CXX=g++ I thought it might have something to do with using a build configuration in my application build that is set up to include Cuda, but it also fails in MPI_Init with a non-Cuda configuraton if I set the FI_PROVIDER variable. Bruce From: Zhou, Hui <zhouh(a)anl.gov> Sent: Friday, June 14, 2024 9:27 AM To: Palmer, Bruce J <Bruce.Palmer(a)pnnl.gov>; discuss(a)mpich.org Subject: Re: Fail on MPI_Wait Never mind. It is v4.2.1. ________________________________ From: Zhou, Hui <zhouh(a)anl.gov<mailto:[email protected]>> Sent: Friday, June 14, 2024 11:26 AM To: Palmer, Bruce J <Bruce.Palmer(a)pnnl.gov<mailto:[email protected]>>; discuss(a)mpich.org<mailto:[email protected]> <discuss(a)mpich.org<mailto:[email protected]>> Subject: Re: Fail on MPI_Wait Bruce, What is the mpich version, BTW? -- Hui ________________________________ From: Zhou, Hui <zhouh(a)anl.gov<mailto:[email protected]>> Sent: Friday, June 14, 2024 10:55 AM To: Palmer, Bruce J <Bruce.Palmer(a)pnnl.gov<mailto:[email protected]>>; discuss(a)mpich.org<mailto:[email protected]> <discuss(a)mpich.org<mailto:[email protected]>> Subject: Re: Fail on MPI_Wait Bruce, You are using the sockets provider. Could try set FI_PROVIDER=tcp to see if it makes a difference? Meanwhile, if you can get a small reproducer – with the sockets provider or any provider, I'll try to debug it. It is difficult to guess the true source of the issue without a reproducer. -- Hui ________________________________ From: Palmer, Bruce J <Bruce.Palmer(a)pnnl.gov<mailto:[email protected]>> Sent: Friday, June 14, 2024 10:47 AM To: Zhou, Hui <zhouh(a)anl.gov<mailto:[email protected]>>; discuss(a)mpich.org<mailto:[email protected]> <discuss(a)mpich.org<mailto:[email protected]>> Subject: Re: Fail on MPI_Wait The output to standard out from running on 2 nodes and one process per node is attached. From: Zhou, Hui <zhouh@ anl. gov> Date: Tuesday, June 11, 2024 at 5: 49 PM To: discuss@ mpich. org <discuss@ mpich. org> Cc: Palmer, Bruce J <Bruce. Palmer@ pnnl. gov> ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd The output to standard out from running on 2 nodes and one process per node is attached. From: Zhou, Hui <zhouh(a)anl.gov<mailto:[email protected]>> Date: Tuesday, June 11, 2024 at 5:49 PM To: discuss(a)mpich.org<mailto:[email protected]> <discuss(a)mpich.org<mailto:[email protected]>> Cc: Palmer, Bruce J <Bruce.Palmer(a)pnnl.gov<mailto:[email protected]>> Subject: Re: Fail on MPI_Wait >MPIDI_OFI_handle_cq_error(793): OFI poll failed (ofi_events.c:793:MPIDI_OFI_handle_cq_error:Input/output error) This is an error coming from the libfabric provider. First we need find out which provider are you using. Try set environment variable MPIR_CVAR_DEBUG_SUMMARY=1 and run a simple MPI_INIT+MPI_Finalize test code. Could post its console output? -- Hui ________________________________ From: Palmer, Bruce J via discuss <discuss(a)mpich.org<mailto:[email protected]>> Sent: Tuesday, June 11, 2024 3:17 PM To: discuss(a)mpich.org<mailto:[email protected]> <discuss(a)mpich.org<mailto:[email protected]>> Cc: Palmer, Bruce J <Bruce.Palmer(a)pnnl.gov<mailto:[email protected]>> Subject: [mpich-discuss] Fail on MPI_Wait Hi, I’m trying to debug a GPU-aware runtime for the Global Arrays library. We had a version of this working a while ago, but it has mysteriously started failing and we are trying to track down why. Currently, we are getting failures in MPI_Wait ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Hi, I’m trying to debug a GPU-aware runtime for the Global Arrays library. We had a version of this working a while ago, but it has mysteriously started failing and we are trying to track down why. Currently, we are getting failures in MPI_Wait and were wondering if anyone could provide some information on what exactly seems to be failing inside the wait call. The error we are getting is Abort(206752655) on node 0: Fatal error in internal_Wait: Other MPI error, error stack: internal_Wait(68205)..........: MPI_Wait(request=0x500847a0, status=0x7ffff9331800) failed MPIR_Wait(780)................: MPIR_Wait_state(737)..........: MPIDI_progress_test(134)......: MPIDI_OFI_handle_cq_error(793): OFI poll failed (ofi_events.c:793:MPIDI_OFI_handle_cq_error:Input/output error) I’ve verified that the handle corresponding to 0x500847a0 is getting set earlier in the code in an MPI_Isend call and that no MPI_Wait or MPI_Test is called on the handle before it crashes with the above error message. I’m using MPICH 4.2.1 using gcc/8.3.0. The MPICH library was configured with ../configure --prefix=/people/d3g293/mpich/mpich-4.2.1/build_newell/install \ --with-device=ch4:ofi:sockets --with-libfabric=embedded \ --without-ucx --enable-threads=multiple --with-slurm \ CC=gcc CXX=g+ I’ve tried building with UCX and gotten the same results. Are these errors indicative of corruption of the request handle or problems with some internal MPI data structures or something else? Any information you can provide would be appreciated. Thanks, Bruce

1 0

Re: [mpich-discuss] Mpich over RDMA sample
by Niyaz Murshed 18 Jun '24

18 Jun '24

Unfortunately, still couldn’t make it work. Any more debugging tips? It does work if I just use 1 host FI_PROVIDER=verbs MPIR_CVAR_DEBUG_SUMMARY=1 /opt/mpich/bin/mpirun -n 5 -hosts 192.168.2.200 ./a.out However, when I see what is on the wire, it is TCP messages. I am looking for RoCE messages on it. root@ampere-altra-2-1:/mpich/examples# FI_PROVIDER=verbs MPIR_CVAR_DEBUG_SUMMARY=1 /opt/mpich/bin/mpirun -n 5 -hosts 192.168.2.200,192.168.2.100 ./a.out ==== Various sizes and limits ==== sizeof(MPIDI_per_vci_t): 192 Required minimum FI_VERSION: 0, current version: 10015 provider: verbs, score = 0, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200 provider: verbs, score = 0, pref = 0, FI_SOCKADDR_IN6 [28] fe80::526b:4bff:fefc:1342 provider: verbs, score = 0, pref = 0, FI_SOCKADDR_IB [48] provider: verbs, score = 0, pref = 0, FI_SOCKADDR_IB [48] provider: verbs, score = 0, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200 provider: verbs, score = 0, pref = 0, FI_SOCKADDR_IN6 [28] fe80::526b:4bff:fefc:1342 provider: verbs, score = 0, pref = 0, FI_SOCKADDR_IB [48] provider: verbs, score = 0, pref = 0, FI_SOCKADDR_IB [48] provider: verbs, score = 0, pref = 0, FI_FORMAT_UNSPEC [32] provider: verbs, score = 0, pref = 0, FI_FORMAT_UNSPEC [32] provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200 provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::526b:4bff:fefc:1342 provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48] provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48] provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200 provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::526b:4bff:fefc:1342 provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48] provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48] provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200 provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::526b:4bff:fefc:1342 provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48] provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48] provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200 provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::526b:4bff:fefc:1342 provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48] provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48] provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200 provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::526b:4bff:fefc:1342 provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48] provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48] provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200 provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::526b:4bff:fefc:1342 provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48] provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48] provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200 provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::526b:4bff:fefc:1342 provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48] provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48] provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200 provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::526b:4bff:fefc:1342 provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48] provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48] provider: verbs;ofi_rxd, score = 5, pref = -2, FI_FORMAT_UNSPEC [32] provider: verbs;ofi_rxd, score = 5, pref = -2, FI_FORMAT_UNSPEC [32] Required minimum FI_VERSION: 10006, current version: 10015 ==== Capability set configuration ==== libfabric provider: verbs;ofi_rxm - IB-0xfe80000000000000 MPIDI_OFI_ENABLE_DATA: 0 MPIDI_OFI_ENABLE_AV_TABLE: 1 MPIDI_OFI_ENABLE_SCALABLE_ENDPOINTS: 0 MPIDI_OFI_ENABLE_SHARED_CONTEXTS: 0 MPIDI_OFI_ENABLE_MR_VIRT_ADDRESS: 1 MPIDI_OFI_ENABLE_MR_ALLOCATED: 1 MPIDI_OFI_ENABLE_MR_REGISTER_NULL: 1 MPIDI_OFI_ENABLE_MR_PROV_KEY: 1 MPIDI_OFI_ENABLE_TAGGED: 1 MPIDI_OFI_ENABLE_AM: 1 MPIDI_OFI_ENABLE_RMA: 1 MPIDI_OFI_ENABLE_ATOMICS: 0 MPIDI_OFI_FETCH_ATOMIC_IOVECS: 1 MPIDI_OFI_ENABLE_DATA_AUTO_PROGRESS: 0 MPIDI_OFI_ENABLE_CONTROL_AUTO_PROGRESS: 0 MPIDI_OFI_ENABLE_PT2PT_NOPACK: 1 MPIDI_OFI_ENABLE_TRIGGERED: 0 MPIDI_OFI_ENABLE_HMEM: 0 MPIDI_OFI_NUM_AM_BUFFERS: 8 MPIDI_OFI_NUM_OPTIMIZED_MEMORY_REGIONS: 0 MPIDI_OFI_CONTEXT_BITS: 16 MPIDI_OFI_SOURCE_BITS: 23 MPIDI_OFI_TAG_BITS: 20 MPIDI_OFI_VNI_USE_DOMAIN: 1 MAXIMUM SUPPORTED RANKS: 8388608 MAXIMUM TAG: 1048576 ==== Provider global thresholds ==== max_buffered_send: 192 max_buffered_write: 192 max_msg_size: 1073741824 max_order_raw: 1073741824 max_order_war: 0 max_order_waw: 1073741824 tx_iov_limit: 4 rx_iov_limit: 4 rma_iov_limit: 1 max_mr_key_size: 4 ==== Various sizes and limits ==== MPIDI_OFI_AM_MSG_HEADER_SIZE: 24 MPIDI_OFI_MAX_AM_HDR_SIZE: 255 sizeof(MPIDI_OFI_am_request_header_t): 416 sizeof(MPIDI_OFI_per_vci_t): 52480 MPIDI_OFI_AM_HDR_POOL_CELL_SIZE: 1024 MPIDI_OFI_DEFAULT_SHORT_SEND_SIZE: 16384 Assertion failed in file src/mpid/ch4/netmod/ofi/init_addrxchg.c at line 151: mapped_table[i] != FI_ADDR_NOTAVAIL Assertion failed in file src/mpid/ch4/netmod/ofi/init_addrxchg.c at line 151: mapped_table[i] != FI_ADDR_NOTAVAIL Assertion failed in file src/mpid/ch4/netmod/ofi/init_addrxchg.c at line 151: mapped_table[i] != FI_ADDR_NOTAVAIL Assertion failed in file src/mpid/ch4/netmod/ofi/init_addrxchg.c at line 151: mapped_table[i] != FI_ADDR_NOTAVAIL Assertion failed in file src/mpid/ch4/netmod/ofi/init_addrxchg.c at line 151: mapped_table[i] != FI_ADDR_NOTAVAIL /opt/mpich/lib/libmpi.so.0(+0x58c1b5) [0x7fdf0919f1b5] /opt/mpich/lib/libmpi.so.0(+0x491858) [0x7fdf090a4858] /opt/mpich/lib/libmpi.so.0(+0x55443c) [0x7fdf0916743c] /opt/mpich/lib/libmpi.so.0(+0x53409d) [0x7fdf0914709d] /opt/mpich/lib/libmpi.so.0(+0x4dc78f) [0x7fdf090ef78f] /opt/mpich/lib/libmpi.so.0(+0x4df10a) [0x7fdf090f210a] /opt/mpich/lib/libmpi.so.0(+0x3deab6) [0x7fdf08ff1ab6] /opt/mpich/lib/libmpi.so.0(+0x3e0732) [0x7fdf08ff3732] /opt/mpich/lib/libmpi.so.0(+0x3dd075) [0x7fdf08ff0075] /opt/mpich/lib/libmpi.so.0(+0x418215) [0x7fdf0902b215] /opt/mpich/lib/libmpi.so.0(+0x4188fa) [0x7fdf0902b8fa] /opt/mpich/lib/libmpi.so.0(+0x58c1b5) [0x7f3a84bfb1b5] /opt/mpich/lib/libmpi.so.0(+0x491858) [0x7f3a84b00858] /opt/mpich/lib/libmpi.so.0(+0x55443c) [0x7f3a84bc343c] /opt/mpich/lib/libmpi.so.0(+0x53409d) [0x7f3a84ba309d] /opt/mpich/lib/libmpi.so.0(+0x4dc78f) [0x7f3a84b4b78f] /opt/mpich/lib/libmpi.so.0(+0x4df10a) [0x7f3a84b4e10a] /opt/mpich/lib/libmpi.so.0(+0x3deab6) [0x7f3a84a4dab6] /opt/mpich/lib/libmpi.so.0(+0x3e0732) [0x7f3a84a4f732] /opt/mpich/lib/libmpi.so.0(+0x3dd075) [0x7f3a84a4c075] /opt/mpich/lib/libmpi.so.0(+0x418215) [0x7f3a84a87215] /opt/mpich/lib/libmpi.so.0(+0x4188fa) [0x7f3a84a878fa] /opt/mpich/lib/libmpi.so.0(MPI_Init+0x34) [0x7f3a84746594] ./a.out(+0x121a) [0x5606031f021a] /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7f3a8446bd90] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80) [0x7f3a8446be40] /opt/mpich/lib/libmpi.so.0(MPI_Init+0x34) [0x7fdf08cea594] ./a.out(+0x121a) [0x55accc98e21a] /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7fdf08a0fd90] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80) [0x7fdf08a0fe40] ./a.out(+0x1125) [0x55accc98e125] Abort(1) on node 2: Internal error /opt/mpich/lib/libmpi.so.0(+0x58c1b5) [0x7fa1131a11b5] /opt/mpich/lib/libmpi.so.0(+0x491858) [0x7fa1130a6858] /opt/mpich/lib/libmpi.so.0(+0x55443c) [0x7fa11316943c] /opt/mpich/lib/libmpi.so.0(+0x53409d) [0x7fa11314909d] /opt/mpich/lib/libmpi.so.0(+0x4dc78f) [0x7fa1130f178f] /opt/mpich/lib/libmpi.so.0(+0x4df10a) [0x7fa1130f410a] /opt/mpich/lib/libmpi.so.0(+0x3deab6) [0x7fa112ff3ab6] /opt/mpich/lib/libmpi.so.0(+0x3e0732) [0x7fa112ff5732] /opt/mpich/lib/libmpi.so.0(+0x3dd075) [0x7fa112ff2075] /opt/mpich/lib/libmpi.so.0(+0x418215) [0x7fa11302d215] /opt/mpich/lib/libmpi.so.0(+0x4188fa) [0x7fa11302d8fa] /opt/mpich/lib/libmpi.so.0(MPI_Init+0x34) [0x7fa112cec594] ./a.out(+0x121a) [0x55a0b562221a] /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7fa112a11d90] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80) [0x7fa112a11e40] ./a.out(+0x1125) [0x55a0b5622125] Abort(1) on node 4: Internal error ./a.out(+0x1125) [0x5606031f0125] Abort(1) on node 0: Internal error /opt/mpich/lib/libmpi.so.0(+0x59d26c) [0xffffb4dbd26c] /opt/mpich/lib/libmpi.so.0(+0x4a6b58) [0xffffb4cc6b58] /opt/mpich/lib/libmpi.so.0(+0x5678b0) [0xffffb4d878b0] /opt/mpich/lib/libmpi.so.0(+0x548c54) [0xffffb4d68c54] /opt/mpich/lib/libmpi.so.0(+0x4f710c) [0xffffb4d1710c] /opt/mpich/lib/libmpi.so.0(+0x4f9890) [0xffffb4d19890] /opt/mpich/lib/libmpi.so.0(+0x59d26c) [0xffffa9e9d26c] /opt/mpich/lib/libmpi.so.0(+0x4a6b58) [0xffffa9da6b58] /opt/mpich/lib/libmpi.so.0(+0x5678b0) [0xffffa9e678b0] /opt/mpich/lib/libmpi.so.0(+0x548c54) [0xffffa9e48c54] /opt/mpich/lib/libmpi.so.0(+0x4f710c) [0xffffa9df710c] /opt/mpich/lib/libmpi.so.0(+0x4f9890) [0xffffa9df9890] /opt/mpich/lib/libmpi.so.0(+0x3ffd2c) [0xffffa9cffd2c] /opt/mpich/lib/libmpi.so.0(+0x4017ec) [0xffffa9d017ec] /opt/mpich/lib/libmpi.so.0(+0x3fe384) [0xffffa9cfe384] /opt/mpich/lib/libmpi.so.0(+0x436a64) [0xffffa9d36a64] /opt/mpich/lib/libmpi.so.0(+0x43700c) [0xffffa9d3700c] /opt/mpich/lib/libmpi.so.0(MPI_Init+0x44) [0xffffa9a2eeb4] ./a.out(+0x9c4) [0xaaaabd4709c4] /lib/aarch64-linux-gnu/libc.so.6(+0x273fc) [0xffffa97773fc] /lib/aarch64-linux-gnu/libc.so.6(__libc_start_main+0x98) [0xffffa97774cc] ./a.out(+0x8b0) [0xaaaabd4708b0] /opt/mpich/lib/libmpi.so.0(+0x3ffd2c) [0xffffb4c1fd2c] /opt/mpich/lib/libmpi.so.0(+0x4017ec) [0xffffb4c217ec] /opt/mpich/lib/libmpi.so.0(+0x3fe384) [0xffffb4c1e384] /opt/mpich/lib/libmpi.so.0(+0x436a64) [0xffffb4c56a64] /opt/mpich/lib/libmpi.so.0(+0x43700c) [0xffffb4c5700c] /opt/mpich/lib/libmpi.so.0(MPI_Init+0x44) [0xffffb494eeb4] ./a.out(+0x9c4) [0xaaaab8e909c4] /lib/aarch64-linux-gnu/libc.so.6(+0x273fc) [0xffffb46973fc] /lib/aarch64-linux-gnu/libc.so.6(__libc_start_main+0x98) [0xffffb46974cc] ./a.out(+0x8b0) [0xaaaab8e908b0] Abort(1) on node 1: Internal error Abort(1) on node 3: Internal error a.out: prov/util/src/util_mem_monitor.c:172: ofi_monitor_cleanup: Assertion `dlist_empty(&monitor->list)' failed. a.out: prov/util/src/util_mem_monitor.c:172: ofi_monitor_cleanup: Assertion `dlist_empty(&monitor->list)' failed. =================================================================================== = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = PID 105 RUNNING AT 192.168.2.100 = EXIT CODE: 134 = CLEANING UP REMAINING PROCESSES = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES =================================================================================== YOUR APPLICATION TERMINATED WITH THE EXIT STRING: Aborted (signal 6) This typically refers to a problem with your application. Please see the FAQ page for debugging suggestions From: Niyaz Murshed via discuss <discuss(a)mpich.org> Date: Tuesday, June 18, 2024 at 11:13 AM To: Zhou, Hui <zhouh(a)anl.gov>, discuss(a)mpich.org <discuss(a)mpich.org> Cc: Niyaz Murshed <Niyaz.Murshed(a)arm.com>, nd <nd(a)arm.com> Subject: Re: [mpich-discuss] Mpich over RDMA sample Okay, let me keep digging. If verbs;ofi_rxm is good, then my understanding is that I should see the RoCE messages over the wire, correct? i. e once I can get the sample application working. Currently, we still see the error as below. I will ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Okay, let me keep digging. If verbs;ofi_rxm is good, then my understanding is that I should see the RoCE messages over the wire, correct? i.e once I can get the sample application working. Currently, we still see the error as below. I will try to use your patch for interface selection to see what happens. ==== Capability set configuration ==== libfabric provider: verbs;ofi_rxm - IB-0xfe80000000000000 Assertion failed in file src/mpid/ch4/netmod/ofi/init_addrxchg.c at line 151: mapped_table[i] != FI_ADDR_NOTAVAIL Assertion failed in file src/mpid/ch4/netmod/ofi/init_addrxchg.c at line 151: mapped_table[i] != FI_ADDR_NOTAVAIL Assertion failed in file src/mpid/ch4/netmod/ofi/init_addrxchg.c at line 151: mapped_table[i] != FI_ADDR_NOTAVAIL Assertion failed in file src/mpid/ch4/netmod/ofi/init_addrxchg.c at line 151: mapped_table[i] != FI_ADDR_NOTAVAIL Assertion failed in file src/mpid/ch4/netmod/ofi/init_addrxchg.c at line 151: mapped_table[i] != FI_ADDR_NOTAVAIL From: Zhou, Hui <zhouh(a)anl.gov> Date: Tuesday, June 18, 2024 at 11:05 AM To: Niyaz Murshed <Niyaz.Murshed(a)arm.com>, discuss(a)mpich.org <discuss(a)mpich.org> Cc: nd <nd(a)arm.com> Subject: Re: Mpich over RDMA sample Niyaz, FYI -https://urldefense.us/v3/__https://ofiwg.github.io/libfabric/v1.21.0/man/fi_provider.7.html__;!!G_uCfscf7eWS!cDbck3MzxE4MNvuedpeu36W-QEWamMuYg13g3KM2baHb-NZlWpKNLupIgqwWVrph458DsJyPiWyBSS3K0ZE$ <https://urldefense.us/v3/__https:/ofiwg.github.io/libfabric/v1.21.0/man/fi_…> Hui ________________________________ From: Niyaz Murshed <Niyaz.Murshed(a)arm.com> Sent: Tuesday, June 18, 2024 11:00 AM To: Zhou, Hui <zhouh(a)anl.gov>; discuss(a)mpich.org <discuss(a)mpich.org> Cc: nd <nd(a)arm.com> Subject: Re: Mpich over RDMA sample Hi Hui, When you say message semantic, do you mean endpoint type message ? Mpich is running over libfabric. When I run sample application on libfabric, it does have message endpoints over RMA. From: Zhou, Hui <zhouh@ anl. gov> Date: Tuesday, ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Hi Hui, When you say message semantic, do you mean endpoint type message ? Mpich is running over libfabric. When I run sample application on libfabric, it does have message endpoints over RMA. From: Zhou, Hui <zhouh(a)anl.gov> Date: Tuesday, June 18, 2024 at 9:44 AM To: Niyaz Murshed <Niyaz.Murshed(a)arm.com>, discuss(a)mpich.org <discuss(a)mpich.org> Cc: nd <nd(a)arm.com> Subject: Re: Mpich over RDMA sample The "verbs" provider alone does not provide the message semantics we needed for MPI. "ofi_rxm" is the utility provider that provides the message semantics. "verbs;ofi_rxm" is the provider combination you want. Hui ________________________________ From: Niyaz Murshed <Niyaz.Murshed(a)arm.com> Sent: Tuesday, June 18, 2024 9:14 AM To: Zhou, Hui <zhouh(a)anl.gov>; discuss(a)mpich.org <discuss(a)mpich.org> Cc: nd <nd(a)arm.com> Subject: Re: Mpich over RDMA sample Hi Hui, I have set the env variable to FI_PROVIDER=verbs , after thich I get the below. libfabric provider: verbs;ofi_rxm - IB-0xfe80000000000000 It does not chose just “verbs” , it selects verbs;ofi_rxf I am wanting to select the below: provider: ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Hi Hui, I have set the env variable to FI_PROVIDER=verbs , after thich I get the below. libfabric provider: verbs;ofi_rxm - IB-0xfe80000000000000 It does not chose just “verbs” , it selects verbs;ofi_rxf I am wanting to select the below: provider: verbs, score = 0, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200 I would really like to thank you for such fast responses. From: Zhou, Hui <zhouh(a)anl.gov> Date: Tuesday, June 18, 2024 at 8:58 AM To: Niyaz Murshed <Niyaz.Murshed(a)arm.com>, discuss(a)mpich.org <discuss(a)mpich.org> Cc: nd <nd(a)arm.com> Subject: Re: Mpich over RDMA sample Niyaz, Set environment variable FI_PROVIDER=verbs to select the verbs provider. Hui ________________________________ From: Niyaz Murshed <Niyaz.Murshed(a)arm.com> Sent: Monday, June 17, 2024 8:57 PM To: Zhou, Hui <zhouh(a)anl.gov>; discuss(a)mpich.org <discuss(a)mpich.org> Cc: nd <nd(a)arm.com> Subject: Re: Mpich over RDMA sample Hi Hui, I would still need some assistance in this. I am not able to make mpich chose verbs provider ☹ With the log : provider: verbs, score = 0, pref = 0, FI_SOCKADDR_IN [16] 192. 168. 2. 200 Is my assumption to think the above log represent ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Hi Hui, I would still need some assistance in this. I am not able to make mpich chose verbs provider ☹ With the log : provider: verbs, score = 0, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200 Is my assumption to think the above log represent that verbs provider knows about a path to 192.168.2.200 ? How does the application hello-world decide which provider to use? Running on server: 192.168.2.100 root@ampere-altra-2-1# FI_PROVIDER=verbs MPIR_CVAR_DEBUG_SUMMARY=1 mpirun -n 5 -hosts 192.168.2.200,192.168.2.100 /mpich/examples/a.out ==== Various sizes and limits ==== sizeof(MPIDI_per_vci_t): 192 Required minimum FI_VERSION: 0, current version: 10015 provider: verbs, score = 0, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200 provider: verbs, score = 0, pref = 0, FI_SOCKADDR_IN6 [28] fe80::526b:4bff:fefc:1342 provider: verbs, score = 0, pref = 0, FI_SOCKADDR_IB [48] provider: verbs, score = 0, pref = 0, FI_SOCKADDR_IB [48] provider: verbs, score = 0, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200 provider: verbs, score = 0, pref = 0, FI_SOCKADDR_IN6 [28] fe80::526b:4bff:fefc:1342 provider: verbs, score = 0, pref = 0, FI_SOCKADDR_IB [48] provider: verbs, score = 0, pref = 0, FI_SOCKADDR_IB [48] provider: verbs, score = 0, pref = 0, FI_FORMAT_UNSPEC [32] provider: verbs, score = 0, pref = 0, FI_FORMAT_UNSPEC [32] provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200 provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::526b:4bff:fefc:1342 provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48] provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48] provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200 provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::526b:4bff:fefc:1342 provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48] provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48] provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200 provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::526b:4bff:fefc:1342 provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48] provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48] provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200 provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::526b:4bff:fefc:1342 provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48] provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48] provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200 provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::526b:4bff:fefc:1342 provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48] provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48] provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200 provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::526b:4bff:fefc:1342 provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48] provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48] provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200 provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::526b:4bff:fefc:1342 provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48] provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48] provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200 provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::526b:4bff:fefc:1342 provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48] provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48] provider: verbs;ofi_rxd, score = 5, pref = -2, FI_FORMAT_UNSPEC [32] provider: verbs;ofi_rxd, score = 5, pref = -2, FI_FORMAT_UNSPEC [32] Required minimum FI_VERSION: 10006, current version: 10015 ==== Capability set configuration ==== libfabric provider: verbs;ofi_rxm - IB-0xfe80000000000000 MPIDI_OFI_ENABLE_DATA: 0 MPIDI_OFI_ENABLE_AV_TABLE: 1 MPIDI_OFI_ENABLE_SCALABLE_ENDPOINTS: 0 MPIDI_OFI_ENABLE_SHARED_CONTEXTS: 0 MPIDI_OFI_ENABLE_MR_VIRT_ADDRESS: 1 MPIDI_OFI_ENABLE_MR_ALLOCATED: 1 MPIDI_OFI_ENABLE_MR_REGISTER_NULL: 1 MPIDI_OFI_ENABLE_MR_PROV_KEY: 1 MPIDI_OFI_ENABLE_TAGGED: 1 MPIDI_OFI_ENABLE_AM: 1 MPIDI_OFI_ENABLE_RMA: 1 MPIDI_OFI_ENABLE_ATOMICS: 0 MPIDI_OFI_FETCH_ATOMIC_IOVECS: 1 MPIDI_OFI_ENABLE_DATA_AUTO_PROGRESS: 0 MPIDI_OFI_ENABLE_CONTROL_AUTO_PROGRESS: 0 MPIDI_OFI_ENABLE_PT2PT_NOPACK: 1 MPIDI_OFI_ENABLE_TRIGGERED: 0 MPIDI_OFI_ENABLE_HMEM: 0 MPIDI_OFI_NUM_AM_BUFFERS: 8 MPIDI_OFI_NUM_OPTIMIZED_MEMORY_REGIONS: 0 MPIDI_OFI_CONTEXT_BITS: 16 MPIDI_OFI_SOURCE_BITS: 23 MPIDI_OFI_TAG_BITS: 20 MPIDI_OFI_VNI_USE_DOMAIN: 1 MAXIMUM SUPPORTED RANKS: 8388608 MAXIMUM TAG: 1048576 ==== Provider global thresholds ==== max_buffered_send: 192 max_buffered_write: 192 max_msg_size: 1073741824 max_order_raw: 1073741824 max_order_war: 0 max_order_waw: 1073741824 tx_iov_limit: 4 rx_iov_limit: 4 rma_iov_limit: 1 max_mr_key_size: 4 ==== Various sizes and limits ==== MPIDI_OFI_AM_MSG_HEADER_SIZE: 24 MPIDI_OFI_MAX_AM_HDR_SIZE: 255 sizeof(MPIDI_OFI_am_request_header_t): 416 sizeof(MPIDI_OFI_per_vci_t): 52480 MPIDI_OFI_AM_HDR_POOL_CELL_SIZE: 1024 MPIDI_OFI_DEFAULT_SHORT_SEND_SIZE: 16384 Assertion failed in file src/mpid/ch4/netmod/ofi/init_addrxchg.c at line 151: mapped_table[i] != FI_ADDR_NOTAVAIL Assertion failed in file src/mpid/ch4/netmod/ofi/init_addrxchg.c at line 151: mapped_table[i] != FI_ADDR_NOTAVAIL Assertion failed in file src/mpid/ch4/netmod/ofi/init_addrxchg.c at line 151: mapped_table[i] != FI_ADDR_NOTAVAIL Assertion failed in file src/mpid/ch4/netmod/ofi/init_addrxchg.c at line 151: mapped_table[i] != FI_ADDR_NOTAVAIL Assertion failed in file src/mpid/ch4/netmod/ofi/init_addrxchg.c at line 151: mapped_table[i] != FI_ADDR_NOTAVAIL If I do not use the FI_PROVIDER=verbs, it will work over sockets. root@ampere-altra-2-1: # MPIR_CVAR_DEBUG_SUMMARY=1 mpirun -n 5 -hosts 192.168.2.200,192.168.2.100 /mpich/examples/a.out ==== Various sizes and limits ==== sizeof(MPIDI_per_vci_t): 192 Required minimum FI_VERSION: 0, current version: 10015 provider: verbs, score = 0, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200 provider: verbs, score = 0, pref = 0, FI_SOCKADDR_IN6 [28] fe80::526b:4bff:fefc:1342 provider: verbs, score = 0, pref = 0, FI_SOCKADDR_IB [48] provider: verbs, score = 0, pref = 0, FI_SOCKADDR_IB [48] provider: verbs, score = 0, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200 provider: verbs, score = 0, pref = 0, FI_SOCKADDR_IN6 [28] fe80::526b:4bff:fefc:1342 provider: verbs, score = 0, pref = 0, FI_SOCKADDR_IB [48] provider: verbs, score = 0, pref = 0, FI_SOCKADDR_IB [48] provider: verbs, score = 0, pref = 0, FI_FORMAT_UNSPEC [32] provider: verbs, score = 0, pref = 0, FI_FORMAT_UNSPEC [32] provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200 provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::526b:4bff:fefc:1342 provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48] provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48] provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200 provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::526b:4bff:fefc:1342 provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48] provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48] provider: tcp, score = 4, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200 provider: tcp, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::526b:4bff:fefc:1342 provider: tcp, score = 4, pref = 0, FI_SOCKADDR_IN [16] 10.118.91.159 provider: tcp, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::be97:e1ff:fe9d:7caa provider: tcp, score = 4, pref = 0, FI_SOCKADDR_IN [16] 127.0.0.1 provider: tcp, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] ::1 provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200 provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::526b:4bff:fefc:1342 provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 10.118.91.159 provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::be97:e1ff:fe9d:7caa provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 127.0.0.1 provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] ::1 provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200 provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::526b:4bff:fefc:1342 provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48] provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48] provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200 provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::526b:4bff:fefc:1342 provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48] provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48] provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200 provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::526b:4bff:fefc:1342 provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 10.118.91.159 provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::be97:e1ff:fe9d:7caa provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 127.0.0.1 provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] ::1 provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200 provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::526b:4bff:fefc:1342 provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48] provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48] provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200 provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::526b:4bff:fefc:1342 provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48] provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48] provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200 provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::526b:4bff:fefc:1342 provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 10.118.91.159 provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::be97:e1ff:fe9d:7caa provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 127.0.0.1 provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] ::1 provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200 provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::526b:4bff:fefc:1342 provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48] provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48] provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200 provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::526b:4bff:fefc:1342 provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48] provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48] provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200 provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::526b:4bff:fefc:1342 provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 10.118.91.159 provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::be97:e1ff:fe9d:7caa provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 127.0.0.1 provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] ::1 provider: verbs;ofi_rxd, score = 5, pref = -2, FI_FORMAT_UNSPEC [32] provider: verbs;ofi_rxd, score = 5, pref = -2, FI_FORMAT_UNSPEC [32] provider: udp;ofi_rxd, score = 5, pref = -2, FI_SOCKADDR_IN [16] 192.168.2.200 provider: udp;ofi_rxd, score = 5, pref = -2, FI_SOCKADDR_IN6 [28] fe80::526b:4bff:fefc:1342 provider: udp;ofi_rxd, score = 5, pref = -2, FI_SOCKADDR_IN [16] 10.118.91.159 provider: udp;ofi_rxd, score = 5, pref = -2, FI_SOCKADDR_IN6 [28] fe80::be97:e1ff:fe9d:7caa provider: udp;ofi_rxd, score = 5, pref = -2, FI_SOCKADDR_IN [16] 127.0.0.1 provider: udp;ofi_rxd, score = 5, pref = -2, FI_SOCKADDR_IN6 [28] ::1 provider: shm, score = 4, pref = -2, FI_ADDR_STR [14] - fi_shm://1177 provider: shm, score = 4, pref = -2, FI_ADDR_STR [14] - fi_shm://1177 provider: udp, score = 0, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200 provider: udp, score = 0, pref = 0, FI_SOCKADDR_IN6 [28] fe80::526b:4bff:fefc:1342 provider: udp, score = 0, pref = 0, FI_SOCKADDR_IN [16] 10.118.91.159 provider: udp, score = 0, pref = 0, FI_SOCKADDR_IN6 [28] fe80::be97:e1ff:fe9d:7caa provider: udp, score = 0, pref = 0, FI_SOCKADDR_IN [16] 127.0.0.1 provider: udp, score = 0, pref = 0, FI_SOCKADDR_IN6 [28] ::1 provider: tcp, score = 0, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200 provider: tcp, score = 0, pref = 0, FI_SOCKADDR_IN6 [28] fe80::526b:4bff:fefc:1342 provider: tcp, score = 0, pref = 0, FI_SOCKADDR_IN [16] 10.118.91.159 provider: tcp, score = 0, pref = 0, FI_SOCKADDR_IN6 [28] fe80::be97:e1ff:fe9d:7caa provider: tcp, score = 0, pref = 0, FI_SOCKADDR_IN [16] 127.0.0.1 provider: tcp, score = 0, pref = 0, FI_SOCKADDR_IN6 [28] ::1 provider: tcp, score = 4, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200 provider: tcp, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::526b:4bff:fefc:1342 provider: tcp, score = 4, pref = 0, FI_SOCKADDR_IN [16] 10.118.91.159 provider: tcp, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::be97:e1ff:fe9d:7caa provider: tcp, score = 4, pref = 0, FI_SOCKADDR_IN [16] 127.0.0.1 provider: tcp, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] ::1 provider: sockets, score = 3, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200 provider: sockets, score = 3, pref = 0, FI_SOCKADDR_IN6 [28] fe80::526b:4bff:fefc:1342 provider: sockets, score = 3, pref = 0, FI_SOCKADDR_IN [16] 10.118.91.159 provider: sockets, score = 3, pref = 0, FI_SOCKADDR_IN6 [28] fe80::be97:e1ff:fe9d:7caa provider: sockets, score = 3, pref = 0, FI_SOCKADDR_IN [16] 127.0.0.1 provider: sockets, score = 3, pref = 0, FI_SOCKADDR_IN6 [28] ::1 provider: sockets, score = 5, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200 provider: sockets, score = 5, pref = 0, FI_SOCKADDR_IN6 [28] fe80::526b:4bff:fefc:1342 provider: sockets, score = 5, pref = 0, FI_SOCKADDR_IN [16] 10.118.91.159 provider: sockets, score = 5, pref = 0, FI_SOCKADDR_IN6 [28] fe80::be97:e1ff:fe9d:7caa provider: sockets, score = 5, pref = 0, FI_SOCKADDR_IN [16] 127.0.0.1 provider: sockets, score = 5, pref = 0, FI_SOCKADDR_IN6 [28] ::1 provider: sockets, score = 5, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200 provider: sockets, score = 5, pref = 0, FI_SOCKADDR_IN6 [28] fe80::526b:4bff:fefc:1342 provider: sockets, score = 5, pref = 0, FI_SOCKADDR_IN [16] 10.118.91.159 provider: sockets, score = 5, pref = 0, FI_SOCKADDR_IN6 [28] fe80::be97:e1ff:fe9d:7caa provider: sockets, score = 5, pref = 0, FI_SOCKADDR_IN [16] 127.0.0.1 provider: sockets, score = 5, pref = 0, FI_SOCKADDR_IN6 [28] ::1 provider: sm2, score = 3, pref = 0, FI_ADDR_STR [13] - fi_sm2://1177 provider: sm2, score = 3, pref = 0, FI_ADDR_STR [13] - fi_sm2://1177 Required minimum FI_VERSION: 10005, current version: 10015 ==== Capability set configuration ==== libfabric provider: sockets - 192.168.2.0/24 MPIDI_OFI_ENABLE_DATA: 1 MPIDI_OFI_ENABLE_AV_TABLE: 1 MPIDI_OFI_ENABLE_SCALABLE_ENDPOINTS: 1 MPIDI_OFI_ENABLE_SHARED_CONTEXTS: 0 MPIDI_OFI_ENABLE_MR_VIRT_ADDRESS: 0 MPIDI_OFI_ENABLE_MR_ALLOCATED: 0 MPIDI_OFI_ENABLE_MR_REGISTER_NULL: 1 MPIDI_OFI_ENABLE_MR_PROV_KEY: 0 MPIDI_OFI_ENABLE_TAGGED: 1 MPIDI_OFI_ENABLE_AM: 1 MPIDI_OFI_ENABLE_RMA: 1 MPIDI_OFI_ENABLE_ATOMICS: 1 MPIDI_OFI_FETCH_ATOMIC_IOVECS: 1 MPIDI_OFI_ENABLE_DATA_AUTO_PROGRESS: 0 MPIDI_OFI_ENABLE_CONTROL_AUTO_PROGRESS: 0 MPIDI_OFI_ENABLE_PT2PT_NOPACK: 1 MPIDI_OFI_ENABLE_TRIGGERED: 0 MPIDI_OFI_ENABLE_HMEM: 0 MPIDI_OFI_NUM_AM_BUFFERS: 8 MPIDI_OFI_NUM_OPTIMIZED_MEMORY_REGIONS: 0 MPIDI_OFI_CONTEXT_BITS: 20 MPIDI_OFI_SOURCE_BITS: 0 MPIDI_OFI_TAG_BITS: 31 MPIDI_OFI_VNI_USE_DOMAIN: 1 MAXIMUM SUPPORTED RANKS: 4294967296 MAXIMUM TAG: 2147483648 ==== Provider global thresholds ==== max_buffered_send: 255 max_buffered_write: 255 max_msg_size: 9223372036854775807 max_order_raw: -1 max_order_war: -1 max_order_waw: -1 tx_iov_limit: 8 rx_iov_limit: 8 rma_iov_limit: 8 max_mr_key_size: 8 ==== Various sizes and limits ==== MPIDI_OFI_AM_MSG_HEADER_SIZE: 24 MPIDI_OFI_MAX_AM_HDR_SIZE: 255 sizeof(MPIDI_OFI_am_request_header_t): 416 sizeof(MPIDI_OFI_per_vci_t): 52480 MPIDI_OFI_AM_HDR_POOL_CELL_SIZE: 1024 MPIDI_OFI_DEFAULT_SHORT_SEND_SIZE: 16384 ==== OFI dynamic settings ==== num_vcis: 1 num_nics: 1 ====================================== error checking : enabled QMPI : disabled debugger support : disabled thread level : MPI_THREAD_SINGLE thread CS : per-vci threadcomm : enabled ==== data structure summary ==== sizeof(MPIR_Comm): 1808 sizeof(MPIR_Request): 512 sizeof(MPIR_Datatype): 280 ================================ Hello world from process 0 of 5 Hello world from process 2 of 5 Hello world from process 1 of 5 Hello world from process 4 of 5 Hello world from process 3 of 5 From: Zhou, Hui <zhouh(a)anl.gov> Date: Wednesday, June 12, 2024 at 7:22 PM To: Niyaz Murshed <Niyaz.Murshed(a)arm.com>, discuss(a)mpich.org <discuss(a)mpich.org> Subject: Re: Mpich over RDMA sample It’s possible that the native path for RMA on verbs are turned off due to verbs does not meet MPI’s full atomic ordering semantics. We’ll check and follow up. -- Hui Zhou From: Niyaz Murshed <Niyaz.Murshed(a)arm.com> Date: Wednesday, June 12, 2024 at 3:59 PM To: Zhou, Hui <zhouh(a)anl.gov>, discuss(a)mpich.org <discuss(a)mpich.org> Cc: nd <nd(a)arm.com> Subject: Re: Mpich over RDMA sample I think image is not gone : https: //ibb. co/RDGnJfy From: Niyaz Murshed <Niyaz. Murshed@ arm. com> Date: Wednesday, June 12, 2024 at 3: 56 PM To: Zhou, Hui <zhouh@ anl. gov>, discuss@ mpich. org <discuss@ mpich. org> Cc: nd <nd@ arm. com> ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd I think image is not gone : https://urldefense.us/v3/__https://ibb.co/RDGnJfy__;!!G_uCfscf7eWS!cDbck3Mz… <https://urldefense.us/v3/__https:/ibb.co/RDGnJfy__;!!G_uCfscf7eWS!YH6L3OHfP…> From: Niyaz Murshed <Niyaz.Murshed(a)arm.com> Date: Wednesday, June 12, 2024 at 3:56 PM To: Zhou, Hui <zhouh(a)anl.gov>, discuss(a)mpich.org <discuss(a)mpich.org> Cc: nd <nd(a)arm.com> Subject: Re: Mpich over RDMA sample Thank you Hui for the reply. I was expecting to see RoCE messages in the packet capture. When I use verbs on libfabric sample tests, I see the RoCE messages as below : [cid:[email protected]] From: Zhou, Hui <zhouh(a)anl.gov> Date: Wednesday, June 12, 2024 at 3:41 PM To: Niyaz Murshed <Niyaz.Murshed(a)arm.com>, discuss(a)mpich.org <discuss(a)mpich.org> Cc: nd <nd(a)arm.com> Subject: Re: Mpich over RDMA sample Niyaz, ofi_rxm:verbs is the correct provider. The ofi_rxm provides the verbs provider additional messaging semantics that are needed to be used in MPI. -- Hui Zhou From: Niyaz Murshed <Niyaz.Murshed(a)arm.com> Date: Wednesday, June 12, 2024 at 2:54 PM To: Zhou, Hui <zhouh(a)anl.gov>, discuss(a)mpich.org <discuss(a)mpich.org> Cc: nd <nd(a)arm.com> Subject: Re: Mpich over RDMA sample Thank you for the reply. I did that; however, it selects ofi_rxm: verbs libfabric: 412098: 1718221892: : ofi_rxm: av: ofi_av_insert_addr(): 313<info> fi_addr: 1 libfabric: 412099: 1718221892: : ofi_rxm: av: ofi_av_insert_addr(): 313<info> fi_addr: ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Thank you for the reply. I did that; however, it selects ofi_rxm:verbs libfabric:412098:1718221892::ofi_rxm:av:ofi_av_insert_addr():313<info> fi_addr: 1 libfabric:412099:1718221892::ofi_rxm:av:ofi_av_insert_addr():313<info> fi_addr: 1 options: backend: cpu iters: 16 warmup_iters: 16 cache: 1 min_elem_count: 1 max_elem_count: 1 elem_counts: [1] validate: last window_size: 64 #------------------------------------------------------------ # Benchmarking: Bandwidth # #processes: 2 #------------------------------------------------------------ #bytes #repetitions Mbytes/sec 4 16 0.02 # All done libfabric:412098:1718221892::ofi_rxm:ep_ctrl:rxm_stop_listen():864<info> stopping CM thread libfabric:412099:1718221892::ofi_rxm:ep_ctrl:rxm_stop_listen():864<info> stopping CM thread As per https://urldefense.us/v3/__https://www.intel.com/content/www/us/en/docs/mpi… <https://urldefense.us/v3/__https:/www.intel.com/content/www/us/en/docs/mpi-…> , we need to add FI_PROVIDER=^ofi_rxm , but if do that, it moves back to sockets provider. Is here a way to combine ^ofi_rxm and verbs From: Zhou, Hui <zhouh(a)anl.gov> Date: Wednesday, June 12, 2024 at 2:41 PM To: Niyaz Murshed <Niyaz.Murshed(a)arm.com>, discuss(a)mpich.org <discuss(a)mpich.org> Cc: nd <nd(a)arm.com> Subject: Re: Mpich over RDMA sample Niyaz, All you need to do is to set an environment variable `FI_PROVIDER=verbs`. -- Hui Zhou From: Niyaz Murshed <Niyaz.Murshed(a)arm.com> Date: Wednesday, June 12, 2024 at 1:23 PM To: Zhou, Hui <zhouh(a)anl.gov>, discuss(a)mpich.org <discuss(a)mpich.org> Cc: nd <nd(a)arm.com> Subject: Re: Mpich over RDMA sample When testing with Libfabric, verbs provider is selected. I did have to use “-e msg -d mlx5_1” so that it selects verbs. I was checking if there is anything like that for mpich sample tests. Else might need to do some hack in the code to force ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd When testing with Libfabric, verbs provider is selected. I did have to use “-e msg -d mlx5_1” so that it selects verbs. I was checking if there is anything like that for mpich sample tests. Else might need to do some hack in the code to force the selection of verbs. From: Zhou, Hui <zhouh(a)anl.gov> Date: Wednesday, June 12, 2024 at 1:15 PM To: discuss(a)mpich.org <discuss(a)mpich.org> Cc: Niyaz Murshed <Niyaz.Murshed(a)arm.com>, nd <nd(a)arm.com> Subject: Re: Mpich over RDMA sample Libfabric support multiple providers. Sounds like it was selecting the sockets or tcp provider rather than a provider that support RoCE. I am not exactly sure whether the verbs provider will do that. If you can confirm the provider using libfabric tests, then you can try forcing MPICH to use that provider by setting the FI_PROVIDER environment variable. -- Hui Zhou From: Niyaz Murshed via discuss <discuss(a)mpich.org> Date: Wednesday, June 12, 2024 at 9:03 AM To: discuss(a)mpich.org <discuss(a)mpich.org> Cc: Niyaz Murshed <Niyaz.Murshed(a)arm.com>, nd <nd(a)arm.com> Subject: [mpich-discuss] Mpich over RDMA sample Hello, I am trying to learn about MPICH and its performance over RDMA. I am using libfabric and installed mpich using the below configure. ./configure --prefix=/opt/mpich/ --with-ofi=/opt/libfabric/ When I run any applications between 2 directly ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Hello, I am trying to learn about MPICH and its performance over RDMA. I am using libfabric and installed mpich using the below configure. ./configure --prefix=/opt/mpich/ --with-ofi=/opt/libfabric/ When I run any applications between 2 directly connected servers having Mellanox NICs, I see that communication is happening over tcp and not over RoCE. Is there any way to test commination over RoCE ? For eg. I was able to test it for libfabric using the below sample that comes along with libfabric to test RMA. Is there something similar for MPICH ? or use the current sample to use RoCE by some parameter? Server : fi_rma_bw -s 192.168.1.100 -e msg -d mlx5_1 -S 1024 -I 1 Client : fi_rma_bw -s 192.168.1.200 -e msg -d mlx5_3 192.168.1.100 -S 1024 -I 1 Regards, Niyaz

1 0