June 2024 - discuss - mailman3.cels.anl.gov

Re: [mpich-discuss] Fail on MPI_Wait
by Zhou, Hui 14 Jun '24

14 Jun '24

Never mind. It is v4.2.1. ________________________________ From: Zhou, Hui <zhouh(a)anl.gov> Sent: Friday, June 14, 2024 11:26 AM To: Palmer, Bruce J <Bruce.Palmer(a)pnnl.gov>; discuss(a)mpich.org <discuss(a)mpich.org> Subject: Re: Fail on MPI_Wait Bruce, What is the mpich version, BTW? -- Hui ________________________________ From: Zhou, Hui <zhouh(a)anl.gov> Sent: Friday, June 14, 2024 10:55 AM To: Palmer, Bruce J <Bruce.Palmer(a)pnnl.gov>; discuss(a)mpich.org <discuss(a)mpich.org> Subject: Re: Fail on MPI_Wait Bruce, You are using the sockets provider. Could try set FI_PROVIDER=tcp to see if it makes a difference? Meanwhile, if you can get a small reproducer – with the sockets provider or any provider, I'll try to debug it. It is difficult to guess the true source of the issue without a reproducer. -- Hui ________________________________ From: Palmer, Bruce J <Bruce.Palmer(a)pnnl.gov> Sent: Friday, June 14, 2024 10:47 AM To: Zhou, Hui <zhouh(a)anl.gov>; discuss(a)mpich.org <discuss(a)mpich.org> Subject: Re: Fail on MPI_Wait The output to standard out from running on 2 nodes and one process per node is attached. From: Zhou, Hui <zhouh@ anl. gov> Date: Tuesday, June 11, 2024 at 5: 49 PM To: discuss@ mpich. org <discuss@ mpich. org> Cc: Palmer, Bruce J <Bruce. Palmer@ pnnl. gov> ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd The output to standard out from running on 2 nodes and one process per node is attached. From: Zhou, Hui <zhouh(a)anl.gov> Date: Tuesday, June 11, 2024 at 5:49 PM To: discuss(a)mpich.org <discuss(a)mpich.org> Cc: Palmer, Bruce J <Bruce.Palmer(a)pnnl.gov> Subject: Re: Fail on MPI_Wait >MPIDI_OFI_handle_cq_error(793): OFI poll failed (ofi_events.c:793:MPIDI_OFI_handle_cq_error:Input/output error) This is an error coming from the libfabric provider. First we need find out which provider are you using. Try set environment variable MPIR_CVAR_DEBUG_SUMMARY=1 and run a simple MPI_INIT+MPI_Finalize test code. Could post its console output? -- Hui ________________________________ From: Palmer, Bruce J via discuss <discuss(a)mpich.org> Sent: Tuesday, June 11, 2024 3:17 PM To: discuss(a)mpich.org <discuss(a)mpich.org> Cc: Palmer, Bruce J <Bruce.Palmer(a)pnnl.gov> Subject: [mpich-discuss] Fail on MPI_Wait Hi, I’m trying to debug a GPU-aware runtime for the Global Arrays library. We had a version of this working a while ago, but it has mysteriously started failing and we are trying to track down why. Currently, we are getting failures in MPI_Wait ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Hi, I’m trying to debug a GPU-aware runtime for the Global Arrays library. We had a version of this working a while ago, but it has mysteriously started failing and we are trying to track down why. Currently, we are getting failures in MPI_Wait and were wondering if anyone could provide some information on what exactly seems to be failing inside the wait call. The error we are getting is Abort(206752655) on node 0: Fatal error in internal_Wait: Other MPI error, error stack: internal_Wait(68205)..........: MPI_Wait(request=0x500847a0, status=0x7ffff9331800) failed MPIR_Wait(780)................: MPIR_Wait_state(737)..........: MPIDI_progress_test(134)......: MPIDI_OFI_handle_cq_error(793): OFI poll failed (ofi_events.c:793:MPIDI_OFI_handle_cq_error:Input/output error) I’ve verified that the handle corresponding to 0x500847a0 is getting set earlier in the code in an MPI_Isend call and that no MPI_Wait or MPI_Test is called on the handle before it crashes with the above error message. I’m using MPICH 4.2.1 using gcc/8.3.0. The MPICH library was configured with ../configure --prefix=/people/d3g293/mpich/mpich-4.2.1/build_newell/install \ --with-device=ch4:ofi:sockets --with-libfabric=embedded \ --without-ucx --enable-threads=multiple --with-slurm \ CC=gcc CXX=g+ I’ve tried building with UCX and gotten the same results. Are these errors indicative of corruption of the request handle or problems with some internal MPI data structures or something else? Any information you can provide would be appreciated. Thanks, Bruce

1 0

Re: [mpich-discuss] Fail on MPI_Wait
by Zhou, Hui 14 Jun '24

14 Jun '24

Bruce, What is the mpich version, BTW? -- Hui ________________________________ From: Zhou, Hui <zhouh(a)anl.gov> Sent: Friday, June 14, 2024 10:55 AM To: Palmer, Bruce J <Bruce.Palmer(a)pnnl.gov>; discuss(a)mpich.org <discuss(a)mpich.org> Subject: Re: Fail on MPI_Wait Bruce, You are using the sockets provider. Could try set FI_PROVIDER=tcp to see if it makes a difference? Meanwhile, if you can get a small reproducer – with the sockets provider or any provider, I'll try to debug it. It is difficult to guess the true source of the issue without a reproducer. -- Hui ________________________________ From: Palmer, Bruce J <Bruce.Palmer(a)pnnl.gov> Sent: Friday, June 14, 2024 10:47 AM To: Zhou, Hui <zhouh(a)anl.gov>; discuss(a)mpich.org <discuss(a)mpich.org> Subject: Re: Fail on MPI_Wait The output to standard out from running on 2 nodes and one process per node is attached. From: Zhou, Hui <zhouh@ anl. gov> Date: Tuesday, June 11, 2024 at 5: 49 PM To: discuss@ mpich. org <discuss@ mpich. org> Cc: Palmer, Bruce J <Bruce. Palmer@ pnnl. gov> ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd The output to standard out from running on 2 nodes and one process per node is attached. From: Zhou, Hui <zhouh(a)anl.gov> Date: Tuesday, June 11, 2024 at 5:49 PM To: discuss(a)mpich.org <discuss(a)mpich.org> Cc: Palmer, Bruce J <Bruce.Palmer(a)pnnl.gov> Subject: Re: Fail on MPI_Wait >MPIDI_OFI_handle_cq_error(793): OFI poll failed (ofi_events.c:793:MPIDI_OFI_handle_cq_error:Input/output error) This is an error coming from the libfabric provider. First we need find out which provider are you using. Try set environment variable MPIR_CVAR_DEBUG_SUMMARY=1 and run a simple MPI_INIT+MPI_Finalize test code. Could post its console output? -- Hui ________________________________ From: Palmer, Bruce J via discuss <discuss(a)mpich.org> Sent: Tuesday, June 11, 2024 3:17 PM To: discuss(a)mpich.org <discuss(a)mpich.org> Cc: Palmer, Bruce J <Bruce.Palmer(a)pnnl.gov> Subject: [mpich-discuss] Fail on MPI_Wait Hi, I’m trying to debug a GPU-aware runtime for the Global Arrays library. We had a version of this working a while ago, but it has mysteriously started failing and we are trying to track down why. Currently, we are getting failures in MPI_Wait ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Hi, I’m trying to debug a GPU-aware runtime for the Global Arrays library. We had a version of this working a while ago, but it has mysteriously started failing and we are trying to track down why. Currently, we are getting failures in MPI_Wait and were wondering if anyone could provide some information on what exactly seems to be failing inside the wait call. The error we are getting is Abort(206752655) on node 0: Fatal error in internal_Wait: Other MPI error, error stack: internal_Wait(68205)..........: MPI_Wait(request=0x500847a0, status=0x7ffff9331800) failed MPIR_Wait(780)................: MPIR_Wait_state(737)..........: MPIDI_progress_test(134)......: MPIDI_OFI_handle_cq_error(793): OFI poll failed (ofi_events.c:793:MPIDI_OFI_handle_cq_error:Input/output error) I’ve verified that the handle corresponding to 0x500847a0 is getting set earlier in the code in an MPI_Isend call and that no MPI_Wait or MPI_Test is called on the handle before it crashes with the above error message. I’m using MPICH 4.2.1 using gcc/8.3.0. The MPICH library was configured with ../configure --prefix=/people/d3g293/mpich/mpich-4.2.1/build_newell/install \ --with-device=ch4:ofi:sockets --with-libfabric=embedded \ --without-ucx --enable-threads=multiple --with-slurm \ CC=gcc CXX=g+ I’ve tried building with UCX and gotten the same results. Are these errors indicative of corruption of the request handle or problems with some internal MPI data structures or something else? Any information you can provide would be appreciated. Thanks, Bruce

1 0

Re: [mpich-discuss] Fail on MPI_Wait
by Palmer, Bruce J 14 Jun '24

14 Jun '24

Zhou, This is the output to standard error if I turn off the debug flags and run the test code with the FI_PROVIDER flag set to tcp. Abort(136963471): Fatal error in internal_Init: Other MPI error, error stack: internal_Init(48306).......: MPI_Init(argc=0x7fffc7757f00, argv=0x7fffc7757f08) failed MPII_Init_thread(242)......: MPID_Init(552).............: MPIDI_OFI_init_local(761)..: MPIDI_OFI_find_provider(84): find_provider(178).........: OFI fi_getinfo() failure Abort(808052111): Fatal error in internal_Init: Other MPI error, error stack: internal_Init(48306).......: MPI_Init(argc=0x7ffff046c410, argv=0x7ffff046c418) failed MPII_Init_thread(242)......: MPID_Init(552).............: MPIDI_OFI_init_local(761)..: MPIDI_OFI_find_provider(84): find_provider(178).........: OFI fi_getinfo() failure From: Zhou, Hui <zhouh(a)anl.gov> Date: Friday, June 14, 2024 at 8:55 AM To: Palmer, Bruce J <Bruce.Palmer(a)pnnl.gov>, discuss(a)mpich.org <discuss(a)mpich.org> Subject: Re: Fail on MPI_Wait Bruce, You are using the sockets provider. Could try set FI_PROVIDER=tcp to see if it makes a difference? Meanwhile, if you can get a small reproducer – with the sockets provider or any provider, I'll try to debug it. It is difficult to guess the true source of the issue without a reproducer. -- Hui ________________________________ From: Palmer, Bruce J <Bruce.Palmer(a)pnnl.gov> Sent: Friday, June 14, 2024 10:47 AM To: Zhou, Hui <zhouh(a)anl.gov>; discuss(a)mpich.org <discuss(a)mpich.org> Subject: Re: Fail on MPI_Wait The output to standard out from running on 2 nodes and one process per node is attached. From: Zhou, Hui <zhouh@ anl. gov> Date: Tuesday, June 11, 2024 at 5: 49 PM To: discuss@ mpich. org <discuss@ mpich. org> Cc: Palmer, Bruce J <Bruce. Palmer@ pnnl. gov> ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd The output to standard out from running on 2 nodes and one process per node is attached. From: Zhou, Hui <zhouh(a)anl.gov> Date: Tuesday, June 11, 2024 at 5:49 PM To: discuss(a)mpich.org <discuss(a)mpich.org> Cc: Palmer, Bruce J <Bruce.Palmer(a)pnnl.gov> Subject: Re: Fail on MPI_Wait >MPIDI_OFI_handle_cq_error(793): OFI poll failed (ofi_events.c:793:MPIDI_OFI_handle_cq_error:Input/output error) This is an error coming from the libfabric provider. First we need find out which provider are you using. Try set environment variable MPIR_CVAR_DEBUG_SUMMARY=1 and run a simple MPI_INIT+MPI_Finalize test code. Could post its console output? -- Hui ________________________________ From: Palmer, Bruce J via discuss <discuss(a)mpich.org> Sent: Tuesday, June 11, 2024 3:17 PM To: discuss(a)mpich.org <discuss(a)mpich.org> Cc: Palmer, Bruce J <Bruce.Palmer(a)pnnl.gov> Subject: [mpich-discuss] Fail on MPI_Wait Hi, I’m trying to debug a GPU-aware runtime for the Global Arrays library. We had a version of this working a while ago, but it has mysteriously started failing and we are trying to track down why. Currently, we are getting failures in MPI_Wait ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Hi, I’m trying to debug a GPU-aware runtime for the Global Arrays library. We had a version of this working a while ago, but it has mysteriously started failing and we are trying to track down why. Currently, we are getting failures in MPI_Wait and were wondering if anyone could provide some information on what exactly seems to be failing inside the wait call. The error we are getting is Abort(206752655) on node 0: Fatal error in internal_Wait: Other MPI error, error stack: internal_Wait(68205)..........: MPI_Wait(request=0x500847a0, status=0x7ffff9331800) failed MPIR_Wait(780)................: MPIR_Wait_state(737)..........: MPIDI_progress_test(134)......: MPIDI_OFI_handle_cq_error(793): OFI poll failed (ofi_events.c:793:MPIDI_OFI_handle_cq_error:Input/output error) I’ve verified that the handle corresponding to 0x500847a0 is getting set earlier in the code in an MPI_Isend call and that no MPI_Wait or MPI_Test is called on the handle before it crashes with the above error message. I’m using MPICH 4.2.1 using gcc/8.3.0. The MPICH library was configured with ../configure --prefix=/people/d3g293/mpich/mpich-4.2.1/build_newell/install \ --with-device=ch4:ofi:sockets --with-libfabric=embedded \ --without-ucx --enable-threads=multiple --with-slurm \ CC=gcc CXX=g+ I’ve tried building with UCX and gotten the same results. Are these errors indicative of corruption of the request handle or problems with some internal MPI data structures or something else? Any information you can provide would be appreciated. Thanks, Bruce

1 0

Re: [mpich-discuss] Fail on MPI_Wait
by Zhou, Hui 14 Jun '24

14 Jun '24

Bruce, You are using the sockets provider. Could try set FI_PROVIDER=tcp to see if it makes a difference? Meanwhile, if you can get a small reproducer – with the sockets provider or any provider, I'll try to debug it. It is difficult to guess the true source of the issue without a reproducer. -- Hui ________________________________ From: Palmer, Bruce J <Bruce.Palmer(a)pnnl.gov> Sent: Friday, June 14, 2024 10:47 AM To: Zhou, Hui <zhouh(a)anl.gov>; discuss(a)mpich.org <discuss(a)mpich.org> Subject: Re: Fail on MPI_Wait The output to standard out from running on 2 nodes and one process per node is attached. From: Zhou, Hui <zhouh@ anl. gov> Date: Tuesday, June 11, 2024 at 5: 49 PM To: discuss@ mpich. org <discuss@ mpich. org> Cc: Palmer, Bruce J <Bruce. Palmer@ pnnl. gov> ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd The output to standard out from running on 2 nodes and one process per node is attached. From: Zhou, Hui <zhouh(a)anl.gov> Date: Tuesday, June 11, 2024 at 5:49 PM To: discuss(a)mpich.org <discuss(a)mpich.org> Cc: Palmer, Bruce J <Bruce.Palmer(a)pnnl.gov> Subject: Re: Fail on MPI_Wait >MPIDI_OFI_handle_cq_error(793): OFI poll failed (ofi_events.c:793:MPIDI_OFI_handle_cq_error:Input/output error) This is an error coming from the libfabric provider. First we need find out which provider are you using. Try set environment variable MPIR_CVAR_DEBUG_SUMMARY=1 and run a simple MPI_INIT+MPI_Finalize test code. Could post its console output? -- Hui ________________________________ From: Palmer, Bruce J via discuss <discuss(a)mpich.org> Sent: Tuesday, June 11, 2024 3:17 PM To: discuss(a)mpich.org <discuss(a)mpich.org> Cc: Palmer, Bruce J <Bruce.Palmer(a)pnnl.gov> Subject: [mpich-discuss] Fail on MPI_Wait Hi, I’m trying to debug a GPU-aware runtime for the Global Arrays library. We had a version of this working a while ago, but it has mysteriously started failing and we are trying to track down why. Currently, we are getting failures in MPI_Wait ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Hi, I’m trying to debug a GPU-aware runtime for the Global Arrays library. We had a version of this working a while ago, but it has mysteriously started failing and we are trying to track down why. Currently, we are getting failures in MPI_Wait and were wondering if anyone could provide some information on what exactly seems to be failing inside the wait call. The error we are getting is Abort(206752655) on node 0: Fatal error in internal_Wait: Other MPI error, error stack: internal_Wait(68205)..........: MPI_Wait(request=0x500847a0, status=0x7ffff9331800) failed MPIR_Wait(780)................: MPIR_Wait_state(737)..........: MPIDI_progress_test(134)......: MPIDI_OFI_handle_cq_error(793): OFI poll failed (ofi_events.c:793:MPIDI_OFI_handle_cq_error:Input/output error) I’ve verified that the handle corresponding to 0x500847a0 is getting set earlier in the code in an MPI_Isend call and that no MPI_Wait or MPI_Test is called on the handle before it crashes with the above error message. I’m using MPICH 4.2.1 using gcc/8.3.0. The MPICH library was configured with ../configure --prefix=/people/d3g293/mpich/mpich-4.2.1/build_newell/install \ --with-device=ch4:ofi:sockets --with-libfabric=embedded \ --without-ucx --enable-threads=multiple --with-slurm \ CC=gcc CXX=g+ I’ve tried building with UCX and gotten the same results. Are these errors indicative of corruption of the request handle or problems with some internal MPI data structures or something else? Any information you can provide would be appreciated. Thanks, Bruce

1 0

Re: [mpich-discuss] Fail on MPI_Wait
by Palmer, Bruce J 14 Jun '24

14 Jun '24

The output to standard out from running on 2 nodes and one process per node is attached. From: Zhou, Hui <zhouh(a)anl.gov> Date: Tuesday, June 11, 2024 at 5:49 PM To: discuss(a)mpich.org <discuss(a)mpich.org> Cc: Palmer, Bruce J <Bruce.Palmer(a)pnnl.gov> Subject: Re: Fail on MPI_Wait >MPIDI_OFI_handle_cq_error(793): OFI poll failed (ofi_events.c:793:MPIDI_OFI_handle_cq_error:Input/output error) This is an error coming from the libfabric provider. First we need find out which provider are you using. Try set environment variable MPIR_CVAR_DEBUG_SUMMARY=1 and run a simple MPI_INIT+MPI_Finalize test code. Could post its console output? -- Hui ________________________________ From: Palmer, Bruce J via discuss <discuss(a)mpich.org> Sent: Tuesday, June 11, 2024 3:17 PM To: discuss(a)mpich.org <discuss(a)mpich.org> Cc: Palmer, Bruce J <Bruce.Palmer(a)pnnl.gov> Subject: [mpich-discuss] Fail on MPI_Wait Hi, I’m trying to debug a GPU-aware runtime for the Global Arrays library. We had a version of this working a while ago, but it has mysteriously started failing and we are trying to track down why. Currently, we are getting failures in MPI_Wait ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Hi, I’m trying to debug a GPU-aware runtime for the Global Arrays library. We had a version of this working a while ago, but it has mysteriously started failing and we are trying to track down why. Currently, we are getting failures in MPI_Wait and were wondering if anyone could provide some information on what exactly seems to be failing inside the wait call. The error we are getting is Abort(206752655) on node 0: Fatal error in internal_Wait: Other MPI error, error stack: internal_Wait(68205)..........: MPI_Wait(request=0x500847a0, status=0x7ffff9331800) failed MPIR_Wait(780)................: MPIR_Wait_state(737)..........: MPIDI_progress_test(134)......: MPIDI_OFI_handle_cq_error(793): OFI poll failed (ofi_events.c:793:MPIDI_OFI_handle_cq_error:Input/output error) I’ve verified that the handle corresponding to 0x500847a0 is getting set earlier in the code in an MPI_Isend call and that no MPI_Wait or MPI_Test is called on the handle before it crashes with the above error message. I’m using MPICH 4.2.1 using gcc/8.3.0. The MPICH library was configured with ../configure --prefix=/people/d3g293/mpich/mpich-4.2.1/build_newell/install \ --with-device=ch4:ofi:sockets --with-libfabric=embedded \ --without-ucx --enable-threads=multiple --with-slurm \ CC=gcc CXX=g+ I’ve tried building with UCX and gotten the same results. Are these errors indicative of corruption of the request handle or problems with some internal MPI data structures or something else? Any information you can provide would be appreciated. Thanks, Bruce

1 0

Re: [mpich-discuss] Mpich over RDMA sample
by Zhou, Hui 12 Jun '24

12 Jun '24

It’s possible that the native path for RMA on verbs are turned off due to verbs does not meet MPI’s full atomic ordering semantics. We’ll check and follow up. -- Hui Zhou From: Niyaz Murshed <Niyaz.Murshed(a)arm.com> Date: Wednesday, June 12, 2024 at 3:59 PM To: Zhou, Hui <zhouh(a)anl.gov>, discuss(a)mpich.org <discuss(a)mpich.org> Cc: nd <nd(a)arm.com> Subject: Re: Mpich over RDMA sample I think image is not gone : https: //ibb. co/RDGnJfy From: Niyaz Murshed <Niyaz. Murshed@ arm. com> Date: Wednesday, June 12, 2024 at 3: 56 PM To: Zhou, Hui <zhouh@ anl. gov>, discuss@ mpich. org <discuss@ mpich. org> Cc: nd <nd@ arm. com> ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd I think image is not gone : https://urldefense.us/v3/__https://ibb.co/RDGnJfy__;!!G_uCfscf7eWS!dPlBFaJJ… <https://urldefense.us/v3/__https:/ibb.co/RDGnJfy__;!!G_uCfscf7eWS!YH6L3OHfP…> From: Niyaz Murshed <Niyaz.Murshed(a)arm.com> Date: Wednesday, June 12, 2024 at 3:56 PM To: Zhou, Hui <zhouh(a)anl.gov>, discuss(a)mpich.org <discuss(a)mpich.org> Cc: nd <nd(a)arm.com> Subject: Re: Mpich over RDMA sample Thank you Hui for the reply. I was expecting to see RoCE messages in the packet capture. When I use verbs on libfabric sample tests, I see the RoCE messages as below : [cid:[email protected]] From: Zhou, Hui <zhouh(a)anl.gov> Date: Wednesday, June 12, 2024 at 3:41 PM To: Niyaz Murshed <Niyaz.Murshed(a)arm.com>, discuss(a)mpich.org <discuss(a)mpich.org> Cc: nd <nd(a)arm.com> Subject: Re: Mpich over RDMA sample Niyaz, ofi_rxm:verbs is the correct provider. The ofi_rxm provides the verbs provider additional messaging semantics that are needed to be used in MPI. -- Hui Zhou From: Niyaz Murshed <Niyaz.Murshed(a)arm.com> Date: Wednesday, June 12, 2024 at 2:54 PM To: Zhou, Hui <zhouh(a)anl.gov>, discuss(a)mpich.org <discuss(a)mpich.org> Cc: nd <nd(a)arm.com> Subject: Re: Mpich over RDMA sample Thank you for the reply. I did that; however, it selects ofi_rxm: verbs libfabric: 412098: 1718221892: : ofi_rxm: av: ofi_av_insert_addr(): 313<info> fi_addr: 1 libfabric: 412099: 1718221892: : ofi_rxm: av: ofi_av_insert_addr(): 313<info> fi_addr: ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Thank you for the reply. I did that; however, it selects ofi_rxm:verbs libfabric:412098:1718221892::ofi_rxm:av:ofi_av_insert_addr():313<info> fi_addr: 1 libfabric:412099:1718221892::ofi_rxm:av:ofi_av_insert_addr():313<info> fi_addr: 1 options: backend: cpu iters: 16 warmup_iters: 16 cache: 1 min_elem_count: 1 max_elem_count: 1 elem_counts: [1] validate: last window_size: 64 #------------------------------------------------------------ # Benchmarking: Bandwidth # #processes: 2 #------------------------------------------------------------ #bytes #repetitions Mbytes/sec 4 16 0.02 # All done libfabric:412098:1718221892::ofi_rxm:ep_ctrl:rxm_stop_listen():864<info> stopping CM thread libfabric:412099:1718221892::ofi_rxm:ep_ctrl:rxm_stop_listen():864<info> stopping CM thread As per https://urldefense.us/v3/__https://www.intel.com/content/www/us/en/docs/mpi… <https://urldefense.us/v3/__https:/www.intel.com/content/www/us/en/docs/mpi-…> , we need to add FI_PROVIDER=^ofi_rxm , but if do that, it moves back to sockets provider. Is here a way to combine ^ofi_rxm and verbs From: Zhou, Hui <zhouh(a)anl.gov> Date: Wednesday, June 12, 2024 at 2:41 PM To: Niyaz Murshed <Niyaz.Murshed(a)arm.com>, discuss(a)mpich.org <discuss(a)mpich.org> Cc: nd <nd(a)arm.com> Subject: Re: Mpich over RDMA sample Niyaz, All you need to do is to set an environment variable `FI_PROVIDER=verbs`. -- Hui Zhou From: Niyaz Murshed <Niyaz.Murshed(a)arm.com> Date: Wednesday, June 12, 2024 at 1:23 PM To: Zhou, Hui <zhouh(a)anl.gov>, discuss(a)mpich.org <discuss(a)mpich.org> Cc: nd <nd(a)arm.com> Subject: Re: Mpich over RDMA sample When testing with Libfabric, verbs provider is selected. I did have to use “-e msg -d mlx5_1” so that it selects verbs. I was checking if there is anything like that for mpich sample tests. Else might need to do some hack in the code to force ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd When testing with Libfabric, verbs provider is selected. I did have to use “-e msg -d mlx5_1” so that it selects verbs. I was checking if there is anything like that for mpich sample tests. Else might need to do some hack in the code to force the selection of verbs. From: Zhou, Hui <zhouh(a)anl.gov> Date: Wednesday, June 12, 2024 at 1:15 PM To: discuss(a)mpich.org <discuss(a)mpich.org> Cc: Niyaz Murshed <Niyaz.Murshed(a)arm.com>, nd <nd(a)arm.com> Subject: Re: Mpich over RDMA sample Libfabric support multiple providers. Sounds like it was selecting the sockets or tcp provider rather than a provider that support RoCE. I am not exactly sure whether the verbs provider will do that. If you can confirm the provider using libfabric tests, then you can try forcing MPICH to use that provider by setting the FI_PROVIDER environment variable. -- Hui Zhou From: Niyaz Murshed via discuss <discuss(a)mpich.org> Date: Wednesday, June 12, 2024 at 9:03 AM To: discuss(a)mpich.org <discuss(a)mpich.org> Cc: Niyaz Murshed <Niyaz.Murshed(a)arm.com>, nd <nd(a)arm.com> Subject: [mpich-discuss] Mpich over RDMA sample Hello, I am trying to learn about MPICH and its performance over RDMA. I am using libfabric and installed mpich using the below configure. ./configure --prefix=/opt/mpich/ --with-ofi=/opt/libfabric/ When I run any applications between 2 directly ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Hello, I am trying to learn about MPICH and its performance over RDMA. I am using libfabric and installed mpich using the below configure. ./configure --prefix=/opt/mpich/ --with-ofi=/opt/libfabric/ When I run any applications between 2 directly connected servers having Mellanox NICs, I see that communication is happening over tcp and not over RoCE. Is there any way to test commination over RoCE ? For eg. I was able to test it for libfabric using the below sample that comes along with libfabric to test RMA. Is there something similar for MPICH ? or use the current sample to use RoCE by some parameter? Server : fi_rma_bw -s 192.168.1.100 -e msg -d mlx5_1 -S 1024 -I 1 Client : fi_rma_bw -s 192.168.1.200 -e msg -d mlx5_3 192.168.1.100 -S 1024 -I 1 Regards, Niyaz

1 0

Re: [mpich-discuss] Mpich over RDMA sample
by Niyaz Murshed 12 Jun '24

12 Jun '24

I think image is not gone : https://urldefense.us/v3/__https://ibb.co/RDGnJfy__;!!G_uCfscf7eWS!ZufMq3Gf… From: Niyaz Murshed <Niyaz.Murshed(a)arm.com> Date: Wednesday, June 12, 2024 at 3:56 PM To: Zhou, Hui <zhouh(a)anl.gov>, discuss(a)mpich.org <discuss(a)mpich.org> Cc: nd <nd(a)arm.com> Subject: Re: Mpich over RDMA sample Thank you Hui for the reply. I was expecting to see RoCE messages in the packet capture. When I use verbs on libfabric sample tests, I see the RoCE messages as below : [cid:[email protected]] From: Zhou, Hui <zhouh(a)anl.gov> Date: Wednesday, June 12, 2024 at 3:41 PM To: Niyaz Murshed <Niyaz.Murshed(a)arm.com>, discuss(a)mpich.org <discuss(a)mpich.org> Cc: nd <nd(a)arm.com> Subject: Re: Mpich over RDMA sample Niyaz, ofi_rxm:verbs is the correct provider. The ofi_rxm provides the verbs provider additional messaging semantics that are needed to be used in MPI. -- Hui Zhou From: Niyaz Murshed <Niyaz.Murshed(a)arm.com> Date: Wednesday, June 12, 2024 at 2:54 PM To: Zhou, Hui <zhouh(a)anl.gov>, discuss(a)mpich.org <discuss(a)mpich.org> Cc: nd <nd(a)arm.com> Subject: Re: Mpich over RDMA sample Thank you for the reply. I did that; however, it selects ofi_rxm: verbs libfabric: 412098: 1718221892: : ofi_rxm: av: ofi_av_insert_addr(): 313<info> fi_addr: 1 libfabric: 412099: 1718221892: : ofi_rxm: av: ofi_av_insert_addr(): 313<info> fi_addr: ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Thank you for the reply. I did that; however, it selects ofi_rxm:verbs libfabric:412098:1718221892::ofi_rxm:av:ofi_av_insert_addr():313<info> fi_addr: 1 libfabric:412099:1718221892::ofi_rxm:av:ofi_av_insert_addr():313<info> fi_addr: 1 options: backend: cpu iters: 16 warmup_iters: 16 cache: 1 min_elem_count: 1 max_elem_count: 1 elem_counts: [1] validate: last window_size: 64 #------------------------------------------------------------ # Benchmarking: Bandwidth # #processes: 2 #------------------------------------------------------------ #bytes #repetitions Mbytes/sec 4 16 0.02 # All done libfabric:412098:1718221892::ofi_rxm:ep_ctrl:rxm_stop_listen():864<info> stopping CM thread libfabric:412099:1718221892::ofi_rxm:ep_ctrl:rxm_stop_listen():864<info> stopping CM thread As per https://urldefense.us/v3/__https://www.intel.com/content/www/us/en/docs/mpi… <https://urldefense.us/v3/__https:/www.intel.com/content/www/us/en/docs/mpi-…> , we need to add FI_PROVIDER=^ofi_rxm , but if do that, it moves back to sockets provider. Is here a way to combine ^ofi_rxm and verbs From: Zhou, Hui <zhouh(a)anl.gov> Date: Wednesday, June 12, 2024 at 2:41 PM To: Niyaz Murshed <Niyaz.Murshed(a)arm.com>, discuss(a)mpich.org <discuss(a)mpich.org> Cc: nd <nd(a)arm.com> Subject: Re: Mpich over RDMA sample Niyaz, All you need to do is to set an environment variable `FI_PROVIDER=verbs`. -- Hui Zhou From: Niyaz Murshed <Niyaz.Murshed(a)arm.com> Date: Wednesday, June 12, 2024 at 1:23 PM To: Zhou, Hui <zhouh(a)anl.gov>, discuss(a)mpich.org <discuss(a)mpich.org> Cc: nd <nd(a)arm.com> Subject: Re: Mpich over RDMA sample When testing with Libfabric, verbs provider is selected. I did have to use “-e msg -d mlx5_1” so that it selects verbs. I was checking if there is anything like that for mpich sample tests. Else might need to do some hack in the code to force ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd When testing with Libfabric, verbs provider is selected. I did have to use “-e msg -d mlx5_1” so that it selects verbs. I was checking if there is anything like that for mpich sample tests. Else might need to do some hack in the code to force the selection of verbs. From: Zhou, Hui <zhouh(a)anl.gov> Date: Wednesday, June 12, 2024 at 1:15 PM To: discuss(a)mpich.org <discuss(a)mpich.org> Cc: Niyaz Murshed <Niyaz.Murshed(a)arm.com>, nd <nd(a)arm.com> Subject: Re: Mpich over RDMA sample Libfabric support multiple providers. Sounds like it was selecting the sockets or tcp provider rather than a provider that support RoCE. I am not exactly sure whether the verbs provider will do that. If you can confirm the provider using libfabric tests, then you can try forcing MPICH to use that provider by setting the FI_PROVIDER environment variable. -- Hui Zhou From: Niyaz Murshed via discuss <discuss(a)mpich.org> Date: Wednesday, June 12, 2024 at 9:03 AM To: discuss(a)mpich.org <discuss(a)mpich.org> Cc: Niyaz Murshed <Niyaz.Murshed(a)arm.com>, nd <nd(a)arm.com> Subject: [mpich-discuss] Mpich over RDMA sample Hello, I am trying to learn about MPICH and its performance over RDMA. I am using libfabric and installed mpich using the below configure. ./configure --prefix=/opt/mpich/ --with-ofi=/opt/libfabric/ When I run any applications between 2 directly ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Hello, I am trying to learn about MPICH and its performance over RDMA. I am using libfabric and installed mpich using the below configure. ./configure --prefix=/opt/mpich/ --with-ofi=/opt/libfabric/ When I run any applications between 2 directly connected servers having Mellanox NICs, I see that communication is happening over tcp and not over RoCE. Is there any way to test commination over RoCE ? For eg. I was able to test it for libfabric using the below sample that comes along with libfabric to test RMA. Is there something similar for MPICH ? or use the current sample to use RoCE by some parameter? Server : fi_rma_bw -s 192.168.1.100 -e msg -d mlx5_1 -S 1024 -I 1 Client : fi_rma_bw -s 192.168.1.200 -e msg -d mlx5_3 192.168.1.100 -S 1024 -I 1 Regards, Niyaz

1 0

Re: [mpich-discuss] Mpich over RDMA sample
by Niyaz Murshed 12 Jun '24

12 Jun '24

Thank you Hui for the reply. I was expecting to see RoCE messages in the packet capture. When I use verbs on libfabric sample tests, I see the RoCE messages as below : [cid:[email protected]] From: Zhou, Hui <zhouh(a)anl.gov> Date: Wednesday, June 12, 2024 at 3:41 PM To: Niyaz Murshed <Niyaz.Murshed(a)arm.com>, discuss(a)mpich.org <discuss(a)mpich.org> Cc: nd <nd(a)arm.com> Subject: Re: Mpich over RDMA sample Niyaz, ofi_rxm:verbs is the correct provider. The ofi_rxm provides the verbs provider additional messaging semantics that are needed to be used in MPI. -- Hui Zhou From: Niyaz Murshed <Niyaz.Murshed(a)arm.com> Date: Wednesday, June 12, 2024 at 2:54 PM To: Zhou, Hui <zhouh(a)anl.gov>, discuss(a)mpich.org <discuss(a)mpich.org> Cc: nd <nd(a)arm.com> Subject: Re: Mpich over RDMA sample Thank you for the reply. I did that; however, it selects ofi_rxm: verbs libfabric: 412098: 1718221892: : ofi_rxm: av: ofi_av_insert_addr(): 313<info> fi_addr: 1 libfabric: 412099: 1718221892: : ofi_rxm: av: ofi_av_insert_addr(): 313<info> fi_addr: ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Thank you for the reply. I did that; however, it selects ofi_rxm:verbs libfabric:412098:1718221892::ofi_rxm:av:ofi_av_insert_addr():313<info> fi_addr: 1 libfabric:412099:1718221892::ofi_rxm:av:ofi_av_insert_addr():313<info> fi_addr: 1 options: backend: cpu iters: 16 warmup_iters: 16 cache: 1 min_elem_count: 1 max_elem_count: 1 elem_counts: [1] validate: last window_size: 64 #------------------------------------------------------------ # Benchmarking: Bandwidth # #processes: 2 #------------------------------------------------------------ #bytes #repetitions Mbytes/sec 4 16 0.02 # All done libfabric:412098:1718221892::ofi_rxm:ep_ctrl:rxm_stop_listen():864<info> stopping CM thread libfabric:412099:1718221892::ofi_rxm:ep_ctrl:rxm_stop_listen():864<info> stopping CM thread As per https://urldefense.us/v3/__https://www.intel.com/content/www/us/en/docs/mpi… <https://urldefense.us/v3/__https:/www.intel.com/content/www/us/en/docs/mpi-…> , we need to add FI_PROVIDER=^ofi_rxm , but if do that, it moves back to sockets provider. Is here a way to combine ^ofi_rxm and verbs From: Zhou, Hui <zhouh(a)anl.gov> Date: Wednesday, June 12, 2024 at 2:41 PM To: Niyaz Murshed <Niyaz.Murshed(a)arm.com>, discuss(a)mpich.org <discuss(a)mpich.org> Cc: nd <nd(a)arm.com> Subject: Re: Mpich over RDMA sample Niyaz, All you need to do is to set an environment variable `FI_PROVIDER=verbs`. -- Hui Zhou From: Niyaz Murshed <Niyaz.Murshed(a)arm.com> Date: Wednesday, June 12, 2024 at 1:23 PM To: Zhou, Hui <zhouh(a)anl.gov>, discuss(a)mpich.org <discuss(a)mpich.org> Cc: nd <nd(a)arm.com> Subject: Re: Mpich over RDMA sample When testing with Libfabric, verbs provider is selected. I did have to use “-e msg -d mlx5_1” so that it selects verbs. I was checking if there is anything like that for mpich sample tests. Else might need to do some hack in the code to force ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd When testing with Libfabric, verbs provider is selected. I did have to use “-e msg -d mlx5_1” so that it selects verbs. I was checking if there is anything like that for mpich sample tests. Else might need to do some hack in the code to force the selection of verbs. From: Zhou, Hui <zhouh(a)anl.gov> Date: Wednesday, June 12, 2024 at 1:15 PM To: discuss(a)mpich.org <discuss(a)mpich.org> Cc: Niyaz Murshed <Niyaz.Murshed(a)arm.com>, nd <nd(a)arm.com> Subject: Re: Mpich over RDMA sample Libfabric support multiple providers. Sounds like it was selecting the sockets or tcp provider rather than a provider that support RoCE. I am not exactly sure whether the verbs provider will do that. If you can confirm the provider using libfabric tests, then you can try forcing MPICH to use that provider by setting the FI_PROVIDER environment variable. -- Hui Zhou From: Niyaz Murshed via discuss <discuss(a)mpich.org> Date: Wednesday, June 12, 2024 at 9:03 AM To: discuss(a)mpich.org <discuss(a)mpich.org> Cc: Niyaz Murshed <Niyaz.Murshed(a)arm.com>, nd <nd(a)arm.com> Subject: [mpich-discuss] Mpich over RDMA sample Hello, I am trying to learn about MPICH and its performance over RDMA. I am using libfabric and installed mpich using the below configure. ./configure --prefix=/opt/mpich/ --with-ofi=/opt/libfabric/ When I run any applications between 2 directly ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Hello, I am trying to learn about MPICH and its performance over RDMA. I am using libfabric and installed mpich using the below configure. ./configure --prefix=/opt/mpich/ --with-ofi=/opt/libfabric/ When I run any applications between 2 directly connected servers having Mellanox NICs, I see that communication is happening over tcp and not over RoCE. Is there any way to test commination over RoCE ? For eg. I was able to test it for libfabric using the below sample that comes along with libfabric to test RMA. Is there something similar for MPICH ? or use the current sample to use RoCE by some parameter? Server : fi_rma_bw -s 192.168.1.100 -e msg -d mlx5_1 -S 1024 -I 1 Client : fi_rma_bw -s 192.168.1.200 -e msg -d mlx5_3 192.168.1.100 -S 1024 -I 1 Regards, Niyaz

1 0

Re: [mpich-discuss] Mpich over RDMA sample
by Zhou, Hui 12 Jun '24

12 Jun '24

Niyaz, ofi_rxm:verbs is the correct provider. The ofi_rxm provides the verbs provider additional messaging semantics that are needed to be used in MPI. -- Hui Zhou From: Niyaz Murshed <Niyaz.Murshed(a)arm.com> Date: Wednesday, June 12, 2024 at 2:54 PM To: Zhou, Hui <zhouh(a)anl.gov>, discuss(a)mpich.org <discuss(a)mpich.org> Cc: nd <nd(a)arm.com> Subject: Re: Mpich over RDMA sample Thank you for the reply. I did that; however, it selects ofi_rxm: verbs libfabric: 412098: 1718221892: : ofi_rxm: av: ofi_av_insert_addr(): 313<info> fi_addr: 1 libfabric: 412099: 1718221892: : ofi_rxm: av: ofi_av_insert_addr(): 313<info> fi_addr: ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Thank you for the reply. I did that; however, it selects ofi_rxm:verbs libfabric:412098:1718221892::ofi_rxm:av:ofi_av_insert_addr():313<info> fi_addr: 1 libfabric:412099:1718221892::ofi_rxm:av:ofi_av_insert_addr():313<info> fi_addr: 1 options: backend: cpu iters: 16 warmup_iters: 16 cache: 1 min_elem_count: 1 max_elem_count: 1 elem_counts: [1] validate: last window_size: 64 #------------------------------------------------------------ # Benchmarking: Bandwidth # #processes: 2 #------------------------------------------------------------ #bytes #repetitions Mbytes/sec 4 16 0.02 # All done libfabric:412098:1718221892::ofi_rxm:ep_ctrl:rxm_stop_listen():864<info> stopping CM thread libfabric:412099:1718221892::ofi_rxm:ep_ctrl:rxm_stop_listen():864<info> stopping CM thread As per https://urldefense.us/v3/__https://www.intel.com/content/www/us/en/docs/mpi… <https://urldefense.us/v3/__https:/www.intel.com/content/www/us/en/docs/mpi-…> , we need to add FI_PROVIDER=^ofi_rxm , but if do that, it moves back to sockets provider. Is here a way to combine ^ofi_rxm and verbs From: Zhou, Hui <zhouh(a)anl.gov> Date: Wednesday, June 12, 2024 at 2:41 PM To: Niyaz Murshed <Niyaz.Murshed(a)arm.com>, discuss(a)mpich.org <discuss(a)mpich.org> Cc: nd <nd(a)arm.com> Subject: Re: Mpich over RDMA sample Niyaz, All you need to do is to set an environment variable `FI_PROVIDER=verbs`. -- Hui Zhou From: Niyaz Murshed <Niyaz.Murshed(a)arm.com> Date: Wednesday, June 12, 2024 at 1:23 PM To: Zhou, Hui <zhouh(a)anl.gov>, discuss(a)mpich.org <discuss(a)mpich.org> Cc: nd <nd(a)arm.com> Subject: Re: Mpich over RDMA sample When testing with Libfabric, verbs provider is selected. I did have to use “-e msg -d mlx5_1” so that it selects verbs. I was checking if there is anything like that for mpich sample tests. Else might need to do some hack in the code to force ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd When testing with Libfabric, verbs provider is selected. I did have to use “-e msg -d mlx5_1” so that it selects verbs. I was checking if there is anything like that for mpich sample tests. Else might need to do some hack in the code to force the selection of verbs. From: Zhou, Hui <zhouh(a)anl.gov> Date: Wednesday, June 12, 2024 at 1:15 PM To: discuss(a)mpich.org <discuss(a)mpich.org> Cc: Niyaz Murshed <Niyaz.Murshed(a)arm.com>, nd <nd(a)arm.com> Subject: Re: Mpich over RDMA sample Libfabric support multiple providers. Sounds like it was selecting the sockets or tcp provider rather than a provider that support RoCE. I am not exactly sure whether the verbs provider will do that. If you can confirm the provider using libfabric tests, then you can try forcing MPICH to use that provider by setting the FI_PROVIDER environment variable. -- Hui Zhou From: Niyaz Murshed via discuss <discuss(a)mpich.org> Date: Wednesday, June 12, 2024 at 9:03 AM To: discuss(a)mpich.org <discuss(a)mpich.org> Cc: Niyaz Murshed <Niyaz.Murshed(a)arm.com>, nd <nd(a)arm.com> Subject: [mpich-discuss] Mpich over RDMA sample Hello, I am trying to learn about MPICH and its performance over RDMA. I am using libfabric and installed mpich using the below configure. ./configure --prefix=/opt/mpich/ --with-ofi=/opt/libfabric/ When I run any applications between 2 directly ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Hello, I am trying to learn about MPICH and its performance over RDMA. I am using libfabric and installed mpich using the below configure. ./configure --prefix=/opt/mpich/ --with-ofi=/opt/libfabric/ When I run any applications between 2 directly connected servers having Mellanox NICs, I see that communication is happening over tcp and not over RoCE. Is there any way to test commination over RoCE ? For eg. I was able to test it for libfabric using the below sample that comes along with libfabric to test RMA. Is there something similar for MPICH ? or use the current sample to use RoCE by some parameter? Server : fi_rma_bw -s 192.168.1.100 -e msg -d mlx5_1 -S 1024 -I 1 Client : fi_rma_bw -s 192.168.1.200 -e msg -d mlx5_3 192.168.1.100 -S 1024 -I 1 Regards, Niyaz

1 0

Re: [mpich-discuss] Mpich over RDMA sample
by Niyaz Murshed 12 Jun '24

12 Jun '24

Thank you for the reply. I did that; however, it selects ofi_rxm:verbs libfabric:412098:1718221892::ofi_rxm:av:ofi_av_insert_addr():313<info> fi_addr: 1 libfabric:412099:1718221892::ofi_rxm:av:ofi_av_insert_addr():313<info> fi_addr: 1 options: backend: cpu iters: 16 warmup_iters: 16 cache: 1 min_elem_count: 1 max_elem_count: 1 elem_counts: [1] validate: last window_size: 64 #------------------------------------------------------------ # Benchmarking: Bandwidth # #processes: 2 #------------------------------------------------------------ #bytes #repetitions Mbytes/sec 4 16 0.02 # All done libfabric:412098:1718221892::ofi_rxm:ep_ctrl:rxm_stop_listen():864<info> stopping CM thread libfabric:412099:1718221892::ofi_rxm:ep_ctrl:rxm_stop_listen():864<info> stopping CM thread As per https://urldefense.us/v3/__https://www.intel.com/content/www/us/en/docs/mpi… , we need to add FI_PROVIDER=^ofi_rxm , but if do that, it moves back to sockets provider. Is here a way to combine ^ofi_rxm and verbs From: Zhou, Hui <zhouh(a)anl.gov> Date: Wednesday, June 12, 2024 at 2:41 PM To: Niyaz Murshed <Niyaz.Murshed(a)arm.com>, discuss(a)mpich.org <discuss(a)mpich.org> Cc: nd <nd(a)arm.com> Subject: Re: Mpich over RDMA sample Niyaz, All you need to do is to set an environment variable `FI_PROVIDER=verbs`. -- Hui Zhou From: Niyaz Murshed <Niyaz.Murshed(a)arm.com> Date: Wednesday, June 12, 2024 at 1:23 PM To: Zhou, Hui <zhouh(a)anl.gov>, discuss(a)mpich.org <discuss(a)mpich.org> Cc: nd <nd(a)arm.com> Subject: Re: Mpich over RDMA sample When testing with Libfabric, verbs provider is selected. I did have to use “-e msg -d mlx5_1” so that it selects verbs. I was checking if there is anything like that for mpich sample tests. Else might need to do some hack in the code to force ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd When testing with Libfabric, verbs provider is selected. I did have to use “-e msg -d mlx5_1” so that it selects verbs. I was checking if there is anything like that for mpich sample tests. Else might need to do some hack in the code to force the selection of verbs. From: Zhou, Hui <zhouh(a)anl.gov> Date: Wednesday, June 12, 2024 at 1:15 PM To: discuss(a)mpich.org <discuss(a)mpich.org> Cc: Niyaz Murshed <Niyaz.Murshed(a)arm.com>, nd <nd(a)arm.com> Subject: Re: Mpich over RDMA sample Libfabric support multiple providers. Sounds like it was selecting the sockets or tcp provider rather than a provider that support RoCE. I am not exactly sure whether the verbs provider will do that. If you can confirm the provider using libfabric tests, then you can try forcing MPICH to use that provider by setting the FI_PROVIDER environment variable. -- Hui Zhou From: Niyaz Murshed via discuss <discuss(a)mpich.org> Date: Wednesday, June 12, 2024 at 9:03 AM To: discuss(a)mpich.org <discuss(a)mpich.org> Cc: Niyaz Murshed <Niyaz.Murshed(a)arm.com>, nd <nd(a)arm.com> Subject: [mpich-discuss] Mpich over RDMA sample Hello, I am trying to learn about MPICH and its performance over RDMA. I am using libfabric and installed mpich using the below configure. ./configure --prefix=/opt/mpich/ --with-ofi=/opt/libfabric/ When I run any applications between 2 directly ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Hello, I am trying to learn about MPICH and its performance over RDMA. I am using libfabric and installed mpich using the below configure. ./configure --prefix=/opt/mpich/ --with-ofi=/opt/libfabric/ When I run any applications between 2 directly connected servers having Mellanox NICs, I see that communication is happening over tcp and not over RoCE. Is there any way to test commination over RoCE ? For eg. I was able to test it for libfabric using the below sample that comes along with libfabric to test RMA. Is there something similar for MPICH ? or use the current sample to use RoCE by some parameter? Server : fi_rma_bw -s 192.168.1.100 -e msg -d mlx5_1 -S 1024 -I 1 Client : fi_rma_bw -s 192.168.1.200 -e msg -d mlx5_3 192.168.1.100 -S 1024 -I 1 Regards, Niyaz

1 0