Libfabric support multiple providers. Sounds like it was selecting the sockets or tcp provider rather than a provider that support RoCE. I am not exactly sure whether the verbs provider will do that. If you can confirm the provider using libfabric tests, then you can try forcing MPICH to use that provider by setting the FI_PROVIDER environment variable.

-- 
Hui Zhou

From: Niyaz Murshed via discuss <discuss@mpich.org>
Date: Wednesday, June 12, 2024 at 9:03 AM
To: discuss@mpich.org <discuss@mpich.org>
Cc: Niyaz Murshed <Niyaz.Murshed@arm.com>, nd <nd@arm.com>
Subject: [mpich-discuss] Mpich over RDMA sample

Hello, I am trying to learn about MPICH and its performance over RDMA. I am using libfabric and installed mpich using the below configure. ./configure --prefix=/opt/mpich/ --with-ofi=/opt/libfabric/ When I run any applications between 2 directly

ZjQcmQRYFpfptBannerStart

This Message Is From an External Sender

This message came from outside your organization.

ZjQcmQRYFpfptBannerEnd

Hello, 

I am trying to learn about MPICH and its performance over RDMA.

I am using libfabric and installed mpich using the below configure.

./configure --prefix=/opt/mpich/  --with-ofi=/opt/libfabric/

When I run any applications between 2 directly connected servers having Mellanox NICs, I see that communication is happening over tcp and not over RoCE.

Is there any way to test commination over RoCE ?

For eg. I was able to test it for libfabric using the below sample that comes along with libfabric to test RMA.

Is there something similar for MPICH ? or use the current sample to use RoCE by some parameter?

Server : 

fi_rma_bw -s   192.168.1.100  -e msg   -d mlx5_1 -S 1024 -I 1 

Client :

fi_rma_bw -s   192.168.1.200  -e msg   -d mlx5_3  192.168.1.100  -S 1024 -I 1 

Regards,

Niyaz