Hello, 

I am trying to learn about MPICH and its performance over RDMA.

I am using libfabric and installed mpich using the below configure.

./configure --prefix=/opt/mpich/  --with-ofi=/opt/libfabric/

When I run any applications between 2 directly connected servers having Mellanox NICs, I see that communication is happening over tcp and not over RoCE.

Is there any way to test commination over RoCE ?

For eg. I was able to test it for libfabric using the below sample that comes along with libfabric to test RMA.

Is there something similar for MPICH ? or use the current sample to use RoCE by some parameter?

Server : 

fi_rma_bw -s   192.168.1.100  -e msg   -d mlx5_1 -S 1024 -I 1 

Client :

fi_rma_bw -s   192.168.1.200  -e msg   -d mlx5_3  192.168.1.100  -S 1024 -I 1 

Regards,

Niyaz