This Message Is From an External Sender

This message came from outside your organization.

Hi Hui,

I would still need some assistance in this. I am not able to make mpich chose verbs provider ☹

With the log :

provider: verbs, score = 0, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200

Is my assumption to think the above log represent that verbs provider knows about a path to 192.168.2.200 ?

How does the application hello-world decide which provider to use?

Running on server: 192.168.2.100

root@ampere-altra-2-1# FI_PROVIDER=verbs MPIR_CVAR_DEBUG_SUMMARY=1 mpirun -n 5 -hosts 192.168.2.200,192.168.2.100 /mpich/examples/a.out

==== Various sizes and limits ====

sizeof(MPIDI_per_vci_t): 192

Required minimum FI_VERSION: 0, current version: 10015

provider: verbs, score = 0, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200

provider: verbs, score = 0, pref = 0, FI_SOCKADDR_IN6 [28] fe80::526b:4bff:fefc:1342

provider: verbs, score = 0, pref = 0, FI_SOCKADDR_IB [48]

provider: verbs, score = 0, pref = 0, FI_SOCKADDR_IB [48]

provider: verbs, score = 0, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200

provider: verbs, score = 0, pref = 0, FI_SOCKADDR_IN6 [28] fe80::526b:4bff:fefc:1342

provider: verbs, score = 0, pref = 0, FI_SOCKADDR_IB [48]

provider: verbs, score = 0, pref = 0, FI_SOCKADDR_IB [48]

provider: verbs, score = 0, pref = 0, FI_FORMAT_UNSPEC [32]

provider: verbs, score = 0, pref = 0, FI_FORMAT_UNSPEC [32]

provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200

provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::526b:4bff:fefc:1342

provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48]

provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48]

provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200

provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::526b:4bff:fefc:1342

provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48]

provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48]

provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200

provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::526b:4bff:fefc:1342

provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48]

provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48]

provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200

provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::526b:4bff:fefc:1342

provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48]

provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48]

provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200

provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::526b:4bff:fefc:1342

provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48]

provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48]

provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200

provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::526b:4bff:fefc:1342

provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48]

provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48]

provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200

provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::526b:4bff:fefc:1342

provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48]

provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48]

provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200

provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::526b:4bff:fefc:1342

provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48]

provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48]

provider: verbs;ofi_rxd, score = 5, pref = -2, FI_FORMAT_UNSPEC [32]

provider: verbs;ofi_rxd, score = 5, pref = -2, FI_FORMAT_UNSPEC [32]

Required minimum FI_VERSION: 10006, current version: 10015

==== Capability set configuration ====

libfabric provider: verbs;ofi_rxm - IB-0xfe80000000000000

MPIDI_OFI_ENABLE_DATA: 0

MPIDI_OFI_ENABLE_AV_TABLE: 1

MPIDI_OFI_ENABLE_SCALABLE_ENDPOINTS: 0

MPIDI_OFI_ENABLE_SHARED_CONTEXTS: 0

MPIDI_OFI_ENABLE_MR_VIRT_ADDRESS: 1

MPIDI_OFI_ENABLE_MR_ALLOCATED: 1

MPIDI_OFI_ENABLE_MR_REGISTER_NULL: 1

MPIDI_OFI_ENABLE_MR_PROV_KEY: 1

MPIDI_OFI_ENABLE_TAGGED: 1

MPIDI_OFI_ENABLE_AM: 1

MPIDI_OFI_ENABLE_RMA: 1

MPIDI_OFI_ENABLE_ATOMICS: 0

MPIDI_OFI_FETCH_ATOMIC_IOVECS: 1

MPIDI_OFI_ENABLE_DATA_AUTO_PROGRESS: 0

MPIDI_OFI_ENABLE_CONTROL_AUTO_PROGRESS: 0

MPIDI_OFI_ENABLE_PT2PT_NOPACK: 1

MPIDI_OFI_ENABLE_TRIGGERED: 0

MPIDI_OFI_ENABLE_HMEM: 0

MPIDI_OFI_NUM_AM_BUFFERS: 8

MPIDI_OFI_NUM_OPTIMIZED_MEMORY_REGIONS: 0

MPIDI_OFI_CONTEXT_BITS: 16

MPIDI_OFI_SOURCE_BITS: 23

MPIDI_OFI_TAG_BITS: 20

MPIDI_OFI_VNI_USE_DOMAIN: 1

MAXIMUM SUPPORTED RANKS: 8388608

MAXIMUM TAG: 1048576

==== Provider global thresholds ====

max_buffered_send: 192

max_buffered_write: 192

max_msg_size: 1073741824

max_order_raw: 1073741824

max_order_war: 0

max_order_waw: 1073741824

tx_iov_limit: 4

rx_iov_limit: 4

rma_iov_limit: 1

max_mr_key_size: 4

==== Various sizes and limits ====

MPIDI_OFI_AM_MSG_HEADER_SIZE: 24

MPIDI_OFI_MAX_AM_HDR_SIZE: 255

sizeof(MPIDI_OFI_am_request_header_t): 416

sizeof(MPIDI_OFI_per_vci_t): 52480

MPIDI_OFI_AM_HDR_POOL_CELL_SIZE: 1024

MPIDI_OFI_DEFAULT_SHORT_SEND_SIZE: 16384

Assertion failed in file src/mpid/ch4/netmod/ofi/init_addrxchg.c at line 151: mapped_table[i] != FI_ADDR_NOTAVAIL

Assertion failed in file src/mpid/ch4/netmod/ofi/init_addrxchg.c at line 151: mapped_table[i] != FI_ADDR_NOTAVAIL

Assertion failed in file src/mpid/ch4/netmod/ofi/init_addrxchg.c at line 151: mapped_table[i] != FI_ADDR_NOTAVAIL

Assertion failed in file src/mpid/ch4/netmod/ofi/init_addrxchg.c at line 151: mapped_table[i] != FI_ADDR_NOTAVAIL

Assertion failed in file src/mpid/ch4/netmod/ofi/init_addrxchg.c at line 151: mapped_table[i] != FI_ADDR_NOTAVAIL

If I do not use the FI_PROVIDER=verbs, it will work over sockets.

root@ampere-altra-2-1: # MPIR_CVAR_DEBUG_SUMMARY=1 mpirun -n 5 -hosts 192.168.2.200,192.168.2.100 /mpich/examples/a.out

==== Various sizes and limits ====

sizeof(MPIDI_per_vci_t): 192

Required minimum FI_VERSION: 0, current version: 10015

provider: verbs, score = 0, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200

provider: verbs, score = 0, pref = 0, FI_SOCKADDR_IN6 [28] fe80::526b:4bff:fefc:1342

provider: verbs, score = 0, pref = 0, FI_SOCKADDR_IB [48]

provider: verbs, score = 0, pref = 0, FI_SOCKADDR_IB [48]

provider: verbs, score = 0, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200

provider: verbs, score = 0, pref = 0, FI_SOCKADDR_IN6 [28] fe80::526b:4bff:fefc:1342

provider: verbs, score = 0, pref = 0, FI_SOCKADDR_IB [48]

provider: verbs, score = 0, pref = 0, FI_SOCKADDR_IB [48]

provider: verbs, score = 0, pref = 0, FI_FORMAT_UNSPEC [32]

provider: verbs, score = 0, pref = 0, FI_FORMAT_UNSPEC [32]

provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200

provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::526b:4bff:fefc:1342

provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48]

provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48]

provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200

provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::526b:4bff:fefc:1342

provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48]

provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48]

provider: tcp, score = 4, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200

provider: tcp, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::526b:4bff:fefc:1342

provider: tcp, score = 4, pref = 0, FI_SOCKADDR_IN [16] 10.118.91.159

provider: tcp, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::be97:e1ff:fe9d:7caa

provider: tcp, score = 4, pref = 0, FI_SOCKADDR_IN [16] 127.0.0.1

provider: tcp, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] ::1

provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200

provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::526b:4bff:fefc:1342

provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 10.118.91.159

provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::be97:e1ff:fe9d:7caa

provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 127.0.0.1

provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] ::1

provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200

provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::526b:4bff:fefc:1342

provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48]

provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48]

provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200

provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::526b:4bff:fefc:1342

provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48]

provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48]

provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200

provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::526b:4bff:fefc:1342

provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 10.118.91.159

provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::be97:e1ff:fe9d:7caa

provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 127.0.0.1

provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] ::1

provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200

provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::526b:4bff:fefc:1342

provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48]

provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48]

provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200

provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::526b:4bff:fefc:1342

provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48]

provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48]

provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200

provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::526b:4bff:fefc:1342

provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 10.118.91.159

provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::be97:e1ff:fe9d:7caa

provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 127.0.0.1

provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] ::1

provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200

provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::526b:4bff:fefc:1342

provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48]

provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48]

provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200

provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::526b:4bff:fefc:1342

provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48]

provider: verbs;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IB [48]

provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200

provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::526b:4bff:fefc:1342

provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 10.118.91.159

provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::be97:e1ff:fe9d:7caa

provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN [16] 127.0.0.1

provider: tcp;ofi_rxm, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] ::1

provider: verbs;ofi_rxd, score = 5, pref = -2, FI_FORMAT_UNSPEC [32]

provider: verbs;ofi_rxd, score = 5, pref = -2, FI_FORMAT_UNSPEC [32]

provider: udp;ofi_rxd, score = 5, pref = -2, FI_SOCKADDR_IN [16] 192.168.2.200

provider: udp;ofi_rxd, score = 5, pref = -2, FI_SOCKADDR_IN6 [28] fe80::526b:4bff:fefc:1342

provider: udp;ofi_rxd, score = 5, pref = -2, FI_SOCKADDR_IN [16] 10.118.91.159

provider: udp;ofi_rxd, score = 5, pref = -2, FI_SOCKADDR_IN6 [28] fe80::be97:e1ff:fe9d:7caa

provider: udp;ofi_rxd, score = 5, pref = -2, FI_SOCKADDR_IN [16] 127.0.0.1

provider: udp;ofi_rxd, score = 5, pref = -2, FI_SOCKADDR_IN6 [28] ::1

provider: shm, score = 4, pref = -2, FI_ADDR_STR [14] - fi_shm://1177

provider: shm, score = 4, pref = -2, FI_ADDR_STR [14] - fi_shm://1177

provider: udp, score = 0, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200

provider: udp, score = 0, pref = 0, FI_SOCKADDR_IN6 [28] fe80::526b:4bff:fefc:1342

provider: udp, score = 0, pref = 0, FI_SOCKADDR_IN [16] 10.118.91.159

provider: udp, score = 0, pref = 0, FI_SOCKADDR_IN6 [28] fe80::be97:e1ff:fe9d:7caa

provider: udp, score = 0, pref = 0, FI_SOCKADDR_IN [16] 127.0.0.1

provider: udp, score = 0, pref = 0, FI_SOCKADDR_IN6 [28] ::1

provider: tcp, score = 0, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200

provider: tcp, score = 0, pref = 0, FI_SOCKADDR_IN6 [28] fe80::526b:4bff:fefc:1342

provider: tcp, score = 0, pref = 0, FI_SOCKADDR_IN [16] 10.118.91.159

provider: tcp, score = 0, pref = 0, FI_SOCKADDR_IN6 [28] fe80::be97:e1ff:fe9d:7caa

provider: tcp, score = 0, pref = 0, FI_SOCKADDR_IN [16] 127.0.0.1

provider: tcp, score = 0, pref = 0, FI_SOCKADDR_IN6 [28] ::1

provider: tcp, score = 4, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200

provider: tcp, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::526b:4bff:fefc:1342

provider: tcp, score = 4, pref = 0, FI_SOCKADDR_IN [16] 10.118.91.159

provider: tcp, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] fe80::be97:e1ff:fe9d:7caa

provider: tcp, score = 4, pref = 0, FI_SOCKADDR_IN [16] 127.0.0.1

provider: tcp, score = 4, pref = 0, FI_SOCKADDR_IN6 [28] ::1

provider: sockets, score = 3, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200

provider: sockets, score = 3, pref = 0, FI_SOCKADDR_IN6 [28] fe80::526b:4bff:fefc:1342

provider: sockets, score = 3, pref = 0, FI_SOCKADDR_IN [16] 10.118.91.159

provider: sockets, score = 3, pref = 0, FI_SOCKADDR_IN6 [28] fe80::be97:e1ff:fe9d:7caa

provider: sockets, score = 3, pref = 0, FI_SOCKADDR_IN [16] 127.0.0.1

provider: sockets, score = 3, pref = 0, FI_SOCKADDR_IN6 [28] ::1

provider: sockets, score = 5, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200

provider: sockets, score = 5, pref = 0, FI_SOCKADDR_IN6 [28] fe80::526b:4bff:fefc:1342

provider: sockets, score = 5, pref = 0, FI_SOCKADDR_IN [16] 10.118.91.159

provider: sockets, score = 5, pref = 0, FI_SOCKADDR_IN6 [28] fe80::be97:e1ff:fe9d:7caa

provider: sockets, score = 5, pref = 0, FI_SOCKADDR_IN [16] 127.0.0.1

provider: sockets, score = 5, pref = 0, FI_SOCKADDR_IN6 [28] ::1

provider: sockets, score = 5, pref = 0, FI_SOCKADDR_IN [16] 192.168.2.200

provider: sockets, score = 5, pref = 0, FI_SOCKADDR_IN6 [28] fe80::526b:4bff:fefc:1342

provider: sockets, score = 5, pref = 0, FI_SOCKADDR_IN [16] 10.118.91.159

provider: sockets, score = 5, pref = 0, FI_SOCKADDR_IN6 [28] fe80::be97:e1ff:fe9d:7caa

provider: sockets, score = 5, pref = 0, FI_SOCKADDR_IN [16] 127.0.0.1

provider: sockets, score = 5, pref = 0, FI_SOCKADDR_IN6 [28] ::1

provider: sm2, score = 3, pref = 0, FI_ADDR_STR [13] - fi_sm2://1177

provider: sm2, score = 3, pref = 0, FI_ADDR_STR [13] - fi_sm2://1177

Required minimum FI_VERSION: 10005, current version: 10015

==== Capability set configuration ====

libfabric provider: sockets - 192.168.2.0/24

MPIDI_OFI_ENABLE_DATA: 1

MPIDI_OFI_ENABLE_AV_TABLE: 1

MPIDI_OFI_ENABLE_SCALABLE_ENDPOINTS: 1

MPIDI_OFI_ENABLE_SHARED_CONTEXTS: 0

MPIDI_OFI_ENABLE_MR_VIRT_ADDRESS: 0

MPIDI_OFI_ENABLE_MR_ALLOCATED: 0

MPIDI_OFI_ENABLE_MR_REGISTER_NULL: 1

MPIDI_OFI_ENABLE_MR_PROV_KEY: 0

MPIDI_OFI_ENABLE_TAGGED: 1

MPIDI_OFI_ENABLE_AM: 1

MPIDI_OFI_ENABLE_RMA: 1

MPIDI_OFI_ENABLE_ATOMICS: 1

MPIDI_OFI_FETCH_ATOMIC_IOVECS: 1

MPIDI_OFI_ENABLE_DATA_AUTO_PROGRESS: 0

MPIDI_OFI_ENABLE_CONTROL_AUTO_PROGRESS: 0

MPIDI_OFI_ENABLE_PT2PT_NOPACK: 1

MPIDI_OFI_ENABLE_TRIGGERED: 0

MPIDI_OFI_ENABLE_HMEM: 0

MPIDI_OFI_NUM_AM_BUFFERS: 8

MPIDI_OFI_NUM_OPTIMIZED_MEMORY_REGIONS: 0

MPIDI_OFI_CONTEXT_BITS: 20

MPIDI_OFI_SOURCE_BITS: 0

MPIDI_OFI_TAG_BITS: 31

MPIDI_OFI_VNI_USE_DOMAIN: 1

MAXIMUM SUPPORTED RANKS: 4294967296

MAXIMUM TAG: 2147483648

==== Provider global thresholds ====

max_buffered_send: 255

max_buffered_write: 255

max_msg_size: 9223372036854775807

max_order_raw: -1

max_order_war: -1

max_order_waw: -1

tx_iov_limit: 8

rx_iov_limit: 8

rma_iov_limit: 8

max_mr_key_size: 8

==== Various sizes and limits ====

MPIDI_OFI_AM_MSG_HEADER_SIZE: 24

MPIDI_OFI_MAX_AM_HDR_SIZE: 255

sizeof(MPIDI_OFI_am_request_header_t): 416

sizeof(MPIDI_OFI_per_vci_t): 52480

MPIDI_OFI_AM_HDR_POOL_CELL_SIZE: 1024

MPIDI_OFI_DEFAULT_SHORT_SEND_SIZE: 16384

==== OFI dynamic settings ====

num_vcis: 1

num_nics: 1

======================================

error checking : enabled

QMPI : disabled

debugger support : disabled

thread level : MPI_THREAD_SINGLE

thread CS : per-vci

threadcomm : enabled

==== data structure summary ====

sizeof(MPIR_Comm): 1808

sizeof(MPIR_Request): 512

sizeof(MPIR_Datatype): 280

================================

Hello world from process 0 of 5

Hello world from process 2 of 5

Hello world from process 1 of 5

Hello world from process 4 of 5

Hello world from process 3 of 5

From: Zhou, Hui <zhouh@anl.gov>
Date: Wednesday, June 12, 2024 at 7:22 PM
To: Niyaz Murshed <Niyaz.Murshed@arm.com>, discuss@mpich.org <discuss@mpich.org>
Subject: Re: Mpich over RDMA sample

It’s possible that the native path for RMA on verbs are turned off due to verbs does not meet MPI’s full atomic ordering semantics. We’ll check and follow up.

--
Hui Zhou

From: Niyaz Murshed <Niyaz.Murshed@arm.com>
Date: Wednesday, June 12, 2024 at 3:59 PM
To: Zhou, Hui <zhouh@anl.gov>, discuss@mpich.org <discuss@mpich.org>
Cc: nd <nd@arm.com>
Subject: Re: Mpich over RDMA sample

I think image is not gone : https: //ibb. co/RDGnJfy From: Niyaz Murshed <Niyaz. Murshed@ arm. com> Date: Wednesday, June 12, 2024 at 3: 56 PM To: Zhou, Hui <zhouh@ anl. gov>, discuss@ mpich. org <discuss@ mpich. org> Cc: nd <nd@ arm. com>

ZjQcmQRYFpfptBannerStart

This Message Is From an External Sender

This message came from outside your organization.

ZjQcmQRYFpfptBannerEnd

I think image is not gone : https://ibb.co/RDGnJfy

From: Niyaz Murshed <Niyaz.Murshed@arm.com>
Date: Wednesday, June 12, 2024 at 3:56 PM
To: Zhou, Hui <zhouh@anl.gov>, discuss@mpich.org <discuss@mpich.org>
Cc: nd <nd@arm.com>
Subject: Re: Mpich over RDMA sample

Thank you Hui for the reply.

I was expecting to see RoCE messages in the packet capture.

When I use verbs on libfabric sample tests, I see the RoCE messages as below :

From: Zhou, Hui <zhouh@anl.gov>
Date: Wednesday, June 12, 2024 at 3:41 PM
To: Niyaz Murshed <Niyaz.Murshed@arm.com>, discuss@mpich.org <discuss@mpich.org>
Cc: nd <nd@arm.com>
Subject: Re: Mpich over RDMA sample

Niyaz,

ofi_rxm:verbs is the correct provider. The ofi_rxm provides the verbs provider additional messaging semantics that are needed to be used in MPI.

--
Hui Zhou

From: Niyaz Murshed <Niyaz.Murshed@arm.com>
Date: Wednesday, June 12, 2024 at 2:54 PM
To: Zhou, Hui <zhouh@anl.gov>, discuss@mpich.org <discuss@mpich.org>
Cc: nd <nd@arm.com>
Subject: Re: Mpich over RDMA sample

Thank you for the reply. I did that; however, it selects ofi_rxm: verbs libfabric: 412098: 1718221892: : ofi_rxm: av: ofi_av_insert_addr(): 313<info> fi_addr: 1 libfabric: 412099: 1718221892: : ofi_rxm: av: ofi_av_insert_addr(): 313<info> fi_addr:

ZjQcmQRYFpfptBannerStart

This Message Is From an External Sender

This message came from outside your organization.

ZjQcmQRYFpfptBannerEnd

Thank you for the reply.

I did that; however, it selects ofi_rxm:verbs

libfabric:412098:1718221892::ofi_rxm:av:ofi_av_insert_addr():313<info> fi_addr: 1

libfabric:412099:1718221892::ofi_rxm:av:ofi_av_insert_addr():313<info> fi_addr: 1

options:

backend: cpu

iters: 16

warmup_iters: 16

cache: 1

min_elem_count: 1

max_elem_count: 1

elem_counts: [1]

validate: last

window_size: 64

#------------------------------------------------------------

# Benchmarking: Bandwidth

# #processes: 2

#------------------------------------------------------------

#bytes #repetitions Mbytes/sec

4 16 0.02

# All done

libfabric:412098:1718221892::ofi_rxm:ep_ctrl:rxm_stop_listen():864<info> stopping CM thread

libfabric:412099:1718221892::ofi_rxm:ep_ctrl:rxm_stop_listen():864<info> stopping CM thread

As per https://www.intel.com/content/www/us/en/docs/mpi-library/developer-guide-linux/2021-6/ofi-providers-support.html , we need to add FI_PROVIDER=^ofi_rxm , but if do that, it moves back to sockets provider.

Is here a way to combine ^ofi_rxm and verbs

From: Zhou, Hui <zhouh@anl.gov>
Date: Wednesday, June 12, 2024 at 2:41 PM
To: Niyaz Murshed <Niyaz.Murshed@arm.com>, discuss@mpich.org <discuss@mpich.org>
Cc: nd <nd@arm.com>
Subject: Re: Mpich over RDMA sample

Niyaz,

All you need to do is to set an environment variable `FI_PROVIDER=verbs`.

--
Hui Zhou

From: Niyaz Murshed <Niyaz.Murshed@arm.com>
Date: Wednesday, June 12, 2024 at 1:23 PM
To: Zhou, Hui <zhouh@anl.gov>, discuss@mpich.org <discuss@mpich.org>
Cc: nd <nd@arm.com>
Subject: Re: Mpich over RDMA sample

When testing with Libfabric, verbs provider is selected. I did have to use “-e msg -d mlx5_1” so that it selects verbs. I was checking if there is anything like that for mpich sample tests. Else might need to do some hack in the code to force

ZjQcmQRYFpfptBannerStart

This Message Is From an External Sender

This message came from outside your organization.

ZjQcmQRYFpfptBannerEnd

When testing with Libfabric, verbs provider is selected. I did have to use “-e msg -d mlx5_1” so that it selects verbs.

I was checking if there is anything like that for mpich sample tests. Else might need to do some hack in the code to force the selection of verbs.

From: Zhou, Hui <zhouh@anl.gov>
Date: Wednesday, June 12, 2024 at 1:15 PM
To: discuss@mpich.org <discuss@mpich.org>
Cc: Niyaz Murshed <Niyaz.Murshed@arm.com>, nd <nd@arm.com>
Subject: Re: Mpich over RDMA sample

Libfabric support multiple providers. Sounds like it was selecting the sockets or tcp provider rather than a provider that support RoCE. I am not exactly sure whether the verbs provider will do that. If you can confirm the provider using libfabric tests, then you can try forcing MPICH to use that provider by setting the FI_PROVIDER environment variable.

--
Hui Zhou

From: Niyaz Murshed via discuss <discuss@mpich.org>
Date: Wednesday, June 12, 2024 at 9:03 AM
To: discuss@mpich.org <discuss@mpich.org>
Cc: Niyaz Murshed <Niyaz.Murshed@arm.com>, nd <nd@arm.com>
Subject: [mpich-discuss] Mpich over RDMA sample

Hello, I am trying to learn about MPICH and its performance over RDMA. I am using libfabric and installed mpich using the below configure. ./configure --prefix=/opt/mpich/ --with-ofi=/opt/libfabric/ When I run any applications between 2 directly

ZjQcmQRYFpfptBannerStart

This Message Is From an External Sender

This message came from outside your organization.

ZjQcmQRYFpfptBannerEnd

Hello,

I am trying to learn about MPICH and its performance over RDMA.

I am using libfabric and installed mpich using the below configure.

./configure --prefix=/opt/mpich/ --with-ofi=/opt/libfabric/

When I run any applications between 2 directly connected servers having Mellanox NICs, I see that communication is happening over tcp and not over RoCE.

Is there any way to test commination over RoCE ?

For eg. I was able to test it for libfabric using the below sample that comes along with libfabric to test RMA.

Is there something similar for MPICH ? or use the current sample to use RoCE by some parameter?

Server :

fi_rma_bw -s 192.168.1.100 -e msg -d mlx5_1 -S 1024 -I 1

Client :

fi_rma_bw -s 192.168.1.200 -e msg -d mlx5_3 192.168.1.100 -S 1024 -I 1

Regards,

Niyaz