Those warning messages are not coming from MPICH. Can you run ldd one of the benchmark binaries and confirm all the expected libraries are linked to it? Are you perhaps also linked against an hcoll library? Ken On 11/27/19 12:58 PM, Sajid Ali wrote:

Hi Kenneth,

Running the same test with mpich@develop, I still see the following errors in a typical outfile :

|[sas4990@quser10 mdb6i63]$ cat lat_10 ---------------------------------------- ... ---------------------------------------- slurmstepd: error: mpi/pmi2: no value for key in req slurmstepd: error: mpi/pmi2: no value for key in req slurmstepd: error: mpi/pmi2: no value for key in req [1574880837.368994] [qnode5038:4743 :0] ucp_context.c:1131 UCX WARN UCP version is incompatible, required: 1.5, actual: 1.4 (release 0) slurmstepd: error: mpi/pmi2: no value for key in req slurmstepd: error: mpi/pmi2: no value for key in req [1574880837.377860] [qnode5039:4546 :0] ucp_context.c:1131 UCX WARN UCP version is incompatible, required: 1.5, actual: 1.4 (release 0) slurmstepd: error: mpi/pmi2: no value for key in req slurmstepd: error: mpi/pmi2: no value for key in req slurmstepd: error: mpi/pmi2: no value for key in req slurmstepd: error: mpi/pmi2: no value for key in req slurmstepd: error: mpi/pmi2: no value for key in req # OSU MPI_Get latency Test v5.6.1 # Window creation: MPI_Win_allocate # Synchronization: MPI_Win_flush # Size Latency (us) 1 2.13 .... |

Since UCX is a vendored dependency of MPICH, why is MPICH complaining about an incompatible UCX version ?

Thank You, Sajid Ali | PhD Candidate Applied Physics Northwestern University s-sajid-ali.github.io <http://s-sajid-ali.github.io>