Hi Kenneth,

Running the same test with mpich@develop, I still see the following errors in a typical outfile :

[sas4990@quser10 mdb6i63]$ cat lat_10
----------------------------------------    
...                                         
----------------------------------------    
slurmstepd: error: mpi/pmi2: no value for key  in req 
slurmstepd: error: mpi/pmi2: no value for key  in req                                                                                                                  
slurmstepd: error: mpi/pmi2: no value for key  in req                                                                                                                  
[1574880837.368994] [qnode5038:4743 :0]    ucp_context.c:1131 UCX  WARN  UCP version is incompatible, required: 1.5, actual: 1.4 
(release 0)                           slurmstepd: error: mpi/pmi2: no value for key  in req                                                                                                                  
slurmstepd: error: mpi/pmi2: no value for key  in req                                                                                                                  
[1574880837.377860] [qnode5039:4546 :0]    ucp_context.c:1131 UCX  WARN  UCP version is incompatible, required: 1.5, actual: 1.4 
(release 0)                           slurmstepd: error: mpi/pmi2: no value for key  in req                                                                                                                  
slurmstepd: error: mpi/pmi2: no value for key  in req                                                                                                                  
slurmstepd: error: mpi/pmi2: no value for key  in req                                                                                                                  
slurmstepd: error: mpi/pmi2: no value for key  in req                                                                                                                  
slurmstepd: error: mpi/pmi2: no value for key  in req                                                                                                                  
# OSU MPI_Get latency Test v5.6.1                                                                                                                                      
# Window creation: MPI_Win_allocate                                                                                                                                    
# Synchronization: MPI_Win_flush                                                                                                                                       
# Size          Latency (us)                                                                                                                                           
1                       2.13                                                                                                                                           
....

Since UCX is a vendored dependency of MPICH, why is MPICH complaining about an incompatible UCX version ?

Thank You,
Sajid Ali | PhD Candidate
Applied Physics
Northwestern University
s-sajid-ali.github.io