Hi, System: - AMD cluster (AMD Opteron(tm) Processor 6344) with InfiniPath_QLE7240 - CentOS Linux release 7.7.1908 (Core) Scheduler: - slurm MPICH version: 3.3.2 MPICH configuration: ./configure --prefix=/home/yaser/bin/mpich/3.3.2/intel CC=icc CXX=icpc FC=ifort --with-hwloc=/opt/HWLOC/2.2.0 --with-ucx=/opt/UCX/1.8.0 --with-knem=/opt/KNEM/1.1.3 --with-device=ch4:ucx --enable-mpi-cxx --enable-mpi1-compatibility --enable-threads=multiple --with-pmi --with-pm I am testing using the OSU microbenchmark ( http://mvapich.cse.ohio-state.edu/benchmarks) running the job on `srun` would succeed with no problem. ``` srun --mpi=pmi2 ./osu_bw ``` When I use the `mpiexec`, or `mpirun` it fails with: ``` [proxy:0:[email protected]] HYDU_sock_connect (utils/sock/sock.c:145): unable to connect from "pd-compute-3-40.local" to "pd-compute-1-6.local" (Connection refused) [proxy:0:[email protected]] main (pm/pmiserv/pmip.c:183): unable to connect to server pd-compute-1-6.local at port 42216 (check for firewalls!) srun: error: pd-compute-3-40: task 1: Exited with exit code 5 ``` The firewall is off, so it is not the reason. ```
systemctl status firewalld ● firewalld.service Loaded: masked (/dev/null; bad) Active: inactive (dead)
I could not find any hint on MPICH FAQ nor anything useful anywhere else.
Would you help me to resolve this issue?
Many thanks.