We are running a small cluster using slurm 20.11.2.

Our application is built with mpich 3.3.2.

If we submit a 2 batch jobs, each on it’s own single node, one

using srun and the other using mpiexec to run the application,

the mpiexec takes much longer. This is an 8 processor job

on a 32 core single node.

Comparison times, for the same input file and the same

number of processes (8), are

srun 0.006366 s
mpiexec 22.21 s

Something is clearly wrong.

From the FAQ for mpich, it says the default build of MPICH

SLURM is an external process manager that uses MPICH's PMI interface as well.

If I look at the process tree (ps axjf) when the mpiexec job is running:

      1 1427683 1427682 1427682 ?             -1 Sl       0   0:00 slurmstepd: [5435.batch]

1427683 1427687 1427687 1427682 ?             -1 S        0   0:00  \_ /bin/bash /var/spool/slurmd/job05435/slurm_script

1427687 1427702 1427687 1427682 ?             -1 S        0   0:00      \_ mpiexec -n 8 -machinefile ./machinefile myapp -i electron.in

1427702 1427703 1427703 1427703 ?             -1 Ssl      0   0:00          \_ /bin/srun --nodelist ne04 -N 1 -n 1 --input none /home/common/myapp/myapp-11.0-2021-

1427703 1427704 1427703 1427703 ?             -1 S        0   0:00              \_ /bin/srun --nodelist ne04 -N 1 -n 1 --input none /home/common/myapp/myapp-11.0-2

      1 1427712 1427711 1427711 ?             -1 Sl       0   0:00 slurmstepd: [5435.0]

1427712 1427718 1427718 1427711 ?             -1 S        0   0:00  \_ /home/common/myapp/myapp-11.0-2021-03-30-05.02/Contents/engine/bin/hydra_pmi_proxy --control

1427718 1427719 1427719 1427719 ?             -1 Rsl      0   0:10      \_ myapp -i electron.in

1427718 1427720 1427720 1427720 ?             -1 Rsl      0   0:11      \_ myapp -i electron.in

1427718 1427721 1427721 1427721 ?             -1 Rsl      0   0:06      \_ myapp -i electron.in

1427718 1427722 1427722 1427722 ?             -1 Rsl      0   0:06      \_ myapp -i electron.in

1427718 1427723 1427723 1427723 ?             -1 Rsl      0   0:06      \_ myapp -i electron.in

1427718 1427724 1427724 1427724 ?             -1 Rsl      0   0:10      \_ myapp -i electron.in

1427718 1427725 1427725 1427725 ?             -1 Rsl      0   0:08      \_ myapp -i electron.in

1427718 1427726 1427726 1427726 ?             -1 Rsl      0   0:11      \_ myapp -i electron.in

I see that mpiexec is calling /bin/srun with -N 1 -n 1. I don’t know if this matters, because

there are 8 myapp processes running under slurmstepd.

The problem is that mpiexec is taking so much more time.

Process Manager: pmi

Is pmi2 different from pmi?

The issue could also be with slurm. In /etc/slurm/slurm.conf, we define MpiDefault to be pmi2:

If anyone can point me where to look to solve this problem, I would be very thankful.

Note that the default build of MPICH will work fine in SLURM environments. No extra steps are needed.