Environment variable forwarding using Hydra and "-launcher ssh"
We're in the process of moving from mpich-3.x to mpich-4.1.2. We've run into some odd behaviour on SLURM related to environment variable forwarding by mpiexec. It looks like mpiexec now propagates only SLURM_* environment variables, instead of filtering them out (or intending to). Consider something like this: $ mpiexec -launcher slurm printenv HOME SLURM_JOBID Using mpich-3.x, the HOME variable gets forward. Using mpich-4.1.2, it does not. I believe that mpich-3.x intends to filter out SLURM_JOBID, but the value still seems to be present, maybe srun forwards that. It's the fact that HOME doesn't get through using mpich-4.1.2 that is causing us problems. Running mpich- Here's what I think is the relevant change for SLURM: https://urldefense.us/v3/__https://github.com/pmodels/mpich/commit/95ba4ddc7... . Am I reading that correctly? The doc here https://urldefense.us/v3/__https://github.com/pmodels/mpich/blob/main/doc/wi... states that SLURM_ things should be filtered out, but that doesn't appear to be happening? For reference, here's what mpich-4.1.2 "mpiexec -verbose -launcher slurm" prints: mpiexec options: ---------------- Base path: /path/to/mpich-4.1.2 Launcher: slurm Debug level: 1 Enable X: -1 Global environment: ------------------- SLURM_JOBID=102437 SLURM_JOB_USER=eellis SLURM_JOB_QOS=normal SLURM_JOB_NUM_NODES=2 SLURM_TASKS_PER_NODE=1(x2) SLURM_TOPOLOGY_ADDR_PATTERN=node ... many more SLURM_* And here's what mpich-3.x prints: mpiexec options: ---------------- Base path: /path/to/mpich-3.x Launcher: slurm Debug level: 1 Enable X: -1 Global environment: ------------------- ALTERNATE_EDITOR=emacs MAIL=/var/mail/eellis USER=eellis SLURM_JOB_USER=eellis l=/local/eellis XDG_SESSION_TYPE=unspecified SLURM_JOB_QOS=normal ... no SLURM_JOBID Cheers, Edric.
participants (1)
-
Edric Ellis