Try rebuilding your MPICH omitting the "--with-pmi=slurm". Instead, you can use hydra's built-in support for slurm. Some information is available here: http://wiki.mpich.org/mpich/index.php/Using_the_Hydra_Process_Manager I can't remember exactly what will be auto detected and what won't (Pavan, can you help me out here?). You might need to set these environment variables: HYDRA_LAUNCHER=slurm HYDRA_RMK=slurm Then use "mpiexec" instead of "srun". -Dave On Feb 4, 2013, at 9:51 AM CST, Biddiscombe, John A. wrote:
Hello, I’m working on someone else’s cluster and the mpi is not well setup (i.e. not working).
I compiled my own mpich2 using ./configure --with-pmi=slurm --with-pm=no --prefix=/home/biddisco/apps/mpich2-1.5 --enable-shared --disable-static And a simple hello world executed using srun, works as expected and I get the ranks from all nodes responding correctly.
When I used mpiexec previously, I ran an MPMD command like this
mpiexec -n 4 -env DISPLAY :0.0 /home/biddisco/build/pv-38/bin/pvserver -rc -ch=128.178.187.20 -sp=11111 : -n 4 -env DISPLAY :0.1 /home/biddisco/build/pv-38/bin/pvserver -rc -ch=128.178.187.20 -sp=11111 : -n 4 -env DISPLAY :0.2 /home/biddisco/build/pv-38/bin/pvserver -rc -ch=128.178.187.20 -sp=11111
I need each of the 3 sets of mpi ranks to have a different DISPLAY environment variable to correctly utilise the GPUs.
Unfortunately, the mpiexec on the machine does not work with slurm and I’d like to use srun –n XXX –env XXX /program : –n XXX –env YYY /program ... etc etc
but srun doesn’t have an option to pass command line environment vars to the mpi processes. I just can’t find a way of launching an MPMD job using srun with different env vars.
Is there another way?
Thanks
JB
-- John Biddiscombe, email:biddisco @.at.@ cscs.ch http://www.cscs.ch/ CSCS, Swiss National Supercomputing Centre | Tel: +41 (91) 610.82.07 Via Trevano 131, 6900 Lugano, Switzerland | Fax: +41 (91) 610.82.82
_______________________________________________ discuss mailing list [email protected] To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss