I wonder if your MPICH build was able to detect and build in Slurm support. Can you send the src/pm/hydra/config.log file from your MPICH build directory? Ken On 05/23/2015 08:41 PM, Bill Broadley wrote:
I have an ubuntu 14.04 cluster where slurm-14.11 and OpenMPI 1.8.5 work well. I'm trying to get mpich-3.1.4 working. An OpenMPI + slurm example: $ mpicc.openmpi relay.c -o relay $ mpicc.openmpi hello.c -o relay
A run on a single node: $ salloc -N 1 -n 2 mpirun ./relay 1 salloc: Granted job allocation 682 c7-13 c7-13 size= 1, 16384 hops, 2 nodes in 0.01 sec ( 0.37 us/hop) 10468 KB/sec
A run on 2 nodes: $ salloc -N 2 -n 2 mpirun ./relay 1 salloc: Granted job allocation 683 c7-13 c7-14 size= 1, 16384 hops, 2 nodes in 0.10 sec ( 5.95 us/hop) 657 KB/sec
I compiled mpich-3.14, ./configure setting only the --prefix. I just used the ubuntu default gcc-4.8.2. From config.log: $ ./configure --prefix=/share/apps/mpich-3.1.4/gcc
Now to test: $ export PATH=/share/apps/mpich-3.1.4/gcc/bin:$PATH $ export LD_LIBRARY_PATH=/share/apps/mpich-3.1.4/gcc/lib:$LD_LIBRARY_PATH $ which mpicc /share/apps/mpich-3.1.4/gcc/bin/mpicc $ which mpiexec /share/apps/mpich-3.1.4/gcc/bin/mpiexec $ mpicc relay.c -o relay $ mpicc hello.c -o hello $ salloc -N 1 -n 2 mpiexec ./hello salloc: Granted job allocation 688 Hello world from process 0 of 2 Hello world from process 1 of 2 $ salloc -N 2 -n 2 mpiexec ./hello salloc: Granted job allocation 689 Hello world from process 0 of 2 Hello world from process 1 of 2
Great, the basics are working. But no actual communications are happening.
Communications within a single node seems to work: $ salloc -N 1 -n 2 mpiexec ./relay 1 salloc: Granted job allocation 690 c7-13 c7-13 size= 1, 16384 hops, 2 nodes in 0.01 sec ( 0.55 us/hop) 7074 KB/sec
But not between more than one node: bill@hpc1:~/src/relay$ salloc -N 2 -n 2 mpiexec ./relay 1 salloc: Granted job allocation 691 c7-13 Fatal error in MPI_Recv: Unknown error class, error stack: MPI_Recv(187)...................: MPI_Recv(buf=0x7ffdc56aa630, count=129, MPI_CHAR, src=1, tag=1, MPI_COMM_WORLD, status=0x7ffdc56aa610) failed MPIDI_CH3U_Recvq_FDU_or_AEP(636): Communication error with rank 1
=================================================================================== = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = PID 9734 RUNNING AT c7-13 = EXIT CODE: 1 = CLEANING UP REMAINING PROCESSES = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES =================================================================================== [proxy:0:1@c7-14] HYD_pmcd_pmip_control_cmd_cb (pm/pmiserv/pmip_cb.c:885): assert (!closed) failed [proxy:0:1@c7-14] HYDT_dmxu_poll_wait_for_event (tools/demux/demux_poll.c:76): callback returned error status [proxy:0:1@c7-14] main (pm/pmiserv/pmip.c:206): demux engine error waiting for event srun: error: c7-14: task 1: Exited with exit code 7 [mpiexec@hpc1] HYDT_bscu_wait_for_completion (tools/bootstrap/utils/bscu_wait.c:76): one of the processes terminated badly; aborting [mpiexec@hpc1] HYDT_bsci_wait_for_completion (tools/bootstrap/src/bsci_wait.c:23): launcher returned error waiting for completion [mpiexec@hpc1] HYD_pmci_wait_for_completion (pm/pmiserv/pmiserv_pmci.c:218): launcher returned error waiting for completion [mpiexec@hpc1] main (ui/mpich/mpiexec.c:344): process manager error waiting for completion salloc: Relinquishing job allocation 691
DNS, and ssh seem to be fine: $ ssh c7-13 "hostname; ssh c7-14 'hostname'" c7-13 c7-14
Any ideas? _______________________________________________ discuss mailing list [email protected] To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss
_______________________________________________ discuss mailing list [email protected] To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss