Follow up: I rebuilt MPICH2 3.0.1 without any link to SLURM and repeated the experiment with the same result: Fatal error in PMPI_Reduce: A process has failed, error stack: PMPI_Reduce(1217)...............: MPI_Reduce(sbuf=0x7fffee6f1ca0, rbuf=0x7fffee6f1ca8, count=1, MPI_DOUBLE, MPI_SUM, root=0, MPI_COMM_WORLD) failed MPIR_Reduce_impl(1029)..........: MPIR_Reduce_intra(779)..........: MPIR_Reduce_impl(1029)..........: MPIR_Reduce_intra(835)..........: MPIR_Reduce_binomial(144).......: MPIDI_CH3U_Recvq_FDU_or_AEP(612): Communication error with rank 16 MPIR_Reduce_intra(799)..........: MPIR_Reduce_impl(1029)..........: MPIR_Reduce_intra(835)..........: MPIR_Reduce_binomial(206).......: Failure during collective ============================================================================ ======= = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = EXIT CODE: 1 = CLEANING UP REMAINING PROCESSES = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES ============================================================================ ======= [proxy:0:1@n2] HYD_pmcd_pmip_control_cmd_cb (./pm/pmiserv/pmip_cb.c:886): assert (!closed) failed [proxy:0:1@n2] HYDT_dmxu_poll_wait_for_event (./tools/demux/demux_poll.c:77): callback returned error status [proxy:0:1@n2] main (./pm/pmiserv/pmip.c:206): demux engine error waiting for event [mpiexec@n1] HYDT_bscu_wait_for_completion (./tools/bootstrap/utils/bscu_wait.c:76): one of the processes terminated badly; aborting [mpiexec@n1] HYDT_bsci_wait_for_completion (./tools/bootstrap/src/bsci_wait.c:23): launcher returned error waiting for completion [mpiexec@n1] HYD_pmci_wait_for_completion (./pm/pmiserv/pmiserv_pmci.c:217): launcher returned error waiting for completion [mpiexec@n1] main (./ui/mpich/mpiexec.c:330): process manager error waiting for completion This is now a vanilla install trying to run the cxxcpi example code. Any help is appreciated. Thanks, ~Mike C. From: Michael Colonno [mailto:[email protected]] Sent: Friday, January 11, 2013 1:32 PM To: '[email protected]' Subject: Fatal error in PMPI_Reduce Hi All ~ I've compiled MPICH2 3.0 with the Intel compiler (v. 13) on a CentOS 6.3 x64 system using SLURM as the process manager. My configure was simply: ./configure --with-pmi=slurm --with-pm=no --prefix=/usr/local/apps/MPICH2 No errors during build or install. When I compile and run the example program cxxcpi I get (truncated): $ srun -n32 /usr/local/apps/cxxcpi Fatal error in PMPI_Reduce: A process has failed, error stack: PMPI_Reduce(1217)...............: MPI_Reduce(sbuf=0x7fff4ad18120, rbuf=0x7fff4ad18128, count=1, MPI_DOUBLE, MPI_SUM, root=0, MPI_COMM_WORLD) failed MPIR_Reduce_impl(1029)..........: MPIR_Reduce_intra(779)..........: MPIR_Reduce_impl(1029)..........: MPIR_Reduce_intra(835)..........: MPIR_Reduce_binomial(144).......: MPIDI_CH3U_Recvq_FDU_or_AEP(612): Communication error with rank 16 MPIR_Reduce_intra(799)..........: MPIR_Reduce_impl(1029)..........: MPIR_Reduce_intra(835)..........: MPIR_Reduce_binomial(206).......: Failure during collective srun: error: task 0: Exited with exit code 1 This error is experienced with many of my MPI programs. A different application yields: PMPI_Bcast(1525)......: MPI_Bcast(buf=0x7fff545be5fc, count=1, MPI_INT, root=0, MPI_COMM_WORLD) failed MPIR_Bcast_impl(1369).: MPIR_Bcast_intra(1160): MPIR_SMP_Bcast(1077)..: Failure during collective Can anyone point me in the right direction? Thanks, ~Mike C.