Follow up: I rebuilt MPICH2 3.0.1 without any link to SLURM and repeated the experiment with the same result:

Fatal error in PMPI_Reduce: A process has failed, error stack:

PMPI_Reduce(1217)...............: MPI_Reduce(sbuf=0x7fffee6f1ca0, rbuf=0x7fffee6f1ca8, count=1, MPI_DOUBLE, MPI_SUM, root=0, MPI_COMM_WORLD) failed

MPIR_Reduce_impl(1029)..........:

MPIR_Reduce_intra(779)..........:

MPIR_Reduce_impl(1029)..........:

MPIR_Reduce_intra(835)..........:

MPIR_Reduce_binomial(144).......:

MPIDI_CH3U_Recvq_FDU_or_AEP(612): Communication error with rank 16

MPIR_Reduce_intra(799)..........:

MPIR_Reduce_impl(1029)..........:

MPIR_Reduce_intra(835)..........:

MPIR_Reduce_binomial(206).......: Failure during collective

===================================================================================

=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES

=   EXIT CODE: 1

=   CLEANING UP REMAINING PROCESSES

=   YOU CAN IGNORE THE BELOW CLEANUP MESSAGES

===================================================================================

[proxy:0:1@n2] HYD_pmcd_pmip_control_cmd_cb (./pm/pmiserv/pmip_cb.c:886): assert (!closed) failed

[proxy:0:1@n2] HYDT_dmxu_poll_wait_for_event (./tools/demux/demux_poll.c:77): callback returned error status

[proxy:0:1@n2] main (./pm/pmiserv/pmip.c:206): demux engine error waiting for event

[mpiexec@n1] HYDT_bscu_wait_for_completion (./tools/bootstrap/utils/bscu_wait.c:76): one of the processes terminated badly; aborting

[mpiexec@n1] HYDT_bsci_wait_for_completion (./tools/bootstrap/src/bsci_wait.c:23): launcher returned error waiting for completion

[mpiexec@n1] HYD_pmci_wait_for_completion (./pm/pmiserv/pmiserv_pmci.c:217): launcher returned error waiting for completion

[mpiexec@n1] main (./ui/mpich/mpiexec.c:330): process manager error waiting for completion

            This is now a vanilla install trying to run the cxxcpi example code. Any help is appreciated.

            Thanks,

            ~Mike C.

From: Michael Colonno [mailto:mcolonno@stanford.edu]
Sent: Friday, January 11, 2013 1:32 PM
To: 'discuss@mpich.org'
Subject: Fatal error in PMPI_Reduce

            Hi All ~

            I've compiled MPICH2 3.0 with the Intel compiler (v. 13) on a CentOS 6.3 x64 system using SLURM as the process manager. My configure was simply:

./configure --with-pmi=slurm --with-pm=no --prefix=/usr/local/apps/MPICH2

No errors during build or install. When I compile and run the example program cxxcpi I get (truncated):

$ srun -n32 /usr/local/apps/cxxcpi

Fatal error in PMPI_Reduce: A process has failed, error stack:

PMPI_Reduce(1217)...............: MPI_Reduce(sbuf=0x7fff4ad18120, rbuf=0x7fff4ad18128, count=1, MPI_DOUBLE, MPI_SUM, root=0, MPI_COMM_WORLD) failed

MPIR_Reduce_impl(1029)..........:

MPIR_Reduce_intra(779)..........:

MPIR_Reduce_impl(1029)..........:

MPIR_Reduce_intra(835)..........:

MPIR_Reduce_binomial(144).......:

MPIDI_CH3U_Recvq_FDU_or_AEP(612): Communication error with rank 16

MPIR_Reduce_intra(799)..........:

MPIR_Reduce_impl(1029)..........:

MPIR_Reduce_intra(835)..........:

MPIR_Reduce_binomial(206).......: Failure during collective

srun: error: task 0: Exited with exit code 1

            This error is experienced with many of my MPI programs. A different application yields:

PMPI_Bcast(1525)......: MPI_Bcast(buf=0x7fff545be5fc, count=1, MPI_INT, root=0, MPI_COMM_WORLD) failed

MPIR_Bcast_impl(1369).:

MPIR_Bcast_intra(1160):

MPIR_SMP_Bcast(1077)..: Failure during collective

            Can anyone point me in the right direction?

            Thanks,

            ~Mike C.