Ok, this doesn't seem to have anything to do with slurm. Can you try running simple programs from the mpich install to make sure it's correctly installed? For example, can you try examples/cpi in mpich? -- Pavan On 01/11/2013 05:47 PM US Central Time, Michael Colonno wrote:

Follow up: I rebuilt MPICH2 3.0.1 without any link to SLURM and repeated the experiment with the same result:

Fatal error in PMPI_Reduce: A process has failed, error stack:

PMPI_Reduce(1217)...............: MPI_Reduce(sbuf=0x7fffee6f1ca0, rbuf=0x7fffee6f1ca8, count=1, MPI_DOUBLE, MPI_SUM, root=0, MPI_COMM_WORLD) failed

MPIR_Reduce_impl(1029)..........:

MPIR_Reduce_intra(779)..........:

MPIR_Reduce_impl(1029)..........:

MPIR_Reduce_intra(835)..........:

MPIR_Reduce_binomial(144).......:

MPIDI_CH3U_Recvq_FDU_or_AEP(612): Communication error with rank 16

MPIR_Reduce_intra(799)..........:

MPIR_Reduce_impl(1029)..........:

MPIR_Reduce_intra(835)..........:

MPIR_Reduce_binomial(206).......: Failure during collective

===================================================================================

= BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES

= EXIT CODE: 1

= CLEANING UP REMAINING PROCESSES

= YOU CAN IGNORE THE BELOW CLEANUP MESSAGES

===================================================================================

[proxy:0:1@n2] HYD_pmcd_pmip_control_cmd_cb (./pm/pmiserv/pmip_cb.c:886): assert (!closed) failed

[proxy:0:1@n2] HYDT_dmxu_poll_wait_for_event (./tools/demux/demux_poll.c:77): callback returned error status

[proxy:0:1@n2] main (./pm/pmiserv/pmip.c:206): demux engine error waiting for event

[mpiexec@n1] HYDT_bscu_wait_for_completion (./tools/bootstrap/utils/bscu_wait.c:76): one of the processes terminated badly; aborting

[mpiexec@n1] HYDT_bsci_wait_for_completion (./tools/bootstrap/src/bsci_wait.c:23): launcher returned error waiting for completion

[mpiexec@n1] HYD_pmci_wait_for_completion (./pm/pmiserv/pmiserv_pmci.c:217): launcher returned error waiting for completion

[mpiexec@n1] main (./ui/mpich/mpiexec.c:330): process manager error waiting for completion

This is now a vanilla install trying to run the cxxcpi example code. Any help is appreciated.

Thanks,

~Mike C.

*From:*Michael Colonno [mailto:[email protected]] *Sent:* Friday, January 11, 2013 1:32 PM *To:* '[email protected]' *Subject:* Fatal error in PMPI_Reduce

Hi All ~

I've compiled MPICH2 3.0 with the Intel compiler (v. 13) on a CentOS 6.3 x64 system using SLURM as the process manager. My configure was simply:

./configure --with-pmi=slurm --with-pm=no --prefix=/usr/local/apps/MPICH2

No errors during build or install. When I compile and run the example program cxxcpi I get (truncated):

$ srun -n32 /usr/local/apps/cxxcpi

Fatal error in PMPI_Reduce: A process has failed, error stack:

PMPI_Reduce(1217)...............: MPI_Reduce(sbuf=0x7fff4ad18120, rbuf=0x7fff4ad18128, count=1, MPI_DOUBLE, MPI_SUM, root=0, MPI_COMM_WORLD) failed

MPIR_Reduce_impl(1029)..........:

MPIR_Reduce_intra(779)..........:

MPIR_Reduce_impl(1029)..........:

MPIR_Reduce_intra(835)..........:

MPIR_Reduce_binomial(144).......:

MPIDI_CH3U_Recvq_FDU_or_AEP(612): Communication error with rank 16

MPIR_Reduce_intra(799)..........:

MPIR_Reduce_impl(1029)..........:

MPIR_Reduce_intra(835)..........:

MPIR_Reduce_binomial(206).......: Failure during collective

srun: error: task 0: Exited with exit code 1

This error is experienced with many of my MPI programs. A different application yields:

PMPI_Bcast(1525)......: MPI_Bcast(buf=0x7fff545be5fc, count=1, MPI_INT, root=0, MPI_COMM_WORLD) failed

MPIR_Bcast_impl(1369).:

MPIR_Bcast_intra(1160):

MPIR_SMP_Bcast(1077)..: Failure during collective

Can anyone point me in the right direction?

Thanks,

~Mike C.

_______________________________________________ discuss mailing list [email protected] To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

-- Pavan Balaji http://www.mcs.anl.gov/~balaji