Hi Pavan ~
Thanks for the reply. I did verify connectivity prior to trying the program. In this test case I have two systems, each with 16 cores (cv-hpcn1 and cv-hpcn2). ssh between the two systems appears to be working as far as I can tell:
cv-hpcn1$ ssh cv-hpcn1 date
Sat Jan 12 10:20:01 PST 2013
cv-hpcn1$ ssh cv-hpcn2 date
Sat Jan 12 10:19:23 PST 2013
cv-hpcn1$ ssh cv-hpcn2
cv-hpcn2$ ssh cv-hpcn1 date
Sat Jan 12 10:20:43 PST 2013
Here's the full test run of cxxcpi:
cv-hpcn1$ /usr/local/apps/MPICH2/standalone/bin/mpirun -n 32 -f /usr/local/apps/hosts /usr/local/apps/cxxcpi
Process 17 of 32 is on cv-hpcn2
Process 19 of 32 is on cv-hpcn2
Process 20 of 32 is on cv-hpcn2
Process 21 of 32 is on cv-hpcn2
Process 22 of 32 is on cv-hpcn2
Process 23 of 32 is on cv-hpcn2
Process 24 of 32 is on cv-hpcn2
Process 26 of 32 is on cv-hpcn2
Process 28 of 32 is on cv-hpcn2
Process 29 of 32 is on cv-hpcn2
Process 30 of 32 is on cv-hpcn2
Process 31 of 32 is on cv-hpcn2
Process 16 of 32 is on cv-hpcn2
Process 18 of 32 is on cv-hpcn2
Process 25 of 32 is on cv-hpcn2
Process 27 of 32 is on cv-hpcn2
Process 1 of 32 is on cv-hpcn1
Process 2 of 32 is on cv-hpcn1
Process 10 of 32 is on cv-hpcn1
Process 12 of 32 is on cv-hpcn1
Process 14 of 32 is on cv-hpcn1
Process 15 of 32 is on cv-hpcn1
Process 0 of 32 is on cv-hpcn1
Process 5 of 32 is on cv-hpcn1
Process 8 of 32 is on cv-hpcn1
Process 13 of 32 is on cv-hpcn1
Process 11 of 32 is on cv-hpcn1
Process 3 of 32 is on cv-hpcn1
Process 9 of 32 is on cv-hpcn1
Process 7 of 32 is on cv-hpcn1
Process 4 of 32 is on cv-hpcn1
Process 6 of 32 is on cv-hpcn1
Fatal error in PMPI_Reduce: A process has failed, error stack:
PMPI_Reduce(1217)...............: MPI_Reduce(sbuf=0x7ffff0ab2320, rbuf=0x7ffff0ab2328, count=1, MPI_DOUBLE, MPI_SUM, root=0, MPI_COMM_WORLD) failed
MPIR_Reduce_impl(1029)..........:
MPIR_Reduce_intra(779)..........:
MPIR_Reduce_impl(1029)..........:
MPIR_Reduce_intra(835)..........:
MPIR_Reduce_binomial(144).......:
MPIDI_CH3U_Recvq_FDU_or_AEP(612): Communication error with rank 16
MPIR_Reduce_intra(799)..........:
MPIR_Reduce_impl(1029)..........:
MPIR_Reduce_intra(835)..........:
MPIR_Reduce_binomial(206).......: Failure during collective
===================================================================================
= BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
= EXIT CODE: 1
= CLEANING UP REMAINING PROCESSES
= YOU CAN IGNORE THE BELOW CLEANUP MESSAGES
===================================================================================
[proxy:0:1@cv-hpcn2] HYD_pmcd_pmip_control_cmd_cb (./pm/pmiserv/pmip_cb.c:886): assert (!closed) failed
[proxy:0:1@cv-hpcn2] HYDT_dmxu_poll_wait_for_event (./tools/demux/demux_poll.c:77): callback returned error status
[proxy:0:1@cv-hpcn2] main (./pm/pmiserv/pmip.c:206): demux engine error waiting for event
[mpiexec@cv-hpcn2] HYDT_bscu_wait_for_completion (./tools/bootstrap/utils/bscu_wait.c:76): one of the processes terminated badly; aborting
[mpiexec@cv-hpcn2] HYDT_bsci_wait_for_completion (./tools/bootstrap/src/bsci_wait.c:23): launcher returned error waiting for completion
[mpiexec@cv-hpcn2] HYD_pmci_wait_for_completion (./pm/pmiserv/pmiserv_pmci.c:217): launcher returned error waiting for completion
[mpiexec@cv-hpcn2] main (./ui/mpich/mpiexec.c:330): process manager error waiting for completion
Thanks for any advice. I’ve built many different versions of MPICH many times and this one has me scratching me head. I’m rebuilding everything identically on a pair of little VMs now to see if I can repeat the behavior outside of these systems.
~Mike C.
-----Original Message-----
From: Pavan Balaji [mailto:balaji@mcs.anl.gov]
Sent: Friday, January 11, 2013 9:20 PM
To: Michael Colonno
Cc: discuss@mpich.org
Subject: Re: [mpich-discuss] Fatal error in PMPI_Reduce
On 01/11/2013 09:18 PM US Central Time, Michael Colonno wrote:
> The first output below is from cxxcpi; I can also run cpi is it's helpful.
Ah, sorry, I didn't realize you were in fact running one of the test programs distributed in MPICH.
Based on the error, some of the nodes are not able to connect to each other based on the hostnames they are publishing. Did you try the solutions listed on the FAQ:
-- Pavan
--
Pavan Balaji