Hi All

We have included the MPIEXEC_PORT_RANGE and MPIR_CVAR_CH3_PORT_RANGE fields and 

have adjusted our iptables accordingly and in line with the “FAQ” guidance.

Our passwordless SSH works fine between the machines.

But all of this gives us momentary success with the cpi and fpi MPICH test programs.  But they crash with the firewall up. (but of course run happily with the firewall down).

An example of the basic output is below (node short sends one process to “this.machine” and one to remote “that.machine”

[this.machine]% mpiexec -n 2 -f nodesshort cpi.exe

Process 0 of 2 is on this.machine

Process 1 of 2 is on that.machine 

Fatal error in PMPI_Reduce: A process has failed, error stack:

PMPI_Reduce(1217)...............: MPI_Reduce(sbuf=0x7fff466a94d0, rbuf=0x7fff466a94d8, count=1, MPI_DOUBLE, MPI_SUM, root=0, MPI_COMM_WORLD) failed

MPIR_Reduce_impl(1029)..........: 

MPIR_Reduce_intra(835)..........: 

MPIR_Reduce_binomial(144).......: 

MPIDI_CH3U_Recvq_FDU_or_AEP(667): Communication error with rank 1

===================================================================================

=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES

=   EXIT CODE: 1

=   CLEANING UP REMAINING PROCESSES

=   YOU CAN IGNORE THE BELOW CLEANUP MESSAGES

===================================================================================

[proxy:0:1@that.machine] HYD_pmcd_pmip_control_cmd_cb (./pm/pmiserv/pmip_cb.c:886): assert (!closed) failed

[proxy:0:1@that.machine] HYDT_dmxu_poll_wait_for_event (./tools/demux/demux_poll.c:77): callback returned error status

[proxy:0:1@that.machine] main (./pm/pmiserv/pmip.c:206): demux engine error waiting for event

[mpiexec@this.machine] HYDT_bscu_wait_for_completion (./tools/bootstrap/utils/bscu_wait.c:76): one of the processes terminated badly; aborting

[mpiexec@this.machine] HYDT_bsci_wait_for_completion (./tools/bootstrap/src/bsci_wait.c:23): launcher returned error waiting for completion

[mpiexec@this.machine] HYD_pmci_wait_for_completion (./pm/pmiserv/pmiserv_pmci.c:217): launcher returned error waiting for completion

[mpiexec@this.machine] main (./ui/mpich/mpiexec.c:331): process manager error waiting for completion

But later we get output like this:

[mpiexec@this.machine] PMI response to fd 6 pid 4: cmd=keyval_cache P0-businesscard=description#{this.machine’s.ip.address}$port#54105$ifname#{this.machine’s.ip.address}$ P1-businesscard=description#{that.machine’s.ip.address}$port#47302$ifname#{that.machine’s.ip.address}$ 

Does this mean that we have missed a firewall setting either in the environment variables or in the ip tables themselves?

Ideas?

Thanks Much

Bill