Hi All We're running MPICH on a couple machines with a brand new UNIX distro (SL 6.5) and that are on a vulnerable network and rather than leave the firewalls dropped we would like to run it through the firewall. We have included the MPIEXEC_PORT_RANGE and MPIR_CVAR_CH3_PORT_RANGE fields and have adjusted our iptables accordingly and in line with the "FAQ" guidance. Our passwordless SSH works fine between the machines. But all of this gives us momentary success with the cpi and fpi MPICH test programs. But they crash with the firewall up. (but of course run happily with the firewall down). An example of the basic output is below (node short sends one process to "this.machine" and one to remote "that.machine" [this.machine]% mpiexec -n 2 -f nodesshort cpi.exe Process 0 of 2 is on this.machine Process 1 of 2 is on that.machine Fatal error in PMPI_Reduce: A process has failed, error stack: PMPI_Reduce(1217)...............: MPI_Reduce(sbuf=0x7fff466a94d0, rbuf=0x7fff466a94d8, count=1, MPI_DOUBLE, MPI_SUM, root=0, MPI_COMM_WORLD) failed MPIR_Reduce_impl(1029)..........: MPIR_Reduce_intra(835)..........: MPIR_Reduce_binomial(144).......: MPIDI_CH3U_Recvq_FDU_or_AEP(667): Communication error with rank 1 =================================================================================== = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = EXIT CODE: 1 = CLEANING UP REMAINING PROCESSES = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES =================================================================================== [proxy:0:[email protected]] HYD_pmcd_pmip_control_cmd_cb (./pm/pmiserv/pmip_cb.c:886): assert (!closed) failed [proxy:0:[email protected]] HYDT_dmxu_poll_wait_for_event (./tools/demux/demux_poll.c:77): callback returned error status [proxy:0:[email protected]] main (./pm/pmiserv/pmip.c:206): demux engine error waiting for event [[email protected]] HYDT_bscu_wait_for_completion (./tools/bootstrap/utils/bscu_wait.c:76): one of the processes terminated badly; aborting [[email protected]] HYDT_bsci_wait_for_completion (./tools/bootstrap/src/bsci_wait.c:23): launcher returned error waiting for completion [[email protected]] HYD_pmci_wait_for_completion (./pm/pmiserv/pmiserv_pmci.c:217): launcher returned error waiting for completion [[email protected]] main (./ui/mpich/mpiexec.c:331): process manager error waiting for completion In debug mode it affirms that it is at least *starting with the first available port as listed in MPIEXEC_PORT_RANGE But later we get output like this: [[email protected]] PMI response to fd 6 pid 4: cmd=keyval_cache P0-businesscard=description#{this.machine's.ip.address}$port#54105$ifname#{this.machine's.ip.address}$ P1-businesscard=description#{that.machine's.ip.address}$port#47302$ifname#{that.machine's.ip.address}$ Does this mean that we have missed a firewall setting either in the environment variables or in the ip tables themselves? Ideas? Thanks Much Bill