I disabled the whole firewall in those machines but, still get the same problem. connection refuse. I run the program in another set of totally different machines that we have, but still same problem. Any other thought where can be the problem? Thanks. Amin Hassani, CIS department at UAB, Birmingham, AL, USA. On Wed, Nov 26, 2014 at 9:25 AM, Kenneth Raffenetti <[email protected]> wrote:
The connection refused makes me think a firewall is getting in the way. Is TCP communication limited to specific ports on the cluster? If so, you can use this envvar to enforce a range of ports in MPICH.
MPIR_CVAR_CH3_PORT_RANGE Description: The MPIR_CVAR_CH3_PORT_RANGE environment variable allows you to specify the range of TCP ports to be used by the process manager and the MPICH library. The format of this variable is <low>:<high>. To specify any available port, use 0:0. Default: {0,0}
On 11/25/2014 11:50 PM, Amin Hassani wrote:
Tried with the new configure too. same problem :(
$ mpirun -hostfile hosts-hydra -np 2 test_dup Fatal error in MPI_Send: Unknown error class, error stack: MPI_Send(174)..............: MPI_Send(buf=0x7fffd90c76c8, count=1, MPI_INT, dest=1, tag=0, MPI_COMM_WORLD) failed MPID_nem_tcp_connpoll(1832): Communication error with rank 1: Connection refused
============================================================ ======================= = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = PID 5459 RUNNING AT oakmnt-0-a = EXIT CODE: 1 = CLEANING UP REMAINING PROCESSES = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES ============================================================ ======================= [proxy:0:1@oakmnt-0-b] HYD_pmcd_pmip_control_cmd_cb (../../../../src/pm/hydra/pm/pmiserv/pmip_cb.c:885): assert (!closed) failed [proxy:0:1@oakmnt-0-b] HYDT_dmxu_poll_wait_for_event (../../../../src/pm/hydra/tools/demux/demux_poll.c:76): callback returned error status [proxy:0:1@oakmnt-0-b] main (../../../../src/pm/hydra/pm/pmiserv/pmip.c:206): demux engine error waiting for event [mpiexec@oakmnt-0-a] HYDT_bscu_wait_for_completion (../../../../src/pm/hydra/tools/bootstrap/utils/bscu_wait.c:76): one of the processes terminated badly; aborting [mpiexec@oakmnt-0-a] HYDT_bsci_wait_for_completion (../../../../src/pm/hydra/tools/bootstrap/src/bsci_wait.c:23): launcher returned error waiting for completion [mpiexec@oakmnt-0-a] HYD_pmci_wait_for_completion (../../../../src/pm/hydra/pm/pmiserv/pmiserv_pmci.c:218): launcher returned error waiting for completion [mpiexec@oakmnt-0-a] main (../../../../src/pm/hydra/ui/mpich/mpiexec.c:344): process manager error waiting for completion
Amin Hassani, CIS department at UAB, Birmingham, AL, USA.
On Tue, Nov 25, 2014 at 11:44 PM, Lu, Huiwei <[email protected] <mailto:[email protected]>> wrote:
So the error only happens when there is communication.
It may be caused by IB as your guessed before. Could you try to reconfigure MPICH using "./configure --with-device=ch3:nemesis:tcp” and try again?
— Huiwei
> On Nov 25, 2014, at 11:23 PM, Amin Hassani <[email protected] <mailto:[email protected]>> wrote: > > Yes it works. > output: > > $ mpirun -hostfile hosts-hydra -np 2 test > rank 1 > rank 0 > > > Amin Hassani, > CIS department at UAB, > Birmingham, AL, USA. > > On Tue, Nov 25, 2014 at 11:20 PM, Lu, Huiwei <[email protected] <mailto:[email protected]>> wrote: > Could you try to run the following simple code to see if it works? > > #include <mpi.h> > #include <stdio.h> > int main(int argc, char** argv) > { > int rank, size; > MPI_Init(&argc, &argv); > MPI_Comm_rank(MPI_COMM_WORLD, &rank); > printf("rank %d\n", rank); > MPI_Finalize(); > return 0; > } > > — > Huiwei > > > On Nov 25, 2014, at 11:11 PM, Amin Hassani <[email protected] <mailto:[email protected]>> wrote: > > > > No, I checked. Also I always install my MPI's in /nethome/students/ahassani/usr/mpi. I never install them in /nethome/students/ahassani/usr. So MPI files will never get there. Even if put the /usr/mpi/bin in front of /usr/bin, it won't affect anything. There has never been any mpi installed in /usr/bin. > > > > Thank you. > > _______________________________________________ > > discuss mailing list [email protected] <mailto:[email protected]
> > To manage subscription options or unsubscribe: > > https://lists.mpich.org/mailman/listinfo/discuss > > _______________________________________________ > discuss mailing list [email protected] <mailto:[email protected]> > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > > _______________________________________________ > discuss mailing list [email protected] <mailto:[email protected]> > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss
_______________________________________________ discuss mailing list [email protected] <mailto:[email protected]> To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss
_______________________________________________ discuss mailing list [email protected] To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss
_______________________________________________ discuss mailing list [email protected] To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss
_______________________________________________ discuss mailing list [email protected] To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss