Tried with the new configure too. same problem :(

$ mpirun -hostfile hosts-hydra -np 2 test_dup

Fatal error in MPI_Send: Unknown error class, error stack:

MPI_Send(174)..............: MPI_Send(buf=0x7fffd90c76c8, count=1, MPI_INT, dest=1, tag=0, MPI_COMM_WORLD) failed

MPID_nem_tcp_connpoll(1832): Communication error with rank 1: Connection refused

===================================================================================

= BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES

= PID 5459 RUNNING AT oakmnt-0-a

= EXIT CODE: 1

= CLEANING UP REMAINING PROCESSES

= YOU CAN IGNORE THE BELOW CLEANUP MESSAGES

===================================================================================

[proxy:0:1@oakmnt-0-b] HYD_pmcd_pmip_control_cmd_cb (../../../../src/pm/hydra/pm/pmiserv/pmip_cb.c:885): assert (!closed) failed

[proxy:0:1@oakmnt-0-b] HYDT_dmxu_poll_wait_for_event (../../../../src/pm/hydra/tools/demux/demux_poll.c:76): callback returned error status

[proxy:0:1@oakmnt-0-b] main (../../../../src/pm/hydra/pm/pmiserv/pmip.c:206): demux engine error waiting for event

[mpiexec@oakmnt-0-a] HYDT_bscu_wait_for_completion (../../../../src/pm/hydra/tools/bootstrap/utils/bscu_wait.c:76): one of the processes terminated badly; aborting

[mpiexec@oakmnt-0-a] HYDT_bsci_wait_for_completion (../../../../src/pm/hydra/tools/bootstrap/src/bsci_wait.c:23): launcher returned error waiting for completion

[mpiexec@oakmnt-0-a] HYD_pmci_wait_for_completion (../../../../src/pm/hydra/pm/pmiserv/pmiserv_pmci.c:218): launcher returned error waiting for completion

[mpiexec@oakmnt-0-a] main (../../../../src/pm/hydra/ui/mpich/mpiexec.c:344): process manager error waiting for completion

Amin Hassani,
CIS department at UAB,
Birmingham, AL, USA.

On Tue, Nov 25, 2014 at 11:44 PM, Lu, Huiwei <huiweilu@mcs.anl.gov> wrote:

So the error only happens when there is communication.

It may be caused by IB as your guessed before. Could you try to reconfigure MPICH using "./configure --with-device=ch3:nemesis:tcp” and try again?

—
Huiwei

> On Nov 25, 2014, at 11:23 PM, Amin Hassani <ahassani@cis.uab.edu> wrote:
>
> Yes it works.
> output:
>
> $ mpirun -hostfile hosts-hydra -np 2 test
> rank 1
> rank 0
>
>
> Amin Hassani,
> CIS department at UAB,
> Birmingham, AL, USA.
>
> On Tue, Nov 25, 2014 at 11:20 PM, Lu, Huiwei <huiweilu@mcs.anl.gov> wrote:
> Could you try to run the following simple code to see if it works?
>
> #include <mpi.h>
> #include <stdio.h>
> int main(int argc, char** argv)
> {
> int rank, size;
> MPI_Init(&argc, &argv);
> MPI_Comm_rank(MPI_COMM_WORLD, &rank);
> printf("rank %d\n", rank);
> MPI_Finalize();
> return 0;
> }
>
> —
> Huiwei
>
> > On Nov 25, 2014, at 11:11 PM, Amin Hassani <ahassani@cis.uab.edu> wrote:
> >
> > No, I checked. Also I always install my MPI's in /nethome/students/ahassani/usr/mpi. I never install them in /nethome/students/ahassani/usr. So MPI files will never get there. Even if put the /usr/mpi/bin in front of /usr/bin, it won't affect anything. There has never been any mpi installed in /usr/bin.
> >
> > Thank you.
> > _______________________________________________
> > discuss mailing list discuss@mpich.org
> > To manage subscription options or unsubscribe:
> > https://lists.mpich.org/mailman/listinfo/discuss
>
> _______________________________________________
> discuss mailing list discuss@mpich.org
> To manage subscription options or unsubscribe:
> https://lists.mpich.org/mailman/listinfo/discuss
>
> _______________________________________________
> discuss mailing list discuss@mpich.org
> To manage subscription options or unsubscribe:
> https://lists.mpich.org/mailman/listinfo/discuss

_______________________________________________
discuss mailing list discuss@mpich.org
To manage subscription options or unsubscribe:
https://lists.mpich.org/mailman/listinfo/discuss