Hi Bill, I just tried the program with MPICH-3.2.1/gfortran/ssh, it works fine. I guess it is an issue related to the low level network connection. We might need more information from you to figure out the problem. Could you please first check: - Make sure `mpiexec` is the one installed with MPICH-3.2.1, but not the old one installed with MPICH-1.2.7 - Send us the output of `mpichversion` And try: mpiexec -f machines -launcher rsh –n 3 hostname Min On 2018/06/19 16:28, Bill Petrick wrote:
Hi all,
For the past 9 years I've been running MPICH-1.2.7 with the g95 fortran compiler. I just upgraded to MPICH-3.2.1 with a gfortran compiler. My first tests running on 2 nodes were fine. When I ran the program on 3 or more nodes it crashes at the first MPI_BARRIER call. As a test I ran the following "helloworld" program. It runs on all nodes just fine until I add the MPI_BARRIER statement. With the MPI_BARRIER statement it also runs fine on 2 nodes but crashes when run on 3 or more nodes. I execute with the following statement.
mpiexec -f machines -launcher rsh –n 2 ./helloworld
This program runs fine on all nodes without the MPI_BARRIER call but crashes on more than two nodes with the BARRIER call. ____________________________________________________ program main use mpi integer ( kind = 4 ) error integer ( kind = 4 ) id integer ( kind = 4 ) p real ( kind = 8 ) wtime ! Initialize MPI. call MPI_Init ( error ) ! Get the number of processes. call MPI_Comm_size ( MPI_COMM_WORLD, p, error ) ! Get the individual process ID. call MPI_Comm_rank ( MPI_COMM_WORLD, id, error ) call MPI_BARRIER(MPI_COMM_WORLD,error) ! ! Print a message. ! .................etc__________________________________________
However when I run it on 3 or more nodes it crashes at MPI_BARRIER with the following error message __________________________________________________________________________
iic@node0:/clusterfiles$ mpiexec -f machines -launcher rsh -n 3 ./helloworld Fatal error in PMPI_Barrier: Unknown error class, error stack: PMPI_Barrier(425)......: MPI_Barrier(MPI_COMM_WORLD) failed MPIR_Barrier_impl(332).: Failure during collective MPIR_Barrier_impl(327).: MPIR_Barrier(292)......: MPIR_Barrier_intra(180): Failure during collective Fatal error in PMPI_Barrier: Unknown error class, error stack: PMPI_Barrier(425)...............: MPI_Barrier(MPI_COMM_WORLD) failed MPIR_Barrier_impl(332)..........: Failure during collective MPIR_Barrier_impl(327)..........: MPIR_Barrier(292)...............: MPIR_Barrier_intra(169).........: MPID_nem_tcp_connpoll(1845).....: Communication error with rank 2: Connection refused MPIR_Barrier_intra(169).........: MPIDI_CH3U_Recvq_FDU_or_AEP(629): Communication error with rank 2 iic@node0:/clusterfiles$ ______________________________________________________
/etc/hosts is consistent across all nodes. /usr/bin/rsh exists on all nodes the "machines" file is the same on all nodes.
Anyone have any ideas about what could be wrong? Thanks, Bill
_______________________________________________ discuss mailing list [email protected] To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss
_______________________________________________ discuss mailing list [email protected] To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss