Pavan Unfortunately yes. The remote hosts info comes from the server via NIS so each node's /etc/hosts just has its local info in it (see below). I have had this working for years including with mpich. Just to be sure I made a new /etc/hosts for each node with both nodes in it. But the error still occurred. Besides I can ssh back and forth (without password) using the hostnames so I don't think this is the problem. I tried everything in the FAQ before I posted. Tony # Host table for ladd domain 127.0.0.1 pc5 localhost 192.168.1.105 pc5.ladd pc5 # Localhost 192.168.1.1 svr.ladd svr # Server Output from ypcat -k hosts. Note: nsswitch has the line hosts: files nis dns svr.che.ufl.edu 10.227.108.60 svr.che.ufl.edu pc4 192.168.1.104 pc4.ladd pc4 pc6.ladd 192.168.1.106 pc6.ladd pc6 svr.ladd 192.168.1.1 svr.ladd svr pc9 192.168.1.109 pc9.ladd pc9 pc7 192.168.1.107 pc7.ladd pc7 pc3 192.168.1.103 pc3.ladd pc3 pc1 192.168.1.101 pc1.ladd pc1 pc5.ladd 192.168.1.105 pc5.ladd pc5 localhost 127.0.0.1 svr localhost pc3.ladd 192.168.1.103 pc3.ladd pc3 pc8 192.168.1.108 pc8.ladd pc8 pc6 192.168.1.106 pc6.ladd pc6 checs 10.227.121.221 checs.che.ufl.edu checs pc2 192.168.1.102 pc2.ladd pc2 prn.ladd 192.168.1.11 prn.ladd prn pc8.ladd 192.168.1.108 pc8.ladd pc8 svr 192.168.1.1 svr.ladd svr pc4.ladd 192.168.1.104 pc4.ladd pc4 pc2.ladd 192.168.1.102 pc2.ladd pc2 pc5 192.168.1.105 pc5.ladd pc5 pc9.ladd 192.168.1.109 pc9.ladd pc9 pc7.ladd 192.168.1.107 pc7.ladd pc7 checs.che.ufl.edu 10.227.121.221 checs.che.ufl.edu checs pc1.ladd 192.168.1.101 pc1.ladd pc1 prn 192.168.1.11 prn.ladd prn On 03/27/2014 12:05 PM, Balaji, Pavan wrote:
Tony,
You didn�t quite mention it, but I assume you meant the problem exists even with two identical nodes?
What about the /etc/hosts files? Are they consistent?
Did you try the options on this FAQ post:
http://wiki.mpich.org/mpich/index.php/Frequently_Asked_Questions#Q:_My_MPI_p...
� Pavan
On Mar 27, 2014, at 10:14 AM, Tony Ladd <[email protected]> wrote:
Pavan
Same OS - but it was different hardware. So I tried the cpi example on two identical nodes (Dell Optiplex 745) this morning. The OS is Centos 6.5 and the installation on these client nodes is entirely automated so I am sure the configurations on the two boxes are identical (the install is new and the boxes have not been used so far). I used the version of cpi compiled during the installation of mpich. Here is the log file.
I am also including the installation logs in case that helps - I have separate logs of the configure, make, and install stages.
Tony
On 03/27/2014 02:02 AM, Balaji, Pavan wrote:
Are both the nodes similar in architecture and OS configuration?
Are the /etc/hosts files on both machines consistent?
� Pavan
On Mar 26, 2014, at 9:01 PM, Tony Ladd <[email protected]> wrote:
Rajeev
There is a firewall on svr but its configured to accept all packets on the interface connected to the internal domain (where pc5 lives). I had already checked that stopping iptables off made no difference, but I just tried it again on the cpi example. The result was the same.
Tony
On 03/26/2014 09:45 PM, Rajeev Thakur wrote:
Is there a firewall on either machine that is in the way of communication?
Rajeev
On Mar 26, 2014, at 8:28 PM, Tony Ladd <[email protected]> wrote:
No - you get the same error - it looks as if process 1 (on the remote node) is not starting
svr:tladd(netbench)> mpirun -n 2 -f hosts /global/usr/src/mpich-3.0.4/examples/cpi Process 0 of 2 is on svr.che.ufl.edu Fatal error in PMPI_Reduce: A process has failed, error stack: PMPI_Reduce(1217)...............: MPI_Reduce(sbuf=0x7fff30ecced8, rbuf=0x7fff30ecced0, count=1, MPI_DOUBLE,
But if I reverse the order in the host file (pc5 first and then svr) apparently both processes start
svr:tladd(netbench)> mpirun -n 2 -f hosts /global/usr/src/mpich-3.0.4/examples/cpi Process 1 of 2 is on svr.che.ufl.edu Process 0 of 2 is on pc5 Fatal error in PMPI_Reduce: A process has failed, error stack: PMPI_Reduce(1217)...............: MPI_Reduce(sbuf=0x7fff4d776348, rbuf=0x7fff4d776340, count=1, MPI_DOUBLE,
But with the same result in the end.
Tony
On 03/26/2014 08:18 PM, Rajeev Thakur wrote:
Does the cpi example run across two machines?
Rajeev
On Mar 26, 2014, at 7:13 PM, Tony Ladd <[email protected]> wrote:
Rajeev
Sorry about that. I was switching back and forth from openmpi to mpich. But it does not make a difference. Here is a clean log from a fresh terminal - no mention of openmpi
Tony
PS - its a CentOS 6.5install - should have mentioned it before.
-- Tony Ladd
Chemical Engineering Department University of Florida Gainesville, Florida 32611-6005 USA
Email: tladd-"(AT)"-che.ufl.edu Web http://ladd.che.ufl.edu
Tel: (352)-392-6509 FAX: (352)-392-9514
<mpich.log>_______________________________________________ discuss mailing list [email protected] To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss
discuss mailing list [email protected] To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss -- Tony Ladd
Chemical Engineering Department University of Florida Gainesville, Florida 32611-6005 USA
Email: tladd-"(AT)"-che.ufl.edu Web http://ladd.che.ufl.edu
Tel: (352)-392-6509 FAX: (352)-392-9514
_______________________________________________ discuss mailing list [email protected] To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss
discuss mailing list [email protected] To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss -- Tony Ladd
Chemical Engineering Department University of Florida Gainesville, Florida 32611-6005 USA
Email: tladd-"(AT)"-che.ufl.edu Web http://ladd.che.ufl.edu
Tel: (352)-392-6509 FAX: (352)-392-9514
_______________________________________________ discuss mailing list [email protected] To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss
discuss mailing list [email protected] To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss -- Tony Ladd
Chemical Engineering Department University of Florida Gainesville, Florida 32611-6005 USA
Email: tladd-"(AT)"-che.ufl.edu Web http://ladd.che.ufl.edu
Tel: (352)-392-6509 FAX: (352)-392-9514
<mpich.log><config.log><configure.log><install.log><make.log>_______________________________________________ discuss mailing list [email protected] To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss
discuss mailing list [email protected] To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss
-- Tony Ladd Chemical Engineering Department University of Florida Gainesville, Florida 32611-6005 USA Email: tladd-"(AT)"-che.ufl.edu Web http://ladd.che.ufl.edu Tel: (352)-392-6509 FAX: (352)-392-9514