March 2014 - discuss - mailman3.cels.anl.gov

Re: [mpich-discuss] Fatal error in PMPI_Barrier: A process has failed, error stack:
by Kenneth Raffenetti 27 Mar '14

27 Mar '14

Try using pc5.ladd (or another fully qualified name) instead of pc5 in your MPI hosts file. Communication with that machine is being bound to the localhost interface from the look of it. Ken On 03/27/2014 12:12 PM, Tony Ladd wrote: > Pavan > > Unfortunately yes. > > The remote hosts info comes from the server via NIS so each node's > /etc/hosts just has its local info in it (see below). I have had this > working for years including with mpich. Just to be sure I made a new > /etc/hosts for each node with both nodes in it. But the error still > occurred. Besides I can ssh back and forth (without password) using the > hostnames so I don't think this is the problem. > > I tried everything in the FAQ before I posted. > > Tony > > > # Host table for ladd domain > > 127.0.0.1 pc5 localhost > > 192.168.1.105 pc5.ladd pc5 # Localhost > > 192.168.1.1 svr.ladd svr # Server > > Output from ypcat -k hosts. Note: nsswitch has the line > hosts: files nis dns > > svr.che.ufl.edu 10.227.108.60 svr.che.ufl.edu > pc4 192.168.1.104 pc4.ladd pc4 > pc6.ladd 192.168.1.106 pc6.ladd pc6 > svr.ladd 192.168.1.1 svr.ladd svr > pc9 192.168.1.109 pc9.ladd pc9 > pc7 192.168.1.107 pc7.ladd pc7 > pc3 192.168.1.103 pc3.ladd pc3 > pc1 192.168.1.101 pc1.ladd pc1 > pc5.ladd 192.168.1.105 pc5.ladd pc5 > localhost 127.0.0.1 svr localhost > pc3.ladd 192.168.1.103 pc3.ladd pc3 > pc8 192.168.1.108 pc8.ladd pc8 > pc6 192.168.1.106 pc6.ladd pc6 > checs 10.227.121.221 checs.che.ufl.edu checs > pc2 192.168.1.102 pc2.ladd pc2 > prn.ladd 192.168.1.11 prn.ladd prn > pc8.ladd 192.168.1.108 pc8.ladd pc8 > svr 192.168.1.1 svr.ladd svr > pc4.ladd 192.168.1.104 pc4.ladd pc4 > pc2.ladd 192.168.1.102 pc2.ladd pc2 > pc5 192.168.1.105 pc5.ladd pc5 > pc9.ladd 192.168.1.109 pc9.ladd pc9 > pc7.ladd 192.168.1.107 pc7.ladd pc7 > checs.che.ufl.edu 10.227.121.221 checs.che.ufl.edu checs > pc1.ladd 192.168.1.101 pc1.ladd pc1 > prn 192.168.1.11 prn.ladd prn > > > > > On 03/27/2014 12:05 PM, Balaji, Pavan wrote: >> Tony, >> >> You didn’t quite mention it, but I assume you meant the problem exists >> even with two identical nodes? >> >> What about the /etc/hosts files? Are they consistent? >> >> Did you try the options on this FAQ post: >> >> http://wiki.mpich.org/mpich/index.php/Frequently_Asked_Questions#Q:_My_MPI_… >> >> >> — Pavan >> >> On Mar 27, 2014, at 10:14 AM, Tony Ladd <tladd(a)che.ufl.edu> wrote: >> >>> Pavan >>> >>> Same OS - but it was different hardware. So I tried the cpi example >>> on two identical nodes (Dell Optiplex 745) this morning. The OS is >>> Centos 6.5 and the installation on these client nodes is entirely >>> automated so I am sure the configurations on the two boxes are >>> identical (the install is new and the boxes have not been used so >>> far). I used the version of cpi compiled during the installation of >>> mpich. Here is the log file. >>> >>> I am also including the installation logs in case that helps - I have >>> separate logs of the configure, make, and install stages. >>> >>> Tony >>> >>> >>> On 03/27/2014 02:02 AM, Balaji, Pavan wrote: >>>> Are both the nodes similar in architecture and OS configuration? >>>> >>>> Are the /etc/hosts files on both machines consistent? >>>> >>>> — Pavan >>>> >>>> On Mar 26, 2014, at 9:01 PM, Tony Ladd <tladd(a)che.ufl.edu> wrote: >>>> >>>>> Rajeev >>>>> >>>>> There is a firewall on svr but its configured to accept all packets >>>>> on the interface connected to the internal domain (where pc5 >>>>> lives). I had already checked that stopping iptables off made no >>>>> difference, but I just tried it again on the cpi example. The >>>>> result was the same. >>>>> >>>>> Tony >>>>> >>>>> >>>>> On 03/26/2014 09:45 PM, Rajeev Thakur wrote: >>>>>> Is there a firewall on either machine that is in the way of >>>>>> communication? >>>>>> >>>>>> Rajeev >>>>>> >>>>>> On Mar 26, 2014, at 8:28 PM, Tony Ladd <tladd(a)che.ufl.edu> >>>>>> wrote: >>>>>> >>>>>>> No - you get the same error - it looks as if process 1 (on the >>>>>>> remote node) is not starting >>>>>>> >>>>>>> svr:tladd(netbench)> mpirun -n 2 -f hosts >>>>>>> /global/usr/src/mpich-3.0.4/examples/cpi >>>>>>> Process 0 of 2 is on svr.che.ufl.edu >>>>>>> Fatal error in PMPI_Reduce: A process has failed, error stack: >>>>>>> PMPI_Reduce(1217)...............: MPI_Reduce(sbuf=0x7fff30ecced8, >>>>>>> rbuf=0x7fff30ecced0, count=1, MPI_DOUBLE, >>>>>>> >>>>>>> But if I reverse the order in the host file (pc5 first and then >>>>>>> svr) apparently both processes start >>>>>>> >>>>>>> svr:tladd(netbench)> mpirun -n 2 -f hosts >>>>>>> /global/usr/src/mpich-3.0.4/examples/cpi >>>>>>> Process 1 of 2 is on svr.che.ufl.edu >>>>>>> Process 0 of 2 is on pc5 >>>>>>> Fatal error in PMPI_Reduce: A process has failed, error stack: >>>>>>> PMPI_Reduce(1217)...............: MPI_Reduce(sbuf=0x7fff4d776348, >>>>>>> rbuf=0x7fff4d776340, count=1, MPI_DOUBLE, >>>>>>> >>>>>>> But with the same result in the end. >>>>>>> >>>>>>> Tony >>>>>>> >>>>>>> >>>>>>> >>>>>>> On 03/26/2014 08:18 PM, Rajeev Thakur wrote: >>>>>>>> Does the cpi example run across two machines? >>>>>>>> >>>>>>>> Rajeev >>>>>>>> >>>>>>>> On Mar 26, 2014, at 7:13 PM, Tony Ladd <tladd(a)che.ufl.edu> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Rajeev >>>>>>>>> >>>>>>>>> Sorry about that. I was switching back and forth from openmpi >>>>>>>>> to mpich. But it does not make a difference. Here is a clean >>>>>>>>> log from a fresh terminal - no mention of openmpi >>>>>>>>> >>>>>>>>> Tony >>>>>>>>> >>>>>>>>> PS - its a CentOS 6.5install - should have mentioned it before. >>>>>>>>> >>>>>>>>> -- >>>>>>>>> Tony Ladd >>>>>>>>> >>>>>>>>> Chemical Engineering Department >>>>>>>>> University of Florida >>>>>>>>> Gainesville, Florida 32611-6005 >>>>>>>>> USA >>>>>>>>> >>>>>>>>> Email: tladd-"(AT)"-che.ufl.edu >>>>>>>>> Web http://ladd.che.ufl.edu >>>>>>>>> >>>>>>>>> Tel: (352)-392-6509 >>>>>>>>> FAX: (352)-392-9514 >>>>>>>>> >>>>>>>>> <mpich.log>_______________________________________________ >>>>>>>>> discuss mailing list discuss(a)mpich.org >>>>>>>>> To manage subscription options or unsubscribe: >>>>>>>>> https://lists.mpich.org/mailman/listinfo/discuss >>>>>>>> _______________________________________________ >>>>>>>> discuss mailing list discuss(a)mpich.org >>>>>>>> To manage subscription options or unsubscribe: >>>>>>>> https://lists.mpich.org/mailman/listinfo/discuss >>>>>>> -- >>>>>>> Tony Ladd >>>>>>> >>>>>>> Chemical Engineering Department >>>>>>> University of Florida >>>>>>> Gainesville, Florida 32611-6005 >>>>>>> USA >>>>>>> >>>>>>> Email: tladd-"(AT)"-che.ufl.edu >>>>>>> Web http://ladd.che.ufl.edu >>>>>>> >>>>>>> Tel: (352)-392-6509 >>>>>>> FAX: (352)-392-9514 >>>>>>> >>>>>>> _______________________________________________ >>>>>>> discuss mailing list discuss(a)mpich.org >>>>>>> To manage subscription options or unsubscribe: >>>>>>> https://lists.mpich.org/mailman/listinfo/discuss >>>>>> _______________________________________________ >>>>>> discuss mailing list discuss(a)mpich.org >>>>>> To manage subscription options or unsubscribe: >>>>>> https://lists.mpich.org/mailman/listinfo/discuss >>>>> -- >>>>> Tony Ladd >>>>> >>>>> Chemical Engineering Department >>>>> University of Florida >>>>> Gainesville, Florida 32611-6005 >>>>> USA >>>>> >>>>> Email: tladd-"(AT)"-che.ufl.edu >>>>> Web http://ladd.che.ufl.edu >>>>> >>>>> Tel: (352)-392-6509 >>>>> FAX: (352)-392-9514 >>>>> >>>>> _______________________________________________ >>>>> discuss mailing list discuss(a)mpich.org >>>>> To manage subscription options or unsubscribe: >>>>> https://lists.mpich.org/mailman/listinfo/discuss >>>> _______________________________________________ >>>> discuss mailing list discuss(a)mpich.org >>>> To manage subscription options or unsubscribe: >>>> https://lists.mpich.org/mailman/listinfo/discuss >>> -- >>> Tony Ladd >>> >>> Chemical Engineering Department >>> University of Florida >>> Gainesville, Florida 32611-6005 >>> USA >>> >>> Email: tladd-"(AT)"-che.ufl.edu >>> Web http://ladd.che.ufl.edu >>> >>> Tel: (352)-392-6509 >>> FAX: (352)-392-9514 >>> >>> <mpich.log><config.log><configure.log><install.log><make.log>_______________________________________________ >>> >>> discuss mailing list discuss(a)mpich.org >>> To manage subscription options or unsubscribe: >>> https://lists.mpich.org/mailman/listinfo/discuss >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss >

1 0

Re: [mpich-discuss] Fatal error in PMPI_Barrier: A process has failed, error stack:
by Tony Ladd 27 Mar '14

27 Mar '14

Pavan Unfortunately yes. The remote hosts info comes from the server via NIS so each node's /etc/hosts just has its local info in it (see below). I have had this working for years including with mpich. Just to be sure I made a new /etc/hosts for each node with both nodes in it. But the error still occurred. Besides I can ssh back and forth (without password) using the hostnames so I don't think this is the problem. I tried everything in the FAQ before I posted. Tony # Host table for ladd domain 127.0.0.1 pc5 localhost 192.168.1.105 pc5.ladd pc5 # Localhost 192.168.1.1 svr.ladd svr # Server Output from ypcat -k hosts. Note: nsswitch has the line hosts: files nis dns svr.che.ufl.edu 10.227.108.60 svr.che.ufl.edu pc4 192.168.1.104 pc4.ladd pc4 pc6.ladd 192.168.1.106 pc6.ladd pc6 svr.ladd 192.168.1.1 svr.ladd svr pc9 192.168.1.109 pc9.ladd pc9 pc7 192.168.1.107 pc7.ladd pc7 pc3 192.168.1.103 pc3.ladd pc3 pc1 192.168.1.101 pc1.ladd pc1 pc5.ladd 192.168.1.105 pc5.ladd pc5 localhost 127.0.0.1 svr localhost pc3.ladd 192.168.1.103 pc3.ladd pc3 pc8 192.168.1.108 pc8.ladd pc8 pc6 192.168.1.106 pc6.ladd pc6 checs 10.227.121.221 checs.che.ufl.edu checs pc2 192.168.1.102 pc2.ladd pc2 prn.ladd 192.168.1.11 prn.ladd prn pc8.ladd 192.168.1.108 pc8.ladd pc8 svr 192.168.1.1 svr.ladd svr pc4.ladd 192.168.1.104 pc4.ladd pc4 pc2.ladd 192.168.1.102 pc2.ladd pc2 pc5 192.168.1.105 pc5.ladd pc5 pc9.ladd 192.168.1.109 pc9.ladd pc9 pc7.ladd 192.168.1.107 pc7.ladd pc7 checs.che.ufl.edu 10.227.121.221 checs.che.ufl.edu checs pc1.ladd 192.168.1.101 pc1.ladd pc1 prn 192.168.1.11 prn.ladd prn On 03/27/2014 12:05 PM, Balaji, Pavan wrote: > Tony, > > You didnï¿½t quite mention it, but I assume you meant the problem exists even with two identical nodes? > > What about the /etc/hosts files? Are they consistent? > > Did you try the options on this FAQ post: > > http://wiki.mpich.org/mpich/index.php/Frequently_Asked_Questions#Q:_My_MPI_… > > ï¿½ Pavan > > On Mar 27, 2014, at 10:14 AM, Tony Ladd <tladd(a)che.ufl.edu> wrote: > >> Pavan >> >> Same OS - but it was different hardware. So I tried the cpi example on two identical nodes (Dell Optiplex 745) this morning. The OS is Centos 6.5 and the installation on these client nodes is entirely automated so I am sure the configurations on the two boxes are identical (the install is new and the boxes have not been used so far). I used the version of cpi compiled during the installation of mpich. Here is the log file. >> >> I am also including the installation logs in case that helps - I have separate logs of the configure, make, and install stages. >> >> Tony >> >> >> On 03/27/2014 02:02 AM, Balaji, Pavan wrote: >>> Are both the nodes similar in architecture and OS configuration? >>> >>> Are the /etc/hosts files on both machines consistent? >>> >>> ï¿½ Pavan >>> >>> On Mar 26, 2014, at 9:01 PM, Tony Ladd <tladd(a)che.ufl.edu> wrote: >>> >>>> Rajeev >>>> >>>> There is a firewall on svr but its configured to accept all packets on the interface connected to the internal domain (where pc5 lives). I had already checked that stopping iptables off made no difference, but I just tried it again on the cpi example. The result was the same. >>>> >>>> Tony >>>> >>>> >>>> On 03/26/2014 09:45 PM, Rajeev Thakur wrote: >>>>> Is there a firewall on either machine that is in the way of communication? >>>>> >>>>> Rajeev >>>>> >>>>> On Mar 26, 2014, at 8:28 PM, Tony Ladd <tladd(a)che.ufl.edu> >>>>> wrote: >>>>> >>>>>> No - you get the same error - it looks as if process 1 (on the remote node) is not starting >>>>>> >>>>>> svr:tladd(netbench)> mpirun -n 2 -f hosts /global/usr/src/mpich-3.0.4/examples/cpi >>>>>> Process 0 of 2 is on svr.che.ufl.edu >>>>>> Fatal error in PMPI_Reduce: A process has failed, error stack: >>>>>> PMPI_Reduce(1217)...............: MPI_Reduce(sbuf=0x7fff30ecced8, rbuf=0x7fff30ecced0, count=1, MPI_DOUBLE, >>>>>> >>>>>> But if I reverse the order in the host file (pc5 first and then svr) apparently both processes start >>>>>> >>>>>> svr:tladd(netbench)> mpirun -n 2 -f hosts /global/usr/src/mpich-3.0.4/examples/cpi >>>>>> Process 1 of 2 is on svr.che.ufl.edu >>>>>> Process 0 of 2 is on pc5 >>>>>> Fatal error in PMPI_Reduce: A process has failed, error stack: >>>>>> PMPI_Reduce(1217)...............: MPI_Reduce(sbuf=0x7fff4d776348, rbuf=0x7fff4d776340, count=1, MPI_DOUBLE, >>>>>> >>>>>> But with the same result in the end. >>>>>> >>>>>> Tony >>>>>> >>>>>> >>>>>> >>>>>> On 03/26/2014 08:18 PM, Rajeev Thakur wrote: >>>>>>> Does the cpi example run across two machines? >>>>>>> >>>>>>> Rajeev >>>>>>> >>>>>>> On Mar 26, 2014, at 7:13 PM, Tony Ladd <tladd(a)che.ufl.edu> >>>>>>> wrote: >>>>>>> >>>>>>>> Rajeev >>>>>>>> >>>>>>>> Sorry about that. I was switching back and forth from openmpi to mpich. But it does not make a difference. Here is a clean log from a fresh terminal - no mention of openmpi >>>>>>>> >>>>>>>> Tony >>>>>>>> >>>>>>>> PS - its a CentOS 6.5install - should have mentioned it before. >>>>>>>> >>>>>>>> -- >>>>>>>> Tony Ladd >>>>>>>> >>>>>>>> Chemical Engineering Department >>>>>>>> University of Florida >>>>>>>> Gainesville, Florida 32611-6005 >>>>>>>> USA >>>>>>>> >>>>>>>> Email: tladd-"(AT)"-che.ufl.edu >>>>>>>> Web http://ladd.che.ufl.edu >>>>>>>> >>>>>>>> Tel: (352)-392-6509 >>>>>>>> FAX: (352)-392-9514 >>>>>>>> >>>>>>>> <mpich.log>_______________________________________________ >>>>>>>> discuss mailing list discuss(a)mpich.org >>>>>>>> To manage subscription options or unsubscribe: >>>>>>>> https://lists.mpich.org/mailman/listinfo/discuss >>>>>>> _______________________________________________ >>>>>>> discuss mailing list discuss(a)mpich.org >>>>>>> To manage subscription options or unsubscribe: >>>>>>> https://lists.mpich.org/mailman/listinfo/discuss >>>>>> -- >>>>>> Tony Ladd >>>>>> >>>>>> Chemical Engineering Department >>>>>> University of Florida >>>>>> Gainesville, Florida 32611-6005 >>>>>> USA >>>>>> >>>>>> Email: tladd-"(AT)"-che.ufl.edu >>>>>> Web http://ladd.che.ufl.edu >>>>>> >>>>>> Tel: (352)-392-6509 >>>>>> FAX: (352)-392-9514 >>>>>> >>>>>> _______________________________________________ >>>>>> discuss mailing list discuss(a)mpich.org >>>>>> To manage subscription options or unsubscribe: >>>>>> https://lists.mpich.org/mailman/listinfo/discuss >>>>> _______________________________________________ >>>>> discuss mailing list discuss(a)mpich.org >>>>> To manage subscription options or unsubscribe: >>>>> https://lists.mpich.org/mailman/listinfo/discuss >>>> -- >>>> Tony Ladd >>>> >>>> Chemical Engineering Department >>>> University of Florida >>>> Gainesville, Florida 32611-6005 >>>> USA >>>> >>>> Email: tladd-"(AT)"-che.ufl.edu >>>> Web http://ladd.che.ufl.edu >>>> >>>> Tel: (352)-392-6509 >>>> FAX: (352)-392-9514 >>>> >>>> _______________________________________________ >>>> discuss mailing list discuss(a)mpich.org >>>> To manage subscription options or unsubscribe: >>>> https://lists.mpich.org/mailman/listinfo/discuss >>> _______________________________________________ >>> discuss mailing list discuss(a)mpich.org >>> To manage subscription options or unsubscribe: >>> https://lists.mpich.org/mailman/listinfo/discuss >> -- >> Tony Ladd >> >> Chemical Engineering Department >> University of Florida >> Gainesville, Florida 32611-6005 >> USA >> >> Email: tladd-"(AT)"-che.ufl.edu >> Web http://ladd.che.ufl.edu >> >> Tel: (352)-392-6509 >> FAX: (352)-392-9514 >> >> <mpich.log><config.log><configure.log><install.log><make.log>_______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss -- Tony Ladd Chemical Engineering Department University of Florida Gainesville, Florida 32611-6005 USA Email: tladd-"(AT)"-che.ufl.edu Web http://ladd.che.ufl.edu Tel: (352)-392-6509 FAX: (352)-392-9514

1 0

Re: [mpich-discuss] Fatal error in PMPI_Barrier: A process has failed, error stack:
by Balaji, Pavan 27 Mar '14

27 Mar '14

Tony, You didn’t quite mention it, but I assume you meant the problem exists even with two identical nodes? What about the /etc/hosts files? Are they consistent? Did you try the options on this FAQ post: http://wiki.mpich.org/mpich/index.php/Frequently_Asked_Questions#Q:_My_MPI_… — Pavan On Mar 27, 2014, at 10:14 AM, Tony Ladd <tladd(a)che.ufl.edu> wrote: > Pavan > > Same OS - but it was different hardware. So I tried the cpi example on two identical nodes (Dell Optiplex 745) this morning. The OS is Centos 6.5 and the installation on these client nodes is entirely automated so I am sure the configurations on the two boxes are identical (the install is new and the boxes have not been used so far). I used the version of cpi compiled during the installation of mpich. Here is the log file. > > I am also including the installation logs in case that helps - I have separate logs of the configure, make, and install stages. > > Tony > > > On 03/27/2014 02:02 AM, Balaji, Pavan wrote: >> Are both the nodes similar in architecture and OS configuration? >> >> Are the /etc/hosts files on both machines consistent? >> >> — Pavan >> >> On Mar 26, 2014, at 9:01 PM, Tony Ladd <tladd(a)che.ufl.edu> wrote: >> >>> Rajeev >>> >>> There is a firewall on svr but its configured to accept all packets on the interface connected to the internal domain (where pc5 lives). I had already checked that stopping iptables off made no difference, but I just tried it again on the cpi example. The result was the same. >>> >>> Tony >>> >>> >>> On 03/26/2014 09:45 PM, Rajeev Thakur wrote: >>>> Is there a firewall on either machine that is in the way of communication? >>>> >>>> Rajeev >>>> >>>> On Mar 26, 2014, at 8:28 PM, Tony Ladd <tladd(a)che.ufl.edu> >>>> wrote: >>>> >>>>> No - you get the same error - it looks as if process 1 (on the remote node) is not starting >>>>> >>>>> svr:tladd(netbench)> mpirun -n 2 -f hosts /global/usr/src/mpich-3.0.4/examples/cpi >>>>> Process 0 of 2 is on svr.che.ufl.edu >>>>> Fatal error in PMPI_Reduce: A process has failed, error stack: >>>>> PMPI_Reduce(1217)...............: MPI_Reduce(sbuf=0x7fff30ecced8, rbuf=0x7fff30ecced0, count=1, MPI_DOUBLE, >>>>> >>>>> But if I reverse the order in the host file (pc5 first and then svr) apparently both processes start >>>>> >>>>> svr:tladd(netbench)> mpirun -n 2 -f hosts /global/usr/src/mpich-3.0.4/examples/cpi >>>>> Process 1 of 2 is on svr.che.ufl.edu >>>>> Process 0 of 2 is on pc5 >>>>> Fatal error in PMPI_Reduce: A process has failed, error stack: >>>>> PMPI_Reduce(1217)...............: MPI_Reduce(sbuf=0x7fff4d776348, rbuf=0x7fff4d776340, count=1, MPI_DOUBLE, >>>>> >>>>> But with the same result in the end. >>>>> >>>>> Tony >>>>> >>>>> >>>>> >>>>> On 03/26/2014 08:18 PM, Rajeev Thakur wrote: >>>>>> Does the cpi example run across two machines? >>>>>> >>>>>> Rajeev >>>>>> >>>>>> On Mar 26, 2014, at 7:13 PM, Tony Ladd <tladd(a)che.ufl.edu> >>>>>> wrote: >>>>>> >>>>>>> Rajeev >>>>>>> >>>>>>> Sorry about that. I was switching back and forth from openmpi to mpich. But it does not make a difference. Here is a clean log from a fresh terminal - no mention of openmpi >>>>>>> >>>>>>> Tony >>>>>>> >>>>>>> PS - its a CentOS 6.5install - should have mentioned it before. >>>>>>> >>>>>>> -- >>>>>>> Tony Ladd >>>>>>> >>>>>>> Chemical Engineering Department >>>>>>> University of Florida >>>>>>> Gainesville, Florida 32611-6005 >>>>>>> USA >>>>>>> >>>>>>> Email: tladd-"(AT)"-che.ufl.edu >>>>>>> Web http://ladd.che.ufl.edu >>>>>>> >>>>>>> Tel: (352)-392-6509 >>>>>>> FAX: (352)-392-9514 >>>>>>> >>>>>>> <mpich.log>_______________________________________________ >>>>>>> discuss mailing list discuss(a)mpich.org >>>>>>> To manage subscription options or unsubscribe: >>>>>>> https://lists.mpich.org/mailman/listinfo/discuss >>>>>> _______________________________________________ >>>>>> discuss mailing list discuss(a)mpich.org >>>>>> To manage subscription options or unsubscribe: >>>>>> https://lists.mpich.org/mailman/listinfo/discuss >>>>> -- >>>>> Tony Ladd >>>>> >>>>> Chemical Engineering Department >>>>> University of Florida >>>>> Gainesville, Florida 32611-6005 >>>>> USA >>>>> >>>>> Email: tladd-"(AT)"-che.ufl.edu >>>>> Web http://ladd.che.ufl.edu >>>>> >>>>> Tel: (352)-392-6509 >>>>> FAX: (352)-392-9514 >>>>> >>>>> _______________________________________________ >>>>> discuss mailing list discuss(a)mpich.org >>>>> To manage subscription options or unsubscribe: >>>>> https://lists.mpich.org/mailman/listinfo/discuss >>>> _______________________________________________ >>>> discuss mailing list discuss(a)mpich.org >>>> To manage subscription options or unsubscribe: >>>> https://lists.mpich.org/mailman/listinfo/discuss >>> -- >>> Tony Ladd >>> >>> Chemical Engineering Department >>> University of Florida >>> Gainesville, Florida 32611-6005 >>> USA >>> >>> Email: tladd-"(AT)"-che.ufl.edu >>> Web http://ladd.che.ufl.edu >>> >>> Tel: (352)-392-6509 >>> FAX: (352)-392-9514 >>> >>> _______________________________________________ >>> discuss mailing list discuss(a)mpich.org >>> To manage subscription options or unsubscribe: >>> https://lists.mpich.org/mailman/listinfo/discuss >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss > > -- > Tony Ladd > > Chemical Engineering Department > University of Florida > Gainesville, Florida 32611-6005 > USA > > Email: tladd-"(AT)"-che.ufl.edu > Web http://ladd.che.ufl.edu > > Tel: (352)-392-6509 > FAX: (352)-392-9514 > > <mpich.log><config.log><configure.log><install.log><make.log>_______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Fatal error in PMPI_Barrier: A process has failed, error stack:
by Tony Ladd 27 Mar '14

27 Mar '14

Pavan I did wonder if my environment (the Python stuff) might be causing a problem. So I tried again with a minimal .bashrc. Same problem but the log file is a bit cleaner (and there is no firewall on either of these nodes) Tony On 03/27/2014 11:14 AM, Tony Ladd wrote: > Pavan > > Same OS - but it was different hardware. So I tried the cpi example on > two identical nodes (Dell Optiplex 745) this morning. The OS is Centos > 6.5 and the installation on these client nodes is entirely automated > so I am sure the configurations on the two boxes are identical (the > install is new and the boxes have not been used so far). I used the > version of cpi compiled during the installation of mpich. Here is the > log file. > > I am also including the installation logs in case that helps - I have > separate logs of the configure, make, and install stages. > > Tony > > > On 03/27/2014 02:02 AM, Balaji, Pavan wrote: >> Are both the nodes similar in architecture and OS configuration? >> >> Are the /etc/hosts files on both machines consistent? >> >> ï¿½ Pavan >> >> On Mar 26, 2014, at 9:01 PM, Tony Ladd <tladd(a)che.ufl.edu> wrote: >> >>> Rajeev >>> >>> There is a firewall on svr but its configured to accept all packets >>> on the interface connected to the internal domain (where pc5 lives). >>> I had already checked that stopping iptables off made no difference, >>> but I just tried it again on the cpi example. The result was the same. >>> >>> Tony >>> >>> >>> On 03/26/2014 09:45 PM, Rajeev Thakur wrote: >>>> Is there a firewall on either machine that is in the way of >>>> communication? >>>> >>>> Rajeev >>>> >>>> On Mar 26, 2014, at 8:28 PM, Tony Ladd <tladd(a)che.ufl.edu> >>>> wrote: >>>> >>>>> No - you get the same error - it looks as if process 1 (on the >>>>> remote node) is not starting >>>>> >>>>> svr:tladd(netbench)> mpirun -n 2 -f hosts >>>>> /global/usr/src/mpich-3.0.4/examples/cpi >>>>> Process 0 of 2 is on svr.che.ufl.edu >>>>> Fatal error in PMPI_Reduce: A process has failed, error stack: >>>>> PMPI_Reduce(1217)...............: MPI_Reduce(sbuf=0x7fff30ecced8, >>>>> rbuf=0x7fff30ecced0, count=1, MPI_DOUBLE, >>>>> >>>>> But if I reverse the order in the host file (pc5 first and then >>>>> svr) apparently both processes start >>>>> >>>>> svr:tladd(netbench)> mpirun -n 2 -f hosts >>>>> /global/usr/src/mpich-3.0.4/examples/cpi >>>>> Process 1 of 2 is on svr.che.ufl.edu >>>>> Process 0 of 2 is on pc5 >>>>> Fatal error in PMPI_Reduce: A process has failed, error stack: >>>>> PMPI_Reduce(1217)...............: MPI_Reduce(sbuf=0x7fff4d776348, >>>>> rbuf=0x7fff4d776340, count=1, MPI_DOUBLE, >>>>> >>>>> But with the same result in the end. >>>>> >>>>> Tony >>>>> >>>>> >>>>> >>>>> On 03/26/2014 08:18 PM, Rajeev Thakur wrote: >>>>>> Does the cpi example run across two machines? >>>>>> >>>>>> Rajeev >>>>>> >>>>>> On Mar 26, 2014, at 7:13 PM, Tony Ladd <tladd(a)che.ufl.edu> >>>>>> wrote: >>>>>> >>>>>>> Rajeev >>>>>>> >>>>>>> Sorry about that. I was switching back and forth from openmpi to >>>>>>> mpich. But it does not make a difference. Here is a clean log >>>>>>> from a fresh terminal - no mention of openmpi >>>>>>> >>>>>>> Tony >>>>>>> >>>>>>> PS - its a CentOS 6.5install - should have mentioned it before. >>>>>>> >>>>>>> -- >>>>>>> Tony Ladd >>>>>>> >>>>>>> Chemical Engineering Department >>>>>>> University of Florida >>>>>>> Gainesville, Florida 32611-6005 >>>>>>> USA >>>>>>> >>>>>>> Email: tladd-"(AT)"-che.ufl.edu >>>>>>> Web http://ladd.che.ufl.edu >>>>>>> >>>>>>> Tel: (352)-392-6509 >>>>>>> FAX: (352)-392-9514 >>>>>>> >>>>>>> <mpich.log>_______________________________________________ >>>>>>> discuss mailing list discuss(a)mpich.org >>>>>>> To manage subscription options or unsubscribe: >>>>>>> https://lists.mpich.org/mailman/listinfo/discuss >>>>>> _______________________________________________ >>>>>> discuss mailing list discuss(a)mpich.org >>>>>> To manage subscription options or unsubscribe: >>>>>> https://lists.mpich.org/mailman/listinfo/discuss >>>>> -- >>>>> Tony Ladd >>>>> >>>>> Chemical Engineering Department >>>>> University of Florida >>>>> Gainesville, Florida 32611-6005 >>>>> USA >>>>> >>>>> Email: tladd-"(AT)"-che.ufl.edu >>>>> Web http://ladd.che.ufl.edu >>>>> >>>>> Tel: (352)-392-6509 >>>>> FAX: (352)-392-9514 >>>>> >>>>> _______________________________________________ >>>>> discuss mailing list discuss(a)mpich.org >>>>> To manage subscription options or unsubscribe: >>>>> https://lists.mpich.org/mailman/listinfo/discuss >>>> _______________________________________________ >>>> discuss mailing list discuss(a)mpich.org >>>> To manage subscription options or unsubscribe: >>>> https://lists.mpich.org/mailman/listinfo/discuss >>> -- >>> Tony Ladd >>> >>> Chemical Engineering Department >>> University of Florida >>> Gainesville, Florida 32611-6005 >>> USA >>> >>> Email: tladd-"(AT)"-che.ufl.edu >>> Web http://ladd.che.ufl.edu >>> >>> Tel: (352)-392-6509 >>> FAX: (352)-392-9514 >>> >>> _______________________________________________ >>> discuss mailing list discuss(a)mpich.org >>> To manage subscription options or unsubscribe: >>> https://lists.mpich.org/mailman/listinfo/discuss >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss > > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss -- Tony Ladd Chemical Engineering Department University of Florida Gainesville, Florida 32611-6005 USA Email: tladd-"(AT)"-che.ufl.edu Web http://ladd.che.ufl.edu Tel: (352)-392-6509 FAX: (352)-392-9514

1 0

Re: [mpich-discuss] Fatal error in PMPI_Barrier: A process has failed, error stack:
by Tony Ladd 27 Mar '14

27 Mar '14

Pavan Same OS - but it was different hardware. So I tried the cpi example on two identical nodes (Dell Optiplex 745) this morning. The OS is Centos 6.5 and the installation on these client nodes is entirely automated so I am sure the configurations on the two boxes are identical (the install is new and the boxes have not been used so far). I used the version of cpi compiled during the installation of mpich. Here is the log file. I am also including the installation logs in case that helps - I have separate logs of the configure, make, and install stages. Tony On 03/27/2014 02:02 AM, Balaji, Pavan wrote: > Are both the nodes similar in architecture and OS configuration? > > Are the /etc/hosts files on both machines consistent? > > ï¿½ Pavan > > On Mar 26, 2014, at 9:01 PM, Tony Ladd <tladd(a)che.ufl.edu> wrote: > >> Rajeev >> >> There is a firewall on svr but its configured to accept all packets on the interface connected to the internal domain (where pc5 lives). I had already checked that stopping iptables off made no difference, but I just tried it again on the cpi example. The result was the same. >> >> Tony >> >> >> On 03/26/2014 09:45 PM, Rajeev Thakur wrote: >>> Is there a firewall on either machine that is in the way of communication? >>> >>> Rajeev >>> >>> On Mar 26, 2014, at 8:28 PM, Tony Ladd <tladd(a)che.ufl.edu> >>> wrote: >>> >>>> No - you get the same error - it looks as if process 1 (on the remote node) is not starting >>>> >>>> svr:tladd(netbench)> mpirun -n 2 -f hosts /global/usr/src/mpich-3.0.4/examples/cpi >>>> Process 0 of 2 is on svr.che.ufl.edu >>>> Fatal error in PMPI_Reduce: A process has failed, error stack: >>>> PMPI_Reduce(1217)...............: MPI_Reduce(sbuf=0x7fff30ecced8, rbuf=0x7fff30ecced0, count=1, MPI_DOUBLE, >>>> >>>> But if I reverse the order in the host file (pc5 first and then svr) apparently both processes start >>>> >>>> svr:tladd(netbench)> mpirun -n 2 -f hosts /global/usr/src/mpich-3.0.4/examples/cpi >>>> Process 1 of 2 is on svr.che.ufl.edu >>>> Process 0 of 2 is on pc5 >>>> Fatal error in PMPI_Reduce: A process has failed, error stack: >>>> PMPI_Reduce(1217)...............: MPI_Reduce(sbuf=0x7fff4d776348, rbuf=0x7fff4d776340, count=1, MPI_DOUBLE, >>>> >>>> But with the same result in the end. >>>> >>>> Tony >>>> >>>> >>>> >>>> On 03/26/2014 08:18 PM, Rajeev Thakur wrote: >>>>> Does the cpi example run across two machines? >>>>> >>>>> Rajeev >>>>> >>>>> On Mar 26, 2014, at 7:13 PM, Tony Ladd <tladd(a)che.ufl.edu> >>>>> wrote: >>>>> >>>>>> Rajeev >>>>>> >>>>>> Sorry about that. I was switching back and forth from openmpi to mpich. But it does not make a difference. Here is a clean log from a fresh terminal - no mention of openmpi >>>>>> >>>>>> Tony >>>>>> >>>>>> PS - its a CentOS 6.5install - should have mentioned it before. >>>>>> >>>>>> -- >>>>>> Tony Ladd >>>>>> >>>>>> Chemical Engineering Department >>>>>> University of Florida >>>>>> Gainesville, Florida 32611-6005 >>>>>> USA >>>>>> >>>>>> Email: tladd-"(AT)"-che.ufl.edu >>>>>> Web http://ladd.che.ufl.edu >>>>>> >>>>>> Tel: (352)-392-6509 >>>>>> FAX: (352)-392-9514 >>>>>> >>>>>> <mpich.log>_______________________________________________ >>>>>> discuss mailing list discuss(a)mpich.org >>>>>> To manage subscription options or unsubscribe: >>>>>> https://lists.mpich.org/mailman/listinfo/discuss >>>>> _______________________________________________ >>>>> discuss mailing list discuss(a)mpich.org >>>>> To manage subscription options or unsubscribe: >>>>> https://lists.mpich.org/mailman/listinfo/discuss >>>> -- >>>> Tony Ladd >>>> >>>> Chemical Engineering Department >>>> University of Florida >>>> Gainesville, Florida 32611-6005 >>>> USA >>>> >>>> Email: tladd-"(AT)"-che.ufl.edu >>>> Web http://ladd.che.ufl.edu >>>> >>>> Tel: (352)-392-6509 >>>> FAX: (352)-392-9514 >>>> >>>> _______________________________________________ >>>> discuss mailing list discuss(a)mpich.org >>>> To manage subscription options or unsubscribe: >>>> https://lists.mpich.org/mailman/listinfo/discuss >>> _______________________________________________ >>> discuss mailing list discuss(a)mpich.org >>> To manage subscription options or unsubscribe: >>> https://lists.mpich.org/mailman/listinfo/discuss >> -- >> Tony Ladd >> >> Chemical Engineering Department >> University of Florida >> Gainesville, Florida 32611-6005 >> USA >> >> Email: tladd-"(AT)"-che.ufl.edu >> Web http://ladd.che.ufl.edu >> >> Tel: (352)-392-6509 >> FAX: (352)-392-9514 >> >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss -- Tony Ladd Chemical Engineering Department University of Florida Gainesville, Florida 32611-6005 USA Email: tladd-"(AT)"-che.ufl.edu Web http://ladd.che.ufl.edu Tel: (352)-392-6509 FAX: (352)-392-9514

1 0

Re: [mpich-discuss] MPICH in Debian Jessie
by Lu, Huiwei 27 Mar '14

27 Mar '14

Hi, Mark, Installing MPICH from apt is encouraged for MPICH users on Debian. Only if that fails, you then try to compile from source. You can learn more of MPICH at http://www.mpich.org/documentation/guides/. There are a bunch of tutorials and documents at that link. — Huiwei Lu Postdoc Appointee Mathematics and Computer Science Division, Argonne National Laboratory http://www.mcs.anl.gov/~huiweilu/ On Mar 26, 2014, at 9:58 AM, mark <dimitsas.markos(a)gmail.com> wrote: > Hello. I have to imlpement some clustering algorithms for a bachelor essay and i was wondering if the Debian's MPICH package [1] will be enough for this, or will i need to compile MPICH from the source in mpich.org . Also i am new to MPICH and i would like some starting points and tutorials that will guide me with ease into algorithmic impementations (parallel and linear) and mpi programming and functions. Thank you ! > > [1] https://packages.debian.org/jessie/mpich . > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Fatal error in PMPI_Barrier: A process has failed, error stack:
by Balaji, Pavan 27 Mar '14

27 Mar '14

Are both the nodes similar in architecture and OS configuration? Are the /etc/hosts files on both machines consistent? — Pavan On Mar 26, 2014, at 9:01 PM, Tony Ladd <tladd(a)che.ufl.edu> wrote: > Rajeev > > There is a firewall on svr but its configured to accept all packets on the interface connected to the internal domain (where pc5 lives). I had already checked that stopping iptables off made no difference, but I just tried it again on the cpi example. The result was the same. > > Tony > > > On 03/26/2014 09:45 PM, Rajeev Thakur wrote: >> Is there a firewall on either machine that is in the way of communication? >> >> Rajeev >> >> On Mar 26, 2014, at 8:28 PM, Tony Ladd <tladd(a)che.ufl.edu> >> wrote: >> >>> No - you get the same error - it looks as if process 1 (on the remote node) is not starting >>> >>> svr:tladd(netbench)> mpirun -n 2 -f hosts /global/usr/src/mpich-3.0.4/examples/cpi >>> Process 0 of 2 is on svr.che.ufl.edu >>> Fatal error in PMPI_Reduce: A process has failed, error stack: >>> PMPI_Reduce(1217)...............: MPI_Reduce(sbuf=0x7fff30ecced8, rbuf=0x7fff30ecced0, count=1, MPI_DOUBLE, >>> >>> But if I reverse the order in the host file (pc5 first and then svr) apparently both processes start >>> >>> svr:tladd(netbench)> mpirun -n 2 -f hosts /global/usr/src/mpich-3.0.4/examples/cpi >>> Process 1 of 2 is on svr.che.ufl.edu >>> Process 0 of 2 is on pc5 >>> Fatal error in PMPI_Reduce: A process has failed, error stack: >>> PMPI_Reduce(1217)...............: MPI_Reduce(sbuf=0x7fff4d776348, rbuf=0x7fff4d776340, count=1, MPI_DOUBLE, >>> >>> But with the same result in the end. >>> >>> Tony >>> >>> >>> >>> On 03/26/2014 08:18 PM, Rajeev Thakur wrote: >>>> Does the cpi example run across two machines? >>>> >>>> Rajeev >>>> >>>> On Mar 26, 2014, at 7:13 PM, Tony Ladd <tladd(a)che.ufl.edu> >>>> wrote: >>>> >>>>> Rajeev >>>>> >>>>> Sorry about that. I was switching back and forth from openmpi to mpich. But it does not make a difference. Here is a clean log from a fresh terminal - no mention of openmpi >>>>> >>>>> Tony >>>>> >>>>> PS - its a CentOS 6.5install - should have mentioned it before. >>>>> >>>>> -- >>>>> Tony Ladd >>>>> >>>>> Chemical Engineering Department >>>>> University of Florida >>>>> Gainesville, Florida 32611-6005 >>>>> USA >>>>> >>>>> Email: tladd-"(AT)"-che.ufl.edu >>>>> Web http://ladd.che.ufl.edu >>>>> >>>>> Tel: (352)-392-6509 >>>>> FAX: (352)-392-9514 >>>>> >>>>> <mpich.log>_______________________________________________ >>>>> discuss mailing list discuss(a)mpich.org >>>>> To manage subscription options or unsubscribe: >>>>> https://lists.mpich.org/mailman/listinfo/discuss >>>> _______________________________________________ >>>> discuss mailing list discuss(a)mpich.org >>>> To manage subscription options or unsubscribe: >>>> https://lists.mpich.org/mailman/listinfo/discuss >>> -- >>> Tony Ladd >>> >>> Chemical Engineering Department >>> University of Florida >>> Gainesville, Florida 32611-6005 >>> USA >>> >>> Email: tladd-"(AT)"-che.ufl.edu >>> Web http://ladd.che.ufl.edu >>> >>> Tel: (352)-392-6509 >>> FAX: (352)-392-9514 >>> >>> _______________________________________________ >>> discuss mailing list discuss(a)mpich.org >>> To manage subscription options or unsubscribe: >>> https://lists.mpich.org/mailman/listinfo/discuss >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss > > -- > Tony Ladd > > Chemical Engineering Department > University of Florida > Gainesville, Florida 32611-6005 > USA > > Email: tladd-"(AT)"-che.ufl.edu > Web http://ladd.che.ufl.edu > > Tel: (352)-392-6509 > FAX: (352)-392-9514 > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Fatal error in PMPI_Barrier: A process has failed, error stack:
by Tony Ladd 26 Mar '14

26 Mar '14

Rajeev There is a firewall on svr but its configured to accept all packets on the interface connected to the internal domain (where pc5 lives). I had already checked that stopping iptables off made no difference, but I just tried it again on the cpi example. The result was the same. Tony On 03/26/2014 09:45 PM, Rajeev Thakur wrote: > Is there a firewall on either machine that is in the way of communication? > > Rajeev > > On Mar 26, 2014, at 8:28 PM, Tony Ladd <tladd(a)che.ufl.edu> > wrote: > >> No - you get the same error - it looks as if process 1 (on the remote node) is not starting >> >> svr:tladd(netbench)> mpirun -n 2 -f hosts /global/usr/src/mpich-3.0.4/examples/cpi >> Process 0 of 2 is on svr.che.ufl.edu >> Fatal error in PMPI_Reduce: A process has failed, error stack: >> PMPI_Reduce(1217)...............: MPI_Reduce(sbuf=0x7fff30ecced8, rbuf=0x7fff30ecced0, count=1, MPI_DOUBLE, >> >> But if I reverse the order in the host file (pc5 first and then svr) apparently both processes start >> >> svr:tladd(netbench)> mpirun -n 2 -f hosts /global/usr/src/mpich-3.0.4/examples/cpi >> Process 1 of 2 is on svr.che.ufl.edu >> Process 0 of 2 is on pc5 >> Fatal error in PMPI_Reduce: A process has failed, error stack: >> PMPI_Reduce(1217)...............: MPI_Reduce(sbuf=0x7fff4d776348, rbuf=0x7fff4d776340, count=1, MPI_DOUBLE, >> >> But with the same result in the end. >> >> Tony >> >> >> >> On 03/26/2014 08:18 PM, Rajeev Thakur wrote: >>> Does the cpi example run across two machines? >>> >>> Rajeev >>> >>> On Mar 26, 2014, at 7:13 PM, Tony Ladd <tladd(a)che.ufl.edu> >>> wrote: >>> >>>> Rajeev >>>> >>>> Sorry about that. I was switching back and forth from openmpi to mpich. But it does not make a difference. Here is a clean log from a fresh terminal - no mention of openmpi >>>> >>>> Tony >>>> >>>> PS - its a CentOS 6.5install - should have mentioned it before. >>>> >>>> -- >>>> Tony Ladd >>>> >>>> Chemical Engineering Department >>>> University of Florida >>>> Gainesville, Florida 32611-6005 >>>> USA >>>> >>>> Email: tladd-"(AT)"-che.ufl.edu >>>> Web http://ladd.che.ufl.edu >>>> >>>> Tel: (352)-392-6509 >>>> FAX: (352)-392-9514 >>>> >>>> <mpich.log>_______________________________________________ >>>> discuss mailing list discuss(a)mpich.org >>>> To manage subscription options or unsubscribe: >>>> https://lists.mpich.org/mailman/listinfo/discuss >>> _______________________________________________ >>> discuss mailing list discuss(a)mpich.org >>> To manage subscription options or unsubscribe: >>> https://lists.mpich.org/mailman/listinfo/discuss >> -- >> Tony Ladd >> >> Chemical Engineering Department >> University of Florida >> Gainesville, Florida 32611-6005 >> USA >> >> Email: tladd-"(AT)"-che.ufl.edu >> Web http://ladd.che.ufl.edu >> >> Tel: (352)-392-6509 >> FAX: (352)-392-9514 >> >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss -- Tony Ladd Chemical Engineering Department University of Florida Gainesville, Florida 32611-6005 USA Email: tladd-"(AT)"-che.ufl.edu Web http://ladd.che.ufl.edu Tel: (352)-392-6509 FAX: (352)-392-9514

1 0

Re: [mpich-discuss] Fatal error in PMPI_Barrier: A process has failed, error stack:
by Rajeev Thakur 26 Mar '14

26 Mar '14

Is there a firewall on either machine that is in the way of communication? Rajeev On Mar 26, 2014, at 8:28 PM, Tony Ladd <tladd(a)che.ufl.edu> wrote: > No - you get the same error - it looks as if process 1 (on the remote node) is not starting > > svr:tladd(netbench)> mpirun -n 2 -f hosts /global/usr/src/mpich-3.0.4/examples/cpi > Process 0 of 2 is on svr.che.ufl.edu > Fatal error in PMPI_Reduce: A process has failed, error stack: > PMPI_Reduce(1217)...............: MPI_Reduce(sbuf=0x7fff30ecced8, rbuf=0x7fff30ecced0, count=1, MPI_DOUBLE, > > But if I reverse the order in the host file (pc5 first and then svr) apparently both processes start > > svr:tladd(netbench)> mpirun -n 2 -f hosts /global/usr/src/mpich-3.0.4/examples/cpi > Process 1 of 2 is on svr.che.ufl.edu > Process 0 of 2 is on pc5 > Fatal error in PMPI_Reduce: A process has failed, error stack: > PMPI_Reduce(1217)...............: MPI_Reduce(sbuf=0x7fff4d776348, rbuf=0x7fff4d776340, count=1, MPI_DOUBLE, > > But with the same result in the end. > > Tony > > > > On 03/26/2014 08:18 PM, Rajeev Thakur wrote: >> Does the cpi example run across two machines? >> >> Rajeev >> >> On Mar 26, 2014, at 7:13 PM, Tony Ladd <tladd(a)che.ufl.edu> >> wrote: >> >>> Rajeev >>> >>> Sorry about that. I was switching back and forth from openmpi to mpich. But it does not make a difference. Here is a clean log from a fresh terminal - no mention of openmpi >>> >>> Tony >>> >>> PS - its a CentOS 6.5install - should have mentioned it before. >>> >>> -- >>> Tony Ladd >>> >>> Chemical Engineering Department >>> University of Florida >>> Gainesville, Florida 32611-6005 >>> USA >>> >>> Email: tladd-"(AT)"-che.ufl.edu >>> Web http://ladd.che.ufl.edu >>> >>> Tel: (352)-392-6509 >>> FAX: (352)-392-9514 >>> >>> <mpich.log>_______________________________________________ >>> discuss mailing list discuss(a)mpich.org >>> To manage subscription options or unsubscribe: >>> https://lists.mpich.org/mailman/listinfo/discuss >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss > > -- > Tony Ladd > > Chemical Engineering Department > University of Florida > Gainesville, Florida 32611-6005 > USA > > Email: tladd-"(AT)"-che.ufl.edu > Web http://ladd.che.ufl.edu > > Tel: (352)-392-6509 > FAX: (352)-392-9514 > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Fatal error in PMPI_Barrier: A process has failed, error stack:
by Tony Ladd 26 Mar '14

26 Mar '14

No - you get the same error - it looks as if process 1 (on the remote node) is not starting svr:tladd(netbench)> mpirun -n 2 -f hosts /global/usr/src/mpich-3.0.4/examples/cpi Process 0 of 2 is on svr.che.ufl.edu Fatal error in PMPI_Reduce: A process has failed, error stack: PMPI_Reduce(1217)...............: MPI_Reduce(sbuf=0x7fff30ecced8, rbuf=0x7fff30ecced0, count=1, MPI_DOUBLE, But if I reverse the order in the host file (pc5 first and then svr) apparently both processes start svr:tladd(netbench)> mpirun -n 2 -f hosts /global/usr/src/mpich-3.0.4/examples/cpi Process 1 of 2 is on svr.che.ufl.edu Process 0 of 2 is on pc5 Fatal error in PMPI_Reduce: A process has failed, error stack: PMPI_Reduce(1217)...............: MPI_Reduce(sbuf=0x7fff4d776348, rbuf=0x7fff4d776340, count=1, MPI_DOUBLE, But with the same result in the end. Tony On 03/26/2014 08:18 PM, Rajeev Thakur wrote: > Does the cpi example run across two machines? > > Rajeev > > On Mar 26, 2014, at 7:13 PM, Tony Ladd <tladd(a)che.ufl.edu> > wrote: > >> Rajeev >> >> Sorry about that. I was switching back and forth from openmpi to mpich. But it does not make a difference. Here is a clean log from a fresh terminal - no mention of openmpi >> >> Tony >> >> PS - its a CentOS 6.5install - should have mentioned it before. >> >> -- >> Tony Ladd >> >> Chemical Engineering Department >> University of Florida >> Gainesville, Florida 32611-6005 >> USA >> >> Email: tladd-"(AT)"-che.ufl.edu >> Web http://ladd.che.ufl.edu >> >> Tel: (352)-392-6509 >> FAX: (352)-392-9514 >> >> <mpich.log>_______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss -- Tony Ladd Chemical Engineering Department University of Florida Gainesville, Florida 32611-6005 USA Email: tladd-"(AT)"-che.ufl.edu Web http://ladd.che.ufl.edu Tel: (352)-392-6509 FAX: (352)-392-9514

1 0