[Bug 1377] New: Connection to workers lost/Channel timeout on OSG
https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=1377 Summary: Connection to workers lost/Channel timeout on OSG Product: Swift Version: 0.96 Platform: PC OS/Version: Linux Status: NEW Severity: normal Priority: P2 Component: General AssignedTo: [email protected] ReportedBy: [email protected] For long running jobs (workers are up for ~2hrs), I'm seeing several workers fail with channel timeouts, and the following error message in the swift.log: 2014-11-10 11:30:41,367-0600 DEBUG swift APPLICATION_EXCEPTION jobid=bash-1ip8c30m - Application exception: exception @ swift-int-staging.k, line: 160 Caused by: Block task failed: Connection to worker lost java.io.EOFException at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.readFromChannel(AbstractStreamCoasterChannel.java:253) at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:186) at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:116) at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:75) Of the 956 Condor*submit scripts, I see 31 end with Channeltimeouts in the stderr logs. The concern is that due to these worker failures, even with lazyErrors=true and executionRetries set to 2, the application ends up failing. I have not dug deep enough to understand if there is a bug in the worker code that is resulting in the failures, or if this is a reasonable level of reliability from OSG. (31/956 =~ 3%failure). Here are the logs : http://stash.osgconnect.net/+yadunand/run033.tar.gz http://stash.osgconnect.net/+yadunand/worker-logs.tar.gz -- Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=1377 --- Comment #1 from Yadu Nand <[email protected]> 2014-11-18 15:29:15 --- Here are more complete logs. In this run, 1000 tasks were launched of which 10 did not get completed even with 2 retries. There are 956 condor submit scripts and corresponding worker logs. http://stash.osgconnect.net/+yadunand/failing_run_osg.tar.gz I'm running a similar test, with smaller number of tasks on midway with trunk, and will report if I see failures there. -- Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=1377 Yadu Nand <[email protected]> changed: What |Removed |Added ---------------------------------------------------------------------------- Summary|Connection to workers |Connection to workers lost |lost/Channel timeout on OSG |on OSG --- Comment #2 from Yadu Nand <[email protected]> 2014-11-20 11:53:12 --- I have a test running on OSG, with a netcat server listening on nodes uc3-c001 to uc3-c022, the headnode starts a netcat client which connects and stays up, waiting to see if the connection breaks. The test has been running for ~20hours, and not one connection has broken. I ran the same swift test which has 200, 1hour +/- 10min tasks on Midway, and they completed with no errors at all. -- Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=1377 --- Comment #3 from Mihael Hategan <[email protected]> 2014-11-20 13:40:33 --- (In reply to comment #2)
I have a test running on OSG, with a netcat server listening on nodes uc3-c001 to uc3-c022, the headnode starts a netcat client which connects and stays up, waiting to see if the connection breaks. The test has been running for ~20hours, and not one connection has broken.
It might need some data flowing around. What is clear is that it does happen under certain circumstances. Here are a few more things that you can do: - a ping test to some of the nodes while swift is running - run the netcat server on the head node (to simulate better what swift is doing) and do an actual transfer.
I ran the same swift test which has 200, 1hour +/- 10min tasks on Midway, and they completed with no errors at all.
So this is some OSG issue. -- Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
participants (1)
-
bugzilla-daemon＠mcs.anl.gov