[Bug 1091] New: Channel timeouts cause file transfer failures
https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=1091 Summary: Channel timeouts cause file transfer failures Product: Swift Version: 0.94 Platform: PC OS/Version: Linux Status: NEW Severity: normal Priority: P2 Component: General AssignedTo: [email protected] ReportedBy: [email protected] I'm seeing channel timeouts again in the test stage_to_remoteNx2 to the MCS machines. The test connects to the MCS machines and starts 50 apps, each requiring staging in 10mb files as input over provider staging. The logs and all details for the test can be accessed from here : Run from Sep 08: http://swift.rcc.uchicago.edu:8042/swift-0.94.1/run-2013-09-08-020801/stage_... Run from Sep 09: http://swift.rcc.uchicago.edu:8042/swift-0.94.1/run-2013-09-09-003621/stage_... Here's a snippet of the error from the log: org.globus.cog.karajan.workflow.service.TimeoutException: Channel timed out. lastTime=130910-044024.801, now=130910-044225.104, channel=GSSChannel [type: service, contact: 11127766120, id: 3, context: service-55633] at org.globus.cog.karajan.workflow.service.channels.AbstractKarajanChannel.checkTimeouts(AbstractKarajanChannel.java:133) at org.globus.cog.karajan.workflow.service.channels.AbstractKarajanChannel$1.run(AbstractKarajanChannel.java:124) at java.util.TimerThread.mainLoop(Timer.java:555) at java.util.TimerThread.run(Timer.java:505) A similar issue was reported earlier Bug1006, which was not seen for a while now. -- Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=1091 David Kelly <[email protected]> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |[email protected] --- Comment #1 from David Kelly <[email protected]> 2013-09-10 14:21:25 --- Just noticed this while taking a look at the logs: 2013-09-09 04:37:37,243+0000 DEBUG JobSubmissionTaskHandler STDERR from job: cannot get sockname for fd ssh_keysign: no reply key_sign failed I don't remember seeing these errors before. I suspect whatever is causing this might be the reason it's timing out. -- Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=1091 Mihael Hategan <[email protected]> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |ASSIGNED --- Comment #2 from Mihael Hategan <[email protected]> 2013-09-16 22:13:10 --- Channel timeouts SHOULD cause transfer failures! If you get a valid channel timeout, it means you somehow lost the TCP connection. I am, however seeing something similar on beagle and I'm trying to troubleshoot it. -- Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=1091 --- Comment #3 from Mihael Hategan <[email protected]> 2013-09-17 18:10:13 --- I am seeing walltime exceeded messages in the PBS stderr files, so that is probably the cause for the timeouts. -- Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=1091 --- Comment #4 from Yadu Nand <[email protected]> 2013-09-17 19:04:08 --- These are from the tests on MCS machines, using ssh-cl:local. There is no PBS submission mechanism in this particular case. If you check the results from the same run on MCS but with lower stress, you'd see that they've passed: http://swift.rcc.uchicago.edu:8042/swift-0.94.1/run-2013-09-08-020801/tests-... stage_to_remoteNx1 uses 10 apps each moving one 10mb file (passes) stage_to_remoteNx2 uses 50 apps each moving one 10mb file (Fails) -- Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=1091 Mihael Hategan <[email protected]> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|ASSIGNED |RESOLVED Resolution| |FIXED --- Comment #5 from Mihael Hategan <[email protected]> 2013-09-20 23:01:29 --- What I ran into was a deadlock that would occur in the coaster network code, so that would cause the communication to stop and the channels to time out. That particular deadlock has now been fixed in trunk and 0.94. -- Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
participants (1)
-
bugzilla-daemon＠mcs.anl.gov