[Bug 1267] New: Channel Timeouts with provider staging
https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=1267 Summary: Channel Timeouts with provider staging Product: Swift Version: 0.95 Platform: PC OS/Version: Mac OS Status: NEW Severity: normal Priority: P2 Component: General AssignedTo: [email protected] ReportedBy: [email protected] Just ran some of the provider staging tests and ran into some errors. The failing test I ran was tests/stress/IO/bagOnodes/stage_to_remoteNx2 (unmodified). The tests were running on Midway and staging data to/from MCS. It seems things are more likely to fail as the number of files and the size of the files increase. Full log at http://web.ci.uchicago.edu/~davidk/logs/channel_timeouts.log Execution failed: Exception in bash: Arguments: [wrapper.sh, dummy] Host: steamroller Directory: stage_to_remoteNx2-run001/jobs/e/bash-ekdq0xpl exception @ swift-int-staging.k, line: 181 Caused by: exception @ swift-int-staging.k, line: 177 Caused by: Error staging in file: Channel timed out. lastTime=140429-042946.162, now=140429-043146.286, channel=GSSChannel [type: client, contact: https://128.135.112.73:50004, id: 1, context: https://128.135.112.73:50004] org.globus.cog.abstraction.impl.common.execution.JobException: Error staging in file: Channel timed out. lastTime=140429-042946.162, now=140429-043146.286, channel=GSSChannel [type: client, contact: https://128.135.112.73:50004, id: 1, context: https://128.135.112.73:50004] (exit code: 520) at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:40) at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88) at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527) at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238) at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97) at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56) Caused by: org.globus.cog.abstraction.impl.common.execution.JobException: Error staging in file: Channel timed out. lastTime=140429-042946.162, now=140429-043146.286, channel=GSSChannel [type: client, contact: https://128.135.112.73:50004, id: 1, context: https://128.135.112.73:50004] org.globus.cog.abstraction.impl.common.execution.JobException: Error staging in file: Channel timed out. lastTime=140429-042946.162, now=140429-043146.286, channel=GSSChannel [type: client, contact: https://128.135.112.73:50004, id: 1, context: https://128.135.112.73:50004] (exit code: 520) at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:40) at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88) at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527) at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238) at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97) at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56) (exit code: 520) Progress: Tue, 29 Apr 2014 09:31:46+0000 Selecting site:11 Stage in:88 Failed:2 -- Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=1267 Mihael Hategan <[email protected]> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |ASSIGNED --- Comment #1 from Mihael Hategan <[email protected]> 2014-11-01 21:09:26 --- I suspect a deadlock somewhere, since it's the client-service connection that times out. However, I can't tell exactly if that's the case. If you see this again, can you get jstack dumps of the client and service please? -- Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=1267 Mihael Hategan <[email protected]> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|ASSIGNED |WAITING FOR USER INPUT -- Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
participants (1)
-
bugzilla-daemon@mcs.anl.gov