[Bug 1311] New: Waiting threads at end of run using cloud providers
https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=1311 Summary: Waiting threads at end of run using cloud providers Product: Swift Version: trunk Platform: PC OS/Version: Linux Status: NEW Severity: normal Priority: P2 Component: General AssignedTo: [email protected] ReportedBy: [email protected] When running tasks using cloud providers (aws/gce), I see tasks complete followed by a strack trace and list of waiting threads. This happens in almost half the runs. The timestamps from the cancel logs show that around 2s were spent for cancelling the nodes on the cloud. The script exits with exitcode 0 once the jobs are sent cancel requests. Here's the log : http://users.rcc.uchicago.edu/~yadunand/run023_waiting_threads.log Here's a debug log from the provider : http://users.rcc.uchicago.edu/~yadunand/run023_waiting_threads.debug -- Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=1311 --- Comment #1 from Yadu Nand <[email protected]> 2014-08-13 13:21:41 --- I'm seeing the behavior described above on Tukey with local:cobalt. Swift.log -> http://users.rcc.uchicago.edu/~yadunand/hanging_threads.log Run dir -> http://users.rcc.uchicago.edu/~yadunand/run016.tar.gz The application fails due to the summarize array of results, stage failing due to the app not being defined. Following the error is a large trace : ''No events in 1s. Finding dependency loops... Waiting threads: ---- No dependency loops found. Waiting JVM threads: Thread "Thread-11" (0x0000003b) TIMED_WAITING Stack trace: java.lang.Thread.sleep org.globus.cog.abstraction.coaster.service.CoasterService$1.run:373 Thread "I/O Queue" (0x00000053) TIMED_WAITING waiting for org.globus.cog.abstraction.impl.file.coaster.buffers.Buffers (0x3dc48444) Stack trace: java.lang.Object.wait org.globus.cog.abstraction.impl.file.coaster.buffers.Buffers.run:211 Thread "I/O Queue" (0x00000052) TIMED_WAITING waiting for org.globus.cog.abstraction.impl.file.coaster.buffers.Buffers (0x7cf60a7c) Stack trace: java.lang.Object.wait org.globus.cog.abstraction.impl.file.coaster.buffers.Buffers.run:211 -- Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=1311 Yadu Nand <[email protected]> changed: What |Removed |Added ---------------------------------------------------------------------------- Summary|Waiting threads at end of |Waiting threads at end of |run using cloud providers |run [cloud, cobalt] -- Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=1311 --- Comment #2 from Mihael Hategan <[email protected]> 2014-08-16 03:30:33 --- It looks like in both cases things hang here: java.lang.Object.wait java.lang.Object.wait:503 java.lang.UNIXProcess.waitFor:210 org.globus.cog.abstraction.impl.scheduler.common.AbstractExecutor.cancel:225 I can't reasonably see a possibility other than the cancel script failing to exit. What makes you say that the script exits with exit code 0? Did you run a ps to confirm that the script really isn't running? -- Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=1311 Mihael Hategan <[email protected]> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |ASSIGNED -- Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=1311 Mihael Hategan <[email protected]> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|ASSIGNED |WAITING FOR USER INPUT -- Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=1311 --- Comment #3 from Yadu Nand <[email protected]> 2014-09-03 18:00:47 --- In the case of the cloud provider for Google compute engine, the stcancel.sh scripts is taking around 30s to confirm the shutdown of the node. This probably explains the threads hanging. I'm trying to fix this from the python script which is taking most of the time. If that isn't possible, do you have any recommendations, Mihael ? -- Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
participants (1)
-
bugzilla-daemon＠mcs.anl.gov