[Bug 752] New: Coaster provider does not retry worker launches on qsub failures
https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=752 Summary: Coaster provider does not retry worker launches on qsub failures Product: Swift Version: 0.93 Platform: PC OS/Version: Mac OS Status: ASSIGNED Severity: normal Priority: P2 Component: Providers AssignedTo: [email protected] ReportedBy: [email protected] CC: [email protected], [email protected], [email protected] If a qsub to start a worker.pl job fails (in this case, due to the user having exceeded the maximum number of concurrently allowed job submissions, 32 on Eureka) then the coaster provider never tries again to start workers for those coaster slots. Hence, when the user is running multiple swift runs concurrently, and they fluctuate in the number of active workers, the swift run that encounters the worker failures stays forever at a very low active worker count. In the current situation, it stays so low (eg 1 worker out of an expected 32) that the run needs to be killed and restarted in order to get a sufficient number of workers. Logs are on eureka in /home/wilde/LZ/runNNN where NNN is between 020 and 058. -- Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=752 Mihael Hategan <[email protected]> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|ASSIGNED |WAITING FOR USER INPUT --- Comment #1 from Mihael Hategan <[email protected]> 2014-10-18 18:28:26 --- I'm not entirely sure I understand what is being described here. However, here is my take on what I do understand: 1. worker jobs not being retried is by design. It is the design part that says that any error, even transient, should make its way back to the client, and if the client (swift in this case) wants to re-try a job, it can do so. If the error was transient, a subsequent try will eventually succeed. The point is that, like many other job submission services, coasters is not fault-tolerant. Swift handles that part, and it does not need to be duplicated in coasters. 2. As long as there are more jobs than workers available to run them, the coaster service should try to start more workers. Is this not happening in this case? -- Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
participants (1)
-
bugzilla-daemon＠mcs.anl.gov