https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=907 Summary: Dangling coaster workers interfere with subsequent Swift runs Product: Swift Version: 0.94 Platform: PC OS/Version: Mac OS Status: ASSIGNED Severity: normal Priority: P2 Component: Providers AssignedTo: [email protected] ReportedBy: [email protected] CC: [email protected], [email protected], [email protected] Two problems here: - workers are not always cleaned up - such "dangling: workers interfere with subsequent runs Lorenzo has a log file (first message below) showing at least one manifestation of this problem. Second message indicates killing the abandoned workers solves the problems. Can we: - do a better job cleaning up abandoned workers? - make it so an abandoned worker cant connect to a subsequent server (or that if they do, no bad things, and only good things, happen?) --- ----- Forwarded Message ----- From: "Lorenzo Pesce" <[email protected]> To: "Michael Wilde" <[email protected]> Sent: Saturday, December 15, 2012 12:43:06 PM Subject: 500 nodes failed again failed 500 node run: lpesce@login3:/lustre/beagle/GCNet/RG/Oreo/o080522_BS1> pwd /lustre/beagle/GCNet/RG/Oreo/o080522_BS1 -rw-rw-r-- 1 lpesce ci-users 549729889 2012-12-15 12:40 PackAndCausalTestCore_app-20121215-1801-2d9bnr57.log -rw-rw-r-- 1 lpesce ci-users 569522 2012-12-15 12:40 PackAndCausalTestCore_app.swift_2012-12-15_12:01:12_screen.log drwxrwxr-x 2 lpesce ci-users 4096 2012-12-15 12:40 PackAndCausalTestCore_app-20121215-1801-2d9bnr57.d In case you are interested in doing some post mortem. ----- Forwarded Message ----- From: "Lorenzo Pesce" <[email protected]> To: "Michael Wilde" <[email protected]> Sent: Saturday, December 15, 2012 7:40:19 PM Subject: I know why it doesn't work sometimes I just realized that sometimes when swift gets killed or stops some jobs are left queued or running. These jobs create interference when the new jobs are sent. I tried a couple of times and if I clean up the queue, there seem to be no issues. I sent out a 500 nodes job and it has been happily churning for a while. -- Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.