[Bug 869] New: Enable multiple local coaster provider pools
https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=869 Summary: Enable multiple local coaster provider pools Product: Swift Version: trunk Platform: PC OS/Version: Mac OS Status: ASSIGNED Severity: normal Priority: P2 Component: Providers AssignedTo: [email protected] ReportedBy: [email protected] CC: [email protected], [email protected], [email protected] For mixed MPI-serial scripts, we need to have at least two pools on the local cluster (eg, midway): one for serial jobs, and one for MPI jobs. This does not seem to work for automatic coasters. There were 3 cases of interest here: Case 1: The preferred config here is case 1: to use two pools which both specify local:slurm. This case failed: it seemed that different app jobs, differentiated by tc entry, were sent indiscriminately to the wrong pool (ie they got the wrong maxtime entries etc). Case 2: Next I tried using two pools with ssh-cl:slurm, hoping that Swift would start two separate external coaster service processes. This failed with complaints that required ports were already bound. I think it also demanded that a valid proxy was in place. This solution would be fine, and quite transparent to the user, if it works without requiring the user to create an X509 proxy. Case 3: I think I finally got a third config working with two manually started coaster services, a valid proxy, and then automatic worker launch did seem to work. This would have been acceptable (but not so great for the user) if it didnt require the proxy. I will try to replicate and document the failing test cases. Mihael, it would be great if you can make cases 1 or 2 work. Or provide a simple way for the user to create the proxy needed for case 3. -- Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=869 --- Comment #1 from Mihael Hategan <[email protected]> 2012-11-18 17:12:28 --- (In reply to comment #0)
Case 1: The preferred config here is case 1: to use two pools which both specify local:slurm. This case failed: it seemed that different app jobs, differentiated by tc entry, were sent indiscriminately to the wrong pool (ie they got the wrong maxtime entries etc).
This is because coaster services are identified based on their URL, so there was really only one coaster service. Combine that with coaster settings being only sent once and you see the results you have. I suggested using two different urls, such as: <execution provider="coaster" jobmanager="local:local" url="localhost:1" /> <execution provider="coaster" jobmanager="local:local" url="localhost:2" /> But that resulted in further errors. Those errors are now fixed in trunk (cog r3515) and you should be able to get this going without the need for 2 or 3 below.
Case 2: Next I tried using two pools with ssh-cl:slurm, hoping that Swift would start two separate external coaster service processes. This failed with complaints that required ports were already bound. I think it also demanded that a valid proxy was in place. This solution would be fine, and quite transparent to the user, if it works without requiring the user to create an X509 proxy.
I will have to look into this. The port assignment may not work very well with a port range specified and two jvms competing for the same port range.
Case 3: I think I finally got a third config working with two manually started coaster services, a valid proxy, and then automatic worker launch did seem to work. This would have been acceptable (but not so great for the user) if it didnt require the proxy.
-nosec?
I will try to replicate and document the failing test cases. Mihael, it would be great if you can make cases 1 or 2 work. Or provide a simple way for the user to create the proxy needed for case 3.
-- Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=869 --- Comment #2 from Michael Wilde <[email protected]> 2012-11-20 20:52:51 --- Regarding case 3: I *did* use -nosec; thats why I was surprised to get a security complaint from Swift. -- Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=869 Mihael Hategan <[email protected]> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|ASSIGNED |RESOLVED Resolution| |FIXED --- Comment #3 from Mihael Hategan <[email protected]> 2014-07-12 16:43:39 --- Multiple coaster pools should now be supported in trunk even without the url trick. -- Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
participants (1)
-
bugzilla-daemon＠mcs.anl.gov