[Bug 684] New: Feature request: -nosec option for SSH with automatic coasters
https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=684 Summary: Feature request: -nosec option for SSH with automatic coasters Product: Swift Version: trunk Platform: PC OS/Version: Linux Status: NEW Severity: enhancement Priority: Medium Component: Providers AssignedTo: [email protected] ReportedBy: [email protected] CC: [email protected], [email protected], [email protected] This may be related to the changes slated for SSH in 0.94. Currently, when we use the persistent coaster service, we can provide a -nosec option which disables the requirement for security certificates. Could we have a setting that also disables the need for certificates with ssh and automatic coasters? -- Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=684 David Kelly <[email protected]> changed: What |Removed |Added ---------------------------------------------------------------------------- Priority|Medium |P1 Status|NEW |ASSIGNED --- Comment #1 from David Kelly <[email protected]> 2012-07-26 17:54:37 --- This would be very useful to have for some things we are doing now with CMTS. -- Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=684 Mihael Hategan <[email protected]> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|ASSIGNED |RESOLVED Resolution| |FIXED --- Comment #2 from Mihael Hategan <[email protected]> 2012-08-07 00:05:47 --- An alternative solution (an automatic CA) is now in trunk. -- Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=684 --- Comment #3 from David Kelly <[email protected]> 2012-08-07 12:07:29 --- Should it work with ssh-cl too? I'm trying to test this with a setup we created for CMTS that uses ssh-cl:pbs. This is the error I get: Caused by: org.globus.cog.karajan.workflow.service.channels.ChannelException: org.globus.cog.abstraction.impl.common.task.InvalidSecurityContextException: GSSException: Defective credential detected [Caused by: [JGLOBUS-5] Proxy file (/tmp/x509up_u1310) not found.] at org.globus.cog.karajan.workflow.service.ChannelFactory.newChannel(ChannelFactory.java:59) at org.globus.cog.karajan.workflow.service.Client.connect(Client.java:116) at org.globus.cog.karajan.workflow.service.Client.newClient(Client.java:72) at org.globus.cog.karajan.workflow.service.channels.ChannelManager.connect(ChannelManager.java:236) at org.globus.cog.karajan.workflow.service.channels.ChannelManager.reserveChannel(ChannelManager.java:256) at org.globus.cog.karajan.workflow.service.channels.ChannelManager.reserveChannel(ChannelManager.java:211) at org.globus.cog.abstraction.coaster.service.CoasterService.start(CoasterService.java:163) ... 1 more The sites.xml: <config> <pool handle="makena-ssh"> <execution provider="coaster" url="makena.uchicago.edu" jobmanager="ssh-cl:pbs"/> <filesystem provider="local" url="none" /> <profile namespace="globus" key="HighOverAllocation">100</profile> <profile namespace="globus" key="LowOverAllocation">100</profile> <profile namespace="globus" key="maxWallTime">00:50:00</profile> <profile namespace="globus" key="maxTime">3600</profile> <profile key="jobsPerNode" namespace="globus">1</profile> <profile key="nodeGranularity" namespace="globus">1</profile> <profile key="maxNodes" namespace="globus">2</profile> <profile key="queue" namespace="globus">devel</profile> <profile key="jobThrottle" namespace="karajan">1.01</profile> <profile key="initialScore" namespace="karajan">10000</profile> <profile key="count" namespace="globus">2</profile> <profile key="jobType" namespace="globus">single</profile> <profile key="ppn" namespace="globus">8</profile> <workdirectory>/home/davidk/rmsd/run042/swiftwork</workdirectory> </pool> </config> The full logs are in /home/davidk/rmsd/run039 on CI. -- Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=684 --- Comment #4 from Mihael Hategan <[email protected]> 2012-08-07 14:10:01 --- This does not work with ssh-cl (because ssh-cl cannot fake GSI delegation). I can take a look and see if I can convince it to. -- Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=684 --- Comment #5 from David Kelly <[email protected]> 2012-08-07 14:30:43 --- Cool, thanks, I think ssh-cl would be useful to have if it's relatively easy to do. I tried testing with the regular ssh provider, and seem to be getting this: Caused by: org.globus.cog.abstraction.impl.common.task.TaskSubmissionException: Could not submit job Caused by: org.globus.cog.abstraction.impl.common.task.TaskSubmissionException: Could not start coaster service Caused by: org.globus.cog.abstraction.impl.common.task.InvalidSecurityContextException: Failed to find root CA certificate (C=US,O=JavaCoG,OU=AutoCA,CN=Certificate Authority) Execution failed: Failed to find root CA certificate (C=US,O=JavaCoG,OU=AutoCA,CN=Certificate Authority) (/home/davidk/rmsd/run044) Is there something else I need? -- Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=684 Mihael Hategan <[email protected]> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|RESOLVED |REOPENED Resolution|FIXED | --- Comment #6 from Mihael Hategan <[email protected]> 2012-08-07 16:19:54 --- You shouldn't need anything else. Can I have the log and the output of 'ls -alR ~/.globus' on both client and remote sites? -- Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=684 --- Comment #7 from David Kelly <[email protected]> 2012-08-08 09:14:45 --- The ls output and swift log is at: http://www.ci.uchicago.edu/~davidk/logs/logs-ssh-20120808.tar.gz -- Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=684 --- Comment #8 from David Kelly <[email protected]> 2012-08-08 16:12:51 --- I think this may be causing some issues when using coaster/local:local. Without a proxy, it fails with a message about a missing proxy. http://www.ci.uchicago.edu/~davidk/logs/serialforeach-20120808-1557-c7kl6jbb... With a proxy, I see a message that says "Discarding connection" and seems to make no progress. http://www.ci.uchicago.edu/~davidk/logs/serialforeach-20120808-1605-thp0au3e... -- Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=684 --- Comment #9 from Mihael Hategan <[email protected]> 2012-08-08 20:41:24 --- (In reply to comment #8)
I think this may be causing some issues when using coaster/local:local. Without a proxy, it fails with a message about a missing proxy.
http://www.ci.uchicago.edu/~davidk/logs/serialforeach-20120808-1557-c7kl6jbb...
The above should now be fixed.
With a proxy, I see a message that says "Discarding connection" and seems to make no progress.
http://www.ci.uchicago.edu/~davidk/logs/serialforeach-20120808-1605-thp0au3e...
That shouldn't be happening, but check with the last update and let me know if it's still there. As for the SSH root CA issue, I suspect it might be a problem with the way trusted certificates are loaded in jglobus. Can you update and run with the latest code? I added some code to print a list of trusted CAs so we can confirm if that's the case. -- Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=684 --- Comment #10 from David Kelly <[email protected]> 2012-08-09 01:41:19 --- The issue with coasters/local:local seem to be fine now. The new logs are at: http://www.ci.uchicago.edu/~davidk/logs/logs-ssh-20120809.tar.gz -- Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=684 --- Comment #11 from Mihael Hategan <[email protected]> 2012-08-09 03:29:17 --- (In reply to comment #10)
The issue with coasters/local:local seem to be fine now.
The new logs are at: http://www.ci.uchicago.edu/~davidk/logs/logs-ssh-20120809.tar.gz
Sorry. CoGProperties ignores setProxyFile and setCaCertLocations if the respective system properties are set (and those are set if the env variables are set). The latest version sets the system properties instead. -- Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=684 --- Comment #12 from David Kelly <[email protected]> 2012-08-09 13:57:17 --- It looks like it's getting further - the error I am seeing now is: No signing policy file found http://www.ci.uchicago.edu/~davidk/logs/logs-ssh-20120809-2.tar.gz -- Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=684 --- Comment #13 from Mihael Hategan <[email protected]> 2012-08-09 14:03:12 --- (In reply to comment #12)
It looks like it's getting further - the error I am seeing now is: No signing policy file found
http://www.ci.uchicago.edu/~davidk/logs/logs-ssh-20120809-2.tar.gz
Ah, yes. Quick workaround: add the following line to ~/.globus/cog.properties (on both sites): java.security.gsi.signing.policy=false I'll try to add signing policies for the dummy CA. -- Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=684 --- Comment #14 from David Kelly <[email protected]> 2012-08-09 14:08:51 --- That worked - it ran 10 small jobs and everything completed fine. -- Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=684 --- Comment #15 from Mihael Hategan <[email protected]> 2012-08-11 11:04:46 --- I committed code to generate and forward signing policies. Things should now work with default setting (no disabling of signing policies in cog.properties). Please give it a shot. Might be a good idea to clean ~/.globus/coasters. -- Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=684 --- Comment #16 from David Kelly <[email protected]> 2012-09-10 15:20:06 --- Hello, Just curious if there are any updates on this for ssh-cl? This would be very useful for a CMTS script that I'm working on. Thanks, David -- Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=684 Mihael Hategan <[email protected]> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|REOPENED |RESOLVED Resolution| |FIXED --- Comment #17 from Mihael Hategan <[email protected]> 2012-09-20 20:03:02 --- SSH-CL now also has proxy forwarding and can use the automatic CA (cog trunk r3473). -- Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=684 David Kelly <[email protected]> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|RESOLVED |REOPENED Resolution|FIXED | --- Comment #18 from David Kelly <[email protected]> 2012-09-21 01:40:46 --- Thanks for committing this! It seems to work well when using a single site. When I try to use multiple pools/sites together I run into an error. The full log is at: http://www.ci.uchicago.edu/~davidk/logs/rmsd-20120921-0129-zdez34v0.log. Swift trunk swift-r5942 (swift modified locally) cog-r3474 RunID: 20120921-0129-zdez34v0 Progress: time: Fri, 21 Sep 2012 01:29:55 -0500 Failed to lock CA dir java.io.IOException: Failed to delete /home/davidk/.globus/coasters/locking.entering.30217 at org.globus.cog.util.concurrent.FileLock.write(Unknown Source) at org.globus.cog.util.concurrent.FileLock.lock(Unknown Source) at org.globus.cog.abstraction.impl.execution.coaster.AutoCA.ensureCACertsExist(AutoCA.java:125) at org.globus.cog.abstraction.impl.execution.coaster.AutoCA.createProxy(AutoCA.java:116) at org.globus.cog.abstraction.impl.execution.coaster.ServiceManager.setupGSIProxy(ServiceManager.java:216) at org.globus.cog.abstraction.impl.execution.coaster.ServiceManager.startService(ServiceManager.java:179) at org.globus.cog.abstraction.impl.execution.coaster.ServiceManager.reserveService(ServiceManager.java:122) at org.globus.cog.abstraction.impl.execution.coaster.ServiceManager.reserveService(ServiceManager.java:136) at org.globus.cog.abstraction.impl.execution.coaster.JobSubmissionTaskHandler.getChannel(JobSubmissionTaskHandler.java:118) at org.globus.cog.abstraction.impl.execution.coaster.JobSubmissionTaskHandler.submit(JobSubmissionTaskHandler.java:104) at org.globus.cog.abstraction.impl.common.AbstractTaskHandler.submit(AbstractTaskHandler.java:45) at org.globus.cog.karajan.scheduler.submitQueue.NonBlockingSubmit.run(NonBlockingSubmit.java:97) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Execution failed: Failed to delete /home/davidk/.globus/coasters/locking.number.30217 -- Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=684 --- Comment #19 from David Kelly <[email protected]> 2012-09-21 01:48:00 --- This seems to happen with both ssh and ssh-cl (probably the first time I've tested with a multisite configuration since the automatic CA updates) -- Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=684 --- Comment #20 from Mihael Hategan <[email protected]> 2012-09-21 02:45:56 --- I'm an idiot. The file lock was working with multiple concurrent jvms but not with multiple threads in the same jvm. Try now. -- Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=684 --- Comment #21 from David Kelly <[email protected]> 2012-09-21 12:20:22 --- I can run multisite jobs like this now, but maybe 1 out of 5 times it fails with the unable to delete error. http://www.ci.uchicago.edu/~davidk/logs/rmsd-20120921-1210-08grfqx9.log -- Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=684 Michael Wilde <[email protected]> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |[email protected] --- Comment #22 from Michael Wilde <[email protected]> 2012-09-22 11:59:18 --- This is great progress on this enhancement! I think the remaining intermittent failure though is a showstopper for getting multi-site scripts working well for the CMTS-Voth lab, so can you move it near or to the top of your to-do list, Mihael? Thanks! Not sure if you need any logs for the failures from David. -- Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=684 --- Comment #23 from Mihael Hategan <[email protected]> 2012-09-22 19:08:26 --- Third time is the charm! r3476. -- Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=684 David Kelly <[email protected]> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|REOPENED |RESOLVED Resolution| |FIXED --- Comment #24 from David Kelly <[email protected]> 2012-09-24 16:18:20 --- I was able to do some multisite runs this morning and everything seems to be working great. Thank you! -- Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
participants (1)
-
bugzilla-daemon＠mcs.anl.gov