May 2014 - swift-bugs - mailman3.cels.anl.gov

[Bug 1143] New: Enable coaster provider staging to batch output files
by bugzilla-daemon＠mcs.anl.gov 29 May '14

29 May '14

https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=1143 Summary: Enable coaster provider staging to batch output files Product: Swift Version: trunk Platform: PC OS/Version: Mac OS Status: ASSIGNED Severity: enhancement Priority: P2 Component: Providers AssignedTo: hategan(a)mcs.anl.gov ReportedBy: wilde(a)mcs.anl.gov CC: hategan(a)mcs.anl.gov, davidk(a)ci.uchicago.edu, wozniak(a)mcs.anl.gov Many scripts generate large numbers of tiny files. This can cause very high overhead on slow high-traffic shared filesystems, and can also result in a lot of wasted space if the files are much smaller than the minimum file block/fragment size. This enhancement request is to build into the coaster service the ability to request that specific files or stdout/err streams from apps be batched by the swift or coaster client. One suggestion for how this would be specified by the user is that its requested by using a specific format of URI in the mapping for a file. Specifically if a file is mapped to URIs of the form: batch://client/batch/file/name//app/file/name Then the app would write to the file "app/file/name" in the task sandbox directory, and Swift would label the file using the same name in the batch file. The batch file would be named "client/batch/file/name" in the batch file. Batching of stdout and stderr streams from an app should happen in the same manner (i.e., the files specified in the stdout= and stderr= parameters of an app task command template could be mapped in the same manner as other batched files, and get labelled in the same manner in the batch file. Each file in the batch would be preceded by the line: START-BATCHED-FILE app/file/name and followed by the line: END-BATCHED-FILE app/file/name For fully qualified pathnames, either or both // above would be specified as ///. // and /// thus have special meaning and should not be inserted inadvertently. The initial mechanism would be that batched files would be buffered in RAM by the client, and written as a contiguous set of bytes to the local file. Swift can assume that such files will typically be small, and could truncate files above a limit set in a new property like max.batched.file.size. The mechanism would be most useful for (small) text files. Future extensions could create tar files or other archive formats like xar. An example batched output file client/batch/file/name would look like: START-BATCHED-FILE app/file/name001 txt txt txt END-BATCHED-FILE app/file/name001 START-BATCHED-FILE app/file/name002 txt txt txt END-BATCHED-FILE app/file/name002 -- Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 1

[Bug 1167] New: Unable to have file extensions in app names
by bugzilla-daemon＠mcs.anl.gov 29 May '14

29 May '14

https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=1167 Summary: Unable to have file extensions in app names Product: Swift Version: trunk Platform: PC OS/Version: Mac OS Status: NEW Severity: normal Priority: P2 Component: General AssignedTo: hategan(a)mcs.anl.gov ReportedBy: davidk(a)ci.uchicago.edu I'm not sure if this is a bug or by design, but Swift does not seem to allow file extensions in app names. For example: app (file o) simulation () { simulate.sh stdout=filename(o); } This results in: Swift trunk swift-r7349 (swift modified locally) cog-r3846 RunID: 20131202-1525-s8xkny3d Could not compile SwiftScript source: line 5:11: expecting a semicolon, found '.' Trunk has wildcards in tc.data now by default. That makes things very nice, but for apps that have extensions (.py, pl, .sh) it still requires an app definition. -- Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 1

[Bug 1229] New: Directory mapper
by bugzilla-daemon＠mcs.anl.gov 29 May '14

29 May '14

https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=1229 Summary: Directory mapper Product: Swift Version: 0.96 Platform: PC OS/Version: Mac OS Status: NEW Severity: normal Priority: P2 Component: General AssignedTo: hategan(a)mcs.anl.gov ReportedBy: davidk(a)ci.uchicago.edu Suppose there is a directory called data. It contains: data/data.txt data/a/a.dat data/b/b.dat data/c/x/foo.txt As far as I know the only way to get all of this data into an array is to write an ext mapper. I think it would be nice to do something like: file inputs[] <directory_mapper; path="/home/davidk/data", pattern="*">; -- Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 1

[Bug 1275] New: Add some basic array operations
by bugzilla-daemon＠mcs.anl.gov 19 May '14

19 May '14

https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=1275 Summary: Add some basic array operations Product: Swift Version: unspecified Platform: PC OS/Version: Linux Status: NEW Severity: normal Priority: P2 Component: SwiftScript language AssignedTo: hategan(a)mcs.anl.gov ReportedBy: hategan(a)mcs.anl.gov There is no way to slice an array if the indices are not know and/or not sequential. A split function should be added: T[int][KT] split(T[KT] array, int n) This would split array into chunks of size n (except for the last element). a reverse function should also be added: T[KT] join(T[KT1][KT2]) T[KT] join(T[int][KT2]) -> this version would join in order of first index Additional array operations: T[int] slice(T[int] array, low, high) -> return sub array with indices between low and high. Array can be sparse, in which case the returned slice's size would not necessarily be T[int] compact(T[KT] array) -> returns and array with consecutive int indices and values taken (in some random order) from array. -- Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 1

[Bug 1197] New: Automatic coaster service fails to start on a machine with rocks cluster
by bugzilla-daemon＠mcs.anl.gov 18 May '14

18 May '14

https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=1197 Summary: Automatic coaster service fails to start on a machine with rocks cluster Product: Swift Version: unspecified Platform: PC OS/Version: Linux Status: NEW Severity: normal Priority: P2 Component: Providers AssignedTo: hategan(a)mcs.anl.gov ReportedBy: hategan(a)mcs.anl.gov CC: hategan(a)mcs.anl.gov, davidk(a)ci.uchicago.edu, wozniak(a)mcs.anl.gov /etc/profile.d/ssh-key.sh in rocks cluster does a shady test to see if the shell is a login shell or not. It checks $SHLVL and if greater than 2 it decides that it must not be a login shell (despite sh having possibly been invoked with -l). If that script decides it is not a login shell, it exits with "exit", otherwise with "return". This causes any invocations of "bash -l" that are not top-level to abort during initialization with no indication of what happened. Needless to say, this prevents the coaster service from starting. -- Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 2

[Bug 1201] New: Feature request: Force an app to succeed after execution.retries failures
by bugzilla-daemon＠mcs.anl.gov 18 May '14

18 May '14

https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=1201 Summary: Feature request: Force an app to succeed after execution.retries failures Product: Swift Version: 0.95 Platform: PC OS/Version: Mac OS Status: NEW Severity: normal Priority: P2 Component: General AssignedTo: hategan(a)mcs.anl.gov ReportedBy: davidk(a)ci.uchicago.edu This is a feature request from Jason. Suppose you have a two stage workflow. Stage 1 processes 100 files, and passes the results in an array to stage two. You have execution.retries=5 and lazy.errors=true. In stage 1, there is some scientific problem with 1 of the 100 files. It fails 5 times. Even though lazy errors is true, it cannot get to stage 2 of the workflow. The suggested change was to force the task to succeed after execution.retries failures. Empty output files would be created and stage 2 could start. The forced success would be recorded somehow in the logs or on standard out/err. -- Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 6

[Bug 1219] New: Robots are killing Swift
by bugzilla-daemon＠mcs.anl.gov 18 May '14

18 May '14

https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=1219 Summary: Robots are killing Swift Product: Swift Version: 0.94 Platform: PC OS/Version: Mac OS Status: NEW Severity: normal Priority: P2 Component: General AssignedTo: hategan(a)mcs.anl.gov ReportedBy: davidk(a)ci.uchicago.edu I saw an interesting failure captured in the test suite last night. While a test was running, a robot connected to the coaster port and ran "GET / robots.txt" causing a failure. Is it possible to improve our defenses against the robots? (log at http://web.ci.uchicago.edu/~davidk/logs/modis-20140311-0954-r3q2bh9f.log) Progress: time: Tue, 11 Mar 2014 09:56:39 +0000 Submitted:10 (NIO) Header checksum failed. Computed checksum: 6c484307, checksum: 20747874 Tag: 542393671, flags: 1651470895, len: 779318383, data: GET /robots.txt HTTP Failed to shut down block: Block 0311-5409390-000000 (16x3540.000s) org.globus.cog.abstraction.impl.common.task.TaskSubmissionException: Can only cancel an active task at org.globus.cog.abstraction.impl.scheduler.common.AbstractExecutor.cancel(AbstractExecutor.java:196) at org.globus.cog.abstraction.impl.scheduler.common.AbstractJobSubmissionTaskHandler.cancel(AbstractJobSubmissionTaskHandler.java:85) at org.globus.cog.abstraction.impl.common.AbstractTaskHandler.cancel(AbstractTaskHandler.java:69) at org.globus.cog.abstraction.impl.common.task.ExecutionTaskHandler.cancel(ExecutionTaskHandler.java:106) at org.globus.cog.abstraction.impl.common.task.ExecutionTaskHandler.cancel(ExecutionTaskHandler.java:95) at org.globus.cog.abstraction.coaster.service.job.manager.BlockTaskSubmitter.cancel(BlockTaskSubmitter.java:46) at org.globus.cog.abstraction.coaster.service.job.manager.Block.forceShutdown(Block.java:333) at org.globus.cog.abstraction.coaster.service.job.manager.Block.shutdown(Block.java:313) at org.globus.cog.abstraction.coaster.service.job.manager.BlockQueueProcessor.shutdownBlocks(BlockQueueProcessor.java:800) at org.globus.cog.abstraction.coaster.service.job.manager.BlockQueueProcessor.shutdown(BlockQueueProcessor.java:789) at org.globus.cog.abstraction.coaster.service.job.manager.JobQueue.shutdown(JobQueue.java:119) at org.globus.cog.abstraction.coaster.service.CoasterService.shutdown(CoasterService.java:318) at org.globus.cog.abstraction.coaster.service.ServiceShutdownHandler.requestComplete(ServiceShutdownHandler.java:29) at org.globus.cog.karajan.workflow.service.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88) at org.globus.cog.karajan.workflow.service.channels.AbstractKarajanChannel.handleRequest(AbstractKarajanChannel.java:527) at org.globus.cog.karajan.workflow.service.channels.AbstractPipedChannel.actualSend(AbstractPipedChannel.java:86) at org.globus.cog.karajan.workflow.service.channels.AbstractPipedChannel$Sender.run(AbstractPipedChannel.java:115) -- Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 3

[Bug 1207] New: Array append fails to compile on 0.94 & trunk
by bugzilla-daemon＠mcs.anl.gov 18 May '14

18 May '14

https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=1207 Summary: Array append fails to compile on 0.94 & trunk Product: Swift Version: 0.94 Platform: PC OS/Version: Linux Status: NEW Severity: normal Priority: P2 Component: General AssignedTo: hategan(a)mcs.anl.gov ReportedBy: yadunandb(a)ci.uchicago.edu The following snippet of code from the 0.94 userguide, fails to compile on 0.94 and trunk. int[auto] array; foreach i in [1:100] { append(array, i * 2); } foreach v in array { trace(v); } Here's the error that I see : swift append.swift Could not start execution Compile error in foreach statement at line 3 Compile error in procedure invocation at line 4 Procedure append is not declared. -- Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 5

[Bug 1233] New: Nosuchmethod zip.DeflaterOutputStream failure
by bugzilla-daemon＠mcs.anl.gov 17 May '14

17 May '14

https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=1233 Summary: Nosuchmethod zip.DeflaterOutputStream failure Product: Swift Version: trunk Platform: PC URL: http://swift.rcc.uchicago.edu:8043/swift-trunk/run-201 4-04-03-014857/remote_driver-013122/remote_driver.stdo ut OS/Version: Linux Status: NEW Severity: normal Priority: P2 Component: General AssignedTo: hategan(a)mcs.anl.gov ReportedBy: yadudoc1729(a)gmail.com Swift trunk tests are failing with the following error: java.lang.NoSuchMethodError: java.util.zip.DeflaterOutputStream.<init>(Ljava/io/OutputStream;Z)V at org.globus.cog.coaster.channels.GSSChannel.setOutputStream(GSSChannel.java:169) at org.globus.cog.coaster.channels.AbstractTCPChannel.setSocket(AbstractTCPChannel.java:44) at org.globus.cog.coaster.channels.GSSChannel.reconnect(GSSChannel.java:101) at org.globus.cog.coaster.channels.GSSChannel.start(GSSChannel.java:66) at org.globus.cog.coaster.ChannelFactory.newChannel(ChannelFactory.java:55) at org.globus.cog.coaster.Client.connect(Client.java:115) at org.globus.cog.coaster.Client.newClient(Client.java:71) at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:236) at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255) at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:211) at org.globus.cog.abstraction.coaster.service.CoasterService.start(CoasterService.java:198) at org.globus.cog.abstraction.coaster.service.CoasterService.main(CoasterService.java:430) Logs from the complete run is here : http://swift.rcc.uchicago.edu:8043/bridled_sanity-141011/ -- Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 2

[Bug 1243] New: Handling of swiftx and kml files
by bugzilla-daemon＠mcs.anl.gov 17 May '14

17 May '14

https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=1243 Summary: Handling of swiftx and kml files Product: Swift Version: 0.95 Platform: PC OS/Version: Mac OS Status: NEW Severity: normal Priority: P2 Component: General AssignedTo: hategan(a)mcs.anl.gov ReportedBy: davidk(a)ci.uchicago.edu A recent discussion on swift-devel about swiftx and kml files. I feel like the ideal solution here is to keep swiftx and kml files after they're created, but make them hidden: [Swift-devel] .swiftx and .kml files --- Mihael Hategan via uchicago.edu Mar 27 Hi, The .swiftx and .kml files get deleted by bin/swift after a run. They are maybe not very user friendly, but they are useful for debugging and swift skips re-compiling things if they are already there. Python does something similar. So I would vote against having them deleted. Mihael --- David Kelly davidkelly(a)uchicago.edu Mar 27 to Mihael, Swift I think the problem with keeping *.xml and *.swiftx is that it clutters up your working directory with files that the majority of Swift users will never (directly) use. Is it possible to make the xml and swiftx files hidden? You'd get any possible compilation benefits without them getting in the way. Otherwise I'd say it makes sense to have a property to control this: If script.debug.keep=false (default) and a run directory is created, move them to the run directory at the end of the run so they are available for debugging If script.debug.keep=false and no run directory is created, remove them If script.debug.keep=true, keep them untouched in the current working directory --- Yadu Nand Mar 27 to David, Mihael, Swift We could make the kml and swiftx files dot files, or we could move them to the run directory instead of deleting them. --- Mihael Hategan Mar 28 to David, Swift On Thu, 2014-03-27 at 23:11 -0500, David Kelly wrote: > I think the problem with keeping *.xml and *.swiftx is that it clutters up > your working directory with files that the majority of Swift users will > never (directly) use. The even larger majority of python users don't use the .pyc or .pyo files. I would agree that log files that accumulate in the same directory clutter things. But .swiftx and .kml (should probably rename that to .swiftk) don't multiply, so it isn't such a big problem. So I agree with you in principle. But I don't think the benefits outweigh the cost. --- -- Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 1