August 2013 - swift-bugs - mailman3.cels.anl.gov

[Bug 1052] New: readData() returning an array fails when called as an array_mapper files= argument
by bugzilla-daemon＠mcs.anl.gov 27 Aug '13

27 Aug '13

https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=1052 Summary: readData() returning an array fails when called as an array_mapper files= argument Product: Swift Version: 0.94 Platform: PC OS/Version: Mac OS Status: ASSIGNED Severity: normal Priority: P2 Component: SwiftScript language AssignedTo: hategan(a)mcs.anl.gov ReportedBy: wilde(a)mcs.anl.gov This script: --- type file; app (file output) getFileNames() { sh "-c" @strcat("find /tmp/f*.test") stdout=@output; } file f[] <array_mapper; files=readData(getFileNames())>; --- Gives this error: Could not start execution Failed to convert .swiftx to .kml for readdata.bad.swift Not typed properly: <variable>swift#callintermediate</variable> When coded like this, with readData() assigned to an intermediate array, it works: --- type file; app (file output) getFileNames() { sh "-c" @strcat("find /tmp/f*.test") stdout=@output; } string fileNames[] = readData(getFileNames()); trace(fileNames); file f[] <array_mapper; files=fileNames>; --- when assigned to array fileNames[], readData seems to know its supposed to return an array. When assigned to the mapper argument files=, it somehow does not realize that its in an array context. Seems like files= should have Array type??? -- Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 1

[Bug 1046] New: Interactive shell through tui/coasters
by bugzilla-daemon＠mcs.anl.gov 27 Aug '13

27 Aug '13

https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=1046 Summary: Interactive shell through tui/coasters Product: Swift Version: 0.94 Platform: PC OS/Version: Mac OS Status: NEW Severity: normal Priority: P2 Component: General AssignedTo: hategan(a)mcs.anl.gov ReportedBy: davidk(a)ci.uchicago.edu 1. Start swift with -tui 2. F4 to go to list of jobs. 3. Press enter on an active job 4. A window pops up with job properties. Hit tab and select terminal. Then you'll see something like this: /000000$ ls java.lang.NullPointerException /000000$ pwd java.lang.NullPointerException -- Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 3

[Bug 1073] New: readData data type problem
by bugzilla-daemon＠mcs.anl.gov 26 Aug '13

26 Aug '13

https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=1073 Summary: readData data type problem Product: Swift Version: 0.94 Platform: PC OS/Version: Mac OS Status: ASSIGNED Severity: normal Priority: P2 Component: SwiftScript language AssignedTo: hategan(a)mcs.anl.gov ReportedBy: wilde(a)mcs.anl.gov file seedfile<"output/seed.dat">; # Dynamically generated bias for simulation ensemble seedfile = genseed(genseed_prog, 1); # int s = @toInt(readData(seedfile)); tracef("Generated seed=%i\n", readData(seedfile)); yields: Could not start execution Failed to convert .swiftx to .kml for p6.swift Not typed properly: <variable>swift#callintermediate</variable> -- Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 4

[Bug 1063] New: Parsing error on a simple expression
by bugzilla-daemon＠mcs.anl.gov 26 Aug '13

26 Aug '13

https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=1063 Summary: Parsing error on a simple expression Product: Swift Version: 0.94 Platform: PC OS/Version: Mac OS Status: ASSIGNED Severity: major Priority: P1 Component: SwiftScript language AssignedTo: hategan(a)mcs.anl.gov ReportedBy: wilde(a)mcs.anl.gov The initialized declaration: int iscale = 1000 * incr(100); works, but putting the function call first: int jscale = incr(100) * 1000; yields this (I think erroneous) parsing error: Could not compile SwiftScript source: line 10:24: expecting a semicolon, found '*' (Unless Im missing something obvious...) Full example here: uc3$ cat t.swift (int j) incr (int i) { j = i + 1; } int iscale = 1000 * incr(100); tracef("iscale=%i\n", iscale); int jscale = incr(100) * 1000; tracef("jscale=%i\n", jscale); uc3$ swift t.swift Could not compile SwiftScript source: line 10:24: expecting a semicolon, found '*' uc3$ cp t.swift t2.swift uc3$ e t2.swift [3]+ Stopped emacs t2.swift uc3$ cat t2.swift (int j) incr (int i) { j = i + 1; } int iscale = 1000 * incr(100); tracef("iscale=%i\n", iscale); /* int jscale = incr(100) * 1000; tracef("jscale=%i\n", jscale); */ uc3$ swift t2.swift Swift 0.94.1-RC1 swift-r6829 cog-r3758 RunID: 20130817-1800-ig5ra7z6 Progress: time: Sat, 17 Aug 2013 18:00:20 -0500 iscale=101000 Final status: Sat, 17 Aug 2013 18:00:20 -0500 uc3$ -- Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 3

[Bug 1071] New: Apps returning multiple files to arrays fail to compile on [Trunk]
by bugzilla-daemon＠mcs.anl.gov 25 Aug '13

25 Aug '13

https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=1071 Summary: Apps returning multiple files to arrays fail to compile on [Trunk] Product: Swift Version: trunk Platform: PC OS/Version: Linux Status: NEW Severity: major Priority: P2 Component: General AssignedTo: hategan(a)mcs.anl.gov ReportedBy: yadunandb(a)ci.uchicago.edu The following script fails to compile on trunk swift-r6922 with cog-r3766. The same script compiles and runs on 0.94 : type file; app (file out, file err) remote_driver () { echo "hello" stdout=@out stderr=@err; } file driver_out[] <simple_mapper; prefix="driver", suffix=".out">; file driver_err[] <simple_mapper; prefix="driver", suffix=".err">; (driver_out[0],driver_err[0]) = remote_driver(); I see that the app must return more than one result, and the result must go into an array for this bug to manifest. When the script is run, swift fails with the error : org.globus.cog.karajan.parser.ParsingException: /scratch/midway/yadunand/swift-trunk/run-2013-08-21/beagle_tests/remote_driver.kml, line 92 column 58 swift:partialCloseDataset(driver_out)swift:partialCloseDataset(driver_err) ^ Expected ) A complete log and stdout from the run is in the attachment. I am assigning this Bug a higher priority than normal because, this bug could potentially affect a large number of workflows. The site-testing swiftscript and the current map-reduce examples are affected. -- Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 1

[Bug 1075] New: Add new sites for test automation
by bugzilla-daemon＠mcs.anl.gov 23 Aug '13

23 Aug '13

https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=1075 Summary: Add new sites for test automation Product: Swift Version: 0.94 Platform: PC OS/Version: Linux Status: ASSIGNED Severity: normal Priority: P2 Component: Testing AssignedTo: yadunandb(a)ci.uchicago.edu ReportedBy: yadunandb(a)ci.uchicago.edu CC: wilde(a)mcs.anl.gov, skenny(a)uchicago.edu, davidk(a)ci.uchicago.edu, ketan(a)mcs.anl.gov Add support for testing for the following sites: Stampede Raven Any machine which grants access with ssh-keys are suitable candidates for testing at this stage. So far, we've looked at machines with ssh-cl alone, other provider mechanisms such as gram need to be tested. For this pass (for 0.94.1 RC2, we are not going for machines that require cryptocard for authentication. -- Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 1

[Bug 1067] New: Swift error:
by bugzilla-daemon＠mcs.anl.gov 20 Aug '13

20 Aug '13

https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=1067 Summary: Swift error: Product: Swift Version: 0.94 Platform: PC OS/Version: Mac OS Status: ASSIGNED Severity: normal Priority: P2 Component: Runtime AssignedTo: hategan(a)mcs.anl.gov ReportedBy: wilde(a)mcs.anl.gov Im not sure how many cases of this we have seen. I vaguely recall encountering it myself, but at the moment I can only fine email trails regarding it in Here is the test case for this bug: Test case sketch from Mihael to swift-devel: f = quicklyFailingApp(); foreach i in [0:sufficientlyLargeNumber] { a[i] = someApp(f); } dummy = sleep(15 seconds or so) // to give the other stuff time to do its thing. Translated this into the test below. Not yet able to reproduce the bug. bri$ cat onename.swift type file; app (file ofile) quicklyFailingApp() { sh "-c" "echo foo; sleep 2; false" stdout=@ofile; } app (file ofile) someApp(file ifile) { sh "-c" @strcat("cat ",@filename(ifile)) stdout=@ofile; } app sleep (int sec) { sh "-c" @strcat("sleep ",sec); } int sufficientlyLargeNumber = 100000; file f = quicklyFailingApp(); file a[]; foreach i in [0:sufficientlyLargeNumber] { a[i] = someApp(f); } sleep(5); // to give the other stuff time to do its thing. bri$ bri$ cat ~/.swift/swift.properties sites.file=sites.xml tc.file=apps wrapperlog.always.transfer=true sitedir.keep=true file.gc.enabled=false status.mode=provider execution.retries=5 lazy.errors=true use.wrapper.staging=false use.provider.staging=false provider.staging.pin.swiftfiles=false bri$ bri$ cat apps localhost sh sh bri$ cat sites.xml <config> <pool handle="localhost"> <execution provider="local" /> <profile namespace="karajan" key="jobThrottle">.19</profile> <profile namespace="karajan" key="initialScore">10000</profile> <filesystem provider="local"/> <workdirectory>/scratch/local/wilde/test/swiftwork</workdirectory> </pool> </config> bri$ bri$ swift onename.swift Swift 0.94 swift-r6659 cog-r3743 (cog modified locally) RunID: 20130819-1056-7zs21fu9 Progress: time: Mon, 19 Aug 2013 10:56:09 -0500 Progress: time: Mon, 19 Aug 2013 10:56:11 -0500 Selecting site:1 Initializing site shared directory:1 Progress: time: Mon, 19 Aug 2013 10:56:13 -0500 Active:1 Failed but can retry:1 Progress: time: Mon, 19 Aug 2013 10:56:15 -0500 Active:1 Failed but can retry:1 Progress: time: Mon, 19 Aug 2013 10:56:18 -0500 Finished successfully:1 Failed but can retry:1 Progress: time: Mon, 19 Aug 2013 10:56:20 -0500 Finished successfully:1 Failed but can retry:1 Progress: time: Mon, 19 Aug 2013 10:56:22 -0500 Finished successfully:1 Failed but can retry:1 Progress: time: Mon, 19 Aug 2013 10:56:24 -0500 Finished successfully:1 Failed but can retry:1 Exception in sh: Arguments: [-c, echo foo; sleep 2; false] Host: localhost Directory: onename-20130819-1056-7zs21fu9/jobs/p/sh-psci91el stderr.txt: stdout.txt: foo Caused by: Application /bin/sh failed with an exit code of 1 Progress: time: Mon, 19 Aug 2013 10:56:25 -0500 Initializing:3 Failed:1 Finished successfully:1 Progress: time: Mon, 19 Aug 2013 10:56:26 -0500 Initializing:3 Failed:1 Finished successfully:1 Progress: time: Mon, 19 Aug 2013 10:56:32 -0500 Initializing:16385 Failed:1 Finished successfully:1 Progress: time: Mon, 19 Aug 2013 10:56:38 -0500 Initializing:32771 Failed:1 Finished successfully:1 Progress: time: Mon, 19 Aug 2013 10:56:39 -0500 Initializing:49152 Failed:1 Finished successfully:1 Progress: time: Mon, 19 Aug 2013 10:56:43 -0500 Initializing:49154 Failed:1 Finished successfully:1 Progress: time: Mon, 19 Aug 2013 10:56:49 -0500 Initializing:65537 Failed:1 Finished successfully:1 Progress: time: Mon, 19 Aug 2013 10:56:56 -0500 Initializing:81921 Failed:1 Finished successfully:1 Progress: time: Mon, 19 Aug 2013 10:56:59 -0500 Initializing:98306 Failed:1 Finished successfully:1 Final status: Mon, 19 Aug 2013 10:56:59 -0500 Initializing:100001 Failed:1 Finished successfully:1 The following errors have occurred: 1. Exception in sh: Arguments: [-c, echo foo; sleep 2; false] Host: localhost Directory: onename-20130819-1056-7zs21fu9/jobs/p/sh-psci91el stderr.txt: stdout.txt: foo Caused by: Application /bin/sh failed with an exit code of 1 2. Application sh not executed due to errors in dependencies (100001 times) bri$ -- Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 10

[Bug 1058] New: Improve tunneling in start-coaster-service
by bugzilla-daemon＠mcs.anl.gov 19 Aug '13

19 Aug '13

https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=1058 Summary: Improve tunneling in start-coaster-service Product: Swift Version: unspecified Platform: PC OS/Version: Mac OS Status: ASSIGNED Severity: normal Priority: P2 Component: General AssignedTo: davidk(a)ci.uchicago.edu ReportedBy: davidk(a)ci.uchicago.edu When tunneling is enabled in start-coaster-service, it currently runs "sleep 999". Remove the sleep, and see if we can create the tunnel in the same ssh command where we start worker.pl. -- Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 1

[Bug 1065] New: start-coaster-service fails to set jobsPerNode if echo app is not defined
by bugzilla-daemon＠mcs.anl.gov 19 Aug '13

19 Aug '13

https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=1065 Summary: start-coaster-service fails to set jobsPerNode if echo app is not defined Product: Swift Version: 0.94 Platform: PC OS/Version: Mac OS Status: ASSIGNED Severity: normal Priority: P2 Component: Providers AssignedTo: davidk(a)ci.uchicago.edu ReportedBy: wilde(a)mcs.anl.gov CC: hategan(a)mcs.anl.gov, davidk(a)ci.uchicago.edu, wozniak(a)mcs.anl.gov If echo is missing from coaster-service.conf, the "hi.swift" job used to force the setting of jobsPerNode fails, and jobsPerNode is left at "1". Maybe priming should use its own trivial tc.data file. This only needs fixing if the "priming" script is still needed, else this ticket can be closed. -- Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 1

[Bug 1048] New: Null pointer exception in 0.94 from MS-String runs
by bugzilla-daemon＠mcs.anl.gov 19 Aug '13

19 Aug '13

https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=1048 Summary: Null pointer exception in 0.94 from MS-String runs Product: Swift Version: 0.94 Platform: PC OS/Version: Mac OS Status: ASSIGNED Severity: major Priority: P1 Component: Runtime AssignedTo: hategan(a)mcs.anl.gov ReportedBy: wilde(a)mcs.anl.gov The CMTS MD app MS-String is getting occasional random NPEs on midway under Java 1.7. The messages appear at the start of a run, and then Swift hangs. We dont have a log yet from the user, and will try to get a jstack as well. --- mid$ module load swift mid$ swift -version Swift 0.94 swift-r6492 cog-r3658 mid$ java -version java version "1.7.0_17" Java(TM) SE Runtime Environment (build 1.7.0_17-b02) Java HotSpot(TM) 64-Bit Server VM (build 23.7-b01, mixed mode) makena002$ more runtask.68.out Swift 0.94 swift-r6492 cog-r3658 RunID: 20130719-0925-jv69o387 Progress: time: Fri, 19 Jul 2013 09:25:53 +0000 Another uncaught exception while handling an uncaught exception. java.lang.NullPointerException at org.globus.cog.karajan.workflow.FlowElementWrapper.hasProperty(FlowElementWrapper.java:146) at org.globus.cog.karajan.workflow.FlowElementWrapper.toString(FlowElementWrapper.java:319) at java.lang.String.valueOf(String.java:2854) at java.lang.StringBuilder.append(StringBuilder.java:128) at org.globus.cog.karajan.workflow.events.EventBus.start(EventBus.java:120) at org.globus.cog.karajan.workflow.events.EventTargetPair.run(EventTargetPair.java:40) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:722) The initial exception was java.lang.NullPointerException at org.globus.cog.karajan.workflow.nodes.FlowNode.setProperties(FlowNode.java:398) at org.globus.cog.karajan.workflow.FlowElementWrapper.populate(FlowElementWrapper.java:308) at org.globus.cog.karajan.workflow.FlowElementWrapper.bind(FlowElementWrapper.java:275) at org.globus.cog.karajan.workflow.FlowElementWrapper.start(FlowElementWrapper.java:224) at org.globus.cog.karajan.workflow.events.EventBus.start(EventBus.java:103) at org.globus.cog.karajan.workflow.events.EventTargetPair.run(EventTargetPair.java:40) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:722) Another uncaught exception while handling an uncaught exception. java.lang.NullPointerException at org.globus.cog.karajan.workflow.FlowElementWrapper.hasProperty(FlowElementWrapper.java:146) at org.globus.cog.karajan.workflow.FlowElementWrapper.toString(FlowElementWrapper.java:319) at java.lang.String.valueOf(String.java:2854) at java.lang.StringBuilder.append(StringBuilder.java:128) at org.globus.cog.karajan.workflow.events.EventBus.start(EventBus.java:120) at org.globus.cog.karajan.workflow.events.EventTargetPair.run(EventTargetPair.java:40) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:722) The initial exception was java.lang.NullPointerException at org.globus.cog.karajan.workflow.nodes.FlowNode.setProperties(FlowNode.java:398) at org.globus.cog.karajan.workflow.FlowElementWrapper.populate(FlowElementWrapper.java:308) at org.globus.cog.karajan.workflow.FlowElementWrapper.bind(FlowElementWrapper.java:275) at org.globus.cog.karajan.workflow.FlowElementWrapper.start(FlowElementWrapper.java:224) at org.globus.cog.karajan.workflow.events.EventBus.start(EventBus.java:103) at org.globus.cog.karajan.workflow.events.EventTargetPair.run(EventTargetPair.java:40) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) makena002$ ls -lt runtask.68.out -rw-rw-r-- 1 mvolche mvolche 7 -- Michael Wilde Computation Institute, University of Chicago Mathematics and Computer Science Division Argonne National Laboratory -- Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 3