August 2014 - swift-bugs - mailman3.cels.anl.gov

[Bug 1287] New: simple mapper throws error if mapped variable not used
by bugzilla-daemon＠mcs.anl.gov 16 Aug '14

16 Aug '14

https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=1287 Summary: simple mapper throws error if mapped variable not used Product: Swift Version: trunk Platform: All OS/Version: Linux Status: NEW Severity: normal Priority: P2 Component: General AssignedTo: hategan(a)mcs.anl.gov ReportedBy: ketan(a)mcs.anl.gov For instance, If there is a simple mapper declaration as: file result[] <simple_mapper; location="outdir", prefix="res", suffix=".dat">; and the variable is unused in the subsequent calls to apps, the error is: Execution failed: java.lang.IllegalArgumentException: Directory not found: outdir Variable declaration @ model_lmp, line: 14 Caused by: java.lang.IllegalArgumentException: Directory not found: outdir at org.griphyn.vdl.mapping.file.AbstractFileMapper.existing(AbstractFileMapper.java:232) at org.griphyn.vdl.mapping.file.AbstractFileMapper.existing(AbstractFileMapper.java:214) at org.griphyn.vdl.mapping.nodes.InitMapper.checkInputs(InitMapper.java:179) at org.griphyn.vdl.mapping.nodes.InitMapper.initialize(InitMapper.java:125) at org.griphyn.vdl.mapping.nodes.RootFutureArrayDataNode.init(RootFutureArrayDataNode.java:56) at org.griphyn.vdl.karajan.lib.New.initHandle(New.java:232) at org.griphyn.vdl.karajan.lib.New.function(New.java:180) at org.griphyn.vdl.karajan.lib.SwiftFunction.runBody(SwiftFunction.java:110) at org.globus.cog.karajan.compiled.nodes.InternalFunction.run(InternalFunction.java:157) at org.globus.cog.karajan.compiled.nodes.CompoundNode.runChild(CompoundNode.java:91) at org.globus.cog.karajan.compiled.nodes.SetVarK.run(SetVarK.java:51) at org.globus.cog.karajan.compiled.nodes.CompoundNode.runChild(CompoundNode.java:91) at org.globus.cog.karajan.compiled.nodes.FramedInternalFunction.run(FramedInternalFunction.java:63) at k.thr.LWThread.run(LWThread.java:228) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:722) I think this should be a warning. -- Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 3

[Bug 1289] New: Dependency loop detector issue
by bugzilla-daemon＠mcs.anl.gov 16 Aug '14

16 Aug '14

https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=1289 Summary: Dependency loop detector issue Product: Swift Version: unspecified Platform: PC OS/Version: Linux Status: NEW Severity: normal Priority: P2 Component: Runtime AssignedTo: wozniak(a)mcs.anl.gov ReportedBy: hategan(a)mcs.anl.gov The dependency loop detector doesn't catch the following loop: (numFaultSitesFile, numFaultSitesDbg) = runNOF (execFIFile, numFaultSitesFile); numFaultSites = readData(numFaultSitesFile); It detects a deadlock but does not explicitly mention the loop. It should. -- Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 1

[Bug 1313] New: Array return fails on trunk
by bugzilla-daemon＠mcs.anl.gov 16 Aug '14

16 Aug '14

https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=1313 Summary: Array return fails on trunk Product: Swift Version: trunk Platform: PC OS/Version: Linux Status: NEW Severity: normal Priority: P2 Component: General AssignedTo: hategan(a)mcs.anl.gov ReportedBy: yadunandb(a)ci.uchicago.edu I'm trying to return an array which would contain a random number of files from the app "remote_driver". I ran the following code locally, and it executes without any error/warnings but I do not see a returns directory in the folder I ran the script from. I see FILE_STAGE_OUT for sanity.out, sanity.err but not for the return/????.out files. type file; app (file ret[], file out, file err) remote_driver (file wrapper) { bash @wrapper stdout=@out stderr=@err; } int loops = toInt(arg("loops", "5")); file out <simple_mapper; prefix="output/sanity", suffix=".out">; file err <simple_mapper; prefix="output/sanity", suffix=".err">; file ret[] <simple_mapper; prefix="returns/", suffix=".out" >; file wrapper <"wrapper.sh">; (ret, out, err) = remote_driver(wrapper); The entire package I used to test this is available here: http://users.rcc.uchicago.edu/~yadunand/array_return.tar.gz -- Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 3

[Bug 1321] New: NullPointerException when running local:cobalt
by bugzilla-daemon＠mcs.anl.gov 16 Aug '14

16 Aug '14

https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=1321 Summary: NullPointerException when running local:cobalt Product: Swift Version: 0.95 Platform: PC OS/Version: Linux Status: NEW Severity: normal Priority: P2 Component: General AssignedTo: hategan(a)mcs.anl.gov ReportedBy: yadunandb(a)ci.uchicago.edu Here's the output on stdout, when running Swift 0.95 RC6 on Tukey with local:cobalt. The configs are from the ATPESC 2013 tutorial, which were running correctly at the time with Swift 0.94. Swift 0.95 RC6 swift-r7900 cog-r3908 RunID: run010 [Error] sites.xml, line 1, col 9: cvc-elt.1: Cannot find the declaration of element 'config'. Progress: Wed, 13 Aug 2014 16:38:11+0000 Exception in thread "Scheduler" java.lang.NullPointerException at org.globus.cog.abstraction.impl.common.task.TaskImpl.hashCode(TaskImpl.java:364) at java.util.HashMap.hash(HashMap.java:136) at java.util.HashMap.getEntry(HashMap.java:587) at java.util.HashMap.get(HashMap.java:575) at org.griphyn.vdl.karajan.VDSAdaptiveScheduler.failTask(VDSAdaptiveScheduler.java:400) at org.globus.cog.karajan.scheduler.LateBindingScheduler.run(LateBindingScheduler.java:266) Progress: Wed, 13 Aug 2014 16:38:12+0000 Selecting site:10 Tarred run folder : http://users.rcc.uchicago.edu/~yadunand/npe_0.95.tar.gz -- Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 1

[Bug 1319] New: Negative coaster worker walltimes
by bugzilla-daemon＠mcs.anl.gov 16 Aug '14

16 Aug '14

https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=1319 Summary: Negative coaster worker walltimes Product: Swift Version: trunk Platform: PC OS/Version: Linux Status: NEW Severity: normal Priority: P1 Component: General AssignedTo: hategan(a)mcs.anl.gov ReportedBy: yadunandb(a)ci.uchicago.edu On Swift trunk git-rev: 3669febcb1d6f74bf32f9429eab4e1731412b726 heads/master 5952, jobs submitted to ssh-cl:local using coasters, remain in submitted status without making any progress. This is blocking the remote site tests, and coasters remote site functionality , so I'm marking this is as a high priority bug. The swift.log shows 71 BLOCK_ACTIVE log messages each followed immediately by BLOCK_SHUTDOWN log messages. On the coaster-log on the remote machine (crush.mcs.anl.gov) the walltimes for workers are negative: 2014-08-11 16:44:48,781-0500 INFO BlockQueueProcessor Allocating blocks for a total walltime of: 1s 2014-08-11 16:44:48,781-0500 INFO BlockQueueProcessor Considering: Job(id:0 300.000s) 2014-08-11 16:44:48,781-0500 INFO BlockQueueProcessor Max Walltime (seconds): 300 2014-08-11 16:44:48,782-0500 INFO BlockQueueProcessor Time estimate (seconds): -755914305 2014-08-11 16:44:48,782-0500 INFO BlockQueueProcessor Total for this new Block (est. seconds): 0 2014-08-11 16:44:48,785-0500 INFO BlockQueueProcessor index: 0, last: 0, holding.size(): 1 2014-08-11 16:44:48,785-0500 INFO BlockQueueProcessor Queued: 0 jobs to new Block 2014-08-11 16:44:48,785-0500 INFO BlockQueueProcessor index: 0, last: 0, ii: 0, holding.size(): 1 2014-08-11 16:44:48,785-0500 INFO Block Starting block: workers=8, walltime=-755914305.000s swift.log : http://users.rcc.uchicago.edu/~yadunand/swift_worker_negative_walltime.log coaster.log : http://users.rcc.uchicago.edu/~yadunand/coaster-negative-walltime.log -- Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 1

[Bug 1305] New: Direct mode for staging
by bugzilla-daemon＠mcs.anl.gov 16 Aug '14

16 Aug '14

https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=1305 Summary: Direct mode for staging Product: Swift Version: trunk Platform: PC OS/Version: Linux Status: NEW Severity: enhancement Priority: P2 Component: General AssignedTo: hategan(a)mcs.anl.gov ReportedBy: yadunandb(a)ci.uchicago.edu Current staging mechanisms either copy(SFS, proxy) to local disk, or create links(CDM) to files when on a shared filesystem. The simple alternative that was discussed is to allow to direct access to files on shared filesystems. Mihael mentioned earlier that this could be done by changing the behavior of filename(s) when the stagingMethod is set to direct. -- Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 1

[Bug 1323] New: Record Scheduler submit spec in log
by bugzilla-daemon＠mcs.anl.gov 15 Aug '14

15 Aug '14

https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=1323 Summary: Record Scheduler submit spec in log Product: Swift Version: 0.96 Platform: All OS/Version: All Status: NEW Severity: enhancement Priority: P2 Component: Runtime AssignedTo: hategan(a)mcs.anl.gov ReportedBy: ketan(a)mcs.anl.gov In the case of submission batch schedulers (LRM, DRM), root cause of errors is often misconfiguration in sites or config properties. It would be nice to have the scheduler submit specification recorded in Swift log for speedy diagnosis of such errors. -- Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 0

[Bug 1309] New: initialParallelTasks not starting with the set number of tasks
by bugzilla-daemon＠mcs.anl.gov 08 Aug '14

08 Aug '14

https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=1309 Summary: initialParallelTasks not starting with the set number of tasks Product: Swift Version: trunk Platform: PC OS/Version: Linux Status: NEW Severity: normal Priority: P2 Component: General AssignedTo: hategan(a)mcs.anl.gov ReportedBy: yadunandb(a)ci.uchicago.edu I'm running trunk swift-r8073 (swift modified locally) cog-r4037, with the following config settings : maxJobs: 10 tasksPerNode: 4 initialParallelTasks: 20 maxParallelTasks: 20 I am expecting to see 5 jobs, but I see atmost 2, when I'm using ec2-cloud provider. Here's logs from the run: http://users.rcc.uchicago.edu/~yadunand/run016.tar.gz I retried with the local provider and I see 2 active at the beginning, with 15 towards the end. Here's the log : http://users.rcc.uchicago.edu/~yadunand/run018.tar.gz -- Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 1

[Bug 1307] New: Occasional NullPointerException when running the same script with trunk
by bugzilla-daemon＠mcs.anl.gov 08 Aug '14

08 Aug '14

https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=1307 Summary: Occasional NullPointerException when running the same script with trunk Product: Swift Version: trunk Platform: Macintosh OS/Version: Mac OS Status: NEW Severity: normal Priority: P2 Component: General AssignedTo: hategan(a)mcs.anl.gov ReportedBy: yangyangxinye(a)gmail.com I saw indeterministic behaviors when running a modis script using trunk. When running "swift -config swift.conf modis06.swift" on a local setting I got a NullPointerException error and if I ran the same command for several times (normally 2-5 times), it would then finish the job successfully. Here is the error message I got from one run: ~/MODIS-demo/original/modis/tutorial$ swift -config swift.conf modis06.swift Swift trunk swift-r8073 cog-r4037 RunID: run009 Progress: Wed, 30 Jul 2014 15:29:35-0500 Execution failed: java.lang.NullPointerException assignment @ modis06, line: 35 Caused by: java.lang.NullPointerException at org.griphyn.vdl.mapping.nodes.AbstractFutureArrayDataNode.addListener(AbstractFutureArrayDataNode.java:86) at org.griphyn.vdl.mapping.nodes.InitMapper.initialize(InitMapper.java:110) at org.griphyn.vdl.mapping.nodes.InitMapper.futureUpdated(InitMapper.java:135) at org.griphyn.vdl.mapping.nodes.AbstractFutureDataNode.notifyListeners(AbstractFutureDataNode.java:52) at org.griphyn.vdl.mapping.nodes.AbstractFutureArrayDataNode.createField(AbstractFutureArrayDataNode.java:132) at org.griphyn.vdl.mapping.nodes.AbstractFutureArrayDataNode.getField(AbstractFutureArrayDataNode.java:121) at org.griphyn.vdl.mapping.nodes.AbstractDataNode.getField(AbstractDataNode.java:238) at org.griphyn.vdl.mapping.nodes.InitMapper.addExisting(InitMapper.java:225) at org.griphyn.vdl.mapping.nodes.InitMapper.checkInputs(InitMapper.java:179) at org.griphyn.vdl.mapping.nodes.InitMapper.initialize(InitMapper.java:125) at org.griphyn.vdl.mapping.nodes.InitMapper.futureUpdated(InitMapper.java:135) at org.griphyn.vdl.mapping.nodes.AbstractFutureDataNode.notifyListeners(AbstractFutureDataNode.java:52) at org.griphyn.vdl.mapping.nodes.AbstractDataNode.postCloseActions(AbstractDataNode.java:285) at org.griphyn.vdl.mapping.nodes.AbstractFutureNonCompositeDataNode.setValue(AbstractFutureNonCompositeDataNode.java:82) at org.griphyn.vdl.karajan.lib.SetFieldValue.deepCopy(SetFieldValue.java:198) at org.griphyn.vdl.karajan.lib.SetFieldValue.function(SetFieldValue.java:148) at org.griphyn.vdl.karajan.lib.SwiftFunction.runBody(SwiftFunction.java:86) at org.globus.cog.karajan.compiled.nodes.InternalFunction.run(InternalFunction.java:158) at org.globus.cog.karajan.compiled.nodes.CompoundNode.runChild(CompoundNode.java:91) at org.globus.cog.karajan.compiled.nodes.Sequential.run(Sequential.java:22) at org.globus.cog.karajan.compiled.nodes.CompoundNode.runChild(CompoundNode.java:91) at org.globus.cog.karajan.compiled.nodes.UParallel$1.run(UParallel.java:72) at k.thr.LWThread.run(LWThread.java:228) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) at java.lang.Thread.run(Thread.java:695) I also attached the swift.log file. Anyone can tell me what might be causing this? Thanks! -- Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 2

[Bug 1315] New: Calculate total number of CPU hours used
by bugzilla-daemon＠mcs.anl.gov 07 Aug '14

07 Aug '14

https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=1315 Summary: Calculate total number of CPU hours used Product: Swift Version: unspecified Platform: PC OS/Version: Mac OS Status: NEW Severity: enhancement Priority: P2 Component: General AssignedTo: hategan(a)mcs.anl.gov ReportedBy: davidkelly(a)uchicago.edu Would be nice to have a way to automatically calculate the total number of real CPU hours used in a run -- Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 0