April 2012 - swift-bugs - mailman3.cels.anl.gov

[Bug 740] New: Swift continues if the project you specify has no allocation
by bugzilla-daemon＠mcs.anl.gov 06 Dec '13

06 Dec '13

https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=740 Summary: Swift continues if the project you specify has no allocation Product: Swift Version: 0.93 Platform: PC OS/Version: Mac OS Status: NEW Severity: normal Priority: P2 Component: Providers AssignedTo: wozniak(a)mcs.anl.gov ReportedBy: jonmon(a)utexas.edu CC: hategan(a)mcs.anl.gov, wilde(a)mcs.anl.gov, davidk(a)ci.uchicago.edu, wozniak(a)mcs.anl.gov So currently(at least on Fusion) if the project you are trying to use has no allocation, Swift sits there with the same message printing to stdout. If you check the logs there should be a message saying your the project does not have an allocation. In my case, this message 2012-02-23 16:07:02,892-0600 DEBUG AbstractExecutor Output from qsub is: "ERROR: Project "startup-jonmon" has no allocation; can't run job." 2012-02-23 16:07:02,892-0600 INFO BlockTaskSubmitter Error submitting block task: Cannot submit job: Could not submit job (qsub reported an exit code of 1). ERROR: Project "startup-jonmon" has no allocation; can't run job. It would probably be best if Swift just haults after receiving this message and reports this message to the user, so the user does not think that jobs are actually waiting in the queue. -- Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 2

[Bug 732] New: malformed xml in the sites file was used
by bugzilla-daemon＠mcs.anl.gov 06 Sep '13

06 Sep '13

https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=732 Summary: malformed xml in the sites file was used Product: Swift Version: 0.93 Platform: PC OS/Version: Mac OS Status: NEW Severity: normal Priority: P2 Component: General AssignedTo: hategan(a)mcs.anl.gov ReportedBy: jonmon(a)utexas.edu I had this line in my sites.xml file: <pool> handle="local" > That line was accepted by Swift no problem(kind of), no error reporting or nothing. What happened looked like is Swift just rejected the sites file and defaulted to the one provided by Swift. This caused problems as Swift could not find the application in my tc with that pool entry. Does Swift do any schema validation of the XML? Even then, shouldn't the Java XML parser have complained about that line? That looks like malformed XML to me. -- Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 1

[Bug 766] New: swift command gives Java stacktrace when run in non-writable directory
by bugzilla-daemon＠mcs.anl.gov 30 Aug '13

30 Aug '13

https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=766 Summary: swift command gives Java stacktrace when run in non-writable directory Product: Swift Version: trunk Platform: PC OS/Version: Mac OS Status: ASSIGNED Severity: minor Priority: P3 Component: SwiftScript language AssignedTo: davidk(a)ci.uchicago.edu ReportedBy: wilde(a)mcs.anl.gov sandbox$ swift -version no sites file specified, setting to default: /home/wilde/swift/rev/trunk/etc/sites.xml log4j:ERROR setFile(null,true) call failed. java.io.FileNotFoundException: swift.log (Permission denied) at java.io.FileOutputStream.openAppend(Native Method) at java.io.FileOutputStream.<init>(FileOutputStream.java:188) at java.io.FileOutputStream.<init>(FileOutputStream.java:113) at org.apache.log4j.FileAppender.setFile(FileAppender.java:294) at org.apache.log4j.FileAppender.activateOptions(FileAppender.java:165) at org.apache.log4j.config.PropertySetter.activate(PropertySetter.java:307) at org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:172) at org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:104) at org.apache.log4j.PropertyConfigurator.parseAppender(PropertyConfigurator.java:809) at org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:735) at org.apache.log4j.PropertyConfigurator.configureRootCategory(PropertyConfigurator.java:615) at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:502) at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:547) at org.apache.log4j.helpers.OptionConverter.selectAndConfigure(OptionConverter.java:483) at org.apache.log4j.LogManager.<clinit>(LogManager.java:127) at java.lang.J9VMInternals.initializeImpl(Native Method) at java.lang.J9VMInternals.initialize(J9VMInternals.java:200) at org.apache.log4j.Logger.getLogger(Logger.java:117) at org.globus.cog.karajan.Loader.<clinit>(Loader.java:47) at java.lang.J9VMInternals.initializeImpl(Native Method) at java.lang.J9VMInternals.initialize(J9VMInternals.java:200) at java.lang.J9VMInternals.initialize(J9VMInternals.java:167) Swift trunk swift-r5739 cog-r3368 (cog modified locally) sandbox$ pwd /home/sinitskiy/energylandscape sandbox$ -- Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 5

[Bug 746] New: Cobalt provider on Eureka only uses 8 cores in multi-node jobs
by bugzilla-daemon＠mcs.anl.gov 30 Aug '13

30 Aug '13

https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=746 Summary: Cobalt provider on Eureka only uses 8 cores in multi-node jobs Product: Swift Version: 0.93 Platform: PC OS/Version: Mac OS Status: ASSIGNED Severity: normal Priority: P2 Component: Providers AssignedTo: davidk(a)ci.uchicago.edu ReportedBy: wilde(a)mcs.anl.gov CC: hategan(a)mcs.anl.gov, davidk(a)ci.uchicago.edu, wozniak(a)mcs.anl.gov On Eureka, when I request coaster slots with nodegranularity=2 and have 8 Cobalt jobs running (for a total of 16 8-core nodes), is that I never see more than 64 jobs active. My throttle should allow 128 jobs. It looks to me like, possibly, we are only starting a coaster worker on one of the two nodes. I.e., our "multinode" logic in the submit file may be broken. Here's my sites file: <pool handle="cobalt"> <execution provider="coaster" jobmanager="local:cobalt" url="eureka.alcf.anl.gov" /> <filesystem provider="local" />  <profile namespace="globus" key="project">MTCScienceApps</profile> <profile namespace="globus" key="queue">default</profile> <profile namespace="karajan" key="jobthrottle">1.28</profile> <profile namespace="karajan" key="initialScore">10000</profile> <profile namespace="globus" key="jobsPerNode">8</profile>   <profile namespace="globus" key="slots">8</profile> <profile namespace="globus" key="maxTime">7200</profile>  <profile namespace="globus" key="maxWallTime">00:10:00</profile> <profile namespace="globus" key="nodeGranularity">2</profile> <profile namespace="globus" key="maxNodes">2</profile> <workdirectory>/home/wilde/LZ/run012/swiftwork</workdirectory> </pool> -- Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 5

[Bug 770] New: Hang checker triggered during filesystem reads
by bugzilla-daemon＠mcs.anl.gov 18 Mar '13

18 Mar '13

https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=770 Summary: Hang checker triggered during filesystem reads Product: Swift Version: trunk Platform: PC OS/Version: Linux Status: NEW Severity: normal Priority: P2 Component: General AssignedTo: hategan(a)mcs.anl.gov ReportedBy: davidk(a)ci.uchicago.edu The hang checker gets triggered during large filesystem reads. Once the read is done, work seems to progress as usual. Here is an excerpt from a script I was working on: ----- foreach script in scripts { file output_file <single_file_mapper; file=@strcat(@filename(script), ".out")>; file error_file <single_file_mapper; file=@strcat(@filename(script), ".err")>; (output_file, error_file) = bash(script); } ----- [davidk@communicado scec-sim]$ swift -sites.file sites.grid-ps.xml -tc.file tc.data -config cf scec-sim.swift Swift trunk swift-r5746 cog-r3371 RunID: 20120412-0914-0dsnyia7 No events in 10s. Registered futures: ---- Waiting threads: ---- (input): found 5938 files Progress: time: Thu, 12 Apr 2012 09:14:34 -0500 Progress: time: Thu, 12 Apr 2012 09:14:40 -0500 Initializing:1 Find: http://localhost:50000 Find: keepalive(120), reconnect - http://localhost:50000 Passive queue processor initialized. Callback URI is null Progress: time: Thu, 12 Apr 2012 09:14:42 -0500 Selecting site:25 Submitting:998 Submitted:1 -- Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 2

[Bug 696] New: Swift fails to close array initialized within nested control structures
by bugzilla-daemon＠mcs.anl.gov 23 Nov '12

23 Nov '12

https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=696 Summary: Swift fails to close array initialized within nested control structures Product: Swift Version: 0.93 Platform: PC OS/Version: Linux Status: NEW Severity: major Priority: High Component: SwiftScript language AssignedTo: hategan(a)mcs.anl.gov ReportedBy: wilde(a)mcs.anl.gov In the following example, t2.swift, the array ia[] is not correctly closed when passed to another function, when its set within an if() within foreach loops. When its set within simpler nested control structures (example t3.swift) it seems to close OK. In t2.swift has the additional nested if statement "if(true)", otherwise the programs are similar. This logic was extracted from a much more complex science app in which this failure-to-close was observed. com$ cat t2.swift main() { foreach i in [0:0] { int ia[]; if(true) { foreach j in [0:1] { if (j==0) { ia[j] = 100; } else { ia[j] = 101; } } sub(ia); } } } sub(int iap[]) { tracef("in sub: entered\n"); tracef("in sub: iap=%q\n",iap); } main(); com$ swift t2.swift no sites file specified, setting to default: /home/wilde/swift/src/0.93/cog/modules/swift/dist/swift-svn/etc/sites.xml Swift 0.93 swift-r5501 (swift modified locally) cog-r3350 RunID: 20120122-1907-uzipsekg Progress: time: Sun, 22 Jan 2012 19:07:03 -0600 in sub: entered No events in 10s. Registered futures: int[] ia Open, 2 elements, 1 listeners ---- Waiting threads: 0-1-1-0-1-1-1 ---- com$ com$ cat t3.swift main() { foreach i in [0:0] { int ia[]; foreach j in [0:1] { if (j==0) { ia[j] = 100; } else { ia[j] = 101; } } sub(ia); } } sub(int iap[]) { tracef("in sub: iap=%q\n",iap); } main(); com$ swift t3.swift no sites file specified, setting to default: /home/wilde/swift/src/0.93/cog/modules/swift/dist/swift-svn/etc/sites.xml Swift 0.93 swift-r5501 (swift modified locally) cog-r3350 RunID: 20120122-1908-zyyzuogd Progress: time: Sun, 22 Jan 2012 19:08:07 -0600 in sub: iap=[100.0,101.0] Final status: time: Sun, 22 Jan 2012 19:08:07 -0600 com$ -- Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 3

[Bug 684] New: Feature request: -nosec option for SSH with automatic coasters
by bugzilla-daemon＠mcs.anl.gov 24 Sep '12

24 Sep '12

https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=684 Summary: Feature request: -nosec option for SSH with automatic coasters Product: Swift Version: trunk Platform: PC OS/Version: Linux Status: NEW Severity: enhancement Priority: Medium Component: Providers AssignedTo: hategan(a)mcs.anl.gov ReportedBy: davidk(a)ci.uchicago.edu CC: hategan(a)mcs.anl.gov, davidk(a)ci.uchicago.edu, wozniak(a)mcs.anl.gov This may be related to the changes slated for SSH in 0.94. Currently, when we use the persistent coaster service, we can provide a -nosec option which disables the requirement for security certificates. Could we have a setting that also disables the need for certificates with ssh and automatic coasters? -- Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 24

[Bug 692] New: Add traceback for waiting threads when lockup detector triggers
by bugzilla-daemon＠mcs.anl.gov 07 Aug '12

07 Aug '12

https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=692 Summary: Add traceback for waiting threads when lockup detector triggers Product: Swift Version: 0.94 Platform: PC OS/Version: Mac OS Status: NEW Severity: enhancement Priority: P2 Component: Runtime AssignedTo: hategan(a)mcs.anl.gov ReportedBy: wilde(a)mcs.anl.gov Would be great to get a symbolic traceback, with argument values, of all threads that are hung when the "No events in 10 sec" timer goes off: No events in 10s. Registered futures: float[] try_x Open, 0 elements, 1 listeners float rejection - F/rejection[1][3]:float - Open float[] try_x Open, 0 elements, 1 listeners float rejection - F/rejection[1][4]:float - Open float[] try_x Open, 0 elements, 1 listeners ---- Waiting threads: 0-3-1-4-0-1-0-1-4-0-3-3-3-1-0 0-3-1-4-0-1-0-1-4-0-3-4-3-6-1 0-3-1-4-0-1-0-1-4-0-3-3-3-6-1 0-3-1-4-0-1-0-1-4-0-3-4-3-1-0 0-3-1-4-0-1-0-1-4-0-3-2-3-1-0 I'd like to have Swift tell me what these threads are doing and how they relate to the registered futures listed above. How much of this can be gleaned via log analysis, from the above data? -- Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 1

[Bug 756] New: Swift array closing ambiguity
by bugzilla-daemon＠mcs.anl.gov 17 Jul '12

17 Jul '12

https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=756 Summary: Swift array closing ambiguity Product: Swift Version: 0.93 Platform: PC OS/Version: Mac OS Status: NEW Severity: normal Priority: P2 Component: SwiftScript language AssignedTo: hategan(a)mcs.anl.gov ReportedBy: wilde(a)mcs.anl.gov I think from tests its the case that Swift fully closes an array when its returned from a function, but also allows modifications to that array by statements outside of the function that returns the array. I think this can lead to race conditions but dont yet have a test case that shows this. Im filing this so we dont loose track of this potential bug. The example below is one of a few variations. I think Tim may have reported similar issues when he first started testing Swift array semantics. ============================ com$ cat t1.race.swift (int a[]) fill() { a = [1:9]; } int out[]; if (1==1) { foreach i in [1] { out[0] = 123; } trace("in if",out); } if(2==2) { out = fill(); trace("in if-2",out); } if(3==3) { trace("END",out); } com$ swift t1.race.swift no sites file specified, setting to default: /home/wilde/swift/rev/trunk/etc/sites.xml Swift trunk swift-r5651 cog-r3361 RunID: 20120316-1731-z81sk1fd Progress: time: Fri, 16 Mar 2012 17:31:15 -0500 Execution failed: out is closed with a value of 123 com$ swift t1.race.swift no sites file specified, setting to default: /home/wilde/swift/rev/trunk/etc/sites.xml Swift trunk swift-r5651 cog-r3361 RunID: 20120316-1731-kvy0rel1 Progress: time: Fri, 16 Mar 2012 17:31:25 -0500 Execution failed: out is closed with a value of 123 com$ swift t1.race.swift no sites file specified, setting to default: /home/wilde/swift/rev/trunk/etc/sites.xml Swift trunk swift-r5651 cog-r3361 RunID: 20120316-1731-fxpisskc Progress: time: Fri, 16 Mar 2012 17:31:29 -0500 Execution failed: out is closed with a value of 123 com$ swift t1.race.swift no sites file specified, setting to default: /home/wilde/swift/rev/trunk/etc/sites.xml Swift trunk swift-r5651 cog-r3361 RunID: 20120316-1731-8jf41jq7 Progress: time: Fri, 16 Mar 2012 17:31:33 -0500 SwiftScript trace: in if-2, {0 = 1, 1 = 2, 2 = 3, 3 = 4, 4 = 5, 5 = 6, 6 = 7, 7 = 8, 8 = 9} SwiftScript trace: END, {0 = 1, 1 = 2, 2 = 3, 3 = 4, 4 = 5, 5 = 6, 6 = 7, 7 = 8, 8 = 9} SwiftScript trace: in if, {0 = 1, 1 = 2, 2 = 3, 3 = 4, 4 = 5, 5 = 6, 6 = 7, 7 = 8, 8 = 9} Execution failed: out is closed with a value of 1 com$ swift t1.race.swift no sites file specified, setting to default: /home/wilde/swift/rev/trunk/etc/sites.xml Swift trunk swift-r5651 cog-r3361 RunID: 20120316-1731-a5vb891f Progress: time: Fri, 16 Mar 2012 17:31:56 -0500 Execution failed: out is closed with a value of 123 com$ swift t1.race.swift no sites file specified, setting to default: /home/wilde/swift/rev/trunk/etc/sites.xml Swift trunk swift-r5651 cog-r3361 RunID: 20120316-1731-hhkppxn0 Progress: time: Fri, 16 Mar 2012 17:32:00 -0500 Execution failed: out is closed with a value of 123 com$ swift t1.race.swift no sites file specified, setting to default: /home/wilde/swift/rev/trunk/etc/sites.xml Swift trunk swift-r5651 cog-r3361 RunID: 20120316-1732-tupc3xi9 Progress: time: Fri, 16 Mar 2012 17:32:04 -0500 Execution failed: out is closed with a value of 123 com$ com$ -- Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 1

[Bug 700] New: Concurrent mapper deletes files
by bugzilla-daemon＠mcs.anl.gov 06 Jul '12

06 Jul '12

https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=700 Summary: Concurrent mapper deletes files Product: Swift Version: trunk Platform: PC OS/Version: Linux Status: NEW Severity: normal Priority: P2 Component: SwiftScript language AssignedTo: hategan(a)mcs.anl.gov ReportedBy: jonmon(a)utexas.edu CC: wilde(a)mcs.anl.gov, jon.monette(a)gmail.com Currently the concurrent mapper has is garbage collected to avoid a file system fill up of unused files. Tom Uram has mentioned that he would like to keep these files. Should we provide a setting/command line option to turn off garbage collection? -- Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 1