October 2012 - swift-bugs - mailman3.cels.anl.gov

[Bug 682] New: Make location and naming of execution provider logs and scripts more manageable flexible
by bugzilla-daemon＠mcs.anl.gov 06 Dec '13

06 Dec '13

https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=682 Summary: Make location and naming of execution provider logs and scripts more manageable flexible Product: Swift Version: 0.93 Platform: All OS/Version: All Status: ASSIGNED Severity: enhancement Priority: P2 Component: Providers AssignedTo: davidk(a)ci.uchicago.edu ReportedBy: wilde(a)mcs.anl.gov CC: hategan(a)mcs.anl.gov, davidk(a)ci.uchicago.edu, wozniak(a)mcs.anl.gov Responding to this suggestion from Glen in email: I like that idea. Maybe we should have the location of the scripts and logs be settable from a swift.config property and/or a provider.properties setting. Ive never likes these files going under .globus, and that almost guarantees them to be unreadable by Swift support people. Although I see the desire for keeping these private by default. I'll open a bugzilla ticket for this. Thanks, Glen. - Mike ----- Original Message ----- > From: "Glen Hocky" <hockyg(a)uchicago.edu> > To: "Mike Wilde" <wilde(a)mcs.anl.gov>, "swift-devel" <swift-devel(a)ci.uchicago.edu> > Sent: Thursday, January 5, 2012 4:05:43 PM > Subject: Re: failure on stage out [PADS Support #17814] > Mike, > Ti fixed the problem I was having with PBS staging out on pads > This reminded me of a small suggestion I had, so I'm copying it to > swift devel. > > Right now job submission scripts are placed in ~/.globus/scripts > This eventually makes it very hard to look in ~/.globus/scripts and > find a specific submit file if there is ever an error > > I would suggest putting submit scripts in ~/.globus/scripts/DATE or > ~/.globus/scripts/YEAR/MONTH/DAY > > A secondary benefit is it makes it very easy to clean up this folder > by just tarring up each month separately > > -Glen -- Michael Wilde Computation Institute, University of Chicago Mathematics and Computer Science Division Argonne National Laboratory -- Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 1

[Bug 776] New: swift waits on failed job with no messages to user
by bugzilla-daemon＠mcs.anl.gov 06 Dec '13

06 Dec '13

https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=776 Summary: swift waits on failed job with no messages to user Product: Swift Version: 0.93 Platform: PC OS/Version: Other Status: ASSIGNED Severity: normal Priority: P2 Component: Error messages AssignedTo: davidk(a)ci.uchicago.edu ReportedBy: wilde(a)mcs.anl.gov CC: wozniak(a)mcs.anl.gov In debugging bug 774, I noticed that Swift kept saying "1 job submitted" in the progress indicator line on stdout/err, even though the Beagle PBS job had already failed. This is with a slightly old trunk; needs to be reproduced to see if its an additional problem. If you can find these in time, logs are in /lustre/beagle/wilde/swift/lab, dated evening of Apr 13 and morning of Apr 14. Its possible that these failures were due to the fact that I had commented out the project tag in my sites file, thinking that my default project was sufficient. Later I noticed in the log that my job was rejected by qsub due to invalid project. But this was not reported back to me on stdout/err. So Swift seemed to think the job was still queued, even though the failure was recorded in the detailed .log file. -- Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 1

[Bug 758] New: Cobalt provider on Eureka fails silently if user project is invalid
by bugzilla-daemon＠mcs.anl.gov 06 Dec '13

06 Dec '13

https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=758 Summary: Cobalt provider on Eureka fails silently if user project is invalid Product: Swift Version: 0.93 Platform: Other OS/Version: Linux Status: NEW Severity: normal Priority: P2 Component: Providers AssignedTo: davidk(a)ci.uchicago.edu ReportedBy: wilde(a)mcs.anl.gov CC: hategan(a)mcs.anl.gov, davidk(a)ci.uchicago.edu, wozniak(a)mcs.anl.gov A run with an invalid project in the sites file (an active project but invalid for the user) put 400 jobs in the submitted state even though the cobalt submission had silently failed. Look for the log on cobalt under /home/lzamboni/???/HiRAMTools/run00(1?) - I will help find this log if the problem is not trivial to replicate. -- Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 2

[Bug 740] New: Swift continues if the project you specify has no allocation
by bugzilla-daemon＠mcs.anl.gov 06 Dec '13

06 Dec '13

https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=740 Summary: Swift continues if the project you specify has no allocation Product: Swift Version: 0.93 Platform: PC OS/Version: Mac OS Status: NEW Severity: normal Priority: P2 Component: Providers AssignedTo: wozniak(a)mcs.anl.gov ReportedBy: jonmon(a)utexas.edu CC: hategan(a)mcs.anl.gov, wilde(a)mcs.anl.gov, davidk(a)ci.uchicago.edu, wozniak(a)mcs.anl.gov So currently(at least on Fusion) if the project you are trying to use has no allocation, Swift sits there with the same message printing to stdout. If you check the logs there should be a message saying your the project does not have an allocation. In my case, this message 2012-02-23 16:07:02,892-0600 DEBUG AbstractExecutor Output from qsub is: "ERROR: Project "startup-jonmon" has no allocation; can't run job." 2012-02-23 16:07:02,892-0600 INFO BlockTaskSubmitter Error submitting block task: Cannot submit job: Could not submit job (qsub reported an exit code of 1). ERROR: Project "startup-jonmon" has no allocation; can't run job. It would probably be best if Swift just haults after receiving this message and reports this message to the user, so the user does not think that jobs are actually waiting in the queue. -- Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 2

[Bug 732] New: malformed xml in the sites file was used
by bugzilla-daemon＠mcs.anl.gov 06 Sep '13

06 Sep '13

https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=732 Summary: malformed xml in the sites file was used Product: Swift Version: 0.93 Platform: PC OS/Version: Mac OS Status: NEW Severity: normal Priority: P2 Component: General AssignedTo: hategan(a)mcs.anl.gov ReportedBy: jonmon(a)utexas.edu I had this line in my sites.xml file: <pool> handle="local" > That line was accepted by Swift no problem(kind of), no error reporting or nothing. What happened looked like is Swift just rejected the sites file and defaulted to the one provided by Swift. This caused problems as Swift could not find the application in my tc with that pool entry. Does Swift do any schema validation of the XML? Even then, shouldn't the Java XML parser have complained about that line? That looks like malformed XML to me. -- Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 1

[Bug 861] New: Test suite fails on a mac
by bugzilla-daemon＠mcs.anl.gov 30 Aug '13

30 Aug '13

https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=861 Summary: Test suite fails on a mac Product: Swift Version: trunk Platform: PC OS/Version: Mac OS Status: ASSIGNED Severity: minor Priority: Medium Component: Testing AssignedTo: davidk(a)ci.uchicago.edu ReportedBy: davidk(a)ci.uchicago.edu CC: wilde(a)mcs.anl.gov, skenny(a)uchicago.edu, davidk(a)ci.uchicago.edu, ketan(a)mcs.anl.gov Several tests fail on a mac that pass on Linux. The list of tests that fail: 075551-ext-mapper-twostruct-singleproducer.swift 0759-ext-mapper-array.swift 07591-ext-mapper-multidimensional-array.swift 07592-ext-mapper-multidimensional-array-structs.swift 142-space-and-quotes.swift 1421-space-and-quotes.swift -- Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 1

[Bug 766] New: swift command gives Java stacktrace when run in non-writable directory
by bugzilla-daemon＠mcs.anl.gov 30 Aug '13

30 Aug '13

https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=766 Summary: swift command gives Java stacktrace when run in non-writable directory Product: Swift Version: trunk Platform: PC OS/Version: Mac OS Status: ASSIGNED Severity: minor Priority: P3 Component: SwiftScript language AssignedTo: davidk(a)ci.uchicago.edu ReportedBy: wilde(a)mcs.anl.gov sandbox$ swift -version no sites file specified, setting to default: /home/wilde/swift/rev/trunk/etc/sites.xml log4j:ERROR setFile(null,true) call failed. java.io.FileNotFoundException: swift.log (Permission denied) at java.io.FileOutputStream.openAppend(Native Method) at java.io.FileOutputStream.<init>(FileOutputStream.java:188) at java.io.FileOutputStream.<init>(FileOutputStream.java:113) at org.apache.log4j.FileAppender.setFile(FileAppender.java:294) at org.apache.log4j.FileAppender.activateOptions(FileAppender.java:165) at org.apache.log4j.config.PropertySetter.activate(PropertySetter.java:307) at org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:172) at org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:104) at org.apache.log4j.PropertyConfigurator.parseAppender(PropertyConfigurator.java:809) at org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:735) at org.apache.log4j.PropertyConfigurator.configureRootCategory(PropertyConfigurator.java:615) at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:502) at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:547) at org.apache.log4j.helpers.OptionConverter.selectAndConfigure(OptionConverter.java:483) at org.apache.log4j.LogManager.<clinit>(LogManager.java:127) at java.lang.J9VMInternals.initializeImpl(Native Method) at java.lang.J9VMInternals.initialize(J9VMInternals.java:200) at org.apache.log4j.Logger.getLogger(Logger.java:117) at org.globus.cog.karajan.Loader.<clinit>(Loader.java:47) at java.lang.J9VMInternals.initializeImpl(Native Method) at java.lang.J9VMInternals.initialize(J9VMInternals.java:200) at java.lang.J9VMInternals.initialize(J9VMInternals.java:167) Swift trunk swift-r5739 cog-r3368 (cog modified locally) sandbox$ pwd /home/sinitskiy/energylandscape sandbox$ -- Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 5

[Bug 746] New: Cobalt provider on Eureka only uses 8 cores in multi-node jobs
by bugzilla-daemon＠mcs.anl.gov 30 Aug '13

30 Aug '13

https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=746 Summary: Cobalt provider on Eureka only uses 8 cores in multi-node jobs Product: Swift Version: 0.93 Platform: PC OS/Version: Mac OS Status: ASSIGNED Severity: normal Priority: P2 Component: Providers AssignedTo: davidk(a)ci.uchicago.edu ReportedBy: wilde(a)mcs.anl.gov CC: hategan(a)mcs.anl.gov, davidk(a)ci.uchicago.edu, wozniak(a)mcs.anl.gov On Eureka, when I request coaster slots with nodegranularity=2 and have 8 Cobalt jobs running (for a total of 16 8-core nodes), is that I never see more than 64 jobs active. My throttle should allow 128 jobs. It looks to me like, possibly, we are only starting a coaster worker on one of the two nodes. I.e., our "multinode" logic in the submit file may be broken. Here's my sites file: <pool handle="cobalt"> <execution provider="coaster" jobmanager="local:cobalt" url="eureka.alcf.anl.gov" /> <filesystem provider="local" />  <profile namespace="globus" key="project">MTCScienceApps</profile> <profile namespace="globus" key="queue">default</profile> <profile namespace="karajan" key="jobthrottle">1.28</profile> <profile namespace="karajan" key="initialScore">10000</profile> <profile namespace="globus" key="jobsPerNode">8</profile>   <profile namespace="globus" key="slots">8</profile> <profile namespace="globus" key="maxTime">7200</profile>  <profile namespace="globus" key="maxWallTime">00:10:00</profile> <profile namespace="globus" key="nodeGranularity">2</profile> <profile namespace="globus" key="maxNodes">2</profile> <workdirectory>/home/wilde/LZ/run012/swiftwork</workdirectory> </pool> -- Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 5

[Bug 770] New: Hang checker triggered during filesystem reads
by bugzilla-daemon＠mcs.anl.gov 18 Mar '13

18 Mar '13

https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=770 Summary: Hang checker triggered during filesystem reads Product: Swift Version: trunk Platform: PC OS/Version: Linux Status: NEW Severity: normal Priority: P2 Component: General AssignedTo: hategan(a)mcs.anl.gov ReportedBy: davidk(a)ci.uchicago.edu The hang checker gets triggered during large filesystem reads. Once the read is done, work seems to progress as usual. Here is an excerpt from a script I was working on: ----- foreach script in scripts { file output_file <single_file_mapper; file=@strcat(@filename(script), ".out")>; file error_file <single_file_mapper; file=@strcat(@filename(script), ".err")>; (output_file, error_file) = bash(script); } ----- [davidk@communicado scec-sim]$ swift -sites.file sites.grid-ps.xml -tc.file tc.data -config cf scec-sim.swift Swift trunk swift-r5746 cog-r3371 RunID: 20120412-0914-0dsnyia7 No events in 10s. Registered futures: ---- Waiting threads: ---- (input): found 5938 files Progress: time: Thu, 12 Apr 2012 09:14:34 -0500 Progress: time: Thu, 12 Apr 2012 09:14:40 -0500 Initializing:1 Find: http://localhost:50000 Find: keepalive(120), reconnect - http://localhost:50000 Passive queue processor initialized. Callback URI is null Progress: time: Thu, 12 Apr 2012 09:14:42 -0500 Selecting site:25 Submitting:998 Submitted:1 -- Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 2

[Bug 823] New: EventBus exception causes workflow to stop
by bugzilla-daemon＠mcs.anl.gov 18 Mar '13

18 Mar '13

https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=823 Summary: EventBus exception causes workflow to stop Product: Swift Version: trunk Platform: PC OS/Version: Mac OS Status: NEW Severity: normal Priority: P2 Component: General AssignedTo: hategan(a)mcs.anl.gov ReportedBy: davidk(a)ci.uchicago.edu http://www.ci.uchicago.edu/~davidk/logs/rmsd-20120828-1503-nfv8ze24.log This is the first time I've seen this error. I saw it while running swift in a loop 100 times looking for another issue. If I run into it again I will get a jstack. 2012-08-28 15:03:47,804-0500 WARN EventBus Another uncaught exception while handling an uncaught exception. java.lang.NullPointerException at org.globus.cog.karajan.workflow.FlowElementWrapper.hasProperty(FlowElementWrapper.java:146) at org.globus.cog.karajan.workflow.FlowElementWrapper.toString(FlowElementWrapper.java:319) at java.lang.String.valueOf(String.java:2902) at java.lang.StringBuilder.append(StringBuilder.java:128) at org.globus.cog.karajan.workflow.events.EventBus.start(EventBus.java:115) at org.globus.cog.karajan.workflow.events.EventTargetPair.run(EventTargetPair.java:40) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:722) 2012-08-28 15:03:47,809-0500 WARN EventBus The initial exception was java.lang.NullPointerException at org.globus.cog.karajan.workflow.nodes.FlowNode.setProperties(FlowNode.java:398) at org.globus.cog.karajan.workflow.FlowElementWrapper.populate(FlowElementWrapper.java:308) at org.globus.cog.karajan.workflow.FlowElementWrapper.bind(FlowElementWrapper.java:275) at org.globus.cog.karajan.workflow.FlowElementWrapper.start(FlowElementWrapper.java:224) at org.globus.cog.karajan.workflow.events.EventBus.start(EventBus.java:98) at org.globus.cog.karajan.workflow.events.EventTargetPair.run(EventTargetPair.java:40) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:722) -- Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 2