October 2014 - swift-bugs - mailman3.cels.anl.gov

[Bug 1357] New: Swift usage
by bugzilla-daemon＠mcs.anl.gov 01 Nov '14

01 Nov '14

https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=1357 Summary: Swift usage Product: Swift Version: trunk Platform: All OS/Version: Linux Status: NEW Severity: normal Priority: P2 Component: General AssignedTo: hategan(a)mcs.anl.gov ReportedBy: ketan(a)mcs.anl.gov On typing Swift without any argument, it looks like Swift shows usage options for git instead of Swift. I get this output: % /home/ketan/swift-k/dist/swift-svn/bin/swift Swift usage: git rev-list [OPTION] <commit-id>... [ -- paths... ] limiting output: --max-count=nr --max-age=epoch --min-age=epoch --sparse --no-merges --remove-empty --all --branches --tags --remotes --stdin --quiet ordering output: --topo-order --date-order --reverse formatting output: --parents --children --objects | --objects-edge --unpacked --header | --pretty --abbrev=nr | --no-abbrev --abbrev-commit --left-right special purpose: --bisect --bisect-vars --bisect-all trunk git-rev: 870bd2114e6bdcab18278d44208defbf7c077fdc heads/master No Swift script specified -- Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 5

[Bug 1361] New: app executable path
by bugzilla-daemon＠mcs.anl.gov 01 Nov '14

01 Nov '14

https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=1361 Summary: app executable path Product: Swift Version: trunk Platform: PC OS/Version: Mac OS Status: NEW Severity: normal Priority: P2 Component: Configuration AssignedTo: hategan(a)mcs.anl.gov ReportedBy: ketan(a)mcs.anl.gov In the new config, Swift complains about the app executable not in system PATH despite the full path of executable specified in config. Here is the error message: Exception in bgsh: Arguments: [/home/ketan/SwiftApps/subjobs/mpicatnap, data.txt, outdir/f.0014.out, 1] Host: local Directory: catsnsleepmpi-run017/jobs/1/bgsh-1azv87zl exception @ swift-int.k, line: 467 Caused by: Cannot find executable bgsh on site system path throw @ swift-int.k, line: 78 Here is the config file: site.cluster { execution { type: "coaster" URL: "localhost" jobManager: "local:cobalt" options { tasksPerNode: 32 # Option ignored: globus:mode = script workerLoggingLevel: "DEBUG" nodeGranularity: 512 workerLoggingDirectory: "/home/ketan/workerlogs" maxJobs: 2 maxNodesPerJob: 512 # Option ignored: globus:walltime = 2050 } } filesystem { type: "local" URL: "localhost" } workDirectory: "/home/ketan/swiftwork" maxParallelTasks: 300 initialParallelTasks: 299 app.bgsh { executable: "/home/ketan/SwiftApps/subjobs/bg.sh" maxWallTime: "00:40:00" } } executionRetries: 0 keepSiteDir: false providerStagingPinSwiftFiles: false alwaysTransferWrapperLog: false -- Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 2

[Bug 1347] New: Channel timeouts on Midway
by bugzilla-daemon＠mcs.anl.gov 01 Nov '14

01 Nov '14

https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=1347 Summary: Channel timeouts on Midway Product: Swift Version: 0.95 Platform: PC OS/Version: Mac OS Status: NEW Severity: normal Priority: P2 Component: General AssignedTo: hategan(a)mcs.anl.gov ReportedBy: davidkelly(a)uchicago.edu We started to see this just recently on Midway after upgrading to 0.95. Halfway through a run, all of the active nodes will drop at once. In the logs, there are messages about channel timeouts that seem to happen about the same time. The messages look like this: globus.cog.coaster.TimeoutException: Channel timed out. lastTime=141010-130754.016, now=141010-130954.196, channel=TCPChannel [type: server, contact: 1010-0811220-000095-000000] at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133) at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124) at java.util.TimerThread.mainLoop(Timer.java:555) at java.util.TimerThread.run(Timer.java:505) Swift log at http://web.ci.uchicago.edu/~davidk/logs/RunpSIMS-20141010-1107-xgj80xg2.log This has happened for a few different large runs we've done recently in 0.95. I'll try to get a good set of worker logs soon. -- Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 5

[Bug 1325] New: userHomeOverride not honored by coasters
by bugzilla-daemon＠mcs.anl.gov 24 Oct '14

24 Oct '14

https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=1325 Summary: userHomeOverride not honored by coasters Product: Swift Version: trunk Platform: PC OS/Version: Linux Status: NEW Severity: normal Priority: P2 Component: General AssignedTo: hategan(a)mcs.anl.gov ReportedBy: yadunandb(a)ci.uchicago.edu The userHomeOverride config is not being honored with coasters using ssh-cl. I'm seeing the beagle_sanity test fail when jobOptions.pbs.aprun=true : http://users.rcc.uchicago.edu/~yadunand/override_present_pbsatrr.log When no pbs options are set, the same test fails : http://users.rcc.uchicago.edu/~yadunand/override_no_pbsatrr.log Again, in both of these cases, the userHomeOverride is set to /lustre/beagle/yadunandb/swiftwork, but no .globus folder and .globus/coaster/coaster*log gets generated. Instead the coaster*logs are going to ~/.globus. -- Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 4

[Bug 1365] New: Add and document a compile-only flag
by bugzilla-daemon＠mcs.anl.gov 23 Oct '14

23 Oct '14

https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=1365 Summary: Add and document a compile-only flag Product: Swift Version: 0.96 Platform: All OS/Version: Linux Status: ASSIGNED Severity: enhancement Priority: P2 Component: SwiftScript language AssignedTo: hategan(a)mcs.anl.gov ReportedBy: wilde(a)mcs.anl.gov When submitting Swift for batch execution it would be handy to do a syntax check before submission. A -c[compile] flag that just runs the parser and as much semantic analysis as possible, would be handy for general development as well. -- Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 0

[Bug 1363] New: Add ability to specify the run dir name for swift and coaster-service commands
by bugzilla-daemon＠mcs.anl.gov 22 Oct '14

22 Oct '14

https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=1363 Summary: Add ability to specify the run dir name for swift and coaster-service commands Product: Swift Version: 0.96 Platform: All OS/Version: Linux Status: ASSIGNED Severity: enhancement Priority: P2 Component: Configuration AssignedTo: hategan(a)mcs.anl.gov ReportedBy: wilde(a)mcs.anl.gov This is needed for running the swift and coaster-service commands in settings where some higher-level framework (or a user who wants to customize) needs to group logs etc into a specific directory. -- Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 0

[Bug 1353] New: worker log directory creation
by bugzilla-daemon＠mcs.anl.gov 22 Oct '14

22 Oct '14

https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=1353 Summary: worker log directory creation Product: Swift Version: 0.95 Platform: All OS/Version: Linux Status: NEW Severity: normal Priority: P2 Component: Configuration AssignedTo: davidkelly(a)uchicago.edu ReportedBy: ketan(a)mcs.anl.gov I have local:cobalt coaster runs on ALCF systems with workerlogging enabled via enable_worker_logging which will generate worker log files and put in the path specified in workerLoggingDirectory. If the directory specified the the path above does not exist, Swift exits with block task failed error message as follows: Execution failed: Exception in bgsh: Arguments: [/home/ketan/SwiftApps/subjobs/mpicatnap, data.txt, outdir/f.0146.out, 200] Host: cluster Directory: catsnsleepmpi-run012/jobs/q/bgsh-qm5g85zl exception @ swift-int.k, line: 530 Caused by: Block task failed: LOG: /home/ketan/workerlogs/worker-1021-2602130-000000.log After creating the workerlogs directory the error was gone. I think it would be useful to indicate the cause of the problem in this situation or if possible create the directory if it does not exist (similar to swift workdir). -- Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 2

[Bug 1359] New: config not read
by bugzilla-daemon＠mcs.anl.gov 22 Oct '14

22 Oct '14

https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=1359 Summary: config not read Product: Swift Version: trunk Platform: PC OS/Version: Mac OS Status: NEW Severity: normal Priority: P2 Component: Configuration AssignedTo: hategan(a)mcs.anl.gov ReportedBy: ketan(a)mcs.anl.gov After converting my old config files with swift-convert-config tool, I still get the same message: "Swift does not use site files any more". Here is my commandline: % /home/ketan/swift-k/dist/swift-svn/bin/swift -config swift.conf catsnsleepmpi.swift -n=600 -s=200 Here is the swift.conf file: % cat swift.conf site.cluster { execution { type: "coaster" URL: "localhost" jobManager: "local:cobalt" options { tasksPerNode: 32 # Option ignored: globus:mode = script workerLoggingLevel: "DEBUG" nodeGranularity: 512 workerLoggingDirectory: "/home/ketan/workerlogs" maxJobs: 2 maxNodesPerJob: 512 # Option ignored: globus:walltime = 2050 } } filesystem { type: "local" URL: "localhost" } workDirectory: "/home/ketan/swiftwork" maxParallelTasks: 300 initialParallelTasks: 299 app.bgsh { executable: "/home/ketan/SwiftApps/subjobs/bg.sh" maxWallTime: "00:40:00" } } executionRetries: 0 keepSiteDir: false providerStagingPinSwiftFiles: false alwaysTransferWrapperLog: false -- Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 5

[Bug 913] New: Large coaster block on beagle is missing workers
by bugzilla-daemon＠mcs.anl.gov 19 Oct '14

19 Oct '14

https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=913 Summary: Large coaster block on beagle is missing workers Product: Swift Version: 0.94 Platform: PC OS/Version: Mac OS Status: ASSIGNED Severity: blocker Priority: P2 Component: Providers AssignedTo: hategan(a)mcs.anl.gov ReportedBy: wilde(a)mcs.anl.gov CC: hategan(a)mcs.anl.gov, davidk(a)ci.uchicago.edu, wozniak(a)mcs.anl.gov A Swift run using a single 520-node block of workers on Beagle only sees 503 workers. This is running 1 worker per node, 1 job per node (running 24-core OpenMP jobs). Only 503 worker connections seem to be seen by Swift. Logs are on Beagle in: /lustre/beagle/wilde/scicol/svn/run028 An earlier 420-node run with a similar configuration saw all its workers and ran the expected 420 concurrent jobs. -- Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 2

[Bug 893] New: Consider an ext mapper for arrays that takes a full command string
by bugzilla-daemon＠mcs.anl.gov 19 Oct '14

19 Oct '14

https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=893 Summary: Consider an ext mapper for arrays that takes a full command string Product: Swift Version: 0.94 Platform: PC OS/Version: Mac OS Status: NEW Severity: enhancement Priority: P2 Component: SwiftScript language AssignedTo: wozniak(a)mcs.anl.gov ReportedBy: wilde(a)mcs.anl.gov (we may want to defer this to Swift/T) -- Per a swift-support thread with Neil Best: I need to qualify what I said in the prior posting. Per the User Guide, an ext mapper needs to return two columns for the mapping, so for this format you almost always need a wrapper script to produce that format. Neil, your suggestion would work for a mapper that mapped N elements of an array, with column 1 implicitly being [0], [1], ... With that caveat, your suggestion makes sense for some other new but useful mapper type. In Swift/T ( ) the approach to mappers is evolving somewhat differently, and we will discuss that with the community and provide some kind of backward compatibility option or conversion guide. You might want to do this as follows, instead: - declare "sh" as an app. - pass your find command to the sh as the value of the -c argument, and have the app return stdout as its result. - pass that to readData() into an array, and map the dataset with this array using the array_mapper: string s[] = [ "a.txt", "b.txt", "c.txt" ]; file f[] <array_mapper;files=s>; A simple example of the above belongs in the User Guide. I'll create a ticket for that as well. - Mike ----- Original Message ----- > From: "Michael Wilde" <wilde(a)mcs.anl.gov> > To: "Neil Best" <nbest(a)ci.uchicago.edu> > Cc: swift-user(a)ci.uchicago.edu > Sent: Wednesday, December 5, 2012 6:46:51 PM > Subject: Re: [Swift-user] Unix find as external mapper > Neil, > > While your suggested syntax sounds reasonable, thats not how the ext > mapper currently works. You do indeed need to create a small wrapper > script for such situations. The exec parameter is only used to specify > the pathname of the mapper script (absolute or relative to the working > dir in which you are running the swift command). > > I'll file an enhancement ticket to record your suggestion, which in > hindsight seems obvious, and which I like :) > > Thanks, > > - Mike > > ----- Original Message ----- > > From: "Neil Best" <nbest(a)ci.uchicago.edu> > > To: swift-user(a)ci.uchicago.edu > > Sent: Wednesday, December 5, 2012 4:17:22 PM > > Subject: [Swift-user] Unix find as external mapper > > I tried to use find as an external mapper in a way that seemed > > fairly > > natural to me: > > > > file monthly[] <ext; exec="find data/nc -mindepth 1 -type d -printf > > '[%f] %p.nc\n'\|.txt'">; > > > > When I run this find by itself at the command line the output looks > > like this: > > > > $ find data/nc -mindepth 1 -type d -printf '[%f] %p.nc\n' | head > > [197901] data/nc/197901.nc > > [197903] data/nc/197903.nc > > [197904] data/nc/197904.nc > > [197902] data/nc/197902.nc > > [197905] data/nc/197905.nc > > [197906] data/nc/197906.nc > > [197907] data/nc/197907.nc > > [197908] data/nc/197908.nc > > [197909] data/nc/197909.nc > > [197910] data/nc/197910.nc > > > > The find actually looks at directories. These .nc files don't exist > > yet, but each one will be an aggregation of the .nc files within the > > folder of the same name using this command: > > > > cdo mergetime data/nc/197901/*.single.nc data/nc/197901.nc > > > > I thought I could then do a nested foreach over years and months to > > execute the invocations. > > > > I presume that the external mapper is treating the exec argument as > > a > > command name and not parsing the arguments. Using the other > > parameters in the ext construct would force me to write a wrapper > > script for the find invocation since all of the arguments must have > > names apparently. > > > > Do I have this right? Maybe there is a more straightforward > > approach. > > _______________________________________________ > > Swift-user mailing list > > Swift-user(a)ci.uchicago.edu > > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user > > -- > Michael Wilde > Computation Institute, University of Chicago > Mathematics and Computer Science Division > Argonne National Laboratory > > _______________________________________________ > Swift-user mailing list > Swift-user(a)ci.uchicago.edu > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user -- Michael Wilde Computation Institute, University of Chicago Mathematics and Computer Science Division Argonne National Laboratory _______________________________________________ Swift-user mailing list Swift-user(a)ci.uchicago.edu https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user -- Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 1