March 2014 - swift-bugs - mailman3.cels.anl.gov

[Bug 949] New: Document the scratch tag
by bugzilla-daemon＠mcs.anl.gov 12 Jul '14

12 Jul '14

https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=949 Summary: Document the scratch tag Product: Swift Version: unspecified Platform: PC OS/Version: Mac OS Status: NEW Severity: normal Priority: P2 Component: Documentation AssignedTo: davidk(a)ci.uchicago.edu ReportedBy: davidk(a)ci.uchicago.edu Document the scratch tag -- Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 1

[Bug 921] New: swift -typecheck proceeds to execute script
by bugzilla-daemon＠mcs.anl.gov 12 Jul '14

12 Jul '14

https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=921 Summary: swift -typecheck proceeds to execute script Product: Swift Version: 0.94 Platform: PC OS/Version: Mac OS Status: ASSIGNED Severity: minor Priority: Low Component: SwiftScript language AssignedTo: davidk(a)ci.uchicago.edu ReportedBy: wilde(a)mcs.anl.gov -typecheck is advertised as doing a compile and type check, but not executing. When no errors are detected, it seems to try to execute: login1$ swift -typecheck annealing.swift no sites file specified, setting to default: /home/wilde/swift/src/0.93/cog/modules/swift/dist/swift-svn/etc/sites.xml Swift 0.93 swift-r6113 (swift modified locally) cog-r3532 RunID: 20130106-1153-c5sv56ca Progress: time: Sun, 06 Jan 2013 11:54:03 -0600 multi_loss appCalls=1 [0.0,0.0,4.0,50.0,-1.0] multi_loss: entered: ci=0 cj=0 target_innov=58.0 evolve_reruns=100 x=[0.0,0.0,4.0,50.0,-1.0] multi_loss: i=1 calling evolve, args=[0,0,4,50,-1,58,40000,20,100,2,1,2,0.01,50,0.3,2.3,0,0,0,0,0,m,4,0] multi_annealing: AR: i=1 ....T = 2.0 multi_annealing: AR: 0.4402700913512941 0 Execution failed: sumloss/T58.0.0.sumloss (No such file or directory) login1$ -- Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 2

[Bug 869] New: Enable multiple local coaster provider pools
by bugzilla-daemon＠mcs.anl.gov 12 Jul '14

12 Jul '14

https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=869 Summary: Enable multiple local coaster provider pools Product: Swift Version: trunk Platform: PC OS/Version: Mac OS Status: ASSIGNED Severity: normal Priority: P2 Component: Providers AssignedTo: hategan(a)mcs.anl.gov ReportedBy: wilde(a)mcs.anl.gov CC: hategan(a)mcs.anl.gov, davidk(a)ci.uchicago.edu, wozniak(a)mcs.anl.gov For mixed MPI-serial scripts, we need to have at least two pools on the local cluster (eg, midway): one for serial jobs, and one for MPI jobs. This does not seem to work for automatic coasters. There were 3 cases of interest here: Case 1: The preferred config here is case 1: to use two pools which both specify local:slurm. This case failed: it seemed that different app jobs, differentiated by tc entry, were sent indiscriminately to the wrong pool (ie they got the wrong maxtime entries etc). Case 2: Next I tried using two pools with ssh-cl:slurm, hoping that Swift would start two separate external coaster service processes. This failed with complaints that required ports were already bound. I think it also demanded that a valid proxy was in place. This solution would be fine, and quite transparent to the user, if it works without requiring the user to create an X509 proxy. Case 3: I think I finally got a third config working with two manually started coaster services, a valid proxy, and then automatic worker launch did seem to work. This would have been acceptable (but not so great for the user) if it didnt require the proxy. I will try to replicate and document the failing test cases. Mihael, it would be great if you can make cases 1 or 2 work. Or provide a simple way for the user to create the proxy needed for case 3. -- Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 3

[Bug 666] New: Ability to have sitewise provider-staging policy in a single run
by bugzilla-daemon＠mcs.anl.gov 12 Jul '14

12 Jul '14

https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=666 Summary: Ability to have sitewise provider-staging policy in a single run Product: Swift Version: 0.93 Platform: All OS/Version: Linux Status: NEW Severity: normal Priority: P2 Component: Providers AssignedTo: hategan(a)mcs.anl.gov ReportedBy: ketan(a)mcs.anl.gov CC: hategan(a)mcs.anl.gov, davidk(a)ci.uchicago.edu, wozniak(a)mcs.anl.gov In the SCEC application ported to TG and OSG, the TG part runs using the SGE provider which does not need provider staging while the OSG part of the script needs provider staging. Currently, a script-wide single config file dictates whether to have provider-staging or not. However, in cases like this, it is desirable to have provider staging set site-wise. So that, with a sites.xml property, one may specify provider staging policy. -- Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 3

[Bug 1040] New: Swift fails with error 127 and no clarification if workdirectory is not fully qualified
by bugzilla-daemon＠mcs.anl.gov 12 Jul '14

12 Jul '14

https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=1040 Summary: Swift fails with error 127 and no clarification if workdirectory is not fully qualified Product: Swift Version: 0.94 Platform: PC OS/Version: Mac OS Status: ASSIGNED Severity: normal Priority: P2 Component: Runtime AssignedTo: hategan(a)mcs.anl.gov ReportedBy: wilde(a)mcs.anl.gov When the sites file <workdirectory> value was inadvertently left as _WORK_, a dir by that name is created in the run directory (ie the cwd of the dir in which swift is invoked), but then the app invocation dies with error code 127. Ideally this should be made to work(when the relative dir is in fact writable by both client and compute node, as it was in this case) If thats not feasible or desirable, then a clear error message should be given that the workdirectory must be fully qualified *and* writable by both client and compute node. Error log is in http://www.ci.uchicago.edu/~wilde/catsn-20130716-1913-8556sr60.log -- Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 5

[Bug 787] New: Coasters looses slots on large-scale Cray tests
by bugzilla-daemon＠mcs.anl.gov 29 May '14

29 May '14

https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=787 Summary: Coasters looses slots on large-scale Cray tests Product: Swift Version: 0.93 Platform: Other OS/Version: Linux Status: NEW Severity: normal Priority: P2 Component: Providers AssignedTo: hategan(a)mcs.anl.gov ReportedBy: wilde(a)mcs.anl.gov CC: hategan(a)mcs.anl.gov, davidk(a)ci.uchicago.edu, wozniak(a)mcs.anl.gov Created an attachment (id=299) --> (http://bugzilla.mcs.anl.gov/swift/attachment.cgi?id=299) Image of nodes running during test on 18K-core Cray "hera" See attached slide - lost 2K of 18K cores. -- Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 3

[Bug 1024] New: scp (ssh) data provider hangs while staging out
by bugzilla-daemon＠mcs.anl.gov 29 May '14

29 May '14

https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=1024 Summary: scp (ssh) data provider hangs while staging out Product: Swift Version: 0.94 Platform: PC OS/Version: Linux Status: ASSIGNED Severity: blocker Priority: P1 Component: Providers AssignedTo: hategan(a)mcs.anl.gov ReportedBy: wilde(a)mcs.anl.gov CC: hategan(a)mcs.anl.gov, davidk(a)ci.uchicago.edu, wozniak(a)mcs.anl.gov Moving this issue from email to this ticket. Original email report is first, followed by a second incident with a log pointer. ===== Here's a snapshot of the scp failure. Script profile is: 50 jobs submitted to beagle from midway; each job is about 30 secs CPU, 2 x 16MB files in, 1 x 16 MB file out (plus some small ones in each direction). Script runs OK at 25 jobs; seems to fail consistently (2X now ;) at 50 jobs. exec provider is coasters; transport provider is ssh Point of failure looks like this in the log: 2013-06-13 17:52:26,503+0000 INFO TransportProtocolCommon Starting key exchange 2013-06-13 17:52:26,504+0000 INFO DhGroup1Sha1 Starting client side key exchange. 2013-06-13 17:52:26,506+0000 INFO TransportProtocolCommon Sending SSH_MSG_KEXDH_INIT 2013-06-13 17:52:26,507+0000 INFO TransportProtocolCommon The Transport Protocol thread failed com.sshtools.j2ssh.transport.kex.KeyExchangeException: Failed to send key exchange initailaztion message at com.sshtools.j2ssh.transport.kex.DhGroup1Sha1.performClientExchange(DhGroup1Sha1.java:180) at com.sshtools.j2ssh.transport.TransportProtocolClient.performKeyExchange(TransportProtocolClient.java:347) at com.sshtools.j2ssh.transport.TransportProtocolCommon.beginKeyExchange(TransportProtocolCommon.java:698) at com.sshtools.j2ssh.transport.TransportProtocolCommon.onMsgKexInit(TransportProtocolCommon.java:1298) at com.sshtools.j2ssh.transport.TransportProtocolCommon.startBinaryPacketProtocol(TransportProtocolCommon.java:1040) at com.sshtools.j2ssh.transport.TransportProtocolCommon.run(TransportProtocolCommon.java:393) at java.lang.Thread.run(Thread.java:722) 2013-06-13 17:52:26,509+0000 INFO Service ssh-connection thread is exiting 2013-06-13 17:52:31,172+0000 INFO Multiplexer No streams 2013-06-13 17:52:33,231+0000 INFO RuntimeStats$ProgressTicker Active:2 Stage out:48 2013-06-13 17:52:33,392+0000 INFO LateBindingScheduler jobs queued: 81 2013-06-13 17:52:33,392+0000 INFO RequestHandler Handler(tag: 107, JOBSTATUS) unregistering (send) 2013-06-13 17:52:33,392+0000 DEBUG swift STAGING_OUT jobid=sh-b5ph6val 2013-06-13 17:52:33,392+0000 INFO swift START jobid=sh-b5ph6val - Staging out files 2013-06-13 17:52:33,392+0000 DEBUG swift stageouts: [[image, aligned:Volume - Open], [header, aligned:Volume - Open]] 2013-06-13 17:52:33,393+0000 DEBUG swift FILE_STAGE_OUT_START srcname=aligned17.hdr srcdir=genatlas-20130613-1750-id2mhskd/shared/wo\ rk srchost=beagle-scp destdir=work desthost=localhost provider=file 2013-06-13 17:52:33,393+0000 DEBUG swift FILE_STAGE_OUT_START srcname=aligned17.img srcdir=genatlas-20130613-1750-id2mhskd/shared/wo\ rk srchost=beagle-scp destdir=work desthost=localhost provider=file 2013-06-13 17:52:33,393+0000 INFO LateBindingScheduler jobs queued: 81 2013-06-13 17:52:33,393+0000 INFO LateBindingScheduler jobs queued: 81 2013-06-13 17:52:33,393+0000 DEBUG swift CDM: file://localhost/work/aligned17.hdr : DEFAULT 2013-06-13 17:52:33,393+0000 DEBUG swift CDM: file://localhost/work/aligned17.img : DEFAULT 2013-06-13 17:52:33,647+0000 INFO LateBindingScheduler jobs queued: 83 2013-06-13 17:52:33,648+0000 INFO RequestHandler Handler(tag: 108, JOBSTATUS) unregistering (send) 2013-06-13 17:52:33,648+0000 INFO Sender Sender GSSChannel queue size: 0 2013-06-13 17:52:33,648+0000 DEBUG swift STAGING_OUT jobid=sh-26ph6val 2013-06-13 17:52:33,648+0000 INFO swift START jobid=sh-26ph6val - Staging out files 2013-06-13 17:52:33,648+0000 DEBUG swift stageouts: [[image, aligned:Volume - Open], [header, aligned:Volume - Open]] 2013-06-13 17:52:33,648+0000 DEBUG swift FILE_STAGE_OUT_START srcname=aligned6.hdr srcdir=genatlas-20130613-1750-id2mhskd/shared/wor\ k srchost=beagle-scp destdir=work desthost=localhost provider=file 2013-06-13 17:52:33,648+0000 DEBUG swift FILE_STAGE_OUT_START srcname=aligned6.img srcdir=genatlas-20130613-1750-id2mhskd/shared/wor\ k srchost=beagle-scp destdir=work desthost=localhost provider=file 2013-06-13 17:52:33,648+0000 INFO LateBindingScheduler jobs queued: 83 2013-06-13 17:52:33,648+0000 INFO LateBindingScheduler jobs queued: 83 2013-06-13 17:52:33,648+0000 DEBUG swift CDM: file://localhost/work/aligned6.hdr : DEFAULT 2013-06-13 17:52:33,649+0000 DEBUG swift CDM: file://localhost/work/aligned6.img : DEFAULT 2013-06-13 17:52:41,181+0000 INFO Multiplexer No streams -= ============ Here's another case of it. retries was to 3. Logs for this run are on midway at: swift$ pwd /home/wilde/Projects/OHBM2013/svn swift$ ls -lt | head total 315296 -rw-rw-r-- 1 wilde wilde 48 Jun 13 16:31 genatlas-20130613-2125-z8eenga2.0.rlog -rw-rw-r-- 1 wilde wilde 32896755 Jun 13 16:31 genatlas-20130613-2125-z8eenga2.log drwxrwxr-x 2 wilde wilde 32768 Jun 13 16:25 genatlas-20130613-2125-z8eenga2.d/ swift$ swift -config swift.properties -tc.file apps.beagle-scp genatlas.swift -d=data_100 Swift 0.94 swift-r6414 (swift modified locally) cog-r3648 RunID: 20130613-2125-z8eenga2 Progress: time: Thu, 13 Jun 2013 21:25:26 +0000 [wilde] wilde [/home/wilde/.ssh/id_rsa-swift] /home/wilde/.ssh/id_rsa-swift [null] RV]adkor{ "&068=CEGNPTX[`hux{) Progress: time: Thu, 13 Jun 2013 21:25:54 +0000 Selecting site:96 Initializing site shared directory:1 Stage in:3 Progress: time: Thu, 13 Jun 2013 21:25:56 +0000 Stage in:100 Progress: time: Thu, 13 Jun 2013 21:26:21 +0000 Stage in:99 Submitting:1 Progress: time: Thu, 13 Jun 2013 21:26:26 +0000 Stage in:97 Submitting:3 Progress: time: Thu, 13 Jun 2013 21:26:31 +0000 Stage in:96 Submitting:3 Submitted:1 Progress: time: Thu, 13 Jun 2013 21:26:34 +0000 Stage in:95 Submitting:1 Submitted:4 Progress: time: Thu, 13 Jun 2013 21:26:38 +0000 Stage in:94 Submitting:1 Submitted:5 Progress: time: Thu, 13 Jun 2013 21:26:39 +0000 Stage in:93 Submitting:1 Submitted:6 Progress: time: Thu, 13 Jun 2013 21:26:46 +0000 Stage in:91 Submitting:1 Active:8 Progress: time: Thu, 13 Jun 2013 21:26:51 +0000 Stage in:89 Submitting:1 Active:10 Progress: time: Thu, 13 Jun 2013 21:26:56 +0000 Stage in:87 Submitting:1 Active:12 Progress: time: Thu, 13 Jun 2013 21:26:58 +0000 Stage in:85 Submitting:1 Active:14 Progress: time: Thu, 13 Jun 2013 21:27:04 +0000 Stage in:83 Submitting:1 Active:16 Progress: time: Thu, 13 Jun 2013 21:27:09 +0000 Stage in:82 Submitting:1 Active:17 Progress: time: Thu, 13 Jun 2013 21:27:13 +0000 Stage in:79 Submitting:1 Active:20 Progress: time: Thu, 13 Jun 2013 21:27:14 +0000 Stage in:74 Submitting:1 Submitted:1 Active:24 Progress: time: Thu, 13 Jun 2013 21:27:19 +0000 Stage in:73 Submitting:1 Submitted:2 Active:24 Progress: time: Thu, 13 Jun 2013 21:27:21 +0000 Stage in:70 Submitting:1 Submitted:5 Active:24 Progress: time: Thu, 13 Jun 2013 21:27:22 +0000 Stage in:67 Active:33 Progress: time: Thu, 13 Jun 2013 21:27:26 +0000 Stage in:67 Active:33 Progress: time: Thu, 13 Jun 2013 21:27:28 +0000 Stage in:64 Submitting:1 Active:35 Progress: time: Thu, 13 Jun 2013 21:27:34 +0000 Stage in:60 Submitting:1 Active:31 Stage out:8 Progress: time: Thu, 13 Jun 2013 21:27:35 +0000 Stage in:59 Submitting:1 Active:32 Stage out:8 Progress: time: Thu, 13 Jun 2013 21:27:37 +0000 Stage in:59 Active:32 Checking status:1 Stage out:8 Progress: time: Thu, 13 Jun 2013 21:27:42 +0000 Stage in:59 Active:30 Checking status:1 Stage out:10 Progress: time: Thu, 13 Jun 2013 21:27:45 +0000 Stage in:59 Active:28 Checking status:1 Stage out:12 Progress: time: Thu, 13 Jun 2013 21:27:49 +0000 Stage in:59 Active:26 Checking status:1 Stage out:14 Progress: time: Thu, 13 Jun 2013 21:27:53 +0000 Stage in:59 Active:24 Checking status:1 Stage out:16 Progress: time: Thu, 13 Jun 2013 21:27:56 +0000 Stage in:59 Active:24 Stage out:17 Progress: time: Thu, 13 Jun 2013 21:27:59 +0000 Stage in:59 Active:23 Checking status:1 Stage out:17 Progress: time: Thu, 13 Jun 2013 21:28:02 +0000 Stage in:59 Active:20 Checking status:1 Stage out:20 Progress: time: Thu, 13 Jun 2013 21:28:04 +0000 Stage in:59 Active:17 Checking status:1 Stage out:23 Progress: time: Thu, 13 Jun 2013 21:28:11 +0000 Stage in:59 Active:16 Checking status:1 Stage out:24 Progress: time: Thu, 13 Jun 2013 21:28:18 +0000 Stage in:59 Active:7 Checking status:1 Stage out:33 Progress: time: Thu, 13 Jun 2013 21:28:23 +0000 Stage in:59 Active:1 Checking status:1 Stage out:39 Progress: time: Thu, 13 Jun 2013 21:28:26 +0000 Stage in:59 Checking status:1 Stage out:40 Progress: time: Thu, 13 Jun 2013 21:28:56 +0000 Stage in:59 Stage out:41 Progress: time: Thu, 13 Jun 2013 21:29:26 +0000 Stage in:59 Stage out:41 Progress: time: Thu, 13 Jun 2013 21:29:56 +0000 Stage in:59 Stage out:41 Progress: time: Thu, 13 Jun 2013 21:30:26 +0000 Stage in:59 Stage out:41 Progress: time: Thu, 13 Jun 2013 21:30:56 +0000 Stage in:59 Stage out:41 Progress: time: Thu, 13 Jun 2013 21:31:26 +0000 Stage in:59 Stage out:41 ^Cswift$ swift$ swift$ ^C swift$ -- Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 2

[Bug 957] New: Requirements for monitoring tools
by bugzilla-daemon＠mcs.anl.gov 29 May '14

29 May '14

https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=957 Summary: Requirements for monitoring tools Product: Swift Version: trunk Platform: All OS/Version: All Status: NEW Severity: enhancement Priority: P2 Component: Design Decision AssignedTo: hategan(a)mcs.anl.gov ReportedBy: hategan(a)mcs.anl.gov This is a bug to keep track of requirements for the TUI/text interface/HTML interface. The idea is that swift could be launched with a -ui parameter with various possible implementations. Currently the two possibilities are the summary or the TUI. Mike has mentioned a desire for a simple text interface and a web-browser-accessible interface. We need to discuss what run-time information these tools should convey to the user. All kinds of ideas are welcome, so please feel free to comment. -- Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 5

[Bug 1181] New: Batching (zip) provider
by bugzilla-daemon＠mcs.anl.gov 29 May '14

29 May '14

https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=1181 Summary: Batching (zip) provider Product: Swift Version: trunk Platform: PC OS/Version: Linux Status: NEW Severity: enhancement Priority: P2 Component: Providers AssignedTo: wozniak(a)mcs.anl.gov ReportedBy: hategan(a)mcs.anl.gov CC: hategan(a)mcs.anl.gov, davidk(a)ci.uchicago.edu, wozniak(a)mcs.anl.gov Implement a zip provider that reads and writes files to and from a zip archive and make sure it can be used with coasters. -- Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 2

[Bug 1143] New: Enable coaster provider staging to batch output files
by bugzilla-daemon＠mcs.anl.gov 29 May '14

29 May '14

https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=1143 Summary: Enable coaster provider staging to batch output files Product: Swift Version: trunk Platform: PC OS/Version: Mac OS Status: ASSIGNED Severity: enhancement Priority: P2 Component: Providers AssignedTo: hategan(a)mcs.anl.gov ReportedBy: wilde(a)mcs.anl.gov CC: hategan(a)mcs.anl.gov, davidk(a)ci.uchicago.edu, wozniak(a)mcs.anl.gov Many scripts generate large numbers of tiny files. This can cause very high overhead on slow high-traffic shared filesystems, and can also result in a lot of wasted space if the files are much smaller than the minimum file block/fragment size. This enhancement request is to build into the coaster service the ability to request that specific files or stdout/err streams from apps be batched by the swift or coaster client. One suggestion for how this would be specified by the user is that its requested by using a specific format of URI in the mapping for a file. Specifically if a file is mapped to URIs of the form: batch://client/batch/file/name//app/file/name Then the app would write to the file "app/file/name" in the task sandbox directory, and Swift would label the file using the same name in the batch file. The batch file would be named "client/batch/file/name" in the batch file. Batching of stdout and stderr streams from an app should happen in the same manner (i.e., the files specified in the stdout= and stderr= parameters of an app task command template could be mapped in the same manner as other batched files, and get labelled in the same manner in the batch file. Each file in the batch would be preceded by the line: START-BATCHED-FILE app/file/name and followed by the line: END-BATCHED-FILE app/file/name For fully qualified pathnames, either or both // above would be specified as ///. // and /// thus have special meaning and should not be inserted inadvertently. The initial mechanism would be that batched files would be buffered in RAM by the client, and written as a contiguous set of bytes to the local file. Swift can assume that such files will typically be small, and could truncate files above a limit set in a new property like max.batched.file.size. The mechanism would be most useful for (small) text files. Future extensions could create tar files or other archive formats like xar. An example batched output file client/batch/file/name would look like: START-BATCHED-FILE app/file/name001 txt txt txt END-BATCHED-FILE app/file/name001 START-BATCHED-FILE app/file/name002 txt txt txt END-BATCHED-FILE app/file/name002 -- Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 1