August 2013 - swift-bugs - mailman3.cels.anl.gov

[Bug 1040] New: Swift fails with error 127 and no clarification if workdirectory is not fully qualified
by bugzilla-daemon＠mcs.anl.gov 12 Jul '14

12 Jul '14

https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=1040 Summary: Swift fails with error 127 and no clarification if workdirectory is not fully qualified Product: Swift Version: 0.94 Platform: PC OS/Version: Mac OS Status: ASSIGNED Severity: normal Priority: P2 Component: Runtime AssignedTo: hategan(a)mcs.anl.gov ReportedBy: wilde(a)mcs.anl.gov When the sites file <workdirectory> value was inadvertently left as _WORK_, a dir by that name is created in the run directory (ie the cwd of the dir in which swift is invoked), but then the app invocation dies with error code 127. Ideally this should be made to work(when the relative dir is in fact writable by both client and compute node, as it was in this case) If thats not feasible or desirable, then a clear error message should be given that the workdirectory must be fully qualified *and* writable by both client and compute node. Error log is in http://www.ci.uchicago.edu/~wilde/catsn-20130716-1913-8556sr60.log -- Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 5

[Bug 787] New: Coasters looses slots on large-scale Cray tests
by bugzilla-daemon＠mcs.anl.gov 29 May '14

29 May '14

https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=787 Summary: Coasters looses slots on large-scale Cray tests Product: Swift Version: 0.93 Platform: Other OS/Version: Linux Status: NEW Severity: normal Priority: P2 Component: Providers AssignedTo: hategan(a)mcs.anl.gov ReportedBy: wilde(a)mcs.anl.gov CC: hategan(a)mcs.anl.gov, davidk(a)ci.uchicago.edu, wozniak(a)mcs.anl.gov Created an attachment (id=299) --> (http://bugzilla.mcs.anl.gov/swift/attachment.cgi?id=299) Image of nodes running during test on 18K-core Cray "hera" See attached slide - lost 2K of 18K cores. -- Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 3

[Bug 1024] New: scp (ssh) data provider hangs while staging out
by bugzilla-daemon＠mcs.anl.gov 29 May '14

29 May '14

https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=1024 Summary: scp (ssh) data provider hangs while staging out Product: Swift Version: 0.94 Platform: PC OS/Version: Linux Status: ASSIGNED Severity: blocker Priority: P1 Component: Providers AssignedTo: hategan(a)mcs.anl.gov ReportedBy: wilde(a)mcs.anl.gov CC: hategan(a)mcs.anl.gov, davidk(a)ci.uchicago.edu, wozniak(a)mcs.anl.gov Moving this issue from email to this ticket. Original email report is first, followed by a second incident with a log pointer. ===== Here's a snapshot of the scp failure. Script profile is: 50 jobs submitted to beagle from midway; each job is about 30 secs CPU, 2 x 16MB files in, 1 x 16 MB file out (plus some small ones in each direction). Script runs OK at 25 jobs; seems to fail consistently (2X now ;) at 50 jobs. exec provider is coasters; transport provider is ssh Point of failure looks like this in the log: 2013-06-13 17:52:26,503+0000 INFO TransportProtocolCommon Starting key exchange 2013-06-13 17:52:26,504+0000 INFO DhGroup1Sha1 Starting client side key exchange. 2013-06-13 17:52:26,506+0000 INFO TransportProtocolCommon Sending SSH_MSG_KEXDH_INIT 2013-06-13 17:52:26,507+0000 INFO TransportProtocolCommon The Transport Protocol thread failed com.sshtools.j2ssh.transport.kex.KeyExchangeException: Failed to send key exchange initailaztion message at com.sshtools.j2ssh.transport.kex.DhGroup1Sha1.performClientExchange(DhGroup1Sha1.java:180) at com.sshtools.j2ssh.transport.TransportProtocolClient.performKeyExchange(TransportProtocolClient.java:347) at com.sshtools.j2ssh.transport.TransportProtocolCommon.beginKeyExchange(TransportProtocolCommon.java:698) at com.sshtools.j2ssh.transport.TransportProtocolCommon.onMsgKexInit(TransportProtocolCommon.java:1298) at com.sshtools.j2ssh.transport.TransportProtocolCommon.startBinaryPacketProtocol(TransportProtocolCommon.java:1040) at com.sshtools.j2ssh.transport.TransportProtocolCommon.run(TransportProtocolCommon.java:393) at java.lang.Thread.run(Thread.java:722) 2013-06-13 17:52:26,509+0000 INFO Service ssh-connection thread is exiting 2013-06-13 17:52:31,172+0000 INFO Multiplexer No streams 2013-06-13 17:52:33,231+0000 INFO RuntimeStats$ProgressTicker Active:2 Stage out:48 2013-06-13 17:52:33,392+0000 INFO LateBindingScheduler jobs queued: 81 2013-06-13 17:52:33,392+0000 INFO RequestHandler Handler(tag: 107, JOBSTATUS) unregistering (send) 2013-06-13 17:52:33,392+0000 DEBUG swift STAGING_OUT jobid=sh-b5ph6val 2013-06-13 17:52:33,392+0000 INFO swift START jobid=sh-b5ph6val - Staging out files 2013-06-13 17:52:33,392+0000 DEBUG swift stageouts: [[image, aligned:Volume - Open], [header, aligned:Volume - Open]] 2013-06-13 17:52:33,393+0000 DEBUG swift FILE_STAGE_OUT_START srcname=aligned17.hdr srcdir=genatlas-20130613-1750-id2mhskd/shared/wo\ rk srchost=beagle-scp destdir=work desthost=localhost provider=file 2013-06-13 17:52:33,393+0000 DEBUG swift FILE_STAGE_OUT_START srcname=aligned17.img srcdir=genatlas-20130613-1750-id2mhskd/shared/wo\ rk srchost=beagle-scp destdir=work desthost=localhost provider=file 2013-06-13 17:52:33,393+0000 INFO LateBindingScheduler jobs queued: 81 2013-06-13 17:52:33,393+0000 INFO LateBindingScheduler jobs queued: 81 2013-06-13 17:52:33,393+0000 DEBUG swift CDM: file://localhost/work/aligned17.hdr : DEFAULT 2013-06-13 17:52:33,393+0000 DEBUG swift CDM: file://localhost/work/aligned17.img : DEFAULT 2013-06-13 17:52:33,647+0000 INFO LateBindingScheduler jobs queued: 83 2013-06-13 17:52:33,648+0000 INFO RequestHandler Handler(tag: 108, JOBSTATUS) unregistering (send) 2013-06-13 17:52:33,648+0000 INFO Sender Sender GSSChannel queue size: 0 2013-06-13 17:52:33,648+0000 DEBUG swift STAGING_OUT jobid=sh-26ph6val 2013-06-13 17:52:33,648+0000 INFO swift START jobid=sh-26ph6val - Staging out files 2013-06-13 17:52:33,648+0000 DEBUG swift stageouts: [[image, aligned:Volume - Open], [header, aligned:Volume - Open]] 2013-06-13 17:52:33,648+0000 DEBUG swift FILE_STAGE_OUT_START srcname=aligned6.hdr srcdir=genatlas-20130613-1750-id2mhskd/shared/wor\ k srchost=beagle-scp destdir=work desthost=localhost provider=file 2013-06-13 17:52:33,648+0000 DEBUG swift FILE_STAGE_OUT_START srcname=aligned6.img srcdir=genatlas-20130613-1750-id2mhskd/shared/wor\ k srchost=beagle-scp destdir=work desthost=localhost provider=file 2013-06-13 17:52:33,648+0000 INFO LateBindingScheduler jobs queued: 83 2013-06-13 17:52:33,648+0000 INFO LateBindingScheduler jobs queued: 83 2013-06-13 17:52:33,648+0000 DEBUG swift CDM: file://localhost/work/aligned6.hdr : DEFAULT 2013-06-13 17:52:33,649+0000 DEBUG swift CDM: file://localhost/work/aligned6.img : DEFAULT 2013-06-13 17:52:41,181+0000 INFO Multiplexer No streams -= ============ Here's another case of it. retries was to 3. Logs for this run are on midway at: swift$ pwd /home/wilde/Projects/OHBM2013/svn swift$ ls -lt | head total 315296 -rw-rw-r-- 1 wilde wilde 48 Jun 13 16:31 genatlas-20130613-2125-z8eenga2.0.rlog -rw-rw-r-- 1 wilde wilde 32896755 Jun 13 16:31 genatlas-20130613-2125-z8eenga2.log drwxrwxr-x 2 wilde wilde 32768 Jun 13 16:25 genatlas-20130613-2125-z8eenga2.d/ swift$ swift -config swift.properties -tc.file apps.beagle-scp genatlas.swift -d=data_100 Swift 0.94 swift-r6414 (swift modified locally) cog-r3648 RunID: 20130613-2125-z8eenga2 Progress: time: Thu, 13 Jun 2013 21:25:26 +0000 [wilde] wilde [/home/wilde/.ssh/id_rsa-swift] /home/wilde/.ssh/id_rsa-swift [null] RV]adkor{ "&068=CEGNPTX[`hux{) Progress: time: Thu, 13 Jun 2013 21:25:54 +0000 Selecting site:96 Initializing site shared directory:1 Stage in:3 Progress: time: Thu, 13 Jun 2013 21:25:56 +0000 Stage in:100 Progress: time: Thu, 13 Jun 2013 21:26:21 +0000 Stage in:99 Submitting:1 Progress: time: Thu, 13 Jun 2013 21:26:26 +0000 Stage in:97 Submitting:3 Progress: time: Thu, 13 Jun 2013 21:26:31 +0000 Stage in:96 Submitting:3 Submitted:1 Progress: time: Thu, 13 Jun 2013 21:26:34 +0000 Stage in:95 Submitting:1 Submitted:4 Progress: time: Thu, 13 Jun 2013 21:26:38 +0000 Stage in:94 Submitting:1 Submitted:5 Progress: time: Thu, 13 Jun 2013 21:26:39 +0000 Stage in:93 Submitting:1 Submitted:6 Progress: time: Thu, 13 Jun 2013 21:26:46 +0000 Stage in:91 Submitting:1 Active:8 Progress: time: Thu, 13 Jun 2013 21:26:51 +0000 Stage in:89 Submitting:1 Active:10 Progress: time: Thu, 13 Jun 2013 21:26:56 +0000 Stage in:87 Submitting:1 Active:12 Progress: time: Thu, 13 Jun 2013 21:26:58 +0000 Stage in:85 Submitting:1 Active:14 Progress: time: Thu, 13 Jun 2013 21:27:04 +0000 Stage in:83 Submitting:1 Active:16 Progress: time: Thu, 13 Jun 2013 21:27:09 +0000 Stage in:82 Submitting:1 Active:17 Progress: time: Thu, 13 Jun 2013 21:27:13 +0000 Stage in:79 Submitting:1 Active:20 Progress: time: Thu, 13 Jun 2013 21:27:14 +0000 Stage in:74 Submitting:1 Submitted:1 Active:24 Progress: time: Thu, 13 Jun 2013 21:27:19 +0000 Stage in:73 Submitting:1 Submitted:2 Active:24 Progress: time: Thu, 13 Jun 2013 21:27:21 +0000 Stage in:70 Submitting:1 Submitted:5 Active:24 Progress: time: Thu, 13 Jun 2013 21:27:22 +0000 Stage in:67 Active:33 Progress: time: Thu, 13 Jun 2013 21:27:26 +0000 Stage in:67 Active:33 Progress: time: Thu, 13 Jun 2013 21:27:28 +0000 Stage in:64 Submitting:1 Active:35 Progress: time: Thu, 13 Jun 2013 21:27:34 +0000 Stage in:60 Submitting:1 Active:31 Stage out:8 Progress: time: Thu, 13 Jun 2013 21:27:35 +0000 Stage in:59 Submitting:1 Active:32 Stage out:8 Progress: time: Thu, 13 Jun 2013 21:27:37 +0000 Stage in:59 Active:32 Checking status:1 Stage out:8 Progress: time: Thu, 13 Jun 2013 21:27:42 +0000 Stage in:59 Active:30 Checking status:1 Stage out:10 Progress: time: Thu, 13 Jun 2013 21:27:45 +0000 Stage in:59 Active:28 Checking status:1 Stage out:12 Progress: time: Thu, 13 Jun 2013 21:27:49 +0000 Stage in:59 Active:26 Checking status:1 Stage out:14 Progress: time: Thu, 13 Jun 2013 21:27:53 +0000 Stage in:59 Active:24 Checking status:1 Stage out:16 Progress: time: Thu, 13 Jun 2013 21:27:56 +0000 Stage in:59 Active:24 Stage out:17 Progress: time: Thu, 13 Jun 2013 21:27:59 +0000 Stage in:59 Active:23 Checking status:1 Stage out:17 Progress: time: Thu, 13 Jun 2013 21:28:02 +0000 Stage in:59 Active:20 Checking status:1 Stage out:20 Progress: time: Thu, 13 Jun 2013 21:28:04 +0000 Stage in:59 Active:17 Checking status:1 Stage out:23 Progress: time: Thu, 13 Jun 2013 21:28:11 +0000 Stage in:59 Active:16 Checking status:1 Stage out:24 Progress: time: Thu, 13 Jun 2013 21:28:18 +0000 Stage in:59 Active:7 Checking status:1 Stage out:33 Progress: time: Thu, 13 Jun 2013 21:28:23 +0000 Stage in:59 Active:1 Checking status:1 Stage out:39 Progress: time: Thu, 13 Jun 2013 21:28:26 +0000 Stage in:59 Checking status:1 Stage out:40 Progress: time: Thu, 13 Jun 2013 21:28:56 +0000 Stage in:59 Stage out:41 Progress: time: Thu, 13 Jun 2013 21:29:26 +0000 Stage in:59 Stage out:41 Progress: time: Thu, 13 Jun 2013 21:29:56 +0000 Stage in:59 Stage out:41 Progress: time: Thu, 13 Jun 2013 21:30:26 +0000 Stage in:59 Stage out:41 Progress: time: Thu, 13 Jun 2013 21:30:56 +0000 Stage in:59 Stage out:41 Progress: time: Thu, 13 Jun 2013 21:31:26 +0000 Stage in:59 Stage out:41 ^Cswift$ swift$ swift$ ^C swift$ -- Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 2

[Bug 957] New: Requirements for monitoring tools
by bugzilla-daemon＠mcs.anl.gov 29 May '14

29 May '14

https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=957 Summary: Requirements for monitoring tools Product: Swift Version: trunk Platform: All OS/Version: All Status: NEW Severity: enhancement Priority: P2 Component: Design Decision AssignedTo: hategan(a)mcs.anl.gov ReportedBy: hategan(a)mcs.anl.gov This is a bug to keep track of requirements for the TUI/text interface/HTML interface. The idea is that swift could be launched with a -ui parameter with various possible implementations. Currently the two possibilities are the summary or the TUI. Mike has mentioned a desire for a simple text interface and a web-browser-accessible interface. We need to discuss what run-time information these tools should convey to the user. All kinds of ideas are welcome, so please feel free to comment. -- Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 5

[Bug 730] New: swift command gives error message when run with explicit path
by bugzilla-daemon＠mcs.anl.gov 26 Feb '14

26 Feb '14

https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=730 Summary: swift command gives error message when run with explicit path Product: Swift Version: 0.93 Platform: PC OS/Version: Mac OS Status: ASSIGNED Severity: normal Priority: P2 Component: General AssignedTo: davidk(a)ci.uchicago.edu ReportedBy: wilde(a)mcs.anl.gov When you run the 0.93 swift command from an explicit path, it gets confused when generating the "no sites file" message: login2$ swift-0.93/bin/swift -version no sites file specified, setting to default: which: no swift in (/soft/apache-ant/1.8.2/etc/sites.xml Error: XML parsing error in which: no swift in (/soft/apache-ant/1.8.2/etc/sites.xml which: no swift in (/soft/apache-ant/1.8.2/etc/sites.xml is [Errno 2] No such file or directory: 'which: no swift in (/soft/apache-ant/1.8.2/etc/sites.xml' Swift 0.93 swift-r5483 cog-r3339 when run from PATH its OK: login2$ PATH=$(pwd)/swift/bin:$PATH login2$ swift -version no sites file specified, setting to default: /home/wilde/AndreysOptimizer/src/swift/etc/sites.xml Swift 0.93 swift-r5483 cog-r3339 login2$ --- While fixing this, also fix: dont generate this message at all if user is just doing -version. Also dont generate swift.log file in this (or any???) case. -- Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 1

[Bug 835] New: Rename jobsPerNode to appsPerNode
by bugzilla-daemon＠mcs.anl.gov 26 Feb '14

26 Feb '14

https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=835 Summary: Rename jobsPerNode to appsPerNode Product: Swift Version: trunk Platform: PC OS/Version: Mac OS Status: NEW Severity: normal Priority: P2 Component: Providers AssignedTo: davidk(a)ci.uchicago.edu ReportedBy: wilde(a)mcs.anl.gov CC: hategan(a)mcs.anl.gov, davidk(a)ci.uchicago.edu, wozniak(a)mcs.anl.gov >From J. Margoliash: > Who should I talk to if I want to recommend that jobsPerNode be > renamed appsPerNode or appCallsPerNode? It really is confusing to > conflate jobs sent to the execution provider (which consist of blocks > of coaster worker initializing code, if I understand correctly) with > appCalls (which circumvent the job-scheduler dilemma via already set > up coaster workers). I'd be happy to voice this opinion on behalf of > swift users if it might make a difference. -- Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 2

[Bug 1081] New: Generate provider scripts, logs and stdout/err in current dir, not .globus/scripts
by bugzilla-daemon＠mcs.anl.gov 07 Dec '13

07 Dec '13

https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=1081 Summary: Generate provider scripts, logs and stdout/err in current dir, not .globus/scripts Product: Swift Version: 0.94 Platform: PC OS/Version: Mac OS Status: ASSIGNED Severity: normal Priority: P2 Component: Providers AssignedTo: davidk(a)ci.uchicago.edu ReportedBy: wilde(a)mcs.anl.gov CC: hategan(a)mcs.anl.gov, davidk(a)ci.uchicago.edu, wozniak(a)mcs.anl.gov Local scheduler providers should generate scripts, logs, and stderr/out files below .swift in the current working dir. For Condor, should generate .log files as well as stdout/err. -- Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 2

[Bug 682] New: Make location and naming of execution provider logs and scripts more manageable flexible
by bugzilla-daemon＠mcs.anl.gov 06 Dec '13

06 Dec '13

https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=682 Summary: Make location and naming of execution provider logs and scripts more manageable flexible Product: Swift Version: 0.93 Platform: All OS/Version: All Status: ASSIGNED Severity: enhancement Priority: P2 Component: Providers AssignedTo: davidk(a)ci.uchicago.edu ReportedBy: wilde(a)mcs.anl.gov CC: hategan(a)mcs.anl.gov, davidk(a)ci.uchicago.edu, wozniak(a)mcs.anl.gov Responding to this suggestion from Glen in email: I like that idea. Maybe we should have the location of the scripts and logs be settable from a swift.config property and/or a provider.properties setting. Ive never likes these files going under .globus, and that almost guarantees them to be unreadable by Swift support people. Although I see the desire for keeping these private by default. I'll open a bugzilla ticket for this. Thanks, Glen. - Mike ----- Original Message ----- > From: "Glen Hocky" <hockyg(a)uchicago.edu> > To: "Mike Wilde" <wilde(a)mcs.anl.gov>, "swift-devel" <swift-devel(a)ci.uchicago.edu> > Sent: Thursday, January 5, 2012 4:05:43 PM > Subject: Re: failure on stage out [PADS Support #17814] > Mike, > Ti fixed the problem I was having with PBS staging out on pads > This reminded me of a small suggestion I had, so I'm copying it to > swift devel. > > Right now job submission scripts are placed in ~/.globus/scripts > This eventually makes it very hard to look in ~/.globus/scripts and > find a specific submit file if there is ever an error > > I would suggest putting submit scripts in ~/.globus/scripts/DATE or > ~/.globus/scripts/YEAR/MONTH/DAY > > A secondary benefit is it makes it very easy to clean up this folder > by just tarring up each month separately > > -Glen -- Michael Wilde Computation Institute, University of Chicago Mathematics and Computer Science Division Argonne National Laboratory -- Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 1

[Bug 1054] New: Implement retry around local scheduler commands
by bugzilla-daemon＠mcs.anl.gov 06 Dec '13

06 Dec '13

https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=1054 Summary: Implement retry around local scheduler commands Product: Swift Version: 0.94 Platform: PC OS/Version: Mac OS Status: ASSIGNED Severity: normal Priority: P2 Component: Providers AssignedTo: davidk(a)ci.uchicago.edu ReportedBy: wilde(a)mcs.anl.gov CC: hategan(a)mcs.anl.gov, davidk(a)ci.uchicago.edu, wozniak(a)mcs.anl.gov We're seeing cases on the Argonne PBS system "fusion" where qstat and qsub will occasionally hang trying to connect to a PBS service node. I suggest we consider retrying all/certain failing scheduler commands say 3 times? (perhaps configurable in provider-nnn.properties? Here's what I reported to LCRC Support on this: --- Both manually and from Swift Im seeing occasional timeouts from the qsub and qstat commands on flogin, trying to connect to fmgt2. For example, manually typing a handful of qstat commends in a row, about in 10 seems to hang, like this: fusion$ qstat -u wilde Connection timed out qstat: cannot connect to server fmgt2.lcrc.anl.gov (errno=110) Connection timed out fusion$ Looking at that process with strace, its indeed hanging in connect(): fusion$ ps -u wilde -H -f UID PID PPID C STIME TTY TIME CMD ... wilde 29449 29447 0 17:19 ? 00:00:00 sshd: wilde@pts/9 wilde 29450 29449 0 17:19 pts/9 00:00:00 -bash wilde 29662 29450 0 17:44 pts/9 00:00:00 emacs t1.sub wilde 29674 29450 0 17:45 pts/9 00:00:00 qstat -u wilde fusion$ strace -p 29674 Process 29674 attached - interrupt to quit connect(3, {sa_family=AF_INET, sin_port=htons(15001), sin_addr=inet_addr("140.221.71.72")}, 16 <unfinished ...> Process 29674 detached fusion$ Similarly, Im seeing connect failures from the swift command to qsub in Clare's science runs on fusion. Is this a known symptom? We'll need to implement some retry logic in Swift, but wondering if you think there's a networking or service problem between flogin2 and fmgt2? --- -- Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 1

[Bug 979] New: Runtime errors in SLURM provider are not halting run and getting sent to user
by bugzilla-daemon＠mcs.anl.gov 06 Dec '13

06 Dec '13

https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=979 Summary: Runtime errors in SLURM provider are not halting run and getting sent to user Product: Swift Version: 0.94 Platform: PC OS/Version: Mac OS Status: NEW Severity: major Priority: P2 Component: Providers AssignedTo: wozniak(a)mcs.anl.gov ReportedBy: wilde(a)mcs.anl.gov CC: hategan(a)mcs.anl.gov, davidk(a)ci.uchicago.edu, wozniak(a)mcs.anl.gov This might be the standard way all providers are running, but errors that the provider enounters in running jobs are only visible in the log and not brought to the user's attention on say stderr. For example, if all SLURM jobs are failing because the user has set an invalid reservation to use, then Swift just keeps indicating lack of progress, but no jobs are submitted and not error is made visible to the user. The log is below. Minimally, errors of this type should make it to the progress log, where the user could see them and adress the problem. --- 2013-04-10 03:36:42,200+0000 INFO Block Block task status changed: Submitting 2013-04-10 03:36:42,246+0000 DEBUG AbstractExecutor Writing Slurm script to /home/wilde/.globus/scripts/Slurm1915108282528394750.sub\ mit 2013-04-10 03:36:42,278+0000 DEBUG AbstractExecutor Wrote Slurm script to /home/wilde/.globus/scripts/Slurm1915108282528394750.submi\ t 2013-04-10 03:36:42,278+0000 DEBUG AbstractExecutor Command line: sbatch /home/wilde/.globus/scripts/Slurm1915108282528394750.submit 2013-04-10 03:36:42,325+0000 DEBUG AbstractExecutor Waiting for output from sbatch 2013-04-10 03:36:42,325+0000 DEBUG AbstractExecutor Output from sbatch is: "" 2013-04-10 03:36:42,325+0000 DEBUG AbstractExecutor Waiting for output from sbatch 2013-04-10 03:36:42,325+0000 DEBUG AbstractExecutor Output from sbatch is: "sbatch: error: Batch job submission failed: Requested re\ servation is invalid" 2013-04-10 03:36:42,325+0000 INFO BlockTaskSubmitter Error submitting block task org.globus.cog.abstraction.impl.common.task.TaskSubmissionException: Cannot submit job: Could not submit job (sbatch reported an exi\ t code of 1). sbatch: error: Batch job submission failed: Requested reservation is invalid at org.globus.cog.abstraction.impl.scheduler.common.AbstractJobSubmissionTaskHandler.submit(AbstractJobSubmissionTaskHandler\ .java:63) at org.globus.cog.abstraction.impl.common.AbstractTaskHandler.submit(AbstractTaskHandler.java:45) at org.globus.cog.abstraction.impl.common.task.ExecutionTaskHandler.submit(ExecutionTaskHandler.java:61) at org.globus.cog.abstraction.coaster.service.job.manager.BlockTaskSubmitter.run(BlockTaskSubmitter.java:70) Caused by: org.globus.cog.abstraction.impl.scheduler.common.ProcessException: Could not submit job (sbatch reported an exit code of \ 1). sbatch: error: Batch job submission failed: Requested reservation is invalid at org.globus.cog.abstraction.impl.scheduler.common.AbstractExecutor.start(AbstractExecutor.java:108) at org.globus.cog.abstraction.impl.scheduler.common.AbstractJobSubmissionTaskHandler.submit(AbstractJobSubmissionTaskHandler\ .java:53) ... 3 more 2013-04-10 03:36:52,022+0000 INFO TaskNotifier Congestion queue size: 0 2013-04-10 03:37:02,022+0000 INFO TaskNotifier Congestion queue size: 0 2013-04-10 03:37:11,711+0000 INFO RuntimeStats$ProgressTicker Submitted:10 2013-04-10 03:37:12,022+0000 INFO TaskNotifier Congestion queue size: 0 2013-04-10 03:37:22,022+0000 INFO TaskNotifier Congestion queue size: 0 2013-04-10 03:37:32,022+0000 INFO TaskNotifier Congestion queue size: 0 2013-04-10 03:37:35,894+0000 INFO ServiceManager Cleaning up... 2013-04-10 03:37:35,894+0000 INFO ServiceManager Shutting down service at https://10.50.180.71:50002 2013-04-10 03:37:35,895+0000 INFO RequestHandler Handler(tag: 12, SHUTDOWNSERVICE) unregistering (send) 2013-04-10 03:37:35,996+0000 INFO RequestHandler Handler(tag: 3, UNREGISTER) unregistering (send) 2013-04-10 03:37:36,019+0000 INFO BlockQueueProcessor Shutting down blocks 2013-04-10 03:37:36,019+0000 INFO Block Shutting down block Block 0410-3603420-000000 (48x1740.000s) 2013-04-10 03:37:36,020+0000 INFO Block Block Block 0410-3603420-000000 (48x1740.000s) not running. Cancelling job. 2013-04-10 03:37:36,020+0000 INFO RemoteLogHandler BLOCK_SHUTDOWN id=0410-3603420-000000 2013-04-10 03:37:36,020+0000 INFO RequestHandler Handler(tag: 4, RLOG) unregistering (send) 2013-04-10 03:37:36,020+0000 WARN Block Failed to shut down block: Block 0410-3603420-000000 (48x1740.000s) org.globus.cog.abstraction.impl.common.task.TaskSubmissionException: Can only cancel an active task at org.globus.cog.abstraction.impl.scheduler.common.AbstractExecutor.cancel(AbstractExecutor.java:196) at org.globus.cog.abstraction.impl.scheduler.common.AbstractJobSubmissionTaskHandler.cancel(AbstractJobSubmissionTaskHandler\ .java:85) at org.globus.cog.abstraction.impl.common.AbstractTaskHandler.cancel(AbstractTaskHandler.java:69) -- Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 3