November 2013 - swift-bugs - mailman3.cels.anl.gov

[Bug 1081] New: Generate provider scripts, logs and stdout/err in current dir, not .globus/scripts
by bugzilla-daemon＠mcs.anl.gov 07 Dec '13

07 Dec '13

https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=1081 Summary: Generate provider scripts, logs and stdout/err in current dir, not .globus/scripts Product: Swift Version: 0.94 Platform: PC OS/Version: Mac OS Status: ASSIGNED Severity: normal Priority: P2 Component: Providers AssignedTo: davidk(a)ci.uchicago.edu ReportedBy: wilde(a)mcs.anl.gov CC: hategan(a)mcs.anl.gov, davidk(a)ci.uchicago.edu, wozniak(a)mcs.anl.gov Local scheduler providers should generate scripts, logs, and stderr/out files below .swift in the current working dir. For Condor, should generate .log files as well as stdout/err. -- Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 2

[Bug 682] New: Make location and naming of execution provider logs and scripts more manageable flexible
by bugzilla-daemon＠mcs.anl.gov 06 Dec '13

06 Dec '13

https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=682 Summary: Make location and naming of execution provider logs and scripts more manageable flexible Product: Swift Version: 0.93 Platform: All OS/Version: All Status: ASSIGNED Severity: enhancement Priority: P2 Component: Providers AssignedTo: davidk(a)ci.uchicago.edu ReportedBy: wilde(a)mcs.anl.gov CC: hategan(a)mcs.anl.gov, davidk(a)ci.uchicago.edu, wozniak(a)mcs.anl.gov Responding to this suggestion from Glen in email: I like that idea. Maybe we should have the location of the scripts and logs be settable from a swift.config property and/or a provider.properties setting. Ive never likes these files going under .globus, and that almost guarantees them to be unreadable by Swift support people. Although I see the desire for keeping these private by default. I'll open a bugzilla ticket for this. Thanks, Glen. - Mike ----- Original Message ----- > From: "Glen Hocky" <hockyg(a)uchicago.edu> > To: "Mike Wilde" <wilde(a)mcs.anl.gov>, "swift-devel" <swift-devel(a)ci.uchicago.edu> > Sent: Thursday, January 5, 2012 4:05:43 PM > Subject: Re: failure on stage out [PADS Support #17814] > Mike, > Ti fixed the problem I was having with PBS staging out on pads > This reminded me of a small suggestion I had, so I'm copying it to > swift devel. > > Right now job submission scripts are placed in ~/.globus/scripts > This eventually makes it very hard to look in ~/.globus/scripts and > find a specific submit file if there is ever an error > > I would suggest putting submit scripts in ~/.globus/scripts/DATE or > ~/.globus/scripts/YEAR/MONTH/DAY > > A secondary benefit is it makes it very easy to clean up this folder > by just tarring up each month separately > > -Glen -- Michael Wilde Computation Institute, University of Chicago Mathematics and Computer Science Division Argonne National Laboratory -- Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 1

[Bug 1054] New: Implement retry around local scheduler commands
by bugzilla-daemon＠mcs.anl.gov 06 Dec '13

06 Dec '13

https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=1054 Summary: Implement retry around local scheduler commands Product: Swift Version: 0.94 Platform: PC OS/Version: Mac OS Status: ASSIGNED Severity: normal Priority: P2 Component: Providers AssignedTo: davidk(a)ci.uchicago.edu ReportedBy: wilde(a)mcs.anl.gov CC: hategan(a)mcs.anl.gov, davidk(a)ci.uchicago.edu, wozniak(a)mcs.anl.gov We're seeing cases on the Argonne PBS system "fusion" where qstat and qsub will occasionally hang trying to connect to a PBS service node. I suggest we consider retrying all/certain failing scheduler commands say 3 times? (perhaps configurable in provider-nnn.properties? Here's what I reported to LCRC Support on this: --- Both manually and from Swift Im seeing occasional timeouts from the qsub and qstat commands on flogin, trying to connect to fmgt2. For example, manually typing a handful of qstat commends in a row, about in 10 seems to hang, like this: fusion$ qstat -u wilde Connection timed out qstat: cannot connect to server fmgt2.lcrc.anl.gov (errno=110) Connection timed out fusion$ Looking at that process with strace, its indeed hanging in connect(): fusion$ ps -u wilde -H -f UID PID PPID C STIME TTY TIME CMD ... wilde 29449 29447 0 17:19 ? 00:00:00 sshd: wilde@pts/9 wilde 29450 29449 0 17:19 pts/9 00:00:00 -bash wilde 29662 29450 0 17:44 pts/9 00:00:00 emacs t1.sub wilde 29674 29450 0 17:45 pts/9 00:00:00 qstat -u wilde fusion$ strace -p 29674 Process 29674 attached - interrupt to quit connect(3, {sa_family=AF_INET, sin_port=htons(15001), sin_addr=inet_addr("140.221.71.72")}, 16 <unfinished ...> Process 29674 detached fusion$ Similarly, Im seeing connect failures from the swift command to qsub in Clare's science runs on fusion. Is this a known symptom? We'll need to implement some retry logic in Swift, but wondering if you think there's a networking or service problem between flogin2 and fmgt2? --- -- Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 1

[Bug 979] New: Runtime errors in SLURM provider are not halting run and getting sent to user
by bugzilla-daemon＠mcs.anl.gov 06 Dec '13

06 Dec '13

https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=979 Summary: Runtime errors in SLURM provider are not halting run and getting sent to user Product: Swift Version: 0.94 Platform: PC OS/Version: Mac OS Status: NEW Severity: major Priority: P2 Component: Providers AssignedTo: wozniak(a)mcs.anl.gov ReportedBy: wilde(a)mcs.anl.gov CC: hategan(a)mcs.anl.gov, davidk(a)ci.uchicago.edu, wozniak(a)mcs.anl.gov This might be the standard way all providers are running, but errors that the provider enounters in running jobs are only visible in the log and not brought to the user's attention on say stderr. For example, if all SLURM jobs are failing because the user has set an invalid reservation to use, then Swift just keeps indicating lack of progress, but no jobs are submitted and not error is made visible to the user. The log is below. Minimally, errors of this type should make it to the progress log, where the user could see them and adress the problem. --- 2013-04-10 03:36:42,200+0000 INFO Block Block task status changed: Submitting 2013-04-10 03:36:42,246+0000 DEBUG AbstractExecutor Writing Slurm script to /home/wilde/.globus/scripts/Slurm1915108282528394750.sub\ mit 2013-04-10 03:36:42,278+0000 DEBUG AbstractExecutor Wrote Slurm script to /home/wilde/.globus/scripts/Slurm1915108282528394750.submi\ t 2013-04-10 03:36:42,278+0000 DEBUG AbstractExecutor Command line: sbatch /home/wilde/.globus/scripts/Slurm1915108282528394750.submit 2013-04-10 03:36:42,325+0000 DEBUG AbstractExecutor Waiting for output from sbatch 2013-04-10 03:36:42,325+0000 DEBUG AbstractExecutor Output from sbatch is: "" 2013-04-10 03:36:42,325+0000 DEBUG AbstractExecutor Waiting for output from sbatch 2013-04-10 03:36:42,325+0000 DEBUG AbstractExecutor Output from sbatch is: "sbatch: error: Batch job submission failed: Requested re\ servation is invalid" 2013-04-10 03:36:42,325+0000 INFO BlockTaskSubmitter Error submitting block task org.globus.cog.abstraction.impl.common.task.TaskSubmissionException: Cannot submit job: Could not submit job (sbatch reported an exi\ t code of 1). sbatch: error: Batch job submission failed: Requested reservation is invalid at org.globus.cog.abstraction.impl.scheduler.common.AbstractJobSubmissionTaskHandler.submit(AbstractJobSubmissionTaskHandler\ .java:63) at org.globus.cog.abstraction.impl.common.AbstractTaskHandler.submit(AbstractTaskHandler.java:45) at org.globus.cog.abstraction.impl.common.task.ExecutionTaskHandler.submit(ExecutionTaskHandler.java:61) at org.globus.cog.abstraction.coaster.service.job.manager.BlockTaskSubmitter.run(BlockTaskSubmitter.java:70) Caused by: org.globus.cog.abstraction.impl.scheduler.common.ProcessException: Could not submit job (sbatch reported an exit code of \ 1). sbatch: error: Batch job submission failed: Requested reservation is invalid at org.globus.cog.abstraction.impl.scheduler.common.AbstractExecutor.start(AbstractExecutor.java:108) at org.globus.cog.abstraction.impl.scheduler.common.AbstractJobSubmissionTaskHandler.submit(AbstractJobSubmissionTaskHandler\ .java:53) ... 3 more 2013-04-10 03:36:52,022+0000 INFO TaskNotifier Congestion queue size: 0 2013-04-10 03:37:02,022+0000 INFO TaskNotifier Congestion queue size: 0 2013-04-10 03:37:11,711+0000 INFO RuntimeStats$ProgressTicker Submitted:10 2013-04-10 03:37:12,022+0000 INFO TaskNotifier Congestion queue size: 0 2013-04-10 03:37:22,022+0000 INFO TaskNotifier Congestion queue size: 0 2013-04-10 03:37:32,022+0000 INFO TaskNotifier Congestion queue size: 0 2013-04-10 03:37:35,894+0000 INFO ServiceManager Cleaning up... 2013-04-10 03:37:35,894+0000 INFO ServiceManager Shutting down service at https://10.50.180.71:50002 2013-04-10 03:37:35,895+0000 INFO RequestHandler Handler(tag: 12, SHUTDOWNSERVICE) unregistering (send) 2013-04-10 03:37:35,996+0000 INFO RequestHandler Handler(tag: 3, UNREGISTER) unregistering (send) 2013-04-10 03:37:36,019+0000 INFO BlockQueueProcessor Shutting down blocks 2013-04-10 03:37:36,019+0000 INFO Block Shutting down block Block 0410-3603420-000000 (48x1740.000s) 2013-04-10 03:37:36,020+0000 INFO Block Block Block 0410-3603420-000000 (48x1740.000s) not running. Cancelling job. 2013-04-10 03:37:36,020+0000 INFO RemoteLogHandler BLOCK_SHUTDOWN id=0410-3603420-000000 2013-04-10 03:37:36,020+0000 INFO RequestHandler Handler(tag: 4, RLOG) unregistering (send) 2013-04-10 03:37:36,020+0000 WARN Block Failed to shut down block: Block 0410-3603420-000000 (48x1740.000s) org.globus.cog.abstraction.impl.common.task.TaskSubmissionException: Can only cancel an active task at org.globus.cog.abstraction.impl.scheduler.common.AbstractExecutor.cancel(AbstractExecutor.java:196) at org.globus.cog.abstraction.impl.scheduler.common.AbstractJobSubmissionTaskHandler.cancel(AbstractJobSubmissionTaskHandler\ .java:85) at org.globus.cog.abstraction.impl.common.AbstractTaskHandler.cancel(AbstractTaskHandler.java:69) -- Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 3

[Bug 776] New: swift waits on failed job with no messages to user
by bugzilla-daemon＠mcs.anl.gov 06 Dec '13

06 Dec '13

https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=776 Summary: swift waits on failed job with no messages to user Product: Swift Version: 0.93 Platform: PC OS/Version: Other Status: ASSIGNED Severity: normal Priority: P2 Component: Error messages AssignedTo: davidk(a)ci.uchicago.edu ReportedBy: wilde(a)mcs.anl.gov CC: wozniak(a)mcs.anl.gov In debugging bug 774, I noticed that Swift kept saying "1 job submitted" in the progress indicator line on stdout/err, even though the Beagle PBS job had already failed. This is with a slightly old trunk; needs to be reproduced to see if its an additional problem. If you can find these in time, logs are in /lustre/beagle/wilde/swift/lab, dated evening of Apr 13 and morning of Apr 14. Its possible that these failures were due to the fact that I had commented out the project tag in my sites file, thinking that my default project was sufficient. Later I noticed in the log that my job was rejected by qsub due to invalid project. But this was not reported back to me on stdout/err. So Swift seemed to think the job was still queued, even though the failure was recorded in the detailed .log file. -- Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 1

[Bug 758] New: Cobalt provider on Eureka fails silently if user project is invalid
by bugzilla-daemon＠mcs.anl.gov 06 Dec '13

06 Dec '13

https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=758 Summary: Cobalt provider on Eureka fails silently if user project is invalid Product: Swift Version: 0.93 Platform: Other OS/Version: Linux Status: NEW Severity: normal Priority: P2 Component: Providers AssignedTo: davidk(a)ci.uchicago.edu ReportedBy: wilde(a)mcs.anl.gov CC: hategan(a)mcs.anl.gov, davidk(a)ci.uchicago.edu, wozniak(a)mcs.anl.gov A run with an invalid project in the sites file (an active project but invalid for the user) put 400 jobs in the submitted state even though the cobalt submission had silently failed. Look for the log on cobalt under /home/lzamboni/???/HiRAMTools/run00(1?) - I will help find this log if the problem is not trivial to replicate. -- Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 2

[Bug 740] New: Swift continues if the project you specify has no allocation
by bugzilla-daemon＠mcs.anl.gov 06 Dec '13

06 Dec '13

https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=740 Summary: Swift continues if the project you specify has no allocation Product: Swift Version: 0.93 Platform: PC OS/Version: Mac OS Status: NEW Severity: normal Priority: P2 Component: Providers AssignedTo: wozniak(a)mcs.anl.gov ReportedBy: jonmon(a)utexas.edu CC: hategan(a)mcs.anl.gov, wilde(a)mcs.anl.gov, davidk(a)ci.uchicago.edu, wozniak(a)mcs.anl.gov So currently(at least on Fusion) if the project you are trying to use has no allocation, Swift sits there with the same message printing to stdout. If you check the logs there should be a message saying your the project does not have an allocation. In my case, this message 2012-02-23 16:07:02,892-0600 DEBUG AbstractExecutor Output from qsub is: "ERROR: Project "startup-jonmon" has no allocation; can't run job." 2012-02-23 16:07:02,892-0600 INFO BlockTaskSubmitter Error submitting block task: Cannot submit job: Could not submit job (qsub reported an exit code of 1). ERROR: Project "startup-jonmon" has no allocation; can't run job. It would probably be best if Swift just haults after receiving this message and reports this message to the user, so the user does not think that jobs are actually waiting in the queue. -- Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 2

[Bug 1139] New: OSG/Condor failing in trunk
by bugzilla-daemon＠mcs.anl.gov 06 Dec '13

06 Dec '13

https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=1139 Summary: OSG/Condor failing in trunk Product: Swift Version: unspecified Platform: PC OS/Version: Mac OS Status: NEW Severity: major Priority: P2 Component: Providers AssignedTo: hategan(a)mcs.anl.gov ReportedBy: davidk(a)ci.uchicago.edu CC: hategan(a)mcs.anl.gov, davidk(a)ci.uchicago.edu, wozniak(a)mcs.anl.gov To run on OSG/UC3 we set the work directory set to ' . ' This is because on OSG, we have no idea of what the work directory will be in advance (condor creates a new work directory for each job). In 0.94 this configuration works well. In trunk, runs on OSG are failing. I believe the reason is related to the work directory. It seems like ' . ' is being interpreted locally where Swift is running, then it tries to create that same directory structure on the worker node, which fails. The errors I see are: Application exception: exception @ swift-int-staging.k, line: 183 2013-11-22 13:38:27,613-0600 DEBUG Cpu Failed mkdir /share/home/davidk/test: Permission denied at cscript8257449448675410471.pl line 1202 (/share/home/davidk/test is the uc3 directory where I am running swift) Full log at: http://web.ci.uchicago.edu/~davidk/logs/osg-trunk-run002.log -- Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 3

[Bug 1163] New: Lingering processes on remote systems
by bugzilla-daemon＠mcs.anl.gov 02 Dec '13

02 Dec '13

https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=1163 Summary: Lingering processes on remote systems Product: Swift Version: trunk Platform: PC OS/Version: Linux Status: NEW Severity: normal Priority: P2 Component: Testing AssignedTo: yadunandb(a)ci.uchicago.edu ReportedBy: yadunandb(a)ci.uchicago.edu CC: wilde(a)mcs.anl.gov, skenny(a)uchicago.edu, davidk(a)ci.uchicago.edu, ketan(a)mcs.anl.gov, yadunandb(a)ci.uchicago.edu Originally reported by Paul Fisher on the MCS Compute Nodes. 1. The test suite leaves several java instance on MCS nodes which are unresponsive and waste system memory. 2. David also reported a lingering worker process on UC3 which has been running for 6 days. This is probably unrelated to the above issue. * Add nice/renice to the remote scripts * Make sure the tests cleanup correctly (I killed the jvms before running jstack, so no logs to attach) -- Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 1

[Bug 1165] New: worker.pl failure causes it to use all memory available on system
by bugzilla-daemon＠mcs.anl.gov 30 Nov '13

30 Nov '13

https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=1165 Summary: worker.pl failure causes it to use all memory available on system Product: Swift Version: trunk Platform: PC OS/Version: Mac OS Status: NEW Severity: major Priority: P2 Component: General AssignedTo: hategan(a)mcs.anl.gov ReportedBy: davidk(a)ci.uchicago.edu When worker.pl fails, it gets stuck in a loop where it uses gigabytes of memory until a ulimit is eventually hit. This message keeps repeating forever in the worker log: Use of uninitialized value $SOFT_IMAGE_DST in concatenation (.) or string at /home/davidkelly999/swift-trunk/cog/modules/swift/dist/swift-svn/bin/worker.pl line 1889. flock() on closed filehandle $lock at /home/davidkelly999/swift-trunk/cog/modules/swift/dist/swift-svn/bin/worker.pl line 1890. Use of uninitialized value $SOFT_IMAGE_DST in concatenation (.) or string at /home/davidkelly999/swift-trunk/cog/modules/swift/dist/swift-svn/bin/worker.pl line 1889. flock() on closed filehandle at /home/davidkelly999/swift-trunk/cog/modules/swift/dist/swift-svn/bin/worker.pl line 1890. Use of uninitialized value $SOFT_IMAGE_DST in concatenation (.) or string at /home/davidkelly999/swift-trunk/cog/modules/swift/dist/swift-svn/bin/worker.pl line 1889. >From what I can tell, the function cleanSoftImage() fails when SOFT_IMAGE_DST is undefined, which then triggers a call to dieNicely(). dieNicely() then calls cleanSoftImage() and it gets stuck. -- Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 1