August 2014 - swift-bugs - mailman3.cels.anl.gov

[Bug 1299] New: initialParallelTasks not starting the set number of tasks
by bugzilla-daemon＠mcs.anl.gov 20 Sep '14

20 Sep '14

https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=1299 Summary: initialParallelTasks not starting the set number of tasks Product: Swift Version: trunk Platform: PC OS/Version: Linux Status: NEW Severity: normal Priority: P2 Component: General AssignedTo: hategan(a)mcs.anl.gov ReportedBy: yadunandb(a)ci.uchicago.edu I'm using the following configuration to get swift to start with 10 parallel tasks, which would require 5 jobs each with 2 tasks per node. The logs show that only 2 jobs start and remain active for the first several minutes. site.cloud { execution { type:"coaster" URL: "127.0.0.1" jobManager: "local:ec2-cloud" options { maxJobs: 10 tasksPerNode: 2 workerLoggingLevel: TRACE workerLoggingDirectory: /tmp #worker_image: t1.micro } } initialParallelTasks: 10 maxParallelTasks: 20 filesystem.type: swift workDirectory: /tmp/swift-trunk-test staging: "local" app.ALL {executable: "*"} } Swift log : http://users.rcc.uchicago.edu/~yadunand/swift.log Swift output : http://pastebin.com/zuJrRssa -- Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 5

[Bug 1177] New: Coaster-C-Client automatic coasters [enhancement]
by bugzilla-daemon＠mcs.anl.gov 08 Sep '14

08 Sep '14

https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=1177 Summary: Coaster-C-Client automatic coasters [enhancement] Product: Swift Version: trunk Platform: PC OS/Version: Linux Status: NEW Severity: normal Priority: P2 Component: General AssignedTo: hategan(a)mcs.anl.gov ReportedBy: yadunandb(a)ci.uchicago.edu Splitting bug 1133, to move Automatic coasters as a separate enhancement request. Currently the Coaster-C-Client goes through a separately started persistent coaster service. This comes with a few limitations: 1. A persistent coaster service needs to be started separately. 2. Missing the key coaster feature of expanding and shrinking worker pool. 3. Coaster's and workers remain open after execution, which can be undesirable in some cases. These features would be needed for SwiftT to take full advantage of coasters and also for language unification in the future. Currently we use scripts to submit SwiftT to resources which is very clumsy compared to what coasters can do. -- Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 3

[Bug 1249] New: Passive coaster-service failure on Trunk
by bugzilla-daemon＠mcs.anl.gov 08 Sep '14

08 Sep '14

https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=1249 Summary: Passive coaster-service failure on Trunk Product: Swift Version: trunk Platform: PC OS/Version: Linux Status: NEW Severity: normal Priority: P2 Component: General AssignedTo: hategan(a)mcs.anl.gov ReportedBy: yadunandb(a)ci.uchicago.edu I'm running Swift trunk swift-r7775 cog-r3892 on a couple of nodes with Swift running a small script connected to coaster-service in passive mode, and a single worker on a remote node (running worker.pl from the same version). This fails fairly reliably with the following error. 2014-04-18 21:54:52,853+0000 WARN RequestReply Failed to de-serialize remote exception java.io.EOFException at java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2598) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1318) at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370) at org.globus.cog.coaster.RequestReply.errorReceived(RequestReply.java:325) at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleReply(AbstractCoasterChannel.java:422) at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:234) at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97) at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56) 2014-04-18 21:54:53,630+0000 INFO RuntimeStats$ProgressTicker Submitting:4 Failed but can retry:1 2014-04-18 21:54:53,630+0000 INFO RuntimeStats$ProgressTicker HeapMax: 1037959168, CrtHeap: 27066368, UsedHeap: 8795736 -- Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 4

[Bug 907] New: Dangling coaster workers interfere with subsequent Swift runs
by bugzilla-daemon＠mcs.anl.gov 08 Sep '14

08 Sep '14

https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=907 Summary: Dangling coaster workers interfere with subsequent Swift runs Product: Swift Version: 0.94 Platform: PC OS/Version: Mac OS Status: ASSIGNED Severity: normal Priority: P2 Component: Providers AssignedTo: hategan(a)mcs.anl.gov ReportedBy: wilde(a)mcs.anl.gov CC: hategan(a)mcs.anl.gov, davidk(a)ci.uchicago.edu, wozniak(a)mcs.anl.gov Two problems here: - workers are not always cleaned up - such "dangling: workers interfere with subsequent runs Lorenzo has a log file (first message below) showing at least one manifestation of this problem. Second message indicates killing the abandoned workers solves the problems. Can we: - do a better job cleaning up abandoned workers? - make it so an abandoned worker cant connect to a subsequent server (or that if they do, no bad things, and only good things, happen?) --- ----- Forwarded Message ----- From: "Lorenzo Pesce" <lpesce(a)uchicago.edu> To: "Michael Wilde" <wilde(a)mcs.anl.gov> Sent: Saturday, December 15, 2012 12:43:06 PM Subject: 500 nodes failed again failed 500 node run: lpesce@login3:/lustre/beagle/GCNet/RG/Oreo/o080522_BS1> pwd /lustre/beagle/GCNet/RG/Oreo/o080522_BS1 -rw-rw-r-- 1 lpesce ci-users 549729889 2012-12-15 12:40 PackAndCausalTestCore_app-20121215-1801-2d9bnr57.log -rw-rw-r-- 1 lpesce ci-users 569522 2012-12-15 12:40 PackAndCausalTestCore_app.swift_2012-12-15_12:01:12_screen.log drwxrwxr-x 2 lpesce ci-users 4096 2012-12-15 12:40 PackAndCausalTestCore_app-20121215-1801-2d9bnr57.d In case you are interested in doing some post mortem. ----- Forwarded Message ----- From: "Lorenzo Pesce" <lpesce(a)uchicago.edu> To: "Michael Wilde" <wilde(a)mcs.anl.gov> Sent: Saturday, December 15, 2012 7:40:19 PM Subject: I know why it doesn't work sometimes I just realized that sometimes when swift gets killed or stops some jobs are left queued or running. These jobs create interference when the new jobs are sent. I tried a couple of times and if I clean up the queue, there seem to be no issues. I sent out a 500 nodes job and it has been happily churning for a while. -- Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 1

[Bug 1265] New: Errors at end of execution
by bugzilla-daemon＠mcs.anl.gov 08 Sep '14

08 Sep '14

https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=1265 Summary: Errors at end of execution Product: Swift Version: 0.95 Platform: PC OS/Version: Mac OS Status: NEW Severity: normal Priority: P2 Component: General AssignedTo: hategan(a)mcs.anl.gov ReportedBy: davidk(a)ci.uchicago.edu Saw this on Midway after doing a few tests, seems to happen sporadically at the end of a run. Full log at http://web.ci.uchicago.edu/~davidk/logs/errors-at-end.log. Swift 0.95 Branch SVN swift-r7787 (swift modified locally) cog-r3894 RunID: run001 Progress: Mon, 28 Apr 2014 17:48:08+0000 Progress: Mon, 28 Apr 2014 17:48:09+0000 Submitted:2 Final status:Mon, 28 Apr 2014 17:48:15+0000 Finished successfully:2 No events in 1s. Finding dependency loops... Waiting threads: ---- No dependency loops found. Waiting JVM threads: Thread "Thread-13" (0x00000036) TIMED_WAITING Stack trace: java.lang.Thread.sleep org.globus.cog.abstraction.coaster.service.CoasterService$1.run:338 Thread "pool-1-thread-24" (0x0000004b) WAITING waiting for java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject (0x1019399e) Stack trace: sun.misc.Unsafe.park java.util.concurrent.locks.LockSupport.park:186 java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await:2043 java.util.concurrent.LinkedBlockingQueue.take:442 java.util.concurrent.ThreadPoolExecutor.getTask:1068 java.util.concurrent.ThreadPoolExecutor.runWorker:1130 java.util.concurrent.ThreadPoolExecutor$Worker.run:615 java.lang.Thread.run:745 Thread "pool-1-thread-23" (0x0000004a) WAITING waiting for java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject (0x1019399e) Stack trace: sun.misc.Unsafe.park java.util.concurrent.locks.LockSupport.park:186 java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await:2043 java.util.concurrent.LinkedBlockingQueue.take:442 java.util.concurrent.ThreadPoolExecutor.getTask:1068 java.util.concurrent.ThreadPoolExecutor.runWorker:1130 java.util.concurrent.ThreadPoolExecutor$Worker.run:615 java.lang.Thread.run:745 Thread "pool-1-thread-22" (0x00000049) WAITING waiting for java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject (0x1019399e) Stack trace: sun.misc.Unsafe.park java.util.concurrent.locks.LockSupport.park:186 java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await:2043 java.util.concurrent.LinkedBlockingQueue.take:442 java.util.concurrent.ThreadPoolExecutor.getTask:1068 java.util.concurrent.ThreadPoolExecutor.runWorker:1130 java.util.concurrent.ThreadPoolExecutor$Worker.run:615 java.lang.Thread.run:745 Thread "pool-1-thread-21" (0x00000048) WAITING waiting for java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject (0x1019399e) Stack trace: sun.misc.Unsafe.park java.util.concurrent.locks.LockSupport.park:186 java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await:2043 java.util.concurrent.LinkedBlockingQueue.take:442 java.util.concurrent.ThreadPoolExecutor.getTask:1068 java.util.concurrent.ThreadPoolExecutor.runWorker:1130 java.util.concurrent.ThreadPoolExecutor$Worker.run:615 java.lang.Thread.run:745 Thread "pool-1-thread-20" (0x00000047) WAITING waiting for java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject (0x1019399e) Stack trace: sun.misc.Unsafe.park java.util.concurrent.locks.LockSupport.park:186 java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await:2043 java.util.concurrent.LinkedBlockingQueue.take:442 java.util.concurrent.ThreadPoolExecutor.getTask:1068 java.util.concurrent.ThreadPoolExecutor.runWorker:1130 java.util.concurrent.ThreadPoolExecutor$Worker.run:615 java.lang.Thread.run:745 Thread "NBS11" (0x00000046) WAITING waiting for java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject (0x6815bc53) Stack trace: sun.misc.Unsafe.park java.util.concurrent.locks.LockSupport.park:186 java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await:2043 java.util.concurrent.LinkedBlockingQueue.take:442 java.util.concurrent.ThreadPoolExecutor.getTask:1068 java.util.concurrent.ThreadPoolExecutor.runWorker:1130 java.util.concurrent.ThreadPoolExecutor$Worker.run:615 java.lang.Thread.run:745 Thread "pool-1-thread-19" (0x00000045) WAITING waiting for java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject (0x1019399e) Stack trace: sun.misc.Unsafe.park java.util.concurrent.locks.LockSupport.park:186 java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await:2043 java.util.concurrent.LinkedBlockingQueue.take:442 java.util.concurrent.ThreadPoolExecutor.getTask:1068 java.util.concurrent.ThreadPoolExecutor.runWorker:1130 java.util.concurrent.ThreadPoolExecutor$Worker.run:615 java.lang.Thread.run:745 Thread "PullThread" (0x00000043) TIMED_WAITING waiting for org.globus.cog.abstraction.coaster.service.job.manager.PullThread (0x1688e9ff) Stack trace: java.lang.Object.wait org.globus.cog.abstraction.coaster.service.job.manager.PullThread.mwait:151 org.globus.cog.abstraction.coaster.service.job.manager.PullThread.run:122 Thread "Timer-3" (0x00000041) TIMED_WAITING waiting for java.util.TaskQueue (0x11c58239) Stack trace: java.lang.Object.wait java.util.TimerThread.mainLoop:552 java.util.TimerThread.run:505 Thread "Slurm provider queue poller" (0x00000040) TIMED_WAITING Stack trace: java.lang.Thread.sleep org.globus.cog.abstraction.impl.scheduler.common.AbstractQueuePoller.run:90 java.lang.Thread.run:745 Thread "Block Submitter" (0x0000003e) WAITING waiting for java.util.LinkedList (0x3de8754c) Stack trace: java.lang.Object.wait java.lang.Object.wait:503 org.globus.cog.abstraction.coaster.service.job.manager.BlockTaskSubmitter.run:56 Thread "Block Queue Processor" (0x0000003d) TIMED_WAITING waiting for java.util.ArrayList (0x686cf054) Stack trace: java.lang.Object.wait org.globus.cog.abstraction.coaster.service.job.manager.BlockQueueProcessor.run:149 Thread "Timer-2" (0x0000003c) WAITING waiting for java.util.TaskQueue (0x4816505b) Stack trace: java.lang.Object.wait java.lang.Object.wait:503 java.util.TimerThread.mainLoop:526 java.util.TimerThread.run:505 Thread "Piped Channel Sender" (0x0000003b) WAITING waiting for java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject (0x30f5ab56) Stack trace: sun.misc.Unsafe.park java.util.concurrent.locks.LockSupport.park:186 java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await:2043 java.util.concurrent.LinkedBlockingQueue.take:442 org.globus.cog.coaster.channels.AbstractPipedChannel$Sender.run:113 Thread "Piped Channel Sender" (0x0000003a) WAITING waiting for java.lang.UNIXProcess (0x6ed485be) Monitors held: org.globus.cog.abstraction.impl.scheduler.slurm.execution.JobSubmissionTaskHandler (0x50c3b436) java.util.ArrayList (0x606c6ea6) Stack trace: java.lang.Object.wait java.lang.Object.wait:503 java.lang.UNIXProcess.waitFor:263 org.globus.cog.abstraction.impl.scheduler.common.AbstractExecutor.cancel:208 org.globus.cog.abstraction.impl.scheduler.common.AbstractJobSubmissionTaskHandler.cancel:84 -> locked org.globus.cog.abstraction.impl.scheduler.slurm.execution.JobSubmissionTaskHandler (0x50c3b436) org.globus.cog.abstraction.impl.common.AbstractTaskHandler.cancel:69 org.globus.cog.abstraction.impl.common.task.ExecutionTaskHandler.cancel:106 org.globus.cog.abstraction.impl.common.task.ExecutionTaskHandler.cancel:95 org.globus.cog.abstraction.coaster.service.job.manager.BlockTaskSubmitter.cancel:46 org.globus.cog.abstraction.coaster.service.job.manager.Block.forceShutdown:342 -> locked java.util.ArrayList (0x606c6ea6) org.globus.cog.abstraction.coaster.service.job.manager.Block.shutdown:322 org.globus.cog.abstraction.coaster.service.job.manager.BlockQueueProcessor.shutdownBlocks:807 org.globus.cog.abstraction.coaster.service.job.manager.BlockQueueProcessor.shutdown:796 org.globus.cog.abstraction.coaster.service.job.manager.JobQueue.shutdown:119 org.globus.cog.abstraction.coaster.service.CoasterService.shutdown:318 org.globus.cog.abstraction.coaster.service.ServiceShutdownHandler.requestComplete:28 org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted:88 org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest:527 org.globus.cog.coaster.channels.AbstractPipedChannel.actualSend:86 org.globus.cog.coaster.channels.AbstractPipedChannel$Sender.run:115 Thread "Local Queue Processor" (0x00000039) WAITING waiting for java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject (0x4298b5ca) Stack trace: sun.misc.Unsafe.park java.util.concurrent.locks.LockSupport.park:186 java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await:2043 java.util.concurrent.LinkedBlockingQueue.take:442 org.globus.cog.abstraction.coaster.service.job.manager.AbstractQueueProcessor.take:43 org.globus.cog.abstraction.coaster.service.job.manager.LocalQueueProcessor.run:37 Thread "Timer-1" (0x00000035) TIMED_WAITING waiting for java.util.TaskQueue (0x6a9173dd) Stack trace: java.lang.Object.wait java.util.TimerThread.mainLoop:552 java.util.TimerThread.run:505 Thread "Thread-11" (0x00000034) TIMED_WAITING waiting for org.globus.cog.abstraction.coaster.service.CoasterService (0x734de8d3) Stack trace: java.lang.Object.wait org.globus.cog.abstraction.coaster.service.CoasterService.waitFor:251 org.globus.cog.abstraction.coaster.service.CoasterService.main:432 org.globus.cog.abstraction.impl.execution.coaster.ServiceManager$1.run:208 java.lang.Thread.run:745 Thread "NBS10" (0x0000002f) WAITING waiting for java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject (0x6815bc53) Stack trace: sun.misc.Unsafe.park java.util.concurrent.locks.LockSupport.park:186 java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await:2043 java.util.concurrent.LinkedBlockingQueue.take:442 java.util.concurrent.ThreadPoolExecutor.getTask:1068 java.util.concurrent.ThreadPoolExecutor.runWorker:1130 java.util.concurrent.ThreadPoolExecutor$Worker.run:615 java.lang.Thread.run:745 Thread "NBS9" (0x0000002e) WAITING waiting for java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject (0x6815bc53) Stack trace: sun.misc.Unsafe.park java.util.concurrent.locks.LockSupport.park:186 java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await:2043 java.util.concurrent.LinkedBlockingQueue.take:442 java.util.concurrent.ThreadPoolExecutor.getTask:1068 java.util.concurrent.ThreadPoolExecutor.runWorker:1130 java.util.concurrent.ThreadPoolExecutor$Worker.run:615 java.lang.Thread.run:745 Thread "pool-1-thread-18" (0x0000002d) WAITING waiting for java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject (0x1019399e) Stack trace: sun.misc.Unsafe.park java.util.concurrent.locks.LockSupport.park:186 java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await:2043 java.util.concurrent.LinkedBlockingQueue.take:442 java.util.concurrent.ThreadPoolExecutor.getTask:1068 java.util.concurrent.ThreadPoolExecutor.runWorker:1130 java.util.concurrent.ThreadPoolExecutor$Worker.run:615 java.lang.Thread.run:745 Thread "pool-1-thread-17" (0x0000002c) WAITING waiting for java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject (0x1019399e) Stack trace: sun.misc.Unsafe.park java.util.concurrent.locks.LockSupport.park:186 java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await:2043 java.util.concurrent.LinkedBlockingQueue.take:442 java.util.concurrent.ThreadPoolExecutor.getTask:1068 java.util.concurrent.ThreadPoolExecutor.runWorker:1130 java.util.concurrent.ThreadPoolExecutor$Worker.run:615 java.lang.Thread.run:745 Thread "NBS8" (0x0000002b) WAITING waiting for java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject (0x6815bc53) Stack trace: sun.misc.Unsafe.park java.util.concurrent.locks.LockSupport.park:186 java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await:2043 java.util.concurrent.LinkedBlockingQueue.take:442 java.util.concurrent.ThreadPoolExecutor.getTask:1068 java.util.concurrent.ThreadPoolExecutor.runWorker:1130 java.util.concurrent.ThreadPoolExecutor$Worker.run:615 java.lang.Thread.run:745 Thread "pool-1-thread-16" (0x0000002a) WAITING waiting for java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject (0x1019399e) Stack trace: sun.misc.Unsafe.park java.util.concurrent.locks.LockSupport.park:186 java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await:2043 java.util.concurrent.LinkedBlockingQueue.take:442 java.util.concurrent.ThreadPoolExecutor.getTask:1068 java.util.concurrent.ThreadPoolExecutor.runWorker:1130 java.util.concurrent.ThreadPoolExecutor$Worker.run:615 java.lang.Thread.run:745 Thread "NBS7" (0x00000029) WAITING waiting for java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject (0x6815bc53) Stack trace: sun.misc.Unsafe.park java.util.concurrent.locks.LockSupport.park:186 java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await:2043 java.util.concurrent.LinkedBlockingQueue.take:442 java.util.concurrent.ThreadPoolExecutor.getTask:1068 java.util.concurrent.ThreadPoolExecutor.runWorker:1130 java.util.concurrent.ThreadPoolExecutor$Worker.run:615 java.lang.Thread.run:745 Thread "pool-1-thread-15" (0x00000028) WAITING waiting for java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject (0x1019399e) Stack trace: sun.misc.Unsafe.park java.util.concurrent.locks.LockSupport.park:186 java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await:2043 java.util.concurrent.LinkedBlockingQueue.take:442 java.util.concurrent.ThreadPoolExecutor.getTask:1068 java.util.concurrent.ThreadPoolExecutor.runWorker:1130 java.util.concurrent.ThreadPoolExecutor$Worker.run:615 java.lang.Thread.run:745 Thread "NBS6" (0x00000027) WAITING waiting for java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject (0x6815bc53) Stack trace: sun.misc.Unsafe.park java.util.concurrent.locks.LockSupport.park:186 java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await:2043 java.util.concurrent.LinkedBlockingQueue.take:442 java.util.concurrent.ThreadPoolExecutor.getTask:1068 java.util.concurrent.ThreadPoolExecutor.runWorker:1130 java.util.concurrent.ThreadPoolExecutor$Worker.run:615 java.lang.Thread.run:745 Thread "pool-1-thread-14" (0x00000026) WAITING waiting for java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject (0x1019399e) Stack trace: sun.misc.Unsafe.park java.util.concurrent.locks.LockSupport.park:186 java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await:2043 java.util.concurrent.LinkedBlockingQueue.take:442 java.util.concurrent.ThreadPoolExecutor.getTask:1068 java.util.concurrent.ThreadPoolExecutor.runWorker:1130 java.util.concurrent.ThreadPoolExecutor$Worker.run:615 java.lang.Thread.run:745 Thread "NBS5" (0x00000024) WAITING waiting for java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject (0x6815bc53) Stack trace: sun.misc.Unsafe.park java.util.concurrent.locks.LockSupport.park:186 java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await:2043 java.util.concurrent.LinkedBlockingQueue.take:442 java.util.concurrent.ThreadPoolExecutor.getTask:1068 java.util.concurrent.ThreadPoolExecutor.runWorker:1130 java.util.concurrent.ThreadPoolExecutor$Worker.run:615 java.lang.Thread.run:745 Thread "pool-1-thread-13" (0x00000023) WAITING waiting for java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject (0x1019399e) Stack trace: sun.misc.Unsafe.park java.util.concurrent.locks.LockSupport.park:186 java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await:2043 java.util.concurrent.LinkedBlockingQueue.take:442 java.util.concurrent.ThreadPoolExecutor.getTask:1068 java.util.concurrent.ThreadPoolExecutor.runWorker:1130 java.util.concurrent.ThreadPoolExecutor$Worker.run:615 java.lang.Thread.run:745 Thread "NBS4" (0x00000021) WAITING waiting for java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject (0x6815bc53) Stack trace: sun.misc.Unsafe.park java.util.concurrent.locks.LockSupport.park:186 java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await:2043 java.util.concurrent.LinkedBlockingQueue.take:442 java.util.concurrent.ThreadPoolExecutor.getTask:1068 java.util.concurrent.ThreadPoolExecutor.runWorker:1130 java.util.concurrent.ThreadPoolExecutor$Worker.run:615 java.lang.Thread.run:745 Thread "pool-1-thread-12" (0x00000020) WAITING waiting for java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject (0x1019399e) Stack trace: sun.misc.Unsafe.park java.util.concurrent.locks.LockSupport.park:186 java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await:2043 java.util.concurrent.LinkedBlockingQueue.take:442 java.util.concurrent.ThreadPoolExecutor.getTask:1068 java.util.concurrent.ThreadPoolExecutor.runWorker:1130 java.util.concurrent.ThreadPoolExecutor$Worker.run:615 java.lang.Thread.run:745 Thread "NBS3" (0x0000001f) WAITING waiting for java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject (0x6815bc53) Stack trace: sun.misc.Unsafe.park java.util.concurrent.locks.LockSupport.park:186 java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await:2043 java.util.concurrent.LinkedBlockingQueue.take:442 java.util.concurrent.ThreadPoolExecutor.getTask:1068 java.util.concurrent.ThreadPoolExecutor.runWorker:1130 java.util.concurrent.ThreadPoolExecutor$Worker.run:615 java.lang.Thread.run:745 Thread "pool-1-thread-11" (0x0000001e) WAITING waiting for java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject (0x1019399e) Stack trace: sun.misc.Unsafe.park java.util.concurrent.locks.LockSupport.park:186 java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await:2043 java.util.concurrent.LinkedBlockingQueue.take:442 java.util.concurrent.ThreadPoolExecutor.getTask:1068 java.util.concurrent.ThreadPoolExecutor.runWorker:1130 java.util.concurrent.ThreadPoolExecutor$Worker.run:615 java.lang.Thread.run:745 Thread "pool-1-thread-10" (0x0000001d) WAITING waiting for java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject (0x1019399e) Stack trace: sun.misc.Unsafe.park java.util.concurrent.locks.LockSupport.park:186 java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await:2043 java.util.concurrent.LinkedBlockingQueue.take:442 java.util.concurrent.ThreadPoolExecutor.getTask:1068 java.util.concurrent.ThreadPoolExecutor.runWorker:1130 java.util.concurrent.ThreadPoolExecutor$Worker.run:615 java.lang.Thread.run:745 Thread "NBS2" (0x0000001c) WAITING waiting for java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject (0x6815bc53) Stack trace: sun.misc.Unsafe.park java.util.concurrent.locks.LockSupport.park:186 java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await:2043 java.util.concurrent.LinkedBlockingQueue.take:442 java.util.concurrent.ThreadPoolExecutor.getTask:1068 java.util.concurrent.ThreadPoolExecutor.runWorker:1130 java.util.concurrent.ThreadPoolExecutor$Worker.run:615 java.lang.Thread.run:745 Thread "pool-1-thread-9" (0x0000001b) WAITING waiting for java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject (0x1019399e) Stack trace: sun.misc.Unsafe.park java.util.concurrent.locks.LockSupport.park:186 java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await:2043 java.util.concurrent.LinkedBlockingQueue.take:442 java.util.concurrent.ThreadPoolExecutor.getTask:1068 java.util.concurrent.ThreadPoolExecutor.runWorker:1130 java.util.concurrent.ThreadPoolExecutor$Worker.run:615 java.lang.Thread.run:745 Thread "NBS1" (0x0000001a) WAITING waiting for java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject (0x6815bc53) Stack trace: sun.misc.Unsafe.park java.util.concurrent.locks.LockSupport.park:186 java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await:2043 java.util.concurrent.LinkedBlockingQueue.take:442 java.util.concurrent.ThreadPoolExecutor.getTask:1068 java.util.concurrent.ThreadPoolExecutor.runWorker:1130 java.util.concurrent.ThreadPoolExecutor$Worker.run:615 java.lang.Thread.run:745 Thread "pool-1-thread-8" (0x00000019) WAITING waiting for java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject (0x1019399e) Stack trace: sun.misc.Unsafe.park java.util.concurrent.locks.LockSupport.park:186 java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await:2043 java.util.concurrent.LinkedBlockingQueue.take:442 java.util.concurrent.ThreadPoolExecutor.getTask:1068 java.util.concurrent.ThreadPoolExecutor.runWorker:1130 java.util.concurrent.ThreadPoolExecutor$Worker.run:615 java.lang.Thread.run:745 Thread "pool-1-thread-7" (0x00000018) WAITING waiting for java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject (0x1019399e) Stack trace: sun.misc.Unsafe.park java.util.concurrent.locks.LockSupport.park:186 java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await:2043 java.util.concurrent.LinkedBlockingQueue.take:442 java.util.concurrent.ThreadPoolExecutor.getTask:1068 java.util.concurrent.ThreadPoolExecutor.runWorker:1130 java.util.concurrent.ThreadPoolExecutor$Worker.run:615 java.lang.Thread.run:745 Thread "pool-1-thread-6" (0x00000017) WAITING waiting for java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject (0x1019399e) Stack trace: sun.misc.Unsafe.park java.util.concurrent.locks.LockSupport.park:186 java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await:2043 java.util.concurrent.LinkedBlockingQueue.take:442 java.util.concurrent.ThreadPoolExecutor.getTask:1068 java.util.concurrent.ThreadPoolExecutor.runWorker:1130 java.util.concurrent.ThreadPoolExecutor$Worker.run:615 java.lang.Thread.run:745 Thread "pool-1-thread-5" (0x00000016) WAITING waiting for java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject (0x1019399e) Stack trace: sun.misc.Unsafe.park java.util.concurrent.locks.LockSupport.park:186 java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await:2043 java.util.concurrent.LinkedBlockingQueue.take:442 java.util.concurrent.ThreadPoolExecutor.getTask:1068 java.util.concurrent.ThreadPoolExecutor.runWorker:1130 java.util.concurrent.ThreadPoolExecutor$Worker.run:615 java.lang.Thread.run:745 Thread "pool-1-thread-4" (0x00000015) WAITING waiting for java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject (0x1019399e) Stack trace: sun.misc.Unsafe.park java.util.concurrent.locks.LockSupport.park:186 java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await:2043 java.util.concurrent.LinkedBlockingQueue.take:442 java.util.concurrent.ThreadPoolExecutor.getTask:1068 java.util.concurrent.ThreadPoolExecutor.runWorker:1130 java.util.concurrent.ThreadPoolExecutor$Worker.run:615 java.lang.Thread.run:745 Thread "pool-1-thread-3" (0x00000014) WAITING waiting for java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject (0x1019399e) Stack trace: sun.misc.Unsafe.park java.util.concurrent.locks.LockSupport.park:186 java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await:2043 java.util.concurrent.LinkedBlockingQueue.take:442 java.util.concurrent.ThreadPoolExecutor.getTask:1068 java.util.concurrent.ThreadPoolExecutor.runWorker:1130 java.util.concurrent.ThreadPoolExecutor$Worker.run:615 java.lang.Thread.run:745 Thread "Restart Log Sync" (0x00000011) WAITING waiting for org.globus.cog.karajan.compiled.nodes.restartLog.SyncThread (0x18d03cf8) Stack trace: java.lang.Object.wait java.lang.Object.wait:503 org.globus.cog.karajan.compiled.nodes.restartLog.SyncThread.run:47 Thread "Scheduler" (0x0000000f) TIMED_WAITING waiting for org.griphyn.vdl.karajan.VDSAdaptiveScheduler (0x7f5c17fa) Stack trace: java.lang.Object.wait org.globus.cog.karajan.scheduler.LateBindingScheduler.sleep:298 org.globus.cog.karajan.scheduler.LateBindingScheduler.run:250 Thread "Overloaded Host Monitor" (0x00000010) TIMED_WAITING Stack trace: java.lang.Thread.sleep org.globus.cog.karajan.scheduler.OverloadedHostMonitor.run:47 Thread "pool-1-thread-2" (0x0000000e) WAITING waiting for java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject (0x1019399e) Stack trace: sun.misc.Unsafe.park java.util.concurrent.locks.LockSupport.park:186 java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await:2043 java.util.concurrent.LinkedBlockingQueue.take:442 java.util.concurrent.ThreadPoolExecutor.getTask:1068 java.util.concurrent.ThreadPoolExecutor.runWorker:1130 java.util.concurrent.ThreadPoolExecutor$Worker.run:615 java.lang.Thread.run:745 Thread "Timer-0" (0x0000000d) TIMED_WAITING waiting for java.util.TaskQueue (0x0f52ca82) Stack trace: java.lang.Object.wait java.util.TimerThread.mainLoop:552 java.util.TimerThread.run:505 Thread "NBS0" (0x0000000c) WAITING waiting for java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject (0x6815bc53) Stack trace: sun.misc.Unsafe.park java.util.concurrent.locks.LockSupport.park:186 java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await:2043 java.util.concurrent.LinkedBlockingQueue.take:442 java.util.concurrent.ThreadPoolExecutor.getTask:1068 java.util.concurrent.ThreadPoolExecutor.runWorker:1130 java.util.concurrent.ThreadPoolExecutor$Worker.run:615 java.lang.Thread.run:745 Thread "pool-1-thread-1" (0x0000000b) WAITING waiting for java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject (0x1019399e) Stack trace: sun.misc.Unsafe.park java.util.concurrent.locks.LockSupport.park:186 java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await:2043 java.util.concurrent.LinkedBlockingQueue.take:442 java.util.concurrent.ThreadPoolExecutor.getTask:1068 java.util.concurrent.ThreadPoolExecutor.runWorker:1130 java.util.concurrent.ThreadPoolExecutor$Worker.run:615 java.lang.Thread.run:745 Thread "LWT Scheduler Timer" (0x0000000a) WAITING waiting for java.util.TaskQueue (0x046d24aa) Stack trace: java.lang.Object.wait java.lang.Object.wait:503 java.util.TimerThread.mainLoop:526 java.util.TimerThread.run:505 Thread "AsyncAppender-Dispatcher-Thread-0" (0x00000008) WAITING waiting for java.util.ArrayList (0x1982172b) Stack trace: java.lang.Object.wait java.lang.Object.wait:503 org.apache.log4j.AsyncAppender$Dispatcher.run:548 java.lang.Thread.run:745 Thread "Finalizer" (0x00000003) WAITING waiting for java.lang.ref.ReferenceQueue$Lock (0x2bbf1522) Stack trace: java.lang.Object.wait java.lang.ref.ReferenceQueue.remove:135 java.lang.ref.ReferenceQueue.remove:151 java.lang.ref.Finalizer$FinalizerThread.run:189 Thread "Reference Handler" (0x00000002) WAITING waiting for java.lang.ref.Reference$Lock (0x0e04819e) Stack trace: java.lang.Object.wait java.lang.Object.wait:503 java.lang.ref.Reference$ReferenceHandler.run:133 Thread "main" (0x00000001) WAITING waiting for org.globus.cog.abstraction.coaster.service.CoasterService$1 (0x68f559e5) Monitors held: java.lang.Class (0x49d9b342) Stack trace: java.lang.Object.wait java.lang.Thread.join:1281 java.lang.Thread.join:1355 java.lang.ApplicationShutdownHooks.runHooks:106 java.lang.ApplicationShutdownHooks$1.run:46 java.lang.Shutdown.runHooks:123 java.lang.Shutdown.sequence:167 java.lang.Shutdown.exit:212 -> locked java.lang.Class (0x49d9b342) java.lang.Runtime.exit:109 java.lang.System.exit:962 org.griphyn.vdl.karajan.Loader.main:198 -- Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 2

[Bug 1327] New: swift-convert-config error
by bugzilla-daemon＠mcs.anl.gov 08 Sep '14

08 Sep '14

https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=1327 Summary: swift-convert-config error Product: Swift Version: trunk Platform: All OS/Version: All Status: NEW Severity: normal Priority: P2 Component: Error messages AssignedTo: hategan(a)mcs.anl.gov ReportedBy: ketan(a)mcs.anl.gov CC: davidkelly(a)uchicago.edu, wozniak(a)mcs.anl.gov Trying to convert to new config with swift-convert-config which is giving the following error: $ swift-convert-config -sites.file sites.xml -tc.file tc -config cf -out swift.conf [Error] sites.xml, line 2, col 61: cvc-elt.1: Cannot find the declaration of element 'config'. [Error] sites.xml, line 2, col 61: cvc-elt.1: Cannot find the declaration of element 'config'. 2014-08-29 10:49:21,876-0500 INFO SiteCatalogParser [Error] sites.xml, line 2, col 61: cvc-elt.1: Cannot find the declaration of element 'config'. The sites file is: <?xml version="1.0" encoding="UTF-8"?> <config xmlns="http://www.ci.uchicago.edu/swift/SwiftSites"> <pool handle="localhost"> <filesystem provider="local" /> <execution provider="local" /> <workdirectory>/tmp/swift.workdir</workdirectory> <profile namespace="karajan" key="jobThrottle">0</profile> </pool> </config> -- Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 7

[Bug 871] New: Multiple client support for persistent coaster service
by bugzilla-daemon＠mcs.anl.gov 08 Sep '14

08 Sep '14

https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=871 Summary: Multiple client support for persistent coaster service Product: Swift Version: trunk Platform: All OS/Version: All Status: NEW Severity: enhancement Priority: Medium Component: Providers AssignedTo: hategan(a)mcs.anl.gov ReportedBy: hategan(a)mcs.anl.gov CC: hategan(a)mcs.anl.gov, davidk(a)ci.uchicago.edu, wozniak(a)mcs.anl.gov A persistent coaster service should support multiple clients. There should be three modes of operation: 1. Manual workers - jobsPerNode (and possibly other relevant settings) should be specifiable either when the service is started or when the workers are started 2. Shared pool - clients would be able to re-use existing workers - all settings should be specified when starting the service and clients should not be able to override them 3. Per-client pool - each client will get its own pool of workers that is only used for one client connection - settings are specified by the client -- Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 2

[Bug 1311] New: Waiting threads at end of run using cloud providers
by bugzilla-daemon＠mcs.anl.gov 03 Sep '14

03 Sep '14

https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=1311 Summary: Waiting threads at end of run using cloud providers Product: Swift Version: trunk Platform: PC OS/Version: Linux Status: NEW Severity: normal Priority: P2 Component: General AssignedTo: hategan(a)mcs.anl.gov ReportedBy: yadunandb(a)ci.uchicago.edu When running tasks using cloud providers (aws/gce), I see tasks complete followed by a strack trace and list of waiting threads. This happens in almost half the runs. The timestamps from the cancel logs show that around 2s were spent for cancelling the nodes on the cloud. The script exits with exitcode 0 once the jobs are sent cancel requests. Here's the log : http://users.rcc.uchicago.edu/~yadunand/run023_waiting_threads.log Here's a debug log from the provider : http://users.rcc.uchicago.edu/~yadunand/run023_waiting_threads.debug -- Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 6

[Bug 1223] New: Hang checker messages after task was completed
by bugzilla-daemon＠mcs.anl.gov 16 Aug '14

16 Aug '14

https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=1223 Summary: Hang checker messages after task was completed Product: Swift Version: 0.94 Platform: PC OS/Version: Mac OS Status: NEW Severity: normal Priority: P2 Component: General AssignedTo: hategan(a)mcs.anl.gov ReportedBy: davidk(a)ci.uchicago.edu I was just testing a simple script that runs hostname once and records the output to a file. The job was submitted to condor and ran, but the printed hang checker messages after it was finished. Log at http://web.ci.uchicago.edu/~davidk/logs/hostname-20140320-0141-8dscb5n4.log ... Final status: Thu, 20 Mar 2014 01:52:05 +0000 Finished successfully:1 No events in 10s. Finding dependency loops... Registered futures: Waiting threads: ---- No dependency loops found. Waiting JVM threads: Thread "Thread-12" (0x00000020) TIMED_WAITING Stack trace: java.lang.Thread.sleep org.globus.cog.abstraction.coaster.service.CoasterService$1.run:323 Thread "PullThread" (0x000000a5) TIMED_WAITING waiting for org.globus.cog.abstraction.coaster.service.job.manager.PullThread (0x4b8bbb61) Stack trace: java.lang.Object.wait org.globus.cog.abstraction.coaster.service.job.manager.PullThread.mwait:151 org.globus.cog.abstraction.coaster.service.job.manager.PullThread.run:122 Thread "Timer-3" (0x000000a3) TIMED_WAITING waiting for java.util.TaskQueue (0x2d9f90e3) Stack trace: java.lang.Object.wait java.util.TimerThread.mainLoop:531 java.util.TimerThread.run:484 Thread "Condor provider queue poller" (0x0000002a) TIMED_WAITING Stack trace: java.lang.Thread.sleep org.globus.cog.abstraction.impl.scheduler.common.AbstractQueuePoller.run:90 java.lang.Thread.run:679 Thread "Block Submitter" (0x00000028) WAITING waiting for java.util.LinkedList (0x46e13e7b) Stack trace: java.lang.Object.wait java.lang.Object.wait:502 org.globus.cog.abstraction.coaster.service.job.manager.BlockTaskSubmitter.run:56 Thread "Block Queue Processor" (0x00000027) TIMED_WAITING waiting for java.util.ArrayList (0x6dcd2197) Stack trace: java.lang.Object.wait org.globus.cog.abstraction.coaster.service.job.manager.BlockQueueProcessor.run:148 Thread "Timer-2" (0x00000026) WAITING waiting for java.util.TaskQueue (0x365878d2) Stack trace: java.lang.Object.wait java.lang.Object.wait:502 java.util.TimerThread.mainLoop:505 java.util.TimerThread.run:484 Thread "Piped Channel Sender" (0x00000025) WAITING waiting for java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject (0x6a643edd) Stack trace: sun.misc.Unsafe.park java.util.concurrent.locks.LockSupport.park:186 java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await:2043 java.util.concurrent.LinkedBlockingQueue.take:386 org.globus.cog.karajan.workflow.service.channels.AbstractPipedChannel$Sender.run:113 Thread "Local Queue Processor" (0x00000023) WAITING waiting for java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject (0x29949f78) Stack trace: sun.misc.Unsafe.park java.util.concurrent.locks.LockSupport.park:186 java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await:2043 java.util.concurrent.LinkedBlockingQueue.take:386 org.globus.cog.abstraction.coaster.service.job.manager.AbstractQueueProcessor.take:43 org.globus.cog.abstraction.coaster.service.job.manager.LocalQueueProcessor.run:37 Thread "Timer-1" (0x0000001f) WAITING waiting for java.util.TaskQueue (0x24d37b87) Stack trace: java.lang.Object.wait java.lang.Object.wait:502 java.util.TimerThread.mainLoop:505 java.util.TimerThread.run:484 Thread "Thread-10" (0x0000001e) TIMED_WAITING waiting for org.globus.cog.abstraction.coaster.service.CoasterService (0x3852eef4) Stack trace: java.lang.Object.wait org.globus.cog.abstraction.coaster.service.CoasterService.waitFor:251 org.globus.cog.abstraction.coaster.service.CoasterService.main:417 org.globus.cog.abstraction.impl.execution.coaster.ServiceManager$1.run:206 java.lang.Thread.run:679 Thread "NBS3" (0x00000017) WAITING waiting for java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject (0x2b302b69) Stack trace: sun.misc.Unsafe.park java.util.concurrent.locks.LockSupport.park:186 java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await:2043 java.util.concurrent.LinkedBlockingQueue.take:386 java.util.concurrent.ThreadPoolExecutor.getTask:1069 java.util.concurrent.ThreadPoolExecutor.runWorker:1131 java.util.concurrent.ThreadPoolExecutor$Worker.run:615 java.lang.Thread.run:679 Thread "NBS2" (0x00000016) WAITING waiting for java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject (0x2b302b69) Stack trace: sun.misc.Unsafe.park java.util.concurrent.locks.LockSupport.park:186 java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await:2043 java.util.concurrent.LinkedBlockingQueue.take:386 java.util.concurrent.ThreadPoolExecutor.getTask:1069 java.util.concurrent.ThreadPoolExecutor.runWorker:1131 java.util.concurrent.ThreadPoolExecutor$Worker.run:615 java.lang.Thread.run:679 Thread "NBS1" (0x00000015) WAITING waiting for java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject (0x2b302b69) Stack trace: sun.misc.Unsafe.park java.util.concurrent.locks.LockSupport.park:186 java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await:2043 java.util.concurrent.LinkedBlockingQueue.take:386 java.util.concurrent.ThreadPoolExecutor.getTask:1069 java.util.concurrent.ThreadPoolExecutor.runWorker:1131 java.util.concurrent.ThreadPoolExecutor$Worker.run:615 java.lang.Thread.run:679 Thread "Scheduler" (0x00000010) TIMED_WAITING waiting for org.griphyn.vdl.karajan.VDSAdaptiveScheduler (0x6ac67a88) Stack trace: java.lang.Object.wait org.globus.cog.karajan.scheduler.LateBindingScheduler.sleep:303 org.globus.cog.karajan.scheduler.LateBindingScheduler.run:257 Thread "Progress ticker" (0x00000014) TIMED_WAITING Stack trace: java.lang.Thread.sleep org.griphyn.vdl.karajan.lib.RuntimeStats$ProgressTicker.run:183 Thread "Restart Log Sync" (0x00000012) WAITING waiting for org.globus.cog.karajan.workflow.nodes.restartLog.SyncThread (0x677210a5) Stack trace: java.lang.Object.wait java.lang.Object.wait:502 org.globus.cog.karajan.workflow.nodes.restartLog.SyncThread.run:47 Thread "Overloaded Host Monitor" (0x00000011) TIMED_WAITING Stack trace: java.lang.Thread.sleep org.globus.cog.karajan.scheduler.OverloadedHostMonitor.run:47 Thread "Timer-0" (0x0000000f) TIMED_WAITING waiting for java.util.TaskQueue (0x71a2f5b1) Stack trace: java.lang.Object.wait java.util.TimerThread.mainLoop:531 java.util.TimerThread.run:484 Thread "NBS0" (0x0000000e) WAITING waiting for java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject (0x2b302b69) Stack trace: sun.misc.Unsafe.park java.util.concurrent.locks.LockSupport.park:186 java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await:2043 java.util.concurrent.LinkedBlockingQueue.take:386 java.util.concurrent.ThreadPoolExecutor.getTask:1069 java.util.concurrent.ThreadPoolExecutor.runWorker:1131 java.util.concurrent.ThreadPoolExecutor$Worker.run:615 java.lang.Thread.run:679 Thread "pool-1-thread-4" (0x0000000d) WAITING waiting for java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject (0x01b071c0) Stack trace: sun.misc.Unsafe.park java.util.concurrent.locks.LockSupport.park:186 java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await:2043 java.util.concurrent.LinkedBlockingQueue.take:386 java.util.concurrent.ThreadPoolExecutor.getTask:1069 java.util.concurrent.ThreadPoolExecutor.runWorker:1131 java.util.concurrent.ThreadPoolExecutor$Worker.run:615 java.lang.Thread.run:679 Thread "pool-1-thread-3" (0x0000000c) WAITING waiting for java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject (0x01b071c0) Stack trace: sun.misc.Unsafe.park java.util.concurrent.locks.LockSupport.park:186 java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await:2043 java.util.concurrent.LinkedBlockingQueue.take:386 java.util.concurrent.ThreadPoolExecutor.getTask:1069 java.util.concurrent.ThreadPoolExecutor.runWorker:1131 java.util.concurrent.ThreadPoolExecutor$Worker.run:615 java.lang.Thread.run:679 Thread "pool-1-thread-2" (0x0000000b) WAITING waiting for java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject (0x01b071c0) Stack trace: sun.misc.Unsafe.park java.util.concurrent.locks.LockSupport.park:186 java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await:2043 java.util.concurrent.LinkedBlockingQueue.take:386 java.util.concurrent.ThreadPoolExecutor.getTask:1069 java.util.concurrent.ThreadPoolExecutor.runWorker:1131 java.util.concurrent.ThreadPoolExecutor$Worker.run:615 java.lang.Thread.run:679 Thread "pool-1-thread-1" (0x0000000a) WAITING waiting for java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject (0x01b071c0) Stack trace: sun.misc.Unsafe.park java.util.concurrent.locks.LockSupport.park:186 java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await:2043 java.util.concurrent.LinkedBlockingQueue.take:386 java.util.concurrent.ThreadPoolExecutor.getTask:1069 java.util.concurrent.ThreadPoolExecutor.runWorker:1131 java.util.concurrent.ThreadPoolExecutor$Worker.run:615 java.lang.Thread.run:679 Thread "AsyncAppender-Dispatcher-Thread-0" (0x00000008) WAITING waiting for java.util.ArrayList (0x66fbf51d) Stack trace: java.lang.Object.wait java.lang.Object.wait:502 org.apache.log4j.AsyncAppender$Dispatcher.run:548 java.lang.Thread.run:679 Thread "Finalizer" (0x00000003) WAITING waiting for java.lang.ref.ReferenceQueue$Lock (0x279977bd) Stack trace: java.lang.Object.wait java.lang.ref.ReferenceQueue.remove:133 java.lang.ref.ReferenceQueue.remove:149 java.lang.ref.Finalizer$FinalizerThread.run:189 Thread "Reference Handler" (0x00000002) WAITING waiting for java.lang.ref.Reference$Lock (0x4c842d9a) Stack trace: java.lang.Object.wait java.lang.Object.wait:502 java.lang.ref.Reference$ReferenceHandler.run:133 Thread "main" (0x00000001) WAITING waiting for org.globus.cog.abstraction.coaster.service.CoasterService$1 (0x5b6b9e62) Monitors held: java.lang.Class (0x79ce00b4) Stack trace: java.lang.Object.wait java.lang.Thread.join:1203 java.lang.Thread.join:1256 java.lang.ApplicationShutdownHooks.run:97 java.lang.Shutdown.runHooks:106 java.lang.Shutdown.sequence:150 java.lang.Shutdown.exit:195 -> locked java.lang.Class (0x79ce00b4) java.lang.Runtime.exit:107 java.lang.System.exit:923 org.griphyn.vdl.karajan.Loader.main:247 -- Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 3

[Bug 1317] New: Time configs in swift.conf
by bugzilla-daemon＠mcs.anl.gov 16 Aug '14

16 Aug '14

https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=1317 Summary: Time configs in swift.conf Product: Swift Version: trunk Platform: PC OS/Version: Linux Status: NEW Severity: normal Priority: P2 Component: General AssignedTo: hategan(a)mcs.anl.gov ReportedBy: yadunandb(a)ci.uchicago.edu maxJobTime is not documented. When I do set a maxJobTime and maxWallTime I get the following error: Execution failed: Exception in date: Arguments: [] Host: blues Directory: blues_sanity-run009/jobs/a/date-av8x8mul exception @ swift-int.k, line: 469 Caused by: Job walltime > maxTime - reserve (00:05:00 > 00:00:00) site.blues.execution.options.maxJobTime : "60" site.blues.app.date.maxWallTime : "5" I could not find a config variable that matches with maxTime in that error message, and Job walltime appears to be taken from the app maxwalltime rather that the maxJobTime. I also find that the tests are using jobMaxTime while the config schema uses maxJobTime. Documentation would really help. For now, I'm fixing the remote site test configs with maxJobTime. When I set maxJobTime to "05:00:00", I get the following error: Caused by: null Caused by: org.globus.cog.abstraction.impl.common.task.TaskSubmissionException: Could not submit job Caused by: org.globus.cog.coaster.ProtocolException: Failed to set configuration: For input string: "05:00:00" Caused by: org.globus.cog.coaster.RemoteException: Failed to set configuration: For input string: "05:00:00" Caused by: java.lang.NumberFormatException: For input string: "05:00:00" at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65) at java.lang.Integer.parseInt(Integer.java:492) at java.lang.Integer.valueOf(Integer.java:582) at org.globus.cog.abstraction.coaster.service.job.manager.Settings.set(Settings.java:610) at org.globus.cog.abstraction.coaster.service.job.manager.Settings.set(Settings.java:584) at org.globus.cog.abstraction.coaster.service.ServiceConfigurationHandler.requestComplete(ServiceConfigurationHandler.java:37) at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88) at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527) at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:173) at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70) -- Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 2