https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=793 Summary: VDSAdaptiveScheduler deadlock Product: Swift Version: trunk Platform: PC OS/Version: Linux Status: NEW Severity: normal Priority: P2 Component: General AssignedTo: [email protected] ReportedBy: [email protected] Ran into this while testing with the DSSAT scripts.. 2012-06-10 03:27:34 Full thread dump Java HotSpot(TM) 64-Bit Server VM (20.1-b02 mixed mode): "Attach Listener" daemon prio=10 tid=0x000000005f80b800 nid=0x1144 waiting on condition [0x0000000000000000] java.lang.Thread.State: RUNNABLE Locked ownable synchronizers: - None "Timer-3" daemon prio=10 tid=0x000000005f87d000 nid=0x449b in Object.wait() [0x0000000043949000] java.lang.Thread.State: TIMED_WAITING (on object monitor) at java.lang.Object.wait(Native Method) at java.util.TimerThread.mainLoop(Timer.java:509) - locked <0x0000000784270b98> (a java.util.TaskQueue) at java.util.TimerThread.run(Timer.java:462) Locked ownable synchronizers: - None "I/O Queue" daemon prio=10 tid=0x0000000060d02000 nid=0x4499 in Object.wait() [0x0000000043848000] java.lang.Thread.State: TIMED_WAITING (on object monitor) at java.lang.Object.wait(Native Method) at org.globus.cog.abstraction.impl.file.coaster.buffers.Buffers.run(Buffers.java:188) - locked <0x00000007844e9170> (a org.globus.cog.abstraction.impl.file.coaster.buffers.Buffers) Locked ownable synchronizers: - None "I/O Queue" daemon prio=10 tid=0x000000005f1f8800 nid=0x4498 in Object.wait() [0x0000000043747000] java.lang.Thread.State: TIMED_WAITING (on object monitor) at java.lang.Object.wait(Native Method) at org.globus.cog.abstraction.impl.file.coaster.buffers.Buffers.run(Buffers.java:188) - locked <0x00000007844e9618> (a org.globus.cog.abstraction.impl.file.coaster.buffers.Buffers) Locked ownable synchronizers: - None "NBS7" daemon prio=10 tid=0x00002aaab9345000 nid=0x4497 waiting for monitor entry [0x0000000043646000] java.lang.Thread.State: BLOCKED (on object monitor) at org.globus.cog.karajan.scheduler.LateBindingScheduler.getContacts(LateBindingScheduler.java:714) - waiting to lock <0x0000000781a03768> (a org.griphyn.vdl.karajan.VDSAdaptiveScheduler) at org.globus.cog.karajan.scheduler.WeightedHostScoreScheduler.statusChanged(WeightedHostScoreScheduler.java:395) at org.griphyn.vdl.karajan.VDSAdaptiveScheduler.statusChanged(VDSAdaptiveScheduler.java:421) at org.globus.cog.abstraction.impl.common.task.TaskImpl.notifyListeners(TaskImpl.java:240) at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:228) at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:257) at org.globus.cog.abstraction.impl.execution.coaster.JobSubmissionTaskHandler.submit(JobSubmissionTaskHandler.java:104) at org.globus.cog.abstraction.impl.common.AbstractTaskHandler.submit(AbstractTaskHandler.java:45) at org.globus.cog.karajan.scheduler.submitQueue.NonBlockingSubmit.run(NonBlockingSubmit.java:97) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Locked ownable synchronizers: - <0x00000007844e9c80> (a java.util.concurrent.locks.ReentrantLock$NonfairSync) "NBS6" daemon prio=10 tid=0x00002aaab48a8800 nid=0x4496 waiting on condition [0x0000000043545000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x0000000781aa53a8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) at java.lang.Thread.run(Thread.java:662) Locked ownable synchronizers: - None "NBS5" daemon prio=10 tid=0x000000005fd0a000 nid=0x4495 waiting on condition [0x0000000043444000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x0000000781aa53a8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) at java.lang.Thread.run(Thread.java:662) Locked ownable synchronizers: - None "NIO Sender" daemon prio=10 tid=0x00002aaab824a000 nid=0x4494 runnable [0x0000000043343000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) - locked <0x00000007844598e8> (a sun.nio.ch.Util$2) - locked <0x00000007844598d8> (a java.util.Collections$UnmodifiableSet) - locked <0x00000007844596b0> (a sun.nio.ch.EPollSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84) at org.globus.cog.karajan.workflow.service.channels.NIOSender.run(NIOSender.java:91) Locked ownable synchronizers: - None "Timer-2" daemon prio=10 tid=0x000000005fd09000 nid=0x4493 in Object.wait() [0x0000000043242000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on <0x0000000784271070> (a java.util.TaskQueue) at java.lang.Object.wait(Object.java:485) at java.util.TimerThread.mainLoop(Timer.java:483) - locked <0x0000000784271070> (a java.util.TaskQueue) at java.util.TimerThread.run(Timer.java:462) Locked ownable synchronizers: - None "NIO Multiplexer" daemon prio=10 tid=0x000000005f8db000 nid=0x4492 waiting for monitor entry [0x0000000043141000] java.lang.Thread.State: BLOCKED (on object monitor) at org.globus.cog.karajan.scheduler.LateBindingScheduler.getContacts(LateBindingScheduler.java:714) - waiting to lock <0x0000000781a03768> (a org.griphyn.vdl.karajan.VDSAdaptiveScheduler) at org.globus.cog.karajan.scheduler.WeightedHostScoreScheduler.statusChanged(WeightedHostScoreScheduler.java:395) at org.griphyn.vdl.karajan.VDSAdaptiveScheduler.statusChanged(VDSAdaptiveScheduler.java:421) at org.globus.cog.abstraction.impl.common.task.TaskImpl.notifyListeners(TaskImpl.java:240) at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:228) at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.setStatus(NotificationManager.java:117) at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:88) at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:50) at org.globus.cog.karajan.workflow.service.handlers.RequestHandler.receiveCompleted(RequestHandler.java:91) at org.globus.cog.karajan.workflow.service.channels.AbstractKarajanChannel.handleRequest(AbstractKarajanChannel.java:505) at org.globus.cog.karajan.workflow.service.channels.AbstractStreamKarajanChannel.stepNIO(AbstractStreamKarajanChannel.java:239) at org.globus.cog.karajan.workflow.service.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97) at org.globus.cog.karajan.workflow.service.channels.NIOMultiplexer.run(NIOMultiplexer.java:56) Locked ownable synchronizers: - None "Timer-1" daemon prio=10 tid=0x000000005fdfd000 nid=0x4491 waiting for monitor entry [0x0000000043040000] java.lang.Thread.State: BLOCKED (on object monitor) at org.globus.cog.karajan.scheduler.LateBindingScheduler.getContacts(LateBindingScheduler.java:714) - waiting to lock <0x0000000781a03768> (a org.griphyn.vdl.karajan.VDSAdaptiveScheduler) at org.globus.cog.karajan.scheduler.WeightedHostScoreScheduler.statusChanged(WeightedHostScoreScheduler.java:395) at org.griphyn.vdl.karajan.VDSAdaptiveScheduler.statusChanged(VDSAdaptiveScheduler.java:421) at org.globus.cog.abstraction.impl.common.task.TaskImpl.notifyListeners(TaskImpl.java:240) at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:228) at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:257) at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:218) at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:257) at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:218) at org.globus.cog.abstraction.impl.execution.coaster.JobSubmissionTaskHandler.errorReceived(JobSubmissionTaskHandler.java:239) at org.globus.cog.karajan.workflow.service.commands.Command.errorReceived(Command.java:203) at org.globus.cog.karajan.workflow.service.commands.Command.handleTimeout(Command.java:258) at org.globus.cog.karajan.workflow.service.channels.AbstractKarajanChannel.checkTimeouts(AbstractKarajanChannel.java:134) at org.globus.cog.karajan.workflow.service.channels.AbstractKarajanChannel.checkTimeouts(AbstractKarajanChannel.java:124) at org.globus.cog.karajan.workflow.service.channels.AbstractKarajanChannel$1.run(AbstractKarajanChannel.java:118) at java.util.TimerThread.mainLoop(Timer.java:512) at java.util.TimerThread.run(Timer.java:462) Locked ownable synchronizers: - None "NBS4" daemon prio=10 tid=0x00002aaab8249000 nid=0x4490 waiting on condition [0x0000000042f3f000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x0000000781aa53a8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) at java.lang.Thread.run(Thread.java:662) Locked ownable synchronizers: - None "NBS3" daemon prio=10 tid=0x00002aaab8d1f000 nid=0x448f waiting on condition [0x0000000042c3c000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x0000000781aa53a8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) at java.lang.Thread.run(Thread.java:662) Locked ownable synchronizers: - None "NBS2" daemon prio=10 tid=0x00002aaab84e0800 nid=0x448d waiting on condition [0x0000000042d3d000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x0000000781aa53a8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) at java.lang.Thread.run(Thread.java:662) Locked ownable synchronizers: - None "NBS1" daemon prio=10 tid=0x00002aaab8fa9800 nid=0x448b waiting for monitor entry [0x0000000042b3b000] java.lang.Thread.State: BLOCKED (on object monitor) at org.globus.cog.karajan.scheduler.LateBindingScheduler.getContacts(LateBindingScheduler.java:714) - waiting to lock <0x0000000781a03768> (a org.griphyn.vdl.karajan.VDSAdaptiveScheduler) at org.globus.cog.karajan.scheduler.WeightedHostScoreScheduler.statusChanged(WeightedHostScoreScheduler.java:395) at org.griphyn.vdl.karajan.VDSAdaptiveScheduler.statusChanged(VDSAdaptiveScheduler.java:421) at org.globus.cog.abstraction.impl.common.task.TaskImpl.notifyListeners(TaskImpl.java:240) at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:228) at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:257) at org.globus.cog.abstraction.impl.execution.coaster.JobSubmissionTaskHandler.submit(JobSubmissionTaskHandler.java:104) at org.globus.cog.abstraction.impl.common.AbstractTaskHandler.submit(AbstractTaskHandler.java:45) at org.globus.cog.karajan.scheduler.submitQueue.NonBlockingSubmit.run(NonBlockingSubmit.java:97) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Locked ownable synchronizers: - <0x00000007844eb270> (a java.util.concurrent.locks.ReentrantLock$NonfairSync) "Scheduler" prio=10 tid=0x00002aaab9a93800 nid=0x448a waiting for monitor entry [0x0000000042e3e000] java.lang.Thread.State: BLOCKED (on object monitor) at org.globus.cog.karajan.scheduler.OverloadedHostMonitor.add(OverloadedHostMonitor.java:38) - waiting to lock <0x0000000781a08788> (a java.util.HashSet) at org.globus.cog.karajan.scheduler.WeightedHostSet.checkOverloaded(WeightedHostSet.java:94) at org.globus.cog.karajan.scheduler.WeightedHostSet.changeScore(WeightedHostSet.java:61) - locked <0x0000000781a08478> (a java.util.TreeSet) at org.globus.cog.karajan.scheduler.WeightedHostScoreScheduler.factorScore(WeightedHostScoreScheduler.java:149) - locked <0x0000000781a03768> (a org.griphyn.vdl.karajan.VDSAdaptiveScheduler) at org.globus.cog.karajan.scheduler.WeightedHostScoreScheduler.factorMultiple(WeightedHostScoreScheduler.java:491) at org.globus.cog.karajan.scheduler.WeightedHostScoreScheduler.factorSubmission(WeightedHostScoreScheduler.java:464) at org.globus.cog.karajan.scheduler.WeightedHostScoreScheduler.submitBoundToServices(WeightedHostScoreScheduler.java:387) at org.globus.cog.karajan.scheduler.LateBindingScheduler.submitUnbound(LateBindingScheduler.java:384) at org.globus.cog.karajan.scheduler.LateBindingScheduler.run(LateBindingScheduler.java:270) Locked ownable synchronizers: - None "Progress ticker" daemon prio=10 tid=0x000000005f927000 nid=0x3945 waiting on condition [0x0000000042a3a000] java.lang.Thread.State: TIMED_WAITING (sleeping) at java.lang.Thread.sleep(Native Method) at org.griphyn.vdl.karajan.lib.RuntimeStats$ProgressTicker.run(RuntimeStats.java:180) Locked ownable synchronizers: - None "Restart Log Sync" daemon prio=10 tid=0x000000005fbab000 nid=0x3944 in Object.wait() [0x0000000042939000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:485) at org.globus.cog.karajan.workflow.nodes.restartLog.SyncThread.run(SyncThread.java:47) - locked <0x00000007819d7d60> (a org.globus.cog.karajan.workflow.nodes.restartLog.SyncThread) Locked ownable synchronizers: - None "Overloaded Host Monitor" daemon prio=10 tid=0x000000005f160000 nid=0x3943 waiting for monitor entry [0x0000000042838000] java.lang.Thread.State: BLOCKED (on object monitor) at org.globus.cog.karajan.scheduler.WeightedHostScoreScheduler.raiseTasksFinished(WeightedHostScoreScheduler.java:513) - waiting to lock <0x0000000781a03768> (a org.griphyn.vdl.karajan.VDSAdaptiveScheduler) at org.globus.cog.karajan.scheduler.WeightedHostScoreScheduler.removeOverloaded(WeightedHostScoreScheduler.java:509) at org.globus.cog.karajan.scheduler.OverloadedHostMonitor.run(OverloadedHostMonitor.java:54) - locked <0x0000000781a08788> (a java.util.HashSet) Locked ownable synchronizers: - None "Timer-0" daemon prio=10 tid=0x000000005fa5d000 nid=0x3942 in Object.wait() [0x0000000042737000] java.lang.Thread.State: TIMED_WAITING (on object monitor) at java.lang.Object.wait(Native Method) at java.util.TimerThread.mainLoop(Timer.java:509) - locked <0x0000000781aa49c0> (a java.util.TaskQueue) at java.util.TimerThread.run(Timer.java:462) Locked ownable synchronizers: - None "NBS0" daemon prio=10 tid=0x00002aaab44e9800 nid=0x3941 waiting on condition [0x0000000042636000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x0000000781aa53a8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) at java.lang.Thread.run(Thread.java:662) Locked ownable synchronizers: - None "pool-1-thread-8" prio=10 tid=0x00002aaab452f000 nid=0x393c waiting for monitor entry [0x0000000042535000] java.lang.Thread.State: BLOCKED (on object monitor) at org.globus.cog.karajan.scheduler.LateBindingScheduler.enqueue(LateBindingScheduler.java:195) - waiting to lock <0x0000000781a03768> (a org.griphyn.vdl.karajan.VDSAdaptiveScheduler) at org.griphyn.vdl.karajan.VDSAdaptiveScheduler.enqueue(VDSAdaptiveScheduler.java:128) at org.griphyn.vdl.karajan.lib.Execute.submitScheduled(Execute.java:67) at org.globus.cog.karajan.workflow.nodes.grid.GridExec.submitTask(GridExec.java:218) at org.globus.cog.karajan.workflow.nodes.grid.AbstractGridNode.post(AbstractGridNode.java:61) at org.globus.cog.karajan.workflow.nodes.AbstractSequentialWithArguments.completed(AbstractSequentialWithArguments.java:194) at org.globus.cog.karajan.workflow.nodes.FlowNode.complete(FlowNode.java:214) at org.globus.cog.karajan.workflow.nodes.FlowContainer.post(FlowContainer.java:58) at org.globus.cog.karajan.workflow.nodes.AbstractSequentialWithArguments.completed(AbstractSequentialWithArguments.java:194) at org.globus.cog.karajan.workflow.nodes.user.UserDefinedElement.completed(UserDefinedElement.java:286) at org.globus.cog.karajan.workflow.nodes.user.SequentialImplicitExecutionUDE.completed(SequentialImplicitExecutionUDE.java:89) at org.globus.cog.karajan.workflow.nodes.FlowNode.complete(FlowNode.java:214) at org.globus.cog.karajan.workflow.nodes.FlowContainer.post(FlowContainer.java:58) at org.globus.cog.karajan.workflow.nodes.AbstractSequentialWithArguments.completed(AbstractSequentialWithArguments.java:194) at org.globus.cog.karajan.workflow.nodes.PartialArgumentsContainer.nonArgChildCompleted(PartialArgumentsContainer.java:105) at org.globus.cog.karajan.workflow.nodes.PartialArgumentsContainer.completed(PartialArgumentsContainer.java:86) at org.globus.cog.karajan.workflow.nodes.FlowNode.complete(FlowNode.java:214) at org.globus.cog.karajan.workflow.nodes.FlowContainer.post(FlowContainer.java:58) at org.globus.cog.karajan.workflow.nodes.functions.AbstractFunction.post(AbstractFunction.java:28) at org.globus.cog.karajan.workflow.nodes.Sequential.startNext(Sequential.java:29) at org.globus.cog.karajan.workflow.nodes.Sequential.executeChildren(Sequential.java:20) at org.globus.cog.karajan.workflow.nodes.FlowContainer.execute(FlowContainer.java:63) at org.globus.cog.karajan.workflow.nodes.FlowNode.restart(FlowNode.java:139) at org.globus.cog.karajan.workflow.nodes.FlowNode.start(FlowNode.java:197) at org.globus.cog.karajan.workflow.events.EventBus.start(EventBus.java:98) at org.globus.cog.karajan.workflow.events.EventTargetPair.run(EventTargetPair.java:40) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Locked ownable synchronizers: - <0x0000000781aa56e0> (a java.util.concurrent.locks.ReentrantLock$NonfairSync) "pool-1-thread-7" prio=10 tid=0x000000005f1f7800 nid=0x393b waiting for monitor entry [0x0000000042434000] java.lang.Thread.State: BLOCKED (on object monitor) at org.globus.cog.karajan.scheduler.LateBindingScheduler.enqueue(LateBindingScheduler.java:195) - waiting to lock <0x0000000781a03768> (a org.griphyn.vdl.karajan.VDSAdaptiveScheduler) at org.griphyn.vdl.karajan.VDSAdaptiveScheduler.enqueue(VDSAdaptiveScheduler.java:128) at org.griphyn.vdl.karajan.lib.Execute.submitScheduled(Execute.java:67) at org.globus.cog.karajan.workflow.nodes.grid.GridExec.submitTask(GridExec.java:218) at org.globus.cog.karajan.workflow.nodes.grid.AbstractGridNode.post(AbstractGridNode.java:61) at org.globus.cog.karajan.workflow.nodes.AbstractSequentialWithArguments.completed(AbstractSequentialWithArguments.java:194) at org.globus.cog.karajan.workflow.nodes.FlowNode.complete(FlowNode.java:214) at org.globus.cog.karajan.workflow.nodes.FlowContainer.post(FlowContainer.java:58) at org.globus.cog.karajan.workflow.nodes.AbstractSequentialWithArguments.completed(AbstractSequentialWithArguments.java:194) at org.globus.cog.karajan.workflow.nodes.user.UserDefinedElement.completed(UserDefinedElement.java:286) at org.globus.cog.karajan.workflow.nodes.user.SequentialImplicitExecutionUDE.completed(SequentialImplicitExecutionUDE.java:89) at org.globus.cog.karajan.workflow.nodes.FlowNode.complete(FlowNode.java:214) at org.globus.cog.karajan.workflow.nodes.FlowContainer.post(FlowContainer.java:58) at org.globus.cog.karajan.workflow.nodes.AbstractSequentialWithArguments.completed(AbstractSequentialWithArguments.java:194) at org.globus.cog.karajan.workflow.nodes.PartialArgumentsContainer.nonArgChildCompleted(PartialArgumentsContainer.java:105) at org.globus.cog.karajan.workflow.nodes.PartialArgumentsContainer.completed(PartialArgumentsContainer.java:86) at org.globus.cog.karajan.workflow.nodes.FlowNode.complete(FlowNode.java:214) at org.globus.cog.karajan.workflow.nodes.FlowContainer.post(FlowContainer.java:58) at org.globus.cog.karajan.workflow.nodes.functions.AbstractFunction.post(AbstractFunction.java:28) at org.globus.cog.karajan.workflow.nodes.Sequential.startNext(Sequential.java:29) at org.globus.cog.karajan.workflow.nodes.Sequential.executeChildren(Sequential.java:20) at org.globus.cog.karajan.workflow.nodes.FlowContainer.execute(FlowContainer.java:63) at org.globus.cog.karajan.workflow.nodes.FlowNode.restart(FlowNode.java:139) at org.globus.cog.karajan.workflow.nodes.FlowNode.start(FlowNode.java:197) at org.globus.cog.karajan.workflow.events.EventBus.start(EventBus.java:98) at org.globus.cog.karajan.workflow.events.EventTargetPair.run(EventTargetPair.java:40) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Locked ownable synchronizers: - <0x0000000781af29c8> (a java.util.concurrent.locks.ReentrantLock$NonfairSync) "pool-1-thread-6" prio=10 tid=0x000000005f1f6000 nid=0x393a waiting for monitor entry [0x0000000042333000] java.lang.Thread.State: BLOCKED (on object monitor) at org.globus.cog.karajan.scheduler.LateBindingScheduler.enqueue(LateBindingScheduler.java:195) - waiting to lock <0x0000000781a03768> (a org.griphyn.vdl.karajan.VDSAdaptiveScheduler) at org.griphyn.vdl.karajan.VDSAdaptiveScheduler.enqueue(VDSAdaptiveScheduler.java:128) at org.globus.cog.karajan.workflow.nodes.grid.AllocateHost.partialArgumentsEvaluated(AllocateHost.java:67) at org.globus.cog.karajan.workflow.nodes.PartialArgumentsContainer.completed(PartialArgumentsContainer.java:78) at org.globus.cog.karajan.workflow.nodes.FlowNode.complete(FlowNode.java:214) at org.globus.cog.karajan.workflow.nodes.FlowContainer.post(FlowContainer.java:58) at org.globus.cog.karajan.workflow.nodes.functions.Argument.post(Argument.java:48) at org.globus.cog.karajan.workflow.nodes.AbstractSequentialWithArguments.completed(AbstractSequentialWithArguments.java:194) at org.globus.cog.karajan.workflow.nodes.FlowNode.complete(FlowNode.java:214) at org.globus.cog.karajan.workflow.nodes.FlowContainer.post(FlowContainer.java:58) at org.griphyn.vdl.karajan.lib.VDLFunction.post(VDLFunction.java:83) at org.globus.cog.karajan.workflow.nodes.AbstractSequentialWithArguments.completed(AbstractSequentialWithArguments.java:194) at org.globus.cog.karajan.workflow.nodes.FlowNode.complete(FlowNode.java:214) at org.globus.cog.karajan.workflow.nodes.FlowContainer.post(FlowContainer.java:58) at org.globus.cog.karajan.workflow.nodes.functions.AbstractFunction.post(AbstractFunction.java:28) at org.globus.cog.karajan.workflow.nodes.Sequential.startNext(Sequential.java:29) at org.globus.cog.karajan.workflow.nodes.Sequential.executeChildren(Sequential.java:20) at org.globus.cog.karajan.workflow.nodes.FlowContainer.execute(FlowContainer.java:63) at org.globus.cog.karajan.workflow.nodes.FlowNode.restart(FlowNode.java:139) at org.globus.cog.karajan.workflow.nodes.FlowNode.start(FlowNode.java:197) at org.globus.cog.karajan.workflow.events.EventBus.start(EventBus.java:98) at org.globus.cog.karajan.workflow.events.EventTargetPair.run(EventTargetPair.java:40) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Locked ownable synchronizers: - <0x0000000781af2400> (a java.util.concurrent.locks.ReentrantLock$NonfairSync) "pool-1-thread-5" prio=10 tid=0x000000005f165000 nid=0x3939 waiting on condition [0x0000000042232000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x000000078014b5c0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) at java.lang.Thread.run(Thread.java:662) Locked ownable synchronizers: - None "pool-1-thread-4" prio=10 tid=0x000000005f162000 nid=0x3938 waiting for monitor entry [0x0000000042131000] java.lang.Thread.State: BLOCKED (on object monitor) at org.globus.cog.karajan.scheduler.LateBindingScheduler.enqueue(LateBindingScheduler.java:195) - waiting to lock <0x0000000781a03768> (a org.griphyn.vdl.karajan.VDSAdaptiveScheduler) at org.griphyn.vdl.karajan.VDSAdaptiveScheduler.enqueue(VDSAdaptiveScheduler.java:128) at org.griphyn.vdl.karajan.lib.Execute.submitScheduled(Execute.java:67) at org.globus.cog.karajan.workflow.nodes.grid.GridExec.submitTask(GridExec.java:218) at org.globus.cog.karajan.workflow.nodes.grid.AbstractGridNode.post(AbstractGridNode.java:61) at org.globus.cog.karajan.workflow.nodes.AbstractSequentialWithArguments.completed(AbstractSequentialWithArguments.java:194) at org.globus.cog.karajan.workflow.nodes.FlowNode.complete(FlowNode.java:214) at org.globus.cog.karajan.workflow.nodes.FlowContainer.post(FlowContainer.java:58) at org.globus.cog.karajan.workflow.nodes.AbstractSequentialWithArguments.completed(AbstractSequentialWithArguments.java:194) at org.globus.cog.karajan.workflow.nodes.user.UserDefinedElement.completed(UserDefinedElement.java:286) at org.globus.cog.karajan.workflow.nodes.user.SequentialImplicitExecutionUDE.completed(SequentialImplicitExecutionUDE.java:89) at org.globus.cog.karajan.workflow.nodes.FlowNode.complete(FlowNode.java:214) at org.globus.cog.karajan.workflow.nodes.FlowContainer.post(FlowContainer.java:58) at org.globus.cog.karajan.workflow.nodes.AbstractSequentialWithArguments.completed(AbstractSequentialWithArguments.java:194) at org.globus.cog.karajan.workflow.nodes.PartialArgumentsContainer.nonArgChildCompleted(PartialArgumentsContainer.java:105) at org.globus.cog.karajan.workflow.nodes.PartialArgumentsContainer.completed(PartialArgumentsContainer.java:86) at org.globus.cog.karajan.workflow.nodes.FlowNode.complete(FlowNode.java:214) at org.globus.cog.karajan.workflow.nodes.FlowContainer.post(FlowContainer.java:58) at org.globus.cog.karajan.workflow.nodes.functions.AbstractFunction.post(AbstractFunction.java:28) at org.globus.cog.karajan.workflow.nodes.Sequential.startNext(Sequential.java:29) at org.globus.cog.karajan.workflow.nodes.Sequential.executeChildren(Sequential.java:20) at org.globus.cog.karajan.workflow.nodes.FlowContainer.execute(FlowContainer.java:63) at org.globus.cog.karajan.workflow.nodes.FlowNode.restart(FlowNode.java:139) at org.globus.cog.karajan.workflow.nodes.FlowNode.start(FlowNode.java:197) at org.globus.cog.karajan.workflow.events.EventBus.start(EventBus.java:98) at org.globus.cog.karajan.workflow.events.EventTargetPair.run(EventTargetPair.java:40) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Locked ownable synchronizers: - <0x0000000781af2160> (a java.util.concurrent.locks.ReentrantLock$NonfairSync) "pool-1-thread-3" prio=10 tid=0x000000005f161000 nid=0x3937 waiting for monitor entry [0x0000000042030000] java.lang.Thread.State: BLOCKED (on object monitor) at org.globus.cog.karajan.scheduler.LateBindingScheduler.enqueue(LateBindingScheduler.java:195) - waiting to lock <0x0000000781a03768> (a org.griphyn.vdl.karajan.VDSAdaptiveScheduler) at org.griphyn.vdl.karajan.VDSAdaptiveScheduler.enqueue(VDSAdaptiveScheduler.java:128) at org.globus.cog.karajan.workflow.nodes.grid.AllocateHost.partialArgumentsEvaluated(AllocateHost.java:67) at org.globus.cog.karajan.workflow.nodes.PartialArgumentsContainer.completed(PartialArgumentsContainer.java:78) at org.globus.cog.karajan.workflow.nodes.FlowNode.complete(FlowNode.java:214) at org.globus.cog.karajan.workflow.nodes.FlowContainer.post(FlowContainer.java:58) at org.globus.cog.karajan.workflow.nodes.functions.Argument.post(Argument.java:48) at org.globus.cog.karajan.workflow.nodes.AbstractSequentialWithArguments.completed(AbstractSequentialWithArguments.java:194) at org.globus.cog.karajan.workflow.nodes.FlowNode.complete(FlowNode.java:214) at org.globus.cog.karajan.workflow.nodes.FlowContainer.post(FlowContainer.java:58) at org.griphyn.vdl.karajan.lib.VDLFunction.post(VDLFunction.java:83) at org.globus.cog.karajan.workflow.nodes.AbstractSequentialWithArguments.completed(AbstractSequentialWithArguments.java:194) at org.globus.cog.karajan.workflow.nodes.FlowNode.complete(FlowNode.java:214) at org.globus.cog.karajan.workflow.nodes.FlowContainer.post(FlowContainer.java:58) at org.globus.cog.karajan.workflow.nodes.functions.AbstractFunction.post(AbstractFunction.java:28) at org.globus.cog.karajan.workflow.nodes.Sequential.startNext(Sequential.java:29) at org.globus.cog.karajan.workflow.nodes.Sequential.executeChildren(Sequential.java:20) at org.globus.cog.karajan.workflow.nodes.FlowContainer.execute(FlowContainer.java:63) at org.globus.cog.karajan.workflow.nodes.FlowNode.restart(FlowNode.java:139) at org.globus.cog.karajan.workflow.nodes.FlowNode.start(FlowNode.java:197) at org.globus.cog.karajan.workflow.events.EventBus.start(EventBus.java:98) at org.globus.cog.karajan.workflow.events.EventTargetPair.run(EventTargetPair.java:40) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Locked ownable synchronizers: - <0x0000000781af1ec0> (a java.util.concurrent.locks.ReentrantLock$NonfairSync) "pool-1-thread-2" prio=10 tid=0x000000005fbe0000 nid=0x3936 waiting for monitor entry [0x0000000041a58000] java.lang.Thread.State: BLOCKED (on object monitor) at org.globus.cog.karajan.scheduler.LateBindingScheduler.enqueue(LateBindingScheduler.java:195) - waiting to lock <0x0000000781a03768> (a org.griphyn.vdl.karajan.VDSAdaptiveScheduler) at org.griphyn.vdl.karajan.VDSAdaptiveScheduler.enqueue(VDSAdaptiveScheduler.java:128) at org.griphyn.vdl.karajan.lib.Execute.submitScheduled(Execute.java:67) at org.globus.cog.karajan.workflow.nodes.grid.GridExec.submitTask(GridExec.java:218) at org.globus.cog.karajan.workflow.nodes.grid.AbstractGridNode.post(AbstractGridNode.java:61) at org.globus.cog.karajan.workflow.nodes.AbstractSequentialWithArguments.completed(AbstractSequentialWithArguments.java:194) at org.globus.cog.karajan.workflow.nodes.FlowNode.complete(FlowNode.java:214) at org.globus.cog.karajan.workflow.nodes.FlowContainer.post(FlowContainer.java:58) at org.globus.cog.karajan.workflow.nodes.AbstractSequentialWithArguments.completed(AbstractSequentialWithArguments.java:194) at org.globus.cog.karajan.workflow.nodes.user.UserDefinedElement.completed(UserDefinedElement.java:286) at org.globus.cog.karajan.workflow.nodes.user.SequentialImplicitExecutionUDE.completed(SequentialImplicitExecutionUDE.java:89) at org.globus.cog.karajan.workflow.nodes.FlowNode.complete(FlowNode.java:214) at org.globus.cog.karajan.workflow.nodes.FlowContainer.post(FlowContainer.java:58) at org.globus.cog.karajan.workflow.nodes.AbstractSequentialWithArguments.completed(AbstractSequentialWithArguments.java:194) at org.globus.cog.karajan.workflow.nodes.PartialArgumentsContainer.nonArgChildCompleted(PartialArgumentsContainer.java:105) at org.globus.cog.karajan.workflow.nodes.PartialArgumentsContainer.completed(PartialArgumentsContainer.java:86) at org.globus.cog.karajan.workflow.nodes.FlowNode.complete(FlowNode.java:214) at org.globus.cog.karajan.workflow.nodes.FlowContainer.post(FlowContainer.java:58) at org.globus.cog.karajan.workflow.nodes.functions.AbstractFunction.post(AbstractFunction.java:28) at org.globus.cog.karajan.workflow.nodes.Sequential.startNext(Sequential.java:29) at org.globus.cog.karajan.workflow.nodes.Sequential.executeChildren(Sequential.java:20) at org.globus.cog.karajan.workflow.nodes.FlowContainer.execute(FlowContainer.java:63) at org.globus.cog.karajan.workflow.nodes.FlowNode.restart(FlowNode.java:139) at org.globus.cog.karajan.workflow.nodes.FlowNode.start(FlowNode.java:197) at org.globus.cog.karajan.workflow.events.EventBus.start(EventBus.java:98) at org.globus.cog.karajan.workflow.events.EventTargetPair.run(EventTargetPair.java:40) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Locked ownable synchronizers: - <0x0000000781af1c20> (a java.util.concurrent.locks.ReentrantLock$NonfairSync) "pool-1-thread-1" prio=10 tid=0x00002aaab48a3800 nid=0x3935 waiting on condition [0x0000000041957000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x000000078014b5c0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) at java.lang.Thread.run(Thread.java:662) Locked ownable synchronizers: - None "Hang checker" prio=10 tid=0x00002aaab4586000 nid=0x3933 waiting for monitor entry [0x0000000041856000] java.lang.Thread.State: BLOCKED (on object monitor) at org.globus.cog.karajan.scheduler.WeightedHostScoreScheduler.allOverloaded(WeightedHostScoreScheduler.java:256) - waiting to lock <0x0000000781a03768> (a org.griphyn.vdl.karajan.VDSAdaptiveScheduler) at org.griphyn.vdl.karajan.HangChecker.run(HangChecker.java:54) at java.util.TimerThread.mainLoop(Timer.java:512) at java.util.TimerThread.run(Timer.java:462) Locked ownable synchronizers: - None "Low Memory Detector" daemon prio=10 tid=0x000000005edd6000 nid=0x392c runnable [0x0000000000000000] java.lang.Thread.State: RUNNABLE Locked ownable synchronizers: - None "C2 CompilerThread1" daemon prio=10 tid=0x000000005edd4000 nid=0x392b waiting on condition [0x0000000000000000] java.lang.Thread.State: RUNNABLE Locked ownable synchronizers: - None "C2 CompilerThread0" daemon prio=10 tid=0x000000005edce800 nid=0x392a waiting on condition [0x0000000000000000] java.lang.Thread.State: RUNNABLE Locked ownable synchronizers: - None "Signal Dispatcher" daemon prio=10 tid=0x000000005edcc800 nid=0x3929 runnable [0x0000000000000000] java.lang.Thread.State: RUNNABLE Locked ownable synchronizers: - None "Finalizer" daemon prio=10 tid=0x000000005edaa000 nid=0x3928 in Object.wait() [0x0000000041f2f000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118) - locked <0x0000000780213d28> (a java.lang.ref.ReferenceQueue$Lock) at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134) at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159) Locked ownable synchronizers: - None "Reference Handler" daemon prio=10 tid=0x000000005eda8000 nid=0x3927 in Object.wait() [0x0000000041452000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:485) at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116) - locked <0x0000000780213ce8> (a java.lang.ref.Reference$Lock) Locked ownable synchronizers: - None "main" prio=10 tid=0x000000005ed43800 nid=0x3921 in Object.wait() [0x0000000040ffa000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on <0x00000007815c28b8> (a org.griphyn.vdl.karajan.VDL2ExecutionContext) at java.lang.Object.wait(Object.java:485) at org.globus.cog.karajan.workflow.ExecutionContext.waitFor(ExecutionContext.java:234) - locked <0x00000007815c28b8> (a org.griphyn.vdl.karajan.VDL2ExecutionContext) at org.griphyn.vdl.karajan.Loader.main(Loader.java:224) Locked ownable synchronizers: - None "VM Thread" prio=10 tid=0x000000005eda1000 nid=0x3926 runnable "GC task thread#0 (ParallelGC)" prio=10 tid=0x000000005ed56800 nid=0x3922 runnable "GC task thread#1 (ParallelGC)" prio=10 tid=0x000000005ed58800 nid=0x3923 runnable "GC task thread#2 (ParallelGC)" prio=10 tid=0x000000005ed5a000 nid=0x3924 runnable "GC task thread#3 (ParallelGC)" prio=10 tid=0x000000005ed5c000 nid=0x3925 runnable "VM Periodic Task Thread" prio=10 tid=0x000000005ede1000 nid=0x392d waiting on condition JNI global references: 1543 Found one Java-level deadlock: ============================= "NBS7": waiting to lock monitor 0x000000005ffe3a10 (object 0x0000000781a03768, a org.griphyn.vdl.karajan.VDSAdaptiveScheduler), which is held by "Scheduler" "Scheduler": waiting to lock monitor 0x0000000060c008e0 (object 0x0000000781a08788, a java.util.HashSet), which is held by "Overloaded Host Monitor" "Overloaded Host Monitor": waiting to lock monitor 0x000000005ffe3a10 (object 0x0000000781a03768, a org.griphyn.vdl.karajan.VDSAdaptiveScheduler), which is held by "Scheduler" Java stack information for the threads listed above: =================================================== "NBS7": at org.globus.cog.karajan.scheduler.LateBindingScheduler.getContacts(LateBindingScheduler.java:714) - waiting to lock <0x0000000781a03768> (a org.griphyn.vdl.karajan.VDSAdaptiveScheduler) at org.globus.cog.karajan.scheduler.WeightedHostScoreScheduler.statusChanged(WeightedHostScoreScheduler.java:395) at org.griphyn.vdl.karajan.VDSAdaptiveScheduler.statusChanged(VDSAdaptiveScheduler.java:421) at org.globus.cog.abstraction.impl.common.task.TaskImpl.notifyListeners(TaskImpl.java:240) at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:228) at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:257) at org.globus.cog.abstraction.impl.execution.coaster.JobSubmissionTaskHandler.submit(JobSubmissionTaskHandler.java:104) at org.globus.cog.abstraction.impl.common.AbstractTaskHandler.submit(AbstractTaskHandler.java:45) at org.globus.cog.karajan.scheduler.submitQueue.NonBlockingSubmit.run(NonBlockingSubmit.java:97) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) "Scheduler": at org.globus.cog.karajan.scheduler.OverloadedHostMonitor.add(OverloadedHostMonitor.java:38) - waiting to lock <0x0000000781a08788> (a java.util.HashSet) at org.globus.cog.karajan.scheduler.WeightedHostSet.checkOverloaded(WeightedHostSet.java:94) at org.globus.cog.karajan.scheduler.WeightedHostSet.changeScore(WeightedHostSet.java:61) - locked <0x0000000781a08478> (a java.util.TreeSet) at org.globus.cog.karajan.scheduler.WeightedHostScoreScheduler.factorScore(WeightedHostScoreScheduler.java:149) - locked <0x0000000781a03768> (a org.griphyn.vdl.karajan.VDSAdaptiveScheduler) at org.globus.cog.karajan.scheduler.WeightedHostScoreScheduler.factorMultiple(WeightedHostScoreScheduler.java:491) at org.globus.cog.karajan.scheduler.WeightedHostScoreScheduler.factorSubmission(WeightedHostScoreScheduler.java:464) at org.globus.cog.karajan.scheduler.WeightedHostScoreScheduler.submitBoundToServices(WeightedHostScoreScheduler.java:387) at org.globus.cog.karajan.scheduler.LateBindingScheduler.submitUnbound(LateBindingScheduler.java:384) at org.globus.cog.karajan.scheduler.LateBindingScheduler.run(LateBindingScheduler.java:270) "Overloaded Host Monitor": at org.globus.cog.karajan.scheduler.WeightedHostScoreScheduler.raiseTasksFinished(WeightedHostScoreScheduler.java:513) - waiting to lock <0x0000000781a03768> (a org.griphyn.vdl.karajan.VDSAdaptiveScheduler) at org.globus.cog.karajan.scheduler.WeightedHostScoreScheduler.removeOverloaded(WeightedHostScoreScheduler.java:509) at org.globus.cog.karajan.scheduler.OverloadedHostMonitor.run(OverloadedHostMonitor.java:54) - locked <0x0000000781a08788> (a java.util.HashSet) Found 1 deadlock. -- Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.