https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=1123 Summary: New deadlock in 0.94.1 Product: Swift Version: 0.94 Platform: PC OS/Version: Mac OS Status: NEW Severity: critical Priority: P1 Component: General AssignedTo: [email protected] ReportedBy: [email protected] CC: [email protected] Running dssat, I saw a large number of apps fail simultaneously (possibly due to exceeded walltimes). I tried to kill Swift by hitting ctrl-c to find out what was happening, but it would not die. I ran jstack and it reported a deadlock. http://web.ci.uchicago.edu/~davidk/logs/jstack-2013-10-25.log http://web.ci.uchicago.edu/~davidk/logs/RunpSIMS-20131025-1458-zvo73lae.log.... Found one Java-level deadlock: ============================= "NIO Multiplexer": waiting to lock monitor 0x00007f1a2c8159d8 (object 0x00000006c1279f88, a java.util.TreeMap), which is held by "Block Queue Processor" "Block Queue Processor": waiting to lock monitor 0x00007f1a2c33dd08 (object 0x00000006c135aea0, a java.util.ArrayList), which is held by "Timer-1" "Timer-1": waiting to lock monitor 0x00007f1a2c8159d8 (object 0x00000006c1279f88, a java.util.TreeMap), which is held by "Block Queue Processor" Java stack information for the threads listed above: =================================================== "NIO Multiplexer": at org.globus.cog.abstraction.coaster.service.job.manager.BlockQueueProcessor.getBlock(BlockQueueProcessor.java:823) - waiting to lock <0x00000006c1279f88> (a java.util.TreeMap) at org.globus.cog.abstraction.coaster.service.job.manager.BlockQueueProcessor.nextId(BlockQueueProcessor.java:842) at org.globus.cog.abstraction.coaster.service.job.manager.JobQueue.nextId(JobQueue.java:134) at org.globus.cog.abstraction.coaster.service.LocalTCPService.registrationReceived(LocalTCPService.java:73) at org.globus.cog.abstraction.coaster.service.local.RegistrationHandler.requestComplete(RegistrationHandler.java:59) at org.globus.cog.karajan.workflow.service.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88) at org.globus.cog.karajan.workflow.service.channels.AbstractKarajanChannel.handleRequest(AbstractKarajanChannel.java:527) at org.globus.cog.karajan.workflow.service.channels.AbstractStreamKarajanChannel.stepNIO(AbstractStreamKarajanChannel.java:238) at org.globus.cog.karajan.workflow.service.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97) at org.globus.cog.karajan.workflow.service.channels.NIOMultiplexer.run(NIOMultiplexer.java:56) "Block Queue Processor": at org.globus.cog.abstraction.coaster.service.job.manager.Block.fits(Block.java:164) - waiting to lock <0x00000006c135aea0> (a java.util.ArrayList) at org.globus.cog.abstraction.coaster.service.job.manager.BlockQueueProcessor.fits(BlockQueueProcessor.java:263) - locked <0x00000006c1279f88> (a java.util.TreeMap) at org.globus.cog.abstraction.coaster.service.job.manager.BlockQueueProcessor.queueToExistingBlocks(BlockQueueProcessor.java:283) at org.globus.cog.abstraction.coaster.service.job.manager.BlockQueueProcessor.updatePlan(BlockQueueProcessor.java:710) at org.globus.cog.abstraction.coaster.service.job.manager.BlockQueueProcessor.run(BlockQueueProcessor.java:138) "Timer-1": at org.globus.cog.abstraction.coaster.service.job.manager.BlockQueueProcessor.blockTaskFinished(BlockQueueProcessor.java:809) - waiting to lock <0x00000006c1279f88> (a java.util.TreeMap) at org.globus.cog.abstraction.coaster.service.job.manager.Block.statusChanged(Block.java:449) - locked <0x00000006c135aea0> (a java.util.ArrayList) at org.globus.cog.abstraction.impl.common.task.TaskImpl.notifyListeners(TaskImpl.java:240) at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:228) at org.globus.cog.abstraction.impl.scheduler.common.AbstractJobSubmissionTaskHandler.cancel(AbstractJobSubmissionTaskHandler.java:86) - locked <0x00000006c14e1b28> (a org.globus.cog.abstraction.impl.scheduler.slurm.execution.JobSubmissionTaskHandler) at org.globus.cog.abstraction.impl.common.AbstractTaskHandler.cancel(AbstractTaskHandler.java:69) at org.globus.cog.abstraction.impl.common.task.ExecutionTaskHandler.cancel(ExecutionTaskHandler.java:106) at org.globus.cog.abstraction.impl.common.task.ExecutionTaskHandler.cancel(ExecutionTaskHandler.java:95) at org.globus.cog.abstraction.coaster.service.job.manager.BlockTaskSubmitter.cancel(BlockTaskSubmitter.java:46) at org.globus.cog.abstraction.coaster.service.job.manager.Block.forceShutdown(Block.java:333) - locked <0x00000006c135aea0> (a java.util.ArrayList) at org.globus.cog.abstraction.coaster.service.job.manager.Block$1.run(Block.java:323) at java.util.TimerThread.mainLoop(Timer.java:555) at java.util.TimerThread.run(Timer.java:505) Found 1 deadlock. -- Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.