October 2012 - swift-bugs - mailman3.cels.anl.gov

[Bug 829] New: Swift hangs when ext mapper returns error and no output
by bugzilla-daemon＠mcs.anl.gov 18 Mar '13

18 Mar '13

https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=829 Summary: Swift hangs when ext mapper returns error and no output Product: Swift Version: trunk Platform: PC OS/Version: Mac OS Status: NEW Severity: normal Priority: P2 Component: SwiftScript language AssignedTo: hategan(a)mcs.anl.gov ReportedBy: wilde(a)mcs.anl.gov Created an attachment (id=301) --> (http://bugzilla.mcs.anl.gov/swift/attachment.cgi?id=301) Tar file with failing examples from Robin Weiss Robin, I looked into this problem a bit. What may be happening here is that your external mapper scripts are not returning anything on standard output (and an error message on stderr) for the arguments that you are passing in the failing cases. Here's what my tests show: $ ./clusMapper_name.sh [0] out/.0.out [1] out/.1.out [2] out/.2.out [3] out/.3.out $ $ ./clusMapper_name.sh -name=name_here ./clusMapper_name.sh: bad mapper args $ $ ./clusMapper_loop.sh -nfiles=3 ./clusMapper_loop.sh: bad mapper args $ I dont see exactly why Swift is hanging in the cases where the mappers return nothing. I would have thought a nicer behavior would be to (a) exit complaining that the external mapper didnt return with exit code 0, or (b) proceed as if the arrays were not mapped, and use the default (concurrent) mapper. I suspect whats happening is that Swift is not closing the array when the mapper fails, or some similar synchronization error. We'll need to look into this. I've filed this as ticket # - Mike ----- Original Message ----- > From: "Robin Weiss" <robinweiss(a)uchicago.edu> > To: swift-user(a)ci.uchicago.edu > Sent: Monday, September 10, 2012 4:56:45 PM > Subject: [Swift-user] Custom/External Mappers > Howdy swifters, > > > I am having some issues getting external mappers to work. In > particular, swift appears to hang when you use a custom mapper that > takes in one or more command line arguments. > > > Attached is a tar ball with a basic example of the issue. You can > uncomment each line in the mapper.swift script to see the behavior. I > have included 4 versions of my external mapper, two work and two do > not (see mapper.swift). Also included are the config, sites (using > localhost), and tc files I'm using. > > > I noticed this issue after moving from version 0.93 to trunk (Swift > trunk swift-r5917 cog-r3463) when some scripts I had been using > started to hang. > > > Thanks in advance, > Robin > > > > > -- > > Robin M. Weiss > Research Programmer > Research Computing Center > The University of Chicago > 6030 S. Ellis Ave., Suite 289C > Chicago, IL 60637 > robinweiss(a)uchicago.edu > 773.702.9030 > _______________________________________________ > Swift-user mailing list > Swift-user(a)ci.uchicago.edu > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user -- Michael Wilde Computation Institute, University of Chicago Mathematics and Computer Science Division Argonne National Laboratory -- Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 7

[Bug 813] New: OutOfMemoryError with DSSAT and provider staging
by bugzilla-daemon＠mcs.anl.gov 11 Feb '13

11 Feb '13

https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=813 Summary: OutOfMemoryError with DSSAT and provider staging Product: Swift Version: trunk Platform: PC OS/Version: Mac OS Status: NEW Severity: normal Priority: P2 Component: Providers AssignedTo: hategan(a)mcs.anl.gov ReportedBy: davidk(a)ci.uchicago.edu CC: hategan(a)mcs.anl.gov, davidk(a)ci.uchicago.edu, wozniak(a)mcs.anl.gov A user reported seeing the following error while running the DSSAT application. Error in NIO multiplexer java.lang.OutOfMemoryError: Java heap space at org.globus.cog.karajan.workflow.service.channels.AbstractStreamKarajanChannel.stepNIO(AbstractStreamKarajanChannel.java:210) at org.globus.cog.karajan.workflow.service.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97) at org.globus.cog.karajan.workflow.service.channels.NIOMultiplexer.run(NIOMultiplexer.java:56) java.lang.OutOfMemoryError: Java heap space at org.globus.cog.karajan.workflow.service.channels.AbstractStreamKarajanChannel.stepNIO(AbstractStreamKarajanChannel.java:210) at org.globus.cog.karajan.workflow.service.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97) at org.globus.cog.karajan.workflow.service.channels.NIOMultiplexer.run(NIOMultiplexer.java:56) The log files for this run at at http://www.ci.uchicago.edu/~davidk/logs/dssat-oom.tar.gz This application attempts to run a large number of tasks (~100K). Each task transfers a few small files totaling about 5MB per task. The error was reported by the user who saw it on stderr presumably. I've increased the heap size to 4GB for now. This may be the solution, but wanted to report the message just to be safe. -- Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 1

[Bug 696] New: Swift fails to close array initialized within nested control structures
by bugzilla-daemon＠mcs.anl.gov 23 Nov '12

23 Nov '12

https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=696 Summary: Swift fails to close array initialized within nested control structures Product: Swift Version: 0.93 Platform: PC OS/Version: Linux Status: NEW Severity: major Priority: High Component: SwiftScript language AssignedTo: hategan(a)mcs.anl.gov ReportedBy: wilde(a)mcs.anl.gov In the following example, t2.swift, the array ia[] is not correctly closed when passed to another function, when its set within an if() within foreach loops. When its set within simpler nested control structures (example t3.swift) it seems to close OK. In t2.swift has the additional nested if statement "if(true)", otherwise the programs are similar. This logic was extracted from a much more complex science app in which this failure-to-close was observed. com$ cat t2.swift main() { foreach i in [0:0] { int ia[]; if(true) { foreach j in [0:1] { if (j==0) { ia[j] = 100; } else { ia[j] = 101; } } sub(ia); } } } sub(int iap[]) { tracef("in sub: entered\n"); tracef("in sub: iap=%q\n",iap); } main(); com$ swift t2.swift no sites file specified, setting to default: /home/wilde/swift/src/0.93/cog/modules/swift/dist/swift-svn/etc/sites.xml Swift 0.93 swift-r5501 (swift modified locally) cog-r3350 RunID: 20120122-1907-uzipsekg Progress: time: Sun, 22 Jan 2012 19:07:03 -0600 in sub: entered No events in 10s. Registered futures: int[] ia Open, 2 elements, 1 listeners ---- Waiting threads: 0-1-1-0-1-1-1 ---- com$ com$ cat t3.swift main() { foreach i in [0:0] { int ia[]; foreach j in [0:1] { if (j==0) { ia[j] = 100; } else { ia[j] = 101; } } sub(ia); } } sub(int iap[]) { tracef("in sub: iap=%q\n",iap); } main(); com$ swift t3.swift no sites file specified, setting to default: /home/wilde/swift/src/0.93/cog/modules/swift/dist/swift-svn/etc/sites.xml Swift 0.93 swift-r5501 (swift modified locally) cog-r3350 RunID: 20120122-1908-zyyzuogd Progress: time: Sun, 22 Jan 2012 19:08:07 -0600 in sub: iap=[100.0,101.0] Final status: time: Sun, 22 Jan 2012 19:08:07 -0600 com$ -- Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 3

[Bug 855] New: Create shell-level test for Coaster C API
by bugzilla-daemon＠mcs.anl.gov 28 Oct '12

28 Oct '12

https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=855 Summary: Create shell-level test for Coaster C API Product: Swift Version: trunk Platform: PC OS/Version: Mac OS Status: ASSIGNED Severity: enhancement Priority: P2 Component: Providers AssignedTo: davidk(a)ci.uchicago.edu ReportedBy: wilde(a)mcs.anl.gov CC: hategan(a)mcs.anl.gov, davidk(a)ci.uchicago.edu, wozniak(a)mcs.anl.gov ...as a base for Justin to use to integrate Coasters into Swift/T. Moving this to David's plate from Mihael's. Justin's request for this is below. Since the API has been developed, I suspect, David, with your knowledge of, and tools for launching coaster servers and workers, that you could readily create the test lab that Justin needs in order to get the C client integrated into Swift/T. ----- This looks really good. I think the a big thing that would help get me started testing would be a couple of test shell scripts. Could we have one for local execution and one for some scheduler that we both have access to, say, PADS? BTW, do you use one of the Eclipse built-in styles? Justin On 09/11/2012 04:41 AM, Mihael Hategan wrote: > In the mean time, I committed the C client. It's in > cog/modules/provider-coaster-c-client. > > See src/CoasterClientTest.cpp for the basic idea. Make sure you get the > latest java service from trunk. > > It doesn't have support for file GET/PUT, so you can only use sfs or > local staging (and I haven't tested staging at all), but it seems to > otherwise submit jobs and get their status back fine. > > And I would like to use this opportunity to mention how nasty the lack > of a GC is when you write concurrent programs. So expect memory leaks. > > Mihael > > On Tue, 2012-08-28 at 14:43 -0500, Justin M Wozniak wrote: >> I was assuming the perl client was for reference. Again, the >> stand-alone java client / shell script would be just as good as a >> reference. >> >> On 08/28/2012 02:16 PM, Michael Wilde wrote: >>> Would a perl client that was used from Turbine as a command line client work? >>> >>> ie: >>> system("coaster-run"); >>> system ("coaster-status") // uggh... >>> >>> ----- Original Message ----- >>>> From: "Mihael Hategan" <hategan(a)mcs.anl.gov> >>>> To: "Justin M Wozniak" <wozniak(a)mcs.anl.gov> >>>> Cc: "Michael Wilde" <wilde(a)mcs.anl.gov> >>>> Sent: Tuesday, August 28, 2012 1:27:05 PM >>>> Subject: Re: [Swift-devel] coaster documentation >>>> On Tue, 2012-08-28 at 10:49 -0500, Justin M Wozniak wrote: >>>>> I can take a look at the C/C++ client. However, I'm not as concerned >>>>> about the protocol as the architecture: how to recompose the >>>>> Coasters >>>>> features for access by a generic clients. If I could see a shell >>>>> script >>>>> that does this, even with a java client, that would be a big boost. >>>> I can probably have a perl client done a lot quicker than a C client. >> > -- Justin M Wozniak -- Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 1

[Bug 859] New: Coaster shutdown deadlock
by bugzilla-daemon＠mcs.anl.gov 27 Oct '12

27 Oct '12

https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=859 Summary: Coaster shutdown deadlock Product: Swift Version: 0.94 Platform: PC OS/Version: Mac OS Status: NEW Severity: normal Priority: P2 Component: General AssignedTo: hategan(a)mcs.anl.gov ReportedBy: davidk(a)ci.uchicago.edu I saw this deadlock last night while working on some DSSAT issues. In this case, I manually ended the workflow before it was done by hitting Ctrl-C on swift, and killing coaster-service. I realized the coaster-service process was still running, at which point I ran a jstack on it. I later killed it with -9. The full set of logs and configuration files are in /scratch/local/davidk/DSSAT/run007 on communicado. The jstack output is pasted below: [davidk@communicado run007]$ jstack -l 27185 2012-10-22 22:39:16 Full thread dump Java HotSpot(TM) 64-Bit Server VM (20.1-b02 mixed mode): "Attach Listener" daemon prio=10 tid=0x0000000050d12000 nid=0x35db waiting on condition [0x0000000000000000] java.lang.Thread.State: RUNNABLE Locked ownable synchronizers: - None "Thread-4" prio=10 tid=0x0000000050af9000 nid=0x1fab waiting for monitor entry [0x0000000042efa000] java.lang.Thread.State: BLOCKED (on object monitor) at org.globus.cog.abstraction.coaster.service.job.manager.Cpu.shutdown(Cpu.java:278) - waiting to lock <0x0000000724f04b38> (a org.globus.cog.abstraction.coaster.service.job.manager.Cpu) at org.globus.cog.abstraction.coaster.service.job.manager.Block.shutdown(Block.java:274) - locked <0x0000000724df0e18> (a java.util.ArrayList) at org.globus.cog.abstraction.coaster.service.job.manager.BlockQueueProcessor.shutdownBlocks(BlockQueueProcessor.java:799) at org.globus.cog.abstraction.coaster.service.job.manager.BlockQueueProcessor.shutdown(BlockQueueProcessor.java:788) at org.globus.cog.abstraction.coaster.service.job.manager.JobQueue.shutdown(JobQueue.java:119) at org.globus.cog.abstraction.coaster.service.CoasterService.shutdown(CoasterService.java:271) at org.globus.cog.abstraction.coaster.service.CoasterPersistentService$2.run(CoasterPersistentService.java:236) Locked ownable synchronizers: - None "SIGTERM handler" daemon prio=10 tid=0x0000000050af5000 nid=0x1fa9 in Object.wait() [0x0000000042cf8000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) at java.lang.Thread.join(Thread.java:1186) - locked <0x0000000724dca610> (a org.globus.cog.abstraction.coaster.service.CoasterPersistentService$2) at java.lang.Thread.join(Thread.java:1239) at java.lang.ApplicationShutdownHooks.runHooks(ApplicationShutdownHooks.java:79) at java.lang.ApplicationShutdownHooks$1.run(ApplicationShutdownHooks.java:24) at java.lang.Shutdown.runHooks(Shutdown.java:79) at java.lang.Shutdown.sequence(Shutdown.java:123) at java.lang.Shutdown.exit(Shutdown.java:168) - locked <0x000000071fdda778> (a java.lang.Class for java.lang.Shutdown) at java.lang.Terminator$1.handle(Terminator.java:35) at sun.misc.Signal$1.run(Signal.java:195) at java.lang.Thread.run(Thread.java:662) Locked ownable synchronizers: - None "Sender WorkerChannel" daemon prio=10 tid=0x0000000050bc8000 nid=0x7be2 waiting on condition [0x0000000041547000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x0000000724f57748> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399) at org.globus.cog.karajan.workflow.service.channels.Sender.run(Sender.java:56) Locked ownable synchronizers: - None "Timer-3" daemon prio=10 tid=0x0000000050cb0800 nid=0x74b5 in Object.wait() [0x000000004125d000] java.lang.Thread.State: TIMED_WAITING (on object monitor) at java.lang.Object.wait(Native Method) at java.util.TimerThread.mainLoop(Timer.java:509) - locked <0x0000000724e58248> (a java.util.TaskQueue) at java.util.TimerThread.run(Timer.java:462) Locked ownable synchronizers: - None "I/O Queue" daemon prio=10 tid=0x00002aaab44e6000 nid=0x74b1 in Object.wait() [0x0000000042bf7000] java.lang.Thread.State: TIMED_WAITING (on object monitor) at java.lang.Object.wait(Native Method) at org.globus.cog.abstraction.impl.file.coaster.buffers.Buffers.run(Buffers.java:188) - locked <0x0000000724e584b8> (a org.globus.cog.abstraction.impl.file.coaster.buffers.Buffers) Locked ownable synchronizers: - None "I/O Queue" daemon prio=10 tid=0x00002aaab44e5800 nid=0x74b0 in Object.wait() [0x0000000042af6000] java.lang.Thread.State: TIMED_WAITING (on object monitor) at java.lang.Object.wait(Native Method) at org.globus.cog.abstraction.impl.file.coaster.buffers.Buffers.run(Buffers.java:188) - locked <0x0000000724dfd5f0> (a org.globus.cog.abstraction.impl.file.coaster.buffers.Buffers) Locked ownable synchronizers: - None "Timer-2" daemon prio=10 tid=0x0000000050b0d000 nid=0x74ad in Object.wait() [0x00000000429f5000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on <0x0000000724dd2480> (a java.util.TaskQueue) at java.lang.Object.wait(Object.java:485) at java.util.TimerThread.mainLoop(Timer.java:483) - locked <0x0000000724dd2480> (a java.util.TaskQueue) at java.util.TimerThread.run(Timer.java:462) Locked ownable synchronizers: - None "Sender TCPChannel" daemon prio=10 tid=0x0000000050a69800 nid=0x74a9 waiting on condition [0x00000000428f4000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x0000000724dd8940> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399) at org.globus.cog.karajan.workflow.service.channels.Sender.run(Sender.java:56) Locked ownable synchronizers: - None "Channel multiplexer 1" daemon prio=10 tid=0x0000000050a68000 nid=0x74a6 sleeping[0x00000000427f3000] java.lang.Thread.State: TIMED_WAITING (sleeping) at java.lang.Thread.sleep(Native Method) at org.globus.cog.karajan.workflow.service.channels.Multiplexer.run(Multiplexer.java:109) Locked ownable synchronizers: - None "Channel multiplexer 0" daemon prio=10 tid=0x0000000050a67000 nid=0x74a5 sleeping[0x00000000426f2000] java.lang.Thread.State: TIMED_WAITING (sleeping) at java.lang.Thread.sleep(Native Method) at org.globus.cog.karajan.workflow.service.channels.Multiplexer.run(Multiplexer.java:109) Locked ownable synchronizers: - None "NIO Sender" daemon prio=10 tid=0x00002aaab44f7800 nid=0x71cf waiting for monitor entry [0x000000004115c000] java.lang.Thread.State: BLOCKED (on object monitor) at org.globus.cog.abstraction.coaster.service.job.manager.Block.forceShutdown(Block.java:323) - waiting to lock <0x0000000724df0e18> (a java.util.ArrayList) at org.globus.cog.abstraction.coaster.service.job.manager.Node.errorReceived(Node.java:100) at org.globus.cog.karajan.workflow.service.commands.Command.errorReceived(Command.java:203) at org.globus.cog.karajan.workflow.service.channels.ChannelContext.notifyListeners(ChannelContext.java:237) at org.globus.cog.karajan.workflow.service.channels.ChannelContext.notifyRegisteredCommandsAndHandlers(ChannelContext.java:225) at org.globus.cog.karajan.workflow.service.channels.ChannelContext.channelShutDown(ChannelContext.java:318) - locked <0x0000000724e5df80> (a org.globus.cog.karajan.workflow.service.channels.ChannelContext) at org.globus.cog.karajan.workflow.service.channels.ChannelManager.handleChannelException(ChannelManager.java:293) at org.globus.cog.karajan.workflow.service.channels.AbstractKarajanChannel.handleChannelException(AbstractKarajanChannel.java:552) - locked <0x0000000724e5df08> (a org.globus.cog.abstraction.coaster.service.LocalTCPService$WorkerChannel) at org.globus.cog.karajan.workflow.service.channels.NIOSender.run(NIOSender.java:140) Locked ownable synchronizers: - None "PullThread" daemon prio=10 tid=0x00002aaab44d0000 nid=0x71cc waiting for monitor entry [0x000000004030a000] java.lang.Thread.State: BLOCKED (on object monitor) at org.globus.cog.abstraction.coaster.service.job.manager.Block.add(Block.java:191) - waiting to lock <0x0000000724df0e18> (a java.util.ArrayList) at org.globus.cog.abstraction.coaster.service.job.manager.Cpu.launchSequential(Cpu.java:224) at org.globus.cog.abstraction.coaster.service.job.manager.Cpu.launch(Cpu.java:211) at org.globus.cog.abstraction.coaster.service.job.manager.Cpu.pull(Cpu.java:185) - locked <0x0000000724f79670> (a org.globus.cog.abstraction.coaster.service.job.manager.Cpu) at org.globus.cog.abstraction.coaster.service.job.manager.PullThread.run(PullThread.java:131) Locked ownable synchronizers: - None "NIO Multiplexer" daemon prio=10 tid=0x0000000050b04000 nid=0x71c6 waiting for monitor entry [0x00000000425f1000] java.lang.Thread.State: BLOCKED (on object monitor) at org.globus.cog.abstraction.coaster.service.job.manager.Block.remove(Block.java:179) - waiting to lock <0x0000000724df0e18> (a java.util.ArrayList) at org.globus.cog.abstraction.coaster.service.job.manager.Cpu.jobTerminated(Cpu.java:97) at org.globus.cog.abstraction.coaster.service.job.manager.Cpu.statusChanged(Cpu.java:378) - locked <0x0000000724f04b38> (a org.globus.cog.abstraction.coaster.service.job.manager.Cpu) at org.globus.cog.abstraction.impl.common.task.TaskImpl.notifyListeners(TaskImpl.java:240) at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:228) at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.setStatus(NotificationManager.java:117) at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:88) at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:50) at org.globus.cog.karajan.workflow.service.handlers.RequestHandler.receiveCompleted(RequestHandler.java:91) at org.globus.cog.karajan.workflow.service.channels.AbstractKarajanChannel.handleRequest(AbstractKarajanChannel.java:505) at org.globus.cog.karajan.workflow.service.channels.AbstractStreamKarajanChannel.stepNIO(AbstractStreamKarajanChannel.java:238) at org.globus.cog.karajan.workflow.service.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97) at org.globus.cog.karajan.workflow.service.channels.NIOMultiplexer.run(NIOMultiplexer.java:56) Locked ownable synchronizers: - None "Timer-1" daemon prio=10 tid=0x0000000050b03000 nid=0x71c3 waiting for monitor entry [0x00000000424f0000] java.lang.Thread.State: BLOCKED (on object monitor) at org.globus.cog.karajan.workflow.service.channels.ChannelContext.removeOldIgnoredRequests(ChannelContext.java:348) - waiting to lock <0x0000000724e5df80> (a org.globus.cog.karajan.workflow.service.channels.ChannelContext) at org.globus.cog.karajan.workflow.service.channels.AbstractKarajanChannel.checkTimeouts(AbstractKarajanChannel.java:126) at org.globus.cog.karajan.workflow.service.channels.AbstractKarajanChannel$1.run(AbstractKarajanChannel.java:118) at java.util.TimerThread.mainLoop(Timer.java:512) at java.util.TimerThread.run(Timer.java:462) Locked ownable synchronizers: - None "Passive Queue Processor" prio=10 tid=0x00002aaab44d4800 nid=0x6a43 in Object.wait() [0x00000000423ef000] java.lang.Thread.State: TIMED_WAITING (on object monitor) at java.lang.Object.wait(Native Method) at org.globus.cog.abstraction.coaster.service.job.manager.BlockQueueProcessor.run(BlockQueueProcessor.java:143) - locked <0x0000000724e59618> (a java.util.ArrayList) Locked ownable synchronizers: - None "Local Queue Processor" prio=10 tid=0x00002aaab44d3800 nid=0x6a42 waiting on condition [0x00000000422ee000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x0000000724dda348> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399) at org.globus.cog.abstraction.coaster.service.job.manager.AbstractQueueProcessor.take(AbstractQueueProcessor.java:43) at org.globus.cog.abstraction.coaster.service.job.manager.LocalQueueProcessor.run(LocalQueueProcessor.java:37) Locked ownable synchronizers: - None "Local TCP service" daemon prio=10 tid=0x00002aaab44a3000 nid=0x6a40 runnable [0x0000000040627000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:152) - locked <0x0000000724dda658> (a java.lang.Object) at org.globus.cog.abstraction.coaster.service.LocalTCPService.run(LocalTCPService.java:127) at java.lang.Thread.run(Thread.java:662) Locked ownable synchronizers: - None "Timer-0" prio=10 tid=0x00002aaab4496000 nid=0x6a3f waiting for monitor entry [0x0000000040209000] java.lang.Thread.State: BLOCKED (on object monitor) at java.lang.Shutdown.exit(Shutdown.java:168) - waiting to lock <0x000000071fdda778> (a java.lang.Class for java.lang.Shutdown) at java.lang.Runtime.exit(Runtime.java:90) at java.lang.System.exit(System.java:904) at org.globus.cog.abstraction.coaster.service.CoasterService$2.run(CoasterService.java:321) at java.util.TimerThread.mainLoop(Timer.java:512) at java.util.TimerThread.run(Timer.java:462) Locked ownable synchronizers: - None "Low Memory Detector" daemon prio=10 tid=0x0000000050aac800 nid=0x6a3d runnable [0x0000000000000000] java.lang.Thread.State: RUNNABLE Locked ownable synchronizers: - None "C2 CompilerThread1" daemon prio=10 tid=0x0000000050aaa000 nid=0x6a3c waiting on condition [0x0000000000000000] java.lang.Thread.State: RUNNABLE Locked ownable synchronizers: - None "C2 CompilerThread0" daemon prio=10 tid=0x0000000050aa4800 nid=0x6a3b waiting on condition [0x0000000000000000] java.lang.Thread.State: RUNNABLE Locked ownable synchronizers: - None "Signal Dispatcher" daemon prio=10 tid=0x0000000050aa2800 nid=0x6a3a runnable [0x0000000000000000] java.lang.Thread.State: RUNNABLE Locked ownable synchronizers: - None "Finalizer" daemon prio=10 tid=0x0000000050a80000 nid=0x6a39 in Object.wait() [0x0000000041c36000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118) - locked <0x0000000724ddcc80> (a java.lang.ref.ReferenceQueue$Lock) at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134) at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159) Locked ownable synchronizers: - None "Reference Handler" daemon prio=10 tid=0x0000000050a7e000 nid=0x6a38 in Object.wait() [0x0000000041b35000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:485) at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116) - locked <0x0000000724dd8ec0> (a java.lang.ref.Reference$Lock) Locked ownable synchronizers: - None "main" prio=10 tid=0x0000000050a19800 nid=0x6a32 in Object.wait() [0x00000000417cf000] java.lang.Thread.State: TIMED_WAITING (on object monitor) at java.lang.Object.wait(Native Method) at org.globus.cog.abstraction.coaster.service.CoasterService.waitFor(CoasterService.java:219) - locked <0x0000000724dca778> (a org.globus.cog.abstraction.coaster.service.CoasterPersistentService) at org.globus.cog.abstraction.coaster.service.CoasterPersistentService.main(CoasterPersistentService.java:179) Locked ownable synchronizers: - None "VM Thread" prio=10 tid=0x0000000050a77800 nid=0x6a37 runnable "GC task thread#0 (ParallelGC)" prio=10 tid=0x0000000050a2c800 nid=0x6a33 runnable "GC task thread#1 (ParallelGC)" prio=10 tid=0x0000000050a2e800 nid=0x6a34 runnable "GC task thread#2 (ParallelGC)" prio=10 tid=0x0000000050a30000 nid=0x6a35 runnable "GC task thread#3 (ParallelGC)" prio=10 tid=0x0000000050a32000 nid=0x6a36 runnable "VM Periodic Task Thread" prio=10 tid=0x0000000050ab7000 nid=0x6a3e waiting on condition JNI global references: 1020 Found one Java-level deadlock: ============================= "Thread-4": waiting to lock monitor 0x0000000050b29178 (object 0x0000000724f04b38, a org.globus.cog.abstraction.coaster.service.job.manager.Cpu), which is held by "NIO Multiplexer" "NIO Multiplexer": waiting to lock monitor 0x0000000050bc62f0 (object 0x0000000724df0e18, a java.util.ArrayList), which is held by "Thread-4" Java stack information for the threads listed above: =================================================== "Thread-4": at org.globus.cog.abstraction.coaster.service.job.manager.Cpu.shutdown(Cpu.java:278) - waiting to lock <0x0000000724f04b38> (a org.globus.cog.abstraction.coaster.service.job.manager.Cpu) at org.globus.cog.abstraction.coaster.service.job.manager.Block.shutdown(Block.java:274) - locked <0x0000000724df0e18> (a java.util.ArrayList) at org.globus.cog.abstraction.coaster.service.job.manager.BlockQueueProcessor.shutdownBlocks(BlockQueueProcessor.java:799) at org.globus.cog.abstraction.coaster.service.job.manager.BlockQueueProcessor.shutdown(BlockQueueProcessor.java:788) at org.globus.cog.abstraction.coaster.service.job.manager.JobQueue.shutdown(JobQueue.java:119) at org.globus.cog.abstraction.coaster.service.CoasterService.shutdown(CoasterService.java:271) at org.globus.cog.abstraction.coaster.service.CoasterPersistentService$2.run(CoasterPersistentService.java:236) "NIO Multiplexer": at org.globus.cog.abstraction.coaster.service.job.manager.Block.remove(Block.java:179) - waiting to lock <0x0000000724df0e18> (a java.util.ArrayList) at org.globus.cog.abstraction.coaster.service.job.manager.Cpu.jobTerminated(Cpu.java:97) at org.globus.cog.abstraction.coaster.service.job.manager.Cpu.statusChanged(Cpu.java:378) - locked <0x0000000724f04b38> (a org.globus.cog.abstraction.coaster.service.job.manager.Cpu) at org.globus.cog.abstraction.impl.common.task.TaskImpl.notifyListeners(TaskImpl.java:240) at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:228) at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.setStatus(NotificationManager.java:117) at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:88) at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:50) at org.globus.cog.karajan.workflow.service.handlers.RequestHandler.receiveCompleted(RequestHandler.java:91) at org.globus.cog.karajan.workflow.service.channels.AbstractKarajanChannel.handleRequest(AbstractKarajanChannel.java:505) at org.globus.cog.karajan.workflow.service.channels.AbstractStreamKarajanChannel.stepNIO(AbstractStreamKarajanChannel.java:238) at org.globus.cog.karajan.workflow.service.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97) at org.globus.cog.karajan.workflow.service.channels.NIOMultiplexer.run(NIOMultiplexer.java:56) Found 1 deadlock. -- Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 1

[Bug 863] New: Generalize auth.defaults to allow auth options to be passed to swift command
by bugzilla-daemon＠mcs.anl.gov 25 Oct '12

25 Oct '12

https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=863 Summary: Generalize auth.defaults to allow auth options to be passed to swift command Product: Swift Version: trunk Platform: PC OS/Version: Mac OS Status: NEW Severity: enhancement Priority: P2 Component: Providers AssignedTo: hategan(a)mcs.anl.gov ReportedBy: wilde(a)mcs.anl.gov CC: hategan(a)mcs.anl.gov, davidk(a)ci.uchicago.edu, wozniak(a)mcs.anl.gov For executing the swift command from a portal like GPSI, we need to be able to have the invoker of swift specify auth.defaults options in a manner not tied to a single user's auth.defaults file. Eg, allow passing of an auth.defaults file on the command line, and/or other secure ways to pass keys and passwords. -- Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 0

[Bug 857] New: Failed to convert .swiftx to .kml
by bugzilla-daemon＠mcs.anl.gov 24 Oct '12

24 Oct '12

https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=857 Summary: Failed to convert .swiftx to .kml Product: Swift Version: 0.94 Platform: PC OS/Version: Mac OS Status: NEW Severity: normal Priority: P2 Component: General AssignedTo: hategan(a)mcs.anl.gov ReportedBy: davidk(a)ci.uchicago.edu Many tests in the test suite are failing to do the error below. An example of this can be found at http://www.ci.uchicago.edu/swift/tests/swift-trunk/run-2012-10-20/008-forea… 2012-10-20 00:36:53,700-0500 DEBUG Loader arguments: [-wrapperlog.always.transfer, true, -sitedir.keep, true, -config, swift.properties, -sites.file, sites.xml, -tc.file, tc.data, 008-foreach.swift] 2012-10-20 00:36:53,700-0500 DEBUG Loader Max heap: 954466304 2012-10-20 00:36:53,701-0500 DEBUG textfiles BEGIN SWIFTSCRIPT: echo(int i) { app { echo i; }} int sa[]; foreach s in sa { echo(s); } 2012-10-20 00:36:53,701-0500 DEBUG textfiles END SWIFTSCRIPT: 2012-10-20 00:36:53,701-0500 INFO Loader 008-foreach.swift: source file is new. Recompiling. 2012-10-20 00:36:54,326-0500 DEBUG Loader Detailed exception: org.griphyn.vdl.karajan.CompilationException: Failed to convert .swiftx to .kml for 008-foreach.swift at org.griphyn.vdl.karajan.Loader.compile(Loader.java:361) at org.griphyn.vdl.karajan.Loader.main(Loader.java:170) Caused by: java.lang.NullPointerException at org.griphyn.vdl.engine.Karajan.allVariables(Karajan.java:894) at org.griphyn.vdl.engine.Karajan.call(Karajan.java:865) at org.griphyn.vdl.engine.Karajan.statement(Karajan.java:663) at org.griphyn.vdl.engine.Karajan.statements(Karajan.java:626) at org.griphyn.vdl.engine.Karajan.foreachStat(Karajan.java:969) at org.griphyn.vdl.engine.Karajan.statement(Karajan.java:666) at org.griphyn.vdl.engine.Karajan.statements(Karajan.java:626) at org.griphyn.vdl.engine.Karajan.program(Karajan.java:328) at org.griphyn.vdl.engine.Karajan.compile(Karajan.java:133) at org.griphyn.vdl.karajan.Loader.compile(Loader.java:342) ... 1 more -- Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 5

[Bug 853] New: DSSAT: Failed to abort transfer error
by bugzilla-daemon＠mcs.anl.gov 20 Oct '12

20 Oct '12

https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=853 Summary: DSSAT: Failed to abort transfer error Product: Swift Version: trunk Platform: PC OS/Version: Mac OS Status: NEW Severity: normal Priority: P2 Component: General AssignedTo: hategan(a)mcs.anl.gov ReportedBy: davidk(a)ci.uchicago.edu When trying to complete a large DSSAT run, within about 20 minutes of starting the run I see this error in the standard output: Failed to abort transfer java.lang.NullPointerException at org.globus.cog.abstraction.impl.file.coaster.handlers.providers.LocalIOProvider.abort(LocalIOProvider.java:35) at org.globus.cog.abstraction.impl.file.coaster.handlers.GetFileHandler.abort(GetFileHandler.java:194) at org.globus.cog.abstraction.impl.file.coaster.handlers.GetFileHandler.errorReceived(GetFileHandler.java:114) at org.globus.cog.karajan.workflow.service.channels.ChannelContext.notifyListeners(ChannelContext.java:237) at org.globus.cog.karajan.workflow.service.channels.ChannelContext.notifyRegisteredCommandsAndHandlers(ChannelContext.java:224) at org.globus.cog.karajan.workflow.service.channels.ChannelContext.channelShutDown(ChannelContext.java:318) at org.globus.cog.karajan.workflow.service.channels.ChannelManager.handleChannelException(ChannelManager.java:293) at org.globus.cog.karajan.workflow.service.commands.HeartBeatCommand.handleTimeout(HeartBeatCommand.java:51) at org.globus.cog.karajan.workflow.service.channels.AbstractKarajanChannel.checkTimeouts(AbstractKarajanChannel.java:134) at org.globus.cog.karajan.workflow.service.channels.AbstractKarajanChannel.checkTimeouts(AbstractKarajanChannel.java:124) at org.globus.cog.karajan.workflow.service.channels.AbstractKarajanChannel$1.run(AbstractKarajanChannel.java:118) at java.util.TimerThread.mainLoop(Timer.java:512) at java.util.TimerThread.run(Timer.java:462) Once this error occurs, no further progress is made. The swift log, coaster log, standard output, and jstacks of swift and coaster-service can be found at /scratch/local/davidk/DSSAT/run005 on communicado. -- Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 1

[Bug 467] Persistent coaster service runs only one job per node at a time
by bugzilla-daemon＠mcs.anl.gov 20 Oct '12

20 Oct '12

https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=467 --- Comment #4 from ketan <ketan(a)mcs.anl.gov> 2012-10-20 09:24:44 --- This one seems to be persisting in trunk. I tried with 10 8-core nodes on Redcloud but the workers start only 1 job per node. In the source of start_coaster_service, the lines to generate a dummy swift and run it before starting worker was commented off. Uncommenting and restarting the coaster service did not seem to make any change in the behavior. Is there a workaround? The same behavior is also seen in 0.93.1: cog r3354, swift r5521 Is there a workaround to this? -- Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 0

[Bug 849] New: Prevent Swift from overwriting user files with compiler output files
by bugzilla-daemon＠mcs.anl.gov 15 Oct '12

15 Oct '12

https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=849 Summary: Prevent Swift from overwriting user files with compiler output files Product: Swift Version: trunk Platform: PC OS/Version: Mac OS Status: ASSIGNED Severity: minor Priority: P4 Component: Error messages AssignedTo: davidk(a)ci.uchicago.edu ReportedBy: wilde(a)mcs.anl.gov CC: wozniak(a)mcs.anl.gov Swift generates a file called <your_script>.xml during compilation. In your case that would be R_ROC.xml. That overwrites your sites file. Mihael On Wed, 2012-10-10 at 16:39 +0000, Yahui Peng wrote: > I012-10-09 20:24:45,873+0000 DEBUG Loader arguments: [-config, R_ROC.cf, -sites.file, R_ROC.xml, -tc.file, R_ROC.tc, R_ROC.swift] -- to prevent that we could: Check that the scriptname.xml file will not overwrite the -sites file Ensure that if scriptname.xml or .kml exist, they are files from the Swift compiler (eg by a magic number check, appearance check, etc). -- Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 3