April 2014 - swift-bugs - mailman3.cels.anl.gov

[Bug 801] New: Built-in mappers should give an error when an unrecognized parameter is specified.
by bugzilla-daemon＠mcs.anl.gov 18 Oct '14

18 Oct '14

https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=801 Summary: Built-in mappers should give an error when an unrecognized parameter is specified. Product: Swift Version: 0.93 Platform: All OS/Version: All Status: NEW Severity: normal Priority: P2 Component: SwiftScript language AssignedTo: davidk(a)ci.uchicago.edu ReportedBy: wilde(a)mcs.anl.gov >From user R. Weiss: "So like I said, it turns out this whole issue boils down to a typo. I would suggest that swift should throw up some sort of warning if you pass something to one of the pre-defined mappers that is not defined as one of its parameters. I would have expected swift to complain that "sufix" is an unknown parameter of filesys_mapper." -- Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 1

[Bug 752] New: Coaster provider does not retry worker launches on qsub failures
by bugzilla-daemon＠mcs.anl.gov 18 Oct '14

18 Oct '14

https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=752 Summary: Coaster provider does not retry worker launches on qsub failures Product: Swift Version: 0.93 Platform: PC OS/Version: Mac OS Status: ASSIGNED Severity: normal Priority: P2 Component: Providers AssignedTo: hategan(a)mcs.anl.gov ReportedBy: wilde(a)mcs.anl.gov CC: hategan(a)mcs.anl.gov, davidk(a)ci.uchicago.edu, wozniak(a)mcs.anl.gov If a qsub to start a worker.pl job fails (in this case, due to the user having exceeded the maximum number of concurrently allowed job submissions, 32 on Eureka) then the coaster provider never tries again to start workers for those coaster slots. Hence, when the user is running multiple swift runs concurrently, and they fluctuate in the number of active workers, the swift run that encounters the worker failures stays forever at a very low active worker count. In the current situation, it stays so low (eg 1 worker out of an expected 32) that the run needs to be killed and restarted in order to get a sufficient number of workers. Logs are on eureka in /home/wilde/LZ/runNNN where NNN is between 020 and 058. -- Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 1

[Bug 722] New: fprintf presents hanging situation
by bugzilla-daemon＠mcs.anl.gov 18 Oct '14

18 Oct '14

https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=722 Summary: fprintf presents hanging situation Product: Swift Version: 0.93 Platform: PC OS/Version: Mac OS Status: NEW Severity: normal Priority: P2 Component: General AssignedTo: hategan(a)mcs.anl.gov ReportedBy: jonmon(a)utexas.edu CC: wilde(a)mcs.anl.gov If using %i and passing a float variable in the argument list, seems to have fprintf hang -- Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 2

[Bug 688] New: Testing env interactions
by bugzilla-daemon＠mcs.anl.gov 18 Oct '14

18 Oct '14

https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=688 Summary: Testing env interactions Product: Swift Version: trunk Platform: PC OS/Version: Mac OS Status: NEW Severity: normal Priority: P2 Component: Testing AssignedTo: jonmon(a)utexas.edu ReportedBy: jonmon(a)utexas.edu CC: wilde(a)mcs.anl.gov, skenny(a)uchicago.edu, davidk(a)ci.uchicago.edu, wozniak(a)mcs.anl.gov, ketan(a)mcs.anl.gov Testing and documenting interactions with setting environment variables using the env profile. Justin I believe had specific testing situation and questions. -- Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 1

[Bug 923] New: @toFloat() error gives no line number
by bugzilla-daemon＠mcs.anl.gov 12 Oct '14

12 Oct '14

https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=923 Summary: @toFloat() error gives no line number Product: Swift Version: 0.93 Platform: PC OS/Version: Mac OS Status: ASSIGNED Severity: minor Priority: P4 Component: Error messages AssignedTo: davidk(a)ci.uchicago.edu ReportedBy: wilde(a)mcs.anl.gov CC: wozniak(a)mcs.anl.gov In this expression, -startingjump was null, and the error message from @tofloat() gave no line number: (outfile,lossfile) = multi_annealing(best_filename, @tofloat(@arg("tstart", "2.0")), @tofloat(@arg("tend", "0.01")), @tofloat(@arg("trejection", "0.3")), @toint(@arg("evoreruns", "100")), @tofloat(@arg("startingjump", "2.3")), [ @tofloat(@arg("alphai", "0.0")), @tofloat(@arg("alpham", "0.0")), @tofloat(@arg("beta", "4.0")), @tofloat(@arg("gamma", "50.0")), @tofloat(@arg("delta", "-1.0")) ], @tofloat(target_innov), @toint(@arg("annealingcycles", "50")) ); } Need to see if this is still the case in 0.94. Log said: 2013-01-06 11:57:21,066-0600 INFO ExecutionContext Detailed exception: Could not convert value "" to type float at org.griphyn.vdl.karajan.lib.swiftscript.Misc.swiftscript_tofloat(Misc.java:318) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:601) stdout/err said: Swift 0.93 swift-r6113 (swift modified locally) cog-r3532 RunID: 20130106-1157-z0voc3r9 Progress: time: Sun, 06 Jan 2013 11:57:20 -0600 Execution failed: Could not convert value "" to type float -- Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 2

[Bug 919] New: Misleading error message when a structure type field is missing
by bugzilla-daemon＠mcs.anl.gov 12 Oct '14

12 Oct '14

https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=919 Summary: Misleading error message when a structure type field is missing Product: Swift Version: 0.94 Platform: PC OS/Version: Mac OS Status: ASSIGNED Severity: minor Priority: P2 Component: Error messages AssignedTo: davidk(a)ci.uchicago.edu ReportedBy: wilde(a)mcs.anl.gov CC: wozniak(a)mcs.anl.gov If an expression refers to a non-existent structure field, swift gives an error like: Compile error in assignment at line 134: Type Checkpoint is not defined. instead of: in expression ckpt.ref1: field ref1 of Type Checkpoint is not defined. -- Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 2

[Bug 1257] New: Swift submits cleanup job before main job is complete
by bugzilla-daemon＠mcs.anl.gov 24 Sep '14

24 Sep '14

https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=1257 Summary: Swift submits cleanup job before main job is complete Product: Swift Version: 0.95 Platform: All OS/Version: All Status: ASSIGNED Severity: blocker Priority: P1 Component: Runtime AssignedTo: hategan(a)mcs.anl.gov ReportedBy: wilde(a)mcs.anl.gov I'm preparing an XSEDE demo on stampede, and using the development queue, which is limited to one job at a time per user. Ive got maxJobs set to 1, using this 0.95 site def: site.stampede { jobManager=slurm jobQueue=development jobProject=TG-CCR130051 jobWallTime=00:05:00 taskWallTime=00:00:30 maxJobs=1 tasksPerWorker=16 initialScore=10000 filesystem=local workdir=$SCRATCH } It seems like Swift is submitting a cleanup job to SLURM before the primary job has completed. So I get this on termination: Swift 0.95 Branch SVN swift-r7785 cog-r3894 RunID: run004 ... Progress: Tue, 22 Apr 2014 14:56:56-0500 Active:1 Progress: Tue, 22 Apr 2014 14:56:57-0500 Stage out:1 Final status:Tue, 22 Apr 2014 14:56:57-0500 Finished successfully:1 org.globus.cog.abstraction.impl.common.task.TaskSubmissionException: Could not submit job (sbatch reported an exit code of 1). ----------------------------------------------------------------- Welcome to the Stampede Supercomputer -------------------------------------------------------------------> Verifying valid submit host (login1)...OK--> Verifying valid jobname...OK--> Enforcing max jobs per user...OK--> Verifying availability of your home dir (/home1/00306/tg455797)...OK--> Verifying availability of your work dir (/work/00306/tg455797)...OK--> Verifying availability of your scratch dir (/scratch/00306/tg455797)...OK--> Verifying valid ssh keys...OK--> Verifying access to desired queue (development)...OK--> Verifying job request is within current queue limits...FAILED [*] Too many simultaneous jobs in queue. --> Max job limits for development = 1 jobs at org.globus.cog.abstraction.impl.scheduler.common.AbstractJobSubmissionTaskHandler.submit(AbstractJobSubmissionTaskHandler.java:63) at org.globus.cog.abstraction.impl.common.AbstractTaskHandler.submit(AbstractTaskHandler.java:45) at org.globus.cog.abstraction.impl.common.task.ExecutionTaskHandler.submit(ExecutionTaskHandler.java:61) at org.globus.cog.abstraction.coaster.service.job.manager.LocalQueueProcessor.run(LocalQueueProcessor.java:40) Then I get about 2 more repeats of this error, making the termination of the run look very messy. Ive marked this as a blocker because I have to do an important XSEDE demo this Friday, and use of the development queue is pretty necessary, The log is at: www.ci.uchicago.edu/~wilde/stampede.run004.failed-cleanup.log -- Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 2

[Bug 1177] New: Coaster-C-Client automatic coasters [enhancement]
by bugzilla-daemon＠mcs.anl.gov 08 Sep '14

08 Sep '14

https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=1177 Summary: Coaster-C-Client automatic coasters [enhancement] Product: Swift Version: trunk Platform: PC OS/Version: Linux Status: NEW Severity: normal Priority: P2 Component: General AssignedTo: hategan(a)mcs.anl.gov ReportedBy: yadunandb(a)ci.uchicago.edu Splitting bug 1133, to move Automatic coasters as a separate enhancement request. Currently the Coaster-C-Client goes through a separately started persistent coaster service. This comes with a few limitations: 1. A persistent coaster service needs to be started separately. 2. Missing the key coaster feature of expanding and shrinking worker pool. 3. Coaster's and workers remain open after execution, which can be undesirable in some cases. These features would be needed for SwiftT to take full advantage of coasters and also for language unification in the future. Currently we use scripts to submit SwiftT to resources which is very clumsy compared to what coasters can do. -- Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 3

[Bug 1249] New: Passive coaster-service failure on Trunk
by bugzilla-daemon＠mcs.anl.gov 08 Sep '14

08 Sep '14

https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=1249 Summary: Passive coaster-service failure on Trunk Product: Swift Version: trunk Platform: PC OS/Version: Linux Status: NEW Severity: normal Priority: P2 Component: General AssignedTo: hategan(a)mcs.anl.gov ReportedBy: yadunandb(a)ci.uchicago.edu I'm running Swift trunk swift-r7775 cog-r3892 on a couple of nodes with Swift running a small script connected to coaster-service in passive mode, and a single worker on a remote node (running worker.pl from the same version). This fails fairly reliably with the following error. 2014-04-18 21:54:52,853+0000 WARN RequestReply Failed to de-serialize remote exception java.io.EOFException at java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2598) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1318) at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370) at org.globus.cog.coaster.RequestReply.errorReceived(RequestReply.java:325) at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleReply(AbstractCoasterChannel.java:422) at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:234) at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97) at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56) 2014-04-18 21:54:53,630+0000 INFO RuntimeStats$ProgressTicker Submitting:4 Failed but can retry:1 2014-04-18 21:54:53,630+0000 INFO RuntimeStats$ProgressTicker HeapMax: 1037959168, CrtHeap: 27066368, UsedHeap: 8795736 -- Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 4

[Bug 907] New: Dangling coaster workers interfere with subsequent Swift runs
by bugzilla-daemon＠mcs.anl.gov 08 Sep '14

08 Sep '14

https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=907 Summary: Dangling coaster workers interfere with subsequent Swift runs Product: Swift Version: 0.94 Platform: PC OS/Version: Mac OS Status: ASSIGNED Severity: normal Priority: P2 Component: Providers AssignedTo: hategan(a)mcs.anl.gov ReportedBy: wilde(a)mcs.anl.gov CC: hategan(a)mcs.anl.gov, davidk(a)ci.uchicago.edu, wozniak(a)mcs.anl.gov Two problems here: - workers are not always cleaned up - such "dangling: workers interfere with subsequent runs Lorenzo has a log file (first message below) showing at least one manifestation of this problem. Second message indicates killing the abandoned workers solves the problems. Can we: - do a better job cleaning up abandoned workers? - make it so an abandoned worker cant connect to a subsequent server (or that if they do, no bad things, and only good things, happen?) --- ----- Forwarded Message ----- From: "Lorenzo Pesce" <lpesce(a)uchicago.edu> To: "Michael Wilde" <wilde(a)mcs.anl.gov> Sent: Saturday, December 15, 2012 12:43:06 PM Subject: 500 nodes failed again failed 500 node run: lpesce@login3:/lustre/beagle/GCNet/RG/Oreo/o080522_BS1> pwd /lustre/beagle/GCNet/RG/Oreo/o080522_BS1 -rw-rw-r-- 1 lpesce ci-users 549729889 2012-12-15 12:40 PackAndCausalTestCore_app-20121215-1801-2d9bnr57.log -rw-rw-r-- 1 lpesce ci-users 569522 2012-12-15 12:40 PackAndCausalTestCore_app.swift_2012-12-15_12:01:12_screen.log drwxrwxr-x 2 lpesce ci-users 4096 2012-12-15 12:40 PackAndCausalTestCore_app-20121215-1801-2d9bnr57.d In case you are interested in doing some post mortem. ----- Forwarded Message ----- From: "Lorenzo Pesce" <lpesce(a)uchicago.edu> To: "Michael Wilde" <wilde(a)mcs.anl.gov> Sent: Saturday, December 15, 2012 7:40:19 PM Subject: I know why it doesn't work sometimes I just realized that sometimes when swift gets killed or stops some jobs are left queued or running. These jobs create interference when the new jobs are sent. I tried a couple of times and if I clean up the queue, there seem to be no issues. I sent out a 500 nodes job and it has been happily churning for a while. -- Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 1