August 2014 - swift-bugs - mailman3.cels.anl.gov

[Bug 867] New: Tracing
by bugzilla-daemon＠mcs.anl.gov 18 Oct '14

18 Oct '14

https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=867 Summary: Tracing Product: Swift Version: trunk Platform: PC OS/Version: Linux Status: NEW Severity: enhancement Priority: Medium Component: Design Decision AssignedTo: hategan(a)mcs.anl.gov ReportedBy: hategan(a)mcs.anl.gov This is to keep track and discuss tracing until it's finalized. A few general things about it: - it's not supposed to be fast; instead the aim is to provide information about what happens at the source code level to help with troubleshooting scripts - it should not affect performance when it's disabled; this is currently done in a log4j style, by wrapping tracing calls in a fast isEnabled test. The format: In the log, trace lines will appear with "TRACE" as source. This way a trace can be quickly extracted with grep. The general format is (literals are unquoted): {sourceType}, line {line}, thread {thread}, {info} where: {sourceType} - one of: DECLARE, VARIABLE, ASSIGN, ITERATE, APPCALL, CALL, SWIFTCALL, FOREACH, ITERATION or a swiftscript function (such as 'swiftscript:readData'). {line} - the line in the source code associated with this trace line. {thread} - the karajan level thread. The thread name is made of tokens separated by '-'. A thread A is a parent of thread B if the name of B is the name of A with a token added to the end. For example A = "0-0-22" is the parent of "0-0-22-1". {info} - varies by type as explained below: DECLARE - {variableName} [<{mapperType}; [{paramName1}={paramValue1}[, {paramName2}={paramValue2}[, ...]]]> ASSIGN - {variableName} = {unwrappedHandle} {unwrappedHandle} ca be: 1. a primitive value (if known at the time the line is printed) 2. a file name in the form <"{name}"> if the source is a non-composite mapped type 3. the variable name if the source is a closed composite type 4. the variable name prefixed by "?" if the source is open ITERATE - {iterationVariableName} = {value} FOREACH - no info ITERATION - [{keyVariableName} = {key}, ]{valueVariableName} = {value} Whether the key part appears depends on whether the 'foreach v, k' or 'foreach v' variant was used APPCALL 1. {procName}([{argName1} = {unwrappedHandle}[, {argName2} = ...]]) - logged when an app is invoked. {unwrappedHandle} is as explained above. 2. {procName} WAIT {variable} - logged when an app needs to wait for a variable CALL {procName}([{argName1} = {unwrappedHandle}[, {argName2} = ...]]) - for compound procedure calls SWIFTCALL {procName}([{argName1} = {unwrappedHandle}[, {argName2} = ...]]) - for swift internal function calls (such as readData). Few of these are logged currently VARIABLE 1. {varName} INITIALIZED - when a variable is initialized (i.e. all the mapping dependencies have been satisfied) 2. {varName} WAIT {var2} - when a variable needs to wait for a variable that is part of a mapping dependency -- Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 3

[Bug 801] New: Built-in mappers should give an error when an unrecognized parameter is specified.
by bugzilla-daemon＠mcs.anl.gov 18 Oct '14

18 Oct '14

https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=801 Summary: Built-in mappers should give an error when an unrecognized parameter is specified. Product: Swift Version: 0.93 Platform: All OS/Version: All Status: NEW Severity: normal Priority: P2 Component: SwiftScript language AssignedTo: davidk(a)ci.uchicago.edu ReportedBy: wilde(a)mcs.anl.gov >From user R. Weiss: "So like I said, it turns out this whole issue boils down to a typo. I would suggest that swift should throw up some sort of warning if you pass something to one of the pre-defined mappers that is not defined as one of its parameters. I would have expected swift to complain that "sufix" is an unknown parameter of filesys_mapper." -- Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 1

[Bug 752] New: Coaster provider does not retry worker launches on qsub failures
by bugzilla-daemon＠mcs.anl.gov 18 Oct '14

18 Oct '14

https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=752 Summary: Coaster provider does not retry worker launches on qsub failures Product: Swift Version: 0.93 Platform: PC OS/Version: Mac OS Status: ASSIGNED Severity: normal Priority: P2 Component: Providers AssignedTo: hategan(a)mcs.anl.gov ReportedBy: wilde(a)mcs.anl.gov CC: hategan(a)mcs.anl.gov, davidk(a)ci.uchicago.edu, wozniak(a)mcs.anl.gov If a qsub to start a worker.pl job fails (in this case, due to the user having exceeded the maximum number of concurrently allowed job submissions, 32 on Eureka) then the coaster provider never tries again to start workers for those coaster slots. Hence, when the user is running multiple swift runs concurrently, and they fluctuate in the number of active workers, the swift run that encounters the worker failures stays forever at a very low active worker count. In the current situation, it stays so low (eg 1 worker out of an expected 32) that the run needs to be killed and restarted in order to get a sufficient number of workers. Logs are on eureka in /home/wilde/LZ/runNNN where NNN is between 020 and 058. -- Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 1

[Bug 722] New: fprintf presents hanging situation
by bugzilla-daemon＠mcs.anl.gov 18 Oct '14

18 Oct '14

https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=722 Summary: fprintf presents hanging situation Product: Swift Version: 0.93 Platform: PC OS/Version: Mac OS Status: NEW Severity: normal Priority: P2 Component: General AssignedTo: hategan(a)mcs.anl.gov ReportedBy: jonmon(a)utexas.edu CC: wilde(a)mcs.anl.gov If using %i and passing a float variable in the argument list, seems to have fprintf hang -- Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 2

[Bug 688] New: Testing env interactions
by bugzilla-daemon＠mcs.anl.gov 18 Oct '14

18 Oct '14

https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=688 Summary: Testing env interactions Product: Swift Version: trunk Platform: PC OS/Version: Mac OS Status: NEW Severity: normal Priority: P2 Component: Testing AssignedTo: jonmon(a)utexas.edu ReportedBy: jonmon(a)utexas.edu CC: wilde(a)mcs.anl.gov, skenny(a)uchicago.edu, davidk(a)ci.uchicago.edu, wozniak(a)mcs.anl.gov, ketan(a)mcs.anl.gov Testing and documenting interactions with setting environment variables using the env profile. Justin I believe had specific testing situation and questions. -- Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 1

[Bug 923] New: @toFloat() error gives no line number
by bugzilla-daemon＠mcs.anl.gov 12 Oct '14

12 Oct '14

https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=923 Summary: @toFloat() error gives no line number Product: Swift Version: 0.93 Platform: PC OS/Version: Mac OS Status: ASSIGNED Severity: minor Priority: P4 Component: Error messages AssignedTo: davidk(a)ci.uchicago.edu ReportedBy: wilde(a)mcs.anl.gov CC: wozniak(a)mcs.anl.gov In this expression, -startingjump was null, and the error message from @tofloat() gave no line number: (outfile,lossfile) = multi_annealing(best_filename, @tofloat(@arg("tstart", "2.0")), @tofloat(@arg("tend", "0.01")), @tofloat(@arg("trejection", "0.3")), @toint(@arg("evoreruns", "100")), @tofloat(@arg("startingjump", "2.3")), [ @tofloat(@arg("alphai", "0.0")), @tofloat(@arg("alpham", "0.0")), @tofloat(@arg("beta", "4.0")), @tofloat(@arg("gamma", "50.0")), @tofloat(@arg("delta", "-1.0")) ], @tofloat(target_innov), @toint(@arg("annealingcycles", "50")) ); } Need to see if this is still the case in 0.94. Log said: 2013-01-06 11:57:21,066-0600 INFO ExecutionContext Detailed exception: Could not convert value "" to type float at org.griphyn.vdl.karajan.lib.swiftscript.Misc.swiftscript_tofloat(Misc.java:318) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:601) stdout/err said: Swift 0.93 swift-r6113 (swift modified locally) cog-r3532 RunID: 20130106-1157-z0voc3r9 Progress: time: Sun, 06 Jan 2013 11:57:20 -0600 Execution failed: Could not convert value "" to type float -- Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 2

[Bug 919] New: Misleading error message when a structure type field is missing
by bugzilla-daemon＠mcs.anl.gov 12 Oct '14

12 Oct '14

https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=919 Summary: Misleading error message when a structure type field is missing Product: Swift Version: 0.94 Platform: PC OS/Version: Mac OS Status: ASSIGNED Severity: minor Priority: P2 Component: Error messages AssignedTo: davidk(a)ci.uchicago.edu ReportedBy: wilde(a)mcs.anl.gov CC: wozniak(a)mcs.anl.gov If an expression refers to a non-existent structure field, swift gives an error like: Compile error in assignment at line 134: Type Checkpoint is not defined. instead of: in expression ckpt.ref1: field ref1 of Type Checkpoint is not defined. -- Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 2

[Bug 1277] New: exists function should accept a file instead of needing filename()
by bugzilla-daemon＠mcs.anl.gov 12 Oct '14

12 Oct '14

https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=1277 Summary: exists function should accept a file instead of needing filename() Product: Swift Version: unspecified Platform: PC OS/Version: Linux Status: NEW Severity: normal Priority: P2 Component: SwiftScript language AssignedTo: hategan(a)mcs.anl.gov ReportedBy: hategan(a)mcs.anl.gov The reasoning is that filename() converts the path to a remote path, so it will only work correctly if the argument is mapped to a relative path that does not contain parent directory references (".."). If exists() accepted mapped data as an argument, it could use a local version of filename() that does not do the remote path conversion. -- Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 1

[Bug 1293] New: Swift log files too large
by bugzilla-daemon＠mcs.anl.gov 26 Sep '14

26 Sep '14

https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=1293 Summary: Swift log files too large Product: Swift Version: 0.94 Platform: PC OS/Version: Mac OS Status: NEW Severity: normal Priority: P2 Component: General AssignedTo: hategan(a)mcs.anl.gov ReportedBy: davidkelly(a)uchicago.edu >From a discussion on swift-devel a few months ago. There's currently no easy way to reduce very large Swift log files generated while running psim. For full scale runs, the logs were >200GB. I tried minimal and reduced logging, but it didn't make any noticeable difference. log4j can't be easily modified since Swift is coming from a module. We ended up disabling Swift logging altogether by using -logfile /dev/null. Unfortunately by doing this, we're unable to use the log plotters, which would be nice to have. --- I'm running a script in 0.94.1 that generates a large log file. The script only runs 41 tasks and finishes in under 5 minutes, but the log file is 2.7 gigabytes. I have: provenance.log=false tracing.enabled=false I've tried "-minimal.logging" and "-reduced.logging" but it doesn't make much of a difference. What else can I do to reduce the log file size? I'm running Swift from a module so I can't easily edit any swift-wide log4j settings. The log is at http://web.ci.uchicago.edu/~davidk/logs/combine-20140403-2122-csdnssc1.log.… (17mb compressed, 2.7 gigabytes uncompressed) Thanks, David --- -- Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 4

[Bug 1257] New: Swift submits cleanup job before main job is complete
by bugzilla-daemon＠mcs.anl.gov 24 Sep '14

24 Sep '14

https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=1257 Summary: Swift submits cleanup job before main job is complete Product: Swift Version: 0.95 Platform: All OS/Version: All Status: ASSIGNED Severity: blocker Priority: P1 Component: Runtime AssignedTo: hategan(a)mcs.anl.gov ReportedBy: wilde(a)mcs.anl.gov I'm preparing an XSEDE demo on stampede, and using the development queue, which is limited to one job at a time per user. Ive got maxJobs set to 1, using this 0.95 site def: site.stampede { jobManager=slurm jobQueue=development jobProject=TG-CCR130051 jobWallTime=00:05:00 taskWallTime=00:00:30 maxJobs=1 tasksPerWorker=16 initialScore=10000 filesystem=local workdir=$SCRATCH } It seems like Swift is submitting a cleanup job to SLURM before the primary job has completed. So I get this on termination: Swift 0.95 Branch SVN swift-r7785 cog-r3894 RunID: run004 ... Progress: Tue, 22 Apr 2014 14:56:56-0500 Active:1 Progress: Tue, 22 Apr 2014 14:56:57-0500 Stage out:1 Final status:Tue, 22 Apr 2014 14:56:57-0500 Finished successfully:1 org.globus.cog.abstraction.impl.common.task.TaskSubmissionException: Could not submit job (sbatch reported an exit code of 1). ----------------------------------------------------------------- Welcome to the Stampede Supercomputer -------------------------------------------------------------------> Verifying valid submit host (login1)...OK--> Verifying valid jobname...OK--> Enforcing max jobs per user...OK--> Verifying availability of your home dir (/home1/00306/tg455797)...OK--> Verifying availability of your work dir (/work/00306/tg455797)...OK--> Verifying availability of your scratch dir (/scratch/00306/tg455797)...OK--> Verifying valid ssh keys...OK--> Verifying access to desired queue (development)...OK--> Verifying job request is within current queue limits...FAILED [*] Too many simultaneous jobs in queue. --> Max job limits for development = 1 jobs at org.globus.cog.abstraction.impl.scheduler.common.AbstractJobSubmissionTaskHandler.submit(AbstractJobSubmissionTaskHandler.java:63) at org.globus.cog.abstraction.impl.common.AbstractTaskHandler.submit(AbstractTaskHandler.java:45) at org.globus.cog.abstraction.impl.common.task.ExecutionTaskHandler.submit(ExecutionTaskHandler.java:61) at org.globus.cog.abstraction.coaster.service.job.manager.LocalQueueProcessor.run(LocalQueueProcessor.java:40) Then I get about 2 more repeats of this error, making the termination of the run look very messy. Ive marked this as a blocker because I have to do an important XSEDE demo this Friday, and use of the development queue is pretty necessary, The log is at: www.ci.uchicago.edu/~wilde/stampede.run004.failed-cleanup.log -- Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 2