http://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=1427 --- Comment #2 from Max Hutchinson <[email protected]> 2015-06-18 16:00:13 --- Moving wrapperParameterMode:"files" into each site declaration in the conf file instead of using it to set the default outside of the sites makes the immediate errors go away. Running long to create a large argument list to make sure its actually using files. --- Comment #3 from Max Hutchinson <[email protected]> 2015-06-18 16:00:23 --- Moving wrapperParameterMode:"files" into each site declaration in the conf file instead of using it to set the default outside of the sites makes the immediate errors go away. Running long to create a large argument list to make sure its actually using files. -- Configure bugmail: http://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

http://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=1427 --- Comment #4 from Max Hutchinson <[email protected]> 2015-06-19 09:02:29 --- Created an attachment (id=313) --> (http://bugzilla.mcs.anl.gov/swift/attachment.cgi?id=313) conf file This conf file has wrapperParameterMode:"files" set but still results in errors about argument errors being too long. -- Configure bugmail: http://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

http://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=1427 Mihael Hategan <[email protected]> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |ASSIGNED -- Configure bugmail: http://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

http://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=1427 Max Hutchinson <[email protected]> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|RESOLVED |REOPENED Resolution|FIXED | --- Comment #7 from Max Hutchinson <[email protected]> 2015-06-20 09:12:49 --- I had staging:direct in every site. The behavior has definitely changed, but now it doesn't seem to be passing the arguments quite right: [maxhutch@cetuslac1 nek-swift]$ ./cet debug Swift trunk git-rev: cabdb3a36b863d659cc6f9f05b5cfbbf06419348 heads/master 6387 RunID: run008 Progress: Sat, 20 Jun 2015 14:05:50+0000 Progress: Sat, 20 Jun 2015 14:05:51+0000 Submitted:1 Execution failed: Exception in genrun: Arguments: [-d, /gpfs/mira-fs0/projects/alpha-nek/nek-swift/./debug.json, -u, /gpfs/mira-fs0/projects/alpha-nek/nek-swift/./single_mode.tusr, --tdir, /projects/alpha-nek/nek-swift/debug_visc_1.0E-4, ./debug_visc_1.0E-4, --no-make, --override={"visc": 1.0E-4}] Host: cooley Directory: debug-run008/jobs/g/genrun-kjf1biam exception @ swift-int-staging.k, line: 205 providerStaging:execute2 @ swift.k, line: 142 executeSelect @ swift.k, line: 206 Caused by: Application /home/maxhutch/simple/nek-tools/genrun/genrun.py failed with an exit code of 2 ------- Application STDERR -------- usage: genrun.py [-h] [-d CONFIG] [-u USR] [-n NP] [-m] [--makenek MAKENEK] [-c] [-l] [--tdir TDIR] [--no-make] [--override OVERRIDE] name genrun.py: error: the following arguments are required: name ----------------------------------- exception @ swift-int-staging.k, line: 200 Caused by: Process completed with exit code 2 k:assign @ swift.k, line: 173 swift:execute @ debug, line: 248 app_genrun_new @ debug, line: 26 genrun @ debug, line: 24 sweep @ debug, line: 30 [where name is the only positional argument]. I put a print(argv) line at the top of genrun and got this: ['/home/maxhutch/simple/nek-tools/genrun/genrun.py', '-d /gpfs/mira-fs0/projects/alpha-nek/nek-swift/./debug.json -u /gpfs/mira-fs0/projects/alpha-nek/nek-swift/./single_mode.tusr --tdir /projects/alpha-nek/nek-swift/debug_visc_1.0E-4 ./debug_visc_1.0E-4 --no-make', '--override={"visc": 1.0E-4}'] Reverting to Swift trunk git-rev: 1d512cbda070d363e8ec3cf16e77dfcddc6e66cb heads/master 6377 yields: ['/home/maxhutch/simple/nek-tools/genrun/genrun.py', '-d', '/gpfs/mira-fs0/projects/alpha-nek/nek-swift/./debug.json', '-u', '/gpfs/mira-fs0/projects/alpha-nek/nek-swift/./single_mode.tusr', '--tdir', '/projects/alpha-nek/nek-swift/debug_visc_1.0E-4', './debug_visc_1.0E-4', '--no-make', '--override={"visc": 1.0E-4}'] So the change is grouping together arguments in longer strings, which is causing argparse to fail. -- Configure bugmail: http://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

http://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=1427 --- Comment #9 from Mihael Hategan <[email protected]> 2015-06-20 13:33:33 --- (In reply to comment #8)

(In reply to comment #7) [...]

...

The behavior has definitely changed, but now it doesn't seem to be passing the arguments quite right:

[...]

I'll take a look at that.

I can't reproduce it. Do you know what version of bash is installed on the compute nodes? -- Configure bugmail: http://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

http://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=1427 --- Comment #11 from Mihael Hategan <[email protected]> 2015-06-22 15:58:08 --- I have no clue what's happening. I tried this with bash 4.1.2 and still can't reproduce the issue. I've added some debugging statements to the part of the code that builds the argument list. Maybe that would shed some light on the issue. Can you run with the latest trunk and send me the run directory? -- Configure bugmail: http://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

http://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=1427 --- Comment #13 from Mihael Hategan <[email protected]> 2015-06-22 18:02:42 --- (In reply to comment #12)

Could this be python and/or coreutils version mismatch issue?

Max is using python3. ALCF systems has coreutils V 8.4

Is there any evidence to support that suspicion? Because I can't quite see it. "grep" works as expected, and there does not seem to be a way to convince "cut" to merge output from separate lines into fewer lines when space is used as an input field delimiter. -- Configure bugmail: http://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

http://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=1427 Max Hutchinson <[email protected]> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #313 is|0 |1 obsolete| | --- Comment #15 from Max Hutchinson <[email protected]> 2015-06-25 11:41:34 --- Created an attachment (id=315) --> (http://bugzilla.mcs.anl.gov/swift/attachment.cgi?id=315) Output logs for new failure I reran from trunk at got an error (below). It is reproducible and occurs whether or not wrapperParameterMode:"files" is set. Progress: Thu, 25 Jun 2015 16:36:34+0000 Active:1 Finished successfully:3 Execution failed: Failed to start file copy Caused by: java.lang.NullPointerException at org.griphyn.vdl.mapping.file.SymLinker.getPath(SymLinker.java:216) at org.griphyn.vdl.karajan.lib.FileCopier.toPath(FileCopier.java:131) at org.griphyn.vdl.karajan.lib.FileCopier.start(FileCopier.java:94) at org.griphyn.vdl.karajan.lib.SetFieldValue.copyNonComposite(SetFieldValue.java:367) at org.griphyn.vdl.karajan.lib.SetFieldValue.deepCopy(SetFieldValue.java:285) at org.griphyn.vdl.karajan.lib.SetFieldValue.copyArray(SetFieldValue.java:405) at org.griphyn.vdl.karajan.lib.SetFieldValue.deepCopy(SetFieldValue.java:279) at org.griphyn.vdl.karajan.lib.SetFieldValue.deepCopy(SetFieldValue.java:237) at org.griphyn.vdl.karajan.lib.SetFieldValue.function(SetFieldValue.java:174) at org.griphyn.vdl.karajan.lib.SwiftFunction.runBody(SwiftFunction.java:82) at org.globus.cog.karajan.compiled.nodes.InternalFunction.run(InternalFunction.java:175) at org.globus.cog.karajan.compiled.nodes.CompoundNode.runChild(CompoundNode.java:110) at org.globus.cog.karajan.compiled.nodes.Sequential.run(Sequential.java:41) at org.globus.cog.karajan.compiled.nodes.CompoundNode.runChild(CompoundNode.java:110) at org.globus.cog.karajan.compiled.nodes.UParallel$1.run(UParallel.java:91) at k.thr.LWThread.run(LWThread.java:247) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:906) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:929) at java.lang.Thread.run(Thread.java:761) assignment @ debug, line: 77 sweep @ debug, line: 33 -- Configure bugmail: http://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

http://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=1427 Max Hutchinson <[email protected]> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|REOPENED |RESOLVED Resolution| |FIXED --- Comment #17 from Max Hutchinson <[email protected]> 2015-06-25 15:24:48 --- Works! -- Configure bugmail: http://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.