April 2012 - swift-bugs - mailman3.cels.anl.gov

[Bug 768] New: Determine optimal retry behavior
by bugzilla-daemon＠mcs.anl.gov 05 Jul '12

05 Jul '12

https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=768 Summary: Determine optimal retry behavior Product: Swift Version: trunk Platform: PC OS/Version: Linux Status: NEW Severity: normal Priority: P2 Component: Design Decision AssignedTo: wozniak(a)mcs.anl.gov ReportedBy: davidk(a)ci.uchicago.edu CC: hategan(a)mcs.anl.gov, wilde(a)mcs.anl.gov Determine optimal retry behavior. The user guide says this about retries: ----- Site selection will occur for retried jobs in the same way that it happens for new jobs. Retried jobs may run on the same site or may run on a different site. ----- There was some initial discussion of this on swift-devel (titled "Question about retry behavior"). My suggestion would be to never re-run a failed task on the node where it just failed. It would also be nice to automatically remove "bad" nodes when many tasks are failing. -- Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 1

[Bug 690] New: Timeout
by bugzilla-daemon＠mcs.anl.gov 12 Jun '12

12 Jun '12

https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=690 Summary: Timeout Product: Swift Version: 0.93 Platform: All OS/Version: Linux Status: NEW Severity: normal Priority: P2 Component: Providers AssignedTo: hategan(a)mcs.anl.gov ReportedBy: ketan(a)mcs.anl.gov CC: hategan(a)mcs.anl.gov, davidk(a)ci.uchicago.edu, wozniak(a)mcs.anl.gov Brief Desc: When running Swift catsn example from bridled to OSG via coasters and provider staging, after about 5-15 minutes, Swift throws a timeout exception. This is followed by a series of "handling reply timeout" messages during which the workflow progress seems to have been halted. The same progress status message continues on the screen. Setup: provider staging; manual coasters; passive submit-host: bridled.ci.uchicago.edu data source: /gpfs/pads data destination: /gpfs/pads execution site: OSG site "UCR" worker submission: Condor GlideinWMS Application: catsn Jobs: 100 data size: 10MB per catsn Error message: Timeout org.globus.cog.karajan.workflow.service.TimeoutException: Handler(78, PUT): timed out receiving request. Last time 940817-011255.807, now: 120119-113300.193 at org.globus.cog.karajan.workflow.service.handlers.RequestHandler.handleTimeout(RequestHandler.java:124) at org.globus.cog.karajan.workflow.service.channels.AbstractKarajanChannel.checkTimeouts(AbstractKarajanChannel.java:131) at org.globus.cog.karajan.workflow.service.channels.AbstractKarajanChannel.checkTimeouts(AbstractKarajanChannel.java:123) at org.globus.cog.karajan.workflow.service.channels.AbstractKarajanChannel$1.run(AbstractKarajanChannel.java:116) at java.util.TimerThread.mainLoop(Timer.java:512) at java.util.TimerThread.run(Timer.java:462) Command(267, HEARTBEAT): handling reply timeout; sendReqTime=120119-133503.813, sendTime=691231-180000.000, now=120119-133704.275, channel=SC-null Service log: http://ci.uchicago.edu/~ketan/timeout-bug-service.log Service stdout: http://ci.uchicago.edu/~ketan/timeout-bug-service.out Worker log: http://ci.uchicago.edu/~ketan/timeout_worker_log.txt -- Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 63

[Bug 784] Concurrent modification exception in trunk
by bugzilla-daemon＠mcs.anl.gov 30 Apr '12

30 Apr '12

https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=784 Mihael Hategan <hategan(a)mcs.anl.gov> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |RESOLVED Resolution| |FIXED --- Comment #1 from Mihael Hategan <hategan(a)mcs.anl.gov> 2012-04-30 02:31:08 --- Sorry about that. Fixed. -- Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 0

[Bug 782] New: worker.pl fails to run on some systems
by bugzilla-daemon＠mcs.anl.gov 25 Apr '12

25 Apr '12

https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=782 Summary: worker.pl fails to run on some systems Product: Swift Version: trunk Platform: PC OS/Version: Linux Status: NEW Severity: normal Priority: P2 Component: Providers AssignedTo: hategan(a)mcs.anl.gov ReportedBy: davidk(a)ci.uchicago.edu CC: hategan(a)mcs.anl.gov, davidk(a)ci.uchicago.edu, wozniak(a)mcs.anl.gov I see this while trying to run trunk locally on communicado. On other systems it seems fine. This is the error I see in the swift log (causes the application to fail with an error code of 255) 2012-04-24 14:43:37,707-0500 INFO Block Worker task failed: Bareword "POSIX::EWOULDBLOCK" not allowed while "strict subs" in use at /home/davidk/.globus/coasters/cscript1534233173477550576.pl line 466. Execution of /home/davidk/.globus/coasters/cscript1534233173477550576.pl aborted due to compilation errors. -- Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 2

[Bug 780] New: Add ability to pause a worker when an app detects a node problem
by bugzilla-daemon＠mcs.anl.gov 21 Apr '12

21 Apr '12

https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=780 Summary: Add ability to pause a worker when an app detects a node problem Product: Swift Version: 0.93 Platform: PC OS/Version: Mac OS Status: NEW Severity: enhancement Priority: P2 Component: Providers AssignedTo: davidk(a)ci.uchicago.edu ReportedBy: wilde(a)mcs.anl.gov CC: hategan(a)mcs.anl.gov, davidk(a)ci.uchicago.edu, wozniak(a)mcs.anl.gov When a node has a system problem, that node can cause many app() calls to rapidly fail. We need a way to "stun" such a node so that it stops accepting work, reports the node identity and failure, and holds the node idle for the duration of the script (workflow) execution. A possible approach is: - when app detects a failure, it calls a utility script "stopnode" - this create a file /tmp/$USER/stopnode.txt with a 1-line action code and N lines of message. - stopnode() then finds the app's worker in its process ancestry and sends the worker.pl script a signal - the worker accepts the signal subject to critical section logic, and based on the code either: -- quiesces the worker and sleeps -- causes the worker to immediately exit -- possible fails any other app() calls being run by the worker - if a new worker tries to start in the node, it blocks or fails, depending on the recency of the stopnode.txt file, etc. This logic will take some experimentation to get right. The specific case that drove this is that one node one the Eureka cluster is missing its /scratch filesystem, causing my jobs to fail. Other nodes by the way seem to have transient hard disk failures that cause things to run way slower due to retries. We'd want some similar logic to detect slow nodes and take them out of service. -- Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 0

[Bug 778] New: With status.mode=file, app non-zero exit code is returned as success
by bugzilla-daemon＠mcs.anl.gov 21 Apr '12

21 Apr '12

https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=778 Summary: With status.mode=file, app non-zero exit code is returned as success Product: Swift Version: 0.93 Platform: PC OS/Version: Mac OS Status: NEW Severity: normal Priority: P2 Component: Runtime AssignedTo: davidk(a)ci.uchicago.edu ReportedBy: wilde(a)mcs.anl.gov This occurs on Eureka with automatic coasters and cobalt. Same test succeeds with status.mode=provider. Replicate using single app call to /bin/false. (see /home/wilde/swift/lab/fail.swift and cf, cf2, cff, if needed). Not sure if this happens on other providers with coasters. sites entry was: <pool handle="cobalt"> <execution provider="coaster" jobmanager="local:cobalt" url="eureka.alcf.anl.gov" /> <filesystem provider="local" />  <profile namespace="globus" key="project">MTCScienceApps</profile> <profile namespace="globus" key="queue">default</profile> <profile namespace="karajan" key="jobthrottle">2.56</profile> <profile namespace="karajan" key="initialScore">10000</profile> <profile namespace="globus" key="jobsPerNode">8</profile>   <profile namespace="globus" key="slots">32</profile> <profile namespace="globus" key="maxTime">28000</profile>  <profile namespace="globus" key="maxWallTime">00:20:00</profile> <profile namespace="globus" key="lowoverallocation">100</profile> <profile namespace="globus" key="highoverallocation">100</profile> <profile namespace="globus" key="nodeGranularity">1</profile> <profile namespace="globus" key="maxNodes">1</profile> <workdirectory>$PWD/$rundir/swiftwork</workdirectory> </pool> -- Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 0

[Bug 774] New: Need way to set user.home when $HOME is not accessible to compute nodes
by bugzilla-daemon＠mcs.anl.gov 17 Apr '12

17 Apr '12

https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=774 Summary: Need way to set user.home when $HOME is not accessible to compute nodes Product: Swift Version: 0.93 Platform: Other OS/Version: Linux Status: ASSIGNED Severity: normal Priority: P2 Component: Runtime AssignedTo: davidk(a)ci.uchicago.edu ReportedBy: wilde(a)mcs.anl.gov When access to /home by compute nodes was removed on Beagle, coasters could not start, as the $HOME/.globus/coasters/longnameworker.pl script was not readable by the compute node. Below is the swift-user email thread on this. One resolution is to enable USER_HOME as a swift command env var. This needs to be resolved in Beagle's swift module as well as a fix in trunk. --- From email thread: OK, here's a workaround for this problem: You need to add this line to the swift command bin/swift in your Swift release. After: updateOptions "$SWIFT_HOME" "swift.home" Add: updateOptions "$USER_HOME" "user.home" This is near line 92 in the version I tested, Swift trunk swift-r5739 cog-r3368. Then you can do: USER_HOME=/lustre/beagle/wilde swift -config cf -tc.file tc -sites.file pbs.xml catsn.swift -n=1 Lorenzo, if you are using "module load swift" we'll need to update that, or you can copy the swift release directory structure that module load points you to, then modify the swift command there, and put that modified release first in your PATH. We'll work out a way to get something like this into the production module and trunk. I dont know of other systems that are currently affected by this, but Im sure they will come up. - Mike ----- Original Message ----- > From: "Michael Wilde" <wilde(a)mcs.anl.gov> > To: "Jonathan Monette" <jonmon(a)mcs.anl.gov> > Cc: swift-user(a)ci.uchicago.edu > Sent: Saturday, April 14, 2012 10:13:40 AM > Subject: Re: [Swift-user] Error message on Cray XE6 > stackoverflow says this should work: > > java -Duser.home=<new_location> <your_program> > > Need to get that in via the swift command. > > - Mike > > > ----- Original Message ----- > > From: "Michael Wilde" <wilde(a)mcs.anl.gov> > > To: "Jonathan Monette" <jonmon(a)mcs.anl.gov> > > Cc: "Lorenzo Pesce" <lpesce(a)uchicago.edu>, > > swift-user(a)ci.uchicago.edu > > Sent: Saturday, April 14, 2012 10:10:00 AM > > Subject: Re: [Swift-user] Error message on Cray XE6 > > I just tried both setting HOME=/lustre/beagle/wilde and setting > > user.home to the same thing. Neither works. I think user.home is > > coming from the Java property, and that doesnt seem to be influenced > > by the HOME env var. I was about to look if Java can be asked to > > change home. Maybe by setting a command line arg to Java. > > > > - Mike > > > > ----- Original Message ----- > > > From: "Jonathan Monette" <jonmon(a)mcs.anl.gov> > > > To: "Michael Wilde" <wilde(a)mcs.anl.gov> > > > Cc: "Lorenzo Pesce" <lpesce(a)uchicago.edu>, > > > swift-user(a)ci.uchicago.edu > > > Sent: Saturday, April 14, 2012 10:02:14 AM > > > Subject: Re: [Swift-user] Error message on Cray XE6 > > > That is an easy fix I believe. I know where the code is so I will > > > change and test. > > > > > > In the mean time could you try something? Try setting > > > user.home=<someplace.on.lustre> > > > in your config file and try again. > > > > > > On Apr 14, 2012, at 9:58, Michael Wilde <wilde(a)mcs.anl.gov> wrote: > > > > > > > /home is no longer mounted by the compute nodes, per the > > > > post-maitenance summary: > > > > > > > > "External filesystem dependencies minimized: Compute nodes and > > > > the > > > > scheduler should now continue to process and complete jobs > > > > without > > > > the threat of interference of external filesystem outages. > > > > /gpfs/pads is only available on login1 through login5; /home is > > > > on > > > > login and mom nodes only." > > > > > > > > So we need to (finally) remove Swift's dependence on > > > > $HOME/.globus > > > > and $HOME/.globus/scripts in particular. > > > > > > > > I suggest - since the swift command already needs to write to > > > > "." > > > > - > > > > that we create a scripts/ directory in "." instead of > > > > $HOME/.globus. > > > > And this should be used by any provider that would have > > > > previously > > > > created files below .globus. > > > > > > > > I'll echo this to swift-devel and start a thread there to > > > > discuss. > > > > Its possible there's already a property to cause scripts/ to be > > > > created elsewhere. If not, I think we should make one. I think > > > > grouping the scripts created by a run into the current dir, > > > > along > > > > with the swift log, _concurrent, and (in the conventions I use > > > > in > > > > my > > > > run scripts) swiftwork/. > > > > > > > > Lorenzo, hopefully we can at least get you a workaround for this > > > > soon. > > > > > > > > You *might* be able to trick swift into doing this by setting > > > > HOME=/lustre/beagle/$USER. I already tried a symlink under > > > > .globus > > > > and that didnt work, as /home is not even readable by the > > > > compute > > > > nodes, which in this case need to run the coaster worker (.pl) > > > > script. > > > > > > > > - Mike > > > > > > > > > > > > ----- Original Message ----- > > > >> From: "Lorenzo Pesce" <lpesce(a)uchicago.edu> > > > >> To: "Jonathan Monette" <jonmon(a)mcs.anl.gov> > > > >> Cc: swift-user(a)ci.uchicago.edu > > > >> Sent: Saturday, April 14, 2012 8:15:39 AM > > > >> Subject: Re: [Swift-user] Error message on Cray XE6 > > > >> In principle the access to the /home filesystem should still be > > > >> there. > > > >> > > > >> The only thing I did was to chance the cf file to remove some > > > >> errors I > > > >> had into it, so that might also be the source of the problem. > > > >> This > > > >> is > > > >> what it looks like now: > > > >> (BTW, the comments are not mine, I run swift only from lustre) > > > >> > > > >> > > > >> # Whether to transfer the wrappers from the compute nodes > > > >> # I like to launch from my home dir, but keep everything on > > > >> # lustre > > > >> wrapperlog.always.transfer=false > > > >> > > > >> #Indicates whether the working directory on the remote site > > > >> # should be left intact even when a run completes successfully > > > >> sitedir.keep=true > > > >> > > > >> #try only once > > > >> execution.retries=1 > > > >> > > > >> # Attempt to run as much as possible, i.g., ignore non-fatal > > > >> errors > > > >> lazy.errors=true > > > >> > > > >> # to reduce filesystem access > > > >> status.mode=provider > > > >> > > > >> use.provider.staging=false > > > >> > > > >> provider.staging.pin.swiftfiles=false > > > >> > > > >> foreach.max.threads=100 > > > >> > > > >> provenance.log=false > > > >> > > > >> > > > >> > > > >> > > > >> On Apr 14, 2012, at 12:10 AM, Jonathan Monette wrote: > > > >> > > > >>> The perl script is the worker script that is submitted with > > > >>> PBS. > > > >>> I > > > >>> have not tried to run on Beagle since the maintenance period > > > >>> has > > > >>> ended so I am not exactly sure why the error popped up. One > > > >>> reason > > > >>> could be that the home file system is no longer mounted on the > > > >>> compute nodes. I know they spoke about that being a > > > >>> possibility > > > >>> but > > > >>> not sure they implemented that during the maintenance period. > > > >>> Do > > > >>> you > > > >>> know if the home file system is still mounted on the compute > > > >>> nodes? > > > >>> > > > >>> On Apr 13, 2012, at 17:18, Lorenzo Pesce <lpesce(a)uchicago.edu> > > > >>> wrote: > > > >>> > > > >>>> Hi -- > > > >>>> I haven't seen this one before: > > > >>>> > > > >>>> Can't open perl script > > > >>>> "/home/lpesce/.globus/coasters/cscript7176272791806289394.pl": > > > >>>> No > > > >>>> such file or directory > > > >>>> > > > >>>> The config of the cray has changed, might this have anything > > > >>>> to > > > >>>> do > > > >>>> with it? > > > >>>> I have no idea what perl script is it talking about and why > > > >>>> it > > > >>>> is > > > >>>> looking to home. > > > >>>> > > > >>>> Thanks a lot, > > > >>>> > > > >>>> Lorenzo > > > >>>> > > > >>>> > > > >>>> > > > >>>> _______________________________________________ > > > >>>> Swift-user mailing list > > > >>>> Swift-user(a)ci.uchicago.edu > > > >>>> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user > > > >> > > > >> _______________________________________________ > > > >> Swift-user mailing list > > > >> Swift-user(a)ci.uchicago.edu > > > >> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user > > > > > > > > -- > > > > Michael Wilde > > > > Computation Institute, University of Chicago > > > > Mathematics and Computer Science Division > > > > Argonne National Laboratory > > > > > > > > -- > > Michael Wilde > > Computation Institute, University of Chicago > > Mathematics and Computer Science Division > > Argonne National Laboratory > > -- > Michael Wilde > Computation Institute, University of Chicago > Mathematics and Computer Science Division > Argonne National Laboratory > > _______________________________________________ > Swift-user mailing list > Swift-user(a)ci.uchicago.edu > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user -- Michael Wilde Computation Institute, University of Chicago Mathematics and Computer Science Division Argonne National Laboratory _______________________________________________ Swift-user mailing list Swift-user(a)ci.uchicago.edu https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user -- Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 2

[Bug 760] New: Tutorial example for iterate needs improvement
by bugzilla-daemon＠mcs.anl.gov 17 Apr '12

17 Apr '12

https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=760 Summary: Tutorial example for iterate needs improvement Product: Swift Version: 0.93 Platform: PC OS/Version: Mac OS Status: NEW Severity: normal Priority: P2 Component: Documentation AssignedTo: davidk(a)ci.uchicago.edu ReportedBy: wilde(a)mcs.anl.gov See private email to David: - example need more clarity - retry should be 0 - no clear error message when app executable path was invalid - users think that path has flexibility that is in fact lacking: - no ~ or $HOME - rel path doesnt work without further env specs based on experience of Heather Stoller -- Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 1

[Bug 772] New: coaster-service dies when a non-worker connects
by bugzilla-daemon＠mcs.anl.gov 14 Apr '12

14 Apr '12

https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=772 Summary: coaster-service dies when a non-worker connects Product: Swift Version: trunk Platform: PC OS/Version: Linux Status: NEW Severity: normal Priority: P2 Component: Providers AssignedTo: wozniak(a)mcs.anl.gov ReportedBy: davidk(a)ci.uchicago.edu CC: hategan(a)mcs.anl.gov, davidk(a)ci.uchicago.edu, wozniak(a)mcs.anl.gov When running coaster-service on an open network port, some type of web crawler connected to the coaster port and ran the command "GET /robots.txt". This caused coaster-service to go crazy and write about a zillion checksum error messages. A few minutes later I saw some coaster timeouts, then the entire coaster-service died with an exit code of 3. I was using -nosec, but I believe this also happens when using security too.. needs a little more testing. -- Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 0

[Bug 762] New: Swift dies with IOException for too many open files
by bugzilla-daemon＠mcs.anl.gov 10 Apr '12

10 Apr '12

https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=762 Summary: Swift dies with IOException for too many open files Product: Swift Version: 0.93 Platform: Other OS/Version: Mac OS Status: ASSIGNED Severity: critical Priority: P2 Component: Providers AssignedTo: jonmon(a)utexas.edu ReportedBy: wilde(a)mcs.anl.gov CC: hategan(a)mcs.anl.gov, davidk(a)ci.uchicago.edu, wozniak(a)mcs.anl.gov Jon and David are both seeing this. Jon, on Raven: suspected cause is sockets from workers to (embedded) coaster service lingering in CLOSE_WAIT state. Q: why are they closing? And if thats valid, can we make then NO_LINGER? David, on ibicluster: suspect cause is missing file close/cleanup on pipes to SGE commands (qstat, qsub) and local executions. Jon and David to diagnose these; Mihael to assist or do fix: please work a plan out among the 3 of you. -- Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 2