https://bugzilla.mcs.anl.gov/swift/show_bug.cgi?id=1054 Summary: Implement retry around local scheduler commands Product: Swift Version: 0.94 Platform: PC OS/Version: Mac OS Status: ASSIGNED Severity: normal Priority: P2 Component: Providers AssignedTo: [email protected] ReportedBy: [email protected] CC: [email protected], [email protected], [email protected] We're seeing cases on the Argonne PBS system "fusion" where qstat and qsub will occasionally hang trying to connect to a PBS service node. I suggest we consider retrying all/certain failing scheduler commands say 3 times? (perhaps configurable in provider-nnn.properties? Here's what I reported to LCRC Support on this: --- Both manually and from Swift Im seeing occasional timeouts from the qsub and qstat commands on flogin, trying to connect to fmgt2. For example, manually typing a handful of qstat commends in a row, about in 10 seems to hang, like this: fusion$ qstat -u wilde Connection timed out qstat: cannot connect to server fmgt2.lcrc.anl.gov (errno=110) Connection timed out fusion$ Looking at that process with strace, its indeed hanging in connect(): fusion$ ps -u wilde -H -f UID PID PPID C STIME TTY TIME CMD ... wilde 29449 29447 0 17:19 ? 00:00:00 sshd: wilde@pts/9 wilde 29450 29449 0 17:19 pts/9 00:00:00 -bash wilde 29662 29450 0 17:44 pts/9 00:00:00 emacs t1.sub wilde 29674 29450 0 17:45 pts/9 00:00:00 qstat -u wilde fusion$ strace -p 29674 Process 29674 attached - interrupt to quit connect(3, {sa_family=AF_INET, sin_port=htons(15001), sin_addr=inet_addr("140.221.71.72")}, 16 <unfinished ...> Process 29674 detached fusion$ Similarly, Im seeing connect failures from the swift command to qsub in Clare's science runs on fusion. Is this a known symptom? We'll need to implement some retry logic in Swift, but wondering if you think there's a networking or service problem between flogin2 and fmgt2? --- -- Configure bugmail: https://bugzilla.mcs.anl.gov/swift/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.