Given the high profile of these VMs, I'd rather it be off hours. -- Craig On Jul 23, 2012, at 9:15 AM, Dan Olson wrote:
There was an opportunity to do this on Hooray this morning as there were no sessions. It came back up ok with just a normal reboot worth of downtime. Lets let it burn in for a least a few days.
Will next Tuesday during the normal ALCF window work for MCS web downtime?
---- Daniel Murphy-Olson Systems Administrator Mathematics & Computer Science Division Argonne National Laboratory 630-252-0055
----- Original Message ----- From: "Craig Stacey" <[email protected]> To: "Dan Olson" <[email protected]> Cc: "Core Admins" <[email protected]> Sent: Sunday, July 22, 2012 5:23:44 PM Subject: Re: Notifications for wilbur
Cool.
-- Craig
On Jul 22, 2012, at 4:28 PM, Dan Olson wrote:
I would like to flash the latest code on Hooray tomorrow, at noon. It has the same controller. Depending on how that goes we can do vserver2-221 the next time we can get downtime.
---- Daniel Murphy-Olson Systems Administrator Mathematics & Computer Science Division Argonne National Laboratory 630-252-0055
----- Original Message ----- From: "Dan Olson" <[email protected]> To: "Craig Stacey" <[email protected]> Cc: "Core Admins" <[email protected]> Sent: Sunday, July 22, 2012 2:25:53 PM Subject: Re: Notifications for wilbur
We're two firmware revs back (9.5.3), on firmware that was flashed when we installed the controller. In the release notes for 1 fw release back 9.5.4 is a fix for soft resets --
Unexpected soft resets (SCR-2196)
---- Daniel Murphy-Olson Systems Administrator Mathematics & Computer Science Division Argonne National Laboratory 630-252-0055
----- Original Message ----- From: "Dan Olson" <[email protected]> To: "Craig Stacey" <[email protected]> Cc: "Core Admins" <[email protected]> Sent: Sunday, July 22, 2012 2:07:33 PM Subject: Re: Notifications for wilbur
I think the 3ware controller may have reset for some reason causing this. Theres nothing good in the regular system logs, but in the hw log in the controller there is a recent soft reset recorded. I'm having trouble coordinating the timestamps with the system time though.
---- Daniel Murphy-Olson Systems Administrator Mathematics & Computer Science Division Argonne National Laboratory 630-252-0055
----- Original Message ----- From: "Craig Stacey" <[email protected]> To: "Dan Olson" <[email protected]> Cc: "Core Admins" <[email protected]> Sent: Sunday, July 22, 2012 1:41:28 PM Subject: Re: Notifications for wilbur
Okay, cool. I can't connect to their consoles, but I 'm not going to sweat that right now. -- Craig
On Jul 22, 2012, at 1:35 PM, Dan Olson wrote:
auth-console as well.
---- Daniel Murphy-Olson Systems Administrator Mathematics & Computer Science Division Argonne National Laboratory 630-252-0055
----- Original Message ----- From: "Dan Olson" <[email protected]> To: "Craig Stacey" <[email protected]> Cc: "Core Admins" <[email protected]> Sent: Sunday, July 22, 2012 1:34:22 PM Subject: Re: Notifications for wilbur
auth2 is back.. probably a fsck kept it down also.
---- Daniel Murphy-Olson Systems Administrator Mathematics & Computer Science Division Argonne National Laboratory 630-252-0055
----- Original Message ----- From: "Craig Stacey" <[email protected]> To: "Core Admins" <[email protected]> Sent: Sunday, July 22, 2012 11:52:46 AM Subject: Re: Notifications for wilbur
Wilbur seems to have come back. I'm not entirely sure why, but I suspect a particularly long fsck, given the machine uptime and when syslog starts are over an hour apart. It would be really helpful to have visibility into what's going on when the machines are booting.
auth2.mcs.anl.gov and auth-console are both still down, but neither are critical. -- Craig
On Jul 22, 2012, at 11:13 AM, Craig Stacey wrote:
I've put wilbur.mcs.anl.gov into a known down state in nagios for the next two hours so it stops spamming about all the services that are down on it. Still working to bring it up - couldn't raise Dan via phone. If anyone reading this has ideas, bring 'em. -- Craig