From: Corby Schmitz <schmitz@anl.gov>
To: Dan Olson <dolson@mcs.anl.gov>, Ken Raffenetti <raffenet@mcs.anl.gov>
Cc: Hunter Matthews <hunter@mcs.anl.gov>, Core Admins <core-admins@mcs.anl.gov>, Narayan Desai <desai@mcs.anl.gov>
Sent: Sat, Aug 13, 2011 00:22:14 GMT+00:00
Subject: Re: ** PROBLEM alert - localhost/3WARE-RAID is CRITICAL **

The rebuild just completed, so it's back to normal. We can deal with the online spare on Monday. Thanks again everyone for helping get this system put back together.

Corby Schmitz
Sent from my mobile office

On Aug 12, 2011, at 13:42, Corby Schmitz wrote:

> I'm in 221 now doing some prep work.
>
> Corby Schmitz
> Sent from my mobile office
>
> On Aug 12, 2011, at 13:13, Corby Schmitz wrote:
>
>> Much appreciated.
>>
>> Corby Schmitz
>> Sent from my mobile office
>>
>> On Aug 12, 2011, at 13:02, Dan Olson wrote:
>>
>>> OK.. Ken and I are heading over there to rack a server in a little bit anyway.. I'll dig up a drive and have it ready.
>>>
>>> ----
>>> Daniel Murphy-Olson
>>> Systems Administrator
>>> Mathematics & Computer Science Division
>>> Argonne National Laboratory
>>> 630-252-0055
>>>
>>> ----- Original Message -----
>>> From: "Corby Schmitz"
>>> To: "Dan Olson"
>>> Cc: "Core Admins" , "Narayan Desai" , "Hunter Matthews"
>>> Sent: Friday, August 12, 2011 1:00:14 PM
>>> Subject: Re: ** PROBLEM alert - localhost/3WARE-RAID is CRITICAL **
>>>
>>> 221. I will be in around 2.
>>>
>>>
>>> Corby Schmitz
>>> Sent from my mobile office
>>>
>>> On Aug 12, 2011, at 12:59, Dan Olson wrote:
>>>
>>>> I'm available. Where is this server?
>>>>
>>>> ----
>>>> Daniel Murphy-Olson
>>>> Systems Administrator
>>>> Mathematics & Computer Science Division
>>>> Argonne National Laboratory
>>>> 630-252-0055
>>>>
>>>> ----- Original Message -----
>>>> From: "Schmitz Corby"
>>>> To: "Narayan Desai" , "Dan Olson" , "Hunter Matthews"
>>>> Cc: "Core Admins"
>>>> Sent: Friday, August 12, 2011 12:31:29 PM
>>>> Subject: Re: ** PROBLEM alert - localhost/3WARE-RAID is CRITICAL **
>>>>
>>>> -----BEGIN PGP SIGNED MESSAGE-----
>>>> Hash: SHA1
>>>>
>>>> Anyone have cycles today?
>>>>
>>>> - -corby
>>>>
>>>>
>>>>
>>>> On Aug 11, 2011, at 11:27 PM, Schmitz Corby wrote:
>>>>
>>>>> -----BEGIN PGP SIGNED MESSAGE-----
>>>>> Hash: SHA1
>>>>>
>>>>> If we have a drive, I will come in tomorrow and deal with this. It won't be till the afternoon as I am coming in on the red-eye and need to catch some sleep before surgery. If one of you will be around and are willing to give me a hand, that would be great.
>>>>>
>>>>> Thanks all.
>>>>>
>>>>> - -corby
>>>>>
>>>>>
>>>>>
>>>>> On Aug 10, 2011, at 11:14 PM, Corby Schmitz wrote:
>>>>>
>>>>>> Sata.
>>>>>>
>>>>>> Corby Schmitz
>>>>>> Sent from my mobile office
>>>>>>
>>>>>> On Aug 10, 2011, at 18:50, Hunter Matthews wrote:
>>>>>>
>>>>>>> I can. I'll try and remember to see what I have in stock as well.
>>>>>>>
>>>>>>> I assume these are 3.0Gb/s SATA drives? Or is that SAS?
>>>>>>>
>>>>>>> --
>>>>>>> Hunter Matthews Unix Administrator
>>>>>>> Office: Bldg 221 Room B240 Argonne National Labs, MCS
>>>>>>> Key: F0F88438 / FFB5 34C0 B350 99A4 BB02 9779 A5DB 8B09 F0F8 8438
>>>>>>> Never take candy from strangers. Especially on the internet.
>>>>>>>
>>>>>>> On Aug 10, 2011, at 8:43 PM, Narayan Desai wrote:
>>>>>>>
>>>>>>>> I'd hope so. Can anyone verify?
>>>>>>>> -nld
>>>>>>>>
>>>>>>>> On Aug 10, 2011, at 4:48 PM, Schmitz Corby wrote:
>>>>>>>>
>>>>>>>>> -----BEGIN PGP SIGNED MESSAGE-----
>>>>>>>>> Hash: SHA1
>>>>>>>>>
>>>>>>>>> the controller will just treat it like a 300, right?
>>>>>>>>>
>>>>>>>>> - -corby
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Aug 10, 2011, at 2:47 PM, Narayan Desai wrote:
>>>>>>>>>
>>>>>>>>>> I don't have any this small. I could probably set you up with the 750 though.
>>>>>>>>>> -nld
>>>>>>>>>>
>>>>>>>>>> On Aug 10, 2011, at 9:52 AM, Dan Olson wrote:
>>>>>>>>>>
>>>>>>>>>>> If its a raid 10 you with no hot spare you should replace it ASAP. One more failure and the raid is gone. If it didn't get remounted the raid controller did its job and the os never saw the failure.
>>>>>>>>>>>
>>>>>>>>>>> Its a good time to verify the backup logs also.
>>>>>>>>>>>
>>>>>>>>>>> ----
>>>>>>>>>>> Daniel Murphy-Olson
>>>>>>>>>>> Systems Administrator
>>>>>>>>>>> Mathematics & Computer Science Division
>>>>>>>>>>> Argonne National Laboratory
>>>>>>>>>>> 630-252-0055
>>>>>>>>>>>
>>>>>>>>>>> ----- Original Message -----
>>>>>>>>>>> From: "Schmitz Corby"
>>>>>>>>>>> To: "Core Admins"
>>>>>>>>>>> Sent: Wednesday, August 10, 2011 9:37:28 AM
>>>>>>>>>>> Subject: Fwd: ** PROBLEM alert - localhost/3WARE-RAID is CRITICAL **
>>>>>>>>>>>
>>>>>>>>>>> -----BEGIN PGP SIGNED MESSAGE-----
>>>>>>>>>>> Hash: SHA1
>>>>>>>>>>>
>>>>>>>>>>> Fluke just started throwing these errors last night. Looking at the system, the volume has not been remounted as RO. Does this mean that I have time and can the disk when I am back onsite next week. Anyone have thoughts on this?
>>>>>>>>>>>
>>>>>>>>>>> cschmitz@fluke:~$ df -h
>>>>>>>>>>> Filesystem Size Used Avail Use% Mounted on
>>>>>>>>>>> /dev/sda1 19G 3.2G 15G 18% /
>>>>>>>>>>> varrun 1007M 168K 1007M 1% /var/run
>>>>>>>>>>> varlock 1007M 0 1007M 0% /var/lock
>>>>>>>>>>> udev 1007M 52K 1007M 1% /dev
>>>>>>>>>>> devshm 1007M 0 1007M 0% /dev/shm
>>>>>>>>>>> /dev/sdb1 512G 1.1G 486G 1% /disks
>>>>>>>>>>> /dev/sda7 50G 218M 48G 1% /sandbox
>>>>>>>>>>> /dev/sda5 3.7G 72M 3.5G 2% /tmp
>>>>>>>>>>> /dev/sda6 3.7G 678M 2.9G 19% /var/log
>>>>>>>>>>>
>>>>>>>>>>> cschmitz@fluke:~$ mount
>>>>>>>>>>> /dev/sda1 on / type ext3 (rw,relatime,errors=remount-ro)
>>>>>>>>>>> proc on /proc type proc (rw,noexec,nosuid,nodev)
>>>>>>>>>>> /sys on /sys type sysfs (rw,noexec,nosuid,nodev)
>>>>>>>>>>> varrun on /var/run type tmpfs (rw,noexec,nosuid,nodev,mode=0755)
>>>>>>>>>>> varlock on /var/lock type tmpfs (rw,noexec,nosuid,nodev,mode=1777)
>>>>>>>>>>> udev on /dev type tmpfs (rw,mode=0755)
>>>>>>>>>>> devshm on /dev/shm type tmpfs (rw)
>>>>>>>>>>> devpts on /dev/pts type devpts (rw,gid=5,mode=620)
>>>>>>>>>>> /dev/sdb1 on /disks type ext3 (rw,relatime)
>>>>>>>>>>> /dev/sda7 on /sandbox type ext3 (rw,relatime)
>>>>>>>>>>> /dev/sda5 on /tmp type ext3 (rw,relatime)
>>>>>>>>>>> /dev/sda6 on /var/log type ext3 (rw,relatime)
>>>>>>>>>>> securityfs on /sys/kernel/security type securityfs (rw)
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> - -corby
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Begin forwarded message:
>>>>>>>>>>>
>>>>>>>>>>>> From: nagios@mcs.anl.gov (Nagios on Fluke)
>>>>>>>>>>>> Date: August 10, 2011 6:35:21 AM PDT
>>>>>>>>>>>> To: cschmitz@anl.gov
>>>>>>>>>>>> Subject: ** PROBLEM alert - localhost/3WARE-RAID is CRITICAL **
>>>>>>>>>>>>
>>>>>>>>>>>> ***** Nagios Running on fluke.anchor.anl.gov *****
>>>>>>>>>>>>
>>>>>>>>>>>> Notification Type: PROBLEM
>>>>>>>>>>>>
>>>>>>>>>>>> Service: 3WARE-RAID
>>>>>>>>>>>> Host: localhost
>>>>>>>>>>>> Address: 127.0.0.1
>>>>>>>>>>>> State: CRITICAL
>>>>>>>>>>>>
>>>>>>>>>>>> Date/Time: Wed Aug 10 08:35:21 CDT 2011
>>>>>>>>>>>>
>>>>>>>>>>>> Additional Info:
>>>>>>>>>>>>
>>>>>>>>>>>> RAID CRITICAL: 1 array not OK - Array 0 status is DEGRADED(RAID-10 on adapter 8)
>>>>>>>>>>>> URL: https://fluke.anchor.anl.gov/anchor/nagios/cgi-bin/extinfo.cgi?type=2&host=localhost&service=3WARE-RAID
>>>>>>>>>>>
>>>>>>>>>>> -----BEGIN PGP SIGNATURE-----
>>>>>>>>>>> Version: GnuPG/MacGPG2 v2.0.16 (Darwin)
>>>>>>>>>>>
>>>>>>>>>>> iD8DBQFOQpeoQhpwH3ALVFERAh40AKCh6S42E0EZ96RbKhTzFNOOT3VWmwCeOiiG
>>>>>>>>>>> ecKSeHI0FPY2GANILo+Ej2U=
>>>>>>>>>>> =FuHI
>>>>>>>>>>> -----END PGP SIGNATURE-----
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> -----BEGIN PGP SIGNATURE-----
>>>>>>>>> Version: GnuPG/MacGPG2 v2.0.16 (Darwin)
>>>>>>>>>
>>>>>>>>> iD8DBQFOQvyuQhpwH3ALVFERAuFRAKCOw9bIO6NFMpQdaQHNsbTk4FTv3QCeMjMl
>>>>>>>>> Fs3bMNGCBxSlbA5OE33JikI=
>>>>>>>>> =pFiR
>>>>>>>>> -----END PGP SIGNATURE-----
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>> -----BEGIN PGP SIGNATURE-----
>>>>> Version: GnuPG/MacGPG2 v2.0.16 (Darwin)
>>>>>
>>>>> iD8DBQFORKvDQhpwH3ALVFERApqKAJoCstK+D803FHQAV0PSVp61IGwFvACgwPKK
>>>>> /1JFOm914s/uWeP/W4Gb+tg=
>>>>> =g0S2
>>>>> -----END PGP SIGNATURE-----
>>>>
>>>> -----BEGIN PGP SIGNATURE-----
>>>> Version: GnuPG/MacGPG2 v2.0.16 (Darwin)
>>>>
>>>> iD8DBQFORWNxQhpwH3ALVFERAmrmAKCwDYTi48q7HZQx0Un/CWzTE1YWSwCgnNo5
>>>> if5IL0u/9YnXNTMfcMpMtXM=
>>>> =TLgA
>>>> -----END PGP SIGNATURE-----
>>
>>