Sharing some lessons learned with the bigger group. Begin forwarded message:

From: Greg Cross <[email protected]> Subject: Re: Status Date: June 9, 2013 4:09:37 PM CDT To: CI Core Admins <[email protected]>

P.S. I have labeled the two root disk data cables with "P0: c7" and "P1: c8" so future replacements do not necessitate the manual procedure of identifying which cable is associated with the failed drive. (P for SATA path, c for Solaris controller number.)

Greg Cross wrote:

...

Things are okay now. There was quite a bit of minutiae to address:

* The secondary disk (SATA path 1, Solaris c8) was in the disabled boot device in the BIOS; initially I got around this by swapping controller cables, but this change was reverted so Solaris's device order matched that of the BIOS. Currently, the system is set to boot from the secondary disk before the primary disk (SATA path 0, Solaris c7). This shouldn't hurt anything, but it should probably be swapped back eventually in the BIOS.

* Also, replacing a disk necessitated the "-r" (rescan devices) flag be included as an argument to the kernel boot parameters in GRUB. I discovered this after adding various verbosity flags in GRUB "-v -m verbose" and seeing the boot hang at "[ milestone/devices:default starting (device configuration milestone) ]"; this happened regardless of whether or not I included the "-s" (single user) flag. I remembered that Solaris generally wants a device rescan if there are significant hardware changes since the last boot; in this case, the dissimilar geometries of the failed and replacement disks necessitated this step.

* Once I got into the system in single user mode, I had to create a new Solaris partition table and create an appropriately-sized s0 (partition/"slice" 0) before zpool would accept the new disk.

* sal9000's serial console port (port 2) was used by one of the VM servers (vms1). I have reclaimed it since ts-1 and con were configured for its use; vms1's serial connection has been moved to port 10 of ts-1.

The disk is now resilvering with an estimated completion in two to three hours. Once this is complete I will need to apply changes to the GRUB configuration and reinstall it on both disks (the system needs to boot from the first entry, not second, and I want to permanently configure serial output).

At some point I may request another sanity reboot to make sure the system will boot up on its own without hand-holding, but we're fine to operate indefinitely for the time being.

-- Greg

David Forero wrote:

...

I just talked to Greg. He's had a rough afternoon. After battling for a while he finally has the box in single user mode. He's currently trying to get the new disk to be recognized. He thinks optimistically another hour or two and hopefully it will be all done.

David

-- Craig