June 2013 - Core-other - mailman3.cels.anl.gov

Monitoring 5000 servers at Booking.com
by Craig Stacey 14 Jun '13

14 Jun '13

http://blog.booking.com/ok-monitoring-enabled.html Nice light read of setting up a monitoring service. -- Craig

1 0

Skipping lunch and heading out at 4
by Tina Stanton-Piersanti 13 Jun '13

13 Jun '13

1 0

Disk Swap in a bambi box
by Craig Stacey 12 Jun '13

12 Jun '13

Sharing some lessons learned with the bigger group. Begin forwarded message: > From: Greg Cross <grog(a)ci.uchicago.edu> > Subject: Re: Status > Date: June 9, 2013 4:09:37 PM CDT > To: CI Core Admins <core-admins(a)ci.uchicago.edu> > > P.S. I have labeled the two root disk data cables with "P0: c7" and "P1: c8" so > future replacements do not necessitate the manual procedure of identifying which > cable is associated with the failed drive. (P for SATA path, c for Solaris > controller number.) > > Greg Cross wrote: >> Things are okay now. There was quite a bit of minutiae to address: >> >> * The secondary disk (SATA path 1, Solaris c8) was in the disabled boot device >> in the BIOS; initially I got around this by swapping controller cables, but this >> change was reverted so Solaris's device order matched that of the BIOS. >> Currently, the system is set to boot from the secondary disk before the primary >> disk (SATA path 0, Solaris c7). This shouldn't hurt anything, but it should >> probably be swapped back eventually in the BIOS. >> >> * Also, replacing a disk necessitated the "-r" (rescan devices) flag be included >> as an argument to the kernel boot parameters in GRUB. I discovered this after >> adding various verbosity flags in GRUB "-v -m verbose" and seeing the boot hang >> at "[ milestone/devices:default starting (device configuration milestone) ]"; >> this happened regardless of whether or not I included the "-s" (single user) >> flag. I remembered that Solaris generally wants a device rescan if there are >> significant hardware changes since the last boot; in this case, the dissimilar >> geometries of the failed and replacement disks necessitated this step. >> >> * Once I got into the system in single user mode, I had to create a new Solaris >> partition table and create an appropriately-sized s0 (partition/"slice" 0) >> before zpool would accept the new disk. >> >> * sal9000's serial console port (port 2) was used by one of the VM servers >> (vms1). I have reclaimed it since ts-1 and con were configured for its use; >> vms1's serial connection has been moved to port 10 of ts-1. >> >> The disk is now resilvering with an estimated completion in two to three hours. >> Once this is complete I will need to apply changes to the GRUB configuration >> and reinstall it on both disks (the system needs to boot from the first entry, >> not second, and I want to permanently configure serial output). >> >> At some point I may request another sanity reboot to make sure the system will >> boot up on its own without hand-holding, but we're fine to operate indefinitely >> for the time being. >> >> -- Greg >> >> David Forero wrote: >>> I just talked to Greg. He's had a rough afternoon. After battling for a while he finally has the box in single user mode. He's currently trying to get the new disk to be recognized. He thinks optimistically another hour or two and hopefully it will be all done. >>> >>> David >>> >> > -- Craig

1 0

Where's Tina - Need to leave early next week
by Tina Stanton-Piersanti 12 Jun '13

12 Jun '13

Next week is Vinnie's play off week for baseball. Joe has training in Gurnee, Monday, Tuesday and Wed. He will not be able to get home in time to get Vinnie to his game. I will need to leave around 3:30 on the days he will be playing below so I can get him to the field by 5:00. My little man has to get his skills back after his fishing week, cause he made the 8 year old All Star Team. He will be playing that game after the Championship game on the 22nd. T. Begin forwarded message: > From: Michael Hogan <mikehogan25(a)yahoo.com> > Date: June 12, 2013 8:41:47 AM CDT > To: Tina Piersanti <tstanton(a)mcs.anl.gov> > Subject: 2013 Pinto Tournament.pdf > > > > > > > Sent from my iPhone

1 0

Hunter out
by Hunter Matthews 12 Jun '13

12 Jun '13

Don't feel well at all. Please put me down for a sick day. Sent from my iPad

1 0

Warning on solaris system configurations
by Hunter Matthews 11 Jun '13

11 Jun '13

All, Today I completed a maintenance of our big fileserver (345TB). This was caused by an earlier maintenance going bad and trying to get back to a stable/reliable configuration. Long story short - do NOT do a zpool upgrade on your rpool (I think I've mentioned this before but it bears repeating). This caused our system to not reboot without manual intervention that was deep and unreliable. The benefits of a zpool upgrade on your rpool are minuscule at best and not worth the risk. In fixing that issue I've learned another lesson - make certain that your operating system (boot) drives are either 1) attached to the motherboard ports OR 2) sized differently than your data drives. Let me explain. In attempting to fix the "system won't boot past grub command line problem" I tried pretty much everything to be found in google. Nothing worked. Since our data is on a different pool (and different disks) than the OS plan D (yes, 3 times failed) was to simply replace the OS drives and reinstall the os. Did that and would you know- you can't tell the damn data drives from the os drives if they're on the same controller(s) and the same size. Openindiana helpfully displays the device id - but only the first 8 or 10 chars. For SATA disks attached to pci hba cards thats .... all 178 of them. And remember - solaris can't tell you which are the boot drives anyway. Only the BIOS can do that. So - either make them sized differently (I'm using 1TB os drives in a system with 2,3 and 4TB data drives) or attach them direct to the motherboard. My personal preference is to go direct to the motherboard but thats not always an option and not critical. -- Hunter Matthews Unix Administrator Office: Bldg 240 Room 2139 Argonne National Labs, MCS Key: F0F88438 / FFB5 34C0 B350 99A4 BB02 9779 A5DB 8B09 F0F8 8438 Never take candy from strangers. Especially on the internet.

1 0

Where's Dan
by Dan Olson 11 Jun '13

11 Jun '13

I'll be heading out at 3pm tomorrow afternoon for a Dr's appointment. ---- Daniel Murphy-Olson Systems Administrator Mathematics & Computer Science Division Argonne National Laboratory 630-252-0055

1 0

Where's JohnR?
by John Roberts 11 Jun '13

11 Jun '13

Haven't been well for a couple days really. Please put me down for a sick day today. --- John Roberts

1 0

Where's JohnR?
by John Roberts 10 Jun '13

10 Jun '13

Stuck behind accident --- John Roberts Sent from my iPhone

1 0

Where's Dan?
by Dan Olson 09 Jun '13

09 Jun '13

I'm planning on attending the mpi tutorial in the conference center tomorrow. Since we're still tracking down connections to coredb from ALCF, that change will be delayed until Tuesday morning at the earliest. ---- Daniel Murphy-Olson Systems Administrator Mathematics & Computer Science Division Argonne National Laboratory 630-252-0055

1 0