February 2011 - Core-students - mailman3.cels.anl.gov

Tactical meeting tomorrow
by Craig Stacey 15 Feb '11

15 Feb '11

After a project rundown, I want to have a discussion on the services we're providing to other divisions, and what a reasonable expectation of service is for them. I've been asked to come up with Service Level Agreements for some of the services we're doing. It would be easy for me to arbitrarily set some expectation in terms of response time, notification methods, and such, but it's important to me that we do this as a group. My ass might be the one that gets called into someone's office, but it's the rest of you who will be the ones working on issues when they arise. The services in question are listed below. If you think of others, bring it up in the meeting. * CryptoCard * Network Connectivity * Cluster operations * RT * Mailman * Userbase (V2, when deployed) * Externally facing websites * Internally facing websites * Revision Control Systems (svn, git) * Core file servers * Desktop support (incl. laptop rebuilds) The types of things I'm looking for in each of these is what is within our scope of support for each service, what type of service monitoring we'll do, what reasonable actions can we promise in what time frame, etc. Thanks, and sorry for the late notice on this.

2 2

where's beth?
by Beth Cerny Patino 15 Feb '11

15 Feb '11

Hi, I'm at UChicago, but not for long. I forgot storage bags so I'll need to go home after my meeting with Vas. I'll have to store some extra items here so this doesn't happen again. Beth

1 1

use of login.mcs.anl.gov
by Kenneth Raffenetti 15 Feb '11

15 Feb '11

Marta, I see you are running some memory intensive processes on login2.mcs.anl.gov. Please kill these jobs and move them to one of the general use compute machines listed here: http://wiki.mcs.anl.gov/IT/index.php/General_MCS_Questions#computeservers The login servers should be used only to hop through, as they have limited resources and are shared by many users. Thanks, Ken

1 0

Network Optimization Exercise and Intermapper
by Schmitz Corby 15 Feb '11

15 Feb '11

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 I started working on this early this morning and I think I have things settled. I created an IP point-to-point connection between HPC and MCS to offload some of the traffic that currently has to leave the building in order to come right back. This was intended to reduce some latency as well as the number of hops necessary to go from one set of resources to another. I started with systems net and the network management network that lives on HPC. After some testing, I determined it was safe to allow other nets. I currently allow everything to flow across the link (as long as its local to the box, we won't transit to the internet via HPC from MCS) I had originally configured this for specific nets, but that is a nightmare of routing policy that we don't want to live with. This is all controllable by prefix-lists that exist on HPC and MCS tied to the OSPF process. I created vlan2000 that constitutes the point-to-point using 140.221.47.32/30 (.33 on hpc, .34 on mcs). So why am I telling you this? Well, When we split things up during the move, I promised myself that we would keep HPC and MCS separate. This has proven to be somewhat costly on the performance side. While there are good security reasons for keeping things truly separate, I have concluded that at the cost of performance, this is a no-win situation. Unless we find some security concern related to this, I would like to keep it wide open. There are no DMZ acls between the HPC and MCS side, though we could make that change if necessary. So why am I really telling you this? So you can help me keep an eye out for any strange problems that crop up during the day. While nagios is clean and things look good, I just want to be sure that things are really happy. In the case that things are really broken, we can simply shut down vlan2000 on either side of the point-to-point and we go back to the way things were. I hope to have data at the sync on how much traffic we are actually offloading from our external connections. As a side note, some of my planning and analysis was done with the aid of intermapper. I have this running on the mac-mini out in front of the HD. There is a web-interface for those that want to be able to check things out remotely. It can be found at: http://intermapper.mcs.anl.gov It requires credentials to log in and see the maps. systems/systems will get you in so you can poke around if you like. - -corby -----BEGIN PGP SIGNATURE----- Version: GnuPG/MacGPG2 v2.0.16 (Darwin) iD8DBQFNWomKQhpwH3ALVFERAscaAJ495Jr91QKJIB5UA8sRiErEmnQcVwCfYB0D wU2fe2qsVDNJ4VlqvQnxRhI= =r5Zz -----END PGP SIGNATURE-----

1 0

sec.pl
by Ti Leggett 14 Feb '11

14 Feb '11

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Chatted with Rick and Ken about this, but here's the sec.conf I've been using for the CI: [root@admin ~]# cat /etc/sec.conf # Documentation found at http://kodu.neti.ee/~risto/sec/ type=single ptype=substr pattern=SEC_STARTUP context=SEC_INTERNAL_EVENT desc=Assign email address action=assign %e pads-support(a)ci.uchicago.edu # Alert on all sudo attempts #type=single #ptype=RegExp #pattern=(\w*) sudo: *(\w*) #desc=$0 #action=pipe '$0' /bin/mailx -s "$1: SUDO $2" %e # Mark MOM's offline when an error is found type=SingleWithSuppress window=86400 ptype=RegExp pattern=(\S+) MOM-Error: (\S+) desc=MOM Error on $1 action=pipe '$0' /bin/mailx -s "MOM-Error: taking $1 offline" %e # Check for MCE errors type=Single ptype=RegExp pattern=(\S+) MCE: HARDWARE ERROR context=!MCE_CONTEXT desc=$0 action=create MCE_CONTEXT;\ add MCE_CONTEXT $0; type=Single ptype=RegExp pattern=(\S+) MCE: HARDWARE ERROR context=MCE_CONTEXT desc=$0 action=report MCE_CONTEXT /bin/cat | mailx -s "Hardware Error: $1" %e type=Single ptype=RegExp pattern=(\S+) MCE: desc=$0 action=add MCE_CONTEXT $0 # Watch for I/O buffer errors type=SingleWithSuppress window=86400 ptype=RegExp pattern=(\S+) kernel: kernel: Buffer I/O error on device (\S+) desc=$0 action=pipe '$0' /bin/mailx -s "Host: $1 Drive: $2 IO Errors found" %e # Check for almost full IPMI System Event Logs type=SingleWithSuppress window=604800 ptype=RegExp pattern=(\S+) ipmievd: SEL buffer is 9\d\% full desc=IPMI SEL log full on $1 action=pipe '$0' /bin/mailx -s "SEL buffer is almost full on $1" %e # Check for full IPMI System Event Logs type=SingleWithSuppress window=604800 ptype=RegExp pattern=(\S+) ipmievd: SEL buffer used at 100\% desc=IPMI SEL log full on $1 action=pipe '$0' /bin/mailx -s "SEL buffer is full on $1" %e # Check for failed Power Supply type=SingleWithSuppress window=604800 ptype=RegExp pattern=(\S+) ipmievd: Power Supply sensor (Power Supply|PS) \d+( Status)? - Failure desc=Failed Power Supply on $1 action=pipe '$0' /bin/mailx -s "Failed Power Supply on $1" %e # Watch for ipmievd memory errors type=SingleWithSuppress window=86400 ptype=RegExp pattern=(\S+) ipmievd: Memory sensor desc=Memory Error on $1 action=pipe '$0' /bin/mailx -s "Memory error on $1" %e # Watch for ipmievd temperature errors type=SingleWithSuppress window=86400 ptype=RegExp pattern=(\S+) ipmievd: Temperature sensor .* - Transition to Critical desc=Temperature Warning on $1 action=pipe '$0' /bin/mailx -s "Temperature Warning on $1" %e # Watch mptbase raid cards for changes type=SingleWithSuppress window=86400 ptype=RegExp pattern=(\S+) mptbase: ioc0: RAID STATUS CHANGE for PhysDisk desc=RAID Error on $1 action=pipe '$0' /bin/mailx -s "RAID error on $1" %e # Watch for DDN temperature errors type=SingleWithSuppress window=900 ptype=RegExp pattern=(\S+) SES_(\w+):\s+EncID:\w*: Temp Sensor 0 : (\d+ C) : OVERTMP \w+ set desc=Temperature Warning on DDN, Enclosure $2 action=pipe '$0' /bin/mailx -s "Temperature Warning on DDN, Enclosure $2 - $3" %e # Watch for DDN FC link errors type=SingleWithSuppress window=900 ptype=RegExp pattern=(\S+) DISKMAIN: No FC Link (\w+) desc=FC Link error on disk $2 action=pipe '$0' /bin/mailx -s "FC Link error on disk $2" %e # Watch for DDN disk errors #type=SingleWithSuppress #window=900 #ptype=RegExp #pattern=(\S+)\s+(\S+\s+)+(Failing|Hardware Error|Medium Error) Disk (\w+) #desc=Hardware error on disk $4 #action=pipe '$0' /bin/mailx -s "Hardware error on DDN disk $4" %e # Watch for DDN power supply failures type=SingleWithSuppress window=86400 ptype=RegExp pattern=(\S+\.)(\S+) ALRT: AVR_MON (.*) Power Supply Failure desc=$3 Power Supply Failure action=pipe '$0' /bin/mailx -s "$3 Power Supply Failure on DDN Ctrl $2 " %e # Watch for Voltaire port errors #type=SingleWithSuppress #window=900 #ptype=RegExp #pattern=(\S+) .* "ISR2004" (\S+ \d+) port (\d+) .* Link-Downed counter threshold was exceeded #desc=Link Down on Voltaire card $2, port $3 #action=pipe '$0' /bin/mailx -s "Link Down on Voltaire card $2, port $3" %e # Watch for ZFS errors type=Single ptype=RegExp pattern=DESC: context=ZFS_CONTEXT desc=$0 action=add ZFS_CONTEXT $0 type=Single ptype=RegExp pattern=IMPACT: context=ZFS_CONTEXT desc=$0 action=add ZFS_CONTEXT $0 type=Single ptype=RegExp pattern= (\w+) REC-ACTION: context=ZFS_CONTEXT desc=$0 action=add ZFS_CONTEXT $0;\ report ZFS_CONTEXT /bin/cat | mailx -s "ZFS Error on $1" %e;\ delete ZFS_CONTEXT type=Single ptype=RegExp pattern=AUTO-RESPONSE: context=ZFS_CONTEXT desc=$0 action=add ZFS_CONTEXT $0 type=Single ptype=RegExp pattern=(\w+) fmd:.[^\]](.*SEVERITY:.*)$ desc=$0 action=create ZFS_CONTEXT; add ZFS_CONTEXT $0 # Watch areca errors on opensolaris type=Single context=ARECA_$1 ptype=RegExp desc=$0 pattern=(\w+) arcmsr: (.*)$ action=add ARECA_$1 $0; set ARECA_$1 10 (report ARECA_$1 /bin/cat | mailx -s "Areca controller error on $1" %e) type=Single continue=TakeNext ptype=RegExp desc=$0 pattern=(\w+) arcmsr: .*(WARNING.*)$ action=create ARECA_$1 [root@admin ~]# -----BEGIN PGP SIGNATURE----- Version: GnuPG/MacGPG2 v2.0.14 (Darwin) iEYEARECAAYFAk1Zm4wACgkQ4RgdOxQVi0BN0QCeMN4L1AooKQcyEpZ6+TMAbQe/ gzcAn3cjo8YUmdNNvgCCRYCoUNxCttPj =q6py -----END PGP SIGNATURE-----

1 0

vacation
by Linda Winkler 14 Feb '11

14 Feb '11

I will be taking 1/2 day vacation the afternoon of 2/25 lw

1 0

Where's JohnR?
by John Roberts 14 Feb '11

14 Feb '11

I will likely be taking off 2/28 and 3/1. I should be moving those days. -- John Roberts Argonne National Laboratory MCS Division roberts(a)mcs.anl.gov

1 0

SSF Temps
by Ti Leggett 14 Feb '11

14 Feb '11

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Is there any explanation why the cold aisles in the back half of the room are so warm. At least now I have a reason why PADS computes constantly throw temp alerts. -----BEGIN PGP SIGNATURE----- Version: GnuPG/MacGPG2 v2.0.14 (Darwin) iEYEARECAAYFAk1ZQAkACgkQ4RgdOxQVi0AJ0QCaAzGEuZLtXqMMbc1J/kn26jLO zdoAn1ukTQI1svQBaay9yQ6BhjcH73xS =GPEs -----END PGP SIGNATURE-----

2 1

Where's Tina
by Tina Stanton-Piersanti 14 Feb '11

14 Feb '11

I forgot to send this on Friday. I am working from home today to attend Vinnie's Valentine's Day party. The teacher sent a note home stating that since today is a 1/2 day she needs the volunteers help to start earlier then expected since they have to have the party in the morn. I will be leaving here about 8:20 and return about 10:45. I will be online until I leave and back on after I return to finish up my 8 hours. T.

1 0

cryptocard server problems
by Dan Olson 13 Feb '11

13 Feb '11

Another automated process filled up the logs on the auth server, this time from bacon. I believe the spamming has stopped, but not before the server ran out of space and corrupted my token, along with 3 others. When someone gets in can they enable my token? ---- Daniel Murphy-Olson Systems Administrator Mathematics & Computer Science Division Argonne National Laboratory 630-252-0055

1 0