January 2009 - Core-sysadmins - mailman3.cels.anl.gov

Weekly Accounting (1/26-1/30)
by Schmitz Corby 30 Jan '09

30 Jan '09

Weekly Accounting (1/26-1/30) Goals: -Active Strategic *ALCF Firewall migration (P1) - scheduled for monday - complete 1/26 *Watcher - Nagios Refresh (P1) - nearly complete - will be done next week ... Wednesday *Automated tools for server configuration *Got the mac-address/ip/hostname search stuff moved here (just the visual - rest is on netman) *Integrated switch and server automated tools into one unit *Rewrote the monitoring integration between nrpe-clients and nrpe- servers *Netman - network management server (P1) - nearly complete - will be done Monday *Moving last of switches over to this server and removing need for NMS221 - done (NMS221 tacacs and radius services turned off) *Migrated config management and mac/cam analysis tools here *Package management *Got nagios-plugins built for hardy systems and rolled it out using bcfg with configuration *Fixed problems with nagios-server as well *Working on dapper binary installers - Completed Friday *Looking at sarge rollout ... this will be difficult *Redundant network tunnel for DC (P3) - need to put this in the testbed *Rewrite ACL MGR with SVN backend and diff pushing (P3) - stalling for next week *Optical Testbed - rack installed, need to order breaker panel soon (P2) - complete -Active Reactive *Helpdesk Tickets (P2) *MREN/SL Mailing Lists and Answering Service (P2) -Stalled *Strong authentication for watcher/nagios web (P1) - stalled on problems with mod_auth_pam failing *NMS221 return to CIS (with new ubuntu build and services tacacs+ and freeradius and nagios - waiting for free hardware (new NMS) *Route monitoring in anchor space (waiting on nagios refresh) Monday (~9hrs) -Removed ALCF firewall from BMR - moved to CIS (2hrs) -Migrating remaining NMS221 services (arp and net-usage tools) (3hrs) -Rewriting get_proc in the nagios-plugins package (4hrs) Tuesday (~9.5hrs) -Building binary package for nagios-plugins mcs1 (2hrs) -Writing get_mem module for nagios-plugins mcs2 (4hrs) -Building binary package for distribution (30min) -Rollout of NRPE to certain servers (3hrs) Wednesday (~9.25hrs) -NRPE rollout to additional servers (4hrs) -Migrated more switches off of NMS221 reliance (2hrs) -Started on server automation tool (2.5hrs) -Built nagios-plugins mcs2.2 for distribution (45min) Thursday (~10hrs) -Server automation tool complete (3hrs) -Rolling out updated nrpe.cfg and nagios-plugins mcs2.2 on hardy systems (3hrs) -ACL rules update and ticket cleanup (45min) -Migrated remaining switches off of NMS221 reliance (1.25hrs) -Added all compute nodes other than lemma (2hrs) Friday (~8.5hrs) -Added 3 network drops in AM for cameras and a new mac-mini (2hrs) -Installing a dapper VM for building nagios-plugins mcs2.2 (45min) -built nagios-plugins on dapper for 4.11 and 4.2 mcs-2.2 with all my changes (2.5hrs) -Built version of nagios-nrpe-server with a buffer init patch to fix a server problem, added to local repo and pushed to hardy systems (1.25hrs) -Completed and rolled out automated web server configuration scripts with flat-file seed, will add to bcfg for all-admin administration (2hrs) -- Corby Schmitz Network Engineer CIS/MCS/MREN/Starlight Argonne National Laboratory 9700 S. Cass Ave. Argonne, IL 60439 Desk: 630-252-7664 Cell: 630-795-9232 E-mail: schmitz(a)anl.gov

1 0

Status Update for Max
by Max trefonides 26 Jan '09

26 Jan '09

mercury -> quicksilver migration The migration is going well. I have nfs/nis services up and have finished the first rsync pass of home dirs. I have proof of concept for sharing cifs (SMB) direct from quicksilver's zfs filesystems, and a much more manageable automounting system. Automounts: Since we are now mounting every home dir as a separate filesystem, we gain a number of nice wins: * We can make snaps of each homedir on it's own * Eliminates the need for the ugly BACKUPS link hack, snaps will be accessible under $HOME/.zfs/ * Allows us to impose quotas on an individual's filesystem which means if they fill it up only they suffer * Allows us to lock the snaps down to the owner of the parent filesystem. This addresses behavior that we got complaints about early on with mercur: if someone set the perms two liberally on a given file then even after they fixed it on the "real" file it was available with the original perms in the snap. * Backups are on a per user basis, we can run backups more efficiently, in parallel, at a finer grain Testing: Rick B's desktop "shagrath" now has the automount bits in place to mount home dirs from quicksilver. It currently mounts them under the directory /home. /nfs/mcs-homefarm is not being used. Please log in and mess around. BE AWARE!!! The files in your home dir on shagrath are currently an just an rsync from mercury. Changes to those files will not be reflected on your regular home-dir. At this point this is just for testing. Subsequent rsync's could wipe out your changes. Project directories: I am working on the problem of where/how to mount project dirs. I think there is a way to get them automounted in sub dirs, but that might not be what the users want. I suppose I can just have them mounted under home, which is where they appear to be now. As with the home directories much of the naming and mounting was inherited from systems that had different limitations to our current systems. For example, proj-fischer0[0-3] were each seperate luns on one of the AIX fileservers. They could all probably be folded into one filesystem, I'll be contacting Paul about that one. RSYNC: There are a number of home dirs owned by users not in nis. I need some userbase foo to find out how long ago they were removed to see if they should be archived and removed. Also want to know who their sponsors are so I could make the data available to them. The attached is a list of the directories on mercury that are not homes or proj. there are a pile of www dirs that I think have been moved or are in process, etc. Please go through this list and do what you can to tell me what needs to migrate and what can be archived and removed? directories of note: sandboxes www.*

1 0

status update for ken (1/26)
by Kenneth Raffenetti 26 Jan '09

26 Jan '09

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 * installed Mailman/htDig integration patch for all list domains * imported majordomo archives for lists that have moved to Mailman * fixed WordpressMU SSL issues on press.mcs.anl.gov * moved 2 websites from bravo to wind, will try to finish moving the rest this week and then be able to turn off bravo * With John, replaced the mobo in the FL's Dell XPS and got the machine back up and in place Ken -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkl+EFUACgkQvRJB9CBtvwikvwCfW/AhOedrLhph/E52NzBLB6g2 OEcAnRMBNOE/IHn65C4zcAFRL4wklTFW =1qTe -----END PGP SIGNATURE-----

2 5

Bradshaw weekly report 1/23/09
by Rick Bradshaw 23 Jan '09

23 Jan '09

Radix: GPU machines: building,debugging,benchmarking Sicortex fileserver crash debugging Bcfg: 1.0 release testing/new concept code preparing repos for the future Virtual Help desk: first real meeting of the year. short term plans/before summer goals Systems/core: work with Corby to get machine monitoring working with nagios tid bits wih Max about data migration Help Desk week Rick

1 0

Weekly Accounting (1/19-1/23)
by Schmitz Corby 23 Jan '09

23 Jan '09

Weekly Accounting (1/19-1/23) Goals: -Active Strategic *ALCF Firewall migration (P1) - scheduled for monday *Strong authentication for watcher/nagios web (P1) - stalled on problems with mod_auth_pam failing *Watcher - Nagios Refresh (P1) *Working on CC authentication for the web *Bcfg managed with tivoli backups *Netman - network management server (P1) *Moving last of switches over to this server and removing need for NMS221 *Bcfg managed with tivoli backups *Redundant network tunnel for DC (P3) - need to put this in the testbed *Rewrite ACL MGR with SVN backend and diff pushing (P3) - stalling for next week *Optical Testbed - rack installed, need to order breaker panel soon (P2) - complete -Active Reactive *Helpdesk Tickets (P2) *MREN/SL Mailing Lists and Answering Service (P2) -Stalled *NMS221 return to CIS (with new ubuntu build and services tacacs+ and freeradius and nagios - waiting for free hardware (new NMS) *Route monitoring in anchor space (waiting on nagios refresh) Monday (~8hrs) -CC authentication via apache (using mod_auth_pam and mod_auth_radius seem to be failing) (7hrs) -Grounded new 23" rack in BMR for optical testbed (1hr) Tuesday (~9.75hrs) -Removed f10bg from monitoring (30min) -CIS Group Meeting (2hr) -Getting quotes and details on 308 expansion plan for MNS/GPE funding (30min) -Getting details together on purchase of panel refresh for i-Wire install in BMR (45min) -Installed and configured NRPE on netman (monitoring from watcher) (4hr) - done by hand -Working on package install for distribution via bcfg (should be doable) (2hr) - and was Wednesday (~8.5hrs) IT managers meeting (1.5hr) Installed and powered equipment for optical testbed (1.25) Working on tweaking bcfg Bundle for NRPE and initial rollout to key servers and creating templates to accept and graph data (5hrs) Worked with Hunter on getting 10GigE capacity for cosmea (45min) Thursday (~8.5hrs) NRPE rollout to servers (some are working fine, others are not) (8hrs) - still having issues with sarge, feisty and some other systems, but its getting there Meeting on ALCF firewall migration (30min) Friday (~9hrs) Meeting on ALCF firewall migration (45min) Arp flush web-page migration from NMS221 (1hr) NRPE rollout to more servers (login nodes and manageable web-servers done, moved on to misc servers) (7hrs) Added security (with ken's help) to the arp flush (15min)

1 0

Hunters WSR Jan-19 to Jan-23 2009
by Hunter Matthews 23 Jan '09

23 Jan '09

KBT: Worked with IBM some more from last week on closing out the backlog of hardware tickets. IBM was done by wed, but I updated and closed the RT tickets thursday. Began reading/understanding modules (like softenv, but C/tcl) for eventually switching KBT over. Need to reinstall one node (its HD was replaced) and make time to do more work on the new storage nodes (they are racked, but need installing, configuring, etc) Cosmea: Did minimal work on getting bcfg2 going again. Distracted by whatami updates. Met with users/owners on storage issues. Talked with Corby and have our first quote(s) for possible networking changes. Need to get two more quotes one for kbt/bb style storage server(s) and one for BlueArc. Stalled on quote for gpfs from IBM. MCS: More updates, so called "fixes" to whatami. I want to be done with this iteration of whatami - at this point, backwards compatibility retards most any forward progress. Checked updates into svn and announced it for testing. https://svn.mcs.anl.gov/repos/whatami/bin if people want it. -- Hunter Matthews Unix Administrator Office: Bldg 221 Room B240 Argonne National Labs, MCS Key: F0F88438 / FFB5 34C0 B350 99A4 BB02 9779 A5DB 8B09 F0F8 8438 Never take candy from strangers. Especially on the internet.

1 0

Status updates & Group syncs
by Craig Stacey 23 Jan '09

23 Jan '09

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 First up, core-sysadmins is Max, Rick, Me, Ken, Corby, Hunter -- the systems administrators (who have projects). I'm trying to find this balance between too many group meetings and too few. In-person meetings are useful, and should happen with some regularity, but I want to sync more than that once a month and there are enough meetings in our lives. The weekly status updates are helping me, but they're a one-way communication street, and don't improve group communication. I'm finding more and more groups are using the status updates to the group as their general method, and I want to go this route. Here's what I'd like to do. Every week, status updates are due by Monday, and should be sent to this mailing list (core-sysadmins(a)lists.mcs.anl.gov) If you'd rather send them on the previous Friday, that's fine. As for group syncs, we're currently doing a req buzz on the last Tuesday of the month. We're going to do group syncs on the second Tuesday of the month, same time. You should have an invitation via Zimbra for that by the time you read this. The first of these is 2/10. If you want to add something to the agenda for any of these meetings, it's all on the systems wiki under "Group Meetings". Let me know if you have any questions or concerns. == Craig -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.8 (Darwin) iEYEARECAAYFAkl2O6kACgkQbVNGA3HOzQsewACfZqpmDhX2d4OxzJppuvyCCguL WRAAn2zBaRxuVVaEbyU+10dubjIzVxiJ =VNii -----END PGP SIGNATURE-----

1 1

Fwd: Mathematica 7 Presentation: January 29, 2009 10:30 a.m. Bldg 222 rm A253
by Craig Stacey 22 Jan '09

22 Jan '09

One of the problems we have in the group right now is a lack of understanding of some of the tools we are supporting. In light of that, I think it would be beneficial for those who aren't familiar with this package to attend this. == Craig Begin forwarded message: > Subject: Mathematica 7 Presentation: January 29, 2009 10:30 a.m. > Bldg 222 rm A253 > > Good afternoon, > > Please join me in a presentation by representatives from Wolfram > Research Inc. on "A Technical Introduction of Mathematica 7" > scheduled for next week Thursday, January 29, 2009 from 10:30 a.m. > to 11:30 a.m. CST in building 222 room A253. > > I have attached the agenda for this presentation for your review. > > Please feel free to forward this email to those currently using or > is interested in Mathematica. > > Just to provide the speakers with an estimated number of attendees, > please let me know how many from your group are planning on attending. > > > Best Regards, > > > Edy Haus > IT Asset Manager > Office of the CIO > CIS Division > Argonne National Laboratory > Ph: 630-252-6565 > > A Technical Introduction to Mathematica 7 > > Thursday, January 29, 2009 > 10:30 a.m. – 11:30 a.m. > Building 222 Conference Room A253 > > By Eric Wort > Wolfram Research Technical Group > > Introduction to Speaker > by Shalini Dewan, Commercial Account Executive for Argonne > > Mathematica is a computer program used widely in scientific, > engineering and mathematical fields. It is mainly known as a > computer algebra system. Its’ first release was in 1988 by Wolfram > Research in Champaign. Version 6 brought a revolution in the concept > of interactive computing and allowed dynamic interfaces to be > created instantly as a routine part of everyday work. Mathematica 7 > continues the momentum of innovation to deliver an array of new > capabilities, greatly extending the state of the art in many areas, > and bringing a dozen major new application fields into the > integrated framework of Mathematica. > Seminar Outline > > Introduction to Mathematica v7 and its new features: > Parallel Processing > Image Processing > > Key Technologies: > Document interface > Symbolic computation > Graphics > Data handling > Symbolic programming > Numeric computation > Interactivity > > Deployment possibilities; Mathematica, gridMathematica > > Additional Resources > Mathematic-specific resources > General mathematics web resources at WRI

1 0

Re: mozilla cache
by Rick Bradshaw 21 Jan '09

21 Jan '09

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 So I think Max and I have come up with what could aid in our NFS performance issues. and here is the solution I have come up with. The downside is that every user must click and do this themselves. Or we can force a reboot of everyones machine and we sed the files ourselves( yeah right ). open firefox visit about:config modify the below entries. ( you can type browser.cache into "Filter" to aid in finding the entries. ) browser.cache.check_doc_frequency: 1( check everytime "don't look for cache" ) browser.cache.disk.capacity: 0 browser.cache.disk.enable: false browser.cache.disk_cache_ssl: false ( security reasons ) browser.cache.memory.enable: true ( will cache pages in memory ) browser.cache.offline.enable: false close and reopen firefox to ensure changes take effect. this will make a big performace change on our NFS home filesystems as firefox has aggressive caching strategies which cause a constant write stream to our server. Rick Max trefonides wrote: > to relocate the cache directory to the local disk the > browser.cache.disk.parent_directory preference can be set in prefs.js: > > http://kb.mozillazine.org/Browser.cache.disk.parent_directory -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFJd1thdavihOKkCCERAlfyAKCDruqKRx3anWwsJQAEsGVLOLfDdQCglzHH wo7sb+E5tEKBoKcyT7HQTCo= =1Gpx -----END PGP SIGNATURE-----

3 5

Problems authenticating
by Craig Stacey 21 Jan '09

21 Jan '09

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 I've got a user who cannot authenticate to Zimbra, but his credentials are fine. We've tried logging in as "fischer", "fischer-mcs", "fischer(a)zimbra.mcs.anl.gov ", and it always fails with this error: 2009-01-21 13:15:59,271 INFO [btpool0-27296] [name=fisher(a)zimbra.mcs.anl.gov ;oip=130.202.99.55;ua=zclient/5.0.11_GA_2695.UBUNTU6;] SoapEngine - handler exception: authentication failed for fischer-mcs, external LDAP auth failed, [LDAP: error code 49 - 80090308: LdapErr: DSID-0C090334, comment: AcceptSecurityContext error, data 525, vece] His credentials test works just dandy. His password was changed yesterday. Anyone have any clues? == Craig -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.8 (Darwin) iEYEARECAAYFAkl3dZMACgkQbVNGA3HOzQu2OQCeKtQp/vA3pkMs55gh3RMK9yQD tkcAoIVHUMN/Ex0i/BFDzzR51PoH/tO3 =dn+7 -----END PGP SIGNATURE-----

2 4