October 2011 - Core-other - mailman3.cels.anl.gov

Where's Tina (Friday, 10/14)
by Tina Stanton-Piersanti 14 Oct '11

14 Oct '11

I will be working remotely in the morning and then I will need to attend to a personal matter. Thanks! Tina

2 3

Backup-Failure-Notice
by root＠lal.mcs.anl.gov 14 Oct '11

14 Oct '11

ATTENTION The following systems have passed their scheduled backup times. Listed are the date and time of the last backup completed. NODE FILESPACE DATE TIME HOMES.MCS.ANL.GOV /exports/sharp 2011-10-06 19:31:14.000000 HOMES.MCS.ANL.GOV /exports/shared 2011-10-06 19:27:54.000000 HOMES.MCS.ANL.GOV /exports/proj-sysbio 2011-10-06 19:25:33.000000 HOMES.MCS.ANL.GOV /exports/proj-lans1 2011-10-06 19:23:46.000000 HOMES.MCS.ANL.GOV /exports/proj-ira 2011-10-06 19:20:40.000000 HOMES.MCS.ANL.GOV /exports/proj-genomics 2011-10-06 19:20:39.000000 HOMES.MCS.ANL.GOV /exports/proj-genogrp 2011-10-06 18:58:52.000000 HOMES.MCS.ANL.GOV /exports/proj-fluids 2011-10-06 18:37:34.000000 HOMES.MCS.ANL.GOV /exports/proj-flash 2011-10-06 18:34:29.000000 HOMES.MCS.ANL.GOV /exports/proj-fischer03 2011-10-06 18:32:31.000000 HOMES.MCS.ANL.GOV /exports/proj-fischer02 2011-10-06 18:31:25.000000 HOMES.MCS.ANL.GOV /exports/proj-fischer01 2011-10-06 18:31:19.000000 HOMES.MCS.ANL.GOV /exports/proj-fischer00 2011-10-06 18:25:54.000000 HOMES.MCS.ANL.GOV /exports/proj-dsl 2011-10-06 18:24:40.000000 HOMES.MCS.ANL.GOV /exports/proj-climate1 2011-10-06 18:22:04.000000 HOMES.MCS.ANL.GOV /exports/noaa 2011-10-06 18:01:12.000000 HOMES.MCS.ANL.GOV /exports/nmpdr-homes2 2011-10-06 18:01:10.000000 HOMES.MCS.ANL.GOV /exports/nfs4test 2011-10-06 18:01:07.000000 HOMES.MCS.ANL.GOV /exports/ms-users 2011-10-06 18:01:06.000000 HOMES.MCS.ANL.GOV /exports/ms-software 2011-10-06 18:01:02.000000 HOMES.MCS.ANL.GOV /exports/mcs-proj-source 2011-10-06 18:01:01.000000 HOMES.MCS.ANL.GOV /exports/mcs-proj-magnetic2 2011-10-06 17:58:51.000000 HOMES.MCS.ANL.GOV /exports/mcs-proj-magnetic 2011-10-06 17:57:33.000000 HOMES.MCS.ANL.GOV /exports/mcs-proj-dynamics 2011-10-06 17:57:02.000000 HOMES.MCS.ANL.GOV /exports/mcs-proj-climate 2011-10-06 17:54:33.000000 HOMES.MCS.ANL.GOV /exports/gtr.globus.org 2011-10-06 17:53:35.000000 HOMES.MCS.ANL.GOV /exports/mac-nfs-testing 2011-10-06 17:53:35.000000 HOMES.MCS.ANL.GOV /exports/geassm 2011-10-06 17:53:30.000000 HOMES.MCS.ANL.GOV /exports/UncertaintyClimate 2011-10-06 17:51:15.000000 HOMES.MCS.ANL.GOV /exports/alcf-admins 2011-10-06 17:51:15.000000 HOMES.MCS.ANL.GOV /exports/uso 2011-10-06 17:51:15.000000 AHA.MCS.ANL.GOV /failover/var 2011-10-05 05:55:16.000000 AHA.MCS.ANL.GOV /failover/tmp 2011-10-05 05:55:15.000000 AHA.MCS.ANL.GOV /failover 2011-10-05 05:55:14.000000 AHA.MCS.ANL.GOV /u99 2011-10-05 05:54:20.000000 AHA.MCS.ANL.GOV /u02 2011-10-05 05:54:19.000000 ANNO-3.NMPDR.ORG /disks/space0 2011-10-05 05:47:34.000000 HOMES.MCS.ANL.GOV /exports/software 2011-09-11 06:01:17.000000 HOMES.MCS.ANL.GOV /exports/ruppert-share 2011-09-11 05:29:13.000000 HOMES.MCS.ANL.GOV /exports/request 2011-09-11 05:29:12.000000 HOMES.MCS.ANL.GOV /exports/nt-homes001 2011-09-11 05:29:07.000000 HOMES.MCS.ANL.GOV /exports/nt-homes001/maxadam-admin2011-09-11 05:29:07.000000 HOMES.MCS.ANL.GOV /exports/mcs-homes67 2011-09-11 05:04:57.000000 HOMES.MCS.ANL.GOV /exports/mcs-homes65 2011-09-11 05:00:00.000000 HOMES.MCS.ANL.GOV /exports/mcs-homes64 2011-09-11 04:53:37.000000 HOMES.MCS.ANL.GOV /exports/mcs-homes63 2011-09-11 01:25:39.000000 HOMES.MCS.ANL.GOV /exports/mcs-homes62 2011-09-11 01:25:19.000000 HOMES.MCS.ANL.GOV /exports/mcs-homes56 2011-09-11 01:17:04.000000 HOMES.MCS.ANL.GOV /exports/mcs-homes55 2011-09-11 01:11:00.000000 HOMES.MCS.ANL.GOV /exports/mcs-homes46 2011-09-11 01:09:44.000000 HOMES.MCS.ANL.GOV /exports/mcs-homes30 2011-09-11 00:43:09.000000 HOMES.MCS.ANL.GOV /exports/mcs-homes28 2011-09-11 00:41:54.000000 HOMES.MCS.ANL.GOV /exports/mcs-homes29 2011-09-11 00:41:54.000000 HOMES.MCS.ANL.GOV /exports/mcs-homes27 2011-09-11 00:03:50.000000 HOMES.MCS.ANL.GOV /exports/mcs-homes26 2011-09-10 23:41:22.000000 HOMES.MCS.ANL.GOV /exports/mcs-homes25 2011-09-10 21:58:27.000000 HOMES.MCS.ANL.GOV /exports/mcs-homes23 2011-09-10 20:50:06.000000 HOMES.MCS.ANL.GOV /exports/mcs-homes19 2011-09-10 20:34:13.000000 HOMES.MCS.ANL.GOV /exports/mcs-homes18 2011-09-10 20:24:07.000000 HOMES.MCS.ANL.GOV /exports/mcs-homes17 2011-09-10 20:13:01.000000 HOMES.MCS.ANL.GOV /exports/mcs-homes16 2011-09-10 19:54:39.000000 HOMES.MCS.ANL.GOV /exports/mcs-homes15 2011-09-10 19:29:54.000000 HOMES.MCS.ANL.GOV /exports/mcs-homes66 2011-09-09 18:19:07.000000 HOMES.MCS.ANL.GOV /exports/CELSAdmin 2011-09-02 18:12:59.000000 BUNTINAS-MAC.MCS.ANL.GOV / 2011-08-05 16:30:44.000000 REPO.UC.TERAGRID.ORG /var/lib/cvs 2011-05-06 00:44:55.000000 PORTAL1.UC.TERAGRID.ORG /var 2011-05-05 23:49:26.000000 PORTAL1.UC.TERAGRID.ORG / 2011-05-05 23:49:25.000000 PORTAL1.UC.TERAGRID.ORG /home 2011-05-05 23:48:59.000000 PORTAL2.UC.TERAGRID.ORG /var 2011-05-05 20:47:42.000000 PORTAL2.UC.TERAGRID.ORG / 2011-05-05 20:47:41.000000 PORTAL2.UC.TERAGRID.ORG /home 2011-05-05 20:47:18.000000 HOMES.MCS.ANL.GOV /exports/media 2011-02-14 16:26:17.000000 HOMES.MCS.ANL.GOV /exports/linux-debian_sarge 2011-02-14 10:52:41.000000 HOMES.MCS.ANL.GOV /exports/www.accessgrid.org 2011-02-14 10:22:22.000000 HOMES.MCS.ANL.GOV /exports/www.bgl.mcs.anl.gov 2011-02-14 10:20:42.000000 HOMES.MCS.ANL.GOV /exports/www.ivdgl.org 2011-02-14 10:19:34.000000 HOMES.MCS.ANL.GOV /exports/www.gridforum.org 2011-02-14 10:18:00.000000 HOMES.MCS.ANL.GOV /disks/weblogs 2011-02-14 09:13:24.000000 HOMES.MCS.ANL.GOV /disks/nt-homes001/maxadam-admin2011-02-14 06:36:39.000000 HOMES.MCS.ANL.GOV /disks/root 2011-02-14 00:57:04.000000 If any of these file systems no longer exist, you can stop receiving emails by contacting systems(a)mcs.anl.gov. The tables and software are still under development and may change.

1 0

Fwd: Unauthorized access to MCS machines
by Ti Leggett 14 Oct '11

14 Oct '11

I want to say no, but what do the admins keep on NFS? Begin forwarded message: > From: Marc Snir <snir(a)mcs.anl.gov> > Subject: Re: Unauthorized access to MCS machines > Date: October 14, 2011 1:47:30 AM CDT > To: Ti Leggett <leggett(a)mcs.anl.gov> > Cc: Ian Foster <foster(a)anl.gov> > > did you check the content of those compromised machines to make sure they did not have any sensitive, unencrypted material? > > On Oct 14, 2011, at 12:47 AM, Ti Leggett wrote: > >> Here's an update on where we are and what we're doing. am-mac1 and am-mac2.mcs.anl.gov were both rooted this morning. These boxes were in the process of being decommissioned but had been neglected of updates for a while and had several outstanding exploitable holes. It looks like a poorly secured account at the CI was the bridge from the CI into these MCS machines. >> >> About 1.5 hours ago we tracked down that login.ci.uchicago.edu has also been rooted. This machine seems to have used the same rootkit as the am-mac machines, but the troubling aspect about it is that it was patched to all latest about 3 weeks ago. So it *seems* we're looking at a fairly new exploit. I'm taking this machine down later tonight to do more extensive forensics to discover how it was rooted. We're also trying to track down how the attacker got on the box to begin with. >> >> In any case, I will be sending out an announcement to the CI later explaining what has occurred. Once we have identified and patched the exploit we will be requiring all CI users to change their password. Any private keys that aren't passphrase protected will be blacklisted. At MCS we will be encouraging all users to change their password especially if they used the same password at the CI and any unprotected private keys will be blacklisted. >> >> We may require stricter measures if we find the intrusion to be deeper or more serious than we currently are aware. >> >> >> On Oct 13, 2011, at 2:39 PM, Ti Leggett wrote: >> >>> I notified Mike Papka of this this morning, but since Craig is on vacation and I'm not familiar with the procedures in this case, I overlooked informing you. My apologies. >>> >>> This morning at about 4:00am there were some failed attempts on some machines in the 4th floor lab to gain elevated privileges. One of the systems staff saw the attempts, thought it looked suspicious and started investigating. By about 9:00 or so we had determined that someone had gain unauthorized access to 2 machines in the 4th floor lab. It doesn't appear that the intruder gained full elevated privileges but we are still investigating. So far it seems only the one user account and 2 lab machines are compromised. The user's account has been disabled while we continue to perform forensics. We're still investigating the full extent of the intrusion and how access was initially gained. When we have more information, I will keep you updated. >> >

1 0

Today's user termination list
by stace＠mcs.anl.gov 14 Oct '11

14 Oct '11

The following accounts have badge numbers of terminated employees: =========================================================================== username, argonne_username, badge_number, status, account_type, deact date Full list of people who no longer appear on the active employee list =========================================================================== 40741 PSC Rahman, Talat 41809 EVS Veil, John A. 202284 CIS Campbell, Scott 83706 PSC Chae, Sejung 205655 ES Winkler, Kary Lauren 205656 ES Chambers, Bryan 205724 PSC Nascimento, Von B. 216727 CIS Hallquist, Roy Stuart 216728 PSC Murray, Ian P. 216737 CIS Strait, Craig 216739 CIS Vollans, Daniel Joseph 216740 ES Heffington, Brent 216746 CIS Smith, Cathy Lynn 216748 PSC Field, Talia C. 212884 EVS Chaudhari, Anita

1 0

Where's Hunter
by Hunter Matthews 13 Oct '11

13 Oct '11

Power is out at my house (maybe the block - not sure). I may be in late in the morning dealing with this. -- Hunter Matthews Unix Administrator Office: Bldg 221 Room B240 Argonne National Labs, MCS Key: F0F88438 / FFB5 34C0 B350 99A4 BB02 9779 A5DB 8B09 F0F8 8438 Never take candy from strangers. Especially on the internet.

1 0

Where's JohnR (10/18/11)
by John Roberts 13 Oct '11

13 Oct '11

I will be taking a vacation day on Tuesday, 10/18/11. -- John Roberts Systems Administrator Jr. Argonne National Laboratory MCS Division roberts(a)mcs.anl.gov

1 0

possible hack attack
by Dan Olson 13 Oct '11

13 Oct '11

Around 4:45am this morning we had several user accounts attempt to sudo on multiple mcs machines. We are still investigating the root cause. Please check for anything unusual on the machines you manage. ---- Daniel Murphy-Olson Systems Administrator Mathematics & Computer Science Division Argonne National Laboratory 630-252-0055

3 4

Where's max?
by Max trefonides 13 Oct '11

13 Oct '11

Waiting for furnace repair guy. He should be here in an hour. I expect to be at the lab this afternoon. Sorry. -- Max

1 0

Backup-Failure-Notice
by root＠lal.mcs.anl.gov 13 Oct '11

13 Oct '11

ATTENTION The following systems have passed their scheduled backup times. Listed are the date and time of the last backup completed. NODE FILESPACE DATE TIME AHA.MCS.ANL.GOV /failover/var 2011-10-05 05:55:16.000000 AHA.MCS.ANL.GOV /failover/tmp 2011-10-05 05:55:15.000000 AHA.MCS.ANL.GOV /failover 2011-10-05 05:55:14.000000 AHA.MCS.ANL.GOV /u99 2011-10-05 05:54:20.000000 AHA.MCS.ANL.GOV /u02 2011-10-05 05:54:19.000000 ANNO-3.NMPDR.ORG /disks/space0 2011-10-05 05:47:34.000000 HOMES.MCS.ANL.GOV /exports/software 2011-09-11 06:01:17.000000 HOMES.MCS.ANL.GOV /exports/ruppert-share 2011-09-11 05:29:13.000000 HOMES.MCS.ANL.GOV /exports/request 2011-09-11 05:29:12.000000 HOMES.MCS.ANL.GOV /exports/nt-homes001 2011-09-11 05:29:07.000000 HOMES.MCS.ANL.GOV /exports/nt-homes001/maxadam-admin2011-09-11 05:29:07.000000 HOMES.MCS.ANL.GOV /exports/mcs-homes67 2011-09-11 05:04:57.000000 HOMES.MCS.ANL.GOV /exports/mcs-homes65 2011-09-11 05:00:00.000000 HOMES.MCS.ANL.GOV /exports/mcs-homes64 2011-09-11 04:53:37.000000 HOMES.MCS.ANL.GOV /exports/mcs-homes63 2011-09-11 01:25:39.000000 HOMES.MCS.ANL.GOV /exports/mcs-homes62 2011-09-11 01:25:19.000000 HOMES.MCS.ANL.GOV /exports/mcs-homes56 2011-09-11 01:17:04.000000 HOMES.MCS.ANL.GOV /exports/mcs-homes55 2011-09-11 01:11:00.000000 HOMES.MCS.ANL.GOV /exports/mcs-homes46 2011-09-11 01:09:44.000000 HOMES.MCS.ANL.GOV /exports/mcs-homes30 2011-09-11 00:43:09.000000 HOMES.MCS.ANL.GOV /exports/mcs-homes28 2011-09-11 00:41:54.000000 HOMES.MCS.ANL.GOV /exports/mcs-homes29 2011-09-11 00:41:54.000000 HOMES.MCS.ANL.GOV /exports/mcs-homes27 2011-09-11 00:03:50.000000 HOMES.MCS.ANL.GOV /exports/mcs-homes26 2011-09-10 23:41:22.000000 HOMES.MCS.ANL.GOV /exports/mcs-homes25 2011-09-10 21:58:27.000000 HOMES.MCS.ANL.GOV /exports/mcs-homes23 2011-09-10 20:50:06.000000 HOMES.MCS.ANL.GOV /exports/mcs-homes19 2011-09-10 20:34:13.000000 HOMES.MCS.ANL.GOV /exports/mcs-homes18 2011-09-10 20:24:07.000000 HOMES.MCS.ANL.GOV /exports/mcs-homes17 2011-09-10 20:13:01.000000 HOMES.MCS.ANL.GOV /exports/mcs-homes16 2011-09-10 19:54:39.000000 HOMES.MCS.ANL.GOV /exports/mcs-homes15 2011-09-10 19:29:54.000000 HOMES.MCS.ANL.GOV /exports/mcs-homes66 2011-09-09 18:19:07.000000 HOMES.MCS.ANL.GOV /exports/CELSAdmin 2011-09-02 18:12:59.000000 BUNTINAS-MAC.MCS.ANL.GOV / 2011-08-05 16:30:44.000000 REPO.UC.TERAGRID.ORG /var/lib/cvs 2011-05-06 00:44:55.000000 PORTAL1.UC.TERAGRID.ORG /var 2011-05-05 23:49:26.000000 PORTAL1.UC.TERAGRID.ORG / 2011-05-05 23:49:25.000000 PORTAL1.UC.TERAGRID.ORG /home 2011-05-05 23:48:59.000000 PORTAL2.UC.TERAGRID.ORG /var 2011-05-05 20:47:42.000000 PORTAL2.UC.TERAGRID.ORG / 2011-05-05 20:47:41.000000 PORTAL2.UC.TERAGRID.ORG /home 2011-05-05 20:47:18.000000 HOMES.MCS.ANL.GOV /exports/media 2011-02-14 16:26:17.000000 HOMES.MCS.ANL.GOV /exports/linux-debian_sarge 2011-02-14 10:52:41.000000 HOMES.MCS.ANL.GOV /exports/www.accessgrid.org 2011-02-14 10:22:22.000000 HOMES.MCS.ANL.GOV /exports/www.bgl.mcs.anl.gov 2011-02-14 10:20:42.000000 HOMES.MCS.ANL.GOV /exports/www.ivdgl.org 2011-02-14 10:19:34.000000 HOMES.MCS.ANL.GOV /exports/www.gridforum.org 2011-02-14 10:18:00.000000 HOMES.MCS.ANL.GOV /disks/weblogs 2011-02-14 09:13:24.000000 HOMES.MCS.ANL.GOV /disks/nt-homes001/maxadam-admin2011-02-14 06:36:39.000000 HOMES.MCS.ANL.GOV /disks/root 2011-02-14 00:57:04.000000 If any of these file systems no longer exist, you can stop receiving emails by contacting systems(a)mcs.anl.gov. The tables and software are still under development and may change.

1 0

Where's Chris?
by Chris Walker 13 Oct '11

13 Oct '11

Feeling rather contagious today, so I'm working from home.. Normally I'd be more than happy to come in and share with everyone, but I'm not in a sharing mood.. Today I will be working on the mcs, lcrc, and kbase front-ends for userbase, and hopefully testing them all working together in the staging environment, with the ultimate goal of getting kbase ready to deploy, presuming that kbase hasn't decided to move to an all-windows environment by the end of the day...

1 0