May 2011 - Core-students - mailman3.cels.anl.gov

RADIUS
by Ti Leggett 12 May '11

12 May '11

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 After we get con squared away we need to change the RADIUS secret keys we use. -----BEGIN PGP SIGNATURE----- Version: GnuPG/MacGPG2 v2.0.14 (Darwin) iEYEARECAAYFAk3MoKcACgkQ4RgdOxQVi0CRCACeIeHr1pVNAnCXclEt+R9Mzx0+ psQAmgNhCuSzvqZ/zQq+D0tHVC++/Kc/ =LMaj -----END PGP SIGNATURE-----

1 0

Me
by Ti Leggett 12 May '11

12 May '11

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 I just got backup.ci.anl.gov fully updated and it's rebooting. Since my parents just got in, I'm heading offline for a bit to eat dinner. Call me on my cell if you need me for anything in the meantime: 630-854-9335. -----BEGIN PGP SIGNATURE----- Version: GnuPG/MacGPG2 v2.0.14 (Darwin) iEYEARECAAYFAk3MZOEACgkQ4RgdOxQVi0BvXQCggqjv3P2H/D6LYvooP+9LYLef 2b0An0Vl4QLDe5A+1HKkSwGOTxmqhB8g =ufl2 -----END PGP SIGNATURE-----

1 0

Break-in information
by Craig Stacey 12 May '11

12 May '11

What follows is a quick summary of what we've discovered and what we're doing. More details will follow. Findings: We believe backup.ci.anl.gov has been compromised since February 7, based on the ctime of a suspected bad sshd. We believe tp-login2.ci.uchicago.edu was compromised since March based on the ctime of a suspected bad sshd. con.ci.uchicago.edu shows no signs yet that it was modified except for some log scrubbing, but we know a CI sysadmin's password was sniffed and the infiltrator had root access. Based on the times of the first two entries, we believe the old con.ci.uchicago.edu was compromised (it was rebuilt in late April), and that passwords were sniffed, including an old MCS root password. It is possible the old con.ci.uchicago.edu was the initial point of entry for the attack. An MCS sysadmin, in administering the CI backup server, inadvertently exposed an old root password by typing it while logged in on the old, compromised con.ci.uchicago.edu while administering a machine that did not have sudo (this requiring typing a root password). We see activity of SSH scanning from backup.ci.anl.gov in the early morning of May 11. The attacker used the old MCS root password on the one externally exposed server that was using the old root password (it did not get the password change due to it incorrectly not automatically running our configuration management tool) and was able to login as it was also mistakenly configured to allow root logins. This happened between 6:38 and 6:48AM on May 11. He also installed a malicious sshd at this time before network blocks knocked him off. We haven't found evidence of other MCS hosts. In none of the above cases do we know what the malicious sshd does. There's no overt signs of phoning home or keylogging. Actions: We have turned off tp-login2.ci.uchicago.edu. We have pulled con.ci.uchicago.edu to do forensics and ensure it is not compromised. We have sequestered backup.ci.anl.gov to repair its sshd and any other packages out of compliance (a simple rebuild is not doable at this point), and are locking it down to only allow traffic from our secure management hosts in MCS (namby/pamby). We have turned off git.mcs.anl.gov VM and are building a replacement. We are instructing CI users to change their passwords before COB Tuesday. We will monitor CI machines for suspicious activity. CI servers will be configured to behave like MCS servers (only allow logins from secure bastion host requiring OTP, require OTP for sudo). More details to follow

1 0

Possible break-in
by Craig Stacey 12 May '11

12 May '11

No standard syncs today This is very alarming to me. From git.mcs.anl.gov's auth logs on syslog.mcs.anl.gov yesterday. May 11 06:35:22 git sshd[3098]: Accepted keyboard-interactive/pam for root from 130.83.204.10 port 11565 ssh2 May 11 06:38:48 git sshd[3400]: Accepted keyboard-interactive/pam for root from 130.83.204.10 port 11567 ssh2 May 11 06:48:07 git sshd[13649]: Failed hostbased for root from 128.135.125.172 port 55219 ssh2 May 11 06:48:11 git sshd[13649]: Failed hostbased for root from 128.135.125.172 port 55219 ssh2 May 11 06:48:11 git sshd[13649]: Accepted keyboard-interactive/pam for root from 128.135.125.172 port 55219 ssh2 MCS sysadmins, we need to gather and investigate this immediately. Meet in the fishbowl when you get this. CI Admins, why am I seeing keyboard interactive attempts from con.ci? Does con.ci log anywhere other than itself? Are there any logs we can investigate that we trust? At this point I'm led to believe that we have owned machines in MCS and CI. I immediately suspect git.mcs.anl.gov, but I also suspect backup.ci.anl.gov based on this: May 11 06:27:21 git sshd[2983]: Connection from 140.221.8.40 port 58489 May 11 06:27:21 git sshd[2983]: Did not receive identification string from 140.221.8.40 == Craig

1 0

Where's Max?
by Max trefonides 12 May '11

12 May '11

I have some family business to take care of this afternoon. Please put me down for 1/2 day vacation.

1 0

Wheres hunter
by Hunter Matthews 12 May '11

12 May '11

Please put me down for one more sick day - I'm gonna try to beat this thing. -- Hunter Matthews Unix Administrator Office: Bldg 221 Room B240 Argonne National Labs, MCS Key: F0F88438 / FFB5 34C0 B350 99A4 BB02 9779 A5DB 8B09 F0F8 8438 Never take candy from strangers. Especially on the internet.

1 0

where's Linda 5/12
by Linda Winkler 12 May '11

12 May '11

working at home this morning waiting for insurance adjustor. will be at the Lab after lunch. lw

1 0

Cron <stace@namby> /sandbox/stace/check-for-terminations.sh
by root＠mcs.anl.gov 12 May '11

12 May '11

Employees no longer listed as active ===================================== 212814 FMS Hill, Gerald 34013 NE Douglas, Barbara J. 38027 NE Oikle, Christine L. 38935 NE Hamilton, Sunaree 58105 NE Holiski, Allison M. 58445 NE Zordan, Diane N. 82980 PSC Rustan, Gustav E. 86685 CNM Noyan, Ismail C. Entries: 10561 New employees on the list ========================== 203864 PSC Kerlin, William M. 219330 PSC An, Sanghun 219331 PSC Ou, Fang 219340 PSC Hou, Haifeng 219341 PSC Zhang, Bing 219342 PSC Doutch, James J. 219355 PSC Plumb, Kemp W. 219373 PSC Huang, Ching-Mao 219374 PSC Elegheert, Jonathan Andre Georges 219375 PSC Sun, Cheng 34013 DEP Douglas, Barbara J. 38027 DEP Oikle, Christine L. 38935 DEP Hamilton, Sunaree 45889 PHY Sato, Toru 58105 DEP Holiski, Allison M. 58445 DEP Zordan, Diane N. 70507 DOE Soriano, Michael D. Entries: 10570

1 0

Fwd: Where's JohnR
by John Roberts 11 May '11

11 May '11

After a Dr visit this afternoon the verdict is a high flu. Ill be home tomorrow to avoid spreading it and online too. Ill see what Friday holds as I'd like to come in if possible. -------- Original Message -------- Subject: Where's JohnR From: John Roberts <roberts(a)mcs.anl.gov> To: [email protected],[email protected] CC: Isn't normal for me but I seem to be sick again. Instead of the stomach flu I'm feeling strep throat/real flu symptoms since all day yesterday. I'm going to my doctor as soon as They open today. I was planning on being in first but I don't want to risk being contagious. Ill send a message when I find out.

1 0

Today's empoloyee report
by Craig Stacey 11 May '11

11 May '11

Apologies for the mongo spam this morning. I made a change in the script yesterday to make it less spammy, but forgot to fix the first report. Here's today's report, in the less spammy version: Employees no longer listed as active ===================================== 208329 PSC Peng, Chunqing 208790 CIS Varela, Victoria 208813 PSC Noll, Richard K. 208818 PSC Summerton, Jean C. 210149 DIS Borla, Joanna B. 56344 PHY Daniel', Andrey V. 57394 CLS Bartels, Daniela 58977 CNM Wang, Yiliang 66997 PSC Cantrell, Kirk 67023 PSC Hou, Binyang 87131 PSC Merkel, Sebastien 97432 CIS Schellberg, Bruce Entries: 10560 New employees on the list ========================== 202162 PHY Kamano, Hiroyuki 210149 DIS Borla, Joanna 219056 PSC Mankidy, Bijith D. 219356 PSC Tom, Alan R. 219357 DIS Pardo, Julio 219358 PSC Gunderman, Peter R. 219359 PSC Santa Maria, Peter J. 219361 HR Spann, Alice 219362 PSC Theryo, Grayce C. 57249 PHY Cloet, Ian C. 67253 PSC Nimigean, Crina M. 69159 PSC Choi, Soo Hyung 86436 PSC Chattopadhyay, Debasish Entries: 10561

1 0