What follows is a quick summary of what we've discovered and what we're doing. More details will follow. Findings: We believe backup.ci.anl.gov has been compromised since February 7, based on the ctime of a suspected bad sshd. We believe tp-login2.ci.uchicago.edu was compromised since March based on the ctime of a suspected bad sshd. con.ci.uchicago.edu shows no signs yet that it was modified except for some log scrubbing, but we know a CI sysadmin's password was sniffed and the infiltrator had root access. Based on the times of the first two entries, we believe the old con.ci.uchicago.edu was compromised (it was rebuilt in late April), and that passwords were sniffed, including an old MCS root password. It is possible the old con.ci.uchicago.edu was the initial point of entry for the attack. An MCS sysadmin, in administering the CI backup server, inadvertently exposed an old root password by typing it while logged in on the old, compromised con.ci.uchicago.edu while administering a machine that did not have sudo (this requiring typing a root password). We see activity of SSH scanning from backup.ci.anl.gov in the early morning of May 11. The attacker used the old MCS root password on the one externally exposed server that was using the old root password (it did not get the password change due to it incorrectly not automatically running our configuration management tool) and was able to login as it was also mistakenly configured to allow root logins. This happened between 6:38 and 6:48AM on May 11. He also installed a malicious sshd at this time before network blocks knocked him off. We haven't found evidence of other MCS hosts. In none of the above cases do we know what the malicious sshd does. There's no overt signs of phoning home or keylogging. Actions: We have turned off tp-login2.ci.uchicago.edu. We have pulled con.ci.uchicago.edu to do forensics and ensure it is not compromised. We have sequestered backup.ci.anl.gov to repair its sshd and any other packages out of compliance (a simple rebuild is not doable at this point), and are locking it down to only allow traffic from our secure management hosts in MCS (namby/pamby). We have turned off git.mcs.anl.gov VM and are building a replacement. We are instructing CI users to change their passwords before COB Tuesday. We will monitor CI machines for suspicious activity. CI servers will be configured to behave like MCS servers (only allow logins from secure bastion host requiring OTP, require OTP for sudo). More details to follow