No, but I think we're ok now. Craig Stacey <[email protected]> wrote: Just landed. Had anyone pinged Corby? -- Craig (mobile) -----Original Message----- From: Ti Leggett [[email protected]] Received: Monday, 31 Dec 2012, 12:10AM To: Dan Olson [[email protected]] CC: William E. Allcock [[email protected]]; Matthew A. Kwiatkowski [[email protected]]; Tisha Stacey [[email protected]]; [email protected] Rackow [[email protected]]; [email protected] Admins [[email protected]] Subject: Re: crypto auth That IP is still hitting LCRC and LCF as I type this. On Dec 30, 2012, at 11:26 PM, Dan Olson <[email protected]> wrote:
I see this in the block list now, but I'm still seeing the auth traffic from that address.
Here is the entry: 60.166.48.158 2012-12-30 20:27:15 2013-01-06 23:07:00 mattk Request from ALCF 1 161 t1man Both
Maybe something is wrong with manual blocks?
---- Daniel Murphy-Olson Systems Administrator Mathematics & Computer Science Division Argonne National Laboratory 630-252-0055
----- Original Message -----
From: "Ti Leggett" <[email protected]> To: "[email protected] Admins" <[email protected]> Cc: "Tisha Stacey" <[email protected]>, "William E. Allcock" <[email protected]>, "[email protected] Rackow" <[email protected]> Sent: Sunday, December 30, 2012 11:06:52 PM Subject: Re: crypto auth So lots of places - LCRC, kBt, Magellan, ALCF, among others - are getting brute force attempts from 60.166.48.158 and the sheer number of attempts for invalid users is overwhelming the Crypto servers causing them to fall over. I've put in a request to cyber (and cyber911) as well as called Gene to put a block on that IP, but so far I haven't gotten a hold of anyone. My fear is that unless a block goes in soon-ish, the servers will fall over again in an hour or so. If anyone has any other ideas I'm open to them.
On Dec 30, 2012, at 10:33 PM, Ti Leggett <[email protected]> wrote:
Dan and I got auth 'working' but there are some SQL things afoot that Dan's looking in to right now.
On Dec 30, 2012, at 10:12 PM, Ti Leggett <[email protected]> wrote:
I just got a page about crypto auth. I can't get in to auth or auth2. I rebooted auth2 and got a recovery message but I still can't ssh in with CC though I can't rule out if my token is out of sync. If auth needs to be physically booted I don't think I have access to 221 to do so. Can someone take a look and give me a call at 630-854-9335? Thanks.
All, I apologize for the lateness of this. The root cause of the failure of the IDS system was identified as an aged piece of hardware we call "trigger switch". It is end of life and we have had a replacement on order, but could not get it in place before the break. The old trigger switch has hardware failures which we detect, but requires a physical person to reboot the system. The hardware had failed the day the IP was attacking. The trigger router provides the cyber office a persistent SSH connection into the border routers and Firewalls which allow real-time updates of the block tables. In reality, the types of brute force attacks happen all the time, but they are usually stopped quickly. With this piece of hardware down our automated system did not put in the block (they were all queued and put in when the system went back online). Corby was able to reboot the system the next morning and all has returned to normal. We sincerely apologize for the inconvenience and have an action plan in place to prevent this from happening that should be completed in the January timeframe. -- Matt Kwiatkowski M.S.I.S. Cyber Security Operations Manager Aviation Safety Officer Argonne Cyber Security Program Office Argonne National Laboratory http://www.anl.gov 9700 South Cass Avenue Bldg 240 Office 5310 Argonne, IL 60439 [email protected] Phone (630) 252-6465 Fax (630) 252-9689 HEALTH WARNING: Care should be taken when lifting this e-mail, since its mass, and thus its weight, is dependent on its velocity relative to the user. -----Original Message----- From: Ti Leggett [mailto:[email protected]] Sent: Monday, December 31, 2012 12:19 AM To: Stacey, Craig J. Cc: Olson, Daniel E.; Ti Leggett; Allcock, William E.; Kwiatkowski, Matthew A.; Tisha Stacey; Rackow, Eugene A.; [email protected] Subject: Re: crypto auth No, but I think we're ok now. Craig Stacey <[email protected]> wrote: Just landed. Had anyone pinged Corby? -- Craig (mobile) -----Original Message----- From: Ti Leggett [[email protected]] Received: Monday, 31 Dec 2012, 12:10AM To: Dan Olson [[email protected]] CC: William E. Allcock [[email protected]]; Matthew A. Kwiatkowski [[email protected]]; Tisha Stacey [[email protected]]; [email protected] Rackow [[email protected]]; [email protected] Admins [[email protected]] Subject: Re: crypto auth That IP is still hitting LCRC and LCF as I type this. On Dec 30, 2012, at 11:26 PM, Dan Olson <[email protected]> wrote:
I see this in the block list now, but I'm still seeing the auth traffic from that address.
Here is the entry: 60.166.48.158 2012-12-30 20:27:15 2013-01-06 23:07:00 mattk Request from ALCF 1 161 t1man Both
Maybe something is wrong with manual blocks?
---- Daniel Murphy-Olson Systems Administrator Mathematics & Computer Science Division Argonne National Laboratory 630-252-0055
----- Original Message -----
From: "Ti Leggett" <[email protected]> To: "[email protected] Admins" <[email protected]> Cc: "Tisha Stacey" <[email protected]>, "William E. Allcock" <[email protected]>, "[email protected] Rackow" <[email protected]> Sent: Sunday, December 30, 2012 11:06:52 PM Subject: Re: crypto auth So lots of places - LCRC, kBt, Magellan, ALCF, among others - are getting brute force attempts from 60.166.48.158 and the sheer number of attempts for invalid users is overwhelming the Crypto servers causing them to fall over. I've put in a request to cyber (and cyber911) as well as called Gene to put a block on that IP, but so far I haven't gotten a hold of anyone. My fear is that unless a block goes in soon-ish, the servers will fall over again in an hour or so. If anyone has any other ideas I'm open to them.
On Dec 30, 2012, at 10:33 PM, Ti Leggett <[email protected]> wrote:
Dan and I got auth 'working' but there are some SQL things afoot that Dan's looking in to right now.
On Dec 30, 2012, at 10:12 PM, Ti Leggett <[email protected]> wrote:
I just got a page about crypto auth. I can't get in to auth or auth2. I rebooted auth2 and got a recovery message but I still can't ssh in with CC though I can't rule out if my token is out of sync. If auth needs to be physically booted I don't think I have access to 221 to do so. Can someone take a look and give me a call at 630-854-9335? Thanks.
Thanks for the note. I totally understand these kinds of failures. My main concern was understanding why, and it seems to be well in hand. -- Craig On Jan 2, 2013, at 11:12 AM, "Kwiatkowski, Matthew A." <[email protected]> wrote:
All,
I apologize for the lateness of this. The root cause of the failure of the IDS system was identified as an aged piece of hardware we call "trigger switch". It is end of life and we have had a replacement on order, but could not get it in place before the break. The old trigger switch has hardware failures which we detect, but requires a physical person to reboot the system. The hardware had failed the day the IP was attacking. The trigger router provides the cyber office a persistent SSH connection into the border routers and Firewalls which allow real-time updates of the block tables. In reality, the types of brute force attacks happen all the time, but they are usually stopped quickly. With this piece of hardware down our automated system did not put in the block (they were all queued and put in when the system went back online). Corby was able to reboot the system the next morning and all has returned to normal.
We sincerely apologize for the inconvenience and have an action plan in place to prevent this from happening that should be completed in the January timeframe.
-- Matt Kwiatkowski M.S.I.S. Cyber Security Operations Manager Aviation Safety Officer Argonne Cyber Security Program Office
Argonne National Laboratory http://www.anl.gov 9700 South Cass Avenue Bldg 240 Office 5310 Argonne, IL 60439
[email protected] Phone (630) 252-6465 Fax (630) 252-9689
HEALTH WARNING: Care should be taken when lifting this e-mail, since its mass, and thus its weight, is dependent on its velocity relative to the user.
-----Original Message----- From: Ti Leggett [mailto:[email protected]] Sent: Monday, December 31, 2012 12:19 AM To: Stacey, Craig J. Cc: Olson, Daniel E.; Ti Leggett; Allcock, William E.; Kwiatkowski, Matthew A.; Tisha Stacey; Rackow, Eugene A.; [email protected] Subject: Re: crypto auth
No, but I think we're ok now.
Craig Stacey <[email protected]> wrote:
Just landed. Had anyone pinged Corby?
-- Craig (mobile)
-----Original Message----- From: Ti Leggett [[email protected]] Received: Monday, 31 Dec 2012, 12:10AM To: Dan Olson [[email protected]] CC: William E. Allcock [[email protected]]; Matthew A. Kwiatkowski [[email protected]]; Tisha Stacey [[email protected]]; [email protected] Rackow [[email protected]]; [email protected] Admins [[email protected]] Subject: Re: crypto auth
That IP is still hitting LCRC and LCF as I type this.
On Dec 30, 2012, at 11:26 PM, Dan Olson <[email protected]> wrote:
I see this in the block list now, but I'm still seeing the auth traffic from that address.
Here is the entry: 60.166.48.158 2012-12-30 20:27:15 2013-01-06 23:07:00 mattk Request from ALCF 1 161 t1man Both
Maybe something is wrong with manual blocks?
---- Daniel Murphy-Olson Systems Administrator Mathematics & Computer Science Division Argonne National Laboratory 630-252-0055
----- Original Message -----
From: "Ti Leggett" <[email protected]> To: "[email protected] Admins" <[email protected]> Cc: "Tisha Stacey" <[email protected]>, "William E. Allcock" <[email protected]>, "[email protected] Rackow" <[email protected]> Sent: Sunday, December 30, 2012 11:06:52 PM Subject: Re: crypto auth So lots of places - LCRC, kBt, Magellan, ALCF, among others - are getting brute force attempts from 60.166.48.158 and the sheer number of attempts for invalid users is overwhelming the Crypto servers causing them to fall over. I've put in a request to cyber (and cyber911) as well as called Gene to put a block on that IP, but so far I haven't gotten a hold of anyone. My fear is that unless a block goes in soon-ish, the servers will fall over again in an hour or so. If anyone has any other ideas I'm open to them.
On Dec 30, 2012, at 10:33 PM, Ti Leggett <[email protected]> wrote:
Dan and I got auth 'working' but there are some SQL things afoot that Dan's looking in to right now.
On Dec 30, 2012, at 10:12 PM, Ti Leggett <[email protected]> wrote:
I just got a page about crypto auth. I can't get in to auth or auth2. I rebooted auth2 and got a recovery message but I still can't ssh in with CC though I can't rule out if my token is out of sync. If auth needs to be physically booted I don't think I have access to 221 to do so. Can someone take a look and give me a call at 630-854-9335? Thanks.
participants (3)
-
Craig Stacey -
Kwiatkowski, Matthew A. -
Ti Leggett