Fwd: [operations] flood of auth requests from trebuchet and catapult.
-- Craig Begin forwarded message:
From: Matt Kemp <[email protected]> Date: August 25, 2012 10:56:01 AM CDT To: [email protected] Cc: ALCF Operations <[email protected]> Subject: Re: [operations] flood of auth requests from trebuchet and catapult.
Ah crap.
This is my bad. Friday evening, Ryan and I were configuring our new monitoring service for ALCF2 and some of the ssh-based attempts at monitoring were using a non-existent user 'nagios', which I temporarily changed to my username to see if they'd work. I thought they would have been using host-based auth to check and shouldn't have hit radius, but clearly there was a miscommunication. I've disabled the monitoring service temporarily and will fix it on Monday.
Thanks for the heads-up. Apologies.
Matt
Craig Stacey wrote:
-- Craig
----- Original Message ----- Also, while they are looking - there are requests coming from a non existent user, ping_user. I had provided our test account infomation, nagios-radius-test-user with a static password. Maybe they have a bad config on the server?
I can confirm that I can log into intrepid, so it doesn't appear to be causing a problem, just a quirk in the config.
---- Daniel Murphy-Olson Systems Administrator Mathematics& Computer Science Division Argonne National Laboratory 630-252-0055
----- Original Message ----- From: "Dan Olson"<[email protected]> To: [email protected] Cc: "Linda Winkler"<[email protected]>, [email protected] Sent: Saturday, August 25, 2012 10:13:15 AM Subject: Re: Auth is hosed?
There is a flood of requests from mkemp from 140.221.69.2 and 140.221.69.3. I'm still seeing them trickle in.
Can we get someone from alcf to trace the requests back to the source and put a stop to it?
---- Daniel Murphy-Olson Systems Administrator Mathematics& Computer Science Division Argonne National Laboratory 630-252-0055
----- Original Message ----- From: "Craig Stacey"<[email protected]> To: "Linda Winkler"<[email protected]> Cc: [email protected], "Dan Olson"<[email protected]> Sent: Saturday, August 25, 2012 10:01:32 AM Subject: Re: Auth is hosed?
Okay, now I'm in.
-- Craig
----- Original Message ----- From: "Craig Stacey"<[email protected]> To: "Linda Winkler"<[email protected]> Cc: [email protected], "Dan Olson"<[email protected]> Sent: Saturday, August 25, 2012 10:00:53 AM Subject: Re: Auth is hosed?
Unsure. It's not letting me in.
-- Craig
----- Original Message ----- From: "Linda Winkler"<[email protected]> To: "Dan Olson"<[email protected]> Cc: [email protected], [email protected] Sent: Saturday, August 25, 2012 9:59:11 AM Subject: Re: Auth is hosed?
I confirm we can use CC to access anchor device. What changed?
lw
On Aug 25, 2012, at 9:57 AM, Dan Olson wrote:
hmm.. my cryptocard just let me into namby.
---- Daniel Murphy-Olson Systems Administrator Mathematics& Computer Science Division Argonne National Laboratory 630-252-0055
----- Original Message ----- From: "Dan Olson"<[email protected]> To: [email protected] Cc: [email protected] Sent: Saturday, August 25, 2012 9:56:40 AM Subject: Re: Auth is hosed?
I'll just head in.
---- Daniel Murphy-Olson Systems Administrator Mathematics& Computer Science Division Argonne National Laboratory 630-252-0055
----- Original Message ----- From: "Craig Stacey"<[email protected]> To: "Dan Olson"<[email protected]> Cc: [email protected], "Corby B. Schmitz"<[email protected]> Sent: Saturday, August 25, 2012 9:47:47 AM Subject: Re: Auth is hosed?
I can be onsite in 20 minutes if it would help to have a physical presence.
-- Craig
----- Original Message ----- From: "Dan Olson"<[email protected]> To: [email protected] Cc: [email protected], "Corby B. Schmitz"<[email protected]> Sent: Saturday, August 25, 2012 9:46:50 AM Subject: Re: Auth is hosed?
I left him a voicemail.
---- Daniel Murphy-Olson Systems Administrator Mathematics& Computer Science Division Argonne National Laboratory 630-252-0055
----- Original Message ----- From: "Craig Stacey"<[email protected]> To: "Dan Olson"<[email protected]> Cc: [email protected], "Corby B. Schmitz"<[email protected]> Sent: Saturday, August 25, 2012 9:45:31 AM Subject: Re: Auth is hosed?
I don't. Do you have Ken's number?
-- Craig
----- Original Message ----- From: "Dan Olson"<[email protected]> To: "Craig Stacey"<[email protected]> Cc: [email protected], "Corby B. Schmitz"<[email protected]> Sent: Saturday, August 25, 2012 9:44:38 AM Subject: Re: Auth is hosed?
Does anyone have the current root password for the ipmi consoles? I believe it was last changed when we configured them for radius auth.
---- Daniel Murphy-Olson Systems Administrator Mathematics& Computer Science Division Argonne National Laboratory 630-252-0055
----- Original Message ----- From: "Craig Stacey"<[email protected]> To: "Corby B. Schmitz"<[email protected]> Cc: [email protected] Sent: Saturday, August 25, 2012 9:20:32 AM Subject: Re: Auth is hosed?
Dan's been invoked.
-- Craig
----- Original Message ----- From: "Craig Stacey"<[email protected]> To: "Corby B. Schmitz"<[email protected]> Cc: [email protected] Sent: Saturday, August 25, 2012 9:16:14 AM Subject: Re: Auth is hosed?
I noticed this, too. Seeing what I can do, otherwise I'm ringing Dan.
-- Craig
----- Original Message ----- From: "Corby B. Schmitz"<[email protected]> To: [email protected] Sent: Saturday, August 25, 2012 9:13:17 AM Subject: Auth is hosed?
My MCS CC is not working and I saw alerts from Auth last night. Can someone take a look ASAP?
-- corby
_______________________________________________ operations mailing list [email protected] https://lists.alcf.anl.gov/cgi-bin/mailman/listinfo/operations
-- Matt Kemp HPC Systems Administration Associate Argonne National Laboratory Office Number: (630) 252-0024
I put into place a really crude logfile watcher that will tell us if someone attempts to auth more than 100 times in an hour, since this has become a real problem. Right now it only runs at 59 past each hour, and only sends the offending username. Alerts go to the crypto-admins list. ---- Daniel Murphy-Olson Systems Administrator Mathematics & Computer Science Division Argonne National Laboratory 630-252-0055 ----- Original Message ----- From: "Craig Stacey" <[email protected]> To: "Core Admins" <[email protected]> Sent: Saturday, August 25, 2012 10:58:26 AM Subject: Fwd: [operations] flood of auth requests from trebuchet and catapult. -- Craig Begin forwarded message: From: Matt Kemp < [email protected] > Date: August 25, 2012 10:56:01 AM CDT To: [email protected] Cc: ALCF Operations < [email protected] > Subject: Re: [operations] flood of auth requests from trebuchet and catapult. Ah crap. This is my bad. Friday evening, Ryan and I were configuring our new monitoring service for ALCF2 and some of the ssh-based attempts at monitoring were using a non-existent user 'nagios', which I temporarily changed to my username to see if they'd work. I thought they would have been using host-based auth to check and shouldn't have hit radius, but clearly there was a miscommunication. I've disabled the monitoring service temporarily and will fix it on Monday. Thanks for the heads-up. Apologies. Matt Craig Stacey wrote: -- Craig ----- Original Message ----- Also, while they are looking - there are requests coming from a non existent user, ping_user. I had provided our test account infomation, nagios-radius-test-user with a static password. Maybe they have a bad config on the server? I can confirm that I can log into intrepid, so it doesn't appear to be causing a problem, just a quirk in the config. ---- Daniel Murphy-Olson Systems Administrator Mathematics& Computer Science Division Argonne National Laboratory 630-252-0055 ----- Original Message ----- From: "Dan Olson"< [email protected] > To: [email protected] Cc: "Linda Winkler"< [email protected] >, [email protected] Sent: Saturday, August 25, 2012 10:13:15 AM Subject: Re: Auth is hosed? There is a flood of requests from mkemp from 140.221.69.2 and 140.221.69.3. I'm still seeing them trickle in. Can we get someone from alcf to trace the requests back to the source and put a stop to it? ---- Daniel Murphy-Olson Systems Administrator Mathematics& Computer Science Division Argonne National Laboratory 630-252-0055 ----- Original Message ----- From: "Craig Stacey"< [email protected] > To: "Linda Winkler"< [email protected] > Cc: [email protected] , "Dan Olson"< [email protected] > Sent: Saturday, August 25, 2012 10:01:32 AM Subject: Re: Auth is hosed? Okay, now I'm in. -- Craig ----- Original Message ----- From: "Craig Stacey"< [email protected] > To: "Linda Winkler"< [email protected] > Cc: [email protected] , "Dan Olson"< [email protected] > Sent: Saturday, August 25, 2012 10:00:53 AM Subject: Re: Auth is hosed? Unsure. It's not letting me in. -- Craig ----- Original Message ----- From: "Linda Winkler"< [email protected] > To: "Dan Olson"< [email protected] > Cc: [email protected] , [email protected] Sent: Saturday, August 25, 2012 9:59:11 AM Subject: Re: Auth is hosed? I confirm we can use CC to access anchor device. What changed? lw On Aug 25, 2012, at 9:57 AM, Dan Olson wrote: hmm.. my cryptocard just let me into namby. ---- Daniel Murphy-Olson Systems Administrator Mathematics& Computer Science Division Argonne National Laboratory 630-252-0055 ----- Original Message ----- From: "Dan Olson"< [email protected] > To: [email protected] Cc: [email protected] Sent: Saturday, August 25, 2012 9:56:40 AM Subject: Re: Auth is hosed? I'll just head in. ---- Daniel Murphy-Olson Systems Administrator Mathematics& Computer Science Division Argonne National Laboratory 630-252-0055 ----- Original Message ----- From: "Craig Stacey"< [email protected] > To: "Dan Olson"< [email protected] > Cc: [email protected] , "Corby B. Schmitz"< [email protected] > Sent: Saturday, August 25, 2012 9:47:47 AM Subject: Re: Auth is hosed? I can be onsite in 20 minutes if it would help to have a physical presence. -- Craig ----- Original Message ----- From: "Dan Olson"< [email protected] > To: [email protected] Cc: [email protected] , "Corby B. Schmitz"< [email protected] > Sent: Saturday, August 25, 2012 9:46:50 AM Subject: Re: Auth is hosed? I left him a voicemail. ---- Daniel Murphy-Olson Systems Administrator Mathematics& Computer Science Division Argonne National Laboratory 630-252-0055 ----- Original Message ----- From: "Craig Stacey"< [email protected] > To: "Dan Olson"< [email protected] > Cc: [email protected] , "Corby B. Schmitz"< [email protected] > Sent: Saturday, August 25, 2012 9:45:31 AM Subject: Re: Auth is hosed? I don't. Do you have Ken's number? -- Craig ----- Original Message ----- From: "Dan Olson"< [email protected] > To: "Craig Stacey"< [email protected] > Cc: [email protected] , "Corby B. Schmitz"< [email protected] > Sent: Saturday, August 25, 2012 9:44:38 AM Subject: Re: Auth is hosed? Does anyone have the current root password for the ipmi consoles? I believe it was last changed when we configured them for radius auth. ---- Daniel Murphy-Olson Systems Administrator Mathematics& Computer Science Division Argonne National Laboratory 630-252-0055 ----- Original Message ----- From: "Craig Stacey"< [email protected] > To: "Corby B. Schmitz"< [email protected] > Cc: [email protected] Sent: Saturday, August 25, 2012 9:20:32 AM Subject: Re: Auth is hosed? Dan's been invoked. -- Craig ----- Original Message ----- From: "Craig Stacey"< [email protected] > To: "Corby B. Schmitz"< [email protected] > Cc: [email protected] Sent: Saturday, August 25, 2012 9:16:14 AM Subject: Re: Auth is hosed? I noticed this, too. Seeing what I can do, otherwise I'm ringing Dan. -- Craig ----- Original Message ----- From: "Corby B. Schmitz"< [email protected] > To: [email protected] Sent: Saturday, August 25, 2012 9:13:17 AM Subject: Auth is hosed? My MCS CC is not working and I saw alerts from Auth last night. Can someone take a look ASAP? -- corby _______________________________________________ operations mailing list [email protected] https://lists.alcf.anl.gov/cgi-bin/mailman/listinfo/operations -- Matt Kemp HPC Systems Administration Associate Argonne National Laboratory Office Number: (630) 252-0024
participants (2)
-
Craig Stacey -
Dan Olson