You are being copied on an IT ticket request. A new comment has been added to an open ticket. Please details below.

If you have additional information that may help us in resolving this request, please reply to this message, and your comments will be added to the ticket.

If you would like to speak to a member of the Argonne Service Desk team, call 630-252-9999, option 2, and refer to ticket 62221.

Thank you for submitting your request to the Argonne Service Desk.

Entered on 10/03/2012 at 11:42:15 CDT (GMT-0500) by David Salbego:
Dmitriy,
He provided that information.
Look at his command line cut-and-pastes.
--dave

Open Footprints Ticket: Assignees [Duplicate message snipped]

Entered on 10/03/2012 at 11:39:12 CDT (GMT-0500) by Dmitriy Sklyar:
Can you please verify which GSA server these excessive scans are coming from, http://search.anl.gov/ or http://public-search.anl.gov ?

Entered on 10/03/2012 at 11:17:45 CDT (GMT-0500) by Denice DiGiacomo:

Entered on 09/24/2012 at 17:25:47 CDT (GMT-0500) by Dana Stasiak:
Thanks Ken.
 
We'll wait to hear back from you before closing this ticket.
 
Dana

Entered on 09/20/2012 at 09:37:12 CDT (GMT-0500) by raffenet@mcs.anl.gov:
I haven't gotten around to unblocking the bots. I will try to get to it
today and monitor for excessive scanning.

Ken Raffenetti
Systems Administration Associate
MCS Division - Argonne National Laboratory

On 09/17/2012 05:13 PM, Argonne Service Desk wrote:
> [Duplicate message snipped]

Entered on 09/17/2012 at 17:13:04 CDT (GMT-0500) by Dana Stasiak:
Ken,
 
have you seen any more evidence of this behavior?
 
Dana

Entered on 07/27/2012 at 14:28:07 CDT (GMT-0500) by Dmitriy Sklyar:
I have switched both internal and external GSAs to scheduled crawl and specified minimum load for collab.mcs.anl.gov and www.alcf.anl.gov.
 
Please remove the blocks and let me know if the situation improves. If the blocks remain in place, no results from these sites will show up in the internal/external site search.

Entered on 07/27/2012 at 14:09:19 CDT (GMT-0500) by Malikah Reed:
assigning to web services

Entered on 07/27/2012 at 14:06:08 CDT (GMT-0500) by Dana Stasiak:
Craig, Ken,
Please accept my apologies for the difficulties you have been experiencing.
I have asked Dmitry to turn down the host load schedule for these servers. I am also creating a ticket to track this to completion.
We have been recrawling the the collection due to some changes we are making for inside Argonne.
Dana

On Jul 27, 2012, at 2:32 PM, "Craig Stacey" <stace@mcs.anl.gov> wrote:

I'd go ahead and block it until it gets figured out.
-- Craig (from my mobile)

Ken Raffenetti <raffenet@mcs.anl.gov> wrote: This scanning is still out of control. In yesterday's logs I see about
300,000 requests from each of the lab's google bots. Is there something
I should be doing to these sites to prevent this excessive scanning?

root@blizzard:/var/log/apache2# grep 146.137.96.101
blizzard.access_log.2012Jul26 | wc -l
277670
root@blizzard:/var/log/apache2# grep 130.202.84.131
blizzard.access_log.2012Jul26 | wc -l
292253

There are only about 200 pages on this entire site. Can we get support
from Google with regards to this problem? A similar issue caused me to
block the ANL google bots from www.alcf.anl.gov because it was affecting
site performance.

Ken Raffenetti
Systems Administration Associate
MCS Division - Argonne National Laboratory

On 07/10/2012 09:29 AM, Ken Raffenetti wrote:
> Google Admins,
>
> I'm seeing a pretty outrageous number of requests to
> http://collab.mcs.anl.gov. Is there any way the scanning frequency can
> be turned down on this site?
>
> In July alone, I see 1.5 million requests. This is similar to what was
> happening with www.alcf.anl.gov, which I eventually just blocked from
> being crawled by the internal Google bot.
>
> root@blizzard:/var/log/apache2# grep 146.137.96.101
> *.access_log.2012Jul* | wc -l | uniq -c
> 1 1522255
>

View Details Online: Click here to view Incident in Browser