When replying, type your text above this line. ---------------------------------------------- Notification of Incident Change If the pending code above shows “Waiting on Customer”, followup action is required on your part as requested in the Description text below. Incident: Re: excessive scanning of collab.mcs.anl.gov Incident Number: 62221 Priority: Medium Status: Pending Date: 09/17/2012 Time: 17:13:04 Creation Date: 07/27/2012 Creation Time: 14:06:08 Created By: Dana Stasiak View Incident in Browser: https://tickets.anl.gov/MRcgi/MRlogin.pl?DL=62221DA12 Detail: Entered on 09/17/2012 at 17:13:04 CDT (GMT-0500) by Dana Stasiak: Ken, have you seen any more evidence of this behavior? Dana Entered on 07/27/2012 at 14:28:07 CDT (GMT-0500) by Dmitriy Sklyar: I have switched both internal and external GSAs to scheduled crawl and specified minimum load for Normal 0 false false false EN-US X-NONE X-NONE /* Style Definitions */ table.MsoNormalTable {mso-style-name:"Table Normal"; mso-tstyle-rowband-size:0; mso-tstyle-colband-size:0; mso-style-noshow:yes; mso-style-priority:99; mso-style-parent:""; mso-padding-alt:0in 5.4pt 0in 5.4pt; mso-para-margin:0in; mso-para-margin-bottom:.0001pt; mso-pagination:widow-orphan; font-size:10.0pt; font-family:"Times New Roman","serif";} collab.mcs.anl.gov and Normal 0 false false false EN-US X-NONE X-NONE /* Style Definitions */ table.MsoNormalTable {mso-style-name:"Table Normal"; mso-tstyle-rowband-size:0; mso-tstyle-colband-size:0; mso-style-noshow:yes; mso-style-priority:99; mso-style-parent:""; mso-padding-alt:0in 5.4pt 0in 5.4pt; mso-para-margin:0in; mso-para-margin-bottom:.0001pt; mso-pagination:widow-orphan; font-size:10.0pt; font-family:"Times New Roman","serif";} www.alcf.anl.gov. Please remove the blocks and let me know if the situation improves. If the blocks remain in place, no results from these sites will show up in the internal/external site search. Entered on 07/27/2012 at 14:09:19 CDT (GMT-0500) by Malikah Reed: assigning to web services Entered on 07/27/2012 at 14:06:08 CDT (GMT-0500) by Dana Stasiak: Craig, Ken, Please accept my apologies for the difficulties you have been experiencing. I have asked Dmitry to turn down the host load schedule for these servers. I am also creating a ticket to track this to completion. We have been recrawling the the collection due to some changes we are making for inside Argonne. Dana On Jul 27, 2012, at 2:32 PM, "Craig Stacey" <[email protected]> wrote: I'd go ahead and block it until it gets figured out. -- Craig (from my mobile) Ken Raffenetti <[email protected]> wrote: This scanning is still out of control. In yesterday's logs I see about 300,000 requests from each of the lab's google bots. Is there something I should be doing to these sites to prevent this excessive scanning? root@blizzard:/var/log/apache2# grep 146.137.96.101 blizzard.access_log.2012Jul26 | wc -l 277670 root@blizzard:/var/log/apache2# grep 130.202.84.131 blizzard.access_log.2012Jul26 | wc -l 292253 There are only about 200 pages on this entire site. Can we get support from Google with regards to this problem? A similar issue caused me to block the ANL google bots from www.alcf.anl.gov because it was affecting site performance. Ken Raffenetti Systems Administration Associate MCS Division - Argonne National Laboratory On 07/10/2012 09:29 AM, Ken Raffenetti wrote:

Google Admins,

I'm seeing a pretty outrageous number of requests to http://collab.mcs.anl.gov. Is there any way the scanning frequency can be turned down on this site?

In July alone, I see 1.5 million requests. This is similar to what was happening with www.alcf.anl.gov, which I eventually just blocked from being crawled by the internal Google bot.

root@blizzard:/var/log/apache2# grep 146.137.96.101 *.access_log.2012Jul* | wc -l | uniq -c 1 1522255

Current Assignees: Web Services CC(s): (permanent) [email protected], [email protected], [email protected] Customer Information: Last Name: Raffenetti First Name: Kenneth Email Address: [email protected] Badge: 56518 Login ID: b56518 ANL Division: MCS Costcenter: 145 Building: 240 Room: 2135 Phone: (630) 252-3358 This is an automated message sent by the Numara FootPrints ticket tracking system, which is operated by CIS.