When replying, type your text above this line. ---------------------------------------------- Notification of Incident Change Thank You for allowing the CIS team to serve your IT needs. Your ticket request 62213 has been closed. Details of your original request and the actions taken to resolve your request are below. If you feel that your request has not been completed to your satisfaction, please use the “reply” button on your email client to reply to this message and ticket 62213 will automatically be reopened. Otherwise, please do not reply to this message. Incident: Re: excessive scanning of collab.mcs.anl.gov Incident Number: 62213 Priority: Medium Status: Closed Date: 08/09/2012 Time: 11:58:43 Creation Date: 07/27/2012 Creation Time: 13:30:06 Created By: [email protected] View Incident in Browser: https://tickets.anl.gov/MRcgi/MRlogin.pl?DL=62213DA12 Detail: Entered on 08/09/2012 at 11:58:43 CDT (GMT-0500) by Matthew Kwiatkowski: Duh forgot to close ticket! Entered on 08/09/2012 at 11:58:10 CDT (GMT-0500) by Matthew Kwiatkowski: Closing ticket....but would like to hear either from this ticket or offline from Ken if the scanning is less agressive. Entered on 07/30/2012 at 11:06:05 CDT (GMT-0500) by Dmitriy Sklyar: The scanning has been switched to a less aggressive schedule last week on both internal and external GSAs. They are now performing scans only during the nights and weekends with less load on the servers scanned. Please monitor your systems and let me know if you see an improvement. Dmitriy. Entered on 07/27/2012 at 15:54:06 CDT (GMT-0500) by Matthew Kwiatkowski: We need to add Dana’s group to this ticket. Cyber has the same concern. The google appliances index internal sites WAY too often. Entered on 07/27/2012 at 13:38:25 CDT (GMT-0500) by Malikah Reed: assigning to cyber? Entered on 07/27/2012 at 13:30:06 CDT (GMT-0500) by [email protected]: This scanning is still out of control. In yesterday's logs I see about 300,000 requests from each of the lab's google bots. Is there something I should be doing to these sites to prevent this excessive scanning? root@blizzard:/var/log/apache2# grep 146.137.96.101 blizzard.access_log.2012Jul26 | wc -l 277670 root@blizzard:/var/log/apache2# grep 130.202.84.131 blizzard.access_log.2012Jul26 | wc -l 292253 There are only about 200 pages on this entire site. Can we get support from Google with regards to this problem? A similar issue caused me to block the ANL google bots from www.alcf.anl.gov because it was affecting site performance. Ken Raffenetti Systems Administration Associate MCS Division - Argonne National Laboratory On 07/10/2012 09:29 AM, Ken Raffenetti wrote:

Google Admins,

I'm seeing a pretty outrageous number of requests to http://collab.mcs.anl.gov. Is there any way the scanning frequency can be turned down on this site?

In July alone, I see 1.5 million requests. This is similar to what was happening with www.alcf.anl.gov, which I eventually just blocked from being crawled by the internal Google bot.

root@blizzard:/var/log/apache2# grep 146.137.96.101 *.access_log.2012Jul* | wc -l | uniq -c 1 1522255

Current Assignees: Cyber Security, Inside Argonne CC(s): (permanent) [email protected], [email protected] Customer Information: Last Name: Raffenetti First Name: Kenneth Email Address: [email protected] Badge: 56518 Login ID: b56518 ANL Division: MCS Costcenter: 145 Building: 240 Room: 2135 Phone: (630) 252-3358 Attachments: smime_20120727155405_611572.p7s This is an automated message sent by the Numara FootPrints ticket tracking system, which is operated by CIS.