Open main menu

Changes

Robots Are Our Friends

352 bytes added, 15:30, 7 November 2012
added details about crawl-delay
== Throttling ==
=== crawl-delay === Several major search engines support the [https://en.wikipedia.org/wiki/Robots.txt#Crawl-delay_directive crawl delay directive, which you can put in your robots.txt file. This directive lets you tell web crawlers the minimum delay to wait between two successive requests.  <pre>User-agent: * Crawl-delay: 3</pre>  === Google===
# Log in to [https://www.google.com/webmasters/tools Google Webmaster Tools]
# Select your site
Note: This may take a day or two to go into effect and '''it only lasts for 90 days''', at which point it will revert back to Google selecting the crawl rate.
* === Bing===
# Log in to [http://bing.com/webmaster Bing Webmaster Tools]
# Select your site