Jump to: navigation, search

Robots Are Our Friends

352 bytes added, 15:30, 7 November 2012
added details about crawl-delay
== Throttling ==
=== crawl-delay === Several major search engines support the [ crawl delay directive, which you can put in your robots.txt file. This directive lets you tell web crawlers the minimum delay to wait between two successive requests.  <pre>User-agent: * Crawl-delay: 3</pre>  === Google===
# Log in to [ Google Webmaster Tools]
# Select your site
Note: This may take a day or two to go into effect and '''it only lasts for 90 days''', at which point it will revert back to Google selecting the crawl rate.
* === Bing===
# Log in to [ Bing Webmaster Tools]
# Select your site

Navigation menu