Page history
17 November 2012
→Typical Reasons for not allowing Robots
+247
no edit summary
+1
no edit summary
+4
no edit summary
+2
no edit summary
-2
no edit summary
-1
7 November 2012
no edit summary
+487
no edit summary
+34
no edit summary
+506
no edit summary
+526
no edit summary
m-1
added details about crawl-delay
+352
→Throttling
+433
→Throttling
+520
no edit summary
+148
6 November 2012
no edit summary
+299
RobotsAreOurFriends moved to Robots Are Our Friends
m→Robots Are Our Friends
-28
→Robots Are Our Friends
+4
→Robots Are Our Friends
-4
New page: = Robots Are Our Friends = For a variety of reasons cultural heritage organizations can often have [http://www.robotstxt.org/ robots.txt] pages that restrict what web crawlers (aka robots...
+951
