Difference between revisions of "Robots Are Our Friends"

From Code4Lib
Jump to: navigation, search
(New page: = Robots Are Our Friends = For a variety of reasons cultural heritage organizations can often have [http://www.robotstxt.org/ robots.txt] pages that restrict what web crawlers (aka robots...)
 
(Robots Are Our Friends)
Line 1: Line 1:
 
= Robots Are Our Friends =
 
= Robots Are Our Friends =
  
For a variety of reasons cultural heritage organizations can often have [http://www.robotstxt.org/ robots.txt] pages that restrict what web crawlers (aka robots) can see on a website. This is a bad thing because it means that the content that libraries, archives and museums are putting online becomes virtually invisible to search engines like Google, Bing, Yahoo, is less likely to be shared in social media sites like Facebook, Twitter, Pinterest and stands less of a chance of being incorporated into datasets such as Wikipedia. The Robots Are Our Friends campaign aims to help promote an understanding of the role that robots.txt plays in determining the footprint our cultural heritage collections have on the Web.
+
For a variety of reasons cultural heritage organizations often have [http://www.robotstxt.org/ robots.txt] pages that restrict what web crawlers (aka robots) can see on a website. This is a bad thing because it means that the content that libraries, archives and museums are putting online becomes virtually invisible to search engines like Google, Bing, Yahoo, is less likely to be shared in social media sites like Facebook, Twitter, Pinterest and stands less of a chance of being incorporated into datasets such as Wikipedia. The Robots Are Our Friends campaign aims to help promote an understanding of the role that robots.txt plays in determining the footprint our cultural heritage collections have on the Web.
  
 
== Background ==  
 
== Background ==  

Revision as of 22:39, 6 November 2012

Robots Are Our Friends

For a variety of reasons cultural heritage organizations often have robots.txt pages that restrict what web crawlers (aka robots) can see on a website. This is a bad thing because it means that the content that libraries, archives and museums are putting online becomes virtually invisible to search engines like Google, Bing, Yahoo, is less likely to be shared in social media sites like Facebook, Twitter, Pinterest and stands less of a chance of being incorporated into datasets such as Wikipedia. The Robots Are Our Friends campaign aims to help promote an understanding of the role that robots.txt plays in determining the footprint our cultural heritage collections have on the Web.

Background

Typical Reasons for not allowing Robots

Throttling

Sitemaps

HTML5 Microdata

Form Letter