Robots Are Our Friends - Revision history

Edsu: /* Typical Reasons for not allowing Robots */

2012-11-17T12:37:30Z

‎Typical Reasons for not allowing Robots

Edsu at 12:30, 17 November 2012

2012-11-17T12:30:43Z

Edsu at 12:29, 17 November 2012

2012-11-17T12:29:15Z

Edsu at 12:26, 17 November 2012

2012-11-17T12:26:55Z

Edsu at 12:26, 17 November 2012

2012-11-17T12:26:22Z

Edsu at 12:25, 17 November 2012

2012-11-17T12:25:48Z

Bibliotechy at 18:15, 7 November 2012

2012-11-07T18:15:55Z

MrDys at 16:01, 7 November 2012

2012-11-07T16:01:54Z

Bibliotechy at 15:57, 7 November 2012

2012-11-07T15:57:22Z

Bibliotechy at 15:36, 7 November 2012

2012-11-07T15:36:12Z

← Older revision		Revision as of 12:37, 17 November 2012
Line 10:		Line 10:

	* While indexing a dynamic site, robots can put an extra strain on the server, causing a slow response, or in some cases, pegging the CPU at 100%.		* While indexing a dynamic site, robots can put an extra strain on the server, causing a slow response, or in some cases, pegging the CPU at 100%.
		+	* Some content is intentionally shielded from search engines to help shape how a websites resources are presented in search results. For example, if an organization has put a lot of PDFs online and doesn't want those to turn up in search results.

	== Throttling ==		== Throttling ==

← Older revision		Revision as of 12:30, 17 November 2012
Line 1:		Line 1:
−	[[Image:Friendly-Robut.png\|~~left~~]]	+	[[Image:Friendly-Robut.png\|right]]

	For a variety of reasons cultural heritage organizations often have [http://www.robotstxt.org/ robots.txt] documents that restrict what web crawlers (aka robots) can see on a website. This is a bad thing because it means that the content that libraries, archives and museums are putting online becomes virtually invisible to search engines like Google, Bing, Yahoo, is less likely to be shared in social media sites like Facebook, Twitter, Flickr, Pinterest and stands less of a chance of being used in educational sites like Wikipedia. The Robots Are Our Friends campaign aims to help promote an understanding of the role that robots.txt plays in determining the footprint our cultural heritage collections have on the Web.		For a variety of reasons cultural heritage organizations often have [http://www.robotstxt.org/ robots.txt] documents that restrict what web crawlers (aka robots) can see on a website. This is a bad thing because it means that the content that libraries, archives and museums are putting online becomes virtually invisible to search engines like Google, Bing, Yahoo, is less likely to be shared in social media sites like Facebook, Twitter, Flickr, Pinterest and stands less of a chance of being used in educational sites like Wikipedia. The Robots Are Our Friends campaign aims to help promote an understanding of the role that robots.txt plays in determining the footprint our cultural heritage collections have on the Web.

@@ Line 1: / Line 1: @@
 [[Image:Friendly-Robut.png|left]]
-For a variety of reasons cultural heritage organizations often have [http://www.robotstxt.org/ robots.txt] documents that restrict what web crawlers (aka robots) can see on a website. This is a bad thing because it means that the content that libraries, archives and museums are putting online becomes virtually invisible to search engines like Google, Bing, Yahoo, is less likely to be shared in social media sites like Facebook, Twitter, Pinterest and stands less of a chance of being incorporated into datasets such as Wikipedia. The Robots Are Our Friends campaign aims to help promote an understanding of the role that robots.txt plays in determining the footprint our cultural heritage collections have on the Web.
+For a variety of reasons cultural heritage organizations often have [http://www.robotstxt.org/ robots.txt] documents that restrict what web crawlers (aka robots) can see on a website. This is a bad thing because it means that the content that libraries, archives and museums are putting online becomes virtually invisible to search engines like Google, Bing, Yahoo, is less likely to be shared in social media sites like Facebook, Twitter, Flickr, Pinterest and stands less of a chance of being used in educational sites like Wikipedia. The Robots Are Our Friends campaign aims to help promote an understanding of the role that robots.txt plays in determining the footprint our cultural heritage collections have on the Web.
 == Background ==

← Older revision		Revision as of 12:26, 17 November 2012
Line 1:		Line 1:
−	[[Image:Friendly-Robut.png\|left]]For a variety of reasons cultural heritage organizations often have [http://www.robotstxt.org/ robots.txt] documents that restrict what web crawlers (aka robots) can see on a website. This is a bad thing because it means that the content that libraries, archives and museums are putting online becomes virtually invisible to search engines like Google, Bing, Yahoo, is less likely to be shared in social media sites like Facebook, Twitter, Pinterest and stands less of a chance of being incorporated into datasets such as Wikipedia. The Robots Are Our Friends campaign aims to help promote an understanding of the role that robots.txt plays in determining the footprint our cultural heritage collections have on the Web.	+	[[Image:Friendly-Robut.png\|left]]
		+
		+	For a variety of reasons cultural heritage organizations often have [http://www.robotstxt.org/ robots.txt] documents that restrict what web crawlers (aka robots) can see on a website. This is a bad thing because it means that the content that libraries, archives and museums are putting online becomes virtually invisible to search engines like Google, Bing, Yahoo, is less likely to be shared in social media sites like Facebook, Twitter, Pinterest and stands less of a chance of being incorporated into datasets such as Wikipedia. The Robots Are Our Friends campaign aims to help promote an understanding of the role that robots.txt plays in determining the footprint our cultural heritage collections have on the Web.

	== Background ==		== Background ==

@@ Line 50: / Line 50: @@
 *[http://vufind.org/wiki/search_engine_optimization#sitemaps VUfind]
 === Telling Search Engine about your sitemap ===
 Sitemaps can be added into your '''robots.txt''', or submitted directly to Google Webmaster Tools or Bing.

← Older revision		Revision as of 12:25, 17 November 2012
Line 1:		Line 1:
−	[[Image:Friendly-Robut.png\|~~right~~]]For a variety of reasons cultural heritage organizations often have [http://www.robotstxt.org/ robots.txt] documents that restrict what web crawlers (aka robots) can see on a website. This is a bad thing because it means that the content that libraries, archives and museums are putting online becomes virtually invisible to search engines like Google, Bing, Yahoo, is less likely to be shared in social media sites like Facebook, Twitter, Pinterest and stands less of a chance of being incorporated into datasets such as Wikipedia. The Robots Are Our Friends campaign aims to help promote an understanding of the role that robots.txt plays in determining the footprint our cultural heritage collections have on the Web.	+	[[Image:Friendly-Robut.png\|left]]For a variety of reasons cultural heritage organizations often have [http://www.robotstxt.org/ robots.txt] documents that restrict what web crawlers (aka robots) can see on a website. This is a bad thing because it means that the content that libraries, archives and museums are putting online becomes virtually invisible to search engines like Google, Bing, Yahoo, is less likely to be shared in social media sites like Facebook, Twitter, Pinterest and stands less of a chance of being incorporated into datasets such as Wikipedia. The Robots Are Our Friends campaign aims to help promote an understanding of the role that robots.txt plays in determining the footprint our cultural heritage collections have on the Web.

	== Background ==		== Background ==

@@ Line 41: / Line 41: @@
 "Sitemaps are an easy way for webmasters to inform search engines about pages on their sites that are available for crawling. In its simplest form, a Sitemap is an XML file that lists URLs for a site along with additional metadata about each URL (when it was last updated, how often it usually changes, and how important it is, relative to other URLs in the site) so that search engines can more intelligently crawl the site." -- www.sitemap.org
-The full specification is available at [http://www.sitemaps.org/ sitemap.org]
+The full specification is available at [http://www.sitemaps.org/ sitemaps.org]
 Sitemaps can be added into your '''robots.txt''', or submitted directly to Google Webmaster Tools or Bing.
-=== robots.txt ===
+==== robots.txt ====
-add the line:
+add a line like this to your robots.txt file:
 <pre>
@@ Line 52: / Line 62: @@
 </pre>
-=== Google ===
+==== Google ====
 # Log in to [https://www.google.com/webmasters/tools Google Webmaster Tools]
 # Select your site

← Older revision		Revision as of 16:01, 7 November 2012
Line 1:		Line 1:
−	For a variety of reasons cultural heritage organizations often have [http://www.robotstxt.org/ robots.txt] documents that restrict what web crawlers (aka robots) can see on a website. This is a bad thing because it means that the content that libraries, archives and museums are putting online becomes virtually invisible to search engines like Google, Bing, Yahoo, is less likely to be shared in social media sites like Facebook, Twitter, Pinterest and stands less of a chance of being incorporated into datasets such as Wikipedia. The Robots Are Our Friends campaign aims to help promote an understanding of the role that robots.txt plays in determining the footprint our cultural heritage collections have on the Web.	+	[[Image:Friendly-Robut.png\|right]]For a variety of reasons cultural heritage organizations often have [http://www.robotstxt.org/ robots.txt] documents that restrict what web crawlers (aka robots) can see on a website. This is a bad thing because it means that the content that libraries, archives and museums are putting online becomes virtually invisible to search engines like Google, Bing, Yahoo, is less likely to be shared in social media sites like Facebook, Twitter, Pinterest and stands less of a chance of being incorporated into datasets such as Wikipedia. The Robots Are Our Friends campaign aims to help promote an understanding of the role that robots.txt plays in determining the footprint our cultural heritage collections have on the Web.

	== Background ==		== Background ==

← Older revision		Revision as of 15:57, 7 November 2012
Line 42:		Line 42:

	The full specification is available at [http://www.sitemaps.org/ sitemap.org]		The full specification is available at [http://www.sitemaps.org/ sitemap.org]
		+
		+	Sitemaps can be added into your '''robots.txt''', or submitted directly to Google Webmaster Tools or Bing.
		+
		+	=== robots.txt ===
		+	add the line:
		+
		+	<pre>
		+	Sitemap: http://yourdomain.tld/path/to/sitemap.xml
		+	</pre>
		+
		+	=== Google ===
		+	# Log in to [https://www.google.com/webmasters/tools Google Webmaster Tools]
		+	# Select your site
		+	# In the left-hand side, select '''Optimization''' and then '''Sitemaps'''
		+	# Click '''Add/Test Sitemap'''
		+	# Enter the path to your sitemap on your server and click '''Submit Sitemap'''
		+
		+

← Older revision		Revision as of 15:36, 7 November 2012
Line 39:		Line 39:

	== Sitemaps ==		== Sitemaps ==
		+	"Sitemaps are an easy way for webmasters to inform search engines about pages on their sites that are available for crawling. In its simplest form, a Sitemap is an XML file that lists URLs for a site along with additional metadata about each URL (when it was last updated, how often it usually changes, and how important it is, relative to other URLs in the site) so that search engines can more intelligently crawl the site." -- www.sitemap.org
		+
		+	The full specification is available at [http://www.sitemaps.org/ sitemap.org]
		+

	== HTML5 Microdata ==		== HTML5 Microdata ==