2008 Mar 27
Date : 2008 03 27 Google 
Official Google Webmaster Central Blog
Written by Sean Harding, Webmaster Tools TeamWe all know how friendly Googlebot is. And like all benevolent robots, he listens to us and respects our wishes about parts of our site that we don't want crawled. We can just give him a robots.txt file explaining what we want, and he'll happily comply. But what if read more »
Official Google Webmaster Central BlogWritten by Sean Harding, Webmaster Tools TeamWe all know how friendly Googlebot is. And like all benevolent robots, he listens to us and respects our wishes about parts of our site that we don't want crawled. We can just give him a robots.txt file explaining what we want, and he'll happily comply. But what if read more »