Yahoo! Search Support for X-Robots-Tag Directive to Simplify Webmaster's Control and Weather Update
Date : 2007 12 05 Category : Yahoo Yahoo! Search BlogToday we're announcing support for tags that give webmasters even more flexibility over which pages and documents are crawled and indexed by Yahoo! Search. Specifically, we're extending our support of page level exclusion tags -- NOINDEX, NOARCHIVE, NOSNIPPET, NOFOLLOW -- to provide additional control for archiving and summarization of ANY file type. Previously, these page level tags could only be expressed within html pages through the META directive (for e.g. <META NAME="Slurp" CONTENT="NOARCHIVE">), but based on feedback from our webmasters, Yahoo! now enables these tags to be expressed through X-Robots-Tag directive in the http header, giving webmasters the flexibility to achieve exclusions on PDF, Word documents, PowerPoint, video, and other file types, including html files, and increasing their coverage through a simplified process. Additionally, webmasters no longer need access to html templates in order to express exclusions for html files. To take advantage of this feature, simply add the following page level tags to the X-Robots-Tag directive in the HTTP Header. Here are a few examples:
X-Robots-Tag: NOINDEX -- If you don't want to show the URL in the Yahoo! Search results.Note: We'll still need to crawl the page to see and apply the tag, so if you don't wish to have the page crawled, use robots disallow on robots.txt.
X-Robots-Tag: NOARCHIVE -- If you don't want to display cache link in the search results page.
X-Robots-Tag: NOSNIPPET -- If you don't want to display summary in the search results page.
X-Robots-Tag: NOFOLLOW -- If you don't want Yahoo! to crawl links in the page.
Along with this change, we'll be rolling out additional changes to our crawling, indexing and ranking algorithms over the next few days. We expect the update will be completed early next week, but you may see some changes in ranking as well as some shuffling of the pages in the index during this process.
We're at SES in Chicago and WebmasterWorld's PubCon in Las Vegas, participating in a few different panels this week. Please find us if you have any questions or suggestions or drop us your feedback here.
Sharad Verma
Yahoo! Search