User-agent: * Disallow: /sales/ Disallow: /images/
User-agent: googlebot Disallow: /sales
User-Agent: * Allow: /
<meta name="robots" content="noindex" />
<meta name="robots" content="noindex" /> <meta name="googlebot" content="index" />
<meta name="robots" content="nofollow" />
<a href="mypage.html" rel="nofollow" />
x-robots-tag
X-Robots-Tag: noindex
{{<meta name="robots" content="noarchive" />}}
{{<meta name="googlebot" content="nosnippet" />}}
{{<meta name="robots" content="noodp" />}}
{{<meta name="slurp" content="noydir">}}
nopreview
x-robots-tag: nopreview
{{User-agent: searchpreview Disallow: /}}
{{<meta name="robots" content="noimageindex,nomediaindex" />}}
{{<meta name="robots" content="unavailable_after: 21-Jul-2037 14:30:00 CET" />}}
{{X-Robots-Tag: unavailable_after: 7 Jul 2037 16:30:00 GMT}}
Pro
Con
crawl-delay
{{ User-agent: Slurp Crawl-delay: 0.5 User-agent: msnbot Crawl-delay: 4 }}{{}}
Crawl-delay
This information is treated as a removal request: it will take about a day after the removal date passes for the page to disappear from the search results. We currently only support unavailable_after for Google web search results.