X-Robots-Tag
Controlling How Search Engines Index Any File
Summary
The X-Robots-Tag is an HTTP response header that gives search engines indexing instructions, such as noindex or nofollow, for a URL. It works like the robots meta tag but is sent by the server, so it also covers non-HTML files. Teams use it to keep PDFs, images and other assets out of search results.
The X-Robots-Tag is an HTTP response header that tells search engine crawlers how to index and display a URL. It supports the same directives as the robots meta tag, such as noindex and nofollow, but the server sends it with the response instead of it being placed in a page's HTML. That makes it the standard method for controlling non-HTML files such as PDFs, images, videos and spreadsheets, which cannot contain a meta tag.
Common X-Robots-Tag directives include:
- noindex: Keeps the URL out of search results.
- nofollow: Tells crawlers not to follow links found in the resource.
- nosnippet: Prevents a text snippet or video preview from appearing in results.
- noimageindex: Prevents images on the page from being indexed.
- unavailable_after: Asks search engines to stop showing the URL after a specified date.
- max-snippet and max-image-preview: Limit the length of text snippets and the size of image previews.
Several directives can be combined in one header, for example "noindex, nofollow", and a rule can target a specific crawler by naming its user agent.
Why the X-Robots-Tag Matters
- Control over non-HTML files: It is the established way to keep PDFs, documents and media out of search results.
- Protecting gated content: Downloadable lead magnets such as whitepapers can be kept from ranking on their own, so visitors arrive through the landing page and form.
- Scalable rules: Server configuration can apply directives to whole folders or file types at once rather than page by page.
- A cleaner index: Keeping low-value files, staging assets and duplicates out of results helps search engines focus on the pages that matter, supporting healthy indexing.
Real-World Example of the X-Robots-Tag
A healthcare company publishes clinical guides as PDFs behind a sign-up form. Over time, several PDFs start appearing in search results, letting people bypass the form and miss the context on the landing page. The development team adds a server rule that sends "X-Robots-Tag: noindex" for every PDF in the downloads folder. After search engines recrawl the files, they drop out of results, while the landing pages continue to rank and capture leads.
How to Implement the X-Robots-Tag
The header is set at the server or edge level. On Apache, teams typically add rules to the server configuration or an .htaccess file; on Nginx, they use the add_header directive; and many CDNs and edge platforms let you add response headers by path or file type. On hosted platforms that do not expose header settings, an edge layer or proxy is often the practical route.
Two rules prevent the most common mistakes. First, do not block the same URLs in robots.txt: if crawlers cannot fetch a file, they never see its noindex header. Second, verify the header after deployment using browser developer tools, a command-line request or the URL Inspection tool in Google Search Console. Keep noindexed URLs out of your XML sitemap to avoid mixed signals, and include header checks in regular technical SEO audits.
Conclusion
The X-Robots-Tag gives you precise, scalable control over how search engines treat any file on your server. It is especially valuable for PDFs and other assets that HTML meta tags cannot reach.
Need Help With the X-Robots-Tag?
Artifact helps teams audit and fine-tune technical SEO, from crawl directives and sitemaps to site architecture. If unwanted files are showing up in search, or key pages are missing, we can help. Tell us about your project