{"id":23220,"date":"2024-08-30T10:42:35","date_gmt":"2024-08-30T10:42:35","guid":{"rendered":"https:\/\/kwebby.com\/blog\/?p=23220"},"modified":"2024-08-30T10:43:08","modified_gmt":"2024-08-30T10:43:08","slug":"what-is-x-robots-tag-how-and-when-to-use","status":"publish","type":"post","link":"https:\/\/kwebby.com\/blog\/what-is-x-robots-tag-how-and-when-to-use\/","title":{"rendered":"What is X-Robots-Tag? How and When to Use?"},"content":{"rendered":"\n

In the digital marketing world, controlling your website\u2019s visibility to search engines is key. Enter the X-Robots-Tag<\/a>, a powerful but often overlooked tool to fine-tune how your content interacts with search engine crawlers<\/a>.<\/p>\n\n\n\n

Whether you want to stop certain pages from being indexed or tell search engines how to handle your content, the X-Robots-Tag is your new best friend for optimising your online presence.<\/p>\n\n\n\n

In this post, we\u2019ll go into what the X-Robots-Tag is, how it works and best practices for using it so you can get the most out of your website\u2019s SEO.<\/p>\n\n\n\n

What is the X-Robots-Tag?<\/h2>\n\n\n\n

The X-Robots-Tag might sound technical but it\u2019s a simple yet powerful tool to optimise your website\u2019s SEO. It\u2019s an optional part of the HTTP response header<\/a> that tells search engines how to crawl and index your web pages.<\/p>\n\n\n\n

Unlike the robots meta tag which can only be applied to HTML pages the X-Robots-Tag can be applied to non-HTML files including images, text files and PDFs. So you can manage the visibility of different content types with ease.<\/p>\n\n\n\n

For example if you have a PDF document with sensitive information that shouldn\u2019t be in search results you can use the X-Robots-Tag in the HTTP response to tell search engines not to index it. Imagine your server responding with something like this:<\/p>\n\n\n\n

<\/circle><\/circle><\/circle><\/g><\/svg><\/span><\/path><\/path><\/svg><\/span>
HTTP<\/span>\/<\/span>1.1<\/span> <\/span>200<\/span> <\/span>OK<\/span><\/span>\nDate<\/span>:<\/span> <\/span>Tue<\/span>,<\/span> <\/span>30<\/span> <\/span>Aug<\/span> <\/span>2024<\/span> <\/span>11<\/span>:<\/span>38<\/span>:<\/span>17<\/span> <\/span>GMT<\/span><\/span>\nContent<\/span>-<\/span>encoding<\/span>:<\/span> <\/span>gzip<\/span><\/span>\nX<\/span>-<\/span>Robots<\/span>-<\/span>Tag<\/span>:<\/span> <\/span>noindex<\/span><\/span><\/code><\/pre><\/div>\n\n\n\n
This lets you control not just how your website pages are indexed but also how external files are treated by search crawlers so you have a more comprehensive approach to managing your online presence. You can even specify directives for specific user agents as in this example:<\/p>\n\n\n\n
<\/circle><\/circle><\/circle><\/g><\/svg><\/span><\/path><\/path><\/svg><\/span>
HTTP<\/span>\/<\/span>1.1<\/span> <\/span>200<\/span> <\/span>OK<\/span><\/span>\nDate<\/span>:<\/span> <\/span>Tue<\/span>,<\/span> <\/span>30<\/span> <\/span>Aug<\/span> <\/span>2024<\/span> <\/span>11<\/span>:<\/span>38<\/span>:<\/span>17<\/span> <\/span>GMT<\/span><\/span>\nContent<\/span>-<\/span>encoding<\/span>:<\/span> <\/span>gzip<\/span><\/span>\nX<\/span>-<\/span>Robots<\/span>-<\/span>Tag<\/span>:<\/span> googlebot<\/span>:<\/span> <\/span>noarchive<\/span>,<\/span> <\/span>nofollow<\/span><\/span><\/code><\/pre><\/div>\n\n\n\n
By using the X-Robots-Tag you get an extra layer of control that can really help your digital presence.<\/p>\n\n\n\n
X-Robots-Tags Derivatives<\/h2>\n\n\n\n
As we get into the X-Robots-Tag more we need to understand its derivatives. Google\u2019s specifications say that any directive used in a robots meta tag can also be used as an X-Robots-Tag. While the full list of valid indexing and serving directives is long the most common ones are:<\/p>\n\n\n\n