Robots.txt is a text file which resides within your root directory that informs robots, dispatched by search engines, which pages to crawl and which to just overlook.<\/p>\n\n\n\n
By this, you have a little idea about how powerful this tool is! If you utilise it correctly then it will increase crawl frequency<\/a> which can impact SEO efforts.<\/p>\n\n\n\n So, How to use this file? What to avoid? How does this work? Here is the blog to find the answers to your questions.<\/p>\n\n\n\n When you create a new website, search engines will send their search bots to crawl through your website and make a sitemap<\/a> of what that page contains. In this way, they\u2019ll know what exact pages to show when someone searches using the related keywords. <\/p>\n\n\n\n Robot exclusion protocol<\/a> (REP), is a set of web standards that regulates how robots crawl to the web, index content and serve the content to users and robots.txt is a part of it. Robots.txt is an execution of this protocol that delineates the guideline that every robot should follow, including Google bots. <\/p>\n\n\n\n Be careful while making changes in the robots.txt file, that file must have the potential to make big parts of your website inaccessible to search engines<\/a>. a robots.txt is only valid if you have a full domain, including protocol<\/a>.<\/p>\n\n\n\n You can also take a quick sneak peek of any website by typing any URL and adding\/robots.txt file at the end, simple.<\/p>\n\n\n\n Robots.txt file controls the unnecessary crawler to get access to certain areas of your sites. If you ever disallow Google bots<\/a> to crawl your site this might lead to extreme danger.<\/p>\n\n\n\n Some of the uses are:<\/p>\n\n\n\n In your site, if there is no area you want to control user agents’ access, then you\u2019ll not need a robots.txt file.<\/p>\n\n\n\n Robots.txt is a text-based file, so it is super easy to create. You can simply create it on your notepad, like a text editor. Open that sheet and just save it as a robots.txt file.<\/p>\n\n\n\n Then log in to your cPanel and locate a public HTML folder to access your site’s root directory. After that drag those files into it.<\/p>\n\n\n\n Then ensure that you have the correct permission for your file. As an owner, you need to read, write and execute the file and no one can get control over it.<\/p>\n\n\n\n A file will display the \u201c644\u201d permission code. If it is not showing this code then you\u2019ll have to change it through file permission. <\/p>\n\n\n\n Now yes, your robots.txt file is ready! <\/p>\n\n\n\n How you can find your robots.txt file is another question. For that, you have to open FTP cPanel and there you’ll find the file of your public_html website directory.<\/p>\n\n\n\n Once you open the file you\u2019ll be greeted by some message, which will look like this.<\/p>\n\n\n\n Some popular sites use robots.txt to guide the search engine crawlers that what they should do and what not.<\/p>\n\n\n\n One of the examples is Facebook.com; if you open their robots.txt <\/a>file, You will see how they want different bots to navigate their website and for the most part they disallow various section of their website like “share”, “plugins” etc;<\/p>\n\n\n\n Another example is Amazon.com; if you try to access their robots.txt<\/a> file, You will see different patter they use if we compare to the Facebook above, they use a global variable i.e. * for all bots and direct all of them to disallow certain directories which are mentioned in their file;<\/p>\n\n\n\n There are some pros and cons of using the robots.txt file. Pros are:<\/p>\n\n\n\n Cons are :<\/p>\n\n\n\n A robots.txt file is made up of different directives and each of them begins with user agents. <\/p>\n\n\n\n There are various search engines and different search engines interpret them differently. <\/p>\n\n\n\n First, matching directives always wins, but the chance with Bing and Google is always high!<\/p>\n\n\n\n We can say robots.txt syntax<\/a> is the language of the robots.txt file. There are five common terms you may come across – user agents, allow, disallow, crawl delay and sitemap. <\/p>\n\n\n\n Sitemap directive tells search engines where to find an XML sitemap<\/a>. <\/p>\n\n\n\n Each search engine identifies itself by user agents as Google<\/a> identifies as Googlebot, Yahoo identifies as Slurp and Bing as BingBot and the list goes on. User agents mean a specific crawler bot to which you\u2019re instructing, this can be a search engine.<\/p>\n\n\n\n Any program on the internet has a ‘user agent’ or we can say the assigned name. For human users, this contains information like operating system version and browsing type that does not contain personal information.<\/p>\n\n\n\n If you see in the first few lines in the block, that is user agents. It will pinpoint specific bots. User agents identify which crawler the rules apply to.<\/p>\n\n\n\n There are thousands of web crawlers<\/a> and user agents wandering the internet, but these are some most common ones:<\/p>\n\n\n\n If you see any second line in a block of directives then it’s called a Disallow directive<\/a>. You can use this to specify that user agents are not allowed to crawl this URL. <\/p>\n\n\n\n Here’s an example of a The disallow directive Robots.txt file:<\/p>\n\n\n\n User-agent: *<\/p>\n\n\n\n Disallow: \/cgi-bin\/<\/p>\n\n\n\n Disallow: \/admin\/<\/p>\n\n\n\n Disallow: \/private-directory\/<\/p>\n<\/blockquote>\n\n\n\n It means these particular areas are disallowed for bots. You can simply tell search engines not to access certain areas, files or pages. <\/strong><\/p>\n\n\n\n Some leading crawlers support an Allow directive, which can compensate for disallow directive. You can use this to specify that user agents are allowed to crawl this URL. It’s only applicable to Google bots<\/a>.<\/p>\n\n\n\n Here’s an example of a The allow directive Robots.txt file:<\/p>\n\n\n\n User-agent: *<\/p>\n\n\n\n Disallow: \/cgi-bin\/<\/p>\n\n\n\n Allow: \/cgi-bin\/forum\/<\/p>\n<\/blockquote>\n\n\n\n It is used to counteract a disallowed directory. It is supported by Bing and Google. Bing uses allow and disallow directive, whichever is more distinct, based on length, like Google.<\/p>\n\n\n\n The noindex directive is not a part of the Robots.txt standard, but it’s widely used by webmasters<\/a>. It is a meta tag that can be placed in the HTML header of a page to instruct search engines not to index the page.<\/p>\n\n\n\n Here’s an example of a The noindex directive Robots.txt file:<\/p>\n\n\n\n User-agent: *<\/p>\n\n\n\n Disallow: \/cgi-bin\/<\/p>\n\n\n\n Noindex: \/private.html<\/p>\n<\/blockquote>\n\n\n\n The Robots META Tag<\/a> has an attribute called “nofollow” that can be used to instruct some search engines that a hyperlink should not influence the link target’s search engine ranking.<\/p>\n\n\n\n Here’s an example of a The nofollow directive Robots.txt file:<\/p>\n\n\n\n User-agent: *<\/p>\n\n\n\n Disallow: \/cgi-bin\/<\/p>\n\n\n\n Nofollow: \/private.html<\/p>\n<\/blockquote>\n\n\n\n The Sitemap directive is not part of the Robots.txt standard, but it’s widely supported by all major search engines. It allows webmasters to include a sitemap location in Robots.txt file.<\/p>\n\n\n\n Here’s an example of a The sitemap directive Robots.txt file:<\/p>\n\n\n\n User-agent: *<\/p>\n\n\n\n Disallow: \/cgi-bin\/<\/p>\n\n\n\n Sitemap: http:\/\/www.example.com\/sitemap.xml<\/p>\n<\/blockquote>\n\n\n\n The Host directive is not part of the Robots.txt standard, but it’s supported by all major search engines. It allows webmasters to specify the preferred domain name for a site that has multiple domain names.<\/p>\n\n\n\n Here’s an example of a The host directive Robots.txt file:<\/p>\n\n\n\n User-agent: *<\/p>\n\n\n\n Disallow: \/cgi-bin\/<\/p>\n\n\n\n Host: www.example.com<\/p>\n<\/blockquote>\n\n\n\n Crawl-delay directive means how many seconds the crawler can wait to load any content and crawl web page content. Yahoo, Yandex and Bing are good when it comes to crawling and they respond to crawl-delay<\/a> directives.<\/p>\n\n\n\n Here’s an example of a crawl delay directive Robots.txt file:<\/p>\n\n\n\n User-agent: *<\/p>\n\n\n\n Disallow: \/cgi-bin\/<\/p>\n\n\n\n Crawl-delay: 10<\/p>\n<\/blockquote>\n\n\n\n This means you can make the search engine wait for 5 seconds to crawl your site or 5 seconds before re-accessing your site, it is the same, but just has slight differences depending upon the search engine.<\/p>\n\n\n\n Googlebot does not respond to this directive, but the crawl rate can be set in Google Search Console<\/a>. It says that, avoid using crawl-delay directives for search engines.<\/p>\n\n\n\n So now we understand what Robots.txt file is and where you should put your file and their agents too. But now you might have questions like,\u2019\u201d I have my eCommerce website and I would like to disallow all the pages which contain question marks or any other unwanted signs in their URLs\u201d.<\/p>\n\n\n\n Here then comes our wildcard to help. There are some things you have considered. First, you don’t need to append wildcards to every string in your robots.txt file and the second thing is you have to know that there are two types of wildcards:<\/p>\n\n\n\n *wildcard<\/strong>: The *wildcards will simply match any sequence character. This type of wildcard will be great for URLs which have the same pattern.<\/p>\n\n\n\n Example of Using *wildcard :<\/p>\n\n\n\n User-agent: *<\/p>\n\n\n\n Disallow: \/*?<\/p>\n<\/blockquote>\n\n\n\n The above Robots.txt will disallow all the pages which contain question mark in their URLs. So, if your website has a lot of pages with question mark then this Robots.txt will be good for you. But what if your website has some pages with question mark and some pages without question mark?<\/p>\n\n\n\n $wildcard:<\/strong> The $wildcard is used to show the end of the URL. If you see your robots are not allowing the bots from accessing the PDFs, then this wildcard might come in handy.<\/p>\n\n\n\n Example of using $Wildcard:<\/p>\n\n\n\n User-agent: *<\/p>\n\n\n\n Disallow: \/*.pdf$<\/p>\n<\/blockquote>\n\n\n\n This Robots.txt will help you to disallow bots from accessing all the PDFs of your website. So, this is how you can use Robots.txt to control what search engine bots can and cannot access your site. Robots.txt is a helpful tool that can be used to improve your website’s SEO.<\/p>\n\n\n\n Wildcards are not just used for defining the user agents, but also can be used to match URLs. Wildcards are supported by Google, Yahoo and Ask. Remember, it’s always great to double-check your robots.txt wildcards before publishing them.<\/p>\n\n\n\n If you are using robots.txt then you have to test it in the robots.txt tester for whether it is blocking Google web crawls from any URLs on your sites. <\/p>\n\n\n\n You can use the robots.txt tester tool <\/a>to check whether the Googlebot – Image crawler can crawl the file you want to block from Google Image Search.<\/p>\n\n\n\n You can use any program that produces the text file or you can simply use an available option that is Google Webmaster<\/a>. Once you create your robots.txt file then add this to your top directory of the server. After that make sure you set the correct permission for the visitors to read anytime.<\/p>\n\n\n\n You can also test if your URL is blocked by Googlebot or not by using the form just below the tool;<\/p>\n\n\n\n The Robots.txt file is important for all websites, especially eCommerce sites which have a lot of products and categories. If you don’t have a Robots.txt file or if it’s not correctly configured, search engines might index all your pages, which can result in lower rankings because of duplicate content.A Robots.txt file should be placed in the root directory of your website. The Robots.txt file is a text file and must be named “robots.txt”.<\/p>\n\n\n\n Here’s an example of a Robots.txt file:<\/p>\n\n\n\n User-agent: *<\/p>\n\n\n\n Disallow: \/cgi-bin\/<\/p>\n\n\n\n Disallow: \/tmp\/<\/p>\n\n\n\n Disallow: \/~joe<\/p>\n\n\n\n \/<\/p>\n\n\n\n Allow: \/~joe\/public_html\/<\/p>\n\n\n\n Sitemap: http:\/\/www.example.com\/sitemap.xml<\/p>\n<\/blockquote>\n\n\n\n The above Robots.txt file will block all bots from the \/cgi-bin\/, \/tmp\/ and \/~joe\/ directories, and it will allow the bot to crawl the pages in the \/~joe\/public_html\/ directory. It will also tell the bots where to find your sitemap.<\/p>\n\n\n\n If you want to learn more about Robots.txt or if you need help creating a Robots.txt file for your website, we recommend that you contact a professional SEO company.<\/p>\n\n\n\n You can generate your Robots.txt file using our very own Robots.txt File Generator tool<\/a>;<\/p>\n\n\n\n Here are the options you need to consider;<\/p>\n\n\n\n Option 1<\/strong>: All robots are either allowed or refused, If you have a private app which don’t have information available for public then you can select “Refused” otherwise you can allow it;<\/p>\n\n\n\n Option 2<\/strong>: Crawl Delay option enables to delay the bots for some interval time to reduce server loading, If your site is big and gets lots of traffic then you should enable it for 10 seconds otherwise there’s no point of delaying it for the starter or medium sized websites;<\/p>\n\n\n\n Option 3<\/strong>: This option allows you to include Sitemap URL of your website to let the robot knows where your sitemap is and crawl only the necessary URLs which is mentioned in the sitemap itself.<\/p>\n\n\n\n Generally, Your sitemap file is situated at root folder i.e. example.com\/sitemap.xml or example.com\/sitemap_xml, copy the same URL and paste it in the option;<\/p>\n\n\n\n Option 4<\/strong>: In this section, you can allow or disallow some search robots you don’t want to crawl your website. Just look at the list of bots and let select either “allow” or “refused” to enable or disable them to crawl your website;<\/p>\n\n\n\n Option 5<\/strong>: Restricted directories let you disallow some directories which you don’t want bots to crawl, which is generally files URLs or private administrative directories i.e. “\/admin”, “\/login” etc.<\/p>\n\n\n\n Now, you have an option to create or save the robots.txt file and paste it to your root folder.<\/p>\n\n\n\n Robots.txt file plays a vital role in SEO. A robots.txt file tells the search engine crawls which URLs crawler<\/a> can have access on your site. This factor is used to avoid overloading your site. If you want to keep the page out of Google, then block indexing with no index or password to protect your page.<\/p>\n\n\n\n A big part of SEO is sending the right signals at the right time to search engines<\/a>, and robots.txt is the best way to communicate your crawling preference to search engines.<\/p>\n\n\n\n For SEO’s best results, make sure you are not blocking any of the content<\/a> that you want to be crawled. Search engines will cache the robots.txt content but usually update that content once a day. If you want this processor to get faster then you can submit your robots.txt URL to Google.<\/p>\n\n\n\n Do not use robots.txt<\/a> for any private data, because others can get direct access to your information, it may still get indexed. For that use some page passwords to protect your data. Robots.txt file is case sensitive, that’s why the file must be named in lower case like robots.txt, not a ROBOTS.TXT.<\/p>\n\n\n\n We have talked about every detailed function of robots.txt and now let’s move a little deeper into it and understand how each may turn into an SEO disaster if not utilised properly. <\/p>\n\n\n\n You mustn’t block any good content that you want to present publicly by robots.txt or noindex tag<\/a> and being crawled. If you do it wrongly, it might hurt SEO. And as we previously said don’t overuse crawl- delay as it leads to limiting your pages crawled by the bots.<\/p>\n\n\n\n This can be good for some websites but if you have a huge website then you are doing trouble by losing all solid traffic. Robots.txt file is case sensitive as mentioned early, you have to call it ‘robots.txt’ in lower case otherwise, it won’t work!<\/p>\n\n\n\n The question is asked on SEO Work Hours August 2024 Edition<\/a>. It was noticed by the user that the robots.txt <\/a>URL had been shown as a Soft 404 error<\/a> on the Google search console. <\/p>\n\n\n\n John Mueller Answered this;<\/p>\n\n\n\n This one’s easy. That’s fine. You don’t need to do anything. The robots.txt file generally doesn’t need to be indexed. It’s fine to have it be seen as a soft 404.<\/p>\n<\/blockquote>\n\n\n\n Therefore, If you see such errors, then do not bother, its fine to be in soft 404 error. <\/p>\n\n\n\n You can do this effortlessly using Robots.txt, hide your login page from search crawlers, and prevent them from indexing. <\/p>\n\n\n\n For example, if your admin login page is located at “www.example.com\/admin\/login”, you can add the following line in your Robots.txt file:<\/p>\n\n\n\nWhat is a robots.txt file?<\/h2>\n\n\n\n
![\"\"](\"https:\/\/kwebby.com\/blog\/wp-content\/uploads\/2022\/07\/CleanShot-2022-07-06-at-13.57.16@2x-1024x612.png\" "\\"\\"")
<\/figure>\n\n\n\n![\"\"](\"https:\/\/kwebby.com\/blog\/wp-content\/uploads\/2022\/07\/CleanShot-2022-07-06-at-13.57.50.gif\" "\\"\\"")
<\/figure>\n\n\n\nWhat does the robots.txt file do?<\/h2>\n\n\n\n
![\"\"](\"https:\/\/kwebby.com\/blog\/wp-content\/uploads\/2022\/07\/robots.txt-1024x1024.png\" "\\"\\"")
<\/figure>\n\n\n\n\n
Where should I put or find my robots.txt file?<\/h2>\n\n\n\n
![\"\"](\"https:\/\/kwebby.com\/blog\/wp-content\/uploads\/2022\/07\/CleanShot-2022-07-06-at-14.11.48@2x-1024x669.png\" "\\"\\"")
<\/figure>\n\n\n\n![\"\"](\"https:\/\/kwebby.com\/blog\/wp-content\/uploads\/2022\/07\/CleanShot-2022-07-06-at-14.12.43@2x-1024x669.png\" "\\"\\"")
<\/figure>\n\n\n\nHow big sites use robots.txt?<\/h2>\n\n\n\n
![\"\"](\"https:\/\/kwebby.com\/blog\/wp-content\/uploads\/2022\/07\/CleanShot-2022-07-06-at-14.13.49@2x-1024x600.png\" "\\"\\"")
<\/figure>\n\n\n\n![\"\"](\"https:\/\/kwebby.com\/blog\/wp-content\/uploads\/2022\/07\/CleanShot-2022-07-06-at-14.14.03@2x-1024x599.png\" "\\"\\"")
<\/figure>\n\n\n\nPros and cons of using robots.txt<\/h2>\n\n\n\n
\n
\n
Robots.txt syntax<\/h2>\n\n\n\n
The user-agents<\/h2>\n\n\n\n
The most common user agents for search engine spiders<\/h3>\n\n\n\n
\n
The disallow directive<\/h3>\n\n\n\n
\n
The allow directive<\/h3>\n\n\n\n
\n
The noindex directive<\/h3>\n\n\n\n
\n
The nofollow directive<\/h3>\n\n\n\n
\n
The sitemap directive<\/h3>\n\n\n\n
\n
The host directive<\/h3>\n\n\n\n
\n
The crawl-delay directive<\/h3>\n\n\n\n
\n
How to use wildcard\/ regular expressions<\/h2>\n\n\n\n
\n
\n
Validate your robots.txt<\/h2>\n\n\n\n
![\"\"](\"https:\/\/kwebby.com\/blog\/wp-content\/uploads\/2022\/07\/CleanShot-2022-07-06-at-14.19.02@2x-1024x469.png\" "\\"\\"")
<\/figure>\n\n\n\n![\"\"](\"https:\/\/kwebby.com\/blog\/wp-content\/uploads\/2022\/07\/CleanShot-2022-07-06-at-14.21.58@2x-1024x311.png\" "\\"\\"")
<\/figure>\n\n\n\nFinal code<\/h2>\n\n\n\n
\n
Generate robots.txt file using our tool Kwebby Robot TXT Generator<\/h2>\n\n\n\n
![\"\"](\"https:\/\/kwebby.com\/blog\/wp-content\/uploads\/2022\/07\/CleanShot-2022-07-06-at-14.25.15@2x-1024x548.png\" "\\"\\"")
<\/figure>\n\n\n\n![\"\"](\"https:\/\/kwebby.com\/blog\/wp-content\/uploads\/2022\/07\/CleanShot-2022-07-06-at-14.31.21@2x-1024x452.png\" "\\"\\"")
<\/figure>\n\n\n\n![\"\"](\"https:\/\/kwebby.com\/blog\/wp-content\/uploads\/2022\/07\/CleanShot-2022-07-06-at-14.33.22@2x-1024x467.png\" "\\"\\"")
<\/figure>\n\n\n\n![\"\"](\"https:\/\/kwebby.com\/blog\/wp-content\/uploads\/2022\/07\/CleanShot-2022-07-06-at-14.36.00@2x-1024x464.png\" "\\"\\"")
<\/figure>\n\n\n\n![\"\"](\"https:\/\/kwebby.com\/blog\/wp-content\/uploads\/2022\/07\/CleanShot-2022-07-06-at-14.37.01@2x-1024x464.png\" "\\"\\"")
<\/figure>\n\n\n\n![\"\"](\"https:\/\/kwebby.com\/blog\/wp-content\/uploads\/2022\/07\/CleanShot-2022-07-06-at-14.41.05@2x-1024x421.png\" "\\"\\"")
<\/figure>\n\n\n\n![\"\"](\"https:\/\/kwebby.com\/blog\/wp-content\/uploads\/2022\/07\/CleanShot-2022-07-06-at-14.41.56@2x-1024x177.png\" "\\"\\"")
<\/figure>\n\n\n\nWhy is robots.txt important for SEO?<\/h2>\n\n\n\n
Other tips for robots.txt<\/h2>\n\n\n\n
Frequently Asked Questions (FAQs)<\/h2>\n\n\n\n
Why Is the Robots.txt file shown as a Soft 404 Error in Google Search Console?<\/h3>\n\n\n\n
\n
Can I Hide My Admin Login Page Using Robots.txt file from Search Crawlers?<\/h3>\n\n\n\n