Robots.txt GeneratorBuild Rules for Search and AI Bots

A robots.txt file is a short, public text file that tells crawlers which URLs they may fetch. One wrong line can block your whole site, and a missing line can let AI crawlers take content you meant to protect. Kwebby's Robots.txt Generator builds the file for you. Choose a preset, switch bots on or off from a list of 36, add your own rules and copy the result. This page explains how the tool works, what robots.txt can and cannot do, and the checks to run before you upload.

Key Takeaways

• Kwebby's Robots.txt Generator offers four presets, an on and off switch for 36 named bots in five groups, custom rules, a sitemap line and an AI tab that drafts a file from a description of your site.
• Robots.txt controls crawling, not indexing. Google says a disallowed page can still be indexed if other sites link to it, and that it is not a way to keep a page out of Google.
• Google supports four fields: user-agent, allow, disallow and sitemap. It ignores Crawl-delay, and Host is a Yandex field.
• The Block AI Training preset blocks the nine bots in the AI Training group, including GPTBot, ClaudeBot and Google-Extended. Claude-SearchBot and Claude-User sit in the AI Search group.
• Only blocked bots get their own group in the file. Allowed bots follow your User-agent * rules, so your /admin/ style paths apply to them too.
• The Block Everything preset writes User-agent: * with Disallow: /, which asks every crawler to stay out.
• The tool is free and needs no account.

What Is the Kwebby Robots.txt Generator?

Kwebby's Robots.txt Generator is a free web tool that writes a robots.txt file from your choices. The Builder tab lists search engine, social, research and AI crawlers and lets you allow or block each one. A Custom Rules block handles paths such as /admin/. The AI Generate tab drafts a file from a short description of your site. Copy the output or download it as robots.txt.

You do not need an account.

Key Features and How It Works

The tool has two tabs, and the Builder tab does most of the work.

Quick Presets

Four buttons set every bot at once. Allow All sets all 36 bots to Allow. Block AI Training sets the nine AI Training bots to Block and leaves the rest allowed. Block All AI blocks 23 bots, which is the AI Training, AI Search / Citation and Research groups. Block Everything sets all 36 to Block and changes the User-agent * row to Disallow: /, so the file blocks every crawler, including Googlebot and bots the list does not name. Use it only for a site that must not be crawled, such as a staging copy.

Bot Groups and Toggles

Bots sit in five groups. AI Training has GPTBot, ClaudeBot, anthropic-ai, Google-Extended, Applebot-Extended, cohere-ai, Bytespider, CCBot and Diffbot. AI Search / Citation has OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, claude-web, PerplexityBot, Perplexity-User, DuckAssistBot, YouBot, MistralAI-User and GoogleAgent-Mariner. The anthropic-ai and claude-web rows are older Anthropic tokens kept for files that still use them. Research has AI2Bot, omgili and Timpibot. Social Media has FacebookBot, meta-externalagent, LinkedInBot and Twitterbot. Traditional Search has Googlebot, Bingbot, Applebot, DuckDuckBot, YandexBot, Baiduspider, Amazonbot, Slurp and ia_archiver.

Each bot has an Allow or Block toggle with a short note on its purpose, and each group has Allow all and Block all buttons. A counter at the top shows how many bots are blocked and how many are allowed.

Custom Rules

Custom Rules starts with a row for User-agent * that disallows /admin/ and /private/. Each row has boxes for User-agent, an optional Crawl-delay, Allow and Disallow, and you can add more rows with + Add Custom Rule. Use one path per line.

Sitemap URL and Host

The Sitemap URL box adds a Sitemap line to the file when you fill it. It starts empty, so no placeholder address reaches your file. The Host box is optional and labelled for Yandex.

Generated robots.txt

The output shows one Disallow: / group for each blocked bot, with a comment naming the vendor and purpose, followed by your custom rules and the Sitemap line. Allowed bots get no group of their own, because a crawler with its own group ignores the User-agent * group. Leaving them out means they follow your * rules. Click Copy or click Download robots.txt. The tool does not test the file against your site.

AI Generate Tab

The AI Generate tab asks for your Domain, a description of your site and any extra preferences, such as protecting /api/. It sends these to Kwebby's AI service, which returns a draft. Read every line before you use it.

Why Robots.txt Matters

Robots.txt manages crawler traffic, but it is not a privacy or indexing tool. Google's robots.txt introduction says a robots.txt file tells crawlers which URLs they can access, and that it is not a mechanism for keeping a web page out of Google. To keep a page out of search results, use noindex or password protection. A page disallowed in robots.txt can still be indexed if other sites link to it.

That has two practical effects. First, never rely on robots.txt to hide private content, because anyone can read the file, and a Disallow line for /private/ tells readers the folder exists. Second, do not combine Disallow with noindex on the same page. Google's robots meta tag documentation says a page blocked from crawling cannot have its noindex read.

How Google Reads the File

Google's file guidelines say the file must be named robots.txt, sit at the root of the site host and be UTF-8 text. The interpretation rules add a 500 KiB size limit and a short list of supported fields: user-agent, allow, disallow and sitemap. Google ignores other fields, such as crawl-delay.

When rules overlap, Google applies the most specific rule by path length, and it uses the least restrictive rule if two are equally specific. Paths are case-sensitive, and a crawler with its own group follows that group instead of the User-agent * group. You can check your uploaded file in the robots.txt report in Search Console.

Which AI Crawlers Should You Block?

There is no single right answer, because AI crawlers do different jobs. Vendors split them into three kinds:

• Training crawlers collect content for model training. OpenAI says disallowing GPTBot signals that content should not be used to train its generative models.
• Search crawlers build the index behind an AI search product. OpenAI says sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers. Perplexity recommends allowing PerplexityBot and says it is not used for AI foundation models.
• User-triggered fetchers visit a page because a person asked. OpenAI says robots.txt rules may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores robots.txt.

Google-Extended is a control token for Gemini training and grounding. Google says it does not affect inclusion in Google Search or act as a ranking signal, and it has no separate user agent string.

Anthropic runs three bots. It describes ClaudeBot as collecting content that could contribute to model training, Claude-SearchBot as improving search results, and Claude-User as fetching pages when a person asks Claude. The tool files ClaudeBot under AI Training and the other two under AI Search / Citation, so you can block training and still appear in Claude's answers.

How to Use the Robots.txt Generator

Pick a starting point, adjust the bots and paths, then replace the placeholders. The steps below match the labels on the page.

1. Open the Builder tab. Check that Builder, not AI Generate, is selected.
2. Choose a Quick Preset. Start with Allow All, Block AI Training or Block All AI.
3. Adjust single bots. Use the Allow and Block toggles or the group buttons, and watch the blocked and allowed counter.
4. Edit Custom Rules. Change the /admin/ and /private/ lines to the paths you really want crawlers to skip, or clear them.
5. Enter your Sitemap URL. Type your full sitemap address, such as https://yoursite.com/sitemap.xml, or leave the box empty.
6. Read the Generated robots.txt. Check that every Disallow: / line sits under a bot you meant to block.
7. Copy or download the file. Click Copy, or click Download robots.txt.
8. Upload and test. Place the file at your domain root, open yoursite.com/robots.txt in a browser and check it in the Search Console robots.txt report.

Best Practices for Robots.txt

A short, readable file is easier to get right.

Block Paths, Not Pages You Want Indexed

Use Disallow for areas that waste crawl time, such as internal search results, carts and admin screens. Keep pages that should rank crawlable, and use noindex when a page must stay out of results.

Do Not Block Rendering Resources

Google's JavaScript SEO basics say Google Search will not render JavaScript from blocked files or on blocked pages. If a page depends on a script, leave that script's folder open unless you know why it is blocked.

Understand Block Everything

The Block Everything preset writes User-agent: * with Disallow: /. That one group covers every crawler, named or not. Clicking another preset afterwards restores the default * row.

Remember Crawl-Delay Limits

Google ignores Crawl-delay, and the Host field is for Yandex. Anthropic says it supports the non-standard Crawl-delay extension, so the setting can matter for some bots. Set it only if a crawler is straining your server.

Common Mistakes

• Pointing to a sitemap that does not exist. Open the Sitemap URL in a browser before you upload.
• Using Block Everything on a live site. It blocks Googlebot.
• Treating robots.txt as security. Use authentication for private content.
• Forgetting that rules are per host. Each subdomain needs its own file.

Common Use Cases

• Site owners set up a first robots.txt for a new site.
• Publishers opt out of AI training crawlers while staying in search.
• Developers block a staging site from crawlers.

Related Kwebby Tools

The Robots.txt Generator is part of Kwebby's SEO Tools hub. These tools work with it:

• XML Sitemap Generator builds the sitemap you point to in the Sitemap URL box.
• Meta Tag Generator writes the noindex and canonical tags that robots.txt cannot replace.
• Meta Tags Analyzer reads a live page's robots meta tag.
• Google Index Checker shows whether a page is in Google.
• HTTP Headers Checker shows the status code and headers a URL returns, which helps when your robots.txt fails to load.
• GEO Checklist covers crawler access as part of preparing for AI search.

Final Thoughts

Start with the preset closest to your policy, change the Custom Rules to your own paths and add your sitemap address. Read the output line by line, upload it to your domain root and open it in a browser. Then check it in Search Console and test a few URLs.