Robots.txt Generator
Create a correct robots.txt in seconds: block private folders, decide what AI crawlers may do, add your sitemap — and test any URL against the file with Google's matching rules before it goes live. Dangerous rules are flagged loudly.
robots.txt to the root of your site: https://example.com/robots.txt.Need the sitemap file itself? XML Sitemap Generator →What Is robots.txt?
A plain-text file that tells crawlers which parts of your site they may fetch.
robots.txt lives at the root of a site (https://example.com/robots.txt) and follows the Robots Exclusion Protocol (RFC 9309). It contains groups of rules: each group names one or more crawlers with User-agent and lists paths they may not (Disallow) or may (Allow) crawl. It can also point crawlers to your Sitemap.
Important: robots.txt controls crawling, not indexing. A blocked page can still appear in Google if other sites link to it — to keep a page out of search results, use a noindex meta tag and let Google crawl it. And robots.txt is public and voluntary: never use it to hide private content.
AI crawlers in 2026. Most AI companies now run separate crawlers for model training (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot…) and for search and answers (OAI-SearchBot, Claude-SearchBot, PerplexityBot…). The “Block AI training” policy blocks the first kind while keeping your site visible in AI search and answer engines. Blocking Google-Extended doesn't affect Google Search.
A crawler obeys only the most specific group that matches it. If you add a group for Googlebot, Googlebot ignores the User-agent: * rules entirely — the generator warns you when that drops a rule you probably wanted.
How Google Reads robots.txt
The matching rules the built-in tester uses.
One group per crawler
The group with the most specific matching User-agent wins; groups aren't combined. * is used only if no other group matches. User-agent names are case-insensitive.
Longest rule wins
When several rules match a URL, the longest path wins. If an Allow and a Disallow are equally long, Allow wins. Paths are case-sensitive and start with /.
Wildcards
* matches any sequence of characters; $ marks the end of the URL. /*.pdf$ blocks every URL ending in .pdf.
Not supported by Google
Crawl-delay, Noindex and Host are ignored by Google. Files over 500 KiB are cut off.
How to Use the Robots.txt Generator
Get your result in under ten seconds.
Set the defaults
Allow the whole site and list folders to block, like /admin/ or /cart.
Choose an AI policy
Keep AI crawlers on the default rules, block only training, or block them all. Add groups for other crawlers if needed.
Test and upload
Test important URLs in the tester, then upload the file as robots.txt to your site's root.
Why Use the Robots.txt Generator on Developers Web Tools
Fast, accurate and private — built for developers.
URL tester
Check any URL against the generated file for Googlebot, Bingbot, GPTBot and more — with the rule that decided it.
AI crawler policies
Up-to-date lists of AI training and AI search crawlers, grouped correctly.
Danger warnings
Flags rules that block your whole site, CSS/JS or search engines.
Valid syntax
Paths, wildcards, sitemaps and crawl-delay values are checked.
Ready to upload
Download a plain robots.txt file with the right line endings.
Private
Runs entirely in your browser.
Frequently Asked Questions
In the root of your site, so it's reachable at https://yourdomain.com/robots.txt. Each subdomain (blog.example.com) and each protocol needs its own file. Crawlers don't look for it in subfolders.
No. Disallow stops crawling, but the URL can still be indexed if other pages link to it. To remove a page from search results, allow crawling and add a noindex meta tag, or protect it with a password.
Choose “Block AI training” or “Block all AI crawlers”. Neither touches Googlebot, so Google Search is unaffected. Blocking Google-Extended stops Gemini training use but doesn't change your Google Search visibility.
No. Google needs CSS and JavaScript to render pages and judge mobile-friendliness. Blocking them can hurt how your pages are understood and ranked.
“Disallow:” with no path means nothing is blocked — the crawler may fetch everything. “Disallow: /” blocks the entire site.
No, Google ignores it. Bing and Yandex respect it. To slow Googlebot down in an emergency, return 503 or 429 status codes temporarily.
Reputable search engines and the major AI companies say they do. Malicious bots ignore it — robots.txt is a request, not security. Use server rules or a firewall to actually block access.