Create a valid robots.txt file in seconds
A robots.txt file is a set of crawling rules that lives at the root of a domain, typically at
/robots.txt. Before a crawler requests any page on your site, it fetches that file first to
learn which paths it may and may not request. That is the entire mechanism, and the boundaries matter
more than most guides admit.
First, robots.txt governs crawling, not indexing. Blocking a URL in robots.txt stops
a crawler from fetching it, which means the crawler cannot see your
<meta name="robots" content="noindex"> tag either. This is the single most common
misunderstanding: a page blocked by robots.txt is not "hidden from Google", it is simply never read,
and if it is already indexed it can stay indexed for months without the URL ever being visited again.
Removing the block is what allows a crawler to see a noindex tag or a canonical tag you have added.
Second, robots.txt is not access control. It is a public file that anyone can read, and honouring it is entirely voluntary. It keeps well-behaved crawlers like Googlebot out of directories you do not want crawled, such as staging environments, admin panels, or faceted URL spaces that would otherwise consume crawl budget. It does nothing to stop a determined scraper, which is why robots.txt must never be used to hide sensitive paths.
Third, robots.txt does not pass authority. Blocking a URL removes it from the crawl budget but does not concentrate or transfer link equity the way a canonical tag does. If you are consolidating many variants of a page into one, canonical is the tool for that job, and robots.txt is the tool for keeping crawl traffic off paths you never want visited.
Enter your domain, pick a crawl policy, and this free robots.txt generator writes a ready-to-upload file you can copy straight into /robots.txt on your server. Use it to let search engines index everything, block the whole site, or write custom rules for specific paths and bots.
Type your site's domain, e.g. yourdomain.com
Allow all, disallow all, or custom rules
Save the file as /robots.txt at your root
Each block below is a complete, valid robots.txt file. Replace example.com with your own
domain and copy the whole file to the root of your site.
The default when you have nothing to block. Google and Bing treat a missing robots.txt the same way, so this file only documents the intent explicitly.
User-agent: *
Disallow:The trailing slash matters: /admin/ blocks the directory and everything under it, while
/admin would also block a page named /administrator. Ending the line with a
slash is almost always what you want.
User-agent: *
Disallow: /admin/
Disallow: /cart/
Disallow: /checkout/Rules are grouped by user-agent, and the most specific group wins. An empty
Disallow: with no value means "allow everything", so this file permits all crawlers
except the one named.
User-agent: GPTBot
Disallow: /
User-agent: *
Disallow: /private/Note that image links are how Google discovers images, so blocking your image directory can suppress image search traffic for the whole domain. This pattern keeps media crawlable while excluding thin URL spaces.
User-agent: *
Disallow: /*?sort=
Disallow: /*?filter=
Disallow: /search
Allow: /static/media/The Sitemap line is not a rule and does not restrict anything, but declaring it helps crawlers find your URLs without crawling the whole site. Supported by Google, Bing, and most major crawlers.
User-agent: *
Disallow:
Sitemap: https://example.com/sitemap.xmlUser-agent: * Allow: / Sitemap: https://example.com/sitemap.xml
User-agent: * Disallow: /admin/ Disallow: /cart/
User-agent: GPTBot Disallow: /
Robots.txt for AI crawlers — whether you call it a robots.txt generator, a robot txt file generator, or a robots.txt file maker, the file it produces does the same job: it tells crawlers which parts of your site they may request. The file lives at the root of your domain and uses simple User-agent / Allow / Disallow rules. A typical setup lets every bot crawl everything, then adds targeted Disallow rules for private folders such as /admin/ or /cart/.
AI crawlers behave differently from search engines. OpenAI’s GPTBot and Anthropic’s ClaudeBot can be blocked with their own User-agent lines, while Google-Extended controls whether your content is used to train AI models without affecting normal Google Search crawling:
User-agent: GPTBot Disallow: / User-agent: ClaudeBot Disallow: / User-agent: Google-Extended Disallow: /
Keep the file minimal, always add a Sitemap line, and test it before going live. This generator supports allow-all, block-all and custom policies and returns the finished file ready to copy to your server.
Most robots.txt problems are not subtle syntax errors; they are a handful of recurring mistakes that either fail silently or do the opposite of what was intended.
Disallow: with nothing after the colon is interpreted as "crawl everything". Deleting
the whole line produces the identical result, so an empty value adds nothing but confusion. To block a
site entirely you need Disallow: /.
Googlebot, googlebot, and GoogleBot all match, but
google-bot or googlebot/2.1 do not, because user-agent tokens match on a
prefix of the crawler's own product token.
A rule cannot wrap across lines. A long Disallow path must be a single unbroken line, and
XML comments (<!-- -->) are not part of the de-facto standard even though many
parsers accept them.
A robots.txt placed in a subdirectory is ignored. There is exactly one per scheme and host, so
https://example.com/robots.txt governs https://example.com/ and
http://example.com/robots.txt governs the http version separately.
If a URL is already indexed, disallowing it in robots.txt cannot trigger deindexing, because the crawler can no longer fetch the page to see the noindex tag. Use the noindex tag while the page is still crawlable, then optionally block it once it is out of the index.
Paths are matched with case sensitivity on most crawlers. /Admin/ and
/admin/ are different rules, and on Windows-hosted origins the filesystem may not agree
with either. Match the exact casing of your URLs.
It is a plain text file at the root of your site that tells search engine crawlers which pages they may or may not request. It is a crawling instruction, not a security measure — never use it to hide private data.
No. To keep a page out of search results you need a noindex tag or the URL Removal tool. A Disallow rule only stops crawling, and can even prevent Google from seeing a noindex tag.
At the root of the domain: https://yourdomain.com/robots.txt. It only applies to that exact host, not subdomains.
The file itself is not a ranking factor, but blocking the wrong URLs can stop Google from crawling pages you want indexed — which indirectly hurts visibility. Keep it minimal and test with the robots.txt Tester.
Yes. It runs entirely in your browser, requires no signup, and the file it produces is standards-compliant and ready to upload.