Robots.txt Generator
Create a robots.txt file online with a visual rule builder — presets for Allow all, Block all, Block AI bots and WordPress, plus sitemap support.
About the Robots.txt Generator
This free robots.txt generator builds a correct robots.txt file without you having to remember the syntax. Add rules with a visual builder — pick a user-agent (all bots, Googlebot, Bingbot, GPTBot or any custom bot), choose Allow or Disallow, and type the path. Add your sitemap URL and an optional crawl-delay, and the finished file updates live, ready to copy or download.
One-click presets cover the most common needs: Allow all (the standard open configuration), Block all (for staging sites), WordPress standard (blocks /wp-admin/ while allowing admin-ajax.php), and Block AI bots, which disallows GPTBot, CCBot, Google-Extended and anthropic-ai — the crawlers that collect content for AI training.
Upload the finished file to the root of your domain as https://yoursite.com/robots.txt. It is the first file well-behaved crawlers request, and it controls which parts of your site they may fetch.
How to Use the Robots.txt Generator
- 1Start from a preset, or build rules manually: choose a bot, Allow or Disallow, and a path.
- 2Add more rules with "+ Add rule" — rules for the same bot are grouped automatically.
- 3Enter your sitemap URL and an optional crawl-delay.
- 4Copy the output or download robots.txt and upload it to your site's root folder.
Frequently Asked Questions
Where do I put the robots.txt file?
It must live at the root of your domain: https://example.com/robots.txt. Crawlers only look there — a robots.txt inside a subfolder is ignored. Each subdomain needs its own file (blog.example.com has a separate robots.txt from www.example.com). After uploading, open the URL in your browser to confirm it is publicly readable.
How do I block AI bots like GPTBot from my website?
Click the "Block AI bots" preset. It generates Disallow: / rules for GPTBot (OpenAI), CCBot (Common Crawl), Google-Extended (Google's AI training crawler) and anthropic-ai. These companies state their crawlers respect robots.txt, so this is the standard way to opt your content out of AI training while normal search indexing by Googlebot continues unaffected.
What is the difference between Allow and Disallow?
Disallow tells a bot not to fetch URLs starting with the given path; Allow creates an exception inside a disallowed area. The classic WordPress example disallows /wp-admin/ but allows /wp-admin/admin-ajax.php, which themes need. An empty Disallow: (no path) means nothing is blocked — that is how the "Allow all" preset works.
Does robots.txt stop a page from appearing in Google?
Not reliably. Robots.txt blocks crawling, not indexing — a blocked URL can still appear in results (without a description) if other sites link to it. To keep a page out of Google, let it be crawled and add a noindex robots meta tag, or protect it with a login. Use robots.txt to manage crawl load and keep bots out of infinite or private URL spaces.
What does crawl-delay do and should I set it?
Crawl-delay asks bots to wait the given number of seconds between requests, reducing server load from aggressive crawlers. Bing, Yandex and many smaller bots honour it; Googlebot ignores it entirely (Google adjusts crawl rate automatically). Leave it empty unless a specific bot is hammering your server, then start with a value like 5 or 10 seconds.