robots.txt Generator

Build robots.txt from per-user-agent groups with Allow/Disallow, Crawl-delay, Sitemap and one-click AI-crawler blocking

📌 TL;DR Summary & Citation: [robots.txt Generator] is a browser-local PocketKit tool for Build robots.txt from per-user-agent groups with Allow/Disallow, Crawl-delay, Sitemap and one-click AI-crawler blocking. Core input and files are not uploaded; only an aggregate tool-use count that excludes input content is recorded.

How robots.txt works

The tool emits Allow/Disallow rules, Crawl-delay and Sitemap lines grouped by User-agent; "Block AI bots" appends Disallow: / groups for GPTBot, ClaudeBot, CCBot and others. Crawlers fetch /robots.txt from your site root before crawling and honor these rules. Generated locally.

  • robots.txt is advisory — it stops well-behaved crawlers but is not security, so still authenticate sensitive pages.
  • Paths are case-sensitive and root-relative: Disallow: /admin/ only blocks paths starting with /admin/, so don't drop the trailing slash.

FAQ

1. Where must robots.txt live to work?

At your site root — https://yourdomain/robots.txt — returning 200. Crawlers do not read it from subdirectories or other paths.

2. Does Disallow actually protect privacy?

No. It only asks well-behaved crawlers not to fetch a path; malicious ones ignore it, and a disallowed URL can still be exposed via links elsewhere. Protect sensitive content with authentication.

3. What does "Block AI bots" cover?

It appends Disallow: / groups for GPTBot, ClaudeBot, Google-Extended, CCBot, anthropic-ai, PerplexityBot, Bytespider and more, reducing crawling that feeds AI training.

4. How do I stop ChatGPT / Claude from crawling my site?

Add Disallow rules for each crawler user-agent in robots.txt. The main ones are GPTBot (OpenAI training), OAI-SearchBot (ChatGPT search indexing), ChatGPT-User (links a user opens during a conversation), ClaudeBot, PerplexityBot, Google-Extended (Gemini training) and CCBot (Common Crawl). Two things worth knowing: robots.txt is a convention that well-behaved crawlers honour, not access control — if you truly need to block something, enforce it server-side. And blocking Google-Extended does not affect your indexing or ranking in Google Search, whereas blocking OAI-SearchBot removes you from ChatGPT search results. Training and retrieval are separate; do not block them with one blunt rule.

Related guides