Generate an llms.txt Markdown file (llmstxt.org proposal) from your site name, summary and key page links.
robots.txt Generator
Runs locally
Use robots.txt Generator
What it does
Build a robots.txt with your own rules for all crawlers, a Sitemap line and a Disallow: / group for each AI or SEO crawler you tick, from GPTBot and ClaudeBot to AhrefsBot.
It runs entirely in your browser: nothing you enter is uploaded.
Example
Rules for all crawlers
Disallow: /search
Disallow: /cart/
Allow: /User-agent: *
Disallow: /search
Disallow: /cart/
Allow: /
User-agent: GPTBot
Disallow: /
User-agent: ChatGPT-User
Disallow: /
User-agent: OAI-SearchBot
Disallow: /
User-agent: anthropic-ai
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: PerplexityBot
Disallow: /
User-agent: Amazonbot
Disallow: /
User-agent: cohere-ai
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: AhrefsBot
Disallow: /
User-agent: SemrushBot
Disallow: /
User-agent: DotBot
Disallow: /
User-agent: MJ12bot
Disallow: /The output starts with User-agent: * and these three lines, followed by a User-agent and Disallow: / group for each of the 14 ticked bots.
FAQ
Does robots.txt actually block bots?
No, it is a request that well-behaved crawlers follow; scrapers, phishing-kit cloners and many bad bots ignore it. Never list secret paths in it, because robots.txt is public and attackers read it to find admin and login pages.
What does Google-Extended do?
It is a control token, not a crawler: it tells Google not to use your pages to train or ground Gemini models. It does not affect crawling, indexing or ranking in Google Search.
Is anything uploaded?
No. The file is generated in your browser and nothing is sent anywhere. The tool does not validate your rules, so check that each line is a valid Allow: or Disallow: rule.