# robots.txt Generator

URL: https://phishdestroy.io/developer-tools/robots-txt-generator
Category: General dev
Where it runs: Runs locally (nothing you enter is uploaded)

Build a robots.txt with your own rules for all crawlers, a Sitemap line and a Disallow: / group for each AI or SEO crawler you tick, from GPTBot and ClaudeBot to AhrefsBot.

## Example input

Rules for all crawlers

```
Disallow: /search
Disallow: /cart/
Allow: /
```

The output starts with User-agent: * and these three lines, followed by a User-agent and Disallow: / group for each of the 14 ticked bots.

## FAQ

### Does robots.txt actually block bots?

No, it is a request that well-behaved crawlers follow; scrapers, phishing-kit cloners and many bad bots ignore it. Never list secret paths in it, because robots.txt is public and attackers read it to find admin and login pages.

### What does Google-Extended do?

It is a control token, not a crawler: it tells Google not to use your pages to train or ground Gemini models. It does not affect crawling, indexing or ranking in Google Search.

### Is anything uploaded?

No. The file is generated in your browser and nothing is sent anywhere. The tool does not validate your rules, so check that each line is a valid Allow: or Disallow: rule.

## Related tools

- [llms.txt Generator](https://phishdestroy.io/developer-tools/llms-txt-generator)
- [Meta Robots Tag Builder](https://phishdestroy.io/developer-tools/meta-robots-builder)
- [Sitemap XML Generator](https://phishdestroy.io/developer-tools/sitemap-xml-generator)
- [security.txt Generator](https://phishdestroy.io/developer-tools/security-txt-generator)
- [Apache .htaccess Snippets](https://phishdestroy.io/developer-tools/htaccess-snippets)

All tools: https://phishdestroy.io/developer-tools