Free Online Tool

Robots.txt Generator

Pick a site preset and customize. Get a clean robots.txt ready to upload to your site root in seconds.

What goes in a robots.txt

robots.txt tells search engine crawlers and AI bots what paths they can access on your site. It lives at your domain root: https://yoursite.com/robots.txt. Every search engine reads it before crawling. It's the single cheapest SEO control you have.

User-agent: identifies which bot the rule applies to. * means all.
Allow / Disallow: explicitly let or block crawler access to a path.
Sitemap: points crawlers to your sitemap.xml so they discover pages faster.
Crawl-delay: not honored by Google but useful for Bing/Yandex on slow servers.

Common mistakes to avoid

Don't block CSS / JS: Google needs them to render your page properly. Blocking breaks SEO.
Don't rely on robots.txt for security: it's public and crawlers may ignore it. Use real auth.
Don't forget the sitemap line: Google can find your sitemap without it, but it's the fastest discovery path.
Test before deploying: use Google Search Console's robots.txt tester to verify.

How to use

Pick a site type preset (start with the closest match).
Add your sitemap URL.
Add any extra paths to disallow.
Optionally block AI crawlers from training on your content.
Click Generate and upload the file to your site root.

FAQ

Where do I put robots.txt on my server?

At the root of your domain so it's accessible at https://yoursite.com/robots.txt — for most CMS this means uploading to the public root or static folder.

Should I block AI bots?

Depends on your goals. Blocking GPTBot, ClaudeBot, etc. prevents your content from training future LLMs. If you want your content to be cited by AI assistants, leave them open. Many publishers block AI bots; many SEO experts leave them open for visibility.

Will blocking AI bots hurt my Google ranking?

Google-Extended is separate from Googlebot. You can block Google-Extended (Bard/Gemini training) without affecting search rankings. Same for Bingbot vs other Microsoft AI crawlers.

What about a noindex meta tag instead?

robots.txt blocks crawling. Noindex blocks indexing. They're different. If you want a page not to appear in search but still crawled (so internal links work), use noindex in HTML, not robots.txt.

How do I test my robots.txt?

1) Upload to your site. 2) Visit yoursite.com/robots.txt to confirm it loads. 3) Use Google Search Console > Settings > robots.txt tester. 4) Wait 24h for Google to refresh its cached copy.