6XC

Robots.txt Generator

Build a correct robots.txt with presets, custom rules, AI-bot blocking and sitemap line — download ready.

806 views

What robots.txt Actually Does

It asks compliant crawlers not to fetch certain paths — saving crawl budget and keeping admin/search/filter URLs out of the queue. Common myths corrected: it is not access control (the file is public and bad bots ignore it), and blocking a URL does not remove it from the index — use noindex for that; a blocked page can still be indexed from external links, just without content.

The tool covers today's real decisions: standard disallows, WordPress preset, per-bot rules, the sitemap line, and the modern question — whether to block AI training crawlers (GPTBot, CCBot, Google-Extended, etc.). Blocking them keeps content out of future model training but also out of AI answers that might cite you; both choices are legitimate.

Frequently Asked Questions

Where does the file go?

Exactly at the domain root: example.com/robots.txt. Subdirectory copies are ignored; each subdomain needs its own.

Should I block AI bots?

Trade-off: blocking (GPTBot & friends) protects content from training sets but forfeits visibility inside AI assistants. Publishers split roughly evenly — decide by whether AI referrals matter to you.

Similar Tools

Meta Tag Generator (Title & Description) Write your title and description with live Google-preview and length guides — copy clean HTML meta tags. Generate Open Graph & Twitter Card Generator Generate og: and twitter: meta tags with a live share-card preview — control how links look on social platforms. Generate Google Search Result Preview (SERP Snippet) See how your title, URL and meta description will look in Google search results before you publish — with live character/pixel truncation warnings. Run URL Slug Generator Turn any title into a clean URL slug — accents transliterated, punctuation stripped, multi-line batch supported. Generate