Robots.txt Generator
Create a perfect robots.txt file — control how search engines crawl your site
Quick Presets
Global Settings
Site URL, sitemaps, crawl delay & options
seconds (0 = disabled)
Applies as global Crawl-delay directive in each block that doesn’t override it.
User-Agent Blocks (0)
File Health
Common Bots Quick-Add
Click to add a pre-configured block for that bot.
Statistics
Generated robots.txt