Robots.txt Generator — Control Search Engine Crawling Easily
Free online robots.txt generator to create robots.txt files that control how search engines crawl and index your website. Add allow and disallow rules for specific paths, set crawl delay, specify sitemap location, and add rules for individual user-agents like Googlebot and Bingbot.
Quick Presets
Crawl Rules
*
0 disallow · 0 allow
No explicit allows — optional override
Note: Googlebot ignores crawl-delay
Sitemaps
No sitemaps added — recommended to include your sitemap URL
Host (optional — for Yandex)
# robots.txt
# Generated: 2026-06-15
User-agent: *
How to Use Robots.txt Generator
› Always include a User-agent directive before adding rules
› Use * for rules that apply to all crawlers
› Test your robots.txt in Google Search Console
› Remember to upload the file to your root directory
Upload robots.txt to your website root at https://yourdomain.com/robots.txt. Changes take effect on next crawl.
How to Use Robots.txt Generator
Add User-Agent
Specify the search engine crawler (e.g., Googlebot, Bingbot, or * for all)
Set Crawl Rules
Add Disallow and Allow rules to control which paths each crawler can access
Add Sitemaps
Include your sitemap URLs so crawlers can discover all your pages
Download or Copy
Download the robots.txt file and upload it to your website root directory
Frequently Asked Questions
Robots.txt is a text file placed in a website's root directory that instructs search engine crawlers which pages or sections to crawl and which to ignore.
No, robots.txt only prevents crawling, not indexing. If external sites link to a blocked page, it may still appear in search results. Use noindex tags to prevent indexing.
Robots.txt must be placed in the root directory of your website, typically at example.com/robots.txt. It cannot be placed in subdirectories.
While you can set Disallow for all user-agents, malicious crawlers may ignore robots.txt. It is a voluntary protocol and not a security measure.
Try an Example
Sample robots.txt content
User-agent: * Allow: / User-agent: GPTBot Disallow: / Sitemap: https://example.com/sitemap.xml
3 directives · 2 user-agents · 1 sitemap reference
✓ Valid robots.txt — allows all crawlers except GPTBot
Stop searching.
Start doing.
Everything you need is already here — free, fast, and ready to use. No accounts. No limits. No excuses.