Robots.txt Generator
Robots.txt Generator: Paste your data to get a clean result instantly. Everything runs locally in your browser, so nothing is uploaded.
🌐 Website Information
🤖 Crawler Control
Select which crawlers to configure:
🚫 Path Rules
⏱️ Crawl Delay (Optional)
Set crawl delay for specific bots (seconds between requests):
📄 robots.txt Preview
⚠️ Important: Place your robots.txt file in the root directory of your domain (e.g., https://example.com/robots.txt)
What is a Robots.txt File?
A robots.txt file is a text file located in your website's root directory that tells search engine crawlers which pages or files they can or cannot request from your site.
It's part of the Robots Exclusion Protocol (REP), a group of web standards that regulate how robots crawl and index the web.
While robots.txt is not a security measure (savvy users can still access disallowed pages), it's essential for:
- Preventing duplicate content issues
- Managing crawl budget
- Keeping internal pages private
- Directing crawlers to your sitemap
Basic Syntax
User-agent: * Allow: / Disallow: /admin/ Sitemap: https://example.com/sitemap.xml
Robots.txt Best Practices
Do
- Place in root directory
- Use consistent casing
- Reference your sitemap
- Test with Google Search Console
Don't
- Block CSS or JS files
- Use noindex as a substitute
- Disallow entire site
- Block search result pages
Pro Tips
- Wildcards (*) are supported
- Use $ for end of URL
- Crawl-delay is a suggestion only
- Check logs for bot activity