Robots.txt Guide for SEO

Learn robots.txt syntax, best practices, and how to test URL blocking with our free generator.

Robots.txt tells crawlers which pages to visit and which to skip. A good file protects crawl budget. Use our Robots.txt Generator to create one.

Basic Syntax

User-agent: *
Disallow: /admin/
Sitemap: https://example.com/sitemap.xml

Admin panels, search results, staging sites. Never block CSS or JS files.

Use our Robots.txt Tester or Google Search Console.

Blocking CSS/JS, using Disallow: / for everything, forgetting sitemap location.

Block low-value paths, test rules, add sitemap. Robots.txt is not a security tool.

Generate your robots.txt with our free tool.