robots.txt
robots.txt is a text file at the root of a website that instructs search engine crawlers which pages or sections to crawl or not crawl, using the Robots Exclusion Protocol.
Reviewed by Alexander Yarovenko · Updated: 2026-09-18
What is robots.txt?
robots.txt is a plain text file placed at the root of a website (e.g. example.com/robots.txt) that tells web crawlers which parts of the site they are allowed or not allowed to crawl. It follows the Robots Exclusion Protocol (REP) — an industry-wide standard supported by Google, Bing, and most other crawlers.
Important: robots.txt controls crawling, not indexing. A page blocked by robots.txt can still appear in search results if other sites link to it. Use noindex to prevent indexing.
robots.txt Syntax
User-agent: * # Applies to all crawlers
Disallow: /admin/ # Block the /admin/ directory
Disallow: /search? # Block all search result URLs
User-agent: Googlebot
Disallow: /staging/ # Block staging only for Googlebot
Sitemap: https://example.com/sitemap.xmlCommon Use Cases
- Block admin panels (
/admin/,/wp-admin/) - Block duplicate parameter URLs (
?sort=,?page=) - Block staging or development environments
- Block internal search result pages
- Declare your sitemap location
Critical Mistakes to Avoid
- Blocking CSS and JavaScript — prevents Google from rendering pages properly
- Accidentally blocking the entire site —
Disallow: /blocks everything - Confusing Disallow with noindex — Disallow only prevents crawling, not indexing
- Blocking your sitemap — counterproductive
Testing robots.txt
Use Google Search Console's robots.txt tester or the URL Inspection tool to verify that Googlebot can access your important pages.
How to use this concept in SEO
robots.txt controls crawling, not guaranteed removal from search. Keep CSS and JavaScript needed for rendering crawlable, and never use robots.txt as protection for private information.
Audit checklist
- Confirm that the implementation matches the page purpose and the user task.
- Check the HTTP response, rendered HTML, canonical, robots rules, and internal links together.
- Use Search Console and analytics to verify the result instead of relying on one crawler signal.
- Test a representative URL on mobile and desktop after deployment.
- Document the expected outcome so regressions can be detected in the next audit.
Practical example
A staging area is protected with authentication. Parameter traps may be disallowed only after confirming that Google can still crawl canonical pages and their resources.
How to validate the result
Record the current crawl, indexation, performance, and traffic state before changing anything. Recheck the affected template and a representative URL after deployment, then monitor Search Console over the following crawl cycle. A technical change is complete only when the live response and Google’s observed state agree.