Use precise rules
Start with User-agent: *, then allow the public site and disallow only paths that should not be crawled. robots.txt is not an access-control system and does not guarantee de-indexing.
Add your sitemap URL and test the file after publishing. Use the robots.txt generator to prepare a starting file.
How crawler rules work
Rules are grouped by user agent. A general group can describe the default policy, while more specific groups can target a particular crawler. Keep the file readable and test every disallow rule against the paths you actually intend to restrict.
Disallowing a path is not the same as removing a URL from search results, and robots.txt should never protect private information. Use authentication for private areas and use appropriate indexing directives where you control the page response.
Common mistakes to avoid
- Blocking CSS or JavaScript needed to render the page.
- Blocking the entire site with a rule copied from a staging environment.
- Using robots.txt as an access-control mechanism.
- Listing a sitemap URL that does not match the live site.
- Forgetting that each path rule should be tested after deployment.
After publishing, inspect the file directly, check important URLs and monitor crawl or indexing reports. Make the smallest rule change that solves the actual crawl problem.