What is robots.txt and how do you write one?
robots.txt is a small text file on your server root that tells search engine crawlers which parts of your site they are allowed to crawl. Written correctly, it protects private pages and saves crawl budget; written wrongly, it can hide your whole site from Google.
How do robots.txt rules work?
The file uses simple rules. User-agent selects the crawler, Disallow blocks a path, Allow re-opens one, and Sitemap points at your sitemap. An empty Disallow (Disallow: with nothing after it) means everything is allowed.
- User-agent: * — applies to all crawlers.
- Disallow: /admin/ — blocks the admin folder.
- Allow: /admin/open/ — lets one path back in.
- Sitemap: https://example.com/sitemap.xml — announces your sitemap.
Common mistakes that hurt rankings
Blocking CSS and JavaScript files is the most frequent error — Google then renders a broken page. robots.txt also cannot remove a page from the index; use a noindex tag for that. And remember: a broken Disallow line can accidentally block the entire site.
Generate a clean robots.txt with FreetoolsY
The Robots.txt Generator builds the file from your allowed and blocked paths, then lets you copy it in one click. Keep the file small, test it in Search Console afterwards, and combine it with a noindex tag and canonical URLs — together they give you full control over how Google crawls your site.