robots.txt generator
Create a robots.txt file with allowed and disallowed paths.
robots.txt
User-agent: * Disallow: /admin Disallow: /cart Allow: / Sitemap: https://example.com/sitemap.xml
How to use it
- 1Choose the general policy for your site.
- 2Write the disallowed paths line by line.
- 3Place the output in the robots.txt file at your site's root.
Worked examples
Sample calculation
"Disallow: /admin" means bots won't crawl the admin section.
Frequently asked questions
Where should robots.txt be placed?
At the root of the domain, i.e. example.com/robots.txt
Does Disallow prevent indexing?
Not always; use the noindex meta tag to definitively prevent indexing.