robots.txt generator
Create a robots.txt file with allowed and disallowed paths.
User-agent: * Disallow: /admin Disallow: /cart Allow: / Sitemap: https://example.com/sitemap.xml
About the robots.txt generator
The robots.txt file is the very first thing search engine crawlers read before they crawl your site, and it tells them which sections are open to crawling and which should be left alone. Writing this file by hand, with its specific syntax like User-agent, Disallow and Allow, confuses many users, and one small mistake can accidentally hide your whole site from Google. Abzario's robots.txt generator produces a standard, ready file from a few simple choices — such as allowing indexing overall or blocking the whole site, a list of disallowed paths, and your sitemap URL. Just save the output as a text file named robots.txt and upload it to your domain's root. This tool is especially handy for site owners who want to keep crawlers away from admin areas, shopping carts or staging pages, while still correctly pointing crawlers to their sitemap.
How to use it
- 1Choose the general policy for your site.
- 2Write the disallowed paths line by line.
- 3Place the output in the robots.txt file at your site's root.
Why use this tool?
- Quickly builds a standard robots.txt file without learning the manual syntax
- Choose between allowing indexing with exceptions or blocking the whole site
- Add several Disallow paths at once, one per line
- Automatically appends the Sitemap URL at the end of the file for crawlers
- Supports setting a Crawl-delay to control crawl speed on lower-capacity servers
- Copy-ready output that can be uploaded directly to your domain root
Worked examples
Blocking the admin panel
Listing /admin and /wp-admin in the disallowed paths produces lines like "Disallow: /admin" and "Disallow: /wp-admin" so crawlers never enter your admin panel.
Pointing to your sitemap
Entering https://example.com/sitemap.xml in the sitemap field appends "Sitemap: https://example.com/sitemap.xml" to the file so Google can discover all your pages faster.
Blocking the entire site from crawlers
For a site still under development that shouldn't be indexed yet, choosing the "block entire site" option produces "User-agent: *" and "Disallow: /", closing the whole domain to crawlers.
Allowing indexing with a few exceptions
For a store that wants product pages indexed but not the cart or checkout, listing /cart and /checkout alongside "Allow: /" blocks only those two paths.
Frequently asked questions
Where should robots.txt be placed?
At the root of the domain, i.e. example.com/robots.txt
Does Disallow prevent indexing?
Not always; use the noindex meta tag to definitively prevent indexing.
What happens if my site has no robots.txt file at all?
Without this file, crawlers generally treat the whole site as crawlable; missing the file isn't a problem in itself, but you lose the ability to fine-tune what gets crawled.
Can I set different rules for different bots?
Yes, by defining several separate User-agent blocks — for example one for Googlebot and one for everyone else — you can set different rules; this tool produces a base output with one general rule for all bots.
Why does a page I blocked in robots.txt still show up on Google?
Disallow only stops crawling, not indexing; if other pages link to that URL, Google may still show the address without its content. To fully prevent this, use a noindex meta tag instead.
What is Crawl-delay used for?
This directive tells crawlers how many seconds to wait between requests, reducing load on your server; it's useful for high-traffic sites with limited resources, though Google ignores it.