Robots.txt Generator
Build a robots.txt from presets or by hand, see exactly what each rule does, and get warned about the mistakes that quietly block a whole site from being crawled.
One per line, or * for every crawler.
0 omits the line
robots.txt
What this tool does
This builds a robots.txt: the plain-text file at your site root that tells crawlers which paths they may request. Pick a preset or write the rules yourself, and the file is assembled line by line as you type, ready to copy or download.
The limitation that matters most is what robots.txt cannot do. It controls crawling, not indexing. A page you disallow can still appear in search results when other sites link to it, because Google is allowed to index a URL it discovered elsewhere — it simply cannot fetch the page to write a snippet. To keep a page out of the index you need a noindex tag on the page itself, and that page has to stay crawlable for Google to see the tag. Blocking it here prevents exactly that.
How to use it
- Start from a preset if one fits, then edit the fields — choosing Custom leaves them exactly as you left them.
- List one path per line in Disallow and Allow. The order in the file does not matter: crawlers apply the most specific rule that matches, and an Allow that is longer than the matching Disallow wins.
- Add your sitemap URL if you have one. It is a hint rather than a rule, and it goes at the end of the file.
- Read the notes before you upload, then save the file as robots.txt in your site root and open yourdomain.com/robots.txt to confirm it is being served.
Common use cases
-
Blocking a staging site
A copy of the site on a test domain can outrank the real one if crawlers find it. Disallow everything there, and keep the block in place until the domain is retired.
-
An online store
Cart, checkout, order confirmation and internal search results are pages no shopper should ever land on from Google. Blocking them keeps crawlers on the pages that can actually be found.
-
A WordPress site
The WordPress preset covers the usual set: the admin area, the login page, xmlrpc.php and the readme that advertises your version number.
-
Faceted and filtered URLs
Colour, size and sort parameters can generate thousands of near-identical URLs. A wildcard rule on the query string stops crawlers spending the budget on all of them.
Frequently asked questions
Does a Disallow rule remove a page from Google?
No. Robots.txt controls crawling, not indexing. If other sites link to a disallowed URL, Google can still list it, because it learned the address somewhere else — it just cannot fetch the page, so the result shows a bare URL with no title or snippet. To keep a page out of the index, leave it crawlable and put a noindex tag on it. Blocking it in robots.txt stops Google from ever seeing that tag.
Where does the file go?
In your site root, at yourdomain.com/robots.txt, and nowhere else. A robots.txt in a subfolder is ignored, and each subdomain is a separate site with its own file — shop.example.com needs one even if example.com already has it. The file is public, so anything you list in it is a list of paths you would rather people did not look at.
What does User-agent: * cover?
It is the group every crawler follows if it has no group of its own. As soon as you add a group naming a specific crawler, that crawler ignores the wildcard group entirely and follows only its own rules. This is the mistake that takes sites out of search: a group for one bot with a Disallow that the wildcard group was supposed to override. Test the file with Google's robots.txt tester before you upload it.