⚙Robots.txt Generator
Build a robots.txt file with presets, custom allow and disallow paths, and a sitemap line.
Save as robots.txt in your site root. Blocking a page here does not remove it from search - use a noindex tag for that.
Robots.txt does not keep pages out of Google. It stops crawling, which is a different thing, and confusing the two is how pages end up indexed with no description under a note saying no information is available.
If a URL is blocked here but another site links to it, Google can still list it — it simply cannot see what is on it. And if you have put a noindex tag on that page, blocking it in robots.txt guarantees Google never reads the tag. The two work against each other.
To keep a page out of the index: let it be crawled, serve noindex. To stop a crawler wasting time on a directory: use this file.
The presets
The WordPress preset blocks the admin and search paths but keeps admin-ajax.php reachable. That exception matters — blocking it breaks front-end features on a surprising number of sites, because plugins route legitimate public requests through it.
Block everything is for staging sites, and it is worth saying that it is not a security measure either.
What the directives mean
User-agent names which crawler a block applies to; an asterisk means all of them. Disallow lists paths not to request, and an empty Disallow allows everything. Allow carves an exception out of a broader Disallow. Sitemap is independent of the rest and can sit anywhere in the file.
Order matters less than people assume. Google applies the most specific matching rule, not the first one it encounters.
Two practical notes
The file is public. Anyone can read it, so a Disallow line is effectively an announcement of the paths you would rather people did not visit. Use authentication for anything genuinely private.
And it only works at the domain root. A robots.txt sitting in a subdirectory is ignored completely.
This tool is provided free of charge for general information and convenience only. It runs in your browser and we do not store the data you enter. We make no guarantee as to the accuracy, completeness, or suitability of any result, and accept no liability for any loss, damage, or decision arising from its use — use it at your own risk. See our Terms & Conditions.