SEO tools
Robots.txt Generator
Generate a valid robots.txt with as many user-agent groups as you need. Presets cover the common cases, and the tool warns you when a rule blocks more than you probably meant.
Blocks admin, cart and search pages — the usual thin or private areas.
* means every crawler that has no more specific group.
One path per line. Leave empty to allow everything.
Exceptions carved out of a Disallow. More specific rules win.
Ignored by Google. Bing and Yandex honour it.
One per line. Must be absolute URLs.
Only Yandex uses this. Safe to leave blank.
noindex meta tag.robots.txt
User-agent: *
Disallow: /admin/
Disallow: /cart
Disallow: /checkout
Disallow: /search
Disallow: /*?s=
Disallow: /api/
Sitemap: https://example.com/sitemap.xmlUpload to the root of your domain — it must be reachable at /robots.txt. A file in a subdirectory is ignored entirely.
Crawling is not indexing
Almost every robots.txt mistake traces back to one confusion. These are two separate systems and they are controlled by two different mechanisms.
| Goal | Correct tool | Why |
|---|---|---|
| Save crawl budget on junk URLs | robots.txt Disallow | Stops the fetch before it happens |
| Keep a page out of search results | noindex meta tag | Page must be crawlable for this to be read |
| Hide private data | Authentication | robots.txt is public and advertises the path |
| Consolidate duplicate URLs | rel=canonical | Blocking hides the duplicate signal instead of resolving it |
The third row deserves emphasis. robots.txt is world-readable at a predictable URL, so listing Disallow: /internal-admin-panel/ publishes the existence of that path to anyone curious enough to look. It is a signpost, not a lock.
Related tools
- Sitemap Generator — build the sitemap you reference from this file.
- Canonical Checker — reads the robots meta tag and X-Robots-Tag on a live page.
- Redirect Checker — make sure crawlable URLs are not wasted on redirect chains.
Frequently asked questions
Does robots.txt stop a page appearing in Google?
No, and this is the single most costly misunderstanding in technical SEO. robots.txt controls crawling, not indexing. If other sites link to a blocked URL, Google can still list it — typically with no description, because it was never allowed to read the page. To keep something out of the index you must allow crawling and serve a noindex meta tag.
So blocking a page in robots.txt makes noindex stop working?
Exactly. The noindex tag lives in the page's HTML, and a crawler that is forbidden from fetching the page never sees it. Combining Disallow with noindex is self-defeating: pick one. If the page is already indexed, allow crawling until Google re-reads it and drops it, then block if you still want to.
Where does the file go?
The root of the host, at /robots.txt exactly. It applies per host and per protocol, so https://example.com and https://shop.example.com need separate files. A robots.txt in a subdirectory is ignored completely.
Does Google honour Crawl-delay?
No. Google ignores Crawl-delay entirely; crawl rate is managed in Search Console. Bing and Yandex do honour it, so the directive is not useless — it just does nothing for the crawler most people are worried about.
Can I block ChatGPT and other AI crawlers?
You can block the ones that publish a user-agent and choose to obey robots.txt — GPTBot, ClaudeBot, Google-Extended, CCBot and PerplexityBot among them. Compliance is voluntary. Blocking here is a request, not an enforcement mechanism, and it does nothing against a scraper that ignores the file.
How do Allow and Disallow interact?
The most specific matching rule wins, judged by path length, not by the order the lines appear. Disallow: /wp-admin/ with Allow: /wp-admin/admin-ajax.php blocks the directory but keeps that one file reachable, because the Allow path is longer. If two rules are equally specific, the least restrictive wins.