Developer Tools New Tool

Free Online Robots.txt Generator and Rule Tester

Write user-agent groups with Allow and Disallow paths, a crawl delay and your sitemap lines, starting from a preset if you like. Then paste a path into the tester and read which rule wins and why, using the real rule that the longest matching pattern takes precedence.

Free Forever Nothing Uploaded Tests a path against your rules Runs in your browser

Free Online Robots.txt Generator and Rule Tester

Advertisement Slot (Top Banner) Google AdSense Unit • Responsive Banner

Robots.txt Generator Everything happens in this tab. Nothing you paste is sent anywhere.

Rule tester

Path to test As this crawler

Add a rule to see a verdict.

robots.txt

Start from

Sitemaps One full URL per line. These apply to every crawler, wherever they sit in the file.

Buy Us A Coffee

Enjoying WizTools123? Help keep our server infrastructure 100% free and open for everyone.

Buy Us A Coffee

Sponsored Content (Below Tool) Google AdSense Placement

How a Crawler Decides, Step by Step

One group, then the longest match, then Allow on a tie.

First the crawler picks one group, and only one. If the file has a group naming that crawler, it uses that group and ignores the * group completely. So a file with a Googlebot group and a * group does not give Googlebot both sets of rules; it gives it only its own. This is the single most common surprise in robots.txt, and it is why a rule added to the * group sometimes appears to do nothing.

Then, within that group, every Allow and Disallow whose pattern matches the path is collected, and the one with the longest pattern wins. Order in the file is irrelevant. If two matching patterns are the same length, Allow wins. That is how Disallow: /wp-admin/ with Allow: /wp-admin/admin-ajax.php works: the allow is longer, so that one file stays crawlable while the folder does not.

Two wildcards are supported by the big crawlers: * matches any run of characters, and $ at the end of a pattern anchors it to the end of the URL. So Disallow: /*.pdf$ blocks PDFs but not /notes.pdf.html. The tester on this page implements all of this, including the anchoring, so you can check a path rather than reason it out.

How to Write a Robots.txt File

A few steps, and nothing is uploaded.

Start from a preset Allow everything, block everything, a staging site, the usual WordPress rules, or a group blocking the AI crawlers. Then edit it.

Edit the groups Each group names one or more crawlers, separated by commas, and holds its Allow and Disallow paths and an optional crawl delay.

Add your sitemaps One full URL per line. Sitemap lines are not part of any group, so they are written once at the end of the file.

Test a path Type a path and a crawler name into the panel. It tells you which group applies, which rule wins, and why that rule beat the others.

What to Know About Robots.txt

Including why it will not keep a page out of search results.

Robots.txt is not security and it is not a way out of search results. The file is public, anyone can read it, and it asks politely rather than enforcing anything. A page you disallow can still appear in search results if other pages link to it, because the crawler never needs to fetch a page to list its URL. The actual answer is a noindex meta tag or an X-Robots-Tag header, and for that to be read the page must not be blocked in robots.txt. Blocking and noindexing the same URL cancels itself out.

Order in the file means nothing; pattern length does. Crawlers collect every matching rule in the one group that applies and take the longest pattern, with Allow winning a tie. Moving a line higher up changes nothing at all, which is why the tester here reports the winning rule rather than the first match.

Crawl-delay is ignored by Google. Bing, Yandex and several smaller crawlers honour it; Googlebot does not, and adjusts its own rate instead. It is written into the file if you set it, because it does something for the crawlers that read it, but it is not a way to slow Google down.

This page writes a file and tests it; it cannot fetch yours. There is no import, so an existing robots.txt cannot be pasted in for editing, and nothing here checks that the file is reachable at the right address. It must sit at the root of the host, as /robots.txt, and a copy in a subfolder is never read.

Key Features & Capabilities

What this tool does, and what it deliberately does not.

Real user-agent groups Several crawlers per group, each with its own rules.

Five presets Open, closed, staging, WordPress, and the AI crawlers.

A tester that is correct Longest match wins, Allow breaks a tie, with wildcards.

It shows its working Every rule that matched is listed, so you can see the loser too.

Sitemap lines Kept out of the groups, because they are not group scoped.

Nothing is fetched No request is made to your site. The page works offline.

About the Robots.txt Generator

Robots.txt is a plain text file at the root of a host that tells crawlers which paths they may fetch. It is old, it is simple, and it is misunderstood more than almost anything else in web publishing, because the rules for choosing between two conflicting lines are not written anywhere near the file itself.

This page has two halves. The builder writes groups of rules, with presets for the situations that come up constantly: a staging site nobody should index, the two or three WordPress paths worth blocking, and the growing list of crawlers that collect text for language models. The tester then runs a path through the file you just built and reports which rule decided the answer.

The tester was the part worth getting right. It implements group selection properly, so a named crawler ignores the * group entirely, and it implements precedence properly, so the longest matching pattern wins and Allow breaks a tie. It also lists the rules that matched and lost, because when a path is blocked and you did not expect it, the useful information is which line did it.

Frequently Asked Questions

Precedence, wildcards, noindex and where the file must live.

No. It stops the page being fetched, not listed. If other sites link to a blocked URL, Google can index the URL with no description, which is usually worse than indexing it properly. To keep a page out of results, allow crawling and add a noindex meta tag or an X-Robots-Tag header, because a blocked page can never be read to discover the noindex.

No. Within the one group that applies, crawlers take the rule with the longest matching pattern, and Allow wins when two matches are the same length. That is why Allow: /wp-admin/admin-ajax.php beats Disallow: /wp-admin/ no matter which line comes first.

Because a crawler uses exactly one group: the one that names it, if there is one, otherwise *. If your file has a Googlebot group, Googlebot never reads the * group at all, so a rule added there does nothing for it. Repeat the rule in the named group, or remove the named group.

Google, Bing and the other major crawlers support two: * for any run of characters and $ to anchor the pattern to the end of the URL. So /*.pdf$ matches a URL ending in .pdf. These are extensions rather than part of the original standard, so an obscure crawler may treat them literally.

At the root of the host, reachable as https://example.com/robots.txt, and it applies only to that host and protocol. A subdomain needs its own file, and a copy in a subfolder is never read. Keep it under 500 kibibytes, which is where Google stops reading.