Robots.txt Tester

Check your robots.txt like Google. This tool uses the official library.
It’s a replacement for the cancelled tester in the Search Console. Enjoy :).

Continue with robots monitoring


You will get an e-mail alert about every robots.txt change.

Why is this tester special?

Based on Google’s official robots.txt parser

Our tool uses the same open-source library that Google relies on to parse and interpret robots.txt files. This means the results here are as close as possible to what Googlebot would do.

Real Google behaviour

Some Google user-agents, like AdsBot-Google, ignore global User-agent: * rules. Our tester accurately reflects this behaviour, so you see exactly what Google sees.

Intelligent handling of special characters

Google first decodes the URL (based on RFC 3986) and then applies robots.txt rules following a different standard (RFC 9309). This can cause unexpected results when working with special or encoded characters like commas (,) or accented letters (é). This tester handles these cases, helping you avoid mistakes when blocking or allowing URLS.

Proper crawl-delay handling

Google completely ignores the Crawl-delay directive. If you mistakenly add only a crawl-delay for Googlebot, it will behave as if you wrote nothing, or worse, accidentally allow or disallow things you didn’t intend.

How useful was this post?

Click on a star to rate it, please.

Average rating 4.8 / 5. Vote count: 26

No votes so far! Be the first to rate this post.

We are sorry that this post was not useful for you.

Let us improve this post.

Tell us how we can improve this post?