Check your robots.txt like Google. This tool uses the official library.
It’s a replacement for the cancelled tester in the Search Console. Enjoy :).
Continue with robots monitoring
You will get an e-mail alert about every robots.txt change.
Why is this tester special?
Based on Google’s official robots.txt parser
Our tool uses the same open-source library that Google relies on to parse and interpret robots.txt files. This means the results here are as close as possible to what Googlebot would do.
Real Google behaviour
Some Google user-agents, like AdsBot-Google, ignore global User-agent: *
rules. Our tester accurately reflects this behaviour, so you see exactly what Google sees.
Intelligent handling of special characters
Google first decodes the URL (based on RFC 3986) and then applies robots.txt rules following a different standard (RFC 9309). This can cause unexpected results when working with special or encoded characters like commas (,
) or accented letters (é
). This tester handles these cases, helping you avoid mistakes when blocking or allowing URLS.
Proper crawl-delay handling
Google completely ignores the Crawl-delay
directive. If you mistakenly add only a crawl-delay for Googlebot, it will behave as if you wrote nothing, or worse, accidentally allow or disallow things you didn’t intend.