Manchester & the North West hello@whatisseo.co.uk
Free SEO Audit
Home / Free SEO Tools / AI Crawler Access Checker
AI & Structured Data

AI Crawler Access Checker

Send a real request as GPTBot, ClaudeBot, PerplexityBot and Googlebot, and see what your server actually returns.

Free, no signup. We fetch the page once and read what your server sends back.

Fetching the page…

What this checks

This is the check almost nothing else performs, and it finds real problems. Reading robots.txt tells you what you have permitted. It does not tell you what your CDN or firewall does when an AI crawler actually knocks. We have tested live sites that allow AI bots in robots.txt while returning 403 at the edge, with the owners entirely unaware.

  • A live request sent with each bot's real user agent
  • The status code your server returns to each
  • What robots.txt says about that bot
  • Where the two disagree

How to act on the result

  • A 403 with an allow in robots.txt means your CDN or WAF is blocking. Fix it in the firewall, not the file.
  • Cloudflare, AWS WAF and several security plugins block AI bots by default.
  • Blocking is a legitimate choice. Blocking by accident, while trying to be recommended, is not.

Frequently asked

Why does robots.txt not tell me this?

robots.txt is a request you publish. It is honoured voluntarily and it has no bearing on whether your server, CDN or firewall actually serves the request. Those are separate layers, and they frequently disagree.

Should I allow AI crawlers?

It depends on your business. Publishers whose product is the content often block them. Businesses that want to be recommended by ChatGPT or Perplexity almost always should not. What you should avoid is blocking them by accident.

Which bots does this test?

GPTBot (OpenAI), ClaudeBot (Anthropic), PerplexityBot and Googlebot as a control. If Googlebot is blocked too, you have a much larger problem.

Will being crawled get me recommended?

Access is necessary but not sufficient. Being cited also depends on what the wider web says about you and how cleanly your content can be extracted. Access is simply the prerequisite people miss.

Want all 40 checks at once?

Run the full free report, or have us go through it with you on a call.