Best tool to check if my website is accessible to AI crawlers: how we test client sites
How we check whether a client's website is accessible to AI crawlers: robots.txt review, readiness checks, and IP-verified logs that prove what bots actually get.
"Can AI crawlers read our site?" is the first technical question on most of our engagements, and the answer is more often "partly" than "yes". A site can look open on paper and still turn bots away in practice. Here is how we test, and the tools we use at each step.
Why a robots.txt check is not enough
Permission and access are different things. Robots.txt says what you would like bots to do. A CDN challenge, a bot-scoring rule, a rate limit or a JavaScript-only page decides what actually happens. We have seen sites where robots.txt welcomed every AI crawler while the firewall quietly returned errors to the lot. So we test in layers.
Layer one: read the rules
We review robots.txt against the bots that matter: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User, Claude-SearchBot, PerplexityBot, Perplexity-User and Google-Agent. They do different jobs, so we make a deliberate choice per bot rather than a blanket allow or block. That is the client's decision, and we explain the trade-offs.
Promptwatch's free tools help here. The AI Bots directory lists 440+ crawlers, so we can look up an unfamiliar user agent. The robots.txt and llms.txt generators give a clean starting file. The Agentic Readiness Check is a fast outside-in pass that we use early in an engagement.
Layer two: outside-in checks
We fetch key pages the way a bot would and see what comes back. A command-line request with a bot's user agent string is a rough test. It catches simple blocks and misses IP-based or score-based rules, so a pass proves little. We also check whether the key text exists in the server-rendered HTML, because a bot that receives an empty shell cannot use the page.
Foglift's free technical audit is another outside-in option. It covers SEO, AI readiness, performance, security and accessibility.
Layer three: logs, which are the proof
Only logs tell you what bots experienced. We run client programs on Promptwatch for this reason. Agent Analytics provides real-time AI crawler logs that are IP-verified, so spoofed user agents are filtered out. Error tracking explains failures, path filters let us look at a section of the site, and the crawl-to-citation path shows which fetched pages get cited.
Setup depends on the client's stack. Logs flow in via Cloudflare (Logpush or an auto-deployed Worker), AWS CloudFront, Fastly, Vercel, Netlify, Akamai, Google Cloud CDN, a custom HTTP route or the WordPress plugin. Crawler logs start on Professional ($245 a month), so on Essential clients we rely on layers one and two and tell them what that cannot prove.
Other tools we point to
Cloudflare AI Crawl Control is free on every Cloudflare plan, and it shows which AI crawlers request pages, with status codes and robots.txt violations. It only covers traffic through Cloudflare, and on the Free plan detection depends on user-agent strings.
Known Agents has a free plan and offers agent analytics, referral tracking from AI chats and robots.txt tooling. It does not track prompts.
Seology, Encited and SEO MCP fix problems after you find them. We use them selectively.
Reading the results
We sort findings into three buckets.
- Blocked. A bot is refused or challenged. This is a rule change, and the fastest win.
- Unreadable. The bot gets the page but the useful content is not in the HTML. This is a rendering fix.
- Fetched but not cited. Access is fine. The page needs editorial work, and that sits in a different workstream.
Keeping those apart stops a content team from rewriting pages that were never reachable.
After the fix
We re-check logs a few weeks later. Fixing access removes an obstacle and nothing more. It does not make a model cite a page. We track visibility on a prompt panel as well, and the Action Board holds follow-up tasks, with the Discovery agent suggesting up to five evidence-backed actions per weekly run.
For agencies, every self-serve agency plan includes crawler logs and site audits from Kick-off at $199 a month, so access checks can be part of onboarding for every client.
What we tell clients
An accessible site gives you a chance, not a result. If you want to start with the free checks and move to logs once you know what to look for, Promptwatch has both.