Crawl Errors That Keep Good Pages Out of AI Answers

When a client's best page never appears in AI answers, the first theory is always content. It needs more depth, a better angle, more authority. Sometimes that is true. In our experience, an unexpectedly large share of the time, the page was simply never read.

The fix lives in the infrastructure, not the editorial calendar. Here are the access problems we run into most, and how we find them.

Start from the log, not the page

Before reading a single paragraph, we look at what the relevant bots did when they asked for the URL. Did they ask at all? What status code did they get? How often?

If the answer is "never", the problem is discovery or policy. If the answer is "yes, with a 403", the problem is a block. If the answer is "yes, with 200 and repeatedly", the page is being read and the problem is somewhere else.

Promptwatch's Agent Analytics gives this per page, with error tracking and a failure explanation attached to each error. That last part saves hours. Instead of "403" we get a reason we can hand to an engineer.

The usual suspects

Edge or firewall rules. A CDN or WAF can block automated traffic by category. These rules are often added for good reasons, such as stopping scrapers or protecting against load, and then forgotten. They affect AI bots because bots look automated. The log shows 403 or challenge responses for specific bots. Fixing it means adding allow rules for verified bots, not turning protection off.

Robots.txt conflicts. A wildcard disallow, a leftover staging rule, or a block on a bot name that was added during a training-data debate. Check each bot name against what you actually want. Note that there is no bot named "ChatGPTBot", but some robots.txt files include one, which does nothing useful.

Rate limits. A site that throttles aggressively can return 429 responses to bots. The bot may back off and fetch less. For sites with many pages, that means slow or partial coverage.

Redirect chains and loops. After migrations, old URLs often redirect through several hops. Some bots give up. We cover migrations in a separate post, but the symptom here is a log with many 301 or 302 responses and few 200s.

Soft 404s and thin error pages. A page returns 200 with a "not found" message, or a template that looks empty. The bot reads it, finds nothing, and moves on. Logs look healthy while the content is not.

Client-side rendering. The HTML a bot receives is a shell, and the real content arrives via JavaScript. Some bots do not run scripts. The log shows a 200 and the citation never comes. We check by viewing the raw HTML response.

Login, consent, and geo walls. A cookie wall or country redirect that bots cannot pass leaves them looking at the wall.

Slow responses and timeouts. Pages that take too long to respond can be dropped. Time-to-first-byte matters for bots too.

Canonical and noindex mistakes. A page that canonicalizes to another URL, or carries a noindex directive, tells engines to skip it. This often comes from a template carried over from a staging site.

How we trace a case

Take a hypothetical client whose pricing guide is never cited. Our steps:

Open the crawler data for the URL and filter by the bots tied to ChatGPT Search and Perplexity, such as OAI-SearchBot and PerplexityBot.
If hits are absent, check the internal links and sitemap. A page that nothing links to may not be found.
If hits are present with errors, read the explanation, then check the CDN and robots rules for that bot.
If hits are 200, view the HTML the bot would see and confirm the content is in it.
Fix, then watch the log for a subsequent clean fetch.
Only then look at content.

This order is not romantic, but it prevents the most common waste: a rewrite for a page nobody could read.

Site audits as a second net

Promptwatch also runs weekly Site Health audits with 21 checks, a health score, and a summary of what changed since the last run. Together with Site Content and Site Linking reports and crawlability checks, they catch issues like missing canonicals or broken links that never show up in the bot logs because the bots never reached them. We treat the audit as a weekly smoke test and the crawler logs as the post-mortem.

What to tell the client

Access fixes are usually small and cheap relative to content work, but they often need the client's engineers. We write the ticket carefully:

The URL and the bot.
The status code and explanation from the log.
The rule we believe is responsible.
The change we propose.
How we will verify.

Engineers respond better to a precise ticket than to "AI cannot see our site".

Verify before celebrating

After the fix, we do not declare victory. We wait for a fresh crawl, confirm a 200 in the log, and then watch citations over the next weeks. Many pages begin to appear once the access barrier is gone. Some do not, and then the content question is fair.

Prevention

We add crawler checks to the client's release process. Any change to CDN rules, robots.txt, redirects, or templates triggers a quick look at the logs a few days later. It is cheaper than finding out a quarter later.

Our recommendation

When a page is missing from AI answers, read the crawler log before you read the page. We do this in Promptwatch with Agent Analytics for the errors and the weekly Site Health audit as a second check. Crawler logs start on the Professional plan, so plan for that. See the platform at promptwatch.com.