Why isn't my website showing up in AI search results? What we check first
When a client's website isn't showing up in AI search results, we check crawler access, indexing and third-party presence in a fixed order. Here is that order.
"We cannot find ourselves in ChatGPT." It is one of the most common openers in a first client call. The diagnosis follows a pattern, and the order matters, because the cheap causes come first and they are often the whole answer.
We start with whether the bots can get in
Before looking at content, we check whether AI crawlers can read the site. That means robots.txt first. We search for OAI-SearchBot, GPTBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, PerplexityBot, Googlebot and Bingbot. OpenAI's documentation says sites that disallow OAI-SearchBot may not appear in ChatGPT search answers, so a rule someone added to stop training use can also remove a site from search answers. When we find one, we do not just delete it. We ask the client which uses they want to allow, because it is a business decision.
Then we look at the firewall and CDN, which is where less obvious blocking happens. Bot-protection rules block unknown agents regardless of what robots.txt says. We only find these by reading logs.
How we read the logs
We use Agent Analytics in Promptwatch. It shows real-time AI crawler logs, IP-verified so spoofed user agents drop out, with the status of each request and an explanation when one fails. It also shows a citation rate per page. That lets us say something specific: "OAI-SearchBot reached the homepage, but the product pages return errors."
Cost is worth flagging. Crawler logs start on the Professional plan ($245 a month). Essential ($95) does not include them. For a smaller client, we sometimes check access by hand first and move to the logs if the cause is not obvious.
Next, indexing
AI engines that search the live web lean on search indexes. We verify Google Search Console and Bing Webmaster Tools for every client. Copilot draws on Bing, and a brand missing from Bing is easy to overlook. We look for noindex tags, canonical conflicts, thin or orphaned pages, and content that exists only after JavaScript executes. Viewing source rather than the rendered page settles that last one quickly.
Promptwatch's weekly Site Health audit (21 checks, a health score, and a note on what changed) catches many of these without a manual pass.
Then, does the page answer the question?
If bots can read the page and it is indexed, we look at whether it deserves to be quoted. Engines pick pages that state the answer clearly. A page that opens with three paragraphs of preamble is harder to lift than one that gives the answer under a heading that matches the question. We rewrite the top of important pages before we touch anything else.
Then, the rest of the web
The cause that surprises clients most is that answers depend heavily on what other sites say about them. If competitors appear in forum threads, review sites and comparison articles, and the client does not, engines have more material about the competitors.
Citation analytics in Promptwatch show which domains and pages engines cite for each prompt, including Reddit and YouTube reports, and the offsite mentions list shows where a brand is named without a link. That tells us whether to spend effort on the client's own pages or on earning presence elsewhere.
And finally, are they actually missing?
Sometimes the client has been checking by hand. One manual search proves little. Answers vary between runs, by country and by phrasing. We set up a fixed prompt set tagged by topic and intent, and Promptwatch runs it daily per model, country and language. Often we find that the client appears for some prompts and not others, which changes the plan completely.
What we do not rely on
Clients ask about llms.txt. We add one when it costs nothing, but we know of no engine that has committed to requiring it. It is never our fix for blocked crawlers or missing indexing.
Our output
After the diagnosis, we hand the client a short list in priority order:
- Access fixes (robots.txt, firewall rules, server errors).
- Indexing fixes (Bing and Google verification, canonicals, rendering).
- Content fixes on the pages tied to high-value prompts.
- Offsite work to earn mentions where engines already look.
The Action Board helps us keep that list evidence-based. Its Discovery agent suggests actions from the data each week, and we filter and order them.
What we are honest about
Fixing access removes the biggest obstacle. It does not guarantee an engine will name the brand. Answers shift between runs and models update. We promise a clear diagnosis and a measured baseline, not a ranking.
If you are doing this yourself
Spend your first thirty minutes on robots.txt, logs, and Search Console plus Bing Webmaster Tools. Most invisibility problems are solved or ruled out right there. If they are not, set up prompt tracking and look at citations, because the answer is probably not on your own site.