The Mechanics
How AI answer engines
decide who to cite
Every AI answer is built in stages — retrieve candidates, select trustworthy sources, synthesize, corroborate. Each stage silently eliminates sites. Once you see the pipeline, AEO stops being a mystery and becomes a checklist.
The four stages of every answer
1 · Retrieval
The engine turns the user's question into search queries and pulls a shortlist of candidate pages — usually via a traditional search index, sometimes its own crawl. If AI crawlers are blocked from your site, you drop out here, silently, before any judging happens.
Rewards: crawler access, indexable HTML, SEO fundamentals, llms.txt and sitemaps.
2 · Selection
The shortlist gets read and filtered. The engine keeps sources it can parse and verify: clear page structure, schema.org markup that says what the business is, consistent facts, visible dates and authorship. Ambiguous or contradictory sources are cheap to discard — there are always others.
Rewards: structured data, semantic headings, entity consistency, freshness.
3 · Synthesis & citation
The model composes one answer from the surviving sources and attributes claims to a handful of them. Passages that state the answer directly — a definition, a price, a recommendation with reasons — get quoted. Everything vaguer becomes unattributed background, or nothing.
Rewards: direct answers up front, question-based headings, FAQ coverage, liftable facts.
4 · Corroboration
Engines cross-check. When your site, your directory profiles, your reviews, and third-party mentions all agree about who you are and what you do, citing you is safe. When they conflict, the engine hedges — or recommends the competitor whose story checks out.
Rewards: consistent NAP/entity data, reviews, real third-party mentions.
Platform by platform
The engines differ in where they retrieve from — which changes nothing about the fundamentals and everything about who you need to let in:
ChatGPT + ChatGPT search
Blends model knowledge with live web search. Its crawlers (GPTBot for training, OAI-SearchBot / ChatGPT-User for search and browsing) must be allowed separately in robots.txt — many sites block one without realizing it affects the other.
Perplexity
Retrieval-first: every answer is built from live sources with prominent citations. The most transparent engine to monitor — ask it your customers' questions and you can see exactly who it cites, and who it doesn't.
Google AI Overviews & AI Mode
Sits on top of Google's index, so classic SEO carries the most weight here — but the Overview cites far fewer sources than a results page shows links. Google-Extended in robots.txt controls Gemini training, not regular indexing.
Microsoft Copilot
Built on Bing's index plus OpenAI models. Bing SEO — often neglected — directly feeds Copilot answers, and Bing's crawler must be able to see your content.
Claude, Grok & Meta AI
Claude browses with ClaudeBot / Claude-User; Grok runs live web and X search with xAI's crawlers; Meta AI leans on its own crawlers and partnerships. Different bots, same fundamentals: if the crawler can't read it, the assistant can't cite it.
Practical guides: the AI crawler allowlist · getting cited by ChatGPT · what is llms.txt?
Answer engine FAQ
Do AI answers come from training data or from the live web?
Both, and the mix matters. Models carry general knowledge from training, but for current, specific, or commercial questions — 'best X near me', prices, comparisons — modern engines run live retrieval: they search the web, read a shortlist of pages, and compose the answer from what they just read, with citations. Retrieval is where AEO acts, because it happens fresh on every question.
How does an engine decide which pages to read?
Retrieval usually starts from a search index — the engine issues queries and takes the strongest candidates. That's why SEO fundamentals still matter. From that shortlist, engines favor pages they can access (AI crawler not blocked), parse cleanly, and trust: clear structure, explicit facts, consistent identity, dates, and authorship.
Why do engines cite some pages from the shortlist and skip others?
Because citation requires extraction. The model needs a passage that answers the question directly enough to quote or paraphrase with attribution. Pages that bury the answer in marketing prose, split it across tabs and accordions the crawler can't see, or contradict themselves between pages get read — and then skipped in favor of a source that just says the thing.
Can I pay to appear in AI answers?
Ads are appearing around some AI experiences, but the organic citations inside answers are not for sale on any major platform. They're earned the same way across engines: be crawlable, structured, answerable, and corroborated. That's good news — it means the playing field responds to work, not just budget.
How often do answer engines update what they know about my site?
Retrieval-based engines effectively update as often as they re-crawl — days, sometimes hours. That cuts both ways: fixes show up fast, and regressions do too. It's why we pair remediation with monthly monitoring rather than treating AEO as a one-time project.
Which stage is eliminating your site? The free audit tells you.
Run the Free Audit