Testing AI Search Retrieval: A Practical Protocol for Claims and Citations

A citation is not a receipt. It tells you a system pointed at your page, not that it read the page correctly, not that the passage it used says what the answer claims, and not that the answer is true. Most audits stop at the citation and call it a win.

I have watched this failure mode repeat across content teams: someone queries an answer engine, sees the domain cited, screenshots it, and reports success. Nobody opens the cited passage. Nobody checks whether the number in the answer matches the number on the page. The audit measured the easiest observable signal and quietly assumed it stood in for the two harder ones.

This protocol separates three questions that most audits collapse into one: did the system cite your content, does the cited passage actually support the answer, and is the answer correct. Each is independently testable. Each fails in a different way. A test that measures only one of them will produce confident, wrong conclusions.

Four Variables, Only Three of Them Observable

Before any querying, fix the vocabulary. The rest of this protocol depends on keeping four variables apart, and on being honest about which ones you can actually see.

Retrieval asks whether your content was surfaced inside the system's pipeline for a query where it was relevant. You cannot observe this directly. You cannot see which chunks were selected, which candidates were ranked, or which passages were retrieved and then discarded during synthesis. Retrieval is the variable everyone wants to measure and the one an external audit cannot score.

Citation presence asks whether the system attributed something to your domain. This is observable. It is also the signal most teams over-trust.

Citation support asks whether the cited passage actually contains the claim the answer attaches to it. This is observable, because you can open the URL and read the passage. It is a different question from citation presence, and it is where most audits find their real failures.

Answer correctness asks whether the answer is true, current, and appropriately scoped — independent of whether you were cited. A system can produce a correct answer without you and a wrong answer with you.

The reason citation presence is not evidence of retrieval quality is simple: a system can cite a source it retrieved poorly. It can pull a passage out of context, attribute a claim the source does not make, or cite your homepage for a fact that lives on a different page. Citation accuracy testing exists precisely because the citation and the support are two different things. And neither one tells you whether retrieval happened at all.

The atomic unit of testing is the claim, not the page and not the domain. A page is a container. A domain is a namespace. A claim is the smallest unit you can verify: a number, a definition, a causal statement, a recommendation. If you test at the page level, you learn almost nothing actionable. If you test at the claim level, every result points at a specific fix.

This assumes you already have a working model of what is observable in AI search — impressions, referrals, and the unobservable exposure of an answer you never see. That measurement framework is background here. The protocol below is what you run once you have decided to stop guessing and start sampling.

State the honest scope up front: you are sampling a non-deterministic system, not certifying it. The goal is not a verdict. The goal is a repeatable procedure that produces evidence you can act on and re-run.

What You Can and Cannot Observe

Set evidence boundaries before the protocol, or you will over-trust your own results.

Directly observable: whether a citation is present, which URL was surfaced, the full answer text, the position of the citation in the answer, and referral traffic if you can attribute it. These are facts you can log.

Inferred, not observed: whether your passage was actually retrieved, which chunk was selected, why a competitor was chosen over you, and how the system ranked candidates internally. You can build hypotheses about these. You cannot read them off the answer. If you have access to retrieval traces or instrumented logs from a system you operate, you can observe more — but that is a different audit than the external one this protocol describes.

Answer systems vary by query phrasing, session state, model version, and time. A single run is an anecdote, not a measurement. This is the single most common source of false confidence in an AI search audit: one clean result, treated as a stable property of the system.

Vendor-reported quality metrics deserve a specific caution. When a platform publishes evaluation numbers — groundedness scores, citation quality rates, answer relevance across large query sets — treat those as vendor claims about the vendor's own system under the vendor's own test conditions. They are useful as a signal about what the vendor considers measurable. They are not an independent benchmark, and they are not a guarantee for your content. Published methodology tells you which dimensions the vendor tracks; it does not tell you how your specific claims will fare.

The decision rule: any conclusion you cannot reproduce across repeated runs should be labeled provisional. Write that label into your records. A provisional finding is still useful — it tells you where to look next — but it should never drive a content investment on its own.

Build the Claim Inventory First

You cannot test claims you have not extracted. This step turns a content library into a testable list before any querying begins.

Pull claims that are specific, checkable, and consequential. Numbers, definitions, causal statements, and recommendations qualify. Marketing language does not. "Our platform is fast" is not a claim you can test. "The median response time is under 200 milliseconds" is.

For each claim, record four fields:

• The exact source passage, quoted verbatim
• The canonical URL where it lives
• The date the claim was published or last verified
• What a correct answer would need to say to count as accurate

That fourth field is the one teams skip, and it is the one that makes scoring possible later. If you do not write down what "correct" looks like before you see the answer, you will rationalize whatever the system produces. For normative or rapidly changing claims, the criterion may need an explicit "not decidable from available evidence" outcome rather than a forced true-or-false.

Mark claims by risk tier. A wrong price, dosage, legal threshold, or safety limit matters more than a wrong stylistic preference. Risk tier determines how much a failure costs you and therefore how much attention it deserves in the audit.

One upstream assumption: claim clarity and source linkage affect retrievability. Content that states claims plainly and links them to evidence is easier to retrieve and verify. Assume that work is done. This protocol tests the outcome, not the input.

Version the inventory. When results change between runs, you need to know whether the system changed or you edited the source. An unversioned inventory makes that question unanswerable.

Design the Query Set

Ad-hoc prompting produces ad-hoc conclusions. A defensible query set is sampled, not improvised.

Cover four query classes:

1. Direct claim lookup — queries that target a specific claim you made, phrased the way a user would ask it.
2. Comparative or evaluative queries — "is X better than Y," "what is the best option for Z." These test whether you appear in judgment-shaped answers.
3. Multi-hop questions — queries that require combining sources. These are where retrieval pipelines decompose a question into subqueries, and where your content may be retrieved but not cited.
4. Queries where you expect to be absent — the negative controls.

Include the queries your audience actually types, not only the queries you wish they typed. Pull them from search console data, support tickets, sales calls, and community questions. The gap between the queries you want to win and the queries people ask is usually where the real findings live.

Negative controls are not optional. A query where your content should not appear, and does not, validates your scoring. A query where your content should not appear, and does, tells you your scoring is picking up false positives. Without controls, you cannot tell a real citation from a coincidental match.

Vary phrasing deliberately. Ask the same question three ways and see whether the answer and citation hold. If results swing on phrasing alone, that instability is itself a finding — and it caps how much any single result can tell you.

Size the set to what you can actually re-run. A small set you repeat beats a large set you run once and abandon. Twenty to fifty queries is a reasonable starting heuristic for many teams, not a standard: tie the number to how much of your high-risk claim surface you need to cover and how much repetition you can sustain. The exact count matters less than the commitment to re-run.

Run the Queries and Record Raw Output

Make the capture step mechanical. If it requires judgment, it will not survive contact with a busy week.

Log for every query:

• The full answer text, copied verbatim
• Every citation, with its position in the answer
• The cited URL for each citation
• The timestamp and the system version or interface you used
• The exact query string, including phrasing

Capture omissions explicitly. A claim that should have been cited and was not is data, not a null result. Record it in the same structure as a citation. The absence is the finding.

Record the answer's disposition: refused, partially answered, or answered without any citation. Each implies a different interpretation. A refusal is not the same as a wrong answer, and an uncited answer is not the same as an unsupported one.

Standardize the capture format before you start. This is where audits die — teams collect screenshots for three weeks, then try to retrofit structure onto images and give up. A spreadsheet with fixed columns, filled in during the run, is worth more than a perfect schema designed afterward.

One structural note: multi-turn and agentic retrieval pipelines may decompose a single query into several subqueries. A modern retrieval system can break a complex question into focused subqueries, run them in parallel, and synthesize the results. That means one prompt you type can produce several retrieval events you never see. You are observing the output of a pipeline, not a single lookup. This is a limit on interpretation, not a reason to skip the test.

Score Three Observable Outputs Separately

This is the analytical core. Three scores, computed independently — and none of them is a retrieval score.

Citation presence rate. For each query where your content was relevant, was it cited at all? Binary per query, aggregated as a rate. This is a proxy for source visibility, not a measurement of retrieval. It cannot distinguish "retrieved and cited" from "retrieved and dropped" from "never retrieved." Track it because it is easy and directional, not because it answers the retrieval question.

Citation support score. For each citation, does the cited passage actually contain the claim the answer attributes to it? Open the URL. Read the passage. Compare it to the sentence in the answer. This is where most audits find their real failures — not in missing citations, but in citations that do not support what they are attached to.

Answer correctness score. Is the answer true, current, and appropriately scoped, independent of whether you were cited? Score this against the "what a correct answer would need to say" field you wrote in the inventory. A correct answer from a competitor is still a correct answer. A wrong answer that cites you is still wrong.

Lay citation presence against answer correctness and you get four cells. Citation support is a separate annotation on the cited cells, not a third axis:

	Answer correct	Answer wrong
Cited	You won the citation and the answer is right. Check citation support before protecting this.	The answer is wrong and it cites you. Diagnose before fixing.
Not cited	The answer is right but you were not part of it. Source-selection or coverage hypothesis.	The answer is wrong and you were absent. Escalate only if the claim is high-risk.

Each cell implies a different next step, and the steps are diagnostic, not prescriptive. Cited-but-unsupported is not automatically a content-clarity problem. It can be a system attribution error, a synthesis error, a scope mismatch between your passage and the question, or an outdated source. Before you rewrite anything, verify the passage scope and freshness, then decide whether the source is genuinely ambiguous or the system misread a clear source. Only the first case is yours to fix.

Uncited-but-correct points at source selection or coverage — the system answered well without you. That is a competitive or retrieval hypothesis, not a content-accuracy problem, and it may be outside your control. Cited-and-correct is the outcome you are trying to reproduce. Uncited-and-wrong is the cell most teams over-invest in, because it feels like an injustice; for low-risk claims it is usually the least actionable, but for high-risk claims it can be the most urgent.

The error to avoid: treating a citation as a correctness signal. A fluent answer with a plausible-looking citation can still be wrong. Fluency and citation presence are presentation. Support and accuracy are substance. Score them apart.

Where This Protocol Breaks

Name the failure modes before you generalize from your results.

Non-determinism. Identical queries can return different answers and citations across runs. Single-run conclusions are unreliable. If you cannot reproduce a result, do not build a strategy on it.

Personalization and session context. Your results may not represent what other users see. Location, history, and session state can shift both the answer and the sources. Your audit is a sample of your conditions, not a census of the system.

Judge-based scoring is model-dependent. Automated scoring of groundedness and citation quality typically uses a language model as a judge, often by decomposing answers into atomic factual claims and checking which are supported by retrieved content. That approach is useful and it scales, but the judge is itself a model with its own error rate. Treat automated scores as directional, not authoritative. Spot-check them by hand.

Small samples cannot support large claims. Twenty queries do not tell you how a whole content library performs, and they certainly do not tell you how a whole market behaves. Report your sample size alongside every rate.

Absence of a citation does not prove absence of retrieval. Your content may have been retrieved and discarded during synthesis. You cannot distinguish "never retrieved" from "retrieved and dropped" from the outside. This is the hardest limit in the protocol, and it is why citation presence is a proxy, not a retrieval measurement.

Turn the Audit Into a Recurring Loop

A one-off test is a story. A recurring loop is an instrument.

Re-run the same query set on a fixed schedule and diff the results rather than re-scoring from scratch. The diff is where the signal lives: what changed, when, and in which direction. Pick the cadence from your update frequency, how often the systems you track change, and how much a stale finding would cost you. Weekly is a common default for fast-moving topics; slower topics can justify a slower loop.

Track three trends over time — citation presence rate, citation support rate, and answer correctness rate. They move independently. Citation presence can rise while support rate falls, which looks like progress and is actually the opposite. Watching them as separate lines is the whole point of separating them.

Route each failure type to the right next step, conditioned on claim risk:

• Cited but unsupported → diagnose first. Check passage scope, freshness, and whether the source is genuinely ambiguous. Rewrite only when the source itself is unclear or incomplete.
• Uncited but correct → investigate as a source-selection or coverage hypothesis. This may be a content problem, a competitive problem, or a system behavior outside your control.
• Uncited and wrong → escalate when the claim is high-risk or the wrong answer could materially harm users; otherwise monitor. You cannot fix a system's coverage of a topic you do not own, but you can decide whether silence on a dangerous claim is acceptable.

Keep the raw logs. They are the only way to answer "when did this change" six months from now, and that question will come up.

Decide in advance what result would change your content investment. If no possible outcome would alter what you publish, fix, or stop, the audit is a report, not a decision tool. Write the decision rule down before the first run.

What to Learn Next

Build the minimum viable version first: a spreadsheet, a fixed query list, and a recurring re-run. Do that before any dashboard, any scoring automation, or any vendor tool. The dashboard is a scaling decision; the spreadsheet is a learning decision. Most teams need the second one and buy the first.

The skills worth acquiring are claim extraction, evaluation design, and enough retrieval literacy to know which stage failed. Claim extraction teaches you to see your own content as testable assertions. Evaluation design teaches you to separate what you measured from what you assumed. Retrieval literacy — understanding how a pipeline chunks, retrieves, ranks, and synthesizes — is what lets you read a failure and guess where it happened.

If you want to go deeper, the productive directions are retrieval architecture, citation failure modes, and how answer systems handle uncertainty. Each one sharpens a different part of the audit.

Start with the leverage question: which single claim, if consistently misrepresented, would cost you the most? That claim is where the audit begins. Run the smallest reproducible test you can sustain, score citation presence, citation support, and answer correctness on separate axes, and treat every single-run result as a hypothesis until it survives a second run.

The audit is worth doing only if a result would change what you publish, fix, or stop investing in. If it would not, you have already answered the question — and you did not need the spreadsheet.