Synthetic Data for Evaluation: Finding the Boundary Before It Misleads You

A synthetic eval set can raise your score without lowering your production error rate. That gap is the whole problem.

The team that ships on a synthetic benchmark and then watches production degrade is not usually careless. They built a generation pipeline, filtered for diversity, checked for leakage, and confirmed the examples were grounded in their own documentation. Every prerequisite test passed. The model still failed on real traffic.

Here is the mechanism behind that failure. A generator conditioned on your documentation, your prompt templates, or your own taxonomy produces fluent, well-formed examples that inherit your framing of the problem. Evaluation exists to find the malformed, the ambiguous, the adversarial, and the failure mode you have not named yet. Those are not the same objective, and the gap between them is where false confidence lives.

If you have already worked through the risk side of synthetic data — diversity, realism, leakage, and downstream performance testing — you have the foundation. This article picks up at the next question, which is harder: which evaluation questions is a generated example allowed to answer, and which ones does it quietly corrupt?

My working rule is short. Synthetic data is a coverage instrument, not a validity instrument. It tells you whether your system has been exercised. It cannot, by itself, tell you whether your system is correct under the conditions that actually matter. Confusing those two jobs is how a green dashboard becomes a false alibi.

The Coverage-Confidence Trap

Coverage and validity sound like the same property. They are not, and the difference decides whether your eval set is evidence or decoration.

Coverage asks a spatial question: does the set touch the input space? Did you generate examples across your intents, formats, languages, and tool-call shapes? Validity asks a representational question: does the set reflect the distribution that actually matters — the one your users produce, with all its mess?

Generation is very good at the first and structurally weak at the second. A generator conditioned on your own materials inherits your blind spots. It may miss the failure mode you have not imagined, because the model producing examples shares your framing of the problem. The eval set becomes a mirror of your assumptions rather than a test of them.

There is a specific symptom worth memorizing, because it is easy to miss in a dashboard that only trends upward:

Eval scores improve as you add synthetic examples, while production error rates stay flat or get worse.

That divergence is the signature. It means the new examples are measuring the generator's fluency, not the system's behavior under real conditions. The score went up because the test got easier, not because the system got better.

The trap is seductive because it feels like progress. More data, broader coverage, higher numbers. But a large, diverse synthetic eval set is not evidence of real-world readiness. It is evidence that you can generate a large, diverse set. Those are different claims, and only one of them is about your model.

What Synthetic Examples Are Actually Good At

I am not arguing against generation. I use it, and the boundary later in this article only earns credibility if I first grant the case where it works. Synthetic evaluation data is genuinely strong in four situations, and the reason is always the same: it is strongest where you can name the axis you want to test.

Rare or expensive conditions. Long-tail intents, malformed inputs, multilingual variants, and edge-case tool-call sequences are hard to source at volume. Real traffic gives you a handful of examples per month; generation gives you hundreds per hour. When the condition is rare but nameable, generation buys you coverage you could not otherwise afford.

Rapid iteration on retrieval and ranking. Grounded question-answer generation with an answerability filter can approximate human-annotated recall behavior closely enough to guide embedding and reranker changes. NVIDIA's NeMo Curator pipeline is a concrete example of this pattern: an LLM generates QA pairs from seed documents, an embedding model filters by difficulty, and an answerability filter removes questions the source material does not actually support. In their reported experiments on the FiQA benchmark, synthetic datasets produced by that pipeline landed within roughly ten percent average deviation in recall at 5 compared to manually annotated datasets. That is a vendor-reported result, not an independent replication, but it illustrates the shape of the case: when the task is "did this change improve ranking," synthetic data can steer the decision.

Cold-start evaluation. Before any production traffic exists, the alternative to synthetic data is no measurement at all. A generated set is a weak instrument, but a weak instrument beats a blank page.

Controllable axes. This is the mechanism underneath all three. Generation is cheap and steerable along dimensions you can specify — domain, difficulty, format, language. It is strongest exactly where you can articulate the axis you want to probe.

Notice what these have in common. In every case, you already know what you are testing. Generation fills in volume along a known dimension. It does not discover the dimension for you.

Where the Boundary Sits: Four Failure Modes

The boundary is set by what the generator can represent and what it can guarantee, not by how much data you generate. Four mechanisms turn synthetic evaluation data into false confidence. Each one is a different way for the same structural problem to surface.

Distribution shift by construction

The generator samples from a model of the world, not the world. That model is smooth where reality is jagged. Rare, messy, or contradictory real inputs get averaged into fluent, well-formed ones, because fluency is what the generator was trained to produce.

This is synthetic data distribution shift, and it is not a bug you can prompt away. The generator has no reliable access to the tail it is smoothing over. If your production traffic contains malformed histories, missing fields, or half-typed requests, a generator conditioned on clean documentation is unlikely to produce them — and your eval set will never test them.

Ambiguity erasure

Real user requests are underspecified, self-contradictory, or dependent on context the system has to infer. "Make it shorter" — shorter than what, for whom, in which format? Generators tend to resolve ambiguity to produce clean examples, because clean examples are what the generation objective rewards.

The consequence is subtle and expensive. Your eval set never tests the disambiguation your system actually has to perform. You measure the model's ability to answer a well-formed question and call it the model's ability to handle users.

Bias laundering

A generator trained on skewed data reproduces the skew in a cleaner-looking form. The output is more polished, more consistent, and more confident than the source — and the eval set then certifies the skew as correct behavior. The bias did not disappear. It got a better suit.

This is the most dangerous of the four, because the cleaned-up version is harder to spot than the original. A messy dataset with an obvious imbalance invites scrutiny. A fluent synthetic set with the same imbalance looks like a standard.

Contamination and self-reference

If the generator, the judge, and the system under test share a model family or prompt lineage, agreement between them measures shared bias, not correctness. The generator produces examples the judge likes, the judge approves outputs the system produces, and the system was tuned on data from the same lineage. Everyone agrees. Nothing was tested.

This is why judge-model agreement is such a weak signal, and why it deserves its own section below.

The reason this is a boundary and not a tuning problem is worth stating plainly: unconstrained generation gives you no guarantee of discovering the failure mode you need to measure, and no guarantee that the failures it does produce resemble real ones. Instruction-following can steer generation toward a named axis. It cannot recover information the generator never had, and it cannot certify that the resulting examples match production. The boundary is about guarantees and provenance, not about the size of the model.

Designing the Mixed Set

Diagnosis is cheap. Construction is where the work happens. The architecture I use separates evaluation by purpose, then assigns each source the job it is actually suited for.

Split coverage slices from validity slices

Coverage slices are synthetic-heavy, cheap, and broad. They answer: has the system been exercised across the input space? Validity slices are real or production-derived, small, expensive, and authoritative. They answer: does the system behave correctly on the distribution that matters?

Keep them separate. Report them separately. A blended score hides which question you answered.

Classify each slice by the claim it is allowed to support

Before you generate anything, name the claim. The claim type determines the evidence source, and the mapping is not negotiable.

• Specified coverage. You want to know whether a named behavior, format, or edge case is exercised. Synthetic-only is acceptable, because the claim is about the constructed contract, not the world.
• Representativeness, prevalence, or drift. You want to know how often something happens, or whether the input distribution has moved. Real or production-derived data is required. A synthetic set cannot detect a shift it was generated inside of.
• Correctness under real conditions. You want to know whether the system behaves well for actual users. Real or production-derived data carries the claim. Synthetic variants may probe the anchor, but the validity claim stays attached to the anchor.

If a slice cannot be classified into one of these three, it does not belong in the report yet.

Never let a synthetic example carry a validity claim on its own

This is the hard rule. A generated example can establish that a behavior is possible under the contract you constructed — a known schema, a specified perturbation, a deliberate edge case. It cannot establish correctness, prevalence, or reliability under real conditions, because the conditions were generated too. Local test validity is not external validity, and the difference is exactly what a synthetic-only slice cannot close.

Stratify by failure mode, not by topic

Topic stratification tells you that you have examples about billing, onboarding, and search. Failure-mode stratification tells you that you have examples probing hallucination under missing context, tool-call loops, and refusal boundaries. The second is what you actually need.

Tag every example with the specific behavior it probes. Then check that each high-risk behavior has at least one real-data anchor. A behavior with only synthetic coverage is a behavior you have not validated.

Use production-derived evaluation where you can

The strongest validity anchor is de-identified real traffic with the final response stripped and re-sampled. OpenAI's production-evaluation pipeline is a documented example of this approach: a representative subset of de-identified traffic, personally identifiable information replaced with synthetic substitutes, the final assistant response removed, and the prefix re-sampled against the model under test. Because the pipeline draws from live traffic, it can be refreshed as behavior drifts — a property a static generated set does not have automatically.

The privacy design is worth noting because it is the part teams get wrong. In that pipeline, the system generating synthetic substitutes never sees the original data. If you build something similar, the de-identification step has to be architecturally separated from the generation step, not just prompt-instructed.

Budget the mix explicitly

Write down the ratio, the source of each slice, and the claim each slice is allowed to support. A mix you cannot describe is a mix you cannot defend in a review. The number matters less than the discipline of naming it.

Judging Synthetic Examples Without Fooling Yourself

Once you have a mixed set, you face a second-order problem: how do you tell whether a generated example is a legitimate test case or a plausible-looking non-test? This is the evaluation-of-the-evaluation problem, and it is where most of the remaining risk lives.

Grounding filters matter more than volume. An answerability or source-grounding check removes questions the source material does not actually support. This is the single most common source of inflated scores, because an ungrounded question has no correct answer — so any confident response looks acceptable. The NVIDIA pipeline's answerability filter exists for exactly this reason: to retain only questions directly supported by the seed document.

Difficulty calibration is a separate axis from correctness. A generated example can be perfectly grounded and still trivially easy. Easy examples make the eval set look strong while testing nothing. Difficulty needs its own measurement, not an assumption that generation produced something hard.

Prefer reasoning-based coverage metrics over embedding distance. Embedding-based cosine distance gives a high-level signal with little actionable detail. It tells you two things are far apart; it does not tell you what is missing. Google Research's Simula work makes this argument directly, proposing taxonomic coverage and calibrated complexity scoring — including LLM-driven batch comparisons that assign Elo-style ratings to individual data points — as alternatives that produce a finer-grained, more actionable view of what a dataset covers and where the gaps are. Treat this as a research signal, not a settled standard, but the critique of embedding-distance metrics is sound.

Watch for judge-model agreement as a false signal. If the same model family generates, filters, and judges, high agreement is expected and uninformative. It measures lineage, not correctness. The fix is structural: use a different model family for judging than for generation, or anchor the judge against human labels.

Run the correlation check. Hold out a small human-labeled slice and measure how well synthetic-set scores predict performance on it. If the correlation is weak, the synthetic set is measuring itself. This is the single most valuable diagnostic in the whole workflow, and it is the one teams skip because it requires human labeling — which is the expensive thing they were trying to avoid.

The uncomfortable implication: the validation loop that keeps synthetic evaluation honest depends on real human judgment. There is no version of this that removes the human from the loop. There is only a version that hides where the human went.

When Synthetic Evaluation Is the Wrong Tool

Some evaluation questions should refuse generation outright. Stating these explicitly is more useful than another caution, because it lets you stop rather than proceed carefully.

Safety and misalignment-adjacent evaluation. These depend on behaviors that are rare, adversarial, and often deliberately hidden. A generator produces what is plausible; misalignment hides in what is implausible. OpenAI's production-evaluation work frames this as "evaluation awareness" — the risk that models behave differently when they know they are being tested — and argues for grounding evaluation in real production traffic rather than constructed sets. Synthetic tests can supplement controlled hazard coverage, but claims about observed behavior and risk should be carried by real or independently grounded evidence.

Regulated or contractual claims. If the evaluation result feeds a compliance statement, a vendor claim, or a customer commitment, the evidence standard is real-world data. A generated set is not sufficient evidence for a claim about real-world behavior. This is a boundary, not a preference.

High-stakes slices where a false pass costs more than collection. The economics flip. When the cost of missing a failure exceeds the cost of collecting real examples, manual collection wins. Run the arithmetic explicitly rather than defaulting to generation because it is cheaper per example.

Any evaluation whose purpose is to detect distribution shift. A synthetic set cannot detect a shift it was generated inside of. If the question is "has the input distribution moved," the answer has to come from production data. This is the cleanest case of the boundary, because the failure is definitional rather than empirical.

A Working Checklist and What to Learn Next

Here is the operational version. It is short on purpose.

Label every eval slice by source and permitted claim. Synthetic or real. Coverage or validity. If a slice cannot name the claim it supports, it does not belong in the report.

Require at least one real-data anchor per high-risk behavior. A behavior with only synthetic coverage is unvalidated, regardless of how many examples it has.

Hold out a human-labeled slice. Measure how well synthetic-set scores predict performance on it. Re-run the correlation after every generator or judge change, because a model swap silently invalidates the previous calibration.

Log the mix ratio alongside every reported score. A score without its mix is not interpretable. The ratio is part of the result.

Build a slice registry. Record provenance, the claim each slice supports, and the last time it was validated against real data. This is the reusable asset. The generated dataset is disposable; the registry is what tells you when a slice has gone stale.

The skills worth building next follow from the registry. Designing stratified eval sets. Writing grounding and answerability filters that actually reject ungrounded examples. Building production-traffic sampling pipelines with de-identification separated from generation. None of these are generation skills. All of them are evaluation skills, and they compound.

The leverage point is worth naming clearly. The reusable asset is not the synthetic dataset — that regenerates in an afternoon and expires just as fast. The reusable asset is the slice registry and the validation loop that tells you when a slice has gone stale. That loop is what keeps the mix honest as models and traffic drift, and it is the thing competitors cannot copy by reading your blog post.

So here is the sentence to carry into your next eval review: a synthetic slice may prove behavior under the contract you constructed, but only an anchored evaluation can support claims about the world that produced the contract. Synthetic data earns its place in the coverage layer. It does not earn one in the validity layer. The work is building the loop that keeps those two layers from quietly merging.