Enterprise AI Vendor Evaluation: Questions Beyond the Demo

A demo is a result under conditions the vendor chose. Reliability is what remains when your inputs, your delays, and your failures show up.

Every enterprise AI purchase starts the same way. A polished session, a live agent completing a multi-step task, a chart showing a benchmark lead. The room nods. Someone says "this is exactly what we need."

Then production starts, and the questions change. The retrieval layer misses on the queries your users actually type. A tool call times out and nobody defined what should happen next. A model version changes underneath you and quality drifts for a week before anyone notices. The demo was a controlled performance. The contract is an operating commitment, and the two are not the same artifact.

This article is a set of questions that force a vendor to describe the mechanism behind the outcome, not the outcome alone. I use seven dimensions as an evaluation spine: workload evidence, data controls, portability, incident handling, support, total cost, and exit options. The decision rule up front: score vendors on the evidence they can produce, not the confidence they project.

One boundary rule keeps the rest of this honest. Every requirement below belongs to one of three owners: the vendor controls it, you control it, or you share it. A model provider can commit to change notices; it cannot detect that your retrieval index went stale. A platform can keep an API up; it cannot fix a workflow you wired wrong. I mark the ownership where it matters, because the most expensive procurement mistakes come from assuming a vendor promised something that was always your job.

The Demo Is a Controlled Experiment You Did Not Design

A benchmark score is a result under agreed test conditions. Reliability is what remains when ordinary inputs, missing data, delays, and recoverable failure disturb those conditions.

That distinction matters because vendor demos and leaderboard rankings answer a narrow question: how did this system perform on a task set someone else assembled? Benchmark rank tells you about a test distribution. It says very little about your task distribution, your data shape, your latency budget, or your tolerance for a wrong answer in front of a customer.

Vendor-published case studies deserve the same treatment. They are vendor claims, not independent findings. That does not make them useless. It makes them directional signals. When a vendor cites a deployment, ask what was measured, on what volume, over what period, and who verified the number. A support automation vendor reporting faster resolution on calls that escalate to humans is telling you something real about a specific workflow — but it is not telling you what will happen in your workflow, with your escalation policy, on your call mix.

I have watched teams treat a strong demo as a proxy for production readiness, then spend the first quarter of the contract discovering which assumptions the demo quietly held. The demo assumed clean inputs. The demo assumed the retrieval index was fresh. The demo assumed a human was watching. Production assumes none of that.

So reframe the evaluation task. Stop asking "does it work?" Start asking: under what conditions does it stop working, how would we know, and who pays when it does? Every section below is a way to make a vendor answer that question with specifics instead of adjectives.

Start With Your Workload, Not the Vendor's Feature List

Before you compare vendors, write the workload spec. This is the single highest-leverage hour in the whole process, because it converts a vague aspiration into a testable requirement.

A usable workload spec names the task distribution (what kinds of requests arrive, and in what rough proportion), input variability (how messy the real inputs are), acceptable failure rate (per task type, not overall), latency ceiling, context needs, human review points, and volume trajectory. If you cannot state your acceptable failure rate, you cannot evaluate any vendor's accuracy claim, because accuracy without a threshold is a number without a decision.

One distinction does a lot of work here: single-turn assistance versus multi-step agentic workflows. An agentic workflow is one where the system takes a sequence of actions — calling tools, retrieving data, deciding what to do next — rather than producing one response. These fail differently and are priced differently. A single-turn assistant that is wrong 10% of the time is an annoyance. A multi-step workflow that is wrong 10% of the time at step three may have already written to a downstream system. The blast radius is not comparable, so the evidence bar should not be either.

Define the evidence you will accept before you talk to anyone. Three forms are common: a scoped pilot on your data, a reference deployment in a comparable vertical, or a documented evaluation harness you can inspect. Rank them. A pilot on your data beats a reference call; a reference call beats a slide.

Name your failure modes before the vendor names their strengths. If you let the vendor open with capabilities, the conversation drifts to their best case and you spend the meeting reacting. If you open with "here are the five ways this breaks for us," you learn quickly whether they have seen those failures before or are hearing about them for the first time.

If you have already worked through how to test models against workload fit, treat that as settled input. This section is about turning that testing discipline into a vendor-facing requirement — a document you hand over, not a feeling you carry into the room.

What Counts as Real Workload Evidence

Most vendors will show you a success rate. Fewer will show you the method behind it. The method is where the information lives.

Ask for the evaluation method in plain terms: what was the task set, what was the scoring rubric, who reviewed the outputs, and how were disagreements between model output and ground truth resolved? That last question is the one that separates a real evaluation from a marketing number. If a human reviewer and the model disagreed, and the disagreement was resolved in the model's favor without a documented rule, the success rate is softer than it looks.

Ask what changed between the demo configuration and the production configuration. Demos often run with curated context, a warm index, generous timeouts, and a human operator. Production runs with stale data, cold caches, tight timeouts, and no operator. Someone owns that gap. Find out whether it is you.

Ask for failure behavior, not just success rate. What does the system do when retrieval misses? When a tool times out? When the model's confidence is low? A system that fails loudly and routes to a human is a different product from one that fails silently and produces a confident wrong answer. The second is more expensive, because you pay for the output and then pay again to catch it.

Ask how quality is measured over time as your data distribution shifts. Enterprise query distributions drift as your customer base changes and existing users mature. A one-time evaluation is a snapshot; what you need is a measurement loop you can run yourself, or at least inspect. Note the ownership split: the vendor may supply the harness and the model, but the drift signal usually lives in your logs, on your query mix.

There is a useful research signal here. Multi-dimensional evaluation frameworks — the kind that weight cost, latency, accuracy, and reliability differently depending on the application — are a better model for your scorecard than any single composite number. A framework that lets a customer-facing workflow weight latency heavily while a compliance workflow weights accuracy and traceability heavily is describing how evaluation should actually work. Treat that as method guidance for building your own scorecard. It is not proof of any vendor's performance.

Data Controls, Residency, and the Questions Vendors Answer Vaguely

Compliance checkboxes are where careful evaluation goes to die. A certification badge tells you a process existed at a point in time. It does not tell you where your data goes at runtime.

Trace the data path. What leaves your boundary? What is retained, for how long, and under whose control? Ask for a diagram, not a paragraph. If the vendor cannot produce a data-flow diagram, that is itself an answer.

Then separate the permissions that get bundled into the word "use." Training use, fine-tuning use, logging, and human review access are four different things with four different risk profiles. A vendor may never train on your data but still retain prompts for debugging, and a human reviewer may still be able to read them. Ask about each separately. The vague answer usually appears when these are collapsed into one reassuring sentence.

Ask for tenant isolation, encryption, key management, and audit trail specifics rather than a certification badge alone. Who holds the keys? Can you bring your own? What does the audit trail record, and can you export it? These are mechanism questions, and mechanism questions are hard to answer vaguely without it being obvious.

The governing principle: ask what the vendor can prove versus what the vendor asserts. Request the artifact — the architecture document, the retention schedule, the access-control matrix — not the assurance. Assurance is free to give. Artifacts cost something to produce, which is exactly why they are informative.

One open question worth planning around: retention and review policies change. A policy that is acceptable at signing may not be acceptable after a vendor updates its terms or changes subprocessors. Build periodic re-verification into the relationship rather than treating data governance as a one-time sign-off.

Portability: What You Keep If the Relationship Ends

Assume you have already mapped your switching costs. The question here is narrower: which of those costs will the vendor contractually reduce before you sign?

Separate reversible dependencies from load-bearing ones. Prompts and thin API wrappers are cheap to move — often a weekend of work, though that estimate assumes the wrapper is genuinely thin and not hiding orchestration. Embedded workflow logic, accumulated evaluation data, and fine-tuned artifacts are not. The first category should not drive your decision. The second should.

Ask what the vendor exposes. Model-agnostic interfaces matter because they let you swap the model without rewriting the workflow. Export formats matter because they determine whether your data leaves in a usable shape or a PDF. Evaluation history matters because it is the record of how the system behaved on your tasks, and rebuilding it costs months. And if you fine-tune, ask directly whether the resulting artifact is portable, and in what form.

Ask whether the platform supports multiple models behind one governance layer, and what that actually costs in operational consistency. Multi-model routing sounds like freedom until you discover that each model has different failure modes, different latency profiles, and different evaluation results, and your governance layer now has to reconcile all of them. That is a real cost, not a footnote.

The decision rule: invest in portability proportional to how load-bearing the workflow is, not to how much you like the vendor. An internal research assistant can be tightly coupled. A customer-facing system that touches revenue should be built so that the model is a replaceable part.

Incident Handling, Support, and the Contract Behind the SLA

The relationship is judged on the day the system breaks, not the day it demos. So evaluate the vendor's ability to operate, not just to sell.

Ask for the incident taxonomy. What counts as a severity-one event for a model regression? For a data exposure? For a silent quality drop — the case where the system keeps returning answers but they get worse? That third category is the one most contracts miss, and it is the one that erodes trust fastest, because nothing alerts and nobody owns it. Be explicit about who detects it: the vendor can notify you of a model change, but only your evaluation loop can tell you the change hurt your task.

Ask who responds, with what authority, and how fast — and whether that changes between business hours and off-hours. A support engineer who can acknowledge a ticket is not the same as an engineer who can roll back a model version. Find out which one you are buying.

Ask how quality regressions are detected and communicated when the vendor changes a model version underneath you. Model updates are a normal part of the product. Silent model updates are a governance problem. You want a notification path, a rollback option, and a documented change history.

Ask what the escalation path looks like when the answer is "the model did that" and no one owns the fix. This is the failure mode that turns a vendor relationship sour: the vendor treats the output as probabilistic and therefore not their responsibility, while you treat it as a product defect because it is in front of your customer. Resolve that philosophical gap in the contract, not in the incident channel.

Finally, distinguish support for the platform from support for your workflow. Platform support keeps the API up. Workflow support helps when your retrieval pipeline returns the wrong document for a query type you did not anticipate. The second is usually where the real cost hides, and it is usually not included.

Total Cost: Price Per Token Is the Wrong Number

A lower-priced interaction that produces an unusable output is not good value. A higher-cost workflow that completes a deliverable across multiple steps, with acceptable accuracy and security, can be cheaper overall. Cost per outcome, not cost per call.

That reframe changes what you put in the spreadsheet. Include integration engineering, evaluation harness maintenance, human review labor, retraining when the data shifts, and the internal ownership cost after launch. The last one is the most commonly omitted and the most commonly underestimated: someone inside your company has to own this workflow's quality after the vendor's implementation team leaves.

Ask how usage-based pricing behaves at your volume ceiling. Can administrators forecast, cap, and govern spend at the user, group, or tenant level? Usage-based billing is reasonable, but it is only reasonable if you can see where the meter is and stop it before it runs.

Ask what is bundled versus billed separately, including professional services. Then ask the question that reveals whether you are buying a product or funding delivery labor: what got faster on the last repeat deployment? A credible vendor can name a specific delta — fewer engineering hours, fewer weeks to value, a higher reuse rate — and how it was measured. General claims about "playbooks" and "learnings" are not enough.

This is where the productization lag shows up. If repeat deployments do not get faster, the vendor is accumulating bespoke delivery work rather than turning field learning into reusable capability. That is not automatically disqualifying — some complex deployments legitimately require embedded engineering. But you should know which one you are paying for, because one compounds and the other recurs. Services are fine when you buy them on purpose. They are a problem when you mistake them for product capability.

Exit Options: Write the Divorce Terms Before the Wedding

Exit planning is a pre-signature requirement, not a post-mortem exercise. If you cannot describe your exit in one page, you do not yet understand your dependency.

Ask for data export in a usable format, and be specific about what "data" includes: your raw data, derived artifacts, embeddings, evaluation sets, and configuration. Embeddings matter more than people expect — they are expensive to regenerate and often encode tuning you have forgotten you did.

Ask what happens to your integrations, prompts, and workflow logic at termination, and how long the transition window is. A thirty-day window sounds generous until you try to migrate a production workflow in thirty days.

Ask whether the vendor will commit to a migration assistance clause and what it covers. Some will. Some will offer it as a paid service. Both are acceptable; silence is not.

Then do the honest arithmetic: what would you need to rebuild in-house, and what would that cost? Estimate it before signing, not during the renewal conversation when your leverage is lowest. That number is the real price of the dependency, and it belongs in the total-cost model from the previous section.

A Scorecard You Can Actually Run

Compress everything above into one instrument and run it the same way for every vendor.

Score each vendor across the seven dimensions — workload evidence, data controls, portability, incident handling, support, total cost, exit options. But do not score on vibes. Score on evidence status, using four states:

• Verified — you inspected the artifact or ran the test yourself.
• Demonstrated — the vendor showed a working artifact under conditions you could question.
• Asserted — a claim with no inspectable artifact behind it.
• Unanswered — the question was asked and not answered.

An "asserted" score is not a zero, but it is not evidence either. Treat it as a risk to price, not a box to check. An "unanswered" question is a finding in itself, and it should be recorded with the date it was asked.

Then apply the hard-gate rule. Some requirements are not weighted averages; they are pass/fail. A data-residency violation, a missing export path, or a refusal to commit to change notification can disqualify a vendor regardless of how well it scores elsewhere. Weighted scoring is for tradeoffs. Hard gates are for constraints you cannot trade away. Decide which of your seven dimensions are gates before you score anyone, because deciding after the scores come in is how teams talk themselves into a vendor they already know is wrong.

Record four fields per dimension: the owner (vendor, buyer, or shared), the artifact you accepted, the date you verified it, and the trigger that would force a re-test. That last field is what keeps the scorecard alive. Model version changes, volume growth past a threshold, or a quality regression should reopen the decision rather than being absorbed silently.

Weight the dimensions by workload criticality. A customer-facing workflow weights reliability and latency higher. An internal research tool weights cost and flexibility higher. A compliance workflow weights data controls and audit trail above everything else. The weights are yours; the vendor does not get to set them.

Run the same scorecard on your build-versus-buy option. This is the step teams skip, and it is the one that keeps the comparison honest. If building in-house scores badly on incident handling because you have no on-call rotation for it, that is real information, and it belongs on the same page as the vendor scores.

The Vendor That Wins Is the One That Shows Its Work

The vendor that wins your evaluation is not the one with the best demo. It is the one that can show you the mechanism, the failure path, and the exit — and can do it before you sign, not after.

That is a higher bar than most procurement processes set, and it is the right bar, because the demo was never the hard part. The hard part is the first week of production, when ordinary inputs arrive, a tool times out, and someone has to decide what happens next. A vendor who has thought about that moment will have an answer. A vendor who has not will have a slide.

The practical next step is small and specific: write the workload spec and build the scorecard before your next vendor conversation. Not after the demo, not during the pilot. Before. The document you bring into the room determines whether you are evaluating a vendor or being evaluated by one.