An AI product can score well and still fail the person using it.
The answer may be accurate but arrive too late. A recommendation may be reasonable but impossible to verify. A draft may save typing while creating more review work. An average score may hide a small category of failures that destroys trust.
Product evaluation must connect model behavior to human behavior and real-world operation.
Use this when
- A team has a working AI prototype and is deciding whether it is ready for a pilot or launch.
- “Quality” is discussed as one average score.
- Generic benchmarks are being treated as proof of product value.
- The team cannot explain which failures matter most or what happens after launch.
The outcome
By the end, you will have a living evaluation plan with a baseline, realistic task set, severity-weighted failure taxonomy, release thresholds, human-outcome measures, and a production learning loop.
What you need
- Inputs: Realistic examples, the current workflow, known user segments, model traces or outputs, latency and cost estimates.
- People: A product owner, domain reviewers, an engineering or data partner, and someone accountable for release risk.
- Timebox: One focused workshop to design the plan, then recurring evaluation runs.
- Tooling: A versioned task set, a shared rubric, and a simple place to record results. Automate only after the judgment is clear.
The playbook
1. Define the decision being evaluated
State what the complete product should help a person do—not what the model should generate.
Compare:
- “Produce a good customer-feedback summary.”
- “Help a product manager identify which feedback themes deserve follow-up, with enough evidence to verify the recommendation.”
The second statement tells the team what behavior, evidence, and outcome to inspect.
Output: One product decision and the person who makes it.
2. Establish the current baseline
The alternative to AI is usually not perfection. It is the workflow people use today.
Measure what is practical:
- Time to complete the task.
- Error or rework rate.
- Escalation and abandonment.
- Confidence and verification effort.
- Cost and operational burden.
- Downstream outcome, when observable.
An AI system should earn its place against that baseline.
Output: A small baseline scorecard for the current workflow.
3. Build a severity-weighted failure taxonomy
List the ways the product can fail, then separate them by consequence.
- Low: Cosmetic or easy to correct.
- Medium: Creates rework or confusion.
- High: Changes an important decision or undermines trust.
- Critical: Creates material harm, violates a hard boundary, or cannot be safely recovered.
Do not allow frequent harmless errors to cancel out a rare critical one inside an average.
Output: Named failure categories, severity, examples, and an owner.
4. Assemble a representative task set
Include more than polished happy paths.
Your set should cover:
- Ordinary high-volume cases.
- Important user or market segments.
- Ambiguous and incomplete inputs.
- Long-tail conditions.
- Known historical failures.
- Inputs designed to test the product boundaries.
Keep a holdout set so repeated tuning does not quietly teach the team to the test.
Output: A versioned task set with source, segment, and difficulty labels.
5. Evaluate three layers
The answer: Is the output correct, relevant, grounded, complete enough, and appropriately uncertain?
The person: Does it improve the decision, save meaningful time, reduce effort, increase confidence appropriately, or change follow-through?
The product: Is it reliable, fast, affordable, observable, and recoverable in the real workflow?
All three matter. A strong answer inside a fragile product is not ready.
Output: Measures and review methods for answer, person, and product.
6. Set release thresholds before reading the result
Write down what must be true for:
- An internal test.
- A limited pilot.
- A broader launch.
- More responsibility or automation.
Use slice-level gates for important segments and critical failures. “The average improved” is not enough when one high-consequence context regressed.
Output: Release, expansion, rollback, and stop thresholds.
7. Turn production evidence into new tests
The evaluation set should learn from reality.
Capture meaningful corrections, overrides, escalations, incidents, and unexpected inputs. Review them on a defined cadence. Add durable examples to the task set and revise the taxonomy when a new kind of failure appears.
Avoid adding every unusual case. Add the cases that teach the product something important.
Output: A feedback intake, review owner, and evaluation refresh cadence.
Worked example
Scenario: An AI system groups customer feedback and recommends themes for product review.
The baseline is the existing manual review: time spent, themes found, evidence attached, and which themes lead to follow-up.
The evaluation set includes common feedback, regional language, mixed sentiment, very short messages, conflicting requests, and feedback that mentions a serious issue indirectly. A high-severity failure is inventing a theme without supporting feedback. Another is burying a low-volume but high-impact signal.
The team evaluates theme quality, whether product managers verify the cited evidence, time saved after review, corrections, missed important signals, latency, and cost. The pilot proceeds only if useful coverage improves without exceeding the threshold for unsupported or missed high-impact themes.
Copy the evaluation card
- Decision or task:
- User and context:
- Current baseline:
- Representative input:
- Expected useful behavior:
- Unacceptable failure:
- Failure severity: Low / Medium / High / Critical
- Review method: Expert / rubric / automated check / observed outcome
- Human action to observe:
- Release threshold:
- Segment-specific threshold:
- Production signal:
- Owner and review cadence:
Pre-launch quality check
- Representative real-world tasks are included.
- Critical failures have their own thresholds.
- Human reviewers use a shared rubric.
- The existing workflow is measured as a baseline.
- Model output and user behavior are evaluated together.
- Cost and latency are tested at plausible volume.
- Launch, rollback, and expansion rules are written down.
- Production corrections can become future tests.
Common failure modes
- Reporting only one aggregate accuracy score.
- Testing polished happy paths.
- Treating a generic benchmark as product evaluation.
- Using an AI judge without calibration or expert review.
- Moving thresholds after seeing the results.
- Ignoring important segment-level failures.
- Allowing the evaluation set to become stale.
What this does not solve
An evaluation plan does not remove judgment or risk. High-consequence applications still require appropriate domain, safety, security, privacy, and legal expertise. Some outcomes take time to observe. The plan makes those limits visible instead of hiding them behind one number.
Take it with you
Evaluate the answer, the person, and the product—and never let an average hide the failure that matters.