How many test cases does an AI feature need before launch?.

Most AI features are approved the same way: someone tries a dozen examples, the answers look good, and the feature ships. A few weeks later, a customer finds the case that nobody tried. The problem is not that the team skipped testing. It is that a dozen examples cannot tell you much, and it is worth knowing exactly how little.

A score is an estimate

When you run an AI feature on a set of test cases and 90% of the answers are correct, that 90% is not a property of the system. It is an estimate, based on a sample, of how the system will behave on the inputs it meets in production. Like any estimate, it comes with a margin of error, and the margin depends mostly on how many cases you tested.

Figure 1How precise a score is depends on how many cases you test

Margin of error, in percentage points

±0±5±10±15±20±25

±5.9 pts±2.9 pts

02004006008001,000

Number of test cases

Score around 50%Score around 90%

Margin of error by number of cases

Number of test cases	Score around 50%	Score around 90%
20	±22	±13
40	±15	±9
60	±13	±8
80	±11	±7
100	±10	±6
120	±9	±5
140	±8	±5
160	±8	±5
180	±7	±4
200	±7	±4
220	±7	±4
240	±6	±4
260	±6	±4
280	±6	±4
300	±6	±3
320	±5	±3
340	±5	±3
360	±5	±3
380	±5	±3
400	±5	±3
420	±5	±3
440	±5	±3
460	±5	±3
480	±4	±3
500	±4	±3
520	±4	±3
540	±4	±3
560	±4	±2
580	±4	±2
600	±4	±2
620	±4	±2
640	±4	±2
660	±4	±2
680	±4	±2
700	±4	±2
720	±4	±2
740	±4	±2
760	±4	±2
780	±4	±2
800	±3	±2
820	±3	±2
840	±3	±2
860	±3	±2
880	±3	±2
900	±3	±2
920	±3	±2
940	±3	±2
960	±3	±2
980	±3	±2
1,000	±3	±2

With twenty cases, a measured 90% is compatible with a true rate anywhere from the high 70s to 100%. With one hundred cases, the margin narrows to about six points either way. It takes around four hundred cases to get within three points. Because the margin shrinks with the square root of the number of cases, halving it requires four times as many.

This changes how to read a demo. A feature that answers ten questions out of ten has not shown that it is reliable. It has shown that it is not obviously broken.

Comparing two versions is harder

The question teams ask most often is not "how good is it?" but "is the new version better?". A new model, a revised prompt or a different retrieval setup produces a new score, and the temptation is to keep whichever number is higher.

Detecting a difference takes many more cases than measuring a single score. The smaller the improvement, the more cases you need, and the requirement grows quickly.

Figure 2Cases needed to tell two versions apart

90% vs 80%10-point difference197 cases

90% vs 85%5-point difference683 cases

90% vs 88%2-point difference3,839 cases

A ten-point improvement shows up clearly with about two hundred cases per version. A five-point improvement needs nearly seven hundred. A two-point improvement needs several thousand. With a set of fifty cases, most of the differences you observe between two versions are noise, and teams that act on them end up switching back and forth between changes that do nothing.

Two practical consequences follow. First, score every version on the same set of cases, so that the comparison is paired. This removes much of the noise and lowers the number of cases you need. Second, decide in advance what size of improvement is worth acting on, and size the evaluation set for it.

What to put in the set

The number of cases matters, but so does what they cover. An evaluation set made only of easy, typical questions will give a high score that says nothing about the cases that cause incidents. A good set mixes four kinds of cases.

Figure 3What a first evaluation set can look like

Common cases, from real data 100Edge cases 50Past failures 30Out of scope or adversarial 20

• Common cases, from real data. Sampled from actual requests, documents or tickets, so that the score reflects what users really send. Anonymize them first.
• Edge cases. Long inputs, missing information, ambiguous wording, other languages, unusual formats. These are where behavior changes, and where users lose trust.
• Past failures. Every incident and every bad answer reported by a user becomes a test case. This is how the set grows over time, and it guarantees that a fixed problem stays fixed.
• Out-of-scope and adversarial requests. Questions the feature should decline, and attempts to make it reveal data or ignore its instructions. The correct answer here is often a refusal.

How to score answers

Each case needs a way to decide whether the answer is right. Where possible, use checks that code can run: an exact value, a classification label, a valid schema, a citation that points to the right source document. These are fast, cheap and consistent.

For open-ended answers, such as summaries or drafted replies, use a written rubric with a small number of explicit criteria, and have people grade a sample. A second model can apply the same rubric at scale, but only after you have checked, on a few dozen cases, that its grades agree with those of your reviewers. An unchecked automated grader adds its own errors to the measurement.

Make it part of delivery

An evaluation set is most useful when it runs automatically. Run it on every change to the prompt, the model, the retrieval setup or the surrounding code, and block a release when the score drops beyond the margin you have decided matters. Report the score with its margin, not as a bare number, so that nobody celebrates a one-point gain measured on fifty cases.

After launch, keep measuring. Sample production traffic, have it reviewed regularly, and add what you learn to the set. Inputs drift, models are updated, and a feature that scored well at launch can degrade without any change on your side.

Where to start

You do not need thousands of cases on day one. Collect one hundred to two hundred real examples, write down what a correct answer looks like for each, and add the edge cases and failures you already know about. That set will tell you more than any demo, and it will grow with every problem you find. Our applied AI work starts with this step, and you can tell us about the feature you are planning.