Skip to content

AI engineering

How to tell whether your LLM feature still works next quarter

Prompts drift, models get deprecated, and your data changes underneath you. An evaluation set is the only thing that tells you before a customer does.

20 November 2026 · 7 min

A feature built on a language model is not finished when it ships. The provider will release a new version, deprecate the one you pinned, and change default behaviour in between. Your own data will shift. Someone will improve a prompt and break two cases they never looked at.

The only practical defence is an evaluation set, and the only time to build it is before the feature.

What an evaluation set actually is

A few hundred real inputs with known correct outputs, drawn from the actual workflow rather than invented. For invoice coding that is real invoices and the account a human assigned. For document triage it is real documents and the queue they belonged in.

The work is in the labelling, and it is worth doing properly: an evaluation set assembled carelessly will give you confident numbers about nothing.

Score more than accuracy

  • Accuracy on the set, broken down by case type rather than averaged into one comforting figure
  • The shape of the failures: a model that is wrong in consistent ways is fixable, one that is wrong randomly is not
  • Cost and latency per case, which decide whether it survives contact with real volume
  • Refusal and escalation rate: a system that correctly declines is often better than one that always answers

Run it on every change

Prompt edits, retrieval changes, model upgrades, a new document type in the corpus — all of them go through the same suite, and the result goes in the pull request. This is ordinary engineering discipline applied to a component that happens to be probabilistic.

If someone cannot show you a number, they are telling you it feels better.

Watch for the quiet failures

The dangerous regressions are not the ones that crash. They are the ones where the system keeps answering, at the same speed, slightly worse. Nobody notices for a month because nothing looks broken. That is precisely the failure mode an evaluation suite exists to catch, and the reason it needs to run on a schedule rather than only when someone remembers.

What this costs

A week or two up front, and a small amount of ongoing attention. Set against the cost of discovering through a customer complaint that a core workflow has been degrading since the last model update, it is among the cheapest insurance in the stack.

Facing this in your own business?

Book a 30-minute call