AI Testing and Validation
Testing for systems that do not give you the same answer twice.
Pass or fail is the wrong question
Ordinary software is deterministic. The same input gives the same output, so a test that passes today means something tomorrow. AI systems do not work that way, and testing them as though they do produces a green tick that proves almost nothing.
What matters instead is the spread. How often is it right, how badly wrong is it when it is wrong, and where does that sit against what you are willing to accept? Those are numbers, and they can be measured, tracked and argued about. They are also what a demo never shows you.
Related, and often confused with this: production AI covers monitoring and fallbacks once a system is live. This is the work that decides whether it should go live at all.
Four ways we get brought in
Two of these are about something already built, and one is about something bought.
Before an AI feature launches
An independent read on whether it behaves well enough to put in front of customers.
Building the evaluation set
Real cases from your own history, frozen, so a score means the same thing in six months.
Judging a product you are buying
A vendor's benchmark was picked by the vendor. We put the product on your cases instead.
Watching a live system
Behaviour drifts as models, prompts and data change underneath it. The same evaluation, on a schedule.
How we test something that changes its mind
- Real cases, frozen before the run
- The set is built from your history, and it includes the edge cases and adversarial inputs production will eventually supply. It is fixed before anyone sees a result, with the reasoning written down first. Nothing is added or dropped afterwards, because a set edited in light of the scores stops measuring anything.
- Thresholds, not ticks
- Agreed in advance: how often right is right enough, and what an unacceptable answer looks like. A system that is correct 92% of the time is either excellent or unusable depending entirely on what the answer is used for.
- People where the consequences are
- Automated grading sifts volume and is worth having. It does not get the last word where an answer moves money, data or a person's case, because its own mistakes are the quiet kind.
- Fairness, bias and the regulator
- Where fairness, bias and explainability are in scope, evaluation runs against defined criteria and the regulatory frameworks now forming around AI. The same work builds the evidence base a compliance conversation needs, starting with where your system actually sits in those frameworks.
What lands on your desk
- An evaluation set that is yours
- Built from your cases and kept, so it goes on being useful after we leave.
- Thresholds written down
- With the reasoning for each.
- The results with their limits stated
- What the run establishes, and what it is not big enough to tell you - written to be read by your engineers and your executives alike.
- A view on whether it is fit to ship
- Said plainly.
We break our own tests before we trust them
Running this on our own AI work has been humbling, which is the point. When we audited an automated grader we had built, roughly a third of cases that looked perfectly valid turned out not to be scoreable at all, and the graders had marked correct work as wrong four times in thirteen. One of those was a machine insisting on an exact form of words while the system had expressed the same rule better, and in a safer place. A separate check, written specifically to catch a misconfigured model, turned out to be missing it, which we only found by deliberately pointing it at a model that did not exist. None of that shows up in a passing test run. It shows up when you try to break your own instruments, which is now a fixed step in how we work.
Common questions
Why can't we test AI the way we test everything else?
Because the same input does not reliably give the same output. A single pass tells you the system managed it once. What you need to know is how often it gets it right, how wrong it is when it is wrong, and whether that is acceptable for what the answer is used for.
What is an evaluation set?
A fixed collection of real cases with an agreed view of what a good answer looks like. Built from your own history, frozen before anyone sees results, and kept unchanged so today's score means the same thing as last month's.
Can't the AI grade itself?
It can help, and it is genuinely useful for sifting volume. It cannot be the final word, because an automated grader has its own failure modes and they are quiet ones. Ours have marked correct work as wrong for expressing the right rule in different words. Where the consequences are real, a person reviews against the specification.
What if we are buying an AI product rather than building one?
The work is much the same and often more valuable. A vendor's benchmark was chosen by the vendor. An evaluation set built from your cases tells you how the product behaves on your work, which is the only figure that decides anything.
How do we know the evaluation itself is any good?
You break it on purpose. Every check has to be shown to fail when it should before it is trusted to pass anything. It is the part most often skipped, and the part that has caught the most in our own work.