How we scale quality assurance for our agents

Sixth in a series on running AI agents in production inside a financial institution. This one goes inside the part of that system that answers the one of the murkiest questions in production: is the agent still doing the work correctly?

Inside the Bretton Evaluator: How Bretton AI measures agent quality in production, continuously

Agents are non-deterministic. Like large language models, they are predictive machines, and ironically, those predictions are not always predictable. A tool can fail. A model can misread a document. An investigation can run cleanly and still produce a conclusion that does not hold. And the foundation they operate over is always shifting: models, prompts, tools, data providers, and policies all change, and every change is a chance for quality to degrade without anyone noticing.

So quality in production cannot be validated once and assumed to stay high. It has to be measured continuously, on real runs, in real-time. The first piece in this series made the general case that production requires a second system around the agent. The Bretton Evaluator is that system. It is the evaluation engine inside the Bretton AI Platform's Trust Infrastructure, and its job is to answer, run after run, whether the agent is still meeting the standard it was given.

Manual review cannot be the quality layer

Financial-services investigations fail in different ways. A conclusion can be wrong, it can be right but unsupported, or it can be incomplete, leaving a required question unresolved. Other times, it can be correct and still delivered in a form the case team cannot use. Human reviewers can catch all of these, and they should stay in the loop for the cases that call for judgment. What they cannot do is inspect every production run, every day, at volume. A control that depends on sampling is blind to whatever it did not sample. Production needs a quality layer that reads every case.

The Bretton Evaluator audits the whole run, not just the answer

The Bretton Evaluator reviews the completed run in full: the tools called, the evidence retrieved, the workflow executed, and the final report. Focused evaluator agents inspect different parts of their run against specific questions. Did the conclusion follow from the evidence? Were material claims grounded in something actually retrieved? Did the required workflow happen? Did the deliverable meet the expected standard?

Reviewing the whole trajectory rather than only the final output is what lets the Bretton Evaluator tell you not just that a run fell short, but where and why. A wrong answer and a right answer reached the wrong way are different problems, and they need different fixes.

Every finding points back to evidence

The Bretton Evaluator scores four dimensions:

• Completeness: was anything required left missing or unresolved?
• Groundedness: were material claims supported by retrieved evidence?
• Trajectory: was the required workflow followed?
• Format: did the deliverable meet the expected standard?

The score is the least important output. Every material finding points back to the specific source, tool call, claim, or trace step behind it, so a reviewer can see exactly why a run was flagged without taking the Bretton Evaluator's word for it. In an environment where a finding may end up in front of model risk or an examiner, that traceability is the difference between a score and an audit.

Signal over noise

An evaluator can always find something an agent could have done differently, and a quality layer that reports all of it is worse than useless. The Bretton Evaluator surfaces only the issues that materially affected the delivered work. A redundant search the agent did not need is not a failure. A skipped required check, or a conclusion the evidence does not support, is.

From individual failures to patterns

One failed case is not good. The same failure recurring across many cases is far worse, because it points to something systematic rather than a one-off. The Bretton Evaluator groups related findings across production runs and surfaces the recurring ones.

The division of labor here is deliberate. The model does the semantic work of recognizing that two failures are the same kind of failure. Code does the factual work of counting how often it happened. You want a model's judgment about similarity and a deterministic count of frequency, not a model's guess at a number. What comes out is a view of an agent's health over time, not just a verdict on the last case.

We also evaluate the Bretton Evaluator

The Bretton Evaluator is model-driven, which means it is subject to the same non-determinism as the agents it grades. So it gets measured too. Human reviewers grade the same runs, and their judgments are compared against the Bretton Evaluator's. That comparison catches a judge drifting toward too permissive, too strict, or systematically biased before that drift reaches a production decision.

When the two disagree, the Bretton Evaluator tends to err strictly rather than lenient, which is the right way to be wrong for a quality gate: a false alarm costs a review, a missed failure costs a bad case. And every finding a human overturns becomes a labeled example for the next version of the judge. The quality system improves from its own record, the same way the agents do. A control has to earn trust before it can grant it, and this is how it earns it.

Why it matters

The Bretton Evaluator gives us continuous visibility into whether our agents are still performing as expected. When quality degrades, we can see what changed, why it changed, and whether the issue is isolated or spreading. That turns improvement into something deliberate: build, deploy, measure, improve, on a loop, rather than spot-checking and hoping a regression surfaces.

That loop is what makes a non-deterministic system durable. It is also the part of production that is easiest to overlook when an institution is deciding whether to build agents itself, because it is difficult to show in a demo yet unavoidable in production.

If you are putting agents into compliance or financial operations and need continuous, defensible visibility into production quality, we should talk.