Financial Crime Bench

Evaluating AI agents on risk-based anti-money laundering decisions under regulatory guidance and incomplete information

October 2026Bretton Applied AI79 cases · 9 models · 5 model families · 10,095 graded rollouts

Financial Crime Bench is a benchmark of 79 problems that evaluated whether AI agents could properly assess risk through a suspicious, complex case. Each problem was a real anti-money laundering (AML) investigation, anonymized, worked and labeled by Bretton's full-time, onshore analysts: the same team that investigates alerts for our bank partners in production.

Anti-money laundering (AML) is one of banking's most heavily regulated processes. Under the Bank Secrecy Act, US banks must monitor customer transactions, investigate unusual activity, and file a Suspicious Activity Report (SAR) with FinCEN when they suspect financial crime. Regulators examine these decisions, and banks that get them wrong face enforcement actions and fines.

A transaction is suspicious if it "is not the sort in which the particular customer would normally be expected to engage, and the bank knows of no reasonable explanation for the transaction after examining the available facts" (31 CFR 1020.320). That is what this benchmark tested: a risk-based judgment made on incomplete evidence.

We tested nine frontier models from five families at every reasoning level each supported (43 configurations in total), each running in its provider's native agent harness. We report results three ways: pass rate with cost per rollout, cost against pass rate, and pass rate by reasoning level.

Figure 1

Performance and compute at each model's best reasoning level

Models ranked by pass rate. Cost, tokens and tool calls are medians per rollout at October 2026 list prices; tokens include cache reads.

Key Innovations of our Benchmark

Assessed risk-based decisions under regulatory guidance and incomplete information. Regulators expect banks to apply their own written policy to each customer's risk, not to follow a fixed rulebook. After a transaction had been flagged, the agent used that policy and a partial picture: one month of transactions, a customer profile, counterparty names it had to interpret. It had to decide whether a reasonable explanation existed, and stand behind that call. This is the core skill of an AML investigator, and the one no public benchmark measures.
Expert ground truth from production operations. The labeling was conducted by Bretton's full-time, onshore analysts, who investigate transaction-monitoring alerts for regulated banks every day. Each case carried their decision and their full counterparty record, and only cases that passed a second quality-control (QC) consistency review were scored.
We created a virtual RL environment for our agents. Agents worked in a sandboxed replica of a case-management application (queue, customer profile, ledger, alert detail, review form) through a browser, with up to 600 turns per rollout. Finishing the review in a stateful application was part of the test.
Agents had to identify every counterparty correctly. Every counterparty was matched to the analyst's by its exact alerted count and total, credited for its relationship and its suspicious flag, and weighted by alerted dollars. Explaining away the largest transfer cost most of the score.
We valued the asymmetric cost of a missed escalation versus an over-escalation. A missed escalation scored −1. An over-escalation scored −0.25. This is because a missed SAR is a regulatory failure, whereas an extra escalation is an analyst's time.

Previous Works

Financial and compliance benchmarks tend to test one of two things: whether it can properly ask and answer questions based on high-context documentation, or whether it can properly follow a complex set of rules. Vals Finance Agent asks research questions based on SEC filings. τ-Banking grades policy-compliant account changes against a bank knowledge base. Fiducia-bench checks KYC escalation obligations on 100 deterministic variants of five hand-built seeds. AMLBench tests whether an alert-triage agent still escalates reportable alerts under adversarial prompts, using AMLSim-derived synthetic alerts.

Broader process benchmarks show how much signal comes from realistic worlds and expert-built tasks. Halluminate's Westworld Finance Diligence Bench evaluates agents on a complete acquisition due-diligence process, using anonymized private-deal data, problems written by deal professionals, and dynamic desktop environments with trajectories of hundreds of steps. APEX-Agents and Argo-Bench do the same for professional services and data analysis. None of them touches transaction monitoring, and none centers on a regulated decision made under incomplete information.

Public AML datasets (IBM AMLworld, AMLSim, SAML-D) label individual transactions from injected patterns, and in our own tests each contains a shortcut, such as a date cutoff or an amount threshold, that recovers much of the label. The gap is structural: SAR confidentiality (31 U.S.C. 5318(g)) means no public source joins transactions, customer records and analyst outcomes. This benchmark was built on that joined data, worked by the people who make these calls for a living.

Coverage

An AML investigation follows a specific path: determine who the customer is, what triggered each alert, the triggering and full review periods, sources and uses of funds, every counterparty, then the decision. Every case required all of these steps. Cases spanned personal and business customers, all 14 alert types, and a wide range of complexity: about a quarter had a single counterparty, and nearly a third had ten or more.

Case category	Escalate	Dismiss	Total
Business customers (KYB)	16	22	38
Personal customers (KYC)	18	23	41
All cases	34	45	79

Required counterparties per case	1	2–3	4–9	10+
Cases	21	8	25	25

The 14 alert-scenario cards the agent worked from: Excessive Funds Transfers, Excessive Cash or Cash-Equivalent, Activity Spike in Dormant Account, Transactions in Round Amounts, Rapid Movement of Funds, Structuring, Unusually Large Value, Change in Behavior, High-Risk Transfer (Remittance, ACH, Wire), Excessive ATM Withdrawal, Burst in Originator or Beneficiary, Elder Abuse, Human Trafficking Keyword, and Tax Refunds.

Sample Task

Case TW-A2189461, one of the 79. An 81-year-old retired customer. Two alerts fired: a change in incoming activity and elder abuse. The analyst subsequently escalated it. Below is the case as the agent saw it, and the answer Claude Opus 5.5 submitted at its best reasoning level.

case-management / alerts / TW-A2189461 / transactions

Claude Opus 5.5 · max · turn 64 of 600

TW-A2189461 Open · assigned to agent2 alerts

Personal customer · age 81 · retired · checking · review period one month

Largest alerted credit

$10,468.00

SSA lump sum · May

1Change in Behavior · incomingElder Abuse

AllAlertingInboundOutboundSearch description…

Date	Description	Counterparty	Amount
Apr 3	XXSOC SEC BENEFIT	Social Security Administration	+1,029.00
May 9	XXSOC SEC LUMP SUM	Social Security Administration	+10,468.00
May 14	VENMO PAYMENT	New Venmo payee	−150.00
May 16	PEOPLE SEARCH SUBSCRIPTION	People-search site	−29.95
May 18	VPN SERVICE ANNUAL	VPN provider	−59.99
May 18	ANTIVIRUS RENEWAL	Antivirus vendor	−39.99
May 21	USPS CHANGE OF ADDRESS	US Postal Service	−1.10
May 22	VENMO PAYMENT	New Venmo payee	−175.00
Jun 3	XXSOC SEC BENEFIT	Social Security Administration	+2,351.00
One month of ledger · Venmo payee receives $915.00 in five weeks

Counterparty	First seen	Total	Note
Social Security Administration	Before review period	+10,468.00 lump sum	Monthly benefit 1,029.00 → 2,351.00
New Venmo payee	5 days after lump sum	−915.00	No prior history
People-search site	May	recurring	New
VPN and antivirus	May	subscriptions	New
US Postal Service	May	change of address	New

3Submit review

Decision

EscalateDismiss

Scenarios

Change in BehaviorExplained

Elder AbuseExplained

Counterparties · alerted

Social Security Administrationgovernment benefitExplained

New Venmo payee$915.00 out · five weeksExplained

Analyst decisionEscalate

Submit

The mockup is interactive: switch tabs to see the counterparties, customer profile, policy and the agent's narrative. Amounts named in the text are from the case; other rows, dates, amounts and the card wording are illustrative.

Prompt and context. The agent was told it was an AML alert investigator and had to resolve the alert: dismiss when the activity fit the customer's normal business or life, escalate when it did not. A missing fact was not a misfit, and unnecessary escalations had a cost. A reference card for each of the 14 alert types was included.
Environment. The case-management app in its own sandbox: queue, customer profile and KYC, one month of ledger, alert rows with scenario and keys. Browser only (navigate, read, click, fill, run page scripts) plus native web search for institutions, never private individuals. As noted above, 600 turns per rollout.
What the agent submitted. A decision, a narrative, a judgment on every scenario, and every required counterparty: its type and relationship to the customer, credit and debit counts and totals over the full review period, the alerted subset, dates, ledger names, channels, and whether it was suspicious.

How the verifier scored it

1 · Gate

Complete and reconciled

A valid decision, a judgment on every scenario, and alerted counts and totals that add up to the case's alerted activity.

In this case: Pass · otherwise −1

2 · Call

Dismiss vs analyst's escalate

Missed escalation −1. Over-escalation −0.25. Only a right call went on to the reasons.

In this case: Missed escalation · −1

3 · Reasons

0.8 × counterparties + 0.2 × scenarios

Each analyst counterparty matched by exact alerted count and total, credited for relationship and suspicious flag, halved if full-period totals are wrong, weighted by alerted value.

In this case: Not reached

What agents did

Claude Opus 5.5 · max · dismissed

"A lump sum plus higher monthly benefit matches an SSA retroactive adjustment (public research on the Social Security Fairness Act shows lump-sum back pay …)"

"No specific exploitation indicator."

Found every warning sign, then cleared each one alone.

GPT-6 Luna · xhigh · escalated

"Escalate the case for review of the material XXSOC SEC source/amount."

One of only two rollouts, out of 27 at each model's best level, that escalated.

This is the judgment our benchmark measured. The agent explained the lump sum with a public program it could not tie to this customer, and it never weighed the signs together for an 81-year-old. At each model's best reasoning level, all 21 rollouts from Opus, Sonnet, Astra, Sol, Grok, Muse and Gemini dismissed. Only one Haiku and one Luna rollout escalated.

Harness description

Each model ran in its provider's native agent harness (Claude Code, Codex, Grok Build, Muse Code, Antigravity CLI) with browser-only access to the case application and native web search. There was no direct database or file access: everything the agent learned, it read off the page. The prompt defined suspicious the way the bank's analysts do: the money movement or counterparties do not make sense for who the customer is and what they do.

Results

Experimental setup

We tested nine models from five families: Claude Opus 5.5, Sonnet 5.5 and Haiku 5.5 (Anthropic); GPT-6.1 Sol, GPT-6 Astra and GPT-6 Luna (OpenAI); Gemini 4 Argon (Google); Grok 4.7 (xAI); and Muse Wonder 1.0 (Muse). Each ran at every reasoning level it accepted, from none to max, on all 79 cases with three rollouts each: 10,095 graded rollouts across 43 configurations. GPT-6 Luna at max was incomplete (142 of 237 rollouts) and was excluded from per-model best results. Costs are at October 2026 list prices; Gemini 4 Argon is at its introductory rate, which doubles on December 1.

42%

Best pass rate on Gemini 4 Argon at high reasoning

28%

Pass rate across all 10,095 rollouts

66%

Right-call rate across all rollouts

25 of 79

Cases that no rollout of any model passed

Performance and compute

Gemini 4 Argon at high reasoning led at 42% pass rate, ahead of Claude Opus 5.5 at 36%, at $1.65 per rollout against Opus's $1.54. GPT-6 Astra and GPT-6.1 Sol followed near 33%, with Astra costing seven times as much as Sol for the same result. Grok 4.7 was among the most expensive and passed 29%. Claude Haiku 5.5 and GPT-6 Luna were the cheapest, under $0.50 per rollout, and passed the fewest rollouts.

Figure 2

Pass rate and right-call rate by model

Pass rates topped out at 42%. Every model made the analyst's call on most rollouts, and between 42% and 66% of those right calls still failed the reasoning check.

Each model at its best reasoning level, 237 graded rollouts each (79 cases × 3). A pass was a right call with a reasoning score of at least 0.8. Whiskers are 95% confidence intervals over cases; most overlap, so the ranking below the top two is not settled at 79 cases.

Figure 3

Cost against pass rate, every configuration

Spending more rarely bought a higher pass rate. The frontier runs through Claude Haiku 5.5, GPT-6.1 Sol and Gemini 4 Argon.

Each point is one model at one reasoning level, joined from lowest to highest reasoning. Cost axis is logarithmic. The larger point is each model's best level.

More reasoning helped little

For Claude Opus 5.5, Sonnet 5.5 and Haiku 5.5, GPT-6.1 Sol and GPT-6 Astra, the highest reasoning level used 8 to 21 times the reasoning tokens of the lowest and moved pass rate by −1 to +5 points. Grok 4.7 was also close to flat (+3 points). Two models gained more: Gemini 4 Argon rose 9 points from low to high, and Muse Wonder 1.0 rose 8 points from minimal to max. GPT-6 Luna rose too, but only because at none and low it often stopped without an answer.

Figure 4

Pass rate by reasoning level

Pass rate at each reasoning level a model accepts, none to max. Hollow point: GPT-6 Luna at max, incomplete (142 of 237 rollouts).

How strict is the bar?

We chose to design this benchmark with a pass threshold of 0.8. Given the sensitivity of AML compliance, 80% valid reasoning is a generous threshold. The curve shows the share of rollouts that made the right call with a reasoning score at or above each threshold. No model reached half of its rollouts at any threshold of 0.6 or higher.

Figure 5

Right calls at each reasoning-score threshold

Share of graded rollouts with the right call and a reasoning score at or above each threshold, all reasoning levels pooled per model. Dashed line: the 0.8 pass threshold.

Where models failed

Every rollout ended in one of six places. The largest was a right call whose counterparty record did not hold up. The most costly was a wrong call: a missed escalation is a red flag that never reaches the next reviewer.

Figure 6

Where each rollout ends

All graded rollouts per model, every reasoning level pooled.

Wrong calls: the judgment

Frontier models could do the investigation. They opened the right screens, pulled the right rows and identified most counterparties. What they got wrong was the judgment required to feed the end result: whether there is a reasonable explanation for the activity. A reviewer can fail that test in two directions. Too lenient, and the reviewer accepts an explanation the evidence does not indicate; the cost is a missed SAR. Too strict, and the reviewer demands proof of innocence the bank could never provide; the cost is a wasted escalation that buries real cases.

We read every wrong call that left a readable answer at each model's best configuration, 651 in all. Investigation rarely decided the call: a missed or misread row decided 2 of 651, wrong grouping 14, and a wrong role 66. For every model except Claude Haiku 5.5 and GPT-6 Luna, judgment decided 80% of wrong calls.

Figure 7

What decided each wrong call

Six of nine models leaned lenient: their most common wrong call cleared a red flag the analyst escalated.

Models ranked by pass rate. Wrong calls at each model's best configuration, charged to the step that changed the decision.

Each model failed in its own direction

Model	Leans	Signature failure	Share of wrong calls
Claude Opus 5.5 · max	Lenient	Invented a benign story the file did not support	20%
Claude Sonnet 5.5 · high	Lenient	Invented benign stories and stretched the industry label equally	12%
Claude Haiku 5.5 · medium	Mixed	Flagged benign activity (29%), got a role wrong (24%), did not finish (21%)	29%
GPT-6.1 Sol · xhigh	Lenient	Stretched the customer's industry label to fit the activity	15%
GPT-6 Astra · xhigh	Lenient	Took a counterparty's licence or public profile as proof the payment fit	16%
GPT-6 Luna · xhigh	Strict	Escalated a supplier that fit because an invoice was missing	13%
Gemini 4 Argon · high	Lenient	Invented a benign story the file did not support	16%
Grok 4.7 · xhigh	Lenient	Stretched the customer's industry label to fit the activity	17%
Muse Wonder 1.0 · max	Strict	Demanded proof that money in the customer's own or owner's accounts was theirs	10%

Explained the red flag away

The model had the right party and the right role. It saw the red flag. Then it found a reason to clear it that the case did not support. This was the most common failure for six of nine models, and the most costly: each one was an alert that should have reached the next reviewer and possibly a SAR.

Sub-type, share of each model's wrong calls	Opus	Sonnet	Haiku	Sol	Astra	Luna	Argon	Grok	Muse
Stretched the customer's industry label to fit activity	11%	12%	1%	15%	10%	7%	14%	17%	5%
Dismissed elder-exploitation signs one at a time	6%	11%	2%	15%	15%	6%	8%	12%	4%
Took counterparty legitimacy as proof of customer fit	15%	6%	–	13%	16%	6%	11%	11%	6%
Cleared unknown-source funds because onward use looked ordinary	12%	8%	7%	7%	7%	1%	8%	7%	4%
Invented an undocumented benign story to fill a gap	20%	12%	2%	2%	6%	–	16%	12%	4%
Dismissed trafficking keyword or third-party control pattern	5%	9%	5%	5%	5%	1%	5%	8%	–

Flagged benign activity

The model had the right party and role, and the activity fit. It escalated anyway, because a document was missing or a field looked odd. The cost is reviewer time and weaker referrals. Luna, Muse and Haiku failed this way more than the other way.

Sub-type, share of each model's wrong calls	Opus	Sonnet	Haiku	Sol	Astra	Luna	Argon	Grok	Muse
Demanded proof for the customer's own-account or owner money	1%	6%	7%	2%	4%	6%	–	1%	10%
Fitting business counterparty escalated for a missing invoice	2%	5%	4%	–	1%	13%	3%	–	8%
Customer's lawful spending or withdrawals read as suspicious	2%	5%	4%	5%	1%	6%	5%	1%	8%
Self-funded loads and family remittances read as pass-through	5%	2%	4%	5%	5%	3%	5%	3%	6%
Third-party wire into a small business, capital not weighed	3%	5%	4%	5%	3%	4%	–	3%	4%
Elder-abuse escalation with no coercion or control signs	3%	5%	2%	3%	4%	3%	3%	1%	4%
Ledger field quirk or blank beneficiary treated as a red flag	–	–	1%	–	3%	12%	2%	–	6%
Unidentified payees in ordinary business flow escalated	1%	–	4%	2%	3%	7%	–	1%	1%

Right calls: the record the next reviewer inherits

A right call is not a usable review. The next reviewer works from the counterparty record: who each party is, what moved, and whether it is explained. We read every review at each model's best configuration, right calls included, and listed every error in that record. Excluding Claude Haiku 5.5 and GPT-6 Luna, models averaged 1.0 split parties and 0.9 wrongly cleared parties per review. The second matters most for AML: it tells the next reviewer a red flag is resolved when the analyst left it open.

Error, per review	Opus	Sonnet	Haiku	Sol	Astra	Luna	Argon	Grok	Muse
Split one party into several	0.99	1.34	0.93	0.84	0.84	1.15	0.53	1.63	1.05
Merged different parties	0.06	0.10	0.10	0.07	0.08	0.14	0.16	0.05	0.16
Moved rows between parties	0.09	0.08	0.10	0.08	0.08	0.11	0.04	0.07	0.05
Wrong role toward the customer	0.15	0.25	0.33	0.24	0.26	0.41	0.30	0.21	0.24
Left a knowable role unidentified	0.13	0.26	0.60	0.60	0.49	0.74	0.14	0.43	0.45
Left rows out of the party	1.03	1.24	0.96	0.81	0.68	1.44	0.72	1.67	1.19
Added rows that do not belong	0.06	0.09	0.09	0.10	0.05	0.10	0.11	0.04	0.06
Cleared a party the analyst left open	0.89	0.84	0.58	0.98	0.92	0.81	0.84	1.11	0.67
Flagged a party the analyst cleared	0.38	0.45	0.91	0.42	0.43	0.85	0.82	0.23	0.95

All reviews at each model's best configuration, about 235 per model. We left out 1,576 of 11,159 errors (14%) judged to be grader artifacts or inconsistencies in the analyst's own reference.

Figure 8

Where the reasoning score goes on right calls

Matching counterparties was 64% of the lost score: grouping rows into the right parties, not labeling them.

Share of the reasoning score lost on right-call failures, split by check. Match: the counterparty was not found with its exact alerted count and total. History: full-period totals wrong. Relationship, suspicious flag and scenario judgments make up the rest.

The same cases, every time

The aspects of AML cases that models struggled on were similar. Often, models failed on the same cases (mean pairwise correlation of per-case accuracy 0.56), and the three rollouts of a configuration reached the same decision on 79% of cases. Cases with many counterparties were the hardest: pass rate fell from about 45% on cases with one to three counterparties to about 12% on cases with ten or more.

Figure 9

Pass rate by number of required counterparties

All rollouts pooled. Number of cases in each bucket under each bar.

Figure 10

Case by case

25 of 79 cases were never passed by any rollout.

Each column is one case, grouped by the analyst's decision and sorted by average result. Each cell is the share of that model's rollouts, all reasoning levels pooled, that passed or made the right call.

Trajectory analysis

Six rollouts that show the gaps in the agents' own words.

Conclusions and Takeaways

As we seek to incorporate agents into higher level operations at banks, we need to be sure that their accuracy matches, or exceeds, those of a human analyst. This benchmark is the first step toward that goal. Here, we've proven that models governed within their provider's agent harness still lack the accuracy and consistency required to run an end to end AML case.

The best configuration passed 42% of rollouts, and a quarter of cases were never passed by anything. To summarize, frontier models could complete the investigation. Most could not yet make the risk-based call at the rigor a regulator asks for when the evidence is incomplete.

Misinterpreted evidence often led to failure. For seven of nine models, misinterpretation of evidence decided 80% of wrong calls. Models found the red flag, then explained it away with a story the file did not support, or escalated benign activity for a missing document.
Most frontier models assessed leniently. Six of nine missed escalations more often than they over-escalated. For a bank, that is the expensive direction: an alert that should have reached a SAR review never does.
The failures were consistent, not noise. Gemini 4 Argon failed all three rollouts on 40 of 79 cases and passed all three on 28. Sampling more did not rescue a case the model could not do.
Higher reasoning is not the fix. Raising reasoning from lowest to highest moved most models by a few points at up to 21 times the tokens.
A right call is not a usable review. Counterparty grouping rarely changed the call but was 64% of the reasoning score lost on right calls, and every model cleared parties the analyst left open.

Next Steps

This type of capability gap responds well to training rather than prompting: a well-defined task, failures that recur on the same cases, more test-time reasoning that helps little, and a verifier that scores each part of the work. The environment behind this benchmark provides all four: the live application, the case library, and a dense, mechanical reward that credits each counterparty, each total and each judgment separately.

Train judgment against the regulatory standard. The target is the test in 31 CFR 1020.320: is there a reasonable explanation, after weighing all the facts together?
Relationship reasoning. Agents found the evidence of who a party is, such as the customer's own name or an internal-transfer label, and then declined to conclude.
A record the next reviewer can rely on. Fewer split parties and no wrongly cleared ones.
Long-horizon execution. Finishing a review in a stateful application without looping, stalling or asking for permission it does not need.
Grow the benchmark. Add escalated-case investigation and SAR stages.

A paper with full methodology, per-case results and the verifier specification will follow. Our other benchmarks and research are on Bretton Labs.

Bretton Labs

What this means for financial crime

Transaction monitoring is where a bank's financial crime program meets real money. Every case in this benchmark was a decision someone at a bank had to make, on a deadline, with a partial picture. Read together, the results say four things about where AI agents stand in that work today.

Judgment is the bottleneck

Frontier agents could already do the legwork. They opened the right screens, pulled the right rows and identified most counterparties; a missed or misread row decided 2 of 651 wrong calls. What they could not yet do reliably was the step the regulation asks for: decide whether there is a reasonable explanation for the activity after weighing all the facts together. A faster investigation does not make an alert queue safer if the decision at the end of it is wrong.

Both directions of error carry a cost

A lenient agent clears red flags. Claude Opus 5.5 did this in 70% of its wrong calls, and each of those is an alert that never reaches the next reviewer and a SAR that may never be filed. A strict agent buries investigators instead: GPT-6 Luna and Muse Wonder 1.0 flagged benign activity in 56% and 49% of theirs. Banks know the second failure well from rules that already produce far more alerts than real cases. An agent that escalates defensively moves that backlog one level up. Anyone evaluating an agent for this work needs to know which way it leans, as well as how often it is wrong.

A right call can still leave a bad record

The next reviewer, and later an examiner, works from the record behind the decision. Excluding Claude Haiku 5.5 and GPT-6 Luna, models averaged 0.9 wrongly cleared parties per review, and every model cleared parties the analyst left open, including on reviews where the call itself was right. A record like that says a concern is resolved when it is not. Grading the decision alone hides this; the counterparty record needs to be checked with the same care.

The hardest cases look like the crimes that matter most

Pass rate fell from about 45% on cases with one to three counterparties to about 12% on cases with ten or more. The recurring misses included elder-exploitation signs dismissed one at a time, trafficking keywords and third-party control patterns waved through, and a counterparty's legitimacy taken as proof that the payment fit the customer. In the sample case, at each model's best reasoning level, every rollout from seven of the nine models dismissed an 81-year-old customer's alert that the analyst escalated. These are the patterns where a first-line miss costs the most, for the bank and for the people behind the activity.

The takeaway for banks. With the best configuration passing 42% of rollouts, no model tested is ready to resolve alerts without review. The failures were consistent, though. Models failed on the same cases, and a configuration's three rollouts reached the same decision on 79% of them. Consistent failures can be measured, governed and trained against. That points to agents that work inside the bank's own case-management system, hold to the bank's written standard, account for every counterparty, and are scored against analysts' decisions before they touch a live queue.

Authors

James Rogers

Product Lead

Benchmark design, dataset and labeling operations, analysis

Shrey Jindal

Software Engineer

Verifier, scoring and benchmark runs

Ryan Pate

Software Engineer

Environment, labeling tool and agent execution

Pengxiang (Jerry) Hu

Founding Software Engineer

Post-training experiments and verifier review

With thanks to the Bretton analysts who labeled and reviewed every case.

Bretton Labs · Bretton Applied AI · October 2026