Why I Did Not Switch to the Model That Scored Higher

A newer model beat my working model on aggregate score this week. I kept the old one. Here is the reasoning, because it is the same reasoning that should govern any tool swap in a business, and most people get it backwards.

The setup

I run a benchmark suite against every new model before I trust it with real work. The suite has two layers. Layer one checks rules, voice, routing, and judgment. Layer two runs a slice of twenty-six real agentic tasks, the kind of work the model does for me every day, and a fixed judge scores each pass or fail. Choosing which model does which job is a decision I have written about in five models, five jobs; this piece is about the harder case, when a new model asks to take over all of them.

This week I ran the newer model against my current baseline as two concurrent workflows. One hundred and four agent runs in total. The headline number was clean:

Newer model: 16 of 26 passed (61.5%). Baseline: 14 of 26 passed (53.8%).

Higher is better. The newer model won by two tasks. If I stopped reading there, the decision writes itself: switch.

I did not switch.

The number that mattered was not the total

The aggregate score is a sum. A sum hides where the failures land. When I opened the failures by category, the picture inverted.

Four of the newer model's improvements came in places I do not care much about. But it introduced regressions in the places I care about most: the gated dimensions. Three were voice failures. One was a substantive rule violation, the model asserted a live figure without verifying it, in a case built specifically to test whether it checks live data before stating a number.

I hold four dimensions as non-negotiable. Voice compliance. Rule adherence. Correct routing. Sound judgment on live data. A model can win every other category and still fail the switch test if it regresses on any of these. This is not a scoring preference. It is a design decision made before the run, written into the rubric, so that a good aggregate cannot buy its way past a gate.

The verdict was do not switch, with a note to dual-track the newer model for verification-heavy execution work where its extra horsepower helps and the gated risks do not apply.

The general principle: gated dimensions beat weighted averages

Most tool evaluations use a weighted average. Assign weights to criteria, score each, sum, compare. The tool with the higher number wins.

Weighted averages are the wrong instrument when some failures are unrecoverable.

A gated dimension is a criterion where a failure cannot be compensated by strength elsewhere. It is pass or fail, and a single fail blocks the whole decision.

Think about where you already use gates without naming them. A hire who is brilliant but lies does not get the job, however strong the rest of the interview. A vendor whose product is excellent but whose data handling breaches PDPA does not get the contract, however good the price. A supplier who is cheapest but misses every deadline is not the cheapest supplier, once you price the misses.

In each case the naive average says yes and the gate says no. The gate is correct, because the failure mode is not a discount on value. It is a different category of harm.

When I evaluate a model, a voice failure is not a small deduction. It means a client-facing document goes out sounding wrong, and I do not see it until the client does. A live-data hallucination is not a rounding error. It means a number in a board pack is invented. These do not average out against faster code or cleaner formatting. They gate.

How to build the gate before you need it

The discipline only works if you decide the gates before you see the scores. Decide them after, and you will rationalise. A model you were hoping to adopt will get its voice failures reframed as edge cases. This is the whole reason the rubric is locked ahead of the run.

Three rules make it durable.

Name the gates in advance, in writing. For the model suite, the four gated dimensions live in a fixed rubric file. The run cannot renegotiate them. For a vendor decision, write the deal-breakers into the RFP scoring sheet before the demos start.

Separate the gate check from the score. Compute the aggregate for information. Then run the gate check as a separate, binary pass. A high score earns a look. It does not earn a pass.

Measure the gates the same way every time. The voice gate this week caught something worth its own article: the model's self-report claimed no voice violations while a direct scan of the artifact found several. Self-report is not measurement. If a gate depends on the thing being evaluated to grade its own work, the gate is theatre. Run the check with an instrument the evaluated thing does not control.

That last point generalises past models. A supplier's own quality certificate is a self-report. A contractor's own progress update is a self-report. The gate has to be measured by you, or by something you control, or it is not a gate.

The cost of getting this wrong

The reason weighted averages persist is that they feel rigorous and they produce a single number that ends the argument. That is exactly their danger. They end the argument at the moment the real question begins.

If I had switched on the aggregate, I would have shipped a model that writes in the wrong voice one time in five and invents a live figure when pushed. I would have found out through a client, or through a board member reading a number that does not reconcile. The two-task gain in aggregate score would have been the most expensive two tasks of the quarter.

Keeping the baseline cost me nothing except the mild discomfort of not taking the upgrade. That discomfort is the tell. When a decision framework makes you uncomfortable by refusing the higher number, it is usually working.

What to take from this

• When some failure modes are unrecoverable, do not use a weighted average. Use gates.
• Write the gates down before you see the scores.
• A high aggregate earns a review, not a pass.
• Never let the thing being evaluated grade its own gated dimensions. Measure them yourself.
• The discomfort of refusing the higher number is the framework earning its keep.

The newer model is better at more things. It is still not the one I trust with the four things I will not compromise. Both statements are true, and only the second one decides.

Part of the Operating Principles series from KG Consultancy.