AI Gives You Plausible Defaults, Not Correct Answers

Premise: The failure mode of working with capable AI is not that it produces nonsense. It is that it produces fluent, confident, plausible output that is wrong in ways only the actual data or system will reveal. The verification layer is not overhead. It is the job.

A productive week of AI-assisted building threw up the same lesson three times, in three unrelated places. Each time, a capable model gave a competent, well-structured recommendation. Each time, the recommendation was wrong against the specifics, and only checking against the real thing caught it. Three bites in one week is a pattern, not bad luck.

Bite one: the build flags

A data pipeline needed to convert millions of geographic points into map tiles. The tool that does this takes a long list of flags. A research agent was asked to recommend the right flags, and it produced a clean, sensible-looking set. One of them told the tool to keep adding zoom levels until it stopped dropping detail.

For sparse data, that flag is reasonable. For dense data, which this was, it runs away: the build kept generating finer and finer zoom levels, ballooning to gigabytes before failing. The flag was a plausible default. It was wrong for the actual shape of this data.

A recommendation correct in general can be wrong for your specific case, and the model has no way to know which case you are in unless you tell it.

The agent had never seen the data. It recommended against the typical case. The fix was not a better agent. It was sanity-checking the agent's flags against the actual density of the dataset, which a glance at the point count would have flagged immediately.

Bite two: the classification

A separate task asked agents to classify a long list of items and propose which were already complete and could be closed. The agents returned confident verdicts: these are done, close them.

A substring check against the live source proved several of the proposed closures were already marked done, and the agents had matched on the wrong signal entirely. Some had matched lines that merely contained a label string rather than the status they claimed to read. The reported line numbers were unreliable. One item flagged at one location was actually somewhere else in the file entirely.

The verdicts were plausible. The work looked done. But classification is a hypothesis, not a fact, and the only way to test the hypothesis is to check the live source and confirm each claim by unique content match before acting on it. Trusting the verdict, or worse, trusting the line numbers, would have corrupted the source.

Bite three: the recovery

A server would not start because a connection string had gone missing. The fast fix was to reset the credential and move on. The fast fix is also the one with blast radius: reset a shared secret and you may break every other system that quietly depends on it.

Instead, the credential was recovered non-destructively from a configuration file on another box, verified, and persisted. Nothing else broke, because nothing else had to. The instinct to reset is the plausible move. The correct move is to verify what already exists before you overwrite it.

The common thread

Three different surfaces, one underlying rule: verify external state before you assert anything about it.

The reason AI assistance makes this rule more important, not less, is that the assistant removes the friction that used to force verification. When producing a recommendation was slow and effortful, you naturally checked it, because you had invested in it. When a model produces a complete, confident answer in seconds, the answer arrives pre-packaged as finished. The fluency substitutes for the verification that used to be implicit in the effort.

It should not. The model optimises for a plausible answer to the question as posed. It does not, and cannot, optimise for correctness against a system or dataset it has not seen. That gap between plausible and correct is exactly the gap your verification layer exists to close. I have written before about treating the verification loop as load-bearing infrastructure; this is the same principle applied at the level of daily working habits.

The discipline, stated plainly

For build configuration:

• Sanity-check any tool flags or parameters an agent recommends against the actual shape of your data, not the typical case the agent assumed.

For classification and status verdicts:

• Never trust an agent's "this is done" without a content-substring check against the live source.
• Never trust an agent's reported line numbers. Match by unique content, and guard for exactly one match before you act.

For destructive or overwriting actions:

• Verify what already exists before you reset, delete, or overwrite. Recovery beats reset. Reset beats nothing, but only after recovery has failed.

None of these are sophisticated. They are cheap, mechanical checks. The reason they get skipped is precisely that the AI output looked finished, so the check felt redundant. It is not redundant. It is the part of the work that the model structurally cannot do for you, because it requires knowledge of your specific system that the model does not have.

The same logic governs any delegated work. A management accountant does not book a journal because the schedule "looked right." The schedule is agreed to the ledger, the ledger to the bank, and only then does the entry post. Reconciliation exists because confident-looking numbers are not the same as correct numbers. AI output deserves the same treatment the trial balance gets.

What to take from this

Working well with capable AI is not about better prompts or better models. It is about institutionalising the verification step that the model's fluency tempts you to skip.

The model gives you a plausible default. Your edge is knowing your specific data, your specific system, your specific blast radius. The verification layer is where that edge gets applied. Treat it as the deliverable, not the chore.

Part of the Operating Principles series from KG Consultancy.