Skip to main content
Steve Bolton

An average expert answer is now cheap. Knowing when it's wrong isn't.

AI gives everyone the same confident answer to the typical problem. Two small experiments on what agents assume, what they lock in, and where judgement now sits.


Java would never catch on against C++, because it ran too slowly. That was the verdict I overheard from a lecturer in 1996 in the computer lab, while I was building a project in Java around the time version 1.0 was cut. The project pulled stock option data from the internet and graphed it.

He was right about the speed, and wrong about what it meant. He was ranking two languages on one axis, and could not see either that the axis would move (it did) or that it was not the axis most problems turn on. My program spent its time waiting on the network, not using the CPU. What mattered was networking and drawing, and Java's network and canvas classes let me focus on the problem instead of fighting the X UI libraries in C++. Learning the language along the way was a bonus. C++ gave far more control over memory, but was that really something I needed to control?

I backed neural networks around the same time, for the same reason. They were unfashionable and widely held to be inferior to other techniques, and the objection was about compute. Compute is the kind of constraint that expires; it seemed there was much more they could do given a bit more processing power.

In both cases the consensus was right about the thing it was measuring, and the thing it was measuring was not what bound the problem.

I have been thinking about that while working with coding agents, though the parallel is not exact, and the gap is the interesting part. The lecturer was one person's compressed experience, and I could have argued with him; he might have asked what my program actually did. What is different now is that the same confident ranking arrives for everyone at once, there is nobody in the room to argue with, and it never says which axis it ranked on.

What the default looks like

I ran two experiments with AI agents to see this for myself.

I asked twelve fresh agents, across three models, to answer the standard senior system design questions as a candidate would. Half got the prompts as they are usually posed, with no scale stated: design Ticketmaster, a URL shortener, a news feed. Every one of those designs assumed web scale (hundreds of millions of users) and averaged around 14 components: caches, queues, a CDN, distributed locks. The other half got the same three problems at a stated small scale: a regional ticketing site at 50 purchases a second, an internal shortener for 2,000 staff, a community feed with 20,000 daily users. Their designs dropped to about 10 components. Every ticketing design held seats with a simple conditional update in Postgres, and most said outright that no distributed lock was needed.

What none of them did was treat the scale as an open question. Given the number they used it well, so the capability was never in doubt; what was missing was the move of asking. My old lecturer never made it either. The canonical Ticketmaster answer is easy to practise and easy to repeat; the part worth anything is knowing why you wouldn't reach for Redis by reflex, and how far the data model will carry the load before extra components earn their place.

That gap is fixable, and the fix is instructive. Tell an agent in its instructions file to establish scale before it designs anything, and it will. But look at what just happened: I supplied the constraint, and to supply it I had to already know that scale is the variable deciding this class of design. The fix moved my judgement up a layer.

None of this is a complaint about capability. I have never been attached to the lines themselves. Formatting, linting and static analysis have always done as much of that as I could hand them, and agents are the next step in the same direction: they get ideas out of my head faster, so more of my time goes on the design, the decisions and the abstractions that get value to the product without adding debt. Given a problem it has seen a thousand variants of, a model produces a clean implementation faster than I will, and follows the practices I would otherwise have had to remind someone about.

It is also worth being precise about what the default is, because it is easy to grant it too much. It approximates the best answer only where the corpus it came from was subject to selection. In well-trodden technical domains it usually is close, because bad answers get selected out: systems fall over and people stop doing that. Where no such pressure operated, it is simply the most common answer, which is a different thing wearing the same clothes. The recognisable look of an AI-built website did not win on merit. It won on frequency.

Variation is cheap. Selection is where knowledge comes in.

AI can generate options without limit. More samples, more agents and more randomness give you more variety, but not more knowledge. Knowledge only comes in when an option is checked against something outside the model: a test, a real user, an experiment, or a person's judgement grounded in context the model doesn't have.

The idea isn't new. Donald Campbell described creativity as blind variation and selective retention in 1960. What's new is how cheap the variation has become, and how easy it is to forget the selection.

One experiment made this concrete for me. I set up a small pricing module (invented, so no model could have seen it before) and planted six bugs: a loyalty discount starting at 6 months instead of 12, two discounts adding together instead of the larger one applying, free shipping from £40 instead of £50, and so on. Then I asked AI agents to write tests for it, the way you'd ask any assistant to add tests to existing code.

I ran it twelve times, across three models. With the code alone, five of the six runs locked in four of the six bugs (the sixth, three): they wrote tests asserting the wrong behaviour was correct. Then I added one line per rule to the code, stating what each rule should be, such as when the loyalty discount starts. With that one line, all four Sonnet and Opus runs caught all six bugs and flagged them. Haiku, the smallest model, listed every mismatch in its reply but left most of the calls to me, and one run pinned the code's behaviour anyway.

  • Code alone4 of 6
  • Code plus one line per rule (Sonnet, Opus)0 of 6
Planted bugs the agents' tests locked in as correct

The AI wasn't the weak link. It had nothing outside the code to check against. The one line told it the answer, and that is the point: the reference has to come from somewhere other than the thing being checked. Published research finds the same pattern at much larger scale. Tests written by language models tend to assert what the code does rather than what it should do (Konstantinou and colleagues, 2024).

There is a loop here, and the near end of it needs no theory. Had those tests gone green and had I trusted them, four wrong behaviours would have been locked in as correct and two more left untested, with a passing suite to prove it. The more we trust the output, the less selection pressure we apply, and the less there is to catch the next thing.

The level you have to direct at keeps rising. Two years ago I was reviewing lines; now I review designs and the model fills in what I would have written. I expect that to continue, and at each step more of the work is genuinely done without me. What does not change is that the ceiling sits at the top of whatever you can specify. Nor do the layers underneath stop mattering. Most Java developers never think about garbage collection, and a low-latency trading team lives inside it. The skill is knowing which kind of problem you have, and keeping enough understanding of the layer below to question what comes up from it.

I should own something here. While working on this I kept a dated record of the ideas I thought were mine, then checked them by putting the same questions to fresh models, blind. Two that I had started to think of as my own came straight back: review the change in behaviour rather than the diff, and a self-improving system is capped by whatever checks its work. Knowing about the pull towards the average did not exempt me from it. What caught it was not better instinct but the check. I trust neither the model's answers nor my own. The work is a debate, and neither side gets the last word. The experiments do.

What this means for teams

The first question is whether your problem is actually average. Most are. Most software recreates something that already exists, and for that the default is simply the right answer arrived at faster. The practices below are for the cases where it isn't, and I'm testing them as I go:

  • Write down what the rule should be, independently of the code. Wherever your team keeps intent (a spec, a ticket, a docstring), one line per business rule, written by the person who owns the rule, is one of the cheapest outside references you can create. The line has to come from the decision, not from reading the code. Generate it from the code and it will simply agree with the bugs.
  • Decide before you ask. Write down your decisions and constraints, scale and priorities included, before the AI proposes anything. Unstated, they default to the canonical case. Adding "use best practices" does not change that: it returns the modal set of precautions for the modal threat model.
  • Ask what would have to be true for this to be the right answer. Run it on the output, not the input. If the answer is "that you are serving a hundred million users" and you are serving fifty a second, you have found the assumption nobody stated. This one works when deciding in advance cannot, because it does not require you to have known the question beforehand.
  • Get review that brings different information. A second copy of the same model mostly agrees with the first. A reviewer with the logs, the user complaints or the old design notes adds something.
  • Record what would change your mind. Whenever you record a decision, as you would in an architecture decision record, also record what would reopen it: the constraint it rests on, and what would have to change for that to stop holding. The same goes for "that won't work because…" from a person or an AI: write down the constraint and when it should be checked again. "Too slow" was true of Java in 1996 for some classes of problem. It stopped being true for most of those, and it never applied to mine. It matters more now because things change faster than models are retrained. The signal that a constraint has lifted often appears months before it reaches a model's training snapshot, and until then the model will keep giving you the old answer with full confidence.

The pull towards the average isn't new

Best practice, textbooks and published research always pulled us towards the consensus. They just did it slowly, which left time for people who saw past it. AI speeds that pull up enormously, and most of what it produces is good. That is precisely the problem: if it is cheap for you, it is cheap for everyone you compete with, and an excellent average is not an advantage.

We have always valued work partly by how hard the skill behind it was to acquire. People even rate the same poem as better when told it took longer to write (Kruger and colleagues, 2004). A simple idea carried by deep craft can become a masterpiece, and the craft doubled as a proxy: clean code suggested someone who understood the problem. AI lifts everyone's apparent skill towards the average, so more of us can be a good cover band, and there is nothing wrong with that. But it cheapens the skill it lifts and breaks the proxy. Slop becomes easy to make and hard to spot, because a well-executed copy looks the same as a considered answer.

None of this closes the layers below. There is still room for craft and invention at every layer; it is just harder to stand out there. Some authors still write on a typewriter, and some engineers still hand-write assembly. For the rest of us, time no longer spent on the layer below goes to the one above, where there is more room for a choice of our own. That is why the level we direct at keeps rising, and why telling the copy from the considered answer is a skill in its own right.

Human consensus does the same thing, and the skill is noticing which axis it is using. A few years ago I moved a build off Gradle onto Maven. Gradle is the more functionally capable tool and the consensus is right about that. It was simply not what bound us. The build was legible to about three people, more of the team already knew Maven, and Maven's constraints forced a decomposition that permissiveness had not. The capability ranking was answering a question we did not have.

What it bought us was consistency: modules built to one convention instead of each one configured its own way, and best-practice plugins applied the same way everywhere. Nor was it meant to be permanent. The tool was never the problem; the logic that had accumulated inside the build was. Once Maven's conventions had forced that logic out into a standard structure, moving back to Gradle later would be easier than it had ever been.

The far end of the loop is slower, and I can cite it rather than test it. As more of what models train on is itself model output, the worry is that the distribution narrows. The evidence is more careful than the headlines: Shumailov and colleagues showed in Nature in 2024 that models trained recursively on their own output degrade, with the damage falling hardest on the tails. But Gerstgrasser and colleagues showed the same year that this depends on synthetic data replacing real data; where it accumulates alongside, the degradation stays bounded. So the catastrophic version looks unlikely. What holds is the part that matters here, and it is not a race to the bottom. It is a race to the average: the middle gets reinforced and better served while the tails get thinner. There is a human version of the same finding. In a 2024 study, stories written with AI-suggested ideas were judged more creative one by one, but were more similar to each other (Doshi and Hauser). And because the middle keeps getting better, there is less and less reason to notice what the tails have lost. Creativity is still possible, but now it has to be deliberate.

There is a market signal here I did not expect. MIT Technology Review reported in October 2026 that AI labs have begun buying the archives of defunct startups: Slack and Teams histories, emails, documents and code. The demand appeared as labs reached the limits of public and synthetic data. And what they are paying for is not the software, since these are companies that failed, but the record of people deciding things. A request arrives, people argue, someone opens a ticket, the code changes. Variation was never the scarce input. Selection is, and it is now being bought. Hold that loosely: nobody has published results comparing models trained on this material with the alternatives, so the premium is a bet rather than a finding. The direction of the bet is still informative.

So the default gets better every year, and that is genuinely good news: more of the work is done well without me. What does not commoditise is the direction. Knowing which problem you actually have, what to leave out, and what to check the answer against. That part got more valuable because everything underneath it got cheap.

The lecturer wasn't wrong about Java. He just never said which axis he was ranking on. The model won't say either, and there is nobody in the room to argue with. Asking has become the job, and that is what I'll be writing about here: how teams keep that judgement when the average has never been easier to get.

This article will end up in a training set too, so it carries its experiments and its doubts as well as its conclusions. These were small experiments, run with one vendor's models, and I treat them as pilots rather than proof. I also check my ideas against blind model baselines, and cut or cite whatever comes straight back. A larger test on real bugs from real projects is next, with its method published before it runs. If you've seen something that cuts the other way, I'd like to hear it.