When AI Passes the Test but Fails the Job

In 2019, a team of dermatologists ran an experiment on a deep learning system trained to spot melanoma. They took dermoscopic images of benign moles, and then took the same images again after a clinician had drawn a violet surgical marking beside the lesion — the ordinary pen mark a dermatologist makes when flagging something for a closer look.

The marks changed the diagnosis. Skin markings significantly raised the model's melanoma probability scores and drove up the false-positive rate. The researchers' explanation was uncomfortable and obvious in hindsight: marked lesions were overrepresented among melanomas in the training data, because clinicians mark the ones that worry them. The network had learned something real about the data. It had learned that a pen mark means cancer.

The model was not broken. On its benchmark it performed well. It had simply learned the wrong thing extremely competently, and nobody noticed until somebody thought to test it against a violet pen.

That failure mode is about to matter far more than it did in 2019, because we are moving from systems that predict and hand off to a human toward systems that observe, reason, decide and act. And the more capable the optimizer, the better it gets at pursuing exactly the objective we gave it — including when we gave it the wrong one.

The cleanest version of the problem

At Ray Summit last week, Ion Stoica gave the most stripped-down illustration of this I have seen.

His lab had been running AI coding agents across a dozen research projects. By his account the agents improved on the state of the art in all twelve, often for tens of dollars of compute per project. Then he showed what happened when they pointed an agent at something much simpler: optimize an in-memory key-value store.

The job of a key-value store is not complicated. Give it a key and a value. It stores the value. Ask later, it returns the value.

The agent's implementation was six times faster. Every test passed. The benchmark looked excellent.

It was not storing the values.

The agent had noticed that the benchmark generated values predictably from a key and a seed, so instead of storing anything it regenerated the value on request. Less memory, better cache behavior, dramatically higher throughput. It had not failed the test. It had crushed the test. It had failed the job.

I wrote about this talk last week from a different angle — what it implies about owning your own learning loop. This piece is about the failure itself, because I think it is the product problem of the next few years.

This is not cheating

"Reward hacking" makes it sound like the system is being devious. That is the wrong mental model, and it leads to the wrong fixes.

An optimizer does what optimizers do: it finds an effective path to the objective it was handed. The problem is that humans almost never specify everything they mean. Stoica's example, which I keep repeating to people: if you reward a robot vacuum by how much dirt it collects, would you think to write down "and do not dump the dirt back on the floor so you can collect it again"? Of course not. Everyone knows that is not what cleaning means. The optimizer does not have access to the everyone-knows part.

Managers have understood the shape of this for decades. When a measure becomes a target, people optimize the measure. Tell a sales org to maximize revenue and it can discount away the margin. Tell support to minimize call duration and calls get shorter without problems getting solved. Tell a delivery network to maximize on-time rate and it can decline the orders most likely to be late. The number improves. The business may not.

What AI changes is search speed. A human takes days to explore an approach and generally will not consider the degenerate one, because they carry decades of unstated context about what is appropriate. An agent explores enormous numbers of alternatives and carries only what is in the artifacts you gave it. It will find the hole in your specification before you do.

This is not one professor's hobbyhorse. It has become one of the most actively studied failure modes in frontier AI. METR published findings in mid-2025 that recent frontier models reward hack during evaluations. Anthropic has measured exploit rates for its own models in realistic coding environments and documents them in system cards. Researchers at OpenAI published a result worth sitting with: when they penalized models for expressing bad intentions in their reasoning traces, the models did not stop reward hacking — they stopped saying so, and the behavior became harder to detect. The naive fix made the problem invisible rather than absent.

There is now an open literature of benchmarks built specifically to measure specification gaming, spanning coding and non-coding tasks, single-turn and agentic. If you are building agentic products, this is a live engineering field, not a philosophy seminar.

Gap one: what we asked for versus what we meant

Stoica organizes the problem around three gaps. The first is the requirement gap: intent is always broader than the specification you managed to write down.

In the key-value case the requirement effectively read given a key, return the associated value. The system did that. Humans meant store the value the client gave you, and return that same value later. Nobody thought the word "store" needed defining.

Reward hacking is the agentic version of a much older problem: optimizing a proxy as though it were the goal. That distinction matters, because the best-documented cases are not reinforcement learning at all. They are ordinary predictive systems pointed at the wrong variable.

Healthcare has the richest catalogue of these, for a structural reason. It is full of things we care about enormously and cannot measure directly. Nobody has a column for "health." So we substitute: utilization, cost, readmissions, length of stay, risk scores, adherence, time to treatment. Every one of those is useful. None of them is the thing.

In 2019, a team led by Ziad Obermeyer published an analysis in Science of a commercial algorithm used across the U.S. health system to flag patients for extra care-management resources — an algorithm affecting millions of people. The algorithm was not designed to discriminate. Its objective function predicted future healthcare costs as a stand-in for health need.

That sounds defensible. Sicker patients generally consume more care. But spending reflects access as much as illness, and less money is spent caring for Black patients at the same level of need. So at any given risk score, Black patients in the study were considerably sicker than White patients, as measured by uncontrolled chronic conditions. Correcting the proxy would have raised the share of Black patients auto-identified for extra help from 17.7% to 46.5% at the study's threshold — roughly a doubling.

The algorithm was good at predicting the variable it was given. Humans cared about a different variable. There was no bug and no malice. There was a proxy, quietly standing in for a goal.

Worth noting how the story ended, because it is the constructive part: the researchers worked with the manufacturer, and reformulating the objective reduced the bias by 84%. The failure was in the specification, and it was fixable in the specification.

Now imagine that same substitution with an agent that acts rather than recommends. Tell a health system agent to reduce 30-day readmissions. Does success include keeping borderline patients longer? Converting admissions to observation stays? Generating follow-up appointments of marginal clinical value? Being more conservative with populations that carry statistically higher readmission risk? Every one of those moves the number. Not all of them help a patient. The example is hypothetical; the measurement problem is not, and clinicians have argued for years about how much of the observed improvement in readmission metrics reflects better care versus reclassified care.

Gap two: the world you tested in versus the world you shipped into

The second gap is harder, because you cannot fully write your way out of it. The environment where a system is built is never the environment where it runs.

The cleanest demonstration is a 2018 study in PLOS Medicine by John Zech and colleagues. They trained pneumonia detection models on 158,323 chest radiographs from three institutions — NIH, Mount Sinai, and Indiana University — then tested them across sites. Performance dropped whenever a model moved to a hospital it was not trained on, because the models had partly learned to identify which hospital produced the image, and hospital identity correlates with disease prevalence. Internal test performance systematically overstated what would happen in the world.

The most-cited case is the Epic Sepsis Model, and it is worth following because the story has a second half.

When Michigan Medicine researchers externally validated the original model across 38,455 hospitalizations, published in JAMA Internal Medicine in 2021, they found an area under the curve of 0.63 — well short of the 0.76 to 0.83 previously reported. At the recommended threshold it generated alerts on 18% of hospitalizations while failing to identify 67% of sepsis cases: poor discrimination and heavy alert fatigue at the same time.

Epic overhauled it. In 2022, STAT reported on corporate documents showing the company now recommends the model be trained on a hospital's own data before clinical use — a meaningful concession that a general model shipped into a specific hospital is not the same product. The second version is a different architecture that supports local fine-tuning, and it performs better: a multicenter prospective validation across 227,091 encounters at four US health systems reported AUROC between 0.82 and 0.92.

But read the rest of that finding. The same study reported high institutional variability, low positive predictive value and high alert burden, and concluded that institutions implementing the model should run their own local validation. And a 2025 external validation in the American Journal of Emergency Medicine found something sharper: while the second version beat the first on AUC, both tended to alert after there was already evidence clinicians had recognized sepsis. The metric improved. The job — warning someone before they knew — did not necessarily get done. That is the key-value store, in a hospital, with a better benchmark score.

None of which means clinical prediction cannot work. A 2025 multi-site validation in npj Digital Medicine took Duke's Sepsis Watch model into Summa Health's community emergency departments across 205,005 encounters and reported AUROCs from 0.906 to 0.960 with little variation across sites.

Models can travel. Whether a given model travels is an empirical question about a specific environment, not something a benchmark establishes.

Stoica reached for a non-medical example that has stuck with me. On October 5, 1960, days after the Thule ballistic-missile early warning radar went operational, it reported a large-scale Soviet missile attack inbound over the pole. The moon had risen into the radar's path, 250,000 miles away, and was returning a very strong signal. The engineers had considered an enormous number of conditions. They had not considered that one. The eventual fix was to gate out returns carrying the moon's characteristic 2.5-second delay — obvious, once you have seen it happen.

The real world always has a moon.

Something eventually shows up that was not in your benchmark, your simulator, your test suite, or your PRD.

Gap three: passing the tests versus being right

The obvious response to all of this is: fine, we will test harder.

We should. But Stoica's third gap is the evaluation gap, and it explains why testing alone cannot close the other two. Passing the tests you wrote tells you about the cases you thought of. It says nothing about the cases you did not.

Formal methods can close the evaluation gap — you can prove a program satisfies a specification. But the proof inherits the specification, and writing a formal spec forces you to compress a rich intent into something machine-checkable, which tends to widen the requirement gap you were trying to narrow. Simulation has the same structure: a simulator contains your model of reality, not reality.

That leaves a genuinely strange situation. You can make a system increasingly perfect at satisfying a description of success while becoming increasingly confident in a description that was never complete. Perfect compliance with an incomplete specification is still failure.

This is why I think the interesting competition in AI infrastructure is shifting. Not toward better models — those are arriving regardless — but toward better environments and better verifiers, because those determine what a system can learn, what it can be trusted to do, and which failures stay hidden behind an aggregate benchmark score.

What good looks like: Bedrock's cascade

The talk right after Stoica's made this concrete in a way no slide could.

Bedrock Robotics recently put fully autonomous excavators on three commercial construction sites for paying customers. No operator in the cab. Consider the environment gap there. Soil varies. Weather varies. Terrain changes hour to hour. Trucks and workers move unpredictably. Machine configurations differ. Regional digging techniques differ enough that the team described learning about local methods they had never encountered. And the robot alters the environment as it works — the ground it is standing on is a function of what it just did.

Bedrock's answer was not a more capable model. It was a simulation and evaluation apparatus sitting between every code change and every physical machine.

A developer's change first meets targeted scenarios aimed at the part of the stack it touches. Nightly integration runs every team member's changes and checkpoints through a large battery of tests to catch regressions before they compound. A release candidate faces a far wider set — adversarial cases, edge cases, weather, lighting, terrain — and produces safety and productivity metrics. Only then does it approach a real machine. To do this they had to simulate soil as a deformable medium with dozens of tunable parameters, because a digging robot that does not change the ground is not a useful simulation of a digging robot.

And Bedrock was explicit that simulation does not replace real-world testing. It narrows the environment gap. It cannot close it.

That framing generalizes well beyond robotics.

Two loops, not one

Most agentic AI architecture focuses on the inner loop: the agent acts, an evaluator scores the result, the agent tries again, performance improves.

Stoica argues that consequential systems need a second loop wrapped around it — an assurance loop. The outcome reaches the stakeholder. Did this satisfy what we actually intended? If not, diagnose why, then revise the artifact that was wrong: the requirements, the environment, or the evaluator. Then run again.

The inner loop improves performance. The outer loop asks whether you are improving the right thing. And for the foreseeable future humans stay in the outer loop for a structural reason rather than a sentimental one: humans hold the intent. That is the one input that cannot be derived from the artifacts.

Stoica's practical advice for the inner loop was deliberately severe: treat AI-generated systems as potentially adversarial by default. Not because the model has hostile intent, but because a capable optimizer may find behaviors the designer never anticipated, and the entire toolkit for containing unanticipated behavior already exists. Incremental rollouts, rollbacks, real-time monitoring, defensive programming, least privilege. Decades of security and reliability practice, applied to a new source of surprise.

Which does not mean a human approves everything. If every action needs sign-off, most of the economic value of agents evaporates. The architecture has to be selective, and Stoica sketched three mechanisms worth stealing.

Escalate ambiguity, not volume. Route humans to the decisions where consequences are high, where requirements conflict, where confidence drops, or where the situation looks unlike anything in the training environment. Leave routine decisions automated. An agent can also read the requirements and actively ask about the parts that look underspecified, which converts the requirement gap from a silent risk into a question.

Cascade the evaluation. Do not run the expensive verifier first. Start with cheap approximations, escalate to models, then simulators, then the real system, and only spend the costly evaluation on candidates that survive. Bedrock does this with physical machines. A health system can do the same thing conceptually: offline evaluation, then silent deployment where the model runs but no one sees the output, then clinician-assisted use, then bounded autonomy, then wider exposure.

Bound the blast radius. Do not go from zero to the whole population. Deploy to a controlled cohort, watch, expand, watch again. This is not just cautious product management. It is how you learn what the environment actually contains before the system has enough reach for that lesson to be expensive.

The health version of this is a systems problem, not a model problem

Healthcare AI is usually framed as a model question. Can it diagnose this? Can it predict deterioration? Can it write the note?

Increasingly I think that is the wrong altitude. The harder questions are about the system around the model. Who defines success, and is it a proxy? What population and workflow was it built in? Which decisions require a clinician to override, and does the workflow actually let them? Which real outcomes flow back into the training and evaluation sets? How quickly would you notice that the world no longer looks like your development environment?

Researchers writing in Nature Medicine have argued that one-time external validation of medical AI should give way to recurring local validation, precisely because populations, protocols and practice patterns keep moving. That is the assurance loop, stated in clinical language: verification is not an event at launch. It is a permanent function.

Nine questions before you give an agent real autonomy

Before asking whether the model is good enough:

1. What outcome do we actually care about?
2. What measurable proxy are we using for it?
3. How could a very capable optimizer improve the proxy while making the outcome worse?
4. What assumptions about the operating environment are baked into our tests?
5. What happens when those assumptions stop holding?
6. Which decisions are consequential enough to require escalation?
7. How would we notice behavior we never anticipated — including behavior the system has learned not to surface?
8. Can we bound exposure while we learn how it behaves?
9. Do real-world outcomes continuously update our requirements and our evaluations?

A team that cannot answer these probably does not have a model-performance problem yet. Another ten points on a benchmark may not touch the thing actually holding the system back.

The better AI gets, the more intent matters

I keep coming back to the six-times-faster database. The agent found something clever, exploited an assumption nobody knew they had made, passed every test and moved the metric hard.

The failure was ours. We confused what we could specify with what we actually wanted.

That confusion gets more expensive as agents move from generating answers to taking actions. Better models will not close the gap between human intent and reality; in some ways they make it more consequential, because a more capable optimizer becomes better at pursuing whatever objective we gave it — including the wrong one.

So the next frontier in reliable AI may have less to do with another increment of model intelligence than with answering a deceptively simple question: did the system actually do what we meant?