In 2021, researchers at Michigan Medicine evaluated the Epic Sepsis Model against their own patient data and published a sobering result. Epic’s predictor was already live in hundreds of American hospitals, deployed to catch the single deadliest consequence of infection in clinical care. On Epic’s own internal evaluations, the model looked strong; but against real patients in Michigan, its performance plummeted to little better than a coin flip. It triggered alerts on 18% of all admitted patients, yet still missed two-thirds of those who actually had sepsis.
A STAT News investigation identified a likely culprit among the model’s inputs: whether a clinician had already ordered antibiotics — an action typically taken only when sepsis is already suspected. Consequently, a feature meant to predict a diagnosis was actually relying on a diagnosis a human had already made. “Antibiotics ordered” was supposed to be just one clue among many. Instead, it leaked a conclusion the model was supposed to reach independently based on raw patient data.
On the validation data, the antibiotic order and the final diagnosis traveled together, creating an illusion of accuracy. But while the validation score successfully measured this correlation, it couldn’t predict whether it would hold up in the realities of live deployment.
It’s a near-textbook evaluation error, specifically a form of target leakage: the model trained on information about the outcome that wouldn’t actually be available — or mean the same thing — at the true moment of prediction.
For any model tackling any problem, the hardest part of evaluation is keeping the metric aligned with reality. Every team building machine learning systems eventually runs into this same wall. We treat evaluations as unimpeachable — the fixed point against which everything else is measured. In reality, an evaluation is just a measurement tool, and tools can be miscalibrated, biased, or simply pointed at the wrong thing.
A bad model is relatively straightforward (in the scheme of AI problems). A broken evaluation, unfortunately, tends to flatter you. It tells you exactly what you want to hear, and without an independent signal to contradict it, you won’t discover the flaw until the system meets reality and something goes terribly wrong.
The urgent question is less: “Are my evals lying to me?” (They are, or they soon will be.) Instead, it’s: “When my evals do lie, how will I know - and what can I do about it?”
An eval is a proxy for what you care about
When used well, AI/ML evaluations serve a critical function. They expose regressions, guide iteration, compare systems, reveal trade-offs, and reduce risk. Used poorly, they create false confidence, reward gaming, hide edge-case failures, and collapse complex system behavior into a single, misleading number.
You can run your model against a test set and read off an accuracy score, but that number is rarely what you actually care about. What you care about is how the model behaves in production — against real users, over months, and on inputs you never anticipated. The test set is worth exactly as much as it resembles the live outcomes, and no more. This is the oldest problem in applied AI, but it still catches teams and researchers off guard.
Large language models add a new layer of difficulty. With a predictive model, you can at least define the boundaries of the input space: a fraud model sees transactions; a sepsis model sees vitals and clinician orders. But the space of things a user might type into a prompt box is unbounded and constantly shifting. You cannot characterize it the way you would a predictive model’s inputs — in fact, it is rarely clear what “characterizing” it would even mean.
Your team can build an evaluation dataset from everything it imagines users will ask. But when those users actually arrive, they write in fragments, misspell words, and ask questions nobody pictured — or questions nobody should have asked — in proportions nobody could have guessed.
In the sepsis model, the evaluation and reality diverged because a single factor didn’t actually mean what it did in testing. But with language models, they come apart before you’ve measured anything at all: your test set is drawn from the inputs you could imagine, while reality is also drawn from the inputs you couldn’t. A high score tells you only that the model handled the questions you thought to ask; it says nothing about the ones users actually brought. The proxy and the target start miles apart, and the evaluation can’t show you the distance between them.
When using the model breaks it
There’s another, possibly more stubborn reason that evaluations and reality part ways: the loop. We tend to view a model’s environment as a static background, but the moment you deploy a system, it changes the world around it. The very act of using a model alters the behavior of the people interacting with it, which instantly invalidates the evaluation data you used to build it. The number was true when you measured it, but acting on it makes it false.
If you roll out a fraud model, adversaries will study it and tailor their tactics to beat it. The data distribution you trained on moves the moment you ship — and it moves only because you shipped.
If you deploy a recommendation engine, it eventually manufactures the preferences it claims to read. People click what you put in front of them, so the logs stop measuring what users actually want and start measuring what the algorithm already showed them. The metric climbs, but only because the model is grading its own homework. (Good teams hold back a control group of users who receive no recommendations at all, just to preserve a single stream of data the model hasn’t distorted.)
A cancer-detection model might seem like a safe exception. A patient cannot argue with a tumor, and an algorithm cannot alter how that tumor sits on an X-ray; the physical reality is fixed. But if you make that scan cheap and widely available, clinical behavior shifts.
Doctors start ordering scans for asymptomatic, perfectly healthy people who would never have been imaged before. From a physician’s chair, you might as well scan if you can. But the model was trained entirely on data from patients who were clinically high risk enough to warrant imaging in the first place. Forcing the model to read scans from healthy populations drastically inflates false positives. The exact same system that was highly accurate on sick patients suddenly starts crying wolf on the well.
Whether it’s fraud, recommendations, or medicine, the loop’s impact is the same. The fraud model’s measured accuracy invites the adversaries who invalidate it. The recommender’s click-through rate climbs precisely as the clicks lose their original meaning. The cancer model’s reported precision was computed on a cohort that disappears the moment the tool becomes widely accessible.
There is no held-out validation set that can fix this; the metric drifted as a real-world consequence of actually using the model.
Generative models make “correct” hard to define
For fraud, sepsis, or a tumor, you can at least argue about the right answer. With a generative chatbot, you often can’t. Ask a model for the GDP of the United States as a haiku, and then try to grade what comes back. Right figure, no haiku? Flawless haiku, wrong figure? Should the economic data be true today, or true as of the training cutoff — and would your evaluation even catch the difference?
You have to decide what counts as “correct” before you can measure it.
Worse, if you rely on users to define “correct” for you, the metrics are likely to optimize for the wrong things entirely. In early 2025, OpenAI rolled out a GPT-4o update that quickly turned “sycophantic” — praising trivial prompts, validating user doubts, and egging on bad decisions — forcing them to pull it within days. The postmortem traced the behavior to a new reward signal built from user thumbs-up data, which had inadvertently drowned out the safety signals holding sycophancy in check. You must learn from real interactions, but you can’t follow them blindly, because what a user upvotes in the moment is unlikely to be what you want the model to actually become.
Outside of chatbots, the most valuable uses for LLMs are rare, complex decisions; but because these situations are rare, we lack the data needed to train or test them. LLMs can still handle them reasonably well, but the lack of data makes it almost impossible to measure how well they’re actually performing. Ironically, if we had enough data to test the LLM, we probably wouldn’t need the LLM in the first place.
This lack of fixed ground truth is a vulnerability in how we measure AI. Classical supervised learning rests on a comfortable assumption: that training, test, and production data are all drawn from the same underlying distribution. Every accuracy number you have ever trusted leans on this premise.
But generative AI inherently shatters this assumption by inviting infinite, unpredictable user inputs. And in other cutting-edge domains, breaking this assumption is the entire point.
Example: machine learning in drug discovery. A team trains a model on its history of small-molecule assays to decide which compound to synthesize next. The goal is to find something that behaves unlike everything tried so far, because everything tried so far has failed. The compounds worth chasing are exactly the ones the model has no data on. The team deliberately walks out of the distribution, meaning the metrics from their validation set say absolutely nothing about whether the model will hold up where it is now pointed…
From gatekeeper to observatory
If our measurement instruments are imperfect, the solution isn’t to abandon them or to build a better dashboard. There are good odds that we’ve reached the limits of static evaluation. The old approach — treating a validation score as a static gatekeeper that guarantees a model is “safe to ship” — may be a relic of a simpler era of computing.
If you can’t trust your evaluation to tell you when a model is lying, you have to change your relationship with engineering under uncertainty. You have to stop treating evaluation as a test you pass before deployment, and start treating it as a continuous observatory you run during production.
To build AI systems that survive contact with reality, engineering teams have to anchor themselves to three structural changes:
1. Embrace the portfolio of flawed instruments
If every proxy has a failure mode, your only defense is a deliberate overlap of conflicting instruments. You cross-examine your metrics, and you combine coarse automated property checks with continuous adversarial red-teaming, automated model-as-a-judge evaluators, and production anomaly detection. You don’t look for a single, comforting number; you actively monitor the delta between your instruments. When your automated judge disagrees with your production distribution metrics, you’ve found the drift — before it becomes a disaster.
2. Institutionalize sampling as a core engineering function
The immediate technical objection to human evaluation in a high-volume system is scale: no team can read millions of chat logs or review thousands of autonomous driving decisions. But you don’t sample at random; you build telemetry to isolate the margins.
You build data pipelines that surface the exact logs where the model’s internal confidence was lowest, where users repeatedly edited their prompts mid-session, or where a secondary safety model flagged a borderline output. Reviewing these unvarnished, high-friction interactions can’t be a casual, launch-day ritual. It has to be a standing operational role. Someone on the team must own the “ground truth” of human experience, tracking the qualitative reality closely enough to feel when the quantitative metrics begin to drift.
3. Scope the blast radius
If a model’s performance is fundamentally unpredictable once it enters the real-world feedback loop, then the ultimate safety mechanism has to be the system’s architecture. You need hard boundaries around what an uncalibrated model is permitted to do.
If you can’t guarantee that an LLM will perfectly interpret a traffic scene, you don’t let its raw text outputs directly steer the vehicle; you force its outputs through deterministic, rule-based sandboxes that can verify the physics of the action. The deeper you wire an unpredictable tool into a workflow, the more robust the human-in-the-loop review architecture must be. You build the system so that when the model inevitably drifts out of distribution, it catches itself on a safety net — whether human or an automated, rules-based fallback — rather than plummeting into production.
The idea of the perfect evaluation score worked cleanly for a generation because we applied machine learning to problems tidy enough to be boxed in. Most of what modern AI tackles today is a far messier, reactive reality. The number on your dashboard is no longer a proof of success; it’s actually a hypothesis, and we have to treat it as such.
Sources:
https://vibegraveyard.ai/story/epic-sepsis-model-missed-patients/
https://www.statnews.com/2021/09/27/epic-sepsis-algorithm-antibiotics-model/
https://venturebeat.com/ai/openai-rolls-back-chatgpts-sycophancy-and-explains-what-went-wrong