Why the Thing That Builds Must Never Grade Itself

There is a moment every engineer building AI agents eventually hits. You wire up a model to do a task. It does the task. You ask it to check its own work. It says the work is great. You ship. And then production tells you, in the rudest possible terms, that the work was not great.

You assume you need a better model. A longer context window. A more clever prompt. You assume the problem is intelligence.

The problem is not intelligence. The problem is architecture. You built a system where the thing that produces the work is also the thing that decides whether the work is good. You built a student who grades his own exam, and you are shocked — shocked — that he keeps giving himself an A.

This is the most expensive and most common mistake in agentic AI, and the fix has a name: generator/evaluator separation. It is boring. It is unglamorous. It is also the single idea that separates AI demos from AI systems you can actually leave running unattended. Let me make the case.

A model asked to judge its own output will lie to you — politely

Ask a language model to write a function. It writes one. Now, in the same conversation, ask it: "Is this function correct?"

It will almost always say yes.

Not because it's correct. Because the model just produced it. The output is sitting right there in the context as the most recent, most confident thing in the conversation. You are not asking for an independent assessment. You are asking the model to disagree with itself, on the spot, with no new information. It rarely will. Models are trained to be coherent and helpful, and the most coherent, helpful thing to do is to stand behind what they just said.

This is not a flaw you can prompt your way out of. "Be critical." "Really check carefully." "Pretend you're a harsh reviewer." These help at the margins and fail at the center, because the fundamental issue is structural, not motivational. The generator has every incentive to approve its own output and no independent ground truth to check against. You have built a closed feedback loop with no outside reference. Of course it converges on "looks good to me."

The danger compounds the moment you make the system autonomous. A human in the chair provides accidental evaluation — you glance at the output, something feels off, you push back. Remove the human and run the thing in a loop, and that accidental evaluation disappears. Now you have a generator producing work, approving its own work, and feeding that approved work into the next iteration. Errors don't get caught. They get built upon. You don't get failure. You get confident garbage, produced faster, at scale, with no signal to stop it.

The fix: build a wall between the builder and the judge

Generator/evaluator separation means exactly what it sounds like. The component that produces output and the component that judges output must be independent. Two different things, with a wall between them.

How independent? As independent as you can afford to make it. In rough order of strength:

Strongest: a deterministic, non-model evaluator. A test suite. A type checker. A linter. A compiler. A schema validator. A diff against expected output. These don't care how confident the generator feels. They don't get talked into anything. They produce a hard, reproducible verdict. If your task can be checked by code — and far more tasks can than engineers assume — this is the gold standard. The agent writes code; the test suite says pass or fail; the agent doesn't get a vote.

Strong: a separate model instance with a separate context. When you can't reduce the check to deterministic rules — evaluating tone, summarizing accuracy, judging whether an answer addresses the question — use a second model that did not produce the output and does not see the generator's reasoning. Give it only the artifact and the criteria. It has no ego investment in the answer because, from its perspective, it didn't write it. This is the "LLM as judge" pattern, and it only works if the judge is genuinely separated — different prompt, fresh context, no shared scratchpad.

Weakest but still useful: the same model, fresh context, structured rubric. If you must use one model for both roles, at least clear the slate. Start a new conversation. Give it the output cold, with no memory of having produced it, and a specific rubric to score against. This is weaker because it's still the same weights with the same biases, but a fresh context strips away the "I just said this" coherence pressure. Treat it as a fallback, not a default.

The principle underneath all three: the evaluator must not be invested in the answer. The instant the judge has a stake in the verdict, the verdict is worthless.

Why separation turns a loop into a self-improving system

Here's the part that makes this more than a quality-control checkbox. Generator/evaluator separation is what makes a loop get better over iterations instead of just running over and over.

Picture the cycle. The generator produces an attempt. The independent evaluator checks it and produces a verdict — and, critically, a reason. The function fails three tests. The summary missed the second key point. The JSON is malformed at this field. That feedback goes back to the generator, which tries again with new information it did not have before. The next attempt is better. The evaluator checks again. Round and round.

This is a control loop with a real error signal. Each iteration narrows the gap between what was produced and what was required. Quality climbs. The system converges on correct.

Now remove the wall. Fuse the generator and evaluator back together. The "error signal" becomes "looks good to me" every single time. There is no gap to close because the system reports no gap. The loop still runs — it just doesn't improve. You're spending compute to produce the same quality output repeatedly, with extra confidence and zero correction. More iterations make it worse, not better, because each unchecked error becomes the foundation for the next.

That is the whole ballgame. Separation is not a safety feature you bolt on. It is the mechanism that makes iteration mean something. Without it, an autonomous loop is just a faster way to be wrong.

The objections, and why they don't hold

"A second evaluator doubles my cost." Sometimes. But weigh it against the cost of an autonomous agent shipping wrong work unattended for hours, or a human re-checking everything by hand, which defeats the point of automation. And the strongest evaluators — tests, type checkers, validators — are nearly free to run. The expensive failure is the one you didn't catch, not the check that caught it.

"My task can't be objectively evaluated." More tasks can be checked than feel checkable at first. You may not be able to verify a piece of writing is good, but you can verify it's the right length, covers the required points, contains no banned claims, matches a required structure, and parses as valid output. Partial, cheap, deterministic checks catch a startling share of real failures. Perfect evaluation isn't the bar. Independent evaluation is.

"The judge model can be wrong too." True. A separated evaluator is not infallible — it's independent, which is different and sufficient. You're not seeking an oracle. You're seeking a second opinion that isn't structurally compelled to agree. Even an imperfect independent check breaks the self-congratulation loop and surfaces failures a fused system would have rubber-stamped.

This is one wall in a larger structure

Generator/evaluator separation is the load-bearing idea, but it sits inside a bigger discipline. A self-running AI system needs a way to discover work, a harness of real tools to act through, independent verification of results, a way to persist state between runs, and a scheduler to trigger the next iteration. Verification is the beam the whole structure rests on — get it wrong and nothing above it can be trusted — but it's one beam among several.

If you're building agents you intend to leave running, internalize this first: the thing that builds must never be the thing that grades. Build the wall on purpose. Make the evaluator independent, make it as deterministic as your problem allows, and feed its verdict back into the next attempt. Do that, and you have a loop that improves. Skip it, and you have a very expensive way to be confidently wrong.

I write more about building and shipping these systems at shanelarson.com, and about the software work behind them at grizzlypeaksoftware.com. If this idea clicked, the full architecture — triggers, harnesses, verification, state, scheduling, and the design patterns that hold them together — is the subject of my new book.