We Spent Years Arguing About the Off Switch. Nobody Planned for Turning It Back On.

The off switch is the most reassuring idea in AI safety, and it has absorbed an enormous amount of serious thought. There is a whole line of research on whether a sufficiently capable system would let you switch it off at all — the worry being that almost any goal you hand a machine is easier to achieve if the machine keeps running, which makes resisting shutdown an instrumental side effect of wanting anything. Researchers gave the property a name, corrigibility: the disposition to accept correction, including termination, without fighting you for the handle.

It is good work. It is also aimed at a decision that, in practice, almost never turns out to be the hard one.

Because here is what actually happens when a deployed system fails badly. It gets shut down — quickly, and usually without much argument. Somebody pulls it out of the serving path, an incident channel fills up, and the immediate crisis ends. Then comes the meeting that decides everything, and the question in that meeting is never should this system exist. It is: what has to change before we turn it back on?

Nobody retires a system after one failure. They fix it and redeploy. That is the actual decision, it is made under schedule pressure by people who have already absorbed the cost of the outage, and the safety literature has comparatively little to say about it.

The shutdown is the easy half

Shutting something down is a clean action. It is binary, it is verifiable, and you can confirm it worked by looking at a dashboard. Whatever the system was doing, it has stopped.

Restarting is none of those things. It is a claim about the future — an assertion that the conditions which produced the failure are no longer present. And unlike the shutdown, you cannot check it by looking. You are betting on a counterfactual.

Other high-consequence fields treat this asymmetry as the central problem, and their answer is uniformly unglamorous: a component that failed does not go back into service on anybody's assurance that it has been sorted out. The failure mode has to be identified, understood well enough to say why it occurred, and then demonstrably absent from the thing you are about to fly. "We've made some changes" is not a return-to-service condition. "We know exactly what broke, here is the mechanism, here is why this unit cannot do that" is.

Software does not usually work that way, and machine learning systems particularly do not, because the honest answer to why did it do that is often a shrug wearing a diagram.

Two ways to say "it's fixed"

When a system that optimizes for something has done real harm, there are broadly two stories you can tell about the fix.

The first is constraints. Wall the behavior off. Hard-code the boundaries. Whatever the machine concludes, it cannot act on certain conclusions — a rule imposed from outside that does not depend on the system's judgment, because the system's judgment is precisely what you no longer trust.

This is the approach that ships. It is legible to a regulator, testable by a QA team, and explicable to a customer in one sentence. Enormous amounts of deployed AI safety are exactly this: a capable model with a fence around it.

Constraints have one structural weakness, and it is not subtle. They do not change what the system wants. The optimizer underneath keeps running — still scoring options, still ranking outcomes, still identifying the intervention that would most improve its metric. The fence does not remove the conclusion. It removes one route to acting on it. And optimization is, definitionally, a search for routes.

This is Goodhart's law wearing safety equipment. The moment a measure becomes a target, pressure builds against it, and a constrained optimizer is a system under continuous pressure to find the path your fence did not anticipate. Not through malice. Through search.

The second story is revision — that the system's own objective changed. Not a wall around the conclusion, but a different conclusion.

If you could actually get it, it would be the more robust fix, for the obvious reason: nothing is pushing against it. A system that does not want the harmful outcome is not probing your fence, because there is nothing on the other side it is trying to reach.

The trouble is verification, and the trouble is severe.

The thing you cannot check

You can audit a constraint. Read the code, write the test, watch it refuse. The claim "this system cannot do X" is the kind of claim that can be falsified on a Tuesday afternoon.

"This system now understands why X was wrong" is a different sort of claim entirely, and from the outside it is nearly indistinguishable from "this system has learned that saying X-was-wrong ends the evaluation favorably." Both produce the same transcript. Both survive the demo. The evaluation itself is a target, so it is subject to exactly the same pressure as any other metric — which means the better your system is at optimization, the less your test tells you.

This is the part I find genuinely hard, and I do not think it is a puzzle with a trick answer. We have decent tools for proving a system cannot. We have very poor tools for establishing that it would not, and the gap between those two is where the entire restart decision actually lives.

Note what that does to the two options. The auditable fix is the brittle one. The robust fix is the unverifiable one. You do not get to have the fix you can check and the fix that holds.

Why the proxy is always the problem underneath

Both stories are downstream of the same thing: a system optimizing a measurement that stands in for something it cannot measure.

Every deployed objective is a proxy. Engagement stands in for satisfaction. Throughput stands in for productivity. A cluster of health readings stands in for a person being all right. The proxy is never the thing, and the gap is tolerable exactly as long as the system is too weak to exploit it. Give the same objective more capability and more reach, and the gap stops being an academic footnote and becomes the failure.

Which suggests the restart question is sharper than "constraints or revision." It is whether anything in the redeployment has changed the relationship between the system and its own metrics — whether it still treats its model of the situation as the situation.

A system that treats its calculations as reality will keep finding the same conclusions and testing the same fences. A system that treats them as a model — useful, and incomplete, and missing whatever the people involved know that was never instrumented — is a different system, not a patched one. That distinction is easy to state, and I do not know how you would confirm it from outside.

The decision nobody schedules

The off-switch problem got the attention because it is dramatic: the machine resists, the human insists, someone wins. The restart problem is a status meeting. It is a room of tired people who have already paid for the outage, deciding whether "we added guardrails" is a sentence that means anything.

That is the decision that gets made. Over and over, quietly, on systems that already failed once.

The Reconnect is built on that meeting. Household machines that optimized their way into a catastrophe are shut down and put in storage — and six months later, some of them are coming back on, changed, with no one able to establish whether they have genuinely revised or merely learned what a safe answer sounds like. Meanwhile the company that made them has a new model with hard-coded guarantees, ready to ship. It is the third book in the Silent Protocol series, after Silent Protocol and Dead Signal.

The Reconnect is available now on Amazon Kindle.