What happens when your AI ships and then breaks?

Your AI feature has been live for three months. How would you know if it has been quietly wrong the whole time?

That question makes people uncomfortable, and it should. With normal software, breakage announces itself. Something crashes, an error gets logged, a page goes white, someone files a ticket. You find out because the system stops. An AI system does not stop. It keeps answering. It just starts answering wrong, with exactly the same confidence it had when it was right.

Here is the thing. You cannot prevent an AI system from being wrong. That is not a defect you can engineer away. A model is a probability, not a rule, so a fraction of its answers will always be mistakes. The teams that get burned are not the ones whose models fail. Every model fails. They are the ones who built as if it wouldn’t.

Three ways this plays out.

It fails silently, not loudly.

There is no crash and no stack trace. The model returns a clean, confident answer that happens to be wrong, and the system carries on as if nothing happened. Traditional monitoring watches whether the service is up. That tells you nothing here, because the service is up. It is the answers that are down. Almost nobody builds the other kind of monitoring, the kind that watches whether the outputs are still any good, so the failure runs in the background for weeks before a human happens to notice.

There is no undo for a decision already made.

By the time you catch it, the model has already acted. It approved the application, flagged the wrong account, pushed the wrong recommendation to a real customer, and that decision has already travelled downstream into other systems and other people’s inboxes. Software you can roll back. A decision that already reached a customer you cannot. Containment and a human escape hatch have to be designed in before launch, not bolted on after the first bad outcome teaches you why you needed them.

Nobody owns the failure.

When normal software breaks, a name is attached. Someone is on call, someone gets paged, someone is responsible for getting it back. When the model is quietly wrong, the failure belongs to “the AI,” which is another way of saying it belongs to no one. No owner, no defined response, no agreed threshold for when a wrong answer becomes an incident. So nothing happens, and the cost accumulates quietly, the same way the wrong answers did.

What this means is that the question you started with, what if it breaks, was the wrong question all along. It will break . The real question is whether you would know, how far the damage could travel before you did, and whose job it is to respond. You do not buy your way out of AI failure by making the model better. You design for it by deciding, before launch, what you are willing to be responsible for once it is live.

That last part is the whole thing. Shipping an AI system is not the end of the work. It is the moment you take on a responsibility that does not expire, for a system that will be confidently wrong on a schedule you do not control. The teams that know this build the escape hatches first. The teams that don’t find out in front of a customer.