How do you know if your AI project is working?

Let’s picture the situation. The data scientist is proud, the number is real, and the room is uneasy in a way nobody can quite name. Someone finally says it out loud. The model is accurate, but is it actually doing anything for us?

Here is the thing. “Working” is not a property you discover about a model after you build it. It is a definition you write down before you build one. And the teams that cannot answer the question almost always skipped that step.

Three patterns show up when the definition is missing.

The metric was never tied to an outcome.

Accuracy, F1, precision, none of these are goals. They are proxies. Someone has to decide what business outcome the number stands in for, and that decision usually never gets made. So the team optimises the proxy and forgets the thing the proxy was supposed to represent. The model reaches 94% and still misses the cases that matter, because the 6% it gets wrong are the high-value ones, and nobody told the model those were the ones to protect.

It is a thermostat reading the temperature of the wrong room. The number is correct. It is just answering a question nobody asked.

Demo metrics are not production metrics.

The pilot reports a number on a frozen test set, in a notebook, under perfect conditions. The business needs to know something completely different. Did the prediction change a decision. Did it save someone an hour. Did it remove a cost that shows up on a real budget line. The distance between “the model is accurate” and “the model is useful” is enormous, and it is exactly the distance many projects never measure.

Nobody defined what an acceptable failure looks like.

Every model is wrong sometimes. That is not the problem. The problem is that if no one drew the line in advance between an expected miss and an unacceptable one, then every failure becomes an argument. Is this broken, or is this just the model being a model? You cannot tell, because there is no agreed line. Improvement and noise start to look identical, and a team that cannot separate the two cannot improve anything.

So what does working actually require? A definition, written before the build, that says: here is the business outcome, here is the metric that proxies it, here is the threshold that counts as success, and here is the failure we will tolerate versus the one we will not. That document has a name. It is the evaluation framework, and it should exist before a single model is trained.

What this means is that the evaluation framework is not a QA step you bolt on at the end. It is the product spec for an AI system. Write it first and the project has a destination. Skip it and you have funded a science experiment that everyone in the room quietly assumed was a product.

The question “is it working” is only unanswerable when nobody decided, at kickoff, what working would mean. Define it first, and you are not guessing anymore. You are just checking.