How to Know If a Change Actually Worked

You made a change in March. Enquiries were up in April. The change worked.

Maybe. April is also a better month than March in most businesses, you hired somebody in February, a competitor closed, and you happened to change two things at once. Any of those explains it as well as the change does.

This matters because businesses keep expensive things that never worked and abandon cheap things that did, and both errors come from the same missing habit.

The four explanations to rule out

Seasonality. The single most common false positive. Almost every business has a rhythm, and comparing a good month to a bad one produces an improvement from nothing. Compare against the same period last year, not against last month.

Something else changed. Businesses rarely change one thing. New person, new price, a supplier problem, a large client leaving. If three things moved, attributing the result to the one you are proud of is a decision rather than a finding.

Regression to the mean. You changed it because it was bad, and bad periods are usually followed by ordinary ones whether or not anybody intervenes. This one is invisible and it fools experienced people constantly.

The measurement changed. New system, new definition, new person entering the data. A rise that is actually better recording is the most embarrassing version, because it can persist for years.

Write the prediction down first

The cheapest possible protection: before making the change, write one sentence saying what should happen, by how much, and by when.

“Quote turnaround should fall from three days to one within six weeks.” Then check it against reality on the date, not whenever the result looks good.

This works because it removes the two ways people fool themselves after the fact: moving the goalposts, and choosing the window that flatters the result. If it does not appear by the date, that is information, and it is information you can only have if the date existed beforehand.

Find a comparison group

The strongest method available to a small business, and it costs nothing: change it in one place and not another.

One branch, one team, one region, one product line. Both are exposed to the same season, the same market and the same economy, so the difference between them is much closer to being the effect of your change than any before and after comparison can be.

It feels slow, and it is slower by a few weeks. It is dramatically faster than spending two years rolling out something that never worked.

Measure the thing, not the proxy

A change to your quoting process should be measured on quote turnaround and win rate, not on revenue.

Revenue is affected by everything, so it is the noisiest measure you have and the least able to tell you about one intervention. The closer the metric sits to the thing you changed, the faster and more clearly it will move, and the harder it is to argue with.

Keep the revenue question for the annual review, where it belongs.

What AI makes possible here

Most of this was known and skipped, because doing it properly meant somebody assembling data from three systems every time anybody wanted to know whether something worked. At two days of effort per question, the questions stopped being asked.

AI collapsed that cost. Pulling the same period last year, splitting by branch, controlling for the three other things that changed and writing the comparison in plain language is now a routine operation rather than a project.

There is a second, less obvious use, and it is the one that changes the quality of the answer. An AI step reading your operational history can tell you what else changed in the window you are looking at, which is the step humans skip almost every time. You remember the change you made. You do not remember that the same fortnight is when a large client doubled their order and somebody went on leave for three weeks. The record does.

And AI can watch continuously rather than waiting to be asked. A change that worked for two months and then stopped working is extremely common and almost never noticed, because nobody re-runs the analysis. Something that does re-run it will tell you.

Ask the people, not only the data

One source consistently gets ignored, and it is the fastest of all: ask the team whether anything is different.

They will tell you things no metric contains. That the new process works but they are doing an extra step outside it to make it work. That the improvement is real and comes from something unrelated. That the number moved because they changed how they record it, not because anything changed.

That last one is worth asking about explicitly every time, because it is the failure people are least likely to volunteer and it invalidates everything downstream.

Do this after the numbers, not before, so their account and the data are independent of each other. Where they agree, you have something solid. Where they disagree, the disagreement is the most interesting finding in the exercise.

When the answer is that you cannot tell

Sometimes the honest conclusion is that the data will not support a finding either way. Say so.

That is a better outcome than a confident wrong answer, and it usually points at the real problem: nobody recorded the baseline, or the volume is too low for any change to be detectable in less than a year. Both of those are fixable from today, and neither gets fixed while everybody pretends the last analysis was conclusive.

The four questions

• What did you predict, in writing, before you started.

• What is the comparison, and is it the same period last year rather than last month.

• What else changed in that window, including things nobody thought were relevant.

• Is the metric close to the change, or is it revenue standing in for everything.

Four questions, ten minutes, and they will save you from keeping at least one expensive thing that has never worked.

AI Optimize builds the comparison properly, tells you what else changed in the same window, and keeps checking whether it still holds six months later. That work sits under Reporting & Data.