Why Your Forecasts Need a Scoring Algorithm

Loren Marti · November 6, 2026 · 7 min read

Picture a sales analyst at a consumer-goods company whose biggest customer is Walmart. Every week brings the same questions. How many extra units will next month’s promotion move? What promoted price makes the company the most profit and still gets the Walmart buyer to say yes? How much of the lift is just stolen from our own other items? Get it wrong and you are either sitting on extra inventory or explaining an empty shelf to the buyer.

The answers usually come from a model someone built by hand, under deadline, with no time to check whether it would have held up last year.

Hand two capable analysts the same sales and promotion history and the same list of economic indicators, and you will usually get back two different forecasting models. Ask the same analyst again next quarter, after a few more months of data, and you may get a third. Each one will come with a confident explanation.

Then the numbers get adjusted. In a study of more than 60,000 supply-chain forecasts, planners who nudged forecasts upward often made them worse. At one retailer, upward overrides roughly doubled the average error, from 32% to 65%. (Fildes et al., 2009)

That is not a people problem you can hire your way out of. It is how human judgment works. If your forecasts and the insights you draw from them depend on someone picking the “best” model by eye, they carry that person’s mood, habits and blind spots into every number you plan around.

A model scoring algorithm fixes this. It judges every candidate model against the same measured standard, every time, and it can judge thousands of them before lunch. Here is why that matters, what the research says, and what we learned building one.

Seventy years of research says people are noisy judges

The evidence that a consistent rule beats unaided expert judgment is among the most repeated findings in social science.

Experts disagree with each other far more than their bosses think. In noise audits at two financial-services firms, professionals judging the same cases differed by an average of 48% and 60%. Executives had expected 5 to 10%. Staff with five or more years of experience were no more consistent. (HBR, 2016)

Experts disagree with themselves. Software developers asked to estimate the same task on two different days gave answers 71% apart. Pathologists rating the same biopsy twice agreed with their own earlier rating at a correlation of only about 0.6. (HBR, 2016)

Same data, different analysts, different answers. In one study, 29 research teams were given the same data set and the same question. Their estimated effects ranged from 0.89 to 2.93, and the teams used 21 different combinations of variables. That is exactly what happens when people hand-pick forecasting models. (Silberzahn et al., 2018)

A formula wins or ties almost every time. A review of 136 studies found mechanical prediction about 10% more accurate on average. It clearly won in 33 to 47% of studies, and human judgment clearly won in only 6 to 16%. (Grove et al., 2000) Even a crude equal-weight rule beat an admissions committee: 0.48 correlation with later performance, against the committee’s 0.19. (Dawes, 1979)

Overrides usually hurt. When judges were given a rule’s answer and allowed to override it, they still did worse than the rule alone. People spot too many “special cases.” (Dawes, Faust & Meehl, 1989) The same pattern shows up in business forecasting. Across more than 60,000 supply-chain forecasts, small and upward adjustments by planners often made accuracy worse. Large, well-informed adjustments did help, which is why judgment still has a place. (Fildes et al., 2009; working paper)

The lesson is not that experts are foolish. It is that nobody weighs a dozen factors the same way twice.

What we saw building forecasts for large companies

I have built forecasts and marketing mix models since 2002, and automated model-building systems since 2010, for national consumer-goods, food, retail and industrial companies. In all that time, nothing has amazed me more than how inconsistent people are. Every problem in that research showed up in our own work, in plain view.

Even the experts don’t agree on the rules. Textbook rules of thumb for a “good” model contradict each other, and most come from papers built on a model or two. We replaced them with measurements of what actually moves forecast error and stability, taken across hundreds of millions of models.

I challenge you to ask your forecaster or model builder if they have ever:

Updated a model with a few more months of data and watched it fall apart.
Struggled to explain why “the answer keeps changing.”
Taken two attempts at the same problem and gotten different answers, in very different amounts of time.
Felt less than fully confident defending their model to a client or boss.
Described their method as “I like to look at…” rather than as a standard.

How a scoring algorithm judges a model

Model Score turns the calls an analyst makes by eye into measurement. It asks one question of every candidate: would this model have forecast well, with believable drivers, if we had used it in the past?

Every statistical predictive model has a set of attributes that decide whether you can trust it. How well does it forecast periods it has never seen? Do its drivers hold steady as new data arrives? Do they point in a direction that makes economic sense? Are a few unusual points bending the fit? Do its errors drift over time? We know how each of these attributes affects forecast quality, because we have measured it across hundreds of millions of models.

The algorithm quantifies every one of those attributes for every candidate, the same way every time, and turns them into a single Model Score. Each model is rebuilt as history rolls forward, so it is judged on what it would actually have done, not on how well it fits the past. That is also why searching thousands of models does not hand you an overfit winner. A model that only fits the past scores poorly by design, so it never rises to the top. Models whose drivers point the wrong way, or that cannot see far enough ahead, never make the ranked list.

What changed when we scored instead of picked

The biggest change was that the same data now gives the same answer, and that answer can be checked.

The score separates good models from bad ones. See a pattern in the residuals, the misses a model leaves behind? Model Score goes down. Drivers that wander every time new data arrives? Down. A few odd promotion weeks bending the whole fit? Down. Because the weights are measured rather than set by opinion, a better score means lower future forecast error. Models with better scores also vary less from one another, so the search and the analysts get a clear signal instead of a confusing one.

It does the work no team could. One search scores thousands of candidate models in under a minute. Each one is rebuilt dozens of times as history rolls forward.

Where human judgment still belongs

A scoring algorithm does not replace the analyst. It moves the analyst to the jobs people are good at, and takes away the jobs people do badly.

People still decide:

Which drivers make the final cut. Our screening tool, Winnow, narrows thousands of candidate indicators down to a few. An expert makes the final call on which of those make economic sense for the business.
Which way each driver should push. The analyst says up front that, say, rising fuel prices should lower demand. The algorithm then enforces that rule on every model without exception.
What the forecast is for. How far ahead you need to see, and which unusual periods, like a pandemic shutdown, should not count against a model.
What to do with the answer. A score ranks models within one problem. It does not tell you how to run the business.

The algorithm takes over the part where people drift: weighing a dozen diagnostics at once, the same way, across thousands of candidates. It also has to be held to account itself. We test every change to the score against an independent rebuild, so that a bug cannot hide behind a confident number.

That leaves one inconsistent piece: the expert’s own rules. Two economists will still shortlist different drivers and expect different directions. Making that expertise as consistent as the scoring is where we are headed next.

Start by scoring the forecast you already have

If your plan rests on a forecast someone picked by hand, ask three questions:

Would it have forecast well if you had used it over the last two years, built only on the data available at the time?
Do its drivers stay put when new data arrives, or do they wander and flip?
Did anyone compare it with thousands of alternatives, or with a handful someone liked?

If nobody can answer with numbers, your forecast is carrying one person’s judgment into every decision you make with it.

At Analytics That Work, Model Score answers those questions for you. We search thousands of candidate models built from your history and economic indicators, score every one the same way, and show you why the winners won. You don’t need new data. We start with the sales and promotion history you already have, and the outside economic factors come from free public sources like the Federal Reserve’s FRED database. Your experts still set the rules. The algorithm makes sure they are applied every time.

Talk to us about scoring your current forecast

Coming in our next article: the expert in a box

Scoring makes the judging consistent. The next step makes the expertise consistent, and cheap. What if every analyst on your team had an expert model builder on call, through the $20 or $100 a month AI subscription you may already pay for?

In our next article, I’ll show Model Score connected to an AI assistant through an MCP connector. Have a conversation with your AI agent and tell it your objective: “I want a 12-month forecast for our frozen pizza line. Consider outside economic factors as well as our usual sales drivers.” It handles the rest. Ask it why, push back, have it rerun, or fire up half a dozen agents to work on six different problems at once, from next month’s promotion forecast to a pricing question. One person, the output of a team. An economist in a box, a CPG expert in a box, whatever you need.

The technology is here now, so why wait? Every quarter you do is another quarter of promotion plans built on numbers no one can defend when the buyer asks why, while your competitors sit in meetings arguing over whose numbers are right.

Watch for it here on our blog.

Sources

Kahneman, Rosenfield, Gandhi and Blaser, “Noise: How to Overcome the High, Hidden Cost of Inconsistent Decision Making”, Harvard Business Review, October 2016.
Silberzahn, Uhlmann, Martin et al., “Many analysts, one data set”, Advances in Methods and Practices in Psychological Science, 2018.
Grove, Zald, Lebow, Snitz and Nelson, “Clinical versus mechanical prediction: A meta-analysis”, Psychological Assessment, 2000. The count of 136 studies is as reported by Dietvorst, Simmons and Massey, 2015.
Dawes, “The robust beauty of improper linear models in decision making”, American Psychologist, 1979.
Dawes, Faust and Meehl, “Clinical Versus Actuarial Judgment”, Science, 1989.
Fildes, Goodwin, Lawrence and Nikolopoulos, “Effective forecasting and judgmental adjustments”, International Journal of Forecasting, 2009. The retailer figures come from the authors’ 2006 working paper on the same data.