The Expert in a Box: Let the AI Drive, Let the Process Decide
Last week I argued that people are inconsistent judges of forecast models, and that a scoring algorithm fixes it by judging every model the same way, every time. Read that one first if you missed it.
I promised to show the next step: an AI assistant driving that scoring engine for you. You tell it what you need, it does the work of an expert model builder, and you can ask it why, push back and have it rerun.
Before I show it, a warning that sounds like it contradicts the whole idea: don’t ask an AI to build your forecast.
Why “build me a forecast” goes wrong
Ask a general-purpose AI assistant for a sales forecast and it will happily give you one. It will write the code, fit a regression in seconds and explain the result with complete confidence.
The trouble is that fitting a model and building a good one are different jobs. A good model takes a process: knowing what the data really means, following the steps in order, and testing every candidate against a standard for what counts as acceptable. An AI improvising on the spot has none of that. It picks drivers that look good on the history it can see, and it can’t tell you how the model would have done on months it hadn’t seen, because it never checked.
So the answer isn’t a smarter AI. It’s giving the AI a proven process to run, and a score it can’t argue with.
The setup: Model Score on the other end of a connector
The link between the two is an MCP connector. MCP, the Model Context Protocol, is an open standard that lets an AI assistant use outside tools safely. (Anthropic, 2024) Our connector gives the assistant the Model Score engine’s tools: load the data, screen economic indicators, start a model search, read the results, build a model by hand, and produce the final forecast.
The assistant decides which tool to use next. It doesn’t decide what counts as a good model. Model Score does that, the same way for every candidate, using weights measured across hundreds of millions of models.
A real session, on data where we know the answer
My client work stays private, so instead of the frozen pizza example I teased last week, here is a session on the built-in sample that ships with Model Score. It’s made-up monthly sales for a home-goods brand, nine years of history, plus twelve candidate indicators. I ran it on November 7, and the numbers below are what came back.
The sample has one big advantage over real data: we know the right answer. Sales were built to rise with consumer confidence three months earlier and housing starts six months earlier, fall with the mortgage rate, and follow the seasons. The other indicators are noise, or near-copies of the real ones planted to fool a model builder: gasoline, copper, lumber, shipping costs, the weather and a sentiment survey that tracks confidence closely. A good process should find the real drivers and leave the rest.
Here is the conversation, condensed. The AI’s answers are summarized from what the connector returned.
I need a six-month forecast for monthly sales. Consider the outside economic factors in the file as well as seasonality and our November promotion.
Starting a full search on all fourteen candidates. Each model is rebuilt 24 times on growing history, so it’s judged on what it would actually have forecast.
Done in 11 seconds. 2,760 models scored. 138 passed the ranking rules. The best, with a Model Score of 0.92, uses consumer confidence (three months ahead), housing starts (six months ahead), the mortgage rate, seasonality and the November promotion. All the effects point the way the economics says they should: confidence and housing up, mortgage rates down. None of the noise series made it into the best model.
That is the right answer, found without anyone picking variables by eye. But the interesting part is what got turned away.
What the score turned away
Were there models that fit just as well but didn’t make the list?
Yes. One model swapped in the copper price, which has nothing to do with home-goods sales. It fit the history almost exactly as well as the winner. But copper can’t be known far enough ahead, so the model can only forecast two of the six months you asked for. Model Score kept it off the ranked list.
Judged on fit alone, you could not have told those two models apart. That is exactly the choice an analyst, or an improvising AI, gets wrong. The score caught it because it asks a different question: not “does it fit?” but “would it have forecast what you need?”
Pushing back
An expert on call is only useful if you can argue with it. So I did.
Sentiment and confidence measure the same thing. Wouldn’t a model with both be stronger? Build it and score it.
Built and scored by the same rules as the search. It scores 0.33 against the winner’s 0.92. Adding sentiment didn’t help: as the model was rebuilt on growing history, its effect flipped sign twice, and on the full history it came out slightly negative. Confidence already carries that information. The model also reaches only three of your six months.
This is the conversation you want with an expert. You bring the business question and the hunch. It tests the hunch properly, in seconds, and shows you why the answer is what it is. Nobody’s opinion wins by default, including mine.
Why it’s safe to let the AI drive
Notice what the AI did and didn’t do in that session.
- It didn’t choose the drivers. The search tried thousands of combinations, and Model Score ranked them.
- It didn’t decide which way each driver should push. That rule was set up front, by a person: confidence and housing starts raise sales, mortgage rates lower them. Every model was held to it.
- It didn’t decide what “good” means. The score did, using the same measured weights for every candidate.
- It did the work in between: running the steps in order, reading the results, testing my pushback and explaining it all in plain English.
With your own data, there is one more step before the search: finding the outside factors. The assistant screens the Federal Reserve’s FRED data, about 5,700 national and state series, keeps only those with a believable economic link to your business, and narrows them with our screening tool, Winnow. It agrees the list with you before anything goes into a model. That is the same expert judgment I described last week. The difference is that it now happens in a conversation instead of a three-week project.
One person, the output of a team
Put those pieces together and one analyst can do what used to take a team. Ask for the promotion forecast, then a pricing question, then a what-if on next quarter’s mortgage rates. Each one runs the same process and is judged by the same score, so the answers are consistent with each other and you can defend every one.
That is what I mean by an expert in a box. Not an AI that thinks it’s an economist, but an AI that runs an economist’s process and can’t skip the parts that keep the answer honest.
Try it on your own forecast
If your plan rests on a forecast someone built by hand, the fastest way to see the difference is to score it. Send us the sales history you already have. We’ll run it through Model Score, connected to an AI assistant such as Claude, and show you which drivers hold up, which don’t, and how far ahead the forecast can really see.
Talk to us about scoring your current forecast
Coming in our next article: the forecast that shows up for work
A model you have to go and ask is a model most people never ask. Next time I’ll show scored models built into the dashboard your team already opens every morning. It tells you what moved overnight, whether that’s normal or a real change, why it happened and what to do about it, with the model’s track record attached.
Watch for it here on our blog.
Sources
- Anthropic, “Introducing the Model Context Protocol”, November 2024.
- Session data: Model Score’s built-in tutorial sample (synthetic monthly home-goods sales and twelve synthetic indicators), run through the Model Score MCP connector on November 7, 2026, with the Standard search settings.