Every predictive vendor will show you a backtest. Almost none will show you the misses, and that gap is what separates evidence from marketing. The chart in the sales deck hugs the actuals because somebody chose that chart; nobody shows a prospect the model that missed.
You do not have to look far for the tell. The mechanics of honest evaluation are usually documented in detail, well away from the claims. AWS publishes its forecast accuracy metrics and backtesting windows in developer documentation, and other vendors describe their validation methods with similar care. The marketing above then reports outcomes with the misses removed, and buyers are left to bridge the gap between a rigorous method and a curated result.
What honest backtesting requires
The discipline is old and unglamorous, which is partly why it gets skipped in public.
- Frozen slices. Test the model against historical periods it never saw, cut before the test begins, so nothing can drift towards the answer.
- No peeking. Every input the model uses must have existed at prediction time. Leakage from the future is the commonest way a backtest flatters, and you cannot see it in a chart.
- The misses retained. Keep, score and report every prediction, including the ones the vendor would rather bury.
A track record you curated is a brochure.
One honourable exception
Nixtla runs open benchmark arenas where its own models compete in public, results can be reproduced, and rebuttals happen in the open. If other vendors' numbers are real, ask why the same openness would not serve them.
The uncomfortable answer is that a published miss rate can be checked, compared and held against you next quarter, and a curated chart cannot. The vendors are not lying, mostly. They are choosing claims that cannot be checked, and buyers should read that choice as information.
What to ask for
Put one sentence to any predictive vendor, ours included. Show me the backtest with the misses in it, on frozen data, against a naive baseline. A vendor who cannot produce that within days does not lack time; they lack the artefact, and without the artefact there is no track record.
Every Prophesee model ships with a Model Passport that carries its backtest whole. The passport holds the frozen slices, the naive baseline alongside, and every miss. That discipline is hard to copy because it cannot be faked after the fact. To see a passport on your own data, start here.