[{"data":1,"prerenderedAt":4},["ShallowReactive",2],{"article-body-the-backtest-includes-the-misses":3},"\nEvery predictive vendor will show you a backtest. Almost none will\nshow you the misses, and that gap is what separates evidence from\nmarketing. The chart in the sales deck hugs the actuals because\nsomebody chose that chart; nobody shows a prospect the model that\nmissed.\n\nYou do not have to look far for the tell. The mechanics of honest\nevaluation are usually documented in detail, well away from the\nclaims. AWS publishes its\n[forecast accuracy metrics and backtesting windows](https://docs.aws.amazon.com/forecast/latest/dg/metrics.html)\nin developer documentation, and other vendors describe their\nvalidation methods with similar care. The marketing above then\nreports outcomes with the misses removed, and buyers are left to\nbridge the gap between a rigorous method and a curated result.\n\n## What honest backtesting requires\n\nThe discipline is old and unglamorous, which is partly why it gets\nskipped in public.\n\n- **Frozen slices.** Test the model against historical periods it\n  never saw, cut before the test begins, so nothing can drift\n  towards the answer.\n- **No peeking.** Every input the model uses must have existed at\n  prediction time. Leakage from the future is the commonest way a\n  backtest flatters, and you cannot see it in a chart.\n- **The misses retained.** Keep, score and report every prediction,\n  including the ones the vendor would rather bury.\n\n*A track record you curated is a brochure.*\n\n## One honourable exception\n\nNixtla runs\n[open benchmark arenas](https://github.com/Nixtla/nixtla/tree/main/experiments/foundation-time-series-arena)\nwhere its own models compete in public, results can be reproduced,\nand rebuttals happen in the open. If other vendors' numbers are\nreal, ask why the same openness would not serve them.\n\nThe uncomfortable answer is that a published miss rate can be\nchecked, compared and held against you next quarter, and a curated\nchart cannot. The vendors are not lying, mostly. They are choosing\nclaims that cannot be checked, and buyers should read that choice as\ninformation.\n\n## What to ask for\n\nPut one sentence to any predictive vendor, ours included. Show me\nthe backtest with the misses in it, on frozen data, against a naive\nbaseline. A vendor who cannot produce that within days does not lack\ntime; they lack the artefact, and without the artefact there is no\ntrack record.\n\nEvery Prophesee model ships with a\n[Model Passport](/insights/the-model-passport) that carries its\nbacktest whole. The passport holds the frozen slices, the naive\nbaseline alongside, and every miss. That discipline is hard to copy\nbecause it cannot be faked after the fact. To see a passport on your\nown data, [start here](/contact).\n",1786984937349]