Overfitting

Almanak launched its Agentic DeFi platform, where natural language lets 18 AI agents divide up the work to produce auditable, backtestable quant strategies that deploy on-chain, with funds staying in the user’s own Safe wallet the whole time. But the author warns that the Zodiac Role permission system only stops agents from moving funds around recklessly - it can’t guarantee the strategy itself is sound, and pretty backtest numbers can still just be an illusion of overfitting to historical price action.

A paper trading strategy with 87.5% apparent win rate fails statistical validation—Z-score yields p=0.24, no significant difference from coin flipping. Using Bayesian adjustment and Overfitting Index (OFI) with 33 real trades to establish a strategy validation logic that avoids the small-sample high-win-rate trap.

We developed a mean reversion strategy. Backtesting showed 3 out of 8 combinations hitting 100% win rate. Then we ran Out-of-Sample validation, and 100% crashed to 25%. Here’s what happened.