The hostile test
Most machine-learning evaluation is run on data the model has at least seen the shape of. Ours is built from the cases designed to fool us — and the thing it measures is not how often the tool acts, but how reliably it refuses.
Ch. 01 — Evaluation
The ordinary way, and why it flatters
The standard practice is to develop on one recent span of history and test on the span immediately after it. The two look alike. The distribution is similar, the market character is similar, and the result comes back looking excellent. Then the tool meets a market event that behaves nothing like either window, and every assumption underneath the test goes with it.
Similarity between the development window and the test window is not a technicality. It is the single most reliable way to produce a number that means nothing.
What we built instead
We assemble a deliberately hostile evaluation set. The construction rule is simple and unforgiving:
If a situation looked like an obvious opportunity on the surface but the outcome that actually followed was a loss, it goes in the set.
Construction rule — hostile evaluation setIs that cherry-picked? Yes, deliberately. The purpose is not to estimate typical behaviour. The purpose is to put pressure on exactly the capability we claim to have: knowing when not to act. A tool that looks fine on average can still be catastrophic if all of its failures arrive at the same time, in the same conditions, for the same reason. Averages hide that. A hostile set does not.
What the hostile set is used to measure
Three things, none of which is a performance figure:
- Refusal coverage. Of the situations in the hostile set, how many did the system correctly decline to act on? This is the headline question, and it is a question about restraint, not about returns.
- Behaviour on the ones it took. Where the system did act despite the hostile framing, was its behaviour meaningfully different from its behaviour on ordinary conditions? If the hostile set degrades it, we want that visible.
- Whether stated confidence survives. Under hostile conditions, does the system's own sense of how sure it is stay honest — or does it spike into overconfidence precisely when it should be hedging? Overconfidence under stress is the failure mode that costs the most.
Most conditions are not worth acting on
Set against a plain out-of-sample record — not the hostile set, just ordinary held-out history — the large majority of moments we evaluate carry no positive expected value once the cost of transacting is subtracted. The market is mostly chop. The minority of moments that are genuinely worth acting on have to clear a high bar before anything happens: the stated confidence has to be honest, the surrounding conditions have to be readable, scheduled events have to be clear, and the broader market character has to be one the system can actually interpret.
This is the opposite of how automated market tools are usually sold. The honest version is unglamorous: most of the time, the correct move is no move.
We publish no performance figures. Any return, win-rate or accuracy number attributed to Qovaryx did not come from us. What we will describe is the method — and the method is mostly rejection.
If a tool produces fifty opportunities a day, it is selling you opportunities. It is not selling you an edge.
On volume as a product feature