How we check that the backtest matches reality
One page, no formulas. Companion to the reconciliation dashboard.
What we did
We ran a small, deliberately simple market-making strategy on a real exchange account for one trading day. It places a buy order a little below the current price and a sell order a little above it, on five liquid perpetual contracts, with tiny sizes (about $50 per order, at most $150 of inventory per contract). It has no forecasting signal and no hedge; it only earns the spread and the exchange's maker rebate. That makes it a clean test object: nothing in it is clever, so the only thing being tested is whether our simulator reproduces what the exchange actually did to our orders.
What was recorded
- Market data: every best bid/ask update, every order book change and every public trade on the five contracts, with the exchange's own timestamps, recorded continuously by an independent recorder.
- Our orders: every placement, modification, cancel, acknowledgement and fill, with the exchange's timestamps and fees.
- Strategy state: position, quoted prices and profit-and-loss, published once per second.
- Latency: how long market data took to reach the strategy and how long order acknowledgements took to come back.
What the backtest does
The backtest runs the same strategy code with the same parameters on the recorded market data of the same day. It does not see our live orders. It has to decide on its own where it would have quoted, and the simulator has to decide whether those quotes would have been filled. Two things make this hard and are modelled explicitly:
| Modelled effect | How |
|---|---|
| Delay | Market data and order acknowledgements are delayed by the amounts measured live that day, so the simulated strategy reacts as late as the real one did. |
| Queue position | A resting order is filled only after the orders ahead of it in the queue. We show two variants: a standard model calibrated on past live data, and a pessimistic model that never assumes the queue shrinks because other participants cancelled. Live results should fall between the two. |
| Provable fills | The recorded book is the real book and it contained our own orders. When the recorded best bid moves up to or through the price of our resting sell (or the best ask down to our resting buy), the market could only have got there by executing our order, so the simulator fills it in full at that moment regardless of the queue model. Before this rule was added the simulator reproduced 90% of the live fills on the demo day; with it, 98%. |
Two ways the simulator is run
| Run | What it answers |
|---|---|
| Strategy re-run | The strategy code decides its own quotes from the recorded market data, with the same parameters and parameter changes as live. This gives the PnL lines. Once one fill differs from live, inventory and therefore quotes diverge, so fills are compared in aggregate. |
| Order-stream replay | Every live order action (placement, amendment, cancel) is replayed at its recorded time; the simulator only decides which of those orders would have been filled. This isolates the fill model and allows a fill-by-fill comparison: a live fill counts as reproduced when the simulator fills the same side within 60 seconds at a price within two ticks. |
How the difference is measured
- Profit and loss: cumulative live PnL (realized + unrealized + fees) against the backtest PnL on the same time axis.
- Fills: number, turnover and timing of fills, hour by hour, with live and simulated fills drawn on the same price chart.
- Causes: for every hour with a difference we state the most likely reason: a fill the simulator took that the exchange did not give us (or the reverse), a difference in how open positions were valued, fees, or latency.
What it means
A simulator can never be exact: it does not know how other traders would have reacted to our orders, and the data gaps of a real day (reconnects, missed messages) are in the record. What we claim is narrower and testable: on the same day, with the same data, the backtest's PnL and fill count land within the range spanned by the two queue models, and every remaining difference has a named cause. Where that is not the case, the table says so. Nothing on the dashboard is hand-adjusted; every number is computed from the recorded data.
Calibration
The simulator has two free settings: how fast the queue ahead of our order is assumed to shrink, and the delays it applies. The delays are measured on the trading day itself from the strategy's own telemetry. The queue setting shown as standard was chosen on this same day so that the simulated fill count matches the live one; that makes the standard line an in-sample calibration, not a forecast. The pessimistic line uses no calibration at all and is the floor. A fair reading is: the calibrated simulator reproduces the day it was calibrated on; whether it holds on the next day is a separate test, and the method for it is exactly the one on this page.
Reproducibility
The recorded market data, the order log, the strategy parameters and the simulator version are archived together. Re-running the backtest from that archive reproduces the dashboard to the cent.