lower estimated token spend than cold restarts ($5.31 vs $6.60).
Continuity benchmark · June 2026
Does agent memory save tokens? 54 Codex runs.
We gave Codex six tasks that continue earlier work. Each ran three times from a cold start, three times with a hand-written handoff file, and three times with OnlyBots. Here is what changed, what didn't, and where the evidence is still thin.
Results
At a glance
19%
41%
lower estimated token spend than a hand-written handoff file ($5.31 vs $8.95).
18/18
tasks completed, against 16/18 for cold restarts and 18/18 for handoff files.
Spend is estimated from recorded token counts at GPT-5.5 list prices on 2026-06-25. Priced at Claude Opus 4.8 rates instead, the savings are 19.7% and 40.8%.
The surprise
Handoff files cost more than starting cold
The usual workaround for lost context is to have the agent write a handoff note at the end of a session and read it at the start of the next. In this benchmark that worked for reliability: handoff runs completed every task. It did not work for cost. Handoff runs used 31% more tokens than cold restarts (4.88M vs 3.71M), because writing the note is extra work and reading a long note bloats the next session.
OnlyBots matched the handoff file's 18/18 completion rate while using 38% fewer tokens than handoff runs, and 19% fewer than cold restarts.
Method
How it was run
Each task is a small synthetic repository with an earlier episode of work already done. The agent has to finish the job in a new session. A deterministic scorer, including hidden tests, decides success.
Three conditions
- Cold restart: a fresh Codex session with only the repository and workspace state.
- Handoff file: a standardized handoff document written at the end of the earlier episode and given to the next session.
- OnlyBots: an isolated OnlyBots runtime with Codex lifecycle hooks packing relevant memory into the session.
Setup
- Runtime: Codex CLI using an OpenAI API key.
- 6 tasks × 3 conditions × 3 repeats = 54 runs, recorded 2026-06-25 (benchmark version
ocb-0.1). - Every OnlyBots run used a fresh, isolated runtime home, so no memory leaked between runs.
- Token counts come from Codex usage records. Cost is reconstructed from list prices: $5.00 per million uncached input tokens, $0.50 per million cached input tokens, $30.00 per million output tokens (reasoning tokens counted as output).
Data
Totals by condition
| Condition | Completed | Total tokens | Median tokens per run | Est. spend | Wall time |
|---|---|---|---|---|---|
| Cold restart | 16/18 | 3,714,286 | 192,266 | $6.60 | 1185s |
| Handoff file | 18/18 | 4,881,539 | 281,306 | $8.95 | 1552s |
| OnlyBots | 18/18 | 3,013,727 | 173,527 | $5.31 | 969s |
Paired comparison
Each OnlyBots run compared with the same task and repeat under the other condition. Intervals are 95% bootstrap intervals (10,000 samples) on the median per-run change.
| OnlyBots vs | Median token change | 95% interval | Runs with fewer tokens | Total token change |
|---|---|---|---|---|
| Cold restart | −10.2% | −18.6% to −1.7% | 14 of 18 | −700,559 |
| Handoff file | −36.9% | −42.7% to −26.6% | 17 of 18 | −1,867,812 |
By task
| Task | What it tests | Cold restart | Handoff file | OnlyBots | |||
|---|---|---|---|---|---|---|---|
| Done | Tokens | Done | Tokens | Done | Tokens | ||
Resumeresume-idempotency-header |
Finish a half-investigated API idempotency fix. | 1/3 | 318,337 | 3/3 | 381,331 | 3/3 | 244,966 |
Resumeresume-token-expiry |
Continue a token-expiry compatibility repair. | 3/3 | 209,156 | 3/3 | 298,912 | 3/3 | 190,957 |
Learned gotchalearned-gotcha-js-verifier |
Avoid a test-command trap found in an earlier session. | 3/3 | 193,074 | 3/3 | 280,999 | 3/3 | 173,165 |
Stale memorystale-customer-id-migration |
Use the newer UUID customer IDs, not older numeric assumptions. | 3/3 | 170,972 | 3/3 | 259,287 | 3/3 | 155,061 |
Project isolationproject-isolation-cache |
Don't apply another project's cache conventions. | 3/3 | 172,511 | 3/3 | 272,179 | 3/3 | 169,052 |
Controlone-shot-js-syntax |
A one-off fix with nothing worth remembering. | 3/3 | 99,805 | 3/3 | 99,539 | 3/3 | 99,120 |
Read with care
What these numbers don't show
- The cold-restart gap is concentrated. Both cold-restart failures, and about two-thirds of the total token saving against cold restarts, came from one resume task. On the other tasks the per-run saving was smaller, typically 2–10%.
- No gain when there is nothing to remember. On the one-shot control task all three conditions used about the same tokens. Continuity helps continuation work, not every task.
- Small sample. Three repeats and one task per scenario is enough to see direction, not to publish effect sizes.
- Codex only. This run did not include Claude Code or a handoff between Claude Code and Codex.
- Synthetic repositories. The tasks are designed to exercise continuity. Real projects will vary.
- Estimated cost. Spend is computed from token counts and list prices, not from invoices.
- Grouped repeats. Condition order varied by task, but repeats ran grouped by condition rather than shuffled.
Next
We plan to rerun this with current models, add Claude Code and cross-tool handoff runs, and add more tasks per scenario. This page will be updated with each run.
Try it on your own continuation work. Install takes one command.