A token ledger tells you how much a model cost. It does not tell you whether the spend was productive, whether a cheaper recipe would work, or whether projected savings survived deployment. The visual below shows what an outcomes-oriented view looks like for one Python bug-fix workload.
What the diagram says, in plain terms
We start with real system receipts. Then we connect the spend to a durable outcome: a merged, passing Python bug fix that stays in place for 14 days. That denominator turns tokens into cost per durable success.
Next we run a controlled A/B experiment on comparable work. We compare the current execution package against a candidate package and decide which to promote. The outcome stage separates the estimated effect from what is actually realized after rollout.
Why this is useful
- Business: You see cost per real outcome, not just consumption. A savings number is either estimated, approved, or realized.
- Engineering: The controllable object is the execution package — model, reasoning level, context, tools, skills, cache, harness, and guardrails — not just “use a cheaper model.”
- Risk: The experiment, owner, rollout plan, and escalation path are visible. The recommendation is to keep the frontier model for complex or high-risk issues.
- Platform: Repeated reasoning can become a reusable repository skill, so the same work is cheaper next time.
What the numbers mean
The example uses a one-month slice of Python bug fixes in a mature repository. The current package costs $39.67 per durable success; the candidate package costs $20.26 in the experiment. The $2.9M annual savings is an estimate attached to an approved rollout plan, not a booked result.
Research basis and interpretation. The numbers come from Proverify.ai’s AI Factory Control Plane research prototype, not a reported customer result. They are intended to show what decision-grade TokenOps reporting can look like: observed inputs, an attributed durable outcome, an experimental comparison between execution packages, a named recommendation and rollout plan, and an estimated effect to be verified in production.