Evaluating every trace, at a cost that finally works
Zeebu reconciles carrier invoices and usage records at scale, where one wrong settlement is a wrong payout. Evaluating every trace with a frontier model was too expensive, so most went unchecked. Triage checks and Trodo’s own models changed the bill.
Evaluation cost forced a sample, so failures were found by customers
Running every trace through a frontier model as a judge made the evaluation bill larger than the problem it solved, so the team sampled. Evaluations, where they existed, were a notebook someone ran by hand on a slice of traffic, not a score attached to every trace. Which version of the agent had produced a failure lived in a spreadsheet that was often a version behind.
So failures surfaced when a customer flagged a bad reconciliation, and the first hour after that went on rebuilding context that should already have existed:which trace, which score, which change, in what order.
Cheap checks first, Trodo’s models for the hard cases, every trace covered
Cheap code and semantic checks settle most traces first. Only the ones they cannot settle go to Trodo’s own models, which return several checks in one request at a fraction of a frontier model’s cost. That is what made evaluating all of it affordable, and failures now group into issues by cause. A failing reconciliation now opens straight into the score that flagged it, the tool calls it made, and the version of the agent that produced it.
What made it stick day to day was that nobody had to build a wrapper to make it usable. Engineers see what failed, diff a regression against the last known-good version, and roll back or ship the fix without leaving Trodo.
| Before Trodo | With Trodo |
|---|---|
| A sample of traces evaluated, because judging all of them cost too much | Every trace evaluated, then score to cause to fix in one view |
| Changes shipped without a clear audit trail | Every version tied to the traces and scores it produced |
| Incident reviews needed two separate data exports, one per team | Engineering and operations pull from the same shared view |
Fixing and rolling back stopped being a project
Indexed: what it cost to judge a trace with a frontier model before, set to 100.
Minutes, measured from the decision to roll back to the previous version being live.
Hours per developer per week, from the team's own before-and-after estimate.
Judging every trace with a frontier model cost us far too much, so we sampled. Triage checks and Trodo’s own models cut the cost of an evaluation by about 80%, and now every trace is covered.
Every past failure and its fix became the onboarding material
- New engineers ramping onto the agent codebase get productive faster, because every past regression and the fix for it are still attached to the trace that caused them.
- Cross-functional incident reviews went from two separate exports to one view both teams read directly.
- Time that used to go into finding context now goes into shipping, which is where the eight hours actually show up.