OrbtCatching the failures that never threw
Orbt has no sales team and no onboarding call. The assistant in the Orbt Hub is the only thing between a first-time user and a funded position, and when it was wrong nothing in the trace said so.
It did not fail loudly. It almost never failed loudly.
The assistant answers balance and rate questions, walks people through allocation flows, and calls live into the protocol to do it. It answered fluently, immediately, and often enough with a number that had nothing to do with the tool output it had just received, or a walkthrough of a flow redesigned three days earlier.
A conventional trace looked identical whether the trace was right or quietly wrong. No failed span, no alert, nothing telling anyone to go and look.
So the bottleneck was never fixing bugs. It was finding them. Detection depended on a user getting frustrated enough to complain, which for a self-serve financial product mostly meant they left instead.
Every trace checked, not a sample
Orbt put Trodo’s checks on every production trace, testing whether a number in the response actually traces back to a tool call in that same trace, and whether the right tool was selected at all.
Failures that used to be invisible now surface as flagged, clustered issues the moment they happen, ranked by how many users they are touching.
| Before Trodo | With Trodo |
|---|---|
| A user noticed, got annoyed, and said something, or quietly left | A check flags the trace as it happens |
| Silent failures arrived one support ticket at a time | Grouped by root cause, so one fix closes a whole cluster |
| The failing span had to be reconstructed before work could start | The span and the reason it failed arrive together |
Detection stopped waiting on a complaint
Hours. 'Before' is a conservative reading of days. The true figure depended on how long it took a user to complain.
Indexed: what it cost to judge a trace before, set to 100.
Percent of traces with a check attached.
Every trace is evaluated now, at about 60% less than it cost us to judge a sample. Silent failures arrive grouped by cause, so one fix closes the whole set.
One fix now closes a cluster, not a ticket
- Silent failures that used to surface one complaint at a time are grouped by root cause, so a single fix clears the whole set rather than chasing each report.
- Engineers get the failing span and the reason it failed together, so there is no reconstruction step before the actual work begins.
- For a product with no onboarding call, catching a bad answer before the user does is the difference between a fix and a churned account.