*This is the markdown version of this page, for agents. To see the web page, open https://trodo.ai/models/latest/benchmarks?view=html (or use the Human / Agent switch at the bottom)*

# Hartley 1 benchmarks: accuracy, price and speed

> Hartley 1 against Jev 1.13, Microsoft Decision-1, CLM-8B and the frontier models: accuracy on three benchmark families, price per million tokens, and speed.

## Accuracy: Real agent traces. Eleven points ahead.

| Benchmark | What it covers | Questions | Hartley 1 | Jev 1.13 | Microsoft Decision-1 | CLM-8B |
|---|---|---|---|---|---|---|
| Agent-trace judgment | Real production traces: instructions followed, tools used right, answers grounded, tasks finished | 1,860 | 73.7% | 62.7% | 64.3% | 50.3% |
| Typed decisions | AG News, Banking77, BoolQ, MNLI, SST-5, TREC, DBpedia and more | 1,264 | 86.9% | 84.7% | 85.6% | 44.1% |
| Coding-agent verification | SWE-Gym, DeepSWE and Terminal-Bench: did the run pass its tests? | 1,413 | 71.8% | 69.1% | 72.8% | 66.7% |

## Where the gap opens: judging the details.

| Question | Hartley 1 | Jev 1.13 | Microsoft Decision-1 | CLM-8B |
|---|---|---|---|---|
| Followed its system prompt | 34.7% | 12% | 20% | 6.7% |
| Efficient path | 83% | 16% | 31.9% | 12.8% |
| Answer supported by tools | 62.8% | 23.4% | 17% | 7.4% |
| Instruction adherence | 62.8% | 24.5% | 22.3% | 11.7% |
| Primary failure mode | 76.7% | 61.6% | 68.6% | 18.6% |
| Context sufficient | 68.1% | 64.8% | 65.9% | 80.2% |
| Task completed | 51.1% | 63.8% | 62.8% | 71.3% |
| Repeated action loop | 87.1% | 67.1% | 82.4% | 36.5% |
| Needs human review | 89.4% | 68.1% | 74.5% | 55.3% |
| Tool output used | 72.4% | 69% | 70.1% | 71.3% |
| Custom eval criteria | 68.1% | 66.8% | 68.1% | 41.9% |
| Numbers match the tools | 70.6% | 70.6% | 70.6% | 66.7% |
| Tool arguments grounded | 70.1% | 64.9% | 58.4% | 57.1% |

## Sure when it’s right.

- Right on its most confident half: Hartley 1 91.1%, Jev 1.13 82.5%, Microsoft Decision-1 82.7%, CLM-8B 52.5%
- Wrong at 90%+ confidence: Hartley 1 0.8%, Jev 1.13 5.1%, Microsoft Decision-1 9.6%, CLM-8B 12.8%

## Longer traces. Same accuracy.

| Trace length | Hartley | Jev 1.13 |
|---|---|---|
| ~1.1K | 74.5% | 63% |
| ~4K | 73.8% | 63.9% |
| ~8K | 74.2% | 63.8% |
| ~16K | 72.5% | 61.9% |

## Price: The price of intelligence. Per million tokens.

| Model | Input per million tokens | Output per million tokens | Input price against Hartley |
|---|---|---|---|
| Claude Fable 5.1 (Anthropic) | $10 | $50 | 250× |
| GPT-6 Astra (OpenAI) | $10 | $50 | 250× |
| Llama 4.1 405B (Meta · on Microsoft Foundry) | $5 | $15 | 125× |
| Claude Opus 5.5 (Anthropic) | $4 | $20 | 100× |
| Claude Sonnet 5.5 (Anthropic) | $2 | $10 | 50× |
| GPT-6.1 Sol (OpenAI) | $2 | $10 | 50× |
| Gemini 3.1 Pro (Google) | $2 | $12 | 50× |
| DeepSeek-V4 Pro (DeepSeek · on Microsoft Foundry) | $2 | $8 | 50× |
| Claude Haiku 4.5 (Anthropic) | $1 | $5 | 25× |
| Kimi K2.6 (Moonshot · on Microsoft Foundry) | $0.95 | $4 | 24× |
| GPT-5.4 mini (OpenAI) | $0.75 | $4.5 | 19× |
| Gemini 3.8 Flash (Google) | $0.75 | $3.75 | 19× |
| Gemini 3.1 Flash-Lite (Google) | $0.25 | $1.5 | 6.3× |
| Clef (Cloudflare · Workers AI) | $0.24 | $0 | 6× |
| GPT-5.4 nano (OpenAI) | $0.2 | $1.25 | 5× |
| DeepSeek-V4 Flash (DeepSeek · on Microsoft Foundry) | $0.2 | $0.8 | 5× |
| Llama 4 Maverick (Meta · on Together AI) | $0.15 | $0.6 | 3.8× |
| Clef-flash (Cloudflare · Workers AI) | $0.09 | $0 | 2.3× |
| Jev 1.13 (TypeSafe) | $0.042 | $0 | 1.1× |
| Hartley 1 (Trodo) | $0.04 | $0 | 1× |

List prices, 4 October 2026.

## Speed: up to 25× faster

Against a frontier model that reasons for about 1,700 tokens before three short verdicts, at 90 tokens a second: about 22.6 s, against Hartley’s ~0.9 s of model time on the live API.

- **~0.5 s** Model time on a typical trace, five questions
- **< 3 s** On traces up to about 7,000 tokens
- **16K** Tokens per state, accuracy holds
- **Up to 16×** Faster when you ask again about the same trace

| State | First read | Asked again |
|---|---|---|
| ~1.2K tokens (typical trace) | 0.53 s | 189 ms |
| 3.3K tokens | 1.3 s | 210 ms |
| 7.3K tokens | 2.9 s | 288 ms |
| 11.5K tokens | 6.6 s | 400 ms |

## Twenty questions. One read.

Hartley reads the input once and answers every question from that read: twenty questions about one trace cost barely more than one.

## Pages

- Hartley 1: https://trodo.ai/models/latest.md
- Benchmarks (accuracy, price, speed): https://trodo.ai/models/latest/benchmarks.md
- API access (private beta): https://console.trodo.ai/signup
- API docs: https://docs.trodo.ai/console

---

Web version: https://trodo.ai/models/latest/benchmarks?view=html · Everything about Trodo for agents: https://trodo.ai/index.md