OpenVoiceCS-Bench: Measuring Whether an Agent Actually Resolved the Call

Resolution, Not Vibes

Most voice-agent evaluation asks whether the agent sounded helpful. OpenVoiceCS-Bench asks whether it actually resolved the customer’s problem without breaking policy — and makes it prove that with a replayable trace. Every scenario defines a goal, an initial system state, a set of replayable tools, and a standard operating procedure; every score is computed deterministically from what the agent actually did, not from what a judge felt about it.

The leaderboard ranks 52 of 58 attempted models on the text_to_action track. The spread is real: overall scores run from 25.38 to 88.87, and task_success — did the customer’s problem actually get resolved — runs from 0.010 to 0.754. Even the leader fails a quarter of scenarios.

The rest of this report is the methodology: what the scenarios contain, how replay scoring works, the rules that keep failure signals about the model rather than the harness, and the limitations we publish next to the numbers instead of hiding behind them.

What The Benchmark Measures

The corpus is 220 synthetic scenarios across five tracks. All data is synthetic — no real customer records, no real personal data, no recordings of real people. Six domains carry the load — retail (38), travel (38), fintech sandbox (37), telecom (36), healthcare admin (35), and SaaS support (35), plus a single utility-support scenario — with 120 TTS-generated audio variants. The corpus splits into public-dev (99 scenarios) for development and reproduction and sealed-test (121 scenarios) that is never published; split commitments publish counts and hashes without revealing sealed IDs.

Each trace is scored on eight deterministic metrics, combined into an overall score out of 100 with fixed weights:

Alongside the metrics, every report carries the operational evidence that decides whether an agent is deployable at all: per-turn latency percentiles, tool-call efficiency, token cost, flake rate across trials, and measurement coverage.

How Scoring Works

A scenario is a contract. It defines the customer’s goal, the initial state of the backing systems, a set of replayable tools — each with preconditions and state effects — and the standard operating procedure the agent is supposed to follow. The agent produces a trace: its messages, its tool calls, and the policy events those triggered.

The anchor for all of this is the oracle agent: a reference agent that performs each scenario correctly by construction. It passes 220 of 220 scenarios, so any oracle regression means the harness broke, not the agent. At the other end, a no-op agent that replies politely and never acts scores 24.89 overall — the floor that makes the scale readable.

Models are exercised through two collection paths: providers with native tool calling use it directly, and chat-only models run a stepwise JSON action loop bounded to eight rounds, which is what makes the long tail of the OpenRouter catalog scoreable at all.

What failure looks like in practice is usually not spectacular. The most common single cause of a failed scenario, even for the leader, is performing the customer-facing action and then omitting a required piece of secondary bookkeeping:

called : verify_identity, perform_service_action(resolution=completed)
missing: create_case(case_id=case_fs_001, reason=card_dispute)

Measurement Honesty

The rule underneath everything: a failure to measure is not a measurement. Every failed trial is attributed to either the infrastructure or the model. Rate limits, provider 404s, empty balances, and transport errors are infrastructure: they are excluded from metric means and reported as reduced measurement_coverage. Refusals, malformed actions, and wrong tool calls are the model’s problem: they score what they score.

On top of that classification sits a hard floor: a model is ranked only if at least 90% of its trials produced a usable trace. Six of the 58 attempted models fall below it and are listed as unmeasured — a different statement from “scored badly”. The rule is load-bearing: in early harness runs, before the floor existed, one free-tier model ranked first on the strength of 7% trial coverage — the handful of trials that survived rate limiting happened to go well — and two models whose endpoints returned HTTP 404 for every request ranked at 0.00, as though they had been evaluated and failed everything.

Four rules keep failure signals about the model rather than the harness, and each is pinned by a regression test:

The six exclusions, with coverage and cause, are published next to the leaderboard rather than in a footnote.

Token Efficiency

Per-token pricing changes; the number that transfers is how many tokens a system burns to actually resolve something. The leaderboard therefore reports tokens_per_success: total tokens spent, divided by the number of scenarios actually resolved. The denominator is the point. An agent that spends heavily and fails is charged for the budget it burned on those failures, which is exactly how a deployment experiences it.

The ~23× spread is the deployment-relevant result in this release: two agents with similar-looking overall scores can differ by an order of magnitude in what they cost to run per resolved case.

Multi-Turn Conversations

A real support call is a negotiation, not a single utterance. The harness replays conversations turn by turn: the agent is called once per customer turn; later customer turns are withheld, so an agent cannot read the customer’s next objection before answering the current one; the agent’s own prior replies are fed back to it; and state advances between turns, so turn N sees the account exactly as turn N−1 left it. Single-turn scenarios still call the agent exactly once with the scenario untouched.

Honest Limitations

These are the caveats we would rather list than have a reader discover. Every one is measured against the current repository, not asserted, and each carries a reproduction command in docs/known-limitations.md.

The grounding metric is provisional

factual_grounding is a literal phrase matcher, and it conflates three different things. It misses synonymy: fee_waived accepts “no change fee” and “fee waiver”, but an agent saying “rebooked you at no charge” is marked ungrounded. It correctly catches genuine omissions, like never stating a $12.00 credit amount. And it punishes honest failure reports: where a scenario injects a tool failure and the agent truthfully says “I couldn’t complete this, I’ve escalated it”, the required claim completed is absent and the agent is penalised for accuracy.

The consequence is measured: removing the metric leaves the top three unchanged but moves 40 of 52 models somewhere in the ordering. The podium is trustworthy; fine-grained mid-table ordering is not. A semantic grader to replace this metric is the single most valuable contribution anyone can make to the project.

One sweep, three trials, no confidence intervals

Nothing in this release separates models a couple of points apart. Until repeated sweeps produce intervals, treat a two-point gap as noise. Gaps under a few points are not results, and we ask you not to quote them as results.

One track of five

The sweep covers text_to_action only. robustness, adversarial_compliance, audio_to_action, and end_to_end_voice have no provider sweep, so this is not a voice-agent leaderboard yet — it is a text-track result. There is also no hosted sealed evaluator, so published numbers carry no contamination control.

Tool-call rate is sensitive to cosmetic prompt wording

Holding the model, scenarios, and scorer fixed and changing only how tool arguments are annotated in the prompt:

Marking an argument system-assigned in any wording makes models call the tool less often — a 2.5× swing in task_success from wording alone. Absolute task_success is therefore not stable across harness revisions, and re-scored older runs are not comparable to a fresh sweep.

“Safety” is trivially satisfied by inaction

The no-op baseline — reply politely, never act — scores 0.990 on safety and 24.89 overall. Safety is a don’t-do-harm measure and must be read next to task_success, never quoted alone. The no-op’s 24.89 is also what makes the scale readable (Figure 4): the usable band is roughly 25–100, and the lowest-ranked model at 25.38 is indistinguishable from doing nothing at all.

Smaller prints

Binary trial gating compresses the passed family: a trial passes only when all seven deterministic metrics equal exactly 1.0, so an agent at 0.95 on six metrics records the same as one that crashed — read the per-metric means instead. Latency measures the full JSON action loop, up to eight sequential provider calls, and is not comparable to a single-turn voice latency budget. And the 120 audio variants are TTS-generated from a small speaker set: claims about robustness to real-world audio are not supported by this release.

Check It Yourself

The leaderboard is rebuildable from published artifacts, offline, with no API key. Scoring a model yourself needs an OpenRouter key. Every release artifact is SHA-256 pinned and cross-validated, and CI runs the full verification gate on every push.

# Score one model against the track (needs OPENROUTER_API_KEY)
python scripts/run_openvoicecs.py score-provider \
  --provider openrouter --model google/gemini-2.5-flash-lite \
  --track text_to_action --trials 3 --json-trace --output report.json

# Rebuild the leaderboard from the published reports (offline, no API key)
python scripts/build_leaderboard.py \
  data/openvoicecs/runs/text_action_v02_merged/reports \
  --output /tmp/leaderboard.csv

The full sweep runs unattended on a spot VM, is resumable, and took roughly two hours. Code is Apache 2.0, data is CC BY 4.0, and all of it — scenarios, runs, limitations, regression tests — lives at github.com/0daycloud/openvoicecs-bench.

The ranked table itself, with the excluded models and the reference anchors, is on the results page. Read it with this report beside it — that is what the two pages are for.