Youssef

voice-evals

Reproducible evaluation for voice agents: replay scoring, scripted live probes over WebSocket, and CI gates for WER, latency, barge-in behavior and task outcomes.

Code on GitHub pypi.org PyPIKaggle

quality gates on every release
4

Word error rate, response time, behaviour when interrupted, and whether the caller's task got done.

Language
Python
Licence
Apache-2.0
Last push
4 Oct 2026
Stars
1

What it is

Reproducible evaluation for voice agents. Score recorded calls, run scripted conversations over WebSocket, and turn transcription, latency, interruptions and task outcomes into CI gates.

Watch the demo · Quick start · Kaggle notebooks · Live probe · Documentation

How it works

View full-size diagram · Mermaid source

One evaluator, two entry points. A scored probe exports the same replay format used for offline calls. Replay reproduces the legacy scores from that recording; a new live call can vary with the agent, provider and network.

Results

You have… Start here What you get
Recorded transcripts and timings voice-eval run calls.jsonl Metrics, per-call details and configurable gates
A live WebSocket voice agent voice-eval probe scenario.json A scripted call, interruption observations and replayable artifacts
No agent or credentials yet Offline quick start or Kaggle A complete offline run you can reproduce yourself

Try it

git clone https://github.com/Gjusev/voice-evals.git
cd voice-evals
python -m pip install voice-evals
voice-eval run evals/data/demo_calls.jsonl