Pipecat Evals

Test agent behavior with scripted conversations and simulated callers, run against the same agent you deploy.

Pipecat Evals is the framework’s built-in system for testing agent behavior. You describe a conversation and the behavior you expect, and Pipecat runs it against your real agent (the same pipeline, the same services, the same code) and tells you whether the expectation still holds.

There are two ways to describe that conversation. A scripted scenario writes the user’s turns out and checks what the agent does after each one: an exact phrase, a function call, a latency budget, or a natural-language eval: the judge decides.

capital_question.yaml

name: capital_question

turns:
  - user: "What is the capital of Germany?"
    expect:
      - event: response
        eval: "the response says the capital of Germany is Berlin"

A simulated scenario hands the user’s side to an LLM with a persona and a goal, and the judge decides from the whole conversation whether the agent got the job done and how well:

book_table.yaml

name: book_table

persona: |
  Jamie, booking dinner for two tonight at 6 PM. Gives a name and phone number
  when asked (Jamie Lee, 555-0142). Polite, answers one question at a time.
goal: "Book a table for two at 6 PM, then end the call."
success: "the bot confirmed a reservation for two at 6 PM"

metrics:
  - name: politeness
    criterion: "the reply is courteous, never curt or dismissive"
    min_score: 1

Both run with the same command:

pipecat eval run capital_question.yaml book_table.yaml

Why evals matter

Voice agents are probabilistic systems. The same agent can answer differently run to run, and a prompt tweak, a model upgrade, or a service swap can quietly break behavior that used to work: a function that no longer gets called, context that stops carrying across turns, an interruption that derails the conversation. Manual testing catches some of this, but it’s slow, unrepeatable, and impractical to run on every change.

Evals make agent behavior testable the way unit tests make code testable:

• Regression safety: run your scenarios after every prompt, model, or pipeline change and catch breakage before users do.
• Fast iteration: text-mode evals skip STT and TTS entirely, so a full conversation test runs in seconds with no audio service cost.
• Semantic assertions: an LLM judge checks meaning (“the response says the capital is Berlin”), not exact strings, so tests don’t break when wording changes.
• Whole-flow coverage: a simulated caller pursues a goal in its own words, so you learn whether the agent finishes the job, not just whether one scripted path still works.
• A feedback signal for AI coding assistants: evals give a coding assistant a command it can run and a pass/fail result it can read, closing the loop between writing agent code and verifying it. See The Eval Loop.

Pipecat itself relies on this framework: before every release, an eval suite drives 100+ example agents end to end, with scripted scenarios and simulations side by side.

Two kinds of scenario

Every scenario has a kind, decided by its top-level key: turns: makes it scripted, persona: makes it simulated.

Kind	You write	Best for
Scripted	The user’s turns, each with the events expected back: text_contains, a function_call with its args, within_ms, or an eval: criterion.	Checks where you control the user’s side exactly. You know what the user says, so you can assert exactly what the agent must do: this tool with these arguments, this phrase, within this budget, after this interruption.
Simulated	A persona:, a goal:, a success: criterion, and metrics:. An LLM plays the caller and hangs up when done.	Checks of a whole goal. The caller adapts to whatever the agent says, so one file covers the many paths a real conversation can take, and the judge decides whether the goal was reached and how the replies held up.

The two answer different questions. A scripted scenario asks “does the agent do this exact thing when the user says this?”, and its strength is control: the input is fixed, so the assertions can be precise. A simulation asks “does the agent get this caller to their goal?”, and its strength is flexibility: the caller adjusts to the agent’s replies, so you don’t have to write a scenario for every way the conversation could go.

Scripted Scenarios

Turns, events, and assertions: eval:, text_contains:, function calls, latency budgets, DTMF, vision, and interruptions.

Simulated Scenarios

A persona and a goal, a success criterion, judged and measured metrics, and runs.

How it works

Pipecat Evals has two halves:

1. The eval transport. Your agent runs unchanged with the eval transport. If your agent uses create_transport() and the development runner, this is already built in: start it with -t eval and it hosts a local WebSocket server speaking RTVI, instead of connecting to Daily, WebRTC, or telephony.

2. The eval harness. The harness connects to that transport as an RTVI client and plays the user’s side of the conversation: a scripted scenario’s turns, or a persona LLM’s replies. It collects the events your agent emits (transcriptions, LLM responses, spoken output, function calls, and timing) and either asserts on them in order, for a script, or hands the whole transcript to a judge, for a simulation.

When a scenario asserts on meaning rather than exact text, a judge LLM evaluates the agent’s output against a natural-language criterion: each eval: assertion in a scripted scenario, and the success: criterion plus every judged metric in a simulation. The judge runs locally with Ollama by default, or against OpenAI or any OpenAI-compatible endpoint.

Text and audio modes

Every scenario of either kind also has a modality:

Mode	User input	Agent output	Best for
Text (default)	Sent as text, bypassing the STT	LLM text; TTS is skipped automatically	Fast, cheap iteration on prompts, logic, and function calling
Audio	Synthesized by a TTS the harness runs (local by default)	Real synthesized speech, transcribed by an STT the harness runs	True end-to-end coverage of the full STT, LLM, and TTS pipeline

Text mode exercises your agent’s actual pipeline and context handling while skipping the audio services, so it costs nothing in TTS or STT usage and runs fast. Audio mode synthesizes the user’s voice (a scripted turn or a persona’s reply), streams it through your agent’s real STT, and transcribes the agent’s actual spoken audio for judging. This catches problems that only show up with real speech: turn detection, words that sound alike, barge-in. A speech-to-speech agent has no separate text LLM step, so it is evaluated in audio mode only.

Kind and modality are independent, which gives four ways to run a scenario:

	Text	Audio
Scripted	The inner loop: prompts, tool calls and their args, multi-turn context, barge-in. Run it constantly.	One behavior on the real speech path: how numbers, names, and accents transcribe; turn-taking and VAD timing; how the agent’s speech sounds.
Simulated	A flow end to end, against a caller who phrases things their own way. Conversation-level quality.	What a real caller experiences. The pre-ship pass, and the only option for speech-to-speech agents.

What you can test

• Response content: substring checks (text_contains) or semantic judging (eval) of the agent’s replies.
• Multi-turn context: verify the agent remembers earlier turns.
• Function calling: assert that specific tools were called, with specific arguments, in a scripted turn or across a whole simulated conversation.
• Interruptions: barge in mid-response and verify the agent recovers (send_after).
• Latency: per-event budgets with within_ms, or the slowest reply of a simulation with the latency measure.
• Goals and outcomes: whether a simulated caller got what they came for, or was turned down correctly when they shouldn’t.
• Conversation quality: per-reply criteria such as politeness or brevity, scored across every reply of a simulation.
• Vision: serve an image when the agent requests one and judge its description.

YAML or Python

Scenarios are YAML files, so they’re easy to write, review, and share. Everything is also available as a library: load and run scenarios of either kind programmatically, build them in code, inject a custom judge or persona LLM, or orchestrate whole suites from your own tooling. See Using the Library.

Requirements

• Pipecat CLI: the pipecat eval commands ship with the CLI extra: uv tool install "pipecat-ai[cli]". If you’ve added pipecat-ai[cli] to your project instead, run them with uv run pipecat eval (just like uv run bot.py). The same commands are also available as python -m pipecat.evals.
• A judge LLM (for eval: assertions, and for every simulation): Ollama by default (ollama pull gemma4:12b), or point the scenario’s judge: block at OpenAI or any OpenAI-compatible endpoint.
• A persona LLM (simulations only): by default the same local Ollama model as the judge, so a simulation needs no API key. A simulator: block can name another Ollama model, or any OpenAI-compatible model that supports function calling through a factory:.
• Audio services (audio mode only): the harness needs a TTS to synthesize the user’s voice and an STT to transcribe the agent’s speech. Both can be local models or HTTP-based services; the defaults are local (Kokoro and Moonshine or Whisper, installed with uv add "pipecat-ai[kokoro,moonshine]" or uv add "pipecat-ai[kokoro,whisper]"), which download once on first use and run with no keys and no per-run cost. WebSocket-streaming services aren’t supported here, which keeps the harness simple.
• Your agent’s own credentials: the agent under test is your real agent, so it needs the same service API keys it normally would.

Production evaluation

Pipecat Evals is built for development: fast, local, repeatable, and run on every change. Once your agent is deployed, third-party evaluation platforms complement it with testing and monitoring at production scale:

• Testing over the deployed path: test calls over API, WebSocket, or telephony, with varied caller populations, real phone-network conditions, and load, exercising the transport your users actually hit.
• Observability: continuous evaluation of live traffic, with automated quality scoring of calls and transcripts, audio-signal metrics, and trends tracked over time to catch quality drift.

Bluejay

Simulation, observability, and evaluation platform with native Pipecat Cloud integration. Supports no-code API, WebSocket, and telephony testing.

Cekura

Automated testing and monitoring platform with native Pipecat Integration for WebRTC/Text based testing and support for Mock Tools, Custom Dynamic Variables and more!

Coval

AI-native simulation and evaluation platform for voice agents, trusted by QA, Engineering, Operations, AI, and Executive teams.

Roark

Simulation, observability, tracing, and metrics with native Pipecat Cloud integration over the Daily transport: personas, flows, built-in metrics, and a drop-in observer for capturing production calls.

Arize

Observability and online evaluation for voice agents. Auto-instrument Pipecat with OpenInference (OpenTelemetry) to trace every turn, then run LLM-as-judge evals on live traffic in Arize AX or open-source Phoenix.

Building an evaluation integration for Pipecat? We welcome contributions to this page. Open a PR on the docs repository.

Pipecat’s other building blocks feed into any evaluation workflow: Metrics for TTFB, processing time, and usage; Saving Transcripts for offline analysis; OpenTelemetry for latency traces; and Observers for custom instrumentation.

Next steps

Quickstart

Run a scripted scenario and a simulation against an existing agent in a few minutes.

Scripted Scenarios

The full scripted format: turns, expectations, modalities, and the judge.

Simulated Scenarios

Personas, goals, success criteria, judged and measured metrics, and runs.

Eval Suites

Spawn multiple agents and run many scenarios of both kinds concurrently from a manifest.

The Eval Loop

Let a coding assistant write agent code, run evals, and iterate automatically until the agent is better.