> ## Documentation Index > Fetch the complete documentation index at: https://daily-docs-flows-declarative.mintlify.site/llms.txt > Use this file to discover all available pages before exploring further. # Pipecat Evals > Test agent behavior with scripted conversations and simulated callers, run against the same agent you deploy. Pipecat Evals is the framework's built-in system for testing agent behavior. You describe a conversation and the behavior you expect, and Pipecat runs it against your real agent (the same pipeline, the same services, the same code) and tells you whether the expectation still holds. There are two ways to describe that conversation. A **scripted** scenario writes the user's turns out and checks what the agent does after each one: an exact phrase, a function call, a latency budget, or a natural-language `eval:` the judge decides. ```yaml capital_question.yaml theme={null} name: capital_question turns: - user: "What is the capital of Germany?" expect: - event: response eval: "the response says the capital of Germany is Berlin" ``` A **simulated** scenario hands the user's side to an LLM with a persona and a goal, and the judge decides from the whole conversation whether the agent got the job done and how well: ```yaml book_table.yaml theme={null} name: book_table persona: | Jamie, booking dinner for two tonight at 6 PM. Gives a name and phone number when asked (Jamie Lee, 555-0142). Polite, answers one question at a time. goal: "Book a table for two at 6 PM, then end the call." success: "the bot confirmed a reservation for two at 6 PM" metrics: - name: politeness criterion: "the reply is courteous, never curt or dismissive" min_score: 1 ``` Both run with the same command: ```bash theme={null} pipecat eval run capital_question.yaml book_table.yaml ``` ## Why evals matter Voice agents are probabilistic systems. The same agent can answer differently run to run, and a prompt tweak, a model upgrade, or a service swap can quietly break behavior that used to work: a function that no longer gets called, context that stops carrying across turns, an interruption that derails the conversation. Manual testing catches some of this, but it's slow, unrepeatable, and impractical to run on every change. Evals make agent behavior testable the way unit tests make code testable: * **Regression safety**: run your scenarios after every prompt, model, or pipeline change and catch breakage before users do. * **Fast iteration**: text-mode evals skip STT and TTS entirely, so a full conversation test runs in seconds with no audio service cost. * **Semantic assertions**: an LLM judge checks meaning ("the response says the capital is Berlin"), not exact strings, so tests don't break when wording changes. * **Whole-flow coverage**: a simulated caller pursues a goal in its own words, so you learn whether the agent finishes the job, not just whether one scripted path still works. * **A feedback signal for AI coding assistants**: evals give a coding assistant a command it can run and a pass/fail result it can read, closing the loop between writing agent code and verifying it. See [The Eval Loop](/pipecat/evals/the-eval-loop). Pipecat itself relies on this framework: before every release, an eval suite drives 100+ example agents end to end, with scripted scenarios and simulations side by side. ## Two kinds of scenario Every scenario has a **kind**, decided by its top-level key: `turns:` makes it scripted, `persona:` makes it simulated. | Kind | You write | Best for | | ------------- | --------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Scripted** | The user's turns, each with the events expected back: `text_contains`, a `function_call` with its args, `within_ms`, or an `eval:` criterion. | Checks where you control the user's side exactly. You know what the user says, so you can assert exactly what the agent must do: this tool with these arguments, this phrase, within this budget, after this interruption. | | **Simulated** | A `persona:`, a `goal:`, a `success:` criterion, and `metrics:`. An LLM plays the caller and hangs up when done. | Checks of a whole goal. The caller adapts to whatever the agent says, so one file covers the many paths a real conversation can take, and the judge decides whether the goal was reached and how the replies held up. | The two answer different questions. A scripted scenario asks "does the agent do this exact thing when the user says this?", and its strength is control: the input is fixed, so the assertions can be precise. A simulation asks "does the agent get this caller to their goal?", and its strength is flexibility: the caller adjusts to the agent's replies, so you don't have to write a scenario for every way the conversation could go. Turns, events, and assertions: `eval:`, `text_contains:`, function calls, latency budgets, DTMF, vision, and interruptions. A persona and a goal, a success criterion, judged and measured metrics, and runs. ## How it works Pipecat Evals has two halves: 1. **The eval transport.** Your agent runs unchanged with the eval transport. If your agent uses `create_transport()` and the development runner, this is already built in: start it with `-t eval` and it hosts a local WebSocket server speaking RTVI, instead of connecting to Daily, WebRTC, or telephony. 2. **The eval harness.** The harness connects to that transport as an RTVI client and plays the user's side of the conversation: a scripted scenario's turns, or a persona LLM's replies. It collects the events your agent emits (transcriptions, LLM responses, spoken output, function calls, and timing) and either asserts on them in order, for a script, or hands the whole transcript to a judge, for a simulation. When a scenario asserts on meaning rather than exact text, a **judge LLM** evaluates the agent's output against a natural-language criterion: each `eval:` assertion in a scripted scenario, and the `success:` criterion plus every judged metric in a simulation. The judge runs locally with [Ollama](https://ollama.com) by default, or against OpenAI or any OpenAI-compatible endpoint. ### Text and audio modes Every scenario of either kind also has a **modality**: | Mode | User input | Agent output | Best for | | ------------------ | -------------------------------------------------------- | --------------------------------------------------------------- | --------------------------------------------------------------- | | **Text** (default) | Sent as text, bypassing the STT | LLM text; TTS is skipped automatically | Fast, cheap iteration on prompts, logic, and function calling | | **Audio** | Synthesized by a TTS the harness runs (local by default) | Real synthesized speech, transcribed by an STT the harness runs | True end-to-end coverage of the full STT, LLM, and TTS pipeline | Text mode exercises your agent's actual pipeline and context handling while skipping the audio services, so it costs nothing in TTS or STT usage and runs fast. Audio mode synthesizes the user's voice (a scripted turn or a persona's reply), streams it through your agent's real STT, and transcribes the agent's actual spoken audio for judging. This catches problems that only show up with real speech: turn detection, words that sound alike, barge-in. A speech-to-speech agent has no separate text LLM step, so it is evaluated in audio mode only. Kind and modality are independent, which gives four ways to run a scenario: | | Text | Audio | | ------------- | ---------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------- | | **Scripted** | The inner loop: prompts, tool calls and their args, multi-turn context, barge-in. Run it constantly. | One behavior on the real speech path: how numbers, names, and accents transcribe; turn-taking and VAD timing; how the agent's speech sounds. | | **Simulated** | A flow end to end, against a caller who phrases things their own way. Conversation-level quality. | What a real caller experiences. The pre-ship pass, and the only option for speech-to-speech agents. | ## What you can test * **Response content**: substring checks (`text_contains`) or semantic judging (`eval`) of the agent's replies. * **Multi-turn context**: verify the agent remembers earlier turns. * **Function calling**: assert that specific tools were called, with specific arguments, in a scripted turn or across a whole simulated conversation. * **Interruptions**: barge in mid-response and verify the agent recovers (`send_after`). * **Latency**: per-event budgets with `within_ms`, or the slowest reply of a simulation with the `latency` measure. * **Goals and outcomes**: whether a simulated caller got what they came for, or was turned down correctly when they shouldn't. * **Conversation quality**: per-reply criteria such as politeness or brevity, scored across every reply of a simulation. * **Vision**: serve an image when the agent requests one and judge its description. ## YAML or Python Scenarios are YAML files, so they're easy to write, review, and share. Everything is also available as a library: load and run scenarios of either kind programmatically, build them in code, inject a custom judge or persona LLM, or orchestrate whole suites from your own tooling. See [Using the Library](/pipecat/evals/library). ## Requirements * **Pipecat CLI**: the `pipecat eval` commands ship with the CLI extra: `uv tool install "pipecat-ai[cli]"`. If you've added `pipecat-ai[cli]` to your project instead, run them with `uv run pipecat eval` (just like `uv run bot.py`). The same commands are also available as `python -m pipecat.evals`. * **A judge LLM** (for `eval:` assertions, and for every simulation): Ollama by default (`ollama pull gemma4:12b`), or point the scenario's `judge:` block at OpenAI or any OpenAI-compatible endpoint. * **A persona LLM** (simulations only): by default the same local Ollama model as the judge, so a simulation needs no API key. A `simulator:` block can name another Ollama model, or any OpenAI-compatible model that supports function calling through a `factory:`. * **Audio services** (audio mode only): the harness needs a TTS to synthesize the user's voice and an STT to transcribe the agent's speech. Both can be local models or HTTP-based services; the defaults are local (Kokoro and Moonshine or Whisper, installed with `uv add "pipecat-ai[kokoro,moonshine]"` or `uv add "pipecat-ai[kokoro,whisper]"`), which download once on first use and run with no keys and no per-run cost. WebSocket-streaming services aren't supported here, which keeps the harness simple. * **Your agent's own credentials**: the agent under test is your real agent, so it needs the same service API keys it normally would. ## Production evaluation Pipecat Evals is built for development: fast, local, repeatable, and run on every change. Once your agent is deployed, third-party evaluation platforms complement it with testing and monitoring at production scale: * **Testing over the deployed path**: test calls over API, WebSocket, or telephony, with varied caller populations, real phone-network conditions, and load, exercising the transport your users actually hit. * **Observability**: continuous evaluation of live traffic, with automated quality scoring of calls and transcripts, audio-signal metrics, and trends tracked over time to catch quality drift. Simulation, observability, and evaluation platform with native Pipecat Cloud integration. Supports no-code API, WebSocket, and telephony testing. Automated testing and monitoring platform with native Pipecat Integration for WebRTC/Text based testing and support for Mock Tools, Custom Dynamic Variables and more! AI-native simulation and evaluation platform for voice agents, trusted by QA, Engineering, Operations, AI, and Executive teams. Simulation, observability, tracing, and metrics with native Pipecat Cloud integration over the Daily transport: personas, flows, built-in metrics, and a drop-in observer for capturing production calls. Observability and online evaluation for voice agents. Auto-instrument Pipecat with OpenInference (OpenTelemetry) to trace every turn, then run LLM-as-judge evals on live traffic in Arize AX or open-source Phoenix. Building an evaluation integration for Pipecat? We welcome contributions to this page. Open a PR on the [docs repository](https://github.com/pipecat-ai/docs). Pipecat's other building blocks feed into any evaluation workflow: [Metrics](/pipecat/fundamentals/metrics) for TTFB, processing time, and usage; [Saving Transcripts](/pipecat/fundamentals/saving-transcripts) for offline analysis; [OpenTelemetry](/api-reference/server/utilities/opentelemetry) for latency traces; and [Observers](/api-reference/server/utilities/observers/observer-pattern) for custom instrumentation. ## Next steps Run a scripted scenario and a simulation against an existing agent in a few minutes. The full scripted format: turns, expectations, modalities, and the judge. Personas, goals, success criteria, judged and measured metrics, and runs. Spawn multiple agents and run many scenarios of both kinds concurrently from a manifest. Let a coding assistant write agent code, run evals, and iterate automatically until the agent is better.