Evals Quickstart

Run your first Pipecat behavioral evals against an existing agent: a scripted scenario, then a simulated caller with a goal.

This guide takes an existing agent, starts it with the eval transport, runs a two-turn scripted scenario against it, and then lets a simulated caller hold a conversation with it. Total time: a few minutes.

Prerequisites

• A working Pipecat agent that uses create_transport() and the development runner (the standard pattern from the quickstart and all Pipecat examples), with its usual service API keys in .env.
• The Pipecat CLI: uv tool install "pipecat-ai[cli]" (or add pipecat-ai[cli] to your project and run the commands below with uv run pipecat eval).
• Ollama, with ollama pull gemma4:12b. It runs the judge, and in the last step the simulated caller, locally with no API key. (It’s the default; any other LLM works through a factory:, see Scenario Configuration.)

1. Run your agent with the eval transport

   If your agent uses create_transport(), it supports the eval transport with a one-line addition to its transport_params:

   from pipecat.evals.transport import EvalTransportParams
   
   transport_params = {
       "eval": lambda: EvalTransportParams(
           audio_in_enabled=True,
           audio_out_enabled=True,
       ),
       # ... your other transports (daily, webrtc, twilio, ...)
   }
   

   Then start the agent with -t eval:

   uv run bot.py -t eval
   

   🚀 Bot ready! (eval transport on ws://localhost:7860)
   

   Instead of connecting to Daily or WebRTC, the agent now hosts a local WebSocket server and waits for the eval harness to connect. Nothing else in the agent changes: same pipeline, same services, same event handlers.

   The harness talks to your agent over RTVI. PipelineWorker adds an RTVIProcessor and RTVIObserver automatically, so the standard agent setup needs no extra wiring. All Pipecat example agents already include the "eval" transport entry.

2. Write a scripted scenario

   A scripted scenario is a YAML file that writes the user’s turns out, each with the behavior you expect back. Save this as scenarios/capital_question.yaml:

   name: capital_question
   
   turns:
     # The agent greets on connect; wait for the greeting before speaking.
     - expect:
         - event: response
           eval: "the bot opens the conversation with a greeting or an offer to help"
   
     - user: "What is the capital of Germany?"
       expect:
         - event: response
           eval: "the response says the capital of Germany is Berlin"
   

   Each turn optionally sends a user utterance and lists the events expected in response. The eval: field is a natural-language criterion checked by the judge LLM, so the test passes whether the agent says “Berlin is the capital of Germany” or “That would be Berlin!”.

   This scenario runs in text mode (the default): the user turn is sent as text and the agent’s TTS is skipped automatically, so the whole conversation costs nothing in audio services and finishes in seconds.

   Ollama with gemma4:12b is the default judge, which is why there is no judge: block. To judge with another LLM, see Scenario Configuration.

3. Run the eval

   With the agent still running, run the scenario from another terminal:

   pipecat eval run scenarios/capital_question.yaml
   

   The harness connects to ws://localhost:7860 (override with --bot-url), drives the conversation, and reports the result. Pass -v to watch each turn resolve:

         turn 0 → (observe)
           ✓ llm_response — "Hello! How can I help you today?"
         turn 1 → "What is the capital of Germany?"
           ✓ llm_response — "The capital of Germany is Berlin."
   
     ✓ ws://localhost:7860 capital_question (3402ms)
   
     1/1 passed  ·  3.4s
   

   The command exits 0 when everything passes and 1 otherwise, so it slots directly into scripts and CI. Each scenario also writes a decision trace to <scenario>.eval.log, which shows every event the harness saw and why each assertion passed or failed.

4. Make it fail (optional but recommended)

   Change the criterion to something false, for example "the response says the capital of Germany is Madrid", and run again:

     ✗ ws://localhost:7860 capital_question
   
     Failed (1):
     ✗ ws://localhost:7860 capital_question
         • turn 1 expectation 0 (llm_response): judge said no: the reply says the capital is Berlin, not Madrid
   
     0/1 passed, 1 failed  ·  4.1s
   

   A failing eval tells you which turn, which expectation, and why. That message (plus the .eval.log trace) is what you, or your AI coding assistant, iterate against.

5. Let a simulated caller talk to it

   A scripted scenario walks one path. A simulated scenario hands the user’s side to an LLM with a persona and a goal, and a judge decides from the whole conversation whether the agent did its job. Save this as scenarios/capital_curious.yaml:

   name: capital_curious
   
   persona: |
     A friendly, curious traveler planning a trip to Europe. Polite and concise:
     asks one question at a time, and thanks the assistant once satisfied.
   
   goal: "Find out what the capital of Germany is, then end the call."
   
   success: "the bot told the caller that the capital of Germany is Berlin"
   
   metrics:
     - name: politeness
       criterion: "the reply is courteous and helpful, never curt or dismissive"
       min_score: 1
     - measure: words
       max_value: 60
   
   max_turns: 6
   

   There are no turns to write. persona: and goal: tell the LLM playing the caller who it is and what it wants, and success: tells the judge what the agent must have done. The caller runs on the same local Ollama model as the judge, so this needs no API key; a simulator: block would name another LLM. The politeness metric is judged on every reply, and words bounds the longest one.

   Run it with -v to watch the conversation as it happens:

   pipecat eval run scenarios/capital_curious.yaml -v
   

         bot: Hello! How can I help you today?
         user: Hi! I'm planning a trip to Europe. Could you tell me what the capital of Germany is?
         bot: The capital of Germany is Berlin.
         user: Great, thank you so much!
         bot: You're welcome! Enjoy your trip.
   
         ended by end_call after 2 persona turn(s)
   
       judge: the bot told the caller the capital of Germany is Berlin
   
       metrics:
         politeness: 1.00 (min 1.00) | 3/3 turns
         words: 1.00 | longest reply 7 words, at most 60
   
       persona: succeeded: I learned that the capital of Germany is Berlin.
   
     ✓ ws://localhost:7860 capital_curious end_call (9814ms)
   
     1/1 passed  ·  9.8s
   

   The caller says something different every run, which is the point: it checks that the agent gets a real caller to the goal, without you writing a scenario for every way the conversation could go. When you need exact control over what the user says and what the agent must do in return, write a scripted scenario instead.

Where to go next

• Learn the full scripted format, including multi-turn conversations, function call assertions, interruptions, latency budgets, and text vs audio modes, in Scripted Scenarios.
• Learn how personas, success criteria, judged and measured metrics, and runs: work in Simulated Scenarios.
• Have many scenarios or agents? Let Pipecat spawn the agents for you with Eval Suites.
• Want your coding assistant to run these for you? See The Eval Loop.