Skip to main content
pipecat eval run tests scenarios against an agent you started yourself. A suite goes one step further: you list agents and scenarios in a manifest, and pipecat eval suite spawns each agent with its eval transport on its own port, runs its scenarios, tears it down, and aggregates the results, several runs at a time. Suites are the right tool when you have more than one agent, more than a handful of scenarios, or want a single command for CI. They are also how a simulation runs more than once: each run gets a fresh agent. Pipecat’s own release evals are a manifest with 100+ example agents plus this command.

​
The manifest

manifest.yaml
Paths in the manifest (bots_dir, scenarios_dir, runs_dir, the bot: entries) resolve relative to the manifest file, so a manifest is portable: check it into your repo and run it from anywhere. A scenario name can include a subfolder (simulated/book_table), and a name ending in .yaml is a path relative to the manifest. Scripted and simulated scenarios are listed the same way, and the file says which it is: turns: makes it scripted, persona: makes it a simulation. Pipecat’s release evals keep the two in scenarios/scripted/ and scenarios/simulated/ folders, which is a convention worth copying. Scenarios are reusable across agents. One greeting scenario can cover every agent in the suite.
An optional runner_body: points at a JSON file passed to the agent as --runner-body. It supplies session data the agent would normally receive in a /start request body (for example, a vision agent’s image path).

​
Running a suite

In a terminal, a live dashboard shows each run’s status, a running tally, and total time. When piped (in CI, or driven by a coding assistant), it streams one plain result line per run instead. The command exits 0 only if every run passes. Useful flags:
Everything except the suite: list can live in the manifest or be passed on the command line (the command line wins), so a manifest can be as minimal as a suite: list.

​
Run output

Each invocation writes to <runs_dir>/<name>/ (a timestamp when -n is omitted):
results.jsonl carries one line per run, and every line names its kind, script or simulation. A scripted record carries the bot, scenario, attempt, outcome, duration, failures (each with a kind), per-turn results, and paths to its artifacts. A simulation record carries the outcome (passed, succeeded, error), how the run ended (ended_by), the persona’s turn count, every metric’s score, value, failure_kind, and per-turn verdicts (yes, no, or none when the judge gave none), the judge’s reason, the persona’s own end_call claim, and the whole conversation as messages. Lines are appended as each run finishes, so an interrupted sweep keeps everything already done. Runs that didn’t pass also carry events_seen. Each run executes in its own process, so a harness that loads local audio models does so on its own, and a crash in one run doesn’t stop the others.
results.jsonl is written by pipecat eval suite. pipecat eval run doesn’t produce one.
When a run fails, start with the .eval.log decision trace: it’s a timestamped record of every event the harness saw, what it matched, what the judge said, and why an assertion failed. The agent’s own log sits next to it.

​
Testing one agent with many scenarios

If you just want to run a batch of scenarios against an agent you already have running, you don’t need a manifest. pipecat eval run accepts multiple scenario files and shares the suite’s dashboard and tally:
A directory expands to its .yaml and .yml files in filename order, non-recursively, and files and directories can be mixed in one invocation. A directory holding no scenarios is an error rather than an empty run. By default the agent is left running afterward so it can serve more evals; pass --stop-bot to shut it down when the batch finishes.

​
Running a simulation more than once

A persona doesn’t say the same thing twice, so one run of a simulation proves little. A simulation’s own runs: field tells the suite how many times to run it, and every run must pass. The suite reports a pass rate per simulation with a ✓ or ✗ beside it, and exits 1 when any run failed:
A run that errored (the agent never came up, the persona’s LLM failed, the judge gave no verdict on the goal) is reported but kept out of the rate: it says nothing about whether the agent did its job.

​
Repeating a run

A behavior with a race in it passes sometimes. repeat: in the manifest, or --repeat / -r on the command line, runs every (bot, scenario) pair N times and reports a pass rate per pair instead of a single verdict:
Attempts interleave across bots rather than running back to back, and artifact filenames gain an attempt suffix (__001) only when repeat is above 1, so a single pass keeps the filenames it always had.
A repeated sweep always exits 0. A pass rate isn’t a pass or a fail, so the threshold is yours to choose: read results.jsonl and decide. Leave repeat unset for the CI gate below.
A repeat set on the command line or in the manifest applies to every run, simulations included, even when it is 1. It overrides a simulation’s runs: and turns the requirement into a measurement: rates are reported and the exit code stays 0. Leave repeat unset to let each simulation run its own runs: and gate on them.

​
Suites in CI

The exit code makes suites CI-ready with no extra glue:
For deterministic, key-free CI runs, prefer text-mode scripted scenarios and an OpenAI-compatible judge endpoint you control. Simulations run their persona on the local Ollama model by default, so they need no extra key, but they say something different each run by design, so give them runs: 3 and expect them to take longer. -k script runs the scripted half alone when you want a fast gate on every push and the simulations on a schedule. Audio-mode scenarios work in CI too, but need the harness’s TTS and STT services available (local models by default, which also need more CPU).