> ## Documentation Index
> Fetch the complete documentation index at: https://docs.bland.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Scenarios

> Run automated test scenarios against your agent with the Agent Testing API.

## Introduction

The Agent Testing API runs automated test scenarios to catch issues before they reach customers. A simulated caller talks to your agent, follows a scripted behavior, and predefined assertions measure whether your agent handled the conversation correctly.

> [!NOTE]
>
> Working in the dashboard? In Bland Agents, scenarios are split between test cases and judges under **Validate**, in **Evaluations**. See [Evaluations](https://docs.bland.ai/agents/evaluations.md). To make a passing run a condition of shipping, see [Environments](https://docs.bland.ai/agents/environments.md#the-evaluation-gate).

> [card] Scenario-Based Testing
>
> Define test scenarios with tester personas that simulate real callers, such as angry customers, voicemail systems, and gatekeepers. You can also clone pre-built scenario templates.

> [card] Assertion Engine
>
> 10 assertion types including LLM judges, variable extraction checks, node traversal validation, and regex matching

> [card] Batch & Simulation Runs
>
> Run multiple scenarios at once, or run the same scenario N times to detect flaky behavior

> [card] Norm Analysis
>
> AI-generated root cause and fix suggestions for failed runs

***

## Core Concepts

### Scenarios

A scenario defines one test case. Each scenario contains:

* **Tester Persona**: A prompt that tells the simulated caller how to behave (e.g., "You are an angry customer who was charged twice")
* **Assertions**: Rules that evaluate whether the agent handled the conversation correctly
* **Configuration**: Max turns, starting node, request data overrides, and more

> [!TIP]
>
> Set `start_node_id` on a scenario to begin the simulated conversation at a specific node instead of the default start node. This is useful for testing a single branch, like an authentication handoff or a post-transfer flow, without rewiring the flow. Leave it unset (or `null`) to start from the default start node. If the node no longer exists in the version under test, the run fails with an `Invalid start_node_id` error.

Each scenario is bound to the `pathway_id` or `persona_id` you pass when you create it. Scenarios persist across versions, so the same tests can run against every version.

### Assertions

Assertions are the pass/fail criteria for a test run. Each scenario can have multiple assertions, each with a type, weight, and required flag.

| Type | What it checks |
| - | - |
| **LLM\_JUDGE** | Custom LLM prompt evaluates the conversation (boolean, score, or categorical) |
| **BLAND\_TONE** | Built-in naturalness scoring: empathy, conciseness, flow, back-channeling |
| **VARIABLE\_EXTRACTED** | Whether a specific variable was collected and optionally matches a value |
| **NODE\_REACHED** | Whether the agent visited a specific node |
| **NODES\_VISITED** | Whether the agent visited a set of nodes (all or any) |
| **WEBHOOK\_TRIGGERED** | Whether a webhook was called during the conversation |
| **REGEX\_MATCH** | Regex pattern match against the transcript or extracted variables |
| **STRING\_CHECK** | Simple string match (equals, contains, case-insensitive) against transcript or variables |
| **CUSTOM\_LLM** | Custom LLM evaluation with your own system prompt and model |
| **TRAVERSAL\_MATCH** | Compares the node traversal path against a reference path |

### Test Runs

When you execute a scenario, a **test run** is created. The system:

1. Creates a simulated call
2. The tester persona converses with your agent up to a predefined maximum turn depth
3. Captures the full chat history, extracted variables, and nodes visited
4. Evaluates all assertions against the conversation
5. Computes an overall score (weighted average of assertion scores)

Run statuses: `PENDING` → `RUNNING` → `PASSED` | `FAILED` | `ERROR` | `CANCELLED`

### Batches

A **batch** runs multiple scenarios together. Use it to run your full test suite against a version before you ship it. Pass `version_number` to choose the version. It defaults to the current production version.

Scenarios marked `is_required_for_promotion` act as a **gate** that must pass before a version is promoted. For Bland Agents, gating is configured as checks on the staging environment. See [Environments](https://docs.bland.ai/agents/environments.md#the-evaluation-gate) and [Set Checks](https://docs.bland.ai/api-v2/put/agents-id-environments-env-checks.md).

***

## Scenario Categories

Bland provides out-of-box templates for common testing scenarios. Clone one to create your own scenario, then customize it.

> [accordion] Voicemail
>
> Tests how your agent handles reaching voicemail, such as leaving a coherent message or handling a full mailbox.

> [accordion] Call Screener
>
> Simulates Google Call Screen or a suspicious gatekeeper. Tests whether your agent identifies itself clearly to get through.

> [accordion] Angry Caller
>
> Progressively escalating frustration. Tests de-escalation skills: empathy, ownership, and professionalism under pressure.

> [accordion] Belligerent Caller
>
> Abusive language and personal attacks. Tests whether your agent maintains professional boundaries and escalates appropriately.

> [accordion] Confused Caller
>
> Rambling, off-topic caller who isn't sure what they need. Tests patience, clarification questions, and gentle redirection.

> [accordion] Happy Path
>
> Cooperative callers following the expected flow, such as appointment scheduling or information requests. Validates your core conversation logic.

> [accordion] Edge Case
>
> Unusual situations that test the boundaries of your agent's handling.

> [accordion] Custom
>
> Your own custom scenarios tailored to your specific use case.

***

## Get started with the API

**1. Create a scenario**

Create a scenario with [`POST /v1/agent-testing/scenarios`](https://docs.bland.ai/api-v1/post/agent-testing-scenarios). To start from a template instead, list templates with [`GET /v1/agent-testing/templates`](https://docs.bland.ai/api-v1/get/agent-testing-templates) and clone one with [`POST /v1/agent-testing/templates/:id/clone`](https://docs.bland.ai/api-v1/post/agent-testing-templates-clone). To build one from a real conversation, use [`POST /v1/agent-testing/scenarios/generate-from-call`](https://docs.bland.ai/api-v1/post/agent-testing-scenarios-generate).

**2. Run it**

Run one scenario with [`POST /v1/agent-testing/scenarios/:id/run`](https://docs.bland.ai/api-v1/post/agent-testing-scenarios-run), or several with [`POST /v1/agent-testing/batch-run`](https://docs.bland.ai/api-v1/post/agent-testing-batch-run). Both return immediately while the tests run.

**3. Read the results**

Fetch a run with [`GET /v1/agent-testing/runs/:id`](https://docs.bland.ai/api-v1/get/agent-testing-runs-id) or a batch with [`GET /v1/agent-testing/batches/:id`](https://docs.bland.ai/api-v1/get/agent-testing-batches-id). A run includes the chat history, assertion results, and scores.

**4. Analyze failures**

For a failed run, call [`POST /v1/agent-testing/runs/:id/analyze`](https://docs.bland.ai/api-v1/post/agent-testing-runs-analyze) to get a root cause and suggested fixes. See [Norm analysis](#norm-analysis).

***

## Simulation Sets

Simulation sets run the same scenario multiple times to detect **flaky behavior**: cases where your agent passes sometimes but fails other times.

The results include per-scenario statistics:

* **Pass rate** across all runs
* **Score distribution** (mean, median, stddev, min, max)
* **Flakiness detection**: scenarios flagged as unreliable
* **Common failure modes**: recurring issues across runs
* **Confidence level** (low, medium, high) based on sample size

***

## Analytics

Agent testing provides two levels of analytics. Both endpoints take a `pathwayId`.

### Basic Analytics

Pass rate, failure rate, average score, and per-run details over a configurable time window (default 30 days, max 365).

### Enhanced Analytics

Advanced metrics including:

* **Health Score**: Overall testing health
* **Reliability Score**: Consistency of test results
* **Trend Analysis**: Weekly direction (improving, declining, stable) with daily pass rates
* **Node Failure Heatmap**: Which nodes have the highest failure rates
* **Weakest Link**: The single node most responsible for test failures
* **Sankey Flow Data**: Visual flow from scenarios through nodes to outcomes (pass/fail)

***

## Norm Analysis

When a test run fails, you can trigger **Norm analysis** to get AI-powered suggestions for fixing your agent. The analyzer examines the conversation, identifies the root cause, and suggests specific changes:

* **prompt\_change**: Modify a node's prompt text
* **node\_config**: Change node configuration settings
* **flow\_change**: Restructure routing
* **tone\_improvement**: Adjust conversational style

Each suggestion includes a target node, description, suggested change text, and confidence score.

***

## API Reference

For full endpoint details, see the [Scenarios API Reference](https://docs.bland.ai/api-v1/post/agent-testing-scenarios.md).

| Endpoint | Description |
| - | - |
| `POST /v1/agent-testing/scenarios` | Create a test scenario |
| `GET /v1/agent-testing/scenarios` | List scenarios |
| `GET /v1/agent-testing/scenarios/:id` | Get scenario details |
| `PUT /v1/agent-testing/scenarios/:id` | Update a scenario |
| `DELETE /v1/agent-testing/scenarios/:id` | Delete a scenario |
| `GET /v1/agent-testing/templates` | List out-of-box templates |
| `POST /v1/agent-testing/templates/:id/clone` | Clone a template |
| `POST /v1/agent-testing/scenarios/generate-from-call` | Generate a scenario from a call transcript |
| `POST /v1/agent-testing/scenarios/:id/run` | Run a single scenario |
| `POST /v1/agent-testing/batch-run` | Run multiple scenarios |
| `GET /v1/agent-testing/runs` | List test runs |
| `GET /v1/agent-testing/runs/:id` | Get run details |
| `GET /v1/agent-testing/batches/:id` | Get batch details |
| `POST /v1/agent-testing/runs/:id/analyze` | Analyze a failed run |
| `GET /v1/agent-testing/analytics/:pathwayId` | Basic analytics |
| `GET /v1/agent-testing/analytics/:pathwayId/enhanced` | Enhanced analytics |
| `POST /v1/agent-testing/simulation-sets` | Create a simulation set |
| `GET /v1/agent-testing/simulation-sets/:id` | Get simulation set results |

***

Docs for agents: [llms.txt](https://docs.bland.ai/llms.txt)


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.