Skip to main content

​
Introduction

The Agent Testing API runs automated test scenarios to catch issues before they reach customers. A simulated caller talks to your agent, follows a scripted behavior, and predefined assertions measure whether your agent handled the conversation correctly.
Working in the dashboard? In Bland Agents, scenarios are split between test cases and judges under Validate, in Evaluations. See Evaluations. To make a passing run a condition of shipping, see Environments.

Scenario-Based Testing

Define test scenarios with tester personas that simulate real callers, such as angry customers, voicemail systems, and gatekeepers. You can also clone pre-built scenario templates.

Assertion Engine

10 assertion types including LLM judges, variable extraction checks, node traversal validation, and regex matching

Batch & Simulation Runs

Run multiple scenarios at once, or run the same scenario N times to detect flaky behavior

Norm Analysis

AI-generated root cause and fix suggestions for failed runs

​
Core Concepts

​
Scenarios

A scenario defines one test case. Each scenario contains:
  • Tester Persona: A prompt that tells the simulated caller how to behave (e.g., “You are an angry customer who was charged twice”)
  • Assertions: Rules that evaluate whether the agent handled the conversation correctly
  • Configuration: Max turns, starting node, request data overrides, and more
Set start_node_id on a scenario to begin the simulated conversation at a specific node instead of the default start node. This is useful for testing a single branch, like an authentication handoff or a post-transfer flow, without rewiring the flow. Leave it unset (or null) to start from the default start node. If the node no longer exists in the version under test, the run fails with an Invalid start_node_id error.
Each scenario is bound to the pathway_id or persona_id you pass when you create it. Scenarios persist across versions, so the same tests can run against every version.

​
Assertions

Assertions are the pass/fail criteria for a test run. Each scenario can have multiple assertions, each with a type, weight, and required flag.

​
Test Runs

When you execute a scenario, a test run is created. The system:
  1. Creates a simulated call
  2. The tester persona converses with your agent up to a predefined maximum turn depth
  3. Captures the full chat history, extracted variables, and nodes visited
  4. Evaluates all assertions against the conversation
  5. Computes an overall score (weighted average of assertion scores)
Run statuses: PENDING → RUNNING → PASSED | FAILED | ERROR | CANCELLED

​
Batches

A batch runs multiple scenarios together. Use it to run your full test suite against a version before you ship it. Pass version_number to choose the version. It defaults to the current production version. Scenarios marked is_required_for_promotion act as a gate that must pass before a version is promoted. For Bland Agents, gating is configured as checks on the staging environment. See Environments and Set Checks.

​
Scenario Categories

Bland provides out-of-box templates for common testing scenarios. Clone one to create your own scenario, then customize it.
Tests how your agent handles reaching voicemail, such as leaving a coherent message or handling a full mailbox.
Simulates Google Call Screen or a suspicious gatekeeper. Tests whether your agent identifies itself clearly to get through.
Progressively escalating frustration. Tests de-escalation skills: empathy, ownership, and professionalism under pressure.
Abusive language and personal attacks. Tests whether your agent maintains professional boundaries and escalates appropriately.
Rambling, off-topic caller who isn’t sure what they need. Tests patience, clarification questions, and gentle redirection.
Cooperative callers following the expected flow, such as appointment scheduling or information requests. Validates your core conversation logic.
Unusual situations that test the boundaries of your agent’s handling.
Your own custom scenarios tailored to your specific use case.

​
Get started with the API

1

Create a scenario

Create a scenario with POST /v1/agent-testing/scenarios. To start from a template instead, list templates with GET /v1/agent-testing/templates and clone one with POST /v1/agent-testing/templates/:id/clone. To build one from a real conversation, use POST /v1/agent-testing/scenarios/generate-from-call.
2

Run it

Run one scenario with POST /v1/agent-testing/scenarios/:id/run, or several with POST /v1/agent-testing/batch-run. Both return immediately while the tests run.
3

Read the results

Fetch a run with GET /v1/agent-testing/runs/:id or a batch with GET /v1/agent-testing/batches/:id. A run includes the chat history, assertion results, and scores.
4

Analyze failures

For a failed run, call POST /v1/agent-testing/runs/:id/analyze to get a root cause and suggested fixes. See Norm analysis.

​
Simulation Sets

Simulation sets run the same scenario multiple times to detect flaky behavior: cases where your agent passes sometimes but fails other times. The results include per-scenario statistics:
  • Pass rate across all runs
  • Score distribution (mean, median, stddev, min, max)
  • Flakiness detection: scenarios flagged as unreliable
  • Common failure modes: recurring issues across runs
  • Confidence level (low, medium, high) based on sample size

​
Analytics

Agent testing provides two levels of analytics. Both endpoints take a pathwayId.

​
Basic Analytics

Pass rate, failure rate, average score, and per-run details over a configurable time window (default 30 days, max 365).

​
Enhanced Analytics

Advanced metrics including:
  • Health Score: Overall testing health
  • Reliability Score: Consistency of test results
  • Trend Analysis: Weekly direction (improving, declining, stable) with daily pass rates
  • Node Failure Heatmap: Which nodes have the highest failure rates
  • Weakest Link: The single node most responsible for test failures
  • Sankey Flow Data: Visual flow from scenarios through nodes to outcomes (pass/fail)

​
Norm Analysis

When a test run fails, you can trigger Norm analysis to get AI-powered suggestions for fixing your agent. The analyzer examines the conversation, identifies the root cause, and suggests specific changes:
  • prompt_change: Modify a node’s prompt text
  • node_config: Change node configuration settings
  • flow_change: Restructure routing
  • tone_improvement: Adjust conversational style
Each suggestion includes a target node, description, suggested change text, and confidence score.

​
API Reference

For full endpoint details, see the Scenarios API Reference.
Docs for agents: llms.txt