Run a scenario

A scenario is a named set of test cases for your agent. Each run posts pass/fail results to the dashboard so you can track quality over time. Scenarios have two item types:

Type	How it works	Cost
Synthetic	Live LLM, fresh input you define	Real API cost
Trace-backed	Replays a recorded production trace: live model, recorded tool data	Model calls only

RunScenarioAsync

var result = await veval.RunScenarioAsync(
    scenarioName:       "my-scenario",
    agent:              service.ExecuteAsync,
    scenarioAssertions: [
        TraceAssert.NoErrors(),
        TraceAssert.MaxCost(0.10m),
        TraceAssert.StepExists("classify"),
    ],
    items: [
        new ScenarioItem { Name = "question 1", Input = "What is prompt caching?" },
        new ScenarioItem { Name = "question 2", Input = "What is tool use?" },
    ]
);

result = await veval.run_scenario_async(
    scenario_name="my-scenario",
    agent=service.execute_async,
    scenario_assertions=[
        TraceAssert.no_errors(),
        TraceAssert.max_cost(0.10),
        TraceAssert.step_exists("classify"),
    ],
    items=[
        ScenarioItem(name="question 1", input="What is prompt caching?"),
        ScenarioItem(name="question 2", input="What is tool use?"),
    ],
)

const result = await veval.runScenarioAsync(
  "my-scenario",
  (ctx) => service.execute(ctx),
  [
    TraceAssert.noErrors(),
    TraceAssert.maxCost(0.10),
    TraceAssert.stepExists("classify"),
  ],
  [
    { name: "question 1", input: "What is prompt caching?", assertions: [] },
    { name: "question 2", input: "What is tool use?", assertions: [] },
  ]
);

Parameter	Description
`scenarioName`	Identifies the scenario in the dashboard
`agent`	Your agent — same signature as `RunAsync`
`scenarioAssertions`	Assertions that apply to every item
`items`	List of `ScenarioItem` — inline or fetched from dashboard

Synthetic items

Use these for new inputs you want to test with a live LLM.

new ScenarioItem
{
    Name  = "prompt caching question",
    Input = "Explain prompt caching in one sentence.",
}

ScenarioItem(
    name="prompt caching question",
    input="Explain prompt caching in one sentence.",
)

{ name: "prompt caching question", input: "Explain prompt caching in one sentence.", assertions: [] }

Trace-backed items

Use these to turn a recorded production trace into a regression test. Steps recorded as llm run live with your current prompt, so a prompt change that makes the output worse fails the item. Every other step (tools, untyped steps) returns its recorded output, so the replay sees the same data as the original run and never reaches a real system. Mark your model steps when you record the trace, with step.SetMeta("type", "llm") (set_meta / setMeta), and your tools with "tool". If no step of a trace-backed item ran live, the item fails and tells you to mark the model steps.

new ScenarioItem
{
    Name    = "production trace replay",
    TraceId = "tr_4ec4e79c5d03...",
    Assertions = [TraceAssert.MaxCost(0.05m)],  // per-item assertion
}

ScenarioItem(
    name="production trace replay",
    trace_id="tr_4ec4e79c5d03...",
    assertions=[TraceAssert.max_cost(0.05)],  # per-item assertion
)

{
  name: "production trace replay",
  trace_id: "tr_4ec4e79c5d03...",
  assertions: [TraceAssert.maxCost(0.05)],  // per-item assertion
}

The trace must have recorded steps. If it has none, Veval throws rather than silently calling the live LLM.

Mocked trace-backed items

Set MockModelSteps to serve model steps from the recording too. The item is then free and deterministic, which suits checking a code change (routing, parsing, tool wiring), but it can’t catch a prompt regression: every model call returns what the old prompt produced.

new ScenarioItem { TraceId = "tr_4ec4e79c5d03...", MockModelSteps = true }

ScenarioItem(trace_id="tr_4ec4e79c5d03...", mock_model_steps=True)

{ trace_id: "tr_4ec4e79c5d03...", mock_model_steps: true, assertions: [] }

To mock every trace-backed item in a run, including items loaded from the dashboard, pass the option to RunScenarioAsync:

await veval.RunScenarioAsync("my-scenario", agent, assertions, mockModelSteps: true);

await veval.run_scenario_async("my-scenario", agent, assertions, mock_model_steps=True)

await veval.runScenarioAsync("my-scenario", agent, assertions, undefined, { mock_model_steps: true });

VevalTestSdk always replays trace-backed items mocked, so a test built on it never makes a live model call.

Keeping the trace

Traces are deleted after your plan’s retention window. Each time a scenario replays a trace, its retention restarts, so a trace your tests replay at least once per window stays. To keep a trace no matter how often it runs, open it in the dashboard and click Use as fixture. This pins the trace and shows the ScenarioItem for it in each language, ready to copy. Pin on its own keeps the trace without showing the item. Each plan allows a set number of pinned traces (25 on Free); unpin one to free a slot. If the trace is gone, the item fails with trace '<id>' not found and the scenario’s other items still run.

Per-item assertions

Each ScenarioItem can carry its own assertions on top of the scenario-level ones:

new ScenarioItem
{
    Name       = "expensive question",
    Input      = "Write me a novel.",
    Assertions = [TraceAssert.MaxCost(0.50m)],  // only for this item
}

ScenarioItem(
    name="expensive question",
    input="Write me a novel.",
    assertions=[TraceAssert.max_cost(0.50)],  # only for this item
)

{
  name: "expensive question",
  input: "Write me a novel.",
  assertions: [TraceAssert.maxCost(0.50)],  // only for this item
}

Fetching items from the dashboard

If items is null, Veval fetches items for the scenario from the API automatically. This lets you manage test cases in the dashboard without redeploying code.

// items: null → fetched from dashboard by scenarioName
var result = await veval.RunScenarioAsync(
    scenarioName:       "my-scenario",
    agent:              service.ExecuteAsync,
    scenarioAssertions: [TraceAssert.NoErrors()]
);

# items=None → fetched from dashboard by scenario_name
result = await veval.run_scenario_async(
    scenario_name="my-scenario",
    agent=service.execute_async,
    scenario_assertions=[TraceAssert.no_errors()],
)

// items omitted → fetched from dashboard by scenarioName
const result = await veval.runScenarioAsync(
  "my-scenario",
  (ctx) => service.execute(ctx),
  [TraceAssert.noErrors()]
);

ScenarioRunResult

Console.WriteLine($"{result.PassCount}/{result.Results.Count} passed");

foreach (var item in result.Results)
{
    Console.WriteLine($"{(item.Passed ? "✓" : "✗")} {item.Item.Name}");
    foreach (var failure in item.Failures)
        Console.WriteLine($"  → {failure}");
}

print(f"{result.pass_count}/{len(result.results)} passed")

for item in result.results:
    print(f"{'✓' if item.passed else '✗'} {item.item.name}")
    for failure in item.failures:
        print(f"  → {failure}")

console.log(`${result.pass_count}/${result.results.length} passed`);

for (const item of result.results) {
  console.log(`${item.passed ? "✓" : "✗"} ${item.item.name}`);
  for (const failure of item.failures)
    console.log(`  → ${failure}`);
}

Member	Description
`Passed`	True if all items passed
`PassCount`	Number of passing items
`FailCount`	Number of failing items
`Results`	List of `ItemRunResult`

Get started

Concepts

Guides

Development

RunScenarioAsync

Synthetic items

Trace-backed items

Mocked trace-backed items

Keeping the trace

Per-item assertions

Fetching items from the dashboard

ScenarioRunResult

What’s next

Write assertions

Test with replay

​RunScenarioAsync

​Synthetic items

​Trace-backed items

​Mocked trace-backed items

​Keeping the trace

​Per-item assertions

​Fetching items from the dashboard

​ScenarioRunResult

​What’s next

Write assertions

Test with replay

RunScenarioAsync

Synthetic items

Trace-backed items

Mocked trace-backed items

Keeping the trace

Per-item assertions

Fetching items from the dashboard

ScenarioRunResult

What’s next