Catch regressions with snapshots

A snapshot is a known-good run, stored in Veval: every step in order, with the input your agent sent to it and the output it got back. Comparing a new run against it reports:

Change	Example
Input changed	The system prompt was edited; a tool now gets different arguments
Added	A new step ran — or a known step ran again (e.g. a second LLM call)
Removed	A step stopped running
Moved	A step ran at a different point in the sequence
Output changed	Only when you opt in (`CompareOutputs`) — for deterministic steps

Every change comes with its position and, for input/output changes, a line diff — so a failing test tells you what changed, not just that something did.

Snapshots and the judge are complementary. In replay, model outputs are recorded, so the step inputs are where a code or prompt change shows up — a snapshot catches it for free, with no LLM call. Whether the change made the output better or worse is a question for a live TraceAssert.Judge run.

Save a baseline

Pick a known-good production run and save it under a name. The baseline is stored self-contained, and its source trace is pinned, so neither is removed by trace retention.

await veval.SaveSnapshotAsync("briefing-baseline", "tr_9bc2024e84324464...");

await veval.save_snapshot_async("briefing-baseline", "tr_9bc2024e84324464...")

await veval.saveSnapshotAsync("briefing-baseline", "tr_9bc2024e84324464...");

You can also save from a run you just executed (e.g. a replay or test run) by passing its context instead of a trace ID. Saving again under the same name replaces the baseline — do that when you change the agent intentionally.

Traces are ingested asynchronously. Saving by trace ID right after a run can return 404 until the trace lands — retry shortly, or save from the run’s context instead.

Check a run in tests

Use MatchesSnapshot like any other assertion — in a replay, a scenario, or on its own. A missing baseline, or one that can’t be loaded, fails the assertion; it never passes silently.

var trace = await veval.GetTraceAsync("tr_9bc2024e84324464...");
var testSdk = new VevalTestSdk(new VevalOptions { ApiKey = TestApiKey }).WithReplay(trace!);
var agent = new MyAgent(testSdk);

var result = await testSdk.ReplayAsync(trace!, agent.ExecuteAsync, new ReplayOptions
{
    MockLlmResponses = true,
    Assertions = [TraceAssert.MatchesSnapshot(testSdk, "briefing-baseline")],
});

Assert.True(result.Passed, string.Join("\n", result.Failures));

trace = await veval.get_trace_async("tr_9bc2024e84324464...")
test_sdk = VevalTestSdk(VevalOptions(api_key=TEST_API_KEY)).with_replay(trace)
agent = MyAgent(test_sdk)

result = await test_sdk.replay_async(trace, agent.execute_async, ReplayOptions(
    mock_llm_responses=True,
    assertions=[TraceAssert.matches_snapshot(test_sdk, "briefing-baseline")],
))

assert result.passed, "\n".join(result.failures)

const trace = await veval.getTraceAsync("tr_9bc2024e84324464...");
const testSdk = new VevalTestSdk({ apiKey: TEST_API_KEY }).withReplay(trace);
const agent = new MyAgent(testSdk);

const result = await testSdk.replayAsync(trace, (ctx) => agent.execute(ctx), {
  mock_llm_responses: true,
  assertions: [TraceAssert.matchesSnapshot(testSdk, "briefing-baseline")],
});

expect(result.failures).toEqual([]);

A failure reads like this:

MatchesSnapshot: Snapshot 'briefing-baseline': 2 change(s)
  ~ input changed for 'write-briefing'
        Write a concise, friendly briefing...
      - 3. **Tasks** — overdue items first, then what's due today.
      + 3. **Tasks** — list every open task alphabetically by title.
  + added 'write-briefing' (call #2) at position 5

To run fully offline in CI, register the baseline locally with WithSnapshot(name, snapshot) (with_snapshot / withSnapshot) on the test SDK.

Guard every replay against its recording

A mocked replay already fails when your agent adds a call or skips a recorded step, and it reports calls whose input differs from the recording without failing on them (how calls are matched). Turn on CompareWithRecording and the replay also fails when its sequence or inputs drift from the trace it’s replaying:

new ReplayOptions { MockLlmResponses = true, CompareWithRecording = new SnapshotOptions() }
// result.RecordingDiff holds the details

ReplayOptions(mock_llm_responses=True, compare_with_recording=SnapshotOptions())
# result.recording_diff holds the details

{ mock_llm_responses: true, compare_with_recording: {} }
// result.recording_diff holds the details

Production monitoring

Compare live runs against the baseline and record every result in the dashboard, where each run shows expected and actual steps side by side:

var baseline = await veval.GetSnapshotAsync("briefing-baseline");   // load once, cache it

var result = await veval.RunAsync("my-agent", async ctx =>
{
    var answer = await myService.ExecuteAsync(ctx);
    var diff = await veval.CompareSnapshotAsync("briefing-baseline", baseline!, ctx);
    if (diff.HasChanges) logger.LogWarning("{Report}", diff.Summary());
    return answer;
});

baseline = await veval.get_snapshot_async("briefing-baseline")   # load once, cache it

async def my_agent(ctx):
    answer = await my_service.execute_async(ctx)
    diff = await veval.compare_snapshot_async("briefing-baseline", baseline, ctx)
    if diff.has_changes:
        logger.warning(diff.summary())
    return answer

result = await veval.run_async("my-agent", my_agent)

const baseline = await veval.getSnapshotAsync("briefing-baseline"); // load once, cache it

const result = await veval.runAsync("my-agent", async (ctx) => {
  const answer = await myService.execute(ctx);
  const diff = await veval.compareSnapshotAsync("briefing-baseline", baseline, ctx);
  if (diff.has_changes) console.warn(diff.summary());
  return answer;
});

In live runs, inputs built from live data (today’s date, fetched records) differ from the baseline’s. Use IgnoreFields, IgnoreSteps, or Normalize to leave those out, or compare only in replay where inputs are reproducible.

Options

Option	Default	Description
`CompareInputs`	`true`	Compare what the agent sent to each step
`CompareOutputs`	`false`	Compare each step’s output — turn on for deterministic steps
`IgnoreSteps`	none	Step names left out of the comparison entirely
`IgnoreFields`	none	JSON property names dropped from inputs/outputs at any depth (e.g. `timestamp`)
`Normalize`	none	`(stepName, value) => value` rewrite applied before comparing

Python uses snake_case (compare_inputs, ignore_fields, …); Node uses camelCase (compareInputs, ignoreFields, …). Values are compared structurally: key order doesn’t matter, and a value loaded from the API compares equal to the same object built in your code.

SnapshotDiff

Member	Description
`HasChanges`	`true` if any difference was found
`Changes`	Every change in run order: kind, step name, which call of that step, positions, and a line diff
`Alignment`	Expected and actual steps side by side, in run order
`Summary()`	A readable report of every change — made for test failure messages
`AddedSteps` / `RemovedSteps` / `OrderChanges`	Name-level views, kept for existing code

Baselines saved before step inputs were recorded (older snapshots) still load. They’re compared on the step sequence only, and the summary says so. Save the baseline again to get input comparison.

Choosing a baseline

Pick a run that represents the expected “happy path”:

All expected steps completed successfully, with no error steps
Representative of typical production behavior
Recent — save a new baseline whenever you change the agent on purpose

What’s next

Test with replay

Run your agent against recorded traces in tests.

Write assertions

Combine snapshots with the LLM judge and other checks.

Get started

Concepts

Guides

Development

Catch regressions with snapshots

Save a baseline

Check a run in tests

Guard every replay against its recording

Production monitoring

Options

SnapshotDiff

Choosing a baseline

What’s next

Test with replay

Write assertions

​Save a baseline

​Check a run in tests

​Guard every replay against its recording

​Production monitoring

​Options

​SnapshotDiff

​Choosing a baseline

​What’s next

Test with replay

Write assertions

Save a baseline

Check a run in tests

Guard every replay against its recording

Production monitoring

Options

SnapshotDiff

Choosing a baseline

What’s next