SDK reference

This page covers every SDK API in all three languages. It’s also built as a single-page context drop for AI coding agents. See Build with AI.

What Veval does

Veval wraps your agent runs and LLM calls to record traces: inputs, outputs, step timing, token counts, and cost. Traces appear in the dashboard. You run scenarios and assertions against them in CI to catch regressions before they reach production. Three SDKs, same concept: C# (Veval.Sdk), Python (veval), Node (@veval/sdk).

Install

# C#
dotnet add package Veval.Sdk

# Python
pip install veval-sdk

# Node
npm install @veval/sdk

Initialize

using Veval.Sdk;

var veval = new VevalSdk(new VevalOptions { ApiKey = "YOUR_API_KEY" });

Python

from veval import VevalSdk, VevalOptions

veval = VevalSdk(VevalOptions(api_key="YOUR_API_KEY"))

Node

import { VevalSdk } from "@veval/sdk";

const veval = new VevalSdk({ apiKey: "YOUR_API_KEY" });

Options

C#	Python	Node	Default	Meaning
`ApiKey`	`api_key`	`apiKey`	—	Required. Your API key. It also selects the workspace. Without it, every send fails with 401.
`ThrowOnSendFailure`	`raise_on_send_failure`	`throwOnSendFailure`	`false`	Throw when a trace, scenario run or snapshot comparison can’t be posted. Turn it on in tests and CI.

Delivery failures

Sending results (traces, scenario runs, snapshot comparisons) never throws unless strict mode is on. A rejected or failed send (401 wrong API key, 429 quota used up, network error) logs a warning that names the status and, when the API gives one, the reason:

C#: standard error
Python: the veval logger at WARNING
Node: console.warn

Python prints the warning to standard error when the app hasn’t configured logging. If it raises the level above WARNING or silences library loggers, the warning is hidden. Calls that return data (SaveSnapshotAsync, GetSnapshotAsync, JudgeAsync, GetTraceAsync, LoadSnapshotAsync, and RunScenarioAsync loading items or saved traces from the dashboard) are not sends: they throw on failure regardless of strict mode. Loading a trace or a scenario’s items throws VevalRequestException (C#, StatusCode) or VevalRequestError (Python and Node, status); the status is null when there was no response. GetTraceAsync returns null only for a trace that doesn’t exist (404). A scenario whose items can’t be loaded throws rather than running zero items. With ThrowOnSendFailure / raise_on_send_failure / throwOnSendFailure on, the send throws VevalSendException (C#, StatusCode) or VevalSendError (Python and Node, status). The status is null when there was no response. If the agent itself threw, its exception wins and the send failure is only logged. CompareSnapshotAsync throws without returning the diff if posting the comparison fails. In a scenario, every item runs, then the run is posted. If that post fails, strict mode throws that failure. Otherwise it throws one error for any item traces that couldn’t be stored (N trace(s) from scenario '<name>' could not be stored). That error’s status is always null; the per-item warnings have the statuses. If traces don’t appear in the dashboard, look for these warnings first.

Core API

RunAsync — wrap a complete agent run

var result = await veval.RunAsync("agent-name", async ctx =>
{
    // your agent logic using ctx
    return output;
}, input: userMessage);  // input is optional

Python

result = await veval.run_async("agent-name", my_agent, user_message)
# my_agent is async def my_agent(ctx): ...
# input is optional (positional or keyword)

# Synchronous agent: plain functions, no event loop
result = veval.run("agent-name", my_sync_agent, user_message)
# inside it: ctx.track_step("step-name", text, lambda: call_model(text))

Node

const result = await veval.runAsync("agent-name", async (ctx) => {
  // your agent logic using ctx
  return output;
}, userMessage);  // input is optional

Sends a trace on both success and error. The ctx object is a VevalExecutionContext. Python also has synchronous run, track_step, get_trace, replay and run_scenario for agents written as plain functions. Call them from synchronous code; inside a running event loop, use the _async versions.

TrackStepAsync — record one LLM call or sub-operation

Simple overload (no metadata):

var output = await ctx.TrackStepAsync("step-name", input: text, async () =>
{
    return await llm.Call(text);
});

Python

async def step():
    return await llm.call(text)

output = await ctx.track_step_async("step-name", text, step)

Node

const output = await ctx.trackStepAsync("step-name", text, async () => {
  return await llm.call(text);
});

Handle overload (attach LLM metadata):

var output = await ctx.TrackStepAsync("step-name", input: text, async handle =>
{
    var response = await llm.Call(text);
    handle.SetMeta("model",      response.Model);
    handle.SetMeta("tokens_in",  response.Usage.InputTokens);
    handle.SetMeta("tokens_out", response.Usage.OutputTokens);
    handle.SetMeta("cost_usd",   0.0012m);
    return response.Content[0].Text;
});

Python

async def step(handle):
    response = await llm.call(text)
    handle.set_meta("model",      response.model)
    handle.set_meta("tokens_in",  response.usage.input_tokens)
    handle.set_meta("tokens_out", response.usage.output_tokens)
    handle.set_meta("cost_usd",   0.0012)
    return response.content[0].text

output = await ctx.track_step_async("step-name", text, step)

Node

const output = await ctx.trackStepAsync("step-name", text, async (handle) => {
  const response = await llm.call(text);
  handle.setMeta("model",      response.model);
  handle.setMeta("tokens_in",  response.usage.input_tokens);
  handle.setMeta("tokens_out", response.usage.output_tokens);
  handle.setMeta("cost_usd",   0.0012);
  return response.content[0].text;
});

StepHandle well-known keys (all SDKs use the same string keys):

Key	Type	Effect
`model`	string	Stored on `step.model`
`tokens_in`	int	Stored on `step.tokens_in`
`tokens_out`	int	Stored on `step.tokens_out`
`cost_usd`	float	Stored on `step.cost_usd`
`type`	string	Use `"tool"` for tool-call steps
(any other key)	any	Stored in `step.metadata` dict

Nested steps

Pass the parent handle’s context to nest steps in the dashboard tree:

var result = await ctx.TrackStepAsync("pipeline", query, async handle =>
{
    var a = await ctx.TrackStepAsync("classify", query, async () => await Classify(query));
    var b = await ctx.TrackStepAsync("answer",   a,     async () => await Answer(a));
    return b;
});

Python

async def pipeline(handle):
    a = await ctx.track_step_async("classify", query, lambda: classify(query))
    b = await ctx.track_step_async("answer",   a,     lambda: answer(a))
    return b

result = await ctx.track_step_async("pipeline", query, pipeline)

Node

const result = await ctx.trackStepAsync("pipeline", query, async (handle) => {
  const a = await ctx.trackStepAsync("classify", query, async () => await classify(query));
  const b = await ctx.trackStepAsync("answer",   a,     async () => await answer(a));
  return b;
});

Trace-level metadata

ctx.SetMetadata("user_id", userId);
ctx.SetMetadata("region",  "us-east-1");

Python

ctx.set_metadata("user_id", user_id)
ctx.set_metadata("region",  "us-east-1")

Node

ctx.setMetadata("user_id", userId);
ctx.setMetadata("region",  "us-east-1");

Replay / Test SDK

VevalTestSdk is a drop-in test double. It mocks LLM step outputs from a recorded trace so no real API calls are made. Throws if a step name isn’t found in the trace (strict mode). Each call gets the recorded step with the same name and an identical input, else the next unused one in recorded order (reported as diverged in ReplayResult.Fidelity). A replay fails if a recorded step is never called.

var trace   = await veval.GetTraceAsync("tr_...");
var testSdk = new VevalTestSdk(new VevalOptions { ApiKey = "..." }).WithReplay(trace);
var service = new MyAgentService(testSdk, mockLlm);

var result = await testSdk.RunAsync("agent-name", service.ExecuteAsync);

Assert.Equal("success", testSdk.LastStatus);
Assert.Null(testSdk.LastError);

Python

trace    = await veval.get_trace_async("tr_...")
test_sdk = VevalTestSdk(VevalOptions(api_key="...")).with_replay(trace)
service  = MyAgentService(test_sdk, mock_llm)

result = await test_sdk.run_async("agent-name", service.execute_async)

assert test_sdk.last_status == "success"
assert test_sdk.last_error is None

Node

const trace   = await veval.getTraceAsync("tr_...");
const testSdk = new VevalTestSdk({ apiKey: "..." }).withReplay(trace);
const service = new MyAgentService(testSdk, mockLlm);

const result = await testSdk.runAsync("agent-name", (ctx) => service.execute(ctx));

expect(testSdk.lastStatus).toBe("success");
expect(testSdk.lastError).toBeNull();

ReplayAsync (lower-level)

var r = await testSdk.ReplayAsync(trace, service.ExecuteAsync,
    new ReplayOptions { MockLlmResponses = true, Assertions = [TraceAssert.NoErrors()] });
// r.Failures, r.ReplayedContext, r.Output, r.Status, r.Error

Python

r = await test_sdk.replay_async(trace, service.execute_async,
    ReplayOptions(mock_llm_responses=True, assertions=[TraceAssert.no_errors()]))
# r.failures, r.replayed_context, r.output, r.status, r.error

Node

const r = await testSdk.replayAsync(trace, (ctx) => service.execute(ctx),
  { mock_llm_responses: true, assertions: [TraceAssert.noErrors()] });
// r.failures, r.replayed_context, r.output, r.status, r.error

ctx.IsReplay (is_replay, isReplay) is true while a recorded trace is replayed (any mode, scenario trace-backed items, WithReplay). Only steps are served from the recording; check it before side effects outside a step, such as sending an email.

Assertions

All assertions implement ITraceAssertion. They return null on pass or a failure string. Built-in factory — all three languages:

C#	Python	Node	Description
`TraceAssert.NoErrors()`	`TraceAssert.no_errors()`	`TraceAssert.noErrors()`	Fail if any step has error status
`TraceAssert.MaxSteps(n)`	`TraceAssert.max_steps(n)`	`TraceAssert.maxSteps(n)`	Fail if total step count > n
`TraceAssert.MaxCost(n)`	`TraceAssert.max_cost(n)`	`TraceAssert.maxCost(n)`	Fail if total cost_usd > n
`TraceAssert.MaxDuration(ms)`	`TraceAssert.max_duration(ms)`	`TraceAssert.maxDuration(ms)`	Fail if total duration > ms
`TraceAssert.StepExists("name")`	`TraceAssert.step_exists("name")`	`TraceAssert.stepExists("name")`	Fail if named step not found
`TraceAssert.OutputContains("s")`	`TraceAssert.output_contains("s")`	`TraceAssert.outputContains("s")`	Fail if no step output contains s
`TraceAssert.ToolCalled("name")`	`TraceAssert.tool_called("name")`	`TraceAssert.toolCalled("name")`	Fail if no tool step with that name
`TraceAssert.Judge(veval, "rubric")`	`TraceAssert.judge(veval, "rubric")`	`TraceAssert.judge(veval, "rubric")`	Fail if the LLM judge’s verdict is not passed

Custom assertion:

public class MyAssertion : ITraceAssertion
{
    public string? Evaluate(VevalExecutionContext ctx)
    {
        // inspect ctx.Steps, return null to pass or a message to fail
        return null;
    }
}

Python

from veval import ITraceAssertion

class MyAssertion(ITraceAssertion):
    def evaluate(self, ctx) -> str | None:
        # inspect ctx.steps, return None to pass or a message to fail
        return None

Node

class MyAssertion {
  evaluate(ctx) {
    // inspect ctx.steps, return null to pass or a message to fail
    return null;
  }
}

Scenarios

Run your agent against multiple inputs and post pass/fail to the dashboard.

var result = await veval.RunScenarioAsync(
    scenarioName:       "my-scenario",
    agent:              service.ExecuteAsync,
    scenarioAssertions: [TraceAssert.NoErrors(), TraceAssert.MaxCost(0.10m)],
    items: [
        new ScenarioItem { Name = "q1", Input = "What is prompt caching?" },
        new ScenarioItem { Name = "replay", TraceId = "tr_...", Assertions = [TraceAssert.MaxCost(0.01m)] },
    ]
);
// result.Passed, result.PassCount, result.FailCount, result.Results

Python

result = await veval.run_scenario_async(
    scenario_name="my-scenario",
    agent=service.execute_async,
    scenario_assertions=[TraceAssert.no_errors(), TraceAssert.max_cost(0.10)],
    items=[
        ScenarioItem(name="q1", input="What is prompt caching?"),
        ScenarioItem(name="replay", trace_id="tr_...", assertions=[TraceAssert.max_cost(0.01)]),
    ],
)
# result.passed, result.pass_count, result.fail_count, result.results

Node

const result = await veval.runScenarioAsync(
  "my-scenario",
  (ctx) => service.execute(ctx),
  [TraceAssert.noErrors(), TraceAssert.maxCost(0.10)],
  [
    { name: "q1", input: "What is prompt caching?", assertions: [] },
    { name: "replay", trace_id: "tr_...", assertions: [TraceAssert.maxCost(0.01)] },
  ]
);
// result.passed, result.pass_count, result.fail_count, result.results

ScenarioItem fields: name (string), input (any), trace_id (string) — provide either input (live LLM) or trace_id (live replay: steps recorded as llm run live, all other steps return their recorded output), not both. assertions is always an array (can be empty). MockModelSteps / mock_model_steps (bool) makes a trace-backed item a fully mocked replay: free, but blind to prompt changes. RunScenarioAsync(..., mockModelSteps: true) (mock_model_steps=True; Node: 5th argument { mock_model_steps: true }) does it for every item, including dashboard-defined ones. VevalTestSdk always replays trace-backed items mocked. In ReplayOptions, LiveModelSteps / live_model_steps selects live replay; it fails if no step ran live or the agent calls a step the trace never recorded. A trace-backed item’s trace follows the plan’s retention window, which restarts each time the trace is replayed. Pinning the trace in the dashboard (Use as fixture or Pin on the trace page) keeps it indefinitely, up to the plan’s pinned-trace limit (Free 25, Starter 100, Growth 500, Pro 2,000). If the trace no longer exists, that item fails with trace '<id>' not found and the other items still run. Pass items: null / omit items to fetch them from the dashboard by scenarioName.

Snapshots

A snapshot is a stored known-good run: every step in order with its input and output. Comparing a run against it reports added, removed, and moved steps (including an extra call of a known step) and changed step inputs (prompts, tool arguments), each with a line diff. Saving by trace ID pins the trace, so baselines survive retention.

// save once (or again, after an intentional change)
await veval.SaveSnapshotAsync("briefing-baseline", "tr_golden...");

// in tests: an assertion like any other; missing baseline = failure
TraceAssert.MatchesSnapshot(veval, "briefing-baseline")

// guard a replay against its own recording (also fails on changed step inputs)
new ReplayOptions { MockLlmResponses = true, CompareWithRecording = new SnapshotOptions() }

// production: compare and record the result in the dashboard
var baseline = await veval.GetSnapshotAsync("briefing-baseline");
var diff = await veval.CompareSnapshotAsync("briefing-baseline", baseline!, ctx);
if (diff.HasChanges) logger.LogWarning(diff.Summary());

Python

await veval.save_snapshot_async("briefing-baseline", "tr_golden...")
TraceAssert.matches_snapshot(veval, "briefing-baseline")
ReplayOptions(mock_llm_responses=True, compare_with_recording=SnapshotOptions())
baseline = await veval.get_snapshot_async("briefing-baseline")
diff = await veval.compare_snapshot_async("briefing-baseline", baseline, ctx)

Node

await veval.saveSnapshotAsync("briefing-baseline", "tr_golden...");
TraceAssert.matchesSnapshot(veval, "briefing-baseline");
({ mock_llm_responses: true, compare_with_recording: {} });
const baseline = await veval.getSnapshotAsync("briefing-baseline");
const diff = await veval.compareSnapshotAsync("briefing-baseline", baseline, ctx);

SnapshotOptions: CompareInputs (default true), CompareOutputs (default false), IgnoreSteps, IgnoreFields (JSON keys dropped at any depth), Normalize ((stepName, value) => value). Python snake_case, Node camelCase. SnapshotDiff: has_changes, changes (kind Added/Removed/Moved/InputChanged/OutputChanged, step_name, occurrence, positions, detail line diff), alignment, summary(), plus legacy added_steps, removed_steps, order_changes. C# uses PascalCase (HasChanges, Changes, Summary()). LoadSnapshotAsync(traceId) still builds a snapshot from a trace on the fly, but it isn’t pinned — prefer SaveSnapshotAsync.

Naming convention cheat sheet

Concept	C#	Python	Node
SDK class	`VevalSdk`	`VevalSdk`	`VevalSdk`
Test SDK	`VevalTestSdk`	`VevalTestSdk`	`VevalTestSdk`
Options	`VevalOptions { ApiKey }`	`VevalOptions(api_key=)`	`{ apiKey: }`
Strict sends	`ThrowOnSendFailure`	`raise_on_send_failure`	`throwOnSendFailure`
Send failure	`VevalSendException`	`VevalSendError`	`VevalSendError`
Run agent	`RunAsync`	`run_async`	`runAsync`
Record step	`TrackStepAsync`	`track_step_async`	`trackStepAsync`
Sync run / step (Python only)	—	`run` / `track_step`	—
Replay check	`ctx.IsReplay`	`ctx.is_replay`	`ctx.isReplay`
Trace metadata	`SetMetadata`	`set_metadata`	`setMetadata`
Step metadata	`handle.SetMeta`	`handle.set_meta`	`handle.setMeta`
Load replay	`WithReplay`	`with_replay`	`withReplay`
Run replay	`ReplayAsync`	`replay_async`	`replayAsync`
Run scenario	`RunScenarioAsync`	`run_scenario_async`	`runScenarioAsync`
Save snapshot	`SaveSnapshotAsync`	`save_snapshot_async`	`saveSnapshotAsync`
Get snapshot	`GetSnapshotAsync`	`get_snapshot_async`	`getSnapshotAsync`
Snapshot assertion	`TraceAssert.MatchesSnapshot`	`TraceAssert.matches_snapshot`	`TraceAssert.matchesSnapshot`
Compare snapshot	`CompareSnapshotAsync`	`compare_snapshot_async`	`compareSnapshotAsync`
Assertion factory	`TraceAssert.NoErrors()`	`TraceAssert.no_errors()`	`TraceAssert.noErrors()`
Context input	`ctx.Input`	`ctx.input`	`ctx.input`
Context trace ID	`ctx.TraceId`	`ctx.trace_id`	`ctx.traceId`
Last run status	`testSdk.LastStatus`	`test_sdk.last_status`	`testSdk.lastStatus`

Get started

Concepts

Guides

Development

What Veval does

Install

Initialize

Options

Delivery failures

Core API

RunAsync — wrap a complete agent run

TrackStepAsync — record one LLM call or sub-operation

Nested steps

Trace-level metadata

Replay / Test SDK

ReplayAsync (lower-level)

Assertions

Scenarios

Snapshots

Naming convention cheat sheet

​What Veval does

​Install

​Initialize

​Options

​Delivery failures

​Core API

​RunAsync — wrap a complete agent run

​TrackStepAsync — record one LLM call or sub-operation

​Nested steps

​Trace-level metadata

​Replay / Test SDK

​ReplayAsync (lower-level)

​Assertions

​Scenarios

​Snapshots

​Naming convention cheat sheet

What Veval does

Install

Initialize

Options

Delivery failures

Core API

RunAsync — wrap a complete agent run

TrackStepAsync — record one LLM call or sub-operation

Nested steps

Trace-level metadata

Replay / Test SDK

ReplayAsync (lower-level)

Assertions

Scenarios

Snapshots

Naming convention cheat sheet