Instrument your agent

Every agent run is a trace: a record of inputs, outputs, timing, cost, and the steps in between.

Wrap the run

RunAsync starts a trace, runs your agent, then ships the trace to Veval.

Track each step

Your agent calls ctx.TrackStepAsync for each LLM call or sub-operation.

Open the dashboard

The trace appears in your dashboard with full step detail.

RunAsync

Wraps a complete agent invocation. Sends a trace on success or error.

var result = await veval.RunAsync(
    agentName: "my-agent",
    callback:  ctx => myService.ExecuteAsync(ctx),
    input:     userMessage   // optional — stored on the trace
);

result = await veval.run_async(
    "my-agent",
    lambda ctx: my_service.execute_async(ctx),
    user_message,   # optional — stored on the trace
)

const result = await veval.runAsync(
  "my-agent",
  (ctx) => myService.execute(ctx),
  userMessage   // optional — stored on the trace
);

Parameter	Type	Description
`agentName`	`string`	Identifies the agent in the dashboard
`callback`	`Func<VevalExecutionContext, Task<T>>`	Your agent logic
`input`	`object?`	Arbitrary input stored on the trace

Synchronous Python agents

If your Python agent is plain synchronous code, use run and track_step. They take ordinary functions and need no event loop. run sends the trace on the calling thread, so in async code, such as an async def web handler, use run_async instead. C# and Node agents use the async API.

Python

def answer(ctx):
    def call_model(handle):
        response = anthropic.messages.create(...)
        handle.set_meta("type", "llm")
        return response.content[0].text

    return ctx.track_step("answer", ctx.input, call_model)

result = veval.run("my-agent", answer, user_message)

Tests can stay synchronous too: get_trace, replay and run_scenario mirror their _async counterparts. Call them from synchronous code; inside a running event loop, await the _async versions instead. The async methods also accept a plain function as the agent.

Delivery failures

Tracing never breaks your app. If Veval can’t store a trace (wrong API key, quota used up, network down), the SDK logs a warning and your agent’s result comes back as usual. The warning names the HTTP status and, when the API gives one, the reason:

Veval: Sending trace failed with status 429: {"error":"Trace quota exceeded"}

Each SDK logs it in that language’s usual place:

SDK	Where the warning goes
C#	Standard error
Python	The `veval` logger, at `WARNING` level. With no logging set up, Python prints it to standard error. If your app raises the log level above `WARNING` or silences library loggers, re-enable `veval` to see it.
Node	`console.warn`

Posting a scenario run or a snapshot comparison to the dashboard fails the same way: a warning, and no exception unless strict mode is on. This only covers sending results. Calls that return something you asked for throw when they fail, whether or not strict mode is on. That includes SaveSnapshotAsync, GetSnapshotAsync, JudgeAsync and GetTraceAsync. It also includes RunScenarioAsync when it loads a scenario’s items or a saved trace to replay. Loading a trace or a scenario’s items throws VevalRequestException (C#) or VevalRequestError (Python, Node). Like the send error, it carries the HTTP status, or null when there was no response. GetTraceAsync returns null only when the trace doesn’t exist. A scenario whose items can’t be loaded throws instead of running zero items and passing.

Throw in tests and CI

In a test suite, a run that never reached the dashboard should fail the build. Turn on strict mode and a failed send throws VevalSendException (C#) or VevalSendError (Python, Node). The exception carries the HTTP status (StatusCode in C#, status in Python and Node). The status is null when the request got no response, and on the single error a scenario throws for lost item traces (see below).

var veval = new VevalSdk(new VevalOptions
{
    ApiKey             = "YOUR_API_KEY",
    ThrowOnSendFailure = true,
});

veval = VevalSdk(VevalOptions(api_key="YOUR_API_KEY", raise_on_send_failure=True))

const veval = new VevalSdk({ apiKey: "YOUR_API_KEY", throwOnSendFailure: true });

If your agent throws, you get your agent’s exception, not the send failure. The send failure is still logged. With strict mode on, CompareSnapshotAsync throws if it can’t post the comparison, so you don’t get the diff back. The comparison itself ran; only reporting it failed. In a scenario, a trace that can’t be stored doesn’t stop the remaining items. Every item runs, then the scenario run is posted. If posting the run fails, strict mode throws that failure. Otherwise, if any item traces were lost, it throws once for all of them: Veval: 3 trace(s) from scenario 'my-scenario' could not be stored. That error has no status; the warnings logged for each item have the details.

VevalExecutionContext

The context object passed into your agent. Do not instantiate directly — Veval creates it for you.

Member	Description
`TraceId`	Unique ID for this run (`tr_...`)
`Input`	The input passed to `RunAsync`
`TrackStepAsync(...)`	Records a step
`SetMetadata(key, value)`	Attaches trace-level metadata

TrackStepAsync

Records a single step within a trace.

// Simple overload — no handle needed
var summary = await ctx.TrackStepAsync("summarize", input: text, async () =>
{
    return await llm.Summarize(text);
});

// Handle overload — attach LLM metadata
var answer = await ctx.TrackStepAsync("answer", input: question, async handle =>
{
    var response = await claude.Messages.CreateAsync(...);

    handle.SetMeta("type",       "llm");

    handle.SetMeta("model",      response.Model);
    handle.SetMeta("tokens_in",  response.Usage.InputTokens);
    handle.SetMeta("tokens_out", response.Usage.OutputTokens);
    handle.SetMeta("cost_usd",   0.0012m);

    return response.Content[0].Text;
});

# Simple overload — no handle needed
async def summarize():
    return await llm.summarize(text)

summary = await ctx.track_step_async("summarize", text, summarize)

# Handle overload — attach LLM metadata
async def answer(handle):
    response = await anthropic.messages.create(...)

    handle.set_meta("type",       "llm")

    handle.set_meta("model",      response.model)
    handle.set_meta("tokens_in",  response.usage.input_tokens)
    handle.set_meta("tokens_out", response.usage.output_tokens)
    handle.set_meta("cost_usd",   0.0012)

    return response.content[0].text

answer_text = await ctx.track_step_async("answer", question, answer)

// Simple overload — no handle needed
const summary = await ctx.trackStepAsync("summarize", text, async () => {
  return await llm.summarize(text);
});

// Handle overload — attach LLM metadata
const answer = await ctx.trackStepAsync("answer", question, async (handle) => {
  const response = await anthropic.messages.create({ ... });

  handle.setMeta("type",       "llm");

  handle.setMeta("model",      response.model);
  handle.setMeta("tokens_in",  response.usage.input_tokens);
  handle.setMeta("tokens_out", response.usage.output_tokens);
  handle.setMeta("cost_usd",   0.0012);

  return response.content[0].text;
});

StepHandle metadata keys

handle.SetMeta(key, value) accepts these well-known keys, plus any custom string:

Key	Type	Description
`model`	`string`	Model name (e.g. `claude-sonnet-4-6`)
`tokens_in`	`int`	Input token count
`tokens_out`	`int`	Output token count
`cost_usd`	`decimal`	Cost in USD
`type`	`string`	Step type: `"llm"` for model calls, `"tool"` for tool calls. Live replay runs `llm` steps for real and serves every other step from the recording, so mark both
(custom)	`object`	Any other key stored in step metadata

Nested steps

Steps can be nested by passing the StepHandle to a child TrackStepAsync call.

var result = await ctx.TrackStepAsync("pipeline", input: query, async handle =>
{
    var classified = await ctx.TrackStepAsync("classify", input: query, async () =>
        await llm.Classify(query));

    var answered = await ctx.TrackStepAsync("answer", input: classified, async () =>
        await llm.Answer(classified));

    return answered;
});

async def pipeline(handle):
    async def classify():
        return await llm.classify(query)

    classified = await ctx.track_step_async("classify", query, classify)

    async def answer():
        return await llm.answer(classified)

    return await ctx.track_step_async("answer", classified, answer)

result = await ctx.track_step_async("pipeline", query, pipeline)

const result = await ctx.trackStepAsync("pipeline", query, async (handle) => {
  const classified = await ctx.trackStepAsync("classify", query, async () =>
    await llm.classify(query));

  const answered = await ctx.trackStepAsync("answer", classified, async () =>
    await llm.answer(classified));

  return answered;
});

Nested steps appear as a tree in the dashboard, letting you see exactly where time and cost are spent.

Trace-level metadata

Attach arbitrary key/value pairs to the whole trace (not a step):

ctx.SetMetadata("user_id",  userId);
ctx.SetMetadata("session",  sessionId);
ctx.SetMetadata("region",   "us-east-1");

ctx.set_metadata("user_id",  user_id)
ctx.set_metadata("session",  session_id)
ctx.set_metadata("region",   "us-east-1")

ctx.setMetadata("user_id",  userId);
ctx.setMetadata("session",  sessionId);
ctx.setMetadata("region",   "us-east-1");

Get started

Concepts

Guides

Development

Instrument your agent

RunAsync

Synchronous Python agents

Delivery failures

Throw in tests and CI

VevalExecutionContext

TrackStepAsync

StepHandle metadata keys

Nested steps

Trace-level metadata

What’s next

Test with replay

Run a scenario

​RunAsync

​Synchronous Python agents

​Delivery failures

​Throw in tests and CI

​VevalExecutionContext

​TrackStepAsync

​StepHandle metadata keys

​Nested steps

​Trace-level metadata

​What’s next

Test with replay

Run a scenario

RunAsync

Synchronous Python agents

Delivery failures

Throw in tests and CI

VevalExecutionContext

TrackStepAsync

StepHandle metadata keys

Nested steps

Trace-level metadata

What’s next