Evaluation overview

Evaluation on Tanomis is built on the run you already have. Every run produces an execution trace; evaluators score traces into assessments; a set of assessments over a dataset is an experiment; and a comparison between experiments becomes a verdict that can gate a rollout.

Availability: Evaluation is enabled per tenant, shadow-first. See What is available where.

At a glance

• Trace: one OpenTelemetry trace per run, on the tenant’s own trace plane, with content captured at the level tenant policy allows.
• Assessment: a scorer’s verdict on one trace, with a reason and the evidence it used.
• Experiment: a candidate version replayed over a dataset version, with aggregate metrics.
• Gate: a candidate-versus-baseline comparison whose verdict can block promotion.

The flow

Step	What it is	Where you see it
Run	Any run, from the Console, the CLI, or your application	Sessions
Trace	The run’s model calls, tool calls, timing, and captured content	Evidence, Evaluations
Assessment	A scorer’s verdict on a trace	Evaluations
Experiment	A candidate replayed over a dataset, with rolled-up metrics	Evaluations
Verdict	Candidate compared with baseline against a threshold	tanomis eval compare, the promotion gate

Nothing in this flow copies content into the audit ledger. The ledger records that a gate decision was made and references the trace content by hash; the trace store holds the content under its own capture, retention, and access rules. See Audit and evidence.

Offline and online

Offline evaluation scores a candidate before it serves traffic: pin a target, replay a dataset, compare with the baseline. This is the path in Evaluate a version.

Online evaluation samples live runs and scores them continuously so drift shows up early. It is planned; see Online monitoring.

What decides a score

Scorers run over traces. Deterministic scorers check structure and outcomes; judge-model scorers ask a model, through the same gateway agents use and under an allowlist your administrator controls. Human feedback recorded beside an assessment is how a judge is calibrated before a gate is allowed to enforce.

Where to go next

• Capture execution traces: what a trace contains and how much content it carries.
• Evaluate a version: the offline path, step by step.
• Datasets and experiments and Scorers and judge models: planned pages for the workbench.