Evaluate a version

Before a version serves real traffic, replay real conversations through it and through the version you already trust, grade both with the agent’s checks, and see what got worse. One command does it, so it drops into the pipeline that builds the agent.

Availability: Evaluation is enabled per tenant. See What is available where.

At a glance

• Checks: the agent’s checks (Quality > Checks) grade both versions, the same checks that grade its live runs.
• Dataset: runs you picked in the Console with Add to dataset. The replay reads each run’s first message again; nothing from the conversation is copied into the dataset.
• One command: tanomis eval ci replays the dataset through both versions at once, compares them, and replays what got worse once more before calling the candidate worse.
• Result: a Markdown summary for the pull request, and an exit code the pipeline can act on.

Before you start

• The agent has checks. tanomis eval ci grades with them and refuses to run without them.

• A dataset of the agent’s runs. On the Quality screen, open a run and choose Add to dataset; collect the conversations you want every new version to keep handling well.

• The candidate version is built and registered in the tenant. Stage it so nobody is served it yet:

  Terminal

  tanomis agent deploy agents/support-copilot --stage

  This stages the version in agent.yaml, say 0.4.0. A staged version can start sessions, so it can be replayed, but the tenant keeps serving the version it served. Release it within 14 days with tanomis agent promote support-copilot --version 0.4.0 --scope-tenant TENANT; a staged version nobody releases is then rejected and its storage reclaimed.

Run it

Terminal

tanomis eval ci support-copilot --dataset refunds --candidate 0.4.0 --summary summary.md

The baseline is the version serving the tenant; name another with --baseline. Pin a dataset version with --dataset refunds@3, otherwise the latest is used. --threshold 0.05 lets up to 5% of the comparable examples get worse before the test fails; the default is none.

The command prints the summary and exits with:

Exit code	Meaning
0	No worse than the baseline.
1	Worse: some examples that passed on the baseline fail on the candidate.
2	Not enough to tell, for example when too few examples could be replayed.
3	The test could not run (no checks, unknown dataset or version, lost connection).

An answer of “worse” is checked before it is given. Each example that passed on the baseline and failed on the candidate is replayed once more on the candidate; it counts only if it fails again. One that passes on the second try is listed as flaky and not counted, and the verdict is worked out again from the examples that failed twice. Turn this off with --confirm=false.

The summary lists each check’s pass rate on both versions, each example that got worse with a link to its run in the Console, and, separately, the flaky ones. The Console’s address is taken from the API’s (api.example.com gives console.example.com); name it with --console-url if yours differs. When it cannot be worked out, the summary shows trace ids instead.

A replay is a real run

Each example runs through the production run path on both versions, with the tenant’s real connections: a tool call a replay makes really happens. Test in a tenant whose connections point at test systems, such as your sandbox tenant, and build the dataset from that tenant’s runs.

What a replay cannot do

A replay runs with nobody there. If a version stops to ask a person, for example because a rule holds a shell command for approval, the replay is stopped at once, nothing is left in your approvals inbox, and the summary counts that example as one that could not be replayed and says what it asked for. A run that had attached files cannot be replayed either.

In a pull request

The job stages the pull request’s version in the sandbox tenant that holds the dataset, then tests it against the version that tenant serves. Staging changes nothing anyone is served; the replays themselves are real runs (see above).

It uses two automation clients: a deploy client to stage the version, and an invoke client for the test (invoke can replay and grade; it cannot deploy). A deploy client could also release a version, so run the job only for pull requests from branches of your own repository, in a protected environment. It runs on a GitHub-hosted runner, which already has yq and gh; the first step installs the tanomis CLI, pinned to a release so a new CLI version cannot change the job under you. It posts the summary on the pull request:

.github/workflows/agent-eval.yml

name: support-copilot eval
on:
  pull_request:
    paths: ["agents/support-copilot/**"]

jobs:
  eval:
    if: github.event.pull_request.head.repo.full_name == github.repository
    runs-on: ubuntu-latest
    environment: eval
    permissions:
      pull-requests: write
    env:
      TANOMIS_API_URL: ${{ vars.TANOMIS_API_URL }}
      TANOMIS_TENANT: northwind-sandbox
      AGENT: support-copilot
      CANDIDATE: 0.0.0-pr${{ github.event.pull_request.number }}.${{ github.run_id }}
    steps:
      - uses: actions/checkout@v4
      - name: Install tanomis
        run: |
          curl -fsSL https://get.tanomis.com/install.sh | TANOMIS_VERSION=v0.6.0 TANOMIS_INSTALL_DIR="$HOME/.local/bin" sh
          echo "$HOME/.local/bin" >> "$GITHUB_PATH"
      - name: Stage the candidate
        run: |
          yq -i '.version = env(CANDIDATE)' "agents/$AGENT/agent.yaml"
          tanomis agent deploy "agents/$AGENT" --stage --no-stream-logs
        env:
          TANOMIS_CLIENT_ID: ${{ secrets.TANOMIS_DEPLOY_CLIENT_ID }}
          TANOMIS_CLIENT_SECRET: ${{ secrets.TANOMIS_DEPLOY_CLIENT_SECRET }}
      - name: Test the candidate
        id: eval
        run: |
          set +e
          tanomis eval ci "$AGENT" --dataset refunds --candidate "$CANDIDATE" --summary summary.md
          echo "code=$?" >> "$GITHUB_OUTPUT"
        env:
          TANOMIS_CLIENT_ID: ${{ secrets.TANOMIS_EVAL_CLIENT_ID }}
          TANOMIS_CLIENT_SECRET: ${{ secrets.TANOMIS_EVAL_CLIENT_SECRET }}
      - name: Comment on the pull request
        if: always() && hashFiles('summary.md') != ''
        run: gh pr comment "${{ github.event.pull_request.number }}" --body-file summary.md
        env:
          GH_TOKEN: ${{ github.token }}
      - name: Fail unless the candidate passed
        if: steps.eval.outputs.code != '0'
        run: exit ${{ steps.eval.outputs.code }}

The last step fails the check for any exit code other than 0: worse (1), not enough to tell (2), or could not run (3). Each run stages a new version under its own label; staged versions serve nobody, and one not released within 14 days is rejected and its storage reclaimed. The version you release is built from the merged source by your deploy pipeline (see Deploy from CI).

The test replays each example on both versions at the same time, so it takes minutes, not seconds. --timeout bounds it (45 minutes by default), and a machine client’s token is renewed while it waits.

The baseline is replayed only when needed. If the same baseline version was graded with the same checks on the same dataset version within the last 7 days, that result is reused and only the candidate is replayed; the summary says so and when the baseline was graded. --baseline-max-age changes the 7 days (--baseline-max-age 0 always replays the baseline), and --baseline-experiment names a finished baseline experiment to reuse; it must have been graded with the same checks on the same dataset version.

Step by step

tanomis eval ci is built from commands you can also run yourself:

Terminal

tanomis eval target create --agent-version VERSION_ID --dataset-version DATASET_VERSION_ID
tanomis eval target run TARGET_ID --max-examples 50
tanomis eval compare BASELINE_EXPERIMENT --to CANDIDATE_EXPERIMENT --threshold 0.02

A target pins one version to one dataset version; running it replays the examples through the production run path. tanomis eval compare exits non-zero unless the verdict is pass.

Checkpoint

The Evaluations screen in the Console lists each experiment’s runs with their results, cost, and latency, and opens the trace behind each one.

Where to go next

• Roll out and roll back: what to do once the candidate passes.
• Deploy from CI: the deploy half of the pipeline.