# Evaluate a version

Test a candidate version against the version serving a tenant on real past runs, and post the result on the pull request.

Before a version serves real traffic, replay real conversations through it and through the version
you already trust, grade both with the agent's checks, and see what got worse. One command does it,
so it drops into the pipeline that builds the agent.

> **Availability:** Evaluation is enabled per tenant. See [What is available where](/reference/availability).

- **Checks:** the agent's checks (Quality > Checks) grade both versions, the same checks that
  grade its live runs.
- **Dataset:** runs you picked in the Console with **Add to dataset**. The replay reads each run's
  first message again; nothing from the conversation is copied into the dataset.
- **One command:** `tanomis eval ci` replays the dataset through both versions at once, compares
  them, and replays what got worse once more before calling the candidate worse.
- **Result:** a Markdown summary for the pull request, and an exit code the pipeline can act on.

## Before you start

- The agent has checks. `tanomis eval ci` grades with them and refuses to run without them.
- A dataset of the agent's runs. On the Quality screen, open a run and choose **Add to dataset**;
  collect the conversations you want every new version to keep handling well.
- The candidate version is built and registered in the tenant. Stage it so nobody is served it
  yet:

  ```bash title="Terminal"
  tanomis agent deploy agents/support-copilot --stage
  ```

  This stages the version in `agent.yaml`, say `0.4.0`. A staged version can start sessions, so it
  can be replayed, but the tenant keeps serving the version it served. Release it within 14 days
  with `tanomis agent promote support-copilot --version 0.4.0 --scope-tenant TENANT`; a staged
  version nobody releases is then rejected and its storage reclaimed.

## Run it

```bash title="Terminal"
tanomis eval ci support-copilot --dataset refunds --candidate 0.4.0 --summary summary.md
```

The baseline is the version serving the tenant; name another with `--baseline`. Pin a dataset
version with `--dataset refunds@3`, otherwise the latest is used. `--threshold 0.05` lets up to 5%
of the comparable examples get worse before the test fails; the default is none.

The command prints the summary and exits with:

| Exit code | Meaning |
| --- | --- |
| `0` | No worse than the baseline. |
| `1` | Worse: some examples that passed on the baseline fail on the candidate. |
| `2` | Not enough to tell, for example when too few examples could be replayed. |
| `3` | The test could not run (no checks, unknown dataset or version, lost connection). |

An answer of "worse" is checked before it is given. Each example that passed on the baseline and
failed on the candidate is replayed once more on the candidate; it counts only if it fails again.
One that passes on the second try is listed as flaky and not counted, and the verdict is worked
out again from the examples that failed twice. Turn this off with `--confirm=false`.

The summary lists each check's pass rate on both versions, each example that got worse with a link
to its run in the Console, and, separately, the flaky ones. The Console's address is taken from
the API's (`api.example.com` gives `console.example.com`); name it with `--console-url` if yours
differs. When it cannot be worked out, the summary shows trace ids instead.

Each example runs through the production run path on both versions, with the tenant's real
connections: a tool call a replay makes really happens. Test in a tenant whose connections point
at test systems, such as your sandbox tenant, and build the dataset from that tenant's runs.

A replay runs with nobody there. If a version stops to ask a person, for example because a rule
holds a shell command for approval, the replay is stopped at once, nothing is left in your
approvals inbox, and the summary counts that example as one that could not be replayed and says
what it asked for. A run that had attached files cannot be replayed either.

## In a pull request

The job stages the pull request's version in the sandbox tenant that holds the dataset, then tests
it against the version that tenant serves. Staging changes nothing anyone is served; the replays
themselves are real runs (see above).

It uses two [automation clients](/guides/deploy/automation-credentials): a `deploy` client to stage
the version, and an `invoke` client for the test (`invoke` can replay and grade; it cannot deploy).
A deploy client could also release a version, so run the job only for pull requests from branches
of your own repository, in a protected environment. It runs on a GitHub-hosted runner, which
already has `yq` and `gh`; the first step installs the `tanomis` CLI, pinned to a release so a new
CLI version cannot change the job under you. It posts the summary on the pull request:

```yaml title=".github/workflows/agent-eval.yml"
name: support-copilot eval
on:
  pull_request:
    paths: ["agents/support-copilot/**"]

jobs:
  eval:
    if: github.event.pull_request.head.repo.full_name == github.repository
    runs-on: ubuntu-latest
    environment: eval
    permissions:
      pull-requests: write
    env:
      TANOMIS_API_URL: ${{ vars.TANOMIS_API_URL }}
      TANOMIS_TENANT: northwind-sandbox
      AGENT: support-copilot
      CANDIDATE: 0.0.0-pr${{ github.event.pull_request.number }}.${{ github.run_id }}
    steps:
      - uses: actions/checkout@v4
      - name: Install tanomis
        run: |
          curl -fsSL https://get.tanomis.com/install.sh | TANOMIS_VERSION=v0.6.0 TANOMIS_INSTALL_DIR="$HOME/.local/bin" sh
          echo "$HOME/.local/bin" >> "$GITHUB_PATH"
      - name: Stage the candidate
        run: |
          yq -i '.version = env(CANDIDATE)' "agents/$AGENT/agent.yaml"
          tanomis agent deploy "agents/$AGENT" --stage --no-stream-logs
        env:
          TANOMIS_CLIENT_ID: ${{ secrets.TANOMIS_DEPLOY_CLIENT_ID }}
          TANOMIS_CLIENT_SECRET: ${{ secrets.TANOMIS_DEPLOY_CLIENT_SECRET }}
      - name: Test the candidate
        id: eval
        run: |
          set +e
          tanomis eval ci "$AGENT" --dataset refunds --candidate "$CANDIDATE" --summary summary.md
          echo "code=$?" >> "$GITHUB_OUTPUT"
        env:
          TANOMIS_CLIENT_ID: ${{ secrets.TANOMIS_EVAL_CLIENT_ID }}
          TANOMIS_CLIENT_SECRET: ${{ secrets.TANOMIS_EVAL_CLIENT_SECRET }}
      - name: Comment on the pull request
        if: always() && hashFiles('summary.md') != ''
        run: gh pr comment "${{ github.event.pull_request.number }}" --body-file summary.md
        env:
          GH_TOKEN: ${{ github.token }}
      - name: Fail unless the candidate passed
        if: steps.eval.outputs.code != '0'
        run: exit ${{ steps.eval.outputs.code }}
```

The last step fails the check for any exit code other than `0`: worse (`1`), not enough to tell
(`2`), or could not run (`3`). Each run stages a new version under its own label; staged versions
serve nobody, and one not released within 14 days is rejected and its storage reclaimed. The version you release is built from the merged source by your
deploy pipeline (see [Deploy from CI](/guides/deploy/deploy-from-ci)).

The test replays each example on both versions at the same time, so it takes minutes, not seconds.
`--timeout` bounds it (45 minutes by default), and a machine client's token is renewed while it
waits.

The baseline is replayed only when needed. If the same baseline version was graded with the same
checks on the same dataset version within the last 7 days, that result is reused and only the
candidate is replayed; the summary says so and when the baseline was graded. `--baseline-max-age`
changes the 7 days (`--baseline-max-age 0` always replays the baseline), and
`--baseline-experiment` names a finished baseline experiment to reuse; it must have been graded
with the same checks on the same dataset version.

## Step by step

`tanomis eval ci` is built from commands you can also run yourself:

```bash title="Terminal"
tanomis eval target create --agent-version VERSION_ID --dataset-version DATASET_VERSION_ID
tanomis eval target run TARGET_ID --max-examples 50
tanomis eval compare BASELINE_EXPERIMENT --to CANDIDATE_EXPERIMENT --threshold 0.02
```

A target pins one version to one dataset version; running it replays the examples through the
production run path. `tanomis eval compare` exits non-zero unless the verdict is `pass`.

The Evaluations screen in the Console lists each experiment's runs with their results, cost, and
latency, and opens the trace behind each one.

## Where to go next

- [Roll out and roll back](/guides/deploy/rollout-and-rollback): what to do once the candidate
  passes.
- [Deploy from CI](/guides/deploy/deploy-from-ci): the deploy half of the pipeline.