# Roll out and roll back

Move a global agent version to customers one tenant at a time with a canary, graduate it, and restore the known-good baseline in one command.

Move a new version of a global agent into service one customer at a time. The loop is always the
same: **inspect → preview → change → verify**. You will canary `support-copilot@v2` for two
customer tenants, graduate it to their baseline, and, if the release must be reversed, repoint both
to the still-serveable known-good `v1`.

- **Inspect** what each tenant serves, **preview** with `--dry-run`, **change** one tenant at a time, **verify** with `versions --by-tenant`.
- A canary adds a version at a share while the baseline keeps serving; graduation makes it the baseline.
- Rolling back is enabling the still-serveable previous version; nothing is rebuilt and versions are never changed in place.
- Command output is provisional. Only the follow-up version view proves what a tenant serves.

Before: `acme-eu` and `acme-us` use baseline `v1` with no canary → Canary: both keep baseline `v1`
and add `v2` at 10% → Graduated: both use baseline `v2`, and unreferenced `v1` may become deprecated
→ Rolled back: both use serveable, deprecated `v1` with no canary. Version build and manifest
contents stay immutable, while lifecycle status and tenant assignments change separately.

## What each command changes

A version's lifecycle status and a tenant's traffic assignment are separate controls. Promotion
decides whether a global version may serve; enablement decides where it serves.

| Command | What it changes | Tenant traffic |
|---|---|---|
| `tanomis agent deploy` | Builds and registers an immutable version and, depending on scope, requests enablement. | Follow the deploy result. |
| `tanomis agent promote` | Moves a global version to `production` after its gates pass. | None. |
| `tanomis agent enable` | Writes a tenant's baseline and optional canary directly. Accepts any serveable version, including deprecated. | The named tenants. |
| `tanomis agent rollout` | Promotes the version if needed, then enables it for the named tenants. | Only the named tenants. |
| `tanomis agent disable` | Removes the named tenants' enablement; the agent and versions stay. | Stops service there. It is a kill switch, not version rollback. |

A **baseline** is the version a tenant normally receives; a **canary** is an optional second
version with a tenant-specific share.

## 1. Check access and current assignments

You need an authenticated operator context authorized for `read:agents`, `promote:agents`, and every
named target tenant, plus an existing global agent, a known-good production version, a candidate
version, and explicit tenant IDs. Scopes alone do not grant platform support access to a customer
workspace: that path also requires a valid, short-lived support assertion bound to the target
customer tenant, obtained through your organization's authorized support workflow. A
support assertion is bound to exactly one target tenant. Platform support acting into
customer workspaces must run one tenant per command under that tenant's assertion; comma-separated
batches are only for an operator whose ordinary authority already covers every named tenant.

```bash title="Terminal"
tanomis auth status
tanomis agent versions support-copilot --by-tenant
```

`tanomis auth status` confirms the context and login; it does not list granted scopes, tenant
grants, or a support assertion, so a permission error can mean any of the three is missing.

Confirm that `v1` is `production`, both tenants list `v1` under **BASELINE TENANTS**, and `v2` is
the candidate. Record `v1`: that is the rollback target. The Registry may deprecate an older production version
once a newer one supersedes it and nothing references it. Built-in `deprecated` is terminal for status
transitions but remains serveable: tenants can keep using it, and rollback can repoint tenants to it
with `agent enable`; `agent rollout` cannot perform that rollback because it first tries to return
every non-production target to `production`. A tenant with no baseline needs one before a canary;
canary mode cannot create a tenant's first assignment.

## 2. Preview one tenant's canary

Start with one tenant and a 10% share. A canary share must be from 1 through 99; omitting
`--canary-share` means a full baseline assignment.

```bash title="Terminal"
tanomis agent rollout support-copilot \
  --version v2 \
  --tenants acme-eu \
  --canary-share 10 \
  --dry-run
```

```text
dry-run: rollout support-copilot@v2 (would first promote v2 registered -> production)
TENANT          BEFORE (base/canary)      AFTER (base/canary)
acme-eu         v1 / -                    v1 / v2 (10%)
no changes made (dry-run)
```

Dry run reads the current state and prints the proposed matrix without
running promotion gates, promoting, or enabling anything. Stop if the tenant, baseline, candidate,
or share differs from your plan. It does not predict application health or move existing sessions.

## 3. Start the canary and verify it

```bash title="Terminal"
tanomis agent rollout support-copilot \
  --version v2 \
  --tenants acme-eu \
  --canary-share 10
tanomis agent versions support-copilot --by-tenant
```

Rollout first ensures `v2` is `production`; if a required gate fails, the command stops before enabling any tenant.
It then uses the `v1` baseline observed for `acme-eu` when the command began and adds `v2` as the
10% canary. Serialize assignment changes for each tenant: canary does not use a compare-and-set precondition, so a
concurrent baseline change after the initial read can be overwritten by the canary's cached baseline.

Expect `acme-eu` beside `v1` in **BASELINE TENANTS** and `acme-eu (10%)` beside `v2` in **CANARY
TENANTS**. Treat canary output and process exit as provisional: a missing baseline is printed as a
skip, but another tenant assignment failure can produce an apparent success line and a zero-failure
summary. Only this follow-up version view proves the assignment changed. If it does not match,
retry only the tenant that did not change.

## 4. Expand to another named tenant

After your own health and outcome checks pass, preview and apply the same candidate and share for
the next tenant, then verify:

```bash title="Terminal"
tanomis agent rollout support-copilot \
  --version v2 \
  --tenants acme-us \
  --canary-share 10 \
  --dry-run
tanomis agent rollout support-copilot \
  --version v2 \
  --tenants acme-us \
  --canary-share 10
tanomis agent versions support-copilot --by-tenant
```

The share belongs to each tenant's assignment; it is not a global split across the tenant list.
Keep naming tenants explicitly as the rollout expands.

## 5. Graduate the candidate

When the canary meets your release criteria, preview a baseline rollout for the same tenants
without `--canary-share`. The batch form assumes ordinary operator authority over both tenants;
support act-as previews and applies each tenant separately.

```bash title="Terminal"
tanomis agent rollout support-copilot \
  --version v2 \
  --tenants acme-eu,acme-us \
  --dry-run
```

The **AFTER** cells must read `v2 / -`. Then apply and verify:

```bash title="Terminal"
tanomis agent rollout support-copilot \
  --version v2 \
  --tenants acme-eu,acme-us
tanomis agent versions support-copilot --by-tenant
```

For each successful tenant, the plain rollout makes `v2` the 100% baseline and clears its canary.
A partially failed baseline rollout returns an error, but still read per-tenant results and the
version view before declaring it complete. If these were the last `v1` assignments, expect `v1` to
become `deprecated`: superseded, not deleted or unservable.

## 6. Roll back to the known-good version

Rollback repoints enablement to the serveable version recorded in step 1; it is not a lifecycle
transition. Re-check that `v1` is `production` or built-in `deprecated`. `agent enable` has no dry-run mode,
so write the preview down first:

```text
TENANT          BEFORE (base/canary)      AFTER (base/canary)
acme-eu         v2 / -                    v1 / -
acme-us         v2 / -                    v1 / -
```

```bash title="Terminal"
tanomis agent enable support-copilot \
  --tenants acme-eu,acme-us \
  --baseline v1
tanomis agent versions support-copilot --by-tenant
```

Each successful tenant now serves `v1` with no canary, even while `v1` is `deprecated`; the
candidate remains an immutable version. Like canary mode, `agent enable` prints per-tenant failures
without a failing exit for a partial result, so require a summary with zero failed tenants and
verify both under the `v1` baseline. Do not use `tanomis agent disable` for this: it removes service
rather than returning a tenant to a known-good version.

## Interpret failures before retrying

| Signal | What changed | Next action |
|---|---|---|
| Permission denied | Nothing. The scope, tenant grant, support assertion, or its tenant binding may be missing. | Have an administrator confirm all of them, then retry. |
| Version not found | Nothing. | Re-run the version view and use an exact version label. |
| Required promotion gate failed | The candidate did not reach `production`; rollout stopped before enablement. | Resolve the gate, then repeat the dry run. |
| Known-good version is built-in `deprecated` | It is serveable, but `rollout` cannot transition it back. | Use named-tenant `agent enable --baseline`. |
| Known-good version is not serveable | The Registry rejects the enablement. | Build the known-good source as a new immutable version. |
| `cannot canary — tenant has no baseline` | That tenant was skipped; canary mode may still exit successfully. | Assign a baseline first, or drop the tenant from the list. |
| Canary output looks successful but the version view disagrees | The tenant did not change. | Retry only that tenant. |
| Another operator changed the baseline during a canary | Canary may have restored the baseline it observed at start. | Serialize assignment changes; repeat the preview. |
| Baseline rollout reports a partial failure | Successful tenants changed; a failed tenant's final assignment may be unchanged or may have committed. | Read the version view before retrying. |
| Rollback enable reports failed tenants | Process exit may still be successful, and a failed tenant's final assignment is unknown until read back. | Read the version view, then re-enable the failed tenants. |

The safe stopping point is always explicit state: the named tenants under the intended baseline
and canary columns in `tanomis agent versions support-copilot --by-tenant`.

## Where to go next

- [Quickstart](/getting-started/quickstart): the first deploy.
- [Deploy from CI](/guides/deploy/deploy-from-ci): the rollout job behind an approval.
- [Journey commands](/cli#roll-out-and-roll-back): compact command lookup.