Roll out and roll back

6 min readUpdated 6 Sep 2026

Move a new version of a global agent into service one customer at a time. The loop is always the same: inspect → preview → change → verify. You will canary support-copilot@v2 for two customer tenants, graduate it to their baseline, and, if the release must be reversed, repoint both to the still-serveable known-good v1.

Versions stay fixed. Tenant assignments move.Baseline / optional canary

Tenant	Beforev1 production · v2 registered	Canaryv1 + v2 production	Graduatedv1 deprecated · v2 production	Rolled backv1 deprecated · v2 production
acme-eu	baseline v1; canary none	baseline v1; canary v2 at 10%	baseline v2; canary none	baseline v1; canary none
acme-us	baseline v1; canary none	baseline v1; canary v2 at 10%	baseline v2; canary none	baseline v1; canary none

Before: acme-eu and acme-us use baseline v1 with no canary → Canary: both keep baseline v1 and add v2 at 10% → Graduated: both use baseline v2, and unreferenced v1 may become deprecated → Rolled back: both use serveable, deprecated v1 with no canary. Version build and manifest contents stay immutable, while lifecycle status and tenant assignments change separately.

What each command changes

A version’s lifecycle status and a tenant’s traffic assignment are separate controls. Promotion decides whether a global version may serve; enablement decides where it serves.

Command	What it changes	Tenant traffic
`tanomis agent deploy`	Builds and registers an immutable version and, depending on scope, requests enablement.	Follow the deploy result.
`tanomis agent promote`	Moves a global version to `production` after its gates pass.	None.
`tanomis agent enable`	Writes a tenant’s baseline and optional canary directly. Accepts any serveable version, including deprecated.	The named tenants.
`tanomis agent rollout`	Promotes the version if needed, then enables it for the named tenants.	Only the named tenants.
`tanomis agent disable`	Removes the named tenants’ enablement; the agent and versions stay.	Stops service there. It is a kill switch, not version rollback.

A baseline is the version a tenant normally receives; a canary is an optional second version with a tenant-specific share.

1. Check access and current assignments

You need an authenticated operator context authorized for read:agents, promote:agents, and every named target tenant, plus an existing global agent, a known-good production version, a candidate version, and explicit tenant IDs. Scopes alone do not grant platform support access to a customer workspace: that path also requires a valid, short-lived support assertion bound to the target customer tenant, obtained through your organization’s authorized support workflow. A support assertion is bound to exactly one target tenant. Platform support acting into customer workspaces must run one tenant per command under that tenant’s assertion; comma-separated batches are only for an operator whose ordinary authority already covers every named tenant.

tanomis auth status
tanomis agent versions support-copilot --by-tenant

tanomis auth status confirms the context and login; it does not list granted scopes, tenant grants, or a support assertion, so a permission error can mean any of the three is missing.

Confirm that v1 is production, both tenants list v1 under BASELINE TENANTS, and v2 is the candidate. Record v1: that is the rollback target. The Registry may deprecate an older production version once a newer one supersedes it and nothing references it. Built-in deprecated is terminal for status transitions but remains serveable: tenants can keep using it, and rollback can repoint tenants to it with agent enable; agent rollout cannot perform that rollback because it first tries to return every non-production target to production. A tenant with no baseline needs one before a canary; canary mode cannot create a tenant’s first assignment.

2. Preview one tenant’s canary

Start with one tenant and a 10% share. A canary share must be from 1 through 99; omitting --canary-share means a full baseline assignment.

tanomis agent rollout support-copilot \
  --version v2 \
  --tenants acme-eu \
  --canary-share 10 \
  --dry-run

dry-run: rollout support-copilot@v2 (would first promote v2 registered -> production)
TENANT          BEFORE (base/canary)      AFTER (base/canary)
acme-eu         v1 / -                    v1 / v2 (10%)
no changes made (dry-run)

Dry run reads the current state and prints the proposed matrix without running promotion gates, promoting, or enabling anything. Stop if the tenant, baseline, candidate, or share differs from your plan. It does not predict application health or move existing sessions.

3. Start the canary and verify it

tanomis agent rollout support-copilot \
  --version v2 \
  --tenants acme-eu \
  --canary-share 10
tanomis agent versions support-copilot --by-tenant

Rollout first ensures v2 is production; if a required gate fails, the command stops before enabling any tenant. It then uses the v1 baseline observed for acme-eu when the command began and adds v2 as the 10% canary. Serialize assignment changes for each tenant: canary does not use a compare-and-set precondition, so a concurrent baseline change after the initial read can be overwritten by the canary’s cached baseline.

Expect acme-eu beside v1 in BASELINE TENANTS and acme-eu (10%) beside v2 in CANARY TENANTS. Treat canary output and process exit as provisional: a missing baseline is printed as a skip, but another tenant assignment failure can produce an apparent success line and a zero-failure summary. Only this follow-up version view proves the assignment changed. If it does not match, retry only the tenant that did not change.

4. Expand to another named tenant

After your own health and outcome checks pass, preview and apply the same candidate and share for the next tenant, then verify:

tanomis agent rollout support-copilot \
  --version v2 \
  --tenants acme-us \
  --canary-share 10 \
  --dry-run
tanomis agent rollout support-copilot \
  --version v2 \
  --tenants acme-us \
  --canary-share 10
tanomis agent versions support-copilot --by-tenant

The share belongs to each tenant’s assignment; it is not a global split across the tenant list. Keep naming tenants explicitly as the rollout expands.

5. Graduate the candidate

When the canary meets your release criteria, preview a baseline rollout for the same tenants without --canary-share. The batch form assumes ordinary operator authority over both tenants; support act-as previews and applies each tenant separately.

tanomis agent rollout support-copilot \
  --version v2 \
  --tenants acme-eu,acme-us \
  --dry-run

The AFTER cells must read v2 / -. Then apply and verify:

tanomis agent rollout support-copilot \
  --version v2 \
  --tenants acme-eu,acme-us
tanomis agent versions support-copilot --by-tenant

For each successful tenant, the plain rollout makes v2 the 100% baseline and clears its canary. A partially failed baseline rollout returns an error, but still read per-tenant results and the version view before declaring it complete. If these were the last v1 assignments, expect v1 to become deprecated: superseded, not deleted or unservable.

6. Roll back to the known-good version

Rollback repoints enablement to the serveable version recorded in step 1; it is not a lifecycle transition. Re-check that v1 is production or built-in deprecated. agent enable has no dry-run mode, so write the preview down first:

TENANT          BEFORE (base/canary)      AFTER (base/canary)
acme-eu         v2 / -                    v1 / -
acme-us         v2 / -                    v1 / -

tanomis agent enable support-copilot \
  --tenants acme-eu,acme-us \
  --baseline v1
tanomis agent versions support-copilot --by-tenant

Each successful tenant now serves v1 with no canary, even while v1 is deprecated; the candidate remains an immutable version. Like canary mode, agent enable prints per-tenant failures without a failing exit for a partial result, so require a summary with zero failed tenants and verify both under the v1 baseline. Do not use tanomis agent disable for this: it removes service rather than returning a tenant to a known-good version.

Interpret failures before retrying

Signal	What changed	Next action
Permission denied	Nothing. The scope, tenant grant, support assertion, or its tenant binding may be missing.	Have an administrator confirm all of them, then retry.
Version not found	Nothing.	Re-run the version view and use an exact version label.
Required promotion gate failed	The candidate did not reach `production`; rollout stopped before enablement.	Resolve the gate, then repeat the dry run.
Known-good version is built-in `deprecated`	It is serveable, but `rollout` cannot transition it back.	Use named-tenant `agent enable --baseline`.
Known-good version is not serveable	The Registry rejects the enablement.	Build the known-good source as a new immutable version.
`cannot canary — tenant has no baseline`	That tenant was skipped; canary mode may still exit successfully.	Assign a baseline first, or drop the tenant from the list.
Canary output looks successful but the version view disagrees	The tenant did not change.	Retry only that tenant.
Another operator changed the baseline during a canary	Canary may have restored the baseline it observed at start.	Serialize assignment changes; repeat the preview.
Baseline rollout reports a partial failure	Successful tenants changed; a failed tenant’s final assignment may be unchanged or may have committed.	Read the version view before retrying.
Rollback enable reports failed tenants	Process exit may still be successful, and a failed tenant’s final assignment is unknown until read back.	Read the version view, then re-enable the failed tenants.

The safe stopping point is always explicit state: the named tenants under the intended baseline and canary columns in tanomis agent versions support-copilot --by-tenant.

Where to go next

Quickstart: the first deploy.
Deploy from CI: the rollout job behind an approval.
Journey commands: compact command lookup.