Attestation Mode Rollout

Audience: operators turning on the runtime attestation gate (#725 PR6) on a fleet that has been running with AUTONOMY_ATTESTATION_MODE=off. The gate denies tool execution when a node’s enrollment or rollout state is not coherent with the bundle’s declared provenance. Flipping straight to enforce without soak time means any operator misalignment surfaces as production 403s. This runbook is the safe sequence.

Prerequisites

  • All nodes in the target fleet have been enrolled via autonomy node enroll --node-id <id> --enrollment-ref <ref>.

  • Bundle authors have started declaring bundle.Manifest.Provenance.EnrollmentRef on signed bundles (schema v1.2+, see Security Model — Attestation join key).

  • Each runtime has AUTONOMY_NODE_ID (or identity.node_id in config) set to the same string used at enrollment.

  • Operator has access to the WAL via autonomy wal inspect and the orchestrator read APIs via autonomy attestation status.

Procedure

1. Verify enrollment substrate fleet-wide

For each node, confirm the enrollment row matches the bundle’s declared binding:

autonomy attestation status --node-id <id>

Expected output: ENROLLMENT_REF populated, STATUS=active, ROLLOUT_STATE either <none> or one of pending / acknowledged / active. A <none> ENROLLMENT means the node was missed — re-enroll before proceeding.

For CI / scripted verification, drive autonomy attestation eval against every node + candidate bundle pair:

autonomy bundle pull <ref> /tmp/bundle.tar
autonomy attestation eval --bundle /tmp/bundle.tar --node-id <id>

Non-zero exit on deny means the fleet is not ready for enforce. Fix the misalignment before flipping the mode.

2. Flip to advisory

On every runtime:

AUTONOMY_ATTESTATION_MODE=advisory <restart runtime process>

Advisory mode runs the gate but never returns 403. Every would-be deny is written to the WAL as a second autonomy.decision frame under the same audit_id as the policy-layer allow.

3. Soak

Let traffic run for at least one full directive cycle (typically 24h). Scrape advisory denies from the WAL:

autonomy wal inspect --kind autonomy.decision \
  | jq 'select(.attrs.reason | startswith("attestation:"))'

Expected results, in order of severity:

Reason prefix

Action

enrollment_revoked

Investigate why the node was revoked; re-enroll if intentional

enrollment_mismatch

The bundle’s binding doesn’t match the enrollment row — verify the operator typed the same string at both ends, re-enroll if needed

rollout_state_invalid

Node is in a rollback or failed state; resolve the rollout incident before flipping enforce

Do not flip enforce while ANY non-zero advisory denies are appearing unless you intentionally want those denies to start returning 403.

4. Flip to enforce

When the advisory denies are zero (or only on nodes you’ve intentionally revoked), flip:

AUTONOMY_ATTESTATION_MODE=enforce <restart runtime process>

The runtime now returns 403 on any sub-check failure. The wire response carries the canonical reason prefix (attestation: <sub-check>) so client code can branch on it.

5. Verify the layered WAL

Pick a known-good audit_id (any tool call that succeeded post- enforce) and confirm the dual-emit shape:

autonomy audit get <audit_id>

Expected: outcome=allow, reason from the policy layer. No attestation frame means the gate didn’t object — the happy path.

For a deny path, force a known mismatch (revoke a test node, attempt a request) and confirm:

autonomy audit get <audit_id>

Expected: outcome=deny, reason starts with attestation:.

Recovery

If enforce surfaces unexpected denies in production:

  1. Immediate: flip back to advisory and restart. Traffic resumes; the WAL still records the would-be denies for investigation.

  2. Diagnose: autonomy attestation status per affected node; confirm enrollment + rollout state match expectations. For targeted WAL inspection, use the layer-filtered grep recipes in the next section.

  3. Re-test: re-run autonomy attestation eval --bundle <new> against each node before flipping enforce again.

Diagnose with layer-filtered WAL grep

When a deny surfaces in production, the WAL is the canonical answer to which layer blocked and what the reason said. Every autonomy.decision frame written by the runtime carries the layer prefix in the reason field (attestation:, runtime allowlist:, or — for plain policy denies — the Rego evaluator’s text without a layer prefix). The shape is stable, CI-pinned, and grep-friendly. See Security Model — Layered governance for the contract.

The recipes below apply to any layer, not just attestation; this section lives here because attestation denies are the most common operator-facing surface today.

Recipe 1 — recent denies across all layers

autonomy wal inspect --kind autonomy.decision --since 1h --json \
  | jq -c 'select(.event.attrs.outcome == "deny")
           | {seq, ts: .written_at, reason: .event.attrs.reason,
              audit_id: .event.attrs.audit_id, tool: .event.attrs.tool}'

Shows every deny in the last hour with seq, timestamp, layer-tagged reason, audit_id, and tool kind — typically the right starting point for an unexpected-deny investigation.

Recipe 2 — attestation denies only

autonomy wal inspect --kind autonomy.decision --json \
  | jq -c 'select(.event.attrs.reason | startswith("attestation:"))'

Useful when narrowing to the gate after autonomy attestation status flags a node. Sub-check vocabulary today: enrollment_mismatch, enrollment_revoked, rollout_state_invalid, window_expired, window_not_yet_active, source_unavailable. Each has a different operator remediation — see the Security Model attestation section for the per-subreason fix paths.

Recipe 3 — runtime allowlist denies only

autonomy wal inspect --kind autonomy.decision --json \
  | jq -c 'select(.event.attrs.reason | startswith("runtime allowlist:"))'

These come from tool.http_get calls against endpoints not present in ServerOptions.AllowedDomains. The dynamic tail (the endpoint key) follows the layer prefix; remediation is to either add the endpoint to the allowlist (with operator review) or to fix the agent to stop calling it.

Recipe 4 — reconstruct the layered-governance chain for one audit_id

autonomy wal inspect --kind autonomy.decision --json \
  | jq -c 'select(.event.attrs.audit_id == "<id>")'

Returns every WAL frame written under one audit_id, ordered by seq. When the policy layer allowed and a later layer denied, both frames surface — the chain tells you exactly which layer ended the request and what the earlier layers had decided. This is the canonical procedure for answering “why did this specific call get blocked?” — strictly stronger than the single-row GET /v1/audit/{id} endpoint, which returns only the canonical last frame.

Recipe 5 — count denies by layer over a window

autonomy wal inspect --kind autonomy.decision --since 24h --json \
  | jq -r 'select(.event.attrs.outcome == "deny")
           | .event.attrs.reason
           | if   startswith("runtime allowlist:") then "runtime allowlist"
             elif startswith("attestation:")      then "attestation"
             else                                      "policy (no prefix)"
             end' \
  | sort | uniq -c | sort -rn

Useful for triage when you suspect drift — a sudden spike in attestation vs. baseline tells you whether the issue is enrollment state, bundle provenance, or windowing.

Why explicit prefixes, not a <layer>: regex. Plain policy-layer denies surface the Rego evaluator’s text without a layer prefix (see Security Model contract) — so the recipe has to bucket those rows somewhere. But a regex like ^[a-z]+( [a-z]+)*: would also false-positive on operator-authored Rego text that happens to start with lowercase words and a colon ("policy denied by rule: agents-may-not-shell"), inventing fake layer buckets. The explicit startswith ladder pins the rollup to the documented post-policy layers (runtime allowlist, attestation) and routes everything else — including any plain Rego deny text, regardless of shape — into policy (no prefix). When a new enforcement layer lands its exported Reason* constant, add an elif line above the else and the rollup picks it up; the Security Model layer table is the canonical list to keep in sync.

Operator escalation

If the layer prefix on the deny is one your runbook doesn’t cover, the constant taxonomy is the source of truth:

  • runtime/tools.go — runtime allowlist constants

  • runtime/attestation/types.go — attestation constants

Any new layer that lands a Reason* constant must match the <layer>: <subreason> shape (CI-enforced by runtime/reasons_shape_test.go), so a deny prefix that grep finds in production is guaranteed to map to a documented constant.

Optional: time-bounded authorization (ExecutionWindow)

Bundles signed at schema v1.3 can declare an ExecutionWindow block in the manifest:

{
  "schema_version": "1.3",
  "execution_window": {
    "not_before": "2026-05-15T00:00:00Z",
    "not_after":  "2026-08-15T00:00:00Z"
  }
}

The gate evaluates not_before and not_after against the runtime’s wall clock (time.Now) on every decision once enforce is active. Operational notes:

  • The window is half-open [NotBefore, NotAfter). A bundle whose not_after is 2026-08-15T00:00:00Z denies at exactly that instant — no one-tick grace.

  • Both endpoints are independently optional. A bundle with only not_after runs from any starting point up to the deadline (cert-style expiry). A bundle with only not_before runs from that moment forever (scheduled release).

  • An expired bundle surfaces attestation: window_expired on the wire and in the WAL; a future-dated bundle surfaces attestation: window_not_yet_active. Operator remediation is to ship a new (or renewed) bundle — different from the enrollment- side remediations.

  • Build-time validation rejects malformed RFC3339 timestamps and inverted / zero-length windows, so a bundle that passes bundle inspect won’t surface as a runtime-only window failure.

  • Pre-promotion check: autonomy attestation eval --bundle <path> reports the verdict against the local wall clock; for windows starting in the future the dry-run shows window_not_yet_active until activation day.

References