autonomy policy verify cases

Run fixed (scene → expected) cases, or replay recorded scenes (–from-wal) against a bundle

Synopsis

Two modes, both through the SAME runtime engine as the edge:

FIXTURE CASES (positional <case.json|dir>): asserts each case's verdict equals its
"expected" (the DESIRED verdict) — any mismatch fails the suite (nonzero exit),
gating CI. Mind the polarity: an exported failure (from "run --export-failures")
froze the OBSERVED verdict, which for a violation is the wrong answer — do NOT drop
raw exported failures in unchanged. For the "did my fix eliminate the failure?"
loop use "policy verify replay --fail-on-reproduced".

RECORDED-SCENE REPLAY (--from-wal <wal-dir> --action-kind <kind> [--source X]):
re-evaluates the scenes the fleet actually observed (autonomy.scene_state frames)
and reports the outcome distribution — "how would this bundle decide on the scenes
we actually saw?". No expected answer (scenes are observations); gate with
--fail-on-outcome DENY[,DEFER] (nonzero exit if any observed scene decides that
way) and/or --invariants. Caveat: the scene channel is not tied to a decision, so
--action-kind is operator-supplied and params-driven rules aren't exercised (that
is faithful full-action replay, #1313).

  autonomy policy verify cases --bundle bundle.tar.gz ./regression-suite
  autonomy policy verify cases --bundle bundle.tar.gz --from-wal ./wal --action-kind tool.ros2.topic.publish --fail-on-outcome DENY

Usage

autonomy policy verify cases --bundle <dir|.tar.gz> {<case.json|dir> | --from-wal <wal-dir> --action-kind <kind>} [flags]

Options

      --action-kind string           action.kind to replay observed scenes under (required with --from-wal)
      --bundle string                path to the policy bundle directory or .tar.gz (required)
      --deployment-manifest string   signed deployment manifest whose execution_decisions vocabulary resolves graded verdicts, so a case can assert expected.verdict and --from-wal invariants can use expect_verdict_in (#1369)
      --fail-on-outcome string       comma-separated outcomes (ALLOW,DEFER,DENY) that fail the run if any observed scene decides that way (--from-wal)
      --format string                output format: "terminal" or "json" (JSON is the default when --output is set) (default "terminal")
      --from-wal string              replay recorded scene-state frames from this telemetry WAL directory instead of fixture cases (#1312)
      --invariants string            invariants YAML to check each observed decision against (--from-wal)
      --output string                write the report to this file (default: stdout)
      --source string                restrict --from-wal replay to one scene source

Options inherited from parent commands

      --require-signed-manifest   Refuse to start if the SHA256SUMS release manifest or its cosign signature is missing/invalid (Tier 2 binary-integrity check; default: build-time compile-in via -ldflags -X, build-hardened artifacts=true, all others=false)

See also