AutonomyOps ADK — Tutorial Pack¶

Story: Seed once, update everywhere. A node receives a signed, policy-verified update in an airgapped environment. It propagates that update to peers via deterministic outbound relay, survives a full OS replacement, reconstructs its mission runtime state from persistent storage, and continues policy-governed operation — without ever needing a network connection to a central authority.

What You Will Learn¶

Tutorial	Capability
01 — Single Node: Receive, Verify, Activate	Build a policy bundle, attach it to an OCI artifact, sign the supply chain, load it into the autonomy runtime, and observe policy enforcement at the tool boundary.
02 — Multi-Node: Seed Once, Update Everywhere	Publish a desired-state release to the control plane; watch multiple nodes poll, detect the candidate, and emit verifiable lifecycle telemetry.
03 — Crash and Recovery: WAL + Safe-Point	Kill the runtime mid-operation; restart it; demonstrate that the Write-Ahead Log survives and drains in priority order with no event loss.
04 — OS Replacement Survival and Runtime Reconstruction	Simulate a kernel/OS update via the edge daemon’s precheck flow; force a fingerprint mismatch; observe `RunReconstruction` restore runtime state; verify `BootEpoch` advances.
05 — Portability and “Run Everywhere”	Current build targets, CGO status, container images, and a roadmap for RISC-V support.

Prerequisites¶

Requirement	Version	Notes
Docker + Compose Plugin	24+ / 2.20+	`docker compose version`
Go	1.23+	`go version` (workspace uses 1.25.7)
cosign	2.x	`cosign version` — required for signed-chain tutorials only
uv (Python)	0.4+	`uv --version` — required for Python agent demo
curl	any	HTTP demo calls
jq	any	JSON parsing in examples
Linux x86-64	kernel 5.15+	All demos tested on Ubuntu 22.04 LTS

Repository root:

export REPO=/home/ubuntu/vsc_workstation/autonomyops  # adjust to your checkout
cd $REPO

Build the CLI once (used in every tutorial):

make build
export PATH=$REPO/bin:$PATH
autonomy --help      # verify it's on PATH

Quickstart Path (Recommended Order)¶

Tutorial 01  →  Tutorial 02  →  Tutorial 03  →  Tutorial 04
(supply chain)   (fleet update)   (crash recovery)   (OS survival)

Tutorial 05 (portability) is standalone and can be read at any time.

For a 15-minute “show me it works” run:

make build
make demo-up           # start the full Docker Compose stack
make demo-run          # Tutorial 01 end-to-end (requires cosign)
#   or:
make demo-run-unsigned # Tutorial 01 without cosign requirement
make demo-offline-drain  # Tutorial 03 offline + drain
make demo-down

Evidence Map¶

Every major product claim is backed by a code path, test, or script:

Claim	Evidence
Activation is atomic	`os.Rename` in `executeCopyBinary` (`edge/bootstrap/install.go:123-130`)
Policy deny cannot be overridden	`TestInterceptorDenyCannotBeOverridden` (`runtime/interceptor_test.go`)
Fail-closed: no policy → deny all	`TestInterceptorFailClosed` (`runtime/interceptor_test.go`); `denyAllEvaluator{}` in `cmd/autonomy/commands/runtime.go`
WAL survives collector outage	`TestWALSurvivesCollectorDown` (`telemetry/buffer_test.go`); Drill 3 in `demo/scripts/05_failure_drills.sh`
WAL is fail-hard on corruption	`causeSeqGap`, `causeSafeSeqNotFound` constants; `OpenWAL` in `telemetry/wal.go`
OS fingerprint is BLAKE3-deterministic	`TestFingerprintChange_KernelOnly_TriggersReconstruction` (`edge/bootstrap/bootstrap_test.go`)
Kernel-only change triggers reconstruction	`CompositeHash` includes `uname().Release`; GAP-B4 comment in `edge/bootstrap/osfingerprint.go`
Reconstruction is TOCTOU-safe	`RunReconstruction` executes the caller-verified bytes without re-reading disk; GAP-B1 comment in `edge/bootstrap/bootstrap.go:143`
Relay uses atomic file writes	`os.WriteFile` + `os.Rename` in `executeCopyBinary` (`edge/bootstrap/install.go:119-130`)
Relay bounded retry / dead-letter	`TestFailRelay_Deadletter` (`edge/relay/executor_test.go:464`); INV-12
Supply-chain tamper detected	Drill 4, `demo/scripts/05_failure_drills.sh`; `TestVerify_TamperedAgentDigest` (`oci/sign/verify_tamper_test.go`)
Release polling emits lifecycle events	Phases `polled → candidate_detected → verify_*`; `runtime/poller.go:142-234`

Glossary¶

Term	Definition
Segment	Atomic unit of content in the edge relay. Identified by a content-addressed `SegmentID`. Stored in the local segment store under `{state_root}/edge/segments/`.
Activation Lock (`autonomy.lock.json`)	A signed, content-addressed JSON artifact binding an agent image, policy bundle, and behavioral fingerprint. Hash algorithm: BLAKE3. Stored as an OCI sidecar referrer.
Behavioral Fingerprint	BLAKE3 hash of the canonical (alphabetically sorted) JSON representation of the lock. CI gate: `make check-golden`. Computed by `lock.ComputeFingerprint()`.
Policy Bundle	A `.tar.gz` archive of Rego policy files + `manifest.json`. Loaded into the runtime via `autonomy policy load`. Evaluated per tool call.
WAL (Write-Ahead Log)	Length-prefixed JSONL file (`telemetry.wal`) written before any OTLP export attempt. Guarantees events are not lost if the collector is unreachable.
Safe-Point	An 8-byte little-endian uint64 in `telemetry.safe_seq` recording the last WAL sequence confirmed durable. Recovery truncates to this point on restart.
Desired-State Release	A control-plane record (`POST /v1/releases`) containing a channel, `target_lock_fingerprint`, `artifact_ref`, and `policy_ref`. Advisory only — does not change runtime policy directly.
Release Polling	The `runtime.Poller` background goroutine that calls `GET /v1/releases/latest?channel=stable` at a configurable interval and emits lifecycle telemetry.
Precheck	The `edged precheck` subcommand that runs before `edged` starts: validates config, checks cert expiry, detects OS fingerprint changes, and optionally runs `RunReconstruction`.
State Root	The persistent partition mounted at `{state_root}`. Survives OS replacement. Contains segments, relay ledger, identity certs, epoch state, and bootstrap configuration.
Boot Epoch	A monotonic uint64 counter in `{state_root}/bootstrap/os_fingerprint.json`. Incremented by `edged precheck` after each successful runtime reconstruction.
BootEpoch Evidence	A file written to `{state_root}/epoch/current/evidence.json` after each reconstruction, containing the epoch number, previous/new fingerprint, timestamp, and an optional binary BLAKE3 hash.
LKG (Last-Known-Good)	The policy slot promoted when `LoadBundle` succeeds. Preserved if the new bundle fails the compatibility check.
mTLS	Mutual TLS used by `edged` for all outbound connections to peers (INV-09). Cert expiry is pre-checked (exit 3 if within 7 days).
Success Condition	The relay delivery guarantee: `one_peer` (at least one ACK) or `all_peers` (all configured peers ACK). Configured in `edge.toml` under `relay.success_condition`.

Repository Structure (Key Paths)¶

autonomyops/
├── cmd/autonomy/       ← CLI entry point (all subcommands)
├── orchestrator/       ← Event ingestion API + HA pgstore backend + rollout APIs
├── demo/               ← Docker Compose stack + demo scripts
│   ├── docker-compose.yml
│   ├── locks/          ← example.lock.json (template)
│   ├── policies/       ← echo_allow.rego, shell_deny.rego
│   ├── scripts/        ← 01_build.sh … 07_poll_loop.sh + lib.sh
│   └── keys/           ← cosign.key / cosign.pub (demo use only)
├── edge/               ← edged daemon + edgectl control tool
│   ├── bootstrap/      ← OS fingerprinting + RunReconstruction
│   ├── config/         ← TOML config loader + validation
│   ├── relay/          ← BoltDB ledger + executor
│   ├── storage/        ← Disk-ceiling enforcement + segment store
│   └── stateroot/      ← Persistent-partition path helpers
├── lock/               ← Lock schema + BLAKE3 fingerprint + canonicalization
├── oci/                ← OCI sidecar attach/pull + referrers probe
├── policy/             ← Bundle builder + manager (current/LKG slots)
├── runtime/            ← ToolServer + Interceptor + Poller
├── telemetry/          ← WAL + OTLP sender + bridge
└── Makefile

Supporting Documents¶

Document	Purpose
Story Script	90–150 second live demo narration with exact output markers
Repo Findings Checklist	Every product claim mapped to file + symbol; known gaps documented
Orchestrator HA Runbook	Replicated datastore operations, leader epochs, health/readiness, and failover diagnostics
Edge Relay Deadletter Lab	Reproducible local lab for `edgectl relay deadletter` workflows, `relay status`, and bandwidth verification, with checked-in evidence bundles
CLI Audit Lab	Reproducible local lab for PR-17 CLI audit emission, PR-18 support-bundle verification, PR-19 audit query/export verification, PR-20 RBAC role CLI verification, PR-21 RBAC enforcement verification, PR-22 metrics list/query verification, and PR-27 config migration verification across rollout, HA, cert, auth, relay, metrics, and config commands