Operator Runbook

Note

In-repo only. This page documents the failure-injection harness debug flow and the proof-summary interpretation procedure for the in-repo FI suite — it assumes a git clone checkout with make fi, make fi-root, and the /tmp/edge-fi-report/ outputs. Operators following the install flow (curl … install.sh | bash) should follow the Demo Runbook for the operator-facing supply-chain demo and failure drills that DO work from the install path.

What you’re proving

  • You can classify failures by layer (toolchain, environment, capability tests, FI).

  • You can map proof summaries to actionable follow-ups.

  • You can interpret “pass/fail/not run” without over-claiming capability coverage.

Prereqs

  • Repo root: <repo-root>

  • Latest generated captures under docs/_generated/test-outputs/

Steps

  1. Regenerate evidence artifacts.

GOCACHE=/tmp/go-build FI_TARGET= make docs-gen

  1. Inspect proof-oriented outputs.

sed -n '1,200p' docs/_generated/test-outputs/demo-output.txt
sed -n '1,220p' docs/_generated/test-outputs/fi-traceability-output.txt

  1. Build docs strictly before release.

GOCACHE=/tmp/go-build FI_TARGET= make docs

Expected outputs (from real run)

VALIDATION PASS (checks=32, assurance=full (cgroup v2 ok), ...)
...
transport socket tests unavailable in this environment; running deterministic transport unit subset
...

=== Edge FI Test Summary ===
Pass:    2
Fail:    0
Not run: 1
...
- Overall: **PASS**

Verification

  • make docs-gen exits 0 and refreshes generated outputs.

  • make docs exits 0 (sphinx-build -W).

  • docs/_generated/tbd-report.md remains empty of unresolved items.

Common failure patterns and debug actions

  • Go cache permission denied:

    • Symptom: open /home/ubuntu/.cache/go-build/... permission denied

    • Action: set GOCACHE=/tmp/go-build.

  • Wrong Go toolchain in shell FI scripts:

    • Symptom: invalid go version '1.23.0' / unknown directive: toolchain

    • Action: prepend pinned Go bindir to PATH.

  • Socket-restricted environment:

    • Symptom: transport tests report socket: operation not permitted

    • Action: rely on deterministic transport unit subset and document fallback evidence.

  • Integration prereq unavailable (registry/cosign):

    • Symptom: integrity tamper tests show SKIP with registry/cosign guidance.

    • Action: start prereqs (make demo-up / make demo-registry-test) before rerun.

Interpreting proof summaries

  • PASS: all executed checks passed.

  • NOT RUN: intentionally excluded (e.g., root-required FI), not equivalent to failure.

  • SKIP: prereq-gated test acknowledged by framework; evaluate if prereqs are required for your release gate.

Non-goals

  • This runbook does not authorize remote execution from control plane.

  • This runbook does not imply orchestration, leader election, or convergence.

Evidence

  • docs/_generated/test-outputs/demo-output.txt

  • docs/_generated/test-outputs/fi-traceability-output.txt

  • docs/_generated/test-outputs/policy-deny-output.txt

  • docs/_generated/test-outputs/offline-drain-output.txt

  • docs/_generated/tbd-report.md

  • Makefile (docs-gen, docs)