VAL28 — Cross-Cutting Proof Report Generator

Audience: engineering leads, product managers, and external reviewers who need a consolidated, evidence-backed assessment of the AutonomyOps ADK security posture, observability pipeline, and operational diagnostics.

VAL28 is a report generator, not a test runner. It reads evidence from six completed cross-cutting validation slices (VAL01–VAL06) produced by a single run_cli_audit_lab.sh run and produces a two-part report: an executive summary followed by a detailed appendix covering the Security, Observability, and Operations surfaces.

1. Scope

VAL28 consolidates evidence from:

Slice

Name

What it proves

VAL01

Zero-Downtime Cert Rotation

Near-expiry detection, mTLS continuity during rotation, timing bound ≤ 300 s, audit capture

VAL02

Trust-Chain Rejection

mTLS rejection of missing cert, invalid chain, expired cert, revoked cert, wrong server trust

VAL03

RBAC Permission Enforcement

5 DENY, 5 ALLOW, 3 NOT_GUARDED, 1 AUDIT check; 14-permission matrix

VAL04

Audit Completeness

25 wired event types present, 6 categories validated, schema per-category, query latency ≤ 2,000 ms

VAL05

OTel Integration

Prometheus /metrics, telemetry WAL pipeline, OTLP flush, trace ID propagation (JSONL + OTLP)

VAL06

Support Bundle

Archive creation, timing ≤ 30 s, 6 collectors, secrets redacted, no private keys, degraded mode

Branch rule: coverage by existing runner

Existing asset

Coverage

run_cli_audit_lab.sh

Runs VAL01–VAL06 as embedded slices and produces per-slice reports — no cross-slice aggregation

run_ha_proof_report_val26.sh

HA slices (VAL13–VAL17) only

run_fleet_rollout_proof_report_val25.sh

Fleet rollout slices (VAL07–VAL11) only

run_relay_proof_report_val27.sh

Relay slices (VAL19–VAL23) only

New aggregator required. No existing script combines VAL01–VAL06 into a single cross-cutting proof artifact with an executive summary and design-partner readiness conclusion.

Note: mixed report formats

VAL01 and VAL02 produce text-only reports (cert-rotation-val01-report.txt, cert-rejection-val02-report.txt). VAL28 parses pass/fail results via regex (VAL0N-N PASS|FAIL pattern) and extracts the rotation timing bound from the VAL01 text. VAL03–VAL06 produce JSON reports that VAL28 reads directly.

Out of scope

  • PostgreSQL-backed audit store query performance (VAL04 uses SQLite)

  • Production OTel collector validation (VAL05 uses a local test sink on :14318)

  • VAL03 full 14-check coverage when HA server is unavailable (SKIP ≠ FAIL)

  • External security audit of cert management and RBAC enforcement surfaces

  • Penetration testing of mTLS trust-chain boundaries

  • Compliance audit of audit completeness against applicable standards

2. Evidence Structure

VAL28 reads from the evidence directory produced by run_cli_audit_lab.sh (default: evidence/cli-audit-lab-YYYY-MM-DD).

Input file

Produced by

Format

Contents

cert-rotation-val01-report.txt

run_cert_lab() Phase 8

Text

6 check outcomes, rotation elapsed time

cert-rejection-val02-report.txt

run_cert_lab() Phase 9

Text

5 check outcomes

val03/val03-report.json

run_rbac_val03_lab()

JSON

pass_count, skip_count, fail_count, checks dict

val04/val04-report.json

run_audit_completeness_val04_lab()

JSON

pass_count, fail_count, latency_ms, coverage_found/expected, checks dict

val05/val05-report.json

run_otel_val05_lab()

JSON

pass_count, fail_count, checks dict (VAL05-01..09)

val06/val06-report.json

run_support_bundle_val06_lab()

JSON

pass_count, fail_count, elapsed_s, checks dict

VAL28 expects all six slices to come from one coherent run_cli_audit_lab.sh run. It checks the spread of timestamps (JSON timestamp fields and text file mtimes) and requires all found evidence to fall within a single 6-hour evidence window before issuing a Design Partner readiness conclusion.

Missing slices are reported as MISSING in the coverage table without aborting. JSON reports that fail schema validation are degraded to MISSING with a schema-mismatch detail.

3. Metric Definitions and Targets

Certificate Rotation (VAL01)

Metric

Target

Source

Rotation elapsed time

≤ 300 s

VAL01 / workplan cert-rotation target

Expiry detection

Near-expiry cert appears in cert list

VAL01-1

Pre-rotation mTLS

Old cert accepted before rotation

VAL01-2

Rotation timing bound

Cert issuance completes ≤ 300 s

VAL01-3

Expiry window cleared

New cert no longer in 5-day expiry window

VAL01-4

Post-rotation mTLS

New cert accepted without CP restart

VAL01-5

Audit captured

cert.rotated event retained in audit store

VAL01-6

Note: Rotation is sub-second in the local lab. 300 s is a conservative SLA floor for production deployments with slower CA issuance paths.

Trust-Chain Rejection (VAL02)

Rejection gate

Check

Missing client cert

CLI exits non-zero; stderr contains “certificate required”

Invalid chain (rogue CA)

Rejected; stderr contains “unknown authority”

Expired certificate

Rejected; stderr contains “expired”

Revoked certificate

CRL VerifyPeerCertificate callback rejects the cert

Wrong server trust

Client cannot verify server certificate with wrong CA bundle

Note: A cert from a trusted CA with an unexpected CN is accepted by design. Identity authentication is RBAC-based (actor string), not CN-based.

RBAC Enforcement (VAL03)

Check category

Count

Examples

DENY (blocked as expected)

5

unassigned/analyst blocked from fleet:read, cert:manage, rbac:manage

ALLOW (permitted as expected)

5

operator/analyst/auditor permitted for appropriate commands

NOT_GUARDED (no RBAC gate)

3

rbac role list, rollout plan list, support-bundle generate

AUDIT (denied events captured)

1

auth.access.denied events present in audit store

VAL03 checks can be SKIP (not FAIL) when the HA server is unavailable. SKIP checks are excluded from FAIL count. Full 14-check coverage requires a running HA server.

Audit Completeness (VAL04)

Metric

Target

Wired event types present

25 / 25

Categories populated

All 6: rollout, ha, cert, relay, auth, rollback

Schema fields validated per event

event_name, category, action, outcome, source, timestamp

Query latency

≤ 2,000 ms

25 wired event types covers: rollout (3), ha (8), cert (5), relay (3), auth (4), rollback (2). 6 deferred events are intentionally excluded from the 25-event target.

OTel Integration (VAL05)

Sub-check

Target

Prometheus /metrics

4 families, non-zero observations

Telemetry WAL populated

JSONL export non-empty, fields valid

OTLP flush to test sink

flush_ok=true, payloads > 0 at 127.0.0.1:14318

trace_id in JSONL export

Propagated in telemetry WAL export

traceId in OTLP log records

Correlation IDs present at OTLP sink

Support Bundle (VAL06)

Metric

Target

Bundle generation time

≤ 30 s

Core files present

manifest.json, system_info.json, build_info.json

Collectors recorded

6: system_info, build_info, config, ha_status, audit_recent, logs

fleet_salt redacted

<REDACTED> in bundle output

postgres password redacted

REDACTED in bundle output

No private keys

Zero PEM blocks in archive

Degraded mode

Bundle exits 0; manifest records status: failed for failing collectors

4. Readiness Level Definitions

Cross-Cutting Design Partner Ready

Criteria (all must hold):

  1. VAL01 all 6 checks pass (cert rotation text report found and parsed)

  2. VAL02 all 5 checks pass (trust-chain rejection text report found and parsed)

  3. VAL03 all non-SKIP checks pass (fail_count == 0); at least 1 DENY + 1 ALLOW confirmed

  4. VAL04 all 10 checks pass; coverage_found = coverage_expected = 25; latency_ms ≤ 2,000

  5. VAL05 all 9 checks pass; trace correlation IDs confirmed in JSONL + OTLP

  6. VAL06 all 10 checks pass; timing ≤ 30 s; secrets redacted

  7. Cert rotation timing ≤ 300 s (VAL01-3)

  8. Audit event coverage ≥ 25 wired types (VAL04)

  9. OTel trace correlation IDs propagated: VAL05-08 + VAL05-09 both pass

  10. Evidence timestamps fall within a single 6-hour evidence window

Cross-Cutting GA Ready

Not evaluated by VAL28. The report always emits NOT EVALUATED IN VAL28 SCOPE for GA because VAL28 aggregates only VAL01–VAL06 and does not ingest the additional evidence required for a GA claim. Those additional requirements are:

  1. PostgreSQL-backed audit store query performance validation under load

  2. OTel pipeline validated against a production-grade OTLP collector (Grafana, Datadog, etc.)

  3. VAL03 full 14-check coverage with HA server (no SKIP checks)

  4. Redaction coverage independently audited against production secret inventory

Cross-Cutting Public Production Claim

Not evaluated by VAL28. The report always emits NOT EVALUATED IN VAL28 SCOPE for Public Production because the generator does not ingest the external evidence needed for that claim. Required evidence is GA criteria PLUS:

  1. External security audit of cert management and RBAC enforcement surfaces

  2. Penetration testing of mTLS trust-chain boundaries

  3. Compliance audit of audit event completeness against applicable standards (SOC 2, etc.)

5. 10-Check Matrix

ID

When

Description

Pass criterion

VAL28-01

Setup

VAL01 cert rotation: all 6 checks pass

s01 == "PASS" (6/6 text checks)

VAL28-02

Setup

VAL02 trust-chain rejection: all 5 checks pass

s02 == "PASS" (5/5 text checks)

VAL28-03

Setup

VAL03 RBAC enforcement: full 14-check accounting, no FAIL, SKIP acceptable when HA unavailable

fc03 == 0, pc03 + sc03 == total == 14, and at least 1 DENY + 1 ALLOW confirmed

VAL28-04

Setup

VAL04 audit completeness: all 10 checks pass

s04 == "PASS" (10/10 checks)

VAL28-05

Setup

VAL05 OTel integration: all 9 checks pass

s05 == "PASS" (9/9 checks)

VAL28-06

Setup

VAL06 support bundle: all 10 checks pass

s06 == "PASS" (10/10 checks)

VAL28-07

Metric

Cert rotation timing bound: elapsed ≤ 300 s

elapsed01_s <= 300

VAL28-08

Metric

Audit event coverage: 25/25 wired event types present

coverage_found04 == coverage_expected04 == 25

VAL28-09

Metric

OTel trace correlation IDs propagate: JSONL + OTLP both confirmed

VAL05-08 PASS AND VAL05-09 PASS

VAL28-10

Summary

Cross-cutting design partner readiness — all above pass and evidence coherent

VAL28-01..09 all PASS + 6-hour window

6. Run the Report

Prerequisites

Run the full cli-audit-lab to produce all six slices:

export GOROOT=/home/ubuntu/.local/go1.25.7
export PATH="$GOROOT/bin:$PATH"
export GOTOOLCHAIN=local

bash scripts/labs/run_cli_audit_lab.sh

The evidence directory is printed at the end: evidence/cli-audit-lab-YYYY-MM-DD.

Generate the proof report

bash scripts/labs/run_crosscut_proof_report_val28.sh \
  evidence/cli-audit-lab-2026-03-23

Output files

File

Contents

stdout

Executive summary + detailed appendix

val28/val28-proof-report.txt

Same content as stdout

val28/val28-proof-report.json

Machine-readable JSON with slices, checks, readiness

7. Final Report Format

VAL28 — Cross-Cutting Proof Report
Generated:    <YYYY-MM-DDTHH:MM:SSZ>
Evidence dir: <path>
Evidence:     timestamp spread=<N>s

========================================================================
EXECUTIVE SUMMARY
========================================================================

This report covers six cross-cutting validation slices that prove the
security posture, observability pipeline, and operational diagnostics of
the AutonomyOps ADK control plane.  ...

Slice Results:
  VAL01  Zero-Downtime Cert Rotation    PASS     (6/6 checks)
  VAL02  Trust-Chain Rejection          PASS     (5/5 checks)
  VAL03  RBAC Enforcement              PASS     (14 pass, 0 skip, 0 fail)
  VAL04  Audit Completeness             PASS     (10/10 checks)
  VAL05  OTel Integration               PASS     (9/9 checks)
  VAL06  Support Bundle                 PASS     (10/10 checks)

Key metrics:
  Cert rotation timing:   42 s   [target <= 300 s]   PASS
  Audit query latency:    18 ms  [target <= 2,000 ms]   PASS
  Audit event coverage:   25/25 wired types   PASS
  Bundle generation:      3 s   [target <= 30 s]   PASS

Overall: PASS=10/10  FAIL=0/10
Verdict: CROSS-CUTTING DESIGN PARTNER READY

GA / Public Production:
  NOT EVALUATED IN VAL28 SCOPE

========================================================================
DETAILED APPENDIX
========================================================================

── Security Surface ──────────────────────────────────────────────────

A. Certificate Rotation  (VAL01)

   Status:   PASS  (6/6 checks)
   Rotation elapsed:  42 s   [target <= 300 s]
   Checks validated:
     VAL01-1  Expiry detection: near-expiry cert appears in cert list output
     VAL01-2  Pre-rotation mTLS: old cert accepted over live mTLS connection
     VAL01-3  Rotation timing: cert issuance completes within 300 s bound
     VAL01-4  Expiry window cleared: new cert no longer in 5-day expiry window
     VAL01-5  Post-rotation mTLS: new cert accepted without CP restart
     VAL01-6  Audit captured: cert.rotated event in retained audit store
   Note: Rotation is sub-second in local lab; 300 s is a practical SLA floor.

B. Trust-Chain Rejection  (VAL02)

   Status:   PASS  (5/5 checks)
   Rejection gates confirmed:
     VAL02-1  Missing client cert: exit non-zero, stderr: 'certificate required'
     VAL02-2  Invalid chain (rogue CA): rejected, stderr: 'unknown authority'
     VAL02-3  Expired certificate: rejected, stderr: 'expired'
     VAL02-4  Revoked certificate: CRL VerifyPeerCertificate callback rejects
     VAL02-5  Wrong server trust: client cannot verify server with wrong CA
   Note: right_ca_wrong_cn (trusted CA, unexpected CN) is ACCEPTED by design —
         identity auth is RBAC-based (actor string), not CN-based.

C. RBAC Permission Enforcement  (VAL03)

   Status:   PASS  (pass=14, skip=0, fail=0 of 14 checks)
   DENY  checks:        5/5 pass  (unassigned/operator/analyst blocked)
   ALLOW checks:        5/5 pass  (operator/analyst/auditor permitted)
   NOT_GUARDED checks:  3/3 pass  (rbac_role_list, rollout_plan_list, support-bundle)
   AUDIT check:         PASS  (auth.access.denied events present)
   Roles tested: unassigned, operator, analyst, auditor
   Permissions tested: fleet:read, audit_history:read, rbac:manage (14 permissions total)

── Observability Surface ────────────────────────────────────────────

D. Audit Completeness  (VAL04)

   Status:   PASS  (10/10 checks)
   Event coverage:  25/25 wired event types   [target >= 25]
   Query latency:   18 ms   [target <= 2,000 ms]
   Categories populated: rollout(3), ha(8), cert(5), relay(3), auth(4), rollback(2)
   Schema validated: event_name, category, action, outcome, source, timestamp
   ...

E. OTel Integration  (VAL05)

   Status:   PASS  (9/9 checks)
   Prometheus endpoint:     PASS  (/metrics, 4 families, non-zero observations)
   Telemetry WAL pipeline:  PASS  (WAL populated, JSONL export non-empty, fields valid)
   OTLP flush to sink:      PASS  (127.0.0.1:14318, flush_ok=true, payloads > 0)
   Trace ID in JSONL:       PASS  (trace_id/span_id propagated in telemetry export)
   TraceId in OTLP:         PASS  (correlation IDs in OTLP log records at sink)
   ...

── Operations Surface ───────────────────────────────────────────────

F. Support Bundle  (VAL06)

   Status:   PASS  (10/10 checks)
   Generation time:  3 s   [target <= 30 s]   PASS
   Core files present:        PASS  (manifest.json, system_info.json, build_info.json)
   All 6 collectors recorded: PASS  (system_info, build_info, config, ha_status, audit_recent, logs)
   fleet_salt redacted:       PASS  (deadbeef… → <REDACTED>)
   postgres password redacted:PASS  (val06-secret-pass → REDACTED)
   No private keys in archive:PASS  (zero PEM blocks)
   Degraded mode (ha_status fails): PASS  (bundle exits 0, manifest records status:failed)

── 10-Check Matrix ──────────────────────────────────────────────────

  VAL28-01 PASS  VAL01 cert rotation: all 6 checks pass
           val01 6/6 checks pass
  VAL28-02 PASS  VAL02 trust-chain rejection: all 5 checks pass
           val02 5/5 checks pass
  VAL28-03 PASS  VAL03 RBAC enforcement: 14/14 checks pass (SKIP acceptable when HA unavailable)
           val03 pass=14 fail=0 skip=0
  ...

Overall: PASS=10/10  FAIL=0/10

── Known Limitations ────────────────────────────────────────────────

  Security scope:
    - VAL01/02 produce text reports only (no machine-readable JSON).
    - Cert rotation tested with SQLite-backed CP.
    - CRL is loaded at CP start; runtime revocation requires CP restart.
    - VAL03 HA-dependent checks are SKIPped when HA server is unavailable.

  Observability scope:
    - VAL04 measures audit query latency with SQLite backend.
    - VAL04 measures 25 wired event types; 6 deferred events excluded.
    - VAL05 Prometheus uses custom client_golang exporter (not OTel SDK).
    - VAL05 OTLP sink is a local test server (127.0.0.1:14318).

  Operations scope:
    - VAL06 generation timing is not representative of production deployments.
    - Bundle tested with synthetic secrets; production scanning not audited.

── Readiness Conclusion ─────────────────────────────────────────────

  CROSS-CUTTING DESIGN PARTNER READY  ✓
    All six cross-cutting slices passed.  Security, observability, and
    operations surfaces are validated for Design Partner deployment.

  CROSS-CUTTING GA READY  ✗  (NOT YET)
    ...

  PUBLIC PRODUCTION CLAIM  ✗  (NOT YET)
    ...

Verdict: CROSS-CUTTING DESIGN PARTNER READY

8. Tooling

File

Role

scripts/labs/run_crosscut_proof_report_val28.sh

VAL28 cross-cutting proof report generator

scripts/labs/run_cli_audit_lab.sh

Source of VAL01–VAL06 evidence

docs/tutorials/cert-rotation-validation.md

VAL01 formal plan

docs/tutorials/cert-rejection-validation.md

VAL02 formal plan

docs/tutorials/rbac-enforcement-validation.md

VAL03 formal plan

docs/tutorials/audit-completeness-validation.md

VAL04 formal plan

docs/tutorials/otel-integration-validation.md

VAL05 formal plan

docs/tutorials/support-bundle-validation.md

VAL06 formal plan

scripts/labs/run_fleet_rollout_proof_report_val25.sh

VAL25 fleet rollout proof report

scripts/labs/run_ha_proof_report_val26.sh

VAL26 HA proof report

scripts/labs/run_relay_proof_report_val27.sh

VAL27 relay proof report