HA Operations — Rollout Leader Election and Streaming Promoter¶

This document covers the high-availability (HA) deployment model for the control-plane rollout subsystem.

Note: this page describes rollout evaluator leadership in the rollout store path (orchestrator/rollout/*), which uses an advisory lease table. The replicated PostgreSQL control-plane backend and epoch-fenced write authority for the HA datastore path are documented in self-hosted/tutorials/orchestrator-ha-runbook.md (orchestrator/pgstore/*).

Overview¶

The rollout promotion evaluator runs on a single leader node at any time. Leadership is coordinated via a lease table (leader_lease) in the rollout store. HA semantics are advisory (spec Appendix D): no robot participates in leader election, and no quorum is required for robot safety.

Two promoter instances run on each control-plane node:

Promoter	Mode	Frequency
BatchPromoter	Tick-based	Every 10 seconds
StreamingPromoter	Event-driven	Per EventBus event

The BatchPromoter is the correctness fallback. If the StreamingPromoter misses events (leader failover, EventBus drops, process restart), the BatchPromoter catches up on its next tick.

Leader Lease Table¶

CREATE TABLE IF NOT EXISTS leader_lease (
    id          TEXT NOT NULL PRIMARY KEY DEFAULT 'promotion_leader',
    holder      TEXT NOT NULL,
    acquired_at TEXT NOT NULL,
    expires_at  TEXT NOT NULL
);

Single-row table keyed on 'promotion_leader'. At most one row exists.

Election Protocol¶

Campaign (acquire/renew)¶

INSERT INTO leader_lease (id, holder, acquired_at, expires_at)
VALUES ('promotion_leader', ?, ?, ?)
ON CONFLICT(id) DO UPDATE
SET holder = excluded.holder,
    acquired_at = excluded.acquired_at,
    expires_at = excluded.expires_at
WHERE leader_lease.holder = excluded.holder
   OR leader_lease.expires_at <= ?

A Campaign succeeds (RowsAffected > 0) when:

No lease row exists (first election), or
The caller already holds the lease (renewal), or
The existing lease has expired.

A Campaign fails (RowsAffected == 0) when another node holds an unexpired lease.

IsLeader (check)¶

Queries holder and expires_at from the lease row. Returns true when holder == self.holderID AND now < expires_at. Results are cached for 1 second to reduce DB queries under high call rates.

Resign (voluntary release)¶

Deletes the lease row where holder == self.holderID. After Resign, IsLeader returns false until a new Campaign succeeds.

Configuration¶

DBLeaderElectorConfig fields:

Field	Type	Default	Description
`DB`	`*sql.DB`	required	Shared SQLite database connection
`HolderID`	`string`	required	Node identity (hostname or UUID)
`TTL`	`time.Duration`	30s	Lease duration

Single-Node Deployment¶

For single-node deployments, use StaticLeaderElector — it always reports leadership without touching the database. This is the default until HA is explicitly enabled.

leader := cprollout.NewStaticLeaderElector()

Multi-Node Deployment¶

leader := cprollout.NewDBLeaderElector(cprollout.DBLeaderElectorConfig{
    DB:       db,
    HolderID: hostname,
    TTL:      30 * time.Second,
})

Each node calls Campaign(ctx, holderID) on startup. Only the winning node’s StreamingPromoter processes events; the losing node’s handleEvent returns immediately when IsLeader returns false.

Promotion Safety Under Leader Transitions¶

Monotonic promotion is guaranteed by the store’s PromoteStage query:

UPDATE stage_status SET phase = 'promoted' ... WHERE phase != 'promoted'

If two nodes briefly overlap (old leader’s last evaluation, new leader’s first), the WHERE clause prevents double-promotion. The promoted stage is never reverted.

Telemetry¶

Leader election does not emit its own telemetry events. The StreamingPromoter and BatchPromoter emit rollout lifecycle events (ai.rollout.*) only when they perform observable actions (stage promotion, rollback detection).

Failure Modes¶

Scenario	Behavior
Leader crashes without Resign	Lease expires after TTL; another node acquires on next Campaign
DB unavailable	`IsLeader` returns false (fail-closed); no promotions occur
Both promoters miss a promotion	Next BatchPromoter tick (≤10s) catches up
Split-brain (both nodes think leader)	Impossible with single SQLite DB; `PromoteStage` WHERE clause prevents double-promotion regardless