VAL23 — Relay Bandwidth Management Validation

Audience: engineers and reviewers who want a reproducible local lab proving the relay bandwidth controls (relay config set-bandwidth) against a live edged daemon, including rate-limit enforcement, daily-quota enforcement, counter accuracy, and hot-reload behavior.

1. Scope

VAL23 validates five operational goals:

  1. Unlimited baseline — with no bandwidth config, all segments deliver and relay status reports unlimited=true, with peer evidence matching the exact seeded segment ID set.

  2. Rate-limit enforcement — bytes_per_second throttles delivery to at most one segment per token-refill cycle; ThrottleCount accumulates, QuotaDropCount stays zero.

  3. Daily-quota enforcement — daily_quota_bytes caps total bytes delivered per 24-hour window; segments beyond the quota are rejected and eventually deadletter; QuotaDropCount accumulates; the terminal delivered IDs plus deadletter IDs reconcile exactly to the seeded set.

  4. Hot-reload — relay config set-bandwidth applied to a running daemon takes effect on the next executor cycle without a restart; applied=true is returned and subsequent peer evidence matches the exact seeded segment ID set.

  5. Daily quota reset — the 24-hour rolling window resets correctly, verified via the TestBandwidthLimiter_DailyQuota_ResetsAfter24h unit test with an injected clock (live 24h advance is not practical in a lab).

Branch rule: coverage by existing runner

The existing run_edge_deadletter_lab.sh --with-bandwidth mode covers relay config set-bandwidth command execution and the relay.bandwidth.configured audit event, but has three gaps that VAL23 addresses:

Gap

Existing lab

VAL23

Rate-only vs quota-only isolation

No — always sets both simultaneously

Yes — S-B (rate only), S-C (quota only)

ThrottleCount / QuotaDropCount counter accuracy

No

Yes — S-B verifies ThrottleCount >= 4, QuotaDropCount = 0; S-C verifies QuotaDropCount >= 4

Hot-reload (set-bandwidth mid-run)

No

Yes — S-D verifies applied=true + delivery resumes

New standalone runner is justified because:

  • The existing lab hardcodes a combined rate+quota scenario that cannot isolate which limit is responsible for throttling

  • Port isolation (19054–19055) prevents interference with VAL19–VAL22

No impairment proxy is required: bandwidth throttling is enforced inside edged before the outbound TCP dial, so a direct peer address is sufficient.

Out of scope

  • Bandwidth interaction with deadletter retry (covered by PR-16 evidence)

  • Per-peer bandwidth configuration (not implemented as of VAL23)

  • Quota persistence across edged restarts (quota window is in-memory; restart resets daily usage — documented behavior)

  • Live 24-hour quota reset window (covered by unit test in S-E)

2. Architecture

edged (relay executor, bandwidth limiter)
  → edge_deadletter_lab_peer:19055  ← mTLS receive + JSON evidence
edged listen addr: 127.0.0.1:19054
edged control socket: $WORKDIR/ctl.sock

No impairment proxy. Bandwidth throttling is an in-process gate that runs before transport.Dial().

Port assignments (isolated from VAL19–VAL22)

Component

Address

edged

127.0.0.1:19054

peer server

127.0.0.1:19055

3. Scenario Matrix

Scenario

Bandwidth config

Segments seeded

Expected delivery

Key counters

S-A Unlimited baseline

none

4 × 64B

4/4

unlimited=true

S-B Rate only

bytes_per_second=64, no quota

5 × 64B

5/5 (throttled then delivered)

throttle_count >= 4, quota_drop_count = 0

S-C Quota only

daily_quota=128, no rate

6 × 64B

2/6 (128B / 64B = 2)

quota_drop_count >= 4, daily_used_bytes <= 128

S-D Hot-reload

start: bytes_per_second=8 → live update to unlimited

6 × 64B

6/6 (blocked then released)

applied=true

S-E Daily reset

unit test (injected clock)

—

PASS

—

Segment naming

Scenario

Segment IDs

S-A

val23-sa-001..004

S-B

val23-sb-001..005

S-C

val23-sc-001..006

S-D

val23-sd-001..006

All segments are 64 bytes and target peer-val23.

4. Token-Bucket Behaviour (S-B Timing Reference)

With bytes_per_second=64 and segment size 64B:

Time

Tokens

Event

t=0

64

First scheduler tick. Segment 1: Allow(64) → OK (tokens → 0). Segments 2–5: Allow(64) → FAIL (throttle_count += 4).

t=1s

64

Segment 2 retry (backoff=1s). Allow(64) → OK. Segments 3–5 retry: FAIL (throttle_count += 3).

t=2s

64

Segment 3: OK. Segments 4–5: FAIL.

t=3s

64

Segment 4: OK. Segment 5: FAIL.

t=4s

64

Segment 5: OK.

Total: throttle_count >= 4; all 5 delivered by t=5s. Runner waits 20s.

5. Daily-Quota Drop Path (S-C)

With daily_quota=128 and segment size 64B:

  1. Segment 1: daily_used + 64 <= 128 → OK. daily_used = 64.

  2. Segment 2: 64 + 64 <= 128 → OK. daily_used = 128.

  3. Segments 3–6: 128 + 64 > 128 → FAIL. quota_drop_count += 1 each attempt.

Each quota-rejected segment records a failed attempt and backs off per the retry schedule. With max_retry_count=3 and backoff_base_seconds=1:

  • Attempt 1: fail (quota drop)

  • Backoff 1s → attempt 2: fail (quota drop)

  • Backoff 2s → attempt 3: fail (quota drop) → deadletter

Total time to deadletter all 4 over-quota segments: ~4–7s. Runner waits 20s.

6. Hot-Reload Path (S-D)

With bytes_per_second=8 and segment size 64B (initial tokens=8 < 64):

  1. t=1s: attempt 1, all 6 fail (tokens=8 < 64), throttle_count += 6.

  2. t=2s: attempt 2 (backoff=1s), tokens=16 < 64, fail again.

  3. t=3s: unlimited config applied (bytes_per_second=0, daily_quota=0).

  4. t=4s: attempt 3 (backoff=2s from t=2) fires. Bandwidth is now unlimited → Allow(64) returns true → all 6 segments deliver.

max_retry_count=5 ensures segments are not deadlettered before the unlimited config is applied at t=3s.

7. 10-Check Matrix

ID

Scenario

Description

Pass criterion

VAL23-01

S-A

Unlimited baseline: exact 4 seeded segments delivered

peer-received.json matches the exact val23-sa-* ID set

VAL23-02

S-A

relay status: unlimited=true when no bandwidth configured

relay-status-baseline.json has bandwidth_status.unlimited = true

VAL23-03

S-B

set-bandwidth audit event emitted

audit-bandwidth-set-rate.log contains relay.bandwidth.configured

VAL23-04

S-B

Rate throttle: ThrottleCount >= 4, QuotaDropCount = 0

relay status after S-B shows throttle_count >= 4, quota_drop_count = 0

VAL23-05

S-B

Rate-limited delivery: exact 5 seeded segments delivered

peer-received.json matches the exact val23-sb-* ID set

VAL23-06

S-C

Quota enforcement: first 2 deliver, remaining 4 deadletter

peer has exact val23-sc-001..002; deadletter has exact val23-sc-003..006

VAL23-07

S-C

QuotaDropCount >= 4 and daily_used_bytes <= daily_quota_bytes

relay status after S-C: quota_drop_count >= 4, daily_used_bytes <= 128

VAL23-08

S-D

Hot-reload: applied=true returned immediately

relay-bandwidth-set-unlimited.txt contains applied=true

VAL23-09

S-D

Post-hot-reload delivery: exact 6 seeded segments delivered

peer-received.json matches the exact val23-sd-* ID set after hot-reload

VAL23-10

S-E

Daily quota reset via deterministic clock

daily-quota-reset-test.txt contains --- PASS: TestBandwidthLimiter_DailyQuota_ResetsAfter24h

Operator-visible output checks

Evidence file

What to verify

sa-unlimited/peer-received.json

exact val23-sa-* segment ID set present once each

sa-unlimited/relay-status-baseline.json

bandwidth_status.unlimited = true

sb-rate-only/relay-bandwidth-set-rate.txt

applied=true

sb-rate-only/audit-bandwidth-set-rate.log

contains relay.bandwidth.configured

sb-rate-only/relay-status-final.json

throttle_count >= 4, quota_drop_count = 0

sb-rate-only/peer-received.json

exact val23-sb-* segment ID set present once each

sc-quota-only/relay-bandwidth-set-quota.txt

applied=true

sc-quota-only/relay-status-final.json

quota_drop_count >= 4, daily_used_bytes <= 128

sc-quota-only/peer-received.json

exact val23-sc-001..002 IDs present

sc-quota-only/deadletter-list-final.txt

exact val23-sc-003..006 IDs present in deadletter

sd-hot-reload/relay-bandwidth-set-unlimited.txt

applied=true

sd-hot-reload/peer-received.json

exact val23-sd-* segment ID set present once each

se-quota-reset/daily-quota-reset-test.txt

--- PASS: TestBandwidthLimiter_DailyQuota_ResetsAfter24h

8. Run the Lab

export GOROOT=/home/ubuntu/.local/go1.25.7
export PATH="$GOROOT/bin:$PATH"
export GOTOOLCHAIN=local

bash scripts/labs/run_relay_bandwidth_val23_lab.sh

Optional custom evidence directory:

bash scripts/labs/run_relay_bandwidth_val23_lab.sh \
  "$PWD/evidence/val23-relay-bandwidth-local-$(date +%F)"

Expected runtime: 2–3 minutes.

  • S-A: ~15s (4-segment delivery with no throttle)

  • S-B: ~25s (rate-throttled delivery, 5 segments at 64B/s)

  • S-C: ~25s (quota exhaustion + deadletter wait)

  • S-D: ~15s (block + hot-reload + delivery)

  • S-E: ~5s (unit test)

9. Final Report Format

VAL23 — Relay Bandwidth Management Validation
Date:           <YYYY-MM-DD>
Environment:    <OS, Go version>
Evidence dir:   <path>

Scenario summary:
  S-A unlimited baseline:     4/4 delivered, unlimited=true
  S-B rate only (64B/s):      5/5 delivered, throttle_count=<N>, quota_drop_count=0
  S-C quota only (128B):      2/6 delivered, quota_drop_count=<N>, daily_used=<N>B
  S-D hot-reload (8→∞ B/s):   6/6 delivered after unlimited applied
  S-E daily reset unit test:  PASS

10-check matrix:
  VAL23-01 PASS/FAIL  unlimited baseline: <count>/4 delivered
  VAL23-02 PASS/FAIL  unlimited status: unlimited=<value>
  VAL23-03 PASS/FAIL  set-bandwidth audit: <found/not found>
  VAL23-04 PASS/FAIL  rate throttle counters: throttle=<N>, quota_drops=<N>
  VAL23-05 PASS/FAIL  rate-limited delivery: <count>/5 delivered
  VAL23-06 PASS/FAIL  quota enforcement: <count>/6 delivered
  VAL23-07 PASS/FAIL  quota drop counters: quota_drops=<N>, daily_used=<N>B
  VAL23-08 PASS/FAIL  hot-reload applied: <true/false>
  VAL23-09 PASS/FAIL  post-hotreload delivery: <count>/6
  VAL23-10 PASS/FAIL  daily quota reset: <PASS/FAIL>

Overall: PASS=<N> FAIL=<N>

10. Tooling

File

Role

scripts/labs/run_relay_bandwidth_val23_lab.sh

Lab runner (entry point)

scripts/labs/edge_relay_bandwidth_val23_setup.go

Setup binary — seeds PENDING segments, generates TLS + edge.toml

scripts/labs/edge_deadletter_lab_peer.go

Reused from PR-14 — mTLS peer server

scripts/labs/edge_deadletter_lab_dump.go

Reused — final ledger state dump