diff --git a/docs/superpowers/specs/2026-06-26-stress-suite-report-integration-design.md b/docs/superpowers/specs/2026-06-26-stress-suite-report-integration-design.md new file mode 100644 index 0000000..dfa6e76 --- /dev/null +++ b/docs/superpowers/specs/2026-06-26-stress-suite-report-integration-design.md @@ -0,0 +1,126 @@ +# Stress Suite Report Integration — Design + +Date: 2026-06-26 + +## Context + +A capacity/stress suite already exists in `Test/E2E/chain/` but is **untracked in +git** and **not wired into the report**: + +- `stress.py` — declarative matrix, 27 cases across 7 load axes, all valid. +- `stress_run.py` — orchestrator: runs each case, captures cpu/peakRSS via + `proc_perf.snapshot()`, zoom p50/p95 latency, gates survival+liveness (hard) and + RSS/zoom-p95 (soft), writes `stress_results.json`. +- `run_stress.sh` — standalone wrapper (build + env + invoke `stress_run.py`). +- `client/main.go` — `stress` mode (liveness + zoom-latency recorder). + +The suite is functional (smoke-tested `ds_size_1000` → PASS end-to-end). The report +pipeline (`run_chain_e2e.sh`, `collect.py`, `report_build.py`, `E2E_Report.typ`) has +**zero** stress references. + +This work covers the user's three requested stress axes — **signal size**, **client +count**, **source count** — which already map onto existing matrix axes +(`{ds,hub}_signal_elements`, `hub_ws_clients` / `ds_subscriber_hubs`, +`hub_source_count`), plus the report integration. + +Separately, the UDPSClient reassembly cap was just raised from 64 KB to 1 MiB +(`UDPS_CLIENT_MAX_PACKET_BYTES = 1048576`), with exact `assembledBytes` delivery. +The stress matrix still enforces a now-outdated sub-64 KB single-datagram ceiling. + +## Goals + +1. Extend the signal-size axis into the multi-fragment regime (exercise the fixed + reassembly path) and lift the outdated 64 KB validation cap. +2. Run the stress suite as an opt-in phase of `run_chain_e2e.sh` (`--stress`). +3. Surface stress results in the PDF: a table, per-axis scaling curves, and + progression/regression vs the previous run (tracked in `history.jsonl`). + +Non-goals: waveform fidelity under stress (UDP loss is expected; the hub decimates +to PushRate), and changing the stress harness's gating semantics. + +## Design + +### 1. Matrix extension (`stress.py`) + +- Extend `_DS_SIZE` and `_HUB_SIZE` with multi-fragment cases: add `50000`, + `100000`, `250000` elements (≈200 KB, 400 KB, ~1 MB float32 packets). +- Large packets get a **slower producer** to keep bandwidth realistic (mirroring + s51's 100 Hz / 10 ms cycle): set `producer_hz`/`row_dt` per case so a ~1 MB packet + is not emitted at 1 kHz (= 1 GB/s). `sampling_rate` stays `elements / row_dt` so + the FirstSample per-cycle window equals one producer cycle (keeps the hub ring's + time axis monotonic — same constraint `scenarios.py` enforces). +- Lift the single-datagram validation in `validate_case()` from 64 KB to the 1 MiB + deliverable cap (`UDPS_CLIENT_MAX_PACKET_BYTES = 1048576`). Update the module + docstring (lines 44–47) to reflect multi-fragment support. + +### 2. Report data wiring (`run_chain_e2e.sh`, `report_build.py`) + +- `run_chain_e2e.sh` gains `--stress` (default off). When set, after the correctness + phase and before `collect.py`/`report_build.py`, it runs the stress phase, reusing + `run_stress.sh`'s env/LD_LIBRARY_PATH setup and the `stress_run.py` invocation, + producing `stress_results.json` in `Build/x86-linux/E2E/chain/stress/`. Honors + `--skip-build`. +- `report_build.py` reads `stress_results.json` *if present* and folds a `stress` + block into `report_data.json`: + - `stress.overall`, `stress.cases[]` (id, axis, level, status, survival, + marte_cpu_s/rss, hub_cpu_s/rss, zoom p50/p95, fails). + - `stress.by_axis` — cases grouped by axis, sorted by level (for the curves). +- When stress was not run, `report_data.json` has no `stress` block and the PDF omits + the section (no stale data). + +### 3. PDF section + scaling plots (`report_build.py`, `E2E_Report.typ`) + +- New `= Stress Tests` section in `E2E_Report.typ`, after the Performance section. +- **Table** — one row per case: axis, level, status, marteCPU, marteRSS, hubRSS, + zoom p50/p95 (matching the existing Performance table style). +- **Scaling curves** — `report_build.py` generates one PNG per axis (reusing its + matplotlib trend-plot helper): load level (x) vs the relevant metric(s) — RSS & CPU + vs level for size/count/sources, zoom-p95 vs reqrate, frames/RSS vs ws_clients. + PNGs land alongside the existing trend plots and are embedded in the Typst section. + +### 4. Regression + history (`report_build.py`) + +- Append a `stress` summary to each `history.jsonl` run record: per-case key metrics + (marte/hub RSS & CPU, zoom-p95, min_frames). +- Extend the `_DIRECTION` map: RSS/CPU/zoom-p95 → lower-is-better; min_frames → + higher-is-better. Surface stress regressions in the existing headline/delta + mechanism (▲/▼ vs last run), consistent with E2E metrics. +- First-run guard: when no prior stress history exists, show baseline values with no + delta (existing `is_first_run` handling). + +### 5. `--stress` flag plumbing (`run_chain_e2e.sh`) + +- Add `--stress` to the arg parser; default off (correctness-only stays the fast + default). +- Update the `run_chain_e2e.sh` header comment and the `CLAUDE.md` E2E paragraph to + document `--stress`. + +## Full flow + +``` +run_chain_e2e.sh [--stress] + ├─ correctness phase → results.json + perf JSON (unchanged) + ├─ [if --stress] stress phase → stress_results.json + ├─ collect.py → unit tests + coverage (unchanged) + └─ report_build.py → report_data.json (+stress block, +history/regression) + └─ E2E_Report.typ → PDF (Stress Tests: table + per-axis scaling curves) +``` + +## Verification + +- `python3 stress.py` validates the extended matrix (multi-fragment cases pass the + lifted cap). +- A multi-fragment size case (e.g. `ds_size_100000`) runs PASS via `run_stress.sh + --only ds_size_100000` (exercises reassembly). +- `run_chain_e2e.sh --stress --skip-build` produces a PDF with the Stress Tests + section (table + curves); a run without `--stress` omits it. +- Second `--stress` run shows progression/regression deltas vs the first. + +## Risks + +- **Bandwidth at large element counts** — mitigated by the slower producer; verify the + hub keeps up (survival gate) and doesn't OOM (RSS gate; ~1 MB/slot ×4 slots ×N + sources). +- **Runtime** — 27+ cases at ~6–8 s each adds minutes; mitigated by opt-in `--stress`. +- **history.jsonl already polluted** by prior single-scenario runs — stress history + starts fresh (new key); no rewrite of existing E2E history.