docs(e2e-stress): design for stress suite report integration
Track the existing (untracked) stress matrix, extend the signal-size axis into the multi-fragment regime now that the UDPSClient reassembly cap is 1 MiB, and wire stress results into the E2E PDF report via an opt-in --stress flag (table + per-axis scaling curves + regression vs prior run). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
Martino Ferrari
@@ -0,0 +1,126 @@
|
||||
# Stress Suite Report Integration — Design
|
||||
|
||||
Date: 2026-06-26
|
||||
|
||||
## Context
|
||||
|
||||
A capacity/stress suite already exists in `Test/E2E/chain/` but is **untracked in
|
||||
git** and **not wired into the report**:
|
||||
|
||||
- `stress.py` — declarative matrix, 27 cases across 7 load axes, all valid.
|
||||
- `stress_run.py` — orchestrator: runs each case, captures cpu/peakRSS via
|
||||
`proc_perf.snapshot()`, zoom p50/p95 latency, gates survival+liveness (hard) and
|
||||
RSS/zoom-p95 (soft), writes `stress_results.json`.
|
||||
- `run_stress.sh` — standalone wrapper (build + env + invoke `stress_run.py`).
|
||||
- `client/main.go` — `stress` mode (liveness + zoom-latency recorder).
|
||||
|
||||
The suite is functional (smoke-tested `ds_size_1000` → PASS end-to-end). The report
|
||||
pipeline (`run_chain_e2e.sh`, `collect.py`, `report_build.py`, `E2E_Report.typ`) has
|
||||
**zero** stress references.
|
||||
|
||||
This work covers the user's three requested stress axes — **signal size**, **client
|
||||
count**, **source count** — which already map onto existing matrix axes
|
||||
(`{ds,hub}_signal_elements`, `hub_ws_clients` / `ds_subscriber_hubs`,
|
||||
`hub_source_count`), plus the report integration.
|
||||
|
||||
Separately, the UDPSClient reassembly cap was just raised from 64 KB to 1 MiB
|
||||
(`UDPS_CLIENT_MAX_PACKET_BYTES = 1048576`), with exact `assembledBytes` delivery.
|
||||
The stress matrix still enforces a now-outdated sub-64 KB single-datagram ceiling.
|
||||
|
||||
## Goals
|
||||
|
||||
1. Extend the signal-size axis into the multi-fragment regime (exercise the fixed
|
||||
reassembly path) and lift the outdated 64 KB validation cap.
|
||||
2. Run the stress suite as an opt-in phase of `run_chain_e2e.sh` (`--stress`).
|
||||
3. Surface stress results in the PDF: a table, per-axis scaling curves, and
|
||||
progression/regression vs the previous run (tracked in `history.jsonl`).
|
||||
|
||||
Non-goals: waveform fidelity under stress (UDP loss is expected; the hub decimates
|
||||
to PushRate), and changing the stress harness's gating semantics.
|
||||
|
||||
## Design
|
||||
|
||||
### 1. Matrix extension (`stress.py`)
|
||||
|
||||
- Extend `_DS_SIZE` and `_HUB_SIZE` with multi-fragment cases: add `50000`,
|
||||
`100000`, `250000` elements (≈200 KB, 400 KB, ~1 MB float32 packets).
|
||||
- Large packets get a **slower producer** to keep bandwidth realistic (mirroring
|
||||
s51's 100 Hz / 10 ms cycle): set `producer_hz`/`row_dt` per case so a ~1 MB packet
|
||||
is not emitted at 1 kHz (= 1 GB/s). `sampling_rate` stays `elements / row_dt` so
|
||||
the FirstSample per-cycle window equals one producer cycle (keeps the hub ring's
|
||||
time axis monotonic — same constraint `scenarios.py` enforces).
|
||||
- Lift the single-datagram validation in `validate_case()` from 64 KB to the 1 MiB
|
||||
deliverable cap (`UDPS_CLIENT_MAX_PACKET_BYTES = 1048576`). Update the module
|
||||
docstring (lines 44–47) to reflect multi-fragment support.
|
||||
|
||||
### 2. Report data wiring (`run_chain_e2e.sh`, `report_build.py`)
|
||||
|
||||
- `run_chain_e2e.sh` gains `--stress` (default off). When set, after the correctness
|
||||
phase and before `collect.py`/`report_build.py`, it runs the stress phase, reusing
|
||||
`run_stress.sh`'s env/LD_LIBRARY_PATH setup and the `stress_run.py` invocation,
|
||||
producing `stress_results.json` in `Build/x86-linux/E2E/chain/stress/`. Honors
|
||||
`--skip-build`.
|
||||
- `report_build.py` reads `stress_results.json` *if present* and folds a `stress`
|
||||
block into `report_data.json`:
|
||||
- `stress.overall`, `stress.cases[]` (id, axis, level, status, survival,
|
||||
marte_cpu_s/rss, hub_cpu_s/rss, zoom p50/p95, fails).
|
||||
- `stress.by_axis` — cases grouped by axis, sorted by level (for the curves).
|
||||
- When stress was not run, `report_data.json` has no `stress` block and the PDF omits
|
||||
the section (no stale data).
|
||||
|
||||
### 3. PDF section + scaling plots (`report_build.py`, `E2E_Report.typ`)
|
||||
|
||||
- New `= Stress Tests` section in `E2E_Report.typ`, after the Performance section.
|
||||
- **Table** — one row per case: axis, level, status, marteCPU, marteRSS, hubRSS,
|
||||
zoom p50/p95 (matching the existing Performance table style).
|
||||
- **Scaling curves** — `report_build.py` generates one PNG per axis (reusing its
|
||||
matplotlib trend-plot helper): load level (x) vs the relevant metric(s) — RSS & CPU
|
||||
vs level for size/count/sources, zoom-p95 vs reqrate, frames/RSS vs ws_clients.
|
||||
PNGs land alongside the existing trend plots and are embedded in the Typst section.
|
||||
|
||||
### 4. Regression + history (`report_build.py`)
|
||||
|
||||
- Append a `stress` summary to each `history.jsonl` run record: per-case key metrics
|
||||
(marte/hub RSS & CPU, zoom-p95, min_frames).
|
||||
- Extend the `_DIRECTION` map: RSS/CPU/zoom-p95 → lower-is-better; min_frames →
|
||||
higher-is-better. Surface stress regressions in the existing headline/delta
|
||||
mechanism (▲/▼ vs last run), consistent with E2E metrics.
|
||||
- First-run guard: when no prior stress history exists, show baseline values with no
|
||||
delta (existing `is_first_run` handling).
|
||||
|
||||
### 5. `--stress` flag plumbing (`run_chain_e2e.sh`)
|
||||
|
||||
- Add `--stress` to the arg parser; default off (correctness-only stays the fast
|
||||
default).
|
||||
- Update the `run_chain_e2e.sh` header comment and the `CLAUDE.md` E2E paragraph to
|
||||
document `--stress`.
|
||||
|
||||
## Full flow
|
||||
|
||||
```
|
||||
run_chain_e2e.sh [--stress]
|
||||
├─ correctness phase → results.json + perf JSON (unchanged)
|
||||
├─ [if --stress] stress phase → stress_results.json
|
||||
├─ collect.py → unit tests + coverage (unchanged)
|
||||
└─ report_build.py → report_data.json (+stress block, +history/regression)
|
||||
└─ E2E_Report.typ → PDF (Stress Tests: table + per-axis scaling curves)
|
||||
```
|
||||
|
||||
## Verification
|
||||
|
||||
- `python3 stress.py` validates the extended matrix (multi-fragment cases pass the
|
||||
lifted cap).
|
||||
- A multi-fragment size case (e.g. `ds_size_100000`) runs PASS via `run_stress.sh
|
||||
--only ds_size_100000` (exercises reassembly).
|
||||
- `run_chain_e2e.sh --stress --skip-build` produces a PDF with the Stress Tests
|
||||
section (table + curves); a run without `--stress` omits it.
|
||||
- Second `--stress` run shows progression/regression deltas vs the first.
|
||||
|
||||
## Risks
|
||||
|
||||
- **Bandwidth at large element counts** — mitigated by the slower producer; verify the
|
||||
hub keeps up (survival gate) and doesn't OOM (RSS gate; ~1 MB/slot ×4 slots ×N
|
||||
sources).
|
||||
- **Runtime** — 27+ cases at ~6–8 s each adds minutes; mitigated by opt-in `--stress`.
|
||||
- **history.jsonl already polluted** by prior single-scenario runs — stress history
|
||||
starts fresh (new key); no rewrite of existing E2E history.
|
||||
Reference in New Issue
Block a user