Martino Ferrari
168 lines
9.0 KiB
Markdown
168 lines
9.0 KiB
Markdown
# Streaming-Chain E2E Test Suite — Design
|
||
|
||
**Date:** 2026-06-25
|
||
**Status:** Approved (autonomous implementation authorised)
|
||
**Sub-project:** A (StreamHub full-chain waveform E2E)
|
||
|
||
## Goal
|
||
|
||
A curated end-to-end test suite that validates the streaming chain
|
||
**MARTe2 App (`UDPStreamer`) → `StreamHub` → client** with *waveform* validation,
|
||
across a covering matrix of `UDPStreamer` configuration options, plus
|
||
StreamHub↔client behavioural tests (live stream, zoom in/out, time window, all
|
||
trigger modes). Output: one combined PDF report **and** a machine-readable
|
||
`results.json`.
|
||
|
||
## Decisions (from brainstorming)
|
||
|
||
1. **Matrix strategy** — curated "covering" set (~40–70 scenarios): every option
|
||
value appears at least once, plus deliberately chosen high-risk interactions.
|
||
2. **Oracle** — hybrid, applied per case:
|
||
- **Analytic (A)**: clean cases reconstruct expected values from the
|
||
generator formula on the received timestamps; tolerance
|
||
`max|err| ≤ quant_step/2 + eps`; decimated streams assert only the samples
|
||
that should survive; LTTB-decimated views use a shape metric (RMSE /
|
||
correlation), not point equality.
|
||
- **Fed-reference (B)**: subtle modes (Accumulate, FullArray, First/LastSample)
|
||
compare received-vs-fed using a `FileWriter` tap of the exact signals MARTe
|
||
fed into the `UDPStreamer`.
|
||
3. **Client testing** — Go mock client (grown from `Test/E2E/streamhub/main.go`)
|
||
is the authoritative functional/waveform gate; plus a thin Qt `QTest` GUI
|
||
smoke test (offscreen) for GUI-level confidence.
|
||
4. **"to disk to client"** — the mock client persists the stream it received to
|
||
disk as a report artifact for offline comparison/plots.
|
||
5. **Output** — one orchestrator + one combined PDF **+** `results.json`.
|
||
|
||
## Architecture
|
||
|
||
New directory `Test/E2E/chain/`. A shell entry point delegates the matrix to a
|
||
Python driver, matching the existing `Test/E2E/datasources/` and
|
||
`Test/E2E/recorder/` conventions (shell + Python + Typst). Artifacts go to
|
||
`Build/x86-linux/E2E/chain/` (never the source tree).
|
||
|
||
### Two-process stack (per scenario)
|
||
```
|
||
gen_data.py -> /tmp/chain_e2e/input_<id>.bin (ground truth + fed reference)
|
||
gen_cfg.py -> marte_<id>.cfg (LinuxTimer + FileReader -> IOGAM -> UDPStreamer
|
||
[+ optional FileWriter tap])
|
||
-> hub_<id>.cfg (StreamHub Source pointing at the UDPStreamer)
|
||
|
||
StreamHub.ex -cfg hub_<id>.cfg [process 1, started first]
|
||
MARTeApp.ex -l RealTimeLoader -f marte_<id>.cfg -s Running [process 2]
|
||
chain-client (Go) -hub 127.0.0.1:<wsport> -scenario <id> -out <dir>
|
||
-> records live pushes, runs zoom/window/trigger checks,
|
||
dumps received_<id>.bin + checks_<id>.json
|
||
validate_waveform.py input_<id>.bin received_<id>.bin [+ tap_<id>.bin]
|
||
-> metrics_<id>.json (+ overlay/diff plots)
|
||
```
|
||
|
||
### Components & responsibilities
|
||
|
||
| File | Responsibility |
|
||
|------|----------------|
|
||
| `Test/E2E/chain/scenarios.py` | Declarative list of ~50 scenario dicts (the covering set) + validity rules |
|
||
| `Test/E2E/chain/gen_data.py` | scenario → deterministic `input_<id>.bin` for all 10 MARTe2 types & shapes; knows the analytic formula per signal |
|
||
| `Test/E2E/chain/gen_cfg.py` | scenario → MARTe2 app `.cfg` and StreamHub `.cfg` (unicast/multicast, Strict/Accumulate/Decimate, per-signal type/shape/timemode/quant, optional FileWriter tap) |
|
||
| `Test/E2E/chain/client/` | Go mock client (grown from `streamhub/main.go`): record live, zoom, window, all trigger modes; dump `received_<id>.bin` + `checks_<id>.json` |
|
||
| `Test/E2E/chain/validate_waveform.py` | ground-truth/fed-reference vs received; per-effect tolerance; emit `metrics_<id>.json` |
|
||
| `Test/E2E/chain/plots.py` | matplotlib overlays (truth/received/diff), trigger-capture, zoom/window figures |
|
||
| `Test/E2E/chain/run_chain_e2e.sh` | orchestrator: build, loop scenarios over the two-process stack, validate, aggregate `results.json`, compile PDF |
|
||
| `Test/E2E/chain/E2E_Report.typ` | Typst report template → `E2E_Report.pdf` |
|
||
| `Client/streamhub-qt/test/` | Qt `QTest` smoke (offscreen), run via ctest |
|
||
|
||
## Covering matrix (~50 scenarios)
|
||
|
||
Grouped so each option value is hit at least once, with targeted interactions.
|
||
Config-surface reference (verified against source):
|
||
|
||
- **Publishing modes**: `Strict`, `Accumulate` (+`MinRefreshRate`), `Decimate` (+`Ratio`).
|
||
- **Network**: unicast (`Port`); multicast (`MulticastGroup` + `DataPort`).
|
||
- **Signal types**: uint8/int8/uint16/int16/uint32/int32/uint64/int64/float32/float64.
|
||
- **Time modes** (per signal): `PacketTime`, `FullArray`, `FirstSample`,
|
||
`LastSample` (+ `TimeSignal`, `SamplingRate`).
|
||
- **Quantisation** (float only): `none`, `uint8`, `int8`, `uint16`, `int16`
|
||
(+`RangeMin`/`RangeMax`).
|
||
- **Shapes**: scalar, 100-element array, 5000-element array (fragmentation).
|
||
- **Payload**: default 1400, small (512, forces fragmentation), jumbo (65507).
|
||
- **Sources**: single and multiple sources (two UDPStreamer blocks → two hub sources).
|
||
|
||
Representative buckets (illustrative, ~50 total):
|
||
- Per-type scalar Strict unicast (10) — type fidelity.
|
||
- Time-mode sweep on arrays (FullArray/FirstSample/LastSample/PacketTime × float32/uint64-time) (~8).
|
||
- Quantisation sweep (none/uint8/int8/uint16/int16 with matching RangeMin/Max) (~6).
|
||
- Publishing modes (Strict/Accumulate@{10,100,1000Hz}/Decimate@{1,5,10}) (~8).
|
||
- Multicast variants of the above (~6).
|
||
- Shapes/fragmentation (scalar/100/5000 × payload 512/1400/65507) (~6).
|
||
- Multi-source (2–3 sources, mixed configs) (~3).
|
||
- Edge/robustness (mixed-type multi-signal source, large array + quant) (~3).
|
||
|
||
## Oracle details
|
||
|
||
Ground truth is the analytic generator. For a received sample at time `t` for
|
||
signal `s`, the expected value is `f_s(t)` reconstructed from the generator's
|
||
per-signal formula and the row/cycle clock.
|
||
|
||
- **Type fidelity (no quant)**: integer/float types must be bit-exact after the
|
||
type round-trip (`err == 0`).
|
||
- **Quantised float**: `quant_step = (RangeMax-RangeMin)/levels` (levels = 255,
|
||
254, 65535, 65534 for uint8/int8/uint16/int16). Require
|
||
`max|recv − truth| ≤ quant_step/2 + 1e-6·range`.
|
||
- **Decimate (Ratio R)**: only every R-th producer cycle is transmitted; assert
|
||
the surviving subsequence matches truth; received count ≈ produced/R.
|
||
- **Accumulate**: batched samples must reconstruct the original per-cycle
|
||
sequence within float eps; validated against the **fed reference** tap.
|
||
- **Time modes**: reconstructed timestamps must be monotonic, within wall-clock
|
||
bounds, and spaced by `1/SamplingRate` (First/LastSample) or equal the time
|
||
array (FullArray); compared to fed reference where reconstruction is subtle.
|
||
- **LTTB live/zoom views**: shape metric — Pearson correlation ≥ 0.99 and
|
||
normalised RMSE ≤ tolerance against truth resampled onto returned timestamps.
|
||
|
||
## Client behavioural validation (Go mock, authoritative)
|
||
|
||
Per scenario the client performs and records:
|
||
- **Live**: ≥N binary v1 frames; per-signal wall-clock + monotonic time; dump
|
||
`received_<id>.bin`; waveform vs truth (shape metric).
|
||
- **Zoom**: ≥2 ranges (narrow "in" and wide "out"); points within `[t0,t1]`;
|
||
count ≤ requested `n`; shape vs truth. Also `historyZoom` when history enabled.
|
||
- **Time window**: set `windowSec`; assert returned/clamped span matches.
|
||
- **Triggers**: matrix of `edge ∈ {rising,falling,both}` × `mode ∈
|
||
{normal,single}` on an oscillating signal; validate triggerState transitions,
|
||
v2 capture window `[trigTime−pre, trigTime+post]`, the edge actually crosses
|
||
threshold in the correct direction at `trigTime`, and captured waveform vs
|
||
truth. Normal mode must re-arm; single must not until `rearm`.
|
||
- Emit `checks_<id>.json` (per-check pass/fail + measured values).
|
||
|
||
## Qt GUI smoke (thin)
|
||
|
||
`Client/streamhub-qt/test/smoke_test.cpp` using `QTest`, `QT_QPA_PLATFORM=offscreen`:
|
||
construct `MainWindow`, point at a live StreamHub, spin the event loop, assert it
|
||
received ≥1 source and rendered ≥1 frame, exercise a layout change and pause
|
||
toggle without crashing. Built via the existing CMake; run via `ctest`.
|
||
|
||
## Report
|
||
|
||
`E2E_Report.typ` → `E2E_Report.pdf`:
|
||
- Title + environment + matrix summary table (scenario, config, oracle, result).
|
||
- Per-scenario waveform overlays (truth / received / diff) for a representative
|
||
subset; aggregate pass/fail.
|
||
- Trigger-capture plots (one per edge/mode).
|
||
- Zoom/window plots.
|
||
- Qt smoke result.
|
||
|
||
`results.json` — aggregate machine-readable: list of scenarios with config
|
||
summary, per-check metrics, pass/fail, and overall status (for CI gating).
|
||
|
||
## Failure & robustness
|
||
|
||
- Each scenario runs under `timeout`; stack always torn down via trap.
|
||
- A scenario that fails to produce data fails that scenario only; the suite
|
||
continues and the report records it.
|
||
- Multicast scenarios are skipped (recorded as SKIP) if the loopback multicast
|
||
route is unavailable, rather than hanging.
|
||
|
||
## Out of scope (covered elsewhere)
|
||
|
||
- Per-component standalone Typst reports → Sub-project C.
|
||
- DebugService E2E → Sub-project B.
|
||
- Real GUI click-automation beyond the QTest smoke.
|