# Streaming-Chain E2E Test Suite — Design

**Date:** 2026-06-25
**Status:** Approved (autonomous implementation authorised)
**Sub-project:** A (StreamHub full-chain waveform E2E)

## Goal

A curated end-to-end test suite that validates the streaming chain
**MARTe2 App (`UDPStreamer`) → `StreamHub` → client** with *waveform* validation,
across a covering matrix of `UDPStreamer` configuration options, plus
StreamHub↔client behavioural tests (live stream, zoom in/out, time window, all
trigger modes). Output: one combined PDF report **and** a machine-readable
`results.json`.

## Decisions (from brainstorming)

1. **Matrix strategy** — curated "covering" set (~40–70 scenarios): every option
   value appears at least once, plus deliberately chosen high-risk interactions.
2. **Oracle** — hybrid, applied per case:
   - **Analytic (A)**: clean cases reconstruct expected values from the
     generator formula on the received timestamps; tolerance
     `max|err| ≤ quant_step/2 + eps`; decimated streams assert only the samples
     that should survive; LTTB-decimated views use a shape metric (RMSE /
     correlation), not point equality.
   - **Fed-reference (B)**: subtle modes (Accumulate, FullArray, First/LastSample)
     compare received-vs-fed using a `FileWriter` tap of the exact signals MARTe
     fed into the `UDPStreamer`.
3. **Client testing** — Go mock client (grown from `Test/E2E/streamhub/main.go`)
   is the authoritative functional/waveform gate; plus a thin Qt `QTest` GUI
   smoke test (offscreen) for GUI-level confidence.
4. **"to disk to client"** — the mock client persists the stream it received to
   disk as a report artifact for offline comparison/plots.
5. **Output** — one orchestrator + one combined PDF **+** `results.json`.

## Architecture

New directory `Test/E2E/chain/`. A shell entry point delegates the matrix to a
Python driver, matching the existing `Test/E2E/datasources/` and
`Test/E2E/recorder/` conventions (shell + Python + Typst). Artifacts go to
`Build/x86-linux/E2E/chain/` (never the source tree).

### Two-process stack (per scenario)
```
gen_data.py  -> /tmp/chain_e2e/input_<id>.bin   (ground truth + fed reference)
gen_cfg.py   -> marte_<id>.cfg  (LinuxTimer + FileReader -> IOGAM -> UDPStreamer
                                  [+ optional FileWriter tap])
             -> hub_<id>.cfg    (StreamHub Source pointing at the UDPStreamer)

StreamHub.ex -cfg hub_<id>.cfg            [process 1, started first]
MARTeApp.ex  -l RealTimeLoader -f marte_<id>.cfg -s Running   [process 2]
chain-client (Go) -hub 127.0.0.1:<wsport> -scenario <id> -out <dir>
             -> records live pushes, runs zoom/window/trigger checks,
                dumps received_<id>.bin + checks_<id>.json
validate_waveform.py  input_<id>.bin  received_<id>.bin  [+ tap_<id>.bin]
             -> metrics_<id>.json  (+ overlay/diff plots)
```

### Components & responsibilities

| File | Responsibility |
|------|----------------|
| `Test/E2E/chain/scenarios.py` | Declarative list of ~50 scenario dicts (the covering set) + validity rules |
| `Test/E2E/chain/gen_data.py` | scenario → deterministic `input_<id>.bin` for all 10 MARTe2 types & shapes; knows the analytic formula per signal |
| `Test/E2E/chain/gen_cfg.py` | scenario → MARTe2 app `.cfg` and StreamHub `.cfg` (unicast/multicast, Strict/Accumulate/Decimate, per-signal type/shape/timemode/quant, optional FileWriter tap) |
| `Test/E2E/chain/client/` | Go mock client (grown from `streamhub/main.go`): record live, zoom, window, all trigger modes; dump `received_<id>.bin` + `checks_<id>.json` |
| `Test/E2E/chain/validate_waveform.py` | ground-truth/fed-reference vs received; per-effect tolerance; emit `metrics_<id>.json` |
| `Test/E2E/chain/plots.py` | matplotlib overlays (truth/received/diff), trigger-capture, zoom/window figures |
| `Test/E2E/chain/run_chain_e2e.sh` | orchestrator: build, loop scenarios over the two-process stack, validate, aggregate `results.json`, compile PDF |
| `Test/E2E/chain/E2E_Report.typ` | Typst report template → `E2E_Report.pdf` |
| `Client/streamhub-qt/test/` | Qt `QTest` smoke (offscreen), run via ctest |

## Covering matrix (~50 scenarios)

Grouped so each option value is hit at least once, with targeted interactions.
Config-surface reference (verified against source):

- **Publishing modes**: `Strict`, `Accumulate` (+`MinRefreshRate`), `Decimate` (+`Ratio`).
- **Network**: unicast (`Port`); multicast (`MulticastGroup` + `DataPort`).
- **Signal types**: uint8/int8/uint16/int16/uint32/int32/uint64/int64/float32/float64.
- **Time modes** (per signal): `PacketTime`, `FullArray`, `FirstSample`,
  `LastSample` (+ `TimeSignal`, `SamplingRate`).
- **Quantisation** (float only): `none`, `uint8`, `int8`, `uint16`, `int16`
  (+`RangeMin`/`RangeMax`).
- **Shapes**: scalar, 100-element array, 5000-element array (fragmentation).
- **Payload**: default 1400, small (512, forces fragmentation), jumbo (65507).
- **Sources**: single and multiple sources (two UDPStreamer blocks → two hub sources).

Representative buckets (illustrative, ~50 total):
- Per-type scalar Strict unicast (10) — type fidelity.
- Time-mode sweep on arrays (FullArray/FirstSample/LastSample/PacketTime × float32/uint64-time) (~8).
- Quantisation sweep (none/uint8/int8/uint16/int16 with matching RangeMin/Max) (~6).
- Publishing modes (Strict/Accumulate@{10,100,1000Hz}/Decimate@{1,5,10}) (~8).
- Multicast variants of the above (~6).
- Shapes/fragmentation (scalar/100/5000 × payload 512/1400/65507) (~6).
- Multi-source (2–3 sources, mixed configs) (~3).
- Edge/robustness (mixed-type multi-signal source, large array + quant) (~3).

## Oracle details

Ground truth is the analytic generator. For a received sample at time `t` for
signal `s`, the expected value is `f_s(t)` reconstructed from the generator's
per-signal formula and the row/cycle clock.

- **Type fidelity (no quant)**: integer/float types must be bit-exact after the
  type round-trip (`err == 0`).
- **Quantised float**: `quant_step = (RangeMax-RangeMin)/levels` (levels = 255,
  254, 65535, 65534 for uint8/int8/uint16/int16). Require
  `max|recv − truth| ≤ quant_step/2 + 1e-6·range`.
- **Decimate (Ratio R)**: only every R-th producer cycle is transmitted; assert
  the surviving subsequence matches truth; received count ≈ produced/R.
- **Accumulate**: batched samples must reconstruct the original per-cycle
  sequence within float eps; validated against the **fed reference** tap.
- **Time modes**: reconstructed timestamps must be monotonic, within wall-clock
  bounds, and spaced by `1/SamplingRate` (First/LastSample) or equal the time
  array (FullArray); compared to fed reference where reconstruction is subtle.
- **LTTB live/zoom views**: shape metric — Pearson correlation ≥ 0.99 and
  normalised RMSE ≤ tolerance against truth resampled onto returned timestamps.

## Client behavioural validation (Go mock, authoritative)

Per scenario the client performs and records:
- **Live**: ≥N binary v1 frames; per-signal wall-clock + monotonic time; dump
  `received_<id>.bin`; waveform vs truth (shape metric).
- **Zoom**: ≥2 ranges (narrow "in" and wide "out"); points within `[t0,t1]`;
  count ≤ requested `n`; shape vs truth. Also `historyZoom` when history enabled.
- **Time window**: set `windowSec`; assert returned/clamped span matches.
- **Triggers**: matrix of `edge ∈ {rising,falling,both}` × `mode ∈
  {normal,single}` on an oscillating signal; validate triggerState transitions,
  v2 capture window `[trigTime−pre, trigTime+post]`, the edge actually crosses
  threshold in the correct direction at `trigTime`, and captured waveform vs
  truth. Normal mode must re-arm; single must not until `rearm`.
- Emit `checks_<id>.json` (per-check pass/fail + measured values).

## Qt GUI smoke (thin)

`Client/streamhub-qt/test/smoke_test.cpp` using `QTest`, `QT_QPA_PLATFORM=offscreen`:
construct `MainWindow`, point at a live StreamHub, spin the event loop, assert it
received ≥1 source and rendered ≥1 frame, exercise a layout change and pause
toggle without crashing. Built via the existing CMake; run via `ctest`.

## Report

`E2E_Report.typ` → `E2E_Report.pdf`:
- Title + environment + matrix summary table (scenario, config, oracle, result).
- Per-scenario waveform overlays (truth / received / diff) for a representative
  subset; aggregate pass/fail.
- Trigger-capture plots (one per edge/mode).
- Zoom/window plots.
- Qt smoke result.

`results.json` — aggregate machine-readable: list of scenarios with config
summary, per-check metrics, pass/fail, and overall status (for CI gating).

## Failure & robustness

- Each scenario runs under `timeout`; stack always torn down via trap.
- A scenario that fails to produce data fails that scenario only; the suite
  continues and the report records it.
- Multicast scenarios are skipped (recorded as SKIP) if the loopback multicast
  route is unavailable, rather than hanging.

## Out of scope (covered elsewhere)

- Per-component standalone Typst reports → Sub-project C.
- DebugService E2E → Sub-project B.
- Real GUI click-automation beyond the QTest smoke.