Streaming-Chain E2E Test Suite — Design

Date: 2026-06-25 Status: Approved (autonomous implementation authorised) Sub-project: A (StreamHub full-chain waveform E2E)

Goal

A curated end-to-end test suite that validates the streaming chain MARTe2 App (UDPStreamer) → StreamHub → client with waveform validation, across a covering matrix of UDPStreamer configuration options, plus StreamHub↔client behavioural tests (live stream, zoom in/out, time window, all trigger modes). Output: one combined PDF report and a machine-readable results.json.

Decisions (from brainstorming)

Matrix strategy — curated "covering" set (~40–70 scenarios): every option value appears at least once, plus deliberately chosen high-risk interactions.
Oracle — hybrid, applied per case:
- Analytic (A): clean cases reconstruct expected values from the generator formula on the received timestamps; tolerance max|err| ≤ quant_step/2 + eps; decimated streams assert only the samples that should survive; LTTB-decimated views use a shape metric (RMSE / correlation), not point equality.
- Fed-reference (B): subtle modes (Accumulate, FullArray, First/LastSample) compare received-vs-fed using a FileWriter tap of the exact signals MARTe fed into the UDPStreamer.
Client testing — Go mock client (grown from Test/E2E/streamhub/main.go) is the authoritative functional/waveform gate; plus a thin Qt QTest GUI smoke test (offscreen) for GUI-level confidence.
"to disk to client" — the mock client persists the stream it received to disk as a report artifact for offline comparison/plots.
Output — one orchestrator + one combined PDF + results.json.

Architecture

New directory Test/E2E/chain/. A shell entry point delegates the matrix to a Python driver, matching the existing Test/E2E/datasources/ and Test/E2E/recorder/ conventions (shell + Python + Typst). Artifacts go to Build/x86-linux/E2E/chain/ (never the source tree).

Two-process stack (per scenario)

gen_data.py  -> /tmp/chain_e2e/input_<id>.bin   (ground truth + fed reference)
gen_cfg.py   -> marte_<id>.cfg  (LinuxTimer + FileReader -> IOGAM -> UDPStreamer
                                  [+ optional FileWriter tap])
             -> hub_<id>.cfg    (StreamHub Source pointing at the UDPStreamer)

StreamHub.ex -cfg hub_<id>.cfg            [process 1, started first]
MARTeApp.ex  -l RealTimeLoader -f marte_<id>.cfg -s Running   [process 2]
chain-client (Go) -hub 127.0.0.1:<wsport> -scenario <id> -out <dir>
             -> records live pushes, runs zoom/window/trigger checks,
                dumps received_<id>.bin + checks_<id>.json
validate_waveform.py  input_<id>.bin  received_<id>.bin  [+ tap_<id>.bin]
             -> metrics_<id>.json  (+ overlay/diff plots)

Components & responsibilities

File	Responsibility
`Test/E2E/chain/scenarios.py`	Declarative list of ~50 scenario dicts (the covering set) + validity rules
`Test/E2E/chain/gen_data.py`	scenario → deterministic `input_<id>.bin` for all 10 MARTe2 types & shapes; knows the analytic formula per signal
`Test/E2E/chain/gen_cfg.py`	scenario → MARTe2 app `.cfg` and StreamHub `.cfg` (unicast/multicast, Strict/Accumulate/Decimate, per-signal type/shape/timemode/quant, optional FileWriter tap)
`Test/E2E/chain/client/`	Go mock client (grown from `streamhub/main.go`): record live, zoom, window, all trigger modes; dump `received_<id>.bin` + `checks_<id>.json`
`Test/E2E/chain/validate_waveform.py`	ground-truth/fed-reference vs received; per-effect tolerance; emit `metrics_<id>.json`
`Test/E2E/chain/plots.py`	matplotlib overlays (truth/received/diff), trigger-capture, zoom/window figures
`Test/E2E/chain/run_chain_e2e.sh`	orchestrator: build, loop scenarios over the two-process stack, validate, aggregate `results.json`, compile PDF
`Test/E2E/chain/E2E_Report.typ`	Typst report template → `E2E_Report.pdf`
`Client/streamhub-qt/test/`	Qt `QTest` smoke (offscreen), run via ctest

Covering matrix (~50 scenarios)

Grouped so each option value is hit at least once, with targeted interactions. Config-surface reference (verified against source):

Publishing modes: Strict, Accumulate (+MinRefreshRate), Decimate (+Ratio).
Network: unicast (Port); multicast (MulticastGroup + DataPort).
Signal types: uint8/int8/uint16/int16/uint32/int32/uint64/int64/float32/float64.
Time modes (per signal): PacketTime, FullArray, FirstSample, LastSample (+ TimeSignal, SamplingRate).
Quantisation (float only): none, uint8, int8, uint16, int16 (+RangeMin/RangeMax).
Shapes: scalar, 100-element array, 5000-element array (fragmentation).
Payload: default 1400, small (512, forces fragmentation), jumbo (65507).
Sources: single and multiple sources (two UDPStreamer blocks → two hub sources).

Representative buckets (illustrative, ~50 total):

Per-type scalar Strict unicast (10) — type fidelity.
Time-mode sweep on arrays (FullArray/FirstSample/LastSample/PacketTime × float32/uint64-time) (~8).
Quantisation sweep (none/uint8/int8/uint16/int16 with matching RangeMin/Max) (~6).
Publishing modes (Strict/Accumulate@{10,100,1000Hz}/Decimate@{1,5,10}) (~8).
Multicast variants of the above (~6).
Shapes/fragmentation (scalar/100/5000 × payload 512/1400/65507) (~6).
Multi-source (2–3 sources, mixed configs) (~3).
Edge/robustness (mixed-type multi-signal source, large array + quant) (~3).

Oracle details

Ground truth is the analytic generator. For a received sample at time t for signal s, the expected value is f_s(t) reconstructed from the generator's per-signal formula and the row/cycle clock.

Type fidelity (no quant): integer/float types must be bit-exact after the type round-trip (err == 0).
Quantised float: quant_step = (RangeMax-RangeMin)/levels (levels = 255, 254, 65535, 65534 for uint8/int8/uint16/int16). Require max|recv − truth| ≤ quant_step/2 + 1e-6·range.
Decimate (Ratio R): only every R-th producer cycle is transmitted; assert the surviving subsequence matches truth; received count ≈ produced/R.
Accumulate: batched samples must reconstruct the original per-cycle sequence within float eps; validated against the fed reference tap.
Time modes: reconstructed timestamps must be monotonic, within wall-clock bounds, and spaced by 1/SamplingRate (First/LastSample) or equal the time array (FullArray); compared to fed reference where reconstruction is subtle.
LTTB live/zoom views: shape metric — Pearson correlation ≥ 0.99 and normalised RMSE ≤ tolerance against truth resampled onto returned timestamps.

Client behavioural validation (Go mock, authoritative)

Per scenario the client performs and records:

Live: ≥N binary v1 frames; per-signal wall-clock + monotonic time; dump received_<id>.bin; waveform vs truth (shape metric).
Zoom: ≥2 ranges (narrow "in" and wide "out"); points within [t0,t1]; count ≤ requested n; shape vs truth. Also historyZoom when history enabled.
Time window: set windowSec; assert returned/clamped span matches.
Triggers: matrix of edge ∈ {rising,falling,both} × mode ∈ {normal,single} on an oscillating signal; validate triggerState transitions, v2 capture window [trigTime−pre, trigTime+post], the edge actually crosses threshold in the correct direction at trigTime, and captured waveform vs truth. Normal mode must re-arm; single must not until rearm.
Emit checks_<id>.json (per-check pass/fail + measured values).

Qt GUI smoke (thin)

Client/streamhub-qt/test/smoke_test.cpp using QTest, QT_QPA_PLATFORM=offscreen: construct MainWindow, point at a live StreamHub, spin the event loop, assert it received ≥1 source and rendered ≥1 frame, exercise a layout change and pause toggle without crashing. Built via the existing CMake; run via ctest.

Report

E2E_Report.typ → E2E_Report.pdf:

Title + environment + matrix summary table (scenario, config, oracle, result).
Per-scenario waveform overlays (truth / received / diff) for a representative subset; aggregate pass/fail.
Trigger-capture plots (one per edge/mode).
Zoom/window plots.
Qt smoke result.

results.json — aggregate machine-readable: list of scenarios with config summary, per-check metrics, pass/fail, and overall status (for CI gating).

Failure & robustness

Each scenario runs under timeout; stack always torn down via trap.
A scenario that fails to produce data fails that scenario only; the suite continues and the report records it.
Multicast scenarios are skipped (recorded as SKIP) if the loopback multicast route is unavailable, rather than hanging.

Out of scope (covered elsewhere)

Per-component standalone Typst reports → Sub-project C.
DebugService E2E → Sub-project B.
Real GUI click-automation beyond the QTest smoke.

9.0 KiB Raw Permalink Blame History Unescape Escape