21 KiB
21 KiB
E2E Test Suite
The streaming-chain end-to-end suite (Test/E2E/suite/) validates the full data path from
MARTe2 real-time application through the UDPS wire protocol to StreamHub and client consumers.
It also covers the debug/trace path (DebugService, TCPLogger) and the direct
UDPStreamer-to-UDPStreamerClient round-trip.
Overview
The suite is driven by a single orchestrator script:
source env.sh
./Test/E2E/suite/run_e2e.sh [flags]
For each scenario defined in scenarios.py, the orchestrator:
- Generates input data (
gen_data.py) — deterministic typed/shaped binary in MARTe2 FileReader format, plus a ground-truth dict for the validator. - Generates configs (
gen_cfg.py) — MARTe2 app config (LinuxTimer + FileReader + IOGAM + UDPStreamer) and StreamHub config, per scenario. - Launches the server stack — MARTe2 app + StreamHub (for chain/recorder scenarios) or MARTe2 app alone (for direct/debug scenarios).
- Drives mock clients — the Go
chain-client(chain scenarios) ordebugclient(debug/tcplogger scenarios) connects, records data, and runs behavioural checks. - Validates (
validate_waveform.py) — compares the recorded stream against the analytic ground truth and/or the fed-reference tap file. - Renders plots (
plots.py) — waveform, trigger, and zoom overlay PNGs per scenario. - Runs unit tests + coverage (
collect.py) — C++ GTest, Go, and Python suites with optional lcov C++ line coverage. - Runs stress matrix (
stress_run.py/stress.py) — capacity sweeps (signal size, count, fan-out, zoom rate) with survival/liveness/RSS/latency gates. - Builds the report (
report_build.py) — consolidates everything intoreport_data.jsonwith regression tracking against the previous run, trend plots, and a Typst PDF (E2E_Report.typ).
Flags
| Flag | Effect |
|---|---|
--skip-build |
Skip C++ component rebuild |
--only <id> |
Run a single scenario by ID |
--pdf-only |
Just compile the Typst PDF report (no tests) |
--cpp-coverage |
Instrumented gcov rebuild + lcov capture (on by default) |
--skip-coverage |
Disable the coverage pass |
--skip-stress |
Skip the stress matrix |
--skip-datasources |
Skip direct scenarios |
--skip-recorder |
Skip recorder scenarios |
--skip-debug |
Skip debug and debug_pause_resume scenarios |
--skip-tcplogger |
Skip tcplogger scenarios |
Scenario Kinds
chain
Full streaming pipeline: MARTe2 (FileReader -> IOGAM -> UDPStreamer) -> StreamHub -> Go
chain-client. The client records the live binary stream and runs behavioural checks
(live, zoom, window, trigger). The validator compares the recording against the analytic
ground truth (fidelity, sine shape fit, continuity) and optionally a fed-reference tap.
direct
MARTe2 FileReader -> UDPStreamer -> UDPStreamerClient -> FileWriter round-trip. Validates that the written binary matches the input binary (bit-exact for each signal type).
recorder
MARTe2 -> UDPStreamer -> StreamHub with BinaryRecorder enabled. Validates the .bin file
written to disk by the recorder against the original input.
debug / debug_pause_resume
DebugService scenarios exercising FORCE, TRACE, and BREAK commands over TCP (port 8080) with
trace telemetry on UDP (port 8081). The Go debugclient scripts a fixed command sequence and
verifies real acknowledgements. The debug_pause_resume variant additionally verifies that
PAUSE halts the RT loop and RESUME restarts it via live VALUE polling.
tcplogger
TCPLogger delivery: verifies that a triggered DebugService event produces a log line on the TCPLogger TCP port (8082/9090).
Validation Oracles
Each chain scenario specifies an oracle mode:
- analytic — ground truth is reconstructed from
gen_data.py's deterministic formulas (sine, ramp, counter, time_us, time_ns). No reference file needed. - fed — a second IOGAM branch in the MARTe config taps the same signals into a FileWriter ("tap file"). The validator compares recordings against this tap.
- both — both oracles are applied.
Per-signal checks (validate_waveform.py):
| Check | Description |
|---|---|
| Fidelity | Every received value within tolerance of some ground-truth value. Tolerance is 0 for raw integers, float epsilon for raw floats, quant_step/2 + 1e-6*range for quantised floats. |
| Shape | Sine signals (>= 8 points): least-squares fit of a*sin(wt)+b*cos(wt)+c. Requires correlation >= 0.99 and low normalised RMSE (relaxed by quant step). |
| Fed reference | When --tap is given, each received value must also match the tap. |
| Continuity | Flags inter-sample gaps > 10x median spacing. Fails when summed gap duration exceeds 5% of capture span. |
Client Checks
The Go chain-client (Test/E2E/suite/client/) performs behavioural checks specified per
scenario in client_checks:
| Check | What it verifies |
|---|---|
live |
WebSocket connection succeeds and live binary pushes arrive with monotonic timestamps. |
zoom |
A zoom WS command returns a valid binary response covering the requested time range. |
window |
A window WS command returns data within the specified time bounds. |
trigger |
A trigger WS command on the specified signal fires and returns data around the trigger point. |
Stress Matrix
The stress module (stress.py + stress_run.py) exercises capacity by sweeping one load axis
at a time:
| Axis | What is scaled |
|---|---|
| Signal size | Bytes per packet (array element count) |
| Signal count | Number of signals per source |
| Subscriber fan-out | Number of StreamHub instances subscribing to one UDPStreamer |
| WS client count | Parallel WebSocket clients on one StreamHub |
| Zoom request rate | Concurrent zoom queries per second per client |
Gates:
- Survival (hard) — neither server crashed or hung.
- Liveness (hard) — every client received monotonic, timestamped pushes.
- Peak RSS (soft) — MARTe and StreamHub memory stayed under case ceilings.
- Zoom p95 latency (soft) — round-trip zoom query latency under load.
Results are written to stress_results.json with axis/level for scaling-curve plots.
Artifacts
| Path | Content |
|---|---|
Build/x86-linux/E2E/chain/results.json |
Per-scenario status (PASS/FAIL/SKIP/XFAIL/XPASS) + waveform metrics |
Build/x86-linux/E2E/chain/report_data.json |
Full report data including regression diffs |
Build/x86-linux/E2E/chain/history.jsonl |
One-line-per-run headline metrics for trend tracking |
Build/x86-linux/E2E/chain/trend_*.png |
Pass-rate / coverage / fidelity / memory trend plots |
Build/x86-linux/E2E/chain/E2E_Report.pdf |
Compiled Typst PDF report |
Build/x86-linux/E2E/chain/unit_tests.json |
Per-suite test results (GTest, Go, Python) |
Build/x86-linux/E2E/chain/coverage.json |
Per-language coverage percentages |
Build/x86-linux/E2E/chain/stress/ |
Stress matrix results |
Build/x86-linux/E2E/chain/hub_<id>.log |
StreamHub stdout/stderr per scenario |
Build/x86-linux/E2E/chain/marte_<id>.log |
MARTe2 app stdout/stderr per scenario |
Build/x86-linux/E2E/chain/client_<id>.log |
Client stdout/stderr per scenario |
/tmp/chain_e2e/ |
Scratch: input binaries, configs, recordings, metrics, plots |
XFAIL / XPASS Handling
Scenarios may carry a known_issue marker (a human-readable string describing a documented,
not-yet-fixed chain gap). When present:
- A raw FAIL is reclassified as XFAIL (expected failure) — does not break the green baseline.
- A raw PASS becomes XPASS (unexpectedly fixed) — surfaced as a failure to prompt removal of the stale marker.
Overall status is PASS when there are no hard FAILs and no XPASSes.
Framework Files
| File | Role |
|---|---|
run_e2e.sh |
Top-level orchestrator (build, run scenarios, coverage, report) |
scenarios.py |
Declarative scenario matrix + validation |
gen_data.py |
Deterministic input binary generator |
gen_cfg.py |
MARTe2 + StreamHub config generator |
validate_waveform.py |
Waveform comparison (fidelity, shape, continuity) |
plots.py |
Per-scenario PNG figure renderer |
collect.py |
Unit test runner + coverage collector (GTest, Go, Python, lcov) |
report_build.py |
Report data consolidator + trend plots + history |
stress.py |
Declarative stress case matrix |
stress_run.py |
Stress matrix orchestrator |
proc_perf.py |
Live-process CPU/RSS snapshot from /proc |
E2E_Report.typ |
Typst template for the PDF report |
tests_py.py |
Python framework unit tests (python3 -m unittest tests_py) |
client/main.go |
Go chain-client (live record + zoom/window/trigger checks) |
debugclient/main.go |
Go debug/tcplogger client (command scripting + verification) |
Scenario Matrix
| ID | Kind | Description |
|---|---|---|
s01_scalar_uint32 |
chain | Single uint32 scalar counter, Strict unicast (type fidelity) |
s02_array_float32_fullarray |
chain | 100-elem float32 array, FullArray time mode, uint64 ns time array |
s03_quant_uint16 |
chain | float32 scalar quantised to uint16 over [-5,5], Strict unicast |
s04_int8_scalar |
chain | int8 scalar counter, type fidelity |
s05_uint8_scalar |
chain | uint8 scalar counter, type fidelity |
s06_int16_scalar |
chain | int16 scalar ramp, type fidelity |
s07_uint16_scalar |
chain | uint16 scalar ramp, type fidelity |
s08_int32_scalar |
chain | int32 scalar counter, type fidelity |
s09_int64_scalar |
chain | int64 scalar counter, type fidelity |
s10_uint64_scalar |
chain | uint64 scalar counter, type fidelity |
s11_float64_scalar |
chain | float64 scalar sine 5 Hz (double-precision path) |
s12_f32_arr8 |
chain | float32 8-elem array sine 5 Hz |
s13_f32_arr32 |
chain | float32 32-elem array sine 10 Hz |
s14_f64_arr64 |
chain | float64 64-elem array ramp |
s15_i16_arr16 |
chain | int16 16-elem array counter |
s16_f32_arr256 |
chain | float32 256-elem array sine 5 Hz (large frame) |
s17_lastsample |
chain | float32 8-elem LastSample, uint64 ns scalar anchor |
s18_firstsample |
chain | float32 8-elem FirstSample, uint32 us scalar anchor |
s19_fullarray_f64 |
chain | float64 50-elem FullArray sine 5 Hz, uint64 ns time |
s20_quant_uint8 |
chain | float32 scalar quant uint8 [-1,1] sine 5 Hz |
s21_quant_int8 |
chain | float32 scalar quant int8 [-10,10] sine 5 Hz |
s22_quant_int16 |
chain | float32 scalar quant int16 [-100,100] ramp |
s23_quant_f64_arr |
chain | float64 16-elem quant uint16 [-2,2] sine 5 Hz |
s24_accumulate |
chain | float32 scalar sine 5 Hz, Accumulate @50 Hz refresh |
s25_decimate4 |
chain | float32 scalar sine 5 Hz, Decimate ratio 4 |
s26_decimate10_arr |
chain | float32 8-elem counter, Decimate ratio 10 |
s27_frag_f64_128 |
chain | float64 128-elem ramp, MaxPayload 512 (fragmented) |
s28_frag_f32_100 |
chain | float32 100-elem sine 5 Hz, MaxPayload 256 (fragmented) |
s29_mcast_scalar |
chain | multicast float32 scalar sine 5 Hz |
s30_mcast_arr_fullarray |
chain | multicast float32 32-elem FullArray sine 5 Hz |
s31_two_src |
chain | two unicast sources: float32 sine + uint32 counter |
s32_three_src |
chain | three unicast sources: int16 ramp / float64 sine / uint8 counter |
s33_dec_arr_quant |
chain | Decimate 2 + 16-elem quant uint16 sine 5 Hz |
s34_acc_fullarray |
chain | Accumulate @100 Hz: accumulated scalar + 32-elem FullArray sine passenger |
s35_mcast_decimate |
chain | multicast + Decimate ratio 5, float32 scalar sine 5 Hz |
s36_big_frag_dec |
chain | float64 64-elem ramp, MaxPayload 256 + Decimate 4 |
s37_trig_ramp_i32 |
chain | trigger on int32 ramp scalar |
s38_trig_f64_sine |
chain | trigger on float64 sine 5 Hz scalar |
s39_uint8_arr32 |
chain | uint8 32-elem array counter (wrap fidelity) |
s40_int8_arr16 |
chain | int8 16-elem array counter (wrap fidelity) |
s41_f32_unit |
chain | float32 scalar ramp with Unit=V |
s42_f64_counter |
chain | float64 scalar counter (large integer values) |
s43_fullarray_quant |
chain | float32 16-elem FullArray quant uint16 sine 5 Hz |
s44_window_check |
chain | float32 sine 5 Hz scalar, window time-range check |
s45_decimate_multisig |
chain | Decimate ratio 2 over a 2-signal source |
s46_accumulate_arr |
chain | Accumulate @200 Hz: accumulated scalar sine + 16-elem array passenger |
s47_mcast_multisrc |
chain | multicast, two sources (scalar each) |
s48_f64_arr_big_payload |
chain | float64 100-elem ramp, MaxPayload 65490 (single frame) |
s49_mixed_quant_raw |
chain | one source: quant uint8 sine + raw float32 sine |
s50_trig_quant |
chain | trigger on quantised uint16 sine 10 Hz |
s51_8x1msps_100hz |
chain | 8x float32 10k-elem arrays @1 MSps, FirstSample, 100 Hz packets (~32 MB/s) |
s52_direct_unicast |
direct | Direct UDPStreamer->UDPStreamerClient round-trip, unicast |
s53_direct_multicast |
direct | Direct UDPStreamer->UDPStreamerClient round-trip, multicast |
s54_recorder |
recorder | StreamHub BinaryRecorder disk-output round-trip |
s55_debug_force_trace_break |
debug | DebugService FORCE/TRACE/BREAK over real TCP 8080 + UDP 8081 |
s56_tcplogger_delivery |
tcplogger | TCPLogger delivers a log line for a triggered DebugService event |
s57_debug_pause_resume |
debug_pause_resume | DebugService PAUSE/RESUME halts and resumes the RT loop, verified via live VALUE polling |
Coverage Goals
The chain scenario matrix is a curated covering set: every configurable UDPStreamer option value appears in at least one scenario:
- All 10 MARTe2 types: int8, uint8, int16, uint16, int32, uint32, int64, uint64, float32, float64
- Scalar and array shapes: elements 1, 8, 16, 32, 50, 64, 100, 128, 256, 1000, 10000
- All four TimeModes: PacketTime, FullArray, FirstSample, LastSample
- All five QuantizedTypes: none, uint8, int8, uint16, int16
- All three PublishingModes: Strict, Accumulate, Decimate
- Both network modes: unicast and multicast
- Fragmentation: small MaxPayloadSize forcing multi-fragment datagrams
- Multi-source: 1, 2, and 3 independent UDPStreamer feeds into one StreamHub
- High-risk interactions: decimate+quant+array, accumulate+fullarray, multicast+decimate, fragmentation+decimate, mixed quant+raw signals