# Unified test/E2E/reporting/coverage pipeline — design Date: 2026-07-01 ## Problem Test/coverage/reporting infrastructure is fragmented across at least seven independent entry points with no shared report: - `Test/E2E/chain/run_chain_e2e.sh` — the mature suite: 51-scenario matrix, `collect.py` (C++ GTest, C++ Integration, 3x Go, Python), lcov coverage, `report_build.py` → `report_data.json`/`history.jsonl`, Typst PDF. - `Test/E2E/chain/run_stress.sh` (+`stress.py`/`stress_run.py`) — a 27-case scaling matrix, fully standalone, own JSON/PNG output, no report/history integration. A design+plan+branch (`feature/stress-report-integration`) already exists for wiring it in but never landed on `main` and has since diverged (predates the `UDPStreamerClient` test dir and `DebugServiceGTest` additions). - `Test/E2E/streamhub/` (Go smoke test) + root `run_e2e_test.sh` — a strict subset of what `Test/E2E/chain/client` already checks. - `Test/E2E/datasources/` (+`run_e2e_report.sh`, its own `E2E_Report.typ`) — narrower UDPStreamer↔UDPStreamerClient direct round-trip predecessor to the chain suite; has a stale synthetic (`np.random`) latency histogram in its report generation. - `Test/E2E/recorder/` (+`run_recorder_e2e.sh`) — BinaryRecorder disk-output check, depends on `datasources/`'s generator/validator by relative path. - `run_combined_test.sh` — manual, no-assertion demo launcher for the DebugService + TCPLogger + UDPStreamer combined config. - TCPLogger has **zero** automated tests of any kind today. Additionally, `Test/Applications/StreamHub`'s build is currently broken: two untracked GTest files (`BoundsCheckTest.cpp`, `WSServerBufferTest.cpp` — regression tests for `BUG_FIX_PLAN.md` items HI-1/HI-4/CR-1) `#include `, which fails under the project's `-std=c++98` build even during a bare `-MM` dependency scan (MARTe2's generic `dependsRaw` rule globs *all* `.cpp` files in a test dir regardless of `OBJSX` membership). This currently breaks `make test` repo-wide. Coverage is also incomplete: it's rebuilt and collected *after* E2E scenarios already ran on normal (non-instrumented) binaries, so none of the code paths scenarios actually exercise count toward the C++ coverage percentage. ## Goal One command, one report. Running `Test/E2E/suite/run_e2e.sh` (renamed from `run_chain_e2e.sh`) executes every test suite in the repo — unit tests (GTest, Integration), the chain/stress/datasources/recorder/debug/tcplogger E2E scenario families, and Go/Python framework tests — and produces one `results.json`, one `unit_tests.json`/`coverage.json`, and one `report_data.json`/`history.jsonl`/Typst PDF covering all of it, with accurate coverage (including E2E-exercised code) and uncontaminated performance metrics. ## Non-goals - No changes to the actual product code paths being tested (this is test-infrastructure-only), except the minimal `` → MARTe2-types fix needed to unblock the build. - No new CI wiring (out of scope; this only needs to produce artifacts a CI job *could* consume later). - No decision here about merging/resolving `BUG_FIX_PLAN.md`'s broader remediation phases — only unblocking the two orphaned test files enough to compile and run as part of the unified suite. ## Design ### 1. Layout and entry point - `Test/E2E/chain/` → `Test/E2E/suite/` (git `mv`), `run_chain_e2e.sh` → `run_e2e.sh`. `CLAUDE.md`/`AGENTS.md` references updated. - New flags on `run_e2e.sh`: `--skip-coverage`, `--skip-stress`, `--skip-datasources`, `--skip-recorder`, `--skip-debug`, `--skip-tcplogger` (all six suites run by default; existing `--skip-build`/`--only`/`--pdf-only`/`--cpp-coverage` retained, `--cpp-coverage` now governs the single instrumented phase below instead of a second rebuild). - `Test/E2E/datasources/` and `Test/E2E/recorder/` directories stay in place (cfgs, `gen_test_data.py`, `validate_binary.py`); `run_e2e.sh` imports them as Python modules (same pattern it already uses for `scenarios`/ `gen_data`/`gen_cfg`/`validate_waveform`). - Deleted: `Test/E2E/streamhub/` (dir + Go module/binary), root `run_e2e_test.sh`, `Test/E2E/datasources/run_e2e_report.sh` + its standalone `E2E_Report.typ`, `Test/E2E/recorder/run_recorder_e2e.sh`, root `run_combined_test.sh`. ### 2. Build & coverage flow Flow when coverage is enabled (default): 1. Build **normal** binaries (`make core apps` + test dirs, no `--coverage`), unless `--skip-build`. 2. **Authoritative pass**: run chain + datasources + recorder + debug + tcplogger scenarios (any not individually skipped) on normal binaries. This is the sole source of `results.json` (pass/fail/oracle) and `perf_*.json` (cpu/RSS/throughput — unchanged from today, feeds `trend_perf.png`/regression). 3. Run the **stress matrix** (unless `--skip-stress`) on the same normal binaries. Stress is *never* instrumented, regardless of `--skip-coverage`, because gcov overhead skews its scaling/perf measurements. 4. Rebuild **instrumented** binaries (`--coverage`): `core`, `apps`, and test dirs (`Test/GTest`, `Test/Integration`, `Test/Components/DataSources/ UDPStreamer{,Client}`, `Test/Applications/StreamHub`). 5. Re-run chain + datasources + recorder + debug + tcplogger scenarios on instrumented binaries for **coverage accumulation only**. Their pass/fail is compared against step 2's results as a sanity check (a mismatch is logged as a warning, not a report-blocking failure — it flags nondeterminism without duplicating the correctness gate); their perf metrics are discarded. 6. Run C++ GTest + Integration (instrumented) and the Go/Python suites (unaffected by instrumentation, run once). 7. Collect coverage once (`lcov --capture` + `--extract Source/* Test/*`, same restricted scope as today), aggregating gcov data from steps 5 and 6. 8. Restore a clean, non-instrumented `core apps` build so the repo ends in its normal runnable state. When `--skip-coverage`: steps 4/5/7 are skipped; step 6 runs GTest/ Integration once on normal binaries; `coverage.json`/the report's coverage section is marked `"skipped": true`. ### 3. Unified scenario model `scenarios.py` entries gain a `kind` field: - `"chain"` — the existing 51 scenarios, unchanged. - `"direct"` — new; the 2 existing datasources cfgs (`E2ETest.cfg`, `E2EMulticastTest.cfg`), UDPStreamer→UDPStreamerClient only, no StreamHub/WS hop, validated via `validate_binary.py` (byte-identical round-trip). - `"recorder"` — new; `RecorderStreamer.cfg`/`StreamHubRec.cfg`, chain path + `BinaryRecorder`, validated via `validate_binary.py` against the recorded disk file. - `"debug"` — new; see §4. - `"tcplogger"` — new; see §4. `run_e2e.sh`'s per-scenario loop dispatches on `kind`: `chain` keeps today's launch/record/validate sequence; `direct` skips the StreamHub/chain-client hop (launch `MARTeApp.ex` with `UDPStreamerClient` receiving directly, then byte-compare); `recorder` runs the chain sequence plus a post-run disk-file check; `debug`/`tcplogger` launch `MARTeApp.ex` with a DebugService/TCPLogger config and drive it with `debugclient` (§4). All five write into the same `results.json` (with `kind` as a discriminator field), so `known_issue`/ XFAIL/XPASS handling and PASS/FAIL aggregation need no branching beyond dispatch. Stress stays structurally separate (`stress_results.json`, produced by `stress_run.py` as today, read by `report_build.py` as a distinct input — the same pattern already used for `perf_*.json`), because it's a scaling matrix, not a per-signal pass/fail scenario. ### 4. DebugService & TCPLogger E2E scenarios `Client/debugger/martecontrol.go`'s `MarteController` already implements the needed protocol clients as library code: `Connect`/`runTCP`/`SendCommand` (TCP 8080 command protocol), `runDebugUDP` (UDP 8081 trace telemetry), and `runLog` (TCP 8082 TCPLogger stream) — currently wired to a WebSocket hub for the browser UI. Refactor the hub-broadcast calls behind a small callback interface so this logic is reusable headless, and add a new Go binary `Test/E2E/suite/debugclient/` that imports it, scripted instead of browser-driven. - **`kind="debug"`**: launch `MARTeApp.ex` with a DebugService config derived from `Test/Configurations/combined_test.cfg` (trimmed to what's needed). `debugclient` connects to TCP 8080 + UDP 8081 and scripts a sequence (force a signal, arm a breakpoint, trace a signal) mirroring what `Test/Integration`'s `TraceTest.cpp`/`ValidationTest.cpp` already exercise in-process — but validated end-to-end as a separate OS process pair over real sockets, catching wire-format/serialization bugs the in-process suite cannot. - **`kind="tcplogger"`**: same launch, `debugclient` connects to TCP 8082, triggers known log-worthy events (a forced value, a config error), asserts expected log lines arrive with correct formatting/ordering. - Both are functional (pass/fail), not perf-sensitive: they follow the chain/datasources/recorder double-run pattern from §2 for coverage consistency, but skip `proc_perf.py` capture. - `run_combined_test.sh` becomes redundant once these scenarios exist and is retired, folding its config into the new scenario cfgs. ### 5. Reporting `E2E_Report.typ`/`report_data.json` gain sections alongside the existing Progression/Unit tests/Coverage/Performance/Scenarios: **Stress Tests** (per-axis scaling tables/plots + regression, per the existing `2026-06-26-stress-suite-report-integration-design.md`), **Direct Round-Trip** (datasources pass/fail table), **Recorder** (pass/fail table), **Debug Service E2E** and **TCPLogger E2E** (pass/fail per scenario). `history.jsonl`/`headline()`/`regression()` extended with fields from each new section so trend plots and regression-vs-previous-run cover everything in one place. ### 6. `Test/Applications/StreamHub` build fix `BoundsCheckTest.cpp`/`WSServerBufferTest.cpp` replace `#include ` with MARTe2 types (`uint8`/`uint32`/etc. from `GeneralDefinitions.h`), matching the sibling GTest files' style and the repo's MARTe2-style convention. Both are registered in `Makefile.inc`'s `OBJSX` (as `BoundsCheckTest.x`/`WSServerBufferTest.x`) alongside the existing four GTest `.x` entries, so they compile deliberately rather than only being incidentally scanned by the `-MM` dependency glob. ### 7. Stale branch reconciliation `git diff main...feature/stress-report-integration` shows a concrete, mostly-still-relevant 8-commit diff (`report_build.py` +124/-lines, `run_chain_e2e.sh` +27, `stress.py` +69 for the multi-fragment size-axis extension, plus a new `test_report_stress.py`, a `CLAUDE.md` doc line, and a stray `TODO.md`). Port the relevant logic (stress table/plots/regression wiring, the multi-fragment size-axis fix) onto current `main` as part of implementing §5, adapting for the scenario-model/build-flow changes in this spec (the branch predates both). Once superseded, delete the branch (local + `origin`) — confirm before deleting the remote ref. ## Testing - All 76 existing GTests plus the two newly-registered ones must pass. - Full `run_e2e.sh` (default flags) must complete with `results.json` reporting pass for all scenario kinds, `stress_results.json` produced, `coverage.json` showing a higher C++ % than today's GTest/Integration-only baseline, and a single `E2E_Report.pdf` containing all six sections. - `run_e2e.sh --skip-coverage --skip-stress --only ` must still work for fast single-scenario iteration (unchanged developer workflow). - `Test/E2E/chain/tests_py.py` (framework self-tests) extended to cover the new `kind` dispatch logic and the `direct`/`recorder`/`debug`/`tcplogger` scenario definitions the same way it already covers `chain` scenarios. ## Risks - Double-running chain/datasources/recorder/debug/tcplogger scenarios for coverage roughly doubles that portion of total E2E wall-clock time (stress is unaffected, running once). - Refactoring `MarteController` to be headless-usable touches `Client/debugger`'s production code, not just test infra — needs care not to regress the browser UI. - The `feature/stress-report-integration` branch's `stress.py` multi-fragment extension needs re-validation against the current `UDPSClient` 1 MiB reassembly cap mentioned in its own design doc, since main has moved on since that branch diverged.