# Unified test/E2E/reporting/coverage pipeline — design

Date: 2026-07-01

## Problem

Test/coverage/reporting infrastructure is fragmented across at least seven
independent entry points with no shared report:

- `Test/E2E/chain/run_chain_e2e.sh` — the mature suite: 51-scenario matrix,
  `collect.py` (C++ GTest, C++ Integration, 3x Go, Python), lcov coverage,
  `report_build.py` → `report_data.json`/`history.jsonl`, Typst PDF.
- `Test/E2E/chain/run_stress.sh` (+`stress.py`/`stress_run.py`) — a 27-case
  scaling matrix, fully standalone, own JSON/PNG output, no report/history
  integration. A design+plan+branch (`feature/stress-report-integration`)
  already exists for wiring it in but never landed on `main` and has since
  diverged (predates the `UDPStreamerClient` test dir and `DebugServiceGTest`
  additions).
- `Test/E2E/streamhub/` (Go smoke test) + root `run_e2e_test.sh` — a strict
  subset of what `Test/E2E/chain/client` already checks.
- `Test/E2E/datasources/` (+`run_e2e_report.sh`, its own `E2E_Report.typ`) —
  narrower UDPStreamer↔UDPStreamerClient direct round-trip predecessor to the
  chain suite; has a stale synthetic (`np.random`) latency histogram in its
  report generation.
- `Test/E2E/recorder/` (+`run_recorder_e2e.sh`) — BinaryRecorder disk-output
  check, depends on `datasources/`'s generator/validator by relative path.
- `run_combined_test.sh` — manual, no-assertion demo launcher for the
  DebugService + TCPLogger + UDPStreamer combined config.
- TCPLogger has **zero** automated tests of any kind today.

Additionally, `Test/Applications/StreamHub`'s build is currently broken:
two untracked GTest files (`BoundsCheckTest.cpp`, `WSServerBufferTest.cpp` —
regression tests for `BUG_FIX_PLAN.md` items HI-1/HI-4/CR-1) `#include
<cstdint>`, which fails under the project's `-std=c++98` build even during a
bare `-MM` dependency scan (MARTe2's generic `dependsRaw` rule globs *all*
`.cpp` files in a test dir regardless of `OBJSX` membership). This currently
breaks `make test` repo-wide.

Coverage is also incomplete: it's rebuilt and collected *after* E2E scenarios
already ran on normal (non-instrumented) binaries, so none of the code paths
scenarios actually exercise count toward the C++ coverage percentage.

## Goal

One command, one report. Running `Test/E2E/suite/run_e2e.sh` (renamed from
`run_chain_e2e.sh`) executes every test suite in the repo — unit tests
(GTest, Integration), the chain/stress/datasources/recorder/debug/tcplogger
E2E scenario families, and Go/Python framework tests — and produces one
`results.json`, one `unit_tests.json`/`coverage.json`, and one
`report_data.json`/`history.jsonl`/Typst PDF covering all of it, with
accurate coverage (including E2E-exercised code) and uncontaminated
performance metrics.

## Non-goals

- No changes to the actual product code paths being tested (this is
  test-infrastructure-only), except the minimal `<cstdint>` → MARTe2-types
  fix needed to unblock the build.
- No new CI wiring (out of scope; this only needs to produce artifacts a CI
  job *could* consume later).
- No decision here about merging/resolving `BUG_FIX_PLAN.md`'s broader
  remediation phases — only unblocking the two orphaned test files enough to
  compile and run as part of the unified suite.

## Design

### 1. Layout and entry point

- `Test/E2E/chain/` → `Test/E2E/suite/` (git `mv`), `run_chain_e2e.sh` →
  `run_e2e.sh`. `CLAUDE.md`/`AGENTS.md` references updated.
- New flags on `run_e2e.sh`: `--skip-coverage`, `--skip-stress`,
  `--skip-datasources`, `--skip-recorder`, `--skip-debug`,
  `--skip-tcplogger` (all six suites run by default; existing
  `--skip-build`/`--only`/`--pdf-only`/`--cpp-coverage` retained,
  `--cpp-coverage` now governs the single instrumented phase below instead
  of a second rebuild).
- `Test/E2E/datasources/` and `Test/E2E/recorder/` directories stay in place
  (cfgs, `gen_test_data.py`, `validate_binary.py`); `run_e2e.sh` imports them
  as Python modules (same pattern it already uses for `scenarios`/
  `gen_data`/`gen_cfg`/`validate_waveform`).
- Deleted: `Test/E2E/streamhub/` (dir + Go module/binary), root
  `run_e2e_test.sh`, `Test/E2E/datasources/run_e2e_report.sh` + its
  standalone `E2E_Report.typ`, `Test/E2E/recorder/run_recorder_e2e.sh`, root
  `run_combined_test.sh`.

### 2. Build & coverage flow

Flow when coverage is enabled (default):

1. Build **normal** binaries (`make core apps` + test dirs, no
   `--coverage`), unless `--skip-build`.
2. **Authoritative pass**: run chain + datasources + recorder + debug +
   tcplogger scenarios (any not individually skipped) on normal binaries.
   This is the sole source of `results.json` (pass/fail/oracle) and
   `perf_*.json` (cpu/RSS/throughput — unchanged from today, feeds
   `trend_perf.png`/regression).
3. Run the **stress matrix** (unless `--skip-stress`) on the same normal
   binaries. Stress is *never* instrumented, regardless of
   `--skip-coverage`, because gcov overhead skews its scaling/perf
   measurements.
4. Rebuild **instrumented** binaries (`--coverage`): `core`, `apps`, and test
   dirs (`Test/GTest`, `Test/Integration`, `Test/Components/DataSources/
   UDPStreamer{,Client}`, `Test/Applications/StreamHub`).
5. Re-run chain + datasources + recorder + debug + tcplogger scenarios on
   instrumented binaries for **coverage accumulation only**. Their pass/fail
   is compared against step 2's results as a sanity check (a mismatch is
   logged as a warning, not a report-blocking failure — it flags
   nondeterminism without duplicating the correctness gate); their perf
   metrics are discarded.
6. Run C++ GTest + Integration (instrumented) and the Go/Python suites
   (unaffected by instrumentation, run once).
7. Collect coverage once (`lcov --capture` + `--extract Source/* Test/*`,
   same restricted scope as today), aggregating gcov data from steps 5 and 6.
8. Restore a clean, non-instrumented `core apps` build so the repo ends in
   its normal runnable state.

When `--skip-coverage`: steps 4/5/7 are skipped; step 6 runs GTest/
Integration once on normal binaries; `coverage.json`/the report's coverage
section is marked `"skipped": true`.

### 3. Unified scenario model

`scenarios.py` entries gain a `kind` field:

- `"chain"` — the existing 51 scenarios, unchanged.
- `"direct"` — new; the 2 existing datasources cfgs (`E2ETest.cfg`,
  `E2EMulticastTest.cfg`), UDPStreamer→UDPStreamerClient only, no
  StreamHub/WS hop, validated via `validate_binary.py` (byte-identical
  round-trip).
- `"recorder"` — new; `RecorderStreamer.cfg`/`StreamHubRec.cfg`, chain path
  + `BinaryRecorder`, validated via `validate_binary.py` against the
  recorded disk file.
- `"debug"` — new; see §4.
- `"tcplogger"` — new; see §4.

`run_e2e.sh`'s per-scenario loop dispatches on `kind`: `chain` keeps today's
launch/record/validate sequence; `direct` skips the StreamHub/chain-client
hop (launch `MARTeApp.ex` with `UDPStreamerClient` receiving directly, then
byte-compare); `recorder` runs the chain sequence plus a post-run disk-file
check; `debug`/`tcplogger` launch `MARTeApp.ex` with a DebugService/TCPLogger
config and drive it with `debugclient` (§4). All five write into the same
`results.json` (with `kind` as a discriminator field), so `known_issue`/
XFAIL/XPASS handling and PASS/FAIL aggregation need no branching beyond
dispatch.

Stress stays structurally separate (`stress_results.json`, produced by
`stress_run.py` as today, read by `report_build.py` as a distinct input —
the same pattern already used for `perf_*.json`), because it's a scaling
matrix, not a per-signal pass/fail scenario.

### 4. DebugService & TCPLogger E2E scenarios

`Client/debugger/martecontrol.go`'s `MarteController` already implements the
needed protocol clients as library code: `Connect`/`runTCP`/`SendCommand`
(TCP 8080 command protocol), `runDebugUDP` (UDP 8081 trace telemetry), and
`runLog` (TCP 8082 TCPLogger stream) — currently wired to a WebSocket hub for
the browser UI. Refactor the hub-broadcast calls behind a small callback
interface so this logic is reusable headless, and add a new Go binary
`Test/E2E/suite/debugclient/` that imports it, scripted instead of
browser-driven.

- **`kind="debug"`**: launch `MARTeApp.ex` with a DebugService config derived
  from `Test/Configurations/combined_test.cfg` (trimmed to what's needed).
  `debugclient` connects to TCP 8080 + UDP 8081 and scripts a sequence
  (force a signal, arm a breakpoint, trace a signal) mirroring what
  `Test/Integration`'s `TraceTest.cpp`/`ValidationTest.cpp` already exercise
  in-process — but validated end-to-end as a separate OS process pair over
  real sockets, catching wire-format/serialization bugs the in-process suite
  cannot.
- **`kind="tcplogger"`**: same launch, `debugclient` connects to TCP 8082,
  triggers known log-worthy events (a forced value, a config error), asserts
  expected log lines arrive with correct formatting/ordering.
- Both are functional (pass/fail), not perf-sensitive: they follow the
  chain/datasources/recorder double-run pattern from §2 for coverage
  consistency, but skip `proc_perf.py` capture.
- `run_combined_test.sh` becomes redundant once these scenarios exist and is
  retired, folding its config into the new scenario cfgs.

### 5. Reporting

`E2E_Report.typ`/`report_data.json` gain sections alongside the existing
Progression/Unit tests/Coverage/Performance/Scenarios: **Stress Tests**
(per-axis scaling tables/plots + regression, per the existing
`2026-06-26-stress-suite-report-integration-design.md`), **Direct
Round-Trip** (datasources pass/fail table), **Recorder** (pass/fail table),
**Debug Service E2E** and **TCPLogger E2E** (pass/fail per scenario).
`history.jsonl`/`headline()`/`regression()` extended with fields from each
new section so trend plots and regression-vs-previous-run cover everything
in one place.

### 6. `Test/Applications/StreamHub` build fix

`BoundsCheckTest.cpp`/`WSServerBufferTest.cpp` replace `#include <cstdint>`
with MARTe2 types (`uint8`/`uint32`/etc. from `GeneralDefinitions.h`),
matching the sibling GTest files' style and the repo's MARTe2-style
convention. Both are registered in `Makefile.inc`'s `OBJSX` (as
`BoundsCheckTest.x`/`WSServerBufferTest.x`) alongside the existing four
GTest `.x` entries, so they compile deliberately rather than only being
incidentally scanned by the `-MM` dependency glob.

### 7. Stale branch reconciliation

`git diff main...feature/stress-report-integration` shows a concrete,
mostly-still-relevant 8-commit diff (`report_build.py` +124/-lines,
`run_chain_e2e.sh` +27, `stress.py` +69 for the multi-fragment size-axis
extension, plus a new `test_report_stress.py`, a `CLAUDE.md` doc line, and a
stray `TODO.md`). Port the relevant logic (stress table/plots/regression
wiring, the multi-fragment size-axis fix) onto current `main` as part of
implementing §5, adapting for the scenario-model/build-flow changes in this
spec (the branch predates both). Once superseded, delete the branch (local +
`origin`) — confirm before deleting the remote ref.

## Testing

- All 76 existing GTests plus the two newly-registered ones must pass.
- Full `run_e2e.sh` (default flags) must complete with `results.json`
  reporting pass for all scenario kinds, `stress_results.json` produced,
  `coverage.json` showing a higher C++ % than today's GTest/Integration-only
  baseline, and a single `E2E_Report.pdf` containing all six sections.
- `run_e2e.sh --skip-coverage --skip-stress --only <id>` must still work for
  fast single-scenario iteration (unchanged developer workflow).
- `Test/E2E/chain/tests_py.py` (framework self-tests) extended to cover the
  new `kind` dispatch logic and the `direct`/`recorder`/`debug`/`tcplogger`
  scenario definitions the same way it already covers `chain` scenarios.

## Risks

- Double-running chain/datasources/recorder/debug/tcplogger scenarios for
  coverage roughly doubles that portion of total E2E wall-clock time (stress
  is unaffected, running once).
- Refactoring `MarteController` to be headless-usable touches
  `Client/debugger`'s production code, not just test infra — needs care not
  to regress the browser UI.
- The `feature/stress-report-integration` branch's `stress.py` multi-fragment
  extension needs re-validation against the current `UDPSClient` 1 MiB
  reassembly cap mentioned in its own design doc, since main has moved on
  since that branch diverged.
