docs(testing): design for unified test/E2E/reporting/coverage pipeline

Consolidates the fragmented chain/stress/streamhub/datasources/recorder E2E suites, unit-test collection, and coverage into one entry point and one report, adds new DebugService/TCPLogger E2E coverage, and fixes the Test/Applications/StreamHub build break blocking `make test`. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-07-01 18:14:09 +02:00
parent 0bea41f866
commit dcaa466736
1 changed files with 237 additions and 0 deletions
@@ -0,0 +1,237 @@
+# Unified test/E2E/reporting/coverage pipeline — design
+
+Date: 2026-07-01
+
+## Problem
+
+Test/coverage/reporting infrastructure is fragmented across at least seven
+independent entry points with no shared report:
+
+- `Test/E2E/chain/run_chain_e2e.sh` — the mature suite: 51-scenario matrix,
+  `collect.py` (C++ GTest, C++ Integration, 3x Go, Python), lcov coverage,
+  `report_build.py` → `report_data.json`/`history.jsonl`, Typst PDF.
+- `Test/E2E/chain/run_stress.sh` (+`stress.py`/`stress_run.py`) — a 27-case
+  scaling matrix, fully standalone, own JSON/PNG output, no report/history
+  integration. A design+plan+branch (`feature/stress-report-integration`)
+  already exists for wiring it in but never landed on `main` and has since
+  diverged (predates the `UDPStreamerClient` test dir and `DebugServiceGTest`
+  additions).
+- `Test/E2E/streamhub/` (Go smoke test) + root `run_e2e_test.sh` — a strict
+  subset of what `Test/E2E/chain/client` already checks.
+- `Test/E2E/datasources/` (+`run_e2e_report.sh`, its own `E2E_Report.typ`) —
+  narrower UDPStreamer↔UDPStreamerClient direct round-trip predecessor to the
+  chain suite; has a stale synthetic (`np.random`) latency histogram in its
+  report generation.
+- `Test/E2E/recorder/` (+`run_recorder_e2e.sh`) — BinaryRecorder disk-output
+  check, depends on `datasources/`'s generator/validator by relative path.
+- `run_combined_test.sh` — manual, no-assertion demo launcher for the
+  DebugService + TCPLogger + UDPStreamer combined config.
+- TCPLogger has **zero** automated tests of any kind today.
+
+Additionally, `Test/Applications/StreamHub`'s build is currently broken:
+two untracked GTest files (`BoundsCheckTest.cpp`, `WSServerBufferTest.cpp` —
+regression tests for `BUG_FIX_PLAN.md` items HI-1/HI-4/CR-1) `#include
+<cstdint>`, which fails under the project's `-std=c++98` build even during a
+bare `-MM` dependency scan (MARTe2's generic `dependsRaw` rule globs *all*
+`.cpp` files in a test dir regardless of `OBJSX` membership). This currently
+breaks `make test` repo-wide.
+
+Coverage is also incomplete: it's rebuilt and collected *after* E2E scenarios
+already ran on normal (non-instrumented) binaries, so none of the code paths
+scenarios actually exercise count toward the C++ coverage percentage.
+
+## Goal
+
+One command, one report. Running `Test/E2E/suite/run_e2e.sh` (renamed from
+`run_chain_e2e.sh`) executes every test suite in the repo — unit tests
+(GTest, Integration), the chain/stress/datasources/recorder/debug/tcplogger
+E2E scenario families, and Go/Python framework tests — and produces one
+`results.json`, one `unit_tests.json`/`coverage.json`, and one
+`report_data.json`/`history.jsonl`/Typst PDF covering all of it, with
+accurate coverage (including E2E-exercised code) and uncontaminated
+performance metrics.
+
+## Non-goals
+
+- No changes to the actual product code paths being tested (this is
+  test-infrastructure-only), except the minimal `<cstdint>` → MARTe2-types
+  fix needed to unblock the build.
+- No new CI wiring (out of scope; this only needs to produce artifacts a CI
+  job *could* consume later).
+- No decision here about merging/resolving `BUG_FIX_PLAN.md`'s broader
+  remediation phases — only unblocking the two orphaned test files enough to
+  compile and run as part of the unified suite.
+
+## Design
+
+### 1. Layout and entry point
+
+- `Test/E2E/chain/` → `Test/E2E/suite/` (git `mv`), `run_chain_e2e.sh` →
+  `run_e2e.sh`. `CLAUDE.md`/`AGENTS.md` references updated.
+- New flags on `run_e2e.sh`: `--skip-coverage`, `--skip-stress`,
+  `--skip-datasources`, `--skip-recorder`, `--skip-debug`,
+  `--skip-tcplogger` (all six suites run by default; existing
+  `--skip-build`/`--only`/`--pdf-only`/`--cpp-coverage` retained,
+  `--cpp-coverage` now governs the single instrumented phase below instead
+  of a second rebuild).
+- `Test/E2E/datasources/` and `Test/E2E/recorder/` directories stay in place
+  (cfgs, `gen_test_data.py`, `validate_binary.py`); `run_e2e.sh` imports them
+  as Python modules (same pattern it already uses for `scenarios`/
+  `gen_data`/`gen_cfg`/`validate_waveform`).
+- Deleted: `Test/E2E/streamhub/` (dir + Go module/binary), root
+  `run_e2e_test.sh`, `Test/E2E/datasources/run_e2e_report.sh` + its
+  standalone `E2E_Report.typ`, `Test/E2E/recorder/run_recorder_e2e.sh`, root
+  `run_combined_test.sh`.
+
+### 2. Build & coverage flow
+
+Flow when coverage is enabled (default):
+
+1. Build **normal** binaries (`make core apps` + test dirs, no
+   `--coverage`), unless `--skip-build`.
+2. **Authoritative pass**: run chain + datasources + recorder + debug +
+   tcplogger scenarios (any not individually skipped) on normal binaries.
+   This is the sole source of `results.json` (pass/fail/oracle) and
+   `perf_*.json` (cpu/RSS/throughput — unchanged from today, feeds
+   `trend_perf.png`/regression).
+3. Run the **stress matrix** (unless `--skip-stress`) on the same normal
+   binaries. Stress is *never* instrumented, regardless of
+   `--skip-coverage`, because gcov overhead skews its scaling/perf
+   measurements.
+4. Rebuild **instrumented** binaries (`--coverage`): `core`, `apps`, and test
+   dirs (`Test/GTest`, `Test/Integration`, `Test/Components/DataSources/
+   UDPStreamer{,Client}`, `Test/Applications/StreamHub`).
+5. Re-run chain + datasources + recorder + debug + tcplogger scenarios on
+   instrumented binaries for **coverage accumulation only**. Their pass/fail
+   is compared against step 2's results as a sanity check (a mismatch is
+   logged as a warning, not a report-blocking failure — it flags
+   nondeterminism without duplicating the correctness gate); their perf
+   metrics are discarded.
+6. Run C++ GTest + Integration (instrumented) and the Go/Python suites
+   (unaffected by instrumentation, run once).
+7. Collect coverage once (`lcov --capture` + `--extract Source/* Test/*`,
+   same restricted scope as today), aggregating gcov data from steps 5 and 6.
+8. Restore a clean, non-instrumented `core apps` build so the repo ends in
+   its normal runnable state.
+
+When `--skip-coverage`: steps 4/5/7 are skipped; step 6 runs GTest/
+Integration once on normal binaries; `coverage.json`/the report's coverage
+section is marked `"skipped": true`.
+
+### 3. Unified scenario model
+
+`scenarios.py` entries gain a `kind` field:
+
+- `"chain"` — the existing 51 scenarios, unchanged.
+- `"direct"` — new; the 2 existing datasources cfgs (`E2ETest.cfg`,
+  `E2EMulticastTest.cfg`), UDPStreamer→UDPStreamerClient only, no
+  StreamHub/WS hop, validated via `validate_binary.py` (byte-identical
+  round-trip).
+- `"recorder"` — new; `RecorderStreamer.cfg`/`StreamHubRec.cfg`, chain path
+  + `BinaryRecorder`, validated via `validate_binary.py` against the
+  recorded disk file.
+- `"debug"` — new; see §4.
+- `"tcplogger"` — new; see §4.
+
+`run_e2e.sh`'s per-scenario loop dispatches on `kind`: `chain` keeps today's
+launch/record/validate sequence; `direct` skips the StreamHub/chain-client
+hop (launch `MARTeApp.ex` with `UDPStreamerClient` receiving directly, then
+byte-compare); `recorder` runs the chain sequence plus a post-run disk-file
+check; `debug`/`tcplogger` launch `MARTeApp.ex` with a DebugService/TCPLogger
+config and drive it with `debugclient` (§4). All five write into the same
+`results.json` (with `kind` as a discriminator field), so `known_issue`/
+XFAIL/XPASS handling and PASS/FAIL aggregation need no branching beyond
+dispatch.
+
+Stress stays structurally separate (`stress_results.json`, produced by
+`stress_run.py` as today, read by `report_build.py` as a distinct input —
+the same pattern already used for `perf_*.json`), because it's a scaling
+matrix, not a per-signal pass/fail scenario.
+
+### 4. DebugService & TCPLogger E2E scenarios
+
+`Client/debugger/martecontrol.go`'s `MarteController` already implements the
+needed protocol clients as library code: `Connect`/`runTCP`/`SendCommand`
+(TCP 8080 command protocol), `runDebugUDP` (UDP 8081 trace telemetry), and
+`runLog` (TCP 8082 TCPLogger stream) — currently wired to a WebSocket hub for
+the browser UI. Refactor the hub-broadcast calls behind a small callback
+interface so this logic is reusable headless, and add a new Go binary
+`Test/E2E/suite/debugclient/` that imports it, scripted instead of
+browser-driven.
+
+- **`kind="debug"`**: launch `MARTeApp.ex` with a DebugService config derived
+  from `Test/Configurations/combined_test.cfg` (trimmed to what's needed).
+  `debugclient` connects to TCP 8080 + UDP 8081 and scripts a sequence
+  (force a signal, arm a breakpoint, trace a signal) mirroring what
+  `Test/Integration`'s `TraceTest.cpp`/`ValidationTest.cpp` already exercise
+  in-process — but validated end-to-end as a separate OS process pair over
+  real sockets, catching wire-format/serialization bugs the in-process suite
+  cannot.
+- **`kind="tcplogger"`**: same launch, `debugclient` connects to TCP 8082,
+  triggers known log-worthy events (a forced value, a config error), asserts
+  expected log lines arrive with correct formatting/ordering.
+- Both are functional (pass/fail), not perf-sensitive: they follow the
+  chain/datasources/recorder double-run pattern from §2 for coverage
+  consistency, but skip `proc_perf.py` capture.
+- `run_combined_test.sh` becomes redundant once these scenarios exist and is
+  retired, folding its config into the new scenario cfgs.
+
+### 5. Reporting
+
+`E2E_Report.typ`/`report_data.json` gain sections alongside the existing
+Progression/Unit tests/Coverage/Performance/Scenarios: **Stress Tests**
+(per-axis scaling tables/plots + regression, per the existing
+`2026-06-26-stress-suite-report-integration-design.md`), **Direct
+Round-Trip** (datasources pass/fail table), **Recorder** (pass/fail table),
+**Debug Service E2E** and **TCPLogger E2E** (pass/fail per scenario).
+`history.jsonl`/`headline()`/`regression()` extended with fields from each
+new section so trend plots and regression-vs-previous-run cover everything
+in one place.
+
+### 6. `Test/Applications/StreamHub` build fix
+
+`BoundsCheckTest.cpp`/`WSServerBufferTest.cpp` replace `#include <cstdint>`
+with MARTe2 types (`uint8`/`uint32`/etc. from `GeneralDefinitions.h`),
+matching the sibling GTest files' style and the repo's MARTe2-style
+convention. Both are registered in `Makefile.inc`'s `OBJSX` (as
+`BoundsCheckTest.x`/`WSServerBufferTest.x`) alongside the existing four
+GTest `.x` entries, so they compile deliberately rather than only being
+incidentally scanned by the `-MM` dependency glob.
+
+### 7. Stale branch reconciliation
+
+`git diff main...feature/stress-report-integration` shows a concrete,
+mostly-still-relevant 8-commit diff (`report_build.py` +124/-lines,
+`run_chain_e2e.sh` +27, `stress.py` +69 for the multi-fragment size-axis
+extension, plus a new `test_report_stress.py`, a `CLAUDE.md` doc line, and a
+stray `TODO.md`). Port the relevant logic (stress table/plots/regression
+wiring, the multi-fragment size-axis fix) onto current `main` as part of
+implementing §5, adapting for the scenario-model/build-flow changes in this
+spec (the branch predates both). Once superseded, delete the branch (local +
+`origin`) — confirm before deleting the remote ref.
+
+## Testing
+
+- All 76 existing GTests plus the two newly-registered ones must pass.
+- Full `run_e2e.sh` (default flags) must complete with `results.json`
+  reporting pass for all scenario kinds, `stress_results.json` produced,
+  `coverage.json` showing a higher C++ % than today's GTest/Integration-only
+  baseline, and a single `E2E_Report.pdf` containing all six sections.
+- `run_e2e.sh --skip-coverage --skip-stress --only <id>` must still work for
+  fast single-scenario iteration (unchanged developer workflow).
+- `Test/E2E/chain/tests_py.py` (framework self-tests) extended to cover the
+  new `kind` dispatch logic and the `direct`/`recorder`/`debug`/`tcplogger`
+  scenario definitions the same way it already covers `chain` scenarios.
+
+## Risks
+
+- Double-running chain/datasources/recorder/debug/tcplogger scenarios for
+  coverage roughly doubles that portion of total E2E wall-clock time (stress
+  is unaffected, running once).
+- Refactoring `MarteController` to be headless-usable touches
+  `Client/debugger`'s production code, not just test infra — needs care not
+  to regress the browser UI.
+- The `feature/stress-report-integration` branch's `stress.py` multi-fragment
+  extension needs re-validation against the current `UDPSClient` 1 MiB
+  reassembly cap mentioned in its own design doc, since main has moved on
+  since that branch diverged.