From dcaa466736ccd4c6e9055f429714ae2fc520ff61 Mon Sep 17 00:00:00 2001 From: Martino Ferrari Date: Wed, 1 Jul 2026 18:14:09 +0200 Subject: [PATCH] docs(testing): design for unified test/E2E/reporting/coverage pipeline Consolidates the fragmented chain/stress/streamhub/datasources/recorder E2E suites, unit-test collection, and coverage into one entry point and one report, adds new DebugService/TCPLogger E2E coverage, and fixes the Test/Applications/StreamHub build break blocking `make test`. Co-Authored-By: Claude Opus 4.6 --- ...026-07-01-unified-test-reporting-design.md | 237 ++++++++++++++++++ 1 file changed, 237 insertions(+) create mode 100644 docs/superpowers/specs/2026-07-01-unified-test-reporting-design.md diff --git a/docs/superpowers/specs/2026-07-01-unified-test-reporting-design.md b/docs/superpowers/specs/2026-07-01-unified-test-reporting-design.md new file mode 100644 index 0000000..80612b2 --- /dev/null +++ b/docs/superpowers/specs/2026-07-01-unified-test-reporting-design.md @@ -0,0 +1,237 @@ +# Unified test/E2E/reporting/coverage pipeline — design + +Date: 2026-07-01 + +## Problem + +Test/coverage/reporting infrastructure is fragmented across at least seven +independent entry points with no shared report: + +- `Test/E2E/chain/run_chain_e2e.sh` — the mature suite: 51-scenario matrix, + `collect.py` (C++ GTest, C++ Integration, 3x Go, Python), lcov coverage, + `report_build.py` → `report_data.json`/`history.jsonl`, Typst PDF. +- `Test/E2E/chain/run_stress.sh` (+`stress.py`/`stress_run.py`) — a 27-case + scaling matrix, fully standalone, own JSON/PNG output, no report/history + integration. A design+plan+branch (`feature/stress-report-integration`) + already exists for wiring it in but never landed on `main` and has since + diverged (predates the `UDPStreamerClient` test dir and `DebugServiceGTest` + additions). +- `Test/E2E/streamhub/` (Go smoke test) + root `run_e2e_test.sh` — a strict + subset of what `Test/E2E/chain/client` already checks. +- `Test/E2E/datasources/` (+`run_e2e_report.sh`, its own `E2E_Report.typ`) — + narrower UDPStreamer↔UDPStreamerClient direct round-trip predecessor to the + chain suite; has a stale synthetic (`np.random`) latency histogram in its + report generation. +- `Test/E2E/recorder/` (+`run_recorder_e2e.sh`) — BinaryRecorder disk-output + check, depends on `datasources/`'s generator/validator by relative path. +- `run_combined_test.sh` — manual, no-assertion demo launcher for the + DebugService + TCPLogger + UDPStreamer combined config. +- TCPLogger has **zero** automated tests of any kind today. + +Additionally, `Test/Applications/StreamHub`'s build is currently broken: +two untracked GTest files (`BoundsCheckTest.cpp`, `WSServerBufferTest.cpp` — +regression tests for `BUG_FIX_PLAN.md` items HI-1/HI-4/CR-1) `#include +`, which fails under the project's `-std=c++98` build even during a +bare `-MM` dependency scan (MARTe2's generic `dependsRaw` rule globs *all* +`.cpp` files in a test dir regardless of `OBJSX` membership). This currently +breaks `make test` repo-wide. + +Coverage is also incomplete: it's rebuilt and collected *after* E2E scenarios +already ran on normal (non-instrumented) binaries, so none of the code paths +scenarios actually exercise count toward the C++ coverage percentage. + +## Goal + +One command, one report. Running `Test/E2E/suite/run_e2e.sh` (renamed from +`run_chain_e2e.sh`) executes every test suite in the repo — unit tests +(GTest, Integration), the chain/stress/datasources/recorder/debug/tcplogger +E2E scenario families, and Go/Python framework tests — and produces one +`results.json`, one `unit_tests.json`/`coverage.json`, and one +`report_data.json`/`history.jsonl`/Typst PDF covering all of it, with +accurate coverage (including E2E-exercised code) and uncontaminated +performance metrics. + +## Non-goals + +- No changes to the actual product code paths being tested (this is + test-infrastructure-only), except the minimal `` → MARTe2-types + fix needed to unblock the build. +- No new CI wiring (out of scope; this only needs to produce artifacts a CI + job *could* consume later). +- No decision here about merging/resolving `BUG_FIX_PLAN.md`'s broader + remediation phases — only unblocking the two orphaned test files enough to + compile and run as part of the unified suite. + +## Design + +### 1. Layout and entry point + +- `Test/E2E/chain/` → `Test/E2E/suite/` (git `mv`), `run_chain_e2e.sh` → + `run_e2e.sh`. `CLAUDE.md`/`AGENTS.md` references updated. +- New flags on `run_e2e.sh`: `--skip-coverage`, `--skip-stress`, + `--skip-datasources`, `--skip-recorder`, `--skip-debug`, + `--skip-tcplogger` (all six suites run by default; existing + `--skip-build`/`--only`/`--pdf-only`/`--cpp-coverage` retained, + `--cpp-coverage` now governs the single instrumented phase below instead + of a second rebuild). +- `Test/E2E/datasources/` and `Test/E2E/recorder/` directories stay in place + (cfgs, `gen_test_data.py`, `validate_binary.py`); `run_e2e.sh` imports them + as Python modules (same pattern it already uses for `scenarios`/ + `gen_data`/`gen_cfg`/`validate_waveform`). +- Deleted: `Test/E2E/streamhub/` (dir + Go module/binary), root + `run_e2e_test.sh`, `Test/E2E/datasources/run_e2e_report.sh` + its + standalone `E2E_Report.typ`, `Test/E2E/recorder/run_recorder_e2e.sh`, root + `run_combined_test.sh`. + +### 2. Build & coverage flow + +Flow when coverage is enabled (default): + +1. Build **normal** binaries (`make core apps` + test dirs, no + `--coverage`), unless `--skip-build`. +2. **Authoritative pass**: run chain + datasources + recorder + debug + + tcplogger scenarios (any not individually skipped) on normal binaries. + This is the sole source of `results.json` (pass/fail/oracle) and + `perf_*.json` (cpu/RSS/throughput — unchanged from today, feeds + `trend_perf.png`/regression). +3. Run the **stress matrix** (unless `--skip-stress`) on the same normal + binaries. Stress is *never* instrumented, regardless of + `--skip-coverage`, because gcov overhead skews its scaling/perf + measurements. +4. Rebuild **instrumented** binaries (`--coverage`): `core`, `apps`, and test + dirs (`Test/GTest`, `Test/Integration`, `Test/Components/DataSources/ + UDPStreamer{,Client}`, `Test/Applications/StreamHub`). +5. Re-run chain + datasources + recorder + debug + tcplogger scenarios on + instrumented binaries for **coverage accumulation only**. Their pass/fail + is compared against step 2's results as a sanity check (a mismatch is + logged as a warning, not a report-blocking failure — it flags + nondeterminism without duplicating the correctness gate); their perf + metrics are discarded. +6. Run C++ GTest + Integration (instrumented) and the Go/Python suites + (unaffected by instrumentation, run once). +7. Collect coverage once (`lcov --capture` + `--extract Source/* Test/*`, + same restricted scope as today), aggregating gcov data from steps 5 and 6. +8. Restore a clean, non-instrumented `core apps` build so the repo ends in + its normal runnable state. + +When `--skip-coverage`: steps 4/5/7 are skipped; step 6 runs GTest/ +Integration once on normal binaries; `coverage.json`/the report's coverage +section is marked `"skipped": true`. + +### 3. Unified scenario model + +`scenarios.py` entries gain a `kind` field: + +- `"chain"` — the existing 51 scenarios, unchanged. +- `"direct"` — new; the 2 existing datasources cfgs (`E2ETest.cfg`, + `E2EMulticastTest.cfg`), UDPStreamer→UDPStreamerClient only, no + StreamHub/WS hop, validated via `validate_binary.py` (byte-identical + round-trip). +- `"recorder"` — new; `RecorderStreamer.cfg`/`StreamHubRec.cfg`, chain path + + `BinaryRecorder`, validated via `validate_binary.py` against the + recorded disk file. +- `"debug"` — new; see §4. +- `"tcplogger"` — new; see §4. + +`run_e2e.sh`'s per-scenario loop dispatches on `kind`: `chain` keeps today's +launch/record/validate sequence; `direct` skips the StreamHub/chain-client +hop (launch `MARTeApp.ex` with `UDPStreamerClient` receiving directly, then +byte-compare); `recorder` runs the chain sequence plus a post-run disk-file +check; `debug`/`tcplogger` launch `MARTeApp.ex` with a DebugService/TCPLogger +config and drive it with `debugclient` (§4). All five write into the same +`results.json` (with `kind` as a discriminator field), so `known_issue`/ +XFAIL/XPASS handling and PASS/FAIL aggregation need no branching beyond +dispatch. + +Stress stays structurally separate (`stress_results.json`, produced by +`stress_run.py` as today, read by `report_build.py` as a distinct input — +the same pattern already used for `perf_*.json`), because it's a scaling +matrix, not a per-signal pass/fail scenario. + +### 4. DebugService & TCPLogger E2E scenarios + +`Client/debugger/martecontrol.go`'s `MarteController` already implements the +needed protocol clients as library code: `Connect`/`runTCP`/`SendCommand` +(TCP 8080 command protocol), `runDebugUDP` (UDP 8081 trace telemetry), and +`runLog` (TCP 8082 TCPLogger stream) — currently wired to a WebSocket hub for +the browser UI. Refactor the hub-broadcast calls behind a small callback +interface so this logic is reusable headless, and add a new Go binary +`Test/E2E/suite/debugclient/` that imports it, scripted instead of +browser-driven. + +- **`kind="debug"`**: launch `MARTeApp.ex` with a DebugService config derived + from `Test/Configurations/combined_test.cfg` (trimmed to what's needed). + `debugclient` connects to TCP 8080 + UDP 8081 and scripts a sequence + (force a signal, arm a breakpoint, trace a signal) mirroring what + `Test/Integration`'s `TraceTest.cpp`/`ValidationTest.cpp` already exercise + in-process — but validated end-to-end as a separate OS process pair over + real sockets, catching wire-format/serialization bugs the in-process suite + cannot. +- **`kind="tcplogger"`**: same launch, `debugclient` connects to TCP 8082, + triggers known log-worthy events (a forced value, a config error), asserts + expected log lines arrive with correct formatting/ordering. +- Both are functional (pass/fail), not perf-sensitive: they follow the + chain/datasources/recorder double-run pattern from §2 for coverage + consistency, but skip `proc_perf.py` capture. +- `run_combined_test.sh` becomes redundant once these scenarios exist and is + retired, folding its config into the new scenario cfgs. + +### 5. Reporting + +`E2E_Report.typ`/`report_data.json` gain sections alongside the existing +Progression/Unit tests/Coverage/Performance/Scenarios: **Stress Tests** +(per-axis scaling tables/plots + regression, per the existing +`2026-06-26-stress-suite-report-integration-design.md`), **Direct +Round-Trip** (datasources pass/fail table), **Recorder** (pass/fail table), +**Debug Service E2E** and **TCPLogger E2E** (pass/fail per scenario). +`history.jsonl`/`headline()`/`regression()` extended with fields from each +new section so trend plots and regression-vs-previous-run cover everything +in one place. + +### 6. `Test/Applications/StreamHub` build fix + +`BoundsCheckTest.cpp`/`WSServerBufferTest.cpp` replace `#include ` +with MARTe2 types (`uint8`/`uint32`/etc. from `GeneralDefinitions.h`), +matching the sibling GTest files' style and the repo's MARTe2-style +convention. Both are registered in `Makefile.inc`'s `OBJSX` (as +`BoundsCheckTest.x`/`WSServerBufferTest.x`) alongside the existing four +GTest `.x` entries, so they compile deliberately rather than only being +incidentally scanned by the `-MM` dependency glob. + +### 7. Stale branch reconciliation + +`git diff main...feature/stress-report-integration` shows a concrete, +mostly-still-relevant 8-commit diff (`report_build.py` +124/-lines, +`run_chain_e2e.sh` +27, `stress.py` +69 for the multi-fragment size-axis +extension, plus a new `test_report_stress.py`, a `CLAUDE.md` doc line, and a +stray `TODO.md`). Port the relevant logic (stress table/plots/regression +wiring, the multi-fragment size-axis fix) onto current `main` as part of +implementing §5, adapting for the scenario-model/build-flow changes in this +spec (the branch predates both). Once superseded, delete the branch (local + +`origin`) — confirm before deleting the remote ref. + +## Testing + +- All 76 existing GTests plus the two newly-registered ones must pass. +- Full `run_e2e.sh` (default flags) must complete with `results.json` + reporting pass for all scenario kinds, `stress_results.json` produced, + `coverage.json` showing a higher C++ % than today's GTest/Integration-only + baseline, and a single `E2E_Report.pdf` containing all six sections. +- `run_e2e.sh --skip-coverage --skip-stress --only ` must still work for + fast single-scenario iteration (unchanged developer workflow). +- `Test/E2E/chain/tests_py.py` (framework self-tests) extended to cover the + new `kind` dispatch logic and the `direct`/`recorder`/`debug`/`tcplogger` + scenario definitions the same way it already covers `chain` scenarios. + +## Risks + +- Double-running chain/datasources/recorder/debug/tcplogger scenarios for + coverage roughly doubles that portion of total E2E wall-clock time (stress + is unaffected, running once). +- Refactoring `MarteController` to be headless-usable touches + `Client/debugger`'s production code, not just test infra — needs care not + to regress the browser UI. +- The `feature/stress-report-integration` branch's `stress.py` multi-fragment + extension needs re-validation against the current `UDPSClient` 1 MiB + reassembly cap mentioned in its own design doc, since main has moved on + since that branch diverged.