MARTe-Integrated-Components/docs/superpowers/specs/2026-07-01-unified-test-reporting-design.md

Unified test/E2E/reporting/coverage pipeline — design

Date: 2026-07-01

Problem

Test/coverage/reporting infrastructure is fragmented across at least seven independent entry points with no shared report:

• Test/E2E/chain/run_chain_e2e.sh — the mature suite: 51-scenario matrix, collect.py (C++ GTest, C++ Integration, 3x Go, Python), lcov coverage, report_build.py → report_data.json/history.jsonl, Typst PDF.
• Test/E2E/chain/run_stress.sh (+stress.py/stress_run.py) — a 27-case scaling matrix, fully standalone, own JSON/PNG output, no report/history integration. A design+plan+branch (feature/stress-report-integration) already exists for wiring it in but never landed on main and has since diverged (predates the UDPStreamerClient test dir and DebugServiceGTest additions).
• Test/E2E/streamhub/ (Go smoke test) + root run_e2e_test.sh — a strict subset of what Test/E2E/chain/client already checks.
• Test/E2E/datasources/ (+run_e2e_report.sh, its own E2E_Report.typ) — narrower UDPStreamer↔UDPStreamerClient direct round-trip predecessor to the chain suite; has a stale synthetic (np.random) latency histogram in its report generation.
• Test/E2E/recorder/ (+run_recorder_e2e.sh) — BinaryRecorder disk-output check, depends on datasources/'s generator/validator by relative path.
• run_combined_test.sh — manual, no-assertion demo launcher for the DebugService + TCPLogger + UDPStreamer combined config.
• TCPLogger has zero automated tests of any kind today.

Additionally, Test/Applications/StreamHub's build is currently broken: two untracked GTest files (BoundsCheckTest.cpp, WSServerBufferTest.cpp — regression tests for BUG_FIX_PLAN.md items HI-1/HI-4/CR-1) #include <cstdint>, which fails under the project's -std=c++98 build even during a bare -MM dependency scan (MARTe2's generic dependsRaw rule globs all .cpp files in a test dir regardless of OBJSX membership). This currently breaks make test repo-wide.

Coverage is also incomplete: it's rebuilt and collected after E2E scenarios already ran on normal (non-instrumented) binaries, so none of the code paths scenarios actually exercise count toward the C++ coverage percentage.

Goal

One command, one report. Running Test/E2E/suite/run_e2e.sh (renamed from run_chain_e2e.sh) executes every test suite in the repo — unit tests (GTest, Integration), the chain/stress/datasources/recorder/debug/tcplogger E2E scenario families, and Go/Python framework tests — and produces one results.json, one unit_tests.json/coverage.json, and one report_data.json/history.jsonl/Typst PDF covering all of it, with accurate coverage (including E2E-exercised code) and uncontaminated performance metrics.

Non-goals

• No changes to the actual product code paths being tested (this is test-infrastructure-only), except the minimal <cstdint> → MARTe2-types fix needed to unblock the build.
• No new CI wiring (out of scope; this only needs to produce artifacts a CI job could consume later).
• No decision here about merging/resolving BUG_FIX_PLAN.md's broader remediation phases — only unblocking the two orphaned test files enough to compile and run as part of the unified suite.

Design

1. Layout and entry point

• Test/E2E/chain/ → Test/E2E/suite/ (git mv), run_chain_e2e.sh → run_e2e.sh. CLAUDE.md/AGENTS.md references updated.
• New flags on run_e2e.sh: --skip-coverage, --skip-stress, --skip-datasources, --skip-recorder, --skip-debug, --skip-tcplogger (all six suites run by default; existing --skip-build/--only/--pdf-only/--cpp-coverage retained, --cpp-coverage now governs the single instrumented phase below instead of a second rebuild).
• Test/E2E/datasources/ and Test/E2E/recorder/ directories stay in place (cfgs, gen_test_data.py, validate_binary.py); run_e2e.sh imports them as Python modules (same pattern it already uses for scenarios/ gen_data/gen_cfg/validate_waveform).
• Deleted: Test/E2E/streamhub/ (dir + Go module/binary), root run_e2e_test.sh, Test/E2E/datasources/run_e2e_report.sh + its standalone E2E_Report.typ, Test/E2E/recorder/run_recorder_e2e.sh, root run_combined_test.sh.

2. Build & coverage flow

Flow when coverage is enabled (default):

1. Build normal binaries (make core apps + test dirs, no --coverage), unless --skip-build.
2. Authoritative pass: run chain + datasources + recorder + debug + tcplogger scenarios (any not individually skipped) on normal binaries. This is the sole source of results.json (pass/fail/oracle) and perf_*.json (cpu/RSS/throughput — unchanged from today, feeds trend_perf.png/regression).
3. Run the stress matrix (unless --skip-stress) on the same normal binaries. Stress is never instrumented, regardless of --skip-coverage, because gcov overhead skews its scaling/perf measurements.
4. Rebuild instrumented binaries (--coverage): core, apps, and test dirs (Test/GTest, Test/Integration, Test/Components/DataSources/ UDPStreamer{,Client}, Test/Applications/StreamHub).
5. Re-run chain + datasources + recorder + debug + tcplogger scenarios on instrumented binaries for coverage accumulation only. Their pass/fail is compared against step 2's results as a sanity check (a mismatch is logged as a warning, not a report-blocking failure — it flags nondeterminism without duplicating the correctness gate); their perf metrics are discarded.
6. Run C++ GTest + Integration (instrumented) and the Go/Python suites (unaffected by instrumentation, run once).
7. Collect coverage once (lcov --capture + --extract Source/* Test/*, same restricted scope as today), aggregating gcov data from steps 5 and 6.
8. Restore a clean, non-instrumented core apps build so the repo ends in its normal runnable state.

When --skip-coverage: steps 4/5/7 are skipped; step 6 runs GTest/ Integration once on normal binaries; coverage.json/the report's coverage section is marked "skipped": true.

3. Unified scenario model

scenarios.py entries gain a kind field:

• "chain" — the existing 51 scenarios, unchanged.
• "direct" — new; the 2 existing datasources cfgs (E2ETest.cfg, E2EMulticastTest.cfg), UDPStreamer→UDPStreamerClient only, no StreamHub/WS hop, validated via validate_binary.py (byte-identical round-trip).
• "recorder" — new; RecorderStreamer.cfg/StreamHubRec.cfg, chain path
  • BinaryRecorder, validated via validate_binary.py against the recorded disk file.
• "debug" — new; see §4.
• "tcplogger" — new; see §4.

run_e2e.sh's per-scenario loop dispatches on kind: chain keeps today's launch/record/validate sequence; direct skips the StreamHub/chain-client hop (launch MARTeApp.ex with UDPStreamerClient receiving directly, then byte-compare); recorder runs the chain sequence plus a post-run disk-file check; debug/tcplogger launch MARTeApp.ex with a DebugService/TCPLogger config and drive it with debugclient (§4). All five write into the same results.json (with kind as a discriminator field), so known_issue/ XFAIL/XPASS handling and PASS/FAIL aggregation need no branching beyond dispatch.

Stress stays structurally separate (stress_results.json, produced by stress_run.py as today, read by report_build.py as a distinct input — the same pattern already used for perf_*.json), because it's a scaling matrix, not a per-signal pass/fail scenario.

4. DebugService & TCPLogger E2E scenarios

Client/debugger/martecontrol.go's MarteController already implements the needed protocol clients as library code: Connect/runTCP/SendCommand (TCP 8080 command protocol), runDebugUDP (UDP 8081 trace telemetry), and runLog (TCP 8082 TCPLogger stream) — currently wired to a WebSocket hub for the browser UI. Refactor the hub-broadcast calls behind a small callback interface so this logic is reusable headless, and add a new Go binary Test/E2E/suite/debugclient/ that imports it, scripted instead of browser-driven.

• kind="debug": launch MARTeApp.ex with a DebugService config derived from Test/Configurations/combined_test.cfg (trimmed to what's needed). debugclient connects to TCP 8080 + UDP 8081 and scripts a sequence (force a signal, arm a breakpoint, trace a signal) mirroring what Test/Integration's TraceTest.cpp/ValidationTest.cpp already exercise in-process — but validated end-to-end as a separate OS process pair over real sockets, catching wire-format/serialization bugs the in-process suite cannot.
• kind="tcplogger": same launch, debugclient connects to TCP 8082, triggers known log-worthy events (a forced value, a config error), asserts expected log lines arrive with correct formatting/ordering.
• Both are functional (pass/fail), not perf-sensitive: they follow the chain/datasources/recorder double-run pattern from §2 for coverage consistency, but skip proc_perf.py capture.
• run_combined_test.sh becomes redundant once these scenarios exist and is retired, folding its config into the new scenario cfgs.

5. Reporting

E2E_Report.typ/report_data.json gain sections alongside the existing Progression/Unit tests/Coverage/Performance/Scenarios: Stress Tests (per-axis scaling tables/plots + regression, per the existing 2026-06-26-stress-suite-report-integration-design.md), Direct Round-Trip (datasources pass/fail table), Recorder (pass/fail table), Debug Service E2E and TCPLogger E2E (pass/fail per scenario). history.jsonl/headline()/regression() extended with fields from each new section so trend plots and regression-vs-previous-run cover everything in one place.

6. Test/Applications/StreamHub build fix

BoundsCheckTest.cpp/WSServerBufferTest.cpp replace #include <cstdint> with MARTe2 types (uint8/uint32/etc. from GeneralDefinitions.h), matching the sibling GTest files' style and the repo's MARTe2-style convention. Both are registered in Makefile.inc's OBJSX (as BoundsCheckTest.x/WSServerBufferTest.x) alongside the existing four GTest .x entries, so they compile deliberately rather than only being incidentally scanned by the -MM dependency glob.

7. Stale branch reconciliation

git diff main...feature/stress-report-integration shows a concrete, mostly-still-relevant 8-commit diff (report_build.py +124/-lines, run_chain_e2e.sh +27, stress.py +69 for the multi-fragment size-axis extension, plus a new test_report_stress.py, a CLAUDE.md doc line, and a stray TODO.md). Port the relevant logic (stress table/plots/regression wiring, the multi-fragment size-axis fix) onto current main as part of implementing §5, adapting for the scenario-model/build-flow changes in this spec (the branch predates both). Once superseded, delete the branch (local + origin) — confirm before deleting the remote ref.

Testing

• All 76 existing GTests plus the two newly-registered ones must pass.
• Full run_e2e.sh (default flags) must complete with results.json reporting pass for all scenario kinds, stress_results.json produced, coverage.json showing a higher C++ % than today's GTest/Integration-only baseline, and a single E2E_Report.pdf containing all six sections.
• run_e2e.sh --skip-coverage --skip-stress --only <id> must still work for fast single-scenario iteration (unchanged developer workflow).
• Test/E2E/chain/tests_py.py (framework self-tests) extended to cover the new kind dispatch logic and the direct/recorder/debug/tcplogger scenario definitions the same way it already covers chain scenarios.

Risks

• Double-running chain/datasources/recorder/debug/tcplogger scenarios for coverage roughly doubles that portion of total E2E wall-clock time (stress is unaffected, running once).
• Refactoring MarteController to be headless-usable touches Client/debugger's production code, not just test infra — needs care not to regress the browser UI.
• The feature/stress-report-integration branch's stress.py multi-fragment extension needs re-validation against the current UDPSClient 1 MiB reassembly cap mentioned in its own design doc, since main has moved on since that branch diverged.