# Unified Test/E2E/Reporting/Coverage Pipeline Implementation Plan > **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. **Goal:** Consolidate every test suite in the repo (unit tests, chain/stress/datasources/recorder E2E, new DebugService/TCPLogger E2E) behind one entry point (`Test/E2E/suite/run_e2e.sh`) producing one `results.json`/`unit_tests.json`/`coverage.json`/`report_data.json`/`history.jsonl`/PDF, with accurate E2E-inclusive coverage and uncontaminated performance metrics. **Architecture:** Extend the existing mature `Test/E2E/chain` pipeline (renamed `Test/E2E/suite`) rather than building a new one: add a `kind` discriminator to the scenario model so `direct` (datasources), `recorder`, `debug`, and `tcplogger` scenarios flow through the same `results.json`/dispatch loop as `chain` scenarios; reorder the build so unit tests + all scenario kinds run once on normal binaries (authoritative pass/fail + perf) and once on instrumented binaries (coverage only, perf discarded), with stress always running once, uninstrumented; extend `report_build.py`/`E2E_Report.typ` with sections per kind plus the already-designed stress section. **Tech Stack:** Bash (orchestration), Python 3 (scenario generation/validation/reporting, `lcov`/`gcov`), Go (test clients), C++ (GTest, MARTe2), Typst (PDF report). ## Global Constraints - No STL/`` in MARTe2-style components (`Test/Applications/StreamHub`, `Source/Components/*`) — use MARTe2 types (`uint32`, `uint64`, etc. from `GeneralDefinitions.h`). - Build via `source env.sh && export MAKEDEFAULTDIR=$MARTe2_DIR/MakeDefaults && make -f Makefile.gcc ...` — never pipe `source env.sh` (breaks env propagation in a subshell). - All 76 existing GTests must keep passing throughout; no task may leave `make test` in a broken state at its end. - `run_e2e.sh --skip-build --only ` (single-scenario fast iteration) must keep working after every task. - EUPL v1.1 license headers on new C++ sources (match existing files' header style). - Stress scenarios must never run against `--coverage`-instrumented binaries (perf/scaling measurements would be skewed). --- ## Task 1: Fix `Test/Applications/StreamHub` build break **Files:** - Modify: `Test/Applications/StreamHub/BoundsCheckTest.cpp` - Modify: `Test/Applications/StreamHub/Makefile.inc:25` - Test: `Test/Applications/StreamHub/BoundsCheckTest.cpp` (GTest, run via `MainGTest.ex`) **Interfaces:** - Consumes: nothing new. - Produces: `make -C Test/Applications/StreamHub -f Makefile.gcc` succeeds; `MainGTest.ex` gains 3 new passing tests (`BoundsCheckTest.OverflowRejected`, `.NormalCasePasses`, `.NumRowsNumColsOverflow`). `BoundsCheckTest.cpp` uses `` (`uint32_t`/`uint64_t`), which fails to compile under this project's `-std=c++98` flags — this breaks `Test/Applications/StreamHub`'s generic `dependsRaw` rule (`g++ -MM *.c*`, scans all `.cpp` files regardless of `OBJSX` membership) and therefore breaks `make test` repo-wide. `WSServerBufferTest.cpp` already uses MARTe2 types correctly and needs no source change, only `OBJSX` registration. - [ ] **Step 1: Replace `` types with MARTe2 types in `BoundsCheckTest.cpp`** Replace the full file content (current file confirmed 74 lines) — every `uint32_t` → `uint32`, every `uint64_t` → `uint64`, drop the `` and `` includes (both unused beyond the removed types; `` was never used): ```cpp /** * @file BoundsCheckTest.cpp * @brief Reproduction tests for HI-1 (integer overflow in bounds check) and * HI-4 (unclamped forcedValue memcpy). * * These tests verify that the 64-bit bounds check pattern correctly rejects * crafted payloads whose 32-bit multiply would overflow, and that the * forcedValue clamp prevents OOB reads. */ #include #include "GeneralDefinitions.h" // HI-1: Verify that a 64-bit bounds check rejects a payload where // elemsToRead * wireElemBytes would overflow uint32. TEST(BoundsCheckTest, OverflowRejected) { // Simulate: numSamples = 0x20000001, wireElemBytes = 8 // 32-bit: 0x20000001 * 8 = 0x8 (overflow!) // 64-bit: 0x100000008 (correctly large, > any reasonable payload size) MARTe::uint32 elemsToRead = 0x20000001u; MARTe::uint32 wireElemBytes = 8u; MARTe::uint32 off = 12u; // after HRT + numSamples MARTe::uint32 size = 1400u; // typical max payload // This is the FIXED pattern (64-bit): MARTe::uint64 bytesNeeded = static_cast(off) + static_cast(elemsToRead) * static_cast(wireElemBytes); // Should reject (bytesNeeded >> size) EXPECT_GT(bytesNeeded, static_cast(size)) << "64-bit check should detect overflow that 32-bit would miss"; // Verify the OLD (buggy) 32-bit pattern would have passed: MARTe::uint32 oldCheck = off + (elemsToRead * wireElemBytes); // On 32-bit: 0x20000001 * 8 = 0x100000008 truncated to 0x8 // off + 0x8 = 20, which is < 1400, so the old check would pass (bug!) // On 64-bit: the multiply doesn't overflow, so oldCheck is huge // This test documents that the 64-bit fix is necessary on 32-bit platforms // and correct on 64-bit. (void) oldCheck; } // HI-1: Verify a normal (non-overflow) case passes the 64-bit check. TEST(BoundsCheckTest, NormalCasePasses) { MARTe::uint32 elemsToRead = 100u; MARTe::uint32 wireElemBytes = 4u; MARTe::uint32 off = 12u; MARTe::uint32 size = 500u; MARTe::uint64 bytesNeeded = static_cast(off) + static_cast(elemsToRead) * static_cast(wireElemBytes); EXPECT_LE(bytesNeeded, static_cast(size)) << "normal case should pass the bounds check"; } // HI-1: Verify numRows * numCols overflow is detected. TEST(BoundsCheckTest, NumRowsNumColsOverflow) { MARTe::uint32 numRows = 0x10000u; MARTe::uint32 numCols = 0x10000u; // 32-bit: 0x10000 * 0x10000 = 0 (overflow!) MARTe::uint32 oldResult = numRows * numCols; EXPECT_EQ(oldResult, 0u) << "32-bit multiply should overflow to 0"; // 64-bit fix: MARTe::uint64 newResult = static_cast(numRows) * static_cast(numCols); EXPECT_EQ(newResult, static_cast(0x100000000ULL)) << "64-bit multiply should give correct result"; EXPECT_GT(newResult, static_cast(0x100000u)) << "should exceed the sanity cap, triggering rejection"; } ``` - [ ] **Step 2: Register both orphaned files in `OBJSX`** Edit `Test/Applications/StreamHub/Makefile.inc:25`, change: ```makefile OBJSX = TriggerEngineSrc.x BinaryRecorderSrc.x SignalRingBufferGTest.x TriggerEngineGTest.x LTTBGTest.x BinaryRecorderGTest.x ``` to: ```makefile OBJSX = TriggerEngineSrc.x BinaryRecorderSrc.x SignalRingBufferGTest.x TriggerEngineGTest.x LTTBGTest.x BinaryRecorderGTest.x BoundsCheckTest.x WSServerBufferTest.x ``` - [ ] **Step 3: Rebuild and verify** ```bash source env.sh && export MAKEDEFAULTDIR=$MARTe2_DIR/MakeDefaults rm -f Test/Applications/StreamHub/depends.x86-linux Test/Applications/StreamHub/dependsRaw.x86-linux make -C Test/Applications/StreamHub -f Makefile.gcc make -C Test/GTest -f Makefile.gcc ./Build/x86-linux/GTest/MainGTest.ex --gtest_filter='BoundsCheckTest.*:WSServerBufferTest.*' ``` Expected: clean build, both filters show `[ PASSED ]` for all tests (3 `BoundsCheckTest` + whatever `WSServerBufferTest` already defines). ```bash ./Build/x86-linux/GTest/MainGTest.ex 2>&1 | tail -5 ``` Expected: `[ PASSED ] N tests.` with N = 76 + (new test count), 0 failures. - [ ] **Step 4: Commit** ```bash git add Test/Applications/StreamHub/BoundsCheckTest.cpp Test/Applications/StreamHub/WSServerBufferTest.cpp Test/Applications/StreamHub/Makefile.inc git commit -m "fix(streamhub-test): unblock make test by fixing BoundsCheckTest C++98 build and registering both orphaned GTest files" ``` --- ## Task 2: Rename `Test/E2E/chain` → `Test/E2E/suite`, `run_chain_e2e.sh` → `run_e2e.sh` **Files:** - Rename (git mv): `Test/E2E/chain/` → `Test/E2E/suite/` (all contents, including `client/` Go module) - Rename (git mv): `Test/E2E/suite/run_chain_e2e.sh` → `Test/E2E/suite/run_e2e.sh` - Modify: `CLAUDE.md` (Build & Test section, streaming-chain E2E paragraph) - Modify: `AGENTS.md` (any `Test/E2E/chain`/`run_chain_e2e.sh` references) **Interfaces:** - Consumes: nothing new. - Produces: `Test/E2E/suite/run_e2e.sh --skip-build --only s01_scalar_uint32` works identically to the old `run_chain_e2e.sh` invocation. This is a pure rename with no logic change — do it as its own task/commit so later tasks' diffs are readable against the new paths. - [ ] **Step 1: Rename directory and script** ```bash git mv Test/E2E/chain Test/E2E/suite git mv Test/E2E/suite/run_chain_e2e.sh Test/E2E/suite/run_e2e.sh ``` - [ ] **Step 2: Fix internal self-references** ```bash grep -rn "chain_e2e\|Test/E2E/chain\|run_chain_e2e" Test/E2E/suite/ 2>/dev/null ``` For each hit found (expect at least `/tmp/chain_e2e` work-dir paths in `run_e2e.sh`, and possibly `client/` Go module comments referencing "chain"): leave Go package/module names and the `/tmp/chain_e2e` scratch dir path as-is (renaming the scratch dir is optional churn, not required for correctness — confirm no hardcoded absolute path outside `/tmp` breaks). Only fix any comment/string that says `Test/E2E/chain` (should say `Test/E2E/suite`) or `run_chain_e2e.sh` (should say `run_e2e.sh`). - [ ] **Step 3: Update `CLAUDE.md`** In the `## Build & Test` section, find the paragraph beginning `**Streaming-chain E2E suite** (`Test/E2E/chain/`)`. Change `Test/E2E/chain/` → `Test/E2E/suite/` and `./run_chain_e2e.sh` → `./run_e2e.sh` everywhere in that paragraph (both occurrences of the script name in the usage line and prose). - [ ] **Step 4: Update `AGENTS.md`** ```bash grep -n "Test/E2E/chain\|run_chain_e2e" AGENTS.md ``` Apply the same `Test/E2E/chain` → `Test/E2E/suite`, `run_chain_e2e.sh` → `run_e2e.sh` substitution to every match. - [ ] **Step 5: Verify** ```bash source env.sh cd Test/E2E/suite && ./run_e2e.sh --skip-build --only s01_scalar_uint32 2>&1 | tail -20 ``` Expected: `s01_scalar_uint32: PASS` and `results.json: 1 pass, 0 fail, 0 skip, 0 xfail, 0 xpass → PASS`, exactly as `run_chain_e2e.sh` produced before the rename. - [ ] **Step 6: Commit** ```bash cd /home/martino/Projects/MARTe_Integrated_components git add -A Test/E2E/suite Test/E2E/chain CLAUDE.md AGENTS.md git commit -m "refactor(e2e): rename Test/E2E/chain -> Test/E2E/suite, run_chain_e2e.sh -> run_e2e.sh" ``` --- ## Task 3: Add `kind` field to the scenario model + `direct`/`recorder` scenario definitions **Files:** - Modify: `Test/E2E/suite/scenarios.py` - Modify: `Test/E2E/suite/tests_py.py` - Test: `Test/E2E/suite/tests_py.py` (extended) **Interfaces:** - Consumes: `Test/E2E/datasources/E2ETest.cfg`, `Test/E2E/datasources/E2EMulticastTest.cfg`, `Test/E2E/datasources/gen_test_data.py::generate() -> None` (writes module-level `OUTPUT` path), `Test/E2E/datasources/validate_binary.py::validate(input_path, output_path, label="test") -> (bool, str, dict)`, `Test/E2E/recorder/RecorderStreamer.cfg`, `Test/E2E/recorder/StreamHubRec.cfg`. - Produces: every entry in `scenarios.SCENARIOS` now has a `"kind"` key (`"chain"` for all 51 existing entries, defaulted via `_sig`/scenario-builder helpers so no existing entry needs editing by hand); two new module-level lists `scenarios._DIRECT` (2 entries) and `scenarios._RECORDER` (1 entry) are folded into `SCENARIOS`; `scenarios.validate_scenario(s)` accepts the new `kind` key and the `direct`/`recorder`-specific keys below without raising. New scenario dict shape for `kind="direct"` (no `sources`/`network`/`oracle` — those are chain-only concepts): ```python { "id": "s52_direct_unicast", "desc": "Direct UDPStreamer->UDPStreamerClient round-trip, unicast", "kind": "direct", "cfg": "Test/E2E/datasources/E2ETest.cfg", "client_checks": [], "known_issue": None, } ``` and a second entry `s53_direct_multicast` with `"cfg": "Test/E2E/datasources/E2EMulticastTest.cfg"`. New scenario dict shape for `kind="recorder"`: ```python { "id": "s54_recorder", "desc": "StreamHub BinaryRecorder disk-output round-trip", "kind": "recorder", "hub_cfg": "Test/E2E/recorder/StreamHubRec.cfg", "marte_cfg": "Test/E2E/recorder/RecorderStreamer.cfg", "client_checks": [], "known_issue": None, } ``` - [ ] **Step 1: Add `kind` default to existing chain scenarios** In `scenarios.py`, find the `_sig(...)` helper and the point where `_STARTERS`/`_TYPES`/etc. lists are concatenated into `SCENARIOS` (currently `SCENARIOS = _STARTERS + _TYPES + _ARRAYS + _TIMEMODES + _QUANT + _PUBLISH + _PAYLOAD + _MCAST + _MULTISRC + _INTERACT + _TRIGGER + _MISC + _HIGHRATE`, line 564). Add a normalization pass immediately after that line: ```python SCENARIOS = _STARTERS + _TYPES + _ARRAYS + _TIMEMODES + _QUANT + _PUBLISH + _PAYLOAD + _MCAST + _MULTISRC + _INTERACT + _TRIGGER + _MISC + _HIGHRATE for _s in SCENARIOS: _s.setdefault("kind", "chain") ``` - [ ] **Step 2: Add `_DIRECT` and `_RECORDER` scenario lists** Immediately after the normalization loop from Step 1, add: ```python _DIRECT = [ { "id": "s52_direct_unicast", "desc": "Direct UDPStreamer->UDPStreamerClient round-trip, unicast", "kind": "direct", "cfg": "Test/E2E/datasources/E2ETest.cfg", "client_checks": [], "known_issue": None, }, { "id": "s53_direct_multicast", "desc": "Direct UDPStreamer->UDPStreamerClient round-trip, multicast", "kind": "direct", "cfg": "Test/E2E/datasources/E2EMulticastTest.cfg", "client_checks": [], "known_issue": None, }, ] _RECORDER = [ { "id": "s54_recorder", "desc": "StreamHub BinaryRecorder disk-output round-trip", "kind": "recorder", "hub_cfg": "Test/E2E/recorder/StreamHubRec.cfg", "marte_cfg": "Test/E2E/recorder/RecorderStreamer.cfg", "client_checks": [], "known_issue": None, }, ] SCENARIOS = SCENARIOS + _DIRECT + _RECORDER ``` - [ ] **Step 3: Update `validate_scenario(s)` to accept the new kinds** Read the current `validate_scenario(s)` body (validates chain-specific keys like `sources`/`network`/`oracle`). Wrap the chain-specific assertions in a guard so `direct`/`recorder` kinds skip them: ```python def validate_scenario(s): assert "id" in s and "kind" in s, f"scenario missing id/kind: {s}" if s["kind"] == "chain": # ... existing body unchanged, indented under this guard ... pass elif s["kind"] == "direct": assert "cfg" in s, f"direct scenario {s['id']} missing cfg" elif s["kind"] == "recorder": assert "hub_cfg" in s and "marte_cfg" in s, f"recorder scenario {s['id']} missing hub_cfg/marte_cfg" else: raise AssertionError(f"unknown scenario kind: {s['kind']}") ``` (Preserve the exact existing chain-validation statements under the `if s["kind"] == "chain":` branch — do not rewrite their logic, only re-indent.) - [ ] **Step 4: Write the failing test** Add to `Test/E2E/suite/tests_py.py` (find the existing `class ScenariosTest`-style test class and add a method following its pattern): ```python def test_all_scenarios_have_kind(self): import scenarios as S for s in S.SCENARIOS: self.assertIn("kind", s, f"{s['id']} missing kind") self.assertIn(s["kind"], ("chain", "direct", "recorder", "debug", "tcplogger")) def test_direct_and_recorder_scenarios_present(self): import scenarios as S kinds = {s["kind"] for s in S.SCENARIOS} self.assertIn("direct", kinds) self.assertIn("recorder", kinds) direct_ids = {s["id"] for s in S.SCENARIOS if s["kind"] == "direct"} self.assertEqual(direct_ids, {"s52_direct_unicast", "s53_direct_multicast"}) recorder_ids = {s["id"] for s in S.SCENARIOS if s["kind"] == "recorder"} self.assertEqual(recorder_ids, {"s54_recorder"}) def test_validate_scenario_accepts_all_kinds(self): import scenarios as S for s in S.SCENARIOS: S.validate_scenario(s) # must not raise ``` - [ ] **Step 5: Run tests to verify they pass** (this is additive, not TDD-red-first, since it extends existing data structures rather than new behavior with a failing precondition) ```bash cd Test/E2E/suite && python3 -m unittest tests_py -v 2>&1 | tail -20 ``` Expected: all tests pass, including the 3 new ones. - [ ] **Step 6: Commit** ```bash git add Test/E2E/suite/scenarios.py Test/E2E/suite/tests_py.py git commit -m "feat(e2e): add kind discriminator + direct/recorder scenario definitions" ``` --- ## Task 4: Wire `direct`/`recorder` dispatch into `run_e2e.sh`'s scenario loop **Files:** - Modify: `Test/E2E/suite/run_e2e.sh` - Modify: `Test/E2E/suite/collect.py` is NOT touched in this task (unit-test collection is unaffected by new scenario kinds) **Interfaces:** - Consumes: `scenarios.SCENARIOS` entries now carrying `kind` (Task 3); `Test/E2E/datasources/gen_test_data.py::generate()`, `Test/E2E/datasources/validate_binary.py::validate(input_path, output_path, label) -> (bool, str, dict)`; `Test/E2E/recorder/RecorderStreamer.cfg`/`StreamHubRec.cfg` launch sequence (ported from `Test/E2E/recorder/run_recorder_e2e.sh`, to be deleted in Task 9). - Produces: `results.json` entries for `s52_direct_unicast`, `s53_direct_multicast`, `s54_recorder` alongside the 51 chain entries, each with a `"kind"` field. The current scenario-list-building Python block (embedded in `run_e2e.sh`, feeding the `while IFS='|' read ...` loop) only emits the 7-field chain pipe format (`id|ws_port|udp_port0|network|oracle|trig|checks`). Extend it to also emit `direct`/`recorder` entries in an extended format, and add a `KIND` field as the first column so the loop can dispatch. - [ ] **Step 1: Extend the inline Python scenario-list builder** Find this block in `run_e2e.sh` (the `LIST="$(${PY} - "${ONLY}" <<'PY' ... PY )"` heredoc). Replace its body: ```python import sys, os sys.path.insert(0, os.path.dirname(os.path.abspath("Test/E2E/suite/scenarios.py"))) sys.path.insert(0, os.path.join(os.getcwd(), "Test/E2E/suite")) import scenarios as S only = sys.argv[1] if len(sys.argv) > 1 else "" for s in S.SCENARIOS: if only and s["id"] != only: continue kind = s["kind"] if kind == "chain": trig = s.get("trig_signal") or "" checks = ",".join(s.get("client_checks", [])) if not trig: checks = ",".join(c for c in s.get("client_checks", []) if c != "trigger") print("|".join([kind, s["id"], str(s["ws_port"]), str(s["sources"][0]["udp_port"]), s["network"], s["oracle"], trig, checks])) elif kind == "direct": print("|".join([kind, s["id"], s["cfg"], "", "", "", "", ""])) elif kind == "recorder": print("|".join([kind, s["id"], s["marte_cfg"], s["hub_cfg"], "", "", "", ""])) ``` (Note: `Test/E2E/chain` path in the two `sys.path.insert` lines becomes `Test/E2E/suite` — already done by Task 2's rename; confirm those two lines read `Test/E2E/suite/scenarios.py` before proceeding.) - [ ] **Step 2: Change the loop's `read` line and add dispatch** Find `while IFS='|' read -r ID WSPORT UDPPORT NET ORACLE TRIG CHECKS; do` and change to: ```bash while IFS='|' read -r KIND ID F2 F3 F4 F5 F6 F7; do [ -z "${ID}" ] && continue case "${KIND}" in chain) WSPORT="${F2}"; UDPPORT="${F3}"; NET="${F4}"; ORACLE="${F5}"; TRIG="${F6}"; CHECKS="${F7}" # ... existing chain scenario body unchanged, using $WSPORT/$UDPPORT/$NET/$ORACLE/$TRIG/$CHECKS ... ;; direct) CFG="${F2}" echo "══ scenario ${ID} (kind=direct cfg=${CFG}) ══" INPUT_BIN="/tmp/udpstreamer_test_input.bin" OUTPUT_BIN="/tmp/udpstreamer_test_output_${ID}.bin" rm -f "${INPUT_BIN}" "${OUTPUT_BIN}" (cd "${SCRIPT_DIR}/../datasources" && ${PY} -c "import gen_test_data; gen_test_data.generate()") timeout 15 "${MARTE_APP}" -l RealTimeLoader -f "${REPO_ROOT}/${CFG}" -s Running \ > "${OUT_DIR}/marte_${ID}.log" 2>&1 ${PY} -c " import sys; sys.path.insert(0, '${SCRIPT_DIR}/../datasources') import validate_binary as V ok, msg, details = V.validate('${INPUT_BIN}', '${OUTPUT_BIN}', label='${ID}') print(f'{\"${ID}\"}: {\"PASS\" if ok else \"FAIL\"} - {msg}') sys.exit(0 if ok else 1) " && STATUS=PASS || STATUS=FAIL echo "${STATUS}" > "${OUT_DIR}/status_${ID}.txt" ;; recorder) MCFG="${F2}"; HCFG="${F3}" echo "══ scenario ${ID} (kind=recorder marte_cfg=${MCFG} hub_cfg=${HCFG}) ══" rm -rf /tmp/streamhub_rec_e2e && mkdir -p /tmp/streamhub_rec_e2e "${STREAMHUB_EX}" -cfg "${REPO_ROOT}/${HCFG}" > "${OUT_DIR}/hub_${ID}.log" 2>&1 & HUB_PID=$! sleep 1 (cd "${SCRIPT_DIR}/../datasources" && ${PY} -c "import gen_test_data; gen_test_data.generate()") timeout 8 "${MARTE_APP}" -l RealTimeLoader -f "${REPO_ROOT}/${MCFG}" -s Running \ > "${OUT_DIR}/marte_${ID}.log" 2>&1 sleep 2 kill "${HUB_PID}" 2>/dev/null; wait "${HUB_PID}" 2>/dev/null REC_FILE="$(ls -t /tmp/streamhub_rec_e2e/*.bin 2>/dev/null | head -1)" if [ -z "${REC_FILE}" ]; then echo "${ID}: FAIL - no recorder output file found" echo "FAIL" > "${OUT_DIR}/status_${ID}.txt" else ${PY} -c " import sys; sys.path.insert(0, '${SCRIPT_DIR}/../datasources') import validate_binary as V ok, msg, details = V.validate('/tmp/udpstreamer_test_input.bin', '${REC_FILE}', label='${ID}') print(f'{\"${ID}\"}: {\"PASS\" if ok else \"FAIL\"} - {msg}') sys.exit(0 if ok else 1) " && STATUS=PASS || STATUS=FAIL echo "${STATUS}" > "${OUT_DIR}/status_${ID}.txt" fi ;; esac done <<< "${LIST}" ``` Keep the existing chain-scenario body (launch StreamHub, launch MARTeApp, run `chain-client`, `validate_waveform.py`, `plots.py`) exactly as it is today, just moved inside the `chain)` case arm with variable names unchanged (`$WSPORT`/`$UDPPORT`/etc. already match what the existing body uses). - [ ] **Step 3: Extend `results.json` aggregation to record `kind`** Find wherever `run_e2e.sh` builds the final `results.json` (reads `status_.txt` files per scenario, likely in a trailing Python block). Add `"kind"` to each scenario's record by re-deriving it from `scenarios.SCENARIOS` keyed by `id` (import `scenarios`, build `{s["id"]: s["kind"] for s in scenarios.SCENARIOS}`, look up per status file). - [ ] **Step 4: Verify** ```bash source env.sh cd Test/E2E/suite && ./run_e2e.sh --skip-build --only s52_direct_unicast 2>&1 | tail -20 ./run_e2e.sh --skip-build --only s53_direct_multicast 2>&1 | tail -20 ./run_e2e.sh --skip-build --only s54_recorder 2>&1 | tail -20 ``` Expected: each prints `: PASS`, and `results.json` shows `1 pass, 0 fail`. ```bash ./run_e2e.sh --skip-build --only s01_scalar_uint32 2>&1 | tail -10 ``` Expected: unaffected, still `PASS` — confirms the `chain)` case arm refactor didn't break existing behavior. - [ ] **Step 5: Commit** ```bash git add Test/E2E/suite/run_e2e.sh git commit -m "feat(e2e): dispatch direct/recorder scenario kinds in run_e2e.sh" ``` --- ## Task 5: Refactor `MarteController` for headless reuse + build `debugclient` **Files:** - Modify: `Client/debugger/martecontrol.go` - Create: `Test/E2E/suite/debugclient/main.go` - Create: `Test/E2E/suite/debugclient/go.mod` **Interfaces:** - Consumes: `Client/debugger`'s exported (post-refactor) `NewHeadlessMarteController(sink func(v any)) *MarteController`, and its existing `(*MarteController).Connect(host string, cmdPort, udpPort, logPort int)`, `(*MarteController).SendCommand(cmd string)`, `(*MarteController).Disconnect()`, `(*MarteController).IsConnected() bool` (all already defined in `martecontrol.go`, unchanged by this task). - Produces: `debugclient` binary accepting `-host -cmdport -udpport -logport -scenario -out -mode -dur `; exits 0 on script success, non-zero + JSON detail file (`result_.json`) on failure. `MarteController` currently has a single constructor `NewMarteController(hub *wshub.Hub) *MarteController` and routes every event through the module-level helper `broadcastHub(hub *wshub.Hub, v any)`, called at 16 sites across `Connect`/`runTCP`/`runDebugUDP`/`runLog`/`readLoop`. Introduce a `sink func(v any)` field so the same struct/methods work with or without a WebSocket hub. - [ ] **Step 1: Add a `sink` field and headless constructor** In `Client/debugger/martecontrol.go`, add to the `MarteController` struct definition (alongside the existing `hub *wshub.Hub` field): ```go sink func(v any) ``` Change `broadcastHub`'s call sites: `broadcastHub` itself stays (used by `NewMarteController`'s default sink), but every one of the 16 `broadcastHub(m.hub, X)` call sites inside `MarteController` methods becomes `m.sink(X)`. This is a mechanical find-and-replace — verify with: ```bash grep -n "broadcastHub(m.hub" Client/debugger/martecontrol.go ``` Expected before: 16 matches. Replace each with `m.sink(...)` (same arguments minus `m.hub,`). After replacement: ```bash grep -c "broadcastHub(m.hub" Client/debugger/martecontrol.go # expect 0 grep -c "m.sink(" Client/debugger/martecontrol.go # expect 16 ``` - [ ] **Step 2: Update both constructors** The current `NewMarteController` (`martecontrol.go:130-143`) reads: ```go func NewMarteController(hub *wshub.Hub) *MarteController { mc := &MarteController{ hub: hub, signals: make(map[uint32]*SignalMeta), tracedNames: make(map[string]bool), forcedState: make(map[string]string), stopCh: make(chan struct{}), } hub.SetOnClientConnect(mc.replayStateToClient) return mc } ``` Change it to set `sink` and keep the hub-specific `SetOnClientConnect` registration (only headless instances skip that): ```go func NewMarteController(hub *wshub.Hub) *MarteController { mc := &MarteController{ hub: hub, signals: make(map[uint32]*SignalMeta), tracedNames: make(map[string]bool), forcedState: make(map[string]string), stopCh: make(chan struct{}), } mc.sink = func(v any) { broadcastHub(mc.hub, v) } hub.SetOnClientConnect(mc.replayStateToClient) return mc } ``` Add a new headless constructor immediately after it, reusing the same field initializers but with `hub: nil` and no `SetOnClientConnect` call (there is no hub to register a callback on): ```go // NewHeadlessMarteController creates a MarteController with no WebSocket hub, // routing all events through sink instead (used by the debugclient E2E test tool). func NewHeadlessMarteController(sink func(v any)) *MarteController { mc := &MarteController{ hub: nil, signals: make(map[uint32]*SignalMeta), tracedNames: make(map[string]bool), forcedState: make(map[string]string), stopCh: make(chan struct{}), } mc.sink = sink return mc } ``` - [ ] **Step 3: Build to verify the refactor didn't break the browser UI path** ```bash cd Client/debugger && go build ./... && go vet ./... ``` Expected: clean build, no errors (confirms `NewMarteController`'s existing callers in `main.go` are unaffected by the added field). ```bash go test ./... 2>&1 | tail -20 ``` Expected: existing `cmd_gate_test.go` tests still pass. - [ ] **Step 4: Commit the martecontrol.go refactor** ```bash git add Client/debugger/martecontrol.go git commit -m "refactor(debugger): extract MarteController sink so it can run headless" ``` - [ ] **Step 5: Create the `debugclient` Go module** ```bash mkdir -p Test/E2E/suite/debugclient cd Test/E2E/suite/debugclient go mod init debugclient ``` Edit the generated `go.mod` to add a `replace` directive pointing at the sibling module and require it: ``` module debugclient go 1.21 require debugger v0.0.0 replace debugger => ../../../Client/debugger ``` (Confirm the actual module name declared in `Client/debugger/go.mod`'s `module` line and use that exact name in place of `debugger` above — read the file first: `head -1 Client/debugger/go.mod`.) - [ ] **Step 6: Write `debugclient/main.go`** ```go // Command debugclient drives a running MARTeApp.ex DebugService+TCPLogger // instance over TCP/UDP for the Test/E2E/suite "debug" and "tcplogger" // scenario kinds. It scripts a fixed command sequence, captures responses // and log/trace output, and reports PASS/FAIL as JSON to -out. package main import ( "encoding/json" "flag" "fmt" "os" "sync" "time" debugger "debugger" // see go.mod replace directive; import path must match Client/debugger/go.mod's module line ) type event struct { kind string data any } func main() { host := flag.String("host", "127.0.0.1", "MARTeApp host") cmdPort := flag.Int("cmdport", 8080, "DebugService TCP command port") udpPort := flag.Int("udpport", 8081, "DebugService UDP trace port") logPort := flag.Int("logport", 9090, "TCPLogger TCP port") scenario := flag.String("scenario", "", "scenario id (for output naming)") outDir := flag.String("out", "/tmp/debug_e2e", "output directory") mode := flag.String("mode", "debug", "debug|tcplogger") dur := flag.Duration("dur", 4*time.Second, "how long to run the script/collection") flag.Parse() if *scenario == "" { fmt.Fprintln(os.Stderr, "missing -scenario") os.Exit(2) } os.MkdirAll(*outDir, 0o755) var mu sync.Mutex var events []event sink := func(v any) { mu.Lock() defer mu.Unlock() events = append(events, event{kind: fmt.Sprintf("%T", v), data: v}) } mc := debugger.NewHeadlessMarteController(sink) mc.Connect(*host, *cmdPort, *udpPort, *logPort) defer mc.Disconnect() time.Sleep(500 * time.Millisecond) // allow TCP/UDP/log connects to establish var ok bool var msg string switch *mode { case "debug": ok, msg = runDebugScript(mc) case "tcplogger": ok, msg = runTcpLoggerScript(mc, *dur) default: fmt.Fprintf(os.Stderr, "unknown -mode %q\n", *mode) os.Exit(2) } time.Sleep(300 * time.Millisecond) // drain final events result := map[string]any{ "id": *scenario, "pass": ok, "detail": msg, } f, _ := os.Create(fmt.Sprintf("%s/result_%s.json", *outDir, *scenario)) json.NewEncoder(f).Encode(result) f.Close() status := "FAIL" if ok { status = "PASS" } fmt.Printf("%s: %s - %s\n", *scenario, status, msg) os.WriteFile(fmt.Sprintf("%s/status_%s.txt", *outDir, *scenario), []byte(status), 0o644) if !ok { os.Exit(1) } } // runDebugScript exercises FORCE/TRACE/BREAK over the TCP 8080 command // protocol (grammar confirmed against Source/Components/Interfaces/ // DebugService/DebugServiceBase.cpp's HandleCommand: "FORCE ", // "TRACE <0|1> [decim]", "BREAK ", // each replying "OK \n"). func runDebugScript(mc *debugger.MarteController) (bool, string) { mc.SendCommand("FORCE App.Data.Timer.Counter 42") time.Sleep(200 * time.Millisecond) mc.SendCommand("TRACE App.Data.Timer.Counter 1 1") time.Sleep(200 * time.Millisecond) mc.SendCommand("BREAK App.Data.Timer.Counter > 1000") time.Sleep(500 * time.Millisecond) mc.SendCommand("BREAK App.Data.Timer.Counter OFF") mc.SendCommand("UNFORCE App.Data.Timer.Counter") if !mc.IsConnected() { return false, "lost connection to DebugService during script" } return true, "FORCE/TRACE/BREAK sequence completed without disconnect" } // runTcpLoggerScript triggers a log-worthy event (an invalid FORCE) and // waits for a log line to arrive over the TCPLogger TCP port within dur. func runTcpLoggerScript(mc *debugger.MarteController, dur time.Duration) (bool, string) { mc.SendCommand("FORCE App.Data.DoesNotExist 1") deadline := time.Now().Add(dur) for time.Now().Before(deadline) { time.Sleep(100 * time.Millisecond) } if !mc.IsConnected() { return false, "lost connection during tcplogger wait" } return true, "connected to TCPLogger for full duration without disconnect" } ``` **Note for implementer:** `runTcpLoggerScript`'s pass condition here only checks connection liveness because log-line content isn't exposed by `MarteController`'s public API in the reviewed code — the `sink` callback (`events` slice in `main`) already captures every `{"type":"log",...}` event `runLog` emits. Before finalizing this task, inspect the exact JSON shape `runLog` sends (read `Client/debugger/martecontrol.go`'s `runLog` body around its `broadcastHub`/`m.sink` call, now at the location from Step 1) and tighten `runTcpLoggerScript` to assert on `events` containing a `"type":"log"` entry whose message references the triggered event (e.g. the unknown-signal FORCE failure), rather than only checking liveness. Do this as part of Step 7 below, not deferred further. - [ ] **Step 7: Tighten the TCPLogger assertion using the real event shape** Read `runLog`'s exact `m.sink(...)` payload shape (post Step 1 refactor) and update `runTcpLoggerScript` to inspect `events` (passed in via a new parameter) for a matching log entry: ```go func runTcpLoggerScript(mc *debugger.MarteController, dur time.Duration, events *[]event, mu *sync.Mutex) (bool, string) { mc.SendCommand("FORCE App.Data.DoesNotExist 1") deadline := time.Now().Add(dur) for time.Now().Before(deadline) { mu.Lock() for _, e := range *events { if m, ok := e.data.(map[string]any); ok && m["type"] == "log" { mu.Unlock() return true, fmt.Sprintf("received log event: %v", m) } } mu.Unlock() time.Sleep(100 * time.Millisecond) } return false, "no log event received within duration" } ``` Adjust the `sink` closure in `main()` to store `map[string]any` (unmarshal-then-remarshal, or type-assert directly if `runLog` already passes a `map[string]any`/struct — match whatever type `runLog` actually sends) and update the `runDebugScript`/`runTcpLoggerScript` call sites in `main()` to pass `&events`/`&mu`. - [ ] **Step 8: Build and smoke-test standalone** ```bash cd Test/E2E/suite/debugclient && go build -o debugclient . && go vet ./... ``` Expected: clean build. (Full end-to-end exercise against a running `MARTeApp.ex` happens in Task 6's verification step, once the debug/tcplogger cfgs exist.) - [ ] **Step 9: Commit** ```bash git add Test/E2E/suite/debugclient git commit -m "feat(e2e): add debugclient Go tool for DebugService/TCPLogger E2E scenarios" ``` --- ## Task 6: Add `debug`/`tcplogger` scenario kinds + trimmed cfgs + `run_e2e.sh` dispatch **Files:** - Create: `Test/E2E/suite/debug_e2e.cfg` - Modify: `Test/E2E/suite/scenarios.py` - Modify: `Test/E2E/suite/tests_py.py` - Modify: `Test/E2E/suite/run_e2e.sh` **Interfaces:** - Consumes: `debugclient` binary from Task 5 (`-host -cmdport -udpport -logport -scenario -out -mode`). - Produces: `results.json` entries for `s55_debug_force_trace_break` (`kind="debug"`) and `s56_tcplogger_delivery` (`kind="tcplogger"`). - [ ] **Step 1: Create the trimmed debug/tcplogger cfg** Trimmed from `Test/Configurations/combined_test.cfg`'s Thread1 (drop the Sine GAMs and `Streamer1`/`StreamerGAM1` entirely — DebugService/TCPLogger don't depend on the streaming path) plus its `+DebugService` block verbatim. Both `s55`/`s56` scenarios reuse this same cfg (only `debugclient -mode` differs). Write the full file: ``` Test/E2E/suite/debug_e2e.cfg ``` ``` /** * Trimmed single-thread configuration for the "debug"/"tcplogger" E2E * scenario kinds: exercises DebugService FORCE/TRACE/BREAK over TCP 8080 / * UDP 8081 and TCPLogger delivery over TCP 9090, with no UDPStreamer path. */ $App = { Class = RealTimeApplication +Functions = { Class = ReferenceContainer +TimerGAM = { Class = IOGAM InputSignals = { Counter = { DataSource = Timer Type = uint32 Frequency = 1000 } Time = { DataSource = Timer Type = uint32 } } OutputSignals = { Counter = { DataSource = DDB1 Type = uint32 } Time = { DataSource = DDB1 Type = uint32 } } } } +Data = { Class = ReferenceContainer DefaultDataSource = DDB1 +DDB1 = { Class = GAMDataSource } +Timer = { Class = LinuxTimer SleepNature = "Default" Signals = { Counter = { Type = uint32 } Time = { Type = uint32 } } } +Timings = { Class = TimingDataSource } } +States = { Class = ReferenceContainer +Running = { Class = RealTimeState +Threads = { Class = ReferenceContainer +Thread1 = { Class = RealTimeThread CPUs = 0x1 Functions = {TimerGAM} } } } } +Scheduler = { Class = GAMScheduler TimingDataSource = Timings } } // ── DebugService ────────────────────────────────────────────────────────────── // Patches the broker registry at startup so every signal is traceable. // TcpLogger is auto-injected on LogPort (9090) — no explicit DataSource needed. +DebugService = { Class = DebugService ControlPort = 8080 StreamPort = 8081 LogPort = 9090 StreamIP = "127.0.0.1" } ``` The `App.Data.Timer.Counter` signal path used by `debugclient`'s `FORCE`/`TRACE`/`BREAK` commands matches this file's `+Data.+Timer.Counter` signal, mirroring `combined_test.cfg`'s naming. - [ ] **Step 2: Add scenario definitions** In `scenarios.py`, after the `_RECORDER` list from Task 3, add: ```python _DEBUG = [ { "id": "s55_debug_force_trace_break", "desc": "DebugService FORCE/TRACE/BREAK over real TCP 8080 + UDP 8081", "kind": "debug", "cfg": "Test/E2E/suite/debug_e2e.cfg", "cmd_port": 8080, "udp_port": 8081, "log_port": 9090, "client_checks": [], "known_issue": None, }, ] _TCPLOGGER = [ { "id": "s56_tcplogger_delivery", "desc": "TCPLogger delivers a log line for a triggered DebugService event", "kind": "tcplogger", "cfg": "Test/E2E/suite/debug_e2e.cfg", "cmd_port": 8080, "udp_port": 8081, "log_port": 9090, "client_checks": [], "known_issue": None, }, ] SCENARIOS = SCENARIOS + _DEBUG + _TCPLOGGER ``` Extend `validate_scenario(s)` (Task 3, Step 3) with two more branches: ```python elif s["kind"] in ("debug", "tcplogger"): assert "cfg" in s and "cmd_port" in s and "udp_port" in s and "log_port" in s, \ f"{s['kind']} scenario {s['id']} missing cfg/cmd_port/udp_port/log_port" ``` - [ ] **Step 3: Extend `tests_py.py`** ```python def test_debug_and_tcplogger_scenarios_present(self): import scenarios as S kinds = {s["kind"] for s in S.SCENARIOS} self.assertIn("debug", kinds) self.assertIn("tcplogger", kinds) ``` Also update `test_all_scenarios_have_kind`'s allowed-kinds tuple if it was written differently in Task 3 (it already includes `"debug"`/`"tcplogger"` per Task 3 Step 4 — no change needed if so, just confirm). - [ ] **Step 4: Extend `run_e2e.sh`'s Python scenario-list builder** In the heredoc from Task 4 Step 1, add two more `elif` branches: ```python elif kind == "debug": print("|".join([kind, s["id"], s["cfg"], str(s["cmd_port"]), str(s["udp_port"]), str(s["log_port"]), "", ""])) elif kind == "tcplogger": print("|".join([kind, s["id"], s["cfg"], str(s["cmd_port"]), str(s["udp_port"]), str(s["log_port"]), "", ""])) ``` - [ ] **Step 5: Extend the `case "${KIND}"` dispatch in `run_e2e.sh`** ```bash debug|tcplogger) CFG="${F2}"; CMDPORT="${F3}"; UDPPORT2="${F4}"; LOGPORT="${F5}" echo "══ scenario ${ID} (kind=${KIND} cfg=${CFG}) ══" timeout 15 "${MARTE_APP}" -l RealTimeLoader -f "${REPO_ROOT}/${CFG}" -s Running \ > "${OUT_DIR}/marte_${ID}.log" 2>&1 & APP_PID=$! sleep 1 "${SCRIPT_DIR}/debugclient/debugclient" -host 127.0.0.1 \ -cmdport "${CMDPORT}" -udpport "${UDPPORT2}" -logport "${LOGPORT}" \ -mode "${KIND}" -scenario "${ID}" -out "${OUT_DIR}" -dur 4s \ > "${OUT_DIR}/client_${ID}.log" 2>&1 CLIENT_RC=$? kill "${APP_PID}" 2>/dev/null; wait "${APP_PID}" 2>/dev/null if [ "${CLIENT_RC}" -eq 0 ]; then echo "PASS" > "${OUT_DIR}/status_${ID}.txt" else echo "FAIL" > "${OUT_DIR}/status_${ID}.txt" fi ;; ``` (`debugclient` already writes its own `status_${ID}.txt`/`result_${ID}.json` per Task 5 Step 6 — this arm's own `status_${ID}.txt` write is redundant with that but harmless/idempotent; keep both for robustness in case `debugclient` crashes before writing its file, in which case this arm's fallback based on `$CLIENT_RC` is authoritative.) Also build `debugclient` alongside `chain-client` in the "build Go tools" section near the top of `run_e2e.sh` (find where `(cd "${SCRIPT_DIR}/client" && go build -o chain-client .)` runs and add an equivalent block for `${SCRIPT_DIR}/debugclient`). - [ ] **Step 6: Verify end-to-end** ```bash source env.sh cd Test/E2E/suite && ./run_e2e.sh --skip-build --only s55_debug_force_trace_break 2>&1 | tail -20 ./run_e2e.sh --skip-build --only s56_tcplogger_delivery 2>&1 | tail -20 ``` Expected: both `PASS`. If `s56` fails, inspect `client_s56_tcplogger_delivery.log` and `Build/x86-linux/E2E/suite/marte_s56_tcplogger_delivery.log` — the most likely failure mode is the trimmed cfg not producing the expected log line format from `runLog`'s parser; adjust `debug_e2e.cfg`'s triggered event or `runTcpLoggerScript`'s match condition accordingly, re-run until green (do not mark this task complete with a failing scenario). - [ ] **Step 7: Commit** ```bash git add Test/E2E/suite/debug_e2e.cfg Test/E2E/suite/scenarios.py Test/E2E/suite/tests_py.py Test/E2E/suite/run_e2e.sh git commit -m "feat(e2e): add debug/tcplogger E2E scenario kinds using debugclient" ``` --- ## Task 7: Restructure build/coverage flow (instrument-first, double-run, new skip flags) + port stress multi-fragment fix **Files:** - Modify: `Test/E2E/suite/run_e2e.sh` - Modify: `Test/E2E/suite/stress.py` - Modify: `Test/E2E/suite/collect.py` (only the `main()` CLI surface — add no new suites, just confirm invocation still matches new call sites) **Interfaces:** - Consumes: `Test/E2E/suite/stress_run.py::main()` CLI (`--marte --hub --client --work --out --only --axis`, unchanged). - Produces: `run_e2e.sh` flags `--skip-coverage`, `--skip-stress`, `--skip-datasources`, `--skip-recorder`, `--skip-debug`, `--skip-tcplogger` (all suites run by default); `stress.py`'s `STRESS_CASES` includes multi-fragment size cases; a `stress_results.json` is always produced (unless `--skip-stress`) at `${OUT_DIR}/stress/stress_results.json`. - [ ] **Step 1: Port `stress.py`'s multi-fragment extension from the stale branch** ```bash git show feature/stress-report-integration:Test/E2E/chain/stress.py > /tmp/stress_feature_branch.py diff /tmp/stress_feature_branch.py Test/E2E/suite/stress.py ``` Apply these specific changes to `Test/E2E/suite/stress.py` (per the pre-gathered diff summary — re-derive exact line numbers from the `diff` output above since paths/line numbers shifted after Task 2's rename): 1. Add module-level constant `UDPS_CLIENT_MAX_PACKET_BYTES = 1048576` (matching `Source/Components/Interfaces/UDPStream/UDPSClient.h`'s `UDPS_CLIENT_MAX_PACKET_BYTES`). 2. `_source()` gains a `row_dt=S.ROW_DT` parameter; change its rate formula from the hardcoded `elements * 1000.0` to `elements / row_dt`. 3. `mk_stress()` gains `row_dt=None, num_rows=None, producer_hz=None` parameters, wired into the returned case dict (previously always `None`). 4. Add `_BIG_SIZE` cases (50000/100000/250000 elements, slowed `producer_hz`) appended to both `_DS_SIZE` and `_HUB_SIZE`. 5. `validate_case()`'s packet-size ceiling check changes from `>= 65536` to `>= UDPS_CLIENT_MAX_PACKET_BYTES`. - [ ] **Step 2: Verify stress.py still validates** ```bash cd Test/E2E/suite && python3 -c " import stress as ST for c in ST.STRESS_CASES: ST.validate_case(c) print(f'{len(ST.STRESS_CASES)} stress cases validated OK') " ``` Expected: no assertion errors, case count is 27 (original) + 6 (2 axes x 3 `_BIG_SIZE` levels) = 33. - [ ] **Step 3: Add new flags to `run_e2e.sh`'s arg parser** Extend the `case "$1" in` block (currently lines 30-36) to: ```bash --skip-build) SKIP_BUILD=1 ;; --only) shift; ONLY="$1" ;; --pdf-only) PDF_ONLY=1 ;; --cpp-coverage) CPP_COV=1 ;; --skip-coverage) CPP_COV=0 ;; --skip-stress) SKIP_STRESS=1 ;; --skip-datasources) SKIP_DATASOURCES=1 ;; --skip-recorder) SKIP_RECORDER=1 ;; --skip-debug) SKIP_DEBUG=1 ;; --skip-tcplogger) SKIP_TCPLOGGER=1 ;; --help|-h) echo "Usage: $0 [--skip-build] [--only ] [--pdf-only] [--skip-coverage] [--skip-stress] [--skip-datasources] [--skip-recorder] [--skip-debug] [--skip-tcplogger]"; exit 0 ;; *) echo "unknown arg $1" >&2; exit 2 ;; ``` Add the new variable defaults alongside the existing `SKIP_BUILD=0` etc.: ```bash SKIP_STRESS=0 SKIP_DATASOURCES=0 SKIP_RECORDER=0 SKIP_DEBUG=0 SKIP_TCPLOGGER=0 ``` Wire `SKIP_DATASOURCES`/`SKIP_RECORDER`/`SKIP_DEBUG`/`SKIP_TCPLOGGER` into the Python scenario-list builder's filter (skip emitting those `kind`s when set) — add to the heredoc's env/argv: ```python import sys, os sys.path.insert(0, os.path.join(os.getcwd(), "Test/E2E/suite")) import scenarios as S only = sys.argv[1] if len(sys.argv) > 1 else "" skip_kinds = set(sys.argv[2].split(",")) if len(sys.argv) > 2 and sys.argv[2] else set() for s in S.SCENARIOS: if only and s["id"] != only: continue if s["kind"] in skip_kinds: continue ... ``` and change the invocation to pass a comma-joined skip-kinds argument built from the shell flags: ```bash SKIP_KINDS="" [ "${SKIP_DATASOURCES}" -eq 1 ] && SKIP_KINDS="${SKIP_KINDS}direct," [ "${SKIP_RECORDER}" -eq 1 ] && SKIP_KINDS="${SKIP_KINDS}recorder," [ "${SKIP_DEBUG}" -eq 1 ] && SKIP_KINDS="${SKIP_KINDS}debug," [ "${SKIP_TCPLOGGER}" -eq 1 ] && SKIP_KINDS="${SKIP_KINDS}tcplogger," LIST="$(${PY} - "${ONLY}" "${SKIP_KINDS}" <<'PY' ... PY )" ``` - [ ] **Step 4: Restructure the top-level flow into named phases** Replace the existing single-pass "build → scenarios → coverage-rebuild → collect → restore" sequence with: ```bash # Phase 1: normal build if [ "${SKIP_BUILD}" -eq 0 ]; then make -C "${REPO_ROOT}" -f Makefile.gcc core apps TARGET="${TARGET}" 2>&1 | tail -1 fi # Phase 2: authoritative scenario pass (normal binaries) - existing scenario loop from Task 4/6, unchanged run_scenario_matrix() { local RESULTS_SUFFIX="$1" # ... existing `while IFS='|' read -r KIND ID ...; do ... done <<< "${LIST}"` loop body, unchanged ... } run_scenario_matrix "primary" # Phase 3: stress (normal binaries, always, never instrumented) if [ "${SKIP_STRESS}" -eq 0 ]; then ${PY} "${SCRIPT_DIR}/stress_run.py" --marte "${MARTE_APP}" --hub "${STREAMHUB_EX}" \ --client "${CLIENT}" --work "/tmp/chain_stress" --out "${OUT_DIR}/stress" || true fi # Phase 4: instrumented rebuild + coverage-only scenario re-run + unit tests if [ "${CPP_COV}" -eq 1 ]; then COV_O="--coverage" COV_L="-Wl,--no-as-needed -fPIC --coverage" make -C "${REPO_ROOT}" -f Makefile.gcc clean >/dev/null 2>&1 || true make -C "${REPO_ROOT}" -f Makefile.gcc core TARGET="${TARGET}" OPTIM="${COV_O}" LFLAGS="${COV_L}" 2>&1 | tail -1 for d in Test/Components/DataSources/UDPStreamer Test/Components/DataSources/UDPStreamerClient Test/Applications/StreamHub Test/GTest Test/Integration; do make -C "${REPO_ROOT}/${d}" -f Makefile.gcc TARGET="${TARGET}" OPTIM="${COV_O}" LFLAGS="${COV_L}" 2>&1 | tail -1 done OUT_DIR="${OUT_DIR}/coverage_pass" run_scenario_matrix "coverage" || true ${PY} "${SCRIPT_DIR}/collect.py" --repo "${REPO_ROOT}" --target "${TARGET}" \ --out "${OUT_DIR}" --work "${WORK}" --cpp-coverage || true make -C "${REPO_ROOT}" -f Makefile.gcc clean >/dev/null 2>&1 || true make -C "${REPO_ROOT}" -f Makefile.gcc core apps TARGET="${TARGET}" 2>&1 | tail -1 (cd "${SCRIPT_DIR}/client" && go build -o chain-client . >/dev/null 2>&1) || true (cd "${SCRIPT_DIR}/debugclient" && go build -o debugclient . >/dev/null 2>&1) || true else ${PY} "${SCRIPT_DIR}/collect.py" --repo "${REPO_ROOT}" --target "${TARGET}" \ --out "${OUT_DIR}" --work "${WORK}" || true fi # Phase 5: report ${PY} "${SCRIPT_DIR}/report_build.py" --repo "${REPO_ROOT}" --results "${OUT_DIR}/results.json" \ --work "${WORK}" --out "${OUT_DIR}" --stress-results "${OUT_DIR}/stress/stress_results.json" # ... existing Typst PDF compile block, unchanged ... ``` Wrapping the existing scenario loop body in a `run_scenario_matrix()` shell function requires moving the `LIST="$(...)"` build step (Step 3, already parameterized by `SKIP_KINDS`) to run once before Phase 2, and having `run_scenario_matrix` iterate over the already-computed `${LIST}` variable (read-only reuse — no need to recompute `LIST` for the coverage pass since it's the same scenario set). The `OUT_DIR="${OUT_DIR}/coverage_pass" run_scenario_matrix "coverage"` line's env-var-prefix trick only affects the subshell's view of `OUT_DIR` if `run_scenario_matrix` is a function (not a subshell) reading a global — since bash functions share the parent shell's variables, use a local override inside the call instead: ```bash ( OUT_DIR="${OUT_DIR}/coverage_pass" mkdir -p "${OUT_DIR}" run_scenario_matrix "coverage" ) || true ``` This runs `run_scenario_matrix` in a subshell so the outer `OUT_DIR` is unaffected afterward, while the coverage-only pass writes its (discarded) results/logs to a separate subdirectory. - [ ] **Step 5: Verify default run and skip flags** ```bash source env.sh cd Test/E2E/suite ./run_e2e.sh --skip-build --only s01_scalar_uint32 --skip-stress --skip-datasources --skip-recorder --skip-debug --skip-tcplogger --skip-coverage 2>&1 | tail -20 ``` Expected: fast single-scenario run, no coverage rebuild, no stress phase — same behavior as the pre-Task-7 fast-iteration workflow. ```bash ./run_e2e.sh --skip-build --skip-stress 2>&1 | tail -40 ``` Expected: full scenario matrix (chain+direct+recorder+debug+tcplogger) runs once for results, then again (coverage_pass subdir) under instrumented binaries, `stress_results.json` absent/skipped, final `report_data.json`/PDF produced without a stress section (or with a "stress: skipped" placeholder — verify `report_build.py` from Task 8 handles a missing `--stress-results` file gracefully; if Task 8 hasn't landed yet in execution order, this check is deferred to that task's verification instead). - [ ] **Step 6: Commit** ```bash git add Test/E2E/suite/run_e2e.sh Test/E2E/suite/stress.py git commit -m "feat(e2e): instrument-first coverage flow with double-run + new skip flags; port stress multi-fragment sizing" ``` --- ## Task 8: Extend `report_build.py` + `E2E_Report.typ` with per-kind and stress sections **Files:** - Modify: `Test/E2E/suite/report_build.py` - Modify: `Test/E2E/suite/E2E_Report.typ` - Modify: `Test/E2E/suite/tests_py.py` **Interfaces:** - Consumes: `results.json` entries now carrying `kind` (Tasks 3/4/6); `stress_results.json` = `{"overall","cases":[...]}` (from `stress_run.py`, unchanged shape). - Produces: `report_data.json` gains `"direct"`, `"recorder"`, `"debug"`, `"tcplogger"`, `"stress"`, `"stress_plots"` top-level keys; `headline()`'s return dict gains matching scalar fields; `regression()` compares them against `history.jsonl`; `E2E_Report.typ` renders one section per kind. - [ ] **Step 1: Port the stress reporting functions from the stale branch** ```bash git show feature/stress-report-integration:Test/E2E/chain/report_build.py > /tmp/report_build_feature_branch.py diff /tmp/report_build_feature_branch.py Test/E2E/suite/report_build.py ``` Port these specific additions (adapting names/paths for the current file — the stale branch predates the `kind`-based scenario model, so integrate its logic as new standalone functions rather than replacing existing ones): - `_STRESS_LABELS`, `_STRESS_DIRECTION` (module-level dicts). - `build_stress(sr)` — reads `stress_results.json`'s `cases` list, groups by `axis`, returns `{"overall", "by_axis": {axis: [case,...]}, "n_cases"}`. - `stress_headline(stress)` — flattens key scalar stress metrics (e.g. `stress_pass_rate`, `stress_max_rss_mb`) into the same shape `headline()` returns. - `_STRESS_AXIS_METRICS` — per-axis field-pairs table used by `stress_plots`. - `stress_plots(by_axis, out)` — writes per-axis scaling PNGs to `out`, returns list of paths (mirrors `trend_plots`'s return-list-of-paths pattern). - [ ] **Step 2: Add `build_direct`/`build_recorder`/`build_debug`/`build_tcplogger`** These are simpler than `build_stress` since they read the same `results.json` shape `build_e2e()` already consumes, just filtered by `kind`. Add: ```python def build_by_kind(results, kind): """Filter results.json's scenario list to one kind, mirroring build_e2e()'s per-scenario shape but without the chain-specific corr/nrmse/perf fields.""" scenarios = [s for s in results.get("scenarios", []) if s.get("kind") == kind] n_pass = sum(1 for s in scenarios if s.get("status") == "PASS") n_fail = sum(1 for s in scenarios if s.get("status") == "FAIL") return { "kind": kind, "n_pass": n_pass, "n_fail": n_fail, "n_total": len(scenarios), "scenarios": scenarios, } ``` (Confirm `results.json`'s actual top-level shape — re-read `build_e2e(results, work)`'s existing body to see whether scenario records live under `results["scenarios"]` or a differently-named key, and match that exactly; adjust the `.get("scenarios", [])` key name if it differs.) - [ ] **Step 3: Wire everything into `main()`** Extend the argument parser: ```python ap.add_argument("--stress-results", default=None) ``` After the existing `e2e = build_e2e(results, work)` call, add: ```python direct = build_by_kind(results, "direct") recorder = build_by_kind(results, "recorder") debug = build_by_kind(results, "debug") tcplogger = build_by_kind(results, "tcplogger") stress = None stress_plot_paths = [] if args.stress_results and os.path.exists(args.stress_results): with open(args.stress_results) as f: sr = json.load(f) stress = build_stress(sr) stress_plot_paths = stress_plots(stress["by_axis"], out) ``` Extend `headline(e2e, ut, cov)`'s call site and signature to also fold in the new sections: ```python hl = headline(e2e, ut, cov) hl.update({ "direct_pass": direct["n_pass"], "direct_fail": direct["n_fail"], "recorder_pass": recorder["n_pass"], "recorder_fail": recorder["n_fail"], "debug_pass": debug["n_pass"], "debug_fail": debug["n_fail"], "tcplogger_pass": tcplogger["n_pass"], "tcplogger_fail": tcplogger["n_fail"], }) if stress: hl.update(stress_headline(stress)) ``` Extend `_DIRECTION` (module-level) with the new headline keys: ```python _DIRECTION.update({ "direct_pass": True, "direct_fail": False, "recorder_pass": True, "recorder_fail": False, "debug_pass": True, "debug_fail": False, "tcplogger_pass": True, "tcplogger_fail": False, }) _DIRECTION.update(_STRESS_DIRECTION) ``` Extend the final `report_data` dict and `history.jsonl` append to include the new sections: ```python report_data = { "meta": meta, "e2e": e2e, "unit_tests": ut, "coverage": cov, "direct": direct, "recorder": recorder, "debug": debug, "tcplogger": tcplogger, "stress": stress, "stress_plots": stress_plot_paths, "regression": regression(hl, prev_hl), "headline": hl, "trend_plots": trend_plots(history, out), "history_len": len(history), "is_first_run": is_first_run, } ``` (Match whatever the existing `main()` already names its local variables for `meta`/`prev_hl`/`history`/`is_first_run` — read the current file's `main()` body first and adapt variable names exactly rather than introducing new ones.) - [ ] **Step 4: Add Typst sections** In `E2E_Report.typ`, after the existing "Scenarios" section (around line 224 per the pre-gathered survey), add: ```typst #if report.direct != none [ == Direct Round-Trip (UDPStreamer ↔ UDPStreamerClient) #table( columns: 3, [*Scenario*], [*Status*], [*Detail*], ..report.direct.scenarios.map(s => (s.id, s.status, s.at("detail", default: ""))).flatten() ) ] #if report.recorder != none [ == Recorder (BinaryRecorder disk output) #table( columns: 3, [*Scenario*], [*Status*], [*Detail*], ..report.recorder.scenarios.map(s => (s.id, s.status, s.at("detail", default: ""))).flatten() ) ] #if report.debug != none [ == Debug Service E2E #table( columns: 3, [*Scenario*], [*Status*], [*Detail*], ..report.debug.scenarios.map(s => (s.id, s.status, s.at("detail", default: ""))).flatten() ) ] #if report.tcplogger != none [ == TCPLogger E2E #table( columns: 3, [*Scenario*], [*Status*], [*Detail*], ..report.tcplogger.scenarios.map(s => (s.id, s.status, s.at("detail", default: ""))).flatten() ) ] #if report.stress != none [ == Stress Tests #for (axis, cases) in report.stress.by_axis [ === #axis #table( columns: 4, [*Level*], [*Status*], [*Peak RSS (MB)*], [*Zoom p95 (ms)*], ..cases.map(c => (str(c.level), c.status, str(c.at("marte_rss_mb", default: 0)), str(c.at("zoom_p95_ms", default: 0)))).flatten() ) ] #for p in report.stress_plots [ #image(p) ] ] else [ == Stress Tests _Not run this session (use without --skip-stress to include)._ ] ``` (Confirm Typst's exact JSON-access syntax matches the version in use — check the existing template's patterns for accessing `report.e2e.scenarios` or similar nested fields, and mirror that exact syntax rather than the illustrative dot-access above if the template uses a different accessor style, e.g. `report.at("e2e")`.) - [ ] **Step 5: Extend `tests_py.py`** ```python def test_build_by_kind_filters_correctly(self): import report_build as R results = {"scenarios": [ {"id": "a", "kind": "direct", "status": "PASS"}, {"id": "b", "kind": "chain", "status": "PASS"}, {"id": "c", "kind": "direct", "status": "FAIL"}, ]} d = R.build_by_kind(results, "direct") self.assertEqual(d["n_pass"], 1) self.assertEqual(d["n_fail"], 1) self.assertEqual(d["n_total"], 2) ``` - [ ] **Step 6: Run the full pipeline and inspect the PDF** ```bash source env.sh cd Test/E2E/suite && ./run_e2e.sh 2>&1 | tail -60 ``` Expected: completes with all six scenario kinds + stress + coverage, final line `PDF: .../E2E_Report.pdf`. ```bash python3 -m unittest tests_py -v 2>&1 | tail -10 ``` Expected: all pass, including the new `test_build_by_kind_filters_correctly`. Open/inspect `Build/x86-linux/E2E/suite/E2E_Report.pdf` (or extract text) and confirm it contains "Direct Round-Trip", "Recorder", "Debug Service E2E", "TCPLogger E2E", and "Stress Tests" section headers: ```bash pdftotext Build/x86-linux/E2E/suite/E2E_Report.pdf - | grep -E "Direct Round-Trip|Recorder|Debug Service E2E|TCPLogger E2E|Stress Tests" ``` Expected: all 5 strings found. - [ ] **Step 7: Commit** ```bash git add Test/E2E/suite/report_build.py Test/E2E/suite/E2E_Report.typ Test/E2E/suite/tests_py.py git commit -m "feat(e2e): render direct/recorder/debug/tcplogger/stress sections in the unified report" ``` --- ## Task 9: Retire duplicate/superseded suites **Files:** - Delete: `Test/E2E/streamhub/` (entire directory) - Delete: `run_e2e_test.sh` - Delete: `Test/E2E/datasources/run_e2e_report.sh` - Delete: `Test/E2E/datasources/E2E_Report.typ` - Delete: `Test/E2E/recorder/run_recorder_e2e.sh` - Delete: `run_combined_test.sh` - Modify: `CLAUDE.md` / `AGENTS.md` (remove references to the deleted scripts) **Interfaces:** - Consumes: nothing (this task only removes files whose functionality is now covered by Tasks 4/6/8). - Produces: no dangling references to deleted files anywhere in the repo. - [ ] **Step 1: Confirm no remaining references before deleting** ```bash grep -rln "run_e2e_test.sh\|run_recorder_e2e.sh\|run_e2e_report.sh\|run_combined_test.sh\|Test/E2E/streamhub" \ --include="*.md" --include="*.sh" --include="*.py" --include="*.go" . 2>/dev/null ``` Review every hit. Expected hits at this point: `CLAUDE.md`/`AGENTS.md` (to be edited in Step 3), and possibly this plan document itself (ignore). If any *other* script still shells out to one of these (e.g. a CI config file not yet seen), stop and flag it rather than deleting blindly — the spec's retirement list assumed no other consumers exist based on the Task-0 survey, but re-verify before executing. - [ ] **Step 2: Delete the files** ```bash git rm -r Test/E2E/streamhub git rm run_e2e_test.sh git rm Test/E2E/datasources/run_e2e_report.sh git rm Test/E2E/datasources/E2E_Report.typ git rm Test/E2E/recorder/run_recorder_e2e.sh git rm run_combined_test.sh ``` - [ ] **Step 3: Clean up doc references** ```bash grep -n "run_e2e_test.sh\|run_recorder_e2e.sh\|run_e2e_report.sh\|run_combined_test.sh\|Test/E2E/streamhub" CLAUDE.md AGENTS.md ``` For each match, remove the reference (delete the sentence/line if it exists solely to describe the deleted script) rather than leaving a dangling mention. If `CLAUDE.md`'s demo-scripts line currently reads something like `` `./run_combined_test.sh`, `./run_streamhub.sh` ``, drop `` `./run_combined_test.sh`, `` and keep `` `./run_streamhub.sh` `` (that one is NOT being deleted — it's a pure interactive demo launcher unrelated to any retired test suite, confirmed in the original survey). - [ ] **Step 4: Verify nothing else breaks** ```bash source env.sh cd Test/E2E/suite && ./run_e2e.sh --skip-build --only s01_scalar_uint32 --skip-coverage --skip-stress 2>&1 | tail -10 ``` Expected: still passes (confirms the deletions didn't remove anything `run_e2e.sh` itself depends on). ```bash cd /home/martino/Projects/MARTe_Integrated_components grep -rn "Test/E2E/streamhub\|run_e2e_test\|run_recorder_e2e\|run_e2e_report\|run_combined_test" --include="*.md" --include="*.sh" . ``` Expected: no output (all references cleaned up). - [ ] **Step 5: Commit** ```bash git add -A CLAUDE.md AGENTS.md git commit -m "chore(e2e): retire streamhub/datasources/recorder standalone scripts superseded by run_e2e.sh" ``` --- ## Task 10: Full pipeline verification and stale branch cleanup **Files:** none (verification-only task). **Interfaces:** none. - [ ] **Step 1: Full clean run from scratch** ```bash source env.sh && export MAKEDEFAULTDIR=$MARTe2_DIR/MakeDefaults cd Test/E2E/suite && ./run_e2e.sh 2>&1 | tee /tmp/full_run.log | tail -100 ``` Expected: exits 0 (or check `results.json`'s overall status explicitly), produces `Build/x86-linux/E2E/suite/E2E_Report.pdf`. - [ ] **Step 2: Verify GTest count and full pass** ```bash ./Build/x86-linux/GTest/MainGTest.ex 2>&1 | tail -5 ``` Expected: `[ PASSED ] N tests.` with N = 79 (76 original + 3 `BoundsCheckTest` from Task 1; adjust if `WSServerBufferTest.cpp` also contributed previously-uncounted tests — check the actual delta and confirm 0 failures either way). - [ ] **Step 3: Verify `report_data.json` structure** ```bash python3 -c " import json d = json.load(open('Build/x86-linux/E2E/suite/report_data.json')) for k in ('e2e','unit_tests','coverage','direct','recorder','debug','tcplogger','stress','regression','headline'): print(k, 'present' if k in d else 'MISSING') " ``` Expected: all keys print `present`. - [ ] **Step 4: Verify coverage rose vs. the pre-unification baseline** ```bash python3 -c " import json d = json.load(open('Build/x86-linux/E2E/suite/report_data.json')) print('C++ coverage:', d['coverage']) " ``` Compare against the coverage percentage recorded in `Build/x86-linux/E2E/suite/history.jsonl`'s second-to-last entry (the last pre-unification run). Expected: the new C++ coverage % is higher (E2E-exercised code now counts), or at minimum not lower — if it's lower, investigate before declaring this task done (likely cause: the coverage-pass `run_scenario_matrix` subshell in Task 7 not actually exercising the instrumented binaries — check that `${MARTE_APP}`/`${STREAMHUB_EX}` path variables still point at the just-rebuilt instrumented binaries at that point in the script, not stale paths from Phase 1's normal build). - [ ] **Step 5: Confirm fast single-scenario iteration still works** ```bash ./run_e2e.sh --skip-build --only s01_scalar_uint32 --skip-coverage --skip-stress --skip-datasources --skip-recorder --skip-debug --skip-tcplogger 2>&1 | tail -10 ``` Expected: runs in a few seconds, `PASS`, no coverage/stress/other-kind overhead — confirms the default-heavy `run_e2e.sh` still supports the fast dev-loop workflow via skip flags. - [ ] **Step 6: Reconcile and delete the stale branch** ```bash git log --oneline main..feature/stress-report-integration git branch -D feature/stress-report-integration ``` Ask the user for explicit confirmation before running the following (deletes the remote ref, a shared/visible action): ```bash git push origin --delete feature/stress-report-integration ``` - [ ] **Step 7: Final full-suite commit (if any stray files remain uncommitted)** ```bash git status ``` If clean, no action needed. If any generated artifacts (e.g. `depends.x86-linux` regenerated files) show as modified, follow the repo's existing convention (per `AGENTS.md`/prior session history: these are build-generated and typically not re-committed unless the repo's `.gitignore` doesn't already exclude them — check `git status` output against `.gitignore` before adding anything).