martino/MARTe-Integrated-Components
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
462b05b71ac3c6606f94009c74108185cd95792d
MARTe-Integrated-Components/docs/superpowers/plans/2026-07-01-unified-test-reporting.md
T
Martino Ferrari 462b05b71a docs(testing): implementation plan for unified test/E2E/reporting/coverage pipeline 
Plan derived from the approved 2026-07-01 design spec; covers scenario
kind unification (chain/direct/recorder/debug/tcplogger), instrument-first
double-run coverage, and DebugService/TCPLogger E2E via debugclient.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-07-01 18:29:29 +02:00

66 KiB
Raw Blame History

Unified Test/E2E/Reporting/Coverage Pipeline Implementation Plan

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: Consolidate every test suite in the repo (unit tests, chain/stress/datasources/recorder E2E, new DebugService/TCPLogger E2E) behind one entry point (Test/E2E/suite/run_e2e.sh) producing one results.json/unit_tests.json/coverage.json/report_data.json/history.jsonl/PDF, with accurate E2E-inclusive coverage and uncontaminated performance metrics.

Architecture: Extend the existing mature Test/E2E/chain pipeline (renamed Test/E2E/suite) rather than building a new one: add a kind discriminator to the scenario model so direct (datasources), recorder, debug, and tcplogger scenarios flow through the same results.json/dispatch loop as chain scenarios; reorder the build so unit tests + all scenario kinds run once on normal binaries (authoritative pass/fail + perf) and once on instrumented binaries (coverage only, perf discarded), with stress always running once, uninstrumented; extend report_build.py/E2E_Report.typ with sections per kind plus the already-designed stress section.

Tech Stack: Bash (orchestration), Python 3 (scenario generation/validation/reporting, lcov/gcov), Go (test clients), C++ (GTest, MARTe2), Typst (PDF report).

Global Constraints

  • No STL/<cstdint> in MARTe2-style components (Test/Applications/StreamHub, Source/Components/*) — use MARTe2 types (uint32, uint64, etc. from GeneralDefinitions.h).
  • Build via source env.sh && export MAKEDEFAULTDIR=$MARTe2_DIR/MakeDefaults && make -f Makefile.gcc ... — never pipe source env.sh (breaks env propagation in a subshell).
  • All 76 existing GTests must keep passing throughout; no task may leave make test in a broken state at its end.
  • run_e2e.sh --skip-build --only <id> (single-scenario fast iteration) must keep working after every task.
  • EUPL v1.1 license headers on new C++ sources (match existing files' header style).
  • Stress scenarios must never run against --coverage-instrumented binaries (perf/scaling measurements would be skewed).

Task 1: Fix Test/Applications/StreamHub build break

Files:

  • Modify: Test/Applications/StreamHub/BoundsCheckTest.cpp
  • Modify: Test/Applications/StreamHub/Makefile.inc:25
  • Test: Test/Applications/StreamHub/BoundsCheckTest.cpp (GTest, run via MainGTest.ex)

Interfaces:

  • Consumes: nothing new.
  • Produces: make -C Test/Applications/StreamHub -f Makefile.gcc succeeds; MainGTest.ex gains 3 new passing tests (BoundsCheckTest.OverflowRejected, .NormalCasePasses, .NumRowsNumColsOverflow).

BoundsCheckTest.cpp uses <cstdint> (uint32_t/uint64_t), which fails to compile under this project's -std=c++98 flags — this breaks Test/Applications/StreamHub's generic dependsRaw rule (g++ -MM *.c*, scans all .cpp files regardless of OBJSX membership) and therefore breaks make test repo-wide. WSServerBufferTest.cpp already uses MARTe2 types correctly and needs no source change, only OBJSX registration.

  • Step 1: Replace <cstdint> types with MARTe2 types in BoundsCheckTest.cpp

Replace the full file content (current file confirmed 74 lines) — every uint32_t → uint32, every uint64_t → uint64, drop the <cstdint> and <cstring> includes (both unused beyond the removed types; <cstring> was never used):

/**
 * @file BoundsCheckTest.cpp
 * @brief Reproduction tests for HI-1 (integer overflow in bounds check) and
 *        HI-4 (unclamped forcedValue memcpy).
 *
 * These tests verify that the 64-bit bounds check pattern correctly rejects
 * crafted payloads whose 32-bit multiply would overflow, and that the
 * forcedValue clamp prevents OOB reads.
 */

#include <gtest/gtest.h>
#include "GeneralDefinitions.h"

// HI-1: Verify that a 64-bit bounds check rejects a payload where
// elemsToRead * wireElemBytes would overflow uint32.
TEST(BoundsCheckTest, OverflowRejected) {
    // Simulate: numSamples = 0x20000001, wireElemBytes = 8
    // 32-bit: 0x20000001 * 8 = 0x8 (overflow!)
    // 64-bit: 0x100000008 (correctly large, > any reasonable payload size)
    MARTe::uint32 elemsToRead = 0x20000001u;
    MARTe::uint32 wireElemBytes = 8u;
    MARTe::uint32 off = 12u; // after HRT + numSamples
    MARTe::uint32 size = 1400u; // typical max payload

    // This is the FIXED pattern (64-bit):
    MARTe::uint64 bytesNeeded = static_cast<MARTe::uint64>(off) +
                           static_cast<MARTe::uint64>(elemsToRead) *
                           static_cast<MARTe::uint64>(wireElemBytes);
    // Should reject (bytesNeeded >> size)
    EXPECT_GT(bytesNeeded, static_cast<MARTe::uint64>(size))
        << "64-bit check should detect overflow that 32-bit would miss";

    // Verify the OLD (buggy) 32-bit pattern would have passed:
    MARTe::uint32 oldCheck = off + (elemsToRead * wireElemBytes);
    // On 32-bit: 0x20000001 * 8 = 0x100000008 truncated to 0x8
    // off + 0x8 = 20, which is < 1400, so the old check would pass (bug!)
    // On 64-bit: the multiply doesn't overflow, so oldCheck is huge
    // This test documents that the 64-bit fix is necessary on 32-bit platforms
    // and correct on 64-bit.
    (void) oldCheck;
}

// HI-1: Verify a normal (non-overflow) case passes the 64-bit check.
TEST(BoundsCheckTest, NormalCasePasses) {
    MARTe::uint32 elemsToRead = 100u;
    MARTe::uint32 wireElemBytes = 4u;
    MARTe::uint32 off = 12u;
    MARTe::uint32 size = 500u;

    MARTe::uint64 bytesNeeded = static_cast<MARTe::uint64>(off) +
                           static_cast<MARTe::uint64>(elemsToRead) *
                           static_cast<MARTe::uint64>(wireElemBytes);
    EXPECT_LE(bytesNeeded, static_cast<MARTe::uint64>(size))
        << "normal case should pass the bounds check";
}

// HI-1: Verify numRows * numCols overflow is detected.
TEST(BoundsCheckTest, NumRowsNumColsOverflow) {
    MARTe::uint32 numRows = 0x10000u;
    MARTe::uint32 numCols = 0x10000u;

    // 32-bit: 0x10000 * 0x10000 = 0 (overflow!)
    MARTe::uint32 oldResult = numRows * numCols;
    EXPECT_EQ(oldResult, 0u) << "32-bit multiply should overflow to 0";

    // 64-bit fix:
    MARTe::uint64 newResult = static_cast<MARTe::uint64>(numRows) *
                         static_cast<MARTe::uint64>(numCols);
    EXPECT_EQ(newResult, static_cast<MARTe::uint64>(0x100000000ULL))
        << "64-bit multiply should give correct result";
    EXPECT_GT(newResult, static_cast<MARTe::uint64>(0x100000u))
        << "should exceed the sanity cap, triggering rejection";
}
  • Step 2: Register both orphaned files in OBJSX

Edit Test/Applications/StreamHub/Makefile.inc:25, change:

OBJSX = TriggerEngineSrc.x BinaryRecorderSrc.x SignalRingBufferGTest.x TriggerEngineGTest.x LTTBGTest.x BinaryRecorderGTest.x

to:

OBJSX = TriggerEngineSrc.x BinaryRecorderSrc.x SignalRingBufferGTest.x TriggerEngineGTest.x LTTBGTest.x BinaryRecorderGTest.x BoundsCheckTest.x WSServerBufferTest.x
  • Step 3: Rebuild and verify
source env.sh && export MAKEDEFAULTDIR=$MARTe2_DIR/MakeDefaults
rm -f Test/Applications/StreamHub/depends.x86-linux Test/Applications/StreamHub/dependsRaw.x86-linux
make -C Test/Applications/StreamHub -f Makefile.gcc
make -C Test/GTest -f Makefile.gcc
./Build/x86-linux/GTest/MainGTest.ex --gtest_filter='BoundsCheckTest.*:WSServerBufferTest.*'

Expected: clean build, both filters show [ PASSED ] for all tests (3 BoundsCheckTest + whatever WSServerBufferTest already defines).

./Build/x86-linux/GTest/MainGTest.ex 2>&1 | tail -5

Expected: [ PASSED ] N tests. with N = 76 + (new test count), 0 failures.

  • Step 4: Commit
git add Test/Applications/StreamHub/BoundsCheckTest.cpp Test/Applications/StreamHub/WSServerBufferTest.cpp Test/Applications/StreamHub/Makefile.inc
git commit -m "fix(streamhub-test): unblock make test by fixing BoundsCheckTest C++98 build and registering both orphaned GTest files"

Task 2: Rename Test/E2E/chain → Test/E2E/suite, run_chain_e2e.sh → run_e2e.sh

Files:

  • Rename (git mv): Test/E2E/chain/ → Test/E2E/suite/ (all contents, including client/ Go module)
  • Rename (git mv): Test/E2E/suite/run_chain_e2e.sh → Test/E2E/suite/run_e2e.sh
  • Modify: CLAUDE.md (Build & Test section, streaming-chain E2E paragraph)
  • Modify: AGENTS.md (any Test/E2E/chain/run_chain_e2e.sh references)

Interfaces:

  • Consumes: nothing new.
  • Produces: Test/E2E/suite/run_e2e.sh --skip-build --only s01_scalar_uint32 works identically to the old run_chain_e2e.sh invocation.

This is a pure rename with no logic change — do it as its own task/commit so later tasks' diffs are readable against the new paths.

  • Step 1: Rename directory and script
git mv Test/E2E/chain Test/E2E/suite
git mv Test/E2E/suite/run_chain_e2e.sh Test/E2E/suite/run_e2e.sh
  • Step 2: Fix internal self-references
grep -rn "chain_e2e\|Test/E2E/chain\|run_chain_e2e" Test/E2E/suite/ 2>/dev/null

For each hit found (expect at least /tmp/chain_e2e work-dir paths in run_e2e.sh, and possibly client/ Go module comments referencing "chain"): leave Go package/module names and the /tmp/chain_e2e scratch dir path as-is (renaming the scratch dir is optional churn, not required for correctness — confirm no hardcoded absolute path outside /tmp breaks). Only fix any comment/string that says Test/E2E/chain (should say Test/E2E/suite) or run_chain_e2e.sh (should say run_e2e.sh).

  • Step 3: Update CLAUDE.md

In the ## Build & Test section, find the paragraph beginning **Streaming-chain E2E suite** (Test/E2E/chain/). Change Test/E2E/chain/ → Test/E2E/suite/ and ./run_chain_e2e.sh → ./run_e2e.sh everywhere in that paragraph (both occurrences of the script name in the usage line and prose).

  • Step 4: Update AGENTS.md
grep -n "Test/E2E/chain\|run_chain_e2e" AGENTS.md

Apply the same Test/E2E/chain → Test/E2E/suite, run_chain_e2e.sh → run_e2e.sh substitution to every match.

  • Step 5: Verify
source env.sh
cd Test/E2E/suite && ./run_e2e.sh --skip-build --only s01_scalar_uint32 2>&1 | tail -20

Expected: s01_scalar_uint32: PASS and results.json: 1 pass, 0 fail, 0 skip, 0 xfail, 0 xpass → PASS, exactly as run_chain_e2e.sh produced before the rename.

  • Step 6: Commit
cd /home/martino/Projects/MARTe_Integrated_components
git add -A Test/E2E/suite Test/E2E/chain CLAUDE.md AGENTS.md
git commit -m "refactor(e2e): rename Test/E2E/chain -> Test/E2E/suite, run_chain_e2e.sh -> run_e2e.sh"

Task 3: Add kind field to the scenario model + direct/recorder scenario definitions

Files:

  • Modify: Test/E2E/suite/scenarios.py
  • Modify: Test/E2E/suite/tests_py.py
  • Test: Test/E2E/suite/tests_py.py (extended)

Interfaces:

  • Consumes: Test/E2E/datasources/E2ETest.cfg, Test/E2E/datasources/E2EMulticastTest.cfg, Test/E2E/datasources/gen_test_data.py::generate() -> None (writes module-level OUTPUT path), Test/E2E/datasources/validate_binary.py::validate(input_path, output_path, label="test") -> (bool, str, dict), Test/E2E/recorder/RecorderStreamer.cfg, Test/E2E/recorder/StreamHubRec.cfg.
  • Produces: every entry in scenarios.SCENARIOS now has a "kind" key ("chain" for all 51 existing entries, defaulted via _sig/scenario-builder helpers so no existing entry needs editing by hand); two new module-level lists scenarios._DIRECT (2 entries) and scenarios._RECORDER (1 entry) are folded into SCENARIOS; scenarios.validate_scenario(s) accepts the new kind key and the direct/recorder-specific keys below without raising.

New scenario dict shape for kind="direct" (no sources/network/oracle — those are chain-only concepts):

{
    "id": "s52_direct_unicast",
    "desc": "Direct UDPStreamer->UDPStreamerClient round-trip, unicast",
    "kind": "direct",
    "cfg": "Test/E2E/datasources/E2ETest.cfg",
    "client_checks": [],
    "known_issue": None,
}

and a second entry s53_direct_multicast with "cfg": "Test/E2E/datasources/E2EMulticastTest.cfg".

New scenario dict shape for kind="recorder":

{
    "id": "s54_recorder",
    "desc": "StreamHub BinaryRecorder disk-output round-trip",
    "kind": "recorder",
    "hub_cfg": "Test/E2E/recorder/StreamHubRec.cfg",
    "marte_cfg": "Test/E2E/recorder/RecorderStreamer.cfg",
    "client_checks": [],
    "known_issue": None,
}
  • Step 1: Add kind default to existing chain scenarios

In scenarios.py, find the _sig(...) helper and the point where _STARTERS/_TYPES/etc. lists are concatenated into SCENARIOS (currently SCENARIOS = _STARTERS + _TYPES + _ARRAYS + _TIMEMODES + _QUANT + _PUBLISH + _PAYLOAD + _MCAST + _MULTISRC + _INTERACT + _TRIGGER + _MISC + _HIGHRATE, line 564). Add a normalization pass immediately after that line:

SCENARIOS = _STARTERS + _TYPES + _ARRAYS + _TIMEMODES + _QUANT + _PUBLISH + _PAYLOAD + _MCAST + _MULTISRC + _INTERACT + _TRIGGER + _MISC + _HIGHRATE

for _s in SCENARIOS:
    _s.setdefault("kind", "chain")
  • Step 2: Add _DIRECT and _RECORDER scenario lists

Immediately after the normalization loop from Step 1, add:

_DIRECT = [
    {
        "id": "s52_direct_unicast",
        "desc": "Direct UDPStreamer->UDPStreamerClient round-trip, unicast",
        "kind": "direct",
        "cfg": "Test/E2E/datasources/E2ETest.cfg",
        "client_checks": [],
        "known_issue": None,
    },
    {
        "id": "s53_direct_multicast",
        "desc": "Direct UDPStreamer->UDPStreamerClient round-trip, multicast",
        "kind": "direct",
        "cfg": "Test/E2E/datasources/E2EMulticastTest.cfg",
        "client_checks": [],
        "known_issue": None,
    },
]

_RECORDER = [
    {
        "id": "s54_recorder",
        "desc": "StreamHub BinaryRecorder disk-output round-trip",
        "kind": "recorder",
        "hub_cfg": "Test/E2E/recorder/StreamHubRec.cfg",
        "marte_cfg": "Test/E2E/recorder/RecorderStreamer.cfg",
        "client_checks": [],
        "known_issue": None,
    },
]

SCENARIOS = SCENARIOS + _DIRECT + _RECORDER
  • Step 3: Update validate_scenario(s) to accept the new kinds

Read the current validate_scenario(s) body (validates chain-specific keys like sources/network/oracle). Wrap the chain-specific assertions in a guard so direct/recorder kinds skip them:

def validate_scenario(s):
    assert "id" in s and "kind" in s, f"scenario missing id/kind: {s}"
    if s["kind"] == "chain":
        # ... existing body unchanged, indented under this guard ...
        pass
    elif s["kind"] == "direct":
        assert "cfg" in s, f"direct scenario {s['id']} missing cfg"
    elif s["kind"] == "recorder":
        assert "hub_cfg" in s and "marte_cfg" in s, f"recorder scenario {s['id']} missing hub_cfg/marte_cfg"
    else:
        raise AssertionError(f"unknown scenario kind: {s['kind']}")

(Preserve the exact existing chain-validation statements under the if s["kind"] == "chain": branch — do not rewrite their logic, only re-indent.)

  • Step 4: Write the failing test

Add to Test/E2E/suite/tests_py.py (find the existing class ScenariosTest-style test class and add a method following its pattern):

def test_all_scenarios_have_kind(self):
    import scenarios as S
    for s in S.SCENARIOS:
        self.assertIn("kind", s, f"{s['id']} missing kind")
        self.assertIn(s["kind"], ("chain", "direct", "recorder", "debug", "tcplogger"))

def test_direct_and_recorder_scenarios_present(self):
    import scenarios as S
    kinds = {s["kind"] for s in S.SCENARIOS}
    self.assertIn("direct", kinds)
    self.assertIn("recorder", kinds)
    direct_ids = {s["id"] for s in S.SCENARIOS if s["kind"] == "direct"}
    self.assertEqual(direct_ids, {"s52_direct_unicast", "s53_direct_multicast"})
    recorder_ids = {s["id"] for s in S.SCENARIOS if s["kind"] == "recorder"}
    self.assertEqual(recorder_ids, {"s54_recorder"})

def test_validate_scenario_accepts_all_kinds(self):
    import scenarios as S
    for s in S.SCENARIOS:
        S.validate_scenario(s)  # must not raise
  • Step 5: Run tests to verify they pass (this is additive, not TDD-red-first, since it extends existing data structures rather than new behavior with a failing precondition)
cd Test/E2E/suite && python3 -m unittest tests_py -v 2>&1 | tail -20

Expected: all tests pass, including the 3 new ones.

  • Step 6: Commit
git add Test/E2E/suite/scenarios.py Test/E2E/suite/tests_py.py
git commit -m "feat(e2e): add kind discriminator + direct/recorder scenario definitions"

Task 4: Wire direct/recorder dispatch into run_e2e.sh's scenario loop

Files:

  • Modify: Test/E2E/suite/run_e2e.sh
  • Modify: Test/E2E/suite/collect.py is NOT touched in this task (unit-test collection is unaffected by new scenario kinds)

Interfaces:

  • Consumes: scenarios.SCENARIOS entries now carrying kind (Task 3); Test/E2E/datasources/gen_test_data.py::generate(), Test/E2E/datasources/validate_binary.py::validate(input_path, output_path, label) -> (bool, str, dict); Test/E2E/recorder/RecorderStreamer.cfg/StreamHubRec.cfg launch sequence (ported from Test/E2E/recorder/run_recorder_e2e.sh, to be deleted in Task 9).
  • Produces: results.json entries for s52_direct_unicast, s53_direct_multicast, s54_recorder alongside the 51 chain entries, each with a "kind" field.

The current scenario-list-building Python block (embedded in run_e2e.sh, feeding the while IFS='|' read ... loop) only emits the 7-field chain pipe format (id|ws_port|udp_port0|network|oracle|trig|checks). Extend it to also emit direct/recorder entries in an extended format, and add a KIND field as the first column so the loop can dispatch.

  • Step 1: Extend the inline Python scenario-list builder

Find this block in run_e2e.sh (the LIST="$(${PY} - "${ONLY}" <<'PY' ... PY )" heredoc). Replace its body:

import sys, os
sys.path.insert(0, os.path.dirname(os.path.abspath("Test/E2E/suite/scenarios.py")))
sys.path.insert(0, os.path.join(os.getcwd(), "Test/E2E/suite"))
import scenarios as S
only = sys.argv[1] if len(sys.argv) > 1 else ""
for s in S.SCENARIOS:
    if only and s["id"] != only:
        continue
    kind = s["kind"]
    if kind == "chain":
        trig = s.get("trig_signal") or ""
        checks = ",".join(s.get("client_checks", []))
        if not trig:
            checks = ",".join(c for c in s.get("client_checks", []) if c != "trigger")
        print("|".join([kind, s["id"], str(s["ws_port"]), str(s["sources"][0]["udp_port"]),
                        s["network"], s["oracle"], trig, checks]))
    elif kind == "direct":
        print("|".join([kind, s["id"], s["cfg"], "", "", "", "", ""]))
    elif kind == "recorder":
        print("|".join([kind, s["id"], s["marte_cfg"], s["hub_cfg"], "", "", "", ""]))

(Note: Test/E2E/chain path in the two sys.path.insert lines becomes Test/E2E/suite — already done by Task 2's rename; confirm those two lines read Test/E2E/suite/scenarios.py before proceeding.)

  • Step 2: Change the loop's read line and add dispatch

Find while IFS='|' read -r ID WSPORT UDPPORT NET ORACLE TRIG CHECKS; do and change to:

while IFS='|' read -r KIND ID F2 F3 F4 F5 F6 F7; do
    [ -z "${ID}" ] && continue
    case "${KIND}" in
        chain)
            WSPORT="${F2}"; UDPPORT="${F3}"; NET="${F4}"; ORACLE="${F5}"; TRIG="${F6}"; CHECKS="${F7}"
            # ... existing chain scenario body unchanged, using $WSPORT/$UDPPORT/$NET/$ORACLE/$TRIG/$CHECKS ...
            ;;
        direct)
            CFG="${F2}"
            echo "══ scenario ${ID}  (kind=direct cfg=${CFG}) ══"
            INPUT_BIN="/tmp/udpstreamer_test_input.bin"
            OUTPUT_BIN="/tmp/udpstreamer_test_output_${ID}.bin"
            rm -f "${INPUT_BIN}" "${OUTPUT_BIN}"
            (cd "${SCRIPT_DIR}/../datasources" && ${PY} -c "import gen_test_data; gen_test_data.generate()")
            timeout 15 "${MARTE_APP}" -l RealTimeLoader -f "${REPO_ROOT}/${CFG}" -s Running \
                > "${OUT_DIR}/marte_${ID}.log" 2>&1
            ${PY} -c "
import sys; sys.path.insert(0, '${SCRIPT_DIR}/../datasources')
import validate_binary as V
ok, msg, details = V.validate('${INPUT_BIN}', '${OUTPUT_BIN}', label='${ID}')
print(f'{\"${ID}\"}: {\"PASS\" if ok else \"FAIL\"} - {msg}')
sys.exit(0 if ok else 1)
" && STATUS=PASS || STATUS=FAIL
            echo "${STATUS}" > "${OUT_DIR}/status_${ID}.txt"
            ;;
        recorder)
            MCFG="${F2}"; HCFG="${F3}"
            echo "══ scenario ${ID}  (kind=recorder marte_cfg=${MCFG} hub_cfg=${HCFG}) ══"
            rm -rf /tmp/streamhub_rec_e2e && mkdir -p /tmp/streamhub_rec_e2e
            "${STREAMHUB_EX}" -cfg "${REPO_ROOT}/${HCFG}" > "${OUT_DIR}/hub_${ID}.log" 2>&1 &
            HUB_PID=$!
            sleep 1
            (cd "${SCRIPT_DIR}/../datasources" && ${PY} -c "import gen_test_data; gen_test_data.generate()")
            timeout 8 "${MARTE_APP}" -l RealTimeLoader -f "${REPO_ROOT}/${MCFG}" -s Running \
                > "${OUT_DIR}/marte_${ID}.log" 2>&1
            sleep 2
            kill "${HUB_PID}" 2>/dev/null; wait "${HUB_PID}" 2>/dev/null
            REC_FILE="$(ls -t /tmp/streamhub_rec_e2e/*.bin 2>/dev/null | head -1)"
            if [ -z "${REC_FILE}" ]; then
                echo "${ID}: FAIL - no recorder output file found"
                echo "FAIL" > "${OUT_DIR}/status_${ID}.txt"
            else
                ${PY} -c "
import sys; sys.path.insert(0, '${SCRIPT_DIR}/../datasources')
import validate_binary as V
ok, msg, details = V.validate('/tmp/udpstreamer_test_input.bin', '${REC_FILE}', label='${ID}')
print(f'{\"${ID}\"}: {\"PASS\" if ok else \"FAIL\"} - {msg}')
sys.exit(0 if ok else 1)
" && STATUS=PASS || STATUS=FAIL
                echo "${STATUS}" > "${OUT_DIR}/status_${ID}.txt"
            fi
            ;;
    esac
done <<< "${LIST}"

Keep the existing chain-scenario body (launch StreamHub, launch MARTeApp, run chain-client, validate_waveform.py, plots.py) exactly as it is today, just moved inside the chain) case arm with variable names unchanged ($WSPORT/$UDPPORT/etc. already match what the existing body uses).

  • Step 3: Extend results.json aggregation to record kind

Find wherever run_e2e.sh builds the final results.json (reads status_<id>.txt files per scenario, likely in a trailing Python block). Add "kind" to each scenario's record by re-deriving it from scenarios.SCENARIOS keyed by id (import scenarios, build {s["id"]: s["kind"] for s in scenarios.SCENARIOS}, look up per status file).

  • Step 4: Verify
source env.sh
cd Test/E2E/suite && ./run_e2e.sh --skip-build --only s52_direct_unicast 2>&1 | tail -20
./run_e2e.sh --skip-build --only s53_direct_multicast 2>&1 | tail -20
./run_e2e.sh --skip-build --only s54_recorder 2>&1 | tail -20

Expected: each prints <id>: PASS, and results.json shows 1 pass, 0 fail.

./run_e2e.sh --skip-build --only s01_scalar_uint32 2>&1 | tail -10

Expected: unaffected, still PASS — confirms the chain) case arm refactor didn't break existing behavior.

  • Step 5: Commit
git add Test/E2E/suite/run_e2e.sh
git commit -m "feat(e2e): dispatch direct/recorder scenario kinds in run_e2e.sh"

Task 5: Refactor MarteController for headless reuse + build debugclient

Files:

  • Modify: Client/debugger/martecontrol.go
  • Create: Test/E2E/suite/debugclient/main.go
  • Create: Test/E2E/suite/debugclient/go.mod

Interfaces:

  • Consumes: Client/debugger's exported (post-refactor) NewHeadlessMarteController(sink func(v any)) *MarteController, and its existing (*MarteController).Connect(host string, cmdPort, udpPort, logPort int), (*MarteController).SendCommand(cmd string), (*MarteController).Disconnect(), (*MarteController).IsConnected() bool (all already defined in martecontrol.go, unchanged by this task).
  • Produces: debugclient binary accepting -host -cmdport -udpport -logport -scenario <id> -out <dir> -mode <debug|tcplogger> -dur <duration>; exits 0 on script success, non-zero + JSON detail file (result_<id>.json) on failure.

MarteController currently has a single constructor NewMarteController(hub *wshub.Hub) *MarteController and routes every event through the module-level helper broadcastHub(hub *wshub.Hub, v any), called at 16 sites across Connect/runTCP/runDebugUDP/runLog/readLoop. Introduce a sink func(v any) field so the same struct/methods work with or without a WebSocket hub.

  • Step 1: Add a sink field and headless constructor

In Client/debugger/martecontrol.go, add to the MarteController struct definition (alongside the existing hub *wshub.Hub field):

	sink func(v any)

Change broadcastHub's call sites: broadcastHub itself stays (used by NewMarteController's default sink), but every one of the 16 broadcastHub(m.hub, X) call sites inside MarteController methods becomes m.sink(X). This is a mechanical find-and-replace — verify with:

grep -n "broadcastHub(m.hub" Client/debugger/martecontrol.go

Expected before: 16 matches. Replace each with m.sink(...) (same arguments minus m.hub,). After replacement:

grep -c "broadcastHub(m.hub" Client/debugger/martecontrol.go   # expect 0
grep -c "m.sink(" Client/debugger/martecontrol.go               # expect 16
  • Step 2: Update both constructors

The current NewMarteController (martecontrol.go:130-143) reads:

func NewMarteController(hub *wshub.Hub) *MarteController {
	mc := &MarteController{
		hub:         hub,
		signals:     make(map[uint32]*SignalMeta),
		tracedNames: make(map[string]bool),
		forcedState: make(map[string]string),
		stopCh:      make(chan struct{}),
	}
	hub.SetOnClientConnect(mc.replayStateToClient)
	return mc
}

Change it to set sink and keep the hub-specific SetOnClientConnect registration (only headless instances skip that):

func NewMarteController(hub *wshub.Hub) *MarteController {
	mc := &MarteController{
		hub:         hub,
		signals:     make(map[uint32]*SignalMeta),
		tracedNames: make(map[string]bool),
		forcedState: make(map[string]string),
		stopCh:      make(chan struct{}),
	}
	mc.sink = func(v any) { broadcastHub(mc.hub, v) }
	hub.SetOnClientConnect(mc.replayStateToClient)
	return mc
}

Add a new headless constructor immediately after it, reusing the same field initializers but with hub: nil and no SetOnClientConnect call (there is no hub to register a callback on):

// NewHeadlessMarteController creates a MarteController with no WebSocket hub,
// routing all events through sink instead (used by the debugclient E2E test tool).
func NewHeadlessMarteController(sink func(v any)) *MarteController {
	mc := &MarteController{
		hub:         nil,
		signals:     make(map[uint32]*SignalMeta),
		tracedNames: make(map[string]bool),
		forcedState: make(map[string]string),
		stopCh:      make(chan struct{}),
	}
	mc.sink = sink
	return mc
}
  • Step 3: Build to verify the refactor didn't break the browser UI path
cd Client/debugger && go build ./... && go vet ./...

Expected: clean build, no errors (confirms NewMarteController's existing callers in main.go are unaffected by the added field).

go test ./... 2>&1 | tail -20

Expected: existing cmd_gate_test.go tests still pass.

  • Step 4: Commit the martecontrol.go refactor
git add Client/debugger/martecontrol.go
git commit -m "refactor(debugger): extract MarteController sink so it can run headless"
  • Step 5: Create the debugclient Go module
mkdir -p Test/E2E/suite/debugclient
cd Test/E2E/suite/debugclient
go mod init debugclient

Edit the generated go.mod to add a replace directive pointing at the sibling module and require it:

module debugclient

go 1.21

require debugger v0.0.0

replace debugger => ../../../Client/debugger

(Confirm the actual module name declared in Client/debugger/go.mod's module line and use that exact name in place of debugger above — read the file first: head -1 Client/debugger/go.mod.)

  • Step 6: Write debugclient/main.go
// Command debugclient drives a running MARTeApp.ex DebugService+TCPLogger
// instance over TCP/UDP for the Test/E2E/suite "debug" and "tcplogger"
// scenario kinds. It scripts a fixed command sequence, captures responses
// and log/trace output, and reports PASS/FAIL as JSON to -out.
package main

import (
	"encoding/json"
	"flag"
	"fmt"
	"os"
	"sync"
	"time"

	debugger "debugger" // see go.mod replace directive; import path must match Client/debugger/go.mod's module line
)

type event struct {
	kind string
	data any
}

func main() {
	host := flag.String("host", "127.0.0.1", "MARTeApp host")
	cmdPort := flag.Int("cmdport", 8080, "DebugService TCP command port")
	udpPort := flag.Int("udpport", 8081, "DebugService UDP trace port")
	logPort := flag.Int("logport", 9090, "TCPLogger TCP port")
	scenario := flag.String("scenario", "", "scenario id (for output naming)")
	outDir := flag.String("out", "/tmp/debug_e2e", "output directory")
	mode := flag.String("mode", "debug", "debug|tcplogger")
	dur := flag.Duration("dur", 4*time.Second, "how long to run the script/collection")
	flag.Parse()

	if *scenario == "" {
		fmt.Fprintln(os.Stderr, "missing -scenario")
		os.Exit(2)
	}
	os.MkdirAll(*outDir, 0o755)

	var mu sync.Mutex
	var events []event
	sink := func(v any) {
		mu.Lock()
		defer mu.Unlock()
		events = append(events, event{kind: fmt.Sprintf("%T", v), data: v})
	}

	mc := debugger.NewHeadlessMarteController(sink)
	mc.Connect(*host, *cmdPort, *udpPort, *logPort)
	defer mc.Disconnect()

	time.Sleep(500 * time.Millisecond) // allow TCP/UDP/log connects to establish

	var ok bool
	var msg string
	switch *mode {
	case "debug":
		ok, msg = runDebugScript(mc)
	case "tcplogger":
		ok, msg = runTcpLoggerScript(mc, *dur)
	default:
		fmt.Fprintf(os.Stderr, "unknown -mode %q\n", *mode)
		os.Exit(2)
	}

	time.Sleep(300 * time.Millisecond) // drain final events

	result := map[string]any{
		"id":     *scenario,
		"pass":   ok,
		"detail": msg,
	}
	f, _ := os.Create(fmt.Sprintf("%s/result_%s.json", *outDir, *scenario))
	json.NewEncoder(f).Encode(result)
	f.Close()

	status := "FAIL"
	if ok {
		status = "PASS"
	}
	fmt.Printf("%s: %s - %s\n", *scenario, status, msg)
	os.WriteFile(fmt.Sprintf("%s/status_%s.txt", *outDir, *scenario), []byte(status), 0o644)

	if !ok {
		os.Exit(1)
	}
}

// runDebugScript exercises FORCE/TRACE/BREAK over the TCP 8080 command
// protocol (grammar confirmed against Source/Components/Interfaces/
// DebugService/DebugServiceBase.cpp's HandleCommand: "FORCE <name> <val>",
// "TRACE <name> <0|1> [decim]", "BREAK <name> <op|OFF> <threshold>",
// each replying "OK <TOKEN> <count>\n").
func runDebugScript(mc *debugger.MarteController) (bool, string) {
	mc.SendCommand("FORCE App.Data.Timer.Counter 42")
	time.Sleep(200 * time.Millisecond)
	mc.SendCommand("TRACE App.Data.Timer.Counter 1 1")
	time.Sleep(200 * time.Millisecond)
	mc.SendCommand("BREAK App.Data.Timer.Counter > 1000")
	time.Sleep(500 * time.Millisecond)
	mc.SendCommand("BREAK App.Data.Timer.Counter OFF")
	mc.SendCommand("UNFORCE App.Data.Timer.Counter")
	if !mc.IsConnected() {
		return false, "lost connection to DebugService during script"
	}
	return true, "FORCE/TRACE/BREAK sequence completed without disconnect"
}

// runTcpLoggerScript triggers a log-worthy event (an invalid FORCE) and
// waits for a log line to arrive over the TCPLogger TCP port within dur.
func runTcpLoggerScript(mc *debugger.MarteController, dur time.Duration) (bool, string) {
	mc.SendCommand("FORCE App.Data.DoesNotExist 1")
	deadline := time.Now().Add(dur)
	for time.Now().Before(deadline) {
		time.Sleep(100 * time.Millisecond)
	}
	if !mc.IsConnected() {
		return false, "lost connection during tcplogger wait"
	}
	return true, "connected to TCPLogger for full duration without disconnect"
}

Note for implementer: runTcpLoggerScript's pass condition here only checks connection liveness because log-line content isn't exposed by MarteController's public API in the reviewed code — the sink callback (events slice in main) already captures every {"type":"log",...} event runLog emits. Before finalizing this task, inspect the exact JSON shape runLog sends (read Client/debugger/martecontrol.go's runLog body around its broadcastHub/m.sink call, now at the location from Step 1) and tighten runTcpLoggerScript to assert on events containing a "type":"log" entry whose message references the triggered event (e.g. the unknown-signal FORCE failure), rather than only checking liveness. Do this as part of Step 7 below, not deferred further.

  • Step 7: Tighten the TCPLogger assertion using the real event shape

Read runLog's exact m.sink(...) payload shape (post Step 1 refactor) and update runTcpLoggerScript to inspect events (passed in via a new parameter) for a matching log entry:

func runTcpLoggerScript(mc *debugger.MarteController, dur time.Duration, events *[]event, mu *sync.Mutex) (bool, string) {
	mc.SendCommand("FORCE App.Data.DoesNotExist 1")
	deadline := time.Now().Add(dur)
	for time.Now().Before(deadline) {
		mu.Lock()
		for _, e := range *events {
			if m, ok := e.data.(map[string]any); ok && m["type"] == "log" {
				mu.Unlock()
				return true, fmt.Sprintf("received log event: %v", m)
			}
		}
		mu.Unlock()
		time.Sleep(100 * time.Millisecond)
	}
	return false, "no log event received within duration"
}

Adjust the sink closure in main() to store map[string]any (unmarshal-then-remarshal, or type-assert directly if runLog already passes a map[string]any/struct — match whatever type runLog actually sends) and update the runDebugScript/runTcpLoggerScript call sites in main() to pass &events/&mu.

  • Step 8: Build and smoke-test standalone
cd Test/E2E/suite/debugclient && go build -o debugclient . && go vet ./...

Expected: clean build. (Full end-to-end exercise against a running MARTeApp.ex happens in Task 6's verification step, once the debug/tcplogger cfgs exist.)

  • Step 9: Commit
git add Test/E2E/suite/debugclient
git commit -m "feat(e2e): add debugclient Go tool for DebugService/TCPLogger E2E scenarios"

Task 6: Add debug/tcplogger scenario kinds + trimmed cfgs + run_e2e.sh dispatch

Files:

  • Create: Test/E2E/suite/debug_e2e.cfg
  • Modify: Test/E2E/suite/scenarios.py
  • Modify: Test/E2E/suite/tests_py.py
  • Modify: Test/E2E/suite/run_e2e.sh

Interfaces:

  • Consumes: debugclient binary from Task 5 (-host -cmdport -udpport -logport -scenario -out -mode).

  • Produces: results.json entries for s55_debug_force_trace_break (kind="debug") and s56_tcplogger_delivery (kind="tcplogger").

  • Step 1: Create the trimmed debug/tcplogger cfg

Trimmed from Test/Configurations/combined_test.cfg's Thread1 (drop the Sine GAMs and Streamer1/StreamerGAM1 entirely — DebugService/TCPLogger don't depend on the streaming path) plus its +DebugService block verbatim. Both s55/s56 scenarios reuse this same cfg (only debugclient -mode differs). Write the full file:

Test/E2E/suite/debug_e2e.cfg

/**
 * Trimmed single-thread configuration for the "debug"/"tcplogger" E2E
 * scenario kinds: exercises DebugService FORCE/TRACE/BREAK over TCP 8080 /
 * UDP 8081 and TCPLogger delivery over TCP 9090, with no UDPStreamer path.
 */

$App = {
    Class = RealTimeApplication

    +Functions = {
        Class = ReferenceContainer

        +TimerGAM = {
            Class = IOGAM
            InputSignals = {
                Counter = {
                    DataSource = Timer
                    Type = uint32
                    Frequency = 1000
                }
                Time = {
                    DataSource = Timer
                    Type = uint32
                }
            }
            OutputSignals = {
                Counter = { DataSource = DDB1  Type = uint32 }
                Time    = { DataSource = DDB1  Type = uint32 }
            }
        }
    }

    +Data = {
        Class = ReferenceContainer
        DefaultDataSource = DDB1

        +DDB1 = { Class = GAMDataSource }

        +Timer = {
            Class = LinuxTimer
            SleepNature = "Default"
            Signals = {
                Counter = { Type = uint32 }
                Time    = { Type = uint32 }
            }
        }

        +Timings = { Class = TimingDataSource }
    }

    +States = {
        Class = ReferenceContainer
        +Running = {
            Class = RealTimeState
            +Threads = {
                Class = ReferenceContainer

                +Thread1 = {
                    Class = RealTimeThread
                    CPUs = 0x1
                    Functions = {TimerGAM}
                }
            }
        }
    }

    +Scheduler = {
        Class = GAMScheduler
        TimingDataSource = Timings
    }
}

// ── DebugService ──────────────────────────────────────────────────────────────
// Patches the broker registry at startup so every signal is traceable.
// TcpLogger is auto-injected on LogPort (9090) — no explicit DataSource needed.
+DebugService = {
    Class = DebugService
    ControlPort = 8080
    StreamPort  = 8081
    LogPort     = 9090
    StreamIP    = "127.0.0.1"
}

The App.Data.Timer.Counter signal path used by debugclient's FORCE/TRACE/BREAK commands matches this file's +Data.+Timer.Counter signal, mirroring combined_test.cfg's naming.

  • Step 2: Add scenario definitions

In scenarios.py, after the _RECORDER list from Task 3, add:

_DEBUG = [
    {
        "id": "s55_debug_force_trace_break",
        "desc": "DebugService FORCE/TRACE/BREAK over real TCP 8080 + UDP 8081",
        "kind": "debug",
        "cfg": "Test/E2E/suite/debug_e2e.cfg",
        "cmd_port": 8080, "udp_port": 8081, "log_port": 9090,
        "client_checks": [],
        "known_issue": None,
    },
]

_TCPLOGGER = [
    {
        "id": "s56_tcplogger_delivery",
        "desc": "TCPLogger delivers a log line for a triggered DebugService event",
        "kind": "tcplogger",
        "cfg": "Test/E2E/suite/debug_e2e.cfg",
        "cmd_port": 8080, "udp_port": 8081, "log_port": 9090,
        "client_checks": [],
        "known_issue": None,
    },
]

SCENARIOS = SCENARIOS + _DEBUG + _TCPLOGGER

Extend validate_scenario(s) (Task 3, Step 3) with two more branches:

    elif s["kind"] in ("debug", "tcplogger"):
        assert "cfg" in s and "cmd_port" in s and "udp_port" in s and "log_port" in s, \
            f"{s['kind']} scenario {s['id']} missing cfg/cmd_port/udp_port/log_port"
  • Step 3: Extend tests_py.py
def test_debug_and_tcplogger_scenarios_present(self):
    import scenarios as S
    kinds = {s["kind"] for s in S.SCENARIOS}
    self.assertIn("debug", kinds)
    self.assertIn("tcplogger", kinds)

Also update test_all_scenarios_have_kind's allowed-kinds tuple if it was written differently in Task 3 (it already includes "debug"/"tcplogger" per Task 3 Step 4 — no change needed if so, just confirm).

  • Step 4: Extend run_e2e.sh's Python scenario-list builder

In the heredoc from Task 4 Step 1, add two more elif branches:

    elif kind == "debug":
        print("|".join([kind, s["id"], s["cfg"], str(s["cmd_port"]), str(s["udp_port"]), str(s["log_port"]), "", ""]))
    elif kind == "tcplogger":
        print("|".join([kind, s["id"], s["cfg"], str(s["cmd_port"]), str(s["udp_port"]), str(s["log_port"]), "", ""]))
  • Step 5: Extend the case "${KIND}" dispatch in run_e2e.sh
        debug|tcplogger)
            CFG="${F2}"; CMDPORT="${F3}"; UDPPORT2="${F4}"; LOGPORT="${F5}"
            echo "══ scenario ${ID}  (kind=${KIND} cfg=${CFG}) ══"
            timeout 15 "${MARTE_APP}" -l RealTimeLoader -f "${REPO_ROOT}/${CFG}" -s Running \
                > "${OUT_DIR}/marte_${ID}.log" 2>&1 &
            APP_PID=$!
            sleep 1
            "${SCRIPT_DIR}/debugclient/debugclient" -host 127.0.0.1 \
                -cmdport "${CMDPORT}" -udpport "${UDPPORT2}" -logport "${LOGPORT}" \
                -mode "${KIND}" -scenario "${ID}" -out "${OUT_DIR}" -dur 4s \
                > "${OUT_DIR}/client_${ID}.log" 2>&1
            CLIENT_RC=$?
            kill "${APP_PID}" 2>/dev/null; wait "${APP_PID}" 2>/dev/null
            if [ "${CLIENT_RC}" -eq 0 ]; then
                echo "PASS" > "${OUT_DIR}/status_${ID}.txt"
            else
                echo "FAIL" > "${OUT_DIR}/status_${ID}.txt"
            fi
            ;;

(debugclient already writes its own status_${ID}.txt/result_${ID}.json per Task 5 Step 6 — this arm's own status_${ID}.txt write is redundant with that but harmless/idempotent; keep both for robustness in case debugclient crashes before writing its file, in which case this arm's fallback based on $CLIENT_RC is authoritative.)

Also build debugclient alongside chain-client in the "build Go tools" section near the top of run_e2e.sh (find where (cd "${SCRIPT_DIR}/client" && go build -o chain-client .) runs and add an equivalent block for ${SCRIPT_DIR}/debugclient).

  • Step 6: Verify end-to-end
source env.sh
cd Test/E2E/suite && ./run_e2e.sh --skip-build --only s55_debug_force_trace_break 2>&1 | tail -20
./run_e2e.sh --skip-build --only s56_tcplogger_delivery 2>&1 | tail -20

Expected: both PASS. If s56 fails, inspect client_s56_tcplogger_delivery.log and Build/x86-linux/E2E/suite/marte_s56_tcplogger_delivery.log — the most likely failure mode is the trimmed cfg not producing the expected log line format from runLog's parser; adjust debug_e2e.cfg's triggered event or runTcpLoggerScript's match condition accordingly, re-run until green (do not mark this task complete with a failing scenario).

  • Step 7: Commit
git add Test/E2E/suite/debug_e2e.cfg Test/E2E/suite/scenarios.py Test/E2E/suite/tests_py.py Test/E2E/suite/run_e2e.sh
git commit -m "feat(e2e): add debug/tcplogger E2E scenario kinds using debugclient"

Task 7: Restructure build/coverage flow (instrument-first, double-run, new skip flags) + port stress multi-fragment fix

Files:

  • Modify: Test/E2E/suite/run_e2e.sh
  • Modify: Test/E2E/suite/stress.py
  • Modify: Test/E2E/suite/collect.py (only the main() CLI surface — add no new suites, just confirm invocation still matches new call sites)

Interfaces:

  • Consumes: Test/E2E/suite/stress_run.py::main() CLI (--marte --hub --client --work --out --only --axis, unchanged).

  • Produces: run_e2e.sh flags --skip-coverage, --skip-stress, --skip-datasources, --skip-recorder, --skip-debug, --skip-tcplogger (all suites run by default); stress.py's STRESS_CASES includes multi-fragment size cases; a stress_results.json is always produced (unless --skip-stress) at ${OUT_DIR}/stress/stress_results.json.

  • Step 1: Port stress.py's multi-fragment extension from the stale branch

git show feature/stress-report-integration:Test/E2E/chain/stress.py > /tmp/stress_feature_branch.py
diff /tmp/stress_feature_branch.py Test/E2E/suite/stress.py

Apply these specific changes to Test/E2E/suite/stress.py (per the pre-gathered diff summary — re-derive exact line numbers from the diff output above since paths/line numbers shifted after Task 2's rename):

  1. Add module-level constant UDPS_CLIENT_MAX_PACKET_BYTES = 1048576 (matching Source/Components/Interfaces/UDPStream/UDPSClient.h's UDPS_CLIENT_MAX_PACKET_BYTES).
  2. _source() gains a row_dt=S.ROW_DT parameter; change its rate formula from the hardcoded elements * 1000.0 to elements / row_dt.
  3. mk_stress() gains row_dt=None, num_rows=None, producer_hz=None parameters, wired into the returned case dict (previously always None).
  4. Add _BIG_SIZE cases (50000/100000/250000 elements, slowed producer_hz) appended to both _DS_SIZE and _HUB_SIZE.
  5. validate_case()'s packet-size ceiling check changes from >= 65536 to >= UDPS_CLIENT_MAX_PACKET_BYTES.
  • Step 2: Verify stress.py still validates
cd Test/E2E/suite && python3 -c "
import stress as ST
for c in ST.STRESS_CASES:
    ST.validate_case(c)
print(f'{len(ST.STRESS_CASES)} stress cases validated OK')
"

Expected: no assertion errors, case count is 27 (original) + 6 (2 axes x 3 _BIG_SIZE levels) = 33.

  • Step 3: Add new flags to run_e2e.sh's arg parser

Extend the case "$1" in block (currently lines 30-36) to:

        --skip-build) SKIP_BUILD=1 ;;
        --only) shift; ONLY="$1" ;;
        --pdf-only) PDF_ONLY=1 ;;
        --cpp-coverage) CPP_COV=1 ;;
        --skip-coverage) CPP_COV=0 ;;
        --skip-stress) SKIP_STRESS=1 ;;
        --skip-datasources) SKIP_DATASOURCES=1 ;;
        --skip-recorder) SKIP_RECORDER=1 ;;
        --skip-debug) SKIP_DEBUG=1 ;;
        --skip-tcplogger) SKIP_TCPLOGGER=1 ;;
        --help|-h) echo "Usage: $0 [--skip-build] [--only <id>] [--pdf-only] [--skip-coverage] [--skip-stress] [--skip-datasources] [--skip-recorder] [--skip-debug] [--skip-tcplogger]"; exit 0 ;;
        *) echo "unknown arg $1" >&2; exit 2 ;;

Add the new variable defaults alongside the existing SKIP_BUILD=0 etc.:

SKIP_STRESS=0
SKIP_DATASOURCES=0
SKIP_RECORDER=0
SKIP_DEBUG=0
SKIP_TCPLOGGER=0

Wire SKIP_DATASOURCES/SKIP_RECORDER/SKIP_DEBUG/SKIP_TCPLOGGER into the Python scenario-list builder's filter (skip emitting those kinds when set) — add to the heredoc's env/argv:

import sys, os
sys.path.insert(0, os.path.join(os.getcwd(), "Test/E2E/suite"))
import scenarios as S
only = sys.argv[1] if len(sys.argv) > 1 else ""
skip_kinds = set(sys.argv[2].split(",")) if len(sys.argv) > 2 and sys.argv[2] else set()
for s in S.SCENARIOS:
    if only and s["id"] != only:
        continue
    if s["kind"] in skip_kinds:
        continue
    ...

and change the invocation to pass a comma-joined skip-kinds argument built from the shell flags:

SKIP_KINDS=""
[ "${SKIP_DATASOURCES}" -eq 1 ] && SKIP_KINDS="${SKIP_KINDS}direct,"
[ "${SKIP_RECORDER}" -eq 1 ] && SKIP_KINDS="${SKIP_KINDS}recorder,"
[ "${SKIP_DEBUG}" -eq 1 ] && SKIP_KINDS="${SKIP_KINDS}debug,"
[ "${SKIP_TCPLOGGER}" -eq 1 ] && SKIP_KINDS="${SKIP_KINDS}tcplogger,"
LIST="$(${PY} - "${ONLY}" "${SKIP_KINDS}" <<'PY'
...
PY
)"
  • Step 4: Restructure the top-level flow into named phases

Replace the existing single-pass "build → scenarios → coverage-rebuild → collect → restore" sequence with:

# Phase 1: normal build
if [ "${SKIP_BUILD}" -eq 0 ]; then
    make -C "${REPO_ROOT}" -f Makefile.gcc core apps TARGET="${TARGET}" 2>&1 | tail -1
fi

# Phase 2: authoritative scenario pass (normal binaries) - existing scenario loop from Task 4/6, unchanged
run_scenario_matrix() {
    local RESULTS_SUFFIX="$1"
    # ... existing `while IFS='|' read -r KIND ID ...; do ... done <<< "${LIST}"` loop body, unchanged ...
}
run_scenario_matrix "primary"

# Phase 3: stress (normal binaries, always, never instrumented)
if [ "${SKIP_STRESS}" -eq 0 ]; then
    ${PY} "${SCRIPT_DIR}/stress_run.py" --marte "${MARTE_APP}" --hub "${STREAMHUB_EX}" \
        --client "${CLIENT}" --work "/tmp/chain_stress" --out "${OUT_DIR}/stress" || true
fi

# Phase 4: instrumented rebuild + coverage-only scenario re-run + unit tests
if [ "${CPP_COV}" -eq 1 ]; then
    COV_O="--coverage"
    COV_L="-Wl,--no-as-needed -fPIC --coverage"
    make -C "${REPO_ROOT}" -f Makefile.gcc clean >/dev/null 2>&1 || true
    make -C "${REPO_ROOT}" -f Makefile.gcc core TARGET="${TARGET}" OPTIM="${COV_O}" LFLAGS="${COV_L}" 2>&1 | tail -1
    for d in Test/Components/DataSources/UDPStreamer Test/Components/DataSources/UDPStreamerClient Test/Applications/StreamHub Test/GTest Test/Integration; do
        make -C "${REPO_ROOT}/${d}" -f Makefile.gcc TARGET="${TARGET}" OPTIM="${COV_O}" LFLAGS="${COV_L}" 2>&1 | tail -1
    done
    OUT_DIR="${OUT_DIR}/coverage_pass" run_scenario_matrix "coverage" || true
    ${PY} "${SCRIPT_DIR}/collect.py" --repo "${REPO_ROOT}" --target "${TARGET}" \
        --out "${OUT_DIR}" --work "${WORK}" --cpp-coverage || true
    make -C "${REPO_ROOT}" -f Makefile.gcc clean >/dev/null 2>&1 || true
    make -C "${REPO_ROOT}" -f Makefile.gcc core apps TARGET="${TARGET}" 2>&1 | tail -1
    (cd "${SCRIPT_DIR}/client" && go build -o chain-client . >/dev/null 2>&1) || true
    (cd "${SCRIPT_DIR}/debugclient" && go build -o debugclient . >/dev/null 2>&1) || true
else
    ${PY} "${SCRIPT_DIR}/collect.py" --repo "${REPO_ROOT}" --target "${TARGET}" \
        --out "${OUT_DIR}" --work "${WORK}" || true
fi

# Phase 5: report
${PY} "${SCRIPT_DIR}/report_build.py" --repo "${REPO_ROOT}" --results "${OUT_DIR}/results.json" \
    --work "${WORK}" --out "${OUT_DIR}" --stress-results "${OUT_DIR}/stress/stress_results.json"
# ... existing Typst PDF compile block, unchanged ...

Wrapping the existing scenario loop body in a run_scenario_matrix() shell function requires moving the LIST="$(...)" build step (Step 3, already parameterized by SKIP_KINDS) to run once before Phase 2, and having run_scenario_matrix iterate over the already-computed ${LIST} variable (read-only reuse — no need to recompute LIST for the coverage pass since it's the same scenario set). The OUT_DIR="${OUT_DIR}/coverage_pass" run_scenario_matrix "coverage" line's env-var-prefix trick only affects the subshell's view of OUT_DIR if run_scenario_matrix is a function (not a subshell) reading a global — since bash functions share the parent shell's variables, use a local override inside the call instead:

(
    OUT_DIR="${OUT_DIR}/coverage_pass"
    mkdir -p "${OUT_DIR}"
    run_scenario_matrix "coverage"
) || true

This runs run_scenario_matrix in a subshell so the outer OUT_DIR is unaffected afterward, while the coverage-only pass writes its (discarded) results/logs to a separate subdirectory.

  • Step 5: Verify default run and skip flags
source env.sh
cd Test/E2E/suite
./run_e2e.sh --skip-build --only s01_scalar_uint32 --skip-stress --skip-datasources --skip-recorder --skip-debug --skip-tcplogger --skip-coverage 2>&1 | tail -20

Expected: fast single-scenario run, no coverage rebuild, no stress phase — same behavior as the pre-Task-7 fast-iteration workflow.

./run_e2e.sh --skip-build --skip-stress 2>&1 | tail -40

Expected: full scenario matrix (chain+direct+recorder+debug+tcplogger) runs once for results, then again (coverage_pass subdir) under instrumented binaries, stress_results.json absent/skipped, final report_data.json/PDF produced without a stress section (or with a "stress: skipped" placeholder — verify report_build.py from Task 8 handles a missing --stress-results file gracefully; if Task 8 hasn't landed yet in execution order, this check is deferred to that task's verification instead).

  • Step 6: Commit
git add Test/E2E/suite/run_e2e.sh Test/E2E/suite/stress.py
git commit -m "feat(e2e): instrument-first coverage flow with double-run + new skip flags; port stress multi-fragment sizing"

Task 8: Extend report_build.py + E2E_Report.typ with per-kind and stress sections

Files:

  • Modify: Test/E2E/suite/report_build.py
  • Modify: Test/E2E/suite/E2E_Report.typ
  • Modify: Test/E2E/suite/tests_py.py

Interfaces:

  • Consumes: results.json entries now carrying kind (Tasks 3/4/6); stress_results.json = {"overall","cases":[...]} (from stress_run.py, unchanged shape).

  • Produces: report_data.json gains "direct", "recorder", "debug", "tcplogger", "stress", "stress_plots" top-level keys; headline()'s return dict gains matching scalar fields; regression() compares them against history.jsonl; E2E_Report.typ renders one section per kind.

  • Step 1: Port the stress reporting functions from the stale branch

git show feature/stress-report-integration:Test/E2E/chain/report_build.py > /tmp/report_build_feature_branch.py
diff /tmp/report_build_feature_branch.py Test/E2E/suite/report_build.py

Port these specific additions (adapting names/paths for the current file — the stale branch predates the kind-based scenario model, so integrate its logic as new standalone functions rather than replacing existing ones):

  • _STRESS_LABELS, _STRESS_DIRECTION (module-level dicts).

  • build_stress(sr) — reads stress_results.json's cases list, groups by axis, returns {"overall", "by_axis": {axis: [case,...]}, "n_cases"}.

  • stress_headline(stress) — flattens key scalar stress metrics (e.g. stress_pass_rate, stress_max_rss_mb) into the same shape headline() returns.

  • _STRESS_AXIS_METRICS — per-axis field-pairs table used by stress_plots.

  • stress_plots(by_axis, out) — writes per-axis scaling PNGs to out, returns list of paths (mirrors trend_plots's return-list-of-paths pattern).

  • Step 2: Add build_direct/build_recorder/build_debug/build_tcplogger

These are simpler than build_stress since they read the same results.json shape build_e2e() already consumes, just filtered by kind. Add:

def build_by_kind(results, kind):
    """Filter results.json's scenario list to one kind, mirroring build_e2e()'s
    per-scenario shape but without the chain-specific corr/nrmse/perf fields."""
    scenarios = [s for s in results.get("scenarios", []) if s.get("kind") == kind]
    n_pass = sum(1 for s in scenarios if s.get("status") == "PASS")
    n_fail = sum(1 for s in scenarios if s.get("status") == "FAIL")
    return {
        "kind": kind,
        "n_pass": n_pass,
        "n_fail": n_fail,
        "n_total": len(scenarios),
        "scenarios": scenarios,
    }

(Confirm results.json's actual top-level shape — re-read build_e2e(results, work)'s existing body to see whether scenario records live under results["scenarios"] or a differently-named key, and match that exactly; adjust the .get("scenarios", []) key name if it differs.)

  • Step 3: Wire everything into main()

Extend the argument parser:

    ap.add_argument("--stress-results", default=None)

After the existing e2e = build_e2e(results, work) call, add:

    direct = build_by_kind(results, "direct")
    recorder = build_by_kind(results, "recorder")
    debug = build_by_kind(results, "debug")
    tcplogger = build_by_kind(results, "tcplogger")

    stress = None
    stress_plot_paths = []
    if args.stress_results and os.path.exists(args.stress_results):
        with open(args.stress_results) as f:
            sr = json.load(f)
        stress = build_stress(sr)
        stress_plot_paths = stress_plots(stress["by_axis"], out)

Extend headline(e2e, ut, cov)'s call site and signature to also fold in the new sections:

    hl = headline(e2e, ut, cov)
    hl.update({
        "direct_pass": direct["n_pass"], "direct_fail": direct["n_fail"],
        "recorder_pass": recorder["n_pass"], "recorder_fail": recorder["n_fail"],
        "debug_pass": debug["n_pass"], "debug_fail": debug["n_fail"],
        "tcplogger_pass": tcplogger["n_pass"], "tcplogger_fail": tcplogger["n_fail"],
    })
    if stress:
        hl.update(stress_headline(stress))

Extend _DIRECTION (module-level) with the new headline keys:

_DIRECTION.update({
    "direct_pass": True, "direct_fail": False,
    "recorder_pass": True, "recorder_fail": False,
    "debug_pass": True, "debug_fail": False,
    "tcplogger_pass": True, "tcplogger_fail": False,
})
_DIRECTION.update(_STRESS_DIRECTION)

Extend the final report_data dict and history.jsonl append to include the new sections:

    report_data = {
        "meta": meta, "e2e": e2e, "unit_tests": ut, "coverage": cov,
        "direct": direct, "recorder": recorder, "debug": debug, "tcplogger": tcplogger,
        "stress": stress, "stress_plots": stress_plot_paths,
        "regression": regression(hl, prev_hl), "headline": hl,
        "trend_plots": trend_plots(history, out),
        "history_len": len(history), "is_first_run": is_first_run,
    }

(Match whatever the existing main() already names its local variables for meta/prev_hl/history/is_first_run — read the current file's main() body first and adapt variable names exactly rather than introducing new ones.)

  • Step 4: Add Typst sections

In E2E_Report.typ, after the existing "Scenarios" section (around line 224 per the pre-gathered survey), add:

#if report.direct != none [
  == Direct Round-Trip (UDPStreamer ↔ UDPStreamerClient)
  #table(
    columns: 3,
    [*Scenario*], [*Status*], [*Detail*],
    ..report.direct.scenarios.map(s => (s.id, s.status, s.at("detail", default: ""))).flatten()
  )
]

#if report.recorder != none [
  == Recorder (BinaryRecorder disk output)
  #table(
    columns: 3,
    [*Scenario*], [*Status*], [*Detail*],
    ..report.recorder.scenarios.map(s => (s.id, s.status, s.at("detail", default: ""))).flatten()
  )
]

#if report.debug != none [
  == Debug Service E2E
  #table(
    columns: 3,
    [*Scenario*], [*Status*], [*Detail*],
    ..report.debug.scenarios.map(s => (s.id, s.status, s.at("detail", default: ""))).flatten()
  )
]

#if report.tcplogger != none [
  == TCPLogger E2E
  #table(
    columns: 3,
    [*Scenario*], [*Status*], [*Detail*],
    ..report.tcplogger.scenarios.map(s => (s.id, s.status, s.at("detail", default: ""))).flatten()
  )
]

#if report.stress != none [
  == Stress Tests
  #for (axis, cases) in report.stress.by_axis [
    === #axis
    #table(
      columns: 4,
      [*Level*], [*Status*], [*Peak RSS (MB)*], [*Zoom p95 (ms)*],
      ..cases.map(c => (str(c.level), c.status, str(c.at("marte_rss_mb", default: 0)), str(c.at("zoom_p95_ms", default: 0)))).flatten()
    )
  ]
  #for p in report.stress_plots [
    #image(p)
  ]
] else [
  == Stress Tests
  _Not run this session (use without --skip-stress to include)._
]

(Confirm Typst's exact JSON-access syntax matches the version in use — check the existing template's patterns for accessing report.e2e.scenarios or similar nested fields, and mirror that exact syntax rather than the illustrative dot-access above if the template uses a different accessor style, e.g. report.at("e2e").)

  • Step 5: Extend tests_py.py
def test_build_by_kind_filters_correctly(self):
    import report_build as R
    results = {"scenarios": [
        {"id": "a", "kind": "direct", "status": "PASS"},
        {"id": "b", "kind": "chain", "status": "PASS"},
        {"id": "c", "kind": "direct", "status": "FAIL"},
    ]}
    d = R.build_by_kind(results, "direct")
    self.assertEqual(d["n_pass"], 1)
    self.assertEqual(d["n_fail"], 1)
    self.assertEqual(d["n_total"], 2)
  • Step 6: Run the full pipeline and inspect the PDF
source env.sh
cd Test/E2E/suite && ./run_e2e.sh 2>&1 | tail -60

Expected: completes with all six scenario kinds + stress + coverage, final line PDF: .../E2E_Report.pdf.

python3 -m unittest tests_py -v 2>&1 | tail -10

Expected: all pass, including the new test_build_by_kind_filters_correctly.

Open/inspect Build/x86-linux/E2E/suite/E2E_Report.pdf (or extract text) and confirm it contains "Direct Round-Trip", "Recorder", "Debug Service E2E", "TCPLogger E2E", and "Stress Tests" section headers:

pdftotext Build/x86-linux/E2E/suite/E2E_Report.pdf - | grep -E "Direct Round-Trip|Recorder|Debug Service E2E|TCPLogger E2E|Stress Tests"

Expected: all 5 strings found.

  • Step 7: Commit
git add Test/E2E/suite/report_build.py Test/E2E/suite/E2E_Report.typ Test/E2E/suite/tests_py.py
git commit -m "feat(e2e): render direct/recorder/debug/tcplogger/stress sections in the unified report"

Task 9: Retire duplicate/superseded suites

Files:

  • Delete: Test/E2E/streamhub/ (entire directory)
  • Delete: run_e2e_test.sh
  • Delete: Test/E2E/datasources/run_e2e_report.sh
  • Delete: Test/E2E/datasources/E2E_Report.typ
  • Delete: Test/E2E/recorder/run_recorder_e2e.sh
  • Delete: run_combined_test.sh
  • Modify: CLAUDE.md / AGENTS.md (remove references to the deleted scripts)

Interfaces:

  • Consumes: nothing (this task only removes files whose functionality is now covered by Tasks 4/6/8).

  • Produces: no dangling references to deleted files anywhere in the repo.

  • Step 1: Confirm no remaining references before deleting

grep -rln "run_e2e_test.sh\|run_recorder_e2e.sh\|run_e2e_report.sh\|run_combined_test.sh\|Test/E2E/streamhub" \
    --include="*.md" --include="*.sh" --include="*.py" --include="*.go" . 2>/dev/null

Review every hit. Expected hits at this point: CLAUDE.md/AGENTS.md (to be edited in Step 3), and possibly this plan document itself (ignore). If any other script still shells out to one of these (e.g. a CI config file not yet seen), stop and flag it rather than deleting blindly — the spec's retirement list assumed no other consumers exist based on the Task-0 survey, but re-verify before executing.

  • Step 2: Delete the files
git rm -r Test/E2E/streamhub
git rm run_e2e_test.sh
git rm Test/E2E/datasources/run_e2e_report.sh
git rm Test/E2E/datasources/E2E_Report.typ
git rm Test/E2E/recorder/run_recorder_e2e.sh
git rm run_combined_test.sh
  • Step 3: Clean up doc references
grep -n "run_e2e_test.sh\|run_recorder_e2e.sh\|run_e2e_report.sh\|run_combined_test.sh\|Test/E2E/streamhub" CLAUDE.md AGENTS.md

For each match, remove the reference (delete the sentence/line if it exists solely to describe the deleted script) rather than leaving a dangling mention. If CLAUDE.md's demo-scripts line currently reads something like `./run_combined_test.sh`, `./run_streamhub.sh`, drop `./run_combined_test.sh`, and keep `./run_streamhub.sh` (that one is NOT being deleted — it's a pure interactive demo launcher unrelated to any retired test suite, confirmed in the original survey).

  • Step 4: Verify nothing else breaks
source env.sh
cd Test/E2E/suite && ./run_e2e.sh --skip-build --only s01_scalar_uint32 --skip-coverage --skip-stress 2>&1 | tail -10

Expected: still passes (confirms the deletions didn't remove anything run_e2e.sh itself depends on).

cd /home/martino/Projects/MARTe_Integrated_components
grep -rn "Test/E2E/streamhub\|run_e2e_test\|run_recorder_e2e\|run_e2e_report\|run_combined_test" --include="*.md" --include="*.sh" .

Expected: no output (all references cleaned up).

  • Step 5: Commit
git add -A CLAUDE.md AGENTS.md
git commit -m "chore(e2e): retire streamhub/datasources/recorder standalone scripts superseded by run_e2e.sh"

Task 10: Full pipeline verification and stale branch cleanup

Files: none (verification-only task).

Interfaces: none.

  • Step 1: Full clean run from scratch
source env.sh && export MAKEDEFAULTDIR=$MARTe2_DIR/MakeDefaults
cd Test/E2E/suite && ./run_e2e.sh 2>&1 | tee /tmp/full_run.log | tail -100

Expected: exits 0 (or check results.json's overall status explicitly), produces Build/x86-linux/E2E/suite/E2E_Report.pdf.

  • Step 2: Verify GTest count and full pass
./Build/x86-linux/GTest/MainGTest.ex 2>&1 | tail -5

Expected: [ PASSED ] N tests. with N = 79 (76 original + 3 BoundsCheckTest from Task 1; adjust if WSServerBufferTest.cpp also contributed previously-uncounted tests — check the actual delta and confirm 0 failures either way).

  • Step 3: Verify report_data.json structure
python3 -c "
import json
d = json.load(open('Build/x86-linux/E2E/suite/report_data.json'))
for k in ('e2e','unit_tests','coverage','direct','recorder','debug','tcplogger','stress','regression','headline'):
    print(k, 'present' if k in d else 'MISSING')
"

Expected: all keys print present.

  • Step 4: Verify coverage rose vs. the pre-unification baseline
python3 -c "
import json
d = json.load(open('Build/x86-linux/E2E/suite/report_data.json'))
print('C++ coverage:', d['coverage'])
"

Compare against the coverage percentage recorded in Build/x86-linux/E2E/suite/history.jsonl's second-to-last entry (the last pre-unification run). Expected: the new C++ coverage % is higher (E2E-exercised code now counts), or at minimum not lower — if it's lower, investigate before declaring this task done (likely cause: the coverage-pass run_scenario_matrix subshell in Task 7 not actually exercising the instrumented binaries — check that ${MARTE_APP}/${STREAMHUB_EX} path variables still point at the just-rebuilt instrumented binaries at that point in the script, not stale paths from Phase 1's normal build).

  • Step 5: Confirm fast single-scenario iteration still works
./run_e2e.sh --skip-build --only s01_scalar_uint32 --skip-coverage --skip-stress --skip-datasources --skip-recorder --skip-debug --skip-tcplogger 2>&1 | tail -10

Expected: runs in a few seconds, PASS, no coverage/stress/other-kind overhead — confirms the default-heavy run_e2e.sh still supports the fast dev-loop workflow via skip flags.

  • Step 6: Reconcile and delete the stale branch
git log --oneline main..feature/stress-report-integration
git branch -D feature/stress-report-integration

Ask the user for explicit confirmation before running the following (deletes the remote ref, a shared/visible action):

git push origin --delete feature/stress-report-integration
  • Step 7: Final full-suite commit (if any stray files remain uncommitted)
git status

If clean, no action needed. If any generated artifacts (e.g. depends.x86-linux regenerated files) show as modified, follow the repo's existing convention (per AGENTS.md/prior session history: these are build-generated and typically not re-committed unless the repo's .gitignore doesn't already exclude them — check git status output against .gitignore before adding anything).

Reference in New Issue View Git Blame Copy Permalink