refactor(e2e): rename Test/E2E/chain -> Test/E2E/suite, run_chain_e2e.sh -> run_e2e.sh
This commit is contained in:
Martino Ferrari
@@ -89,7 +89,7 @@ clangd. The CMake-based clients (ImGui, Qt) also export it into their `build/`.
|
|||||||
cd Common/Client/go && go build ./...
|
cd Common/Client/go && go build ./...
|
||||||
cd Client/debugger && go build ./...
|
cd Client/debugger && go build ./...
|
||||||
cd Client/udpstreamer && go build ./... # or: cd Client/webui && go build
|
cd Client/udpstreamer && go build ./... # or: cd Client/webui && go build
|
||||||
cd Test/E2E/chain/client && go build ./... # chain-client (E2E driver) + its tests
|
cd Test/E2E/suite/client && go build ./... # chain-client (E2E driver) + its tests
|
||||||
|
|
||||||
# ImGui desktop client (needs SDL2; fetches Dear ImGui + ImPlot via FetchContent)
|
# ImGui desktop client (needs SDL2; fetches Dear ImGui + ImPlot via FetchContent)
|
||||||
cd Client/streamhub && cmake -B build -DCMAKE_BUILD_TYPE=Release && cmake --build build
|
cd Client/streamhub && cmake -B build -DCMAKE_BUILD_TYPE=Release && cmake --build build
|
||||||
@@ -113,13 +113,13 @@ cd Client/streamhub-qt && cmake -B build && cmake --build build
|
|||||||
### Go tests
|
### Go tests
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
cd Test/E2E/chain/client && go test ./...
|
cd Test/E2E/suite/client && go test ./...
|
||||||
```
|
```
|
||||||
|
|
||||||
### Python framework tests (E2E framework logic, no live stack needed)
|
### Python framework tests (E2E framework logic, no live stack needed)
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
cd Test/E2E/chain && python3 -m unittest tests_py
|
cd Test/E2E/suite && python3 -m unittest tests_py
|
||||||
```
|
```
|
||||||
|
|
||||||
### E2E / demo scripts (build + launch the full stack)
|
### E2E / demo scripts (build + launch the full stack)
|
||||||
@@ -129,8 +129,8 @@ cd Test/E2E/chain && python3 -m unittest tests_py
|
|||||||
| `./run_streamhub.sh` | Launch MARTe2 app (UDPStreamer) + StreamHub, optionally web UI (`-w`) and ImGui client (`-g`). Ports documented in its header. |
|
| `./run_streamhub.sh` | Launch MARTe2 app (UDPStreamer) + StreamHub, optionally web UI (`-w`) and ImGui client (`-g`). Ports documented in its header. |
|
||||||
| `./run_combined_test.sh` | Combined streaming + debug integration test. `-d` auto-starts the debugger web UI. |
|
| `./run_combined_test.sh` | Combined streaming + debug integration test. `-d` auto-starts the debugger web UI. |
|
||||||
| `./run_e2e_test.sh` | Older standalone StreamHub E2E (`-s` skips build). |
|
| `./run_e2e_test.sh` | Older standalone StreamHub E2E (`-s` skips build). |
|
||||||
| `./Test/E2E/chain/run_chain_e2e.sh` | **Streaming-chain E2E suite**: per-scenario generates data + cfgs, runs MARTe2+StreamHub, drives the Go `chain-client` (live/zoom/window/trigger), validates the recorded waveform against an analytic/fed oracle, runs unit suites + coverage, builds a Typst PDF report. Flags: `--skip-build`, `--only <id>`, `--cpp-coverage`, `--pdf-only`, `--stress`. |
|
| `./Test/E2E/suite/run_e2e.sh` | **Streaming-chain E2E suite**: per-scenario generates data + cfgs, runs MARTe2+StreamHub, drives the Go `chain-client` (live/zoom/window/trigger), validates the recorded waveform against an analytic/fed oracle, runs unit suites + coverage, builds a Typst PDF report. Flags: `--skip-build`, `--only <id>`, `--cpp-coverage`, `--pdf-only`, `--stress`. |
|
||||||
| `./Test/E2E/chain/run_stress.sh` | **Capacity/stress harness**: sweeps one load axis at a time (signal size/count, subscriber fan-out, source count, WS-client count, zoom rate), gates on survival+liveness (hard) and RSS+zoom-p95 latency (soft). Flags: `--skip-build`, `--only <id>`, `--axis <axis>`. |
|
| `./Test/E2E/suite/run_stress.sh` | **Capacity/stress harness**: sweeps one load axis at a time (signal size/count, subscriber fan-out, source count, WS-client count, zoom rate), gates on survival+liveness (hard) and RSS+zoom-p95 latency (soft). Flags: `--skip-build`, `--only <id>`, `--axis <axis>`. |
|
||||||
|
|
||||||
The E2E suite produces `Build/x86-linux/E2E/chain/` artifacts: `report_data.json`,
|
The E2E suite produces `Build/x86-linux/E2E/chain/` artifacts: `report_data.json`,
|
||||||
`history.jsonl` (per-run headline metrics for trend/regression tracking),
|
`history.jsonl` (per-run headline metrics for trend/regression tracking),
|
||||||
@@ -246,7 +246,7 @@ default 1, `Decimation` default 1, `FlushIntervalSec` default 5,
|
|||||||
|
|
||||||
### E2E scenario / stress matrix
|
### E2E scenario / stress matrix
|
||||||
|
|
||||||
`Test/E2E/chain/scenarios.py` is a *curated covering set*: every configurable
|
`Test/E2E/suite/scenarios.py` is a *curated covering set*: every configurable
|
||||||
UDPStreamer option value appears in at least one scenario, plus high-risk
|
UDPStreamer option value appears in at least one scenario, plus high-risk
|
||||||
interactions. `stress.py` is the capacity sibling — it sweeps one load axis at
|
interactions. `stress.py` is the capacity sibling — it sweeps one load axis at
|
||||||
a time and records survival/liveness (hard gates) and RSS/zoom-p95 latency
|
a time and records survival/liveness (hard gates) and RSS/zoom-p95 latency
|
||||||
|
|||||||
@@ -39,7 +39,7 @@ cd Client/streamhub-qt && cmake -B build && cmake --build build
|
|||||||
|
|
||||||
End-to-end demo scripts (build + launch full stack, see headers for ports/options): `./run_combined_test.sh`, `./run_streamhub.sh`.
|
End-to-end demo scripts (build + launch full stack, see headers for ports/options): `./run_combined_test.sh`, `./run_streamhub.sh`.
|
||||||
|
|
||||||
**Streaming-chain E2E suite** (`Test/E2E/chain/`): `./run_chain_e2e.sh [--skip-build] [--only <id>] [--cpp-coverage]` drives the full chain per scenario (`scenarios.py`) — generates typed/shaped input + both cfgs, runs MARTe2+StreamHub, records via the Go `chain-client` (live/zoom/window/trigger), and validates the recorded waveform against an analytic/fed oracle (`validate_waveform.py`: fidelity gates correctness, sine shape-fit is a gross-sanity gate + tracked metric pending Phase-A timestamp calibration). It then runs the unit suites + coverage (`collect.py`: C++ GTest, Go, Python; `--cpp-coverage` does an instrumented `--coverage` rebuild, captures with lcov restricted to `Source/*`+`Test/*`, then restores the clean build), consolidates everything into `report_data.json` with per-field progression/regression vs the previous run and trend plots (`report_build.py`, history in `Build/x86-linux/E2E/chain/history.jsonl`), and compiles a Typst PDF (`E2E_Report.typ`). Python framework unit tests: `python3 -m unittest tests_py` (in `Test/E2E/chain/`).
|
**Streaming-chain E2E suite** (`Test/E2E/suite/`): `./run_e2e.sh [--skip-build] [--only <id>] [--cpp-coverage]` drives the full chain per scenario (`scenarios.py`) — generates typed/shaped input + both cfgs, runs MARTe2+StreamHub, records via the Go `chain-client` (live/zoom/window/trigger), and validates the recorded waveform against an analytic/fed oracle (`validate_waveform.py`: fidelity gates correctness, sine shape-fit is a gross-sanity gate + tracked metric pending Phase-A timestamp calibration). It then runs the unit suites + coverage (`collect.py`: C++ GTest, Go, Python; `--cpp-coverage` does an instrumented `--coverage` rebuild, captures with lcov restricted to `Source/*`+`Test/*`, then restores the clean build), consolidates everything into `report_data.json` with per-field progression/regression vs the previous run and trend plots (`report_build.py`, history in `Build/x86-linux/E2E/chain/history.jsonl`), and compiles a Typst PDF (`E2E_Report.typ`). Python framework unit tests: `python3 -m unittest tests_py` (in `Test/E2E/suite/`).
|
||||||
|
|
||||||
Build output goes to `Build/x86-linux/` (shared libs per component, `.ex` executables).
|
Build output goes to `Build/x86-linux/` (shared libs per component, `.ex` executables).
|
||||||
|
|
||||||
|
|||||||
BIN
Binary file not shown.
Binary file not shown.
@@ -109,7 +109,7 @@ def integration_suite(int_bin, work, timeout=220):
|
|||||||
def go_all_suites(repo, work):
|
def go_all_suites(repo, work):
|
||||||
"""Run Go test suites across all project modules and aggregate results."""
|
"""Run Go test suites across all project modules and aggregate results."""
|
||||||
modules = [
|
modules = [
|
||||||
(os.path.join(repo, "Test/E2E/chain/client"),
|
(os.path.join(repo, "Test/E2E/suite/client"),
|
||||||
"Go (chain-client)"),
|
"Go (chain-client)"),
|
||||||
(os.path.join(repo, "Common/Client/go"),
|
(os.path.join(repo, "Common/Client/go"),
|
||||||
"Go (common udpsprotocol + wshub)"),
|
"Go (common udpsprotocol + wshub)"),
|
||||||
@@ -1,5 +1,5 @@
|
|||||||
#!/usr/bin/env bash
|
#!/usr/bin/env bash
|
||||||
# run_chain_e2e.sh — Full-chain E2E orchestrator for the streaming chain
|
# run_e2e.sh — Full-chain E2E orchestrator for the streaming chain
|
||||||
#
|
#
|
||||||
# MARTe2 app (FileReader -> IOGAM -> UDPStreamer)
|
# MARTe2 app (FileReader -> IOGAM -> UDPStreamer)
|
||||||
# -> UDPS -> StreamHub -> chain-client (record + zoom/window/trigger)
|
# -> UDPS -> StreamHub -> chain-client (record + zoom/window/trigger)
|
||||||
@@ -10,7 +10,7 @@
|
|||||||
# against the analytic/fed oracle, renders plots, and aggregates results.json.
|
# against the analytic/fed oracle, renders plots, and aggregates results.json.
|
||||||
# Artifacts: Build/x86-linux/E2E/chain/ (report) and /tmp/chain_e2e/ (scratch).
|
# Artifacts: Build/x86-linux/E2E/chain/ (report) and /tmp/chain_e2e/ (scratch).
|
||||||
#
|
#
|
||||||
# Usage: ./run_chain_e2e.sh [--skip-build] [--only <id>] [--pdf-only]
|
# Usage: ./run_e2e.sh [--skip-build] [--only <id>] [--pdf-only]
|
||||||
set -u
|
set -u
|
||||||
|
|
||||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||||
@@ -88,8 +88,8 @@ fi
|
|||||||
# ── Scenario list (id|ws_port|udp_port0|network|oracle|trig|checks) ──────────
|
# ── Scenario list (id|ws_port|udp_port0|network|oracle|trig|checks) ──────────
|
||||||
LIST="$(${PY} - "${ONLY}" <<'PY'
|
LIST="$(${PY} - "${ONLY}" <<'PY'
|
||||||
import sys, os
|
import sys, os
|
||||||
sys.path.insert(0, os.path.dirname(os.path.abspath("Test/E2E/chain/scenarios.py")))
|
sys.path.insert(0, os.path.dirname(os.path.abspath("Test/E2E/suite/scenarios.py")))
|
||||||
sys.path.insert(0, os.path.join(os.getcwd(), "Test/E2E/chain"))
|
sys.path.insert(0, os.path.join(os.getcwd(), "Test/E2E/suite"))
|
||||||
import scenarios as S
|
import scenarios as S
|
||||||
only = sys.argv[1] if len(sys.argv) > 1 else ""
|
only = sys.argv[1] if len(sys.argv) > 1 else ""
|
||||||
for s in S.SCENARIOS:
|
for s in S.SCENARIOS:
|
||||||
@@ -8,7 +8,7 @@
|
|||||||
# It sweeps one load axis at a time (signal size/count, subscriber fan-out, source
|
# It sweeps one load axis at a time (signal size/count, subscriber fan-out, source
|
||||||
# count, WS-client count, zoom request rate — see stress.py) and gates each case on
|
# count, WS-client count, zoom request rate — see stress.py) and gates each case on
|
||||||
# survival + liveness (hard) and RSS + zoom-p95 latency (soft). This is the
|
# survival + liveness (hard) and RSS + zoom-p95 latency (soft). This is the
|
||||||
# capacity sibling of run_chain_e2e.sh (which gates waveform correctness).
|
# capacity sibling of run_e2e.sh (which gates waveform correctness).
|
||||||
#
|
#
|
||||||
# Usage: ./run_stress.sh [--skip-build] [--only <id>] [--axis <axis>]
|
# Usage: ./run_stress.sh [--skip-build] [--only <id>] [--axis <axis>]
|
||||||
set -u
|
set -u
|
||||||
@@ -2,7 +2,7 @@
|
|||||||
"""
|
"""
|
||||||
stress_run.py — Orchestrator for the streaming-chain stress matrix (stress.py).
|
stress_run.py — Orchestrator for the streaming-chain stress matrix (stress.py).
|
||||||
|
|
||||||
Where run_chain_e2e.sh drives scenarios.py for *correctness*, this drives
|
Where run_e2e.sh drives scenarios.py for *correctness*, this drives
|
||||||
STRESS_CASES for *capacity*. Per case it:
|
STRESS_CASES for *capacity*. Per case it:
|
||||||
|
|
||||||
1. generates the FileReader input + MARTe app cfg + 1..M StreamHub cfgs
|
1. generates the FileReader input + MARTe app cfg + 1..M StreamHub cfgs
|
||||||
Reference in New Issue
Block a user