MARTe-Integrated-Components/AGENTS.md

# AGENTS.md

Guide for agents working in the MARTe2 Integrated Components repository. Read
`ARCHITECTURE.md` and `CLAUDE.md` for deeper detail; this file focuses on
non-obvious knowledge, commands, and conventions that are not self-evident from
a single file read.

---

## Repository at a glance

MARTe2 component library with two independent real-time data paths:

1. **Streaming path** — `UDPStreamer` DataSource (serialises signals to UDPS
   binary packets on UDP 44500) → `StreamHub` (headless C++ hub: ring buffers,
   LTTB decimation, trigger FSM, history writer) → WebSocket 8090 → clients
   (browser SPA, native ImGui, native Qt).
2. **Debug path** — `DebugService` Interface patches `ClassRegistryDatabase` at
   `Initialise()` so `ConfigureApplication()` wraps all `MemoryMap*Broker` types
   with `DebugBrokerWrapper<T>`. Zero application code changes. Exposes TCP 8080
   (text commands), UDP 8081 (trace telemetry), TCP 8082 (`TcpLogger`).

Key shared artefact: `Common/UDP/UDPSProtocol.h` — the UDPS binary wire format
(17-byte packed header, 136-byte signal descriptors, little-endian). It is
deliberately MARTe2-free and is mirrored across **four** codebases that must stay
in sync: C++ producers (`UDPStreamer`, `DebugService`), C++ consumer
(`Source/Components/Interfaces/UDPStream/UDPSClient`), Go decoder
(`Common/Client/go/udpsprotocol`), and the JS client parsers. Any protocol
change must be mirrored in all of them.

StreamHub WebSocket protocol: JSON text frames for commands/events, binary
frames for data pushes (spec in `ARCHITECTURE.md` §6). The Go hub
(`Client/udpstreamer`) and the C++ StreamHub implement the identical protocol;
browser JS, ImGui, and Qt clients must stay compatible with both.

---

## Environment setup (do this first, always)

`source env.sh` is **required** before building *or* running anything. It sets:

- `MARTe2_DIR` (default `~/workspace/MARTe2`) and `MARTe2_Components_DIR`
  (default `~/workspace/MARTe2-components`) — external dependencies, not in
  this repo. Edit `env.sh` if your MARTe2 install lives elsewhere.
- `TARGET=x86-linux`
- `LD_LIBRARY_PATH` — prepends MARTe2 Core, MARTe2-components (LinuxTimer,
  FileDataSource), and this repo's built `.so`s (UDPStreamer, SineArrayGAM,
  TimeArrayGAM, DebugService, TCPLogger).

Without sourcing `env.sh`, builds fail (can't find `MakeDefaults`) and binaries
fail to load shared libs. The E2E scripts source it themselves, but a bare
`make` or `./MainGTest.ex` from a fresh shell will not.

---

## Build commands

All C++ builds use the MARTe2 `Makefile.gcc` wrapper system. There is one
top-level `Makefile.gcc` and one per component directory. `TARGET` defaults to
`x86-linux`; pass `TARGET=x86-linux` explicitly when invoking `make -C` from
elsewhere.

```bash
source env.sh

make -f Makefile.gcc core    # all C++ MARTe2 components (UDPStreamer, GAMs, DebugService, TCPLogger, UDPStream)
make -f Makefile.gcc apps    # StreamHub standalone app
make -f Makefile.gcc test    # build GTest + Integration test binaries
make -f Makefile.gcc clean
make -f Makefile.gcc         # = all: core apps test

# Build a single component (each component dir has its own Makefile.gcc):
make -C Source/Components/DataSources/UDPStreamer -f Makefile.gcc
make -C Source/Applications/StreamHub -f Makefile.gcc
```

Build output goes to `Build/x86-linux/` — shared libs under
`Build/x86-linux/Components/<...>/`, the StreamHub executable at
`Build/x86-linux/StreamHub/StreamHub.ex`, test binaries under
`Build/x86-linux/GTest/` and `Build/x86-linux/Test/Integration/`.

`compile_commands.json` is generated at the repo root (gitignored) for LSP /
clangd. The CMake-based clients (ImGui, Qt) also export it into their `build/`.

### Non-MARTe2 clients (separate build systems, no env.sh needed)

```bash
# Go clients (UDPS web client, debug client, E2E chain-client)
cd Common/Client/go && go build ./...
cd Client/debugger   && go build ./...
cd Client/udpstreamer && go build ./...        # or: cd Client/webui && go build
cd Test/E2E/suite/client && go build ./...     # chain-client (E2E driver) + its tests

# ImGui desktop client (needs SDL2; fetches Dear ImGui + ImPlot via FetchContent)
cd Client/streamhub && cmake -B build -DCMAKE_BUILD_TYPE=Release && cmake --build build

# Qt desktop client (Qt5 or Qt6 Widgets + WebSockets; autodetects, prefers Qt6)
cd Client/streamhub-qt && cmake -B build && cmake --build build
```

---

## Run / test commands

### Unit & integration tests (C++)

```bash
./Build/x86-linux/GTest/MainGTest.ex                              # UDPStreamer unit tests
./Build/x86-linux/GTest/MainGTest.ex --gtest_filter='Name*'       # single test
./Build/x86-linux/Test/Integration/Integration/IntegrationTests.ex   # DebugService integration tests
```

### Go tests

```bash
cd Test/E2E/suite/client && go test ./...
```

### Python framework tests (E2E framework logic, no live stack needed)

```bash
cd Test/E2E/suite && python3 -m unittest tests_py
```

### E2E / demo scripts (build + launch the full stack)

| Script | What it does |
|---|---|
| `./run_streamhub.sh` | Launch MARTe2 app (UDPStreamer) + StreamHub, optionally web UI (`-w`) and ImGui client (`-g`). Ports documented in its header. |
| `./Test/E2E/suite/run_e2e.sh` | **Unified E2E suite**: per-scenario generates data + cfgs, runs MARTe2+StreamHub, drives the Go `chain-client` (live/zoom/window/trigger) for `chain` scenarios plus `direct`/`recorder`/`debug`/`tcplogger` scenario kinds, runs a stress matrix, unit suites + coverage, builds a Typst PDF report. Flags: `--skip-build`, `--only <id>`, `--pdf-only`, `--cpp-coverage`, `--skip-coverage`, `--skip-stress`, `--skip-datasources`, `--skip-recorder`, `--skip-debug`, `--skip-tcplogger`. |
| `./Test/E2E/suite/run_stress.sh` | **Capacity/stress harness**: sweeps one load axis at a time (signal size/count, subscriber fan-out, source count, WS-client count, zoom rate), gates on survival+liveness (hard) and RSS+zoom-p95 latency (soft). Flags: `--skip-build`, `--only <id>`, `--axis <axis>`. |

The E2E suite produces `Build/x86-linux/E2E/chain/` artifacts: `report_data.json`,
`history.jsonl` (per-run headline metrics for trend/regression tracking),
`trend_*.png` plots, and `E2E_Report.pdf` (compiled from `E2E_Report.typ` via
`typst`). `--cpp-coverage` triggers an instrumented `--coverage` rebuild, captures
with `lcov` (restricted to `Source/*` + `Test/*`), then restores a clean build.

---

## Code organization

```
Common/
  UDP/UDPSProtocol.h            # shared UDPS binary protocol (header-only, MARTe2-free)
  Client/go/                    # Go packages: udpsprotocol (decoder), wshub (WS hub client)
Source/
  Components/                   # MARTe2 components — NO STL allowed here
    DataSources/UDPStreamer/    # real-time UDP signal streaming DataSource
    DataSources/UDPStreamerClient/   # (MARTe2-side UDPS client DataSource)
    GAMs/SineArrayGAM/          # sine-wave array generator GAM
    GAMs/TimeArrayGAM/          # time-reference array GAM
    Interfaces/DebugService/    # tracing/forcing/breakpoint Interface (registry-patched brokers)
    Interfaces/TCPLogger/       # REPORT_ERROR → TCP forwarder (LoggerConsumerI)
    Interfaces/UDPStream/       # UDPSClient/UDPSServer (C++ consumer/producer of UDPS)
  Applications/StreamHub/       # headless C++ hub app (links MARTe2 core but follows MARTe2 style)
Client/
  debugger/                     # Go web client for DebugService (browser UI in static/)
  udpstreamer/                  # Go web server + static SPA for UDPStreamer (legacy direct-UDP path)
  webui/                        # Go StreamHub WebSocket web UI (newer, hub-based)
  streamhub/                    # native ImGui desktop client (SDL2 + OpenGL + ImPlot, C++17, no MARTe2)
  streamhub-qt/                 # native Qt Widgets desktop client (C++17, Qt5/Qt6, no MARTe2)
Test/
  GTest/                        # GTest harness (MainGTest.cpp) for UDPStreamer unit tests
  Integration/                  # DebugService integration tests (link against DebugService .so)
  Components/DataSources/UDPStreamer/   # UDPStreamer unit test sources
  Configurations/               # MARTe2 .cfg files for tests and demos
  E2E/
    chain/                      # streaming-chain E2E + stress suite (Python orchestrator + Go client)
    datasources/ recorder/ streamhub/   # older per-component E2E scripts
Docs/                           # per-component reference docs (Protocol, UDPStreamer, StreamHub-*, DebugService, ...)
docs/superpowers/{specs,plans}/ # design specs + implementation plans (dated)
ARCHITECTURE.md                 # full architecture (data flow diagrams, protocol tables, WS protocol)
```

### Component layout convention

Every MARTe2 component directory contains:

- `Makefile.gcc` — thin wrapper (`TARGET = x86-linux` then `include Makefile.inc`).
- `Makefile.inc` — real build definition: `OBJSX`, `PACKAGE`, `INCLUDES`,
  `LIBRARIES`, includes `$(MARTe2_DIR)/MakeDefaults/MakeStdLibDefs.$(TARGET)`
  and `MakeStdLibRules.$(TARGET)`, and `depends.$(TARGET)`.
- `depends.x86-linux` / `dependsRaw.x86-linux` — generated header dependency
  files (committed; regenerated by the build). Do not hand-edit.
- `<ClassName>.{h,cpp}` — the implementation, with `CLASS_REGISTER(ClassName,
  "1.0")` at the bottom of the `.cpp` (MARTe2 class-registration macro).

To add a new component, copy an existing one and adapt `Makefile.inc`'s
`OBJSX` / `PACKAGE` / `INCLUDES` / `LIBRARIES`.

---

## Conventions and gotchas

### No STL in MARTe2 components

`Source/Components/**` must not use the C++ standard library. Use MARTe2 types:
`StreamString` (not `std::string`), `FastPollingMutexSem` (not `std::mutex`),
fixed arrays / `StaticList` (not `std::vector`), MARTe2 error macros
(`REPORT_ERROR`, `REPORT_ERROR_STATIC`) instead of exceptions. Includes come
from `$(MARTe2_DIR)/Source/Core/...` (BareMetal, Scheduler, FileSystem).

`Source/Applications/StreamHub/` links MARTe2 core and follows the same
no-STL-on-RT-paths style, but is a standalone app (not a loadable component).

STL / C++17 is **fine** in `Client/streamhub/` and `Client/streamhub-qt/`
(framework-free C++17, no MARTe2 dependency).

### RT hot-path mutex rule

Use `FastPollingMutexSem` on real-time hot paths — never OS mutexes
(`pthread_mutex`, `std::mutex`). The RT cycle must not block on the scheduler.

### Protocol change checklist

Changing `Common/UDP/UDPSProtocol.h` (or the Go mirror) requires mirroring
across all four consumers listed in §1, plus the JS client parsers in
`Client/udpstreamer/static/`. The Go decoder lives in
`Common/Client/go/udpsprotocol/protocol.go` and re-declares the constants
(`HeaderSize`, `SigDescSize`, packet types, quant types, time modes) — it does
**not** include the C header.

### Qt client: `QT_NO_KEYWORDS`

`Client/streamhub-qt/` reuses `../streamhub/Protocol.{h,cpp}` and
`SignalBuffer.h` verbatim. Those structs have members named `signals`, which
collide with Qt's `signals`/`slots`/`emit` macros. The Qt build sets
`QT_NO_KEYWORDS` and all Qt classes use `Q_SIGNALS:`/`Q_SLOTS:`/`Q_EMIT`. Do
not remove that define, and do not use the lowercase Qt keywords in Qt-client
code. Single GUI thread — `QWebSocket` signals arrive on the GUI thread, no
locks; a 60 Hz `QTimer` drives repaint. Run with long options:
`./build/StreamHubQtClient --host HOST --port 8090` (single-dash `-host` is
misparsed as clustered short flags).

### StreamHub history

`HistoryWriter` (`Source/Applications/StreamHub/HistoryWriter.{h,cpp}`) does
disk-backed circular storage: per-signal `.shist` files with a 64-byte header
and a pre-allocated circular region of `(float64 time, float64 value)` pairs.
Configured via a `+History` block (`Directory` required; `DurationHours`
default 1, `Decimation` default 1, `FlushIntervalSec` default 5,
`MinDiskFreeMB` default 500). WS commands: `historyInfo`, `historyZoom`.

### E2E scenario / stress matrix

`Test/E2E/suite/scenarios.py` is a *curated covering set*: every configurable
UDPStreamer option value appears in at least one scenario, plus high-risk
interactions. `stress.py` is the capacity sibling — it sweeps one load axis at
a time and records survival/liveness (hard gates) and RSS/zoom-p95 latency
(soft gates). Both feed the same `gen_data.py` / `gen_cfg.py` generators. When
adding a new UDPStreamer option, add a scenario that exercises it and update
`validate_scenario` / the stress matrix accordingly.

`validate_waveform.py` has two gates: **fidelity** (every received value within
`tol` of some ground-truth value; `tol` is 0 for un-quantised ints, a float
epsilon for un-quantised floats, `quant_step/2 + 1e-6·range` for quantised
floats) — this is the **correctness** gate. **Shape** (least-squares sine fit,
correlation ≥ 0.99) is a gross-sanity gate + tracked metric, pending Phase-A
timestamp calibration. Do not tighten the shape gate to a correctness gate.

### Gitignored build artefacts

`Build/`, `compile_commands.json`, `*.gcov`/`*.gcda`/`*.gcno` (coverage), Go
binaries (`Client/*/marte2debugger`, `udpstreamer-webui`, `streamhub-e2e`,
`Client/*/bin/`), and the `udp_standalone_webui/` scratch dir are all
gitignored. `vgore.*` core dumps (e.g. `vgcore.26968`) are not — remove them.

---

## Documentation map

- `ARCHITECTURE.md` — full architecture: data-flow diagrams, UDPS packet/header
  tables, CONFIG payload, StreamHub WebSocket protocol (commands, events,
  binary frames), threading model.
- `Docs/` — per-component reference docs (`Protocol.md`, `UDPStreamer.md`,
  `SineArrayGAM.md`, `DebugService.md`, `StreamHub-{UserGuide,API,Developer}.md`,
  `Tutorial.md`, `WebUI.md`).
- `docs/superpowers/specs/` — design specs (dated, e.g.
  `2026-06-26-stress-suite-report-integration-design.md`).
- `docs/superpowers/plans/` — implementation plans paired with specs.
- `.superpowers/sdd/` — task briefs, reports, and a `progress.md` ledger for
  in-flight feature work (not always present; per-branch).
- `README.md` — high-level overview + quick-start config snippets.
- `CLAUDE.md` — condensed agent guidance (overlaps with this file).

---

## Ports reference (defaults)

| Port | Protocol | Component | Purpose |
|------|----------|-----------|---------|
| 44500 | UDP | UDPStreamer | scalar signals (unicast control) |
| 44501 | UDP | UDPStreamer | packed arrays (FirstSample/LastSample) |
| 44502 | UDP | UDPStreamer | packed arrays (FullArray) |
| 44503 | UDP | UDPStreamer | multicast data (group 239.0.0.1) |
| 8080 | TCP | DebugService | text command channel |
| 8081 | UDP | DebugService | trace telemetry stream |
| 8082 | TCP | TcpLogger | REPORT_ERROR log forward |
| 8090 | TCP/WS | StreamHub | WebSocket (commands + binary data) |
| 8080 | TCP | udpstreamer-webui | web UI listen (legacy direct-UDP path) |
| 9090 | TCP | debugger | debug web UI listen |

Note the 8080 collision: DebugService control vs. the legacy web UI listen port.
In combined demos that run both, the scripts adjust one of them — check the
script header before assuming.

---

## License

EUPL v1.1 — license headers are present on all C++ sources. Preserve them on
new files; do not relicense.