245 lines
15 KiB
Markdown
245 lines
15 KiB
Markdown
# Repository Guidelines
|
||
|
||
Guide for AI assistants working in the MARTe2 Integrated Components repository.
|
||
Focuses on non-obvious facts: commands, conventions, cross-module contracts, and
|
||
gotchas that are not self-evident from a single file read.
|
||
|
||
## Project Overview
|
||
|
||
MARTe2 component library with **two independent real-time data paths** sharing
|
||
one binary wire protocol (`Common/UDP/UDPSProtocol.h`):
|
||
|
||
1. **Streaming path** — `UDPStreamer` DataSource serialises DDB signals into UDPS
|
||
binary packets on UDP → `StreamHub` (headless C++ hub: ring buffers, LTTB
|
||
decimation, trigger FSM, history writer, binary recorder) → WebSocket 8090 →
|
||
clients (browser SPA, native ImGui, native Qt).
|
||
2. **Debug path** — `DebugService` patches `ClassRegistryDatabase` at
|
||
`Initialise()` so `ConfigureApplication()` wraps all `MemoryMap*Broker` types
|
||
with `DebugBrokerWrapper<T>` — **zero application code changes**. Exposes
|
||
TCP 8080 (text commands), UDP 8081 (UDPS trace telemetry), TCP 8082
|
||
(`TcpLogger` log forward).
|
||
|
||
## Architecture & Data Flow
|
||
|
||
```
|
||
[SineArrayGAM/TimeArrayGAM] → DDB → UDPStreamer (UDPS over UDP)
|
||
├─→ UDPStreamerClient (input DS back into a MARTe2 RT app, round-trip)
|
||
└─→ StreamHub: UDPSourceSession (receive thread → SignalRingBuffer)
|
||
→ push loop @30Hz: LTTB decimate temporal sigs → WS binary frames → clients
|
||
DebugService: patches broker builders at Initialise(); TCP 8080 commands,
|
||
UDP 8081 telemetry, TcpLogger 8082 (REPORT_ERROR → "LOG <LEVEL> <desc>" lines)
|
||
```
|
||
|
||
- **Wire protocol**: `Common/UDP/UDPSProtocol.h` is the canonical spec (17-byte
|
||
packed header, magic `0x53504455` 'UDPS', 136-byte signal descriptors,
|
||
CONFIG/DATA/ACK/CONNECT/DISCONNECT packet types, quant/time/publish modes).
|
||
Deliberately MARTe2-free so Go clients reuse it. **Mirrored across four
|
||
codebases that must stay in sync**: C++ producers (UDPStreamer, DebugService),
|
||
C++ consumer (`Source/Components/Interfaces/UDPStream/UDPSClient`), Go decoder
|
||
(`Common/Client/go/udpsprotocol/protocol.go`), and JS parsers
|
||
(`Client/udpstreamer/static/`, `Client/debugger/static/`). Any protocol change
|
||
must be mirrored in all of them.
|
||
- **WS protocol** has two implementations — Go hub (`Common/Client/go/wshub`) and
|
||
C++ StreamHub — that must behave identically; every client (SPA, ImGui, Qt)
|
||
must satisfy both. JSON text frames for commands/events (`addSource`,
|
||
`removeSource`, `setTrigger`, `arm`, `zoom`, `historyZoom`, `recStart`…), binary
|
||
frames for data pushes (live v1 + trigger capture v2).
|
||
- **Threading model**: RT threads only spinlock+memcpy (`FastPollingMutexSem`);
|
||
all socket I/O, fragmentation, and reassembly lives on background
|
||
`SingleThreadService` threads. StreamHub: per-session UDPSClient receive
|
||
threads + WS accept/read threads + one push loop.
|
||
- **DebugService patching**: `PatchRegistry()` replaces the ObjectBuilder for 11
|
||
`MemoryMap*Broker` classes; runs only when `ControlPort > 0`; static guard
|
||
against double-patching; wrappers persist for process lifetime.
|
||
|
||
## Key Directories
|
||
|
||
| Path | Purpose |
|
||
|---|---|
|
||
| `Source/Components/DataSources/UDPStreamer/` | Output DataSource; UDP I/O on bg thread, RT thread only spinlock+memcpy in `Synchronise()` |
|
||
| `Source/Components/DataSources/UDPStreamerClient/` | Input DataSource (shared `UDPSClient`), double-buffered ready/scratch |
|
||
| `Source/Components/GAMs/` | `SineArrayGAM` (float32 sine, continuous phase), `TimeArrayGAM` (us-timer → per-sample timestamp array; `Anchor = FirstSample|LastSample|Continuous`, use `Continuous` for contiguous sources so a lost RT cycle cannot hole the time base) |
|
||
| `Source/Components/Interfaces/DebugService/` | Registry patching, `DebugBrokerWrapper.h`, TCP/UDP services |
|
||
| `Source/Components/Interfaces/TCPLogger/` | `LoggerConsumerI` forwarding `REPORT_ERROR` to ≤8 TCP clients |
|
||
| `Source/Components/Interfaces/UDPStream/` | Plain-C++ helpers (not MARTe2 Objects): `UDPSClient` (auto-reconnect + fragment reassembly), `UDPSServer` (not thread-safe — owner's Execute thread only) |
|
||
| `Source/Applications/StreamHub/` | Standalone app (links MARTe2 core): `StreamHub`, `UDPSourceSession`, `WSServer`, `TriggerEngine`, `HistoryWriter`, `BinaryRecorder`, `LTTB`, `SignalRingBuffer` |
|
||
| `Common/UDP/` | Canonical wire protocol (header-only, MARTe2-free) |
|
||
| `Common/Client/go/` | Go mirror: `udpsprotocol` (decoder), `wshub` (WS hub client) |
|
||
| `Client/udpstreamer/` | Go legacy direct-UDP oscilloscope web UI (connects straight to UDPStreamer, no StreamHub) |
|
||
| `Client/webui/` | Go thin static server; SPA talks WS directly to C++ StreamHub (discovers via `GET /hub`) |
|
||
| `Client/debugger/` | Go debug web UI for DebugService |
|
||
| `Client/streamhub/` | Native ImGui+SDL2+OpenGL oscilloscope (C++17, no MARTe2) |
|
||
| `Client/streamhub-qt/` | Native Qt Widgets oscilloscope (Qt6 preferred, Qt5 fallback) |
|
||
| `Test/` | GTest, legacy Integration tests, Configurations (.cfg), E2E suite |
|
||
| `Docs/` | Per-component reference: `Protocol.md`, `UDPStreamer.md`, `StreamHub-{API,UserGuide,Developer}.md`, `DebugService.md`, `WebUI.md`, `Tutorial.md`, `E2E-Suite.md` |
|
||
|
||
## Development Commands
|
||
|
||
`source env.sh` is **mandatory** before any MARTe2 build or run (sets
|
||
`MARTe2_DIR`, `MARTe2_Components_DIR`, `TARGET=x86-linux`, `LD_LIBRARY_PATH`).
|
||
The E2E scripts source it themselves; a bare `make` from a fresh shell will not
|
||
work. `run_streamhub.sh` hard-errors if `MARTe2_DIR` is unset.
|
||
|
||
```bash
|
||
source env.sh
|
||
|
||
make -f Makefile.gcc core # 7 components (UDPStream interface FIRST, then UDPStreamer, UDPStreamerClient, GAMs, TCPLogger, DebugService)
|
||
make -f Makefile.gcc apps # StreamHub standalone app → Build/x86-linux/StreamHub/StreamHub.ex
|
||
make -f Makefile.gcc test # GTest + Integration test binaries + component test libs
|
||
make -f Makefile.gcc all # core + apps + test
|
||
make -f Makefile.gcc clean
|
||
|
||
# Single component:
|
||
make -C Source/Components/GAMs/SineArrayGAM -f Makefile.gcc
|
||
```
|
||
|
||
Build output → `Build/x86-linux/` mirroring `PACKAGE` paths (both `libX.so` and
|
||
`X.so` are produced). `compile_commands.json` (repo root, gitignored) feeds
|
||
LSP/clangd; CMake clients export their own into `Client/*/build/`.
|
||
|
||
### Non-MARTe2 clients (no env.sh needed)
|
||
|
||
```bash
|
||
cd Common/Client/go && go build ./...
|
||
cd Client/debugger && go build ./...
|
||
cd Client/streamhub && cmake -B build -DCMAKE_BUILD_TYPE=Release && cmake --build build
|
||
cd Client/streamhub-qt && cmake -B build && cmake --build build
|
||
```
|
||
|
||
### Key scripts
|
||
|
||
| Script | Purpose |
|
||
|---|---|
|
||
| `./run_streamhub.sh` | Demo stack: build + launch MARTe2 app + StreamHub, optional web (`-w`) / ImGui (`-g`) clients. Flags `-m/-c` MARTe2 dirs, `-b TARGET`, `-p WS_PORT`, `-n MAX_POINTS` (actual default 1000000, header says 10000), `-s` skip build. Generates temp hub cfg with `+History`/`+Recorder` blocks in `/tmp`. Ctrl-C kills all. |
|
||
| `./Test/E2E/suite/run_e2e.sh` | Full E2E: 57-scenario matrix + stress + unit suites + gcov coverage + Typst PDF report. Flags: `--skip-build`, `--only <id>`, `--pdf-only`, `--skip-coverage`, `--skip-stress`, `--skip-datasources`, `--skip-recorder`, `--skip-debug`, `--skip-tcplogger` |
|
||
| `./Test/E2E/suite/run_stress.sh` | Capacity harness: sweeps one load axis at a time (`--axis`), hard gates survival+liveness, soft gates RSS+zoom-p95 |
|
||
|
||
## Code Conventions & Common Patterns
|
||
|
||
- **No STL in `Source/Components/**` (and StreamHub)**: use `StreamString` (not
|
||
`std::string`), `FastPollingMutexSem`/`EventSem` (not `std::mutex`/threads),
|
||
fixed arrays / MARTe2 `Vector<T>` (not `std::vector`), `REPORT_ERROR` /
|
||
`REPORT_ERROR_STATIC` macros (no exceptions). C stdlib is fine. Heap
|
||
`new`/`delete[]` is normal. STL/C++17 is fine in `Client/streamhub/` and
|
||
`Client/streamhub-qt/`.
|
||
- **RT hot-path rule**: `FastPollingMutexSem` on real-time hot paths, never OS
|
||
mutexes; RT cycle must not block on the scheduler.
|
||
- **Class registration**: `CLASS_REGISTER_DECLARATION()` in the class `public:`
|
||
section of the header; `CLASS_REGISTER(Name, "1.0")` at the end of the `.cpp`
|
||
inside `namespace MARTe`. Every component `.cpp` ends with it.
|
||
- **EUPL v1.1 license headers** on all C++ sources and `Makefile.inc` — preserve
|
||
on new files.
|
||
- **Per-component build**: each dir has one-line `Makefile.gcc` wrapper
|
||
(`include Makefile.inc`) + `Makefile.inc` declaring `OBJSX`, `PACKAGE`,
|
||
`ROOT_DIR`, `INCLUDES` (re-declared per file, ~12 MARTe2 layer dirs),
|
||
`LIBRARIES`, including `MakeStdLibDefs.$(TARGET)` then
|
||
`MakeStdLibRules.$(TARGET)`. Generated `depends.x86-linux` (gcc -MM) is
|
||
committed but **never hand-edited** — delete to regenerate.
|
||
- **Qt client**: `QT_NO_KEYWORDS` is required (reused `Protocol.h` structs have
|
||
members named `signals`); Qt classes use `Q_SIGNALS`/`Q_SLOTS`/`Q_EMIT`. Run
|
||
with long options: `--host HOST --port 8090` (single-dash misparsed). Single
|
||
GUI thread, 60 Hz QTimer repaint.
|
||
- **StreamHub config** is *not* a MARTe2 `RealTimeApplication`: `Hub = { WSPort
|
||
MaxPoints PushRate MaxPushPoints RingTemporal RingScalar RingMaxMB AllowedOrigins
|
||
+Recorder{...} Sources={id={Label Addr Port}} }`. `AllowedOrigins` is the
|
||
WebSocket Origin allowlist — without it a browser serving the SPA from a
|
||
different port than the hub is rejected 403.
|
||
`+History` keys: `Directory` (required),
|
||
`DurationHours` (1), `Decimation` (1), `FlushIntervalSec` (5),
|
||
`MinDiskFreeMB` (500). `.shist` files: 64-byte header ('SHR1') + circular
|
||
(t,v) float64 pairs.
|
||
- **UDPStreamer config**: `Port` (44500; multicast data = `DataPort`, default
|
||
`Port+1`), `MaxPayloadSize` (1400), `PublishingMode` `Strict`/`Accumulate`,
|
||
per-signal `Signals={Name={Type,Unit,NumberOfDimensions,NumberOfElements,
|
||
TimeMode}}` with `TimeMode` `PacketTime`/`FirstSample`/`LastSample`/`FullArray`;
|
||
multicast needs `MulticastGroup` + `Interface`.
|
||
|
||
## Important Files
|
||
|
||
- `env.sh` — environment; source first, always.
|
||
- `Makefile.gcc` / `Makefile.inc` (root) — build orchestration.
|
||
- `Common/UDP/UDPSProtocol.h` — canonical wire format; changing it triggers the
|
||
4-way mirror checklist above.
|
||
- `Source/Applications/StreamHub/main.cpp` — hub entry (`[-cfg file.cfg]
|
||
[-port N] [-maxPoints N]`); hub **must be heap-allocated** (~128 MB, exceeds
|
||
the 8 MB stack).
|
||
- `Test/Configurations/*.cfg` — MARTe2 app configs (`$App = { Class =
|
||
RealTimeApplication }` with `+Functions`, `+DataSources`, `+States`, `+Timings`
|
||
blocks); `streamhub_demo.cfg` and `TestApp.cfg` are good templates.
|
||
- `Test/E2E/suite/{scenarios,gen_data,gen_cfg,validate_waveform,stress}.py` —
|
||
declarative scenario matrix and generators consumed identically by the Go
|
||
chain-client and validators.
|
||
- `Client/debugger/main.go` — `-addr :7777` default, `-enable-dangerous-commands`
|
||
safety gate (CR-4) for FORCE/PAUSE/RESUME/STEP/BREAK/MSG.
|
||
|
||
## Runtime/Tooling Preferences
|
||
|
||
- **OS**: Linux x86_64 (`TARGET=x86-linux`). External deps live outside this
|
||
repo: `MARTe2_DIR` (default `~/workspace/MARTe2`) and
|
||
`MARTe2_Components_DIR` (default `~/workspace/MARTe2-components`) — edit
|
||
`env.sh` if they differ. `env.sh`'s `LD_LIBRARY_PATH` does **not** cover
|
||
UDPStreamerClient/UDPStream lib dirs.
|
||
- **C++**: MARTe2 `Makefile.gcc` wrapper system, gtest-1.7.0 for tests.
|
||
- **Go**: `go 1.21`; modules use `replace marte2/common => ../../Common/Client/go`
|
||
(`gorilla/websocket` v1.5.1). Go binaries are gitignored.
|
||
- **ImGui client**: needs SDL2; CMake FetchContent pins Dear ImGui **v1.91.8** +
|
||
ImPlot **v0.17** (`implot_items.cpp` is a slow -O3 TU, ~2 min rebuild).
|
||
- **Qt client**: Qt6 preferred, Qt5 fallback, Widgets + WebSockets, custom
|
||
QPainter plotting (no QtCharts).
|
||
- **E2E report**: `typst compile E2E_Report.typ`; Python 3 + numpy for the suite.
|
||
- Remove `vgore.*` core dumps when you see them; they are not gitignored.
|
||
|
||
## Testing & QA
|
||
|
||
Four test layers; `env.sh` + built stack required for all but the standalone
|
||
ones. Only `tests_py.py`, Go tests, and the built C++ test binaries run
|
||
standalone.
|
||
|
||
```bash
|
||
./Build/x86-linux/GTest/MainGTest.ex --gtest_filter='Name*' # C++ GTest
|
||
./Build/x86-linux/Test/Integration/Integration/IntegrationTests.ex # legacy DebugService runtime tests
|
||
cd Test/E2E/suite/client && go test ./... # Go chain-client unit tests
|
||
cd Test/E2E/suite && python3 -m unittest tests_py # framework logic, standalone
|
||
```
|
||
|
||
- **GTest**: `MainGTest.ex` currently holds only `DebugServiceGTest`
|
||
(TraceRingBuffer SPSC, DebugSignalInfo, BreakOp). Component GTests
|
||
(`UDPStreamerGTest.cpp` ~46 cases, `StreamHubTest.a`, `UDPStreamerClientTest.a`)
|
||
compile **as libraries only — no standalone executable**.
|
||
- **Legacy IntegrationTests.ex**: 9 printf-narrated DebugService runtime tests,
|
||
always returns 0; `collect.py` parses stdout blocks.
|
||
- **E2E suite** (`run_e2e.sh`): 57 curated scenarios (s01–s57) across kinds
|
||
`chain`/`direct`/`recorder`/`debug`/`debug_pause_resume`/`tcplogger`, driven
|
||
against live MARTeApp.ex + StreamHub.ex + Go chain-client. `scenarios.py` is a
|
||
curated covering set: **every configurable UDPStreamer option value appears in
|
||
≥1 scenario** — add a scenario when adding an option.
|
||
- **Oracle gates** (`validate_waveform.py`): **fidelity** (every received value
|
||
within `tol` of ground truth; 0 for un-quantised ints, float epsilon for
|
||
un-quantised floats, `quant_step/2 + 1e-6·range` for quantised) is the
|
||
**correctness gate**. **Shape** is a *gross* sanity gate + tracked metric
|
||
(`corr >= 0.5`, `nRMSE <= 0.30` relaxed by quant step, frequency searched
|
||
±5% band); a correct sinusoid yields corr ~0.82–0.98, wrong frequency
|
||
collapses to ~0.00. Do **not** tighten shape into a correctness gate —
|
||
timestamp calibration (Phase-A) is pending.
|
||
- **Stress** (`run_stress.sh`): 7 axes (signal size/count/fan-out/sources/WS
|
||
clients/zoom rate), hard gates survival+liveness, soft gates RSS+zoom-p95.
|
||
- **Coverage**: `--cpp-coverage` rebuilds with gcov, captures via `lcov`
|
||
restricted to `Source/*` + `Test/*`, then restores a clean build.
|
||
- Artifacts → `Build/x86-linux/E2E/chain/`: `results.json` (XFAIL/XPASS for
|
||
`known_issue` markers), `report_data.json`, `history.jsonl`, `trend_*.png`,
|
||
`E2E_Report.pdf`; stress → `stress/stress_results.json`.
|
||
|
||
## Ports Reference (defaults)
|
||
|
||
| Port | Protocol | Component | Purpose |
|
||
|---|---|---|---|
|
||
| 44500 | UDP | UDPStreamer | scalar signals (unicast control + data) |
|
||
| 44501/44502 | UDP | UDPStreamer | packed arrays (FirstSample/LastSample, FullArray) |
|
||
| 44503 | UDP | UDPStreamer | multicast data (group 239.0.0.1) |
|
||
| 8080 | TCP | DebugService | text command channel (one client at a time, newline-terminated) |
|
||
| 8081 | UDP | DebugService | trace telemetry (UDPS format) |
|
||
| 8082 | TCP | TcpLogger | REPORT_ERROR log forward |
|
||
| 8090 | TCP/WS | StreamHub | WebSocket (commands + binary data) |
|
||
| 7777 | TCP | Client/debugger | debug web UI (older docs say 9090; current flag is `-addr`) |
|
||
| 8080 | TCP | Client/udpstreamer, Client/webui | web UI listen (collides with DebugService in combined demos — scripts adjust) |
|