MARTe-Integrated-Components/Docs/StreamHub-Developer.md

# StreamHub — Developer Guide

`Source/Applications/StreamHub/` is a headless C++ application (MARTe2-linked,
MARTe2 coding style, no STL) that aggregates one or more UDPS sources
(`UDPStreamer` DataSources) and serves them to oscilloscope clients over
WebSocket. The Go hub (`Client/udpstreamer`) is the feature-complete reference
implementation of the same protocol and is kept untouched.

Wire protocols: [Protocol.md](Protocol.md) (UDPS, source → hub) and
[StreamHub-API.md](StreamHub-API.md) (WebSocket, hub → clients).

---

## 1. Source layout

| File | Role |
|------|------|
| `main.cpp` | CLI entry (`-cfg file.cfg -port N -maxPoints N`), signal handling |
| `StreamHub.{h,cpp}` | Top-level object: config, push loop, WS command dispatch, broadcasts |
| `UDPSourceSession.{h,cpp}` | One UDPS source: `UDPSClient` listener, payload decode, wall-clock calibration, ring buffers, stats |
| `SignalRingBuffer.h` | Per-signal (t,v) ring: monotonic write counter, `ReadSince` cursor reads, binary-search `ReadRange` |
| `TriggerEngine.{h,cpp}` | Trigger FSM (IDLE/ARMED/COLLECTING/TRIGGERED), edge detection per decoded sample |
| `UDPSourceStats.h` | 512-entry cycle/frag/byte rings → avg/std/min/max, rate, 20-bin histogram |
| `HistoryWriter.{h,cpp}` | Disk-backed circular history: per-signal `.shist` files, `WriteTick`, `ReadRange` (binary search via `pread`), disk space monitoring |
| `WSServer.{h,cpp}` | RFC 6455 server: handshake, framing, per-client write mutex, broadcast/unicast |
| `WSFrame.h`, `SHA1.h`, `Base64.h` | Header-only WS plumbing, shared with the ImGui client |
| `LTTB.h` | Largest-Triangle-Three-Buckets decimation |

The UDPS client itself lives in the shared library
`Source/Components/Interfaces/UDPStream/` (`UDPSClient`, also used by
`DebugService`). Note: in multicast mode the server delivers CONFIG over the
**TCP control connection**, so `UDPSClient` selects on both the multicast UDP
socket and the TCP socket and frames TCP reads.

## 2. Thread model

| Thread | Created by | Work |
|--------|-----------|------|
| main / push loop | `StreamHub::Run()` | At `PushRate` Hz: `PushData()` (serialise v1 frames), trigger servicing (capture finalisation, auto-rearm), `PushStats()` at `StatsRate` Hz |
| WS accept | `WSServer::Start()` | `accept()` + handshake, spawns client readers |
| WS client reader ×16 | `WSServer` | Reads frames, may contain several coalesced frames per TCP read; dispatches JSON to `StreamHub::OnWSCommand(json, len, slotIdx)` |
| UDPS receive ×32 | `UDPSClient::Start()` (one per session) | select() on UDP (+TCP in multicast), reassembles fragments, calls `UDPSourceSession` listener callbacks |

Synchronisation:
- Each `SignalRingBuffer` has its own `FastPollingMutexSem`; writers are the
  UDPS receive threads, readers are the push loop and zoom handlers.
- `WSServer` has a per-client write mutex (push loop and command replies can
  write concurrently).
- WS commands run on reader threads, but mutating operations (ring resize via
  `setMaxPoints`) are deferred to the push loop through pending atomics.
- Session slots use an `active` flag; removal never compacts the array, so
  indices stay stable.

## 3. Time base (wall-clock calibration)

All timestamps exposed to clients are **Unix wall-clock seconds** (float64).
Each session calibrates per time-source:

- First DATA packet anchors `pktCalibOffset = wallNow − hrt/hrtFreq`; thereafter
  `packetT = pktCalibOffset + hrt/hrtFreq`.
- Each referenced time signal gets its own offset on first value;
  `timerToSec = 1e-9` for `uint64` time signals, `1e-6` otherwise.
- The time-signal offset is **snapped** only on a genuine discontinuity in the
  source: reconnect, CONFIG change, or the source clock jumping backward (a
  looping/rewinding producer such as a rewinding `FileReader`).
- Plain *drift* — a source free-running on its own clock, or remote-vs-local HRT
  frequency error — is **slewed**, not snapped. Past a 2 s threshold the offset
  is nudged toward wall clock by at most 10 % of the packet's own duration.
  Snapping instead would shift the whole published timeline in one step and so
  tear a hole of exactly the drift into a stream that is in fact continuous;
  a source drifting past the threshold repeatedly used to produce a train of
  2 s holes. Drift is the honest reading, and the trade-off `TimeArrayGAM`'s
  `Anchor = Continuous` explicitly asks for: a producer that cannot sustain its
  nominal sample rate will fall progressively behind wall clock, and the hub
  reports that rather than hiding it. The Go hub anchors once and never
  re-anchors, so it never had the hole.

Per `timeMode`:

| timeMode | Timestamping |
|----------|-------------|
| FIRST/LAST_SAMPLE | anchor = calibrated time-signal value (fallback `packetT`); `t = anchor ± k·dt` from `samplingRate` |
| FULL_ARRAY | `t[k]` = calibrated time array element |
| PACKET, n=1 | `packetT` |
| PACKET, n>1 | elements span `(lastPktWall, wallNow]` — backward anchoring, deliberately different from the Go hub (which extrapolates forward and overlaps the next packet under jitter); keeps ring time strictly monotonic |

Multi-element PACKET signals are exposed per element as `name[i]`.

## 4. Push path (only-new samples)

`SignalRingBuffer` keeps a monotonic `totalWritten`; the hub keeps a
per-(session, signal) cursor and uses `ReadSince(cursor, …)` (clamped to the
oldest sample on overrun). Each tick serialises only new samples, LTTB-capped
to `MaxPushPoints` (default 50) per signal — LTTB is applied only to temporal
signals (multi-element, timeMode ≠ PACKET). Cursors advance even with zero WS
clients so a connecting client never receives a backlog burst. Cursors are
reset on (re)CONFIG and on ring resize.

This replaces the original "re-send the last N points each tick" design, which
caused visible trace corruption (LTTB picked different points per overlapping
window).

## 5. Trigger engine

Hub-side, web-client semantics (`setTrigger` fields in
[StreamHub-API.md](StreamHub-API.md)):

```
IDLE --arm--> ARMED --edge crossing--> COLLECTING --every source past trigTime+postSec+0.15s--> TRIGGERED
TRIGGERED --rearm (single) / auto ~200ms (normal, unless stopped)--> ARMED
any --disarm--> IDLE
```

`UDPSourceSession` calls `TriggerEngine::CheckSample` for every decoded sample
of the configured signal (signal index cached per config epoch). Each source is
read `[trigTime−preSec, trigTime+postSec]`, LTTB-capped to 20 000 pts/signal and
appended to a binary **version 2** capture frame; every FSM transition
broadcasts a `triggerState` event.

Once fired, that event carries `trigTime` **and** the window latched at fire
time (`preSec`/`postSec`). Clients draw the still-filling capture from their own
buffers on that axis long before the v2 frame arrives — for a long window at a
high rate the hub stays silent for seconds — and the trigger bar's window and
pre-% are editable, so without the latched values a client would place the
filling trace on whatever window the operator happened to be typing. Older hubs
omit both fields; clients fall back to their local config.

The COLLECTING deadline is on the **data's** clock, via
`UDPSourceSession::ProducerNewestTime()` — `trigTime` comes from sample
timestamps, and a source free-running on its own clock sits seconds away from
`clock_gettime()`, so a wall-clock deadline chops exactly that offset off every
capture's tail. Only signals actually timestamped from a time signal count
toward that reading: PACKET-timed ones (including the time array itself) are
stamped on arrival and would just report "now".

Sources are harvested independently — `BeginTriggerCapture`,
`HarvestTriggerCapture` per source as *it* becomes ready, `FinishTriggerCapture`
once all are in — with the frame accumulating in `capBuf_` across push ticks.
Waiting for the slowest source before reading any of them lets the leaders'
rings roll past the pre-trigger region first, losing the head of their traces. A
2 s wall-clock watchdog bounds the wait for a source that stopped advancing: it
is harvested short, with a warning naming the source and how far it got.

`setTrigger` also records the requested window, and each stats tick the push
loop runs `GrowRingsForTrigger()`. A ring whose measured rate
(`Count() / TimeSpan()`, since UDPS sources usually advertise
`samplingRate = 0`) cannot hold `window + 0.5 s` is grown in place to
`rate × (window + 0.5) × 1.2` points, clamped to `RingMaxMB` per signal.
`SignalRingBuffer::Grow()` copies oldest→newest and leaves `count` /
`totalWritten` untouched so the per-client push cursors survive the resize.
Rings never shrink; a hub left with a 5 s window on a 5 MSps source will sit at
the ceiling.

## 6. Configuration

```
WSPort        = 8090
MaxPoints     = 20000      // legacy global cap (overridable with -maxPoints)
PushRate      = 30         // Hz
MaxPushPoints = 50         // per signal per push
StatsRate     = 1          // Hz
RingTemporal  = 1000000    // initial ring capacity, temporal signals (pts)
RingScalar    = 100000     // ring capacity, scalar/PACKET signals (pts)
RingMaxMB     = 128        // per-signal growth ceiling (MiB) for trigger windows
SourcesFile   = "streamhub_sources.json"   // saveSources persistence
AllowedOrigins = "http://127.0.0.1:8099,http://localhost:8099"
              // comma/space-separated WebSocket Origin allowlist (max 8 × 128 chars).
              // Without it the handshake only accepts an Origin whose host matches
              // the request Host, so a browser serving the SPA from another port
              // (run_streamhub.sh: SPA 8099, hub 8090) gets 403. Non-browser
              // clients send no Origin and are unaffected.
Sources = {
    Src1 = { Label = "PSU"  Addr = "127.0.0.1" Port = 44500
             MulticastGroup = "239.0.0.1" DataPort = 44503 }  // multicast optional
}
```

Sources from `SourcesFile` are loaded after the static `Sources` block;
sources added at runtime via WS `addSource` get ids `s1, s2, …`.

### History configuration

An optional `+History` block enables disk-backed circular storage
(`HistoryWriter`). The `+` prefix is MARTe2 `StandardParser` syntax for a
child node; the hub looks for both `+History` and `History` as the node name.

```
+History = {
    Directory      = "/data/streamhub_history"   // required
    DurationHours  = 1          // hours of data to retain per signal (default 1)
    Decimation     = 10         // keep every Nth sample (default 1)
    FlushIntervalSec = 5        // header flush period in seconds (default 5)
    MinDiskFreeMB  = 500        // pause writes below this threshold (default 500)
}
```

Per-signal file capacity is computed at source CONFIG time:
`capacity = ceil(DurationHours × 3600 × samplingRate / Decimation)`, minimum
1000 pairs.

The Go hub (`Client/udpstreamer`) carries the same archive and the same file
format, configured with flags instead of a config node: `-history-dir`
(defaults to `<tmp>/udpstreamer-history`; empty disables),
`-history-window-sec`, `-history-decimation`, `-history-flush-sec`,
`-history-min-free-mb` (negative disables the check; 0 means the 500 MB default,
where the C++ `MinDiskFreeMB = 0` disables it) and `-history-max-mpts`, a
per-signal budget in millions of stored points, defaulting to 16 MPts (256 MB).
The budget exists because the timespan alone cannot bound the file: 600 s of a
1 MSps signal is 9.6 GB.

**The Go hub sizes its files from the window, not from a retention period.** The
archive exists to answer a zoom or a trigger capture after the in-memory rings
have rolled past it, and neither ever asks for more than the live or trigger
window — so a file holds `windowSec × rate` samples (plus 25 % headroom, since a
capture is read back a window after its first sample was written), and never
hours of them. Retaining an hour instead meant a 1 s live window was archived at
a thousandth of the resolution the same budget could have bought.

The budget is therefore spent on resolution, not on span. A signal too fast to
archive sample-for-sample within it is stored as a **min/max envelope**: `bucket`
source samples collapse to their two extremes, with `bucket` the narrowest that
makes the window fit. The `.shist` header's `decimation` field carries
`bucket × Decimation`, so a reader knows the stored resolution, and a file is
only reopened when it matches.

`Hub.retuneRings` re-sizes the files once a second alongside the rings, from the
same `activeWindowSec()`. A file's capacity and bucket are fixed at creation, so
a re-size discards what it held; two rules keep that rare. A file is only grown
when it no longer covers the window, and only shrunk when it is enveloped
(`bucket > 1`), covers more than twice the window, and a narrower bucket is
actually available — a file already at full resolution is left alone however
short the window becomes, so arming a 1 s trigger does not throw away the
seconds the capture is about to ask for. `historyInfo` is re-broadcast whenever a
re-size happens.

The budget is also settable at runtime from the web UI (the history badge in the
status bar) via the `setHistoryBudget` WS command; `historyInfo` reports it as
`maxMPtsPerSignal` and reports each signal's `bucket`. Changing it re-creates the
files, so the archived samples are lost — a file's capacity and bucket width are
fixed at creation and an existing envelope cannot be re-bucketed into a different
one.

History is on by default in the Go hub because it is what holds a trigger capture
at full resolution — see *Trigger captures* below. Signals whose producer
declares `samplingRate = 0` — every UDPS source — are not sized from a guess: the
file is opened only once the hub has measured the rate off the live stream, which
it retries once a second.

### `.shist` binary file format

Each signal gets one file: `<Directory>/<sourceId>/<signalName>.shist`.
The file size is fixed at creation (`64 + capacity × 16` bytes) and never grows.

```
Offset  Size  Field
0       4     Magic: "SHR1"
4       4     uint32 version (1)
8       4     uint32 capacity (max pairs)
12      4     uint32 head (next write position, 0-based, wraps at capacity)
16      4     uint32 count (valid entries, ≤ capacity)
20      4     uint32 decimation
24      8     float64 tOldest (Unix seconds)
32      8     float64 tNewest (Unix seconds)
40      24    reserved (zero-padded to 64 bytes)
64      …     data: capacity × 16 bytes (float64 time + float64 value per pair)
```

Data is written at the `head` position and wraps circularly. The oldest valid
entry is at logical index `(head + capacity − count) % capacity`. All I/O
uses `pwrite`/`pread` (no `mmap`), so concurrent reads from WS threads are
safe without locking.

Headers are flushed to disk every `FlushIntervalSec` seconds and on shutdown.
On restart, if an existing file has matching magic, version and capacity, it is
reopened — head/count/time bounds are restored from the on-disk header.

### History query path

`historyZoom` requests (see [StreamHub-API.md](StreamHub-API.md)) call
`HistoryWriter::ReadRange` which performs binary search over the circular file
using `pread` to locate the `[t0, t1]` window, then copies matching pairs.
If the result exceeds the requested `n`, decimation is applied (same decimator
as in-memory zoom: `LTTBDecimate` in the C++ hub, `minMaxDecimate` in the Go
one).

Both the web SPA and ImGui client issue `historyZoom` in parallel with regular
`zoom` and merge the results: history covers the older part of the visible
window, the in-memory ring covers the recent part.

A range wider than the read budget is thinned across its whole width with a
stride, not truncated at the front: answering a 10 s query with its first
few milliseconds reads as an empty plot to a client and sends it back to its
own coarse copy of the data.

### In-memory buffer policy (Go hub)

Each temporal signal gets one ring holding a fixed **budget** of `(t, v)` pairs:
10 M points, 160 MB, settable with `-ring-mpts`. Scalar signals keep a flat
100 000-packet ring, where a megasample budget would be waste. Rings start at
250 k points and are grown to the budget on demand, so a source that is
configured but never sends costs nothing.

Like the disk archive, the budget buys **resolution, not span**. Once a second
`retuneRings` compares the measured source rate against the window being
displayed and picks each ring's min/max `bucket`:

| condition | bucket | effect |
|---|---|---|
| `Sps × window ≤ budget` | 1 | stored verbatim; the ring reaches further back than the window, which is free zoom headroom |
| `Sps × window > budget` | `⌈2 × Sps × window × 1.25 ÷ capacity⌉` | `bucket` samples collapse to their two extremes, so the whole window fits |

A bucket costs two points (its minimum and its maximum), hence the factor 2 —
and why a bucket of 2 covers no more ground than a bucket of 1.

The window is the **trigger's** while a trigger is armed: its pre-window has to
already be in the ring when the trigger fires, or the capture has nothing to
back-fill from. Otherwise it is the widest window any connected client has
reported with the `setWindow` command, defaulting to 10 s for clients that never
send one. Sizing for the live window matters as much as for a capture: a fixed
sample-count ring covers ~6 s at 1 MSps, so a zoom on a 60 s timescale used to
come back with only its tail.

Retuning is hysteretic — a bucket is held while it covers the window without
covering more than twice it. Sharing one threshold for up and down makes a rate
jittering across a bucket boundary halve and double the stored resolution every
second.

Live pushes, the disk archive and the trigger comparator all see every sample:
`ingest` hands the raw batch to each, and only the ring's own copy is reduced.

### Trigger captures (Go hub)

A trigger capture is delivered as a decimated snapshot (20 000 points), so a
zoom into it has to come from full-resolution storage. The rings are tuned to
~1.25× the trigger window, so they roll past a captured window shortly after the
capture — and the trigger rearms and starts refilling them immediately.

**In-memory double buffer.** The rings are the write half; `captureHold`
(`capturehold.go`) is the read half. As `buildTriggerCapture` lifts each signal's
window out of its ring it publishes the *undecimated* slice into the hold, and
`zoomSlice` answers from the hold rather than the ring for any range the held
window fully contains. The swap happens only once the next capture is complete —
which is also the moment the client stops displaying the previous one — so the
shot being explored is never overwritten by the acquisition running behind it. A
capture that came back empty does not swap, so it cannot blank the window on
screen.

**Waiting for the buffer.** An armed trigger ignores edges until its buffers
reach back far enough for a capture taken now to come back whole (`fillLocked`
in `trigger.go`, fed by `refreshTriggerFill` from the trigger signal's own ring
— once per tick, and again on every trigger command so that an `arm` cannot
fire on a stale measurement). Firing earlier can only produce a capture whose
front was never recorded, which is what made the first shot after a widened
window come back short.

What must hold is that the buffer spans the whole window *at harvest time* — its
newest sample is then `trigTime + post`, so anything less has lost the front of
the capture. It keeps filling while the post-window is collected, so the
shortfall it may start with is what it will make up in that time, measured
rather than assumed:

```
need = windowSec − growth × postSec        (floored at the pre-trigger window)
```

`growth` is the ring's span growth in seconds per second, sampled over at least
`bufGrowthIntervalSec` and smoothed. The three regimes fall out of the one
formula:

| ring | growth | needs |
|---|---|---|
| still filling | 1 | the pre-trigger window — everything after the trigger is yet to be recorded anyway |
| full, re-bucketing for a longer window | 0…1 | in between: it drops dense old samples to take sparse new ones, so it fills slower than real time and the front of the capture recedes while the post-window elapses |
| full, settled | 0 | the whole window — which a ring tuned for that window already exceeds, so nothing actually waits |

Measured at 1 MSps, widening 10 s → 30 s with 50 % pre: growth settles at ~0.65,
so `need` converges on ~20.4 s of the 30 s and the trigger fires ~12 s after
arming with a capture that is 100 % complete. Requiring the whole window instead
would have waited 26 s for the same result.

The gate measures the trigger signal's ring, not the narrowest of all of them: a
signal that never reaches back that far would otherwise stop the trigger from
ever firing. It is disabled outright when there is no ring to measure or nothing
is needed, and `forceTrigger` overrides it. While it holds off, `triggerState`
carries `bufferFill`/`bufferNeedSec` and is re-broadcast as the fraction climbs,
so the UI shows `ARMED 42%` rather than a trigger that looks stuck.

**Back-filling a short capture.** A ring only spans the window once it has
rolled over completely at its current min/max bucket, which takes as long as the
window itself; widen the window, or arm right after setting it, and the first
captures start late and the client draws a blank front half.
`backfillCaptureHead` (`trigger.go`) therefore prepends whatever of
`[t0, ring's first sample)` the archive still holds, budgeting the read by the
share of the window being filled and trimming the overlap so the frame's
timestamps stay ascending. It needs history enabled; without it the capture is
simply short, and the hub logs by how much. The hold declines any range its own
samples do not actually cover, so a stretch neither source could supply falls
through to the archive instead of being redrawn as the same hole on every zoom
and every *fit*.

The hold declines ranges reaching outside its window: those are live zooms, and
only the rings still track the stream. Inside the window it needs no
trigger-state gating, because retuning never rewrites stored samples — a ring
that still covers the range holds the very same points. It is cleared when
`updateConfig` rebuilds the rings, since a restarted producer can replay the same
timestamps.

Budget: the hold costs one window per signal on top of the ring budget, up to a
further ~0.8 × `-ring-mpts`. Nothing is held until the first capture fires.

**On disk.** The archive covers what the hold cannot: ranges wider than the
capture window, and sessions where the hub restarted. It is circular and sized
from that same window, so it too wraps over a captured shot within a window of
delivering it. When a capture is delivered, the hub therefore copies
`[trigTime − pre, trigTime + post]` out of each `.shist` into a
`<signalName>.cap` file, laid out as a full non-wrapping `.shist`
(`capacity == count`, `head == 0`) so the same `readRange` reads it. A
`historyZoom` whose range the capture file fully contains is answered from it;
anything wider is answered from the archive. The copy is replaced by the next
trigger and by nothing else — rearming keeps it, because the client is still
showing that capture.

Budget: a capture costs one window's worth of disk per signal on top of
`-history-max-mpts`.

Protecting the window in place instead — pinning the region and refusing to
wrap onto it — does not work, and was tried: a capture held for longer than the
archive covers stops the archive dead, and the resulting hole lands exactly
where the *next* capture's pre-trigger window belongs.

## 7. Build & test

```bash
source env.sh                                    # always, for build and run

make -f Makefile.gcc apps                        # builds StreamHub.ex
# or: make -C Source/Applications/StreamHub -f Makefile.gcc
./Build/x86-linux/StreamHub/StreamHub.ex -cfg hub.cfg

make -f Makefile.gcc test
./Build/x86-linux/GTest/MainGTest.ex             # 71 unit tests, incl.
   # SignalRingBuffer (ReadSince / binary-search ReadRange / wrap),
   # TriggerEngine FSM, LTTB — sources in Test/Applications/StreamHub/

cd Test/E2E/suite && ./run_e2e.sh                # full-stack E2E (see below)
./run_streamhub.sh -w -g                         # interactive demo stack
```

### End-to-end test

`Test/E2E/suite/run_e2e.sh` is the unified E2E suite covering the whole
streaming + debug chain (`chain`/`direct`/`recorder`/`debug`/`tcplogger`
scenario kinds, see `Test/E2E/suite/scenarios.py`), including StreamHub live
push, zoom, window and trigger checks via the Go `chain-client`. It builds
everything, runs the scenario matrix plus the stress matrix, and produces a
consolidated `report_data.json` + Typst PDF report
(`Test/E2E/suite/E2E_Report.typ`). See the script's `--help` for options.

When changing the WS protocol, update **in lockstep**: this hub, the Go hub
(`Common/Client/go/wshub`), the browser SPA (`Client/udpstreamer/static`), the
ImGui client (`Client/streamhub/Protocol.cpp`), the E2E `chain-client`
(`Test/E2E/suite/client`), and [StreamHub-API.md](StreamHub-API.md).

## 8. Gotchas

- **No STL** in this directory (MARTe2 style); use `StreamString`,
  `FastPollingMutexSem`, fixed arrays.
- Link against UDPStream **dynamically** (`LIBRARIES += -lUDPStream`), never
  `LIBRARIES_STATIC` — the GTest binary links every `.a` it finds and would get
  duplicate symbols.
- WS text frames may arrive coalesced; never NUL-terminate a payload in place
  without restoring the byte (it is the first header byte of the next frame).
- Zoom replies must print `t` with `%.17g`: at Unix-epoch magnitudes `%.6f`
  destroys µs resolution.