martino/MARTe-Integrated-Components
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Implemented and fixed many issues

Browse Source
This commit is contained in:
Martino Ferrari
2026-08-21 23:24:48 +02:00
parent 14d5351a81
commit e03c60db25
52 changed files with 7726 additions and 769 deletions
Download Patch File Download Diff File Expand all files Collapse all files
Docs/StreamHub-Developer.md
+253 -10
View File
@@ -60,8 +60,20 @@ Each session calibrates per time-source:
`packetT = pktCalibOffset + hrt/hrtFreq`.
- Each referenced time signal gets its own offset on first value;
`timerToSec = 1e-9` for `uint64` time signals, `1e-6` otherwise.
- Re-anchoring on reconnect, CONFIG change, or if computed time drifts > 2 s
from wall clock (source restart / remote-vs-local HRT frequency drift).
- The time-signal offset is **snapped** only on a genuine discontinuity in the
source: reconnect, CONFIG change, or the source clock jumping backward (a
looping/rewinding producer such as a rewinding `FileReader`).
- Plain *drift* — a source free-running on its own clock, or remote-vs-local HRT
frequency error — is **slewed**, not snapped. Past a 2 s threshold the offset
is nudged toward wall clock by at most 10 % of the packet's own duration.
Snapping instead would shift the whole published timeline in one step and so
tear a hole of exactly the drift into a stream that is in fact continuous;
a source drifting past the threshold repeatedly used to produce a train of
2 s holes. Drift is the honest reading, and the trade-off `TimeArrayGAM`'s
`Anchor = Continuous` explicitly asks for: a producer that cannot sustain its
nominal sample rate will fall progressively behind wall clock, and the hub
reports that rather than hiding it. The Go hub anchors once and never
re-anchors, so it never had the hole.
Per `timeMode`:
@@ -94,16 +106,50 @@ Hub-side, web-client semantics (`setTrigger` fields in
[StreamHub-API.md](StreamHub-API.md)):
```
IDLE --arm--> ARMED --edge crossing--> COLLECTING --wallNow ≥ trigTime+postSec+0.15s--> TRIGGERED
IDLE --arm--> ARMED --edge crossing--> COLLECTING --every source past trigTime+postSec+0.15s--> TRIGGERED
TRIGGERED --rearm (single) / auto ~200ms (normal, unless stopped)--> ARMED
any --disarm--> IDLE
```
`UDPSourceSession` calls `TriggerEngine::CheckSample` for every decoded sample
of the configured signal (signal index cached per config epoch). On
finalisation the push loop reads `[trigTime−preSec, trigTime+postSec]` from all
rings, LTTB-caps to 20 000 pts/signal and broadcasts a binary **version 2**
capture frame; every FSM transition broadcasts a `triggerState` event.
of the configured signal (signal index cached per config epoch). Each source is
read `[trigTime−preSec, trigTime+postSec]`, LTTB-capped to 20 000 pts/signal and
appended to a binary **version 2** capture frame; every FSM transition
broadcasts a `triggerState` event.
Once fired, that event carries `trigTime` **and** the window latched at fire
time (`preSec`/`postSec`). Clients draw the still-filling capture from their own
buffers on that axis long before the v2 frame arrives — for a long window at a
high rate the hub stays silent for seconds — and the trigger bar's window and
pre-% are editable, so without the latched values a client would place the
filling trace on whatever window the operator happened to be typing. Older hubs
omit both fields; clients fall back to their local config.
The COLLECTING deadline is on the **data's** clock, via
`UDPSourceSession::ProducerNewestTime()` — `trigTime` comes from sample
timestamps, and a source free-running on its own clock sits seconds away from
`clock_gettime()`, so a wall-clock deadline chops exactly that offset off every
capture's tail. Only signals actually timestamped from a time signal count
toward that reading: PACKET-timed ones (including the time array itself) are
stamped on arrival and would just report "now".
Sources are harvested independently — `BeginTriggerCapture`,
`HarvestTriggerCapture` per source as *it* becomes ready, `FinishTriggerCapture`
once all are in — with the frame accumulating in `capBuf_` across push ticks.
Waiting for the slowest source before reading any of them lets the leaders'
rings roll past the pre-trigger region first, losing the head of their traces. A
2 s wall-clock watchdog bounds the wait for a source that stopped advancing: it
is harvested short, with a warning naming the source and how far it got.
`setTrigger` also records the requested window, and each stats tick the push
loop runs `GrowRingsForTrigger()`. A ring whose measured rate
(`Count() / TimeSpan()`, since UDPS sources usually advertise
`samplingRate = 0`) cannot hold `window + 0.5 s` is grown in place to
`rate × (window + 0.5) × 1.2` points, clamped to `RingMaxMB` per signal.
`SignalRingBuffer::Grow()` copies oldest→newest and leaves `count` /
`totalWritten` untouched so the per-client push cursors survive the resize.
Rings never shrink; a hub left with a 5 s window on a 5 MSps source will sit at
the ceiling.
## 6. Configuration
@@ -113,9 +159,16 @@ MaxPoints = 20000 // legacy global cap (overridable with -maxPoints)
PushRate = 30 // Hz
MaxPushPoints = 50 // per signal per push
StatsRate = 1 // Hz
RingTemporal = 1000000 // ring capacity, temporal signals (pts)
RingTemporal = 1000000 // initial ring capacity, temporal signals (pts)
RingScalar = 100000 // ring capacity, scalar/PACKET signals (pts)
RingMaxMB = 128 // per-signal growth ceiling (MiB) for trigger windows
SourcesFile = "streamhub_sources.json" // saveSources persistence
AllowedOrigins = "http://127.0.0.1:8099,http://localhost:8099"
// comma/space-separated WebSocket Origin allowlist (max 8 × 128 chars).
// Without it the handshake only accepts an Origin whose host matches
// the request Host, so a browser serving the SPA from another port
// (run_streamhub.sh: SPA 8099, hub 8090) gets 403. Non-browser
// clients send no Origin and are unaffected.
Sources = {
Src1 = { Label = "PSU" Addr = "127.0.0.1" Port = 44500
MulticastGroup = "239.0.0.1" DataPort = 44503 } // multicast optional
@@ -145,6 +198,54 @@ Per-signal file capacity is computed at source CONFIG time:
`capacity = ceil(DurationHours × 3600 × samplingRate / Decimation)`, minimum
1000 pairs.
The Go hub (`Client/udpstreamer`) carries the same archive and the same file
format, configured with flags instead of a config node: `-history-dir`
(defaults to `<tmp>/udpstreamer-history`; empty disables),
`-history-window-sec`, `-history-decimation`, `-history-flush-sec`,
`-history-min-free-mb` (negative disables the check; 0 means the 500 MB default,
where the C++ `MinDiskFreeMB = 0` disables it) and `-history-max-mpts`, a
per-signal budget in millions of stored points, defaulting to 16 MPts (256 MB).
The budget exists because the timespan alone cannot bound the file: 600 s of a
1 MSps signal is 9.6 GB.
**The Go hub sizes its files from the window, not from a retention period.** The
archive exists to answer a zoom or a trigger capture after the in-memory rings
have rolled past it, and neither ever asks for more than the live or trigger
window — so a file holds `windowSec × rate` samples (plus 25 % headroom, since a
capture is read back a window after its first sample was written), and never
hours of them. Retaining an hour instead meant a 1 s live window was archived at
a thousandth of the resolution the same budget could have bought.
The budget is therefore spent on resolution, not on span. A signal too fast to
archive sample-for-sample within it is stored as a **min/max envelope**: `bucket`
source samples collapse to their two extremes, with `bucket` the narrowest that
makes the window fit. The `.shist` header's `decimation` field carries
`bucket × Decimation`, so a reader knows the stored resolution, and a file is
only reopened when it matches.
`Hub.retuneRings` re-sizes the files once a second alongside the rings, from the
same `activeWindowSec()`. A file's capacity and bucket are fixed at creation, so
a re-size discards what it held; two rules keep that rare. A file is only grown
when it no longer covers the window, and only shrunk when it is enveloped
(`bucket > 1`), covers more than twice the window, and a narrower bucket is
actually available — a file already at full resolution is left alone however
short the window becomes, so arming a 1 s trigger does not throw away the
seconds the capture is about to ask for. `historyInfo` is re-broadcast whenever a
re-size happens.
The budget is also settable at runtime from the web UI (the history badge in the
status bar) via the `setHistoryBudget` WS command; `historyInfo` reports it as
`maxMPtsPerSignal` and reports each signal's `bucket`. Changing it re-creates the
files, so the archived samples are lost — a file's capacity and bucket width are
fixed at creation and an existing envelope cannot be re-bucketed into a different
one.
History is on by default in the Go hub because it is what holds a trigger capture
at full resolution — see *Trigger captures* below. Signals whose producer
declares `samplingRate = 0` — every UDPS source — are not sized from a guess: the
file is opened only once the hub has measured the rate off the live stream, which
it retries once a second.
### `.shist` binary file format
Each signal gets one file: `<Directory>/<sourceId>/<signalName>.shist`.
@@ -178,13 +279,155 @@ reopened — head/count/time bounds are restored from the on-disk header.
`historyZoom` requests (see [StreamHub-API.md](StreamHub-API.md)) call
`HistoryWriter::ReadRange` which performs binary search over the circular file
using `pread` to locate the `[t0, t1]` window, then copies matching pairs.
If the result exceeds the requested `n`, LTTB decimation is applied (same
`LTTBDecimate` as in-memory zoom).
If the result exceeds the requested `n`, decimation is applied (same decimator
as in-memory zoom: `LTTBDecimate` in the C++ hub, `minMaxDecimate` in the Go
one).
Both the web SPA and ImGui client issue `historyZoom` in parallel with regular
`zoom` and merge the results: history covers the older part of the visible
window, the in-memory ring covers the recent part.
A range wider than the read budget is thinned across its whole width with a
stride, not truncated at the front: answering a 10 s query with its first
few milliseconds reads as an empty plot to a client and sends it back to its
own coarse copy of the data.
### In-memory buffer policy (Go hub)
Each temporal signal gets one ring holding a fixed **budget** of `(t, v)` pairs:
10 M points, 160 MB, settable with `-ring-mpts`. Scalar signals keep a flat
100 000-packet ring, where a megasample budget would be waste. Rings start at
250 k points and are grown to the budget on demand, so a source that is
configured but never sends costs nothing.
Like the disk archive, the budget buys **resolution, not span**. Once a second
`retuneRings` compares the measured source rate against the window being
displayed and picks each ring's min/max `bucket`:
| condition | bucket | effect |
|---|---|---|
| `Sps × window ≤ budget` | 1 | stored verbatim; the ring reaches further back than the window, which is free zoom headroom |
| `Sps × window > budget` | `⌈2 × Sps × window × 1.25 ÷ capacity⌉` | `bucket` samples collapse to their two extremes, so the whole window fits |
A bucket costs two points (its minimum and its maximum), hence the factor 2 —
and why a bucket of 2 covers no more ground than a bucket of 1.
The window is the **trigger's** while a trigger is armed: its pre-window has to
already be in the ring when the trigger fires, or the capture has nothing to
back-fill from. Otherwise it is the widest window any connected client has
reported with the `setWindow` command, defaulting to 10 s for clients that never
send one. Sizing for the live window matters as much as for a capture: a fixed
sample-count ring covers ~6 s at 1 MSps, so a zoom on a 60 s timescale used to
come back with only its tail.
Retuning is hysteretic — a bucket is held while it covers the window without
covering more than twice it. Sharing one threshold for up and down makes a rate
jittering across a bucket boundary halve and double the stored resolution every
second.
Live pushes, the disk archive and the trigger comparator all see every sample:
`ingest` hands the raw batch to each, and only the ring's own copy is reduced.
### Trigger captures (Go hub)
A trigger capture is delivered as a decimated snapshot (20 000 points), so a
zoom into it has to come from full-resolution storage. The rings are tuned to
~1.25× the trigger window, so they roll past a captured window shortly after the
capture — and the trigger rearms and starts refilling them immediately.
**In-memory double buffer.** The rings are the write half; `captureHold`
(`capturehold.go`) is the read half. As `buildTriggerCapture` lifts each signal's
window out of its ring it publishes the *undecimated* slice into the hold, and
`zoomSlice` answers from the hold rather than the ring for any range the held
window fully contains. The swap happens only once the next capture is complete —
which is also the moment the client stops displaying the previous one — so the
shot being explored is never overwritten by the acquisition running behind it. A
capture that came back empty does not swap, so it cannot blank the window on
screen.
**Waiting for the buffer.** An armed trigger ignores edges until its buffers
reach back far enough for a capture taken now to come back whole (`fillLocked`
in `trigger.go`, fed by `refreshTriggerFill` from the trigger signal's own ring
— once per tick, and again on every trigger command so that an `arm` cannot
fire on a stale measurement). Firing earlier can only produce a capture whose
front was never recorded, which is what made the first shot after a widened
window come back short.
What must hold is that the buffer spans the whole window *at harvest time* — its
newest sample is then `trigTime + post`, so anything less has lost the front of
the capture. It keeps filling while the post-window is collected, so the
shortfall it may start with is what it will make up in that time, measured
rather than assumed:
```
need = windowSec − growth × postSec (floored at the pre-trigger window)
```
`growth` is the ring's span growth in seconds per second, sampled over at least
`bufGrowthIntervalSec` and smoothed. The three regimes fall out of the one
formula:
| ring | growth | needs |
|---|---|---|
| still filling | 1 | the pre-trigger window — everything after the trigger is yet to be recorded anyway |
| full, re-bucketing for a longer window | 0…1 | in between: it drops dense old samples to take sparse new ones, so it fills slower than real time and the front of the capture recedes while the post-window elapses |
| full, settled | 0 | the whole window — which a ring tuned for that window already exceeds, so nothing actually waits |
Measured at 1 MSps, widening 10 s → 30 s with 50 % pre: growth settles at ~0.65,
so `need` converges on ~20.4 s of the 30 s and the trigger fires ~12 s after
arming with a capture that is 100 % complete. Requiring the whole window instead
would have waited 26 s for the same result.
The gate measures the trigger signal's ring, not the narrowest of all of them: a
signal that never reaches back that far would otherwise stop the trigger from
ever firing. It is disabled outright when there is no ring to measure or nothing
is needed, and `forceTrigger` overrides it. While it holds off, `triggerState`
carries `bufferFill`/`bufferNeedSec` and is re-broadcast as the fraction climbs,
so the UI shows `ARMED 42%` rather than a trigger that looks stuck.
**Back-filling a short capture.** A ring only spans the window once it has
rolled over completely at its current min/max bucket, which takes as long as the
window itself; widen the window, or arm right after setting it, and the first
captures start late and the client draws a blank front half.
`backfillCaptureHead` (`trigger.go`) therefore prepends whatever of
`[t0, ring's first sample)` the archive still holds, budgeting the read by the
share of the window being filled and trimming the overlap so the frame's
timestamps stay ascending. It needs history enabled; without it the capture is
simply short, and the hub logs by how much. The hold declines any range its own
samples do not actually cover, so a stretch neither source could supply falls
through to the archive instead of being redrawn as the same hole on every zoom
and every *fit*.
The hold declines ranges reaching outside its window: those are live zooms, and
only the rings still track the stream. Inside the window it needs no
trigger-state gating, because retuning never rewrites stored samples — a ring
that still covers the range holds the very same points. It is cleared when
`updateConfig` rebuilds the rings, since a restarted producer can replay the same
timestamps.
Budget: the hold costs one window per signal on top of the ring budget, up to a
further ~0.8 × `-ring-mpts`. Nothing is held until the first capture fires.
**On disk.** The archive covers what the hold cannot: ranges wider than the
capture window, and sessions where the hub restarted. It is circular and sized
from that same window, so it too wraps over a captured shot within a window of
delivering it. When a capture is delivered, the hub therefore copies
`[trigTime − pre, trigTime + post]` out of each `.shist` into a
`<signalName>.cap` file, laid out as a full non-wrapping `.shist`
(`capacity == count`, `head == 0`) so the same `readRange` reads it. A
`historyZoom` whose range the capture file fully contains is answered from it;
anything wider is answered from the archive. The copy is replaced by the next
trigger and by nothing else — rearming keeps it, because the client is still
showing that capture.
Budget: a capture costs one window's worth of disk per signal on top of
`-history-max-mpts`.
Protecting the window in place instead — pinning the region and refusing to
wrap onto it — does not work, and was tried: a capture held for longer than the
archive covers stops the archive dead, and the resulting hole lands exactly
where the *next* capture's pre-trigger window belongs.
## 7. Build & test
```bash