fixed and improved ui

Client/udpstreamer/BUFFER_AND_TRIGGER.md

@@ -0,0 +1,425 @@
+# Buffer time-window & trigger logic in `Client/udpstreamer`
+
+How the UDP Scope client acquires, buffers, times and triggers waveforms.
+The pipeline has two halves that must be read together:
+
+- the **Go hub** (`Common/Client/go/wshub/`) — owns the UDP sockets, the
+  full-resolution sample storage, the disk history, and the trigger FSM;
+- the **browser SPA** (`static/app.js`) — owns the display buffers, the rolling
+  window, and the trigger capture rendering.
+
+The same SPA is also served by `Client/webui` and talks to the C++ StreamHub,
+which mirrors the Go hub's behaviour (same trigger FSM states, same binary
+frames). Everything below describes the Go-hub path; the wire contracts are
+identical on both.
+
+---
+
+## 1. End-to-end data flow
+
+```
+MARTe2 RT app ──UDP/UDPS──▶ sources.go: runSession()
+   │  CONFIG + DATA packets, 17-byte header, HRT timestamp per frame
+   ▼
+udpsprotocol.ParseData()  →  []DataSample{HRTTimestamp, WallTime, Values}
+   ▼
+Hub.Run()  dataCh → pending[sourceID]   (drained every 30 Hz tick)
+   ▼
+buildBinaryDataMessageForSource()
+   ├─ rebuild per-sample timestamps from TimeMode / calibration / monotonic snap
+   ├─ h.ingest(key, n, t, v)   ← FULL rate: ring.write + hist.write + trigger.feed
+   └─ minMaxDecimate(…, maxPushPoints=50)  →  WS binary v1 frame to clients
+   ▼
+browser: onBinaryData() → pushBuffer() into per-signal circular buffers
+   ▼
+renderDirtyPlots() (rAF loop) → buildUPlotData() → uPlot
+```
+
+The 30 Hz push is the **only** live path to the browser and it is decimated to
+≤50 points/signal/tick. Everything that needs full resolution — zoom, trigger
+captures, disk history — is fed independently through `ingest()` and never goes
+over the wire until asked.
+
+---
+
+## 2. Hub-side buffers: `sigRing` (`wshub/ringbuf.go`)
+
+One ring per `"sourceId:signalName"` key. A fixed-capacity circular buffer of
+Float64 `(t, v)` pairs with a `sync.RWMutex` (writes from `Hub.Run()`, reads
+from HTTP/WS handler goroutines).
+
+### 2.1 Min/max bucketing
+
+`bucket` is how many source samples collapse into **one min/max pair** on the
+way in:
+
+- `bucket == 1` — the stream is stored verbatim;
+- `bucket > 1` — each group contributes its minimum and its maximum, emitted in
+  time order (`flushBucketLocked`), so the stored timestamps stay
+  non-decreasing (reads binary-search `rb.t`).
+
+Bucketing is what lets an arbitrarily long window fit a fixed per-signal memory
+budget at a megasample rate. Samples already stored keep the resolution they
+were written at; the ring converges on a new bucket as it rolls
+(`setBucket`).
+
+### 2.2 Source-rate measurement
+
+`sigRing` keeps its own source-sample accounting (`srcCount`, `srcT0`, `srcT1`,
+reset every `srcRateWindowSec = 10 s`), because once `bucket > 1` neither `size`
+nor the stored timespan measures the real incoming rate. `sourceRate()` is used
+by the tuning sweep and by the history writer.
+
+### 2.3 Ring tuning (`retuneRings`, every 1 s)
+
+`activeWindowSec()` decides how far back the rings must reach:
+
+1. an **armed trigger** owns the window: `cfg.windowSec + captureLagSec`
+   (`captureLagSec = captureMarginSec + 1/30 ≈ 0.183 s` — the capture is read
+   out a post-window + margin + one push tick after the trigger, so the rings
+   must hold that much extra or the front of the capture has already rolled);
+2. otherwise the **widest window any connected client is displaying**
+   (`wsClient.displayWindowSec`, set by the SPA's `setWindow` command), with a
+   `defaultLiveWindowSec = 10 s` fallback while nobody has said;
+
+Then per ring, with `budget = ringBudget()` (default `defaultRingPts = 10 M`,
+floor `ringCapInitial = 250 k`):
+
+- grow to the budget first (`grow()` preserves all samples, never shrinks);
+- compute the needed bucket with `ringBucketFor(rate, window, capacity)`
+  (`ceil(2·rate·window·ringHeadroom / capacity)`, `ringHeadroom = 1.25`);
+- apply it with **hysteresis**: keep the current bucket while its coverage is
+  between `need` and `2·need`, so a rate jittering across the boundary does not
+  flip the resolution every second.
+
+The history archive is re-sized from the same window (`hist.setWindow`) so a
+zoom or capture that outlives the rings can fall back to it.
+
+### 2.4 Reading: `slice(t0, t1)`
+
+Binary search for `t0` then `t1` over the circular layout, returning copies of
+the pairs in `[t0, t1]`. Safe to use without holding the lock.
+
+---
+
+## 3. Disk history (`wshub/history.go`)
+
+Optional (`EnableHistory`, `CloseHistory`), enabled by the hub configuration.
+Every sample goes to disk through `ingest → hist.write` at full rate, in files
+sized for the *current* window (not a retention period). It exists to back
+three things the rings cannot:
+
+- **zoom past the window**: `readRange(key, t0, t1, maxOut)`;
+- **captures the rings have rolled past**: `captureRange(trigTime−pre, trigTime+post)`
+  lifts each capture into a file of its own so nothing overwrites it before the
+  next trigger;
+- **short captures**: `backfillCaptureHead` prepends the front of the window the
+  ring no longer holds (the ring only *becomes* as long as the window after a
+  re-tune; the archive was written straight through).
+
+---
+
+## 4. Live push to the browser
+
+`Hub.Run()` drains `pending[sourceID]` on a 30 Hz ticker. Even with no client
+connected the frame is built: that is what keeps feeding rings, history and the
+trigger, and keeps push cursors advancing so a late client does not get a
+backlog burst.
+
+`buildBinaryDataMessageForSource` reconstructs per-sample timestamps per signal
+`TimeMode`:
+
+| Mode | Timestamp reconstruction |
+|---|---|
+| `FirstSample` / `LastSample` | scalar TimeSignal value × `timerToSec` (µs→s or ns→s for u64), calibrated once against `WallTime`; samples spaced by `1/SamplingRate` |
+| `FullArray` | per-element TimeSignal array, calibrated once against `WallTime` |
+| scalar (`n == 1`) | `WallTime` of the UDP arrival |
+| `PacketTime` (default, n>1) | inter-packet wall-clock gaps divided by n (single-packet ticks use the gap from the previous tick) |
+
+**Monotonic snapping** (optional, `setMonotonic` command / "Sync TS" checkbox):
+when enabled, the inter-frame anchor gap is smoothed with an EMA
+(`monotonicEMAAlpha = 0.01`, initialised from the nominal `n·dt`) and small
+deviations (< `monotonicTolerance = 5 ms`) are snapped to the smoothed gap,
+removing the software-dispatch jitter overlaps/gaps described in the StreamHub
+docs while tracking the true hardware rate (no accumulated drift).
+
+The live frame is a **binary v1** WS message:
+
+```
+[u8 1][u8 srcIdLen][srcId][u32 nSigs]
+{[u16 keyLen][key][u32 N][f64 t×N][f64 v×N]}
+```
+
+with each signal min/max-decimated to `maxPushPoints = 50` (`minMaxDecimate`:
+the range is split into `threshold/2` buckets, each contributing its min and max
+in time order — a scope-style envelope that keeps glitches on screen).
+
+---
+
+## 5. Browser-side buffers (`static/app.js`)
+
+### 5.1 Capacity & growth
+
+- `MAX_CAP = 2 000 000` — hard ceiling per buffer (~32 MB/signal at Float64 t+v);
+- `DEFAULT_CAP = 100 000` — starting size for scalars;
+- `TEMPORAL_CAP = 500 000` — starting size for array signals (the hub pushes
+  ≤50 pts/signal/tick, so this already covers ~5 min);
+- `growBufferForWindow(buf, windowSec)` — **sizes from the buffer's own span**,
+  not the signal's sampling rate: the incoming rate here is the hub-decimated
+  ~1.5 kpts/s regardless of the source rate, so rate-based sizing overshot by
+  three orders of magnitude. Grows only when the buffer is full, to
+  `windowSec × 1.5` headroom, capped at `MAX_CAP`.
+- `growBuffer` copies all existing samples into a larger array (preserving
+  circular order).
+
+### 5.2 The window
+
+`windowSec` (default 5 s, options 1 s … 10 min) is the rolling viewport.
+Changing it:
+
+1. updates `windowSec`;
+2. `sendWindow()` → WS `setWindow` → hub `displayWindowSec` → ring re-tune;
+3. grows every local buffer via `growBufferForWindow`;
+4. evicts the decimation cache (a different window invalidates all cached
+   renderings).
+
+The rolling "now" anchor is **data-driven, not wall-clock**:
+`computePlotNow(p)` takes the newest timestamp of each contributing source and
+uses the min-of-max over sources that are still active (a source lagging the
+fastest by more than `windowSec` is treated as stale and excluded). This keeps
+the window tracking real data regardless of clock skew between hub and browser.
+
+### 5.3 Slicing & rendering
+
+- `getBufferSliceRange(buf, t0, t1)` — binary search on the circular layout,
+  O(log n + window size);
+- `getBufferSliceRangeWithBrackets` — same plus one point on each side so lines
+  still cross a nearly-empty zoom window;
+- `supplementWithBrackets` — same bracketing for sparse server-fetched zoom data.
+
+`buildLiveData(p)`:
+
+1. slices every trace in `[t0, t1]`;
+2. picks the **master** signal: highest `SamplingRate`, then most points;
+3. decimates the master to ~2× plot width (`DECIM_MIN = 200` floor) via a
+   background worker (`decimateAsync`, stale-while-revalidate cache keyed per
+   plot/range/data-generation);
+4. resamples every other trace onto the master grid with `resampleLinear`;
+5. normalises Y (`applyVScaleNorm`: calibration `v·scale+offset`, then
+   `(y − offset)/div`).
+
+### 5.4 Zoom
+
+A zoom pins `p.xRange` and asks the hub for hi-res data over the exact range
+(WS `zoom` request or HTTP `/api/zoom`). The hub answers from the full-res
+rings — or from the **held copy of the last trigger capture** (`captureHold`
+double buffer) while that window is still relevant — decimated to the requested
+point budget. The browser prefers the fetched data when it exists, falls back
+to its own circular buffers otherwise, and always brackets with local points.
+
+---
+
+## 6. Hub-side trigger FSM (`wshub/trigger.go`)
+
+### 6.1 States and configuration
+
+```
+idle ──arm──▶ armed ──edge──▶ collecting ──window elapsed──▶ triggered
+ ▲  ▲                         (pre/post latched)                │
+ │  └───────────── rearm (normal mode, after holdoff) ◀─────────┘
+ └────────────── disarm / single mode stays triggered
+```
+
+Configuration (`trigConfig`, client-settable via WS `setTrigger`):
+
+| field | meaning | clamp |
+|---|---|---|
+| `signalKey` | `"src:sig"` or `"src:sig[i]"` | — |
+| `edge` | `rising` / `falling` / `both` | — |
+| `threshold` | **raw** units (SPA converts calibrated → raw) | — |
+| `windowSec` | capture window | `[1e-4, 600]` |
+| `prePercent` | pre-trigger share | `[0, 100]` |
+| `mode` | `normal` (auto-rearm) / `single` | — |
+| `holdoffSec` | re-arm delay after a capture, double-trigger guard | `[0, 60]` |
+
+### 6.2 Edge detection (`feed`)
+
+Called from `ingest` with every full-resolution batch for the trigger signal.
+Level tracking (`prevValue`/`prevValid`) compares consecutive samples against
+the threshold; `[i]`-suffixed keys stride the flattened batch by `nElem` to
+watch one column. On a qualifying edge in `armed` state: `latchWindowLocked`
+freezes `trigTime` and the pre/post split (so later config edits cannot move a
+capture's axis).
+
+### 6.3 Buffer-fill gate
+
+Before accepting an edge, the FSM checks that the trigger signal's ring reaches
+back far enough that the capture will come back whole (`fillLocked`):
+
+```
+need = windowSec − growth × postSec, floored at the pre-window
+```
+
+`growth` is the measured span-growth rate of the ring (`setBuffered`, refreshed
+by `refreshTriggerFill` from the tick and from trigger commands). A still-filling
+ring grows 1 s of span per second, so the gate reduces to the pre-window; a
+full ring at a long window needs the whole window. While holding off, the level
+is still tracked so the first edge after the gate opens is measured against the
+right predecessor. The SPA shows the hold-off as an armed trigger with a
+`bufferFill %` badge.
+
+### 6.4 Window timing and the pending edge
+
+`dueCapture` waits for the window on the **sample clock**, not the wall clock:
+`lastT ≥ trigTime + post + captureMarginSec(0.15)`. This avoids cutting a
+capture short when the stream's timestamps lag real time. Three ways it fires:
+
+1. the samples themselves covered the window;
+2. wall-clock fallback when no sample was ever seen (Force from idle);
+3. `captureStallSec = 2 s` of stream silence — deliver what was collected
+   rather than leaving the client stuck in "collecting".
+
+While a capture is in flight the comparator keeps running. The **first**
+qualifying edge at/after `notBefore = trigTime + max(post, holdoffSec)` is
+remembered (`pendingT`/`pendingValid`) and fired immediately on the automatic
+`rearm()`. Without this the trigger was deaf through the whole post-window +
+holdoff, which rounded sparse pulse trains up to whole periods (a 1 Hz train at
+a 1 s window was caught at 0.5 Hz).
+
+### 6.5 Holdoff and rearm
+
+`markTriggered` moves `collecting → triggered` and, in `normal` mode (not
+stopped), schedules `rearmAt = now + cfg.holdoffSec`. `dueRearm` consumes it;
+`rearm()` re-arms immediately on a pending edge or returns to `armed`. The
+holdoff is measured from the trigger point, overlapping the post-window rather
+than adding to it.
+
+`Force()` fires immediately at the most recent sample time (wall clock if no
+sample yet) — the "Force" button.
+
+### 6.6 Capture assembly (`buildTriggerCapture`)
+
+On a due capture the hub builds the **binary v2** frame:
+
+```
+[u8 2][f64 trigTime][f64 preSec][f64 postSec][u32 nSig]
+{[u16 keyLen][fullKey][u32 N][f64 t×N][f64 v×N]}
+```
+
+For every ring:
+
+1. `slice(trigTime−pre, trigTime+post)`;
+2. `backfillCaptureHead` from disk history for the front the ring lost;
+3. if it is still short by more than `shortCaptureTol = 1 %` of the window,
+   log it explicitly (nothing can recover data the ring never held);
+4. keep the **full-resolution** slice in the `captureHold` double buffer
+   (so a zoom into the capture can be answered after the rings roll past);
+5. min/max-decimate to `trigCapturePts = 20 000` per signal for the wire —
+   a 60 s window at 1 MSps is ~960 MB raw per signal and would be dropped by
+   the send path anyway.
+
+The double buffer is published (`capture.publish`) only once the frame is known
+good, so a shot that yielded nothing leaves the previous capture on screen.
+Dropped frames (client send-queue full) are logged.
+
+`triggerTick` (every push tick) drives the whole FSM: re-tune rings → open
+pending history files → refresh the fill measurement → due capture (send + mark
+triggered + `hist.captureRange`) or due rearm → broadcast state only when it
+changed (`stateUnsent`).
+
+### 6.7 WS commands
+
+| message | effect |
+|---|---|
+| `setTrigger {signal, edge, threshold, windowSec, prePercent, mode, holdoffSec}` | replace config |
+| `arm` / `rearm` | explicit arm (discards pending edge) |
+| `disarm` | → idle |
+| `trigStop {stopped}` | pause/resume auto-rearm |
+| `forceTrigger` | fire now |
+
+Every command also refreshes the buffer-fill measurement synchronously — at
+1 MSps the ring crosses the fill threshold many times inside one 33 ms tick, so
+waiting for the next tick would fire on a stale measurement.
+
+---
+
+## 7. Browser-side trigger (`static/app.js`)
+
+### 7.1 State handling
+
+`onTriggerState(msg)` tracks the FSM broadcast:
+
+- **armed** — shows `bufferFill %` while the hub is holding off on the fill
+  gate, so a trigger that is not yet fireable does not look broken;
+- **collecting** — clears the previous snapshot, latches `trigTime` (and
+  `preSec`/`postSec` if the hub sent them), and lets live data sweep into the
+  trigger axis (see 7.3);
+- **triggered / idle** — bookkeeping for the Rearm/Stop buttons.
+
+### 7.2 Capture handling
+
+`onTriggerCapture` parses the v2 frame into `trig.snapshot[key] = {t, v}` plus
+the latched `_preS`/`_postS`. It is **ignored when the client did not enable
+the trigger** (`trig.enabled`), because the hub keeps an armed trigger across
+client sessions and applying a foreign capture would clobber this client's zoom
+and scales. On receipt: the horizontal zoom is dropped so the whole capture is
+visible, but **vertical scales (V/div, offset) persist** — they are user
+settings and must survive from shot to shot.
+
+### 7.3 Rendering modes (`buildUPlotData`)
+
+| state | renderer | source |
+|---|---|---|
+| collecting, no snapshot yet | `buildTrigFillData` | live buffers, drawn on the *final* trigger axis (relative seconds, `[-pre, +post]`) so the trace sweeps in from the left |
+| armed, not fired | freeze last frame | — |
+| snapshot present | `buildTrigData` | the capture (or a hi-res zoom reply that covers ≥98 % of the view, else the snapshot) |
+| otherwise | `buildLiveData` | rolling window |
+
+`buildTrigData` converts to trigger-relative time (`t − trigT`), picks the
+master by rate/count, decimates (cached per range+source-tag), resamples the
+other traces, and normalises Y.
+
+### 7.4 Threshold in calibrated units
+
+The trigger threshold is held in **calibrated units** (what the user sees on
+the Y axis). `sendTrigConfig()` inverts it through the signal's calibration
+before sending: `raw = (calibrated − offset)/scale`, so the hub's raw
+comparator fires exactly when `GAIN·signal + OFFSET` crosses the threshold.
+The threshold line (`drawTriggerMarker`) maps the same calibrated threshold
+through the signal's vscale: `y_norm = (threshold − offset)/div`.
+
+---
+
+## 8. Key constants
+
+| constant | value | file |
+|---|---|---|
+| push rate | 30 Hz | `hub.go` |
+| `maxPushPoints` (live) | 50 pts/signal/tick | `hub.go` |
+| `trigCapturePts` (capture) | 20 000 pts/signal | `trigger.go` |
+| `captureMarginSec` | 0.15 s | `trigger.go` |
+| `captureStallSec` | 2.0 s | `trigger.go` |
+| `autoRearmDelaySec` (default holdoff) | 0.2 s | `trigger.go` |
+| `maxTriggerWindowSec` | 600 s | `trigger.go` |
+| `ringBudget` default | 10 000 000 pts/signal | `hub.go` |
+| `ringCapInitial` | 250 000 | `hub.go` |
+| `ringCapScalar` | 100 000 | `hub.go` |
+| `ringHeadroom` | 1.25 | `ringbuf.go` |
+| `defaultLiveWindowSec` | 10 s | `ringbuf.go` |
+| `captureLagSec` | 0.15 + 1/30 ≈ 0.183 s | `ringbuf.go` |
+| `monotonicTolerance` | 5 ms | `hub.go` |
+| `monotonicEMAAlpha` | 0.01 | `hub.go` |
+| `MAX_CAP` (browser) | 2 000 000 pts/signal | `app.js` |
+| `DEFAULT_CAP` / `TEMPORAL_CAP` | 100 000 / 500 000 | `app.js` |
+| `DECIM_MIN` | 200 | `app.js` |
+
+---
+
+## 9. The C++ StreamHub mirror
+
+The Go hub and the C++ StreamHub implement the same WS contracts and must stay
+in sync (`AGENTS.md`): same `triggerState` FSM strings, same v2 capture frame,
+same command set (`setTrigger` including `holdoffSec`, `arm`, `disarm`,
+`trigStop`, `forceTrigger`), same `trigCapturePts`/`kTrigCapturePts` cap, and
+the same ring/history windowing intent (`Source/Applications/StreamHub/`). A
+protocol change on one side must be mirrored on the other.