4 Commits
Author SHA1 Message Date
Martino FerrariandClaude Opus 4.6 5562877c99 fix(udps): publish the producer's HRT frequency so timestamps survive the hop
DATA packets timestamp with the raw value of the producer's high-resolution
counter, and the wire never said how fast that counter runs. The hub divided by
its own timer's frequency instead, which is only the same number while producer
and hub share a machine — on x86 it is the TSC frequency and differs from model
to model. Off-box, every accumulated batch was therefore laid out over the wrong
span of time: the samples in it drift away from where they belong and start
colliding with the next packet's, which is the "same" symptom as a stale time
base even though nothing is out of order.

CONFIG now carries the rate as a trailing uint64, alongside the publish-mode
byte and read the same tolerant way: absent or zero means the producer did not
say, and the hub falls back to its own timer as before. Anything below 1 kHz is
not a high-resolution timer and is refused, so a mis-parsed payload cannot
stretch a millisecond batch across seconds.

The Accumulate DATA payload is unchanged, so this costs nothing per packet and
the period *within* a batch is still estimated from the gap between packets.

The Go, C and browser parsers already ignore trailer bytes they do not know,
so they read the new CONFIG unchanged; none of them uses the HRT timestamp.

Also corrects the Accumulate DATA layout in all three protocol documents: they
described it as one snapshot per array signal, where it has always been one per
accumulated cycle.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-09-02 02:49:50 +02:00
Martino FerrariandClaude Opus 4.6 092fd3c775 fix(streamhub): stop the trigger going deaf between captures
TriggerEngine::CheckSample returned early in every state but ARMED, so an
edge arriving while a capture was being collected or handed out was
dropped, and the automatic rearm then waited for a FRESH edge. The engine
was therefore blind from its own trigger point until the capture had been
harvested — a post-window — and for the holdoff on top of that.

On a sparse pulse train that rounds the capture spacing up to a whole
pulse period: at the default 1 s window the blind stretch is 1 s, so a
1 Hz train was caught at 0.5 Hz and a wider window lost whole multiples.

The comparator now keeps running through COLLECTING and TRIGGERED and
remembers the first edge at or past trigTime + max(postSec, holdoffSec).
The holdoff guards against re-triggering on the ringing of the same
event and is measured from the trigger point, so it overlaps the
post-window rather than adding to it. Rearm() fires on the remembered
edge; it also keeps the tracked level, so the first sample after it has
a real predecessor instead of being spent seeding one.

Arm() stays the operator's arm and discards the held edge — they asked
for the next event, not one already been and gone — and SetConfig() and
Disarm() drop it too, since it was never judged against the new window.

This is the same defect and the same remedy already validated in the Go
hub (wshub/trigger.go, trigger_sporadic_test.go); the C++ hub had been
left with the original semantics.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-09-02 01:45:13 +02:00
Martino FerrariandClaude Opus 4.6 fbae7d712c fix(udps): stop packets being dated from an earlier time base
Reported as samples sporadically carrying a previous packet's timestamp:
holes on one side of the stream and collisions on the other, in both the
Go and the MARTe2 receiver. That it appeared in both is what located it
-- the shared cause is upstream of either client. Four independent
defects, all of which end in a packet's values being placed at a time
that is not theirs.

Reassembly slot exhaustion (the "Reassembly slots full; evicting oldest"
flood). Chunk size was learnt only from fragment 0, so an out-of-order
burst destroyed a packet whose bytes had all arrived and left the slot
occupied until the 2 s GC. Slots were keyed on the counter alone, but
DATA and CONFIG number independently, so equal counters merged the two
streams. The 32-byte received-mask covered 256 of the 512 fragments the
client accepts, so a duplicate above 255 was counted as new and the
packet was delivered with a hole of stale bytes in it. And one datagram
was read per Execute(), which cannot drain a fast producer. Fixed with a
pendingTail deferral, (counter, type) keying, a 64-byte mask, a
256-datagram drain, counter-age slot reclamation, and a 1 Hz aggregated
warning in place of the per-eviction flood.

UDPStreamer dropping whole Accumulate batches. EventSem::ResetWait is
Reset-then-Wait, so a Post() landing while the sender thread was inside
ServiceClients()/SendData() was destroyed by the next Reset. The batch
was then skipped with dataReady false, readyFill was never cleared, and
the following flush overwrote it: an entire run of RT cycles never
reached the wire. The record of pending work now lives in the buffers
rather than in the semaphore edge, which also removes up to
UDPS_DATA_WAIT_MS of latency; genuine backpressure overwrites are
counted and reported. Against the unfixed code the new test sees
2999/3000 batches never consumed.

Period inflation after loss. Accumulated scalars carry no SamplingRate,
so the receiver derives dt from the sender-clock gap -- but dividing it
by the previous packet's sample count is only right while nothing is
lost. One loss doubles the reported period, which spreads a batch a full
batch past its own end and into the range the next packet claims. That
is the hole and the collision, exactly. Inferring the cycle count from
the estimate's own period is not a way out: it has a stable fixed point
wherever gap/dt is an integer, so a real rate change locks it at the old
one for good (AccumDtGTest.FollowsSustainedRateChange).

The packet counter removes the ambiguity, so all three receivers now
order on it: a DATA packet that does not advance the counter is dropped
rather than delivered, because its values are older than data already
handed over. Ordering is on the signed difference so it survives the
uint32 wrap, and the sequence resets on reconnect, where the producer's
counter restarts independently of ours. The loss count that falls out of
the same delta feeds the period estimate as cycles = prevN * (1 + lost),
which reduces exactly to gap/prevN when nothing is lost and therefore
still tracks a genuine rate change. UDPSClient::AcceptDataCounter (C++),
udpsprotocol.SequenceGate (Go), decode_data (C).

The C client's existing gap counter was wrap-unsafe and let a stale
packet rewind last_counter, which made every subsequent gap wrong; it
uses the same code now. Docs/Protocol.md gains an Ordering DATA section
stating the requirement for any receiver, including ones outside this
repository.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-09-02 01:18:49 +02:00
Martino FerrariandClaude Opus 4.6 f334995865 feat(webui): sporadic-trigger capture, CSV export and UI rework
Brings the Go hub and web SPA work developed on feature/udpscope onto
main, without the udpscope client itself.

The trigger engine could not capture a sporadic event: it armed on the
live tail only, so a burst shorter than one push window was already past
by the time the FSM looked for it. It now searches the ring history for
the crossing, which also makes a capture reproducible from the same data
rather than dependent on push timing (wshub/trigger.go, ringbuf.go,
history.go).

Adds CSV/JSON export of the visible window (wshub/export.go) and reworks
the SPA: per-signal axis controls, a readable trigger panel, and a fix
for the flicker caused by repainting on every push instead of on a frame
tick (static/app.js, index.html, style.css).

BUFFER_AND_TRIGGER.md documents the ring/decimation/trigger interaction,
which is otherwise only inferable from the three files that implement it.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-09-02 01:18:46 +02:00
50 changed files with 4568 additions and 377 deletions
+11 -4
View File
@@ -117,9 +117,16 @@ Sent when the signal set changes or a client connects:
```
[uint32 numSigs]
numSigs × UDPSSignalDescriptor (136 bytes each, packed)
[uint8 publishMode] 0=Strict/Decimate, 1=Accumulate
[uint8 publishMode] 0=Strict, 1=Accumulate, 2=Decimate
[uint64 hrtFrequency] producer's HRT ticks per second; 0 = unknown
```
Everything after the descriptors is an optional trailer: a receiver accepts a
payload that stops early and ignores bytes it does not know. `hrtFrequency` is
what lets a receiver on another host turn the raw counter in DATA into seconds
— without it the only option is the receiver's own timer, which agrees with the
producer only when the two share a machine.
### DATA Payload (Strict / Decimate modes)
```
@@ -130,9 +137,9 @@ per-signal data in CONFIG order (quantised or raw, no inter-signal padding)
### DATA Payload (Accumulate mode)
```
[uint64 HRT timestamp]
[uint32 numSamples]
for each signal: if scalar → numSamples elements; else → NumElements once
[uint64 HRT timestamp of the first slot in the batch]
[uint32 numSamples] RT cycles accumulated into this packet
for each signal, in CONFIG order: numSamples × NumElements values
```
### Quantization / Dequantization
+425
View File
@@ -0,0 +1,425 @@
# Buffer time-window & trigger logic in `Client/udpstreamer`
How the UDP Scope client acquires, buffers, times and triggers waveforms.
The pipeline has two halves that must be read together:
- the **Go hub** (`Common/Client/go/wshub/`) — owns the UDP sockets, the
full-resolution sample storage, the disk history, and the trigger FSM;
- the **browser SPA** (`static/app.js`) — owns the display buffers, the rolling
window, and the trigger capture rendering.
The same SPA is also served by `Client/webui` and talks to the C++ StreamHub,
which mirrors the Go hub's behaviour (same trigger FSM states, same binary
frames). Everything below describes the Go-hub path; the wire contracts are
identical on both.
---
## 1. End-to-end data flow
```
MARTe2 RT app ──UDP/UDPS──▶ sources.go: runSession()
│ CONFIG + DATA packets, 17-byte header, HRT timestamp per frame
udpsprotocol.ParseData() → []DataSample{HRTTimestamp, WallTime, Values}
Hub.Run() dataCh → pending[sourceID] (drained every 30 Hz tick)
buildBinaryDataMessageForSource()
├─ rebuild per-sample timestamps from TimeMode / calibration / monotonic snap
├─ h.ingest(key, n, t, v) ← FULL rate: ring.write + hist.write + trigger.feed
└─ minMaxDecimate(…, maxPushPoints=50) → WS binary v1 frame to clients
browser: onBinaryData() → pushBuffer() into per-signal circular buffers
renderDirtyPlots() (rAF loop) → buildUPlotData() → uPlot
```
The 30 Hz push is the **only** live path to the browser and it is decimated to
≤50 points/signal/tick. Everything that needs full resolution — zoom, trigger
captures, disk history — is fed independently through `ingest()` and never goes
over the wire until asked.
---
## 2. Hub-side buffers: `sigRing` (`wshub/ringbuf.go`)
One ring per `"sourceId:signalName"` key. A fixed-capacity circular buffer of
Float64 `(t, v)` pairs with a `sync.RWMutex` (writes from `Hub.Run()`, reads
from HTTP/WS handler goroutines).
### 2.1 Min/max bucketing
`bucket` is how many source samples collapse into **one min/max pair** on the
way in:
- `bucket == 1` — the stream is stored verbatim;
- `bucket > 1` — each group contributes its minimum and its maximum, emitted in
time order (`flushBucketLocked`), so the stored timestamps stay
non-decreasing (reads binary-search `rb.t`).
Bucketing is what lets an arbitrarily long window fit a fixed per-signal memory
budget at a megasample rate. Samples already stored keep the resolution they
were written at; the ring converges on a new bucket as it rolls
(`setBucket`).
### 2.2 Source-rate measurement
`sigRing` keeps its own source-sample accounting (`srcCount`, `srcT0`, `srcT1`,
reset every `srcRateWindowSec = 10 s`), because once `bucket > 1` neither `size`
nor the stored timespan measures the real incoming rate. `sourceRate()` is used
by the tuning sweep and by the history writer.
### 2.3 Ring tuning (`retuneRings`, every 1 s)
`activeWindowSec()` decides how far back the rings must reach:
1. an **armed trigger** owns the window: `cfg.windowSec + captureLagSec`
(`captureLagSec = captureMarginSec + 1/30 ≈ 0.183 s` — the capture is read
out a post-window + margin + one push tick after the trigger, so the rings
must hold that much extra or the front of the capture has already rolled);
2. otherwise the **widest window any connected client is displaying**
(`wsClient.displayWindowSec`, set by the SPA's `setWindow` command), with a
`defaultLiveWindowSec = 10 s` fallback while nobody has said;
Then per ring, with `budget = ringBudget()` (default `defaultRingPts = 10 M`,
floor `ringCapInitial = 250 k`):
- grow to the budget first (`grow()` preserves all samples, never shrinks);
- compute the needed bucket with `ringBucketFor(rate, window, capacity)`
(`ceil(2·rate·window·ringHeadroom / capacity)`, `ringHeadroom = 1.25`);
- apply it with **hysteresis**: keep the current bucket while its coverage is
between `need` and `2·need`, so a rate jittering across the boundary does not
flip the resolution every second.
The history archive is re-sized from the same window (`hist.setWindow`) so a
zoom or capture that outlives the rings can fall back to it.
### 2.4 Reading: `slice(t0, t1)`
Binary search for `t0` then `t1` over the circular layout, returning copies of
the pairs in `[t0, t1]`. Safe to use without holding the lock.
---
## 3. Disk history (`wshub/history.go`)
Optional (`EnableHistory`, `CloseHistory`), enabled by the hub configuration.
Every sample goes to disk through `ingest → hist.write` at full rate, in files
sized for the *current* window (not a retention period). It exists to back
three things the rings cannot:
- **zoom past the window**: `readRange(key, t0, t1, maxOut)`;
- **captures the rings have rolled past**: `captureRange(trigTimepre, trigTime+post)`
lifts each capture into a file of its own so nothing overwrites it before the
next trigger;
- **short captures**: `backfillCaptureHead` prepends the front of the window the
ring no longer holds (the ring only *becomes* as long as the window after a
re-tune; the archive was written straight through).
---
## 4. Live push to the browser
`Hub.Run()` drains `pending[sourceID]` on a 30 Hz ticker. Even with no client
connected the frame is built: that is what keeps feeding rings, history and the
trigger, and keeps push cursors advancing so a late client does not get a
backlog burst.
`buildBinaryDataMessageForSource` reconstructs per-sample timestamps per signal
`TimeMode`:
| Mode | Timestamp reconstruction |
|---|---|
| `FirstSample` / `LastSample` | scalar TimeSignal value × `timerToSec` (µs→s or ns→s for u64), calibrated once against `WallTime`; samples spaced by `1/SamplingRate` |
| `FullArray` | per-element TimeSignal array, calibrated once against `WallTime` |
| scalar (`n == 1`) | `WallTime` of the UDP arrival |
| `PacketTime` (default, n>1) | inter-packet wall-clock gaps divided by n (single-packet ticks use the gap from the previous tick) |
**Monotonic snapping** (optional, `setMonotonic` command / "Sync TS" checkbox):
when enabled, the inter-frame anchor gap is smoothed with an EMA
(`monotonicEMAAlpha = 0.01`, initialised from the nominal `n·dt`) and small
deviations (< `monotonicTolerance = 5 ms`) are snapped to the smoothed gap,
removing the software-dispatch jitter overlaps/gaps described in the StreamHub
docs while tracking the true hardware rate (no accumulated drift).
The live frame is a **binary v1** WS message:
```
[u8 1][u8 srcIdLen][srcId][u32 nSigs]
{[u16 keyLen][key][u32 N][f64 t×N][f64 v×N]}
```
with each signal min/max-decimated to `maxPushPoints = 50` (`minMaxDecimate`:
the range is split into `threshold/2` buckets, each contributing its min and max
in time order — a scope-style envelope that keeps glitches on screen).
---
## 5. Browser-side buffers (`static/app.js`)
### 5.1 Capacity & growth
- `MAX_CAP = 2 000 000` — hard ceiling per buffer (~32 MB/signal at Float64 t+v);
- `DEFAULT_CAP = 100 000` — starting size for scalars;
- `TEMPORAL_CAP = 500 000` — starting size for array signals (the hub pushes
≤50 pts/signal/tick, so this already covers ~5 min);
- `growBufferForWindow(buf, windowSec)`**sizes from the buffer's own span**,
not the signal's sampling rate: the incoming rate here is the hub-decimated
~1.5 kpts/s regardless of the source rate, so rate-based sizing overshot by
three orders of magnitude. Grows only when the buffer is full, to
`windowSec × 1.5` headroom, capped at `MAX_CAP`.
- `growBuffer` copies all existing samples into a larger array (preserving
circular order).
### 5.2 The window
`windowSec` (default 5 s, options 1 s … 10 min) is the rolling viewport.
Changing it:
1. updates `windowSec`;
2. `sendWindow()` → WS `setWindow` → hub `displayWindowSec` → ring re-tune;
3. grows every local buffer via `growBufferForWindow`;
4. evicts the decimation cache (a different window invalidates all cached
renderings).
The rolling "now" anchor is **data-driven, not wall-clock**:
`computePlotNow(p)` takes the newest timestamp of each contributing source and
uses the min-of-max over sources that are still active (a source lagging the
fastest by more than `windowSec` is treated as stale and excluded). This keeps
the window tracking real data regardless of clock skew between hub and browser.
### 5.3 Slicing & rendering
- `getBufferSliceRange(buf, t0, t1)` — binary search on the circular layout,
O(log n + window size);
- `getBufferSliceRangeWithBrackets` — same plus one point on each side so lines
still cross a nearly-empty zoom window;
- `supplementWithBrackets` — same bracketing for sparse server-fetched zoom data.
`buildLiveData(p)`:
1. slices every trace in `[t0, t1]`;
2. picks the **master** signal: highest `SamplingRate`, then most points;
3. decimates the master to ~2× plot width (`DECIM_MIN = 200` floor) via a
background worker (`decimateAsync`, stale-while-revalidate cache keyed per
plot/range/data-generation);
4. resamples every other trace onto the master grid with `resampleLinear`;
5. normalises Y (`applyVScaleNorm`: calibration `v·scale+offset`, then
`(y offset)/div`).
### 5.4 Zoom
A zoom pins `p.xRange` and asks the hub for hi-res data over the exact range
(WS `zoom` request or HTTP `/api/zoom`). The hub answers from the full-res
rings — or from the **held copy of the last trigger capture** (`captureHold`
double buffer) while that window is still relevant — decimated to the requested
point budget. The browser prefers the fetched data when it exists, falls back
to its own circular buffers otherwise, and always brackets with local points.
---
## 6. Hub-side trigger FSM (`wshub/trigger.go`)
### 6.1 States and configuration
```
idle ──arm──▶ armed ──edge──▶ collecting ──window elapsed──▶ triggered
▲ ▲ (pre/post latched) │
│ └───────────── rearm (normal mode, after holdoff) ◀─────────┘
└────────────── disarm / single mode stays triggered
```
Configuration (`trigConfig`, client-settable via WS `setTrigger`):
| field | meaning | clamp |
|---|---|---|
| `signalKey` | `"src:sig"` or `"src:sig[i]"` | — |
| `edge` | `rising` / `falling` / `both` | — |
| `threshold` | **raw** units (SPA converts calibrated → raw) | — |
| `windowSec` | capture window | `[1e-4, 600]` |
| `prePercent` | pre-trigger share | `[0, 100]` |
| `mode` | `normal` (auto-rearm) / `single` | — |
| `holdoffSec` | re-arm delay after a capture, double-trigger guard | `[0, 60]` |
### 6.2 Edge detection (`feed`)
Called from `ingest` with every full-resolution batch for the trigger signal.
Level tracking (`prevValue`/`prevValid`) compares consecutive samples against
the threshold; `[i]`-suffixed keys stride the flattened batch by `nElem` to
watch one column. On a qualifying edge in `armed` state: `latchWindowLocked`
freezes `trigTime` and the pre/post split (so later config edits cannot move a
capture's axis).
### 6.3 Buffer-fill gate
Before accepting an edge, the FSM checks that the trigger signal's ring reaches
back far enough that the capture will come back whole (`fillLocked`):
```
need = windowSec growth × postSec, floored at the pre-window
```
`growth` is the measured span-growth rate of the ring (`setBuffered`, refreshed
by `refreshTriggerFill` from the tick and from trigger commands). A still-filling
ring grows 1 s of span per second, so the gate reduces to the pre-window; a
full ring at a long window needs the whole window. While holding off, the level
is still tracked so the first edge after the gate opens is measured against the
right predecessor. The SPA shows the hold-off as an armed trigger with a
`bufferFill %` badge.
### 6.4 Window timing and the pending edge
`dueCapture` waits for the window on the **sample clock**, not the wall clock:
`lastT ≥ trigTime + post + captureMarginSec(0.15)`. This avoids cutting a
capture short when the stream's timestamps lag real time. Three ways it fires:
1. the samples themselves covered the window;
2. wall-clock fallback when no sample was ever seen (Force from idle);
3. `captureStallSec = 2 s` of stream silence — deliver what was collected
rather than leaving the client stuck in "collecting".
While a capture is in flight the comparator keeps running. The **first**
qualifying edge at/after `notBefore = trigTime + max(post, holdoffSec)` is
remembered (`pendingT`/`pendingValid`) and fired immediately on the automatic
`rearm()`. Without this the trigger was deaf through the whole post-window +
holdoff, which rounded sparse pulse trains up to whole periods (a 1 Hz train at
a 1 s window was caught at 0.5 Hz).
### 6.5 Holdoff and rearm
`markTriggered` moves `collecting → triggered` and, in `normal` mode (not
stopped), schedules `rearmAt = now + cfg.holdoffSec`. `dueRearm` consumes it;
`rearm()` re-arms immediately on a pending edge or returns to `armed`. The
holdoff is measured from the trigger point, overlapping the post-window rather
than adding to it.
`Force()` fires immediately at the most recent sample time (wall clock if no
sample yet) — the "Force" button.
### 6.6 Capture assembly (`buildTriggerCapture`)
On a due capture the hub builds the **binary v2** frame:
```
[u8 2][f64 trigTime][f64 preSec][f64 postSec][u32 nSig]
{[u16 keyLen][fullKey][u32 N][f64 t×N][f64 v×N]}
```
For every ring:
1. `slice(trigTimepre, trigTime+post)`;
2. `backfillCaptureHead` from disk history for the front the ring lost;
3. if it is still short by more than `shortCaptureTol = 1 %` of the window,
log it explicitly (nothing can recover data the ring never held);
4. keep the **full-resolution** slice in the `captureHold` double buffer
(so a zoom into the capture can be answered after the rings roll past);
5. min/max-decimate to `trigCapturePts = 20 000` per signal for the wire —
a 60 s window at 1 MSps is ~960 MB raw per signal and would be dropped by
the send path anyway.
The double buffer is published (`capture.publish`) only once the frame is known
good, so a shot that yielded nothing leaves the previous capture on screen.
Dropped frames (client send-queue full) are logged.
`triggerTick` (every push tick) drives the whole FSM: re-tune rings → open
pending history files → refresh the fill measurement → due capture (send + mark
triggered + `hist.captureRange`) or due rearm → broadcast state only when it
changed (`stateUnsent`).
### 6.7 WS commands
| message | effect |
|---|---|
| `setTrigger {signal, edge, threshold, windowSec, prePercent, mode, holdoffSec}` | replace config |
| `arm` / `rearm` | explicit arm (discards pending edge) |
| `disarm` | → idle |
| `trigStop {stopped}` | pause/resume auto-rearm |
| `forceTrigger` | fire now |
Every command also refreshes the buffer-fill measurement synchronously — at
1 MSps the ring crosses the fill threshold many times inside one 33 ms tick, so
waiting for the next tick would fire on a stale measurement.
---
## 7. Browser-side trigger (`static/app.js`)
### 7.1 State handling
`onTriggerState(msg)` tracks the FSM broadcast:
- **armed** — shows `bufferFill %` while the hub is holding off on the fill
gate, so a trigger that is not yet fireable does not look broken;
- **collecting** — clears the previous snapshot, latches `trigTime` (and
`preSec`/`postSec` if the hub sent them), and lets live data sweep into the
trigger axis (see 7.3);
- **triggered / idle** — bookkeeping for the Rearm/Stop buttons.
### 7.2 Capture handling
`onTriggerCapture` parses the v2 frame into `trig.snapshot[key] = {t, v}` plus
the latched `_preS`/`_postS`. It is **ignored when the client did not enable
the trigger** (`trig.enabled`), because the hub keeps an armed trigger across
client sessions and applying a foreign capture would clobber this client's zoom
and scales. On receipt: the horizontal zoom is dropped so the whole capture is
visible, but **vertical scales (V/div, offset) persist** — they are user
settings and must survive from shot to shot.
### 7.3 Rendering modes (`buildUPlotData`)
| state | renderer | source |
|---|---|---|
| collecting, no snapshot yet | `buildTrigFillData` | live buffers, drawn on the *final* trigger axis (relative seconds, `[-pre, +post]`) so the trace sweeps in from the left |
| armed, not fired | freeze last frame | — |
| snapshot present | `buildTrigData` | the capture (or a hi-res zoom reply that covers ≥98 % of the view, else the snapshot) |
| otherwise | `buildLiveData` | rolling window |
`buildTrigData` converts to trigger-relative time (`t trigT`), picks the
master by rate/count, decimates (cached per range+source-tag), resamples the
other traces, and normalises Y.
### 7.4 Threshold in calibrated units
The trigger threshold is held in **calibrated units** (what the user sees on
the Y axis). `sendTrigConfig()` inverts it through the signal's calibration
before sending: `raw = (calibrated offset)/scale`, so the hub's raw
comparator fires exactly when `GAIN·signal + OFFSET` crosses the threshold.
The threshold line (`drawTriggerMarker`) maps the same calibrated threshold
through the signal's vscale: `y_norm = (threshold offset)/div`.
---
## 8. Key constants
| constant | value | file |
|---|---|---|
| push rate | 30 Hz | `hub.go` |
| `maxPushPoints` (live) | 50 pts/signal/tick | `hub.go` |
| `trigCapturePts` (capture) | 20 000 pts/signal | `trigger.go` |
| `captureMarginSec` | 0.15 s | `trigger.go` |
| `captureStallSec` | 2.0 s | `trigger.go` |
| `autoRearmDelaySec` (default holdoff) | 0.2 s | `trigger.go` |
| `maxTriggerWindowSec` | 600 s | `trigger.go` |
| `ringBudget` default | 10 000 000 pts/signal | `hub.go` |
| `ringCapInitial` | 250 000 | `hub.go` |
| `ringCapScalar` | 100 000 | `hub.go` |
| `ringHeadroom` | 1.25 | `ringbuf.go` |
| `defaultLiveWindowSec` | 10 s | `ringbuf.go` |
| `captureLagSec` | 0.15 + 1/30 ≈ 0.183 s | `ringbuf.go` |
| `monotonicTolerance` | 5 ms | `hub.go` |
| `monotonicEMAAlpha` | 0.01 | `hub.go` |
| `MAX_CAP` (browser) | 2 000 000 pts/signal | `app.js` |
| `DEFAULT_CAP` / `TEMPORAL_CAP` | 100 000 / 500 000 | `app.js` |
| `DECIM_MIN` | 200 | `app.js` |
---
## 9. The C++ StreamHub mirror
The Go hub and the C++ StreamHub implement the same WS contracts and must stay
in sync (`AGENTS.md`): same `triggerState` FSM strings, same v2 capture frame,
same command set (`setTrigger` including `holdoffSec`, `arm`, `disarm`,
`trigStop`, `forceTrigger`), same `trigCapturePts`/`kTrigCapturePts` cap, and
the same ring/history windowing intent (`Source/Applications/StreamHub/`). A
protocol change on one side must be mirrored on the other.
+11 -1
View File
@@ -1,12 +1,22 @@
module udpstreamer-webui
go 1.21
go 1.24.9
require marte2/common v0.0.0
require (
github.com/andybalholm/brotli v1.1.1 // indirect
github.com/google/uuid v1.6.0 // indirect
github.com/gorilla/websocket v1.5.1 // indirect
github.com/klauspost/compress v1.17.9 // indirect
github.com/parquet-go/bitpack v1.0.0 // indirect
github.com/parquet-go/jsonlite v1.0.0 // indirect
github.com/parquet-go/parquet-go v0.32.0 // indirect
github.com/pierrec/lz4/v4 v4.1.21 // indirect
github.com/twpayne/go-geom v1.6.1 // indirect
golang.org/x/net v0.17.0 // indirect
golang.org/x/sys v0.38.0 // indirect
google.golang.org/protobuf v1.34.2 // indirect
)
replace marte2/common => ../../Common/Client/go
+34
View File
@@ -1,4 +1,38 @@
github.com/DATA-DOG/go-sqlmock v1.5.2 h1:OcvFkGmslmlZibjAjaHm3L//6LiuBgolP7OputlJIzU=
github.com/DATA-DOG/go-sqlmock v1.5.2/go.mod h1:88MAG/4G7SMwSE3CeA0ZKzrT5CiOU3OJ+JlNzwDqpNU=
github.com/alecthomas/assert/v2 v2.10.0 h1:jjRCHsj6hBJhkmhznrCzoNpbA3zqy0fYiUcYZP/GkPY=
github.com/alecthomas/assert/v2 v2.10.0/go.mod h1:Bze95FyfUr7x34QZrjL+XP+0qgp/zg8yS+TtBj1WA3k=
github.com/alecthomas/repr v0.4.0 h1:GhI2A8MACjfegCPVq9f1FLvIBS+DrQ2KQBFZP1iFzXc=
github.com/alecthomas/repr v0.4.0/go.mod h1:Fr0507jx4eOXV7AlPV6AVZLYrLIuIeSOWtW57eE/O/4=
github.com/andybalholm/brotli v1.1.1 h1:PR2pgnyFznKEugtsUo0xLdDop5SKXd5Qf5ysW+7XdTA=
github.com/andybalholm/brotli v1.1.1/go.mod h1:05ib4cKhjx3OQYUY22hTVd34Bc8upXjOLL2rKwwZBoA=
github.com/google/go-cmp v0.5.5 h1:Khx7svrCpmxxtHBq5j2mp/xVjsi8hQMfNLvJFAlrGgU=
github.com/google/go-cmp v0.5.5/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE=
github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0=
github.com/google/uuid v1.6.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
github.com/gorilla/websocket v1.5.1 h1:gmztn0JnHVt9JZquRuzLw3g4wouNVzKL15iLr/zn/QY=
github.com/gorilla/websocket v1.5.1/go.mod h1:x3kM2JMyaluk02fnUJpQuwD2dCS5NDG2ZHL0uE0tcaY=
github.com/hexops/gotextdiff v1.0.3 h1:gitA9+qJrrTCsiCl7+kh75nPqQt1cx4ZkudSTLoUqJM=
github.com/hexops/gotextdiff v1.0.3/go.mod h1:pSWU5MAI3yDq+fZBTazCSJysOMbxWL1BSow5/V2vxeg=
github.com/klauspost/compress v1.17.9 h1:6KIumPrER1LHsvBVuDa0r5xaG0Es51mhhB9BQB2qeMA=
github.com/klauspost/compress v1.17.9/go.mod h1:Di0epgTjJY877eYKx5yC51cX2A2Vl2ibi7bDH9ttBbw=
github.com/parquet-go/bitpack v1.0.0 h1:AUqzlKzPPXf2bCdjfj4sTeacrUwsT7NlcYDMUQxPcQA=
github.com/parquet-go/bitpack v1.0.0/go.mod h1:XnVk9TH+O40eOOmvpAVZ7K2ocQFrQwysLMnc6M/8lgs=
github.com/parquet-go/jsonlite v1.0.0 h1:87QNdi56wOfsE5bdgas0vRzHPxfJgzrXGml1zZdd7VU=
github.com/parquet-go/jsonlite v1.0.0/go.mod h1:nDjpkpL4EOtqs6NQugUsi0Rleq9sW/OtC1NnZEnxzF0=
github.com/parquet-go/parquet-go v0.32.0 h1:NWDqTUHfrCS4cJP/Fj2HlxvqsrVedWG3sayMkf+znzM=
github.com/parquet-go/parquet-go v0.32.0/go.mod h1:navtkAYr2LGoJVp141oXPlO/sxLvaOe3la2JEoD8+rg=
github.com/pierrec/lz4/v4 v4.1.21 h1:yOVMLb6qSIDP67pl/5F7RepeKYu/VmTyEXvuMI5d9mQ=
github.com/pierrec/lz4/v4 v4.1.21/go.mod h1:gZWDp/Ze/IJXGXf23ltt2EXimqmTUXEy0GFuRQyBid4=
github.com/twpayne/go-geom v1.6.1 h1:iLE+Opv0Ihm/ABIcvQFGIiFBXd76oBIar9drAwHFhR4=
github.com/twpayne/go-geom v1.6.1/go.mod h1:Kr+Nly6BswFsKM5sd31YaoWS5PeDDH2NftJTK7Gd028=
github.com/xyproto/randomstring v1.0.5 h1:YtlWPoRdgMu3NZtP45drfy1GKoojuR7hmRcnhZqKjWU=
github.com/xyproto/randomstring v1.0.5/go.mod h1:rgmS5DeNXLivK7YprL0pY+lTuhNQW3iGxZ18UQApw/E=
golang.org/x/net v0.17.0 h1:pVaXccu2ozPjCXewfr1S7xza/zcXTity9cCdXQYSjIM=
golang.org/x/net v0.17.0/go.mod h1:NxSsAGuq816PNPmqtQdLE42eU2Fs7NoRIZrHJAlaCOE=
golang.org/x/sys v0.38.0 h1:3yZWxaJjBmCWXqhN1qh02AkOnCQ1poK6oF+a7xWL6Gc=
golang.org/x/sys v0.38.0/go.mod h1:OgkHotnGiDImocRcuBABYBEXf8A9a87e/uXjp9XT3ks=
golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543 h1:E7g+9GITq07hpfrRu66IVDexMakfv52eLZ2CXBWiKr4=
golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
google.golang.org/protobuf v1.34.2 h1:6xV6lTsCfpGD21XK49h7MhtcApnLqkfYgPcdHftf6hg=
google.golang.org/protobuf v1.34.2/go.mod h1:qYOHts0dSfpeUzUFpOMr/WGzszTmLH+DiWniOlNbLDw=
+1
View File
@@ -94,6 +94,7 @@ func main() {
http.Handle("/", http.FileServer(http.FS(sub)))
http.HandleFunc("/ws", hub.HandleWebSocket)
http.HandleFunc("/api/zoom", hub.HandleZoom)
http.HandleFunc("/api/export", hub.HandleExport)
http.HandleFunc("/version", func(w http.ResponseWriter, r *http.Request) {
fmt.Fprint(w, buildVersion)
})
+581 -111
View File
@@ -476,14 +476,24 @@ let cursorsDirty = false; // if true, redraw all plots to update cursor lines
// Rolling-window anchor used to keep cursors visually fixed while live data scrolls.
let _cursorAnchorNow = null;
// Horizontal value rulers — stored in normalized division units (the shared
// y scale, -4.5…4.5) so one pair applies to every plot regardless of V/div.
const rulers = { mode: 'off', yA: null, yB: null };
// Horizontal value rulers. The on/off toggle is global, but each plot keeps its
// own pair of normalized-division positions (rulerState), so dragging Y1 in
// one plot does not move it in the others.
const rulers = { mode: 'off', plotId: null };
const rulerState = {}; // plotId → { yA, yB }
// Layout — [label, cssClass, cols, rows]
function getRulerState(plotId) {
if (!rulerState[plotId]) rulerState[plotId] = { yA: null, yB: null };
return rulerState[plotId];
}
// Layout — [label, cssClass, cols, rows, (optional) plotCount].
// Custom (non-uniform) layouts carry an explicit plotCount; the grid template
// and the spanning cells are defined in style.css under #plot-grid.<class>.
const LAYOUTS = [
['1×1', 'l1x1', 1, 1], ['1×2', 'l1x2', 1, 2], ['2×1', 'l2x1', 2, 1], ['1×3', 'l1x3', 1, 3],
['3×1', 'l3x1', 3, 1], ['2×2', 'l2x2', 2, 2], ['1×4', 'l1x4', 1, 4], ['4×1', 'l4x1', 4, 1],
['1+2', 'l1p2', 2, 2, 3], // one plot spanning the top row, two below
];
let currentLayout = 'l1x1';
let colFrs = [1]; // fractional column sizes (sum = cols)
@@ -816,6 +826,7 @@ function onConfig(msg) {
}
buildSidebar();
buildTrigSignalSelect();
maybeRestoreViewLate();
}
/* ════════════════════════════════════════════════════════════════
@@ -925,12 +936,21 @@ function wsSend(obj) {
function sendWindow() {
wsSend({ type: 'setWindow', seconds: windowSec });
}
// trig.threshold is held in calibrated units. The hub's comparator runs on raw
// trig.threshold is held in calibrated units. The hub.s comparator runs on raw
// samples, so invert on the way out: raw = (calibrated - offset) / scale.
function sendTrigConfig() {
const cal = trig.signal ? calForKey(trig.signal) : Calib.IDENTITY;
// A negative calibration gain flips the signal on screen (v_cal = v_raw·scale
// + offset with scale < 0), so a calibrated rising edge is a raw FALLING
// edge. The hub compares raw samples, so send the raw direction that matches
// the edge the user picked on the calibrated trace.
let edge = trig.edge;
if (cal.scale < 0) {
if (edge === 'rising') edge = 'falling';
else if (edge === 'falling') edge = 'rising';
}
wsSend({
type: 'setTrigger', signal: trig.signal, edge: trig.edge,
type: 'setTrigger', signal: trig.signal, edge: edge,
threshold: Calib.invertCal(trig.threshold, cal), windowSec: trig.windowSec,
prePercent: trig.prePercent, mode: trig.mode, holdoffSec: trig.holdoffSec,
});
@@ -1332,7 +1352,11 @@ function decimateAsync(cacheKey, t, v, threshold, gen) {
return result;
}
}
return cached || null; // stale entry, or nothing to draw yet
// Never hand out a stale decimation: drawing it (at its old timestamps) and
// then the fresh one a frame later is what makes the trace jump/shimmer on
// every push. Return null instead — the caller holds the previous render
// until the worker's fresh result lands (it flags the plot for redraw).
return null;
}
// Evict stale decimation cache entries for a plot (call when zoom range changes).
@@ -1838,15 +1862,9 @@ function drawCursorLines(u, p) {
if (vNorm === null) return;
const cy = u.valToPos(vNorm, 'y', true);
if (cy < bbox.top || cy > bbox.top + bbox.height) return;
// Un-transform normalized value back to real units for display
// y_norm = (y_raw - offset) / divValue → y_raw = y_norm * divValue + offset
const vs = sigVScale[vsKeyFor(p.id, key)];
let vReal = vNorm;
if (vs) {
const dv = vs._resolvedDiv || vs.divValue || 1;
const ofs = vs._resolvedOffset != null ? vs._resolvedOffset : (vs.offset || 0);
vReal = vNorm * dv + ofs;
}
// Calibrated value at the cursor time, from the raw source (matches
// the hover and the cursor readouts in every display mode).
const vReal = calibratedValueAt(key, val);
const tc = getSigStyle(key).color;
// Diamond marker at intersection
ctx.fillStyle = tc;
@@ -1860,7 +1878,7 @@ function drawCursorLines(u, p) {
ctx.closePath();
ctx.fill();
// Value text next to diamond (real units)
const str = Math.abs(vReal) >= 10000 ? vReal.toExponential(2) : parseFloat(vReal.toPrecision(4)).toString();
const str = vReal === null ? '—' : (Math.abs(vReal) >= 10000 ? vReal.toExponential(2) : parseFloat(vReal.toPrecision(4)).toString());
ctx.fillStyle = tc;
ctx.font = '11px monospace';
const currentAlign = ctx.textAlign;
@@ -1898,6 +1916,8 @@ function rulerRawValue(p, yNorm) {
// Draw the horizontal value rulers (called from the draw hook).
function drawRulerLines(u, p) {
if (rulers.mode !== 'on') return;
const rs = rulerState[p.id];
if (!rs) return;
const { ctx, bbox } = u;
if (!bbox) return;
@@ -1926,8 +1946,8 @@ function drawRulerLines(u, p) {
ctx.restore();
};
drawLine(rulers.yA, 'rgba(166,227,161,0.85)', 'Y1');
drawLine(rulers.yB, 'rgba(243,139,168,0.85)', 'Y2');
drawLine(rs.yA, 'rgba(166,227,161,0.85)', 'Y1');
drawLine(rs.yB, 'rgba(243,139,168,0.85)', 'Y2');
}
// Compute the rolling-window anchor ("newest common timestamp") for a plot.
@@ -2133,8 +2153,9 @@ function createUPlot(p) {
const rect = p.uplot.over.getBoundingClientRect();
const { min, max } = p.uplot.scales.y;
const toY = val => rect.top + (1 - (val - min) / (max - min)) * rect.height;
if (rulers.yA !== null && Math.abs(clientY - toY(rulers.yA)) <= CURSOR_SNAP_PX) return 'A';
if (rulers.yB !== null && Math.abs(clientY - toY(rulers.yB)) <= CURSOR_SNAP_PX) return 'B';
const rs = rulerState[p.id];
if (rs && rs.yA !== null && Math.abs(clientY - toY(rs.yA)) <= CURSOR_SNAP_PX) return 'A';
if (rs && rs.yB !== null && Math.abs(clientY - toY(rs.yB)) <= CURSOR_SNAP_PX) return 'B';
return null;
}
@@ -2171,8 +2192,10 @@ function createUPlot(p) {
// Set cursor position immediately on mousedown
if (yTarget) {
if (yTarget === 'A') rulers.yA = _rulerValFromEvent(e);
else rulers.yB = _rulerValFromEvent(e);
rulers.plotId = p.id; // the readout follows the plot whose rulers moved
const rs = getRulerState(p.id);
if (yTarget === 'A') rs.yA = _rulerValFromEvent(e);
else rs.yB = _rulerValFromEvent(e);
} else if (target === 'A') cursors.tA = _cursorValFromEvent(e);
else cursors.tB = _cursorValFromEvent(e);
updateCursorReadout();
@@ -2180,8 +2203,9 @@ function createUPlot(p) {
const onMove = ev => {
if (yTarget) {
if (yTarget === 'A') rulers.yA = _rulerValFromEvent(ev);
else rulers.yB = _rulerValFromEvent(ev);
const rs = getRulerState(p.id);
if (yTarget === 'A') rs.yA = _rulerValFromEvent(ev);
else rs.yB = _rulerValFromEvent(ev);
} else if (target === 'A') cursors.tA = _cursorValFromEvent(ev);
else cursors.tB = _cursorValFromEvent(ev);
updateCursorReadout();
@@ -2447,8 +2471,12 @@ function buildLiveData(p) {
let dec;
if (cached) {
dec = cached;
} else if (p.uplot && p.uplot.data && p.uplot.data[0] && p.uplot.data[0].length) {
// Fresh decimation not ready yet — hold the previous render so the trace
// does not flicker between a stale decimation and the fresh one.
return p.uplot.data;
} else {
// Worker job submitted — sync fallback this frame so the plot isn't blank.
// First render: worker job submitted, nothing on screen yet — sync.
dec = decimate(masterRaw.t, masterRaw.v, targetPts);
}
sharedT = dec.t;
@@ -2521,7 +2549,14 @@ function buildTrigData(p) {
// same-length snapshot slice for the same range, so it is tagged separately.
const cacheKey = `${p.id}:${masterKey}:${t0.toFixed(6)}:${t1.toFixed(6)}:${masterRaw.t.length}:${usedFetched ? 'hi' : 'snap'}`;
const cachedDec = decimateAsync(cacheKey, masterRaw.t, masterRaw.v, targetPts);
const dec = cachedDec || decimate(masterRaw.t, masterRaw.v, targetPts);
let dec;
if (cachedDec) {
dec = cachedDec;
} else if (p.uplot && p.uplot.data && p.uplot.data[0] && p.uplot.data[0].length) {
return p.uplot.data; // hold the previous render until the fresh decimation lands
} else {
dec = decimate(masterRaw.t, masterRaw.v, targetPts);
}
// Convert absolute → relative seconds
const sharedT = new Float64Array(dec.t.length);
for (let i = 0; i < dec.t.length; i++) sharedT[i] = dec.t[i] - trigT;
@@ -2578,8 +2613,15 @@ function buildTrigFillData(p) {
masterV = masterRaw.v;
} else {
const cacheKey = `${p.id}:${masterKey}:trigfill`;
const dec = decimateAsync(cacheKey, masterRaw.t, masterRaw.v, targetPts, _dataGen) ||
decimate(masterRaw.t, masterRaw.v, targetPts);
const decd = decimateAsync(cacheKey, masterRaw.t, masterRaw.v, targetPts, _dataGen);
let dec;
if (decd) {
dec = decd;
} else if (p.uplot && p.uplot.data && p.uplot.data[0] && p.uplot.data[0].length) {
return p.uplot.data; // hold until the fresh decimation is ready
} else {
dec = decimate(masterRaw.t, masterRaw.v, targetPts);
}
sharedAbsT = dec.t;
masterV = dec.v;
}
@@ -2753,7 +2795,19 @@ function updateCursorBtnVisibility() {
under one — so a zoom, a pan or a new capture can leave them outside the
viewport entirely, with no way to get them back: they are dragged by grabbing
their line, and an off-screen line cannot be grabbed. */
function resetRulers() {
// Re-place every plot's rulers at the default ±2 divisions, like
// resetCursors re-places the vertical cursors.
plots.forEach(p => {
const rs = getRulerState(p.id);
rs.yA = -2; rs.yB = 2;
});
updateCursorReadout();
cursorsDirty = true;
}
function resetCursors() {
resetRulers();
const refPlot = plots.find(p => p.uplot);
if (!refPlot) return;
const { min, max } = refPlot.uplot.scales.x;
@@ -2790,9 +2844,13 @@ document.getElementById('btn-ruler').addEventListener('click', () => {
rulers.mode = rulers.mode === 'off' ? 'on' : 'off';
const btn = document.getElementById('btn-ruler');
btn.classList.toggle('active', rulers.mode === 'on');
if (rulers.mode === 'on' && rulers.yA === null && rulers.yB === null) {
// Auto-place at ±2 divisions from the centre on first use.
rulers.yA = -2; rulers.yB = 2;
if (rulers.mode === 'on') {
// Auto-place every plot at ±2 divisions from the centre on first use;
// afterwards each plot keeps its own positions.
plots.forEach(pl => {
const rs = getRulerState(pl.id);
if (rs.yA === null && rs.yB === null) { rs.yA = -2; rs.yB = 2; }
});
}
updateCursorReadout();
cursorsDirty = true;
@@ -2809,16 +2867,9 @@ function getValueAtCursor(p, t) {
if (!p.uplot || t === null) return null;
const key = plotActiveSignal[p.id] || (p.traces.length === 1 ? p.traces[0] : null);
if (!key) return null;
const idx = p.traces.indexOf(key);
if (idx < 0) return null;
const vNorm = interpAtTime(p.uplot, idx + 1, t);
if (vNorm === null) return null;
// Un-normalize: y_norm = (y_raw - offset) / divValue
const vs = sigVScale[p.id + ':' + key];
if (!vs) return vNorm;
const dv = vs._resolvedDiv != null ? vs._resolvedDiv : (vs.divValue || 1);
const ofs = vs._resolvedOffset != null ? vs._resolvedOffset : (vs.offset || 0);
return vNorm * dv + ofs;
// Interpolate the raw wire value and apply the calibration explicitly, so
// cursor readouts match the hover in every display mode.
return calibratedValueAt(key, t);
}
// Update per-plot cursor value readouts (A, B, ΔV) for all plots.
@@ -2851,6 +2902,64 @@ function rawFromNorm(p, key, vNorm) {
const ofs = vs._resolvedOffset != null ? vs._resolvedOffset : (vs.offset || 0);
return vNorm * dv + ofs;
}
// Linear interpolation of a sorted (t, v) pair at absolute time tAbs. Returns
// null outside the data's range — never fabricated, so an export or readout
// cannot invent samples the signal never had.
function interpSortedRaw(t, v, tAbs) {
if (!t || t.length === 0) return null;
if (tAbs < t[0] || tAbs > t[t.length - 1]) return null;
let lo = 0, hi = t.length - 1;
while (lo < hi) { const m = (lo + hi) >> 1; if (t[m] < tAbs) lo = m + 1; else hi = m; }
if (lo === 0) return v[0] ?? null;
const t0 = t[lo - 1], t1 = t[lo];
const v0 = v[lo - 1], v1 = v[lo];
if (v0 == null || v1 == null) return v0 ?? v1 ?? null;
return v0 + (tAbs - t0) / (t1 - t0) * (v1 - v0);
}
// Binary-search linear interpolation of a circular buffer at time t.
function interpCircular(buf, t) {
if (!buf || buf.size === 0) return null;
const { cap, size, head } = buf;
const start = (size === cap) ? head : 0;
const physAt = k => (start + k) % cap;
let lo = 0, hi = size;
while (lo < hi) { const m = (lo + hi) >> 1; if (buf.t[physAt(m)] < t) lo = m + 1; else hi = m; }
if (lo === 0) return buf.v[physAt(0)] ?? null;
if (lo >= size) return buf.v[physAt(size - 1)] ?? null;
const t0 = buf.t[physAt(lo - 1)], t1 = buf.t[physAt(lo)];
const v0 = buf.v[physAt(lo - 1)], v1 = buf.v[physAt(lo)];
if (v0 == null || v1 == null) return v0 ?? v1 ?? null;
return v0 + (t - t0) / (t1 - t0) * (v1 - v0);
}
// Raw (uncalibrated) value of `key` at absolute time tAbs, from the best
// available raw source: trigger snapshot → fetched zoom data → live push
// buffer. All three store wire values, so calibration is applied here, at the
// point of display, exactly once.
function rawAtAbsTime(key, tAbs) {
if (trig.snapshot) {
const s = trig.snapshot[key];
if (s && s.t.length) { const v = interpSortedRaw(s.t, s.v, tAbs); if (v != null) return v; }
}
for (const p of plots) {
const zd = zoomData[p.id];
if (!zd) continue;
const s = zd.signals[key];
if (s && s.t.length) { const v = interpSortedRaw(s.t, s.v, tAbs); if (v != null) return v; }
}
const buf = buffers[key];
if (buf && buf.size) { const v = interpCircular(buf, tAbs); if (v != null) return v; }
return null;
}
// Calibrated value of `key` at axis time t. Under a trigger the axis is
// relative to the trigger instant, so convert to absolute first.
function calibratedValueAt(key, t) {
const tAbs = (inTrigWindow() && trig.trigTime != null) ? trig.trigTime + t : t;
const raw = rawAtAbsTime(key, tAbs);
return raw === null ? null : Calib.applyCal(raw, calForKey(key));
}
function hideHoverReadout() {
document.getElementById('hover-readout').style.display = 'none';
@@ -2870,13 +2979,14 @@ function showHoverReadout(p, e) {
const tStr = inTrigWindow() ? fmtDuration(t, span, true) : fmtLiveTime(t, span);
let html = '<div class="hov-time">' + escHtml(tStr) + '</div>';
p.traces.forEach((key, idx) => {
const vNorm = interpAtTime(p.uplot, idx + 1, t);
const name = key.includes(':') ? key.slice(key.indexOf(':') + 1) : key;
// rawFromNorm inverts the vscale transform, which Task 7 made operate on
// calibrated values — so this is already in calibrated units.
const unit = unitForKey(key);
const val = vNorm === null ? '—'
: (_fmtVal(rawFromNorm(p, key, vNorm)) + (unit ? ' ' + unit : ''));
// Interpolate the raw wire value and apply the calibration explicitly,
// so the hover is correct in every display mode (analog, digital,
// mixed) and independent of the vscale state.
const vCal = calibratedValueAt(key, t);
const val = vCal === null ? '—'
: (_fmtVal(vCal) + (unit ? ' ' + unit : ''));
html += '<div class="hov-row"><span class="hov-dot" style="background:' +
escHtml(getSigStyle(key).color) + '"></span>' +
'<span class="hov-name">' + escHtml(name) + '</span>' +
@@ -2894,17 +3004,25 @@ function showHoverReadout(p, e) {
el.style.top = Math.max(4, y) + 'px';
}
// Update the Y1/Y2/ΔY ruler readout, expressed in the raw units of the first
// plot that has an active (or sole) signal.
// Update the Y1/Y2/ΔY ruler readout, expressed in the raw units of the plot
// whose rulers were last moved, falling back to the first plot with a signal.
function updateRulerReadout() {
const box = document.getElementById('ruler-readout');
const on = rulers.mode === 'on';
box.style.display = on ? '' : 'none';
if (!on) return;
const ref = plots.find(p => p.uplot && p.traces.length > 0 &&
rulerRawValue(p, 0) !== null);
const conv = y => (y === null || !ref) ? null : rulerRawValue(ref, y);
const vA = conv(rulers.yA), vB = conv(rulers.yB);
let ref = null;
if (rulers.plotId !== null) {
const pl = plots.find(p => p.id === rulers.plotId);
if (pl && pl.uplot && pl.traces.length > 0) ref = pl;
}
if (!ref) {
ref = plots.find(p => p.uplot && p.traces.length > 0 &&
rulerRawValue(p, 0) !== null) || null;
}
const rs = ref ? rulerState[ref.id] : null;
const conv = y => (y === null || !ref || !rs) ? null : rulerRawValue(ref, y);
const vA = conv(rs ? rs.yA : null), vB = conv(rs ? rs.yB : null);
document.getElementById('cur-y1').textContent = 'Y1: ' + fmtVal(vA);
document.getElementById('cur-y2').textContent = 'Y2: ' + fmtVal(vB);
document.getElementById('cur-dy').textContent =
@@ -3355,18 +3473,36 @@ function initPlotCfgBar(plotId, p) {
/* ════════════════════════════════════════════════════════════════
Layout management
════════════════════════════════════════════════════════════════ */
// Returns the number of plot cells in a layout (cols × rows).
// Returns the number of plot cells in a layout. Custom layouts carry an
// explicit plotCount; uniform ones are cols × rows.
function layoutPlotCount(cls) {
const entry = LAYOUTS.find(l => l[1] === cls);
if (entry) {
if (entry.length >= 5) return entry[4];
return entry[2] * entry[3];
}
const m = cls.match(/^l(\d+)x(\d+)$/);
return m ? parseInt(m[1]) * parseInt(m[2]) : 1;
}
// Build a small SVG grid thumbnail for a given cols×rows layout.
function layoutSVG(cols, rows) {
// Build a small SVG grid thumbnail for a layout entry. Custom (non-uniform)
// layouts draw their own cell arrangement.
function layoutSVG(entry) {
const W = 28, H = 20, GAP = 1.5, PAD = 1.5;
let rects = '';
if (entry[1] === 'l1p2') {
// 1+2: one full-width cell on top, two side by side below.
const cw = (W - PAD * 2 - GAP) / 2;
const ch = (H - PAD * 2 - GAP) / 2;
const y2 = (PAD + ch + GAP).toFixed(1);
const x2 = (PAD + cw + GAP).toFixed(1);
rects += `<rect x="${PAD}" y="${PAD}" width="${(W - PAD * 2).toFixed(1)}" height="${ch.toFixed(1)}" rx="1.5"/>`;
rects += `<rect x="${PAD}" y="${y2}" width="${cw.toFixed(1)}" height="${ch.toFixed(1)}" rx="1.5"/>`;
rects += `<rect x="${x2}" y="${y2}" width="${cw.toFixed(1)}" height="${ch.toFixed(1)}" rx="1.5"/>`;
} else {
const [, , cols, rows] = entry;
const cw = (W - PAD * 2 - GAP * (cols - 1)) / cols;
const ch = (H - PAD * 2 - GAP * (rows - 1)) / rows;
let rects = '';
for (let r = 0; r < rows; r++) {
for (let c = 0; c < cols; c++) {
const x = (PAD + c * (cw + GAP)).toFixed(1);
@@ -3374,6 +3510,7 @@ function layoutSVG(cols, rows) {
rects += `<rect x="${x}" y="${y}" width="${cw.toFixed(1)}" height="${ch.toFixed(1)}" rx="1.5"/>`;
}
}
}
return `<svg xmlns="http://www.w3.org/2000/svg" width="${W}" height="${H}" viewBox="0 0 ${W} ${H}">`
+ `<rect width="${W}" height="${H}" rx="2" fill="#11111b"/>`
+ `<g fill="#45475a">${rects}</g></svg>`;
@@ -3401,7 +3538,7 @@ function applyLayout(cls) {
// Update button label
const btn = document.getElementById('btn-layout');
if (btn) btn.innerHTML = layoutSVG(cols, rows) + ' <span>' + label + '</span> ▾';
if (btn) btn.innerHTML = layoutSVG(entry) + ' <span>' + label + '</span> ▾';
// Update active state in menu
document.querySelectorAll('.layout-menu-item')
@@ -3436,11 +3573,12 @@ function applyLayout(cls) {
function buildLayoutMenu() {
const menu = document.getElementById('layout-menu');
LAYOUTS.forEach(([label, cls, cols, rows]) => {
LAYOUTS.forEach(entry => {
const [label, cls] = entry;
const item = document.createElement('button');
item.className = 'layout-menu-item' + (cls === currentLayout ? ' active' : '');
item.dataset.layout = cls;
item.innerHTML = layoutSVG(cols, rows) + '<span>' + label + '</span>';
item.innerHTML = layoutSVG(entry) + '<span>' + label + '</span>';
item.addEventListener('click', () => {
applyLayout(cls);
menu.classList.remove('open');
@@ -3467,9 +3605,20 @@ function buildLayoutMenu() {
/* ════════════════════════════════════════════════════════════════
Export CSV (all plots) — fetches full-resolution data from ring
════════════════════════════════════════════════════════════════ */
// Shared busy state for the export dropdown: prevents re-entry and shows
// progress on the selector while a (possibly large) export runs.
let exportBusy = false;
function setExportBusy(busy) {
exportBusy = busy;
const sel = document.getElementById('export-select');
if (!sel) return;
sel.disabled = busy;
const ph = sel.querySelector('option[value=""]');
if (ph) ph.textContent = busy ? '\u23f3 Exporting\u2026' : '\u23ea Export';
}
async function exportAllCSV() {
const btn = document.getElementById('btn-csv-all');
if (btn.disabled) return;
if (exportBusy) return;
const inTrigMode = trig.enabled && trig.snapshot !== null;
@@ -3482,8 +3631,8 @@ async function exportAllCSV() {
let t0, t1, relOffset = 0;
if (inTrigMode) {
// Export the full trigger window around the trigger event.
t0 = trig.trigTime - trigPreSec();
t1 = trig.trigTime + trigPostSec();
t0 = trig.trigTime - activePreSec();
t1 = trig.trigTime + activePostSec();
relOffset = trig.trigTime;
} else {
// Use the current zoom range if active, else the rolling window.
@@ -3504,60 +3653,52 @@ async function exportAllCSV() {
t1 = plotNow;
}
}
if (!(t1 > t0)) return;
// Show loading state.
const origLabel = btn.textContent;
btn.textContent = '⏳ Downloading…';
btn.disabled = true;
exportBusy = true;
// Cap the export. A full window at a megasample rate is hundreds of MB raw
// (the old exact-timestamp merge exploded into millions of rows and crashed
// the tab); ask the hub for a min/max-decimated envelope — the same scope
// style reduction the live view uses — and cap the number of rows.
const BUDGET = 100000; // max rows per signal
setExportBusy(true);
// Fetch full-resolution ring data (n=0 → no decimation).
let ringSignals = null;
if (!inTrigMode) {
try {
ringSignals = await wsZoomRequest(t0, t1, 0, keys);
ringSignals = await wsZoomRequest(t0, t1, BUDGET, keys);
} catch (e) {
console.warn('CSV export: ring fetch failed, falling back to push buffer', e);
} finally {
btn.textContent = origLabel;
btn.disabled = false;
console.warn('CSV export: ring fetch failed, falling back to local data', e);
}
}
setExportBusy(false);
// Build per-signal time/value arrays.
// Priority: ring buffer (full res) → trigger snapshot → push buffer.
// Per-signal raw source: hub ring (whole window, decimated) → trigger
// snapshot (already \u226420k pts) → local push buffer.
const slices = keys.map(key => {
if (!inTrigMode) {
const rd = ringSignals && ringSignals[key];
if (rd && rd.t && rd.t.length > 0) {
const t = rd.t, v = rd.v;
if (inTrigMode) {
return { t: Array.from(t).map(ts => ts - relOffset), v: Array.from(v) };
if (rd && rd.t && rd.t.length > 0) return { key, t: rd.t, v: rd.v };
}
return { t: Array.from(t), v: Array.from(v) };
}
// Fallback: push buffer or trigger snapshot.
if (inTrigMode) {
const raw = trig.snapshot[key] || { t: new Float64Array(0), v: new Float64Array(0) };
return { t: Array.from(raw.t).map(ts => ts - relOffset), v: Array.from(raw.v) };
return { key, t: raw.t, v: raw.v };
}
const buf = buffers[key]; if (!buf) return { t: [], v: [] };
const sl = getBufferSliceRange(buf, t0, t1);
return { t: Array.from(sl.t), v: Array.from(sl.v) };
const buf = buffers[key];
const sl = buf ? getBufferSliceRange(buf, t0, t1) : { t: new Float64Array(0), v: new Float64Array(0) };
return { key, t: sl.t, v: sl.v };
});
const present = slices.filter(s => s.t.length > 0);
if (!present.length) return;
// Merge all timestamps and build aligned rows.
const allT = new Set();
slices.forEach(s => s.t.forEach(t => allT.add(t)));
const sortedT = Array.from(allT).sort((a, b) => a - b);
if (!sortedT.length) return;
// Master time grid = the signal with the most samples; every other signal is
// resampled onto it (linear, no extrapolation). Cells outside a signal's own
// span stay empty rather than being fabricated, so continuous signals export
// without holes and no value is invented.
let master = present[0];
present.forEach(s => { if (s.t.length > master.t.length) master = s; });
const lookups = slices.map(s => {
const m = new Map();
s.t.forEach((t, i) => m.set(t, s.v[i]));
return m;
});
// Strip "sourceId:" prefix from column headers for readability, and append
// the effective unit. These values come straight from the ring/history/
// snapshot and never pass through applyVScaleNorm, so calibrate them here.
const cals = keys.map(k => calForKey(k));
const cals = new Map(keys.map(k => [k, calForKey(k)]));
const displayKeys = keys.map(k => {
const name = k.includes(':') ? k.split(':').slice(1).join(':') : k;
const u = unitForKey(k);
@@ -3566,10 +3707,21 @@ async function exportAllCSV() {
});
const timeCol = '"' + (inTrigMode ? 'time_rel_s' : 'time_s') + '"';
const hdr = [timeCol, ...displayKeys].join(',');
const rows = sortedT.map(t =>
[t.toFixed(9), ...lookups.map((lk, i) =>
lk.has(t) ? Calib.applyCal(lk.get(t), cals[i]) : '')].join(',')
);
const rows = new Array(master.t.length);
for (let i = 0; i < master.t.length; i++) {
const tAbs = master.t[i];
const cells = present.map(s => {
if (s === master) {
return Calib.applyCal(master.v[i], cals.get(s.key));
}
const v = interpSortedRaw(s.t, s.v, tAbs);
return v === null ? '' : Calib.applyCal(v, cals.get(s.key));
});
const tt = inTrigMode ? tAbs - relOffset : tAbs;
rows[i] = [tt.toFixed(9), ...cells].join(',');
}
const blob = new Blob([hdr + '\n' + rows.join('\n')], { type: 'text/csv' });
const a = document.createElement('a');
a.href = URL.createObjectURL(blob);
@@ -3754,6 +3906,10 @@ function deletePlot(plotId) {
let _dbgTick = 0;
let _dataGen = 0; // incremented each time new data arrives
function renderDirtyPlots() {
// Schedule the next frame FIRST: an exception below must never kill the
// animation loop, or every plot would freeze until a page refresh.
requestAnimationFrame(renderDirtyPlots);
try {
// Compute global "now" once — shared by all rolling-window plots this frame.
const globalPlotNow = getGlobalNow();
@@ -3823,7 +3979,7 @@ function renderDirtyPlots() {
plots.forEach(p => {
if (!p.needsRedraw || !p.uplot || p.traces.length === 0) return;
try {
const inTrigModeNow = inTrigWindow();
// The x tick formatter and the cursor-sync group are baked into the uPlot
// options at construction. A plot built in live mode therefore keeps
@@ -3838,7 +3994,10 @@ function renderDirtyPlots() {
if (isRolling && _dataGen === p.lastDataGen && p.uplot.data && p.uplot.data[0] && p.uplot.data[0].length > 0) {
p.needsRedraw = false;
zoomGuard = true;
p.uplot.setScale('x', { min: globalPlotNow - windowSec, max: globalPlotNow });
// Use the same per-plot anchor as the rebuild path, so the rolling window
// does not jump when the frame switches between the two.
const plotNow = computePlotNow(p);
p.uplot.setScale('x', { min: plotNow - windowSec, max: plotNow });
zoomGuard = false;
return;
}
@@ -3875,12 +4034,25 @@ function renderDirtyPlots() {
p.uplot.setScale('x', { min: plotNow - windowSec, max: plotNow });
}
zoomGuard = false;
p._errCount = 0;
} catch (e) {
// One bad plot must not kill the whole render loop. Track consecutive
// failures and self-heal by rebuilding the uPlot instance.
p._errCount = (p._errCount || 0) + 1;
console.error(`[render] plot ${p.id}:`, e);
p.needsRedraw = true; // retry next frame
if (p._errCount >= 30) {
p._errCount = 0;
try { createUPlot(p); } catch (e2) { console.error(`[render] rebuild plot ${p.id}:`, e2); }
}
}
});
// Keep per-plot cursor value readouts in sync with live data.
if (cursors.mode === 'on') updatePlotCursorReadouts();
requestAnimationFrame(renderDirtyPlots);
} catch (e) {
console.error('[render]', e);
}
}
@@ -3958,6 +4130,7 @@ function onSources(msg) {
});
buildSidebar();
if (statsOpen) _refreshStatsSelector();
maybeRestoreViewLate();
}
function addSourceWS(label, addr, multicastGroup, dataPort) {
@@ -4582,7 +4755,303 @@ initSignalMenu();
const cb = document.getElementById('cb-monotonic');
if (cb) cb.checked = localStorage.getItem('udpscope.monotonic') === '1';
}
document.getElementById('btn-csv-all').addEventListener('click', exportAllCSV);
// Export every stored sample of the plotted signals as a Parquet file, served
// by the Go hub's /api/export. Full resolution (no decimation) and hole-free
// (each signal keeps its own timestamps — long format). The file can be huge
// (hundreds of MB at high rates), so stream it to disk when the File System
// Access API is available instead of holding it in a Blob.
async function exportParquet() {
if (exportBusy) return;
const inTrigMode = trig.enabled && trig.snapshot !== null;
const keys = [];
plots.forEach(p => p.traces.forEach(k => { if (!keys.includes(k)) keys.push(k); }));
if (!keys.length) return;
// Same range resolution as the CSV export.
let t0, t1;
if (inTrigMode) {
t0 = trig.trigTime - activePreSec();
t1 = trig.trigTime + activePostSec();
} else {
const refPlot = plots.find(p => p.xRange);
if (refPlot) {
[t0, t1] = refPlot.xRange;
} else {
let plotNow = -Infinity;
keys.forEach(k => {
const buf = buffers[k];
if (buf && buf.size > 0) {
const t = buf.t[(buf.head - 1 + buf.cap) % buf.cap];
if (t > plotNow) plotNow = t;
}
});
if (!isFinite(plotNow)) plotNow = Date.now() / 1000;
t0 = plotNow - windowSec;
t1 = plotNow;
}
}
if (!(t1 > t0)) return;
exportBusy = true;
setExportBusy(true);
try {
const url = '/api/export?t0=' + t0.toFixed(9) + '&t1=' + t1.toFixed(9) +
'&signals=' + encodeURIComponent(keys.join(','));
const resp = await fetch(url);
if (!resp.ok) {
alert('Parquet export failed (HTTP ' + resp.status + ').\n\n' +
'The /api/export endpoint is provided by the Go hub; the C++ ' +
'StreamHub does not serve it.');
return;
}
const filename = 'signals_' + Date.now() + '.parquet';
if (window.showSaveFilePicker && resp.body) {
try {
const handle = await window.showSaveFilePicker({
suggestedName: filename,
types: [{ description: 'Parquet', accept: { 'application/vnd.apache.parquet': ['.parquet'] } }],
});
const writable = await handle.createWritable();
await resp.body.pipeTo(writable);
return;
} catch (e) {
if (e && e.name === 'AbortError') return; // user cancelled the picker
console.warn('parquet export: file picker failed, falling back to Blob', e);
}
}
const blob = await resp.blob();
const a = document.createElement('a');
a.href = URL.createObjectURL(blob);
a.download = filename;
a.click();
URL.revokeObjectURL(a.href);
} catch (e) {
console.warn('parquet export failed', e);
alert('Parquet export failed: ' + e.message);
} finally {
setExportBusy(false);
}
}
// Export dropdown: dispatch on selection, then reset to the placeholder so the
// same format can be chosen again.
document.getElementById('export-select').addEventListener('change', () => {
const sel = document.getElementById('export-select');
const fmt = sel.value;
sel.value = '';
if (fmt === 'csv') exportAllCSV();
else if (fmt === 'parquet') exportParquet();
});
/* ════════════════════════════════════════════════════════════════
View-state persistence (cookie)
════════════════════════════════════════════════════════════════ */
// The whole client view — layout, plots (traces/titles/modes), window, trigger
// configuration, rulers, sources — is serialised into one cookie so a reload
// restores the previous view. Cookies are size-limited, so the state degrades
// gracefully (rulers → trigger → sources → traces) when it would not fit.
const VIEW_COOKIE = 'udpscope.view';
const VIEW_COOKIE_MAX = 3500; // encoded chars; browsers cap cookies at ~4 KiB
function packViewState() {
const state = {
v: 1,
windowSec: windowSec,
layout: currentLayout,
plots: plots.map(p => ({
title: p.title,
mode: p.mode,
traces: p.traces.map(k => {
const colon = k.indexOf(':');
const name = colon >= 0 ? k.slice(colon + 1) : k;
return { key: k, label: srcLabelForKey(k), name };
}),
})),
trig: {
enabled: trig.enabled, signal: trig.signal, edge: trig.edge,
threshold: trig.threshold, windowSec: trig.windowSec,
prePercent: trig.prePercent, mode: trig.mode, holdoffSec: trig.holdoffSec,
},
rulers: {
mode: rulers.mode,
plotId: plots.findIndex(p => p.id === rulers.plotId),
states: plots.map(p => {
const rs = rulerState[p.id];
return rs ? { yA: rs.yA, yB: rs.yB } : { yA: null, yB: null };
}),
},
sources: Object.values(sourcesMap).map(s => ({
label: s.label || s.addr || s.id, addr: s.addr,
})),
};
let s = JSON.stringify(state);
const tooBig = () => encodeURIComponent(s).length > VIEW_COOKIE_MAX;
if (tooBig()) { delete state.rulers; s = JSON.stringify(state); }
if (tooBig()) { delete state.trig; s = JSON.stringify(state); }
if (tooBig()) { delete state.sources; s = JSON.stringify(state); }
if (tooBig()) {
state.plots = state.plots.map(p => ({ title: p.title, mode: p.mode }));
s = JSON.stringify(state);
}
if (tooBig()) { state.plots = []; s = JSON.stringify(state); }
return s;
}
// Saves are gated until the saved view has been re-applied (phase 2) or the
// grace timeout fires: otherwise the very first periodic save would overwrite
// the cookie with the not-yet-restored (empty) state and destroy it.
let _viewSaveReady = false;
function saveViewState() {
if (!_viewSaveReady) return;
try {
const s = packViewState();
document.cookie = VIEW_COOKIE + '=' + encodeURIComponent(s) +
'; path=/; max-age=31536000; SameSite=Lax';
} catch (e) {
console.warn('view cookie save failed', e);
}
}
function readViewState() {
try {
const prefix = VIEW_COOKIE + '=';
const m = document.cookie.split('; ').find(c => c.startsWith(prefix));
if (!m) return null;
const st = JSON.parse(decodeURIComponent(m.slice(prefix.length)));
return (st && st.v === 1) ? st : null;
} catch (e) {
return null;
}
}
// Phase 1 (init): layout, plot cards (titles/modes), window, rulers. Traces and
// the trigger need sources/signals loaded, so they are applied in phase 2.
function restoreViewState() {
const st = readViewState();
if (!st) return;
if (st.layout && LAYOUTS.some(l => l[1] === st.layout)) applyLayout(st.layout);
const plotState = st.plots || [];
plotState.forEach((ps, i) => {
const p = plots[i];
if (!p) return;
if (ps.title && ps.title !== 'Plot ' + p.id) {
p.title = ps.title;
const tEl = document.getElementById('ptitle-' + p.id);
if (tEl) tEl.textContent = ps.title;
const inp = document.querySelector('#pcfg-' + p.id + ' .pcfg-title-input');
if (inp) inp.value = ps.title;
}
if (ps.mode && ps.mode !== p.mode) {
p.mode = ps.mode;
document.querySelectorAll('#pcfg-' + p.id + ' .pcfg-mode-btn')
.forEach(b => b.classList.toggle('active', b.dataset.mode === ps.mode));
}
});
if (st.windowSec != null) {
windowSec = st.windowSec;
const sel = document.getElementById('window-select');
if (sel && [...sel.options].some(o => o.value === String(st.windowSec))) {
sel.value = String(st.windowSec);
}
}
if (st.rulers) {
rulers.mode = st.rulers.mode === 'on' ? 'on' : 'off';
const btn = document.getElementById('btn-ruler');
if (btn) btn.classList.toggle('active', rulers.mode === 'on');
(st.rulers.states || []).forEach((rs, i) => {
const p = plots[i];
if (!p || !rs) return;
const cur = getRulerState(p.id);
cur.yA = rs.yA; cur.yB = rs.yB;
});
if (st.rulers.plotId != null && plots[st.rulers.plotId]) {
rulers.plotId = plots[st.rulers.plotId].id;
}
}
}
// Rebuild a saved trace key against the current sources: source ids change
// across restarts, so match by label and fall back to the saved id-key.
function restoreTraceKey(entry) {
const src = Object.values(sourcesMap).find(s => (s.label || s.id) === entry.label);
return src ? (src.id + ':' + entry.name) : entry.key;
}
// Phase 2 (first sources + signals): reconcile sources, re-apply traces, then
// the trigger configuration and the window to the hub.
let _viewLateRestored = false;
function maybeRestoreViewLate() {
if (_viewLateRestored) return;
const st = readViewState();
if (!st) { _viewLateRestored = true; return; }
// Add any saved sources the hub does not already have (it persists its own).
const known = Object.values(sourcesMap).map(s => (s.label || s.addr || s.id) + '\u0000' + s.addr);
let added = false;
(st.sources || []).forEach(sv => {
const k = (sv.label || sv.addr) + '\u0000' + (sv.addr || '');
if (!known.includes(k)) { addSourceWS(sv.label, sv.addr, sv.multicastGroup, sv.dataPort); added = true; }
});
if (added) return; // re-enter when the new sources appear
// Traces and the trigger selector need at least one source with signals.
if (!Object.values(sourcesMap).some(s => (s.signals || []).length > 0)) return;
_viewLateRestored = true;
_viewSaveReady = true;
(st.plots || []).forEach((ps, i) => {
const p = plots[i];
if (!p) return;
(ps.traces || []).forEach(t => addTraceTo(p.id, restoreTraceKey(t)));
});
if (st.windowSec != null) {
windowSec = st.windowSec;
sendWindow();
}
const t = st.trig;
if (t) {
trig.edge = t.edge || trig.edge;
if (t.threshold != null) trig.threshold = t.threshold;
if (t.windowSec != null) trig.windowSec = t.windowSec;
if (t.prePercent != null) trig.prePercent = t.prePercent;
trig.mode = t.mode || trig.mode;
if (t.holdoffSec != null) trig.holdoffSec = t.holdoffSec;
trig.signal = t.signal || '';
const el = id => document.getElementById(id);
if (el('trig-edge')) el('trig-edge').value = trig.edge;
if (el('trig-window')) el('trig-window').value = String(trig.windowSec);
if (el('trig-mode')) el('trig-mode').value = trig.mode;
if (el('trig-holdoff')) el('trig-holdoff').value = trig.holdoffSec;
if (el('trig-pre')) el('trig-pre').value = String(trig.prePercent);
if (el('trig-pre-val')) el('trig-pre-val').textContent = trig.prePercent + '%';
refreshTrigThresholdField();
const selSig = document.getElementById('trig-signal');
if (selSig && trig.signal) {
const base = trig.signal.replace(/\[\d+\]$/, '');
if ([...selSig.options].some(o => o.value === base)) selSig.value = base;
}
if (t.enabled) openTrigBar(true);
else updateTrigStatusBadge('idle');
}
}
// Periodic save keeps the cookie current without wiring every control; the
// pagehide save captures the final state on close/reload.
setInterval(saveViewState, 3000);
window.addEventListener('pagehide', saveViewState);
// Hub down / never connected: stop gating after 10 s so layout, window and
// rulers still persist even though traces and the trigger could not be
// restored.
setTimeout(() => { _viewSaveReady = true; }, 10000);
document.getElementById('history-badge').addEventListener('click', toggleHistoryPanel);
document.getElementById('btn-hist-cancel').addEventListener('click', toggleHistoryPanel);
document.getElementById('btn-hist-apply').addEventListener('click', applyHistoryBudget);
@@ -4592,6 +5061,7 @@ document.getElementById('stats-source-sel').addEventListener('change', e => {
statsSelectedSrc = e.target.value || null;
renderStats();
});
restoreViewState();
resolveHub().then(connectWS);
requestAnimationFrame(renderDirtyPlots);
fetch('/version').then(r => r.text()).then(v => {
+5 -1
View File
@@ -41,7 +41,11 @@
<button id="btn-ruler" class="ctrl-btn" title="Horizontal value rulers">Rulers</button>
<button id="btn-zoom-back" class="ctrl-btn" style="display:none">← Back</button>
<button id="btn-zoom-fit" class="ctrl-btn">Fit</button>
<button id="btn-csv-all" class="ctrl-btn" title="Export all signals to CSV">⬇ CSV</button>
<select id="export-select" class="ctrl-select" title="Export the visible signals">
<option value="" disabled selected>⬇ Export</option>
<option value="csv" title="Export the visible signals as CSV (decimated to a bounded row count)">CSV</option>
<option value="parquet" title="Export every stored sample (full resolution, no holes) as Parquet — requires the Go hub">Parquet</option>
</select>
<button id="btn-sync-resume" class="ctrl-btn resume-btn" style="display:none">↺ Auto</button>
<button id="btn-trigger" class="ctrl-btn">⚡ Trigger</button>
<button id="btn-pause-global" class="ctrl-btn">⏸ Pause</button>
+8
View File
@@ -14,6 +14,11 @@
*, *::before, *::after { box-sizing: border-box; margin: 0; padding: 0; }
html, body { height:100%; background:var(--bg); color:var(--text);
font-family:'Segoe UI',system-ui,sans-serif; font-size:14px; overflow:hidden; }
/* Uniform 0.9x compaction — scales every element (fonts, bars, plots,
spacing) while reflowing layout. `zoom` (Chrome/Edge/Safari, Firefox 126+)
is preferred over `transform: scale` because it reflows, so fixed-position
bars and JS-computed offsets stay aligned. */
html { zoom: 0.9; }
::-webkit-scrollbar { width:6px; }
::-webkit-scrollbar-track { background:var(--mantle); }
::-webkit-scrollbar-thumb { background:var(--surface1); border-radius:3px; }
@@ -256,6 +261,9 @@ input[type=range].trig-range::-webkit-slider-thumb {
#plot-grid.l2x3 { grid-template-columns:1fr 1fr; grid-template-rows:1fr 1fr 1fr; }
#plot-grid.l1x4 { grid-template-columns:1fr; grid-template-rows:1fr 1fr 1fr 1fr; }
#plot-grid.l4x1 { grid-template-columns:1fr 1fr 1fr 1fr; grid-template-rows:1fr; }
/* 1+2 layout: one plot spanning the top row, two side by side below. */
#plot-grid.l1p2 { grid-template-columns:1fr 1fr; grid-template-rows:1fr 1fr; }
#plot-grid.l1p2 .plot-card:first-child { grid-column: 1 / -1; }
/* ── Plot card ────────────────────────────────────────────────── */
.plot-card {
+26 -5
View File
@@ -482,11 +482,36 @@ static int decode_data(udps_client_t *c, const uint8_t *pl, size_t len,
size_t total = 0u;
size_t written = 0u;
uint32_t i;
uint32_t lost = 0u;
udps_frame_t frame;
if (c->num_sigs == 0u) {
return 0; /* DATA before CONFIG: nothing to decode against. */
}
/* Order the sequence before spending anything on the payload.
*
* Reassembly completes in arrival order, not counter order, so a packet
* delayed or duplicated on the wire surfaces after a newer one has already
* been delivered. Its samples carry an older time base: they land on top
* of data the consumer already has and leave the span they should have
* filled empty. Nothing in the payload distinguishes such a packet from a
* good one, only the counter does.
*
* The counter is a wrapping uint32, so it is ordered by the signed
* difference; comparing the values directly would call the first packet
* after the wrap stale and reject the stream from then on. */
if (c->have_counter) {
int32_t delta = (int32_t)(counter - c->last_counter);
if (delta <= 0) {
c->stats.stale_packets++;
return 0;
}
lost = (uint32_t)delta - 1u;
c->stats.counter_gaps += lost;
}
c->last_counter = counter;
c->have_counter = 1;
if (len < 8u) {
return fail(c, "DATA payload too short (%lu bytes)", (unsigned long)len);
}
@@ -528,15 +553,11 @@ static int decode_data(udps_client_t *c, const uint8_t *pl, size_t len,
written += count;
}
if (c->have_counter && counter > c->last_counter + 1u) {
c->stats.counter_gaps += counter - c->last_counter - 1u;
}
c->last_counter = counter;
c->have_counter = 1;
c->stats.frames_delivered++;
if (c->on_data != NULL) {
frame.counter = counter;
frame.lost = lost;
frame.hrt = rd_u64(pl);
frame.recv_time = recv_time;
frame.publish_mode = c->publish_mode;
+16
View File
@@ -140,6 +140,16 @@ typedef struct {
/** One fully decoded DATA packet. */
typedef struct {
uint32_t counter; /**< Packet counter; gaps mean lost datagrams. */
/**
* DATA packets missing immediately before this one, from the counter.
*
* Needed to space samples correctly: the elapsed time since the previous
* frame covers the lost packets' cycles too, so dividing it by this
* frame's sample count alone gives a period too long by exactly
* @c lost + 1, which walks the samples past their own end and into the
* range the next frame claims.
*/
uint32_t lost;
uint64_t hrt; /**< Producer's high-resolution timer at send. */
double recv_time; /**< Wall-clock seconds (CLOCK_REALTIME) at arrival. */
uint8_t publish_mode; /**< UDPS_PUBLISH_*. */
@@ -164,6 +174,12 @@ typedef struct {
uint64_t config_updates;
uint64_t fragments_dropped; /**< Duplicate, stale or unplaceable fragments. */
uint64_t counter_gaps; /**< DATA packets missing from the sequence. */
/**
* DATA packets dropped for not advancing the counter: reordered or
* duplicated on the wire. Delivering one would stamp its values with a
* time base older than data already handed over.
*/
uint64_t stale_packets;
uint64_t reconnects;
} udps_stats_t;
+14 -2
View File
@@ -1,7 +1,19 @@
module marte2/common
go 1.21
go 1.24.9
require github.com/gorilla/websocket v1.5.1
require golang.org/x/net v0.17.0 // indirect
require (
github.com/andybalholm/brotli v1.1.1 // indirect
github.com/google/uuid v1.6.0 // indirect
github.com/klauspost/compress v1.17.9 // indirect
github.com/parquet-go/bitpack v1.0.0 // indirect
github.com/parquet-go/jsonlite v1.0.0 // indirect
github.com/parquet-go/parquet-go v0.32.0 // indirect
github.com/pierrec/lz4/v4 v4.1.21 // indirect
github.com/twpayne/go-geom v1.6.1 // indirect
golang.org/x/net v0.17.0 // indirect
golang.org/x/sys v0.38.0 // indirect
google.golang.org/protobuf v1.34.2 // indirect
)
+21
View File
@@ -1,4 +1,25 @@
github.com/andybalholm/brotli v1.1.1 h1:PR2pgnyFznKEugtsUo0xLdDop5SKXd5Qf5ysW+7XdTA=
github.com/andybalholm/brotli v1.1.1/go.mod h1:05ib4cKhjx3OQYUY22hTVd34Bc8upXjOLL2rKwwZBoA=
github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0=
github.com/google/uuid v1.6.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
github.com/gorilla/websocket v1.5.1 h1:gmztn0JnHVt9JZquRuzLw3g4wouNVzKL15iLr/zn/QY=
github.com/gorilla/websocket v1.5.1/go.mod h1:x3kM2JMyaluk02fnUJpQuwD2dCS5NDG2ZHL0uE0tcaY=
github.com/klauspost/compress v1.17.9 h1:6KIumPrER1LHsvBVuDa0r5xaG0Es51mhhB9BQB2qeMA=
github.com/klauspost/compress v1.17.9/go.mod h1:Di0epgTjJY877eYKx5yC51cX2A2Vl2ibi7bDH9ttBbw=
github.com/parquet-go/bitpack v1.0.0 h1:AUqzlKzPPXf2bCdjfj4sTeacrUwsT7NlcYDMUQxPcQA=
github.com/parquet-go/bitpack v1.0.0/go.mod h1:XnVk9TH+O40eOOmvpAVZ7K2ocQFrQwysLMnc6M/8lgs=
github.com/parquet-go/jsonlite v1.0.0 h1:87QNdi56wOfsE5bdgas0vRzHPxfJgzrXGml1zZdd7VU=
github.com/parquet-go/jsonlite v1.0.0/go.mod h1:nDjpkpL4EOtqs6NQugUsi0Rleq9sW/OtC1NnZEnxzF0=
github.com/parquet-go/parquet-go v0.32.0 h1:NWDqTUHfrCS4cJP/Fj2HlxvqsrVedWG3sayMkf+znzM=
github.com/parquet-go/parquet-go v0.32.0/go.mod h1:navtkAYr2LGoJVp141oXPlO/sxLvaOe3la2JEoD8+rg=
github.com/pierrec/lz4/v4 v4.1.21 h1:yOVMLb6qSIDP67pl/5F7RepeKYu/VmTyEXvuMI5d9mQ=
github.com/pierrec/lz4/v4 v4.1.21/go.mod h1:gZWDp/Ze/IJXGXf23ltt2EXimqmTUXEy0GFuRQyBid4=
github.com/twpayne/go-geom v1.6.1 h1:iLE+Opv0Ihm/ABIcvQFGIiFBXd76oBIar9drAwHFhR4=
github.com/twpayne/go-geom v1.6.1/go.mod h1:Kr+Nly6BswFsKM5sd31YaoWS5PeDDH2NftJTK7Gd028=
github.com/xyproto/randomstring v1.0.5/go.mod h1:rgmS5DeNXLivK7YprL0pY+lTuhNQW3iGxZ18UQApw/E=
golang.org/x/net v0.17.0 h1:pVaXccu2ozPjCXewfr1S7xza/zcXTity9cCdXQYSjIM=
golang.org/x/net v0.17.0/go.mod h1:NxSsAGuq816PNPmqtQdLE42eU2Fs7NoRIZrHJAlaCOE=
golang.org/x/sys v0.38.0 h1:3yZWxaJjBmCWXqhN1qh02AkOnCQ1poK6oF+a7xWL6Gc=
golang.org/x/sys v0.38.0/go.mod h1:OgkHotnGiDImocRcuBABYBEXf8A9a87e/uXjp9XT3ks=
google.golang.org/protobuf v1.34.2 h1:6xV6lTsCfpGD21XK49h7MhtcApnLqkfYgPcdHftf6hg=
google.golang.org/protobuf v1.34.2/go.mod h1:qYOHts0dSfpeUzUFpOMr/WGzszTmLH+DiWniOlNbLDw=
@@ -0,0 +1,146 @@
package udpsprotocol
// Accumulate mode ships one full snapshot of EVERY signal per RT cycle —
// arrays included. See UDPStreamer.cpp pass 5 ("ALL signals (scalars and
// arrays alike) are tagged accumulated = true") and SerializeAccumulated,
// which writes, for each signal in CONFIG order, numSamples consecutive
// snapshots of that signal's full element set.
//
// The tests below build a payload byte-for-byte the way the C++ producer
// does, so a decoding regression shows up here rather than as a mangled
// waveform three components downstream.
import (
"encoding/binary"
"math"
"testing"
"time"
)
// buildAccumulatePayload lays out an Accumulate DATA payload exactly as
// UDPStreamer::SerializeAccumulated does:
//
// [8 HRT][4 numSamples] then, per signal, numSamples × NumElements float64.
//
// slots[i][k] holds signal i's element set for cycle k.
func buildAccumulatePayload(hrt uint64, slots [][][]float64) []byte {
numSamples := 0
if len(slots) > 0 {
numSamples = len(slots[0])
}
out := make([]byte, 12)
binary.LittleEndian.PutUint64(out[0:8], hrt)
binary.LittleEndian.PutUint32(out[8:12], uint32(numSamples))
for _, sig := range slots {
for _, elems := range sig {
for _, v := range elems {
var b [8]byte
binary.LittleEndian.PutUint64(b[:], math.Float64bits(v))
out = append(out, b[:]...)
}
}
}
return out
}
// TestParseDataAccumulateGivesEachSlotItsOwnArray pins the array case: with an
// accumulated batch, slot k's array signal must decode to the values the
// producer captured on cycle k, not to some other cycle's. Handing every slot
// slot 0's array would stamp one cycle's data with every slot's timestamp —
// the same samples drawn repeatedly at advancing times, with the cycles they
// displaced missing entirely.
func TestParseDataAccumulateGivesEachSlotItsOwnArray(t *testing.T) {
sigs := []SignalInfo{
{Name: "Time", TypeCode: 9, NumRows: 1, NumCols: 1, QuantType: QuantNone},
{Name: "Wave", TypeCode: 9, NumRows: 4, NumCols: 1, QuantType: QuantNone},
}
// Three RT cycles. "Wave" carries a different ramp each cycle so a
// mix-up is unambiguous.
timeSlots := [][]float64{{10}, {20}, {30}}
waveSlots := [][]float64{
{1, 2, 3, 4},
{5, 6, 7, 8},
{9, 10, 11, 12},
}
payload := buildAccumulatePayload(777, [][][]float64{timeSlots, waveSlots})
samples, err := ParseData(payload, sigs, PublishModeAccumulate, time.Now())
if err != nil {
t.Fatalf("ParseData: %v", err)
}
if len(samples) != 3 {
t.Fatalf("expected 3 slots, got %d", len(samples))
}
for k, s := range samples {
got := s.Values["Wave"]
want := waveSlots[k]
if len(got) != len(want) {
t.Fatalf("slot %d: Wave has %d elements, want %d", k, len(got), len(want))
}
for i := range want {
if got[i] != want[i] {
t.Fatalf("slot %d: Wave = %v, want %v (slot %d's data has been "+
"served for this slot's timestamp)", k, got, want,
indexOfSlot(waveSlots, got))
}
}
if tv := s.Values["Time"]; len(tv) != 1 || tv[0] != timeSlots[k][0] {
t.Fatalf("slot %d: Time = %v, want %v", k, tv, timeSlots[k])
}
}
}
// TestParseDataAccumulateConsumesTheWholeArrayBlock catches the same defect
// from the other side: a signal following an array must be read at the right
// offset. Under-reading the array block slides every later signal backwards
// into the array's tail, which decodes as plausible-looking but wrong values
// rather than as an error.
func TestParseDataAccumulateConsumesTheWholeArrayBlock(t *testing.T) {
sigs := []SignalInfo{
{Name: "Wave", TypeCode: 9, NumRows: 4, NumCols: 1, QuantType: QuantNone},
{Name: "Tail", TypeCode: 9, NumRows: 1, NumCols: 1, QuantType: QuantNone},
}
waveSlots := [][]float64{
{1, 2, 3, 4},
{5, 6, 7, 8},
{9, 10, 11, 12},
}
tailSlots := [][]float64{{100}, {200}, {300}}
payload := buildAccumulatePayload(0, [][][]float64{waveSlots, tailSlots})
samples, err := ParseData(payload, sigs, PublishModeAccumulate, time.Now())
if err != nil {
t.Fatalf("ParseData: %v", err)
}
if len(samples) != 3 {
t.Fatalf("expected 3 slots, got %d", len(samples))
}
for k, s := range samples {
tv := s.Values["Tail"]
if len(tv) != 1 || tv[0] != tailSlots[k][0] {
t.Fatalf("slot %d: Tail = %v, want %v — the array block before it "+
"was not fully consumed", k, tv, tailSlots[k])
}
}
}
// indexOfSlot reports which slot's data a decoded array actually matches, so a
// failure message can name the culprit instead of just showing numbers.
func indexOfSlot(slots [][]float64, got []float64) int {
for k, want := range slots {
if len(want) != len(got) {
continue
}
same := true
for i := range want {
if want[i] != got[i] {
same = false
break
}
}
if same {
return k
}
}
return -1
}
+33 -28
View File
@@ -290,28 +290,37 @@ type DataSample struct {
HRTTimestamp uint64
WallTime time.Time // wall-clock time at UDP arrival; used as x-axis
Values map[string][]float64 // key = signal name, value = []float64 with NumElements entries
// Lost is the number of DATA packets missing between the previous sample
// and this one, taken from the producer's packet counter (see
// SequenceGate). Consumers that derive a per-element period from the
// inter-packet gap need it: the gap widens with every lost packet, and
// dividing it by this packet's element count alone reports a period too
// long by exactly that factor — which walks the packet's elements past
// their own end and into the range the next packet claims.
Lost uint32
}
// parseElems reads n elements for sig from payload at offset, advancing offset.
// Returns the slice of float64 values and the new offset.
func parseElems(payload []byte, offset, n int, sig SignalInfo) ([]float64, int, error) {
elems := make([]float64, n)
if sig.QuantType == QuantNone {
sz := rawTypeSize(sig.TypeCode)
needed := n * sz
if offset+needed > len(payload) {
if sig.QuantType != QuantNone {
sz = quantSize(sig.QuantType)
}
// Bounds-check before allocating. In Accumulate mode n is numSamples ×
// NumElements, so a malformed packet could otherwise ask for an allocation
// far larger than its own payload could ever justify.
if n < 0 || n > (len(payload)-offset)/sz {
return nil, offset, fmt.Errorf("data payload truncated for signal %q", sig.Name)
}
elems := make([]float64, n)
needed := n * sz
if sig.QuantType == QuantNone {
for i := 0; i < n; i++ {
elems[i] = readRawElement(payload, offset+i*sz, sig.TypeCode)
}
offset += needed
} else {
sz := quantSize(sig.QuantType)
needed := n * sz
if offset+needed > len(payload) {
return nil, offset, fmt.Errorf("data payload truncated (quant) for signal %q", sig.Name)
}
for i := 0; i < n; i++ {
var raw uint16
if sz == 1 {
@@ -331,7 +340,13 @@ func parseElems(payload []byte, offset, n int, sig SignalInfo) ([]float64, int,
//
// For PublishModeAccumulate the payload format is:
//
// [8 HRT][4 numSamples][for each signal: accumulated scalars → numSamples elems; arrays → NumElements elems]
// [8 HRT][4 numSamples][for each signal: numSamples × NumElements elems]
//
// Every signal is accumulated, arrays included: the producer captures one full
// snapshot of the whole signal set per RT cycle and lays the cycles out
// contiguously per signal (UDPStreamer::SerializeAccumulated). Reading only
// NumElements for an array would hand every slot the first cycle's data and
// slide all later signals into that array's tail.
//
// The function returns one DataSample per accumulated snapshot so the hub can
// process each slot independently with its own timestamp.
@@ -357,28 +372,18 @@ func ParseData(payload []byte, sigs []SignalInfo, publishMode uint8, arrivalTime
}
// Parse per-signal data blocks (all slots for a signal are contiguous).
accumVals := make(map[string][]float64, len(sigs)) // scalars: numSamples values
fixedVals := make(map[string][]float64, len(sigs)) // arrays: NumElements values
accumVals := make(map[string][]float64, len(sigs)) // numSamples × NumElements
accumElems := make(map[string]int, len(sigs))
for _, sig := range sigs {
n := sig.NumElements()
if n == 1 {
// Accumulated scalar: read numSamples back-to-back elements.
elems, newOff, err := parseElems(payload, offset, numSamples, sig)
elems, newOff, err := parseElems(payload, offset, numSamples*n, sig)
if err != nil {
return nil, err
}
offset = newOff
accumVals[sig.Name] = elems
} else {
// Fixed array (non-accumulated): one set of NumElements values.
elems, newOff, err := parseElems(payload, offset, n, sig)
if err != nil {
return nil, err
}
offset = newOff
fixedVals[sig.Name] = elems
}
accumElems[sig.Name] = n
}
// Build one DataSample per slot.
@@ -386,10 +391,10 @@ func ParseData(payload []byte, sigs []SignalInfo, publishMode uint8, arrivalTime
for k := 0; k < numSamples; k++ {
vals := make(map[string][]float64, len(sigs))
for sigName, av := range accumVals {
vals[sigName] = []float64{av[k]}
}
for sigName, fv := range fixedVals {
vals[sigName] = fv // shared read-only reference; hub does not modify
n := accumElems[sigName]
// Sub-slice of the decoded block; the hub treats values as
// read-only, so no copy is needed.
vals[sigName] = av[k*n : (k+1)*n : (k+1)*n]
}
samples[k] = DataSample{HRTTimestamp: hrt, WallTime: arrivalTime, Values: vals}
}
+49
View File
@@ -0,0 +1,49 @@
package udpsprotocol
// SequenceGate orders DATA packets by the producer's packet counter.
//
// Reassembly completes in arrival order, not counter order, so a packet that
// was delayed or duplicated on the wire is handed up after a newer one has
// already been consumed. Its samples then carry an older time base than the
// data already in the ring: they land on top of samples that are already
// there, and the span they should have filled stays empty. That is a hole on
// one side and a collision on the other, from a packet that is entirely
// well-formed — the counter is the only thing that distinguishes it.
//
// A SequenceGate is not safe for concurrent use; each receive loop owns one.
type SequenceGate struct {
last uint32
valid bool
// Stale counts packets rejected for not advancing the counter (reordered
// or duplicated), for diagnostics.
Stale uint64
}
// Reset forgets the sequence. Call it on (re)connect: the producer's counter
// restarts independently of ours, so a counter carried over from the previous
// connection would reject the whole new stream.
func (g *SequenceGate) Reset() {
g.last = 0
g.valid = false
}
// Accept reports whether a DATA packet with this counter should be delivered,
// and how many packets went missing immediately before it.
//
// The counter is a wrapping uint32, so ordering is done on the signed
// difference: a plain comparison would call the first packet after the wrap
// stale and reject everything from then on.
func (g *SequenceGate) Accept(counter uint32) (ok bool, lost uint32) {
if !g.valid {
g.valid = true
g.last = counter
return true, 0
}
delta := int32(counter - g.last)
if delta <= 0 {
g.Stale++
return false, 0
}
g.last = counter
return true, uint32(delta) - 1
}
@@ -0,0 +1,82 @@
package udpsprotocol
import "testing"
// A packet older than one already delivered carries an older time base. Its
// samples land on top of data that is already in the ring and leave the span
// they should have filled empty, so it must not get through.
func TestSequenceGateRejectsStaleAndDuplicate(t *testing.T) {
var g SequenceGate
if ok, lost := g.Accept(10); !ok || lost != 0 {
t.Fatalf("first packet: got (%v, %d), want (true, 0)", ok, lost)
}
if ok, _ := g.Accept(11); !ok {
t.Fatal("counter 11 advances past 10 and must be accepted")
}
if ok, _ := g.Accept(9); ok {
t.Error("counter 9 is older than the delivered 11 and must be dropped")
}
if ok, _ := g.Accept(11); ok {
t.Error("a repeat of the delivered counter must be dropped")
}
if g.Stale != 2 {
t.Errorf("Stale = %d, want 2", g.Stale)
}
// The rejections must not have moved the sequence on.
if ok, lost := g.Accept(12); !ok || lost != 0 {
t.Errorf("after rejections: got (%v, %d), want (true, 0)", ok, lost)
}
}
// The loss count is what lets a consumer tell a widened gap from a slowed
// producer, so it must exclude the packet being delivered and must not persist
// into the next one.
func TestSequenceGateReportsLoss(t *testing.T) {
var g SequenceGate
g.Accept(100)
if _, lost := g.Accept(104); lost != 3 {
t.Errorf("101..103 missing: lost = %d, want 3", lost)
}
if _, lost := g.Accept(105); lost != 0 {
t.Errorf("consecutive packet: lost = %d, want 0", lost)
}
if g.Stale != 0 {
t.Errorf("Stale = %d, want 0", g.Stale)
}
}
// The counter is a wrapping uint32. Ordering it by plain comparison would call
// every packet after the wrap older than 0xFFFFFFFF and kill the stream.
func TestSequenceGateSurvivesWraparound(t *testing.T) {
var g SequenceGate
for _, c := range []uint32{0xFFFFFFFD, 0xFFFFFFFE, 0xFFFFFFFF, 0, 1, 2} {
ok, lost := g.Accept(c)
if !ok {
t.Fatalf("counter %#x rejected across the wrap", c)
}
if lost != 0 {
t.Errorf("counter %#x: lost = %d, want 0", c, lost)
}
}
// Loss must still be measured correctly across the wrap.
var h SequenceGate
h.Accept(0xFFFFFFFE)
if _, lost := h.Accept(1); lost != 2 {
t.Errorf("0xFFFFFFFF and 0 missing: lost = %d, want 2", lost)
}
}
// A reconnect restarts the producer's counter independently of ours; a carried
// over counter would reject the entire new stream.
func TestSequenceGateResetAcceptsLowerCounter(t *testing.T) {
var g SequenceGate
g.Accept(5000)
g.Reset()
if ok, lost := g.Accept(3); !ok || lost != 0 {
t.Errorf("after Reset: got (%v, %d), want (true, 0)", ok, lost)
}
}
@@ -0,0 +1,75 @@
package wshub
import (
"math"
"testing"
)
// A short window at a high sample rate fits in a ring's initial capacity, so the
// retune sweep used to leave it there — and a ring holding exactly the window has
// already rolled past the front of a capture by the time that capture is read,
// which happens a post-window plus captureMarginSec after the trigger fires.
//
// 1 MSps over a 200 ms window: 200 k points fit in the 250 k initial ring, and
// every shot came back missing its first 123 ms.
func TestCaptureWholeAtHighRateShortWindow(t *testing.T) {
const (
key = "s1:Ch1"
rate = 1e6
window = 0.2
prePct = 20.0
batchSec = 1.0 / 30.0
simSec = 6.0
)
h := NewHub()
h.SetRingBudget(defaultRingPts)
h.rings[key] = newSigRing(ringCapInitial)
h.trigger.SetConfig(trigConfig{signalKey: key, edge: "rising", threshold: 0,
windowSec: window, prePercent: prePct, mode: "normal", holdoffSec: 0.2})
rateHz := float64(rate)
nBatch := int(rateHz * batchSec)
ts := make([]float64, nBatch)
vs := make([]float64, nBatch)
armed, shots := false, 0
for now := 0.0; now < simSec; now += batchSec {
for i := range ts {
ts[i] = now + float64(i)/rateHz
vs[i] = math.Sin(2 * math.Pi * 5 * ts[i]) // a rising crossing every 200 ms
}
h.ingest(key, 1, ts, vs)
h.retuneRings(now)
h.refreshTriggerFill()
if !armed && now > 2 {
h.trigger.Arm()
armed = true
}
trigTime, pre, post, ok := h.trigger.dueCapture(now + batchSec)
if !ok {
if h.trigger.dueRearm(now + batchSec) {
h.trigger.Arm()
}
continue
}
t0 := trigTime - pre
buf := h.buildTriggerCapture(trigTime, pre, post)
if buf == nil {
t.Fatalf("shot at t=%.4f produced no frame at all", trigTime)
}
first, last, n := decodeCaptureSpan(t, buf, key)
shots++
if lost := first - t0; lost > shortCaptureTol*window {
_, span := h.rings[key].stats()
t.Errorf("shot at t=%.4f is missing %.0f ms at the front of its %.0f ms window "+
"(got [%.4f,%.4f], %d pts; ring holds %.4f s in %d points)",
trigTime, 1e3*lost, 1e3*window, first, last, n, span, h.rings[key].capacity())
}
h.trigger.markTriggered(now + batchSec)
}
if shots < 3 {
t.Fatalf("only %d shots in %.0f s", shots, simSec)
}
}
+119
View File
@@ -0,0 +1,119 @@
package wshub
import (
"fmt"
"net/http"
"sort"
"strconv"
"strings"
"time"
"github.com/parquet-go/parquet-go"
)
// ExportSample is one row of the binary export: a single stored sample, in
// long ("tidy") form, keyed by source and signal with its own timestamp.
//
// Keeping each signal's samples as its own rows — rather than resampling onto a
// shared time grid — is what makes the export hole-free: per-signal streams of
// different lengths export exactly as stored, nothing is fabricated, and
// nothing is dropped.
type ExportSample struct {
Source string `parquet:"source"`
Signal string `parquet:"signal"`
Time float64 `parquet:"time"`
Value float64 `parquet:"value"`
}
// exportChunkRows bounds each batched write and, via MaxRowsPerRowGroup, the
// size of each parquet row group: memory stays bounded however large the
// export is, because a finished row group is flushed to the HTTP stream.
const exportChunkRows = 65536
// exportWriteBuffer is the parquet writer's output buffer: larger than the
// 32KiB default means fewer writes on the HTTP stream for a multi-GB export.
const exportWriteBuffer = 1 << 20
// HandleExport serves GET /api/export?t0=..&t1=..[&signals=a,b] as a Parquet
// file containing every stored sample of the named signals in [t0, t1].
//
// Unlike /api/zoom there is no decimation: the file holds the full contents of
// the rings. At rates above the ring budget those contents are min/max buckets
// (the finest resolution the hub retains); at lower rates they are verbatim.
func (h *Hub) HandleExport(w http.ResponseWriter, r *http.Request) {
q := r.URL.Query()
t0, err0 := strconv.ParseFloat(q.Get("t0"), 64)
t1, err1 := strconv.ParseFloat(q.Get("t1"), 64)
if err0 != nil || err1 != nil || t1 <= t0 {
http.Error(w, "invalid t0/t1", http.StatusBadRequest)
return
}
var keys []string
if s := strings.TrimSpace(q.Get("signals")); s != "" {
keys = strings.Split(s, ",")
for i := range keys {
keys[i] = strings.TrimSpace(keys[i])
}
}
// Snapshot the rings we will read. A signal removed mid-export must not
// silently drop rows from the file.
h.ringsMu.RLock()
refs := make(map[string]*sigRing)
if keys == nil {
for k, rb := range h.rings {
refs[k] = rb
}
} else {
for _, k := range keys {
if rb, ok := h.rings[k]; ok {
refs[k] = rb
}
}
}
h.ringsMu.RUnlock()
if len(refs) == 0 {
http.Error(w, "no signals", http.StatusNotFound)
return
}
// Deterministic column order.
names := make([]string, 0, len(refs))
for k := range refs {
names = append(names, k)
}
sort.Strings(names)
w.Header().Set("Content-Type", "application/vnd.apache.parquet")
w.Header().Set("Content-Disposition",
fmt.Sprintf("attachment; filename=\"signals_%d.parquet\"", time.Now().Unix()))
writer := parquet.NewGenericWriter[ExportSample](w,
parquet.MaxRowsPerRowGroup(exportChunkRows),
parquet.WriteBufferSize(exportWriteBuffer),
)
batch := make([]ExportSample, 0, exportChunkRows)
for _, key := range names {
st, sv := refs[key].slice(t0, t1)
colon := strings.IndexByte(key, ':')
source, signal := key, key
if colon >= 0 {
source = key[:colon]
signal = key[colon+1:]
}
for i := range st {
batch = append(batch, ExportSample{Source: source, Signal: signal, Time: st[i], Value: sv[i]})
if len(batch) >= exportChunkRows {
if _, err := writer.Write(batch); err != nil {
// Client went away or the stream broke; stop writing.
return
}
batch = batch[:0]
}
}
}
if len(batch) > 0 {
_, _ = writer.Write(batch)
}
_ = writer.Close()
}
+87
View File
@@ -0,0 +1,87 @@
package wshub
import (
"bytes"
"net/http/httptest"
"testing"
"github.com/parquet-go/parquet-go"
)
func TestHandleExportParquetFullResolution(t *testing.T) {
h := NewHub()
// Two signals with different lengths and offset time bases: the export must
// keep every sample of each, on its own timestamps (no holes, no
// resampling, no decimation).
sig1 := newSigRing(10000)
sig2 := newSigRing(10000)
t1, v1 := make([]float64, 1000), make([]float64, 1000)
for i := range t1 {
t1[i] = float64(i) * 0.001
v1[i] = float64(i) * 2
}
sig1.write(t1, v1)
t2, v2 := make([]float64, 500), make([]float64, 500)
for i := range t2 {
t2[i] = 0.1 + float64(i)*0.002
v2[i] = -float64(i)
}
sig2.write(t2, v2)
h.rings["s1:Ch1"] = sig1
h.rings["s1:Ch2"] = sig2
req := httptest.NewRequest("GET", "/api/export?t0=0&t1=2&signals=s1:Ch1,s1:Ch2", nil)
rec := httptest.NewRecorder()
h.HandleExport(rec, req)
if rec.Code != 200 {
t.Fatalf("status = %d, want 200 (body: %s)", rec.Code, rec.Body.String())
}
reader := parquet.NewGenericReader[ExportSample](bytes.NewReader(rec.Body.Bytes()))
defer reader.Close()
var got []ExportSample
buf := make([]ExportSample, 1000)
for {
n, err := reader.Read(buf)
got = append(got, buf[:n]...)
if err != nil {
break
}
}
if len(got) != 1500 {
t.Fatalf("rows = %d, want 1500 (every sample of both signals)", len(got))
}
ch1 := filterExportSamples(got, "s1", "Ch1")
ch2 := filterExportSamples(got, "s1", "Ch2")
if len(ch1) != 1000 || len(ch2) != 500 {
t.Fatalf("ch1=%d ch2=%d rows, want 1000/500 (no holes, no resampling)", len(ch1), len(ch2))
}
if ch1[0].Time != 0 || ch1[0].Value != 0 || ch1[999].Time != 0.999 || ch1[999].Value != 1998 {
t.Fatalf("ch1 endpoints wrong: first=%+v last=%+v", ch1[0], ch1[999])
}
if ch2[0].Time != 0.1 || ch2[499].Time != 0.1+499*0.002 || ch2[499].Value != -499 {
t.Fatalf("ch2 endpoints wrong: first=%+v last=%+v", ch2[0], ch2[499])
}
}
func filterExportSamples(rows []ExportSample, source, signal string) []ExportSample {
out := make([]ExportSample, 0, len(rows))
for _, r := range rows {
if r.Source == source && r.Signal == signal {
out = append(out, r)
}
}
return out
}
func TestHandleExportParquetBadRange(t *testing.T) {
h := NewHub()
h.rings["s1:Ch1"] = newSigRing(10)
req := httptest.NewRequest("GET", "/api/export?t0=2&t1=1", nil)
rec := httptest.NewRecorder()
h.HandleExport(rec, req)
if rec.Code != 400 {
t.Fatalf("status = %d, want 400 for inverted range", rec.Code)
}
}
+26 -17
View File
@@ -422,6 +422,26 @@ func (hw *historyWriter) window() float64 {
return hw.windowSec
}
// coversWindow reports whether the archive file for key currently spans at
// least sec seconds. When true, backfillCaptureHead can reconstruct a capture's
// front out of the archive, so the trigger need not wait for the ring to cover
// the whole window on its own.
func (hw *historyWriter) coversWindow(key string, sec float64) bool {
if !hw.enabled() || !(sec > 0) {
return false
}
hw.mu.RLock()
hf, ok := hw.files[key]
hw.mu.RUnlock()
if !ok {
return false
}
hf.mu.RLock()
span := hf.tNewest - hf.tOldest
hf.mu.RUnlock()
return span >= sec
}
// setWindow points the archive at the timespan the clients are looking at, and
// re-sizes the files that no longer match it. It reports whether any file's
// geometry changed, which invalidates what clients know about the archive.
@@ -842,10 +862,7 @@ func (hf *histFile) readAfter(after, t0, t1 float64, max int) ([]byte, float64,
// The run wraps at most once, so it costs at most two reads.
buf := make([]byte, n*histPairSize)
start := (oldest + lo) % capacity
head := int(capacity-start) * histPairSize
if head > len(buf) {
head = len(buf)
}
head := min(int(capacity-start)*histPairSize, len(buf))
if _, err := hf.f.ReadAt(buf[:head], int64(histHeaderSize)+int64(start)*histPairSize); err != nil {
return nil, 0, err
}
@@ -898,10 +915,8 @@ func (hf *histFile) writePairs(t, v []float64) error {
binary.LittleEndian.PutUint64(buf[i*histPairSize+8:], math.Float64bits(v[i]))
}
first := int(hf.capacity - hf.head)
if first > n {
first = n
}
first := min(int(hf.capacity-hf.head), n)
off := int64(histHeaderSize) + int64(hf.head)*histPairSize
if _, err := hf.f.WriteAt(buf[:first*histPairSize], off); err != nil {
return err
@@ -1089,10 +1104,7 @@ func (hw *historyWriter) readRange(key string, t0, t1 float64, maxOut int) ([]fl
// Read in contiguous runs: the range wraps at most once.
buf := make([]byte, n*histPairSize)
start := (oldest + lo) % capacity
first := int(capacity - start)
if first > n {
first = n
}
first := min(int(capacity-start), n)
if _, err := hf.f.ReadAt(buf[:first*histPairSize],
int64(histHeaderSize)+int64(start)*histPairSize); err != nil {
return nil, nil
@@ -1265,7 +1277,7 @@ func (h *Hub) handleSetHistoryBudget(env map[string]interface{}) {
// handleHistoryZoom answers a historyZoom request from disk. Same request and
// reply shape as "zoom", so clients can fall back to it transparently when a
// window reaches further back than the in-memory rings hold.
func (h *Hub) handleHistoryZoom(c *wsClient, env map[string]interface{}) {
func (h *Hub) handleHistoryZoom(c *wsClient, env map[string]any) {
if !h.hist.enabled() {
msg, _ := json.Marshal(map[string]any{
"type": "historyZoom", "reqId": env["reqId"],
@@ -1287,10 +1299,7 @@ func (h *Hub) handleHistoryZoom(c *wsClient, env map[string]interface{}) {
// oversampled relative to the plot's point budget and thinned afterwards.
// The cap keeps a request for "no decimation" over a multi-hour window from
// pulling the whole file into memory.
readCap := n * histReadOversample
if readCap > histMaxReadPoints {
readCap = histMaxReadPoints
}
readCap := min(n*histReadOversample, histDefaultMaxPoints)
signals := make(map[string]sigData)
for _, k := range strings.Split(sigCSV, ",") {
+12 -3
View File
@@ -1216,15 +1216,24 @@ func (h *Hub) buildBinaryDataMessageForSource(src *sourceHubState, batch []udpsp
wallNs := s.WallTime.UnixNano()
wallSec := float64(wallNs) / 1e9
var dtSec float64
// A gap spans the elements of every packet that went missing
// inside it as well as this packet's own, so the divisor has
// to widen with it. Without this a single loss halves the
// apparent rate and the elements overrun into the next
// packet's range. The loss count belongs to the packet the
// gap ends at.
if bi+1 < len(batch) {
// Two consecutive packets in this tick → exact dt.
dtSec = (float64(batch[bi+1].WallTime.UnixNano()) - float64(wallNs)) / 1e9 / float64(n)
span := float64(n) * float64(1+batch[bi+1].Lost)
dtSec = (float64(batch[bi+1].WallTime.UnixNano()) - float64(wallNs)) / 1e9 / span
} else if bi > 0 {
// Last of multiple packets → use diff from previous.
dtSec = (float64(wallNs) - float64(batch[bi-1].WallTime.UnixNano())) / 1e9 / float64(n)
span := float64(n) * float64(1+s.Lost)
dtSec = (float64(wallNs) - float64(batch[bi-1].WallTime.UnixNano())) / 1e9 / span
} else if prevNs, ok2 := src.lastPktNs[sig.Name]; ok2 && prevNs > 0 && wallNs > prevNs {
// Single packet this tick → gap from the previous tick's packet.
dtSec = (float64(wallNs) - float64(prevNs)) / 1e9 / float64(n)
span := float64(n) * float64(1+s.Lost)
dtSec = (float64(wallNs) - float64(prevNs)) / 1e9 / span
} else {
// Truly first packet ever — inter-packet timing unknown.
// Skip to avoid poisoning the ring with wrongly-spaced timestamps;
@@ -0,0 +1,139 @@
package wshub
import (
"math"
"testing"
"time"
"marte2/common/udpsprotocol"
)
// pktSignal is a 4-element array with no time signal and no declared sampling
// rate, i.e. the TimeModePacket path where dt has to be inferred from the gap
// between packets.
func pktSignal(name string) udpsprotocol.SignalInfo {
return udpsprotocol.SignalInfo{
Name: name,
TypeCode: 8, // float64
NumDimensions: 1,
NumRows: 4,
NumCols: 1,
TimeMode: udpsprotocol.TimeModePacket,
TimeSignalIdx: udpsprotocol.NoTimeSignal,
}
}
// newPacketDtHub builds a Hub with a ring for one packet-timed array signal and
// returns both. It does not start Run(): buildBinaryDataMessageForSource is
// called directly so the timestamps it produces can be read back verbatim.
func newPacketDtHub(t *testing.T, sigName string) (*Hub, *sourceHubState, *sigRing) {
t.Helper()
h := NewHub()
src := &sourceHubState{
id: "s1",
signals: []udpsprotocol.SignalInfo{pktSignal(sigName)},
timeSigCalib: map[string]float64{},
lastPktNs: map[string]int64{},
lastFrameMeasured: map[string]float64{},
lastFrameEndT: map[string]float64{},
gapEMA: map[string]float64{},
}
rb := newSigRing(4096)
h.rings["s1:"+sigName] = rb
return h, src, rb
}
// packet builds a one-signal batch entry arriving at t0 with the given loss
// count; the values are irrelevant, only the timestamps are under test.
func packet(sigName string, at time.Time, lost uint32, n int) udpsprotocol.DataSample {
vals := make([]float64, n)
return udpsprotocol.DataSample{WallTime: at, Values: map[string][]float64{sigName: vals}, Lost: lost}
}
// ringTimes returns the timestamps written to the ring, in order.
func ringTimes(rb *sigRing) []float64 {
rb.mu.RLock()
defer rb.mu.RUnlock()
out := make([]float64, 0, rb.size)
start := (rb.head - rb.size + rb.cap) % rb.cap
for i := 0; i < rb.size; i++ {
out = append(out, rb.t[(start+i)%rb.cap])
}
return out
}
// A lost packet widens the inter-packet gap without adding elements to the
// packet that follows it. Dividing the gap by that packet's element count
// alone reports a period too long by exactly the number of packets missing,
// which walks the elements past their own end and into the range the next
// packet claims: they collide there, and the span they vacated stays empty.
func TestPacketDtIgnoresLostPacketWidening(t *testing.T) {
const sig = "Wave"
const n = 4
const dt = 1 * time.Millisecond
base := time.Unix(1700000000, 0)
h, src, rb := newPacketDtHub(t, sig)
// One clean packet establishes lastPktNs.
h.buildBinaryDataMessageForSource(src, []udpsprotocol.DataSample{
packet(sig, base, 0, n)})
// The next producer packet is lost, so the one after it arrives a full
// extra batch later and reports Lost=1.
arrival := base.Add(2 * n * dt)
h.buildBinaryDataMessageForSource(src, []udpsprotocol.DataSample{
packet(sig, arrival, 1, n)})
ts := ringTimes(rb)
if len(ts) != n {
t.Fatalf("ring holds %d points, want %d (the first packet is skipped: no gap yet)", len(ts), n)
}
got := ts[1] - ts[0]
if !nearSec(got, dt.Seconds()) {
t.Errorf("dt = %v s, want %v s (the gap spans two batches, not one)", got, dt.Seconds())
}
// Elements run forward from the packet's own arrival, so a doubled dt
// would stretch this batch across two batch periods and into the range the
// next packet claims.
span := ts[len(ts)-1] - ts[0]
if !nearSec(span, float64(n-1)*dt.Seconds()) {
t.Errorf("batch spans %v s, want %v s: it overruns into the next packet's range",
span, float64(n-1)*dt.Seconds())
}
}
// nearSec compares two intervals in seconds. The hub carries timestamps as
// float64 seconds derived from UnixNano, whose spacing near the current epoch
// is a couple of hundred nanoseconds, so exact equality is not available. The
// defect under test moves the period by a factor of two, three orders of
// magnitude outside this tolerance.
func nearSec(got, want float64) bool { return math.Abs(got-want) <= 1e-6 }
// The correction must be driven by the reported loss and nothing else: with no
// packet missing the period still comes straight from the gap, so a producer
// that genuinely slows down is followed rather than second-guessed.
func TestPacketDtFollowsGapWhenNothingIsLost(t *testing.T) {
const sig = "Wave"
const n = 4
base := time.Unix(1700000000, 0)
h, src, rb := newPacketDtHub(t, sig)
h.buildBinaryDataMessageForSource(src, []udpsprotocol.DataSample{
packet(sig, base, 0, n)})
// Same widened gap as the test above, but reported as no loss: the
// producer really is running at half the rate.
slowDt := 2 * time.Millisecond
h.buildBinaryDataMessageForSource(src, []udpsprotocol.DataSample{
packet(sig, base.Add(n*slowDt), 0, n)})
ts := ringTimes(rb)
if len(ts) != n {
t.Fatalf("ring holds %d points, want %d", len(ts), n)
}
if got := ts[1] - ts[0]; !nearSec(got, slowDt.Seconds()) {
t.Errorf("dt = %v s, want %v s: a real rate change must be followed", got, slowDt.Seconds())
}
}
+14 -1
View File
@@ -221,6 +221,19 @@ func ringCoverage(bucket, capacity int) int {
return capacity / 2 * bucket
}
// captureLagSec is how much further back than the window itself a ring has to
// reach to deliver a capture of it.
//
// A capture is not read out when its last sample arrives but captureMarginSec
// later, and then only on the next push tick — so by the time the window is
// extracted, its oldest sample is that much deeper in the ring. A ring holding
// exactly the window has already overwritten the front of its own capture, which
// is what made every shot at a short window come back missing its head. The
// pre/post split does not enter into it: the harvest is a post-window after the
// trigger and the read reaches a pre-window before it, so the two sum to the
// window whatever the split.
const captureLagSec = captureMarginSec + 1.0/30.0
// activeWindowSec is the timespan the buffers must cover. An armed trigger owns
// it: its pre-window has to already be in the ring when the trigger fires or
// there is nothing to back-fill the capture from. Otherwise it is the widest
@@ -228,7 +241,7 @@ func ringCoverage(bucket, capacity int) int {
func (h *Hub) activeWindowSec() float64 {
if h.trigger != nil && h.trigger.Active() {
if cfg := h.trigger.Config(); cfg.windowSec > 0 {
return cfg.windowSec
return cfg.windowSec + captureLagSec
}
}
widest := 0.0
+6 -3
View File
@@ -120,7 +120,9 @@ func TestActiveWindowSecTakesTheWidestClientWindow(t *testing.T) {
}
// An armed trigger owns the window: its pre-window has to be in the buffer
// before the trigger fires or the capture has nothing to back-fill from.
// before the trigger fires or the capture has nothing to back-fill from. The
// buffers must reach back past the window itself, because the capture is read
// out a margin and a tick after its last sample lands.
func TestActiveWindowSecPrefersTheArmedTrigger(t *testing.T) {
h := NewHub()
c := &wsClient{}
@@ -128,8 +130,9 @@ func TestActiveWindowSecPrefersTheArmedTrigger(t *testing.T) {
h.clients[c] = true
h.trigger.SetConfig(trigConfig{signalKey: "s1:sig", windowSec: 45, mode: "normal"})
if got := h.activeWindowSec(); got != 45 {
t.Fatalf("activeWindowSec = %v, want the trigger's 45", got)
if got := h.activeWindowSec(); got != 45+captureLagSec {
t.Fatalf("activeWindowSec = %v, want the trigger's 45 plus the %v harvest lag",
got, captureLagSec)
}
}
+22
View File
@@ -341,6 +341,9 @@ func (u *UDPClient) runSession() error {
}
reassembler := udpsprotocol.NewReassembler(2 * time.Second)
// Per-session: the producer's counter restarts independently of ours, so
// the gate must not carry a counter over from the previous connection.
var gate udpsprotocol.SequenceGate
buf := make([]byte, readBufSize)
var currentSigs []udpsprotocol.SignalInfo
var currentPublishMode uint8
@@ -414,11 +417,20 @@ func (u *UDPClient) runSession() error {
if len(currentSigs) == 0 {
continue
}
fresh, lost := gate.Accept(hdr.Counter)
if !fresh {
continue
}
samples, err := udpsprotocol.ParseData(complete, currentSigs, currentPublishMode, arrivalTime)
if err != nil {
log.Printf("[%s] udp: parse data: %v", u.sourceID, err)
continue
}
// The gap precedes the packet, so it belongs to its first slot only;
// the slots after it are consecutive cycles of the same batch.
if len(samples) > 0 {
samples[0].Lost = lost
}
for _, s := range samples {
u.hub.PushDataForSource(u.sourceID, s)
}
@@ -589,6 +601,9 @@ func (u *UDPClient) runMulticastSession() error {
}()
reassembler := udpsprotocol.NewReassembler(2 * time.Second)
// Per-session, as in runSession(): a counter from the previous connection
// would reject the whole new stream.
var gate udpsprotocol.SequenceGate
buf := make([]byte, readBufSize)
for {
@@ -629,11 +644,18 @@ func (u *UDPClient) runMulticastSession() error {
if len(currentSigs) == 0 {
continue
}
fresh, lost := gate.Accept(hdr.Counter)
if !fresh {
continue
}
samples, parseErr := udpsprotocol.ParseData(complete, currentSigs, currentPublishMode, arrivalTime)
if parseErr != nil {
log.Printf("[%s] multicast: parse data: %v", u.sourceID, parseErr)
continue
}
if len(samples) > 0 {
samples[0].Lost = lost
}
for _, s := range samples {
u.hub.PushDataForSource(u.sourceID, s)
}
+119 -13
View File
@@ -92,6 +92,14 @@ type triggerEngine struct {
bufGrowth float64
bufKnown bool
bufRateOK bool
// bufCoverage is the maximum span (seconds) the ring can reach at its
// current bucket and capacity — the gate must never demand more than this,
// or a ring whose coverage is below the window can never satisfy it. 0 =
// unknown (no measurable rate).
bufCoverage float64
// bufArchived is true when the disk history already spans the trigger
// window, so a short capture's front can be back-filled from it.
bufArchived bool
// Reference point the growth is measured against.
bufRefSpan, bufRefWall float64
@@ -108,6 +116,22 @@ type triggerEngine struct {
firedPost float64
firedValid bool
// The edge to fire on as soon as the FSM rearms, in sample time. Recorded
// while a capture is still being collected or handed out, for edges late
// enough that a capture of them would not overlap the one in flight.
//
// Without this the trigger is deaf from its own trigger point until the
// capture has been harvested — a post-window plus captureMarginSec — and
// then for the holdoff on top of that, and afterwards waits for a FRESH
// edge. On a sparse pulse train that rounds the capture spacing up to a
// whole pulse period: at the default 1 s window the blind stretch comes to
// 1.15 s, so a 1 Hz train was caught at 0.5 Hz and a wider window lost whole
// multiples. Remembering the edge instead makes the blind stretch exactly
// the post-window it has to be, since the capture is built from the edge's
// own timestamp and the ring still holds everything around it.
pendingT float64
pendingValid bool
rearmAt float64 // wall-clock seconds; 0 when no rearm is pending
}
@@ -163,10 +187,14 @@ func (te *triggerEngine) SetConfig(cfg trigConfig) {
if base != te.baseKey {
// The buffer measurement belongs to the old signal's ring.
te.bufKnown, te.bufRateOK = false, false
te.bufCoverage, te.bufArchived = 0, false
}
te.baseKey, te.elemIdx = base, idx
te.prevValid = false
te.prevValue = 0
// An edge held over from the old configuration would be latched against the
// new window, whose fill the gate has not vouched for.
te.pendingValid = false
}
func (te *triggerEngine) Config() trigConfig {
@@ -175,15 +203,37 @@ func (te *triggerEngine) Config() trigConfig {
return te.cfg
}
// Arm starts a fresh acquisition. It is the user's own arm, so it discards any
// edge remembered during the previous capture: the user asked for the next
// event, not for one that has already been and gone.
func (te *triggerEngine) Arm() {
te.mu.Lock()
te.state = trigArmed
te.prevValid = false
te.prevValue = 0
te.pendingValid = false
te.rearmAt = 0
te.mu.Unlock()
}
// rearm is the automatic arm at the end of a capture. Unlike Arm it honours an
// edge that arrived while the capture was being collected, firing on it at once
// rather than waiting for the next one — see pendingT. It also keeps the level
// tracked through the dead time, so the first sample after rearming is compared
// against its real predecessor instead of being spent seeding one.
func (te *triggerEngine) rearm() {
te.mu.Lock()
te.rearmAt = 0
if te.pendingValid {
t := te.pendingT
te.pendingValid = false
te.latchWindowLocked(t)
} else {
te.state = trigArmed
}
te.mu.Unlock()
}
func (te *triggerEngine) Disarm() {
te.mu.Lock()
te.state = trigIdle
@@ -191,6 +241,7 @@ func (te *triggerEngine) Disarm() {
te.prevValid = false
te.prevValue = 0
te.firedValid = false
te.pendingValid = false
te.rearmAt = 0
te.mu.Unlock()
}
@@ -243,13 +294,16 @@ const bufGrowthIntervalSec = 0.5
const bufGrowthSmooth = 0.5
// setBuffered records how far back the trigger signal's ring reaches, at wall
// clock now, and derives how fast that is growing. Pass known=false when there
// is no such ring.
func (te *triggerEngine) setBuffered(span float64, known bool, now float64) {
// clock now, and derives how fast that is growing. coverage is the maximum
// span (seconds) the ring can reach at its current bucket/capacity; archived
// says the disk history already spans the trigger window. Pass known=false when
// there is no ring to measure.
func (te *triggerEngine) setBuffered(span, coverage float64, archived, known bool, now float64) {
te.mu.Lock()
defer te.mu.Unlock()
if !known {
te.bufKnown, te.bufRateOK = false, false
te.bufCoverage, te.bufArchived = 0, false
return
}
if !te.bufKnown {
@@ -257,6 +311,8 @@ func (te *triggerEngine) setBuffered(span float64, known bool, now float64) {
te.bufRefSpan, te.bufRefWall = span, now
}
te.bufSpan = span
te.bufCoverage = coverage
te.bufArchived = archived
dt := now - te.bufRefWall
if dt < bufGrowthIntervalSec {
return
@@ -294,9 +350,17 @@ func (te *triggerEngine) setBuffered(span float64, known bool, now float64) {
// anyway. A full one grows only as fast as its incoming samples free space —
// re-bucketing to a longer window replaces dense old samples with sparse new
// ones — and it is that case, growth well below 1, where firing on the
// pre-window alone delivers a capture whose front has been overwritten by the
// time it is read. In the steady state growth is 0 and need is the whole
// window, which a ring tuned for that window already exceeds, so nothing waits.
//
// Two escapes keep an armed trigger from staying deaf forever:
//
// - archived — the disk history already spans the window, so the front of a
// capture can be back-filled from it; the ring only needs to
// hold the pre-window worth of recent data.
// - coverage — never demand more than the ring can physically reach. If its
// coverage saturates below the window (a measured source rate
// that over-estimates the true one), the gate opens once the
// ring is full anyway and a short capture is delivered instead
// of deafness.
func (te *triggerEngine) fillNeedLocked() float64 {
pre := te.cfg.windowSec * te.cfg.prePercent / 100
growth := 0.0 // until measured, assume the buffer will not fill on its own
@@ -307,6 +371,14 @@ func (te *triggerEngine) fillNeedLocked() float64 {
if need < pre {
need = pre
}
if te.bufArchived {
// The archive back-fills the front; the ring holds the post-trigger
// window live, so the pre-window is all it needs to have reached.
return pre
}
if te.bufCoverage > 0 && need > te.bufCoverage {
need = te.bufCoverage
}
return need
}
@@ -366,7 +438,11 @@ func (te *triggerEngine) feed(key string, nElem int, t, v []float64) {
te.lastT = t[len(t)-1]
te.lastTOK = true
te.lastFeedWall = float64(time.Now().UnixNano()) / 1e9
if te.state != trigArmed {
// A capture in flight does not stop the comparator; it only changes what an
// edge does. See pendingT.
inFlight := te.state == trigCollecting || te.state == trigTriggered
if te.state != trigArmed && !inFlight {
return
}
step, start := 1, 0
@@ -381,12 +457,20 @@ func (te *triggerEngine) feed(key string, nElem int, t, v []float64) {
// which is what made the first shot after a window change come back short.
// Track the level meanwhile, so the first edge once the buffer is deep
// enough is still measured against the right previous sample.
if te.fillLocked() < 1 {
if !inFlight && te.fillLocked() < 1 {
for i := start; i < len(v); i += step {
te.prevValue, te.prevValid = v[i], true
}
return
}
// The earliest trigger point a new capture may take. The one in flight owns
// everything up to the end of its own post-window, and the holdoff — a guard
// against re-triggering on the ringing of the SAME event — is measured from
// its trigger point too, so the two overlap rather than add.
notBefore := math.Inf(1)
if inFlight && te.firedValid {
notBefore = te.trigTime + math.Max(te.firedPost, te.cfg.holdoffSec)
}
thr := te.cfg.threshold
for i := start; i < len(t); i += step {
if !te.prevValid {
@@ -406,10 +490,19 @@ func (te *triggerEngine) feed(key string, nElem int, t, v []float64) {
default:
fired = up
}
if fired {
if !fired {
continue
}
if !inFlight {
te.latchWindowLocked(t[i])
return
}
// Keep the FIRST qualifying edge and go on tracking the level: a later
// one would be no more use, and stopping here would leave prevValue
// stale by the time the FSM rearms.
if !te.pendingValid && t[i] >= notBefore {
te.pendingT, te.pendingValid = t[i], true
}
}
}
@@ -592,19 +685,32 @@ func (h *Hub) refreshTriggerFill() {
return
}
now := float64(time.Now().UnixNano()) / 1e9
key := h.trigger.baseSignalKey()
var rb *sigRing
if key := h.trigger.baseSignalKey(); key != "" {
if key != "" {
rb = h.getRing(key)
}
if rb == nil {
// Nothing to measure. Do not gate on a signal the hub does not carry:
// that would leave the trigger armed forever, which is worse than a
// short capture.
h.trigger.setBuffered(0, false, now)
h.trigger.setBuffered(0, 0, false, false, now)
return
}
_, span := rb.stats()
h.trigger.setBuffered(span, true, now)
// Maximum span the ring can ever reach at its current bucket/capacity, in
// seconds. The gate must never demand more than this, or a ring whose
// coverage is below the window (a measured source rate that over-estimates
// the true one) can never satisfy it.
coverage := 0.0
if rate := rb.sourceRate(); rate > 0 {
coverage = float64(ringCoverage(rb.bucketSize(), rb.capacity())) / rate
}
// If the disk archive already spans the trigger window, the front of a
// short capture can be back-filled from it, so the ring need not cover the
// whole window on its own.
archived := h.hist.coversWindow(key, h.trigger.Config().windowSec)
h.trigger.setBuffered(span, coverage, archived, true, now)
}
// triggerTick services the trigger FSM; called from Hub.Run() on every push tick.
@@ -640,7 +746,7 @@ func (h *Hub) triggerTick() {
// file of its own, where nothing overwrites it until the next trigger.
h.hist.captureRange(trigTime-pre, trigTime+post)
} else if h.trigger.dueRearm(nowSec) {
h.trigger.Arm()
h.trigger.rearm()
}
if h.trigger.stateUnsent() {
@@ -0,0 +1,367 @@
package wshub
import (
"encoding/binary"
"math"
"testing"
)
/*
Sporadic-signal trigger coverage.
Every other trigger test in this package feeds a periodic waveform, or a
hand-built two-sample batch. Neither can show a trigger that is blind most of
the time: a sine crosses the threshold again a few milliseconds after every
missed crossing, so a trigger losing 80 % of its edges still fires steadily and
looks healthy. A sparse train 0000000111000000000000, one short burst in a
long flat run has nothing to fall back on, so every missed edge is a missed
capture and the yield is a direct measure of how long the FSM was deaf.
That deafness is what these tests pin down. It is not a bug in itself: a capture
cannot be harvested before the samples after its trigger point exist, so the
trigger is necessarily blind for its own post-trigger window. What must NOT
happen is for edges arriving after that window to be thrown away as well.
*/
// pulseTrainSim drives a Hub the way Run() does — ingest on one side, the
// trigger tick on the other — on a simulated clock.
type pulseTrainSim struct {
rateHz float64
batchSec float64
pulsePeriod float64
pulseSamples int
simSec float64
windowSec float64
prePercent float64
holdoffSec float64
armAt float64
// When windowChangeAt > 0 the window is switched to windowChangeTo at that
// time and the trigger re-armed, as a user editing the trigger bar would.
windowChangeAt float64
windowChangeTo float64
}
type pulseTrainResult struct {
pulses int // pulse starts presented after the trigger was armed
shots int // captures actually delivered
trigTimes []float64 // the sample time each capture triggered on
worstCov float64 // smallest fraction of its window a capture spanned
holdDeclined int // captures the zoom hold would not answer for
drawnPulses int // pulses visible in the delivered frames
wantPulses int // pulses those frames' windows really contained
gatedPulses int // pulses that arrived armed but with the fill gate shut
}
// yield is the fraction of presented pulses that produced a capture.
func (r pulseTrainResult) yield() float64 {
if r.pulses == 0 {
return 0
}
return float64(r.shots) / float64(r.pulses)
}
// run executes the simulation and returns what the trigger caught.
func (s pulseTrainSim) run(t *testing.T, key string) pulseTrainResult {
t.Helper()
h := NewHub()
h.rings[key] = newSigRing(ringCapInitial)
h.trigger.SetConfig(trigConfig{
signalKey: key, edge: "rising", threshold: 0.5,
windowSec: s.windowSec, prePercent: s.prePercent,
mode: "normal", holdoffSec: s.holdoffSec,
})
res := pulseTrainResult{worstCov: 1}
nBatch := int(s.rateHz * s.batchSec)
ts := make([]float64, nBatch)
vs := make([]float64, nBatch)
armed, changed := false, false
for now := 0.0; now < s.simSec; now += s.batchSec {
nPulseStarts := 0
for i := range ts {
ts[i] = now + float64(i)/s.rateHz
// Position within the current pulse period, in samples.
k := int((ts[i] - math.Floor(ts[i]/s.pulsePeriod)*s.pulsePeriod) * s.rateHz)
if k < s.pulseSamples {
vs[i] = 1
if k == 0 {
nPulseStarts++
}
} else {
vs[i] = 0
}
}
if !armed && now >= s.armAt {
h.trigger.Arm()
armed = true
}
if s.windowChangeAt > 0 && !changed && now >= s.windowChangeAt {
cfg := h.trigger.Config()
cfg.windowSec = s.windowChangeTo
h.trigger.SetConfig(cfg)
h.trigger.Arm()
changed = true
}
if armed {
res.pulses += nPulseStarts
if nPulseStarts > 0 && h.trigger.State() == trigArmed {
h.trigger.mu.Lock()
f := h.trigger.fillLocked()
h.trigger.mu.Unlock()
if f < 1 {
res.gatedPulses += nPulseStarts
}
}
}
h.ingest(key, 1, ts, vs)
// Mirror triggerTick, on the simulated clock.
tick := now + s.batchSec
h.retuneRings(tick)
_, span := h.rings[key].stats()
h.trigger.setBuffered(span, 0, false, true, tick)
if trigTime, pre, post, ok := h.trigger.dueCapture(tick); ok {
if buf := h.buildTriggerCapture(trigTime, pre, post); buf != nil {
res.shots++
res.trigTimes = append(res.trigTimes, trigTime)
first, last, _ := decodeCaptureSpan(t, buf, key)
if cov := (last - first) / (pre + post); cov < res.worstCov {
res.worstCov = cov
}
if _, _, ok := h.capture.slice(key, trigTime-pre, trigTime+post); !ok {
res.holdDeclined++
}
// What the client would actually draw, against what the window
// really contained. A wide window holds several pulses, and a
// frame showing only the one it triggered on has lost the rest
// between the ring, the bucketing and the decimation.
_, fv := decodeCaptureSig(t, buf, key)
res.drawnPulses += countPulses(fv, 0.5)
res.wantPulses += countPulseStarts(trigTime-pre, trigTime+post,
s.pulsePeriod, 1/s.rateHz)
}
h.trigger.markTriggered(tick)
} else if h.trigger.dueRearm(tick) {
h.trigger.rearm()
}
}
return res
}
// decodeCaptureSig pulls one signal's samples out of a v2 capture frame.
func decodeCaptureSig(t *testing.T, buf []byte, key string) (ts, vs []float64) {
t.Helper()
off := 1 + 8 + 8 + 8
nSig := int(binary.LittleEndian.Uint32(buf[off:]))
off += 4
for i := 0; i < nSig; i++ {
kl := int(binary.LittleEndian.Uint16(buf[off:]))
off += 2
k := string(buf[off : off+kl])
off += kl
cnt := int(binary.LittleEndian.Uint32(buf[off:]))
off += 4
if k == key {
ts = make([]float64, cnt)
vs = make([]float64, cnt)
for j := 0; j < cnt; j++ {
ts[j] = math.Float64frombits(binary.LittleEndian.Uint64(buf[off+j*8:]))
vs[j] = math.Float64frombits(binary.LittleEndian.Uint64(buf[off+cnt*8+j*8:]))
}
}
off += cnt * 16
}
return
}
// countPulses counts runs of samples at or above thr.
func countPulses(v []float64, thr float64) int {
n, in := 0, false
for _, x := range v {
if x >= thr {
if !in {
n, in = n+1, true
}
} else {
in = false
}
}
return n
}
// countPulseStarts is how many pulse starts fall inside [t0, t1]. A pulse
// starting within one sample of t1 is not counted: only its first sample is
// inside the window, and the ring's min/max bucket for it may put that sample's
// extremum just past the edge, which is a boundary artefact rather than a loss.
func countPulseStarts(t0, t1, period, dt float64) int {
n := 0
for k := math.Floor(t0 / period); k*period <= t1; k++ {
if p := k * period; p >= t0 && p < t1-2*dt {
n++
}
}
return n
}
// deadTimeSec is how long the FSM is blind after firing at t: it must acquire
// the post-trigger window before the capture can be harvested, and the holdoff
// guards against re-triggering on the same event. Both are measured from the
// trigger point, so they overlap rather than add.
func deadTimeSec(windowSec, prePercent, holdoffSec float64) float64 {
return math.Max(windowSec*(1-prePercent/100), holdoffSec)
}
// A trigger cannot show two windows at once, so pulses closer together than its
// post-trigger window are necessarily lost. Pulses spaced FURTHER apart than
// that are not: nothing about the acquisition prevents catching every one.
//
// This is the reported failure. The FSM used to go deaf from the trigger point
// until the capture had been harvested (a post-window plus captureMarginSec)
// and the holdoff had then elapsed on top of that, then wait for a fresh edge —
// so the effective spacing was rounded UP to a whole pulse period. At the
// default 1 s window and 0.2 s holdoff the blind stretch came to 1.15 s, which
// is longer than a 1 s pulse period by a hair, and a pulse train at 1 Hz was
// caught at 0.5 Hz. Widening the window made it worse in whole multiples.
func TestSporadicPulsesWiderThanThePostWindowAreAllCaught(t *testing.T) {
const key = "s1:Ch1"
cases := []struct{ window, period float64 }{
{0.5, 0.5}, // post 0.4 s
{1.0, 1.0}, // post 0.8 s — the case the report was made against
{2.0, 2.0}, // post 1.6 s
{5.0, 5.0}, // post 4.0 s
}
for _, c := range cases {
sim := pulseTrainSim{
rateHz: 1000, batchSec: 1.0 / 30.0,
pulsePeriod: c.period, pulseSamples: 3,
simSec: 41 * c.period, windowSec: c.window, prePercent: 20,
holdoffSec: autoRearmDelaySec, armAt: c.period,
}
res := sim.run(t, key)
// Two pulses are always in flight rather than caught: the one that lands
// as the trigger arms, and the one still being collected when the run
// ends.
if got := res.yield(); got < 0.94 {
t.Errorf("window %.1f s, pulse every %.1f s: caught %d of %d (%.0f%%); "+
"the post-trigger window is only %.2f s, so every pulse fits",
c.window, c.period, res.shots, res.pulses, 100*got,
c.window*0.8)
}
}
}
// The loss that remains must be the loss that has to remain. A capture cannot
// start before the previous one's post-window is acquired, and it can only start
// on a pulse, so consecutive captures are a dead time apart rounded UP to the
// next pulse — never further. Any longer gap means an edge that the acquisition
// no longer needed was thrown away anyway.
//
// The bound is stated as dead + period rather than ceil(dead/period)*period
// because when the two divide exactly, whether the pulse at the boundary counts
// comes down to the last bit of the sample timestamp. Both answers are correct;
// a gap beyond either is not.
func TestSporadicCaptureGapsStayWithinTheDeadTime(t *testing.T) {
const key = "s1:Ch1"
for _, window := range []float64{0.5, 1.0, 2.0, 5.0} {
for _, period := range []float64{0.25, 0.5, 1.0, 2.0} {
sim := pulseTrainSim{
rateHz: 1000, batchSec: 1.0 / 30.0,
pulsePeriod: period, pulseSamples: 3,
simSec: 60, windowSec: window, prePercent: 20,
holdoffSec: autoRearmDelaySec, armAt: 1.0,
}
res := sim.run(t, key)
dead := deadTimeSec(window, 20, autoRearmDelaySec)
limit := dead + period + 2*sim.batchSec
worst, worstAt := 0.0, 0.0
for i := 1; i < len(res.trigTimes); i++ {
if g := res.trigTimes[i] - res.trigTimes[i-1]; g > worst {
worst, worstAt = g, res.trigTimes[i-1]
}
}
if worst > limit {
t.Errorf("window %.1f s, pulse every %.2f s: %.2f s between the captures "+
"at %.2f s and %.2f s; the dead time is only %.2f s, so %.2f s is the most "+
"that can be missed",
window, period, worst, worstAt, worstAt+worst, dead, limit)
}
t.Logf("window %.1f s, pulse every %.2f s: %d/%d captures (%.0f%%), "+
"dead time %.2f s, worst gap %.2f s, gated %d",
window, period, res.shots, res.pulses, 100*res.yield(), dead,
worst, res.gatedPulses)
}
}
}
// Whatever the trigger does catch has to come back whole: a window wide enough
// to hold several pulses must show all of them, at every rate, including the
// rates that force the ring into min/max bucketing.
func TestSporadicCaptureShowsEveryPulseInItsWindow(t *testing.T) {
const key = "s1:Ch1"
for _, rate := range []float64{1000, 200e3} {
for _, window := range []float64{1.0, 2.0, 5.0} {
sim := pulseTrainSim{
rateHz: rate, batchSec: 1.0 / 30.0,
pulsePeriod: 0.5, pulseSamples: 3,
simSec: 40, windowSec: window, prePercent: 20,
holdoffSec: autoRearmDelaySec, armAt: 1.0,
}
res := sim.run(t, key)
if res.shots == 0 {
t.Fatalf("rate %.0f window %.1f s: no captures at all", rate, window)
}
// wantPulses excludes the pulse straddling each window's far edge,
// whose bucket may place its extremum just past it, so the frames
// may legitimately draw a few more than that — but never fewer.
if res.drawnPulses < res.wantPulses {
t.Errorf("rate %.0f window %.1f s: frames drew %d pulses, their windows held %d",
rate, window, res.drawnPulses, res.wantPulses)
}
if res.worstCov < 0.98 {
t.Errorf("rate %.0f window %.1f s: worst capture spanned %.0f%% of its window",
rate, window, 100*res.worstCov)
}
if res.holdDeclined > 0 {
t.Errorf("rate %.0f window %.1f s: the zoom hold declined %d of %d captures",
rate, window, res.holdDeclined, res.shots)
}
}
}
}
// Widening the window mid-run is the gesture the report came from. The fill gate
// holds the first shot off until the ring reaches back far enough, which is
// correct; what it must not do is stay shut, nor leave the trigger losing pulses
// once the ring has caught up.
func TestSporadicYieldRecoversAfterAWindowChange(t *testing.T) {
const key = "s1:Ch1"
for _, w := range []float64{1.0, 2.0, 5.0} {
sim := pulseTrainSim{
rateHz: 200e3, batchSec: 1.0 / 30.0,
pulsePeriod: w, pulseSamples: 3,
simSec: 30 * w, windowSec: 0.2, prePercent: 20,
holdoffSec: autoRearmDelaySec, armAt: 1.0,
windowChangeAt: 10 * w, windowChangeTo: w,
}
res := sim.run(t, key)
// Count only what happened after the change settled.
after, want := 0, 0
for _, tt := range res.trigTimes {
if tt > 11*w {
after++
}
}
for p := 11 * w; p < 30*w; p += w {
want++
}
if float64(after) < 0.9*float64(want) {
t.Errorf("window 0.2 -> %.1f s: %d captures in the %d pulses after the change",
w, after, want)
}
}
}
+58 -6
View File
@@ -297,9 +297,9 @@ func TestCollectingIsBroadcast(t *testing.T) {
// later. It forgets any earlier measurement first, so the rate is the one
// asked for rather than a blend with it.
func setFill(te *triggerEngine, span, growth, now float64) {
te.setBuffered(0, false, now)
te.setBuffered(span-growth, true, now)
te.setBuffered(span, true, now+1)
te.setBuffered(0, 0, false, false, now)
te.setBuffered(span-growth, 0, false, true, now)
te.setBuffered(span, 0, false, true, now+1)
}
// What has to hold is that the buffer spans the whole window by the time the
@@ -417,9 +417,9 @@ func TestForceIgnoresFillGate(t *testing.T) {
// interval, so they refresh the span and leave the seeded rate alone.
func seedFillNow(te *triggerEngine, span, growth float64) {
now := float64(time.Now().UnixNano()) / 1e9
te.setBuffered(0, false, now-1)
te.setBuffered(span-growth, true, now-1)
te.setBuffered(span, true, now)
te.setBuffered(0, 0, false, false, now-1)
te.setBuffered(span-growth, 0, false, true, now-1)
te.setBuffered(span, 0, false, true, now)
}
// While it holds off, the trigger looks identical to one that is ignoring
@@ -505,3 +505,55 @@ func drainStates(t *testing.T, h *Hub) []map[string]any {
}
}
}
// A ring whose coverage saturates below the window (measured source rate that
// over-estimates the true one) can never satisfy the full-window need. The
// coverage clamp must open the gate once the ring is full, delivering a short
// capture rather than staying deaf forever.
func TestFillNeedClampedToCoverage(t *testing.T) {
te := newTriggerEngine()
te.SetConfig(trigConfig{signalKey: "s:x", windowSec: 60, prePercent: 20, mode: "normal", holdoffSec: 0.2})
setFill(te, 50, 0, 100) // ring full at 50 s, no growth
te.mu.Lock()
te.bufCoverage = 50 // the ring can never reach further back
te.mu.Unlock()
if need := te.fillNeedLocked(); need != 50 {
t.Errorf("need = %v, want 50 (clamped to coverage, not the 60 s window)", need)
}
if f := te.fillLocked(); f < 1 {
t.Errorf("fillLocked = %v, want >= 1: a full ring below the window must still open the gate", f)
}
// Without the clamp the gate would stay shut forever.
te.mu.Lock()
te.bufCoverage = 0
te.mu.Unlock()
if f := te.fillLocked(); f >= 1 {
t.Errorf("baseline: fillLocked = %v, want < 1 without a coverage clamp", f)
}
}
// When the disk archive already spans the window it can back-fill the front of
// a capture, so the gate must only require the ring to have reached the
// pre-window, not the whole window.
func TestFillNeedArchiveLowersToPreWindow(t *testing.T) {
te := newTriggerEngine()
te.SetConfig(trigConfig{signalKey: "s:x", windowSec: 60, prePercent: 20, mode: "normal", holdoffSec: 0.2})
setFill(te, 30, 0, 100) // ring holds only 30 s, no growth → need 60 without archive
te.mu.Lock()
te.bufArchived = true
te.mu.Unlock()
if want := 12.0; te.fillNeedLocked() != want { // 60 * 0.20
t.Errorf("need = %v, want %v (archive lowers to the pre-window)", te.fillNeedLocked(), want)
}
// A ring holding just the pre-window opens the gate once archived.
te.mu.Lock()
te.bufSpan = 12
te.mu.Unlock()
if f := te.fillLocked(); f < 1 {
t.Errorf("fillLocked = %v, want >= 1 with pre-window buffered and the archive available", f)
}
}
+24 -3
View File
@@ -23,15 +23,22 @@
* [uint32 numSigs]
* numSigs × UDPSSignalDescriptor (136 bytes each, packed)
* [uint8 publishMode] (PublishModeStrict / Accumulate / Decimate)
* [uint64 hrtFrequency] ticks per second of the producer's HRT
*
* Everything after the descriptors is an optional trailer: a receiver must
* accept a payload that stops early and must ignore bytes it does not know.
* publishMode defaults to Strict when absent, hrtFrequency to
* UDPS_HRT_FREQUENCY_UNKNOWN.
*
* DATA payload (Strict / Decimate):
* [uint64 HRT timestamp]
* per-signal data in CONFIG order (quantised or raw, no padding)
*
* DATA payload (Accumulate):
* [uint64 HRT timestamp]
* [uint32 numSamples]
* for each signal: if scalar numSamples elements; else NumElements once
* [uint64 HRT timestamp of the first slot in the batch]
* [uint32 numSamples] RT cycles accumulated into this packet
* for each signal, in CONFIG order: numSamples × NumElements values
* (signal-major, one full snapshot per accumulated cycle)
*/
#ifndef UDPS_PROTOCOL_H_
@@ -123,6 +130,20 @@ static const uint8 UDPS_PUBLISH_STRICT = 0u; ///< One packet per Synchronise
static const uint8 UDPS_PUBLISH_ACCUMULATE = 1u; ///< Variable batch; flush on size or time
static const uint8 UDPS_PUBLISH_DECIMATE = 2u; ///< One packet per Ratio calls
/*---------------------------------------------------------------------------*/
/* HRT frequency (CONFIG trailing uint64) */
/*---------------------------------------------------------------------------*/
/**
* Sentinel for a CONFIG that carries no HRT frequency, either because the
* trailer is absent (producer older than this field) or because the producer
* could not determine it. DATA timestamps are raw ticks of the producer's
* high-resolution timer, so without this a receiver on another host has no
* way to turn them into seconds and can only fall back to its own timer's
* frequency which is right only while the two happen to agree.
*/
static const uint64 UDPS_HRT_FREQUENCY_UNKNOWN = 0u;
/*---------------------------------------------------------------------------*/
/* CONFIG payload — per-signal descriptor */
/*---------------------------------------------------------------------------*/
+56 -1
View File
@@ -84,8 +84,28 @@ Offset Size Type Field
0xFFFFFFFF = PacketTime (no reference)
104 32 char[32] unit null-terminated physical unit string
── (total per signal: 136 bytes) ────────────────────────────
── trailer, immediately after the last descriptor ───────────
0 1 uint8 publishMode 0 = Strict, 1 = Accumulate, 2 = Decimate
1 8 uint64 hrtFrequency producer's HRT ticks per second;
0 = unknown
```
### CONFIG trailer
Everything after the descriptors is a trailer that grew field by field, so a
receiver must accept a payload that stops early and must ignore bytes it does
not recognise. An absent `publishMode` means Strict; an absent or zero
`hrtFrequency` means the producer did not publish its tick rate.
`hrtFrequency` is what makes DATA timestamps interpretable off-box. DATA
carries the raw value of the producer's high-resolution counter, and on x86
that counter runs at the TSC frequency — a different number on every model. A
receiver that divides by its own timer's frequency instead is right only while
producer and consumer sit on the same host; anywhere else every batch is laid
out over the wrong span of time. Fall back to the local frequency only when the
field is missing, and reject implausible values (nothing below 1 kHz is a
high-resolution timer).
### Type Codes
| Code | C type | Bytes/element |
@@ -129,7 +149,9 @@ After reassembly, the DATA payload layout is:
```
Offset Size Type Field
────── ──── ────── ────────────────────────────────────────────────────
0 8 uint64 hrtTimestamp hardware reference timer count at Synchronise()
0 8 uint64 hrtTimestamp producer's high-resolution counter at
Synchronise(); divide by the CONFIG
hrtFrequency to get seconds
── for each signal (in config order) ────────────────────────────────────
varies N×sz — signal data N = numRows×numCols, sz = element size
(wire size if quantized, raw size otherwise)
@@ -171,6 +193,39 @@ the client to reassemble them in any order.
---
## Ordering DATA (required of every receiver)
DATA carries its own `counter` sequence, incremented once per sent packet
(CONFIG is numbered independently). Reassembly completes in arrival order, not
counter order, so a packet reordered or duplicated on the wire surfaces after a
newer one has already been consumed. Its values are well-formed but carry an
older time base: accepting it writes them over samples the consumer already
holds and leaves the span they should have filled empty — a collision on one
side and a hole on the other.
A receiver must therefore drop any DATA packet that does not advance the
counter, and must order it by the *signed* difference:
```c
int32_t delta = (int32_t)(counter - lastCounter); /* survives the uint32 wrap */
if (delta <= 0) { /* stale or duplicate: drop */ }
lost = (uint32_t)delta - 1u; /* packets missing before this one */
```
Comparing the values directly would call the first packet after the wrap stale
and reject the stream from then on.
`lost` matters beyond diagnostics. A consumer that spaces batched samples from
the elapsed time since the previous packet must divide that gap by `lost + 1`
batches; dividing by one batch reports a period too long by exactly that factor
and walks the samples past their own end into the next packet's range. Reset
the sequence on (re)connect: the producer's counter restarts independently.
Implemented in `UDPSClient::AcceptDataCounter` (C++),
`udpsprotocol.SequenceGate` (Go) and `decode_data` (C).
---
## Minimal Python Client Example
```python
+21 -2
View File
@@ -107,12 +107,31 @@ Hub-side, web-client semantics (`setTrigger` fields in
```
IDLE --arm--> ARMED --edge crossing--> COLLECTING --every source past trigTime+postSec+0.15s--> TRIGGERED
TRIGGERED --rearm (single) / auto ~200ms (normal, unless stopped)--> ARMED
TRIGGERED --rearm (single) / auto after holdoffSec (normal, unless stopped)--> ARMED
└─ or straight to COLLECTING on a held edge
any --disarm--> IDLE
```
`UDPSourceSession` calls `TriggerEngine::CheckSample` for every decoded sample
of the configured signal (signal index cached per config epoch). Each source is
of the configured signal (signal index cached per config epoch).
The comparator keeps running through COLLECTING and TRIGGERED. It cannot fire
there — the capture in flight owns that stretch — but it remembers the first
edge at or past `trigTime + max(postSec, holdoffSec)`, and `Rearm()` fires on
that remembered edge instead of waiting for a fresh one. Without this the engine
is deaf from its own trigger point until the capture has been harvested and the
holdoff has run, which on a sparse pulse train rounds the capture spacing up to
a whole pulse period: at a 1 s window a 1 Hz train was caught at 0.5 Hz, and a
wider window lost whole multiples. The capture is built from the edge's own
timestamp out of rings that still hold everything around it, so honouring it
costs nothing.
`Arm()` and `Rearm()` differ only in this: `Arm()` is the operator's own arm and
discards the held edge (they asked for the next event), while `Rearm()` is the
automatic end-of-capture arm and consumes it. `Rearm()` also keeps the tracked
level, so the first sample after it is compared against its real predecessor
rather than being spent seeding one. `SetConfig()` and `Disarm()` drop the held
edge as well — it was never judged against the new window. Each source is
read `[trigTimepreSec, trigTime+postSec]`, LTTB-capped to 20 000 pts/signal and
appended to a binary **version 2** capture frame; every FSM transition
broadcasts a `triggerState` event.
+9
View File
@@ -175,6 +175,7 @@ replayed traffic can be decoded without a client.
```c
typedef struct {
uint32_t counter; /* gaps in this sequence are lost datagrams */
uint32_t lost; /* DATA packets missing immediately before this one */
uint64_t hrt; /* producer's high-resolution timer at send */
double recv_time; /* CLOCK_REALTIME seconds at arrival */
uint8_t publish_mode;
@@ -194,6 +195,13 @@ scalar signal in Accumulate mode, where the producer batches several RT cycles i
and `count == num_samples` — one value per cycle. Arrays are not batched: they appear once and
apply to the whole packet. `udps_frame_value(f, sig, sample, elem)` applies that rule for you.
**Ordering.** Frames reach `on_data` in counter order: a DATA packet that does not advance the
counter — reordered or duplicated on the wire — is dropped rather than delivered, because its
values carry a time base older than data you already have, and placing them would overwrite live
samples while leaving their own span empty. `lost` reports how many packets went missing just
before the frame. If you space samples yourself from the elapsed time since the previous frame,
divide by `lost + 1` batches, not one: the gap covers the missing packets' cycles too.
**Timestamps.** The protocol does not put a timestamp on every element; how to date them depends
on the signal's `time_mode` (see [Protocol.md](Protocol.md#time-mode-codes)):
@@ -223,6 +231,7 @@ udps_client_stats(cli, &s);
| `frames_delivered` | DATA packets decoded and passed to `on_data`. |
| `config_updates` | CONFIG packets applied. |
| `counter_gaps` | Missing packet counters — datagrams lost on the wire or in the kernel. |
| `stale_packets` | DATA packets dropped for not advancing the counter: reordered or duplicated on the wire. |
| `fragments_dropped` | Duplicate, stale or unplaceable fragments; a non-zero value with `counter_gaps` means fragmented updates are arriving incomplete. |
| `reconnects` | Sessions re-established after a silence timeout. |
+5 -1
View File
@@ -1736,7 +1736,11 @@ void StreamHub::TriggerTick(float64 wallNowS) {
(wallNowS >= rearmAtWallS_)) {
rearmPending_ = false;
if (!trigger_.GetStopped()) {
trigger_.Arm();
/* Rearm, not Arm: an edge that arrived while this capture was being
* collected is fired on at once instead of being thrown away, which
* is what kept sparse pulse trains from being caught at their own
* rate. */
trigger_.Rearm();
}
}
+64 -12
View File
@@ -19,7 +19,17 @@ TriggerEngine::TriggerEngine()
trigTime_(0.0),
firedPreSec_(0.0),
firedPostSec_(0.0),
firedValid_(false) {
firedValid_(false),
pendingTime_(0.0),
pendingValid_(false) {
}
void TriggerEngine::LatchWindowLocked(float64 t) {
state_ = kTrigCollecting;
trigTime_ = t;
firedPreSec_ = config_.windowSec * config_.prePercent / 100.0;
firedPostSec_ = config_.windowSec - firedPreSec_;
firedValid_ = true;
}
void TriggerEngine::SetConfig(const TriggerConfig &cfg) {
@@ -40,6 +50,9 @@ void TriggerEngine::SetConfig(const TriggerConfig &cfg) {
epoch_++;
prevValid_ = false;
prevValue_ = 0.0;
/* An edge held over from the old configuration would be latched against the
* new window, which it was never judged against. */
pendingValid_ = false;
mutex_.FastUnLock();
}
@@ -62,6 +75,26 @@ void TriggerEngine::Arm() {
state_ = kTrigArmed;
prevValid_ = false;
prevValue_ = 0.0;
pendingValid_ = false;
mutex_.FastUnLock();
}
void TriggerEngine::Rearm() {
(void) mutex_.FastLock();
if (pendingValid_) {
const float64 t = pendingTime_;
pendingValid_ = false;
LatchWindowLocked(t);
REPORT_ERROR_STATIC(MARTe::ErrorManagement::Information,
"TriggerEngine: rearmed onto the edge held at t=%.6f "
"(pre=%.4fs post=%.4fs)",
t, firedPreSec_, firedPostSec_);
}
else {
/* prevValue_/prevValid_ are deliberately kept: the comparator ran right
* through the dead time, so the next sample has a real predecessor. */
state_ = kTrigArmed;
}
mutex_.FastUnLock();
}
@@ -72,6 +105,7 @@ void TriggerEngine::Disarm() {
prevValid_ = false;
prevValue_ = 0.0;
firedValid_ = false;
pendingValid_ = false;
mutex_.FastUnLock();
}
@@ -96,7 +130,10 @@ void TriggerEngine::CheckSample(float64 t, float64 v) {
lastTime_ = t;
lastTimeValid_ = true;
if (state_ != kTrigArmed) {
/* A capture in flight does not stop the comparator; it only changes what an
* edge does. See pendingTime_. */
const bool inFlight = (state_ == kTrigCollecting) || (state_ == kTrigTriggered);
if ((state_ != kTrigArmed) && !inFlight) {
mutex_.FastUnLock();
return;
}
@@ -121,17 +158,36 @@ void TriggerEngine::CheckSample(float64 t, float64 v) {
}
if (fired) {
state_ = kTrigCollecting;
trigTime_ = t;
if (!inFlight) {
/* Latch the window at fire time so later config edits do not
* affect this capture (web client snap._preS/_postS). */
firedPreSec_ = config_.windowSec * config_.prePercent / 100.0;
firedPostSec_ = config_.windowSec - firedPreSec_;
firedValid_ = true;
LatchWindowLocked(t);
REPORT_ERROR_STATIC(MARTe::ErrorManagement::Information,
"TriggerEngine: fired at t=%.6f (pre=%.4fs post=%.4fs)",
t, firedPreSec_, firedPostSec_);
}
else if (!pendingValid_ && firedValid_) {
/* The earliest trigger point a new capture may take. The one in
* flight owns everything up to the end of its own post-window, and
* the holdoff a guard against re-triggering on the ringing of the
* SAME event is measured from its trigger point too, so the two
* overlap rather than add.
*
* Keep only the FIRST qualifying edge: a later one would deliver
* the same capture a pulse further on and skip the one between. */
float64 guard = firedPostSec_;
if (config_.holdoffSec > guard) {
guard = config_.holdoffSec;
}
if (t >= (trigTime_ + guard)) {
pendingTime_ = t;
pendingValid_ = true;
}
}
else {
/* Already holding an edge, or no window latched to measure against. */
}
}
mutex_.FastUnLock();
}
@@ -141,11 +197,7 @@ bool TriggerEngine::Force() {
bool ok = lastTimeValid_ && (state_ != kTrigCollecting);
if (ok) {
state_ = kTrigCollecting;
trigTime_ = lastTime_;
firedPreSec_ = config_.windowSec * config_.prePercent / 100.0;
firedPostSec_ = config_.windowSec - firedPreSec_;
firedValid_ = true;
LatchWindowLocked(lastTime_);
REPORT_ERROR_STATIC(MARTe::ErrorManagement::Information,
"TriggerEngine: forced at t=%.6f (pre=%.4fs post=%.4fs)",
trigTime_, firedPreSec_, firedPostSec_);
+35 -3
View File
@@ -88,9 +88,24 @@ public:
*/
uint32 GetConfigEpoch() const;
/** @brief Arm: any state → ARMED (resets edge detection). */
/**
* @brief Arm: any state ARMED (resets edge detection).
* This is the user's own arm, so it discards any edge remembered during the
* previous capture: the user asked for the next event, not for one that has
* already been and gone.
*/
void Arm();
/**
* @brief The automatic arm at the end of a capture (normal mode).
* Unlike Arm() it honours an edge seen while the capture was being
* collected, firing on it at once rather than waiting for the next one, and
* it keeps the tracked level so the first sample afterwards is compared
* against its real predecessor. TRIGGERED COLLECTING when an edge was
* remembered, otherwise ARMED.
*/
void Rearm();
/** @brief Disarm: any state → IDLE; clears the stopped flag. */
void Disarm();
@@ -102,8 +117,10 @@ public:
/**
* @brief Edge-detect one decoded sample of the configured signal.
* Receive-thread context. Only acts in ARMED state; on a matching edge
* latches trigTime and the pre/post window and moves to COLLECTING.
* Receive-thread context. In ARMED state a matching edge latches trigTime
* and the pre/post window and moves to COLLECTING. While a capture is in
* flight (COLLECTING/TRIGGERED) the comparator keeps running and the first
* edge clear of that capture is remembered for the next Rearm().
*/
void CheckSample(float64 t, float64 v);
@@ -143,6 +160,21 @@ private:
float64 firedPreSec_; ///< Window pre-part latched at fire time
float64 firedPostSec_; ///< Window post-part latched at fire time
bool firedValid_; ///< true after a fire, until Disarm()
/**
* The edge to fire on as soon as the FSM rearms, in sample time, recorded
* while a capture is still being collected or handed out. Without it the
* trigger is deaf from its own trigger point until the capture has been
* harvested and the holdoff has run, and then waits for a fresh edge, which
* on a sparse pulse train rounds the capture spacing up to a whole pulse
* period. Remembering the edge instead makes the blind stretch exactly the
* guard interval it has to be, since the capture is built from the edge's
* own timestamp and the rings still hold everything around it.
*/
float64 pendingTime_;
bool pendingValid_;
/** @brief Freeze the pre/post split at fire time; caller holds the mutex. */
void LatchWindowLocked(float64 t);
};
inline TriggerConfig::TriggerConfig()
@@ -99,6 +99,8 @@ void UDPSourceSession::ResetCalibration() {
lastPktWallValid_[i] = false;
lastPktWallS_[i] = 0.0;
accScalarPrevN_[i] = 0u;
accScalarDtValid_[i] = false;
accScalarDtEMA_[i] = 0.0;
}
}
@@ -197,7 +199,8 @@ void UDPSourceSession::OnUDPSConfig(const uint8 *payload, uint32 payloadSize) {
}
void UDPSourceSession::OnUDPSData(const uint8 *payload, uint32 payloadSize) {
ParseDataPayload(payload, payloadSize);
/* Valid only for the duration of this callback. */
ParseDataPayload(payload, payloadSize, client_.GetLastDataGap());
}
/*---------------------------------------------------------------------------*/
@@ -225,6 +228,9 @@ void UDPSourceSession::ParseConfigPayload(const uint8 *payload, uint32 size) {
sigDescs_[i].unit[sizeof(sigDescs_[i].unit) - 1u] = '\0';
}
publishMode_ = payload[4u + numSigs * UDPS_SIGNAL_DESC_SIZE];
hrtFreq_ = UDPSConfigHrtFrequency(
payload, size, numSigs,
static_cast<float64>(MARTe::HighResolutionTimer::Frequency()));
numSignals_ = numSigs;
configured_ = true;
@@ -273,8 +279,9 @@ void UDPSourceSession::ParseConfigPayload(const uint8 *payload, uint32 size) {
}
REPORT_ERROR_STATIC(MARTe::ErrorManagement::Information,
"UDPSourceSession[%s]: CONFIG received — %u signals.",
id_.Buffer(), numSigs);
"UDPSourceSession[%s]: CONFIG received — %u signals, "
"producer HRT %.0f Hz.",
id_.Buffer(), numSigs, hrtFreq_);
}
void UDPSourceSession::AllocateRingBuffers() {
@@ -376,7 +383,8 @@ float64 UDPSourceSession::ProducerNewestTime() const {
/* DATA parsing */
/*---------------------------------------------------------------------------*/
void UDPSourceSession::ParseDataPayload(const uint8 *payload, uint32 size) {
void UDPSourceSession::ParseDataPayload(const uint8 *payload, uint32 size,
uint32 lostPackets) {
if (size < 8u) { return; }
/* Copy metadata under lock */
@@ -568,9 +576,9 @@ void UDPSourceSession::ParseDataPayload(const uint8 *payload, uint32 size) {
* immune to this because it is sampled at acquisition.
*
* hrtTimestamp is the HRT counter of sample 0; hrtFreq_ (the
* local HRT frequency, identical to the sender on the same
* host) converts it to seconds, then a one-time calibration
* maps the sender clock onto wall-clock. */
* producer's tick rate, taken from the CONFIG trailer) converts
* it to seconds, then a one-time calibration maps the sender
* clock onto wall-clock. */
const float64 hrt0Sec = static_cast<float64>(hrtTimestamp) /
hrtFreq_;
if ((!timeSigCalibValid_[s]) ||
@@ -581,16 +589,25 @@ void UDPSourceSession::ParseDataPayload(const uint8 *payload, uint32 size) {
}
/* Per-sample dt: samplingRate if present, else derive it from
* the sender-HRT gap to the previous packet divided by that
* packet's sample count (the flushes carry contiguous RT
* cycles, so this is exactly one cycle period). */
* the sender-HRT gap to the previous packet.
*
* The gap is divided by the number of RT cycles it actually
* spans, not by the previous packet's sample count. Those two
* agree only while nothing is lost; once a packet goes missing
* the gap covers cycles the previous count never saw, and
* dividing by that count inflates dt until this packet's
* samples overrun into the next packet's range. lostPackets
* comes from the producer's packet counter, so the divisor
* widens with the gap and dt is unchanged. */
float64 dt;
if (desc.samplingRate > 0.0) {
dt = 1.0 / desc.samplingRate;
} else if (lastPktWallValid_[s] && (accScalarPrevN_[s] > 0u) &&
(hrt0Sec > lastPktWallS_[s])) {
dt = (hrt0Sec - lastPktWallS_[s]) /
static_cast<float64>(accScalarPrevN_[s]);
dt = UDPSEstimateAccumDt(hrt0Sec - lastPktWallS_[s],
accScalarPrevN_[s], lostPackets,
accScalarDtEMA_[s],
accScalarDtValid_[s]);
} else {
dt = 1.0e-3; /* 1 kHz default until the gap is known */
}
@@ -39,6 +39,104 @@ using MARTe::ConfigurationDatabase;
/** Maximum number of signals per source session. */
static const uint32 UDPSS_MAX_SIGNALS = 256u;
/* Accumulated-scalar dt estimator tuning. */
/** Weight of a new observation; slow enough that one bad gap barely moves it. */
static const float64 UDPSS_DT_EMA_ALPHA = 0.05;
/** Observations outside [lo, hi] x the current estimate are treated as a
* mis-counted gap and discarded rather than smoothed in. */
static const float64 UDPSS_DT_ACCEPT_LO = 0.5;
static const float64 UDPSS_DT_ACCEPT_HI = 2.0;
/**
* @brief Per-sample period of an accumulated scalar packet, robust to loss.
*
* An Accumulate producer batches consecutive RT cycles, so the sender-clock
* gap between two packets' first samples covers exactly as many cycles as the
* earlier packet carried but only while nothing is lost in between. Over UDP
* (and with a producer that can overwrite a batch the sender never took) that
* assumption fails, and dividing the gap by the previous packet's sample count
* then inflates the period. The packet's own samples are laid out as
* base + e*dt, so an inflated dt walks them past their real end and into the
* span the next packet will claim: samples collide there and leave a hole
* behind them.
*
* The number of packets that went missing is not guessed from the gap that
* is circular, and an estimator that infers the cycle count from its own
* period has a stable fixed point wherever gap/dt is an integer, so a genuine
* rate change locks it at the old period forever. It comes instead from the
* UDPS packet counter, which the producer increments once per sent packet. The
* gap then spans (1 + lost) batches, each assumed to be prevN cycles, and with
* nothing lost the formula reduces exactly to gap/prevN.
*
* @param gap Sender-clock seconds since the previous packet's first sample.
* Must be > 0.
* @param prevN Samples in the previous packet. Must be > 0.
* @param lost Packets missing between the previous packet and this one,
* from the producer's counter.
* @param[in,out] dtEMA Smoothed period. Seeded on the first call.
* @param[in,out] dtValid False until dtEMA holds an estimate.
* @return The period to space this packet's samples by.
*/
inline float64 UDPSEstimateAccumDt(const float64 gap, const uint32 prevN,
const uint32 lost, float64 &dtEMA,
bool &dtValid) {
float64 cycles = static_cast<float64>(prevN) *
(1.0 + static_cast<float64>(lost));
if (cycles < 1.0) {
cycles = 1.0;
}
const float64 dtObs = gap / cycles;
if (!dtValid) {
dtEMA = dtObs;
dtValid = true;
} else if ((dtObs > (dtEMA * UDPSS_DT_ACCEPT_LO)) &&
(dtObs < (dtEMA * UDPSS_DT_ACCEPT_HI))) {
/* Track slow drift, but ignore observations far outside the current
* estimate: those are the signature of a mis-counted gap, and folding
* one in would drag the estimate towards the very error it exists to
* absorb. */
dtEMA = ((1.0 - UDPSS_DT_EMA_ALPHA) * dtEMA) +
(UDPSS_DT_EMA_ALPHA * dtObs);
}
return dtEMA;
}
/**
* @brief Pick the tick rate to divide a producer's DATA timestamps by.
*
* DATA packets carry the raw value of the producer's high-resolution counter,
* which is meaningless without the rate it runs at. The rate is published in
* the CONFIG trailer, after the descriptors and the publish-mode byte. When it
* is missing an older producer, or one that could not determine it the
* only remaining option is this host's own timer, which is right only while
* the two machines agree; on x86 that is the TSC frequency, so it is a
* different number on every model.
*
* @param payload Reassembled CONFIG payload.
* @param size Bytes in @p payload.
* @param numSigs Signal count already read from the payload, capped to what
* the receiver will store.
* @param localFreq This host's HRT frequency, used as the fallback.
* @return Ticks per second to convert DATA timestamps with; never 0.
*/
inline float64 UDPSConfigHrtFrequency(const uint8 *payload, const uint32 size,
const uint32 numSigs,
const float64 localFreq) {
const uint32 offset = 4u + (numSigs * MARTe::UDPS_SIGNAL_DESC_SIZE) + 1u;
if ((payload != NULL_PTR(const uint8 *)) && (size >= (offset + 8u))) {
uint64 wireFreq = 0u;
memcpy(&wireFreq, payload + offset, 8u);
/* Anything below 1 kHz is not a high-resolution timer; the field is
* either absent, unset, or the payload was mis-parsed, and adopting it
* would stretch every timestamp far enough to make the trace useless. */
if (wireFreq >= 1000u) {
return static_cast<float64>(wireFreq);
}
}
return localFreq;
}
/**
* @brief One connected UDPStreamer source.
*
@@ -229,7 +327,13 @@ private:
/* DATA payload parsing */
void ParseConfigPayload(const uint8 *payload, uint32 size);
void ParseDataPayload(const uint8 *payload, uint32 size);
/**
* @param lostPackets DATA packets missing immediately before this one, from
* the producer's counter; the accumulated-scalar period estimate
* needs it to know how many cycles the sender-clock gap spans.
*/
void ParseDataPayload(const uint8 *payload, uint32 size,
uint32 lostPackets);
void AllocateRingBuffers();
/** @brief Invalidate all wall-clock calibration state (receive thread only). */
@@ -394,13 +498,20 @@ private:
float64 lastPktWallS_[UDPSS_MAX_SIGNALS];
bool lastPktWallValid_[UDPSS_MAX_SIGNALS];
/* Accumulated-scalar timing: HRT counter frequency (local == sender on the
* same host) and the previous packet's sample count, used to reconstruct
* per-sample timestamps from the embedded sender HRT instead of the (UDP
* burst-sensitive) packet arrival time. */
/* Accumulated-scalar timing: the producer's HRT counter frequency and the
* previous packet's sample count, used to reconstruct per-sample timestamps
* from the embedded sender HRT instead of the (UDP burst-sensitive) packet
* arrival time. Seeded from this host's timer and replaced by the rate the
* producer publishes in CONFIG; see UDPSConfigHrtFrequency. */
float64 hrtFreq_;
uint32 accScalarPrevN_[UDPSS_MAX_SIGNALS];
/* Per-signal state of UDPSEstimateAccumDt (see above): the smoothed
* per-sample period for accumulated scalars whose descriptor carries no
* SamplingRate. */
float64 accScalarDtEMA_[UDPSS_MAX_SIGNALS];
bool accScalarDtValid_[UDPSS_MAX_SIGNALS];
/* Scratch buffers for decoding arrays (receive thread only). */
float64 *timeScratch_; ///< Time values scratch
float64 *valScratch_; ///< Data values scratch
@@ -107,6 +107,9 @@ UDPStreamer::UDPStreamer()
readyTimestamps = NULL_PTR(uint64 *);
scratchTimestamps = NULL_PTR(uint64 *);
readyFill = 0u;
readySnapshotPending = false;
droppedPublications = 0u;
lastDropReportTicks = 0u;
decimateRatio = 1u;
decimateCounter = 0u;
@@ -807,7 +810,9 @@ bool UDPStreamer::PrepareNextState(const char8 *const currentStateName,
* receives it immediately. The config is static for the lifetime of this
* state. */
if (ok) {
uint32 configBufSize = 4u + (numSigs * UDPS_SIGNAL_DESC_SIZE) + 32u + 1u;
/* numSigs + descriptors + publishMode + hrtFrequency (+ slack). */
uint32 configBufSize =
4u + (numSigs * UDPS_SIGNAL_DESC_SIZE) + 1u + 8u + 32u;
HeapI *heap = GlobalObjectsDatabase::Instance()->GetStandardHeap();
uint8 *cfgBuf = reinterpret_cast<uint8 *>(heap->Malloc(configBufSize));
if (cfgBuf != NULL_PTR(uint8 *)) {
@@ -871,6 +876,11 @@ bool UDPStreamer::Synchronise() {
/* HI-3: if accumFill reached maxBatchCount, force-flush before writing */
if (accumFill >= maxBatchCount) {
uint32 filled = accumFill;
if (readyFill > 0u) {
/* The sender has not taken the previous batch: it is about to be
* overwritten and its cycles will never reach any receiver. */
droppedPublications++;
}
(void)MemoryOperationsHelper::Copy(readyBuffer, accumBuffer,
filled * totalSrcBytes);
(void)MemoryOperationsHelper::Copy(
@@ -901,6 +911,9 @@ bool UDPStreamer::Synchronise() {
if (sizeCondition || timeCondition) {
bufMutex.FastLock(TTInfiniteWait);
if (readyFill > 0u) {
droppedPublications++;
}
(void)MemoryOperationsHelper::Copy(readyBuffer, accumBuffer,
filled * totalSrcBytes);
(void)MemoryOperationsHelper::Copy(
@@ -922,16 +935,24 @@ bool UDPStreamer::Synchronise() {
if (decimateCounter >= decimateRatio) {
decimateCounter = 0u;
bufMutex.FastLock(TTInfiniteWait);
if (readySnapshotPending) {
droppedPublications++;
}
(void)MemoryOperationsHelper::Copy(readyBuffer, memory, totalSrcBytes);
syncTimestamp = ts;
readySnapshotPending = true;
bufMutex.FastUnLock();
(void)dataSem.Post();
}
} else {
/* --- Strict path: post every call --- */
bufMutex.FastLock(TTInfiniteWait);
if (readySnapshotPending) {
droppedPublications++;
}
(void)MemoryOperationsHelper::Copy(readyBuffer, memory, totalSrcBytes);
syncTimestamp = ts;
readySnapshotPending = true;
bufMutex.FastUnLock();
(void)dataSem.Post();
}
@@ -955,25 +976,29 @@ ErrorManagement::ErrorType UDPStreamer::Execute(ExecutionInfo &info) {
}
if (info.GetStage() == ExecutionInfo::MainStage) {
/* --- Wait for RT thread to post new data ---
* ResetWait sleeps the background thread until the RT thread calls
* Synchronise() and posts dataSem, or until the timeout expires.
* Doing this FIRST means the thread spends nearly all its time here
* instead of spinning on the non-blocking select() below.
* Command latency is bounded by UDPS_DATA_WAIT_MS (acceptable for
* CONNECT / DISCONNECT). */
ErrorManagement::ErrorType waitErr =
dataSem.ResetWait(TimeoutType(UDPS_DATA_WAIT_MS));
bool dataReady = (waitErr == ErrorManagement::NoError);
/* --- Wait for the RT thread to publish new data ---
* dataSem is only a wake-up hint, never the record of pending work:
* EventSem::ResetWait resets the semaphore before waiting, so a Post that
* landed while this thread was inside ServiceClients()/SendData() is
* destroyed by the next Reset. Deciding what to send from the wait result
* would then skip that publication entirely, and the next flush would
* overwrite it the receiver sees the batch's whole time span missing.
* The buffers therefore carry the state, and are only waited on when they
* are empty (which also avoids paying the wait when work is already
* queued). */
if (!HasPendingPublication()) {
(void)dataSem.ResetWait(TimeoutType(UDPS_DATA_WAIT_MS));
}
/* --- Poll for incoming control commands (CONNECT / DISCONNECT / ACK) ---
*/
server.ServiceClients();
if (dataReady && server.HasClients()) {
/* Synchronise() already gates posting dataSem to the correct rate
* (size/time for Accumulate, every-Nth for Decimate, every call for
* Strict). Execute() just sends whatever is in the ready buffers. */
/* Synchronise() already gates publication to the correct rate (size/time
* for Accumulate, every-Nth for Decimate, every call for Strict). The
* pending publication is consumed whether or not anyone is listening, so
* that a client-less streamer neither spins here nor delivers a stale
* snapshot to the next client that connects. */
if (publishMode == UDPStreamerPublishAccumulate) {
/* --- Accumulate batch send --- */
uint32 fill = 0u;
@@ -986,10 +1011,11 @@ ErrorManagement::ErrorType UDPStreamer::Execute(ExecutionInfo &info) {
reinterpret_cast<uint8 *>(scratchTimestamps),
reinterpret_cast<const uint8 *>(readyTimestamps),
fill * static_cast<uint32>(sizeof(uint64)));
readyFill = 0u;
}
bufMutex.FastUnLock();
if (fill > 0u) {
if ((fill > 0u) && server.HasClients()) {
SerializeAccumulated(scratchBuffer, scratchTimestamps, fill);
uint32 sendBytes =
UDPS_TIMESTAMP_BYTES + 4u + fill * singleCycleWireBytes;
@@ -1003,12 +1029,18 @@ ErrorManagement::ErrorType UDPStreamer::Execute(ExecutionInfo &info) {
} else {
/* --- Single-snapshot send (Strict or Decimate) --- */
uint64 ts = 0u;
bool pending = false;
bufMutex.FastLock(TTInfiniteWait);
pending = readySnapshotPending;
if (pending) {
(void)MemoryOperationsHelper::Copy(scratchBuffer, readyBuffer,
totalSrcBytes);
ts = syncTimestamp;
readySnapshotPending = false;
}
bufMutex.FastUnLock();
if (pending && server.HasClients()) {
QuantizeAndSerialize(scratchBuffer, ts);
packetCounter++;
@@ -1019,6 +1051,8 @@ ErrorManagement::ErrorType UDPStreamer::Execute(ExecutionInfo &info) {
}
}
}
ReportDroppedPublications();
}
if (info.GetStage() == ExecutionInfo::TerminationStage) {
@@ -1212,6 +1246,16 @@ bool UDPStreamer::BuildConfigPayload(uint8 *buf, uint32 bufSize,
buf[payloadSize] = static_cast<uint8>(publishMode);
payloadSize += 1u;
/* 8 bytes: this host's HRT tick rate. DATA packets carry raw counter
* values, so a receiver on another machine cannot turn them into seconds
* without it. */
if ((payloadSize + 8u) > bufSize) {
return false;
}
uint64 hrtFrequency = HighResolutionTimer::Frequency();
(void)MemoryOperationsHelper::Copy(buf + payloadSize, &hrtFrequency, 8u);
payloadSize += 8u;
return true;
}
@@ -1327,6 +1371,36 @@ bool UDPStreamer::IsClientConnected() const { return server.HasClients(); }
bool UDPStreamer::IsMulticast() const { return server.IsMulticast(); }
uint32 UDPStreamer::GetDroppedPublications() const { return droppedPublications; }
bool UDPStreamer::HasPendingPublication() {
bool pending = false;
bufMutex.FastLock(TTInfiniteWait);
pending = (readyFill > 0u) || readySnapshotPending;
bufMutex.FastUnLock();
return pending;
}
void UDPStreamer::ReportDroppedPublications() {
uint64 now = HighResolutionTimer::Counter();
if ((now - lastDropReportTicks) < HighResolutionTimer::Frequency()) {
return;
}
lastDropReportTicks = now;
uint32 dropped = 0u;
bufMutex.FastLock(TTInfiniteWait);
dropped = droppedPublications;
bufMutex.FastUnLock();
if (dropped > 0u) {
REPORT_ERROR(ErrorManagement::Warning,
"Dropped %u unsent publication(s) so far: the sender thread is "
"not keeping up with the RT cycle.",
dropped);
}
}
CLASS_REGISTER(UDPStreamer, "1.0")
} /* namespace MARTe */
@@ -322,6 +322,17 @@ public:
*/
bool IsMulticast() const;
/**
* @brief Number of publications the sender thread never put on the wire.
* @details Synchronise() promotes a snapshot (Strict/Decimate) or a batch
* (Accumulate) to the ready buffer for the sender thread. If the next
* promotion arrives before the sender has taken the previous one, that
* publication is overwritten and its cycles never reach any receiver
* which a consumer sees as a hole in the time series. Counts those, so the
* loss is measurable rather than inferred from the plot.
*/
uint32 GetDroppedPublications() const;
private:
/**
* @brief Serializes the CONFIG payload into buf and sets payloadSize.
@@ -349,6 +360,18 @@ private:
*/
static uint8 TypeDescriptorToCode(TypeDescriptor td);
/**
* @brief True when the ready buffer holds data the sender has not taken yet.
* @details Read under bufMutex. The sender must consult this rather than
* rely on the dataSem edge, which ResetWait can destroy.
*/
bool HasPendingPublication();
/**
* @brief Emits at most one warning per second about overwritten publications.
*/
void ReportDroppedPublications();
/* Configuration parameters */
uint16 port; /**< UDP server port */
uint32 maxPayloadSize; /**< Max payload bytes per UDP packet (excluding header) */
@@ -367,6 +390,13 @@ private:
uint64 *readyTimestamps; /**< Heap: [maxBatchCount] HRT for completed ready batch */
uint64 *scratchTimestamps; /**< Heap: [maxBatchCount] background-thread local copy */
uint32 readyFill; /**< Snapshot count in the ready batch */
/** Strict/Decimate: readyBuffer holds a snapshot the sender has not taken
* yet. Publication state must live here rather than in dataSem, because
* EventSem::ResetWait resets before waiting and so destroys any Post that
* landed while the sender was busy. */
bool readySnapshotPending;
uint32 droppedPublications; /**< Publications overwritten before being sent */
uint64 lastDropReportTicks; /**< Sender-thread rate limit for the drop warning */
/* Decimate mode */
uint32 decimateRatio; /**< Send 1 packet every decimateRatio Synchronise() calls */
uint32 decimateCounter; /**< Current decimate cycle counter */
@@ -674,6 +674,14 @@ bool DebugService::SendUDPSConfig() {
payloadOffset++;
}
// Write this host's HRT tick rate: DATA packets carry raw counter values,
// which a receiver on another machine cannot convert to seconds without it.
if ((payloadOffset + 8u) <= CFG_BUF_SIZE) {
uint64 hrtFrequency = HighResolutionTimer::Frequency();
memcpy(payload + payloadOffset, &hrtFrequency, 8u);
payloadOffset += 8u;
}
udpsNumSlots = newNumSlots;
mutex.FastUnLock();
@@ -36,7 +36,13 @@ UDPSClient::UDPSClient()
disconnectTick(0u),
lastKeepAliveTicks(0u),
localPort(0u),
lastGcTicks(0u) {
lastGcTicks(0u),
lastDropWarnTicks(0u),
droppedSinceWarn(0u),
lastDataCounter(0u),
lastDataCounterValid(false),
lastDataGap(0u),
staleDataPackets(0u) {
for (uint32 i = 0u; i < UDPS_CLIENT_MAX_REASSEMBLY_SLOTS; i++) {
reassemblySlots[i].counter = 0u;
@@ -46,7 +52,11 @@ UDPSClient::UDPSClient()
reassemblySlots[i].active = false;
reassemblySlots[i].firstSeenTicks = 0u;
reassemblySlots[i].chunkSize = 0u;
(void) MemoryOperationsHelper::Set(reassemblySlots[i].recvMask, 0, 32u);
reassemblySlots[i].assembledBytes = 0u;
reassemblySlots[i].pendingTailBytes = 0u;
reassemblySlots[i].pendingTailValid = false;
(void) MemoryOperationsHelper::Set(reassemblySlots[i].recvMask, 0,
UDPS_CLIENT_RECV_MASK_BYTES);
}
}
@@ -228,6 +238,12 @@ bool UDPSClient::Connect() {
connected = true;
lastDataTicks = HighResolutionTimer::Counter();
lastKeepAliveTicks = lastDataTicks;
/* The producer's packetCounter restarts independently of ours, so a
* counter carried over from the previous connection would make the
* sequence gate reject the whole new stream as stale. */
lastDataCounterValid = false;
lastDataCounter = 0u;
lastDataGap = 0u;
if (listener != NULL_PTR(UDPSClientListener *)) {
listener->OnUDPSConnected();
}
@@ -498,6 +514,14 @@ bool UDPSClient::ReceiveAndProcess() {
return true; // only the TCP socket was readable
}
/* Drain the socket rather than taking one datagram per Execute() iteration:
* a fragmented high-rate source delivers datagrams far faster than the
* select/read round trip retires them, and the resulting kernel-buffer
* overflow shows up as lost fragments i.e. as packets that can never be
* reassembled. Bounded so the silence and keepalive checks in Execute()
* still run under a sustained flood. */
uint32 drained = 0u;
while (drained < UDPS_CLIENT_MAX_DATAGRAMS_PER_CYCLE) {
uint32 recvSize = static_cast<uint32>(sizeof(recvBuf));
bool ok;
if (useMulticast) {
@@ -517,6 +541,18 @@ bool UDPSClient::ReceiveAndProcess() {
lastDataTicks = HighResolutionTimer::Counter();
ProcessDatagram(recvBuf, recvSize);
drained++;
/* Stop as soon as the socket runs dry: Read() would otherwise block. */
fd_set dset;
FD_ZERO(&dset);
FD_SET(fd, &dset);
struct timeval zero;
zero.tv_sec = 0; zero.tv_usec = 0;
if (select(fd + 1, &dset, NULL, NULL, &zero) <= 0) {
break;
}
}
return true;
}
@@ -599,7 +635,7 @@ void UDPSClient::ProcessDatagram(const uint8 *buf, uint32 size) {
if (hdr->type == UDPS_TYPE_CONFIG) {
listener->OnUDPSConfig(pl, payloadBytes);
}
else {
else if (AcceptDataCounter(hdr->counter)) {
listener->OnUDPSData(pl, payloadBytes);
}
}
@@ -616,21 +652,83 @@ void UDPSClient::ProcessDatagram(const uint8 *buf, uint32 size) {
// Private: PlaceFragment
// ---------------------------------------------------------------------------
uint32 UDPSClient::AcquireReassemblySlot(uint32 counter, uint8 type) {
for (uint32 i = 0u; i < UDPS_CLIENT_MAX_REASSEMBLY_SLOTS; i++) {
if (!reassemblySlots[i].active) {
return i;
}
}
/* All slots busy. The producer emits packets sequentially, so a slot
* holding an OLDER counter of the SAME stream is provably dead: its
* missing fragments were sent before the ones arriving now and will never
* turn up. Reclaiming it immediately instead of waiting out the 2 s GC
* is what keeps a handful of lost fragments from wedging the whole table.
* The counter is a wrapping uint32, so compare via the signed difference. */
uint32 victim = UDPS_CLIENT_MAX_REASSEMBLY_SLOTS;
int32 bestDist = 0;
for (uint32 i = 0u; i < UDPS_CLIENT_MAX_REASSEMBLY_SLOTS; i++) {
if (reassemblySlots[i].type != type) {
continue;
}
int32 dist = static_cast<int32>(counter - reassemblySlots[i].counter);
if ((dist > 0) && (dist > bestDist)) {
bestDist = dist;
victim = i;
}
}
if (victim >= UDPS_CLIENT_MAX_REASSEMBLY_SLOTS) {
/* Nothing is provably dead (e.g. the other stream owns every slot):
* fall back to the least recently started. */
uint64 oldestTick = 0xFFFFFFFFFFFFFFFFuLL;
victim = 0u;
for (uint32 i = 0u; i < UDPS_CLIENT_MAX_REASSEMBLY_SLOTS; i++) {
if (reassemblySlots[i].firstSeenTicks < oldestTick) {
oldestTick = reassemblySlots[i].firstSeenTicks;
victim = i;
}
}
}
NoteDroppedIncomplete(reassemblySlots[victim].counter);
return victim;
}
void UDPSClient::NoteDroppedIncomplete(uint32 counter) {
droppedSinceWarn++;
uint64 now = HighResolutionTimer::Counter();
if ((now - lastDropWarnTicks) >= HighResolutionTimer::Frequency()) {
REPORT_ERROR_STATIC(ErrorManagement::Warning,
"UDPSClient: dropped %u incomplete packet(s) in the last "
"second (latest counter %u); fragments are being lost.",
droppedSinceWarn, counter);
droppedSinceWarn = 0u;
lastDropWarnTicks = now;
}
}
bool UDPSClient::PlaceFragment(const UDPSPacketHeader *hdr,
const uint8 *payload,
uint32 payloadBytes) {
uint32 counter = hdr->counter;
uint8 type = hdr->type;
uint16 fragIdx = hdr->fragmentIdx;
uint16 totalFrags = hdr->totalFragments;
if ((fragIdx >= totalFrags) || (totalFrags > 512u)) {
if ((fragIdx >= totalFrags) ||
(static_cast<uint32>(totalFrags) > UDPS_CLIENT_MAX_FRAGMENTS)) {
return false; // sanity check
}
// Find existing slot for this counter
/* Slots are keyed on (counter, type): DATA and CONFIG carry independent
* counter sequences, so the same counter value legitimately appears on
* both, and matching on the counter alone merges the two streams into one
* slot one payload is delivered under the wrong type, the other is lost. */
uint32 slot = UDPS_CLIENT_MAX_REASSEMBLY_SLOTS;
for (uint32 i = 0u; i < UDPS_CLIENT_MAX_REASSEMBLY_SLOTS; i++) {
if (reassemblySlots[i].active && (reassemblySlots[i].counter == counter)) {
if (reassemblySlots[i].active && (reassemblySlots[i].counter == counter) &&
(reassemblySlots[i].type == type)) {
slot = i;
break;
}
@@ -638,34 +736,20 @@ bool UDPSClient::PlaceFragment(const UDPSPacketHeader *hdr,
// Allocate new slot if not found
if (slot >= UDPS_CLIENT_MAX_REASSEMBLY_SLOTS) {
for (uint32 i = 0u; i < UDPS_CLIENT_MAX_REASSEMBLY_SLOTS; i++) {
if (!reassemblySlots[i].active) {
slot = i;
break;
}
}
if (slot >= UDPS_CLIENT_MAX_REASSEMBLY_SLOTS) {
// All slots occupied — evict the oldest
uint64 oldestTick = 0xFFFFFFFFFFFFFFFFuLL;
for (uint32 i = 0u; i < UDPS_CLIENT_MAX_REASSEMBLY_SLOTS; i++) {
if (reassemblySlots[i].firstSeenTicks < oldestTick) {
oldestTick = reassemblySlots[i].firstSeenTicks;
slot = i;
}
}
REPORT_ERROR_STATIC(ErrorManagement::Warning,
"UDPSClient: Reassembly slots full; evicting oldest.");
}
slot = AcquireReassemblySlot(counter, type);
reassemblySlots[slot].counter = counter;
reassemblySlots[slot].type = hdr->type;
reassemblySlots[slot].type = type;
reassemblySlots[slot].totalFragments = totalFrags;
reassemblySlots[slot].receivedFragments = 0u;
reassemblySlots[slot].active = true;
reassemblySlots[slot].firstSeenTicks = HighResolutionTimer::Counter();
reassemblySlots[slot].chunkSize = 0u;
reassemblySlots[slot].assembledBytes = 0u;
(void) MemoryOperationsHelper::Set(reassemblySlots[slot].recvMask, 0, 32u);
reassemblySlots[slot].pendingTailBytes = 0u;
reassemblySlots[slot].pendingTailValid = false;
(void) MemoryOperationsHelper::Set(reassemblySlots[slot].recvMask, 0,
UDPS_CLIENT_RECV_MASK_BYTES);
}
UDPSReassemblySlot &s = reassemblySlots[slot];
@@ -673,27 +757,56 @@ bool UDPSClient::PlaceFragment(const UDPSPacketHeader *hdr,
// Skip duplicate
uint32 byteIdx = fragIdx / 8u;
uint8 bitMask = static_cast<uint8>(1u << (fragIdx % 8u));
if (byteIdx < 32u) {
if ((s.recvMask[byteIdx] & bitMask) != 0u) {
return false; // already have this fragment
}
const bool isLastFragment = ((static_cast<uint32>(fragIdx) + 1u) ==
static_cast<uint32>(totalFrags));
/* Every fragment but the last carries a full chunk, so any of them reveals
* the chunk size waiting specifically for fragment 0 means a merely
* reordered burst, with nothing lost, destroys the packet. */
if ((s.chunkSize == 0u) && !isLastFragment) {
s.chunkSize = payloadBytes;
}
// Compute placement offset
uint32 chunkSize = s.chunkSize;
if (chunkSize == 0u) {
// Learn chunk size from first non-last fragment
if (fragIdx == 0u) {
chunkSize = payloadBytes;
s.chunkSize = chunkSize;
}
else {
// Can't place yet without knowing chunk size — drop (rare edge case)
if (s.chunkSize == 0u) {
/* The last fragment arrived before any full-size one: its offset is
* not computable yet, so hold it until the chunk size is known. */
if (payloadBytes > UDPS_CLIENT_PENDING_TAIL_BYTES) {
return false;
}
if (payloadBytes > 0u) {
(void) MemoryOperationsHelper::Copy(s.pendingTail, payload, payloadBytes);
}
s.pendingTailBytes = payloadBytes;
s.pendingTailValid = true;
s.recvMask[byteIdx] |= bitMask;
s.receivedFragments++;
return false; // totalFrags > 1 here, so this can never complete a packet
}
uint32 offset = static_cast<uint32>(fragIdx) * chunkSize;
// Flush a deferred last fragment now that the chunk size is known.
if (s.pendingTailValid) {
uint32 tailOffset = (static_cast<uint32>(s.totalFragments) - 1u) * s.chunkSize;
if ((tailOffset + s.pendingTailBytes) > static_cast<uint32>(sizeof(s.payload))) {
s.active = false;
NoteDroppedIncomplete(s.counter);
return false; // overflow guard
}
if (s.pendingTailBytes > 0u) {
(void) MemoryOperationsHelper::Copy(s.payload + tailOffset,
s.pendingTail, s.pendingTailBytes);
}
if ((tailOffset + s.pendingTailBytes) > s.assembledBytes) {
s.assembledBytes = tailOffset + s.pendingTailBytes;
}
s.pendingTailValid = false;
s.pendingTailBytes = 0u;
}
uint32 offset = static_cast<uint32>(fragIdx) * s.chunkSize;
if ((offset + payloadBytes) > static_cast<uint32>(sizeof(s.payload))) {
return false; // overflow guard
}
@@ -709,9 +822,7 @@ bool UDPSClient::PlaceFragment(const UDPSPacketHeader *hdr,
s.assembledBytes = offset + payloadBytes;
}
if (byteIdx < 32u) {
s.recvMask[byteIdx] |= bitMask;
}
s.receivedFragments++;
// Check if complete
@@ -727,6 +838,30 @@ bool UDPSClient::PlaceFragment(const UDPSPacketHeader *hdr,
// Private: DeliverAssembled
// ---------------------------------------------------------------------------
bool UDPSClient::AcceptDataCounter(uint32 counter) {
if (!lastDataCounterValid) {
lastDataCounterValid = true;
lastDataCounter = counter;
lastDataGap = 0u;
return true;
}
// The counter is a wrapping uint32, so order it by the signed difference:
// that stays correct across the wrap, where a plain comparison would call
// the first packet after it stale and reject the stream from then on.
int32 delta = static_cast<int32>(counter - lastDataCounter);
if (delta <= 0) {
staleDataPackets++;
return false;
}
lastDataGap = static_cast<uint32>(delta) - 1u;
lastDataCounter = counter;
return true;
}
uint32 UDPSClient::GetLastDataGap() const { return lastDataGap; }
uint32 UDPSClient::GetStaleDataPackets() const { return staleDataPackets; }
void UDPSClient::DeliverAssembled(UDPSReassemblySlot &s) {
if (listener == NULL_PTR(UDPSClientListener *)) {
return;
@@ -739,7 +874,7 @@ void UDPSClient::DeliverAssembled(UDPSReassemblySlot &s) {
if (s.type == UDPS_TYPE_CONFIG) {
listener->OnUDPSConfig(s.payload, totalSize);
}
else {
else if (AcceptDataCounter(s.counter)) {
listener->OnUDPSData(s.payload, totalSize);
}
}
@@ -757,10 +892,8 @@ void UDPSClient::GcReassemblySlots() {
continue;
}
if ((now - reassemblySlots[i].firstSeenTicks) > staleThreshold) {
REPORT_ERROR_STATIC(ErrorManagement::Warning,
"UDPSClient: Discarding stale reassembly slot (counter %u).",
reassemblySlots[i].counter);
reassemblySlots[i].active = false;
NoteDroppedIncomplete(reassemblySlots[i].counter);
}
}
}
@@ -82,10 +82,27 @@ public:
* update for one source is fragmented into MaxPayloadSize chunks; this is
* the ceiling on the reassembled total, so it bounds the largest multi-
* fragment packet the client can deliver. Sized for large array bursts
* (e.g. 8x10000 float32 ~= 320 KiB) with headroom; stays well within the
* 256-fragment span the recvMask[32] tracks at typical chunk sizes. */
* (e.g. 8x10000 float32 ~= 320 KiB) with headroom. */
static const uint32 UDPS_CLIENT_MAX_PACKET_BYTES = 1048576u; // 1 MiB
/** Maximum fragment count accepted for one packet. The received-fragment
* bitmask must cover this whole span: a fragment index the mask cannot
* represent has no duplicate detection, so a duplicated datagram counts
* twice and the packet is delivered with a fragment still missing. */
static const uint32 UDPS_CLIENT_MAX_FRAGMENTS = 512u;
/** Bytes of received-fragment bitmask (one bit per fragment). */
static const uint32 UDPS_CLIENT_RECV_MASK_BYTES =
UDPS_CLIENT_MAX_FRAGMENTS / 8u;
/** Size of the per-slot buffer that holds a last fragment which arrived
* before the chunk size was known. Fragments larger than this cannot be
* deferred and are dropped (the packet then fails to reassemble). */
static const uint32 UDPS_CLIENT_PENDING_TAIL_BYTES = 8192u;
/** Maximum datagrams drained from the socket per Execute() iteration. */
static const uint32 UDPS_CLIENT_MAX_DATAGRAMS_PER_CYCLE = 256u;
/** Default silence timeout before reconnect (seconds); sub-second values allowed. */
static const float32 UDPS_CLIENT_DEFAULT_SILENCE_TIMEOUT_S = 1.0f;
@@ -155,6 +172,22 @@ public:
*/
virtual ErrorManagement::ErrorType Execute(ExecutionInfo &info);
/**
* @brief DATA packets that went missing immediately before the one being
* delivered, from the gap in the producer's packet counter.
* @details Valid for the duration of the OnUDPSData() callback. A listener
* that reconstructs per-sample timestamps needs this: without it, the time
* elapsed since the previous packet looks like it covers only that
* packet's samples, so the inferred sample period comes out too long and
* the samples are spread past where they belong.
*/
uint32 GetLastDataGap() const;
/**
* @brief DATA packets discarded for arriving after a newer one.
*/
uint32 GetStaleDataPackets() const;
private:
// -------------------------------------------------------------------------
@@ -165,12 +198,19 @@ private:
uint8 type; ///< UDPS_TYPE_DATA or UDPS_TYPE_CONFIG
uint16 totalFragments; ///< Expected fragment count
uint16 receivedFragments; ///< How many we have so far
uint8 recvMask[32]; ///< Bitmask: bit f set iff fragment f received
uint8 recvMask[UDPS_CLIENT_RECV_MASK_BYTES]; ///< Bit f set iff fragment f received
uint8 payload[UDPS_CLIENT_MAX_PACKET_BYTES]; ///< Assembled payload buffer
uint64 firstSeenTicks; ///< For GC (2 s stale detection)
bool active; ///< Slot in use
uint32 chunkSize; ///< Payload bytes per fragment (from first fragment)
uint32 chunkSize; ///< Payload bytes per fragment (any non-last fragment)
uint32 assembledBytes; ///< Exact total payload bytes placed so far
/** Last fragment received before chunkSize was known: its offset is
* not yet computable, so it waits here until a full-size fragment
* reveals the chunk size. Only the last fragment can ever be short,
* hence one deferred fragment per slot is enough. */
uint8 pendingTail[UDPS_CLIENT_PENDING_TAIL_BYTES];
uint32 pendingTailBytes;
bool pendingTailValid;
};
// -------------------------------------------------------------------------
@@ -195,8 +235,39 @@ private:
bool ReadExactTCP(uint8 *dst, uint32 n);
/** @return true iff this fragment completed the reassembly (payload delivered). */
bool PlaceFragment(const UDPSPacketHeader *hdr, const uint8 *payload, uint32 payloadBytes);
/**
* @brief Reserve a reassembly slot for (@p counter, @p type), reclaiming
* one if none is free.
* @return the slot index (always valid).
*/
uint32 AcquireReassemblySlot(uint32 counter, uint8 type);
/**
* @brief Account one packet abandoned with fragments missing, and report
* it at most once per second.
* @details Fragment loss on a busy stream is chronic, not exceptional: an
* unconditional message per drop buries every other log line.
*/
void NoteDroppedIncomplete(uint32 counter);
void GcReassemblySlots();
void DeliverAssembled(UDPSReassemblySlot &slot);
/**
* @brief Sequence gate for DATA packets, applied just before delivery.
* @details The producer numbers DATA packets consecutively, so the counter
* reveals both how many packets went missing and whether this one is late.
* A late packet must not be delivered: its samples predate what the
* listener has already stored, so they land behind the current write
* position and collide with data that is already there which is what a
* consumer sees as two signals occupying the same instant. Reassembly
* completes in arrival order, not counter order, so this ordering is not
* guaranteed upstream.
*
* Also records the number of packets missing immediately before this one,
* for GetLastDataGap().
*
* @param counter The candidate packet's UDPS counter.
* @return true if the packet is newer than the last delivered one.
*/
bool AcceptDataCounter(uint32 counter);
// -------------------------------------------------------------------------
// Configuration
@@ -236,6 +307,14 @@ private:
// Reassembly
UDPSReassemblySlot reassemblySlots[UDPS_CLIENT_MAX_REASSEMBLY_SLOTS];
uint64 lastGcTicks; ///< Ticks at last GC run
uint64 lastDropWarnTicks;///< Ticks at last incomplete-packet report
uint32 droppedSinceWarn; ///< Incomplete packets since that report
// DATA sequencing (see AcceptDataCounter)
uint32 lastDataCounter; ///< Counter of the last delivered DATA packet
bool lastDataCounterValid; ///< False until the first DATA packet
uint32 lastDataGap; ///< Packets missing before the current one
uint32 staleDataPackets; ///< DATA packets discarded as late
// Receive scratch buffer
uint8 recvBuf[65535u + UDPS_HEADER_SIZE];
@@ -0,0 +1,162 @@
/**
* @file AccumDtGTest.cpp
* @brief Tests UDPSEstimateAccumDt, the per-sample period estimator used for
* accumulated scalars that carry no SamplingRate.
*
* The estimator exists because the natural formula sender-clock gap divided
* by the previous packet's sample count is only correct while no packet is
* lost. When one is, the gap covers cycles that count never saw and the period
* comes out too large, which spreads the packet's samples past their real end
* and into the range the next packet claims. The loss count comes from the
* producer's packet counter rather than being inferred from the gap itself, so
* these tests pin both sides: the estimate must not move when packets go
* missing, and it must still follow a genuine rate change a cycle count
* inferred from the estimate's own period would lock onto the old one.
*
* @copyright Copyright 2015 F4E | European Joint Undertaking for ITER and
* the Development of Fusion Energy ('Fusion for Energy').
* Licensed under the EUPL, Version 1.1 or - as soon they will be approved
* by the European Commission - subsequent versions of the EUPL (the "Licence")
* You may not use this work except in compliance with the Licence.
* You may obtain a copy of the Licence at: http://ec.europa.eu/idabc/eupl
*
* @warning Unless required by applicable law or agreed to in writing,
* software distributed under the Licence is distributed on an "AS IS"
* basis, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
* or implied. See the Licence permissions and limitations under the Licence.
*/
#include <gtest/gtest.h>
#include "UDPSourceSession.h"
using MARTe::float64;
using MARTe::uint32;
using StreamHub::UDPSEstimateAccumDt;
namespace {
/** A producer emitting batches of BATCH cycles at a period of DT seconds. */
const float64 kDt = 1.0e-3;
const uint32 kBatch = 10u;
const float64 kGap = kDt * static_cast<float64>(kBatch);
/** Feeds n clean packets and returns the settled estimate. */
float64 Warmup(uint32 n, float64 &dtEMA, bool &dtValid) {
float64 dt = 0.0;
for (uint32 i = 0u; i < n; i++) {
dt = UDPSEstimateAccumDt(kGap, kBatch, 0u, dtEMA, dtValid);
}
return dt;
}
} // namespace
/* The first packet has nothing to go on but the previous sample count, so it
* must fall back to gap/prevN rather than to some fixed default. */
TEST(AccumDtGTest, BootstrapsFromPreviousSampleCount) {
float64 dtEMA = 0.0;
bool dtValid = false;
const float64 dt = UDPSEstimateAccumDt(kGap, kBatch, 0u, dtEMA, dtValid);
EXPECT_TRUE(dtValid);
EXPECT_NEAR(kDt, dt, 1.0e-12);
}
/* A clean stream must hold the period steady, not drift. */
TEST(AccumDtGTest, SteadyStreamStaysOnPeriod) {
float64 dtEMA = 0.0;
bool dtValid = false;
const float64 dt = Warmup(50u, dtEMA, dtValid);
EXPECT_NEAR(kDt, dt, 1.0e-9);
}
/* The regression this whole estimator is for: one packet is lost, so the gap
* doubles while prevN does not. Dividing by prevN would report 2x the true
* period enough to walk a 10-sample batch a full batch past its own end. */
TEST(AccumDtGTest, LostPacketDoesNotInflatePeriod) {
float64 dtEMA = 0.0;
bool dtValid = false;
(void) Warmup(50u, dtEMA, dtValid);
const float64 dt = UDPSEstimateAccumDt(2.0 * kGap, kBatch, 1u, dtEMA, dtValid);
/* What the naive formula would have produced. */
const float64 naive = (2.0 * kGap) / static_cast<float64>(kBatch);
EXPECT_NEAR(2.0 * kDt, naive, 1.0e-12);
EXPECT_NEAR(kDt, dt, 1.0e-6);
}
/* Several consecutive losses are the same situation, just wider. */
TEST(AccumDtGTest, MultiplePacketLossDoesNotInflatePeriod) {
float64 dtEMA = 0.0;
bool dtValid = false;
(void) Warmup(50u, dtEMA, dtValid);
for (uint32 missing = 1u; missing <= 5u; missing++) {
const float64 span = static_cast<float64>(missing + 1u) * kGap;
const float64 dt = UDPSEstimateAccumDt(span, kBatch, missing, dtEMA,
dtValid);
EXPECT_NEAR(kDt, dt, 1.0e-6) << "after " << missing << " lost packet(s)";
}
}
/* Loss must not leave the estimator poisoned for the packets that follow. */
TEST(AccumDtGTest, RecoversToCleanStreamAfterLoss) {
float64 dtEMA = 0.0;
bool dtValid = false;
(void) Warmup(50u, dtEMA, dtValid);
(void) UDPSEstimateAccumDt(3.0 * kGap, kBatch, 2u, dtEMA, dtValid);
const float64 dt = Warmup(20u, dtEMA, dtValid);
EXPECT_NEAR(kDt, dt, 1.0e-6);
}
/* A real, sustained rate change must still be followed — the estimator is a
* smoother, not a latch. Half the period is exactly on the rejection boundary,
* so use a change that lands inside the accepted band. */
TEST(AccumDtGTest, FollowsSustainedRateChange) {
float64 dtEMA = 0.0;
bool dtValid = false;
(void) Warmup(50u, dtEMA, dtValid);
const float64 newDt = kDt * 0.75;
const float64 newGap = newDt * static_cast<float64>(kBatch);
float64 dt = 0.0;
for (uint32 i = 0u; i < 400u; i++) {
dt = UDPSEstimateAccumDt(newGap, kBatch, 0u, dtEMA, dtValid);
}
EXPECT_NEAR(newDt, dt, 1.0e-6);
}
/* A batch that carries fewer cycles than usual (a time-triggered flush) is not
* loss: the gap shrinks with it, so the period must not shrink too. */
TEST(AccumDtGTest, ShortBatchDoesNotDeflatePeriod) {
float64 dtEMA = 0.0;
bool dtValid = false;
(void) Warmup(50u, dtEMA, dtValid);
const uint32 shortBatch = 3u;
const float64 dt = UDPSEstimateAccumDt(
kDt * static_cast<float64>(shortBatch), shortBatch, 0u, dtEMA, dtValid);
EXPECT_NEAR(kDt, dt, 1.0e-6);
}
/* A gap shorter than one period cannot mean zero cycles; the divisor is
* clamped so the estimate can never be driven to infinity. */
TEST(AccumDtGTest, SubPeriodGapDoesNotExplode) {
float64 dtEMA = 0.0;
bool dtValid = false;
(void) Warmup(50u, dtEMA, dtValid);
const float64 dt = UDPSEstimateAccumDt(kDt * 1.0e-3, 1u, 0u, dtEMA, dtValid);
EXPECT_NEAR(kDt, dt, 1.0e-6);
}
@@ -0,0 +1,151 @@
/**
* @file ConfigHrtFreqGTest.cpp
* @brief Tests UDPSConfigHrtFrequency, which picks the tick rate used to turn
* a producer's DATA timestamps into seconds.
*
* DATA packets carry the raw value of the producer's high-resolution counter.
* Until the rate was published in CONFIG the hub divided by its own timer's
* frequency, which is only right while the producer runs on the same host
* on x86 that number is the TSC frequency and differs from model to model, so
* off-box every accumulated batch was laid out over the wrong span of time.
*
* The field is a trailer, so these tests pin both directions: a payload that
* carries it must be believed, and one that stops early an older producer
* must still decode against the local fallback rather than against zero.
*
* @copyright Copyright 2015 F4E | European Joint Undertaking for ITER and
* the Development of Fusion Energy ('Fusion for Energy').
* Licensed under the EUPL, Version 1.1 or - as soon they will be approved
* by the European Commission - subsequent versions of the EUPL (the "Licence")
* You may not use this work except in compliance with the Licence.
* You may obtain a copy of the Licence at: http://ec.europa.eu/idabc/eupl
*
* @warning Unless required by applicable law or agreed to in writing,
* software distributed under the Licence is distributed on an "AS IS"
* basis, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
* or implied. See the Licence permissions and limitations under the Licence.
*/
#include <gtest/gtest.h>
#include <string.h>
#include "UDPSourceSession.h"
using MARTe::uint8;
using MARTe::uint32;
using MARTe::uint64;
using MARTe::float64;
using MARTe::UDPS_SIGNAL_DESC_SIZE;
using StreamHub::UDPSConfigHrtFrequency;
namespace {
/** This hub's own timer rate, i.e. what the code must fall back to. */
const float64 kLocalFreq = 1.0e9;
/** A plausible producer rate that is deliberately not kLocalFreq. */
const uint64 kWireFreq = 2400000000ULL;
const uint32 kNumSigs = 2u;
/** Offset of the CONFIG trailer that follows the publish-mode byte. */
uint32 TrailerOffset(uint32 numSigs) {
return 4u + (numSigs * UDPS_SIGNAL_DESC_SIZE) + 1u;
}
/**
* Builds a CONFIG payload for kNumSigs signals.
* @param withFreq Append the 8-byte HRT frequency trailer.
* @param freq Value to append when @p withFreq.
* @param[out] size Bytes written.
*/
const uint8 *BuildConfig(bool withFreq, uint64 freq, uint32 &size) {
static uint8 buf[4u + (kNumSigs * UDPS_SIGNAL_DESC_SIZE) + 1u + 8u];
(void) memset(buf, 0, sizeof(buf));
(void) memcpy(buf, &kNumSigs, 4u);
size = TrailerOffset(kNumSigs);
if (withFreq) {
(void) memcpy(buf + size, &freq, 8u);
size += 8u;
}
return buf;
}
} // namespace
/* The whole point of the field: a producer that publishes its rate is believed
* even when the hub's own timer runs at a different one. */
TEST(ConfigHrtFreqGTest, AdoptsThePublishedRate) {
uint32 size = 0u;
const uint8 *cfg = BuildConfig(true, kWireFreq, size);
EXPECT_DOUBLE_EQ(static_cast<float64>(kWireFreq),
UDPSConfigHrtFrequency(cfg, size, kNumSigs, kLocalFreq));
}
/* A producer older than the field stops after the publish-mode byte. Reading
* past it would take whatever follows in the receive buffer as a frequency. */
TEST(ConfigHrtFreqGTest, FallsBackWhenTheTrailerIsAbsent) {
uint32 size = 0u;
const uint8 *cfg = BuildConfig(false, 0u, size);
EXPECT_DOUBLE_EQ(kLocalFreq,
UDPSConfigHrtFrequency(cfg, size, kNumSigs, kLocalFreq));
}
/* A trailer cut short mid-field is not a frequency either; taking the bytes
* that are there would assemble one out of whatever the rest of the buffer
* holds. */
TEST(ConfigHrtFreqGTest, FallsBackOnATruncatedTrailer) {
uint32 size = 0u;
const uint8 *cfg = BuildConfig(true, kWireFreq, size);
EXPECT_DOUBLE_EQ(kLocalFreq,
UDPSConfigHrtFrequency(cfg, size - 1u, kNumSigs,
kLocalFreq));
}
/* Zero is the protocol's "I do not know my own rate". Dividing by it yields
* infinities that propagate into every timestamp. */
TEST(ConfigHrtFreqGTest, FallsBackOnTheUnknownSentinel) {
uint32 size = 0u;
const uint8 *cfg = BuildConfig(true, MARTe::UDPS_HRT_FREQUENCY_UNKNOWN,
size);
EXPECT_DOUBLE_EQ(kLocalFreq,
UDPSConfigHrtFrequency(cfg, size, kNumSigs, kLocalFreq));
}
/* No high-resolution timer ticks slower than 1 kHz, so a value that low means
* the payload was misread. Adopting it would stretch a millisecond batch
* across whole seconds. */
TEST(ConfigHrtFreqGTest, FallsBackOnAnImplausiblyLowRate) {
uint32 size = 0u;
const uint8 *cfg = BuildConfig(true, 999u, size);
EXPECT_DOUBLE_EQ(kLocalFreq,
UDPSConfigHrtFrequency(cfg, size, kNumSigs, kLocalFreq));
}
/* The trailer sits after the descriptors, so its offset moves with the signal
* count; a fixed offset would read descriptor bytes on any other config. */
TEST(ConfigHrtFreqGTest, LocatesTheTrailerAfterTheDescriptors) {
const uint32 numSigs = 7u;
const uint32 size = TrailerOffset(numSigs) + 8u;
uint8 buf[4u + (7u * UDPS_SIGNAL_DESC_SIZE) + 1u + 8u];
/* Fill the descriptor area with a byte pattern that would decode as a
* plausible frequency if the offset were wrong. */
(void) memset(buf, 0x11, sizeof(buf));
(void) memcpy(buf, &numSigs, 4u);
(void) memcpy(buf + TrailerOffset(numSigs), &kWireFreq, 8u);
EXPECT_DOUBLE_EQ(static_cast<float64>(kWireFreq),
UDPSConfigHrtFrequency(buf, size, numSigs, kLocalFreq));
}
/* A null payload must not be dereferenced: CONFIG arrives from the network. */
TEST(ConfigHrtFreqGTest, FallsBackOnANullPayload) {
EXPECT_DOUBLE_EQ(kLocalFreq,
UDPSConfigHrtFrequency(NULL_PTR(const uint8 *), 1024u,
kNumSigs, kLocalFreq));
}
+1 -1
View File
@@ -22,7 +22,7 @@
#
#############################################################
OBJSX = TriggerEngineSrc.x BinaryRecorderSrc.x SignalRingBufferGTest.x TriggerEngineGTest.x LTTBGTest.x BinaryRecorderGTest.x BoundsCheckTest.x WSServerBufferTest.x
OBJSX = TriggerEngineSrc.x BinaryRecorderSrc.x SignalRingBufferGTest.x TriggerEngineGTest.x LTTBGTest.x BinaryRecorderGTest.x BoundsCheckTest.x WSServerBufferTest.x AccumDtGTest.x ConfigHrtFreqGTest.x
PACKAGE=Applications
ROOT_DIR=../../..
@@ -196,6 +196,183 @@ TEST(TriggerEngineGTest, TestRearmResetsEdgeDetection) {
EXPECT_DOUBLE_EQ(4.0, tt);
}
/* An edge that arrives while a capture is still being collected, or while it is
* being handed out, used to be dropped on the floor: CheckSample returned early
* in every state but ARMED, and the automatic rearm then waited for a FRESH
* edge. The engine is therefore deaf from its own trigger point until the
* capture has been harvested a post-window and then for the holdoff on top.
*
* On a sparse pulse train that rounds the capture spacing up to a whole pulse
* period: at the default 1 s window the blind stretch is 1 s, so a 1 Hz train
* was caught at 0.5 Hz and a wider window lost whole multiples. Remembering the
* edge costs nothing, because the capture is built from the edge's own
* timestamp out of a ring that still holds everything around it. */
TEST(TriggerEngineGTest, TestEdgeDuringCaptureFiresOnRearm) {
TriggerEngine eng;
eng.SetConfig(MakeConfig(kEdgeRising, 0.5, 1.0, 20.0)); /* post = 0.8 */
eng.Arm();
eng.CheckSample(1.0, 0.0);
eng.CheckSample(1.1, 1.0);
ASSERT_EQ(kTrigCollecting, eng.GetState());
/* A second pulse, clear of the capture in flight (1.1 + 0.8 = 1.9). */
eng.CheckSample(2.4, 0.0);
eng.CheckSample(2.5, 1.0);
eng.MarkTriggered();
eng.Rearm();
EXPECT_EQ(kTrigCollecting, eng.GetState());
float64 tt, pre, post;
ASSERT_TRUE(eng.GetFiredWindow(tt, pre, post));
EXPECT_DOUBLE_EQ(2.5, tt); /* the remembered edge, not the rearm instant */
}
/* The remembered edge must not be one the capture in flight already covers, nor
* one inside the holdoff that guard exists to stop the ringing of a single
* event re-triggering on itself, and it is measured from the trigger point, so
* the two overlap rather than add. */
TEST(TriggerEngineGTest, TestEdgeInsideOwnCaptureIsNotRemembered) {
TriggerEngine eng;
eng.SetConfig(MakeConfig(kEdgeRising, 0.5, 1.0, 20.0)); /* post = 0.8 */
eng.Arm();
eng.CheckSample(1.0, 0.0);
eng.CheckSample(1.1, 1.0);
ASSERT_EQ(kTrigCollecting, eng.GetState());
/* Inside 1.1 + max(0.8, 0.2 holdoff) = 1.9: the capture owns this stretch. */
eng.CheckSample(1.4, 0.0);
eng.CheckSample(1.5, 1.0);
eng.MarkTriggered();
eng.Rearm();
EXPECT_EQ(kTrigArmed, eng.GetState());
}
/* A holdoff longer than the post-window is what decides the guard interval. */
TEST(TriggerEngineGTest, TestHoldoffOutlastingPostWindowGovernsRearm) {
TriggerEngine eng;
TriggerConfig cfg = MakeConfig(kEdgeRising, 0.5, 1.0, 80.0); /* post = 0.2 */
cfg.holdoffSec = 2.0;
eng.SetConfig(cfg);
eng.Arm();
eng.CheckSample(1.0, 0.0);
eng.CheckSample(1.1, 1.0);
ASSERT_EQ(kTrigCollecting, eng.GetState());
/* Past the post-window but inside the holdoff (1.1 + 2.0 = 3.1): ignored. */
eng.CheckSample(1.9, 0.0);
eng.CheckSample(2.0, 1.0);
/* Clear of it: remembered. */
eng.CheckSample(3.4, 0.0);
eng.CheckSample(3.5, 1.0);
eng.MarkTriggered();
eng.Rearm();
ASSERT_EQ(kTrigCollecting, eng.GetState());
float64 tt, pre, post;
ASSERT_TRUE(eng.GetFiredWindow(tt, pre, post));
EXPECT_DOUBLE_EQ(3.5, tt);
}
/* Only the first qualifying edge is worth keeping; a later one would deliver
* the same capture a pulse further on and skip the one in between. */
TEST(TriggerEngineGTest, TestFirstQualifyingEdgeWins) {
TriggerEngine eng;
eng.SetConfig(MakeConfig(kEdgeRising, 0.5, 1.0, 20.0));
eng.Arm();
eng.CheckSample(1.0, 0.0);
eng.CheckSample(1.1, 1.0);
ASSERT_EQ(kTrigCollecting, eng.GetState());
eng.CheckSample(2.4, 0.0);
eng.CheckSample(2.5, 1.0); /* first past 1.9 */
eng.CheckSample(3.4, 0.0);
eng.CheckSample(3.5, 1.0); /* later, must not displace it */
eng.MarkTriggered();
eng.Rearm();
float64 tt, pre, post;
ASSERT_TRUE(eng.GetFiredWindow(tt, pre, post));
EXPECT_DOUBLE_EQ(2.5, tt);
}
/* Arm() is the user's own arm: it asks for the next event, not for one that has
* already been and gone, so it drops anything remembered. */
TEST(TriggerEngineGTest, TestUserArmDiscardsRememberedEdge) {
TriggerEngine eng;
eng.SetConfig(MakeConfig(kEdgeRising, 0.5, 1.0, 20.0));
eng.Arm();
eng.CheckSample(1.0, 0.0);
eng.CheckSample(1.1, 1.0);
eng.CheckSample(2.4, 0.0);
eng.CheckSample(2.5, 1.0);
eng.MarkTriggered();
eng.Arm();
EXPECT_EQ(kTrigArmed, eng.GetState());
}
/* Reconfiguring drops it too: the edge would be latched against a window it was
* never judged against. */
TEST(TriggerEngineGTest, TestSetConfigDiscardsRememberedEdge) {
TriggerEngine eng;
eng.SetConfig(MakeConfig(kEdgeRising, 0.5, 1.0, 20.0));
eng.Arm();
eng.CheckSample(1.0, 0.0);
eng.CheckSample(1.1, 1.0);
eng.CheckSample(2.4, 0.0);
eng.CheckSample(2.5, 1.0);
eng.MarkTriggered();
eng.SetConfig(MakeConfig(kEdgeRising, 0.5, 2.0, 20.0));
eng.Rearm();
EXPECT_EQ(kTrigArmed, eng.GetState());
}
/* The comparator keeps running through the dead time, so the first sample after
* an automatic rearm is measured against its real predecessor rather than being
* spent seeding one. A rearm landing mid-pulse would otherwise miss that
* pulse's edge as well as the ones it slept through. */
TEST(TriggerEngineGTest, TestRearmKeepsTrackingTheLevel) {
TriggerEngine eng;
eng.SetConfig(MakeConfig(kEdgeRising, 0.5, 1.0, 20.0));
eng.Arm();
eng.CheckSample(1.0, 0.0);
eng.CheckSample(1.1, 1.0);
ASSERT_EQ(kTrigCollecting, eng.GetState());
/* Falls back low during the capture: no rising edge, nothing remembered,
* but the level is now known to be 0. */
eng.CheckSample(2.0, 0.0);
eng.MarkTriggered();
eng.Rearm();
ASSERT_EQ(kTrigArmed, eng.GetState());
eng.CheckSample(2.1, 1.0); /* 0.0 → 1.0 across 0.5, on the very first sample */
EXPECT_EQ(kTrigCollecting, eng.GetState());
float64 tt, pre, post;
ASSERT_TRUE(eng.GetFiredWindow(tt, pre, post));
EXPECT_DOUBLE_EQ(2.1, tt);
}
/* Idle is genuinely deaf: nothing is tracked and nothing is remembered, so a
* disarmed hub cannot fire the moment it is armed again. */
TEST(TriggerEngineGTest, TestDisarmDiscardsRememberedEdge) {
TriggerEngine eng;
eng.SetConfig(MakeConfig(kEdgeRising, 0.5, 1.0, 20.0));
eng.Arm();
eng.CheckSample(1.0, 0.0);
eng.CheckSample(1.1, 1.0);
eng.CheckSample(2.4, 0.0);
eng.CheckSample(2.5, 1.0);
eng.MarkTriggered();
eng.Disarm();
eng.Rearm();
EXPECT_EQ(kTrigArmed, eng.GetState());
}
TEST(TriggerEngineGTest, TestStoppedFlag) {
TriggerEngine eng;
EXPECT_FALSE(eng.GetStopped());
@@ -225,3 +225,8 @@ TEST(UDPStreamerGTest, TestExecute_MulticastConnectDataDisconnect) {
UDPStreamerTest test;
ASSERT_TRUE(test.TestExecute_MulticastConnectDataDisconnect());
}
TEST(UDPStreamerGTest, TestAccumulate_EveryPublishedCycleReachesTheWire) {
UDPStreamerTest test;
ASSERT_TRUE(test.TestAccumulate_EveryPublishedCycleReachesTheWire());
}
@@ -36,11 +36,13 @@
#include "ConfigurationDatabase.h"
#include "GAM.h"
#include "GAMScheduler.h"
#include "HighResolutionTimer.h"
#include "MemoryOperationsHelper.h"
#include "ObjectRegistryDatabase.h"
#include "RealTimeApplication.h"
#include "Sleep.h"
#include "StandardParser.h"
#include "UDPSClient.h"
#include "UDPStreamer.h"
#include "UDPStreamerTest.h"
@@ -906,6 +908,23 @@ bool UDPStreamerTest::TestExecute_ConnectDataDisconnect() {
reinterpret_cast<const UDPSPacketHeader *>(recvBuf);
ok &= (hdr->magic == UDPS_MAGIC);
ok &= (hdr->type == UDPS_TYPE_CONFIG);
/* The CONFIG trailer must carry this host's HRT tick rate: DATA
* packets timestamp with the raw counter, so a receiver on another
* machine has nothing to convert it with otherwise. */
const uint8 *payload = recvBuf + UDPS_HEADER_SIZE;
uint32 numSigs = 0u;
if (ok && (hdr->payloadBytes >= 4u)) {
(void) memcpy(&numSigs, payload, 4u);
}
const uint32 freqOff =
4u + (numSigs * UDPS_SIGNAL_DESC_SIZE) + 1u;
ok &= (hdr->payloadBytes >= (freqOff + 8u));
if (ok) {
uint64 wireFreq = 0u;
(void) memcpy(&wireFreq, payload + freqOff, 8u);
ok &= (wireFreq == HighResolutionTimer::Frequency());
}
}
}
@@ -1845,3 +1864,298 @@ bool UDPStreamerTest::TestExecute_MulticastConnectDataDisconnect() {
ObjectRegistryDatabase::Instance()->Purge();
return ok;
}
/*---------------------------------------------------------------------------*/
/* Accumulate publication continuity */
/*---------------------------------------------------------------------------*/
/* Four float64 scalars, no quantisation: 32 wire bytes per RT cycle.
* With MaxPayloadSize = 60 the accumulate header (8 B HRT + 4 B count) leaves
* room for exactly one cycle, so the size condition flushes on every single
* Synchronise() the maximum number of hand-offs to the sender thread, each
* one a chance for a promoted batch to be skipped. */
#define ACC_FUNCTIONS_BLOCK \
" +Functions = {\n" \
" Class = ReferenceContainer\n" \
" +Writer = {\n" \
" Class = UDPStreamerTestOutputGAM\n" \
" OutputSignals = {\n" \
" A = {\n" \
" DataSource = Streamer\n" \
" Type = float64\n" \
" }\n" \
" B = {\n" \
" DataSource = Streamer\n" \
" Type = float64\n" \
" }\n" \
" C = {\n" \
" DataSource = Streamer\n" \
" Type = float64\n" \
" }\n" \
" D = {\n" \
" DataSource = Streamer\n" \
" Type = float64\n" \
" }\n" \
" }\n" \
" }\n" \
" }\n"
static const MARTe::char8 *const ACC_CFG_CONTINUITY =
"+Test = {\n"
" Class = RealTimeApplication\n"
ACC_FUNCTIONS_BLOCK
" +Data = {\n"
" Class = ReferenceContainer\n"
" +Streamer = {\n"
" Class = UDPStreamer\n"
" Port = 44680\n"
" MaxPayloadSize = 60\n"
" PublishingMode = Accumulate\n"
" MinRefreshRate = 1000.0\n"
" Signals = {\n"
" A = {\n"
" Type = float64\n"
" }\n"
" B = {\n"
" Type = float64\n"
" }\n"
" C = {\n"
" Type = float64\n"
" }\n"
" D = {\n"
" Type = float64\n"
" }\n"
" }\n"
" }\n"
HF_TAIL_BLOCK;
namespace {
/** Cycles driven by TestAccumulate_EveryPublishedCycleReachesTheWire. */
static const MARTe::uint32 ACC_CONTINUITY_CYCLES = 3000u;
/**
* @brief Records which RT cycles reached the wire, and how often.
*
* The test stamps signal A with the cycle index before every Synchronise(),
* and the config is sized so each Accumulate batch carries exactly one cycle.
* The payload is [8 B HRT][4 B numSamples][A][B][C][D], so A of the single
* slot sits at offset 12 and identifies the cycle unambiguously.
*
* Counting distinct cycles (rather than summing numSamples) is what makes this
* able to tell a lost publication from a re-sent one: a sender that never
* consumes its ready buffer emits the right *number* of packets while
* repeating a stale batch, which shows up here as duplicates plus missing
* cycles instead of a clean tally.
*/
class AccumRampRecorder: public MARTe::UDPSClientListener {
public:
AccumRampRecorder() :
packets(0u), duplicates(0u), malformed(0u) {
mux.Create();
for (MARTe::uint32 i = 0u; i < ACC_CONTINUITY_CYCLES; i++) {
seen[i] = false;
}
}
virtual void OnUDPSData(const MARTe::uint8 *payload, MARTe::uint32 payloadSize) {
MARTe::uint32 n = 0u;
MARTe::float64 v = 0.0;
if (payloadSize >= 20u) {
(void) MARTe::MemoryOperationsHelper::Copy(&n, &payload[8], 4u);
(void) MARTe::MemoryOperationsHelper::Copy(&v, &payload[12], 8u);
}
(void) mux.FastLock();
packets++;
if ((payloadSize < 20u) || (n != 1u)) {
malformed++;
}
else {
MARTe::uint32 idx = static_cast<MARTe::uint32>(v);
if ((static_cast<MARTe::float64>(idx) != v) || (idx >= ACC_CONTINUITY_CYCLES)) {
malformed++;
}
else if (seen[idx]) {
duplicates++;
}
else {
seen[idx] = true;
}
}
mux.FastUnLock();
}
MARTe::uint32 DistinctCycles() {
(void) mux.FastLock();
MARTe::uint32 n = 0u;
for (MARTe::uint32 i = 0u; i < ACC_CONTINUITY_CYCLES; i++) {
if (seen[i]) {
n++;
}
}
mux.FastUnLock();
return n;
}
MARTe::uint32 Packets() {
(void) mux.FastLock();
MARTe::uint32 n = packets;
mux.FastUnLock();
return n;
}
MARTe::uint32 Duplicates() {
(void) mux.FastLock();
MARTe::uint32 n = duplicates;
mux.FastUnLock();
return n;
}
MARTe::uint32 Malformed() {
(void) mux.FastLock();
MARTe::uint32 n = malformed;
mux.FastUnLock();
return n;
}
private:
MARTe::FastPollingMutexSem mux;
bool seen[ACC_CONTINUITY_CYCLES];
MARTe::uint32 packets;
MARTe::uint32 duplicates;
MARTe::uint32 malformed;
};
} // namespace
bool UDPStreamerTest::TestAccumulate_EveryPublishedCycleReachesTheWire() {
using namespace MARTe;
/* One-cycle batches every 200 us: ~5000 small packets/s, which the sender
* thread handles comfortably. The period has to be this short because a
* wake-up can only be swallowed while the sender is mid-send; at 1 ms the
* sender is always back in its wait before the next Synchronise() and the
* defect never fires at all. */
const uint32 CYCLES = ACC_CONTINUITY_CYCLES;
static const float64 CYCLE_SEC = 200e-6;
/* Tolerance, as a fraction of CYCLES, for cycles that never reach the wire.
* It is not zero: this is an ordinary userspace thread on a general-purpose
* kernel, so it can occasionally be descheduled past a 200 us slot, and the
* last batch may still be in the accumulation buffer when the loop ends.
* It is small because the defect this guards against is not marginal a
* sender that decides what to send from the semaphore edge fails to consume
* essentially every batch (~100% here), so a 1% ceiling separates the two
* regimes with three orders of magnitude to spare. */
const uint32 MAX_LOST = CYCLES / 100u;
ReferenceT<RealTimeApplication> app = LoadApplication(ACC_CFG_CONTINUITY);
bool ok = app.IsValid();
if (ok) {
ok = (app->PrepareNextState("State1") == ErrorManagement::NoError);
}
Sleep::MSec(50u);
AccumRampRecorder counter;
UDPSClient client;
ReferenceT<UDPStreamer> ds;
if (ok) {
ConfigurationDatabase clientCfg;
ok = clientCfg.Write("ServerAddr", "127.0.0.1");
ok = ok && clientCfg.Write("Port", 44680u);
ok = ok && clientCfg.Write("SilenceTimeout", 0.0f);
ok = ok && clientCfg.Write("KeepAliveInterval", 0u);
client.SetListener(&counter);
ok = ok && client.Initialise(clientCfg);
ok = ok && client.Start();
}
/* Wait for the CONNECT to register on the streamer side. */
if (ok) {
ds = ObjectRegistryDatabase::Instance()->Find("Test.Data.Streamer");
ok = ds.IsValid();
}
if (ok) {
uint32 waited = 0u;
while ((waited < 3000u) && !ds->IsClientConnected()) {
Sleep::MSec(20u);
waited += 20u;
}
ok = ds->IsClientConnected();
}
/* Signal A carries the cycle index, so every packet identifies exactly
* which RT cycle produced it. Synchronise() snapshots the DataSource
* memory, so writing straight into it is equivalent to a GAM having
* produced the value. */
float64 *sigA = NULL_PTR(float64 *);
if (ok) {
void *addr = NULL_PTR(void *);
ok = ds->GetSignalMemoryBuffer(0u, 0u, addr);
sigA = reinterpret_cast<float64 *>(addr);
ok = ok && (sigA != NULL_PTR(float64 *));
}
/* Drive the RT cycles. */
if (ok) {
for (uint32 i = 0u; (i < CYCLES) && ok; i++) {
*sigA = static_cast<float64>(i);
ok = ds->Synchronise();
Sleep::Sec(CYCLE_SEC);
}
}
/* Let the last packets drain. */
Sleep::MSec(300u);
uint32 distinct = counter.DistinctCycles();
uint32 packets = counter.Packets();
uint32 duplicates = counter.Duplicates();
uint32 malformed = counter.Malformed();
uint32 dropped = (ds.IsValid()) ? ds->GetDroppedPublications() : 0u;
if (ok) {
ok = (malformed == 0u);
if (!ok) {
REPORT_ERROR_STATIC(ErrorManagement::FatalError,
"%u of %u DATA packets did not carry exactly one "
"decodable cycle index.", malformed, packets);
}
}
if (ok) {
/* A cycle that never arrives is a hole in the consumer's time series. */
ok = (distinct + MAX_LOST) >= CYCLES;
if (!ok) {
REPORT_ERROR_STATIC(ErrorManagement::FatalError,
"Accumulate lost cycles: %u of %u reached the wire "
"in %u packets (%u duplicates, %u publications "
"overwritten before being sent).",
distinct, CYCLES, packets, duplicates, dropped);
}
}
if (ok) {
/* A cycle that arrives twice means the sender re-sent a ready buffer it
* had already transmitted, which lands the same samples on the receiver
* under two different time bases. */
ok = (duplicates == 0u);
if (!ok) {
REPORT_ERROR_STATIC(ErrorManagement::FatalError,
"%u of %u DATA packets repeated a cycle already sent.",
duplicates, packets);
}
}
if (ok) {
/* Same ceiling from the producer's side: it sees the overwrite directly
* and does not depend on the packet reaching the loopback socket. */
ok = (dropped <= MAX_LOST);
if (!ok) {
REPORT_ERROR_STATIC(ErrorManagement::FatalError,
"%u of %u publications were overwritten before the "
"sender thread took them.", dropped, CYCLES);
}
}
(void) client.Stop();
ObjectRegistryDatabase::Instance()->Purge();
return ok;
}
@@ -234,6 +234,16 @@ public:
* @brief Tests full TCP CONNECT CONFIG DATA via multicast DISCONNECT on loopback.
*/
bool TestExecute_MulticastConnectDataDisconnect();
/**
* @brief Tests that Accumulate publishes every RT cycle it batches.
* @details Drives 600 cycles at a rate the sender thread trivially keeps up
* with, and sums the numSamples field of every DATA packet that arrives.
* A batch promoted to the ready buffer but never sent because the wake-up
* announcing it was swallowed shows up here as missing cycles, which a
* consumer sees as a hole in the time series.
*/
bool TestAccumulate_EveryPublishedCycleReachesTheWire();
};
#endif /* UDPSTREAMERTEST_H_ */
+416
View File
@@ -41,6 +41,7 @@
/*---------------------------------------------------------------------------*/
#include "BasicUDPSocket.h"
#include "ConfigurationDatabase.h"
#include "FastPollingMutexSem.h"
#include "InternetHost.h"
#include "Sleep.h"
#include "UDPSClient.h"
@@ -131,6 +132,147 @@ bool WaitForClient(UDPSServer &server, uint32 timeoutMs) {
return false;
}
/*---------------------------------------------------------------------------*/
/* Fragment-reassembly test harness */
/*---------------------------------------------------------------------------*/
/** Largest reassembled payload the recording listener keeps a copy of. */
const uint32 kMaxRecordedBytes = 8192u;
/** How many reassembled payloads the recording listener keeps. */
const uint32 kMaxRecorded = 16u;
/**
* @brief Listener that records every reassembled DATA/CONFIG payload.
*
* Callbacks run on the UDPSClient receive thread; the test thread reads the
* records after a settle sleep, so both sides take the same lock.
*/
class RecordingListener: public UDPSClientListener {
public:
RecordingListener() :
dataCount(0u), configCount(0u) {
mux.Create();
}
virtual void OnUDPSData(const uint8 *payload, uint32 payloadSize) {
Record(dataPayloads, dataSizes, dataCount, payload, payloadSize);
}
virtual void OnUDPSConfig(const uint8 *payload, uint32 payloadSize) {
Record(configPayloads, configSizes, configCount, payload, payloadSize);
}
uint32 DataCount() {
(void) mux.FastLock();
uint32 n = dataCount;
mux.FastUnLock();
return n;
}
uint32 ConfigCount() {
(void) mux.FastLock();
uint32 n = configCount;
mux.FastUnLock();
return n;
}
/** @return true iff record @p idx matches @p expected byte for byte. */
bool DataMatches(uint32 idx, const uint8 *expected, uint32 expectedSize) {
return Matches(dataPayloads, dataSizes, dataCount, idx, expected,
expectedSize);
}
bool ConfigMatches(uint32 idx, const uint8 *expected, uint32 expectedSize) {
return Matches(configPayloads, configSizes, configCount, idx, expected,
expectedSize);
}
uint32 DataSize(uint32 idx) {
(void) mux.FastLock();
uint32 n = (idx < dataCount) ? dataSizes[idx] : 0u;
mux.FastUnLock();
return n;
}
private:
void Record(uint8 (&dst)[kMaxRecorded][kMaxRecordedBytes],
uint32 (&sizes)[kMaxRecorded], uint32 &count,
const uint8 *payload, uint32 payloadSize) {
(void) mux.FastLock();
if (count < kMaxRecorded) {
sizes[count] = payloadSize;
uint32 n = (payloadSize < kMaxRecordedBytes) ? payloadSize
: kMaxRecordedBytes;
memcpy(dst[count], payload, n);
count++;
}
mux.FastUnLock();
}
bool Matches(uint8 (&src)[kMaxRecorded][kMaxRecordedBytes],
uint32 (&sizes)[kMaxRecorded], uint32 &count, uint32 idx,
const uint8 *expected, uint32 expectedSize) {
(void) mux.FastLock();
bool ok = (idx < count) && (sizes[idx] == expectedSize) &&
(expectedSize <= kMaxRecordedBytes) &&
(memcmp(src[idx], expected, expectedSize) == 0);
mux.FastUnLock();
return ok;
}
FastPollingMutexSem mux;
uint8 dataPayloads[kMaxRecorded][kMaxRecordedBytes];
uint32 dataSizes[kMaxRecorded];
uint32 dataCount;
uint8 configPayloads[kMaxRecorded][kMaxRecordedBytes];
uint32 configSizes[kMaxRecorded];
uint32 configCount;
};
/** Fill @p buf with a position-dependent pattern so misplacement is visible. */
void FillPattern(uint8 *buf, uint32 n, uint8 seed) {
for (uint32 i = 0u; i < n; i++) {
buf[i] = static_cast<uint8>((i * 7u) + seed);
}
}
/** Send one UDPS fragment datagram to 127.0.0.1:@p dstPort. */
bool SendFragment(BasicUDPSocket &sock, uint16 dstPort, uint8 type,
uint32 counter, uint16 fragIdx, uint16 totalFrags,
const uint8 *payload, uint32 payloadBytes) {
uint8 buf[UDPS_HEADER_SIZE + 2048u];
if (payloadBytes > 2048u) {
return false;
}
UDPSBuildHeader(buf, type, counter, fragIdx, totalFrags, payloadBytes);
memcpy(&buf[UDPS_HEADER_SIZE], payload, payloadBytes);
InternetHost dst(dstPort, "127.0.0.1");
(void) sock.SetDestination(dst);
uint32 n = UDPS_HEADER_SIZE + payloadBytes;
return sock.Write(reinterpret_cast<const char8 *>(buf), n);
}
/**
* @brief Bring up a UDPSClient pointed at @p server and learn the ephemeral
* port it receives DATA on (the source port of its CONNECT).
*
* Silence timeout and keepalive are disabled so the session never churns
* underneath the fragments the test injects.
*/
bool StartClientAndLearnPort(UDPSClient &client, ConfigurationDatabase &cfg,
BasicUDPSocket &server, uint16 serverPort,
uint16 &clientPort) {
if (!cfg.Write("ServerAddr", "127.0.0.1")) { return false; }
if (!cfg.Write("Port", static_cast<uint32>(serverPort))) { return false; }
if (!cfg.Write("SilenceTimeout", 0.0f)) { return false; }
if (!cfg.Write("KeepAliveInterval", 0u)) { return false; }
if (!client.Initialise(cfg)) { return false; }
if (!client.Start()) { return false; }
uint8 type = 0xFFu;
if (!WaitDatagram(server, 3000, type, clientPort)) { return false; }
return (type == UDPS_TYPE_CONNECT) && (clientPort != 0u);
}
} // namespace
/*---------------------------------------------------------------------------*/
@@ -346,3 +488,277 @@ TEST(UDPSClientGTest, TestSilenceTimeoutSubSecondTriggersReconnect) {
client.Stop();
server.Close();
}
TEST(UDPSClientGTest, TestReorderedFragmentsAreReassembled) {
/* UDP gives no ordering guarantee: the fragments of one packet may arrive
* in any order, with nothing lost. Reassembly must not depend on fragment
* 0 arriving first if it does, an out-of-order burst destroys a packet
* whose bytes all arrived, and leaves a slot occupied until the 2 s GC,
* which is how four slots end up permanently full. */
BasicUDPSocket server;
ASSERT_TRUE(server.Open());
ASSERT_TRUE(server.Listen(0u));
uint16 serverPort = GetBoundPort(server);
ASSERT_NE(serverPort, 0u);
RecordingListener listener;
UDPSClient client;
client.SetListener(&listener);
ConfigurationDatabase cfg;
uint16 clientPort = 0u;
ASSERT_TRUE(StartClientAndLearnPort(client, cfg, server, serverPort,
clientPort));
/* 20-byte payload over three 8-byte chunks: the last one is short, which
* is exactly why chunk size has to be learnt from a non-last fragment. */
uint8 expected[20];
FillPattern(expected, sizeof(expected), 3u);
ASSERT_TRUE(SendFragment(server, clientPort, UDPS_TYPE_DATA, 7u, 1u, 3u,
&expected[8], 8u));
ASSERT_TRUE(SendFragment(server, clientPort, UDPS_TYPE_DATA, 7u, 2u, 3u,
&expected[16], 4u));
ASSERT_TRUE(SendFragment(server, clientPort, UDPS_TYPE_DATA, 7u, 0u, 3u,
&expected[0], 8u));
Sleep::MSec(400u);
ASSERT_EQ(listener.DataCount(), 1u)
<< "no fragment was lost, yet the packet was not delivered";
EXPECT_EQ(listener.DataSize(0u), 20u);
EXPECT_TRUE(listener.DataMatches(0u, expected, sizeof(expected)));
client.Stop();
server.Close();
}
TEST(UDPSClientGTest, TestDataAndConfigWithSameCounterDoNotCollide) {
/* DATA and CONFIG carry independent counter sequences, so the same counter
* value legitimately appears on both. A reassembly slot keyed on the
* counter alone merges the two streams: one payload is delivered under the
* wrong type and the other is silently dropped. */
BasicUDPSocket server;
ASSERT_TRUE(server.Open());
ASSERT_TRUE(server.Listen(0u));
uint16 serverPort = GetBoundPort(server);
ASSERT_NE(serverPort, 0u);
RecordingListener listener;
UDPSClient client;
client.SetListener(&listener);
ConfigurationDatabase cfg;
uint16 clientPort = 0u;
ASSERT_TRUE(StartClientAndLearnPort(client, cfg, server, serverPort,
clientPort));
uint8 dataPayload[16];
uint8 cfgPayload[16];
FillPattern(dataPayload, sizeof(dataPayload), 11u);
FillPattern(cfgPayload, sizeof(cfgPayload), 200u);
/* Same counter (42), interleaved, two fragments each. */
ASSERT_TRUE(SendFragment(server, clientPort, UDPS_TYPE_CONFIG, 42u, 0u, 2u,
&cfgPayload[0], 8u));
ASSERT_TRUE(SendFragment(server, clientPort, UDPS_TYPE_DATA, 42u, 0u, 2u,
&dataPayload[0], 8u));
ASSERT_TRUE(SendFragment(server, clientPort, UDPS_TYPE_CONFIG, 42u, 1u, 2u,
&cfgPayload[8], 8u));
ASSERT_TRUE(SendFragment(server, clientPort, UDPS_TYPE_DATA, 42u, 1u, 2u,
&dataPayload[8], 8u));
Sleep::MSec(400u);
EXPECT_EQ(listener.ConfigCount(), 1u);
EXPECT_TRUE(listener.ConfigMatches(0u, cfgPayload, sizeof(cfgPayload)));
ASSERT_EQ(listener.DataCount(), 1u)
<< "the DATA packet was swallowed by the CONFIG slot sharing its counter";
EXPECT_TRUE(listener.DataMatches(0u, dataPayload, sizeof(dataPayload)));
client.Stop();
server.Close();
}
TEST(UDPSClientGTest, TestDuplicateHighIndexFragmentDoesNotFakeCompletion) {
/* Completion is decided by counting fragments, with a received-bitmask to
* reject duplicates. If the mask is narrower than the fragment count the
* client accepts, a duplicated high-index fragment is counted twice and
* the packet is delivered while a fragment is still missing a payload
* with a hole of stale bytes, reported as valid. */
BasicUDPSocket server;
ASSERT_TRUE(server.Open());
ASSERT_TRUE(server.Listen(0u));
uint16 serverPort = GetBoundPort(server);
ASSERT_NE(serverPort, 0u);
RecordingListener listener;
UDPSClient client;
client.SetListener(&listener);
ConfigurationDatabase cfg;
uint16 clientPort = 0u;
ASSERT_TRUE(StartClientAndLearnPort(client, cfg, server, serverPort,
clientPort));
/* 300 fragments — past the 256 a 32-byte mask covers, but well inside the
* 512 the client's own sanity check permits. */
const uint16 kTotalFrags = 300u;
const uint32 kChunk = 8u;
const uint32 kLastChunk = 4u;
const uint32 kTotalBytes = ((kTotalFrags - 1u) * kChunk) + kLastChunk;
uint8 expected[((kTotalFrags - 1u) * kChunk) + kLastChunk];
FillPattern(expected, kTotalBytes, 5u);
/* Everything except the final fragment, plus one duplicate above 255. */
for (uint16 f = 0u; f < (kTotalFrags - 1u); f++) {
ASSERT_TRUE(SendFragment(server, clientPort, UDPS_TYPE_DATA, 9u, f,
kTotalFrags, &expected[f * kChunk], kChunk));
}
ASSERT_TRUE(SendFragment(server, clientPort, UDPS_TYPE_DATA, 9u, 260u,
kTotalFrags, &expected[260u * kChunk], kChunk));
Sleep::MSec(500u);
ASSERT_EQ(listener.DataCount(), 0u)
<< "delivered with a fragment still missing (a duplicate was counted "
"as a new fragment)";
/* The genuinely missing fragment completes it, with the right bytes. */
ASSERT_TRUE(SendFragment(server, clientPort, UDPS_TYPE_DATA, 9u,
kTotalFrags - 1u, kTotalFrags,
&expected[(kTotalFrags - 1u) * kChunk],
kLastChunk));
Sleep::MSec(400u);
ASSERT_EQ(listener.DataCount(), 1u);
EXPECT_EQ(listener.DataSize(0u), kTotalBytes);
EXPECT_TRUE(listener.DataMatches(0u, expected, kTotalBytes));
client.Stop();
server.Close();
}
TEST(UDPSClientGTest, TestStaleDataPacketIsNotDelivered) {
/* A DATA packet that arrives after a newer one has already been delivered
* carries an older time base. Delivering it makes the consumer place its
* samples behind the ones it has: they collide with what is already
* plotted, and the range they should have occupied stays empty. The
* counter is the only thing that tells the two apart, so the client must
* drop anything that does not advance it. */
BasicUDPSocket server;
ASSERT_TRUE(server.Open());
ASSERT_TRUE(server.Listen(0u));
uint16 serverPort = GetBoundPort(server);
ASSERT_NE(serverPort, 0u);
RecordingListener listener;
UDPSClient client;
client.SetListener(&listener);
ConfigurationDatabase cfg;
uint16 clientPort = 0u;
ASSERT_TRUE(StartClientAndLearnPort(client, cfg, server, serverPort,
clientPort));
uint8 pkt[8];
FillPattern(pkt, sizeof(pkt), 1u);
/* 10 and 11 advance the counter; 9 and the repeat of 11 do not. */
ASSERT_TRUE(SendFragment(server, clientPort, UDPS_TYPE_DATA, 10u, 0u, 1u,
pkt, sizeof(pkt)));
ASSERT_TRUE(SendFragment(server, clientPort, UDPS_TYPE_DATA, 11u, 0u, 1u,
pkt, sizeof(pkt)));
ASSERT_TRUE(SendFragment(server, clientPort, UDPS_TYPE_DATA, 9u, 0u, 1u,
pkt, sizeof(pkt)));
ASSERT_TRUE(SendFragment(server, clientPort, UDPS_TYPE_DATA, 11u, 0u, 1u,
pkt, sizeof(pkt)));
Sleep::MSec(400u);
EXPECT_EQ(listener.DataCount(), 2u)
<< "a packet older than one already delivered reached the listener";
EXPECT_EQ(client.GetStaleDataPackets(), 2u);
client.Stop();
server.Close();
}
TEST(UDPSClientGTest, TestCounterGapIsReported) {
/* Consumers that infer a sample period from the sender-clock gap need to
* know how many packets that gap spans; without it a single loss reads as
* a halved rate. The gap comes from the counter, and must exclude the
* packet being delivered. */
BasicUDPSocket server;
ASSERT_TRUE(server.Open());
ASSERT_TRUE(server.Listen(0u));
uint16 serverPort = GetBoundPort(server);
ASSERT_NE(serverPort, 0u);
RecordingListener listener;
UDPSClient client;
client.SetListener(&listener);
ConfigurationDatabase cfg;
uint16 clientPort = 0u;
ASSERT_TRUE(StartClientAndLearnPort(client, cfg, server, serverPort,
clientPort));
uint8 pkt[8];
FillPattern(pkt, sizeof(pkt), 2u);
ASSERT_TRUE(SendFragment(server, clientPort, UDPS_TYPE_DATA, 100u, 0u, 1u,
pkt, sizeof(pkt)));
Sleep::MSec(200u);
EXPECT_EQ(client.GetLastDataGap(), 0u) << "the first packet lost nothing";
/* 101, 102 and 103 never arrive. */
ASSERT_TRUE(SendFragment(server, clientPort, UDPS_TYPE_DATA, 104u, 0u, 1u,
pkt, sizeof(pkt)));
Sleep::MSec(200u);
EXPECT_EQ(client.GetLastDataGap(), 3u);
ASSERT_TRUE(SendFragment(server, clientPort, UDPS_TYPE_DATA, 105u, 0u, 1u,
pkt, sizeof(pkt)));
Sleep::MSec(200u);
EXPECT_EQ(client.GetLastDataGap(), 0u) << "the gap must not persist";
EXPECT_EQ(listener.DataCount(), 3u);
EXPECT_EQ(client.GetStaleDataPackets(), 0u);
client.Stop();
server.Close();
}
TEST(UDPSClientGTest, TestCounterWraparoundDoesNotRejectStream) {
/* The counter is a uint32 that wraps. Ordering it by plain comparison
* would call every packet after the wrap older than 0xFFFFFFFF and reject
* the stream permanently, so the ordering has to be done on the signed
* difference. */
BasicUDPSocket server;
ASSERT_TRUE(server.Open());
ASSERT_TRUE(server.Listen(0u));
uint16 serverPort = GetBoundPort(server);
ASSERT_NE(serverPort, 0u);
RecordingListener listener;
UDPSClient client;
client.SetListener(&listener);
ConfigurationDatabase cfg;
uint16 clientPort = 0u;
ASSERT_TRUE(StartClientAndLearnPort(client, cfg, server, serverPort,
clientPort));
uint8 pkt[8];
FillPattern(pkt, sizeof(pkt), 4u);
const uint32 counters[4] = { 0xFFFFFFFEu, 0xFFFFFFFFu, 0u, 1u };
for (uint32 i = 0u; i < 4u; i++) {
ASSERT_TRUE(SendFragment(server, clientPort, UDPS_TYPE_DATA,
counters[i], 0u, 1u, pkt, sizeof(pkt)));
Sleep::MSec(150u);
}
EXPECT_EQ(listener.DataCount(), 4u)
<< "the stream was rejected across the counter wrap";
EXPECT_EQ(client.GetStaleDataPackets(), 0u);
EXPECT_EQ(client.GetLastDataGap(), 0u);
client.Stop();
server.Close();
}