diff --git a/README.md b/README.md index dbb301c..f1aa36a 100644 --- a/README.md +++ b/README.md @@ -26,7 +26,7 @@ Options: `--server`, `--port`, `--multicast`, `--interface`, `--buffer-mb` (size of the sample buffer, default 200), `--max-window` (largest time window the toolbar offers, default 600), `--holdoff` (default trigger hold-off), `--recv-buffer`, `--plots` (the number of stacked plots to start with, 1..8; rows are -added and removed from the **Layout editor** afterwards). +added and removed from the **Layout editor** afterwards), `--relative-time` (see below). Every CONFIG is dumped to stderr once -- publish mode and, per signal, its name, element count, rate, time mode and time-signal index -- because a signal that @@ -363,6 +363,50 @@ latch, a pan and a zoom do **not** reset: they change which samples are in view, exactly what a statistic over many windows is meant to average. The toggle itself does not reset either, so turning the line off to read the plot and back on again keeps the history. +## Resequencing and relative timing + +**A shaded band marks samples the stream says exist and have not arrived.** When a packet is +received that reveals a gap in the counter, a run of NaN samples is reserved for the missing +packets; the span is shaded in the plot and the counters in the status bar show it. The band +carries three states: + +- **Solid** while the hole still lacks packets. +- **Faint** once the late packets arrive and repair the values — the data is now correct, but + the record remains that this region arrived out of order. +- **Absent** once that part of the stream scrolls out of the ring's buffer. + +Two things are **not** repaired when a late packet arrives: **math channels** and **the trigger**. +A backfilled slot's value is corrected in the ring and in the pyramid, so the raw trace and the +readouts are right. The math engine and the trigger already ran over the placeholder, however, and +are not replayed: a math trace keeps its gap across a repaired hole, and if the trigger latched +adjacent to one, that latch was on the data the engine actually saw. + +**Known limitations.** The fade is all-or-nothing at the hole level — a hole with one packet still +missing shades solidly, even if all but one slot are already repaired. And the two tints are not +yet configurable; the preferences dialog in `todo.md` is where that would land. Do not use a +command-line flag for either; none exists. + +**`--relative-time` makes the x axis follow the sample count and the declared rate**, rather than +the stream's own clock. The packet counter says exactly how many samples each gap spans, so a +reserved hole is a precise arithmetic range. In stream mode the same hole is estimated from the +declared cadence, which is the existing glitch-bridge mechanism reused. + +The history cost of relative mode is measured rather than computed: + +| mode | retained span | +| --- | --- | +| stream time (default) | **83.886 s** | +| `--relative-time` | **41.943 s** | + +Conditions: the mock at 100 kS/s, 4 channels at stride 1, the default `--buffer-mb 200`. +Relative mode retains **exactly half** the history — not the ~22% the original spec claimed, and +not the 75% a per-byte calculation suggests. The reason is the floor: relative mode keeps a second +f64 ring for absolute time beside the stream clock's, so a tick's cost goes from +`8 + 4×4 = 24` bytes to `16 + 4×4 = 32` bytes — a third more. But `Partition::time_capacity` is +floored to a **power of two** (`src/model.cpp:83`, `:110`), so a third more weight does not cost a +third of the span; it costs a whole halving. The measured figures above are the correct ones; a +per-byte computation gives the wrong answer for this reason. + ## The session layout file What you arranged is remembered per data source, in diff --git a/SPECS.md b/SPECS.md index df5ee78..67c22bb 100644 --- a/SPECS.md +++ b/SPECS.md @@ -36,3 +36,16 @@ - the time should be computed from the time associated with the signal (not from the client but from the streamed data) - in trigger mode data visualisation should not be refreshed till next event, that's why 2 identical buffer, one is the currently viewed event, the other used to acquire live data for next event + +## Deviations from the spec + +The following were built differently from what the spec states, for documented reasons in `README.md`: + +- **`todo.md` item 2 — relative timing.** Implemented as `--relative-time`. The retained history + in this mode is **41.943 s** against the default **83.886 s** (measured at 100 kS/s, 4 channels, + `--buffer-mb 200`): exactly half, because a second f64 ring for absolute time raises a tick's + cost from 24 to 32 bytes — a third more — and `Partition::time_capacity` is floored to a power + of two, so a third more weight costs a full halving. The spec and the original plan both quoted + ~22%, which is wrong; the per-byte figure of 75% is also wrong for the same reason. + Absolute stream time is preserved in a separate ring and used for the export's `abs_seconds` + column, exactly as `todo.md` requires. diff --git a/docs/claude/build-and-run.md b/docs/claude/build-and-run.md index 4f6c9c5..083060f 100644 --- a/docs/claude/build-and-run.md +++ b/docs/claude/build-and-run.md @@ -62,6 +62,28 @@ handler keyed by a pointer-to-pointer (`gpointer slot` cast to `AdwDialog**`), s handler clears either pointer without knowing which dialog it is. Presenting either on this app's plain `GtkApplicationWindow` is supported and warning-free; it was checked. +`deps/udpsclient` is a vendored C99 source pair and its diff from the upstream source +is intentional. Four things were changed: + +1. **A reorder ring of reassembled payloads.** When a packet arrives with a higher counter + than expected, the raw datagram is held in a fixed-size ring rather than decoded immediately. + A deadline (`now + reorder_window_s`, default 0.020 s) is set for the gap; the deadline is + served by a third `clamp_deadline()` call on the `select()` budget + (`udps_client.c:1145-1152`) so a quiet stream does not wait for the next arrival. +2. **Late delivery replacing the outright discard.** A packet that arrives past the window — or + when the ring is drained by the deadline — is delivered to the callback flagged `late` rather + than discarded with `stats.stale_packets++; return 0`. That flag is what makes backfill + possible at all; without it the samples are gone before udpscope can act. +3. **The third `clamp_deadline()`.** The existing pair handles the keepalive and the silence + timeout; the reorder deadline is a third call to the same helper, which already understood + the pattern. +4. **`reorder_window_s = 0` as the switch.** `udps_client_config_t::reorder_window_s`, + defaulted to `0.020` in `udps_client_default_config()` (`deps/udpsclient/udps_client.c:1190`), + is the only knob. Setting it to zero disables resequencing and restores the previous + discard-on-late behaviour exactly — so a reported regression is bisectable without reverting + the source. **There is no command-line flag for the reorder window.** It is not surfaced on + udpscope's command line; do not document a `--reorder-window-ms`. + `deps/plotter` is upstream's own repository and this app's figure support was added there rather than worked around here: `Figure::set_min_y_gutter()` (a floor under the common left gutter), `Figure::set_right_gutter()` / `Plot::set_right_gutter()` (an empty diff --git a/docs/claude/invariants-data-path.md b/docs/claude/invariants-data-path.md index 1f7d25d..40c7d89 100644 --- a/docs/claude/invariants-data-path.md +++ b/docs/claude/invariants-data-path.md @@ -96,6 +96,26 @@ Part of [CLAUDE.md](../../CLAUDE.md). These are rules that cost a defect to lear the export guard in [probe-readings.md](probe-readings.md) for why it is no longer exactly one value). Three steps at 2, 2.5 and 3 ULP of *the stream's uptime* — microseconds, not femtoseconds — is this defect. +- **Append is not the only write.** A backfill amends slots that a reservation wrote as NaN, and + the amend is split across threads for a reason: + - **Ring amend — producer thread.** `RingBuffer::amend(absolute, v)` is a single relaxed + store, bounds-checked against `[produced_ - capacity, produced_)`. The same race class as a + push, and an `f32` cannot tear. Measurements and export read the ring directly, so they see + a repair immediately and no second notification is needed. + - **Pyramid amend — consumer (GTK main thread), inside `ChannelBuffer::acquire()`.** The + producer collects amended values in a plain `std::vector` member and the consumer + drains it after its own append pass. `acquire()` is called under `model.mutex()`, and the + producer holds that same mutex for all of `on_data`, so the vector is correctly serialised + with no lock-free queue. The reason the pyramid amend cannot also be on the producer: a + bucket is a 32-byte non-atomic aggregate that the renderer reads **unlocked at paint time**, + so a producer writing it could be caught mid-bucket — a `count` that does not match its + `sum`, producing a wrong mean from a structure whose readers trust `count == 0` to mean + *absent*. On the consumer that cannot happen. + - **A hole is stamped at reservation; backfill changes values only.** The f64 time ring is + never amended. `neighbours()' binary search requires a monotonic array, and `window_x0_` + must remain valid; pre-stamping the hole at the declared cadence (exact in relative mode, + the bridge mechanism in stream mode) is what keeps both invariants, and it is why a late + packet's counter, not its arrival stamp, is what addresses the hole. - **A dropped time slot is clamped, never answered with a sentinel.** `acquire()` pins the published window to the value ring's base and both rings keep sliding afterwards — their capacities are equal, so there is no slack — which means the oldest few samples diff --git a/docs/claude/invariants-plot-and-layout.md b/docs/claude/invariants-plot-and-layout.md index 50b2963..cd4cd38 100644 --- a/docs/claude/invariants-plot-and-layout.md +++ b/docs/claude/invariants-plot-and-layout.md @@ -104,6 +104,29 @@ Part of [CLAUDE.md](../../CLAUDE.md). These are rules that cost a defect to lear figure rebuild — which `assign()` already forces. A signal on no plot gets `RowSet::unplotted_color()` (grey) rather than no chip, so the panel's rows stay the same height. +- **The invalid-span band is data-anchored and reserves no layout space.** `Plot::set_invalid_spans()` + draws a tinted quad over the span it names; unlike `Plot::set_x_note()` (the `t0` caption), which + is chrome drawn at a fixed corner of the axes regardless of the data, a band pans and zooms with + the samples it accuses. The two are the complementary cases: the caption is chrome that data does + not move, the band is annotation that data does. Both are alike in the second count: neither + reserves layout space and neither can be picked, so a hole cannot move the layout the labels were + measured into. + + The band has a **one-device-pixel floor on its width**. A hole of one sample in a window of hundreds + of thousands is a hundredth of a column, and a fill rounded to zero would drop exactly the case + the band is most needed for — a sub-pixel hole is invisible as a gap in the polyline and would be + invisible as a band too without the floor. + + The band carries **three states** and they are a property of the hole, not of the plot: solid + (`Style::invalid_fill`) while the hole still lacks packets, faint (`Style::invalid_repaired_fill`) + once `Hole::repaired()` is true, absent once the hole's slots age out of the ring and + `forget_holes_before()` drops it from the table. A repaired hole is **not** retired early: the band + is a record of what the window was missing, and retiring it on repair would erase the report the + moment it became explicable — the faint tier is how the display says "this was missing but arrived + late". `RowSet` sends one `plot::InvalidSpan` per hole to every row — a gap is in the stream, not + in one signal — and only in `Live` and `Capturing`, because a `Held` or `Paused` snapshot carries + its own stamps that the live hole table does not address. + - **`RowSet::signal_color()` is the only route to a trace's colour.** The user can override the palette entry per signal (clicking its chip; `set_signal_color()` / `clear_signal_color()`, keyed by **name**, because a palette index is positional and a CONFIG renumbers), and that diff --git a/docs/claude/invariants-trigger-and-pinning.md b/docs/claude/invariants-trigger-and-pinning.md index fafe005..f74f703 100644 --- a/docs/claude/invariants-trigger-and-pinning.md +++ b/docs/claude/invariants-trigger-and-pinning.md @@ -122,6 +122,10 @@ Part of [CLAUDE.md](../../CLAUDE.md). These are rules that cost a defect to lear export.** Both need the same absolute index range over the time ring and every value ring, or an export and a capture of the same window would hold different samples. Call it under `mutex()`. +- **The trigger engine is not fed a reserved hole's samples.** A NaN comparison is false anyway, + so the engine would not latch; but the real reason the samples are withheld is the other one: + feeding them would advance the hold-off clock across time nothing was sampled in. The probe + asserts `trigger_in_hole=0` rather than relying on NaN's arithmetic properties. - **`EventDetector` owns every condition shape and holds no arming state; `TriggerEngine` owns the arming, the hold-off and the post window and holds no condition.** The split is what lets the buffer search ask the trigger's exact question without owning a trigger, and it is only diff --git a/docs/claude/probe-readings.md b/docs/claude/probe-readings.md index bf0e965..753cda0 100644 --- a/docs/claude/probe-readings.md +++ b/docs/claude/probe-readings.md @@ -586,6 +586,53 @@ Two lines of the self-check output are assertions in disguise: it afterwards. - `[keys] a: armed 0->1 | space: paused 0->1 | t: trigger panel: 0->1 | … | editable focused: 'm' refused=1` (stderr, `UDPSCOPE_PROBE_KEYS_MS`) — **every `X->Y` pair must differ** and **`refused` must be 1**. The refusal is the whole reason the controller is in the bubble phase: every bare-letter accel is also a character, so an accel that fires while a `GtkEditable` has focus makes the window entry unusable. A pair reading `0->0` is an action whose dispatch does not reach the handler a click reaches. Note the format is `: : ->` — the name passed to `before_after()` includes the widget's description, so `t: trigger panel: 0->1` is the correct form (not `t: trigger panel 0->1` without the colon before the values). - `[color] '' at row R slot S: trace=#RRGGBB chip_agrees=1 (want 1)`, then `[color] override: trace=#123456 chip_agrees=1`, `[color] session: entries=1 rgb=#123456` and `[color] default: trace=#RRGGBB chip_agrees=1` (stderr, `UDPSCOPE_PROBE_COLOR_MS`) — **all three `chip_agrees` must be 1**, the override line's `trace` must be `#123456`, the session line must read `entries=1 rgb=#123456`, and the default line's `trace` must be back to the first line's. Run it with `UDPSCOPE_ENABLE_ALL_MS` ahead of it, or there is no bound trace and the probe prints `no bound trace to recolour` instead of asserting anything; on the standard mock the first line reads `'sine' at row 0 slot 0: trace=#4FA3FF`. Two things make this an assertion rather than a tautology. The colours are read out of **`plot::Plot::trace().color`** — what the renderer will stroke, reached through `Row::plot` *after* `relayout_areas()`, since a new colour is a new figure — and out of the chip's own `` markup, not by asking `RowSet::signal_color()` twice, which would pass against a chip that never reached the trace. And the override is `#123456`, deliberately in no palette slot: an override equal to the entry it replaces passes against an implementation that ignores overrides entirely. The session leg round-trips `collect_layout()` → `save_layout()` → `load_layout()` on a temp path and deletes it, because a probe must not leave a colour in the real session file for a later run to inherit. +- `[reseq-probe] reseq=N late=N stale=N holes=N hole_slots=N backfilled=N open_holes=N bands=N + bands_repaired=N nan_slots=N scanned=N trigger_in_hole=D` (stderr, `UDPSCOPE_PROBE_RESEQ_MS`) — + the resequencing path's mid-run state; the measured readings and the per-case command are in + [verification.md](verification.md). Six things to read carefully: + + 1. **`backfilled` and `hole_slots` are not comparable directly.** `hole_slots` counts + *time-ring* slots once per hole; `backfilled` counts *value-ring* slots repaired and + therefore sums over every non-math channel. On a stream with 4 non-math channels they + stand a factor of 4 apart. `hole_slots=4200 backfilled=16800` in the `late` case is + healthy, not an error; `backfilled` below `4 × hole_slots` would mean some channels + went unrepaired. + + 2. **`nan_slots=0` beside a large `holes` is healthy, not a contradiction.** A hole is + retired from the table only when its slots age out of the ring + (`ScopeModel::forget_holes_before()`), *never* on repair. So the `late` case reads + `holes=42 nan_slots=0`: every hole was fully repaired, and all 42 entries stay in the + table with no placeholder left in the data. What *would* be a defect is `nan_slots > 0` + with `bands_repaired == bands`, which would mean a band claimed data was recovered while + placeholders remained. The discriminator is `bands_repaired`, not `holes`. + + 3. **`bands` is the visible spans, `open_holes` the whole table.** `bands` counts what the + plot is actually shading in the visible window; `open_holes` counts every live hole entry + whether or not it is in view. `bands` below `open_holes` is normal — in the `mixed` case + `open_holes=43` against `bands=42` because the 43rd hole has scrolled out of the visible + window and `build_invalid_spans()` drops it. `bands` *above* `open_holes` would name a + stale span left on a plot after its hole was retired. + + 4. **`bands_repaired` is the faint tier.** A hole whose late packets have all arrived is + still drawn, in `Style::invalid_repaired_fill` rather than `Style::invalid_fill`, so the + display distinguishes "there is no data here" from "there was none when this window was + first drawn". The three states are: solid while the hole still lacks packets, faint once + `Hole::repaired()` is true, absent once the slots age out and `forget_holes_before()` + drops it. `bands_repaired == bands` with `nan_slots=0` is the fully recovered reading; + `bands_repaired=0` with a non-zero `backfilled` would mean the repair never reached the + display. + + 5. **`trigger_in_hole` must be 0, always.** A `1` means the trigger engine was fed a + reserved span's samples. The engine is specifically not fed them; a NaN comparison is + false anyway, but the real reason is that feeding them would advance the hold-off clock + across time nothing was sampled in. + + 6. **The discriminator for whether a reservation sized the hole right is the exit dump's + per-channel `stored=` line, not `frames=`.** `frames` counts delivered packets and is + blind to the slots a reservation pushed on their behalf. `stored` reflects what the ring + actually holds, so a reservation that mismeasured the hole shows up there and nowhere + else. + - `[zoomkeys] ctrl+plus x=[..]->[..] y_held=1 | alt+plus y=[..]->[..] x_held=1` (stderr, `UDPSCOPE_PROBE_ZOOMKEYS_MS`) — **both `_held` must be 1 and both zooms must still happen.** Run it inside a pinned state, where a frame cannot re-frame the other axis for its own reasons: `UDPSCOPE_PROBE_PAUSE_MS=1500 UDPSCOPE_PROBE_ZOOMKEYS_MS=2200`. A `_held=0` is the modifier not reaching the plotter's controller; an unchanged range on the *zoomed* axis is the opposite failure, the key doing nothing. Ctrl restricts to x and Alt to y; Ctrl+Alt zooms neither (no special case needed — neither flag alone fires). - `status bar: glance=[● live · 1.09 MS/s · 2 s window] max_glance_chars=53 detail_fields=11 popover_rebuilds=0` (exit dump) — **`max_glance_chars` must stay small** (under about 60) and **`popover_rebuilds` must be 0**. The status row is directly above the plot, so this is the "layout must not depend on content" rule read from the bar's own side: the state field is one word from a fixed set (`connecting`, `live`, `paused`, `armed`, `capturing`, `event held`, `disarmed` — a word added to it must be **no wider than the widest already there**, which is what let `connecting` join at ten characters beside `event held`; the dead-port glance reads `max_glance_chars=51` against a live run's 54), the rate field is `%.3g` (so `10 kS/s`, never `10.0 kS/s`), and the two fields a *state* adds are both elapsed times — trigger mode's `last event … ago` and a pause's `for ` — which step units rather than growing digits (`format_elapsed()`). A **paused** glance is not in the dump at all: it is read by `[pause-probe]`'s `glance=` field, since the dump's own line is taken live. `detail_fields` is the popover's label count, a literal, so adding a label without thinking about layout shows up here. The glance rate format matters: `%.3g` drops the trailing zero that a `%.4g` or a fixed-precision format would keep — a rate that reads differently in the dump and in the help page is a format disagreement, and `14-status.txt`'s example must match it. diff --git a/docs/claude/verification.md b/docs/claude/verification.md index 7eff937..274d62f 100644 --- a/docs/claude/verification.md +++ b/docs/claude/verification.md @@ -201,6 +201,59 @@ as distinct from the fault injection below, which exercises ingest: - `--quiet-at S` freezes `sine` at its held value from stream second `S` onwards, so it stops crossing zero: the dropout trigger's input. +- `UDPSCOPE_PROBE_RESEQ_MS` — one mid-run reading of the resequencing path. Four cases, + **run each as its own process** — four back to back starve the mock past its 2 s silence + timeout and every case then reads a reconnect instead of what it was testing. Common + conditions for all four: the mock at `--rate 10000 --batch 100 --freq 5`, udpscope with + no extra options, under `xvfb-run` with + `UDPSCOPE_EXIT_AFTER_MS=8000 UDPSCOPE_ENABLE_ALL_MS=600 UDPSCOPE_EXIT_DUMP=1 + UDPSCOPE_PROBE_RESEQ_MS=5000`. + + | case | mock extra flags | key reading | + | --- | --- | --- | + | **base** | *(none)* | every counter 0; proves the machinery costs nothing | + | **reorder** | `--reorder-every 10` | `reseq=51` (mid-run) / `102` (dump); `holes=0`; resequencing inside the window leaves no reservation | + | **late** | `--reorder-every 10 --reorder-delay-ms 50` | `holes=42 bands=42 bands_repaired=42 nan_slots=0`; every band is the faint, repaired tier | + | **mixed** | `--drop-every 20 --reorder-every 10 --reorder-delay-ms 50` | `bands=42 bands_repaired=20 nan_slots=2300`; all three band states at once | + + Full measured readings (probe line first, then exit-dump; `bands_repaired` was added after the + original probe and any earlier reading without it is obsolete): + + ``` + === base mock: none + [reseq-probe] reseq=0 late=0 stale=0 holes=0 hole_slots=0 backfilled=0 open_holes=0 bands=0 + bands_repaired=0 nan_slots=0 scanned=44000 trigger_in_hole=0 + [udpscope] frames=1024 packets=1025 gaps=0 channels=4 span=838.861s stride=1 reseq=0 late=0 + stale=0 holes=0 hole_slots=0 backfilled=0 + [udpscope] channel 0 'sine' stored=95200 published=95000 + + === reorder mock: --reorder-every 10 + [reseq-probe] reseq=51 late=0 stale=0 holes=0 hole_slots=0 backfilled=0 open_holes=0 bands=0 + bands_repaired=0 nan_slots=0 scanned=44000 trigger_in_hole=0 + [udpscope] frames=1028 packets=1029 gaps=0 channels=4 span=838.861s stride=1 reseq=102 late=0 + stale=0 holes=0 hole_slots=0 backfilled=0 + [udpscope] channel 0 'sine' stored=95400 published=95100 + + === late mock: --reorder-every 10 --reorder-delay-ms 50 + [reseq-probe] reseq=142 late=48 stale=0 holes=42 hole_slots=4200 backfilled=16800 open_holes=42 + bands=42 bands_repaired=42 nan_slots=0 scanned=44000 trigger_in_hole=0 + [udpscope] frames=1026 packets=1027 gaps=100 channels=4 span=838.861s stride=1 reseq=288 + late=100 stale=0 holes=94 hole_slots=9400 backfilled=37600 + [udpscope] channel 0 'sine' stored=95700 published=95500 + + === mixed mock: --drop-every 20 --reorder-every 10 --reorder-delay-ms 50 + [reseq-probe] reseq=163 late=24 stale=0 holes=43 hole_slots=4300 backfilled=8000 open_holes=43 + bands=42 bands_repaired=20 nan_slots=2300 scanned=44100 trigger_in_hole=0 + [udpscope] frames=974 packets=975 gaps=101 channels=4 span=838.861s stride=1 reseq=314 late=50 + stale=0 holes=94 hole_slots=9400 backfilled=18400 + [udpscope] channel 0 'sine' stored=95100 published=94900 + ``` + + The `mixed` case's `open_holes=43` against `bands=42` is correct: the 43rd hole has scrolled + out of the visible window, and `build_invalid_spans()` drops a span outside the window rather + than pinning it to the edge. The per-field reference is in + [probe-readings.md](probe-readings.md). + The mock also carries three **fault injections** that no probe schedules, because what they exercise is the ingest rather than the UI. The first: `--time-glitch-at --time-glitch-by ` makes **one** packet's time signal jump ahead and then carries on from where the true timeline was, which @@ -229,12 +282,12 @@ session for that port; the figures above are the default 2 s window (50 % pre), `~/.config/udpscope/_.toml` aside to reproduce them. The other two are **packet loss** (`--drop-every N`, `--drop-burst K`) and **wire inversion** -(`--reorder-every N`). Both withhold a datagram the counter and the stream clock have already -advanced over, which is what makes them loss and reordering rather than a slower producer: the -client orders the stream by the header's counter alone (`udps_client.c:504`), counting a forward -jump as `counter_gaps` and rejecting anything that does not advance it as `stale_packets`. There is -**no resequencing and no interpolation** anywhere below the ring — an inverted packet's samples are -discarded and the hole they leave is a jump in x, which is the behaviour these options are for. +(`--reorder-every N`). Loss withholds a datagram the counter and the stream clock have already +advanced over; the client counts the forward jump as `counter_gaps`. An inversion withholds a +packet until the one behind it is sent, so the receiver sees a later counter first; a packet that +arrives within the 20 ms reorder window is resequenced with no hole, and one that arrives past it +leaves a committed hole that it then backfills (`UDPSCOPE_PROBE_RESEQ_MS` exercises both paths). +There is **no interpolation** anywhere below the ring. All of the readings below are off `UDPSCOPE_EXIT_DUMP`'s `[udpscope] frames= packets= gaps=` line, over an 8 s run against `--rate 10000 --batch 100 --freq 5` (100 packets a second) except where the @@ -247,11 +300,10 @@ including the clean one, then read a reconnect instead of what it was testing: - `--drop-every 10`: `frames=925 packets=926 gaps=102`. Every tenth packet never reaches the wire, so `packets` falls with `frames` and `gaps` is the count of them. No warning: a one-packet hole is a one-packet jump in the stamps, far inside the glitch guard. -- `--reorder-every 10`: `frames=925 packets=1028 gaps=102`. This is the signature to look for — - `packets` is the clean figure (every datagram arrived) while `frames` is the dropped one, because - the overtaken packet is decoded far enough to be read as stale and then thrown away. One inversion - costs one `gaps` (for the packet that overtook it) **and** one `stale_packets`, which udpscope does - not currently display; the ~104 missing frames are the only place it shows. +- `--reorder-every 10`: `frames=1028 packets=1029 gaps=0`. The inverted packet lands inside the + 20 ms reorder window and is resequenced: every datagram arrives and `gaps` is 0. The pre-resequencing + signature was `frames=925 packets=1028 gaps=102`; that is now the reading for + `--reorder-every 10 --reorder-delay-ms 50`, which forces a late arrival past the window. - `--drop-every 1000 --drop-burst 200` at `--rate 100000 --batch 100` (1000 packets a second): `frames=7000 packets=7001 gaps=1600`, eight bursts of 200, and still no warning. - `--drop-every 1000 --drop-burst 300`, same stream: `frames=6203 packets=6204 gaps=2400` **and the diff --git a/docs/superpowers/specs/2026-10-08-resequencing-and-relative-timing-design.md b/docs/superpowers/specs/2026-10-08-resequencing-and-relative-timing-design.md index aa00eef..32f2076 100644 --- a/docs/superpowers/specs/2026-10-08-resequencing-and-relative-timing-design.md +++ b/docs/superpowers/specs/2026-10-08-resequencing-and-relative-timing-design.md @@ -140,12 +140,18 @@ cannot be misdiagnosed. `todo.md` requires absolute stream time to remain available for export. There is exactly one f64 time ring per `BufferStore`, shared by every channel of that buffer, and relative mode takes it. Export therefore needs a **second** f64 ring carrying the absolute stamp under the same group-mean -rule, written in relative mode only and read by export and by nothing else — eight bytes per -stored slot. +rule, written in **relative mode only** and read by export and by nothing else — eight bytes per +stored slot. The absolute ring is not kept unconditionally; it is budgeted along with the main +ring and the value rings, which is why it halves the retained span rather than costing a +proportional fraction (see SPECS.md for the measured figures and the power-of-two explanation). This cost is paid rather than avoided. Without it, selecting relative mode silently degrades the one output `todo.md` says must keep absolute time. +The measured history cost: **83.886 s** in stream mode and **41.943 s** in relative mode, at +100 kS/s, 4 channels, `--buffer-mb 200`. The plan's ~22% figure and the per-byte 75% figure are +both wrong because `Partition::time_capacity` is floored to a power of two. + ## The receive path ### In the client: a reorder ring of reassembled payloads @@ -226,9 +232,11 @@ thread folds what was pushed into the pyramid. That split carries straight over `[produced_ - capacity, produced_)`, a single relaxed store. The same race class as a push, and an `f32` cannot tear. Measurements and export read the ring directly, so they see a repair immediately. -- **Pyramid amend — main thread, inside `acquire()`.** The producer enqueues `(slot, value)` on an - SPSC queue; `ChannelBuffer::acquire()` drains it **after** its append pass, so every amended - index is already published. +- **Pyramid amend — main thread, inside `acquire()`.** The producer appends amended `(slot, value)` + pairs to a plain `std::vector` member; `ChannelBuffer::acquire()` drains it **after** its + append pass, so every amended index is already published. No SPSC queue is needed: `acquire()` is + called under `model.mutex()` and the producer holds that same mutex for all of `on_data`, so the + vector is correctly serialised without lock-free machinery. The thread split is not a stylistic choice. A bucket is 32 bytes and is not atomic, so amending it from the producer would let a reader see a half-updated bucket whose `sum` does not match its diff --git a/todo.md b/todo.md index 117ae3a..2a8a534 100644 --- a/todo.md +++ b/todo.md @@ -2,13 +2,14 @@ 1. the antialisaing strategy with phosphore effect should be applied with more stregth 2. the lines should be more tick 3. the sample points should be bigger -2. relative timing: +~~2. relative timing: - in order to avoid problem with time jitter user can decide to go in relative time mode - in this mode absolute signal time is only use for export - the plot time depends just on signal sample rate and frame number, e.g.: - user is acquiring a signal with 1MSps and 100_000 samples per udp frame - last frame as number and previous in this case a 100_000 indexes must be skipped - - time is simply: `(( - ) * - ) / ` (or something like that) + - time is simply: `(( - ) * - ) / ` (or something like that)~~ + done: `--relative-time`; see README and SPECS.md for the measured history cost. 3. named configurations: in order to have multiples configurations for the same sources configuration can also have comments 4. optional udp frame storage for debugging of strange behaviours (store last , default = 1000, udp packets binary content on disk for forensic analyss) 5. open exported binary waveforms to view/analyse, in the source selection dialog there should be the option to open exported waveform (maybe a tab with latest opened file and option to open new ones?) diff --git a/tools/mock_udpstreamer.py b/tools/mock_udpstreamer.py index eb18559..c798abb 100644 --- a/tools/mock_udpstreamer.py +++ b/tools/mock_udpstreamer.py @@ -271,10 +271,11 @@ def main(): type=int, default=0, help="hold every Nth packet back and send it after the next one that goes out -- " - "an inversion on the wire. The receiver takes the later counter first, so the held " - "packet no longer advances the counter and is stale by definition: one inversion " - "costs one counter gap, one stale packet, and that packet's samples for good. A " - "--drop-every that swallows the packet after a held one deepens the inversion " + "an inversion on the wire. What the receiver does depends on how late the held " + "packet arrives: inside the 20 ms reorder window it is resequenced with no hole; " + "past it a hole is committed first and the packet backfills it. " + "--reorder-delay-ms controls which path is taken (see its help). " + "A --drop-every that swallows the packet after a held one deepens the inversion " "rather than cancelling it, since the held packet waits for the next packet " "actually sent. 0 = never", )