udpscope/docs/claude/invariants-data-path.md

# Invariants: the data path — buffers, decimation and the rings

What rebuilds a buffer, why decimation is an average, and the two rules the time ring's precision and its first stamp impose.

Part of [CLAUDE.md](../../CLAUDE.md). These are rules that cost a defect to learn; read the ones touching the area you are editing before you edit it.

- **Any structural change rebuilds the buffer and drops history**: a CONFIG, an
  enable/disable, a calibration change, or a **change of display window that moves the
  decimation factor**. Calibration is applied at ingest, which is why it re-partitions.
  `ScopeModel::set_target_span()` is the window's path in, and it re-plans *only* when
  `time_stride` actually changes, so walking the window inside one decade is free; it asks
  `plan_partition()` rather than recomputing the factor, because two expressions for that
  decision is how the buffer and the view end up disagreeing about the stored rate. It
  takes `mutex()` itself — do not call it holding the lock.
- **The shared clock is decimated to fit the requested window; channels never are.** A
  window too long to store at the wire rate used to be refused (the control was capped to
  `Partition::span_s`), which made a 600 s view of a 1 MS/s stream unreachable.
  `plan_partition()` now picks `time_stride`, a power of two, **after** the growth loop has
  bought all the ring the budget allows — deciding it earlier over-decimates by whatever
  factor growth was about to add. Per-channel `stride[i]` stays 1: a stride of
  `round(dt_i / dt_fastest)` keeps one sample per stride, and when that equals the signal's
  own period — a 2 kHz sine on a 1 MHz clock, stride 500 — every kept sample sits at the
  same phase and the channel plots flat.
- **Decimation at ingest is an average, never a stride.** `Acquisition::on_data()` reads
  and accumulates *every* sample and pushes the mean of each group of `time_stride`
  (`value_acc_`/`value_acc_n_`, and `time_acc_` for the clock). That boxcar's first null is
  at the stored rate, so content that would fold back is attenuated instead of reappearing
  as a false low frequency — which is the whole reason decimation was once dropped
  entirely. Skipping the read for unstored samples re-introduces the aliasing; the
  `UDPSCOPE_PROBE_WINDOW_MS` run under `--freq 2000` is the guard (the anti-alias check in
  [probe-readings.md](probe-readings.md)).
  The stamp pushed is the group's **mean**, not its last member: that is where the boxcar's
  group delay puts the value, and stamping at the end of the group shifts every trace half
  a stored sample late. The accumulators are reset whenever `time_stride` or the channel
  count changes, so a re-plan starts a fresh group rather than finishing one at the wrong
  length. The LOD min/max pyramid still preserves the extremes *of what is stored*; a spike
  narrower than a group is now filtered away rather than aliased, which is the correct
  trade and is why the status line's `rate` field is worth reading.
- **One bad packet is bridged; only a run of them changes the time source.** The stream-time
  checks in `build_stamps()` reject three things: stamps that are not finite, a packet with no
  span at all, and a packet whose stamps do not advance past the previous packet's last. A
  fourth was added because a real device's clock glitches: a packet whose stamps jump **more
  than 256 packet spans** ahead is a fault, not a gap, and must be rejected — letting it
  through poisons `last_stamp_`, and the monotonic clamp (`max(stamps[k] - epoch,
  last_stamp_)`) then pins every later packet flat at the glitched value, so the whole stream
  freezes at one x and never recovers. A rejected packet is **bridged**: its stamps are
  replaced by a continuation of the accepted timeline at the declared cadence, so neither the
  monotonic mark nor `last_stream_anchor_` is poisoned and the next good packet is accepted
  normally. Believing the glitch instead is what condemned the *next*, perfectly good packet
  (it now sits below the poisoned anchor, so it fails the advance test) and threw the whole
  stream onto the arrival clock. Only **four consecutive** rejected packets abandon the
  declared time signal, because that is what a signal which is not a time base at all looks
  like; a first packet with nothing to bridge from still falls back immediately. The unit
  scale ('ns', 'us', 'ms') is therefore resolved **before** these checks rather than after:
  they weigh the stamps against a cadence in seconds, and on a 'ns' stream an unscaled
  comparison is off by 1e9, so every packet would read as an implausible jump. The guard is
  `tools/mock_udpstreamer.py --time-glitch-at`/`--time-glitch-by` (see
  [verification.md](verification.md)).
- **The arrival-clock fallback does not drop history.** `arrival_epoch_` is derived from
  `last_stamp_`, so the arrival clock *continues* from the monotonic mark and the two
  timelines share a zero by construction — the recorded stamps are therefore still on one
  timeline and are kept. The fallback used to call `reset_live_history_locked()` on the
  reasoning that the old stamps belonged to a dead clock; they do not, and that wipe is what
  made a single glitching packet look like "the live stream restarted from 0 with no action
  from the user". The continuity is also what makes a rebase of `TriggerEngine` unnecessary:
  the engine holds four absolute stamps (`holdoff_until_s_`, `pending_start_s_`,
  `last_trigger_s_`, `last_t_s_`) and they stay valid across the change. Anything that makes
  the clock *discontinuous* breaks that and has to reset them.
- **The trigger sees the stored (filtered) signal, not the wire signal.** `engine.process()`
  is fed the averaged value, inside the `if` that pushes it. An edge found on a sample that
  is not in the buffer would be cut from data that does not contain it.
- **The time ring is `f64` and the value rings are `f32`, and the asymmetry is the point.**
  A value's precision is a property of the signal (24 bits is plenty for a sample); a
  timestamp's is a property of *how long the scope has been running*, because the stamps are
  stored relative to an epoch fixed at the first sample ever seen and so grow without bound
  while the interval they must resolve stays one sample period. `f32`'s ULP is 1.9 µs at 16 s
  of uptime, 3.8 µs at 32 s, 7.6 µs at 64 s and 61 µs over a 600 s window — so at 1 MS/s two
  to four consecutive samples shared a stamp after half a minute, eight after a minute, and
  the LOD envelope over such a plateau drew a zero-width segment, i.e. a vertical spike where
  the waveform should be. That is "zoom in to ~10 µs and three to five points share a
  timestamp, and min/max produces artefacts". **Do not widen the values to match**: they are
  the memory budget, and nothing about them is magnitude-bound this way. Three consequences:
  `plan_partition()` can no longer count slots in its growth loop (a time slot and a value
  sample are different sizes, so it weighs the budget in **bytes** — `sizeof(f64)` for the
  clock plus `sizeof(f32) / stride[i]` per channel — and `bytes_for()` is also what bounds
  the doubling against overflow); `live_window_locked()` compares its bounds in the ring's own
  type with **no rounding step**, the old `f32` round-trip having existed only so that a
  sample sitting exactly on a bound was kept whatever the stream's absolute time was; and
  `EventSnapshot::stamps` is `f64` because it is cut from this ring and is therefore in the same
  units, whatever a given snapshot's own origin is (both kinds are rebased now — an event's
  stamps are edge-relative, a pause's freeze-relative — so neither is magnitude-bound the way the
  ring is, and the width is for the cut rather than for the coordinate). `RingBuffer<T>` stores
  slots as `std::atomic<T>` behind a `static_assert(is_always_lock_free)`, which `double`
  satisfies on every supported target. The guard is the **export**: over a multi-million-row
  `.npy`, `np.diff(t)` must hold one step to within a few ULP of the column's magnitude (see
  the export guard in [probe-readings.md](probe-readings.md) for why it is no longer exactly one
  value). Three steps at 2, 2.5 and
  3 ULP of *the stream's uptime* — microseconds, not femtoseconds — is this defect.
- **Append is not the only write.** A backfill amends slots that a reservation wrote as NaN, and
  the amend is split across threads for a reason:
  - **Ring amend — producer thread.** `RingBuffer<T>::amend(absolute, v)` is a single relaxed
    store, bounds-checked against `[produced_ - capacity, produced_)`. The same race class as a
    push, and an `f32` cannot tear. Measurements and export read the ring directly, so they see
    a repair immediately and no second notification is needed.
  - **Pyramid amend — consumer (GTK main thread), inside `ChannelBuffer::acquire()`.** The
    producer collects amended values in a plain `std::vector<Amend>` member and the consumer
    drains it after its own append pass. `acquire()` is called under `model.mutex()`, and the
    producer holds that same mutex for all of `on_data`, so the vector is correctly serialised
    with no lock-free queue. The reason the pyramid amend cannot also be on the producer: a
    bucket is a 32-byte non-atomic aggregate that the renderer reads **unlocked at paint time**,
    so a producer writing it could be caught mid-bucket — a `count` that does not match its
    `sum`, producing a wrong mean from a structure whose readers trust `count == 0` to mean
    *absent*. On the consumer that cannot happen.
  - **A hole is stamped at reservation; backfill changes values only.** The f64 time ring is
    never amended. `neighbours()' binary search requires a monotonic array, and `window_x0_`
    must remain valid; pre-stamping the hole at the declared cadence (exact in relative mode,
    the bridge mechanism in stream mode) is what keeps both invariants, and it is why a late
    packet's counter, not its arrival stamp, is what addresses the hole. The reservation cap
    (`kGlitchPackets`, 256) is counted in **packets, not samples**, so one hole's sample cost
    is `256 × batch_` and scales with the stream's batch — 25 600 samples at a batch of 100,
    2.56 M at a batch of 10 000 — which is the number to weigh against the ring, not the 256.
  - **A re-plan drops every hole and restarts the producer's tick.** `rebuild_locked()` replaces
    the store, so `produced()` restarts at 0 and a hole's `slot_begin`/`tick_begin`/`x_begin`
    describe a buffer that is gone; it therefore clears `holes_` and bumps
    `ScopeModel::plan_generation()`, which the producer compares against the value it last saw
    and resets `tick_` on. That is what keeps `tick_ / period == produced()`, the relation
    `backfill_from()` derives a slot index from — without it the amend falls outside the ring
    and repairs nothing while the hole is still marked repaired, a band saying the opposite of
    the truth. The generation is the signal and the stride/channel-count comparison is not: a
    reconnect with an identical CONFIG rebuilds the store leaving both unchanged. `slot_base_`
    is **not** reset — it is relative mode's clock origin, samples since the stream began, and
    relative time must not jump backwards when the user retypes the window.
- **A dropped time slot is clamped, never answered with a sentinel.** `acquire()` pins
  the published window to the value ring's base and both rings keep sliding afterwards —
  their capacities are equal, so there is no slack — which means the oldest few samples
  of a saturated window routinely point at time slots the producer has already
  overwritten. `ChannelBuffer::time_of()` therefore clamps to the oldest *readable*
  slot: a stamp of `0` would reach `x_bounds()`, and from there `dx()` and `x0()`, which
  is the renderer's whole index mapping — one unreadable slot inflated `dx()` by the
  ratio of elapsed stream time to the retained window and clipped the older part of
  every live view. The library's `StreamSeries` is immune for a reason worth copying:
  its x is `x0 + i*dx`, a function of the absolute index, so an overwritten slot costs a
  *y* value, never the geometry. Clamping buys the same bargain here.
- **The window's *first* stamp is cached by `acquire()` and never read at paint time.** The
  clamp above makes an unreadable slot *survivable*; it does not make reading it *safe*, and
  `absolute(0)` is not an ordinary slot — `acquire()` pins the window to `values_->base()`, so
  sample 0 is the ring's oldest, the very slot the producer is about to overwrite. `time_of()`
  checks the bounds and then copies, and when the producer laps the slot in between the stamp it
  returns is **one whole ring-span too new**. Everything else on a `ChannelBuffer` is read at
  paint time with no lock, and `x0()` *was* the origin the renderer reconstructed every trace
  point from (`gather[i] + series.x0()`), read both inside its increasing-suffix filter and again
  at `stage_xy()` — so one torn read of that one slot produced three pictures, all of them "the
  waveform is gone while the grid and the axes stay": `neighbours()` tests against `x_at(0)` and
  refused the whole visible range (the renderer then gathered the *entire* ring at a coarse level
  and drew it off the axis — `idx=[0..8388608) level=7 cover=320%`); `x0()` changing *during* the
  filter loop dropped three quarters of the points; and `stage_xy()` shifted the finished polyline
  a ring-span off its axis. The gather origin is the *view's* minimum now (see the `gather()`
  invariant), a local of the renderer's, so the second and third of those are structurally
  impossible — but **the first is not**, `x_at(0)` is still what `neighbours()` tests the visible
  range against, and that is the one that gathered the whole ring. The cache is load-bearing for
  that reason; do not read the slot at paint time because two of its three symptoms are gone.
  That is the "with measurements and cursors active the plot flickers as
  if there were no data, at no constant frequency" defect. It is **live only** (a snapshot is
  immutable, so `x0()` is constant for `Held`/`Paused`), it is a race against packet arrival so it
  is irregular, and the measurements pane makes it both likelier and larger: the pane holds
  `mutex()` for each `MeasureWindowReader`'s life, so the released producer bursts several packets
  at once. `acquire()` runs on the consumer under `ScopeModel::mutex()` — the same lock the
  producer's pushes hold — and the window is pinned on the line above, so that is the one point in
  the object's life where the slot is safe to read; `window_x0_` is also the *right* answer rather
  than merely a stable one (it is the stamp sample `window_base` actually carries, where a
  paint-time clamp answered with whatever slot happened to be oldest *now*). `x_at(0)` is served
  from it too, for the same reason, and `invalidate()` clears it with the window. The guard is
  `trace ink:` / `[ink]` (see [probe-readings.md](probe-readings.md)). The same torn read still exists for a small `i > 0`
  — the oldest frame or so of samples — but there its observable is a *prefix plateau*, which is
  exactly what the renderer's increasing-suffix filter was written to absorb, so it is a different
  symptom and deliberately not fixed here.