udpscope/docs/claude/invariants-data-path.md

Invariants: the data path — buffers, decimation and the rings

What rebuilds a buffer, why decimation is an average, and the two rules the time ring's precision and its first stamp impose.

Part of CLAUDE.md. These are rules that cost a defect to learn; read the ones touching the area you are editing before you edit it.

• Any structural change rebuilds the buffer and drops history: a CONFIG, an enable/disable, a calibration change, or a change of display window that moves the decimation factor. Calibration is applied at ingest, which is why it re-partitions. ScopeModel::set_target_span() is the window's path in, and it re-plans only when time_stride actually changes, so walking the window inside one decade is free; it asks plan_partition() rather than recomputing the factor, because two expressions for that decision is how the buffer and the view end up disagreeing about the stored rate. It takes mutex() itself — do not call it holding the lock.
• The shared clock is decimated to fit the requested window; channels never are. A window too long to store at the wire rate used to be refused (the control was capped to Partition::span_s), which made a 600 s view of a 1 MS/s stream unreachable. plan_partition() now picks time_stride, a power of two, after the growth loop has bought all the ring the budget allows — deciding it earlier over-decimates by whatever factor growth was about to add. Per-channel stride[i] stays 1: a stride of round(dt_i / dt_fastest) keeps one sample per stride, and when that equals the signal's own period — a 2 kHz sine on a 1 MHz clock, stride 500 — every kept sample sits at the same phase and the channel plots flat.
• Decimation at ingest is an average, never a stride. Acquisition::on_data() reads and accumulates every sample and pushes the mean of each group of time_stride (value_acc_/value_acc_n_, and time_acc_ for the clock). That boxcar's first null is at the stored rate, so content that would fold back is attenuated instead of reappearing as a false low frequency — which is the whole reason decimation was once dropped entirely. Skipping the read for unstored samples re-introduces the aliasing; the UDPSCOPE_PROBE_WINDOW_MS run under --freq 2000 is the guard (the anti-alias check in probe-readings.md). The stamp pushed is the group's mean, not its last member: that is where the boxcar's group delay puts the value, and stamping at the end of the group shifts every trace half a stored sample late. The accumulators are reset whenever time_stride or the channel count changes, so a re-plan starts a fresh group rather than finishing one at the wrong length. The LOD min/max pyramid still preserves the extremes of what is stored; a spike narrower than a group is now filtered away rather than aliased, which is the correct trade and is why the status line's rate field is worth reading.
• One bad packet is bridged; only a run of them changes the time source. The stream-time checks in build_stamps() reject three things: stamps that are not finite, a packet with no span at all, and a packet whose stamps do not advance past the previous packet's last. A fourth was added because a real device's clock glitches: a packet whose stamps jump more than 256 packet spans ahead is a fault, not a gap, and must be rejected — letting it through poisons last_stamp_, and the monotonic clamp (max(stamps[k] - epoch, last_stamp_)) then pins every later packet flat at the glitched value, so the whole stream freezes at one x and never recovers. A rejected packet is bridged: its stamps are replaced by a continuation of the accepted timeline at the declared cadence, so neither the monotonic mark nor last_stream_anchor_ is poisoned and the next good packet is accepted normally. Believing the glitch instead is what condemned the next, perfectly good packet (it now sits below the poisoned anchor, so it fails the advance test) and threw the whole stream onto the arrival clock. Only four consecutive rejected packets abandon the declared time signal, because that is what a signal which is not a time base at all looks like; a first packet with nothing to bridge from still falls back immediately. The unit scale ('ns', 'us', 'ms') is therefore resolved before these checks rather than after: they weigh the stamps against a cadence in seconds, and on a 'ns' stream an unscaled comparison is off by 1e9, so every packet would read as an implausible jump. The guard is tools/mock_udpstreamer.py --time-glitch-at/--time-glitch-by (see verification.md).
• The arrival-clock fallback does not drop history. arrival_epoch_ is derived from last_stamp_, so the arrival clock continues from the monotonic mark and the two timelines share a zero by construction — the recorded stamps are therefore still on one timeline and are kept. The fallback used to call reset_live_history_locked() on the reasoning that the old stamps belonged to a dead clock; they do not, and that wipe is what made a single glitching packet look like "the live stream restarted from 0 with no action from the user". The continuity is also what makes a rebase of TriggerEngine unnecessary: the engine holds four absolute stamps (holdoff_until_s_, pending_start_s_, last_trigger_s_, last_t_s_) and they stay valid across the change. Anything that makes the clock discontinuous breaks that and has to reset them.
• The trigger sees the stored (filtered) signal, not the wire signal. engine.process() is fed the averaged value, inside the if that pushes it. An edge found on a sample that is not in the buffer would be cut from data that does not contain it.
• The time ring is f64 and the value rings are f32, and the asymmetry is the point. A value's precision is a property of the signal (24 bits is plenty for a sample); a timestamp's is a property of how long the scope has been running, because the stamps are stored relative to an epoch fixed at the first sample ever seen and so grow without bound while the interval they must resolve stays one sample period. f32's ULP is 1.9 µs at 16 s of uptime, 3.8 µs at 32 s, 7.6 µs at 64 s and 61 µs over a 600 s window — so at 1 MS/s two to four consecutive samples shared a stamp after half a minute, eight after a minute, and the LOD envelope over such a plateau drew a zero-width segment, i.e. a vertical spike where the waveform should be. That is "zoom in to ~10 µs and three to five points share a timestamp, and min/max produces artefacts". Do not widen the values to match: they are the memory budget, and nothing about them is magnitude-bound this way. Three consequences: plan_partition() can no longer count slots in its growth loop (a time slot and a value sample are different sizes, so it weighs the budget in bytes — sizeof(f64) for the clock plus sizeof(f32) / stride[i] per channel — and bytes_for() is also what bounds the doubling against overflow); live_window_locked() compares its bounds in the ring's own type with no rounding step, the old f32 round-trip having existed only so that a sample sitting exactly on a bound was kept whatever the stream's absolute time was; and EventSnapshot::stamps is f64 because it is cut from this ring and is therefore in the same units, whatever a given snapshot's own origin is (both kinds are rebased now — an event's stamps are edge-relative, a pause's freeze-relative — so neither is magnitude-bound the way the ring is, and the width is for the cut rather than for the coordinate). RingBuffer<T> stores slots as std::atomic<T> behind a static_assert(is_always_lock_free), which double satisfies on every supported target. The guard is the export: over a multi-million-row .npy, np.diff(t) must hold one step to within a few ULP of the column's magnitude (see the export guard in probe-readings.md for why it is no longer exactly one value). Three steps at 2, 2.5 and 3 ULP of the stream's uptime — microseconds, not femtoseconds — is this defect.
• Append is not the only write. A backfill amends slots that a reservation wrote as NaN, and the amend is split across threads for a reason:
  • Ring amend — producer thread. RingBuffer<T>::amend(absolute, v) is a single relaxed store, bounds-checked against [produced_ - capacity, produced_). The same race class as a push, and an f32 cannot tear. Measurements and export read the ring directly, so they see a repair immediately and no second notification is needed.
  • Pyramid amend — consumer (GTK main thread), inside ChannelBuffer::acquire(). The producer collects amended values in a plain std::vector<Amend> member and the consumer drains it after its own append pass. acquire() is called under model.mutex(), and the producer holds that same mutex for all of on_data, so the vector is correctly serialised with no lock-free queue. The reason the pyramid amend cannot also be on the producer: a bucket is a 32-byte non-atomic aggregate that the renderer reads unlocked at paint time, so a producer writing it could be caught mid-bucket — a count that does not match its sum, producing a wrong mean from a structure whose readers trust count == 0 to mean absent. On the consumer that cannot happen.
  • A hole is stamped at reservation; backfill changes values only. The f64 time ring is never amended. neighbours()' binary search requires a monotonic array, and window_x0_ must remain valid; pre-stamping the hole at the declared cadence (exact in relative mode, the bridge mechanism in stream mode) is what keeps both invariants, and it is why a late packet's counter, not its arrival stamp, is what addresses the hole. The reservation cap (kGlitchPackets, 256) is counted in **packets, not samples**, so one hole's sample cost is 256 × batch_` and scales with the stream's batch — 25 600 samples at a batch of 100, 2.56 M at a batch of 10 000 — which is the number to weigh against the ring, not the 256.
  • A re-plan drops every hole and restarts the producer's tick. rebuild_locked() replaces the store, so produced() restarts at 0 and a hole's slot_begin/tick_begin/x_begin describe a buffer that is gone; it therefore clears holes_ and bumps ScopeModel::plan_generation(), which the producer compares against the value it last saw and resets tick_ on. That is what keeps tick_ / period == produced(), the relation backfill_from() derives a slot index from — without it the amend falls outside the ring and repairs nothing while the hole is still marked repaired, a band saying the opposite of the truth. The generation is the signal and the stride/channel-count comparison is not: a reconnect with an identical CONFIG rebuilds the store leaving both unchanged. slot_base_ is not reset — it is relative mode's clock origin, samples since the stream began, and relative time must not jump backwards when the user retypes the window.
• A dropped time slot is clamped, never answered with a sentinel. acquire() pins the published window to the value ring's base and both rings keep sliding afterwards — their capacities are equal, so there is no slack — which means the oldest few samples of a saturated window routinely point at time slots the producer has already overwritten. ChannelBuffer::time_of() therefore clamps to the oldest readable slot: a stamp of 0 would reach x_bounds(), and from there dx() and x0(), which is the renderer's whole index mapping — one unreadable slot inflated dx() by the ratio of elapsed stream time to the retained window and clipped the older part of every live view. The library's StreamSeries is immune for a reason worth copying: its x is x0 + i*dx, a function of the absolute index, so an overwritten slot costs a y value, never the geometry. Clamping buys the same bargain here.
• The window's first stamp is cached by acquire() and never read at paint time. The clamp above makes an unreadable slot survivable; it does not make reading it safe, and absolute(0) is not an ordinary slot — acquire() pins the window to values_->base(), so sample 0 is the ring's oldest, the very slot the producer is about to overwrite. time_of() checks the bounds and then copies, and when the producer laps the slot in between the stamp it returns is one whole ring-span too new. Everything else on a ChannelBuffer is read at paint time with no lock, and x0() was the origin the renderer reconstructed every trace point from (gather[i] + series.x0()), read both inside its increasing-suffix filter and again at stage_xy() — so one torn read of that one slot produced three pictures, all of them "the waveform is gone while the grid and the axes stay": neighbours() tests against x_at(0) and refused the whole visible range (the renderer then gathered the entire ring at a coarse level and drew it off the axis — idx=[0..8388608) level=7 cover=320%); x0() changing during the filter loop dropped three quarters of the points; and stage_xy() shifted the finished polyline a ring-span off its axis. The gather origin is the view's minimum now (see the gather() invariant), a local of the renderer's, so the second and third of those are structurally impossible — but the first is not, x_at(0) is still what neighbours() tests the visible range against, and that is the one that gathered the whole ring. The cache is load-bearing for that reason; do not read the slot at paint time because two of its three symptoms are gone. That is the "with measurements and cursors active the plot flickers as if there were no data, at no constant frequency" defect. It is live only (a snapshot is immutable, so x0() is constant for Held/Paused), it is a race against packet arrival so it is irregular, and the measurements pane makes it both likelier and larger: the pane holds mutex() for each MeasureWindowReader's life, so the released producer bursts several packets at once. acquire() runs on the consumer under ScopeModel::mutex() — the same lock the producer's pushes hold — and the window is pinned on the line above, so that is the one point in the object's life where the slot is safe to read; window_x0_ is also the right answer rather than merely a stable one (it is the stamp sample window_base actually carries, where a paint-time clamp answered with whatever slot happened to be oldest now). x_at(0) is served from it too, for the same reason, and invalidate() clears it with the window. The guard is trace ink: / [ink] (see probe-readings.md). The same torn read still exists for a small i > 0 — the oldest frame or so of samples — but there its observable is a prefix plateau, which is exactly what the renderer's increasing-suffix filter was written to absorb, so it is a different symptom and deliberately not fixed here.