udpscope/docs/claude/invariants-measurements.md

Invariants: the measurements pane

Why the pane is refreshed on a cadence, why neither pass may read its window raw, how a crossing is counted, and which mean over a split is which.

Part of CLAUDE.md. These are rules that cost a defect to learn; read the ones touching the area you are editing before you edit it.

• The measurements pane is refreshed on a cadence, and its cost is the window's, not the frame's. A measurement is a statistic of the visible window and a live view moves that window every frame, so "recompute when the window moves" is "recompute every frame" at O(visible samples) per measurement. With the pane holding four measurements over a 4.7M-sample window that measured 13.66 ms of tick mean (baseline 0.41 ms) and cost the run a third of its frames. Three things fixed it, and all three matter: update_measures() recomputes at most every kMeasureRefreshUs (100 ms; 400 ms when a pass costs more than kMeasureHeavyUs), once when the window settles and once per mode change -- and reading that backwards is what made it worse first, 457 recomputes in one run, because a pinned window then recomputed every frame; the window is materialized in bounded chunks (never two whole-window f32 temporaries) and the loop holds it by reference (copying both vectors per measurement was 300 MB of memcpy a frame); and measure_stats() is one shared pass carrying both extremes with their first positions, the sum and the count, so average/min/max/peak-to-peak need no pass of their own -- top/bottom/amplitude take a second read of that same bucket list and never of the window (the flat-level invariant below). Adding a type must not add a pass -- that is what compute_measure()'s stats argument is for. The window a value is computed over is the pin while the view is pinned, and the viewport otherwise. A pinned view -- a held event, a paused stream, an opened file -- is a still frame: the data behind it does not change, so a pan or a zoom only looks closer. Measuring the pin (RowSet::pinned_range(), the same accessor hold_pinned_window() clamps into, so "what frames this view" has one expression) is what makes a gesture change no value at all -- and it is decided in update_measures(), at the call site, rather than inside MeasureWindowReader, because the decision is also booked: the window MeasureWindowReader is handed is the same one the "it moved" test compares against App::measure_window, so a gesture inside the pin spends no recompute either. A reader that chose its own window would leave a pinned view recomputing on every gesture frame for a number that cannot change (457 passes in one run is the shape that mistake took). This test has been wrong in both directions and the reasoning is worth keeping. Suppressing it while held (moved = ... && !held) left the pane describing the whole 2 s event (for=[0.47..2.47]) while a 50 ms slice of it was on screen; removing the exemption was the opposite mistake, and it is the reported "when measuring a still frame (a triggered event, or a paused stream) moving the zoom makes measurements such as average or top change, which should never happen". The two are one question asked of two different windows: which window is this pane about (the pin, while pinned) and has it moved (asked of that window in every mode). Answers: a pinned view still recomputes when it should -- a fresh latch, a re-stated pin, a new snapshot, the held/live change -- and a settled one satisfies swin == measure_window exactly and so reads moved == false on its own. [measurepin-probe] is the reading: on a latched event zoomed to a quarter of itself, same=4/4 for_same=1 pass_delta=0 with the values bit-identical and for= still at the pin; the same on a pause taken from a held event. Live is the case that must not be swallowed, and it reads same=0/4 for_same=0 with a positive pass_delta, span_before=2.000000, span_after=0.125000 and follows=1 -- the results window narrowed with the viewport and the values moved with it. The one path that is not this rule is the window control: apply_time_window() unpins (set_time_window() clears pinned_), so for the frames before the held branch re-states the pin the view really is live-framed and the pane measures the viewport -- [measure-trace] prints pin=off for exactly those frames, which is what makes them legible rather than looking like the defect again. The marks' ink is set where they are created (install() in plot_host.cpp): 0.75 px and dashed 4/4 through Plot::set_cursor_stroke(), which is per cursor because the trigger marker (also a cursor) keeps the solid line_width_px stroke. A new measure mark must take that stroke -- it is created in the same loop, so it does; a new instrument should not. The marks the pane draws (set_measure_marks() -- full-width h_rulers and full-height cursors, i.e. the "lines across the plot") are re-placed every frame from the result, so a throttled pass is a stepped ruler: acceptable, because the ruler and the number beside it come from the same pass and agree. What is not acceptable is a ruler that appears and disappears, so an invalid pass falls back to the last valid result for its level marks and only drops its time marks: a level is a property of the signal (a mean, a top, an amplitude) and outlives the window it was measured in, while a time mark is a position inside that window and may not. Both are emitted in one pass after the numbers, never inside the loop that computes them: that loop has three continues (an unresolvable signal, a channel with no window) and each one was a way for the rulers to blink off and back. Fixing one and not the others is how that bug survived a round of "fixes". clamp fires: N in the dump is the other half of this: clamp_to_data() compares with a relative tolerance, because clamping to exactly window_s_ can leave the group's span an ULP above it and a clamp that fires every frame is a re-frame loop, not a clamp.

• A measurement must not read its window raw. Both passes go through plot::data::MinMaxPyramid, which the producer already maintains at amortised O(1) -- so the pane inherits the summary of the new samples rather than computing one, which is what the "evaluate only on new samples" ask comes to here. Throttling the cadence (above) bounded how often; this bounds how much, and it is the larger factor: 19.7 ms per recompute to 1.2--1.6 ms, reading 0.1 % of a 2 M-sample window. Pass one (measure_scan_buckets()) adds whole level-0 buckets -- 64 samples each, carrying min, max an exact sum and a finite count -- and locates each extreme to a bucket; one raw measure_scan() of that bucket alone then pins it to a sample, two buckets at most whatever the window's size. Pass two (measure_accumulate_bucket()) folds a bucket's sums only when it lies strictly on one side of its level (k.lo > level or k.hi < level): then every sample falls in the same bin and none is at the level, so the fold is exact and not an approximation. It asks separately whether the bucket enters a side of the hysteresis band (see the invariant below), because a single test against level ± h refuses exactly the buckets a crossing passes through. A bucket wholly past a threshold holds that crossing at its first sample, exactly -- which is where the raw walk would have found it. It refuses a bucket that straddles the level, one that enters a side somewhere inside itself, and one holding a non-finite sample (the raw walk takes the first finite sample of the new side and an aggregate cannot say which that is); the caller then opens that bucket. Nothing may be folded before a refusal -- a refused bucket goes to measure_accumulate(), which adds the same samples again, and sums added above the return false double-counted every opened bucket. That read as top=0.614304 on a unit sine against the 2/pi a mean over a half then gave; a bucketed window no longer reports top from those sums at all (see the flat-level invariant below), which makes the rule easier to break silently rather than less binding -- the sums are the fallback a window with no histogram reads, so a wrong one is a wrong answer waiting for that window. Four things this rests on, each of which breaks it silently:

  • Exactness is the point, so nothing may stride or decimate. peak to peak must still read exactly 2 on a unit sine, and it and average still come straight out of the one shared measure_stats() pass. A bucket's sum is a sum. (top has no exact reading to protect any more: it is a density estimate and not a mean, so the assertion that used to be top == 0.636620 on a sine now belongs to above 50, which is a mean over a side and reads exactly 2/pi again -- see the band-average invariant below, and the flat-level one for what top reads instead. Everything above is about the bucket path being a fast path and not an approximation, and that is unchanged.)
  • The bucket index space must be the window's. ChannelBuffer::acquire() calls pyramid_.drop_before(values_->base()), so a channel's pyramid is in the value ring's absolute index space -- the same space live_window_locked() answers in, since a channel's stride is always 1. The snapshot's pyramid is built from sample 0 and never dropped from, so its space is the snapshot's own. MeasureWindowReader::origin_ is what turns one into a window offset, and a mismatch there measures the wrong samples rather than failing.
  • The raw path stays, and stays exact, for the window's partial edge buckets, for the samples produced since this frame's acquire_all() (one frame of production, so bounded), and for the straddling buckets. A window shorter than a bucket degrades to O(window) and must still be right.
  • Both passes walk the same fetched bucket list (App::measure_bucket_scratch, reused across recomputes -- a 2 M-sample window is 32 k buckets). Fetching twice is a second walk of the pyramid for nothing. measure cost:'s covered/buckets/raw is the guard, and it needs the fast mock to say anything.

• A pass costs the window's buckets times the pane's measured channels and its distinct levels, and the window's own size multiplies all of it. The cadence above bounds how often; this is how much, and it is the term a big time scale buys. Measured on the mock at the user's scale -- a 15 s window, --buffer-mb 4000 (so time_stride stays 1 and the window is 11.8 M samples, 184 k level-0 buckets), four signals, one row -- with [levels-probe] (see verification.md), measure cost: reading recomputes/each/covered/buckets/raw:

  pane	each	raw	tick mean
  empty	--	--	1.34 ms
  1 measurement, 1 level	6.58 ms	0.1 %	1.99 ms
  4 measurements, 4 shares on one channel	10.28 ms	0.4 %	2.38 ms
  16 measurements, 16 shares on one channel	24.55 ms	1.2 %	3.86 ms
  4 measurements, 1 share each on four channels	35.69 ms	2.6 %	4.58 ms

  So a pane is priced per measured channel (one MeasureWindowReader, one fetch, one scan) and per distinct share (one accumulator), and the window's samples are the multiplier on both: the same pane over a 2 M-sample window is the 1.2--1.6 ms a recompute this pane's budget was fixed at, and a machine's ten noisy channels at a 15 s window is tens of ms a pass, several times a second. That lands on the tick -- update_measures() runs inside it -- so frame cost:'s tick mean is where a pane's cost shows (1.34 -> 4.58 ms above), and it is not an ingest problem: gaps=/packets= stay exactly clean even though each MeasureWindowReader holds model->mutex() for its whole life, because a socket buffer absorbs a few tens of ms of producer. Read the tick mean and each before blaming the producer. Inside a pass the terms are, instrumented per window at that scale: the fetch 3.3 ms (three out-of-line MinMaxPyramid accessors and a 48-byte MeasureBucket per bucket), measure_scan_buckets() 0.8 ms, and pass two 7.5 ms -- dominated by the per-(bucket, level) fold. That fold is why measure_accumulate_bucket() is inline in measure.hpp: it is the innermost loop of a walk whose iteration count is the window's buckets times the pane's levels, and as an out-of-line call in its own translation unit it cost 30 % of the pass (sixteen shares on one channel: 24.55 -> 17.25 ms a recompute). A caller adding a per-bucket type keeps it in the header, and a caller adding a bulk one may not let it grow a call per bucket either. The third term is raw, and it is the signal's: a level that falls inside a bucket's [lo, hi] has to be walked sample by sample (the fold cannot split a bucket), so a signal whose local range covers the level everywhere -- white noise, a noisy machine channel -- degrades the pass to O(window) per level: measured raw=4.6 % on the mock's noise channel against 0.1 % on its sine at the same window. It is the field to read before blaming the pane for a slow run, and no cheaper walk fixes it.

• Top/Bottom/Amplitude read a flat level, and the density that finds it is the buckets', not the samples'. A band-limited edge overshoots and rings -- the mock's square at 30 harmonics reaches ±1.179 where its plateau is ±1 -- so a measurement that reads the extremes, or a mean over a side of a level, reports the ringing rather than the level the signal dwells at: that is what made this change (amplitude read 1.179350 and top 0.986496 on the rippled square, against 1.179350 of overshoot where the plateau is 1). A flat level is a density question: the plateau holds most of the samples and the ringing a few percent. So measure_scan_buckets() folds a 64-bin histogram of the whole buckets' midpoints while it has the list in hand -- one observation per bucket, each bin carrying its buckets' exact sum and finite count -- and measure_flat_levels() takes, either side of the split, the densest bin that holds at least half the densest one's buckets, the outermost of those, and the exact mean of the samples of the buckets within 5 % of the window's peak-to-peak of it. 5 % is under the ~9 %-of-swing Gibbs overshoot, so the ringing's excursion falls outside the band while a plateau's own ripple stays inside it; the outermost-of-the-equally-dense rule is what makes a waveform with no plateau -- a sine, a ramp -- read its peak instead of an arbitrary mid-range bin that a shifted window would move. Amplitude is then top - bottom, a bench scope's Vamplitude, and not the half-swing it used to be, nor the peak-to-peak swing. The readings, fast mock, 2 s window, [measure-trace], harmonics 0 → 30: top 1.000000 → 0.999983, bottom -1.000000 → -0.999983, amplitude 2.000000 → 1.999966, against peak to peak 2.000000 → 2.358701 -- the extremes keep the overshoot, and that separation is the measurement. On the probe's 1 Hz ±1 sine: top 0.957988 (it was 0.636620, the half-wave mean), amplitude 1.915983, and peak to peak still exactly 2; [stats-probe] reads top mean=0.957905 sd=0.0014 lo=0.951967 hi=0.962348 over a scrolling window, which is the stability the value band buys -- the answer does not follow the window's alignment. Five things this rests on, each of which breaks something silently:

  • Nothing may be read raw for it. The histogram is the same fetched bucket list, walked a second time inside measure_scan_buckets(); a histogram of the window's samples would be a second O(window) walk, which is exactly what the pyramid path exists to avoid. measure cost: is the guard and its raw does not move: the probe's pane reads covered=59684972 buckets=932446 raw=47640 (0.1%) against the baseline's covered=61183221 buckets=955856 raw=47402 (0.1%) -- the same raw, 0.44 → 0.50 ms a pass (and --noise 0.1, 2.1% both sides, 0.52 → 0.65 ms).
  • It is gated, and the gate is what keeps an unrelated pane free. measure_scan_buckets()'s flat_hist comes from App::MeasureWindow::flat_wanted, set by the first loop over the pane and only for one of the three types. A pane of means and extremes pays nothing.
  • The bin range is the buckets', not the window's. The window's partial edge buckets and the samples produced since acquire_all() are read raw and are not in the list, so measure_scan_buckets() tracks its own blo/bhi over the non-empty buckets only -- an all-non-finite bucket is an empty aggregate whose lo/hi say nothing, and taking them would put a bucket at some arbitrary value. A degenerate range (bhi == blo) leaves folded at 0, which is the caller's "no histogram at all".
  • The window's peak-to-peak is what the band is a share of, and the split is sized exactly as measure_set_level() sizes a level, so "the top" and "the level a period is counted about" are placed by one rule. The band is a value band, which is why the reading does not move when the window does.
  • The fallback is the split, and it is what a filtered or sub-bucket window reads. hf_fc > 0 disables the pyramid (above) and a window shorter than a bucket has no bucket to fold, so top/bottom fall back to the mean of each side of the split -- and amplitude to the window's own peak to peak, which is the same number on a waveform with nothing to exclude. There is no flat level in that mode to subtract, so amplitude == top - bottom is not an invariant to lean on there. Measured on the rippled square at hf_fc=100: top=0.973824 bottom=-0.991731 amplitude=2.354173 peak to peak=2.354173, buckets=0 raw=200.0% -- two raw walks and no density estimate, which is the honest reading of a window the pyramid cannot summarize.

• Above/Below/Between are the means over a side of a share, and they are the readings Top/Bottom gave before a flat level replaced them. That change kept the split's two means as its internal fallback for a window the pyramid cannot summarize (above); these three types are that fallback, exposed and named, which is what lets the two different questions about one split be asked side by side -- on a unit sine above 50 reads 0.636620 (2/pi) where top reads 0.957988, and on the mock's band-limited square above 50 reads 0.986496 where the plateau is 1.000000 and top 0.999983: the edge's ring is inside this mean and not inside that level. Five rules, and the first two are what a reader is most likely to assume wrongly:

  • A band is not a wider split, and needs no walk of its own. Above/Below are MeasureLevel::sum[1]/count[1] and sum[0]/count[0] at the measurement's own share. Between is the difference of two levels' upper sides -- sum[1](lo) - sum[1](hi) over count[1](lo) - count[1](hi), i.e. the samples above the lower edge and not above the upper one -- and no single level's two sums can bound a band, because a band's two edges are two thresholds. Both accumulators come out of one pass two over one window (the pane registers one level per edge, w.pct, and both are deduplicated into it), so the difference is exact rather than approximate. v == level is the below side, as everywhere, so the band is (lo, hi].
  • It rides the bucket fast path unchanged. A bucket wholly inside the band is strictly above the lower level and strictly below the upper one, so both accumulators fold it whole; only the buckets that straddle either edge are opened, exactly as for one level. measure cost: is the guard: adding all three types to a pane of every type left raw at 0.1% of covered (113966 of 240719031 with the ten types, 134978 of 255372033 with the thirteen), and a band is two entries of the level list, not a second pass.
  • measure_level_pcts() is the only place the shares are read, and it clamps both to [0,100] and puts Between's pair in order: "between 80 % and 20 %" is the band between 20 % and 80 %. Normalized rather than refused because the shares name two edges, and the band does not depend on which the user typed first; the pane's dedup key and both accumulators are keyed on the ordered pair, so one band is one walk however it was written.
  • Nothing to measure reads --, exactly as for every other type: a side holding no sample (a share at 100 %, whose level is the window's own maximum) and an interior holding none are both invalid/-- and never 0. The ideal --square-harmonics 0 square is where the second shows -- every sample is exactly +/-1, so the band (0, 0.5] holds nothing -- against the same band on the band-limited square, which holds the edges' transitions (between 50 75 = 0.302744, between 25 75 = 0.000000: the symmetric band's two halves cancel, the transition being odd about its own midpoint).
  • The reading is a mean over the window's samples, so its sixth decimal follows the window's boundary. A live window is closed at both ends, so its oldest and newest sample share a phase and the mean carries one sample that is not in an open interval -- about one unit in the sixth decimal out of 2.6 M samples. Measured over one fast-mock run, [measure-trace]: above 50 on the sine printed 0.636620 on 48 lines and 0.636621 on 2, below 50 -0.636620 on 37 and -0.636619 on 18, above 50 on the square 0.986496 on 32 and 0.986495 on 18, and between 50 75 on the ramp 0.250000/0.250001/0.249999. The old Top/Bottom spread the same way -- it is a property of a mean over a finite window, not of these types, and a reading that has to be exact to six figures is read off [bandavg-probe]'s nine. Measured on the fast mock, 2 s window, the mock's default 30-harmonic square, [bandavg-probe] (nine significant digits, then what the pane prints): above 50 on the sine 0.636620262 -> 636.62 mV; below 50 on the sine -0.636619283; between 50 75 on the sine 0.255873597 -- the analytic 3(2-sqrt 3)/pi = 0.255873, which is neither the window's mean (0) nor the band's edge (0.5); above 50 on the square 0.986494746 / 0.986495505 (one or two samples at the window's end); between 50 75 on the ramp 0.250000385 and above 50 0.500000 -- a uniform density makes every band's mean that band's own centre, so a quarter and a half of the range are the shape's prediction and not a coincidence; above 100 on the ramp --; and the ideal square's between 50 75 --. Editing the sine's band through the dialog to 50/90 takes it to 0.431361697 (0.8 / (2 asin 0.8)), which is the check that the second share is the one the engine reads.

• MeasureOptions::noise_pct is a percentage, clamped in measure_set_level() to half the room to the nearer extreme, so a band can never invert its own thresholds. It moves the edge decision only: the sums still split at the bare level, which is the fallback Top/Bottom read on a window with no bucket histogram -- a band that moved that split would change what a filtered or sub-bucket window reports, for no gain to a crossing count.

• measure_accumulate loops samples outer, levels inner, and that is not backwards. The filter holds one state for the whole walk, so levels-outer would step it once per level per sample and hand every level but the last a different filter state.

• A non-zero hf_fc disables both bucket fast paths, with the single if (!filtered && reader.bucket_range(...)). An empty buckets makes bulk_from == bulk_to == n and leaves lo_search / hi_search at 0, so pin() becomes a no-op -- and that is the right answer, because a bucket is an aggregate of unfiltered samples and cannot be mixed into a filtered walk in either direction.

• A crossing is counted with hysteresis, and the band moves the edge decision only. MeasureLevel::hysteresis is a Schmitt trigger about level: an edge is counted once the signal has passed level + h going up or level - h going down, and a sample inside the band decides nothing. Without it period, frequency and pulse duration are unusable on anything real -- the mid level is exactly where a waveform moves fastest, so an interferer faster than the fundamental recrosses a single level several times per true crossing. Measured on a 5 Hz sine with a 500 Hz tone at a tenth of the amplitude (tools/mock_udpstreamer.py --noise 0.1): 9 true crossings counted as 79, period=0.025622 freq=39.029 for a true 0.2 s / 5 Hz, against period=0.200188 with the band. Four rules, each of which was wrong first:

  • measure_set_level() is the only place the band is sized, and src/ui.cpp calls it rather than assigning .level itself: two expressions for the width is how the raw walk and the bucket path end up disagreeing about where a crossing was. h = min(0.10 * (hi - lo), 0.5 * min(hi - level, level - lo)). The 0.10 is measured -- 5 % reads a 0.1 interferer and fails at 0.2 (period=0.075), 10 % holds to 0.3 and to white noise at sigma=0.05.
  • The clamp is load-bearing. PulseDuration lets the user put the split near an extreme, and a fixed tenth of the span there puts the upper threshold above every sample: the walk never decides that side and the measurement reads nothing at all, which is worse than reading something noisy. On a ringing 3.3 V logic signal at a 90 % split: enters=0 with a fixed band, enters=4 period=0.200000 with the clamp.
  • The sum/count split stays at the bare level. Four things read it and none may be moved by a band: a crossing's side is decided by the sample's own value against the level (v == level is the below side, as it always was), Top/Bottom fall back to the mean of each side on a window with no bucket histogram, and Above/Below/Between read it on every window -- their whole definition is that split, so a band folded into it would move a documented exact reading (2/pi on a sine) for no gain to any crossing count. The band is about deciding a crossing, not about where a sample lives.
  • h = 0 must reduce exactly to the single-level walk, including the v == level skip, so the band is a parameter of the engine rather than a mode of it. The cost is the noise's, not the band's: the clean fast mock reads raw=119412 (0.1%) of 215 M at 1.41 ms a recompute and with the 0.1 interferer raw=2.0% at 1.96 ms. A /tmp/check_measure.cpp-style check is how the walk itself was verified (g++ -std=c++20 -O2 -I src ... src/measure.cpp): over a clean sine, sines with 0.1 and 0.3 interferers, 10 %/50 % duty squares and a 90 % threshold, the bucket walk must equal the raw walk exactly -- same enters, same count[], identical first_enter/last_enter to 0.0 -- which is the assertion that the fast path is a fast path and not an approximation. The app's reading is [measures-probe] against --noise 0.1.

• A statistic's observation is one recompute, not one sample. The pane already refreshes on a cadence (at most one pass per 100 ms while the window scrolls, exactly once when it settles), and the accumulator is fed from that pass's result — so n counts windows measured. Feeding it per sample would make n a sample count and the mean a mean of samples, which is not what any of these measurements measure: a frequency over a window is not a property of a sample. The reading that names this defect is [stats-probe]'s n in the thousands rather than the tens.

• The accumulator is Welford's, and the reason is the same one the time ring is f64 for. A measurement's values are routinely large with a small spread — an absolute stream time is the obvious case — and Σx² - (Σx)²/n cancels to noise there. The standalone check is that the spread of {2,4,4,4,5,5,7,9} survives a +1e9 offset unchanged.

• What resets the statistics is "what is being measured changed", not "what is in view changed". An explicit Reset, an add/edit/delete (all three funnel through rebuild_measures_pane()), a CONFIG, and a change of display window each change the question, so the history is cleared. A pause, a resume, a trigger latch, a pan and a zoom do not: on a live view they change which samples answer it, which is precisely what averaging over many windows is for -- and on a pinned one they change nothing, not even a value (the window invariant above), so there is nothing for the history to see either way. The rule is about the accumulator only; it is not a claim that a gesture leaves a value alone, which is a property of the window a value is computed over and is the other half of this change. The pane-wide toggle does not reset either — it only shows and hides labels that are always created and always fed, so reading the plot for a moment does not cost the history.