diff --git a/README.md b/README.md index 1e3465d..2007139 100644 --- a/README.md +++ b/README.md @@ -1638,6 +1638,35 @@ Measured on the same stream and window: tick mean **1.39 ms** with the pane hold measurements against **0.73 ms** without (595 frames against 610), the pass itself 11 ms every 400 ms — and the 82 ms worst tick is the *control's* too, i.e. the publish stall, not this. +**...and the same pane costs what the *window* costs, which a big time scale buys.** The three +fixes above bound *how often* and *how much per sample*; what is left is the arithmetic of the +scale: a pass is priced per measured **channel** (one fetch of the window's level-0 buckets, one +statistics scan) and per **distinct share** in the pane (one level accumulator, walked over those +buckets), and the window's own sample count multiplies both. Measured with the new +`[levels-probe]` sweep on the mock at a **15 s** window and `--buffer-mb 4000` (so `time_stride` +stays 1 and the window holds 11.8 M samples, 184 k buckets), four signals, one row, on llvmpipe +— `measure cost:`'s `each` per recompute, and the tick's mean: + +| pane | `each` | `raw` | tick mean | +|---|---|---|---| +| empty | — | — | 1.34 ms | +| 1 measurement, 1 level, 1 channel | 6.58 ms | 0.1 % | 1.99 ms | +| 4 measurements, 4 shares, 1 channel | 10.28 ms | 0.4 % | 2.38 ms | +| 16 measurements, 16 shares, 1 channel | 24.55 ms | 1.2 % | 3.86 ms | +| 4 measurements, 1 share each, **4 channels** | 35.69 ms | 2.6 % | 4.58 ms | + +So a machine's ten measured signals at a 15 s window is tens of milliseconds a pass, several +times a second, on the **tick** — the pane is a *frame* cost and not an ingest one (`gaps` and +`packets` stay clean even though each reader holds the model mutex, which is what "the plot is +slow with measurements on" actually is). One of those terms was a pure call-overhead defect: +pass two folds each bucket into each level through `measure_accumulate_bucket()`, the innermost +loop of a walk whose iteration count is the window's buckets *times* the pane's levels, and as an +out-of-line function in its own translation unit it cost **30 % of the pass** — sixteen shares on +one channel read `24.55 ms` a recompute with every value in `measurements:` unchanged at +**17.25 ms** once it was moved into `measure.hpp` as `inline`. The fetch's three out-of-line +pyramid accessors per bucket (3.3 ms of an 11.8 M-sample window's 11.6 ms pass) are the next +lever, and `raw` is the last one and belongs to the signal rather than to the pane. + **"Show on plot" draws full-width rulers, and they must not blink.** A level measurement draws an `h_ruler` across the whole plot and a time one a full-height cursor, so a pane holding four measurements can put six lines over the data — that is the feature, and the lines move with the diff --git a/docs/claude/invariants-measurements.md b/docs/claude/invariants-measurements.md index 6dc1947..3e39764 100644 --- a/docs/claude/invariants-measurements.md +++ b/docs/claude/invariants-measurements.md @@ -118,6 +118,47 @@ Part of [CLAUDE.md](../../CLAUDE.md). These are rules that cost a defect to lear the pyramid for nothing. `measure cost:`'s `covered`/`buckets`/`raw` is the guard, and it needs the fast mock to say anything. +- **A pass costs the window's buckets times the pane's *measured channels* and its *distinct + levels*, and the window's own size multiplies all of it.** The cadence above bounds *how + often*; this is *how much*, and it is the term a big time scale buys. Measured on the mock at + the user's scale -- a **15 s** window, `--buffer-mb 4000` (so `time_stride` stays 1 and the + window is 11.8 M samples, 184 k level-0 buckets), four signals, one row -- with + `[levels-probe]` (see [verification.md](verification.md)), `measure cost:` reading + `recomputes`/`each`/`covered`/`buckets`/`raw`: + + | pane | `each` | `raw` | tick mean | + |---|---|---|---| + | empty | -- | -- | 1.34 ms | + | 1 measurement, 1 level | 6.58 ms | 0.1 % | 1.99 ms | + | 4 measurements, 4 shares on **one** channel | 10.28 ms | 0.4 % | 2.38 ms | + | 16 measurements, 16 shares on one channel | 24.55 ms | 1.2 % | 3.86 ms | + | 4 measurements, 1 share each on **four** channels | 35.69 ms | 2.6 % | 4.58 ms | + + So a pane is priced per measured **channel** (one `MeasureWindowReader`, one fetch, one scan) + and per **distinct share** (one accumulator), and the window's samples are the multiplier on + both: the same pane over a 2 M-sample window is the 1.2--1.6 ms a recompute this pane's + budget was fixed at, and a machine's ten noisy channels at a 15 s window is **tens of ms a + pass**, several times a second. That lands on the **tick** -- `update_measures()` runs inside + it -- so `frame cost:`'s tick mean is where a pane's cost shows (1.34 -> 4.58 ms above), and + it is *not* an ingest problem: `gaps=`/`packets=` stay exactly clean even though each + `MeasureWindowReader` holds `model->mutex()` for its whole life, because a socket buffer + absorbs a few tens of ms of producer. Read the tick mean and `each` before blaming the + producer. + Inside a pass the terms are, instrumented per window at that scale: the **fetch** 3.3 ms + (three out-of-line `MinMaxPyramid` accessors and a 48-byte `MeasureBucket` per bucket), + `measure_scan_buckets()` 0.8 ms, and **pass two** 7.5 ms -- dominated by the per-(bucket, + level) fold. That fold is why `measure_accumulate_bucket()` is `inline` in `measure.hpp`: it + is the innermost loop of a walk whose iteration count is the window's buckets *times* the + pane's levels, and as an out-of-line call in its own translation unit it cost **30 % of the + pass** (sixteen shares on one channel: `24.55 -> 17.25 ms` a recompute). A caller adding a + per-bucket type keeps it in the header, and a caller adding a bulk one may not let it grow a + call per bucket either. + The third term is `raw`, and it is the **signal's**: a level that falls inside a bucket's + `[lo, hi]` has to be walked sample by sample (the fold cannot split a bucket), so a signal + whose local range covers the level everywhere -- white noise, a noisy machine channel -- + degrades the pass to O(window) *per level*: measured `raw=4.6 %` on the mock's `noise` channel + against `0.1 %` on its `sine` at the same window. It is the field to read before blaming the + pane for a slow run, and no cheaper walk fixes it. - **`Top`/`Bottom`/`Amplitude` read a *flat* level, and the density that finds it is the buckets', not the samples'.** A band-limited edge overshoots and rings -- the mock's `square` at 30 harmonics reaches ±1.179 where its plateau is ±1 -- so a measurement that reads the extremes, or diff --git a/docs/claude/probe-readings.md b/docs/claude/probe-readings.md index 79fef9b..e5998f2 100644 --- a/docs/claude/probe-readings.md +++ b/docs/claude/probe-readings.md @@ -714,6 +714,33 @@ Two lines of the self-check output are assertions in disguise: reported `gaps=73` with a pause against 0 without, which is what made the stall worth measuring; it has not reproduced and did not scale with the publish length — 79 gaps with the pyramid build removed against 73 with it — so it was incidental load on one run. Do not quote it as a cost.) +- `[levels-probe] N measurement(s) over K channel(s), shares='...' -> W window(s), L level + walk(s)` — then the exit dump's `measure cost:` and `frame cost:` beside it. This is how a + pane's cost is read, and the three fields are the three factors: **`W` is the pane's price + for the window**, one fetch and one scan per *measured channel*; **`L` is its price for the + pane's shares**, one accumulator per distinct one, walked over the window's buckets; and the + window's samples are the multiplier on both, so a sweep at one scale says nothing about + another. Measured at the user's scale (15 s window, `--buffer-mb 4000` -> `stride=1`, 11.8 M + samples in the window = 184 k buckets, four signals, mock at ~0.5 MS/s per signal, llvmpipe), + with `UDPSCOPE_PROBE_LEVELS_CHANNELS` set: + * `50`, 1 channel -> `1 window, 1 level`, `measure cost: 226 recomputes, 6.58 ms each`, `raw=0.1%`, + `tick 1.99 ms` against an empty pane's **1.34**; + * `5,10,15,20`, 1 channel -> `4 levels`, **10.28 ms** each, `raw=0.4%`, `tick 2.38 ms`; + * the sixteen shares `5..80`, 1 channel -> `16 levels`, **24.55 ms** each, `raw=1.2%`, `tick 3.86 ms`; + * `5,10,15,20` over **4 channels** -> 4 windows and 4 levels, **35.69 ms** each, `raw=2.6%`, + `tick 4.58 ms`, and still `gaps=0 packets=30580` — the pane is a *tick* cost and not an + ingest one, which is the thing to say before blaming the producer. + `measure cost:`'s `each` is the pass and `covered`/`recomputes` is the mean window it ran + over (11.8 M samples here), so `each` is only comparable between runs at the same window and + the same pane; `raw` is the *signal's* share (the mock's `noise` channel reads `4.6%` against + its `sine`'s `0.1%` at the same window, because a level inside a bucket's own range has to be + walked sample by sample). A pass that reads **tens of ms** on a 15 s window with a handful of + measurements is the curve in + [invariants-measurements.md](invariants-measurements.md) and not a defect, and the same pane + at a 2 s window reads the 1.2--1.6 ms the pane's budget was fixed at. The guard for the walk + itself is the exit dump's `frame cost:`: sixteen shares on one channel read `24.55 ms` a pass + and `3.86 ms` of tick mean before `measure_accumulate_bucket()` was made `inline`, and + **17.25 ms** a pass after it, with every value in `measurements:` unchanged. - `trace ink: N paints checked, C collapsed, S staircase, H short, worst cover P%, worst drawn-vs-quoted D px over R raw paints` plus a `[ink] paint N mode=.. axis=[..] n=.. idx=[..) spp=.. level=.. pts=.. kept=.. diff --git a/docs/claude/verification.md b/docs/claude/verification.md index c179a81..1823012 100644 --- a/docs/claude/verification.md +++ b/docs/claude/verification.md @@ -76,6 +76,22 @@ precedent, not an exception. with, so a synchronous sequence would read every position one action stale. It clicks by emitting `::clicked`, never `gtk_widget_activate()`, which holds a `GtkButton` for its 250 ms `ACTIVATE_TIMEOUT` and then emits into a button the rebuild has already destroyed. +- `[levels-probe] N measurement(s) over K channel(s), shares='...' -> W window(s), L level + walk(s)` (stderr, `UDPSCOPE_PROBE_LEVELS_MS`) — **prices a measurement pane**, which is what a + report of "it is slow with many measurements" has to become before it is a number. It adds one + `above` per share in `UDPSCOPE_PROBE_LEVELS` (comma separated, default `50`), spread in turn + over the first `UDPSCOPE_PROBE_LEVELS_CHANNELS` channels (default 1), and reports the two + things a count does not name: how many **windows** (one `MeasureWindowReader`, one fetch, one + scan per measured channel) and how many **level walks** (one accumulator per *distinct* share) + the pass then had to take. `50,50,50,50` on one channel is four pane rows and **one** walk; + `5,10,15,20` is four rows and **four**; the same four shares over four channels is four + windows *and* four walks. Read the `measure cost:` line it leaves in the exit dump and the + sweep in [invariants-measurements.md](invariants-measurements.md) — and read it against a run + with `UDPSCOPE_PROBE_LEVELS_MS` unset, because the pane's cost is a *difference* between two + tick means. **Run it at the scale being asked about**: the cost is the window's, so a 2 s + window prices a 15 s one at nothing (four measurements over four channels read `35.69 ms` a + recompute and a 4.58 ms tick mean at a 15 s window against `6.58 ms`/1.34 ms for one window at + the same window). - `[measure-probe] N measurements on 'X'` (stderr, `UDPSCOPE_PROBE_MEASURE_MS`) — adds one measurement of each statistical type (average, top, peak to peak, amplitude) on the first *channel* and forces a recompute. It must name a **channel**, not `signals().front()`: the diff --git a/src/measure.cpp b/src/measure.cpp index 9ef7f4e..0d9bfae 100644 --- a/src/measure.cpp +++ b/src/measure.cpp @@ -14,20 +14,6 @@ namespace { /// in. Shared by the raw walk and the bucket walk so the two cannot disagree about what an edge /// records -- an interval's width is the distance from the edge that entered it to the edge that /// left it, which is the pairing the pulse duration averages. -void enter_side(MeasureLevel& acc, std::size_t e, double t) { - const std::size_t other = 1 - e; - if (acc.enters[e] == 0) acc.first_enter[e] = t; - acc.last_enter[e] = t; - ++acc.enters[e]; - if (acc.open[other]) { - acc.width_sum[other] += t - acc.open_at[other]; - ++acc.widths[other]; - acc.open[other] = false; - } - acc.open_at[e] = t; - acc.open[e] = true; -} - // The hysteresis band and its rationale moved to `MeasureOptions::noise_pct`: it is a per-measurement // setting now, and a second copy of the number here would be a second thing to keep in step. constexpr double kPi = 3.14159265358979323846; @@ -240,51 +226,6 @@ void measure_accumulate(const MeasureChunk& chunk, } } -bool measure_accumulate_bucket(const MeasureBucket& k, - const std::function& time_of, - MeasureLevel& acc) { - if (k.count == 0) return true; - // A non-finite sample in the bucket would make the edge's position a guess: the raw walk takes - // the first *finite* sample of the new side, and an aggregate cannot say which that is. - if (k.count != k.span) return false; - // The *sums* need only the bare level: strictly on one side of it, so every sample decides the - // same bin and none is *at* it, which is what makes the fold exact rather than an approximation. - const bool above = k.lo > acc.level; - const bool below = k.hi < acc.level; - if (!above && !below) return false; - const std::size_t e = above ? 1u : 0u; - // The band is deliberately *not* folded into the test above. Tightening the whole condition to - // `level +/- h` refuses every bucket lying inside the band -- a tenth of the amplitude either side - // of the mid level is where a waveform moves fastest, so those are exactly the buckets a crossing - // passes through -- which hands the raw walk a share of the window rather than a bucket or two per - // crossing, and breaks the "`raw` stays at a fraction of a percent of `covered`" invariant. - // Instead: a bucket on one side of the level can always be *binned*, and only a *side change* - // whose position is inside the bucket is refused. - const double top = acc.level + acc.hysteresis; - const double bot = acc.level - acc.hysteresis; - // Does the bucket reach past the band at all? One wholly inside it decides no side, exactly as a - // raw sample inside it decides none, so the walk state is left alone. - const bool reaches = above ? (k.hi > top) : (k.lo < bot); - const bool change = reaches && acc.have_side && e != (acc.side ? 1u : 0u); - // A bucket that *enters* a side while straddling its own threshold holds the crossing somewhere - // inside itself, and only the raw walk can say where. One wholly past the threshold holds it at - // the bucket's first sample, exactly, which is where the raw walk would have found it. - const bool fully_outside = above ? (k.lo > top) : (k.hi < bot); - // Nothing may be folded before this point: a refusal hands the bucket to `measure_accumulate()`, - // which adds the same samples again. Sums added above the refusal double-counted every opened - // bucket -- which read as `top` 0.614304 on a unit sine against the 2/pi a mean over a half then - // gave, and which is now the sums `Top`/`Bottom` fall back to on a window with no histogram, so - // a wrong one is a wrong answer waiting for that window. - if (change && !fully_outside) return false; - acc.sum[e] += k.sum; - acc.count[e] += k.count; - if (change) enter_side(acc, e, time_of(k.offset)); - if (reaches) { - acc.side = e != 0; - acc.have_side = true; - } - return true; -} MeasureFlatLevels measure_flat_levels(const MeasureStats& b, double pct) { MeasureFlatLevels out; @@ -375,7 +316,6 @@ int measure_level_pcts(MeasureType type, const MeasureOptions& options, double p } } - const char* measure_type_name(MeasureType type) { switch (type) { case MeasureType::Average: return "average"; @@ -598,5 +538,4 @@ MeasureResult compute_measure(const MeasureStats& b, const MeasureLevel* const l return out; } - } // namespace udpscope diff --git a/src/measure.hpp b/src/measure.hpp index ec72944..d31642a 100644 --- a/src/measure.hpp +++ b/src/measure.hpp @@ -349,6 +349,66 @@ bool measure_accumulate_bucket(const MeasureBucket& bucket, const std::function& time_of, MeasureLevel& level); +inline void enter_side(MeasureLevel& acc, std::size_t e, double t) { + const std::size_t other = 1 - e; + if (acc.enters[e] == 0) acc.first_enter[e] = t; + acc.last_enter[e] = t; + ++acc.enters[e]; + if (acc.open[other]) { + acc.width_sum[other] += t - acc.open_at[other]; + ++acc.widths[other]; + acc.open[other] = false; + } + acc.open_at[e] = t; + acc.open[e] = true; +} + +inline bool measure_accumulate_bucket(const MeasureBucket& k, + const std::function& time_of, + MeasureLevel& acc) { + if (k.count == 0) return true; + // A non-finite sample in the bucket would make the edge's position a guess: the raw walk takes + // the first *finite* sample of the new side, and an aggregate cannot say which that is. + if (k.count != k.span) return false; + // The *sums* need only the bare level: strictly on one side of it, so every sample decides the + // same bin and none is *at* it, which is what makes the fold exact rather than an approximation. + const bool above = k.lo > acc.level; + const bool below = k.hi < acc.level; + if (!above && !below) return false; + const std::size_t e = above ? 1u : 0u; + // The band is deliberately *not* folded into the test above. Tightening the whole condition to + // `level +/- h` refuses every bucket lying inside the band -- a tenth of the amplitude either side + // of the mid level is where a waveform moves fastest, so those are exactly the buckets a crossing + // passes through -- which hands the raw walk a share of the window rather than a bucket or two per + // crossing, and breaks the "`raw` stays at a fraction of a percent of `covered`" invariant. + // Instead: a bucket on one side of the level can always be *binned*, and only a *side change* + // whose position is inside the bucket is refused. + const double top = acc.level + acc.hysteresis; + const double bot = acc.level - acc.hysteresis; + // Does the bucket reach past the band at all? One wholly inside it decides no side, exactly as a + // raw sample inside it decides none, so the walk state is left alone. + const bool reaches = above ? (k.hi > top) : (k.lo < bot); + const bool change = reaches && acc.have_side && e != (acc.side ? 1u : 0u); + // A bucket that *enters* a side while straddling its own threshold holds the crossing somewhere + // inside itself, and only the raw walk can say where. One wholly past the threshold holds it at + // the bucket's first sample, exactly, which is where the raw walk would have found it. + const bool fully_outside = above ? (k.lo > top) : (k.hi < bot); + // Nothing may be folded before this point: a refusal hands the bucket to `measure_accumulate()`, + // which adds the same samples again. Sums added above the refusal double-counted every opened + // bucket -- which read as `top` 0.614304 on a unit sine against the 2/pi a mean over a half then + // gave, and which is now the sums `Top`/`Bottom` fall back to on a window with no histogram, so + // a wrong one is a wrong answer waiting for that window. + if (change && !fully_outside) return false; + acc.sum[e] += k.sum; + acc.count[e] += k.count; + if (change) enter_side(acc, e, time_of(k.offset)); + if (reaches) { + acc.side = e != 0; + acc.have_side = true; + } + return true; +} + /// The levels a type reads, as shares of the window's own min..max, **lower first**, or 0 when it /// reads none (the pure statistics, answered from `MeasureStats` alone). `pct` must hold two: a /// level type fills `pct[0]` and answers 1, `Between` fills both and answers 2. diff --git a/src/ui.cpp b/src/ui.cpp index c20847c..1e27c1b 100644 --- a/src/ui.cpp +++ b/src/ui.cpp @@ -5831,6 +5831,9 @@ int run_ui(int argc, char** argv, const Config& cfg, ScopeModel& model, Acquisit if (const char* env = g_getenv("UDPSCOPE_PROBE_MEASURE_MS")) { g_timeout_add(static_cast(std::atoi(env)), measure_probe, &app); } + if (const char* env = g_getenv("UDPSCOPE_PROBE_LEVELS_MS")) { + g_timeout_add(static_cast(std::atoi(env)), levels_probe, &app); + } if (const char* env = g_getenv("UDPSCOPE_PROBE_MEASUREPIN_MS")) { g_timeout_add(static_cast(std::atoi(env)), measurepin_probe, &app); } diff --git a/src/ui_app.hpp b/src/ui_app.hpp index 55c898e..3cff1b9 100644 --- a/src/ui_app.hpp +++ b/src/ui_app.hpp @@ -1730,6 +1730,7 @@ gboolean mathwrap_probe(gpointer user); gboolean mathwrap_probe_wait(gpointer user); gboolean mathwrap_probe_report(gpointer user); gboolean measure_probe(gpointer user); +gboolean levels_probe(gpointer user); gboolean measurepin_probe(gpointer user); gboolean measurepin_held_wait(gpointer user); gboolean measurepin_pause_wait(gpointer user); diff --git a/src/ui_probe.cpp b/src/ui_probe.cpp index a2e6a41..276321d 100644 --- a/src/ui_probe.cpp +++ b/src/ui_probe.cpp @@ -1326,6 +1326,71 @@ gboolean measure_probe(gpointer user) { return G_SOURCE_REMOVE; } +/// `UDPSCOPE_PROBE_LEVELS_MS`: puts a pane of **N** measurements on screen -- one `above` per +/// share in `UDPSCOPE_PROBE_LEVELS` (comma separated, default `50`) -- spread in turn over the +/// first `UDPSCOPE_PROBE_LEVELS_CHANNELS` channels (default 1), and says how many *windows* and +/// *levels* the pass then had to walk. +/// +/// The pane's cost has three factors and a count alone names none of them: the window's samples +/// (one fetch and one scan per **measured channel**, whatever the pane holds), the number of +/// *distinct levels* in it (measurements that share a share share the walk), and the rows that +/// are only labels over the two. `50,50,50,50` is four rows and one walk; `5,10,15,20` is four +/// rows and four; and the same four shares on four signals is four windows *and* four walks. +/// This probe is the only way to vary the three apart, which is what a report of "it is slow +/// with many measurements" needs before it is a number. +gboolean levels_probe(gpointer user) { + App* a = static_cast(user); + if (a->window == nullptr) return G_SOURCE_REMOVE; + std::vector names; + { + std::lock_guard lock(a->model->mutex()); + for (const ChannelInfo& c : a->model->channels()) names.push_back(c.name); + } + if (names.empty()) return G_SOURCE_REMOVE; + const char* chan_env = g_getenv("UDPSCOPE_PROBE_LEVELS_CHANNELS"); + const long want_channels = chan_env != nullptr ? std::atol(chan_env) : 1; + const std::size_t channels = + want_channels > 0 ? std::min(static_cast(want_channels), + names.size()) + : 1u; + const char* list = g_getenv("UDPSCOPE_PROBE_LEVELS"); + const std::string shares = list != nullptr && list[0] != '\0' ? list : "50"; + std::size_t rows = 0; + for (std::size_t at = 0; at <= shares.size();) { + const std::size_t comma = shares.find(',', at); + const std::string field = + shares.substr(at, comma == std::string::npos ? std::string::npos : comma - at); + at = comma == std::string::npos ? shares.size() + 1u : comma + 1u; + char* end = nullptr; + const double pct = std::strtod(field.c_str(), &end); + if (end == field.c_str() || !(pct >= 0.0)) continue; + Measure m; + m.signal = names[rows % channels]; + m.type = MeasureType::Above; + m.options.threshold_pct = pct; + a->measures.push_back(std::move(m)); + ++rows; + } + rebuild_measures_pane(*a); + update_measures(*a, /*force=*/true, /*accumulate=*/true); + // What that pass walked: the level accumulators the pane deduplicated to, and the windows they + // hang off. Read without the model lock -- `measure_samples` is the *tick*'s own state, not the + // producer's. + std::size_t levels = 0; + std::size_t windows = 0; + for (const App::MeasureWindow& w : a->measure_samples) { + ++windows; + levels += w.pct.size(); + } + std::fprintf(stderr, + "[levels-probe] %zu measurement(s) over %zu channel(s), shares='%s' -> %zu " + "window(s), %zu level walk(s)\n", + rows, channels, shares.c_str(), windows, levels); + std::fflush(stderr); + g_timeout_add(250, measure_trace, a); + return G_SOURCE_REMOVE; +} + /// `UDPSCOPE_PROBE_MEASUREPIN_MS`: on a **still frame** the pane describes the frame and not the /// viewport, so a pan or a zoom changes no number -- and on a live view the numbers still follow /// their window.