fix(data): draw the partial edge buckets at LOD levels, correct the pyramid's memory and precondition docs

Findings from the review of the sliding-window change:

- A level >= 1 gather emitted only complete, wholly-retained buckets, so the
  geometry stopped at the last bucket boundary inside the range: the newest
  samples of a live window -- up to `span - 1` of them, about a pixel at the
  level level_for() picks -- were not drawn, on the right edge, which is the
  part of a scrolling trace that gets watched, and the oldest retained samples
  fell outside the geometry too (the level-1 gather of a sliding window started
  51 samples in). The two partial edge buckets are now gathered as raw samples
  through copy_raw() and bracketed around the envelope -- the same exactness
  split reduce/sum/sumsq already use -- so the emitted geometry spans the
  requested range exactly. The pyramid still stores no partial bucket, so append
  stays amortised O(1); at 1 MS/s the demo stages 511 extra sub-pixel segments
  and still draws at 1.05 ms/frame, and the screenshot's trace now reaches one
  column further at each end.
- `drop_before`'s doc claimed a no-op "for an empty pyramid": false, and
  backwards -- the restart branch is exactly what an empty pyramid hits, which is
  what StreamSeries::acquire() depends on when the ring saturated before the
  first acquire. The precondition is `absolute_sample <= window_base()`, and the
  restart is documented as what it is.
- The pyramid's memory figure was 3x optimistic: an entry is 24 bytes (MinMax
  plus the exact sum/sumsq doubles), so level 0 costs 0.375n bytes and the whole
  pyramid ~0.75n -- about 19% of 4-byte raw samples, not the 6% a min/max-only
  index would cost. Corrected in the header, spec 4.2 and 4.5, with the trade
  stated (the doubles buy O(log n) sum/mean/RMS/integral from whole buckets),
  and the plan's Task 3 sketch no longer describes per-level prefix sums, a
  level() span or an unaligned-end widening that does not exist.

Tests: `gather_at_a_lod_level_reaches_both_ends_of_the_window` pins that the
first and last emitted points are the oldest and newest retained samples -- with
spike values no bucket envelope can produce -- at level 1, at the level a dense
view picks, and for a static series whose end is unaligned. It fails against the
previous commit (first point at the first whole bucket), and so does the sliding
gather mapping check, which now accounts for the raw edge pairs.
This commit is contained in:
Martino Ferrari committed 2026-09-28 00:19:59 +02:00
1 parent c8a97e4a1b
commit fb2c2f5e4d
6 files changed
+199 -56

No files matched your search

@@ -718,12 +718,19 @@ class MinMaxPyramid {
public:
void reserve(u64 n);
void append(f32 v);
void drop_before(u64 absolute_sample); // slide the retained window to an absolute sample
void clear();
u64 size() const noexcept { return size_; }
u32 level_count() const noexcept { return u32(levels_.size()); }
std::span<const MinMax> level(u32 l) const;
u64 size() const noexcept; // retained-window samples
u64 window_base() const noexcept; // absolute index of window index 0
u64 window_end() const noexcept; // absolute index one past the last append
u32 level_count() const noexcept;
u64 level_base(u32 l) const noexcept; // absolute level-l bucket index of entry 0
u64 level_size(u32 l) const noexcept;
MinMax bucket(u32 l, u64 i) const; // absolute level-l bucket i
// Whole-bucket only: [begin, end) is intersected with whole level-0 buckets and
// partial edges are excluded, never approximated. An empty result has span()==0.
// Window coordinates: the grid is absolute, so the intersection runs through
// window_base().
Range reduce_buckets(u64 begin, u64 end) const;
f64 sum_buckets(u64 begin, u64 end) const;
f64 sumsq_buckets(u64 begin, u64 end) const;
@@ -731,23 +738,33 @@ class MinMaxPyramid {
u64 snap(f64 x, f64 x0, f64 dx) const;
private:
u64 size_ = 0;
std::vector<std::vector<MinMax>> levels_; // levels_[0] = kBucket granularity
std::vector<std::vector<f64>> sums_, sumsqs_; // inclusive prefix sums per level
// One bucket: its extremes plus its own exact sum and sum of squares. The two
// doubles take a bucket from 8 bytes to 24 -- level 0 costs 0.375n bytes, the
// whole pyramid ≈0.75n, ≈19% of 4-byte raw samples -- and are what let
// mean/RMS/integral come from whole buckets. Absolute prefix sums cannot: they
// grow without bound and cannot survive a sliding front.
struct Entry { MinMax mm; f64 sum = 0.0; f64 sumsq = 0.0; };
struct Level { std::deque<Entry> entries; u64 base = 0; }; // absolute bucket space
u64 count_ = 0; // samples ever appended
u64 base_ = 0; // absolute index of the window's oldest sample
u32 live_levels_ = 1;
std::vector<Level> levels_{1};
Entry pending_; // the in-progress level-0 bucket
};
} // namespace plot::data
```
Implementation rules for `src/data/minmax_pyramid.cpp`:
- `append(v)`: write/merge into the last bucket of `levels_[0]`, then for each level `l`, if level `l` gained a complete pair, merge the last two entries of level `l` into a new entry of level `l+1`; otherwise stop early. Amortised O(1). The same walk maintains `sums_`/`sumsqs_` (inclusive prefix sums per level).
- `reduce_buckets(begin, end)`: `if (end <= begin) return {0,0};` intersect `[begin, end)` with whole level-0 buckets, then cover the result by descending to the coarsest level whose buckets lie entirely inside it, finishing the remainder at level 0. It must NEVER scan raw samples — it stores none — and a partial edge bucket is excluded, not approximated. `sum_buckets`/`sumsq_buckets` use the same decomposition over `sums_`/`sumsqs_`.
- `append(v)`: fold the sample into `pending_`, the in-progress level-0 bucket; when that bucket completes, push it as an entry and merge upward while `kBucket << l` divides the sample count, stopping at the first level that gained no entry. Amortised O(1) -- `1/kBucket + 1/(2·kBucket) + … = 2/kBucket` merges per sample, independent of the number of live levels -- and no entry is ever rewritten, which is what lets a window slide over the pyramid instead of forcing a rebuild.
- `drop_before(u)`: set the window base and discard, per level, the leading entries whose span starts before `u`; level 0 also drops its in-progress bucket. Memory then tracks the window, not the samples ever pushed.
- `reduce_buckets(begin, end)`: `if (end <= begin) return {0,0};` intersect `[begin, end)` with whole level-0 buckets, then cover the result by descending to the coarsest level whose buckets lie entirely inside it, finishing the remainder at level 0. It must NEVER scan raw samples — it stores none — and a partial edge bucket is excluded, not approximated. `sum_buckets`/`sumsq_buckets` use the same decomposition, folding each covered entry's own `sum`/`sumsq`.
- `Series::reduce`/`sum`/`sumsq` in `src/data/series.cpp` are the exact entry points every other module uses: they take the bucket-aligned interior from `reduce_buckets`/`sum_buckets`/`sumsq_buckets` and scan the at-most-two partial edge buckets obtained via `copy_raw` (a stack buffer of at most 2*kBucket floats, so no allocation on the query path).
- `level_for(n, spp)`: `if (spp <= f64(kBucket)) return 0;` else `u32 l = u32(std::floor(std::log2(spp / f64(kBucket))));` clamp to `level_count() - 1`, and additionally clamp so that `level(l).size() >= 2`.
- `level_for(n, spp)`: `if (spp <= f64(kBucket)) return 0;` else `u32 l = u32(std::floor(std::log2(spp / f64(kBucket))));` clamp to `level_count() - 1`, and additionally clamp so that `level_size(l) >= 2`.
- `snap(x, x0, dx)`: `if (dx <= 0.0 || size_ == 0) return 0;` compute `i = llround((x - x0) / dx)` and clamp to `[0, size_ - 1]`.
`include/plot/data/ring_buffer.hpp`: `RingBuffer<T>` with power-of-two capacity, atomic producer index, `push(T)` overwriting the oldest entry when full, `size()`, `operator[](u64)` indexing oldest→newest, `clear()`.
`include/plot/data/series.hpp`: the abstract `Series` from spec §6. `virtual f64 x0() const` and `virtual f64 dx() const` expose the uniform x grid (measurement and `snap` need them). `virtual void gather(u32 level, u64 begin, u64 end, std::vector<f32>& out_xy) const = 0;` is documented as "appends interleaved (x, y) pairs in data coordinates for bucket-aligned level `level`; level 0 emits raw samples". `reduce`/`sum`/`sumsq` are non-virtual, implemented once in `src/data/series.cpp` over the protected pure virtual `copy_raw(u64 begin, u64 end, f32* out) const`, which copies `end - begin` raw y values and handles any wraparound.
`include/plot/data/series.hpp`: the abstract `Series` from spec §6. `virtual f64 x0() const` and `virtual f64 dx() const` expose the uniform x grid (measurement and `snap` need them). `virtual void gather(u32 level, u64 begin, u64 end, std::vector<f32>& out_xy) const = 0;` is documented as "level 0 emits the raw samples of [begin, end); level >= 1 emits one (min, max) envelope pair per complete retained bucket, bracketed by the partial edge buckets drawn as raw samples", so the geometry spans the requested range exactly without the pyramid storing a partial bucket. `reduce`/`sum`/`sumsq` are non-virtual, implemented once in `src/data/series.cpp` over the protected pure virtual `copy_raw(u64 begin, u64 end, f32* out) const`, which copies `end - begin` raw y values and handles any wraparound.
`ArraySeries`: holds `std::vector<f32> y_`, `f64 x0_`, `f64 dx_`, a `MinMaxPyramid`, and cached `Range y_bounds_`/`Range x_bounds_`. Constructor and `reset()` fill the pyramid and bounds; `commit()` rebuilds them (for when the caller mutated the vector out of band — document that the vector is owned and mutated only via `reset`). `copy_raw` is a bounds-checked `std::copy` over `y_`.