martino/plotter
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

perf(data): absolute-space pyramid so acquire() is O(produced), not O(window)

Browse Source 
Frame cost was O(capacity x levels) once the ring saturated: acquire() saw the
window base move (it always moves once the ring wraps), cleared the pyramid and
re-appended all 1,048,576 retained samples, and append() walked every live level
with no early stop -- 13,647,936 bucket merges and 73.8 ms per frame at a
1M-sample window, scaling with the window, which is what capped the 1 MS/s demo
at 32-58 ms/frame while it was still filling.

Index the pyramid in absolute bucket space instead. Level l bucket k covers the
samples [k * (kBucket << l), (k + 1) * (kBucket << l)), so a bucket's span never
depends on where the window starts and the window can slide over the aggregate
instead of forcing it to be rebuilt. Each level keeps its complete buckets in a
deque plus the absolute base of its first entry; append() folds a sample into the
single in-progress level-0 bucket and, only when that bucket completes, cascades
upward, stopping at the first level that gains no new entry; a new drop_before()
slides the window by discarding leading buckets and remembers where the window
begins. acquire() is now: drop to the new base, then append exactly what was
produced -- 256 merges and 0.034 ms per frame for 8192 new samples, independent
of the window, and memory bounded by the window (32,767 entries after 50M samples
through a 1M ring, the same as after 10M). A producer that laps the consumer by
more than capacity() between two acquires restarts the window rather than
splicing a gap.

Interface: `level(l) -> std::span<const MinMax>` cannot survive a ring and is
replaced by level_base/level_size/bucket in absolute bucket coordinates, plus
window_base/window_end/drop_before. The *_buckets queries stay in window
coordinates (index 0 = oldest retained sample), so Series keeps its window
semantics for callers; its exactness split now aligns in absolute space
(window_base() + the ceil/floor) because the bucket grid is absolute. The sums
move from per-level prefix sums to per-bucket ones -- the only representation a
sliding front can keep exact -- and StreamSeries::y_bounds() becomes
reduce(0, size()), exact in O(log n + kBucket) per frame instead of a scan of the
window. The pyramid owns the published window now, so there is no second copy of
base/end to keep in step, and gather() emits only buckets that are both complete
and wholly retained.

Tests: test-only merge counters (mirroring PLOT_PYRAMID_COUNT_COVERS, absent from
every non-test build) pin an acquire proportional to production and independent
of capacity, an append cost amortised over 16 live levels, and bounded memory;
brute-force checks pin absolute bucket aggregates across slides, sliding-window
reduce/sum/sumsq over unaligned ranges, and the level-1 gather mapping. The demo
now draws 946 samples/px at 1.03 ms/frame (967 fps) where the saturated window
cost 54.96 ms/frame (18 fps).
This commit is contained in:
Martino Ferrari committed 2026-09-28 00:05:15 +02:00
1 parent 4b649a2783
commit 0ac5d47fa6
13 files changed
+769 -244
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

src/data/minmax_pyramid.cpp
+149 -114
View File
@@ -1,8 +1,10 @@
#include <plot/data/detail/cover_count.hpp>
#include <plot/data/detail/merge_count.hpp>
#include <plot/data/minmax_pyramid.hpp>
#include <algorithm>
#include <cmath>
#include <cstddef>
namespace plot::data {
namespace {
@@ -16,18 +18,19 @@ constexpr u64 buckets_for(u64 n, u32 l) {
return n / span + (n % span == 0 ? u64{0} : u64{1});
}
/// Ceiling division: first level-l bucket whose span starts at or after v.
constexpr u64 ceil_div(u64 v, u64 d) { return v / d + (v % d == 0 ? u64{0} : u64{1}); }
/// First level-0 bucket boundary at or after v.
constexpr u64 ceil_to_bucket(u64 v) {
return (v / kBucket + (v % kBucket == 0 ? u64{0} : u64{1})) * kBucket;
}
constexpr u64 ceil_to_bucket(u64 v) { return ceil_div(v, kBucket) * kBucket; }
/// Canonical, coarsest-first decomposition of the level-0 bucket span [b0, b1)
/// into pyramid entries, calling fn(level, index) for each.
/// into pyramid entries, calling fn(level, absolute_bucket_index) for each.
///
/// A level-l entry is emitted only when the whole level-l bucket exists (all
/// kBucket << l samples are present, so its aggregate is final). Callers
/// guarantee b1 <= ceil(size() / kBucket), so the only possible remainder is the
/// single trailing partial level-0 bucket, which is emitted at level 0.
/// guarantee b0 >= the first whole bucket of the window and b1 <= the number of
/// complete buckets, so every emission both exists and lies inside the window.
///
/// The level is chosen per *emission*, not once per pass: at each cursor the
/// largest level is taken whose bucket is aligned with the cursor, lies wholly
@@ -40,16 +43,18 @@ constexpr u64 ceil_to_bucket(u64 v) {
/// no coarser level is ever reachable from an odd cursor.
template <typename Fn>
void cover_buckets(const MinMaxPyramid& p, u64 b0, u64 b1, Fn&& fn) {
const u64 complete = p.size() / kBucket;
const u64 limit = std::min(b1, complete);
// One past the last complete level-0 bucket: levels sit on top of those, so
// this also bounds every coarser bucket the cursor may take.
const u64 limit = std::min(b1, p.level_base(0) + p.level_size(0));
const u32 top = p.level_count() - 1; // level_count() is at least 1
u64 b = b0;
while (b < b1) {
u32 li = top;
for (;;) {
const u64 lb = u64{1} << li;
if ((b % lb) == 0 && b + lb <= limit &&
(b / lb) < static_cast<u64>(p.level(li).size())) {
const u64 i = b >> li;
if ((b % lb) == 0 && b + lb <= limit && i >= p.level_base(li) &&
i - p.level_base(li) < p.level_size(li)) {
break;
}
if (li == 0) break;
@@ -69,7 +74,7 @@ Range fold_range(const MinMaxPyramid& p, u64 b0, u64 b1) {
Range r{};
bool first = true;
cover_buckets(p, b0, b1, [&](u32 l, u64 i) {
const MinMax m = p.level(l)[static_cast<usize>(i)];
const MinMax m = p.bucket(l, i);
if (first) {
r.min = f64(m.min);
r.max = f64(m.max);
@@ -84,140 +89,170 @@ Range fold_range(const MinMaxPyramid& p, u64 b0, u64 b1) {
} // namespace
void MinMaxPyramid::add_level() {
levels_.emplace_back();
sums_.emplace_back();
sumsqs_.emplace_back();
}
void MinMaxPyramid::add_level() { levels_.emplace_back(); }
void MinMaxPyramid::reserve(u64 n) {
if (n > reserved_n_) reserved_n_ = n;
const u64 want = std::max(std::max(n, size_), u64{2});
// Materialise every level n samples can need -- level 0 plus one per doubling
// of the bucket -- so a run of appends never grows the level vector. See the
// header for why the bucket deques themselves cannot be pre-sized.
u32 want = 1;
for (u64 span = level_span(0); span <= n && want < 63; span <<= 1) ++want;
if (levels_.size() < want) levels_.resize(want);
}
// Materialise every level n samples can need -- buckets *and* prefix sums --
// and size each one up front, so a run of appends allocates nothing at all.
u32 max_levels = 1;
for (u64 span = level_span(0); span <= want && max_levels < 63; span <<= 1) ++max_levels;
while (levels_.size() < max_levels) add_level();
for (u32 l = 0; l < levels_.size(); ++l) {
const usize want_l = static_cast<usize>(buckets_for(want, l));
levels_[l].reserve(want_l);
sums_[l].reserve(want_l);
sumsqs_[l].reserve(want_l);
}
void MinMaxPyramid::push(u32 l, u64 k, const Entry& e) {
Level& lv = levels_[l];
// The first entry of a level fixes its base; every later entry is the next
// absolute bucket, which is what keeps a level one contiguous run.
if (lv.entries.empty()) lv.base = k;
lv.entries.push_back(e);
#ifdef PLOT_PYRAMID_COUNT_MERGES
detail::merge_count().fetch_add(1, std::memory_order_relaxed);
#endif
}
void MinMaxPyramid::append(f32 v) {
const u64 prev = size_;
const f64 vd = f64(v);
// sum_[l].back() is the sum over every sample appended so far, and the newest
// sample always lands in the last bucket of every level, so the last prefix
// entry of every level is exactly the running total.
const f64 total_before = sums_[0].empty() ? 0.0 : sums_[0].back();
const f64 total = total_before + vd;
const f64 sq_before = sumsqs_[0].empty() ? 0.0 : sumsqs_[0].back();
const f64 total_sq = sq_before + vd * vd;
const u64 idx = prev / kBucket;
if (idx == levels_[0].size()) {
levels_[0].push_back(MinMax{v, v});
sums_[0].push_back(total);
sumsqs_[0].push_back(total_sq);
// The new sample always lands in the in-progress level-0 bucket, which is the
// only bucket any append ever touches: a bucket becomes an entry exactly once,
// complete, and is never rewritten.
if (count_ % kBucket == 0) {
pending_.mm.min = v;
pending_.mm.max = v;
pending_.sum = vd;
pending_.sumsq = vd * vd;
} else {
MinMax& cur = levels_[0][static_cast<usize>(idx)];
cur.min = std::min(cur.min, v);
cur.max = std::max(cur.max, v);
sums_[0][static_cast<usize>(idx)] = total;
sumsqs_[0][static_cast<usize>(idx)] = total_sq;
pending_.mm.min = std::min(pending_.mm.min, v);
pending_.mm.max = std::max(pending_.mm.max, v);
pending_.sum += vd;
pending_.sumsq += vd * vd;
}
++count_;
if (count_ % kBucket != 0) return; // the bucket is still partial: nothing to push
const u64 k0 = count_ / kBucket - 1; // the level-0 bucket that just completed
if (k0 < level_base(0)) return; // its span starts before the window: never read
const Entry done = pending_;
push(0, k0, done);
// Cascade: level l gains an entry exactly when kBucket << l divides the sample
// count, so the walk stops at the first level that gained nothing -- amortised
// O(1) per sample (1/kBucket + 1/(2*kBucket) + ... = 1/(kBucket/2) merges).
for (u32 l = 1; l < 63; ++l) {
const u64 span = level_span(l);
if (count_ % span != 0) break;
const u64 k = count_ / span - 1; // the level-l bucket that just completed
const u64 child = 2 * k; // its left child, at level l-1
if (child < level_base(l - 1)) break; // a child is gone: this parent is not retained
if (l == static_cast<u32>(levels_.size())) add_level();
if (l >= live_levels_) live_levels_ = l + 1;
const Entry& a = at(l - 1, child);
const Entry& b = at(l - 1, child + 1);
Entry m;
m.mm.min = std::min(a.mm.min, b.mm.min);
m.mm.max = std::max(a.mm.max, b.mm.max);
m.sum = a.sum + b.sum;
m.sumsq = a.sumsq + b.sumsq;
push(l, k, m);
}
}
void MinMaxPyramid::drop_before(u64 absolute_sample) {
if (absolute_sample <= base_) return;
if (absolute_sample > count_) {
// The producer produced more than the ring held between two acquires: every
// appended sample is gone and the ones in between were never appended, so the
// pyramid restarts empty at the new base rather than splicing a gap.
base_ = absolute_sample;
count_ = absolute_sample;
pending_ = Entry{};
live_levels_ = 1;
for (u32 l = 0; l < levels_.size(); ++l) {
levels_[l].entries.clear();
levels_[l].base = ceil_div(absolute_sample, level_span(l));
}
return;
}
for (u32 l = 1; l < 63; ++l) {
// A level only carries data once the level below has more than one bucket: a
// one-child parent would just duplicate it.
if (levels_[l - 1].size() < 2) break;
if (l == live_levels_) {
++live_levels_;
if (l == levels_.size()) { // no reserve() sized this level up front
add_level();
if (reserved_n_ != 0) {
const usize want = static_cast<usize>(buckets_for(reserved_n_, l));
levels_[l].reserve(want);
sums_[l].reserve(want);
sumsqs_[l].reserve(want);
}
}
}
const u64 ci = prev / level_span(l - 1); // current bucket at level l-1
const u64 parent = ci / 2; // its parent at level l
MinMax m;
if ((ci & 1u) != 0) {
const MinMax& a = levels_[l - 1][static_cast<usize>(ci - 1)];
const MinMax& b = levels_[l - 1][static_cast<usize>(ci)];
m.min = std::min(a.min, b.min);
m.max = std::max(a.max, b.max);
base_ = absolute_sample;
for (u32 l = 0; l < levels_.size(); ++l) {
Level& lv = levels_[l];
// A bucket is dropped once its span starts before the window: the first
// retained bucket of every level is the first one wholly inside the window,
// which is exactly what the queries assume.
const u64 keep = ceil_div(base_, level_span(l));
if (keep <= lv.base) continue;
const u64 have = static_cast<u64>(lv.entries.size());
const u64 drop = std::min(keep - lv.base, have);
lv.entries.erase(lv.entries.begin(),
lv.entries.begin() + static_cast<std::ptrdiff_t>(drop));
// The base is where the next entry goes: past the last one if the level was
// emptied, which is how the cursor catches up with the sliding window.
if (drop == have) {
lv.base = keep;
} else {
m = levels_[l - 1][static_cast<usize>(ci)];
}
if (parent == levels_[l].size()) {
levels_[l].push_back(m);
sums_[l].push_back(total);
sumsqs_[l].push_back(total_sq);
} else {
levels_[l][static_cast<usize>(parent)] = m;
sums_[l][static_cast<usize>(parent)] = total;
sumsqs_[l][static_cast<usize>(parent)] = total_sq;
lv.base += drop;
}
}
size_ = prev + 1;
// The in-progress level-0 bucket may now start before the window's base: its
// aggregate covers samples that are gone. It belongs to the bucket the window
// starts inside, which is never read (the queries take whole buckets and the
// Series scans the two partial edges raw), so discarding it is lossless.
if (ceil_div(base_, kBucket) > count_ / kBucket) pending_ = Entry{};
while (live_levels_ > 1 && level_size(live_levels_ - 1) == 0) --live_levels_;
}
void MinMaxPyramid::clear() {
size_ = 0;
count_ = 0;
base_ = 0;
live_levels_ = 1;
for (auto& lv : levels_) lv.clear();
for (auto& s : sums_) s.clear();
for (auto& s : sumsqs_) s.clear();
levels_.resize(1);
levels_[0].entries.clear();
levels_[0].base = 0;
pending_ = Entry{};
}
std::span<const MinMax> MinMaxPyramid::level(u32 l) const {
if (l >= level_count()) return {};
return levels_[l];
u64 MinMaxPyramid::level_base(u32 l) const noexcept {
return l < levels_.size() ? levels_[l].base : u64{0};
}
u64 MinMaxPyramid::level_size(u32 l) const noexcept {
return l < levels_.size() ? static_cast<u64>(levels_[l].entries.size()) : u64{0};
}
MinMax MinMaxPyramid::bucket(u32 l, u64 i) const { return at(l, i).mm; }
Range MinMaxPyramid::reduce_buckets(u64 begin, u64 end) const {
const u64 e = std::min(end, size_);
const u64 b = std::min(begin, e);
const u64 b0 = ceil_to_bucket(b) / kBucket; // first whole bucket inside
const u64 b1 = e / kBucket; // one past the last whole bucket
const u64 e = std::min(end, size());
if (begin >= e) return Range{0.0, 0.0};
// The bucket grid is absolute while these are window coordinates: window index
// i is absolute window_base() + i, so the first whole bucket inside the range is
// the first absolute bucket boundary at or after that.
const u64 b0 = ceil_to_bucket(base_ + begin) / kBucket;
const u64 b1 = (base_ + e) / kBucket;
if (b1 <= b0) return Range{0.0, 0.0};
return fold_range(*this, b0, b1);
}
f64 MinMaxPyramid::sum_buckets(u64 begin, u64 end) const {
const u64 e = std::min(end, size_);
const u64 b = std::min(begin, e);
const u64 b0 = ceil_to_bucket(b) / kBucket;
const u64 b1 = e / kBucket;
const u64 e = std::min(end, size());
if (begin >= e) return 0.0;
const u64 b0 = ceil_to_bucket(base_ + begin) / kBucket;
const u64 b1 = (base_ + e) / kBucket;
if (b1 <= b0) return 0.0;
f64 total = 0.0;
cover_buckets(*this, b0, b1, [&](u32 l, u64 i) {
const f64 prev = i == 0 ? 0.0 : sums_[l][static_cast<usize>(i - 1)];
total += sums_[l][static_cast<usize>(i)] - prev;
});
cover_buckets(*this, b0, b1, [&](u32 l, u64 i) { total += at(l, i).sum; });
return total;
}
f64 MinMaxPyramid::sumsq_buckets(u64 begin, u64 end) const {
const u64 e = std::min(end, size_);
const u64 b = std::min(begin, e);
const u64 b0 = ceil_to_bucket(b) / kBucket;
const u64 b1 = e / kBucket;
const u64 e = std::min(end, size());
if (begin >= e) return 0.0;
const u64 b0 = ceil_to_bucket(base_ + begin) / kBucket;
const u64 b1 = (base_ + e) / kBucket;
if (b1 <= b0) return 0.0;
f64 total = 0.0;
cover_buckets(*this, b0, b1, [&](u32 l, u64 i) {
const f64 prev = i == 0 ? 0.0 : sumsqs_[l][static_cast<usize>(i - 1)];
total += sumsqs_[l][static_cast<usize>(i)] - prev;
});
cover_buckets(*this, b0, b1, [&](u32 l, u64 i) { total += at(l, i).sumsq; });
return total;
}
@@ -229,7 +264,7 @@ u32 MinMaxPyramid::level_for(u64 n, f64 samples_per_pixel) const {
const f64 lg = std::floor(std::log2(ratio));
u32 l = lg >= f64(62) ? 62u : static_cast<u32>(lg);
while (l > 0) {
const bool exists = l < level_count() && level(l).size() >= 2;
const bool exists = l < level_count() && level_size(l) >= 2;
if (exists && buckets_for(n, l) >= 2) break;
--l;
}
@@ -237,14 +272,14 @@ u32 MinMaxPyramid::level_for(u64 n, f64 samples_per_pixel) const {
}
u64 MinMaxPyramid::snap(f64 x, f64 x0, f64 dx) const {
if (dx <= 0.0 || size_ == 0) return 0;
if (dx <= 0.0 || size() == 0) return 0;
const f64 i = (x - x0) / dx;
// Bound the round before it: llround() is undefined for values a long long
// cannot hold, and this also maps NaN to index 0.
if (!(i > 0.0)) return 0;
if (i >= f64(size_)) return size_ - 1;
if (i >= f64(size())) return size() - 1;
const u64 idx = static_cast<u64>(std::llround(i));
return idx >= size_ ? size_ - 1 : idx;
return idx >= size() ? size() - 1 : idx;
}
} // namespace plot::data