fix(data): level-per-emission pyramid cover, relaxed-atomic ring slots

cover_buckets chose one level per pass, so an odd first whole bucket (half of all

measurement ranges) never reached a level above 0 and folded every remaining

bucket at level 0 -- 15624 entries for a 1e6-sample pyramid. Pick the largest

aligned, in-range, materialised level at each cursor instead; the new cost test

counts folded entries through a PLOT_PYRAMID_COUNT_COVERS-only hook (absent from

normal builds) and fails on the old body.

Ring slots are now relaxed std::atomic: overwrite-on-full means the producer

reuses the slot of a snapshot's oldest sample while the consumer reads it, which

was a data race, not just a stale value. copy_from loads explicitly instead of

std::copy; push stays lock-free. A new producer/consumer case is the one TSan

flags without this.

Also extends the wrapped-window test with x0/bounds/pyramid/sum/sumsq/gather

checks, and corrects the StreamSeries threading note to the sample-count bound.
This commit is contained in:
Martino Ferrari committed 2026-09-27 21:42:53 +02:00
1 parent 138fa478c3
commit d9909f2115
8 files changed
+190 -38

No files matched your search

+27 -17
View File
@@ -1,8 +1,6 @@
#pragma once
#include <algorithm>
#include <atomic>
#include <bit>
#include <cstddef>
#include <vector>
#include <plot/core/types.hpp>
@@ -16,6 +14,15 @@ namespace plot::data {
/// sample count with a release store; a consumer that observes that count with an
/// acquire load is guaranteed to see the slot contents written before it.
///
/// Slots are std::atomic and every slot access is relaxed. Because the ring
/// *overwrites*, the producer's next push reuses the slot of the snapshot's
/// oldest retained element while the consumer may still be reading it, so a plain
/// load/store there would be a data race (UB), not merely a stale value. A
/// relaxed atomic load/store is a plain mov/ldr/str on the supported targets, so
/// the sample path costs the same. Relaxed is enough because the ordering lives in
/// the count handshake above; the count fixes *which* indices a snapshot reads,
/// not that the slots still hold those samples (see StreamSeries).
///
/// A push into a full ring silently discards the oldest element: for a live
/// plotting stream the newest sample is the interesting one and the retained
/// window slides forward.
@@ -29,6 +36,8 @@ namespace plot::data {
/// such a snapshot and then reads only absolute indices.
template <typename T>
class RingBuffer {
static_assert(std::atomic<T>::is_always_lock_free,
"RingBuffer slots are atomic and must be lock-free: the sample path may not lock");
public:
explicit RingBuffer(u64 capacity)
: cap_(std::bit_ceil(capacity < 1 ? u64{1} : capacity)),
@@ -38,7 +47,7 @@ class RingBuffer {
/// Producer thread only.
void push(T v) {
const u64 i = produced_.load(std::memory_order_relaxed);
buf_[static_cast<usize>(i & mask_)] = v;
buf_[static_cast<usize>(i & mask_)].store(v, std::memory_order_relaxed);
produced_.store(i + 1, std::memory_order_release);
}
@@ -66,24 +75,25 @@ class RingBuffer {
}
/// Single-threaded: retained-window index (0 = oldest retained). `i` must be
/// < size().
const T& operator[](u64 i) const noexcept {
return buf_[static_cast<usize>((base() + i) & mask_)];
/// < size(). Returns by value: the slot is an atomic, so a relaxed load is the
/// only way to observe it.
T operator[](u64 i) const noexcept {
return buf_[static_cast<usize>((base() + i) & mask_)].load(std::memory_order_relaxed);
}
/// Consumer: copies the `n` elements at absolute indices [absolute, absolute+n)
/// into `out`, splitting the copy at the ring's wrap point. The caller must hold
/// a snapshot that retains those indices; this never consults the live count, so
/// a pushing producer cannot shift the read.
/// into `out`, masking the index to wrap. The caller must hold a snapshot
/// that retains those indices; this never consults the live count, so
/// a pushing producer cannot shift the read -- but it fixes only *which*
/// indices are read, not that the slots still hold those samples: a producer
/// that overtakes capacity() has already rewritten the oldest ones (see the
/// class note).
void copy_from(u64 absolute, u64 n, T* out) const {
if (n == 0) return;
const u64 s = absolute & mask_;
const u64 run = std::min(n, cap_ - s);
const auto first = buf_.begin() + static_cast<std::ptrdiff_t>(s);
std::copy(first, first + static_cast<std::ptrdiff_t>(run), out);
if (run < n) {
std::copy(buf_.begin(), buf_.begin() + static_cast<std::ptrdiff_t>(n - run),
out + static_cast<std::ptrdiff_t>(run));
// Explicit relaxed loads: std::copy cannot read atomic slots, and the mask
// makes the wrap split unnecessary.
for (u64 i = 0; i < n; ++i) {
out[i] = buf_[static_cast<usize>((absolute + i) & mask_)].load(std::memory_order_relaxed);
}
}
@@ -94,7 +104,7 @@ class RingBuffer {
private:
u64 cap_;
u64 mask_;
std::vector<T> buf_;
std::vector<std::atomic<T>> buf_;
std::atomic<u64> produced_{0};
};