fix: a loaded recording is filtered into its buffer, not truncated

The loader pushed one slot per file row while the plan decimated by
eight, so a 10 M-row file overran the ring and only its last 4.19 s
survived -- a 20 s recording that read as a 4 s one. Each group of
`time_stride` rows is now averaged into one slot at the group's mean
stamp, as the packet path already does, and the load line reports
rows and decimation so a filtered file cannot be mistaken for a
truncated one.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

README.md

@@ -672,6 +672,17 @@ from the file: a `.npy` written by `write_npy()` is a structured array whose *dt
 Only `.npy` is read: it is the format the item asked for and the one that carries names. A `.csv`
 export is for other tools.
 
+**A recording longer than the buffer can hold at its own rate is resampled, never cut short.** The
+file is pushed into the same ring a stream fills, and that ring is sized by `--buffer-mb`: 20 s of ten
+channels at 500 kS/s is 480 MB of rings, which a 200 MB buffer cannot hold. The loader therefore reads
+the decimation the plan chose and pushes one stored sample per group of file rows, carrying the
+group's **mean** — the same anti-aliased filter the live ingest applies, so a long event opens as the
+whole recording at a lower sample rate instead of as a fraction of itself. It used to push every row
+regardless, which overran the ring and left the file as **its own tail**: a 20 s event opened showing
+only its last 4 s, with a correct-looking 20 s header line above it. The load line now names both
+numbers, `samples=31250 rows=1000000 decim=32`, so what is on screen is never in doubt; give the run a
+larger `--buffer-mb` (the source's configuration carries one) to see such a file sample for sample.
+
 ## Named configurations
 
 What you arranged is remembered per data source, and a source can hold **several named

docs/claude/invariants-data-path.md

@@ -63,6 +63,17 @@ Part of [CLAUDE.md](../../CLAUDE.md). These are rules that cost a defect to lear
   length. The LOD min/max pyramid still preserves the extremes *of what is stored*; a spike
   narrower than a group is now filtered away rather than aliased, which is the correct
   trade and is why the status line's `rate` field is worth reading.
+- **A file is ingested too, so the loader decimates by the same factor.** `load_waveform_into_model()`
+  sets the target span to the recording's own length and enables every column, and the plan that
+  comes back may well decimate: 20 s of 10 channels at 500 kS/s is 480 MB of rings, and the budget is
+  `--buffer-mb`. The loader therefore reads `partition().time_stride` **after** the enables and pushes
+  one slot per that many rows, carrying the group's mean at the group's mean stamp — the packet path's
+  filter, written a second time because a file does not arrive in packets. Pushing every row instead
+  overruns the ring and leaves the file as **its own tail**, with nothing to say so: a 20 s event
+  opened as its last 4 s, `stored=10000000` against `published=2097152` in the channel dump, and a
+  trace that looks like a short recording rather than a truncated one. The `waveform loaded:` line
+  carries `rows=` and `decim=` beside `samples=` for exactly this reason — `samples * decim == rows`
+  is the arithmetic that says the whole file is in the buffer.
 - **One bad packet is bridged; only a run of them changes the time source.** The stream-time
   checks in `build_stamps()` reject three things: stamps that are not finite, a packet with no
   span at all, and a packet whose stamps do not advance past the previous packet's last. A

docs/claude/probe-readings.md

@@ -979,7 +979,8 @@ configuration that carries settings, is the defect this exists for — a saved h
 [sources] chooser ui: page='sources' title='Sources' icon=yes page='waveforms' title='Waveforms' icon=yes visible='sources' switcher_wired=1 buttons=[prefs=1 new_host=1 open=1 cancel=1]
 [sources] waveforms: 1 recent /tmp/rt.npy(ok) page=waveforms
 [sources] picked waveform /tmp/rt.npy
-[udpscope] waveform loaded: /tmp/rt.npy signals=4 samples=20001 span=2.000000s rate=10000 time=[-2.000000..0.000000]
+[udpscope] waveform loaded: /tmp/rt.npy signals=4 samples=20001 rows=20001 decim=1 span=2.000000s rate=10000 time=[-2.000000..0.000000]
+[udpscope] waveform loaded: /tmp/long.npy signals=4 samples=31250 rows=1000000 decim=32 span=19.999980s rate=50000 time=[-4.000000..15.999980]
 ```
 
 | field | value it must read | the defect a wrong reading names |
@@ -987,7 +988,9 @@ configuration that carries settings, is the defect this exists for — a saved h
 | `chooser ui:` | both pages with a `title` and an `icon=yes`, `switcher_wired=1`, and all four buttons `1` | the chooser's `.ui` losing an id: the code names the missing one and refuses the window, and this line is the frame that *did* build -- a page with `icon=NO` is one the switcher cannot draw |
 | `waveforms:` | one entry per remembered export, with `(ok)` or `(missing)`, and `page=waveforms` once the page is shown | the Waveforms page not built, or a stale path offered as if it were openable |
 | `picked waveform` | the selected path | a pick that carries no file: `--open` is what the mode reads |
-| `signals=`, `samples=` | the export's `cols=` and `rows=` | a reader mis-slicing its records, or the columns bound to the wrong channels |
+| `signals=`, `rows=` | the export's `cols=` and `rows=` | a reader mis-slicing its records, or the columns bound to the wrong channels |
+| `samples=` × `decim=` | `rows`, to within the last partial group | the loader not filtering the file into the plan it built: every row pushed into a ring that cannot hold them leaves the file as its own tail (`stored=` far above `published=` in the channel dump, and a 20 s recording on screen as its last 4 s) |
+| `decim=` | 1 whenever the recording fits the buffer at its own rate; a power of two above it | a decimation that moved without the span moving — the second line above is the reading to compare against: `samples` fell 32-fold and `span=` did **not** |
 | `span=`, `rate=`, `time=[…]` | the exported window and rate, `[-window, 0]` | the plan not covering the whole file (the tail arrives as the whole picture) or the display epoch not pinned to the file's last stamp |
 | `frames=0 packets=0` beside `channels=N` | 0 — no acquisition | a socket opened for a file, which would also make the starvation notice live |
 

docs/claude/verification.md

@@ -354,12 +354,27 @@ as distinct from the fault injection below, which exercises ingest:
 - **A waveform round-trips.** Export from a live run (`UDPSCOPE_EXPORT_MS` with
   `UDPSCOPE_EXPORT_PATH`, against the mock at 10 kS/s), then open that file with `--open` and an
   exit dump. The load must print
-  `[udpscope] waveform loaded: … signals=4 samples=20001 span=2.000000s rate=10000 time=[-2.000000..0.000000]`
+  `[udpscope] waveform loaded: … signals=4 samples=20001 rows=20001 decim=1 span=2.000000s rate=10000 time=[-2.000000..0.000000]`
   — **the same row count, column count, span and time range the export probe reported**
   (`rows=20001 cols=4 time=[-2.000000..0.000000]`) — and the dump must show `channels=4`,
   `frames=0 packets=0` (no acquisition) and, per channel,
   `20001 samples vs 20001 time slots … y=[-1.0000, 1.0000]`. A file that loads with fewer samples
   than it was written with, or with `NaN` slots, is the reader mis-slicing its records.
+- **A recording that does not fit its buffer is *filtered* into it, not truncated.** The case needs a
+  file bigger than the budget, which the mock cannot export in one window, so write one with NumPy —
+  the dtype is the exporter's own, `[('time','<f8'), ('abs_seconds','<f8'), (<signal>,'<f4'), …]` — and
+  open it against a small buffer:
+  ```bash
+  python3 -c "import numpy as np; n=1000000; t=-4.0+np.arange(n)/50000.0; \
+  a=np.zeros(n, dtype=[('time','<f8'),('abs_seconds','<f8'),('a','<f4'),('b','<f4'),('c','<f4'),('d','<f4')]); \
+  a['time']=t; a['abs_seconds']=1000.0+t; a['d']=np.linspace(-1,1,n); np.save('/tmp/long.npy', a)"
+  xvfb-run -a env UDPSCOPE_EXIT_AFTER_MS=4000 ./build/udpscope --open /tmp/long.npy --buffer-mb 1
+  ```
+  must read `samples=31250 rows=1000000 decim=32 span=19.999980s time=[-4.000000..15.999980]` and
+  `stored=31250 … published=31250` per channel — **the whole span, at a thirty-second of the rate**.
+  The ramp column is the assertion that the *head* is there: `y=[-0.999969, 0.999969]` is the file end
+  to end, where the defect this exists for reads a fraction of it. With every row pushed regardless of
+  the plan, this file arrived as its **last 0.9 s** and nothing in the log said so.
 - **A loaded file can be plotted.** `UDPSCOPE_PROBE_DROP_MS` drives the drop handler the way a drag
   from the signal panel does, on a file opened with `--open`: it must read
   `[drop-probe] signals=4 channels=4 rows=4 slots0=[sine|||]` — **`rows` equal to the column count**,

src/main.cpp

@@ -76,11 +76,15 @@ int main(int argc, char** argv) {
     // The whole file framed from the start: the window is the recording's own span, and nothing
     // afterwards restates it -- a zoom is the user's.
     if (report.span_s > 0.0) rows.set_time_window(report.span_s);
+    // `rows`/`decim` say whether what is on screen is the file sample for sample or a filtered
+    // version of it: `samples` alone below `rows` is how a recording that did not fit its buffer at
+    // its own rate used to read as a truncated one.
     std::fprintf(stderr,
-                 "[udpscope] waveform loaded: %s signals=%zu samples=%zu span=%.6fs rate=%.6g "
-                 "time=[%.6f..%.6f]\n",
-                 cfg.open_waveform.c_str(), report.signals, report.samples, report.span_s,
-                 report.rate_hz, report.first_time, report.last_time);
+                 "[udpscope] waveform loaded: %s signals=%zu samples=%zu rows=%zu decim=%llu "
+                 "span=%.6fs rate=%.6g time=[%.6f..%.6f]\n",
+                 cfg.open_waveform.c_str(), report.signals, report.samples, report.rows,
+                 static_cast<unsigned long long>(report.decimation), report.span_s, report.rate_hz,
+                 report.first_time, report.last_time);
     std::fflush(stderr);
   }
 

src/waveform.cpp

@@ -8,6 +8,7 @@
 #include <cmath>
 #include <cstdlib>
 #include <cstring>
+#include <mutex>
 #include <sys/stat.h>
 
 #include "layout_store.hpp"
@@ -283,18 +284,57 @@ bool load_waveform_into_model(ScopeModel& model, const std::string& path,
     if (error != nullptr) *error = "'" + path + "': the model built no channels for its columns";
     return false;
   }
+  // The plan the enables above built may **decimate**, and then the file has to be filtered into the
+  // ring exactly as the packet path filters the wire: one stored slot per `time_stride` rows,
+  // carrying their mean. A recording longer than a few seconds at a high rate does not fit a buffer
+  // sized in megabytes -- 20 s of 10 channels at 500 kS/s is 480 MB of rings -- so the planner
+  // decimates, and pushing every row regardless overruns the ring and leaves the file as *its own
+  // tail*: `stored=10000000` against a `published=2097152` ring read as "I loaded a 20 s file and
+  // only the last 4 s are there". Averaging and not striding, for the reason
+  // `invariants-data-path.md` gives: a stride aliases whatever is above the stored Nyquist.
+  std::uint64_t decim = 1;
+  {
+    std::lock_guard<std::mutex> lock(model.mutex());
+    if (model.partition().time_stride > 0) decim = model.partition().time_stride;
+  }
   std::vector<float> values(reader.names().size(), 0.0f);
+  std::vector<float> mean(reader.names().size(), 0.0f);
+  std::vector<double> acc(reader.names().size(), 0.0);
   std::size_t pushed = 0;
+  std::size_t rows_read = 0;
+  std::uint64_t in_group = 0;
+  double t_acc = 0.0;
+  double abs_acc = 0.0;
   double stamp = 0.0;
   double abs_stamp = 0.0;
   double first = 0.0;
   double last = 0.0;
-  while (reader.next(&stamp, &abs_stamp, values.data())) {
-    if (pushed == 0) first = stamp;
-    last = stamp;
-    model.push_loaded_slot(values, stamp, abs_stamp);
+  // The group's mean value at its mean stamp, which is where the ingest filter puts it too: a slot
+  // stamped at the group's *first* sample would shift the whole trace half a group early.
+  const auto flush_group = [&]() {
+    const double n = static_cast<double>(in_group);
+    for (std::size_t i = 0; i < acc.size(); ++i) {
+      mean[i] = static_cast<float>(acc[i] / n);
+      acc[i] = 0.0;
+    }
+    model.push_loaded_slot(mean, t_acc / n, abs_acc / n);
     ++pushed;
+    t_acc = 0.0;
+    abs_acc = 0.0;
+    in_group = 0;
+  };
+  while (reader.next(&stamp, &abs_stamp, values.data())) {
+    if (rows_read == 0) first = stamp;
+    last = stamp;
+    ++rows_read;
+    for (std::size_t i = 0; i < acc.size(); ++i) acc[i] += values[i];
+    t_acc += stamp;
+    abs_acc += abs_stamp;
+    if (++in_group >= decim) flush_group();
   }
+  // The last group need not be whole, and dropping it would shorten the record by up to one slot --
+  // which on a one-slot file (`decim` larger than the whole recording) is the entire file.
+  if (in_group > 0) flush_group();
   reader.close();
 
   model.set_epoch_if_unset(first);
@@ -305,6 +345,8 @@ bool load_waveform_into_model(ScopeModel& model, const std::string& path,
   if (report != nullptr) {
     report->signals = channels;
     report->samples = pushed;
+    report->rows = rows_read;
+    report->decimation = decim;
     report->span_s = pushed > 0 ? std::abs(last - first) : 0.0;
     report->rate_hz = rate;
     report->first_time = first;

src/waveform.hpp

@@ -58,7 +58,14 @@ class WaveformReader {
 /// What a load put into the model, for the log and the status line.
 struct WaveformLoadReport {
   std::size_t signals = 0;
+  /// Slots actually stored, which is the file's row count divided by `decimation`.
   std::size_t samples = 0;
+  /// Records the file held, and the group size each stored slot is the mean of. They differ when the
+  /// recording does not fit the buffer at its own rate: `samples * decimation == rows` (up to the
+  /// last, partial group), and a `decimation` above 1 is the only honest way to read a `samples`
+  /// below `rows`.
+  std::size_t rows = 0;
+  std::uint64_t decimation = 1;
   double span_s = 0.0;
   double rate_hz = 0.0;
   double first_time = 0.0;