diff --git a/README.md b/README.md index c3181b2..74ddaa6 100644 --- a/README.md +++ b/README.md @@ -672,6 +672,17 @@ from the file: a `.npy` written by `write_npy()` is a structured array whose *dt Only `.npy` is read: it is the format the item asked for and the one that carries names. A `.csv` export is for other tools. +**A recording longer than the buffer can hold at its own rate is resampled, never cut short.** The +file is pushed into the same ring a stream fills, and that ring is sized by `--buffer-mb`: 20 s of ten +channels at 500 kS/s is 480 MB of rings, which a 200 MB buffer cannot hold. The loader therefore reads +the decimation the plan chose and pushes one stored sample per group of file rows, carrying the +group's **mean** — the same anti-aliased filter the live ingest applies, so a long event opens as the +whole recording at a lower sample rate instead of as a fraction of itself. It used to push every row +regardless, which overran the ring and left the file as **its own tail**: a 20 s event opened showing +only its last 4 s, with a correct-looking 20 s header line above it. The load line now names both +numbers, `samples=31250 rows=1000000 decim=32`, so what is on screen is never in doubt; give the run a +larger `--buffer-mb` (the source's configuration carries one) to see such a file sample for sample. + ## Named configurations What you arranged is remembered per data source, and a source can hold **several named diff --git a/docs/claude/invariants-data-path.md b/docs/claude/invariants-data-path.md index 430e936..9336929 100644 --- a/docs/claude/invariants-data-path.md +++ b/docs/claude/invariants-data-path.md @@ -63,6 +63,17 @@ Part of [CLAUDE.md](../../CLAUDE.md). These are rules that cost a defect to lear length. The LOD min/max pyramid still preserves the extremes *of what is stored*; a spike narrower than a group is now filtered away rather than aliased, which is the correct trade and is why the status line's `rate` field is worth reading. +- **A file is ingested too, so the loader decimates by the same factor.** `load_waveform_into_model()` + sets the target span to the recording's own length and enables every column, and the plan that + comes back may well decimate: 20 s of 10 channels at 500 kS/s is 480 MB of rings, and the budget is + `--buffer-mb`. The loader therefore reads `partition().time_stride` **after** the enables and pushes + one slot per that many rows, carrying the group's mean at the group's mean stamp — the packet path's + filter, written a second time because a file does not arrive in packets. Pushing every row instead + overruns the ring and leaves the file as **its own tail**, with nothing to say so: a 20 s event + opened as its last 4 s, `stored=10000000` against `published=2097152` in the channel dump, and a + trace that looks like a short recording rather than a truncated one. The `waveform loaded:` line + carries `rows=` and `decim=` beside `samples=` for exactly this reason — `samples * decim == rows` + is the arithmetic that says the whole file is in the buffer. - **One bad packet is bridged; only a run of them changes the time source.** The stream-time checks in `build_stamps()` reject three things: stamps that are not finite, a packet with no span at all, and a packet whose stamps do not advance past the previous packet's last. A diff --git a/docs/claude/probe-readings.md b/docs/claude/probe-readings.md index 0df3c22..23a3562 100644 --- a/docs/claude/probe-readings.md +++ b/docs/claude/probe-readings.md @@ -979,7 +979,8 @@ configuration that carries settings, is the defect this exists for — a saved h [sources] chooser ui: page='sources' title='Sources' icon=yes page='waveforms' title='Waveforms' icon=yes visible='sources' switcher_wired=1 buttons=[prefs=1 new_host=1 open=1 cancel=1] [sources] waveforms: 1 recent /tmp/rt.npy(ok) page=waveforms [sources] picked waveform /tmp/rt.npy -[udpscope] waveform loaded: /tmp/rt.npy signals=4 samples=20001 span=2.000000s rate=10000 time=[-2.000000..0.000000] +[udpscope] waveform loaded: /tmp/rt.npy signals=4 samples=20001 rows=20001 decim=1 span=2.000000s rate=10000 time=[-2.000000..0.000000] +[udpscope] waveform loaded: /tmp/long.npy signals=4 samples=31250 rows=1000000 decim=32 span=19.999980s rate=50000 time=[-4.000000..15.999980] ``` | field | value it must read | the defect a wrong reading names | @@ -987,7 +988,9 @@ configuration that carries settings, is the defect this exists for — a saved h | `chooser ui:` | both pages with a `title` and an `icon=yes`, `switcher_wired=1`, and all four buttons `1` | the chooser's `.ui` losing an id: the code names the missing one and refuses the window, and this line is the frame that *did* build -- a page with `icon=NO` is one the switcher cannot draw | | `waveforms:` | one entry per remembered export, with `(ok)` or `(missing)`, and `page=waveforms` once the page is shown | the Waveforms page not built, or a stale path offered as if it were openable | | `picked waveform` | the selected path | a pick that carries no file: `--open` is what the mode reads | -| `signals=`, `samples=` | the export's `cols=` and `rows=` | a reader mis-slicing its records, or the columns bound to the wrong channels | +| `signals=`, `rows=` | the export's `cols=` and `rows=` | a reader mis-slicing its records, or the columns bound to the wrong channels | +| `samples=` × `decim=` | `rows`, to within the last partial group | the loader not filtering the file into the plan it built: every row pushed into a ring that cannot hold them leaves the file as its own tail (`stored=` far above `published=` in the channel dump, and a 20 s recording on screen as its last 4 s) | +| `decim=` | 1 whenever the recording fits the buffer at its own rate; a power of two above it | a decimation that moved without the span moving — the second line above is the reading to compare against: `samples` fell 32-fold and `span=` did **not** | | `span=`, `rate=`, `time=[…]` | the exported window and rate, `[-window, 0]` | the plan not covering the whole file (the tail arrives as the whole picture) or the display epoch not pinned to the file's last stamp | | `frames=0 packets=0` beside `channels=N` | 0 — no acquisition | a socket opened for a file, which would also make the starvation notice live | diff --git a/docs/claude/verification.md b/docs/claude/verification.md index c8fed7f..9545f1b 100644 --- a/docs/claude/verification.md +++ b/docs/claude/verification.md @@ -354,12 +354,27 @@ as distinct from the fault injection below, which exercises ingest: - **A waveform round-trips.** Export from a live run (`UDPSCOPE_EXPORT_MS` with `UDPSCOPE_EXPORT_PATH`, against the mock at 10 kS/s), then open that file with `--open` and an exit dump. The load must print - `[udpscope] waveform loaded: … signals=4 samples=20001 span=2.000000s rate=10000 time=[-2.000000..0.000000]` + `[udpscope] waveform loaded: … signals=4 samples=20001 rows=20001 decim=1 span=2.000000s rate=10000 time=[-2.000000..0.000000]` — **the same row count, column count, span and time range the export probe reported** (`rows=20001 cols=4 time=[-2.000000..0.000000]`) — and the dump must show `channels=4`, `frames=0 packets=0` (no acquisition) and, per channel, `20001 samples vs 20001 time slots … y=[-1.0000, 1.0000]`. A file that loads with fewer samples than it was written with, or with `NaN` slots, is the reader mis-slicing its records. +- **A recording that does not fit its buffer is *filtered* into it, not truncated.** The case needs a + file bigger than the budget, which the mock cannot export in one window, so write one with NumPy — + the dtype is the exporter's own, `[('time',',' 0.0) rows.set_time_window(report.span_s); + // `rows`/`decim` say whether what is on screen is the file sample for sample or a filtered + // version of it: `samples` alone below `rows` is how a recording that did not fit its buffer at + // its own rate used to read as a truncated one. std::fprintf(stderr, - "[udpscope] waveform loaded: %s signals=%zu samples=%zu span=%.6fs rate=%.6g " - "time=[%.6f..%.6f]\n", - cfg.open_waveform.c_str(), report.signals, report.samples, report.span_s, - report.rate_hz, report.first_time, report.last_time); + "[udpscope] waveform loaded: %s signals=%zu samples=%zu rows=%zu decim=%llu " + "span=%.6fs rate=%.6g time=[%.6f..%.6f]\n", + cfg.open_waveform.c_str(), report.signals, report.samples, report.rows, + static_cast(report.decimation), report.span_s, report.rate_hz, + report.first_time, report.last_time); std::fflush(stderr); } diff --git a/src/waveform.cpp b/src/waveform.cpp index 66f762e..f167847 100644 --- a/src/waveform.cpp +++ b/src/waveform.cpp @@ -8,6 +8,7 @@ #include #include #include +#include #include #include "layout_store.hpp" @@ -283,18 +284,57 @@ bool load_waveform_into_model(ScopeModel& model, const std::string& path, if (error != nullptr) *error = "'" + path + "': the model built no channels for its columns"; return false; } + // The plan the enables above built may **decimate**, and then the file has to be filtered into the + // ring exactly as the packet path filters the wire: one stored slot per `time_stride` rows, + // carrying their mean. A recording longer than a few seconds at a high rate does not fit a buffer + // sized in megabytes -- 20 s of 10 channels at 500 kS/s is 480 MB of rings -- so the planner + // decimates, and pushing every row regardless overruns the ring and leaves the file as *its own + // tail*: `stored=10000000` against a `published=2097152` ring read as "I loaded a 20 s file and + // only the last 4 s are there". Averaging and not striding, for the reason + // `invariants-data-path.md` gives: a stride aliases whatever is above the stored Nyquist. + std::uint64_t decim = 1; + { + std::lock_guard lock(model.mutex()); + if (model.partition().time_stride > 0) decim = model.partition().time_stride; + } std::vector values(reader.names().size(), 0.0f); + std::vector mean(reader.names().size(), 0.0f); + std::vector acc(reader.names().size(), 0.0); std::size_t pushed = 0; + std::size_t rows_read = 0; + std::uint64_t in_group = 0; + double t_acc = 0.0; + double abs_acc = 0.0; double stamp = 0.0; double abs_stamp = 0.0; double first = 0.0; double last = 0.0; - while (reader.next(&stamp, &abs_stamp, values.data())) { - if (pushed == 0) first = stamp; - last = stamp; - model.push_loaded_slot(values, stamp, abs_stamp); + // The group's mean value at its mean stamp, which is where the ingest filter puts it too: a slot + // stamped at the group's *first* sample would shift the whole trace half a group early. + const auto flush_group = [&]() { + const double n = static_cast(in_group); + for (std::size_t i = 0; i < acc.size(); ++i) { + mean[i] = static_cast(acc[i] / n); + acc[i] = 0.0; + } + model.push_loaded_slot(mean, t_acc / n, abs_acc / n); ++pushed; + t_acc = 0.0; + abs_acc = 0.0; + in_group = 0; + }; + while (reader.next(&stamp, &abs_stamp, values.data())) { + if (rows_read == 0) first = stamp; + last = stamp; + ++rows_read; + for (std::size_t i = 0; i < acc.size(); ++i) acc[i] += values[i]; + t_acc += stamp; + abs_acc += abs_stamp; + if (++in_group >= decim) flush_group(); } + // The last group need not be whole, and dropping it would shorten the record by up to one slot -- + // which on a one-slot file (`decim` larger than the whole recording) is the entire file. + if (in_group > 0) flush_group(); reader.close(); model.set_epoch_if_unset(first); @@ -305,6 +345,8 @@ bool load_waveform_into_model(ScopeModel& model, const std::string& path, if (report != nullptr) { report->signals = channels; report->samples = pushed; + report->rows = rows_read; + report->decimation = decim; report->span_s = pushed > 0 ? std::abs(last - first) : 0.0; report->rate_hz = rate; report->first_time = first; diff --git a/src/waveform.hpp b/src/waveform.hpp index 006bd9a..39a6a0c 100644 --- a/src/waveform.hpp +++ b/src/waveform.hpp @@ -58,7 +58,14 @@ class WaveformReader { /// What a load put into the model, for the log and the status line. struct WaveformLoadReport { std::size_t signals = 0; + /// Slots actually stored, which is the file's row count divided by `decimation`. std::size_t samples = 0; + /// Records the file held, and the group size each stored slot is the mean of. They differ when the + /// recording does not fit the buffer at its own rate: `samples * decimation == rows` (up to the + /// last, partial group), and a `decimation` above 1 is the only honest way to read a `samples` + /// below `rows`. + std::size_t rows = 0; + std::uint64_t decimation = 1; double span_s = 0.0; double rate_hz = 0.0; double first_time = 0.0;