# Verification How to run the headless self-check, the two baselines every run is read against, and the environment variables that schedule each probe. Part of [CLAUDE.md](../../CLAUDE.md). There is no unit test suite. The only automated check is the app's own headless self-check, driven by env vars and run under a display (`xvfb-run`, llvmpipe) against a live UDPStreamer source: ```sh xvfb-run -a env UDPSCOPE_EXIT_AFTER_MS=8000 UDPSCOPE_ENABLE_ALL_MS=1000 \ ./build/udpscope --server 127.0.0.1 --port 8000 ``` Two baselines every run is read against. A clean build emits **four** warnings and no more: two `-Wmissing-field-initializers` on `MathNode::source` and `-Woverlength-strings` on **both** generated GResource `.c` files — the help pages' and the icons', whose blob crossed C99's 4095-byte string-literal limit when the two app icons joined the four action ones (a note about a generated file, which GCC itself supports). The fifth warning that used to appear here was an unused `output_built` in `math_probe()`; it was **deleted** as dead code — redundant with the `output_after_add` already printed on the same line — and that deletion is what dropped the baseline from five to four. Nothing started reading it. And an exit-dump run emits **no** `Gtk-CRITICAL` at all. A fifth warning, or any critical, is a finding. An *incremental* build reads **two** of those four: the generated GResource `.c` files are not recompiled when only `src/` changed, so their `-Woverlength-strings` do not reappear. That second baseline used to be "exactly two, `gtk_widget_get_height` and `gtk_widget_get_width`, from the dump asking a destroyed widget for its size", and tolerating them was a mistake: the dump runs from the destroy handler, so the widget pointer is **freed memory**, and a critical is only the lucky reading of it. A run that had built and destroyed other windows first — `UDPSCOPE_PROBE_HELP_MS`, which opens three — has had that memory reused, and the same two calls then **SIGSEGV in the middle of the dump**, which reads as a truncated dump rather than as a crash in a debug line. Every size the dump prints is therefore captured on the *tick*, while the window is alive (`App::dbg_*`), and **no `gtk_widget_*` call may appear in the dump** — the `panel controls:` line was already written that way for the same reason, and that comment is the precedent, not an exception. - `UDPSCOPE_EXIT_AFTER_MS` — run the trigger-on → go-live probe, close the window, then dump the whole pipeline state to stdout (frame/sample counts, per-plot painted frames, trace bindings, panel sizes, status line in both states). - `UDPSCOPE_ENABLE_ALL_MS`, `UDPSCOPE_ARM_TRIGGER_MS`, `UDPSCOPE_PROBE_DISARM_MS`, `UDPSCOPE_TRIG_TRACE_MS` — schedule individual probes. `UDPSCOPE_PROBE_DISARM_MS` only *starts* its sequence: it switches the trigger on and then polls for a capture to open, so the Disarm lands ~300 ms into one rather than wherever a fixed delay happened to fall. - `UDPSCOPE_PROBE_TRIGRENUM_MS` — the trigger's capture must survive a **renumbering of its own channel**. It sets the reported geometry through the real widgets (1 s window, 10 % pre share, hold-off from `--holdoff`) on the mock's *square* alone, polls for a capture to open, and then enables the sine — which sorts before the square and so moves the trigger channel's index under the engine. Its three lines are in [probe-readings.md](probe-readings.md); **`t0_held=1` is the assertion**. Run it with `--holdoff 10`, which makes the stolen-t0 reading unmistakable: a second latch barely a second after the first cannot be a hold-off that applied. - `UDPSCOPE_PROBE_FIELDS_MS` — change the trigger panel's edge and Normal/Single dropdowns. As much a crash test as a state check (see the GTK signal-signature invariant below), so time it inside a capture: `[fields-probe] ... capturing=1` followed by no SIGSEGV *is* the assertion. - `UDPSCOPE_PROBE_WINDOW_MS` — drive the window control every way a user can: type a number, change the unit under it, step a decade either way. - `UDPSCOPE_PROBE_LAYOUT_MS` — drive the layout editor: switch it on, insert above the first row, remove a row, append one at the bottom, switch it off. Its steps are 200 ms apart on purpose — the buttons are placed from the rects the *last frame* was laid out with, so a synchronous sequence would read every position one action stale. It clicks by emitting `::clicked`, never `gtk_widget_activate()`, which holds a `GtkButton` for its 250 ms `ACTIVATE_TIMEOUT` and then emits into a button the rebuild has already destroyed. - `[measure-probe] N measurements on 'X'` (stderr, `UDPSCOPE_PROBE_MEASURE_MS`) — adds one measurement of each statistical type (average, top, peak to peak, amplitude) on the first *channel* and forces a recompute. It must name a **channel**, not `signals().front()`: the first signal of a stream is its timestamp, which is not acquired, and naming it adds four measurements that all read `--` *and* makes a performance run look free (each bails before collecting anything -- that is how the first baseline of the stall below read 0.4 ms). The values are in the exit dump as `measurements: =`; on the fast mock (`tools/mock_udpstreamer.py --rate 1300000 --batch 2000`) a unit sine must read `peak to peak=2`, `amplitude=1`, `average=0` and **`top=0.636620`** -- 2/pi, the half-wave mean of a sine, which is the check that the threshold types are not merely plausible. `[measure-trace] view=[..] =... marks=N level:V/time:V...` is a repeating line (250 ms) from the probe: the pane's numbers beside the *current* window and the marks actually placed, which is the only way a stale result or a mark that should not be drawn shows up as a number. It must never print `marks=0` while measurements with `show_on_plot` are on and the data is there -- that was the rulers blinking off at a mode change. `measure cost: N recomputes, X ms total, Y ms each, data D ms (P%), covered=C buckets=B raw=R (F%), N ns/covered sample` follows it: **read it against a run with `UDPSCOPE_PROBE_MEASURE_MS` unset**, because a frame mean cannot attribute a cost. The last three fields are the assertion, and they are a *ratio*, not a time: `covered` is the samples the window spans, `buckets` the LOD buckets the two walks answered from, and `raw` the samples they still had to read. **`raw` must stay at a fraction of a percent of `covered`** on a clean waveform over a window of millions -- 0.1% is what the pyramid-backed walks give (see the measurement-cost invariant in [invariants-measurements.md](invariants-measurements.md)). A `raw` near `covered` means the walks went back to reading the window, which at 1 MS/s is 19.7 ms per recompute and a third of the run's frames. It is *legitimately* higher for a window shorter than a bucket (64 samples) or a signal noisy enough to cross a threshold everywhere, so read it on the fast mock (`tools/mock_udpstreamer.py --rate 1300000 --batch 2000`), where a window is ~2 M samples. `--noise 0.1` there reads **2.0%** at 1.96 ms a recompute against the clean 0.1% at 1.41 ms, and that is the interferer's own cost rather than a regression -- the figure to compare a change against is the **clean** one. Note the **final** `measurements:` line and `[measure-trace]`'s last lines may read `--`: `UDPSCOPE_EXIT_AFTER_MS` arms the trigger before it dumps, so the dump can land in `Waiting` with nothing bound and the axis at its default. The race-free reading is an earlier `[measure-trace]` line, which carries its own `view=`; those lines are also where the level-mark fallback is visible (an invalid pass must still print its `level:` marks). - `[pan-probe] asked=[..] was=[..]`, `[pan-probe] after=[..] held=D asked=[..]`, then `[pan-probe] zoom asked=[..] after=[..] span=S window=W limited=L` (stderr, `UDPSCOPE_PROBE_PAN_MS`) — **`held` must be 1**: a gesture must not move a window that is pinned (trigger mode, or a paused view). The probe writes the panned range the way a gesture does — on the *member* axis, which `Figure::layout()` adopts into the group — and reports the axis the frame after it produced. Time it *inside* one pinned state: a pan during a capture whose check lands after the capture has ended is re-framed by the mode change, not by the pin, and reads `held=1` with the hold removed (that is how the first version of this probe fooled me). The pause probe makes a stable pin, so run them together: `UDPSCOPE_PROBE_PAUSE_MS=1500 UDPSCOPE_PROBE_PAN_MS=2200` puts the pan and its check inside the pause. With `hold_pinned_window()` disabled it reads `held=0` and the axis sits at the asked range. It then asks for a **zoom-in** (a window a quarter of what is on screen) and reports `group=`/`member=` beside it: **`allowed` must be 1, with both equal to the ask** -- that is the single-trigger-mode case, where the wheel has to keep working, and where reading the wrong one of the two made a working clamp look broken. All three verdicts are read from the *tick*, one frame after the ask: a `g_timeout` was the first version and it drifted past the paused/resumed boundary under load. The probe then asks for a **zoom-out** ten times the current span and reports the span the next frame produced: **`limited` must be 1**, i.e. `span <= window`. Run it with a ring that holds *more* than the window (`--buffer-mb 64` past the window's own seconds) — that is the state the old `max(retained, window_s_)` bound leaked in, and it showed up here as `span=2.502550 window=2.000000 limited=0` against `span=2.000000` after the fix. - `[zoom-probe] /x axis=[..] span=.. n=N idx=[a..b) spp=.. level=L pts=P kept=K back=.. ink=[..] x_at_first=.. gap_px=.. end_px=.. rect=[..]` (stderr, `UDPSCOPE_PROBE_EVENTZOOM_MS`) — arms the trigger in **single** mode, polls for a **held event** (a latch happens when the stream says so; a sweep on a timer has nothing to zoom into), then zooms in eight doublings twice: `ges/` writes the row's own axis the way a gesture does, `win/` drives the window control (`apply_time_window`, which unpins and re-arms the live follow before the pin is restated). Each step's reading is taken in the *tick*, after that frame's pin and clamp — a `g_timeout` would report the ask. It recomputes what `Renderer::draw_traces` is handed, through the same calls: `neighbours()` for `idx`, the 64-spp gate for `level`, then `gather()`. **`gap_px` and `end_px` must stay within a pixel or two at every step** — they are how much of the left and right margin no geometry reaches, and a `gap_px` that *doubles* per step is the "zooming into a held event clips it more and more from the left" defect (374 px of a 716 px axis by `ges/x128`), whose cause is a constant offset in data seconds (see the `neighbours()` invariant in [invariants-rendering.md](invariants-rendering.md)). `kept`/`back` are the renderer's own increasing-suffix filter counted rather than applied: `kept` far below `pts` is a staircase, and a non-zero `back` is a non-monotonic gather. **Run it with `--clock-jitter`** (`tools/mock_udpstreamer.py --rate 1300000 --batch 2000 --clock-jitter 0.4`): the ideal mock's stamps sit on a perfect grid, which is exactly what hides this class of bug, and three runs against it reported `gap_px=0.0` throughout while the defect was live. The jitter varies each *packet's* spacing rather than offsetting its start, because the stream's clock has to stay strictly increasing. The `win/` sweep bottoms out at `span=0.05` (`RowSet::set_time_window()`'s floor) and walks to the event's post side, since `set_time_window()` frames the newest span before the pin clamps it — both are expected. - `UDPSCOPE_PROBE_PAUSE_MS` — pause acquisition mid-run so layout numbers are measured on a still picture. It now also asserts what a pause *is* (`[pause-probe] paused=D ring_produced=N event_size=M frozen_x=[..] axis=[..] glance=[..]` on stderr): it presses the **button**, not `Acquisition::set_paused()`, because pausing is a display state as well as an acquisition one and the view snapshot and axis pin live in the handler. Across the three lines **`ring_produced` must grow** (the producer never stops — it fills the live ring exactly as it does with the trigger off), **`event_size` must be non-zero exactly while paused** (the plot is bound to the frozen snapshot, not the ring) and **`frozen_x` must not move between the pause and the resume**; after the resume `event_size` is 0 and the window is back on the newest data. Pausing used to stop the producer, which is why the counts grew by nothing and the resume left a hole. `frozen_x` is the *rebased* snapshot — the pause view's own stamps are relative to the freeze instant now, so the range reads `[-window..0]`-ish rather than stream seconds, and a `frozen_x` in the thousands is the rebase gone. **`glance=` is the assertion for the paused status field**, and this is the only line that can make it: the exit dump's `status bar:` is read live and its two `mode …` lines are the live and triggered states. While paused it must read `paused · · · for `, with the elapsed figure **growing between the two paused lines** and **absent on the resume line**. It is **wall** time (`g_get_monotonic_time()`), not stream time, because the producer never stops while paused and "how long have I been looking at this" is a question about the user, not about the stream; it is written through `format_elapsed()` so the field steps to a coarser unit instead of growing digits (see the layout rule). - `UDPSCOPE_PROBE_COLOR_MS` — recolour one plotted signal and read the colour back out of the trace and the chip, round-trip it through the session writer, then clear it. Schedule `UDPSCOPE_ENABLE_ALL_MS` before it: with nothing bound it has no trace to recolour and asserts nothing. Its four lines are in [probe-readings.md](probe-readings.md). - `UDPSCOPE_EXPORT_MS` (+ optional `UDPSCOPE_EXPORT_PATH`, default `/tmp/udpscope-export`) — collect the export table and write it both ways, skipping only the file chooser. - `UDPSCOPE_PROBE_CONDITION_MS` — arms a qualified trigger against the mock and reports the latched `t0`, the measured pulse or band width, and sample values for the window condition. Schedules its own arm and report stages. `UDPSCOPE_PROBE_CONDITION_SHAPE` picks the leg: `0` pulse width `<` (the glitch, the default, needs `--glitch-at`/`--glitch-width`), `1` pulse width `>` (uses a 1.0 s window and 50 % pre so the opening edge of a 0.2 s pulse is captured), `2` window (uses `sine` with a 0.2 s window and 10 % pre), `3` dropout (needs `--quiet-at`; uses `sine` with a 1.0 s window and 50 % pre so the last crossing before the outage is captured). - `UDPSCOPE_PROBE_STATS_MS` — reports every measurement's accumulator (`n`, mean, sd, lo, hi), pauses, reports again with `dn=` per measurement, then resets and reports a third time. Three phases: `phase=0`, `phase=1` (after pause) and `phase=2` (after Reset). Needs `UDPSCOPE_ENABLE_ALL_MS` ahead of it and `UDPSCOPE_PROBE_MEASURE_MS` to add measurements first. - `UDPSCOPE_PROBE_SEARCH_MS` — runs a buffer search for the armed condition, cross-checks the live trigger's own `t0` against the nearest hit (`agree=1` is the load-bearing reading), and jumps to hit 0. Reports the resulting `mode=`, `axis=` and `marks placed=`. The following mock waveform features are what the qualified-condition probes are checked against, as distinct from the fault injection below, which exercises ingest: - `--glitch-at S` / `--glitch-width W` invert the `square` signal for `W` seconds at the start of every square period from stream second `S` onwards — a runt pulse of a known width, which is what a `<` pulse-width trigger and a glitch search are checked against. - `--quiet-at S` freezes `sine` at its held value from stream second `S` onwards, so it stops crossing zero: the dropout trigger's input. - `UDPSCOPE_PROBE_RESEQ_MS` — one mid-run reading of the resequencing path. Four cases, **run each as its own process** — four back to back starve the mock past its 2 s silence timeout and every case then reads a reconnect instead of what it was testing. Common conditions for all four: the mock at `--rate 10000 --batch 100 --freq 5`, udpscope with no extra options, under `xvfb-run` with `UDPSCOPE_EXIT_AFTER_MS=8000 UDPSCOPE_ENABLE_ALL_MS=600 UDPSCOPE_EXIT_DUMP=1 UDPSCOPE_PROBE_RESEQ_MS=5000`. | case | mock extra flags | key reading | | --- | --- | --- | | **base** | *(none)* | every counter 0; proves the machinery costs nothing | | **reorder** | `--reorder-every 10` | `reseq=51` (mid-run) / `102` (dump); `holes=0`; resequencing inside the window leaves no reservation | | **late** | `--reorder-every 10 --reorder-delay-ms 50` | `holes=42 bands=42 bands_repaired=42 nan_slots=0`; every band is the faint, repaired tier | | **mixed** | `--drop-every 20 --reorder-every 10 --reorder-delay-ms 50` | `bands=42 bands_repaired=20 nan_slots=2300`; all three band states at once | Full measured readings (probe line first, then exit-dump; `bands_repaired` was added after the original probe and any earlier reading without it is obsolete): ``` === base mock: none [reseq-probe] reseq=0 late=0 stale=0 holes=0 hole_slots=0 backfilled=0 open_holes=0 bands=0 bands_repaired=0 nan_slots=0 scanned=44000 trigger_in_hole=0 [udpscope] frames=1024 packets=1025 gaps=0 channels=4 span=838.861s stride=1 reseq=0 late=0 stale=0 holes=0 hole_slots=0 backfilled=0 [udpscope] channel 0 'sine' stored=95200 published=95000 === reorder mock: --reorder-every 10 [reseq-probe] reseq=51 late=0 stale=0 holes=0 hole_slots=0 backfilled=0 open_holes=0 bands=0 bands_repaired=0 nan_slots=0 scanned=44000 trigger_in_hole=0 [udpscope] frames=1028 packets=1029 gaps=0 channels=4 span=838.861s stride=1 reseq=102 late=0 stale=0 holes=0 hole_slots=0 backfilled=0 [udpscope] channel 0 'sine' stored=95400 published=95100 === late mock: --reorder-every 10 --reorder-delay-ms 50 [reseq-probe] reseq=142 late=48 stale=0 holes=42 hole_slots=4200 backfilled=16800 open_holes=42 bands=42 bands_repaired=42 nan_slots=0 scanned=44000 trigger_in_hole=0 [udpscope] frames=1026 packets=1027 gaps=100 channels=4 span=838.861s stride=1 reseq=288 late=100 stale=0 holes=94 hole_slots=9400 backfilled=37600 [udpscope] channel 0 'sine' stored=95700 published=95500 === mixed mock: --drop-every 20 --reorder-every 10 --reorder-delay-ms 50 [reseq-probe] reseq=163 late=24 stale=0 holes=43 hole_slots=4300 backfilled=8000 open_holes=43 bands=42 bands_repaired=20 nan_slots=2300 scanned=44100 trigger_in_hole=0 [udpscope] frames=974 packets=975 gaps=101 channels=4 span=838.861s stride=1 reseq=314 late=50 stale=0 holes=94 hole_slots=9400 backfilled=18400 [udpscope] channel 0 'sine' stored=95100 published=94900 ``` The `mixed` case's `open_holes=43` against `bands=42` is correct: the 43rd hole has scrolled out of the visible window, and `build_invalid_spans()` drops a span outside the window rather than pinning it to the edge. The per-field reference is in [probe-readings.md](probe-readings.md). The mock also carries three **fault injections** that no probe schedules, because what they exercise is the ingest rather than the UI. The first: `--time-glitch-at --time-glitch-by ` makes **one** packet's time signal jump ahead and then carries on from where the true timeline was, which is what a device whose clock glitches puts on the wire. Run it with the trigger armed, e.g. ```sh xvfb-run -a env UDPSCOPE_ENABLE_ALL_MS=400 UDPSCOPE_ARM_TRIGGER_MS=700 UDPSCOPE_EXIT_AFTER_MS=22000 \ ./build/udpscope --server 127.0.0.1 --port 8000 --holdoff 10 # mock: --rate 10000 --batch 100 --freq 5 --time-glitch-at 6.0 --time-glitch-by 10.0 ``` Four readings, all off the once-a-second `[udpscope] trig engine:` line: - exactly one `warning: ... jumped implausibly far ahead of the previous packet (16.01..16.0199 against a previous 6.0099); bridging that packet ...` — the glitch was **caught**; - **no** `timing samples by arrival instead` — one bad packet must not change the time source; - `now_t` crosses the glitch continuously (`5.040 → 6.050 → 7.050`), and `history` keeps **growing** (`1.01 … 23.15`) instead of collapsing — the ring was not wiped; - the latches stay a hold-off apart: `edge at 1.060 / latched at 2.060 / holdoff_until=12.060`, then `edge at 12.260 / latched at 13.260 / holdoff_until=23.260` — **11.2 s** for a 10 s hold-off plus a 0.9 s post window. Before the fix the same run read a fallback warning, `history` collapsing 4.03 → 0.95, `now_t` jumping 5.040 → 16.040, and two latches **1.2 s** apart. Note the window here comes from the saved session for that port; the figures above are the default 2 s window (50 % pre), so move `~/.config/udpscope/_.toml` aside to reproduce them. The other two are **packet loss** (`--drop-every N`, `--drop-burst K`) and **wire inversion** (`--reorder-every N`). Loss withholds a datagram the counter and the stream clock have already advanced over; the client counts the forward jump as `counter_gaps`. An inversion withholds a packet until the one behind it is sent, so the receiver sees a later counter first; a packet that arrives within the 20 ms reorder window is resequenced with no hole, and one that arrives past it leaves a committed hole that it then backfills (`UDPSCOPE_PROBE_RESEQ_MS` exercises both paths). There is **no interpolation** anywhere below the ring. All of the readings below are off `UDPSCOPE_EXIT_DUMP`'s `[udpscope] frames= packets= gaps=` line, over an 8 s run against `--rate 10000 --batch 100 --freq 5` (100 packets a second) except where the burst needs the faster stream. **`gaps` is the exact reading** and repeats to the packet; `frames` and `packets` move by one or two with the run's own timing. Run each of these on its own — four of them back to back in one script starved the mock past the 2 s silence timeout and every case, including the clean one, then read a reconnect instead of what it was testing: - **clean**, as the reference: `frames=1029 packets=1030 gaps=0`, no warning. - `--drop-every 10`: `frames=925 packets=926 gaps=102`. Every tenth packet never reaches the wire, so `packets` falls with `frames` and `gaps` is the count of them. No warning: a one-packet hole is a one-packet jump in the stamps, far inside the glitch guard. - `--reorder-every 10`: `frames=1028 packets=1029 gaps=0`. The inverted packet lands inside the 20 ms reorder window and is resequenced: every datagram arrives and `gaps` is 0. The pre-resequencing signature was `frames=925 packets=1028 gaps=102`; that is now the reading for `--reorder-every 10 --reorder-delay-ms 50`, which forces a late arrival past the window. - `--drop-every 1000 --drop-burst 200` at `--rate 100000 --batch 100` (1000 packets a second): `frames=7000 packets=7001 gaps=1600`, eight bursts of 200, and still no warning. - `--drop-every 1000 --drop-burst 300`, same stream: `frames=6203 packets=6204 gaps=2400` **and the guard fires** — `warning: ... jumped implausibly far ahead of the previous packet (1.299..1.29999 against a previous 0.99899)`, then `timing samples by arrival instead`. 300 packet spans is past `kGlitchPackets` (256), so real loss is read as a clock glitch; and since each bridge advances the anchor by one packet span while the true timeline is 300 ahead, the next four packets fail too and `kFailureLimit` throws the stream onto the arrival clock. A burst between 256 packets and the 2 s `silence_timeout_s` is the only window in which this is reachable, and it is a finding, not a baseline. Keep a burst under 2 s of wire time whatever the rate: past that the client reconnects, the mock resets its counter and clock on the new CONNECT, and the run reads `gaps=0` with a *backwards* stamp warning (`does not advance across packets ... 0..0.0099 against a previous 0.3899`) — the reconnect path, not the loss path. **Run one self-check at a time.** Two concurrent `xvfb-run -a` instances end up on the same display and the second gets no GL context — it dumps `layout over 0 frames` and `valid=-1`, which looks like a pipeline failure and is not one. `tools/mock_udpstreamer.py` also outlives its `--duration`, so kill it after each run or the leftovers pile up and hold ports. The probes are the regression net — when changing UI or plot-host behaviour, extend the probe/dump in `src/ui.cpp` rather than adding a separate harness.