udpscope/docs/claude/verification.md

Verification

How to run the headless self-check, the two baselines every run is read against, and the environment variables that schedule each probe.

Part of CLAUDE.md.

There is no unit test suite. The only automated check is the app's own headless self-check, driven by env vars and run under a display (xvfb-run, llvmpipe) against a live UDPStreamer source:

xvfb-run -a env UDPSCOPE_EXIT_AFTER_MS=8000 UDPSCOPE_ENABLE_ALL_MS=1000 \
  ./build/udpscope --server 127.0.0.1 --port 8000

Two baselines every run is read against. A clean build emits four warnings and no more: two -Wmissing-field-initializers on MathNode::source and -Woverlength-strings on both generated GResource .c files — the help pages' and the icons', whose blob crossed C99's 4095-byte string-literal limit when the two app icons joined the four action ones (a note about a generated file, which GCC itself supports). The fifth warning that used to appear here was an unused output_built in math_probe(); it was deleted as dead code — redundant with the output_after_add already printed on the same line — and that deletion is what dropped the baseline from five to four. Nothing started reading it. And an exit-dump run emits no Gtk-CRITICAL at all. A fifth warning, or any critical, is a finding.

An incremental build reads two of those four: the generated GResource .c files are not recompiled when only src/ changed, so their -Woverlength-strings do not reappear.

That second baseline used to be "exactly two, gtk_widget_get_height and gtk_widget_get_width, from the dump asking a destroyed widget for its size", and tolerating them was a mistake: the dump runs from the destroy handler, so the widget pointer is freed memory, and a critical is only the lucky reading of it. A run that had built and destroyed other windows first — UDPSCOPE_PROBE_HELP_MS, which opens three — has had that memory reused, and the same two calls then SIGSEGV in the middle of the dump, which reads as a truncated dump rather than as a crash in a debug line. Every size the dump prints is therefore captured on the tick, while the window is alive (App::dbg_*), and no gtk_widget_* call may appear in the dump — the panel controls: line was already written that way for the same reason, and that comment is the precedent, not an exception.

• UDPSCOPE_EXIT_AFTER_MS — run the trigger-on → go-live probe, close the window, then dump the whole pipeline state to stdout (frame/sample counts, per-plot painted frames, trace bindings, panel sizes, status line in both states).

• UDPSCOPE_ENABLE_ALL_MS, UDPSCOPE_ARM_TRIGGER_MS, UDPSCOPE_PROBE_DISARM_MS, UDPSCOPE_TRIG_TRACE_MS — schedule individual probes. UDPSCOPE_PROBE_DISARM_MS only starts its sequence: it switches the trigger on and then polls for a capture to open, so the Disarm lands ~300 ms into one rather than wherever a fixed delay happened to fall.

• UDPSCOPE_PROBE_TRIGRENUM_MS — the trigger's capture must survive a renumbering of its own channel. It sets the reported geometry through the real widgets (1 s window, 10 % pre share, hold-off from --holdoff) on the mock's square alone, polls for a capture to open, and then enables the sine — which sorts before the square and so moves the trigger channel's index under the engine. Its three lines are in probe-readings.md; t0_held=1 is the assertion. Run it with --holdoff 10, which makes the stolen-t0 reading unmistakable: a second latch barely a second after the first cannot be a hold-off that applied.

• UDPSCOPE_PROBE_FIELDS_MS — change the trigger panel's edge and Normal/Single dropdowns. As much a crash test as a state check (see the GTK signal-signature invariant below), so time it inside a capture: [fields-probe] ... capturing=1 followed by no SIGSEGV is the assertion.

• UDPSCOPE_PROBE_WINDOW_MS — drive the window control every way a user can: type a number, change the unit under it, step a decade either way.

• UDPSCOPE_PROBE_LAYOUT_MS — drive the layout editor: switch it on, insert above the first row, remove a row, append one at the bottom, switch it off. Its steps are 200 ms apart on purpose — the buttons are placed from the rects the last frame was laid out with, so a synchronous sequence would read every position one action stale. It clicks by emitting ::clicked, never gtk_widget_activate(), which holds a GtkButton for its 250 ms ACTIVATE_TIMEOUT and then emits into a button the rebuild has already destroyed.

• [measure-probe] N measurements on 'X' (stderr, UDPSCOPE_PROBE_MEASURE_MS) — adds one measurement of each statistical type (average, top, peak to peak, amplitude) on the first channel and forces a recompute. It must name a channel, not signals().front(): the first signal of a stream is its timestamp, which is not acquired, and naming it adds four measurements that all read -- and makes a performance run look free (each bails before collecting anything -- that is how the first baseline of the stall below read 0.4 ms). The values are in the exit dump as measurements: <type>=<value>; on the fast mock (tools/mock_udpstreamer.py --rate 1300000 --batch 2000) a unit sine must read peak to peak=2, amplitude=1, average=0 and top=0.636620 -- 2/pi, the half-wave mean of a sine, which is the check that the threshold types are not merely plausible. [measure-trace] view=[..] <type>=<value>... marks=N level:V/time:V... is a repeating line (250 ms) from the probe: the pane's numbers beside the current window and the marks actually placed, which is the only way a stale result or a mark that should not be drawn shows up as a number. It must never print marks=0 while measurements with show_on_plot are on and the data is there -- that was the rulers blinking off at a mode change. measure cost: N recomputes, X ms total, Y ms each, data D ms (P%), covered=C buckets=B raw=R (F%), N ns/covered sample follows it: read it against a run with UDPSCOPE_PROBE_MEASURE_MS unset, because a frame mean cannot attribute a cost. The last three fields are the assertion, and they are a ratio, not a time: covered is the samples the window spans, buckets the LOD buckets the two walks answered from, and raw the samples they still had to read. raw must stay at a fraction of a percent of covered on a clean waveform over a window of millions -- 0.1% is what the pyramid-backed walks give (see the measurement-cost invariant in invariants-measurements.md). A raw near covered means the walks went back to reading the window, which at 1 MS/s is 19.7 ms per recompute and a third of the run's frames. It is legitimately higher for a window shorter than a bucket (64 samples) or a signal noisy enough to cross a threshold everywhere, so read it on the fast mock (tools/mock_udpstreamer.py --rate 1300000 --batch 2000), where a window is ~2 M samples. --noise 0.1 there reads 2.0% at 1.96 ms a recompute against the clean 0.1% at 1.41 ms, and that is the interferer's own cost rather than a regression -- the figure to compare a change against is the clean one. Note the final measurements: line and [measure-trace]'s last lines may read --: UDPSCOPE_EXIT_AFTER_MS arms the trigger before it dumps, so the dump can land in Waiting with nothing bound and the axis at its default. The race-free reading is an earlier [measure-trace] line, which carries its own view=; those lines are also where the level-mark fallback is visible (an invalid pass must still print its level: marks).

• [pan-probe] asked=[..] was=[..], [pan-probe] after=[..] held=D asked=[..], then [pan-probe] zoom asked=[..] after=[..] span=S window=W limited=L (stderr, UDPSCOPE_PROBE_PAN_MS) — held must be 1: a gesture must not move a window that is pinned (trigger mode, or a paused view). The probe writes the panned range the way a gesture does — on the member axis, which Figure::layout() adopts into the group — and reports the axis the frame after it produced. Time it inside one pinned state: a pan during a capture whose check lands after the capture has ended is re-framed by the mode change, not by the pin, and reads held=1 with the hold removed (that is how the first version of this probe fooled me). The pause probe makes a stable pin, so run them together: UDPSCOPE_PROBE_PAUSE_MS=1500 UDPSCOPE_PROBE_PAN_MS=2200 puts the pan and its check inside the pause. With hold_pinned_window() disabled it reads held=0 and the axis sits at the asked range. It then asks for a zoom-in (a window a quarter of what is on screen) and reports group=/member= beside it: allowed must be 1, with both equal to the ask -- that is the single-trigger-mode case, where the wheel has to keep working, and where reading the wrong one of the two made a working clamp look broken. All three verdicts are read from the tick, one frame after the ask: a g_timeout was the first version and it drifted past the paused/resumed boundary under load. The probe then asks for a zoom-out ten times the current span and reports the span the next frame produced: limited must be 1, i.e. span <= window. Run it with a ring that holds more than the window (--buffer-mb 64 past the window's own seconds) — that is the state the old max(retained, window_s_) bound leaked in, and it showed up here as span=2.502550 window=2.000000 limited=0 against span=2.000000 after the fix.

• [zoom-probe] <sweep>/x<step> axis=[..] span=.. n=N idx=[a..b) spp=.. level=L pts=P kept=K back=.. ink=[..] x_at_first=.. gap_px=.. end_px=.. rect=[..] (stderr, UDPSCOPE_PROBE_EVENTZOOM_MS) — arms the trigger in single mode, polls for a held event (a latch happens when the stream says so; a sweep on a timer has nothing to zoom into), then zooms in eight doublings twice: ges/ writes the row's own axis the way a gesture does, win/ drives the window control (apply_time_window, which unpins and re-arms the live follow before the pin is restated). Each step's reading is taken in the tick, after that frame's pin and clamp — a g_timeout would report the ask. It recomputes what Renderer::draw_traces is handed, through the same calls: neighbours() for idx, the 64-spp gate for level, then gather(). gap_px and end_px must stay within a pixel or two at every step — they are how much of the left and right margin no geometry reaches, and a gap_px that doubles per step is the "zooming into a held event clips it more and more from the left" defect (374 px of a 716 px axis by ges/x128), whose cause is a constant offset in data seconds (see the neighbours() invariant in invariants-rendering.md). kept/back are the renderer's own increasing-suffix filter counted rather than applied: kept far below pts is a staircase, and a non-zero back is a non-monotonic gather. Run it with --clock-jitter (tools/mock_udpstreamer.py --rate 1300000 --batch 2000 --clock-jitter 0.4): the ideal mock's stamps sit on a perfect grid, which is exactly what hides this class of bug, and three runs against it reported gap_px=0.0 throughout while the defect was live. The jitter varies each packet's spacing rather than offsetting its start, because the stream's clock has to stay strictly increasing. The win/ sweep bottoms out at span=0.05 (RowSet::set_time_window()'s floor) and walks to the event's post side, since set_time_window() frames the newest span before the pin clamps it — both are expected.

• UDPSCOPE_PROBE_PAUSE_MS — pause acquisition mid-run so layout numbers are measured on a still picture. It now also asserts what a pause is ([pause-probe] <action> paused=D ring_produced=N event_size=M frozen_x=[..] axis=[..] glance=[..] on stderr): it presses the button, not Acquisition::set_paused(), because pausing is a display state as well as an acquisition one and the view snapshot and axis pin live in the handler. Across the three lines ring_produced must grow (the producer never stops — it fills the live ring exactly as it does with the trigger off), event_size must be non-zero exactly while paused (the plot is bound to the frozen snapshot, not the ring) and frozen_x must not move between the pause and the resume; after the resume event_size is 0 and the window is back on the newest data. Pausing used to stop the producer, which is why the counts grew by nothing and the resume left a hole. frozen_x is the rebased snapshot — the pause view's own stamps are relative to the freeze instant now, so the range reads [-window..0]-ish rather than stream seconds, and a frozen_x in the thousands is the rebase gone. glance= is the assertion for the paused status field, and this is the only line that can make it: the exit dump's status bar: is read live and its two mode … lines are the live and triggered states. While paused it must read paused · <rate> · <window> · for <elapsed>, with the elapsed figure growing between the two paused lines and absent on the resume line. It is wall time (g_get_monotonic_time()), not stream time, because the producer never stops while paused and "how long have I been looking at this" is a question about the user, not about the stream; it is written through format_elapsed() so the field steps to a coarser unit instead of growing digits (see the layout rule).

• UDPSCOPE_PROBE_COLOR_MS — recolour one plotted signal and read the colour back out of the trace and the chip, round-trip it through the session writer, then clear it. Schedule UDPSCOPE_ENABLE_ALL_MS before it: with nothing bound it has no trace to recolour and asserts nothing. Its four lines are in probe-readings.md.

• UDPSCOPE_EXPORT_MS (+ optional UDPSCOPE_EXPORT_PATH, default /tmp/udpscope-export) — collect the export table and write it both ways, skipping only the file chooser.

• UDPSCOPE_PROBE_CONDITION_MS — arms a qualified trigger against the mock and reports the latched t0, the measured pulse or band width, and sample values for the window condition. Schedules its own arm and report stages. UDPSCOPE_PROBE_CONDITION_SHAPE picks the leg: 0 pulse width < (the glitch, the default, needs --glitch-at/--glitch-width), 1 pulse width > (uses a 1.0 s window and 50 % pre so the opening edge of a 0.2 s pulse is captured), 2 window (uses sine with a 0.2 s window and 10 % pre), 3 dropout (needs --quiet-at; uses sine with a 1.0 s window and 50 % pre so the last crossing before the outage is captured).

• UDPSCOPE_PROBE_STATS_MS — reports every measurement's accumulator (n, mean, sd, lo, hi), pauses, reports again with dn= per measurement, then resets and reports a third time. Three phases: phase=0, phase=1 (after pause) and phase=2 (after Reset). Needs UDPSCOPE_ENABLE_ALL_MS ahead of it and UDPSCOPE_PROBE_MEASURE_MS to add measurements first.

• UDPSCOPE_PROBE_SEARCH_MS — runs a buffer search for the armed condition, cross-checks the live trigger's own t0 against the nearest hit (agree=1 is the load-bearing reading), and jumps to hit 0. Reports the resulting mode=, axis= and marks placed=.

The following mock waveform features are what the qualified-condition probes are checked against, as distinct from the fault injection below, which exercises ingest:

• --glitch-at S / --glitch-width W invert the square signal for W seconds at the start of every square period from stream second S onwards — a runt pulse of a known width, which is what a < pulse-width trigger and a glitch search are checked against.

• --quiet-at S freezes sine at its held value from stream second S onwards, so it stops crossing zero: the dropout trigger's input.

• UDPSCOPE_PROBE_RESEQ_MS — one mid-run reading of the resequencing path. Four cases, run each as its own process — four back to back starve the mock past its 2 s silence timeout and every case then reads a reconnect instead of what it was testing. Common conditions for all four: the mock at --rate 10000 --batch 100 --freq 5, udpscope with no extra options, under xvfb-run with UDPSCOPE_EXIT_AFTER_MS=8000 UDPSCOPE_ENABLE_ALL_MS=600 UDPSCOPE_EXIT_DUMP=1 UDPSCOPE_PROBE_RESEQ_MS=5000.

  case	mock extra flags	key reading
  base	(none)	every counter 0; proves the machinery costs nothing
  reorder	--reorder-every 10	reseq=51 (mid-run) / 102 (dump); holes=0; resequencing inside the window leaves no reservation
  late	--reorder-every 10 --reorder-delay-ms 50	holes=42 bands=42 bands_repaired=42 nan_slots=0; every band is the faint, repaired tier
  mixed	--drop-every 20 --reorder-every 10 --reorder-delay-ms 50	bands=42 bands_repaired=20 nan_slots=2300; all three band states at once

  Full measured readings (probe line first, then exit-dump; bands_repaired was added after the original probe and any earlier reading without it is obsolete):

  === base   mock: none
  [reseq-probe] reseq=0 late=0 stale=0 holes=0 hole_slots=0 backfilled=0 open_holes=0 bands=0
                bands_repaired=0 nan_slots=0 scanned=44000 trigger_in_hole=0
  [udpscope] frames=1024 packets=1025 gaps=0 channels=4 span=838.861s stride=1 reseq=0 late=0
             stale=0 holes=0 hole_slots=0 backfilled=0
  [udpscope]   channel 0 'sine' stored=95200 published=95000
  
  === reorder   mock: --reorder-every 10
  [reseq-probe] reseq=51 late=0 stale=0 holes=0 hole_slots=0 backfilled=0 open_holes=0 bands=0
                bands_repaired=0 nan_slots=0 scanned=44000 trigger_in_hole=0
  [udpscope] frames=1028 packets=1029 gaps=0 channels=4 span=838.861s stride=1 reseq=102 late=0
             stale=0 holes=0 hole_slots=0 backfilled=0
  [udpscope]   channel 0 'sine' stored=95400 published=95100
  
  === late   mock: --reorder-every 10 --reorder-delay-ms 50
  [reseq-probe] reseq=142 late=48 stale=0 holes=42 hole_slots=4200 backfilled=16800 open_holes=42
                bands=42 bands_repaired=42 nan_slots=0 scanned=44000 trigger_in_hole=0
  [udpscope] frames=1026 packets=1027 gaps=100 channels=4 span=838.861s stride=1 reseq=288
             late=100 stale=0 holes=94 hole_slots=9400 backfilled=37600
  [udpscope]   channel 0 'sine' stored=95700 published=95500
  
  === mixed   mock: --drop-every 20 --reorder-every 10 --reorder-delay-ms 50
  [reseq-probe] reseq=163 late=24 stale=0 holes=43 hole_slots=4300 backfilled=8000 open_holes=43
                bands=42 bands_repaired=20 nan_slots=2300 scanned=44100 trigger_in_hole=0
  [udpscope] frames=974 packets=975 gaps=101 channels=4 span=838.861s stride=1 reseq=314 late=50
             stale=0 holes=94 hole_slots=9400 backfilled=18400
  [udpscope]   channel 0 'sine' stored=95100 published=94900
  

  The mixed case's open_holes=43 against bands=42 is correct: the 43rd hole has scrolled out of the visible window, and build_invalid_spans() drops a span outside the window rather than pinning it to the edge. The per-field reference is in probe-readings.md.

The mock also carries three fault injections that no probe schedules, because what they exercise is the ingest rather than the UI. The first: --time-glitch-at <stream seconds> --time-glitch-by <seconds> makes one packet's time signal jump ahead and then carries on from where the true timeline was, which is what a device whose clock glitches puts on the wire. Run it with the trigger armed, e.g.

xvfb-run -a env UDPSCOPE_ENABLE_ALL_MS=400 UDPSCOPE_ARM_TRIGGER_MS=700 UDPSCOPE_EXIT_AFTER_MS=22000 \
  ./build/udpscope --server 127.0.0.1 --port 8000 --holdoff 10
# mock: --rate 10000 --batch 100 --freq 5 --time-glitch-at 6.0 --time-glitch-by 10.0

Four readings, all off the once-a-second [udpscope] trig engine: line:

• exactly one warning: ... jumped implausibly far ahead of the previous packet (16.01..16.0199 against a previous 6.0099); bridging that packet ... — the glitch was caught;
• no timing samples by arrival instead — one bad packet must not change the time source;
• now_t crosses the glitch continuously (5.040 → 6.050 → 7.050), and history keeps growing (1.01 … 23.15) instead of collapsing — the ring was not wiped;
• the latches stay a hold-off apart: edge at 1.060 / latched at 2.060 / holdoff_until=12.060, then edge at 12.260 / latched at 13.260 / holdoff_until=23.260 — 11.2 s for a 10 s hold-off plus a 0.9 s post window.

Before the fix the same run read a fallback warning, history collapsing 4.03 → 0.95, now_t jumping 5.040 → 16.040, and two latches 1.2 s apart. Note the window here comes from the saved session for that port; the figures above are the default 2 s window (50 % pre), so move ~/.config/udpscope/<host>_<port>.toml aside to reproduce them.

The other two are packet loss (--drop-every N, --drop-burst K) and wire inversion (--reorder-every N). Loss withholds a datagram the counter and the stream clock have already advanced over; the client counts the forward jump as counter_gaps. An inversion withholds a packet until the one behind it is sent, so the receiver sees a later counter first; a packet that arrives within the 20 ms reorder window is resequenced with no hole, and one that arrives past it leaves a committed hole that it then backfills (UDPSCOPE_PROBE_RESEQ_MS exercises both paths). There is no interpolation anywhere below the ring.

All of the readings below are off UDPSCOPE_EXIT_DUMP's [udpscope] frames= packets= gaps= line, over an 8 s run against --rate 10000 --batch 100 --freq 5 (100 packets a second) except where the burst needs the faster stream. gaps is the exact reading and repeats to the packet; frames and packets move by one or two with the run's own timing. Run each of these on its own — four of them back to back in one script starved the mock past the 2 s silence timeout and every case, including the clean one, then read a reconnect instead of what it was testing:

• clean, as the reference: frames=1029 packets=1030 gaps=0, no warning.
• --drop-every 10: frames=925 packets=926 gaps=102. Every tenth packet never reaches the wire, so packets falls with frames and gaps is the count of them. No warning: a one-packet hole is a one-packet jump in the stamps, far inside the glitch guard.
• --reorder-every 10: frames=1028 packets=1029 gaps=0. The inverted packet lands inside the 20 ms reorder window and is resequenced: every datagram arrives and gaps is 0. The pre-resequencing signature was frames=925 packets=1028 gaps=102; that is now the reading for --reorder-every 10 --reorder-delay-ms 50, which forces a late arrival past the window.
• --drop-every 1000 --drop-burst 200 at --rate 100000 --batch 100 (1000 packets a second): frames=7000 packets=7001 gaps=1600, eight bursts of 200, and still no warning.
• --drop-every 1000 --drop-burst 300, same stream: frames=6203 packets=6204 gaps=2400 and the guard fires — warning: ... jumped implausibly far ahead of the previous packet (1.299..1.29999 against a previous 0.99899), then timing samples by arrival instead. 300 packet spans is past kGlitchPackets (256), so real loss is read as a clock glitch; and since each bridge advances the anchor by one packet span while the true timeline is 300 ahead, the next four packets fail too and kFailureLimit throws the stream onto the arrival clock. A burst between 256 packets and the 2 s silence_timeout_s is the only window in which this is reachable, and it is a finding, not a baseline.

Keep a burst under 2 s of wire time whatever the rate: past that the client reconnects, the mock resets its counter and clock on the new CONNECT, and the run reads gaps=0 with a backwards stamp warning (does not advance across packets ... 0..0.0099 against a previous 0.3899) — the reconnect path, not the loss path.

Run one self-check at a time. Two concurrent xvfb-run -a instances end up on the same display and the second gets no GL context — it dumps layout over 0 frames and valid=-1, which looks like a pipeline failure and is not one. tools/mock_udpstreamer.py also outlives its --duration, so kill it after each run or the leftovers pile up and hold ports.

The probes are the regression net — when changing UI or plot-host behaviour, extend the probe/dump in src/ui.cpp rather than adding a separate harness.