fix: a claimed gap must be corroborated before it is believed, and its hole must fit
The user's real host reported "connected and live" with one missing-data band over the whole window
and no trace in it. The counter is the only thing that claims a packet existed, and a reservation
was booked on its word alone: a source whose DATA counter counts samples, or frames of an inner
layer, advances by more than 1 per datagram, the client cannot check that claim (it has no clock of
its own to compare against), so it holds each packet, saturates its reorder ring, delivers it in
order and reports the skipped counters as lost. Every frame therefore reported a phantom gap, and
`reserve_hole()` wrote `lost * batch_` NaN samples with no bound but the 256-PACKET cap -- the
sample form of that rule ("the number to weigh against the ring is 256 * batch_, not the 256") was
never implemented. With relative time on -- which that session carries -- each phantom gap is
materialised as NaN slots by construction: the window filled with reservations, there was never
anything missing to repair them, and the gaps also stretched the relative clock, which is why the
status bar read live at ten times the true rate.
`Acquisition::gap_plausible()` now gates every reservation on two independent tests: the arrival
clock must corroborate the claim (that many missing packets would take that many frame intervals;
deliberately permissive, because a false refusal costs a shading and never data), and the ring must
be able to hold the hole. The verdict is taken before the stamps, and relative mode's slot advance
is skipped with it, since the advance and the reservation are the same claim. Refusals are counted
in `StreamStats::gaps_refused` and said once per connection with the numbers, naming the wire shape
they describe.
tools/mock_udpstreamer.py gains `--counter-step N`: a counter that is not a packet counter on an
intact link, which is the one shape the mock could not produce and the shape the report came from.
Verified in both directions. A counter stepping 1000 per datagram reads `holes=0 hole_slots=0
backfilled=0` with the traces present (3 traces, 1.4 M samples) against 7.9 M claimed gaps, and the
warning above names the cause. Every other packet withheld -- a real loss -- still reads
`holes=3579`, so genuine gaps are still reserved: a fix that stopped believing real losses would
have traded one wrong picture for another. The user's control read as "the math channel is the
problem" and was confounded -- disabling it is a structural change, which rebuilds the buffer and
clears the accumulated NaN.
This commit is contained in:
1 parent
22e7a8efcc
commit
27c9b34277
9 files changed
+249
-15
No files matched your search
@@ -169,7 +169,36 @@ Part of [CLAUDE.md](../../CLAUDE.md). These are rules that cost a defect to lear
|
||||
(`kGlitchPackets`, 256) is counted in **packets, not samples**, so one hole's sample cost
|
||||
is `256 × batch_` and scales with the stream's batch — 25 600 samples at a batch of 100,
|
||||
2.56 M at a batch of 10 000 — which is the number to weigh against the ring, not the 256.
|
||||
- **A re-plan drops every hole and restarts the producer's tick.** `rebuild_locked()` replaces
|
||||
- **A reservation is weighed against the wire, and never booked on the counter's word alone.**
|
||||
The counter is a *claim* about the stream; the arrival clock is a *measurement* of it, and only
|
||||
the two together justify writing NaN into the ring (`Acquisition::gap_plausible()`, called
|
||||
before anything is booked *and* before relative mode's `slot_base_` advance, because the advance
|
||||
and the reservation are the same claim). Two tests, either one refusing is enough:
|
||||
- **The arrival clock.** `missing` packets would have taken `missing` frame intervals to arrive,
|
||||
so a gap whose elapsed arrival time is under half of that is refused. Half and not one because
|
||||
the socket clock jitters and a drain can compress two frames' arrival times -- and a refused
|
||||
gap costs a *shading*, never data, so the test is deliberately permissive.
|
||||
- **The ring.** `missing × batch_` samples become `missing × batch_ / time_stride` slots, and the
|
||||
hole may not lap `time_capacity`. This is the *sample* form of the rule the 256-packet cap
|
||||
states in packets -- the cap's own comment says 256 packet spans in one go is an outage, and
|
||||
the invariant above says the number to weigh against the ring is `256 × batch_` and not the
|
||||
256 -- and it has to exist in samples because a hole that laps the ring erases the history it
|
||||
is meant to annotate, can never be backfilled (its slots are already overwritten) and is never
|
||||
forgotten (`forget_holes_before()` waits for slots the ring has passed), so it leaves a
|
||||
*permanent* band over a buffer that is all NaN.
|
||||
What its absence looks like, which is why it is written down: a device whose counter counts
|
||||
*samples*, or frames of an inner layer, reports a forward jump on every datagram while the wire
|
||||
delivers on time. Every frame then reserves, and with `relative_time` every frame books NaN
|
||||
slots by construction -- one band over the whole window with the samples washed out,
|
||||
`backfilled=0` (no packet was ever missing, so nothing can repair it), and the status bar reading
|
||||
`live` at the right rate the whole time. The refusal is counted in `StreamStats::gaps_refused`
|
||||
and said once per connection with the numbers, because the status bar's `gaps` climbs either way
|
||||
and the state is otherwise invisible; `tools/mock_udpstreamer.py --counter-step N` is the mock
|
||||
shape (its counter is otherwise always +1, so this is the one wire shape it could not produce),
|
||||
and `gaps_refused` rising while `nan_slots` stays 0 is the reading. A refused gap is not a silent
|
||||
one: the timeline is not stretched over loss that did not happen, so it shows as the x
|
||||
discontinuity it is.
|
||||
- **A re-plan drops every hole and restarts the producer's tick.** `rebuild_locked()` replaces
|
||||
the store, so `produced()` restarts at 0 and a hole's `slot_begin`/`tick_begin`/`x_begin`
|
||||
describe a buffer that is gone; it therefore clears `holes_` and bumps
|
||||
`ScopeModel::plan_generation()`, which the producer compares against the value it last saw
|
||||
|
||||
@@ -872,10 +872,11 @@ two also catch a row whose accelerator went missing. `settings=` must be a real
|
||||
The `m`/`g` pairs are the *tab name* before and after, which is what says those two keys select a tab rather than toggling a pane; `t`/`c` read the toolbar toggle's own state, because those two enable a feature and the button is where that state is visible. The refusal is the whole reason the controller is in the bubble phase: every bare-letter accel is also a character, so an accel that fires while a `GtkEditable` has focus makes the window entry unusable. A pair reading `0->0` is an action whose dispatch does not reach the handler a click reaches. Note the format is `<key>: <widget-description>: <before>-><after>` — the name passed to `before_after()` includes the widget's description, so `t: trigger panel: 0->1` is the correct form (not `t: trigger panel 0->1` without the colon before the values).
|
||||
- `[color] '<name>' at row R slot S: trace=#RRGGBB chip_agrees=1 (want 1)`, then `[color] override: trace=#123456 chip_agrees=1`, `[color] session: entries=1 rgb=#123456` and `[color] default: trace=#RRGGBB chip_agrees=1` (stderr, `UDPSCOPE_PROBE_COLOR_MS`) — **all three `chip_agrees` must be 1**, the override line's `trace` must be `#123456`, the session line must read `entries=1 rgb=#123456`, and the default line's `trace` must be back to the first line's. Run it with `UDPSCOPE_ENABLE_ALL_MS` ahead of it, or there is no bound trace and the probe prints `no bound trace to recolour` instead of asserting anything; on the standard mock the first line reads `'sine' at row 0 slot 0: trace=#4FA3FF`. Two things make this an assertion rather than a tautology. The colours are read out of **`plot::Plot::trace().color`** — what the renderer will stroke, reached through `Row::plot` *after* `relayout_areas()`, since a new colour is a new figure — and out of the chip's own `<span foreground="…">` markup, not by asking `RowSet::signal_color()` twice, which would pass against a chip that never reached the trace. And the override is `#123456`, deliberately in no palette slot: an override equal to the entry it replaces passes against an implementation that ignores overrides entirely. The session leg round-trips `collect_layout()` → `save_layout()` → `load_layout()` on a temp path and deletes it, because a probe must not leave a colour in the real session file for a later run to inherit.
|
||||
- `[reseq-probe] reseq=N late=N stale=N overflows=N holes=N hole_slots=N backfilled=N
|
||||
open_holes=N bands=N bands_repaired=N nan_slots=N scanned=N trigger_in_hole=D` (stderr,
|
||||
gaps_refused=N open_holes=N bands=N bands_repaired=N nan_slots=N scanned=N trigger_in_hole=D`
|
||||
(stderr,
|
||||
`UDPSCOPE_PROBE_RESEQ_MS`) —
|
||||
the resequencing path's mid-run state; the measured readings and the per-case command are in
|
||||
[verification.md](verification.md). Eight things to read carefully:
|
||||
[verification.md](verification.md). Nine things to read carefully:
|
||||
|
||||
1. **`backfilled` and `hole_slots` are not comparable directly.** `hole_slots` counts
|
||||
*time-ring* slots once per hole; `backfilled` counts *value-ring* slots repaired and
|
||||
@@ -941,6 +942,29 @@ two also catch a row whose accelerator went missing. `settings=` must be a real
|
||||
slots. A non-zero reading beside a healthy `backfilled` is a note about the ring's size,
|
||||
not a defect.
|
||||
|
||||
9. **`gaps_refused` counts reservations the wire contradicted, and it is the field that reads
|
||||
a counter which is not counting packets.** A gap is booked only if it passes two tests
|
||||
(`Acquisition::gap_plausible()`): the *arrival clock* has to corroborate the elapsed time
|
||||
(`missing` packets would have taken `missing` frame intervals; anything under half of that
|
||||
is refused) and the *ring* has to hold the hole (`missing * batch_` samples must not lap
|
||||
`time_capacity`). The shape this exists for is a device whose counter advances by samples,
|
||||
or by frames of an inner layer: every datagram reports a forward jump while the wire
|
||||
delivers on time, so *every* frame reserves -- and, with `relative_time`, every frame
|
||||
books NaN slots, which is a band over the whole window with the samples washed out and no
|
||||
packet ever missing to backfill it. `tools/mock_udpstreamer.py --counter-step N` produces
|
||||
exactly that on an intact link. The reading's two sides, both measured with
|
||||
`--relative-time` and no session: a *fixed* run reads
|
||||
`holes=1 hole_slots=900 gaps_refused=419 open_holes=0 bands=0 nan_slots=0 scanned=21700`
|
||||
with the data channels full and a once-per-connection stderr warning naming the numbers,
|
||||
while the unfixed one read `holes=420 hole_slots=378000 open_holes=64 bands=64
|
||||
bands_repaired=0 nan_slots=198000 scanned=220000` -- `nan_slots` at 90% of `scanned`, which
|
||||
is the whole-window band. The refusal is **not relative-mode-only**: the same command without
|
||||
`--relative-time` reads `holes=1 gaps_refused=419 nan_slots=0 scanned=21800`, because a
|
||||
counter that lies is not a mode. On the honest shapes
|
||||
(`--drop-every`, `--drop-burst 200`, `--reorder-delay-ms`) this field must stay **0**:
|
||||
a non-zero `gaps_refused` there means the arrival test is refusing real loss, and the cost
|
||||
is a gap with no shading (never lost data).
|
||||
|
||||
- `[zoomkeys] ctrl+plus x=[..]->[..] y_held=1 | alt+plus y=[..]->[..] x_held=1` (stderr, `UDPSCOPE_PROBE_ZOOMKEYS_MS`) — **both `_held` must be 1 and both zooms must still happen.** Run it inside a pinned state, where a frame cannot re-frame the other axis for its own reasons: `UDPSCOPE_PROBE_PAUSE_MS=1500 UDPSCOPE_PROBE_ZOOMKEYS_MS=2200`. A `_held=0` is the modifier not reaching the plotter's controller; an unchanged range on the *zoomed* axis is the opposite failure, the key doing nothing. Ctrl restricts to x and Alt to y; Ctrl+Alt zooms neither (no special case needed — neither flag alone fires).
|
||||
- `status bar: glance=[● live · 1.09 MS/s · 2 s window] max_glance_chars=53 detail_fields=11 popover_rebuilds=0` (exit dump) — **`max_glance_chars` must stay small** (under about 60) and **`popover_rebuilds` must be 0**. The status row is directly above the plot, so this is the "layout must not depend on content" rule read from the bar's own side: the state field is one word from a fixed set (`connecting`, `live`, `paused`, `armed`, `capturing`, `event held`, `disarmed` — a word added to it must be **no wider than the widest already there**, which is what let `connecting` join at ten characters beside `event held`; the dead-port glance reads `max_glance_chars=51` against a live run's 54), the rate field is `%.3g` (so `10 kS/s`, never `10.0 kS/s`), and the two fields a *state* adds are both elapsed times — trigger mode's `last event … ago` and a pause's `for <elapsed>` — which step units rather than growing digits (`format_elapsed()`). A **paused** glance is not in the dump at all: it is read by `[pause-probe]`'s `glance=` field, since the dump's own line is taken live. `detail_fields` is the popover's label count, a literal, so adding a label without thinking about layout shows up here. The glance rate format matters: `%.3g` drops the trailing zero that a `%.4g` or a fixed-precision format would keep — a rate that reads differently in the dump and in the help page is a format disagreement, and `14-status.txt`'s example must match it.
|
||||
|
||||
|
||||
@@ -590,7 +590,7 @@ as distinct from the fault injection below, which exercises ingest:
|
||||
at 64. The `mixed` case's `open_holes=43` against `bands=43` has no such spread because
|
||||
nothing re-planned. The per-field reference is in [probe-readings.md](probe-readings.md).
|
||||
|
||||
The mock also carries three **fault injections** that no probe schedules, because what they exercise
|
||||
The mock also carries four **fault injections** that no probe schedules, because what they exercise
|
||||
is the ingest rather than the UI. The first: `--time-glitch-at <stream seconds> --time-glitch-by <seconds>` makes
|
||||
**one** packet's time signal jump ahead and then carries on from where the true timeline was, which
|
||||
is what a device whose clock glitches puts on the wire. Run it with the trigger armed, e.g.
|
||||
@@ -625,6 +625,41 @@ arrives within the 20 ms reorder window is resequenced with no hole, and one tha
|
||||
leaves a committed hole that it then backfills (`UDPSCOPE_PROBE_RESEQ_MS` exercises both paths).
|
||||
There is **no interpolation** anywhere below the ring.
|
||||
|
||||
The fourth is the counter itself: **`--counter-step N`** advances the header counter by `N` per
|
||||
datagram instead of one -- a device whose counter counts *samples*, or frames of an inner layer,
|
||||
rather than packets. Nothing is withheld and every datagram goes out on schedule, so the link is
|
||||
intact and the status bar is right to say `live`; what is wrong is the counter's claim, which the
|
||||
client cannot check (it has no clock) and only the app can (it has the arrival times). It is the
|
||||
one wire shape the mock could not otherwise produce, and the run that reads it is the reseq probe
|
||||
with relative time on (the mode that frames a counter gap as NaN by construction):
|
||||
|
||||
```sh
|
||||
# mock: --counter-step 10
|
||||
xvfb-run -a env UDPSCOPE_EXIT_AFTER_MS=8000 UDPSCOPE_ENABLE_ALL_MS=600 \
|
||||
UDPSCOPE_PROBE_RESEQ_MS=5000 UDPSCOPE_EXIT_DUMP=1 \
|
||||
./build/udpscope --server 127.0.0.1 --port 8000 --relative-time
|
||||
```
|
||||
|
||||
```
|
||||
# fixed: [reseq-probe] … holes=1 hole_slots=900 backfilled=0 gaps_refused=419 open_holes=0
|
||||
# bands=0 bands_repaired=0 nan_slots=0 scanned=21700 trigger_in_hole=0
|
||||
# status bar: glance=[● live · 9.97 kS/s · 2 s window]
|
||||
# broken: [reseq-probe] … holes=420 hole_slots=378000 backfilled=0 open_holes=64 bands=64
|
||||
# bands_repaired=0 nan_slots=198000 scanned=220000 trigger_in_hole=0
|
||||
# status bar: glance=[● live · 101 kS/s · 2 s window]
|
||||
```
|
||||
|
||||
**`gaps_refused` above 0 with `nan_slots` at 0 is the assertion** — the gap was refused and the
|
||||
samples were kept. The broken reading is `nan_slots` at 90% of `scanned` with `backfilled=0`: a
|
||||
reserved hole per datagram, `relative_time` framing each one as NaN, and nothing ever missing to
|
||||
backfill it, i.e. one missing-data band over the whole window with no trace in it, under a status
|
||||
bar reading `live` at ten times the stream's real rate. The same field
|
||||
must be **0** on every honest shape above (`--drop-every`, `--drop-burst 200`,
|
||||
`--reorder-delay-ms 50`, at 10 kS/s: `holes=210 hole_slots=21000 nan_slots=11000 scanned=22000
|
||||
gaps_refused=0` for `--drop-every 2`, `holes=3 hole_slots=60000 nan_slots=20000` for the 200-packet
|
||||
burst at 100 kS/s) — a non-zero reading there means the arrival test is refusing real loss, whose
|
||||
only cost is a gap with no shading.
|
||||
|
||||
All of the readings below are off `UDPSCOPE_EXIT_DUMP`'s `[udpscope] frames= packets= gaps=` line,
|
||||
over an 8 s run against `--rate 10000 --batch 100 --freq 5` (100 packets a second) except where the
|
||||
burst needs the faster stream. **`gaps` is the exact reading** and repeats to the packet; `frames`
|
||||
|
||||
Reference in new issue
Block a user