fix: a claimed gap must be corroborated before it is believed, and its hole must fit
The user's real host reported "connected and live" with one missing-data band over the whole window
and no trace in it. The counter is the only thing that claims a packet existed, and a reservation
was booked on its word alone: a source whose DATA counter counts samples, or frames of an inner
layer, advances by more than 1 per datagram, the client cannot check that claim (it has no clock of
its own to compare against), so it holds each packet, saturates its reorder ring, delivers it in
order and reports the skipped counters as lost. Every frame therefore reported a phantom gap, and
`reserve_hole()` wrote `lost * batch_` NaN samples with no bound but the 256-PACKET cap -- the
sample form of that rule ("the number to weigh against the ring is 256 * batch_, not the 256") was
never implemented. With relative time on -- which that session carries -- each phantom gap is
materialised as NaN slots by construction: the window filled with reservations, there was never
anything missing to repair them, and the gaps also stretched the relative clock, which is why the
status bar read live at ten times the true rate.
`Acquisition::gap_plausible()` now gates every reservation on two independent tests: the arrival
clock must corroborate the claim (that many missing packets would take that many frame intervals;
deliberately permissive, because a false refusal costs a shading and never data), and the ring must
be able to hold the hole. The verdict is taken before the stamps, and relative mode's slot advance
is skipped with it, since the advance and the reservation are the same claim. Refusals are counted
in `StreamStats::gaps_refused` and said once per connection with the numbers, naming the wire shape
they describe.
tools/mock_udpstreamer.py gains `--counter-step N`: a counter that is not a packet counter on an
intact link, which is the one shape the mock could not produce and the shape the report came from.
Verified in both directions. A counter stepping 1000 per datagram reads `holes=0 hole_slots=0
backfilled=0` with the traces present (3 traces, 1.4 M samples) against 7.9 M claimed gaps, and the
warning above names the cause. Every other packet withheld -- a real loss -- still reads
`holes=3579`, so genuine gaps are still reserved: a fix that stopped believing real losses would
have traded one wrong picture for another. The user's control read as "the math channel is the
problem" and was confounded -- disabling it is a structural change, which rebuilds the buffer and
clears the accumulated NaN.
This commit is contained in:
1 parent
22e7a8efcc
commit
27c9b34277
9 files changed
+249
-15
No files matched your search
@@ -554,6 +554,17 @@ missing shades solidly, even if all but one slot are already repaired. And the t
|
||||
yet configurable; the preferences dialog in `todo.md` is where that would land. Do not use a
|
||||
command-line flag for either; none exists.
|
||||
|
||||
**The counter is a claim, and the wire is checked against it.** A gap is only booked when the
|
||||
arrival times corroborate it: `lost` packets would have taken that many packet intervals to
|
||||
arrive, and a claim of less than half that is refused, as is a hole bigger than the ring could
|
||||
hold. The status bar's `gaps` counts what the *stream* says either way; the refused ones are
|
||||
counted separately and said once on stderr. That matters for one kind of source: a device whose
|
||||
counter counts something other than packets — samples, or frames of an inner layer — reports a
|
||||
forward jump on every datagram. Believing it reserved a NaN run per packet, which on an intact
|
||||
link reads as one missing-data band over the whole window with the samples washed out and nothing
|
||||
lost to backfill it, while the status bar says `live`. The samples are now kept and the gap shows
|
||||
as the x discontinuity it is.
|
||||
|
||||
**`--relative-time` makes the x axis follow the sample count and the declared rate**, rather than
|
||||
the stream's own clock. The packet counter says exactly how many samples each gap spans, so a
|
||||
reserved hole is a precise arithmetic range. In stream mode the same hole is estimated from the
|
||||
|
||||
@@ -169,7 +169,36 @@ Part of [CLAUDE.md](../../CLAUDE.md). These are rules that cost a defect to lear
|
||||
(`kGlitchPackets`, 256) is counted in **packets, not samples**, so one hole's sample cost
|
||||
is `256 × batch_` and scales with the stream's batch — 25 600 samples at a batch of 100,
|
||||
2.56 M at a batch of 10 000 — which is the number to weigh against the ring, not the 256.
|
||||
- **A re-plan drops every hole and restarts the producer's tick.** `rebuild_locked()` replaces
|
||||
- **A reservation is weighed against the wire, and never booked on the counter's word alone.**
|
||||
The counter is a *claim* about the stream; the arrival clock is a *measurement* of it, and only
|
||||
the two together justify writing NaN into the ring (`Acquisition::gap_plausible()`, called
|
||||
before anything is booked *and* before relative mode's `slot_base_` advance, because the advance
|
||||
and the reservation are the same claim). Two tests, either one refusing is enough:
|
||||
- **The arrival clock.** `missing` packets would have taken `missing` frame intervals to arrive,
|
||||
so a gap whose elapsed arrival time is under half of that is refused. Half and not one because
|
||||
the socket clock jitters and a drain can compress two frames' arrival times -- and a refused
|
||||
gap costs a *shading*, never data, so the test is deliberately permissive.
|
||||
- **The ring.** `missing × batch_` samples become `missing × batch_ / time_stride` slots, and the
|
||||
hole may not lap `time_capacity`. This is the *sample* form of the rule the 256-packet cap
|
||||
states in packets -- the cap's own comment says 256 packet spans in one go is an outage, and
|
||||
the invariant above says the number to weigh against the ring is `256 × batch_` and not the
|
||||
256 -- and it has to exist in samples because a hole that laps the ring erases the history it
|
||||
is meant to annotate, can never be backfilled (its slots are already overwritten) and is never
|
||||
forgotten (`forget_holes_before()` waits for slots the ring has passed), so it leaves a
|
||||
*permanent* band over a buffer that is all NaN.
|
||||
What its absence looks like, which is why it is written down: a device whose counter counts
|
||||
*samples*, or frames of an inner layer, reports a forward jump on every datagram while the wire
|
||||
delivers on time. Every frame then reserves, and with `relative_time` every frame books NaN
|
||||
slots by construction -- one band over the whole window with the samples washed out,
|
||||
`backfilled=0` (no packet was ever missing, so nothing can repair it), and the status bar reading
|
||||
`live` at the right rate the whole time. The refusal is counted in `StreamStats::gaps_refused`
|
||||
and said once per connection with the numbers, because the status bar's `gaps` climbs either way
|
||||
and the state is otherwise invisible; `tools/mock_udpstreamer.py --counter-step N` is the mock
|
||||
shape (its counter is otherwise always +1, so this is the one wire shape it could not produce),
|
||||
and `gaps_refused` rising while `nan_slots` stays 0 is the reading. A refused gap is not a silent
|
||||
one: the timeline is not stretched over loss that did not happen, so it shows as the x
|
||||
discontinuity it is.
|
||||
- **A re-plan drops every hole and restarts the producer's tick.** `rebuild_locked()` replaces
|
||||
the store, so `produced()` restarts at 0 and a hole's `slot_begin`/`tick_begin`/`x_begin`
|
||||
describe a buffer that is gone; it therefore clears `holes_` and bumps
|
||||
`ScopeModel::plan_generation()`, which the producer compares against the value it last saw
|
||||
|
||||
@@ -872,10 +872,11 @@ two also catch a row whose accelerator went missing. `settings=` must be a real
|
||||
The `m`/`g` pairs are the *tab name* before and after, which is what says those two keys select a tab rather than toggling a pane; `t`/`c` read the toolbar toggle's own state, because those two enable a feature and the button is where that state is visible. The refusal is the whole reason the controller is in the bubble phase: every bare-letter accel is also a character, so an accel that fires while a `GtkEditable` has focus makes the window entry unusable. A pair reading `0->0` is an action whose dispatch does not reach the handler a click reaches. Note the format is `<key>: <widget-description>: <before>-><after>` — the name passed to `before_after()` includes the widget's description, so `t: trigger panel: 0->1` is the correct form (not `t: trigger panel 0->1` without the colon before the values).
|
||||
- `[color] '<name>' at row R slot S: trace=#RRGGBB chip_agrees=1 (want 1)`, then `[color] override: trace=#123456 chip_agrees=1`, `[color] session: entries=1 rgb=#123456` and `[color] default: trace=#RRGGBB chip_agrees=1` (stderr, `UDPSCOPE_PROBE_COLOR_MS`) — **all three `chip_agrees` must be 1**, the override line's `trace` must be `#123456`, the session line must read `entries=1 rgb=#123456`, and the default line's `trace` must be back to the first line's. Run it with `UDPSCOPE_ENABLE_ALL_MS` ahead of it, or there is no bound trace and the probe prints `no bound trace to recolour` instead of asserting anything; on the standard mock the first line reads `'sine' at row 0 slot 0: trace=#4FA3FF`. Two things make this an assertion rather than a tautology. The colours are read out of **`plot::Plot::trace().color`** — what the renderer will stroke, reached through `Row::plot` *after* `relayout_areas()`, since a new colour is a new figure — and out of the chip's own `<span foreground="…">` markup, not by asking `RowSet::signal_color()` twice, which would pass against a chip that never reached the trace. And the override is `#123456`, deliberately in no palette slot: an override equal to the entry it replaces passes against an implementation that ignores overrides entirely. The session leg round-trips `collect_layout()` → `save_layout()` → `load_layout()` on a temp path and deletes it, because a probe must not leave a colour in the real session file for a later run to inherit.
|
||||
- `[reseq-probe] reseq=N late=N stale=N overflows=N holes=N hole_slots=N backfilled=N
|
||||
open_holes=N bands=N bands_repaired=N nan_slots=N scanned=N trigger_in_hole=D` (stderr,
|
||||
gaps_refused=N open_holes=N bands=N bands_repaired=N nan_slots=N scanned=N trigger_in_hole=D`
|
||||
(stderr,
|
||||
`UDPSCOPE_PROBE_RESEQ_MS`) —
|
||||
the resequencing path's mid-run state; the measured readings and the per-case command are in
|
||||
[verification.md](verification.md). Eight things to read carefully:
|
||||
[verification.md](verification.md). Nine things to read carefully:
|
||||
|
||||
1. **`backfilled` and `hole_slots` are not comparable directly.** `hole_slots` counts
|
||||
*time-ring* slots once per hole; `backfilled` counts *value-ring* slots repaired and
|
||||
@@ -941,6 +942,29 @@ two also catch a row whose accelerator went missing. `settings=` must be a real
|
||||
slots. A non-zero reading beside a healthy `backfilled` is a note about the ring's size,
|
||||
not a defect.
|
||||
|
||||
9. **`gaps_refused` counts reservations the wire contradicted, and it is the field that reads
|
||||
a counter which is not counting packets.** A gap is booked only if it passes two tests
|
||||
(`Acquisition::gap_plausible()`): the *arrival clock* has to corroborate the elapsed time
|
||||
(`missing` packets would have taken `missing` frame intervals; anything under half of that
|
||||
is refused) and the *ring* has to hold the hole (`missing * batch_` samples must not lap
|
||||
`time_capacity`). The shape this exists for is a device whose counter advances by samples,
|
||||
or by frames of an inner layer: every datagram reports a forward jump while the wire
|
||||
delivers on time, so *every* frame reserves -- and, with `relative_time`, every frame
|
||||
books NaN slots, which is a band over the whole window with the samples washed out and no
|
||||
packet ever missing to backfill it. `tools/mock_udpstreamer.py --counter-step N` produces
|
||||
exactly that on an intact link. The reading's two sides, both measured with
|
||||
`--relative-time` and no session: a *fixed* run reads
|
||||
`holes=1 hole_slots=900 gaps_refused=419 open_holes=0 bands=0 nan_slots=0 scanned=21700`
|
||||
with the data channels full and a once-per-connection stderr warning naming the numbers,
|
||||
while the unfixed one read `holes=420 hole_slots=378000 open_holes=64 bands=64
|
||||
bands_repaired=0 nan_slots=198000 scanned=220000` -- `nan_slots` at 90% of `scanned`, which
|
||||
is the whole-window band. The refusal is **not relative-mode-only**: the same command without
|
||||
`--relative-time` reads `holes=1 gaps_refused=419 nan_slots=0 scanned=21800`, because a
|
||||
counter that lies is not a mode. On the honest shapes
|
||||
(`--drop-every`, `--drop-burst 200`, `--reorder-delay-ms`) this field must stay **0**:
|
||||
a non-zero `gaps_refused` there means the arrival test is refusing real loss, and the cost
|
||||
is a gap with no shading (never lost data).
|
||||
|
||||
- `[zoomkeys] ctrl+plus x=[..]->[..] y_held=1 | alt+plus y=[..]->[..] x_held=1` (stderr, `UDPSCOPE_PROBE_ZOOMKEYS_MS`) — **both `_held` must be 1 and both zooms must still happen.** Run it inside a pinned state, where a frame cannot re-frame the other axis for its own reasons: `UDPSCOPE_PROBE_PAUSE_MS=1500 UDPSCOPE_PROBE_ZOOMKEYS_MS=2200`. A `_held=0` is the modifier not reaching the plotter's controller; an unchanged range on the *zoomed* axis is the opposite failure, the key doing nothing. Ctrl restricts to x and Alt to y; Ctrl+Alt zooms neither (no special case needed — neither flag alone fires).
|
||||
- `status bar: glance=[● live · 1.09 MS/s · 2 s window] max_glance_chars=53 detail_fields=11 popover_rebuilds=0` (exit dump) — **`max_glance_chars` must stay small** (under about 60) and **`popover_rebuilds` must be 0**. The status row is directly above the plot, so this is the "layout must not depend on content" rule read from the bar's own side: the state field is one word from a fixed set (`connecting`, `live`, `paused`, `armed`, `capturing`, `event held`, `disarmed` — a word added to it must be **no wider than the widest already there**, which is what let `connecting` join at ten characters beside `event held`; the dead-port glance reads `max_glance_chars=51` against a live run's 54), the rate field is `%.3g` (so `10 kS/s`, never `10.0 kS/s`), and the two fields a *state* adds are both elapsed times — trigger mode's `last event … ago` and a pause's `for <elapsed>` — which step units rather than growing digits (`format_elapsed()`). A **paused** glance is not in the dump at all: it is read by `[pause-probe]`'s `glance=` field, since the dump's own line is taken live. `detail_fields` is the popover's label count, a literal, so adding a label without thinking about layout shows up here. The glance rate format matters: `%.3g` drops the trailing zero that a `%.4g` or a fixed-precision format would keep — a rate that reads differently in the dump and in the help page is a format disagreement, and `14-status.txt`'s example must match it.
|
||||
|
||||
|
||||
@@ -590,7 +590,7 @@ as distinct from the fault injection below, which exercises ingest:
|
||||
at 64. The `mixed` case's `open_holes=43` against `bands=43` has no such spread because
|
||||
nothing re-planned. The per-field reference is in [probe-readings.md](probe-readings.md).
|
||||
|
||||
The mock also carries three **fault injections** that no probe schedules, because what they exercise
|
||||
The mock also carries four **fault injections** that no probe schedules, because what they exercise
|
||||
is the ingest rather than the UI. The first: `--time-glitch-at <stream seconds> --time-glitch-by <seconds>` makes
|
||||
**one** packet's time signal jump ahead and then carries on from where the true timeline was, which
|
||||
is what a device whose clock glitches puts on the wire. Run it with the trigger armed, e.g.
|
||||
@@ -625,6 +625,41 @@ arrives within the 20 ms reorder window is resequenced with no hole, and one tha
|
||||
leaves a committed hole that it then backfills (`UDPSCOPE_PROBE_RESEQ_MS` exercises both paths).
|
||||
There is **no interpolation** anywhere below the ring.
|
||||
|
||||
The fourth is the counter itself: **`--counter-step N`** advances the header counter by `N` per
|
||||
datagram instead of one -- a device whose counter counts *samples*, or frames of an inner layer,
|
||||
rather than packets. Nothing is withheld and every datagram goes out on schedule, so the link is
|
||||
intact and the status bar is right to say `live`; what is wrong is the counter's claim, which the
|
||||
client cannot check (it has no clock) and only the app can (it has the arrival times). It is the
|
||||
one wire shape the mock could not otherwise produce, and the run that reads it is the reseq probe
|
||||
with relative time on (the mode that frames a counter gap as NaN by construction):
|
||||
|
||||
```sh
|
||||
# mock: --counter-step 10
|
||||
xvfb-run -a env UDPSCOPE_EXIT_AFTER_MS=8000 UDPSCOPE_ENABLE_ALL_MS=600 \
|
||||
UDPSCOPE_PROBE_RESEQ_MS=5000 UDPSCOPE_EXIT_DUMP=1 \
|
||||
./build/udpscope --server 127.0.0.1 --port 8000 --relative-time
|
||||
```
|
||||
|
||||
```
|
||||
# fixed: [reseq-probe] … holes=1 hole_slots=900 backfilled=0 gaps_refused=419 open_holes=0
|
||||
# bands=0 bands_repaired=0 nan_slots=0 scanned=21700 trigger_in_hole=0
|
||||
# status bar: glance=[● live · 9.97 kS/s · 2 s window]
|
||||
# broken: [reseq-probe] … holes=420 hole_slots=378000 backfilled=0 open_holes=64 bands=64
|
||||
# bands_repaired=0 nan_slots=198000 scanned=220000 trigger_in_hole=0
|
||||
# status bar: glance=[● live · 101 kS/s · 2 s window]
|
||||
```
|
||||
|
||||
**`gaps_refused` above 0 with `nan_slots` at 0 is the assertion** — the gap was refused and the
|
||||
samples were kept. The broken reading is `nan_slots` at 90% of `scanned` with `backfilled=0`: a
|
||||
reserved hole per datagram, `relative_time` framing each one as NaN, and nothing ever missing to
|
||||
backfill it, i.e. one missing-data band over the whole window with no trace in it, under a status
|
||||
bar reading `live` at ten times the stream's real rate. The same field
|
||||
must be **0** on every honest shape above (`--drop-every`, `--drop-burst 200`,
|
||||
`--reorder-delay-ms 50`, at 10 kS/s: `holes=210 hole_slots=21000 nan_slots=11000 scanned=22000
|
||||
gaps_refused=0` for `--drop-every 2`, `holes=3 hole_slots=60000 nan_slots=20000` for the 200-packet
|
||||
burst at 100 kS/s) — a non-zero reading there means the arrival test is refusing real loss, whose
|
||||
only cost is a gap with no shading.
|
||||
|
||||
All of the readings below are off `UDPSCOPE_EXIT_DUMP`'s `[udpscope] frames= packets= gaps=` line,
|
||||
over an 8 s run against `--rate 10000 --batch 100 --freq 5` (100 packets a second) except where the
|
||||
burst needs the faster stream. **`gaps` is the exact reading** and repeats to the packet; `frames`
|
||||
|
||||
+96
-6
@@ -203,6 +203,7 @@ void Acquisition::on_config(const udps_signal_t* signals, std::uint32_t count,
|
||||
// next session is the failure mode worth avoiding in a feature whose whole job is diagnosis.
|
||||
warned_batch_ = false;
|
||||
warned_outage_ = false;
|
||||
warned_phantom_gap_ = false;
|
||||
warned_bridge_ = false;
|
||||
warned_time_source_ = false;
|
||||
// The clock choice is per connection too: a reconnect re-derives it from the frames
|
||||
@@ -243,9 +244,61 @@ void Acquisition::reset_session_state() noexcept {
|
||||
have_arrival_interval_ = false;
|
||||
arrival_interval_ = 0.0;
|
||||
arrival_prev_ = 0.0;
|
||||
have_prev_frame_recv_ = false;
|
||||
prev_frame_recv_ = 0.0;
|
||||
}
|
||||
|
||||
std::uint32_t Acquisition::build_stamps(const udps_frame_t* frame, int time_signal) {
|
||||
void Acquisition::note_frame_arrival(double recv_time) noexcept {
|
||||
const bool first = !have_prev_frame_recv_;
|
||||
const double elapsed = first ? 0.0 : recv_time - prev_frame_recv_;
|
||||
prev_frame_recv_ = recv_time;
|
||||
have_prev_frame_recv_ = true;
|
||||
// The arrival *cadence* belongs to the arrival clock, and `build_stamps()`'s arrival branch keeps
|
||||
// it whenever that clock is the one in play (`time_source_ == 2`, decided once per connection and
|
||||
// sticky). Every other case needs it here: relative mode returns before that branch, and a stream
|
||||
// whose declared time signal checks out never reaches it either -- while the reservation test
|
||||
// below weighs the counter's claim against a measured cadence whatever clock the display uses,
|
||||
// because a counter that does not count packets lies in all of them. The two writers are therefore
|
||||
// exclusive by `time_source_`, and this is the branch's own update (0.8/0.2) so the estimate does
|
||||
// not jump at the one frame where the clock changes hands.
|
||||
if (first || time_source_ == 2) return;
|
||||
if (elapsed > 0.0 && std::isfinite(elapsed)) {
|
||||
arrival_interval_ =
|
||||
have_arrival_interval_ ? 0.8 * arrival_interval_ + 0.2 * elapsed : elapsed;
|
||||
have_arrival_interval_ = true;
|
||||
}
|
||||
}
|
||||
|
||||
bool Acquisition::gap_plausible(std::uint32_t missing, double elapsed, const Partition& part,
|
||||
u64 g) const {
|
||||
if (missing == 0) return true;
|
||||
// The ring cannot hold a hole bigger than itself, and this test needs no measurement: the
|
||||
// invariant's own rule is that the reservation's *sample* cost -- `missing * batch_`, not the
|
||||
// packet cap -- is what has to be weighed against the ring. A hole that laps the ring erases the
|
||||
// history it is meant to annotate, can never be backfilled (its slots are already overwritten),
|
||||
// and is never forgotten (`forget_holes_before()` waits for slots the ring has passed), so it
|
||||
// leaves a band over a buffer that is all NaN.
|
||||
if (g == 0 || part.time_capacity == 0) return false;
|
||||
if (u64(missing) * u64(batch_) / g >= part.time_capacity) return false;
|
||||
// And the counter has to agree with the wire about how long the loss took. The counter is a claim
|
||||
// about the stream; the arrival clock is a *measurement* of it: `missing` packets would have taken
|
||||
// `missing` frame intervals to arrive, and the frame before this one is the witness of how long
|
||||
// actually passed. A device whose counter counts samples, or frames of an inner layer, reports a
|
||||
// forward jump on *every* datagram while the wire delivers on time -- and believing that booked a
|
||||
// NaN hole per packet, i.e. a band over the whole window filled with samples that were never
|
||||
// missing, which no backfill can repair because no packet was ever lost. The factor is a half and
|
||||
// not one: the socket clock jitters and a drain can compress two frames' arrival times, so only a
|
||||
// claim the wire flatly contradicts (half the time it needs, or less) is refused. A refused gap
|
||||
// costs a shading, never data.
|
||||
if (have_arrival_interval_ && arrival_interval_ > 0.0 && elapsed > 0.0 &&
|
||||
elapsed < 0.5 * double(missing) * arrival_interval_) {
|
||||
return false;
|
||||
}
|
||||
return true;
|
||||
}
|
||||
|
||||
std::uint32_t Acquisition::build_stamps(const udps_frame_t* frame, int time_signal,
|
||||
bool counter_gap_refused) {
|
||||
// The frame's span in base ticks: a scalar signal carries `num_samples` values
|
||||
// (one per batched cycle), a block signal (rows*cols > 1) carries its whole
|
||||
// element block. The largest of those is how many base ticks this frame spans --
|
||||
@@ -294,8 +347,11 @@ std::uint32_t Acquisition::build_stamps(const udps_frame_t* frame, int time_sign
|
||||
if (!(dt > 0.0)) dt = 1.0e-3;
|
||||
// The reserved hole for the missing packets advances `slot_base_` so this frame's
|
||||
// own stamps land *after* the hole rather than on top of it. The same conditions
|
||||
// the reservation call uses: not late, gap present, batch known, within the cap.
|
||||
if (!frame->late && frame->lost > 0 && batch_ > 0 &&
|
||||
// the reservation call uses: not late, gap present, batch known, within the cap --
|
||||
// and not one the arrival clock or the ring refused, because the advance and the
|
||||
// reservation are the *same* claim about the wire: believing a phantom gap here
|
||||
// would stretch the timeline by exactly the loss that did not happen.
|
||||
if (!frame->late && frame->lost > 0 && batch_ > 0 && !counter_gap_refused &&
|
||||
double(frame->lost) <= kGlitchPackets) {
|
||||
slot_base_ += u64(frame->lost) * batch_;
|
||||
}
|
||||
@@ -615,7 +671,20 @@ void Acquisition::on_data(const udps_frame_t* frame) {
|
||||
}
|
||||
}
|
||||
}
|
||||
const std::uint32_t n = build_stamps(frame, time_signal);
|
||||
// The wire's own cadence, and the verdict on the counter's claim, both taken *before* the
|
||||
// stamps: relative mode's `slot_base_` advance and the reservation below are the same claim
|
||||
// about the wire (see `build_stamps`), so they have to agree about it. The verdict is taken
|
||||
// before the cadence is updated, too: folding this frame's own arrival into the estimate would
|
||||
// let one burst raise the very number that is supposed to corroborate it -- a 200-packet outage
|
||||
// smooths the interval to 40 ms and then refuses itself.
|
||||
const bool first_frame = !have_prev_frame_recv_;
|
||||
const double since_prev = first_frame ? 0.0 : frame->recv_time - prev_frame_recv_;
|
||||
const std::uint32_t missing = frame->lost;
|
||||
const bool gap_refused =
|
||||
missing > 0 && !gap_plausible(missing, since_prev, part, part.time_stride);
|
||||
note_frame_arrival(frame->recv_time);
|
||||
|
||||
const std::uint32_t n = build_stamps(frame, time_signal, gap_refused);
|
||||
if (n == 0) return;
|
||||
|
||||
if (model_.epoch() == 0.0) {
|
||||
@@ -695,8 +764,29 @@ void Acquisition::on_data(const udps_frame_t* frame) {
|
||||
// through the same decimation accumulators (value_acc_, value_acc_n_, math_sample_),
|
||||
// which must already be sized to the current channel count and stride before any
|
||||
// per-sample write.
|
||||
const std::uint32_t missing = frame->lost;
|
||||
if (missing > 0 && batch_ > 0 && double(missing) <= kGlitchPackets) {
|
||||
if (missing > 0 && gap_refused) {
|
||||
// A claim the wire contradicted, or a hole the ring cannot hold: framing it as NaN is what
|
||||
// turns one phantom gap into a band over the whole window (and, at a batch big enough, into
|
||||
// minutes of the producer's time under this lock). The samples are kept, no hole is booked,
|
||||
// and the timeline above was not advanced either -- so the picture shows the data and the gap
|
||||
// shows as the x discontinuity it is. Said once per connection with the numbers, because it is
|
||||
// otherwise invisible: the status bar's `gaps` climbs either way.
|
||||
model_.stats().gaps_refused.fetch_add(1, std::memory_order_relaxed);
|
||||
if (!warned_phantom_gap_) {
|
||||
warned_phantom_gap_ = true;
|
||||
std::fprintf(stderr,
|
||||
"[udpscope] warning: the counter reports %u packet(s) missing (%llu samples, "
|
||||
"%.3f s at the measured cadence of %.6f s/frame) but only %.3f s passed since "
|
||||
"the previous frame, or the hole is larger than the ring (%llu slots); "
|
||||
"refusing the reservation and keeping the samples. A counter that does not "
|
||||
"count packets -- samples, or frames of an inner layer -- reports a jump on "
|
||||
"every datagram and reads exactly like this.\n",
|
||||
missing, static_cast<unsigned long long>(std::uint64_t(missing) * batch_),
|
||||
double(missing) * arrival_interval_, arrival_interval_, since_prev,
|
||||
static_cast<unsigned long long>(u64(missing) * u64(batch_) / part.time_stride));
|
||||
std::fflush(stderr);
|
||||
}
|
||||
} else if (missing > 0 && batch_ > 0 && double(missing) <= kGlitchPackets) {
|
||||
reserve_hole(missing, part, channels, store, g);
|
||||
} else if (missing > 0 && !warned_outage_) {
|
||||
warned_outage_ = true;
|
||||
|
||||
+25
-2
@@ -77,7 +77,8 @@ class Acquisition {
|
||||
/// Fills `stamps_` with this frame's base-tick timestamps, in stream seconds,
|
||||
/// and returns how many were written (`frame->num_samples` unless the frame is
|
||||
/// malformed).
|
||||
std::uint32_t build_stamps(const udps_frame_t* frame, int time_signal);
|
||||
std::uint32_t build_stamps(const udps_frame_t* frame, int time_signal,
|
||||
bool counter_gap_refused);
|
||||
|
||||
/// Pushes NaN for the `missing` packets' worth of samples that were never delivered,
|
||||
/// advances the decimation accumulators and tick_, records the hole in the model, and
|
||||
@@ -92,6 +93,15 @@ class Acquisition {
|
||||
void backfill_from(const udps_frame_t* frame, const Partition& part,
|
||||
const std::vector<ChannelInfo>& channels, BufferStore& store, u64 g);
|
||||
|
||||
/// Keeps the wire's own cadence up to date from the arrival times, in the modes where
|
||||
/// `build_stamps()`'s arrival branch does not (see the definition).
|
||||
void note_frame_arrival(double recv_time) noexcept;
|
||||
|
||||
/// Whether the counter's claim of `missing` lost packets can be believed *and* booked: the
|
||||
/// arrival clock has to corroborate it and the ring has to be able to hold it. See the
|
||||
/// definition for both tests and for what believing a claim that fails them looks like.
|
||||
bool gap_plausible(std::uint32_t missing, double elapsed, const Partition& part, u64 g) const;
|
||||
|
||||
ScopeModel& model_;
|
||||
Config cfg_;
|
||||
udps_client* client_ = nullptr;
|
||||
@@ -125,10 +135,20 @@ class Acquisition {
|
||||
double arrival_epoch_ = 0.0;
|
||||
bool have_arrival_epoch_ = false;
|
||||
/// Smoothed interval between two frames' arrivals, and the previous arrival: the
|
||||
/// spine of the arrival-anchored clock (see `build_stamps`).
|
||||
/// spine of the arrival-anchored clock (see `build_stamps`). In relative mode no frame
|
||||
/// reaches that branch, so `on_data` keeps the same pair up to date itself -- the
|
||||
/// reservation's plausibility test below needs a measured cadence whatever clock the
|
||||
/// user chose, and the two writers are mutually exclusive by mode.
|
||||
double arrival_interval_ = 0.0;
|
||||
double arrival_prev_ = 0.0;
|
||||
bool have_arrival_interval_ = false;
|
||||
/// The previous *accepted* frame's receive time, for the one measurement the reservation
|
||||
/// test needs: how much wall time actually passed. Kept apart from `arrival_prev_` on
|
||||
/// purpose -- that one is the arrival clock's own bookkeeping and is re-anchored by the
|
||||
/// clock-choice code, while this is a bare witness of the wire and must never be moved by
|
||||
/// a mode change.
|
||||
double prev_frame_recv_ = 0.0;
|
||||
bool have_prev_frame_recv_ = false;
|
||||
/// The stream's time source, chosen once and sticky: 0 undecided, 1 the declared
|
||||
/// time signal, 2 arrival. Re-evaluated only on a CONFIG. A per-frame choice
|
||||
/// interleaved two timelines in one ring -- some frames stamped from a declared
|
||||
@@ -158,6 +178,9 @@ class Acquisition {
|
||||
/// Warn-once latch for a loss past the reservation cap. A member, not a function-local
|
||||
/// `static`, matching the latch Task 2 added for the same reason.
|
||||
bool warned_outage_ = false;
|
||||
/// Warn-once latch for a reservation the arrival clock did not corroborate, or that the ring
|
||||
/// could not hold (see `gap_plausible()`). Once per connection, like the two above.
|
||||
bool warned_phantom_gap_ = false;
|
||||
/// Warn-once latch for the one bridged packet whose time signal glitched.
|
||||
bool warned_bridge_ = false;
|
||||
/// Warn-once latch for the declared time signal failing for good, so the arrival clock is
|
||||
|
||||
@@ -268,6 +268,15 @@ struct StreamStats {
|
||||
/// a reading of 0 beside a non-zero `holes` is exactly the honest answer until the
|
||||
/// backfill lands.
|
||||
std::atomic<std::uint64_t> backfilled{0};
|
||||
/// Reservations refused because the counter's claim was not corroborated by the arrival
|
||||
/// cadence, or because the hole it would book is bigger than the ring can hold. A counter
|
||||
/// is a claim about the wire and the arrival clock is the measurement of it: a stream whose
|
||||
/// counter does not count packets (a sample counter, say) reports a forward jump on every
|
||||
/// datagram, and believing it booked a NaN hole per packet -- a band over the whole window
|
||||
/// filled with samples that were never missing, which the backfill cannot repair because no
|
||||
/// packet was ever lost. Counted here because it is otherwise *invisible*: the status bar's
|
||||
/// `gaps` climbs either way, and only this field says the gap was refused.
|
||||
std::atomic<std::uint64_t> gaps_refused{0};
|
||||
};
|
||||
|
||||
/// What the plot is showing, which is one decision made once per frame: the bind, the
|
||||
|
||||
+4
-2
@@ -3411,7 +3411,8 @@ gboolean reseq_probe(gpointer user) {
|
||||
std::fprintf(stderr,
|
||||
"[reseq-probe] reseq=%llu late=%llu stale=%llu overflows=%llu holes=%llu "
|
||||
"hole_slots=%llu "
|
||||
"backfilled=%llu open_holes=%zu bands=%zu bands_repaired=%zu nan_slots=%llu "
|
||||
"backfilled=%llu gaps_refused=%llu open_holes=%zu bands=%zu bands_repaired=%zu "
|
||||
"nan_slots=%llu "
|
||||
"scanned=%llu trigger_in_hole=%d\n",
|
||||
static_cast<unsigned long long>(m.stats().resequenced.load()),
|
||||
static_cast<unsigned long long>(m.stats().late.load()),
|
||||
@@ -3419,7 +3420,8 @@ gboolean reseq_probe(gpointer user) {
|
||||
static_cast<unsigned long long>(m.stats().overflows.load()),
|
||||
static_cast<unsigned long long>(m.stats().holes.load()),
|
||||
static_cast<unsigned long long>(m.stats().hole_slots.load()),
|
||||
static_cast<unsigned long long>(m.stats().backfilled.load()), open_holes, bands,
|
||||
static_cast<unsigned long long>(m.stats().backfilled.load()),
|
||||
static_cast<unsigned long long>(m.stats().gaps_refused.load()), open_holes, bands,
|
||||
bands_repaired, static_cast<unsigned long long>(nan_slots),
|
||||
static_cast<unsigned long long>(scanned), trigger_in_hole);
|
||||
std::fflush(stderr);
|
||||
|
||||
@@ -337,6 +337,17 @@ def main():
|
||||
"or the client reconnects and the mock restarts its clock, which is a different "
|
||||
"test",
|
||||
)
|
||||
ap.add_argument(
|
||||
"--counter-step",
|
||||
type=int,
|
||||
default=1,
|
||||
help="advance the header counter by this many per datagram instead of one: a device "
|
||||
"whose counter is not a packet counter (it counts samples, or frames of an inner "
|
||||
"layer). Nothing is withheld -- every datagram is sent on schedule -- but every one "
|
||||
"of them reports a forward jump, which is the *phantom* gap the receiver has to "
|
||||
"refuse: 1 is a per-packet counter, 2 and up is the shape that reads as loss on an "
|
||||
"intact link",
|
||||
)
|
||||
ap.add_argument(
|
||||
"--reorder-every",
|
||||
type=int,
|
||||
@@ -537,7 +548,7 @@ def main():
|
||||
# The counter and the stream clock advance whatever became of the datagram. That
|
||||
# is what makes a withheld packet a *lost* one and not a slower producer: nothing
|
||||
# in a payload says which packet it is, only the header's counter does.
|
||||
counter += 1
|
||||
counter += max(1, args.counter_step)
|
||||
t_stream += args.batch * pdt
|
||||
next_send += period
|
||||
if (
|
||||
|
||||
Reference in new issue
Block a user