StreamHub: loop UTF-8 tail repair to match Go CalConfig.Normalise()
The single-pass repair left invalid bytes when the candidate lead byte had class 0 (illegal 0xF8-0xFF bytes, or a bare continuation byte reached after the 3-byte backward-scan cap). Convert to a loop with a `cut` flag mirroring Go's loop: each iteration either makes no cut (exits) or strictly reduces ulen by >= 1 byte (terminates in <= 16 iterations). Also treat expected==0 as a cut target, matching Go's behaviour of stripping any byte that decodes as an invalid one-byte sequence. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Sonnet 4.6
parent
93e00d0c21
commit
bdc74f5fd2
@@ -189,3 +189,52 @@ Overall: ALL PASS
|
||||
```
|
||||
|
||||
All required cases verified: `"Ω"` survives unchanged, `"µs"` survives unchanged, 20-byte ASCII truncates to 16, a cut mid-rune truncates to the last complete rune, and a 16-byte string ending exactly on a complete multi-byte rune is untouched.
|
||||
|
||||
## Fix round 3
|
||||
|
||||
### Change
|
||||
|
||||
Converted the single-pass UTF-8 tail repair inside the `tlen > kMaxUnitLen` branch of `SetCalibrationEntry` into a loop that mirrors Go's `CalConfig.Normalise()` exactly. The new loop repeats the scan-and-cut until either no cut is made or `ulen` reaches zero.
|
||||
|
||||
Two new cases are now handled that the old single pass missed:
|
||||
|
||||
1. **Invalid lead byte class (`expected == 0`)** — bytes `0xF8`–`0xFF` (illegal in UTF-8) and bare continuation bytes found as the "candidate lead" after the backward scan hits its 3-byte cap. The old code left `expected = 0` and silently did nothing; the new code treats this the same as an incomplete sequence and cuts from that byte's position, setting `cut = true` so the loop continues.
|
||||
|
||||
2. **Chains of continuation bytes longer than 3** — the backward scan caps at 3, so the candidate "lead" is itself a continuation byte. `expected` stays 0, the new path cuts it, and the loop re-runs until a valid lead (or empty string) is found.
|
||||
|
||||
### Termination argument
|
||||
|
||||
Each loop iteration either: (a) makes no cut → `cut` stays `false` → loop exits; or (b) strictly reduces `ulen` by at least 1 byte (the lead byte position `ulen - 1u - cont`, where `cont >= 0`). Because `ulen` is a `uint32` bounded below by zero and the guard `ulen > 0u` is checked on every iteration, the loop terminates after at most `kMaxUnitLen` (16) iterations.
|
||||
|
||||
### Code diff (StreamHub.cpp, repair block)
|
||||
|
||||
Old: single pass, no loop, `expected == 0` → silent no-op.
|
||||
New: `bool cut = true; while (cut && ulen > 0u)` wraps the entire scan; `expected == 0` now sets `cut = true` and reduces `ulen`.
|
||||
|
||||
### Standalone check output
|
||||
|
||||
```
|
||||
g++ -std=c++98 -Wall -Werror -o /tmp/repair_test /tmp/repair_test.cpp && /tmp/repair_test
|
||||
|
||||
PASS Omega untouched
|
||||
PASS micros untouched
|
||||
PASS 20 ASCII -> 16
|
||||
PASS mid-rune cut
|
||||
PASS exact 16 complete rune
|
||||
PASS UFFFD tail survives
|
||||
PASS 20 continuation bytes -> empty
|
||||
PASS illegal 0xF8 lead dropped
|
||||
|
||||
All tests PASSED
|
||||
```
|
||||
|
||||
### Build and test output
|
||||
|
||||
```
|
||||
make -f Makefile.gcc apps → StreamHub.ex linked successfully (0 errors)
|
||||
|
||||
./Build/x86-linux/GTest/MainGTest.ex
|
||||
132 tests from 12 test cases ran.
|
||||
PASSED: 127
|
||||
FAILED: 5 (UDPStreamerGTest multicast — pre-existing, machine-level issue; expected baseline 127/132 or 128/132)
|
||||
```
|
||||
|
||||
Reference in New Issue
Block a user