diff --git a/docs/coverage.md b/docs/coverage.md index e6f2749..5d1884d 100644 --- a/docs/coverage.md +++ b/docs/coverage.md @@ -3,6 +3,13 @@ `plot` measures line coverage of the compiled library bodies under `src/` and enforces a floor on it, so a regression fails a command instead of quietly lowering a number in a document. +**The table and the floor verdict below are generated.** `tools/coverage.sh` rewrites the region +between `` and +`` from the report that run just captured, by calling +`tools/coverage_floor.py --write-doc`. Do not edit inside those markers: the numbers there come +from the command, not from a hand-typed copy, and a run on an unchanged tree rewrites the region +with identical content. + ## The one command ```sh @@ -12,12 +19,10 @@ cmake --build build/cov --target coverage # the same thing, through CMake The script configures `build/cov` from the `linux-coverage` preset, builds it, runs the whole `ctest` suite, captures `src/**` with `lcov` into `build/cov/coverage.info`, renders the HTML report -to `build/cov/html/index.html`, prints the per-file table below, and **exits non-zero when a floor -is violated**. It needs `cmake`, `ninja`, `ctest`, `lcov`, `genhtml` and `python3` -(Arch: `pacman -S cmake ninja lcov python`); it exits `2` with a message when one is missing. - -A clean run of it — no build tree, all 26 translation units under `src/` compiled, 274 cases — takes -about two minutes here, of which the suite is ~107 s. +to `build/cov/html/index.html`, prints the per-file table, rewrites the generated region of this +file, and **exits non-zero when a floor is violated**. It needs `cmake`, `ninja`, `ctest`, `lcov`, +`genhtml` and `python3` (Arch: `pacman -S cmake ninja lcov python`); it exits `2` with a message +when one is missing. ## The floors @@ -35,18 +40,20 @@ about two minutes here, of which the suite is ~107 s. ## The measured table -Measured on 2026-09-28 with GCC 16.2.1 from a clean `build/cov`, over the full test suite. -`coverage.sh` prints exactly this table on every run. + + +``` +coverage_floor: ok -- 23 files under src/, aggregate 95.3% lines (3284/3447), every file at or above 80%, aggregate target 90% +``` - | File | Lines | Line rate | Functions | | --- | ---: | ---: | ---: | | `src/data/array_series.cpp` | 43/43 | 100.0% | 100.0% | | `src/data/minmax_pyramid.cpp` | 163/163 | 100.0% | 100.0% | | `src/data/series.cpp` | 105/107 | 98.1% | 100.0% | | `src/data/stream_series.cpp` | 39/39 | 100.0% | 100.0% | -| `src/gl/handle.cpp` | 214/256 | 83.6% | 100.0% | -| `src/gl/loader.cpp` | 53/81 | 65.4% | 77.8% | +| `src/gl/handle.cpp` | 256/256 | 100.0% | 100.0% | +| `src/gl/loader.cpp` | 81/81 | 100.0% | 100.0% | | `src/input/controller.cpp` | 327/341 | 95.9% | 100.0% | | `src/math/transform.cpp` | 28/28 | 100.0% | 100.0% | | `src/plot/axis.cpp` | 12/12 | 100.0% | 100.0% | @@ -55,25 +62,18 @@ Measured on 2026-09-28 with GCC 16.2.1 from a clean `build/cov`, over the full t | `src/plot/measure.cpp` | 21/21 | 100.0% | 100.0% | | `src/plot/plot.cpp` | 733/753 | 97.3% | 98.9% | | `src/plot/ticks.cpp` | 93/99 | 93.9% | 100.0% | -| `src/render/buffer.cpp` | 87/98 | 88.8% | 100.0% | +| `src/render/buffer.cpp` | 88/98 | 89.8% | 100.0% | | `src/render/embedded_mono_font.cpp` | 45/45 | 100.0% | 100.0% | | `src/render/primitives.cpp` | 71/75 | 94.7% | 100.0% | | `src/render/program.cpp` | 26/27 | 96.3% | 100.0% | | `src/render/renderer.cpp` | 529/604 | 87.6% | 93.7% | | `src/render/text.cpp` | 27/30 | 90.0% | 100.0% | | `src/render/text_atlas.cpp` | 40/41 | 97.6% | 100.0% | -| `src/render/vertex_array.cpp` | 49/59 | 83.1% | 100.0% | +| `src/render/vertex_array.cpp` | 59/59 | 100.0% | 100.0% | | `src/text/true_type_font.cpp` | 270/291 | 92.8% | 96.8% | -| **src/ aggregate** | **3203/3447** | **92.9%** | **97.8%** | - +| **src/ aggregate** | **3284/3447** | **95.3%** | **98.3%** | -**The floor currently fails, and that is the point.** `src/gl/loader.cpp` sits at 65.4% lines -(53/81) — below the 80% per-file floor — so `tools/coverage.sh` exits `1` on today's tree. What is -uncovered there is the loader's error paths and its extension scanner; `src/gl/handle.cpp` (83.6%) -and `src/render/vertex_array.cpp` (83.1%) are the same kind of gap, just above the line. Those tests -are Task 2 of `docs/superpowers/plans/2026-09-28-plot-docs-and-coverage.md`; the floor went in first -so the fix has something to satisfy rather than something to negotiate with, and so the *next* -regression in that area is a failing command rather than a number nobody reads. + ## What the report measures @@ -118,40 +118,35 @@ before `genhtml` re-creates it. per-file `LF`/`LH`/`FNF`/`FNH` tallies that are identical for all 23 files — verified by diffing the two tracefiles — but the per-line hit *counts* differ by a few parts per million, because a handful of tests drive a producer thread whose iteration count follows wall-clock timing. Everything the -floor reads (the `LF`/`LH` tallies, and therefore every rate in the table above) is unaffected. +floor reads (the `LF`/`LH` tallies, and therefore every rate in the generated table above) is +unaffected. ## Adjusting the floors, and seeing the gate fail Both floors are flags, which is how the failure path is demonstrated — and how a later task tightens -the contract. Against `build/cov/coverage.info` from the table above: +the contract. Against the report in the generated table above, a floor raised above every file's +rate fails the check and names each offender: ```sh -$ python3 tools/coverage_floor.py build/cov/coverage.info # the real floors -coverage_floor: FAILED -- 1 floor(s) violated in build/cov/coverage.info: - src/gl/loader.cpp: 65.4% lines (53/81) is below the 80% per-file floor -$ echo $? -1 +# Any file not at 100% fails: a floor that is guaranteed to bite, whatever the +# current rates are. The list of offenders is the files in the table above whose +# rate is below 100%. +python3 tools/coverage_floor.py --per-file 100 build/cov/coverage.info +# coverage_floor: FAILED -- floor(s) violated in build/cov/coverage.info: +# : % lines (/) is below the 100% per-file floor +# ... +# $? == 1 -$ python3 tools/coverage_floor.py --per-file 95 build/cov/coverage.info # floor raised above rates -coverage_floor: FAILED -- 10 floor(s) violated in build/cov/coverage.info: - src/gl/handle.cpp: 83.6% lines (214/256) is below the 95% per-file floor - src/gl/loader.cpp: 65.4% lines (53/81) is below the 95% per-file floor - src/plot/grid.cpp: 93.8% lines (15/16) is below the 95% per-file floor - src/plot/ticks.cpp: 93.9% lines (93/99) is below the 95% per-file floor - src/render/buffer.cpp: 88.8% lines (87/98) is below the 95% per-file floor - src/render/primitives.cpp: 94.7% lines (71/75) is below the 95% per-file floor - src/render/renderer.cpp: 87.6% lines (529/604) is below the 95% per-file floor - src/render/text.cpp: 90.0% lines (27/30) is below the 95% per-file floor - src/render/vertex_array.cpp: 83.1% lines (49/59) is below the 95% per-file floor - src/text/true_type_font.cpp: 92.8% lines (270/291) is below the 95% per-file floor -$ echo $? -1 +# The same, for the aggregate: a target above the measured aggregate fails with one line. +python3 tools/coverage_floor.py --aggregate 99 build/cov/coverage.info +# coverage_floor: FAILED -- 1 floor(s) violated in build/cov/coverage.info: +# src/ aggregate: % lines (/) is below the 99% aggregate floor +# $? == 1 -$ python3 tools/coverage_floor.py --per-file 60 --aggregate 90 build/cov/coverage.info -coverage_floor: ok -- 23 files under src/, aggregate 92.9% lines (3203/3447), every file at or -above 60%, aggregate target 90% -$ echo $? -0 +# A floor at or below the measured rates passes. +python3 tools/coverage_floor.py build/cov/coverage.info +# coverage_floor: ok -- ... +# $? == 0 ``` ## Why the check is a CMake target and not a ctest case diff --git a/tools/coverage.sh b/tools/coverage.sh index af22b3f..a09a020 100755 --- a/tools/coverage.sh +++ b/tools/coverage.sh @@ -6,8 +6,10 @@ # capture src/** with lcov into build/cov/coverage.info, render the HTML report # under build/cov/html/, print the per-file table, and exit non-zero when # tools/coverage_floor.py finds a file below 80% lines or the src/ aggregate -# below 90%. The numbers and the floors are the contract docs/coverage.md -# states; `docs/coverage.md` also records how to re-read the table from here. +# below 90%. The floors are tools/coverage_floor.py's constants; the measured +# table and the floor verdict are written back into docs/coverage.md's generated +# region by `coverage_floor.py --write-doc`, from the very report this run +# captured, so that document cannot drift from the command that measures it. # # Idempotent by construction: gcov counters accumulate in .gcda files, so a # second run that inherited the first run's counters would report a superset of @@ -31,6 +33,7 @@ readonly FLOOR="$ROOT/tools/coverage_floor.py" readonly RAW="$BUILD/coverage.raw.info" readonly INFO="$BUILD/coverage.info" readonly HTML="$BUILD/html" +readonly DOC="$ROOT/docs/coverage.md" log() { printf '==> %s\n' "$*"; } @@ -87,6 +90,13 @@ python3 "$FLOOR" --table "$INFO" readonly FLOOR_STATUS=$? set -e +# The measured table is written into the document that states it, from the very +# report this run captured, so docs/coverage.md cannot drift from the command +# that produces it. This happens before the floor gate below, so a failing tree +# still gets an honest table (and its failure recorded) in the document. +log "regenerating the measured table in $DOC" +python3 "$FLOOR" --write-doc "$DOC" "$INFO" + if (( FLOOR_STATUS != 0 )); then printf '\ncoverage: FLOOR FAILED (exit %d) -- report %s, HTML %s/index.html\n' \ "$FLOOR_STATUS" "$INFO" "$HTML" >&2 diff --git a/tools/coverage_floor.py b/tools/coverage_floor.py index e05fbd7..4dfc5aa 100755 --- a/tools/coverage_floor.py +++ b/tools/coverage_floor.py @@ -184,6 +184,75 @@ def render_table(records: list[Record], markdown: bool) -> str: return "\n".join(lines) +# The generated region docs/coverage.md carries. tools/coverage.sh calls this +# tool with --write-doc on every run, so the table in that document is the one +# the command that measured it just captured, never a number copied by hand. +DOC_BEGIN = "" +DOC_END = "" + + +def render_doc_block(records: list[Record], per_file: float, aggregate: float) -> str: + """The region spliced into docs/coverage.md: the floor verdict and the table.""" + failures = [ + record + for record in records + if record.lines_found > 0 + and record.lines_hit * 100.0 < per_file * record.lines_found + ] + aggregate_hit = sum(record.lines_hit for record in records) + aggregate_found = sum(record.lines_found for record in records) + aggregate_rate = ( + 100.0 * aggregate_hit / aggregate_found if aggregate_found else 0.0 + ) + aggregate_below = aggregate_hit * 100.0 < aggregate * aggregate_found + + lines = [DOC_BEGIN, ""] + lines.append("```") + if failures or aggregate_below: + lines.append( + f"coverage_floor: FAILED -- {len(failures) + (1 if aggregate_below else 0)} " + "floor(s) violated in the report this run captured:" + ) + for record in failures: + lines.append( + f" {record.display}: {record.rate:.1f}% lines " + f"({record.lines_hit}/{record.lines_found}) is below the " + f"{per_file:g}% per-file floor" + ) + if aggregate_below: + lines.append( + f" src/ aggregate: {aggregate_rate:.1f}% lines " + f"({aggregate_hit}/{aggregate_found}) is below the " + f"{aggregate:g}% aggregate floor" + ) + else: + lines.append( + f"coverage_floor: ok -- {len(records)} files under src/, aggregate " + f"{aggregate_rate:.1f}% lines ({aggregate_hit}/{aggregate_found}), every file " + f"at or above {per_file:g}%, aggregate target {aggregate:g}%" + ) + lines.append("```") + lines.append("") + lines.append(render_table(records, markdown=True)) + lines.append("") + lines.append(DOC_END) + return "\n".join(lines) + + +def update_doc(path: Path, block: str) -> None: + """Replace the region between the markers in `path` with `block`.""" + lines = path.read_text(encoding="utf-8").splitlines() + try: + begin = next(i for i, line in enumerate(lines) if line.startswith(DOC_BEGIN)) + end = next(i for i, line in enumerate(lines) if line.startswith(DOC_END)) + except StopIteration: + raise ValueError(f"no {DOC_BEGIN} / {DOC_END} markers") from None + if end < begin: + raise ValueError("the end marker precedes the begin marker") + merged = lines[:begin] + block.splitlines() + lines[end + 1 :] + path.write_text("\n".join(merged) + "\n", encoding="utf-8") + + def main(argv: list[str]) -> int: parser = argparse.ArgumentParser( description="Fail when src/** coverage drops below the project's floors.", @@ -210,6 +279,14 @@ def main(argv: list[str]) -> int: default="table", help="table layout; markdown is what docs/coverage.md carries", ) + parser.add_argument( + "--write-doc", + metavar="PATH", + help=( + "rewrite the generated region of the markdown file at PATH from the " + "report, and exit 0; the floor verdict is written inside that region" + ), + ) args = parser.parse_args(argv) report = Path(args.report) @@ -235,6 +312,21 @@ def main(argv: list[str]) -> int: ) return 2 + if args.write_doc: + try: + update_doc( + Path(args.write_doc), + render_doc_block(records, args.per_file, args.aggregate), + ) + except (OSError, ValueError) as error: + print( + f"coverage_floor: cannot update {args.write_doc}: {error}", + file=sys.stderr, + ) + return 2 + print(f"coverage_floor: wrote the measured table to {args.write_doc}") + return 0 + if args.table: print(render_table(records, args.format == "markdown")) print()