docs(coverage): regenerate the measured table from the command that measures it
This commit is contained in:
1 parent
b7d0f901cc
commit
19c8d4368d
3 files changed
+147
-50
No files matched your search
+43
-48
@@ -3,6 +3,13 @@
|
||||
`plot` measures line coverage of the compiled library bodies under `src/` and enforces a floor on
|
||||
it, so a regression fails a command instead of quietly lowering a number in a document.
|
||||
|
||||
**The table and the floor verdict below are generated.** `tools/coverage.sh` rewrites the region
|
||||
between `<!-- BEGIN GENERATED: tools/coverage.sh -->` and
|
||||
`<!-- END GENERATED: tools/coverage.sh -->` from the report that run just captured, by calling
|
||||
`tools/coverage_floor.py --write-doc`. Do not edit inside those markers: the numbers there come
|
||||
from the command, not from a hand-typed copy, and a run on an unchanged tree rewrites the region
|
||||
with identical content.
|
||||
|
||||
## The one command
|
||||
|
||||
```sh
|
||||
@@ -12,12 +19,10 @@ cmake --build build/cov --target coverage # the same thing, through CMake
|
||||
|
||||
The script configures `build/cov` from the `linux-coverage` preset, builds it, runs the whole
|
||||
`ctest` suite, captures `src/**` with `lcov` into `build/cov/coverage.info`, renders the HTML report
|
||||
to `build/cov/html/index.html`, prints the per-file table below, and **exits non-zero when a floor
|
||||
is violated**. It needs `cmake`, `ninja`, `ctest`, `lcov`, `genhtml` and `python3`
|
||||
(Arch: `pacman -S cmake ninja lcov python`); it exits `2` with a message when one is missing.
|
||||
|
||||
A clean run of it — no build tree, all 26 translation units under `src/` compiled, 274 cases — takes
|
||||
about two minutes here, of which the suite is ~107 s.
|
||||
to `build/cov/html/index.html`, prints the per-file table, rewrites the generated region of this
|
||||
file, and **exits non-zero when a floor is violated**. It needs `cmake`, `ninja`, `ctest`, `lcov`,
|
||||
`genhtml` and `python3` (Arch: `pacman -S cmake ninja lcov python`); it exits `2` with a message
|
||||
when one is missing.
|
||||
|
||||
## The floors
|
||||
|
||||
@@ -35,18 +40,20 @@ about two minutes here, of which the suite is ~107 s.
|
||||
|
||||
## The measured table
|
||||
|
||||
Measured on 2026-09-28 with GCC 16.2.1 from a clean `build/cov`, over the full test suite.
|
||||
`coverage.sh` prints exactly this table on every run.
|
||||
<!-- BEGIN GENERATED: tools/coverage.sh -->
|
||||
|
||||
```
|
||||
coverage_floor: ok -- 23 files under src/, aggregate 95.3% lines (3284/3447), every file at or above 80%, aggregate target 90%
|
||||
```
|
||||
|
||||
<!-- BEGIN GENERATED TABLE (tools/coverage_floor.py --table --format markdown) -->
|
||||
| File | Lines | Line rate | Functions |
|
||||
| --- | ---: | ---: | ---: |
|
||||
| `src/data/array_series.cpp` | 43/43 | 100.0% | 100.0% |
|
||||
| `src/data/minmax_pyramid.cpp` | 163/163 | 100.0% | 100.0% |
|
||||
| `src/data/series.cpp` | 105/107 | 98.1% | 100.0% |
|
||||
| `src/data/stream_series.cpp` | 39/39 | 100.0% | 100.0% |
|
||||
| `src/gl/handle.cpp` | 214/256 | 83.6% | 100.0% |
|
||||
| `src/gl/loader.cpp` | 53/81 | 65.4% | 77.8% |
|
||||
| `src/gl/handle.cpp` | 256/256 | 100.0% | 100.0% |
|
||||
| `src/gl/loader.cpp` | 81/81 | 100.0% | 100.0% |
|
||||
| `src/input/controller.cpp` | 327/341 | 95.9% | 100.0% |
|
||||
| `src/math/transform.cpp` | 28/28 | 100.0% | 100.0% |
|
||||
| `src/plot/axis.cpp` | 12/12 | 100.0% | 100.0% |
|
||||
@@ -55,25 +62,18 @@ Measured on 2026-09-28 with GCC 16.2.1 from a clean `build/cov`, over the full t
|
||||
| `src/plot/measure.cpp` | 21/21 | 100.0% | 100.0% |
|
||||
| `src/plot/plot.cpp` | 733/753 | 97.3% | 98.9% |
|
||||
| `src/plot/ticks.cpp` | 93/99 | 93.9% | 100.0% |
|
||||
| `src/render/buffer.cpp` | 87/98 | 88.8% | 100.0% |
|
||||
| `src/render/buffer.cpp` | 88/98 | 89.8% | 100.0% |
|
||||
| `src/render/embedded_mono_font.cpp` | 45/45 | 100.0% | 100.0% |
|
||||
| `src/render/primitives.cpp` | 71/75 | 94.7% | 100.0% |
|
||||
| `src/render/program.cpp` | 26/27 | 96.3% | 100.0% |
|
||||
| `src/render/renderer.cpp` | 529/604 | 87.6% | 93.7% |
|
||||
| `src/render/text.cpp` | 27/30 | 90.0% | 100.0% |
|
||||
| `src/render/text_atlas.cpp` | 40/41 | 97.6% | 100.0% |
|
||||
| `src/render/vertex_array.cpp` | 49/59 | 83.1% | 100.0% |
|
||||
| `src/render/vertex_array.cpp` | 59/59 | 100.0% | 100.0% |
|
||||
| `src/text/true_type_font.cpp` | 270/291 | 92.8% | 96.8% |
|
||||
| **src/ aggregate** | **3203/3447** | **92.9%** | **97.8%** |
|
||||
<!-- END GENERATED TABLE -->
|
||||
| **src/ aggregate** | **3284/3447** | **95.3%** | **98.3%** |
|
||||
|
||||
**The floor currently fails, and that is the point.** `src/gl/loader.cpp` sits at 65.4% lines
|
||||
(53/81) — below the 80% per-file floor — so `tools/coverage.sh` exits `1` on today's tree. What is
|
||||
uncovered there is the loader's error paths and its extension scanner; `src/gl/handle.cpp` (83.6%)
|
||||
and `src/render/vertex_array.cpp` (83.1%) are the same kind of gap, just above the line. Those tests
|
||||
are Task 2 of `docs/superpowers/plans/2026-09-28-plot-docs-and-coverage.md`; the floor went in first
|
||||
so the fix has something to satisfy rather than something to negotiate with, and so the *next*
|
||||
regression in that area is a failing command rather than a number nobody reads.
|
||||
<!-- END GENERATED: tools/coverage.sh -->
|
||||
|
||||
## What the report measures
|
||||
|
||||
@@ -118,40 +118,35 @@ before `genhtml` re-creates it.
|
||||
per-file `LF`/`LH`/`FNF`/`FNH` tallies that are identical for all 23 files — verified by diffing the
|
||||
two tracefiles — but the per-line hit *counts* differ by a few parts per million, because a handful
|
||||
of tests drive a producer thread whose iteration count follows wall-clock timing. Everything the
|
||||
floor reads (the `LF`/`LH` tallies, and therefore every rate in the table above) is unaffected.
|
||||
floor reads (the `LF`/`LH` tallies, and therefore every rate in the generated table above) is
|
||||
unaffected.
|
||||
|
||||
## Adjusting the floors, and seeing the gate fail
|
||||
|
||||
Both floors are flags, which is how the failure path is demonstrated — and how a later task tightens
|
||||
the contract. Against `build/cov/coverage.info` from the table above:
|
||||
the contract. Against the report in the generated table above, a floor raised above every file's
|
||||
rate fails the check and names each offender:
|
||||
|
||||
```sh
|
||||
$ python3 tools/coverage_floor.py build/cov/coverage.info # the real floors
|
||||
coverage_floor: FAILED -- 1 floor(s) violated in build/cov/coverage.info:
|
||||
src/gl/loader.cpp: 65.4% lines (53/81) is below the 80% per-file floor
|
||||
$ echo $?
|
||||
1
|
||||
# Any file not at 100% fails: a floor that is guaranteed to bite, whatever the
|
||||
# current rates are. The list of offenders is the files in the table above whose
|
||||
# rate is below 100%.
|
||||
python3 tools/coverage_floor.py --per-file 100 build/cov/coverage.info
|
||||
# coverage_floor: FAILED -- <n> floor(s) violated in build/cov/coverage.info:
|
||||
# <file>: <rate>% lines (<hit>/<found>) is below the 100% per-file floor
|
||||
# ...
|
||||
# $? == 1
|
||||
|
||||
$ python3 tools/coverage_floor.py --per-file 95 build/cov/coverage.info # floor raised above rates
|
||||
coverage_floor: FAILED -- 10 floor(s) violated in build/cov/coverage.info:
|
||||
src/gl/handle.cpp: 83.6% lines (214/256) is below the 95% per-file floor
|
||||
src/gl/loader.cpp: 65.4% lines (53/81) is below the 95% per-file floor
|
||||
src/plot/grid.cpp: 93.8% lines (15/16) is below the 95% per-file floor
|
||||
src/plot/ticks.cpp: 93.9% lines (93/99) is below the 95% per-file floor
|
||||
src/render/buffer.cpp: 88.8% lines (87/98) is below the 95% per-file floor
|
||||
src/render/primitives.cpp: 94.7% lines (71/75) is below the 95% per-file floor
|
||||
src/render/renderer.cpp: 87.6% lines (529/604) is below the 95% per-file floor
|
||||
src/render/text.cpp: 90.0% lines (27/30) is below the 95% per-file floor
|
||||
src/render/vertex_array.cpp: 83.1% lines (49/59) is below the 95% per-file floor
|
||||
src/text/true_type_font.cpp: 92.8% lines (270/291) is below the 95% per-file floor
|
||||
$ echo $?
|
||||
1
|
||||
# The same, for the aggregate: a target above the measured aggregate fails with one line.
|
||||
python3 tools/coverage_floor.py --aggregate 99 build/cov/coverage.info
|
||||
# coverage_floor: FAILED -- 1 floor(s) violated in build/cov/coverage.info:
|
||||
# src/ aggregate: <rate>% lines (<hit>/<found>) is below the 99% aggregate floor
|
||||
# $? == 1
|
||||
|
||||
$ python3 tools/coverage_floor.py --per-file 60 --aggregate 90 build/cov/coverage.info
|
||||
coverage_floor: ok -- 23 files under src/, aggregate 92.9% lines (3203/3447), every file at or
|
||||
above 60%, aggregate target 90%
|
||||
$ echo $?
|
||||
0
|
||||
# A floor at or below the measured rates passes.
|
||||
python3 tools/coverage_floor.py build/cov/coverage.info
|
||||
# coverage_floor: ok -- ...
|
||||
# $? == 0
|
||||
```
|
||||
|
||||
## Why the check is a CMake target and not a ctest case
|
||||
|
||||
+12
-2
@@ -6,8 +6,10 @@
|
||||
# capture src/** with lcov into build/cov/coverage.info, render the HTML report
|
||||
# under build/cov/html/, print the per-file table, and exit non-zero when
|
||||
# tools/coverage_floor.py finds a file below 80% lines or the src/ aggregate
|
||||
# below 90%. The numbers and the floors are the contract docs/coverage.md
|
||||
# states; `docs/coverage.md` also records how to re-read the table from here.
|
||||
# below 90%. The floors are tools/coverage_floor.py's constants; the measured
|
||||
# table and the floor verdict are written back into docs/coverage.md's generated
|
||||
# region by `coverage_floor.py --write-doc`, from the very report this run
|
||||
# captured, so that document cannot drift from the command that measures it.
|
||||
#
|
||||
# Idempotent by construction: gcov counters accumulate in .gcda files, so a
|
||||
# second run that inherited the first run's counters would report a superset of
|
||||
@@ -31,6 +33,7 @@ readonly FLOOR="$ROOT/tools/coverage_floor.py"
|
||||
readonly RAW="$BUILD/coverage.raw.info"
|
||||
readonly INFO="$BUILD/coverage.info"
|
||||
readonly HTML="$BUILD/html"
|
||||
readonly DOC="$ROOT/docs/coverage.md"
|
||||
|
||||
log() { printf '==> %s\n' "$*"; }
|
||||
|
||||
@@ -87,6 +90,13 @@ python3 "$FLOOR" --table "$INFO"
|
||||
readonly FLOOR_STATUS=$?
|
||||
set -e
|
||||
|
||||
# The measured table is written into the document that states it, from the very
|
||||
# report this run captured, so docs/coverage.md cannot drift from the command
|
||||
# that produces it. This happens before the floor gate below, so a failing tree
|
||||
# still gets an honest table (and its failure recorded) in the document.
|
||||
log "regenerating the measured table in $DOC"
|
||||
python3 "$FLOOR" --write-doc "$DOC" "$INFO"
|
||||
|
||||
if (( FLOOR_STATUS != 0 )); then
|
||||
printf '\ncoverage: FLOOR FAILED (exit %d) -- report %s, HTML %s/index.html\n' \
|
||||
"$FLOOR_STATUS" "$INFO" "$HTML" >&2
|
||||
|
||||
@@ -184,6 +184,75 @@ def render_table(records: list[Record], markdown: bool) -> str:
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
# The generated region docs/coverage.md carries. tools/coverage.sh calls this
|
||||
# tool with --write-doc on every run, so the table in that document is the one
|
||||
# the command that measured it just captured, never a number copied by hand.
|
||||
DOC_BEGIN = "<!-- BEGIN GENERATED: tools/coverage.sh -->"
|
||||
DOC_END = "<!-- END GENERATED: tools/coverage.sh -->"
|
||||
|
||||
|
||||
def render_doc_block(records: list[Record], per_file: float, aggregate: float) -> str:
|
||||
"""The region spliced into docs/coverage.md: the floor verdict and the table."""
|
||||
failures = [
|
||||
record
|
||||
for record in records
|
||||
if record.lines_found > 0
|
||||
and record.lines_hit * 100.0 < per_file * record.lines_found
|
||||
]
|
||||
aggregate_hit = sum(record.lines_hit for record in records)
|
||||
aggregate_found = sum(record.lines_found for record in records)
|
||||
aggregate_rate = (
|
||||
100.0 * aggregate_hit / aggregate_found if aggregate_found else 0.0
|
||||
)
|
||||
aggregate_below = aggregate_hit * 100.0 < aggregate * aggregate_found
|
||||
|
||||
lines = [DOC_BEGIN, ""]
|
||||
lines.append("```")
|
||||
if failures or aggregate_below:
|
||||
lines.append(
|
||||
f"coverage_floor: FAILED -- {len(failures) + (1 if aggregate_below else 0)} "
|
||||
"floor(s) violated in the report this run captured:"
|
||||
)
|
||||
for record in failures:
|
||||
lines.append(
|
||||
f" {record.display}: {record.rate:.1f}% lines "
|
||||
f"({record.lines_hit}/{record.lines_found}) is below the "
|
||||
f"{per_file:g}% per-file floor"
|
||||
)
|
||||
if aggregate_below:
|
||||
lines.append(
|
||||
f" src/ aggregate: {aggregate_rate:.1f}% lines "
|
||||
f"({aggregate_hit}/{aggregate_found}) is below the "
|
||||
f"{aggregate:g}% aggregate floor"
|
||||
)
|
||||
else:
|
||||
lines.append(
|
||||
f"coverage_floor: ok -- {len(records)} files under src/, aggregate "
|
||||
f"{aggregate_rate:.1f}% lines ({aggregate_hit}/{aggregate_found}), every file "
|
||||
f"at or above {per_file:g}%, aggregate target {aggregate:g}%"
|
||||
)
|
||||
lines.append("```")
|
||||
lines.append("")
|
||||
lines.append(render_table(records, markdown=True))
|
||||
lines.append("")
|
||||
lines.append(DOC_END)
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
def update_doc(path: Path, block: str) -> None:
|
||||
"""Replace the region between the markers in `path` with `block`."""
|
||||
lines = path.read_text(encoding="utf-8").splitlines()
|
||||
try:
|
||||
begin = next(i for i, line in enumerate(lines) if line.startswith(DOC_BEGIN))
|
||||
end = next(i for i, line in enumerate(lines) if line.startswith(DOC_END))
|
||||
except StopIteration:
|
||||
raise ValueError(f"no {DOC_BEGIN} / {DOC_END} markers") from None
|
||||
if end < begin:
|
||||
raise ValueError("the end marker precedes the begin marker")
|
||||
merged = lines[:begin] + block.splitlines() + lines[end + 1 :]
|
||||
path.write_text("\n".join(merged) + "\n", encoding="utf-8")
|
||||
|
||||
|
||||
def main(argv: list[str]) -> int:
|
||||
parser = argparse.ArgumentParser(
|
||||
description="Fail when src/** coverage drops below the project's floors.",
|
||||
@@ -210,6 +279,14 @@ def main(argv: list[str]) -> int:
|
||||
default="table",
|
||||
help="table layout; markdown is what docs/coverage.md carries",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--write-doc",
|
||||
metavar="PATH",
|
||||
help=(
|
||||
"rewrite the generated region of the markdown file at PATH from the "
|
||||
"report, and exit 0; the floor verdict is written inside that region"
|
||||
),
|
||||
)
|
||||
args = parser.parse_args(argv)
|
||||
|
||||
report = Path(args.report)
|
||||
@@ -235,6 +312,21 @@ def main(argv: list[str]) -> int:
|
||||
)
|
||||
return 2
|
||||
|
||||
if args.write_doc:
|
||||
try:
|
||||
update_doc(
|
||||
Path(args.write_doc),
|
||||
render_doc_block(records, args.per_file, args.aggregate),
|
||||
)
|
||||
except (OSError, ValueError) as error:
|
||||
print(
|
||||
f"coverage_floor: cannot update {args.write_doc}: {error}",
|
||||
file=sys.stderr,
|
||||
)
|
||||
return 2
|
||||
print(f"coverage_floor: wrote the measured table to {args.write_doc}")
|
||||
return 0
|
||||
|
||||
if args.table:
|
||||
print(render_table(records, args.format == "markdown"))
|
||||
print()
|
||||
|
||||
Reference in new issue
Block a user