162 lines
9.0 KiB
Markdown
162 lines
9.0 KiB
Markdown
# Coverage
|
|
|
|
`plot` measures line coverage of the compiled library bodies under `src/` and enforces a floor on
|
|
it, so a regression fails a command instead of quietly lowering a number in a document.
|
|
|
|
**The table and the floor verdict below are generated.** `tools/coverage.sh` rewrites the region
|
|
between `<!-- BEGIN GENERATED: tools/coverage.sh -->` and
|
|
`<!-- END GENERATED: tools/coverage.sh -->` from the report that run just captured, by calling
|
|
`tools/coverage_floor.py --write-doc`. Do not edit inside those markers: the numbers there come
|
|
from the command, not from a hand-typed copy, and a run on an unchanged tree rewrites the region
|
|
with identical content.
|
|
|
|
## The one command
|
|
|
|
```sh
|
|
tools/coverage.sh # from the repository root
|
|
cmake --build build/cov --target coverage # the same thing, through CMake
|
|
```
|
|
|
|
The script configures `build/cov` from the `linux-coverage` preset, builds it, runs the whole
|
|
`ctest` suite, captures `src/**` with `lcov` into `build/cov/coverage.info`, renders the HTML report
|
|
to `build/cov/html/index.html`, prints the per-file table, rewrites the generated region of this
|
|
file, and **exits non-zero when a floor is violated**. It needs `cmake`, `ninja`, `ctest`, `lcov`,
|
|
`genhtml` and `python3` (Arch: `pacman -S cmake ninja lcov python`); it exits `2` with a message
|
|
when one is missing.
|
|
|
|
## The floors
|
|
|
|
| Scope | Floor | Enforced by |
|
|
| --- | --- | --- |
|
|
| Every file under `src/` | **80%** of its instrumented lines | `tools/coverage_floor.py` (`--per-file`, default 80) |
|
|
| `src/` as a whole | **90%** of instrumented lines | `tools/coverage_floor.py` (`--aggregate`, default 90) |
|
|
|
|
`tools/coverage_floor.py build/cov/coverage.info` reads the lcov tracefile on its own and exits:
|
|
|
|
* `0` — every floor holds;
|
|
* `1` — a floor is violated; every offending file is printed with its rate and counts;
|
|
* `2` — the report cannot be read at all (missing file, or no `src/` records in it), because a
|
|
report this tool cannot find must never read as a pass.
|
|
|
|
## The measured table
|
|
|
|
<!-- BEGIN GENERATED: tools/coverage.sh -->
|
|
|
|
```
|
|
coverage_floor: ok -- 23 files under src/, aggregate 95.3% lines (3284/3447), every file at or above 80%, aggregate target 90%
|
|
```
|
|
|
|
| File | Lines | Line rate | Functions |
|
|
| --- | ---: | ---: | ---: |
|
|
| `src/data/array_series.cpp` | 43/43 | 100.0% | 100.0% |
|
|
| `src/data/minmax_pyramid.cpp` | 163/163 | 100.0% | 100.0% |
|
|
| `src/data/series.cpp` | 105/107 | 98.1% | 100.0% |
|
|
| `src/data/stream_series.cpp` | 39/39 | 100.0% | 100.0% |
|
|
| `src/gl/handle.cpp` | 256/256 | 100.0% | 100.0% |
|
|
| `src/gl/loader.cpp` | 81/81 | 100.0% | 100.0% |
|
|
| `src/input/controller.cpp` | 327/341 | 95.9% | 100.0% |
|
|
| `src/math/transform.cpp` | 28/28 | 100.0% | 100.0% |
|
|
| `src/plot/axis.cpp` | 12/12 | 100.0% | 100.0% |
|
|
| `src/plot/figure.cpp` | 213/218 | 97.7% | 95.5% |
|
|
| `src/plot/grid.cpp` | 15/16 | 93.8% | 100.0% |
|
|
| `src/plot/measure.cpp` | 21/21 | 100.0% | 100.0% |
|
|
| `src/plot/plot.cpp` | 733/753 | 97.3% | 98.9% |
|
|
| `src/plot/ticks.cpp` | 93/99 | 93.9% | 100.0% |
|
|
| `src/render/buffer.cpp` | 88/98 | 89.8% | 100.0% |
|
|
| `src/render/embedded_mono_font.cpp` | 45/45 | 100.0% | 100.0% |
|
|
| `src/render/primitives.cpp` | 71/75 | 94.7% | 100.0% |
|
|
| `src/render/program.cpp` | 26/27 | 96.3% | 100.0% |
|
|
| `src/render/renderer.cpp` | 529/604 | 87.6% | 93.7% |
|
|
| `src/render/text.cpp` | 27/30 | 90.0% | 100.0% |
|
|
| `src/render/text_atlas.cpp` | 40/41 | 97.6% | 100.0% |
|
|
| `src/render/vertex_array.cpp` | 59/59 | 100.0% | 100.0% |
|
|
| `src/text/true_type_font.cpp` | 270/291 | 92.8% | 96.8% |
|
|
| **src/ aggregate** | **3284/3447** | **95.3%** | **98.3%** |
|
|
|
|
<!-- END GENERATED: tools/coverage.sh -->
|
|
|
|
## What the report measures
|
|
|
|
* `src/**` — the compiled library bodies. These are what a caller of `plot::core` executes.
|
|
* Lines and functions. **Branch coverage is off** (`--rc branch_coverage=0`): the data-structure and
|
|
GL code is dense with short-circuit conditions whose branch counts mostly measure how the tests
|
|
were written, so a branch floor would be noise rather than a contract.
|
|
* Three of the 26 translation units produce no record at all, because gcov only writes counters for
|
|
code that runs: `src/core/status.cpp` (`status.o` is never pulled out of `libplot_core.a` — no
|
|
test calls `plot::to_string`), `src/render/shaders.cpp` (string literals and inline definitions),
|
|
and `src/text/stb_truetype_impl.cpp` (its lines belong to the vendored
|
|
`third_party/stb_truetype.h`, which is filtered out). Should a record ever arrive with `LF:0`, the
|
|
checker exempts that file from the per-file floor rather than dividing by zero.
|
|
* Not measured: the public headers under `include/plot/` (inline and template code, exercised
|
|
through their callers but not held to a per-file rate); `tests/` and the generated header-check
|
|
translation units; `adapters/` — only the two header-only seams the unit suite drives appear in a
|
|
raw capture, and the adapter bodies need a window, which the demos cover and this report does not;
|
|
and the C++ standard library headers gcov also records.
|
|
|
|
## Reproducibility
|
|
|
|
gcov's counters accumulate in `.gcda` files, so a run that inherits the previous run's counters
|
|
reports a superset of what it actually exercised — the classic reason a coverage number drifts on a
|
|
second run with no source change, and here it is measurable: `build/cov` before this script existed
|
|
had been reused across revisions, and its accumulated counters reported **93.0% (3206/3447)** for
|
|
the same tree that a clean single run reports as **92.9% (3203/3447)**. Three lines of hit in
|
|
`plot.cpp`, `renderer.cpp` and `buffer.cpp` were left over from an earlier build.
|
|
|
|
`tools/coverage.sh` therefore deletes every `.gcda` under the build tree
|
|
(`find "$BUILD" -name '*.gcda' -delete`) **before** it runs the suite, so the captured counters are
|
|
exactly the ones that run produced. `lcov --zerocounters --directory "$BUILD"` is the equivalent
|
|
call — on lcov 2.5 it unlinks the `.gcda` files and reports success even on a tree that has none
|
|
(checked) — and the script keeps the `find` because it is one explicit step whose meaning does not
|
|
depend on the lcov version, and because clearing counters must never be the thing that breaks a
|
|
fresh tree. The capture also runs with `--rc geninfo_unexecuted_blocks=1` (the "unexecuted
|
|
block" notes come from libstdc++ headers on this toolchain; this records those blocks as
|
|
unexecuted, which is what they are) and `--ignore-errors inconsistent,inconsistent` for the same
|
|
source. `lcov --extract` then narrows the tracefile to `src/**`, and the HTML directory is removed
|
|
before `genhtml` re-creates it.
|
|
|
|
**The numbers are stable; the raw counts are not.** Two consecutive runs on the same tree produce
|
|
per-file `LF`/`LH`/`FNF`/`FNH` tallies that are identical for all 23 files — verified by diffing the
|
|
two tracefiles — but the per-line hit *counts* differ by a few parts per million, because a handful
|
|
of tests drive a producer thread whose iteration count follows wall-clock timing. Everything the
|
|
floor reads (the `LF`/`LH` tallies, and therefore every rate in the generated table above) is
|
|
unaffected.
|
|
|
|
## Adjusting the floors, and seeing the gate fail
|
|
|
|
Both floors are flags, which is how the failure path is demonstrated — and how a later task tightens
|
|
the contract. Against the report in the generated table above, a floor raised above every file's
|
|
rate fails the check and names each offender:
|
|
|
|
```sh
|
|
# Any file not at 100% fails: a floor that is guaranteed to bite, whatever the
|
|
# current rates are. The list of offenders is the files in the table above whose
|
|
# rate is below 100%.
|
|
python3 tools/coverage_floor.py --per-file 100 build/cov/coverage.info
|
|
# coverage_floor: FAILED -- <n> floor(s) violated in build/cov/coverage.info:
|
|
# <file>: <rate>% lines (<hit>/<found>) is below the 100% per-file floor
|
|
# ...
|
|
# $? == 1
|
|
|
|
# The same, for the aggregate: a target above the measured aggregate fails with one line.
|
|
python3 tools/coverage_floor.py --aggregate 99 build/cov/coverage.info
|
|
# coverage_floor: FAILED -- 1 floor(s) violated in build/cov/coverage.info:
|
|
# src/ aggregate: <rate>% lines (<hit>/<found>) is below the 99% aggregate floor
|
|
# $? == 1
|
|
|
|
# A floor at or below the measured rates passes.
|
|
python3 tools/coverage_floor.py build/cov/coverage.info
|
|
# coverage_floor: ok -- ...
|
|
# $? == 0
|
|
```
|
|
|
|
## Why the check is a CMake target and not a ctest case
|
|
|
|
A ctest entry registered only in the coverage preset would either fail in a freshly configured tree
|
|
— no report exists until `tools/coverage.sh` has run once, so the preset's own clean-tree command
|
|
(`cmake --preset linux-coverage && cmake --build build/cov && ctest --test-dir build/cov`) would
|
|
then fail for a reason that has nothing to do with the code — or it would have to build the library
|
|
and re-run the whole suite inside the test, duplicating the script. A custom target instead: it is
|
|
not part of `all`, so it cannot slow or destabilise the `linux-debug`, `linux-asan` or
|
|
`linux-release` builds, and it is not even created in those presets, because they leave
|
|
`PLOT_COVERAGE` off and the option's guard rejects a tree that carries no `--coverage` flags.
|