All of lore.kernel.org
 help / color / mirror / Atom feed
From: Trevor Woerner <twoerner@gmail.com>
To: yocto-patches@lists.yoctoproject.org
Subject: [wic][PATCH v3 04/10] tests: add optional coverage reporting to run-tests.sh
Date: Wed,  1 Jul 2026 03:40:24 -0400	[thread overview]
Message-ID: <20260701074030.1090807-5-twoerner@gmail.com> (raw)
In-Reply-To: <20260701074030.1090807-1-twoerner@gmail.com>

Knowing which lines a test run actually exercised is the difference
between "the suite is green" and "the suite is green and we know what
it touched". This commit wires branch coverage of the wic source into
the runner, kept entirely opt-in so a plain run stays fast and quiet.

pyproject.toml gains coverage and pytest-cov in the tests extra, so
"pip install -e .[tests]" pulls in what the new flags need.

run-tests.sh gains two options:

  - --coverage measures branch coverage of src/wic during the run and
    prints a terminal report listing the lines that were missed;

  - --html [DIR] additionally writes a browsable HTML report (default
    htmlcov/, or DIR if given) and implies --coverage.

If --coverage is requested but pytest-cov is not installed the runner
fails loudly with the install command rather than running without the
measurement it was asked for.

The coverage data file and the HTML report directory are build
artifacts, so .gitignore learns to ignore .coverage and htmlcov/.

AI-Generated: codex/claude-opus 4.7 (xhigh)
Signed-off-by: Trevor Woerner <twoerner@gmail.com>
---
changes in v3:
- no change in this revision.
changes in v2:
- v1 submitted the entire test suite as a single commit; v2 breaks
  the work into a reviewable series, and this patch is one step of it.
---
 .gitignore         |  4 ++++
 pyproject.toml     |  2 ++
 tests/run-tests.sh | 39 ++++++++++++++++++++++++++++++++++++++-
 3 files changed, 44 insertions(+), 1 deletion(-)

diff --git a/.gitignore b/.gitignore
index 07992096c0fb..534c49538091 100644
--- a/.gitignore
+++ b/.gitignore
@@ -2,3 +2,7 @@
 
 # pytest cache
 /.pytest_cache/
+
+# coverage data and reports
+/.coverage
+/htmlcov/
diff --git a/pyproject.toml b/pyproject.toml
index d660e39007c4..ece2757bb686 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -24,6 +24,8 @@ Repository = "https://git.yoctoproject.org/wic"
 [project.optional-dependencies]
 tests = [
     "pytest >= 7.0",
+    "coverage >= 7.0",
+    "pytest-cov >= 4.0",
 ]
 
 [project.scripts]
diff --git a/tests/run-tests.sh b/tests/run-tests.sh
index 9db6d50338d6..a483da6a63a6 100755
--- a/tests/run-tests.sh
+++ b/tests/run-tests.sh
@@ -13,9 +13,13 @@ set -euo pipefail
 usage() {
     cat <<'USAGE'
 Usage:
-  tests/run-tests.sh [pytest args]
+  tests/run-tests.sh [--coverage] [--html [DIR]] [pytest args]
 
 Options:
+  --coverage    also measure branch coverage of src/wic and print a
+                terminal report listing the lines that were missed
+  --html [DIR]  also write an HTML coverage report (default dir:
+                htmlcov/); implies --coverage
   -h, --help    show this help and exit
 
 Anything else is passed straight through to pytest (for example a
@@ -24,6 +28,9 @@ tests/ is run.
 
 Examples:
   tests/run-tests.sh                  # whole suite
+  tests/run-tests.sh --coverage       # + terminal coverage report
+  tests/run-tests.sh --html           # + HTML report in htmlcov/
+  tests/run-tests.sh --html /tmp/cov  # + HTML report in /tmp/cov
   tests/run-tests.sh -k filemap -v    # pass args through to pytest
   tests/run-tests.sh tests/unit       # a single tier or file
 
@@ -34,9 +41,25 @@ USAGE
 REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
 PY="${PYTHON:-python3}"
 
+coverage=0
+html=0
+html_dir="htmlcov"
 pytest_args=()
 while [ $# -gt 0 ]; do
     case "$1" in
+        --coverage)
+            coverage=1
+            ;;
+        --html)
+            coverage=1
+            html=1
+            # Optional directory argument: consume the next token only if
+            # it is not another option or the pytest pass-through marker.
+            case "${2:-}" in
+                ""|-*|--) ;;
+                *) html_dir="$2"; shift ;;
+            esac
+            ;;
         -h|--help)
             usage
             exit 0
@@ -72,4 +95,18 @@ if [ ${#pytest_args[@]} -eq 0 ]; then
     pytest_args=("tests")
 fi
 
+if [ "$coverage" -eq 1 ]; then
+    if ! "$PY" -c "import pytest_cov" >/dev/null 2>&1; then
+        echo "error: coverage requested but pytest-cov is not installed." >&2
+        echo "       run: $PY -m pip install -e \".[tests]\"" >&2
+        exit 1
+    fi
+    cov_args=(--cov=wic --cov-branch --cov-report=term-missing)
+    if [ "$html" -eq 1 ]; then
+        cov_args+=(--cov-report="html:${html_dir}")
+        echo "HTML coverage report: ${html_dir}/index.html" >&2
+    fi
+    exec "$PY" -m pytest "${pytest_args[@]}" "${cov_args[@]}"
+fi
+
 exec "$PY" -m pytest "${pytest_args[@]}"
-- 
2.50.0.173.g8b6f19ccfc3a



  parent reply	other threads:[~2026-07-01  7:40 UTC|newest]

Thread overview: 21+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-07-01  7:40 [wic][PATCH v3 00/10] tests: standalone test-suite framework plus the first unit test Trevor Woerner
2026-07-01  7:40 ` [wic][PATCH v3 01/10] tests: add the standalone test-suite skeleton Trevor Woerner
2026-07-06  8:18   ` [yocto-patches] " Paul Barker
2026-07-01  7:40 ` [wic][PATCH v3 02/10] tests: add a session banner via conftest.py Trevor Woerner
2026-07-06  8:17   ` [yocto-patches] " Paul Barker
2026-07-01  7:40 ` [wic][PATCH v3 03/10] tests: add the run-tests.sh wrapper Trevor Woerner
2026-07-06  8:21   ` [yocto-patches] " Paul Barker
2026-07-01  7:40 ` Trevor Woerner [this message]
2026-07-06  8:33   ` [yocto-patches] [wic][PATCH v3 04/10] tests: add optional coverage reporting to run-tests.sh Paul Barker
2026-07-01  7:40 ` [wic][PATCH v3 05/10] tests: add ruff linting " Trevor Woerner
2026-07-06  8:40   ` [yocto-patches] " Paul Barker
2026-07-01  7:40 ` [wic][PATCH v3 06/10] tests/docs: add the suite overview README Trevor Woerner
2026-07-06  8:47   ` [yocto-patches] " Paul Barker
2026-07-01  7:40 ` [wic][PATCH v3 07/10] tests/docs: add the test-authoring guide Trevor Woerner
2026-07-06  8:52   ` [yocto-patches] " Paul Barker
2026-07-01  7:40 ` [wic][PATCH v3 08/10] tests: ignore E402 in the test tree for the sys.path bootstrap Trevor Woerner
2026-07-06  8:55   ` [yocto-patches] " Paul Barker
2026-07-01  7:40 ` [wic][PATCH v3 09/10] tests/unit/test_bb_utils: test mkdirhier() and fix its missing errno import Trevor Woerner
2026-07-06  9:06   ` [yocto-patches] " Paul Barker
2026-07-01  7:40 ` [wic][PATCH v3 10/10] tests/docs: add the review rubric Trevor Woerner
2026-07-06  9:08   ` [yocto-patches] " Paul Barker

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20260701074030.1090807-5-twoerner@gmail.com \
    --to=twoerner@gmail.com \
    --cc=yocto-patches@lists.yoctoproject.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.