From: Trevor Woerner <twoerner@gmail.com>
To: yocto-patches@lists.yoctoproject.org
Subject: [wic][PATCH v2 7/9] tests/docs: add the test-authoring guide
Date: Tue, 30 Jun 2026 12:06:10 -0400 [thread overview]
Message-ID: <20260630160612.1005451-8-twoerner@gmail.com> (raw)
In-Reply-To: <20260630160612.1005451-1-twoerner@gmail.com>
A suite is only as good as the tests added to it over time, so it needs
a guide that says where a new test goes, what one looks like, and how
to choose its inputs and assertions. This commit adds
tests/docs/authoring.md.
The guide covers:
- where a test goes: one test_<area>.py per wic module under
tests/unit/, extending an existing file or starting a new one;
- the shape of a test: a small worked example, a preference for
parametrize over in-test loops so each input/output pair is its own
named case, and use of the tmp_path fixture for scratch files;
- asserting the correct behaviour rather than whatever wic currently
does, so a test never bakes in a bug; when a test exposes a defect,
the source fix lands in the same change so the test passes;
- probing the boundaries: a checklist of input classes worth covering
for numbers, strings, paths, types/shape, and state, each asserting
a specific value or exception rather than merely "it did not
crash";
- keeping the assertion strong: never weaken an assertion to get
green;
- how to run just the test you wrote, with and without coverage.
AI-Generated: codex/claude-opus 4.7 (xhigh)
Signed-off-by: Trevor Woerner <twoerner@gmail.com>
---
changes in v2:
- v1 submitted the entire test suite as a single commit; v2 breaks
the work into a reviewable series, and this patch is one step of it.
---
tests/docs/authoring.md | 124 ++++++++++++++++++++++++++++++++++++++++
1 file changed, 124 insertions(+)
create mode 100644 tests/docs/authoring.md
diff --git a/tests/docs/authoring.md b/tests/docs/authoring.md
new file mode 100644
index 000000000000..c60cceb2d32d
--- /dev/null
+++ b/tests/docs/authoring.md
@@ -0,0 +1,124 @@
+# Authoring a unit test
+
+Unit tests live under `tests/unit/`, one module per area of wic, named
+`test_<area>.py`. They import the wic module under test directly and
+run in-process, so they need no host tools and no build environment.
+
+## Contents
+
+- [Where a test goes](#where-a-test-goes)
+- [Shape of a test](#shape-of-a-test)
+- [Assert the correct behaviour](#assert-the-correct-behaviour)
+- [Probe the boundaries](#probe-the-boundaries)
+- [Keep the assertion strong](#keep-the-assertion-strong)
+- [Running what you wrote](#running-what-you-wrote)
+
+## Where a test goes
+
+Group tests by the wic module they exercise, one `test_<area>.py` per
+module. Start a new file when you begin covering a module that has no
+file yet; otherwise extend the existing one.
+
+## Shape of a test
+
+```python
+import pytest
+
+from wic.ksparser import sizetype
+
+
+class TestSizetype:
+ # sizetype(default, size_in_bytes=False) returns a parser f(arg);
+ # with default "M", a bare number is read as mebibytes-in-KiB.
+ def test_plain_value_uses_default_unit(self):
+ parse = sizetype("M")
+ assert parse("100") == 100 * 1024
+
+ @pytest.mark.parametrize("arg,expected", [
+ ("1K", 1),
+ ("1G", 1 * 1024 * 1024),
+ ("0", 0),
+ ])
+ def test_suffixes(self, arg, expected):
+ assert sizetype("M")(arg) == expected
+```
+
+Prefer `@pytest.mark.parametrize` to express each input/output pair as
+its own case rather than looping inside one test, so a single bad
+value shows up as one named failure.
+
+Use `pytest`'s `tmp_path` fixture for any scratch files; a passing
+test's scratch directory is removed automatically (see the retention
+policy in `pyproject.toml`), and a failing one is kept for inspection.
+
+## Assert the correct behaviour
+
+Every assertion states the behaviour wic is *expected* to provide,
+never the behaviour it currently happens to have. A test that locks in
+a wrong result is worse than no test: it gives false confidence and it
+turns red exactly when someone repairs the code.
+
+When a test exposes a defect, the fix to the wic source lands in the
+same change as the test, so the test asserts the correct behaviour and
+passes. Do not commit a test that asserts a known-wrong result.
+
+Wrong -- bakes in the bug:
+
+```python
+# value lost its comma; asserting the broken output
+assert result["file"] == "s3://bucket/a"
+```
+
+Right -- asserts the correct output (and the fix lands with it):
+
+```python
+assert result["file"] == "s3://bucket/a,b.img"
+```
+
+## Probe the boundaries
+
+For a parameter, probe the boundary and just past it rather than only
+the comfortable middle of its range. The classes worth covering:
+
+- numbers: empty, zero, negative, the maximum and one beyond it,
+ non-power-of-two and non-multiple-of-1024/1000 values, primes,
+ fractional values where an integer is assumed, `0` versus `0.0`
+ versus `"0"`, off-by-one at a block or sector boundary, and very
+ large magnitudes near `2**31` and `2**63`
+- strings: the empty string, whitespace only, embedded spaces, tabs,
+ newlines and CRLF endings, non-ASCII characters, shell
+ metacharacters in a value that becomes part of a command line,
+ leading-dash values that look like options, over-delimited or
+ malformed forms (stray commas, doubled or missing separators), and
+ very long values
+- paths: traversal (`../`), doubled slashes, a trailing slash or its
+ absence, `.` and `..` components, broken or looping symlinks, and a
+ non-existent path
+- types and shape: each wrong type the parameter might receive (a
+ string where an integer is expected, a list where a string is
+ expected, `None`, and so on), the wrong argument count, and
+ duplicate keys or entries
+- state: a second call that must not see stale cached data, a
+ `cache=False` path that must evict, calling an operation twice, and
+ finalising a half-constructed object
+
+Each case asserts a specific outcome -- an exact value, or a specific
+exception -- rather than merely "it did not crash." A test that only
+checks for the absence of an exception will pass against badly wrong
+output.
+
+## Keep the assertion strong
+
+If a test is hard to make pass, the answer is in the code or in
+understanding the correct behaviour, never in weakening the assertion
+to something vague enough to pass. Loosening an assertion to get green
+is how a suite quietly stops catching regressions.
+
+## Running what you wrote
+
+```bash
+tests/run-tests.sh tests/unit/test_<area>.py -v
+tests/run-tests.sh --coverage tests/unit/test_<area>.py
+```
+
+A new test should leave the suite green, with zero failures.
--
2.50.0.173.g8b6f19ccfc3a
next prev parent reply other threads:[~2026-06-30 16:07 UTC|newest]
Thread overview: 10+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-06-30 16:06 [wic][PATCH v2 0/9] tests: standalone test-suite framework plus the first unit test Trevor Woerner
2026-06-30 16:06 ` [wic][PATCH v2 1/9] tests: add the standalone test-suite skeleton Trevor Woerner
2026-06-30 16:06 ` [wic][PATCH v2 2/9] tests: add a session banner via conftest.py Trevor Woerner
2026-06-30 16:06 ` [wic][PATCH v2 3/9] tests: add the run-tests.sh wrapper Trevor Woerner
2026-06-30 16:06 ` [wic][PATCH v2 4/9] tests: add optional coverage reporting to run-tests.sh Trevor Woerner
2026-06-30 16:06 ` [wic][PATCH v2 5/9] tests: add ruff linting " Trevor Woerner
2026-06-30 16:06 ` [wic][PATCH v2 6/9] tests/docs: add the suite overview README Trevor Woerner
2026-06-30 16:06 ` Trevor Woerner [this message]
2026-06-30 16:06 ` [wic][PATCH v2 8/9] tests: ignore E402 in the test tree for the sys.path bootstrap Trevor Woerner
2026-06-30 16:06 ` [wic][PATCH v2 9/9] tests/unit/test_bb_utils: test mkdirhier() and fix its missing errno import Trevor Woerner
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20260630160612.1005451-8-twoerner@gmail.com \
--to=twoerner@gmail.com \
--cc=yocto-patches@lists.yoctoproject.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).