Re: [PATCH v2 4/6] automation/qtb: add unit tests for the QTB framework

[Oleksii Kurochko <oleksii.kurochko@gmail.com>]

On 8/27/26 11:42 AM, Baptiste Le Duc wrote:
> The QTB framework is meant to be extended: new test types, new machines and
> new device trees aim to be added by other people.
> 
> Add pytest coverage of the framework's functions, and of the console-test
> type's config validation and expect/retry loop, so that such changes get
> immediate feedback and existing behaviour does not silently regress.

Nothing runs these tests, though: there is no CI job for them, and the
container from patch 1 doesn't even have pytest. So the feedback only
comes if whoever changes the framework has pytest and the QTB fork
installed locally and remembers to run them.

Probably, we want to consider to have a separate job for that (mainly 
question to CI maintainers then)?

Nit: probably "immedidate feedback" will be clearer if this part ...
> 
> The suite covers 100% of the framework's statements, so a new code path
> added without a test shows up as a coverage drop.

Running the suggested pytest-cov command gives 99% here:

Name                                                        Stmts   Miss 
  Cover   Missing
-----------------------------------------------------------------------------------------
...
automation/scripts/qtb/riscv/paths.py                          32      1 
    97%   50
...
automation/scripts/qtb/riscv/xen_dt.py                         26      1 
    96%   21
-----------------------------------------------------------------------------------------
TOTAL                                                         592      2 
    99%

============================== 51 passed in 0.22s 
==============================

Not a problem as
such, but the commit message shouldn't claim 100%.

> 
> The tests are meant to be run locally, from the Xen tree root:
... and part below will be moved up.>
>      python3 -m pytest automation/scripts/qtb/riscv/unit/
> 
> and, with pytest-cov installed, the coverage report is:
> 
>      python3 -m pytest --cov=automation/scripts/qtb \
>          automation/scripts/qtb/riscv/unit/
> 
> The tests drive fakes rather than QEMU, so they need no artifacts to run.
> 
> Assisted-by: Claude:claude-opus-5
> Signed-off-by: Baptiste Le Duc <baptiste.le-duc@vates.tech>
> ---
>   automation/scripts/qtb/riscv/unit/__init__.py |   2 +
>   automation/scripts/qtb/riscv/unit/conftest.py |  42 ++++
>   .../scripts/qtb/riscv/unit/test_config.py     | 121 ++++++++++
>   .../qtb/riscv/unit/test_console_test.py       | 217 ++++++++++++++++++
>   automation/scripts/qtb/riscv/unit/test_dt.py  |  99 ++++++++
>   .../scripts/qtb/riscv/unit/test_machine.py    |  60 +++++
>   .../scripts/qtb/riscv/unit/test_temp_dir.py   |  42 ++++
>   .../scripts/qtb/riscv/unit/test_xen_dt.py     |  46 ++++
>   8 files changed, 629 insertions(+)

[...]

> diff --git a/automation/scripts/qtb/riscv/unit/test_machine.py b/automation/scripts/qtb/riscv/unit/test_machine.py
> new file mode 100644
> index 0000000000..68a1564072
> --- /dev/null
> +++ b/automation/scripts/qtb/riscv/unit/test_machine.py
> @@ -0,0 +1,60 @@
> +# SPDX-License-Identifier: GPL-2.0-only
> +"""Unit tests for RiscvTestMachine QEMU argument assembly."""
> +
> +from __future__ import annotations
> +
> +from stat import S_IEXEC
> +from unittest import mock
> +
> +from ..machine import RiscvTestMachine
> +from ..config import MACHINE_MEMORY, BinariesConfig
> +
> +
> +def _binaries(tmp_path):
> +    xen = tmp_path / "xen"
> +    xen.write_bytes(b"")
> +    return BinariesConfig(xen=xen)
> +
> +
> +def _machine(mc) -> RiscvTestMachine:
> +    """Build the machine with QtbMachine.__init__ stubbed out (it spawns QEMU)."""
> +    with mock.patch("qemu.qtb.QtbMachine.__init__", return_value=None):
> +        return RiscvTestMachine(mc, timeout=30)
> +
> +
> +def test_init_forwards_cpus_to_qtbmachine(tmp_path, make_machine):
> +    mc = make_machine(name="machine", binaries=_binaries(tmp_path))
> +
> +    with mock.patch("qemu.qtb.QtbMachine.__init__", return_value=None) as base_init:
> +        vm = RiscvTestMachine(mc, timeout=30, log_dir="/logs")
> +
> +    assert vm.machine_conf is mc
> +    assert base_init.call_args.kwargs == {
> +        "memory": MACHINE_MEMORY,
> +        "cpus": mc.pcpu,
> +        "mirror_console": False,
> +        "timeout": 30,
> +        "log_dir": "/logs",
> +    }

Many of the tests check how the code is written rather than what it
does: the exact keyword arguments passed to QtbMachine.__init__() here,
the exact dtc argv in test_dt.py, the number of expect_exact() calls of
the attempts loop in test_console_test.py. With everything else mocked
out, these mostly restate the implementation, so each change asked for
in review of patch 3 (mirroring the console, dropping attempts, ...)
has to be repeated here, without the tests catching anything real in
between.

FWIW, all 51 tests pass for me (Debian trixie container from patch 1
plus python3-pytest).

~ Oleksii