Re: [PATCH 00/43] test: Improvements to ut command and test-suite running

[Mattijs Korpershoek <mkorpershoek@baylibre.com>]

Hi Simon,

Thank you for the series.

On mer., janv. 15, 2025 at 06:30, Simon Glass <sjg@chromium.org> wrote:

> The current method of running unit tests relies on subcommands of the
> ut command. Only the code in each subcommand knows how to find the tests
> related to that subcomand.
>
> This is not ideal and we now have quite a few subcommands which do
> nothing but locate the relevant tests in a linker list, then call a
> common function to run them.
>
> This series adds a list of test suites, so that these subcommands can be
> removed.
>
> An issue with 'ut all' is that it doesn't record how many tests failed
> overall, so it is necessary to examine copious amounts of output to look
> for failures. This series adds a new 'total' feature allow recording the
> total number of failed tests.
>
> To help with 'ut all' a new pytest is created which runs it (as well as
> 'ut info') and makes sure that all is well. Due to the 'ut all' failures
> this does not pass, so the test is disabled for now. It is here because
> it provides security against misnaming a test suite and causing it not
> to run.

I wanted to test this, but it seems it does not apply on both master or
next:

With master: 178f6ecb21fe ("Merge patch series "bootstd: Support recording images"")
With next: d6da3dbaef57 ("Merge patch series "cmd: Add support for optee commands."")

When running:
$ b4 shazam -s -l --check 20250115133114.590375-1-sjg@chromium.org

I see:

"""
[...] snip

Total patches: 43
---
Applying: test: Drop unused suite prototypes
Applying: test: Rename test_get_state() to ut_get_state()
Applying: test: Add functions to init and uninit the test state
Applying: test: Pass the test-state into ut_run_list()
Applying: Improve support for linker lists in data structures
Applying: test: Add newlines to hush-test messages
Applying: test: Rename test suites to match their linker-list name
Applying: test: Update ut info to show suites
Applying: test/py: Add a test which runs all unit tests
Applying: test: Drop the _test suffix on linker lists
Applying: test: Introduce a better array of test suites
Patch failed at 0011 test: Introduce a better array of test suites
error: sha1 information is lacking or useless (test/cmd_ut.c).
error: could not build fake ancestor
hint: Use 'git am --show-current-patch=diff' to see the failed patch
hint: When you have resolved this problem, run "git am --continue".
hint: If you prefer to skip this patch, run "git am --skip" instead.
hint: To restore the original branch and stop patching, run "git am --abort".
"""

On what base should this be applied if we want to test this?
Sorry if this is a sily question, usually I get dependencies from the
cover letter but I could not find any here.

>
> Future work may:
> - get 'ut all' passing
> - enable test_suite() in CL, to ensure that 'ut all' keeps passing
> - record duration of each suite
> - allow running the tests in random order to tease out dependencies
> - tweak the output to remove common prefixes
> - getting rid of bootstd, optee and seame 'ut' subcommands
>
>
> Simon Glass (43):
>   test: Drop unused suite prototypes
>   test: Rename test_get_state() to ut_get_state()
>   test: Add functions to init and uninit the test state
>   test: Pass the test-state into ut_run_list()
>   Improve support for linker lists in data structures
>   test: Add newlines to hush-test messages
>   test: Rename test suites to match their linker-list name
>   test: Update ut info to show suites
>   test/py: Add a test which runs all unit tests
>   test: Drop the _test suffix on linker lists
>   test: Introduce a better array of test suites
>   test: Drop the function for running bdinfo tests
>   test: Drop the function for running cmd tests
>   test: Drop the function for running common tests
>   test: Drop the function for running dm tests
>   test: Drop the function for running env tests
>   test: Drop the function for running exit tests
>   test: Drop the function for running fdt tests
>   test: Drop the function for running font tests
>   test: Drop the function for running lib tests
>   test: Drop the function for running log tests
>   test: Drop the function for running mbr tests
>   test: Drop the function for running mem tests
>   test: Drop the function for running setexpr tests
>   test: Drop the function for running measurement tests
>   test: Drop the function for running bloblist tests
>   test: Drop the function for running bootm tests
>   test: Drop the function for running addrmap tests
>   test: Drop the function for running hush tests
>   test: Drop the function for running loadm tests
>   test: Drop the function for running pci_mps tests
>   test: Drop the function for running seama tests
>   test: Drop the function for running upl tests
>   test: Drop the info test from the list
>   test: Drop conditional compilation for suites
>   test: Pass the test state to cmd_ut_category()
>   test: Move stats into a struct
>   test: Keep a track of the numbers of tests run
>   test: Move stat-printing into its own function
>   test: Record and show the totals for all test runs
>   test: Sort the test suites
>   test: Disable test_suite
>   test: Move help into the suite declaration
>
>  arch/sandbox/cpu/spl.c        |   6 +-
>  include/dm/test.h             |   2 +-
>  include/linker_lists.h        |  51 +++++
>  include/test/cmd.h            |   2 +-
>  include/test/common.h         |   2 +-
>  include/test/env.h            |   2 +-
>  include/test/hush.h           |   2 +-
>  include/test/lib.h            |   2 +-
>  include/test/log.h            |   4 +-
>  include/test/optee.h          |   2 +-
>  include/test/overlay.h        |   2 +-
>  include/test/suites.h         |  53 ++---
>  include/test/test.h           |  23 ++-
>  include/test/ut.h             |  45 ++++-
>  test/boot/bootm.c             |  11 +-
>  test/boot/bootstd_common.c    |   9 +-
>  test/boot/bootstd_common.h    |   3 +-
>  test/boot/measurement.c       |  12 +-
>  test/boot/upl.c               |  11 +-
>  test/cmd/Makefile             |   4 +-
>  test/cmd/addrmap.c            |  11 +-
>  test/cmd/bdinfo.c             |  10 +-
>  test/cmd/cmd_ut_cmd.c         |  20 --
>  test/cmd/exit.c               |  11 +-
>  test/cmd/fdt.c                |  10 +-
>  test/cmd/font.c               |  10 +-
>  test/cmd/loadm.c              |  11 +-
>  test/cmd/mbr.c                |  10 +-
>  test/cmd/mem.c                |  19 --
>  test/cmd/mem_copy.c           |   2 +-
>  test/cmd/mem_search.c         |   2 +-
>  test/cmd/pci_mps.c            |  12 +-
>  test/cmd/seama.c              |  11 +-
>  test/cmd/setexpr.c            |  11 +-
>  test/cmd_ut.c                 | 361 +++++++++++++++++++++-------------
>  test/common/Makefile          |   1 -
>  test/common/bloblist.c        |  12 +-
>  test/common/cmd_ut_common.c   |  21 --
>  test/dm/Makefile              |   2 -
>  test/dm/test-dm.c             |  16 --
>  test/dm/test-driver.c         |  10 +-
>  test/dm/test-uclass.c         |   8 +-
>  test/env/cmd_ut_env.c         |  10 -
>  test/hush/Makefile            |   1 -
>  test/hush/cmd_ut_hush.c       |  19 --
>  test/hush/dollar.c            |   7 +-
>  test/hush/loop.c              |   4 +-
>  test/lib/Makefile             |   1 -
>  test/lib/cmd_ut_lib.c         |  19 --
>  test/log/Makefile             |   2 -
>  test/log/log_ut.c             |  20 --
>  test/optee/cmd_ut_optee.c     |   4 +-
>  test/overlay/cmd_ut_overlay.c |  14 +-
>  test/py/conftest.py           |   7 +-
>  test/py/tests/test_spl.py     |   2 +-
>  test/py/tests/test_suite.py   | 182 +++++++++++++++++
>  test/py/tests/test_upl.py     |   2 +-
>  test/py/tests/test_ut.py      |   2 +-
>  test/py/tests/test_vbe.py     |   2 +-
>  test/py/tests/test_vpl.py     |   2 +-
>  test/test-main.c              | 104 ++++++----
>  test/ut.c                     |   4 +-
>  62 files changed, 657 insertions(+), 580 deletions(-)
>  delete mode 100644 test/cmd/cmd_ut_cmd.c
>  delete mode 100644 test/cmd/mem.c
>  delete mode 100644 test/common/cmd_ut_common.c
>  delete mode 100644 test/dm/test-dm.c
>  delete mode 100644 test/hush/cmd_ut_hush.c
>  delete mode 100644 test/lib/cmd_ut_lib.c
>  delete mode 100644 test/log/log_ut.c
>  create mode 100644 test/py/tests/test_suite.py
>
> -- 
> 2.34.1