[PATCH v3 00/41] test: Refactor tests to have a single test runner

[Tom Rini <trini@konsulko.com>]

On Wed, Feb 03, 2021 at 05:44:06AM -0700, Simon Glass wrote:

> At present U-Boot has two broad sets of tests in the C code: driver model
> tests which do a lot of pre-/post-init and command tests which do not.
> 
> This separation makes it slightly harder to write a test, since there are
> two different test-state structures and different rules for running the
> two different test types. At present these rules are determined by where
> the test is (actually its prefix).
> 
> All unit tests can be run from the command line with the 'ut' command.
> Since SPL does not have commands, it currently calls the test runner
> directly and offers no control of which tests are run.
> 
> This seems like a good time to refactor the tests into a unified test
> runner, allowing U-Boot proper and SPL to use the same path, perhaps with
> some different conditions along the way.
> 
> This series sets up a unified runner called ut_run_list(), which runs a
> set of tests from a linker_list. Driver model tests are distinguished by
> a new UT_TESTF_DM flag so that the necessary init and cleanup can still
> be done.
> 
> The runner is modified to support running SPL tests that are not solely
> for driver model. An example test for FIT loading is added as a
> demonstration.
> 
> In addition, some documentation is added to explain how to write tests.
> 
> This series is available at u-boot-dm/test-working
> 
> Changes in v3:
> - Add new patch to re-enable test_ofplatdata
> - Reword the SPL tests section for clarity
> - Use test_set_state() throughout test-main.c instead of direct assignment
> - Update the pytest collector as well
> 
> Changes in v2:
> - Use correct rst format for 'Ad-hoc tests' section
> - Expand docs on how each type of test is marked
> - Put the docs in tests_sandbox since it is more related to sandbox
> - Put in a mention of tests_sandbox in the main testing docs
> - Add a note that SPL tests can in fact be run individualy
> - Document how to run all C tests with 'ut all'
> - Fix 'get list' typo
> - Fix conditions so non-DM SPL tests are actually run
> - Allow for prefix to be NULL, to match function comment
> - Add new patches to cover running an SPL test

There was a small problem applying one or two of these patches that I
fixed up.  But then when testing locally (Ubuntu 18.04, but virtualenv
setup and populated with test/py/requirements.txt) like 200 tests failed
to run for Sandbox.  Please check this and that CI runs.

-- 
Tom
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 659 bytes
Desc: not available
URL: <https://lists.denx.de/pipermail/u-boot/attachments/20210303/f9c2baca/attachment.sig>