[PATCH 00/28] check-parallel: Running tests without check

[Dave Chinner <david@fromorbit.com>]

Hi folks,

This set of patches is intended to move check-parallel away from
using check to execute tests.

To do this, we need to share a bunch of check code between check and
check-parallel. This is mainly the code that parses and builds the
test list, the config section parsing and iteration, and the test
execution loop itself.

To do this, test list parsing and building is factored out of check
into common/test_list. check is converted to use the test list
functions at the same time, and then check-parallel is converted to
use the factored code to directly build it's test list rather than
the open coded grep hack it currently uses.  This allows
check-parallel CLI to use the same group selection interface as
check.

The next change is to factor the config section parsing out of
common/config and move it to common/config-section. This allows
check-parallel to parse and implement section iteration itself
without needing to run all the environment setup code in
common/config. This also allows check-parallel to implement it's own
config section to define the device sizes that it will use
independently of the sections that run tests.

Next, we change check-parallel to use a global test list that runner
scripts can safely dequeue the next test to run. This uses a test
list file and a lock file to serialise access to the file. Hence a
runner can dequeue the next test and remove it from the test list
file without racing with any other runner trying to dequeue the next
test to run. This means we get rid of the static per-runner test
lists that result in many runners finishing and going idle while
other test runners have pending tests still to run. i.e. all test
runners keep executing tests until there are no tests left in the
queue, hence keeping utilisation as high as possible across the test
run.

Then we factor the test execution loop out of check and put it in
common/test_exec. This abstraction makes the results array part of
the test execution, as well using a context defined helper
"_run_seq" to do the actual execution of the test. This allows the
test execution loop to be completely generic, whilst allowing check
and check-parallel to do completely independent things with
individual test execution and overall results reporting.

Finally, we change check-parallel to run tests directly via the
common/test_exec infrastructure rather than executing them via
check. This requires a new helper function that does the test
environment setup in the private mount+pid namespace, but this is
much simpler and faster than using check itself to execute
individual tests. This last bit of functionality is still a work in
progress, so this specific patch is still tagged with [RFC].

There are lots of other bits of changes. The way common/rc and
common/config are used is changed. common/config only sets up the
execution environment now, and should not contain any code that
needs to be executed outside of environment setup. It should only be
sourced once at the highest level to set up the environment, and
never called again.

common/rc is similar - all directly executed code has been removed
from it, and that is now called from the high level code that needs
initialisation work done.  It no longer sources common/config,
either. The test preamble does not need to run init_rc() any more;
they just need to source the generic and fs specific functions the
tests may run. Also, because check does some weird things and lots
of _requires....() functions assume the TEST_DEV is mounted without
first running _require_test(), it also needs to ensure the TEST_DEV
is mounted...

check-parallel can now take a "-t N" parameter to specify how many
execution threads it will use. If this is not specified, it will
default to the number of CPUs in the machine. Testing with 4p
restrictions show that check-parallel will run the quick group 3.5x
faster on a 4p system with 8 execution threads than it will with a
single execution thread. IOWs, even on small test systems,
check-parallel can result in dramatic reductions in test runtime
over check.

On a 64 p machine, testing XFS with the quick group drops from 61
minutes to just under 4 minutes. Testing XFS with the auto group
drops from 246 minutes to just under 8 minutes.

Other miscellaneous stuff in the series:

	- kill non-numeric test name support
	- creating common/exit for all the general test exit
	  functions to fix circular dependencies between common/rc
	  and common/config
	- fix iscratch_mkfs_sized to make USE_EXTERNAL on XFS work
	  the same as ext4.
	- dm-logwrites devices are now created by check-parallel
	- several test conversions from sync() to syncfs()
	- removal of a could of stale .c test source files.
	- address poor CPU count scaling in a couple of tests

I have tried not to cause any regressions for people running plain
check. I've tested that a bit with XFS and ext4, but I can't
guarantee that there aren't issues I haven't uncovered. e.g. btrfs,
as yet, is untested. It is unfortunate that the problem I seek to
address - running exhaustive check testing across many filesystem
types and configurations is prohibitively expensive in terms of time
- is the very reason I can't really adequately test check for
regressions as I develop check-parallel functionality...

Thoughts, comments and code review all welcome!

-Dave.

 .gitignore                          |   1 -
 check                               | 727 ++++--------------------------------
 check-parallel                      | 351 ++++++++++++++---
 common/config                       | 612 +-----------------------------
 common/config-sections              | 461 +++++++++++++++++++++++
 common/dmlogwrites                  |   5 +-
 common/exit                         |  48 +++
 common/preamble                     |  19 +-
 common/rc                           | 253 +++++++++++--
 common/report                       |   2 +-
 common/test_exec                    | 377 +++++++++++++++++++
 common/test_list                    | 308 +++++++++++++++
 common/test_names                   |   8 +-
 new                                 |  24 --
 src/Makefile                        |   4 +-
 src/bulkstat_unlink_test.c          |  12 +-
 src/bulkstat_unlink_test_modified.c | 193 ----------
 src/fsync-tester.c                  |   2 +-
 src/open_by_handle.c                |   6 +-
 src/scaleread.c                     | 224 -----------
 src/scaleread.sh                    |  64 ----
 src/stale_handle.c                  |  15 +-
 tests/generic/531                   |   8 +-
 tests/xfs/259                       |   1 -
 tests/xfs/271                       |   2 -
 tools/run_test.sh                   | 116 ++++++
 26 files changed, 1954 insertions(+), 1889 deletions(-)