[PATCH 00/11] Refactor meta-oe, meta-perl, meta-python ptests with BBCLASSEXTEND parallel execution

[Tim Orling <ticotimo@gmail.com>]

This series implements parallel ptest execution, similar to
"core-image-ptest: Switch to BBCLASSEXTEND parallel execution"

Refactor meta-*-ptest-image.bb for an image per ptest in the form of
meta-*-ptest-image-XXX using class extensions.

* Add meta-*-ptest-all-image.bb depends on ALL meta-*-ptest-image-XXX in meta-*
* Add meta-*-ptest-fast-image.bb depends on meta-* ptests which take less than ~30s

This allows parallel execution of the tests.

The downside to this approach is the parsing time of an image generating
this number of class extensions is slow but making it easier to execute
and collect test data should outweigh that.

A useful advantage is that the dependencies of each ptest package are
checked/tested individually.

Add to local.conf:
IMAGE_CLASSES += "testimage"

All the meta-*-ptest-image-XXX can be built with:
$ bitbake meta-*-ptest-all-image

Subsequently, al the ptest images can be run with:
$ bitbake -c testimage meta-*-ptest-all-image

You will likely want to limit the number of parallel image tests:

BB_NUMBER_THREADS = "4"

based on the number of TAP devices you have defined or the number of simultaneous
QEMU sessions your server can support.

Results can be summarized with:

$ resulttool report tmp/log/oeqa/testresults.json


Call to arms:

1. When upgrading recipes that have ptest enabled, run testimage
   $ bitbake meta-*-ptest-image-<pn>
   $ bitbake -c testimage meta-*-ptest-image-<pn>
2. Help move recipes in PTEST_PROBLEMS_* to a state where all ptests pass
3. Test on architectures you care about and report problems.
4. Enable ptest in more recipes:
   * inherit ptest
   * provide a run-ptest script
   * add to conf/include/ptest-packagelists-meta-*.inc
5. Migrate python recipes which use pytest to the new "pytest --automake" methodology
6. Enable this methodology in more meta-openembedded layers.

The following changes since commit 17243e70c8309751a1129a9214899ec476ab121a:

  python3-betamax: fix ptest failture of fixture and record modes (2023-03-28 10:44:00 -0700)

are available in the Git repository at:

  https://git.openembedded.org/meta-openembedded-contrib timo/ptest-refactoring
  http://cgit.openembedded.org/meta-openembedded-contrib/log/?h=timo/ptest-refactoring

Tim Orling (11):
  cli11: do not inherit ptest
  span-lite: do not inherit ptest
  ptest-packagelists-meta-oe.inc: add ptest recipes
  meta-oe-ptest*-image: enable BBCLASSEXTEND parallel execution
  ptest-packagelists-meta-python.inc: add ptest recipes
  meta-python-ptest*-image: enable BBCLASSEXTEND parallel execution
  python3-aspectlib: fix ptest
  ptest-packagelists-meta-perl.inc: add ptest recipes
  recipes-perl/packagegroups: move to recipes-core/
  recipes-perl/images: move to recipes-core
  meta-perl-ptest*-image: enable BBCLASSEXTEND parallel execution

 .../include/ptest-packagelists-meta-oe.inc    | 60 ++++++++++++++
 .../images/meta-oe-ptest-all-image.bb         | 25 ++++++
 .../images/meta-oe-ptest-fast-image.bb        |  6 ++
 .../images/meta-oe-ptest-image.bb             | 38 ++++++++-
 meta-oe/recipes-support/cli11/cli11_2.3.2.bb  |  3 +-
 .../span-lite/span-lite_0.10.3.bb             |  1 -
 .../include/ptest-packagelists-meta-perl.inc  | 50 +++++++++++
 .../images/meta-perl-base.bb                  |  0
 .../images/meta-perl-image.bb                 |  0
 .../images/meta-perl-ptest-all-image.bb       | 25 ++++++
 .../images/meta-perl-ptest-fast-image.bb      |  6 ++
 .../images/meta-perl-ptest-image.bb           | 41 ++++++++++
 .../packagegroups/packagegroup-meta-perl.bb   |  0
 .../images/meta-perl-ptest-image.bb           |  9 --
 .../ptest-packagelists-meta-python.inc        | 82 +++++++++++++++++++
 .../images/meta-python-ptest-all-image.bb     | 25 ++++++
 .../images/meta-python-ptest-fast-image.bb    |  6 ++
 .../images/meta-python-ptest-image.bb         | 38 ++++++++-
 .../python/python3-aspectlib/run-ptest        |  2 +-
 .../python/python3-aspectlib_2.0.0.bb         |  4 +-
 20 files changed, 405 insertions(+), 16 deletions(-)
 create mode 100644 meta-oe/conf/include/ptest-packagelists-meta-oe.inc
 create mode 100644 meta-oe/recipes-core/images/meta-oe-ptest-all-image.bb
 create mode 100644 meta-oe/recipes-core/images/meta-oe-ptest-fast-image.bb
 create mode 100644 meta-perl/conf/include/ptest-packagelists-meta-perl.inc
 rename meta-perl/{recipes-perl => recipes-core}/images/meta-perl-base.bb (100%)
 rename meta-perl/{recipes-perl => recipes-core}/images/meta-perl-image.bb (100%)
 create mode 100644 meta-perl/recipes-core/images/meta-perl-ptest-all-image.bb
 create mode 100644 meta-perl/recipes-core/images/meta-perl-ptest-fast-image.bb
 create mode 100644 meta-perl/recipes-core/images/meta-perl-ptest-image.bb
 rename meta-perl/{recipes-perl => recipes-core}/packagegroups/packagegroup-meta-perl.bb (100%)
 delete mode 100644 meta-perl/recipes-perl/images/meta-perl-ptest-image.bb
 create mode 100644 meta-python/conf/include/ptest-packagelists-meta-python.inc
 create mode 100644 meta-python/recipes-core/images/meta-python-ptest-all-image.bb
 create mode 100644 meta-python/recipes-core/images/meta-python-ptest-fast-image.bb

-- 
2.34.1