[PATCH 00/11] Refactor meta-oe, meta-perl, meta-python ptests with BBCLASSEXTEND parallel execution

[PATCH 00/11] Refactor meta-oe, meta-perl, meta-python ptests with BBCLASSEXTEND parallel execution

[Tim Orling]

This series implements parallel ptest execution, similar to
"core-image-ptest: Switch to BBCLASSEXTEND parallel execution"

Refactor meta-*-ptest-image.bb for an image per ptest in the form of
meta-*-ptest-image-XXX using class extensions.

* Add meta-*-ptest-all-image.bb depends on ALL meta-*-ptest-image-XXX in meta-*
* Add meta-*-ptest-fast-image.bb depends on meta-* ptests which take less than ~30s

This allows parallel execution of the tests.

The downside to this approach is the parsing time of an image generating
this number of class extensions is slow but making it easier to execute
and collect test data should outweigh that.

A useful advantage is that the dependencies of each ptest package are
checked/tested individually.

Add to local.conf:
IMAGE_CLASSES += "testimage"

All the meta-*-ptest-image-XXX can be built with:
$ bitbake meta-*-ptest-all-image

Subsequently, al the ptest images can be run with:
$ bitbake -c testimage meta-*-ptest-all-image

You will likely want to limit the number of parallel image tests:

BB_NUMBER_THREADS = "4"

based on the number of TAP devices you have defined or the number of simultaneous
QEMU sessions your server can support.

Results can be summarized with:

$ resulttool report tmp/log/oeqa/testresults.json


Call to arms:

1. When upgrading recipes that have ptest enabled, run testimage
   $ bitbake meta-*-ptest-image-<pn>
   $ bitbake -c testimage meta-*-ptest-image-<pn>
2. Help move recipes in PTEST_PROBLEMS_* to a state where all ptests pass
3. Test on architectures you care about and report problems.
4. Enable ptest in more recipes:
   * inherit ptest
   * provide a run-ptest script
   * add to conf/include/ptest-packagelists-meta-*.inc
5. Migrate python recipes which use pytest to the new "pytest --automake" methodology
6. Enable this methodology in more meta-openembedded layers.

The following changes since commit 17243e70c8309751a1129a9214899ec476ab121a:

  python3-betamax: fix ptest failture of fixture and record modes (2023-03-28 10:44:00 -0700)

are available in the Git repository at:

  https://git.openembedded.org/meta-openembedded-contrib timo/ptest-refactoring
  http://cgit.openembedded.org/meta-openembedded-contrib/log/?h=timo/ptest-refactoring

Tim Orling (11):
  cli11: do not inherit ptest
  span-lite: do not inherit ptest
  ptest-packagelists-meta-oe.inc: add ptest recipes
  meta-oe-ptest*-image: enable BBCLASSEXTEND parallel execution
  ptest-packagelists-meta-python.inc: add ptest recipes
  meta-python-ptest*-image: enable BBCLASSEXTEND parallel execution
  python3-aspectlib: fix ptest
  ptest-packagelists-meta-perl.inc: add ptest recipes
  recipes-perl/packagegroups: move to recipes-core/
  recipes-perl/images: move to recipes-core
  meta-perl-ptest*-image: enable BBCLASSEXTEND parallel execution

 .../include/ptest-packagelists-meta-oe.inc    | 60 ++++++++++++++
 .../images/meta-oe-ptest-all-image.bb         | 25 ++++++
 .../images/meta-oe-ptest-fast-image.bb        |  6 ++
 .../images/meta-oe-ptest-image.bb             | 38 ++++++++-
 meta-oe/recipes-support/cli11/cli11_2.3.2.bb  |  3 +-
 .../span-lite/span-lite_0.10.3.bb             |  1 -
 .../include/ptest-packagelists-meta-perl.inc  | 50 +++++++++++
 .../images/meta-perl-base.bb                  |  0
 .../images/meta-perl-image.bb                 |  0
 .../images/meta-perl-ptest-all-image.bb       | 25 ++++++
 .../images/meta-perl-ptest-fast-image.bb      |  6 ++
 .../images/meta-perl-ptest-image.bb           | 41 ++++++++++
 .../packagegroups/packagegroup-meta-perl.bb   |  0
 .../images/meta-perl-ptest-image.bb           |  9 --
 .../ptest-packagelists-meta-python.inc        | 82 +++++++++++++++++++
 .../images/meta-python-ptest-all-image.bb     | 25 ++++++
 .../images/meta-python-ptest-fast-image.bb    |  6 ++
 .../images/meta-python-ptest-image.bb         | 38 ++++++++-
 .../python/python3-aspectlib/run-ptest        |  2 +-
 .../python/python3-aspectlib_2.0.0.bb         |  4 +-
 20 files changed, 405 insertions(+), 16 deletions(-)
 create mode 100644 meta-oe/conf/include/ptest-packagelists-meta-oe.inc
 create mode 100644 meta-oe/recipes-core/images/meta-oe-ptest-all-image.bb
 create mode 100644 meta-oe/recipes-core/images/meta-oe-ptest-fast-image.bb
 create mode 100644 meta-perl/conf/include/ptest-packagelists-meta-perl.inc
 rename meta-perl/{recipes-perl => recipes-core}/images/meta-perl-base.bb (100%)
 rename meta-perl/{recipes-perl => recipes-core}/images/meta-perl-image.bb (100%)
 create mode 100644 meta-perl/recipes-core/images/meta-perl-ptest-all-image.bb
 create mode 100644 meta-perl/recipes-core/images/meta-perl-ptest-fast-image.bb
 create mode 100644 meta-perl/recipes-core/images/meta-perl-ptest-image.bb
 rename meta-perl/{recipes-perl => recipes-core}/packagegroups/packagegroup-meta-perl.bb (100%)
 delete mode 100644 meta-perl/recipes-perl/images/meta-perl-ptest-image.bb
 create mode 100644 meta-python/conf/include/ptest-packagelists-meta-python.inc
 create mode 100644 meta-python/recipes-core/images/meta-python-ptest-all-image.bb
 create mode 100644 meta-python/recipes-core/images/meta-python-ptest-fast-image.bb

-- 
2.34.1

[meta-oe][PATCH 01/11] cli11: do not inherit ptest

[Tim Orling]

This recipe provides no run-ptest script.

Signed-off-by: Tim Orling <tim.orling@konsulko.com>
---
 meta-oe/recipes-support/cli11/cli11_2.3.2.bb | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)

diff --git a/meta-oe/recipes-support/cli11/cli11_2.3.2.bb b/meta-oe/recipes-support/cli11/cli11_2.3.2.bb
index b9ff914da..7c6b2547c 100644
--- a/meta-oe/recipes-support/cli11/cli11_2.3.2.bb
+++ b/meta-oe/recipes-support/cli11/cli11_2.3.2.bb
@@ -15,11 +15,10 @@ S = "${WORKDIR}/git"
 
 do_configure:prepend() {
     mkdir -p ${S}/tests/catch2
-    cp ${DL_DIR}/catch.hpp ${S}/tests/catch2/catch.hpp 
+    cp ${DL_DIR}/catch.hpp ${S}/tests/catch2/catch.hpp
 }
 
 inherit cmake
-inherit ptest
 
 # cli11 is a header only C++ library, so the main package will be empty.
 RDEPENDS:${PN}-dev = ""
-- 
2.34.1

[meta-oe][PATCH 02/11] span-lite: do not inherit ptest

[Tim Orling]

This recipe provides no run-ptest script.

Signed-off-by: Tim Orling <tim.orling@konsulko.com>
---
 meta-oe/recipes-support/span-lite/span-lite_0.10.3.bb | 1 -
 1 file changed, 1 deletion(-)

diff --git a/meta-oe/recipes-support/span-lite/span-lite_0.10.3.bb b/meta-oe/recipes-support/span-lite/span-lite_0.10.3.bb
index 0d9a8c93d..15380f5d7 100644
--- a/meta-oe/recipes-support/span-lite/span-lite_0.10.3.bb
+++ b/meta-oe/recipes-support/span-lite/span-lite_0.10.3.bb
@@ -9,4 +9,3 @@ SRCREV = "cbb9c3c5162a0d7018c7b3e053153a04d4fbbbb9"
 S = "${WORKDIR}/git"
 
 inherit cmake
-inherit ptest
-- 
2.34.1

[meta-oe][PATCH 03/11] ptest-packagelists-meta-oe.inc: add ptest recipes

[Tim Orling]

Similar to ptest-packagelists.inc in oe-core, this list enables
per image ptest (isolated ptests for a given recipe)

* Add recipes to PTEST_FAST_META_OE for ptests which take less than ~30s
* Add recipes to PTEST_SLOW_META_OE for ptests which take longer
* Add recipes to PTEST_PROBLEMS_META_OE for ptests which have failing tests
  or other issues with ptest package

The initial list was taken from:

meta_oe_ptest_recipes=$(bitbake-layers show-recipes --recipes-only \
  --layer meta-oe --inherits ptest --bare | tr '\n' ' ' | \
  pcregrep -o1 '^NOTE:.+===(.+)$')

Signed-off-by: Tim Orling <tim.orling@konsulko.com>
---
 .../include/ptest-packagelists-meta-oe.inc    | 60 +++++++++++++++++++
 1 file changed, 60 insertions(+)
 create mode 100644 meta-oe/conf/include/ptest-packagelists-meta-oe.inc

diff --git a/meta-oe/conf/include/ptest-packagelists-meta-oe.inc b/meta-oe/conf/include/ptest-packagelists-meta-oe.inc
new file mode 100644
index 000000000..e8898ce78
--- /dev/null
+++ b/meta-oe/conf/include/ptest-packagelists-meta-oe.inc
@@ -0,0 +1,60 @@
+#
+# Lists of the ptest in meta-oe, sorted into two sets by the time they take
+# Please keep these sorted in alphabetical order
+#
+# A first pass at getting all meta-oe recipes which inherit ptest
+# meta_oe_ptest_recipes=$(bitbake-layers show-recipes --recipes-only --layer meta-oe --inherits ptest --bare | tr '\n' ' ' | pcregrep -o1 '^NOTE:.+===(.+)$')
+#
+# ptests which take less than ~30s each
+PTESTS_FAST_META_OE = "\
+    cmocka \
+    cunit \
+    duktape \
+    fmt \
+    function2 \
+    fwupd \
+    gcab \
+    jq \
+    leveldb \
+    libcereal \
+    libee \
+    libio-pty-perl \
+    libmanette \
+    libssh \
+    libxml++-5.0 \
+    libxmlb \
+    libyang \
+    lmdb \
+    minicoredumper \
+    neon \
+    nodejs \
+    onig \
+    poco \
+    protobuf \
+    sdbus-c++ \
+    uthash \
+    xmlsec1 \
+    zeromq \
+"
+
+PTESTS_SLOW_META_OE = "\
+    fftw \
+    libusb-compat \
+    rsyslog \
+"
+
+PTESTS_PROBLEMS_META_OE ="\
+    jemalloc \
+    kernel-selftest \
+    keyutils \
+    libdbi-perl \
+    libgpiod \
+    libjcat \
+    libteam \
+    libxml++ \
+    mcelog \
+    oprofile \
+    ostree \
+    psqlodbc \
+    pv \
+"
-- 
2.34.1

[meta-oe][PATCH 04/11] meta-oe-ptest*-image: enable BBCLASSEXTEND parallel execution

[Tim Orling]

Similar to "core-image-ptest: Switch to BBCLASSEXTEND parallel execution"

Refactor meta-oe-ptest-image.bb for an image per ptest in the form of
meta-oe-ptest-image-XXX using class extensions.

* Add meta-oe-ptest-all-image.bb depends on ALL meta-oe-ptest-image-XXX in meta-oe
* Add meta-oe-ptest-fast-image.bb depends on meta-oe ptests which take less than ~30s

This allows parallel execution of the tests.

The downside to this approach is the parsing time of an image generating
this number of class extensions is slow but making it easier to execute
and collect test data should outweigh that.

A useful advantage is that the dependencies of each ptest package are
checked/tested individually.

Add to local.conf:
IMAGE_CLASSES += "testimage"

All the meta-oe-ptest-image-XXX can be built with:
$ bitbake meta-oe-ptest-all-image

Subsequently, all the ptest images can be run with:
$ bitbake -c testimage meta-oe-ptest-all-image

You will likely want to limit the number of parallel image tests:

BB_NUMBER_THREADS = "4"

based on the number of TAP devices you have defined or the number of simultaneous
QEMU sessions your server can support.

Results can be summarized with:

$ resulttool report tmp/log/oeqa/testresult.json

Signed-off-by: Tim Orling <tim.orling@konsulko.com>
---
 .../images/meta-oe-ptest-all-image.bb         | 25 ++++++++++++
 .../images/meta-oe-ptest-fast-image.bb        |  6 +++
 .../images/meta-oe-ptest-image.bb             | 38 ++++++++++++++++++-
 3 files changed, 68 insertions(+), 1 deletion(-)
 create mode 100644 meta-oe/recipes-core/images/meta-oe-ptest-all-image.bb
 create mode 100644 meta-oe/recipes-core/images/meta-oe-ptest-fast-image.bb

diff --git a/meta-oe/recipes-core/images/meta-oe-ptest-all-image.bb b/meta-oe/recipes-core/images/meta-oe-ptest-all-image.bb
new file mode 100644
index 000000000..0ea03dd2e
--- /dev/null
+++ b/meta-oe/recipes-core/images/meta-oe-ptest-all-image.bb
@@ -0,0 +1,25 @@
+DESCRIPTION = "Recipe to trigger execution of all meta-oe ptest images."
+HOMEPAGE = "https://www.openembedded.org/"
+
+LICENSE = "MIT"
+
+inherit features_check nopackages
+REQUIRED_DISTRO_FEATURES = "ptest"
+
+require conf/include/ptest-packagelists-meta-oe.inc
+
+# Include the full set of ptests
+PTESTS_META_OE = "${PTESTS_FAST_META_OE} ${PTESTS_SLOW_META_OE}"
+
+do_testimage[noexec] = "1"
+do_testimage[depends] = "${@' '.join(['meta-oe-ptest-image-'+x+':do_testimage' for x in d.getVar('PTESTS_META_OE').split()])}"
+
+do_build[depends] = "${@' '.join(['meta-oe-ptest-image-'+x+':do_build' for x in d.getVar('PTESTS_META_OE').split()])}"
+
+# normally image.bbclass would do this
+EXCLUDE_FROM_WORLD = "1"
+
+python () {
+    if bb.utils.contains('IMAGE_CLASSES', 'testimage', True, False, d):
+        bb.build.addtask("do_testimage", "", "", d)
+}
diff --git a/meta-oe/recipes-core/images/meta-oe-ptest-fast-image.bb b/meta-oe/recipes-core/images/meta-oe-ptest-fast-image.bb
new file mode 100644
index 000000000..f56e841ff
--- /dev/null
+++ b/meta-oe/recipes-core/images/meta-oe-ptest-fast-image.bb
@@ -0,0 +1,6 @@
+require meta-oe-ptest-all-image.bb
+
+DESCRIPTION = "Recipe to trigger execution of all fast meta-oe ptest images."
+
+PTESTS_META_OE = "${PTESTS_FAST_META_OE}"
+
diff --git a/meta-oe/recipes-core/images/meta-oe-ptest-image.bb b/meta-oe/recipes-core/images/meta-oe-ptest-image.bb
index 0166b9c41..951d91f82 100644
--- a/meta-oe/recipes-core/images/meta-oe-ptest-image.bb
+++ b/meta-oe/recipes-core/images/meta-oe-ptest-image.bb
@@ -1,5 +1,41 @@
+inherit features_check
+REQUIRED_DISTRO_FEATURES = "ptest"
+
+require conf/include/ptest-packagelists-meta-oe.inc
+
 require  meta-oe-image-base.bb
 
 SUMMARY = "meta-oe ptest test image"
 
-IMAGE_INSTALL += "packagegroup-meta-oe"
+DESCRIPTION += "Also including the ${MCNAME} ptest package."
+HOMEPAGE = "https://www.openembedded.org/"
+
+PTESTS_META_OE = "${PTESTS_SLOW_META_OE} ${PTESTS_FAST_META_OE}"
+
+IMAGE_INSTALL:append = " ${MCNAME}-ptest openssh"
+
+BBCLASSEXTEND = "${@' '.join(['mcextend:'+x for x in d.getVar('PTESTS_META_OE').split()])}"
+
+# The image can be sufficiently large (~1.8GB) that we need to be careful that it fits in a live
+# image (which has a 4GB limit), so nullify the overhead factor (1.3x out of the
+# box) and explicitly add up to 1500MB.
+IMAGE_OVERHEAD_FACTOR = "1.0"
+IMAGE_ROOTFS_EXTRA_SPACE = "324288"
+# If a particular ptest needs more space, it can be customized:
+#IMAGE_ROOTFS_EXTRA_SPACE:virtclass-mcextend-<pn> = "1024288"
+
+# ptests need more memory than standard to avoid the OOM killer
+QB_MEM = "-m 1024"
+# If a particular ptest needs more memroy, it can be customized:
+#QB_MEM:virtclass-mcextend-<pn> = "-m 4096"
+
+TEST_SUITES = "ping ssh parselogs ptest"
+
+# Sadly at the moment the full set of ptests is not robust enough and sporadically fails in random places
+PTEST_EXPECT_FAILURE = "1"
+
+python () {
+    if not d.getVar("MCNAME"):
+        raise bb.parse.SkipRecipe("No class extension set")
+}
+
-- 
2.34.1

[meta-python][PATCH 05/11] ptest-packagelists-meta-python.inc: add ptest recipes

[Tim Orling]

Similar to ptest-packagelists.inc in oe-core, this list enables
per image ptest (isolated ptests for a given recipe)

* Add recipes to PTEST_FAST_META_PYTHON for ptests which take less than ~30s
* Add recipes to PTEST_SLOW_META_PYTHON for ptests which take longer
* Add recipes to PTEST_PROBLEMS_META_PYTHON for ptests which have failing tests
  or other issues with ptest package

The initial list was taken from:

meta_python_ptest_recipes=$(bitbake-layers show-recipes --recipes-only \
      --layer meta-python --inherits ptest --bare | tr '\n' ' ' | \
      pcregrep -o1 '^NOTE:.+===(.+)$')

Signed-off-by: Tim Orling <tim.orling@konsulko.com>
---
 .../ptest-packagelists-meta-python.inc        | 82 +++++++++++++++++++
 1 file changed, 82 insertions(+)
 create mode 100644 meta-python/conf/include/ptest-packagelists-meta-python.inc

diff --git a/meta-python/conf/include/ptest-packagelists-meta-python.inc b/meta-python/conf/include/ptest-packagelists-meta-python.inc
new file mode 100644
index 000000000..ef5d20bcd
--- /dev/null
+++ b/meta-python/conf/include/ptest-packagelists-meta-python.inc
@@ -0,0 +1,82 @@
+#
+# Lists of the ptest in meta-python, sorted into two sets by the time they take
+# Please keep these sorted in alphabetical order
+#
+# A first pass at getting all meta-python recipes which inherit ptest
+# meta_python_ptest_recipes=$(bitbake-layers show-recipes --recipes-only --layer meta-python --inherits ptest --bare | tr '\n' ' ' | pcregrep -o1 '^NOTE:.+===(.+)$')
+#
+# ptests which take less than ~30s each
+PTESTS_FAST_META_PYTHON = "\
+    python3-ansicolors \
+    python3-asgiref \
+    python3-aspectlib \
+    python3-blinker \
+    python3-cachetools \
+    python3-cbor2 \
+    python3-click \
+    python3-dominate \
+    python3-execnet \
+    python3-geojson \
+    python3-html2text \
+    python3-inflection \
+    python3-intervals \
+    python3-ipy \
+    python3-iso3166 \
+    python3-parse-type \
+    python3-polyline \
+    python3-precise-runner \
+    python3-prettytable \
+    python3-ptyprocess \
+    python3-pyasn1-modules \
+    python3-pyroute2 \
+    python3-pyserial \
+    python3-pytoml \
+    python3-requests-file \
+    python3-requests-toolbelt \
+    python3-semver \
+    python3-serpent \
+    python3-simpleeval \
+    python3-smpplib \
+    python3-soupsieve \
+    python3-sqlparse \
+    python3-u-msgpack-python \
+    python3-unidiff \
+    python3-uritemplate \
+    python3-xmltodict \
+    python3-xxhash \
+"
+
+PTESTS_SLOW_META_PYTHON = "\
+    python3-lz4 \
+"
+
+PTESTS_PROBLEMS_META_PYTHON ="\
+    python3-appdirs \
+    python3-betamax \
+    python3-dnspython \
+    python3-fastjsonschema \
+    python3-gpiod \
+    python3-gunicorn \
+    python3-inotify \
+    python3-jdcal \
+    python3-jsmin \
+    python3-license-expression \
+    python3-msgpack \
+    python3-multidict \
+    python3-ordered-set \
+    python3-parse \
+    python3-pillow \
+    python3-pint \
+    python3-py-cpuinfo \
+    python3-pytest-lazy-fixture \
+    python3-pyzmq \
+    python3-scrypt \
+    python3-service-identity \
+    python3-trustme \
+    python3-typeguard \
+    python3-whoosh \
+    python3-ujson \
+    python3-xlrd \
+    python3-yappi \
+    python3-yarl \
+"
-- 
2.34.1

[meta-python][PATCH 06/11] meta-python-ptest*-image: enable BBCLASSEXTEND parallel execution

[Tim Orling]

Similar to "core-image-ptest: Switch to BBCLASSEXTEND parallel execution"

Refactor meta-python-ptest-image.bb for an image per ptest in the form of
meta-python-ptest-image-XXX using class extensions.

* Add meta-python-ptest-all-image.bb depends on ALL meta-python-ptest-image-XXX in meta-python
* Add meta-python-ptest-fast-image.bb depends on meta-python ptests which take less than ~30s

This allows parallel execution of the tests.

The downside to this approach is the parsing time of an image generating
this number of class extensions is slow but making it easier to execute
and collect test data should outweigh that.

A useful advantage is that the dependencies of each ptest package are
checked/tested individually.

Add to local.conf:
IMAGE_CLASSES += "testimage"

All the meta-python-ptest-image-XXX can be built with:
$ bitbake meta-python-ptest-all-image

Subsequently, all the ptest images can be run with:
$ bitbake -c testimage meta-python-ptest-all-image

You will likely want to limit the number of parallel image tests:

BB_NUMBER_THREADS = "4"

based on the number of TAP devices you have defined or the number of simultaneous
QEMU sessions your server can support.

Results can be summarized with:

$ resulttool report tmp/log/oeqa/testresults.json

Signed-off-by: Tim Orling <tim.orling@konsulko.com>
---
 .../images/meta-python-ptest-all-image.bb     | 25 ++++++++++++
 .../images/meta-python-ptest-fast-image.bb    |  6 +++
 .../images/meta-python-ptest-image.bb         | 38 ++++++++++++++++++-
 3 files changed, 68 insertions(+), 1 deletion(-)
 create mode 100644 meta-python/recipes-core/images/meta-python-ptest-all-image.bb
 create mode 100644 meta-python/recipes-core/images/meta-python-ptest-fast-image.bb

diff --git a/meta-python/recipes-core/images/meta-python-ptest-all-image.bb b/meta-python/recipes-core/images/meta-python-ptest-all-image.bb
new file mode 100644
index 000000000..fd4dc4248
--- /dev/null
+++ b/meta-python/recipes-core/images/meta-python-ptest-all-image.bb
@@ -0,0 +1,25 @@
+DESCRIPTION = "Recipe to trigger execution of all meta-python ptest images."
+HOMEPAGE = "https://www.openembedded.org/"
+
+LICENSE = "MIT"
+
+inherit features_check nopackages
+REQUIRED_DISTRO_FEATURES = "ptest"
+
+require conf/include/ptest-packagelists-meta-python.inc
+
+# Include the full set of ptests
+PTESTS_META_PYTHON = "${PTESTS_FAST_META_PYTHON} ${PTESTS_SLOW_META_PYTHON}"
+
+do_testimage[noexec] = "1"
+do_testimage[depends] = "${@' '.join(['meta-python-ptest-image-'+x+':do_testimage' for x in d.getVar('PTESTS_META_PYTHON').split()])}"
+
+do_build[depends] = "${@' '.join(['meta-python-ptest-image-'+x+':do_build' for x in d.getVar('PTESTS_META_PYTHON').split()])}"
+
+# normally image.bbclass would do this
+EXCLUDE_FROM_WORLD = "1"
+
+python () {
+    if bb.utils.contains('IMAGE_CLASSES', 'testimage', True, False, d):
+        bb.build.addtask("do_testimage", "", "", d)
+}
diff --git a/meta-python/recipes-core/images/meta-python-ptest-fast-image.bb b/meta-python/recipes-core/images/meta-python-ptest-fast-image.bb
new file mode 100644
index 000000000..4f93a1595
--- /dev/null
+++ b/meta-python/recipes-core/images/meta-python-ptest-fast-image.bb
@@ -0,0 +1,6 @@
+require meta-python-ptest-all-image.bb
+
+DESCRIPTION = "Recipe to trigger execution of all fast meta-python ptest images."
+
+PTESTS_META_PYTHON = "${PTESTS_FAST_META_PYTHON}"
+
diff --git a/meta-python/recipes-core/images/meta-python-ptest-image.bb b/meta-python/recipes-core/images/meta-python-ptest-image.bb
index d497016d4..a649cbb94 100644
--- a/meta-python/recipes-core/images/meta-python-ptest-image.bb
+++ b/meta-python/recipes-core/images/meta-python-ptest-image.bb
@@ -1,5 +1,41 @@
+inherit features_check
+REQUIRED_DISTRO_FEATURES = "ptest"
+
+require conf/include/ptest-packagelists-meta-python.inc
+
 require  meta-python-image-base.bb
 
 SUMMARY = "meta-python ptest test image"
 
-IMAGE_INSTALL += "packagegroup-meta-python3-ptest"
+DESCRIPTION += "Also including the ${MCNAME} ptest package."
+HOMEPAGE = "https://www.openembedded.org/"
+
+PTESTS_META_PYTHON = "${PTESTS_SLOW_META_PYTHON} ${PTESTS_FAST_META_PYTHON}"
+
+IMAGE_INSTALL:append = " ${MCNAME}-ptest openssh"
+
+BBCLASSEXTEND = "${@' '.join(['mcextend:'+x for x in d.getVar('PTESTS_META_PYTHON').split()])}"
+
+# The image can be sufficiently large (~1.8GB) that we need to be careful that it fits in a live
+# image (which has a 4GB limit), so nullify the overhead factor (1.3x out of the
+# box) and explicitly add up to 1500MB.
+IMAGE_OVERHEAD_FACTOR = "1.0"
+IMAGE_ROOTFS_EXTRA_SPACE = "324288"
+# If a particular ptest needs more space, it can be customized:
+#IMAGE_ROOTFS_EXTRA_SPACE:virtclass-mcextend-<pn> = "1024288"
+
+# ptests need more memory than standard to avoid the OOM killer
+QB_MEM = "-m 1024"
+# If a particular ptest needs more memory, it can be customized:
+#QB_MEM:virtclass-mcextend-<pn> = "-m 4096"
+
+TEST_SUITES = "ping ssh parselogs ptest"
+
+# Sadly at the moment the full set of ptests is not robust enough and sporadically fails in random places
+PTEST_EXPECT_FAILURE = "1"
+
+python () {
+    if not d.getVar("MCNAME"):
+        raise bb.parse.SkipRecipe("No class extension set")
+}
+
-- 
2.34.1

[meta-python][PATCH 07/11] python3-aspectlib: fix ptest

[Tim Orling]

* ptest RDEPENDS on pytest
* Use python3-unittest-automake-output instead of using sed to reformat
  the output use this new module.

Signed-off-by: Tim Orling <tim.orling@konsulko.com>
---
 .../recipes-devtools/python/python3-aspectlib/run-ptest       | 2 +-
 .../recipes-devtools/python/python3-aspectlib_2.0.0.bb        | 4 +++-
 2 files changed, 4 insertions(+), 2 deletions(-)

diff --git a/meta-python/recipes-devtools/python/python3-aspectlib/run-ptest b/meta-python/recipes-devtools/python/python3-aspectlib/run-ptest
index b63c4de0d..8d2017d39 100644
--- a/meta-python/recipes-devtools/python/python3-aspectlib/run-ptest
+++ b/meta-python/recipes-devtools/python/python3-aspectlib/run-ptest
@@ -1,3 +1,3 @@
 #!/bin/sh
 
-pytest -o log_cli=true -o log_cli_level=INFO | sed -e 's/\[...%\]//g'| sed -e 's/PASSED/PASS/g'| sed -e 's/FAILED/FAIL/g'|sed -e 's/SKIPPED/SKIP/g'| awk '{if ($NF=="PASS" || $NF=="FAIL" || $NF=="SKIP" || $NF=="XFAIL" || $NF=="XPASS"){printf "%s: %s\n", $NF, $0}else{print}}'| awk '{if ($NF=="PASS" || $NF=="FAIL" || $NF=="SKIP" || $NF=="XFAIL" || $NF=="XPASS") {$NF="";print $0}else{print}}'
+pytest --automake
diff --git a/meta-python/recipes-devtools/python/python3-aspectlib_2.0.0.bb b/meta-python/recipes-devtools/python/python3-aspectlib_2.0.0.bb
index 416b031c5..d2b0864ca 100644
--- a/meta-python/recipes-devtools/python/python3-aspectlib_2.0.0.bb
+++ b/meta-python/recipes-devtools/python/python3-aspectlib_2.0.0.bb
@@ -21,8 +21,10 @@ do_install_ptest() {
 }
 
 RDEPENDS:${PN}-ptest += "\
-    python3-tornado \
     python3-process-tests \
+    python3-pytest \
+    python3-tornado \
+    python3-unittest-automake-output \
 "
 
 RDEPENDS:${PN} += "python3-core python3-fields"
-- 
2.34.1

[meta-perl][PATCH 08/11] ptest-packagelists-meta-perl.inc: add ptest recipes

[Tim Orling]

Similar to ptest-packagelists.inc in oe-core, this list enables
per image ptest (isolated ptests for a given recipe)

* Add recipes to PTEST_FAST_META_PERL for ptests which take less than ~30s
* Add recipes to PTEST_SLOW_META_PERL for ptests which take longer
* Add recipes to PTEST_PROBLEMS_META_PERL for ptests which have failing tests
  or other issues with ptest package

The initial list was taken from:

meta_perl_ptest_recipes=$(bitbake-layers show-recipes --recipes-only \
    --layer meta-perl --inherits ptest --bare | tr '\n' ' ' | \
    pcregrep -o1 '^NOTE:.+===(.+)$')

Signed-off-by: Tim Orling <tim.orling@konsulko.com>
---
 .../include/ptest-packagelists-meta-perl.inc  | 50 +++++++++++++++++++
 1 file changed, 50 insertions(+)
 create mode 100644 meta-perl/conf/include/ptest-packagelists-meta-perl.inc

diff --git a/meta-perl/conf/include/ptest-packagelists-meta-perl.inc b/meta-perl/conf/include/ptest-packagelists-meta-perl.inc
new file mode 100644
index 000000000..1a8c759b1
--- /dev/null
+++ b/meta-perl/conf/include/ptest-packagelists-meta-perl.inc
@@ -0,0 +1,50 @@
+#
+# Lists of the ptest in meta-perl, sorted into two sets by the time they take
+# Please keep these sorted in alphabetical order
+#
+# A first pass at getting all meta-perl recipes which inherit ptest
+# meta_perl_ptest_recipes=$(bitbake-layers show-recipes --recipes-only --layer meta-perl --inherits ptest --bare | tr '\n' ' ' | pcregrep -o1 '^NOTE:.+===(.+)$')
+#
+# ptests which take less than ~30s each
+PTESTS_FAST_META_PERL = "\
+    libcapture-tiny-perl \
+    libcgi-perl \
+    libcrypt-openssl-random-perl \
+    libextutils-config-perl \
+    libhtml-tagset-perl \
+    libmozilla-ca-perl \
+    libstrictures-perl \
+    libterm-readkey-perl \
+    libtest-nowarnings-perl \
+"
+
+PTESTS_SLOW_META_PERL = "\
+    libio-socket-ssl-perl \
+"
+
+PTESTS_PROBLEMS_META_PERL ="\
+    libauthen-sasl-perl \
+    libconfig-autoconf-perl \
+    libcrypt-openssl-guess-perl \
+    libcrypt-openssl-rsa-perl \
+    libdata-hexdump-perl \
+    libdbd-sqlite-perl \
+    libdigest-hmac-perl \
+    libdigest-sha1-perl \
+    libnet-dns-perl \
+    libnet-dns-sec-perl \
+    libnet-ldap-perl \
+    libencode-perl \
+    libextutils-installpaths-perl \
+    libfile-slurper-perl \
+    libhtml-parser-perl \
+    libmime-types-perl \
+    libnet-ssleay-perl \
+    libsub-uplevel-perl \
+    libtest-deep-perl \
+    libtest-harness-perl \
+    libtest-warn-perl \
+    libunicode-linebreak-perl \
+    libunix-statgrab \
+    libxml-libxml-perl \
+"
-- 
2.34.1

[meta-perl][PATCH 09/11] recipes-perl/packagegroups: move to recipes-core/

[Tim Orling]

Be consistent with other layers, like oe-core, meta-oe and meta-python

Signed-off-by: Tim Orling <tim.orling@konsulko.com>
---
 .../packagegroups/packagegroup-meta-perl.bb                       | 0
 1 file changed, 0 insertions(+), 0 deletions(-)
 rename meta-perl/{recipes-perl => recipes-core}/packagegroups/packagegroup-meta-perl.bb (100%)

diff --git a/meta-perl/recipes-perl/packagegroups/packagegroup-meta-perl.bb b/meta-perl/recipes-core/packagegroups/packagegroup-meta-perl.bb
similarity index 100%
rename from meta-perl/recipes-perl/packagegroups/packagegroup-meta-perl.bb
rename to meta-perl/recipes-core/packagegroups/packagegroup-meta-perl.bb
-- 
2.34.1

[meta-perl][PATCH 10/11] recipes-perl/images: move to recipes-core

[Tim Orling]

Be consistent with other layers like oe-core, meta-oe and meta-python

Signed-off-by: Tim Orling <tim.orling@konsulko.com>
---
 meta-perl/{recipes-perl => recipes-core}/images/meta-perl-base.bb | 0
 .../{recipes-perl => recipes-core}/images/meta-perl-image.bb      | 0
 .../images/meta-perl-ptest-image.bb                               | 0
 3 files changed, 0 insertions(+), 0 deletions(-)
 rename meta-perl/{recipes-perl => recipes-core}/images/meta-perl-base.bb (100%)
 rename meta-perl/{recipes-perl => recipes-core}/images/meta-perl-image.bb (100%)
 rename meta-perl/{recipes-perl => recipes-core}/images/meta-perl-ptest-image.bb (100%)

diff --git a/meta-perl/recipes-perl/images/meta-perl-base.bb b/meta-perl/recipes-core/images/meta-perl-base.bb
similarity index 100%
rename from meta-perl/recipes-perl/images/meta-perl-base.bb
rename to meta-perl/recipes-core/images/meta-perl-base.bb
diff --git a/meta-perl/recipes-perl/images/meta-perl-image.bb b/meta-perl/recipes-core/images/meta-perl-image.bb
similarity index 100%
rename from meta-perl/recipes-perl/images/meta-perl-image.bb
rename to meta-perl/recipes-core/images/meta-perl-image.bb
diff --git a/meta-perl/recipes-perl/images/meta-perl-ptest-image.bb b/meta-perl/recipes-core/images/meta-perl-ptest-image.bb
similarity index 100%
rename from meta-perl/recipes-perl/images/meta-perl-ptest-image.bb
rename to meta-perl/recipes-core/images/meta-perl-ptest-image.bb
-- 
2.34.1

[meta-perl][PATCH 11/11] meta-perl-ptest*-image: enable BBCLASSEXTEND parallel execution

[Tim Orling]

Similar to "core-image-ptest: Switch to BBCLASSEXTEND parallel execution"

Refactor meta-perl-ptest-image.bb for an image per ptest in the form of
meta-perl-ptest-image-XXX using class extensions.

* Add meta-perl-ptest-all-image.bb depends on ALL meta-perl-ptest-image-XXX in meta-perl
* Add meta-perl-ptest-fast-image.bb depends on meta-perl ptests which take less than ~30s

This allows parallel execution of the tests.

The downside to this approach is the parsing time of an image generating
this number of class extensions is slow but making it easier to execute
and collect test data should outweigh that.

A useful advantage is that the dependencies of each ptest package are
checked/tested individually.

Add to local.conf:
IMAGE_CLASSES += "testimage"

All the meta-perl-ptest-image-XXX can be built with:
$ bitbake meta-perl-ptest-all-image

Subsequently, all the ptest images can be run with:
$ bitbake -c testimage meta-perl-ptest-all-image

You will likely want to limit the number of parallel image tests:

BB_NUMBER_THREADS = "4"

based on the number of TAP devices you have defined or the number of simultaneous
QEMU sessions your server can support.

Results can be summarized with:

$ resulttool report tmp/log/oeqa/testresults.json

Signed-off-by: Tim Orling <tim.orling@konsulko.com>
---
 .../images/meta-perl-ptest-all-image.bb       | 25 ++++++++++++
 .../images/meta-perl-ptest-fast-image.bb      |  6 +++
 .../images/meta-perl-ptest-image.bb           | 38 +++++++++++++++++--
 3 files changed, 66 insertions(+), 3 deletions(-)
 create mode 100644 meta-perl/recipes-core/images/meta-perl-ptest-all-image.bb
 create mode 100644 meta-perl/recipes-core/images/meta-perl-ptest-fast-image.bb

diff --git a/meta-perl/recipes-core/images/meta-perl-ptest-all-image.bb b/meta-perl/recipes-core/images/meta-perl-ptest-all-image.bb
new file mode 100644
index 000000000..236043c22
--- /dev/null
+++ b/meta-perl/recipes-core/images/meta-perl-ptest-all-image.bb
@@ -0,0 +1,25 @@
+DESCRIPTION = "Recipe to trigger execution of all meta-perl ptest images."
+HOMEPAGE = "https://www.openembedded.org/"
+
+LICENSE = "MIT"
+
+inherit features_check nopackages
+REQUIRED_DISTRO_FEATURES = "ptest"
+
+require conf/include/ptest-packagelists-meta-perl.inc
+
+# Include the full set of ptests
+PTESTS_META_PERL = "${PTESTS_FAST_META_PERL} ${PTESTS_SLOW_META_PERL}"
+
+do_testimage[noexec] = "1"
+do_testimage[depends] = "${@' '.join(['meta-perl-ptest-image-'+x+':do_testimage' for x in d.getVar('PTESTS_META_PERL').split()])}"
+
+do_build[depends] = "${@' '.join(['meta-perl-ptest-image-'+x+':do_build' for x in d.getVar('PTESTS_META_PERL').split()])}"
+
+# normally image.bbclass would do this
+EXCLUDE_FROM_WORLD = "1"
+
+python () {
+    if bb.utils.contains('IMAGE_CLASSES', 'testimage', True, False, d):
+        bb.build.addtask("do_testimage", "", "", d)
+}
diff --git a/meta-perl/recipes-core/images/meta-perl-ptest-fast-image.bb b/meta-perl/recipes-core/images/meta-perl-ptest-fast-image.bb
new file mode 100644
index 000000000..5d8afa08e
--- /dev/null
+++ b/meta-perl/recipes-core/images/meta-perl-ptest-fast-image.bb
@@ -0,0 +1,6 @@
+require meta-perl-ptest-all-image.bb
+
+DESCRIPTION = "Recipe to trigger execution of all fast meta-perl ptest images."
+
+PTESTS_META_PERL = "${PTESTS_FAST_META_PERL}"
+
diff --git a/meta-perl/recipes-core/images/meta-perl-ptest-image.bb b/meta-perl/recipes-core/images/meta-perl-ptest-image.bb
index a9b4cc3c7..7cca6402b 100644
--- a/meta-perl/recipes-core/images/meta-perl-ptest-image.bb
+++ b/meta-perl/recipes-core/images/meta-perl-ptest-image.bb
@@ -1,9 +1,41 @@
+inherit features_check
+REQUIRED_DISTRO_FEATURES += "ptest"
+
 require  meta-perl-base.bb
 
 SUMMARY = "meta-perl build ptest image"
 
-inherit features_check
+require conf/include/ptest-packagelists-meta-perl.inc
 
-REQUIRED_DISTRO_FEATURES += "ptest"
+DESCRIPTION += "Also including the ${MCNAME} ptest package."
+HOMEPAGE = "https://www.openembedded.org/"
+
+PTESTS_META_PERL = "${PTESTS_SLOW_META_PERL} ${PTESTS_FAST_META_PERL}"
+
+IMAGE_INSTALL:append = " ${MCNAME}-ptest openssh"
+
+BBCLASSEXTEND = "${@' '.join(['mcextend:'+x for x in d.getVar('PTESTS_META_PERL').split()])}"
+
+# The image can be sufficiently large (~1.8GB) that we need to be careful that it fits in a live
+# image (which has a 4GB limit), so nullify the overhead factor (1.3x out of the
+# box) and explicitly add up to 1500MB.
+IMAGE_OVERHEAD_FACTOR = "1.0"
+IMAGE_ROOTFS_EXTRA_SPACE = "324288"
+# If a particular ptest needs more space, it can be customized:
+#IMAGE_ROOTFS_EXTRA_SPACE:virtclass-mcextend-<pn> = "1024288"
+
+# ptests need more memory than standard to avoid the OOM killer
+QB_MEM = "-m 1024"
+# If a particular ptest needs more memory, it can be customized:
+#QB_MEM:virtclass-mcextend-<pn> = "-m 4096"
+
+TEST_SUITES = "ping ssh parselogs ptest"
+
+# Sadly at the moment the full set of ptests is not robust enough and sporadically fails in random places
+PTEST_EXPECT_FAILURE = "1"
+
+python () {
+    if not d.getVar("MCNAME"):
+        raise bb.parse.SkipRecipe("No class extension set")
+}
 
-IMAGE_INSTALL += "packagegroup-meta-perl-ptest"
-- 
2.34.1

Re: [oe] [meta-oe][PATCH 04/11] meta-oe-ptest*-image: enable BBCLASSEXTEND parallel execution

[Khem Raj]

Thanks for this patchset Tim, really appreciate it. I have few comments.

On Sun, Apr 2, 2023 at 3:30 PM Tim Orling <ticotimo@gmail.com> wrote:
>
> Similar to "core-image-ptest: Switch to BBCLASSEXTEND parallel execution"
>
> Refactor meta-oe-ptest-image.bb for an image per ptest in the form of
> meta-oe-ptest-image-XXX using class extensions.
>
> * Add meta-oe-ptest-all-image.bb depends on ALL meta-oe-ptest-image-XXX in meta-oe
> * Add meta-oe-ptest-fast-image.bb depends on meta-oe ptests which take less than ~30s
>
> This allows parallel execution of the tests.
>
> The downside to this approach is the parsing time of an image generating
> this number of class extensions is slow but making it easier to execute
> and collect test data should outweigh that.
>
> A useful advantage is that the dependencies of each ptest package are
> checked/tested individually.
>
> Add to local.conf:
> IMAGE_CLASSES += "testimage"
>
> All the meta-oe-ptest-image-XXX can be built with:
> $ bitbake meta-oe-ptest-all-image
>
> Subsequently, all the ptest images can be run with:
> $ bitbake -c testimage meta-oe-ptest-all-image
>
> You will likely want to limit the number of parallel image tests:
>
> BB_NUMBER_THREADS = "4"
>
> based on the number of TAP devices you have defined or the number of simultaneous
> QEMU sessions your server can support.
>
> Results can be summarized with:
>
> $ resulttool report tmp/log/oeqa/testresult.json
>
> Signed-off-by: Tim Orling <tim.orling@konsulko.com>
> ---
>  .../images/meta-oe-ptest-all-image.bb         | 25 ++++++++++++
>  .../images/meta-oe-ptest-fast-image.bb        |  6 +++
>  .../images/meta-oe-ptest-image.bb             | 38 ++++++++++++++++++-
>  3 files changed, 68 insertions(+), 1 deletion(-)
>  create mode 100644 meta-oe/recipes-core/images/meta-oe-ptest-all-image.bb
>  create mode 100644 meta-oe/recipes-core/images/meta-oe-ptest-fast-image.bb

I think these images also need ( and also the ones in meta-python and
meta-perl too I think )

IMAGE_FEATURES += "debug-tweaks"

since some of them add ssh and it will be helpful to be able to do
passwordless access. and I was also wondering
if meta-oe-image-base.bb is really needed. We could simply include
core-image-base.bb here. This change also makes
meta-oe-image.bb and packagegroup-meta-oe.bb also questionable. Since
build part can be taken care by bitbake world.
but we can address that later.

>
> diff --git a/meta-oe/recipes-core/images/meta-oe-ptest-all-image.bb b/meta-oe/recipes-core/images/meta-oe-ptest-all-image.bb
> new file mode 100644
> index 000000000..0ea03dd2e
> --- /dev/null
> +++ b/meta-oe/recipes-core/images/meta-oe-ptest-all-image.bb
> @@ -0,0 +1,25 @@
> +DESCRIPTION = "Recipe to trigger execution of all meta-oe ptest images."
> +HOMEPAGE = "https://www.openembedded.org/"
> +
> +LICENSE = "MIT"
> +
> +inherit features_check nopackages
> +REQUIRED_DISTRO_FEATURES = "ptest"
> +
> +require conf/include/ptest-packagelists-meta-oe.inc
> +
> +# Include the full set of ptests
> +PTESTS_META_OE = "${PTESTS_FAST_META_OE} ${PTESTS_SLOW_META_OE}"
> +
> +do_testimage[noexec] = "1"
> +do_testimage[depends] = "${@' '.join(['meta-oe-ptest-image-'+x+':do_testimage' for x in d.getVar('PTESTS_META_OE').split()])}"
> +
> +do_build[depends] = "${@' '.join(['meta-oe-ptest-image-'+x+':do_build' for x in d.getVar('PTESTS_META_OE').split()])}"
> +
> +# normally image.bbclass would do this
> +EXCLUDE_FROM_WORLD = "1"
> +
> +python () {
> +    if bb.utils.contains('IMAGE_CLASSES', 'testimage', True, False, d):
> +        bb.build.addtask("do_testimage", "", "", d)
> +}
> diff --git a/meta-oe/recipes-core/images/meta-oe-ptest-fast-image.bb b/meta-oe/recipes-core/images/meta-oe-ptest-fast-image.bb
> new file mode 100644
> index 000000000..f56e841ff
> --- /dev/null
> +++ b/meta-oe/recipes-core/images/meta-oe-ptest-fast-image.bb
> @@ -0,0 +1,6 @@
> +require meta-oe-ptest-all-image.bb
> +
> +DESCRIPTION = "Recipe to trigger execution of all fast meta-oe ptest images."
> +
> +PTESTS_META_OE = "${PTESTS_FAST_META_OE}"
> +
> diff --git a/meta-oe/recipes-core/images/meta-oe-ptest-image.bb b/meta-oe/recipes-core/images/meta-oe-ptest-image.bb
> index 0166b9c41..951d91f82 100644
> --- a/meta-oe/recipes-core/images/meta-oe-ptest-image.bb
> +++ b/meta-oe/recipes-core/images/meta-oe-ptest-image.bb
> @@ -1,5 +1,41 @@
> +inherit features_check
> +REQUIRED_DISTRO_FEATURES = "ptest"
> +
> +require conf/include/ptest-packagelists-meta-oe.inc
> +
>  require  meta-oe-image-base.bb
>
>  SUMMARY = "meta-oe ptest test image"
>
> -IMAGE_INSTALL += "packagegroup-meta-oe"
> +DESCRIPTION += "Also including the ${MCNAME} ptest package."
> +HOMEPAGE = "https://www.openembedded.org/"
> +
> +PTESTS_META_OE = "${PTESTS_SLOW_META_OE} ${PTESTS_FAST_META_OE}"
> +
> +IMAGE_INSTALL:append = " ${MCNAME}-ptest openssh"
> +
> +BBCLASSEXTEND = "${@' '.join(['mcextend:'+x for x in d.getVar('PTESTS_META_OE').split()])}"
> +
> +# The image can be sufficiently large (~1.8GB) that we need to be careful that it fits in a live
> +# image (which has a 4GB limit), so nullify the overhead factor (1.3x out of the
> +# box) and explicitly add up to 1500MB.
> +IMAGE_OVERHEAD_FACTOR = "1.0"
> +IMAGE_ROOTFS_EXTRA_SPACE = "324288"
> +# If a particular ptest needs more space, it can be customized:
> +#IMAGE_ROOTFS_EXTRA_SPACE:virtclass-mcextend-<pn> = "1024288"
> +
> +# ptests need more memory than standard to avoid the OOM killer
> +QB_MEM = "-m 1024"
> +# If a particular ptest needs more memroy, it can be customized:
> +#QB_MEM:virtclass-mcextend-<pn> = "-m 4096"
> +
> +TEST_SUITES = "ping ssh parselogs ptest"
> +
> +# Sadly at the moment the full set of ptests is not robust enough and sporadically fails in random places
> +PTEST_EXPECT_FAILURE = "1"
> +
> +python () {
> +    if not d.getVar("MCNAME"):
> +        raise bb.parse.SkipRecipe("No class extension set")
> +}
> +
> --
> 2.34.1
>
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#101882): https://lists.openembedded.org/g/openembedded-devel/message/101882
> Mute This Topic: https://lists.openembedded.org/mt/98023314/1997914
> Group Owner: openembedded-devel+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-devel/unsub [raj.khem@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>

Re: [oe] [meta-oe][PATCH 04/11] meta-oe-ptest*-image: enable BBCLASSEXTEND parallel execution

[Tim Orling]

[-- Attachment #1: Type: text/plain, Size: 8653 bytes --]

On Sun, Apr 2, 2023 at 9:13 PM Khem Raj <raj.khem@gmail.com> wrote:

> Thanks for this patchset Tim, really appreciate it. I have few comments.
>
> On Sun, Apr 2, 2023 at 3:30 PM Tim Orling <ticotimo@gmail.com> wrote:
> >
> > Similar to "core-image-ptest: Switch to BBCLASSEXTEND parallel execution"
> >
> > Refactor meta-oe-ptest-image.bb for an image per ptest in the form of
> > meta-oe-ptest-image-XXX using class extensions.
> >
> > * Add meta-oe-ptest-all-image.bb depends on ALL meta-oe-ptest-image-XXX
> in meta-oe
> > * Add meta-oe-ptest-fast-image.bb depends on meta-oe ptests which take
> less than ~30s
> >
> > This allows parallel execution of the tests.
> >
> > The downside to this approach is the parsing time of an image generating
> > this number of class extensions is slow but making it easier to execute
> > and collect test data should outweigh that.
> >
> > A useful advantage is that the dependencies of each ptest package are
> > checked/tested individually.
> >
> > Add to local.conf:
> > IMAGE_CLASSES += "testimage"
> >
> > All the meta-oe-ptest-image-XXX can be built with:
> > $ bitbake meta-oe-ptest-all-image
> >
> > Subsequently, all the ptest images can be run with:
> > $ bitbake -c testimage meta-oe-ptest-all-image
> >
> > You will likely want to limit the number of parallel image tests:
> >
> > BB_NUMBER_THREADS = "4"
> >
> > based on the number of TAP devices you have defined or the number of
> simultaneous
> > QEMU sessions your server can support.
> >
> > Results can be summarized with:
> >
> > $ resulttool report tmp/log/oeqa/testresult.json
> >
> > Signed-off-by: Tim Orling <tim.orling@konsulko.com>
> > ---
> >  .../images/meta-oe-ptest-all-image.bb         | 25 ++++++++++++
> >  .../images/meta-oe-ptest-fast-image.bb        |  6 +++
> >  .../images/meta-oe-ptest-image.bb             | 38 ++++++++++++++++++-
> >  3 files changed, 68 insertions(+), 1 deletion(-)
> >  create mode 100644 meta-oe/recipes-core/images/
> meta-oe-ptest-all-image.bb
> >  create mode 100644 meta-oe/recipes-core/images/
> meta-oe-ptest-fast-image.bb
>
> I think these images also need ( and also the ones in meta-python and
> meta-perl too I think )
>
> IMAGE_FEATURES += "debug-tweaks"
>
> since some of them add ssh and it will be helpful to be able to do
> passwordless access.


I was mirroring the setup in oe-core. The default local.conf.sample in poky
enables
debug-tweaks and that is what I used as well as what the AutoBuilder uses.

The question is do we want to enforce a hard requirement on 'debug-tweaks'
when
some users may want to test in a different environment with a defined
user/password or
with ssh keys added to known_hosts, etc. Food for thought and I am open to
the change.


> and I was also wondering
> if meta-oe-image-base.bb is really needed. We could simply include
> core-image-base.bb here.


I was trying to do minimal change to what was already there. Open to the
change.


> This change also makes
> meta-oe-image.bb and packagegroup-meta-oe.bb also questionable. Since
> build part can be taken care by bitbake world.
> but we can address that later.


Again, I was trying to make only the changes to enable this new "per image
ptest" methodology. I personally
never used meta-oe-image.bb, meta-python-image.bb nor meta-perl-image.bb. I
am open to dropping them if
we decide to drop the packagegroups (which are TBH a bit of a challenge
since they are not necessarily always
up to date). In case others were using these for their workflows, I did not
want to cause breakage without some
discussion.

I will note that meta-oe-ptest-image.bb, meta-python-ptest-image.bb and
meta-perl-ptest-image.bb are all currently
needed in order to have layer isolation of the ptests. In
core-image-ptest.bb, the PTESTS variable is set and
I did not want to clobber that. Hence the new PTESTS_META_OE and friends
variables.

I'll also note that I tried to enable the "missing-ptest" QA check from
ptest.bbclass, but this is a per recipe check (not
an image level check) and is currently hard coded to only look at
meta/conf/distro/include/ptest-packagelists.inc and the
PTESTS_FAST and PTESTS_SLOW variables defined there. I didn't immediately
come up with a way to modularize that
behavior.


>
>
> > diff --git a/meta-oe/recipes-core/images/meta-oe-ptest-all-image.bb
> b/meta-oe/recipes-core/images/meta-oe-ptest-all-image.bb
> > new file mode 100644
> > index 000000000..0ea03dd2e
> > --- /dev/null
> > +++ b/meta-oe/recipes-core/images/meta-oe-ptest-all-image.bb
> > @@ -0,0 +1,25 @@
> > +DESCRIPTION = "Recipe to trigger execution of all meta-oe ptest images."
> > +HOMEPAGE = "https://www.openembedded.org/"
> > +
> > +LICENSE = "MIT"
> > +
> > +inherit features_check nopackages
> > +REQUIRED_DISTRO_FEATURES = "ptest"
> > +
> > +require conf/include/ptest-packagelists-meta-oe.inc
> > +
> > +# Include the full set of ptests
> > +PTESTS_META_OE = "${PTESTS_FAST_META_OE} ${PTESTS_SLOW_META_OE}"
> > +
> > +do_testimage[noexec] = "1"
> > +do_testimage[depends] = "${@'
> '.join(['meta-oe-ptest-image-'+x+':do_testimage' for x in
> d.getVar('PTESTS_META_OE').split()])}"
> > +
> > +do_build[depends] = "${@' '.join(['meta-oe-ptest-image-'+x+':do_build'
> for x in d.getVar('PTESTS_META_OE').split()])}"
> > +
> > +# normally image.bbclass would do this
> > +EXCLUDE_FROM_WORLD = "1"
> > +
> > +python () {
> > +    if bb.utils.contains('IMAGE_CLASSES', 'testimage', True, False, d):
> > +        bb.build.addtask("do_testimage", "", "", d)
> > +}
> > diff --git a/meta-oe/recipes-core/images/meta-oe-ptest-fast-image.bb
> b/meta-oe/recipes-core/images/meta-oe-ptest-fast-image.bb
> > new file mode 100644
> > index 000000000..f56e841ff
> > --- /dev/null
> > +++ b/meta-oe/recipes-core/images/meta-oe-ptest-fast-image.bb
> > @@ -0,0 +1,6 @@
> > +require meta-oe-ptest-all-image.bb
> > +
> > +DESCRIPTION = "Recipe to trigger execution of all fast meta-oe ptest
> images."
> > +
> > +PTESTS_META_OE = "${PTESTS_FAST_META_OE}"
> > +
> > diff --git a/meta-oe/recipes-core/images/meta-oe-ptest-image.bb
> b/meta-oe/recipes-core/images/meta-oe-ptest-image.bb
> > index 0166b9c41..951d91f82 100644
> > --- a/meta-oe/recipes-core/images/meta-oe-ptest-image.bb
> > +++ b/meta-oe/recipes-core/images/meta-oe-ptest-image.bb
> > @@ -1,5 +1,41 @@
> > +inherit features_check
> > +REQUIRED_DISTRO_FEATURES = "ptest"
> > +
> > +require conf/include/ptest-packagelists-meta-oe.inc
> > +
> >  require  meta-oe-image-base.bb
> >
> >  SUMMARY = "meta-oe ptest test image"
> >
> > -IMAGE_INSTALL += "packagegroup-meta-oe"
> > +DESCRIPTION += "Also including the ${MCNAME} ptest package."
> > +HOMEPAGE = "https://www.openembedded.org/"
> > +
> > +PTESTS_META_OE = "${PTESTS_SLOW_META_OE} ${PTESTS_FAST_META_OE}"
> > +
> > +IMAGE_INSTALL:append = " ${MCNAME}-ptest openssh"
> > +
> > +BBCLASSEXTEND = "${@' '.join(['mcextend:'+x for x in
> d.getVar('PTESTS_META_OE').split()])}"
> > +
> > +# The image can be sufficiently large (~1.8GB) that we need to be
> careful that it fits in a live
> > +# image (which has a 4GB limit), so nullify the overhead factor (1.3x
> out of the
> > +# box) and explicitly add up to 1500MB.
> > +IMAGE_OVERHEAD_FACTOR = "1.0"
> > +IMAGE_ROOTFS_EXTRA_SPACE = "324288"
> > +# If a particular ptest needs more space, it can be customized:
> > +#IMAGE_ROOTFS_EXTRA_SPACE:virtclass-mcextend-<pn> = "1024288"
> > +
> > +# ptests need more memory than standard to avoid the OOM killer
> > +QB_MEM = "-m 1024"
> > +# If a particular ptest needs more memroy, it can be customized:
> > +#QB_MEM:virtclass-mcextend-<pn> = "-m 4096"
> > +
> > +TEST_SUITES = "ping ssh parselogs ptest"
> > +
> > +# Sadly at the moment the full set of ptests is not robust enough and
> sporadically fails in random places
> > +PTEST_EXPECT_FAILURE = "1"
> > +
> > +python () {
> > +    if not d.getVar("MCNAME"):
> > +        raise bb.parse.SkipRecipe("No class extension set")
> > +}
> > +
> > --
> > 2.34.1
> >
> >
> > -=-=-=-=-=-=-=-=-=-=-=-
> > Links: You receive all messages sent to this group.
> > View/Reply Online (#101882):
> https://lists.openembedded.org/g/openembedded-devel/message/101882
> > Mute This Topic: https://lists.openembedded.org/mt/98023314/1997914
> > Group Owner: openembedded-devel+owner@lists.openembedded.org
> > Unsubscribe: https://lists.openembedded.org/g/openembedded-devel/unsub [
> raj.khem@gmail.com]
> > -=-=-=-=-=-=-=-=-=-=-=-
> >
>

[-- Attachment #2: Type: text/html, Size: 13934 bytes --]