[meta-virtualization][PATCH 0/4] Xen Test Framework OEQA

[meta-virtualization][PATCH 0/4] Xen Test Framework OEQA

[Christopher Clark]

This series integrates the Xen Test Framework (XTF) into the
OpenEmbedded QA (OEQA) structure and enables testing of Xen with XTF
test cases in Qemu via the standard OE testimage mechanism.
It builds upon the earlier work to package XTF and the XTF image.

I am aiming for this test to be enabled by default to support test
coverage for the Xen hypervisor and initial virtual machines, the
Xen toolstack and XTF.

Since many XTF test cases have hardware-dependent behaviour, the list of
XTF test cases to run is minimal by default but able to be extended as
appropriate for a particular downstream image or machine via provided
configuration variables documented within the test.

thanks,

Christopher


Christopher Clark (4):
  xen-image-minimal: supply bootloader config for qemux86-64 machine
  xtf-image: fix QB_DEFAULT_FSTYPE: only needed for x86-64 qemu
  xtf-image: update qemuboot settings
  xtf: add testimage integration to run XTF test cases in OEQA

 .gitignore                                   |   1 +
 classes/qemuboot-testimage-network.bbclass   |  17 +++
 lib/oeqa/runtime/cases/xtf_minimal.py        | 116 +++++++++++++++++++
 recipes-extended/images/xen-image-minimal.bb |   3 +-
 recipes-extended/images/xtf-image.bb         |  20 +++-
 wic/qemuboot-xen-x86-64.cfg                  |   8 ++
 wic/qemuboot-xen-x86-64.wks                  |  12 ++
 7 files changed, 171 insertions(+), 6 deletions(-)
 create mode 100644 classes/qemuboot-testimage-network.bbclass
 create mode 100644 lib/oeqa/runtime/cases/xtf_minimal.py
 create mode 100644 wic/qemuboot-xen-x86-64.cfg
 create mode 100644 wic/qemuboot-xen-x86-64.wks

-- 
2.25.1

[meta-virtualization][PATCH 1/4] xen-image-minimal: supply bootloader config for qemux86-64 machine

[Christopher Clark]

Add a dedicated bootloader config for the qemux86-64 machine so that the
'pmtmr=0' kernel command line argument can be provided, which removes an
error message that otherwise occurs in syslog during boot which is
detected by an OEQA test case causing it to fail.

A new kickstart file is provided that applies the new bootloader config
and it is supplied as an override to WKS_FILE for this image.

This is work towards enabling the Xen Test Framework (XTF) in the OEQA
testimage framework.

Signed-off-by: Christopher Clark <christopher.clark@starlab.io>
---
 recipes-extended/images/xen-image-minimal.bb |  1 +
 wic/qemuboot-xen-x86-64.cfg                  |  8 ++++++++
 wic/qemuboot-xen-x86-64.wks                  | 12 ++++++++++++
 3 files changed, 21 insertions(+)
 create mode 100644 wic/qemuboot-xen-x86-64.cfg
 create mode 100644 wic/qemuboot-xen-x86-64.wks

diff --git a/recipes-extended/images/xen-image-minimal.bb b/recipes-extended/images/xen-image-minimal.bb
index b4ddda2..fbdd007 100644
--- a/recipes-extended/images/xen-image-minimal.bb
+++ b/recipes-extended/images/xen-image-minimal.bb
@@ -74,6 +74,7 @@ build_syslinux_cfg () {
 
 # Enable runqemu. eg: runqemu xen-image-minimal nographic slirp
 WKS_FILE:x86-64 = "directdisk-xen.wks"
+WKS_FILE:qemux86-64 = "qemuboot-xen-x86-64.wks"
 QB_MEM ?= "-m 400"
 QB_DEFAULT_KERNEL ?= "none"
 QB_DEFAULT_FSTYPE ?= "wic"
diff --git a/wic/qemuboot-xen-x86-64.cfg b/wic/qemuboot-xen-x86-64.cfg
new file mode 100644
index 0000000..297e19b
--- /dev/null
+++ b/wic/qemuboot-xen-x86-64.cfg
@@ -0,0 +1,8 @@
+DEFAULT boot
+TIMEOUT 0
+PROMPT 0
+SERIAL 0 115200
+
+LABEL boot
+  KERNEL mboot.c32
+  APPEND /xen.gz console=com1,vga com1=115200,8n1 loglvl=all guest_loglvl=all --- /vmlinuz console=hvc0 earlyprintk=xen root=/dev/sda2 pmtmr=0
diff --git a/wic/qemuboot-xen-x86-64.wks b/wic/qemuboot-xen-x86-64.wks
new file mode 100644
index 0000000..593e0a6
--- /dev/null
+++ b/wic/qemuboot-xen-x86-64.wks
@@ -0,0 +1,12 @@
+# short-description: Create a 'pcbios' direct disk image with Xen hypervisor and bootloader config
+# long-description: Creates a partitioned legacy BIOS disk image to boot Xen
+# with a bootloader config that the user can directly dd to boot media.
+# Boot files are located on the first vfat partition.
+
+part /boot --source bootimg-biosxen --ondisk sda --label boot --active --align 1024
+
+# For the main partition, it can be useful to add additional space for VMs;
+# eg. increase partition size by appending: --size 10G
+part / --source rootfs --ondisk sda --use-uuid --fstype=ext4 --label root --align 1024
+
+bootloader --configfile="qemuboot-xen-x86-64.cfg"
-- 
2.25.1

[meta-virtualization][PATCH 2/4] xtf-image: fix QB_DEFAULT_FSTYPE: only needed for x86-64 qemu

[Christopher Clark]

The qemuboot launch method for x86-64 uses wic but Arm does not.

Signed-off-by: Christopher Clark <christopher.clark@starlab.io>
---
 recipes-extended/images/xtf-image.bb | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/recipes-extended/images/xtf-image.bb b/recipes-extended/images/xtf-image.bb
index 6bd5a34..21c0e7f 100644
--- a/recipes-extended/images/xtf-image.bb
+++ b/recipes-extended/images/xtf-image.bb
@@ -16,7 +16,7 @@ IMAGE_NAME="xtf"
 
 IMAGE_INSTALL:append = " xtf"
 
-QB_DEFAULT_FSTYPE = "wic"
+QB_DEFAULT_FSTYPE_x86-64 = "wic"
 
 # Set the dom0 memory level lower than that assigned to qemu so that Xen has
 # some available memory for allocating to the XTF microkernel guests to run:
-- 
2.25.1

[meta-virtualization][PATCH 3/4] xtf-image: update qemuboot settings

[Christopher Clark]

QB_MEM is now set via the qemuboot-xen-defaults bbclass and has the
better default value of 512MB, which fixes some test failures observed
when using the lower previous default value of 400MB.

The SYSLINUX_XEN_ARGS method of setting the intended dom0 memory level
did not take effect in practice, so remove it.

Signed-off-by: Christopher Clark <christopher.clark@starlab.io>
---
 recipes-extended/images/xtf-image.bb | 5 -----
 1 file changed, 5 deletions(-)

diff --git a/recipes-extended/images/xtf-image.bb b/recipes-extended/images/xtf-image.bb
index 21c0e7f..b73e4a5 100644
--- a/recipes-extended/images/xtf-image.bb
+++ b/recipes-extended/images/xtf-image.bb
@@ -17,8 +17,3 @@ IMAGE_NAME="xtf"
 IMAGE_INSTALL:append = " xtf"
 
 QB_DEFAULT_FSTYPE_x86-64 = "wic"
-
-# Set the dom0 memory level lower than that assigned to qemu so that Xen has
-# some available memory for allocating to the XTF microkernel guests to run:
-QB_MEM = "-m 400"
-SYSLINUX_XEN_ARGS:append = " dom0_mem=256M"
-- 
2.25.1

[meta-virtualization][PATCH 4/4] xtf: add testimage integration to run XTF test cases in OEQA

[Christopher Clark]

Add a new minimal OEQA test case to run the Xen Test Framework
test runner in the standard testimage step. Tested with qemux86-64
and designed for compatibility with Arm when XTF supports it.

To enable, append to local.conf:
    INHERIT += "testimage"
    QEMU_USE_SLIRP = "1"
    TEST_SERVER_IP = "127.0.0.1"

To run: bitbake -c testimage xtf-image

For inspection while it runs, at another shell prompt:
 * Observe the image booting:
 tail -f ${TMPDIR}/work/qemux86_64-*/xtf-image/*/testimage/qemu_boot_log.*
 * Observe the tests running once boot has completed:
 tail -f ${TMPDIR}/work/qemux86_64-*/xtf-image/*/temp/log.do_testimage

The XTF test sequence by default is a single XTF test case with minimal
hardware dependency to ensure that Xen is running, the Xen toolstack is
functional and XTF works. Additional XTF test cases for an image can be
configured via variables that are documented in the OEQA test case:
   * XTF_TEST_CASES_POPULATE
   * XTF_TEST_CASES_SKIP
   * XTF_TEST_CASES_REQUIRE

Since testimage requires a functioning network to perform the tests on
the image and the qemu MACHINES do not have networking enabled
this commit provides a new qemuboot-testimage-network bbclass to add an
image postprocess command to enable a functional eth0 for qemu MACHINES.

Signed-off-by: Christopher Clark <christopher.clark@starlab.io>
---
 .gitignore                                   |   1 +
 classes/qemuboot-testimage-network.bbclass   |  17 +++
 lib/oeqa/runtime/cases/xtf_minimal.py        | 116 +++++++++++++++++++
 recipes-extended/images/xen-image-minimal.bb |   2 +-
 recipes-extended/images/xtf-image.bb         |  15 +++
 5 files changed, 150 insertions(+), 1 deletion(-)
 create mode 100644 classes/qemuboot-testimage-network.bbclass
 create mode 100644 lib/oeqa/runtime/cases/xtf_minimal.py

diff --git a/.gitignore b/.gitignore
index 357aefe..a61332e 100644
--- a/.gitignore
+++ b/.gitignore
@@ -6,3 +6,4 @@ pyshtables.py
 /*.patch
 *~
 scripts/lib/wic/plugins/source/__pycache__
+lib/oeqa/runtime/cases/__pycache__
diff --git a/classes/qemuboot-testimage-network.bbclass b/classes/qemuboot-testimage-network.bbclass
new file mode 100644
index 0000000..18af1ee
--- /dev/null
+++ b/classes/qemuboot-testimage-network.bbclass
@@ -0,0 +1,17 @@
+# The recipe for init-ifupdown in core has a special-case for all
+# the Qemu MACHINES: it removes all external network interfaces
+# by default. However, eth0 is needed for testimage, so enable it here.
+enable_runqemu_network() {
+    : # no-op for non-qemu MACHINES
+}
+enable_runqemu_network:qemuall() {
+    if ! grep -q eth0 "${IMAGE_ROOTFS}${sysconfdir}/network/interfaces" ; then
+        cat <<EOF >>${IMAGE_ROOTFS}${sysconfdir}/network/interfaces
+
+# Network for testimage
+auto eth0
+iface eth0 inet dhcp
+EOF
+    fi
+}
+ROOTFS_POSTPROCESS_COMMAND += 'enable_runqemu_network;'
diff --git a/lib/oeqa/runtime/cases/xtf_minimal.py b/lib/oeqa/runtime/cases/xtf_minimal.py
new file mode 100644
index 0000000..6094cd9
--- /dev/null
+++ b/lib/oeqa/runtime/cases/xtf_minimal.py
@@ -0,0 +1,116 @@
+#
+# SPDX-License-Identifier: MIT
+#
+# Author: Christopher Clark
+# Copyright (c) Star Lab Corp, 2021
+#
+# Integration of the Xen Test Framework (XTF) into OpenEmbedded QA
+#
+# Since not all XTF test cases are appropriate for all test environments,
+# images or machine configurations the selection of XTF test cases to run
+# is determined by variables that can be set in an image recipe.
+#
+# * XTF_TEST_CASES_POPULATE
+# Specifies the list of queries passed to xtf-runner to populate the test list.
+# eg. 'pv64 livepatch-priv-check'
+#
+# Since the space character is meaningful and may be required within a populate clause,
+# the '|' character is used for separating multiple queries.
+# eg. 'pv64 livepatch-priv-check|pv32pae selftest'
+#
+# * XTF_TEST_CASES_SKIP
+# A space-separate list of test cases that should be skipped even if returned
+# from the queries specified in XTF_TEST_CASES_POPULATE.
+# eg. 'test-pv64-livepatch-priv-check'
+#
+# * XTF_TEST_CASES_REQUIRE
+# A space-separate list of test cases that must not be skipped even if missing
+# from the results of the queries specified in XTF_TEST_CASES_POPULATE.
+
+#----------
+# The default single test case here is chosen because it exercises XTF
+# and just Xen itself rather than any specifics of the hardware
+# (virtual or not) that Xen is running on.
+# TODO: this is an x86-specific test - revisit this choice when XTF supports Arm
+DEFAULT_POPULATE = 'pv64 livepatch-priv-check'
+#----------
+
+import json
+from oeqa.runtime.case import OERuntimeTestCase
+from oeqa.core.decorator.depends import OETestDepends
+from oeqa.core.decorator.oetimeout import OETimeout
+from oeqa.runtime.decorator.package import OEHasPackage
+
+def xtf_runner_exit_status(state):
+    """ Convert a xtf-runner exit code to a test result. """
+    return { 0: "SUCCESS",
+             1: "sys.exit 1",
+             2: "sys.exit 2",
+             3: "SKIP",
+             4: "ERROR",
+             5: "FAILURE",
+             6: "CRASH",
+    }[state]
+
+xtf_rundir = '/usr/libexec/xtf'
+
+class XTFMinimalTest(OERuntimeTestCase):
+
+    def query_xtf_cases(self, query_item):
+        (status, output) = self.target.run(
+                            'cd %s; ./xtf-runner --list %s' % \
+                            (xtf_rundir, query_item))
+        self.assertTrue(status == 0, msg='XTF runner failed')
+
+        populate_case_lines = output.split('\n')
+        while '' in populate_case_lines:
+            populate_case_lines.remove('')
+
+        return list(map(lambda x: x.lstrip().rstrip(), populate_case_lines))
+
+    def get_xtf_case_list(self):
+        xtf_cases = []
+
+        populate = self.tc.td.get('XTF_TEST_CASES_POPULATE')
+        skip = self.tc.td.get('XTF_TEST_CASES_SKIP')
+        require = self.tc.td.get('XTF_TEST_CASES_REQUIRE')
+
+        if populate is None:
+            populate = DEFAULT_POPULATE
+
+        for query_item in populate.split('|'):
+            xtf_cases.extend( self.query_xtf_cases(query_item) )
+
+        if skip is not None:
+            for skip_item in skip.split(' '):
+                while skip_item in xtf_cases:
+                    xtf_cases.remove(skip_item)
+
+        if require is not None:
+            for require_item in require.split(' '):
+                if require_item == '':
+                    continue
+                if not require_item in xtf_cases:
+                    xtf_cases.append(require_item)
+
+        self.logger.info('XTF cases: %s' % str(xtf_cases))
+        return xtf_cases
+
+    def run_xtf_case(self, xtf_case_name):
+        (status, output) = self.target.run('cd %s; ./xtf-runner %s' % \
+                                           (xtf_rundir, xtf_case_name))
+        self.assertTrue(status == 0, msg='XTF test %s failed: %s' % \
+                        (xtf_case_name, xtf_runner_exit_status(status)))
+
+    @OETimeout(2400)
+    @OETestDepends(['ssh.SSHTest.test_ssh'])
+    @OEHasPackage(['xtf'])
+    @OEHasPackage(['xen-tools'])
+    def test_xtf_minimal(self):
+
+        xtf_cases = self.get_xtf_case_list()
+
+        for xtf_case_name in xtf_cases:
+            self.logger.debug('Running XTF case: %s' % xtf_case_name)
+
+            self.run_xtf_case(xtf_case_name)
diff --git a/recipes-extended/images/xen-image-minimal.bb b/recipes-extended/images/xen-image-minimal.bb
index fbdd007..ea596ce 100644
--- a/recipes-extended/images/xen-image-minimal.bb
+++ b/recipes-extended/images/xen-image-minimal.bb
@@ -31,7 +31,7 @@ XEN_PCIBACK_MODULE:x86-64 = "kernel-module-xen-pciback"
 
 LICENSE = "MIT"
 
-inherit core-image qemuboot-xen-defaults qemuboot-xen-dtb
+inherit core-image qemuboot-xen-defaults qemuboot-xen-dtb qemuboot-testimage-network
 
 do_check_xen_state() {
     if [ "${@bb.utils.contains('DISTRO_FEATURES', 'xen', ' yes', 'no', d)}" = "no" ]; then
diff --git a/recipes-extended/images/xtf-image.bb b/recipes-extended/images/xtf-image.bb
index b73e4a5..a78959d 100644
--- a/recipes-extended/images/xtf-image.bb
+++ b/recipes-extended/images/xtf-image.bb
@@ -11,9 +11,24 @@ DESCRIPTION = "A minimal Xen Test Framework (XTF) image for testing the Xen hype
 #       ./xtf-runner --list pv
 #       # run an example test:
 #       ./xtf-runner test-pv64-livepatch-priv-check
+#
+# This image also supports the OE QA framework, so XTF tests can be
+# run from bitbake by adding the following (or similar) to local.conf:
+#
+#   INHERIT += "testimage"
+#   QEMU_USE_SLIRP = "1"
+#   TEST_SERVER_IP = "127.0.0.1"
+#
+# and the tests that are configured (see the xtf-oeqa-conf package)
+# can be run with: bitbake -c testimage xtf-image
+#
+# For testimage, see the qemu boot log: ${WORKDIR}/testimage/qemu_boot_log.*
+# and the test log: ${WORKDIR}/temp/log.do_testimage
 
 IMAGE_NAME="xtf"
 
 IMAGE_INSTALL:append = " xtf"
 
+DEFAULT_TEST_SUITES:append = " xtf_minimal"
+
 QB_DEFAULT_FSTYPE_x86-64 = "wic"
-- 
2.25.1

Re: [meta-virtualization][PATCH 0/4] Xen Test Framework OEQA

[Bruce Ashfield]

In message: [meta-virtualization][PATCH 0/4] Xen Test Framework OEQA
on 31/08/2021 Christopher Clark wrote:

> This series integrates the Xen Test Framework (XTF) into the
> OpenEmbedded QA (OEQA) structure and enables testing of Xen with XTF
> test cases in Qemu via the standard OE testimage mechanism.
> It builds upon the earlier work to package XTF and the XTF image.
> 
> I am aiming for this test to be enabled by default to support test
> coverage for the Xen hypervisor and initial virtual machines, the
> Xen toolstack and XTF.
> 
> Since many XTF test cases have hardware-dependent behaviour, the list of
> XTF test cases to run is minimal by default but able to be extended as
> appropriate for a particular downstream image or machine via provided
> configuration variables documented within the test.

I'm cycle constrained while I finish up my ELC presentation and demo,
so I haven't had a chance to test this yet.

But i have queued it and pushed it to master-next, so others can
more easily try it out.

Bruce

> 
> thanks,
> 
> Christopher
> 
> 
> Christopher Clark (4):
>   xen-image-minimal: supply bootloader config for qemux86-64 machine
>   xtf-image: fix QB_DEFAULT_FSTYPE: only needed for x86-64 qemu
>   xtf-image: update qemuboot settings
>   xtf: add testimage integration to run XTF test cases in OEQA
> 
>  .gitignore                                   |   1 +
>  classes/qemuboot-testimage-network.bbclass   |  17 +++
>  lib/oeqa/runtime/cases/xtf_minimal.py        | 116 +++++++++++++++++++
>  recipes-extended/images/xen-image-minimal.bb |   3 +-
>  recipes-extended/images/xtf-image.bb         |  20 +++-
>  wic/qemuboot-xen-x86-64.cfg                  |   8 ++
>  wic/qemuboot-xen-x86-64.wks                  |  12 ++
>  7 files changed, 171 insertions(+), 6 deletions(-)
>  create mode 100644 classes/qemuboot-testimage-network.bbclass
>  create mode 100644 lib/oeqa/runtime/cases/xtf_minimal.py
>  create mode 100644 wic/qemuboot-xen-x86-64.cfg
>  create mode 100644 wic/qemuboot-xen-x86-64.wks
> 
> -- 
> 2.25.1
> 

Re: [meta-virtualization][PATCH 0/4] Xen Test Framework OEQA

[Bruce Ashfield]

In message: [meta-virtualization][PATCH 0/4] Xen Test Framework OEQA
on 31/08/2021 Christopher Clark wrote:

> This series integrates the Xen Test Framework (XTF) into the
> OpenEmbedded QA (OEQA) structure and enables testing of Xen with XTF
> test cases in Qemu via the standard OE testimage mechanism.
> It builds upon the earlier work to package XTF and the XTF image.
> 
> I am aiming for this test to be enabled by default to support test
> coverage for the Xen hypervisor and initial virtual machines, the
> Xen toolstack and XTF.
> 
> Since many XTF test cases have hardware-dependent behaviour, the list of
> XTF test cases to run is minimal by default but able to be extended as
> appropriate for a particular downstream image or machine via provided
> configuration variables documented within the test.

These have now been merged to master!

Thanks for the series.

Bruce

> 
> thanks,
> 
> Christopher
> 
> 
> Christopher Clark (4):
>   xen-image-minimal: supply bootloader config for qemux86-64 machine
>   xtf-image: fix QB_DEFAULT_FSTYPE: only needed for x86-64 qemu
>   xtf-image: update qemuboot settings
>   xtf: add testimage integration to run XTF test cases in OEQA
> 
>  .gitignore                                   |   1 +
>  classes/qemuboot-testimage-network.bbclass   |  17 +++
>  lib/oeqa/runtime/cases/xtf_minimal.py        | 116 +++++++++++++++++++
>  recipes-extended/images/xen-image-minimal.bb |   3 +-
>  recipes-extended/images/xtf-image.bb         |  20 +++-
>  wic/qemuboot-xen-x86-64.cfg                  |   8 ++
>  wic/qemuboot-xen-x86-64.wks                  |  12 ++
>  7 files changed, 171 insertions(+), 6 deletions(-)
>  create mode 100644 classes/qemuboot-testimage-network.bbclass
>  create mode 100644 lib/oeqa/runtime/cases/xtf_minimal.py
>  create mode 100644 wic/qemuboot-xen-x86-64.cfg
>  create mode 100644 wic/qemuboot-xen-x86-64.wks
> 
> -- 
> 2.25.1
>