[PATCH 0/9] tests/avocado: ppc additions and other fixes

[PATCH 0/9] tests/avocado: ppc additions and other fixes

[Nicholas Piggin]

This is a set of ppc avocado tests that I have posted before, but
it has also grown a few misc changes, and patch 2 creates a
dependency with the added ppc tests that use the
AVOCADO_ALLOW_LONG_RUNTIME it introduces. I could take patches 2-3
via the ppc tree if they get acked, but happy to rebase ppc tests
if they'd better go via testing tree.

Thanks,
Nick

Nicholas Piggin (9):
  gitlab: fix s390x tag for avocado-system-centos
  tests/avocado: mark boot_linux.py long runtime instead of flaky
  tests/avocado: Mark x86-64 boot_linux.py TCG tests as long runtime
  tests/avocado: Enable replay_linux.py on ppc64 pseries
  tests/avocado: ppc add powernv10 boot_linux_console test
  tests/avocado: Add ppc pseries and powernv hash MMU tests
  tests/avocado: Add pseries KVM boot_linux test
  tests/avocado: ppc add hypervisor tests
  tests/avocado: Add FreeBSD distro boot tests for ppc

 MAINTAINERS                         |   1 +
 docs/devel/testing.rst              |   8 ++
 .gitlab-ci.d/buildtest.yml          |   2 +-
 tests/avocado/boot_freebsd.py       | 106 +++++++++++++++
 tests/avocado/boot_linux.py         |  32 ++---
 tests/avocado/boot_linux_console.py |   8 ++
 tests/avocado/ppc_hv_tests.py       | 201 ++++++++++++++++++++++++++++
 tests/avocado/ppc_powernv.py        |  23 +++-
 tests/avocado/ppc_pseries.py        |  20 ++-
 tests/avocado/replay_linux.py       |  76 ++++++++++-
 10 files changed, 453 insertions(+), 24 deletions(-)
 create mode 100644 tests/avocado/boot_freebsd.py
 create mode 100644 tests/avocado/ppc_hv_tests.py

-- 
2.42.0

[PATCH 1/9] gitlab: fix s390x tag for avocado-system-centos

[Nicholas Piggin]

The 390x tag should be s390x.

Signed-off-by: Nicholas Piggin <npiggin@gmail.com>
---
 .gitlab-ci.d/buildtest.yml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/.gitlab-ci.d/buildtest.yml b/.gitlab-ci.d/buildtest.yml
index 91663946de..cfe737aca2 100644
--- a/.gitlab-ci.d/buildtest.yml
+++ b/.gitlab-ci.d/buildtest.yml
@@ -184,7 +184,7 @@ avocado-system-centos:
   variables:
     IMAGE: centos8
     MAKE_CHECK_ARGS: check-avocado
-    AVOCADO_TAGS: arch:ppc64 arch:or1k arch:390x arch:x86_64 arch:rx
+    AVOCADO_TAGS: arch:ppc64 arch:or1k arch:s390x arch:x86_64 arch:rx
       arch:sh4 arch:nios2
 
 build-system-opensuse:
-- 
2.42.0

[PATCH 2/9] tests/avocado: mark boot_linux.py long runtime instead of flaky

[Nicholas Piggin]

The ppc64 and s390x tests were first marked skipIf GITLAB_CI by commit
c0c8687ef0f ("tests/avocado: disable BootLinuxPPC64 test in CI"), and
commit 0f26d94ec9e ("tests/acceptance: skip s390x_ccw_vrtio_tcg on
GitLab") due to being very heavy-weight for gitlab CI.

Commit 9b45cc99318 ("docs/devel: rationalise unstable gitlab tests under
FLAKY_TESTS") changed this to being flaky but it isn't really, it just
had a long runtime.

So introduce a new AVOCADO_ALLOW_LONG_RUNTIME variable and make these
tests require it. Re-testing the s390x and ppc64 tests on gitlab shows
about 100-150s runtime each, which is similar to the x86-64 tests.
Since these are among the longest running avocado tests, make x86-64
require long runtime as well.

Signed-off-by: Nicholas Piggin <npiggin@gmail.com>
---
 docs/devel/testing.rst      | 8 ++++++++
 tests/avocado/boot_linux.py | 8 ++------
 2 files changed, 10 insertions(+), 6 deletions(-)

diff --git a/docs/devel/testing.rst b/docs/devel/testing.rst
index bd132306c1..3a9c1327be 100644
--- a/docs/devel/testing.rst
+++ b/docs/devel/testing.rst
@@ -1346,6 +1346,14 @@ the environment.
 The definition of *large* is a bit arbitrary here, but it usually means an
 asset which occupies at least 1GB of size on disk when uncompressed.
 
+AVOCADO_ALLOW_LONG_RUNTIME
+^^^^^^^^^^^^^^^^^^^^^^^^^^
+Tests which have a long runtime will not be run unless that
+``AVOCADO_ALLOW_LONG_RUNTIME=1`` is exported on the environment.
+
+The definition of *long* is a bit arbitrary here, but it usually means a
+test which takes more than 100 seconds to complete.
+
 AVOCADO_ALLOW_UNTRUSTED_CODE
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 There are tests which will boot a kernel image or firmware that can be
diff --git a/tests/avocado/boot_linux.py b/tests/avocado/boot_linux.py
index 7c4769904e..6df0fc0489 100644
--- a/tests/avocado/boot_linux.py
+++ b/tests/avocado/boot_linux.py
@@ -93,13 +93,11 @@ class BootLinuxPPC64(LinuxTest):
 
     timeout = 360
 
-    @skipUnless(os.getenv('QEMU_TEST_FLAKY_TESTS'), 'Test is unstable on GitLab')
-
+    @skipUnless(os.getenv('AVOCADO_ALLOW_LONG_RUNTIME'), 'runtime limited')
     def test_pseries_tcg(self):
         """
         :avocado: tags=machine:pseries
         :avocado: tags=accel:tcg
-        :avocado: tags=flaky
         """
         self.require_accelerator("tcg")
         self.vm.add_args("-accel", "tcg")
@@ -113,13 +111,11 @@ class BootLinuxS390X(LinuxTest):
 
     timeout = 240
 
-    @skipUnless(os.getenv('QEMU_TEST_FLAKY_TESTS'), 'Test is unstable on GitLab')
-
+    @skipUnless(os.getenv('AVOCADO_ALLOW_LONG_RUNTIME'), 'runtime limited')
     def test_s390_ccw_virtio_tcg(self):
         """
         :avocado: tags=machine:s390-ccw-virtio
         :avocado: tags=accel:tcg
-        :avocado: tags=flaky
         """
         self.require_accelerator("tcg")
         self.vm.add_args("-accel", "tcg")
-- 
2.42.0

[PATCH 3/9] tests/avocado: Mark x86-64 boot_linux.py TCG tests as long runtime

[Nicholas Piggin]

Re-testing gitlab CI shows the x86-64 TCG tests take ~100s each, are
the longest-running tests. They are close to the ~150s taken by the
disabled ppc64 and s390x tests. From avocado-system-centos:

  boot_linux.py:BootLinuxX8664.test_pc_i440fx_tcg:  PASS (112.34 s)
  boot_linux.py:BootLinuxX8664.test_pc_q35_tcg:  PASS (97.05 s)
  boot_linux.py:BootLinuxPPC64.test_pseries_tcg:  PASS (148.86 s)
  boot_linux.py:BootLinuxS390X.test_s390_ccw_virtio_tcg:  PASS (149.83 s)

So disable the x86-64 tests as well.

Signed-off-by: Nicholas Piggin <npiggin@gmail.com>
---

The other way we could go is enabling them all since ppc64 and s390s are
now much faster than when they were originally disabled; or to only
enable q35, giving at least one boot_linux.py test.

[Test time results from here https://gitlab.com/npiggin/qemu/-/jobs/5842257510]

Thanks,
Nick
---
 tests/avocado/boot_linux.py | 16 +++++++---------
 1 file changed, 7 insertions(+), 9 deletions(-)

diff --git a/tests/avocado/boot_linux.py b/tests/avocado/boot_linux.py
index 6df0fc0489..a4a78122ac 100644
--- a/tests/avocado/boot_linux.py
+++ b/tests/avocado/boot_linux.py
@@ -14,6 +14,9 @@
 
 from avocado import skipUnless
 
+# We don't run TCG tests in CI, as booting the current Fedora OS in TCG tests
+# is very heavyweight (~100s per test). There are lighter weight distros which
+# we use in the machine_aarch64_virt.py, tux_baseline.py, etc.
 
 class BootLinuxX8664(LinuxTest):
     """
@@ -21,6 +24,7 @@ class BootLinuxX8664(LinuxTest):
     """
     timeout = 480
 
+    @skipUnless(os.getenv('AVOCADO_ALLOW_LONG_RUNTIME'), 'runtime limited')
     def test_pc_i440fx_tcg(self):
         """
         :avocado: tags=machine:pc
@@ -39,6 +43,7 @@ def test_pc_i440fx_kvm(self):
         self.vm.add_args("-accel", "kvm")
         self.launch_and_wait(set_up_ssh_connection=False)
 
+    @skipUnless(os.getenv('AVOCADO_ALLOW_LONG_RUNTIME'), 'runtime limited')
     def test_pc_q35_tcg(self):
         """
         :avocado: tags=machine:q35
@@ -58,9 +63,6 @@ def test_pc_q35_kvm(self):
         self.launch_and_wait(set_up_ssh_connection=False)
 
 
-# For Aarch64 we only boot KVM tests in CI as booting the current
-# Fedora OS in TCG tests is very heavyweight. There are lighter weight
-# distros which we use in the machine_aarch64_virt.py tests.
 class BootLinuxAarch64(LinuxTest):
     """
     :avocado: tags=arch:aarch64
@@ -84,14 +86,11 @@ def test_virt_kvm(self):
         self.launch_and_wait(set_up_ssh_connection=False)
 
 
-# See the tux_baseline.py tests for almost the same coverage in a lot
-# less time.
 class BootLinuxPPC64(LinuxTest):
     """
     :avocado: tags=arch:ppc64
     """
-
-    timeout = 360
+    timeout = 480
 
     @skipUnless(os.getenv('AVOCADO_ALLOW_LONG_RUNTIME'), 'runtime limited')
     def test_pseries_tcg(self):
@@ -108,8 +107,7 @@ class BootLinuxS390X(LinuxTest):
     """
     :avocado: tags=arch:s390x
     """
-
-    timeout = 240
+    timeout = 480
 
     @skipUnless(os.getenv('AVOCADO_ALLOW_LONG_RUNTIME'), 'runtime limited')
     def test_s390_ccw_virtio_tcg(self):
-- 
2.42.0

[PATCH 4/9] tests/avocado: Enable replay_linux.py on ppc64 pseries

[Nicholas Piggin]

Add a ppc64 pseries test. This tends to hang in the replay phase near
the end of the trace due to a missing event, so it is marked flaky.

spapr-vscsi IO is extremely slow when running in record-replay modes,
particularly when driven by SLOF. This causes tests to time-out even
after an hour, so this uses guestfish to extract the kernel and initrd
and boot them directly.

Signed-off-by: Nicholas Piggin <npiggin@gmail.com>
---
 tests/avocado/replay_linux.py | 76 ++++++++++++++++++++++++++++++++++-
 1 file changed, 75 insertions(+), 1 deletion(-)

diff --git a/tests/avocado/replay_linux.py b/tests/avocado/replay_linux.py
index 270ccc1eae..82daba9f3f 100644
--- a/tests/avocado/replay_linux.py
+++ b/tests/avocado/replay_linux.py
@@ -11,8 +11,9 @@
 import os
 import logging
 import time
+import subprocess
 
-from avocado import skipUnless
+from avocado import skipIf, skipUnless
 from avocado_qemu import BUILD_DIR
 from avocado.utils import cloudinit
 from avocado.utils import network
@@ -191,3 +192,76 @@ def test_virt_gicv3(self):
         self.run_rr(shift=3,
                     args=(*self.get_common_args(),
                           "-machine", "virt,gic-version=3"))
+
+# ppc64 pseries test.
+#
+# This machine tends to fail replay and hang very close to the end of the
+# trace, with missing events, which is still an open issue.
+#
+# spapr-scsi IO driven by SLOF/grub is extremely slow in record/replay mode,
+# so jump through some hoops to boot the kernel directly. With this, the test
+# runs in about 5 minutes (modulo hang), which suggests other machines may
+# have similar issues and could benefit from bypassing bootloaders.
+#
+ppc_deps = ["guestfish"] # dependent tools needed in the test setup/box.
+
+def which(tool):
+    """ looks up the full path for @tool, returns None if not found
+        or if @tool does not have executable permissions.
+    """
+    paths=os.getenv('PATH')
+    for p in paths.split(os.path.pathsep):
+        p = os.path.join(p, tool)
+        if os.path.exists(p) and os.access(p, os.X_OK):
+            return p
+    return None
+
+def ppc_missing_deps():
+    """ returns True if any of the test dependent tools are absent.
+    """
+    for dep in ppc_deps:
+        if which(dep) is None:
+            return True
+    return False
+
+@skipIf(ppc_missing_deps(), 'dependencies (%s) not installed' % ','.join(ppc_deps))
+@skipUnless(os.getenv('QEMU_TEST_FLAKY_TESTS'), 'known failure in trace replay')
+class ReplayLinuxPPC64(ReplayLinux):
+    """
+    :avocado: tags=arch:ppc64
+    :avocado: tags=accel:tcg
+    """
+
+    hdd = 'scsi-hd'
+    cd = 'scsi-cd'
+    bus = None
+
+    def setUp(self):
+        super().setUp()
+
+        # kernel, initramfs, and kernel cmdline are all taken by hand from
+        # the Fedora image.
+        self.kernel="vmlinuz-5.3.7-301.fc31.ppc64le"
+        self.initramfs="initramfs-5.3.7-301.fc31.ppc64le.img"
+        cmd = "guestfish --ro -a %s run "
+              ": mount /dev/sda2 / "
+              ": copy-out /boot/%s %s "
+              ": copy-out /boot/%s %s "
+              % (self.boot_path, self.kernel, self.workdir,
+                 self.initramfs, self.workdir)
+        subprocess.run(cmd.split())
+
+    def test_pseries(self):
+        """
+        :avocado: tags=machine:pseries
+        """
+        kernel=os.path.normpath(os.path.join(self.workdir, self.kernel))
+        initramfs=os.path.normpath(os.path.join(self.workdir, self.initramfs))
+        cmdline="root=UUID=8a409ee6-3cb3-4b06-a266-39e2dae3e5fa ro "
+                "no_timer_check net.ifnames=0 console=tty1 "
+                "console=ttyS0,115200n8"
+        self.run_rr(shift=1, args=("-device", "spapr-vscsi",
+                                   "-machine", "x-vof=on",
+                                   "-kernel", kernel,
+                                   "-initrd", initramfs,
+                                   "-append", cmdline))
-- 
2.42.0

[PATCH 5/9] tests/avocado: ppc add powernv10 boot_linux_console test

[Nicholas Piggin]

Add test for POWER10.

Signed-off-by: Nicholas Piggin <npiggin@gmail.com>
---
 tests/avocado/boot_linux_console.py | 8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/tests/avocado/boot_linux_console.py b/tests/avocado/boot_linux_console.py
index 3f0180e1f8..4f05bb7441 100644
--- a/tests/avocado/boot_linux_console.py
+++ b/tests/avocado/boot_linux_console.py
@@ -1386,6 +1386,14 @@ def test_ppc_powernv9(self):
         """
         self.do_test_ppc64_powernv('P9')
 
+    def test_ppc_powernv10(self):
+        """
+        :avocado: tags=arch:ppc64
+        :avocado: tags=machine:powernv10
+        :avocado: tags=accel:tcg
+        """
+        self.do_test_ppc64_powernv('P10')
+
     def test_ppc_g3beige(self):
         """
         :avocado: tags=arch:ppc
-- 
2.42.0

[PATCH 6/9] tests/avocado: Add ppc pseries and powernv hash MMU tests

[Nicholas Piggin]

POWER CPUs support hash and radix MMU modes. Linux supports running in
either mode, but defaults to radix. To keep up testing of QEMU's hash
MMU implementation, add some Linux hash boot tests.

Signed-off-by: Nicholas Piggin <npiggin@gmail.com>
---
 tests/avocado/ppc_powernv.py | 23 +++++++++++++++++++----
 tests/avocado/ppc_pseries.py | 20 +++++++++++++++++---
 2 files changed, 36 insertions(+), 7 deletions(-)

diff --git a/tests/avocado/ppc_powernv.py b/tests/avocado/ppc_powernv.py
index d0e5c07bde..4342941d5d 100644
--- a/tests/avocado/ppc_powernv.py
+++ b/tests/avocado/ppc_powernv.py
@@ -12,11 +12,11 @@
 class powernvMachine(QemuSystemTest):
 
     timeout = 90
-    KERNEL_COMMON_COMMAND_LINE = 'printk.time=0 '
+    KERNEL_COMMON_COMMAND_LINE = 'printk.time=0 console=hvc0 '
     panic_message = 'Kernel panic - not syncing'
     good_message = 'VFS: Cannot open root device'
 
-    def do_test_linux_boot(self):
+    def do_test_linux_boot(self, command_line = KERNEL_COMMON_COMMAND_LINE):
         self.require_accelerator("tcg")
         kernel_url = ('https://archives.fedoraproject.org/pub/archive'
                       '/fedora-secondary/releases/29/Everything/ppc64le/os'
@@ -25,9 +25,8 @@ def do_test_linux_boot(self):
         kernel_path = self.fetch_asset(kernel_url, asset_hash=kernel_hash)
 
         self.vm.set_console()
-        kernel_command_line = self.KERNEL_COMMON_COMMAND_LINE + 'console=hvc0'
         self.vm.add_args('-kernel', kernel_path,
-                         '-append', kernel_command_line)
+                         '-append', command_line)
         self.vm.launch()
 
     def test_linux_boot(self):
@@ -54,6 +53,22 @@ def test_linux_smp_boot(self):
         wait_for_console_pattern(self, console_pattern, self.panic_message)
         wait_for_console_pattern(self, self.good_message, self.panic_message)
 
+    def test_linux_smp_hpt_boot(self):
+        """
+        :avocado: tags=arch:ppc64
+        :avocado: tags=machine:powernv
+        :avocado: tags=accel:tcg
+        """
+
+        self.vm.add_args('-smp', '4')
+        self.do_test_linux_boot(self.KERNEL_COMMON_COMMAND_LINE +
+                                'disable_radix')
+        console_pattern = 'smp: Brought up 1 node, 4 CPUs'
+        wait_for_console_pattern(self, 'hash-mmu: Initializing hash mmu',
+                                 self.panic_message)
+        wait_for_console_pattern(self, console_pattern, self.panic_message)
+        wait_for_console_pattern(self, self.good_message, self.panic_message)
+
     def test_linux_smt_boot(self):
         """
         :avocado: tags=arch:ppc64
diff --git a/tests/avocado/ppc_pseries.py b/tests/avocado/ppc_pseries.py
index a8311e6555..74aaa4ac4a 100644
--- a/tests/avocado/ppc_pseries.py
+++ b/tests/avocado/ppc_pseries.py
@@ -12,11 +12,11 @@
 class pseriesMachine(QemuSystemTest):
 
     timeout = 90
-    KERNEL_COMMON_COMMAND_LINE = 'printk.time=0 '
+    KERNEL_COMMON_COMMAND_LINE = 'printk.time=0 console=hvc0 '
     panic_message = 'Kernel panic - not syncing'
     good_message = 'VFS: Cannot open root device'
 
-    def do_test_ppc64_linux_boot(self):
+    def do_test_ppc64_linux_boot(self, kernel_command_line = KERNEL_COMMON_COMMAND_LINE):
         kernel_url = ('https://archives.fedoraproject.org/pub/archive'
                       '/fedora-secondary/releases/29/Everything/ppc64le/os'
                       '/ppc/ppc64/vmlinuz')
@@ -24,7 +24,6 @@ def do_test_ppc64_linux_boot(self):
         kernel_path = self.fetch_asset(kernel_url, asset_hash=kernel_hash)
 
         self.vm.set_console()
-        kernel_command_line = self.KERNEL_COMMON_COMMAND_LINE + 'console=hvc0'
         self.vm.add_args('-kernel', kernel_path,
                          '-append', kernel_command_line)
         self.vm.launch()
@@ -62,6 +61,21 @@ def test_ppc64_linux_smp_boot(self):
         wait_for_console_pattern(self, console_pattern, self.panic_message)
         wait_for_console_pattern(self, self.good_message, self.panic_message)
 
+    def test_ppc64_linux_hpt_smp_boot(self):
+        """
+        :avocado: tags=arch:ppc64
+        :avocado: tags=machine:pseries
+        """
+
+        self.vm.add_args('-smp', '4')
+        self.do_test_ppc64_linux_boot(self.KERNEL_COMMON_COMMAND_LINE +
+                                      'disable_radix')
+        console_pattern = 'smp: Brought up 1 node, 4 CPUs'
+        wait_for_console_pattern(self, 'hash-mmu: Initializing hash mmu',
+                                 self.panic_message)
+        wait_for_console_pattern(self, console_pattern, self.panic_message)
+        wait_for_console_pattern(self, self.good_message, self.panic_message)
+
     def test_ppc64_linux_smt_boot(self):
         """
         :avocado: tags=arch:ppc64
-- 
2.42.0

[PATCH 7/9] tests/avocado: Add pseries KVM boot_linux test

[Nicholas Piggin]

ppc has no avocado tests for the KVM backend. Add a KVM boot_linux.py
test for pseries.

Signed-off-by: Nicholas Piggin <npiggin@gmail.com>
---
 tests/avocado/boot_linux.py | 8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/tests/avocado/boot_linux.py b/tests/avocado/boot_linux.py
index a4a78122ac..b20b2fc620 100644
--- a/tests/avocado/boot_linux.py
+++ b/tests/avocado/boot_linux.py
@@ -102,6 +102,14 @@ def test_pseries_tcg(self):
         self.vm.add_args("-accel", "tcg")
         self.launch_and_wait(set_up_ssh_connection=False)
 
+    def test_pseries_kvm(self):
+        """
+        :avocado: tags=machine:pseries
+        :avocado: tags=accel:kvm
+        """
+        self.require_accelerator("kvm")
+        self.vm.add_args("-accel", "kvm")
+        self.launch_and_wait(set_up_ssh_connection=False)
 
 class BootLinuxS390X(LinuxTest):
     """
-- 
2.42.0

[PATCH 8/9] tests/avocado: ppc add hypervisor tests

[Nicholas Piggin]

The powernv and pseries machines both provide hypervisor facilities
that are supported by KVM. This is a large and complicated set of
features that don't get much system-level testing in ppc tests.

Add a new test case for these which runs QEMU KVM inside the target.
This downloads an Alpine VM image, boots it and downloads and installs
the qemu package, then boots a virtual machine under it, re-using the
original Alpine VM image.

Signed-off-by: Nicholas Piggin <npiggin@gmail.com>
---
 MAINTAINERS                   |   1 +
 tests/avocado/ppc_hv_tests.py | 201 ++++++++++++++++++++++++++++++++++
 2 files changed, 202 insertions(+)
 create mode 100644 tests/avocado/ppc_hv_tests.py

diff --git a/MAINTAINERS b/MAINTAINERS
index 395f26ba86..ffe91a2bc5 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -1509,6 +1509,7 @@ F: tests/qtest/libqos/*spapr*
 F: tests/qtest/rtas*
 F: tests/qtest/libqos/rtas*
 F: tests/avocado/ppc_pseries.py
+F: tests/avocado/ppc_hv_tests.py
 
 PowerNV (Non-Virtualized)
 M: Cédric Le Goater <clg@kaod.org>
diff --git a/tests/avocado/ppc_hv_tests.py b/tests/avocado/ppc_hv_tests.py
new file mode 100644
index 0000000000..2162d6bd68
--- /dev/null
+++ b/tests/avocado/ppc_hv_tests.py
@@ -0,0 +1,201 @@
+# Tests that specifically try to exercise hypervisor features of the
+# target machines. powernv supports the Power hypervisor ISA, and
+# pseries supports the nested-HV hypervisor spec.
+#
+# Copyright (c) 2023 IBM Corporation
+#
+# This work is licensed under the terms of the GNU GPL, version 2 or
+# later.  See the COPYING file in the top-level directory.
+
+from avocado import skipIf, skipUnless
+from avocado.utils import archive
+from avocado_qemu import QemuSystemTest
+from avocado_qemu import wait_for_console_pattern, exec_command
+import os
+import time
+import subprocess
+
+deps = ["xorriso"] # dependent tools needed in the test setup/box.
+
+def which(tool):
+    """ looks up the full path for @tool, returns None if not found
+        or if @tool does not have executable permissions.
+    """
+    paths=os.getenv('PATH')
+    for p in paths.split(os.path.pathsep):
+        p = os.path.join(p, tool)
+        if os.path.exists(p) and os.access(p, os.X_OK):
+            return p
+    return None
+
+def missing_deps():
+    """ returns True if any of the test dependent tools are absent.
+    """
+    for dep in deps:
+        if which(dep) is None:
+            return True
+    return False
+
+# Alpine is a light weight distro that supports QEMU. These tests boot
+# that on the machine then run a QEMU guest inside it in KVM mode,
+# that runs the same Alpine distro image.
+# QEMU packages are downloaded and installed on each test. That's not a
+# large download, but it may be more polite to create qcow2 image with
+# QEMU already installed and use that.
+@skipUnless(os.getenv('AVOCADO_ALLOW_LARGE_STORAGE'), 'storage limited')
+@skipUnless(os.getenv('AVOCADO_ALLOW_LONG_RUNTIME'), 'runtime limited')
+@skipIf(missing_deps(), 'dependencies (%s) not installed' % ','.join(deps))
+class HypervisorTest(QemuSystemTest):
+
+    timeout = 1000
+    KERNEL_COMMON_COMMAND_LINE = 'printk.time=0 console=hvc0 '
+    panic_message = 'Kernel panic - not syncing'
+    good_message = 'VFS: Cannot open root device'
+
+    def extract_from_iso(self, iso, path):
+        """
+        Extracts a file from an iso file into the test workdir
+
+        :param iso: path to the iso file
+        :param path: path within the iso file of the file to be extracted
+        :returns: path of the extracted file
+        """
+        filename = os.path.basename(path)
+
+        cwd = os.getcwd()
+        os.chdir(self.workdir)
+
+        with open(filename, "w") as outfile:
+            cmd = "xorriso -osirrox on -indev %s -cpx %s %s" %
+                  (iso, path, filename)
+            subprocess.run(cmd.split(),
+                           stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
+
+        os.chdir(cwd)
+
+        # Return complete path to extracted file.  Because callers to
+        # extract_from_iso() specify 'path' with a leading slash, it is
+        # necessary to use os.path.relpath() as otherwise os.path.join()
+        # interprets it as an absolute path and drops the self.workdir part.
+        return os.path.normpath(os.path.join(self.workdir, filename))
+
+    def setUp(self):
+        super().setUp()
+
+        iso_url = ('https://dl-cdn.alpinelinux.org/alpine/v3.18/releases/ppc64le/alpine-standard-3.18.4-ppc64le.iso')
+
+        # Alpine use sha256 so I recalculated this myself
+        iso_sha256 = 'c26b8d3e17c2f3f0fed02b4b1296589c2390e6d5548610099af75300edd7b3ff'
+        iso_path = self.fetch_asset(iso_url, asset_hash=iso_sha256,
+                                    algorithm = "sha256")
+
+        self.iso_path = iso_path
+        self.vmlinuz = self.extract_from_iso(iso_path, '/boot/vmlinuz-lts')
+        self.initramfs = self.extract_from_iso(iso_path, '/boot/initramfs-lts')
+
+    def do_start_alpine(self):
+        self.vm.set_console()
+        self.vm.add_args("-kernel", self.vmlinuz)
+        self.vm.add_args("-initrd", self.initramfs)
+        kernel_command_line = self.KERNEL_COMMON_COMMAND_LINE
+        self.require_accelerator("tcg")
+
+        self.vm.add_args("-accel", "tcg,thread=multi")
+        self.vm.add_args("-smp", "4", "-m", "2g")
+        self.vm.add_args("-drive", f"file={self.iso_path},format=raw,if=none,id=drive0")
+
+        self.vm.launch()
+        wait_for_console_pattern(self, 'Welcome to Alpine Linux 3.18')
+        exec_command(self, 'root')
+        wait_for_console_pattern(self, 'localhost login:')
+        wait_for_console_pattern(self, 'You may change this message by editing /etc/motd.')
+        exec_command(self, 'setup-alpine -qe')
+        wait_for_console_pattern(self, 'localhost:~#')
+
+    def do_stop_alpine(self):
+        exec_command(self, 'poweroff')
+        wait_for_console_pattern(self, 'alpine:~#')
+        self.vm.wait()
+
+    def do_setup_kvm(self):
+        exec_command(self, 'echo http://dl-cdn.alpinelinux.org/alpine/v3.18/main > /etc/apk/repositories')
+        wait_for_console_pattern(self, 'alpine:~#')
+        exec_command(self, 'echo http://dl-cdn.alpinelinux.org/alpine/v3.18/community >> /etc/apk/repositories')
+        wait_for_console_pattern(self, 'alpine:~#')
+        exec_command(self, 'apk update')
+        wait_for_console_pattern(self, 'alpine:~#')
+        exec_command(self, 'apk add qemu-system-ppc64')
+        wait_for_console_pattern(self, 'alpine:~#')
+        exec_command(self, 'modprobe kvm-hv')
+        wait_for_console_pattern(self, 'alpine:~#')
+
+    # This uses the host's block device as the source file for guest block
+    # device for install media. This is a bit hacky but allows reuse of the
+    # iso without having a passthrough filesystem configured.
+    def do_test_kvm(self, hpt=False):
+        if hpt:
+            append = 'disable_radix'
+        else:
+            append = ''
+        exec_command(self, 'qemu-system-ppc64 -nographic -smp 2 -m 1g '
+                           '-machine pseries,x-vof=on,accel=kvm '
+                           '-machine cap-cfpc=broken,cap-sbbc=broken,'
+                                    'cap-ibs=broken,cap-ccf-assist=off '
+                           '-drive file=/dev/nvme0n1,format=raw,readonly=on '
+                           '-initrd /media/nvme0n1/boot/initramfs-lts '
+                           '-kernel /media/nvme0n1/boot/vmlinuz-lts '
+                           '-append \'usbcore.nousb ' + append + '\'')
+        # Alpine 3.18 kernel seems to crash in XHCI USB driver.
+        wait_for_console_pattern(self, 'Welcome to Alpine Linux 3.18')
+        exec_command(self, 'root')
+        wait_for_console_pattern(self, 'localhost login:')
+        wait_for_console_pattern(self, 'You may change this message by editing /etc/motd.')
+        exec_command(self, 'poweroff >& /dev/null')
+        wait_for_console_pattern(self, 'localhost:~#')
+        wait_for_console_pattern(self, 'reboot: Power down')
+        time.sleep(1)
+        exec_command(self, '') # console has strange issue after qemu exit
+        exec_command(self, 'reset')
+        exec_command(self, 'echo VM finished')
+        wait_for_console_pattern(self, 'VM finished')
+
+    def test_hv_pseries(self):
+        """
+        :avocado: tags=arch:ppc64
+        :avocado: tags=machine:pseries
+        :avocado: tags=accel:tcg
+        """
+        self.vm.add_args('-device', 'nvme,serial=1234,drive=drive0')
+        self.vm.add_args("-machine", "x-vof=on,cap-nested-hv=on")
+        self.do_start_alpine()
+        self.do_setup_kvm()
+        self.do_test_kvm()
+        self.do_stop_alpine()
+
+    def test_hv_pseries_kvm(self):
+        """
+        :avocado: tags=arch:ppc64
+        :avocado: tags=machine:pseries
+        :avocado: tags=accel:kvm
+        """
+        self.vm.add_args('-device', 'nvme,serial=1234,drive=drive0')
+        self.vm.add_args("-machine", "x-vof=on,cap-nested-hv=on")
+        self.do_start_alpine()
+        self.do_setup_kvm()
+        self.do_test_kvm()
+        self.do_stop_alpine()
+
+    def test_hv_powernv(self):
+        """
+        :avocado: tags=arch:ppc64
+        :avocado: tags=machine:powernv
+        :avocado: tags=accel:tcg
+        """
+        self.vm.add_args('-device', 'nvme,bus=pcie.2,addr=0x0,serial=1234,drive=drive0',
+                         '-device', 'e1000e,netdev=net0,mac=C0:FF:EE:00:00:02,bus=pcie.0,addr=0x0',
+                         '-netdev', 'user,id=net0,hostfwd=::20022-:22,hostname=alpine')
+        self.do_start_alpine()
+        self.do_setup_kvm()
+        self.do_test_kvm()
+        self.do_test_kvm(True)
+        self.do_stop_alpine()
-- 
2.42.0

[PATCH 9/9] tests/avocado: Add FreeBSD distro boot tests for ppc

[Nicholas Piggin]

FreeBSD project provides qcow2 images that work well for testing QEMU.
Add pseries tests for HPT and Radix, KVM and TCG. This uses a short
term VM image, because FreeBSD has not set up long term builds for
ppc64 at present.

Other architectures could be added so this does not get a ppc_ prefix
but is instead named similarly to boot_linux.

Reviewed-by: Warner Losh <imp@bsdimp.com>
Signed-off-by: Nicholas Piggin <npiggin@gmail.com>
---
Unfortunately the latest stable (14.0) x86-64 VM image does not seem to
output to console by default and I've not been able to find a reliable
way to edit the filesystem to change the boot loader options, or use
console input in the test case to change it on the fly.

Thanks,
Nick
---
 tests/avocado/boot_freebsd.py | 106 ++++++++++++++++++++++++++++++++++
 1 file changed, 106 insertions(+)
 create mode 100644 tests/avocado/boot_freebsd.py

diff --git a/tests/avocado/boot_freebsd.py b/tests/avocado/boot_freebsd.py
new file mode 100644
index 0000000000..79c68b149a
--- /dev/null
+++ b/tests/avocado/boot_freebsd.py
@@ -0,0 +1,106 @@
+# Functional tests that boot FreeBSD in various configurations
+#
+# Copyright (c) 2023 IBM Corporation
+#
+# This work is licensed under the terms of the GNU GPL, version 2 or
+# later. See the COPYING file in the top-level directory.
+
+import os
+
+from avocado import skipUnless
+from avocado_qemu import QemuSystemTest
+from avocado_qemu import wait_for_console_pattern
+from avocado_qemu import exec_command
+from avocado.utils import archive
+from avocado.utils import process
+from avocado.utils.path import find_command
+
+@skipUnless(os.getenv('AVOCADO_ALLOW_LARGE_STORAGE'), 'storage limited')
+@skipUnless(os.getenv('AVOCADO_ALLOW_LONG_RUNTIME'), 'runtime limited')
+class BootFreeBSDPPC64(QemuSystemTest):
+    """
+    :avocado: tags=arch:ppc64
+    """
+
+    timeout = 360
+
+    def setUp(self):
+        super().setUp()
+
+        # We need zstd for all the tests
+        # See https://github.com/avocado-framework/avocado/issues/5609
+        zstd = find_command('zstd', False)
+        if zstd is False:
+            self.cancel('Could not find "zstd", which is required to '
+                        'decompress rootfs')
+        drive_url = ('https://artifact.ci.freebsd.org/snapshot/15.0-CURRENT/8a735ffdf04936c6785ac4fa31486639262dd416/powerpc/powerpc64le/disk.qcow2.zst')
+        drive_hash = '95d863dbbc4b60f4899d1ef21d6489fca05bf03d'
+        drive_path_zstd = self.fetch_asset(drive_url, asset_hash=drive_hash)
+        drive_path = os.path.join(self.workdir, 'disk.qcow2')
+
+        cmd = f"{zstd} -d {drive_path_zstd} -o {drive_path}"
+        process.run(cmd)
+
+        self.drive_opt = f"file={drive_path},format=qcow2,if=virtio"
+
+    def run_pseries_test(self, force_HPT=False):
+        if force_HPT:
+            self.vm.add_args('-m', '4g')
+        else:
+            self.vm.add_args('-m', '1g')
+        self.vm.add_args('-smp', '4')
+        self.vm.add_args('-drive', self.drive_opt)
+        self.vm.add_args('-net', 'nic,model=virtio')
+        self.vm.set_console()
+        self.vm.launch()
+
+        wait_for_console_pattern(self, 'Hit [Enter] to boot immediately, or any other key for command prompt.')
+        if force_HPT:
+            exec_command(self, 'x')
+            wait_for_console_pattern(self, 'OK')
+            exec_command(self, 'set radix_mmu=0')
+            exec_command(self, 'boot')
+            wait_for_console_pattern(self, 'cas: selected hash MMU', 'panic:')
+        else:
+            exec_command(self, '')
+            wait_for_console_pattern(self, 'cas: selected radix MMU', 'panic:')
+
+        wait_for_console_pattern(self, 'FreeBSD 15.0-CURRENT', 'panic:')
+        wait_for_console_pattern(self, 'FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs', 'panic:')
+        wait_for_console_pattern(self, 'FreeBSD/powerpc (Amnesiac) (ttyu0)', 'panic:')
+
+    def test_pseries_tcg(self):
+        """
+        :avocado: tags=arch:ppc64
+        :avocado: tags=machine:pseries
+        :avocado: tags=accel:tcg
+        """
+        self.require_accelerator("tcg")
+        self.run_pseries_test()
+
+    def test_pseries_hpt_tcg(self):
+        """
+        :avocado: tags=arch:ppc64
+        :avocado: tags=machine:pseries
+        :avocado: tags=accel:tcg
+        """
+        self.require_accelerator("tcg")
+        self.run_pseries_test(force_HPT=True)
+
+    def test_pseries_kvm(self):
+        """
+        :avocado: tags=arch:ppc64
+        :avocado: tags=machine:pseries
+        :avocado: tags=accel:kvm
+        """
+        self.require_accelerator("kvm")
+        self.run_pseries_test()
+
+    def test_pseries_hpt_kvm(self):
+        """
+        :avocado: tags=arch:ppc64
+        :avocado: tags=machine:pseries
+        :avocado: tags=accel:kvm
+        """
+        self.require_accelerator("kvm")
+        self.run_pseries_test(force_HPT=True)
-- 
2.42.0

Re: [PATCH 1/9] gitlab: fix s390x tag for avocado-system-centos

[Cédric Le Goater]

On 1/7/24 18:01, Nicholas Piggin wrote:
> The 390x tag should be s390x.
> 
> Signed-off-by: Nicholas Piggin <npiggin@gmail.com>


Reviewed-by: Cédric Le Goater <clg@kaod.org>

Thanks,

C.


> ---
>   .gitlab-ci.d/buildtest.yml | 2 +-
>   1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/.gitlab-ci.d/buildtest.yml b/.gitlab-ci.d/buildtest.yml
> index 91663946de..cfe737aca2 100644
> --- a/.gitlab-ci.d/buildtest.yml
> +++ b/.gitlab-ci.d/buildtest.yml
> @@ -184,7 +184,7 @@ avocado-system-centos:
>     variables:
>       IMAGE: centos8
>       MAKE_CHECK_ARGS: check-avocado
> -    AVOCADO_TAGS: arch:ppc64 arch:or1k arch:390x arch:x86_64 arch:rx
> +    AVOCADO_TAGS: arch:ppc64 arch:or1k arch:s390x arch:x86_64 arch:rx
>         arch:sh4 arch:nios2
>   
>   build-system-opensuse:

Re: [PATCH 2/9] tests/avocado: mark boot_linux.py long runtime instead of flaky

[Cédric Le Goater]

On 1/7/24 18:01, Nicholas Piggin wrote:
> The ppc64 and s390x tests were first marked skipIf GITLAB_CI by commit
> c0c8687ef0f ("tests/avocado: disable BootLinuxPPC64 test in CI"), and
> commit 0f26d94ec9e ("tests/acceptance: skip s390x_ccw_vrtio_tcg on
> GitLab") due to being very heavy-weight for gitlab CI.
> 
> Commit 9b45cc99318 ("docs/devel: rationalise unstable gitlab tests under
> FLAKY_TESTS") changed this to being flaky but it isn't really, it just
> had a long runtime.

I agree. The same comment applies to the aspeed tests.
  
> So introduce a new AVOCADO_ALLOW_LONG_RUNTIME variable and make these
> tests require it. Re-testing the s390x and ppc64 tests on gitlab shows
> about 100-150s runtime each, which is similar to the x86-64 tests.
> Since these are among the longest running avocado tests, make x86-64
> require long runtime as well.
> 
> Signed-off-by: Nicholas Piggin <npiggin@gmail.com>
>
> ---
>   docs/devel/testing.rst      | 8 ++++++++
>   tests/avocado/boot_linux.py | 8 ++------
>   2 files changed, 10 insertions(+), 6 deletions(-)
> 
> diff --git a/docs/devel/testing.rst b/docs/devel/testing.rst
> index bd132306c1..3a9c1327be 100644
> --- a/docs/devel/testing.rst
> +++ b/docs/devel/testing.rst
> @@ -1346,6 +1346,14 @@ the environment.
>   The definition of *large* is a bit arbitrary here, but it usually means an
>   asset which occupies at least 1GB of size on disk when uncompressed.
>   
> +AVOCADO_ALLOW_LONG_RUNTIME
> +^^^^^^^^^^^^^^^^^^^^^^^^^^
> +Tests which have a long runtime will not be run unless that
> +``AVOCADO_ALLOW_LONG_RUNTIME=1`` is exported on the environment.
> +
> +The definition of *long* is a bit arbitrary here, but it usually means a
> +test which takes more than 100 seconds to complete.
Defining a limit is the difficult part. The Aspeed SDK tests have been
tagged flaky for CI even though they run in less than 50s. Anyhow,

Reviewed-by: Cédric Le Goater <clg@kaod.org>

Thanks,

C.




> +
>   AVOCADO_ALLOW_UNTRUSTED_CODE
>   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>   There are tests which will boot a kernel image or firmware that can be
> diff --git a/tests/avocado/boot_linux.py b/tests/avocado/boot_linux.py
> index 7c4769904e..6df0fc0489 100644
> --- a/tests/avocado/boot_linux.py
> +++ b/tests/avocado/boot_linux.py
> @@ -93,13 +93,11 @@ class BootLinuxPPC64(LinuxTest):
>   
>       timeout = 360
>   
> -    @skipUnless(os.getenv('QEMU_TEST_FLAKY_TESTS'), 'Test is unstable on GitLab')
> -
> +    @skipUnless(os.getenv('AVOCADO_ALLOW_LONG_RUNTIME'), 'runtime limited')
>       def test_pseries_tcg(self):
>           """
>           :avocado: tags=machine:pseries
>           :avocado: tags=accel:tcg
> -        :avocado: tags=flaky
>           """
>           self.require_accelerator("tcg")
>           self.vm.add_args("-accel", "tcg")
> @@ -113,13 +111,11 @@ class BootLinuxS390X(LinuxTest):
>   
>       timeout = 240
>   
> -    @skipUnless(os.getenv('QEMU_TEST_FLAKY_TESTS'), 'Test is unstable on GitLab')
> -
> +    @skipUnless(os.getenv('AVOCADO_ALLOW_LONG_RUNTIME'), 'runtime limited')
>       def test_s390_ccw_virtio_tcg(self):
>           """
>           :avocado: tags=machine:s390-ccw-virtio
>           :avocado: tags=accel:tcg
> -        :avocado: tags=flaky
>           """
>           self.require_accelerator("tcg")
>           self.vm.add_args("-accel", "tcg")

Re: [PATCH 5/9] tests/avocado: ppc add powernv10 boot_linux_console test

[Cédric Le Goater]

On 1/7/24 18:01, Nicholas Piggin wrote:
> Add test for POWER10.
> 
> Signed-off-by: Nicholas Piggin <npiggin@gmail.com>


Reviewed-by: Cédric Le Goater <clg@kaod.org>

Thanks,

C.


> ---
>   tests/avocado/boot_linux_console.py | 8 ++++++++
>   1 file changed, 8 insertions(+)
> 
> diff --git a/tests/avocado/boot_linux_console.py b/tests/avocado/boot_linux_console.py
> index 3f0180e1f8..4f05bb7441 100644
> --- a/tests/avocado/boot_linux_console.py
> +++ b/tests/avocado/boot_linux_console.py
> @@ -1386,6 +1386,14 @@ def test_ppc_powernv9(self):
>           """
>           self.do_test_ppc64_powernv('P9')
>   
> +    def test_ppc_powernv10(self):
> +        """
> +        :avocado: tags=arch:ppc64
> +        :avocado: tags=machine:powernv10
> +        :avocado: tags=accel:tcg
> +        """
> +        self.do_test_ppc64_powernv('P10')
> +
>       def test_ppc_g3beige(self):
>           """
>           :avocado: tags=arch:ppc

Re: [PATCH 7/9] tests/avocado: Add pseries KVM boot_linux test

[Cédric Le Goater]

On 1/7/24 18:01, Nicholas Piggin wrote:
> ppc has no avocado tests for the KVM backend. Add a KVM boot_linux.py
> test for pseries.
> 
> Signed-off-by: Nicholas Piggin <npiggin@gmail.com>


Reviewed-by: Cédric Le Goater <clg@kaod.org>

Thanks,

C.


> ---
>   tests/avocado/boot_linux.py | 8 ++++++++
>   1 file changed, 8 insertions(+)
> 
> diff --git a/tests/avocado/boot_linux.py b/tests/avocado/boot_linux.py
> index a4a78122ac..b20b2fc620 100644
> --- a/tests/avocado/boot_linux.py
> +++ b/tests/avocado/boot_linux.py
> @@ -102,6 +102,14 @@ def test_pseries_tcg(self):
>           self.vm.add_args("-accel", "tcg")
>           self.launch_and_wait(set_up_ssh_connection=False)
>   
> +    def test_pseries_kvm(self):
> +        """
> +        :avocado: tags=machine:pseries
> +        :avocado: tags=accel:kvm
> +        """
> +        self.require_accelerator("kvm")
> +        self.vm.add_args("-accel", "kvm")
> +        self.launch_and_wait(set_up_ssh_connection=False)
>   
>   class BootLinuxS390X(LinuxTest):
>       """

Re: [PATCH 1/9] gitlab: fix s390x tag for avocado-system-centos

[Philippe Mathieu-Daudé]

On 7/1/24 18:01, Nicholas Piggin wrote:
> The 390x tag should be s390x.
> 
> Signed-off-by: Nicholas Piggin <npiggin@gmail.com>
> ---
>   .gitlab-ci.d/buildtest.yml | 2 +-
>   1 file changed, 1 insertion(+), 1 deletion(-)

Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org>

Re: [PATCH 2/9] tests/avocado: mark boot_linux.py long runtime instead of flaky

[Thomas Huth]

On 07/01/2024 18.01, Nicholas Piggin wrote:
> The ppc64 and s390x tests were first marked skipIf GITLAB_CI by commit
> c0c8687ef0f ("tests/avocado: disable BootLinuxPPC64 test in CI"), and
> commit 0f26d94ec9e ("tests/acceptance: skip s390x_ccw_vrtio_tcg on
> GitLab") due to being very heavy-weight for gitlab CI.
> 
> Commit 9b45cc99318 ("docs/devel: rationalise unstable gitlab tests under
> FLAKY_TESTS") changed this to being flaky but it isn't really, it just
> had a long runtime.
> 
> So introduce a new AVOCADO_ALLOW_LONG_RUNTIME variable and make these
> tests require it. Re-testing the s390x and ppc64 tests on gitlab shows
> about 100-150s runtime each, which is similar to the x86-64 tests.
> Since these are among the longest running avocado tests, make x86-64
> require long runtime as well.
> 
> Signed-off-by: Nicholas Piggin <npiggin@gmail.com>
> ---
>   docs/devel/testing.rst      | 8 ++++++++
>   tests/avocado/boot_linux.py | 8 ++------
>   2 files changed, 10 insertions(+), 6 deletions(-)
> 
> diff --git a/docs/devel/testing.rst b/docs/devel/testing.rst
> index bd132306c1..3a9c1327be 100644
> --- a/docs/devel/testing.rst
> +++ b/docs/devel/testing.rst
> @@ -1346,6 +1346,14 @@ the environment.
>   The definition of *large* is a bit arbitrary here, but it usually means an
>   asset which occupies at least 1GB of size on disk when uncompressed.
>   
> +AVOCADO_ALLOW_LONG_RUNTIME
> +^^^^^^^^^^^^^^^^^^^^^^^^^^
> +Tests which have a long runtime will not be run unless that
> +``AVOCADO_ALLOW_LONG_RUNTIME=1`` is exported on the environment.
> +
> +The definition of *long* is a bit arbitrary here, but it usually means a
> +test which takes more than 100 seconds to complete.

For the qtests (and others), we are using "make check SPEED=slow", so what 
about just re-using the SPEED environment variable instead? IMHO that would 
be more consistent?

  Thomas

Re: [PATCH 9/9] tests/avocado: Add FreeBSD distro boot tests for ppc

[Cédric Le Goater]

On 1/7/24 18:01, Nicholas Piggin wrote:
> FreeBSD project provides qcow2 images that work well for testing QEMU.
> Add pseries tests for HPT and Radix, KVM and TCG. This uses a short
> term VM image, because FreeBSD has not set up long term builds for
> ppc64 at present.
> 
> Other architectures could be added so this does not get a ppc_ prefix
> but is instead named similarly to boot_linux.
> 
> Reviewed-by: Warner Losh <imp@bsdimp.com>
> Signed-off-by: Nicholas Piggin <npiggin@gmail.com>
> ---
> Unfortunately the latest stable (14.0) x86-64 VM image does not seem to
> output to console by default and I've not been able to find a reliable
> way to edit the filesystem to change the boot loader options, or use
> console input in the test case to change it on the fly.

It would be interesting to add similar tests for the Big-Endian pseries
image, may be not all tests, but at least TCG.

Also, booting the kernel on a powernv9 machine would be nice. FreeBSD
supports OPAL on POWER9 and it is not that common. It would exercise
the PowerNV models differently from Linux.

I tried the apple image on a mac99,G5 machine but the kernel panics on
a DSI.


> ---
>   tests/avocado/boot_freebsd.py | 106 ++++++++++++++++++++++++++++++++++
>   1 file changed, 106 insertions(+)
>   create mode 100644 tests/avocado/boot_freebsd.py
> 
> diff --git a/tests/avocado/boot_freebsd.py b/tests/avocado/boot_freebsd.py
> new file mode 100644
> index 0000000000..79c68b149a
> --- /dev/null
> +++ b/tests/avocado/boot_freebsd.py
> @@ -0,0 +1,106 @@
> +# Functional tests that boot FreeBSD in various configurations
> +#
> +# Copyright (c) 2023 IBM Corporation
> +#
> +# This work is licensed under the terms of the GNU GPL, version 2 or
> +# later. See the COPYING file in the top-level directory.
> +
> +import os
> +
> +from avocado import skipUnless
> +from avocado_qemu import QemuSystemTest
> +from avocado_qemu import wait_for_console_pattern
> +from avocado_qemu import exec_command
> +from avocado.utils import archive
> +from avocado.utils import process
> +from avocado.utils.path import find_command
> +
> +@skipUnless(os.getenv('AVOCADO_ALLOW_LARGE_STORAGE'), 'storage limited')
> +@skipUnless(os.getenv('AVOCADO_ALLOW_LONG_RUNTIME'), 'runtime limited')
> +class BootFreeBSDPPC64(QemuSystemTest):
> +    """
> +    :avocado: tags=arch:ppc64
> +    """
> +
> +    timeout = 360
> +
> +    def setUp(self):
> +        super().setUp()
> +
> +        # We need zstd for all the tests
> +        # See https://github.com/avocado-framework/avocado/issues/5609
> +        zstd = find_command('zstd', False)
> +        if zstd is False:
> +            self.cancel('Could not find "zstd", which is required to '
> +                        'decompress rootfs')
> +        drive_url = ('https://artifact.ci.freebsd.org/snapshot/15.0-CURRENT/8a735ffdf04936c6785ac4fa31486639262dd416/powerpc/powerpc64le/disk.qcow2.zst')

The problem is the sustainability of this snapshot. It seems FreeBSD keeps
one year of history.

Thanks,

C.



> +        drive_hash = '95d863dbbc4b60f4899d1ef21d6489fca05bf03d'
> +        drive_path_zstd = self.fetch_asset(drive_url, asset_hash=drive_hash)
> +        drive_path = os.path.join(self.workdir, 'disk.qcow2')
> +
> +        cmd = f"{zstd} -d {drive_path_zstd} -o {drive_path}"
> +        process.run(cmd)
> +
> +        self.drive_opt = f"file={drive_path},format=qcow2,if=virtio"
> +
> +    def run_pseries_test(self, force_HPT=False):
> +        if force_HPT:
> +            self.vm.add_args('-m', '4g')
> +        else:
> +            self.vm.add_args('-m', '1g')
> +        self.vm.add_args('-smp', '4')
> +        self.vm.add_args('-drive', self.drive_opt)
> +        self.vm.add_args('-net', 'nic,model=virtio')
> +        self.vm.set_console()
> +        self.vm.launch()
> +
> +        wait_for_console_pattern(self, 'Hit [Enter] to boot immediately, or any other key for command prompt.')
> +        if force_HPT:
> +            exec_command(self, 'x')
> +            wait_for_console_pattern(self, 'OK')
> +            exec_command(self, 'set radix_mmu=0')
> +            exec_command(self, 'boot')
> +            wait_for_console_pattern(self, 'cas: selected hash MMU', 'panic:')
> +        else:
> +            exec_command(self, '')
> +            wait_for_console_pattern(self, 'cas: selected radix MMU', 'panic:')
> +
> +        wait_for_console_pattern(self, 'FreeBSD 15.0-CURRENT', 'panic:')
> +        wait_for_console_pattern(self, 'FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs', 'panic:')
> +        wait_for_console_pattern(self, 'FreeBSD/powerpc (Amnesiac) (ttyu0)', 'panic:')
> +
> +    def test_pseries_tcg(self):
> +        """
> +        :avocado: tags=arch:ppc64
> +        :avocado: tags=machine:pseries
> +        :avocado: tags=accel:tcg
> +        """
> +        self.require_accelerator("tcg")
> +        self.run_pseries_test()
> +
> +    def test_pseries_hpt_tcg(self):
> +        """
> +        :avocado: tags=arch:ppc64
> +        :avocado: tags=machine:pseries
> +        :avocado: tags=accel:tcg
> +        """
> +        self.require_accelerator("tcg")
> +        self.run_pseries_test(force_HPT=True)
> +
> +    def test_pseries_kvm(self):
> +        """
> +        :avocado: tags=arch:ppc64
> +        :avocado: tags=machine:pseries
> +        :avocado: tags=accel:kvm
> +        """
> +        self.require_accelerator("kvm")
> +        self.run_pseries_test()
> +
> +    def test_pseries_hpt_kvm(self):
> +        """
> +        :avocado: tags=arch:ppc64
> +        :avocado: tags=machine:pseries
> +        :avocado: tags=accel:kvm
> +        """
> +        self.require_accelerator("kvm")
> +        self.run_pseries_test(force_HPT=True)

Re: [PATCH 8/9] tests/avocado: ppc add hypervisor tests

[Cédric Le Goater]

On 1/7/24 18:01, Nicholas Piggin wrote:
> The powernv and pseries machines both provide hypervisor facilities
> that are supported by KVM. This is a large and complicated set of
> features that don't get much system-level testing in ppc tests.
> 
> Add a new test case for these which runs QEMU KVM inside the target.
> This downloads an Alpine VM image, boots it and downloads and installs
> the qemu package, then boots a virtual machine under it, re-using the
> original Alpine VM image.
> 
> Signed-off-by: Nicholas Piggin <npiggin@gmail.com>
> ---
>   MAINTAINERS                   |   1 +
>   tests/avocado/ppc_hv_tests.py | 201 ++++++++++++++++++++++++++++++++++
>   2 files changed, 202 insertions(+)
>   create mode 100644 tests/avocado/ppc_hv_tests.py
> 
> diff --git a/MAINTAINERS b/MAINTAINERS
> index 395f26ba86..ffe91a2bc5 100644
> --- a/MAINTAINERS
> +++ b/MAINTAINERS
> @@ -1509,6 +1509,7 @@ F: tests/qtest/libqos/*spapr*
>   F: tests/qtest/rtas*
>   F: tests/qtest/libqos/rtas*
>   F: tests/avocado/ppc_pseries.py
> +F: tests/avocado/ppc_hv_tests.py
>   
>   PowerNV (Non-Virtualized)
>   M: Cédric Le Goater <clg@kaod.org>
> diff --git a/tests/avocado/ppc_hv_tests.py b/tests/avocado/ppc_hv_tests.py
> new file mode 100644
> index 0000000000..2162d6bd68
> --- /dev/null
> +++ b/tests/avocado/ppc_hv_tests.py
> @@ -0,0 +1,201 @@
> +# Tests that specifically try to exercise hypervisor features of the
> +# target machines. powernv supports the Power hypervisor ISA, and
> +# pseries supports the nested-HV hypervisor spec.
> +#
> +# Copyright (c) 2023 IBM Corporation
> +#
> +# This work is licensed under the terms of the GNU GPL, version 2 or
> +# later.  See the COPYING file in the top-level directory.
> +
> +from avocado import skipIf, skipUnless
> +from avocado.utils import archive
> +from avocado_qemu import QemuSystemTest
> +from avocado_qemu import wait_for_console_pattern, exec_command
> +import os
> +import time
> +import subprocess
> +
> +deps = ["xorriso"] # dependent tools needed in the test setup/box.
> +
> +def which(tool):
> +    """ looks up the full path for @tool, returns None if not found
> +        or if @tool does not have executable permissions.
> +    """
> +    paths=os.getenv('PATH')
> +    for p in paths.split(os.path.pathsep):
> +        p = os.path.join(p, tool)
> +        if os.path.exists(p) and os.access(p, os.X_OK):
> +            return p
> +    return None
> +
> +def missing_deps():
> +    """ returns True if any of the test dependent tools are absent.
> +    """
> +    for dep in deps:
> +        if which(dep) is None:
> +            return True
> +    return False
> +
> +# Alpine is a light weight distro that supports QEMU. These tests boot
> +# that on the machine then run a QEMU guest inside it in KVM mode,
> +# that runs the same Alpine distro image.
> +# QEMU packages are downloaded and installed on each test. That's not a
> +# large download, but it may be more polite to create qcow2 image with
> +# QEMU already installed and use that.
> +@skipUnless(os.getenv('AVOCADO_ALLOW_LARGE_STORAGE'), 'storage limited')
> +@skipUnless(os.getenv('AVOCADO_ALLOW_LONG_RUNTIME'), 'runtime limited')
> +@skipIf(missing_deps(), 'dependencies (%s) not installed' % ','.join(deps))
> +class HypervisorTest(QemuSystemTest):
> +
> +    timeout = 1000
> +    KERNEL_COMMON_COMMAND_LINE = 'printk.time=0 console=hvc0 '
> +    panic_message = 'Kernel panic - not syncing'
> +    good_message = 'VFS: Cannot open root device'
> +
> +    def extract_from_iso(self, iso, path):
> +        """
> +        Extracts a file from an iso file into the test workdir
> +
> +        :param iso: path to the iso file
> +        :param path: path within the iso file of the file to be extracted
> +        :returns: path of the extracted file
> +        """
> +        filename = os.path.basename(path)
> +
> +        cwd = os.getcwd()
> +        os.chdir(self.workdir)
> +
> +        with open(filename, "w") as outfile:
> +            cmd = "xorriso -osirrox on -indev %s -cpx %s %s" %
> +                  (iso, path, filename)
$ build/pyvenv/bin/avocado -V list  build/tests/avocado/ppc_hv_tests.py
Type Test Tag(s)

Resolver             Reference                           Info
avocado-instrumented build/tests/avocado/ppc_hv_tests.py invalid syntax (ppc_hv_tests.py, line 77)

With this change :

@@ -66,8 +74,7 @@ class HypervisorTest(QemuSystemTest):
          os.chdir(self.workdir)
  
          with open(filename, "w") as outfile:
-            cmd = "xorriso -osirrox on -indev %s -cpx %s %s" %
-                  (iso, path, filename)
+            cmd = "xorriso -osirrox on -indev %s -cpx %s %s" % (iso, path, filename)
              subprocess.run(cmd.split(),
                             stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
  
the tests run :

   (1/3) build/tests/avocado/ppc_hv_tests.py:HypervisorTest.test_hv_pseries: PASS (330.62 s)
   (2/3) build/tests/avocado/ppc_hv_tests.py:HypervisorTest.test_hv_pseries_kvm: PASS (327.66 s)
   (3/3) build/tests/avocado/ppc_hv_tests.py:HypervisorTest.test_hv_powernv: /

test 2/3 is not using KVM, TCG is forced in do_start_alpine(), and
the powernv  test fails when starting the HPT VM. See below. Anyhow,
this looks promising. Thanks for these tests.

If you need small images for the L2, you could use :

https://github.com/legoater/qemu-ppc-boot/tree/main/buildroot/qemu_ppc64le_pseries-2023.11-8-gdcd9f0f6eb-20240104

C.


2024-01-08 15:22:06,328 __init__         L0153 DEBUG| (detected by 0, t=2103 jiffies, g=7877, q=12 ncpus=2)
2024-01-08 15:22:06,330 __init__         L0153 DEBUG| Sending NMI from CPU 0 to CPUs 1:
2024-01-08 15:22:06,614 __init__         L0153 DEBUG| NMI backtrace for cpu 1
2024-01-08 15:22:06,618 __init__         L0153 DEBUG| CPU: 1 PID: 2431 Comm: init-early.sh Not tainted 6.1.55-0-lts #1-Alpine
2024-01-08 15:22:06,622 __init__         L0153 DEBUG| Hardware name: IBM pSeries (emulated by qemu) POWER9 (raw) 0x4e1202 0xf000005 hv:linux,kvm pSeries
2024-01-08 15:22:06,625 __init__         L0153 DEBUG| NIP:  c0000000004a6510 LR: c0000000004adb10 CTR: 0000000000000023
2024-01-08 15:22:06,629 __init__         L0153 DEBUG| REGS: c00000000834b440 TRAP: 0500   Not tainted  (6.1.55-0-lts)
2024-01-08 15:22:06,633 __init__         L0153 DEBUG| MSR:  8000000000009033 <SF,EE,ME,IR,DR,RI,LE>  CR: 44844800  XER: 00000000
2024-01-08 15:22:06,635 __init__         L0153 DEBUG| CFAR: 0000000000000000 IRQMASK: 0
2024-01-08 15:22:06,638 __init__         L0153 DEBUG| GPR00: c0000000004adb10 c00000000834b6e0 c000000001502300 c00c0000000fe900
2024-01-08 15:22:06,641 __init__         L0153 DEBUG| GPR04: 0000000000000001 0000000000000400 0000000000000200 0000000000000100
2024-01-08 15:22:06,644 __init__         L0153 DEBUG| GPR08: 0000000000000040 c00000003fa47400 0000000000000080 c0000000025935b0
2024-01-08 15:22:06,647 __init__         L0153 DEBUG| GPR12: 0000000000000180 c00000003ffff300 0000000040000000 c00000003ffb3800
2024-01-08 15:22:06,650 __init__         L0153 DEBUG| GPR16: 0000000000000000 0000000000000000 0000000000000000 c00c0000000fe900
2024-01-08 15:22:06,654 __init__         L0153 DEBUG| GPR20: c00000000834b910 0000000000000000 0000000000000881 c00000003fd92fa8
2024-01-08 15:22:06,656 __init__         L0153 DEBUG| GPR24: 0000000000000000 c00c0000000fe908 0000000000000001 c00000003fd92f80
2024-01-08 15:22:06,659 __init__         L0153 DEBUG| GPR28: c0000000028391d0 0000000000000380 0000000000000300 0000000000000280
2024-01-08 15:22:06,663 __init__         L0153 DEBUG| NIP [c0000000004a6510] kernel_init_pages+0xa0/0x120
2024-01-08 15:22:06,665 __init__         L0153 DEBUG| LR [c0000000004adb10] get_page_from_freelist+0xa00/0x1640
2024-01-08 15:22:06,666 __init__         L0153 DEBUG| Call Trace:
2024-01-08 15:22:06,670 __init__         L0153 DEBUG| [c00000000834b6e0] [c0000000004ad438] get_page_from_freelist+0x328/0x1640 (unreliable)
2024-01-08 15:22:06,674 __init__         L0153 DEBUG| [c00000000834b820] [c0000000004af094] __alloc_pages+0x174/0x12f0
2024-01-08 15:22:06,676 __init__         L0153 DEBUG| [c00000000834ba10] [c0000000004b0bd4] __folio_alloc+0x34/0xb0
2024-01-08 15:22:06,679 __init__         L0153 DEBUG| [c00000000834ba40] [c0000000004e5ef8] vma_alloc_folio+0xd8/0x530
2024-01-08 15:22:06,682 __init__         L0153 DEBUG| [c00000000834baf0] [c000000000470ff8] wp_page_copy+0xc8/0xf10
2024-01-08 15:22:06,685 __init__         L0153 DEBUG| [c00000000834bbd0] [c000000000479188] __handle_mm_fault+0x9d8/0x1940
2024-01-08 15:22:06,688 __init__         L0153 DEBUG| [c00000000834bcc0] [c00000000047a240] handle_mm_fault+0x150/0x3d0
2024-01-08 15:22:06,691 __init__         L0153 DEBUG| [c00000000834bd10] [c00000000008184c] ___do_page_fault+0x32c/0x920
2024-01-08 15:22:06,695 __init__         L0153 DEBUG| [c00000000834bdb0] [c000000000081f10] hash__do_page_fault+0x30/0x100
2024-01-08 15:22:06,698 __init__         L0153 DEBUG| [c00000000834bde0] [c00000000008bd44] do_hash_fault+0x1f4/0x390
2024-01-08 15:22:06,701 __init__         L0153 DEBUG| [c00000000834be10] [c000000000008918] data_access_common_virt+0x198/0x1f0
2024-01-08 15:22:06,703 __init__         L0153 DEBUG| --- interrupt: 300 at 0x7fff9da32a60
2024-01-08 15:22:06,706 __init__         L0153 DEBUG| NIP:  00007fff9da32a60 LR: 00007fff9da37400 CTR: 0000000000000000
2024-01-08 15:22:06,709 __init__         L0153 DEBUG| REGS: c00000000834be80 TRAP: 0300   Not tainted  (6.1.55-0-lts)
2024-01-08 15:22:06,713 __init__         L0153 DEBUG| MSR:  800000000000d033 <SF,EE,PR,ME,IR,DR,RI,LE>  CR: 20000000  XER: 00000000
2024-01-08 15:22:06,716 __init__         L0153 DEBUG| CFAR: c000000000002230 DAR: 00007fff9da7f988 DSISR: 0a000000 IRQMASK: 0
2024-01-08 15:22:06,719 __init__         L0153 DEBUG| GPR00: 00007fff9da37400 00007fffef7f61f0 00007fff9da87e00 00007fff9d9c0000
2024-01-08 15:22:06,723 __init__         L0153 DEBUG| GPR04: 00007fff9da7fbf0 00007fff9da45e84 00007fff9d9d3fb0 00007fffef7f6460
2024-01-08 15:22:06,726 __init__         L0153 DEBUG| GPR08: 0000000000000030 00000000000bf988 00007fff9da7f990 0000000000000000
2024-01-08 15:22:06,728 __init__         L0153 DEBUG| GPR12: 00007fff9da373d8 0000000000000000 0000000000000000 0000000000000000
2024-01-08 15:22:06,731 __init__         L0153 DEBUG| GPR16: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
2024-01-08 15:22:06,735 __init__         L0153 DEBUG| GPR20: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
2024-01-08 15:22:06,738 __init__         L0153 DEBUG| GPR24: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
2024-01-08 15:22:06,741 __init__         L0153 DEBUG| GPR28: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
2024-01-08 15:22:06,743 __init__         L0153 DEBUG| NIP [00007fff9da32a60] 0x7fff9da32a60
2024-01-08 15:22:06,745 __init__         L0153 DEBUG| LR [00007fff9da37400] 0x7fff9da37400
2024-01-08 15:22:06,746 __init__         L0153 DEBUG| --- interrupt: 300
2024-01-08 15:22:06,748 __init__         L0153 DEBUG| Instruction dump:
2024-01-08 15:22:06,751 __init__         L0153 DEBUG| 7d291850 79470fa4 792953e4 79461764 78a9f00e 79451f24 7d8a3a14 7fc73214
2024-01-08 15:22:06,754 __init__         L0153 DEBUG| 7fea3214 7faa2850 7908e8e2 7d0903a6 <7c004fec> 7c0a4fec 7c074fec 7c0c4fec

Re: [PATCH 1/9] gitlab: fix s390x tag for avocado-system-centos

[Philippe Mathieu-Daudé]

On 7/1/24 18:01, Nicholas Piggin wrote:
> The 390x tag should be s390x.
> 
> Signed-off-by: Nicholas Piggin <npiggin@gmail.com>
> ---
>   .gitlab-ci.d/buildtest.yml | 2 +-
>   1 file changed, 1 insertion(+), 1 deletion(-)

FTR this has been merged as commit cdd30f369a.