[PATCH v2 0/7] testing/next: libvirt-ci, tests/vm and vbsa functional test

[PATCH v2 0/7] testing/next: libvirt-ci, tests/vm and vbsa functional test

[Alex Bennée]

While I was looking at Jon's python patches I noticed some failures
with getting OpenBSD up and running. In pulling that thread I found a
few other bits to fix and up revved the image and converted it to use
lci-tool data like FreeBSD does.

I've also added the VBSA conformance test for Arm virt machines to the
functional tests. This just adds the UEFI based tests as the full
suite needs kernel modules and a userspace app which I think I have to
build.

v2
  - added the docker updates
  - added tags
  - fixed up binary and error checking on VBSA test

The following still need review:

  tests/functional: add Arm VBSA uefi conformance test
  tests/docker: migrate legacy-test-cross compilers to trixie

Alex.

Alex Bennée (6):
  tests/docker: migrate legacy-test-cross compilers to trixie
  tests/vm: bump OpenBSD to the current 7.8 release
  tests/vm: remove unused import
  tests/vm: fix interactive boot
  tests/vm: build openbsd from lcitool data (!567)
  tests/functional: add Arm VBSA uefi conformance test

John Snow (1):
  tests/docker: upgrade most non-lcitool debian tests to debian 13

 configure                                     | 10 +-
 .gitlab-ci.d/buildtest.yml                    | 12 ---
 .gitlab-ci.d/container-cross.yml              |  6 --
 .gitlab-ci.d/containers.yml                   |  1 -
 .../dockerfiles/debian-all-test-cross.docker  |  8 +-
 .../dockerfiles/debian-hexagon-cross.docker   | 11 ++-
 .../debian-legacy-test-cross.docker           | 51 ----------
 .../dockerfiles/debian-loongarch-cross.docker | 10 +-
 .../dockerfiles/debian-toolchain.docker       |  8 +-
 .../dockerfiles/debian-tricore-cross.docker   |  5 +-
 .../dockerfiles/debian-xtensa-cross.docker    |  2 +-
 tests/functional/aarch64/meson.build          |  1 +
 tests/functional/aarch64/test_virt_vbsa.py    | 99 +++++++++++++++++++
 tests/lcitool/refresh                         | 13 ++-
 tests/vm/Makefile.include                     |  2 +-
 tests/vm/generated/openbsd.json               | 27 +++++
 tests/vm/openbsd                              | 47 +--------
 17 files changed, 167 insertions(+), 146 deletions(-)
 delete mode 100644 tests/docker/dockerfiles/debian-legacy-test-cross.docker
 create mode 100755 tests/functional/aarch64/test_virt_vbsa.py
 create mode 100644 tests/vm/generated/openbsd.json

-- 
2.47.3

[PATCH v2 1/7] tests/docker: upgrade most non-lcitool debian tests to debian 13

[Alex Bennée]

From: John Snow <jsnow@redhat.com>

Debian 11 was EOL in 2024, and Debian 12 will be EOL this June. This
patch moves all but one of our tests, debian-legacy-test-cross, onto
Debian 13.

This patch does the bare minimum to upgrade these tests and doesn't make
any attempt at optimization or cleanup that may or may not be possible
with this upgrade.

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
[AJB: tweak summary line]
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
---
 tests/docker/dockerfiles/debian-hexagon-cross.docker  | 11 ++++++-----
 .../docker/dockerfiles/debian-loongarch-cross.docker  | 10 +++++-----
 tests/docker/dockerfiles/debian-toolchain.docker      |  8 +++++---
 tests/docker/dockerfiles/debian-tricore-cross.docker  |  5 ++---
 tests/docker/dockerfiles/debian-xtensa-cross.docker   |  2 +-
 5 files changed, 19 insertions(+), 17 deletions(-)

diff --git a/tests/docker/dockerfiles/debian-hexagon-cross.docker b/tests/docker/dockerfiles/debian-hexagon-cross.docker
index 23152b4918b..91d4b71ac95 100644
--- a/tests/docker/dockerfiles/debian-hexagon-cross.docker
+++ b/tests/docker/dockerfiles/debian-hexagon-cross.docker
@@ -5,10 +5,12 @@
 # needs to be able to build QEMU itself in CI we include its
 # build-deps.
 #
-FROM docker.io/library/debian:11-slim
+FROM docker.io/library/debian:13-slim
+
+# Add deb-src repository sources
+RUN sed -i "s/^Types: deb$/Types: deb deb-src/" \
+    /etc/apt/sources.list.d/debian.sources
 
-# Duplicate deb line as deb-src
-RUN cat /etc/apt/sources.list | sed "s/^deb\ /deb-src /" >> /etc/apt/sources.list
 RUN apt-get update && \
     DEBIAN_FRONTEND=noninteractive apt install -yy eatmydata && \
     DEBIAN_FRONTEND=noninteractive eatmydata \
@@ -24,6 +26,7 @@ RUN apt-get update && \
         ninja-build \
         python3-pip \
         python3-setuptools \
+        python3-tomli \
         python3-venv \
         python3-wheel && \
 # Install QEMU build deps for use in CI
@@ -36,8 +39,6 @@ RUN apt-get update && \
     ln -s /usr/bin/ccache /usr/libexec/ccache-wrappers/gcc && \
     dpkg-query --showformat '${Package}_${Version}_${Architecture}\n' --show > /packages.txt
 
-RUN /usr/bin/pip3 install tomli
-
 ENV TOOLCHAIN_INSTALL /opt
 ENV TOOLCHAIN_RELEASE 12.Dec.2023
 ENV TOOLCHAIN_BASENAME "clang+llvm-${TOOLCHAIN_RELEASE}-cross-hexagon-unknown-linux-musl"
diff --git a/tests/docker/dockerfiles/debian-loongarch-cross.docker b/tests/docker/dockerfiles/debian-loongarch-cross.docker
index 538ab534902..55b3dbe4510 100644
--- a/tests/docker/dockerfiles/debian-loongarch-cross.docker
+++ b/tests/docker/dockerfiles/debian-loongarch-cross.docker
@@ -4,10 +4,11 @@
 # This docker target uses prebuilt toolchains for LoongArch64 from:
 # https://github.com/loongson/build-tools/releases
 #
-FROM docker.io/library/debian:11-slim
+FROM docker.io/library/debian:13-slim
 
-# Duplicate deb line as deb-src
-RUN cat /etc/apt/sources.list | sed "s/^deb\ /deb-src /" >> /etc/apt/sources.list
+# Add deb-src repository sources
+RUN sed -i "s/^Types: deb$/Types: deb deb-src/" \
+    /etc/apt/sources.list.d/debian.sources
 
 RUN export DEBIAN_FRONTEND=noninteractive && \
     apt-get update && \
@@ -31,12 +32,11 @@ RUN apt-get update && \
         ninja-build \
         python3-pip \
         python3-setuptools \
+        python3-tomli \
         python3-venv \
         python3-wheel && \
         dpkg-query --showformat '${Package}_${Version}_${Architecture}\n' --show > /packages.txt
 
-RUN /usr/bin/pip3 install tomli
-
 RUN curl -#SL https://github.com/loongson/build-tools/releases/download/2023.08.08/CLFS-loongarch64-8.1-x86_64-cross-tools-gcc-glibc.tar.xz \
     | tar -xJC /opt
 
diff --git a/tests/docker/dockerfiles/debian-toolchain.docker b/tests/docker/dockerfiles/debian-toolchain.docker
index ab4ce29533d..9a256209a78 100644
--- a/tests/docker/dockerfiles/debian-toolchain.docker
+++ b/tests/docker/dockerfiles/debian-toolchain.docker
@@ -4,13 +4,15 @@
 # This dockerfile is used for building a cross-compiler toolchain.
 # The script for building the toolchain is supplied via extra-files.
 #
-FROM docker.io/library/debian:11-slim
+FROM docker.io/library/debian:13-slim
 
 # Install build utilities for building gcc and glibc.
 # ??? The build-dep isn't working, missing a number of
 # minimal build dependiencies, e.g. libmpc.
 
-RUN sed 's/^deb /deb-src /' </etc/apt/sources.list >/etc/apt/sources.list.d/deb-src.list
+# Add deb-src repository sources
+RUN sed -i "s/^Types: deb$/Types: deb deb-src/" \
+    /etc/apt/sources.list.d/debian.sources
 
 RUN apt update && \
     DEBIAN_FRONTEND=noninteractive apt install -yy eatmydata && \
@@ -34,7 +36,7 @@ RUN cd /root && ./build-toolchain.sh
 # Throw away the extra toolchain build deps, the downloaded source,
 # and the build trees by restoring the original image,
 # then copying the built toolchain from stage 0.
-FROM docker.io/library/debian:11-slim
+FROM docker.io/library/debian:13-slim
 RUN apt update && \
     DEBIAN_FRONTEND=noninteractive apt install -yy eatmydata && \
     DEBIAN_FRONTEND=noninteractive eatmydata \
diff --git a/tests/docker/dockerfiles/debian-tricore-cross.docker b/tests/docker/dockerfiles/debian-tricore-cross.docker
index 7e00e870ceb..fd797dc7ee4 100644
--- a/tests/docker/dockerfiles/debian-tricore-cross.docker
+++ b/tests/docker/dockerfiles/debian-tricore-cross.docker
@@ -9,7 +9,7 @@
 #
 # SPDX-License-Identifier: GPL-2.0-or-later
 #
-FROM docker.io/library/debian:11-slim
+FROM docker.io/library/debian:13-slim
 
 RUN apt update && \
     DEBIAN_FRONTEND=noninteractive apt install -yy eatmydata && \
@@ -31,12 +31,11 @@ RUN apt update && \
        pkgconf \
        python3-pip \
        python3-setuptools \
+       python3-tomli \
        python3-wheel \
        python3-venv && \
        dpkg-query --showformat '${Package}_${Version}_${Architecture}\n' --show > /packages.txt
 
-RUN /usr/bin/pip3 install tomli
-
 RUN curl -#SL https://github.com/bkoppelmann/package_940/releases/download/tricore-toolchain-9.40/tricore-toolchain-9.4.0.tar.gz \
     | tar -xzC /usr/local/
 
diff --git a/tests/docker/dockerfiles/debian-xtensa-cross.docker b/tests/docker/dockerfiles/debian-xtensa-cross.docker
index d011eee2ad3..ef63e44e2ef 100644
--- a/tests/docker/dockerfiles/debian-xtensa-cross.docker
+++ b/tests/docker/dockerfiles/debian-xtensa-cross.docker
@@ -5,7 +5,7 @@
 # using a prebuilt toolchains for Xtensa cores from:
 # https://github.com/foss-xtensa/toolchain/releases
 #
-FROM docker.io/library/debian:11-slim
+FROM docker.io/library/debian:13-slim
 
 RUN apt-get update && \
     DEBIAN_FRONTEND=noninteractive apt install -yy eatmydata && \
-- 
2.47.3

[PATCH v2 2/7] tests/docker: migrate legacy-test-cross compilers to trixie

[Alex Bennée]

The bugs have evidently been fixed in the latest release so we can
migrate the laggards into how all-test-cross container and remove the
legacy hacks. They are also packaged for the main architectures so we
don't need to jump through the amd64 hoops.

Suggested-by: John Snow <jsnow@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
---
 configure                                     | 10 +---
 .gitlab-ci.d/buildtest.yml                    | 12 -----
 .gitlab-ci.d/container-cross.yml              |  6 ---
 .gitlab-ci.d/containers.yml                   |  1 -
 .../dockerfiles/debian-all-test-cross.docker  |  8 ++-
 .../debian-legacy-test-cross.docker           | 51 -------------------
 tests/lcitool/refresh                         |  8 ++-
 7 files changed, 13 insertions(+), 83 deletions(-)
 delete mode 100644 tests/docker/dockerfiles/debian-legacy-test-cross.docker

diff --git a/configure b/configure
index c4837eba2b1..683813473e9 100755
--- a/configure
+++ b/configure
@@ -1450,7 +1450,7 @@ probe_target_compiler() {
         container_image=debian-all-test-cross
         container_cross_prefix=aarch64-linux-gnu-
         ;;
-      hppa|m68k|mips|riscv64|sparc64)
+      alpha|hppa|m68k|mips|riscv64|sh4|sparc64)
         container_image=debian-all-test-cross
         ;;
       mips64)
@@ -1462,14 +1462,6 @@ probe_target_compiler() {
         container_cross_prefix=powerpc${target_arch#ppc}-linux-gnu-
         ;;
 
-      # debian-legacy-test-cross architectures (need Debian 11)
-      # - libc6.1-dev-alpha-cross: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1054412
-      # - sh4-linux-user: binaries don't run with bookworm compiler
-
-      alpha|sh4)
-        container_image=debian-legacy-test-cross
-        ;;
-
       # architectures with individual containers
 
       aarch64)
diff --git a/.gitlab-ci.d/buildtest.yml b/.gitlab-ci.d/buildtest.yml
index 1b656b9eb08..fef19c2d5da 100644
--- a/.gitlab-ci.d/buildtest.yml
+++ b/.gitlab-ci.d/buildtest.yml
@@ -376,18 +376,6 @@ build-user-static:
   variables:
     IMAGE: debian-all-test-cross
     CONFIGURE_ARGS: --disable-tools --disable-system --static
-      --target-list-exclude=alpha-linux-user,sh4-linux-user
-    MAKE_CHECK_ARGS: check-tcg
-
-# targets stuck on older compilers
-build-legacy:
-  extends: .native_build_job_template
-  needs:
-    - job: amd64-debian-legacy-cross-container
-  variables:
-    IMAGE: debian-legacy-test-cross
-    TARGETS: alpha-linux-user alpha-softmmu sh4-linux-user
-    CONFIGURE_ARGS: --disable-tools
     MAKE_CHECK_ARGS: check-tcg
 
 build-user-hexagon:
diff --git a/.gitlab-ci.d/container-cross.yml b/.gitlab-ci.d/container-cross.yml
index 2a0cea6ce1a..92a7f1c1b47 100644
--- a/.gitlab-ci.d/container-cross.yml
+++ b/.gitlab-ci.d/container-cross.yml
@@ -10,12 +10,6 @@ amd64-debian-user-cross-container:
   variables:
     NAME: debian-all-test-cross
 
-amd64-debian-legacy-cross-container:
-  extends: .container_job_template
-  stage: containers
-  variables:
-    NAME: debian-legacy-test-cross
-
 arm64-debian-cross-container:
   extends: .container_job_template
   stage: containers
diff --git a/.gitlab-ci.d/containers.yml b/.gitlab-ci.d/containers.yml
index 6aeccf8be0c..c8b8e44ad86 100644
--- a/.gitlab-ci.d/containers.yml
+++ b/.gitlab-ci.d/containers.yml
@@ -45,7 +45,6 @@ weekly-container-builds:
     # cross
     - amd64-debian-cross-container
     - amd64-debian-user-cross-container
-    - amd64-debian-legacy-cross-container
     - arm64-debian-cross-container
     - hexagon-cross-container
     - loongarch-debian-cross-container
diff --git a/tests/docker/dockerfiles/debian-all-test-cross.docker b/tests/docker/dockerfiles/debian-all-test-cross.docker
index 1823233438a..b73776d95b5 100644
--- a/tests/docker/dockerfiles/debian-all-test-cross.docker
+++ b/tests/docker/dockerfiles/debian-all-test-cross.docker
@@ -64,6 +64,8 @@ ENV AVAILABLE_COMPILERS gcc-aarch64-linux-gnu \
   libc6-dev-arm64-cross \
   gcc-arm-linux-gnueabihf \
   libc6-dev-armhf-cross \
+  gcc-alpha-linux-gnu \
+  libc6.1-dev-alpha-cross \
   gcc-mips-linux-gnu \
   libc6-dev-mips-cross \
   gcc-mips64-linux-gnuabi64 \
@@ -77,7 +79,9 @@ ENV AVAILABLE_COMPILERS gcc-aarch64-linux-gnu \
   gcc-riscv64-linux-gnu \
   libc6-dev-riscv64-cross \
   gcc-s390x-linux-gnu \
-  libc6-dev-s390x-cross
+  libc6-dev-s390x-cross\
+  gcc-sh4-linux-gnu \
+  libc6-dev-sh4-cross
 RUN if dpkg-architecture -e amd64; then \
   export AVAILABLE_COMPILERS="${AVAILABLE_COMPILERS} gcc-hppa-linux-gnu libc6-dev-hppa-cross"; \
   export AVAILABLE_COMPILERS="${AVAILABLE_COMPILERS} gcc-m68k-linux-gnu libc6-dev-m68k-cross"; \
@@ -90,7 +94,7 @@ apt install -y --no-install-recommends \
 ${AVAILABLE_COMPILERS} && \
 dpkg-query --showformat '${Package}_${Version}_${Architecture}' --show > /packages.txt
 ENV QEMU_CONFIGURE_OPTS --disable-docs
-ENV DEF_TARGET_LIST aarch64-linux-user,arm-linux-user,hppa-linux-user,i386-linux-user,m68k-linux-user,mips-linux-user,mips64-linux-user,mips64el-linux-user,mipsel-linux-user,ppc-linux-user,ppc64-linux-user,ppc64le-linux-user,riscv64-linux-user,s390x-linux-user,sparc64-linux-user
+ENV DEF_TARGET_LIST aarch64-linux-user,alpha-linux-user,arm-linux-user,hppa-linux-user,i386-linux-user,m68k-linux-user,mips-linux-user,mips64-linux-user,mips64el-linux-user,mipsel-linux-user,ppc-linux-user,ppc64-linux-user,ppc64le-linux-user,riscv64-linux-user,s390x-linux-user,sh4-linux-user,sparc64-linux-user
 # As a final step configure the user (if env is defined)
 ARG USER
 ARG UID
diff --git a/tests/docker/dockerfiles/debian-legacy-test-cross.docker b/tests/docker/dockerfiles/debian-legacy-test-cross.docker
deleted file mode 100644
index 5a6616b7d39..00000000000
--- a/tests/docker/dockerfiles/debian-legacy-test-cross.docker
+++ /dev/null
@@ -1,51 +0,0 @@
-# Docker legacy cross-compiler target (tests and minimal qemu)
-#
-# Compilers for some of our older targets which we cant currently
-# upgrade. Currently:
-#
-#   libc6.1-dev-alpha-cross: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1054412
-#   sh4-linux-user: binaries don't run with bookworm compiler
-#
-# As we are targeting check-tcg here we only need minimal qemu
-# dependencies and the relevant cross compilers.
-
-FROM docker.io/library/debian:11-slim
-
-# Duplicate deb line as deb-src
-RUN cat /etc/apt/sources.list | sed "s/^deb\ /deb-src /" >> /etc/apt/sources.list
-
-RUN export DEBIAN_FRONTEND=noninteractive && \
-    apt-get update && \
-    apt-get install -y eatmydata && \
-    eatmydata apt-get dist-upgrade -y && \
-    apt build-dep -yy qemu
-
-# Add extra build tools and as many cross compilers as we can for testing
-RUN DEBIAN_FRONTEND=noninteractive eatmydata \
-        apt install -y --no-install-recommends \
-        bison \
-        ccache \
-        clang  \
-        flex \
-        git \
-        ninja-build \
-        gcc-alpha-linux-gnu \
-        libc6.1-dev-alpha-cross \
-        gcc-sh4-linux-gnu \
-        libc6-dev-sh4-cross \
-        python3-pip \
-        python3-setuptools \
-        python3-venv \
-        python3-wheel && \
-        dpkg-query --showformat '${Package}_${Version}_${Architecture}\n' --show > /packages.txt
-
-RUN /usr/bin/pip3 install tomli
-
-ENV QEMU_CONFIGURE_OPTS --disable-system --disable-docs --disable-tools
-ENV DEF_TARGET_LIST alpha-linux-user,sh4-linux-user
-ENV MAKE /usr/bin/make
-# As a final step configure the user (if env is defined)
-ARG USER
-ARG UID
-RUN if [ "${USER}" ]; then \
-  id ${USER} 2>/dev/null || useradd -u ${UID} -U ${USER}; fi
diff --git a/tests/lcitool/refresh b/tests/lcitool/refresh
index 8f0aa3fa343..c74345e03d0 100755
--- a/tests/lcitool/refresh
+++ b/tests/lcitool/refresh
@@ -172,6 +172,8 @@ debian_all_test_cross_compilers = [
     "  libc6-dev-arm64-cross \\\n",
     "  gcc-arm-linux-gnueabihf \\\n",
     "  libc6-dev-armhf-cross \\\n",
+    "  gcc-alpha-linux-gnu \\\n"
+    "  libc6.1-dev-alpha-cross \\\n"
     "  gcc-mips-linux-gnu \\\n",
     "  libc6-dev-mips-cross \\\n",
     "  gcc-mips64-linux-gnuabi64 \\\n",
@@ -185,7 +187,9 @@ debian_all_test_cross_compilers = [
     "  gcc-riscv64-linux-gnu \\\n",
     "  libc6-dev-riscv64-cross \\\n",
     "  gcc-s390x-linux-gnu \\\n",
-    "  libc6-dev-s390x-cross\n",
+    "  libc6-dev-s390x-cross\\\n",
+    "  gcc-sh4-linux-gnu \\\n",
+    "  libc6-dev-sh4-cross\n",
     "RUN if dpkg-architecture -e amd64; then \\\n",
     "  export AVAILABLE_COMPILERS=\"${AVAILABLE_COMPILERS} gcc-hppa-linux-gnu libc6-dev-hppa-cross\"; \\\n",
     "  export AVAILABLE_COMPILERS=\"${AVAILABLE_COMPILERS} gcc-m68k-linux-gnu libc6-dev-m68k-cross\"; \\\n",
@@ -198,7 +202,7 @@ debian_all_test_cross_compilers = [
     "${AVAILABLE_COMPILERS} && \\\n",
     "dpkg-query --showformat '${Package}_${Version}_${Architecture}' --show > /packages.txt\n",
     "ENV QEMU_CONFIGURE_OPTS --disable-docs\n",
-    "ENV DEF_TARGET_LIST aarch64-linux-user,arm-linux-user,hppa-linux-user,i386-linux-user,m68k-linux-user,mips-linux-user,mips64-linux-user,mips64el-linux-user,mipsel-linux-user,ppc-linux-user,ppc64-linux-user,ppc64le-linux-user,riscv64-linux-user,s390x-linux-user,sparc64-linux-user\n",
+    "ENV DEF_TARGET_LIST aarch64-linux-user,alpha-linux-user,arm-linux-user,hppa-linux-user,i386-linux-user,m68k-linux-user,mips-linux-user,mips64-linux-user,mips64el-linux-user,mipsel-linux-user,ppc-linux-user,ppc64-linux-user,ppc64le-linux-user,riscv64-linux-user,s390x-linux-user,sh4-linux-user,sparc64-linux-user\n",
 ]
 
 def cross_build(prefix, targets):
-- 
2.47.3

[PATCH v2 3/7] tests/vm: bump OpenBSD to the current 7.8 release

[Alex Bennée]

Reviewed-by: Pierrick Bouvier <pierrick.bouvier@linaro.org>
Message-ID: <20260213154859.1551283-3-alex.bennee@linaro.org>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
---
 tests/vm/openbsd | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/tests/vm/openbsd b/tests/vm/openbsd
index 5222041b855..d1289d78d24 100755
--- a/tests/vm/openbsd
+++ b/tests/vm/openbsd
@@ -22,8 +22,8 @@ class OpenBSDVM(basevm.BaseVM):
     name = "openbsd"
     arch = "x86_64"
 
-    link = "https://cdn.openbsd.org/pub/OpenBSD/7.7/amd64/install77.iso"
-    csum = "da0106e39463f015524dca806f407c37a9bdd17e6dfffe533b06a2dd2edd8a27"
+    link = "https://cdn.openbsd.org/pub/OpenBSD/7.8/amd64/install78.iso"
+    csum = "a228d0a1ef558b4d9ec84c698f0d3ffd13cd38c64149487cba0f1ad873be07b2"
     size = "20G"
     pkgs = [
         # tools
-- 
2.47.3

[PATCH v2 4/7] tests/vm: remove unused import

[Alex Bennée]

Reviewed-by: Pierrick Bouvier <pierrick.bouvier@linaro.org>
Message-ID: <20260213154859.1551283-4-alex.bennee@linaro.org>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
---
 tests/vm/openbsd | 1 -
 1 file changed, 1 deletion(-)

diff --git a/tests/vm/openbsd b/tests/vm/openbsd
index d1289d78d24..9aabfeec963 100755
--- a/tests/vm/openbsd
+++ b/tests/vm/openbsd
@@ -14,7 +14,6 @@
 
 import os
 import sys
-import socket
 import subprocess
 import basevm
 
-- 
2.47.3

[PATCH v2 5/7] tests/vm: fix interactive boot

[Alex Bennée]

For reasons still not clear to me passing the single dashed
-interactive would confuse the argument parsing enough we tried to
pass "nterative" as a string to the launch command causing failure and
head scratching.

Reviewed-by: Pierrick Bouvier <pierrick.bouvier@linaro.org>
Message-ID: <20260213154859.1551283-5-alex.bennee@linaro.org>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
---
 tests/vm/Makefile.include | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/tests/vm/Makefile.include b/tests/vm/Makefile.include
index 8a5aaedce7a..0a53dc9fdd6 100644
--- a/tests/vm/Makefile.include
+++ b/tests/vm/Makefile.include
@@ -154,6 +154,6 @@ vm-boot-ssh-%: $(IMAGES_DIR)/%.img $(VM_VENV)
 		$(if $(EFI_AARCH64),--efi-aarch64 $(EFI_AARCH64)) \
 		$(if $(LOG_CONSOLE),--log-console) \
 		--image "$<" \
-		$(if $(ROOT_USER),--interactive-root,-interactive) \
+		$(if $(ROOT_USER),--interactive-root,--interactive) \
 		false, \
 		"  VM-BOOT-SSH $*") || true
-- 
2.47.3

[PATCH v2 6/7] tests/vm: build openbsd from lcitool data (!567)

[Alex Bennée]

For now only use the minimal decadency set until all the OpenBSD
mappings can be divined.

Currently depends on: https://gitlab.com/libvirt/libvirt-ci/-/merge_requests/567

Reviewed-by: Pierrick Bouvier <pierrick.bouvier@linaro.org>
Message-ID: <20260213154859.1551283-6-alex.bennee@linaro.org>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
---
 tests/lcitool/refresh           |  5 ++--
 tests/vm/generated/openbsd.json | 27 +++++++++++++++++++++
 tests/vm/openbsd                | 42 ++-------------------------------
 3 files changed, 32 insertions(+), 42 deletions(-)
 create mode 100644 tests/vm/generated/openbsd.json

diff --git a/tests/lcitool/refresh b/tests/lcitool/refresh
index c74345e03d0..3e4b026035b 100755
--- a/tests/lcitool/refresh
+++ b/tests/lcitool/refresh
@@ -87,9 +87,9 @@ def generate_cirrus(target, trailer=None):
     generate(filename, cmd, trailer)
 
 
-def generate_pkglist(vm, target):
+def generate_pkglist(vm, target, project="qemu"):
     filename = Path(src_dir, "tests", "vm", "generated", vm + ".json")
-    cmd = lcitool_cmd + ["variables", "--format", "json", target, "qemu"]
+    cmd = lcitool_cmd + ["variables", "--format", "json", target, project]
     generate(filename, cmd, None)
 
 
@@ -316,6 +316,7 @@ try:
     # VM packages lists
     #
     generate_pkglist("freebsd", "freebsd-14")
+    generate_pkglist("openbsd", "openbsd-78", project="qemu-minimal")
 
     #
     # Ansible package lists
diff --git a/tests/vm/generated/openbsd.json b/tests/vm/generated/openbsd.json
new file mode 100644
index 00000000000..0886afd0d3d
--- /dev/null
+++ b/tests/vm/generated/openbsd.json
@@ -0,0 +1,27 @@
+{
+  "ccache": "/usr/local/bin/ccache",
+  "cpan_pkgs": [],
+  "cross_pkgs": [],
+  "make": "/usr/local/bin/gmake",
+  "ninja": "/usr/local/bin/ninja",
+  "packaging_command": "pkg_add",
+  "pip3": "/usr/local/bin/pip3",
+  "pkgs": [
+    "bash",
+    "bison",
+    "bzip2",
+    "ccache",
+    "dtc",
+    "git",
+    "glib2",
+    "gmake",
+    "gsed",
+    "libffi",
+    "meson",
+    "ninja",
+    "pkgconf",
+    "python3"
+  ],
+  "pypi_pkgs": [],
+  "python": "/usr/local/bin/python3"
+}
diff --git a/tests/vm/openbsd b/tests/vm/openbsd
index 9aabfeec963..4f8eeb28e24 100755
--- a/tests/vm/openbsd
+++ b/tests/vm/openbsd
@@ -24,45 +24,6 @@ class OpenBSDVM(basevm.BaseVM):
     link = "https://cdn.openbsd.org/pub/OpenBSD/7.8/amd64/install78.iso"
     csum = "a228d0a1ef558b4d9ec84c698f0d3ffd13cd38c64149487cba0f1ad873be07b2"
     size = "20G"
-    pkgs = [
-        # tools
-        "dtc",
-        "git",
-        "pkgconf",
-        "bzip2", "xz",
-        "ninja",
-
-        # gnu tools
-        "bash",
-        "gmake",
-        "gsed",
-        "gettext-tools",
-
-        # libs: usb
-        "libusb1--",
-
-        # libs: crypto
-        "gnutls",
-
-        # libs: images
-        "jpeg",
-        "png",
-
-        # libs: ui
-        "capstone",
-        "sdl2",
-        "gtk+3",
-        "libxkbcommon",
-
-        # libs: migration
-        "zstd",
-
-        # libs: networking
-        "libslirp",
-
-        # Python stuff
-        "py3-setuptools",
-    ]
 
     BUILD_SCRIPT = """
         set -e;
@@ -198,8 +159,9 @@ class OpenBSDVM(basevm.BaseVM):
         self.console_wait("login:")
         self.wait_ssh()
 
+        pkgs = self.get_qemu_packages_from_lcitool_json()
         self.print_step("Installing packages")
-        self.ssh_root_check("pkg_add %s\n" % " ".join(self.pkgs))
+        self.ssh_root_check("pkg_add %s\n" % " ".join(pkgs))
 
         # shutdown
         self.ssh_root(self.poweroff)
-- 
2.47.3

[PATCH v2 7/7] tests/functional: add Arm VBSA uefi conformance test

[Alex Bennée]

The VBSA test is a subset of the wider Arm architecture compliance
suites (ACS) which validate machines meet particular minimum set of
requirements. The VBSA is for virtual machines so it makes sense we
should check the -M virt machine is compliant.

Fortunately there are prebuilt binaries published via github so all we
need to do is build an EFI partition and place things in the right
place.

There are some additional Linux based tests which are left for later.

Message-ID: <20260213154859.1551283-7-alex.bennee@linaro.org>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>

---
v2
  - use the VBSA prebuilt shell
  - fail the test if we see FAILED in the console stream
---
 tests/functional/aarch64/meson.build       |  1 +
 tests/functional/aarch64/test_virt_vbsa.py | 99 ++++++++++++++++++++++
 2 files changed, 100 insertions(+)
 create mode 100755 tests/functional/aarch64/test_virt_vbsa.py

diff --git a/tests/functional/aarch64/meson.build b/tests/functional/aarch64/meson.build
index 49eca120589..7ea8c22b048 100644
--- a/tests/functional/aarch64/meson.build
+++ b/tests/functional/aarch64/meson.build
@@ -46,6 +46,7 @@ tests_aarch64_system_thorough = [
   'tuxrun',
   'virt',
   'virt_gpu',
+  'virt_vbsa',
   'xen',
   'xlnx_versal',
 ]
diff --git a/tests/functional/aarch64/test_virt_vbsa.py b/tests/functional/aarch64/test_virt_vbsa.py
new file mode 100755
index 00000000000..3da06904ea9
--- /dev/null
+++ b/tests/functional/aarch64/test_virt_vbsa.py
@@ -0,0 +1,99 @@
+#!/usr/bin/env python3
+#
+# Functional test that runs the Arm VBSA conformance tests.
+#
+# Copyright (c) 2026 Linaro Ltd.
+#
+# Author:
+#  Alex Bennée <alex.bennee@linaro.org>
+#
+# SPDX-License-Identifier: GPL-2.0-or-later
+
+import shutil
+from subprocess import check_call, DEVNULL
+
+from qemu_test import QemuSystemTest, Asset
+from qemu_test import wait_for_console_pattern, get_qemu_img, skipIfMissingCommands
+from qemu_test import exec_command_and_wait_for_pattern as ec_and_wait
+
+
+@skipIfMissingCommands("mformat", "mcopy", "mmd")
+class Aarch64VirtMachine(QemuSystemTest):
+    KERNEL_COMMON_COMMAND_LINE = 'printk.time=0 '
+    timeout = 360
+
+    def wait_for_console_pattern(self, success_message, vm=None):
+        wait_for_console_pattern(self, success_message,
+                                 failure_message='FAILED',
+                                 vm=vm)
+
+    ASSET_VBSA_EFI = Asset('https://github.com/ARM-software/sysarch-acs/raw/refs/heads/main'
+                           '/prebuilt_images/VBSA/v25.12_VBSA_0.7.0/Vbsa.efi',
+                           '80f37d2fb86d152d95dec4d05ff099c9e47ee8a89314268e08056b0e1359e1fa')
+
+    ASSET_BSA_SHELL = Asset('https://github.com/ARM-software/sysarch-acs/raw/refs/heads/main'
+                            '/prebuilt_images/VBSA/v25.12_VBSA_0.7.0/Shell.efi',
+                            'e526604f0d329b481c6a1f62f7a0db8ea24ce8178b2c6abda8e247425f38775c')
+
+    def test_aarch64_vbsa_uefi_tests(self):
+        """
+        Launch the UEFI based VBSA test from an EFI file-system
+        """
+
+        self.vm.set_console()
+
+        # virt machine wi
+        self.set_machine('virt')
+        self.vm.add_args('-M', 'virt,gic-version=max,virtualization=on')
+        self.vm.add_args('-cpu', 'max','-m', '1024')
+
+        # We will use the QEMU firmware blobs to boot
+        code_path = self.build_file('pc-bios', 'edk2-aarch64-code.fd')
+        vars_source = self.build_file('pc-bios', 'edk2-arm-vars.fd')
+        vars_path = self.scratch_file('vars.fd')
+        shutil.copy(vars_source, vars_path)
+
+        self.vm.add_args('-drive', f'if=pflash,format=raw,readonly=on,file={code_path}')
+        self.vm.add_args('-drive', f'if=pflash,format=raw,file={vars_path}')
+
+        # Build an EFI FAT32 file-system for the UEFI tests
+        vbsa_efi = self.ASSET_VBSA_EFI.fetch()
+        bsa_shell = self.ASSET_BSA_SHELL.fetch()
+
+        img_path = self.scratch_file('vbsa.img')
+        qemu_img = get_qemu_img(self)
+        check_call([qemu_img, 'create', '-f', 'raw', img_path, '64M'],
+                   stdout=DEVNULL, stderr=DEVNULL)
+
+        check_call(['mformat', '-i', img_path, '-v', 'VBSA', '::'],
+                   stdout=DEVNULL, stderr=DEVNULL)
+
+        check_call(['mmd', '-i', img_path, '::/EFI'],
+                   stdout=DEVNULL, stderr=DEVNULL)
+
+        check_call(['mmd', '-i', img_path, '::/EFI/BOOT'],
+                   stdout=DEVNULL, stderr=DEVNULL)
+
+        check_call(['mcopy', '-i', img_path, bsa_shell, '::/EFI/BOOT/BOOTAA64.EFI'],
+                   stdout=DEVNULL, stderr=DEVNULL)
+
+        check_call(['mcopy', '-i', img_path, vbsa_efi, '::/Vbsa.efi'],
+                   stdout=DEVNULL, stderr=DEVNULL)
+
+        self.vm.add_args('-drive', f'file={img_path},format=raw,if=none,id=drive0')
+        self.vm.add_args('-device', 'virtio-blk-pci,drive=drive0')
+
+        self.vm.launch()
+
+        # wait for EFI prompt
+        self.wait_for_console_pattern('Shell>')
+
+        # Start the VBSA tests
+        ec_and_wait(self, "FS0:Vbsa.efi", 'VBSA Architecture Compliance Suite')
+
+        # could we parse the summary somehow?
+
+        self.wait_for_console_pattern('VBSA tests complete. Reset the system.')
+
+if __name__ == '__main__':
+    QemuSystemTest.main()
-- 
2.47.3

Re: [PATCH v2 7/7] tests/functional: add Arm VBSA uefi conformance test

[Mohamed Mediouni]

> On 26. Feb 2026, at 19:53, Alex Bennée <alex.bennee@linaro.org> wrote:
> 
> The VBSA test is a subset of the wider Arm architecture compliance
> suites (ACS) which validate machines meet particular minimum set of
> requirements. The VBSA is for virtual machines so it makes sense we
> should check the -M virt machine is compliant.
> 
> Fortunately there are prebuilt binaries published via github so all we
> need to do is build an EFI partition and place things in the right
> place.
> 
> There are some additional Linux based tests which are left for later.
> 
> Message-ID: <20260213154859.1551283-7-alex.bennee@linaro.org>
> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Mohamed Mediouni <mohamed@unpredictable.fr>
> ---
> v2
>  - use the VBSA prebuilt shell
>  - fail the test if we see FAILED in the console stream
> ---
> tests/functional/aarch64/meson.build       |  1 +
> tests/functional/aarch64/test_virt_vbsa.py | 99 ++++++++++++++++++++++
> 2 files changed, 100 insertions(+)
> create mode 100755 tests/functional/aarch64/test_virt_vbsa.py
> 
> diff --git a/tests/functional/aarch64/meson.build b/tests/functional/aarch64/meson.build
> index 49eca120589..7ea8c22b048 100644
> --- a/tests/functional/aarch64/meson.build
> +++ b/tests/functional/aarch64/meson.build
> @@ -46,6 +46,7 @@ tests_aarch64_system_thorough = [
>   'tuxrun',
>   'virt',
>   'virt_gpu',
> +  'virt_vbsa',
>   'xen',
>   'xlnx_versal',
> ]
> diff --git a/tests/functional/aarch64/test_virt_vbsa.py b/tests/functional/aarch64/test_virt_vbsa.py
> new file mode 100755
> index 00000000000..3da06904ea9
> --- /dev/null
> +++ b/tests/functional/aarch64/test_virt_vbsa.py
> @@ -0,0 +1,99 @@
> +#!/usr/bin/env python3
> +#
> +# Functional test that runs the Arm VBSA conformance tests.
> +#
> +# Copyright (c) 2026 Linaro Ltd.
> +#
> +# Author:
> +#  Alex Bennée <alex.bennee@linaro.org>
> +#
> +# SPDX-License-Identifier: GPL-2.0-or-later
> +
> +import shutil
> +from subprocess import check_call, DEVNULL
> +
> +from qemu_test import QemuSystemTest, Asset
> +from qemu_test import wait_for_console_pattern, get_qemu_img, skipIfMissingCommands
> +from qemu_test import exec_command_and_wait_for_pattern as ec_and_wait
> +
> +
> +@skipIfMissingCommands("mformat", "mcopy", "mmd")
> +class Aarch64VirtMachine(QemuSystemTest):
> +    KERNEL_COMMON_COMMAND_LINE = 'printk.time=0 '
> +    timeout = 360
> +
> +    def wait_for_console_pattern(self, success_message, vm=None):
> +        wait_for_console_pattern(self, success_message,
> +                                 failure_message='FAILED',
> +                                 vm=vm)
> +
> +    ASSET_VBSA_EFI = Asset('https://github.com/ARM-software/sysarch-acs/raw/refs/heads/main'
> +                           '/prebuilt_images/VBSA/v25.12_VBSA_0.7.0/Vbsa.efi',
> +                           '80f37d2fb86d152d95dec4d05ff099c9e47ee8a89314268e08056b0e1359e1fa')
> +
> +    ASSET_BSA_SHELL = Asset('https://github.com/ARM-software/sysarch-acs/raw/refs/heads/main'
> +                            '/prebuilt_images/VBSA/v25.12_VBSA_0.7.0/Shell.efi',
> +                            'e526604f0d329b481c6a1f62f7a0db8ea24ce8178b2c6abda8e247425f38775c')
> +
> +    def test_aarch64_vbsa_uefi_tests(self):
> +        """
> +        Launch the UEFI based VBSA test from an EFI file-system
> +        """
> +
> +        self.vm.set_console()
> +
> +        # virt machine wi
> +        self.set_machine('virt')
> +        self.vm.add_args('-M', 'virt,gic-version=max,virtualization=on')
> +        self.vm.add_args('-cpu', 'max','-m', '1024')
> +
> +        # We will use the QEMU firmware blobs to boot
> +        code_path = self.build_file('pc-bios', 'edk2-aarch64-code.fd')
> +        vars_source = self.build_file('pc-bios', 'edk2-arm-vars.fd')
> +        vars_path = self.scratch_file('vars.fd')
> +        shutil.copy(vars_source, vars_path)
> +
> +        self.vm.add_args('-drive', f'if=pflash,format=raw,readonly=on,file={code_path}')
> +        self.vm.add_args('-drive', f'if=pflash,format=raw,file={vars_path}')
> +
> +        # Build an EFI FAT32 file-system for the UEFI tests
> +        vbsa_efi = self.ASSET_VBSA_EFI.fetch()
> +        bsa_shell = self.ASSET_BSA_SHELL.fetch()
> +
> +        img_path = self.scratch_file('vbsa.img')
> +        qemu_img = get_qemu_img(self)
> +        check_call([qemu_img, 'create', '-f', 'raw', img_path, '64M'],
> +                   stdout=DEVNULL, stderr=DEVNULL)
> +
> +        check_call(['mformat', '-i', img_path, '-v', 'VBSA', '::'],
> +                   stdout=DEVNULL, stderr=DEVNULL)
> +
> +        check_call(['mmd', '-i', img_path, '::/EFI'],
> +                   stdout=DEVNULL, stderr=DEVNULL)
> +
> +        check_call(['mmd', '-i', img_path, '::/EFI/BOOT'],
> +                   stdout=DEVNULL, stderr=DEVNULL)
> +
> +        check_call(['mcopy', '-i', img_path, bsa_shell, '::/EFI/BOOT/BOOTAA64.EFI'],
> +                   stdout=DEVNULL, stderr=DEVNULL)
> +
> +        check_call(['mcopy', '-i', img_path, vbsa_efi, '::/Vbsa.efi'],
> +                   stdout=DEVNULL, stderr=DEVNULL)
> +
> +        self.vm.add_args('-drive', f'file={img_path},format=raw,if=none,id=drive0')
> +        self.vm.add_args('-device', 'virtio-blk-pci,drive=drive0')
> +
> +        self.vm.launch()
> +
> +        # wait for EFI prompt
> +        self.wait_for_console_pattern('Shell>')
> +
> +        # Start the VBSA tests
> +        ec_and_wait(self, "FS0:Vbsa.efi", 'VBSA Architecture Compliance Suite')
> +
> +        # could we parse the summary somehow?
> +
> +        self.wait_for_console_pattern('VBSA tests complete. Reset the system.')
> +
> +if __name__ == '__main__':
> +    QemuSystemTest.main()
> -- 
> 2.47.3
> 
> 

Re: [PATCH v2 7/7] tests/functional: add Arm VBSA uefi conformance test

[Pierrick Bouvier]

On 2/26/26 10:53 AM, Alex Bennée wrote:
> The VBSA test is a subset of the wider Arm architecture compliance
> suites (ACS) which validate machines meet particular minimum set of
> requirements. The VBSA is for virtual machines so it makes sense we
> should check the -M virt machine is compliant.
> 
> Fortunately there are prebuilt binaries published via github so all we
> need to do is build an EFI partition and place things in the right
> place.
> 
> There are some additional Linux based tests which are left for later.
> 
> Message-ID: <20260213154859.1551283-7-alex.bennee@linaro.org>
> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
> 
> ---
> v2
>    - use the VBSA prebuilt shell
>    - fail the test if we see FAILED in the console stream
> ---
>   tests/functional/aarch64/meson.build       |  1 +
>   tests/functional/aarch64/test_virt_vbsa.py | 99 ++++++++++++++++++++++
>   2 files changed, 100 insertions(+)
>   create mode 100755 tests/functional/aarch64/test_virt_vbsa.py
> 
> diff --git a/tests/functional/aarch64/meson.build b/tests/functional/aarch64/meson.build
> index 49eca120589..7ea8c22b048 100644
> --- a/tests/functional/aarch64/meson.build
> +++ b/tests/functional/aarch64/meson.build
> @@ -46,6 +46,7 @@ tests_aarch64_system_thorough = [
>     'tuxrun',
>     'virt',
>     'virt_gpu',
> +  'virt_vbsa',
>     'xen',
>     'xlnx_versal',
>   ]
> diff --git a/tests/functional/aarch64/test_virt_vbsa.py b/tests/functional/aarch64/test_virt_vbsa.py
> new file mode 100755
> index 00000000000..3da06904ea9
> --- /dev/null
> +++ b/tests/functional/aarch64/test_virt_vbsa.py
> @@ -0,0 +1,99 @@
> +#!/usr/bin/env python3
> +#
> +# Functional test that runs the Arm VBSA conformance tests.
> +#
> +# Copyright (c) 2026 Linaro Ltd.
> +#
> +# Author:
> +#  Alex Bennée <alex.bennee@linaro.org>
> +#
> +# SPDX-License-Identifier: GPL-2.0-or-later
> +
> +import shutil
> +from subprocess import check_call, DEVNULL
> +
> +from qemu_test import QemuSystemTest, Asset
> +from qemu_test import wait_for_console_pattern, get_qemu_img, skipIfMissingCommands
> +from qemu_test import exec_command_and_wait_for_pattern as ec_and_wait
> +
> +
> +@skipIfMissingCommands("mformat", "mcopy", "mmd")
> +class Aarch64VirtMachine(QemuSystemTest):
> +    KERNEL_COMMON_COMMAND_LINE = 'printk.time=0 '
> +    timeout = 360
> +
> +    def wait_for_console_pattern(self, success_message, vm=None):
> +        wait_for_console_pattern(self, success_message,
> +                                 failure_message='FAILED',
> +                                 vm=vm)
> +
> +    ASSET_VBSA_EFI = Asset('https://github.com/ARM-software/sysarch-acs/raw/refs/heads/main'
> +                           '/prebuilt_images/VBSA/v25.12_VBSA_0.7.0/Vbsa.efi',
> +                           '80f37d2fb86d152d95dec4d05ff099c9e47ee8a89314268e08056b0e1359e1fa')
> +
> +    ASSET_BSA_SHELL = Asset('https://github.com/ARM-software/sysarch-acs/raw/refs/heads/main'
> +                            '/prebuilt_images/VBSA/v25.12_VBSA_0.7.0/Shell.efi',
> +                            'e526604f0d329b481c6a1f62f7a0db8ea24ce8178b2c6abda8e247425f38775c')
> +
> +    def test_aarch64_vbsa_uefi_tests(self):
> +        """
> +        Launch the UEFI based VBSA test from an EFI file-system
> +        """
> +
> +        self.vm.set_console()
> +
> +        # virt machine wi
> +        self.set_machine('virt')
> +        self.vm.add_args('-M', 'virt,gic-version=max,virtualization=on')
> +        self.vm.add_args('-cpu', 'max','-m', '1024')
> +
> +        # We will use the QEMU firmware blobs to boot
> +        code_path = self.build_file('pc-bios', 'edk2-aarch64-code.fd')
> +        vars_source = self.build_file('pc-bios', 'edk2-arm-vars.fd')
> +        vars_path = self.scratch_file('vars.fd')
> +        shutil.copy(vars_source, vars_path)
> +
> +        self.vm.add_args('-drive', f'if=pflash,format=raw,readonly=on,file={code_path}')
> +        self.vm.add_args('-drive', f'if=pflash,format=raw,file={vars_path}')
> +
> +        # Build an EFI FAT32 file-system for the UEFI tests
> +        vbsa_efi = self.ASSET_VBSA_EFI.fetch()
> +        bsa_shell = self.ASSET_BSA_SHELL.fetch()
> +
> +        img_path = self.scratch_file('vbsa.img')
> +        qemu_img = get_qemu_img(self)
> +        check_call([qemu_img, 'create', '-f', 'raw', img_path, '64M'],
> +                   stdout=DEVNULL, stderr=DEVNULL)
> +
> +        check_call(['mformat', '-i', img_path, '-v', 'VBSA', '::'],
> +                   stdout=DEVNULL, stderr=DEVNULL)
> +
> +        check_call(['mmd', '-i', img_path, '::/EFI'],
> +                   stdout=DEVNULL, stderr=DEVNULL)
> +
> +        check_call(['mmd', '-i', img_path, '::/EFI/BOOT'],
> +                   stdout=DEVNULL, stderr=DEVNULL)
> +
> +        check_call(['mcopy', '-i', img_path, bsa_shell, '::/EFI/BOOT/BOOTAA64.EFI'],
> +                   stdout=DEVNULL, stderr=DEVNULL)
> +
> +        check_call(['mcopy', '-i', img_path, vbsa_efi, '::/Vbsa.efi'],
> +                   stdout=DEVNULL, stderr=DEVNULL)
> +
> +        self.vm.add_args('-drive', f'file={img_path},format=raw,if=none,id=drive0')
> +        self.vm.add_args('-device', 'virtio-blk-pci,drive=drive0')
> +
> +        self.vm.launch()
> +
> +        # wait for EFI prompt
> +        self.wait_for_console_pattern('Shell>')
> +
> +        # Start the VBSA tests
> +        ec_and_wait(self, "FS0:Vbsa.efi", 'VBSA Architecture Compliance Suite')
> +
> +        # could we parse the summary somehow?
> +
> +        self.wait_for_console_pattern('VBSA tests complete. Reset the system.')
> +

Same comment than on v1:
https://lore.kernel.org/qemu-devel/5befbd0e-5f0f-41c4-99a7-1cf17bfe5454@linaro.org/

In case a failing output would not contain any string easy to identify, 
I'm fine with current patch, but since I didn't receive any answer on 
previous series, I'm not sure if it has been tried.

Regards,
Pierrick

Re: [PATCH v2 7/7] tests/functional: add Arm VBSA uefi conformance test

[Alex Bennée]

Pierrick Bouvier <pierrick.bouvier@linaro.org> writes:

> On 2/26/26 10:53 AM, Alex Bennée wrote:
>> The VBSA test is a subset of the wider Arm architecture compliance
>> suites (ACS) which validate machines meet particular minimum set of
>> requirements. The VBSA is for virtual machines so it makes sense we
>> should check the -M virt machine is compliant.
>> Fortunately there are prebuilt binaries published via github so all
>> we
>> need to do is build an EFI partition and place things in the right
>> place.
>> There are some additional Linux based tests which are left for
>> later.
>> Message-ID: <20260213154859.1551283-7-alex.bennee@linaro.org>
>> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
>> ---
>> v2
>>    - use the VBSA prebuilt shell
>>    - fail the test if we see FAILED in the console stream
>> ---
>>   tests/functional/aarch64/meson.build       |  1 +
>>   tests/functional/aarch64/test_virt_vbsa.py | 99 ++++++++++++++++++++++
>>   2 files changed, 100 insertions(+)
>>   create mode 100755 tests/functional/aarch64/test_virt_vbsa.py
>> diff --git a/tests/functional/aarch64/meson.build
>> b/tests/functional/aarch64/meson.build
>> index 49eca120589..7ea8c22b048 100644
>> --- a/tests/functional/aarch64/meson.build
>> +++ b/tests/functional/aarch64/meson.build
>> @@ -46,6 +46,7 @@ tests_aarch64_system_thorough = [
>>     'tuxrun',
>>     'virt',
>>     'virt_gpu',
>> +  'virt_vbsa',
>>     'xen',
>>     'xlnx_versal',
>>   ]
>> diff --git a/tests/functional/aarch64/test_virt_vbsa.py b/tests/functional/aarch64/test_virt_vbsa.py
>> new file mode 100755
>> index 00000000000..3da06904ea9
>> --- /dev/null
>> +++ b/tests/functional/aarch64/test_virt_vbsa.py
>> @@ -0,0 +1,99 @@
>> +#!/usr/bin/env python3
>> +#
>> +# Functional test that runs the Arm VBSA conformance tests.
>> +#
>> +# Copyright (c) 2026 Linaro Ltd.
>> +#
>> +# Author:
>> +#  Alex Bennée <alex.bennee@linaro.org>
>> +#
>> +# SPDX-License-Identifier: GPL-2.0-or-later
>> +
>> +import shutil
>> +from subprocess import check_call, DEVNULL
>> +
>> +from qemu_test import QemuSystemTest, Asset
>> +from qemu_test import wait_for_console_pattern, get_qemu_img, skipIfMissingCommands
>> +from qemu_test import exec_command_and_wait_for_pattern as ec_and_wait
>> +
>> +
>> +@skipIfMissingCommands("mformat", "mcopy", "mmd")
>> +class Aarch64VirtMachine(QemuSystemTest):
>> +    KERNEL_COMMON_COMMAND_LINE = 'printk.time=0 '
>> +    timeout = 360
>> +
>> +    def wait_for_console_pattern(self, success_message, vm=None):
>> +        wait_for_console_pattern(self, success_message,
>> +                                 failure_message='FAILED',
>> +                                 vm=vm)
>> +
>> +    ASSET_VBSA_EFI = Asset('https://github.com/ARM-software/sysarch-acs/raw/refs/heads/main'
>> +                           '/prebuilt_images/VBSA/v25.12_VBSA_0.7.0/Vbsa.efi',
>> +                           '80f37d2fb86d152d95dec4d05ff099c9e47ee8a89314268e08056b0e1359e1fa')
>> +
>> +    ASSET_BSA_SHELL = Asset('https://github.com/ARM-software/sysarch-acs/raw/refs/heads/main'
>> +                            '/prebuilt_images/VBSA/v25.12_VBSA_0.7.0/Shell.efi',
>> +                            'e526604f0d329b481c6a1f62f7a0db8ea24ce8178b2c6abda8e247425f38775c')
>> +
>> +    def test_aarch64_vbsa_uefi_tests(self):
>> +        """
>> +        Launch the UEFI based VBSA test from an EFI file-system
>> +        """
>> +
>> +        self.vm.set_console()
>> +
>> +        # virt machine wi
>> +        self.set_machine('virt')
>> +        self.vm.add_args('-M', 'virt,gic-version=max,virtualization=on')
>> +        self.vm.add_args('-cpu', 'max','-m', '1024')
>> +
>> +        # We will use the QEMU firmware blobs to boot
>> +        code_path = self.build_file('pc-bios', 'edk2-aarch64-code.fd')
>> +        vars_source = self.build_file('pc-bios', 'edk2-arm-vars.fd')
>> +        vars_path = self.scratch_file('vars.fd')
>> +        shutil.copy(vars_source, vars_path)
>> +
>> +        self.vm.add_args('-drive', f'if=pflash,format=raw,readonly=on,file={code_path}')
>> +        self.vm.add_args('-drive', f'if=pflash,format=raw,file={vars_path}')
>> +
>> +        # Build an EFI FAT32 file-system for the UEFI tests
>> +        vbsa_efi = self.ASSET_VBSA_EFI.fetch()
>> +        bsa_shell = self.ASSET_BSA_SHELL.fetch()
>> +
>> +        img_path = self.scratch_file('vbsa.img')
>> +        qemu_img = get_qemu_img(self)
>> +        check_call([qemu_img, 'create', '-f', 'raw', img_path, '64M'],
>> +                   stdout=DEVNULL, stderr=DEVNULL)
>> +
>> +        check_call(['mformat', '-i', img_path, '-v', 'VBSA', '::'],
>> +                   stdout=DEVNULL, stderr=DEVNULL)
>> +
>> +        check_call(['mmd', '-i', img_path, '::/EFI'],
>> +                   stdout=DEVNULL, stderr=DEVNULL)
>> +
>> +        check_call(['mmd', '-i', img_path, '::/EFI/BOOT'],
>> +                   stdout=DEVNULL, stderr=DEVNULL)
>> +
>> +        check_call(['mcopy', '-i', img_path, bsa_shell, '::/EFI/BOOT/BOOTAA64.EFI'],
>> +                   stdout=DEVNULL, stderr=DEVNULL)
>> +
>> +        check_call(['mcopy', '-i', img_path, vbsa_efi, '::/Vbsa.efi'],
>> +                   stdout=DEVNULL, stderr=DEVNULL)
>> +
>> +        self.vm.add_args('-drive', f'file={img_path},format=raw,if=none,id=drive0')
>> +        self.vm.add_args('-device', 'virtio-blk-pci,drive=drive0')
>> +
>> +        self.vm.launch()
>> +
>> +        # wait for EFI prompt
>> +        self.wait_for_console_pattern('Shell>')
>> +
>> +        # Start the VBSA tests
>> +        ec_and_wait(self, "FS0:Vbsa.efi", 'VBSA Architecture Compliance Suite')
>> +
>> +        # could we parse the summary somehow?
>> +
>> +        self.wait_for_console_pattern('VBSA tests complete. Reset the system.')
>> +
>
> Same comment than on v1:
> https://lore.kernel.org/qemu-devel/5befbd0e-5f0f-41c4-99a7-1cf17bfe5454@linaro.org/
>
> In case a failing output would not contain any string easy to
> identify, I'm fine with current patch, but since I didn't receive any
> answer on previous series, I'm not sure if it has been tried.

That's why I added the FAILED catch.

>
> Regards,
> Pierrick

-- 
Alex Bennée
Virtualisation Tech Lead @ Linaro

Re: [PATCH v2 7/7] tests/functional: add Arm VBSA uefi conformance test

[Pierrick Bouvier]

On 2/26/26 12:48 PM, Alex Bennée wrote:
> Pierrick Bouvier <pierrick.bouvier@linaro.org> writes:
> 
>> On 2/26/26 10:53 AM, Alex Bennée wrote:
>>> The VBSA test is a subset of the wider Arm architecture compliance
>>> suites (ACS) which validate machines meet particular minimum set of
>>> requirements. The VBSA is for virtual machines so it makes sense we
>>> should check the -M virt machine is compliant.
>>> Fortunately there are prebuilt binaries published via github so all
>>> we
>>> need to do is build an EFI partition and place things in the right
>>> place.
>>> There are some additional Linux based tests which are left for
>>> later.
>>> Message-ID: <20260213154859.1551283-7-alex.bennee@linaro.org>
>>> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
>>> ---
>>> v2
>>>     - use the VBSA prebuilt shell
>>>     - fail the test if we see FAILED in the console stream
>>> ---
>>>    tests/functional/aarch64/meson.build       |  1 +
>>>    tests/functional/aarch64/test_virt_vbsa.py | 99 ++++++++++++++++++++++
>>>    2 files changed, 100 insertions(+)
>>>    create mode 100755 tests/functional/aarch64/test_virt_vbsa.py
>>> diff --git a/tests/functional/aarch64/meson.build
>>> b/tests/functional/aarch64/meson.build
>>> index 49eca120589..7ea8c22b048 100644
>>> --- a/tests/functional/aarch64/meson.build
>>> +++ b/tests/functional/aarch64/meson.build
>>> @@ -46,6 +46,7 @@ tests_aarch64_system_thorough = [
>>>      'tuxrun',
>>>      'virt',
>>>      'virt_gpu',
>>> +  'virt_vbsa',
>>>      'xen',
>>>      'xlnx_versal',
>>>    ]
>>> diff --git a/tests/functional/aarch64/test_virt_vbsa.py b/tests/functional/aarch64/test_virt_vbsa.py
>>> new file mode 100755
>>> index 00000000000..3da06904ea9
>>> --- /dev/null
>>> +++ b/tests/functional/aarch64/test_virt_vbsa.py
>>> @@ -0,0 +1,99 @@
>>> +#!/usr/bin/env python3
>>> +#
>>> +# Functional test that runs the Arm VBSA conformance tests.
>>> +#
>>> +# Copyright (c) 2026 Linaro Ltd.
>>> +#
>>> +# Author:
>>> +#  Alex Bennée <alex.bennee@linaro.org>
>>> +#
>>> +# SPDX-License-Identifier: GPL-2.0-or-later
>>> +
>>> +import shutil
>>> +from subprocess import check_call, DEVNULL
>>> +
>>> +from qemu_test import QemuSystemTest, Asset
>>> +from qemu_test import wait_for_console_pattern, get_qemu_img, skipIfMissingCommands
>>> +from qemu_test import exec_command_and_wait_for_pattern as ec_and_wait
>>> +
>>> +
>>> +@skipIfMissingCommands("mformat", "mcopy", "mmd")
>>> +class Aarch64VirtMachine(QemuSystemTest):
>>> +    KERNEL_COMMON_COMMAND_LINE = 'printk.time=0 '
>>> +    timeout = 360
>>> +
>>> +    def wait_for_console_pattern(self, success_message, vm=None):
>>> +        wait_for_console_pattern(self, success_message,
>>> +                                 failure_message='FAILED',
>>> +                                 vm=vm)
>>> +
>>> +    ASSET_VBSA_EFI = Asset('https://github.com/ARM-software/sysarch-acs/raw/refs/heads/main'
>>> +                           '/prebuilt_images/VBSA/v25.12_VBSA_0.7.0/Vbsa.efi',
>>> +                           '80f37d2fb86d152d95dec4d05ff099c9e47ee8a89314268e08056b0e1359e1fa')
>>> +
>>> +    ASSET_BSA_SHELL = Asset('https://github.com/ARM-software/sysarch-acs/raw/refs/heads/main'
>>> +                            '/prebuilt_images/VBSA/v25.12_VBSA_0.7.0/Shell.efi',
>>> +                            'e526604f0d329b481c6a1f62f7a0db8ea24ce8178b2c6abda8e247425f38775c')
>>> +
>>> +    def test_aarch64_vbsa_uefi_tests(self):
>>> +        """
>>> +        Launch the UEFI based VBSA test from an EFI file-system
>>> +        """
>>> +
>>> +        self.vm.set_console()
>>> +
>>> +        # virt machine wi
>>> +        self.set_machine('virt')
>>> +        self.vm.add_args('-M', 'virt,gic-version=max,virtualization=on')
>>> +        self.vm.add_args('-cpu', 'max','-m', '1024')
>>> +
>>> +        # We will use the QEMU firmware blobs to boot
>>> +        code_path = self.build_file('pc-bios', 'edk2-aarch64-code.fd')
>>> +        vars_source = self.build_file('pc-bios', 'edk2-arm-vars.fd')
>>> +        vars_path = self.scratch_file('vars.fd')
>>> +        shutil.copy(vars_source, vars_path)
>>> +
>>> +        self.vm.add_args('-drive', f'if=pflash,format=raw,readonly=on,file={code_path}')
>>> +        self.vm.add_args('-drive', f'if=pflash,format=raw,file={vars_path}')
>>> +
>>> +        # Build an EFI FAT32 file-system for the UEFI tests
>>> +        vbsa_efi = self.ASSET_VBSA_EFI.fetch()
>>> +        bsa_shell = self.ASSET_BSA_SHELL.fetch()
>>> +
>>> +        img_path = self.scratch_file('vbsa.img')
>>> +        qemu_img = get_qemu_img(self)
>>> +        check_call([qemu_img, 'create', '-f', 'raw', img_path, '64M'],
>>> +                   stdout=DEVNULL, stderr=DEVNULL)
>>> +
>>> +        check_call(['mformat', '-i', img_path, '-v', 'VBSA', '::'],
>>> +                   stdout=DEVNULL, stderr=DEVNULL)
>>> +
>>> +        check_call(['mmd', '-i', img_path, '::/EFI'],
>>> +                   stdout=DEVNULL, stderr=DEVNULL)
>>> +
>>> +        check_call(['mmd', '-i', img_path, '::/EFI/BOOT'],
>>> +                   stdout=DEVNULL, stderr=DEVNULL)
>>> +
>>> +        check_call(['mcopy', '-i', img_path, bsa_shell, '::/EFI/BOOT/BOOTAA64.EFI'],
>>> +                   stdout=DEVNULL, stderr=DEVNULL)
>>> +
>>> +        check_call(['mcopy', '-i', img_path, vbsa_efi, '::/Vbsa.efi'],
>>> +                   stdout=DEVNULL, stderr=DEVNULL)
>>> +
>>> +        self.vm.add_args('-drive', f'file={img_path},format=raw,if=none,id=drive0')
>>> +        self.vm.add_args('-device', 'virtio-blk-pci,drive=drive0')
>>> +
>>> +        self.vm.launch()
>>> +
>>> +        # wait for EFI prompt
>>> +        self.wait_for_console_pattern('Shell>')
>>> +
>>> +        # Start the VBSA tests
>>> +        ec_and_wait(self, "FS0:Vbsa.efi", 'VBSA Architecture Compliance Suite')
>>> +
>>> +        # could we parse the summary somehow?
>>> +
>>> +        self.wait_for_console_pattern('VBSA tests complete. Reset the system.')
>>> +
>>
>> Same comment than on v1:
>> https://lore.kernel.org/qemu-devel/5befbd0e-5f0f-41c4-99a7-1cf17bfe5454@linaro.org/
>>
>> In case a failing output would not contain any string easy to
>> identify, I'm fine with current patch, but since I didn't receive any
>> answer on previous series, I'm not sure if it has been tried.
> 
> That's why I added the FAILED catch.
>

Oh right. Sorry I missed you overloaded the method at the top.

>>
>> Regards,
>> Pierrick
> 

Reviewed-by: Pierrick Bouvier <pierrick.bouvier@linaro.org>

Re: [PATCH v2 2/7] tests/docker: migrate legacy-test-cross compilers to trixie

[Thomas Huth]

On 26/02/2026 19.52, Alex Bennée wrote:
> The bugs have evidently been fixed in the latest release so we can
> migrate the laggards into how all-test-cross container and remove the
> legacy hacks. They are also packaged for the main architectures so we
> don't need to jump through the amd64 hoops.
> 
> Suggested-by: John Snow <jsnow@redhat.com>
> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
> ---
>   configure                                     | 10 +---
>   .gitlab-ci.d/buildtest.yml                    | 12 -----
>   .gitlab-ci.d/container-cross.yml              |  6 ---
>   .gitlab-ci.d/containers.yml                   |  1 -
>   .../dockerfiles/debian-all-test-cross.docker  |  8 ++-
>   .../debian-legacy-test-cross.docker           | 51 -------------------
>   tests/lcitool/refresh                         |  8 ++-
>   7 files changed, 13 insertions(+), 83 deletions(-)
>   delete mode 100644 tests/docker/dockerfiles/debian-legacy-test-cross.docker
> 

Nice! Kudos to John for figuring it out!

Reviewed-by: Thomas Huth <thuth@redhat.com>