[PATCH v3 0/4] Add driver domains test

[PATCH v3 0/4] Add driver domains test

[Marek Marczykowski-Górecki]

This is intended to detect issues like recent xl devd crash in domU.

Pipeline:
https://gitlab.com/xen-project/people/marmarek/xen/-/pipelines/2196576805

I did checked that indeed it fails with the fix reverted - that's why the
Debian patch - Alpine version did not detected that regression.

Requires debian in test-artifacts repo - patch series sent separately.

Marek Marczykowski-Górecki (4):
  automation: switch to alpine:3.22
  CI: Add driver domains tests
  CI: Add configure --enable-systemd for full build
  CI: Run driver domains test on Debian too

 automation/build/alpine/3.18-arm64v8.dockerfile |  51 +------
 automation/build/alpine/3.18.dockerfile         |  52 +------
 automation/build/alpine/3.22-arm64v8.dockerfile |  51 ++++++-
 automation/build/alpine/3.22.dockerfile         |  52 ++++++-
 automation/build/debian/13-x86_64.dockerfile    |   4 +-
 automation/gitlab-ci/build.yaml                 |  44 ++---
 automation/gitlab-ci/test.yaml                  | 125 ++++++++------
 automation/scripts/build                        |   3 +-
 automation/scripts/containerize                 |   4 +-
 automation/scripts/qemu-driverdomains-x86_64.sh | 152 +++++++++++++++++-
 10 files changed, 362 insertions(+), 176 deletions(-)
 delete mode 100644 automation/build/alpine/3.18-arm64v8.dockerfile
 delete mode 100644 automation/build/alpine/3.18.dockerfile
 create mode 100644 automation/build/alpine/3.22-arm64v8.dockerfile
 create mode 100644 automation/build/alpine/3.22.dockerfile
 create mode 100755 automation/scripts/qemu-driverdomains-x86_64.sh

base-commit: 62bd4c2a8ee809c181d47098583270dc9db9300e
-- 
git-series 0.9.1

[PATCH v3 2/4] CI: Add driver domains tests

[Marek Marczykowski-Górecki]

Setup a simple two domU system. One with network backend, running
xendriverdomain service, and one with frontend, trying to ping the
backend.

Contrary to other similar tests, use disk image instead of initrd, to
allow bigger rootfs without adding more RAM (for both dom0 and domU).
But keep using pxelinux as a bootloader as it's easier to setup than
installing grub on the disk. Theoretically, it could be started via direct
kernel boot in QEMU, but pxelinux is slightly closer to real-world
deployment.

Use fakeroot to preserve file owners/permissions. This is especially
important for suid binaries like /bin/mount - without fakeroot, they
will end up as suid into non-root user.

Signed-off-by: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com>
---
Changes in v3:
- add fakeroot
- run ldconfig at the disk image creation time, to avoid running it at
  dom0/domU boot time (which is much slower)
Changes in v2:
- use heredoc
- limit ping loop iterations
- use full "backend" / "frontend" in disk image names
- print domU consoles directly to /dev/console, to avoid systemd-added
  messages prefix
- terminate test on failure, don't wait for timeout
---
 automation/build/debian/13-x86_64.dockerfile    |   2 +-
 automation/gitlab-ci/test.yaml                  |   8 +-
 automation/scripts/qemu-driverdomains-x86_64.sh | 138 +++++++++++++++++-
 3 files changed, 148 insertions(+)
 create mode 100755 automation/scripts/qemu-driverdomains-x86_64.sh

diff --git a/automation/build/debian/13-x86_64.dockerfile b/automation/build/debian/13-x86_64.dockerfile
index 2c6c9d4a5098..6382bafbd5bd 100644
--- a/automation/build/debian/13-x86_64.dockerfile
+++ b/automation/build/debian/13-x86_64.dockerfile
@@ -55,7 +55,9 @@ RUN <<EOF
 
         # for test phase, qemu-* jobs
         busybox-static
+        e2fsprogs
         expect
+        fakeroot
         ovmf
         qemu-system-x86
 
diff --git a/automation/gitlab-ci/test.yaml b/automation/gitlab-ci/test.yaml
index 0c4f535cd112..52ffb271951a 100644
--- a/automation/gitlab-ci/test.yaml
+++ b/automation/gitlab-ci/test.yaml
@@ -656,6 +656,14 @@ qemu-alpine-x86_64-gcc:
     - *x86-64-test-needs
     - alpine-3.22-gcc
 
+qemu-alpine-driverdomains-x86_64-gcc:
+  extends: .qemu-x86-64
+  script:
+    - ./automation/scripts/qemu-driverdomains-x86_64.sh 2>&1 | tee ${LOGFILE}
+  needs:
+    - *x86-64-test-needs
+    - alpine-3.22-gcc
+
 qemu-smoke-x86-64-gcc:
   extends: .qemu-smoke-x86-64
   script:
diff --git a/automation/scripts/qemu-driverdomains-x86_64.sh b/automation/scripts/qemu-driverdomains-x86_64.sh
new file mode 100755
index 000000000000..c0241da54168
--- /dev/null
+++ b/automation/scripts/qemu-driverdomains-x86_64.sh
@@ -0,0 +1,138 @@
+#!/bin/bash
+
+set -ex -o pipefail
+
+dom0_rootfs_extra_comp=()
+dom0_rootfs_extra_uncomp=()
+
+cd binaries
+
+# DomU rootfs
+
+mkdir -p rootfs
+cd rootfs
+mkdir -p etc/local.d
+passed="ping test passed"
+failed="TEST FAILED"
+cat > etc/local.d/xen.start << EOF
+#!/bin/bash
+
+set -x
+
+if grep -q test=backend /proc/cmdline; then
+    brctl addbr xenbr0
+    ip link set xenbr0 up
+    ip addr add 192.168.0.1/24 dev xenbr0
+    bash /etc/init.d/xendriverdomain start
+    # log backend-related logs to the console
+    tail -F /var/log/xen/xldevd.log /var/log/xen/xen-hotplug.log >>/dev/console 2>/dev/null &
+else
+    ip link set eth0 up
+    ip addr add 192.168.0.2/24 dev eth0
+    timeout=6 # 6*10s
+    until ping -c 10 192.168.0.1; do
+        sleep 1
+        if [ \$timeout -le 0 ]; then
+            echo "${failed}"
+            exit 1
+        fi
+        ((timeout--))
+    done
+    echo "${passed}"
+fi
+EOF
+chmod +x etc/local.d/xen.start
+fakeroot sh -c "
+    zcat ../rootfs.cpio.gz | cpio -imd
+    zcat ../xen-tools.cpio.gz | cpio -imd
+    ldconfig -r .
+    touch etc/.updated
+    mkfs.ext4 -d . ../domU-rootfs.img 1024M
+"
+cd ..
+rm -rf rootfs
+
+# Dom0 rootfs
+mkdir -p rootfs
+cd rootfs
+fakeroot -s ../fakeroot-save sh -c "
+    zcat ../rootfs.cpio.gz | cpio -imd
+    zcat ../xen-tools.cpio.gz | cpio -imd
+    ldconfig -r .
+    touch etc/.updated
+"
+mkdir -p root etc/local.d
+cat > root/backend.cfg << EOF
+name="backend"
+memory=512
+vcpus=1
+kernel="/root/bzImage"
+extra="console=hvc0 root=/dev/xvda net.ifnames=0 test=backend"
+disk=[ '/root/domU-rootfs-backend.img,raw,xvda,rw' ]
+EOF
+cat > root/frontend.cfg << EOF
+name="frontend"
+memory=512
+vcpus=1
+kernel="/root/bzImage"
+extra="console=hvc0 root=/dev/xvda net.ifnames=0 test=frontend"
+disk=[ '/root/domU-rootfs-frontend.img,raw,xvda,rw' ]
+vif=[ 'bridge=xenbr0,backend=backend' ]
+EOF
+
+cat > etc/local.d/xen.start << EOF
+#!/bin/bash
+
+set -x
+
+bash /etc/init.d/xencommons start
+
+xl list
+
+tail -F /var/log/xen/console/guest-backend.log 2>/dev/null | sed -e "s/^/(backend) /" >>/dev/console &
+tail -F /var/log/xen/console/guest-frontend.log 2>/dev/null | sed -e "s/^/(frontend) /" >>/dev/console &
+xl -vvv create /root/backend.cfg
+xl -vvv create /root/frontend.cfg
+EOF
+chmod +x etc/local.d/xen.start
+
+cp ../domU-rootfs.img ./root/domU-rootfs-backend.img
+cp ../domU-rootfs.img ./root/domU-rootfs-frontend.img
+cp ../bzImage ./root/
+mkdir -p etc/default
+echo 'XENCONSOLED_TRACE=all' >> etc/default/xencommons
+mkdir -p var/log/xen/console
+fakeroot -i ../fakeroot-save mkfs.ext4 -d . ../dom0-rootfs.img 2048M
+cd ..
+rm -rf rootfs
+
+cd ..
+
+cat >> binaries/pxelinux.0 << EOF
+#!ipxe
+
+kernel xen console=com1 console_timestamps=boot
+module bzImage console=hvc0 root=/dev/sda net.ifnames=0
+boot
+EOF
+
+# Run the test
+rm -f smoke.serial
+export TEST_CMD="qemu-system-x86_64 \
+    -cpu qemu64,+svm \
+    -m 2G -smp 2 \
+    -monitor none -serial stdio \
+    -nographic \
+    -device virtio-net-pci,netdev=n0 \
+    -netdev user,id=n0,tftp=binaries,bootfile=/pxelinux.0 \
+    -drive file=binaries/dom0-rootfs.img,format=raw"
+
+export TEST_LOG="smoke.serial"
+export BOOT_MSG="Latest ChangeSet: "
+export LOG_MSG="Domain-0"
+# exit early on test failure too, check if it was success below
+export PASSED="$passed|$failed"
+
+./automation/scripts/console.exp | sed 's/\r\+$//'
+
+grep "$passed" smoke.serial
-- 
git-series 0.9.1

[PATCH v3 3/4] CI: Add configure --enable-systemd for full build

[Marek Marczykowski-Górecki]

This doesn't exclude sysvinit scripts, but allows testing systemd too.

Signed-off-by: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com>
--
Changes in v3:
- switch to trixie

New in v2.

Requires containers rebuild
---
 automation/build/debian/13-x86_64.dockerfile | 1 +
 automation/scripts/build                     | 3 +++
 2 files changed, 4 insertions(+)

diff --git a/automation/build/debian/13-x86_64.dockerfile b/automation/build/debian/13-x86_64.dockerfile
index 6382bafbd5bd..9387da453fea 100644
--- a/automation/build/debian/13-x86_64.dockerfile
+++ b/automation/build/debian/13-x86_64.dockerfile
@@ -26,6 +26,7 @@ RUN <<EOF
         cpio
         git-core
         pkg-config
+        systemd-dev
         wget
         # libxenguest dombuilder
         libbz2-dev
diff --git a/automation/scripts/build b/automation/scripts/build
index 7a81d229decd..e77529cfc8ec 100755
--- a/automation/scripts/build
+++ b/automation/scripts/build
@@ -71,6 +71,9 @@ else
     # Full build.  Figure out our ./configure options
     cfgargs=("--prefix=/usr")
     cfgargs+=("--enable-docs")
+    if pkg-config systemd; then
+        cfgargs+=("--enable-systemd")
+    fi
 
     # booleans for which compiler is in use
     cc_is_gcc="$($cc --version | grep -q gcc && echo "y" || :)"
-- 
git-series 0.9.1

[PATCH v3 4/4] CI: Run driver domains test on Debian too

[Marek Marczykowski-Górecki]

The recent failure affected only glibc-based systems, so do the test on
Debian too.

Signed-off-by: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com>
---
Changes in v3:
- update to trixie
Changes in v2:
- use systemd in Debian
---
 automation/build/debian/13-x86_64.dockerfile    |  1 +-
 automation/gitlab-ci/test.yaml                  | 19 ++++++++++++++++++-
 automation/scripts/qemu-driverdomains-x86_64.sh | 18 +++++++++++++++--
 3 files changed, 36 insertions(+), 2 deletions(-)

diff --git a/automation/build/debian/13-x86_64.dockerfile b/automation/build/debian/13-x86_64.dockerfile
index 9387da453fea..d0b7b8fe69d1 100644
--- a/automation/build/debian/13-x86_64.dockerfile
+++ b/automation/build/debian/13-x86_64.dockerfile
@@ -61,6 +61,7 @@ RUN <<EOF
         fakeroot
         ovmf
         qemu-system-x86
+        systemctl
 
         # for build-each-commit-gcc
         ccache
diff --git a/automation/gitlab-ci/test.yaml b/automation/gitlab-ci/test.yaml
index 52ffb271951a..292e5ddb8de5 100644
--- a/automation/gitlab-ci/test.yaml
+++ b/automation/gitlab-ci/test.yaml
@@ -30,6 +30,17 @@
     job: microcode-x86
     ref: master
 
+.debian-x86-64-test-needs: &debian-x86-64-test-needs
+  - project: xen-project/hardware/test-artifacts
+    job: linux-6.6.56-x86_64
+    ref: master
+  - project: xen-project/hardware/test-artifacts
+    job: debian-13-x86_64-rootfs
+    ref: master
+  - project: xen-project/hardware/test-artifacts
+    job: microcode-x86
+    ref: master
+
 .qemu-arm64:
   extends: .test-jobs-common
   variables:
@@ -664,6 +675,14 @@ qemu-alpine-driverdomains-x86_64-gcc:
     - *x86-64-test-needs
     - alpine-3.22-gcc
 
+qemu-debian-13-driverdomains-x86_64-gcc:
+  extends: .qemu-x86-64
+  script:
+    - ./automation/scripts/qemu-driverdomains-x86_64.sh 2>&1 | tee ${LOGFILE}
+  needs:
+    - *debian-x86-64-test-needs
+    - debian-13-x86_64-gcc-debug
+
 qemu-smoke-x86-64-gcc:
   extends: .qemu-smoke-x86-64
   script:
diff --git a/automation/scripts/qemu-driverdomains-x86_64.sh b/automation/scripts/qemu-driverdomains-x86_64.sh
index c0241da54168..6a69282470ca 100755
--- a/automation/scripts/qemu-driverdomains-x86_64.sh
+++ b/automation/scripts/qemu-driverdomains-x86_64.sh
@@ -23,7 +23,11 @@ if grep -q test=backend /proc/cmdline; then
     brctl addbr xenbr0
     ip link set xenbr0 up
     ip addr add 192.168.0.1/24 dev xenbr0
-    bash /etc/init.d/xendriverdomain start
+    if [ -d /run/systemd ]; then
+        systemctl start xendriverdomain
+    else
+        bash /etc/init.d/xendriverdomain start
+    fi
     # log backend-related logs to the console
     tail -F /var/log/xen/xldevd.log /var/log/xen/xen-hotplug.log >>/dev/console 2>/dev/null &
 else
@@ -85,7 +89,11 @@ cat > etc/local.d/xen.start << EOF
 
 set -x
 
-bash /etc/init.d/xencommons start
+if [ -d /run/systemd ]; then
+    systemctl start xen-init-dom0.service
+else
+    bash /etc/init.d/xencommons start
+fi
 
 xl list
 
@@ -102,6 +110,12 @@ cp ../bzImage ./root/
 mkdir -p etc/default
 echo 'XENCONSOLED_TRACE=all' >> etc/default/xencommons
 mkdir -p var/log/xen/console
+if [ -e etc/systemd/system.conf ]; then
+    systemctl --root=. enable \
+        xenstored.service \
+        xenconsoled.service \
+        xen-init-dom0.service
+fi
 fakeroot -i ../fakeroot-save mkfs.ext4 -d . ../dom0-rootfs.img 2048M
 cd ..
 rm -rf rootfs
-- 
git-series 0.9.1

Re: [PATCH v3 1/4] automation: switch to alpine:3.22

[Andrew Cooper]

On 04/12/2025 4:53 pm, Marek Marczykowski-Górecki wrote:
> Marek Marczykowski-Górecki (4):
>   automation: switch to alpine:3.22

I don't have a copy of 1/4 to reply to, so here will here will have to do.

One minor point to start with, your hunk to containerize[sic] still
references 3.19, not 3.22.

The more major point.  Bumping to new Alpine containers is the point we
need to make it rootless without breaking the older stable branches.

i.e. we need a brand new rootfull qubes-alpine container (or somesuch,
along the lines of xenial-xilinx.dockerfile), that the test job can
switch to.

Then, as part of this patch, we rework the new 3.22 containers to be
clean (heredocs, rootless, drop unnecessary dependencies).

I'm sure I've got a cleaned up copy the alpine containers somewhere, but
it will need finding and dusting off and testing again.  I'm happy for
it to be merged into this patch if you want.

Ordering wise, this depends on getting the 3.22 rootfs artefacts first,
it seems.

~Andrew

Re: [PATCH v3 3/4] CI: Add configure --enable-systemd for full build

[Andrew Cooper]

On 04/12/2025 4:53 pm, Marek Marczykowski-Górecki wrote:
> This doesn't exclude sysvinit scripts, but allows testing systemd too.
>
> Signed-off-by: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com>

Oh - I had a task to make this happen by default, which also missed
4.21.  It's bad that configure does the wrong thing by default for systemd.

> --
> Changes in v3:
> - switch to trixie
>
> New in v2.
>
> Requires containers rebuild
> ---
>  automation/build/debian/13-x86_64.dockerfile | 1 +
>  automation/scripts/build                     | 3 +++
>  2 files changed, 4 insertions(+)
>
> diff --git a/automation/build/debian/13-x86_64.dockerfile b/automation/build/debian/13-x86_64.dockerfile
> index 6382bafbd5bd..9387da453fea 100644
> --- a/automation/build/debian/13-x86_64.dockerfile
> +++ b/automation/build/debian/13-x86_64.dockerfile
> @@ -26,6 +26,7 @@ RUN <<EOF
>          cpio
>          git-core
>          pkg-config
> +        systemd-dev

Why is this needed?  Or rather, does this patch predate my breaking of
the dependency on system-dev?

~Andrew

Re: [PATCH v3 3/4] CI: Add configure --enable-systemd for full build

[Marek Marczykowski-Górecki]

[-- Attachment #1: Type: text/plain, Size: 619 bytes --]

On Thu, Dec 04, 2025 at 09:54:41PM +0000, Andrew Cooper wrote:
> On 04/12/2025 4:53 pm, Marek Marczykowski-Górecki wrote:
> > @@ -26,6 +26,7 @@ RUN <<EOF
> >          cpio
> >          git-core
> >          pkg-config
> > +        systemd-dev
> 
> Why is this needed?  Or rather, does this patch predate my breaking of
> the dependency on system-dev?

It's still needed, without it the build fails:
https://gitlab.com/xen-project/people/marmarek/xen/-/jobs/10959608099

Likely related to e54a6cd6a1f3802274cd3a58b07df5f012342192

-- 
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [PATCH v3 3/4] CI: Add configure --enable-systemd for full build

[Andrew Cooper]

On 04/12/2025 11:11 pm, Marek Marczykowski-Górecki wrote:
> On Thu, Dec 04, 2025 at 09:54:41PM +0000, Andrew Cooper wrote:
>> On 04/12/2025 4:53 pm, Marek Marczykowski-Górecki wrote:
>>> @@ -26,6 +26,7 @@ RUN <<EOF
>>>          cpio
>>>          git-core
>>>          pkg-config
>>> +        systemd-dev
>> Why is this needed?  Or rather, does this patch predate my breaking of
>> the dependency on system-dev?
> It's still needed, without it the build fails:
> https://gitlab.com/xen-project/people/marmarek/xen/-/jobs/10959608099
>
> Likely related to e54a6cd6a1f3802274cd3a58b07df5f012342192

Urgh.  Yeah, that breaks the build if systemd isn't installed.

We also need a build container that does not have systemd anywhere in
it, to block stuff like that getting in in the first place.

~Andrew

Re: [PATCH v3 3/4] CI: Add configure --enable-systemd for full build

[Marek Marczykowski-Górecki]

[-- Attachment #1: Type: text/plain, Size: 1072 bytes --]

On Thu, Dec 04, 2025 at 11:17:03PM +0000, Andrew Cooper wrote:
> On 04/12/2025 11:11 pm, Marek Marczykowski-Górecki wrote:
> > On Thu, Dec 04, 2025 at 09:54:41PM +0000, Andrew Cooper wrote:
> >> On 04/12/2025 4:53 pm, Marek Marczykowski-Górecki wrote:
> >>> @@ -26,6 +26,7 @@ RUN <<EOF
> >>>          cpio
> >>>          git-core
> >>>          pkg-config
> >>> +        systemd-dev
> >> Why is this needed?  Or rather, does this patch predate my breaking of
> >> the dependency on system-dev?
> > It's still needed, without it the build fails:
> > https://gitlab.com/xen-project/people/marmarek/xen/-/jobs/10959608099
> >
> > Likely related to e54a6cd6a1f3802274cd3a58b07df5f012342192
> 
> Urgh.  Yeah, that breaks the build if systemd isn't installed.

Technically, it breaks "just" --enable-systemd build.

> We also need a build container that does not have systemd anywhere in
> it, to block stuff like that getting in in the first place.

That is Alpine container.

-- 
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]