[PATCH v1 0/2] Add driver domains test

[PATCH v1 0/2] Add driver domains test

[Marek Marczykowski-Górecki]

This is intended to detect issues like recent xl devd crash in domU.

Pipeline:
https://gitlab.com/xen-project/people/marmarek/xen/-/pipelines/1970971528

I did checked that indeed it fails with the fix reverted - that's why the
second patch - Alpine version did not detected that regression.

Requires debian in test-artifacts repo - patch sent separately.

Marek Marczykowski-Górecki (2):
  CI: Add driver domains tests
  CI: Run driver domains test on Debian too

 automation/gitlab-ci/test.yaml                  |  27 ++++-
 automation/scripts/qemu-driverdomains-x86_64.sh | 116 +++++++++++++++++-
 2 files changed, 143 insertions(+)
 create mode 100755 automation/scripts/qemu-driverdomains-x86_64.sh

base-commit: dbcbbed4e9dc25faa211d359c2f04a9c70f087c9
-- 
git-series 0.9.1

[PATCH v1 1/2] CI: Add driver domains tests

[Marek Marczykowski-Górecki]

Setup a simple two domU system. One with network backend, running
xendriverdomain service, and one with frontend, trying to ping the
backend.

Contrary to other similar tests, use disk image instead of initrd, to
allow bigger rootfs without adding more RAM (for both dom0 and domU).
But keep using pxelinux as a bootloader as it's easier to setup than
installing grub on the disk. Theoretically, it could be started via direct
kernel boot in QEMU, but pxelinux is slightly closer to real-world
deployment.

Signed-off-by: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com>
---
 automation/gitlab-ci/test.yaml                  |   8 +-
 automation/scripts/qemu-driverdomains-x86_64.sh | 116 +++++++++++++++++-
 2 files changed, 124 insertions(+)
 create mode 100755 automation/scripts/qemu-driverdomains-x86_64.sh

diff --git a/automation/gitlab-ci/test.yaml b/automation/gitlab-ci/test.yaml
index 1f0b27b2378a..5c4b2dc304b4 100644
--- a/automation/gitlab-ci/test.yaml
+++ b/automation/gitlab-ci/test.yaml
@@ -656,6 +656,14 @@ qemu-alpine-x86_64-gcc:
     - *x86-64-test-needs
     - alpine-3.18-gcc
 
+qemu-alpine-driverdomains-x86_64-gcc:
+  extends: .qemu-x86-64
+  script:
+    - ./automation/scripts/qemu-driverdomains-x86_64.sh 2>&1 | tee ${LOGFILE}
+  needs:
+    - *x86-64-test-needs
+    - alpine-3.18-gcc
+
 qemu-smoke-x86-64-gcc:
   extends: .qemu-smoke-x86-64
   script:
diff --git a/automation/scripts/qemu-driverdomains-x86_64.sh b/automation/scripts/qemu-driverdomains-x86_64.sh
new file mode 100755
index 000000000000..1caaede7722e
--- /dev/null
+++ b/automation/scripts/qemu-driverdomains-x86_64.sh
@@ -0,0 +1,116 @@
+#!/bin/bash
+
+set -ex -o pipefail
+
+dom0_rootfs_extra_comp=()
+dom0_rootfs_extra_uncomp=()
+
+cd binaries
+
+# DomU rootfs
+
+mkdir -p rootfs
+cd rootfs
+mkdir -p etc/local.d
+passed="ping test passed"
+echo "#!/bin/bash
+
+set -x
+
+if grep -q test=backend /proc/cmdline; then
+    brctl addbr xenbr0
+    ip link set xenbr0 up
+    ip addr add 192.168.0.1/24 dev xenbr0
+    bash /etc/init.d/xendriverdomain start
+    # log backend-related logs to the console
+    tail -F /var/log/xen/xldevd.log /var/log/xen/xen-hotplug.log >>/dev/console 2>/dev/null &
+else
+    ip link set eth0 up
+    ip addr add 192.168.0.2/24 dev eth0
+    until ping -c 10 192.168.0.1; do
+        sleep 1
+    done
+    echo \"${passed}\"
+fi
+
+" > etc/local.d/xen.start
+chmod +x etc/local.d/xen.start
+zcat ../rootfs.cpio.gz | cpio -imd
+zcat ../xen-tools.cpio.gz | cpio -imd
+mkfs.ext4 -d . ../domU-rootfs.img 1024M
+cd ..
+rm -rf rootfs
+
+# Dom0 rootfs
+mkdir -p rootfs
+cd rootfs
+zcat ../rootfs.cpio.gz | cpio -imd
+zcat ../xen-tools.cpio.gz | cpio -imd
+mkdir -p root etc/local.d
+echo "name=\"backend\"
+memory=512
+vcpus=1
+kernel=\"/root/bzImage\"
+extra=\"console=hvc0 root=/dev/xvda net.ifnames=0 test=backend\"
+disk=[ '/root/domU-rootfs-b.img,raw,xvda,rw' ]
+" > root/backend.cfg
+echo "name=\"frontend\"
+memory=512
+vcpus=1
+kernel=\"/root/bzImage\"
+extra=\"console=hvc0 root=/dev/xvda net.ifnames=0 test=frontend\"
+disk=[ '/root/domU-rootfs-f.img,raw,xvda,rw' ]
+vif=[ 'bridge=xenbr0,backend=backend' ]
+" > root/frontend.cfg
+echo "#!/bin/bash
+
+set -x
+
+bash /etc/init.d/xencommons start
+
+xl list
+
+tail -F /var/log/xen/console/guest-backend.log 2>/dev/null | sed -e \"s/^/(backend) /\" &
+tail -F /var/log/xen/console/guest-frontend.log 2>/dev/null | sed -e \"s/^/(frontend) /\" &
+xl -vvv create /root/backend.cfg
+xl -vvv create /root/frontend.cfg
+" > etc/local.d/xen.start
+chmod +x etc/local.d/xen.start
+
+cp ../domU-rootfs.img ./root/domU-rootfs-b.img
+cp ../domU-rootfs.img ./root/domU-rootfs-f.img
+cp ../bzImage ./root/
+mkdir -p etc/default
+echo 'XENCONSOLED_TRACE=all' >> etc/default/xencommons
+mkdir -p var/log/xen/console
+mkfs.ext4 -d . ../dom0-rootfs.img 2048M
+cd ..
+rm -rf rootfs
+
+cd ..
+
+cat >> binaries/pxelinux.0 << EOF
+#!ipxe
+
+kernel xen console=com1 console_timestamps=boot
+module bzImage console=hvc0 root=/dev/sda net.ifnames=0
+boot
+EOF
+
+# Run the test
+rm -f smoke.serial
+export TEST_CMD="qemu-system-x86_64 \
+    -cpu qemu64,+svm \
+    -m 2G -smp 2 \
+    -monitor none -serial stdio \
+    -nographic \
+    -device virtio-net-pci,netdev=n0 \
+    -netdev user,id=n0,tftp=binaries,bootfile=/pxelinux.0 \
+    -drive file=binaries/dom0-rootfs.img,format=raw"
+
+export TEST_LOG="smoke.serial"
+export BOOT_MSG="Latest ChangeSet: "
+export LOG_MSG="Domain-0"
+export PASSED="$passed"
+
+./automation/scripts/console.exp | sed 's/\r\+$//'
-- 
git-series 0.9.1

Re: [PATCH v1 1/2] CI: Add driver domains tests

[dmkhn]

On Thu, Aug 07, 2025 at 02:06:49AM +0200, Marek Marczykowski-Górecki wrote:
> Setup a simple two domU system. One with network backend, running
> xendriverdomain service, and one with frontend, trying to ping the
> backend.
> 
> Contrary to other similar tests, use disk image instead of initrd, to
> allow bigger rootfs without adding more RAM (for both dom0 and domU).
> But keep using pxelinux as a bootloader as it's easier to setup than
> installing grub on the disk. Theoretically, it could be started via direct
> kernel boot in QEMU, but pxelinux is slightly closer to real-world
> deployment.
> 
> Signed-off-by: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com>
> ---
>  automation/gitlab-ci/test.yaml                  |   8 +-
>  automation/scripts/qemu-driverdomains-x86_64.sh | 116 +++++++++++++++++-
>  2 files changed, 124 insertions(+)
>  create mode 100755 automation/scripts/qemu-driverdomains-x86_64.sh
> 
> diff --git a/automation/gitlab-ci/test.yaml b/automation/gitlab-ci/test.yaml
> index 1f0b27b2378a..5c4b2dc304b4 100644
> --- a/automation/gitlab-ci/test.yaml
> +++ b/automation/gitlab-ci/test.yaml
> @@ -656,6 +656,14 @@ qemu-alpine-x86_64-gcc:
>      - *x86-64-test-needs
>      - alpine-3.18-gcc
> 
> +qemu-alpine-driverdomains-x86_64-gcc:
> +  extends: .qemu-x86-64
> +  script:
> +    - ./automation/scripts/qemu-driverdomains-x86_64.sh 2>&1 | tee ${LOGFILE}
> +  needs:
> +    - *x86-64-test-needs
> +    - alpine-3.18-gcc
> +
>  qemu-smoke-x86-64-gcc:
>    extends: .qemu-smoke-x86-64
>    script:
> diff --git a/automation/scripts/qemu-driverdomains-x86_64.sh b/automation/scripts/qemu-driverdomains-x86_64.sh
> new file mode 100755
> index 000000000000..1caaede7722e
> --- /dev/null
> +++ b/automation/scripts/qemu-driverdomains-x86_64.sh
> @@ -0,0 +1,116 @@
> +#!/bin/bash
> +
> +set -ex -o pipefail
> +
> +dom0_rootfs_extra_comp=()
> +dom0_rootfs_extra_uncomp=()
> +
> +cd binaries
> +
> +# DomU rootfs
> +
> +mkdir -p rootfs
> +cd rootfs
> +mkdir -p etc/local.d
> +passed="ping test passed"
> +echo "#!/bin/bash
> +
> +set -x
> +
> +if grep -q test=backend /proc/cmdline; then
> +    brctl addbr xenbr0
> +    ip link set xenbr0 up
> +    ip addr add 192.168.0.1/24 dev xenbr0
> +    bash /etc/init.d/xendriverdomain start
> +    # log backend-related logs to the console
> +    tail -F /var/log/xen/xldevd.log /var/log/xen/xen-hotplug.log >>/dev/console 2>/dev/null &
> +else
> +    ip link set eth0 up
> +    ip addr add 192.168.0.2/24 dev eth0
> +    until ping -c 10 192.168.0.1; do

I would limit the number of iterations (60?), otherwise in the worst case the
test will execute intil the test job timeout, which AFAIU is undesired if
anything goes wrong.

> +        sleep 1
> +    done
> +    echo \"${passed}\"
> +fi
> +
> +" > etc/local.d/xen.start

For the block generating xen.start: suggest switching to heredoc like for
pxelinux.0 config below, this way escaping will not be needed.

> +chmod +x etc/local.d/xen.start
> +zcat ../rootfs.cpio.gz | cpio -imd
> +zcat ../xen-tools.cpio.gz | cpio -imd
> +mkfs.ext4 -d . ../domU-rootfs.img 1024M
> +cd ..
> +rm -rf rootfs
> +
> +# Dom0 rootfs
> +mkdir -p rootfs
> +cd rootfs
> +zcat ../rootfs.cpio.gz | cpio -imd
> +zcat ../xen-tools.cpio.gz | cpio -imd
> +mkdir -p root etc/local.d
> +echo "name=\"backend\"
> +memory=512
> +vcpus=1
> +kernel=\"/root/bzImage\"
> +extra=\"console=hvc0 root=/dev/xvda net.ifnames=0 test=backend\"
> +disk=[ '/root/domU-rootfs-b.img,raw,xvda,rw' ]
> +" > root/backend.cfg
> +echo "name=\"frontend\"
> +memory=512
> +vcpus=1
> +kernel=\"/root/bzImage\"
> +extra=\"console=hvc0 root=/dev/xvda net.ifnames=0 test=frontend\"
> +disk=[ '/root/domU-rootfs-f.img,raw,xvda,rw' ]
> +vif=[ 'bridge=xenbr0,backend=backend' ]
> +" > root/frontend.cfg

.. and heredoc here ^^ ?

> +echo "#!/bin/bash
> +
> +set -x
> +
> +bash /etc/init.d/xencommons start
> +
> +xl list
> +
> +tail -F /var/log/xen/console/guest-backend.log 2>/dev/null | sed -e \"s/^/(backend) /\" &
> +tail -F /var/log/xen/console/guest-frontend.log 2>/dev/null | sed -e \"s/^/(frontend) /\" &
> +xl -vvv create /root/backend.cfg
> +xl -vvv create /root/frontend.cfg
> +" > etc/local.d/xen.start

.. and heredoc here ^^ ?

> +chmod +x etc/local.d/xen.start
> +
> +cp ../domU-rootfs.img ./root/domU-rootfs-b.img
> +cp ../domU-rootfs.img ./root/domU-rootfs-f.img
                                            ^
Perhaps use "backend" and "frontend" suffixes for clarify?

> +cp ../bzImage ./root/
> +mkdir -p etc/default
> +echo 'XENCONSOLED_TRACE=all' >> etc/default/xencommons
> +mkdir -p var/log/xen/console
> +mkfs.ext4 -d . ../dom0-rootfs.img 2048M
> +cd ..
> +rm -rf rootfs
> +
> +cd ..
> +
> +cat >> binaries/pxelinux.0 << EOF
> +#!ipxe
> +
> +kernel xen console=com1 console_timestamps=boot
> +module bzImage console=hvc0 root=/dev/sda net.ifnames=0
> +boot
> +EOF
> +
> +# Run the test
> +rm -f smoke.serial
> +export TEST_CMD="qemu-system-x86_64 \
> +    -cpu qemu64,+svm \
> +    -m 2G -smp 2 \
> +    -monitor none -serial stdio \
> +    -nographic \
> +    -device virtio-net-pci,netdev=n0 \
> +    -netdev user,id=n0,tftp=binaries,bootfile=/pxelinux.0 \
> +    -drive file=binaries/dom0-rootfs.img,format=raw"
> +
> +export TEST_LOG="smoke.serial"
> +export BOOT_MSG="Latest ChangeSet: "
> +export LOG_MSG="Domain-0"
> +export PASSED="$passed"
> +
> +./automation/scripts/console.exp | sed 's/\r\+$//'
> --
> git-series 0.9.1
> 

[PATCH v1 2/2] CI: Run driver domains test on Debian too

[Marek Marczykowski-Górecki]

The recent failure affected only glibc-based systems, so do the test on
Debian too.

Signed-off-by: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com>
---
 automation/gitlab-ci/test.yaml | 19 +++++++++++++++++++
 1 file changed, 19 insertions(+)

diff --git a/automation/gitlab-ci/test.yaml b/automation/gitlab-ci/test.yaml
index 5c4b2dc304b4..a5ae03b0eee9 100644
--- a/automation/gitlab-ci/test.yaml
+++ b/automation/gitlab-ci/test.yaml
@@ -30,6 +30,17 @@
     job: microcode-x86
     ref: master
 
+.debian-x86-64-test-needs: &debian-x86-64-test-needs
+  - project: xen-project/hardware/test-artifacts
+    job: linux-6.6.56-x86_64
+    ref: master
+  - project: xen-project/hardware/test-artifacts
+    job: debian-12-x86_64-rootfs
+    ref: master
+  - project: xen-project/hardware/test-artifacts
+    job: microcode-x86
+    ref: master
+
 .qemu-arm64:
   extends: .test-jobs-common
   variables:
@@ -664,6 +675,14 @@ qemu-alpine-driverdomains-x86_64-gcc:
     - *x86-64-test-needs
     - alpine-3.18-gcc
 
+qemu-debian-12-driverdomains-x86_64-gcc:
+  extends: .qemu-x86-64
+  script:
+    - ./automation/scripts/qemu-driverdomains-x86_64.sh 2>&1 | tee ${LOGFILE}
+  needs:
+    - *debian-x86-64-test-needs
+    - debian-12-x86_64-gcc-debug
+
 qemu-smoke-x86-64-gcc:
   extends: .qemu-smoke-x86-64
   script:
-- 
git-series 0.9.1