[PATCH v4 0/6] Add driver domains test

[PATCH v4 0/6] Add driver domains test

[Marek Marczykowski-Górecki]

This is intended to detect issues like recent xl devd crash in domU.

Pipeline (on top of Andrew's sd-notify.h fixup):
https://gitlab.com/xen-project/people/marmarek/xen/-/pipelines/2199305972

I did checked that indeed it fails with the fix reverted - that's why the
Debian patch - Alpine version did not detected that regression.

Requires Alpine 3.22 and Debian in test-artifacts repo - patch series sent
separately.
---
Cc: Andrew Cooper <andrew.cooper3@citrix.com>

Marek Marczykowski-Górecki (6):
  automation: switch to alpine:3.22
  CI: extract qubes test container
  CI: cleanup alpine containers
  CI: Add driver domains tests
  CI: Add configure --enable-systemd for full build
  CI: Run driver domains test on Debian too

 automation/build/alpine/3.18-arm64v8.dockerfile       |  51 +----
 automation/build/alpine/3.18.dockerfile               |  52 +----
 automation/build/alpine/3.22-arm64v8-qubes.dockerfile |  18 +-
 automation/build/alpine/3.22-arm64v8.dockerfile       |  46 +++-
 automation/build/alpine/3.22.dockerfile               |  59 ++++-
 automation/build/debian/13-x86_64.dockerfile          |   3 +-
 automation/gitlab-ci/build.yaml                       |  44 +--
 automation/gitlab-ci/test.yaml                        | 125 +++++----
 automation/scripts/build                              |   1 +-
 automation/scripts/containerize                       |   4 +-
 automation/scripts/qemu-driverdomains-x86_64.sh       | 152 +++++++++++-
 11 files changed, 379 insertions(+), 176 deletions(-)
 delete mode 100644 automation/build/alpine/3.18-arm64v8.dockerfile
 delete mode 100644 automation/build/alpine/3.18.dockerfile
 create mode 100644 automation/build/alpine/3.22-arm64v8-qubes.dockerfile
 create mode 100644 automation/build/alpine/3.22-arm64v8.dockerfile
 create mode 100644 automation/build/alpine/3.22.dockerfile
 create mode 100755 automation/scripts/qemu-driverdomains-x86_64.sh

base-commit: 28d1d1c3f5aa6563ecf07136ba064be0261565c1
-- 
git-series 0.9.1

[PATCH v4 4/6] CI: Add driver domains tests

[Marek Marczykowski-Górecki]

Setup a simple two domU system. One with network backend, running
xendriverdomain service, and one with frontend, trying to ping the
backend.

Contrary to other similar tests, use disk image instead of initrd, to
allow bigger rootfs without adding more RAM (for both dom0 and domU).
But keep using pxelinux as a bootloader as it's easier to setup than
installing grub on the disk. Theoretically, it could be started via direct
kernel boot in QEMU, but pxelinux is slightly closer to real-world
deployment.

Use fakeroot to preserve file owners/permissions. This is especially
important for suid binaries like /bin/mount - without fakeroot, they
will end up as suid into non-root user.

Signed-off-by: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com>
---
Changes in v3:
- add fakeroot
- run ldconfig at the disk image creation time, to avoid running it at
  dom0/domU boot time (which is much slower)
Changes in v2:
- use heredoc
- limit ping loop iterations
- use full "backend" / "frontend" in disk image names
- print domU consoles directly to /dev/console, to avoid systemd-added
  messages prefix
- terminate test on failure, don't wait for timeout
---
 automation/build/debian/13-x86_64.dockerfile    |   2 +-
 automation/gitlab-ci/test.yaml                  |   8 +-
 automation/scripts/qemu-driverdomains-x86_64.sh | 138 +++++++++++++++++-
 3 files changed, 148 insertions(+)
 create mode 100755 automation/scripts/qemu-driverdomains-x86_64.sh

diff --git a/automation/build/debian/13-x86_64.dockerfile b/automation/build/debian/13-x86_64.dockerfile
index 2c6c9d4a5098..6382bafbd5bd 100644
--- a/automation/build/debian/13-x86_64.dockerfile
+++ b/automation/build/debian/13-x86_64.dockerfile
@@ -55,7 +55,9 @@ RUN <<EOF
 
         # for test phase, qemu-* jobs
         busybox-static
+        e2fsprogs
         expect
+        fakeroot
         ovmf
         qemu-system-x86
 
diff --git a/automation/gitlab-ci/test.yaml b/automation/gitlab-ci/test.yaml
index 7b36f1e126ca..abc5339a74ab 100644
--- a/automation/gitlab-ci/test.yaml
+++ b/automation/gitlab-ci/test.yaml
@@ -656,6 +656,14 @@ qemu-alpine-x86_64-gcc:
     - *x86-64-test-needs
     - alpine-3.22-gcc
 
+qemu-alpine-driverdomains-x86_64-gcc:
+  extends: .qemu-x86-64
+  script:
+    - ./automation/scripts/qemu-driverdomains-x86_64.sh 2>&1 | tee ${LOGFILE}
+  needs:
+    - *x86-64-test-needs
+    - alpine-3.22-gcc
+
 qemu-smoke-x86-64-gcc:
   extends: .qemu-smoke-x86-64
   script:
diff --git a/automation/scripts/qemu-driverdomains-x86_64.sh b/automation/scripts/qemu-driverdomains-x86_64.sh
new file mode 100755
index 000000000000..c0241da54168
--- /dev/null
+++ b/automation/scripts/qemu-driverdomains-x86_64.sh
@@ -0,0 +1,138 @@
+#!/bin/bash
+
+set -ex -o pipefail
+
+dom0_rootfs_extra_comp=()
+dom0_rootfs_extra_uncomp=()
+
+cd binaries
+
+# DomU rootfs
+
+mkdir -p rootfs
+cd rootfs
+mkdir -p etc/local.d
+passed="ping test passed"
+failed="TEST FAILED"
+cat > etc/local.d/xen.start << EOF
+#!/bin/bash
+
+set -x
+
+if grep -q test=backend /proc/cmdline; then
+    brctl addbr xenbr0
+    ip link set xenbr0 up
+    ip addr add 192.168.0.1/24 dev xenbr0
+    bash /etc/init.d/xendriverdomain start
+    # log backend-related logs to the console
+    tail -F /var/log/xen/xldevd.log /var/log/xen/xen-hotplug.log >>/dev/console 2>/dev/null &
+else
+    ip link set eth0 up
+    ip addr add 192.168.0.2/24 dev eth0
+    timeout=6 # 6*10s
+    until ping -c 10 192.168.0.1; do
+        sleep 1
+        if [ \$timeout -le 0 ]; then
+            echo "${failed}"
+            exit 1
+        fi
+        ((timeout--))
+    done
+    echo "${passed}"
+fi
+EOF
+chmod +x etc/local.d/xen.start
+fakeroot sh -c "
+    zcat ../rootfs.cpio.gz | cpio -imd
+    zcat ../xen-tools.cpio.gz | cpio -imd
+    ldconfig -r .
+    touch etc/.updated
+    mkfs.ext4 -d . ../domU-rootfs.img 1024M
+"
+cd ..
+rm -rf rootfs
+
+# Dom0 rootfs
+mkdir -p rootfs
+cd rootfs
+fakeroot -s ../fakeroot-save sh -c "
+    zcat ../rootfs.cpio.gz | cpio -imd
+    zcat ../xen-tools.cpio.gz | cpio -imd
+    ldconfig -r .
+    touch etc/.updated
+"
+mkdir -p root etc/local.d
+cat > root/backend.cfg << EOF
+name="backend"
+memory=512
+vcpus=1
+kernel="/root/bzImage"
+extra="console=hvc0 root=/dev/xvda net.ifnames=0 test=backend"
+disk=[ '/root/domU-rootfs-backend.img,raw,xvda,rw' ]
+EOF
+cat > root/frontend.cfg << EOF
+name="frontend"
+memory=512
+vcpus=1
+kernel="/root/bzImage"
+extra="console=hvc0 root=/dev/xvda net.ifnames=0 test=frontend"
+disk=[ '/root/domU-rootfs-frontend.img,raw,xvda,rw' ]
+vif=[ 'bridge=xenbr0,backend=backend' ]
+EOF
+
+cat > etc/local.d/xen.start << EOF
+#!/bin/bash
+
+set -x
+
+bash /etc/init.d/xencommons start
+
+xl list
+
+tail -F /var/log/xen/console/guest-backend.log 2>/dev/null | sed -e "s/^/(backend) /" >>/dev/console &
+tail -F /var/log/xen/console/guest-frontend.log 2>/dev/null | sed -e "s/^/(frontend) /" >>/dev/console &
+xl -vvv create /root/backend.cfg
+xl -vvv create /root/frontend.cfg
+EOF
+chmod +x etc/local.d/xen.start
+
+cp ../domU-rootfs.img ./root/domU-rootfs-backend.img
+cp ../domU-rootfs.img ./root/domU-rootfs-frontend.img
+cp ../bzImage ./root/
+mkdir -p etc/default
+echo 'XENCONSOLED_TRACE=all' >> etc/default/xencommons
+mkdir -p var/log/xen/console
+fakeroot -i ../fakeroot-save mkfs.ext4 -d . ../dom0-rootfs.img 2048M
+cd ..
+rm -rf rootfs
+
+cd ..
+
+cat >> binaries/pxelinux.0 << EOF
+#!ipxe
+
+kernel xen console=com1 console_timestamps=boot
+module bzImage console=hvc0 root=/dev/sda net.ifnames=0
+boot
+EOF
+
+# Run the test
+rm -f smoke.serial
+export TEST_CMD="qemu-system-x86_64 \
+    -cpu qemu64,+svm \
+    -m 2G -smp 2 \
+    -monitor none -serial stdio \
+    -nographic \
+    -device virtio-net-pci,netdev=n0 \
+    -netdev user,id=n0,tftp=binaries,bootfile=/pxelinux.0 \
+    -drive file=binaries/dom0-rootfs.img,format=raw"
+
+export TEST_LOG="smoke.serial"
+export BOOT_MSG="Latest ChangeSet: "
+export LOG_MSG="Domain-0"
+# exit early on test failure too, check if it was success below
+export PASSED="$passed|$failed"
+
+./automation/scripts/console.exp | sed 's/\r\+$//'
+
+grep "$passed" smoke.serial
-- 
git-series 0.9.1

[PATCH v4 5/6] CI: Add configure --enable-systemd for full build

[Marek Marczykowski-Górecki]

This doesn't exclude sysvinit scripts, but allows testing systemd too.

Signed-off-by: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com>
--
Changes in v4:
- drop systemd-dev, add --enable-systemd always
Changes in v3:
- switch to trixie

New in v2.
---
 automation/scripts/build | 1 +
 1 file changed, 1 insertion(+)

diff --git a/automation/scripts/build b/automation/scripts/build
index 7a81d229decd..9485f37c7020 100755
--- a/automation/scripts/build
+++ b/automation/scripts/build
@@ -71,6 +71,7 @@ else
     # Full build.  Figure out our ./configure options
     cfgargs=("--prefix=/usr")
     cfgargs+=("--enable-docs")
+    cfgargs+=("--enable-systemd")
 
     # booleans for which compiler is in use
     cc_is_gcc="$($cc --version | grep -q gcc && echo "y" || :)"
-- 
git-series 0.9.1

[PATCH v4 6/6] CI: Run driver domains test on Debian too

[Marek Marczykowski-Górecki]

The recent failure affected only glibc-based systems, so do the test on
Debian too.

Signed-off-by: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com>
---
Changes in v3:
- update to trixie
Changes in v2:
- use systemd in Debian
---
 automation/build/debian/13-x86_64.dockerfile    |  1 +-
 automation/gitlab-ci/test.yaml                  | 19 ++++++++++++++++++-
 automation/scripts/qemu-driverdomains-x86_64.sh | 18 +++++++++++++++--
 3 files changed, 36 insertions(+), 2 deletions(-)

diff --git a/automation/build/debian/13-x86_64.dockerfile b/automation/build/debian/13-x86_64.dockerfile
index 6382bafbd5bd..3f84ba4a8823 100644
--- a/automation/build/debian/13-x86_64.dockerfile
+++ b/automation/build/debian/13-x86_64.dockerfile
@@ -60,6 +60,7 @@ RUN <<EOF
         fakeroot
         ovmf
         qemu-system-x86
+        systemctl
 
         # for build-each-commit-gcc
         ccache
diff --git a/automation/gitlab-ci/test.yaml b/automation/gitlab-ci/test.yaml
index abc5339a74ab..ffb2880f5af3 100644
--- a/automation/gitlab-ci/test.yaml
+++ b/automation/gitlab-ci/test.yaml
@@ -30,6 +30,17 @@
     job: microcode-x86
     ref: master
 
+.debian-x86-64-test-needs: &debian-x86-64-test-needs
+  - project: xen-project/hardware/test-artifacts
+    job: linux-6.6.56-x86_64
+    ref: master
+  - project: xen-project/hardware/test-artifacts
+    job: debian-13-x86_64-rootfs
+    ref: master
+  - project: xen-project/hardware/test-artifacts
+    job: microcode-x86
+    ref: master
+
 .qemu-arm64:
   extends: .test-jobs-common
   variables:
@@ -664,6 +675,14 @@ qemu-alpine-driverdomains-x86_64-gcc:
     - *x86-64-test-needs
     - alpine-3.22-gcc
 
+qemu-debian-13-driverdomains-x86_64-gcc:
+  extends: .qemu-x86-64
+  script:
+    - ./automation/scripts/qemu-driverdomains-x86_64.sh 2>&1 | tee ${LOGFILE}
+  needs:
+    - *debian-x86-64-test-needs
+    - debian-13-x86_64-gcc-debug
+
 qemu-smoke-x86-64-gcc:
   extends: .qemu-smoke-x86-64
   script:
diff --git a/automation/scripts/qemu-driverdomains-x86_64.sh b/automation/scripts/qemu-driverdomains-x86_64.sh
index c0241da54168..6a69282470ca 100755
--- a/automation/scripts/qemu-driverdomains-x86_64.sh
+++ b/automation/scripts/qemu-driverdomains-x86_64.sh
@@ -23,7 +23,11 @@ if grep -q test=backend /proc/cmdline; then
     brctl addbr xenbr0
     ip link set xenbr0 up
     ip addr add 192.168.0.1/24 dev xenbr0
-    bash /etc/init.d/xendriverdomain start
+    if [ -d /run/systemd ]; then
+        systemctl start xendriverdomain
+    else
+        bash /etc/init.d/xendriverdomain start
+    fi
     # log backend-related logs to the console
     tail -F /var/log/xen/xldevd.log /var/log/xen/xen-hotplug.log >>/dev/console 2>/dev/null &
 else
@@ -85,7 +89,11 @@ cat > etc/local.d/xen.start << EOF
 
 set -x
 
-bash /etc/init.d/xencommons start
+if [ -d /run/systemd ]; then
+    systemctl start xen-init-dom0.service
+else
+    bash /etc/init.d/xencommons start
+fi
 
 xl list
 
@@ -102,6 +110,12 @@ cp ../bzImage ./root/
 mkdir -p etc/default
 echo 'XENCONSOLED_TRACE=all' >> etc/default/xencommons
 mkdir -p var/log/xen/console
+if [ -e etc/systemd/system.conf ]; then
+    systemctl --root=. enable \
+        xenstored.service \
+        xenconsoled.service \
+        xen-init-dom0.service
+fi
 fakeroot -i ../fakeroot-save mkfs.ext4 -d . ../dom0-rootfs.img 2048M
 cd ..
 rm -rf rootfs
-- 
git-series 0.9.1

Re: [PATCH v4 0/6] Add driver domains test

[Marek Marczykowski-Górecki]

[-- Attachment #1: Type: text/plain, Size: 2373 bytes --]

On Sat, Dec 06, 2025 at 03:48:02AM +0100, Marek Marczykowski-Górecki wrote:
> This is intended to detect issues like recent xl devd crash in domU.
> 
> Pipeline (on top of Andrew's sd-notify.h fixup):
> https://gitlab.com/xen-project/people/marmarek/xen/-/pipelines/2199305972
> 
> I did checked that indeed it fails with the fix reverted - that's why the
> Debian patch - Alpine version did not detected that regression.
> 
> Requires Alpine 3.22 and Debian in test-artifacts repo - patch series sent
> separately.
> ---
> Cc: Andrew Cooper <andrew.cooper3@citrix.com>
> 
> Marek Marczykowski-Górecki (6):
>   automation: switch to alpine:3.22
>   CI: extract qubes test container
>   CI: cleanup alpine containers

As "expected", patches 1-3 didn't made it into ML... 
The pipeline linked above has them.

>   CI: Add driver domains tests
>   CI: Add configure --enable-systemd for full build
>   CI: Run driver domains test on Debian too
> 
>  automation/build/alpine/3.18-arm64v8.dockerfile       |  51 +----
>  automation/build/alpine/3.18.dockerfile               |  52 +----
>  automation/build/alpine/3.22-arm64v8-qubes.dockerfile |  18 +-
>  automation/build/alpine/3.22-arm64v8.dockerfile       |  46 +++-
>  automation/build/alpine/3.22.dockerfile               |  59 ++++-
>  automation/build/debian/13-x86_64.dockerfile          |   3 +-
>  automation/gitlab-ci/build.yaml                       |  44 +--
>  automation/gitlab-ci/test.yaml                        | 125 +++++----
>  automation/scripts/build                              |   1 +-
>  automation/scripts/containerize                       |   4 +-
>  automation/scripts/qemu-driverdomains-x86_64.sh       | 152 +++++++++++-
>  11 files changed, 379 insertions(+), 176 deletions(-)
>  delete mode 100644 automation/build/alpine/3.18-arm64v8.dockerfile
>  delete mode 100644 automation/build/alpine/3.18.dockerfile
>  create mode 100644 automation/build/alpine/3.22-arm64v8-qubes.dockerfile
>  create mode 100644 automation/build/alpine/3.22-arm64v8.dockerfile
>  create mode 100644 automation/build/alpine/3.22.dockerfile
>  create mode 100755 automation/scripts/qemu-driverdomains-x86_64.sh
> 
> base-commit: 28d1d1c3f5aa6563ecf07136ba064be0261565c1
> -- 
> git-series 0.9.1

-- 
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [PATCH v4 4/6] CI: Add driver domains tests

[Stefano Stabellini]

[-- Attachment #1: Type: text/plain, Size: 7225 bytes --]

On Sat, 6 Dec 2025, Marek Marczykowski-Górecki wrote:
> Setup a simple two domU system. One with network backend, running
> xendriverdomain service, and one with frontend, trying to ping the
> backend.
> 
> Contrary to other similar tests, use disk image instead of initrd, to
> allow bigger rootfs without adding more RAM (for both dom0 and domU).
> But keep using pxelinux as a bootloader as it's easier to setup than
> installing grub on the disk. Theoretically, it could be started via direct
> kernel boot in QEMU, but pxelinux is slightly closer to real-world
> deployment.
> 
> Use fakeroot to preserve file owners/permissions. This is especially
> important for suid binaries like /bin/mount - without fakeroot, they
> will end up as suid into non-root user.
> 
> Signed-off-by: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com>
> ---
> Changes in v3:
> - add fakeroot
> - run ldconfig at the disk image creation time, to avoid running it at
>   dom0/domU boot time (which is much slower)
> Changes in v2:
> - use heredoc
> - limit ping loop iterations
> - use full "backend" / "frontend" in disk image names
> - print domU consoles directly to /dev/console, to avoid systemd-added
>   messages prefix
> - terminate test on failure, don't wait for timeout
> ---
>  automation/build/debian/13-x86_64.dockerfile    |   2 +-
>  automation/gitlab-ci/test.yaml                  |   8 +-
>  automation/scripts/qemu-driverdomains-x86_64.sh | 138 +++++++++++++++++-
>  3 files changed, 148 insertions(+)
>  create mode 100755 automation/scripts/qemu-driverdomains-x86_64.sh
> 
> diff --git a/automation/build/debian/13-x86_64.dockerfile b/automation/build/debian/13-x86_64.dockerfile
> index 2c6c9d4a5098..6382bafbd5bd 100644
> --- a/automation/build/debian/13-x86_64.dockerfile
> +++ b/automation/build/debian/13-x86_64.dockerfile
> @@ -55,7 +55,9 @@ RUN <<EOF
>  
>          # for test phase, qemu-* jobs
>          busybox-static
> +        e2fsprogs
>          expect
> +        fakeroot
>          ovmf
>          qemu-system-x86
>  
> diff --git a/automation/gitlab-ci/test.yaml b/automation/gitlab-ci/test.yaml
> index 7b36f1e126ca..abc5339a74ab 100644
> --- a/automation/gitlab-ci/test.yaml
> +++ b/automation/gitlab-ci/test.yaml
> @@ -656,6 +656,14 @@ qemu-alpine-x86_64-gcc:
>      - *x86-64-test-needs
>      - alpine-3.22-gcc
>  
> +qemu-alpine-driverdomains-x86_64-gcc:
> +  extends: .qemu-x86-64
> +  script:
> +    - ./automation/scripts/qemu-driverdomains-x86_64.sh 2>&1 | tee ${LOGFILE}
> +  needs:
> +    - *x86-64-test-needs
> +    - alpine-3.22-gcc
> +
>  qemu-smoke-x86-64-gcc:
>    extends: .qemu-smoke-x86-64
>    script:
> diff --git a/automation/scripts/qemu-driverdomains-x86_64.sh b/automation/scripts/qemu-driverdomains-x86_64.sh
> new file mode 100755
> index 000000000000..c0241da54168
> --- /dev/null
> +++ b/automation/scripts/qemu-driverdomains-x86_64.sh
> @@ -0,0 +1,138 @@
> +#!/bin/bash
> +
> +set -ex -o pipefail
> +
> +dom0_rootfs_extra_comp=()
> +dom0_rootfs_extra_uncomp=()
> +
> +cd binaries
> +
> +# DomU rootfs
> +
> +mkdir -p rootfs
> +cd rootfs
> +mkdir -p etc/local.d
> +passed="ping test passed"
> +failed="TEST FAILED"
> +cat > etc/local.d/xen.start << EOF
> +#!/bin/bash
> +
> +set -x
> +
> +if grep -q test=backend /proc/cmdline; then
> +    brctl addbr xenbr0
> +    ip link set xenbr0 up
> +    ip addr add 192.168.0.1/24 dev xenbr0
> +    bash /etc/init.d/xendriverdomain start
> +    # log backend-related logs to the console
> +    tail -F /var/log/xen/xldevd.log /var/log/xen/xen-hotplug.log >>/dev/console 2>/dev/null &
> +else
> +    ip link set eth0 up
> +    ip addr add 192.168.0.2/24 dev eth0
> +    timeout=6 # 6*10s
> +    until ping -c 10 192.168.0.1; do
> +        sleep 1
> +        if [ \$timeout -le 0 ]; then
> +            echo "${failed}"
> +            exit 1
> +        fi
> +        ((timeout--))
> +    done
> +    echo "${passed}"
> +fi
> +EOF
> +chmod +x etc/local.d/xen.start
> +fakeroot sh -c "
> +    zcat ../rootfs.cpio.gz | cpio -imd
> +    zcat ../xen-tools.cpio.gz | cpio -imd
> +    ldconfig -r .
> +    touch etc/.updated
> +    mkfs.ext4 -d . ../domU-rootfs.img 1024M

Do we really need 1GB? I would rather use a smaller size if possible.
I would rather use as little resources as possible on the build server
as we might run a few of these jobs in parallel one day soon.

Moreover this script will be run inside a container which means this
data is probably in RAM.

The underlying rootfs is 25M on both ARM and x86. This should be at most
50M.


> +"
> +cd ..
> +rm -rf rootfs
> +
> +# Dom0 rootfs
> +mkdir -p rootfs
> +cd rootfs
> +fakeroot -s ../fakeroot-save sh -c "
> +    zcat ../rootfs.cpio.gz | cpio -imd
> +    zcat ../xen-tools.cpio.gz | cpio -imd
> +    ldconfig -r .
> +    touch etc/.updated
> +"
> +mkdir -p root etc/local.d
> +cat > root/backend.cfg << EOF
> +name="backend"
> +memory=512
> +vcpus=1
> +kernel="/root/bzImage"
> +extra="console=hvc0 root=/dev/xvda net.ifnames=0 test=backend"
> +disk=[ '/root/domU-rootfs-backend.img,raw,xvda,rw' ]
> +EOF
> +cat > root/frontend.cfg << EOF
> +name="frontend"
> +memory=512
> +vcpus=1
> +kernel="/root/bzImage"
> +extra="console=hvc0 root=/dev/xvda net.ifnames=0 test=frontend"
> +disk=[ '/root/domU-rootfs-frontend.img,raw,xvda,rw' ]
> +vif=[ 'bridge=xenbr0,backend=backend' ]
> +EOF
> +
> +cat > etc/local.d/xen.start << EOF
> +#!/bin/bash
> +
> +set -x
> +
> +bash /etc/init.d/xencommons start
> +
> +xl list
> +
> +tail -F /var/log/xen/console/guest-backend.log 2>/dev/null | sed -e "s/^/(backend) /" >>/dev/console &
> +tail -F /var/log/xen/console/guest-frontend.log 2>/dev/null | sed -e "s/^/(frontend) /" >>/dev/console &
> +xl -vvv create /root/backend.cfg
> +xl -vvv create /root/frontend.cfg
> +EOF
> +chmod +x etc/local.d/xen.start
> +
> +cp ../domU-rootfs.img ./root/domU-rootfs-backend.img
> +cp ../domU-rootfs.img ./root/domU-rootfs-frontend.img
> +cp ../bzImage ./root/
> +mkdir -p etc/default
> +echo 'XENCONSOLED_TRACE=all' >> etc/default/xencommons
> +mkdir -p var/log/xen/console
> +fakeroot -i ../fakeroot-save mkfs.ext4 -d . ../dom0-rootfs.img 2048M

Same here. Also 2GB might not be sufficient to contain 2 copies of
domU-rootfs.img, given that domU-rootfs.img is 1GB.

If we bring down domU-rootfs.img to 50M, then this could be 150M.


> +cd ..
> +rm -rf rootfs
> +
> +cd ..
> +
> +cat >> binaries/pxelinux.0 << EOF
> +#!ipxe
> +
> +kernel xen console=com1 console_timestamps=boot
> +module bzImage console=hvc0 root=/dev/sda net.ifnames=0
> +boot
> +EOF
> +
> +# Run the test
> +rm -f smoke.serial
> +export TEST_CMD="qemu-system-x86_64 \
> +    -cpu qemu64,+svm \
> +    -m 2G -smp 2 \
> +    -monitor none -serial stdio \
> +    -nographic \
> +    -device virtio-net-pci,netdev=n0 \
> +    -netdev user,id=n0,tftp=binaries,bootfile=/pxelinux.0 \
> +    -drive file=binaries/dom0-rootfs.img,format=raw"
> +
> +export TEST_LOG="smoke.serial"
> +export BOOT_MSG="Latest ChangeSet: "
> +export LOG_MSG="Domain-0"
> +# exit early on test failure too, check if it was success below
> +export PASSED="$passed|$failed"
> +
> +./automation/scripts/console.exp | sed 's/\r\+$//'
> +
> +grep "$passed" smoke.serial
> -- 
> git-series 0.9.1
> 

Re: [PATCH v4 5/6] CI: Add configure --enable-systemd for full build

[Stefano Stabellini]

[-- Attachment #1: Type: text/plain, Size: 976 bytes --]

On Sat, 6 Dec 2025, Marek Marczykowski-Górecki wrote:
> This doesn't exclude sysvinit scripts, but allows testing systemd too.
> 
> Signed-off-by: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com>

Acked-by: Stefano Stabellini <sstabellini@kernel.org>


> --
> Changes in v4:
> - drop systemd-dev, add --enable-systemd always
> Changes in v3:
> - switch to trixie
> 
> New in v2.
> ---
>  automation/scripts/build | 1 +
>  1 file changed, 1 insertion(+)
> 
> diff --git a/automation/scripts/build b/automation/scripts/build
> index 7a81d229decd..9485f37c7020 100755
> --- a/automation/scripts/build
> +++ b/automation/scripts/build
> @@ -71,6 +71,7 @@ else
>      # Full build.  Figure out our ./configure options
>      cfgargs=("--prefix=/usr")
>      cfgargs+=("--enable-docs")
> +    cfgargs+=("--enable-systemd")
>  
>      # booleans for which compiler is in use
>      cc_is_gcc="$($cc --version | grep -q gcc && echo "y" || :)"
> -- 
> git-series 0.9.1
> 

Re: [PATCH v4 6/6] CI: Run driver domains test on Debian too

[Stefano Stabellini]

[-- Attachment #1: Type: text/plain, Size: 3722 bytes --]

On Sat, 6 Dec 2025, Marek Marczykowski-Górecki wrote:
> The recent failure affected only glibc-based systems, so do the test on
> Debian too.
> 
> Signed-off-by: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com>

Acked-by: Stefano Stabellini <sstabellini@kernel.org>


> ---
> Changes in v3:
> - update to trixie
> Changes in v2:
> - use systemd in Debian
> ---
>  automation/build/debian/13-x86_64.dockerfile    |  1 +-
>  automation/gitlab-ci/test.yaml                  | 19 ++++++++++++++++++-
>  automation/scripts/qemu-driverdomains-x86_64.sh | 18 +++++++++++++++--
>  3 files changed, 36 insertions(+), 2 deletions(-)
> 
> diff --git a/automation/build/debian/13-x86_64.dockerfile b/automation/build/debian/13-x86_64.dockerfile
> index 6382bafbd5bd..3f84ba4a8823 100644
> --- a/automation/build/debian/13-x86_64.dockerfile
> +++ b/automation/build/debian/13-x86_64.dockerfile
> @@ -60,6 +60,7 @@ RUN <<EOF
>          fakeroot
>          ovmf
>          qemu-system-x86
> +        systemctl
>  
>          # for build-each-commit-gcc
>          ccache
> diff --git a/automation/gitlab-ci/test.yaml b/automation/gitlab-ci/test.yaml
> index abc5339a74ab..ffb2880f5af3 100644
> --- a/automation/gitlab-ci/test.yaml
> +++ b/automation/gitlab-ci/test.yaml
> @@ -30,6 +30,17 @@
>      job: microcode-x86
>      ref: master
>  
> +.debian-x86-64-test-needs: &debian-x86-64-test-needs
> +  - project: xen-project/hardware/test-artifacts
> +    job: linux-6.6.56-x86_64
> +    ref: master
> +  - project: xen-project/hardware/test-artifacts
> +    job: debian-13-x86_64-rootfs
> +    ref: master
> +  - project: xen-project/hardware/test-artifacts
> +    job: microcode-x86
> +    ref: master
> +
>  .qemu-arm64:
>    extends: .test-jobs-common
>    variables:
> @@ -664,6 +675,14 @@ qemu-alpine-driverdomains-x86_64-gcc:
>      - *x86-64-test-needs
>      - alpine-3.22-gcc
>  
> +qemu-debian-13-driverdomains-x86_64-gcc:
> +  extends: .qemu-x86-64
> +  script:
> +    - ./automation/scripts/qemu-driverdomains-x86_64.sh 2>&1 | tee ${LOGFILE}
> +  needs:
> +    - *debian-x86-64-test-needs
> +    - debian-13-x86_64-gcc-debug
> +
>  qemu-smoke-x86-64-gcc:
>    extends: .qemu-smoke-x86-64
>    script:
> diff --git a/automation/scripts/qemu-driverdomains-x86_64.sh b/automation/scripts/qemu-driverdomains-x86_64.sh
> index c0241da54168..6a69282470ca 100755
> --- a/automation/scripts/qemu-driverdomains-x86_64.sh
> +++ b/automation/scripts/qemu-driverdomains-x86_64.sh
> @@ -23,7 +23,11 @@ if grep -q test=backend /proc/cmdline; then
>      brctl addbr xenbr0
>      ip link set xenbr0 up
>      ip addr add 192.168.0.1/24 dev xenbr0
> -    bash /etc/init.d/xendriverdomain start
> +    if [ -d /run/systemd ]; then
> +        systemctl start xendriverdomain
> +    else
> +        bash /etc/init.d/xendriverdomain start
> +    fi
>      # log backend-related logs to the console
>      tail -F /var/log/xen/xldevd.log /var/log/xen/xen-hotplug.log >>/dev/console 2>/dev/null &
>  else
> @@ -85,7 +89,11 @@ cat > etc/local.d/xen.start << EOF
>  
>  set -x
>  
> -bash /etc/init.d/xencommons start
> +if [ -d /run/systemd ]; then
> +    systemctl start xen-init-dom0.service
> +else
> +    bash /etc/init.d/xencommons start
> +fi
>  
>  xl list
>  
> @@ -102,6 +110,12 @@ cp ../bzImage ./root/
>  mkdir -p etc/default
>  echo 'XENCONSOLED_TRACE=all' >> etc/default/xencommons
>  mkdir -p var/log/xen/console
> +if [ -e etc/systemd/system.conf ]; then
> +    systemctl --root=. enable \
> +        xenstored.service \
> +        xenconsoled.service \
> +        xen-init-dom0.service
> +fi
>  fakeroot -i ../fakeroot-save mkfs.ext4 -d . ../dom0-rootfs.img 2048M
>  cd ..
>  rm -rf rootfs
> -- 
> git-series 0.9.1
> 

Re: [PATCH v4 4/6] CI: Add driver domains tests

[Marek Marczykowski-Górecki]

[-- Attachment #1: Type: text/plain, Size: 8811 bytes --]

On Tue, Dec 09, 2025 at 04:02:06PM -0800, Stefano Stabellini wrote:
> On Sat, 6 Dec 2025, Marek Marczykowski-Górecki wrote:
> > Setup a simple two domU system. One with network backend, running
> > xendriverdomain service, and one with frontend, trying to ping the
> > backend.
> > 
> > Contrary to other similar tests, use disk image instead of initrd, to
> > allow bigger rootfs without adding more RAM (for both dom0 and domU).
> > But keep using pxelinux as a bootloader as it's easier to setup than
> > installing grub on the disk. Theoretically, it could be started via direct
> > kernel boot in QEMU, but pxelinux is slightly closer to real-world
> > deployment.
> > 
> > Use fakeroot to preserve file owners/permissions. This is especially
> > important for suid binaries like /bin/mount - without fakeroot, they
> > will end up as suid into non-root user.
> > 
> > Signed-off-by: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com>
> > ---
> > Changes in v3:
> > - add fakeroot
> > - run ldconfig at the disk image creation time, to avoid running it at
> >   dom0/domU boot time (which is much slower)
> > Changes in v2:
> > - use heredoc
> > - limit ping loop iterations
> > - use full "backend" / "frontend" in disk image names
> > - print domU consoles directly to /dev/console, to avoid systemd-added
> >   messages prefix
> > - terminate test on failure, don't wait for timeout
> > ---
> >  automation/build/debian/13-x86_64.dockerfile    |   2 +-
> >  automation/gitlab-ci/test.yaml                  |   8 +-
> >  automation/scripts/qemu-driverdomains-x86_64.sh | 138 +++++++++++++++++-
> >  3 files changed, 148 insertions(+)
> >  create mode 100755 automation/scripts/qemu-driverdomains-x86_64.sh
> > 
> > diff --git a/automation/build/debian/13-x86_64.dockerfile b/automation/build/debian/13-x86_64.dockerfile
> > index 2c6c9d4a5098..6382bafbd5bd 100644
> > --- a/automation/build/debian/13-x86_64.dockerfile
> > +++ b/automation/build/debian/13-x86_64.dockerfile
> > @@ -55,7 +55,9 @@ RUN <<EOF
> >  
> >          # for test phase, qemu-* jobs
> >          busybox-static
> > +        e2fsprogs
> >          expect
> > +        fakeroot
> >          ovmf
> >          qemu-system-x86
> >  
> > diff --git a/automation/gitlab-ci/test.yaml b/automation/gitlab-ci/test.yaml
> > index 7b36f1e126ca..abc5339a74ab 100644
> > --- a/automation/gitlab-ci/test.yaml
> > +++ b/automation/gitlab-ci/test.yaml
> > @@ -656,6 +656,14 @@ qemu-alpine-x86_64-gcc:
> >      - *x86-64-test-needs
> >      - alpine-3.22-gcc
> >  
> > +qemu-alpine-driverdomains-x86_64-gcc:
> > +  extends: .qemu-x86-64
> > +  script:
> > +    - ./automation/scripts/qemu-driverdomains-x86_64.sh 2>&1 | tee ${LOGFILE}
> > +  needs:
> > +    - *x86-64-test-needs
> > +    - alpine-3.22-gcc
> > +
> >  qemu-smoke-x86-64-gcc:
> >    extends: .qemu-smoke-x86-64
> >    script:
> > diff --git a/automation/scripts/qemu-driverdomains-x86_64.sh b/automation/scripts/qemu-driverdomains-x86_64.sh
> > new file mode 100755
> > index 000000000000..c0241da54168
> > --- /dev/null
> > +++ b/automation/scripts/qemu-driverdomains-x86_64.sh
> > @@ -0,0 +1,138 @@
> > +#!/bin/bash
> > +
> > +set -ex -o pipefail
> > +
> > +dom0_rootfs_extra_comp=()
> > +dom0_rootfs_extra_uncomp=()
> > +
> > +cd binaries
> > +
> > +# DomU rootfs
> > +
> > +mkdir -p rootfs
> > +cd rootfs
> > +mkdir -p etc/local.d
> > +passed="ping test passed"
> > +failed="TEST FAILED"
> > +cat > etc/local.d/xen.start << EOF
> > +#!/bin/bash
> > +
> > +set -x
> > +
> > +if grep -q test=backend /proc/cmdline; then
> > +    brctl addbr xenbr0
> > +    ip link set xenbr0 up
> > +    ip addr add 192.168.0.1/24 dev xenbr0
> > +    bash /etc/init.d/xendriverdomain start
> > +    # log backend-related logs to the console
> > +    tail -F /var/log/xen/xldevd.log /var/log/xen/xen-hotplug.log >>/dev/console 2>/dev/null &
> > +else
> > +    ip link set eth0 up
> > +    ip addr add 192.168.0.2/24 dev eth0
> > +    timeout=6 # 6*10s
> > +    until ping -c 10 192.168.0.1; do
> > +        sleep 1
> > +        if [ \$timeout -le 0 ]; then
> > +            echo "${failed}"
> > +            exit 1
> > +        fi
> > +        ((timeout--))
> > +    done
> > +    echo "${passed}"
> > +fi
> > +EOF
> > +chmod +x etc/local.d/xen.start
> > +fakeroot sh -c "
> > +    zcat ../rootfs.cpio.gz | cpio -imd
> > +    zcat ../xen-tools.cpio.gz | cpio -imd
> > +    ldconfig -r .
> > +    touch etc/.updated
> > +    mkfs.ext4 -d . ../domU-rootfs.img 1024M
> 
> Do we really need 1GB? I would rather use a smaller size if possible.
> I would rather use as little resources as possible on the build server
> as we might run a few of these jobs in parallel one day soon.

This will be a sparse file, so it won't use really all the space. But
this size is the upper bound of what can be put inside.
That said, it's worth checking if sparse files do work properly on all
runners in /build. AFAIR some older docker versions had issues with that
(was it aufs not supporting sparse files?).

> Moreover this script will be run inside a container which means this
> data is probably in RAM.

Are runners configured to use tmpfs for /build? I don't think it's the
default.

> The underlying rootfs is 25M on both ARM and x86. This should be at most
> 50M.

Rootfs itself is small, but for driver domains it needs to include
toolstack too, and xen-tools.cpio is over 600MB (for debug build).
I might be able to pick just the parts needed for the driver domain (xl
with its deps, maybe some startup scripts, probably few more files), but
it's rather fragile.

> > +"
> > +cd ..
> > +rm -rf rootfs
> > +
> > +# Dom0 rootfs
> > +mkdir -p rootfs
> > +cd rootfs
> > +fakeroot -s ../fakeroot-save sh -c "
> > +    zcat ../rootfs.cpio.gz | cpio -imd
> > +    zcat ../xen-tools.cpio.gz | cpio -imd
> > +    ldconfig -r .
> > +    touch etc/.updated
> > +"
> > +mkdir -p root etc/local.d
> > +cat > root/backend.cfg << EOF
> > +name="backend"
> > +memory=512
> > +vcpus=1
> > +kernel="/root/bzImage"
> > +extra="console=hvc0 root=/dev/xvda net.ifnames=0 test=backend"
> > +disk=[ '/root/domU-rootfs-backend.img,raw,xvda,rw' ]
> > +EOF
> > +cat > root/frontend.cfg << EOF
> > +name="frontend"
> > +memory=512
> > +vcpus=1
> > +kernel="/root/bzImage"
> > +extra="console=hvc0 root=/dev/xvda net.ifnames=0 test=frontend"
> > +disk=[ '/root/domU-rootfs-frontend.img,raw,xvda,rw' ]
> > +vif=[ 'bridge=xenbr0,backend=backend' ]
> > +EOF
> > +
> > +cat > etc/local.d/xen.start << EOF
> > +#!/bin/bash
> > +
> > +set -x
> > +
> > +bash /etc/init.d/xencommons start
> > +
> > +xl list
> > +
> > +tail -F /var/log/xen/console/guest-backend.log 2>/dev/null | sed -e "s/^/(backend) /" >>/dev/console &
> > +tail -F /var/log/xen/console/guest-frontend.log 2>/dev/null | sed -e "s/^/(frontend) /" >>/dev/console &
> > +xl -vvv create /root/backend.cfg
> > +xl -vvv create /root/frontend.cfg
> > +EOF
> > +chmod +x etc/local.d/xen.start
> > +
> > +cp ../domU-rootfs.img ./root/domU-rootfs-backend.img
> > +cp ../domU-rootfs.img ./root/domU-rootfs-frontend.img
> > +cp ../bzImage ./root/
> > +mkdir -p etc/default
> > +echo 'XENCONSOLED_TRACE=all' >> etc/default/xencommons
> > +mkdir -p var/log/xen/console
> > +fakeroot -i ../fakeroot-save mkfs.ext4 -d . ../dom0-rootfs.img 2048M
> 
> Same here. Also 2GB might not be sufficient to contain 2 copies of
> domU-rootfs.img, given that domU-rootfs.img is 1GB.

See the note about sparse files.

> If we bring down domU-rootfs.img to 50M, then this could be 150M.
> 
> 
> > +cd ..
> > +rm -rf rootfs
> > +
> > +cd ..
> > +
> > +cat >> binaries/pxelinux.0 << EOF
> > +#!ipxe
> > +
> > +kernel xen console=com1 console_timestamps=boot
> > +module bzImage console=hvc0 root=/dev/sda net.ifnames=0
> > +boot
> > +EOF
> > +
> > +# Run the test
> > +rm -f smoke.serial
> > +export TEST_CMD="qemu-system-x86_64 \
> > +    -cpu qemu64,+svm \
> > +    -m 2G -smp 2 \
> > +    -monitor none -serial stdio \
> > +    -nographic \
> > +    -device virtio-net-pci,netdev=n0 \
> > +    -netdev user,id=n0,tftp=binaries,bootfile=/pxelinux.0 \
> > +    -drive file=binaries/dom0-rootfs.img,format=raw"
> > +
> > +export TEST_LOG="smoke.serial"
> > +export BOOT_MSG="Latest ChangeSet: "
> > +export LOG_MSG="Domain-0"
> > +# exit early on test failure too, check if it was success below
> > +export PASSED="$passed|$failed"
> > +
> > +./automation/scripts/console.exp | sed 's/\r\+$//'
> > +
> > +grep "$passed" smoke.serial
> > -- 
> > git-series 0.9.1
> > 


-- 
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [PATCH v4 1/6] automation: switch to alpine:3.22

[Marek Marczykowski-Górecki]

[-- Attachment #1: Type: text/plain, Size: 1097 bytes --]

On Tue, Dec 09, 2025 at 03:07:46PM -0800, Stefano Stabellini wrote:
> On Sat, 6 Dec 2025, Marek Marczykowski-Górecki wrote:
> > Alpine 3.18 is EOL. This patch updates Alpine to the latest stable
> > Alpine release - 3.22.
> > 
> > Signed-off-by: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com>
> > ---
> > Changes in v4:
> > - fix containerize script
> > Changes in v2:
> > Originally this patch was updating to 3.19 for upcoming stubdomain tests
> > to get dracut-core package. But 3.19 is getting EOL very soon too.
> 
> This one escaped:
> 
> alpine-3.18-gcc-debug-arm64-mpu:
>   extends: .gcc-arm64-build-debug
>   variables:
>     CONTAINER: alpine:3.18-arm64v8
>     HYPERVISOR_ONLY: y
>     EXTRA_XEN_CONFIG: |
>       CONFIG_XEN_START_ADDRESS=0x0
>       CONFIG_MPU=y
>       CONFIG_UNSUPPORTED=y
> 
> was it on purpose?

Right, I guess it was added after I made previous version of this
patch...

> You also might want to rename 3.18 to3.22 in docs/misc/ci.rst

Yes.

-- 
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [PATCH v4 2/6] CI: extract qubes test container

[Marek Marczykowski-Górecki]

[-- Attachment #1: Type: text/plain, Size: 747 bytes --]

On Tue, Dec 09, 2025 at 03:12:06PM -0800, Stefano Stabellini wrote:
> On Sat, 6 Dec 2025, Marek Marczykowski-Górecki wrote:
> > -  \
> > -  # qemu build deps
> > -  glib-dev \
> > -  libattr \
> > -  libcap-ng-dev \
> > -  pixman-dev \
> 
> it would be good to mention in the commit message that the QEMU
> build dependency packages are not needed anymore

TBH, I'm not sure why they were here in the first place, none of the
build jobs for ARM64 build QEMU, and commit adding those initially (for
Alpine 3.12) doesn't tell either. I guess it's copy&paste from x86 one?

> anyway:
> 
> Reviewed-by: Stefano Stabellini <sstabellini@kernel.org>

Thanks.

-- 
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [PATCH v4 4/6] CI: Add driver domains tests

[Stefano Stabellini]

[-- Attachment #1: Type: text/plain, Size: 10253 bytes --]

On Wed, 10 Dec 2025, Marek Marczykowski-Górecki wrote:
> On Tue, Dec 09, 2025 at 04:02:06PM -0800, Stefano Stabellini wrote:
> > On Sat, 6 Dec 2025, Marek Marczykowski-Górecki wrote:
> > > Setup a simple two domU system. One with network backend, running
> > > xendriverdomain service, and one with frontend, trying to ping the
> > > backend.
> > > 
> > > Contrary to other similar tests, use disk image instead of initrd, to
> > > allow bigger rootfs without adding more RAM (for both dom0 and domU).
> > > But keep using pxelinux as a bootloader as it's easier to setup than
> > > installing grub on the disk. Theoretically, it could be started via direct
> > > kernel boot in QEMU, but pxelinux is slightly closer to real-world
> > > deployment.
> > > 
> > > Use fakeroot to preserve file owners/permissions. This is especially
> > > important for suid binaries like /bin/mount - without fakeroot, they
> > > will end up as suid into non-root user.
> > > 
> > > Signed-off-by: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com>
> > > ---
> > > Changes in v3:
> > > - add fakeroot
> > > - run ldconfig at the disk image creation time, to avoid running it at
> > >   dom0/domU boot time (which is much slower)
> > > Changes in v2:
> > > - use heredoc
> > > - limit ping loop iterations
> > > - use full "backend" / "frontend" in disk image names
> > > - print domU consoles directly to /dev/console, to avoid systemd-added
> > >   messages prefix
> > > - terminate test on failure, don't wait for timeout
> > > ---
> > >  automation/build/debian/13-x86_64.dockerfile    |   2 +-
> > >  automation/gitlab-ci/test.yaml                  |   8 +-
> > >  automation/scripts/qemu-driverdomains-x86_64.sh | 138 +++++++++++++++++-
> > >  3 files changed, 148 insertions(+)
> > >  create mode 100755 automation/scripts/qemu-driverdomains-x86_64.sh
> > > 
> > > diff --git a/automation/build/debian/13-x86_64.dockerfile b/automation/build/debian/13-x86_64.dockerfile
> > > index 2c6c9d4a5098..6382bafbd5bd 100644
> > > --- a/automation/build/debian/13-x86_64.dockerfile
> > > +++ b/automation/build/debian/13-x86_64.dockerfile
> > > @@ -55,7 +55,9 @@ RUN <<EOF
> > >  
> > >          # for test phase, qemu-* jobs
> > >          busybox-static
> > > +        e2fsprogs
> > >          expect
> > > +        fakeroot
> > >          ovmf
> > >          qemu-system-x86
> > >  
> > > diff --git a/automation/gitlab-ci/test.yaml b/automation/gitlab-ci/test.yaml
> > > index 7b36f1e126ca..abc5339a74ab 100644
> > > --- a/automation/gitlab-ci/test.yaml
> > > +++ b/automation/gitlab-ci/test.yaml
> > > @@ -656,6 +656,14 @@ qemu-alpine-x86_64-gcc:
> > >      - *x86-64-test-needs
> > >      - alpine-3.22-gcc
> > >  
> > > +qemu-alpine-driverdomains-x86_64-gcc:
> > > +  extends: .qemu-x86-64
> > > +  script:
> > > +    - ./automation/scripts/qemu-driverdomains-x86_64.sh 2>&1 | tee ${LOGFILE}
> > > +  needs:
> > > +    - *x86-64-test-needs
> > > +    - alpine-3.22-gcc
> > > +
> > >  qemu-smoke-x86-64-gcc:
> > >    extends: .qemu-smoke-x86-64
> > >    script:
> > > diff --git a/automation/scripts/qemu-driverdomains-x86_64.sh b/automation/scripts/qemu-driverdomains-x86_64.sh
> > > new file mode 100755
> > > index 000000000000..c0241da54168
> > > --- /dev/null
> > > +++ b/automation/scripts/qemu-driverdomains-x86_64.sh
> > > @@ -0,0 +1,138 @@
> > > +#!/bin/bash
> > > +
> > > +set -ex -o pipefail
> > > +
> > > +dom0_rootfs_extra_comp=()
> > > +dom0_rootfs_extra_uncomp=()
> > > +
> > > +cd binaries
> > > +
> > > +# DomU rootfs
> > > +
> > > +mkdir -p rootfs
> > > +cd rootfs
> > > +mkdir -p etc/local.d
> > > +passed="ping test passed"
> > > +failed="TEST FAILED"
> > > +cat > etc/local.d/xen.start << EOF
> > > +#!/bin/bash
> > > +
> > > +set -x
> > > +
> > > +if grep -q test=backend /proc/cmdline; then
> > > +    brctl addbr xenbr0
> > > +    ip link set xenbr0 up
> > > +    ip addr add 192.168.0.1/24 dev xenbr0
> > > +    bash /etc/init.d/xendriverdomain start
> > > +    # log backend-related logs to the console
> > > +    tail -F /var/log/xen/xldevd.log /var/log/xen/xen-hotplug.log >>/dev/console 2>/dev/null &
> > > +else
> > > +    ip link set eth0 up
> > > +    ip addr add 192.168.0.2/24 dev eth0
> > > +    timeout=6 # 6*10s
> > > +    until ping -c 10 192.168.0.1; do
> > > +        sleep 1
> > > +        if [ \$timeout -le 0 ]; then
> > > +            echo "${failed}"
> > > +            exit 1
> > > +        fi
> > > +        ((timeout--))
> > > +    done
> > > +    echo "${passed}"
> > > +fi
> > > +EOF
> > > +chmod +x etc/local.d/xen.start
> > > +fakeroot sh -c "
> > > +    zcat ../rootfs.cpio.gz | cpio -imd
> > > +    zcat ../xen-tools.cpio.gz | cpio -imd
> > > +    ldconfig -r .
> > > +    touch etc/.updated
> > > +    mkfs.ext4 -d . ../domU-rootfs.img 1024M
> > 
> > Do we really need 1GB? I would rather use a smaller size if possible.
> > I would rather use as little resources as possible on the build server
> > as we might run a few of these jobs in parallel one day soon.
> 
> This will be a sparse file, so it won't use really all the space. But
> this size is the upper bound of what can be put inside.
> That said, it's worth checking if sparse files do work properly on all
> runners in /build. AFAIR some older docker versions had issues with that
> (was it aufs not supporting sparse files?).

I ran the same command on my local baremetal Ubuntu dev environment
(arm64) and it created a new file of the size passed on the command
line (1GB in this case). It looks like they are not sparse on my end. If
the result depends on versions and configurations, I would rather err on
the side of caution and use the smallest possible number that works.


> > Moreover this script will be run inside a container which means this
> > data is probably in RAM.
> 
> Are runners configured to use tmpfs for /build? I don't think it's the
> default.

I don't know for sure, they are just using the default. My goal was to
make our solution more reliable as defaults and configurations might
change.


> > The underlying rootfs is 25M on both ARM and x86. This should be at most
> > 50M.
> 
> Rootfs itself is small, but for driver domains it needs to include
> toolstack too, and xen-tools.cpio is over 600MB (for debug build).
> I might be able to pick just the parts needed for the driver domain (xl
> with its deps, maybe some startup scripts, probably few more files), but
> it's rather fragile.

My first thought is to avoid creating a 1GB file in all cases when it
might only be needed for certain individual tests. Now, I realize that
this script might end up only used in driver domains tests but if not, I
would say to use the smallest number depending on the tests, especially
as there seems to be use a huge difference, e.g. 25MB versus 600MB.

My second thought is that 600MB for just the Xen tools is way too large.
I have alpine linux rootfs'es with just the Xen tools installed that are
below 50MB total. I am confused on how we get to 600MB. It might be due
to QEMU and its dependencies but still going from 25MB to 600MB is
incredible!


> > > +"
> > > +cd ..
> > > +rm -rf rootfs
> > > +
> > > +# Dom0 rootfs
> > > +mkdir -p rootfs
> > > +cd rootfs
> > > +fakeroot -s ../fakeroot-save sh -c "
> > > +    zcat ../rootfs.cpio.gz | cpio -imd
> > > +    zcat ../xen-tools.cpio.gz | cpio -imd
> > > +    ldconfig -r .
> > > +    touch etc/.updated
> > > +"
> > > +mkdir -p root etc/local.d
> > > +cat > root/backend.cfg << EOF
> > > +name="backend"
> > > +memory=512
> > > +vcpus=1
> > > +kernel="/root/bzImage"
> > > +extra="console=hvc0 root=/dev/xvda net.ifnames=0 test=backend"
> > > +disk=[ '/root/domU-rootfs-backend.img,raw,xvda,rw' ]
> > > +EOF
> > > +cat > root/frontend.cfg << EOF
> > > +name="frontend"
> > > +memory=512
> > > +vcpus=1
> > > +kernel="/root/bzImage"
> > > +extra="console=hvc0 root=/dev/xvda net.ifnames=0 test=frontend"
> > > +disk=[ '/root/domU-rootfs-frontend.img,raw,xvda,rw' ]
> > > +vif=[ 'bridge=xenbr0,backend=backend' ]
> > > +EOF
> > > +
> > > +cat > etc/local.d/xen.start << EOF
> > > +#!/bin/bash
> > > +
> > > +set -x
> > > +
> > > +bash /etc/init.d/xencommons start
> > > +
> > > +xl list
> > > +
> > > +tail -F /var/log/xen/console/guest-backend.log 2>/dev/null | sed -e "s/^/(backend) /" >>/dev/console &
> > > +tail -F /var/log/xen/console/guest-frontend.log 2>/dev/null | sed -e "s/^/(frontend) /" >>/dev/console &
> > > +xl -vvv create /root/backend.cfg
> > > +xl -vvv create /root/frontend.cfg
> > > +EOF
> > > +chmod +x etc/local.d/xen.start
> > > +
> > > +cp ../domU-rootfs.img ./root/domU-rootfs-backend.img
> > > +cp ../domU-rootfs.img ./root/domU-rootfs-frontend.img
> > > +cp ../bzImage ./root/
> > > +mkdir -p etc/default
> > > +echo 'XENCONSOLED_TRACE=all' >> etc/default/xencommons
> > > +mkdir -p var/log/xen/console
> > > +fakeroot -i ../fakeroot-save mkfs.ext4 -d . ../dom0-rootfs.img 2048M
> > 
> > Same here. Also 2GB might not be sufficient to contain 2 copies of
> > domU-rootfs.img, given that domU-rootfs.img is 1GB.
> 
> See the note about sparse files.

I double checked and they don't appear to be sparse on my system.


> > If we bring down domU-rootfs.img to 50M, then this could be 150M.
> > 
> > 
> > > +cd ..
> > > +rm -rf rootfs
> > > +
> > > +cd ..
> > > +
> > > +cat >> binaries/pxelinux.0 << EOF
> > > +#!ipxe
> > > +
> > > +kernel xen console=com1 console_timestamps=boot
> > > +module bzImage console=hvc0 root=/dev/sda net.ifnames=0
> > > +boot
> > > +EOF
> > > +
> > > +# Run the test
> > > +rm -f smoke.serial
> > > +export TEST_CMD="qemu-system-x86_64 \
> > > +    -cpu qemu64,+svm \
> > > +    -m 2G -smp 2 \
> > > +    -monitor none -serial stdio \
> > > +    -nographic \
> > > +    -device virtio-net-pci,netdev=n0 \
> > > +    -netdev user,id=n0,tftp=binaries,bootfile=/pxelinux.0 \
> > > +    -drive file=binaries/dom0-rootfs.img,format=raw"
> > > +
> > > +export TEST_LOG="smoke.serial"
> > > +export BOOT_MSG="Latest ChangeSet: "
> > > +export LOG_MSG="Domain-0"
> > > +# exit early on test failure too, check if it was success below
> > > +export PASSED="$passed|$failed"
> > > +
> > > +./automation/scripts/console.exp | sed 's/\r\+$//'
> > > +
> > > +grep "$passed" smoke.serial
> > > -- 
> > > git-series 0.9.1

Re: [PATCH v4 4/6] CI: Add driver domains tests

[Marek Marczykowski-Górecki]

[-- Attachment #1: Type: text/plain, Size: 9053 bytes --]

On Wed, Dec 10, 2025 at 12:28:19PM -0800, Stefano Stabellini wrote:
> On Wed, 10 Dec 2025, Marek Marczykowski-Górecki wrote:
> > On Tue, Dec 09, 2025 at 04:02:06PM -0800, Stefano Stabellini wrote:
> > > On Sat, 6 Dec 2025, Marek Marczykowski-Górecki wrote:
> > > > Setup a simple two domU system. One with network backend, running
> > > > xendriverdomain service, and one with frontend, trying to ping the
> > > > backend.
> > > > 
> > > > Contrary to other similar tests, use disk image instead of initrd, to
> > > > allow bigger rootfs without adding more RAM (for both dom0 and domU).
> > > > But keep using pxelinux as a bootloader as it's easier to setup than
> > > > installing grub on the disk. Theoretically, it could be started via direct
> > > > kernel boot in QEMU, but pxelinux is slightly closer to real-world
> > > > deployment.
> > > > 
> > > > Use fakeroot to preserve file owners/permissions. This is especially
> > > > important for suid binaries like /bin/mount - without fakeroot, they
> > > > will end up as suid into non-root user.
> > > > 
> > > > Signed-off-by: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com>
> > > > ---
> > > > Changes in v3:
> > > > - add fakeroot
> > > > - run ldconfig at the disk image creation time, to avoid running it at
> > > >   dom0/domU boot time (which is much slower)
> > > > Changes in v2:
> > > > - use heredoc
> > > > - limit ping loop iterations
> > > > - use full "backend" / "frontend" in disk image names
> > > > - print domU consoles directly to /dev/console, to avoid systemd-added
> > > >   messages prefix
> > > > - terminate test on failure, don't wait for timeout
> > > > ---
> > > >  automation/build/debian/13-x86_64.dockerfile    |   2 +-
> > > >  automation/gitlab-ci/test.yaml                  |   8 +-
> > > >  automation/scripts/qemu-driverdomains-x86_64.sh | 138 +++++++++++++++++-
> > > >  3 files changed, 148 insertions(+)
> > > >  create mode 100755 automation/scripts/qemu-driverdomains-x86_64.sh
> > > > 
> > > > diff --git a/automation/build/debian/13-x86_64.dockerfile b/automation/build/debian/13-x86_64.dockerfile
> > > > index 2c6c9d4a5098..6382bafbd5bd 100644
> > > > --- a/automation/build/debian/13-x86_64.dockerfile
> > > > +++ b/automation/build/debian/13-x86_64.dockerfile
> > > > @@ -55,7 +55,9 @@ RUN <<EOF
> > > >  
> > > >          # for test phase, qemu-* jobs
> > > >          busybox-static
> > > > +        e2fsprogs
> > > >          expect
> > > > +        fakeroot
> > > >          ovmf
> > > >          qemu-system-x86
> > > >  
> > > > diff --git a/automation/gitlab-ci/test.yaml b/automation/gitlab-ci/test.yaml
> > > > index 7b36f1e126ca..abc5339a74ab 100644
> > > > --- a/automation/gitlab-ci/test.yaml
> > > > +++ b/automation/gitlab-ci/test.yaml
> > > > @@ -656,6 +656,14 @@ qemu-alpine-x86_64-gcc:
> > > >      - *x86-64-test-needs
> > > >      - alpine-3.22-gcc
> > > >  
> > > > +qemu-alpine-driverdomains-x86_64-gcc:
> > > > +  extends: .qemu-x86-64
> > > > +  script:
> > > > +    - ./automation/scripts/qemu-driverdomains-x86_64.sh 2>&1 | tee ${LOGFILE}
> > > > +  needs:
> > > > +    - *x86-64-test-needs
> > > > +    - alpine-3.22-gcc
> > > > +
> > > >  qemu-smoke-x86-64-gcc:
> > > >    extends: .qemu-smoke-x86-64
> > > >    script:
> > > > diff --git a/automation/scripts/qemu-driverdomains-x86_64.sh b/automation/scripts/qemu-driverdomains-x86_64.sh
> > > > new file mode 100755
> > > > index 000000000000..c0241da54168
> > > > --- /dev/null
> > > > +++ b/automation/scripts/qemu-driverdomains-x86_64.sh
> > > > @@ -0,0 +1,138 @@
> > > > +#!/bin/bash
> > > > +
> > > > +set -ex -o pipefail
> > > > +
> > > > +dom0_rootfs_extra_comp=()
> > > > +dom0_rootfs_extra_uncomp=()
> > > > +
> > > > +cd binaries
> > > > +
> > > > +# DomU rootfs
> > > > +
> > > > +mkdir -p rootfs
> > > > +cd rootfs
> > > > +mkdir -p etc/local.d
> > > > +passed="ping test passed"
> > > > +failed="TEST FAILED"
> > > > +cat > etc/local.d/xen.start << EOF
> > > > +#!/bin/bash
> > > > +
> > > > +set -x
> > > > +
> > > > +if grep -q test=backend /proc/cmdline; then
> > > > +    brctl addbr xenbr0
> > > > +    ip link set xenbr0 up
> > > > +    ip addr add 192.168.0.1/24 dev xenbr0
> > > > +    bash /etc/init.d/xendriverdomain start
> > > > +    # log backend-related logs to the console
> > > > +    tail -F /var/log/xen/xldevd.log /var/log/xen/xen-hotplug.log >>/dev/console 2>/dev/null &
> > > > +else
> > > > +    ip link set eth0 up
> > > > +    ip addr add 192.168.0.2/24 dev eth0
> > > > +    timeout=6 # 6*10s
> > > > +    until ping -c 10 192.168.0.1; do
> > > > +        sleep 1
> > > > +        if [ \$timeout -le 0 ]; then
> > > > +            echo "${failed}"
> > > > +            exit 1
> > > > +        fi
> > > > +        ((timeout--))
> > > > +    done
> > > > +    echo "${passed}"
> > > > +fi
> > > > +EOF
> > > > +chmod +x etc/local.d/xen.start
> > > > +fakeroot sh -c "
> > > > +    zcat ../rootfs.cpio.gz | cpio -imd
> > > > +    zcat ../xen-tools.cpio.gz | cpio -imd
> > > > +    ldconfig -r .
> > > > +    touch etc/.updated
> > > > +    mkfs.ext4 -d . ../domU-rootfs.img 1024M
> > > 
> > > Do we really need 1GB? I would rather use a smaller size if possible.
> > > I would rather use as little resources as possible on the build server
> > > as we might run a few of these jobs in parallel one day soon.
> > 
> > This will be a sparse file, so it won't use really all the space. But
> > this size is the upper bound of what can be put inside.
> > That said, it's worth checking if sparse files do work properly on all
> > runners in /build. AFAIR some older docker versions had issues with that
> > (was it aufs not supporting sparse files?).
> 
> I ran the same command on my local baremetal Ubuntu dev environment
> (arm64) and it created a new file of the size passed on the command
> line (1GB in this case). It looks like they are not sparse on my end. If
> the result depends on versions and configurations, I would rather err on
> the side of caution and use the smallest possible number that works.

Hm, interesting. What filesystem is that on?

On my side it's definitely sparse (ext4):

    [user@disp8129 Downloads]$ du -sch
    12K	.
    12K	total
    [user@disp8129 Downloads]$ mkfs.ext4 -d . ../domU-rootfs.img 1024M
    mke2fs 1.47.2 (1-Jan-2025)
    Creating regular file ../domU-rootfs.img
    Creating filesystem with 262144 4k blocks and 65536 inodes
    Filesystem UUID: f50a5dfe-4dcf-4f3e-82d0-3dc54a788ab0
    Superblock backups stored on blocks: 
        32768, 98304, 163840, 229376

    Allocating group tables: done                            
    Writing inode tables: done                            
    Creating journal (8192 blocks): done
    Copying files into the device: done
    Writing superblocks and filesystem accounting information: done

    [user@disp8129 Downloads]$ ls -lhs ../domU-rootfs.img 
    33M -rw-r--r--. 1 user user 1.0G Dec 10 21:45 ../domU-rootfs.img

> > > Moreover this script will be run inside a container which means this
> > > data is probably in RAM.
> > 
> > Are runners configured to use tmpfs for /build? I don't think it's the
> > default.
> 
> I don't know for sure, they are just using the default. My goal was to
> make our solution more reliable as defaults and configurations might
> change.
> 
> 
> > > The underlying rootfs is 25M on both ARM and x86. This should be at most
> > > 50M.
> > 
> > Rootfs itself is small, but for driver domains it needs to include
> > toolstack too, and xen-tools.cpio is over 600MB (for debug build).
> > I might be able to pick just the parts needed for the driver domain (xl
> > with its deps, maybe some startup scripts, probably few more files), but
> > it's rather fragile.
> 
> My first thought is to avoid creating a 1GB file in all cases when it
> might only be needed for certain individual tests. Now, I realize that
> this script might end up only used in driver domains tests but if not, 

Indeed this script is specifically about driverdomains test.

> I
> would say to use the smallest number depending on the tests, especially
> as there seems to be use a huge difference, e.g. 25MB versus 600MB.
> 
> My second thought is that 600MB for just the Xen tools is way too large.
> I have alpine linux rootfs'es with just the Xen tools installed that are
> below 50MB total. I am confused on how we get to 600MB. It might be due
> to QEMU and its dependencies but still going from 25MB to 600MB is
> incredible!

Indeed it's mostly about QEMU (its main binary itself takes 55MB),
including all bundled firmwares etc (various flavors of edk2 alone take
270MB). There is also usr/lib/debug which takes 85MB.
But then, usr/lib/libxen* combined takes almost 50MB.

OTOH, non-debug xen-tools.cpio takes "just" 130MB.

-- 
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [PATCH v4 4/6] CI: Add driver domains tests

[Stefano Stabellini]

[-- Attachment #1: Type: text/plain, Size: 4626 bytes --]

On Wed, 10 Dec 2025, Marek Marczykowski-Górecki wrote:
> > > > > +    mkfs.ext4 -d . ../domU-rootfs.img 1024M
> > > > 
> > > > Do we really need 1GB? I would rather use a smaller size if possible.
> > > > I would rather use as little resources as possible on the build server
> > > > as we might run a few of these jobs in parallel one day soon.
> > > 
> > > This will be a sparse file, so it won't use really all the space. But
> > > this size is the upper bound of what can be put inside.
> > > That said, it's worth checking if sparse files do work properly on all
> > > runners in /build. AFAIR some older docker versions had issues with that
> > > (was it aufs not supporting sparse files?).
> > 
> > I ran the same command on my local baremetal Ubuntu dev environment
> > (arm64) and it created a new file of the size passed on the command
> > line (1GB in this case). It looks like they are not sparse on my end. If
> > the result depends on versions and configurations, I would rather err on
> > the side of caution and use the smallest possible number that works.
> 
> Hm, interesting. What filesystem is that on?
> 
> On my side it's definitely sparse (ext4):
> 
>     [user@disp8129 Downloads]$ du -sch
>     12K	.
>     12K	total
>     [user@disp8129 Downloads]$ mkfs.ext4 -d . ../domU-rootfs.img 1024M
>     mke2fs 1.47.2 (1-Jan-2025)
>     Creating regular file ../domU-rootfs.img
>     Creating filesystem with 262144 4k blocks and 65536 inodes
>     Filesystem UUID: f50a5dfe-4dcf-4f3e-82d0-3dc54a788ab0
>     Superblock backups stored on blocks: 
>         32768, 98304, 163840, 229376
> 
>     Allocating group tables: done                            
>     Writing inode tables: done                            
>     Creating journal (8192 blocks): done
>     Copying files into the device: done
>     Writing superblocks and filesystem accounting information: done
> 
>     [user@disp8129 Downloads]$ ls -lhs ../domU-rootfs.img 
>     33M -rw-r--r--. 1 user user 1.0G Dec 10 21:45 ../domU-rootfs.img

I went and check two of the runners, one ARM and one x86, and it looks
like they support sparse outside and inside containers. They should have
all the same configuration so I think we can assume they support sparse
files appropriately.

So it looks like it OK. But please could you add an in-code comment to
highlight that the file created will be sparse?


> > > > Moreover this script will be run inside a container which means this
> > > > data is probably in RAM.
> > > 
> > > Are runners configured to use tmpfs for /build? I don't think it's the
> > > default.
> > 
> > I don't know for sure, they are just using the default. My goal was to
> > make our solution more reliable as defaults and configurations might
> > change.
> > 
> > 
> > > > The underlying rootfs is 25M on both ARM and x86. This should be at most
> > > > 50M.
> > > 
> > > Rootfs itself is small, but for driver domains it needs to include
> > > toolstack too, and xen-tools.cpio is over 600MB (for debug build).
> > > I might be able to pick just the parts needed for the driver domain (xl
> > > with its deps, maybe some startup scripts, probably few more files), but
> > > it's rather fragile.
> > 
> > My first thought is to avoid creating a 1GB file in all cases when it
> > might only be needed for certain individual tests. Now, I realize that
> > this script might end up only used in driver domains tests but if not, 
> 
> Indeed this script is specifically about driverdomains test.
> 
> > I
> > would say to use the smallest number depending on the tests, especially
> > as there seems to be use a huge difference, e.g. 25MB versus 600MB.
> > 
> > My second thought is that 600MB for just the Xen tools is way too large.
> > I have alpine linux rootfs'es with just the Xen tools installed that are
> > below 50MB total. I am confused on how we get to 600MB. It might be due
> > to QEMU and its dependencies but still going from 25MB to 600MB is
> > incredible!
> 
> Indeed it's mostly about QEMU (its main binary itself takes 55MB),
> including all bundled firmwares etc (various flavors of edk2 alone take
> 270MB). There is also usr/lib/debug which takes 85MB.
> But then, usr/lib/libxen* combined takes almost 50MB.
> 
> OTOH, non-debug xen-tools.cpio takes "just" 130MB.

Can we use the non-debug xen-tools.cpio and also can we remove all the
bundled firmware? Do we really need EDK2 for instance?

I don't think it is worth doing an in-details analysis of what binaries
to keep and what to remove, but at least removing the unnecessary
in-guest firmware and ideally chosing a non-debug build sounds
reasonable?

Re: [PATCH v4 4/6] CI: Add driver domains tests

[Marek Marczykowski-Górecki]

[-- Attachment #1: Type: text/plain, Size: 5609 bytes --]

On Wed, Dec 10, 2025 at 01:58:44PM -0800, Stefano Stabellini wrote:
> On Wed, 10 Dec 2025, Marek Marczykowski-Górecki wrote:
> > > > > > +    mkfs.ext4 -d . ../domU-rootfs.img 1024M
> > > > > 
> > > > > Do we really need 1GB? I would rather use a smaller size if possible.
> > > > > I would rather use as little resources as possible on the build server
> > > > > as we might run a few of these jobs in parallel one day soon.
> > > > 
> > > > This will be a sparse file, so it won't use really all the space. But
> > > > this size is the upper bound of what can be put inside.
> > > > That said, it's worth checking if sparse files do work properly on all
> > > > runners in /build. AFAIR some older docker versions had issues with that
> > > > (was it aufs not supporting sparse files?).
> > > 
> > > I ran the same command on my local baremetal Ubuntu dev environment
> > > (arm64) and it created a new file of the size passed on the command
> > > line (1GB in this case). It looks like they are not sparse on my end. If
> > > the result depends on versions and configurations, I would rather err on
> > > the side of caution and use the smallest possible number that works.
> > 
> > Hm, interesting. What filesystem is that on?
> > 
> > On my side it's definitely sparse (ext4):
> > 
> >     [user@disp8129 Downloads]$ du -sch
> >     12K	.
> >     12K	total
> >     [user@disp8129 Downloads]$ mkfs.ext4 -d . ../domU-rootfs.img 1024M
> >     mke2fs 1.47.2 (1-Jan-2025)
> >     Creating regular file ../domU-rootfs.img
> >     Creating filesystem with 262144 4k blocks and 65536 inodes
> >     Filesystem UUID: f50a5dfe-4dcf-4f3e-82d0-3dc54a788ab0
> >     Superblock backups stored on blocks: 
> >         32768, 98304, 163840, 229376
> > 
> >     Allocating group tables: done                            
> >     Writing inode tables: done                            
> >     Creating journal (8192 blocks): done
> >     Copying files into the device: done
> >     Writing superblocks and filesystem accounting information: done
> > 
> >     [user@disp8129 Downloads]$ ls -lhs ../domU-rootfs.img 
> >     33M -rw-r--r--. 1 user user 1.0G Dec 10 21:45 ../domU-rootfs.img
> 
> I went and check two of the runners, one ARM and one x86, and it looks
> like they support sparse outside and inside containers. They should have
> all the same configuration so I think we can assume they support sparse
> files appropriately.
> 
> So it looks like it OK. But please could you add an in-code comment to
> highlight that the file created will be sparse?

Sure.

> > > > > Moreover this script will be run inside a container which means this
> > > > > data is probably in RAM.
> > > > 
> > > > Are runners configured to use tmpfs for /build? I don't think it's the
> > > > default.
> > > 
> > > I don't know for sure, they are just using the default. My goal was to
> > > make our solution more reliable as defaults and configurations might
> > > change.
> > > 
> > > 
> > > > > The underlying rootfs is 25M on both ARM and x86. This should be at most
> > > > > 50M.
> > > > 
> > > > Rootfs itself is small, but for driver domains it needs to include
> > > > toolstack too, and xen-tools.cpio is over 600MB (for debug build).
> > > > I might be able to pick just the parts needed for the driver domain (xl
> > > > with its deps, maybe some startup scripts, probably few more files), but
> > > > it's rather fragile.
> > > 
> > > My first thought is to avoid creating a 1GB file in all cases when it
> > > might only be needed for certain individual tests. Now, I realize that
> > > this script might end up only used in driver domains tests but if not, 
> > 
> > Indeed this script is specifically about driverdomains test.
> > 
> > > I
> > > would say to use the smallest number depending on the tests, especially
> > > as there seems to be use a huge difference, e.g. 25MB versus 600MB.
> > > 
> > > My second thought is that 600MB for just the Xen tools is way too large.
> > > I have alpine linux rootfs'es with just the Xen tools installed that are
> > > below 50MB total. I am confused on how we get to 600MB. It might be due
> > > to QEMU and its dependencies but still going from 25MB to 600MB is
> > > incredible!
> > 
> > Indeed it's mostly about QEMU (its main binary itself takes 55MB),
> > including all bundled firmwares etc (various flavors of edk2 alone take
> > 270MB). There is also usr/lib/debug which takes 85MB.
> > But then, usr/lib/libxen* combined takes almost 50MB.
> > 
> > OTOH, non-debug xen-tools.cpio takes "just" 130MB.
> 
> Can we use the non-debug xen-tools.cpio 

I can use non-debug one. While debug build of hypervisor changes quite a
lot in terms of test output details, the purpose of this test is mostly
to test toolstack and frontend drivers - and here debug build doesn't
change much.

> and also can we remove all the
> bundled firmware? Do we really need EDK2 for instance?
> 
> I don't think it is worth doing an in-details analysis of what binaries
> to keep and what to remove, but at least removing the unnecessary
> in-guest firmware and ideally chosing a non-debug build sounds
> reasonable?

Excluding QEMU _for now_ makes sense. But there might be a day when we'd
like to test QEMU backends in a driver domain and/or a domU booted via
UEFI (IIUC such configuration has PV frontend in EDK2 - at least for the
disk - and it makes sense testing if it works with driver domains).

-- 
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [PATCH v4 4/6] CI: Add driver domains tests

[Stefano Stabellini]

[-- Attachment #1: Type: text/plain, Size: 5952 bytes --]

On Thu, 11 Dec 2025, Marek Marczykowski-Górecki wrote:
> On Wed, Dec 10, 2025 at 01:58:44PM -0800, Stefano Stabellini wrote:
> > On Wed, 10 Dec 2025, Marek Marczykowski-Górecki wrote:
> > > > > > > +    mkfs.ext4 -d . ../domU-rootfs.img 1024M
> > > > > > 
> > > > > > Do we really need 1GB? I would rather use a smaller size if possible.
> > > > > > I would rather use as little resources as possible on the build server
> > > > > > as we might run a few of these jobs in parallel one day soon.
> > > > > 
> > > > > This will be a sparse file, so it won't use really all the space. But
> > > > > this size is the upper bound of what can be put inside.
> > > > > That said, it's worth checking if sparse files do work properly on all
> > > > > runners in /build. AFAIR some older docker versions had issues with that
> > > > > (was it aufs not supporting sparse files?).
> > > > 
> > > > I ran the same command on my local baremetal Ubuntu dev environment
> > > > (arm64) and it created a new file of the size passed on the command
> > > > line (1GB in this case). It looks like they are not sparse on my end. If
> > > > the result depends on versions and configurations, I would rather err on
> > > > the side of caution and use the smallest possible number that works.
> > > 
> > > Hm, interesting. What filesystem is that on?
> > > 
> > > On my side it's definitely sparse (ext4):
> > > 
> > >     [user@disp8129 Downloads]$ du -sch
> > >     12K	.
> > >     12K	total
> > >     [user@disp8129 Downloads]$ mkfs.ext4 -d . ../domU-rootfs.img 1024M
> > >     mke2fs 1.47.2 (1-Jan-2025)
> > >     Creating regular file ../domU-rootfs.img
> > >     Creating filesystem with 262144 4k blocks and 65536 inodes
> > >     Filesystem UUID: f50a5dfe-4dcf-4f3e-82d0-3dc54a788ab0
> > >     Superblock backups stored on blocks: 
> > >         32768, 98304, 163840, 229376
> > > 
> > >     Allocating group tables: done                            
> > >     Writing inode tables: done                            
> > >     Creating journal (8192 blocks): done
> > >     Copying files into the device: done
> > >     Writing superblocks and filesystem accounting information: done
> > > 
> > >     [user@disp8129 Downloads]$ ls -lhs ../domU-rootfs.img 
> > >     33M -rw-r--r--. 1 user user 1.0G Dec 10 21:45 ../domU-rootfs.img
> > 
> > I went and check two of the runners, one ARM and one x86, and it looks
> > like they support sparse outside and inside containers. They should have
> > all the same configuration so I think we can assume they support sparse
> > files appropriately.
> > 
> > So it looks like it OK. But please could you add an in-code comment to
> > highlight that the file created will be sparse?
> 
> Sure.
> 
> > > > > > Moreover this script will be run inside a container which means this
> > > > > > data is probably in RAM.
> > > > > 
> > > > > Are runners configured to use tmpfs for /build? I don't think it's the
> > > > > default.
> > > > 
> > > > I don't know for sure, they are just using the default. My goal was to
> > > > make our solution more reliable as defaults and configurations might
> > > > change.
> > > > 
> > > > 
> > > > > > The underlying rootfs is 25M on both ARM and x86. This should be at most
> > > > > > 50M.
> > > > > 
> > > > > Rootfs itself is small, but for driver domains it needs to include
> > > > > toolstack too, and xen-tools.cpio is over 600MB (for debug build).
> > > > > I might be able to pick just the parts needed for the driver domain (xl
> > > > > with its deps, maybe some startup scripts, probably few more files), but
> > > > > it's rather fragile.
> > > > 
> > > > My first thought is to avoid creating a 1GB file in all cases when it
> > > > might only be needed for certain individual tests. Now, I realize that
> > > > this script might end up only used in driver domains tests but if not, 
> > > 
> > > Indeed this script is specifically about driverdomains test.
> > > 
> > > > I
> > > > would say to use the smallest number depending on the tests, especially
> > > > as there seems to be use a huge difference, e.g. 25MB versus 600MB.
> > > > 
> > > > My second thought is that 600MB for just the Xen tools is way too large.
> > > > I have alpine linux rootfs'es with just the Xen tools installed that are
> > > > below 50MB total. I am confused on how we get to 600MB. It might be due
> > > > to QEMU and its dependencies but still going from 25MB to 600MB is
> > > > incredible!
> > > 
> > > Indeed it's mostly about QEMU (its main binary itself takes 55MB),
> > > including all bundled firmwares etc (various flavors of edk2 alone take
> > > 270MB). There is also usr/lib/debug which takes 85MB.
> > > But then, usr/lib/libxen* combined takes almost 50MB.
> > > 
> > > OTOH, non-debug xen-tools.cpio takes "just" 130MB.
> > 
> > Can we use the non-debug xen-tools.cpio 
> 
> I can use non-debug one. While debug build of hypervisor changes quite a
> lot in terms of test output details, the purpose of this test is mostly
> to test toolstack and frontend drivers - and here debug build doesn't
> change much.
> 
> > and also can we remove all the
> > bundled firmware? Do we really need EDK2 for instance?
> > 
> > I don't think it is worth doing an in-details analysis of what binaries
> > to keep and what to remove, but at least removing the unnecessary
> > in-guest firmware and ideally chosing a non-debug build sounds
> > reasonable?
> 
> Excluding QEMU _for now_ makes sense. But there might be a day when we'd
> like to test QEMU backends in a driver domain and/or a domU booted via
> UEFI (IIUC such configuration has PV frontend in EDK2 - at least for the
> disk - and it makes sense testing if it works with driver domains).

Ok, in that case, let's go with excluding QEMU and EDK2. While there
might be cases in the future where one or both are needed I don't think
is a good idea to increase the rootfs size for all tests including the
ones where they are not needed.