[PATCH v3 0/7] Several CI cleanups and improvements around initrd/rootfs

[PATCH v3 0/7] Several CI cleanups and improvements around initrd/rootfs

[Marek Marczykowski-Górecki]

Some of those patches are collected from my stubdomain test series, others are
made while getting the new runner working. This series initially included
adding Zen4 runner, but drop it in this version as there is quite enough stuff
here already, and there are few more dependencies for the new runner (fixing
some PVH issues, getting 6.12 kernel to build etc).

This series also evolved into refactoring initrd/rootfs handling and now
depends on test-artifacts changes posted by Andrew. This significantly reduces
hw test jobs time.

Marek Marczykowski-Górecki (7):
  CI: wait for Xen to start before waiting for test to complete
  CI: fix waiting for final test message
  CI: switch qubes runners to use console.exp
  CI: write whole etc/issue for domU initrd
  CI: avoid repacking initrd as part of the test job
  CI: remove now unused alpine-3.18-arm64-rootfs job and its container
  CI: save toolstack artifact as cpio.gz

 automation/build/alpine/3.18-arm64v8.dockerfile           |  1 +-
 automation/gitlab-ci/build.yaml                           | 11 +-
 automation/gitlab-ci/test.yaml                            |  6 +-
 automation/scripts/build                                  |  2 +-
 automation/scripts/console.exp                            | 40 +++-
 automation/scripts/qemu-alpine-x86_64.sh                  | 12 +-
 automation/scripts/qemu-smoke-dom0-arm32.sh               |  1 +-
 automation/scripts/qemu-smoke-dom0-arm64.sh               | 10 +-
 automation/scripts/qemu-smoke-dom0less-arm32.sh           |  1 +-
 automation/scripts/qemu-smoke-dom0less-arm64.sh           |  9 +-
 automation/scripts/qemu-xtf-dom0less-arm64.sh             |  1 +-
 automation/scripts/qubes-x86-64.sh                        | 79 ++------
 automation/scripts/xilinx-smoke-dom0-x86_64.sh            | 23 +-
 automation/scripts/xilinx-smoke-dom0less-arm64.sh         | 17 +--
 automation/tests-artifacts/alpine/3.18-arm64v8.dockerfile | 66 +-------
 15 files changed, 108 insertions(+), 171 deletions(-)
 delete mode 100644 automation/tests-artifacts/alpine/3.18-arm64v8.dockerfile

base-commit: 7bf777b42cade81d4f20f78cb85c4bc0f638de7b
-- 
git-series 0.9.1

[PATCH v3 1/7] CI: wait for Xen to start before waiting for test to complete

[Marek Marczykowski-Górecki]

Add additional stage in console output parsing - wait for first message
from Xen. The message is defined via BOOT_MSG variable. This has two
effects:
- distinguishes failing Xen to load at all from later test failures
- resets timeout when Xen starts loading

The latter is especially relevant for hardware tests where firmware +
network boot may take some time before Xen starts booting. The two-stage
timeout is more robust solution than increasing the overall timeout.
The issue has been observed on some dom0pvh-hvm jobs, at least on
runners hw3 and hw11. This patch is a first stage before qubes-x86-64.sh
is switched to use expect in the next stage.

While at it, consistently use 'expect -re' for all matches. This
especially allows matching newlines ("\n"), which will become relevant
in the next patch. And document variables used in console.exp.

Signed-off-by: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com>
---
Changes in v3:
- split off "CI: switch qubes runners to use console.exp"
- use BOOT_MSG in more tests (all using network boot)
---
 automation/scripts/console.exp                    | 27 +++++++++++++---
 automation/scripts/qemu-alpine-x86_64.sh          |  1 +-
 automation/scripts/qemu-smoke-dom0-arm32.sh       |  1 +-
 automation/scripts/qemu-smoke-dom0-arm64.sh       |  1 +-
 automation/scripts/qemu-smoke-dom0less-arm32.sh   |  1 +-
 automation/scripts/qemu-xtf-dom0less-arm64.sh     |  1 +-
 automation/scripts/xilinx-smoke-dom0-x86_64.sh    |  1 +-
 automation/scripts/xilinx-smoke-dom0less-arm64.sh |  1 +-
 8 files changed, 29 insertions(+), 5 deletions(-)

diff --git a/automation/scripts/console.exp b/automation/scripts/console.exp
index 31ce97b91b63..c27f893dfba7 100755
--- a/automation/scripts/console.exp
+++ b/automation/scripts/console.exp
@@ -1,4 +1,17 @@
 #!/usr/bin/env expect
+#
+# Variables used by this script:
+# - TEST_TIMEOUT: timeout between each *_MSG match
+# - TEST_TIMEOUT_OVERRIDE: when set, overrides TEST_TIMEOUT
+# - TEST_LOG: save console log to this file
+# - TEST_CMD: commands that prints test system console output to stdout - in
+#   qemu tests that's usually qemu itself (with -serial stdio), in hardware
+#   tests that's a command to read serial console
+# - UBOOT_CMD (optional): command to enter at u-boot prompt
+# - BOOT_MSG (optional): initial Xen message to wait for (aka sign-of-life)
+# - LOG_MSG (optional): final console message to wait for
+# - PASSED: message to look for to consider test a success; if LOG_MSG is set,
+#   both LOG_MSG and PASSED must appear (in any order) for test to succeed
 
 if {[info exists env(TEST_TIMEOUT_OVERRIDE)]} {
     set timeout $env(TEST_TIMEOUT_OVERRIDE)
@@ -28,21 +41,25 @@ if {[info exists env(UBOOT_CMD)]} {
     send "$env(UBOOT_CMD)\r"
 }
 
+if {[info exists env(BOOT_MSG)]} {
+    expect -re "$env(BOOT_MSG)"
+}
+
 if {[info exists env(LOG_MSG)]} {
     expect {
-        "$env(PASSED)" {
-            expect "$env(LOG_MSG)"
+        -re "$env(PASSED)" {
+            expect -re "$env(LOG_MSG)"
             exit 0
         }
-        "$env(LOG_MSG)" {
-            expect "$env(PASSED)"
+        -re "$env(LOG_MSG)" {
+            expect -re "$env(PASSED)"
             exit 0
         }
     }
 }
 
 expect {
-    "$env(PASSED)" {
+    -re "$env(PASSED)" {
         exit 0
     }
 }
diff --git a/automation/scripts/qemu-alpine-x86_64.sh b/automation/scripts/qemu-alpine-x86_64.sh
index 17e2141d625e..89bdb4df7dac 100755
--- a/automation/scripts/qemu-alpine-x86_64.sh
+++ b/automation/scripts/qemu-alpine-x86_64.sh
@@ -85,6 +85,7 @@ export TEST_CMD="qemu-system-x86_64 \
     -netdev user,id=n0,tftp=binaries,bootfile=/pxelinux.0"
 
 export TEST_LOG="smoke.serial"
+export BOOT_MSG="Latest ChangeSet: "
 export LOG_MSG="Domain-0"
 export PASSED="BusyBox"
 
diff --git a/automation/scripts/qemu-smoke-dom0-arm32.sh b/automation/scripts/qemu-smoke-dom0-arm32.sh
index 0c60a66e25e3..4f50eabdef53 100755
--- a/automation/scripts/qemu-smoke-dom0-arm32.sh
+++ b/automation/scripts/qemu-smoke-dom0-arm32.sh
@@ -92,6 +92,7 @@ export TEST_CMD="./qemu-system-arm \
 
 export UBOOT_CMD="virtio scan; dhcp; tftpb 0x40000000 boot.scr; source 0x40000000"
 export TEST_LOG="${serial_log}"
+export BOOT_MSG="Latest ChangeSet: "
 export LOG_MSG="Domain-0"
 export PASSED="/ #"
 
diff --git a/automation/scripts/qemu-smoke-dom0-arm64.sh b/automation/scripts/qemu-smoke-dom0-arm64.sh
index 8774a8701232..51d037b0003e 100755
--- a/automation/scripts/qemu-smoke-dom0-arm64.sh
+++ b/automation/scripts/qemu-smoke-dom0-arm64.sh
@@ -104,6 +104,7 @@ export TEST_CMD="./binaries/qemu-system-aarch64 \
     -bios /usr/lib/u-boot/qemu_arm64/u-boot.bin"
 
 export UBOOT_CMD="virtio scan; dhcp; tftpb 0x40000000 boot.scr; source 0x40000000"
+export BOOT_MSG="Latest ChangeSet: "
 export TEST_LOG="smoke.serial"
 export LOG_MSG="Domain-0"
 export PASSED="BusyBox"
diff --git a/automation/scripts/qemu-smoke-dom0less-arm32.sh b/automation/scripts/qemu-smoke-dom0less-arm32.sh
index 0c94e662aab9..0e2c5496db51 100755
--- a/automation/scripts/qemu-smoke-dom0less-arm32.sh
+++ b/automation/scripts/qemu-smoke-dom0less-arm32.sh
@@ -144,6 +144,7 @@ export TEST_CMD="./qemu-system-arm \
     -bios /usr/lib/u-boot/qemu_arm/u-boot.bin"
 
 export UBOOT_CMD="virtio scan; dhcp; tftpb 0x40000000 boot.scr; source 0x40000000"
+export BOOT_MSG="Latest ChangeSet: "
 export TEST_LOG="${serial_log}"
 export LOG_MSG="${dom0_prompt}"
 export PASSED="${passed}"
diff --git a/automation/scripts/qemu-xtf-dom0less-arm64.sh b/automation/scripts/qemu-xtf-dom0less-arm64.sh
index 9608de6ec033..436f460c3cb6 100755
--- a/automation/scripts/qemu-xtf-dom0less-arm64.sh
+++ b/automation/scripts/qemu-xtf-dom0less-arm64.sh
@@ -61,6 +61,7 @@ export TEST_CMD="./binaries/qemu-system-aarch64 \
     -bios /usr/lib/u-boot/qemu_arm64/u-boot.bin"
 
 export UBOOT_CMD="virtio scan; dhcp; tftpb 0x40000000 boot.scr; source 0x40000000"
+export BOOT_MSG="Latest ChangeSet: "
 export TEST_LOG="smoke.serial"
 export PASSED="${passed}"
 
diff --git a/automation/scripts/xilinx-smoke-dom0-x86_64.sh b/automation/scripts/xilinx-smoke-dom0-x86_64.sh
index 7834ffbe0593..69caabe2d8ed 100755
--- a/automation/scripts/xilinx-smoke-dom0-x86_64.sh
+++ b/automation/scripts/xilinx-smoke-dom0-x86_64.sh
@@ -159,6 +159,7 @@ stty -F ${SERIAL_DEV} 57600
 
 # Capture test result and power off board before exiting.
 export PASSED="${PASS_MSG}"
+export BOOT_MSG="Latest ChangeSet: "
 export TEST_CMD="cat ${SERIAL_DEV}"
 export TEST_LOG="smoke.serial"
 
diff --git a/automation/scripts/xilinx-smoke-dom0less-arm64.sh b/automation/scripts/xilinx-smoke-dom0less-arm64.sh
index b24ad11b8cac..3e1fcf6bf93c 100755
--- a/automation/scripts/xilinx-smoke-dom0less-arm64.sh
+++ b/automation/scripts/xilinx-smoke-dom0less-arm64.sh
@@ -140,6 +140,7 @@ stty -F ${SERIAL_DEV} 115200
 
 # Capture test result and power off board before exiting.
 export PASSED="${passed}"
+export BOOT_MSG="Latest ChangeSet: "
 export LOG_MSG="Welcome to Alpine Linux"
 export TEST_CMD="cat ${SERIAL_DEV}"
 export TEST_LOG="smoke.serial"
-- 
git-series 0.9.1

Re: [PATCH v3 1/7] CI: wait for Xen to start before waiting for test to complete

[Andrew Cooper]

On 11/04/2025 9:32 pm, Marek Marczykowski-Górecki wrote:
> Add additional stage in console output parsing - wait for first message
> from Xen. The message is defined via BOOT_MSG variable. This has two
> effects:
> - distinguishes failing Xen to load at all from later test failures
> - resets timeout when Xen starts loading
>
> The latter is especially relevant for hardware tests where firmware +
> network boot may take some time before Xen starts booting. The two-stage
> timeout is more robust solution than increasing the overall timeout.
> The issue has been observed on some dom0pvh-hvm jobs, at least on
> runners hw3 and hw11. This patch is a first stage before qubes-x86-64.sh
> is switched to use expect in the next stage.
>
> While at it, consistently use 'expect -re' for all matches. This
> especially allows matching newlines ("\n"), which will become relevant
> in the next patch. And document variables used in console.exp.
>
> Signed-off-by: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com>

Reviewed-by: Andrew Cooper <andrew.cooper3@citrix.com>

[PATCH v3 2/7] CI: fix waiting for final test message

[Marek Marczykowski-Górecki]

Expect normally discards initial part of its buffer after matching the
patter, before looking for the next one. If both PASSED and LOG_MSG
happen to be in the buffer at the same time, depending on their order,
only one will be matched and the waiting for the other will timeout.
Example expect -d output of this happening (parts eclipsed for brevity):

    expect: does "\r\r\r\nWelcome to Alpine Linux 3.18\r\r\r\n...\r\r\r\r\n(domU) + echo 'pci test passed'\r\r\r\r\n(domU) pci test passed\r\r\r\r..." (spawn_id exp4) match regular expression "pci test passed"? Gate "pci test passed"? gate=yes re=yes
    ...
    Gate keeper glob pattern for '\nWelcome to Alpine Linux' is '
    Welcome to Alpine Linux'. Activating booster.
    expect: does "'\r\r\r\r\n(domU) pci test passed\r\r\r\r\n(domU)  [ ok ]\r\r\r\r\n(domU)  [ ok ]\r\r\r\r\n(domU) \r\r\r\r\r\n(domU) domU Welcome to Alpine Linux 3.18\r\r\r\r\n(domU) \rKernel 6.6.56 on an x86_64 (/dev/hvc0)\r\r\r\r\n(domU) \r\r\r\r\r\n" (spawn_id exp4) match regular expression "\nWelcome to Alpine Linux"? Gate "\nWelcome to Alpine Linux"? gate=no

Fix this by using -notransfer flag to keep matched part in the buffer.

Signed-off-by: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com>
---
 automation/scripts/console.exp | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/automation/scripts/console.exp b/automation/scripts/console.exp
index c27f893dfba7..834a08db1b95 100755
--- a/automation/scripts/console.exp
+++ b/automation/scripts/console.exp
@@ -48,11 +48,11 @@ if {[info exists env(BOOT_MSG)]} {
 if {[info exists env(LOG_MSG)]} {
     expect {
         -re "$env(PASSED)" {
-            expect -re "$env(LOG_MSG)"
+            expect -notransfer -re "$env(LOG_MSG)"
             exit 0
         }
         -re "$env(LOG_MSG)" {
-            expect -re "$env(PASSED)"
+            expect -notransfer -re "$env(PASSED)"
             exit 0
         }
     }
-- 
git-series 0.9.1

Re: [PATCH v3 2/7] CI: fix waiting for final test message

[Andrew Cooper]

On 11/04/2025 9:32 pm, Marek Marczykowski-Górecki wrote:
> Expect normally discards initial part of its buffer after matching the
> patter, before looking for the next one. If both PASSED and LOG_MSG
> happen to be in the buffer at the same time, depending on their order,
> only one will be matched and the waiting for the other will timeout.
> Example expect -d output of this happening (parts eclipsed for brevity):
>
>     expect: does "\r\r\r\nWelcome to Alpine Linux 3.18\r\r\r\n...\r\r\r\r\n(domU) + echo 'pci test passed'\r\r\r\r\n(domU) pci test passed\r\r\r\r..." (spawn_id exp4) match regular expression "pci test passed"? Gate "pci test passed"? gate=yes re=yes
>     ...
>     Gate keeper glob pattern for '\nWelcome to Alpine Linux' is '
>     Welcome to Alpine Linux'. Activating booster.
>     expect: does "'\r\r\r\r\n(domU) pci test passed\r\r\r\r\n(domU)  [ ok ]\r\r\r\r\n(domU)  [ ok ]\r\r\r\r\n(domU) \r\r\r\r\r\n(domU) domU Welcome to Alpine Linux 3.18\r\r\r\r\n(domU) \rKernel 6.6.56 on an x86_64 (/dev/hvc0)\r\r\r\r\n(domU) \r\r\r\r\r\n" (spawn_id exp4) match regular expression "\nWelcome to Alpine Linux"? Gate "\nWelcome to Alpine Linux"? gate=no
>
> Fix this by using -notransfer flag to keep matched part in the buffer.
>
> Signed-off-by: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com>

Acked-by: Andrew Cooper <andrew.cooper3@citix.com>

[PATCH v3 3/7] CI: switch qubes runners to use console.exp

[Marek Marczykowski-Górecki]

It appears as sometimes it takes more time for Xen even start booting,
mostly due to firmware and fetching large boot files by grub. In some
jobs the current timeout is pretty close to the actual time needed, and
sometimes (rarely for now) test fails due to timeout expiring in the
middle of dom0 booting. This will be happening more often if the
initramfs will grow (and with more complex tests).
This has been observed on some dom0pvh-hvm jobs, at least on runners hw3
and hw11.

Switch to using expect (console.exp) for more robust test output
handling. This allows waiting separately for Xen starting to boot and
then for the test to complete. For now, set both of those to 120s, which
pessimistically bumps timeout for the whole test to 240s (from 120s).

Add S3 handling to console.exp via SUSPEND_MSG + WAKEUP_CMD.

Signed-off-by: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com>
---
Changes in v3:
- split into two patches - generic change is in the previous one
Changes in v2:
- replace previous "ci: increase timeout for hw tests" with changing how
  console is interacted with

This needs a containers rebuild.
---
 automation/build/alpine/3.18-arm64v8.dockerfile |  1 +-
 automation/scripts/console.exp                  | 13 +++++-
 automation/scripts/qubes-x86-64.sh              | 52 ++++--------------
 3 files changed, 27 insertions(+), 39 deletions(-)

diff --git a/automation/build/alpine/3.18-arm64v8.dockerfile b/automation/build/alpine/3.18-arm64v8.dockerfile
index 19fe46f8418f..b8482d5bf43f 100644
--- a/automation/build/alpine/3.18-arm64v8.dockerfile
+++ b/automation/build/alpine/3.18-arm64v8.dockerfile
@@ -48,3 +48,4 @@ RUN apk --no-cache add \
   # qubes test deps
   openssh-client \
   fakeroot \
+  expect \
diff --git a/automation/scripts/console.exp b/automation/scripts/console.exp
index 834a08db1b95..bdb1dd982003 100755
--- a/automation/scripts/console.exp
+++ b/automation/scripts/console.exp
@@ -9,6 +9,10 @@
 #   tests that's a command to read serial console
 # - UBOOT_CMD (optional): command to enter at u-boot prompt
 # - BOOT_MSG (optional): initial Xen message to wait for (aka sign-of-life)
+# - SUSPEND_MSG (optional): message signaling system is going to sleep, it's
+#   trigger for WAKEUP_CMD (see below)
+# - WAKEUP_CMD (optional): command to execute to wakeup the system 30s after
+#   seeing SUSPEND_MSG
 # - LOG_MSG (optional): final console message to wait for
 # - PASSED: message to look for to consider test a success; if LOG_MSG is set,
 #   both LOG_MSG and PASSED must appear (in any order) for test to succeed
@@ -45,6 +49,15 @@ if {[info exists env(BOOT_MSG)]} {
     expect -re "$env(BOOT_MSG)"
 }
 
+if {[info exists env(WAKEUP_CMD)]} {
+    expect -re "$env(SUSPEND_MSG)"
+
+    # keep it suspended a bit, then wakeup
+    sleep 30
+
+    system "$env(WAKEUP_CMD)"
+}
+
 if {[info exists env(LOG_MSG)]} {
     expect {
         -re "$env(PASSED)" {
diff --git a/automation/scripts/qubes-x86-64.sh b/automation/scripts/qubes-x86-64.sh
index a964ac4b7a4e..861e302d845b 100755
--- a/automation/scripts/qubes-x86-64.sh
+++ b/automation/scripts/qubes-x86-64.sh
@@ -1,6 +1,6 @@
 #!/bin/sh
 
-set -ex
+set -ex -o pipefail
 
 # One of:
 #  - ""             PV dom0,  PVH domU
@@ -267,52 +267,26 @@ cp -f binaries/xen $TFTP/xen
 cp -f binaries/bzImage $TFTP/vmlinuz
 cp -f binaries/dom0-rootfs.cpio.gz $TFTP/initrd-dom0
 
-# start logging the serial; this gives interactive console, don't close its
-# stdin to not close it; the 'cat' is important, plain redirection would hang
-# until somebody opens the pipe; opening and closing the pipe is used to close
-# the console
-mkfifo /tmp/console-stdin
-cat /tmp/console-stdin |\
-ssh $CONTROLLER console | tee smoke.serial | sed 's/\r//' &
-
 # start the system pointing at gitlab-ci predefined config
 ssh $CONTROLLER gitlabci poweron
-trap "ssh $CONTROLLER poweroff; : > /tmp/console-stdin" EXIT
+trap "ssh $CONTROLLER poweroff" EXIT
 
 if [ -n "$wait_and_wakeup" ]; then
-    # wait for suspend or a timeout
-    until grep "$wait_and_wakeup" smoke.serial || [ $timeout -le 0 ]; do
-        sleep 1;
-        : $((--timeout))
-    done
-    if [ $timeout -le 0 ]; then
-        echo "ERROR: suspend timeout, aborting"
-        exit 1
-    fi
-    # keep it suspended a bit, then wakeup
-    sleep 30
-    ssh $CONTROLLER wake
+    export SUSPEND_MSG="$wait_and_wakeup"
+    export WAKEUP_CMD="ssh $CONTROLLER wake"
 fi
 
-set +x
-until grep "^Welcome to Alpine Linux" smoke.serial || [ $timeout -le 0 ]; do
-    sleep 1;
-    : $((--timeout))
-done
-set -x
-
-tail -n 100 smoke.serial
-
-if [ $timeout -le 0 ]; then
-    echo "ERROR: test timeout, aborting"
-    exit 1
-fi
+export PASSED="${passed}"
+export BOOT_MSG="Latest ChangeSet: "
+export LOG_MSG="\nWelcome to Alpine Linux"
+export TEST_CMD="ssh $CONTROLLER console"
+export TEST_LOG="smoke.serial"
+export TEST_TIMEOUT="$timeout"
+./automation/scripts/console.exp | sed 's/\r\+$//'
+TEST_RESULT=$?
 
 if [ -n "$retrieve_xml" ]; then
     nc -w 10 "$SUT_ADDR" 8080 > tests-junit.xml </dev/null
 fi
 
-sleep 1
-
-(grep -q "^Welcome to Alpine Linux" smoke.serial && grep -q "${passed}" smoke.serial) || exit 1
-exit 0
+exit "$TEST_RESULT"
-- 
git-series 0.9.1

Re: [PATCH v3 3/7] CI: switch qubes runners to use console.exp

[Andrew Cooper]

On 11/04/2025 9:32 pm, Marek Marczykowski-Górecki wrote:
> diff --git a/automation/scripts/console.exp b/automation/scripts/console.exp
> index 834a08db1b95..bdb1dd982003 100755
> --- a/automation/scripts/console.exp
> +++ b/automation/scripts/console.exp
> @@ -45,6 +49,15 @@ if {[info exists env(BOOT_MSG)]} {
>      expect -re "$env(BOOT_MSG)"
>  }
>  
> +if {[info exists env(WAKEUP_CMD)]} {
> +    expect -re "$env(SUSPEND_MSG)"
> +
> +    # keep it suspended a bit, then wakeup
> +    sleep 30

Do we need 30s here?  Couldn't we get away with 10?

Either way, Acked-by: Andrew Cooper <andrew.cooper3@citrix.com>

Re: [PATCH v3 3/7] CI: switch qubes runners to use console.exp

[Marek Marczykowski-Górecki]

[-- Attachment #1: Type: text/plain, Size: 980 bytes --]

On Fri, Apr 11, 2025 at 09:40:31PM +0100, Andrew Cooper wrote:
> On 11/04/2025 9:32 pm, Marek Marczykowski-Górecki wrote:
> > diff --git a/automation/scripts/console.exp b/automation/scripts/console.exp
> > index 834a08db1b95..bdb1dd982003 100755
> > --- a/automation/scripts/console.exp
> > +++ b/automation/scripts/console.exp
> > @@ -45,6 +49,15 @@ if {[info exists env(BOOT_MSG)]} {
> >      expect -re "$env(BOOT_MSG)"
> >  }
> >  
> > +if {[info exists env(WAKEUP_CMD)]} {
> > +    expect -re "$env(SUSPEND_MSG)"
> > +
> > +    # keep it suspended a bit, then wakeup
> > +    sleep 30
> 
> Do we need 30s here?  Couldn't we get away with 10?

I want to be absolutely sure it got suspended (target system actually
suspends only after printing the message), and I've seen some drivers
taking a bit more time...

> Either way, Acked-by: Andrew Cooper <andrew.cooper3@citrix.com>

-- 
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

[PATCH v3 4/7] CI: write whole etc/issue for domU initrd

[Marek Marczykowski-Górecki]

Upcoming changes won't unpack original rootfs anymore, so sed on
existing file cannot be used. Override the whole file instead.

No functional change intended.

Signed-off-by: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com>
---
 automation/scripts/qubes-x86-64.sh             | 5 ++++-
 automation/scripts/xilinx-smoke-dom0-x86_64.sh | 5 ++++-
 2 files changed, 8 insertions(+), 2 deletions(-)

diff --git a/automation/scripts/qubes-x86-64.sh b/automation/scripts/qubes-x86-64.sh
index 861e302d845b..dd88a1398f2b 100755
--- a/automation/scripts/qubes-x86-64.sh
+++ b/automation/scripts/qubes-x86-64.sh
@@ -182,7 +182,10 @@ ${domU_check}
 " > etc/local.d/xen.start
     chmod +x etc/local.d/xen.start
     echo "rc_verbose=yes" >> etc/rc.conf
-    sed -i -e 's/^Welcome/domU \0/' etc/issue
+    echo "domU Welcome to Alpine Linux 3.18
+Kernel \r on an \m (\l)
+
+" > etc/issue
     find . | fakeroot -i ../fakeroot-save cpio -H newc -o | gzip > ../binaries/domU-rootfs.cpio.gz
     cd ..
     rm -rf rootfs
diff --git a/automation/scripts/xilinx-smoke-dom0-x86_64.sh b/automation/scripts/xilinx-smoke-dom0-x86_64.sh
index 69caabe2d8ed..4db249530823 100755
--- a/automation/scripts/xilinx-smoke-dom0-x86_64.sh
+++ b/automation/scripts/xilinx-smoke-dom0-x86_64.sh
@@ -103,7 +103,10 @@ ${DOMU_CMD}
 " > etc/local.d/xen.start
 chmod +x etc/local.d/xen.start
 echo "rc_verbose=yes" >> etc/rc.conf
-sed -i -e 's/^Welcome/domU \0/' etc/issue
+echo "domU Welcome to Alpine Linux 3.18
+Kernel \r on an \m (\l)
+
+" > etc/issue
 copy_domU_files
 find . | cpio -H newc -o | gzip > ../binaries/domU-rootfs.cpio.gz
 cd ..
-- 
git-series 0.9.1

Re: [PATCH v3 4/7] CI: write whole etc/issue for domU initrd

[Andrew Cooper]

On 11/04/2025 9:32 pm, Marek Marczykowski-Górecki wrote:
> diff --git a/automation/scripts/qubes-x86-64.sh b/automation/scripts/qubes-x86-64.sh
> index 861e302d845b..dd88a1398f2b 100755
> --- a/automation/scripts/qubes-x86-64.sh
> +++ b/automation/scripts/qubes-x86-64.sh
> @@ -182,7 +182,10 @@ ${domU_check}
>  " > etc/local.d/xen.start
>      chmod +x etc/local.d/xen.start
>      echo "rc_verbose=yes" >> etc/rc.conf
> -    sed -i -e 's/^Welcome/domU \0/' etc/issue
> +    echo "domU Welcome to Alpine Linux 3.18

This 3.18 is going to go stale shortly.

Because of LOG_MSG="Welcome to Alpine Linux", the version doesn't matter.

Alpine does have /etc/os-release, but busybox's getty doesn't translate
\S. 
https://github.com/brgl/busybox/blob/abbf17abccbf832365d9acf1c280369ba7d5f8b2/libbb/login.c#L84-L86

I'd suggest just dropping the 3.18.

Otherwise, Reviewed-by: Andrew Cooper <andrew.cooper3@citrix.com>

Re: [PATCH v3 4/7] CI: write whole etc/issue for domU initrd

[Marek Marczykowski-Górecki]

[-- Attachment #1: Type: text/plain, Size: 1137 bytes --]

On Fri, Apr 11, 2025 at 10:07:28PM +0100, Andrew Cooper wrote:
> On 11/04/2025 9:32 pm, Marek Marczykowski-Górecki wrote:
> > diff --git a/automation/scripts/qubes-x86-64.sh b/automation/scripts/qubes-x86-64.sh
> > index 861e302d845b..dd88a1398f2b 100755
> > --- a/automation/scripts/qubes-x86-64.sh
> > +++ b/automation/scripts/qubes-x86-64.sh
> > @@ -182,7 +182,10 @@ ${domU_check}
> >  " > etc/local.d/xen.start
> >      chmod +x etc/local.d/xen.start
> >      echo "rc_verbose=yes" >> etc/rc.conf
> > -    sed -i -e 's/^Welcome/domU \0/' etc/issue
> > +    echo "domU Welcome to Alpine Linux 3.18
> 
> This 3.18 is going to go stale shortly.
> 
> Because of LOG_MSG="Welcome to Alpine Linux", the version doesn't matter.
> 
> Alpine does have /etc/os-release, but busybox's getty doesn't translate
> \S. 
> https://github.com/brgl/busybox/blob/abbf17abccbf832365d9acf1c280369ba7d5f8b2/libbb/login.c#L84-L86
> 
> I'd suggest just dropping the 3.18.

Fine with me.

> Otherwise, Reviewed-by: Andrew Cooper <andrew.cooper3@citrix.com>

-- 
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

[PATCH v3 5/7] CI: avoid repacking initrd as part of the test job

[Marek Marczykowski-Górecki]

The prerequisite for this is building rootfs.cpio.gz, not only
initrd.tar.gz. That's a change in the test-artifacts repository in the
"Rootfs generation" series from Andrew.

Having that, do not repack the whole initrd, but only pack modified
files and rely on Linux handling of concatenated archives.
This allows packing just test-related files (which includes the whole
toolstack), instead of the whole initrd.

DomU initrd handling is a bit more complicated thing. It's sent to the
target host as part of the dom0 initrd. For now include prepared domU
initrd as a whole in dom0's rootfs "overlay", which means compressing it
(again) each job (this takes 3s). This can be later improved by
splitting domU initrd into two parts (base + overlay) and concatenate
them in target dom0 already. For this to be useful, test-artifacts would
need to provide pre-made double-cpio version too (rootfs.cpio.gz wrapper
in yet another cpio.gz as boot/domU-initrd).

Since full initrd is not unpacked now when preparing domU (and dom0)
rootfs, a couple of minor changes are needed to how they are prepared.
This includes creating whole etc/issue file, instead of modifying
existing one, and a need to create a couple directories.

Finally, move adding "rc_verbose=yes" to /etc/rc.conf to initrd
building.

Signed-off-by: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com>
---
Changes in v3:
- adjust for Andrew's version of cpio rootfs artifacts
- don't require double-cpio archive for domU initrd, instead add it via
  dom0 initrd as plain cpio to avoid re-compressing the same thing
New in v2

Jobs xilinx-* are untested, as I don't have necessary access.
---
 automation/gitlab-ci/test.yaml                    |  6 +++--
 automation/scripts/qemu-alpine-x86_64.sh          |  9 +++-----
 automation/scripts/qemu-smoke-dom0-arm64.sh       |  7 ++----
 automation/scripts/qemu-smoke-dom0less-arm64.sh   |  7 ++----
 automation/scripts/qubes-x86-64.sh                | 18 +++++++---------
 automation/scripts/xilinx-smoke-dom0-x86_64.sh    | 15 ++++++-------
 automation/scripts/xilinx-smoke-dom0less-arm64.sh | 14 +++++-------
 7 files changed, 35 insertions(+), 41 deletions(-)

diff --git a/automation/gitlab-ci/test.yaml b/automation/gitlab-ci/test.yaml
index 59a2de28c864..3e02bf0e4d21 100644
--- a/automation/gitlab-ci/test.yaml
+++ b/automation/gitlab-ci/test.yaml
@@ -11,7 +11,9 @@
   - project: xen-project/hardware/test-artifacts
     job: linux-6.6.86-arm64
     ref: master
-  - alpine-3.18-arm64-rootfs-export
+  - project: xen-project/hardware/test-artifacts
+    job: alpine-3.18-arm64-rootfs
+    ref: master
   - qemu-system-aarch64-6.0.0-arm64-export
 
 .arm32-test-needs: &arm32-test-needs
@@ -22,7 +24,7 @@
     job: linux-6.6.56-x86_64
     ref: master
   - project: xen-project/hardware/test-artifacts
-    job: x86_64-rootfs-alpine-3.18
+    job: alpine-3.18-x86_64-rootfs
     ref: master
 
 .qemu-arm64:
diff --git a/automation/scripts/qemu-alpine-x86_64.sh b/automation/scripts/qemu-alpine-x86_64.sh
index 89bdb4df7dac..6e0a67b4018c 100755
--- a/automation/scripts/qemu-alpine-x86_64.sh
+++ b/automation/scripts/qemu-alpine-x86_64.sh
@@ -28,16 +28,15 @@ cd initrd
 find . | cpio --create --format='newc' | gzip > ../initrd.cpio.gz
 cd ..
 
-# initrd.tar.gz is Dom0 rootfs
+# Dom0 rootfs - overlay on top of rootfs.cpio.gz
 mkdir -p rootfs
 cd rootfs
-tar xvzf ../initrd.tar.gz
 mkdir proc
 mkdir run
 mkdir srv
 mkdir sys
-rm var/run
 cp -ar ../dist/install/* .
+mkdir -p root etc/local.d
 mv ../initrd.cpio.gz ./root
 cp ../bzImage ./root
 echo "name=\"test\"
@@ -60,9 +59,9 @@ xl -vvv create -c /root/test.cfg
 
 " > etc/local.d/xen.start
 chmod +x etc/local.d/xen.start
-echo "rc_verbose=yes" >> etc/rc.conf
 # rebuild Dom0 rootfs
-find . |cpio -H newc -o|gzip > ../xen-rootfs.cpio.gz
+cp ../rootfs.cpio.gz ../xen-rootfs.cpio.gz
+find . |cpio -H newc -o|gzip >> ../xen-rootfs.cpio.gz
 cd ../..
 
 cat >> binaries/pxelinux.0 << EOF
diff --git a/automation/scripts/qemu-smoke-dom0-arm64.sh b/automation/scripts/qemu-smoke-dom0-arm64.sh
index 51d037b0003e..86047ccd9e7b 100755
--- a/automation/scripts/qemu-smoke-dom0-arm64.sh
+++ b/automation/scripts/qemu-smoke-dom0-arm64.sh
@@ -29,13 +29,12 @@ cd ..
 
 mkdir -p rootfs
 cd rootfs
-tar xvzf ../initrd.tar.gz
 mkdir proc
 mkdir run
 mkdir srv
 mkdir sys
-rm var/run
 cp -ar ../dist/install/* .
+mkdir -p etc/local.d root
 mv ../initrd.cpio.gz ./root
 cp ../Image ./root
 echo "name=\"test\"
@@ -56,8 +55,8 @@ xl -vvv create -c /root/test.cfg
 
 " > etc/local.d/xen.start
 chmod +x etc/local.d/xen.start
-echo "rc_verbose=yes" >> etc/rc.conf
-find . |cpio -H newc -o|gzip > ../xen-rootfs.cpio.gz
+cp ../rootfs.cpio.gz ../xen-rootfs.cpio.gz
+find . |cpio -H newc -o|gzip >> ../xen-rootfs.cpio.gz
 cd ../..
 
 # XXX QEMU looks for "efi-virtio.rom" even if it is unneeded
diff --git a/automation/scripts/qemu-smoke-dom0less-arm64.sh b/automation/scripts/qemu-smoke-dom0less-arm64.sh
index f72d20936181..b095a5008e30 100755
--- a/automation/scripts/qemu-smoke-dom0less-arm64.sh
+++ b/automation/scripts/qemu-smoke-dom0less-arm64.sh
@@ -117,13 +117,12 @@ cd ..
 # DOM0 rootfs
 mkdir -p rootfs
 cd rootfs
-tar xzf ../binaries/initrd.tar.gz
 mkdir proc
 mkdir run
 mkdir srv
 mkdir sys
-rm var/run
 cp -ar ../binaries/dist/install/* .
+mkdir -p etc/local.d
 
 echo "#!/bin/bash
 
@@ -142,8 +141,8 @@ xl network-attach 1 type=vif
 ${dom0_check}
 " > etc/local.d/xen.start
 chmod +x etc/local.d/xen.start
-echo "rc_verbose=yes" >> etc/rc.conf
-find . | cpio -H newc -o | gzip > ../binaries/dom0-rootfs.cpio.gz
+cp ../binaries/rootfs.cpio.gz ../binaries/dom0-rootfs.cpio.gz
+find . | cpio -H newc -o | gzip >> ../binaries/dom0-rootfs.cpio.gz
 cd ..
 
 # ImageBuilder
diff --git a/automation/scripts/qubes-x86-64.sh b/automation/scripts/qubes-x86-64.sh
index dd88a1398f2b..99c2ece304f0 100755
--- a/automation/scripts/qubes-x86-64.sh
+++ b/automation/scripts/qubes-x86-64.sh
@@ -167,13 +167,11 @@ if [ -n "$domU_check" ]; then
     # DomU
     mkdir -p rootfs
     cd rootfs
-    # fakeroot is needed to preserve device nodes in rootless podman container
-    fakeroot -s ../fakeroot-save tar xzf ../binaries/initrd.tar.gz
     mkdir proc
     mkdir run
     mkdir srv
     mkdir sys
-    rm var/run
+    mkdir -p etc/local.d
     echo "#!/bin/sh
 
 echo 8 > /proc/sys/kernel/printk
@@ -181,26 +179,25 @@ echo 8 > /proc/sys/kernel/printk
 ${domU_check}
 " > etc/local.d/xen.start
     chmod +x etc/local.d/xen.start
-    echo "rc_verbose=yes" >> etc/rc.conf
     echo "domU Welcome to Alpine Linux 3.18
 Kernel \r on an \m (\l)
 
 " > etc/issue
-    find . | fakeroot -i ../fakeroot-save cpio -H newc -o | gzip > ../binaries/domU-rootfs.cpio.gz
+    cp ../binaries/rootfs.cpio.gz ../binaries/domU-rootfs.cpio.gz
+    find . | cpio -H newc -o | gzip >> ../binaries/domU-rootfs.cpio.gz
     cd ..
     rm -rf rootfs
 fi
 
-# DOM0 rootfs
+# DOM0 rootfs - this will be an overlay over alpine's initrd
 mkdir -p rootfs
 cd rootfs
-fakeroot -s ../fakeroot-save tar xzf ../binaries/initrd.tar.gz
 mkdir boot
 mkdir proc
 mkdir run
 mkdir srv
 mkdir sys
-rm var/run
+mkdir -p etc/local.d
 cp -ar ../binaries/dist/install/* .
 cp -ar ../binaries/tests .
 cp -a ../automation/scripts/run-tools-tests tests/
@@ -237,7 +234,6 @@ fi
 chmod +x etc/local.d/xen.start
 echo "$domU_config" > etc/xen/domU.cfg
 
-echo "rc_verbose=yes" >> etc/rc.conf
 echo "XENCONSOLED_TRACE=all" >> etc/default/xencommons
 echo "QEMU_XEN=/bin/false" >> etc/default/xencommons
 mkdir -p var/log/xen/console
@@ -245,7 +241,9 @@ cp ../binaries/bzImage boot/vmlinuz
 if [ -n "$domU_check" ]; then
     cp ../binaries/domU-rootfs.cpio.gz boot/initrd-domU
 fi
-find . | fakeroot -i ../fakeroot-save cpio -H newc -o | gzip > ../binaries/dom0-rootfs.cpio.gz
+# take base initrd and append test-specific files
+cp ../binaries/rootfs.cpio.gz ../binaries/dom0-rootfs.cpio.gz
+find . | cpio -H newc -o | gzip >> ../binaries/dom0-rootfs.cpio.gz
 cd ..
 
 
diff --git a/automation/scripts/xilinx-smoke-dom0-x86_64.sh b/automation/scripts/xilinx-smoke-dom0-x86_64.sh
index 4db249530823..52a0e05e3f1b 100755
--- a/automation/scripts/xilinx-smoke-dom0-x86_64.sh
+++ b/automation/scripts/xilinx-smoke-dom0-x86_64.sh
@@ -80,6 +80,7 @@ argo-exec -l -p 28333 -- /bin/echo
 "
 copy_dom0_files ()
 {
+    mkdir -p root usr/local/lib
     cp "${WORKDIR}/binaries/xen-argo.ko" "root/"
     cp -ar "${WORKDIR}/binaries/lib/"* "usr/local/lib/"
     cp "${WORKDIR}/binaries/argo-exec" "usr/local/bin/"
@@ -92,9 +93,8 @@ fi
 # Set up domU rootfs.
 mkdir -p rootfs
 cd rootfs
-tar xzf ../binaries/initrd.tar.gz
 mkdir proc run srv sys
-rm var/run
+mkdir -p etc/local.d
 echo "#!/bin/sh
 set -x
 export LD_LIBRARY_PATH=/usr/local/lib
@@ -102,23 +102,22 @@ PATH=/usr/local/bin:/usr/local/sbin:\$PATH
 ${DOMU_CMD}
 " > etc/local.d/xen.start
 chmod +x etc/local.d/xen.start
-echo "rc_verbose=yes" >> etc/rc.conf
 echo "domU Welcome to Alpine Linux 3.18
 Kernel \r on an \m (\l)
 
 " > etc/issue
 copy_domU_files
-find . | cpio -H newc -o | gzip > ../binaries/domU-rootfs.cpio.gz
+cp ../binaries/initrd.cpio.gz ../binaries/domU-rootfs.cpio.gz
+find . | cpio -H newc -o | gzip >> ../binaries/domU-rootfs.cpio.gz
 cd ..
 rm -rf rootfs
 
 # Set up dom0 rootfs.
 mkdir -p rootfs
 cd rootfs
-tar xzf ../binaries/initrd.tar.gz
 mkdir boot proc run srv sys
-rm var/run
 cp -ar ../binaries/dist/install/* .
+mkdir -p etc/local.d
 echo "#!/bin/bash
 set -x
 export LD_LIBRARY_PATH=/usr/local/lib
@@ -127,14 +126,14 @@ ${DOM0_CMD}
 " > etc/local.d/xen.start
 chmod +x etc/local.d/xen.start
 echo "${DOMU_CFG}${DOMU_CFG_EXTRA}" > etc/xen/domU.cfg
-echo "rc_verbose=yes" >> etc/rc.conf
 echo "XENCONSOLED_TRACE=all" >> etc/default/xencommons
 echo "QEMU_XEN=/bin/false" >> etc/default/xencommons
 mkdir -p var/log/xen/console
 cp ../binaries/bzImage boot/vmlinuz
 cp ../binaries/domU-rootfs.cpio.gz boot/initrd-domU
 copy_dom0_files
-find . | cpio -H newc -o | gzip > ../binaries/dom0-rootfs.cpio.gz
+cp ../binaries/rootfs.cpio.gz ../binaries/dom0-rootfs.cpio.gz
+find . | cpio -H newc -o | gzip >> ../binaries/dom0-rootfs.cpio.gz
 cd ..
 
 # Load software into TFTP server directory.
diff --git a/automation/scripts/xilinx-smoke-dom0less-arm64.sh b/automation/scripts/xilinx-smoke-dom0less-arm64.sh
index 3e1fcf6bf93c..97190d0cbb9c 100755
--- a/automation/scripts/xilinx-smoke-dom0less-arm64.sh
+++ b/automation/scripts/xilinx-smoke-dom0less-arm64.sh
@@ -41,33 +41,31 @@ fi
 # DomU
 mkdir -p rootfs
 cd rootfs
-tar xzf ../binaries/initrd.tar.gz
 mkdir proc
 mkdir run
 mkdir srv
 mkdir sys
-rm var/run
+mkdir -p etc/local.d
 echo "#!/bin/sh
 
 ${domU_check}
 /bin/sh" > etc/local.d/xen.start
 chmod +x etc/local.d/xen.start
-echo "rc_verbose=yes" >> etc/rc.conf
-find . | cpio -H newc -o | gzip > ../binaries/domU-rootfs.cpio.gz
+cp ../binaries/rootfs.cpio.gz ../binaries/domU-rootfs.cpio.gz
+find . | cpio -H newc -o | gzip >> ../binaries/domU-rootfs.cpio.gz
 cd ..
 rm -rf rootfs
 
 # DOM0 rootfs
 mkdir -p rootfs
 cd rootfs
-tar xzf ../binaries/initrd.tar.gz
 mkdir proc
 mkdir run
 mkdir srv
 mkdir sys
-rm var/run
 cp -ar ../binaries/dist/install/* .
 
+mkdir -p etc/local.d
 echo "#!/bin/bash
 
 export LD_LIBRARY_PATH=/usr/local/lib
@@ -78,8 +76,8 @@ bash /etc/init.d/xencommons start
 ${dom0_check}
 " > etc/local.d/xen.start
 chmod +x etc/local.d/xen.start
-echo "rc_verbose=yes" >> etc/rc.conf
-find . | cpio -H newc -o | gzip > ../binaries/dom0-rootfs.cpio.gz
+cp ../binaries/rootfs.cpio.gz ../binaries/dom0-rootfs.cpio.gz
+find . | cpio -H newc -o | gzip >> ../binaries/dom0-rootfs.cpio.gz
 cd ..
 
 
-- 
git-series 0.9.1

Re: [PATCH v3 5/7] CI: avoid repacking initrd as part of the test job

[Andrew Cooper]

On 11/04/2025 9:32 pm, Marek Marczykowski-Górecki wrote:
> The prerequisite for this is building rootfs.cpio.gz, not only
> initrd.tar.gz. That's a change in the test-artifacts repository in the
> "Rootfs generation" series from Andrew.
>
> Having that, do not repack the whole initrd, but only pack modified
> files and rely on Linux handling of concatenated archives.
> This allows packing just test-related files (which includes the whole
> toolstack), instead of the whole initrd.
>
> DomU initrd handling is a bit more complicated thing. It's sent to the
> target host as part of the dom0 initrd. For now include prepared domU
> initrd as a whole in dom0's rootfs "overlay", which means compressing it
> (again) each job (this takes 3s). This can be later improved by
> splitting domU initrd into two parts (base + overlay) and concatenate
> them in target dom0 already. For this to be useful, test-artifacts would
> need to provide pre-made double-cpio version too (rootfs.cpio.gz wrapper
> in yet another cpio.gz as boot/domU-initrd).
>
> Since full initrd is not unpacked now when preparing domU (and dom0)
> rootfs, a couple of minor changes are needed to how they are prepared.
> This includes creating whole etc/issue file, instead of modifying
> existing one, and a need to create a couple directories.
>
> Finally, move adding "rc_verbose=yes" to /etc/rc.conf to initrd
> building.
>
> Signed-off-by: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com>
> ---
> Changes in v3:
> - adjust for Andrew's version of cpio rootfs artifacts
> - don't require double-cpio archive for domU initrd, instead add it via
>   dom0 initrd as plain cpio to avoid re-compressing the same thing
> New in v2
>
> Jobs xilinx-* are untested, as I don't have necessary access.

I can test these.

> ---
>  automation/gitlab-ci/test.yaml                    |  6 +++--
>  automation/scripts/qemu-alpine-x86_64.sh          |  9 +++-----
>  automation/scripts/qemu-smoke-dom0-arm64.sh       |  7 ++----
>  automation/scripts/qemu-smoke-dom0less-arm64.sh   |  7 ++----
>  automation/scripts/qubes-x86-64.sh                | 18 +++++++---------
>  automation/scripts/xilinx-smoke-dom0-x86_64.sh    | 15 ++++++-------
>  automation/scripts/xilinx-smoke-dom0less-arm64.sh | 14 +++++-------
>  7 files changed, 35 insertions(+), 41 deletions(-)
>
> diff --git a/automation/gitlab-ci/test.yaml b/automation/gitlab-ci/test.yaml
> index 59a2de28c864..3e02bf0e4d21 100644
> --- a/automation/gitlab-ci/test.yaml
> +++ b/automation/gitlab-ci/test.yaml
> @@ -11,7 +11,9 @@
>    - project: xen-project/hardware/test-artifacts
>      job: linux-6.6.86-arm64
>      ref: master
> -  - alpine-3.18-arm64-rootfs-export
> +  - project: xen-project/hardware/test-artifacts
> +    job: alpine-3.18-arm64-rootfs
> +    ref: master
>    - qemu-system-aarch64-6.0.0-arm64-export
>  
>  .arm32-test-needs: &arm32-test-needs
> @@ -22,7 +24,7 @@
>      job: linux-6.6.56-x86_64
>      ref: master
>    - project: xen-project/hardware/test-artifacts
> -    job: x86_64-rootfs-alpine-3.18
> +    job: alpine-3.18-x86_64-rootfs
>      ref: master
>  
>  .qemu-arm64:
> diff --git a/automation/scripts/qemu-alpine-x86_64.sh b/automation/scripts/qemu-alpine-x86_64.sh
> index 89bdb4df7dac..6e0a67b4018c 100755
> --- a/automation/scripts/qemu-alpine-x86_64.sh
> +++ b/automation/scripts/qemu-alpine-x86_64.sh

There are some very poor choices of name in this script.  I'll do a prep
patch to make them consistent.

> @@ -28,16 +28,15 @@ cd initrd
>  find . | cpio --create --format='newc' | gzip > ../initrd.cpio.gz
>  cd ..
>  
> -# initrd.tar.gz is Dom0 rootfs
> +# Dom0 rootfs - overlay on top of rootfs.cpio.gz
>  mkdir -p rootfs
>  cd rootfs
> -tar xvzf ../initrd.tar.gz
>  mkdir proc
>  mkdir run
>  mkdir srv
>  mkdir sys
> -rm var/run

I'm not sure why we need to make proc, run, srv, sys in the overlay
image, but this is repeated everywhere.

srv is just a plain empty dir.  It's missing from $PATHS in the rootfs
generation, so that's easy.

proc, run and sys are more complicated, because we don't want to simply
copy them from the rootfs generation environment.

dev is also in the same boat, and what we do now seems to cause openrc
not to seed it normally, which seems unwise.

I'll see what I can do to clean this all up.

> diff --git a/automation/scripts/xilinx-smoke-dom0-x86_64.sh b/automation/scripts/xilinx-smoke-dom0-x86_64.sh
> index 4db249530823..52a0e05e3f1b 100755
> --- a/automation/scripts/xilinx-smoke-dom0-x86_64.sh
> +++ b/automation/scripts/xilinx-smoke-dom0-x86_64.sh
> @@ -80,6 +80,7 @@ argo-exec -l -p 28333 -- /bin/echo
>  "
>  copy_dom0_files ()
>  {
> +    mkdir -p root usr/local/lib
>      cp "${WORKDIR}/binaries/xen-argo.ko" "root/"
>      cp -ar "${WORKDIR}/binaries/lib/"* "usr/local/lib/"
>      cp "${WORKDIR}/binaries/argo-exec" "usr/local/bin/"

This would be better based on top of my patch to switch to the new argo,
at which point it will simply be to append argo.cpio.gz.

~Andrew

Re: [PATCH v3 5/7] CI: avoid repacking initrd as part of the test job

[Marek Marczykowski-Górecki]

[-- Attachment #1: Type: text/plain, Size: 2438 bytes --]

On Sat, Apr 12, 2025 at 03:05:13PM +0100, Andrew Cooper wrote:
> On 11/04/2025 9:32 pm, Marek Marczykowski-Górecki wrote:
> > @@ -28,16 +28,15 @@ cd initrd
> >  find . | cpio --create --format='newc' | gzip > ../initrd.cpio.gz
> >  cd ..
> >  
> > -# initrd.tar.gz is Dom0 rootfs
> > +# Dom0 rootfs - overlay on top of rootfs.cpio.gz
> >  mkdir -p rootfs
> >  cd rootfs
> > -tar xvzf ../initrd.tar.gz
> >  mkdir proc
> >  mkdir run
> >  mkdir srv
> >  mkdir sys
> > -rm var/run
> 
> I'm not sure why we need to make proc, run, srv, sys in the overlay
> image, but this is repeated everywhere.
> 
> srv is just a plain empty dir.  It's missing from $PATHS in the rootfs
> generation, so that's easy.

srv is indeed likely not needed, but there is enough changes here
already.

> proc, run and sys are more complicated, because we don't want to simply
> copy them from the rootfs generation environment.

As for those mountpoints, they looks to be necessary - in a job that
failed to unpack this overlay part, openrc failed to mount those
filesystems and crashed rather badly (especially without /run...).
I'm not sure if that's something that is missing in the base archive, or
is a problem for another reason, but it seems to be necessary to have it
here, at least without changes elsewhere.
Looking at PATHS in alpine-rootfs.sh they may be simply skipped there.

> dev is also in the same boat, and what we do now seems to cause openrc
> not to seed it normally, which seems unwise.

It mounts devtmpfs there anyway, no?

> I'll see what I can do to clean this all up.
> 
> > diff --git a/automation/scripts/xilinx-smoke-dom0-x86_64.sh b/automation/scripts/xilinx-smoke-dom0-x86_64.sh
> > index 4db249530823..52a0e05e3f1b 100755
> > --- a/automation/scripts/xilinx-smoke-dom0-x86_64.sh
> > +++ b/automation/scripts/xilinx-smoke-dom0-x86_64.sh
> > @@ -80,6 +80,7 @@ argo-exec -l -p 28333 -- /bin/echo
> >  "
> >  copy_dom0_files ()
> >  {
> > +    mkdir -p root usr/local/lib
> >      cp "${WORKDIR}/binaries/xen-argo.ko" "root/"
> >      cp -ar "${WORKDIR}/binaries/lib/"* "usr/local/lib/"
> >      cp "${WORKDIR}/binaries/argo-exec" "usr/local/bin/"
> 
> This would be better based on top of my patch to switch to the new argo,
> at which point it will simply be to append argo.cpio.gz.
> 
> ~Andrew

-- 
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [PATCH v3 5/7] CI: avoid repacking initrd as part of the test job

[Andrew Cooper]

On 11/04/2025 9:32 pm, Marek Marczykowski-Górecki wrote:
> The prerequisite for this is building rootfs.cpio.gz, not only
> initrd.tar.gz. That's a change in the test-artifacts repository in the
> "Rootfs generation" series from Andrew.
>
> Having that, do not repack the whole initrd, but only pack modified
> files and rely on Linux handling of concatenated archives.
> This allows packing just test-related files (which includes the whole
> toolstack), instead of the whole initrd.

It's probably worth saying ", fixed subsequently)" for the toolstack, as
it's the single largest aspect.

> DomU initrd handling is a bit more complicated thing. It's sent to the
> target host as part of the dom0 initrd. For now include prepared domU
> initrd as a whole in dom0's rootfs "overlay", which means compressing it
> (again) each job (this takes 3s). This can be later improved by
> splitting domU initrd into two parts (base + overlay) and concatenate
> them in target dom0 already. For this to be useful, test-artifacts would
> need to provide pre-made double-cpio version too (rootfs.cpio.gz wrapper
> in yet another cpio.gz as boot/domU-initrd).
>
> Since full initrd is not unpacked now when preparing domU (and dom0)
> rootfs, a couple of minor changes are needed to how they are prepared.
> This includes creating whole etc/issue file, instead of modifying
> existing one, and a need to create a couple directories.
>
> Finally, move adding "rc_verbose=yes" to /etc/rc.conf to initrd
> building.

These two paragraphs are a little stale now.  etc/issue has already been
committed, and there's no initd building now I've moved arm into test
artefacts.

However, they're going to need tweaking again if I can get the rootfs
generation working more nicely, so lets hold off for now.

>
> Signed-off-by: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com>
> ---
> Changes in v3:
> - adjust for Andrew's version of cpio rootfs artifacts
> - don't require double-cpio archive for domU initrd, instead add it via
>   dom0 initrd as plain cpio to avoid re-compressing the same thing
> New in v2
>
> Jobs xilinx-* are untested, as I don't have necessary access.
> ---
>  automation/gitlab-ci/test.yaml                    |  6 +++--
>  automation/scripts/qemu-alpine-x86_64.sh          |  9 +++-----
>  automation/scripts/qemu-smoke-dom0-arm64.sh       |  7 ++----
>  automation/scripts/qemu-smoke-dom0less-arm64.sh   |  7 ++----
>  automation/scripts/qubes-x86-64.sh                | 18 +++++++---------
>  automation/scripts/xilinx-smoke-dom0-x86_64.sh    | 15 ++++++-------
>  automation/scripts/xilinx-smoke-dom0less-arm64.sh | 14 +++++-------
>  7 files changed, 35 insertions(+), 41 deletions(-)

One thing that does matter now is the order of appending archives.

I've reworked this locally to be of the form:

# DomU rootfs
cp ../binaries/rootfs.cpio.gz ../binaries/domU-rootfs.cpio.gz

# test-local configuration
mkdir rootfs
...
find . | cpio -H newc -o | gzip >> ../binaries/domU-rootfs.cpio.gz


This then ends up looking rather more clean in followon patches, where
we end up with:

# Dom0 rootfs
cp ../binaries/ucode.cpio ../binaries/dom0-rootfs.cpio.gz
cat ../binaries/rootfs.cpio.gz >> ../binaries/dom0-rootfs.cpio.gz
cat ../binaries/toolstack.cpio.gz >> ../binaries/dom0-rootfs.cpio.gz
if [[ "${TEST}" == argo ]]; then
    cat ../binaries/argo.cpio.gz >> ../binaries/dom0-rootfs.cpio.gz
fi

# test-local configuration
...
find . | cpio -H newc -o | gzip >> ../binaries/dom0-rootfs.cpio.gz


> diff --git a/automation/scripts/xilinx-smoke-dom0-x86_64.sh b/automation/scripts/xilinx-smoke-dom0-x86_64.sh
> index 4db249530823..52a0e05e3f1b 100755
> --- a/automation/scripts/xilinx-smoke-dom0-x86_64.sh
> +++ b/automation/scripts/xilinx-smoke-dom0-x86_64.sh
> @@ -80,6 +80,7 @@ argo-exec -l -p 28333 -- /bin/echo
>  "
>  copy_dom0_files ()
>  {
> +    mkdir -p root usr/local/lib
>      cp "${WORKDIR}/binaries/xen-argo.ko" "root/"
>      cp -ar "${WORKDIR}/binaries/lib/"* "usr/local/lib/"
>      cp "${WORKDIR}/binaries/argo-exec" "usr/local/bin/"
> @@ -92,9 +93,8 @@ fi
>  # Set up domU rootfs.
>  mkdir -p rootfs
>  cd rootfs
> -tar xzf ../binaries/initrd.tar.gz
>  mkdir proc run srv sys
> -rm var/run
> +mkdir -p etc/local.d
>  echo "#!/bin/sh
>  set -x
>  export LD_LIBRARY_PATH=/usr/local/lib
> @@ -102,23 +102,22 @@ PATH=/usr/local/bin:/usr/local/sbin:\$PATH
>  ${DOMU_CMD}
>  " > etc/local.d/xen.start
>  chmod +x etc/local.d/xen.start
> -echo "rc_verbose=yes" >> etc/rc.conf
>  echo "domU Welcome to Alpine Linux 3.18
>  Kernel \r on an \m (\l)
>  
>  " > etc/issue
>  copy_domU_files
> -find . | cpio -H newc -o | gzip > ../binaries/domU-rootfs.cpio.gz
> +cp ../binaries/initrd.cpio.gz ../binaries/domU-rootfs.cpio.gz

Stray initrd.  Found while reworking.

I'll give the whole lot commit-by-commit full runs, when I'm done tinkering.

~Andrew

[PATCH v3 7/7] CI: save toolstack artifact as cpio.gz

[Marek Marczykowski-Górecki]

This avoids the need to re-compress it in every test job.

Signed-off-by: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com>
---
New in v2
---
 automation/scripts/build                          | 2 +-
 automation/scripts/qemu-alpine-x86_64.sh          | 2 +-
 automation/scripts/qemu-smoke-dom0-arm64.sh       | 2 +-
 automation/scripts/qemu-smoke-dom0less-arm64.sh   | 2 +-
 automation/scripts/qubes-x86-64.sh                | 4 +++-
 automation/scripts/xilinx-smoke-dom0-x86_64.sh    | 4 ++--
 automation/scripts/xilinx-smoke-dom0less-arm64.sh | 2 +-
 7 files changed, 10 insertions(+), 8 deletions(-)

diff --git a/automation/scripts/build b/automation/scripts/build
index 522efe774ef3..365534895047 100755
--- a/automation/scripts/build
+++ b/automation/scripts/build
@@ -108,7 +108,7 @@ else
     # Note: Some smoke tests depending on finding binaries/xen on a full build
     # even though dist/ contains everything, while some containers don't even
     # build Xen
-    cp -r dist binaries/
+    (cd dist/install; find | cpio -o -H newc | gzip) > binaries/toolstack.cpio.gz
     cp -r tools/tests binaries/
     collect_xen_artefacts
 fi
diff --git a/automation/scripts/qemu-alpine-x86_64.sh b/automation/scripts/qemu-alpine-x86_64.sh
index 6e0a67b4018c..746fd48e6a0e 100755
--- a/automation/scripts/qemu-alpine-x86_64.sh
+++ b/automation/scripts/qemu-alpine-x86_64.sh
@@ -35,7 +35,6 @@ mkdir proc
 mkdir run
 mkdir srv
 mkdir sys
-cp -ar ../dist/install/* .
 mkdir -p root etc/local.d
 mv ../initrd.cpio.gz ./root
 cp ../bzImage ./root
@@ -61,6 +60,7 @@ xl -vvv create -c /root/test.cfg
 chmod +x etc/local.d/xen.start
 # rebuild Dom0 rootfs
 cp ../rootfs.cpio.gz ../xen-rootfs.cpio.gz
+cat ../toolstack.cpio.gz >> ../xen-rootfs.cpio.gz
 find . |cpio -H newc -o|gzip >> ../xen-rootfs.cpio.gz
 cd ../..
 
diff --git a/automation/scripts/qemu-smoke-dom0-arm64.sh b/automation/scripts/qemu-smoke-dom0-arm64.sh
index 86047ccd9e7b..a3f321b253e2 100755
--- a/automation/scripts/qemu-smoke-dom0-arm64.sh
+++ b/automation/scripts/qemu-smoke-dom0-arm64.sh
@@ -33,7 +33,6 @@ mkdir proc
 mkdir run
 mkdir srv
 mkdir sys
-cp -ar ../dist/install/* .
 mkdir -p etc/local.d root
 mv ../initrd.cpio.gz ./root
 cp ../Image ./root
@@ -56,6 +55,7 @@ xl -vvv create -c /root/test.cfg
 " > etc/local.d/xen.start
 chmod +x etc/local.d/xen.start
 cp ../rootfs.cpio.gz ../xen-rootfs.cpio.gz
+cat ../toolstack.cpio.gz >> ../xen-rootfs.cpio.gz
 find . |cpio -H newc -o|gzip >> ../xen-rootfs.cpio.gz
 cd ../..
 
diff --git a/automation/scripts/qemu-smoke-dom0less-arm64.sh b/automation/scripts/qemu-smoke-dom0less-arm64.sh
index b095a5008e30..a73cb24b7e52 100755
--- a/automation/scripts/qemu-smoke-dom0less-arm64.sh
+++ b/automation/scripts/qemu-smoke-dom0less-arm64.sh
@@ -121,7 +121,6 @@ mkdir proc
 mkdir run
 mkdir srv
 mkdir sys
-cp -ar ../binaries/dist/install/* .
 mkdir -p etc/local.d
 
 echo "#!/bin/bash
@@ -142,6 +141,7 @@ ${dom0_check}
 " > etc/local.d/xen.start
 chmod +x etc/local.d/xen.start
 cp ../binaries/rootfs.cpio.gz ../binaries/dom0-rootfs.cpio.gz
+cat ../binaries/toolstack.cpio.gz >> ../binaries/dom0-rootfs.cpio.gz
 find . | cpio -H newc -o | gzip >> ../binaries/dom0-rootfs.cpio.gz
 cd ..
 
diff --git a/automation/scripts/qubes-x86-64.sh b/automation/scripts/qubes-x86-64.sh
index 99c2ece304f0..a9696d2168c3 100755
--- a/automation/scripts/qubes-x86-64.sh
+++ b/automation/scripts/qubes-x86-64.sh
@@ -198,7 +198,6 @@ mkdir run
 mkdir srv
 mkdir sys
 mkdir -p etc/local.d
-cp -ar ../binaries/dist/install/* .
 cp -ar ../binaries/tests .
 cp -a ../automation/scripts/run-tools-tests tests/
 
@@ -232,8 +231,10 @@ else
 fi
 
 chmod +x etc/local.d/xen.start
+mkdir -p etc/xen
 echo "$domU_config" > etc/xen/domU.cfg
 
+mkdir -p etc/default
 echo "XENCONSOLED_TRACE=all" >> etc/default/xencommons
 echo "QEMU_XEN=/bin/false" >> etc/default/xencommons
 mkdir -p var/log/xen/console
@@ -243,6 +244,7 @@ if [ -n "$domU_check" ]; then
 fi
 # take base initrd and append test-specific files
 cp ../binaries/rootfs.cpio.gz ../binaries/dom0-rootfs.cpio.gz
+cat ../binaries/toolstack.cpio.gz >> ../binaries/dom0-rootfs.cpio.gz
 find . | cpio -H newc -o | gzip >> ../binaries/dom0-rootfs.cpio.gz
 cd ..
 
diff --git a/automation/scripts/xilinx-smoke-dom0-x86_64.sh b/automation/scripts/xilinx-smoke-dom0-x86_64.sh
index 52a0e05e3f1b..40f23f29582e 100755
--- a/automation/scripts/xilinx-smoke-dom0-x86_64.sh
+++ b/automation/scripts/xilinx-smoke-dom0-x86_64.sh
@@ -116,8 +116,7 @@ rm -rf rootfs
 mkdir -p rootfs
 cd rootfs
 mkdir boot proc run srv sys
-cp -ar ../binaries/dist/install/* .
-mkdir -p etc/local.d
+mkdir -p etc/local.d etc/xen etc/default
 echo "#!/bin/bash
 set -x
 export LD_LIBRARY_PATH=/usr/local/lib
@@ -133,6 +132,7 @@ cp ../binaries/bzImage boot/vmlinuz
 cp ../binaries/domU-rootfs.cpio.gz boot/initrd-domU
 copy_dom0_files
 cp ../binaries/rootfs.cpio.gz ../binaries/dom0-rootfs.cpio.gz
+cat ../binaries/toolstack.cpio.gz >> ../binaries/dom0-rootfs.cpio.gz
 find . | cpio -H newc -o | gzip >> ../binaries/dom0-rootfs.cpio.gz
 cd ..
 
diff --git a/automation/scripts/xilinx-smoke-dom0less-arm64.sh b/automation/scripts/xilinx-smoke-dom0less-arm64.sh
index 97190d0cbb9c..ab9205f29689 100755
--- a/automation/scripts/xilinx-smoke-dom0less-arm64.sh
+++ b/automation/scripts/xilinx-smoke-dom0less-arm64.sh
@@ -63,7 +63,6 @@ mkdir proc
 mkdir run
 mkdir srv
 mkdir sys
-cp -ar ../binaries/dist/install/* .
 
 mkdir -p etc/local.d
 echo "#!/bin/bash
@@ -77,6 +76,7 @@ ${dom0_check}
 " > etc/local.d/xen.start
 chmod +x etc/local.d/xen.start
 cp ../binaries/rootfs.cpio.gz ../binaries/dom0-rootfs.cpio.gz
+cat ../binaries/toolstack.cpio.gz >> ../binaries/dom0-rootfs.cpio.gz
 find . | cpio -H newc -o | gzip >> ../binaries/dom0-rootfs.cpio.gz
 cd ..
 
-- 
git-series 0.9.1

[PATCH test-artifacts v3 8/7] Enable CONFIG_USB_RTL8152 in kernel for hw12 runner

[Marek Marczykowski-Górecki]

It uses this USB network interface.

Signed-off-by: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com>
---
 scripts/build-linux.sh | 1 +
 1 file changed, 1 insertion(+)

diff --git a/scripts/build-linux.sh b/scripts/build-linux.sh
index 49b5ebe..af684d6 100755
--- a/scripts/build-linux.sh
+++ b/scripts/build-linux.sh
@@ -32,6 +32,7 @@ case $UNAME in
             | grep 'XEN' \
             | grep '=m' \
             | sed 's/=m/=y/g' >> .config
+        ./scripts/config --enable USB_RTL8152
         ;;
 
     aarch64)
-- 
2.48.1

Re: [PATCH v3 6/7] CI: remove now unused alpine-3.18-arm64-rootfs job and its container

[Andrew Cooper]

On 11/04/2025 9:32 pm, Marek Marczykowski-Górecki wrote:
> This got moved to test-artifacts.
>
> Signed-off-by: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com>

Reviewed-by: Andrew Cooper <andrew.cooper3@citrix.com>