* [PATCH v2 01/12] CI: prevent grub unpacking initramfs
2025-04-07 12:31 [PATCH v2 00/12] Several CI cleanups and improvements, plus yet another new runner Marek Marczykowski-Górecki
@ 2025-04-07 12:31 ` Marek Marczykowski-Górecki
2025-04-07 12:31 ` [PATCH v2 02/12] CI: switch qubes runners to use console.exp Marek Marczykowski-Górecki
` (12 subsequent siblings)
13 siblings, 0 replies; 26+ messages in thread
From: Marek Marczykowski-Górecki @ 2025-04-07 12:31 UTC (permalink / raw)
To: xen-devel
Cc: Marek Marczykowski-Górecki, Stefano Stabellini,
Doug Goldstein
It fails on larger initramfs (~250MB one) and sometimes even smaller
depending on memory size/memory map, let Linux do it.
Signed-off-by: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com>
Reviewed-by: Stefano Stabellini <sstabellini@kernel.org>
---
automation/scripts/qubes-x86-64.sh | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/automation/scripts/qubes-x86-64.sh b/automation/scripts/qubes-x86-64.sh
index f27296445901..8e78b7984e98 100755
--- a/automation/scripts/qubes-x86-64.sh
+++ b/automation/scripts/qubes-x86-64.sh
@@ -248,7 +248,7 @@ CONTROLLER=control@thor.testnet
echo "
multiboot2 (http)/gitlab-ci/xen $CONSOLE_OPTS loglvl=all guest_loglvl=all dom0_mem=4G console_timestamps=boot $extra_xen_opts
module2 (http)/gitlab-ci/vmlinuz console=hvc0 root=/dev/ram0 earlyprintk=xen
-module2 (http)/gitlab-ci/initrd-dom0
+module2 --nounzip (http)/gitlab-ci/initrd-dom0
" > $TFTP/grub.cfg
echo "#!ipxe
--
git-series 0.9.1
^ permalink raw reply related [flat|nested] 26+ messages in thread* [PATCH v2 02/12] CI: switch qubes runners to use console.exp
2025-04-07 12:31 [PATCH v2 00/12] Several CI cleanups and improvements, plus yet another new runner Marek Marczykowski-Górecki
2025-04-07 12:31 ` [PATCH v2 01/12] CI: prevent grub unpacking initramfs Marek Marczykowski-Górecki
@ 2025-04-07 12:31 ` Marek Marczykowski-Górecki
2025-04-07 18:07 ` Andrew Cooper
2025-04-07 12:31 ` [PATCH v2 03/12] CI: enable XHCI console in Xen debug build on Alpine Marek Marczykowski-Górecki
` (11 subsequent siblings)
13 siblings, 1 reply; 26+ messages in thread
From: Marek Marczykowski-Górecki @ 2025-04-07 12:31 UTC (permalink / raw)
To: xen-devel
Cc: Marek Marczykowski-Górecki, Doug Goldstein,
Stefano Stabellini
It appears as sometimes it takes more time for Xen even start booting,
mostly due to firmware and fetching large boot files by grub. In some
jobs the current timeout is pretty close to the actual time needed, and
sometimes (rarely for now) test fails due to timeout expiring in the
middle of dom0 booting. This will be happening more often if the
initramfs will grow (and with more complex tests).
This has been observed on some dom0pvh-hvm jobs, at least on runners hw3
and hw11.
Switch to using expect (console.exp) for more robust test output
handling. This allows waiting separately for Xen starting to boot and
then for the test to complete. For now, set both of those to 120s, which
pessimistically bumps timeout for the whole test to 240s (from 120s).
Some messages use regex, use 'expect -re' for all of them for
consistency, even though not all strictly need that (yet).
Signed-off-by: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com>
---
Changes in v2:
- replace previous "ci: increase timeout for hw tests" with changing how
console is interacted with
This needs a containers rebuild.
---
automation/build/alpine/3.18-arm64v8.dockerfile | 1 +-
automation/scripts/console.exp | 23 ++++++--
automation/scripts/qubes-x86-64.sh | 52 ++++--------------
3 files changed, 32 insertions(+), 44 deletions(-)
diff --git a/automation/build/alpine/3.18-arm64v8.dockerfile b/automation/build/alpine/3.18-arm64v8.dockerfile
index 19fe46f8418f..b8482d5bf43f 100644
--- a/automation/build/alpine/3.18-arm64v8.dockerfile
+++ b/automation/build/alpine/3.18-arm64v8.dockerfile
@@ -48,3 +48,4 @@ RUN apk --no-cache add \
# qubes test deps
openssh-client \
fakeroot \
+ expect \
diff --git a/automation/scripts/console.exp b/automation/scripts/console.exp
index 31ce97b91b63..d1689fa5bf7f 100755
--- a/automation/scripts/console.exp
+++ b/automation/scripts/console.exp
@@ -28,21 +28,34 @@ if {[info exists env(UBOOT_CMD)]} {
send "$env(UBOOT_CMD)\r"
}
+if {[info exists env(BOOT_MSG)]} {
+ expect -re "$env(BOOT_MSG)"
+}
+
+if {[info exists env(WAKEUP_CMD)]} {
+ expect -re "$env(SUSPEND_MSG)"
+
+ # keep it suspended a bit, then wakeup
+ sleep 30
+
+ system "$env(WAKEUP_CMD)"
+}
+
if {[info exists env(LOG_MSG)]} {
expect {
- "$env(PASSED)" {
- expect "$env(LOG_MSG)"
+ -re "$env(PASSED)" {
+ expect -re "$env(LOG_MSG)"
exit 0
}
- "$env(LOG_MSG)" {
- expect "$env(PASSED)"
+ -re "$env(LOG_MSG)" {
+ expect -re "$env(PASSED)"
exit 0
}
}
}
expect {
- "$env(PASSED)" {
+ -re "$env(PASSED)" {
exit 0
}
}
diff --git a/automation/scripts/qubes-x86-64.sh b/automation/scripts/qubes-x86-64.sh
index 8e78b7984e98..0eac410f4168 100755
--- a/automation/scripts/qubes-x86-64.sh
+++ b/automation/scripts/qubes-x86-64.sh
@@ -1,6 +1,6 @@
#!/bin/sh
-set -ex
+set -ex -o pipefail
# One of:
# - "" PV dom0, PVH domU
@@ -263,52 +263,26 @@ cp -f binaries/xen $TFTP/xen
cp -f binaries/bzImage $TFTP/vmlinuz
cp -f binaries/dom0-rootfs.cpio.gz $TFTP/initrd-dom0
-# start logging the serial; this gives interactive console, don't close its
-# stdin to not close it; the 'cat' is important, plain redirection would hang
-# until somebody opens the pipe; opening and closing the pipe is used to close
-# the console
-mkfifo /tmp/console-stdin
-cat /tmp/console-stdin |\
-ssh $CONTROLLER console | tee smoke.serial | sed 's/\r//' &
-
# start the system pointing at gitlab-ci predefined config
ssh $CONTROLLER gitlabci poweron
-trap "ssh $CONTROLLER poweroff; : > /tmp/console-stdin" EXIT
+trap "ssh $CONTROLLER poweroff" EXIT
if [ -n "$wait_and_wakeup" ]; then
- # wait for suspend or a timeout
- until grep "$wait_and_wakeup" smoke.serial || [ $timeout -le 0 ]; do
- sleep 1;
- : $((--timeout))
- done
- if [ $timeout -le 0 ]; then
- echo "ERROR: suspend timeout, aborting"
- exit 1
- fi
- # keep it suspended a bit, then wakeup
- sleep 30
- ssh $CONTROLLER wake
+ export SUSPEND_MSG="$wait_and_wakeup"
+ export WAKEUP_CMD="ssh $CONTROLLER wake"
fi
-set +x
-until grep "^Welcome to Alpine Linux" smoke.serial || [ $timeout -le 0 ]; do
- sleep 1;
- : $((--timeout))
-done
-set -x
-
-tail -n 100 smoke.serial
-
-if [ $timeout -le 0 ]; then
- echo "ERROR: test timeout, aborting"
- exit 1
-fi
+export PASSED="${passed}"
+export BOOT_MSG="Latest ChangeSet: "
+export LOG_MSG="\nWelcome to Alpine Linux"
+export TEST_CMD="ssh $CONTROLLER console"
+export TEST_LOG="smoke.serial"
+export TEST_TIMEOUT="$timeout"
+./automation/scripts/console.exp | sed 's/\r\+$//'
+TEST_RESULT=$?
if [ -n "$retrieve_xml" ]; then
nc -w 10 "$SUT_ADDR" 8080 > tests-junit.xml </dev/null
fi
-sleep 1
-
-(grep -q "^Welcome to Alpine Linux" smoke.serial && grep -q "${passed}" smoke.serial) || exit 1
-exit 0
+exit "$TEST_RESULT"
--
git-series 0.9.1
^ permalink raw reply related [flat|nested] 26+ messages in thread* Re: [PATCH v2 02/12] CI: switch qubes runners to use console.exp
2025-04-07 12:31 ` [PATCH v2 02/12] CI: switch qubes runners to use console.exp Marek Marczykowski-Górecki
@ 2025-04-07 18:07 ` Andrew Cooper
0 siblings, 0 replies; 26+ messages in thread
From: Andrew Cooper @ 2025-04-07 18:07 UTC (permalink / raw)
To: Marek Marczykowski-Górecki, xen-devel
Cc: Doug Goldstein, Stefano Stabellini
On 07/04/2025 1:31 pm, Marek Marczykowski-Górecki wrote:
> diff --git a/automation/scripts/console.exp b/automation/scripts/console.exp
> index 31ce97b91b63..d1689fa5bf7f 100755
> --- a/automation/scripts/console.exp
> +++ b/automation/scripts/console.exp
> @@ -28,21 +28,34 @@ if {[info exists env(UBOOT_CMD)]} {
> send "$env(UBOOT_CMD)\r"
> }
>
> +if {[info exists env(BOOT_MSG)]} {
> + expect -re "$env(BOOT_MSG)"
> +}
We're gaining an increasingly complex set of environment variables to
control this script. In particular, this is more a signs-of-life
message than a completion.
I think this patch wants splitting into two. First doing the -re
changes, and adding BOOT_MSG, and some comments explaining how each part
is supposed to be used. I expect the xilinx- jobs want to use BOOT_MSG
too, whereas I expect we don't want the qemu/cirrus smoke tests to use
it, seeing as they're all direct-boot in qemu.
The, the second patch is converting qubes-x86-64.sh to use it, adding
WAKEUP_CMD/SYSPEND_MSG as they're pretty specific.
~Andrew
^ permalink raw reply [flat|nested] 26+ messages in thread
* [PATCH v2 03/12] CI: enable XHCI console in Xen debug build on Alpine
2025-04-07 12:31 [PATCH v2 00/12] Several CI cleanups and improvements, plus yet another new runner Marek Marczykowski-Górecki
2025-04-07 12:31 ` [PATCH v2 01/12] CI: prevent grub unpacking initramfs Marek Marczykowski-Górecki
2025-04-07 12:31 ` [PATCH v2 02/12] CI: switch qubes runners to use console.exp Marek Marczykowski-Górecki
@ 2025-04-07 12:31 ` Marek Marczykowski-Górecki
2025-04-07 12:31 ` [PATCH v2 04/12] CI: include domU kernel messages in the console output log Marek Marczykowski-Górecki
` (10 subsequent siblings)
13 siblings, 0 replies; 26+ messages in thread
From: Marek Marczykowski-Górecki @ 2025-04-07 12:31 UTC (permalink / raw)
To: xen-devel
Cc: Marek Marczykowski-Górecki, Stefano Stabellini,
Doug Goldstein
This build is used for hardware tests, and some runners use XHCI console
Signed-off-by: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com>
Acked-by: Stefano Stabellini <sstabellini@kernel.org>
---
automation/gitlab-ci/build.yaml | 1 +
1 file changed, 1 insertion(+)
diff --git a/automation/gitlab-ci/build.yaml b/automation/gitlab-ci/build.yaml
index 2513908b059b..169bebe3c775 100644
--- a/automation/gitlab-ci/build.yaml
+++ b/automation/gitlab-ci/build.yaml
@@ -341,6 +341,7 @@ alpine-3.18-gcc-debug:
CONFIG_ARGO=y
CONFIG_UBSAN=y
CONFIG_UBSAN_FATAL=y
+ CONFIG_XHCI=y
debian-12-x86_64-gcc-debug:
extends: .gcc-x86-64-build-debug
--
git-series 0.9.1
^ permalink raw reply related [flat|nested] 26+ messages in thread* [PATCH v2 04/12] CI: include domU kernel messages in the console output log
2025-04-07 12:31 [PATCH v2 00/12] Several CI cleanups and improvements, plus yet another new runner Marek Marczykowski-Górecki
` (2 preceding siblings ...)
2025-04-07 12:31 ` [PATCH v2 03/12] CI: enable XHCI console in Xen debug build on Alpine Marek Marczykowski-Górecki
@ 2025-04-07 12:31 ` Marek Marczykowski-Górecki
2025-04-07 12:31 ` [PATCH v2 05/12] CI: increase verbosity of starting a domain Marek Marczykowski-Górecki
` (9 subsequent siblings)
13 siblings, 0 replies; 26+ messages in thread
From: Marek Marczykowski-Górecki @ 2025-04-07 12:31 UTC (permalink / raw)
To: xen-devel
Cc: Marek Marczykowski-Górecki, Stefano Stabellini,
Doug Goldstein
Signed-off-by: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com>
Acked-by: Stefano Stabellini <sstabellini@kernel.org>
---
automation/scripts/qubes-x86-64.sh | 2 ++
1 file changed, 2 insertions(+)
diff --git a/automation/scripts/qubes-x86-64.sh b/automation/scripts/qubes-x86-64.sh
index 0eac410f4168..7bf42d2e3d31 100755
--- a/automation/scripts/qubes-x86-64.sh
+++ b/automation/scripts/qubes-x86-64.sh
@@ -175,6 +175,8 @@ if [ -n "$domU_check" ]; then
rm var/run
echo "#!/bin/sh
+echo 8 > /proc/sys/kernel/printk
+
${domU_check}
" > etc/local.d/xen.start
chmod +x etc/local.d/xen.start
--
git-series 0.9.1
^ permalink raw reply related [flat|nested] 26+ messages in thread* [PATCH v2 05/12] CI: increase verbosity of starting a domain
2025-04-07 12:31 [PATCH v2 00/12] Several CI cleanups and improvements, plus yet another new runner Marek Marczykowski-Górecki
` (3 preceding siblings ...)
2025-04-07 12:31 ` [PATCH v2 04/12] CI: include domU kernel messages in the console output log Marek Marczykowski-Górecki
@ 2025-04-07 12:31 ` Marek Marczykowski-Górecki
2025-04-07 12:31 ` [PATCH v2 06/12] CI: consistently use DOCKER_CMD in makefiles Marek Marczykowski-Górecki
` (8 subsequent siblings)
13 siblings, 0 replies; 26+ messages in thread
From: Marek Marczykowski-Górecki @ 2025-04-07 12:31 UTC (permalink / raw)
To: xen-devel
Cc: Marek Marczykowski-Górecki, Doug Goldstein,
Stefano Stabellini
And start collecting qemu log earlier, so it isn't lost in case of a
timeout during domain startup.
Signed-off-by: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com>
---
Changes in v2:
- adjust xilinx-smoke-dom0-x86_64.sh too
---
automation/scripts/qemu-alpine-x86_64.sh | 2 +-
automation/scripts/qemu-smoke-dom0-arm32.sh | 2 +-
automation/scripts/qemu-smoke-dom0-arm64.sh | 2 +-
automation/scripts/qubes-x86-64.sh | 4 ++--
automation/scripts/xilinx-smoke-dom0-x86_64.sh | 4 ++--
5 files changed, 7 insertions(+), 7 deletions(-)
diff --git a/automation/scripts/qemu-alpine-x86_64.sh b/automation/scripts/qemu-alpine-x86_64.sh
index 1ff689b577e3..17e2141d625e 100755
--- a/automation/scripts/qemu-alpine-x86_64.sh
+++ b/automation/scripts/qemu-alpine-x86_64.sh
@@ -56,7 +56,7 @@ bash /etc/init.d/xencommons start
xl list
-xl create -c /root/test.cfg
+xl -vvv create -c /root/test.cfg
" > etc/local.d/xen.start
chmod +x etc/local.d/xen.start
diff --git a/automation/scripts/qemu-smoke-dom0-arm32.sh b/automation/scripts/qemu-smoke-dom0-arm32.sh
index e1cd83880928..0c60a66e25e3 100755
--- a/automation/scripts/qemu-smoke-dom0-arm32.sh
+++ b/automation/scripts/qemu-smoke-dom0-arm32.sh
@@ -21,7 +21,7 @@ echo "#!/bin/bash
xl list
-xl create -c /root/test.cfg
+xl -vvv create -c /root/test.cfg
" > ./root/xen.start
echo "bash /root/xen.start" >> ./etc/init.d/xen-watchdog
diff --git a/automation/scripts/qemu-smoke-dom0-arm64.sh b/automation/scripts/qemu-smoke-dom0-arm64.sh
index 4d22a124df11..8774a8701232 100755
--- a/automation/scripts/qemu-smoke-dom0-arm64.sh
+++ b/automation/scripts/qemu-smoke-dom0-arm64.sh
@@ -52,7 +52,7 @@ bash /etc/init.d/xencommons start
xl list
-xl create -c /root/test.cfg
+xl -vvv create -c /root/test.cfg
" > etc/local.d/xen.start
chmod +x etc/local.d/xen.start
diff --git a/automation/scripts/qubes-x86-64.sh b/automation/scripts/qubes-x86-64.sh
index 7bf42d2e3d31..17fcbd8fa7db 100755
--- a/automation/scripts/qubes-x86-64.sh
+++ b/automation/scripts/qubes-x86-64.sh
@@ -122,7 +122,6 @@ echo \"${passed}\"
"
dom0_check="
-tail -F /var/log/xen/qemu-dm-domU.log &
until grep -q \"^domU Welcome to Alpine Linux\" /var/log/xen/console/guest-domU.log; do
sleep 1
done
@@ -222,7 +221,8 @@ if [ -n "$domU_check" ]; then
echo "
# get domU console content into test log
tail -F /var/log/xen/console/guest-domU.log 2>/dev/null | sed -e \"s/^/(domU) /\" &
-xl create /etc/xen/domU.cfg
+tail -F /var/log/xen/qemu-dm-domU.log 2>/dev/null | sed -e \"s/^/(qemu-dm) /\" &
+xl -vvv create /etc/xen/domU.cfg
${dom0_check}
" >> etc/local.d/xen.start
else
diff --git a/automation/scripts/xilinx-smoke-dom0-x86_64.sh b/automation/scripts/xilinx-smoke-dom0-x86_64.sh
index f70cfdc1552a..7834ffbe0593 100755
--- a/automation/scripts/xilinx-smoke-dom0-x86_64.sh
+++ b/automation/scripts/xilinx-smoke-dom0-x86_64.sh
@@ -49,7 +49,7 @@ ifconfig xenbr0 up
ifconfig xenbr0 192.168.0.1
# get domU console content into test log
tail -F /var/log/xen/console/guest-domU.log 2>/dev/null | sed -e \"s/^/(domU) /\" &
-xl create /etc/xen/domU.cfg
+xl -vvv create /etc/xen/domU.cfg
set +x
until grep -q \"${DOMU_MSG}\" /var/log/xen/console/guest-domU.log; do
sleep 1
@@ -75,7 +75,7 @@ done | argo-exec -p 28333 -d 0 -- /bin/echo
"
DOM0_CMD="
insmod /root/xen-argo.ko
-xl create /etc/xen/domU.cfg
+xl -vvv create /etc/xen/domU.cfg
argo-exec -l -p 28333 -- /bin/echo
"
copy_dom0_files ()
--
git-series 0.9.1
^ permalink raw reply related [flat|nested] 26+ messages in thread* [PATCH v2 06/12] CI: consistently use DOCKER_CMD in makefiles
2025-04-07 12:31 [PATCH v2 00/12] Several CI cleanups and improvements, plus yet another new runner Marek Marczykowski-Górecki
` (4 preceding siblings ...)
2025-04-07 12:31 ` [PATCH v2 05/12] CI: increase verbosity of starting a domain Marek Marczykowski-Górecki
@ 2025-04-07 12:31 ` Marek Marczykowski-Górecki
2025-04-07 17:45 ` Andrew Cooper
2025-04-07 12:31 ` [PATCH v2 07/12] CI: wait for the network interface in PCI passthrough tests Marek Marczykowski-Górecki
` (7 subsequent siblings)
13 siblings, 1 reply; 26+ messages in thread
From: Marek Marczykowski-Górecki @ 2025-04-07 12:31 UTC (permalink / raw)
To: xen-devel
Cc: Marek Marczykowski-Górecki, Stefano Stabellini,
Doug Goldstein
This allows rebuilding containers using podman too.
Signed-off-by: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com>
Reviewed-by: Stefano Stabellini <sstabellini@kernel.org>
---
automation/build/Makefile | 4 ++--
automation/tests-artifacts/Makefile | 4 ++--
2 files changed, 4 insertions(+), 4 deletions(-)
diff --git a/automation/build/Makefile b/automation/build/Makefile
index 4df43b040777..fedf7524dacd 100644
--- a/automation/build/Makefile
+++ b/automation/build/Makefile
@@ -31,8 +31,8 @@ clean:
define CLEAN_RULE
.PHONY: clean-$(1)
clean-$(1):
- if [ -n "$$$$(docker image ls -q $(REGISTRY)/$(subst /,:,$(1)))" ]; then \
- docker image rm $(REGISTRY)/$(subst /,:,$(1)); \
+ if [ -n "$$$$($(DOCKER_CMD) image ls -q $(REGISTRY)/$(subst /,:,$(1)))" ]; then \
+ $(DOCKER_CMD) image rm $(REGISTRY)/$(subst /,:,$(1)); \
fi
endef
diff --git a/automation/tests-artifacts/Makefile b/automation/tests-artifacts/Makefile
index d055cd696bed..80a60a94f3f7 100644
--- a/automation/tests-artifacts/Makefile
+++ b/automation/tests-artifacts/Makefile
@@ -10,9 +10,9 @@ help:
@echo "To push container builds, set the env var PUSH"
%: %.dockerfile ## Builds containers
- docker build --pull -t $(REGISTRY)/$(@D):$(@F) -f $< $(<D)
+ $(DOCKER_CMD) build --pull -t $(REGISTRY)/$(@D):$(@F) -f $< $(<D)
@if [ ! -z $${PUSH+x} ]; then \
- docker push $(REGISTRY)/$(@D):$(@F); \
+ $(DOCKER_CMD) push $(REGISTRY)/$(@D):$(@F); \
fi
.PHONY: all
--
git-series 0.9.1
^ permalink raw reply related [flat|nested] 26+ messages in thread* Re: [PATCH v2 06/12] CI: consistently use DOCKER_CMD in makefiles
2025-04-07 12:31 ` [PATCH v2 06/12] CI: consistently use DOCKER_CMD in makefiles Marek Marczykowski-Górecki
@ 2025-04-07 17:45 ` Andrew Cooper
0 siblings, 0 replies; 26+ messages in thread
From: Andrew Cooper @ 2025-04-07 17:45 UTC (permalink / raw)
To: Marek Marczykowski-Górecki, xen-devel
Cc: Stefano Stabellini, Doug Goldstein
On 07/04/2025 1:31 pm, Marek Marczykowski-Górecki wrote:
> This allows rebuilding containers using podman too.
>
> Signed-off-by: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com>
> Reviewed-by: Stefano Stabellini <sstabellini@kernel.org>
> ---
> automation/build/Makefile | 4 ++--
> automation/tests-artifacts/Makefile | 4 ++--
> 2 files changed, 4 insertions(+), 4 deletions(-)
>
> diff --git a/automation/build/Makefile b/automation/build/Makefile
> index 4df43b040777..fedf7524dacd 100644
> --- a/automation/build/Makefile
> +++ b/automation/build/Makefile
> @@ -31,8 +31,8 @@ clean:
> define CLEAN_RULE
> .PHONY: clean-$(1)
> clean-$(1):
> - if [ -n "$$$$(docker image ls -q $(REGISTRY)/$(subst /,:,$(1)))" ]; then \
> - docker image rm $(REGISTRY)/$(subst /,:,$(1)); \
> + if [ -n "$$$$($(DOCKER_CMD) image ls -q $(REGISTRY)/$(subst /,:,$(1)))" ]; then \
> + $(DOCKER_CMD) image rm $(REGISTRY)/$(subst /,:,$(1)); \
> fi
>
> endef
> diff --git a/automation/tests-artifacts/Makefile b/automation/tests-artifacts/Makefile
> index d055cd696bed..80a60a94f3f7 100644
> --- a/automation/tests-artifacts/Makefile
> +++ b/automation/tests-artifacts/Makefile
> @@ -10,9 +10,9 @@ help:
> @echo "To push container builds, set the env var PUSH"
>
> %: %.dockerfile ## Builds containers
> - docker build --pull -t $(REGISTRY)/$(@D):$(@F) -f $< $(<D)
> + $(DOCKER_CMD) build --pull -t $(REGISTRY)/$(@D):$(@F) -f $< $(<D)
> @if [ ! -z $${PUSH+x} ]; then \
> - docker push $(REGISTRY)/$(@D):$(@F); \
> + $(DOCKER_CMD) push $(REGISTRY)/$(@D):$(@F); \
> fi
>
> .PHONY: all
I'm going to apply this commit to the test artefacts repo too, as it's
got a copy of this Makefile in need of the same change.
^ permalink raw reply [flat|nested] 26+ messages in thread
* [PATCH v2 07/12] CI: wait for the network interface in PCI passthrough tests
2025-04-07 12:31 [PATCH v2 00/12] Several CI cleanups and improvements, plus yet another new runner Marek Marczykowski-Górecki
` (5 preceding siblings ...)
2025-04-07 12:31 ` [PATCH v2 06/12] CI: consistently use DOCKER_CMD in makefiles Marek Marczykowski-Górecki
@ 2025-04-07 12:31 ` Marek Marczykowski-Górecki
2025-04-07 12:31 ` [PATCH v2 09/12] CI: adjust resolving network interface into PCI device Marek Marczykowski-Górecki
` (6 subsequent siblings)
13 siblings, 0 replies; 26+ messages in thread
From: Marek Marczykowski-Górecki @ 2025-04-07 12:31 UTC (permalink / raw)
To: xen-devel
Cc: Marek Marczykowski-Górecki, Stefano Stabellini,
Doug Goldstein
The network driver initializes asynchronously, and it may not be ready
yet by the time the startup script is called. This is especially the
case for USB network adapter (where the PCI device is the USB
controller) in the upcoming runner.
Don't bother about separate timeout - test timeout will cover this part
too.
Signed-off-by: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com>
Reviewed-by: Stefano Stabellini <sstabellini@kernel.org>
---
automation/scripts/qubes-x86-64.sh | 1 +
1 file changed, 1 insertion(+)
diff --git a/automation/scripts/qubes-x86-64.sh b/automation/scripts/qubes-x86-64.sh
index 17fcbd8fa7db..9359e8914fb2 100755
--- a/automation/scripts/qubes-x86-64.sh
+++ b/automation/scripts/qubes-x86-64.sh
@@ -104,6 +104,7 @@ on_reboot = "destroy"
domU_check="
set -x -e
interface=eth0
+while ! [ -e \"/sys/class/net/\$interface\" ]; do sleep 1; done
ip link set \"\$interface\" up
timeout 30s udhcpc -i \"\$interface\"
pingip=\$(ip -o -4 r show default|cut -f 3 -d ' ')
--
git-series 0.9.1
^ permalink raw reply related [flat|nested] 26+ messages in thread* [PATCH v2 09/12] CI: adjust resolving network interface into PCI device
2025-04-07 12:31 [PATCH v2 00/12] Several CI cleanups and improvements, plus yet another new runner Marek Marczykowski-Górecki
` (6 preceding siblings ...)
2025-04-07 12:31 ` [PATCH v2 07/12] CI: wait for the network interface in PCI passthrough tests Marek Marczykowski-Górecki
@ 2025-04-07 12:31 ` Marek Marczykowski-Górecki
2025-04-07 12:31 ` [PATCH v2 10/12] CI: add AMD Zen 4 HW runner Marek Marczykowski-Górecki
` (5 subsequent siblings)
13 siblings, 0 replies; 26+ messages in thread
From: Marek Marczykowski-Górecki @ 2025-04-07 12:31 UTC (permalink / raw)
To: xen-devel
Cc: Marek Marczykowski-Górecki, Stefano Stabellini,
Doug Goldstein
Change how PCI device lookup is done to handle also USB devices, in
which case get the USB controller. Instead of taking basename of the
'device' symlink, resolve the full path (example:
/sys/devices/pci0000:00/0000:00:09.0/usb4/4-7/4-7:1.0) and take the
first part after pci0000:00. Theoretically it could be a bridge, but VM
has flat PCI topology.
Signed-off-by: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com>
Reviewed-by: Stefano Stabellini <sstabellini@kernel.org>
---
automation/scripts/qubes-x86-64.sh | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/automation/scripts/qubes-x86-64.sh b/automation/scripts/qubes-x86-64.sh
index 9359e8914fb2..861e302d845b 100755
--- a/automation/scripts/qubes-x86-64.sh
+++ b/automation/scripts/qubes-x86-64.sh
@@ -110,7 +110,8 @@ timeout 30s udhcpc -i \"\$interface\"
pingip=\$(ip -o -4 r show default|cut -f 3 -d ' ')
ping -c 10 \"\$pingip\"
echo domU started
-pcidevice=\$(basename \$(readlink /sys/class/net/\$interface/device))
+pcidevice=\$(realpath /sys/class/net/\$interface/device |
+ sed 's#.*pci0000:00/\\([^/]*\\).*#\\1#')
lspci -vs \$pcidevice
"
if [ -n "$PCIDEV_INTR" ]; then
--
git-series 0.9.1
^ permalink raw reply related [flat|nested] 26+ messages in thread* [PATCH v2 10/12] CI: add AMD Zen 4 HW runner
2025-04-07 12:31 [PATCH v2 00/12] Several CI cleanups and improvements, plus yet another new runner Marek Marczykowski-Górecki
` (7 preceding siblings ...)
2025-04-07 12:31 ` [PATCH v2 09/12] CI: adjust resolving network interface into PCI device Marek Marczykowski-Górecki
@ 2025-04-07 12:31 ` Marek Marczykowski-Górecki
2025-04-07 12:31 ` [PATCH v2 11/12] CI: avoid repacking initrd as part of the test job Marek Marczykowski-Górecki
` (4 subsequent siblings)
13 siblings, 0 replies; 26+ messages in thread
From: Marek Marczykowski-Górecki @ 2025-04-07 12:31 UTC (permalink / raw)
To: xen-devel
Cc: Marek Marczykowski-Górecki, Stefano Stabellini,
Doug Goldstein
This is AMD Ryzen 5 7640U in Framework Laptop AMD.
It has several things different than the other runners.
First of all, the console is using XHCI debug capability.
And then, this system doesn't have normal wired ethernet. But is has one
on USB, and this one is used for booting. For this, enable
CONFIG_USB_RTL8152 in the Linux build.
Include some basic tests, plus PCI passthrough.
This machine doesn't support S3. S0ix pretends to be working with
Qubes's dom0 (kernel + userspace), but it hangs on suspend with Alpine
used in test. But even when it doesn't hang, it doesn't really reach
deep sleep, so skip this test for now here.
Signed-off-by: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com>
Reviewed-by: Stefano Stabellini <sstabellini@kernel.org>
---
Requires containers rebuild, especially tests-artifacts/kernel/6.12.21.
Do not apply yet, until issues found by those tests are fixed. That
would be at least third issue found using this system, so lets have it
in CI.
---
automation/gitlab-ci/test.yaml | 56 +++++++++++++-
automation/tests-artifacts/kernel/6.12.21.dockerfile | 2 +-
2 files changed, 58 insertions(+)
diff --git a/automation/gitlab-ci/test.yaml b/automation/gitlab-ci/test.yaml
index a13dd040bd26..feb2dd10ab30 100644
--- a/automation/gitlab-ci/test.yaml
+++ b/automation/gitlab-ci/test.yaml
@@ -188,6 +188,16 @@
tags:
- qubes-hw11
+.zen4-x86-64:
+ extends: .adl-x86-64
+ variables:
+ PCIDEV: "c3:00.4"
+ PCIDEV_INTR: "MSI-X"
+ CONSOLE_OPTS: "console=xhci dbgp=xhci@pcic1:00.3,share=yes"
+ SUT_ADDR: test-12.testnet
+ tags:
+ - qubes-hw12
+
# Test jobs
build-each-commit-gcc:
extends: .test-jobs-common
@@ -479,6 +489,52 @@ zen3p-tools-tests-pvh-x86-64-gcc-debug:
- *x86-64-test-needs
- alpine-3.18-gcc-debug
+zen4-smoke-x86-64-gcc-debug:
+ extends: .zen4-x86-64
+ script:
+ - ./automation/scripts/qubes-x86-64.sh 2>&1 | tee ${LOGFILE}
+ needs:
+ - *x86-64-test-needs
+ - alpine-3.18-gcc-debug
+
+zen4-smoke-x86-64-dom0pvh gcc-debug:
+ extends: .zen4-x86-64
+ script:
+ - ./automation/scripts/qubes-x86-64.sh dom0pvh 2>&1 | tee ${LOGFILE}
+ needs:
+ - *x86-64-test-needs
+ - alpine-3.18-gcc-debug
+
+zen4-pci-hvm-x86-64-gcc-debug:
+ extends: .zen4-x86-64
+ script:
+ - ./automation/scripts/qubes-x86-64.sh pci-hvm 2>&1 | tee ${LOGFILE}
+ needs:
+ - *x86-64-test-needs
+ - alpine-3.18-gcc-debug
+
+zen4-tools-tests-pv-x86-64-gcc-debug:
+ extends: .zen4-x86-64
+ script:
+ - ./automation/scripts/qubes-x86-64.sh tools-tests-pv 2>&1 | tee ${LOGFILE}
+ artifacts:
+ reports:
+ junit: tests-junit.xml
+ needs:
+ - *x86-64-test-needs
+ - alpine-3.18-gcc-debug
+
+zen4-tools-tests-pvh-x86-64-gcc-debug:
+ extends: .zen4-x86-64
+ script:
+ - ./automation/scripts/qubes-x86-64.sh tools-tests-pvh 2>&1 | tee ${LOGFILE}
+ artifacts:
+ reports:
+ junit: tests-junit.xml
+ needs:
+ - *x86-64-test-needs
+ - alpine-3.18-gcc-debug
+
qemu-smoke-dom0-arm64-gcc:
extends: .qemu-arm64
script:
diff --git a/automation/tests-artifacts/kernel/6.12.21.dockerfile b/automation/tests-artifacts/kernel/6.12.21.dockerfile
index d7d34031cab5..4e0a68ec61eb 100644
--- a/automation/tests-artifacts/kernel/6.12.21.dockerfile
+++ b/automation/tests-artifacts/kernel/6.12.21.dockerfile
@@ -19,6 +19,7 @@ RUN apt-get update && \
flex \
bison \
libelf-dev \
+ libssl-dev \
&& \
apt-get autoremove -y && \
apt-get clean && \
@@ -33,6 +34,7 @@ RUN curl -fsSLO https://cdn.kernel.org/pub/linux/kernel/v6.x/linux-"$LINUX_VERSI
scripts/config --enable BRIDGE && \
scripts/config --enable IGC && \
scripts/config --enable TUN && \
+ scripts/config --enable USB_RTL8152 && \
cp .config .config.orig && \
cat .config.orig | grep XEN | grep =m |sed 's/=m/=y/g' >> .config && \
make -j$(nproc) bzImage && \
--
git-series 0.9.1
^ permalink raw reply related [flat|nested] 26+ messages in thread* [PATCH v2 11/12] CI: avoid repacking initrd as part of the test job
2025-04-07 12:31 [PATCH v2 00/12] Several CI cleanups and improvements, plus yet another new runner Marek Marczykowski-Górecki
` (8 preceding siblings ...)
2025-04-07 12:31 ` [PATCH v2 10/12] CI: add AMD Zen 4 HW runner Marek Marczykowski-Górecki
@ 2025-04-07 12:31 ` Marek Marczykowski-Górecki
2025-04-07 12:31 ` [PATCH v2 12/12] CI: save toolstack artifact as cpio.gz Marek Marczykowski-Górecki
` (3 subsequent siblings)
13 siblings, 0 replies; 26+ messages in thread
From: Marek Marczykowski-Górecki @ 2025-04-07 12:31 UTC (permalink / raw)
To: xen-devel
Cc: Marek Marczykowski-Górecki, Doug Goldstein,
Stefano Stabellini
The prerequisite for this is building initrd.cpio.gz, not only
initrd.tar.gz. That's a change in the test-artifacts repository.
Having that, do not repack the whole initrd, but only pack modified
files and rely on Linux handling of concatenated archives.
This allows packing just test-related files (which includes the whole
toolstack), instead of the whole initrd.
DomU initrd handling is a bit more complicated thing. It's sent to the
target host as part of the dom0 initrd. But to modify/extend it, the
"overlay" need to be applied already on the target system (otherwise
controller host would need to repack it anyway). So, make the
initrd.cpio.gz in test-artifacts include boot/initrd-domU already, and
add test-specific files as boot/initrd-domU-overlay that is concatenated
by dom0 boot script.
Since full initrd is not unpacked now when preparing domU (and dom0)
rootfs, a couple of minor changes are needed to how they are prepared.
This includes creating whole etc/issue file, instead of modifying
existing one, and a need to create a couple directories.
And since there is now initrd.cpio.gz artifact, rename temporary domU
initrd to initrd-domU.cpio.gz
Finally, move adding "rc_verbose=yes" to /etc/rc.conf to initrd
building.
Signed-off-by: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com>
---
New in v2
In fact, alpine/3.18.dockerfile in this repository is not what
is used to build initrd.cpio.gz for test here, but add it anyway in case
there are some other tests using it that I missed. If it's really
unused, it probably should be removed, but that's a separate patch.
Jobs xilinx-* are untested, as I don't have necessary access.
---
automation/gitlab-ci/build.yaml | 5 +-
automation/scripts/qemu-alpine-x86_64.sh | 11 +--
automation/scripts/qemu-smoke-dom0-arm64.sh | 11 +--
automation/scripts/qemu-smoke-dom0less-arm64.sh | 7 +--
automation/scripts/qubes-x86-64.sh | 29 +++-----
automation/scripts/xilinx-smoke-dom0-x86_64.sh | 23 +++---
automation/scripts/xilinx-smoke-dom0less-arm64.sh | 14 +---
automation/tests-artifacts/alpine/3.18-arm64v8.dockerfile | 5 +-
automation/tests-artifacts/alpine/3.18.dockerfile | 1 +-
9 files changed, 55 insertions(+), 51 deletions(-)
diff --git a/automation/gitlab-ci/build.yaml b/automation/gitlab-ci/build.yaml
index d41f27fc94bf..8e1ff0178469 100644
--- a/automation/gitlab-ci/build.yaml
+++ b/automation/gitlab-ci/build.yaml
@@ -268,10 +268,13 @@ alpine-3.18-arm64-rootfs-export:
extends: .test-jobs-artifact-common
image: registry.gitlab.com/xen-project/xen/tests-artifacts/alpine:3.18-arm64v8
script:
- - mkdir binaries && cp /initrd.tar.gz binaries/initrd.tar.gz
+ - mkdir binaries
+ - cp /initrd.tar.gz binaries/initrd.tar.gz
+ - cp /initrd.cpio.gz binaries/initrd.cpio.gz
artifacts:
paths:
- binaries/initrd.tar.gz
+ - binaries/initrd.cpio.gz
tags:
- arm64
diff --git a/automation/scripts/qemu-alpine-x86_64.sh b/automation/scripts/qemu-alpine-x86_64.sh
index 17e2141d625e..9b47b662c3f2 100755
--- a/automation/scripts/qemu-alpine-x86_64.sh
+++ b/automation/scripts/qemu-alpine-x86_64.sh
@@ -25,20 +25,19 @@ mount -t devtmpfs devtmpfs /dev
chmod +x initrd/init
# DomU rootfs
cd initrd
-find . | cpio --create --format='newc' | gzip > ../initrd.cpio.gz
+find . | cpio --create --format='newc' | gzip > ../initrd-domU.cpio.gz
cd ..
# initrd.tar.gz is Dom0 rootfs
mkdir -p rootfs
cd rootfs
-tar xvzf ../initrd.tar.gz
mkdir proc
mkdir run
mkdir srv
mkdir sys
-rm var/run
cp -ar ../dist/install/* .
-mv ../initrd.cpio.gz ./root
+mkdir -p root etc/local.d
+mv ../initrd-domU.cpio.gz ./root/initrd.cpio.gz
cp ../bzImage ./root
echo "name=\"test\"
memory=512
@@ -60,9 +59,9 @@ xl -vvv create -c /root/test.cfg
" > etc/local.d/xen.start
chmod +x etc/local.d/xen.start
-echo "rc_verbose=yes" >> etc/rc.conf
# rebuild Dom0 rootfs
-find . |cpio -H newc -o|gzip > ../xen-rootfs.cpio.gz
+cp ../initrd.cpio.gz ../xen-rootfs.cpio.gz
+find . |cpio -H newc -o|gzip >> ../xen-rootfs.cpio.gz
cd ../..
cat >> binaries/pxelinux.0 << EOF
diff --git a/automation/scripts/qemu-smoke-dom0-arm64.sh b/automation/scripts/qemu-smoke-dom0-arm64.sh
index 8774a8701232..533b286528f1 100755
--- a/automation/scripts/qemu-smoke-dom0-arm64.sh
+++ b/automation/scripts/qemu-smoke-dom0-arm64.sh
@@ -24,19 +24,18 @@ mount -t devtmpfs devtmpfs /dev
/bin/sh" > initrd/init
chmod +x initrd/init
cd initrd
-find . | cpio --create --format='newc' | gzip > ../initrd.cpio.gz
+find . | cpio --create --format='newc' | gzip > ../initrd-domU.cpio.gz
cd ..
mkdir -p rootfs
cd rootfs
-tar xvzf ../initrd.tar.gz
mkdir proc
mkdir run
mkdir srv
mkdir sys
-rm var/run
cp -ar ../dist/install/* .
-mv ../initrd.cpio.gz ./root
+mkdir -p etc/local.d root
+mv ../initrd-domU.cpio.gz ./root/initrd.cpio.gz
cp ../Image ./root
echo "name=\"test\"
memory=512
@@ -56,8 +55,8 @@ xl -vvv create -c /root/test.cfg
" > etc/local.d/xen.start
chmod +x etc/local.d/xen.start
-echo "rc_verbose=yes" >> etc/rc.conf
-find . |cpio -H newc -o|gzip > ../xen-rootfs.cpio.gz
+cp ../initrd.cpio.gz ../xen-rootfs.cpio.gz
+find . |cpio -H newc -o|gzip >> ../xen-rootfs.cpio.gz
cd ../..
# XXX QEMU looks for "efi-virtio.rom" even if it is unneeded
diff --git a/automation/scripts/qemu-smoke-dom0less-arm64.sh b/automation/scripts/qemu-smoke-dom0less-arm64.sh
index f72d20936181..70085e26c139 100755
--- a/automation/scripts/qemu-smoke-dom0less-arm64.sh
+++ b/automation/scripts/qemu-smoke-dom0less-arm64.sh
@@ -117,13 +117,12 @@ cd ..
# DOM0 rootfs
mkdir -p rootfs
cd rootfs
-tar xzf ../binaries/initrd.tar.gz
mkdir proc
mkdir run
mkdir srv
mkdir sys
-rm var/run
cp -ar ../binaries/dist/install/* .
+mkdir -p etc/local.d
echo "#!/bin/bash
@@ -142,8 +141,8 @@ xl network-attach 1 type=vif
${dom0_check}
" > etc/local.d/xen.start
chmod +x etc/local.d/xen.start
-echo "rc_verbose=yes" >> etc/rc.conf
-find . | cpio -H newc -o | gzip > ../binaries/dom0-rootfs.cpio.gz
+cp ../binaries/initrd.cpio.gz ../binaries/dom0-rootfs.cpio.gz
+find . | cpio -H newc -o | gzip >> ../binaries/dom0-rootfs.cpio.gz
cd ..
# ImageBuilder
diff --git a/automation/scripts/qubes-x86-64.sh b/automation/scripts/qubes-x86-64.sh
index 861e302d845b..399d72e0600d 100755
--- a/automation/scripts/qubes-x86-64.sh
+++ b/automation/scripts/qubes-x86-64.sh
@@ -167,13 +167,10 @@ if [ -n "$domU_check" ]; then
# DomU
mkdir -p rootfs
cd rootfs
- # fakeroot is needed to preserve device nodes in rootless podman container
- fakeroot -s ../fakeroot-save tar xzf ../binaries/initrd.tar.gz
mkdir proc
mkdir run
- mkdir srv
mkdir sys
- rm var/run
+ mkdir -p etc/local.d
echo "#!/bin/sh
echo 8 > /proc/sys/kernel/printk
@@ -181,23 +178,23 @@ echo 8 > /proc/sys/kernel/printk
${domU_check}
" > etc/local.d/xen.start
chmod +x etc/local.d/xen.start
- echo "rc_verbose=yes" >> etc/rc.conf
- sed -i -e 's/^Welcome/domU \0/' etc/issue
- find . | fakeroot -i ../fakeroot-save cpio -H newc -o | gzip > ../binaries/domU-rootfs.cpio.gz
+ echo "domU Welcome to Alpine Linux 3.18
+Kernel \r on an \m (\l)
+
+" > etc/issue
+ find . | cpio -H newc -o | gzip > ../binaries/domU-rootfs-overlay.cpio.gz
cd ..
rm -rf rootfs
fi
-# DOM0 rootfs
+# DOM0 rootfs - this will be an overlay over alpine's initrd
mkdir -p rootfs
cd rootfs
-fakeroot -s ../fakeroot-save tar xzf ../binaries/initrd.tar.gz
mkdir boot
mkdir proc
mkdir run
-mkdir srv
mkdir sys
-rm var/run
+mkdir -p etc/local.d
cp -ar ../binaries/dist/install/* .
cp -ar ../binaries/tests .
cp -a ../automation/scripts/run-tools-tests tests/
@@ -221,6 +218,8 @@ fi
if [ -n "$domU_check" ]; then
echo "
+# append test-specific files to domU initrd
+cat /boot/initrd-domU-overlay >> boot/initrd-domU
# get domU console content into test log
tail -F /var/log/xen/console/guest-domU.log 2>/dev/null | sed -e \"s/^/(domU) /\" &
tail -F /var/log/xen/qemu-dm-domU.log 2>/dev/null | sed -e \"s/^/(qemu-dm) /\" &
@@ -234,18 +233,18 @@ fi
chmod +x etc/local.d/xen.start
echo "$domU_config" > etc/xen/domU.cfg
-echo "rc_verbose=yes" >> etc/rc.conf
echo "XENCONSOLED_TRACE=all" >> etc/default/xencommons
echo "QEMU_XEN=/bin/false" >> etc/default/xencommons
mkdir -p var/log/xen/console
cp ../binaries/bzImage boot/vmlinuz
if [ -n "$domU_check" ]; then
- cp ../binaries/domU-rootfs.cpio.gz boot/initrd-domU
+ cp ../binaries/domU-rootfs-overlay.cpio.gz boot/initrd-domU-overlay
fi
-find . | fakeroot -i ../fakeroot-save cpio -H newc -o | gzip > ../binaries/dom0-rootfs.cpio.gz
+# take base initrd and append test-specific files
+cp ../binaries/initrd.cpio.gz ../binaries/dom0-rootfs.cpio.gz
+find . | cpio -H newc -o | gzip >> ../binaries/dom0-rootfs.cpio.gz
cd ..
-
TFTP=/scratch/gitlab-runner/tftp
CONTROLLER=control@thor.testnet
diff --git a/automation/scripts/xilinx-smoke-dom0-x86_64.sh b/automation/scripts/xilinx-smoke-dom0-x86_64.sh
index 7834ffbe0593..6b9f0e4a8dc7 100755
--- a/automation/scripts/xilinx-smoke-dom0-x86_64.sh
+++ b/automation/scripts/xilinx-smoke-dom0-x86_64.sh
@@ -80,6 +80,7 @@ argo-exec -l -p 28333 -- /bin/echo
"
copy_dom0_files ()
{
+ mkdir -p root usr/local/lib
cp "${WORKDIR}/binaries/xen-argo.ko" "root/"
cp -ar "${WORKDIR}/binaries/lib/"* "usr/local/lib/"
cp "${WORKDIR}/binaries/argo-exec" "usr/local/bin/"
@@ -92,9 +93,8 @@ fi
# Set up domU rootfs.
mkdir -p rootfs
cd rootfs
-tar xzf ../binaries/initrd.tar.gz
mkdir proc run srv sys
-rm var/run
+mkdir -p etc/local.d
echo "#!/bin/sh
set -x
export LD_LIBRARY_PATH=/usr/local/lib
@@ -102,36 +102,39 @@ PATH=/usr/local/bin:/usr/local/sbin:\$PATH
${DOMU_CMD}
" > etc/local.d/xen.start
chmod +x etc/local.d/xen.start
-echo "rc_verbose=yes" >> etc/rc.conf
-sed -i -e 's/^Welcome/domU \0/' etc/issue
+echo "domU Welcome to Alpine Linux 3.18
+Kernel \r on an \m (\l)
+
+" > etc/issue
copy_domU_files
-find . | cpio -H newc -o | gzip > ../binaries/domU-rootfs.cpio.gz
+find . | cpio -H newc -o | gzip > ../binaries/domU-rootfs-overlay.cpio.gz
cd ..
rm -rf rootfs
# Set up dom0 rootfs.
mkdir -p rootfs
cd rootfs
-tar xzf ../binaries/initrd.tar.gz
mkdir boot proc run srv sys
-rm var/run
cp -ar ../binaries/dist/install/* .
+mkdir -p etc/local.d
echo "#!/bin/bash
set -x
export LD_LIBRARY_PATH=/usr/local/lib
bash /etc/init.d/xencommons start
+# append test-specific files to domU initrd
+cat /boot/initrd-domU-overlay >> boot/initrd-domU
${DOM0_CMD}
" > etc/local.d/xen.start
chmod +x etc/local.d/xen.start
echo "${DOMU_CFG}${DOMU_CFG_EXTRA}" > etc/xen/domU.cfg
-echo "rc_verbose=yes" >> etc/rc.conf
echo "XENCONSOLED_TRACE=all" >> etc/default/xencommons
echo "QEMU_XEN=/bin/false" >> etc/default/xencommons
mkdir -p var/log/xen/console
cp ../binaries/bzImage boot/vmlinuz
-cp ../binaries/domU-rootfs.cpio.gz boot/initrd-domU
+cp ../binaries/domU-rootfs-overlay.cpio.gz boot/initrd-domU-overlay
copy_dom0_files
-find . | cpio -H newc -o | gzip > ../binaries/dom0-rootfs.cpio.gz
+cp ../binaries/initrd.cpio.gz ../binaries/dom0-rootfs.cpio.gz
+find . | cpio -H newc -o | gzip >> ../binaries/dom0-rootfs.cpio.gz
cd ..
# Load software into TFTP server directory.
diff --git a/automation/scripts/xilinx-smoke-dom0less-arm64.sh b/automation/scripts/xilinx-smoke-dom0less-arm64.sh
index b24ad11b8cac..81be60e2026e 100755
--- a/automation/scripts/xilinx-smoke-dom0less-arm64.sh
+++ b/automation/scripts/xilinx-smoke-dom0less-arm64.sh
@@ -41,33 +41,31 @@ fi
# DomU
mkdir -p rootfs
cd rootfs
-tar xzf ../binaries/initrd.tar.gz
mkdir proc
mkdir run
mkdir srv
mkdir sys
-rm var/run
+mkdir -p etc/local.d
echo "#!/bin/sh
${domU_check}
/bin/sh" > etc/local.d/xen.start
chmod +x etc/local.d/xen.start
-echo "rc_verbose=yes" >> etc/rc.conf
-find . | cpio -H newc -o | gzip > ../binaries/domU-rootfs.cpio.gz
+cp ../binaries/initrd.cpio.gz ../binaries/domU-rootfs.cpio.gz
+find . | cpio -H newc -o | gzip >> ../binaries/domU-rootfs.cpio.gz
cd ..
rm -rf rootfs
# DOM0 rootfs
mkdir -p rootfs
cd rootfs
-tar xzf ../binaries/initrd.tar.gz
mkdir proc
mkdir run
mkdir srv
mkdir sys
-rm var/run
cp -ar ../binaries/dist/install/* .
+mkdir -p etc/local.d
echo "#!/bin/bash
export LD_LIBRARY_PATH=/usr/local/lib
@@ -78,8 +76,8 @@ bash /etc/init.d/xencommons start
${dom0_check}
" > etc/local.d/xen.start
chmod +x etc/local.d/xen.start
-echo "rc_verbose=yes" >> etc/rc.conf
-find . | cpio -H newc -o | gzip > ../binaries/dom0-rootfs.cpio.gz
+cp ../binaries/initrd.cpio.gz ../binaries/dom0-rootfs.cpio.gz
+find . | cpio -H newc -o | gzip >> ../binaries/dom0-rootfs.cpio.gz
cd ..
diff --git a/automation/tests-artifacts/alpine/3.18-arm64v8.dockerfile b/automation/tests-artifacts/alpine/3.18-arm64v8.dockerfile
index a4542f703997..78d465dc6a39 100644
--- a/automation/tests-artifacts/alpine/3.18-arm64v8.dockerfile
+++ b/automation/tests-artifacts/alpine/3.18-arm64v8.dockerfile
@@ -60,7 +60,10 @@ RUN \
echo "ttyS0::respawn:/sbin/getty -L ttyS0 115200 vt100" >> /etc/inittab && \
echo "hvc0::respawn:/sbin/getty -L hvc0 115200 vt100" >> /etc/inittab && \
passwd -d "root" root && \
+ echo "rc_verbose=yes" >> /etc/rc.conf && \
\
# Create rootfs
cd / && \
- tar cvzf /initrd.tar.gz bin dev etc home init lib mnt opt root sbin usr var
+ tar cvzf /initrd.tar.gz bin dev etc home init lib mnt opt root sbin usr var && \
+ find bin dev etc home init lib mnt opt root sbin usr var |\
+ cpio -o -H newc | gzip > /initrd.cpio.gz
diff --git a/automation/tests-artifacts/alpine/3.18.dockerfile b/automation/tests-artifacts/alpine/3.18.dockerfile
index 311a92889b87..ff124fef49a8 100644
--- a/automation/tests-artifacts/alpine/3.18.dockerfile
+++ b/automation/tests-artifacts/alpine/3.18.dockerfile
@@ -61,6 +61,7 @@ RUN \
echo "hvc0::respawn:/sbin/getty -L hvc0 115200 vt100" >> /etc/inittab && \
echo > /etc/modules && \
passwd -d "root" root && \
+ echo "rc_verbose=yes" >> /etc/rc.conf && \
\
# Create rootfs
cd / && \
--
git-series 0.9.1
^ permalink raw reply related [flat|nested] 26+ messages in thread* [PATCH v2 12/12] CI: save toolstack artifact as cpio.gz
2025-04-07 12:31 [PATCH v2 00/12] Several CI cleanups and improvements, plus yet another new runner Marek Marczykowski-Górecki
` (9 preceding siblings ...)
2025-04-07 12:31 ` [PATCH v2 11/12] CI: avoid repacking initrd as part of the test job Marek Marczykowski-Górecki
@ 2025-04-07 12:31 ` Marek Marczykowski-Górecki
2025-04-07 12:31 ` [PATCH test-artifacts v2 13/12] scripts: add rc_verbose=yes to /etc/rc.conf Marek Marczykowski-Górecki
` (2 subsequent siblings)
13 siblings, 0 replies; 26+ messages in thread
From: Marek Marczykowski-Górecki @ 2025-04-07 12:31 UTC (permalink / raw)
To: xen-devel
Cc: Marek Marczykowski-Górecki, Doug Goldstein,
Stefano Stabellini
This avoids the need to re-compress it in every test job.
Signed-off-by: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com>
---
New in v2
---
automation/scripts/build | 2 +-
automation/scripts/qemu-alpine-x86_64.sh | 2 +-
automation/scripts/qemu-smoke-dom0-arm64.sh | 2 +-
automation/scripts/qemu-smoke-dom0less-arm64.sh | 2 +-
automation/scripts/qubes-x86-64.sh | 4 +++-
automation/scripts/xilinx-smoke-dom0-x86_64.sh | 4 ++--
automation/scripts/xilinx-smoke-dom0less-arm64.sh | 2 +-
7 files changed, 10 insertions(+), 8 deletions(-)
diff --git a/automation/scripts/build b/automation/scripts/build
index 522efe774ef3..365534895047 100755
--- a/automation/scripts/build
+++ b/automation/scripts/build
@@ -108,7 +108,7 @@ else
# Note: Some smoke tests depending on finding binaries/xen on a full build
# even though dist/ contains everything, while some containers don't even
# build Xen
- cp -r dist binaries/
+ (cd dist/install; find | cpio -o -H newc | gzip) > binaries/toolstack.cpio.gz
cp -r tools/tests binaries/
collect_xen_artefacts
fi
diff --git a/automation/scripts/qemu-alpine-x86_64.sh b/automation/scripts/qemu-alpine-x86_64.sh
index 9b47b662c3f2..81e8ff24327a 100755
--- a/automation/scripts/qemu-alpine-x86_64.sh
+++ b/automation/scripts/qemu-alpine-x86_64.sh
@@ -35,7 +35,6 @@ mkdir proc
mkdir run
mkdir srv
mkdir sys
-cp -ar ../dist/install/* .
mkdir -p root etc/local.d
mv ../initrd-domU.cpio.gz ./root/initrd.cpio.gz
cp ../bzImage ./root
@@ -61,6 +60,7 @@ xl -vvv create -c /root/test.cfg
chmod +x etc/local.d/xen.start
# rebuild Dom0 rootfs
cp ../initrd.cpio.gz ../xen-rootfs.cpio.gz
+cat ../toolstack.cpio.gz >> ../xen-rootfs.cpio.gz
find . |cpio -H newc -o|gzip >> ../xen-rootfs.cpio.gz
cd ../..
diff --git a/automation/scripts/qemu-smoke-dom0-arm64.sh b/automation/scripts/qemu-smoke-dom0-arm64.sh
index 533b286528f1..2e5246f2c9b6 100755
--- a/automation/scripts/qemu-smoke-dom0-arm64.sh
+++ b/automation/scripts/qemu-smoke-dom0-arm64.sh
@@ -33,7 +33,6 @@ mkdir proc
mkdir run
mkdir srv
mkdir sys
-cp -ar ../dist/install/* .
mkdir -p etc/local.d root
mv ../initrd-domU.cpio.gz ./root/initrd.cpio.gz
cp ../Image ./root
@@ -56,6 +55,7 @@ xl -vvv create -c /root/test.cfg
" > etc/local.d/xen.start
chmod +x etc/local.d/xen.start
cp ../initrd.cpio.gz ../xen-rootfs.cpio.gz
+cat ../toolstack.cpio.gz >> ../xen-rootfs.cpio.gz
find . |cpio -H newc -o|gzip >> ../xen-rootfs.cpio.gz
cd ../..
diff --git a/automation/scripts/qemu-smoke-dom0less-arm64.sh b/automation/scripts/qemu-smoke-dom0less-arm64.sh
index 70085e26c139..93f4fb410690 100755
--- a/automation/scripts/qemu-smoke-dom0less-arm64.sh
+++ b/automation/scripts/qemu-smoke-dom0less-arm64.sh
@@ -121,7 +121,6 @@ mkdir proc
mkdir run
mkdir srv
mkdir sys
-cp -ar ../binaries/dist/install/* .
mkdir -p etc/local.d
echo "#!/bin/bash
@@ -142,6 +141,7 @@ ${dom0_check}
" > etc/local.d/xen.start
chmod +x etc/local.d/xen.start
cp ../binaries/initrd.cpio.gz ../binaries/dom0-rootfs.cpio.gz
+cat ../binaries/toolstack.cpio.gz >> ../binaries/dom0-rootfs.cpio.gz
find . | cpio -H newc -o | gzip >> ../binaries/dom0-rootfs.cpio.gz
cd ..
diff --git a/automation/scripts/qubes-x86-64.sh b/automation/scripts/qubes-x86-64.sh
index 399d72e0600d..4fe59fce8999 100755
--- a/automation/scripts/qubes-x86-64.sh
+++ b/automation/scripts/qubes-x86-64.sh
@@ -195,7 +195,6 @@ mkdir proc
mkdir run
mkdir sys
mkdir -p etc/local.d
-cp -ar ../binaries/dist/install/* .
cp -ar ../binaries/tests .
cp -a ../automation/scripts/run-tools-tests tests/
@@ -231,8 +230,10 @@ else
fi
chmod +x etc/local.d/xen.start
+mkdir -p etc/xen
echo "$domU_config" > etc/xen/domU.cfg
+mkdir -p etc/default
echo "XENCONSOLED_TRACE=all" >> etc/default/xencommons
echo "QEMU_XEN=/bin/false" >> etc/default/xencommons
mkdir -p var/log/xen/console
@@ -242,6 +243,7 @@ if [ -n "$domU_check" ]; then
fi
# take base initrd and append test-specific files
cp ../binaries/initrd.cpio.gz ../binaries/dom0-rootfs.cpio.gz
+cat ../binaries/toolstack.cpio.gz >> ../binaries/dom0-rootfs.cpio.gz
find . | cpio -H newc -o | gzip >> ../binaries/dom0-rootfs.cpio.gz
cd ..
diff --git a/automation/scripts/xilinx-smoke-dom0-x86_64.sh b/automation/scripts/xilinx-smoke-dom0-x86_64.sh
index 6b9f0e4a8dc7..ea0f952975c7 100755
--- a/automation/scripts/xilinx-smoke-dom0-x86_64.sh
+++ b/automation/scripts/xilinx-smoke-dom0-x86_64.sh
@@ -115,8 +115,7 @@ rm -rf rootfs
mkdir -p rootfs
cd rootfs
mkdir boot proc run srv sys
-cp -ar ../binaries/dist/install/* .
-mkdir -p etc/local.d
+mkdir -p etc/local.d etc/xen etc/default
echo "#!/bin/bash
set -x
export LD_LIBRARY_PATH=/usr/local/lib
@@ -134,6 +133,7 @@ cp ../binaries/bzImage boot/vmlinuz
cp ../binaries/domU-rootfs-overlay.cpio.gz boot/initrd-domU-overlay
copy_dom0_files
cp ../binaries/initrd.cpio.gz ../binaries/dom0-rootfs.cpio.gz
+cat ../binaries/toolstack.cpio.gz >> ../binaries/dom0-rootfs.cpio.gz
find . | cpio -H newc -o | gzip >> ../binaries/dom0-rootfs.cpio.gz
cd ..
diff --git a/automation/scripts/xilinx-smoke-dom0less-arm64.sh b/automation/scripts/xilinx-smoke-dom0less-arm64.sh
index 81be60e2026e..dbb955df0478 100755
--- a/automation/scripts/xilinx-smoke-dom0less-arm64.sh
+++ b/automation/scripts/xilinx-smoke-dom0less-arm64.sh
@@ -63,7 +63,6 @@ mkdir proc
mkdir run
mkdir srv
mkdir sys
-cp -ar ../binaries/dist/install/* .
mkdir -p etc/local.d
echo "#!/bin/bash
@@ -77,6 +76,7 @@ ${dom0_check}
" > etc/local.d/xen.start
chmod +x etc/local.d/xen.start
cp ../binaries/initrd.cpio.gz ../binaries/dom0-rootfs.cpio.gz
+cat ../binaries/toolstack.cpio.gz >> ../binaries/dom0-rootfs.cpio.gz
find . | cpio -H newc -o | gzip >> ../binaries/dom0-rootfs.cpio.gz
cd ..
--
git-series 0.9.1
^ permalink raw reply related [flat|nested] 26+ messages in thread* [PATCH test-artifacts v2 13/12] scripts: add rc_verbose=yes to /etc/rc.conf
2025-04-07 12:31 [PATCH v2 00/12] Several CI cleanups and improvements, plus yet another new runner Marek Marczykowski-Górecki
` (10 preceding siblings ...)
2025-04-07 12:31 ` [PATCH v2 12/12] CI: save toolstack artifact as cpio.gz Marek Marczykowski-Górecki
@ 2025-04-07 12:31 ` Marek Marczykowski-Górecki
2025-04-07 16:15 ` Andrew Cooper
2025-04-07 12:31 ` [PATCH test-artifacts v2 14/12] scripts: build initrd cpio Marek Marczykowski-Górecki
[not found] ` <eda734d64c0b73e5de6ca0d70fe3de02d40cf00d.1744028549.git-series.marmarek@invisiblethingslab.com>
13 siblings, 1 reply; 26+ messages in thread
From: Marek Marczykowski-Górecki @ 2025-04-07 12:31 UTC (permalink / raw)
To: xen-devel
Cc: Marek Marczykowski-Górecki, Stefano Stabellini,
Andrew Cooper, Anthony PERARD, Michal Orzel, Jan Beulich,
Julien Grall, Roger Pau Monné
This is done in every single test job, so do it at the initrd build
time.
Signed-off-by: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com>
---
scripts/x86_64-rootfs-alpine.sh | 1 +
1 file changed, 1 insertion(+)
diff --git a/scripts/x86_64-rootfs-alpine.sh b/scripts/x86_64-rootfs-alpine.sh
index f8b04c5..b70b3a5 100755
--- a/scripts/x86_64-rootfs-alpine.sh
+++ b/scripts/x86_64-rootfs-alpine.sh
@@ -50,6 +50,7 @@ echo "ttyS0" >> /etc/securetty
echo "hvc0" >> /etc/securetty
echo "ttyS0::respawn:/sbin/getty -L ttyS0 115200 vt100" >> /etc/inittab
echo "hvc0::respawn:/sbin/getty -L hvc0 115200 vt100" >> /etc/inittab
+echo "rc_verbose=yes" >> /etc/rc.conf
echo > /etc/modules
passwd -d "root" root
--
2.48.1
^ permalink raw reply related [flat|nested] 26+ messages in thread* [PATCH test-artifacts v2 14/12] scripts: build initrd cpio
2025-04-07 12:31 [PATCH v2 00/12] Several CI cleanups and improvements, plus yet another new runner Marek Marczykowski-Górecki
` (11 preceding siblings ...)
2025-04-07 12:31 ` [PATCH test-artifacts v2 13/12] scripts: add rc_verbose=yes to /etc/rc.conf Marek Marczykowski-Górecki
@ 2025-04-07 12:31 ` Marek Marczykowski-Górecki
2025-04-07 16:17 ` Andrew Cooper
2025-04-07 17:11 ` Andrew Cooper
[not found] ` <eda734d64c0b73e5de6ca0d70fe3de02d40cf00d.1744028549.git-series.marmarek@invisiblethingslab.com>
13 siblings, 2 replies; 26+ messages in thread
From: Marek Marczykowski-Górecki @ 2025-04-07 12:31 UTC (permalink / raw)
To: xen-devel
Cc: Marek Marczykowski-Górecki, Stefano Stabellini,
Andrew Cooper, Anthony PERARD, Michal Orzel, Jan Beulich,
Julien Grall, Roger Pau Monné
Build initrd format directly digestable by the kernel. Additionally,
include it itself inside as boot/initrd-domU file, for domU booting in
tests.
This should avoid the need to repack tar -> cpio as part of the test
job.
Keep generating initrd.tar.gz as that's still used by older branches.
Signed-off-by: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com>
---
scripts/x86_64-rootfs-alpine.sh | 5 +++++
1 file changed, 5 insertions(+)
diff --git a/scripts/x86_64-rootfs-alpine.sh b/scripts/x86_64-rootfs-alpine.sh
index b70b3a5..13e85fa 100755
--- a/scripts/x86_64-rootfs-alpine.sh
+++ b/scripts/x86_64-rootfs-alpine.sh
@@ -58,3 +58,8 @@ passwd -d "root" root
cd /
tar cvzf "${WORKDIR}/binaries/initrd.tar.gz" \
bin dev etc home init lib mnt opt root sbin usr var
+mkdir boot
+find bin dev etc home init lib mnt opt root sbin usr var |\
+ cpio -o -H newc | gzip > boot/initrd-domU
+find bin boot dev etc home init lib mnt opt root sbin usr var |\
+ cpio -o -H newc | gzip > "${WORKDIR}/binaries/initrd.cpio.gz" \
--
2.48.1
^ permalink raw reply related [flat|nested] 26+ messages in thread* Re: [PATCH test-artifacts v2 14/12] scripts: build initrd cpio
2025-04-07 12:31 ` [PATCH test-artifacts v2 14/12] scripts: build initrd cpio Marek Marczykowski-Górecki
@ 2025-04-07 16:17 ` Andrew Cooper
2025-04-07 16:22 ` Andrew Cooper
2025-04-07 17:11 ` Andrew Cooper
1 sibling, 1 reply; 26+ messages in thread
From: Andrew Cooper @ 2025-04-07 16:17 UTC (permalink / raw)
To: Marek Marczykowski-Górecki, xen-devel
Cc: Stefano Stabellini, Anthony PERARD, Michal Orzel, Jan Beulich,
Julien Grall, Roger Pau Monné
On 07/04/2025 1:31 pm, Marek Marczykowski-Górecki wrote:
> Build initrd format directly digestable by the kernel. Additionally,
> include it itself inside as boot/initrd-domU file, for domU booting in
> tests.
> This should avoid the need to repack tar -> cpio as part of the test
> job.
>
> Keep generating initrd.tar.gz as that's still used by older branches.
>
> Signed-off-by: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com>
Reviewed-by: Andrew Cooper <andrew.cooper3@citrix.com>
> ---
> scripts/x86_64-rootfs-alpine.sh | 5 +++++
> 1 file changed, 5 insertions(+)
>
> diff --git a/scripts/x86_64-rootfs-alpine.sh b/scripts/x86_64-rootfs-alpine.sh
> index b70b3a5..13e85fa 100755
> --- a/scripts/x86_64-rootfs-alpine.sh
> +++ b/scripts/x86_64-rootfs-alpine.sh
> @@ -58,3 +58,8 @@ passwd -d "root" root
> cd /
> tar cvzf "${WORKDIR}/binaries/initrd.tar.gz" \
> bin dev etc home init lib mnt opt root sbin usr var
What's required to drop the tar version? Presumably merging the rest of
your series?
> +mkdir boot
> +find bin dev etc home init lib mnt opt root sbin usr var |\
> + cpio -o -H newc | gzip > boot/initrd-domU
> +find bin boot dev etc home init lib mnt opt root sbin usr var |\
> + cpio -o -H newc | gzip > "${WORKDIR}/binaries/initrd.cpio.gz" \
^ permalink raw reply [flat|nested] 26+ messages in thread* Re: [PATCH test-artifacts v2 14/12] scripts: build initrd cpio
2025-04-07 16:17 ` Andrew Cooper
@ 2025-04-07 16:22 ` Andrew Cooper
2025-04-07 16:31 ` Marek Marczykowski-Górecki
0 siblings, 1 reply; 26+ messages in thread
From: Andrew Cooper @ 2025-04-07 16:22 UTC (permalink / raw)
To: Marek Marczykowski-Górecki, xen-devel
Cc: Stefano Stabellini, Anthony PERARD, Michal Orzel, Jan Beulich,
Julien Grall, Roger Pau Monné
On 07/04/2025 5:17 pm, Andrew Cooper wrote:
> On 07/04/2025 1:31 pm, Marek Marczykowski-Górecki wrote:
>> Build initrd format directly digestable by the kernel. Additionally,
>> include it itself inside as boot/initrd-domU file, for domU booting in
>> tests.
>> This should avoid the need to repack tar -> cpio as part of the test
>> job.
>>
>> Keep generating initrd.tar.gz as that's still used by older branches.
>>
>> Signed-off-by: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com>
> Reviewed-by: Andrew Cooper <andrew.cooper3@citrix.com>
>
>> ---
>> scripts/x86_64-rootfs-alpine.sh | 5 +++++
>> 1 file changed, 5 insertions(+)
>>
>> diff --git a/scripts/x86_64-rootfs-alpine.sh b/scripts/x86_64-rootfs-alpine.sh
>> index b70b3a5..13e85fa 100755
>> --- a/scripts/x86_64-rootfs-alpine.sh
>> +++ b/scripts/x86_64-rootfs-alpine.sh
>> @@ -58,3 +58,8 @@ passwd -d "root" root
>> cd /
>> tar cvzf "${WORKDIR}/binaries/initrd.tar.gz" \
>> bin dev etc home init lib mnt opt root sbin usr var
> What's required to drop the tar version? Presumably merging the rest of
> your series?
NVM, I can't read. I'm tempted to fold in:
#TODO, drop .tar.gz when Xen 4.20 drops out of testing
That said, I expect we'll be backporting these changes.
~Andrew
^ permalink raw reply [flat|nested] 26+ messages in thread* Re: [PATCH test-artifacts v2 14/12] scripts: build initrd cpio
2025-04-07 16:22 ` Andrew Cooper
@ 2025-04-07 16:31 ` Marek Marczykowski-Górecki
2025-04-08 12:06 ` Anthony PERARD
0 siblings, 1 reply; 26+ messages in thread
From: Marek Marczykowski-Górecki @ 2025-04-07 16:31 UTC (permalink / raw)
To: Andrew Cooper
Cc: xen-devel, Stefano Stabellini, Anthony PERARD, Michal Orzel,
Jan Beulich, Julien Grall, Roger Pau Monné
[-- Attachment #1: Type: text/plain, Size: 2088 bytes --]
On Mon, Apr 07, 2025 at 05:22:05PM +0100, Andrew Cooper wrote:
> On 07/04/2025 5:17 pm, Andrew Cooper wrote:
> > On 07/04/2025 1:31 pm, Marek Marczykowski-Górecki wrote:
> >> Build initrd format directly digestable by the kernel. Additionally,
> >> include it itself inside as boot/initrd-domU file, for domU booting in
> >> tests.
> >> This should avoid the need to repack tar -> cpio as part of the test
> >> job.
> >>
> >> Keep generating initrd.tar.gz as that's still used by older branches.
> >>
> >> Signed-off-by: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com>
> > Reviewed-by: Andrew Cooper <andrew.cooper3@citrix.com>
> >
> >> ---
> >> scripts/x86_64-rootfs-alpine.sh | 5 +++++
> >> 1 file changed, 5 insertions(+)
> >>
> >> diff --git a/scripts/x86_64-rootfs-alpine.sh b/scripts/x86_64-rootfs-alpine.sh
> >> index b70b3a5..13e85fa 100755
> >> --- a/scripts/x86_64-rootfs-alpine.sh
> >> +++ b/scripts/x86_64-rootfs-alpine.sh
> >> @@ -58,3 +58,8 @@ passwd -d "root" root
> >> cd /
> >> tar cvzf "${WORKDIR}/binaries/initrd.tar.gz" \
> >> bin dev etc home init lib mnt opt root sbin usr var
> > What's required to drop the tar version? Presumably merging the rest of
> > your series?
>
> NVM, I can't read. I'm tempted to fold in:
>
> #TODO, drop .tar.gz when Xen 4.20 drops out of testing
>
> That said, I expect we'll be backporting these changes.
There is one more difference: the cpio.gz contains the whole thing
twice. Once as rootfs for dom0 and then another as boot/initrd-domU.
Dropping .tar.gz is probably a good idea at some point, so I'm okay with
such comment added. But I imagine some future tests may benefit from
just one layer, which may want introducing another cpio.gz without
boot/initrd-domU included. In fact, even right now that might be useful
- for example dom0less arm64 test uses just busybox for domU rootfs, not
the whole archive (so with my changes dom0 rootfs has unused
boot/initrd-domU included).
--
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]
^ permalink raw reply [flat|nested] 26+ messages in thread* Re: [PATCH test-artifacts v2 14/12] scripts: build initrd cpio
2025-04-07 16:31 ` Marek Marczykowski-Górecki
@ 2025-04-08 12:06 ` Anthony PERARD
2025-04-08 12:28 ` Marek Marczykowski-Górecki
0 siblings, 1 reply; 26+ messages in thread
From: Anthony PERARD @ 2025-04-08 12:06 UTC (permalink / raw)
To: Marek Marczykowski-Górecki
Cc: Andrew Cooper, xen-devel, Stefano Stabellini, Michal Orzel,
Jan Beulich, Julien Grall, Roger Pau Monné
On Mon, Apr 07, 2025 at 06:31:06PM +0200, Marek Marczykowski-Górecki wrote:
> There is one more difference: the cpio.gz contains the whole thing
> twice. Once as rootfs for dom0 and then another as boot/initrd-domU.
> Dropping .tar.gz is probably a good idea at some point, so I'm okay with
> such comment added. But I imagine some future tests may benefit from
> just one layer, which may want introducing another cpio.gz without
> boot/initrd-domU included. In fact, even right now that might be useful
> - for example dom0less arm64 test uses just busybox for domU rootfs, not
> the whole archive (so with my changes dom0 rootfs has unused
> boot/initrd-domU included).
In such case, would it make sense to prepare several initrd.cpio.gz? A
common one, then having overlays of files we want to add or replace to
the initrd of spefic test. I think osstest used to do something like
that, with command that would be:
cat common.cpio.gz overlay.cpio.gz > initrd.cpio.gz
If it matter, is seem that the "file.cpio" used to be made with
`cpio -Hnewc -o`
Cheers,
--
Anthony Perard | Vates XCP-ng Developer
XCP-ng & Xen Orchestra - Vates solutions
web: https://vates.tech
^ permalink raw reply [flat|nested] 26+ messages in thread* Re: [PATCH test-artifacts v2 14/12] scripts: build initrd cpio
2025-04-08 12:06 ` Anthony PERARD
@ 2025-04-08 12:28 ` Marek Marczykowski-Górecki
0 siblings, 0 replies; 26+ messages in thread
From: Marek Marczykowski-Górecki @ 2025-04-08 12:28 UTC (permalink / raw)
To: Anthony PERARD
Cc: Andrew Cooper, xen-devel, Stefano Stabellini, Michal Orzel,
Jan Beulich, Julien Grall, Roger Pau Monné
[-- Attachment #1: Type: text/plain, Size: 1859 bytes --]
On Tue, Apr 08, 2025 at 12:06:51PM +0000, Anthony PERARD wrote:
> On Mon, Apr 07, 2025 at 06:31:06PM +0200, Marek Marczykowski-Górecki wrote:
> > There is one more difference: the cpio.gz contains the whole thing
> > twice. Once as rootfs for dom0 and then another as boot/initrd-domU.
> > Dropping .tar.gz is probably a good idea at some point, so I'm okay with
> > such comment added. But I imagine some future tests may benefit from
> > just one layer, which may want introducing another cpio.gz without
> > boot/initrd-domU included. In fact, even right now that might be useful
> > - for example dom0less arm64 test uses just busybox for domU rootfs, not
> > the whole archive (so with my changes dom0 rootfs has unused
> > boot/initrd-domU included).
>
> In such case, would it make sense to prepare several initrd.cpio.gz? A
> common one, then having overlays of files we want to add or replace to
> the initrd of spefic test. I think osstest used to do something like
> that, with command that would be:
> cat common.cpio.gz overlay.cpio.gz > initrd.cpio.gz
>
> If it matter, is seem that the "file.cpio" used to be made with
> `cpio -Hnewc -o`
This is already how final initrd is constructed with my changes.
Having said that, maybe test-artifacts should not include
boot/initrd-domU in initrd.cpio.gz initially, but have a separate cpio
that has just boot/initrd-domU there. Most tests do use this domU
initrd, but since they need to concatenate something anyway, maybe
starting with two files instead of one doesn't make much difference?
So, it would be:
- initrd.cpio.gz - plain rootfs, for dom0 (or domU in dom0less tests)
- initrd-in-boot.cpio.gz - the above initrd.cpio.gz packed again as
boot/initrd-domU
What do you think?
--
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [PATCH test-artifacts v2 14/12] scripts: build initrd cpio
2025-04-07 12:31 ` [PATCH test-artifacts v2 14/12] scripts: build initrd cpio Marek Marczykowski-Górecki
2025-04-07 16:17 ` Andrew Cooper
@ 2025-04-07 17:11 ` Andrew Cooper
2025-04-07 17:14 ` Andrew Cooper
2025-04-07 17:15 ` Marek Marczykowski-Górecki
1 sibling, 2 replies; 26+ messages in thread
From: Andrew Cooper @ 2025-04-07 17:11 UTC (permalink / raw)
To: Marek Marczykowski-Górecki, xen-devel
Cc: Stefano Stabellini, Anthony PERARD, Michal Orzel, Jan Beulich,
Julien Grall, Roger Pau Monné
On 07/04/2025 1:31 pm, Marek Marczykowski-Górecki wrote:
> Build initrd format directly digestable by the kernel. Additionally,
> include it itself inside as boot/initrd-domU file, for domU booting in
> tests.
> This should avoid the need to repack tar -> cpio as part of the test
> job.
>
> Keep generating initrd.tar.gz as that's still used by older branches.
>
> Signed-off-by: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com>
> ---
> scripts/x86_64-rootfs-alpine.sh | 5 +++++
> 1 file changed, 5 insertions(+)
>
> diff --git a/scripts/x86_64-rootfs-alpine.sh b/scripts/x86_64-rootfs-alpine.sh
> index b70b3a5..13e85fa 100755
> --- a/scripts/x86_64-rootfs-alpine.sh
> +++ b/scripts/x86_64-rootfs-alpine.sh
> @@ -58,3 +58,8 @@ passwd -d "root" root
> cd /
> tar cvzf "${WORKDIR}/binaries/initrd.tar.gz" \
> bin dev etc home init lib mnt opt root sbin usr var
> +mkdir boot
> +find bin dev etc home init lib mnt opt root sbin usr var |\
> + cpio -o -H newc | gzip > boot/initrd-domU
> +find bin boot dev etc home init lib mnt opt root sbin usr var |\
> + cpio -o -H newc | gzip > "${WORKDIR}/binaries/initrd.cpio.gz" \
So, after looking at this a bit more, a few notes.
Trailing \ needs dropping.
initrd-domU probably ought to be named initrd-domU-base.cpio.gz so it's
clear what it is. These are easy to fix up.
Looking through the current initrd.tar.gz, we've got:
$ ls -lah ./var/cache/apk/
total 1.9M
drwxr-xr-x 2 andrew andrew 4.0K Nov 8 15:12 .
drwxr-xr-x 4 andrew andrew 4.0K Sep 6 2024 ..
-rw-r--r-- 1 andrew andrew 425K Nov 8 15:12 APKINDEX.73fc500b.tar.gz
-rw-r--r-- 1 andrew andrew 1.4M Nov 8 15:12 APKINDEX.eb46142e.tar.gz
which (alone) is 5% of the size of the initrd. I'll submit a patch
separately to drop this.
But doesn't this result in initrd.tar.gz being included in initrd-domU,
and then (doubly) in initrd.cpio.gz ?
~Andrew
^ permalink raw reply [flat|nested] 26+ messages in thread* Re: [PATCH test-artifacts v2 14/12] scripts: build initrd cpio
2025-04-07 17:11 ` Andrew Cooper
@ 2025-04-07 17:14 ` Andrew Cooper
2025-04-07 17:15 ` Marek Marczykowski-Górecki
1 sibling, 0 replies; 26+ messages in thread
From: Andrew Cooper @ 2025-04-07 17:14 UTC (permalink / raw)
To: Marek Marczykowski-Górecki, xen-devel
Cc: Stefano Stabellini, Anthony PERARD, Michal Orzel, Jan Beulich,
Julien Grall, Roger Pau Monné
On 07/04/2025 6:11 pm, Andrew Cooper wrote:
> On 07/04/2025 1:31 pm, Marek Marczykowski-Górecki wrote:
>> Build initrd format directly digestable by the kernel. Additionally,
>> include it itself inside as boot/initrd-domU file, for domU booting in
>> tests.
>> This should avoid the need to repack tar -> cpio as part of the test
>> job.
>>
>> Keep generating initrd.tar.gz as that's still used by older branches.
>>
>> Signed-off-by: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com>
>> ---
>> scripts/x86_64-rootfs-alpine.sh | 5 +++++
>> 1 file changed, 5 insertions(+)
>>
>> diff --git a/scripts/x86_64-rootfs-alpine.sh b/scripts/x86_64-rootfs-alpine.sh
>> index b70b3a5..13e85fa 100755
>> --- a/scripts/x86_64-rootfs-alpine.sh
>> +++ b/scripts/x86_64-rootfs-alpine.sh
>> @@ -58,3 +58,8 @@ passwd -d "root" root
>> cd /
>> tar cvzf "${WORKDIR}/binaries/initrd.tar.gz" \
>> bin dev etc home init lib mnt opt root sbin usr var
>> +mkdir boot
>> +find bin dev etc home init lib mnt opt root sbin usr var |\
>> + cpio -o -H newc | gzip > boot/initrd-domU
>> +find bin boot dev etc home init lib mnt opt root sbin usr var |\
>> + cpio -o -H newc | gzip > "${WORKDIR}/binaries/initrd.cpio.gz" \
> So, after looking at this a bit more, a few notes.
>
> Trailing \ needs dropping.
>
> initrd-domU probably ought to be named initrd-domU-base.cpio.gz so it's
> clear what it is. These are easy to fix up.
>
> Looking through the current initrd.tar.gz, we've got:
>
> $ ls -lah ./var/cache/apk/
> total 1.9M
> drwxr-xr-x 2 andrew andrew 4.0K Nov 8 15:12 .
> drwxr-xr-x 4 andrew andrew 4.0K Sep 6 2024 ..
> -rw-r--r-- 1 andrew andrew 425K Nov 8 15:12 APKINDEX.73fc500b.tar.gz
> -rw-r--r-- 1 andrew andrew 1.4M Nov 8 15:12 APKINDEX.eb46142e.tar.gz
>
> which (alone) is 5% of the size of the initrd. I'll submit a patch
> separately to drop this.
>
> But doesn't this result in initrd.tar.gz being included in initrd-domU,
> and then (doubly) in initrd.cpio.gz ?
No, because $WORKDIR is nothing, and we put it in /binaries at the top
level which is excluded.
~Andrew
^ permalink raw reply [flat|nested] 26+ messages in thread* Re: [PATCH test-artifacts v2 14/12] scripts: build initrd cpio
2025-04-07 17:11 ` Andrew Cooper
2025-04-07 17:14 ` Andrew Cooper
@ 2025-04-07 17:15 ` Marek Marczykowski-Górecki
1 sibling, 0 replies; 26+ messages in thread
From: Marek Marczykowski-Górecki @ 2025-04-07 17:15 UTC (permalink / raw)
To: Andrew Cooper
Cc: xen-devel, Stefano Stabellini, Anthony PERARD, Michal Orzel,
Jan Beulich, Julien Grall, Roger Pau Monné
[-- Attachment #1: Type: text/plain, Size: 2387 bytes --]
On Mon, Apr 07, 2025 at 06:11:29PM +0100, Andrew Cooper wrote:
> On 07/04/2025 1:31 pm, Marek Marczykowski-Górecki wrote:
> > Build initrd format directly digestable by the kernel. Additionally,
> > include it itself inside as boot/initrd-domU file, for domU booting in
> > tests.
> > This should avoid the need to repack tar -> cpio as part of the test
> > job.
> >
> > Keep generating initrd.tar.gz as that's still used by older branches.
> >
> > Signed-off-by: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com>
> > ---
> > scripts/x86_64-rootfs-alpine.sh | 5 +++++
> > 1 file changed, 5 insertions(+)
> >
> > diff --git a/scripts/x86_64-rootfs-alpine.sh b/scripts/x86_64-rootfs-alpine.sh
> > index b70b3a5..13e85fa 100755
> > --- a/scripts/x86_64-rootfs-alpine.sh
> > +++ b/scripts/x86_64-rootfs-alpine.sh
> > @@ -58,3 +58,8 @@ passwd -d "root" root
> > cd /
> > tar cvzf "${WORKDIR}/binaries/initrd.tar.gz" \
> > bin dev etc home init lib mnt opt root sbin usr var
> > +mkdir boot
> > +find bin dev etc home init lib mnt opt root sbin usr var |\
> > + cpio -o -H newc | gzip > boot/initrd-domU
> > +find bin boot dev etc home init lib mnt opt root sbin usr var |\
> > + cpio -o -H newc | gzip > "${WORKDIR}/binaries/initrd.cpio.gz" \
>
> So, after looking at this a bit more, a few notes.
>
> Trailing \ needs dropping.
>
> initrd-domU probably ought to be named initrd-domU-base.cpio.gz so it's
> clear what it is. These are easy to fix up.
Technically, it's useful as is already if you just want to check if domU
starts.
> Looking through the current initrd.tar.gz, we've got:
>
> $ ls -lah ./var/cache/apk/
> total 1.9M
> drwxr-xr-x 2 andrew andrew 4.0K Nov 8 15:12 .
> drwxr-xr-x 4 andrew andrew 4.0K Sep 6 2024 ..
> -rw-r--r-- 1 andrew andrew 425K Nov 8 15:12 APKINDEX.73fc500b.tar.gz
> -rw-r--r-- 1 andrew andrew 1.4M Nov 8 15:12 APKINDEX.eb46142e.tar.gz
>
> which (alone) is 5% of the size of the initrd. I'll submit a patch
> separately to drop this.
Makes sense.
> But doesn't this result in initrd.tar.gz being included in initrd-domU,
> and then (doubly) in initrd.cpio.gz ?
No, find (same as tar earlier) has explicit list of dirs to look at,
which don't include initrd.* nor "binaries".
--
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]
^ permalink raw reply [flat|nested] 26+ messages in thread
[parent not found: <eda734d64c0b73e5de6ca0d70fe3de02d40cf00d.1744028549.git-series.marmarek@invisiblethingslab.com>]
* Re: [PATCH v2 08/12] CI: switch test kernel from 6.1.19 to 6.12.21
[not found] ` <eda734d64c0b73e5de6ca0d70fe3de02d40cf00d.1744028549.git-series.marmarek@invisiblethingslab.com>
@ 2025-04-07 17:56 ` Andrew Cooper
0 siblings, 0 replies; 26+ messages in thread
From: Andrew Cooper @ 2025-04-07 17:56 UTC (permalink / raw)
To: Marek Marczykowski-Górecki, xen-devel
Cc: Doug Goldstein, Stefano Stabellini
On 07/04/2025 1:31 pm, Marek Marczykowski-Górecki wrote:
> The jump is pretty big, but should be enough for a longer time. The
> change right now is motivated adding new runner with AMD Zen4.
>
> Signed-off-by: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com>
> ---
> automation/gitlab-ci/build.yaml | 4 +-
> automation/gitlab-ci/test.yaml | 2 +-
> automation/tests-artifacts/kernel/6.1.19.dockerfile | 41 +-------------
> automation/tests-artifacts/kernel/6.12.21.dockerfile | 41 +++++++++++++-
> 4 files changed, 44 insertions(+), 44 deletions(-)
> delete mode 100644 automation/tests-artifacts/kernel/6.1.19.dockerfile
> create mode 100644 automation/tests-artifacts/kernel/6.12.21.dockerfile
I'd really prefer not to continue extending the *-export mess, but that
in turn needs fixes in the test-artefacts repo first. It should
simplify later parts of this series.
~Andrew
^ permalink raw reply [flat|nested] 26+ messages in thread