[PATCH v1 00/11] Several CI cleanups and improvements, plus yet another new runner

[PATCH v1 00/11] Several CI cleanups and improvements, plus yet another new runner

[Marek Marczykowski-Górecki]

Some of those patches are collected from my stubdomain test series, others are
made while getting the new runner working. All of the cleanups can be applied
independently, but most if not all are needed for the final patch adding new
runner.

As usual, besides the patches, somebody need to click on "hal9012" runner in
the relevant projects.

Marek Marczykowski-Górecki (11):
  ci: prevent grub unpacking initramfs
  ci: increase timeout for hw tests
  ci: enable XHCI console in Xen debug build on Alpine
  ci: include domU kernel messages in the console output log
  ci: increase verbosity of starting a domain
  ci: consistently use DOCKER_CMD in makefiles
  ci: wait for the network interface in PCI passthrough tests
  ci: switch test kernel from 6.1.19 to 6.12.21
  ci: adjust resolving network interface into PCI device
  ci: add AMD Zen 4 HW runner
  [DO NOT MERGE] container

 automation/build/Makefile                            |  4 +-
 automation/gitlab-ci/build.yaml                      |  5 +-
 automation/gitlab-ci/test.yaml                       | 58 ++++++++++++-
 automation/scripts/qemu-alpine-x86_64.sh             |  2 +-
 automation/scripts/qemu-smoke-dom0-arm32.sh          |  2 +-
 automation/scripts/qemu-smoke-dom0-arm64.sh          |  2 +-
 automation/scripts/qubes-x86-64.sh                   | 14 +--
 automation/tests-artifacts/Makefile                  |  4 +-
 automation/tests-artifacts/kernel/6.1.19.dockerfile  | 41 +---------
 automation/tests-artifacts/kernel/6.12.21.dockerfile | 43 ++++++++++-
 10 files changed, 119 insertions(+), 56 deletions(-)
 delete mode 100644 automation/tests-artifacts/kernel/6.1.19.dockerfile
 create mode 100644 automation/tests-artifacts/kernel/6.12.21.dockerfile

base-commit: 35910ed65f1d188774ce90ba9898ca33f7a9fe4d
-- 
git-series 0.9.1

[PATCH v1 01/11] ci: prevent grub unpacking initramfs

[Marek Marczykowski-Górecki]

It fails on larger initramfs (~250MB one) and sometimes even smaller
depending on memory size/memory map, let Linux do it.

Signed-off-by: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com>
---
 automation/scripts/qubes-x86-64.sh | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/automation/scripts/qubes-x86-64.sh b/automation/scripts/qubes-x86-64.sh
index f27296445901..8e78b7984e98 100755
--- a/automation/scripts/qubes-x86-64.sh
+++ b/automation/scripts/qubes-x86-64.sh
@@ -248,7 +248,7 @@ CONTROLLER=control@thor.testnet
 echo "
 multiboot2 (http)/gitlab-ci/xen $CONSOLE_OPTS loglvl=all guest_loglvl=all dom0_mem=4G console_timestamps=boot $extra_xen_opts
 module2 (http)/gitlab-ci/vmlinuz console=hvc0 root=/dev/ram0 earlyprintk=xen
-module2 (http)/gitlab-ci/initrd-dom0
+module2 --nounzip (http)/gitlab-ci/initrd-dom0
 " > $TFTP/grub.cfg
 
 echo "#!ipxe
-- 
git-series 0.9.1

[PATCH v1 02/11] ci: increase timeout for hw tests

[Marek Marczykowski-Górecki]

It appears as sometimes it takes more time for Xen even start booting,
mostly due to firmware and fetching large boot files by grub. In some
jobs the current timeout is pretty close to the actual time needed, and
sometimes (rarely for now) test fails due to timeout expiring in the
middle of dom0 booting. This will be happening more often if the
initramfs will grow (and with more complex tests).
This has been observed on some dom0pvh-hvm jobs, at least on runners hw3
and hw11.

Increase the timeout by yet another 60s (up to 180s now).

Signed-off-by: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com>
---
 automation/scripts/qubes-x86-64.sh | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/automation/scripts/qubes-x86-64.sh b/automation/scripts/qubes-x86-64.sh
index 8e78b7984e98..771c77d6618b 100755
--- a/automation/scripts/qubes-x86-64.sh
+++ b/automation/scripts/qubes-x86-64.sh
@@ -17,7 +17,7 @@ test_variant=$1
 ### defaults
 extra_xen_opts=
 wait_and_wakeup=
-timeout=120
+timeout=180
 domU_type="pvh"
 domU_vif="'bridge=xenbr0',"
 domU_extra_config=
-- 
git-series 0.9.1

[PATCH v1 03/11] ci: enable XHCI console in Xen debug build on Alpine

[Marek Marczykowski-Górecki]

This build is used for hardware tests, and some runners use XHCI console

Signed-off-by: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com>
---
 automation/gitlab-ci/build.yaml | 1 +
 1 file changed, 1 insertion(+)

diff --git a/automation/gitlab-ci/build.yaml b/automation/gitlab-ci/build.yaml
index 2513908b059b..169bebe3c775 100644
--- a/automation/gitlab-ci/build.yaml
+++ b/automation/gitlab-ci/build.yaml
@@ -341,6 +341,7 @@ alpine-3.18-gcc-debug:
       CONFIG_ARGO=y
       CONFIG_UBSAN=y
       CONFIG_UBSAN_FATAL=y
+      CONFIG_XHCI=y
 
 debian-12-x86_64-gcc-debug:
   extends: .gcc-x86-64-build-debug
-- 
git-series 0.9.1

[PATCH v1 04/11] ci: include domU kernel messages in the console output log

[Marek Marczykowski-Górecki]

Signed-off-by: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com>
---
 automation/scripts/qubes-x86-64.sh | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/automation/scripts/qubes-x86-64.sh b/automation/scripts/qubes-x86-64.sh
index 771c77d6618b..d316c17e0c9f 100755
--- a/automation/scripts/qubes-x86-64.sh
+++ b/automation/scripts/qubes-x86-64.sh
@@ -175,6 +175,8 @@ if [ -n "$domU_check" ]; then
     rm var/run
     echo "#!/bin/sh
 
+echo 8 > /proc/sys/kernel/printk
+
 ${domU_check}
 " > etc/local.d/xen.start
     chmod +x etc/local.d/xen.start
-- 
git-series 0.9.1

[PATCH v1 05/11] ci: increase verbosity of starting a domain

[Marek Marczykowski-Górecki]

And start collecting qemu log earlier, so it isn't lost in case of a
timeout during domain startup.

Signed-off-by: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com>
---
 automation/scripts/qemu-alpine-x86_64.sh    | 2 +-
 automation/scripts/qemu-smoke-dom0-arm32.sh | 2 +-
 automation/scripts/qemu-smoke-dom0-arm64.sh | 2 +-
 automation/scripts/qubes-x86-64.sh          | 4 ++--
 4 files changed, 5 insertions(+), 5 deletions(-)

diff --git a/automation/scripts/qemu-alpine-x86_64.sh b/automation/scripts/qemu-alpine-x86_64.sh
index 1ff689b577e3..17e2141d625e 100755
--- a/automation/scripts/qemu-alpine-x86_64.sh
+++ b/automation/scripts/qemu-alpine-x86_64.sh
@@ -56,7 +56,7 @@ bash /etc/init.d/xencommons start
 
 xl list
 
-xl create -c /root/test.cfg
+xl -vvv create -c /root/test.cfg
 
 " > etc/local.d/xen.start
 chmod +x etc/local.d/xen.start
diff --git a/automation/scripts/qemu-smoke-dom0-arm32.sh b/automation/scripts/qemu-smoke-dom0-arm32.sh
index e1cd83880928..0c60a66e25e3 100755
--- a/automation/scripts/qemu-smoke-dom0-arm32.sh
+++ b/automation/scripts/qemu-smoke-dom0-arm32.sh
@@ -21,7 +21,7 @@ echo "#!/bin/bash
 
 xl list
 
-xl create -c /root/test.cfg
+xl -vvv create -c /root/test.cfg
 
 " > ./root/xen.start
 echo "bash /root/xen.start" >> ./etc/init.d/xen-watchdog
diff --git a/automation/scripts/qemu-smoke-dom0-arm64.sh b/automation/scripts/qemu-smoke-dom0-arm64.sh
index 4d22a124df11..8774a8701232 100755
--- a/automation/scripts/qemu-smoke-dom0-arm64.sh
+++ b/automation/scripts/qemu-smoke-dom0-arm64.sh
@@ -52,7 +52,7 @@ bash /etc/init.d/xencommons start
 
 xl list
 
-xl create -c /root/test.cfg
+xl -vvv create -c /root/test.cfg
 
 " > etc/local.d/xen.start
 chmod +x etc/local.d/xen.start
diff --git a/automation/scripts/qubes-x86-64.sh b/automation/scripts/qubes-x86-64.sh
index d316c17e0c9f..822b81d186a7 100755
--- a/automation/scripts/qubes-x86-64.sh
+++ b/automation/scripts/qubes-x86-64.sh
@@ -122,7 +122,6 @@ echo \"${passed}\"
 "
 
         dom0_check="
-tail -F /var/log/xen/qemu-dm-domU.log &
 until grep -q \"^domU Welcome to Alpine Linux\" /var/log/xen/console/guest-domU.log; do
     sleep 1
 done
@@ -222,7 +221,8 @@ if [ -n "$domU_check" ]; then
     echo "
 # get domU console content into test log
 tail -F /var/log/xen/console/guest-domU.log 2>/dev/null | sed -e \"s/^/(domU) /\" &
-xl create /etc/xen/domU.cfg
+tail -F /var/log/xen/qemu-dm-domU.log 2>/dev/null | sed -e \"s/^/(qemu-dm) /\" &
+xl -vvv create /etc/xen/domU.cfg
 ${dom0_check}
 " >> etc/local.d/xen.start
 else
-- 
git-series 0.9.1

[PATCH v1 06/11] ci: consistently use DOCKER_CMD in makefiles

[Marek Marczykowski-Górecki]

This allows rebuilding containers using podman too.

Signed-off-by: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com>
---
 automation/build/Makefile           | 4 ++--
 automation/tests-artifacts/Makefile | 4 ++--
 2 files changed, 4 insertions(+), 4 deletions(-)

diff --git a/automation/build/Makefile b/automation/build/Makefile
index 4df43b040777..fedf7524dacd 100644
--- a/automation/build/Makefile
+++ b/automation/build/Makefile
@@ -31,8 +31,8 @@ clean:
 define CLEAN_RULE
 .PHONY: clean-$(1)
 clean-$(1):
-	if [ -n "$$$$(docker image ls -q $(REGISTRY)/$(subst /,:,$(1)))" ]; then \
-		docker image rm $(REGISTRY)/$(subst /,:,$(1)); \
+	if [ -n "$$$$($(DOCKER_CMD) image ls -q $(REGISTRY)/$(subst /,:,$(1)))" ]; then \
+		$(DOCKER_CMD) image rm $(REGISTRY)/$(subst /,:,$(1)); \
 	fi
 
 endef
diff --git a/automation/tests-artifacts/Makefile b/automation/tests-artifacts/Makefile
index d055cd696bed..80a60a94f3f7 100644
--- a/automation/tests-artifacts/Makefile
+++ b/automation/tests-artifacts/Makefile
@@ -10,9 +10,9 @@ help:
 	@echo "To push container builds, set the env var PUSH"
 
 %: %.dockerfile ## Builds containers
-	docker build --pull -t $(REGISTRY)/$(@D):$(@F) -f $< $(<D)
+	$(DOCKER_CMD) build --pull -t $(REGISTRY)/$(@D):$(@F) -f $< $(<D)
 	@if [ ! -z $${PUSH+x} ]; then \
-		docker push $(REGISTRY)/$(@D):$(@F); \
+		$(DOCKER_CMD) push $(REGISTRY)/$(@D):$(@F); \
 	fi
 
 .PHONY: all
-- 
git-series 0.9.1

[PATCH v1 07/11] ci: wait for the network interface in PCI passthrough tests

[Marek Marczykowski-Górecki]

The network driver initializes asynchronously, and it may not be ready
yet by the time the startup script is called. This is especially the
case for USB network adapter (where the PCI device is the USB
controller) in the upcoming runner.

Don't bother about separate timeout - test timeout will cover this part
too.

Signed-off-by: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com>
---
 automation/scripts/qubes-x86-64.sh | 1 +
 1 file changed, 1 insertion(+)

diff --git a/automation/scripts/qubes-x86-64.sh b/automation/scripts/qubes-x86-64.sh
index 822b81d186a7..6d2c170d1ee1 100755
--- a/automation/scripts/qubes-x86-64.sh
+++ b/automation/scripts/qubes-x86-64.sh
@@ -104,6 +104,7 @@ on_reboot = "destroy"
         domU_check="
 set -x -e
 interface=eth0
+while ! [ -e \"/sys/class/net/\$interface\" ]; do sleep 1; done
 ip link set \"\$interface\" up
 timeout 30s udhcpc -i \"\$interface\"
 pingip=\$(ip -o -4 r show default|cut -f 3 -d ' ')
-- 
git-series 0.9.1

[PATCH v1 09/11] ci: adjust resolving network interface into PCI device

[Marek Marczykowski-Górecki]

Change how PCI device lookup is done to handle also USB devices, in
which case get the USB controller. Instead of taking basename of the
'device' symlink, resolve the full path (example:
/sys/devices/pci0000:00/0000:00:09.0/usb4/4-7/4-7:1.0) and take the
first part after pci0000:00. Theoretically it could be a bridge, but VM
has flat PCI topology.

Signed-off-by: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com>
---
 automation/scripts/qubes-x86-64.sh | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/automation/scripts/qubes-x86-64.sh b/automation/scripts/qubes-x86-64.sh
index 6d2c170d1ee1..dc8013750f5f 100755
--- a/automation/scripts/qubes-x86-64.sh
+++ b/automation/scripts/qubes-x86-64.sh
@@ -110,7 +110,8 @@ timeout 30s udhcpc -i \"\$interface\"
 pingip=\$(ip -o -4 r show default|cut -f 3 -d ' ')
 ping -c 10 \"\$pingip\"
 echo domU started
-pcidevice=\$(basename \$(readlink /sys/class/net/\$interface/device))
+pcidevice=\$(realpath /sys/class/net/\$interface/device |
+             sed 's#.*pci0000:00/\\([^/]*\\).*#\\1#')
 lspci -vs \$pcidevice
 "
         if [ -n "$PCIDEV_INTR" ]; then
-- 
git-series 0.9.1

[PATCH v1 10/11] ci: add AMD Zen 4 HW runner

[Marek Marczykowski-Górecki]

This is AMD Ryzen 5 7640U in Framework Laptop AMD.
It has several things different than the other runners.
First of all, the console is using XHCI debug capability.
And then, this system doesn't have normal wired ethernet. But is has one
on USB, and this one is used for booting. For this, enable
CONFIG_USB_RTL8152 in the Linux build.

Include some basic tests, plus PCI passthrough.

This machine doesn't support S3. S0ix pretends to be working with
Qubes's dom0 (kernel + userspace), but it hangs on suspend with Alpine
used in test. But even when it doesn't hang, it doesn't really reach
deep sleep, so skip this test for now here.

Signed-off-by: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com>
---
Requires containers rebuild, especially tests-artifacts/kernel/6.12.21.

Do not apply yet, until issues found by those tests are fixed. That
would be at least third issue found using this system, so lets have it
in CI.
---
 automation/gitlab-ci/test.yaml                       | 56 +++++++++++++-
 automation/tests-artifacts/kernel/6.12.21.dockerfile |  2 +-
 2 files changed, 58 insertions(+)

diff --git a/automation/gitlab-ci/test.yaml b/automation/gitlab-ci/test.yaml
index a13dd040bd26..feb2dd10ab30 100644
--- a/automation/gitlab-ci/test.yaml
+++ b/automation/gitlab-ci/test.yaml
@@ -188,6 +188,16 @@
   tags:
     - qubes-hw11
 
+.zen4-x86-64:
+  extends: .adl-x86-64
+  variables:
+    PCIDEV: "c3:00.4"
+    PCIDEV_INTR: "MSI-X"
+    CONSOLE_OPTS: "console=xhci dbgp=xhci@pcic1:00.3,share=yes"
+    SUT_ADDR: test-12.testnet
+  tags:
+    - qubes-hw12
+
 # Test jobs
 build-each-commit-gcc:
   extends: .test-jobs-common
@@ -479,6 +489,52 @@ zen3p-tools-tests-pvh-x86-64-gcc-debug:
     - *x86-64-test-needs
     - alpine-3.18-gcc-debug
 
+zen4-smoke-x86-64-gcc-debug:
+  extends: .zen4-x86-64
+  script:
+    - ./automation/scripts/qubes-x86-64.sh 2>&1 | tee ${LOGFILE}
+  needs:
+    - *x86-64-test-needs
+    - alpine-3.18-gcc-debug
+
+zen4-smoke-x86-64-dom0pvh gcc-debug:
+  extends: .zen4-x86-64
+  script:
+    - ./automation/scripts/qubes-x86-64.sh dom0pvh 2>&1 | tee ${LOGFILE}
+  needs:
+    - *x86-64-test-needs
+    - alpine-3.18-gcc-debug
+
+zen4-pci-hvm-x86-64-gcc-debug:
+  extends: .zen4-x86-64
+  script:
+    - ./automation/scripts/qubes-x86-64.sh pci-hvm 2>&1 | tee ${LOGFILE}
+  needs:
+    - *x86-64-test-needs
+    - alpine-3.18-gcc-debug
+
+zen4-tools-tests-pv-x86-64-gcc-debug:
+  extends: .zen4-x86-64
+  script:
+    - ./automation/scripts/qubes-x86-64.sh tools-tests-pv 2>&1 | tee ${LOGFILE}
+  artifacts:
+    reports:
+      junit: tests-junit.xml
+  needs:
+    - *x86-64-test-needs
+    - alpine-3.18-gcc-debug
+
+zen4-tools-tests-pvh-x86-64-gcc-debug:
+  extends: .zen4-x86-64
+  script:
+    - ./automation/scripts/qubes-x86-64.sh tools-tests-pvh 2>&1 | tee ${LOGFILE}
+  artifacts:
+    reports:
+      junit: tests-junit.xml
+  needs:
+    - *x86-64-test-needs
+    - alpine-3.18-gcc-debug
+
 qemu-smoke-dom0-arm64-gcc:
   extends: .qemu-arm64
   script:
diff --git a/automation/tests-artifacts/kernel/6.12.21.dockerfile b/automation/tests-artifacts/kernel/6.12.21.dockerfile
index d7d34031cab5..4e0a68ec61eb 100644
--- a/automation/tests-artifacts/kernel/6.12.21.dockerfile
+++ b/automation/tests-artifacts/kernel/6.12.21.dockerfile
@@ -19,6 +19,7 @@ RUN apt-get update && \
         flex \
         bison \
         libelf-dev \
+        libssl-dev \
         && \
     apt-get autoremove -y && \
     apt-get clean && \
@@ -33,6 +34,7 @@ RUN curl -fsSLO https://cdn.kernel.org/pub/linux/kernel/v6.x/linux-"$LINUX_VERSI
     scripts/config --enable BRIDGE && \
     scripts/config --enable IGC && \
     scripts/config --enable TUN && \
+    scripts/config --enable USB_RTL8152 && \
     cp .config .config.orig && \
     cat .config.orig | grep XEN | grep =m |sed 's/=m/=y/g' >> .config && \
     make -j$(nproc) bzImage && \
-- 
git-series 0.9.1

[PATCH v1 11/11] [DO NOT MERGE] container

[Marek Marczykowski-Górecki]

Switch to my container repo for rebuilt kernel
---
 automation/gitlab-ci/build.yaml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/automation/gitlab-ci/build.yaml b/automation/gitlab-ci/build.yaml
index d41f27fc94bf..ac7aca141010 100644
--- a/automation/gitlab-ci/build.yaml
+++ b/automation/gitlab-ci/build.yaml
@@ -312,7 +312,7 @@ qemu-system-aarch64-6.0.0-arm32-export:
 
 kernel-6.12.21-export:
   extends: .test-jobs-artifact-common
-  image: registry.gitlab.com/xen-project/xen/tests-artifacts/kernel:6.12.21
+  image: registry.gitlab.com/xen-project/people/marmarek/xen/tests-artifacts/kernel:6.12.21
   script:
     - mkdir binaries && cp /bzImage binaries/bzImage
   artifacts:
-- 
git-series 0.9.1

Re: [PATCH v1 01/11] ci: prevent grub unpacking initramfs

[Jan Beulich]

On 03.04.2025 13:04, Marek Marczykowski-Górecki wrote:
> It fails on larger initramfs (~250MB one) and sometimes even smaller
> depending on memory size/memory map, let Linux do it.

Iirc grub only unpacks gzip-ed modules, so wouldn't a yet better approach
be to use a better compressing algorithm, which simply as a side effect
would keep grub from decompressing it, while at the same time moving
farther away from any critical boundaries?

Jan

Re: [PATCH v1 02/11] ci: increase timeout for hw tests

[Jan Beulich]

On 03.04.2025 13:04, Marek Marczykowski-Górecki wrote:
> It appears as sometimes it takes more time for Xen even start booting,
> mostly due to firmware and fetching large boot files by grub. In some
> jobs the current timeout is pretty close to the actual time needed, and
> sometimes (rarely for now) test fails due to timeout expiring in the
> middle of dom0 booting. This will be happening more often if the
> initramfs will grow (and with more complex tests).

With that, ...

> This has been observed on some dom0pvh-hvm jobs, at least on runners hw3
> and hw11.
> 
> Increase the timeout by yet another 60s (up to 180s now).

... is this little a bump going to be sufficient? How about moving straight
to 5min?

As to observed failing jobs - the PV Dom0 boot failure seen today looks to
also be due to too short a timeout.

Jan

Re: [PATCH v1 01/11] ci: prevent grub unpacking initramfs

[Marek Marczykowski-Górecki]

[-- Attachment #1: Type: text/plain, Size: 750 bytes --]

On Thu, Apr 03, 2025 at 01:29:16PM +0200, Jan Beulich wrote:
> On 03.04.2025 13:04, Marek Marczykowski-Górecki wrote:
> > It fails on larger initramfs (~250MB one) and sometimes even smaller
> > depending on memory size/memory map, let Linux do it.
> 
> Iirc grub only unpacks gzip-ed modules, so wouldn't a yet better approach
> be to use a better compressing algorithm, which simply as a side effect
> would keep grub from decompressing it, while at the same time moving
> farther away from any critical boundaries?

That should also make the test faster (if it's a significant
difference). OTOH likely we wouldn't have found the issue with
map_pages_to_xen().

-- 
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [PATCH v1 01/11] ci: prevent grub unpacking initramfs

[Andrew Cooper]

On 03/04/2025 12:29 pm, Jan Beulich wrote:
> On 03.04.2025 13:04, Marek Marczykowski-Górecki wrote:
>> It fails on larger initramfs (~250MB one) and sometimes even smaller
>> depending on memory size/memory map, let Linux do it.
> Iirc grub only unpacks gzip-ed modules, so wouldn't a yet better approach
> be to use a better compressing algorithm, which simply as a side effect
> would keep grub from decompressing it, while at the same time moving
> farther away from any critical boundaries?

Yes and no.

This is going to change anyway when I (or a delgee) moves initrd
generation from the test step itself into the test artefacts repo.

Switching to a slower algorithm will impact every test step right now.

Real systems doesn't see this in general, because it there's an
uncompressed microcode container at the front, and it's not identified
as being compressed at all.

I'd prefer to stick with Marek's patch in the short term.

~Andrew

Re: [PATCH v1 01/11] ci: prevent grub unpacking initramfs

[Jan Beulich]

On 03.04.2025 13:39, Andrew Cooper wrote:
> On 03/04/2025 12:29 pm, Jan Beulich wrote:
>> On 03.04.2025 13:04, Marek Marczykowski-Górecki wrote:
>>> It fails on larger initramfs (~250MB one) and sometimes even smaller
>>> depending on memory size/memory map, let Linux do it.
>> Iirc grub only unpacks gzip-ed modules, so wouldn't a yet better approach
>> be to use a better compressing algorithm, which simply as a side effect
>> would keep grub from decompressing it, while at the same time moving
>> farther away from any critical boundaries?
> 
> Yes and no.
> 
> This is going to change anyway when I (or a delgee) moves initrd
> generation from the test step itself into the test artefacts repo.
> 
> Switching to a slower algorithm will impact every test step right now.
> 
> Real systems doesn't see this in general, because it there's an
> uncompressed microcode container at the front, and it's not identified
> as being compressed at all.
> 
> I'd prefer to stick with Marek's patch in the short term.

Fair enough.

Jan

Re: [PATCH v1 02/11] ci: increase timeout for hw tests

[Marek Marczykowski-Górecki]

[-- Attachment #1: Type: text/plain, Size: 1787 bytes --]

On Thu, Apr 03, 2025 at 01:32:38PM +0200, Jan Beulich wrote:
> On 03.04.2025 13:04, Marek Marczykowski-Górecki wrote:
> > It appears as sometimes it takes more time for Xen even start booting,
> > mostly due to firmware and fetching large boot files by grub. In some
> > jobs the current timeout is pretty close to the actual time needed, and
> > sometimes (rarely for now) test fails due to timeout expiring in the
> > middle of dom0 booting. This will be happening more often if the
> > initramfs will grow (and with more complex tests).
> 
> With that, ...
> 
> > This has been observed on some dom0pvh-hvm jobs, at least on runners hw3
> > and hw11.
> > 
> > Increase the timeout by yet another 60s (up to 180s now).
> 
> ... is this little a bump going to be sufficient? How about moving straight
> to 5min?

I don't like this, as many (most) actual failures are visible as timeout
(for example panic that prevents reaching Alpine prompt). One
improvement I can see is splitting this into two separate timeouts: one
before seeing the first line from Xen and then the second one for
reaching Alpine login prompt. The first one can be longer as its mostly
about firmware+fetching boot files and shouldn't hit on crashes (unless
a crash happen before printing anything on the console - but those are
rare).

> As to observed failing jobs - the PV Dom0 boot failure seen today looks to
> also be due to too short a timeout.

As responded on Matrix, I'm not so sure, there is over 1m wait after
"Built 1 zonelists, mobility grouping on.  Total pages: 8228487" line
from dom0 (or a bit later, due to buffering by sed), while in successful
test next lines follow instantaneously.

-- 
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [PATCH v1 01/11] ci: prevent grub unpacking initramfs

[Stefano Stabellini]

[-- Attachment #1: Type: text/plain, Size: 1145 bytes --]

On Thu, 2 Apr 2025, Marek Marczykowski-Górecki wrote:
> It fails on larger initramfs (~250MB one) and sometimes even smaller
> depending on memory size/memory map, let Linux do it.
> 
> Signed-off-by: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com>

I believe this is the default with U-Boot anyway, if I remember right.

Reviewed-by: Stefano Stabellini <sstabellini@kernel.org>

> ---
>  automation/scripts/qubes-x86-64.sh | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/automation/scripts/qubes-x86-64.sh b/automation/scripts/qubes-x86-64.sh
> index f27296445901..8e78b7984e98 100755
> --- a/automation/scripts/qubes-x86-64.sh
> +++ b/automation/scripts/qubes-x86-64.sh
> @@ -248,7 +248,7 @@ CONTROLLER=control@thor.testnet
>  echo "
>  multiboot2 (http)/gitlab-ci/xen $CONSOLE_OPTS loglvl=all guest_loglvl=all dom0_mem=4G console_timestamps=boot $extra_xen_opts
>  module2 (http)/gitlab-ci/vmlinuz console=hvc0 root=/dev/ram0 earlyprintk=xen
> -module2 (http)/gitlab-ci/initrd-dom0
> +module2 --nounzip (http)/gitlab-ci/initrd-dom0
>  " > $TFTP/grub.cfg
>  
>  echo "#!ipxe
> -- 
> git-series 0.9.1
> 

Re: [PATCH v1 02/11] ci: increase timeout for hw tests

[Stefano Stabellini]

[-- Attachment #1: Type: text/plain, Size: 1701 bytes --]

On Thu, 3 Apr 2025, Marek Marczykowski-Górecki wrote:
> On Thu, Apr 03, 2025 at 01:32:38PM +0200, Jan Beulich wrote:
> > On 03.04.2025 13:04, Marek Marczykowski-Górecki wrote:
> > > It appears as sometimes it takes more time for Xen even start booting,
> > > mostly due to firmware and fetching large boot files by grub. In some
> > > jobs the current timeout is pretty close to the actual time needed, and
> > > sometimes (rarely for now) test fails due to timeout expiring in the
> > > middle of dom0 booting. This will be happening more often if the
> > > initramfs will grow (and with more complex tests).
> > 
> > With that, ...
> > 
> > > This has been observed on some dom0pvh-hvm jobs, at least on runners hw3
> > > and hw11.
> > > 
> > > Increase the timeout by yet another 60s (up to 180s now).
> > 
> > ... is this little a bump going to be sufficient? How about moving straight
> > to 5min?

Hi Marek, would you be up for moving your script to use "expect"?
Something like ./automation/scripts/console.exp?

That way, we would immediately complete the job no matter the timeout
value. It is also nicer :-)


> I don't like this, as many (most) actual failures are visible as timeout
> (for example panic that prevents reaching Alpine prompt). One
> improvement I can see is splitting this into two separate timeouts: one
> before seeing the first line from Xen and then the second one for
> reaching Alpine login prompt. The first one can be longer as its mostly
> about firmware+fetching boot files and shouldn't hit on crashes (unless
> a crash happen before printing anything on the console - but those are
> rare).

This is also something you can very specifically tweak with expect.

Re: [PATCH v1 03/11] ci: enable XHCI console in Xen debug build on Alpine

[Stefano Stabellini]

[-- Attachment #1: Type: text/plain, Size: 800 bytes --]

On Thu, 3 Apr 2025, Marek Marczykowski-Górecki wrote:
> This build is used for hardware tests, and some runners use XHCI console
> 
> Signed-off-by: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com>

Acked-by: Stefano Stabellini <sstabellini@kernel.org>

> ---
>  automation/gitlab-ci/build.yaml | 1 +
>  1 file changed, 1 insertion(+)
> 
> diff --git a/automation/gitlab-ci/build.yaml b/automation/gitlab-ci/build.yaml
> index 2513908b059b..169bebe3c775 100644
> --- a/automation/gitlab-ci/build.yaml
> +++ b/automation/gitlab-ci/build.yaml
> @@ -341,6 +341,7 @@ alpine-3.18-gcc-debug:
>        CONFIG_ARGO=y
>        CONFIG_UBSAN=y
>        CONFIG_UBSAN_FATAL=y
> +      CONFIG_XHCI=y
>  
>  debian-12-x86_64-gcc-debug:
>    extends: .gcc-x86-64-build-debug
> -- 
> git-series 0.9.1
> 

Re: [PATCH v1 04/11] ci: include domU kernel messages in the console output log

[Stefano Stabellini]

[-- Attachment #1: Type: text/plain, Size: 742 bytes --]

On Thu, 3 Apr 2025, Marek Marczykowski-Górecki wrote:
> Signed-off-by: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com>

Acked-by: Stefano Stabellini <sstabellini@kernel.org>

> ---
>  automation/scripts/qubes-x86-64.sh | 2 ++
>  1 file changed, 2 insertions(+)
> 
> diff --git a/automation/scripts/qubes-x86-64.sh b/automation/scripts/qubes-x86-64.sh
> index 771c77d6618b..d316c17e0c9f 100755
> --- a/automation/scripts/qubes-x86-64.sh
> +++ b/automation/scripts/qubes-x86-64.sh
> @@ -175,6 +175,8 @@ if [ -n "$domU_check" ]; then
>      rm var/run
>      echo "#!/bin/sh
>  
> +echo 8 > /proc/sys/kernel/printk
> +
>  ${domU_check}
>  " > etc/local.d/xen.start
>      chmod +x etc/local.d/xen.start
> -- 
> git-series 0.9.1
> 

Re: [PATCH v1 05/11] ci: increase verbosity of starting a domain

[Stefano Stabellini]

[-- Attachment #1: Type: text/plain, Size: 2860 bytes --]

On Thu, 3 Apr 2025, Marek Marczykowski-Górecki wrote:
> And start collecting qemu log earlier, so it isn't lost in case of a
> timeout during domain startup.
> 
> Signed-off-by: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com>

As we are doing this across the board please also do the same for
xilinx-smoke-dom0-x86_64.sh


> ---
>  automation/scripts/qemu-alpine-x86_64.sh    | 2 +-
>  automation/scripts/qemu-smoke-dom0-arm32.sh | 2 +-
>  automation/scripts/qemu-smoke-dom0-arm64.sh | 2 +-
>  automation/scripts/qubes-x86-64.sh          | 4 ++--
>  4 files changed, 5 insertions(+), 5 deletions(-)
> 
> diff --git a/automation/scripts/qemu-alpine-x86_64.sh b/automation/scripts/qemu-alpine-x86_64.sh
> index 1ff689b577e3..17e2141d625e 100755
> --- a/automation/scripts/qemu-alpine-x86_64.sh
> +++ b/automation/scripts/qemu-alpine-x86_64.sh
> @@ -56,7 +56,7 @@ bash /etc/init.d/xencommons start
>  
>  xl list
>  
> -xl create -c /root/test.cfg
> +xl -vvv create -c /root/test.cfg
>  
>  " > etc/local.d/xen.start
>  chmod +x etc/local.d/xen.start
> diff --git a/automation/scripts/qemu-smoke-dom0-arm32.sh b/automation/scripts/qemu-smoke-dom0-arm32.sh
> index e1cd83880928..0c60a66e25e3 100755
> --- a/automation/scripts/qemu-smoke-dom0-arm32.sh
> +++ b/automation/scripts/qemu-smoke-dom0-arm32.sh
> @@ -21,7 +21,7 @@ echo "#!/bin/bash
>  
>  xl list
>  
> -xl create -c /root/test.cfg
> +xl -vvv create -c /root/test.cfg
>  
>  " > ./root/xen.start
>  echo "bash /root/xen.start" >> ./etc/init.d/xen-watchdog
> diff --git a/automation/scripts/qemu-smoke-dom0-arm64.sh b/automation/scripts/qemu-smoke-dom0-arm64.sh
> index 4d22a124df11..8774a8701232 100755
> --- a/automation/scripts/qemu-smoke-dom0-arm64.sh
> +++ b/automation/scripts/qemu-smoke-dom0-arm64.sh
> @@ -52,7 +52,7 @@ bash /etc/init.d/xencommons start
>  
>  xl list
>  
> -xl create -c /root/test.cfg
> +xl -vvv create -c /root/test.cfg
>  
>  " > etc/local.d/xen.start
>  chmod +x etc/local.d/xen.start
> diff --git a/automation/scripts/qubes-x86-64.sh b/automation/scripts/qubes-x86-64.sh
> index d316c17e0c9f..822b81d186a7 100755
> --- a/automation/scripts/qubes-x86-64.sh
> +++ b/automation/scripts/qubes-x86-64.sh
> @@ -122,7 +122,6 @@ echo \"${passed}\"
>  "
>  
>          dom0_check="
> -tail -F /var/log/xen/qemu-dm-domU.log &
>  until grep -q \"^domU Welcome to Alpine Linux\" /var/log/xen/console/guest-domU.log; do
>      sleep 1
>  done
> @@ -222,7 +221,8 @@ if [ -n "$domU_check" ]; then
>      echo "
>  # get domU console content into test log
>  tail -F /var/log/xen/console/guest-domU.log 2>/dev/null | sed -e \"s/^/(domU) /\" &
> -xl create /etc/xen/domU.cfg
> +tail -F /var/log/xen/qemu-dm-domU.log 2>/dev/null | sed -e \"s/^/(qemu-dm) /\" &
> +xl -vvv create /etc/xen/domU.cfg
>  ${dom0_check}
>  " >> etc/local.d/xen.start
>  else
> -- 
> git-series 0.9.1
> 

Re: [PATCH v1 06/11] ci: consistently use DOCKER_CMD in makefiles

[Stefano Stabellini]

[-- Attachment #1: Type: text/plain, Size: 1638 bytes --]

On Thu, 3 Apr 2025, Marek Marczykowski-Górecki wrote:
> This allows rebuilding containers using podman too.
> 
> Signed-off-by: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com>


Reviewed-by: Stefano Stabellini <sstabellini@kernel.org>


> ---
>  automation/build/Makefile           | 4 ++--
>  automation/tests-artifacts/Makefile | 4 ++--
>  2 files changed, 4 insertions(+), 4 deletions(-)
> 
> diff --git a/automation/build/Makefile b/automation/build/Makefile
> index 4df43b040777..fedf7524dacd 100644
> --- a/automation/build/Makefile
> +++ b/automation/build/Makefile
> @@ -31,8 +31,8 @@ clean:
>  define CLEAN_RULE
>  .PHONY: clean-$(1)
>  clean-$(1):
> -	if [ -n "$$$$(docker image ls -q $(REGISTRY)/$(subst /,:,$(1)))" ]; then \
> -		docker image rm $(REGISTRY)/$(subst /,:,$(1)); \
> +	if [ -n "$$$$($(DOCKER_CMD) image ls -q $(REGISTRY)/$(subst /,:,$(1)))" ]; then \
> +		$(DOCKER_CMD) image rm $(REGISTRY)/$(subst /,:,$(1)); \
>  	fi
>  
>  endef
> diff --git a/automation/tests-artifacts/Makefile b/automation/tests-artifacts/Makefile
> index d055cd696bed..80a60a94f3f7 100644
> --- a/automation/tests-artifacts/Makefile
> +++ b/automation/tests-artifacts/Makefile
> @@ -10,9 +10,9 @@ help:
>  	@echo "To push container builds, set the env var PUSH"
>  
>  %: %.dockerfile ## Builds containers
> -	docker build --pull -t $(REGISTRY)/$(@D):$(@F) -f $< $(<D)
> +	$(DOCKER_CMD) build --pull -t $(REGISTRY)/$(@D):$(@F) -f $< $(<D)
>  	@if [ ! -z $${PUSH+x} ]; then \
> -		docker push $(REGISTRY)/$(@D):$(@F); \
> +		$(DOCKER_CMD) push $(REGISTRY)/$(@D):$(@F); \
>  	fi
>  
>  .PHONY: all
> -- 
> git-series 0.9.1
> 

Re: [PATCH v1 02/11] ci: increase timeout for hw tests

[Marek Marczykowski-Górecki]

[-- Attachment #1: Type: text/plain, Size: 539 bytes --]

On Thu, Apr 03, 2025 at 05:21:56PM -0700, Stefano Stabellini wrote:
> Hi Marek, would you be up for moving your script to use "expect"?
> Something like ./automation/scripts/console.exp?
> 
> That way, we would immediately complete the job no matter the timeout
> value. It is also nicer :-)

Oh, this is excellent idea, I'll see how it fits there. It seems I'll
need at least add support for wakeup for the suspend tests, but that
shouldn't be a problem.

-- 
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [PATCH v1 07/11] ci: wait for the network interface in PCI passthrough tests

[Stefano Stabellini]

[-- Attachment #1: Type: text/plain, Size: 1151 bytes --]

On Thu, 3 Apr 2025, Marek Marczykowski-Górecki wrote:
> The network driver initializes asynchronously, and it may not be ready
> yet by the time the startup script is called. This is especially the
> case for USB network adapter (where the PCI device is the USB
> controller) in the upcoming runner.
> 
> Don't bother about separate timeout - test timeout will cover this part
> too.
> 
> Signed-off-by: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com>

Reviewed-by: Stefano Stabellini <sstabellini@kernel.org>


> ---
>  automation/scripts/qubes-x86-64.sh | 1 +
>  1 file changed, 1 insertion(+)
> 
> diff --git a/automation/scripts/qubes-x86-64.sh b/automation/scripts/qubes-x86-64.sh
> index 822b81d186a7..6d2c170d1ee1 100755
> --- a/automation/scripts/qubes-x86-64.sh
> +++ b/automation/scripts/qubes-x86-64.sh
> @@ -104,6 +104,7 @@ on_reboot = "destroy"
>          domU_check="
>  set -x -e
>  interface=eth0
> +while ! [ -e \"/sys/class/net/\$interface\" ]; do sleep 1; done
>  ip link set \"\$interface\" up
>  timeout 30s udhcpc -i \"\$interface\"
>  pingip=\$(ip -o -4 r show default|cut -f 3 -d ' ')
> -- 
> git-series 0.9.1
> 

Re: [PATCH v1 09/11] ci: adjust resolving network interface into PCI device

[Stefano Stabellini]

[-- Attachment #1: Type: text/plain, Size: 1383 bytes --]

On Thu, 3 Apr 2025, Marek Marczykowski-Górecki wrote:
> Change how PCI device lookup is done to handle also USB devices, in
> which case get the USB controller. Instead of taking basename of the
> 'device' symlink, resolve the full path (example:
> /sys/devices/pci0000:00/0000:00:09.0/usb4/4-7/4-7:1.0) and take the
> first part after pci0000:00. Theoretically it could be a bridge, but VM
> has flat PCI topology.
> 
> Signed-off-by: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com>
> ---
>  automation/scripts/qubes-x86-64.sh | 3 ++-
>  1 file changed, 2 insertions(+), 1 deletion(-)
> 
> diff --git a/automation/scripts/qubes-x86-64.sh b/automation/scripts/qubes-x86-64.sh
> index 6d2c170d1ee1..dc8013750f5f 100755
> --- a/automation/scripts/qubes-x86-64.sh
> +++ b/automation/scripts/qubes-x86-64.sh
> @@ -110,7 +110,8 @@ timeout 30s udhcpc -i \"\$interface\"
>  pingip=\$(ip -o -4 r show default|cut -f 3 -d ' ')
>  ping -c 10 \"\$pingip\"
>  echo domU started
> -pcidevice=\$(basename \$(readlink /sys/class/net/\$interface/device))
> +pcidevice=\$(realpath /sys/class/net/\$interface/device |
> +             sed 's#.*pci0000:00/\\([^/]*\\).*#\\1#')

This is still a bit fragile but better than before

Reviewed-by: Stefano Stabellini <sstabellini@kernel.org>


>  lspci -vs \$pcidevice
>  "
>          if [ -n "$PCIDEV_INTR" ]; then
> -- 
> git-series 0.9.1
> 

Re: [PATCH v1 10/11] ci: add AMD Zen 4 HW runner

[Stefano Stabellini]

[-- Attachment #1: Type: text/plain, Size: 4336 bytes --]

On Thu, 3 Apr 2025, Marek Marczykowski-Górecki wrote:
> This is AMD Ryzen 5 7640U in Framework Laptop AMD.
> It has several things different than the other runners.
> First of all, the console is using XHCI debug capability.
> And then, this system doesn't have normal wired ethernet. But is has one
> on USB, and this one is used for booting. For this, enable
> CONFIG_USB_RTL8152 in the Linux build.
> 
> Include some basic tests, plus PCI passthrough.
> 
> This machine doesn't support S3. S0ix pretends to be working with
> Qubes's dom0 (kernel + userspace), but it hangs on suspend with Alpine
> used in test. But even when it doesn't hang, it doesn't really reach
> deep sleep, so skip this test for now here.
> 
> Signed-off-by: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com>
> ---
> Requires containers rebuild, especially tests-artifacts/kernel/6.12.21.
> 
> Do not apply yet, until issues found by those tests are fixed. That
> would be at least third issue found using this system, so lets have it
> in CI.

I'd prefer to wait 1 week before committing. In the meantime:

Reviewed-by: Stefano Stabellini <sstabellini@kernel.org>



> ---
>  automation/gitlab-ci/test.yaml                       | 56 +++++++++++++-
>  automation/tests-artifacts/kernel/6.12.21.dockerfile |  2 +-
>  2 files changed, 58 insertions(+)
> 
> diff --git a/automation/gitlab-ci/test.yaml b/automation/gitlab-ci/test.yaml
> index a13dd040bd26..feb2dd10ab30 100644
> --- a/automation/gitlab-ci/test.yaml
> +++ b/automation/gitlab-ci/test.yaml
> @@ -188,6 +188,16 @@
>    tags:
>      - qubes-hw11
>  
> +.zen4-x86-64:
> +  extends: .adl-x86-64
> +  variables:
> +    PCIDEV: "c3:00.4"
> +    PCIDEV_INTR: "MSI-X"
> +    CONSOLE_OPTS: "console=xhci dbgp=xhci@pcic1:00.3,share=yes"
> +    SUT_ADDR: test-12.testnet
> +  tags:
> +    - qubes-hw12
> +
>  # Test jobs
>  build-each-commit-gcc:
>    extends: .test-jobs-common
> @@ -479,6 +489,52 @@ zen3p-tools-tests-pvh-x86-64-gcc-debug:
>      - *x86-64-test-needs
>      - alpine-3.18-gcc-debug
>  
> +zen4-smoke-x86-64-gcc-debug:
> +  extends: .zen4-x86-64
> +  script:
> +    - ./automation/scripts/qubes-x86-64.sh 2>&1 | tee ${LOGFILE}
> +  needs:
> +    - *x86-64-test-needs
> +    - alpine-3.18-gcc-debug
> +
> +zen4-smoke-x86-64-dom0pvh gcc-debug:
> +  extends: .zen4-x86-64
> +  script:
> +    - ./automation/scripts/qubes-x86-64.sh dom0pvh 2>&1 | tee ${LOGFILE}
> +  needs:
> +    - *x86-64-test-needs
> +    - alpine-3.18-gcc-debug
> +
> +zen4-pci-hvm-x86-64-gcc-debug:
> +  extends: .zen4-x86-64
> +  script:
> +    - ./automation/scripts/qubes-x86-64.sh pci-hvm 2>&1 | tee ${LOGFILE}
> +  needs:
> +    - *x86-64-test-needs
> +    - alpine-3.18-gcc-debug
> +
> +zen4-tools-tests-pv-x86-64-gcc-debug:
> +  extends: .zen4-x86-64
> +  script:
> +    - ./automation/scripts/qubes-x86-64.sh tools-tests-pv 2>&1 | tee ${LOGFILE}
> +  artifacts:
> +    reports:
> +      junit: tests-junit.xml
> +  needs:
> +    - *x86-64-test-needs
> +    - alpine-3.18-gcc-debug
> +
> +zen4-tools-tests-pvh-x86-64-gcc-debug:
> +  extends: .zen4-x86-64
> +  script:
> +    - ./automation/scripts/qubes-x86-64.sh tools-tests-pvh 2>&1 | tee ${LOGFILE}
> +  artifacts:
> +    reports:
> +      junit: tests-junit.xml
> +  needs:
> +    - *x86-64-test-needs
> +    - alpine-3.18-gcc-debug
> +
>  qemu-smoke-dom0-arm64-gcc:
>    extends: .qemu-arm64
>    script:
> diff --git a/automation/tests-artifacts/kernel/6.12.21.dockerfile b/automation/tests-artifacts/kernel/6.12.21.dockerfile
> index d7d34031cab5..4e0a68ec61eb 100644
> --- a/automation/tests-artifacts/kernel/6.12.21.dockerfile
> +++ b/automation/tests-artifacts/kernel/6.12.21.dockerfile
> @@ -19,6 +19,7 @@ RUN apt-get update && \
>          flex \
>          bison \
>          libelf-dev \
> +        libssl-dev \
>          && \
>      apt-get autoremove -y && \
>      apt-get clean && \
> @@ -33,6 +34,7 @@ RUN curl -fsSLO https://cdn.kernel.org/pub/linux/kernel/v6.x/linux-"$LINUX_VERSI
>      scripts/config --enable BRIDGE && \
>      scripts/config --enable IGC && \
>      scripts/config --enable TUN && \
> +    scripts/config --enable USB_RTL8152 && \
>      cp .config .config.orig && \
>      cat .config.orig | grep XEN | grep =m |sed 's/=m/=y/g' >> .config && \
>      make -j$(nproc) bzImage && \
> -- 
> git-series 0.9.1
> 

Re: [PATCH v1 08/11] ci: switch test kernel from 6.1.19 to 6.12.21

[Marek Marczykowski-Górecki]

[-- Attachment #1: Type: text/plain, Size: 645 bytes --]

On Thu, Apr 03, 2025 at 05:40:07PM -0700, Stefano Stabellini wrote:
> On Thu, 3 Apr 2025, Marek Marczykowski-Górecki wrote:
> > The jump is pretty big, but should be enough for a longer time. The
> > change right now is motivated adding new runner with AMD Zen4.
> > 
> > Signed-off-by: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com>
> 
> As long as all the tests (including QEMU and xilinx-x86) pass:

I tested all I can (and it works), but I don't have access to the xilinx runner.

> Reviewed-by: Stefano Stabellini <sstabellini@kernel.org>

-- 
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [PATCH v1 01/11] ci: prevent grub unpacking initramfs

[Marek Marczykowski-Górecki]

[-- Attachment #1: Type: text/plain, Size: 3014 bytes --]

On Thu, Apr 03, 2025 at 12:39:11PM +0100, Andrew Cooper wrote:
> On 03/04/2025 12:29 pm, Jan Beulich wrote:
> > On 03.04.2025 13:04, Marek Marczykowski-Górecki wrote:
> >> It fails on larger initramfs (~250MB one) and sometimes even smaller
> >> depending on memory size/memory map, let Linux do it.
> > Iirc grub only unpacks gzip-ed modules, so wouldn't a yet better approach
> > be to use a better compressing algorithm, which simply as a side effect
> > would keep grub from decompressing it, while at the same time moving
> > farther away from any critical boundaries?
> 
> Yes and no.
> 
> This is going to change anyway when I (or a delgee) moves initrd
> generation from the test step itself into the test artefacts repo.

Out of curiosity, I tried this:
https://gitlab.com/xen-project/people/marmarek/xen/-/commit/51b5fde816806206e6e0c72ec0146c5ab3da4351
pipeline:
https://gitlab.com/xen-project/people/marmarek/xen/-/pipelines/1753304751

It avoids repacking base alpine initrd, but it still needs to pack
per-job stuff (test script, but also toolstack). The gains are not
great, that step still takes over a minute. The overall job time went
down by just 40-50s (for example from 6m50s to 6m03s). This isn't very
surprising, because the base initrd.cpio.gz is 60MB and the final one is
over 200MB, so the part that was needlessly re-compressed is about 30%.
The numbers checks out, as the (debug) build artifacts zip is about
150MB.

One further idea would be to produce cpio.gz archive as part of the
build artifact, to avoid zip (used by gitlab for artifacts) -> cpio.gz
recompression. But that either makes build artifacts bigger for every
job, or need setting a variable for jobs that are used later for tests.
Unless we decide to store toolstack _only_ as cpio.gz in artifacts?

I tried the latter idea here:
https://gitlab.com/xen-project/people/marmarek/xen/-/commit/a4774213d83ca05c0b0b0fff80e49cf3aa32b67b
pipeline:
https://gitlab.com/xen-project/people/marmarek/xen/-/pipelines/1753312631

This helped quite a bit more. The 1m of compressing initrd is gone now.
Compressing remaining test-specific files and concatenating it all
together takes 2-3s. Now that 6m50s job is down to 3m40s. I'm not
exactly sure where that extra minute went (maybe related to number of
files in the build artifact? Fetching/extracting artifacts seems to be
faster a bit), but those times are not very consistent anyway, so it
could be also just random fluctuation. 

Looking further at the test log, some time is used for fetching
artifacts. I've looked into gitlab docs if those can be cached on the
runner, but I haven't found a solution that would really help (you can
cache arbitrary paths, including artifacts, but it doesn't prevent
re-downloading them and unpacking over the cached version...).
So, I'll stop here, polish the above patches and submit them in v2 of
this series.

-- 
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]