Re: [PATCH v3 5/7] CI: avoid repacking initrd as part of the test job

[Andrew Cooper <andrew.cooper3@citrix.com>]

On 11/04/2025 9:32 pm, Marek Marczykowski-Górecki wrote:
> The prerequisite for this is building rootfs.cpio.gz, not only
> initrd.tar.gz. That's a change in the test-artifacts repository in the
> "Rootfs generation" series from Andrew.
>
> Having that, do not repack the whole initrd, but only pack modified
> files and rely on Linux handling of concatenated archives.
> This allows packing just test-related files (which includes the whole
> toolstack), instead of the whole initrd.
>
> DomU initrd handling is a bit more complicated thing. It's sent to the
> target host as part of the dom0 initrd. For now include prepared domU
> initrd as a whole in dom0's rootfs "overlay", which means compressing it
> (again) each job (this takes 3s). This can be later improved by
> splitting domU initrd into two parts (base + overlay) and concatenate
> them in target dom0 already. For this to be useful, test-artifacts would
> need to provide pre-made double-cpio version too (rootfs.cpio.gz wrapper
> in yet another cpio.gz as boot/domU-initrd).
>
> Since full initrd is not unpacked now when preparing domU (and dom0)
> rootfs, a couple of minor changes are needed to how they are prepared.
> This includes creating whole etc/issue file, instead of modifying
> existing one, and a need to create a couple directories.
>
> Finally, move adding "rc_verbose=yes" to /etc/rc.conf to initrd
> building.
>
> Signed-off-by: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com>
> ---
> Changes in v3:
> - adjust for Andrew's version of cpio rootfs artifacts
> - don't require double-cpio archive for domU initrd, instead add it via
>   dom0 initrd as plain cpio to avoid re-compressing the same thing
> New in v2
>
> Jobs xilinx-* are untested, as I don't have necessary access.

I can test these.

> ---
>  automation/gitlab-ci/test.yaml                    |  6 +++--
>  automation/scripts/qemu-alpine-x86_64.sh          |  9 +++-----
>  automation/scripts/qemu-smoke-dom0-arm64.sh       |  7 ++----
>  automation/scripts/qemu-smoke-dom0less-arm64.sh   |  7 ++----
>  automation/scripts/qubes-x86-64.sh                | 18 +++++++---------
>  automation/scripts/xilinx-smoke-dom0-x86_64.sh    | 15 ++++++-------
>  automation/scripts/xilinx-smoke-dom0less-arm64.sh | 14 +++++-------
>  7 files changed, 35 insertions(+), 41 deletions(-)
>
> diff --git a/automation/gitlab-ci/test.yaml b/automation/gitlab-ci/test.yaml
> index 59a2de28c864..3e02bf0e4d21 100644
> --- a/automation/gitlab-ci/test.yaml
> +++ b/automation/gitlab-ci/test.yaml
> @@ -11,7 +11,9 @@
>    - project: xen-project/hardware/test-artifacts
>      job: linux-6.6.86-arm64
>      ref: master
> -  - alpine-3.18-arm64-rootfs-export
> +  - project: xen-project/hardware/test-artifacts
> +    job: alpine-3.18-arm64-rootfs
> +    ref: master
>    - qemu-system-aarch64-6.0.0-arm64-export
>  
>  .arm32-test-needs: &arm32-test-needs
> @@ -22,7 +24,7 @@
>      job: linux-6.6.56-x86_64
>      ref: master
>    - project: xen-project/hardware/test-artifacts
> -    job: x86_64-rootfs-alpine-3.18
> +    job: alpine-3.18-x86_64-rootfs
>      ref: master
>  
>  .qemu-arm64:
> diff --git a/automation/scripts/qemu-alpine-x86_64.sh b/automation/scripts/qemu-alpine-x86_64.sh
> index 89bdb4df7dac..6e0a67b4018c 100755
> --- a/automation/scripts/qemu-alpine-x86_64.sh
> +++ b/automation/scripts/qemu-alpine-x86_64.sh

There are some very poor choices of name in this script.  I'll do a prep
patch to make them consistent.

> @@ -28,16 +28,15 @@ cd initrd
>  find . | cpio --create --format='newc' | gzip > ../initrd.cpio.gz
>  cd ..
>  
> -# initrd.tar.gz is Dom0 rootfs
> +# Dom0 rootfs - overlay on top of rootfs.cpio.gz
>  mkdir -p rootfs
>  cd rootfs
> -tar xvzf ../initrd.tar.gz
>  mkdir proc
>  mkdir run
>  mkdir srv
>  mkdir sys
> -rm var/run

I'm not sure why we need to make proc, run, srv, sys in the overlay
image, but this is repeated everywhere.

srv is just a plain empty dir.  It's missing from $PATHS in the rootfs
generation, so that's easy.

proc, run and sys are more complicated, because we don't want to simply
copy them from the rootfs generation environment.

dev is also in the same boat, and what we do now seems to cause openrc
not to seed it normally, which seems unwise.

I'll see what I can do to clean this all up.

> diff --git a/automation/scripts/xilinx-smoke-dom0-x86_64.sh b/automation/scripts/xilinx-smoke-dom0-x86_64.sh
> index 4db249530823..52a0e05e3f1b 100755
> --- a/automation/scripts/xilinx-smoke-dom0-x86_64.sh
> +++ b/automation/scripts/xilinx-smoke-dom0-x86_64.sh
> @@ -80,6 +80,7 @@ argo-exec -l -p 28333 -- /bin/echo
>  "
>  copy_dom0_files ()
>  {
> +    mkdir -p root usr/local/lib
>      cp "${WORKDIR}/binaries/xen-argo.ko" "root/"
>      cp -ar "${WORKDIR}/binaries/lib/"* "usr/local/lib/"
>      cp "${WORKDIR}/binaries/argo-exec" "usr/local/bin/"

This would be better based on top of my patch to switch to the new argo,
at which point it will simply be to append argo.cpio.gz.

~Andrew