[PATCH TEST-ARTEFACTS v2 0/7] Rootfs/argo cleanup and microcode support

[PATCH TEST-ARTEFACTS v2 0/7] Rootfs/argo cleanup and microcode support

[Andrew Cooper]

This series supersedes the others posted.  Only minor changes from before; see
patches for details.

See the companion Xen series for the testing using these new artefacts.

Andrew Cooper (7):
  Rename and adjust x86 containers
  Overhaul how Argo is built and packged
  Rework rootfs generation to make a cpio archive
  Shrink the rootfs substantially
  Adjust directory layout for rootfs.cpio.gz
  Provide an ARM64 rootfs too
  Package microcode for the x86 hardware runners

 .gitlab-ci.yml                                | 26 ++++-
 containerize                                  |  5 +-
 images/alpine/3.18-arm64-base.dockerfile      |  6 ++
 images/alpine/3.18-x86_64-base.dockerfile     |  6 ++
 ...ockerfile => 3.18-x86_64-build.dockerfile} | 35 ++++---
 images/alpine/x86_64-rootfs.dockerfile        |  4 -
 scripts/alpine-rootfs.sh                      | 95 +++++++++++++++++++
 scripts/build-argo.sh                         | 67 +++++++++++++
 scripts/build-linux.sh                        |  8 +-
 scripts/x86-microcode.sh                      | 42 ++++++++
 scripts/x86_64-rootfs-alpine.sh               | 60 ------------
 11 files changed, 272 insertions(+), 82 deletions(-)
 create mode 100644 images/alpine/3.18-arm64-base.dockerfile
 create mode 100644 images/alpine/3.18-x86_64-base.dockerfile
 rename images/alpine/{x86_64-build.dockerfile => 3.18-x86_64-build.dockerfile} (51%)
 delete mode 100644 images/alpine/x86_64-rootfs.dockerfile
 create mode 100755 scripts/alpine-rootfs.sh
 create mode 100755 scripts/build-argo.sh
 create mode 100755 scripts/x86-microcode.sh
 delete mode 100755 scripts/x86_64-rootfs-alpine.sh

-- 
2.39.5

[PATCH v2 1/7] Rename and adjust x86 containers

[Andrew Cooper]

Insert the Alpine version as we'll need a newer version in due course, and
rename 'rootfs' to 'base' as the container really is not a rootfs on it's own.

In base, add bash.  It's added later by the rootfs generation anyway, but this
way we can write the generation script in bash too.

In build, group and sort the packages.  Drop the ones we don't need, or are
pulled in by dependencies.  Include diffutils, findutils and gawk to avoid
warnings in the kernel build.

Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
---
CC: Anthony PERARD <anthony.perard@vates.tech>
CC: Stefano Stabellini <sstabellini@kernel.org>
CC: Michal Orzel <michal.orzel@amd.com>
CC: Doug Goldstein <cardoe@cardoe.com>
CC: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com>
CC: Jason Andryuk <jason.andryuk@amd.com>
CC: Daniel P. Smith <dpsmith@apertussolutions.com>
---
 .gitlab-ci.yml                                |  4 +--
 containerize                                  |  4 +--
 images/alpine/3.18-x86_64-base.dockerfile     |  6 ++++
 ...ockerfile => 3.18-x86_64-build.dockerfile} | 32 ++++++++++++-------
 images/alpine/x86_64-rootfs.dockerfile        |  4 ---
 5 files changed, 30 insertions(+), 20 deletions(-)
 create mode 100644 images/alpine/3.18-x86_64-base.dockerfile
 rename images/alpine/{x86_64-build.dockerfile => 3.18-x86_64-build.dockerfile} (54%)
 delete mode 100644 images/alpine/x86_64-rootfs.dockerfile

diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index 670f9621e26d..fb997cc62162 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -28,7 +28,7 @@ stages:
   tags:
     - x86_64
   variables:
-    CONTAINER: alpine:x86_64-build
+    CONTAINER: alpine:3.18-x86_64-build
 
 #
 # ARM64 artifacts
@@ -62,7 +62,7 @@ x86_64-rootfs-alpine-3.18:
   script:
     - . scripts/x86_64-rootfs-alpine.sh
   variables:
-    CONTAINER: alpine:x86_64-rootfs
+    CONTAINER: alpine:3.18-x86_64-base
 
 x86_64-argo-linux-6.6.56:
   extends: .x86_64-artifacts
diff --git a/containerize b/containerize
index 7abeb119ff6a..c23f55ead737 100755
--- a/containerize
+++ b/containerize
@@ -25,8 +25,8 @@ die() {
 BASE="registry.gitlab.com/xen-project/hardware/test-artifacts"
 case "_${CONTAINER}" in
     _alpine-3.18-arm64-build) CONTAINER="${BASE}/alpine:3.18-arm64-build" ;;
-    _alpine-x86_64-rootfs) CONTAINER="${BASE}/alpine:x86_64-rootfs" ;;
-    _alpine-x86_64-build|_) CONTAINER="${BASE}/alpine:x86_64-build" ;;
+    _alpine-3.18-x86_64-base) CONTAINER="${BASE}/alpine:3.18-x86_64-base" ;;
+    _alpine-3.18-x86_64-build|_) CONTAINER="${BASE}/alpine:3.18-x86_64-build" ;;
 esac
 
 # Use this variable to control whether root should be used
diff --git a/images/alpine/3.18-x86_64-base.dockerfile b/images/alpine/3.18-x86_64-base.dockerfile
new file mode 100644
index 000000000000..a8cb274d5821
--- /dev/null
+++ b/images/alpine/3.18-x86_64-base.dockerfile
@@ -0,0 +1,6 @@
+# syntax=docker/dockerfile:1
+FROM --platform=linux/amd64 alpine:3.18
+LABEL maintainer.name="The Xen Project"
+LABEL maintainer.email="xen-devel@lists.xenproject.org"
+
+RUN apk --no-cache add bash
diff --git a/images/alpine/x86_64-build.dockerfile b/images/alpine/3.18-x86_64-build.dockerfile
similarity index 54%
rename from images/alpine/x86_64-build.dockerfile
rename to images/alpine/3.18-x86_64-build.dockerfile
index 98969a5be1da..eac0cda4fed3 100644
--- a/images/alpine/x86_64-build.dockerfile
+++ b/images/alpine/3.18-x86_64-build.dockerfile
@@ -1,31 +1,39 @@
 # syntax=docker/dockerfile:1
 FROM --platform=linux/amd64 alpine:3.18
-LABEL maintainer.name="The Xen Project" \
-      maintainer.email="xen-devel@lists.xenproject.org"
+LABEL maintainer.name="The Xen Project"
+LABEL maintainer.email="xen-devel@lists.xenproject.org"
 
 RUN apk --no-cache add bash
 
-RUN bash -ex <<EOF
+RUN <<EOF
+#!/bin/bash
+      set -eu
+
       adduser -D user --shell /bin/bash
 
-      DEPS=(
-            musl-dev
+      DEPS=(# Base environment
             build-base
-            libc6-compat
-            linux-headers
-            bash
-            git
             curl
-            flex
+            git
+
+            # Linux build deps
             bison
+            diffutils
             elfutils-dev
+            findutils
+            flex
+            gawk
+            linux-headers
+            openssl-dev
+            perl
+
+            # Argo build deps
             autoconf
             automake
             libtool
-            openssl-dev
       )
 
-      apk add --no-cache "\${DEPS[@]}"
+      apk add --no-cache "${DEPS[@]}"
 EOF
 
 USER user
diff --git a/images/alpine/x86_64-rootfs.dockerfile b/images/alpine/x86_64-rootfs.dockerfile
deleted file mode 100644
index b912e9c1600d..000000000000
--- a/images/alpine/x86_64-rootfs.dockerfile
+++ /dev/null
@@ -1,4 +0,0 @@
-# syntax=docker/dockerfile:1
-FROM --platform=linux/amd64 alpine:3.18
-LABEL maintainer.name="The Xen Project" \
-      maintainer.email="xen-devel@lists.xenproject.org"
-- 
2.39.5

Re: [PATCH v2 1/7] Rename and adjust x86 containers

[Marek Marczykowski-Górecki]

[-- Attachment #1: Type: text/plain, Size: 774 bytes --]

On Mon, Apr 14, 2025 at 11:18:37AM +0100, Andrew Cooper wrote:
> Insert the Alpine version as we'll need a newer version in due course, and
> rename 'rootfs' to 'base' as the container really is not a rootfs on it's own.
> 
> In base, add bash.  It's added later by the rootfs generation anyway, but this
> way we can write the generation script in bash too.
> 
> In build, group and sort the packages.  Drop the ones we don't need, or are
> pulled in by dependencies.  Include diffutils, findutils and gawk to avoid
> warnings in the kernel build.
> 
> Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>

Reviewed-by: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com>

-- 
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

[PATCH v2 2/7] Overhaul how Argo is built and packged

[Andrew Cooper]

Right now, the argo artefacts are a pile of files which the test has to turn
back into something which resembles a filesystem.  Furthermore, because we do
not build modules for the main kernel, it is extra important to make sure that
xen-argo.ko doesn't get out of sync.

Build argo conditionally as part of the linux artefact.  It's ~100kb all
together, compared to ~14M for the kernel.

Produce a single argo.cpio.gz with xen-argo.ko in the standard location.
Prune userspace down to just the executables and libraries.

This is cribbed from the existing scripts/x86_64-linux-argo.sh, which stays in
place in the short term until Xen can be updated to use the new scheme.

Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
---
CC: Anthony PERARD <anthony.perard@vates.tech>
CC: Stefano Stabellini <sstabellini@kernel.org>
CC: Michal Orzel <michal.orzel@amd.com>
CC: Doug Goldstein <cardoe@cardoe.com>
CC: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com>
CC: Jason Andryuk <jason.andryuk@amd.com>
CC: Daniel P. Smith <dpsmith@apertussolutions.com>

v2:
 * Only build conditionally.  Argo is bust with Linux 6.12, which is needed
   for new hardware runners.
 * Parallel build of xen-argo.ko and libargo.
 * Use -print0

I tried to make MODPOST work properly, but we don't build enough of it for the
kernel, and I didn't feel like adding an extra 10 mins to the build (all
modules) just to get the metadata right.
---
 .gitlab-ci.yml         |  2 ++
 scripts/build-argo.sh  | 67 ++++++++++++++++++++++++++++++++++++++++++
 scripts/build-linux.sh |  8 ++++-
 3 files changed, 76 insertions(+), 1 deletion(-)
 create mode 100755 scripts/build-argo.sh

diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index fb997cc62162..790a6d9f9896 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -47,6 +47,8 @@ linux-6.6.56-x86_64:
   script: ./scripts/build-linux.sh
   variables:
     LINUX_VERSION: 6.6.56
+    ARGO_SHA: "705a7a8a624b42e13e655d3042059b8a85cdf6a3"
+    ARGOEXEC_SHA: "d900429f6640acc6f68a3d3a4c945d7da60625d8"
 
 #
 # The jobs below here are legacy and being phased out.
diff --git a/scripts/build-argo.sh b/scripts/build-argo.sh
new file mode 100755
index 000000000000..ef7057d847d4
--- /dev/null
+++ b/scripts/build-argo.sh
@@ -0,0 +1,67 @@
+#
+# This is a partial script, sourced by build-linux.sh
+# It has expectations about the environment
+#
+
+cd "${WORKDIR}"
+
+#
+# We're going to collect everything in argo.cpio.gz.  Construct it under
+# $ARGODIR as we go.
+#
+ARGODIR="${WORKDIR}/argo-root"
+
+git clone https://github.com/OpenXT/linux-xen-argo.git --depth=1
+git -C "${WORKDIR}/linux-xen-argo" fetch origin "${ARGO_SHA}"
+git -C "${WORKDIR}/linux-xen-argo" switch --detach FETCH_HEAD
+
+# Build xen-argo.ko against the target kernel, and install it.  Install
+# linux/argo.h too because userspace needs it.
+make -j$(nproc) -C "linux-${LINUX_VERSION}" \
+     M="${WORKDIR}/linux-xen-argo/argo-linux" \
+     KBUILD_MODPOST_WARN=1 \
+     CFLAGS_MODULE="-Wno-error" \
+     modules
+install -D -m644 "${WORKDIR}/linux-xen-argo/argo-linux/xen-argo.ko" \
+     "${ARGODIR}/lib/modules/${LINUX_VERSION}/updates/xen-argo.ko"
+install -D -m644 "${WORKDIR}/linux-xen-argo/argo-linux/include/linux/argo.h" \
+     "${ARGODIR}/usr/include/linux/argo.h"
+
+# Build and install libargo, applying fixes to build in Alpine Linux
+cd "${WORKDIR}/linux-xen-argo/libargo"
+sed -e "s|AM_INIT_AUTOMAKE|AC_CONFIG_AUX_DIR(.)\nAM_INIT_AUTOMAKE|" \
+    -i configure.ac
+sed -e "s/__SOCKADDR_COMMON (sxenargo_)/sa_family_t sxenargo_family/" \
+    -e "s/__SOCKADDR_COMMON_SIZE/(sizeof (unsigned short int))/" \
+    -i src/libargo.h
+
+autoreconf --install
+./configure --prefix=/usr CPPFLAGS="-I${PWD}/../argo-linux/include"
+make -j$(nproc)
+make install DESTDIR="${ARGODIR}"
+
+# Build and install argo-exec, modifying for xilinx argo test
+cd "${WORKDIR}"
+curl -fsSLO \
+    https://raw.githubusercontent.com/OpenXT/xenclient-oe/${ARGOEXEC_SHA}/recipes-openxt/argo-exec/argo-exec/argo-exec.c
+sed -e "/#include <xen\/xen.h>/d" \
+    -e "s|ret = shuffle(s, fds\[0\], fds\[1\]);|ret = shuffle(s, 0, 1);|" \
+    -i argo-exec.c
+
+gcc -I"${ARGODIR}/usr/include" -L"${ARGODIR}/usr/lib/" \
+    argo-exec.c -o "${ARGODIR}/usr/bin/argo-exec" -largo
+
+#
+# Building is now complete.  Strip the devel components and the nointerposer
+# lib, which we don't care to deploy to the test system.
+#
+cd $ARGODIR
+rm -r usr/include usr/lib/pkgconfig
+find usr/lib -name \*nointerposer\* -delete
+find usr/lib \( -name \*.a -o -name \*.so -o -name \*.la \) -delete
+
+# Package everything up
+find . -print0 | cpio -0 -R 0:0 -H newc -o | gzip > "$COPYDIR/argo.cpio.gz"
+
+# Print the contents for the build log
+zcat "${COPYDIR}/argo.cpio.gz" | cpio -tv
diff --git a/scripts/build-linux.sh b/scripts/build-linux.sh
index 652fdba7b9d1..441b8721a490 100755
--- a/scripts/build-linux.sh
+++ b/scripts/build-linux.sh
@@ -8,7 +8,7 @@ fi
 set -ex -o pipefail
 
 WORKDIR="${PWD}"
-COPYDIR="${WORKDIR}/binaries/"
+COPYDIR="${WORKDIR}/binaries"
 UNAME=$(uname -m)
 
 # Build Linux
@@ -45,6 +45,12 @@ case $UNAME in
     x86_64)
         make -j$(nproc) bzImage
         cp arch/x86/boot/bzImage "${COPYDIR}"
+
+        # Build argo if requested
+        if [[ -n "${ARGO_SHA}" ]]; then
+            make modules_prepare
+            . "${WORKDIR}/scripts/build-argo.sh"
+        fi
         ;;
 
     aarch64)
-- 
2.39.5

Re: [PATCH v2 2/7] Overhaul how Argo is built and packged

[Marek Marczykowski-Górecki]

[-- Attachment #1: Type: text/plain, Size: 452 bytes --]

On Mon, Apr 14, 2025 at 11:18:38AM +0100, Andrew Cooper wrote:
> --- a/scripts/build-linux.sh
> +++ b/scripts/build-linux.sh
> @@ -8,7 +8,7 @@ fi
>  set -ex -o pipefail
>  
>  WORKDIR="${PWD}"
> -COPYDIR="${WORKDIR}/binaries/"
> +COPYDIR="${WORKDIR}/binaries"

Is this change intentional? It has worse failure mode if "binaries" dir
wouldn't exist for some reason...

-- 
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [PATCH v2 2/7] Overhaul how Argo is built and packged

[Andrew Cooper]

On 14/04/2025 11:35 am, Marek Marczykowski-Górecki wrote:
> On Mon, Apr 14, 2025 at 11:18:38AM +0100, Andrew Cooper wrote:
>> --- a/scripts/build-linux.sh
>> +++ b/scripts/build-linux.sh
>> @@ -8,7 +8,7 @@ fi
>>  set -ex -o pipefail
>>  
>>  WORKDIR="${PWD}"
>> -COPYDIR="${WORKDIR}/binaries/"
>> +COPYDIR="${WORKDIR}/binaries"
> Is this change intentional? It has worse failure mode if "binaries" dir
> wouldn't exist for some reason...

Yes it is intentional.  It causes problems when we derive new variables
from COPYDIR.

binaries/ always exists.  It's in the base repo.

~Andrew

Re: [PATCH v2 2/7] Overhaul how Argo is built and packged

[Daniel P. Smith]

On 4/14/25 06:18, Andrew Cooper wrote:
> Right now, the argo artefacts are a pile of files which the test has to turn
> back into something which resembles a filesystem.  Furthermore, because we do
> not build modules for the main kernel, it is extra important to make sure that
> xen-argo.ko doesn't get out of sync.
> 
> Build argo conditionally as part of the linux artefact.  It's ~100kb all
> together, compared to ~14M for the kernel.
> 
> Produce a single argo.cpio.gz with xen-argo.ko in the standard location.
> Prune userspace down to just the executables and libraries.
> 
> This is cribbed from the existing scripts/x86_64-linux-argo.sh, which stays in
> place in the short term until Xen can be updated to use the new scheme.
> 
> Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>

Reviewed-by: Daniel P. Smith <dpsmith@apertussolutions.com>

Re: [PATCH v2 2/7] Overhaul how Argo is built and packged

[Stefano Stabellini]

On Mon, 14 Apr 2025, Andrew Cooper wrote:
> Right now, the argo artefacts are a pile of files which the test has to turn
> back into something which resembles a filesystem.  Furthermore, because we do
> not build modules for the main kernel, it is extra important to make sure that
> xen-argo.ko doesn't get out of sync.
> 
> Build argo conditionally as part of the linux artefact.  It's ~100kb all
> together, compared to ~14M for the kernel.
> 
> Produce a single argo.cpio.gz with xen-argo.ko in the standard location.
> Prune userspace down to just the executables and libraries.
> 
> This is cribbed from the existing scripts/x86_64-linux-argo.sh, which stays in
> place in the short term until Xen can be updated to use the new scheme.
> 
> Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>

Do we need to remove x86_64-argo-linux.sh?

Re: [PATCH v2 2/7] Overhaul how Argo is built and packged

[Andrew Cooper]

On 16/04/2025 12:48 am, Stefano Stabellini wrote:
> On Mon, 14 Apr 2025, Andrew Cooper wrote:
>> Right now, the argo artefacts are a pile of files which the test has to turn
>> back into something which resembles a filesystem.  Furthermore, because we do
>> not build modules for the main kernel, it is extra important to make sure that
>> xen-argo.ko doesn't get out of sync.
>>
>> Build argo conditionally as part of the linux artefact.  It's ~100kb all
>> together, compared to ~14M for the kernel.
>>
>> Produce a single argo.cpio.gz with xen-argo.ko in the standard location.
>> Prune userspace down to just the executables and libraries.
>>
>> This is cribbed from the existing scripts/x86_64-linux-argo.sh, which stays in
>> place in the short term until Xen can be updated to use the new scheme.
>>
>> Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
> Do we need to remove x86_64-argo-linux.sh?

Yes, but 4.20 still uses it.

One thing I'm not sure about is whether "keep latest artefact" is just
artefacts from the latest run, or the latest of each named artefact.

Now is as good a time to experiment as any.

~Andrew

Re: [PATCH v2 2/7] Overhaul how Argo is built and packged

[Marek Marczykowski-Górecki]

[-- Attachment #1: Type: text/plain, Size: 1486 bytes --]

On Wed, Apr 16, 2025 at 11:15:21AM +0100, Andrew Cooper wrote:
> On 16/04/2025 12:48 am, Stefano Stabellini wrote:
> > On Mon, 14 Apr 2025, Andrew Cooper wrote:
> >> Right now, the argo artefacts are a pile of files which the test has to turn
> >> back into something which resembles a filesystem.  Furthermore, because we do
> >> not build modules for the main kernel, it is extra important to make sure that
> >> xen-argo.ko doesn't get out of sync.
> >>
> >> Build argo conditionally as part of the linux artefact.  It's ~100kb all
> >> together, compared to ~14M for the kernel.
> >>
> >> Produce a single argo.cpio.gz with xen-argo.ko in the standard location.
> >> Prune userspace down to just the executables and libraries.
> >>
> >> This is cribbed from the existing scripts/x86_64-linux-argo.sh, which stays in
> >> place in the short term until Xen can be updated to use the new scheme.
> >>
> >> Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
> > Do we need to remove x86_64-argo-linux.sh?
> 
> Yes, but 4.20 still uses it.
> 
> One thing I'm not sure about is whether "keep latest artefact" is just
> artefacts from the latest run, or the latest of each named artefact.

I think it's latest run for a branch. If newer job (on a branch) doesn't
include an artifact anymore, it will be gone. Jobs referencing it won't
see it immediately, regardless of expire time.

-- 
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

[PATCH v2 3/7] Rework rootfs generation to make a cpio archive

[Andrew Cooper]

Rename the script as we're going to use it for ARM64 shortly, and have it take
a tar or cpio parameter to determine the output format.

Turn it into a proper bash script, and provide the cpio form under the new
artefact naming scheme.

No functional change.

Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
Reviewed-by: Denis Mukhin <dmukhin@ford.com>
---
CC: Anthony PERARD <anthony.perard@vates.tech>
CC: Stefano Stabellini <sstabellini@kernel.org>
CC: Michal Orzel <michal.orzel@amd.com>
CC: Doug Goldstein <cardoe@cardoe.com>
CC: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com>

v2:
 * Use -print0
 * Don't dedup $PATHS.  It's going to change between cpio and tar in
   subsequent patches.
---
 .gitlab-ci.yml                                |  9 +++++++-
 ...6_64-rootfs-alpine.sh => alpine-rootfs.sh} | 21 +++++++++++++++++--
 2 files changed, 27 insertions(+), 3 deletions(-)
 rename scripts/{x86_64-rootfs-alpine.sh => alpine-rootfs.sh} (71%)

diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index 790a6d9f9896..b7d187168df2 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -42,6 +42,13 @@ linux-6.6.86-arm64:
 #
 # x86_64 artifacts
 #
+alpine-3.18-x86_64-rootfs:
+  extends: .x86_64-artifacts
+  script:
+    - ./scripts/alpine-rootfs.sh cpio
+  variables:
+    CONTAINER: alpine:3.18-x86_64-base
+
 linux-6.6.56-x86_64:
   extends: .x86_64-artifacts
   script: ./scripts/build-linux.sh
@@ -62,7 +69,7 @@ x86_64-kernel-linux-6.6.56:
 x86_64-rootfs-alpine-3.18:
   extends: .x86_64-artifacts
   script:
-    - . scripts/x86_64-rootfs-alpine.sh
+    - ./scripts/alpine-rootfs.sh tar
   variables:
     CONTAINER: alpine:3.18-x86_64-base
 
diff --git a/scripts/x86_64-rootfs-alpine.sh b/scripts/alpine-rootfs.sh
similarity index 71%
rename from scripts/x86_64-rootfs-alpine.sh
rename to scripts/alpine-rootfs.sh
index b70b3a54ede5..75e2f8648ce5 100755
--- a/scripts/x86_64-rootfs-alpine.sh
+++ b/scripts/alpine-rootfs.sh
@@ -1,4 +1,9 @@
+#!/bin/bash
+
+set -eu
+
 WORKDIR="${PWD}"
+COPYDIR="${WORKDIR}/binaries"
 
 apk update
 
@@ -56,5 +61,17 @@ passwd -d "root" root
 
 # Create rootfs
 cd /
-tar cvzf "${WORKDIR}/binaries/initrd.tar.gz" \
-    bin dev etc home init lib mnt opt root sbin usr var
+case $1 in
+    cpio)
+        PATHS="bin dev etc home init lib mnt opt root sbin usr var"
+        find $PATHS -print0 | cpio -0 -H newc -o | gzip > "${COPYDIR}/rootfs.cpio.gz"
+
+        # Print the contents for the build log
+        zcat "${COPYDIR}/rootfs.cpio.gz" | cpio -tv
+        ;;
+
+    tar)
+        PATHS="bin dev etc home init lib mnt opt root sbin usr var"
+        tar cvzf "${COPYDIR}/initrd.tar.gz" $PATHS
+        ;;
+esac
-- 
2.39.5

Re: [PATCH v2 3/7] Rework rootfs generation to make a cpio archive

[Marek Marczykowski-Górecki]

[-- Attachment #1: Type: text/plain, Size: 606 bytes --]

On Mon, Apr 14, 2025 at 11:18:39AM +0100, Andrew Cooper wrote:
> Rename the script as we're going to use it for ARM64 shortly, and have it take
> a tar or cpio parameter to determine the output format.
> 
> Turn it into a proper bash script, and provide the cpio form under the new
> artefact naming scheme.
> 
> No functional change.
> 
> Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
> Reviewed-by: Denis Mukhin <dmukhin@ford.com>

Reviewed-by: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com>

-- 
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

[PATCH v2 4/7] Shrink the rootfs substantially

[Andrew Cooper]

bash, busybox, musl and zlib are all in the base container.

python3 and ncurses are in principle used by bits of Xen, but not in anything
we test in CI.  argp-standlone, curl, dbus, libfdt, libgcc and sudo aren't
used at all (for x86 at least).

libbz2 and libuuid were pulled in transitively before, and need to be included
explicitly now.

Use apk --no-cache to avoid keeping a ~2M package index on disk.

Remove the modules scan on boot.  We don't have or build any.  This removes a
chunk of warnings on boot.

This shrinks the rootfs from ~30M down to ~8M.

No practical change.

Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
---
CC: Anthony PERARD <anthony.perard@vates.tech>
CC: Stefano Stabellini <sstabellini@kernel.org>
CC: Michal Orzel <michal.orzel@amd.com>
CC: Doug Goldstein <cardoe@cardoe.com>
CC: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com>
---
 scripts/alpine-rootfs.sh | 60 +++++++++++++++++++++++-----------------
 1 file changed, 34 insertions(+), 26 deletions(-)

diff --git a/scripts/alpine-rootfs.sh b/scripts/alpine-rootfs.sh
index 75e2f8648ce5..72c29e0a0a13 100755
--- a/scripts/alpine-rootfs.sh
+++ b/scripts/alpine-rootfs.sh
@@ -4,33 +4,42 @@ set -eu
 
 WORKDIR="${PWD}"
 COPYDIR="${WORKDIR}/binaries"
+UNAME=$(uname -m)
 
-apk update
+apk --no-cache update
 
-# xen runtime deps
-apk add musl
-apk add libgcc
-apk add openrc
-apk add busybox
-apk add sudo
-apk add dbus
-apk add bash
-apk add python3
-apk add zlib
-apk add lzo
-apk add ncurses
-apk add yajl
-apk add libaio
-apk add xz
-apk add util-linux
-apk add argp-standalone
-apk add libfdt
-apk add glib
-apk add pixman
-apk add curl
-apk add udev
-apk add pciutils
-apk add libelf
+PKGS=(
+    # System
+    openrc
+    udev
+    util-linux
+
+    # Xen toolstack runtime deps
+    libbz2
+    libuuid
+    lzo
+    xz
+    yajl
+
+    # QEMU
+    glib
+    libaio
+    pixman
+    )
+
+case $UNAME in
+    x86_64)
+        PKGS+=(
+            # System
+            pciutils
+
+            # QEMU
+            libelf
+            )
+        ;;
+esac
+
+apk add --no-cache "${PKGS[@]}"
 
 # Xen
 cd /
@@ -45,7 +54,6 @@ rc-update add dmesg sysinit
 rc-update add hostname boot
 rc-update add hwclock boot
 rc-update add hwdrivers sysinit
-rc-update add modules boot
 rc-update add killprocs shutdown
 rc-update add mount-ro shutdown
 rc-update add savecache shutdown
-- 
2.39.5

Re: [PATCH v2 4/7] Shrink the rootfs substantially

[Marek Marczykowski-Górecki]

[-- Attachment #1: Type: text/plain, Size: 3256 bytes --]

On Mon, Apr 14, 2025 at 11:18:40AM +0100, Andrew Cooper wrote:
> bash, busybox, musl and zlib are all in the base container.
> 
> python3 and ncurses are in principle used by bits of Xen, but not in anything
> we test in CI.  argp-standlone, curl, dbus, libfdt, libgcc and sudo aren't
> used at all (for x86 at least).
> 
> libbz2 and libuuid were pulled in transitively before, and need to be included
> explicitly now.
> 
> Use apk --no-cache to avoid keeping a ~2M package index on disk.
> 
> Remove the modules scan on boot.  We don't have or build any.  This removes a
> chunk of warnings on boot.

Strictly speaking there is xen-argo.ko, but that's handled manually
anyway, so probably not worth mentioning.

> This shrinks the rootfs from ~30M down to ~8M.
> 
> No practical change.

This also adds some preparation (the case on `uname -m`) for ARM64
rootfs, would be nice to mention it too. Especially since libfdt removed
here will be re-added for ARM.

> Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
> ---
> CC: Anthony PERARD <anthony.perard@vates.tech>
> CC: Stefano Stabellini <sstabellini@kernel.org>
> CC: Michal Orzel <michal.orzel@amd.com>
> CC: Doug Goldstein <cardoe@cardoe.com>
> CC: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com>
> ---
>  scripts/alpine-rootfs.sh | 60 +++++++++++++++++++++++-----------------
>  1 file changed, 34 insertions(+), 26 deletions(-)
> 
> diff --git a/scripts/alpine-rootfs.sh b/scripts/alpine-rootfs.sh
> index 75e2f8648ce5..72c29e0a0a13 100755
> --- a/scripts/alpine-rootfs.sh
> +++ b/scripts/alpine-rootfs.sh
> @@ -4,33 +4,42 @@ set -eu
>  
>  WORKDIR="${PWD}"
>  COPYDIR="${WORKDIR}/binaries"
> +UNAME=$(uname -m)
>  
> -apk update
> +apk --no-cache update

This is no-op, no? IIUC the only thing `apk update` does is updating the
cache, which you disabled...

> -# xen runtime deps
> -apk add musl
> -apk add libgcc
> -apk add openrc
> -apk add busybox
> -apk add sudo
> -apk add dbus
> -apk add bash
> -apk add python3
> -apk add zlib
> -apk add lzo
> -apk add ncurses
> -apk add yajl
> -apk add libaio
> -apk add xz
> -apk add util-linux
> -apk add argp-standalone
> -apk add libfdt
> -apk add glib
> -apk add pixman
> -apk add curl
> -apk add udev
> -apk add pciutils
> -apk add libelf
> +PKGS=(
> +    # System
> +    openrc
> +    udev
> +    util-linux
> +
> +    # Xen toolstack runtime deps
> +    libbz2
> +    libuuid
> +    lzo
> +    xz
> +    yajl
> +
> +    # QEMU
> +    glib
> +    libaio
> +    pixman
> +    )
> +
> +case $UNAME in
> +    x86_64)
> +        PKGS+=(
> +            # System
> +            pciutils
> +
> +            # QEMU
> +            libelf
> +            )
> +        ;;
> +esac
> +
> +apk add --no-cache "${PKGS[@]}"
>  
>  # Xen
>  cd /
> @@ -45,7 +54,6 @@ rc-update add dmesg sysinit
>  rc-update add hostname boot
>  rc-update add hwclock boot
>  rc-update add hwdrivers sysinit
> -rc-update add modules boot
>  rc-update add killprocs shutdown
>  rc-update add mount-ro shutdown
>  rc-update add savecache shutdown
> -- 
> 2.39.5
> 

-- 
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [PATCH v2 4/7] Shrink the rootfs substantially

[Andrew Cooper]

On 14/04/2025 12:25 pm, Marek Marczykowski-Górecki wrote:
> On Mon, Apr 14, 2025 at 11:18:40AM +0100, Andrew Cooper wrote:
>> bash, busybox, musl and zlib are all in the base container.
>>
>> python3 and ncurses are in principle used by bits of Xen, but not in anything
>> we test in CI.  argp-standlone, curl, dbus, libfdt, libgcc and sudo aren't
>> used at all (for x86 at least).
>>
>> libbz2 and libuuid were pulled in transitively before, and need to be included
>> explicitly now.
>>
>> Use apk --no-cache to avoid keeping a ~2M package index on disk.
>>
>> Remove the modules scan on boot.  We don't have or build any.  This removes a
>> chunk of warnings on boot.
> Strictly speaking there is xen-argo.ko, but that's handled manually
> anyway, so probably not worth mentioning.

Hmm, it's an awkward way around in the series, but yes.  The lack of
doing any of the normal module work in the kernel build is what causes
xen-argo.ko to be more special than I'd like.

>
>> This shrinks the rootfs from ~30M down to ~8M.
>>
>> No practical change.
> This also adds some preparation (the case on `uname -m`) for ARM64
> rootfs, would be nice to mention it too. Especially since libfdt removed
> here will be re-added for ARM.

"Factor out some x86-isms in preparation for ARM64 support."

>> Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
>> ---
>> CC: Anthony PERARD <anthony.perard@vates.tech>
>> CC: Stefano Stabellini <sstabellini@kernel.org>
>> CC: Michal Orzel <michal.orzel@amd.com>
>> CC: Doug Goldstein <cardoe@cardoe.com>
>> CC: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com>
>> ---
>>  scripts/alpine-rootfs.sh | 60 +++++++++++++++++++++++-----------------
>>  1 file changed, 34 insertions(+), 26 deletions(-)
>>
>> diff --git a/scripts/alpine-rootfs.sh b/scripts/alpine-rootfs.sh
>> index 75e2f8648ce5..72c29e0a0a13 100755
>> --- a/scripts/alpine-rootfs.sh
>> +++ b/scripts/alpine-rootfs.sh
>> @@ -4,33 +4,42 @@ set -eu
>>  
>>  WORKDIR="${PWD}"
>>  COPYDIR="${WORKDIR}/binaries"
>> +UNAME=$(uname -m)
>>  
>> -apk update
>> +apk --no-cache update
> This is no-op, no? IIUC the only thing `apk update` does is updating the
> cache, which you disabled...

Lovely.  This ought to be upgrade, so we pull in updates to the packages
in the base image.

~Andrew

[PATCH v2.1 4/7] Shrink the rootfs substantially

[Andrew Cooper]

bash, busybox, musl and zlib are all in the base container.

python3 and ncurses are in principle used by bits of Xen, but not in anything
we test in CI.  argp-standlone, curl, dbus, libfdt, libgcc and sudo aren't
used at all (for x86 at least).

libbz2 and libuuid were pulled in transitively before, and need to be included
explicitly now.

Use apk --no-cache to avoid keeping a ~2M package index on disk.  Use apk
upgrade in case there are changes to the base container.

Remove the modules scan on boot.  We don't have or build any (except argo, and
that's handled specially).  This removes a chunk of warnings on boot.

This shrinks the rootfs from ~30M down to ~8M.

Factor out some x86-isms in preparation for ARM64 support.

No practical change.

Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
---
CC: Anthony PERARD <anthony.perard@vates.tech>
CC: Stefano Stabellini <sstabellini@kernel.org>
CC: Michal Orzel <michal.orzel@amd.com>
CC: Doug Goldstein <cardoe@cardoe.com>
CC: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com>

v2.1:
 * Extend commit message
 * Use apk upgrade

https://gitlab.com/xen-project/hardware/test-artifacts/-/jobs/9713228239
https://gitlab.com/xen-project/hardware/test-artifacts/-/jobs/9713228242
---
 scripts/alpine-rootfs.sh | 60 +++++++++++++++++++++++-----------------
 1 file changed, 34 insertions(+), 26 deletions(-)

diff --git a/scripts/alpine-rootfs.sh b/scripts/alpine-rootfs.sh
index 75e2f8648ce5..b01de9709d02 100755
--- a/scripts/alpine-rootfs.sh
+++ b/scripts/alpine-rootfs.sh
@@ -4,33 +4,42 @@ set -eu
 
 WORKDIR="${PWD}"
 COPYDIR="${WORKDIR}/binaries"
+UNAME=$(uname -m)
 
-apk update
+apk --no-cache upgrade
 
-# xen runtime deps
-apk add musl
-apk add libgcc
-apk add openrc
-apk add busybox
-apk add sudo
-apk add dbus
-apk add bash
-apk add python3
-apk add zlib
-apk add lzo
-apk add ncurses
-apk add yajl
-apk add libaio
-apk add xz
-apk add util-linux
-apk add argp-standalone
-apk add libfdt
-apk add glib
-apk add pixman
-apk add curl
-apk add udev
-apk add pciutils
-apk add libelf
+PKGS=(
+    # System
+    openrc
+    udev
+    util-linux
+
+    # Xen toolstack runtime deps
+    libbz2
+    libuuid
+    lzo
+    xz
+    yajl
+
+    # QEMU
+    glib
+    libaio
+    pixman
+    )
+
+case $UNAME in
+    x86_64)
+        PKGS+=(
+            # System
+            pciutils
+
+            # QEMU
+            libelf
+            )
+        ;;
+esac
+
+apk add --no-cache "${PKGS[@]}"
 
 # Xen
 cd /
@@ -45,7 +54,6 @@ rc-update add dmesg sysinit
 rc-update add hostname boot
 rc-update add hwclock boot
 rc-update add hwdrivers sysinit
-rc-update add modules boot
 rc-update add killprocs shutdown
 rc-update add mount-ro shutdown
 rc-update add savecache shutdown
-- 
2.39.5

Re: [PATCH v2.1 4/7] Shrink the rootfs substantially

[Marek Marczykowski-Górecki]

[-- Attachment #1: Type: text/plain, Size: 1093 bytes --]

On Mon, Apr 14, 2025 at 12:52:26PM +0100, Andrew Cooper wrote:
> bash, busybox, musl and zlib are all in the base container.
> 
> python3 and ncurses are in principle used by bits of Xen, but not in anything
> we test in CI.  argp-standlone, curl, dbus, libfdt, libgcc and sudo aren't
> used at all (for x86 at least).
> 
> libbz2 and libuuid were pulled in transitively before, and need to be included
> explicitly now.
> 
> Use apk --no-cache to avoid keeping a ~2M package index on disk.  Use apk
> upgrade in case there are changes to the base container.
> 
> Remove the modules scan on boot.  We don't have or build any (except argo, and
> that's handled specially).  This removes a chunk of warnings on boot.
> 
> This shrinks the rootfs from ~30M down to ~8M.
> 
> Factor out some x86-isms in preparation for ARM64 support.
> 
> No practical change.
> 
> Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>

Reviewed-by: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com>

-- 
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

[PATCH v2 5/7] Adjust directory layout for rootfs.cpio.gz

[Andrew Cooper]

All tests using the rootfs currently modify it with:

  mkdir proc
  mkdir run
  mkdir srv
  mkdir sys
  rm var/run

This is because the rootfs doesn't have the correct top level directories.
Furthermore, dev is currently included when it oughtn't to be.

srv is easy.  It's an empty directory and can be included in PATHS.

dev, proc, run and sys are all mount points, which OpenRC knows how to
populate on boot.  Create only the top level directories in rootfs, without
`find` recursing into them.

Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
---
CC: Anthony PERARD <anthony.perard@vates.tech>
CC: Stefano Stabellini <sstabellini@kernel.org>
CC: Michal Orzel <michal.orzel@amd.com>
CC: Doug Goldstein <cardoe@cardoe.com>
CC: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com>

v2:
 * New
---
 scripts/alpine-rootfs.sh | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)

diff --git a/scripts/alpine-rootfs.sh b/scripts/alpine-rootfs.sh
index 72c29e0a0a13..53547b5c5282 100755
--- a/scripts/alpine-rootfs.sh
+++ b/scripts/alpine-rootfs.sh
@@ -71,8 +71,11 @@ passwd -d "root" root
 cd /
 case $1 in
     cpio)
-        PATHS="bin dev etc home init lib mnt opt root sbin usr var"
-        find $PATHS -print0 | cpio -0 -H newc -o | gzip > "${COPYDIR}/rootfs.cpio.gz"
+        {
+            PATHS="bin etc home init lib mnt opt root sbin srv usr var"
+            find $PATHS -print0
+            echo -ne "dev\0proc\0run\0sys\0"
+        } | cpio -0 -H newc -o | gzip > "${COPYDIR}/rootfs.cpio.gz"
 
         # Print the contents for the build log
         zcat "${COPYDIR}/rootfs.cpio.gz" | cpio -tv
-- 
2.39.5

Re: [PATCH v2 5/7] Adjust directory layout for rootfs.cpio.gz

[Marek Marczykowski-Górecki]

[-- Attachment #1: Type: text/plain, Size: 827 bytes --]

On Mon, Apr 14, 2025 at 11:18:41AM +0100, Andrew Cooper wrote:
> All tests using the rootfs currently modify it with:
> 
>   mkdir proc
>   mkdir run
>   mkdir srv
>   mkdir sys
>   rm var/run
> 
> This is because the rootfs doesn't have the correct top level directories.
> Furthermore, dev is currently included when it oughtn't to be.
> 
> srv is easy.  It's an empty directory and can be included in PATHS.
> 
> dev, proc, run and sys are all mount points, which OpenRC knows how to
> populate on boot.  Create only the top level directories in rootfs, without
> `find` recursing into them.
> 
> Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>

Reviewed-by: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com>

-- 
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

[PATCH v2 6/7] Provide an ARM64 rootfs too

[Andrew Cooper]

The only extra package needed is libfdt.

The older ARM64 rootfs configured modloop, networking and sysctl, but none of
this is used in any of the testing, so is omitted here for now.

Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
---
CC: Anthony PERARD <anthony.perard@vates.tech>
CC: Stefano Stabellini <sstabellini@kernel.org>
CC: Michal Orzel <michal.orzel@amd.com>
CC: Doug Goldstein <cardoe@cardoe.com>
CC: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com>
---
 .gitlab-ci.yml                           | 7 +++++++
 containerize                             | 1 +
 images/alpine/3.18-arm64-base.dockerfile | 6 ++++++
 scripts/alpine-rootfs.sh                 | 7 +++++++
 4 files changed, 21 insertions(+)
 create mode 100644 images/alpine/3.18-arm64-base.dockerfile

diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index b7d187168df2..d70ddd99e529 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -33,6 +33,13 @@ stages:
 #
 # ARM64 artifacts
 #
+alpine-3.18-arm64-rootfs:
+  extends: .arm64-artifacts
+  script:
+    - ./scripts/alpine-rootfs.sh cpio
+  variables:
+    CONTAINER: alpine:3.18-arm64-base
+
 linux-6.6.86-arm64:
   extends: .arm64-artifacts
   script: ./scripts/build-linux.sh
diff --git a/containerize b/containerize
index c23f55ead737..38a434ab7075 100755
--- a/containerize
+++ b/containerize
@@ -24,6 +24,7 @@ die() {
 #
 BASE="registry.gitlab.com/xen-project/hardware/test-artifacts"
 case "_${CONTAINER}" in
+    _alpine-3.18-arm64-base) CONTAINER="${BASE}/alpine:3.18-arm64-base" ;;
     _alpine-3.18-arm64-build) CONTAINER="${BASE}/alpine:3.18-arm64-build" ;;
     _alpine-3.18-x86_64-base) CONTAINER="${BASE}/alpine:3.18-x86_64-base" ;;
     _alpine-3.18-x86_64-build|_) CONTAINER="${BASE}/alpine:3.18-x86_64-build" ;;
diff --git a/images/alpine/3.18-arm64-base.dockerfile b/images/alpine/3.18-arm64-base.dockerfile
new file mode 100644
index 000000000000..ab597c75b80e
--- /dev/null
+++ b/images/alpine/3.18-arm64-base.dockerfile
@@ -0,0 +1,6 @@
+# syntax=docker/dockerfile:1
+FROM --platform=linux/arm64/v8 alpine:3.18
+LABEL maintainer.name="The Xen Project"
+LABEL maintainer.email="xen-devel@lists.xenproject.org"
+
+RUN apk --no-cache add bash
diff --git a/scripts/alpine-rootfs.sh b/scripts/alpine-rootfs.sh
index 53547b5c5282..e52908a3d588 100755
--- a/scripts/alpine-rootfs.sh
+++ b/scripts/alpine-rootfs.sh
@@ -37,6 +37,13 @@ case $UNAME in
             libelf
             )
         ;;
+
+    aarch64)
+        PKGS+=(
+            # Xen
+            libfdt
+            )
+        ;;
 esac
 
 apk add --no-cache "${PKGS[@]}"
-- 
2.39.5

Re: [PATCH v2 6/7] Provide an ARM64 rootfs too

[Marek Marczykowski-Górecki]

[-- Attachment #1: Type: text/plain, Size: 475 bytes --]

On Mon, Apr 14, 2025 at 11:18:42AM +0100, Andrew Cooper wrote:
> The only extra package needed is libfdt.
> 
> The older ARM64 rootfs configured modloop, networking and sysctl, but none of
> this is used in any of the testing, so is omitted here for now.
> 
> Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>

Reviewed-by: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com>

-- 
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

[PATCH v2 7/7] Package microcode for the x86 hardware runners

[Andrew Cooper]

They are all out of date, to different degrees.

Install jq into the x86_64 build container so we can parse the Github latest
release information in an acceptable way.

The resulting archive must be uncompressed, in order to work during early
boot.

Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
Reviewed-by: Jason Andryuk <jason.andryuk@amd.com>
---
CC: Anthony PERARD <anthony.perard@vates.tech>
CC: Stefano Stabellini <sstabellini@kernel.org>
CC: Michal Orzel <michal.orzel@amd.com>
CC: Doug Goldstein <cardoe@cardoe.com>
CC: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com>
---
 .gitlab-ci.yml                             |  4 +++
 images/alpine/3.18-x86_64-build.dockerfile |  3 ++
 scripts/x86-microcode.sh                   | 42 ++++++++++++++++++++++
 3 files changed, 49 insertions(+)
 create mode 100755 scripts/x86-microcode.sh

diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index d70ddd99e529..74335363d5ed 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -64,6 +64,10 @@ linux-6.6.56-x86_64:
     ARGO_SHA: "705a7a8a624b42e13e655d3042059b8a85cdf6a3"
     ARGOEXEC_SHA: "d900429f6640acc6f68a3d3a4c945d7da60625d8"
 
+microcode-x86:
+  extends: .x86_64-artifacts
+  script: ./scripts/x86-microcode.sh
+
 #
 # The jobs below here are legacy and being phased out.
 #
diff --git a/images/alpine/3.18-x86_64-build.dockerfile b/images/alpine/3.18-x86_64-build.dockerfile
index eac0cda4fed3..c4ff30e1f138 100644
--- a/images/alpine/3.18-x86_64-build.dockerfile
+++ b/images/alpine/3.18-x86_64-build.dockerfile
@@ -27,6 +27,9 @@ RUN <<EOF
             openssl-dev
             perl
 
+            # Microcode
+            jq
+
             # Argo build deps
             autoconf
             automake
diff --git a/scripts/x86-microcode.sh b/scripts/x86-microcode.sh
new file mode 100755
index 000000000000..cb55a3bd2d52
--- /dev/null
+++ b/scripts/x86-microcode.sh
@@ -0,0 +1,42 @@
+#!/bin/bash
+
+set -eux -o pipefail
+
+WORKDIR="${PWD}"
+COPYDIR="${WORKDIR}/binaries"
+
+ROOTDIR="${WORKDIR}/root"
+UCODEDIR="${ROOTDIR}/kernel/x86/microcode"
+mkdir -p "${UCODEDIR}"
+
+#
+# Intel microcode comes from github
+#
+curl -fsSL https://api.github.com/repos/intel/Intel-Linux-Processor-Microcode-Data-Files/releases/latest > intel-latest.json
+TARBALL_URL="$(jq -r .tarball_url intel-latest.json)"
+curl -fsSL "${TARBALL_URL}" > intel-latest.tar
+tar xf intel-latest.tar --strip-components=1
+
+(
+    cd intel-ucode
+    cat 06-97-02 # adl-*
+    cat 06-8e-09 # kbl-*
+) > "${UCODEDIR}/GenuineIntel.bin"
+
+#
+# AMD microcode comes from linux-firmware
+#
+curl -fsSLO https://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git/plain/amd-ucode/microcode_amd_fam17h.bin
+curl -fsSLO https://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git/plain/amd-ucode/microcode_amd_fam19h.bin
+
+(
+    cat microcode_amd_fam17h.bin # zen2-*, xilinux-*-x86_64-*
+    cat microcode_amd_fam19h.bin # zen3p-*
+) > "${UCODEDIR}/AuthenticAMD.bin"
+
+# Package everything up.  It must be uncompressed
+cd "${ROOTDIR}"
+find . | cpio -R 0:0 -H newc -o > "${COPYDIR}/ucode.cpio"
+
+# Print the contents for the build log
+cpio -tv < "${COPYDIR}/ucode.cpio"
-- 
2.39.5