[PATCH test-artifacts v1 0/5] Changes for building arbitrary Linux branch and for hw12 runner

[PATCH test-artifacts v1 0/5] Changes for building arbitrary Linux branch and for hw12 runner

[Marek Marczykowski-Górecki]

This series is necessary for the other one sent in a moment.

Technically, the last patch isn't strictly required, but it eases debugging.

Green pipeline: https://gitlab.com/xen-project/people/marmarek/test-artifacts/-/pipelines/1881057399

Marek Marczykowski-Górecki (5):
  Add linux-6.12.34-x86_64
  Enable CONFIG_USB_RTL8152 in kernel for hw12 runner
  Include git in the ARM64 build container too
  Support building arbitrary Linux branch/tag/commit
  Setup ssh access to test systems

 .gitlab-ci.yml                            | 28 ++++++++++++++++++++++++-
 images/alpine/3.18-arm64-build.dockerfile |  1 +-
 scripts/alpine-rootfs.sh                  |  4 +++-
 scripts/build-linux.sh                    | 19 +++++++++++-----
 4 files changed, 47 insertions(+), 5 deletions(-)

base-commit: d7434697deec41ddf31a2f3d189dee75d4d2486f
-- 
git-series 0.9.1

[PATCH test-artifacts v1 1/5] Add linux-6.12.34-x86_64

[Marek Marczykowski-Górecki]

This is necessary for new Zen4 runner.
Do not include Argo module in this build, as it isn't compatible with
6.12 yet.

Signed-off-by: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com>
---
 .gitlab-ci.yml | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index c698612..ad44fb4 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -64,6 +64,12 @@ linux-6.6.56-x86_64:
     ARGO_SHA: "cf73819cacc945baca1a7421e5836d1bd481739b"
     ARGOEXEC_SHA: "d900429f6640acc6f68a3d3a4c945d7da60625d8"
 
+linux-6.12.34-x86_64:
+  extends: .x86_64-artifacts
+  script: ./scripts/build-linux.sh
+  variables:
+    LINUX_VERSION: 6.12.34
+
 microcode-x86:
   extends: .x86_64-artifacts
   script: ./scripts/x86-microcode.sh
-- 
git-series 0.9.1

Re: [PATCH test-artifacts v1 1/5] Add linux-6.12.34-x86_64

[Andrew Cooper]

On 23/06/2025 2:46 pm, Marek Marczykowski-Górecki wrote:
> This is necessary for new Zen4 runner.
> Do not include Argo module in this build, as it isn't compatible with
> 6.12 yet.
>
> Signed-off-by: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com>

Acked-by: Andrew Cooper <andrew.cooper3@citrix.com>

Re: [PATCH test-artifacts v1 1/5] Add linux-6.12.34-x86_64

[Stefano Stabellini]

[-- Attachment #1: Type: text/plain, Size: 429 bytes --]

On Mon, 23 Jun 2025, Andrew Cooper wrote:
> On 23/06/2025 2:46 pm, Marek Marczykowski-Górecki wrote:
> > This is necessary for new Zen4 runner.
> > Do not include Argo module in this build, as it isn't compatible with
> > 6.12 yet.
> >
> > Signed-off-by: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com>
> 
> Acked-by: Andrew Cooper <andrew.cooper3@citrix.com>

Acked-by: Stefano Stabellini <sstabellini@kernel.org>

[PATCH test-artifacts v1 2/5] Enable CONFIG_USB_RTL8152 in kernel for hw12 runner

[Marek Marczykowski-Górecki]

It uses this USB network interface.

Signed-off-by: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com>
---
This was posted before at https://lore.kernel.org/xen-devel/20250411203336.585215-1-marmarek@invisiblethingslab.com/
---
 scripts/build-linux.sh | 1 +
 1 file changed, 1 insertion(+)

diff --git a/scripts/build-linux.sh b/scripts/build-linux.sh
index 441b872..cf0e744 100755
--- a/scripts/build-linux.sh
+++ b/scripts/build-linux.sh
@@ -32,6 +32,7 @@ case $UNAME in
             | grep 'XEN' \
             | grep '=m' \
             | sed 's/=m/=y/g' >> .config
+        ./scripts/config --enable USB_RTL8152
         ;;
 
     aarch64)
-- 
git-series 0.9.1

Re: [PATCH test-artifacts v1 2/5] Enable CONFIG_USB_RTL8152 in kernel for hw12 runner

[Andrew Cooper]

On 23/06/2025 2:46 pm, Marek Marczykowski-Górecki wrote:
> It uses this USB network interface.
>
> Signed-off-by: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com>

Acked-by: Andrew Cooper <andrew.cooper3@citrix.com>

Re: [PATCH test-artifacts v1 2/5] Enable CONFIG_USB_RTL8152 in kernel for hw12 runner

[Stefano Stabellini]

[-- Attachment #1: Type: text/plain, Size: 338 bytes --]

On Mon, 23 Jun 2025, Andrew Cooper wrote:
> On 23/06/2025 2:46 pm, Marek Marczykowski-Górecki wrote:
> > It uses this USB network interface.
> >
> > Signed-off-by: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com>
> 
> Acked-by: Andrew Cooper <andrew.cooper3@citrix.com>

Acked-by: Stefano Stabellini <sstabellini@kernel.org>

[PATCH test-artifacts v1 3/5] Include git in the ARM64 build container too

[Marek Marczykowski-Górecki]

It will be used for fetching some Linux versions.

Signed-off-by: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com>
---
 images/alpine/3.18-arm64-build.dockerfile | 1 +
 1 file changed, 1 insertion(+)

diff --git a/images/alpine/3.18-arm64-build.dockerfile b/images/alpine/3.18-arm64-build.dockerfile
index 25a8dbd..38464c9 100644
--- a/images/alpine/3.18-arm64-build.dockerfile
+++ b/images/alpine/3.18-arm64-build.dockerfile
@@ -14,6 +14,7 @@ RUN <<EOF
       DEPS=(# Base environment
             build-base
             curl
+            git
 
             # Linux build deps
             bison
-- 
git-series 0.9.1

Re: [PATCH test-artifacts v1 3/5] Include git in the ARM64 build container too

[Andrew Cooper]

On 23/06/2025 2:46 pm, Marek Marczykowski-Górecki wrote:
> It will be used for fetching some Linux versions.
>
> Signed-off-by: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com>

Acked-by: Andrew Cooper <andrew.cooper3@citrix.com>

Re: [PATCH test-artifacts v1 3/5] Include git in the ARM64 build container too

[Stefano Stabellini]

[-- Attachment #1: Type: text/plain, Size: 352 bytes --]

On Mon, 23 Jun 2025, Andrew Cooper wrote:
> On 23/06/2025 2:46 pm, Marek Marczykowski-Górecki wrote:
> > It will be used for fetching some Linux versions.
> >
> > Signed-off-by: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com>
> 
> Acked-by: Andrew Cooper <andrew.cooper3@citrix.com>

Acked-by: Stefano Stabellini <sstabellini@kernel.org>

[PATCH test-artifacts v1 4/5] Support building arbitrary Linux branch/tag/commit

[Marek Marczykowski-Górecki]

If LINUX_URL is set, fetch LINUX_VERSION from there. Go with "git
init" + "git fetch" instead of "git clone" to support any of
branch/tag/commit.

This also defines optional linux-git-* jobs which will build the thing
if LINUX_GIT_VERSION and LINUX_GIT_URL variables are provided for the
pipeline.

Signed-off-by: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com>
---
The script variable and job variable need to have different names, so a
pipeline variable won't override it for all jobs. While LINUX_VERSION /
LINUX_GIT_VERSION is IMO okay, I'm not very happy about LINUX_URL /
LINUX_GIT_URL. Any better ideas?
---
 .gitlab-ci.yml         | 22 ++++++++++++++++++++++
 scripts/build-linux.sh | 18 +++++++++++++-----
 2 files changed, 35 insertions(+), 5 deletions(-)

diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index ad44fb4..60af072 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -1,5 +1,9 @@
 variables:
   REGISTRY: registry.gitlab.com/xen-project/hardware/test-artifacts
+  LINUX_GIT_VERSION:
+    description: "branch/tag/commit for the linux-git jobs"
+  LINUX_GIT_URL:
+    description: "git url for the linux-git jobs"
 
 stages:
   - build
@@ -46,6 +50,15 @@ linux-6.6.86-arm64:
   variables:
     LINUX_VERSION: 6.6.86
 
+linux-git-arm64:
+  extends: .arm64-artifacts
+  script: ./scripts/build-linux.sh
+  variables:
+    LINUX_VERSION: $LINUX_GIT_VERSION
+    LINUX_URL: $LINUX_GIT_URL
+  rules:
+  - if: $LINUX_GIT_VERSION && $LINUX_GIT_URL
+
 #
 # x86_64 artifacts
 #
@@ -70,6 +83,15 @@ linux-6.12.34-x86_64:
   variables:
     LINUX_VERSION: 6.12.34
 
+linux-git-x86_64:
+  extends: .x86_64-artifacts
+  script: ./scripts/build-linux.sh
+  variables:
+    LINUX_VERSION: $LINUX_GIT_VERSION
+    LINUX_URL: $LINUX_GIT_URL
+  rules:
+  - if: $LINUX_GIT_VERSION && $LINUX_GIT_URL
+
 microcode-x86:
   extends: .x86_64-artifacts
   script: ./scripts/x86-microcode.sh
diff --git a/scripts/build-linux.sh b/scripts/build-linux.sh
index cf0e744..1fc96d1 100755
--- a/scripts/build-linux.sh
+++ b/scripts/build-linux.sh
@@ -12,11 +12,19 @@ COPYDIR="${WORKDIR}/binaries"
 UNAME=$(uname -m)
 
 # Build Linux
-MAJOR=${LINUX_VERSION%%.*}
-curl -fsSLO \
-    https://cdn.kernel.org/pub/linux/kernel/v"${MAJOR}".x/linux-"${LINUX_VERSION}".tar.xz
-tar xf linux-"${LINUX_VERSION}".tar.xz
-cd linux-"${LINUX_VERSION}"
+if [[ -n "${LINUX_URL}" ]]; then
+    mkdir linux
+    cd linux
+    git init
+    git fetch --depth=1 "${LINUX_URL}" "${LINUX_VERSION}"
+    git checkout FETCH_HEAD
+else
+    MAJOR=${LINUX_VERSION%%.*}
+    curl -fsSLO \
+        https://cdn.kernel.org/pub/linux/kernel/v"${MAJOR}".x/linux-"${LINUX_VERSION}".tar.xz
+    tar xf linux-"${LINUX_VERSION}".tar.xz
+    cd linux-"${LINUX_VERSION}"
+fi
 
 make defconfig
 ./scripts/config --enable BRIDGE
-- 
git-series 0.9.1

Re: [PATCH test-artifacts v1 4/5] Support building arbitrary Linux branch/tag/commit

[Stefano Stabellini]

[-- Attachment #1: Type: text/plain, Size: 3722 bytes --]

On Mon, 23 Jun 2025, Marek Marczykowski-Górecki wrote:
> If LINUX_URL is set, fetch LINUX_VERSION from there. Go with "git
> init" + "git fetch" instead of "git clone" to support any of
> branch/tag/commit.
> 
> This also defines optional linux-git-* jobs which will build the thing
> if LINUX_GIT_VERSION and LINUX_GIT_URL variables are provided for the
> pipeline.
> 
> Signed-off-by: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com>
> ---
> The script variable and job variable need to have different names, so a
> pipeline variable won't override it for all jobs. While LINUX_VERSION /
> LINUX_GIT_VERSION is IMO okay, I'm not very happy about LINUX_URL /
> LINUX_GIT_URL. Any better ideas?

I understand the usefulness of these two jobs for testing purposes (I
mean personal testing during development, not the CI-loop) and based on
that alone I would accept this.

However, I thought I would mention that for personal testing purposes
these 2 jobs don't actually need to be part of the master branch. The
person could add them to their own branch and git push.

That said, I am not opposed to having them in the tree for convenience.

I can also see you are using the two variables in your xen.git patch
series. Is that idea that it would allow for a much wider Linux versions
testing? If we are going to only test one version, this is not required.


> ---
>  .gitlab-ci.yml         | 22 ++++++++++++++++++++++
>  scripts/build-linux.sh | 18 +++++++++++++-----
>  2 files changed, 35 insertions(+), 5 deletions(-)
> 
> diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
> index ad44fb4..60af072 100644
> --- a/.gitlab-ci.yml
> +++ b/.gitlab-ci.yml
> @@ -1,5 +1,9 @@
>  variables:
>    REGISTRY: registry.gitlab.com/xen-project/hardware/test-artifacts
> +  LINUX_GIT_VERSION:
> +    description: "branch/tag/commit for the linux-git jobs"
> +  LINUX_GIT_URL:
> +    description: "git url for the linux-git jobs"
>  
>  stages:
>    - build
> @@ -46,6 +50,15 @@ linux-6.6.86-arm64:
>    variables:
>      LINUX_VERSION: 6.6.86
>  
> +linux-git-arm64:
> +  extends: .arm64-artifacts
> +  script: ./scripts/build-linux.sh
> +  variables:
> +    LINUX_VERSION: $LINUX_GIT_VERSION
> +    LINUX_URL: $LINUX_GIT_URL
> +  rules:
> +  - if: $LINUX_GIT_VERSION && $LINUX_GIT_URL
> +
>  #
>  # x86_64 artifacts
>  #
> @@ -70,6 +83,15 @@ linux-6.12.34-x86_64:
>    variables:
>      LINUX_VERSION: 6.12.34
>  
> +linux-git-x86_64:
> +  extends: .x86_64-artifacts
> +  script: ./scripts/build-linux.sh
> +  variables:
> +    LINUX_VERSION: $LINUX_GIT_VERSION
> +    LINUX_URL: $LINUX_GIT_URL
> +  rules:
> +  - if: $LINUX_GIT_VERSION && $LINUX_GIT_URL
> +
>  microcode-x86:
>    extends: .x86_64-artifacts
>    script: ./scripts/x86-microcode.sh
> diff --git a/scripts/build-linux.sh b/scripts/build-linux.sh
> index cf0e744..1fc96d1 100755
> --- a/scripts/build-linux.sh
> +++ b/scripts/build-linux.sh
> @@ -12,11 +12,19 @@ COPYDIR="${WORKDIR}/binaries"
>  UNAME=$(uname -m)
>  
>  # Build Linux
> -MAJOR=${LINUX_VERSION%%.*}
> -curl -fsSLO \
> -    https://cdn.kernel.org/pub/linux/kernel/v"${MAJOR}".x/linux-"${LINUX_VERSION}".tar.xz
> -tar xf linux-"${LINUX_VERSION}".tar.xz
> -cd linux-"${LINUX_VERSION}"
> +if [[ -n "${LINUX_URL}" ]]; then
> +    mkdir linux
> +    cd linux
> +    git init
> +    git fetch --depth=1 "${LINUX_URL}" "${LINUX_VERSION}"
> +    git checkout FETCH_HEAD
> +else
> +    MAJOR=${LINUX_VERSION%%.*}
> +    curl -fsSLO \
> +        https://cdn.kernel.org/pub/linux/kernel/v"${MAJOR}".x/linux-"${LINUX_VERSION}".tar.xz
> +    tar xf linux-"${LINUX_VERSION}".tar.xz
> +    cd linux-"${LINUX_VERSION}"
> +fi
>  
>  make defconfig
>  ./scripts/config --enable BRIDGE
> -- 
> git-series 0.9.1
> 

[PATCH test-artifacts v1 5/5] Setup ssh access to test systems

[Marek Marczykowski-Górecki]

For this add also bridge package, so xenbr0 can be configured with
/etc/network/interfaces.
This allows extracting more logs out of the test system.

Signed-off-by: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com>
---
This enables passwordless root login. It's okay for qubes runners, as
they are isolated (even from each other). Is that okay in other places
too?
---
 scripts/alpine-rootfs.sh | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/scripts/alpine-rootfs.sh b/scripts/alpine-rootfs.sh
index c999b89..632609b 100755
--- a/scripts/alpine-rootfs.sh
+++ b/scripts/alpine-rootfs.sh
@@ -10,6 +10,8 @@ apk --no-cache upgrade
 
 PKGS=(
     # System
+    bridge
+    dropbear
     libgcc
     openrc
     udev
@@ -62,6 +64,7 @@ rc-update add loopback sysinit
 rc-update add bootmisc boot
 rc-update add devfs sysinit
 rc-update add dmesg sysinit
+rc-update add dropbear boot
 rc-update add hostname boot
 rc-update add hwclock boot
 rc-update add hwdrivers sysinit
@@ -74,6 +77,7 @@ echo "ttyS0" >> /etc/securetty
 echo "hvc0" >> /etc/securetty
 echo "ttyS0::respawn:/sbin/getty -L ttyS0 115200 vt100" >> /etc/inittab
 echo "hvc0::respawn:/sbin/getty -L hvc0 115200 vt100" >> /etc/inittab
+echo "DROPBEAR_OPTS=\"-B -R\"" >> /etc/conf.d/dropbear
 echo "rc_verbose=yes" >> /etc/rc.conf
 echo > /etc/modules
 passwd -d "root" root
-- 
git-series 0.9.1

Re: [PATCH test-artifacts v1 5/5] Setup ssh access to test systems

[Andrew Cooper]

On 23/06/2025 2:46 pm, Marek Marczykowski-Górecki wrote:
> For this add also bridge package, so xenbr0 can be configured with
> /etc/network/interfaces.
> This allows extracting more logs out of the test system.
>
> Signed-off-by: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com>
> ---
> This enables passwordless root login. It's okay for qubes runners, as
> they are isolated (even from each other). Is that okay in other places
> too?

It's potentially a problem on a corporate network.

Can't we have each job generate a random password an insert it via the
dom0-rootfs overlay?

Or alternatively have the runner drop a public key in
/root/.ssh/authorised_keys ?

~Andrew

Re: [PATCH test-artifacts v1 5/5] Setup ssh access to test systems

[Marek Marczykowski-Górecki]

[-- Attachment #1: Type: text/plain, Size: 963 bytes --]

On Mon, Jun 23, 2025 at 02:56:00PM +0100, Andrew Cooper wrote:
> On 23/06/2025 2:46 pm, Marek Marczykowski-Górecki wrote:
> > For this add also bridge package, so xenbr0 can be configured with
> > /etc/network/interfaces.
> > This allows extracting more logs out of the test system.
> >
> > Signed-off-by: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com>
> > ---
> > This enables passwordless root login. It's okay for qubes runners, as
> > they are isolated (even from each other). Is that okay in other places
> > too?
> 
> It's potentially a problem on a corporate network.
> 
> Can't we have each job generate a random password an insert it via the
> dom0-rootfs overlay?
> 
> Or alternatively have the runner drop a public key in
> /root/.ssh/authorised_keys ?

That can work, yes. And is preferred to a password, as easier to do
non-interactively.

-- 
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [PATCH test-artifacts v1 5/5] Setup ssh access to test systems

[Stefano Stabellini]

[-- Attachment #1: Type: text/plain, Size: 971 bytes --]

On Mon, 23 Jun 2025, Marek Marczykowski-Górecki wrote:
> On Mon, Jun 23, 2025 at 02:56:00PM +0100, Andrew Cooper wrote:
> > On 23/06/2025 2:46 pm, Marek Marczykowski-Górecki wrote:
> > > For this add also bridge package, so xenbr0 can be configured with
> > > /etc/network/interfaces.
> > > This allows extracting more logs out of the test system.
> > >
> > > Signed-off-by: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com>
> > > ---
> > > This enables passwordless root login. It's okay for qubes runners, as
> > > they are isolated (even from each other). Is that okay in other places
> > > too?
> > 
> > It's potentially a problem on a corporate network.
> > 
> > Can't we have each job generate a random password an insert it via the
> > dom0-rootfs overlay?
> > 
> > Or alternatively have the runner drop a public key in
> > /root/.ssh/authorised_keys ?
> 
> That can work, yes. And is preferred to a password, as easier to do
> non-interactively.

+1