[PATCH TEST-ARTEFACTS 0/3] Rootfs generation (ARM and x86)

All of lore.kernel.org
 help / color / mirror / Atom feed

* [PATCH TEST-ARTEFACTS 0/3] Rootfs generation (ARM and x86)
@ 2025-04-11 10:52 Andrew Cooper
  2025-04-11 10:52 ` [PATCH 1/3] Rework rootfs generation to make a cpio archive Andrew Cooper
                   ` (2 more replies)
  0 siblings, 3 replies; 15+ messages in thread
From: Andrew Cooper @ 2025-04-11 10:52 UTC (permalink / raw)
  To: Xen-devel
  Cc: Andrew Cooper, Anthony PERARD, Stefano Stabellini, Michal Orzel,
	Doug Goldstein, Marek Marczykowski-Górecki

Part 3 of the cleanup, reworking rootfs generation.

The Xen side of the change isn't quite ready yet, but this is in a good enough
state for review.

Andrew Cooper (3):
  Rework rootfs generation to make a cpio archive
  Shrink the rootfs substantially
  Provide an ARM64 rootfs too

 .gitlab-ci.yml                           | 16 ++++-
 containerize                             |  1 +
 images/alpine/3.18-arm64-base.dockerfile |  6 ++
 scripts/alpine-rootfs.sh                 | 92 ++++++++++++++++++++++++
 scripts/x86_64-rootfs-alpine.sh          | 60 ----------------
 5 files changed, 114 insertions(+), 61 deletions(-)
 create mode 100644 images/alpine/3.18-arm64-base.dockerfile
 create mode 100755 scripts/alpine-rootfs.sh
 delete mode 100755 scripts/x86_64-rootfs-alpine.sh

-- 
2.39.5



^ permalink raw reply	[flat|nested] 15+ messages in thread

* [PATCH 1/3] Rework rootfs generation to make a cpio archive
  2025-04-11 10:52 [PATCH TEST-ARTEFACTS 0/3] Rootfs generation (ARM and x86) Andrew Cooper
@ 2025-04-11 10:52 ` Andrew Cooper
  2025-04-11 11:06   ` Marek Marczykowski-Górecki
                     ` (3 more replies)
  2025-04-11 10:52 ` [PATCH 2/3] Shrink the rootfs substantially Andrew Cooper
  2025-04-11 10:52 ` [PATCH 3/3] Provide an ARM64 rootfs too Andrew Cooper
  2 siblings, 4 replies; 15+ messages in thread
From: Andrew Cooper @ 2025-04-11 10:52 UTC (permalink / raw)
  To: Xen-devel
  Cc: Andrew Cooper, Anthony PERARD, Stefano Stabellini, Michal Orzel,
	Doug Goldstein, Marek Marczykowski-Górecki

Rename the script as we're going to use it for ARM64 shortly, and have it take
a tar or cpio parameter to determine the output format.

Turn it into a proper bash script, and provide the cpio form under the new
artefact naming scheme.

No functional change.

Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
---
CC: Anthony PERARD <anthony.perard@vates.tech>
CC: Stefano Stabellini <sstabellini@kernel.org>
CC: Michal Orzel <michal.orzel@amd.com>
CC: Doug Goldstein <cardoe@cardoe.com>
CC: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com>
---
 .gitlab-ci.yml                                |  9 +++++++-
 ...6_64-rootfs-alpine.sh => alpine-rootfs.sh} | 21 +++++++++++++++++--
 2 files changed, 27 insertions(+), 3 deletions(-)
 rename scripts/{x86_64-rootfs-alpine.sh => alpine-rootfs.sh} (75%)

diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index 1d2c72b268a3..916c5ae9d508 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -42,6 +42,13 @@ linux-6.6.86-arm64:
 #
 # x86_64 artifacts
 #
+alpine-3.18-x86_64-rootfs:
+  extends: .x86_64-artifacts
+  script:
+    - ./scripts/alpine-rootfs.sh cpio
+  variables:
+    CONTAINER: alpine:3.18-x86_64-base
+
 linux-6.6.56-x86_64:
   extends: .x86_64-artifacts
   script: ./scripts/build-linux.sh
@@ -62,7 +69,7 @@ x86_64-kernel-linux-6.6.56:
 x86_64-rootfs-alpine-3.18:
   extends: .x86_64-artifacts
   script:
-    - . scripts/x86_64-rootfs-alpine.sh
+    - ./scripts/alpine-rootfs.sh tar
   variables:
     CONTAINER: alpine:3.18-x86_64-base
 
diff --git a/scripts/x86_64-rootfs-alpine.sh b/scripts/alpine-rootfs.sh
similarity index 75%
rename from scripts/x86_64-rootfs-alpine.sh
rename to scripts/alpine-rootfs.sh
index b70b3a54ede5..c29c92d1c682 100755
--- a/scripts/x86_64-rootfs-alpine.sh
+++ b/scripts/alpine-rootfs.sh
@@ -1,4 +1,9 @@
+#!/bin/bash
+
+set -eu
+
 WORKDIR="${PWD}"
+COPYDIR="${WORKDIR}/binaries"
 
 apk update
 
@@ -56,5 +61,17 @@ passwd -d "root" root
 
 # Create rootfs
 cd /
-tar cvzf "${WORKDIR}/binaries/initrd.tar.gz" \
-    bin dev etc home init lib mnt opt root sbin usr var
+PATHS="bin dev etc home init lib mnt opt root sbin usr var"
+
+case $1 in
+    cpio)
+        find $PATHS | cpio -o -H newc | gzip > "${COPYDIR}/rootfs.cpio.gz"
+
+        # Print the contents for the build log
+        zcat "${COPYDIR}/rootfs.cpio.gz" | cpio -tv
+        ;;
+
+    tar)
+        tar cvzf "${COPYDIR}/initrd.tar.gz" $PATHS
+        ;;
+esac
-- 
2.39.5



^ permalink raw reply related	[flat|nested] 15+ messages in thread

* Re: [PATCH 1/3] Rework rootfs generation to make a cpio archive
  2025-04-11 10:52 ` [PATCH 1/3] Rework rootfs generation to make a cpio archive Andrew Cooper
@ 2025-04-11 11:06   ` Marek Marczykowski-Górecki
  2025-04-11 11:09     ` Andrew Cooper
  2025-04-11 12:25   ` Frediano Ziglio
                     ` (2 subsequent siblings)
  3 siblings, 1 reply; 15+ messages in thread
From: Marek Marczykowski-Górecki @ 2025-04-11 11:06 UTC (permalink / raw)
  To: Andrew Cooper
  Cc: Xen-devel, Anthony PERARD, Stefano Stabellini, Michal Orzel,
	Doug Goldstein

[-- Attachment #1: Type: text/plain, Size: 2878 bytes --]

On Fri, Apr 11, 2025 at 11:52:15AM +0100, Andrew Cooper wrote:
> Rename the script as we're going to use it for ARM64 shortly, and have it take
> a tar or cpio parameter to determine the output format.
> 
> Turn it into a proper bash script, and provide the cpio form under the new
> artefact naming scheme.
> 
> No functional change.
> 
> Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
> ---
> CC: Anthony PERARD <anthony.perard@vates.tech>
> CC: Stefano Stabellini <sstabellini@kernel.org>
> CC: Michal Orzel <michal.orzel@amd.com>
> CC: Doug Goldstein <cardoe@cardoe.com>
> CC: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com>
> ---
>  .gitlab-ci.yml                                |  9 +++++++-
>  ...6_64-rootfs-alpine.sh => alpine-rootfs.sh} | 21 +++++++++++++++++--
>  2 files changed, 27 insertions(+), 3 deletions(-)
>  rename scripts/{x86_64-rootfs-alpine.sh => alpine-rootfs.sh} (75%)
> 
> diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
> index 1d2c72b268a3..916c5ae9d508 100644
> --- a/.gitlab-ci.yml
> +++ b/.gitlab-ci.yml
> @@ -42,6 +42,13 @@ linux-6.6.86-arm64:
>  #
>  # x86_64 artifacts
>  #
> +alpine-3.18-x86_64-rootfs:
> +  extends: .x86_64-artifacts
> +  script:
> +    - ./scripts/alpine-rootfs.sh cpio
> +  variables:
> +    CONTAINER: alpine:3.18-x86_64-base
> +
>  linux-6.6.56-x86_64:
>    extends: .x86_64-artifacts
>    script: ./scripts/build-linux.sh
> @@ -62,7 +69,7 @@ x86_64-kernel-linux-6.6.56:
>  x86_64-rootfs-alpine-3.18:
>    extends: .x86_64-artifacts
>    script:
> -    - . scripts/x86_64-rootfs-alpine.sh
> +    - ./scripts/alpine-rootfs.sh tar
>    variables:
>      CONTAINER: alpine:3.18-x86_64-base
>  
> diff --git a/scripts/x86_64-rootfs-alpine.sh b/scripts/alpine-rootfs.sh
> similarity index 75%
> rename from scripts/x86_64-rootfs-alpine.sh
> rename to scripts/alpine-rootfs.sh
> index b70b3a54ede5..c29c92d1c682 100755
> --- a/scripts/x86_64-rootfs-alpine.sh
> +++ b/scripts/alpine-rootfs.sh
> @@ -1,4 +1,9 @@
> +#!/bin/bash
> +
> +set -eu
> +
>  WORKDIR="${PWD}"
> +COPYDIR="${WORKDIR}/binaries"
>  
>  apk update
>  
> @@ -56,5 +61,17 @@ passwd -d "root" root
>  
>  # Create rootfs
>  cd /
> -tar cvzf "${WORKDIR}/binaries/initrd.tar.gz" \
> -    bin dev etc home init lib mnt opt root sbin usr var
> +PATHS="bin dev etc home init lib mnt opt root sbin usr var"
> +
> +case $1 in
> +    cpio)
> +        find $PATHS | cpio -o -H newc | gzip > "${COPYDIR}/rootfs.cpio.gz"
> +
> +        # Print the contents for the build log
> +        zcat "${COPYDIR}/rootfs.cpio.gz" | cpio -tv

add 'v' to cpio above instead?

> +        ;;
> +
> +    tar)
> +        tar cvzf "${COPYDIR}/initrd.tar.gz" $PATHS
> +        ;;
> +esac
> -- 
> 2.39.5
> 

-- 
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: [PATCH 1/3] Rework rootfs generation to make a cpio archive
  2025-04-11 11:06   ` Marek Marczykowski-Górecki
@ 2025-04-11 11:09     ` Andrew Cooper
  2025-04-11 11:33       ` Andrew Cooper
  0 siblings, 1 reply; 15+ messages in thread
From: Andrew Cooper @ 2025-04-11 11:09 UTC (permalink / raw)
  To: Marek Marczykowski-Górecki
  Cc: Xen-devel, Anthony PERARD, Stefano Stabellini, Michal Orzel,
	Doug Goldstein

On 11/04/2025 12:06 pm, Marek Marczykowski-Górecki wrote:
> On Fri, Apr 11, 2025 at 11:52:15AM +0100, Andrew Cooper wrote:
>> diff --git a/scripts/x86_64-rootfs-alpine.sh b/scripts/alpine-rootfs.sh
>> similarity index 75%
>> rename from scripts/x86_64-rootfs-alpine.sh
>> rename to scripts/alpine-rootfs.sh
>> index b70b3a54ede5..c29c92d1c682 100755
>> --- a/scripts/x86_64-rootfs-alpine.sh
>> +++ b/scripts/alpine-rootfs.sh
>> @@ -56,5 +61,17 @@ passwd -d "root" root
>>  
>>  # Create rootfs
>>  cd /
>> -tar cvzf "${WORKDIR}/binaries/initrd.tar.gz" \
>> -    bin dev etc home init lib mnt opt root sbin usr var
>> +PATHS="bin dev etc home init lib mnt opt root sbin usr var"
>> +
>> +case $1 in
>> +    cpio)
>> +        find $PATHS | cpio -o -H newc | gzip > "${COPYDIR}/rootfs.cpio.gz"
>> +
>> +        # Print the contents for the build log
>> +        zcat "${COPYDIR}/rootfs.cpio.gz" | cpio -tv
> add 'v' to cpio above instead?

Oh, of course.  There's another instance of this pattern in argo, which
I'll fix up too.

~Andrew


^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: [PATCH 1/3] Rework rootfs generation to make a cpio archive
  2025-04-11 11:09     ` Andrew Cooper
@ 2025-04-11 11:33       ` Andrew Cooper
  2025-04-11 11:47         ` Marek Marczykowski-Górecki
  0 siblings, 1 reply; 15+ messages in thread
From: Andrew Cooper @ 2025-04-11 11:33 UTC (permalink / raw)
  To: Marek Marczykowski-Górecki
  Cc: Xen-devel, Anthony PERARD, Stefano Stabellini, Michal Orzel,
	Doug Goldstein

On 11/04/2025 12:09 pm, Andrew Cooper wrote:
> On 11/04/2025 12:06 pm, Marek Marczykowski-Górecki wrote:
>> On Fri, Apr 11, 2025 at 11:52:15AM +0100, Andrew Cooper wrote:
>>> diff --git a/scripts/x86_64-rootfs-alpine.sh b/scripts/alpine-rootfs.sh
>>> similarity index 75%
>>> rename from scripts/x86_64-rootfs-alpine.sh
>>> rename to scripts/alpine-rootfs.sh
>>> index b70b3a54ede5..c29c92d1c682 100755
>>> --- a/scripts/x86_64-rootfs-alpine.sh
>>> +++ b/scripts/alpine-rootfs.sh
>>> @@ -56,5 +61,17 @@ passwd -d "root" root
>>>  
>>>  # Create rootfs
>>>  cd /
>>> -tar cvzf "${WORKDIR}/binaries/initrd.tar.gz" \
>>> -    bin dev etc home init lib mnt opt root sbin usr var
>>> +PATHS="bin dev etc home init lib mnt opt root sbin usr var"
>>> +
>>> +case $1 in
>>> +    cpio)
>>> +        find $PATHS | cpio -o -H newc | gzip > "${COPYDIR}/rootfs.cpio.gz"
>>> +
>>> +        # Print the contents for the build log
>>> +        zcat "${COPYDIR}/rootfs.cpio.gz" | cpio -tv
>> add 'v' to cpio above instead?
> Oh, of course.  There's another instance of this pattern in argo, which
> I'll fix up too.

Apparently not. 
https://gitlab.com/xen-project/hardware/test-artifacts/-/jobs/9695952322

I expect the -v is getting swallowed by the pipe.

~Andrew


^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: [PATCH 1/3] Rework rootfs generation to make a cpio archive
  2025-04-11 11:33       ` Andrew Cooper
@ 2025-04-11 11:47         ` Marek Marczykowski-Górecki
  0 siblings, 0 replies; 15+ messages in thread
From: Marek Marczykowski-Górecki @ 2025-04-11 11:47 UTC (permalink / raw)
  To: Andrew Cooper
  Cc: Xen-devel, Anthony PERARD, Stefano Stabellini, Michal Orzel,
	Doug Goldstein

[-- Attachment #1: Type: text/plain, Size: 907 bytes --]

On Fri, Apr 11, 2025 at 12:33:50PM +0100, Andrew Cooper wrote:
> On 11/04/2025 12:09 pm, Andrew Cooper wrote:
> > On 11/04/2025 12:06 pm, Marek Marczykowski-Górecki wrote:
> >> On Fri, Apr 11, 2025 at 11:52:15AM +0100, Andrew Cooper wrote:
> >>> +        find $PATHS | cpio -o -H newc | gzip > "${COPYDIR}/rootfs.cpio.gz"
> >>> +
> >>> +        # Print the contents for the build log
> >>> +        zcat "${COPYDIR}/rootfs.cpio.gz" | cpio -tv
> >> add 'v' to cpio above instead?
> > Oh, of course.  There's another instance of this pattern in argo, which
> > I'll fix up too.
> 
> Apparently not. 
> https://gitlab.com/xen-project/hardware/test-artifacts/-/jobs/9695952322
> 
> I expect the -v is getting swallowed by the pipe.

Great, busybox's cpio prints it to stdout instead of stderr (as normal
cpio do)...

-- 
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: [PATCH 1/3] Rework rootfs generation to make a cpio archive
  2025-04-11 10:52 ` [PATCH 1/3] Rework rootfs generation to make a cpio archive Andrew Cooper
  2025-04-11 11:06   ` Marek Marczykowski-Górecki
@ 2025-04-11 12:25   ` Frediano Ziglio
  2025-04-11 12:39     ` Andrew Cooper
  2025-04-11 15:53   ` Denis Mukhin
  2025-04-11 18:36   ` Denis Mukhin
  3 siblings, 1 reply; 15+ messages in thread
From: Frediano Ziglio @ 2025-04-11 12:25 UTC (permalink / raw)
  To: Andrew Cooper
  Cc: Xen-devel, Anthony PERARD, Stefano Stabellini, Michal Orzel,
	Doug Goldstein, Marek Marczykowski-Górecki

On Fri, Apr 11, 2025 at 11:52 AM Andrew Cooper
<andrew.cooper3@citrix.com> wrote:
>
> Rename the script as we're going to use it for ARM64 shortly, and have it take
> a tar or cpio parameter to determine the output format.
>
> Turn it into a proper bash script, and provide the cpio form under the new
> artefact naming scheme.
>
> No functional change.
>
> Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
> ---
> CC: Anthony PERARD <anthony.perard@vates.tech>
> CC: Stefano Stabellini <sstabellini@kernel.org>
> CC: Michal Orzel <michal.orzel@amd.com>
> CC: Doug Goldstein <cardoe@cardoe.com>
> CC: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com>
> ---
>  .gitlab-ci.yml                                |  9 +++++++-
>  ...6_64-rootfs-alpine.sh => alpine-rootfs.sh} | 21 +++++++++++++++++--
>  2 files changed, 27 insertions(+), 3 deletions(-)
>  rename scripts/{x86_64-rootfs-alpine.sh => alpine-rootfs.sh} (75%)
>
> diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
> index 1d2c72b268a3..916c5ae9d508 100644
> --- a/.gitlab-ci.yml
> +++ b/.gitlab-ci.yml
> @@ -42,6 +42,13 @@ linux-6.6.86-arm64:
>  #
>  # x86_64 artifacts
>  #
> +alpine-3.18-x86_64-rootfs:
> +  extends: .x86_64-artifacts
> +  script:
> +    - ./scripts/alpine-rootfs.sh cpio
> +  variables:
> +    CONTAINER: alpine:3.18-x86_64-base
> +
>  linux-6.6.56-x86_64:
>    extends: .x86_64-artifacts
>    script: ./scripts/build-linux.sh
> @@ -62,7 +69,7 @@ x86_64-kernel-linux-6.6.56:
>  x86_64-rootfs-alpine-3.18:
>    extends: .x86_64-artifacts
>    script:
> -    - . scripts/x86_64-rootfs-alpine.sh
> +    - ./scripts/alpine-rootfs.sh tar
>    variables:
>      CONTAINER: alpine:3.18-x86_64-base
>
> diff --git a/scripts/x86_64-rootfs-alpine.sh b/scripts/alpine-rootfs.sh
> similarity index 75%
> rename from scripts/x86_64-rootfs-alpine.sh
> rename to scripts/alpine-rootfs.sh
> index b70b3a54ede5..c29c92d1c682 100755
> --- a/scripts/x86_64-rootfs-alpine.sh
> +++ b/scripts/alpine-rootfs.sh
> @@ -1,4 +1,9 @@
> +#!/bin/bash
> +
> +set -eu
> +
>  WORKDIR="${PWD}"
> +COPYDIR="${WORKDIR}/binaries"
>
>  apk update
>
> @@ -56,5 +61,17 @@ passwd -d "root" root
>
>  # Create rootfs
>  cd /
> -tar cvzf "${WORKDIR}/binaries/initrd.tar.gz" \
> -    bin dev etc home init lib mnt opt root sbin usr var
> +PATHS="bin dev etc home init lib mnt opt root sbin usr var"
> +
> +case $1 in
> +    cpio)
> +        find $PATHS | cpio -o -H newc | gzip > "${COPYDIR}/rootfs.cpio.gz"
> +

Paranoia mode: -print0 for find and -0 for cpio to support weird file names?

> +        # Print the contents for the build log
> +        zcat "${COPYDIR}/rootfs.cpio.gz" | cpio -tv

I saw other comments about -v and busybox compatibility.
I suppose it depends how important are performances here, otherwise I
could suggest the usage of mkfifo before, use -F option to the fifo
and use an additional pipeline to compress the file.

mypipe=$(mktemp -u)
mkfifo $mypipe
gzip -c > "${COPYDIR}/rootfs.cpio.gz" < $mypipe &
find $PATHS -print0 | cpio -o -H newc -F $mypipe -0
wait # for gzip
rm $mypipe

> +        ;;
> +
> +    tar)
> +        tar cvzf "${COPYDIR}/initrd.tar.gz" $PATHS
> +        ;;
> +esac
> --
> 2.39.5
>
>

Frediano


^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: [PATCH 1/3] Rework rootfs generation to make a cpio archive
  2025-04-11 12:25   ` Frediano Ziglio
@ 2025-04-11 12:39     ` Andrew Cooper
  0 siblings, 0 replies; 15+ messages in thread
From: Andrew Cooper @ 2025-04-11 12:39 UTC (permalink / raw)
  To: Frediano Ziglio
  Cc: Xen-devel, Anthony PERARD, Stefano Stabellini, Michal Orzel,
	Doug Goldstein, Marek Marczykowski-Górecki

On 11/04/2025 1:25 pm, Frediano Ziglio wrote:
> On Fri, Apr 11, 2025 at 11:52 AM Andrew Cooper
> <andrew.cooper3@citrix.com> wrote:
>> Rename the script as we're going to use it for ARM64 shortly, and have it take
>> a tar or cpio parameter to determine the output format.
>>
>> Turn it into a proper bash script, and provide the cpio form under the new
>> artefact naming scheme.
>>
>> No functional change.
>>
>> Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
>> ---
>> CC: Anthony PERARD <anthony.perard@vates.tech>
>> CC: Stefano Stabellini <sstabellini@kernel.org>
>> CC: Michal Orzel <michal.orzel@amd.com>
>> CC: Doug Goldstein <cardoe@cardoe.com>
>> CC: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com>
>> ---
>>  .gitlab-ci.yml                                |  9 +++++++-
>>  ...6_64-rootfs-alpine.sh => alpine-rootfs.sh} | 21 +++++++++++++++++--
>>  2 files changed, 27 insertions(+), 3 deletions(-)
>>  rename scripts/{x86_64-rootfs-alpine.sh => alpine-rootfs.sh} (75%)
>>
>> diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
>> index 1d2c72b268a3..916c5ae9d508 100644
>> --- a/.gitlab-ci.yml
>> +++ b/.gitlab-ci.yml
>> @@ -42,6 +42,13 @@ linux-6.6.86-arm64:
>>  #
>>  # x86_64 artifacts
>>  #
>> +alpine-3.18-x86_64-rootfs:
>> +  extends: .x86_64-artifacts
>> +  script:
>> +    - ./scripts/alpine-rootfs.sh cpio
>> +  variables:
>> +    CONTAINER: alpine:3.18-x86_64-base
>> +
>>  linux-6.6.56-x86_64:
>>    extends: .x86_64-artifacts
>>    script: ./scripts/build-linux.sh
>> @@ -62,7 +69,7 @@ x86_64-kernel-linux-6.6.56:
>>  x86_64-rootfs-alpine-3.18:
>>    extends: .x86_64-artifacts
>>    script:
>> -    - . scripts/x86_64-rootfs-alpine.sh
>> +    - ./scripts/alpine-rootfs.sh tar
>>    variables:
>>      CONTAINER: alpine:3.18-x86_64-base
>>
>> diff --git a/scripts/x86_64-rootfs-alpine.sh b/scripts/alpine-rootfs.sh
>> similarity index 75%
>> rename from scripts/x86_64-rootfs-alpine.sh
>> rename to scripts/alpine-rootfs.sh
>> index b70b3a54ede5..c29c92d1c682 100755
>> --- a/scripts/x86_64-rootfs-alpine.sh
>> +++ b/scripts/alpine-rootfs.sh
>> @@ -1,4 +1,9 @@
>> +#!/bin/bash
>> +
>> +set -eu
>> +
>>  WORKDIR="${PWD}"
>> +COPYDIR="${WORKDIR}/binaries"
>>
>>  apk update
>>
>> @@ -56,5 +61,17 @@ passwd -d "root" root
>>
>>  # Create rootfs
>>  cd /
>> -tar cvzf "${WORKDIR}/binaries/initrd.tar.gz" \
>> -    bin dev etc home init lib mnt opt root sbin usr var
>> +PATHS="bin dev etc home init lib mnt opt root sbin usr var"
>> +
>> +case $1 in
>> +    cpio)
>> +        find $PATHS | cpio -o -H newc | gzip > "${COPYDIR}/rootfs.cpio.gz"
>> +
> Paranoia mode: -print0 for find and -0 for cpio to support weird file names?

This is a tiny alpine rootfs.  There aren't any, but weird filenames
would be a problem elsewhere too.

>
>> +        # Print the contents for the build log
>> +        zcat "${COPYDIR}/rootfs.cpio.gz" | cpio -tv
> I saw other comments about -v and busybox compatibility.
> I suppose it depends how important are performances here, otherwise I
> could suggest the usage of mkfifo before, use -F option to the fifo
> and use an additional pipeline to compress the file.
>
> mypipe=$(mktemp -u)
> mkfifo $mypipe
> gzip -c > "${COPYDIR}/rootfs.cpio.gz" < $mypipe &
> find $PATHS -print0 | cpio -o -H newc -F $mypipe -0
> wait # for gzip
> rm $mypipe

These artefacts are built once, then reused indefinitely.  Generating
this artefact takes 18s, most of which is the gitlab runner starting up.

~Andrew


^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: [PATCH 1/3] Rework rootfs generation to make a cpio archive
  2025-04-11 10:52 ` [PATCH 1/3] Rework rootfs generation to make a cpio archive Andrew Cooper
  2025-04-11 11:06   ` Marek Marczykowski-Górecki
  2025-04-11 12:25   ` Frediano Ziglio
@ 2025-04-11 15:53   ` Denis Mukhin
  2025-04-11 17:50     ` Andrew Cooper
  2025-04-11 18:36   ` Denis Mukhin
  3 siblings, 1 reply; 15+ messages in thread
From: Denis Mukhin @ 2025-04-11 15:53 UTC (permalink / raw)
  To: Andrew Cooper
  Cc: Xen-devel, Anthony PERARD, Stefano Stabellini, Michal Orzel,
	Doug Goldstein, Marek Marczykowski-Górecki

On Friday, April 11th, 2025 at 3:52 AM, Andrew Cooper <andrew.cooper3@citrix.com> wrote:

> 
> 
> Rename the script as we're going to use it for ARM64 shortly, and have it take
> a tar or cpio parameter to determine the output format.
> 
> Turn it into a proper bash script, and provide the cpio form under the new
> artefact naming scheme.
> 
> No functional change.
> 
> Signed-off-by: Andrew Cooper andrew.cooper3@citrix.com
> 
> ---
> CC: Anthony PERARD anthony.perard@vates.tech
> 
> CC: Stefano Stabellini sstabellini@kernel.org
> 
> CC: Michal Orzel michal.orzel@amd.com
> 
> CC: Doug Goldstein cardoe@cardoe.com
> 
> CC: Marek Marczykowski-Górecki marmarek@invisiblethingslab.com
> 
> ---
> .gitlab-ci.yml | 9 +++++++-
> ...6_64-rootfs-alpine.sh => alpine-rootfs.sh} | 21 +++++++++++++++++--
> 
> 2 files changed, 27 insertions(+), 3 deletions(-)
> rename scripts/{x86_64-rootfs-alpine.sh => alpine-rootfs.sh} (75%)
> 
> 
> diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
> index 1d2c72b268a3..916c5ae9d508 100644
> --- a/.gitlab-ci.yml
> +++ b/.gitlab-ci.yml
> @@ -42,6 +42,13 @@ linux-6.6.86-arm64:
> #
> # x86_64 artifacts
> #
> +alpine-3.18-x86_64-rootfs:
> + extends: .x86_64-artifacts
> + script:
> + - ./scripts/alpine-rootfs.sh cpio
> + variables:
> + CONTAINER: alpine:3.18-x86_64-base
> +
> linux-6.6.56-x86_64:
> extends: .x86_64-artifacts
> script: ./scripts/build-linux.sh
> @@ -62,7 +69,7 @@ x86_64-kernel-linux-6.6.56:
> x86_64-rootfs-alpine-3.18:
> extends: .x86_64-artifacts
> script:
> - - . scripts/x86_64-rootfs-alpine.sh
> + - ./scripts/alpine-rootfs.sh tar
> variables:
> CONTAINER: alpine:3.18-x86_64-base
> 
> diff --git a/scripts/x86_64-rootfs-alpine.sh b/scripts/alpine-rootfs.sh
> similarity index 75%
> rename from scripts/x86_64-rootfs-alpine.sh
> rename to scripts/alpine-rootfs.sh
> index b70b3a54ede5..c29c92d1c682 100755
> --- a/scripts/x86_64-rootfs-alpine.sh
> +++ b/scripts/alpine-rootfs.sh
> @@ -1,4 +1,9 @@
> +#!/bin/bash
> +
> +set -eu
> +
> WORKDIR="${PWD}"
> +COPYDIR="${WORKDIR}/binaries"
> 
> apk update
> 
> @@ -56,5 +61,17 @@ passwd -d "root" root
> 
> # Create rootfs
> cd /
> -tar cvzf "${WORKDIR}/binaries/initrd.tar.gz" \
> - bin dev etc home init lib mnt opt root sbin usr var
> +PATHS="bin dev etc home init lib mnt opt root sbin usr var"
> +
> +case $1 in
> + cpio)
> + find $PATHS | cpio -o -H newc | gzip > "${COPYDIR}/rootfs.cpio.gz"
> 
> +
> + # Print the contents for the build log
> + zcat "${COPYDIR}/rootfs.cpio.gz" | cpio -tv

Maybe allow $1 to be a filename? E. g. rootfs.cpio.gz or initrd.tar.gz.
This way $1 will contain an explicit output artifact format along with
the output filename.

What do you think?

> + ;;
> +
> + tar)
> + tar cvzf "${COPYDIR}/initrd.tar.gz" $PATHS
> + ;;

I would add the default case and error out in case of possible mis-use.

> +esac
> --
> 2.39.5


^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: [PATCH 1/3] Rework rootfs generation to make a cpio archive
  2025-04-11 15:53   ` Denis Mukhin
@ 2025-04-11 17:50     ` Andrew Cooper
  2025-04-11 18:26       ` Denis Mukhin
  0 siblings, 1 reply; 15+ messages in thread
From: Andrew Cooper @ 2025-04-11 17:50 UTC (permalink / raw)
  To: Denis Mukhin
  Cc: Xen-devel, Anthony PERARD, Stefano Stabellini, Michal Orzel,
	Doug Goldstein, Marek Marczykowski-Górecki

On 11/04/2025 4:53 pm, Denis Mukhin wrote:
> On Friday, April 11th, 2025 at 3:52 AM, Andrew Cooper <andrew.cooper3@citrix.com> wrote:
>
>>
>> Rename the script as we're going to use it for ARM64 shortly, and have it take
>> a tar or cpio parameter to determine the output format.
>>
>> Turn it into a proper bash script, and provide the cpio form under the new
>> artefact naming scheme.
>>
>> No functional change.
>>
>> Signed-off-by: Andrew Cooper andrew.cooper3@citrix.com
>>
>> ---
>> CC: Anthony PERARD anthony.perard@vates.tech
>>
>> CC: Stefano Stabellini sstabellini@kernel.org
>>
>> CC: Michal Orzel michal.orzel@amd.com
>>
>> CC: Doug Goldstein cardoe@cardoe.com
>>
>> CC: Marek Marczykowski-Górecki marmarek@invisiblethingslab.com
>>
>> ---
>> .gitlab-ci.yml | 9 +++++++-
>> ...6_64-rootfs-alpine.sh => alpine-rootfs.sh} | 21 +++++++++++++++++--
>>
>> 2 files changed, 27 insertions(+), 3 deletions(-)
>> rename scripts/{x86_64-rootfs-alpine.sh => alpine-rootfs.sh} (75%)
>>
>>
>> diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
>> index 1d2c72b268a3..916c5ae9d508 100644
>> --- a/.gitlab-ci.yml
>> +++ b/.gitlab-ci.yml
>> @@ -42,6 +42,13 @@ linux-6.6.86-arm64:
>> #
>> # x86_64 artifacts
>> #
>> +alpine-3.18-x86_64-rootfs:
>> + extends: .x86_64-artifacts
>> + script:
>> + - ./scripts/alpine-rootfs.sh cpio
>> + variables:
>> + CONTAINER: alpine:3.18-x86_64-base
>> +
>> linux-6.6.56-x86_64:
>> extends: .x86_64-artifacts
>> script: ./scripts/build-linux.sh
>> @@ -62,7 +69,7 @@ x86_64-kernel-linux-6.6.56:
>> x86_64-rootfs-alpine-3.18:
>> extends: .x86_64-artifacts
>> script:
>> - - . scripts/x86_64-rootfs-alpine.sh
>> + - ./scripts/alpine-rootfs.sh tar
>> variables:
>> CONTAINER: alpine:3.18-x86_64-base
>>
>> diff --git a/scripts/x86_64-rootfs-alpine.sh b/scripts/alpine-rootfs.sh
>> similarity index 75%
>> rename from scripts/x86_64-rootfs-alpine.sh
>> rename to scripts/alpine-rootfs.sh
>> index b70b3a54ede5..c29c92d1c682 100755
>> --- a/scripts/x86_64-rootfs-alpine.sh
>> +++ b/scripts/alpine-rootfs.sh
>> @@ -1,4 +1,9 @@
>> +#!/bin/bash
>> +
>> +set -eu
>> +
>> WORKDIR="${PWD}"
>> +COPYDIR="${WORKDIR}/binaries"
>>
>> apk update
>>
>> @@ -56,5 +61,17 @@ passwd -d "root" root
>>
>> # Create rootfs
>> cd /
>> -tar cvzf "${WORKDIR}/binaries/initrd.tar.gz" \
>> - bin dev etc home init lib mnt opt root sbin usr var
>> +PATHS="bin dev etc home init lib mnt opt root sbin usr var"
>> +
>> +case $1 in
>> + cpio)
>> + find $PATHS | cpio -o -H newc | gzip > "${COPYDIR}/rootfs.cpio.gz"
>>
>> +
>> + # Print the contents for the build log
>> + zcat "${COPYDIR}/rootfs.cpio.gz" | cpio -tv
> Maybe allow $1 to be a filename? E. g. rootfs.cpio.gz or initrd.tar.gz.
> This way $1 will contain an explicit output artifact format along with
> the output filename.
>
> What do you think?
>
>> + ;;
>> +
>> + tar)
>> + tar cvzf "${COPYDIR}/initrd.tar.gz" $PATHS
>> + ;;
> I would add the default case and error out in case of possible mis-use.

This is bespoke tooling, used as part of Xen's GitlabCI infrastructure.

The parameter is going to last for all of about a week, until we can
retire the tests using the tarball, because they're creating routine but
intermittent test failure.

I'm not looking to over-engineer the result.

~Andrew


^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: [PATCH 1/3] Rework rootfs generation to make a cpio archive
  2025-04-11 17:50     ` Andrew Cooper
@ 2025-04-11 18:26       ` Denis Mukhin
  0 siblings, 0 replies; 15+ messages in thread
From: Denis Mukhin @ 2025-04-11 18:26 UTC (permalink / raw)
  To: Andrew Cooper
  Cc: Xen-devel, Anthony PERARD, Stefano Stabellini, Michal Orzel,
	Doug Goldstein, Marek Marczykowski-Górecki

On Friday, April 11th, 2025 at 10:50 AM, Andrew Cooper <andrew.cooper3@citrix.com> wrote:

> 
> 
> On 11/04/2025 4:53 pm, Denis Mukhin wrote:
> 
> > On Friday, April 11th, 2025 at 3:52 AM, Andrew Cooper andrew.cooper3@citrix.com wrote:
> > 
> > > Rename the script as we're going to use it for ARM64 shortly, and have it take
> > > a tar or cpio parameter to determine the output format.
> > > 
> > > Turn it into a proper bash script, and provide the cpio form under the new
> > > artefact naming scheme.
> > > 
> > > No functional change.
> > > 
> > > Signed-off-by: Andrew Cooper andrew.cooper3@citrix.com
> > > 
> > > ---
> > > CC: Anthony PERARD anthony.perard@vates.tech
> > > 
> > > CC: Stefano Stabellini sstabellini@kernel.org
> > > 
> > > CC: Michal Orzel michal.orzel@amd.com
> > > 
> > > CC: Doug Goldstein cardoe@cardoe.com
> > > 
> > > CC: Marek Marczykowski-Górecki marmarek@invisiblethingslab.com
> > > 
> > > ---
> > > .gitlab-ci.yml | 9 +++++++-
> > > ...6_64-rootfs-alpine.sh => alpine-rootfs.sh} | 21 +++++++++++++++++--
> > > 
> > > 2 files changed, 27 insertions(+), 3 deletions(-)
> > > rename scripts/{x86_64-rootfs-alpine.sh => alpine-rootfs.sh} (75%)
> > > 
> > > diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
> > > index 1d2c72b268a3..916c5ae9d508 100644
> > > --- a/.gitlab-ci.yml
> > > +++ b/.gitlab-ci.yml
> > > @@ -42,6 +42,13 @@ linux-6.6.86-arm64:
> > > #
> > > # x86_64 artifacts
> > > #
> > > +alpine-3.18-x86_64-rootfs:
> > > + extends: .x86_64-artifacts
> > > + script:
> > > + - ./scripts/alpine-rootfs.sh cpio
> > > + variables:
> > > + CONTAINER: alpine:3.18-x86_64-base
> > > +
> > > linux-6.6.56-x86_64:
> > > extends: .x86_64-artifacts
> > > script: ./scripts/build-linux.sh
> > > @@ -62,7 +69,7 @@ x86_64-kernel-linux-6.6.56:
> > > x86_64-rootfs-alpine-3.18:
> > > extends: .x86_64-artifacts
> > > script:
> > > - - . scripts/x86_64-rootfs-alpine.sh
> > > + - ./scripts/alpine-rootfs.sh tar
> > > variables:
> > > CONTAINER: alpine:3.18-x86_64-base
> > > 
> > > diff --git a/scripts/x86_64-rootfs-alpine.sh b/scripts/alpine-rootfs.sh
> > > similarity index 75%
> > > rename from scripts/x86_64-rootfs-alpine.sh
> > > rename to scripts/alpine-rootfs.sh
> > > index b70b3a54ede5..c29c92d1c682 100755
> > > --- a/scripts/x86_64-rootfs-alpine.sh
> > > +++ b/scripts/alpine-rootfs.sh
> > > @@ -1,4 +1,9 @@
> > > +#!/bin/bash
> > > +
> > > +set -eu
> > > +
> > > WORKDIR="${PWD}"
> > > +COPYDIR="${WORKDIR}/binaries"
> > > 
> > > apk update
> > > 
> > > @@ -56,5 +61,17 @@ passwd -d "root" root
> > > 
> > > # Create rootfs
> > > cd /
> > > -tar cvzf "${WORKDIR}/binaries/initrd.tar.gz" \
> > > - bin dev etc home init lib mnt opt root sbin usr var
> > > +PATHS="bin dev etc home init lib mnt opt root sbin usr var"
> > > +
> > > +case $1 in
> > > + cpio)
> > > + find $PATHS | cpio -o -H newc | gzip > "${COPYDIR}/rootfs.cpio.gz"
> > > 
> > > +
> > > + # Print the contents for the build log
> > > + zcat "${COPYDIR}/rootfs.cpio.gz" | cpio -tv
> > > Maybe allow $1 to be a filename? E. g. rootfs.cpio.gz or initrd.tar.gz.
> > > This way $1 will contain an explicit output artifact format along with
> > > the output filename.
> > 
> > What do you think?
> > 
> > > + ;;
> > > +
> > > + tar)
> > > + tar cvzf "${COPYDIR}/initrd.tar.gz" $PATHS
> > > + ;;
> > > I would add the default case and error out in case of possible mis-use.
> 
> 
> This is bespoke tooling, used as part of Xen's GitlabCI infrastructure.
> 
> The parameter is going to last for all of about a week, until we can
> retire the tests using the tarball, because they're creating routine but
> intermittent test failure.

Oh, I see, thanks for the details!

> 
> I'm not looking to over-engineer the result.
> 
> ~Andrew


^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: [PATCH 1/3] Rework rootfs generation to make a cpio archive
  2025-04-11 10:52 ` [PATCH 1/3] Rework rootfs generation to make a cpio archive Andrew Cooper
                     ` (2 preceding siblings ...)
  2025-04-11 15:53   ` Denis Mukhin
@ 2025-04-11 18:36   ` Denis Mukhin
  3 siblings, 0 replies; 15+ messages in thread
From: Denis Mukhin @ 2025-04-11 18:36 UTC (permalink / raw)
  To: Andrew Cooper
  Cc: Xen-devel, Anthony PERARD, Stefano Stabellini, Michal Orzel,
	Doug Goldstein, Marek Marczykowski-Górecki

On Friday, April 11th, 2025 at 3:52 AM, Andrew Cooper <andrew.cooper3@citrix.com> wrote:

> 
> 
> Rename the script as we're going to use it for ARM64 shortly, and have it take
> a tar or cpio parameter to determine the output format.
> 
> Turn it into a proper bash script, and provide the cpio form under the new
> artefact naming scheme.
> 
> No functional change.
> 
> Signed-off-by: Andrew Cooper andrew.cooper3@citrix.com

Reviewed-by: Denis Mukhin <dmukhin@ford.com>

> 
> ---
> CC: Anthony PERARD anthony.perard@vates.tech
> 
> CC: Stefano Stabellini sstabellini@kernel.org
> 
> CC: Michal Orzel michal.orzel@amd.com
> 
> CC: Doug Goldstein cardoe@cardoe.com
> 
> CC: Marek Marczykowski-Górecki marmarek@invisiblethingslab.com
> 
> ---
> .gitlab-ci.yml | 9 +++++++-
> ...6_64-rootfs-alpine.sh => alpine-rootfs.sh} | 21 +++++++++++++++++--
> 
> 2 files changed, 27 insertions(+), 3 deletions(-)
> rename scripts/{x86_64-rootfs-alpine.sh => alpine-rootfs.sh} (75%)
> 
> 
> diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
> index 1d2c72b268a3..916c5ae9d508 100644
> --- a/.gitlab-ci.yml
> +++ b/.gitlab-ci.yml
> @@ -42,6 +42,13 @@ linux-6.6.86-arm64:
> #
> # x86_64 artifacts
> #
> +alpine-3.18-x86_64-rootfs:
> + extends: .x86_64-artifacts
> + script:
> + - ./scripts/alpine-rootfs.sh cpio
> + variables:
> + CONTAINER: alpine:3.18-x86_64-base
> +
> linux-6.6.56-x86_64:
> extends: .x86_64-artifacts
> script: ./scripts/build-linux.sh
> @@ -62,7 +69,7 @@ x86_64-kernel-linux-6.6.56:
> x86_64-rootfs-alpine-3.18:
> extends: .x86_64-artifacts
> script:
> - - . scripts/x86_64-rootfs-alpine.sh
> + - ./scripts/alpine-rootfs.sh tar
> variables:
> CONTAINER: alpine:3.18-x86_64-base
> 
> diff --git a/scripts/x86_64-rootfs-alpine.sh b/scripts/alpine-rootfs.sh
> similarity index 75%
> rename from scripts/x86_64-rootfs-alpine.sh
> rename to scripts/alpine-rootfs.sh
> index b70b3a54ede5..c29c92d1c682 100755
> --- a/scripts/x86_64-rootfs-alpine.sh
> +++ b/scripts/alpine-rootfs.sh
> @@ -1,4 +1,9 @@
> +#!/bin/bash
> +
> +set -eu
> +
> WORKDIR="${PWD}"
> +COPYDIR="${WORKDIR}/binaries"
> 
> apk update
> 
> @@ -56,5 +61,17 @@ passwd -d "root" root
> 
> # Create rootfs
> cd /
> -tar cvzf "${WORKDIR}/binaries/initrd.tar.gz" \
> - bin dev etc home init lib mnt opt root sbin usr var
> +PATHS="bin dev etc home init lib mnt opt root sbin usr var"
> +
> +case $1 in
> + cpio)
> + find $PATHS | cpio -o -H newc | gzip > "${COPYDIR}/rootfs.cpio.gz"
> 
> +
> + # Print the contents for the build log
> + zcat "${COPYDIR}/rootfs.cpio.gz" | cpio -tv
> + ;;
> +
> + tar)
> + tar cvzf "${COPYDIR}/initrd.tar.gz" $PATHS
> + ;;
> +esac
> --
> 2.39.5


^ permalink raw reply	[flat|nested] 15+ messages in thread

* [PATCH 2/3] Shrink the rootfs substantially
  2025-04-11 10:52 [PATCH TEST-ARTEFACTS 0/3] Rootfs generation (ARM and x86) Andrew Cooper
  2025-04-11 10:52 ` [PATCH 1/3] Rework rootfs generation to make a cpio archive Andrew Cooper
@ 2025-04-11 10:52 ` Andrew Cooper
  2025-04-11 12:35   ` Frediano Ziglio
  2025-04-11 10:52 ` [PATCH 3/3] Provide an ARM64 rootfs too Andrew Cooper
  2 siblings, 1 reply; 15+ messages in thread
From: Andrew Cooper @ 2025-04-11 10:52 UTC (permalink / raw)
  To: Xen-devel
  Cc: Andrew Cooper, Anthony PERARD, Stefano Stabellini, Michal Orzel,
	Doug Goldstein, Marek Marczykowski-Górecki

bash, busybox, musl and zlib are all in the base container.

python3 and ncurses are in principle used by bits of Xen, but not in anything
we test in CI.  argp-standlone, curl, dbus, libfdt, libgcc and sudo aren't
used at all (for x86 at least).

libbz2 and libuuid were pulled in transitively before, and need to be included
explicitly now.

Use apk --no-cache to avoid keeping a ~2M package index on disk.

Remove the modules scan on boot.  We don't have or build any.  This removes a
chunk of warnings on boot.

This shrinks the rootfs from ~30M down to ~8M.

No practical change.

Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
---
CC: Anthony PERARD <anthony.perard@vates.tech>
CC: Stefano Stabellini <sstabellini@kernel.org>
CC: Michal Orzel <michal.orzel@amd.com>
CC: Doug Goldstein <cardoe@cardoe.com>
CC: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com>

I'm still working on the integration of the new rootfs's int Xen, and will
hold off until committing that's ready but
https://gitlab.com/xen-project/hardware/xen-staging/-/pipelines/1763510164
shows good early progress.
---
 scripts/alpine-rootfs.sh | 60 +++++++++++++++++++++++-----------------
 1 file changed, 34 insertions(+), 26 deletions(-)

diff --git a/scripts/alpine-rootfs.sh b/scripts/alpine-rootfs.sh
index c29c92d1c682..30d426d6e88f 100755
--- a/scripts/alpine-rootfs.sh
+++ b/scripts/alpine-rootfs.sh
@@ -4,33 +4,42 @@ set -eu
 
 WORKDIR="${PWD}"
 COPYDIR="${WORKDIR}/binaries"
+UNAME=$(uname -m)
 
-apk update
+apk --no-cache update
 
-# xen runtime deps
-apk add musl
-apk add libgcc
-apk add openrc
-apk add busybox
-apk add sudo
-apk add dbus
-apk add bash
-apk add python3
-apk add zlib
-apk add lzo
-apk add ncurses
-apk add yajl
-apk add libaio
-apk add xz
-apk add util-linux
-apk add argp-standalone
-apk add libfdt
-apk add glib
-apk add pixman
-apk add curl
-apk add udev
-apk add pciutils
-apk add libelf
+PKGS=(
+    # System
+    openrc
+    udev
+    util-linux
+
+    # Xen toolstack runtime deps
+    libbz2
+    libuuid
+    lzo
+    xz
+    yajl
+
+    # QEMU
+    glib
+    libaio
+    pixman
+    )
+
+case $UNAME in
+    x86_64)
+        PKGS+=(
+            # System
+            pciutils
+
+            # QEMU
+            libelf
+            )
+        ;;
+esac
+
+apk add --no-cache "${PKGS[@]}"
 
 # Xen
 cd /
@@ -45,7 +54,6 @@ rc-update add dmesg sysinit
 rc-update add hostname boot
 rc-update add hwclock boot
 rc-update add hwdrivers sysinit
-rc-update add modules boot
 rc-update add killprocs shutdown
 rc-update add mount-ro shutdown
 rc-update add savecache shutdown
-- 
2.39.5



^ permalink raw reply related	[flat|nested] 15+ messages in thread

* Re: [PATCH 2/3] Shrink the rootfs substantially
  2025-04-11 10:52 ` [PATCH 2/3] Shrink the rootfs substantially Andrew Cooper
@ 2025-04-11 12:35   ` Frediano Ziglio
  0 siblings, 0 replies; 15+ messages in thread
From: Frediano Ziglio @ 2025-04-11 12:35 UTC (permalink / raw)
  To: Andrew Cooper
  Cc: Xen-devel, Anthony PERARD, Stefano Stabellini, Michal Orzel,
	Doug Goldstein, Marek Marczykowski-Górecki

On Fri, Apr 11, 2025 at 11:52 AM Andrew Cooper
<andrew.cooper3@citrix.com> wrote:
>
> bash, busybox, musl and zlib are all in the base container.
>
> python3 and ncurses are in principle used by bits of Xen, but not in anything
> we test in CI.  argp-standlone, curl, dbus, libfdt, libgcc and sudo aren't
> used at all (for x86 at least).
>
> libbz2 and libuuid were pulled in transitively before, and need to be included
> explicitly now.
>
> Use apk --no-cache to avoid keeping a ~2M package index on disk.
>
> Remove the modules scan on boot.  We don't have or build any.  This removes a
> chunk of warnings on boot.
>
> This shrinks the rootfs from ~30M down to ~8M.
>
> No practical change.
>
> Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
> ---
> CC: Anthony PERARD <anthony.perard@vates.tech>
> CC: Stefano Stabellini <sstabellini@kernel.org>
> CC: Michal Orzel <michal.orzel@amd.com>
> CC: Doug Goldstein <cardoe@cardoe.com>
> CC: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com>
>
> I'm still working on the integration of the new rootfs's int Xen, and will
> hold off until committing that's ready but
> https://gitlab.com/xen-project/hardware/xen-staging/-/pipelines/1763510164
> shows good early progress.
> ---
>  scripts/alpine-rootfs.sh | 60 +++++++++++++++++++++++-----------------
>  1 file changed, 34 insertions(+), 26 deletions(-)
>
> diff --git a/scripts/alpine-rootfs.sh b/scripts/alpine-rootfs.sh
> index c29c92d1c682..30d426d6e88f 100755
> --- a/scripts/alpine-rootfs.sh
> +++ b/scripts/alpine-rootfs.sh
> @@ -4,33 +4,42 @@ set -eu
>
>  WORKDIR="${PWD}"
>  COPYDIR="${WORKDIR}/binaries"
> +UNAME=$(uname -m)
>

Minor: maybe ARCH/MACHINE ?

> -apk update
> +apk --no-cache update
>
> -# xen runtime deps
> -apk add musl
> -apk add libgcc
> -apk add openrc
> -apk add busybox
> -apk add sudo
> -apk add dbus
> -apk add bash
> -apk add python3
> -apk add zlib
> -apk add lzo
> -apk add ncurses
> -apk add yajl
> -apk add libaio
> -apk add xz
> -apk add util-linux
> -apk add argp-standalone
> -apk add libfdt
> -apk add glib
> -apk add pixman
> -apk add curl
> -apk add udev
> -apk add pciutils
> -apk add libelf
> +PKGS=(
> +    # System
> +    openrc
> +    udev
> +    util-linux
> +
> +    # Xen toolstack runtime deps
> +    libbz2
> +    libuuid
> +    lzo
> +    xz
> +    yajl
> +
> +    # QEMU
> +    glib
> +    libaio
> +    pixman
> +    )

Minor: indent with "PKGS" ?

> +
> +case $UNAME in
> +    x86_64)
> +        PKGS+=(
> +            # System
> +            pciutils
> +
> +            # QEMU
> +            libelf
> +            )
> +        ;;
> +esac
> +
> +apk add --no-cache "${PKGS[@]}"
>
>  # Xen
>  cd /
> @@ -45,7 +54,6 @@ rc-update add dmesg sysinit
>  rc-update add hostname boot
>  rc-update add hwclock boot
>  rc-update add hwdrivers sysinit
> -rc-update add modules boot
>  rc-update add killprocs shutdown
>  rc-update add mount-ro shutdown
>  rc-update add savecache shutdown
> --
> 2.39.5
>
>

It looks good to me.

Frediano


^ permalink raw reply	[flat|nested] 15+ messages in thread

* [PATCH 3/3] Provide an ARM64 rootfs too
  2025-04-11 10:52 [PATCH TEST-ARTEFACTS 0/3] Rootfs generation (ARM and x86) Andrew Cooper
  2025-04-11 10:52 ` [PATCH 1/3] Rework rootfs generation to make a cpio archive Andrew Cooper
  2025-04-11 10:52 ` [PATCH 2/3] Shrink the rootfs substantially Andrew Cooper
@ 2025-04-11 10:52 ` Andrew Cooper
  2 siblings, 0 replies; 15+ messages in thread
From: Andrew Cooper @ 2025-04-11 10:52 UTC (permalink / raw)
  To: Xen-devel
  Cc: Andrew Cooper, Anthony PERARD, Stefano Stabellini, Michal Orzel,
	Doug Goldstein, Marek Marczykowski-Górecki

The only extra package needed is libfdt.

The older ARM64 rootfs configured modloop, networking and sysctl, but none of
this is used in any of the testing, so is omitted here for now.

Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
---
CC: Anthony PERARD <anthony.perard@vates.tech>
CC: Stefano Stabellini <sstabellini@kernel.org>
CC: Michal Orzel <michal.orzel@amd.com>
CC: Doug Goldstein <cardoe@cardoe.com>
CC: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com>
---
 .gitlab-ci.yml                           | 7 +++++++
 containerize                             | 1 +
 images/alpine/3.18-arm64-base.dockerfile | 6 ++++++
 scripts/alpine-rootfs.sh                 | 7 +++++++
 4 files changed, 21 insertions(+)
 create mode 100644 images/alpine/3.18-arm64-base.dockerfile

diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index 916c5ae9d508..3587d660aa62 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -33,6 +33,13 @@ stages:
 #
 # ARM64 artifacts
 #
+alpine-3.18-arm64-rootfs:
+  extends: .arm64-artifacts
+  script:
+    - ./scripts/alpine-rootfs.sh cpio
+  variables:
+    CONTAINER: alpine:3.18-arm64-base
+
 linux-6.6.86-arm64:
   extends: .arm64-artifacts
   script: ./scripts/build-linux.sh
diff --git a/containerize b/containerize
index c23f55ead737..38a434ab7075 100755
--- a/containerize
+++ b/containerize
@@ -24,6 +24,7 @@ die() {
 #
 BASE="registry.gitlab.com/xen-project/hardware/test-artifacts"
 case "_${CONTAINER}" in
+    _alpine-3.18-arm64-base) CONTAINER="${BASE}/alpine:3.18-arm64-base" ;;
     _alpine-3.18-arm64-build) CONTAINER="${BASE}/alpine:3.18-arm64-build" ;;
     _alpine-3.18-x86_64-base) CONTAINER="${BASE}/alpine:3.18-x86_64-base" ;;
     _alpine-3.18-x86_64-build|_) CONTAINER="${BASE}/alpine:3.18-x86_64-build" ;;
diff --git a/images/alpine/3.18-arm64-base.dockerfile b/images/alpine/3.18-arm64-base.dockerfile
new file mode 100644
index 000000000000..ab597c75b80e
--- /dev/null
+++ b/images/alpine/3.18-arm64-base.dockerfile
@@ -0,0 +1,6 @@
+# syntax=docker/dockerfile:1
+FROM --platform=linux/arm64/v8 alpine:3.18
+LABEL maintainer.name="The Xen Project"
+LABEL maintainer.email="xen-devel@lists.xenproject.org"
+
+RUN apk --no-cache add bash
diff --git a/scripts/alpine-rootfs.sh b/scripts/alpine-rootfs.sh
index 30d426d6e88f..220a8aedf4a1 100755
--- a/scripts/alpine-rootfs.sh
+++ b/scripts/alpine-rootfs.sh
@@ -37,6 +37,13 @@ case $UNAME in
             libelf
             )
         ;;
+
+    aarch64)
+        PKGS+=(
+            # Xen
+            libfdt
+            )
+        ;;
 esac
 
 apk add --no-cache "${PKGS[@]}"
-- 
2.39.5



^ permalink raw reply related	[flat|nested] 15+ messages in thread

end of thread, other threads:[~2025-04-11 18:37 UTC | newest]

Thread overview: 15+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2025-04-11 10:52 [PATCH TEST-ARTEFACTS 0/3] Rootfs generation (ARM and x86) Andrew Cooper
2025-04-11 10:52 ` [PATCH 1/3] Rework rootfs generation to make a cpio archive Andrew Cooper
2025-04-11 11:06   ` Marek Marczykowski-Górecki
2025-04-11 11:09     ` Andrew Cooper
2025-04-11 11:33       ` Andrew Cooper
2025-04-11 11:47         ` Marek Marczykowski-Górecki
2025-04-11 12:25   ` Frediano Ziglio
2025-04-11 12:39     ` Andrew Cooper
2025-04-11 15:53   ` Denis Mukhin
2025-04-11 17:50     ` Andrew Cooper
2025-04-11 18:26       ` Denis Mukhin
2025-04-11 18:36   ` Denis Mukhin
2025-04-11 10:52 ` [PATCH 2/3] Shrink the rootfs substantially Andrew Cooper
2025-04-11 12:35   ` Frediano Ziglio
2025-04-11 10:52 ` [PATCH 3/3] Provide an ARM64 rootfs too Andrew Cooper

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.