Re: [PATCH v1 01/11] ci: prevent grub unpacking initramfs

["Marek Marczykowski-Górecki" <marmarek@invisiblethingslab.com>]

[-- Attachment #1: Type: text/plain, Size: 3014 bytes --]

On Thu, Apr 03, 2025 at 12:39:11PM +0100, Andrew Cooper wrote:
> On 03/04/2025 12:29 pm, Jan Beulich wrote:
> > On 03.04.2025 13:04, Marek Marczykowski-Górecki wrote:
> >> It fails on larger initramfs (~250MB one) and sometimes even smaller
> >> depending on memory size/memory map, let Linux do it.
> > Iirc grub only unpacks gzip-ed modules, so wouldn't a yet better approach
> > be to use a better compressing algorithm, which simply as a side effect
> > would keep grub from decompressing it, while at the same time moving
> > farther away from any critical boundaries?
> 
> Yes and no.
> 
> This is going to change anyway when I (or a delgee) moves initrd
> generation from the test step itself into the test artefacts repo.

Out of curiosity, I tried this:
https://gitlab.com/xen-project/people/marmarek/xen/-/commit/51b5fde816806206e6e0c72ec0146c5ab3da4351
pipeline:
https://gitlab.com/xen-project/people/marmarek/xen/-/pipelines/1753304751

It avoids repacking base alpine initrd, but it still needs to pack
per-job stuff (test script, but also toolstack). The gains are not
great, that step still takes over a minute. The overall job time went
down by just 40-50s (for example from 6m50s to 6m03s). This isn't very
surprising, because the base initrd.cpio.gz is 60MB and the final one is
over 200MB, so the part that was needlessly re-compressed is about 30%.
The numbers checks out, as the (debug) build artifacts zip is about
150MB.

One further idea would be to produce cpio.gz archive as part of the
build artifact, to avoid zip (used by gitlab for artifacts) -> cpio.gz
recompression. But that either makes build artifacts bigger for every
job, or need setting a variable for jobs that are used later for tests.
Unless we decide to store toolstack _only_ as cpio.gz in artifacts?

I tried the latter idea here:
https://gitlab.com/xen-project/people/marmarek/xen/-/commit/a4774213d83ca05c0b0b0fff80e49cf3aa32b67b
pipeline:
https://gitlab.com/xen-project/people/marmarek/xen/-/pipelines/1753312631

This helped quite a bit more. The 1m of compressing initrd is gone now.
Compressing remaining test-specific files and concatenating it all
together takes 2-3s. Now that 6m50s job is down to 3m40s. I'm not
exactly sure where that extra minute went (maybe related to number of
files in the build artifact? Fetching/extracting artifacts seems to be
faster a bit), but those times are not very consistent anyway, so it
could be also just random fluctuation. 

Looking further at the test log, some time is used for fetching
artifacts. I've looked into gitlab docs if those can be cached on the
runner, but I haven't found a solution that would really help (you can
cache arbitrary paths, including artifacts, but it doesn't prevent
re-downloading them and unpacking over the cached version...).
So, I'll stop here, polish the above patches and submit them in v2 of
this series.

-- 
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]