[PATCH] gitlab: purge msys pacman cache

[PATCH] gitlab: purge msys pacman cache

[Daniel P. Berrangé]

For the Windows msys2 CI job we install many packages using pacman
and use the GitLab cache to preserve the pacman cache across CI
runs. While metadata still needs downloading, this avoids pacman
re-downloading packages from msys2 if they have not changed.

The problem is that pacman never automatically purges anything
from its package cache. Thus the GitLab cache is growing without
bound and packing/unpacking the cache is consuming an increasing
amount of time in the CI job.

If we run 'pacman -Sc' /after/ installing our desired package set,
it will purge any cached downloaded packages that are not matching
any installed package.

This will (currently) cap the pacman download cache at approx
256 MB.

Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
---
 .gitlab-ci.d/windows.yml | 1 +
 1 file changed, 1 insertion(+)

See a test job with this change, plus a find across the msys
pacman cache, showing the cleanup effects....

Before cleanup:

  https://gitlab.com/berrange/qemu/-/jobs/11679136531#L34

After cleanup:

  https://gitlab.com/berrange/qemu/-/jobs/11679136531#L1126

diff --git a/.gitlab-ci.d/windows.yml b/.gitlab-ci.d/windows.yml
index 1e6a01bd9a..6e1135d8b8 100644
--- a/.gitlab-ci.d/windows.yml
+++ b/.gitlab-ci.d/windows.yml
@@ -87,6 +87,7 @@ msys2-64bit:
       mingw-w64-x86_64-pkgconf
       mingw-w64-x86_64-python
       mingw-w64-x86_64-zstd"
+  - .\msys64\usr\bin\bash -lc "pacman -Sc --noconfirm"
   - Write-Output "Running build at $(Get-Date -Format u)"
   - $env:JOBS = $(.\msys64\usr\bin\bash -lc nproc)
   - $env:CHERE_INVOKING = 'yes'  # Preserve the current working directory
-- 
2.50.1

Re: [PATCH] gitlab: purge msys pacman cache

[Peter Maydell]

On Fri, 10 Oct 2025 at 17:05, Daniel P. Berrangé <berrange@redhat.com> wrote:
>
> For the Windows msys2 CI job we install many packages using pacman
> and use the GitLab cache to preserve the pacman cache across CI
> runs. While metadata still needs downloading, this avoids pacman
> re-downloading packages from msys2 if they have not changed.
>
> The problem is that pacman never automatically purges anything
> from its package cache. Thus the GitLab cache is growing without
> bound and packing/unpacking the cache is consuming an increasing
> amount of time in the CI job.
>
> If we run 'pacman -Sc' /after/ installing our desired package set,
> it will purge any cached downloaded packages that are not matching
> any installed package.
>
> This will (currently) cap the pacman download cache at approx
> 256 MB.
>
> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
> ---
>  .gitlab-ci.d/windows.yml | 1 +
>  1 file changed, 1 insertion(+)
>
> See a test job with this change, plus a find across the msys
> pacman cache, showing the cleanup effects....
>
> Before cleanup:
>
>   https://gitlab.com/berrange/qemu/-/jobs/11679136531#L34
>
> After cleanup:
>
>   https://gitlab.com/berrange/qemu/-/jobs/11679136531#L1126
>
> diff --git a/.gitlab-ci.d/windows.yml b/.gitlab-ci.d/windows.yml
> index 1e6a01bd9a..6e1135d8b8 100644
> --- a/.gitlab-ci.d/windows.yml
> +++ b/.gitlab-ci.d/windows.yml
> @@ -87,6 +87,7 @@ msys2-64bit:
>        mingw-w64-x86_64-pkgconf
>        mingw-w64-x86_64-python
>        mingw-w64-x86_64-zstd"
> +  - .\msys64\usr\bin\bash -lc "pacman -Sc --noconfirm"
>    - Write-Output "Running build at $(Get-Date -Format u)"
>    - $env:JOBS = $(.\msys64\usr\bin\bash -lc nproc)
>    - $env:CHERE_INVOKING = 'yes'  # Preserve the current working directory
> --

Reviewed-by: Peter Maydell <peter.maydell@linaro.org>

Hopefully this will make the CI job take less time (for the
record, currently in the main CI run it takes about 70 minutes,
of which 10 minutes is "unpack the cache" at the start and
20 minutes is "repack the cache" at the end).

thanks
-- PMM

Re: [PATCH] gitlab: purge msys pacman cache

[Daniel P. Berrangé]

On Fri, Oct 10, 2025 at 06:23:22PM +0100, Peter Maydell wrote:
> On Fri, 10 Oct 2025 at 17:05, Daniel P. Berrangé <berrange@redhat.com> wrote:
> >
> > For the Windows msys2 CI job we install many packages using pacman
> > and use the GitLab cache to preserve the pacman cache across CI
> > runs. While metadata still needs downloading, this avoids pacman
> > re-downloading packages from msys2 if they have not changed.
> >
> > The problem is that pacman never automatically purges anything
> > from its package cache. Thus the GitLab cache is growing without
> > bound and packing/unpacking the cache is consuming an increasing
> > amount of time in the CI job.
> >
> > If we run 'pacman -Sc' /after/ installing our desired package set,
> > it will purge any cached downloaded packages that are not matching
> > any installed package.
> >
> > This will (currently) cap the pacman download cache at approx
> > 256 MB.
> >
> > Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
> > ---
> >  .gitlab-ci.d/windows.yml | 1 +
> >  1 file changed, 1 insertion(+)
> >
> > See a test job with this change, plus a find across the msys
> > pacman cache, showing the cleanup effects....
> >
> > Before cleanup:
> >
> >   https://gitlab.com/berrange/qemu/-/jobs/11679136531#L34
> >
> > After cleanup:
> >
> >   https://gitlab.com/berrange/qemu/-/jobs/11679136531#L1126
> >
> > diff --git a/.gitlab-ci.d/windows.yml b/.gitlab-ci.d/windows.yml
> > index 1e6a01bd9a..6e1135d8b8 100644
> > --- a/.gitlab-ci.d/windows.yml
> > +++ b/.gitlab-ci.d/windows.yml
> > @@ -87,6 +87,7 @@ msys2-64bit:
> >        mingw-w64-x86_64-pkgconf
> >        mingw-w64-x86_64-python
> >        mingw-w64-x86_64-zstd"
> > +  - .\msys64\usr\bin\bash -lc "pacman -Sc --noconfirm"
> >    - Write-Output "Running build at $(Get-Date -Format u)"
> >    - $env:JOBS = $(.\msys64\usr\bin\bash -lc nproc)
> >    - $env:CHERE_INVOKING = 'yes'  # Preserve the current working directory
> > --
> 
> Reviewed-by: Peter Maydell <peter.maydell@linaro.org>
> 
> Hopefully this will make the CI job take less time (for the
> record, currently in the main CI run it takes about 70 minutes,
> of which 10 minutes is "unpack the cache" at the start and
> 20 minutes is "repack the cache" at the end).

Based on how long it takes to pack/unpack the cache in my job, I'd
be expecting approx a 25 minute win.


With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [PATCH] gitlab: purge msys pacman cache

[Thomas Huth]

On 10/10/2025 18.05, Daniel P. Berrangé wrote:
> For the Windows msys2 CI job we install many packages using pacman
> and use the GitLab cache to preserve the pacman cache across CI
> runs. While metadata still needs downloading, this avoids pacman
> re-downloading packages from msys2 if they have not changed.
> 
> The problem is that pacman never automatically purges anything
> from its package cache. Thus the GitLab cache is growing without
> bound and packing/unpacking the cache is consuming an increasing
> amount of time in the CI job.
> 
> If we run 'pacman -Sc' /after/ installing our desired package set,
> it will purge any cached downloaded packages that are not matching
> any installed package.
> 
> This will (currently) cap the pacman download cache at approx
> 256 MB.
> 
> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
> ---
>   .gitlab-ci.d/windows.yml | 1 +
>   1 file changed, 1 insertion(+)
> 
> See a test job with this change, plus a find across the msys
> pacman cache, showing the cleanup effects....
> 
> Before cleanup:
> 
>    https://gitlab.com/berrange/qemu/-/jobs/11679136531#L34
> 
> After cleanup:
> 
>    https://gitlab.com/berrange/qemu/-/jobs/11679136531#L1126
> 
> diff --git a/.gitlab-ci.d/windows.yml b/.gitlab-ci.d/windows.yml
> index 1e6a01bd9a..6e1135d8b8 100644
> --- a/.gitlab-ci.d/windows.yml
> +++ b/.gitlab-ci.d/windows.yml
> @@ -87,6 +87,7 @@ msys2-64bit:
>         mingw-w64-x86_64-pkgconf
>         mingw-w64-x86_64-python
>         mingw-w64-x86_64-zstd"
> +  - .\msys64\usr\bin\bash -lc "pacman -Sc --noconfirm"
>     - Write-Output "Running build at $(Get-Date -Format u)"
>     - $env:JOBS = $(.\msys64\usr\bin\bash -lc nproc)
>     - $env:CHERE_INVOKING = 'yes'  # Preserve the current working directory

Reviewed-by: Thomas Huth <thuth@redhat.com>

I just tested this with some additional "ls -lR /var/cache/pacman" added, 
and it seems to work as expected:

  https://gitlab.com/thuth/qemu/-/jobs/11680562487#L293

This just removed an old "mingw-w64-x86_64-python-3.12.11-4-any.pkg.tar.zst" 
package, but kept the newer 3.12.12 package (and all other currently 
installed packages) around. Thus also:

Tested-by: Thomas Huth <thuth@redhat.com>

Re: [PATCH] gitlab: purge msys pacman cache

[罗勇刚(Yonggang Luo)]

On Sat, Oct 11, 2025 at 12:05 AM Daniel P. Berrangé <berrange@redhat.com> wrote:
>
> For the Windows msys2 CI job we install many packages using pacman
> and use the GitLab cache to preserve the pacman cache across CI
> runs. While metadata still needs downloading, this avoids pacman
> re-downloading packages from msys2 if they have not changed.
>
> The problem is that pacman never automatically purges anything
> from its package cache. Thus the GitLab cache is growing without
> bound and packing/unpacking the cache is consuming an increasing
> amount of time in the CI job.
>
> If we run 'pacman -Sc' /after/ installing our desired package set,
> it will purge any cached downloaded packages that are not matching
> any installed package.
>
> This will (currently) cap the pacman download cache at approx
> 256 MB.
>
> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
> ---
>  .gitlab-ci.d/windows.yml | 1 +
>  1 file changed, 1 insertion(+)
>
> See a test job with this change, plus a find across the msys
> pacman cache, showing the cleanup effects....
>
> Before cleanup:
>
>   https://gitlab.com/berrange/qemu/-/jobs/11679136531#L34
>
> After cleanup:
>
>   https://gitlab.com/berrange/qemu/-/jobs/11679136531#L1126
>
> diff --git a/.gitlab-ci.d/windows.yml b/.gitlab-ci.d/windows.yml
> index 1e6a01bd9a..6e1135d8b8 100644
> --- a/.gitlab-ci.d/windows.yml
> +++ b/.gitlab-ci.d/windows.yml
> @@ -87,6 +87,7 @@ msys2-64bit:
>        mingw-w64-x86_64-pkgconf
>        mingw-w64-x86_64-python
>        mingw-w64-x86_64-zstd"
> +  - .\msys64\usr\bin\bash -lc "pacman -Sc --noconfirm"
>    - Write-Output "Running build at $(Get-Date -Format u)"
>    - $env:JOBS = $(.\msys64\usr\bin\bash -lc nproc)
>    - $env:CHERE_INVOKING = 'yes'  # Preserve the current working directory
> --
> 2.50.1
>

Reviewed-by: Yonggang Luo <luoyonggang@gmail.com>

-- 
         此致
礼
罗勇刚
Yours
    sincerely,
Yonggang Luo