[RFC PATCH] gitlab: aggressively avoid extra GIT data

[RFC PATCH] gitlab: aggressively avoid extra GIT data

[Alex Bennée]

This avoids fetching blobs and tree references for branches we are not
going to worry about. Also skip tag references which are similarly not
useful and keep the default --prune. This keeps the .git data to
around 100M rather than the ~400M even a shallow clone takes.

So we can check the savings we also run a quick du while setting up
the build.

We also have to have special settings of GIT_FETCH_EXTRA_FLAGS for the
Windows build (git too old?) and the migration legacy test where we
build an older QEMU alongside the main one.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
---
 .gitlab-ci.d/base.yml               | 4 ++++
 .gitlab-ci.d/buildtest-template.yml | 1 +
 .gitlab-ci.d/buildtest.yml          | 2 ++
 .gitlab-ci.d/windows.yml            | 2 ++
 4 files changed, 9 insertions(+)

diff --git a/.gitlab-ci.d/base.yml b/.gitlab-ci.d/base.yml
index ef173a34e6..7cb800fc3b 100644
--- a/.gitlab-ci.d/base.yml
+++ b/.gitlab-ci.d/base.yml
@@ -24,6 +24,10 @@ variables:
     # Each script line from will be in a collapsible section in the job output
     # and show the duration of each line.
     FF_SCRIPT_SECTIONS: 1
+    # The project has a fairly fat GIT repo so we try and avoid bringing in things
+    # we don't need. The --filter options avoid blobs and tree references we aren't going to use
+    # and we also avoid fetching tags.
+    GIT_FETCH_EXTRA_FLAGS: --filter=blob:none --filter=tree:0 --no-tags --prune
 
   interruptible: true
 
diff --git a/.gitlab-ci.d/buildtest-template.yml b/.gitlab-ci.d/buildtest-template.yml
index 4fbfeb6667..22045add80 100644
--- a/.gitlab-ci.d/buildtest-template.yml
+++ b/.gitlab-ci.d/buildtest-template.yml
@@ -14,6 +14,7 @@
     - export CCACHE_DIR="$CCACHE_BASEDIR/ccache"
     - export CCACHE_MAXSIZE="500M"
     - export PATH="$CCACHE_WRAPPERSDIR:$PATH"
+    - du -sh .git
     - mkdir build
     - cd build
     - ccache --zero-stats
diff --git a/.gitlab-ci.d/buildtest.yml b/.gitlab-ci.d/buildtest.yml
index a1c030337b..def469a9dc 100644
--- a/.gitlab-ci.d/buildtest.yml
+++ b/.gitlab-ci.d/buildtest.yml
@@ -187,6 +187,8 @@ build-previous-qemu:
   variables:
     IMAGE: opensuse-leap
     TARGETS: x86_64-softmmu aarch64-softmmu
+    # Override the default flags as we need more to grab the old version
+    GIT_FETCH_EXTRA_FLAGS: --prune
   before_script:
     - export QEMU_PREV_VERSION="$(sed 's/\([0-9.]*\)\.[0-9]*/v\1.0/' VERSION)"
     - git remote add upstream https://gitlab.com/qemu-project/qemu
diff --git a/.gitlab-ci.d/windows.yml b/.gitlab-ci.d/windows.yml
index f116b8012d..f9da745042 100644
--- a/.gitlab-ci.d/windows.yml
+++ b/.gitlab-ci.d/windows.yml
@@ -28,6 +28,8 @@ msys2-64bit:
     # qTests don't run successfully with "--without-default-devices",
     # so let's exclude the qtests from CI for now.
     TEST_ARGS: --no-suite qtest
+    # The Windows git is a bit older so override the default
+    GIT_FETCH_EXTRA_FLAGS: --no-tags --prune
   artifacts:
     name: "$CI_JOB_NAME-$CI_COMMIT_REF_SLUG"
     expire_in: 7 days
-- 
2.39.2

Re: [RFC PATCH] gitlab: aggressively avoid extra GIT data

[Thomas Huth]

On 08/03/2024 16.07, Alex Bennée wrote:
> This avoids fetching blobs and tree references for branches we are not
> going to worry about. Also skip tag references which are similarly not
> useful and keep the default --prune. This keeps the .git data to
> around 100M rather than the ~400M even a shallow clone takes.
> 
> So we can check the savings we also run a quick du while setting up
> the build.
> 
> We also have to have special settings of GIT_FETCH_EXTRA_FLAGS for the
> Windows build (git too old?) and the migration legacy test where we
> build an older QEMU alongside the main one.
> 
> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
> ---
>   .gitlab-ci.d/base.yml               | 4 ++++
>   .gitlab-ci.d/buildtest-template.yml | 1 +
>   .gitlab-ci.d/buildtest.yml          | 2 ++
>   .gitlab-ci.d/windows.yml            | 2 ++
>   4 files changed, 9 insertions(+)
> 
> diff --git a/.gitlab-ci.d/base.yml b/.gitlab-ci.d/base.yml
> index ef173a34e6..7cb800fc3b 100644
> --- a/.gitlab-ci.d/base.yml
> +++ b/.gitlab-ci.d/base.yml
> @@ -24,6 +24,10 @@ variables:
>       # Each script line from will be in a collapsible section in the job output
>       # and show the duration of each line.
>       FF_SCRIPT_SECTIONS: 1
> +    # The project has a fairly fat GIT repo so we try and avoid bringing in things
> +    # we don't need. The --filter options avoid blobs and tree references we aren't going to use
> +    # and we also avoid fetching tags.
> +    GIT_FETCH_EXTRA_FLAGS: --filter=blob:none --filter=tree:0 --no-tags --prune

The original settings included --quiet according to the docs:

 
https://docs.gitlab.com/ee/ci/runners/configure_runners.html#git-fetch-extra-flags

Should we include --quite here, too?

  Thomas