[PATCH 0/7] ci: some speedups for slow tests

[PATCH 0/7] ci: some speedups for slow tests

[Patrick Steinhardt]

Hi,

I was staring at CI way more often recently, and that got me annoyed
real fast because it's just painstakingly slow. So I decided to have a
look and try do something about it, the result of which is this patch
series.

The series contains a couple of bits that are specific to GitLab, and a
couple of fixes that are generic. Some highlights:

  - Installing the Git for Windows SDK speeds up from 3 minutes to 10
    seconds by using tar(1) instead of Extract-Archive.

  - t0027 speeds up from >10 minutes to 6 minutes on Windows by avoiding
    the overhead of spawning redundant processes.

  - t5551 and t5559 both speed up by a factor of 10 on macOS, going from
    20 minutes each to 2 minutes each by using reftables in a repo with
    100k refs. "osx-meson" and "osx-clang" thus go from 29 minutes to 18
    minutes end-to-end.

  - "fedora-breaking-changes-meson" speeds up by a factor of 2, going
    from 35 minutes to 16 minutes. Turns out our SHA256 implementation
    is quite slow with "-O0".

Unfortunately, our Windows runners at GitLab are still atrociously slow
by themselves. So the critical path only improves by about ~10 minutes,
going from 60-70 minutes to 53 minutes end-to-end. But hey, at least the
other jobs now give feedback a bit faster. And I guess on GitHub it may
be a different story because Windows runners are way better there and
are typically not on the critical path.

A test run can be found at [1].

Thanks!

Patrick

[1]: https://gitlab.com/gitlab-org/git/-/merge_requests/734

---
Patrick Steinhardt (7):
      gitlab-ci: speed up installation of Git for Windows SDK
      gitlab-ci: exclude intermediate outputs from Windows artifacts
      ci: build optimized binaries with Meson
      t0027: optimize how we reset repository state
      t0027: cache expected output files
      t1092: set up repositories from templates
      t5551: use reftables for repositories with many refs

 .gitlab-ci.yml                           | 13 ++++++++++--
 ci/install-sdk.ps1                       | 26 ++++++++++++++++++------
 ci/run-build-and-tests.sh                |  1 +
 t/t0027-auto-crlf.sh                     | 33 +++++++++++++++++--------------
 t/t1092-sparse-checkout-compatibility.sh | 34 +++++++++++++++++++-------------
 t/t5551-http-fetch-smart.sh              |  6 ++++--
 6 files changed, 74 insertions(+), 39 deletions(-)


---
base-commit: 0ffa3e35c7c82ab8d0de226149fb03b3f7674d05
change-id: 20261106-pks-gitlab-ci-speedups-09c3841eb178

[PATCH 1/7] gitlab-ci: speed up installation of Git for Windows SDK

[Patrick Steinhardt]

The "build:mingw64" job on GitLab CI downloads the minimal Git for
Windows SDK on every run and then extracts it via `Expand-Archive`,
which is expensive.

Improve this by caching the SDK and by using tar(1) instead of
`Expand-Archive`. Note that caching actually only gives us a win of a
couple seconds. The bigger win here is using tar(1), which speeds up
extracting the archive from 3 minutes to about 10 seconds. Regardless of
that though, caching the SDK feels like a sensible thing to do as it
removes one step that may spuriously fail.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 .gitlab-ci.yml     |  9 ++++++++-
 ci/install-sdk.ps1 | 26 ++++++++++++++++++++------
 2 files changed, 28 insertions(+), 7 deletions(-)

diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index 9b7698a0037..15419a954b3 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -127,9 +127,16 @@ build:mingw64:
     - saas-windows-medium-amd64
   variables:
     NO_PERL: 1
+  cache:
+    key:
+      files:
+        - ci/install-sdk.ps1
+        - ci/install-dependencies.ps1
+    paths:
+      - .dependencies
   before_script:
     - *windows_before_script
-    - ./ci/install-sdk.ps1 -directory "git-sdk"
+    - ./ci/install-sdk.ps1 -Directory "git-sdk"
     - ./ci/install-dependencies.ps1 -Mingw
   script:
     # The minimal SDK's profile resets PATH.
diff --git a/ci/install-sdk.ps1 b/ci/install-sdk.ps1
index 66f24838a40..1dbcef04a18 100755
--- a/ci/install-sdk.ps1
+++ b/ci/install-sdk.ps1
@@ -1,12 +1,26 @@
 param(
-    [string]$directory='git-sdk',
-    [string]$url='https://github.com/git-for-windows/git-sdk-64/releases/download/ci-artifacts/git-sdk-x86_64-minimal.zip'
+    [string]$Directory='git-sdk',
+    [string]$URL='https://github.com/git-for-windows/git-sdk-64/releases/download/ci-artifacts/git-sdk-x86_64-minimal.zip',
+    [string]$DownloadDirectory='.dependencies'
 )
 
-Invoke-WebRequest "$url" -OutFile git-sdk.zip
-Expand-Archive -LiteralPath git-sdk.zip -DestinationPath "$directory"
-Remove-Item -Path git-sdk.zip
+$ErrorActionPreference = 'Stop'
+$ProgressPreference = 'SilentlyContinue'
+
+New-Item -Path $DownloadDirectory -ItemType Directory -Force
+
+$archive = Join-Path $DownloadDirectory 'git-sdk.zip'
+if (-not (Test-Path $archive)) {
+    Invoke-WebRequest "$URL" -OutFile $archive -TimeoutSec 300
+}
+
+New-Item -Path $Directory -ItemType Directory -Force
+tar -xf $archive -C $Directory
+if ($LASTEXITCODE -ne 0) {
+    throw "Failed to extract $archive with exit code $LASTEXITCODE"
+}
 
 New-Item -Path .git/info -ItemType Directory -Force
 New-Item -Path .git/info/exclude -ItemType File -Force
-Add-Content -Path .git/info/exclude -Value "/$directory"
+Add-Content -Path .git/info/exclude -Value "/$Directory"
+Add-Content -Path .git/info/exclude -Value "/$DownloadDirectory"

-- 
2.56.0.500.g5d06dc02cd.dirty

[PATCH 2/7] gitlab-ci: exclude intermediate outputs from Windows artifacts

[Patrick Steinhardt]

As running tests is very slow on Windows we shard our CI tests across 10
runners. To avoid rebuilding Git on each of those runners we have a
preliminary step which builds Git and then passes on the build artifacts
to the test jobs.

This archive contains a ton of intermediate "*.obj" build artifacts that
aren't needed by the test jobs: the artifact contains ~2300 files, out
of which ~1800 are "*.obj" files.

Exclude those files from the build artifact, which reduces the time it
takes to download and extract them from 34 seconds to 17 seconds.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 .gitlab-ci.yml | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index 15419a954b3..036d2f3c7f2 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -194,6 +194,8 @@ build:msvc-meson:
   artifacts:
     paths:
       - build
+    exclude:
+      - build/**/*.obj
 
 test:msvc-meson:
   extends: .msvc-meson

-- 
2.56.0.500.g5d06dc02cd.dirty

[PATCH 3/7] ci: build optimized binaries with Meson

[Patrick Steinhardt]

When taking a look at our CI jobs one will quickly notice that across
the Linux-based jobs, the "fedora-breaking-changes-meson" job is running
significantly longer than any of the other jobs. While most Linux jobs
only run around 15 minutes, that particular job runs roughly twice as
long.

The root cause of this is a combination of two different things:

  - The job uses Meson to compile Git, and Meson by default uses the
    "debug" build type that translates into "-O0". Our Makefile in
    contrast to that uses "-g -O2" by default.

  - The job enables breaking changes and thus runs with SHA256 by
    default with the "block" backend, which benefits heavily from
    compiler optimizations. For example, hashing a 500MB blob with
    SHA256 takes 9.6 seconds with "-O0", whereas it takes only 1.5
    seconds with "-O2". The SHA1 code path only slows down from 0.9
    seconds to 2.1 seconds with "-O0".

Of course, compiling without optimizations also has an impact on all the
other jobs, but the Fedora job is by far the most impacted one.

Speed up executing the tests by using the "debugoptimized" build type
instead. This reduces the runtime of "fedora-breaking-changes-meson"
from 2100 to 1000 seconds. Other Linux- and macOS-based jobs benefit as
well, but they only speed up by around ~10%.

On Windows though this is a bit more muddy, as the build step gets
almost two minutes slower, whereas tests only run ~10% faster. Combined
though the total run time is roughly the same before and after this
change.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 .gitlab-ci.yml            | 2 +-
 ci/run-build-and-tests.sh | 1 +
 2 files changed, 2 insertions(+), 1 deletion(-)

diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index 036d2f3c7f2..df82cd3a5df 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -189,7 +189,7 @@ build:msvc-meson:
   extends: .msvc-meson
   stage: build
   script:
-    - meson setup build --vsenv -Dperl=disabled -Dbackend_max_links=1 -Dcredential_helpers=wincred -Dtest_output_directory="$TEST_OUTPUT_DIRECTORY"
+    - meson setup build --vsenv --buildtype=debugoptimized -Dperl=disabled -Dbackend_max_links=1 -Dcredential_helpers=wincred -Dtest_output_directory="$TEST_OUTPUT_DIRECTORY"
     - meson compile -C build
   artifacts:
     paths:
diff --git a/ci/run-build-and-tests.sh b/ci/run-build-and-tests.sh
index 68b91e31439..fd8136af342 100755
--- a/ci/run-build-and-tests.sh
+++ b/ci/run-build-and-tests.sh
@@ -55,6 +55,7 @@ case "$jobname" in
 		--fatal-meson-warnings \
 		--warnlevel 2 --werror \
 		--wrap-mode nofallback \
+		--buildtype=debugoptimized \
 		-Dfuzzers=true \
 		-Dtest_output_directory="${TEST_OUTPUT_DIRECTORY:-$(pwd)/t}" \
 		$MESONFLAGS

-- 
2.56.0.500.g5d06dc02cd.dirty

[PATCH 4/7] t0027: optimize how we reset repository state

[Patrick Steinhardt]

In t0027 we verify our logic to convert CRLF for all kinds of
combinations of attributes and configuration, which ultimately results
in 2600 test cases. This is of course on the heavier side, so on Windows
this takes more than 10 minutes to run.

One source of slowness is `checkout_files ()`, which gets called for
a lot of our test to set up the initial working directory. As part of
that we both remove the old files that we care about and then check them
out again via git-checkout(1), but we do this one by one for every
single file.

This is of course rather wasteful, as we have to spawn ten different
processes every time the function is invoked. And on top of that, we
also have to rewrite the complete index in git-checkout(1) every time.

Speed this up by instead using a single invocation of rm(1) and
git-checkout(1). This reduces the runtime by about 20% on all platforms,
and on Windows this reduces the runtime by ~2.5 minutes.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 t/t0027-auto-crlf.sh | 16 +++++++++-------
 1 file changed, 9 insertions(+), 7 deletions(-)

diff --git a/t/t0027-auto-crlf.sh b/t/t0027-auto-crlf.sh
index 49dbf09da77..eefb346d11f 100755
--- a/t/t0027-auto-crlf.sh
+++ b/t/t0027-auto-crlf.sh
@@ -303,13 +303,15 @@ checkout_files () {
 		git config core.autocrlf $crlf
 	'
 	pfx=eol_${ceol}_crlf_${crlf}_attr_${attr}_ &&
-	for f in LF CRLF LF_mix_CR CRLF_mix_LF LF_nul
-	do
-		test_expect_success "setup $f checkout ${ceol:+ with -c core.eol=$ceol}"  '
-			rm -f crlf_false_attr__$f.txt &&
-			git ${ceol:+-c core.eol=$ceol} checkout -- crlf_false_attr__$f.txt
-		'
-	done
+	test_expect_success "setup checkout ${ceol:+ with -c core.eol=$ceol}" '
+		rm -f crlf_false_attr__LF.txt crlf_false_attr__CRLF.txt \
+			crlf_false_attr__LF_mix_CR.txt crlf_false_attr__CRLF_mix_LF.txt \
+			crlf_false_attr__LF_nul.txt &&
+		git ${ceol:+-c core.eol=$ceol} checkout -- \
+			crlf_false_attr__LF.txt crlf_false_attr__CRLF.txt \
+			crlf_false_attr__LF_mix_CR.txt crlf_false_attr__CRLF_mix_LF.txt \
+			crlf_false_attr__LF_nul.txt
+	'
 
 	test_expect_success "ls-files --eol attr=$attr $ident aeol=$aeol core.autocrlf=$crlf core.eol=$ceol" '
 		test_when_finished "rm expect actual" &&

-- 
2.56.0.500.g5d06dc02cd.dirty

[PATCH 5/7] t0027: cache expected output files

[Patrick Steinhardt]

In the preceding commit we have optimized one aspect of t0027 already.
Another inefficiency is `compare_files ()` and `compare_ws_file ()`,
which are both called more than a 1000 times.

These functions will repeatedly generate the same expected outcome even
though there's only a small handful of input files that the expected
outcome is computed from. Recomputing those every time is of course
quite pointless.

Improve this by computing the expected outcome once, only, to reduce the
number of spawned processes. This reduces the runtime by around 20-25%,
and on Windows the runtime decreases from 480 to 380 seconds.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 t/t0027-auto-crlf.sh | 17 +++++++++--------
 1 file changed, 9 insertions(+), 8 deletions(-)

diff --git a/t/t0027-auto-crlf.sh b/t/t0027-auto-crlf.sh
index eefb346d11f..25ed6b33704 100755
--- a/t/t0027-auto-crlf.sh
+++ b/t/t0027-auto-crlf.sh
@@ -5,22 +5,23 @@ test_description='CRLF conversion all combinations'
 . ./test-lib.sh
 
 compare_files () {
-	tr '\015\000' QN <"$1" >"$1".expect &&
+	{ test -f "$1".expect || tr '\015\000' QN <"$1" >"$1".expect; } &&
 	tr '\015\000' QN <"$2" | tr -d 'Z' >"$2".actual &&
-	test_cmp "$1".expect "$2".actual &&
-	rm "$1".expect "$2".actual
+	test_cmp "$1".expect "$2".actual
 }
 
 compare_ws_file () {
 	pfx=$1
-	exp=$2.expect
+	exp=$2.ws-expect
 	act=$pfx.actual.$3
-	tr '\015\000abcdef0123456789' QN00000000000000000 <"$2" |
-		sed -e "s/0000*/$ZERO_OID/" >"$exp" &&
+	{
+		test -f "$exp" ||
+		tr '\015\000abcdef0123456789' QN00000000000000000 <"$2" |
+			sed -e "s/0000*/$ZERO_OID/" >"$exp" ;
+	} &&
 	tr '\015\000abcdef0123456789' QN00000000000000000 <"$3" |
 		sed -e "s/0000*/$ZERO_OID/" >"$act" &&
-	test_cmp "$exp" "$act" &&
-	rm "$exp" "$act"
+	test_cmp "$exp" "$act"
 }
 
 create_gitattributes () {

-- 
2.56.0.500.g5d06dc02cd.dirty

[PATCH 6/7] t1092: set up repositories from templates

[Patrick Steinhardt]

In t1092 we verify the behaviour of sparse checkouts. We use three
different repositories to do so: one with a full checkout, one with a
sparse checkout and one with a sparse index, and then verify that the
result is the same across all these configurations.

Those repositories get set up via `init_repos ()`, which initializes
them from scratch every single time with a bunch of processes. The
function is called more than a 100 times though, so the overhead adds up
quite a bit. While the tests only require around 30 seconds on Linux,
it takes more than 10 minutes on Windows.

Optimize this by instead creating the repositories once, only. So
instead of recreating them from scratch, we can then just copy the
template repositories into place and thus save a bunch of processes.
This speeds up tests on Windows by about 3 minutes.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 t/t1092-sparse-checkout-compatibility.sh | 34 +++++++++++++++++++-------------
 1 file changed, 20 insertions(+), 14 deletions(-)

diff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh
index 05b54062b3b..5d81ec54c23 100755
--- a/t/t1092-sparse-checkout-compatibility.sh
+++ b/t/t1092-sparse-checkout-compatibility.sh
@@ -141,28 +141,34 @@ test_expect_success 'setup' '
 	)
 '
 
+test_expect_success 'setup template repositories' '
+	cp -r initial-repo full-checkout.template &&
+	cp -r initial-repo sparse-checkout.template &&
+	cp -r initial-repo sparse-index.template &&
+
+	# initialize sparse-checkout definitions
+	git -C sparse-checkout.template sparse-checkout init --cone &&
+	git -C sparse-checkout.template sparse-checkout set deep &&
+	git -C sparse-index.template sparse-checkout init --cone --sparse-index &&
+	test_cmp_config -C sparse-index.template true index.sparse &&
+	git -C sparse-index.template sparse-checkout set deep &&
+
+	# Disable this message to keep stderr the same.
+	git -C sparse-index.template config advice.sparseIndexExpanded false
+'
+
 init_repos () {
 	rm -rf full-checkout sparse-checkout sparse-index &&
 
 	# create repos in initial state
-	cp -r initial-repo full-checkout &&
+	cp -r full-checkout.template full-checkout &&
 	git -C full-checkout reset --hard &&
 
-	cp -r initial-repo sparse-checkout &&
+	cp -r sparse-checkout.template sparse-checkout &&
 	git -C sparse-checkout reset --hard &&
 
-	cp -r initial-repo sparse-index &&
-	git -C sparse-index reset --hard &&
-
-	# initialize sparse-checkout definitions
-	git -C sparse-checkout sparse-checkout init --cone &&
-	git -C sparse-checkout sparse-checkout set deep &&
-	git -C sparse-index sparse-checkout init --cone --sparse-index &&
-	test_cmp_config -C sparse-index true index.sparse &&
-	git -C sparse-index sparse-checkout set deep &&
-
-	# Disable this message to keep stderr the same.
-	git -C sparse-index config advice.sparseIndexExpanded false
+	cp -r sparse-index.template sparse-index &&
+	git -C sparse-index reset --hard
 }
 
 init_repos_as_submodules () {

-- 
2.56.0.500.g5d06dc02cd.dirty

[PATCH 7/7] t5551: use reftables for repositories with many refs

[Patrick Steinhardt]

Some of the tests in t5551 creates almost a 100,000 references and then
verify that those can be fetched. Naturally, creating this many refs is
quite involved with the "files" backend, as it means that we have to
create this many individual loose files. This is tolerable on Linux,
where the test takes around 2 minutes in our CI. But on macOS, the same
test takes almost 20 minutes to complete. And to add insult to injury,
we execute these tests a second time via t5559.

The system under test here is the HTTP negotiation, so we don't actually
care about whether or not the ref backend handles this alright. So adapt
the test to instead create repositories with the reftable backend, which
is significantly faster in this case as we only need to write a single
table.

And sure enough, this reduces the runtime of this test on macOS by a
factor of 10, going from 20 minutes to 2 minutes. On Linux, the tests at
least speed up by a factor of two.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 t/t5551-http-fetch-smart.sh | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)

diff --git a/t/t5551-http-fetch-smart.sh b/t/t5551-http-fetch-smart.sh
index 130053c57aa..34945059531 100755
--- a/t/t5551-http-fetch-smart.sh
+++ b/t/t5551-http-fetch-smart.sh
@@ -412,7 +412,8 @@ create_tags () {
 }
 
 test_expect_success 'create 2,000 tags in the repo' '
-	git init --bare "$HTTPD_DOCUMENT_ROOT_PATH/many-tags.git" &&
+	git init --bare --ref-format=reftable \
+		"$HTTPD_DOCUMENT_ROOT_PATH/many-tags.git" &&
 	(
 		cd "$HTTPD_DOCUMENT_ROOT_PATH/many-tags.git" &&
 		create_tags 1 2000
@@ -421,7 +422,8 @@ test_expect_success 'create 2,000 tags in the repo' '
 
 test_expect_success CMDLINE_LIMIT \
 	'clone the 2,000 tag repo to check OS command line overflow' '
-	run_with_limited_cmdline git clone $HTTPD_URL/smart/many-tags.git too-many-refs &&
+	run_with_limited_cmdline git clone --ref-format=reftable \
+		$HTTPD_URL/smart/many-tags.git too-many-refs &&
 	(
 		cd too-many-refs &&
 		git for-each-ref refs/tags >actual &&

-- 
2.56.0.500.g5d06dc02cd.dirty

Re: [PATCH 1/7] gitlab-ci: speed up installation of Git for Windows SDK

[Junio C Hamano]

Patrick Steinhardt <ps@pks.im> writes:

> The "build:mingw64" job on GitLab CI downloads the minimal Git for
> Windows SDK on every run and then extracts it via `Expand-Archive`,
> which is expensive.
>
> Improve this by caching the SDK and by using tar(1) instead of
> `Expand-Archive`.
> Note that caching actually only gives us a win of a
> couple seconds. The bigger win here is using tar(1), which speeds up
> extracting the archive from 3 minutes to about 10 seconds. Regardless of

Wow, is a tool to move the bits from one part of a rusted iron
surface to another part really so much more expensive compared to
another tool to move the same bits, to make a difference that is
worth doing this?

> that though, caching the SDK feels like a sensible thing to do as it
> removes one step that may spuriously fail.

One thing that may make readers curious is if we need to worry about
this cached copy going stale.

> Signed-off-by: Patrick Steinhardt <ps@pks.im>
> ---
>  .gitlab-ci.yml     |  9 ++++++++-
>  ci/install-sdk.ps1 | 26 ++++++++++++++++++++------
>  2 files changed, 28 insertions(+), 7 deletions(-)
>
> diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
> index 9b7698a0037..15419a954b3 100644
> --- a/.gitlab-ci.yml
> +++ b/.gitlab-ci.yml
> @@ -127,9 +127,16 @@ build:mingw64:
>      - saas-windows-medium-amd64
>    variables:
>      NO_PERL: 1
> +  cache:
> +    key:
> +      files:
> +        - ci/install-sdk.ps1
> +        - ci/install-dependencies.ps1
> +    paths:
> +      - .dependencies
>    before_script:
>      - *windows_before_script
> -    - ./ci/install-sdk.ps1 -directory "git-sdk"
> +    - ./ci/install-sdk.ps1 -Directory "git-sdk"

Is this change essential to understand this patch, or is it an
unrelated general clean-up to follow some style rules?  I notice
that ...

>      - ./ci/install-dependencies.ps1 -Mingw
>    script:
>      # The minimal SDK's profile resets PATH.
> diff --git a/ci/install-sdk.ps1 b/ci/install-sdk.ps1
> index 66f24838a40..1dbcef04a18 100755
> --- a/ci/install-sdk.ps1
> +++ b/ci/install-sdk.ps1
> @@ -1,12 +1,26 @@
>  param(
> -    [string]$directory='git-sdk',
> -    [string]$url='https://github.com/git-for-windows/git-sdk-64/releases/download/ci-artifacts/git-sdk-x86_64-minimal.zip'
> +    [string]$Directory='git-sdk',
> +    [string]$URL='https://github.com/git-for-windows/git-sdk-64/releases/download/ci-artifacts/git-sdk-x86_64-minimal.zip',
> +    [string]$DownloadDirectory='.dependencies'

... we are making a matching change here to upcase the parameter
names here.

Re: [PATCH 3/7] ci: build optimized binaries with Meson

[Junio C Hamano]

Patrick Steinhardt <ps@pks.im> writes:

> Speed up executing the tests by using the "debugoptimized" build type
> instead. This reduces the runtime of "fedora-breaking-changes-meson"
> from 2100 to 1000 seconds. Other Linux- and macOS-based jobs benefit as
> well, but they only speed up by around ~10%.

That is an impressive improvement.

> On Windows though this is a bit more muddy, as the build step gets
> almost two minutes slower, whereas tests only run ~10% faster. Combined
> though the total run time is roughly the same before and after this
> change.
>
> Signed-off-by: Patrick Steinhardt <ps@pks.im>
> ---
>  .gitlab-ci.yml            | 2 +-
>  ci/run-build-and-tests.sh | 1 +
>  2 files changed, 2 insertions(+), 1 deletion(-)
>
> diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
> index 036d2f3c7f2..df82cd3a5df 100644
> --- a/.gitlab-ci.yml
> +++ b/.gitlab-ci.yml
> @@ -189,7 +189,7 @@ build:msvc-meson:
>    extends: .msvc-meson
>    stage: build
>    script:
> -    - meson setup build --vsenv -Dperl=disabled -Dbackend_max_links=1 -Dcredential_helpers=wincred -Dtest_output_directory="$TEST_OUTPUT_DIRECTORY"
> +    - meson setup build --vsenv --buildtype=debugoptimized -Dperl=disabled -Dbackend_max_links=1 -Dcredential_helpers=wincred -Dtest_output_directory="$TEST_OUTPUT_DIRECTORY"
>      - meson compile -C build
>    artifacts:
>      paths:
> diff --git a/ci/run-build-and-tests.sh b/ci/run-build-and-tests.sh
> index 68b91e31439..fd8136af342 100755
> --- a/ci/run-build-and-tests.sh
> +++ b/ci/run-build-and-tests.sh
> @@ -55,6 +55,7 @@ case "$jobname" in
>  		--fatal-meson-warnings \
>  		--warnlevel 2 --werror \
>  		--wrap-mode nofallback \
> +		--buildtype=debugoptimized \
>  		-Dfuzzers=true \
>  		-Dtest_output_directory="${TEST_OUTPUT_DIRECTORY:-$(pwd)/t}" \
>  		$MESONFLAGS

Re: [PATCH 4/7] t0027: optimize how we reset repository state

[Junio C Hamano]

Patrick Steinhardt <ps@pks.im> writes:

>  	pfx=eol_${ceol}_crlf_${crlf}_attr_${attr}_ &&
> -	for f in LF CRLF LF_mix_CR CRLF_mix_LF LF_nul
> -	do
> -		test_expect_success "setup $f checkout ${ceol:+ with -c core.eol=$ceol}"  '
> -			rm -f crlf_false_attr__$f.txt &&
> -			git ${ceol:+-c core.eol=$ceol} checkout -- crlf_false_attr__$f.txt
> -		'
> -	done
> +	test_expect_success "setup checkout ${ceol:+ with -c core.eol=$ceol}" '
> +		rm -f crlf_false_attr__LF.txt crlf_false_attr__CRLF.txt \
> +			crlf_false_attr__LF_mix_CR.txt crlf_false_attr__CRLF_mix_LF.txt \
> +			crlf_false_attr__LF_nul.txt &&
> +		git ${ceol:+-c core.eol=$ceol} checkout -- \
> +			crlf_false_attr__LF.txt crlf_false_attr__CRLF.txt \
> +			crlf_false_attr__LF_mix_CR.txt crlf_false_attr__CRLF_mix_LF.txt \
> +			crlf_false_attr__LF_nul.txt
> +	'

Hmph, don't we want avoid hand-expand the list of files even though
we may want to use one "rm -f" and another "git checkout" command
to work on them?  IOW,

	paths=
	for f in LF CRLF LF_mix_CR CRLF_mix_LF LF_nul
	do
		paths=${paths:+"$paths "}crlf_false_attr__$f.txt
	done
	test_expect_success "setup checkout ${ceol:+ with -c core.eol=$ceol}" '
		rm -f $paths &&
		git ${ceol:+-c core.eol=$ceol} checkout -- $paths
	'

or something?

>  
>  	test_expect_success "ls-files --eol attr=$attr $ident aeol=$aeol core.autocrlf=$crlf core.eol=$ceol" '
>  		test_when_finished "rm expect actual" &&

Re: [PATCH 0/7] ci: some speedups for slow tests

[Junio C Hamano]

Patrick Steinhardt <ps@pks.im> writes:

> I was staring at CI way more often recently, and that got me annoyed
> real fast because it's just painstakingly slow. So I decided to have a
> look and try do something about it, the result of which is this patch
> series.

Thanks for these patches.  I didn't leave comments on later parts of
the series, but all of them were interesting read and fully made
sense to me.

Re: [PATCH 4/7] t0027: optimize how we reset repository state

[Torsten Bögershausen]

On Fri, Nov 06, 2026 at 09:33:01AM -0800, Junio C Hamano wrote:
> Hmph, don't we want avoid hand-expand the list of files even though
> we may want to use one "rm -f" and another "git checkout" command
> to work on them?  IOW,
> 
> 	paths=
> 	for f in LF CRLF LF_mix_CR CRLF_mix_LF LF_nul
> 	do
> 		paths=${paths:+"$paths "}crlf_false_attr__$f.txt
> 	done
> 	test_expect_success "setup checkout ${ceol:+ with -c core.eol=$ceol}" '
> 		rm -f $paths &&
> 		git ${ceol:+-c core.eol=$ceol} checkout -- $paths
> 	'

"Wow, so it can go" - a well working, but slow code is speed up so much.

I used the following (before reading the suggestion from Junio)
and the execution time here on an older MacOs dropped from 71 seconds to 54 seconds.

    test_expect_success "setup file checkout ${ceol:+ with -c core.eol=$ceol}"  '
        files="\
           crlf_false_attr__LF.txt \
           crlf_false_attr__CRLF.txt \
           crlf_false_attr__LF_mix_CR.txt \
           crlf_false_attr__CRLF_mix_LF.txt \
           crlf_false_attr__LF_nul.txt" &&
        rm -fv $files &&
        git ${ceol:+-c core.eol=$ceol} checkout -- $files
    '

Re: [PATCH 4/7] t0027: optimize how we reset repository state

[Patrick Steinhardt]

On Sun, Nov 08, 2026 at 05:49:01PM +0100, Torsten Bögershausen wrote:
> On Fri, Nov 06, 2026 at 09:33:01AM -0800, Junio C Hamano wrote:
> > Hmph, don't we want avoid hand-expand the list of files even though
> > we may want to use one "rm -f" and another "git checkout" command
> > to work on them?  IOW,
> > 
> > 	paths=
> > 	for f in LF CRLF LF_mix_CR CRLF_mix_LF LF_nul
> > 	do
> > 		paths=${paths:+"$paths "}crlf_false_attr__$f.txt
> > 	done
> > 	test_expect_success "setup checkout ${ceol:+ with -c core.eol=$ceol}" '
> > 		rm -f $paths &&
> > 		git ${ceol:+-c core.eol=$ceol} checkout -- $paths
> > 	'
> 
> "Wow, so it can go" - a well working, but slow code is speed up so much.
> 
> I used the following (before reading the suggestion from Junio)
> and the execution time here on an older MacOs dropped from 71 seconds to 54 seconds.
> 
>     test_expect_success "setup file checkout ${ceol:+ with -c core.eol=$ceol}"  '
>         files="\
>            crlf_false_attr__LF.txt \
>            crlf_false_attr__CRLF.txt \
>            crlf_false_attr__LF_mix_CR.txt \
>            crlf_false_attr__CRLF_mix_LF.txt \
>            crlf_false_attr__LF_nul.txt" &&
>         rm -fv $files &&
>         git ${ceol:+-c core.eol=$ceol} checkout -- $files
>     '

Hmm. I think I like Junio's alternative a bit better, but both are
definitely an improvement over my take. Will add to the next version,
thanks.

Patrick

Re: [PATCH 1/7] gitlab-ci: speed up installation of Git for Windows SDK

[Patrick Steinhardt]

On Fri, Nov 06, 2026 at 09:20:57AM -0800, Junio C Hamano wrote:
> Patrick Steinhardt <ps@pks.im> writes:
> 
> > The "build:mingw64" job on GitLab CI downloads the minimal Git for
> > Windows SDK on every run and then extracts it via `Expand-Archive`,
> > which is expensive.
> >
> > Improve this by caching the SDK and by using tar(1) instead of
> > `Expand-Archive`.
> > Note that caching actually only gives us a win of a
> > couple seconds. The bigger win here is using tar(1), which speeds up
> > extracting the archive from 3 minutes to about 10 seconds. Regardless of
> 
> Wow, is a tool to move the bits from one part of a rusted iron
> surface to another part really so much more expensive compared to
> another tool to move the same bits, to make a difference that is
> worth doing this?

I was surprised by this, as well.

> > that though, caching the SDK feels like a sensible thing to do as it
> > removes one step that may spuriously fail.
> 
> One thing that may make readers curious is if we need to worry about
> this cached copy going stale.

Yes, it may go stale, but I'm not even sure whether that really matters.
I decided to just do this as-is and Cc Johannes to maybe facilitate his
input on this part. But arguably, we can just skip the cache altogether
as it didn't really buy much anyway.

> > diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
> > index 9b7698a0037..15419a954b3 100644
> > --- a/.gitlab-ci.yml
> > +++ b/.gitlab-ci.yml
> > @@ -127,9 +127,16 @@ build:mingw64:
> >      - saas-windows-medium-amd64
> >    variables:
> >      NO_PERL: 1
> > +  cache:
> > +    key:
> > +      files:
> > +        - ci/install-sdk.ps1
> > +        - ci/install-dependencies.ps1
> > +    paths:
> > +      - .dependencies
> >    before_script:
> >      - *windows_before_script
> > -    - ./ci/install-sdk.ps1 -directory "git-sdk"
> > +    - ./ci/install-sdk.ps1 -Directory "git-sdk"
> 
> Is this change essential to understand this patch, or is it an
> unrelated general clean-up to follow some style rules?  I notice
> that ...
> 
> >      - ./ci/install-dependencies.ps1 -Mingw
> >    script:
> >      # The minimal SDK's profile resets PATH.
> > diff --git a/ci/install-sdk.ps1 b/ci/install-sdk.ps1
> > index 66f24838a40..1dbcef04a18 100755
> > --- a/ci/install-sdk.ps1
> > +++ b/ci/install-sdk.ps1
> > @@ -1,12 +1,26 @@
> >  param(
> > -    [string]$directory='git-sdk',
> > -    [string]$url='https://github.com/git-for-windows/git-sdk-64/releases/download/ci-artifacts/git-sdk-x86_64-minimal.zip'
> > +    [string]$Directory='git-sdk',
> > +    [string]$URL='https://github.com/git-for-windows/git-sdk-64/releases/download/ci-artifacts/git-sdk-x86_64-minimal.zip',
> > +    [string]$DownloadDirectory='.dependencies'
> 
> ... we are making a matching change here to upcase the parameter
> names here.

Well, with the new DownloadDirectory parameter everything else would
have been inconsistent, so I made those changes while at it. I'll dropt
he cache in v2 though, and with that we can drop these changes, too.

Patrick

[PATCH v2 0/7] ci: some speedups for slow tests

[Patrick Steinhardt]

Hi,

I was staring at CI way more often recently, and that got me annoyed
real fast because it's just painstakingly slow. So I decided to have a
look and try do something about it, the result of which is this patch
series.

The series contains a couple of bits that are specific to GitLab, and a
couple of fixes that are generic. Some highlights:

  - Installing the Git for Windows SDK speeds up from 3 minutes to 10
    seconds by using tar(1) instead of Extract-Archive.

  - t0027 speeds up from >10 minutes to 6 minutes on Windows by avoiding
    the overhead of spawning redundant processes.

  - t5551 and t5559 both speed up by a factor of 10 on macOS, going from
    20 minutes each to 2 minutes each by using reftables in a repo with
    100k refs. "osx-meson" and "osx-clang" thus go from 29 minutes to 18
    minutes end-to-end.

  - "fedora-breaking-changes-meson" speeds up by a factor of 2, going
    from 35 minutes to 16 minutes. Turns out our SHA256 implementation
    is quite slow with "-O0".

Unfortunately, our Windows runners at GitLab are still atrociously slow
by themselves. So the critical path only improves by about ~10 minutes,
going from 60-70 minutes to 53 minutes end-to-end. But hey, at least the
other jobs now give feedback a bit faster. And I guess on GitHub it may
be a different story because Windows runners are way better there and
are typically not on the critical path.

A test run can be found at [1].

Changes in v2:
  - Drop the cache for the GfW SDK again.
  - Drop repetitive paths in t0027.
  - Link to v1: https://patch.msgid.link/20261106-pks-gitlab-ci-speedups-v1-0-1fa0374c7287@pks.im

Thanks!

Patrick

[1]: https://gitlab.com/gitlab-org/git/-/merge_requests/734

---
Patrick Steinhardt (7):
      gitlab-ci: speed up installation of Git for Windows SDK
      gitlab-ci: exclude intermediate outputs from Windows artifacts
      ci: build optimized binaries with Meson
      t0027: optimize how we reset repository state
      t0027: cache expected output files
      t1092: set up repositories from templates
      t5551: use reftables for repositories with many refs

 .gitlab-ci.yml                           |  4 +++-
 ci/install-sdk.ps1                       |  6 +++++-
 ci/run-build-and-tests.sh                |  1 +
 t/t0027-auto-crlf.sh                     | 26 +++++++++++++-----------
 t/t1092-sparse-checkout-compatibility.sh | 34 +++++++++++++++++++-------------
 t/t5551-http-fetch-smart.sh              |  6 ++++--
 6 files changed, 47 insertions(+), 30 deletions(-)

Range-diff versus v1:

1:  e25adb1b8c8 < -:  ----------- gitlab-ci: speed up installation of Git for Windows SDK
-:  ----------- > 1:  582a9ab6468 gitlab-ci: speed up installation of Git for Windows SDK
2:  116c8b8a716 = 2:  bff0a624127 gitlab-ci: exclude intermediate outputs from Windows artifacts
3:  a035b8a902c = 3:  713e69bfb59 ci: build optimized binaries with Meson
4:  1a1238f4be7 < -:  ----------- t0027: optimize how we reset repository state
-:  ----------- > 4:  e421eb2a275 t0027: optimize how we reset repository state
5:  c652f5b380a = 5:  16905b5be62 t0027: cache expected output files
6:  9a90272c620 = 6:  43f355fbb02 t1092: set up repositories from templates
7:  187128e7d5d = 7:  960716da178 t5551: use reftables for repositories with many refs

---
base-commit: 0ffa3e35c7c82ab8d0de226149fb03b3f7674d05
change-id: 20261106-pks-gitlab-ci-speedups-09c3841eb178

[PATCH v2 1/7] gitlab-ci: speed up installation of Git for Windows SDK

[Patrick Steinhardt]

The "build:mingw64" job on GitLab CI downloads the minimal Git for
Windows SDK on every run and then extracts it via `Expand-Archive`,
which is expensive.

Improve this by using tar(1) instead of `Expand-Archive`. While at it,
make the script error on failures and suppress progress output.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 ci/install-sdk.ps1 | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/ci/install-sdk.ps1 b/ci/install-sdk.ps1
index 66f24838a40..bb332c1c10f 100755
--- a/ci/install-sdk.ps1
+++ b/ci/install-sdk.ps1
@@ -3,8 +3,12 @@ param(
     [string]$url='https://github.com/git-for-windows/git-sdk-64/releases/download/ci-artifacts/git-sdk-x86_64-minimal.zip'
 )
 
+$ErrorActionPreference = 'Stop'
+$ProgressPreference = 'SilentlyContinue'
+
 Invoke-WebRequest "$url" -OutFile git-sdk.zip
-Expand-Archive -LiteralPath git-sdk.zip -DestinationPath "$directory"
+New-Item -Path $directory -ItemType Directory -Force
+tar -xf git-sdk.zip -C $directory
 Remove-Item -Path git-sdk.zip
 
 New-Item -Path .git/info -ItemType Directory -Force

-- 
2.56.0.611.g1030f0b940.dirty

[PATCH v2 2/7] gitlab-ci: exclude intermediate outputs from Windows artifacts

[Patrick Steinhardt]

As running tests is very slow on Windows we shard our CI tests across 10
runners. To avoid rebuilding Git on each of those runners we have a
preliminary step which builds Git and then passes on the build artifacts
to the test jobs.

This archive contains a ton of intermediate "*.obj" build artifacts that
aren't needed by the test jobs: the artifact contains ~2300 files, out
of which ~1800 are "*.obj" files.

Exclude those files from the build artifact, which reduces the time it
takes to download and extract them from 34 seconds to 17 seconds.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 .gitlab-ci.yml | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index 9b7698a0037..3631393a8cc 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -187,6 +187,8 @@ build:msvc-meson:
   artifacts:
     paths:
       - build
+    exclude:
+      - build/**/*.obj
 
 test:msvc-meson:
   extends: .msvc-meson

-- 
2.56.0.611.g1030f0b940.dirty

[PATCH v2 3/7] ci: build optimized binaries with Meson

[Patrick Steinhardt]

When taking a look at our CI jobs one will quickly notice that across
the Linux-based jobs, the "fedora-breaking-changes-meson" job is running
significantly longer than any of the other jobs. While most Linux jobs
only run around 15 minutes, that particular job runs roughly twice as
long.

The root cause of this is a combination of two different things:

  - The job uses Meson to compile Git, and Meson by default uses the
    "debug" build type that translates into "-O0". Our Makefile in
    contrast to that uses "-g -O2" by default.

  - The job enables breaking changes and thus runs with SHA256 by
    default with the "block" backend, which benefits heavily from
    compiler optimizations. For example, hashing a 500MB blob with
    SHA256 takes 9.6 seconds with "-O0", whereas it takes only 1.5
    seconds with "-O2". The SHA1 code path only slows down from 0.9
    seconds to 2.1 seconds with "-O0".

Of course, compiling without optimizations also has an impact on all the
other jobs, but the Fedora job is by far the most impacted one.

Speed up executing the tests by using the "debugoptimized" build type
instead. This reduces the runtime of "fedora-breaking-changes-meson"
from 2100 to 1000 seconds. Other Linux- and macOS-based jobs benefit as
well, but they only speed up by around ~10%.

On Windows though this is a bit more muddy, as the build step gets
almost two minutes slower, whereas tests only run ~10% faster. Combined
though the total run time is roughly the same before and after this
change.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 .gitlab-ci.yml            | 2 +-
 ci/run-build-and-tests.sh | 1 +
 2 files changed, 2 insertions(+), 1 deletion(-)

diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index 3631393a8cc..449fed1b392 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -182,7 +182,7 @@ build:msvc-meson:
   extends: .msvc-meson
   stage: build
   script:
-    - meson setup build --vsenv -Dperl=disabled -Dbackend_max_links=1 -Dcredential_helpers=wincred -Dtest_output_directory="$TEST_OUTPUT_DIRECTORY"
+    - meson setup build --vsenv --buildtype=debugoptimized -Dperl=disabled -Dbackend_max_links=1 -Dcredential_helpers=wincred -Dtest_output_directory="$TEST_OUTPUT_DIRECTORY"
     - meson compile -C build
   artifacts:
     paths:
diff --git a/ci/run-build-and-tests.sh b/ci/run-build-and-tests.sh
index 68b91e31439..fd8136af342 100755
--- a/ci/run-build-and-tests.sh
+++ b/ci/run-build-and-tests.sh
@@ -55,6 +55,7 @@ case "$jobname" in
 		--fatal-meson-warnings \
 		--warnlevel 2 --werror \
 		--wrap-mode nofallback \
+		--buildtype=debugoptimized \
 		-Dfuzzers=true \
 		-Dtest_output_directory="${TEST_OUTPUT_DIRECTORY:-$(pwd)/t}" \
 		$MESONFLAGS

-- 
2.56.0.611.g1030f0b940.dirty

[PATCH v2 4/7] t0027: optimize how we reset repository state

[Patrick Steinhardt]

In t0027 we verify our logic to convert CRLF for all kinds of
combinations of attributes and configuration, which ultimately results
in 2600 test cases. This is of course on the heavier side, so on Windows
this takes more than 10 minutes to run.

One source of slowness is `checkout_files ()`, which gets called for
a lot of our test to set up the initial working directory. As part of
that we both remove the old files that we care about and then check them
out again via git-checkout(1), but we do this one by one for every
single file.

This is of course rather wasteful, as we have to spawn ten different
processes every time the function is invoked. And on top of that, we
also have to rewrite the complete index in git-checkout(1) every time.

Speed this up by instead using a single invocation of rm(1) and
git-checkout(1). This reduces the runtime by about 20% on all platforms,
and on Windows this reduces the runtime by ~2.5 minutes.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 t/t0027-auto-crlf.sh | 9 +++++----
 1 file changed, 5 insertions(+), 4 deletions(-)

diff --git a/t/t0027-auto-crlf.sh b/t/t0027-auto-crlf.sh
index 49dbf09da77..504a7ea54bb 100755
--- a/t/t0027-auto-crlf.sh
+++ b/t/t0027-auto-crlf.sh
@@ -305,11 +305,12 @@ checkout_files () {
 	pfx=eol_${ceol}_crlf_${crlf}_attr_${attr}_ &&
 	for f in LF CRLF LF_mix_CR CRLF_mix_LF LF_nul
 	do
-		test_expect_success "setup $f checkout ${ceol:+ with -c core.eol=$ceol}"  '
-			rm -f crlf_false_attr__$f.txt &&
-			git ${ceol:+-c core.eol=$ceol} checkout -- crlf_false_attr__$f.txt
-		'
+		paths="${paths:+"$paths "}crlf_false_attr__$f.txt"
 	done
+	test_expect_success "setup checkout ${ceol:+ with -c core.eol=$ceol}" '
+		rm -f $paths &&
+		git ${ceol:+-c core.eol=$ceol} checkout -- $paths
+	'
 
 	test_expect_success "ls-files --eol attr=$attr $ident aeol=$aeol core.autocrlf=$crlf core.eol=$ceol" '
 		test_when_finished "rm expect actual" &&

-- 
2.56.0.611.g1030f0b940.dirty

[PATCH v2 5/7] t0027: cache expected output files

[Patrick Steinhardt]

In the preceding commit we have optimized one aspect of t0027 already.
Another inefficiency is `compare_files ()` and `compare_ws_file ()`,
which are both called more than a 1000 times.

These functions will repeatedly generate the same expected outcome even
though there's only a small handful of input files that the expected
outcome is computed from. Recomputing those every time is of course
quite pointless.

Improve this by computing the expected outcome once, only, to reduce the
number of spawned processes. This reduces the runtime by around 20-25%,
and on Windows the runtime decreases from 480 to 380 seconds.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 t/t0027-auto-crlf.sh | 17 +++++++++--------
 1 file changed, 9 insertions(+), 8 deletions(-)

diff --git a/t/t0027-auto-crlf.sh b/t/t0027-auto-crlf.sh
index 504a7ea54bb..b021f0855a8 100755
--- a/t/t0027-auto-crlf.sh
+++ b/t/t0027-auto-crlf.sh
@@ -5,22 +5,23 @@ test_description='CRLF conversion all combinations'
 . ./test-lib.sh
 
 compare_files () {
-	tr '\015\000' QN <"$1" >"$1".expect &&
+	{ test -f "$1".expect || tr '\015\000' QN <"$1" >"$1".expect; } &&
 	tr '\015\000' QN <"$2" | tr -d 'Z' >"$2".actual &&
-	test_cmp "$1".expect "$2".actual &&
-	rm "$1".expect "$2".actual
+	test_cmp "$1".expect "$2".actual
 }
 
 compare_ws_file () {
 	pfx=$1
-	exp=$2.expect
+	exp=$2.ws-expect
 	act=$pfx.actual.$3
-	tr '\015\000abcdef0123456789' QN00000000000000000 <"$2" |
-		sed -e "s/0000*/$ZERO_OID/" >"$exp" &&
+	{
+		test -f "$exp" ||
+		tr '\015\000abcdef0123456789' QN00000000000000000 <"$2" |
+			sed -e "s/0000*/$ZERO_OID/" >"$exp" ;
+	} &&
 	tr '\015\000abcdef0123456789' QN00000000000000000 <"$3" |
 		sed -e "s/0000*/$ZERO_OID/" >"$act" &&
-	test_cmp "$exp" "$act" &&
-	rm "$exp" "$act"
+	test_cmp "$exp" "$act"
 }
 
 create_gitattributes () {

-- 
2.56.0.611.g1030f0b940.dirty

[PATCH v2 6/7] t1092: set up repositories from templates

[Patrick Steinhardt]

In t1092 we verify the behaviour of sparse checkouts. We use three
different repositories to do so: one with a full checkout, one with a
sparse checkout and one with a sparse index, and then verify that the
result is the same across all these configurations.

Those repositories get set up via `init_repos ()`, which initializes
them from scratch every single time with a bunch of processes. The
function is called more than a 100 times though, so the overhead adds up
quite a bit. While the tests only require around 30 seconds on Linux,
it takes more than 10 minutes on Windows.

Optimize this by instead creating the repositories once, only. So
instead of recreating them from scratch, we can then just copy the
template repositories into place and thus save a bunch of processes.
This speeds up tests on Windows by about 3 minutes.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 t/t1092-sparse-checkout-compatibility.sh | 34 +++++++++++++++++++-------------
 1 file changed, 20 insertions(+), 14 deletions(-)

diff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh
index 05b54062b3b..5d81ec54c23 100755
--- a/t/t1092-sparse-checkout-compatibility.sh
+++ b/t/t1092-sparse-checkout-compatibility.sh
@@ -141,28 +141,34 @@ test_expect_success 'setup' '
 	)
 '
 
+test_expect_success 'setup template repositories' '
+	cp -r initial-repo full-checkout.template &&
+	cp -r initial-repo sparse-checkout.template &&
+	cp -r initial-repo sparse-index.template &&
+
+	# initialize sparse-checkout definitions
+	git -C sparse-checkout.template sparse-checkout init --cone &&
+	git -C sparse-checkout.template sparse-checkout set deep &&
+	git -C sparse-index.template sparse-checkout init --cone --sparse-index &&
+	test_cmp_config -C sparse-index.template true index.sparse &&
+	git -C sparse-index.template sparse-checkout set deep &&
+
+	# Disable this message to keep stderr the same.
+	git -C sparse-index.template config advice.sparseIndexExpanded false
+'
+
 init_repos () {
 	rm -rf full-checkout sparse-checkout sparse-index &&
 
 	# create repos in initial state
-	cp -r initial-repo full-checkout &&
+	cp -r full-checkout.template full-checkout &&
 	git -C full-checkout reset --hard &&
 
-	cp -r initial-repo sparse-checkout &&
+	cp -r sparse-checkout.template sparse-checkout &&
 	git -C sparse-checkout reset --hard &&
 
-	cp -r initial-repo sparse-index &&
-	git -C sparse-index reset --hard &&
-
-	# initialize sparse-checkout definitions
-	git -C sparse-checkout sparse-checkout init --cone &&
-	git -C sparse-checkout sparse-checkout set deep &&
-	git -C sparse-index sparse-checkout init --cone --sparse-index &&
-	test_cmp_config -C sparse-index true index.sparse &&
-	git -C sparse-index sparse-checkout set deep &&
-
-	# Disable this message to keep stderr the same.
-	git -C sparse-index config advice.sparseIndexExpanded false
+	cp -r sparse-index.template sparse-index &&
+	git -C sparse-index reset --hard
 }
 
 init_repos_as_submodules () {

-- 
2.56.0.611.g1030f0b940.dirty

[PATCH v2 7/7] t5551: use reftables for repositories with many refs

[Patrick Steinhardt]

Some of the tests in t5551 creates almost a 100,000 references and then
verify that those can be fetched. Naturally, creating this many refs is
quite involved with the "files" backend, as it means that we have to
create this many individual loose files. This is tolerable on Linux,
where the test takes around 2 minutes in our CI. But on macOS, the same
test takes almost 20 minutes to complete. And to add insult to injury,
we execute these tests a second time via t5559.

The system under test here is the HTTP negotiation, so we don't actually
care about whether or not the ref backend handles this alright. So adapt
the test to instead create repositories with the reftable backend, which
is significantly faster in this case as we only need to write a single
table.

And sure enough, this reduces the runtime of this test on macOS by a
factor of 10, going from 20 minutes to 2 minutes. On Linux, the tests at
least speed up by a factor of two.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 t/t5551-http-fetch-smart.sh | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)

diff --git a/t/t5551-http-fetch-smart.sh b/t/t5551-http-fetch-smart.sh
index 130053c57aa..34945059531 100755
--- a/t/t5551-http-fetch-smart.sh
+++ b/t/t5551-http-fetch-smart.sh
@@ -412,7 +412,8 @@ create_tags () {
 }
 
 test_expect_success 'create 2,000 tags in the repo' '
-	git init --bare "$HTTPD_DOCUMENT_ROOT_PATH/many-tags.git" &&
+	git init --bare --ref-format=reftable \
+		"$HTTPD_DOCUMENT_ROOT_PATH/many-tags.git" &&
 	(
 		cd "$HTTPD_DOCUMENT_ROOT_PATH/many-tags.git" &&
 		create_tags 1 2000
@@ -421,7 +422,8 @@ test_expect_success 'create 2,000 tags in the repo' '
 
 test_expect_success CMDLINE_LIMIT \
 	'clone the 2,000 tag repo to check OS command line overflow' '
-	run_with_limited_cmdline git clone $HTTPD_URL/smart/many-tags.git too-many-refs &&
+	run_with_limited_cmdline git clone --ref-format=reftable \
+		$HTTPD_URL/smart/many-tags.git too-many-refs &&
 	(
 		cd too-many-refs &&
 		git for-each-ref refs/tags >actual &&

-- 
2.56.0.611.g1030f0b940.dirty

Re: [PATCH v2 5/7] t0027: cache expected output files

[Torsten Bögershausen]

On Mon, Nov 09, 2026 at 10:58:26AM +0100, Patrick Steinhardt wrote:
> In the preceding commit we have optimized one aspect of t0027 already.
> Another inefficiency is `compare_files ()` and `compare_ws_file ()`,
> which are both called more than a 1000 times.
> 
> These functions will repeatedly generate the same expected outcome even
> though there's only a small handful of input files that the expected
> outcome is computed from. Recomputing those every time is of course
> quite pointless.
> 
> Improve this by computing the expected outcome once, only, to reduce the
> number of spawned processes. This reduces the runtime by around 20-25%,
> and on Windows the runtime decreases from 480 to 380 seconds.
> 
> Signed-off-by: Patrick Steinhardt <ps@pks.im>
> ---
>  t/t0027-auto-crlf.sh | 17 +++++++++--------
>  1 file changed, 9 insertions(+), 8 deletions(-)
> 
> diff --git a/t/t0027-auto-crlf.sh b/t/t0027-auto-crlf.sh
> index 504a7ea54bb..b021f0855a8 100755
> --- a/t/t0027-auto-crlf.sh
> +++ b/t/t0027-auto-crlf.sh
> @@ -5,22 +5,23 @@ test_description='CRLF conversion all combinations'
>  . ./test-lib.sh
>  
>  compare_files () {
> -	tr '\015\000' QN <"$1" >"$1".expect &&
> +	{ test -f "$1".expect || tr '\015\000' QN <"$1" >"$1".expect; } &&
>  	tr '\015\000' QN <"$2" | tr -d 'Z' >"$2".actual &&
> -	test_cmp "$1".expect "$2".actual &&
> -	rm "$1".expect "$2".actual
> +	test_cmp "$1".expect "$2".actual
>  }
>  

Nice.
If you want to save even more CPU/SSD-cycles,
you can prepare the expect files like this:
(No, I didn't do any benchmarks, nor look at ws_compare)


diff --git a/t/t0027-auto-crlf.sh b/t/t0027-auto-crlf.sh
index 49dbf09da7..3563336162 100755
--- a/t/t0027-auto-crlf.sh
+++ b/t/t0027-auto-crlf.sh
@@ -5,10 +5,9 @@ test_description='CRLF conversion all combinations'
 . ./test-lib.sh
 
 compare_files () {
-	tr '\015\000' QN <"$1" >"$1".expect &&
 	tr '\015\000' QN <"$2" | tr -d 'Z' >"$2".actual &&
-	test_cmp "$1".expect "$2".actual &&
-	rm "$1".expect "$2".actual
+	test_cmp "expd/$1".expect "$2".actual &&
+	rm "$2".actual
 }
 
 compare_ws_file () {
@@ -343,6 +342,25 @@ checkout_files () {
 	"
 }
 
+test_expect_success 'create expect files' '
+	printf "\$Id: 0000000000000000000000000000000000000000 \$\nLINEONE\nLINETWO\nLINETHREE"     >LF &&
+	printf "\$Id: 0000000000000000000000000000000000000000 \$\r\nLINEONE\r\nLINETWO\r\nLINETHREE" >CRLF &&
+	printf "\$Id: 0000000000000000000000000000000000000000 \$\nLINEONE\r\nLINETWO\nLINETHREE"   >CRLF_mix_LF &&
+	printf "\$Id: 0000000000000000000000000000000000000000 \$\nLINEONE\nLINETWO\rLINETHREE"     >LF_mix_CR &&
+	printf "\$Id: 0000000000000000000000000000000000000000 \$\r\nLINEONE\r\nLINETWO\rLINETHREE"   >CRLF_mix_CR &&
+	printf "\$Id: 0000000000000000000000000000000000000000 \$\r\nLINEONEQ\r\nLINETWO\r\nLINETHREE" | q_to_nul >CRLF_nul &&
+	printf "\$Id: 0000000000000000000000000000000000000000 \$\nLINEONEQ\nLINETWO\nLINETHREE" | q_to_nul >LF_nul &&
+  mkdir expd &&
+  echo "* -text" >expd/.gitattributes &&
+	tr "\015\000" QN <LF >expd/LF.expect &&
+	tr "\015\000" QN <CRLF >expd/CRLF.expect &&
+	tr "\015\000" QN <CRLF_mix_LF >expd/CRLF_mix_LF.expect &&
+	tr "\015\000" QN <LF_mix_CR >expd/LF_mix_CR.expect &&
+	tr "\015\000" QN <CRLF_mix_CR >expd/CRLF_mix_CR.expect &&
+	tr "\015\000" QN <CRLF_nul >expd/CRLF_nul.expect &&
+	tr "\015\000" QN <LF_nul >expd/LF_nul.expect
+  '
+
 # Test control characters
 # NUL SOH CR EOF==^Z
 test_expect_success 'ls-files --eol -o Text/Binary' '



[snip]

Re: [PATCH v2 1/7] gitlab-ci: speed up installation of Git for Windows SDK

[Junio C Hamano]

Patrick Steinhardt <ps@pks.im> writes:

> +tar -xf git-sdk.zip -C $directory

Wow.  You learn new things every day.

This is in ci/install-sdk.ps1 file.  Right now that script is
referred to only from .gitlab-ci.yml so as long as GitLab CI
environment has tar that works this way we would be fine, but I
thought that ci/* was designed to be portable across CI and other
environments.  At least we would want a comment next to it that says
we assume BSD derived tar that can grok zip archives here, probably?

Re: [PATCH v2 4/7] t0027: optimize how we reset repository state

[Junio C Hamano]

Patrick Steinhardt <ps@pks.im> writes:

> diff --git a/t/t0027-auto-crlf.sh b/t/t0027-auto-crlf.sh
> index 49dbf09da77..504a7ea54bb 100755
> --- a/t/t0027-auto-crlf.sh
> +++ b/t/t0027-auto-crlf.sh
> @@ -305,11 +305,12 @@ checkout_files () {

A micronit.

>  	pfx=eol_${ceol}_crlf_${crlf}_attr_${attr}_ &&
>  	for f in LF CRLF LF_mix_CR CRLF_mix_LF LF_nul
>  	do
> -		test_expect_success "setup $f checkout ${ceol:+ with -c core.eol=$ceol}"  '
> -			rm -f crlf_false_attr__$f.txt &&
> -			git ${ceol:+-c core.eol=$ceol} checkout -- crlf_false_attr__$f.txt
> -		'
> +		paths="${paths:+"$paths "}crlf_false_attr__$f.txt"
>  	done

We may know that nobody has touched variable "$paths" elsewhere in
this program, but I would prefer to have an explicit "let's start
with an empty $paths" assignment very close to this loop to help
future readers.  That way they do not have to wonder and look all
over to find out after what string we are appending these filenames.


> +	test_expect_success "setup checkout ${ceol:+ with -c core.eol=$ceol}" '
> +		rm -f $paths &&
> +		git ${ceol:+-c core.eol=$ceol} checkout -- $paths
> +	'
>  
>  	test_expect_success "ls-files --eol attr=$attr $ident aeol=$aeol core.autocrlf=$crlf core.eol=$ceol" '
>  		test_when_finished "rm expect actual" &&

Re: [PATCH v2 1/7] gitlab-ci: speed up installation of Git for Windows SDK

[Patrick Steinhardt]

On Mon, Nov 09, 2026 at 08:53:53AM -0800, Junio C Hamano wrote:
> Patrick Steinhardt <ps@pks.im> writes:
> 
> > +tar -xf git-sdk.zip -C $directory
> 
> Wow.  You learn new things every day.

Yeah, it's fun. Windows uses bsdtar, which is in turn based on
libarchive, and hence it also supports ZIP and 7zip files [1].

> This is in ci/install-sdk.ps1 file.  Right now that script is
> referred to only from .gitlab-ci.yml so as long as GitLab CI
> environment has tar that works this way we would be fine, but I
> thought that ci/* was designed to be portable across CI and other
> environments.  At least we would want a comment next to it that says
> we assume BSD derived tar that can grok zip archives here, probably?

Well, bsdtar is nowadays distributed with Windows by default, so this
should be portable to other CI environments, too. So I don't think it
really warrants a comment as it should just work everwhere. That's
certainly something to point out in the commit message though.

In fact I wonder whether we should maybe adapt GitHub Workflows to use
that script, too. That was my intent back when introducing this script,
and for a brief time we in fact did that. But we eventually figured that
it was too slow in contrast to the
git-for-windows/setup-git-for-windows-sdk step, so we reverted that
change on GitHub again in 55d62306eea (GitHub ci(windows): speed up
initializing Git for Windows' minimal SDK again, 2024-12-17).

But now that we have found a way to remove the slowness we could unify
both systems again. I intentionally left that out of this patch series
though, as we can still do that as a subsequent step if we want to. In
any case, I want Johannes' opinion on this topic before doing such a
change.

Patrick

[1]: https://learn.microsoft.com/en-us/windows/tar/

Re: [PATCH v2 4/7] t0027: optimize how we reset repository state

[Patrick Steinhardt]

On Mon, Nov 09, 2026 at 08:59:39AM -0800, Junio C Hamano wrote:
> Patrick Steinhardt <ps@pks.im> writes:
> 
> > diff --git a/t/t0027-auto-crlf.sh b/t/t0027-auto-crlf.sh
> > index 49dbf09da77..504a7ea54bb 100755
> > --- a/t/t0027-auto-crlf.sh
> > +++ b/t/t0027-auto-crlf.sh
> > @@ -305,11 +305,12 @@ checkout_files () {
> 
> A micronit.
> 
> >  	pfx=eol_${ceol}_crlf_${crlf}_attr_${attr}_ &&
> >  	for f in LF CRLF LF_mix_CR CRLF_mix_LF LF_nul
> >  	do
> > -		test_expect_success "setup $f checkout ${ceol:+ with -c core.eol=$ceol}"  '
> > -			rm -f crlf_false_attr__$f.txt &&
> > -			git ${ceol:+-c core.eol=$ceol} checkout -- crlf_false_attr__$f.txt
> > -		'
> > +		paths="${paths:+"$paths "}crlf_false_attr__$f.txt"
> >  	done
> 
> We may know that nobody has touched variable "$paths" elsewhere in
> this program, but I would prefer to have an explicit "let's start
> with an empty $paths" assignment very close to this loop to help
> future readers.  That way they do not have to wonder and look all
> over to find out after what string we are appending these filenames.

Yeah, makes sense.

Patrick

Re: [PATCH v2 5/7] t0027: cache expected output files

[Patrick Steinhardt]

On Mon, Nov 09, 2026 at 03:46:51PM +0100, Torsten Bögershausen wrote:
> On Mon, Nov 09, 2026 at 10:58:26AM +0100, Patrick Steinhardt wrote:
> > In the preceding commit we have optimized one aspect of t0027 already.
> > Another inefficiency is `compare_files ()` and `compare_ws_file ()`,
> > which are both called more than a 1000 times.
> > 
> > These functions will repeatedly generate the same expected outcome even
> > though there's only a small handful of input files that the expected
> > outcome is computed from. Recomputing those every time is of course
> > quite pointless.
> > 
> > Improve this by computing the expected outcome once, only, to reduce the
> > number of spawned processes. This reduces the runtime by around 20-25%,
> > and on Windows the runtime decreases from 480 to 380 seconds.
> > 
> > Signed-off-by: Patrick Steinhardt <ps@pks.im>
> > ---
> >  t/t0027-auto-crlf.sh | 17 +++++++++--------
> >  1 file changed, 9 insertions(+), 8 deletions(-)
> > 
> > diff --git a/t/t0027-auto-crlf.sh b/t/t0027-auto-crlf.sh
> > index 504a7ea54bb..b021f0855a8 100755
> > --- a/t/t0027-auto-crlf.sh
> > +++ b/t/t0027-auto-crlf.sh
> > @@ -5,22 +5,23 @@ test_description='CRLF conversion all combinations'
> >  . ./test-lib.sh
> >  
> >  compare_files () {
> > -	tr '\015\000' QN <"$1" >"$1".expect &&
> > +	{ test -f "$1".expect || tr '\015\000' QN <"$1" >"$1".expect; } &&
> >  	tr '\015\000' QN <"$2" | tr -d 'Z' >"$2".actual &&
> > -	test_cmp "$1".expect "$2".actual &&
> > -	rm "$1".expect "$2".actual
> > +	test_cmp "$1".expect "$2".actual
> >  }
> >  
> 
> Nice.
> If you want to save even more CPU/SSD-cycles,
> you can prepare the expect files like this:
> (No, I didn't do any benchmarks, nor look at ws_compare)

Thanks! I benchmarked this and it doesn't seem to make much of a
difference. So I'd propose to stick with the simple change for now, if
that's alright with you.

Patrick

Re: [PATCH v2 5/7] t0027: cache expected output files

[Torsten Bögershausen]

> 
> Thanks! I benchmarked this and it doesn't seem to make much of a
> difference. So I'd propose to stick with the simple change for now, if
> that's alright with you.

Good to hear that the benchmark was done.
OK to stick your patch "as is"

Re: [PATCH v2 1/7] gitlab-ci: speed up installation of Git for Windows SDK

[Junio C Hamano]

Patrick Steinhardt <ps@pks.im> writes:

> On Mon, Nov 09, 2026 at 08:53:53AM -0800, Junio C Hamano wrote:
>> Patrick Steinhardt <ps@pks.im> writes:
>> 
>> > +tar -xf git-sdk.zip -C $directory
>> 
>> Wow.  You learn new things every day.
>
> Yeah, it's fun. Windows uses bsdtar, which is in turn based on
> libarchive, and hence it also supports ZIP and 7zip files [1].
>
>> This is in ci/install-sdk.ps1 file.  Right now that script is
>> referred to only from .gitlab-ci.yml so as long as GitLab CI
>> environment has tar that works this way we would be fine, but I
>> thought that ci/* was designed to be portable across CI and other
>> environments.  At least we would want a comment next to it that says
>> we assume BSD derived tar that can grok zip archives here, probably?
>
> Well, bsdtar is nowadays distributed with Windows by default, so this
> should be portable to other CI environments, too. So I don't think it
> really warrants a comment as it should just work everwhere. That's
> certainly something to point out in the commit message though.

Yeah, this installer is about Windows, so any current or future CI
that uses this script would end up running BSD tar, which means we
can assume readers know what the script is doing and why it works.

> But now that we have found a way to remove the slowness we could unify
> both systems again. I intentionally left that out of this patch series
> though, as we can still do that as a subsequent step if we want to. In
> any case, I want Johannes' opinion on this topic before doing such a
> change.

Yes, indeed.  Thanks for these patches.

[PATCH v3 0/7] ci: some speedups for slow tests

[Patrick Steinhardt]

Hi,

I was staring at CI way more often recently, and that got me annoyed
real fast because it's just painstakingly slow. So I decided to have a
look and try do something about it, the result of which is this patch
series.

The series contains a couple of bits that are specific to GitLab, and a
couple of fixes that are generic. Some highlights:

  - Installing the Git for Windows SDK speeds up from 3 minutes to 10
    seconds by using tar(1) instead of Extract-Archive.

  - t0027 speeds up from >10 minutes to 6 minutes on Windows by avoiding
    the overhead of spawning redundant processes.

  - t5551 and t5559 both speed up by a factor of 10 on macOS, going from
    20 minutes each to 2 minutes each by using reftables in a repo with
    100k refs. "osx-meson" and "osx-clang" thus go from 29 minutes to 18
    minutes end-to-end.

  - "fedora-breaking-changes-meson" speeds up by a factor of 2, going
    from 35 minutes to 16 minutes. Turns out our SHA256 implementation
    is quite slow with "-O0".

Unfortunately, our Windows runners at GitLab are still atrociously slow
by themselves. So the critical path only improves by about ~10 minutes,
going from 60-70 minutes to 53 minutes end-to-end. But hey, at least the
other jobs now give feedback a bit faster. And I guess on GitHub it may
be a different story because Windows runners are way better there and
are typically not on the critical path.

A test run can be found at [1].

Changes in v3:
  - Document why Windows' tar version can extract ZIP archives.
  - Clear `paths` variable.
  - Link to v2: https://patch.msgid.link/20261109-pks-gitlab-ci-speedups-v2-0-508832602d0b@pks.im

Changes in v2:
  - Drop the cache for the GfW SDK again.
  - Drop repetitive paths in t0027.
  - Link to v1: https://patch.msgid.link/20261106-pks-gitlab-ci-speedups-v1-0-1fa0374c7287@pks.im

Thanks!

Patrick

[1]: https://gitlab.com/gitlab-org/git/-/merge_requests/734

---
Patrick Steinhardt (7):
      gitlab-ci: speed up installation of Git for Windows SDK
      gitlab-ci: exclude intermediate outputs from Windows artifacts
      ci: build optimized binaries with Meson
      t0027: optimize how we reset repository state
      t0027: cache expected output files
      t1092: set up repositories from templates
      t5551: use reftables for repositories with many refs

 .gitlab-ci.yml                           |  4 +++-
 ci/install-sdk.ps1                       |  6 +++++-
 ci/run-build-and-tests.sh                |  1 +
 t/t0027-auto-crlf.sh                     | 27 ++++++++++++++-----------
 t/t1092-sparse-checkout-compatibility.sh | 34 +++++++++++++++++++-------------
 t/t5551-http-fetch-smart.sh              |  6 ++++--
 6 files changed, 48 insertions(+), 30 deletions(-)

Range-diff versus v2:

1:  41c3a0e8d7d ! 1:  8fabb336df3 gitlab-ci: speed up installation of Git for Windows SDK
    @@ Commit message
         Windows SDK on every run and then extracts it via `Expand-Archive`,
         which is expensive.
     
    -    Improve this by using tar(1) instead of `Expand-Archive`. While at it,
    -    make the script error on failures and suppress progress output.
    +    Improve this by using tar(1) instead of `Expand-Archive`. Note that
    +    we're using it to extract a ZIP archive, which may seem puzzling at
    +    first. But Windows ships bsdtar by default [1], which also supports ZIP
    +    and 7zip files.
    +
    +    While at it, make the script error on failures and suppress progress
    +    output.
    +
    +    [1]: https://learn.microsoft.com/en-us/windows/tar/
     
         Signed-off-by: Patrick Steinhardt <ps@pks.im>
     
2:  439c12cb15d = 2:  4a4d1e6f2a8 gitlab-ci: exclude intermediate outputs from Windows artifacts
3:  070bef6fecf = 3:  c23b7a7c58d ci: build optimized binaries with Meson
4:  598a143fc24 ! 4:  942d70be750 t0027: optimize how we reset repository state
    @@ Commit message
     
      ## t/t0027-auto-crlf.sh ##
     @@ t/t0027-auto-crlf.sh: checkout_files () {
    + 		git config core.autocrlf $crlf
    + 	'
      	pfx=eol_${ceol}_crlf_${crlf}_attr_${attr}_ &&
    ++	paths=
      	for f in LF CRLF LF_mix_CR CRLF_mix_LF LF_nul
      	do
     -		test_expect_success "setup $f checkout ${ceol:+ with -c core.eol=$ceol}"  '
5:  a325efbaddb = 5:  460cd0c88fd t0027: cache expected output files
6:  a2d33441e50 = 6:  44c77717749 t1092: set up repositories from templates
7:  bfc93fdfb4e = 7:  d64a61efec2 t5551: use reftables for repositories with many refs

---
base-commit: 0ffa3e35c7c82ab8d0de226149fb03b3f7674d05
change-id: 20261106-pks-gitlab-ci-speedups-09c3841eb178

[PATCH v3 1/7] gitlab-ci: speed up installation of Git for Windows SDK

[Patrick Steinhardt]

The "build:mingw64" job on GitLab CI downloads the minimal Git for
Windows SDK on every run and then extracts it via `Expand-Archive`,
which is expensive.

Improve this by using tar(1) instead of `Expand-Archive`. Note that
we're using it to extract a ZIP archive, which may seem puzzling at
first. But Windows ships bsdtar by default [1], which also supports ZIP
and 7zip files.

While at it, make the script error on failures and suppress progress
output.

[1]: https://learn.microsoft.com/en-us/windows/tar/

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 ci/install-sdk.ps1 | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/ci/install-sdk.ps1 b/ci/install-sdk.ps1
index 66f24838a40..bb332c1c10f 100755
--- a/ci/install-sdk.ps1
+++ b/ci/install-sdk.ps1
@@ -3,8 +3,12 @@ param(
     [string]$url='https://github.com/git-for-windows/git-sdk-64/releases/download/ci-artifacts/git-sdk-x86_64-minimal.zip'
 )
 
+$ErrorActionPreference = 'Stop'
+$ProgressPreference = 'SilentlyContinue'
+
 Invoke-WebRequest "$url" -OutFile git-sdk.zip
-Expand-Archive -LiteralPath git-sdk.zip -DestinationPath "$directory"
+New-Item -Path $directory -ItemType Directory -Force
+tar -xf git-sdk.zip -C $directory
 Remove-Item -Path git-sdk.zip
 
 New-Item -Path .git/info -ItemType Directory -Force

-- 
2.56.0.611.g1030f0b940.dirty

[PATCH v3 2/7] gitlab-ci: exclude intermediate outputs from Windows artifacts

[Patrick Steinhardt]

As running tests is very slow on Windows we shard our CI tests across 10
runners. To avoid rebuilding Git on each of those runners we have a
preliminary step which builds Git and then passes on the build artifacts
to the test jobs.

This archive contains a ton of intermediate "*.obj" build artifacts that
aren't needed by the test jobs: the artifact contains ~2300 files, out
of which ~1800 are "*.obj" files.

Exclude those files from the build artifact, which reduces the time it
takes to download and extract them from 34 seconds to 17 seconds.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 .gitlab-ci.yml | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index 9b7698a0037..3631393a8cc 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -187,6 +187,8 @@ build:msvc-meson:
   artifacts:
     paths:
       - build
+    exclude:
+      - build/**/*.obj
 
 test:msvc-meson:
   extends: .msvc-meson

-- 
2.56.0.611.g1030f0b940.dirty

[PATCH v3 3/7] ci: build optimized binaries with Meson

[Patrick Steinhardt]

When taking a look at our CI jobs one will quickly notice that across
the Linux-based jobs, the "fedora-breaking-changes-meson" job is running
significantly longer than any of the other jobs. While most Linux jobs
only run around 15 minutes, that particular job runs roughly twice as
long.

The root cause of this is a combination of two different things:

  - The job uses Meson to compile Git, and Meson by default uses the
    "debug" build type that translates into "-O0". Our Makefile in
    contrast to that uses "-g -O2" by default.

  - The job enables breaking changes and thus runs with SHA256 by
    default with the "block" backend, which benefits heavily from
    compiler optimizations. For example, hashing a 500MB blob with
    SHA256 takes 9.6 seconds with "-O0", whereas it takes only 1.5
    seconds with "-O2". The SHA1 code path only slows down from 0.9
    seconds to 2.1 seconds with "-O0".

Of course, compiling without optimizations also has an impact on all the
other jobs, but the Fedora job is by far the most impacted one.

Speed up executing the tests by using the "debugoptimized" build type
instead. This reduces the runtime of "fedora-breaking-changes-meson"
from 2100 to 1000 seconds. Other Linux- and macOS-based jobs benefit as
well, but they only speed up by around ~10%.

On Windows though this is a bit more muddy, as the build step gets
almost two minutes slower, whereas tests only run ~10% faster. Combined
though the total run time is roughly the same before and after this
change.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 .gitlab-ci.yml            | 2 +-
 ci/run-build-and-tests.sh | 1 +
 2 files changed, 2 insertions(+), 1 deletion(-)

diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index 3631393a8cc..449fed1b392 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -182,7 +182,7 @@ build:msvc-meson:
   extends: .msvc-meson
   stage: build
   script:
-    - meson setup build --vsenv -Dperl=disabled -Dbackend_max_links=1 -Dcredential_helpers=wincred -Dtest_output_directory="$TEST_OUTPUT_DIRECTORY"
+    - meson setup build --vsenv --buildtype=debugoptimized -Dperl=disabled -Dbackend_max_links=1 -Dcredential_helpers=wincred -Dtest_output_directory="$TEST_OUTPUT_DIRECTORY"
     - meson compile -C build
   artifacts:
     paths:
diff --git a/ci/run-build-and-tests.sh b/ci/run-build-and-tests.sh
index 68b91e31439..fd8136af342 100755
--- a/ci/run-build-and-tests.sh
+++ b/ci/run-build-and-tests.sh
@@ -55,6 +55,7 @@ case "$jobname" in
 		--fatal-meson-warnings \
 		--warnlevel 2 --werror \
 		--wrap-mode nofallback \
+		--buildtype=debugoptimized \
 		-Dfuzzers=true \
 		-Dtest_output_directory="${TEST_OUTPUT_DIRECTORY:-$(pwd)/t}" \
 		$MESONFLAGS

-- 
2.56.0.611.g1030f0b940.dirty

[PATCH v3 4/7] t0027: optimize how we reset repository state

[Patrick Steinhardt]

In t0027 we verify our logic to convert CRLF for all kinds of
combinations of attributes and configuration, which ultimately results
in 2600 test cases. This is of course on the heavier side, so on Windows
this takes more than 10 minutes to run.

One source of slowness is `checkout_files ()`, which gets called for
a lot of our test to set up the initial working directory. As part of
that we both remove the old files that we care about and then check them
out again via git-checkout(1), but we do this one by one for every
single file.

This is of course rather wasteful, as we have to spawn ten different
processes every time the function is invoked. And on top of that, we
also have to rewrite the complete index in git-checkout(1) every time.

Speed this up by instead using a single invocation of rm(1) and
git-checkout(1). This reduces the runtime by about 20% on all platforms,
and on Windows this reduces the runtime by ~2.5 minutes.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 t/t0027-auto-crlf.sh | 10 ++++++----
 1 file changed, 6 insertions(+), 4 deletions(-)

diff --git a/t/t0027-auto-crlf.sh b/t/t0027-auto-crlf.sh
index 49dbf09da77..a3904b729b3 100755
--- a/t/t0027-auto-crlf.sh
+++ b/t/t0027-auto-crlf.sh
@@ -303,13 +303,15 @@ checkout_files () {
 		git config core.autocrlf $crlf
 	'
 	pfx=eol_${ceol}_crlf_${crlf}_attr_${attr}_ &&
+	paths=
 	for f in LF CRLF LF_mix_CR CRLF_mix_LF LF_nul
 	do
-		test_expect_success "setup $f checkout ${ceol:+ with -c core.eol=$ceol}"  '
-			rm -f crlf_false_attr__$f.txt &&
-			git ${ceol:+-c core.eol=$ceol} checkout -- crlf_false_attr__$f.txt
-		'
+		paths="${paths:+"$paths "}crlf_false_attr__$f.txt"
 	done
+	test_expect_success "setup checkout ${ceol:+ with -c core.eol=$ceol}" '
+		rm -f $paths &&
+		git ${ceol:+-c core.eol=$ceol} checkout -- $paths
+	'
 
 	test_expect_success "ls-files --eol attr=$attr $ident aeol=$aeol core.autocrlf=$crlf core.eol=$ceol" '
 		test_when_finished "rm expect actual" &&

-- 
2.56.0.611.g1030f0b940.dirty

[PATCH v3 5/7] t0027: cache expected output files

[Patrick Steinhardt]

In the preceding commit we have optimized one aspect of t0027 already.
Another inefficiency is `compare_files ()` and `compare_ws_file ()`,
which are both called more than a 1000 times.

These functions will repeatedly generate the same expected outcome even
though there's only a small handful of input files that the expected
outcome is computed from. Recomputing those every time is of course
quite pointless.

Improve this by computing the expected outcome once, only, to reduce the
number of spawned processes. This reduces the runtime by around 20-25%,
and on Windows the runtime decreases from 480 to 380 seconds.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 t/t0027-auto-crlf.sh | 17 +++++++++--------
 1 file changed, 9 insertions(+), 8 deletions(-)

diff --git a/t/t0027-auto-crlf.sh b/t/t0027-auto-crlf.sh
index a3904b729b3..38ce06cbc1e 100755
--- a/t/t0027-auto-crlf.sh
+++ b/t/t0027-auto-crlf.sh
@@ -5,22 +5,23 @@ test_description='CRLF conversion all combinations'
 . ./test-lib.sh
 
 compare_files () {
-	tr '\015\000' QN <"$1" >"$1".expect &&
+	{ test -f "$1".expect || tr '\015\000' QN <"$1" >"$1".expect; } &&
 	tr '\015\000' QN <"$2" | tr -d 'Z' >"$2".actual &&
-	test_cmp "$1".expect "$2".actual &&
-	rm "$1".expect "$2".actual
+	test_cmp "$1".expect "$2".actual
 }
 
 compare_ws_file () {
 	pfx=$1
-	exp=$2.expect
+	exp=$2.ws-expect
 	act=$pfx.actual.$3
-	tr '\015\000abcdef0123456789' QN00000000000000000 <"$2" |
-		sed -e "s/0000*/$ZERO_OID/" >"$exp" &&
+	{
+		test -f "$exp" ||
+		tr '\015\000abcdef0123456789' QN00000000000000000 <"$2" |
+			sed -e "s/0000*/$ZERO_OID/" >"$exp" ;
+	} &&
 	tr '\015\000abcdef0123456789' QN00000000000000000 <"$3" |
 		sed -e "s/0000*/$ZERO_OID/" >"$act" &&
-	test_cmp "$exp" "$act" &&
-	rm "$exp" "$act"
+	test_cmp "$exp" "$act"
 }
 
 create_gitattributes () {

-- 
2.56.0.611.g1030f0b940.dirty

[PATCH v3 6/7] t1092: set up repositories from templates

[Patrick Steinhardt]

In t1092 we verify the behaviour of sparse checkouts. We use three
different repositories to do so: one with a full checkout, one with a
sparse checkout and one with a sparse index, and then verify that the
result is the same across all these configurations.

Those repositories get set up via `init_repos ()`, which initializes
them from scratch every single time with a bunch of processes. The
function is called more than a 100 times though, so the overhead adds up
quite a bit. While the tests only require around 30 seconds on Linux,
it takes more than 10 minutes on Windows.

Optimize this by instead creating the repositories once, only. So
instead of recreating them from scratch, we can then just copy the
template repositories into place and thus save a bunch of processes.
This speeds up tests on Windows by about 3 minutes.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 t/t1092-sparse-checkout-compatibility.sh | 34 +++++++++++++++++++-------------
 1 file changed, 20 insertions(+), 14 deletions(-)

diff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh
index 05b54062b3b..5d81ec54c23 100755
--- a/t/t1092-sparse-checkout-compatibility.sh
+++ b/t/t1092-sparse-checkout-compatibility.sh
@@ -141,28 +141,34 @@ test_expect_success 'setup' '
 	)
 '
 
+test_expect_success 'setup template repositories' '
+	cp -r initial-repo full-checkout.template &&
+	cp -r initial-repo sparse-checkout.template &&
+	cp -r initial-repo sparse-index.template &&
+
+	# initialize sparse-checkout definitions
+	git -C sparse-checkout.template sparse-checkout init --cone &&
+	git -C sparse-checkout.template sparse-checkout set deep &&
+	git -C sparse-index.template sparse-checkout init --cone --sparse-index &&
+	test_cmp_config -C sparse-index.template true index.sparse &&
+	git -C sparse-index.template sparse-checkout set deep &&
+
+	# Disable this message to keep stderr the same.
+	git -C sparse-index.template config advice.sparseIndexExpanded false
+'
+
 init_repos () {
 	rm -rf full-checkout sparse-checkout sparse-index &&
 
 	# create repos in initial state
-	cp -r initial-repo full-checkout &&
+	cp -r full-checkout.template full-checkout &&
 	git -C full-checkout reset --hard &&
 
-	cp -r initial-repo sparse-checkout &&
+	cp -r sparse-checkout.template sparse-checkout &&
 	git -C sparse-checkout reset --hard &&
 
-	cp -r initial-repo sparse-index &&
-	git -C sparse-index reset --hard &&
-
-	# initialize sparse-checkout definitions
-	git -C sparse-checkout sparse-checkout init --cone &&
-	git -C sparse-checkout sparse-checkout set deep &&
-	git -C sparse-index sparse-checkout init --cone --sparse-index &&
-	test_cmp_config -C sparse-index true index.sparse &&
-	git -C sparse-index sparse-checkout set deep &&
-
-	# Disable this message to keep stderr the same.
-	git -C sparse-index config advice.sparseIndexExpanded false
+	cp -r sparse-index.template sparse-index &&
+	git -C sparse-index reset --hard
 }
 
 init_repos_as_submodules () {

-- 
2.56.0.611.g1030f0b940.dirty

[PATCH v3 7/7] t5551: use reftables for repositories with many refs

[Patrick Steinhardt]

Some of the tests in t5551 creates almost a 100,000 references and then
verify that those can be fetched. Naturally, creating this many refs is
quite involved with the "files" backend, as it means that we have to
create this many individual loose files. This is tolerable on Linux,
where the test takes around 2 minutes in our CI. But on macOS, the same
test takes almost 20 minutes to complete. And to add insult to injury,
we execute these tests a second time via t5559.

The system under test here is the HTTP negotiation, so we don't actually
care about whether or not the ref backend handles this alright. So adapt
the test to instead create repositories with the reftable backend, which
is significantly faster in this case as we only need to write a single
table.

And sure enough, this reduces the runtime of this test on macOS by a
factor of 10, going from 20 minutes to 2 minutes. On Linux, the tests at
least speed up by a factor of two.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 t/t5551-http-fetch-smart.sh | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)

diff --git a/t/t5551-http-fetch-smart.sh b/t/t5551-http-fetch-smart.sh
index 130053c57aa..34945059531 100755
--- a/t/t5551-http-fetch-smart.sh
+++ b/t/t5551-http-fetch-smart.sh
@@ -412,7 +412,8 @@ create_tags () {
 }
 
 test_expect_success 'create 2,000 tags in the repo' '
-	git init --bare "$HTTPD_DOCUMENT_ROOT_PATH/many-tags.git" &&
+	git init --bare --ref-format=reftable \
+		"$HTTPD_DOCUMENT_ROOT_PATH/many-tags.git" &&
 	(
 		cd "$HTTPD_DOCUMENT_ROOT_PATH/many-tags.git" &&
 		create_tags 1 2000
@@ -421,7 +422,8 @@ test_expect_success 'create 2,000 tags in the repo' '
 
 test_expect_success CMDLINE_LIMIT \
 	'clone the 2,000 tag repo to check OS command line overflow' '
-	run_with_limited_cmdline git clone $HTTPD_URL/smart/many-tags.git too-many-refs &&
+	run_with_limited_cmdline git clone --ref-format=reftable \
+		$HTTPD_URL/smart/many-tags.git too-many-refs &&
 	(
 		cd too-many-refs &&
 		git for-each-ref refs/tags >actual &&

-- 
2.56.0.611.g1030f0b940.dirty

Re: [PATCH v2 0/7] ci: some speedups for slow tests

[Johannes Sixt]

Am 09.11.26 um 10:58 schrieb Patrick Steinhardt:
> The series contains a couple of bits that are specific to GitLab, and a
> couple of fixes that are generic. Some highlights:
> 
>   - Installing the Git for Windows SDK speeds up from 3 minutes to 10
>     seconds by using tar(1) instead of Extract-Archive.
> 
>   - t0027 speeds up from >10 minutes to 6 minutes on Windows by avoiding
>     the overhead of spawning redundant processes.
> 
>   - t5551 and t5559 both speed up by a factor of 10 on macOS, going from
>     20 minutes each to 2 minutes each by using reftables in a repo with
>     100k refs. "osx-meson" and "osx-clang" thus go from 29 minutes to 18
>     minutes end-to-end.
> 
>   - "fedora-breaking-changes-meson" speeds up by a factor of 2, going
>     from 35 minutes to 16 minutes. Turns out our SHA256 implementation
>     is quite slow with "-O0".
> 
> Unfortunately, our Windows runners at GitLab are still atrociously slow
> by themselves. So the critical path only improves by about ~10 minutes,
> going from 60-70 minutes to 53 minutes end-to-end. But hey, at least the
> other jobs now give feedback a bit faster. And I guess on GitHub it may
> be a different story because Windows runners are way better there and
> are typically not on the critical path.


I've tested this series on my Windows notebook and it is looking good.

The runtime of t0027 went down from 4:45 to 3:14, and that of t1092 from
6:15 to 5:33. t5551 is skipped in my setup.

Thanks!

-- Hannes