[PATCH 0/7] ci: some speedups for slow tests

[PATCH 0/7] ci: some speedups for slow tests

[Patrick Steinhardt]

Hi,

I was staring at CI way more often recently, and that got me annoyed
real fast because it's just painstakingly slow. So I decided to have a
look and try do something about it, the result of which is this patch
series.

The series contains a couple of bits that are specific to GitLab, and a
couple of fixes that are generic. Some highlights:

  - Installing the Git for Windows SDK speeds up from 3 minutes to 10
    seconds by using tar(1) instead of Extract-Archive.

  - t0027 speeds up from >10 minutes to 6 minutes on Windows by avoiding
    the overhead of spawning redundant processes.

  - t5551 and t5559 both speed up by a factor of 10 on macOS, going from
    20 minutes each to 2 minutes each by using reftables in a repo with
    100k refs. "osx-meson" and "osx-clang" thus go from 29 minutes to 18
    minutes end-to-end.

  - "fedora-breaking-changes-meson" speeds up by a factor of 2, going
    from 35 minutes to 16 minutes. Turns out our SHA256 implementation
    is quite slow with "-O0".

Unfortunately, our Windows runners at GitLab are still atrociously slow
by themselves. So the critical path only improves by about ~10 minutes,
going from 60-70 minutes to 53 minutes end-to-end. But hey, at least the
other jobs now give feedback a bit faster. And I guess on GitHub it may
be a different story because Windows runners are way better there and
are typically not on the critical path.

A test run can be found at [1].

Thanks!

Patrick

[1]: https://gitlab.com/gitlab-org/git/-/merge_requests/734

---
Patrick Steinhardt (7):
      gitlab-ci: speed up installation of Git for Windows SDK
      gitlab-ci: exclude intermediate outputs from Windows artifacts
      ci: build optimized binaries with Meson
      t0027: optimize how we reset repository state
      t0027: cache expected output files
      t1092: set up repositories from templates
      t5551: use reftables for repositories with many refs

 .gitlab-ci.yml                           | 13 ++++++++++--
 ci/install-sdk.ps1                       | 26 ++++++++++++++++++------
 ci/run-build-and-tests.sh                |  1 +
 t/t0027-auto-crlf.sh                     | 33 +++++++++++++++++--------------
 t/t1092-sparse-checkout-compatibility.sh | 34 +++++++++++++++++++-------------
 t/t5551-http-fetch-smart.sh              |  6 ++++--
 6 files changed, 74 insertions(+), 39 deletions(-)


---
base-commit: 0ffa3e35c7c82ab8d0de226149fb03b3f7674d05
change-id: 20261106-pks-gitlab-ci-speedups-09c3841eb178

[PATCH 1/7] gitlab-ci: speed up installation of Git for Windows SDK

[Patrick Steinhardt]

The "build:mingw64" job on GitLab CI downloads the minimal Git for
Windows SDK on every run and then extracts it via `Expand-Archive`,
which is expensive.

Improve this by caching the SDK and by using tar(1) instead of
`Expand-Archive`. Note that caching actually only gives us a win of a
couple seconds. The bigger win here is using tar(1), which speeds up
extracting the archive from 3 minutes to about 10 seconds. Regardless of
that though, caching the SDK feels like a sensible thing to do as it
removes one step that may spuriously fail.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 .gitlab-ci.yml     |  9 ++++++++-
 ci/install-sdk.ps1 | 26 ++++++++++++++++++++------
 2 files changed, 28 insertions(+), 7 deletions(-)

diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index 9b7698a0037..15419a954b3 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -127,9 +127,16 @@ build:mingw64:
     - saas-windows-medium-amd64
   variables:
     NO_PERL: 1
+  cache:
+    key:
+      files:
+        - ci/install-sdk.ps1
+        - ci/install-dependencies.ps1
+    paths:
+      - .dependencies
   before_script:
     - *windows_before_script
-    - ./ci/install-sdk.ps1 -directory "git-sdk"
+    - ./ci/install-sdk.ps1 -Directory "git-sdk"
     - ./ci/install-dependencies.ps1 -Mingw
   script:
     # The minimal SDK's profile resets PATH.
diff --git a/ci/install-sdk.ps1 b/ci/install-sdk.ps1
index 66f24838a40..1dbcef04a18 100755
--- a/ci/install-sdk.ps1
+++ b/ci/install-sdk.ps1
@@ -1,12 +1,26 @@
 param(
-    [string]$directory='git-sdk',
-    [string]$url='https://github.com/git-for-windows/git-sdk-64/releases/download/ci-artifacts/git-sdk-x86_64-minimal.zip'
+    [string]$Directory='git-sdk',
+    [string]$URL='https://github.com/git-for-windows/git-sdk-64/releases/download/ci-artifacts/git-sdk-x86_64-minimal.zip',
+    [string]$DownloadDirectory='.dependencies'
 )
 
-Invoke-WebRequest "$url" -OutFile git-sdk.zip
-Expand-Archive -LiteralPath git-sdk.zip -DestinationPath "$directory"
-Remove-Item -Path git-sdk.zip
+$ErrorActionPreference = 'Stop'
+$ProgressPreference = 'SilentlyContinue'
+
+New-Item -Path $DownloadDirectory -ItemType Directory -Force
+
+$archive = Join-Path $DownloadDirectory 'git-sdk.zip'
+if (-not (Test-Path $archive)) {
+    Invoke-WebRequest "$URL" -OutFile $archive -TimeoutSec 300
+}
+
+New-Item -Path $Directory -ItemType Directory -Force
+tar -xf $archive -C $Directory
+if ($LASTEXITCODE -ne 0) {
+    throw "Failed to extract $archive with exit code $LASTEXITCODE"
+}
 
 New-Item -Path .git/info -ItemType Directory -Force
 New-Item -Path .git/info/exclude -ItemType File -Force
-Add-Content -Path .git/info/exclude -Value "/$directory"
+Add-Content -Path .git/info/exclude -Value "/$Directory"
+Add-Content -Path .git/info/exclude -Value "/$DownloadDirectory"

-- 
2.56.0.500.g5d06dc02cd.dirty

Re: [PATCH 1/7] gitlab-ci: speed up installation of Git for Windows SDK

[Junio C Hamano]

Patrick Steinhardt <ps@pks.im> writes:

> The "build:mingw64" job on GitLab CI downloads the minimal Git for
> Windows SDK on every run and then extracts it via `Expand-Archive`,
> which is expensive.
>
> Improve this by caching the SDK and by using tar(1) instead of
> `Expand-Archive`.
> Note that caching actually only gives us a win of a
> couple seconds. The bigger win here is using tar(1), which speeds up
> extracting the archive from 3 minutes to about 10 seconds. Regardless of

Wow, is a tool to move the bits from one part of a rusted iron
surface to another part really so much more expensive compared to
another tool to move the same bits, to make a difference that is
worth doing this?

> that though, caching the SDK feels like a sensible thing to do as it
> removes one step that may spuriously fail.

One thing that may make readers curious is if we need to worry about
this cached copy going stale.

> Signed-off-by: Patrick Steinhardt <ps@pks.im>
> ---
>  .gitlab-ci.yml     |  9 ++++++++-
>  ci/install-sdk.ps1 | 26 ++++++++++++++++++++------
>  2 files changed, 28 insertions(+), 7 deletions(-)
>
> diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
> index 9b7698a0037..15419a954b3 100644
> --- a/.gitlab-ci.yml
> +++ b/.gitlab-ci.yml
> @@ -127,9 +127,16 @@ build:mingw64:
>      - saas-windows-medium-amd64
>    variables:
>      NO_PERL: 1
> +  cache:
> +    key:
> +      files:
> +        - ci/install-sdk.ps1
> +        - ci/install-dependencies.ps1
> +    paths:
> +      - .dependencies
>    before_script:
>      - *windows_before_script
> -    - ./ci/install-sdk.ps1 -directory "git-sdk"
> +    - ./ci/install-sdk.ps1 -Directory "git-sdk"

Is this change essential to understand this patch, or is it an
unrelated general clean-up to follow some style rules?  I notice
that ...

>      - ./ci/install-dependencies.ps1 -Mingw
>    script:
>      # The minimal SDK's profile resets PATH.
> diff --git a/ci/install-sdk.ps1 b/ci/install-sdk.ps1
> index 66f24838a40..1dbcef04a18 100755
> --- a/ci/install-sdk.ps1
> +++ b/ci/install-sdk.ps1
> @@ -1,12 +1,26 @@
>  param(
> -    [string]$directory='git-sdk',
> -    [string]$url='https://github.com/git-for-windows/git-sdk-64/releases/download/ci-artifacts/git-sdk-x86_64-minimal.zip'
> +    [string]$Directory='git-sdk',
> +    [string]$URL='https://github.com/git-for-windows/git-sdk-64/releases/download/ci-artifacts/git-sdk-x86_64-minimal.zip',
> +    [string]$DownloadDirectory='.dependencies'

... we are making a matching change here to upcase the parameter
names here.

[PATCH 2/7] gitlab-ci: exclude intermediate outputs from Windows artifacts

[Patrick Steinhardt]

As running tests is very slow on Windows we shard our CI tests across 10
runners. To avoid rebuilding Git on each of those runners we have a
preliminary step which builds Git and then passes on the build artifacts
to the test jobs.

This archive contains a ton of intermediate "*.obj" build artifacts that
aren't needed by the test jobs: the artifact contains ~2300 files, out
of which ~1800 are "*.obj" files.

Exclude those files from the build artifact, which reduces the time it
takes to download and extract them from 34 seconds to 17 seconds.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 .gitlab-ci.yml | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index 15419a954b3..036d2f3c7f2 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -194,6 +194,8 @@ build:msvc-meson:
   artifacts:
     paths:
       - build
+    exclude:
+      - build/**/*.obj
 
 test:msvc-meson:
   extends: .msvc-meson

-- 
2.56.0.500.g5d06dc02cd.dirty

[PATCH 3/7] ci: build optimized binaries with Meson

[Patrick Steinhardt]

When taking a look at our CI jobs one will quickly notice that across
the Linux-based jobs, the "fedora-breaking-changes-meson" job is running
significantly longer than any of the other jobs. While most Linux jobs
only run around 15 minutes, that particular job runs roughly twice as
long.

The root cause of this is a combination of two different things:

  - The job uses Meson to compile Git, and Meson by default uses the
    "debug" build type that translates into "-O0". Our Makefile in
    contrast to that uses "-g -O2" by default.

  - The job enables breaking changes and thus runs with SHA256 by
    default with the "block" backend, which benefits heavily from
    compiler optimizations. For example, hashing a 500MB blob with
    SHA256 takes 9.6 seconds with "-O0", whereas it takes only 1.5
    seconds with "-O2". The SHA1 code path only slows down from 0.9
    seconds to 2.1 seconds with "-O0".

Of course, compiling without optimizations also has an impact on all the
other jobs, but the Fedora job is by far the most impacted one.

Speed up executing the tests by using the "debugoptimized" build type
instead. This reduces the runtime of "fedora-breaking-changes-meson"
from 2100 to 1000 seconds. Other Linux- and macOS-based jobs benefit as
well, but they only speed up by around ~10%.

On Windows though this is a bit more muddy, as the build step gets
almost two minutes slower, whereas tests only run ~10% faster. Combined
though the total run time is roughly the same before and after this
change.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 .gitlab-ci.yml            | 2 +-
 ci/run-build-and-tests.sh | 1 +
 2 files changed, 2 insertions(+), 1 deletion(-)

diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index 036d2f3c7f2..df82cd3a5df 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -189,7 +189,7 @@ build:msvc-meson:
   extends: .msvc-meson
   stage: build
   script:
-    - meson setup build --vsenv -Dperl=disabled -Dbackend_max_links=1 -Dcredential_helpers=wincred -Dtest_output_directory="$TEST_OUTPUT_DIRECTORY"
+    - meson setup build --vsenv --buildtype=debugoptimized -Dperl=disabled -Dbackend_max_links=1 -Dcredential_helpers=wincred -Dtest_output_directory="$TEST_OUTPUT_DIRECTORY"
     - meson compile -C build
   artifacts:
     paths:
diff --git a/ci/run-build-and-tests.sh b/ci/run-build-and-tests.sh
index 68b91e31439..fd8136af342 100755
--- a/ci/run-build-and-tests.sh
+++ b/ci/run-build-and-tests.sh
@@ -55,6 +55,7 @@ case "$jobname" in
 		--fatal-meson-warnings \
 		--warnlevel 2 --werror \
 		--wrap-mode nofallback \
+		--buildtype=debugoptimized \
 		-Dfuzzers=true \
 		-Dtest_output_directory="${TEST_OUTPUT_DIRECTORY:-$(pwd)/t}" \
 		$MESONFLAGS

-- 
2.56.0.500.g5d06dc02cd.dirty

Re: [PATCH 3/7] ci: build optimized binaries with Meson

[Junio C Hamano]

Patrick Steinhardt <ps@pks.im> writes:

> Speed up executing the tests by using the "debugoptimized" build type
> instead. This reduces the runtime of "fedora-breaking-changes-meson"
> from 2100 to 1000 seconds. Other Linux- and macOS-based jobs benefit as
> well, but they only speed up by around ~10%.

That is an impressive improvement.

> On Windows though this is a bit more muddy, as the build step gets
> almost two minutes slower, whereas tests only run ~10% faster. Combined
> though the total run time is roughly the same before and after this
> change.
>
> Signed-off-by: Patrick Steinhardt <ps@pks.im>
> ---
>  .gitlab-ci.yml            | 2 +-
>  ci/run-build-and-tests.sh | 1 +
>  2 files changed, 2 insertions(+), 1 deletion(-)
>
> diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
> index 036d2f3c7f2..df82cd3a5df 100644
> --- a/.gitlab-ci.yml
> +++ b/.gitlab-ci.yml
> @@ -189,7 +189,7 @@ build:msvc-meson:
>    extends: .msvc-meson
>    stage: build
>    script:
> -    - meson setup build --vsenv -Dperl=disabled -Dbackend_max_links=1 -Dcredential_helpers=wincred -Dtest_output_directory="$TEST_OUTPUT_DIRECTORY"
> +    - meson setup build --vsenv --buildtype=debugoptimized -Dperl=disabled -Dbackend_max_links=1 -Dcredential_helpers=wincred -Dtest_output_directory="$TEST_OUTPUT_DIRECTORY"
>      - meson compile -C build
>    artifacts:
>      paths:
> diff --git a/ci/run-build-and-tests.sh b/ci/run-build-and-tests.sh
> index 68b91e31439..fd8136af342 100755
> --- a/ci/run-build-and-tests.sh
> +++ b/ci/run-build-and-tests.sh
> @@ -55,6 +55,7 @@ case "$jobname" in
>  		--fatal-meson-warnings \
>  		--warnlevel 2 --werror \
>  		--wrap-mode nofallback \
> +		--buildtype=debugoptimized \
>  		-Dfuzzers=true \
>  		-Dtest_output_directory="${TEST_OUTPUT_DIRECTORY:-$(pwd)/t}" \
>  		$MESONFLAGS

[PATCH 4/7] t0027: optimize how we reset repository state

[Patrick Steinhardt]

In t0027 we verify our logic to convert CRLF for all kinds of
combinations of attributes and configuration, which ultimately results
in 2600 test cases. This is of course on the heavier side, so on Windows
this takes more than 10 minutes to run.

One source of slowness is `checkout_files ()`, which gets called for
a lot of our test to set up the initial working directory. As part of
that we both remove the old files that we care about and then check them
out again via git-checkout(1), but we do this one by one for every
single file.

This is of course rather wasteful, as we have to spawn ten different
processes every time the function is invoked. And on top of that, we
also have to rewrite the complete index in git-checkout(1) every time.

Speed this up by instead using a single invocation of rm(1) and
git-checkout(1). This reduces the runtime by about 20% on all platforms,
and on Windows this reduces the runtime by ~2.5 minutes.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 t/t0027-auto-crlf.sh | 16 +++++++++-------
 1 file changed, 9 insertions(+), 7 deletions(-)

diff --git a/t/t0027-auto-crlf.sh b/t/t0027-auto-crlf.sh
index 49dbf09da77..eefb346d11f 100755
--- a/t/t0027-auto-crlf.sh
+++ b/t/t0027-auto-crlf.sh
@@ -303,13 +303,15 @@ checkout_files () {
 		git config core.autocrlf $crlf
 	'
 	pfx=eol_${ceol}_crlf_${crlf}_attr_${attr}_ &&
-	for f in LF CRLF LF_mix_CR CRLF_mix_LF LF_nul
-	do
-		test_expect_success "setup $f checkout ${ceol:+ with -c core.eol=$ceol}"  '
-			rm -f crlf_false_attr__$f.txt &&
-			git ${ceol:+-c core.eol=$ceol} checkout -- crlf_false_attr__$f.txt
-		'
-	done
+	test_expect_success "setup checkout ${ceol:+ with -c core.eol=$ceol}" '
+		rm -f crlf_false_attr__LF.txt crlf_false_attr__CRLF.txt \
+			crlf_false_attr__LF_mix_CR.txt crlf_false_attr__CRLF_mix_LF.txt \
+			crlf_false_attr__LF_nul.txt &&
+		git ${ceol:+-c core.eol=$ceol} checkout -- \
+			crlf_false_attr__LF.txt crlf_false_attr__CRLF.txt \
+			crlf_false_attr__LF_mix_CR.txt crlf_false_attr__CRLF_mix_LF.txt \
+			crlf_false_attr__LF_nul.txt
+	'
 
 	test_expect_success "ls-files --eol attr=$attr $ident aeol=$aeol core.autocrlf=$crlf core.eol=$ceol" '
 		test_when_finished "rm expect actual" &&

-- 
2.56.0.500.g5d06dc02cd.dirty

Re: [PATCH 4/7] t0027: optimize how we reset repository state

[Junio C Hamano]

Patrick Steinhardt <ps@pks.im> writes:

>  	pfx=eol_${ceol}_crlf_${crlf}_attr_${attr}_ &&
> -	for f in LF CRLF LF_mix_CR CRLF_mix_LF LF_nul
> -	do
> -		test_expect_success "setup $f checkout ${ceol:+ with -c core.eol=$ceol}"  '
> -			rm -f crlf_false_attr__$f.txt &&
> -			git ${ceol:+-c core.eol=$ceol} checkout -- crlf_false_attr__$f.txt
> -		'
> -	done
> +	test_expect_success "setup checkout ${ceol:+ with -c core.eol=$ceol}" '
> +		rm -f crlf_false_attr__LF.txt crlf_false_attr__CRLF.txt \
> +			crlf_false_attr__LF_mix_CR.txt crlf_false_attr__CRLF_mix_LF.txt \
> +			crlf_false_attr__LF_nul.txt &&
> +		git ${ceol:+-c core.eol=$ceol} checkout -- \
> +			crlf_false_attr__LF.txt crlf_false_attr__CRLF.txt \
> +			crlf_false_attr__LF_mix_CR.txt crlf_false_attr__CRLF_mix_LF.txt \
> +			crlf_false_attr__LF_nul.txt
> +	'

Hmph, don't we want avoid hand-expand the list of files even though
we may want to use one "rm -f" and another "git checkout" command
to work on them?  IOW,

	paths=
	for f in LF CRLF LF_mix_CR CRLF_mix_LF LF_nul
	do
		paths=${paths:+"$paths "}crlf_false_attr__$f.txt
	done
	test_expect_success "setup checkout ${ceol:+ with -c core.eol=$ceol}" '
		rm -f $paths &&
		git ${ceol:+-c core.eol=$ceol} checkout -- $paths
	'

or something?

>  
>  	test_expect_success "ls-files --eol attr=$attr $ident aeol=$aeol core.autocrlf=$crlf core.eol=$ceol" '
>  		test_when_finished "rm expect actual" &&

Re: [PATCH 4/7] t0027: optimize how we reset repository state

[Torsten Bögershausen]

On Fri, Nov 06, 2026 at 09:33:01AM -0800, Junio C Hamano wrote:
> Hmph, don't we want avoid hand-expand the list of files even though
> we may want to use one "rm -f" and another "git checkout" command
> to work on them?  IOW,
> 
> 	paths=
> 	for f in LF CRLF LF_mix_CR CRLF_mix_LF LF_nul
> 	do
> 		paths=${paths:+"$paths "}crlf_false_attr__$f.txt
> 	done
> 	test_expect_success "setup checkout ${ceol:+ with -c core.eol=$ceol}" '
> 		rm -f $paths &&
> 		git ${ceol:+-c core.eol=$ceol} checkout -- $paths
> 	'

"Wow, so it can go" - a well working, but slow code is speed up so much.

I used the following (before reading the suggestion from Junio)
and the execution time here on an older MacOs dropped from 71 seconds to 54 seconds.

    test_expect_success "setup file checkout ${ceol:+ with -c core.eol=$ceol}"  '
        files="\
           crlf_false_attr__LF.txt \
           crlf_false_attr__CRLF.txt \
           crlf_false_attr__LF_mix_CR.txt \
           crlf_false_attr__CRLF_mix_LF.txt \
           crlf_false_attr__LF_nul.txt" &&
        rm -fv $files &&
        git ${ceol:+-c core.eol=$ceol} checkout -- $files
    '

[PATCH 5/7] t0027: cache expected output files

[Patrick Steinhardt]

In the preceding commit we have optimized one aspect of t0027 already.
Another inefficiency is `compare_files ()` and `compare_ws_file ()`,
which are both called more than a 1000 times.

These functions will repeatedly generate the same expected outcome even
though there's only a small handful of input files that the expected
outcome is computed from. Recomputing those every time is of course
quite pointless.

Improve this by computing the expected outcome once, only, to reduce the
number of spawned processes. This reduces the runtime by around 20-25%,
and on Windows the runtime decreases from 480 to 380 seconds.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 t/t0027-auto-crlf.sh | 17 +++++++++--------
 1 file changed, 9 insertions(+), 8 deletions(-)

diff --git a/t/t0027-auto-crlf.sh b/t/t0027-auto-crlf.sh
index eefb346d11f..25ed6b33704 100755
--- a/t/t0027-auto-crlf.sh
+++ b/t/t0027-auto-crlf.sh
@@ -5,22 +5,23 @@ test_description='CRLF conversion all combinations'
 . ./test-lib.sh
 
 compare_files () {
-	tr '\015\000' QN <"$1" >"$1".expect &&
+	{ test -f "$1".expect || tr '\015\000' QN <"$1" >"$1".expect; } &&
 	tr '\015\000' QN <"$2" | tr -d 'Z' >"$2".actual &&
-	test_cmp "$1".expect "$2".actual &&
-	rm "$1".expect "$2".actual
+	test_cmp "$1".expect "$2".actual
 }
 
 compare_ws_file () {
 	pfx=$1
-	exp=$2.expect
+	exp=$2.ws-expect
 	act=$pfx.actual.$3
-	tr '\015\000abcdef0123456789' QN00000000000000000 <"$2" |
-		sed -e "s/0000*/$ZERO_OID/" >"$exp" &&
+	{
+		test -f "$exp" ||
+		tr '\015\000abcdef0123456789' QN00000000000000000 <"$2" |
+			sed -e "s/0000*/$ZERO_OID/" >"$exp" ;
+	} &&
 	tr '\015\000abcdef0123456789' QN00000000000000000 <"$3" |
 		sed -e "s/0000*/$ZERO_OID/" >"$act" &&
-	test_cmp "$exp" "$act" &&
-	rm "$exp" "$act"
+	test_cmp "$exp" "$act"
 }
 
 create_gitattributes () {

-- 
2.56.0.500.g5d06dc02cd.dirty

[PATCH 6/7] t1092: set up repositories from templates

[Patrick Steinhardt]

In t1092 we verify the behaviour of sparse checkouts. We use three
different repositories to do so: one with a full checkout, one with a
sparse checkout and one with a sparse index, and then verify that the
result is the same across all these configurations.

Those repositories get set up via `init_repos ()`, which initializes
them from scratch every single time with a bunch of processes. The
function is called more than a 100 times though, so the overhead adds up
quite a bit. While the tests only require around 30 seconds on Linux,
it takes more than 10 minutes on Windows.

Optimize this by instead creating the repositories once, only. So
instead of recreating them from scratch, we can then just copy the
template repositories into place and thus save a bunch of processes.
This speeds up tests on Windows by about 3 minutes.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 t/t1092-sparse-checkout-compatibility.sh | 34 +++++++++++++++++++-------------
 1 file changed, 20 insertions(+), 14 deletions(-)

diff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh
index 05b54062b3b..5d81ec54c23 100755
--- a/t/t1092-sparse-checkout-compatibility.sh
+++ b/t/t1092-sparse-checkout-compatibility.sh
@@ -141,28 +141,34 @@ test_expect_success 'setup' '
 	)
 '
 
+test_expect_success 'setup template repositories' '
+	cp -r initial-repo full-checkout.template &&
+	cp -r initial-repo sparse-checkout.template &&
+	cp -r initial-repo sparse-index.template &&
+
+	# initialize sparse-checkout definitions
+	git -C sparse-checkout.template sparse-checkout init --cone &&
+	git -C sparse-checkout.template sparse-checkout set deep &&
+	git -C sparse-index.template sparse-checkout init --cone --sparse-index &&
+	test_cmp_config -C sparse-index.template true index.sparse &&
+	git -C sparse-index.template sparse-checkout set deep &&
+
+	# Disable this message to keep stderr the same.
+	git -C sparse-index.template config advice.sparseIndexExpanded false
+'
+
 init_repos () {
 	rm -rf full-checkout sparse-checkout sparse-index &&
 
 	# create repos in initial state
-	cp -r initial-repo full-checkout &&
+	cp -r full-checkout.template full-checkout &&
 	git -C full-checkout reset --hard &&
 
-	cp -r initial-repo sparse-checkout &&
+	cp -r sparse-checkout.template sparse-checkout &&
 	git -C sparse-checkout reset --hard &&
 
-	cp -r initial-repo sparse-index &&
-	git -C sparse-index reset --hard &&
-
-	# initialize sparse-checkout definitions
-	git -C sparse-checkout sparse-checkout init --cone &&
-	git -C sparse-checkout sparse-checkout set deep &&
-	git -C sparse-index sparse-checkout init --cone --sparse-index &&
-	test_cmp_config -C sparse-index true index.sparse &&
-	git -C sparse-index sparse-checkout set deep &&
-
-	# Disable this message to keep stderr the same.
-	git -C sparse-index config advice.sparseIndexExpanded false
+	cp -r sparse-index.template sparse-index &&
+	git -C sparse-index reset --hard
 }
 
 init_repos_as_submodules () {

-- 
2.56.0.500.g5d06dc02cd.dirty

[PATCH 7/7] t5551: use reftables for repositories with many refs

[Patrick Steinhardt]

Some of the tests in t5551 creates almost a 100,000 references and then
verify that those can be fetched. Naturally, creating this many refs is
quite involved with the "files" backend, as it means that we have to
create this many individual loose files. This is tolerable on Linux,
where the test takes around 2 minutes in our CI. But on macOS, the same
test takes almost 20 minutes to complete. And to add insult to injury,
we execute these tests a second time via t5559.

The system under test here is the HTTP negotiation, so we don't actually
care about whether or not the ref backend handles this alright. So adapt
the test to instead create repositories with the reftable backend, which
is significantly faster in this case as we only need to write a single
table.

And sure enough, this reduces the runtime of this test on macOS by a
factor of 10, going from 20 minutes to 2 minutes. On Linux, the tests at
least speed up by a factor of two.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 t/t5551-http-fetch-smart.sh | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)

diff --git a/t/t5551-http-fetch-smart.sh b/t/t5551-http-fetch-smart.sh
index 130053c57aa..34945059531 100755
--- a/t/t5551-http-fetch-smart.sh
+++ b/t/t5551-http-fetch-smart.sh
@@ -412,7 +412,8 @@ create_tags () {
 }
 
 test_expect_success 'create 2,000 tags in the repo' '
-	git init --bare "$HTTPD_DOCUMENT_ROOT_PATH/many-tags.git" &&
+	git init --bare --ref-format=reftable \
+		"$HTTPD_DOCUMENT_ROOT_PATH/many-tags.git" &&
 	(
 		cd "$HTTPD_DOCUMENT_ROOT_PATH/many-tags.git" &&
 		create_tags 1 2000
@@ -421,7 +422,8 @@ test_expect_success 'create 2,000 tags in the repo' '
 
 test_expect_success CMDLINE_LIMIT \
 	'clone the 2,000 tag repo to check OS command line overflow' '
-	run_with_limited_cmdline git clone $HTTPD_URL/smart/many-tags.git too-many-refs &&
+	run_with_limited_cmdline git clone --ref-format=reftable \
+		$HTTPD_URL/smart/many-tags.git too-many-refs &&
 	(
 		cd too-many-refs &&
 		git for-each-ref refs/tags >actual &&

-- 
2.56.0.500.g5d06dc02cd.dirty

Re: [PATCH 0/7] ci: some speedups for slow tests

[Junio C Hamano]

Patrick Steinhardt <ps@pks.im> writes:

> I was staring at CI way more often recently, and that got me annoyed
> real fast because it's just painstakingly slow. So I decided to have a
> look and try do something about it, the result of which is this patch
> series.

Thanks for these patches.  I didn't leave comments on later parts of
the series, but all of them were interesting read and fully made
sense to me.