[PATCH 0/5] meson: wire up support for benchmarks

[PATCH 0/5] meson: wire up support for benchmarks

[Patrick Steinhardt]

Hi,

this small patch series implements support for running our benchmarks in
"t/perf" via Meson. The series does not aim to replace "t/perf/run",
which is more fully-featured and allows running benchmarks against
multiple different trees. Instead, this series only allows running the
benchmarks against the current tree. Users are thus expected to continue
using "t/perf/run" for more advanced usecases.

Thanks!

Patrick

---
Patrick Steinhardt (5):
      t/perf: fix benchmarks with alternate repo formats
      t/perf: use configured PERL_PATH
      t/perf: fix benchmarks with out-of-tree builds
      meson: wire up benchmarks
      meson: wire up benchmarking options

 meson.build        |  7 ++---
 meson_options.txt  |  6 +++++
 t/meson.build      | 77 ++++++++++++++++++++++++++++++++++++++++++++++++++++++
 t/perf/perf-lib.sh | 32 ++++++++++++++++++++---
 t/perf/run         |  4 +--
 5 files changed, 117 insertions(+), 9 deletions(-)


---
base-commit: 683c54c999c301c2cd6f715c411407c413b1d84e
change-id: 20250328-pks-meson-benchmarks-a8fac5f69467

[PATCH 1/5] t/perf: fix benchmarks with alternate repo formats

[Patrick Steinhardt]

Many of our benchmarks operate on a user-defined repository that we copy
over before running the benchmarked logic. To keep unintentional side
effects caused by on-disk state at bay we skip copying some files. This
includes for example hooks, but also the repo's configuration.

It is quite sensible to not copy over the configuration, as it is quite
easy to inadvertently carry over configuration that may significantly
impact the performance measurements. But we cannot fully ignore the
configuration either, as it may contain information about the repository
format. This will cause failures when for example using a repository
with SHA256 object format or the reftable ref format.

Fix the issue by parsing the reference and object formats from the
source repository and passing them to git-init(1).

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 t/perf/perf-lib.sh | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/t/perf/perf-lib.sh b/t/perf/perf-lib.sh
index 8ab6d9c4694..1a9a51ca3cc 100644
--- a/t/perf/perf-lib.sh
+++ b/t/perf/perf-lib.sh
@@ -98,6 +98,8 @@ test_perf_create_repo_from () {
 	source_git="$("$MODERN_GIT" -C "$source" rev-parse --git-dir)"
 	objects_dir="$("$MODERN_GIT" -C "$source" rev-parse --git-path objects)"
 	common_dir="$("$MODERN_GIT" -C "$source" rev-parse --git-common-dir)"
+	refformat="$("$MODERN_GIT" -C "$source" rev-parse --show-ref-format)"
+	objectformat="$("$MODERN_GIT" -C "$source" rev-parse --show-object-format)"
 	mkdir -p "$repo/.git"
 	(
 		cd "$source" &&
@@ -114,7 +116,7 @@ test_perf_create_repo_from () {
 	) &&
 	(
 		cd "$repo" &&
-		"$MODERN_GIT" init -q &&
+		"$MODERN_GIT" init -q --ref-format="$refformat" --object-format="$objectformat" &&
 		test_perf_do_repo_symlink_config_ &&
 		mv .git/hooks .git/hooks-disabled 2>/dev/null &&
 		if test -f .git/index.lock

-- 
2.49.0.604.gff1f9ca942.dirty

[PATCH 2/5] t/perf: use configured PERL_PATH

[Patrick Steinhardt]

Our benchmarks use a couple of Perl scripts to compute results. These
Perl scripts get executed directly, and as the shebang is hardcoded to
"/usr/bin/perl" this will fail on any system where the Perl interpreter
is located in a different path.

Our build infrastructure already lets users configure the location of
Perl, which ultimately gets written into the GIT-BUILD-OPTIONS file.
This file is being sourced by "test-lib.sh", and consequently we already
have the "PERL_PATH" variable available that contains its configured
location.

Use "PERL_PATH" to execute Perl scripts, which makes them work on more
esoteric systems like NixOS.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 t/perf/perf-lib.sh | 4 ++--
 t/perf/run         | 4 ++--
 2 files changed, 4 insertions(+), 4 deletions(-)

diff --git a/t/perf/perf-lib.sh b/t/perf/perf-lib.sh
index 1a9a51ca3cc..4173eee4def 100644
--- a/t/perf/perf-lib.sh
+++ b/t/perf/perf-lib.sh
@@ -276,7 +276,7 @@ test_perf_ () {
 	else
 		test_ok_ "$1"
 	fi
-	"$TEST_DIRECTORY"/perf/min_time.perl test_time.* >"$base".result
+	"$PERL_PATH" "$TEST_DIRECTORY"/perf/min_time.perl test_time.* >"$base".result
 	rm test_time.*
 }
 
@@ -324,7 +324,7 @@ test_at_end_hook_ () {
 	if test -z "$GIT_PERF_AGGREGATING_LATER"; then
 		(
 			cd "$TEST_DIRECTORY"/perf &&
-			./aggregate.perl --results-dir="$TEST_RESULTS_DIR" $(basename "$0")
+			"$PERL_PATH" ./aggregate.perl --results-dir="$TEST_RESULTS_DIR" $(basename "$0")
 		)
 	fi
 }
diff --git a/t/perf/run b/t/perf/run
index 486ead21980..073bcb2afff 100755
--- a/t/perf/run
+++ b/t/perf/run
@@ -192,10 +192,10 @@ run_subsection () {
 
 	if test -z "$GIT_PERF_SEND_TO_CODESPEED"
 	then
-		./aggregate.perl --results-dir="$TEST_RESULTS_DIR" $codespeed_opt "$@"
+		"$PERL_PATH" ./aggregate.perl --results-dir="$TEST_RESULTS_DIR" $codespeed_opt "$@"
 	else
 		json_res_file=""$TEST_RESULTS_DIR"/$GIT_PERF_SUBSECTION/aggregate.json"
-		./aggregate.perl --results-dir="$TEST_RESULTS_DIR" --codespeed "$@" | tee "$json_res_file"
+		"$PERL_PATH" ./aggregate.perl --results-dir="$TEST_RESULTS_DIR" --codespeed "$@" | tee "$json_res_file"
 		send_data_url="$GIT_PERF_SEND_TO_CODESPEED/result/add/json/"
 		curl -v --request POST --data-urlencode "json=$(cat "$json_res_file")" "$send_data_url"
 	fi

-- 
2.49.0.604.gff1f9ca942.dirty

Re: [PATCH 2/5] t/perf: use configured PERL_PATH

[Toon Claes]

Patrick Steinhardt <ps@pks.im> writes:

> Our benchmarks use a couple of Perl scripts to compute results. These
> Perl scripts get executed directly, and as the shebang is hardcoded to
> "/usr/bin/perl" this will fail on any system where the Perl interpreter
> is located in a different path.
>
> Our build infrastructure already lets users configure the location of
> Perl, which ultimately gets written into the GIT-BUILD-OPTIONS file.
> This file is being sourced by "test-lib.sh", and consequently we already
> have the "PERL_PATH" variable available that contains its configured
> location.
>
> Use "PERL_PATH" to execute Perl scripts, which makes them work on more
> esoteric systems like NixOS.

I see in `t/perf/README` there's a mention of running `./aggregate.perl`
directly? Shall we inform the user to run that through their Perl as
well?

-    $ ./aggregate.perl . /path/to/other/git ./p0001-rev-list.sh
+    $ perl ./aggregate.perl . /path/to/other/git ./p0001-rev-list.sh

Or do we expect users to know what they are doing when they don't have
Perl installed at /usr/bin/perl?

--
Toon

Re: [PATCH 2/5] t/perf: use configured PERL_PATH

[Patrick Steinhardt]

On Thu, Apr 10, 2025 at 01:43:44PM +0200, Toon Claes wrote:
> Patrick Steinhardt <ps@pks.im> writes:
> 
> > Our benchmarks use a couple of Perl scripts to compute results. These
> > Perl scripts get executed directly, and as the shebang is hardcoded to
> > "/usr/bin/perl" this will fail on any system where the Perl interpreter
> > is located in a different path.
> >
> > Our build infrastructure already lets users configure the location of
> > Perl, which ultimately gets written into the GIT-BUILD-OPTIONS file.
> > This file is being sourced by "test-lib.sh", and consequently we already
> > have the "PERL_PATH" variable available that contains its configured
> > location.
> >
> > Use "PERL_PATH" to execute Perl scripts, which makes them work on more
> > esoteric systems like NixOS.
> 
> I see in `t/perf/README` there's a mention of running `./aggregate.perl`
> directly? Shall we inform the user to run that through their Perl as
> well?
> 
> -    $ ./aggregate.perl . /path/to/other/git ./p0001-rev-list.sh
> +    $ perl ./aggregate.perl . /path/to/other/git ./p0001-rev-list.sh
> 
> Or do we expect users to know what they are doing when they don't have
> Perl installed at /usr/bin/perl?

Another solution could be to switch the shebang to `#!/usr/bin/env
perl`. We also do this for "t/chainlint.pl".

Patrick

Re: [PATCH 2/5] t/perf: use configured PERL_PATH

[Junio C Hamano]

Patrick Steinhardt <ps@pks.im> writes:

>> > Use "PERL_PATH" to execute Perl scripts, which makes them work on more
>> > esoteric systems like NixOS.
>> 
>> I see in `t/perf/README` there's a mention of running `./aggregate.perl`
>> directly? Shall we inform the user to run that through their Perl as
>> well?
>> 
>> -    $ ./aggregate.perl . /path/to/other/git ./p0001-rev-list.sh
>> +    $ perl ./aggregate.perl . /path/to/other/git ./p0001-rev-list.sh

Good.

>> 
>> Or do we expect users to know what they are doing when they don't have
>> Perl installed at /usr/bin/perl?
>
> Another solution could be to switch the shebang to `#!/usr/bin/env
> perl`. We also do this for "t/chainlint.pl".

When you do not have perl installed anywhere, how does this fail?  I
think you would get

    $ ./aggregate.perl ...
    /usr/bin/env: 'perl': No such file or directory

and compared to that, 

    $ perl ./aggregate.perl ...
    bash: perl: command not found

I think it makes it slightly more obvious to those who lack perl on
their $PATH what is going wrong to explicitly tell them to run
"perl" like Toon's suggestion above (primarily because use of
/usr/bin/env is not obvious to those who are told to run
./aggregate.perl script).

Re: [PATCH 2/5] t/perf: use configured PERL_PATH

[Patrick Steinhardt]

On Mon, Apr 14, 2025 at 12:20:40PM -0700, Junio C Hamano wrote:
> Patrick Steinhardt <ps@pks.im> writes:
> 
> >> > Use "PERL_PATH" to execute Perl scripts, which makes them work on more
> >> > esoteric systems like NixOS.
> >> 
> >> I see in `t/perf/README` there's a mention of running `./aggregate.perl`
> >> directly? Shall we inform the user to run that through their Perl as
> >> well?
> >> 
> >> -    $ ./aggregate.perl . /path/to/other/git ./p0001-rev-list.sh
> >> +    $ perl ./aggregate.perl . /path/to/other/git ./p0001-rev-list.sh
> 
> Good.
> 
> >> 
> >> Or do we expect users to know what they are doing when they don't have
> >> Perl installed at /usr/bin/perl?
> >
> > Another solution could be to switch the shebang to `#!/usr/bin/env
> > perl`. We also do this for "t/chainlint.pl".
> 
> When you do not have perl installed anywhere, how does this fail?  I
> think you would get
> 
>     $ ./aggregate.perl ...
>     /usr/bin/env: 'perl': No such file or directory
> 
> and compared to that, 
> 
>     $ perl ./aggregate.perl ...
>     bash: perl: command not found
> 
> I think it makes it slightly more obvious to those who lack perl on
> their $PATH what is going wrong to explicitly tell them to run
> "perl" like Toon's suggestion above (primarily because use of
> /usr/bin/env is not obvious to those who are told to run
> ./aggregate.perl script).

The counterargument is that `#!/usr/bin/env perl` works on systems where
Perl is in PATH, but in a non-standard location. So isn't it preferable
to make things Just Work(TM) on such systems instead of giving slightly
better error messages on systems where it wouldn't work anyway because
Perl is not available? I certainly think so, but if you or others feel
strongly I'll adapt.

Patrick

[PATCH 3/5] t/perf: fix benchmarks with out-of-tree builds

[Patrick Steinhardt]

The "perf-lib.sh" script is sourced by all of our benchmarking suites to
make available common infrastructure. The script assumes that build and
source directory are the same, which works for our Makefile. But the
assumption breaks with both CMake and Meson, where the build directory
can be located in an arbitrary place.

Adapt the script so that it works with out-of-tree builds. This prepares
us for wiring up benchmarks in Meson.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 t/perf/perf-lib.sh | 26 ++++++++++++++++++++++++--
 1 file changed, 24 insertions(+), 2 deletions(-)

diff --git a/t/perf/perf-lib.sh b/t/perf/perf-lib.sh
index 4173eee4def..5406557b7ca 100644
--- a/t/perf/perf-lib.sh
+++ b/t/perf/perf-lib.sh
@@ -25,7 +25,29 @@ TEST_OUTPUT_DIRECTORY=$(pwd)
 TEST_NO_CREATE_REPO=t
 TEST_NO_MALLOC_CHECK=t
 
-. ../test-lib.sh
+# While test-lib.sh computes the build directory for us, we also have to do the
+# same thing in order to locate the script via GIT-BUILD-OPTIONS in the first
+# place.
+GIT_BUILD_DIR="${GIT_BUILD_DIR:-$TEST_DIRECTORY/..}"
+if test -f "$GIT_BUILD_DIR/GIT-BUILD-DIR"
+then
+	GIT_BUILD_DIR="$(cat "$GIT_BUILD_DIR/GIT-BUILD-DIR")" || exit 1
+	# On Windows, we must convert Windows paths lest they contain a colon
+	case "$(uname -s)" in
+	*MINGW*)
+		GIT_BUILD_DIR="$(cygpath -au "$GIT_BUILD_DIR")"
+		;;
+	esac
+fi
+
+if test ! -f "$GIT_BUILD_DIR"/GIT-BUILD-OPTIONS
+then
+	echo >&2 'error: GIT-BUILD-OPTIONS missing (has Git been built?).'
+	exit 1
+fi
+
+. "$GIT_BUILD_DIR"/GIT-BUILD-OPTIONS
+. "$GIT_SOURCE_DIR"/t/test-lib.sh
 
 unset GIT_CONFIG_NOSYSTEM
 GIT_CONFIG_SYSTEM="$TEST_DIRECTORY/perf/config"
@@ -324,7 +346,7 @@ test_at_end_hook_ () {
 	if test -z "$GIT_PERF_AGGREGATING_LATER"; then
 		(
 			cd "$TEST_DIRECTORY"/perf &&
-			"$PERL_PATH" ./aggregate.perl --results-dir="$TEST_RESULTS_DIR" $(basename "$0")
+			"$PERL_PATH" "$GIT_SOURCE_DIR"/t/perf/aggregate.perl --results-dir="$TEST_RESULTS_DIR" $(basename "$0")
 		)
 	fi
 }

-- 
2.49.0.604.gff1f9ca942.dirty

Re: [PATCH 3/5] t/perf: fix benchmarks with out-of-tree builds

[Toon Claes]

Patrick Steinhardt <ps@pks.im> writes:

> The "perf-lib.sh" script is sourced by all of our benchmarking suites to
> make available common infrastructure. The script assumes that build and
> source directory are the same, which works for our Makefile. But the
> assumption breaks with both CMake and Meson, where the build directory
> can be located in an arbitrary place.
>
> Adapt the script so that it works with out-of-tree builds. This prepares
> us for wiring up benchmarks in Meson.
>
> Signed-off-by: Patrick Steinhardt <ps@pks.im>
> ---
>  t/perf/perf-lib.sh | 26 ++++++++++++++++++++++++--
>  1 file changed, 24 insertions(+), 2 deletions(-)
>
> diff --git a/t/perf/perf-lib.sh b/t/perf/perf-lib.sh
> index 4173eee4def..5406557b7ca 100644
> --- a/t/perf/perf-lib.sh
> +++ b/t/perf/perf-lib.sh
> @@ -25,7 +25,29 @@ TEST_OUTPUT_DIRECTORY=$(pwd)
>  TEST_NO_CREATE_REPO=t
>  TEST_NO_MALLOC_CHECK=t
>  
> -. ../test-lib.sh
> +# While test-lib.sh computes the build directory for us, we also have to do the
> +# same thing in order to locate the script via GIT-BUILD-OPTIONS in the first
> +# place.
> +GIT_BUILD_DIR="${GIT_BUILD_DIR:-$TEST_DIRECTORY/..}"
> +if test -f "$GIT_BUILD_DIR/GIT-BUILD-DIR"
> +then
> +	GIT_BUILD_DIR="$(cat "$GIT_BUILD_DIR/GIT-BUILD-DIR")" || exit 1
> +	# On Windows, we must convert Windows paths lest they contain a colon
> +	case "$(uname -s)" in
> +	*MINGW*)
> +		GIT_BUILD_DIR="$(cygpath -au "$GIT_BUILD_DIR")"
> +		;;
> +	esac
> +fi
> +
> +if test ! -f "$GIT_BUILD_DIR"/GIT-BUILD-OPTIONS
> +then
> +	echo >&2 'error: GIT-BUILD-OPTIONS missing (has Git been built?).'
> +	exit 1
> +fi
> +
> +. "$GIT_BUILD_DIR"/GIT-BUILD-OPTIONS
> +. "$GIT_SOURCE_DIR"/t/test-lib.sh
>  
>  unset GIT_CONFIG_NOSYSTEM
>  GIT_CONFIG_SYSTEM="$TEST_DIRECTORY/perf/config"
> @@ -324,7 +346,7 @@ test_at_end_hook_ () {
>  	if test -z "$GIT_PERF_AGGREGATING_LATER"; then
>  		(
>  			cd "$TEST_DIRECTORY"/perf &&
> -			"$PERL_PATH" ./aggregate.perl --results-dir="$TEST_RESULTS_DIR" $(basename "$0")
> +			"$PERL_PATH" "$GIT_SOURCE_DIR"/t/perf/aggregate.perl --results-dir="$TEST_RESULTS_DIR" $(basename "$0")
>  		)
>  	fi
>  }
>
> -- 
> 2.49.0.604.gff1f9ca942.dirty

I'm not sure if it's related to this commit, but this patch series has
broken something:

    $ make && cd t/perf && ./run . master p0005-status.sh

    === Running 1 tests in this tree ===
    ok 1 - setup repo
    perf 2 - read-tree status br_ballast (4629): 1 ok
    # passed all 2 test(s)
    1..2
    === Unpacking 485f5f863615e670fd97ae40af744e14072cfe18 in build/485f5f863615e670fd97ae40af744e14072cfe18 ===
    === Building 485f5f863615e670fd97ae40af744e14072cfe18 (master) ===
    GIT_VERSION=2.49.GIT
        * new build flags
        CC daemon.o
        * new link flags
        CC common-main.o
        CC abspath.o
        CC add-interactive.o
        CC add-patch.o
        [snip]
        CC t/unit-tests/unit-test.o
        CC t/unit-tests/lib-oid.o
        LINK t/unit-tests/bin/unit-tests
        GEN gitweb/gitweb.cgi
        GEN gitweb/static/gitweb.js
    === Running 1 tests in /home/toon/devel/git/t/perf/build/485f5f863615e670fd97ae40af744e14072cfe18/bin-wrappers ===
    ok 1 - setup repo
    perf 2 - read-tree status br_ballast (4629): 1 ok
    # passed all 2 test(s)
    1..2
    cannot open test-results/p0005-status.subtests: No such file or directory at ./aggregate.perl line 159.

--
Toon

Re: [PATCH 3/5] t/perf: fix benchmarks with out-of-tree builds

[Toon Claes]

Toon Claes <toon@iotcl.com> writes:

> I'm not sure if it's related to this commit, but this patch series has
> broken something:
>
>     $ make && cd t/perf && ./run . master p0005-status.sh
>
>     === Running 1 tests in this tree ===
>     ok 1 - setup repo
>     perf 2 - read-tree status br_ballast (4629): 1 ok
>     # passed all 2 test(s)
>     1..2
>     === Unpacking 485f5f863615e670fd97ae40af744e14072cfe18 in build/485f5f863615e670fd97ae40af744e14072cfe18 ===
>     === Building 485f5f863615e670fd97ae40af744e14072cfe18 (master) ===
>     GIT_VERSION=2.49.GIT
>         * new build flags
>         CC daemon.o
>         * new link flags
>         CC common-main.o
>         CC abspath.o
>         CC add-interactive.o
>         CC add-patch.o
>         [snip]
>         CC t/unit-tests/unit-test.o
>         CC t/unit-tests/lib-oid.o
>         LINK t/unit-tests/bin/unit-tests
>         GEN gitweb/gitweb.cgi
>         GEN gitweb/static/gitweb.js
>     === Running 1 tests in /home/toon/devel/git/t/perf/build/485f5f863615e670fd97ae40af744e14072cfe18/bin-wrappers ===
>     ok 1 - setup repo
>     perf 2 - read-tree status br_ballast (4629): 1 ok
>     # passed all 2 test(s)
>     1..2
>     cannot open test-results/p0005-status.subtests: No such file or directory at ./aggregate.perl line 159.
>
> --
> Toon

Euhm, it seems I no longer can reproduce this issue. So you can ignore
this message. Sorry for the noice.

--
Toon

Re: [PATCH 3/5] t/perf: fix benchmarks with out-of-tree builds

[Patrick Steinhardt]

On Mon, Apr 14, 2025 at 08:28:53AM +0200, Toon Claes wrote:
> Toon Claes <toon@iotcl.com> writes:
> 
> > I'm not sure if it's related to this commit, but this patch series has
> > broken something:
> >
> >     $ make && cd t/perf && ./run . master p0005-status.sh
> >
> >     === Running 1 tests in this tree ===
> >     ok 1 - setup repo
> >     perf 2 - read-tree status br_ballast (4629): 1 ok
> >     # passed all 2 test(s)
> >     1..2
> >     === Unpacking 485f5f863615e670fd97ae40af744e14072cfe18 in build/485f5f863615e670fd97ae40af744e14072cfe18 ===
> >     === Building 485f5f863615e670fd97ae40af744e14072cfe18 (master) ===
> >     GIT_VERSION=2.49.GIT
> >         * new build flags
> >         CC daemon.o
> >         * new link flags
> >         CC common-main.o
> >         CC abspath.o
> >         CC add-interactive.o
> >         CC add-patch.o
> >         [snip]
> >         CC t/unit-tests/unit-test.o
> >         CC t/unit-tests/lib-oid.o
> >         LINK t/unit-tests/bin/unit-tests
> >         GEN gitweb/gitweb.cgi
> >         GEN gitweb/static/gitweb.js
> >     === Running 1 tests in /home/toon/devel/git/t/perf/build/485f5f863615e670fd97ae40af744e14072cfe18/bin-wrappers ===
> >     ok 1 - setup repo
> >     perf 2 - read-tree status br_ballast (4629): 1 ok
> >     # passed all 2 test(s)
> >     1..2
> >     cannot open test-results/p0005-status.subtests: No such file or directory at ./aggregate.perl line 159.
> >
> > --
> > Toon
> 
> Euhm, it seems I no longer can reproduce this issue. So you can ignore
> this message. Sorry for the noice.

Okay, great. I wasn't able to reproduce this locally, so thanks for
clarifying!

Patrick

[PATCH 4/5] meson: wire up benchmarks

[Patrick Steinhardt]

Wire up benchmarks in Meson. The setup is mostly the same as how we wire
up our tests. The only difference is that benchmarks get wired up via
the `benchmark()` option instead of via `test()`, which gives them a bit
of special treatment:

  - Benchmarks never run in parallel.

  - Benchmarks aren't run by default when tests are executed.

  - Meson does not inject the `MALLOC_PERTURB` environment variable.

Using benchmarks is quite simple:

    ```
    $ meson setup build
    # Run all benchmarks.
    $ meson test -C build --benchmark
    # Run a specific benchmark.
    $ meson test -C build --benchmark p0000-*
    ```

Other than that the usual command line arguments accepted when running
tests are also accepted when running benchmarks.

Note that the benchmarking target is somewhat limited because it will
only run benchmarks for the current build. Other usecases, like running
benchmarks against multiple different versions of Git, are not currently
supported. Users should continue to use "t/perf/run" for those usecases.
The script should get extended at one point in time to support Meson,
but this is outside of the scope of this series.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 meson.build   |  1 +
 t/meson.build | 77 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 78 insertions(+)

diff --git a/meson.build b/meson.build
index efe2871c9db..9af4a10188d 100644
--- a/meson.build
+++ b/meson.build
@@ -204,6 +204,7 @@ git = find_program('git', dirs: program_path, required: false)
 sed = find_program('sed', dirs: program_path)
 shell = find_program('sh', dirs: program_path)
 tar = find_program('tar', dirs: program_path)
+time = find_program('time', dirs: program_path, required: false)
 
 # Sanity-check that programs required for the build exist.
 foreach tool : ['cat', 'cut', 'grep', 'sort', 'tr', 'uname']
diff --git a/t/meson.build b/t/meson.build
index a59da26be3f..dba327fd7ec 100644
--- a/t/meson.build
+++ b/t/meson.build
@@ -1096,11 +1096,71 @@ integration_tests = [
   't9903-bash-prompt.sh',
 ]
 
+benchmarks = [
+  'perf/p0000-perf-lib-sanity.sh',
+  'perf/p0001-rev-list.sh',
+  'perf/p0002-read-cache.sh',
+  'perf/p0003-delta-base-cache.sh',
+  'perf/p0004-lazy-init-name-hash.sh',
+  'perf/p0005-status.sh',
+  'perf/p0006-read-tree-checkout.sh',
+  'perf/p0007-write-cache.sh',
+  'perf/p0008-odb-fsync.sh',
+  'perf/p0071-sort.sh',
+  'perf/p0090-cache-tree.sh',
+  'perf/p0100-globbing.sh',
+  'perf/p1006-cat-file.sh',
+  'perf/p1400-update-ref.sh',
+  'perf/p1450-fsck.sh',
+  'perf/p1451-fsck-skip-list.sh',
+  'perf/p1500-graph-walks.sh',
+  'perf/p2000-sparse-operations.sh',
+  'perf/p3400-rebase.sh',
+  'perf/p3404-rebase-interactive.sh',
+  'perf/p4000-diff-algorithms.sh',
+  'perf/p4001-diff-no-index.sh',
+  'perf/p4002-diff-color-moved.sh',
+  'perf/p4205-log-pretty-formats.sh',
+  'perf/p4209-pickaxe.sh',
+  'perf/p4211-line-log.sh',
+  'perf/p4220-log-grep-engines.sh',
+  'perf/p4221-log-grep-engines-fixed.sh',
+  'perf/p5302-pack-index.sh',
+  'perf/p5303-many-packs.sh',
+  'perf/p5304-prune.sh',
+  'perf/p5310-pack-bitmaps.sh',
+  'perf/p5311-pack-bitmaps-fetch.sh',
+  'perf/p5312-pack-bitmaps-revs.sh',
+  'perf/p5313-pack-objects.sh',
+  'perf/p5314-name-hash.sh',
+  'perf/p5326-multi-pack-bitmaps.sh',
+  'perf/p5332-multi-pack-reuse.sh',
+  'perf/p5333-pseudo-merge-bitmaps.sh',
+  'perf/p5550-fetch-tags.sh',
+  'perf/p5551-fetch-rescan.sh',
+  'perf/p5600-partial-clone.sh',
+  'perf/p5601-clone-reference.sh',
+  'perf/p6100-describe.sh',
+  'perf/p6300-for-each-ref.sh',
+  'perf/p7000-filter-branch.sh',
+  'perf/p7102-reset.sh',
+  'perf/p7300-clean.sh',
+  'perf/p7519-fsmonitor.sh',
+  'perf/p7527-builtin-fsmonitor.sh',
+  'perf/p7810-grep.sh',
+  'perf/p7820-grep-engines.sh',
+  'perf/p7821-grep-engines-fixed.sh',
+  'perf/p7822-grep-perl-character.sh',
+  'perf/p9210-scalar.sh',
+  'perf/p9300-fast-import-export.sh',
+]
+
 # Sanity check that we are not missing any tests present in 't/'. This check
 # only runs once at configure time and is thus best-effort, only. It is
 # sufficient to catch missing test suites in our CI though.
 foreach glob, tests : {
   't[0-9][0-9][0-9][0-9]-*.sh': integration_tests,
+  'perf/p[0-9][0-9][0-9][0-9]-*.sh': benchmarks,
   'unit-tests/t-*.c': unit_test_programs,
   'unit-tests/u-*.c': clar_test_suites,
 }
@@ -1152,3 +1212,20 @@ foreach integration_test : integration_tests
     timeout: 0,
   )
 endforeach
+
+if time.found()
+  benchmark_environment = test_environment
+  benchmark_environment.set('GTIME', time.full_path())
+
+  foreach benchmark : benchmarks
+    benchmark(fs.stem(benchmark), shell,
+      args: [
+        fs.name(benchmark),
+      ],
+      workdir: meson.current_source_dir() / 'perf',
+      env: benchmark_environment,
+      depends: test_dependencies + bin_wrappers,
+      timeout: 0,
+    )
+  endforeach
+endif

-- 
2.49.0.604.gff1f9ca942.dirty

Re: [PATCH 4/5] meson: wire up benchmarks

[Toon Claes]

Patrick Steinhardt <ps@pks.im> writes:

> Wire up benchmarks in Meson. The setup is mostly the same as how we wire
> up our tests. The only difference is that benchmarks get wired up via
> the `benchmark()` option instead of via `test()`, which gives them a bit
> of special treatment:
>
>   - Benchmarks never run in parallel.
>
>   - Benchmarks aren't run by default when tests are executed.
>
>   - Meson does not inject the `MALLOC_PERTURB` environment variable.
>
> Using benchmarks is quite simple:
>
>     ```
>     $ meson setup build
>     # Run all benchmarks.
>     $ meson test -C build --benchmark
>     # Run a specific benchmark.
>     $ meson test -C build --benchmark p0000-*
>     ```

I really appreciate the efforts, and I'd love to run the perf tests with
Meson, but at the moment I don't know how these changes are useful
because this doesn't print anything relevant:

    $ meson test -C build --benchmark 'p0005-*'
    ninja: Entering directory `/home/toon/devel/git/build'
    [1/28] Generating GIT-VERSION-FILE with a custom command (wrapped by meson to set env)
    1/1 p0005-status        OK              7.39s

    Ok:                 1
    Expected Fail:      0
    Fail:               0
    Unexpected Pass:    0
    Skipped:            0
    Timeout:            0

    Full log written to /home/toon/devel/git/build/meson-logs/testlog.txt

> Other than that the usual command line arguments accepted when running
> tests are also accepted when running benchmarks.
>
> Note that the benchmarking target is somewhat limited because it will
> only run benchmarks for the current build. Other usecases, like running
> benchmarks against multiple different versions of Git, are not currently
> supported. Users should continue to use "t/perf/run" for those usecases.
> The script should get extended at one point in time to support Meson,
> but this is outside of the scope of this series.

Yeah, this is unfortunate, but totally understandable. I've been digging
in `t/perf/run` and `f/perf/aggregate.perl` and it doesn't look easy to
adapt to Meson.

But I was wondering, instead of trying to fully integrate Meson into
those scripts, could we modify the scripts so they work with binaries
built by Meson? I mean, if we could run
`cd t/perf && ./run ../../build1 ../../build2 p0005*` and it would
simply run the benchmarks in those directories (without trying to check
out code and build the sources). I think this would help a lot already.

-- 
Toon

Re: [PATCH 4/5] meson: wire up benchmarks

[Patrick Steinhardt]

On Thu, Apr 10, 2025 at 01:44:55PM +0200, Toon Claes wrote:
> Patrick Steinhardt <ps@pks.im> writes:
> 
> > Wire up benchmarks in Meson. The setup is mostly the same as how we wire
> > up our tests. The only difference is that benchmarks get wired up via
> > the `benchmark()` option instead of via `test()`, which gives them a bit
> > of special treatment:
> >
> >   - Benchmarks never run in parallel.
> >
> >   - Benchmarks aren't run by default when tests are executed.
> >
> >   - Meson does not inject the `MALLOC_PERTURB` environment variable.
> >
> > Using benchmarks is quite simple:
> >
> >     ```
> >     $ meson setup build
> >     # Run all benchmarks.
> >     $ meson test -C build --benchmark
> >     # Run a specific benchmark.
> >     $ meson test -C build --benchmark p0000-*
> >     ```
> 
> I really appreciate the efforts, and I'd love to run the perf tests with
> Meson, but at the moment I don't know how these changes are useful
> because this doesn't print anything relevant:
> 
>     $ meson test -C build --benchmark 'p0005-*'
>     ninja: Entering directory `/home/toon/devel/git/build'
>     [1/28] Generating GIT-VERSION-FILE with a custom command (wrapped by meson to set env)
>     1/1 p0005-status        OK              7.39s
> 
>     Ok:                 1
>     Expected Fail:      0
>     Fail:               0
>     Unexpected Pass:    0
>     Skipped:            0
>     Timeout:            0
> 
>     Full log written to /home/toon/devel/git/build/meson-logs/testlog.txt

You can execute with `meson test -i`, which will cause Meson to print
the benchmark's output.

> > Other than that the usual command line arguments accepted when running
> > tests are also accepted when running benchmarks.
> >
> > Note that the benchmarking target is somewhat limited because it will
> > only run benchmarks for the current build. Other usecases, like running
> > benchmarks against multiple different versions of Git, are not currently
> > supported. Users should continue to use "t/perf/run" for those usecases.
> > The script should get extended at one point in time to support Meson,
> > but this is outside of the scope of this series.
> 
> Yeah, this is unfortunate, but totally understandable. I've been digging
> in `t/perf/run` and `f/perf/aggregate.perl` and it doesn't look easy to
> adapt to Meson.
> 
> But I was wondering, instead of trying to fully integrate Meson into
> those scripts, could we modify the scripts so they work with binaries
> built by Meson? I mean, if we could run
> `cd t/perf && ./run ../../build1 ../../build2 p0005*` and it would
> simply run the benchmarks in those directories (without trying to check
> out code and build the sources). I think this would help a lot already.

This is another thing we could (and eventually should) do, but it's
orthogonal to this patch series from my point of view.

Patrick

Re: [PATCH 4/5] meson: wire up benchmarks

[Toon Claes]

Patrick Steinhardt <ps@pks.im> writes:

> You can execute with `meson test -i`, which will cause Meson to print
> the benchmark's output.

(annoyed grunt) I didn't think about using `-i` here, probably because
it wasn't used in the example in the commit message.

>> But I was wondering, instead of trying to fully integrate Meson into
>> those scripts, could we modify the scripts so they work with binaries
>> built by Meson? I mean, if we could run
>> `cd t/perf && ./run ../../build1 ../../build2 p0005*` and it would
>> simply run the benchmarks in those directories (without trying to check
>> out code and build the sources). I think this would help a lot already.
>
> This is another thing we could (and eventually should) do, but it's
> orthogonal to this patch series from my point of view.

That's true. This series already enables me to use benchmarks with Meson
and is definetely a decent step in the right direction. Let's get this
series merged and we can work on better comparative benchmarks later.

I just reviewed v2 and I approve on this series.

-- 
Toon

[PATCH 5/5] meson: wire up benchmarking options

[Patrick Steinhardt]

Wire up a couple of benchmarking options that we end up writing into our
"GIT-BUILD-OPTIONS" file. These options allow users to control how
exactly benchmarks are executed.

Note that neither `GIT_PERF_MAKE_COMMAND` nor `GIT_PERF_MAKE_OPTS` are
exposed as a build option. Those options are used by "t/perf/run", which
is not used by Meson.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 meson.build       | 6 +++---
 meson_options.txt | 6 ++++++
 2 files changed, 9 insertions(+), 3 deletions(-)

diff --git a/meson.build b/meson.build
index 9af4a10188d..ba128ab0759 100644
--- a/meson.build
+++ b/meson.build
@@ -660,11 +660,11 @@ builtin_sources += custom_target(
 # build options to our tests.
 build_options_config = configuration_data()
 build_options_config.set('GIT_INTEROP_MAKE_OPTS', '')
-build_options_config.set('GIT_PERF_LARGE_REPO', '')
+build_options_config.set_quoted('GIT_PERF_LARGE_REPO', get_option('benchmark_large_repo'))
 build_options_config.set('GIT_PERF_MAKE_COMMAND', '')
 build_options_config.set('GIT_PERF_MAKE_OPTS', '')
-build_options_config.set('GIT_PERF_REPEAT_COUNT', '')
-build_options_config.set('GIT_PERF_REPO', '')
+build_options_config.set_quoted('GIT_PERF_REPEAT_COUNT', get_option('benchmark_repeat_count').to_string())
+build_options_config.set_quoted('GIT_PERF_REPO', get_option('benchmark_repo'))
 build_options_config.set('GIT_TEST_CMP_USE_COPIED_CONTEXT', '')
 build_options_config.set('GIT_TEST_INDEX_VERSION', '')
 build_options_config.set('GIT_TEST_OPTS', '')
diff --git a/meson_options.txt b/meson_options.txt
index 78d172a7401..ca106c43b2e 100644
--- a/meson_options.txt
+++ b/meson_options.txt
@@ -101,6 +101,12 @@ option('docs_backend', type: 'combo', choices: ['asciidoc', 'asciidoctor', 'auto
   description: 'Which backend to use to generate documentation.')
 
 # Testing.
+option('benchmark_repo', type: 'string', value: '',
+  description: 'Repository to copy for the performance tests. Should be at least the size of the Git repository.')
+option('benchmark_large_repo', type: 'string', value: '',
+  description: 'Large repository to copy for the performance tests. Should be at least the size of the Linux repository.')
+option('benchmark_repeat_count', type: 'integer', value: 3,
+  description: 'Number of times a test should be repeated for best-of-N measurements.')
 option('coccinelle', type: 'feature', value: 'auto',
   description: 'Provide a coccicheck target that generates a Coccinelle patch.')
 option('tests', type: 'boolean', value: true,

-- 
2.49.0.604.gff1f9ca942.dirty

[PATCH v2 0/5] meson: wire up support for benchmarks

[Patrick Steinhardt]

Hi,

this small patch series implements support for running our benchmarks in
"t/perf" via Meson. The series does not aim to replace "t/perf/run",
which is more fully-featured and allows running benchmarks against
multiple different trees. Instead, this series only allows running the
benchmarks against the current tree. Users are thus expected to continue
using "t/perf/run" for more advanced usecases.

Changes in v2:
  - Adapt "aggregate.perl" to use a "/usr/bin/env perl" shebang.
  - Link to v1: https://lore.kernel.org/r/20250331-pks-meson-benchmarks-v1-0-b2ace85616a3@pks.im

Thanks!

Patrick

---
Patrick Steinhardt (5):
      t/perf: fix benchmarks with alternate repo formats
      t/perf: use configured PERL_PATH
      t/perf: fix benchmarks with out-of-tree builds
      meson: wire up benchmarks
      meson: wire up benchmarking options

 meson.build           |  7 +++--
 meson_options.txt     |  6 ++++
 t/meson.build         | 77 +++++++++++++++++++++++++++++++++++++++++++++++++++
 t/perf/aggregate.perl |  2 +-
 t/perf/perf-lib.sh    | 32 ++++++++++++++++++---
 t/perf/run            |  4 +--
 6 files changed, 118 insertions(+), 10 deletions(-)

Range-diff versus v1:

1:  fd734280c1d = 1:  844e8bdf81d t/perf: fix benchmarks with alternate repo formats
2:  89a011eec63 ! 2:  d950c828933 t/perf: use configured PERL_PATH
    @@ Commit message
         location.
     
         Use "PERL_PATH" to execute Perl scripts, which makes them work on more
    -    esoteric systems like NixOS.
    +    esoteric systems like NixOS. Furthermore, adapt the shebang to use
    +    env(1) to execute Perl so that users who have Perl in PATH, but in a
    +    non-standard location can execute the script directly.
     
         Signed-off-by: Patrick Steinhardt <ps@pks.im>
     
    + ## t/perf/aggregate.perl ##
    +@@
    +-#!/usr/bin/perl
    ++#!/usr/bin/env perl
    + 
    + use lib '../../perl/build/lib';
    + use strict;
    +
      ## t/perf/perf-lib.sh ##
     @@ t/perf/perf-lib.sh: test_perf_ () {
      	else
3:  efecbd86cd6 = 3:  dc1adf62a86 t/perf: fix benchmarks with out-of-tree builds
4:  feeaaae3ec1 = 4:  8528a775395 meson: wire up benchmarks
5:  fc594fc3dc5 = 5:  c8a4d0b354e meson: wire up benchmarking options

---
base-commit: 683c54c999c301c2cd6f715c411407c413b1d84e
change-id: 20250328-pks-meson-benchmarks-a8fac5f69467

[PATCH v2 1/5] t/perf: fix benchmarks with alternate repo formats

[Patrick Steinhardt]

Many of our benchmarks operate on a user-defined repository that we copy
over before running the benchmarked logic. To keep unintentional side
effects caused by on-disk state at bay we skip copying some files. This
includes for example hooks, but also the repo's configuration.

It is quite sensible to not copy over the configuration, as it is quite
easy to inadvertently carry over configuration that may significantly
impact the performance measurements. But we cannot fully ignore the
configuration either, as it may contain information about the repository
format. This will cause failures when for example using a repository
with SHA256 object format or the reftable ref format.

Fix the issue by parsing the reference and object formats from the
source repository and passing them to git-init(1).

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 t/perf/perf-lib.sh | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/t/perf/perf-lib.sh b/t/perf/perf-lib.sh
index 8ab6d9c4694..1a9a51ca3cc 100644
--- a/t/perf/perf-lib.sh
+++ b/t/perf/perf-lib.sh
@@ -98,6 +98,8 @@ test_perf_create_repo_from () {
 	source_git="$("$MODERN_GIT" -C "$source" rev-parse --git-dir)"
 	objects_dir="$("$MODERN_GIT" -C "$source" rev-parse --git-path objects)"
 	common_dir="$("$MODERN_GIT" -C "$source" rev-parse --git-common-dir)"
+	refformat="$("$MODERN_GIT" -C "$source" rev-parse --show-ref-format)"
+	objectformat="$("$MODERN_GIT" -C "$source" rev-parse --show-object-format)"
 	mkdir -p "$repo/.git"
 	(
 		cd "$source" &&
@@ -114,7 +116,7 @@ test_perf_create_repo_from () {
 	) &&
 	(
 		cd "$repo" &&
-		"$MODERN_GIT" init -q &&
+		"$MODERN_GIT" init -q --ref-format="$refformat" --object-format="$objectformat" &&
 		test_perf_do_repo_symlink_config_ &&
 		mv .git/hooks .git/hooks-disabled 2>/dev/null &&
 		if test -f .git/index.lock

-- 
2.49.0.805.g082f7c87e0.dirty

[PATCH v2 2/5] t/perf: use configured PERL_PATH

[Patrick Steinhardt]

Our benchmarks use a couple of Perl scripts to compute results. These
Perl scripts get executed directly, and as the shebang is hardcoded to
"/usr/bin/perl" this will fail on any system where the Perl interpreter
is located in a different path.

Our build infrastructure already lets users configure the location of
Perl, which ultimately gets written into the GIT-BUILD-OPTIONS file.
This file is being sourced by "test-lib.sh", and consequently we already
have the "PERL_PATH" variable available that contains its configured
location.

Use "PERL_PATH" to execute Perl scripts, which makes them work on more
esoteric systems like NixOS. Furthermore, adapt the shebang to use
env(1) to execute Perl so that users who have Perl in PATH, but in a
non-standard location can execute the script directly.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 t/perf/aggregate.perl | 2 +-
 t/perf/perf-lib.sh    | 4 ++--
 t/perf/run            | 4 ++--
 3 files changed, 5 insertions(+), 5 deletions(-)

diff --git a/t/perf/aggregate.perl b/t/perf/aggregate.perl
index 575d2000cc1..1791c7528a9 100755
--- a/t/perf/aggregate.perl
+++ b/t/perf/aggregate.perl
@@ -1,4 +1,4 @@
-#!/usr/bin/perl
+#!/usr/bin/env perl
 
 use lib '../../perl/build/lib';
 use strict;
diff --git a/t/perf/perf-lib.sh b/t/perf/perf-lib.sh
index 1a9a51ca3cc..4173eee4def 100644
--- a/t/perf/perf-lib.sh
+++ b/t/perf/perf-lib.sh
@@ -276,7 +276,7 @@ test_perf_ () {
 	else
 		test_ok_ "$1"
 	fi
-	"$TEST_DIRECTORY"/perf/min_time.perl test_time.* >"$base".result
+	"$PERL_PATH" "$TEST_DIRECTORY"/perf/min_time.perl test_time.* >"$base".result
 	rm test_time.*
 }
 
@@ -324,7 +324,7 @@ test_at_end_hook_ () {
 	if test -z "$GIT_PERF_AGGREGATING_LATER"; then
 		(
 			cd "$TEST_DIRECTORY"/perf &&
-			./aggregate.perl --results-dir="$TEST_RESULTS_DIR" $(basename "$0")
+			"$PERL_PATH" ./aggregate.perl --results-dir="$TEST_RESULTS_DIR" $(basename "$0")
 		)
 	fi
 }
diff --git a/t/perf/run b/t/perf/run
index 486ead21980..073bcb2afff 100755
--- a/t/perf/run
+++ b/t/perf/run
@@ -192,10 +192,10 @@ run_subsection () {
 
 	if test -z "$GIT_PERF_SEND_TO_CODESPEED"
 	then
-		./aggregate.perl --results-dir="$TEST_RESULTS_DIR" $codespeed_opt "$@"
+		"$PERL_PATH" ./aggregate.perl --results-dir="$TEST_RESULTS_DIR" $codespeed_opt "$@"
 	else
 		json_res_file=""$TEST_RESULTS_DIR"/$GIT_PERF_SUBSECTION/aggregate.json"
-		./aggregate.perl --results-dir="$TEST_RESULTS_DIR" --codespeed "$@" | tee "$json_res_file"
+		"$PERL_PATH" ./aggregate.perl --results-dir="$TEST_RESULTS_DIR" --codespeed "$@" | tee "$json_res_file"
 		send_data_url="$GIT_PERF_SEND_TO_CODESPEED/result/add/json/"
 		curl -v --request POST --data-urlencode "json=$(cat "$json_res_file")" "$send_data_url"
 	fi

-- 
2.49.0.805.g082f7c87e0.dirty

[PATCH v2 3/5] t/perf: fix benchmarks with out-of-tree builds

[Patrick Steinhardt]

The "perf-lib.sh" script is sourced by all of our benchmarking suites to
make available common infrastructure. The script assumes that build and
source directory are the same, which works for our Makefile. But the
assumption breaks with both CMake and Meson, where the build directory
can be located in an arbitrary place.

Adapt the script so that it works with out-of-tree builds. This prepares
us for wiring up benchmarks in Meson.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 t/perf/perf-lib.sh | 26 ++++++++++++++++++++++++--
 1 file changed, 24 insertions(+), 2 deletions(-)

diff --git a/t/perf/perf-lib.sh b/t/perf/perf-lib.sh
index 4173eee4def..5406557b7ca 100644
--- a/t/perf/perf-lib.sh
+++ b/t/perf/perf-lib.sh
@@ -25,7 +25,29 @@ TEST_OUTPUT_DIRECTORY=$(pwd)
 TEST_NO_CREATE_REPO=t
 TEST_NO_MALLOC_CHECK=t
 
-. ../test-lib.sh
+# While test-lib.sh computes the build directory for us, we also have to do the
+# same thing in order to locate the script via GIT-BUILD-OPTIONS in the first
+# place.
+GIT_BUILD_DIR="${GIT_BUILD_DIR:-$TEST_DIRECTORY/..}"
+if test -f "$GIT_BUILD_DIR/GIT-BUILD-DIR"
+then
+	GIT_BUILD_DIR="$(cat "$GIT_BUILD_DIR/GIT-BUILD-DIR")" || exit 1
+	# On Windows, we must convert Windows paths lest they contain a colon
+	case "$(uname -s)" in
+	*MINGW*)
+		GIT_BUILD_DIR="$(cygpath -au "$GIT_BUILD_DIR")"
+		;;
+	esac
+fi
+
+if test ! -f "$GIT_BUILD_DIR"/GIT-BUILD-OPTIONS
+then
+	echo >&2 'error: GIT-BUILD-OPTIONS missing (has Git been built?).'
+	exit 1
+fi
+
+. "$GIT_BUILD_DIR"/GIT-BUILD-OPTIONS
+. "$GIT_SOURCE_DIR"/t/test-lib.sh
 
 unset GIT_CONFIG_NOSYSTEM
 GIT_CONFIG_SYSTEM="$TEST_DIRECTORY/perf/config"
@@ -324,7 +346,7 @@ test_at_end_hook_ () {
 	if test -z "$GIT_PERF_AGGREGATING_LATER"; then
 		(
 			cd "$TEST_DIRECTORY"/perf &&
-			"$PERL_PATH" ./aggregate.perl --results-dir="$TEST_RESULTS_DIR" $(basename "$0")
+			"$PERL_PATH" "$GIT_SOURCE_DIR"/t/perf/aggregate.perl --results-dir="$TEST_RESULTS_DIR" $(basename "$0")
 		)
 	fi
 }

-- 
2.49.0.805.g082f7c87e0.dirty

Re: [PATCH v2 3/5] t/perf: fix benchmarks with out-of-tree builds

[Christian Couder]

On Mon, Apr 14, 2025 at 8:51 AM Patrick Steinhardt <ps@pks.im> wrote:

> --- a/t/perf/perf-lib.sh
> +++ b/t/perf/perf-lib.sh
> @@ -25,7 +25,29 @@ TEST_OUTPUT_DIRECTORY=$(pwd)
>  TEST_NO_CREATE_REPO=t
>  TEST_NO_MALLOC_CHECK=t
>
> -. ../test-lib.sh
> +# While test-lib.sh computes the build directory for us, we also have to do the
> +# same thing in order to locate the script via GIT-BUILD-OPTIONS in the first
> +# place.
> +GIT_BUILD_DIR="${GIT_BUILD_DIR:-$TEST_DIRECTORY/..}"

Right now on 'master' there is:

GIT_BUILD_DIR="${GIT_BUILD_DIR:-${TEST_DIRECTORY%/t}}"
if test "$TEST_DIRECTORY" = "$GIT_BUILD_DIR"
then
       echo "PANIC: Running in a $TEST_DIRECTORY that doesn't end in '/t'?" >&2
       exit 1
fi

so it's not exactly the same thing, even if it still probably works well.

Future readers might wonder if this discrepancy results from changes
that were made to only one of the files or if we really wanted to get
rid of the "/t" check here. In case we do want to get rid of the "/t"
check, I think it might be worth saying it clearly in the comment.

> +if test -f "$GIT_BUILD_DIR/GIT-BUILD-DIR"
> +then
> +       GIT_BUILD_DIR="$(cat "$GIT_BUILD_DIR/GIT-BUILD-DIR")" || exit 1
> +       # On Windows, we must convert Windows paths lest they contain a colon
> +       case "$(uname -s)" in
> +       *MINGW*)
> +               GIT_BUILD_DIR="$(cygpath -au "$GIT_BUILD_DIR")"
> +               ;;
> +       esac
> +fi
> +
> +if test ! -f "$GIT_BUILD_DIR"/GIT-BUILD-OPTIONS
> +then
> +       echo >&2 'error: GIT-BUILD-OPTIONS missing (has Git been built?).'

Maybe something like the following could help debug this:

       echo >&2 "error: GIT-BUILD-OPTIONS file missing from '$GIT_BUILD_DIR'"
       echo >&2 'error: (has Git been built?).'

> +       exit 1
> +fi

Re: [PATCH v2 3/5] t/perf: fix benchmarks with out-of-tree builds

[Patrick Steinhardt]

On Sun, Apr 20, 2025 at 12:00:08PM +0200, Christian Couder wrote:
> On Mon, Apr 14, 2025 at 8:51 AM Patrick Steinhardt <ps@pks.im> wrote:
> 
> > --- a/t/perf/perf-lib.sh
> > +++ b/t/perf/perf-lib.sh
> > @@ -25,7 +25,29 @@ TEST_OUTPUT_DIRECTORY=$(pwd)
> >  TEST_NO_CREATE_REPO=t
> >  TEST_NO_MALLOC_CHECK=t
> >
> > -. ../test-lib.sh
> > +# While test-lib.sh computes the build directory for us, we also have to do the
> > +# same thing in order to locate the script via GIT-BUILD-OPTIONS in the first
> > +# place.
> > +GIT_BUILD_DIR="${GIT_BUILD_DIR:-$TEST_DIRECTORY/..}"
> 
> Right now on 'master' there is:
> 
> GIT_BUILD_DIR="${GIT_BUILD_DIR:-${TEST_DIRECTORY%/t}}"
> if test "$TEST_DIRECTORY" = "$GIT_BUILD_DIR"
> then
>        echo "PANIC: Running in a $TEST_DIRECTORY that doesn't end in '/t'?" >&2
>        exit 1
> fi
> 
> so it's not exactly the same thing, even if it still probably works well.
> 
> Future readers might wonder if this discrepancy results from changes
> that were made to only one of the files or if we really wanted to get
> rid of the "/t" check here. In case we do want to get rid of the "/t"
> check, I think it might be worth saying it clearly in the comment.

The "/.." is intentional here due to the way that `TEST_DIRECTORY` is
constructed. If you extend the context of this patch a bit, you can see
that `TEST_DIRECTORY=$(pwd)/..`. So stripping "/t" from the suffix
wouldn't do anything because it never has that suffix in the first
place. And neither do we  want to strip "/..", because then we'd end up
in "t/perf". So the easiest fix is to just append another "/.." to end
up where we want to.

I'll try to paraphrase this in the commit message.

> > +if test -f "$GIT_BUILD_DIR/GIT-BUILD-DIR"
> > +then
> > +       GIT_BUILD_DIR="$(cat "$GIT_BUILD_DIR/GIT-BUILD-DIR")" || exit 1
> > +       # On Windows, we must convert Windows paths lest they contain a colon
> > +       case "$(uname -s)" in
> > +       *MINGW*)
> > +               GIT_BUILD_DIR="$(cygpath -au "$GIT_BUILD_DIR")"
> > +               ;;
> > +       esac
> > +fi
> > +
> > +if test ! -f "$GIT_BUILD_DIR"/GIT-BUILD-OPTIONS
> > +then
> > +       echo >&2 'error: GIT-BUILD-OPTIONS missing (has Git been built?).'
> 
> Maybe something like the following could help debug this:
> 
>        echo >&2 "error: GIT-BUILD-OPTIONS file missing from '$GIT_BUILD_DIR'"
>        echo >&2 'error: (has Git been built?).'

I'd rather want to keep this as-is for now as we have the same error
message in "test-lib.sh". If we want to change it we should change both
errors, but that feels outside of the scope of this patch series.

Patrick

[PATCH v2 4/5] meson: wire up benchmarks

[Patrick Steinhardt]

Wire up benchmarks in Meson. The setup is mostly the same as how we wire
up our tests. The only difference is that benchmarks get wired up via
the `benchmark()` option instead of via `test()`, which gives them a bit
of special treatment:

  - Benchmarks never run in parallel.

  - Benchmarks aren't run by default when tests are executed.

  - Meson does not inject the `MALLOC_PERTURB` environment variable.

Using benchmarks is quite simple:

    ```
    $ meson setup build
    # Run all benchmarks.
    $ meson test -C build --benchmark
    # Run a specific benchmark.
    $ meson test -C build --benchmark p0000-*
    ```

Other than that the usual command line arguments accepted when running
tests are also accepted when running benchmarks.

Note that the benchmarking target is somewhat limited because it will
only run benchmarks for the current build. Other usecases, like running
benchmarks against multiple different versions of Git, are not currently
supported. Users should continue to use "t/perf/run" for those usecases.
The script should get extended at one point in time to support Meson,
but this is outside of the scope of this series.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 meson.build   |  1 +
 t/meson.build | 77 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 78 insertions(+)

diff --git a/meson.build b/meson.build
index efe2871c9db..9af4a10188d 100644
--- a/meson.build
+++ b/meson.build
@@ -204,6 +204,7 @@ git = find_program('git', dirs: program_path, required: false)
 sed = find_program('sed', dirs: program_path)
 shell = find_program('sh', dirs: program_path)
 tar = find_program('tar', dirs: program_path)
+time = find_program('time', dirs: program_path, required: false)
 
 # Sanity-check that programs required for the build exist.
 foreach tool : ['cat', 'cut', 'grep', 'sort', 'tr', 'uname']
diff --git a/t/meson.build b/t/meson.build
index a59da26be3f..dba327fd7ec 100644
--- a/t/meson.build
+++ b/t/meson.build
@@ -1096,11 +1096,71 @@ integration_tests = [
   't9903-bash-prompt.sh',
 ]
 
+benchmarks = [
+  'perf/p0000-perf-lib-sanity.sh',
+  'perf/p0001-rev-list.sh',
+  'perf/p0002-read-cache.sh',
+  'perf/p0003-delta-base-cache.sh',
+  'perf/p0004-lazy-init-name-hash.sh',
+  'perf/p0005-status.sh',
+  'perf/p0006-read-tree-checkout.sh',
+  'perf/p0007-write-cache.sh',
+  'perf/p0008-odb-fsync.sh',
+  'perf/p0071-sort.sh',
+  'perf/p0090-cache-tree.sh',
+  'perf/p0100-globbing.sh',
+  'perf/p1006-cat-file.sh',
+  'perf/p1400-update-ref.sh',
+  'perf/p1450-fsck.sh',
+  'perf/p1451-fsck-skip-list.sh',
+  'perf/p1500-graph-walks.sh',
+  'perf/p2000-sparse-operations.sh',
+  'perf/p3400-rebase.sh',
+  'perf/p3404-rebase-interactive.sh',
+  'perf/p4000-diff-algorithms.sh',
+  'perf/p4001-diff-no-index.sh',
+  'perf/p4002-diff-color-moved.sh',
+  'perf/p4205-log-pretty-formats.sh',
+  'perf/p4209-pickaxe.sh',
+  'perf/p4211-line-log.sh',
+  'perf/p4220-log-grep-engines.sh',
+  'perf/p4221-log-grep-engines-fixed.sh',
+  'perf/p5302-pack-index.sh',
+  'perf/p5303-many-packs.sh',
+  'perf/p5304-prune.sh',
+  'perf/p5310-pack-bitmaps.sh',
+  'perf/p5311-pack-bitmaps-fetch.sh',
+  'perf/p5312-pack-bitmaps-revs.sh',
+  'perf/p5313-pack-objects.sh',
+  'perf/p5314-name-hash.sh',
+  'perf/p5326-multi-pack-bitmaps.sh',
+  'perf/p5332-multi-pack-reuse.sh',
+  'perf/p5333-pseudo-merge-bitmaps.sh',
+  'perf/p5550-fetch-tags.sh',
+  'perf/p5551-fetch-rescan.sh',
+  'perf/p5600-partial-clone.sh',
+  'perf/p5601-clone-reference.sh',
+  'perf/p6100-describe.sh',
+  'perf/p6300-for-each-ref.sh',
+  'perf/p7000-filter-branch.sh',
+  'perf/p7102-reset.sh',
+  'perf/p7300-clean.sh',
+  'perf/p7519-fsmonitor.sh',
+  'perf/p7527-builtin-fsmonitor.sh',
+  'perf/p7810-grep.sh',
+  'perf/p7820-grep-engines.sh',
+  'perf/p7821-grep-engines-fixed.sh',
+  'perf/p7822-grep-perl-character.sh',
+  'perf/p9210-scalar.sh',
+  'perf/p9300-fast-import-export.sh',
+]
+
 # Sanity check that we are not missing any tests present in 't/'. This check
 # only runs once at configure time and is thus best-effort, only. It is
 # sufficient to catch missing test suites in our CI though.
 foreach glob, tests : {
   't[0-9][0-9][0-9][0-9]-*.sh': integration_tests,
+  'perf/p[0-9][0-9][0-9][0-9]-*.sh': benchmarks,
   'unit-tests/t-*.c': unit_test_programs,
   'unit-tests/u-*.c': clar_test_suites,
 }
@@ -1152,3 +1212,20 @@ foreach integration_test : integration_tests
     timeout: 0,
   )
 endforeach
+
+if time.found()
+  benchmark_environment = test_environment
+  benchmark_environment.set('GTIME', time.full_path())
+
+  foreach benchmark : benchmarks
+    benchmark(fs.stem(benchmark), shell,
+      args: [
+        fs.name(benchmark),
+      ],
+      workdir: meson.current_source_dir() / 'perf',
+      env: benchmark_environment,
+      depends: test_dependencies + bin_wrappers,
+      timeout: 0,
+    )
+  endforeach
+endif

-- 
2.49.0.805.g082f7c87e0.dirty

Re: [PATCH v2 4/5] meson: wire up benchmarks

[Christian Couder]

On Mon, Apr 14, 2025 at 8:51 AM Patrick Steinhardt <ps@pks.im> wrote:
>
> Wire up benchmarks in Meson. The setup is mostly the same as how we wire
> up our tests. The only difference is that benchmarks get wired up via
> the `benchmark()` option instead of via `test()`, which gives them a bit
> of special treatment:
>
>   - Benchmarks never run in parallel.
>
>   - Benchmarks aren't run by default when tests are executed.
>
>   - Meson does not inject the `MALLOC_PERTURB` environment variable.
>
> Using benchmarks is quite simple:
>
>     ```
>     $ meson setup build
>     # Run all benchmarks.
>     $ meson test -C build --benchmark
>     # Run a specific benchmark.
>     $ meson test -C build --benchmark p0000-*
>     ```

Instead of just here, I would have expected that something like this
would appear towards the top of the top level meson.build file, maybe
in a "Benchmarking" section, where there is the rest of the meson
documentation for Git.

> Other than that the usual command line arguments accepted when running
> tests are also accepted when running benchmarks.
>
> Note that the benchmarking target is somewhat limited because it will
> only run benchmarks for the current build. Other usecases, like running
> benchmarks against multiple different versions of Git, are not currently
> supported. Users should continue to use "t/perf/run" for those usecases.

s/usecases/use cases/

> The script should get extended at one point in time to support Meson,
> but this is outside of the scope of this series.

Fair enough.

> +if time.found()
> +  benchmark_environment = test_environment
> +  benchmark_environment.set('GTIME', time.full_path())
> +
> +  foreach benchmark : benchmarks
> +    benchmark(fs.stem(benchmark), shell,
> +      args: [
> +        fs.name(benchmark),
> +      ],
> +      workdir: meson.current_source_dir() / 'perf',
> +      env: benchmark_environment,
> +      depends: test_dependencies + bin_wrappers,
> +      timeout: 0,
> +    )
> +  endforeach
> +endif

I wonder what happens when we run `meson test -C build --benchmark`
but 'time' is not found.

[PATCH v2 5/5] meson: wire up benchmarking options

[Patrick Steinhardt]

Wire up a couple of benchmarking options that we end up writing into our
"GIT-BUILD-OPTIONS" file. These options allow users to control how
exactly benchmarks are executed.

Note that neither `GIT_PERF_MAKE_COMMAND` nor `GIT_PERF_MAKE_OPTS` are
exposed as a build option. Those options are used by "t/perf/run", which
is not used by Meson.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 meson.build       | 6 +++---
 meson_options.txt | 6 ++++++
 2 files changed, 9 insertions(+), 3 deletions(-)

diff --git a/meson.build b/meson.build
index 9af4a10188d..ba128ab0759 100644
--- a/meson.build
+++ b/meson.build
@@ -660,11 +660,11 @@ builtin_sources += custom_target(
 # build options to our tests.
 build_options_config = configuration_data()
 build_options_config.set('GIT_INTEROP_MAKE_OPTS', '')
-build_options_config.set('GIT_PERF_LARGE_REPO', '')
+build_options_config.set_quoted('GIT_PERF_LARGE_REPO', get_option('benchmark_large_repo'))
 build_options_config.set('GIT_PERF_MAKE_COMMAND', '')
 build_options_config.set('GIT_PERF_MAKE_OPTS', '')
-build_options_config.set('GIT_PERF_REPEAT_COUNT', '')
-build_options_config.set('GIT_PERF_REPO', '')
+build_options_config.set_quoted('GIT_PERF_REPEAT_COUNT', get_option('benchmark_repeat_count').to_string())
+build_options_config.set_quoted('GIT_PERF_REPO', get_option('benchmark_repo'))
 build_options_config.set('GIT_TEST_CMP_USE_COPIED_CONTEXT', '')
 build_options_config.set('GIT_TEST_INDEX_VERSION', '')
 build_options_config.set('GIT_TEST_OPTS', '')
diff --git a/meson_options.txt b/meson_options.txt
index 78d172a7401..ca106c43b2e 100644
--- a/meson_options.txt
+++ b/meson_options.txt
@@ -101,6 +101,12 @@ option('docs_backend', type: 'combo', choices: ['asciidoc', 'asciidoctor', 'auto
   description: 'Which backend to use to generate documentation.')
 
 # Testing.
+option('benchmark_repo', type: 'string', value: '',
+  description: 'Repository to copy for the performance tests. Should be at least the size of the Git repository.')
+option('benchmark_large_repo', type: 'string', value: '',
+  description: 'Large repository to copy for the performance tests. Should be at least the size of the Linux repository.')
+option('benchmark_repeat_count', type: 'integer', value: 3,
+  description: 'Number of times a test should be repeated for best-of-N measurements.')
 option('coccinelle', type: 'feature', value: 'auto',
   description: 'Provide a coccicheck target that generates a Coccinelle patch.')
 option('tests', type: 'boolean', value: true,

-- 
2.49.0.805.g082f7c87e0.dirty

Re: [PATCH v2 0/5] meson: wire up support for benchmarks

[Junio C Hamano]

Patrick Steinhardt <ps@pks.im> writes:

> this small patch series implements support for running our benchmarks in
> "t/perf" via Meson. The series does not aim to replace "t/perf/run",
> which is more fully-featured and allows running benchmarks against
> multiple different trees. Instead, this series only allows running the
> benchmarks against the current tree. Users are thus expected to continue
> using "t/perf/run" for more advanced usecases.
>
> Changes in v2:
>   - Adapt "aggregate.perl" to use a "/usr/bin/env perl" shebang.
>   - Link to v1: https://lore.kernel.org/r/20250331-pks-meson-benchmarks-v1-0-b2ace85616a3@pks.im
>
> Thanks!


The previous iteration of this series has been kept out of 'seen'
for some time and I didn't recall why I did so.  With this iteration
merged, all GitHub Actions CI tasks with "meson" in the name are
failing, so does "documentation" jobs (which recently acquired
"let's make sure meson-based build does the docs fine" substep).

Can you help seeing where the merge went wrong (yes, I am suspecting
that there is some stupid merge mistake there)?

Thanks.

Re: [PATCH v2 0/5] meson: wire up support for benchmarks

[Junio C Hamano]

Junio C Hamano <gitster@pobox.com> writes:

> Patrick Steinhardt <ps@pks.im> writes:
>
>> this small patch series implements support for running our benchmarks in
>> "t/perf" via Meson. The series does not aim to replace "t/perf/run",
>> which is more fully-featured and allows running benchmarks against
>> multiple different trees. Instead, this series only allows running the
>> benchmarks against the current tree. Users are thus expected to continue
>> using "t/perf/run" for more advanced usecases.
>>
>> Changes in v2:
>>   - Adapt "aggregate.perl" to use a "/usr/bin/env perl" shebang.
>>   - Link to v1: https://lore.kernel.org/r/20250331-pks-meson-benchmarks-v1-0-b2ace85616a3@pks.im
>>
>> Thanks!
>
>
> The previous iteration of this series has been kept out of 'seen'
> for some time and I didn't recall why I did so.  With this iteration
> merged, all GitHub Actions CI tasks with "meson" in the name are
> failing, so does "documentation" jobs (which recently acquired
> "let's make sure meson-based build does the docs fine" substep).
>
> Can you help seeing where the merge went wrong (yes, I am suspecting
> that there is some stupid merge mistake there)?


Just a few test CI runs.

    https://github.com/git/git/actions/runs/14457387669

is with this topic mergecd in (with alleged mismerge).

This one

    https://github.com/git/git/actions/runs/14406901394

is from last week without the earlier iteration of this topic, which
fails a few meson jobs.

linux-meson job that fails with

meson.build:689:19: ERROR: Command `/usr/bin/git -C /__w/git/git ls-files --deduplicate '*.h' ':!contrib' ':!compat/inet_ntop.c' ':!compat/inet_pton.c' ':!compat/nedmalloc' ':!compat/obstack.*' ':!compat/poll' ':!compat/regex' ':!sha1collisiondetection' ':!sha1dc' ':!t/unit-tests/clar' ':!t/unit-tests/clar' ':!t/t[0-9][0-9][0-9][0-9]*'` failed with status 128.

and

win+Meson test(3) that dies inside Python asyncio both look
problematic.

Thanks.

Re: [PATCH v2 0/5] meson: wire up support for benchmarks

[Patrick Steinhardt]

On Tue, Apr 15, 2025 at 11:18:42AM -0700, Junio C Hamano wrote:
> Junio C Hamano <gitster@pobox.com> writes:
> 
> > Patrick Steinhardt <ps@pks.im> writes:
> >
> >> this small patch series implements support for running our benchmarks in
> >> "t/perf" via Meson. The series does not aim to replace "t/perf/run",
> >> which is more fully-featured and allows running benchmarks against
> >> multiple different trees. Instead, this series only allows running the
> >> benchmarks against the current tree. Users are thus expected to continue
> >> using "t/perf/run" for more advanced usecases.
> >>
> >> Changes in v2:
> >>   - Adapt "aggregate.perl" to use a "/usr/bin/env perl" shebang.
> >>   - Link to v1: https://lore.kernel.org/r/20250331-pks-meson-benchmarks-v1-0-b2ace85616a3@pks.im
> >>
> >> Thanks!
> >
> >
> > The previous iteration of this series has been kept out of 'seen'
> > for some time and I didn't recall why I did so.  With this iteration
> > merged, all GitHub Actions CI tasks with "meson" in the name are
> > failing, so does "documentation" jobs (which recently acquired
> > "let's make sure meson-based build does the docs fine" substep).
> >
> > Can you help seeing where the merge went wrong (yes, I am suspecting
> > that there is some stupid merge mistake there)?
> 
> 
> Just a few test CI runs.
> 
>     https://github.com/git/git/actions/runs/14457387669
> 
> is with this topic mergecd in (with alleged mismerge).
> 
> This one
> 
>     https://github.com/git/git/actions/runs/14406901394
> 
> is from last week without the earlier iteration of this topic, which
> fails a few meson jobs.
> 
> linux-meson job that fails with
> 
> meson.build:689:19: ERROR: Command `/usr/bin/git -C /__w/git/git ls-files --deduplicate '*.h' ':!contrib' ':!compat/inet_ntop.c' ':!compat/inet_pton.c' ':!compat/nedmalloc' ':!compat/obstack.*' ':!compat/poll' ':!compat/regex' ':!sha1collisiondetection' ':!sha1dc' ':!t/unit-tests/clar' ':!t/unit-tests/clar' ':!t/t[0-9][0-9][0-9][0-9]*'` failed with status 128.

Hm, curious. These lines have been moved here by kn/meson-hdr-check, but
the logic already existed beforehand. I've Cc'd Karthik -- it would
probably make sense to investigate (and ideally also fix) the issue as
part of that series.

> and
> 
> win+Meson test(3) that dies inside Python asyncio both look
> problematic.

Yeah, this is a result of cancellation of the job. Johannes already
mentioned this issue to me in the past, and it's known that Python's
asyncio is a bit wonky in Windows. The problem here is mostly that
Windows does not have proper signals.

It's not a huge problem given that this only happens when the job is
being cancelled, but it's not great, either. I've already invested some
time into improving this on Windows [1], but there are a couple of
uncertainties still. Anyway, this should hopefully be fixed soonish.

Patrick

[1]: https://github.com/mesonbuild/meson/pull/14311

Re: [PATCH v2 0/5] meson: wire up support for benchmarks

[Junio C Hamano]

Patrick Steinhardt <ps@pks.im> writes:

>> This one
>> 
>>     https://github.com/git/git/actions/runs/14406901394
>> 
>> is from last week without the earlier iteration of this topic, which
>> fails a few meson jobs.
>> 
>> linux-meson job that fails with
>> 
>> meson.build:689:19: ERROR: Command `/usr/bin/git -C /__w/git/git ls-files --deduplicate '*.h' ':!contrib' ':!compat/inet_ntop.c' ':!compat/inet_pton.c' ':!compat/nedmalloc' ':!compat/obstack.*' ':!compat/poll' ':!compat/regex' ':!sha1collisiondetection' ':!sha1dc' ':!t/unit-tests/clar' ':!t/unit-tests/clar' ':!t/t[0-9][0-9][0-9][0-9]*'` failed with status 128.
>
> Hm, curious. These lines have been moved here by kn/meson-hdr-check, but
> the logic already existed beforehand. I've Cc'd Karthik -- it would
> probably make sense to investigate (and ideally also fix) the issue as
> part of that series.

Indeed.  I tentatively kicked meson-hdr-check topic out of 'seen'
and the CI seems to be a lot happier.  What is curious is that in my
local environment 'seen' with meson-hdr-check topic did not fail for
me.

[PATCH v3 0/5] meson: wire up support for benchmarks

[Patrick Steinhardt]

Hi,

this small patch series implements support for running our benchmarks in
"t/perf" via Meson. The series does not aim to replace "t/perf/run",
which is more fully-featured and allows running benchmarks against
multiple different trees. Instead, this series only allows running the
benchmarks against the current tree. Users are thus expected to continue
using "t/perf/run" for more advanced usecases.

Changes in v2:
  - Adapt "aggregate.perl" to use a "/usr/bin/env perl" shebang.
  - Link to v1: https://lore.kernel.org/r/20250331-pks-meson-benchmarks-v1-0-b2ace85616a3@pks.im

Changes in v3:
  - Document how to run benchmarks in "meson.build".
  - Expand the message for the commit that enables out-of-tree
    benchmarking.
  - Link to v2: https://lore.kernel.org/r/20250414-pks-meson-benchmarks-v2-0-04377080a167@pks.im

Thanks!

Patrick

---
Patrick Steinhardt (5):
      t/perf: fix benchmarks with alternate repo formats
      t/perf: use configured PERL_PATH
      t/perf: fix benchmarks with out-of-tree builds
      meson: wire up benchmarks
      meson: wire up benchmarking options

 meson.build           | 13 +++++++--
 meson_options.txt     |  6 ++++
 t/meson.build         | 77 +++++++++++++++++++++++++++++++++++++++++++++++++++
 t/perf/aggregate.perl |  2 +-
 t/perf/perf-lib.sh    | 32 ++++++++++++++++++---
 t/perf/run            |  4 +--
 6 files changed, 124 insertions(+), 10 deletions(-)

Range-diff versus v2:

1:  dad3c5a3e78 = 1:  85098f35716 t/perf: fix benchmarks with alternate repo formats
2:  42284091b9c = 2:  457ea957acf t/perf: use configured PERL_PATH
3:  d03ca0c0630 ! 3:  a66bc5aaf51 t/perf: fix benchmarks with out-of-tree builds
    @@ Commit message
         assumption breaks with both CMake and Meson, where the build directory
         can be located in an arbitrary place.
     
    -    Adapt the script so that it works with out-of-tree builds. This prepares
    -    us for wiring up benchmarks in Meson.
    +    Adapt the script so that it works with out-of-tree builds. Most
    +    importantly, this requires us to figure out the location of the build
    +    directory:
    +
    +      - When running benchmarks via our Makefile the build directory is the
    +        same as the source directory. We already know to derive the test
    +        directory ("t/") via `$(pwd)/..`, which works because we chdir into
    +        "t/perf" before executing benchmarks. We can thus derive the build
    +        directory by appending another "/.." to that path.
    +
    +      - When running benchmarks via Meson the build directory is located at
    +        an arbitrary location. The build system thus has to make the path
    +        known by exporting the `GIT_BUILD_DIR` environment variable.
    +
    +    This change prepares us for wiring up benchmarks in Meson.
     
         Signed-off-by: Patrick Steinhardt <ps@pks.im>
     
4:  14945278066 ! 4:  386b960f3e1 meson: wire up benchmarks
    @@ Commit message
         tests are also accepted when running benchmarks.
     
         Note that the benchmarking target is somewhat limited because it will
    -    only run benchmarks for the current build. Other usecases, like running
    +    only run benchmarks for the current build. Other use cases, like running
         benchmarks against multiple different versions of Git, are not currently
    -    supported. Users should continue to use "t/perf/run" for those usecases.
    -    The script should get extended at one point in time to support Meson,
    -    but this is outside of the scope of this series.
    +    supported. Users should continue to use "t/perf/run" for those use
    +    cases. The script should get extended at one point in time to support
    +    Meson, but this is outside of the scope of this series.
     
         Signed-off-by: Patrick Steinhardt <ps@pks.im>
     
      ## meson.build ##
    +@@
    + #      # Execute single test interactively such that features like `debug ()` work.
    + #      $ meson test -i --test-args='-ix' t1400-update-ref
    + #
    ++#      # Execute all benchmarks.
    ++#      $ meson test -i --benchmark
    ++#
    ++#      # Execute single benchmark.
    ++#      $ meson test -i --benchmark p0000-*
    ++#
    + #   Test execution is parallelized by default and scales with the number of
    + #   processor cores available. You can change the number of processes by passing
    + #   the `-jN` flag to `meson test`.
     @@ meson.build: git = find_program('git', dirs: program_path, required: false)
      sed = find_program('sed', dirs: program_path)
      shell = find_program('sh', dirs: program_path)
5:  101be1e8efc = 5:  cc3e771ada0 meson: wire up benchmarking options

---
base-commit: 683c54c999c301c2cd6f715c411407c413b1d84e
change-id: 20250328-pks-meson-benchmarks-a8fac5f69467

[PATCH v3 1/5] t/perf: fix benchmarks with alternate repo formats

[Patrick Steinhardt]

Many of our benchmarks operate on a user-defined repository that we copy
over before running the benchmarked logic. To keep unintentional side
effects caused by on-disk state at bay we skip copying some files. This
includes for example hooks, but also the repo's configuration.

It is quite sensible to not copy over the configuration, as it is quite
easy to inadvertently carry over configuration that may significantly
impact the performance measurements. But we cannot fully ignore the
configuration either, as it may contain information about the repository
format. This will cause failures when for example using a repository
with SHA256 object format or the reftable ref format.

Fix the issue by parsing the reference and object formats from the
source repository and passing them to git-init(1).

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 t/perf/perf-lib.sh | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/t/perf/perf-lib.sh b/t/perf/perf-lib.sh
index 8ab6d9c4694..1a9a51ca3cc 100644
--- a/t/perf/perf-lib.sh
+++ b/t/perf/perf-lib.sh
@@ -98,6 +98,8 @@ test_perf_create_repo_from () {
 	source_git="$("$MODERN_GIT" -C "$source" rev-parse --git-dir)"
 	objects_dir="$("$MODERN_GIT" -C "$source" rev-parse --git-path objects)"
 	common_dir="$("$MODERN_GIT" -C "$source" rev-parse --git-common-dir)"
+	refformat="$("$MODERN_GIT" -C "$source" rev-parse --show-ref-format)"
+	objectformat="$("$MODERN_GIT" -C "$source" rev-parse --show-object-format)"
 	mkdir -p "$repo/.git"
 	(
 		cd "$source" &&
@@ -114,7 +116,7 @@ test_perf_create_repo_from () {
 	) &&
 	(
 		cd "$repo" &&
-		"$MODERN_GIT" init -q &&
+		"$MODERN_GIT" init -q --ref-format="$refformat" --object-format="$objectformat" &&
 		test_perf_do_repo_symlink_config_ &&
 		mv .git/hooks .git/hooks-disabled 2>/dev/null &&
 		if test -f .git/index.lock

-- 
2.49.0.901.g37484f566f.dirty

[PATCH v3 2/5] t/perf: use configured PERL_PATH

[Patrick Steinhardt]

Our benchmarks use a couple of Perl scripts to compute results. These
Perl scripts get executed directly, and as the shebang is hardcoded to
"/usr/bin/perl" this will fail on any system where the Perl interpreter
is located in a different path.

Our build infrastructure already lets users configure the location of
Perl, which ultimately gets written into the GIT-BUILD-OPTIONS file.
This file is being sourced by "test-lib.sh", and consequently we already
have the "PERL_PATH" variable available that contains its configured
location.

Use "PERL_PATH" to execute Perl scripts, which makes them work on more
esoteric systems like NixOS. Furthermore, adapt the shebang to use
env(1) to execute Perl so that users who have Perl in PATH, but in a
non-standard location can execute the script directly.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 t/perf/aggregate.perl | 2 +-
 t/perf/perf-lib.sh    | 4 ++--
 t/perf/run            | 4 ++--
 3 files changed, 5 insertions(+), 5 deletions(-)

diff --git a/t/perf/aggregate.perl b/t/perf/aggregate.perl
index 575d2000cc1..1791c7528a9 100755
--- a/t/perf/aggregate.perl
+++ b/t/perf/aggregate.perl
@@ -1,4 +1,4 @@
-#!/usr/bin/perl
+#!/usr/bin/env perl
 
 use lib '../../perl/build/lib';
 use strict;
diff --git a/t/perf/perf-lib.sh b/t/perf/perf-lib.sh
index 1a9a51ca3cc..4173eee4def 100644
--- a/t/perf/perf-lib.sh
+++ b/t/perf/perf-lib.sh
@@ -276,7 +276,7 @@ test_perf_ () {
 	else
 		test_ok_ "$1"
 	fi
-	"$TEST_DIRECTORY"/perf/min_time.perl test_time.* >"$base".result
+	"$PERL_PATH" "$TEST_DIRECTORY"/perf/min_time.perl test_time.* >"$base".result
 	rm test_time.*
 }
 
@@ -324,7 +324,7 @@ test_at_end_hook_ () {
 	if test -z "$GIT_PERF_AGGREGATING_LATER"; then
 		(
 			cd "$TEST_DIRECTORY"/perf &&
-			./aggregate.perl --results-dir="$TEST_RESULTS_DIR" $(basename "$0")
+			"$PERL_PATH" ./aggregate.perl --results-dir="$TEST_RESULTS_DIR" $(basename "$0")
 		)
 	fi
 }
diff --git a/t/perf/run b/t/perf/run
index 486ead21980..073bcb2afff 100755
--- a/t/perf/run
+++ b/t/perf/run
@@ -192,10 +192,10 @@ run_subsection () {
 
 	if test -z "$GIT_PERF_SEND_TO_CODESPEED"
 	then
-		./aggregate.perl --results-dir="$TEST_RESULTS_DIR" $codespeed_opt "$@"
+		"$PERL_PATH" ./aggregate.perl --results-dir="$TEST_RESULTS_DIR" $codespeed_opt "$@"
 	else
 		json_res_file=""$TEST_RESULTS_DIR"/$GIT_PERF_SUBSECTION/aggregate.json"
-		./aggregate.perl --results-dir="$TEST_RESULTS_DIR" --codespeed "$@" | tee "$json_res_file"
+		"$PERL_PATH" ./aggregate.perl --results-dir="$TEST_RESULTS_DIR" --codespeed "$@" | tee "$json_res_file"
 		send_data_url="$GIT_PERF_SEND_TO_CODESPEED/result/add/json/"
 		curl -v --request POST --data-urlencode "json=$(cat "$json_res_file")" "$send_data_url"
 	fi

-- 
2.49.0.901.g37484f566f.dirty

[PATCH v3 3/5] t/perf: fix benchmarks with out-of-tree builds

[Patrick Steinhardt]

The "perf-lib.sh" script is sourced by all of our benchmarking suites to
make available common infrastructure. The script assumes that build and
source directory are the same, which works for our Makefile. But the
assumption breaks with both CMake and Meson, where the build directory
can be located in an arbitrary place.

Adapt the script so that it works with out-of-tree builds. Most
importantly, this requires us to figure out the location of the build
directory:

  - When running benchmarks via our Makefile the build directory is the
    same as the source directory. We already know to derive the test
    directory ("t/") via `$(pwd)/..`, which works because we chdir into
    "t/perf" before executing benchmarks. We can thus derive the build
    directory by appending another "/.." to that path.

  - When running benchmarks via Meson the build directory is located at
    an arbitrary location. The build system thus has to make the path
    known by exporting the `GIT_BUILD_DIR` environment variable.

This change prepares us for wiring up benchmarks in Meson.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 t/perf/perf-lib.sh | 26 ++++++++++++++++++++++++--
 1 file changed, 24 insertions(+), 2 deletions(-)

diff --git a/t/perf/perf-lib.sh b/t/perf/perf-lib.sh
index 4173eee4def..5406557b7ca 100644
--- a/t/perf/perf-lib.sh
+++ b/t/perf/perf-lib.sh
@@ -25,7 +25,29 @@ TEST_OUTPUT_DIRECTORY=$(pwd)
 TEST_NO_CREATE_REPO=t
 TEST_NO_MALLOC_CHECK=t
 
-. ../test-lib.sh
+# While test-lib.sh computes the build directory for us, we also have to do the
+# same thing in order to locate the script via GIT-BUILD-OPTIONS in the first
+# place.
+GIT_BUILD_DIR="${GIT_BUILD_DIR:-$TEST_DIRECTORY/..}"
+if test -f "$GIT_BUILD_DIR/GIT-BUILD-DIR"
+then
+	GIT_BUILD_DIR="$(cat "$GIT_BUILD_DIR/GIT-BUILD-DIR")" || exit 1
+	# On Windows, we must convert Windows paths lest they contain a colon
+	case "$(uname -s)" in
+	*MINGW*)
+		GIT_BUILD_DIR="$(cygpath -au "$GIT_BUILD_DIR")"
+		;;
+	esac
+fi
+
+if test ! -f "$GIT_BUILD_DIR"/GIT-BUILD-OPTIONS
+then
+	echo >&2 'error: GIT-BUILD-OPTIONS missing (has Git been built?).'
+	exit 1
+fi
+
+. "$GIT_BUILD_DIR"/GIT-BUILD-OPTIONS
+. "$GIT_SOURCE_DIR"/t/test-lib.sh
 
 unset GIT_CONFIG_NOSYSTEM
 GIT_CONFIG_SYSTEM="$TEST_DIRECTORY/perf/config"
@@ -324,7 +346,7 @@ test_at_end_hook_ () {
 	if test -z "$GIT_PERF_AGGREGATING_LATER"; then
 		(
 			cd "$TEST_DIRECTORY"/perf &&
-			"$PERL_PATH" ./aggregate.perl --results-dir="$TEST_RESULTS_DIR" $(basename "$0")
+			"$PERL_PATH" "$GIT_SOURCE_DIR"/t/perf/aggregate.perl --results-dir="$TEST_RESULTS_DIR" $(basename "$0")
 		)
 	fi
 }

-- 
2.49.0.901.g37484f566f.dirty

[PATCH v3 4/5] meson: wire up benchmarks

[Patrick Steinhardt]

Wire up benchmarks in Meson. The setup is mostly the same as how we wire
up our tests. The only difference is that benchmarks get wired up via
the `benchmark()` option instead of via `test()`, which gives them a bit
of special treatment:

  - Benchmarks never run in parallel.

  - Benchmarks aren't run by default when tests are executed.

  - Meson does not inject the `MALLOC_PERTURB` environment variable.

Using benchmarks is quite simple:

    ```
    $ meson setup build
    # Run all benchmarks.
    $ meson test -C build --benchmark
    # Run a specific benchmark.
    $ meson test -C build --benchmark p0000-*
    ```

Other than that the usual command line arguments accepted when running
tests are also accepted when running benchmarks.

Note that the benchmarking target is somewhat limited because it will
only run benchmarks for the current build. Other use cases, like running
benchmarks against multiple different versions of Git, are not currently
supported. Users should continue to use "t/perf/run" for those use
cases. The script should get extended at one point in time to support
Meson, but this is outside of the scope of this series.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 meson.build   |  7 ++++++
 t/meson.build | 77 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 84 insertions(+)

diff --git a/meson.build b/meson.build
index efe2871c9db..205b1bb301a 100644
--- a/meson.build
+++ b/meson.build
@@ -70,6 +70,12 @@
 #      # Execute single test interactively such that features like `debug ()` work.
 #      $ meson test -i --test-args='-ix' t1400-update-ref
 #
+#      # Execute all benchmarks.
+#      $ meson test -i --benchmark
+#
+#      # Execute single benchmark.
+#      $ meson test -i --benchmark p0000-*
+#
 #   Test execution is parallelized by default and scales with the number of
 #   processor cores available. You can change the number of processes by passing
 #   the `-jN` flag to `meson test`.
@@ -204,6 +210,7 @@ git = find_program('git', dirs: program_path, required: false)
 sed = find_program('sed', dirs: program_path)
 shell = find_program('sh', dirs: program_path)
 tar = find_program('tar', dirs: program_path)
+time = find_program('time', dirs: program_path, required: false)
 
 # Sanity-check that programs required for the build exist.
 foreach tool : ['cat', 'cut', 'grep', 'sort', 'tr', 'uname']
diff --git a/t/meson.build b/t/meson.build
index a59da26be3f..dba327fd7ec 100644
--- a/t/meson.build
+++ b/t/meson.build
@@ -1096,11 +1096,71 @@ integration_tests = [
   't9903-bash-prompt.sh',
 ]
 
+benchmarks = [
+  'perf/p0000-perf-lib-sanity.sh',
+  'perf/p0001-rev-list.sh',
+  'perf/p0002-read-cache.sh',
+  'perf/p0003-delta-base-cache.sh',
+  'perf/p0004-lazy-init-name-hash.sh',
+  'perf/p0005-status.sh',
+  'perf/p0006-read-tree-checkout.sh',
+  'perf/p0007-write-cache.sh',
+  'perf/p0008-odb-fsync.sh',
+  'perf/p0071-sort.sh',
+  'perf/p0090-cache-tree.sh',
+  'perf/p0100-globbing.sh',
+  'perf/p1006-cat-file.sh',
+  'perf/p1400-update-ref.sh',
+  'perf/p1450-fsck.sh',
+  'perf/p1451-fsck-skip-list.sh',
+  'perf/p1500-graph-walks.sh',
+  'perf/p2000-sparse-operations.sh',
+  'perf/p3400-rebase.sh',
+  'perf/p3404-rebase-interactive.sh',
+  'perf/p4000-diff-algorithms.sh',
+  'perf/p4001-diff-no-index.sh',
+  'perf/p4002-diff-color-moved.sh',
+  'perf/p4205-log-pretty-formats.sh',
+  'perf/p4209-pickaxe.sh',
+  'perf/p4211-line-log.sh',
+  'perf/p4220-log-grep-engines.sh',
+  'perf/p4221-log-grep-engines-fixed.sh',
+  'perf/p5302-pack-index.sh',
+  'perf/p5303-many-packs.sh',
+  'perf/p5304-prune.sh',
+  'perf/p5310-pack-bitmaps.sh',
+  'perf/p5311-pack-bitmaps-fetch.sh',
+  'perf/p5312-pack-bitmaps-revs.sh',
+  'perf/p5313-pack-objects.sh',
+  'perf/p5314-name-hash.sh',
+  'perf/p5326-multi-pack-bitmaps.sh',
+  'perf/p5332-multi-pack-reuse.sh',
+  'perf/p5333-pseudo-merge-bitmaps.sh',
+  'perf/p5550-fetch-tags.sh',
+  'perf/p5551-fetch-rescan.sh',
+  'perf/p5600-partial-clone.sh',
+  'perf/p5601-clone-reference.sh',
+  'perf/p6100-describe.sh',
+  'perf/p6300-for-each-ref.sh',
+  'perf/p7000-filter-branch.sh',
+  'perf/p7102-reset.sh',
+  'perf/p7300-clean.sh',
+  'perf/p7519-fsmonitor.sh',
+  'perf/p7527-builtin-fsmonitor.sh',
+  'perf/p7810-grep.sh',
+  'perf/p7820-grep-engines.sh',
+  'perf/p7821-grep-engines-fixed.sh',
+  'perf/p7822-grep-perl-character.sh',
+  'perf/p9210-scalar.sh',
+  'perf/p9300-fast-import-export.sh',
+]
+
 # Sanity check that we are not missing any tests present in 't/'. This check
 # only runs once at configure time and is thus best-effort, only. It is
 # sufficient to catch missing test suites in our CI though.
 foreach glob, tests : {
   't[0-9][0-9][0-9][0-9]-*.sh': integration_tests,
+  'perf/p[0-9][0-9][0-9][0-9]-*.sh': benchmarks,
   'unit-tests/t-*.c': unit_test_programs,
   'unit-tests/u-*.c': clar_test_suites,
 }
@@ -1152,3 +1212,20 @@ foreach integration_test : integration_tests
     timeout: 0,
   )
 endforeach
+
+if time.found()
+  benchmark_environment = test_environment
+  benchmark_environment.set('GTIME', time.full_path())
+
+  foreach benchmark : benchmarks
+    benchmark(fs.stem(benchmark), shell,
+      args: [
+        fs.name(benchmark),
+      ],
+      workdir: meson.current_source_dir() / 'perf',
+      env: benchmark_environment,
+      depends: test_dependencies + bin_wrappers,
+      timeout: 0,
+    )
+  endforeach
+endif

-- 
2.49.0.901.g37484f566f.dirty

[PATCH v3 5/5] meson: wire up benchmarking options

[Patrick Steinhardt]

Wire up a couple of benchmarking options that we end up writing into our
"GIT-BUILD-OPTIONS" file. These options allow users to control how
exactly benchmarks are executed.

Note that neither `GIT_PERF_MAKE_COMMAND` nor `GIT_PERF_MAKE_OPTS` are
exposed as a build option. Those options are used by "t/perf/run", which
is not used by Meson.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 meson.build       | 6 +++---
 meson_options.txt | 6 ++++++
 2 files changed, 9 insertions(+), 3 deletions(-)

diff --git a/meson.build b/meson.build
index 205b1bb301a..5bba743dc0e 100644
--- a/meson.build
+++ b/meson.build
@@ -666,11 +666,11 @@ builtin_sources += custom_target(
 # build options to our tests.
 build_options_config = configuration_data()
 build_options_config.set('GIT_INTEROP_MAKE_OPTS', '')
-build_options_config.set('GIT_PERF_LARGE_REPO', '')
+build_options_config.set_quoted('GIT_PERF_LARGE_REPO', get_option('benchmark_large_repo'))
 build_options_config.set('GIT_PERF_MAKE_COMMAND', '')
 build_options_config.set('GIT_PERF_MAKE_OPTS', '')
-build_options_config.set('GIT_PERF_REPEAT_COUNT', '')
-build_options_config.set('GIT_PERF_REPO', '')
+build_options_config.set_quoted('GIT_PERF_REPEAT_COUNT', get_option('benchmark_repeat_count').to_string())
+build_options_config.set_quoted('GIT_PERF_REPO', get_option('benchmark_repo'))
 build_options_config.set('GIT_TEST_CMP_USE_COPIED_CONTEXT', '')
 build_options_config.set('GIT_TEST_INDEX_VERSION', '')
 build_options_config.set('GIT_TEST_OPTS', '')
diff --git a/meson_options.txt b/meson_options.txt
index 78d172a7401..ca106c43b2e 100644
--- a/meson_options.txt
+++ b/meson_options.txt
@@ -101,6 +101,12 @@ option('docs_backend', type: 'combo', choices: ['asciidoc', 'asciidoctor', 'auto
   description: 'Which backend to use to generate documentation.')
 
 # Testing.
+option('benchmark_repo', type: 'string', value: '',
+  description: 'Repository to copy for the performance tests. Should be at least the size of the Git repository.')
+option('benchmark_large_repo', type: 'string', value: '',
+  description: 'Large repository to copy for the performance tests. Should be at least the size of the Linux repository.')
+option('benchmark_repeat_count', type: 'integer', value: 3,
+  description: 'Number of times a test should be repeated for best-of-N measurements.')
 option('coccinelle', type: 'feature', value: 'auto',
   description: 'Provide a coccicheck target that generates a Coccinelle patch.')
 option('tests', type: 'boolean', value: true,

-- 
2.49.0.901.g37484f566f.dirty

Re: [PATCH v3 0/5] meson: wire up support for benchmarks

[Christian Couder]

On Tue, Apr 22, 2025 at 8:50 AM Patrick Steinhardt <ps@pks.im> wrote:
>
> Hi,
>
> this small patch series implements support for running our benchmarks in
> "t/perf" via Meson. The series does not aim to replace "t/perf/run",
> which is more fully-featured and allows running benchmarks against
> multiple different trees. Instead, this series only allows running the
> benchmarks against the current tree. Users are thus expected to continue
> using "t/perf/run" for more advanced usecases.
>
> Changes in v2:
>   - Adapt "aggregate.perl" to use a "/usr/bin/env perl" shebang.
>   - Link to v1: https://lore.kernel.org/r/20250331-pks-meson-benchmarks-v1-0-b2ace85616a3@pks.im
>
> Changes in v3:
>   - Document how to run benchmarks in "meson.build".
>   - Expand the message for the commit that enables out-of-tree
>     benchmarking.
>   - Link to v2: https://lore.kernel.org/r/20250414-pks-meson-benchmarks-v2-0-04377080a167@pks.im

In https://lore.kernel.org/git/CAP8UFD2jKwYzmc40knXY7k+FQabjZbGTqs9fowF=-0OqfNYp_w@mail.gmail.com/
I wrote:

"I wonder what happens when we run `meson test -C build --benchmark`
but 'time' is not found."

because I wasn't sure if it would just do nothing in that case which
might not be very user friendly.

Otherwise it looks like my concerns have been addressed.

Thanks.

Re: [PATCH v3 0/5] meson: wire up support for benchmarks

[Patrick Steinhardt]

On Tue, Apr 22, 2025 at 09:27:57AM +0200, Christian Couder wrote:
> On Tue, Apr 22, 2025 at 8:50 AM Patrick Steinhardt <ps@pks.im> wrote:
> >
> > Hi,
> >
> > this small patch series implements support for running our benchmarks in
> > "t/perf" via Meson. The series does not aim to replace "t/perf/run",
> > which is more fully-featured and allows running benchmarks against
> > multiple different trees. Instead, this series only allows running the
> > benchmarks against the current tree. Users are thus expected to continue
> > using "t/perf/run" for more advanced usecases.
> >
> > Changes in v2:
> >   - Adapt "aggregate.perl" to use a "/usr/bin/env perl" shebang.
> >   - Link to v1: https://lore.kernel.org/r/20250331-pks-meson-benchmarks-v1-0-b2ace85616a3@pks.im
> >
> > Changes in v3:
> >   - Document how to run benchmarks in "meson.build".
> >   - Expand the message for the commit that enables out-of-tree
> >     benchmarking.
> >   - Link to v2: https://lore.kernel.org/r/20250414-pks-meson-benchmarks-v2-0-04377080a167@pks.im
> 
> In https://lore.kernel.org/git/CAP8UFD2jKwYzmc40knXY7k+FQabjZbGTqs9fowF=-0OqfNYp_w@mail.gmail.com/
> I wrote:
> 
> "I wonder what happens when we run `meson test -C build --benchmark`
> but 'time' is not found."
> 
> because I wasn't sure if it would just do nothing in that case which
> might not be very user friendly.

Ah, sorry, forgot to answer that question. What Meson does in that case
is to print "No tests defined". We could help improve usability a bit by
printing benchmarks as part of the auto-detected features after setup of
the build directory has finished. That would make it more discoverable
that benchmarks have been disabled.

Patrick

Re: [PATCH v3 0/5] meson: wire up support for benchmarks

[Christian Couder]

On Tue, Apr 22, 2025 at 9:53 AM Patrick Steinhardt <ps@pks.im> wrote:
>
> On Tue, Apr 22, 2025 at 09:27:57AM +0200, Christian Couder wrote:

> > In https://lore.kernel.org/git/CAP8UFD2jKwYzmc40knXY7k+FQabjZbGTqs9fowF=-0OqfNYp_w@mail.gmail.com/
> > I wrote:
> >
> > "I wonder what happens when we run `meson test -C build --benchmark`
> > but 'time' is not found."
> >
> > because I wasn't sure if it would just do nothing in that case which
> > might not be very user friendly.
>
> Ah, sorry, forgot to answer that question. What Meson does in that case
> is to print "No tests defined". We could help improve usability a bit by
> printing benchmarks as part of the auto-detected features after setup of
> the build directory has finished. That would make it more discoverable
> that benchmarks have been disabled.

I was more expecting something simple like:

if time.found()
  ...
else
  error('Benchmarking requires the `time` command')
endif

in the same way as in meson.build elsewhere we have things like:

if not msgfmt.found() and gettext_option.enabled()
  error('Internationalization via libintl requires msgfmt')
endif

Re: [PATCH v3 0/5] meson: wire up support for benchmarks

[Patrick Steinhardt]

On Wed, Apr 23, 2025 at 04:44:20PM +0200, Christian Couder wrote:
> On Tue, Apr 22, 2025 at 9:53 AM Patrick Steinhardt <ps@pks.im> wrote:
> >
> > On Tue, Apr 22, 2025 at 09:27:57AM +0200, Christian Couder wrote:
> 
> > > In https://lore.kernel.org/git/CAP8UFD2jKwYzmc40knXY7k+FQabjZbGTqs9fowF=-0OqfNYp_w@mail.gmail.com/
> > > I wrote:
> > >
> > > "I wonder what happens when we run `meson test -C build --benchmark`
> > > but 'time' is not found."
> > >
> > > because I wasn't sure if it would just do nothing in that case which
> > > might not be very user friendly.
> >
> > Ah, sorry, forgot to answer that question. What Meson does in that case
> > is to print "No tests defined". We could help improve usability a bit by
> > printing benchmarks as part of the auto-detected features after setup of
> > the build directory has finished. That would make it more discoverable
> > that benchmarks have been disabled.
> 
> I was more expecting something simple like:
> 
> if time.found()
>   ...
> else
>   error('Benchmarking requires the `time` command')
> endif
> 
> in the same way as in meson.build elsewhere we have things like:
> 
> if not msgfmt.found() and gettext_option.enabled()
>   error('Internationalization via libintl requires msgfmt')
> endif

But erroring out by default doesn't really feel nice to the general
developer. I'd claim that 99% of the time, developers will only end up
running the test suite, never the benchmarks. So the default should
match that and not require GNU time to be available, if you ask me.

An alternative could be to have a 'benchmarks' feature option wired up.
This feature would be set to 'auto', and then we can detect time like
this:

    time = find_program('time', dirs: program_path, required: get_option('benchmarks'))

This will then auto-enable or -disable the feature depending on the
availability of GNU time. But if the user passes `-Dbenchmarks=enabled`,
we would fail the setup in case the program wasn't found.

Patrick

Re: [PATCH v3 0/5] meson: wire up support for benchmarks

[Christian Couder]

On Thu, Apr 24, 2025 at 6:31 AM Patrick Steinhardt <ps@pks.im> wrote:
>
> On Wed, Apr 23, 2025 at 04:44:20PM +0200, Christian Couder wrote:
> > On Tue, Apr 22, 2025 at 9:53 AM Patrick Steinhardt <ps@pks.im> wrote:
> > >
> > > On Tue, Apr 22, 2025 at 09:27:57AM +0200, Christian Couder wrote:

> > if time.found()
> >   ...
> > else
> >   error('Benchmarking requires the `time` command')
> > endif
> >
> > in the same way as in meson.build elsewhere we have things like:
> >
> > if not msgfmt.found() and gettext_option.enabled()
> >   error('Internationalization via libintl requires msgfmt')
> > endif
>
> But erroring out by default doesn't really feel nice to the general
> developer. I'd claim that 99% of the time, developers will only end up
> running the test suite, never the benchmarks. So the default should
> match that and not require GNU time to be available, if you ask me.

Yeah, right.

> An alternative could be to have a 'benchmarks' feature option wired up.
> This feature would be set to 'auto', and then we can detect time like
> this:
>
>     time = find_program('time', dirs: program_path, required: get_option('benchmarks'))
>
> This will then auto-enable or -disable the feature depending on the
> availability of GNU time. But if the user passes `-Dbenchmarks=enabled`,
> we would fail the setup in case the program wasn't found.

Yeah, that would be great.

An alternative might be a placeholder benchmark that just reports an
error when run, like:

if time.found()
   ...
else
  benchmark('perf-benchmarks-require-time', shell, args: [
    '-c',
    'echo "ERROR: Benchmarks require the \"time\" command to be
available. Please install it and reconfigure." >&2; exit 1'
  ], env: test_environment, )
endif

Re: [PATCH v3 0/5] meson: wire up support for benchmarks

[Junio C Hamano]

Patrick Steinhardt <ps@pks.im> writes:

>> I was more expecting something simple like:
>> 
>> if time.found()
>>   ...
>> else
>>   error('Benchmarking requires the `time` command')
>> endif
>> 
>> in the same way as in meson.build elsewhere we have things like:
>> 
>> if not msgfmt.found() and gettext_option.enabled()
>>   error('Internationalization via libintl requires msgfmt')
>> endif
>
> But erroring out by default doesn't really feel nice to the general
> developer. I'd claim that 99% of the time, developers will only end up
> running the test suite, never the benchmarks. So the default should
> match that and not require GNU time to be available, if you ask me.

Maybe a stupid question, but does "time" a shell built-in count when
"if time.found()" is evaluated?

    $ type --all time
    time is a shell keyword
    time is /usr/bin/time
    time is /bin/time

> An alternative could be to have a 'benchmarks' feature option wired up.
> This feature would be set to 'auto', and then we can detect time like
> this:
>
>     time = find_program('time', dirs: program_path, required: get_option('benchmarks'))
>
> This will then auto-enable or -disable the feature depending on the
> availability of GNU time. But if the user passes `-Dbenchmarks=enabled`,
> we would fail the setup in case the program wasn't found.

Sounds quite sensible.  Do what most users expect by default without
failing, let users give explicit preference, and when the explicitly
expressed preference cannot be fulfilled, loudly error out.  I like
it.

Thanks.

Re: [PATCH v3 0/5] meson: wire up support for benchmarks

[Patrick Steinhardt]

On Thu, Apr 24, 2025 at 04:13:15AM -0700, Junio C Hamano wrote:
> Patrick Steinhardt <ps@pks.im> writes:
> 
> >> I was more expecting something simple like:
> >> 
> >> if time.found()
> >>   ...
> >> else
> >>   error('Benchmarking requires the `time` command')
> >> endif
> >> 
> >> in the same way as in meson.build elsewhere we have things like:
> >> 
> >> if not msgfmt.found() and gettext_option.enabled()
> >>   error('Internationalization via libintl requires msgfmt')
> >> endif
> >
> > But erroring out by default doesn't really feel nice to the general
> > developer. I'd claim that 99% of the time, developers will only end up
> > running the test suite, never the benchmarks. So the default should
> > match that and not require GNU time to be available, if you ask me.
> 
> Maybe a stupid question, but does "time" a shell built-in count when
> "if time.found()" is evaluated?
> 
>     $ type --all time
>     time is a shell keyword
>     time is /usr/bin/time
>     time is /bin/time

No, it doesn't, and that is intentional. Our benchmarks explicitly
require GNU time because it surfaces more information than the shell
builtin.

We even have a `USR_BIN_TIME` prereq in our test lib, which is
unfortunately way too limiting for some platforms like NixOS. Maybe I
should use that as an excuse to also refactor this prereq while at it to
instead use a path-based lookup of time. On the other hand it's only
used by a single test anyway, so it doesn't really feel worth it.

Patrick

Re: [PATCH v3 0/5] meson: wire up support for benchmarks

[Toon Claes]

Patrick Steinhardt <ps@pks.im> writes:

> Hi,
>
> this small patch series implements support for running our benchmarks in
> "t/perf" via Meson. The series does not aim to replace "t/perf/run",
> which is more fully-featured and allows running benchmarks against
> multiple different trees. Instead, this series only allows running the
> benchmarks against the current tree. Users are thus expected to continue
> using "t/perf/run" for more advanced usecases.
>
> Changes in v2:
>   - Adapt "aggregate.perl" to use a "/usr/bin/env perl" shebang.
>   - Link to v1: https://lore.kernel.org/r/20250331-pks-meson-benchmarks-v1-0-b2ace85616a3@pks.im
>
> Changes in v3:
>   - Document how to run benchmarks in "meson.build".

Thanks for adding this.

I have no further comments. I approve.


--
Toon

[PATCH v4 0/5] meson: wire up support for benchmarks

[Patrick Steinhardt]

Hi,

this small patch series implements support for running our benchmarks in
"t/perf" via Meson. The series does not aim to replace "t/perf/run",
which is more fully-featured and allows running benchmarks against
multiple different trees. Instead, this series only allows running the
benchmarks against the current tree. Users are thus expected to continue
using "t/perf/run" for more advanced usecases.

Changes in v2:
  - Adapt "aggregate.perl" to use a "/usr/bin/env perl" shebang.
  - Link to v1: https://lore.kernel.org/r/20250331-pks-meson-benchmarks-v1-0-b2ace85616a3@pks.im

Changes in v3:
  - Document how to run benchmarks in "meson.build".
  - Expand the message for the commit that enables out-of-tree
    benchmarking.
  - Link to v2: https://lore.kernel.org/r/20250414-pks-meson-benchmarks-v2-0-04377080a167@pks.im

Changes in v4:
  - The patch series was rebased on top of f65182a99e5 (The ninth batch,
    2025-04-24). This is due to a conflict with ps/test-wo-perl-prereq.
  - Introduce a 'benchmarks' option. This allows developers to require
    benchmarks as desired. By default, we auto-detect whether the host
    system has all dependencies available and enable or disable them
    accordingly.
  - Report whether or not benchmarks are enabled via `summary()`.
  - Our benchmarks depend on Perl, so add this dependency accordingly.
  - Link to v3: https://lore.kernel.org/r/20250422-pks-meson-benchmarks-v3-0-7aad68bac6fd@pks.im

Thanks!

Patrick

---
Patrick Steinhardt (5):
      t/perf: fix benchmarks with alternate repo formats
      t/perf: use configured PERL_PATH
      t/perf: fix benchmarks with out-of-tree builds
      meson: wire up benchmarks
      meson: wire up benchmarking options

 meson.build           | 16 ++++++++---
 meson_options.txt     |  8 ++++++
 t/meson.build         | 77 +++++++++++++++++++++++++++++++++++++++++++++++++++
 t/perf/aggregate.perl |  2 +-
 t/perf/perf-lib.sh    | 32 ++++++++++++++++++---
 t/perf/run            |  4 +--
 6 files changed, 128 insertions(+), 11 deletions(-)

Range-diff versus v3:

1:  2375a16c2c0 = 1:  174804805c2 t/perf: fix benchmarks with alternate repo formats
2:  69782035246 = 2:  41faa3d9b3b t/perf: use configured PERL_PATH
3:  cca40e9dd0e = 3:  87ea3ab1a45 t/perf: fix benchmarks with out-of-tree builds
4:  50e9b507db4 ! 4:  723a0a2e449 meson: wire up benchmarks
    @@ meson.build
      #   Test execution is parallelized by default and scales with the number of
      #   processor cores available. You can change the number of processes by passing
      #   the `-jN` flag to `meson test`.
    -@@ meson.build: git = find_program('git', dirs: program_path, required: false)
    - sed = find_program('sed', dirs: program_path)
    - shell = find_program('sh', dirs: program_path)
    - tar = find_program('tar', dirs: program_path)
    -+time = find_program('time', dirs: program_path, required: false)
    +@@ meson.build: git = find_program('git', dirs: program_path, native: true, required: false)
    + sed = find_program('sed', dirs: program_path, native: true)
    + shell = find_program('sh', dirs: program_path, native: true)
    + tar = find_program('tar', dirs: program_path, native: true)
    ++time = find_program('time', dirs: program_path, required: get_option('benchmarks'))
      
    - # Sanity-check that programs required for the build exist.
    - foreach tool : ['cat', 'cut', 'grep', 'sort', 'tr', 'uname']
    + target_shell = find_program('sh', dirs: program_path, native: false)
    + 
    +@@ meson.build: endif
    + # features. It is optional if you want to neither execute tests nor use any of
    + # these optional features.
    + perl_required = get_option('perl')
    +-if get_option('gitweb').enabled() or 'netrc' in get_option('credential_helpers') or get_option('docs') != []
    ++if get_option('benchmarks').enabled() or get_option('gitweb').enabled() or 'netrc' in get_option('credential_helpers') or get_option('docs') != []
    +   perl_required = true
    + endif
    + 
    +@@ meson.build: meson.add_dist_script(
    + )
    + 
    + summary({
    ++  'benchmarks': get_option('tests') and perl.found() and time.found(),
    +   'curl': curl.found(),
    +   'expat': expat.found(),
    +   'gettext': intl.found(),
    +
    + ## meson_options.txt ##
    +@@ meson_options.txt: option('docs_backend', type: 'combo', choices: ['asciidoc', 'asciidoctor', 'auto
    +   description: 'Which backend to use to generate documentation.')
    + 
    + # Testing.
    ++option('benchmarks', type: 'feature', value: 'auto',
    ++  description: 'Enable benchmarks. This requires Perl and GNU time.')
    + option('coccinelle', type: 'feature', value: 'auto',
    +   description: 'Provide a coccicheck target that generates a Coccinelle patch.')
    + option('tests', type: 'boolean', value: true,
     
      ## t/meson.build ##
     @@ t/meson.build: integration_tests = [
    @@ t/meson.build: foreach integration_test : integration_tests
        )
      endforeach
     +
    -+if time.found()
    ++if perl.found() and time.found()
     +  benchmark_environment = test_environment
     +  benchmark_environment.set('GTIME', time.full_path())
     +
5:  52cca4b17ae ! 5:  16c0230656f meson: wire up benchmarking options
    @@ meson.build: builtin_sources += custom_target(
     
      ## meson_options.txt ##
     @@ meson_options.txt: option('docs_backend', type: 'combo', choices: ['asciidoc', 'asciidoctor', 'auto
    -   description: 'Which backend to use to generate documentation.')
    - 
      # Testing.
    + option('benchmarks', type: 'feature', value: 'auto',
    +   description: 'Enable benchmarks. This requires Perl and GNU time.')
     +option('benchmark_repo', type: 'string', value: '',
     +  description: 'Repository to copy for the performance tests. Should be at least the size of the Git repository.')
     +option('benchmark_large_repo', type: 'string', value: '',

---
base-commit: f65182a99e545d2f2bc22e6c1c2da192133b16a3
change-id: 20250328-pks-meson-benchmarks-a8fac5f69467

[PATCH v4 1/5] t/perf: fix benchmarks with alternate repo formats

[Patrick Steinhardt]

Many of our benchmarks operate on a user-defined repository that we copy
over before running the benchmarked logic. To keep unintentional side
effects caused by on-disk state at bay we skip copying some files. This
includes for example hooks, but also the repo's configuration.

It is quite sensible to not copy over the configuration, as it is quite
easy to inadvertently carry over configuration that may significantly
impact the performance measurements. But we cannot fully ignore the
configuration either, as it may contain information about the repository
format. This will cause failures when for example using a repository
with SHA256 object format or the reftable ref format.

Fix the issue by parsing the reference and object formats from the
source repository and passing them to git-init(1).

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 t/perf/perf-lib.sh | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/t/perf/perf-lib.sh b/t/perf/perf-lib.sh
index 8ab6d9c4694..1a9a51ca3cc 100644
--- a/t/perf/perf-lib.sh
+++ b/t/perf/perf-lib.sh
@@ -98,6 +98,8 @@ test_perf_create_repo_from () {
 	source_git="$("$MODERN_GIT" -C "$source" rev-parse --git-dir)"
 	objects_dir="$("$MODERN_GIT" -C "$source" rev-parse --git-path objects)"
 	common_dir="$("$MODERN_GIT" -C "$source" rev-parse --git-common-dir)"
+	refformat="$("$MODERN_GIT" -C "$source" rev-parse --show-ref-format)"
+	objectformat="$("$MODERN_GIT" -C "$source" rev-parse --show-object-format)"
 	mkdir -p "$repo/.git"
 	(
 		cd "$source" &&
@@ -114,7 +116,7 @@ test_perf_create_repo_from () {
 	) &&
 	(
 		cd "$repo" &&
-		"$MODERN_GIT" init -q &&
+		"$MODERN_GIT" init -q --ref-format="$refformat" --object-format="$objectformat" &&
 		test_perf_do_repo_symlink_config_ &&
 		mv .git/hooks .git/hooks-disabled 2>/dev/null &&
 		if test -f .git/index.lock

-- 
2.49.0.901.g37484f566f.dirty

[PATCH v4 2/5] t/perf: use configured PERL_PATH

[Patrick Steinhardt]

Our benchmarks use a couple of Perl scripts to compute results. These
Perl scripts get executed directly, and as the shebang is hardcoded to
"/usr/bin/perl" this will fail on any system where the Perl interpreter
is located in a different path.

Our build infrastructure already lets users configure the location of
Perl, which ultimately gets written into the GIT-BUILD-OPTIONS file.
This file is being sourced by "test-lib.sh", and consequently we already
have the "PERL_PATH" variable available that contains its configured
location.

Use "PERL_PATH" to execute Perl scripts, which makes them work on more
esoteric systems like NixOS. Furthermore, adapt the shebang to use
env(1) to execute Perl so that users who have Perl in PATH, but in a
non-standard location can execute the script directly.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 t/perf/aggregate.perl | 2 +-
 t/perf/perf-lib.sh    | 4 ++--
 t/perf/run            | 4 ++--
 3 files changed, 5 insertions(+), 5 deletions(-)

diff --git a/t/perf/aggregate.perl b/t/perf/aggregate.perl
index 575d2000cc1..1791c7528a9 100755
--- a/t/perf/aggregate.perl
+++ b/t/perf/aggregate.perl
@@ -1,4 +1,4 @@
-#!/usr/bin/perl
+#!/usr/bin/env perl
 
 use lib '../../perl/build/lib';
 use strict;
diff --git a/t/perf/perf-lib.sh b/t/perf/perf-lib.sh
index 1a9a51ca3cc..4173eee4def 100644
--- a/t/perf/perf-lib.sh
+++ b/t/perf/perf-lib.sh
@@ -276,7 +276,7 @@ test_perf_ () {
 	else
 		test_ok_ "$1"
 	fi
-	"$TEST_DIRECTORY"/perf/min_time.perl test_time.* >"$base".result
+	"$PERL_PATH" "$TEST_DIRECTORY"/perf/min_time.perl test_time.* >"$base".result
 	rm test_time.*
 }
 
@@ -324,7 +324,7 @@ test_at_end_hook_ () {
 	if test -z "$GIT_PERF_AGGREGATING_LATER"; then
 		(
 			cd "$TEST_DIRECTORY"/perf &&
-			./aggregate.perl --results-dir="$TEST_RESULTS_DIR" $(basename "$0")
+			"$PERL_PATH" ./aggregate.perl --results-dir="$TEST_RESULTS_DIR" $(basename "$0")
 		)
 	fi
 }
diff --git a/t/perf/run b/t/perf/run
index 486ead21980..073bcb2afff 100755
--- a/t/perf/run
+++ b/t/perf/run
@@ -192,10 +192,10 @@ run_subsection () {
 
 	if test -z "$GIT_PERF_SEND_TO_CODESPEED"
 	then
-		./aggregate.perl --results-dir="$TEST_RESULTS_DIR" $codespeed_opt "$@"
+		"$PERL_PATH" ./aggregate.perl --results-dir="$TEST_RESULTS_DIR" $codespeed_opt "$@"
 	else
 		json_res_file=""$TEST_RESULTS_DIR"/$GIT_PERF_SUBSECTION/aggregate.json"
-		./aggregate.perl --results-dir="$TEST_RESULTS_DIR" --codespeed "$@" | tee "$json_res_file"
+		"$PERL_PATH" ./aggregate.perl --results-dir="$TEST_RESULTS_DIR" --codespeed "$@" | tee "$json_res_file"
 		send_data_url="$GIT_PERF_SEND_TO_CODESPEED/result/add/json/"
 		curl -v --request POST --data-urlencode "json=$(cat "$json_res_file")" "$send_data_url"
 	fi

-- 
2.49.0.901.g37484f566f.dirty

[PATCH v4 3/5] t/perf: fix benchmarks with out-of-tree builds

[Patrick Steinhardt]

The "perf-lib.sh" script is sourced by all of our benchmarking suites to
make available common infrastructure. The script assumes that build and
source directory are the same, which works for our Makefile. But the
assumption breaks with both CMake and Meson, where the build directory
can be located in an arbitrary place.

Adapt the script so that it works with out-of-tree builds. Most
importantly, this requires us to figure out the location of the build
directory:

  - When running benchmarks via our Makefile the build directory is the
    same as the source directory. We already know to derive the test
    directory ("t/") via `$(pwd)/..`, which works because we chdir into
    "t/perf" before executing benchmarks. We can thus derive the build
    directory by appending another "/.." to that path.

  - When running benchmarks via Meson the build directory is located at
    an arbitrary location. The build system thus has to make the path
    known by exporting the `GIT_BUILD_DIR` environment variable.

This change prepares us for wiring up benchmarks in Meson.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 t/perf/perf-lib.sh | 26 ++++++++++++++++++++++++--
 1 file changed, 24 insertions(+), 2 deletions(-)

diff --git a/t/perf/perf-lib.sh b/t/perf/perf-lib.sh
index 4173eee4def..5406557b7ca 100644
--- a/t/perf/perf-lib.sh
+++ b/t/perf/perf-lib.sh
@@ -25,7 +25,29 @@ TEST_OUTPUT_DIRECTORY=$(pwd)
 TEST_NO_CREATE_REPO=t
 TEST_NO_MALLOC_CHECK=t
 
-. ../test-lib.sh
+# While test-lib.sh computes the build directory for us, we also have to do the
+# same thing in order to locate the script via GIT-BUILD-OPTIONS in the first
+# place.
+GIT_BUILD_DIR="${GIT_BUILD_DIR:-$TEST_DIRECTORY/..}"
+if test -f "$GIT_BUILD_DIR/GIT-BUILD-DIR"
+then
+	GIT_BUILD_DIR="$(cat "$GIT_BUILD_DIR/GIT-BUILD-DIR")" || exit 1
+	# On Windows, we must convert Windows paths lest they contain a colon
+	case "$(uname -s)" in
+	*MINGW*)
+		GIT_BUILD_DIR="$(cygpath -au "$GIT_BUILD_DIR")"
+		;;
+	esac
+fi
+
+if test ! -f "$GIT_BUILD_DIR"/GIT-BUILD-OPTIONS
+then
+	echo >&2 'error: GIT-BUILD-OPTIONS missing (has Git been built?).'
+	exit 1
+fi
+
+. "$GIT_BUILD_DIR"/GIT-BUILD-OPTIONS
+. "$GIT_SOURCE_DIR"/t/test-lib.sh
 
 unset GIT_CONFIG_NOSYSTEM
 GIT_CONFIG_SYSTEM="$TEST_DIRECTORY/perf/config"
@@ -324,7 +346,7 @@ test_at_end_hook_ () {
 	if test -z "$GIT_PERF_AGGREGATING_LATER"; then
 		(
 			cd "$TEST_DIRECTORY"/perf &&
-			"$PERL_PATH" ./aggregate.perl --results-dir="$TEST_RESULTS_DIR" $(basename "$0")
+			"$PERL_PATH" "$GIT_SOURCE_DIR"/t/perf/aggregate.perl --results-dir="$TEST_RESULTS_DIR" $(basename "$0")
 		)
 	fi
 }

-- 
2.49.0.901.g37484f566f.dirty

[PATCH v4 4/5] meson: wire up benchmarks

[Patrick Steinhardt]

Wire up benchmarks in Meson. The setup is mostly the same as how we wire
up our tests. The only difference is that benchmarks get wired up via
the `benchmark()` option instead of via `test()`, which gives them a bit
of special treatment:

  - Benchmarks never run in parallel.

  - Benchmarks aren't run by default when tests are executed.

  - Meson does not inject the `MALLOC_PERTURB` environment variable.

Using benchmarks is quite simple:

    ```
    $ meson setup build
    # Run all benchmarks.
    $ meson test -C build --benchmark
    # Run a specific benchmark.
    $ meson test -C build --benchmark p0000-*
    ```

Other than that the usual command line arguments accepted when running
tests are also accepted when running benchmarks.

Note that the benchmarking target is somewhat limited because it will
only run benchmarks for the current build. Other use cases, like running
benchmarks against multiple different versions of Git, are not currently
supported. Users should continue to use "t/perf/run" for those use
cases. The script should get extended at one point in time to support
Meson, but this is outside of the scope of this series.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 meson.build       | 10 +++++++-
 meson_options.txt |  2 ++
 t/meson.build     | 77 +++++++++++++++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 88 insertions(+), 1 deletion(-)

diff --git a/meson.build b/meson.build
index 410bbf93dad..2cd05a9b1ba 100644
--- a/meson.build
+++ b/meson.build
@@ -70,6 +70,12 @@
 #      # Execute single test interactively such that features like `debug ()` work.
 #      $ meson test -i --test-args='-ix' t1400-update-ref
 #
+#      # Execute all benchmarks.
+#      $ meson test -i --benchmark
+#
+#      # Execute single benchmark.
+#      $ meson test -i --benchmark p0000-*
+#
 #   Test execution is parallelized by default and scales with the number of
 #   processor cores available. You can change the number of processes by passing
 #   the `-jN` flag to `meson test`.
@@ -235,6 +241,7 @@ git = find_program('git', dirs: program_path, native: true, required: false)
 sed = find_program('sed', dirs: program_path, native: true)
 shell = find_program('sh', dirs: program_path, native: true)
 tar = find_program('tar', dirs: program_path, native: true)
+time = find_program('time', dirs: program_path, required: get_option('benchmarks'))
 
 target_shell = find_program('sh', dirs: program_path, native: false)
 
@@ -836,7 +843,7 @@ endif
 # features. It is optional if you want to neither execute tests nor use any of
 # these optional features.
 perl_required = get_option('perl')
-if get_option('gitweb').enabled() or 'netrc' in get_option('credential_helpers') or get_option('docs') != []
+if get_option('benchmarks').enabled() or get_option('gitweb').enabled() or 'netrc' in get_option('credential_helpers') or get_option('docs') != []
   perl_required = true
 endif
 
@@ -2082,6 +2089,7 @@ meson.add_dist_script(
 )
 
 summary({
+  'benchmarks': get_option('tests') and perl.found() and time.found(),
   'curl': curl.found(),
   'expat': expat.found(),
   'gettext': intl.found(),
diff --git a/meson_options.txt b/meson_options.txt
index 8ac30a52231..7f5bca5c029 100644
--- a/meson_options.txt
+++ b/meson_options.txt
@@ -101,6 +101,8 @@ option('docs_backend', type: 'combo', choices: ['asciidoc', 'asciidoctor', 'auto
   description: 'Which backend to use to generate documentation.')
 
 # Testing.
+option('benchmarks', type: 'feature', value: 'auto',
+  description: 'Enable benchmarks. This requires Perl and GNU time.')
 option('coccinelle', type: 'feature', value: 'auto',
   description: 'Provide a coccicheck target that generates a Coccinelle patch.')
 option('tests', type: 'boolean', value: true,
diff --git a/t/meson.build b/t/meson.build
index bfb744e8863..59438b06266 100644
--- a/t/meson.build
+++ b/t/meson.build
@@ -1097,11 +1097,71 @@ integration_tests = [
   't9903-bash-prompt.sh',
 ]
 
+benchmarks = [
+  'perf/p0000-perf-lib-sanity.sh',
+  'perf/p0001-rev-list.sh',
+  'perf/p0002-read-cache.sh',
+  'perf/p0003-delta-base-cache.sh',
+  'perf/p0004-lazy-init-name-hash.sh',
+  'perf/p0005-status.sh',
+  'perf/p0006-read-tree-checkout.sh',
+  'perf/p0007-write-cache.sh',
+  'perf/p0008-odb-fsync.sh',
+  'perf/p0071-sort.sh',
+  'perf/p0090-cache-tree.sh',
+  'perf/p0100-globbing.sh',
+  'perf/p1006-cat-file.sh',
+  'perf/p1400-update-ref.sh',
+  'perf/p1450-fsck.sh',
+  'perf/p1451-fsck-skip-list.sh',
+  'perf/p1500-graph-walks.sh',
+  'perf/p2000-sparse-operations.sh',
+  'perf/p3400-rebase.sh',
+  'perf/p3404-rebase-interactive.sh',
+  'perf/p4000-diff-algorithms.sh',
+  'perf/p4001-diff-no-index.sh',
+  'perf/p4002-diff-color-moved.sh',
+  'perf/p4205-log-pretty-formats.sh',
+  'perf/p4209-pickaxe.sh',
+  'perf/p4211-line-log.sh',
+  'perf/p4220-log-grep-engines.sh',
+  'perf/p4221-log-grep-engines-fixed.sh',
+  'perf/p5302-pack-index.sh',
+  'perf/p5303-many-packs.sh',
+  'perf/p5304-prune.sh',
+  'perf/p5310-pack-bitmaps.sh',
+  'perf/p5311-pack-bitmaps-fetch.sh',
+  'perf/p5312-pack-bitmaps-revs.sh',
+  'perf/p5313-pack-objects.sh',
+  'perf/p5314-name-hash.sh',
+  'perf/p5326-multi-pack-bitmaps.sh',
+  'perf/p5332-multi-pack-reuse.sh',
+  'perf/p5333-pseudo-merge-bitmaps.sh',
+  'perf/p5550-fetch-tags.sh',
+  'perf/p5551-fetch-rescan.sh',
+  'perf/p5600-partial-clone.sh',
+  'perf/p5601-clone-reference.sh',
+  'perf/p6100-describe.sh',
+  'perf/p6300-for-each-ref.sh',
+  'perf/p7000-filter-branch.sh',
+  'perf/p7102-reset.sh',
+  'perf/p7300-clean.sh',
+  'perf/p7519-fsmonitor.sh',
+  'perf/p7527-builtin-fsmonitor.sh',
+  'perf/p7810-grep.sh',
+  'perf/p7820-grep-engines.sh',
+  'perf/p7821-grep-engines-fixed.sh',
+  'perf/p7822-grep-perl-character.sh',
+  'perf/p9210-scalar.sh',
+  'perf/p9300-fast-import-export.sh',
+]
+
 # Sanity check that we are not missing any tests present in 't/'. This check
 # only runs once at configure time and is thus best-effort, only. It is
 # sufficient to catch missing test suites in our CI though.
 foreach glob, tests : {
   't[0-9][0-9][0-9][0-9]-*.sh': integration_tests,
+  'perf/p[0-9][0-9][0-9][0-9]-*.sh': benchmarks,
   'unit-tests/t-*.c': unit_test_programs,
   'unit-tests/u-*.c': clar_test_suites,
 }
@@ -1153,3 +1213,20 @@ foreach integration_test : integration_tests
     timeout: 0,
   )
 endforeach
+
+if perl.found() and time.found()
+  benchmark_environment = test_environment
+  benchmark_environment.set('GTIME', time.full_path())
+
+  foreach benchmark : benchmarks
+    benchmark(fs.stem(benchmark), shell,
+      args: [
+        fs.name(benchmark),
+      ],
+      workdir: meson.current_source_dir() / 'perf',
+      env: benchmark_environment,
+      depends: test_dependencies + bin_wrappers,
+      timeout: 0,
+    )
+  endforeach
+endif

-- 
2.49.0.901.g37484f566f.dirty

Re: [PATCH v4 4/5] meson: wire up benchmarks

[Christian Couder]

On Fri, Apr 25, 2025 at 9:28 AM Patrick Steinhardt <ps@pks.im> wrote:
>
> Wire up benchmarks in Meson. The setup is mostly the same as how we wire
> up our tests. The only difference is that benchmarks get wired up via
> the `benchmark()` option instead of via `test()`, which gives them a bit
> of special treatment:
>
>   - Benchmarks never run in parallel.
>
>   - Benchmarks aren't run by default when tests are executed.
>
>   - Meson does not inject the `MALLOC_PERTURB` environment variable.

[...]

> diff --git a/meson.build b/meson.build
> index 410bbf93dad..2cd05a9b1ba 100644
> --- a/meson.build
> +++ b/meson.build
> @@ -70,6 +70,12 @@
>  #      # Execute single test interactively such that features like `debug ()` work.
>  #      $ meson test -i --test-args='-ix' t1400-update-ref
>  #
> +#      # Execute all benchmarks.
> +#      $ meson test -i --benchmark
> +#
> +#      # Execute single benchmark.
> +#      $ meson test -i --benchmark p0000-*
> +#
>  #   Test execution is parallelized by default and scales with the number of
>  #   processor cores available. You can change the number of processes by passing
>  #   the `-jN` flag to `meson test`.

Doesn't the above give the impression that benchmarks are also
parallelized by default while the commit message says they are not?

Re: [PATCH v4 4/5] meson: wire up benchmarks

[Patrick Steinhardt]

On Fri, Apr 25, 2025 at 09:57:33AM +0200, Christian Couder wrote:
> On Fri, Apr 25, 2025 at 9:28 AM Patrick Steinhardt <ps@pks.im> wrote:
> >
> > Wire up benchmarks in Meson. The setup is mostly the same as how we wire
> > up our tests. The only difference is that benchmarks get wired up via
> > the `benchmark()` option instead of via `test()`, which gives them a bit
> > of special treatment:
> >
> >   - Benchmarks never run in parallel.
> >
> >   - Benchmarks aren't run by default when tests are executed.
> >
> >   - Meson does not inject the `MALLOC_PERTURB` environment variable.
> 
> [...]
> 
> > diff --git a/meson.build b/meson.build
> > index 410bbf93dad..2cd05a9b1ba 100644
> > --- a/meson.build
> > +++ b/meson.build
> > @@ -70,6 +70,12 @@
> >  #      # Execute single test interactively such that features like `debug ()` work.
> >  #      $ meson test -i --test-args='-ix' t1400-update-ref
> >  #
> > +#      # Execute all benchmarks.
> > +#      $ meson test -i --benchmark
> > +#
> > +#      # Execute single benchmark.
> > +#      $ meson test -i --benchmark p0000-*
> > +#
> >  #   Test execution is parallelized by default and scales with the number of
> >  #   processor cores available. You can change the number of processes by passing
> >  #   the `-jN` flag to `meson test`.
> 
> Doesn't the above give the impression that benchmarks are also
> parallelized by default while the commit message says they are not?

To me it doesn't because it talks about test execution, not benchmark
execution. The comments for each of the invocations also discern between
tests and benchmarks. I'm happy to hear about alternative ways to word
this though.

Patrick

Re: [PATCH v4 4/5] meson: wire up benchmarks

[Christian Couder]

On Fri, Apr 25, 2025 at 10:01 AM Patrick Steinhardt <ps@pks.im> wrote:
>
> On Fri, Apr 25, 2025 at 09:57:33AM +0200, Christian Couder wrote:
> > On Fri, Apr 25, 2025 at 9:28 AM Patrick Steinhardt <ps@pks.im> wrote:
> > >
> > > Wire up benchmarks in Meson. The setup is mostly the same as how we wire
> > > up our tests. The only difference is that benchmarks get wired up via
> > > the `benchmark()` option instead of via `test()`, which gives them a bit
> > > of special treatment:
> > >
> > >   - Benchmarks never run in parallel.
> > >
> > >   - Benchmarks aren't run by default when tests are executed.
> > >
> > >   - Meson does not inject the `MALLOC_PERTURB` environment variable.
> >
> > [...]
> >
> > > diff --git a/meson.build b/meson.build
> > > index 410bbf93dad..2cd05a9b1ba 100644
> > > --- a/meson.build
> > > +++ b/meson.build
> > > @@ -70,6 +70,12 @@
> > >  #      # Execute single test interactively such that features like `debug ()` work.
> > >  #      $ meson test -i --test-args='-ix' t1400-update-ref
> > >  #
> > > +#      # Execute all benchmarks.
> > > +#      $ meson test -i --benchmark
> > > +#
> > > +#      # Execute single benchmark.
> > > +#      $ meson test -i --benchmark p0000-*
> > > +#
> > >  #   Test execution is parallelized by default and scales with the number of
> > >  #   processor cores available. You can change the number of processes by passing
> > >  #   the `-jN` flag to `meson test`.
> >
> > Doesn't the above give the impression that benchmarks are also
> > parallelized by default while the commit message says they are not?
>
> To me it doesn't because it talks about test execution, not benchmark
> execution. The comments for each of the invocations also discern between
> tests and benchmarks. I'm happy to hear about alternative ways to word
> this though.

Maybe something like:

"Test execution (but not benchmark execution) is parallelized by default ..."

Re: [PATCH v4 4/5] meson: wire up benchmarks

[Patrick Steinhardt]

On Fri, Apr 25, 2025 at 10:09:18AM +0200, Christian Couder wrote:
> On Fri, Apr 25, 2025 at 10:01 AM Patrick Steinhardt <ps@pks.im> wrote:
> >
> > On Fri, Apr 25, 2025 at 09:57:33AM +0200, Christian Couder wrote:
> > > On Fri, Apr 25, 2025 at 9:28 AM Patrick Steinhardt <ps@pks.im> wrote:
> > > >
> > > > Wire up benchmarks in Meson. The setup is mostly the same as how we wire
> > > > up our tests. The only difference is that benchmarks get wired up via
> > > > the `benchmark()` option instead of via `test()`, which gives them a bit
> > > > of special treatment:
> > > >
> > > >   - Benchmarks never run in parallel.
> > > >
> > > >   - Benchmarks aren't run by default when tests are executed.
> > > >
> > > >   - Meson does not inject the `MALLOC_PERTURB` environment variable.
> > >
> > > [...]
> > >
> > > > diff --git a/meson.build b/meson.build
> > > > index 410bbf93dad..2cd05a9b1ba 100644
> > > > --- a/meson.build
> > > > +++ b/meson.build
> > > > @@ -70,6 +70,12 @@
> > > >  #      # Execute single test interactively such that features like `debug ()` work.
> > > >  #      $ meson test -i --test-args='-ix' t1400-update-ref
> > > >  #
> > > > +#      # Execute all benchmarks.
> > > > +#      $ meson test -i --benchmark
> > > > +#
> > > > +#      # Execute single benchmark.
> > > > +#      $ meson test -i --benchmark p0000-*
> > > > +#
> > > >  #   Test execution is parallelized by default and scales with the number of
> > > >  #   processor cores available. You can change the number of processes by passing
> > > >  #   the `-jN` flag to `meson test`.
> > >
> > > Doesn't the above give the impression that benchmarks are also
> > > parallelized by default while the commit message says they are not?
> >
> > To me it doesn't because it talks about test execution, not benchmark
> > execution. The comments for each of the invocations also discern between
> > tests and benchmarks. I'm happy to hear about alternative ways to word
> > this though.
> 
> Maybe something like:
> 
> "Test execution (but not benchmark execution) is parallelized by default ..."

Okay. I amended this locally, but will hold off sending another
iteration just for this change. Thanks!

Patrick

[PATCH v4 5/5] meson: wire up benchmarking options

[Patrick Steinhardt]

Wire up a couple of benchmarking options that we end up writing into our
"GIT-BUILD-OPTIONS" file. These options allow users to control how
exactly benchmarks are executed.

Note that neither `GIT_PERF_MAKE_COMMAND` nor `GIT_PERF_MAKE_OPTS` are
exposed as a build option. Those options are used by "t/perf/run", which
is not used by Meson.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 meson.build       | 6 +++---
 meson_options.txt | 6 ++++++
 2 files changed, 9 insertions(+), 3 deletions(-)

diff --git a/meson.build b/meson.build
index 2cd05a9b1ba..5e073403632 100644
--- a/meson.build
+++ b/meson.build
@@ -706,11 +706,11 @@ builtin_sources += custom_target(
 # build options to our tests.
 build_options_config = configuration_data()
 build_options_config.set('GIT_INTEROP_MAKE_OPTS', '')
-build_options_config.set('GIT_PERF_LARGE_REPO', '')
+build_options_config.set_quoted('GIT_PERF_LARGE_REPO', get_option('benchmark_large_repo'))
 build_options_config.set('GIT_PERF_MAKE_COMMAND', '')
 build_options_config.set('GIT_PERF_MAKE_OPTS', '')
-build_options_config.set('GIT_PERF_REPEAT_COUNT', '')
-build_options_config.set('GIT_PERF_REPO', '')
+build_options_config.set_quoted('GIT_PERF_REPEAT_COUNT', get_option('benchmark_repeat_count').to_string())
+build_options_config.set_quoted('GIT_PERF_REPO', get_option('benchmark_repo'))
 build_options_config.set('GIT_TEST_CMP_USE_COPIED_CONTEXT', '')
 build_options_config.set('GIT_TEST_INDEX_VERSION', '')
 build_options_config.set('GIT_TEST_OPTS', '')
diff --git a/meson_options.txt b/meson_options.txt
index 7f5bca5c029..8547c0eb47f 100644
--- a/meson_options.txt
+++ b/meson_options.txt
@@ -103,6 +103,12 @@ option('docs_backend', type: 'combo', choices: ['asciidoc', 'asciidoctor', 'auto
 # Testing.
 option('benchmarks', type: 'feature', value: 'auto',
   description: 'Enable benchmarks. This requires Perl and GNU time.')
+option('benchmark_repo', type: 'string', value: '',
+  description: 'Repository to copy for the performance tests. Should be at least the size of the Git repository.')
+option('benchmark_large_repo', type: 'string', value: '',
+  description: 'Large repository to copy for the performance tests. Should be at least the size of the Linux repository.')
+option('benchmark_repeat_count', type: 'integer', value: 3,
+  description: 'Number of times a test should be repeated for best-of-N measurements.')
 option('coccinelle', type: 'feature', value: 'auto',
   description: 'Provide a coccicheck target that generates a Coccinelle patch.')
 option('tests', type: 'boolean', value: true,

-- 
2.49.0.901.g37484f566f.dirty

Re: [PATCH v4 0/5] meson: wire up support for benchmarks

[Christian Couder]

On Fri, Apr 25, 2025 at 9:28 AM Patrick Steinhardt <ps@pks.im> wrote:

> Changes in v4:
>   - The patch series was rebased on top of f65182a99e5 (The ninth batch,
>     2025-04-24). This is due to a conflict with ps/test-wo-perl-prereq.
>   - Introduce a 'benchmarks' option. This allows developers to require
>     benchmarks as desired. By default, we auto-detect whether the host
>     system has all dependencies available and enable or disable them
>     accordingly.

Nice!

>   - Report whether or not benchmarks are enabled via `summary()`.
>   - Our benchmarks depend on Perl, so add this dependency accordingly.

Nice too!

[...]

> Range-diff versus v3:
>
> 1:  2375a16c2c0 = 1:  174804805c2 t/perf: fix benchmarks with alternate repo formats
> 2:  69782035246 = 2:  41faa3d9b3b t/perf: use configured PERL_PATH
> 3:  cca40e9dd0e = 3:  87ea3ab1a45 t/perf: fix benchmarks with out-of-tree builds

In https://lore.kernel.org/git/aAc8f52rJ8ATncVc@pks.im/ you said you
would try to paraphrase some explanations about how GIT_BUILD_DIR is
computed, but it looks like there aren't any changes in the patch
since the previous version :-(

Re: [PATCH v4 0/5] meson: wire up support for benchmarks

[Patrick Steinhardt]

On Fri, Apr 25, 2025 at 10:06:12AM +0200, Christian Couder wrote:
> On Fri, Apr 25, 2025 at 9:28 AM Patrick Steinhardt <ps@pks.im> wrote:
> > Range-diff versus v3:
> >
> > 1:  2375a16c2c0 = 1:  174804805c2 t/perf: fix benchmarks with alternate repo formats
> > 2:  69782035246 = 2:  41faa3d9b3b t/perf: use configured PERL_PATH
> > 3:  cca40e9dd0e = 3:  87ea3ab1a45 t/perf: fix benchmarks with out-of-tree builds
> 
> In https://lore.kernel.org/git/aAc8f52rJ8ATncVc@pks.im/ you said you
> would try to paraphrase some explanations about how GIT_BUILD_DIR is
> computed, but it looks like there aren't any changes in the patch
> since the previous version :-(

There were, but not in this version:

3:  d03ca0c0630 ! 3:  a66bc5aaf51 t/perf: fix benchmarks with out-of-tree builds
    @@ Commit message
         assumption breaks with both CMake and Meson, where the build directory
         can be located in an arbitrary place.

    -    Adapt the script so that it works with out-of-tree builds. This prepares
    -    us for wiring up benchmarks in Meson.
    +    Adapt the script so that it works with out-of-tree builds. Most
    +    importantly, this requires us to figure out the location of the build
    +    directory:
    +
    +      - When running benchmarks via our Makefile the build directory is the
    +        same as the source directory. We already know to derive the test
    +        directory ("t/") via `$(pwd)/..`, which works because we chdir into
    +        "t/perf" before executing benchmarks. We can thus derive the build
    +        directory by appending another "/.." to that path.
    +
    +      - When running benchmarks via Meson the build directory is located at
    +        an arbitrary location. The build system thus has to make the path
    +        known by exporting the `GIT_BUILD_DIR` environment variable.
    +
    +    This change prepares us for wiring up benchmarks in Meson.

The changes were already part of v3, so you have to look at the diff
versus v2.

Patrick

Re: [PATCH v4 0/5] meson: wire up support for benchmarks

[Christian Couder]

On Fri, Apr 25, 2025 at 10:26 AM Patrick Steinhardt <ps@pks.im> wrote:
>
> On Fri, Apr 25, 2025 at 10:06:12AM +0200, Christian Couder wrote:

> > In https://lore.kernel.org/git/aAc8f52rJ8ATncVc@pks.im/ you said you
> > would try to paraphrase some explanations about how GIT_BUILD_DIR is
> > computed, but it looks like there aren't any changes in the patch
> > since the previous version :-(
>
> There were, but not in this version:
>
> 3:  d03ca0c0630 ! 3:  a66bc5aaf51 t/perf: fix benchmarks with out-of-tree builds
>     @@ Commit message
>          assumption breaks with both CMake and Meson, where the build directory
>          can be located in an arbitrary place.
>
>     -    Adapt the script so that it works with out-of-tree builds. This prepares
>     -    us for wiring up benchmarks in Meson.
>     +    Adapt the script so that it works with out-of-tree builds. Most
>     +    importantly, this requires us to figure out the location of the build
>     +    directory:
>     +
>     +      - When running benchmarks via our Makefile the build directory is the
>     +        same as the source directory. We already know to derive the test
>     +        directory ("t/") via `$(pwd)/..`, which works because we chdir into
>     +        "t/perf" before executing benchmarks. We can thus derive the build
>     +        directory by appending another "/.." to that path.
>     +
>     +      - When running benchmarks via Meson the build directory is located at
>     +        an arbitrary location. The build system thus has to make the path
>     +        known by exporting the `GIT_BUILD_DIR` environment variable.
>     +
>     +    This change prepares us for wiring up benchmarks in Meson.
>
> The changes were already part of v3, so you have to look at the diff
> versus v2.

Oops, sorry for the noise.

[PATCH v5 0/5] meson: wire up support for benchmarks

[Patrick Steinhardt]

Hi,

this small patch series implements support for running our benchmarks in
"t/perf" via Meson. The series does not aim to replace "t/perf/run",
which is more fully-featured and allows running benchmarks against
multiple different trees. Instead, this series only allows running the
benchmarks against the current tree. Users are thus expected to continue
using "t/perf/run" for more advanced usecases.

Changes in v2:
  - Adapt "aggregate.perl" to use a "/usr/bin/env perl" shebang.
  - Link to v1: https://lore.kernel.org/r/20250331-pks-meson-benchmarks-v1-0-b2ace85616a3@pks.im

Changes in v3:
  - Document how to run benchmarks in "meson.build".
  - Expand the message for the commit that enables out-of-tree
    benchmarking.
  - Link to v2: https://lore.kernel.org/r/20250414-pks-meson-benchmarks-v2-0-04377080a167@pks.im

Changes in v4:
  - The patch series was rebased on top of f65182a99e5 (The ninth batch,
    2025-04-24). This is due to a conflict with ps/test-wo-perl-prereq.
  - Introduce a 'benchmarks' option. This allows developers to require
    benchmarks as desired. By default, we auto-detect whether the host
    system has all dependencies available and enable or disable them
    accordingly.
  - Report whether or not benchmarks are enabled via `summary()`.
  - Our benchmarks depend on Perl, so add this dependency accordingly.
  - Link to v3: https://lore.kernel.org/r/20250422-pks-meson-benchmarks-v3-0-7aad68bac6fd@pks.im

Changes in v5:
  - Clarify that benchmarks are not run in parallel by default.
  - Link to v4: https://lore.kernel.org/r/20250425-pks-meson-benchmarks-v4-0-6b89555052c3@pks.im

Thanks!

Patrick

---
Patrick Steinhardt (5):
      t/perf: fix benchmarks with alternate repo formats
      t/perf: use configured PERL_PATH
      t/perf: fix benchmarks with out-of-tree builds
      meson: wire up benchmarks
      meson: wire up benchmarking options

 meson.build           | 22 ++++++++++-----
 meson_options.txt     |  8 ++++++
 t/meson.build         | 77 +++++++++++++++++++++++++++++++++++++++++++++++++++
 t/perf/aggregate.perl |  2 +-
 t/perf/perf-lib.sh    | 32 ++++++++++++++++++---
 t/perf/run            |  4 +--
 6 files changed, 131 insertions(+), 14 deletions(-)

Range-diff versus v4:

1:  bc19180b174 = 1:  03181f601f2 t/perf: fix benchmarks with alternate repo formats
2:  3e128c09333 = 2:  afb2fe1ce63 t/perf: use configured PERL_PATH
3:  f7adc20d14c = 3:  a61e43c6cfb t/perf: fix benchmarks with out-of-tree builds
4:  8b3ea380503 ! 4:  e51b46296ed meson: wire up benchmarks
    @@ meson.build
      #      # Execute single test interactively such that features like `debug ()` work.
      #      $ meson test -i --test-args='-ix' t1400-update-ref
      #
    +-#   Test execution is parallelized by default and scales with the number of
    +-#   processor cores available. You can change the number of processes by passing
    +-#   the `-jN` flag to `meson test`.
     +#      # Execute all benchmarks.
     +#      $ meson test -i --benchmark
     +#
     +#      # Execute single benchmark.
     +#      $ meson test -i --benchmark p0000-*
     +#
    - #   Test execution is parallelized by default and scales with the number of
    - #   processor cores available. You can change the number of processes by passing
    - #   the `-jN` flag to `meson test`.
    ++#   Test execution (but not benchmark execution) is parallelized by default and
    ++#   scales with the number of processor cores available. You can change the
    ++#   number of processes by passing the `-jN` flag to `meson test`.
    + #
    + # 4. Install the Git distribution. Again, this can be done via Meson, Ninja or
    + #    Samurai:
     @@ meson.build: git = find_program('git', dirs: program_path, native: true, required: false)
      sed = find_program('sed', dirs: program_path, native: true)
      shell = find_program('sh', dirs: program_path, native: true)
5:  a0b06cc7017 = 5:  20f9dd86d66 meson: wire up benchmarking options

---
base-commit: f65182a99e545d2f2bc22e6c1c2da192133b16a3
change-id: 20250328-pks-meson-benchmarks-a8fac5f69467

[PATCH v5 1/5] t/perf: fix benchmarks with alternate repo formats

[Patrick Steinhardt]

Many of our benchmarks operate on a user-defined repository that we copy
over before running the benchmarked logic. To keep unintentional side
effects caused by on-disk state at bay we skip copying some files. This
includes for example hooks, but also the repo's configuration.

It is quite sensible to not copy over the configuration, as it is quite
easy to inadvertently carry over configuration that may significantly
impact the performance measurements. But we cannot fully ignore the
configuration either, as it may contain information about the repository
format. This will cause failures when for example using a repository
with SHA256 object format or the reftable ref format.

Fix the issue by parsing the reference and object formats from the
source repository and passing them to git-init(1).

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 t/perf/perf-lib.sh | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/t/perf/perf-lib.sh b/t/perf/perf-lib.sh
index 8ab6d9c4694..1a9a51ca3cc 100644
--- a/t/perf/perf-lib.sh
+++ b/t/perf/perf-lib.sh
@@ -98,6 +98,8 @@ test_perf_create_repo_from () {
 	source_git="$("$MODERN_GIT" -C "$source" rev-parse --git-dir)"
 	objects_dir="$("$MODERN_GIT" -C "$source" rev-parse --git-path objects)"
 	common_dir="$("$MODERN_GIT" -C "$source" rev-parse --git-common-dir)"
+	refformat="$("$MODERN_GIT" -C "$source" rev-parse --show-ref-format)"
+	objectformat="$("$MODERN_GIT" -C "$source" rev-parse --show-object-format)"
 	mkdir -p "$repo/.git"
 	(
 		cd "$source" &&
@@ -114,7 +116,7 @@ test_perf_create_repo_from () {
 	) &&
 	(
 		cd "$repo" &&
-		"$MODERN_GIT" init -q &&
+		"$MODERN_GIT" init -q --ref-format="$refformat" --object-format="$objectformat" &&
 		test_perf_do_repo_symlink_config_ &&
 		mv .git/hooks .git/hooks-disabled 2>/dev/null &&
 		if test -f .git/index.lock

-- 
2.49.0.967.g6a0df3ecc3.dirty

[PATCH v5 2/5] t/perf: use configured PERL_PATH

[Patrick Steinhardt]

Our benchmarks use a couple of Perl scripts to compute results. These
Perl scripts get executed directly, and as the shebang is hardcoded to
"/usr/bin/perl" this will fail on any system where the Perl interpreter
is located in a different path.

Our build infrastructure already lets users configure the location of
Perl, which ultimately gets written into the GIT-BUILD-OPTIONS file.
This file is being sourced by "test-lib.sh", and consequently we already
have the "PERL_PATH" variable available that contains its configured
location.

Use "PERL_PATH" to execute Perl scripts, which makes them work on more
esoteric systems like NixOS. Furthermore, adapt the shebang to use
env(1) to execute Perl so that users who have Perl in PATH, but in a
non-standard location can execute the script directly.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 t/perf/aggregate.perl | 2 +-
 t/perf/perf-lib.sh    | 4 ++--
 t/perf/run            | 4 ++--
 3 files changed, 5 insertions(+), 5 deletions(-)

diff --git a/t/perf/aggregate.perl b/t/perf/aggregate.perl
index 575d2000cc1..1791c7528a9 100755
--- a/t/perf/aggregate.perl
+++ b/t/perf/aggregate.perl
@@ -1,4 +1,4 @@
-#!/usr/bin/perl
+#!/usr/bin/env perl
 
 use lib '../../perl/build/lib';
 use strict;
diff --git a/t/perf/perf-lib.sh b/t/perf/perf-lib.sh
index 1a9a51ca3cc..4173eee4def 100644
--- a/t/perf/perf-lib.sh
+++ b/t/perf/perf-lib.sh
@@ -276,7 +276,7 @@ test_perf_ () {
 	else
 		test_ok_ "$1"
 	fi
-	"$TEST_DIRECTORY"/perf/min_time.perl test_time.* >"$base".result
+	"$PERL_PATH" "$TEST_DIRECTORY"/perf/min_time.perl test_time.* >"$base".result
 	rm test_time.*
 }
 
@@ -324,7 +324,7 @@ test_at_end_hook_ () {
 	if test -z "$GIT_PERF_AGGREGATING_LATER"; then
 		(
 			cd "$TEST_DIRECTORY"/perf &&
-			./aggregate.perl --results-dir="$TEST_RESULTS_DIR" $(basename "$0")
+			"$PERL_PATH" ./aggregate.perl --results-dir="$TEST_RESULTS_DIR" $(basename "$0")
 		)
 	fi
 }
diff --git a/t/perf/run b/t/perf/run
index 486ead21980..073bcb2afff 100755
--- a/t/perf/run
+++ b/t/perf/run
@@ -192,10 +192,10 @@ run_subsection () {
 
 	if test -z "$GIT_PERF_SEND_TO_CODESPEED"
 	then
-		./aggregate.perl --results-dir="$TEST_RESULTS_DIR" $codespeed_opt "$@"
+		"$PERL_PATH" ./aggregate.perl --results-dir="$TEST_RESULTS_DIR" $codespeed_opt "$@"
 	else
 		json_res_file=""$TEST_RESULTS_DIR"/$GIT_PERF_SUBSECTION/aggregate.json"
-		./aggregate.perl --results-dir="$TEST_RESULTS_DIR" --codespeed "$@" | tee "$json_res_file"
+		"$PERL_PATH" ./aggregate.perl --results-dir="$TEST_RESULTS_DIR" --codespeed "$@" | tee "$json_res_file"
 		send_data_url="$GIT_PERF_SEND_TO_CODESPEED/result/add/json/"
 		curl -v --request POST --data-urlencode "json=$(cat "$json_res_file")" "$send_data_url"
 	fi

-- 
2.49.0.967.g6a0df3ecc3.dirty

[PATCH v5 3/5] t/perf: fix benchmarks with out-of-tree builds

[Patrick Steinhardt]

The "perf-lib.sh" script is sourced by all of our benchmarking suites to
make available common infrastructure. The script assumes that build and
source directory are the same, which works for our Makefile. But the
assumption breaks with both CMake and Meson, where the build directory
can be located in an arbitrary place.

Adapt the script so that it works with out-of-tree builds. Most
importantly, this requires us to figure out the location of the build
directory:

  - When running benchmarks via our Makefile the build directory is the
    same as the source directory. We already know to derive the test
    directory ("t/") via `$(pwd)/..`, which works because we chdir into
    "t/perf" before executing benchmarks. We can thus derive the build
    directory by appending another "/.." to that path.

  - When running benchmarks via Meson the build directory is located at
    an arbitrary location. The build system thus has to make the path
    known by exporting the `GIT_BUILD_DIR` environment variable.

This change prepares us for wiring up benchmarks in Meson.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 t/perf/perf-lib.sh | 26 ++++++++++++++++++++++++--
 1 file changed, 24 insertions(+), 2 deletions(-)

diff --git a/t/perf/perf-lib.sh b/t/perf/perf-lib.sh
index 4173eee4def..5406557b7ca 100644
--- a/t/perf/perf-lib.sh
+++ b/t/perf/perf-lib.sh
@@ -25,7 +25,29 @@ TEST_OUTPUT_DIRECTORY=$(pwd)
 TEST_NO_CREATE_REPO=t
 TEST_NO_MALLOC_CHECK=t
 
-. ../test-lib.sh
+# While test-lib.sh computes the build directory for us, we also have to do the
+# same thing in order to locate the script via GIT-BUILD-OPTIONS in the first
+# place.
+GIT_BUILD_DIR="${GIT_BUILD_DIR:-$TEST_DIRECTORY/..}"
+if test -f "$GIT_BUILD_DIR/GIT-BUILD-DIR"
+then
+	GIT_BUILD_DIR="$(cat "$GIT_BUILD_DIR/GIT-BUILD-DIR")" || exit 1
+	# On Windows, we must convert Windows paths lest they contain a colon
+	case "$(uname -s)" in
+	*MINGW*)
+		GIT_BUILD_DIR="$(cygpath -au "$GIT_BUILD_DIR")"
+		;;
+	esac
+fi
+
+if test ! -f "$GIT_BUILD_DIR"/GIT-BUILD-OPTIONS
+then
+	echo >&2 'error: GIT-BUILD-OPTIONS missing (has Git been built?).'
+	exit 1
+fi
+
+. "$GIT_BUILD_DIR"/GIT-BUILD-OPTIONS
+. "$GIT_SOURCE_DIR"/t/test-lib.sh
 
 unset GIT_CONFIG_NOSYSTEM
 GIT_CONFIG_SYSTEM="$TEST_DIRECTORY/perf/config"
@@ -324,7 +346,7 @@ test_at_end_hook_ () {
 	if test -z "$GIT_PERF_AGGREGATING_LATER"; then
 		(
 			cd "$TEST_DIRECTORY"/perf &&
-			"$PERL_PATH" ./aggregate.perl --results-dir="$TEST_RESULTS_DIR" $(basename "$0")
+			"$PERL_PATH" "$GIT_SOURCE_DIR"/t/perf/aggregate.perl --results-dir="$TEST_RESULTS_DIR" $(basename "$0")
 		)
 	fi
 }

-- 
2.49.0.967.g6a0df3ecc3.dirty

[PATCH v5 4/5] meson: wire up benchmarks

[Patrick Steinhardt]

Wire up benchmarks in Meson. The setup is mostly the same as how we wire
up our tests. The only difference is that benchmarks get wired up via
the `benchmark()` option instead of via `test()`, which gives them a bit
of special treatment:

  - Benchmarks never run in parallel.

  - Benchmarks aren't run by default when tests are executed.

  - Meson does not inject the `MALLOC_PERTURB` environment variable.

Using benchmarks is quite simple:

    ```
    $ meson setup build
    # Run all benchmarks.
    $ meson test -C build --benchmark
    # Run a specific benchmark.
    $ meson test -C build --benchmark p0000-*
    ```

Other than that the usual command line arguments accepted when running
tests are also accepted when running benchmarks.

Note that the benchmarking target is somewhat limited because it will
only run benchmarks for the current build. Other use cases, like running
benchmarks against multiple different versions of Git, are not currently
supported. Users should continue to use "t/perf/run" for those use
cases. The script should get extended at one point in time to support
Meson, but this is outside of the scope of this series.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 meson.build       | 16 +++++++++---
 meson_options.txt |  2 ++
 t/meson.build     | 77 +++++++++++++++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 91 insertions(+), 4 deletions(-)

diff --git a/meson.build b/meson.build
index 410bbf93dad..939a2eae326 100644
--- a/meson.build
+++ b/meson.build
@@ -70,9 +70,15 @@
 #      # Execute single test interactively such that features like `debug ()` work.
 #      $ meson test -i --test-args='-ix' t1400-update-ref
 #
-#   Test execution is parallelized by default and scales with the number of
-#   processor cores available. You can change the number of processes by passing
-#   the `-jN` flag to `meson test`.
+#      # Execute all benchmarks.
+#      $ meson test -i --benchmark
+#
+#      # Execute single benchmark.
+#      $ meson test -i --benchmark p0000-*
+#
+#   Test execution (but not benchmark execution) is parallelized by default and
+#   scales with the number of processor cores available. You can change the
+#   number of processes by passing the `-jN` flag to `meson test`.
 #
 # 4. Install the Git distribution. Again, this can be done via Meson, Ninja or
 #    Samurai:
@@ -235,6 +241,7 @@ git = find_program('git', dirs: program_path, native: true, required: false)
 sed = find_program('sed', dirs: program_path, native: true)
 shell = find_program('sh', dirs: program_path, native: true)
 tar = find_program('tar', dirs: program_path, native: true)
+time = find_program('time', dirs: program_path, required: get_option('benchmarks'))
 
 target_shell = find_program('sh', dirs: program_path, native: false)
 
@@ -836,7 +843,7 @@ endif
 # features. It is optional if you want to neither execute tests nor use any of
 # these optional features.
 perl_required = get_option('perl')
-if get_option('gitweb').enabled() or 'netrc' in get_option('credential_helpers') or get_option('docs') != []
+if get_option('benchmarks').enabled() or get_option('gitweb').enabled() or 'netrc' in get_option('credential_helpers') or get_option('docs') != []
   perl_required = true
 endif
 
@@ -2082,6 +2089,7 @@ meson.add_dist_script(
 )
 
 summary({
+  'benchmarks': get_option('tests') and perl.found() and time.found(),
   'curl': curl.found(),
   'expat': expat.found(),
   'gettext': intl.found(),
diff --git a/meson_options.txt b/meson_options.txt
index 8ac30a52231..7f5bca5c029 100644
--- a/meson_options.txt
+++ b/meson_options.txt
@@ -101,6 +101,8 @@ option('docs_backend', type: 'combo', choices: ['asciidoc', 'asciidoctor', 'auto
   description: 'Which backend to use to generate documentation.')
 
 # Testing.
+option('benchmarks', type: 'feature', value: 'auto',
+  description: 'Enable benchmarks. This requires Perl and GNU time.')
 option('coccinelle', type: 'feature', value: 'auto',
   description: 'Provide a coccicheck target that generates a Coccinelle patch.')
 option('tests', type: 'boolean', value: true,
diff --git a/t/meson.build b/t/meson.build
index bfb744e8863..59438b06266 100644
--- a/t/meson.build
+++ b/t/meson.build
@@ -1097,11 +1097,71 @@ integration_tests = [
   't9903-bash-prompt.sh',
 ]
 
+benchmarks = [
+  'perf/p0000-perf-lib-sanity.sh',
+  'perf/p0001-rev-list.sh',
+  'perf/p0002-read-cache.sh',
+  'perf/p0003-delta-base-cache.sh',
+  'perf/p0004-lazy-init-name-hash.sh',
+  'perf/p0005-status.sh',
+  'perf/p0006-read-tree-checkout.sh',
+  'perf/p0007-write-cache.sh',
+  'perf/p0008-odb-fsync.sh',
+  'perf/p0071-sort.sh',
+  'perf/p0090-cache-tree.sh',
+  'perf/p0100-globbing.sh',
+  'perf/p1006-cat-file.sh',
+  'perf/p1400-update-ref.sh',
+  'perf/p1450-fsck.sh',
+  'perf/p1451-fsck-skip-list.sh',
+  'perf/p1500-graph-walks.sh',
+  'perf/p2000-sparse-operations.sh',
+  'perf/p3400-rebase.sh',
+  'perf/p3404-rebase-interactive.sh',
+  'perf/p4000-diff-algorithms.sh',
+  'perf/p4001-diff-no-index.sh',
+  'perf/p4002-diff-color-moved.sh',
+  'perf/p4205-log-pretty-formats.sh',
+  'perf/p4209-pickaxe.sh',
+  'perf/p4211-line-log.sh',
+  'perf/p4220-log-grep-engines.sh',
+  'perf/p4221-log-grep-engines-fixed.sh',
+  'perf/p5302-pack-index.sh',
+  'perf/p5303-many-packs.sh',
+  'perf/p5304-prune.sh',
+  'perf/p5310-pack-bitmaps.sh',
+  'perf/p5311-pack-bitmaps-fetch.sh',
+  'perf/p5312-pack-bitmaps-revs.sh',
+  'perf/p5313-pack-objects.sh',
+  'perf/p5314-name-hash.sh',
+  'perf/p5326-multi-pack-bitmaps.sh',
+  'perf/p5332-multi-pack-reuse.sh',
+  'perf/p5333-pseudo-merge-bitmaps.sh',
+  'perf/p5550-fetch-tags.sh',
+  'perf/p5551-fetch-rescan.sh',
+  'perf/p5600-partial-clone.sh',
+  'perf/p5601-clone-reference.sh',
+  'perf/p6100-describe.sh',
+  'perf/p6300-for-each-ref.sh',
+  'perf/p7000-filter-branch.sh',
+  'perf/p7102-reset.sh',
+  'perf/p7300-clean.sh',
+  'perf/p7519-fsmonitor.sh',
+  'perf/p7527-builtin-fsmonitor.sh',
+  'perf/p7810-grep.sh',
+  'perf/p7820-grep-engines.sh',
+  'perf/p7821-grep-engines-fixed.sh',
+  'perf/p7822-grep-perl-character.sh',
+  'perf/p9210-scalar.sh',
+  'perf/p9300-fast-import-export.sh',
+]
+
 # Sanity check that we are not missing any tests present in 't/'. This check
 # only runs once at configure time and is thus best-effort, only. It is
 # sufficient to catch missing test suites in our CI though.
 foreach glob, tests : {
   't[0-9][0-9][0-9][0-9]-*.sh': integration_tests,
+  'perf/p[0-9][0-9][0-9][0-9]-*.sh': benchmarks,
   'unit-tests/t-*.c': unit_test_programs,
   'unit-tests/u-*.c': clar_test_suites,
 }
@@ -1153,3 +1213,20 @@ foreach integration_test : integration_tests
     timeout: 0,
   )
 endforeach
+
+if perl.found() and time.found()
+  benchmark_environment = test_environment
+  benchmark_environment.set('GTIME', time.full_path())
+
+  foreach benchmark : benchmarks
+    benchmark(fs.stem(benchmark), shell,
+      args: [
+        fs.name(benchmark),
+      ],
+      workdir: meson.current_source_dir() / 'perf',
+      env: benchmark_environment,
+      depends: test_dependencies + bin_wrappers,
+      timeout: 0,
+    )
+  endforeach
+endif

-- 
2.49.0.967.g6a0df3ecc3.dirty

[PATCH v5 5/5] meson: wire up benchmarking options

[Patrick Steinhardt]

Wire up a couple of benchmarking options that we end up writing into our
"GIT-BUILD-OPTIONS" file. These options allow users to control how
exactly benchmarks are executed.

Note that neither `GIT_PERF_MAKE_COMMAND` nor `GIT_PERF_MAKE_OPTS` are
exposed as a build option. Those options are used by "t/perf/run", which
is not used by Meson.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 meson.build       | 6 +++---
 meson_options.txt | 6 ++++++
 2 files changed, 9 insertions(+), 3 deletions(-)

diff --git a/meson.build b/meson.build
index 939a2eae326..d06f82ff9f0 100644
--- a/meson.build
+++ b/meson.build
@@ -706,11 +706,11 @@ builtin_sources += custom_target(
 # build options to our tests.
 build_options_config = configuration_data()
 build_options_config.set('GIT_INTEROP_MAKE_OPTS', '')
-build_options_config.set('GIT_PERF_LARGE_REPO', '')
+build_options_config.set_quoted('GIT_PERF_LARGE_REPO', get_option('benchmark_large_repo'))
 build_options_config.set('GIT_PERF_MAKE_COMMAND', '')
 build_options_config.set('GIT_PERF_MAKE_OPTS', '')
-build_options_config.set('GIT_PERF_REPEAT_COUNT', '')
-build_options_config.set('GIT_PERF_REPO', '')
+build_options_config.set_quoted('GIT_PERF_REPEAT_COUNT', get_option('benchmark_repeat_count').to_string())
+build_options_config.set_quoted('GIT_PERF_REPO', get_option('benchmark_repo'))
 build_options_config.set('GIT_TEST_CMP_USE_COPIED_CONTEXT', '')
 build_options_config.set('GIT_TEST_INDEX_VERSION', '')
 build_options_config.set('GIT_TEST_OPTS', '')
diff --git a/meson_options.txt b/meson_options.txt
index 7f5bca5c029..8547c0eb47f 100644
--- a/meson_options.txt
+++ b/meson_options.txt
@@ -103,6 +103,12 @@ option('docs_backend', type: 'combo', choices: ['asciidoc', 'asciidoctor', 'auto
 # Testing.
 option('benchmarks', type: 'feature', value: 'auto',
   description: 'Enable benchmarks. This requires Perl and GNU time.')
+option('benchmark_repo', type: 'string', value: '',
+  description: 'Repository to copy for the performance tests. Should be at least the size of the Git repository.')
+option('benchmark_large_repo', type: 'string', value: '',
+  description: 'Large repository to copy for the performance tests. Should be at least the size of the Linux repository.')
+option('benchmark_repeat_count', type: 'integer', value: 3,
+  description: 'Number of times a test should be repeated for best-of-N measurements.')
 option('coccinelle', type: 'feature', value: 'auto',
   description: 'Provide a coccicheck target that generates a Coccinelle patch.')
 option('tests', type: 'boolean', value: true,

-- 
2.49.0.967.g6a0df3ecc3.dirty