Re: [PATCH 4/5] meson: wire up benchmarks

[Patrick Steinhardt <ps@pks.im>]

On Thu, Apr 10, 2025 at 01:44:55PM +0200, Toon Claes wrote:
> Patrick Steinhardt <ps@pks.im> writes:
> 
> > Wire up benchmarks in Meson. The setup is mostly the same as how we wire
> > up our tests. The only difference is that benchmarks get wired up via
> > the `benchmark()` option instead of via `test()`, which gives them a bit
> > of special treatment:
> >
> >   - Benchmarks never run in parallel.
> >
> >   - Benchmarks aren't run by default when tests are executed.
> >
> >   - Meson does not inject the `MALLOC_PERTURB` environment variable.
> >
> > Using benchmarks is quite simple:
> >
> >     ```
> >     $ meson setup build
> >     # Run all benchmarks.
> >     $ meson test -C build --benchmark
> >     # Run a specific benchmark.
> >     $ meson test -C build --benchmark p0000-*
> >     ```
> 
> I really appreciate the efforts, and I'd love to run the perf tests with
> Meson, but at the moment I don't know how these changes are useful
> because this doesn't print anything relevant:
> 
>     $ meson test -C build --benchmark 'p0005-*'
>     ninja: Entering directory `/home/toon/devel/git/build'
>     [1/28] Generating GIT-VERSION-FILE with a custom command (wrapped by meson to set env)
>     1/1 p0005-status        OK              7.39s
> 
>     Ok:                 1
>     Expected Fail:      0
>     Fail:               0
>     Unexpected Pass:    0
>     Skipped:            0
>     Timeout:            0
> 
>     Full log written to /home/toon/devel/git/build/meson-logs/testlog.txt

You can execute with `meson test -i`, which will cause Meson to print
the benchmark's output.

> > Other than that the usual command line arguments accepted when running
> > tests are also accepted when running benchmarks.
> >
> > Note that the benchmarking target is somewhat limited because it will
> > only run benchmarks for the current build. Other usecases, like running
> > benchmarks against multiple different versions of Git, are not currently
> > supported. Users should continue to use "t/perf/run" for those usecases.
> > The script should get extended at one point in time to support Meson,
> > but this is outside of the scope of this series.
> 
> Yeah, this is unfortunate, but totally understandable. I've been digging
> in `t/perf/run` and `f/perf/aggregate.perl` and it doesn't look easy to
> adapt to Meson.
> 
> But I was wondering, instead of trying to fully integrate Meson into
> those scripts, could we modify the scripts so they work with binaries
> built by Meson? I mean, if we could run
> `cd t/perf && ./run ../../build1 ../../build2 p0005*` and it would
> simply run the benchmarks in those directories (without trying to check
> out code and build the sources). I think this would help a lot already.

This is another thing we could (and eventually should) do, but it's
orthogonal to this patch series from my point of view.

Patrick