Continuous Benchmarking

[Patrick Steinhardt <ps@pks.im>]

Hi,

due to a couple performance regressions that we have hit over the last
couple Git releases at GitLab, we have started to set up an effort to
implement continuous benchmarking for the Git project. The intent is to
have regular (daily) benchmarking runs against Git's `master` and `next`
branches to be able to spot any performance regressions before they make
it into the next release.

I have started with a relatively simple setup:

  - I have started collection benchmarks that I myself do regularly [1].
    These benchmarks are built on hyperfine and are thus not part of the
    Git repository itself.

  - GitLab CI runs on a nightly basis, executing a subset of these
    benchmarks [2].

  - Results are uploaded with a hyperfine adaptor to Bencher and are
    summarized in dashboards.

This at least gives us some visibility in severe performance outliers,
whether these are improvements or regressions. Some statistics are
applied on this data to automatically generate alerts when things are
significantly changing.

The setup is of course not perfect. It's built on top of CI jobs, which
are by their very nature not really performing consistent. The scripts
are hosted outside of Git. And I'm the only one running this.

So I wonder whether there is a wider interest in the Git community to
have this infrastructure part of the Git project itself. This may
include steps like the following:

  - Extending our performance tests we have in "t/perf" to cover more
    benchmarks.

  - Writing an adaptor that is able to upload the data generated from
    our perf scripts to Bencher.

  - Setting up proper infrastructure to do the benchmarking. We may for
    now also continue to use GitLab CI, but as said they are quite noisy
    overall. Dedicated servers would help here.

  - Sending alerts to the Git mailing list.

I'm happy to hear your thoughts on this. Any ideas are welcome,
including "we're not interested at all". In that case, we'd simply
continue to maintain the setup ourselves at GitLab.

Thanks!

Patrick

[1]: https://gitlab.com/gitlab-org/data-access/git/benchmarks
[2]: https://gitlab.com/gitlab-org/data-access/git/benchmarks/-/blob/main/.gitlab-ci.yml?ref_type=heads
[3]: https://bencher.dev/console/projects/git/plots