[PATCH RFC v6 00/19] Modernize the build system

[Patrick Steinhardt <ps@pks.im>]

Hi,

this patch series modernizes our build infrasturcture. It refactors
various parts of it to make it possible to perform out-of-tree builds in
theory.

The series then continues to wire up Meson as a proposed alternative as
a modern replacement for our current build systems. It provides better
integration into IDEs than our Makefiles, better usability than our
Makefiles and CMake, better configuration and discoverability thereof
than autoconf and CMake (which is of course my personal opinion).

The intent is to have Meson replace autoconf, which is mostly unused and
broken. Eventually, the plan would to also replace CMake and our
Makefiles in case we figure out that this replacement is indeed a net
benefit for the project, but all of this would happen over the course of
multiple releases.

Changes in v6:

  - Add documentation for how to use Meson.

  - Fix compilation in case libintl exists, but required tools like
    msgfmt and related don't. Instead of failing, we now automatically
    compile with the equivalent of NO_GETTEXT=YesPlease.

  - Update last patch that fixes semantic merge conflics with in-flight
    topics in "seen".

  - I've changed the series to be based on "master" at b31fb630c0 (Merge
    https://github.com/j6t/git-gui, 2024-11-11) as all prereqs of this
    series have since landed. This doesn't make any difference though
    as there aren't any merge conflicts or additional changes caused by
    this. I mostly mention this so that there is no confusion around the
    changed base commit.

As before, the tip of this patch series catches up with what is
happening in "seen". If you want to test it without topics in "seen" you
have to revert it.

Links to previous versions:

  - Link to v1: https://lore.kernel.org/r/cover.1727881164.git.ps@pks.im
  - Link to v2: https://lore.kernel.org/r/cover.1728485139.git.ps@pks.im
  - Link to v3: https://lore.kernel.org/r/cover.1729254070.git.ps@pks.im
  - Link to v4: https://lore.kernel.org/r/cover.1729771605.git.ps@pks.im
  - Link to v5: https://lore.kernel.org/r/cover.1731335938.git.ps@pks.im

Thanks!

Patrick

To: git@vger.kernel.org
Cc: Eli Schwartz <eschwartz@gentoo.org>
Cc: Eric Sunshine <sunshine@sunshineco.com>
Cc: Phillip Wood <phillip.wood123@gmail.com>
Cc: Junio C Hamano <gitster@pobox.com>
Cc: Ramsay Jones <ramsay@ramsayjones.plus.com>
Cc: Taylor Blau <me@ttaylorr.com>
Cc: David Aguilar <davvid@gmail.com>
Cc: Jeff King <peff@peff.net>

Patrick Steinhardt (19):
      Makefile: use common template for GIT-BUILD-OPTIONS
      Makefile: consistently use @PLACEHOLDER@ to substitute
      Makefile: consistently use PERL_PATH
      Makefile: extract script to massage Perl scripts
      Makefile: use "generate-perl.sh" to massage Perl library
      Makefile: extract script to massage Shell scripts
      Makefile: extract script to generate gitweb.cgi
      Makefile: refactor GIT-VERSION-GEN to be reusable
      Makefile: refactor generators to be PWD-independent
      Makefile: allow "bin-wrappers/" directory to exist
      Makefile: simplify building of templates
      Documentation: allow sourcing generated includes from separate dir
      Documentation: teach "cmd-list.perl" about out-of-tree builds
      Documentation: extract script to generate a list of mergetools
      t: better support for out-of-tree builds
      t: allow overriding build dir
      Documentation: add comparison of build systems
      Introduce support for the Meson build system
      meson: fix conflicts with in-flight topics

 .gitignore                                         |    1 -
 Documentation/CodingGuidelines                     |    2 +-
 Documentation/Makefile                             |   28 +-
 Documentation/build-docdep.perl                    |    2 +
 Documentation/cmd-list.perl                        |   23 +-
 Documentation/config/diff.txt                      |    2 +-
 Documentation/config/merge.txt                     |    2 +-
 Documentation/generate-mergetool-list.sh           |   17 +
 Documentation/git.txt                              |   24 +-
 Documentation/meson.build                          |  317 ++++
 Documentation/technical/build-systems.txt          |  224 +++
 GIT-BUILD-OPTIONS.in                               |   47 +
 GIT-VERSION-GEN                                    |   12 +-
 Makefile                                           |  209 +--
 bin-wrappers/.gitignore                            |    9 +
 bin-wrappers/meson.build                           |   28 +
 bin-wrappers/wrap-for-bin.sh                       |   37 +
 configure.ac                                       |    2 +-
 contrib/buildsystems/CMakeLists.txt                |  230 +--
 contrib/completion/meson.build                     |    8 +
 contrib/meson.build                                |    1 +
 generate-cmdlist.sh                                |   42 +-
 generate-configlist.sh                             |   20 +-
 generate-hooklist.sh                               |   15 +-
 generate-perl.sh                                   |   36 +
 generate-script.sh                                 |   34 +
 git-cvsserver.perl                                 |    2 +-
 git-instaweb.sh                                    |    8 +-
 git-request-pull.sh                                |    2 +-
 git-send-email.perl                                |    2 +-
 git-sh-i18n.sh                                     |    6 +-
 git-sh-setup.sh                                    |    6 +-
 git-svn.perl                                       |    2 +-
 gitk-git/po/vi.po                                  |    2 +-
 gitweb/GITWEB-BUILD-OPTIONS.in                     |   25 +
 gitweb/Makefile                                    |   58 +-
 gitweb/generate-gitweb.sh                          |   45 +
 gitweb/gitweb.perl                                 |   44 +-
 gitweb/meson.build                                 |   63 +
 meson.build                                        | 1780 ++++++++++++++++++++
 meson_options.txt                                  |   73 +
 perl/FromCPAN/Mail/meson.build                     |    7 +
 perl/FromCPAN/meson.build                          |    9 +
 perl/Git/I18N.pm                                   |    6 +-
 perl/Git/LoadCPAN.pm                               |    6 +-
 perl/Git/LoadCPAN/Mail/meson.build                 |    7 +
 perl/Git/LoadCPAN/meson.build                      |    9 +
 perl/Git/SVN/Memoize/meson.build                   |    7 +
 perl/Git/SVN/meson.build                           |   20 +
 perl/Git/meson.build                               |   18 +
 perl/header_templates/fixed_prefix.template.pl     |    2 +-
 perl/header_templates/runtime_prefix.template.pl   |    8 +-
 perl/meson.build                                   |   12 +
 po/meson.build                                     |   27 +
 subprojects/.gitignore                             |    1 +
 subprojects/curl.wrap                              |   13 +
 subprojects/expat.wrap                             |   13 +
 subprojects/openssl.wrap                           |   15 +
 subprojects/pcre2.wrap                             |   16 +
 subprojects/zlib.wrap                              |   13 +
 t/helper/meson.build                               |   93 +
 t/lib-gettext.sh                                   |    4 +-
 t/meson.build                                      | 1107 ++++++++++++
 t/t7609-mergetool--lib.sh                          |    2 +-
 t/test-lib.sh                                      |   22 +-
 templates/Makefile                                 |   39 +-
 templates/branches--                               |    1 -
 templates/{this--description => description}       |    0
 .../applypatch-msg.sample}                         |    0
 .../commit-msg.sample}                             |    0
 .../fsmonitor-watchman.sample}                     |    0
 templates/hooks/meson.build                        |   24 +
 .../post-update.sample}                            |    0
 .../pre-applypatch.sample}                         |    0
 .../pre-commit.sample}                             |    0
 .../pre-merge-commit.sample}                       |    0
 .../pre-push.sample}                               |    0
 .../pre-rebase.sample}                             |    0
 .../pre-receive.sample}                            |    0
 .../prepare-commit-msg.sample}                     |    0
 .../push-to-checkout.sample}                       |    0
 .../sendemail-validate.sample}                     |    0
 .../{hooks--update.sample => hooks/update.sample}  |    0
 templates/{info--exclude => info/exclude}          |    0
 templates/info/meson.build                         |    5 +
 templates/meson.build                              |   13 +
 unimplemented.sh                                   |    2 +-
 wrap-for-bin.sh                                    |   36 -
 88 files changed, 4600 insertions(+), 447 deletions(-)

Range-diff versus v5:

 1:  ed28e754e8 =  1:  687ae57464 Makefile: use common template for GIT-BUILD-OPTIONS
 2:  178ba0e5d6 =  2:  53d5f0103a Makefile: consistently use @PLACEHOLDER@ to substitute
 3:  66b82c152b =  3:  15a4864450 Makefile: consistently use PERL_PATH
 4:  8382a6b840 =  4:  ed09cc6027 Makefile: extract script to massage Perl scripts
 5:  51491b778d =  5:  56caa7a075 Makefile: use "generate-perl.sh" to massage Perl library
 6:  c3946a69db =  6:  ee2135d178 Makefile: extract script to massage Shell scripts
 7:  a58c901466 =  7:  13fbeee749 Makefile: extract script to generate gitweb.cgi
 8:  a9b886b27a =  8:  f0073e58c8 Makefile: refactor GIT-VERSION-GEN to be reusable
 9:  f2eccc7d65 =  9:  fd412ba3a0 Makefile: refactor generators to be PWD-independent
10:  c891e56e32 = 10:  2e3833a0a3 Makefile: allow "bin-wrappers/" directory to exist
11:  4e9525f64e = 11:  7d93c8afa0 Makefile: simplify building of templates
12:  b209ab75b5 = 12:  8dad1314b2 Documentation: allow sourcing generated includes from separate dir
13:  5e8be2473c = 13:  90d6e0b4eb Documentation: teach "cmd-list.perl" about out-of-tree builds
14:  621d18d281 = 14:  0911b47694 Documentation: extract script to generate a list of mergetools
15:  f56eb0cd86 = 15:  3861eab108 t: better support for out-of-tree builds
16:  9618c7745e = 16:  2226f932d3 t: allow overriding build dir
17:  568287b16a = 17:  a9686e2940 Documentation: add comparison of build systems
18:  42bc858172 ! 18:  a7a7f7efe3 Introduce support for the Meson build system
    @@ Commit message
             symbolic links pointing to the former. This rule is consistent in
             itself and thus easier to reason about.
     
    -      - We do not install dashed binaries into `$libexec/git-core` anymore.
    -        So there won't e.g. be a symlink for git-add(1). These are not
    +      - We do not install dashed binaries into `$libexec/git-core` anymore,
    +        so there won't e.g. be a symlink for git-add(1). These are not
             required by modern Git and there isn't really much of a use case for
             those anymore. By not installing those symlinks we thus start the
             deprecation of this layout.
     
    -      - Documentation does not yet exist. Same here, it will follow if the
    -        project can agree on Meson.
    -
           - We're targeting Meson 1.3.0, which has been released relatively
             recently November 2023. The only feature we use from that version is
             `fs.relative_to()`, which we could replace if necessary. If so, we
    @@ gitweb/meson.build (new)
     
      ## meson.build (new) ##
     @@
    ++# Meson build system
    ++# ==================
    ++#
    ++# The Meson build system is an alternative to our Makefile that you can use to
    ++# build, test and install Git. Using Meson results in a couple of benefits:
    ++#
    ++#  - Out-of-tree builds.
    ++#  - Better integration into IDEs.
    ++#  - Easy-to-use autoconfiguration of available features on your system.
    ++#
    ++# To use Meson from the command line you need to have both Meson and Ninja
    ++# installed. Alternatively, if you do not have Python available on your system,
    ++# you can also use Muon instead of Meson and Samurai instead of Ninja, both of
    ++# which are drop-ins replacement that only depend on C.
    ++#
    ++# Basic usage
    ++# ===========
    ++#
    ++# In the most trivial case, you can configure, build and install Git like this:
    ++#
    ++#  1. Set up the build directory. This only needs to happen once per build
    ++#     directory you want to have. You can also configure multiple different
    ++#     build directories with different configurations.
    ++#
    ++#      $ meson setup build/
    ++#
    ++#     The build directory gets ignored by Git automatically as Meson will write
    ++#     a ".gitignore" file into it. From hereon, we will assume that you execute
    ++#     commands inside this build directory.
    ++#
    ++# 2. Compile Git. You can either use Meson, Ninja or Samurai to do this, so all
    ++#    of the following invocations are equivalent:
    ++#
    ++#      $ meson compile
    ++#      $ ninja
    ++#      $ samu
    ++#
    ++#   The different invocations should ultimately not make much of a difference.
    ++#   Using Meson also works with other generators though, like when the build
    ++#   directory has been set up for use with Microsoft Visual Studio.
    ++#
    ++# 3. Execute tests. Again, you can either use Meson, Ninja or Samurai to do this:
    ++#
    ++#      $ meson test
    ++#      $ ninja test
    ++#      $ samu test
    ++#
    ++#   It is recommended to use Meson in this case though as it also provides you
    ++#   additional features that the other build systems don't have available.
    ++#   You can e.g. pass additional arguments to the test executables or run
    ++#   individual tests:
    ++#
    ++#      # Execute the t0000-basic integration test and t-reftable-stack unit test.
    ++#      $ meson test t0000-basic t-reftable-stack
    ++#
    ++#      # Execute all reftable unit tests.
    ++#      $ meson test t-reftable-*
    ++#
    ++#      # Execute all tests and stop with the first failure.
    ++#      $ meson test --maxfail 1
    ++#
    ++#      # Execute single test interactively such that features like `debug ()` work.
    ++#      $ meson test -i --test-args='-ix' t1400-update-ref
    ++#
    ++# 4. Install the Git distribution. Again, this can be done via Meson, Ninja or
    ++#    Samurai:
    ++#
    ++#      $ meson install
    ++#      $ ninja install
    ++#      $ samu install
    ++#
    ++#    The prefix into which Git shall be installed is defined when setting up
    ++#    the build directory. More on that in the "Configuration" section.
    ++#
    ++# Meson supports multiple backends. The default backend generates Ninja build
    ++# instructions, but it also supports the generation of Microsoft Visual
    ++# Studio solutions as well as Xcode projects. IDEs like Eclipse and Visual
    ++# Studio Code provide plugins to import Meson files directly.
    ++#
    ++# Configuration
    ++# =============
    ++#
    ++# The exact configuration of Git is determined when setting up the Git
    ++# directory. Unless told otherwise, Meson will automatically detect the
    ++# availability of various bits and pieces. There are two different kinds of
    ++# options that can be used to further tweak the build:
    ++#
    ++#   - Built-in options provided by Meson.
    ++#
    ++#   - Options defined by the project in the "meson_options.txt" file.
    ++#
    ++# Both kinds of options can be inspected by running `meson configure` in the
    ++# build directory, which will give you a list of the current value for all
    ++# options.
    ++#
    ++# Options can be configured either when setting up the build directory or can
    ++# be changed in preexisting build directories:
    ++#
    ++#      # Set up a build directory with optimized settings that will be
    ++#      # installed into an alternative prefix.
    ++#      $ meson setup --buildtype release --optimization 3 --strip --prefix=/home/$USER
    ++#
    ++#      # Set up a build directory with 'address' and 'undefined' sanitizers
    ++#      # using Clang.
    ++#      $ CC=clang meson setup -Db_sanitize=address,undefined
    ++#
    ++#      # Disable tests in a preexisting build directory.
    ++#      $ meson configure -Dtests=false
    ++#
    ++#      # Disable features based on Python
    ++#      $ meson configure -Dpython=disabled
    ++#
    ++#      # Disable features based on Python
    ++#      $ meson configure -Dpython=disabled
    ++#
    ++# Options have a type like booleans, choices, strings or features. Features are
    ++# somewhat special as they can have one of three values: enabled, disabled or
    ++# auto. While the first two values are self-explanatory, "auto" will enable or
    ++# disable the feature based on the availability of prerequisites to support it.
    ++# Python-based features for example will be enabled automatically when a Python
    ++# interpreter could be found. The default value of such features can be changed
    ++# globally via `meson setup --auto-features={enabled,disabled,auto}`, which
    ++# will set the value of all features with a value of "auto" to the provided one
    ++# by default.
    ++#
    ++# It is also possible to store a set of configuration options in machine files.
    ++# This can be useful in case you regularly want to reuse the same set of options:
    ++#
    ++#   [binaries]
    ++#   c = ['clang']
    ++#   ar = ['ar']
    ++#
    ++#   [project options]
    ++#   gettext = 'disabled'
    ++#   default_editor = 'vim'
    ++#
    ++#   [built-in options]
    ++#   b_lto = true
    ++#   b_sanitize = 'address,undefined'
    ++#
    ++# These machine files can be passed to Meson via `meson setup --native-file`.
    ++#
    ++# Subproject wrappers
    ++# ===================
    ++#
    ++# Subproject wrappers are a feature provided by Meson that allow the automatic
    ++# fallback to a "wrapped" dependency in case the dependency is not provided by
    ++# the system. For example if the system is lacking curl, then Meson will use
    ++# "subprojects/curl.wrap" to set up curl as a subproject and compile and link
    ++# the dependency into Git itself. This is especially helpful on systems like
    ++# Windows, where you typically don't have such dependencies installed.
    ++#
    ++# The use of subproject wrappers can be disabled by executing `meson setup
    ++# --wrap-mode nofallback`.
    ++
     +project('git', 'c',
     +  meson_version: '>=1.3.0',
     +  # MSVC does not support GNU99, and C99 does not define __STDC_VERSION__
    @@ meson.build (new)
     +  build_options_config.set('NO_PTHREADS', '1')
     +endif
     +
    -+if get_option('gettext').allowed() and host_machine.system() == 'darwin' and get_option('macos_use_homebrew_gettext')
    ++msgfmt = find_program('msgfmt', dirs: program_path, required: false)
    ++gettext_option = get_option('gettext').disable_auto_if(not msgfmt.found())
    ++if not msgfmt.found() and gettext_option.enabled()
    ++  error('Internationalization via libintl requires msgfmt')
    ++endif
    ++
    ++if gettext_option.allowed() and host_machine.system() == 'darwin' and get_option('macos_use_homebrew_gettext')
     +  if host_machine.cpu_family() == 'x86_64'
     +    libintl_prefix = '/usr/local'
     +  elif host_machine.cpu_family() == 'aarch64'
    @@ meson.build (new)
     +    error('Homebrew workaround not supported on current architecture')
     +  endif
     +
    -+  intl = compiler.find_library('intl', dirs: libintl_prefix / 'lib', required: get_option('gettext'))
    ++  intl = compiler.find_library('intl', dirs: libintl_prefix / 'lib', required: gettext_option)
     +  if intl.found()
     +    intl = declare_dependency(
     +      dependencies: intl,
    @@ meson.build (new)
     +    )
     +  endif
     +else
    -+  intl = dependency('intl', required: get_option('gettext'))
    ++  intl = dependency('intl', required: gettext_option)
     +endif
     +if intl.found()
     +  libgit_dependencies += intl
19:  f483916825 ! 19:  8315d432f1 meson: fix conflicts with in-flight topics
    @@ Commit message
         conflicts with topics in "seen". The following conflicts are being
         addressed in this commit:
     
    -      - ej/cat-file-remote-object-info adds t1017.
    +      - ej/cat-file-remote-object-info adds t1017 and "fetch-object-info.c".
     
           - cc/promisor-remote-capability adds t5710.
     
           - ds/path-walk adds t6601 as well as "path-walk.c" and
             "test-path-walk.c".
     
    -      - am/git-blame-ignore-revs-by-default adds t8015 and t8016.
    -
           - ps/reftable-detach adds "reftable/system.c".
     
           - js/libgit-rust adds "common-exit.c" and "common-init.c".
     
    +      - ps/clar-build-improvement adapts the awk(1) script that generates
    +        "clar.suites" to instead be a shell script.
    +
    +      - ds/full-name-hash adds "t/helper/test-name-hash.c".
    +
    +      - cw/worktree-extension deletes t2408.
    +
         This is somewhat painful in the current state where Meson is not yet
         part of the main tree, but we'll have to live with that for the time
         being.
    @@ meson.build: libgit_sources = [
        'compat/nonblock.c',
        'compat/obstack.c',
        'compat/terminal.c',
    +@@ meson.build: libgit_sources = [
    +   'ewah/ewah_rlw.c',
    +   'exec-cmd.c',
    +   'fetch-negotiator.c',
    ++  'fetch-object-info.c',
    +   'fetch-pack.c',
    +   'fmt-merge-msg.c',
    +   'fsck.c',
     @@ meson.build: libgit_sources = [
        'parse-options.c',
        'patch-delta.c',
    @@ meson.build: libgit_sources = [
     
      ## t/helper/meson.build ##
     @@ t/helper/meson.build: test_tool_sources = [
    +   'test-match-trees.c',
    +   'test-mergesort.c',
    +   'test-mktemp.c',
    ++  'test-name-hash.c',
    +   'test-online-cpus.c',
    +   'test-pack-mtimes.c',
    +   'test-parse-options.c',
        'test-parse-pathspec-file.c',
        'test-partial-clone.c',
        'test-path-utils.c',
    @@ t/helper/meson.build: test_tool_sources = [
        'test-proc-receive.c',
     
      ## t/meson.build ##
    +@@ t/meson.build: clar_sources += clar_decls_h
    + clar_sources += custom_target(
    +   input: clar_decls_h,
    +   output: 'clar.suite',
    +-  feed: true,
    +-  capture: true,
    +-  command : [awk, '-f', meson.current_source_dir() + '/unit-tests/clar-generate.awk'],
    ++  command : [shell, meson.current_source_dir() + '/unit-tests/generate-clar-suites.sh', '@INPUT@', '@OUTPUT@'],
    ++  env: script_environment,
    + )
    + 
    + clar_unit_tests = executable('unit-tests',
     @@ t/meson.build: integration_tests = [
        't1014-read-tree-confusing.sh',
        't1015-read-index-unmerged.sh',
    @@ t/meson.build: integration_tests = [
        't1020-subdirectory.sh',
        't1021-rerere-in-workdir.sh',
        't1022-read-tree-partial-clone.sh',
    +@@ t/meson.build: integration_tests = [
    +   't2405-worktree-submodule.sh',
    +   't2406-worktree-repair.sh',
    +   't2407-worktree-heads.sh',
    +-  't2408-worktree-relative.sh',
    +   't2500-untracked-overwriting.sh',
    +   't2501-cwd-empty.sh',
    +   't3000-ls-files-others.sh',
     @@ t/meson.build: integration_tests = [
        't5703-upload-pack-ref-in-want.sh',
        't5704-protocol-violations.sh',
    @@ t/meson.build: integration_tests = [
        't6700-tree-depth.sh',
        't7001-mv.sh',
        't7002-mv-sparse-checkout.sh',
    -@@ t/meson.build: integration_tests = [
    -   't8012-blame-colors.sh',
    -   't8013-blame-ignore-revs.sh',
    -   't8014-blame-ignore-fuzzy.sh',
    -+  't8015-blame-default-ignore-revs.sh',
    -+  't8016-blame-override-ignore-revs.sh',
    -   't9001-send-email.sh',
    -   't9002-column.sh',
    -   't9003-help-autocorrect.sh',

---
base-commit: b31fb630c0fc6869a33ed717163e8a1210460d94
change-id: 20241112-pks-meson-95b665c08f06