Re: [PATCH] meson: distinguish build and target host binaries

[Peter Seiderer <ps.report@gmx.net>]

Hello Patrick,

On Mon, 03 Mar 2025 13:10:59 +0100, Patrick Steinhardt <ps@pks.im> wrote:

> Almost all of the tools we discover during the build process need to be
> native programs. There are only a handful of exceptions, which typically
> are programs whose paths we need to embed into the resulting executable
> so that they can be found on the target system when Git executes. While
> this distinction typically doesn't matter, it does start to matter when
> considering cross-compilation where the build and target machines are
> different.
>
> Meson supports cross-compilation via so-called machine files. These
> machine files allow the user to override parameters for the build
> machine, but also for the target machine when cross-compiling. Part of
> the machine file is a section that allows the user to override the
> location where binaries are to be found in the target system. The
> following machine file would for example override the path of the POSIX
> shell:
>
>     [binaries]
>     sh = '/usr/xpg4/bin/sh'
>
> It can be handed over to Meson via `meson setup --cross-file`.
>
> We do not handle this correctly right now though because we don't know
> to distinguish binaries for the build and target hosts at all. Address
> this by explicitly passing the `native:` parameter to `find_program()`:
>
>   - When set to `true`, we get binaries discovered on the build host.
>
>   - When set to `false`, we get either the path specified in the
>     machine file. Or, if no machine file exists or it doesn't specify
>     the binary path, then we fall back to the binary discovered on the
>     build host.
>
> As mentioned, only a handful of binaries are not native: only the system
> shell, Python and Perl need to be treated specially here.
>
> Signed-off-by: Patrick Steinhardt <ps@pks.im>
> ---
> Hi,
>
> this patch addresses the issue reported at [1], where it is impossible
> to specify the shell, Python and Perl paths during cross-compilation
> when using Meson.

I still believe that is is a 'misuse' of the cross-file (as stated already
here [1]) the given programs in cross-file are to be meant to run while
cross-compiling (at compile time) and not on the target (e.g. it would be
impossible to find a program (as the name find_program indicates) at
compile/configure where the target layout is yet unknown....

I believe the correct solution is an extra configure option for cross-compile
and a sane default (or find_program) in case of native build...

Regards,
Peter

[1] https://public-inbox.org/git/20250210122603.5130e309@gmx.net/

>
> The series is built on top of master at cb0ae672aea (A bit more post
> -rc0, 2025-02-27) with junio/ps/build-meson-fixes-0130 at 9350423982a
> (gitlab-ci: restrict maximum number of link jobs on Windows, 2025-02-26)
> merged into it.
>
> Thanks!
>
> Patrick
>
> [1]: <20250209133027.64a865aa@gmx.net>
> ---
>  Documentation/meson.build | 12 ++++-----
>  gitweb/meson.build        |  2 +-
>  meson.build               | 66 ++++++++++++++++++++++++++++++++++++-----------
>  templates/meson.build     |  4 +--
>  4 files changed, 60 insertions(+), 24 deletions(-)
>
> diff --git a/Documentation/meson.build b/Documentation/meson.build
> index 0a0f2bfa14a..63891fb4455 100644
> --- a/Documentation/meson.build
> +++ b/Documentation/meson.build
> @@ -207,9 +207,9 @@ manpages = {
>
>  docs_backend = get_option('docs_backend')
>  if docs_backend == 'auto'
> -  if find_program('asciidoc', dirs: program_path, required: false).found()
> +  if find_program('asciidoc', dirs: program_path, native: true, required: false).found()
>      docs_backend = 'asciidoc'
> -  elif find_program('asciidoctor', dirs: program_path, required: false).found()
> +  elif find_program('asciidoctor', dirs: program_path, native: true, required: false).found()
>      docs_backend = 'asciidoctor'
>    else
>      error('Neither asciidoc nor asciidoctor were found.')
> @@ -217,7 +217,7 @@ if docs_backend == 'auto'
>  endif
>
>  if docs_backend == 'asciidoc'
> -  asciidoc = find_program('asciidoc', dirs: program_path)
> +  asciidoc = find_program('asciidoc', dirs: program_path, native: true)
>    asciidoc_html = 'xhtml11'
>    asciidoc_docbook = 'docbook'
>    xmlto_extra = [ ]
> @@ -246,7 +246,7 @@ if docs_backend == 'asciidoc'
>      asciidoc_conf,
>    ]
>  elif docs_backend == 'asciidoctor'
> -  asciidoctor = find_program('asciidoctor', dirs: program_path)
> +  asciidoctor = find_program('asciidoctor', dirs: program_path, native: true)
>    asciidoc_html = 'xhtml5'
>    asciidoc_docbook = 'docbook5'
>    xmlto_extra = [
> @@ -284,7 +284,7 @@ elif docs_backend == 'asciidoctor'
>    ]
>  endif
>
> -xmlto = find_program('xmlto', dirs: program_path)
> +xmlto = find_program('xmlto', dirs: program_path, native: true)
>
>  cmd_lists = [
>    'cmds-ancillaryinterrogators.adoc',
> @@ -405,7 +405,7 @@ if get_option('docs').contains('html')
>      pointing_to: 'git.html',
>    )
>
> -  xsltproc = find_program('xsltproc', dirs: program_path)
> +  xsltproc = find_program('xsltproc', dirs: program_path, native: true)
>
>    user_manual_xml = custom_target(
>      command: asciidoc_common_options + [
> diff --git a/gitweb/meson.build b/gitweb/meson.build
> index 89b403dc9de..88a54b4dc99 100644
> --- a/gitweb/meson.build
> +++ b/gitweb/meson.build
> @@ -1,5 +1,5 @@
>  gitweb_config = configuration_data()
> -gitweb_config.set_quoted('PERL_PATH', perl.full_path())
> +gitweb_config.set_quoted('PERL_PATH', target_perl.full_path())
>  gitweb_config.set_quoted('CSSMIN', '')
>  gitweb_config.set_quoted('JSMIN', '')
>  gitweb_config.set_quoted('GIT_BINDIR', get_option('prefix') / get_option('bindir'))
> diff --git a/meson.build b/meson.build
> index 516207f9cfe..0a229f44199 100644
> --- a/meson.build
> +++ b/meson.build
> @@ -155,6 +155,37 @@
>  # These machine files can be passed to `meson setup` via the `--native-file`
>  # option.
>  #
> +# Cross compilation
> +# =================
> +#
> +# Machine files can also be used in the context of cross-compilation to
> +# describe the target machine as well as the cross-compiler toolchain that
> +# shall be used. An example machine file could look like the following:
> +#
> +#   [binaries]
> +#   c = 'x86_64-w64-mingw32-gcc'
> +#   cpp = 'x86_64-w64-mingw32-g++'
> +#   ar = 'x86_64-w64-mingw32-ar'
> +#   windres = 'x86_64-w64-mingw32-windres'
> +#   strip = 'x86_64-w64-mingw32-strip'
> +#   exe_wrapper = 'wine64'
> +#   sh = 'C:/Program Files/Git for Windows/usr/bin/sh.exe'
> +#
> +#   [host_machine]
> +#   system = 'windows'
> +#   cpu_family = 'x86_64'
> +#   cpu = 'x86_64'
> +#   endian = 'little'
> +#
> +# These machine files can be passed to `meson setup` via the `--cross-file`
> +# option.
> +#
> +# Note that next to the cross-compiler toolchain, the `[binaries]` section is
> +# also used to locate a couple of binaries that will be built into Git. This
> +# includes `sh`, `python` and `perl`, so when cross-compiling Git you likely
> +# want to set these binary paths in addition to the cross-compiler toolchain
> +# binaries.
> +#
>  # Subproject wrappers
>  # ===================
>  #
> @@ -173,7 +204,7 @@ project('git', 'c',
>    # The version is only of cosmetic nature, so if we cannot find a shell yet we
>    # simply don't set up a version at all. This may be the case for example on
>    # Windows systems, where we first have to bootstrap the host environment.
> -  version: find_program('sh', required: false).found() ? run_command(
> +  version: find_program('sh', native: true, required: false).found() ? run_command(
>      'GIT-VERSION-GEN', meson.current_source_dir(), '--format=@GIT_VERSION@',
>      capture: true,
>      check: true,
> @@ -198,16 +229,18 @@ elif host_machine.system() == 'windows'
>    program_path = [ 'C:/Program Files/Git/bin', 'C:/Program Files/Git/usr/bin' ]
>  endif
>
> -cygpath = find_program('cygpath', dirs: program_path, required: false)
> -diff = find_program('diff', dirs: program_path)
> -git = find_program('git', dirs: program_path, required: false)
> -sed = find_program('sed', dirs: program_path)
> -shell = find_program('sh', dirs: program_path)
> -tar = find_program('tar', dirs: program_path)
> +cygpath = find_program('cygpath', dirs: program_path, native: true, required: false)
> +diff = find_program('diff', dirs: program_path, native: true)
> +git = find_program('git', dirs: program_path, native: true, required: false)
> +sed = find_program('sed', dirs: program_path, native: true)
> +shell = find_program('sh', dirs: program_path, native: true)
> +tar = find_program('tar', dirs: program_path, native: true)
> +
> +target_shell = find_program('sh', dirs: program_path, native: false)
>
>  # Sanity-check that programs required for the build exist.
>  foreach tool : ['cat', 'cut', 'grep', 'sort', 'tr', 'uname']
> -  find_program(tool, dirs: program_path)
> +  find_program(tool, dirs: program_path, native: true)
>  endforeach
>
>  script_environment = environment()
> @@ -706,7 +739,7 @@ libgit_c_args = [
>    '-DGIT_LOCALE_PATH="' + get_option('localedir') + '"',
>    '-DGIT_MAN_PATH="' + get_option('mandir') + '"',
>    '-DPAGER_ENV="' + get_option('pager_environment') + '"',
> -  '-DSHELL_PATH="' + fs.as_posix(shell.full_path()) + '"',
> +  '-DSHELL_PATH="' + fs.as_posix(target_shell.full_path()) + '"',
>  ]
>  libgit_include_directories = [ '.' ]
>  libgit_dependencies = [ ]
> @@ -761,6 +794,7 @@ endif
>  build_options_config.set_quoted('X', executable_suffix)
>
>  python = import('python').find_installation('python3', required: get_option('python'))
> +target_python = find_program('python3', native: false, required: python.found())
>  if python.found()
>    build_options_config.set('NO_PYTHON', '')
>  else
> @@ -790,9 +824,11 @@ endif
>  # which we can do starting with Meson 1.5.0 and newer, or we have to
>  # match against the minor version.
>  if meson.version().version_compare('>=1.5.0')
> -  perl = find_program('perl', dirs: program_path, required: perl_required, version: '>=5.26.0', version_argument: '-V:version')
> +  perl = find_program('perl', dirs: program_path, native: true, required: perl_required, version: '>=5.26.0', version_argument: '-V:version')
> +  target_perl = find_program('perl', dirs: program_path, native: false, required: perl.found(), version: '>=5.26.0', version_argument: '-V:version')
>  else
> -  perl = find_program('perl', dirs: program_path, required: perl_required, version: '>=26')
> +  perl = find_program('perl', dirs: program_path, native: true, required: perl_required, version: '>=26')
> +  target_perl = find_program('perl', dirs: program_path, native: false, required: perl.found(), version: '>=26')
>  endif
>  perl_features_enabled = perl.found() and get_option('perl').allowed()
>  if perl_features_enabled
> @@ -843,7 +879,7 @@ else
>    build_options_config.set('NO_PTHREADS', '1')
>  endif
>
> -msgfmt = find_program('msgfmt', dirs: program_path, required: false)
> +msgfmt = find_program('msgfmt', dirs: program_path, native: true, required: false)
>  gettext_option = get_option('gettext').disable_auto_if(not msgfmt.found())
>  if not msgfmt.found() and gettext_option.enabled()
>    error('Internationalization via libintl requires msgfmt')
> @@ -1975,9 +2011,9 @@ foreach key, value : {
>    'GIT_TEST_TEMPLATE_DIR': meson.project_build_root() / 'templates',
>    'GIT_TEST_TEXTDOMAINDIR': meson.project_build_root() / 'po',
>    'PAGER_ENV': get_option('pager_environment'),
> -  'PERL_PATH': perl.found() ? perl.full_path() : '',
> -  'PYTHON_PATH': python.found () ? python.full_path() : '',
> -  'SHELL_PATH': shell.full_path(),
> +  'PERL_PATH': target_perl.found() ? target_perl.full_path() : '',
> +  'PYTHON_PATH': target_python.found () ? target_python.full_path() : '',
> +  'SHELL_PATH': target_shell.full_path(),
>    'TAR': tar.full_path(),
>    'TEST_OUTPUT_DIRECTORY': test_output_directory,
>    'TEST_SHELL_PATH': shell.full_path(),
> diff --git a/templates/meson.build b/templates/meson.build
> index 1faf9a44cea..02e6eebe80b 100644
> --- a/templates/meson.build
> +++ b/templates/meson.build
> @@ -1,6 +1,6 @@
>  template_config = configuration_data()
> -template_config.set('PERL_PATH', perl.found() ? fs.as_posix(perl.full_path()) : '')
> -template_config.set('SHELL_PATH', fs.as_posix(shell.full_path()))
> +template_config.set('PERL_PATH', target_perl.found() ? fs.as_posix(target_perl.full_path()) : '')
> +template_config.set('SHELL_PATH', fs.as_posix(target_shell.full_path()))
>  template_config.set('GITWEBDIR', fs.as_posix(get_option('prefix') / get_option('datadir') / 'gitweb'))
>
>  configure_file(
>
> ---
> base-commit: af208620eecbe9b4655e06a28d5146a40738150e
> change-id: 20250303-pks-meson-cross-compiling-7bad3078b9e8
>