Re: [PATCH] meson: port python bindings to build natively via meson and meson-python

[David Gibson <david@gibson.dropbear.id.au>]

[-- Attachment #1: Type: text/plain, Size: 7964 bytes --]

Sorry, I haven't replied to this for ages.  I've had a lot of stuff
going on, both at work and otherwise.

On Wed, Mar 19, 2025 at 02:30:17AM -0400, Eli Schwartz wrote:
> On 3/19/25 1:55 AM, David Gibson wrote:
> 
> >> diff --git a/MANIFEST.in b/MANIFEST.in
> >> deleted file mode 100644
> >> index 904d124..0000000
> >> --- a/MANIFEST.in
> >> +++ /dev/null
> >> @@ -1,12 +0,0 @@
> >> -# SPDX-License-Identifier: (GPL-2.0-or-later OR BSD-2-Clause)
> >> -
> >> -global-exclude *
> >> -include README.md
> >> -include GPL
> >> -include BSD-2-Clause
> >> -include setup.py
> >> -include pylibfdt/libfdt.i
> >> -include libfdt/libfdt.h
> >> -include libfdt/fdt.h
> >> -include libfdt/libfdt_env.h
> >> -include VERSION.txt
> > 
> > We can't just delete the MANIFEST.in and setup.py, for now, because
> > they're still needed for the Makefile build.  I'm certainly
> > considering dropping support for make, but we're not there yet.
> 
> If, as discussed in the previous emails, we drop support for *building
> python from Make* but don't drop Make for C, is that acceptable?

I'm conflicted.  I don't really like the idea of making make work for
only some things, without just removing it completely, or at least
having a concrete plan to do so.  On the other hand, I'd really like
to get rid of the confusing setuptools gunk.

How about this: could you author a patch deprecating make, which
prints a message saying so if you build with make (anything, not just
the Python pieces).  It should say that meson is preferred and give
example meson commands to use instead.

If we start with that, then I'd be ok making the Python build meson
only, followed by the changes you're suggesting here.

> >> diff --git a/libfdt/meson.build b/libfdt/meson.build
> >> index c2f4bd6..9c6ef54 100644
> >> --- a/libfdt/meson.build
> >> +++ b/libfdt/meson.build
> >> @@ -31,7 +31,7 @@ libfdt = library(
> >>    version: meson.project_version(),
> >>    link_args: link_args,
> >>    link_depends: 'version.lds',
> >> -  install: true,
> >> +  install: get_option('default_library') != 'static' or not wheel_only,
> > 
> > The first clause doesn't quite make sense to me.  Why would we not
> > install a static library?
> 
> 
> Read this the other way around.
> 
> We *do* install, if either (or both) conditions are true:
> 
> - we are building a non-static library, so we certainly need it at
>   runtime (thus it has to be installed at runtime)
> 
> - we are NOT building a special wheel-specific build, which is to say,
>   we assume we need this installed if we aren't running from pip install
> 
> 
> i.e. the purpose of this is that we assume the end goal for python
> wheels is as I'm going to describe in the latter half of this email, and
> this specific conditional over here is just trying to detect "PyPI mode".
> 
> 
> >> +[tool.meson-python.args]
> >> +'setup' = ['-Ddefault_library=static', '-Dwheel-only=true']
> > 
> > Shouldn't the python library use the shared library, at least by
> > default?
> 
> 
> This isn't about building the python library. If I'm building the python
> library, I just use "meson setup" like normal, and it builds the full
> dtc stack, including tools, library, python bindings and all.
> 
> meson-python is *specifically* about building a python wheel. Python
> wheels have two choices:
> 
> - the shared library is copied into the wheel
> - don't use the shared library
> 
> Without setting these options, a wheel uploaded to PyPI ends up looking
> like this:
> 
> $ bsdtar -tf dist/libfdt-1.7.2-cp312-cp312-linux_x86_64.whl
> libfdt-1.7.2.dist-info/METADATA
> libfdt-1.7.2.dist-info/WHEEL
> .libfdt.mesonpy.libs/libfdt.so.1.7.2
> .libfdt.mesonpy.libs/pkgconfig/libfdt.pc
> libfdt-1.7.2.data/scripts/convert-dtsv0
> libfdt-1.7.2.data/scripts/dtc
> libfdt-1.7.2.data/scripts/fdtdump
> libfdt-1.7.2.data/scripts/fdtget
> libfdt-1.7.2.data/scripts/fdtput
> libfdt-1.7.2.data/scripts/fdtoverlay
> libfdt-1.7.2.data/scripts/dtdiff
> libfdt.py
> _libfdt.cpython-312-x86_64-linux-gnu.so
> libfdt-1.7.2.data/headers/fdt.h
> libfdt-1.7.2.data/headers/libfdt.h
> libfdt-1.7.2.data/headers/libfdt_env.h
> libfdt-1.7.2.dist-info/RECORD
> 
> And,
> 
> $ readelf -d _libfdt.cpython-312-x86_64-linux-gnu.so
> 
>     Library runpath: [$ORIGIN/.libfdt.mesonpy.libs]
> 
> This is all necessary since pip install can't know that libfdt is
> already installed via a package manager (dpkg, rpm, portage, pacman,
> xbps, apk, etc), so it has to ship its own just in case.
> 
> Building statically simplifies distribution as a PyPI wheel.

Ah, thanks for the explanation.  Thank makes sense.  If you can word
something sufficiently succinct, a comment for posterity explaining
that reasoning would be nice.  But if you can't find a way to explain
that briefly, I won't insist.

> If you really want to build bloated wheels, you can still do it:
> 
> 
> python -m build --wheel \
>     -Csetup-args=-Ddefault_library=static \
>     -Csetup-args=wheel-only=false
> 
> 
> 
> By the way, the locations that the files get installed to when building
> via pip aren't great either. The *.data/headers/ gets installed to
> 
> /usr/include/python3.12/libfdt/
> 
> and the library and pkg-config files end up in
> 
> /usr/lib/python3.12/site-packages/.libfdt.mesonpy.libs/libfdt.so.1.7.2
> /usr/lib/python3.12/site-packages/.libfdt.mesonpy.libs/pkgconfig/libfdt.pc
> 
> The pkg-config file doesn't even get remapped, it still says
> -I/usr/include and -L/usr/lib64
> 
> If you install with pip install --user (default when not running with
> sudo), then you even end up with this:
> 
> 
>     /home/eschwartz/.local/bin/convert-dtsv0
>     /home/eschwartz/.local/bin/dtc
>     /home/eschwartz/.local/bin/dtdiff
>     /home/eschwartz/.local/bin/fdtdump
>     /home/eschwartz/.local/bin/fdtget
>     /home/eschwartz/.local/bin/fdtoverlay
>     /home/eschwartz/.local/bin/fdtput
>     /home/eschwartz/.local/include/python3.12/libfdt/fdt.h
>     /home/eschwartz/.local/include/python3.12/libfdt/libfdt.h
>     /home/eschwartz/.local/include/python3.12/libfdt/libfdt_env.h
> 
> /home/eschwartz/.local/lib/python3.12/site-packages/.libfdt.mesonpy.libs/libfdt.so.1.7.2
> 
> /home/eschwartz/.local/lib/python3.12/site-packages/.libfdt.mesonpy.libs/pkgconfig/libfdt.pc
> 
> /home/eschwartz/.local/lib/python3.12/site-packages/_libfdt.cpython-312-x86_64-linux-gnu.so
>     /home/eschwartz/.local/lib/python3.12/site-packages/libfdt.py
> 
> 
> So yeah, IMO all of this cruft is out of place for pypi.org / pip.
> 
> 
> Python packaging *cannot*, in any way shape or form be used outside of
> its closed-in silo. They literally forbid, via their standards body, the
> possibility of installing outside of the locations I described. The only
> way to install to arbitrary locations the way Make or Meson can, is to
> use what they describe as "the legacy install route" of running
> 
> ```
> python setup.py install
> ```
> 
> and then having the equivalent of Makefile
> 
> `cp XXXX ${DESTDIR}${PREFIX}/bin`
> 
> i.e. completely disrespecting the pip install approach.
> 
> 
> The point of distributing on PyPI is so that people can get a
> python-specific copy of the bindings easily, with a minimum of fuss, and
> nothing else. You can always get the complete set using a real build
> system and installing with "meson install" instead of pip, and usually
> using a distro package manager even. But people installing via pip
> install aren't necessarily interested in that. They just want to
> download a binary package which their python scripts depend on.

Ok, understood.

-- 
David Gibson (he or they)	| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you, not the other way
				| around.
http://www.ozlabs.org/~dgibson

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]