[libgpiod][PATCH v2 0/1] bindings: python: standalone build tooling for tests

[libgpiod][PATCH v2 0/1] bindings: python: standalone build tooling for tests

[Phil Howard]

After much delibration about how in-place test builds should work, I have
pulled out all of the test extension code from setup.py into a new
build_tests.py and updated Makefile.am to invoke it.

This replaces the old `python3 setup.py build_ext --inplace` with enough
setuptools tooling to replicate that behaviour, while also putting it
in a better place to avoid problems arising from the deprecation of direct
setup.py invocation [1].

Since receipients of the gpiod Python bindings (in tar.gz source or .whl 
binary format) won't need the test C extensions (and can't run them anyway
without the accompanying Python modules) the test extension code has been
removed from setup.py rather than duplicated into this new file.

The new build_tests.py creates a temporary directory and redirects all
build-time output into it. A final step - handled by build_ext and internal
to setuptools - then moves the built _ext.<python_version>-<arch>-<os>.so C
extensions into gpiod, tests/gpiosim and tests/procname where they can be
imported.

Since release packages and tests no longer share the build/ directory I
have removed the "rmtree" hack from the "build_ext" override in setup.py.
There should be no cases where test extensions can accidentally end up
being installed into the system/user Python environment.

To build and run tests:
- `make python-tests`
- `(sudo) make python-tests-run` (I need root for the gpiosim module)

Tested with setuptools==68.2.2 (Sep 2023), setuptools==59.8.0 (Dec 2021)
and setuptools==42.0.2 (Dec 2019).

Changes v1 -> v2:
- Add the gpiod ext module to build_tests.py (using glob for sources)
- Formalize the command for running tests by adding it to Makefile.am
- Add build_tests.py output artefacts to Makefile.am's clean-local target
- Sort distutils.log imports (DEBUG before set_verbosity)

Phil Howard (1):
  bindings: python: standalone build tooling for tests

 bindings/python/Makefile.am    | 33 +++++++++---
 bindings/python/build_tests.py | 95 ++++++++++++++++++++++++++++++++++
 bindings/python/setup.py       | 28 +---------
 3 files changed, 121 insertions(+), 35 deletions(-)
 create mode 100644 bindings/python/build_tests.py

-- 
2.34.1

[libgpiod][PATCH v2 1/1] bindings: python: standalone build tooling for tests

[Phil Howard]

Move extension definitions and tooling for building tests into
`build_tests.py` and update Makefile.am to call it with appropriate path
prefixes.

`build_tests.py` will perform a standalone build of the text extensions,
keeping any build noise in a temporary directory and copying the final
built modules automatically out to `tests/gpiosim` and `tests/procname`.

Add "python-tests-run" to Makefile.am so it's clear how to run the tests.

Add .so object files generated by build_test.py to Makefile.am's
clean-local.

Signed-off-by: Phil Howard <phil@gadgetoid.com>
---
 bindings/python/Makefile.am    | 33 +++++++++---
 bindings/python/build_tests.py | 95 ++++++++++++++++++++++++++++++++++
 bindings/python/setup.py       | 28 +---------
 3 files changed, 121 insertions(+), 35 deletions(-)
 create mode 100644 bindings/python/build_tests.py

diff --git a/bindings/python/Makefile.am b/bindings/python/Makefile.am
index 079ceb1..389da9d 100644
--- a/bindings/python/Makefile.am
+++ b/bindings/python/Makefile.am
@@ -7,19 +7,36 @@ EXTRA_DIST = \
 
 if WITH_TESTS
 
-BUILD_TESTS = 1
+python-tests:
+	TOP_SRCDIR=$(abs_top_builddir) \
+	TOP_BUILDDIR=$(abs_top_builddir) \
+	$(PYTHON) build_tests.py
+
+python-tests-run:
+	PYTHONPATH=$(abs_top_srcdir)/bindings/python
+	LD_LIBRARY_PATH=$(abs_top_builddir)/lib/.libs/:\
+$(abs_top_builddir)/tests/gpiosim/.libs/ \
+    python3 -B -m tests
+
+else
+
+python-tests:
 
 endif
 
-all-local:
-	GPIOD_WITH_TESTS=$(BUILD_TESTS) \
-	$(PYTHON) setup.py build_ext --inplace \
-		--include-dirs=$(top_srcdir)/include/:$(top_srcdir)/tests/gpiosim/ \
-		--library-dirs=$(top_builddir)/lib/.libs/:$(top_srcdir)/tests/gpiosim/.libs/
+clean-local:
+	rm -rf dist
+	rm -f gpiod/_ext.*.so
+	rm -f tests/*/_ext.*.so
+
+all-local: python-tests
+	CFLAGS="-I$(abs_top_srcdir)/include/ -I$(abs_top_srcdir)/tests/gpiosim \
+	-L$(abs_top_builddir)/lib/.libs/ -L$(abs_top_builddir)/tests/gpiosim/.libs/" \
+	$(PYTHON) -m build
 
 install-exec-local:
-	GPIOD_WITH_TESTS= \
-	$(PYTHON) setup.py install --prefix=$(DESTDIR)$(prefix)
+	$(PYTHON) -m pip install dist/*.whl \
+	--prefix=$(DESTDIR)$(prefix)
 
 SUBDIRS = gpiod
 
diff --git a/bindings/python/build_tests.py b/bindings/python/build_tests.py
new file mode 100644
index 0000000..1760257
--- /dev/null
+++ b/bindings/python/build_tests.py
@@ -0,0 +1,95 @@
+# SPDX-License-Identifier: GPL-2.0-or-later
+# SPDX-FileCopyrightText: 2023 Phil Howard <phil@gadgetoid.com>
+
+"""
+Bring up just enough of setuptools/distutils in order to build the gpiod
+test module C extensions.
+
+Set "build_temp" and "build_lib" so that our source directory is not
+polluted with artefacts in build/
+
+Builds:
+
+    tests/gpiosim/_ext.<target>.so
+    tests/procname/_ext.<target>.so
+
+"""
+
+import glob
+import tempfile
+from os import getenv, path
+
+from setuptools import Distribution, Extension
+from setuptools.command.build_ext import build_ext
+
+TOP_SRCDIR = getenv("TOP_SRCDIR", "../../")
+TOP_BUILDDIR = getenv("TOP_BUILDDIR", "../../")
+
+# __version__
+with open("gpiod/version.py", "r") as fd:
+    exec(fd.read())
+
+# The tests are run in-place with PYTHONPATH set to bindings/python
+# so we need the gpiod extension module too.
+gpiod_ext = Extension(
+    "gpiod._ext",
+    sources=glob.glob("gpiod/ext/*.c"),
+    define_macros=[("_GNU_SOURCE", "1")],
+    libraries=["gpiod"],
+    extra_compile_args=["-Wall", "-Wextra"],
+    include_dirs=[
+        path.join(TOP_SRCDIR, "include"),
+    ],
+    library_dirs=[
+        path.join(TOP_BUILDDIR, "lib/.libs"),
+    ],
+)
+
+gpiosim_ext = Extension(
+    "tests.gpiosim._ext",
+    sources=["tests/gpiosim/ext.c"],
+    define_macros=[("_GNU_SOURCE", "1")],
+    libraries=["gpiosim"],
+    extra_compile_args=["-Wall", "-Wextra"],
+    include_dirs=[
+        path.join(TOP_SRCDIR, "include"),
+        path.join(TOP_SRCDIR, "tests/gpiosim"),
+    ],
+    library_dirs=[
+        path.join(TOP_BUILDDIR, "lib/.libs"),
+        path.join(TOP_BUILDDIR, "tests/gpiosim/.libs"),
+    ],
+)
+
+procname_ext = Extension(
+    "tests.procname._ext",
+    sources=["tests/procname/ext.c"],
+    define_macros=[("_GNU_SOURCE", "1")],
+    extra_compile_args=["-Wall", "-Wextra"],
+)
+
+dist = Distribution(
+    {
+        "name": "gpiod",
+        "ext_modules": [gpiosim_ext, procname_ext, gpiod_ext],
+        "version": __version__,
+        "platforms": ["linux"],
+    }
+)
+
+try:
+    from setuptools.logging import configure
+
+    configure()
+except ImportError:
+    from distutils.log import DEBUG, set_verbosity
+
+    set_verbosity(DEBUG)
+
+with tempfile.TemporaryDirectory(prefix="libgpiod-") as temp_dir:
+    command = build_ext(dist)
+    command.inplace = True
+    command.build_temp = temp_dir
+    command.build_lib = temp_dir
+    command.finalize_options()
+    command.run()
diff --git a/bindings/python/setup.py b/bindings/python/setup.py
index e8704d5..9607a28 100644
--- a/bindings/python/setup.py
+++ b/bindings/python/setup.py
@@ -13,7 +13,6 @@ from setuptools.errors import BaseError
 LINK_SYSTEM_LIBGPIOD = getenv("LINK_SYSTEM_LIBGPIOD") == "1"
 LIBGPIOD_MINIMUM_VERSION = "2.1"
 LIBGPIOD_VERSION = getenv("LIBGPIOD_VERSION")
-GPIOD_WITH_TESTS = getenv("GPIOD_WITH_TESTS") == "1"
 SRC_BASE_URL = "https://mirrors.edge.kernel.org/pub/software/libs/libgpiod/"
 TAR_FILENAME = "libgpiod-{version}.tar.gz"
 ASC_FILENAME = "sha256sums.asc"
@@ -189,11 +188,6 @@ class build_ext(orig_build_ext):
 
         super().run()
 
-        # We don't ever want the module tests directory in our package
-        # since this might include gpiosim._ext or procname._ext from a
-        # previous dirty build tree.
-        rmtree(path.join(self.build_lib, "tests"), ignore_errors=True)
-
 
 class sdist(orig_sdist):
     """
@@ -226,32 +220,12 @@ gpiod_ext = Extension(
     extra_compile_args=["-Wall", "-Wextra"],
 )
 
-gpiosim_ext = Extension(
-    "tests.gpiosim._ext",
-    sources=["tests/gpiosim/ext.c"],
-    define_macros=[("_GNU_SOURCE", "1")],
-    libraries=["gpiosim"],
-    extra_compile_args=["-Wall", "-Wextra"],
-)
-
-procname_ext = Extension(
-    "tests.procname._ext",
-    sources=["tests/procname/ext.c"],
-    define_macros=[("_GNU_SOURCE", "1")],
-    extra_compile_args=["-Wall", "-Wextra"],
-)
-
-extensions = [gpiod_ext]
-if GPIOD_WITH_TESTS:
-    extensions.append(gpiosim_ext)
-    extensions.append(procname_ext)
-
 setup(
     name="gpiod",
     url="https://git.kernel.org/pub/scm/libs/libgpiod/libgpiod.git",
     packages=find_packages(exclude=["tests", "tests.*"]),
     python_requires=">=3.9.0",
-    ext_modules=extensions,
+    ext_modules=[gpiod_ext],
     cmdclass={"build_ext": build_ext, "sdist": sdist},
     version=__version__,
     author="Bartosz Golaszewski",
-- 
2.34.1

Re: [libgpiod][PATCH v2 1/1] bindings: python: standalone build tooling for tests

[Bartosz Golaszewski]

On Wed, Nov 8, 2023 at 4:52 PM Phil Howard <phil@gadgetoid.com> wrote:
>
> Move extension definitions and tooling for building tests into
> `build_tests.py` and update Makefile.am to call it with appropriate path
> prefixes.
>
> `build_tests.py` will perform a standalone build of the text extensions,
> keeping any build noise in a temporary directory and copying the final
> built modules automatically out to `tests/gpiosim` and `tests/procname`.
>
> Add "python-tests-run" to Makefile.am so it's clear how to run the tests.
>
> Add .so object files generated by build_test.py to Makefile.am's
> clean-local.
>
> Signed-off-by: Phil Howard <phil@gadgetoid.com>
> ---

Thanks, this works quite nicely. I applied it and will release python
bindings v2.1.2 to pypi.

Bart