[libgpiod][PATCH 0/3] bindings: python: optionally include module (...)

[libgpiod][PATCH 0/3] bindings: python: optionally include module (...)

[Phil Howard]

This changeset vendors the gpiod library into the Python package, adds
a pyproject.toml for minimum compatibility with modern Python packaging
standards and sets "python_requires" to 3.9, reflecting the minimum version
required by these bindings as established in commit <b436d05809b1> ("bindings: 
python: replace PyModule_AddObjectRef() with PyModule_AddObject()").

Why?

So that setup.py can produce an sdist that is installable irrespective of the
availability or version of a distro-supplied libgpiod.

This prevents a libgpiod pypi package install balking because the distro
libgpiod is outdated or otherwise incompatible. This happens when attempting to
install the current libgpiod from pypi onto - for example - the Debian Bookworm
based Raspberry Pi OS.

The availability of a distro agnostic package also ensures that libgpiod can be
installed via pypi into an isolated virtual environment, safely specified as a
dependency for Python packages and allows Python developers to target the newest
API version irrespective of their distro supplied libgpiod.

This is essential, since a venv is now widely *required* for user Python
projects due to recommendations in pep-688 - https://peps.python.org/pep-0668/

For Raspberry Pi this sdist can also be converted into a precompiled wheel by
piwheels [1] which is, by default, added to Raspberry Pi OS as a pip index.

How?

If "USE_SYSTEM_GPIOD=1" is not specified then the gpiod._ext C Extension is
amended to include all of the C sources for gpiod, so it can be built as a
standalone module without depending upon a shared distro library.

The gpiod sources are included by symlinking the lib and include directories up
to the parent module, and updating MANIFEST.in to include the source files when
an sdist is built.

The resulting source distribution can then be uploaded to pypi and from there
can be built and installed by any user with python3-dev installed.

[1] - https://www.piwheels.org/

Phil Howard (3):
  bindings: python: optionally include module in sdist
  bindings: python: add pyproject.toml, pep 518
  bindings: python: require python 3.9.0

 bindings/python/MANIFEST.in    |  4 +++
 bindings/python/include        |  1 +
 bindings/python/lib            |  1 +
 bindings/python/pyproject.toml |  2 ++
 bindings/python/setup.py       | 58 ++++++++++++++++++++++++++--------
 5 files changed, 53 insertions(+), 13 deletions(-)
 create mode 120000 bindings/python/include
 create mode 120000 bindings/python/lib
 create mode 100644 bindings/python/pyproject.toml

-- 
2.34.1

[libgpiod][PATCH 1/3] bindings: python: optionally include module in sdist

[Phil Howard]

Build gpiod into Python module.

Optional environment var USE_SYSTEM_GPIO=1 to
generate a module that depends upon system gpiod.

Signed-off-by: Phil Howard <phil@gadgetoid.com>
---
 bindings/python/MANIFEST.in |  4 +++
 bindings/python/include     |  1 +
 bindings/python/lib         |  1 +
 bindings/python/setup.py    | 57 ++++++++++++++++++++++++++++---------
 4 files changed, 50 insertions(+), 13 deletions(-)
 create mode 120000 bindings/python/include
 create mode 120000 bindings/python/lib

diff --git a/bindings/python/MANIFEST.in b/bindings/python/MANIFEST.in
index c7124d4..eff8977 100644
--- a/bindings/python/MANIFEST.in
+++ b/bindings/python/MANIFEST.in
@@ -11,3 +11,7 @@ recursive-include gpiod/ext *.h
 
 recursive-include tests/gpiosim *.c
 recursive-include tests/procname *.c
+
+recursive-include lib *.c
+recursive-include lib *.h
+recursive-include include *.h
diff --git a/bindings/python/include b/bindings/python/include
new file mode 120000
index 0000000..fcffffb
--- /dev/null
+++ b/bindings/python/include
@@ -0,0 +1 @@
+../../include
\ No newline at end of file
diff --git a/bindings/python/lib b/bindings/python/lib
new file mode 120000
index 0000000..58677dd
--- /dev/null
+++ b/bindings/python/lib
@@ -0,0 +1 @@
+../../lib
\ No newline at end of file
diff --git a/bindings/python/setup.py b/bindings/python/setup.py
index 66b7908..2e25981 100644
--- a/bindings/python/setup.py
+++ b/bindings/python/setup.py
@@ -19,19 +19,53 @@ class build_ext(orig_build_ext):
         rmtree(path.join(self.build_lib, "tests"), ignore_errors=True)
 
 
+with open("gpiod/version.py", "r") as fd:
+    exec(fd.read())
+
+
+sources = [
+    # gpiod Python bindings
+    "gpiod/ext/chip.c",
+    "gpiod/ext/common.c",
+    "gpiod/ext/line-config.c",
+    "gpiod/ext/line-settings.c",
+    "gpiod/ext/module.c",
+    "gpiod/ext/request.c",
+]
+
+if "USE_SYSTEM_GPIOD" in environ and environ["USE_SYSTEM_GPIOD"] == "1":
+    libraries = ["gpiod"]
+    include_dirs = ["gpiod"]
+else:
+    sources += [
+        # gpiod library
+        "lib/chip.c",
+        "lib/chip-info.c",
+        "lib/edge-event.c",
+        "lib/info-event.c",
+        "lib/internal.c",
+        "lib/line-config.c",
+        "lib/line-info.c",
+        "lib/line-request.c",
+        "lib/line-settings.c",
+        "lib/misc.c",
+        "lib/request-config.c",
+    ]
+    libraries = []
+    include_dirs = ["include", "lib", "gpiod/ext"]
+
+
 gpiod_ext = Extension(
     "gpiod._ext",
-    sources=[
-        "gpiod/ext/chip.c",
-        "gpiod/ext/common.c",
-        "gpiod/ext/line-config.c",
-        "gpiod/ext/line-settings.c",
-        "gpiod/ext/module.c",
-        "gpiod/ext/request.c",
-    ],
+    libraries=libraries,
+    sources=sources,
     define_macros=[("_GNU_SOURCE", "1")],
-    libraries=["gpiod"],
-    extra_compile_args=["-Wall", "-Wextra"],
+    include_dirs=include_dirs,
+    extra_compile_args=[
+        "-Wall",
+        "-Wextra",
+        '-DGPIOD_VERSION_STR="{}"'.format(__version__),
+    ],
 )
 
 gpiosim_ext = Extension(
@@ -54,9 +88,6 @@ if "GPIOD_WITH_TESTS" in environ and environ["GPIOD_WITH_TESTS"] == "1":
     extensions.append(gpiosim_ext)
     extensions.append(procname_ext)
 
-with open("gpiod/version.py", "r") as fd:
-    exec(fd.read())
-
 setup(
     name="libgpiod",
     packages=find_packages(exclude=["tests", "tests.*"]),
-- 
2.34.1

Re: [libgpiod][PATCH 1/3] bindings: python: optionally include module in sdist

[Andy Shevchenko]

On Wed, Oct 11, 2023 at 01:12:44PM +0100, Phil Howard wrote:
> Build gpiod into Python module.
> 
> Optional environment var USE_SYSTEM_GPIO=1 to
> generate a module that depends upon system gpiod.

...

> --- /dev/null
> +++ b/bindings/python/include
> @@ -0,0 +1 @@
> +../../include

> \ No newline at end of file

These lines are bothering me, why the new line can't be added to all affected files?

-- 
With Best Regards,
Andy Shevchenko

Re: [libgpiod][PATCH 1/3] bindings: python: optionally include module in sdist

[Phil Howard]

On Wed, 11 Oct 2023 at 18:11, Andy Shevchenko
<andriy.shevchenko@linux.intel.com> wrote:
>
> On Wed, Oct 11, 2023 at 01:12:44PM +0100, Phil Howard wrote:
> > Build gpiod into Python module.
> >
> > Optional environment var USE_SYSTEM_GPIO=1 to
> > generate a module that depends upon system gpiod.
>
> ...

*sigh*

>
> > --- /dev/null
> > +++ b/bindings/python/include
> > @@ -0,0 +1 @@
> > +../../include
>
> > \ No newline at end of file
>
> These lines are bothering me, why the new line can't be added to all affected files?

Is it convention for symlinks to include a newline, is it even possible?

I'm not super sure about the symlink approach, actually.

It's the path of least complexity but after some research into the
usage of "shutil.copytree"
to copy dependent and packaged files at build time I'm starting to
wonder if a more complex
setup.py isn't necessarily a bad idea.

>
> --
> With Best Regards,
> Andy Shevchenko
>
>


-- 
Philip Howard
Technology & Lifestyle Writer
gadgetoid.com

Gadgetoid gadg-et-oid [gaj-it-oid]

                                     -adjective

1. having the characteristics or form of a gadget; resembling a
mechanical contrivance or device.

Re: [libgpiod][PATCH 1/3] bindings: python: optionally include module in sdist

[Bartosz Golaszewski]

On Thu, Oct 12, 2023 at 11:03 AM Phil Howard <phil@gadgetoid.com> wrote:
>
> On Wed, 11 Oct 2023 at 18:11, Andy Shevchenko
> <andriy.shevchenko@linux.intel.com> wrote:
> >
> > On Wed, Oct 11, 2023 at 01:12:44PM +0100, Phil Howard wrote:
> > > Build gpiod into Python module.
> > >
> > > Optional environment var USE_SYSTEM_GPIO=1 to
> > > generate a module that depends upon system gpiod.
> >
> > ...
>
> *sigh*
>
> >
> > > --- /dev/null
> > > +++ b/bindings/python/include
> > > @@ -0,0 +1 @@
> > > +../../include
> >
> > > \ No newline at end of file
> >
> > These lines are bothering me, why the new line can't be added to all affected files?
>
> Is it convention for symlinks to include a newline, is it even possible?
>
> I'm not super sure about the symlink approach, actually.
>
> It's the path of least complexity but after some research into the
> usage of "shutil.copytree"
> to copy dependent and packaged files at build time I'm starting to
> wonder if a more complex
> setup.py isn't necessarily a bad idea.
>

How about not having links in the repo but creating them as required
at build-time?

Bart

> >
> > --
> > With Best Regards,
> > Andy Shevchenko
> >
> >
>
>
> --
> Philip Howard
> Technology & Lifestyle Writer
> gadgetoid.com
>
> Gadgetoid gadg-et-oid [gaj-it-oid]
>
>                                      -adjective
>
> 1. having the characteristics or form of a gadget; resembling a
> mechanical contrivance or device.

Re: [libgpiod][PATCH 1/3] bindings: python: optionally include module in sdist

[Bartosz Golaszewski]

On Wed, Oct 11, 2023 at 2:13 PM Phil Howard <phil@gadgetoid.com> wrote:
>
> Build gpiod into Python module.
>
> Optional environment var USE_SYSTEM_GPIO=1 to

Let's call it USE_SYSTEM_GPIOD or USE_SYSTEM_LIBGPIOD or even
LINK_SYSTEM_LIBGPIOD?

> generate a module that depends upon system gpiod.
>
> Signed-off-by: Phil Howard <phil@gadgetoid.com>
> ---
>  bindings/python/MANIFEST.in |  4 +++
>  bindings/python/include     |  1 +
>  bindings/python/lib         |  1 +
>  bindings/python/setup.py    | 57 ++++++++++++++++++++++++++++---------
>  4 files changed, 50 insertions(+), 13 deletions(-)
>  create mode 120000 bindings/python/include
>  create mode 120000 bindings/python/lib
>
> diff --git a/bindings/python/MANIFEST.in b/bindings/python/MANIFEST.in
> index c7124d4..eff8977 100644
> --- a/bindings/python/MANIFEST.in
> +++ b/bindings/python/MANIFEST.in
> @@ -11,3 +11,7 @@ recursive-include gpiod/ext *.h
>
>  recursive-include tests/gpiosim *.c
>  recursive-include tests/procname *.c
> +
> +recursive-include lib *.c
> +recursive-include lib *.h
> +recursive-include include *.h
> diff --git a/bindings/python/include b/bindings/python/include
> new file mode 120000
> index 0000000..fcffffb
> --- /dev/null
> +++ b/bindings/python/include
> @@ -0,0 +1 @@
> +../../include
> \ No newline at end of file

Addressing Andy's remark: I think this is just how git generates diffs
for links, nothing we can do about it.

I would prefer not to have links in the repo anyway, it would be great
if we could create them at build-time.

Bart

> diff --git a/bindings/python/lib b/bindings/python/lib
> new file mode 120000
> index 0000000..58677dd
> --- /dev/null
> +++ b/bindings/python/lib
> @@ -0,0 +1 @@
> +../../lib
> \ No newline at end of file
> diff --git a/bindings/python/setup.py b/bindings/python/setup.py
> index 66b7908..2e25981 100644
> --- a/bindings/python/setup.py
> +++ b/bindings/python/setup.py
> @@ -19,19 +19,53 @@ class build_ext(orig_build_ext):
>          rmtree(path.join(self.build_lib, "tests"), ignore_errors=True)
>
>
> +with open("gpiod/version.py", "r") as fd:
> +    exec(fd.read())
> +
> +
> +sources = [
> +    # gpiod Python bindings
> +    "gpiod/ext/chip.c",
> +    "gpiod/ext/common.c",
> +    "gpiod/ext/line-config.c",
> +    "gpiod/ext/line-settings.c",
> +    "gpiod/ext/module.c",
> +    "gpiod/ext/request.c",
> +]
> +
> +if "USE_SYSTEM_GPIOD" in environ and environ["USE_SYSTEM_GPIOD"] == "1":
> +    libraries = ["gpiod"]
> +    include_dirs = ["gpiod"]
> +else:
> +    sources += [
> +        # gpiod library
> +        "lib/chip.c",
> +        "lib/chip-info.c",
> +        "lib/edge-event.c",
> +        "lib/info-event.c",
> +        "lib/internal.c",
> +        "lib/line-config.c",
> +        "lib/line-info.c",
> +        "lib/line-request.c",
> +        "lib/line-settings.c",
> +        "lib/misc.c",
> +        "lib/request-config.c",
> +    ]
> +    libraries = []
> +    include_dirs = ["include", "lib", "gpiod/ext"]
> +
> +
>  gpiod_ext = Extension(
>      "gpiod._ext",
> -    sources=[
> -        "gpiod/ext/chip.c",
> -        "gpiod/ext/common.c",
> -        "gpiod/ext/line-config.c",
> -        "gpiod/ext/line-settings.c",
> -        "gpiod/ext/module.c",
> -        "gpiod/ext/request.c",
> -    ],
> +    libraries=libraries,
> +    sources=sources,
>      define_macros=[("_GNU_SOURCE", "1")],
> -    libraries=["gpiod"],
> -    extra_compile_args=["-Wall", "-Wextra"],
> +    include_dirs=include_dirs,
> +    extra_compile_args=[
> +        "-Wall",
> +        "-Wextra",
> +        '-DGPIOD_VERSION_STR="{}"'.format(__version__),
> +    ],
>  )
>
>  gpiosim_ext = Extension(
> @@ -54,9 +88,6 @@ if "GPIOD_WITH_TESTS" in environ and environ["GPIOD_WITH_TESTS"] == "1":
>      extensions.append(gpiosim_ext)
>      extensions.append(procname_ext)
>
> -with open("gpiod/version.py", "r") as fd:
> -    exec(fd.read())
> -
>  setup(
>      name="libgpiod",
>      packages=find_packages(exclude=["tests", "tests.*"]),
> --
> 2.34.1
>

[libgpiod][PATCH 2/3] bindings: python: add pyproject.toml, pep 518

[Phil Howard]

Add pyproject.toml to prevent spurious deprecation warnings from pip.

Signed-off-by: Phil Howard <phil@gadgetoid.com>
---
 bindings/python/pyproject.toml | 2 ++
 1 file changed, 2 insertions(+)
 create mode 100644 bindings/python/pyproject.toml

diff --git a/bindings/python/pyproject.toml b/bindings/python/pyproject.toml
new file mode 100644
index 0000000..d1e6ae6
--- /dev/null
+++ b/bindings/python/pyproject.toml
@@ -0,0 +1,2 @@
+[build-system]
+requires = ["setuptools", "wheel"]
-- 
2.34.1

Re: [libgpiod][PATCH 2/3] bindings: python: add pyproject.toml, pep 518

[Bartosz Golaszewski]

On Wed, Oct 11, 2023 at 2:13 PM Phil Howard <phil@gadgetoid.com> wrote:
>
> Add pyproject.toml to prevent spurious deprecation warnings from pip.
>
> Signed-off-by: Phil Howard <phil@gadgetoid.com>
> ---
>  bindings/python/pyproject.toml | 2 ++
>  1 file changed, 2 insertions(+)
>  create mode 100644 bindings/python/pyproject.toml
>
> diff --git a/bindings/python/pyproject.toml b/bindings/python/pyproject.toml
> new file mode 100644
> index 0000000..d1e6ae6
> --- /dev/null
> +++ b/bindings/python/pyproject.toml
> @@ -0,0 +1,2 @@
> +[build-system]
> +requires = ["setuptools", "wheel"]
> --
> 2.34.1
>

Please add licensing and copyright headers like what you see in all
the other files. I know this is a very small configuration file but
otherwise `reuse lint` complains.

Bart

[libgpiod][PATCH 3/3] bindings: python: require python 3.9.0

[Phil Howard]

Required minimum version for PyModule_AddType helper.

Signed-off-by: Phil Howard <phil@gadgetoid.com>
---
 bindings/python/setup.py | 1 +
 1 file changed, 1 insertion(+)

diff --git a/bindings/python/setup.py b/bindings/python/setup.py
index 2e25981..fbba7ab 100644
--- a/bindings/python/setup.py
+++ b/bindings/python/setup.py
@@ -91,6 +91,7 @@ if "GPIOD_WITH_TESTS" in environ and environ["GPIOD_WITH_TESTS"] == "1":
 setup(
     name="libgpiod",
     packages=find_packages(exclude=["tests", "tests.*"]),
+    python_requires=">=3.9.0",
     ext_modules=extensions,
     cmdclass={"build_ext": build_ext},
     version=__version__,
-- 
2.34.1

Re: [libgpiod][PATCH 3/3] bindings: python: require python 3.9.0

[Bartosz Golaszewski]

On Wed, Oct 11, 2023 at 2:13 PM Phil Howard <phil@gadgetoid.com> wrote:
>
> Required minimum version for PyModule_AddType helper.
>
> Signed-off-by: Phil Howard <phil@gadgetoid.com>
> ---
>  bindings/python/setup.py | 1 +
>  1 file changed, 1 insertion(+)
>
> diff --git a/bindings/python/setup.py b/bindings/python/setup.py
> index 2e25981..fbba7ab 100644
> --- a/bindings/python/setup.py
> +++ b/bindings/python/setup.py
> @@ -91,6 +91,7 @@ if "GPIOD_WITH_TESTS" in environ and environ["GPIOD_WITH_TESTS"] == "1":
>  setup(
>      name="libgpiod",
>      packages=find_packages(exclude=["tests", "tests.*"]),
> +    python_requires=">=3.9.0",
>      ext_modules=extensions,
>      cmdclass={"build_ext": build_ext},
>      version=__version__,
> --
> 2.34.1
>

Applied, thanks!

Bart