[PATCH 0/4] Improve pylibfdt python packaging

[PATCH 0/4] Improve pylibfdt python packaging

[Rob Herring]

I'm interested in getting pylibfdt into PyPI and ran into a few issues
with pylibfdt using the python packaging tools. Primarily, pip didn't
work nor did setup.py sdist and bdist_wheel subcommands. This series
fixes those issues.

I've left meson calling setup.py intact for now, but think it's the
wrong way around. In fact, there's actually some efforts to make meson
the backend for pip/setuptools. I made several attempts to completely
eliminate putting files in the source tree without success. Also, I
noticed a meson install builds pylibfdt twice (though make may too).

I don't think I broke anything. Tests and installs both work with make
and meson.

Rob


Rob Herring (4):
  pylibfdt: Use setuptools instead of distutils
  pylibfdt: Use setuptools_scm for the version
  pylibfdt: Split setup.py author name and email
  pylibfdt: Move setup.py to the top level

 .gitignore                    |  4 ++++
 MANIFEST.in                   |  9 +++++++++
 pylibfdt/Makefile.pylibfdt    |  5 ++---
 pylibfdt/meson.build          |  5 ++---
 pylibfdt/setup.py => setup.py | 34 +++++++++++++---------------------
 5 files changed, 30 insertions(+), 27 deletions(-)
 create mode 100644 MANIFEST.in
 rename pylibfdt/setup.py => setup.py (56%)

-- 
2.32.0

[PATCH 1/4] pylibfdt: Use setuptools instead of distutils

[Rob Herring]

The use of setuptools is favored over distutils. setuptools is needed to
support building Python 'wheels' and for pip support.

Signed-off-by: Rob Herring <robh-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org>
---
 pylibfdt/setup.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/pylibfdt/setup.py b/pylibfdt/setup.py
index ef40f1542cf1..f065a5947067 100755
--- a/pylibfdt/setup.py
+++ b/pylibfdt/setup.py
@@ -10,7 +10,7 @@ Copyright (C) 2017 Google, Inc.
 Written by Simon Glass <sjg-F7+t8E8rja9g9hUCZPvPmw@public.gmane.org>
 """
 
-from distutils.core import setup, Extension
+from setuptools import setup, Extension
 import os
 import re
 import sys
-- 
2.32.0

[PATCH 2/4] pylibfdt: Use setuptools_scm for the version

[Rob Herring]

The DTC version in version_gen.h causes a warning with setuptools:

setuptools/dist.py:501: UserWarning: The version specified ('1.6.1-g5454474d') \
is an invalid version, this may not work as expected with newer versions of \
setuptools, pip, and PyPI. Please see PEP 440 for more details.

It also creates an unnecessary dependency on the rest of the build
system(s). Switch to use setuptools_scm instead to get the version for
pylibfdt.

Signed-off-by: Rob Herring <robh-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org>
---
 pylibfdt/Makefile.pylibfdt |  2 +-
 pylibfdt/meson.build       |  1 -
 pylibfdt/setup.py          | 18 ++++--------------
 3 files changed, 5 insertions(+), 16 deletions(-)

diff --git a/pylibfdt/Makefile.pylibfdt b/pylibfdt/Makefile.pylibfdt
index 1b5f23634e30..015a05ec7590 100644
--- a/pylibfdt/Makefile.pylibfdt
+++ b/pylibfdt/Makefile.pylibfdt
@@ -16,7 +16,7 @@ ifndef V
 SETUPFLAGS += --quiet
 endif
 
-$(PYMODULE): $(PYLIBFDT_srcs) $(LIBFDT_archive) $(SETUP) $(VERSION_FILE)
+$(PYMODULE): $(PYLIBFDT_srcs) $(LIBFDT_archive) $(SETUP)
 	@$(VECHO) PYMOD $@
 	$(PYTHON) $(SETUP) $(SETUPFLAGS) build_ext --build-lib=$(PYLIBFDT_dir)
 
diff --git a/pylibfdt/meson.build b/pylibfdt/meson.build
index 088f24934b4f..fad5aa1c8bc3 100644
--- a/pylibfdt/meson.build
+++ b/pylibfdt/meson.build
@@ -5,7 +5,6 @@ custom_target(
   'pylibfdt',
   input: 'libfdt.i',
   output: '_libfdt.so',
-  depends: version_gen_h,
   command: [setup_py, 'build_ext', '--build-lib=' + meson.current_build_dir()],
   build_by_default: true,
 )
diff --git a/pylibfdt/setup.py b/pylibfdt/setup.py
index f065a5947067..52b61b6d7be2 100755
--- a/pylibfdt/setup.py
+++ b/pylibfdt/setup.py
@@ -15,10 +15,6 @@ import os
 import re
 import sys
 
-
-VERSION_PATTERN = '^#define DTC_VERSION "DTC ([^"]*)"$'
-
-
 def get_top_builddir():
     if '--top-builddir' in sys.argv:
         index = sys.argv.index('--top-builddir')
@@ -27,18 +23,9 @@ def get_top_builddir():
     else:
         return os.getcwd()
 
-
 srcdir = os.path.dirname(os.path.abspath(sys.argv[0]))
 top_builddir = get_top_builddir()
 
-
-def get_version():
-    version_file = os.path.join(top_builddir, 'version_gen.h')
-    f = open(version_file, 'rt')
-    m = re.match(VERSION_PATTERN, f.readline())
-    return m.group(1)
-
-
 libfdt_module = Extension(
     '_libfdt',
     sources=[os.path.join(srcdir, 'libfdt.i')],
@@ -50,7 +37,10 @@ libfdt_module = Extension(
 
 setup(
     name='libfdt',
-    version=get_version(),
+    use_scm_version={
+        "root": os.path.join(srcdir, '..'),
+    },
+    setup_requires = ['setuptools_scm'],
     author='Simon Glass <sjg-F7+t8E8rja9g9hUCZPvPmw@public.gmane.org>',
     description='Python binding for libfdt',
     ext_modules=[libfdt_module],
-- 
2.32.0

[PATCH 3/4] pylibfdt: Split setup.py author name and email

[Rob Herring]

The 'author' field in setup.py is supposed to be just the name. The
email address goes in 'author_email' field.

Cc: Simon Glass <sjg-F7+t8E8rja9g9hUCZPvPmw@public.gmane.org>
Signed-off-by: Rob Herring <robh-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org>
---
 pylibfdt/setup.py | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/pylibfdt/setup.py b/pylibfdt/setup.py
index 52b61b6d7be2..75ce09ac8f6d 100755
--- a/pylibfdt/setup.py
+++ b/pylibfdt/setup.py
@@ -41,7 +41,8 @@ setup(
         "root": os.path.join(srcdir, '..'),
     },
     setup_requires = ['setuptools_scm'],
-    author='Simon Glass <sjg-F7+t8E8rja9g9hUCZPvPmw@public.gmane.org>',
+    author='Simon Glass',
+    author_email='sjg-F7+t8E8rja9g9hUCZPvPmw@public.gmane.org',
     description='Python binding for libfdt',
     ext_modules=[libfdt_module],
     package_dir={'': srcdir},
-- 
2.32.0

[PATCH 4/4] pylibfdt: Move setup.py to the top level

[Rob Herring]

Using 'pip' and several setup.py sub-commands currently don't work with
pylibfdt. The primary reason is Python packaging has opinions on the
directory structure of repositories and one of those appears to be the
inability to reference source files outside of setup.py's subtree. This
means a sdist cannot be created with all necessary source components
(i.e. libfdt headers). Moving setup.py to the top-level solves these
problems.

With this change. the following commands now work:

Creating packages for pypi.org:
./setup.py sdist bdist_wheel

Using pip for installs:
pip install .
pip install git+http://github.com/robherring/dtc.git@pypi-v2

Signed-off-by: Rob Herring <robh-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org>
---
 .gitignore                    |  4 ++++
 MANIFEST.in                   |  9 +++++++++
 pylibfdt/Makefile.pylibfdt    |  3 +--
 pylibfdt/meson.build          |  4 ++--
 pylibfdt/setup.py => setup.py | 15 ++++++++-------
 5 files changed, 24 insertions(+), 11 deletions(-)
 create mode 100644 MANIFEST.in
 rename pylibfdt/setup.py => setup.py (75%)

diff --git a/.gitignore b/.gitignore
index 8e332d8d8e07..416fa05be56a 100644
--- a/.gitignore
+++ b/.gitignore
@@ -22,3 +22,7 @@ lex.yy.c
 # cscope files
 cscope.*
 ncscope.*
+
+.eggs/
+build/
+dist/
diff --git a/MANIFEST.in b/MANIFEST.in
new file mode 100644
index 000000000000..9e6c4ac4610c
--- /dev/null
+++ b/MANIFEST.in
@@ -0,0 +1,9 @@
+# SPDX-License-Identifier: (GPL-2.0-or-later OR BSD-2-Clause)
+
+global-exclude *
+include setup.py
+include pylibfdt/libfdt.i
+include pylibfdt/*.py
+include libfdt/libfdt.h
+include libfdt/fdt.h
+include libfdt/libfdt_env.h
diff --git a/pylibfdt/Makefile.pylibfdt b/pylibfdt/Makefile.pylibfdt
index 015a05ec7590..82f565e370fb 100644
--- a/pylibfdt/Makefile.pylibfdt
+++ b/pylibfdt/Makefile.pylibfdt
@@ -9,8 +9,7 @@ PYLIBFDT_CLEANFILES = $(PYLIBFDT_CLEANFILES_L:%=$(PYLIBFDT_dir)/%)
 PYLIBFDT_CLEANDIRS_L = build __pycache__
 PYLIBFDT_CLEANDIRS = $(PYLIBFDT_CLEANDIRS_L:%=$(PYLIBFDT_dir)/%)
 
-SETUP = $(PYLIBFDT_dir)/setup.py
-SETUPFLAGS = --top-builddir .
+SETUP = ./setup.py
 
 ifndef V
 SETUPFLAGS += --quiet
diff --git a/pylibfdt/meson.build b/pylibfdt/meson.build
index fad5aa1c8bc3..f684cbbfb6eb 100644
--- a/pylibfdt/meson.build
+++ b/pylibfdt/meson.build
@@ -1,5 +1,5 @@
-setup_py = find_program('setup.py')
-setup_py = [setup_py.path(), '--quiet', '--top-builddir', meson.current_build_dir() / '..']
+setup_py = find_program('../setup.py')
+setup_py = [setup_py.path(), '--quiet', '--top-builddir', meson.project_build_root()]
 
 custom_target(
   'pylibfdt',
diff --git a/pylibfdt/setup.py b/setup.py
similarity index 75%
rename from pylibfdt/setup.py
rename to setup.py
index 75ce09ac8f6d..4b07be972466 100755
--- a/pylibfdt/setup.py
+++ b/setup.py
@@ -15,36 +15,37 @@ import os
 import re
 import sys
 
+srcdir = os.path.dirname(__file__)
+
 def get_top_builddir():
     if '--top-builddir' in sys.argv:
         index = sys.argv.index('--top-builddir')
         sys.argv.pop(index)
         return sys.argv.pop(index)
     else:
-        return os.getcwd()
+        return srcdir
 
-srcdir = os.path.dirname(os.path.abspath(sys.argv[0]))
 top_builddir = get_top_builddir()
 
 libfdt_module = Extension(
     '_libfdt',
-    sources=[os.path.join(srcdir, 'libfdt.i')],
-    include_dirs=[os.path.join(srcdir, '../libfdt')],
+    sources=[os.path.join(srcdir, 'pylibfdt/libfdt.i')],
+    include_dirs=[os.path.join(srcdir, 'libfdt')],
     libraries=['fdt'],
     library_dirs=[os.path.join(top_builddir, 'libfdt')],
-    swig_opts=['-I' + os.path.join(srcdir, '../libfdt')],
+    swig_opts=['-I' + os.path.join(srcdir, 'libfdt')],
 )
 
 setup(
     name='libfdt',
     use_scm_version={
-        "root": os.path.join(srcdir, '..'),
+        "root": srcdir,
     },
     setup_requires = ['setuptools_scm'],
     author='Simon Glass',
     author_email='sjg-F7+t8E8rja9g9hUCZPvPmw@public.gmane.org',
     description='Python binding for libfdt',
     ext_modules=[libfdt_module],
-    package_dir={'': srcdir},
+    package_dir={'': os.path.join(srcdir, 'pylibfdt')},
     py_modules=['libfdt'],
 )
-- 
2.32.0

Re: [PATCH 0/4] Improve pylibfdt python packaging

[David Gibson]

[-- Attachment #1: Type: text/plain, Size: 2548 bytes --]

On Wed, Nov 10, 2021 at 07:11:31PM -0600, Rob Herring wrote:
> I'm interested in getting pylibfdt into PyPI and ran into a few issues
> with pylibfdt using the python packaging tools. Primarily, pip didn't
> work nor did setup.py sdist and bdist_wheel subcommands. This series
> fixes those issues.
> 
> I've left meson calling setup.py intact for now, but think it's the
> wrong way around. In fact, there's actually some efforts to make meson
> the backend for pip/setuptools. I made several attempts to completely
> eliminate putting files in the source tree without success. Also, I
> noticed a meson install builds pylibfdt twice (though make may too).
> 
> I don't think I broke anything. Tests and installs both work with make
> and meson.

Applied, it certainly looks better to me.

However, I've just spotted another nasty problem.  I think it must
have started with moving to Fedora 35 on my laptop.  A bunch of the
Python tests now fail like this:

======================================================================
ERROR: testGetIntProperties (__main__.PyLibfdtBasicTests)
Test that we can access properties as integers
----------------------------------------------------------------------
SystemError: PY_SSIZE_T_CLEAN macro must be defined for '#' formats

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/dwg/src/dtc/tests/./pylibfdt_tests.py", line 378, in testGetIntProperties
    self.assertEqual(0xdeadbeef, self.get_prop("prop-hex32").as_uint32())
  File "/home/dwg/src/dtc/tests/./pylibfdt_tests.py", line 374, in get_prop
    return self.fdt2.getprop(0, name)
  File "/home/dwg/src/dtc/tests/../pylibfdt/libfdt.py", line 451, in getprop
    pdata = check_err_null(fdt_getprop(self._fdt, nodeoffset, prop_name),
  File "/home/dwg/src/dtc/tests/../pylibfdt/libfdt.py", line 1279, in fdt_getprop
    return _libfdt.fdt_getprop(fdt, nodeoffset, name)
SystemError: <built-in function fdt_getprop> returned a result with an exception set

Any ideas?

Also, Rob, did you have patches to finish the conversion of the
Makefiles to wrappers around meson?  If so, I'm sorry I've lost track
of them.  Can you repost please?  That README addition with meson
native building instructions would also be really good to have.

-- 
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH 0/4] Improve pylibfdt python packaging

[Rob Herring]

On Wed, Nov 10, 2021 at 9:41 PM David Gibson
<david-xT8FGy+AXnRB3Ne2BGzF6laj5H9X9Tb+@public.gmane.org> wrote:
>
> On Wed, Nov 10, 2021 at 07:11:31PM -0600, Rob Herring wrote:
> > I'm interested in getting pylibfdt into PyPI and ran into a few issues
> > with pylibfdt using the python packaging tools. Primarily, pip didn't
> > work nor did setup.py sdist and bdist_wheel subcommands. This series
> > fixes those issues.
> >
> > I've left meson calling setup.py intact for now, but think it's the
> > wrong way around. In fact, there's actually some efforts to make meson
> > the backend for pip/setuptools. I made several attempts to completely
> > eliminate putting files in the source tree without success. Also, I
> > noticed a meson install builds pylibfdt twice (though make may too).
> >
> > I don't think I broke anything. Tests and installs both work with make
> > and meson.
>
> Applied, it certainly looks better to me.
>
> However, I've just spotted another nasty problem.  I think it must
> have started with moving to Fedora 35 on my laptop.  A bunch of the
> Python tests now fail like this:
>
> ======================================================================
> ERROR: testGetIntProperties (__main__.PyLibfdtBasicTests)
> Test that we can access properties as integers
> ----------------------------------------------------------------------
> SystemError: PY_SSIZE_T_CLEAN macro must be defined for '#' formats
>
> The above exception was the direct cause of the following exception:
>
> Traceback (most recent call last):
>   File "/home/dwg/src/dtc/tests/./pylibfdt_tests.py", line 378, in testGetIntProperties
>     self.assertEqual(0xdeadbeef, self.get_prop("prop-hex32").as_uint32())
>   File "/home/dwg/src/dtc/tests/./pylibfdt_tests.py", line 374, in get_prop
>     return self.fdt2.getprop(0, name)
>   File "/home/dwg/src/dtc/tests/../pylibfdt/libfdt.py", line 451, in getprop
>     pdata = check_err_null(fdt_getprop(self._fdt, nodeoffset, prop_name),
>   File "/home/dwg/src/dtc/tests/../pylibfdt/libfdt.py", line 1279, in fdt_getprop
>     return _libfdt.fdt_getprop(fdt, nodeoffset, name)
> SystemError: <built-in function fdt_getprop> returned a result with an exception set
>
> Any ideas?

Python 3.10? Only guessing because I'm on 3.9. Otherwise, I have no clue.

I was going to look at making '.setup.py test' work as testing is
intertwined with meson too. Most python CI testing runs against a
matrix of python versions which would help here.

> Also, Rob, did you have patches to finish the conversion of the
> Makefiles to wrappers around meson?

That was Marc-André...

> If so, I'm sorry I've lost track
> of them.  Can you repost please?

One of the issues you had with Travis CI. Are you still using Travis
CI after their move? I found it easier to just move to GH workflows
than move given I always seem to hit login token issues (maybe that's
just group projects with multiple users).

> That README addition with meson
> native building instructions would also be really good to have.

I can look at that. It's gotten a bit more straight-forward with newer
meson versions as you don't have to run ninja directly. So it is just:

One-time init:
meson <builddir>

And then:
meson compile|test|install -C <builddir>

(I find specifying the builddir every time pretty annoying. Guess I
need a wrapper script.)

Rob

Re: [PATCH 0/4] Improve pylibfdt python packaging

[Bruce Ashfield]

On 11/11/2021 9:08 AM, Rob Herring wrote:
> On Wed, Nov 10, 2021 at 9:41 PM David Gibson
> <david-xT8FGy+AXnRB3Ne2BGzF6laj5H9X9Tb+@public.gmane.org> wrote:
>>
>> On Wed, Nov 10, 2021 at 07:11:31PM -0600, Rob Herring wrote:
>>> I'm interested in getting pylibfdt into PyPI and ran into a few issues
>>> with pylibfdt using the python packaging tools. Primarily, pip didn't
>>> work nor did setup.py sdist and bdist_wheel subcommands. This series
>>> fixes those issues.
>>>
>>> I've left meson calling setup.py intact for now, but think it's the
>>> wrong way around. In fact, there's actually some efforts to make meson
>>> the backend for pip/setuptools. I made several attempts to completely
>>> eliminate putting files in the source tree without success. Also, I
>>> noticed a meson install builds pylibfdt twice (though make may too).
>>>
>>> I don't think I broke anything. Tests and installs both work with make
>>> and meson.
>>
>> Applied, it certainly looks better to me.
>>
>> However, I've just spotted another nasty problem.  I think it must
>> have started with moving to Fedora 35 on my laptop.  A bunch of the
>> Python tests now fail like this:
>>
>> ======================================================================
>> ERROR: testGetIntProperties (__main__.PyLibfdtBasicTests)
>> Test that we can access properties as integers
>> ----------------------------------------------------------------------
>> SystemError: PY_SSIZE_T_CLEAN macro must be defined for '#' formats
>>
>> The above exception was the direct cause of the following exception:
>>
>> Traceback (most recent call last):
>>    File "/home/dwg/src/dtc/tests/./pylibfdt_tests.py", line 378, in testGetIntProperties
>>      self.assertEqual(0xdeadbeef, self.get_prop("prop-hex32").as_uint32())
>>    File "/home/dwg/src/dtc/tests/./pylibfdt_tests.py", line 374, in get_prop
>>      return self.fdt2.getprop(0, name)
>>    File "/home/dwg/src/dtc/tests/../pylibfdt/libfdt.py", line 451, in getprop
>>      pdata = check_err_null(fdt_getprop(self._fdt, nodeoffset, prop_name),
>>    File "/home/dwg/src/dtc/tests/../pylibfdt/libfdt.py", line 1279, in fdt_getprop
>>      return _libfdt.fdt_getprop(fdt, nodeoffset, name)
>> SystemError: <built-in function fdt_getprop> returned a result with an exception set
>>
>> Any ideas?
> 
> Python 3.10? Only guessing because I'm on 3.9. Otherwise, I have no clue.
> 

FWIW, and in case you haven't already sorted it out.

We just fixed this in the yocto integration of dtc/libfdt this
morning (I case the patch hasn't made it upstream to you yet).

It is indeed python 3.10, and in particular:

https://docs.python.org/3.10/whatsnew/3.10.html#id2

(and again, a patch should arrive shortly if it hasn't already).

Cheers,

Bruce

> I was going to look at making '.setup.py test' work as testing is
> intertwined with meson too. Most python CI testing runs against a
> matrix of python versions which would help here.
> 
>> Also, Rob, did you have patches to finish the conversion of the
>> Makefiles to wrappers around meson?
> 
> That was Marc-André...
> 
>> If so, I'm sorry I've lost track
>> of them.  Can you repost please?
> 
> One of the issues you had with Travis CI. Are you still using Travis
> CI after their move? I found it easier to just move to GH workflows
> than move given I always seem to hit login token issues (maybe that's
> just group projects with multiple users).
> 
>> That README addition with meson
>> native building instructions would also be really good to have.
> 
> I can look at that. It's gotten a bit more straight-forward with newer
> meson versions as you don't have to run ninja directly. So it is just:
> 
> One-time init:
> meson <builddir>
> 
> And then:
> meson compile|test|install -C <builddir>
> 
> (I find specifying the builddir every time pretty annoying. Guess I
> need a wrapper script.)
> 
> Rob
> 

Re: [PATCH 0/4] Improve pylibfdt python packaging

[David Gibson]

[-- Attachment #1: Type: text/plain, Size: 4259 bytes --]

On Thu, Nov 11, 2021 at 08:08:08AM -0600, Rob Herring wrote:
> On Wed, Nov 10, 2021 at 9:41 PM David Gibson
> <david-xT8FGy+AXnRB3Ne2BGzF6laj5H9X9Tb+@public.gmane.org> wrote:
> >
> > On Wed, Nov 10, 2021 at 07:11:31PM -0600, Rob Herring wrote:
> > > I'm interested in getting pylibfdt into PyPI and ran into a few issues
> > > with pylibfdt using the python packaging tools. Primarily, pip didn't
> > > work nor did setup.py sdist and bdist_wheel subcommands. This series
> > > fixes those issues.
> > >
> > > I've left meson calling setup.py intact for now, but think it's the
> > > wrong way around. In fact, there's actually some efforts to make meson
> > > the backend for pip/setuptools. I made several attempts to completely
> > > eliminate putting files in the source tree without success. Also, I
> > > noticed a meson install builds pylibfdt twice (though make may too).
> > >
> > > I don't think I broke anything. Tests and installs both work with make
> > > and meson.
> >
> > Applied, it certainly looks better to me.
> >
> > However, I've just spotted another nasty problem.  I think it must
> > have started with moving to Fedora 35 on my laptop.  A bunch of the
> > Python tests now fail like this:
> >
> > ======================================================================
> > ERROR: testGetIntProperties (__main__.PyLibfdtBasicTests)
> > Test that we can access properties as integers
> > ----------------------------------------------------------------------
> > SystemError: PY_SSIZE_T_CLEAN macro must be defined for '#' formats
> >
> > The above exception was the direct cause of the following exception:
> >
> > Traceback (most recent call last):
> >   File "/home/dwg/src/dtc/tests/./pylibfdt_tests.py", line 378, in testGetIntProperties
> >     self.assertEqual(0xdeadbeef, self.get_prop("prop-hex32").as_uint32())
> >   File "/home/dwg/src/dtc/tests/./pylibfdt_tests.py", line 374, in get_prop
> >     return self.fdt2.getprop(0, name)
> >   File "/home/dwg/src/dtc/tests/../pylibfdt/libfdt.py", line 451, in getprop
> >     pdata = check_err_null(fdt_getprop(self._fdt, nodeoffset, prop_name),
> >   File "/home/dwg/src/dtc/tests/../pylibfdt/libfdt.py", line 1279, in fdt_getprop
> >     return _libfdt.fdt_getprop(fdt, nodeoffset, name)
> > SystemError: <built-in function fdt_getprop> returned a result with an exception set
> >
> > Any ideas?
> 
> Python 3.10? Only guessing because I'm on 3.9. Otherwise, I have no clue.

It appears so; I've now merged Ross Burton's fix.

> I was going to look at making '.setup.py test' work as testing is
> intertwined with meson too. Most python CI testing runs against a
> matrix of python versions which would help here.
> 
> > Also, Rob, did you have patches to finish the conversion of the
> > Makefiles to wrappers around meson?
> 
> That was Marc-André...

Oops, sorry, I got confused.

> > If so, I'm sorry I've lost track
> > of them.  Can you repost please?
> 
> One of the issues you had with Travis CI. Are you still using Travis
> CI after their move? I found it easier to just move to GH workflows
> than move given I always seem to hit login token issues (maybe that's
> just group projects with multiple users).

It stopped working after the move, and I haven't looked into what
would be needed to make it go again.  I was actually thinking I'd move
over to GitLan - I'm more familiar with its CI stuff from qemu, and I
like what I've seen.  I haven't actually had a chance to do anything
on that front, though.

> > That README addition with meson
> > native building instructions would also be really good to have.
> 
> I can look at that. It's gotten a bit more straight-forward with newer
> meson versions as you don't have to run ninja directly. So it is just:
> 
> One-time init:
> meson <builddir>
> 
> And then:
> meson compile|test|install -C <builddir>
> 
> (I find specifying the builddir every time pretty annoying. Guess I
> need a wrapper script.)

Yeah, I tend to agree.

-- 
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH 0/4] Improve pylibfdt python packaging

[Rob Herring]

On Thu, Nov 11, 2021 at 8:43 PM David Gibson
<david-xT8FGy+AXnRB3Ne2BGzF6laj5H9X9Tb+@public.gmane.org> wrote:
>
> On Thu, Nov 11, 2021 at 08:08:08AM -0600, Rob Herring wrote:
> > On Wed, Nov 10, 2021 at 9:41 PM David Gibson
> > <david-xT8FGy+AXnRB3Ne2BGzF6laj5H9X9Tb+@public.gmane.org> wrote:
> > >
> > > On Wed, Nov 10, 2021 at 07:11:31PM -0600, Rob Herring wrote:
> > > > I'm interested in getting pylibfdt into PyPI and ran into a few issues
> > > > with pylibfdt using the python packaging tools. Primarily, pip didn't
> > > > work nor did setup.py sdist and bdist_wheel subcommands. This series
> > > > fixes those issues.
> > > >
> > > > I've left meson calling setup.py intact for now, but think it's the
> > > > wrong way around. In fact, there's actually some efforts to make meson
> > > > the backend for pip/setuptools. I made several attempts to completely
> > > > eliminate putting files in the source tree without success. Also, I
> > > > noticed a meson install builds pylibfdt twice (though make may too).
> > > >
> > > > I don't think I broke anything. Tests and installs both work with make
> > > > and meson.
> > >
> > > Applied, it certainly looks better to me.
> > >
> > > However, I've just spotted another nasty problem.  I think it must
> > > have started with moving to Fedora 35 on my laptop.  A bunch of the
> > > Python tests now fail like this:
> > >
> > > ======================================================================
> > > ERROR: testGetIntProperties (__main__.PyLibfdtBasicTests)
> > > Test that we can access properties as integers
> > > ----------------------------------------------------------------------
> > > SystemError: PY_SSIZE_T_CLEAN macro must be defined for '#' formats
> > >
> > > The above exception was the direct cause of the following exception:
> > >
> > > Traceback (most recent call last):
> > >   File "/home/dwg/src/dtc/tests/./pylibfdt_tests.py", line 378, in testGetIntProperties
> > >     self.assertEqual(0xdeadbeef, self.get_prop("prop-hex32").as_uint32())
> > >   File "/home/dwg/src/dtc/tests/./pylibfdt_tests.py", line 374, in get_prop
> > >     return self.fdt2.getprop(0, name)
> > >   File "/home/dwg/src/dtc/tests/../pylibfdt/libfdt.py", line 451, in getprop
> > >     pdata = check_err_null(fdt_getprop(self._fdt, nodeoffset, prop_name),
> > >   File "/home/dwg/src/dtc/tests/../pylibfdt/libfdt.py", line 1279, in fdt_getprop
> > >     return _libfdt.fdt_getprop(fdt, nodeoffset, name)
> > > SystemError: <built-in function fdt_getprop> returned a result with an exception set
> > >
> > > Any ideas?
> >
> > Python 3.10? Only guessing because I'm on 3.9. Otherwise, I have no clue.
>
> It appears so; I've now merged Ross Burton's fix.
>
> > I was going to look at making '.setup.py test' work as testing is
> > intertwined with meson too. Most python CI testing runs against a
> > matrix of python versions which would help here.
> >
> > > Also, Rob, did you have patches to finish the conversion of the
> > > Makefiles to wrappers around meson?
> >
> > That was Marc-André...
>
> Oops, sorry, I got confused.
>
> > > If so, I'm sorry I've lost track
> > > of them.  Can you repost please?
> >
> > One of the issues you had with Travis CI. Are you still using Travis
> > CI after their move? I found it easier to just move to GH workflows
> > than move given I always seem to hit login token issues (maybe that's
> > just group projects with multiple users).
>
> It stopped working after the move, and I haven't looked into what
> would be needed to make it go again.  I was actually thinking I'd move
> over to GitLan - I'm more familiar with its CI stuff from qemu, and I
> like what I've seen.  I haven't actually had a chance to do anything
> on that front, though.

I'm assuming you mean GitLab. I'm using both. The DT spec and dtschema
test/pkging are using workflows. Schema validation runs with the
kernel tree are on GitLab (which actually run in docker on my
machines).

GH workflows have recipes (written by random folks) to do various
things like python version matrix testing or upload to pypi whereas
GitLab is more 'here's a container'. dtc could go either way I think.

Rob

Re: [PATCH 0/4] Improve pylibfdt python packaging

[David Gibson]

[-- Attachment #1: Type: text/plain, Size: 4723 bytes --]

On Thu, Nov 11, 2021 at 10:01:55PM -0600, Rob Herring wrote:
> On Thu, Nov 11, 2021 at 8:43 PM David Gibson
> <david-xT8FGy+AXnRB3Ne2BGzF6laj5H9X9Tb+@public.gmane.org> wrote:
> >
> > On Thu, Nov 11, 2021 at 08:08:08AM -0600, Rob Herring wrote:
> > > On Wed, Nov 10, 2021 at 9:41 PM David Gibson
> > > <david-xT8FGy+AXnRB3Ne2BGzF6laj5H9X9Tb+@public.gmane.org> wrote:
> > > >
> > > > On Wed, Nov 10, 2021 at 07:11:31PM -0600, Rob Herring wrote:
> > > > > I'm interested in getting pylibfdt into PyPI and ran into a few issues
> > > > > with pylibfdt using the python packaging tools. Primarily, pip didn't
> > > > > work nor did setup.py sdist and bdist_wheel subcommands. This series
> > > > > fixes those issues.
> > > > >
> > > > > I've left meson calling setup.py intact for now, but think it's the
> > > > > wrong way around. In fact, there's actually some efforts to make meson
> > > > > the backend for pip/setuptools. I made several attempts to completely
> > > > > eliminate putting files in the source tree without success. Also, I
> > > > > noticed a meson install builds pylibfdt twice (though make may too).
> > > > >
> > > > > I don't think I broke anything. Tests and installs both work with make
> > > > > and meson.
> > > >
> > > > Applied, it certainly looks better to me.
> > > >
> > > > However, I've just spotted another nasty problem.  I think it must
> > > > have started with moving to Fedora 35 on my laptop.  A bunch of the
> > > > Python tests now fail like this:
> > > >
> > > > ======================================================================
> > > > ERROR: testGetIntProperties (__main__.PyLibfdtBasicTests)
> > > > Test that we can access properties as integers
> > > > ----------------------------------------------------------------------
> > > > SystemError: PY_SSIZE_T_CLEAN macro must be defined for '#' formats
> > > >
> > > > The above exception was the direct cause of the following exception:
> > > >
> > > > Traceback (most recent call last):
> > > >   File "/home/dwg/src/dtc/tests/./pylibfdt_tests.py", line 378, in testGetIntProperties
> > > >     self.assertEqual(0xdeadbeef, self.get_prop("prop-hex32").as_uint32())
> > > >   File "/home/dwg/src/dtc/tests/./pylibfdt_tests.py", line 374, in get_prop
> > > >     return self.fdt2.getprop(0, name)
> > > >   File "/home/dwg/src/dtc/tests/../pylibfdt/libfdt.py", line 451, in getprop
> > > >     pdata = check_err_null(fdt_getprop(self._fdt, nodeoffset, prop_name),
> > > >   File "/home/dwg/src/dtc/tests/../pylibfdt/libfdt.py", line 1279, in fdt_getprop
> > > >     return _libfdt.fdt_getprop(fdt, nodeoffset, name)
> > > > SystemError: <built-in function fdt_getprop> returned a result with an exception set
> > > >
> > > > Any ideas?
> > >
> > > Python 3.10? Only guessing because I'm on 3.9. Otherwise, I have no clue.
> >
> > It appears so; I've now merged Ross Burton's fix.
> >
> > > I was going to look at making '.setup.py test' work as testing is
> > > intertwined with meson too. Most python CI testing runs against a
> > > matrix of python versions which would help here.
> > >
> > > > Also, Rob, did you have patches to finish the conversion of the
> > > > Makefiles to wrappers around meson?
> > >
> > > That was Marc-André...
> >
> > Oops, sorry, I got confused.
> >
> > > > If so, I'm sorry I've lost track
> > > > of them.  Can you repost please?
> > >
> > > One of the issues you had with Travis CI. Are you still using Travis
> > > CI after their move? I found it easier to just move to GH workflows
> > > than move given I always seem to hit login token issues (maybe that's
> > > just group projects with multiple users).
> >
> > It stopped working after the move, and I haven't looked into what
> > would be needed to make it go again.  I was actually thinking I'd move
> > over to GitLan - I'm more familiar with its CI stuff from qemu, and I
> > like what I've seen.  I haven't actually had a chance to do anything
> > on that front, though.
> 
> I'm assuming you mean GitLab. I'm using both. The DT spec and dtschema
> test/pkging are using workflows. Schema validation runs with the
> kernel tree are on GitLab (which actually run in docker on my
> machines).

Yes, I meant gitlab.

> GH workflows have recipes (written by random folks) to do various
> things like python version matrix testing or upload to pypi whereas
> GitLab is more 'here's a container'. dtc could go either way I think.

Ok, good to know.

-- 
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]