[PATCH 0/4] Add python_mesonpy ("mesonpy") PEP-517 backend support

[PATCH 0/4] Add python_mesonpy ("mesonpy") PEP-517 backend support

[Tim Orling]

Move the python_mesonpy bbclass and python3-meson-python recipe from meta-python.

https://meson-python.readthedocs.io/en/latest/

This is the declared backend for python-numpy, so it is time to move it to oe-core.

The following changes since commit 4cfd0f7e4e2db19344677999572e5b71ae97dfc4:

  lib/oe/patch: Use git notes to store the filenames for the patches (2024-02-19 16:03:22 +0000)

are available in the Git repository at:

  https://git.yoctoproject.org/poky-contrib timo/python_mesonpy
  https://git.yoctoproject.org/poky-contrib/log/?h=timo/python_mesonpy

Tim Orling (4):
  python3-meson-python: move from meta-python
  python_mesonpy.bbclass: move from meta-python
  python3-numpy: inherit python_mesonpy
  recipetool; add support for python_mesonpy class

 meta/classes-recipe/python_mesonpy.bbclass    | 52 ++++++++++++++++++
 meta/conf/distro/include/maintainers.inc      |  1 +
 meta/lib/oeqa/selftest/cases/recipetool.py    | 19 +++++++
 .../python/python3-meson-python_0.15.0.bb     | 27 ++++++++++
 .../python/python3-numpy_1.26.4.bb            | 54 ++++++++++---------
 .../lib/recipetool/create_buildsys_python.py  |  4 ++
 6 files changed, 131 insertions(+), 26 deletions(-)
 create mode 100644 meta/classes-recipe/python_mesonpy.bbclass
 create mode 100644 meta/recipes-devtools/python/python3-meson-python_0.15.0.bb

-- 
2.34.1

[PATCH 1/4] python3-meson-python: move from meta-python

[Tim Orling]

meson-python is a Python build backend built on top of the Meson
build-system. It enables you to use Meson for your Python packages.

https://meson-python.readthedocs.io/en/latest/

It is used as the PEP-517 build-backend for python3-numpy and python3-scipy.

For other projects using the backend, see:
https://meson-python.readthedocs.io/en/latest/projects-using-meson-python.html

Signed-off-by: Tim Orling <tim.orling@konsulko.com>
---
 meta/conf/distro/include/maintainers.inc      |  1 +
 .../python/python3-meson-python_0.15.0.bb     | 27 +++++++++++++++++++
 2 files changed, 28 insertions(+)
 create mode 100644 meta/recipes-devtools/python/python3-meson-python_0.15.0.bb

diff --git a/meta/conf/distro/include/maintainers.inc b/meta/conf/distro/include/maintainers.inc
index f489fbe275f..1ab69175bb8 100644
--- a/meta/conf/distro/include/maintainers.inc
+++ b/meta/conf/distro/include/maintainers.inc
@@ -657,6 +657,7 @@ RECIPE_MAINTAINER:pn-python3-mako = "Trevor Gamblin <tgamblin@baylibre.com>"
 RECIPE_MAINTAINER:pn-python3-markdown = "Alexander Kanavin <alex.kanavin@gmail.com>"
 RECIPE_MAINTAINER:pn-python3-markupsafe = "Richard Purdie <richard.purdie@linuxfoundation.org>"
 RECIPE_MAINTAINER:pn-python3-maturin = "Tim Orling <tim.orling@konsulko.com>"
+RECIPE_MAINTAINER:pn-python3-meson-python = "Tim Orling <tim.orling@konsulko.com>"
 RECIPE_MAINTAINER:pn-python3-more-itertools = "Tim Orling <tim.orling@konsulko.com>"
 RECIPE_MAINTAINER:pn-python3-ndg-httpsclient = "Tim Orling <tim.orling@konsulko.com>"
 RECIPE_MAINTAINER:pn-python3-numpy = "Trevor Gamblin <tgamblin@baylibre.com>"
diff --git a/meta/recipes-devtools/python/python3-meson-python_0.15.0.bb b/meta/recipes-devtools/python/python3-meson-python_0.15.0.bb
new file mode 100644
index 00000000000..ad3cfe17d9e
--- /dev/null
+++ b/meta/recipes-devtools/python/python3-meson-python_0.15.0.bb
@@ -0,0 +1,27 @@
+SUMMARY = "Meson Python build backend (PEP 517)"
+HOMEPAGE = "https://github.com/mesonbuild/meson-python"
+LICENSE = "MIT"
+LIC_FILES_CHKSUM = "file://LICENSE;md5=d580b27e67cc0892a5b005b0be114b60"
+
+DEPENDS = " \
+	meson-native \
+	ninja-native \
+	patchelf-native \
+	python3-pyproject-metadata-native \
+"
+
+PYPI_PACKAGE = "meson_python"
+
+inherit pypi python_mesonpy
+SRC_URI[sha256sum] = "fddb73eecd49e89c1c41c87937cd89c2d0b65a1c63ba28238681d4bd9484d26f"
+
+DEPENDS:remove:class-native = "python3-meson-python-native"
+
+RDEPENDS:${PN} = " \
+	meson \
+	ninja \
+	patchelf \
+	python3-pyproject-metadata \
+"
+
+BBCLASSEXTEND = "native nativesdk"
-- 
2.34.1

[PATCH 2/4] python_mesonpy.bbclass: move from meta-python

[Tim Orling]

Move the PEP-517 back-end bbclass from meta-python to support:

[build-system]
build-backend = "mesonpy"

This is the declared backend in python3-numpy since:
https://github.com/numpy/numpy/commit/942fb8caf33a65e449fbf198ecf1cd39be953248

Signed-off-by: Tim Orling <tim.orling@konsulko.com>
---
 meta/classes-recipe/python_mesonpy.bbclass | 52 ++++++++++++++++++++++
 1 file changed, 52 insertions(+)
 create mode 100644 meta/classes-recipe/python_mesonpy.bbclass

diff --git a/meta/classes-recipe/python_mesonpy.bbclass b/meta/classes-recipe/python_mesonpy.bbclass
new file mode 100644
index 00000000000..131fa74bede
--- /dev/null
+++ b/meta/classes-recipe/python_mesonpy.bbclass
@@ -0,0 +1,52 @@
+#
+# Copyright OpenEmbedded Contributors
+#
+# SPDX-License-Identifier: MIT
+#
+
+inherit meson setuptools3-base python3targetconfig python_pep517
+
+# meson_do_qa_configure does the wrong thing here because
+# mesonpy runs "meson setup ..." in do_compile context.
+# Make it a dummy function.
+meson_do_qa_configure () {
+    :
+}
+
+# This prevents the meson error:
+# ERROR: Got argument buildtype as both -Dbuildtype and --buildtype. Pick one.
+MESONOPTS:remove = "--buildtype ${MESON_BUILDTYPE}"
+
+CONFIGURE_FILES = "pyproject.toml"
+
+DEPENDS += "python3-wheel-native python3-meson-python-native"
+
+def mesonpy_get_args(d):
+    vars = ['MESONOPTS', 'MESON_CROSS_FILE', 'EXTRA_OEMESON']
+    varlist = []
+    for var in vars:
+        value = d.getVar(var)
+        vallist = value.split()
+        for elem in vallist:
+            varlist.append("-Csetup-args=" + elem)
+    return ' '.join(varlist)
+
+PEP517_BUILD_OPTS = "-Cbuilddir='${B}' ${@mesonpy_get_args(d)}"
+
+# Python pyx -> c -> so build leaves absolute build paths in the code
+INSANE_SKIP:${PN} += "buildpaths"
+INSANE_SKIP:${PN}-src += "buildpaths"
+
+python_mesonpy_do_configure () {
+    python_pep517_do_configure
+}
+
+python_mesonpy_do_compile () {
+    python_pep517_do_compile
+}
+
+python_mesonpy_do_install () {
+    python_pep517_do_install
+}
+
+EXPORT_FUNCTIONS do_configure do_compile do_install
-- 
2.34.1

[PATCH 3/4] python3-numpy: inherit python_mesonpy

[Tim Orling]

Replace the deprecated setuptools3 bbclass with python_mesonpy.

The build-backend has been defined as "mesonpy" since:
https://github.com/numpy/numpy/commit/942fb8caf33a65e449fbf198ecf1cd39be953248

The vendored meson-python was dropped in:
https://github.com/numpy/numpy/commit/6544e33ac7a3a600c2fb565401c811a17ecdb3d5

While we are at it:
* Drop ${PYTHON_PN} and use python3 instead
* Sort RDEPENDS alphabetically

The ptests run, but we still have issues with sufficient memory and
free disk space (the reason python3-numpy is in PTEST_PROBLEMS).

Signed-off-by: Tim Orling <tim.orling@konsulko.com>
---
 .../python/python3-numpy_1.26.4.bb            | 54 ++++++++++---------
 1 file changed, 28 insertions(+), 26 deletions(-)

diff --git a/meta/recipes-devtools/python/python3-numpy_1.26.4.bb b/meta/recipes-devtools/python/python3-numpy_1.26.4.bb
index d11b03efc3d..402d74db72d 100644
--- a/meta/recipes-devtools/python/python3-numpy_1.26.4.bb
+++ b/meta/recipes-devtools/python/python3-numpy_1.26.4.bb
@@ -20,7 +20,7 @@ UPSTREAM_CHECK_REGEX = "releases/tag/v?(?P<pver>\d+(\.\d+)+)$"
 
 DEPENDS += "python3-cython-native"
 
-inherit ptest setuptools3 github-releases
+inherit ptest python_mesonpy github-releases
 
 S = "${WORKDIR}/numpy-${PV}"
 
@@ -33,32 +33,34 @@ do_compile:prepend() {
 FILES:${PN}-staticdev += "${PYTHON_SITEPACKAGES_DIR}/numpy/core/lib/*.a ${PYTHON_SITEPACKAGES_DIR}/numpy/random/lib/*.a"
 
 # install what is needed for numpy.test()
-RDEPENDS:${PN} = "${PYTHON_PN}-unittest \
-                  ${PYTHON_PN}-difflib \
-                  ${PYTHON_PN}-pprint \
-                  ${PYTHON_PN}-pickle \
-                  ${PYTHON_PN}-shell \
-                  ${PYTHON_PN}-doctest \
-                  ${PYTHON_PN}-datetime \
-                  ${PYTHON_PN}-misc \
-                  ${PYTHON_PN}-mmap \
-                  ${PYTHON_PN}-netclient \
-                  ${PYTHON_PN}-numbers \
-                  ${PYTHON_PN}-pydoc \
-                  ${PYTHON_PN}-pkgutil \
-                  ${PYTHON_PN}-email \
-                  ${PYTHON_PN}-compression \
-                  ${PYTHON_PN}-ctypes \
-                  ${PYTHON_PN}-threading \
-                  ${PYTHON_PN}-multiprocessing \
-                  ${PYTHON_PN}-json \
+RDEPENDS:${PN} = "\
+    python3-compression \
+    python3-ctypes \
+    python3-datetime \
+    python3-difflib \
+    python3-doctest \
+    python3-email \
+    python3-json \
+    python3-misc \
+    python3-mmap \
+    python3-multiprocessing \
+    python3-netclient \
+    python3-numbers \
+    python3-pickle \
+    python3-pkgutil \
+    python3-pprint \
+    python3-pydoc \
+    python3-shell \
+    python3-threading \
+    python3-unittest \
 "
-RDEPENDS:${PN}-ptest += "${PYTHON_PN}-pytest \
-                         ${PYTHON_PN}-hypothesis \
-                         ${PYTHON_PN}-sortedcontainers \
-                         ${PYTHON_PN}-resource \
-                         ${PYTHON_PN}-typing-extensions \
-                         ldd \
+RDEPENDS:${PN}-ptest += "\
+    ldd \
+    python3-hypothesis \
+    python3-pytest \
+    python3-resource \
+    python3-sortedcontainers \
+    python3-typing-extensions \
 "
 
 BBCLASSEXTEND = "native nativesdk"
-- 
2.34.1

[PATCH 4/4] recipetool; add support for python_mesonpy class

[Tim Orling]

* Add support to detect the "mesonpy" build-backend for recipetool create.
* Add oe-selftest case for creating a recipe for "siphash24" from pypi.
  https://pypi.org/project/siphash24/

This is by far the simplest recipe using the mesonpy build backend.

Upstream does not provide LICENSE file(s) and we do not detect the
LICENSE so don't check for that result in the test. Likewise, upstream
does not define HOMEPAGE, so skip that result.

Signed-off-by: Tim Orling <tim.orling@konsulko.com>
---
 meta/lib/oeqa/selftest/cases/recipetool.py    | 19 +++++++++++++++++++
 .../lib/recipetool/create_buildsys_python.py  |  4 ++++
 2 files changed, 23 insertions(+)

diff --git a/meta/lib/oeqa/selftest/cases/recipetool.py b/meta/lib/oeqa/selftest/cases/recipetool.py
index 83361814dfd..2eca1800de1 100644
--- a/meta/lib/oeqa/selftest/cases/recipetool.py
+++ b/meta/lib/oeqa/selftest/cases/recipetool.py
@@ -676,6 +676,25 @@ class RecipetoolCreateTests(RecipetoolBase):
 
         self._test_recipe_contents(recipefile, checkvars, inherits)
 
+    def test_recipetool_create_python3_pep517_mesonpy(self):
+        # This test require python 3.11 or above for the tomllib module or tomli module to be installed
+        needTomllib(self)
+
+        # Test creating python3 package from tarball (using mesonpy class)
+        temprecipe = os.path.join(self.tempdir, 'recipe')
+        os.makedirs(temprecipe)
+        pn = 'siphash24'
+        pv = '1.4'
+        recipefile = os.path.join(temprecipe, 'python3-%s_%s.bb' % (pn, pv))
+        srcuri = 'https://files.pythonhosted.org/packages/c2/32/b934a70592f314afcfa86c7f7e388804a8061be65b822e2aa07e573b6477/%s-%s.tar.gz' % (pn, pv)
+        result = runCmd('recipetool create -o %s %s' % (temprecipe, srcuri))
+        self.assertTrue(os.path.isfile(recipefile))
+        checkvars = {}
+        checkvars['SRC_URI[sha256sum]'] = '7fd65e39b2a7c8c4ddc3a168a687f4610751b0ac2ebb518783c0cdfc30bec4a0'
+        inherits = ['python_mesonpy', 'pypi']
+
+        self._test_recipe_contents(recipefile, checkvars, inherits)
+
     def test_recipetool_create_github_tarball(self):
         # Basic test to ensure github URL mangling doesn't apply to release tarballs.
         # Deliberately use an older release of Meson at present so we don't need a toml parser.
diff --git a/scripts/lib/recipetool/create_buildsys_python.py b/scripts/lib/recipetool/create_buildsys_python.py
index a589343cfbf..a807dafae52 100644
--- a/scripts/lib/recipetool/create_buildsys_python.py
+++ b/scripts/lib/recipetool/create_buildsys_python.py
@@ -739,6 +739,7 @@ class PythonPyprojectTomlRecipeHandler(PythonRecipeHandler):
         "flit_core.buildapi": "python_flit_core",
         "hatchling.build": "python_hatchling",
         "maturin": "python_maturin",
+        "mesonpy": "python_mesonpy",
     }
 
     # setuptools.build_meta and flit declare project metadata into the "project" section of pyproject.toml
@@ -779,6 +780,8 @@ class PythonPyprojectTomlRecipeHandler(PythonRecipeHandler):
         "python3-poetry-core-native",
         # already provided by python_flit_core.bbclass
         "python3-flit-core-native",
+        # already provided by python_mesonpy
+        "python3-meson-python-native",
     ]
 
     # add here a list of known and often used packages and the corresponding bitbake package
@@ -790,6 +793,7 @@ class PythonPyprojectTomlRecipeHandler(PythonRecipeHandler):
         "setuptools-scm": "python3-setuptools-scm",
         "hatchling": "python3-hatchling",
         "hatch-vcs": "python3-hatch-vcs",
+        "meson-python" : "python3-meson-python",
     }
 
     def __init__(self):
-- 
2.34.1

Patchtest results for [PATCH 4/4] recipetool; add support for python_mesonpy class

[patchtest]

[-- Attachment #1: Type: text/plain, Size: 3070 bytes --]

Thank you for your submission. Patchtest identified one
or more issues with the patch. Please see the log below for
more information:

---
Testing patch /home/patchtest/share/mboxes/4-4-recipetool-add-support-for-python_mesonpy-class.patch

FAIL: test shortlog format: Commit shortlog (first line of commit message) should follow the format "<target>: <summary>" (test_mbox.TestMbox.test_shortlog_format)

PASS: pretest pylint (test_python_pylint.PyLint.pretest_pylint)
PASS: test Signed-off-by presence (test_mbox.TestMbox.test_signed_off_by_presence)
PASS: test author valid (test_mbox.TestMbox.test_author_valid)
PASS: test commit message presence (test_mbox.TestMbox.test_commit_message_presence)
PASS: test max line length (test_metadata.TestMetadata.test_max_line_length)
PASS: test mbox format (test_mbox.TestMbox.test_mbox_format)
PASS: test non-AUH upgrade (test_mbox.TestMbox.test_non_auh_upgrade)
PASS: test pylint (test_python_pylint.PyLint.test_pylint)
PASS: test shortlog length (test_mbox.TestMbox.test_shortlog_length)

SKIP: pretest src uri left files: No modified recipes, skipping pretest (test_metadata.TestMetadata.pretest_src_uri_left_files)
SKIP: test CVE check ignore: No modified recipes, skipping test (test_metadata.TestMetadata.test_cve_check_ignore)
SKIP: test CVE tag format: No new CVE patches introduced (test_patch.TestPatch.test_cve_tag_format)
SKIP: test Signed-off-by presence: No new CVE patches introduced (test_patch.TestPatch.test_signed_off_by_presence)
SKIP: test Upstream-Status presence: No new CVE patches introduced (test_patch.TestPatch.test_upstream_status_presence_format)
SKIP: test bugzilla entry format: No bug ID found (test_mbox.TestMbox.test_bugzilla_entry_format)
SKIP: test lic files chksum modified not mentioned: No modified recipes, skipping test (test_metadata.TestMetadata.test_lic_files_chksum_modified_not_mentioned)
SKIP: test lic files chksum presence: No added recipes, skipping test (test_metadata.TestMetadata.test_lic_files_chksum_presence)
SKIP: test license presence: No added recipes, skipping test (test_metadata.TestMetadata.test_license_presence)
SKIP: test series merge on head: Merge test is disabled for now (test_mbox.TestMbox.test_series_merge_on_head)
SKIP: test src uri left files: No modified recipes, skipping pretest (test_metadata.TestMetadata.test_src_uri_left_files)
SKIP: test summary presence: No added recipes, skipping test (test_metadata.TestMetadata.test_summary_presence)
SKIP: test target mailing list: Series merged, no reason to check other mailing lists (test_mbox.TestMbox.test_target_mailing_list)

---

Please address the issues identified and
submit a new revision of the patch, or alternatively, reply to this
email with an explanation of why the patch should be accepted. If you
believe these results are due to an error in patchtest, please submit a
bug at https://bugzilla.yoctoproject.org/ (use the 'Patchtest' category
under 'Yocto Project Subprojects'). For more information on specific
failures, see: https://wiki.yoctoproject.org/wiki/Patchtest. Thank
you!

Re: [OE-core] [PATCH 3/4] python3-numpy: inherit python_mesonpy

[Richard Purdie]

On Mon, 2024-02-19 at 09:30 -0800, Tim Orling wrote:
> Replace the deprecated setuptools3 bbclass with python_mesonpy.
> 
> The build-backend has been defined as "mesonpy" since:
> https://github.com/numpy/numpy/commit/942fb8caf33a65e449fbf198ecf1cd39be953248
> 
> The vendored meson-python was dropped in:
> https://github.com/numpy/numpy/commit/6544e33ac7a3a600c2fb565401c811a17ecdb3d5
> 
> While we are at it:
> * Drop ${PYTHON_PN} and use python3 instead
> * Sort RDEPENDS alphabetically
> 
> The ptests run, but we still have issues with sufficient memory and
> free disk space (the reason python3-numpy is in PTEST_PROBLEMS).
> 
> Signed-off-by: Tim Orling <tim.orling@konsulko.com>

This looks to introduce a reproducibility issue:

https://autobuilder.yoctoproject.org/typhoon/#/builders/117/builds/4399

Cheers,

Richard

Re: [OE-core] [PATCH 3/4] python3-numpy: inherit python_mesonpy

[Trevor Gamblin]

On 2024-02-20 06:33, Richard Purdie wrote:
> On Mon, 2024-02-19 at 09:30 -0800, Tim Orling wrote:
>> Replace the deprecated setuptools3 bbclass with python_mesonpy.
>>
>> The build-backend has been defined as "mesonpy" since:
>> https://github.com/numpy/numpy/commit/942fb8caf33a65e449fbf198ecf1cd39be953248
>>
>> The vendored meson-python was dropped in:
>> https://github.com/numpy/numpy/commit/6544e33ac7a3a600c2fb565401c811a17ecdb3d5
>>
>> While we are at it:
>> * Drop ${PYTHON_PN} and use python3 instead
>> * Sort RDEPENDS alphabetically
>>
>> The ptests run, but we still have issues with sufficient memory and
>> free disk space (the reason python3-numpy is in PTEST_PROBLEMS).
>>
>> Signed-off-by: Tim Orling <tim.orling@konsulko.com>
> This looks to introduce a reproducibility issue:
>
> https://autobuilder.yoctoproject.org/typhoon/#/builders/117/builds/4399

Hi,

I've filed a bug for this: 
https://bugzilla.yoctoproject.org/show_bug.cgi?id=15406

Let's stick with the existing backend for now and I'll fix it ASAP.

Trevor

>
> Cheers,
>
> Richard
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#195913): https://lists.openembedded.org/g/openembedded-core/message/195913
> Mute This Topic: https://lists.openembedded.org/mt/104451074/7611679
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [tgamblin@baylibre.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>