* [Buildroot] [PATCH v2 1/2] package/python-pybind: fix upgrade to version 2.6.1
@ 2021-12-06 19:31 guillaume.bressaix
2021-12-06 19:31 ` [Buildroot] [PATCH v2 2/2] package/python-pybind: bump to version 2.8.1 guillaume.bressaix
` (2 more replies)
0 siblings, 3 replies; 21+ messages in thread
From: guillaume.bressaix @ 2021-12-06 19:31 UTC (permalink / raw)
To: buildroot; +Cc: yann.morin.1998, Guillaume W. Bres
From: "Guillaume W. Bres" <guillaume.bressaix@gmail.com>
The python 'setup.py' script needs header files
in $(@D)/pybind11 to work since v2.6.1, and these
files are generated by an internal minimalist cmake build.
Fixes
http://autobuild.buildroot.net/results/b89f1de64b308dffa73675f1f31ccb0b7be5a10d
http://autobuild.buildroot.net/results/d0287b7f64f206b0f074908c5780a3632e0cb799
http://autobuild.buildroot.net/results/27efb545a5a719a5581c8f746d3a3555ff4216ce
http://autobuild.buildroot.net/results/d2f0a0ad8f6c7178517df109e7d885dac9134c3a
http://autobuild.buildroot.net/results/b57e9a3279260dae4a590f9421238fcabb2f7cab
http://autobuild.buildroot.net/results/515e6f2fc6b5780260d98d6bb52b541ce4bf1afe
http://autobuild.buildroot.net/results/d89c4ecc81222d4f80c951da2232d2e393fa1c69
Signed-off-by: Guillaume W. Bres <guillaume.bressaix@gmail.com>
---
This remains a python-package.
'python setup.py' actually hardcodes a system call to cmake now,
which seems dirty at first. But if we have a cmake env when they
do that, the install variables are passed and installation
is smooth. They probably introduced this call to make setup.py
self sufficient, as most people install this package from 'pip'.
---
package/python-pybind/python-pybind.mk | 7 +++++++
1 file changed, 7 insertions(+)
diff --git a/package/python-pybind/python-pybind.mk b/package/python-pybind/python-pybind.mk
index a6a1bdb976..bfd7f6f59a 100644
--- a/package/python-pybind/python-pybind.mk
+++ b/package/python-pybind/python-pybind.mk
@@ -9,5 +9,12 @@ PYTHON_PYBIND_SITE = $(call github,pybind,pybind11,v$(PYTHON_PYBIND_VERSION))
PYTHON_PYBIND_LICENSE = BSD-3-Clause
PYTHON_PYBIND_LICENSE_FILES = LICENSE
PYTHON_PYBIND_SETUP_TYPE = setuptools
+PYTHON_PYBIND_INSTALL_STAGING = YES
+
+# every single 'python setup.py' call actually calls cmake
+# internally, to populate $(@D)/pybind11 with build requirements.
+# If we have a host-cmake env at that moment,
+# then the final installation paths are properly defined
+PYTHON_PYBIND_DEPENDENCIES = host-cmake
$(eval $(python-package))
--
2.20.1
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot
^ permalink raw reply related [flat|nested] 21+ messages in thread* [Buildroot] [PATCH v2 2/2] package/python-pybind: bump to version 2.8.1 2021-12-06 19:31 [Buildroot] [PATCH v2 1/2] package/python-pybind: fix upgrade to version 2.6.1 guillaume.bressaix @ 2021-12-06 19:31 ` guillaume.bressaix 2021-12-06 21:45 ` [Buildroot] [PATCH v2 1/2] package/python-pybind: fix upgrade to version 2.6.1 Yann E. MORIN 2021-12-06 21:57 ` Yann E. MORIN 2 siblings, 0 replies; 21+ messages in thread From: guillaume.bressaix @ 2021-12-06 19:31 UTC (permalink / raw) To: buildroot; +Cc: yann.morin.1998, Guillaume W. Bres From: "Guillaume W. Bres" <guillaume.bressaix@gmail.com> Signed-off-by: Guillaume W. Bres <guillaume.bressaix@gmail.com> --- package/python-pybind/python-pybind.hash | 2 +- package/python-pybind/python-pybind.mk | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/package/python-pybind/python-pybind.hash b/package/python-pybind/python-pybind.hash index a68ac846e2..e1d86a4494 100644 --- a/package/python-pybind/python-pybind.hash +++ b/package/python-pybind/python-pybind.hash @@ -1,4 +1,4 @@ # Locally calculated -sha256 cdbe326d357f18b83d10322ba202d69f11b2f49e2d87ade0dc2be0c5c34f8e2a python-pybind-2.6.1.tar.gz +sha256 f1bcc07caa568eb312411dde5308b1e250bd0e1bc020fae855bf9f43209940cc python-pybind-2.8.1.tar.gz # License files, locally calculated sha256 83965b843b98f670d3a85bd041ed4b372c8ec50d7b4a5995a83ac697ba675dcb LICENSE diff --git a/package/python-pybind/python-pybind.mk b/package/python-pybind/python-pybind.mk index bfd7f6f59a..3aee9cf1e6 100644 --- a/package/python-pybind/python-pybind.mk +++ b/package/python-pybind/python-pybind.mk @@ -4,7 +4,7 @@ # ################################################################################ -PYTHON_PYBIND_VERSION = 2.6.1 +PYTHON_PYBIND_VERSION = 2.8.1 PYTHON_PYBIND_SITE = $(call github,pybind,pybind11,v$(PYTHON_PYBIND_VERSION)) PYTHON_PYBIND_LICENSE = BSD-3-Clause PYTHON_PYBIND_LICENSE_FILES = LICENSE -- 2.20.1 _______________________________________________ buildroot mailing list buildroot@buildroot.org https://lists.buildroot.org/mailman/listinfo/buildroot ^ permalink raw reply related [flat|nested] 21+ messages in thread
* Re: [Buildroot] [PATCH v2 1/2] package/python-pybind: fix upgrade to version 2.6.1 2021-12-06 19:31 [Buildroot] [PATCH v2 1/2] package/python-pybind: fix upgrade to version 2.6.1 guillaume.bressaix 2021-12-06 19:31 ` [Buildroot] [PATCH v2 2/2] package/python-pybind: bump to version 2.8.1 guillaume.bressaix @ 2021-12-06 21:45 ` Yann E. MORIN 2021-12-08 9:09 ` Arnout Vandecappelle 2021-12-06 21:57 ` Yann E. MORIN 2 siblings, 1 reply; 21+ messages in thread From: Yann E. MORIN @ 2021-12-06 21:45 UTC (permalink / raw) To: guillaume.bressaix; +Cc: buildroot Guillaume, All, On 2021-12-06 20:31 +0100, guillaume.bressaix@gmail.com spake thusly: > From: "Guillaume W. Bres" <guillaume.bressaix@gmail.com> > > The python 'setup.py' script needs header files > in $(@D)/pybind11 to work since v2.6.1, and these > files are generated by an internal minimalist cmake build. > > Fixes > http://autobuild.buildroot.net/results/b89f1de64b308dffa73675f1f31ccb0b7be5a10d > http://autobuild.buildroot.net/results/d0287b7f64f206b0f074908c5780a3632e0cb799 > http://autobuild.buildroot.net/results/27efb545a5a719a5581c8f746d3a3555ff4216ce > http://autobuild.buildroot.net/results/d2f0a0ad8f6c7178517df109e7d885dac9134c3a > http://autobuild.buildroot.net/results/b57e9a3279260dae4a590f9421238fcabb2f7cab > http://autobuild.buildroot.net/results/515e6f2fc6b5780260d98d6bb52b541ce4bf1afe > http://autobuild.buildroot.net/results/d89c4ecc81222d4f80c951da2232d2e393fa1c69 > > Signed-off-by: Guillaume W. Bres <guillaume.bressaix@gmail.com> > > --- > This remains a python-package. > > 'python setup.py' actually hardcodes a system call to cmake now, > which seems dirty at first. But if we have a cmake env when they > do that, the install variables are passed and installation > is smooth. They probably introduced this call to make setup.py > self sufficient, as most people install this package from 'pip'. > > --- > package/python-pybind/python-pybind.mk | 7 +++++++ > 1 file changed, 7 insertions(+) > > diff --git a/package/python-pybind/python-pybind.mk b/package/python-pybind/python-pybind.mk > index a6a1bdb976..bfd7f6f59a 100644 > --- a/package/python-pybind/python-pybind.mk > +++ b/package/python-pybind/python-pybind.mk > @@ -9,5 +9,12 @@ PYTHON_PYBIND_SITE = $(call github,pybind,pybind11,v$(PYTHON_PYBIND_VERSION)) > PYTHON_PYBIND_LICENSE = BSD-3-Clause > PYTHON_PYBIND_LICENSE_FILES = LICENSE > PYTHON_PYBIND_SETUP_TYPE = setuptools > +PYTHON_PYBIND_INSTALL_STAGING = YES > + > +# every single 'python setup.py' call actually calls cmake > +# internally, to populate $(@D)/pybind11 with build requirements. > +# If we have a host-cmake env at that moment, > +# then the final installation paths are properly defined > +PYTHON_PYBIND_DEPENDENCIES = host-cmake We do not hard-code a dependency to host-cmake, in case the system cmake is sufficient. Instead, we use $(BR2_CMAKE_HOST_DEPENDENCY) which is set appropriately. Regards, Yann E. MORIN. > $(eval $(python-package)) > -- > 2.20.1 > -- .-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------' _______________________________________________ buildroot mailing list buildroot@buildroot.org https://lists.buildroot.org/mailman/listinfo/buildroot ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [Buildroot] [PATCH v2 1/2] package/python-pybind: fix upgrade to version 2.6.1 2021-12-06 21:45 ` [Buildroot] [PATCH v2 1/2] package/python-pybind: fix upgrade to version 2.6.1 Yann E. MORIN @ 2021-12-08 9:09 ` Arnout Vandecappelle 0 siblings, 0 replies; 21+ messages in thread From: Arnout Vandecappelle @ 2021-12-08 9:09 UTC (permalink / raw) To: Yann E. MORIN, guillaume.bressaix; +Cc: buildroot On 06/12/2021 22:45, Yann E. MORIN wrote: > Guillaume, All, > > On 2021-12-06 20:31 +0100, guillaume.bressaix@gmail.com spake thusly: >> From: "Guillaume W. Bres" <guillaume.bressaix@gmail.com> >> >> The python 'setup.py' script needs header files >> in $(@D)/pybind11 to work since v2.6.1, and these >> files are generated by an internal minimalist cmake build. >> >> Fixes >> http://autobuild.buildroot.net/results/b89f1de64b308dffa73675f1f31ccb0b7be5a10d >> http://autobuild.buildroot.net/results/d0287b7f64f206b0f074908c5780a3632e0cb799 >> http://autobuild.buildroot.net/results/27efb545a5a719a5581c8f746d3a3555ff4216ce >> http://autobuild.buildroot.net/results/d2f0a0ad8f6c7178517df109e7d885dac9134c3a >> http://autobuild.buildroot.net/results/b57e9a3279260dae4a590f9421238fcabb2f7cab >> http://autobuild.buildroot.net/results/515e6f2fc6b5780260d98d6bb52b541ce4bf1afe >> http://autobuild.buildroot.net/results/d89c4ecc81222d4f80c951da2232d2e393fa1c69 >> >> Signed-off-by: Guillaume W. Bres <guillaume.bressaix@gmail.com> >> >> --- >> This remains a python-package. >> >> 'python setup.py' actually hardcodes a system call to cmake now, >> which seems dirty at first. But if we have a cmake env when they >> do that, the install variables are passed and installation >> is smooth. They probably introduced this call to make setup.py >> self sufficient, as most people install this package from 'pip'. >> >> --- >> package/python-pybind/python-pybind.mk | 7 +++++++ >> 1 file changed, 7 insertions(+) >> >> diff --git a/package/python-pybind/python-pybind.mk b/package/python-pybind/python-pybind.mk >> index a6a1bdb976..bfd7f6f59a 100644 >> --- a/package/python-pybind/python-pybind.mk >> +++ b/package/python-pybind/python-pybind.mk >> @@ -9,5 +9,12 @@ PYTHON_PYBIND_SITE = $(call github,pybind,pybind11,v$(PYTHON_PYBIND_VERSION)) >> PYTHON_PYBIND_LICENSE = BSD-3-Clause >> PYTHON_PYBIND_LICENSE_FILES = LICENSE >> PYTHON_PYBIND_SETUP_TYPE = setuptools >> +PYTHON_PYBIND_INSTALL_STAGING = YES >> + >> +# every single 'python setup.py' call actually calls cmake >> +# internally, to populate $(@D)/pybind11 with build requirements. >> +# If we have a host-cmake env at that moment, >> +# then the final installation paths are properly defined >> +PYTHON_PYBIND_DEPENDENCIES = host-cmake > > We do not hard-code a dependency to host-cmake, in case the system cmake > is sufficient. Instead, we use $(BR2_CMAKE_HOST_DEPENDENCY) which is set > appropriately. This turns out not to work in Guillaume's test setup. It turns out that he has a preinstalled cmake which is very old, but also cmake3 which is recent enough. Therefore, BR2_CMAKE is set to /usr/bin/cmake3 and BR2_CMAKE_HOST_DEPENDENCY is empty. However, setup.py hardcodes "cmake". So, possible solutions: - Forget about setup.py and instead hack it into a cmake-package. Not ideal from a maintenance perspective, because later updates may have changes to setup.py that we need to replicate. - Patch setup.py so we can tell it to use BR2_CMAKE instead of cmake-in-path. This is nice because we can use the occasion to also pass in the rest of the cmake options for proper cross-compilation. - In host-skeleton, symlink BR2_CMAKE to host/bin/cmake, so that will be the 'cmake' that is found in $PATH. This has the advantage of also covering other potential cases of the same problem, but I feel it's a bit meh. So I have a slight preference for the second option, but Yann you may have other ideas. Regards, Arnout _______________________________________________ buildroot mailing list buildroot@buildroot.org https://lists.buildroot.org/mailman/listinfo/buildroot ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [Buildroot] [PATCH v2 1/2] package/python-pybind: fix upgrade to version 2.6.1 2021-12-06 19:31 [Buildroot] [PATCH v2 1/2] package/python-pybind: fix upgrade to version 2.6.1 guillaume.bressaix 2021-12-06 19:31 ` [Buildroot] [PATCH v2 2/2] package/python-pybind: bump to version 2.8.1 guillaume.bressaix 2021-12-06 21:45 ` [Buildroot] [PATCH v2 1/2] package/python-pybind: fix upgrade to version 2.6.1 Yann E. MORIN @ 2021-12-06 21:57 ` Yann E. MORIN 2021-12-07 8:04 ` Arnout Vandecappelle 2022-01-03 8:24 ` Tomi Valkeinen 2 siblings, 2 replies; 21+ messages in thread From: Yann E. MORIN @ 2021-12-06 21:57 UTC (permalink / raw) To: guillaume.bressaix; +Cc: buildroot Guillaume, All, On 2021-12-06 20:31 +0100, guillaume.bressaix@gmail.com spake thusly: > From: "Guillaume W. Bres" <guillaume.bressaix@gmail.com> > > The python 'setup.py' script needs header files > in $(@D)/pybind11 to work since v2.6.1, and these > files are generated by an internal minimalist cmake build. > > Fixes > http://autobuild.buildroot.net/results/b89f1de64b308dffa73675f1f31ccb0b7be5a10d > http://autobuild.buildroot.net/results/d0287b7f64f206b0f074908c5780a3632e0cb799 > http://autobuild.buildroot.net/results/27efb545a5a719a5581c8f746d3a3555ff4216ce > http://autobuild.buildroot.net/results/d2f0a0ad8f6c7178517df109e7d885dac9134c3a > http://autobuild.buildroot.net/results/b57e9a3279260dae4a590f9421238fcabb2f7cab > http://autobuild.buildroot.net/results/515e6f2fc6b5780260d98d6bb52b541ce4bf1afe > http://autobuild.buildroot.net/results/d89c4ecc81222d4f80c951da2232d2e393fa1c69 > > Signed-off-by: Guillaume W. Bres <guillaume.bressaix@gmail.com> > > --- > This remains a python-package. > > 'python setup.py' actually hardcodes a system call to cmake now, > which seems dirty at first. But if we have a cmake env when they > do that, the install variables are passed and installation > is smooth. They probably introduced this call to make setup.py > self sufficient, as most people install this package from 'pip'. > > --- > package/python-pybind/python-pybind.mk | 7 +++++++ > 1 file changed, 7 insertions(+) > > diff --git a/package/python-pybind/python-pybind.mk b/package/python-pybind/python-pybind.mk > index a6a1bdb976..bfd7f6f59a 100644 > --- a/package/python-pybind/python-pybind.mk > +++ b/package/python-pybind/python-pybind.mk > @@ -9,5 +9,12 @@ PYTHON_PYBIND_SITE = $(call github,pybind,pybind11,v$(PYTHON_PYBIND_VERSION)) > PYTHON_PYBIND_LICENSE = BSD-3-Clause > PYTHON_PYBIND_LICENSE_FILES = LICENSE > PYTHON_PYBIND_SETUP_TYPE = setuptools > +PYTHON_PYBIND_INSTALL_STAGING = YES > + > +# every single 'python setup.py' call actually calls cmake > +# internally, to populate $(@D)/pybind11 with build requirements. > +# If we have a host-cmake env at that moment, > +# then the final installation paths are properly defined > +PYTHON_PYBIND_DEPENDENCIES = host-cmake I am sorry, but I fail to see how this actually fixes things. As I explained in my previous review, headers are supposed to be in $(STAGING_DIR)/usr/include/pybind11/. That's at least where they are in Debian-based distributions; $ apt-file search 'pybind11/pybind11.h' pybind11-dev: /usr/include/pybind11/pybind11.h However, with the way pybind is currently packaged, the ehaders end up in: $ sort output/build/python-pybind-2.8.1/.files-list.txt python-pybind,./usr/bin/pybind11-config [...] python-pybind,./usr/lib/python3.9/site-packages/pybind11/include/pybind11/pybind11.h [...] So, those headers will most probably not be found by the compiler. Also, since pybind is supposed to be a headers'-only library, there is nothing supposed to go in target, so it should be; PYTHON_PYBIND_INSTALL_TARGET = NO Furthermore, /usr/bin/pybind11-config is a python script which hardcodes its shabang to #!/usr/bin/python, and this is incorrect: 1. /usr/bin/python might be python2 or python3, we can't know 2. anyway, it would not use our host python, so it would miss the proper modules search paths. So, if another package want to call that /usr/bin/pybind11-config to find pybind, they're gonna have incorrect results, if at all... So I think we really, really need a package (like scipy) that actually makes use of pybind, so we can see how it works, and to fix it... Regards, Yann E. MORIN. > $(eval $(python-package)) > -- > 2.20.1 > -- .-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------' _______________________________________________ buildroot mailing list buildroot@buildroot.org https://lists.buildroot.org/mailman/listinfo/buildroot ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [Buildroot] [PATCH v2 1/2] package/python-pybind: fix upgrade to version 2.6.1 2021-12-06 21:57 ` Yann E. MORIN @ 2021-12-07 8:04 ` Arnout Vandecappelle 2021-12-08 11:19 ` Andreas Naumann 2021-12-08 17:18 ` Yann E. MORIN 2022-01-03 8:24 ` Tomi Valkeinen 1 sibling, 2 replies; 21+ messages in thread From: Arnout Vandecappelle @ 2021-12-07 8:04 UTC (permalink / raw) To: Yann E. MORIN, guillaume.bressaix; +Cc: buildroot On 06/12/2021 22:57, Yann E. MORIN wrote: > Guillaume, All, > > On 2021-12-06 20:31 +0100, guillaume.bressaix@gmail.com spake thusly: >> From: "Guillaume W. Bres" <guillaume.bressaix@gmail.com> >> >> The python 'setup.py' script needs header files >> in $(@D)/pybind11 to work since v2.6.1, and these >> files are generated by an internal minimalist cmake build. >> >> Fixes >> http://autobuild.buildroot.net/results/b89f1de64b308dffa73675f1f31ccb0b7be5a10d >> http://autobuild.buildroot.net/results/d0287b7f64f206b0f074908c5780a3632e0cb799 >> http://autobuild.buildroot.net/results/27efb545a5a719a5581c8f746d3a3555ff4216ce >> http://autobuild.buildroot.net/results/d2f0a0ad8f6c7178517df109e7d885dac9134c3a >> http://autobuild.buildroot.net/results/b57e9a3279260dae4a590f9421238fcabb2f7cab >> http://autobuild.buildroot.net/results/515e6f2fc6b5780260d98d6bb52b541ce4bf1afe >> http://autobuild.buildroot.net/results/d89c4ecc81222d4f80c951da2232d2e393fa1c69 >> >> Signed-off-by: Guillaume W. Bres <guillaume.bressaix@gmail.com> >> >> --- >> This remains a python-package. >> >> 'python setup.py' actually hardcodes a system call to cmake now, >> which seems dirty at first. But if we have a cmake env when they >> do that, the install variables are passed and installation >> is smooth. They probably introduced this call to make setup.py >> self sufficient, as most people install this package from 'pip'. >> >> --- >> package/python-pybind/python-pybind.mk | 7 +++++++ >> 1 file changed, 7 insertions(+) >> >> diff --git a/package/python-pybind/python-pybind.mk b/package/python-pybind/python-pybind.mk >> index a6a1bdb976..bfd7f6f59a 100644 >> --- a/package/python-pybind/python-pybind.mk >> +++ b/package/python-pybind/python-pybind.mk >> @@ -9,5 +9,12 @@ PYTHON_PYBIND_SITE = $(call github,pybind,pybind11,v$(PYTHON_PYBIND_VERSION)) >> PYTHON_PYBIND_LICENSE = BSD-3-Clause >> PYTHON_PYBIND_LICENSE_FILES = LICENSE >> PYTHON_PYBIND_SETUP_TYPE = setuptools >> +PYTHON_PYBIND_INSTALL_STAGING = YES >> + >> +# every single 'python setup.py' call actually calls cmake >> +# internally, to populate $(@D)/pybind11 with build requirements. >> +# If we have a host-cmake env at that moment, >> +# then the final installation paths are properly defined >> +PYTHON_PYBIND_DEPENDENCIES = host-cmake > > I am sorry, but I fail to see how this actually fixes things. > > As I explained in my previous review, headers are supposed to be in > $(STAGING_DIR)/usr/include/pybind11/. That's at least where they are in > Debian-based distributions; > > $ apt-file search 'pybind11/pybind11.h' > pybind11-dev: /usr/include/pybind11/pybind11.h > > However, with the way pybind is currently packaged, the ehaders end up > in: > > $ sort output/build/python-pybind-2.8.1/.files-list.txt > python-pybind,./usr/bin/pybind11-config > [...] > python-pybind,./usr/lib/python3.9/site-packages/pybind11/include/pybind11/pybind11.h > [...] > > So, those headers will most probably not be found by the compiler. > > Also, since pybind is supposed to be a headers'-only library, there is > nothing supposed to go in target, so it should be; > > PYTHON_PYBIND_INSTALL_TARGET = NO Note that this is only relevant to make sure that pybind11-config doesn't appear on target. The headers themselves are removed automatically by Buildroot. > > Furthermore, /usr/bin/pybind11-config is a python script which hardcodes > its shabang to #!/usr/bin/python, and this is incorrect: > > 1. /usr/bin/python might be python2 or python3, we can't know > 2. anyway, it would not use our host python, so it would miss the > proper modules search paths. Yeah, this is something that needs to be fixed - in particular, there may not be a /usr/bin/python at all. However, I think that is kind of orthogonal to this patch. This patch fixes a build failure of pybind itself. Your proposed change would be to fix the build of a dependency of pybind. > So, if another package want to call that /usr/bin/pybind11-config to > find pybind, they're gonna have incorrect results, if at all... > > So I think we really, really need a package (like scipy) that actually > makes use of pybind, so we can see how it works, and to fix it... Note that there are already two [1][2] submissions of scipy in patchwork, so feel free to review those. Note also that scipy depends on host-pybind, not pybind itself. I'm not sure how *that* is supposed to work - although, being header-only, it probably doesn't really matter if you link with the host or the target version of the library, and the host-installed pybind11-config makes sure that you pick up the host headers (which are normally not in include path). So conceptually wrong but works in practice. In conclusion: I think this patch is going in the right direction (modulo BR2_CMAKE_HOST_DEPENDENCY). It's still "conceptually wrong but works" because it doesn't set the toolchain file and other cmake options, but that happens to work because it's header-only so it doesn't actually need any of the cmake options. So it's not entirely the correct thing to do, but it does guarantee to fix things. I'm not sure if that means we should merge it or not... The proper fix, I think, would require patching setup.py so we can pass in additional cmake options to it. Regards, Arnout [1] https://patchwork.ozlabs.org/project/buildroot/list/?series=168992 [2] https://patchwork.ozlabs.org/project/buildroot/list/?series=186146 > > Regards, > Yann E. MORIN. > >> $(eval $(python-package)) >> -- >> 2.20.1 >> > _______________________________________________ buildroot mailing list buildroot@buildroot.org https://lists.buildroot.org/mailman/listinfo/buildroot ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [Buildroot] [PATCH v2 1/2] package/python-pybind: fix upgrade to version 2.6.1 2021-12-07 8:04 ` Arnout Vandecappelle @ 2021-12-08 11:19 ` Andreas Naumann 2021-12-08 12:15 ` Guillaume Bres 2022-01-07 21:55 ` Arnout Vandecappelle 2021-12-08 17:18 ` Yann E. MORIN 1 sibling, 2 replies; 21+ messages in thread From: Andreas Naumann @ 2021-12-08 11:19 UTC (permalink / raw) To: Arnout Vandecappelle, Yann E. MORIN, guillaume.bressaix; +Cc: buildroot Hi all, On 07.12.21 09:04, Arnout Vandecappelle wrote: > > > On 06/12/2021 22:57, Yann E. MORIN wrote: >> Guillaume, All, >> >> On 2021-12-06 20:31 +0100, guillaume.bressaix@gmail.com spake thusly: >>> From: "Guillaume W. Bres" <guillaume.bressaix@gmail.com> >>> >>> The python 'setup.py' script needs header files >>> in $(@D)/pybind11 to work since v2.6.1, and these >>> files are generated by an internal minimalist cmake build. >>> >>> Fixes >>> >>> http://autobuild.buildroot.net/results/b89f1de64b308dffa73675f1f31ccb0b7be5a10d >>> >>> >>> http://autobuild.buildroot.net/results/d0287b7f64f206b0f074908c5780a3632e0cb799 >>> >>> >>> http://autobuild.buildroot.net/results/27efb545a5a719a5581c8f746d3a3555ff4216ce >>> >>> >>> http://autobuild.buildroot.net/results/d2f0a0ad8f6c7178517df109e7d885dac9134c3a >>> >>> >>> http://autobuild.buildroot.net/results/b57e9a3279260dae4a590f9421238fcabb2f7cab >>> >>> >>> http://autobuild.buildroot.net/results/515e6f2fc6b5780260d98d6bb52b541ce4bf1afe >>> >>> >>> http://autobuild.buildroot.net/results/d89c4ecc81222d4f80c951da2232d2e393fa1c69 >>> >>> >>> Signed-off-by: Guillaume W. Bres <guillaume.bressaix@gmail.com> >>> >>> --- >>> This remains a python-package. >>> >>> 'python setup.py' actually hardcodes a system call to cmake now, >>> which seems dirty at first. But if we have a cmake env when they >>> do that, the install variables are passed and installation >>> is smooth. They probably introduced this call to make setup.py >>> self sufficient, as most people install this package from 'pip'. >>> >>> --- >>> package/python-pybind/python-pybind.mk | 7 +++++++ >>> 1 file changed, 7 insertions(+) >>> >>> diff --git a/package/python-pybind/python-pybind.mk >>> b/package/python-pybind/python-pybind.mk >>> index a6a1bdb976..bfd7f6f59a 100644 >>> --- a/package/python-pybind/python-pybind.mk >>> +++ b/package/python-pybind/python-pybind.mk >>> @@ -9,5 +9,12 @@ PYTHON_PYBIND_SITE = $(call >>> github,pybind,pybind11,v$(PYTHON_PYBIND_VERSION)) >>> PYTHON_PYBIND_LICENSE = BSD-3-Clause >>> PYTHON_PYBIND_LICENSE_FILES = LICENSE >>> PYTHON_PYBIND_SETUP_TYPE = setuptools >>> +PYTHON_PYBIND_INSTALL_STAGING = YES >>> + >>> +# every single 'python setup.py' call actually calls cmake >>> +# internally, to populate $(@D)/pybind11 with build requirements. >>> +# If we have a host-cmake env at that moment, >>> +# then the final installation paths are properly defined >>> +PYTHON_PYBIND_DEPENDENCIES = host-cmake >> >> I am sorry, but I fail to see how this actually fixes things. >> >> As I explained in my previous review, headers are supposed to be in >> $(STAGING_DIR)/usr/include/pybind11/. That's at least where they are in >> Debian-based distributions; >> >> $ apt-file search 'pybind11/pybind11.h' >> pybind11-dev: /usr/include/pybind11/pybind11.h >> >> However, with the way pybind is currently packaged, the ehaders end up >> in: >> >> $ sort output/build/python-pybind-2.8.1/.files-list.txt >> python-pybind,./usr/bin/pybind11-config >> [...] >> >> python-pybind,./usr/lib/python3.9/site-packages/pybind11/include/pybind11/pybind11.h >> >> [...] >> >> So, those headers will most probably not be found by the compiler. >> >> Also, since pybind is supposed to be a headers'-only library, there is >> nothing supposed to go in target, so it should be; >> >> PYTHON_PYBIND_INSTALL_TARGET = NO > > Note that this is only relevant to make sure that pybind11-config > doesn't appear on target. The headers themselves are removed > automatically by Buildroot. > >> >> Furthermore, /usr/bin/pybind11-config is a python script which hardcodes >> its shabang to #!/usr/bin/python, and this is incorrect: >> >> 1. /usr/bin/python might be python2 or python3, we can't know >> 2. anyway, it would not use our host python, so it would miss the >> proper modules search paths. > > Yeah, this is something that needs to be fixed - in particular, there > may not be a /usr/bin/python at all. > > However, I think that is kind of orthogonal to this patch. This patch > fixes a build failure of pybind itself. Your proposed change would be to > fix the build of a dependency of pybind. > > >> So, if another package want to call that /usr/bin/pybind11-config to >> find pybind, they're gonna have incorrect results, if at all... >> >> So I think we really, really need a package (like scipy) that actually >> makes use of pybind, so we can see how it works, and to fix it... By chance I was playing with such a package just yesterday: zxing-cpp. Not the version buildroot uses (from github glassechidna), but the nu-book one which claims to be faster. (And is the one where the python bindings in the PyPi package zxing-cpp originate from). Until recently they shipped their own pybind11 version, which throws an error when included in a cmake project in a crosscompile environment. This is a known problem, https://github.com/pybind/pybind11/issues/2139 but non of the proposed fixes has been merged yet. However, the unreleased master version of zxing-cpp hast been changed so it can be configured to use the a system pybind11. So I tried to include the br packaged one (2.6.1 as well as 2.8.1). They both built on my system (no problem with the hardcoded host cmake here) but I needed to change three things: 1. Add PYTHON_PYBIND_INSTALL_STAGING = YES 2. Set ZXING_CPP_CONF_OPTS += -Dpybind11_DIR=$(STAGING_DIR)/usr/lib/python$(PYTHON3_VERSION_MAJOR)/site-packages/pybind11/share/cmake/pybind11 3. Apply one of the proposed fixes for the pybind11 crosscompile problem (to tools/pybind11Tools.cmake) So this patch here seems to help my (and possibly others) because 1. is already included. If interested, I could send a patch for 3. > > Note that there are already two [1][2] submissions of scipy in > patchwork, so feel free to review those. > > Note also that scipy depends on host-pybind, not pybind itself. I'm > not sure how *that* is supposed to work - although, being header-only, One assumption why scipy includes the host version of pybind is that it nicely avoids the problems where I need fix 1. and 3. for. > it probably doesn't really matter if you link with the host or the > target version of the library, and the host-installed pybind11-config > makes sure that you pick up the host headers (which are normally not in > include path). So conceptually wrong but works in practice. For getting conceptually right with my zxing-cpp I would appreciate some advice: When compiling it with above modificiations and BUILD_PYTHON_MODULE=ON, cmake happily compiles the zxing pybind wrapper. But of course wrappers/python/setup.py would need to be run as well. However mixing $(eval $(python-package)) and $(eval $(cmake-package)) is not possible. So do I still need to create a second package python-zxingcpp for the bindings only (which might be a pain because the wrappers CMakeLists is not written to use an existant installation of zxing-cpp but rather downloads it again) ? regards, Andreas > > > In conclusion: I think this patch is going in the right direction > (modulo BR2_CMAKE_HOST_DEPENDENCY). It's still "conceptually wrong but > works" because it doesn't set the toolchain file and other cmake > options, but that happens to work because it's header-only so it doesn't > actually need any of the cmake options. So it's not entirely the correct > thing to do, but it does guarantee to fix things. I'm not sure if that > means we should merge it or not... > > The proper fix, I think, would require patching setup.py so we can > pass in additional cmake options to it. > > > Regards, > Arnout > > [1] https://patchwork.ozlabs.org/project/buildroot/list/?series=168992 > [2] https://patchwork.ozlabs.org/project/buildroot/list/?series=186146 So > >> >> Regards, >> Yann E. MORIN. >> >>> $(eval $(python-package)) >>> -- >>> 2.20.1 >>> >> > _______________________________________________ > buildroot mailing list > buildroot@buildroot.org > https://lists.buildroot.org/mailman/listinfo/buildroot > _______________________________________________ buildroot mailing list buildroot@buildroot.org https://lists.buildroot.org/mailman/listinfo/buildroot ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [Buildroot] [PATCH v2 1/2] package/python-pybind: fix upgrade to version 2.6.1 2021-12-08 11:19 ` Andreas Naumann @ 2021-12-08 12:15 ` Guillaume Bres 2022-01-07 21:55 ` Arnout Vandecappelle 1 sibling, 0 replies; 21+ messages in thread From: Guillaume Bres @ 2021-12-08 12:15 UTC (permalink / raw) To: Arnout Vandecappelle, dev; +Cc: buildroot, Yann E. MORIN [-- Attachment #1.1: Type: text/plain, Size: 10062 bytes --] Arnout, I also prefer your second option. I need to run some more tests & inquire what Yann proposed on IRC. This might take a couple of days, I am not submitting until I reach a solution, but will keep in touch on IRC. At least I understand what is going on in all scenarios now Andreas, >They both built on my >but I needed to change three things: >1. Add PYTHON_PYBIND_INSTALL_STAGING = YES >2. Set ZXING_CPP_CONF_OPTS += >-Dpybind11_DIR=$(STAGING_DIR)/usr/lib/python$(PYTHON3_VERSION_MAJOR)/site-packages/pybind11/share/cmake/pybind11 >3. Apply one of the proposed fixes for the pybind11 crosscompile problem >(to tools/pybind11Tools.cmake) Once I come to a solution, my next contribution will allow to install pybind to staging. At that time we should have a solution good enough so other packages will no longer have to internally ship pybind. >Apply one of the proposed fixes for the pybind11 crosscompile problem (to tools/pybind11Tools.cmake) We should keep an eye on the pybind devs topics you pointed out. We might have a near future solution, but it's probably going to need to involve (hopefuly sanitized/simplified) in also a near future Guillaume W. Bres Software engineer <guillaume.bressaix@gmail.com> Le mer. 8 déc. 2021 à 12:19, Andreas Naumann <dev@andin.de> a écrit : > Hi all, > > On 07.12.21 09:04, Arnout Vandecappelle wrote: > > > > > > On 06/12/2021 22:57, Yann E. MORIN wrote: > >> Guillaume, All, > >> > >> On 2021-12-06 20:31 +0100, guillaume.bressaix@gmail.com spake thusly: > >>> From: "Guillaume W. Bres" <guillaume.bressaix@gmail.com> > >>> > >>> The python 'setup.py' script needs header files > >>> in $(@D)/pybind11 to work since v2.6.1, and these > >>> files are generated by an internal minimalist cmake build. > >>> > >>> Fixes > >>> > >>> > http://autobuild.buildroot.net/results/b89f1de64b308dffa73675f1f31ccb0b7be5a10d > >>> > >>> > >>> > http://autobuild.buildroot.net/results/d0287b7f64f206b0f074908c5780a3632e0cb799 > >>> > >>> > >>> > http://autobuild.buildroot.net/results/27efb545a5a719a5581c8f746d3a3555ff4216ce > >>> > >>> > >>> > http://autobuild.buildroot.net/results/d2f0a0ad8f6c7178517df109e7d885dac9134c3a > >>> > >>> > >>> > http://autobuild.buildroot.net/results/b57e9a3279260dae4a590f9421238fcabb2f7cab > >>> > >>> > >>> > http://autobuild.buildroot.net/results/515e6f2fc6b5780260d98d6bb52b541ce4bf1afe > >>> > >>> > >>> > http://autobuild.buildroot.net/results/d89c4ecc81222d4f80c951da2232d2e393fa1c69 > >>> > >>> > >>> Signed-off-by: Guillaume W. Bres <guillaume.bressaix@gmail.com> > >>> > >>> --- > >>> This remains a python-package. > >>> > >>> 'python setup.py' actually hardcodes a system call to cmake now, > >>> which seems dirty at first. But if we have a cmake env when they > >>> do that, the install variables are passed and installation > >>> is smooth. They probably introduced this call to make setup.py > >>> self sufficient, as most people install this package from 'pip'. > >>> > >>> --- > >>> package/python-pybind/python-pybind.mk | 7 +++++++ > >>> 1 file changed, 7 insertions(+) > >>> > >>> diff --git a/package/python-pybind/python-pybind.mk > >>> b/package/python-pybind/python-pybind.mk > >>> index a6a1bdb976..bfd7f6f59a 100644 > >>> --- a/package/python-pybind/python-pybind.mk > >>> +++ b/package/python-pybind/python-pybind.mk > >>> @@ -9,5 +9,12 @@ PYTHON_PYBIND_SITE = $(call > >>> github,pybind,pybind11,v$(PYTHON_PYBIND_VERSION)) > >>> PYTHON_PYBIND_LICENSE = BSD-3-Clause > >>> PYTHON_PYBIND_LICENSE_FILES = LICENSE > >>> PYTHON_PYBIND_SETUP_TYPE = setuptools > >>> +PYTHON_PYBIND_INSTALL_STAGING = YES > >>> + > >>> +# every single 'python setup.py' call actually calls cmake > >>> +# internally, to populate $(@D)/pybind11 with build requirements. > >>> +# If we have a host-cmake env at that moment, > >>> +# then the final installation paths are properly defined > >>> +PYTHON_PYBIND_DEPENDENCIES = host-cmake > >> > >> I am sorry, but I fail to see how this actually fixes things. > >> > >> As I explained in my previous review, headers are supposed to be in > >> $(STAGING_DIR)/usr/include/pybind11/. That's at least where they are in > >> Debian-based distributions; > >> > >> $ apt-file search 'pybind11/pybind11.h' > >> pybind11-dev: /usr/include/pybind11/pybind11.h > >> > >> However, with the way pybind is currently packaged, the ehaders end up > >> in: > >> > >> $ sort output/build/python-pybind-2.8.1/.files-list.txt > >> python-pybind,./usr/bin/pybind11-config > >> [...] > >> > >> > python-pybind,./usr/lib/python3.9/site-packages/pybind11/include/pybind11/pybind11.h > > >> > >> [...] > >> > >> So, those headers will most probably not be found by the compiler. > >> > >> Also, since pybind is supposed to be a headers'-only library, there is > >> nothing supposed to go in target, so it should be; > >> > >> PYTHON_PYBIND_INSTALL_TARGET = NO > > > > Note that this is only relevant to make sure that pybind11-config > > doesn't appear on target. The headers themselves are removed > > automatically by Buildroot. > > > >> > >> Furthermore, /usr/bin/pybind11-config is a python script which hardcodes > >> its shabang to #!/usr/bin/python, and this is incorrect: > >> > >> 1. /usr/bin/python might be python2 or python3, we can't know > >> 2. anyway, it would not use our host python, so it would miss the > >> proper modules search paths. > > > > Yeah, this is something that needs to be fixed - in particular, there > > may not be a /usr/bin/python at all. > > > > However, I think that is kind of orthogonal to this patch. This patch > > fixes a build failure of pybind itself. Your proposed change would be to > > fix the build of a dependency of pybind. > > > > > >> So, if another package want to call that /usr/bin/pybind11-config to > >> find pybind, they're gonna have incorrect results, if at all... > >> > >> So I think we really, really need a package (like scipy) that actually > >> makes use of pybind, so we can see how it works, and to fix it... > > By chance I was playing with such a package just yesterday: zxing-cpp. > Not the version buildroot uses (from github glassechidna), but the > nu-book one which claims to be faster. (And is the one where the python > bindings in the PyPi package zxing-cpp originate from). > > Until recently they shipped their own pybind11 version, which throws an > error when included in a cmake project in a crosscompile environment. > This is a known problem, https://github.com/pybind/pybind11/issues/2139 > but non of the proposed fixes has been merged yet. > > However, the unreleased master version of zxing-cpp hast been changed so > it can be configured to use the a system pybind11. So I tried to include > the br packaged one (2.6.1 as well as 2.8.1). They both built on my > system (no problem with the hardcoded host cmake here) but I needed to > change three things: > 1. Add PYTHON_PYBIND_INSTALL_STAGING = YES > 2. Set ZXING_CPP_CONF_OPTS += > > -Dpybind11_DIR=$(STAGING_DIR)/usr/lib/python$(PYTHON3_VERSION_MAJOR)/site-packages/pybind11/share/cmake/pybind11 > 3. Apply one of the proposed fixes for the pybind11 crosscompile problem > (to tools/pybind11Tools.cmake) > > So this patch here seems to help my (and possibly others) because 1. is > already included. If interested, I could send a patch for 3. > > > > > > Note that there are already two [1][2] submissions of scipy in > > patchwork, so feel free to review those. > > > > Note also that scipy depends on host-pybind, not pybind itself. I'm > > not sure how *that* is supposed to work - although, being header-only, > > > One assumption why scipy includes the host version of pybind is that it > nicely avoids the problems where I need fix 1. and 3. for. > > > it probably doesn't really matter if you link with the host or the > > target version of the library, and the host-installed pybind11-config > > makes sure that you pick up the host headers (which are normally not in > > include path). So conceptually wrong but works in practice. > > For getting conceptually right with my zxing-cpp I would appreciate some > advice: When compiling it with above modificiations and > BUILD_PYTHON_MODULE=ON, cmake happily compiles the zxing pybind wrapper. > But of course wrappers/python/setup.py would need to be run as well. > However mixing $(eval $(python-package)) and $(eval $(cmake-package)) is > not possible. So do I still need to create a second package > python-zxingcpp for the bindings only (which might be a pain because the > wrappers CMakeLists is not written to use an existant installation of > zxing-cpp but rather downloads it again) ? > > > regards, > Andreas > > > > > > > > > In conclusion: I think this patch is going in the right direction > > (modulo BR2_CMAKE_HOST_DEPENDENCY). It's still "conceptually wrong but > > works" because it doesn't set the toolchain file and other cmake > > options, but that happens to work because it's header-only so it doesn't > > actually need any of the cmake options. So it's not entirely the correct > > thing to do, but it does guarantee to fix things. I'm not sure if that > > means we should merge it or not... > > > > The proper fix, I think, would require patching setup.py so we can > > pass in additional cmake options to it. > > > > > > Regards, > > Arnout > > > > [1] https://patchwork.ozlabs.org/project/buildroot/list/?series=168992 > > [2] https://patchwork.ozlabs.org/project/buildroot/list/?series=186146 > > So > > > > >> > >> Regards, > >> Yann E. MORIN. > >> > >>> $(eval $(python-package)) > >>> -- > >>> 2.20.1 > >>> > >> > > _______________________________________________ > > buildroot mailing list > > buildroot@buildroot.org > > https://lists.buildroot.org/mailman/listinfo/buildroot > > > [-- Attachment #1.2: Type: text/html, Size: 14609 bytes --] [-- Attachment #2: Type: text/plain, Size: 150 bytes --] _______________________________________________ buildroot mailing list buildroot@buildroot.org https://lists.buildroot.org/mailman/listinfo/buildroot ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [Buildroot] [PATCH v2 1/2] package/python-pybind: fix upgrade to version 2.6.1 2021-12-08 11:19 ` Andreas Naumann 2021-12-08 12:15 ` Guillaume Bres @ 2022-01-07 21:55 ` Arnout Vandecappelle 1 sibling, 0 replies; 21+ messages in thread From: Arnout Vandecappelle @ 2022-01-07 21:55 UTC (permalink / raw) To: Andreas Naumann, Yann E. MORIN, guillaume.bressaix; +Cc: buildroot On 08/12/2021 12:19, Andreas Naumann wrote: [snip] > By chance I was playing with such a package just yesterday: zxing-cpp. Not the > version buildroot uses (from github glassechidna), but the nu-book one which > claims to be faster. (And is the one where the python bindings in the PyPi > package zxing-cpp originate from). > > Until recently they shipped their own pybind11 version, which throws an error > when included in a cmake project in a crosscompile environment. This is a known > problem, https://github.com/pybind/pybind11/issues/2139 but non of the proposed > fixes has been merged yet. > > However, the unreleased master version of zxing-cpp hast been changed so it can > be configured to use the a system pybind11. So I tried to include the br > packaged one (2.6.1 as well as 2.8.1). They both built on my system (no problem > with the hardcoded host cmake here) but I needed to change three things: > 1. Add PYTHON_PYBIND_INSTALL_STAGING = YES > 2. Set ZXING_CPP_CONF_OPTS += > -Dpybind11_DIR=$(STAGING_DIR)/usr/lib/python$(PYTHON3_VERSION_MAJOR)/site-packages/pybind11/share/cmake/pybind11 > > 3. Apply one of the proposed fixes for the pybind11 crosscompile problem (to > tools/pybind11Tools.cmake) > > So this patch here seems to help my (and possibly others) because 1. is already > included. If interested, I could send a patch for 3. I applied a slightly different approach to master [1]. If you'd like to give that one a go, we'd welcome an updated zxing-cpp package! Regards, Arnout [1] https://git.buildroot.org/buildroot/commit/?id=a19b822239c36cb40a632899bb5c3e848a5fc81e [snip] _______________________________________________ buildroot mailing list buildroot@buildroot.org https://lists.buildroot.org/mailman/listinfo/buildroot ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [Buildroot] [PATCH v2 1/2] package/python-pybind: fix upgrade to version 2.6.1 2021-12-07 8:04 ` Arnout Vandecappelle 2021-12-08 11:19 ` Andreas Naumann @ 2021-12-08 17:18 ` Yann E. MORIN 1 sibling, 0 replies; 21+ messages in thread From: Yann E. MORIN @ 2021-12-08 17:18 UTC (permalink / raw) To: Arnout Vandecappelle; +Cc: buildroot, guillaume.bressaix Arnout, All, On 2021-12-07 09:04 +0100, Arnout Vandecappelle spake thusly: > On 06/12/2021 22:57, Yann E. MORIN wrote: > >Guillaume, All, > > > >On 2021-12-06 20:31 +0100, guillaume.bressaix@gmail.com spake thusly: > >>From: "Guillaume W. Bres" <guillaume.bressaix@gmail.com> > >> > >>The python 'setup.py' script needs header files > >>in $(@D)/pybind11 to work since v2.6.1, and these > >>files are generated by an internal minimalist cmake build. > >> > >>Fixes > >> http://autobuild.buildroot.net/results/b89f1de64b308dffa73675f1f31ccb0b7be5a10d > >> http://autobuild.buildroot.net/results/d0287b7f64f206b0f074908c5780a3632e0cb799 > >> http://autobuild.buildroot.net/results/27efb545a5a719a5581c8f746d3a3555ff4216ce > >> http://autobuild.buildroot.net/results/d2f0a0ad8f6c7178517df109e7d885dac9134c3a > >> http://autobuild.buildroot.net/results/b57e9a3279260dae4a590f9421238fcabb2f7cab > >> http://autobuild.buildroot.net/results/515e6f2fc6b5780260d98d6bb52b541ce4bf1afe > >> http://autobuild.buildroot.net/results/d89c4ecc81222d4f80c951da2232d2e393fa1c69 > >> > >>Signed-off-by: Guillaume W. Bres <guillaume.bressaix@gmail.com> > >I am sorry, but I fail to see how this actually fixes things. [--SNIP--] > >Also, since pybind is supposed to be a headers'-only library, there is > >nothing supposed to go in target, so it should be; > > PYTHON_PYBIND_INSTALL_TARGET = NO > Note that this is only relevant to make sure that pybind11-config doesn't > appear on target. The headers themselves are removed automatically by > Buildroot. It also installs a buncha stuff in /usr/lib/python3.9/site-packages/pybind11/ which are totally useless because that needed only for the packaging of pybind11, not at runtime (as far as I could understand). > >Furthermore, /usr/bin/pybind11-config is a python script which hardcodes > >its shabang to #!/usr/bin/python, and this is incorrect: > > > > 1. /usr/bin/python might be python2 or python3, we can't know > > 2. anyway, it would not use our host python, so it would miss the > > proper modules search paths. > > Yeah, this is something that needs to be fixed - in particular, there may > not be a /usr/bin/python at all. > > However, I think that is kind of orthogonal to this patch. This patch fixes > a build failure of pybind itself. Your proposed change would be to fix the > build of a dependency of pybind. Well, we can also easily fix it with just this: PYTHON_PYBIND_CONFIGURE_CMDS = true PYTHON_PYBIND_BUILD_CMDS = true PYTHON_PYBIND_INSTALL_TARGET_CMDS = true PYTHON_PYBIND_INSTALL_STAGING_CMDS = true HOST_PYTHON_PYBIND_CONFIGURE_CMDS = true HOST_PYTHON_PYBIND_BUILD_CMDS = true HOST_PYTHON_PYBIND_INSTALL_CMDS = true Of course, this won't be very useful... Fixing the build of pybind in a way that does not make it usable by its dependents is useless, imho. If we fix it, we must fix it so that 1. it indeed builds and installs, and 2. that it is usable by its dependents, which means it should install properly. > Note also that scipy depends on host-pybind, not pybind itself. I'm not > sure how *that* is supposed to work - although, being header-only, it > probably doesn't really matter if you link with the host or the target > version of the library, and the host-installed pybind11-config makes sure > that you pick up the host headers (which are normally not in include path). > So conceptually wrong but works in practice. And if we keep it like that without proper explanations why that is, or if we do not fix it so that it behaves properly, it is going to become a maintenance minefield down the road... Already is, anyway... :-] > In conclusion: I think this patch is going in the right direction (modulo > BR2_CMAKE_HOST_DEPENDENCY). It's still "conceptually wrong but works" > because it doesn't set the toolchain file and other cmake options, but that > happens to work because it's header-only so it doesn't actually need any of > the cmake options. So it's not entirely the correct thing to do, but it does > guarantee to fix things. I'm not sure if that means we should merge it or > not... For me, no. But since we can't continue having build failures, and becasue we have no package that depends on pybind, if we eventualyl do not get a solution, then we can still remove it (it's broken!). > The proper fix, I think, would require patching setup.py so we can pass in > additional cmake options to it. Or as we've been discussing in another mail and IRC, by providing a cmake wrapper in $(@D)/bin/cmake (not unlike some packages for which we do a python symlink in $(@D)/bin/pyhon). But now that the cmake issue has been identified, I think that we are narrowing down on to a proper solution (Famous Last Words™). Regards, Yann E. MORIN. -- .-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------' _______________________________________________ buildroot mailing list buildroot@buildroot.org https://lists.buildroot.org/mailman/listinfo/buildroot ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [Buildroot] [PATCH v2 1/2] package/python-pybind: fix upgrade to version 2.6.1 2021-12-06 21:57 ` Yann E. MORIN 2021-12-07 8:04 ` Arnout Vandecappelle @ 2022-01-03 8:24 ` Tomi Valkeinen 2022-01-04 12:38 ` [Buildroot] [PATCH v2 1/3] package/python-pybind: fix upgrade to 2.6.1 guillaume.bressaix 2022-01-07 21:49 ` [Buildroot] [PATCH v2 1/2] package/python-pybind: fix upgrade to version 2.6.1 Arnout Vandecappelle 1 sibling, 2 replies; 21+ messages in thread From: Tomi Valkeinen @ 2022-01-03 8:24 UTC (permalink / raw) To: Yann E. MORIN, guillaume.bressaix; +Cc: buildroot Hi, On 06/12/2021 23:57, Yann E. MORIN wrote: > So I think we really, really need a package (like scipy) that actually > makes use of pybind, so we can see how it works, and to fix it... Jfyi, there are two relatively simple packages in buildroot that can use pybind: kmsxx and rwmem. Both use meson for building. I can test the next version of this series with those packages. Tomi _______________________________________________ buildroot mailing list buildroot@buildroot.org https://lists.buildroot.org/mailman/listinfo/buildroot ^ permalink raw reply [flat|nested] 21+ messages in thread
* [Buildroot] [PATCH v2 1/3] package/python-pybind: fix upgrade to 2.6.1 2022-01-03 8:24 ` Tomi Valkeinen @ 2022-01-04 12:38 ` guillaume.bressaix 2022-01-04 12:39 ` [Buildroot] [PATCH v2 2/3] package/python-pybind: bump to version 2.8.1 guillaume.bressaix ` (2 more replies) 2022-01-07 21:49 ` [Buildroot] [PATCH v2 1/2] package/python-pybind: fix upgrade to version 2.6.1 Arnout Vandecappelle 1 sibling, 3 replies; 21+ messages in thread From: guillaume.bressaix @ 2022-01-04 12:38 UTC (permalink / raw) To: buildroot; +Cc: tomi.valkeinen, yann.morin.1998, Guillaume W. Bres From: "Guillaume W. Bres" <guillaume.bressaix@gmail.com> Fixes http://autobuild.buildroot.net/results/b89f1de64b308dffa73675f1f31ccb0b7be5a10d http://autobuild.buildroot.net/results/d0287b7f64f206b0f074908c5780a3632e0cb799 http://autobuild.buildroot.net/results/27efb545a5a719a5581c8f746d3a3555ff4216ce http://autobuild.buildroot.net/results/d2f0a0ad8f6c7178517df109e7d885dac9134c3a http://autobuild.buildroot.net/results/b57e9a3279260dae4a590f9421238fcabb2f7cab http://autobuild.buildroot.net/results/515e6f2fc6b5780260d98d6bb52b541ce4bf1afe http://autobuild.buildroot.net/results/d89c4ecc81222d4f80c951da2232d2e393fa1c69 Signed-off-by: Guillaume W. Bres <guillaume.bressaix@gmail.com> --- setup.py now calls 'cmake', as is, internally, with DESTDIR=$(@D) to provide some setup.py build requirements. The autobuild failures were actually of different kinds: (1) 'cmake' too old causes an early crash as it is unable to parse their CMakeList.txt (host machine dependent) (2) $(@D)/pybind missing because cmake call was faulty (happened all the time) The provided patch allows us to fully control the hidden cmake context: * which 'cmake' is actually called is important, as most of the time we end up using /usr/bin/cmake. By adding BR2_CMAKE_DEPENDENCY we make sure (1) never happens and (2) is always solved with a complete cmake context control. We could convert python-pybind to a cmake package but it involves so much patching and tweaking (mostly for the python install), that it is now clear that keeping the package as is, is the best option. I also fixed the previous package declaration / handling: * this is a header only lib, so installing to target does not make sense, and installing to staging is mandatory * most of the time, this package will be used at compile time, on host, so having the host-package option is mandatory. This package is then used to customize the host/usr/python capacity, with C++ macro bindings. This patch serie provides an example of use, that does exactly that, and works for both python2 and python3 (refer to following patch) --- ...-py-improve-cmake-context-definition.patch | 48 +++++++++++++++++++ package/python-pybind/python-pybind.mk | 47 ++++++++++++++++++ 2 files changed, 95 insertions(+) create mode 100644 package/python-pybind/0001-setup-py-improve-cmake-context-definition.patch diff --git a/package/python-pybind/0001-setup-py-improve-cmake-context-definition.patch b/package/python-pybind/0001-setup-py-improve-cmake-context-definition.patch new file mode 100644 index 0000000000..0952d44464 --- /dev/null +++ b/package/python-pybind/0001-setup-py-improve-cmake-context-definition.patch @@ -0,0 +1,48 @@ +From aeee170bc90d89973268f286c8bfa12afd8f330c Mon Sep 17 00:00:00 2001 +From: "Guillaume W. Bres" <guillaume.bressaix@gmail.com> +Date: Thu, 9 Dec 2021 20:30:08 +0100 +Subject: [PATCH] setup.py: improve cmake context definition + +Use two env. variables to define which cmake +is internally called and pass a toolchain file +for complex cross-compilation contexts. + +Signed-off-by: Guillaume W. Bres <guillaume.bressaix@gmail.com> +--- + setup.py | 7 +++++-- + 1 file changed, 5 insertions(+), 2 deletions(-) + +diff --git a/setup.py b/setup.py +index 3a03279..2dff721 100644 +--- a/setup.py ++++ b/setup.py +@@ -23,6 +23,8 @@ VERSION_REGEX = re.compile( + # files, and the sys.prefix files (CMake and headers). + + global_sdist = os.environ.get("PYBIND11_GLOBAL_SDIST", False) ++global_cmake = os.environ.get("PYBIND11_CMAKE", "unknown") ++global_cmake_toolchain_file = os.environ.get("PYBIND11_CMAKE_TOOLCHAIN_FILE", "unknown") + + setup_py = "tools/setup_global.py.in" if global_sdist else "tools/setup_main.py.in" + extra_cmd = 'cmdclass["sdist"] = SDist\n' +@@ -101,14 +103,15 @@ def remove_output(*sources): + with remove_output("pybind11/include", "pybind11/share"): + # Generate the files if they are not present. + with TemporaryDirectory() as tmpdir: +- cmd = ["cmake", "-S", ".", "-B", tmpdir] + [ ++ cmd = [global_cmake, "-S", ".", "-B", tmpdir] + [ + "-DCMAKE_INSTALL_PREFIX=pybind11", ++ "-DCMAKE_TOOLCHAIN_FILE={:s}".format(global_cmake_toolchain_file), + "-DBUILD_TESTING=OFF", + "-DPYBIND11_NOPYTHON=ON", + ] + cmake_opts = dict(cwd=DIR, stdout=sys.stdout, stderr=sys.stderr) + subprocess.check_call(cmd, **cmake_opts) +- subprocess.check_call(["cmake", "--install", tmpdir], **cmake_opts) ++ subprocess.check_call([global_cmake, "--install", tmpdir], **cmake_opts) + + txt = get_and_replace(setup_py, version=version, extra_cmd=extra_cmd) + code = compile(txt, setup_py, "exec") +-- +1.8.3.1 + diff --git a/package/python-pybind/python-pybind.mk b/package/python-pybind/python-pybind.mk index a6a1bdb976..442ed3328f 100644 --- a/package/python-pybind/python-pybind.mk +++ b/package/python-pybind/python-pybind.mk @@ -10,4 +10,51 @@ PYTHON_PYBIND_LICENSE = BSD-3-Clause PYTHON_PYBIND_LICENSE_FILES = LICENSE PYTHON_PYBIND_SETUP_TYPE = setuptools +# Only installs header files +PYTHON_PYBIND_INSTALL_TARGET = NO +PYTHON_PYBIND_INSTALL_STAGING = YES + +# their cmakelist.txt file needs a recent cmake +PYTHON_PYBIND_DEPENDENCIES += $(BR2_CMAKE_HOST_DEPENDENCY) +HOST_PYTHON_PYBIND_DEPENDENCIES += $(BR2_CMAKE_HOST_DEPENDENCY) + +# every single setup.py interaction calls 'cmake' (as is) intrinsicly. +# we deliver a patch to customize setup.py to define the cmake context +# using the following variables +PYTHON_PYBIND_MAKE_ENV = $(MAKE_ENV) +PYTHON_PYBIND_SETUPTOOLS_ENV = $(PKG_PYTHON_SETUPTOOLS_ENV) +PYTHON_PYBIND_SETUPTOOLS_ENV += PYBIND11_CMAKE=$(BR2_CMAKE) +PYTHON_PYBIND_SETUPTOOLS_ENV += PYBIND11_CMAKE_TOOLCHAIN_FILE=$(HOST_DIR)/share/buildroot/toolchainfile.cmake + +HOST_PYTHON_PYBIND_MAKE_ENV = $(MAKE_ENV) +HOST_PYTHON_PYBIND_SETUPTOOLS_ENV = $(PKG_PYTHON_SETUPTOOLS_ENV) +HOST_PYTHON_PYBIND_SETUPTOOLS_ENV += PYBIND11_CMAKE=$(BR2_CMAKE) +HOST_PYTHON_PYBIND_SETUPTOOLS_ENV += PYBIND11_CMAKE_TOOLCHAIN_FILE=$(HOST_DIR)/share/buildroot/toolchainfile.cmake + +define PYTHON_PYBIND_BUILD_CMDS + cd $(@D); \ + $(PYTHON_PYBIND_SETUPTOOLS_ENV) $(HOST_DIR)/usr/bin/python setup.py build +endef + +define HOST_PYTHON_PYBIND_BUILD_CMDS + cd $(@D); \ + $(PYTHON_PYBIND_SETUPTOOLS_ENV) $(HOST_DIR)/usr/bin/python setup.py build +endef + +# define installation destinations properly +define PYTHON_PYBIND_INSTALL_STAGING_CMDS + cd $(@D); \ + $(PYTHON_PYBIND_SETUPTOOLS_ENV) \ + PREFIX=$(STAGING_DIR)/usr \ + $(HOST_DIR)/bin/python setup.py install +endef + +define HOST_PYTHON_PYBIND_INSTALL_CMDS + cd $(@D); \ + $(PYTHON_PYBIND_SETUPTOOLS_ENV) \ + PREFIX=$(HOST_DIR)/usr \ + $(HOST_DIR)/bin/python setup.py install +endef + $(eval $(python-package)) +$(eval $(host-python-package)) -- 2.30.2 _______________________________________________ buildroot mailing list buildroot@buildroot.org https://lists.buildroot.org/mailman/listinfo/buildroot ^ permalink raw reply related [flat|nested] 21+ messages in thread
* [Buildroot] [PATCH v2 2/3] package/python-pybind: bump to version 2.8.1 2022-01-04 12:38 ` [Buildroot] [PATCH v2 1/3] package/python-pybind: fix upgrade to 2.6.1 guillaume.bressaix @ 2022-01-04 12:39 ` guillaume.bressaix 2022-01-07 22:10 ` Arnout Vandecappelle 2022-01-04 12:39 ` [Buildroot] [PATCH v2 3/3] support/testing: add test for python-pybind guillaume.bressaix 2022-01-07 21:56 ` [Buildroot] [PATCH v2 1/3] package/python-pybind: fix upgrade to 2.6.1 Arnout Vandecappelle 2 siblings, 1 reply; 21+ messages in thread From: guillaume.bressaix @ 2022-01-04 12:39 UTC (permalink / raw) To: buildroot; +Cc: tomi.valkeinen, yann.morin.1998, Guillaume W. Bres From: "Guillaume W. Bres" <guillaume.bressaix@gmail.com> Signed-off-by: Guillaume W. Bres <guillaume.bressaix@gmail.com> --- package/python-pybind/python-pybind.hash | 2 +- package/python-pybind/python-pybind.mk | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/package/python-pybind/python-pybind.hash b/package/python-pybind/python-pybind.hash index a68ac846e2..e1d86a4494 100644 --- a/package/python-pybind/python-pybind.hash +++ b/package/python-pybind/python-pybind.hash @@ -1,4 +1,4 @@ # Locally calculated -sha256 cdbe326d357f18b83d10322ba202d69f11b2f49e2d87ade0dc2be0c5c34f8e2a python-pybind-2.6.1.tar.gz +sha256 f1bcc07caa568eb312411dde5308b1e250bd0e1bc020fae855bf9f43209940cc python-pybind-2.8.1.tar.gz # License files, locally calculated sha256 83965b843b98f670d3a85bd041ed4b372c8ec50d7b4a5995a83ac697ba675dcb LICENSE diff --git a/package/python-pybind/python-pybind.mk b/package/python-pybind/python-pybind.mk index 442ed3328f..cf6a642922 100644 --- a/package/python-pybind/python-pybind.mk +++ b/package/python-pybind/python-pybind.mk @@ -4,7 +4,7 @@ # ################################################################################ -PYTHON_PYBIND_VERSION = 2.6.1 +PYTHON_PYBIND_VERSION = 2.8.1 PYTHON_PYBIND_SITE = $(call github,pybind,pybind11,v$(PYTHON_PYBIND_VERSION)) PYTHON_PYBIND_LICENSE = BSD-3-Clause PYTHON_PYBIND_LICENSE_FILES = LICENSE -- 2.30.2 _______________________________________________ buildroot mailing list buildroot@buildroot.org https://lists.buildroot.org/mailman/listinfo/buildroot ^ permalink raw reply related [flat|nested] 21+ messages in thread
* Re: [Buildroot] [PATCH v2 2/3] package/python-pybind: bump to version 2.8.1 2022-01-04 12:39 ` [Buildroot] [PATCH v2 2/3] package/python-pybind: bump to version 2.8.1 guillaume.bressaix @ 2022-01-07 22:10 ` Arnout Vandecappelle 0 siblings, 0 replies; 21+ messages in thread From: Arnout Vandecappelle @ 2022-01-07 22:10 UTC (permalink / raw) To: guillaume.bressaix, buildroot; +Cc: tomi.valkeinen, yann.morin.1998 On 04/01/2022 13:39, guillaume.bressaix@gmail.com wrote: > From: "Guillaume W. Bres" <guillaume.bressaix@gmail.com> > > Signed-off-by: Guillaume W. Bres <guillaume.bressaix@gmail.com> I bumped to 2.9.0 instead and applied to master, thanks. Regards, Arnout > --- > package/python-pybind/python-pybind.hash | 2 +- > package/python-pybind/python-pybind.mk | 2 +- > 2 files changed, 2 insertions(+), 2 deletions(-) > > diff --git a/package/python-pybind/python-pybind.hash b/package/python-pybind/python-pybind.hash > index a68ac846e2..e1d86a4494 100644 > --- a/package/python-pybind/python-pybind.hash > +++ b/package/python-pybind/python-pybind.hash > @@ -1,4 +1,4 @@ > # Locally calculated > -sha256 cdbe326d357f18b83d10322ba202d69f11b2f49e2d87ade0dc2be0c5c34f8e2a python-pybind-2.6.1.tar.gz > +sha256 f1bcc07caa568eb312411dde5308b1e250bd0e1bc020fae855bf9f43209940cc python-pybind-2.8.1.tar.gz > # License files, locally calculated > sha256 83965b843b98f670d3a85bd041ed4b372c8ec50d7b4a5995a83ac697ba675dcb LICENSE > diff --git a/package/python-pybind/python-pybind.mk b/package/python-pybind/python-pybind.mk > index 442ed3328f..cf6a642922 100644 > --- a/package/python-pybind/python-pybind.mk > +++ b/package/python-pybind/python-pybind.mk > @@ -4,7 +4,7 @@ > # > ################################################################################ > > -PYTHON_PYBIND_VERSION = 2.6.1 > +PYTHON_PYBIND_VERSION = 2.8.1 > PYTHON_PYBIND_SITE = $(call github,pybind,pybind11,v$(PYTHON_PYBIND_VERSION)) > PYTHON_PYBIND_LICENSE = BSD-3-Clause > PYTHON_PYBIND_LICENSE_FILES = LICENSE > _______________________________________________ buildroot mailing list buildroot@buildroot.org https://lists.buildroot.org/mailman/listinfo/buildroot ^ permalink raw reply [flat|nested] 21+ messages in thread
* [Buildroot] [PATCH v2 3/3] support/testing: add test for python-pybind 2022-01-04 12:38 ` [Buildroot] [PATCH v2 1/3] package/python-pybind: fix upgrade to 2.6.1 guillaume.bressaix 2022-01-04 12:39 ` [Buildroot] [PATCH v2 2/3] package/python-pybind: bump to version 2.8.1 guillaume.bressaix @ 2022-01-04 12:39 ` guillaume.bressaix 2022-01-07 21:57 ` Arnout Vandecappelle 2022-01-07 21:56 ` [Buildroot] [PATCH v2 1/3] package/python-pybind: fix upgrade to 2.6.1 Arnout Vandecappelle 2 siblings, 1 reply; 21+ messages in thread From: guillaume.bressaix @ 2022-01-04 12:39 UTC (permalink / raw) To: buildroot; +Cc: tomi.valkeinen, yann.morin.1998, Guillaume W. Bres From: "Guillaume W. Bres" <guillaume.bressaix@gmail.com> python-pybind is a header only library, most of the time used as a host-package, to customize the host/usr/python interpreter capacity (at build time) with C++ based features. Signed-off-by: Guillaume W. Bres <guillaume.bressaix@gmail.com> --- This test case enhances the /host/usr/python interpreter with a C++ macro. The test delivers a C++ macro contained in its br-external/example.cpp. We compile it against the host context. We install it to $(HOST_DIR)/usr/lib/python3.9 as it is natively contained in the host-python system path (import is easy). The macro is an integer adder, written in C++ that we can now call in host-python. The test infrastructure should actually be declared "host-generic-package" instead of "generic-package", to be consistent with Buildroot infrastructure. But I am not able to get this to actually do anything, if I declare this package as a host-generic and "Config.in.host" instead of "Config.in". To me, this is not too important * TestPythonPybind allows testing for both Py2 and Py3 like it is done in many other packages * TestPythonPybind does not inherit from TestPythonPackageBase as usually done (1) because I need the br2-external feature to ship the example.cpp containing the custom macro (2) because TestPythonPackageBase always involves the emulator, which we do not want in our context * Although TestPythonPybind does have a TestPythonBase.config base at the same time, as it is most convenient to have a valid python context * TestPythonPybind uses subprocess, as similarly done in 'core' on 'infra' test cases, because we do not involve the emulator * TestPythonPybind tests (1+2)=3 written in C++, used in Python 2/3 * TestPythonPybind tests a custom module attribute, written in C++, used in Python 2/3 --- DEVELOPERS | 3 ++ .../br2-external/python-pybind/Config.in | 1 + .../br2-external/python-pybind/external.desc | 1 + .../br2-external/python-pybind/external.mk | 1 + .../package/python-pybind-example/Config.in | 5 ++ .../package/python-pybind-example/example.cpp | 16 +++++++ .../python-pybind-example.mk | 47 ++++++++++++++++++ .../tests/package/sample_python_pybind.py | 4 ++ .../tests/package/test_python_pybind.py | 48 +++++++++++++++++++ 9 files changed, 126 insertions(+) create mode 100644 support/testing/tests/package/br2-external/python-pybind/Config.in create mode 100644 support/testing/tests/package/br2-external/python-pybind/external.desc create mode 100644 support/testing/tests/package/br2-external/python-pybind/external.mk create mode 100644 support/testing/tests/package/br2-external/python-pybind/package/python-pybind-example/Config.in create mode 100644 support/testing/tests/package/br2-external/python-pybind/package/python-pybind-example/example.cpp create mode 100644 support/testing/tests/package/br2-external/python-pybind/package/python-pybind-example/python-pybind-example.mk create mode 100644 support/testing/tests/package/sample_python_pybind.py create mode 100644 support/testing/tests/package/test_python_pybind.py diff --git a/DEVELOPERS b/DEVELOPERS index 21e4f2a84c..130ac3db73 100644 --- a/DEVELOPERS +++ b/DEVELOPERS @@ -1120,6 +1120,9 @@ F: package/liquid-dsp/ F: package/pixiewps/ F: package/python-pybind/ F: package/reaver/ +F: support/testing/tests/package/br2-external/python-pybind +F: support/testing/tests/package/sample_python_pybind.py +F: support/testing/tests/package/test_python_pybind.py N: Guo Ren <ren_guo@c-sky.com> F: arch/Config.in.csky diff --git a/support/testing/tests/package/br2-external/python-pybind/Config.in b/support/testing/tests/package/br2-external/python-pybind/Config.in new file mode 100644 index 0000000000..70c77157b3 --- /dev/null +++ b/support/testing/tests/package/br2-external/python-pybind/Config.in @@ -0,0 +1 @@ +source "$BR2_EXTERNAL_PYTHON_PYBIND_PATH/package/python-pybind-example/Config.in" diff --git a/support/testing/tests/package/br2-external/python-pybind/external.desc b/support/testing/tests/package/br2-external/python-pybind/external.desc new file mode 100644 index 0000000000..eef5e0f5a0 --- /dev/null +++ b/support/testing/tests/package/br2-external/python-pybind/external.desc @@ -0,0 +1 @@ +name: PYTHON_PYBIND diff --git a/support/testing/tests/package/br2-external/python-pybind/external.mk b/support/testing/tests/package/br2-external/python-pybind/external.mk new file mode 100644 index 0000000000..3501f3135e --- /dev/null +++ b/support/testing/tests/package/br2-external/python-pybind/external.mk @@ -0,0 +1 @@ +include $(sort $(wildcard $(BR2_EXTERNAL_PYTHON_PYBIND_PATH)/package/*/*.mk)) diff --git a/support/testing/tests/package/br2-external/python-pybind/package/python-pybind-example/Config.in b/support/testing/tests/package/br2-external/python-pybind/package/python-pybind-example/Config.in new file mode 100644 index 0000000000..773d021a53 --- /dev/null +++ b/support/testing/tests/package/br2-external/python-pybind/package/python-pybind-example/Config.in @@ -0,0 +1,5 @@ +config BR2_PACKAGE_PYTHON_PYBIND_EXAMPLE + bool "python-pybind-example" + depends on BR2_PACKAGE_PYTHON_PYBIND + help + This test creates a cpp macro later used on target in python diff --git a/support/testing/tests/package/br2-external/python-pybind/package/python-pybind-example/example.cpp b/support/testing/tests/package/br2-external/python-pybind/package/python-pybind-example/example.cpp new file mode 100644 index 0000000000..f2eea8e48d --- /dev/null +++ b/support/testing/tests/package/br2-external/python-pybind/package/python-pybind-example/example.cpp @@ -0,0 +1,16 @@ +#include <pybind11/pybind11.h> +namespace py = pybind11; + +int add (int i, int j) { + return i + j; +} + +PYBIND11_MODULE (example, m) { + // optional module description + m.doc() = "pybind11 example plugin"; + // test a module method + m.def("add", &add, "example::add adds two integer numbers"); + // test a module attribute + py::object hello = py::cast("Hello World"); + m.attr("says") = hello; +} diff --git a/support/testing/tests/package/br2-external/python-pybind/package/python-pybind-example/python-pybind-example.mk b/support/testing/tests/package/br2-external/python-pybind/package/python-pybind-example/python-pybind-example.mk new file mode 100644 index 0000000000..85dd09d93f --- /dev/null +++ b/support/testing/tests/package/br2-external/python-pybind/package/python-pybind-example/python-pybind-example.mk @@ -0,0 +1,47 @@ +################################################################################ +# +# python-pybind-example +# +################################################################################ + +# this builds a C++ macro "add(a,b)" +# that we expose to host-python with a custom install +# and that the python test script will later use +PYTHON_PYBIND_EXAMPLE_DEPENDENCIES = host-python-pybind + +PYTHON_PYBIND_EXAMPLE_CXX_FLAGS = $(HOST_CXXFLAGS) +PYTHON_PYBIND_EXAMPLE_CXX_FLAGS += -Wall -shared +PYTHON_PYBIND_EXAMPLE_CXX_FLAGS += -std=c++11 -fPIC + +ifeq ($(BR2_PACKAGE_PYTHON3),y) +PYTHON_PYBIND_PYVER = python$(PYTHON3_VERSION_MAJOR) +PYTHON_PYBIND_PYTHON = $(HOST_DIR)/usr/bin/python3 +else +PYTHON_PYBIND_PYVER = python$(PYTHON_VERSION_MAJOR) +PYTHON_PYBIND_PYTHON = $(HOST_DIR)/usr/bin/python2 +endif + +# retrieve pybind11 include paths +PYTHON_PYBIND_EXAMPLE_CXX_FLAGS += \ + $(shell $(PYTHON_PYBIND_PYTHON) -m pybind11 --includes) + +# .so to be installed must have exact suffix +# otherwise import() in python will not work +HOST_LIB_BINARY_SUFFIX = \ + $(shell $(PYTHON_PYBIND_PYTHON)-config --extension-suffix) + +define PYTHON_PYBIND_EXAMPLE_BUILD_CMDS + $(INSTALL) $(PYTHON_PYBIND_EXAMPLE_PKGDIR)/example.cpp $(@D) + cd $(@D); \ + $(HOSTCXX) $(PYTHON_PYBIND_EXAMPLE_CXX_FLAGS) \ + example.cpp -o example$(HOST_LIB_BINARY_SUFFIX) +endef + +# installs into host/python sys.path +# so the new macro is naturally known +define PYTHON_PYBIND_EXAMPLE_INSTALL_TARGET_CMDS + $(INSTALL) -D -m 755 $(@D)/example$(HOST_LIB_BINARY_SUFFIX) \ + $(HOST_DIR)/usr/lib/$(PYTHON_PYBIND_PYVER)/example$(HOST_LIB_BINARY_SUFFIX) +endef + +$(eval $(generic-package)) diff --git a/support/testing/tests/package/sample_python_pybind.py b/support/testing/tests/package/sample_python_pybind.py new file mode 100644 index 0000000000..605c0bab15 --- /dev/null +++ b/support/testing/tests/package/sample_python_pybind.py @@ -0,0 +1,4 @@ +#!/usr/bin/env python +import example +print(example.add(1, 2)) +print(example.says) diff --git a/support/testing/tests/package/test_python_pybind.py b/support/testing/tests/package/test_python_pybind.py new file mode 100644 index 0000000000..7a68d6512a --- /dev/null +++ b/support/testing/tests/package/test_python_pybind.py @@ -0,0 +1,48 @@ +import os +import infra +import subprocess +from tests.package.test_python import TestPythonPackageBase + +class TestPythonPybind (infra.basetest.BRTest): + config = TestPythonPackageBase.config # python minimal requirements + sample_script = "tests/package/sample_python_pybind.py" + # ship examples macro & installs it to host + br2_external = [infra.filepath("tests/package/br2-external/python-pybind")] + +class TestPythonPy2Pybind (TestPythonPybind): + __test__ = True + config = TestPythonPybind.config + \ + """ + BR2_PACKAGE_PYTHON=y + BR2_PACKAGE_PYTHON_PYBIND=y + BR2_PACKAGE_PYTHON_PYBIND_EXAMPLE=y + """ + + def test_run(self): + host_interpreter = os.path.join(self.builddir,"host","usr","bin","python") + # test new c++->py macro exposed to host interpreter + cmd = [host_interpreter, infra.filepath(self.sample_script)] + output = subprocess.check_output(cmd) + result = int(output.splitlines()[0]) + attribute = output.splitlines()[1].decode("utf-8") + self.assertEqual(result,3) # 2+1 using '+' macro from cpp + self.assertEqual(attribute,"Hello World") # macro special attribute + +class TestPythonPy3Pybind (TestPythonPybind): + __test__ = True + config = TestPythonPybind.config + \ + """ + BR2_PACKAGE_PYTHON3=y + BR2_PACKAGE_PYTHON_PYBIND=y + BR2_PACKAGE_PYTHON_PYBIND_EXAMPLE=y + """ + + def test_run(self): + host_interpreter = os.path.join(self.builddir,"host","usr","bin","python") + # test new c++->py macro exposed to host interpreter + cmd = [host_interpreter, infra.filepath(self.sample_script)] + output = subprocess.check_output(cmd) + result = int(output.splitlines()[0]) + attribute = output.splitlines()[1].decode("utf-8") + self.assertEqual(result,3) # 2+1 using '+' macro from cpp + self.assertEqual(attribute,"Hello World") # macro special attribute -- 2.30.2 _______________________________________________ buildroot mailing list buildroot@buildroot.org https://lists.buildroot.org/mailman/listinfo/buildroot ^ permalink raw reply related [flat|nested] 21+ messages in thread
* Re: [Buildroot] [PATCH v2 3/3] support/testing: add test for python-pybind 2022-01-04 12:39 ` [Buildroot] [PATCH v2 3/3] support/testing: add test for python-pybind guillaume.bressaix @ 2022-01-07 21:57 ` Arnout Vandecappelle 2022-01-08 10:26 ` Guillaume Bres 0 siblings, 1 reply; 21+ messages in thread From: Arnout Vandecappelle @ 2022-01-07 21:57 UTC (permalink / raw) To: guillaume.bressaix, buildroot; +Cc: tomi.valkeinen, yann.morin.1998 On 04/01/2022 13:39, guillaume.bressaix@gmail.com wrote: > From: "Guillaume W. Bres" <guillaume.bressaix@gmail.com> > > python-pybind is a header only library, most of the time used as a host-package, > to customize the host/usr/python interpreter capacity (at build time) with C++ based features. > > Signed-off-by: Guillaume W. Bres <guillaume.bressaix@gmail.com> I applied to master with many changes: - Retain python3 only. - python-pybind is a target package, not host. - Select python-pybind instead of depend. - Simplify python-pybind-example package. - Check in python-pybind-example build if pybind11.get_include() produces output. - Don't use python3 -m pybind11 --includes: it includes the main python includes, which are for the host, not for the target. - Use TestPythonPackageBase instead of open-coding something imported with host python. However, I hadn't noticed that the original author of this test was Adam. In the future, please make sure that you don't claim authorship on a patch that is actually from someone else! Regards, Arnout > --- > This test case enhances the /host/usr/python interpreter with a C++ macro. > > The test delivers a C++ macro contained in its br-external/example.cpp. > We compile it against the host context. > We install it to $(HOST_DIR)/usr/lib/python3.9 as it is natively contained > in the host-python system path (import is easy). > The macro is an integer adder, written in C++ that we can now call in host-python. > > The test infrastructure should actually be declared "host-generic-package" > instead of "generic-package", to be consistent with Buildroot infrastructure. > But I am not able to get this to actually do anything, if I declare this package > as a host-generic and "Config.in.host" instead of "Config.in". To me, this is > not too important > > * TestPythonPybind allows testing for both Py2 and Py3 like it is done > in many other packages > > * TestPythonPybind does not inherit from TestPythonPackageBase as usually done > (1) because I need the br2-external feature to ship the example.cpp containing the custom macro > (2) because TestPythonPackageBase always involves the emulator, which we do not want in our context > > * Although TestPythonPybind does have a TestPythonBase.config base at the same time, > as it is most convenient to have a valid python context > > * TestPythonPybind uses subprocess, as similarly done in 'core' on 'infra' test cases, > because we do not involve the emulator > > * TestPythonPybind tests (1+2)=3 written in C++, used in Python 2/3 > * TestPythonPybind tests a custom module attribute, written in C++, used in Python 2/3 > > --- > DEVELOPERS | 3 ++ > .../br2-external/python-pybind/Config.in | 1 + > .../br2-external/python-pybind/external.desc | 1 + > .../br2-external/python-pybind/external.mk | 1 + > .../package/python-pybind-example/Config.in | 5 ++ > .../package/python-pybind-example/example.cpp | 16 +++++++ > .../python-pybind-example.mk | 47 ++++++++++++++++++ > .../tests/package/sample_python_pybind.py | 4 ++ > .../tests/package/test_python_pybind.py | 48 +++++++++++++++++++ > 9 files changed, 126 insertions(+) > create mode 100644 support/testing/tests/package/br2-external/python-pybind/Config.in > create mode 100644 support/testing/tests/package/br2-external/python-pybind/external.desc > create mode 100644 support/testing/tests/package/br2-external/python-pybind/external.mk > create mode 100644 support/testing/tests/package/br2-external/python-pybind/package/python-pybind-example/Config.in > create mode 100644 support/testing/tests/package/br2-external/python-pybind/package/python-pybind-example/example.cpp > create mode 100644 support/testing/tests/package/br2-external/python-pybind/package/python-pybind-example/python-pybind-example.mk > create mode 100644 support/testing/tests/package/sample_python_pybind.py > create mode 100644 support/testing/tests/package/test_python_pybind.py > > diff --git a/DEVELOPERS b/DEVELOPERS > index 21e4f2a84c..130ac3db73 100644 > --- a/DEVELOPERS > +++ b/DEVELOPERS > @@ -1120,6 +1120,9 @@ F: package/liquid-dsp/ > F: package/pixiewps/ > F: package/python-pybind/ > F: package/reaver/ > +F: support/testing/tests/package/br2-external/python-pybind > +F: support/testing/tests/package/sample_python_pybind.py > +F: support/testing/tests/package/test_python_pybind.py > > N: Guo Ren <ren_guo@c-sky.com> > F: arch/Config.in.csky > diff --git a/support/testing/tests/package/br2-external/python-pybind/Config.in b/support/testing/tests/package/br2-external/python-pybind/Config.in > new file mode 100644 > index 0000000000..70c77157b3 > --- /dev/null > +++ b/support/testing/tests/package/br2-external/python-pybind/Config.in > @@ -0,0 +1 @@ > +source "$BR2_EXTERNAL_PYTHON_PYBIND_PATH/package/python-pybind-example/Config.in" > diff --git a/support/testing/tests/package/br2-external/python-pybind/external.desc b/support/testing/tests/package/br2-external/python-pybind/external.desc > new file mode 100644 > index 0000000000..eef5e0f5a0 > --- /dev/null > +++ b/support/testing/tests/package/br2-external/python-pybind/external.desc > @@ -0,0 +1 @@ > +name: PYTHON_PYBIND > diff --git a/support/testing/tests/package/br2-external/python-pybind/external.mk b/support/testing/tests/package/br2-external/python-pybind/external.mk > new file mode 100644 > index 0000000000..3501f3135e > --- /dev/null > +++ b/support/testing/tests/package/br2-external/python-pybind/external.mk > @@ -0,0 +1 @@ > +include $(sort $(wildcard $(BR2_EXTERNAL_PYTHON_PYBIND_PATH)/package/*/*.mk)) > diff --git a/support/testing/tests/package/br2-external/python-pybind/package/python-pybind-example/Config.in b/support/testing/tests/package/br2-external/python-pybind/package/python-pybind-example/Config.in > new file mode 100644 > index 0000000000..773d021a53 > --- /dev/null > +++ b/support/testing/tests/package/br2-external/python-pybind/package/python-pybind-example/Config.in > @@ -0,0 +1,5 @@ > +config BR2_PACKAGE_PYTHON_PYBIND_EXAMPLE > + bool "python-pybind-example" > + depends on BR2_PACKAGE_PYTHON_PYBIND > + help > + This test creates a cpp macro later used on target in python > diff --git a/support/testing/tests/package/br2-external/python-pybind/package/python-pybind-example/example.cpp b/support/testing/tests/package/br2-external/python-pybind/package/python-pybind-example/example.cpp > new file mode 100644 > index 0000000000..f2eea8e48d > --- /dev/null > +++ b/support/testing/tests/package/br2-external/python-pybind/package/python-pybind-example/example.cpp > @@ -0,0 +1,16 @@ > +#include <pybind11/pybind11.h> > +namespace py = pybind11; > + > +int add (int i, int j) { > + return i + j; > +} > + > +PYBIND11_MODULE (example, m) { > + // optional module description > + m.doc() = "pybind11 example plugin"; > + // test a module method > + m.def("add", &add, "example::add adds two integer numbers"); > + // test a module attribute > + py::object hello = py::cast("Hello World"); > + m.attr("says") = hello; > +} > diff --git a/support/testing/tests/package/br2-external/python-pybind/package/python-pybind-example/python-pybind-example.mk b/support/testing/tests/package/br2-external/python-pybind/package/python-pybind-example/python-pybind-example.mk > new file mode 100644 > index 0000000000..85dd09d93f > --- /dev/null > +++ b/support/testing/tests/package/br2-external/python-pybind/package/python-pybind-example/python-pybind-example.mk > @@ -0,0 +1,47 @@ > +################################################################################ > +# > +# python-pybind-example > +# > +################################################################################ > + > +# this builds a C++ macro "add(a,b)" > +# that we expose to host-python with a custom install > +# and that the python test script will later use > +PYTHON_PYBIND_EXAMPLE_DEPENDENCIES = host-python-pybind > + > +PYTHON_PYBIND_EXAMPLE_CXX_FLAGS = $(HOST_CXXFLAGS) > +PYTHON_PYBIND_EXAMPLE_CXX_FLAGS += -Wall -shared > +PYTHON_PYBIND_EXAMPLE_CXX_FLAGS += -std=c++11 -fPIC > + > +ifeq ($(BR2_PACKAGE_PYTHON3),y) > +PYTHON_PYBIND_PYVER = python$(PYTHON3_VERSION_MAJOR) > +PYTHON_PYBIND_PYTHON = $(HOST_DIR)/usr/bin/python3 > +else > +PYTHON_PYBIND_PYVER = python$(PYTHON_VERSION_MAJOR) > +PYTHON_PYBIND_PYTHON = $(HOST_DIR)/usr/bin/python2 > +endif > + > +# retrieve pybind11 include paths > +PYTHON_PYBIND_EXAMPLE_CXX_FLAGS += \ > + $(shell $(PYTHON_PYBIND_PYTHON) -m pybind11 --includes) > + > +# .so to be installed must have exact suffix > +# otherwise import() in python will not work > +HOST_LIB_BINARY_SUFFIX = \ > + $(shell $(PYTHON_PYBIND_PYTHON)-config --extension-suffix) > + > +define PYTHON_PYBIND_EXAMPLE_BUILD_CMDS > + $(INSTALL) $(PYTHON_PYBIND_EXAMPLE_PKGDIR)/example.cpp $(@D) > + cd $(@D); \ > + $(HOSTCXX) $(PYTHON_PYBIND_EXAMPLE_CXX_FLAGS) \ > + example.cpp -o example$(HOST_LIB_BINARY_SUFFIX) > +endef > + > +# installs into host/python sys.path > +# so the new macro is naturally known > +define PYTHON_PYBIND_EXAMPLE_INSTALL_TARGET_CMDS > + $(INSTALL) -D -m 755 $(@D)/example$(HOST_LIB_BINARY_SUFFIX) \ > + $(HOST_DIR)/usr/lib/$(PYTHON_PYBIND_PYVER)/example$(HOST_LIB_BINARY_SUFFIX) > +endef > + > +$(eval $(generic-package)) > diff --git a/support/testing/tests/package/sample_python_pybind.py b/support/testing/tests/package/sample_python_pybind.py > new file mode 100644 > index 0000000000..605c0bab15 > --- /dev/null > +++ b/support/testing/tests/package/sample_python_pybind.py > @@ -0,0 +1,4 @@ > +#!/usr/bin/env python > +import example > +print(example.add(1, 2)) > +print(example.says) > diff --git a/support/testing/tests/package/test_python_pybind.py b/support/testing/tests/package/test_python_pybind.py > new file mode 100644 > index 0000000000..7a68d6512a > --- /dev/null > +++ b/support/testing/tests/package/test_python_pybind.py > @@ -0,0 +1,48 @@ > +import os > +import infra > +import subprocess > +from tests.package.test_python import TestPythonPackageBase > + > +class TestPythonPybind (infra.basetest.BRTest): > + config = TestPythonPackageBase.config # python minimal requirements > + sample_script = "tests/package/sample_python_pybind.py" > + # ship examples macro & installs it to host > + br2_external = [infra.filepath("tests/package/br2-external/python-pybind")] > + > +class TestPythonPy2Pybind (TestPythonPybind): > + __test__ = True > + config = TestPythonPybind.config + \ > + """ > + BR2_PACKAGE_PYTHON=y > + BR2_PACKAGE_PYTHON_PYBIND=y > + BR2_PACKAGE_PYTHON_PYBIND_EXAMPLE=y > + """ > + > + def test_run(self): > + host_interpreter = os.path.join(self.builddir,"host","usr","bin","python") > + # test new c++->py macro exposed to host interpreter > + cmd = [host_interpreter, infra.filepath(self.sample_script)] > + output = subprocess.check_output(cmd) > + result = int(output.splitlines()[0]) > + attribute = output.splitlines()[1].decode("utf-8") > + self.assertEqual(result,3) # 2+1 using '+' macro from cpp > + self.assertEqual(attribute,"Hello World") # macro special attribute > + > +class TestPythonPy3Pybind (TestPythonPybind): > + __test__ = True > + config = TestPythonPybind.config + \ > + """ > + BR2_PACKAGE_PYTHON3=y > + BR2_PACKAGE_PYTHON_PYBIND=y > + BR2_PACKAGE_PYTHON_PYBIND_EXAMPLE=y > + """ > + > + def test_run(self): > + host_interpreter = os.path.join(self.builddir,"host","usr","bin","python") > + # test new c++->py macro exposed to host interpreter > + cmd = [host_interpreter, infra.filepath(self.sample_script)] > + output = subprocess.check_output(cmd) > + result = int(output.splitlines()[0]) > + attribute = output.splitlines()[1].decode("utf-8") > + self.assertEqual(result,3) # 2+1 using '+' macro from cpp > + self.assertEqual(attribute,"Hello World") # macro special attribute > _______________________________________________ buildroot mailing list buildroot@buildroot.org https://lists.buildroot.org/mailman/listinfo/buildroot ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [Buildroot] [PATCH v2 3/3] support/testing: add test for python-pybind 2022-01-07 21:57 ` Arnout Vandecappelle @ 2022-01-08 10:26 ` Guillaume Bres 0 siblings, 0 replies; 21+ messages in thread From: Guillaume Bres @ 2022-01-08 10:26 UTC (permalink / raw) To: Arnout Vandecappelle; +Cc: buildroot, tomi.valkeinen, Yann E. MORIN [-- Attachment #1.1: Type: text/plain, Size: 14422 bytes --] Hello Arnout, >In the future, please make sure that you don't claim authorship on a patch that is actually from someone else! well I don't understand how the first submitted test/example relates to a real use case or demonstrates something viable. For me pybind is a compile time feature to enhance the compiler capacity. So I was really confused as to what was submitted 2 years ago could actually demonstrate anything. Hence the different and kind of tweaked approach that I proposed. I think we will refer to these notes as the holy bible as of now https://git.busybox.net/buildroot/commit/?id=a19b822239c36cb40a632899bb5c3e848a5fc81e > Generated with scanpypi and converted to host. that seems a useful tool to use > I bumped [python-pybind] to 2.9.0 instead and applied to master, thanks. thanks I did not know it was out there Guillaume W. Bres Software engineer <guillaume.bressaix@gmail.com> Le ven. 7 janv. 2022 à 22:57, Arnout Vandecappelle <arnout@mind.be> a écrit : > > > On 04/01/2022 13:39, guillaume.bressaix@gmail.com wrote: > > From: "Guillaume W. Bres" <guillaume.bressaix@gmail.com> > > > > python-pybind is a header only library, most of the time used as a > host-package, > > to customize the host/usr/python interpreter capacity (at build time) > with C++ based features. > > > > Signed-off-by: Guillaume W. Bres <guillaume.bressaix@gmail.com> > > I applied to master with many changes: > > - Retain python3 only. > - python-pybind is a target package, not host. > - Select python-pybind instead of depend. > - Simplify python-pybind-example package. > - Check in python-pybind-example build if pybind11.get_include() > produces output. > - Don't use python3 -m pybind11 --includes: it includes the main > python > includes, which are for the host, not for the target. > - Use TestPythonPackageBase instead of open-coding something imported > with host python. > > However, I hadn't noticed that the original author of this test was > Adam. In > the future, please make sure that you don't claim authorship on a patch > that is > actually from someone else! > > Regards, > Arnout > > > --- > > This test case enhances the /host/usr/python interpreter with a C++ > macro. > > > > The test delivers a C++ macro contained in its br-external/example.cpp. > > We compile it against the host context. > > We install it to $(HOST_DIR)/usr/lib/python3.9 as it is natively > contained > > in the host-python system path (import is easy). > > The macro is an integer adder, written in C++ that we can now call in > host-python. > > > > The test infrastructure should actually be declared > "host-generic-package" > > instead of "generic-package", to be consistent with Buildroot > infrastructure. > > But I am not able to get this to actually do anything, if I declare this > package > > as a host-generic and "Config.in.host" instead of "Config.in". To me, > this is > > not too important > > > > * TestPythonPybind allows testing for both Py2 and Py3 like it is done > > in many other packages > > > > * TestPythonPybind does not inherit from TestPythonPackageBase as > usually done > > (1) because I need the br2-external feature to ship the example.cpp > containing the custom macro > > (2) because TestPythonPackageBase always involves the emulator, which > we do not want in our context > > > > * Although TestPythonPybind does have a TestPythonBase.config base at > the same time, > > as it is most convenient to have a valid python context > > > > * TestPythonPybind uses subprocess, as similarly done in 'core' on > 'infra' test cases, > > because we do not involve the emulator > > > > * TestPythonPybind tests (1+2)=3 written in C++, used in Python 2/3 > > * TestPythonPybind tests a custom module attribute, written in C++, used > in Python 2/3 > > > > --- > > DEVELOPERS | 3 ++ > > .../br2-external/python-pybind/Config.in | 1 + > > .../br2-external/python-pybind/external.desc | 1 + > > .../br2-external/python-pybind/external.mk | 1 + > > .../package/python-pybind-example/Config.in | 5 ++ > > .../package/python-pybind-example/example.cpp | 16 +++++++ > > .../python-pybind-example.mk | 47 ++++++++++++++++++ > > .../tests/package/sample_python_pybind.py | 4 ++ > > .../tests/package/test_python_pybind.py | 48 +++++++++++++++++++ > > 9 files changed, 126 insertions(+) > > create mode 100644 > support/testing/tests/package/br2-external/python-pybind/Config.in > > create mode 100644 > support/testing/tests/package/br2-external/python-pybind/external.desc > > create mode 100644 > support/testing/tests/package/br2-external/python-pybind/external.mk > > create mode 100644 > support/testing/tests/package/br2-external/python-pybind/package/python-pybind-example/Config.in > > create mode 100644 > support/testing/tests/package/br2-external/python-pybind/package/python-pybind-example/example.cpp > > create mode 100644 > support/testing/tests/package/br2-external/python-pybind/package/python-pybind-example/ > python-pybind-example.mk > > create mode 100644 > support/testing/tests/package/sample_python_pybind.py > > create mode 100644 support/testing/tests/package/test_python_pybind.py > > > > diff --git a/DEVELOPERS b/DEVELOPERS > > index 21e4f2a84c..130ac3db73 100644 > > --- a/DEVELOPERS > > +++ b/DEVELOPERS > > @@ -1120,6 +1120,9 @@ F: package/liquid-dsp/ > > F: package/pixiewps/ > > F: package/python-pybind/ > > F: package/reaver/ > > +F: support/testing/tests/package/br2-external/python-pybind > > +F: support/testing/tests/package/sample_python_pybind.py > > +F: support/testing/tests/package/test_python_pybind.py > > > > N: Guo Ren <ren_guo@c-sky.com> > > F: arch/Config.in.csky > > diff --git > a/support/testing/tests/package/br2-external/python-pybind/Config.in > b/support/testing/tests/package/br2-external/python-pybind/Config.in > > new file mode 100644 > > index 0000000000..70c77157b3 > > --- /dev/null > > +++ b/support/testing/tests/package/br2-external/python-pybind/Config.in > > @@ -0,0 +1 @@ > > +source > "$BR2_EXTERNAL_PYTHON_PYBIND_PATH/package/python-pybind-example/Config.in" > > diff --git > a/support/testing/tests/package/br2-external/python-pybind/external.desc > b/support/testing/tests/package/br2-external/python-pybind/external.desc > > new file mode 100644 > > index 0000000000..eef5e0f5a0 > > --- /dev/null > > +++ > b/support/testing/tests/package/br2-external/python-pybind/external.desc > > @@ -0,0 +1 @@ > > +name: PYTHON_PYBIND > > diff --git a/support/testing/tests/package/br2-external/python-pybind/ > external.mk b/support/testing/tests/package/br2-external/python-pybind/ > external.mk > > new file mode 100644 > > index 0000000000..3501f3135e > > --- /dev/null > > +++ b/support/testing/tests/package/br2-external/python-pybind/ > external.mk > > @@ -0,0 +1 @@ > > +include $(sort $(wildcard > $(BR2_EXTERNAL_PYTHON_PYBIND_PATH)/package/*/*.mk)) > > diff --git > a/support/testing/tests/package/br2-external/python-pybind/package/python-pybind-example/Config.in > b/support/testing/tests/package/br2-external/python-pybind/package/python-pybind-example/Config.in > > new file mode 100644 > > index 0000000000..773d021a53 > > --- /dev/null > > +++ > b/support/testing/tests/package/br2-external/python-pybind/package/python-pybind-example/Config.in > > @@ -0,0 +1,5 @@ > > +config BR2_PACKAGE_PYTHON_PYBIND_EXAMPLE > > + bool "python-pybind-example" > > + depends on BR2_PACKAGE_PYTHON_PYBIND > > + help > > + This test creates a cpp macro later used on target in python > > diff --git > a/support/testing/tests/package/br2-external/python-pybind/package/python-pybind-example/example.cpp > b/support/testing/tests/package/br2-external/python-pybind/package/python-pybind-example/example.cpp > > new file mode 100644 > > index 0000000000..f2eea8e48d > > --- /dev/null > > +++ > b/support/testing/tests/package/br2-external/python-pybind/package/python-pybind-example/example.cpp > > @@ -0,0 +1,16 @@ > > +#include <pybind11/pybind11.h> > > +namespace py = pybind11; > > + > > +int add (int i, int j) { > > + return i + j; > > +} > > + > > +PYBIND11_MODULE (example, m) { > > + // optional module description > > + m.doc() = "pybind11 example plugin"; > > + // test a module method > > + m.def("add", &add, "example::add adds two integer numbers"); > > + // test a module attribute > > + py::object hello = py::cast("Hello World"); > > + m.attr("says") = hello; > > +} > > diff --git > a/support/testing/tests/package/br2-external/python-pybind/package/python-pybind-example/ > python-pybind-example.mk > b/support/testing/tests/package/br2-external/python-pybind/package/python-pybind-example/ > python-pybind-example.mk > > new file mode 100644 > > index 0000000000..85dd09d93f > > --- /dev/null > > +++ > b/support/testing/tests/package/br2-external/python-pybind/package/python-pybind-example/ > python-pybind-example.mk > > @@ -0,0 +1,47 @@ > > > +################################################################################ > > +# > > +# python-pybind-example > > +# > > > +################################################################################ > > + > > +# this builds a C++ macro "add(a,b)" > > +# that we expose to host-python with a custom install > > +# and that the python test script will later use > > +PYTHON_PYBIND_EXAMPLE_DEPENDENCIES = host-python-pybind > > + > > +PYTHON_PYBIND_EXAMPLE_CXX_FLAGS = $(HOST_CXXFLAGS) > > +PYTHON_PYBIND_EXAMPLE_CXX_FLAGS += -Wall -shared > > +PYTHON_PYBIND_EXAMPLE_CXX_FLAGS += -std=c++11 -fPIC > > + > > +ifeq ($(BR2_PACKAGE_PYTHON3),y) > > +PYTHON_PYBIND_PYVER = python$(PYTHON3_VERSION_MAJOR) > > +PYTHON_PYBIND_PYTHON = $(HOST_DIR)/usr/bin/python3 > > +else > > +PYTHON_PYBIND_PYVER = python$(PYTHON_VERSION_MAJOR) > > +PYTHON_PYBIND_PYTHON = $(HOST_DIR)/usr/bin/python2 > > +endif > > + > > +# retrieve pybind11 include paths > > +PYTHON_PYBIND_EXAMPLE_CXX_FLAGS += \ > > + $(shell $(PYTHON_PYBIND_PYTHON) -m pybind11 --includes) > > + > > +# .so to be installed must have exact suffix > > +# otherwise import() in python will not work > > +HOST_LIB_BINARY_SUFFIX = \ > > + $(shell $(PYTHON_PYBIND_PYTHON)-config --extension-suffix) > > + > > +define PYTHON_PYBIND_EXAMPLE_BUILD_CMDS > > + $(INSTALL) $(PYTHON_PYBIND_EXAMPLE_PKGDIR)/example.cpp $(@D) > > + cd $(@D); \ > > + $(HOSTCXX) $(PYTHON_PYBIND_EXAMPLE_CXX_FLAGS) \ > > + example.cpp -o > example$(HOST_LIB_BINARY_SUFFIX) > > +endef > > + > > +# installs into host/python sys.path > > +# so the new macro is naturally known > > +define PYTHON_PYBIND_EXAMPLE_INSTALL_TARGET_CMDS > > + $(INSTALL) -D -m 755 $(@D)/example$(HOST_LIB_BINARY_SUFFIX) \ > > + > $(HOST_DIR)/usr/lib/$(PYTHON_PYBIND_PYVER)/example$(HOST_LIB_BINARY_SUFFIX) > > +endef > > + > > +$(eval $(generic-package)) > > diff --git a/support/testing/tests/package/sample_python_pybind.py > b/support/testing/tests/package/sample_python_pybind.py > > new file mode 100644 > > index 0000000000..605c0bab15 > > --- /dev/null > > +++ b/support/testing/tests/package/sample_python_pybind.py > > @@ -0,0 +1,4 @@ > > +#!/usr/bin/env python > > +import example > > +print(example.add(1, 2)) > > +print(example.says) > > diff --git a/support/testing/tests/package/test_python_pybind.py > b/support/testing/tests/package/test_python_pybind.py > > new file mode 100644 > > index 0000000000..7a68d6512a > > --- /dev/null > > +++ b/support/testing/tests/package/test_python_pybind.py > > @@ -0,0 +1,48 @@ > > +import os > > +import infra > > +import subprocess > > +from tests.package.test_python import TestPythonPackageBase > > + > > +class TestPythonPybind (infra.basetest.BRTest): > > + config = TestPythonPackageBase.config # python minimal requirements > > + sample_script = "tests/package/sample_python_pybind.py" > > + # ship examples macro & installs it to host > > + br2_external = > [infra.filepath("tests/package/br2-external/python-pybind")] > > + > > +class TestPythonPy2Pybind (TestPythonPybind): > > + __test__ = True > > + config = TestPythonPybind.config + \ > > + """ > > + BR2_PACKAGE_PYTHON=y > > + BR2_PACKAGE_PYTHON_PYBIND=y > > + BR2_PACKAGE_PYTHON_PYBIND_EXAMPLE=y > > + """ > > + > > + def test_run(self): > > + host_interpreter = > os.path.join(self.builddir,"host","usr","bin","python") > > + # test new c++->py macro exposed to host interpreter > > + cmd = [host_interpreter, > infra.filepath(self.sample_script)] > > + output = subprocess.check_output(cmd) > > + result = int(output.splitlines()[0]) > > + attribute = output.splitlines()[1].decode("utf-8") > > + self.assertEqual(result,3) # 2+1 using '+' macro from cpp > > + self.assertEqual(attribute,"Hello World") # macro special > attribute > > + > > +class TestPythonPy3Pybind (TestPythonPybind): > > + __test__ = True > > + config = TestPythonPybind.config + \ > > + """ > > + BR2_PACKAGE_PYTHON3=y > > + BR2_PACKAGE_PYTHON_PYBIND=y > > + BR2_PACKAGE_PYTHON_PYBIND_EXAMPLE=y > > + """ > > + > > + def test_run(self): > > + host_interpreter = > os.path.join(self.builddir,"host","usr","bin","python") > > + # test new c++->py macro exposed to host interpreter > > + cmd = [host_interpreter, > infra.filepath(self.sample_script)] > > + output = subprocess.check_output(cmd) > > + result = int(output.splitlines()[0]) > > + attribute = output.splitlines()[1].decode("utf-8") > > + self.assertEqual(result,3) # 2+1 using '+' macro from cpp > > + self.assertEqual(attribute,"Hello World") # macro special > attribute > > > [-- Attachment #1.2: Type: text/html, Size: 18282 bytes --] [-- Attachment #2: Type: text/plain, Size: 150 bytes --] _______________________________________________ buildroot mailing list buildroot@buildroot.org https://lists.buildroot.org/mailman/listinfo/buildroot ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [Buildroot] [PATCH v2 1/3] package/python-pybind: fix upgrade to 2.6.1 2022-01-04 12:38 ` [Buildroot] [PATCH v2 1/3] package/python-pybind: fix upgrade to 2.6.1 guillaume.bressaix 2022-01-04 12:39 ` [Buildroot] [PATCH v2 2/3] package/python-pybind: bump to version 2.8.1 guillaume.bressaix 2022-01-04 12:39 ` [Buildroot] [PATCH v2 3/3] support/testing: add test for python-pybind guillaume.bressaix @ 2022-01-07 21:56 ` Arnout Vandecappelle 2 siblings, 0 replies; 21+ messages in thread From: Arnout Vandecappelle @ 2022-01-07 21:56 UTC (permalink / raw) To: guillaume.bressaix, buildroot; +Cc: tomi.valkeinen, yann.morin.1998 On 04/01/2022 13:38, guillaume.bressaix@gmail.com wrote: > From: "Guillaume W. Bres" <guillaume.bressaix@gmail.com> > > Fixes > http://autobuild.buildroot.net/results/b89f1de64b308dffa73675f1f31ccb0b7be5a10d > http://autobuild.buildroot.net/results/d0287b7f64f206b0f074908c5780a3632e0cb799 > http://autobuild.buildroot.net/results/27efb545a5a719a5581c8f746d3a3555ff4216ce > http://autobuild.buildroot.net/results/d2f0a0ad8f6c7178517df109e7d885dac9134c3a > http://autobuild.buildroot.net/results/b57e9a3279260dae4a590f9421238fcabb2f7cab > http://autobuild.buildroot.net/results/515e6f2fc6b5780260d98d6bb52b541ce4bf1afe > http://autobuild.buildroot.net/results/d89c4ecc81222d4f80c951da2232d2e393fa1c69 > > Signed-off-by: Guillaume W. Bres <guillaume.bressaix@gmail.com> I applied a completely reworked version to master, thanks. Regards, Arnout [snip] _______________________________________________ buildroot mailing list buildroot@buildroot.org https://lists.buildroot.org/mailman/listinfo/buildroot ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [Buildroot] [PATCH v2 1/2] package/python-pybind: fix upgrade to version 2.6.1 2022-01-03 8:24 ` Tomi Valkeinen 2022-01-04 12:38 ` [Buildroot] [PATCH v2 1/3] package/python-pybind: fix upgrade to 2.6.1 guillaume.bressaix @ 2022-01-07 21:49 ` Arnout Vandecappelle 2022-01-08 10:09 ` Tomi Valkeinen 1 sibling, 1 reply; 21+ messages in thread From: Arnout Vandecappelle @ 2022-01-07 21:49 UTC (permalink / raw) To: Tomi Valkeinen, Yann E. MORIN, guillaume.bressaix; +Cc: buildroot On 03/01/2022 09:24, Tomi Valkeinen wrote: > Hi, > > On 06/12/2021 23:57, Yann E. MORIN wrote: > >> So I think we really, really need a package (like scipy) that actually >> makes use of pybind, so we can see how it works, and to fix it... > > Jfyi, there are two relatively simple packages in buildroot that can use pybind: > kmsxx and rwmem. Both use meson for building. > > I can test the next version of this series with those packages. I tried to do that. Unfortunately, it turns out that both of those packages use the CMake way of finding the pybind11 dependency, and apparently our cross-compilation.conf doesn't support cmake yet... So it's not that trivial. I merged the pybind stuff but wasn't able to test the cmake handling, so it's even possible that that is broken as well... Regards, Arnout _______________________________________________ buildroot mailing list buildroot@buildroot.org https://lists.buildroot.org/mailman/listinfo/buildroot ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [Buildroot] [PATCH v2 1/2] package/python-pybind: fix upgrade to version 2.6.1 2022-01-07 21:49 ` [Buildroot] [PATCH v2 1/2] package/python-pybind: fix upgrade to version 2.6.1 Arnout Vandecappelle @ 2022-01-08 10:09 ` Tomi Valkeinen 2022-01-08 10:49 ` Arnout Vandecappelle 0 siblings, 1 reply; 21+ messages in thread From: Tomi Valkeinen @ 2022-01-08 10:09 UTC (permalink / raw) To: Arnout Vandecappelle, Yann E. MORIN, guillaume.bressaix; +Cc: buildroot On 07/01/2022 23:49, Arnout Vandecappelle wrote: > > > On 03/01/2022 09:24, Tomi Valkeinen wrote: >> Hi, >> >> On 06/12/2021 23:57, Yann E. MORIN wrote: >> >>> So I think we really, really need a package (like scipy) that actually >>> makes use of pybind, so we can see how it works, and to fix it... >> >> Jfyi, there are two relatively simple packages in buildroot that can >> use pybind: kmsxx and rwmem. Both use meson for building. >> >> I can test the next version of this series with those packages. > > I tried to do that. Unfortunately, it turns out that both of those > packages use the CMake way of finding the pybind11 dependency, and > apparently our cross-compilation.conf doesn't support cmake yet... So > it's not that trivial. Yes, I had a look at this also. kmsxx and rwmem use the plain meson dependency() to find the package. Works on Ubuntu, as Ubuntu's pybind11 installs the cmake files. I started testing with a more manual approach to finding pybind11, but didn't get it working yet. Tomi _______________________________________________ buildroot mailing list buildroot@buildroot.org https://lists.buildroot.org/mailman/listinfo/buildroot ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [Buildroot] [PATCH v2 1/2] package/python-pybind: fix upgrade to version 2.6.1 2022-01-08 10:09 ` Tomi Valkeinen @ 2022-01-08 10:49 ` Arnout Vandecappelle 0 siblings, 0 replies; 21+ messages in thread From: Arnout Vandecappelle @ 2022-01-08 10:49 UTC (permalink / raw) To: Tomi Valkeinen, Yann E. MORIN, guillaume.bressaix; +Cc: buildroot On 08/01/2022 11:09, Tomi Valkeinen wrote: > On 07/01/2022 23:49, Arnout Vandecappelle wrote: >> >> >> On 03/01/2022 09:24, Tomi Valkeinen wrote: >>> Hi, >>> >>> On 06/12/2021 23:57, Yann E. MORIN wrote: >>> >>>> So I think we really, really need a package (like scipy) that actually >>>> makes use of pybind, so we can see how it works, and to fix it... >>> >>> Jfyi, there are two relatively simple packages in buildroot that can use >>> pybind: kmsxx and rwmem. Both use meson for building. >>> >>> I can test the next version of this series with those packages. >> >> I tried to do that. Unfortunately, it turns out that both of those packages >> use the CMake way of finding the pybind11 dependency, and apparently our >> cross-compilation.conf doesn't support cmake yet... So it's not that trivial. > > Yes, I had a look at this also. kmsxx and rwmem use the plain meson dependency() > to find the package. Works on Ubuntu, as Ubuntu's pybind11 installs the cmake > files. I also made sure that python-pybind installs the cmake files. I just haven't tested if they actually work. I already noticed that there's one of the files that is not cross-compile friendly because it found /usr/bin/python. But maybe that one isn't actually used in practice. > I started testing with a more manual approach to finding pybind11, but didn't > get it working yet. No, the cmake approach should be fine. We just have to update the meson cross-compilation.conf so it points to STAGING_DIR/usr/share/cmake. Regards, Arnout _______________________________________________ buildroot mailing list buildroot@buildroot.org https://lists.buildroot.org/mailman/listinfo/buildroot ^ permalink raw reply [flat|nested] 21+ messages in thread
end of thread, other threads:[~2022-01-08 10:49 UTC | newest] Thread overview: 21+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2021-12-06 19:31 [Buildroot] [PATCH v2 1/2] package/python-pybind: fix upgrade to version 2.6.1 guillaume.bressaix 2021-12-06 19:31 ` [Buildroot] [PATCH v2 2/2] package/python-pybind: bump to version 2.8.1 guillaume.bressaix 2021-12-06 21:45 ` [Buildroot] [PATCH v2 1/2] package/python-pybind: fix upgrade to version 2.6.1 Yann E. MORIN 2021-12-08 9:09 ` Arnout Vandecappelle 2021-12-06 21:57 ` Yann E. MORIN 2021-12-07 8:04 ` Arnout Vandecappelle 2021-12-08 11:19 ` Andreas Naumann 2021-12-08 12:15 ` Guillaume Bres 2022-01-07 21:55 ` Arnout Vandecappelle 2021-12-08 17:18 ` Yann E. MORIN 2022-01-03 8:24 ` Tomi Valkeinen 2022-01-04 12:38 ` [Buildroot] [PATCH v2 1/3] package/python-pybind: fix upgrade to 2.6.1 guillaume.bressaix 2022-01-04 12:39 ` [Buildroot] [PATCH v2 2/3] package/python-pybind: bump to version 2.8.1 guillaume.bressaix 2022-01-07 22:10 ` Arnout Vandecappelle 2022-01-04 12:39 ` [Buildroot] [PATCH v2 3/3] support/testing: add test for python-pybind guillaume.bressaix 2022-01-07 21:57 ` Arnout Vandecappelle 2022-01-08 10:26 ` Guillaume Bres 2022-01-07 21:56 ` [Buildroot] [PATCH v2 1/3] package/python-pybind: fix upgrade to 2.6.1 Arnout Vandecappelle 2022-01-07 21:49 ` [Buildroot] [PATCH v2 1/2] package/python-pybind: fix upgrade to version 2.6.1 Arnout Vandecappelle 2022-01-08 10:09 ` Tomi Valkeinen 2022-01-08 10:49 ` Arnout Vandecappelle
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox