[Buildroot] [git commit] package/python-setuptools: add missing dependency on host-python-wheel

[Buildroot] [git commit] package/python-setuptools: add missing dependency on host-python-wheel

[Arnout Vandecappelle via buildroot]

commit: https://git.buildroot.net/buildroot/commit/?id=3b2913552ee4e304d4d86e577b237652a8f0a99f
branch: https://git.buildroot.net/buildroot/commit/?id=refs/heads/master

While migrating to pep517 build backend the host-python-wheel
dependency has been added to host-python-setuptools but
not to python-setuptools. Note that host-python-wheel is not really
needed during the build of (target) setuptools, but the setup.py script
checks if the dependency is present in the host directory.

To make everything consistent, add host-python-wheel as a build
dependency, and BR2_PACKAGE_PYTHON_WHEEL as a runtime dependency.

Fixes:
https://gitlab.com/buildroot.org/buildroot/-/jobs/4537951715

Signed-off-by: Romain Naour <romain.naour@gmail.com>
Cc: James Hilliard <james.hilliard1@gmail.com>
[Arnout: also add runtime dependency on the target package.]
Signed-off-by: Arnout Vandecappelle <arnout@mind.be>
---
 package/python-setuptools/Config.in            | 3 ++-
 package/python-setuptools/python-setuptools.mk | 1 +
 2 files changed, 3 insertions(+), 1 deletion(-)

diff --git a/package/python-setuptools/Config.in b/package/python-setuptools/Config.in
index 28228534f8..73368c1a49 100644
--- a/package/python-setuptools/Config.in
+++ b/package/python-setuptools/Config.in
@@ -1,7 +1,8 @@
 config BR2_PACKAGE_PYTHON_SETUPTOOLS
 	bool "python-setuptools"
-	select BR2_PACKAGE_PYTHON3_ZLIB # runtime
 	select BR2_PACKAGE_PYTHON3_PYEXPAT # runtime
+	select BR2_PACKAGE_PYTHON3_ZLIB # runtime
+	select BR2_PACKAGE_PYTHON_WHEEL # runtime
 	help
 	  Download, build, install, upgrade, and uninstall Python
 	  packages.
diff --git a/package/python-setuptools/python-setuptools.mk b/package/python-setuptools/python-setuptools.mk
index d957930455..39ead10f55 100644
--- a/package/python-setuptools/python-setuptools.mk
+++ b/package/python-setuptools/python-setuptools.mk
@@ -12,6 +12,7 @@ PYTHON_SETUPTOOLS_LICENSE_FILES = LICENSE
 PYTHON_SETUPTOOLS_CPE_ID_VENDOR = python
 PYTHON_SETUPTOOLS_CPE_ID_PRODUCT = setuptools
 PYTHON_SETUPTOOLS_SETUP_TYPE = pep517
+PYTHON_SETUPTOOLS_DEPENDENCIES = host-python-wheel
 HOST_PYTHON_SETUPTOOLS_DEPENDENCIES = host-python-wheel
 
 $(eval $(python-package))
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [git commit] package/python-setuptools: add missing dependency on host-python-wheel

[Thomas Petazzoni via buildroot]

On Mon, 3 Jul 2023 20:20:31 +0200
Arnout Vandecappelle via buildroot <buildroot@buildroot.org> wrote:

> commit: https://git.buildroot.net/buildroot/commit/?id=3b2913552ee4e304d4d86e577b237652a8f0a99f
> branch: https://git.buildroot.net/buildroot/commit/?id=refs/heads/master
> 
> While migrating to pep517 build backend the host-python-wheel
> dependency has been added to host-python-setuptools but
> not to python-setuptools. Note that host-python-wheel is not really
> needed during the build of (target) setuptools, but the setup.py script
> checks if the dependency is present in the host directory.
> 
> To make everything consistent, add host-python-wheel as a build
> dependency, and BR2_PACKAGE_PYTHON_WHEEL as a runtime dependency.

Hu, this was not in the patch from Romain I believe.

> +	select BR2_PACKAGE_PYTHON_WHEEL # runtime

This is not correct, there is no such symbol in Buildroot, and it
causes a check-symbol error:

https://gitlab.com/buildroot.org/buildroot/-/jobs/4586144848

What was the motivation behind this additional select?

Thomas
-- 
Thomas Petazzoni, co-owner and CEO, Bootlin
Embedded Linux and Kernel engineering and training
https://bootlin.com
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [git commit] package/python-setuptools: add missing dependency on host-python-wheel

[Yann E. MORIN]

Thomas, Arnout, All,

On 2023-07-03 20:56 +0200, Thomas Petazzoni via buildroot spake thusly:
> On Mon, 3 Jul 2023 20:20:31 +0200
> Arnout Vandecappelle via buildroot <buildroot@buildroot.org> wrote:
> 
> > commit: https://git.buildroot.net/buildroot/commit/?id=3b2913552ee4e304d4d86e577b237652a8f0a99f
> > branch: https://git.buildroot.net/buildroot/commit/?id=refs/heads/master
> > 
> > While migrating to pep517 build backend the host-python-wheel
> > dependency has been added to host-python-setuptools but
> > not to python-setuptools. Note that host-python-wheel is not really
> > needed during the build of (target) setuptools, but the setup.py script
> > checks if the dependency is present in the host directory.
> > 
> > To make everything consistent, add host-python-wheel as a build
> > dependency, and BR2_PACKAGE_PYTHON_WHEEL as a runtime dependency.
> Hu, this was not in the patch from Romain I believe.
> > +	select BR2_PACKAGE_PYTHON_WHEEL # runtime
> This is not correct, there is no such symbol in Buildroot, and it
> causes a check-symbol error:
> https://gitlab.com/buildroot.org/buildroot/-/jobs/4586144848
> What was the motivation behind this additional select?

In the meantime, I've pushed a fixup commit that drops this spurious
select; if there was indeed a good rationaly for having python-wheel on
the target, as a runtiem dependency of python-setuptools, this is going
to need a bit more work.

The fixup does not maek the situation worse than it was before either.

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [git commit] package/python-setuptools: add missing dependency on host-python-wheel

[Arnout Vandecappelle via buildroot]

On 03/07/2023 20:56, Thomas Petazzoni wrote:
> On Mon, 3 Jul 2023 20:20:31 +0200
> Arnout Vandecappelle via buildroot <buildroot@buildroot.org> wrote:
> 
>> commit: https://git.buildroot.net/buildroot/commit/?id=3b2913552ee4e304d4d86e577b237652a8f0a99f
>> branch: https://git.buildroot.net/buildroot/commit/?id=refs/heads/master
>>
>> While migrating to pep517 build backend the host-python-wheel
>> dependency has been added to host-python-setuptools but
>> not to python-setuptools. Note that host-python-wheel is not really
>> needed during the build of (target) setuptools, but the setup.py script
>> checks if the dependency is present in the host directory.
>>
>> To make everything consistent, add host-python-wheel as a build
>> dependency, and BR2_PACKAGE_PYTHON_WHEEL as a runtime dependency.
> 
> Hu, this was not in the patch from Romain I believe.
> 
>> +	select BR2_PACKAGE_PYTHON_WHEEL # runtime
> 
> This is not correct, there is no such symbol in Buildroot, and it
> causes a check-symbol error:
> 
> https://gitlab.com/buildroot.org/buildroot/-/jobs/4586144848
> 
> What was the motivation behind this additional select?

  Because setuptools has a runtime dependency on wheel. The pep517 
infrastructure does a build-time check of this runtime dependency, but due to 
the way we run python, the build-time check looks at what is installed for the 
host, not what is installed for the target. Therefore, we have a dependency on 
host-python-wheel instead of target python-wheel.

  Since setuptools clearly defines that it has a dependency on wheel, I thought 
it logical that we would reflect that dependency in Buildroot as well. It is 
really a runtime dependency, not a build-time dependency, so I added this select 
like we do for runtime dependencies.

  Of course, I didn't notice that we don't have a target python-wheel package, 
so it doesn't actually work.

  So, dropping this select I'm pretty sure is going to break something, at some 
point. I've looked a bit closer now, and it turns out that wheel is actually 
only imported in one specific function, in the "editable_wheel" command. So it's 
quite unlikely anyone will ever see this breakage.

  However, now I look a bit more in detail at it, it looks like this commit is 
in fact bogus, and Yann's original comment that the wheel dependency should be 
added to the infrastructure is in fact correct. From the documentation:

    Historically this documentation has unnecessarily listed ``wheel``
    in the ``requires`` list, and many projects still do that. This is
    not recommended. The backend automatically adds ``wheel`` dependency
    when it is required, and listing it explicitly causes it to be
    unnecessarily required for source distribution builds.

And it's pretty clear what is adding this dependency: the pep517 infra calls the 
build with

    python -m build -n -w

The "-w" stands for "build a wheel" and I guess it's obvious that that implies a 
dependency on host-python-wheel? And I guess that the relatively few pep517 
packages we have already have a setuptools-based package (and therefore 
host-python-setuptools and host-python-wheel) somewhere in their dependency chain.

  So, I do think this dependency should move the the pkg-python infra _for 
pep517 packages_, as Yann originally commented.

  Regards,
  Arnout
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [git commit] package/python-setuptools: add missing dependency on host-python-wheel

[James Hilliard]

On Tue, Jul 4, 2023 at 2:28 PM Arnout Vandecappelle <arnout@mind.be> wrote:
>
>
>
> On 03/07/2023 20:56, Thomas Petazzoni wrote:
> > On Mon, 3 Jul 2023 20:20:31 +0200
> > Arnout Vandecappelle via buildroot <buildroot@buildroot.org> wrote:
> >
> >> commit: https://git.buildroot.net/buildroot/commit/?id=3b2913552ee4e304d4d86e577b237652a8f0a99f
> >> branch: https://git.buildroot.net/buildroot/commit/?id=refs/heads/master
> >>
> >> While migrating to pep517 build backend the host-python-wheel
> >> dependency has been added to host-python-setuptools but
> >> not to python-setuptools. Note that host-python-wheel is not really
> >> needed during the build of (target) setuptools, but the setup.py script
> >> checks if the dependency is present in the host directory.
> >>
> >> To make everything consistent, add host-python-wheel as a build
> >> dependency, and BR2_PACKAGE_PYTHON_WHEEL as a runtime dependency.
> >
> > Hu, this was not in the patch from Romain I believe.
> >
> >> +    select BR2_PACKAGE_PYTHON_WHEEL # runtime
> >
> > This is not correct, there is no such symbol in Buildroot, and it
> > causes a check-symbol error:
> >
> > https://gitlab.com/buildroot.org/buildroot/-/jobs/4586144848
> >
> > What was the motivation behind this additional select?
>
>   Because setuptools has a runtime dependency on wheel. The pep517
> infrastructure does a build-time check of this runtime dependency, but due to
> the way we run python, the build-time check looks at what is installed for the
> host, not what is installed for the target. Therefore, we have a dependency on
> host-python-wheel instead of target python-wheel.
>
>   Since setuptools clearly defines that it has a dependency on wheel, I thought
> it logical that we would reflect that dependency in Buildroot as well. It is
> really a runtime dependency, not a build-time dependency, so I added this select
> like we do for runtime dependencies.
>
>   Of course, I didn't notice that we don't have a target python-wheel package,
> so it doesn't actually work.
>
>   So, dropping this select I'm pretty sure is going to break something, at some
> point. I've looked a bit closer now, and it turns out that wheel is actually
> only imported in one specific function, in the "editable_wheel" command. So it's
> quite unlikely anyone will ever see this breakage.
>
>   However, now I look a bit more in detail at it, it looks like this commit is
> in fact bogus, and Yann's original comment that the wheel dependency should be
> added to the infrastructure is in fact correct. From the documentation:
>
>     Historically this documentation has unnecessarily listed ``wheel``
>     in the ``requires`` list, and many projects still do that. This is
>     not recommended. The backend automatically adds ``wheel`` dependency
>     when it is required, and listing it explicitly causes it to be
>     unnecessarily required for source distribution builds.
>
> And it's pretty clear what is adding this dependency: the pep517 infra calls the
> build with
>
>     python -m build -n -w
>
> The "-w" stands for "build a wheel" and I guess it's obvious that that implies a
> dependency on host-python-wheel? And I guess that the relatively few pep517
> packages we have already have a setuptools-based package (and therefore
> host-python-setuptools and host-python-wheel) somewhere in their dependency chain.

So we don't need host-python-wheel for non-setuptools pep517 backends at all.

The host-python-wheel package is basically only designed to be used by
setuptools,
no other pep517 backends/frontends use it AFAIU.

>
>   So, I do think this dependency should move the the pkg-python infra _for
> pep517 packages_, as Yann originally commented.

That doesn't sound right, only host-python-setuptools should need the
host-python-wheel dependency AFAIU. Also host-python-wheel depends
on our pep517 infrastructure itself so host-python-wheel could not be a
dependency for our pep517 infrastructure as that would create a circular
dependency.

FYI I have a patch migrating our setuptools infrastructure to use pep517:
https://patchwork.ozlabs.org/project/buildroot/patch/20230626181531.2312002-5-james.hilliard1@gmail.com/

>
>   Regards,
>   Arnout
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [git commit] package/python-setuptools: add missing dependency on host-python-wheel

[Yann E. MORIN]

Arnout, All,

On 2023-07-04 22:28 +0200, Arnout Vandecappelle spake thusly:
> On 03/07/2023 20:56, Thomas Petazzoni wrote:
> >On Mon, 3 Jul 2023 20:20:31 +0200
> >Arnout Vandecappelle via buildroot <buildroot@buildroot.org> wrote:
> >Hu, this was not in the patch from Romain I believe.
> >>+	select BR2_PACKAGE_PYTHON_WHEEL # runtime
> >This is not correct, there is no such symbol in Buildroot, and it
> >causes a check-symbol error:
> >https://gitlab.com/buildroot.org/buildroot/-/jobs/4586144848
> >What was the motivation behind this additional select?
> 
>  Because setuptools has a runtime dependency on wheel. The pep517
> infrastructure does a build-time check of this runtime dependency, but due
> to the way we run python, the build-time check looks at what is installed
> for the host, not what is installed for the target. Therefore, we have a
> dependency on host-python-wheel instead of target python-wheel.
> 
>  Since setuptools clearly defines that it has a dependency on wheel, I
> thought it logical that we would reflect that dependency in Buildroot as
> well. It is really a runtime dependency, not a build-time dependency, so I
> added this select like we do for runtime dependencies.
> 
>  Of course, I didn't notice that we don't have a target python-wheel
> package, so it doesn't actually work.
> 
>  So, dropping this select I'm pretty sure is going to break something, at

Dropping the select can not break anything: there is no package to
select to begin with. So it can't be more broken than it was before the
select was introduced.

Of course, if we really need wheel as a target package, then it should
be made such a package before we can select it.

Although having sheel on the target jsut because some (most?) packages
have a spurious dependency on it, would be seriously a shame...

> some point. I've looked a bit closer now, and it turns out that wheel is
> actually only imported in one specific function, in the "editable_wheel"
> command. So it's quite unlikely anyone will ever see this breakage.
> 
>  However, now I look a bit more in detail at it, it looks like this commit
> is in fact bogus, and Yann's original comment that the wheel dependency
> should be added to the infrastructure is in fact correct. From the
> documentation:
> 
>    Historically this documentation has unnecessarily listed ``wheel``
>    in the ``requires`` list, and many projects still do that. This is
>    not recommended. The backend automatically adds ``wheel`` dependency
>    when it is required, and listing it explicitly causes it to be
>    unnecessarily required for source distribution builds.
> 
> And it's pretty clear what is adding this dependency: the pep517 infra calls
> the build with
> 
>    python -m build -n -w
> 
> The "-w" stands for "build a wheel" and I guess it's obvious that that
> implies a dependency on host-python-wheel? And I guess that the relatively
> few pep517 packages we have already have a setuptools-based package (and
> therefore host-python-setuptools and host-python-wheel) somewhere in their
> dependency chain.
> 
>  So, I do think this dependency should move the the pkg-python infra _for
> pep517 packages_, as Yann originally commented.

I'd defer to James to find a better solution, because I don't grasp even
partially all this pep517-introduced pain...

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [git commit] package/python-setuptools: add missing dependency on host-python-wheel

[James Hilliard]

On Thu, Jul 6, 2023 at 2:28 PM Yann E. MORIN <yann.morin.1998@free.fr> wrote:
>
> Arnout, All,
>
> On 2023-07-04 22:28 +0200, Arnout Vandecappelle spake thusly:
> > On 03/07/2023 20:56, Thomas Petazzoni wrote:
> > >On Mon, 3 Jul 2023 20:20:31 +0200
> > >Arnout Vandecappelle via buildroot <buildroot@buildroot.org> wrote:
> > >Hu, this was not in the patch from Romain I believe.
> > >>+   select BR2_PACKAGE_PYTHON_WHEEL # runtime
> > >This is not correct, there is no such symbol in Buildroot, and it
> > >causes a check-symbol error:
> > >https://gitlab.com/buildroot.org/buildroot/-/jobs/4586144848
> > >What was the motivation behind this additional select?
> >
> >  Because setuptools has a runtime dependency on wheel. The pep517
> > infrastructure does a build-time check of this runtime dependency, but due
> > to the way we run python, the build-time check looks at what is installed
> > for the host, not what is installed for the target. Therefore, we have a
> > dependency on host-python-wheel instead of target python-wheel.
> >
> >  Since setuptools clearly defines that it has a dependency on wheel, I
> > thought it logical that we would reflect that dependency in Buildroot as
> > well. It is really a runtime dependency, not a build-time dependency, so I
> > added this select like we do for runtime dependencies.
> >
> >  Of course, I didn't notice that we don't have a target python-wheel
> > package, so it doesn't actually work.
> >
> >  So, dropping this select I'm pretty sure is going to break something, at
>
> Dropping the select can not break anything: there is no package to
> select to begin with. So it can't be more broken than it was before the
> select was introduced.
>
> Of course, if we really need wheel as a target package, then it should
> be made such a package before we can select it.

I don't think we need a target wheel as wheel is only really used for
pep517 builds(and we don't have a target pep517 toolchain/frontend).

>
> Although having sheel on the target jsut because some (most?) packages
> have a spurious dependency on it, would be seriously a shame...
>
> > some point. I've looked a bit closer now, and it turns out that wheel is
> > actually only imported in one specific function, in the "editable_wheel"
> > command. So it's quite unlikely anyone will ever see this breakage.
> >
> >  However, now I look a bit more in detail at it, it looks like this commit
> > is in fact bogus, and Yann's original comment that the wheel dependency
> > should be added to the infrastructure is in fact correct. From the
> > documentation:
> >
> >    Historically this documentation has unnecessarily listed ``wheel``
> >    in the ``requires`` list, and many projects still do that. This is
> >    not recommended. The backend automatically adds ``wheel`` dependency
> >    when it is required, and listing it explicitly causes it to be
> >    unnecessarily required for source distribution builds.
> >
> > And it's pretty clear what is adding this dependency: the pep517 infra calls
> > the build with
> >
> >    python -m build -n -w
> >
> > The "-w" stands for "build a wheel" and I guess it's obvious that that
> > implies a dependency on host-python-wheel? And I guess that the relatively
> > few pep517 packages we have already have a setuptools-based package (and
> > therefore host-python-setuptools and host-python-wheel) somewhere in their
> > dependency chain.
> >
> >  So, I do think this dependency should move the the pkg-python infra _for
> > pep517 packages_, as Yann originally commented.
>
> I'd defer to James to find a better solution, because I don't grasp even
> partially all this pep517-introduced pain...

I think the host-python-wheel build dependency is sufficient for
target setuptools,
setuptools uses an in-tree setuptools pep517 based build with wheel as a build
dependency.

See:
https://github.com/pypa/setuptools/blob/v68.0.0/pyproject.toml#L4
https://github.com/pypa/setuptools/blob/v68.0.0/setuptools/build_meta.py#L341

>
> Regards,
> Yann E. MORIN.
>
> --
> .-----------------.--------------------.------------------.--------------------.
> |  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
> | +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
> | +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
> | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
> '------------------------------^-------^------------------^--------------------'
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot