[Buildroot] host-python-2.7 failure to build _hashlib

[Buildroot] host-python-2.7 failure to build _hashlib

[Vladimir Oltean]

Hi Buildroot community,

I have a noob question:

I am trying to complete a build and get this error for host-python:

Python build finished, but the necessary bits to build these modules
were not found:
_bsddb             _tkinter           bsddb185
dbm                dl                 gdbm
imageop            sunaudiodev
To find the necessary bits, look in setup.py in detect_modules() for
the module's name.


Failed to build these modules:
_hashlib

Ok, but I have to ask: why is host-python even built at all? For one
thing, BR2_PACKAGE_HOST_PYTHON is not set.
But with "make graph-depends", I am able to see that host-python is
dragged in by:
- python itself, which I selected
- host-python-setuptools (dragged in by python-lxml, which I selected)
- host-nodejs (dragged in by nodejs, which I selected)

Granted, I don't know much, but I'm baffled that:
- None of the above 3 packages select BR2_PACKAGE_HOST_PYTHON in their Config.in
- The Makefile doesn't complain that they don't (usually it does, when
packages add _DEPENDENCIES that are absent from Config.in)

When I enable BR2_PACKAGE_HOST_PYTHON and BR2_PACKAGE_HOST_PYTHON_SSL
in .config, I am able to carry on with the build.

Can somebody please explain what is going on?
I am not a proficient Python user either, by the way.

Regards,
-Vladimir

[Buildroot] host-python-2.7 failure to build _hashlib

[Thomas Petazzoni]

Hello Vladimir,

On Tue, 28 Jan 2020 00:35:49 +0200
Vladimir Oltean <olteanv@gmail.com> wrote:

> Ok, but I have to ask: why is host-python even built at all? For one
> thing, BR2_PACKAGE_HOST_PYTHON is not set.
> But with "make graph-depends", I am able to see that host-python is
> dragged in by:
> - python itself, which I selected
> - host-python-setuptools (dragged in by python-lxml, which I selected)
> - host-nodejs (dragged in by nodejs, which I selected)

Right, host-python is a dependency of all those things indeed.

> Granted, I don't know much, but I'm baffled that:
> - None of the above 3 packages select BR2_PACKAGE_HOST_PYTHON in their Config.in

Because we don't enforce this for host packages. We do enforce it for
target packages (i.e a target package cannot be built if its Config.in
option is not enabled somehow). But we don't enforce it for host
packages.

For example, host-pkgconf has a visible Config.in option called
BR2_PACKAGE_HOST_PKGCONF. However, almost none of the packages that
depend on host-pkgconf have a select BR2_PACKAGE_HOST_PKGCONF.

> - The Makefile doesn't complain that they don't (usually it does, when
> packages add _DEPENDENCIES that are absent from Config.in)

See above: only for target packages.

> When I enable BR2_PACKAGE_HOST_PYTHON and BR2_PACKAGE_HOST_PYTHON_SSL
> in .config, I am able to carry on with the build.
> 
> Can somebody please explain what is going on?
> I am not a proficient Python user either, by the way.

I don't really understand why your host-python package fails to build.
Could you share the entire build log of the host-python package ?

Best regards,

Thomas
-- 
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

[Buildroot] host-python-2.7 failure to build _hashlib

[Vladimir Oltean]

Hi Thomas,

On Tue, 28 Jan 2020 at 00:40, Thomas Petazzoni
<thomas.petazzoni@bootlin.com> wrote:
>
> Hello Vladimir,
>
> On Tue, 28 Jan 2020 00:35:49 +0200
> Vladimir Oltean <olteanv@gmail.com> wrote:
>
> > Ok, but I have to ask: why is host-python even built at all? For one
> > thing, BR2_PACKAGE_HOST_PYTHON is not set.
> > But with "make graph-depends", I am able to see that host-python is
> > dragged in by:
> > - python itself, which I selected
> > - host-python-setuptools (dragged in by python-lxml, which I selected)
> > - host-nodejs (dragged in by nodejs, which I selected)
>
> Right, host-python is a dependency of all those things indeed.
>
> > Granted, I don't know much, but I'm baffled that:
> > - None of the above 3 packages select BR2_PACKAGE_HOST_PYTHON in their Config.in
>
> Because we don't enforce this for host packages. We do enforce it for
> target packages (i.e a target package cannot be built if its Config.in
> option is not enabled somehow). But we don't enforce it for host
> packages.
>
> For example, host-pkgconf has a visible Config.in option called
> BR2_PACKAGE_HOST_PKGCONF. However, almost none of the packages that
> depend on host-pkgconf have a select BR2_PACKAGE_HOST_PKGCONF.
>

Why not?

> > - The Makefile doesn't complain that they don't (usually it does, when
> > packages add _DEPENDENCIES that are absent from Config.in)
>
> See above: only for target packages.
>
> > When I enable BR2_PACKAGE_HOST_PYTHON and BR2_PACKAGE_HOST_PYTHON_SSL
> > in .config, I am able to carry on with the build.
> >
> > Can somebody please explain what is going on?
> > I am not a proficient Python user either, by the way.
>
> I don't really understand why your host-python package fails to build.
> Could you share the entire build log of the host-python package ?
>

So it looks like the mailing list didn't like my log pasted in here.
Let's try Github gists:
https://gist.github.com/vladimiroltean/4d6b1441b0dbe64dc8e2f9d1ce336e42

> Best regards,
>
> Thomas
> --
> Thomas Petazzoni, CTO, Bootlin
> Embedded Linux and Kernel engineering
> https://bootlin.com

Thanks,
-Vladimir

[Buildroot] host-python-2.7 failure to build _hashlib

[Thomas Petazzoni]

Hello Vladimir,

On Tue, 28 Jan 2020 01:39:57 +0200
Vladimir Oltean <olteanv@gmail.com> wrote:

> > Because we don't enforce this for host packages. We do enforce it for
> > target packages (i.e a target package cannot be built if its Config.in
> > option is not enabled somehow). But we don't enforce it for host
> > packages.
> >
> > For example, host-pkgconf has a visible Config.in option called
> > BR2_PACKAGE_HOST_PKGCONF. However, almost none of the packages that
> > depend on host-pkgconf have a select BR2_PACKAGE_HOST_PKGCONF.
> >  
> 
> Why not?

That's mainly historical. We used to not have any visible Config.in
option for host packages: host packages were only built as build
dependencies of target packages. But then, we realized that some host
packages were not just mere build dependencies of target packages: they
were useful by themselves. This is the case for flashing tools,
debugging tools, filesystem image creation utilities, and more. So we
started adding Config.in options for host packages.

We have been considering adding Config.in options for all host packages
(some visible, some hidden), but adding all those "select" was
considered too much of a burden compared to the benefit.

We had some discussion about this at a previous Buildroot Developers
meeting, after FOSDEM 2019. See the notes of the discussion:
https://elinux.org/Buildroot:DeveloperDaysFOSDEM2019#.22Config.in.host.22_for_all_host_packages.

> > > - The Makefile doesn't complain that they don't (usually it does, when
> > > packages add _DEPENDENCIES that are absent from Config.in)  
> >
> > See above: only for target packages.
> >  
> > > When I enable BR2_PACKAGE_HOST_PYTHON and BR2_PACKAGE_HOST_PYTHON_SSL
> > > in .config, I am able to carry on with the build.
> > >
> > > Can somebody please explain what is going on?
> > > I am not a proficient Python user either, by the way.  
> >
> > I don't really understand why your host-python package fails to build.
> > Could you share the entire build log of the host-python package ?
> >  
> 
> So it looks like the mailing list didn't like my log pasted in here.
> Let's try Github gists:
> https://gist.github.com/vladimiroltean/4d6b1441b0dbe64dc8e2f9d1ce336e42

We're not explicitly disabling building the hashlib module when
building host-python, and hashlib needs OpenSSL, but apparently the
OpenSSL of your system is not compatible with Python 2.x.

Could you try the following change ?

diff --git a/package/python/python.mk b/package/python/python.mk
index 41a981e3d3..cd79be78f6 100644
--- a/package/python/python.mk
+++ b/package/python/python.mk
@@ -33,7 +33,8 @@ HOST_PYTHON_CONF_OPTS += \
        --disable-test-modules \
        --disable-bz2 \
        --disable-ossaudiodev \
-       --disable-pyo-build
+       --disable-pyo-build \
+       --disable-hashlib
 
 # Make sure that LD_LIBRARY_PATH overrides -rpath.
 # This is needed because libpython may be installed at the same time that

Thanks,

Thomas
-- 
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

[Buildroot] host-python-2.7 failure to build _hashlib

[Vladimir Oltean]

Hi Thomas,

On Tue, 28 Jan 2020 at 11:14, Thomas Petazzoni
<thomas.petazzoni@bootlin.com> wrote:
>
> Hello Vladimir,
>
> On Tue, 28 Jan 2020 01:39:57 +0200
> Vladimir Oltean <olteanv@gmail.com> wrote:
>
> > > Because we don't enforce this for host packages. We do enforce it for
> > > target packages (i.e a target package cannot be built if its Config.in
> > > option is not enabled somehow). But we don't enforce it for host
> > > packages.
> > >
> > > For example, host-pkgconf has a visible Config.in option called
> > > BR2_PACKAGE_HOST_PKGCONF. However, almost none of the packages that
> > > depend on host-pkgconf have a select BR2_PACKAGE_HOST_PKGCONF.
> > >
> >
> > Why not?
>
> That's mainly historical. We used to not have any visible Config.in
> option for host packages: host packages were only built as build
> dependencies of target packages. But then, we realized that some host
> packages were not just mere build dependencies of target packages: they
> were useful by themselves. This is the case for flashing tools,
> debugging tools, filesystem image creation utilities, and more. So we
> started adding Config.in options for host packages.
>
> We have been considering adding Config.in options for all host packages
> (some visible, some hidden), but adding all those "select" was
> considered too much of a burden compared to the benefit.
>
> We had some discussion about this at a previous Buildroot Developers
> meeting, after FOSDEM 2019. See the notes of the discussion:
> https://elinux.org/Buildroot:DeveloperDaysFOSDEM2019#.22Config.in.host.22_for_all_host_packages.
>
> > > > - The Makefile doesn't complain that they don't (usually it does, when
> > > > packages add _DEPENDENCIES that are absent from Config.in)
> > >
> > > See above: only for target packages.
> > >
> > > > When I enable BR2_PACKAGE_HOST_PYTHON and BR2_PACKAGE_HOST_PYTHON_SSL
> > > > in .config, I am able to carry on with the build.
> > > >
> > > > Can somebody please explain what is going on?
> > > > I am not a proficient Python user either, by the way.
> > >
> > > I don't really understand why your host-python package fails to build.
> > > Could you share the entire build log of the host-python package ?
> > >
> >
> > So it looks like the mailing list didn't like my log pasted in here.
> > Let's try Github gists:
> > https://gist.github.com/vladimiroltean/4d6b1441b0dbe64dc8e2f9d1ce336e42
>
> We're not explicitly disabling building the hashlib module when
> building host-python, and hashlib needs OpenSSL, but apparently the
> OpenSSL of your system is not compatible with Python 2.x.
>
> Could you try the following change ?
>
> diff --git a/package/python/python.mk b/package/python/python.mk
> index 41a981e3d3..cd79be78f6 100644
> --- a/package/python/python.mk
> +++ b/package/python/python.mk
> @@ -33,7 +33,8 @@ HOST_PYTHON_CONF_OPTS += \
>         --disable-test-modules \
>         --disable-bz2 \
>         --disable-ossaudiodev \
> -       --disable-pyo-build
> +       --disable-pyo-build \
> +       --disable-hashlib

Yes, this works, but I'm not exactly sure what conclusion I should draw from it.
hashlib seems to require BR2_PACKAGE_HOST_PYTHON_SSL, but without a
proper dependency tracking system for host packages, I don't know how
it can be disabled in that case.

>
>  # Make sure that LD_LIBRARY_PATH overrides -rpath.
>  # This is needed because libpython may be installed at the same time that
>
> Thanks,
>
> Thomas
> --
> Thomas Petazzoni, CTO, Bootlin
> Embedded Linux and Kernel engineering
> https://bootlin.com

Thanks,
-Vladimir

[Buildroot] host-python-2.7 failure to build _hashlib

[Thomas Petazzoni]

On Tue, 28 Jan 2020 12:28:01 +0200
Vladimir Oltean <olteanv@gmail.com> wrote:

> > diff --git a/package/python/python.mk b/package/python/python.mk
> > index 41a981e3d3..cd79be78f6 100644
> > --- a/package/python/python.mk
> > +++ b/package/python/python.mk
> > @@ -33,7 +33,8 @@ HOST_PYTHON_CONF_OPTS += \
> >         --disable-test-modules \
> >         --disable-bz2 \
> >         --disable-ossaudiodev \
> > -       --disable-pyo-build
> > +       --disable-pyo-build \
> > +       --disable-hashlib  
> 
> Yes, this works, but I'm not exactly sure what conclusion I should draw from it.

I explained above the patch in my previous e-mail what (I believe) was
happening.

> hashlib seems to require BR2_PACKAGE_HOST_PYTHON_SSL, but without a
> proper dependency tracking system for host packages, I don't know how
> it can be disabled in that case.

Three solutions:

 - We hook into the BR2_PACKAGE_HOST_PYTHON_SSL option, and add
   --disable-hashlib in the "else" case of this condition. This makes
   the BR2_PACKAGE_HOST_PYTHON_SSL option have the semantic "build all
   Python modules that depend on OpenSSL".

 - We create a separate BR2_PACKAGE_HOST_PYTHON_HASHLIB option, so that
   the semantic of BR2_PACKAGE_HOST_PYTHON_SSL remains "build the
   Python ssl module"

 - We simply unconditionally add --disable-hashlib like I proposed, and
   wait until someone needs hashlib in host-python.

I suggest we take the third solution for now.

Best regards,

Thomas
-- 
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com