python3 on build host

python3 on build host

[Gary Thomas]

I have a build host that I've used for years.  It is not
possible to [directly] update python3 on this system to
one which is identified as suitable for use with bitbake.

To try and work around this, I created a meta-toolchain SDK
for my target.  I remember needing to do this years ago when
one of my boxes also became out of date.  Sadly, this process
has not led me to a solution.

First question: is this the proper way to solve this problem?
I simply can't update python3 on this box (I've tried), so I
need another solution so I can continue to use this workhorse.

Here's a list of the stumbling blocks I've run across so far.

* Using a SDK toolchain ostensibly is incompatible with the
normal bitbake flow.  Here's what I did for my 'meta-board' BSP:
   >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
   [gary@thor tmp]$ TEMPLATECONF=meta-board/conf . ./opt/amltd/poky/oe-init-build-env build
   Error: The OE SDK/ADT was detected as already being present in this shell environment. Please use a clean shell when 
sourcing this environment script.
   >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>

* I tried to work around this:
   >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
   [gary@thor tmp]$ unset OECORE_SDK_VERSION
   [gary@thor tmp]$ TEMPLATECONF=meta-board/conf . ./opt/amltd/poky/oe-init-build-env build
   You had no conf/local.conf file. This configuration file has therefore been
   created for you with some default values. You may wish to edit it to, for
   example, select a different MACHINE (target hardware). See conf/local.conf
   for more information as common configuration options are commented.

   You had no conf/bblayers.conf file. This configuration file has therefore been
   created for you with some default values. To add additional metadata layers
   into your configuration please add entries to conf/bblayers.conf.

   The Yocto Project has extensive documentation about OE including a reference
   manual which can be found at:
     http://yoctoproject.org/documentation

   For more information about OpenEmbedded see their website:
     http://www.openembedded.org/


   ### Shell environment set up for builds. ###

   You can now run 'bitbake <target>'
   >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>

   So, it looked like I was making some progress.  Sadly, it was short-lived:
   >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
   [gary@thor build]$ bitbake production-testing-image
   Traceback (most recent call last):
     File "/home/gary/tmp/opt/amltd/poky/bitbake/bin/bitbake", line 31, in <module>
       import bb
     File "/home/gary/tmp/opt/amltd/poky/bitbake/lib/bb/__init__.py", line 43, in <module>
       import logging
     File "/opt/amltd/2.1+snapshot/sysroots/i686-amltdsdk-linux/usr/lib/python3.5/logging/__init__.py", line 26, in <module>
       import sys, os, time, io, traceback, warnings, weakref, collections
     File "/opt/amltd/2.1+snapshot/sysroots/i686-amltdsdk-linux/usr/lib/python3.5/traceback.py", line 3, in <module>
       import collections
     File "/opt/amltd/2.1+snapshot/sysroots/i686-amltdsdk-linux/usr/lib/python3.5/collections/__init__.py", line 10, in 
<module>
       from operator import itemgetter as _itemgetter, eq as _eq
   ImportError: No module named 'operator'
   >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>

   I moved a step further by copying /usr/lib/python3.5/operator.py from one of
   my working build hosts into the appropriate sysroots in the SDK.  That only
   got me one more step along as I then needed _compat_pickle.py.  Then it failed
   with no module 'fcntl' and at this point, I don't know where to find that.

Second question: even if this is not the correct way to solve my initial problem
of providing a suitable python3 on my build host, shouldn't I be able to run
bitbake (or indeed any suitably complex python3 program) using the SDK I installed?

Thanks for your time and I truly hope I can find a working answer for question one.
I have deliveries upcoming to customers I know will not be happy that they have to
update and I've positioned our BSP deliveries solely on Poky/Yocto.

-- 
------------------------------------------------------------
Gary Thomas                 |  Consulting for the
MLB Associates              |    Embedded world
------------------------------------------------------------

Re: python3 on build host

[Burton, Ross]

[-- Attachment #1: Type: text/plain, Size: 1612 bytes --]

On 26 July 2016 at 15:09, Gary Thomas <gary@mlbassoc.com> wrote:

> I have a build host that I've used for years.  It is not
> possible to [directly] update python3 on this system to
> one which is identified as suitable for use with bitbake.
>
> To try and work around this, I created a meta-toolchain SDK
> for my target.  I remember needing to do this years ago when
> one of my boxes also became out of date.  Sadly, this process
> has not led me to a solution.
>
> First question: is this the proper way to solve this problem?
> I simply can't update python3 on this box (I've tried), so I
> need another solution so I can continue to use this workhorse.
>

The buildtools-tarball is what you want: you can either build your own
(bitbake buildtools-tarball) or just download the one we've built for you.
This looks like the right one:

http://downloads.yoctoproject.org/releases/yocto/yocto-2.1/buildtools/poky-glibc-x86_64-buildtools-tarball-core2-64-buildtools-nativesdk-standalone-2.1.sh

Second question: even if this is not the correct way to solve my initial
> problem
> of providing a suitable python3 on my build host, shouldn't I be able to
> run
> bitbake (or indeed any suitably complex python3 program) using the SDK I
> installed?


Looks like this SDK doesn't actually contain a full copy of Python, just
the pieces to make what you have in the SDK work (theoretically, at
least).  This is why we add python3-modules to the buildtools-tarball, so
if you want your SDK to ship its own complete Py3 runtime then add
python3-core python3-modules to it.

Ross

[-- Attachment #2: Type: text/html, Size: 2615 bytes --]

Re: python3 on build host

[Gary Thomas]

On 2016-07-26 16:20, Burton, Ross wrote:
>
> On 26 July 2016 at 15:09, Gary Thomas <gary@mlbassoc.com <mailto:gary@mlbassoc.com>> wrote:
>
>     I have a build host that I've used for years.  It is not
>     possible to [directly] update python3 on this system to
>     one which is identified as suitable for use with bitbake.
>
>     To try and work around this, I created a meta-toolchain SDK
>     for my target.  I remember needing to do this years ago when
>     one of my boxes also became out of date.  Sadly, this process
>     has not led me to a solution.
>
>     First question: is this the proper way to solve this problem?
>     I simply can't update python3 on this box (I've tried), so I
>     need another solution so I can continue to use this workhorse.
>
>
> The buildtools-tarball is what you want: you can either build your own (bitbake buildtools-tarball) or just download the
> one we've built for you.  This looks like the right one:
>
> http://downloads.yoctoproject.org/releases/yocto/yocto-2.1/buildtools/poky-glibc-x86_64-buildtools-tarball-core2-64-buildtools-nativesdk-standalone-2.1.sh
>

Yes, this is just what I needed (except I had to build my own because my box is i686)

My bitbake build is now running and it looks like this will work for me.

Thanks

>     Second question: even if this is not the correct way to solve my initial problem
>     of providing a suitable python3 on my build host, shouldn't I be able to run
>     bitbake (or indeed any suitably complex python3 program) using the SDK I installed?
>
>
> Looks like this SDK doesn't actually contain a full copy of Python, just the pieces to make what you have in the SDK
> work (theoretically, at least).  This is why we add python3-modules to the buildtools-tarball, so if you want your SDK
> to ship its own complete Py3 runtime then add python3-core python3-modules to it.

What would be the best way to do that?

-- 
------------------------------------------------------------
Gary Thomas                 |  Consulting for the
MLB Associates              |    Embedded world
------------------------------------------------------------

Re: python3 on build host

[Burton, Ross]

[-- Attachment #1: Type: text/plain, Size: 764 bytes --]

On 26 July 2016 at 15:57, Gary Thomas <gary@mlbassoc.com> wrote:

> Yes, this is just what I needed (except I had to build my own because my
> box is i686)
>

Oh good, next time we have a bug which only appears on 32-bit build hosts
I'll ping you. :)


> Looks like this SDK doesn't actually contain a full copy of Python, just
>> the pieces to make what you have in the SDK
>> work (theoretically, at least).  This is why we add python3-modules to
>> the buildtools-tarball, so if you want your SDK
>> to ship its own complete Py3 runtime then add python3-core
>> python3-modules to it.
>>
>
> What would be the best way to do that?
>

I think SDK_RDEPENDS += "nativesdk-python3-core nativesdk-python3-modules"
should do the trick.

Ross

[-- Attachment #2: Type: text/html, Size: 1496 bytes --]

Re: python3 on build host

[Paul Eggleton]

On Tue, 26 Jul 2016 15:20:34 Burton, Ross wrote:
> On 26 July 2016 at 15:09, Gary Thomas <gary@mlbassoc.com> wrote:
> > I have a build host that I've used for years.  It is not
> > possible to [directly] update python3 on this system to
> > one which is identified as suitable for use with bitbake.
> > 
> > To try and work around this, I created a meta-toolchain SDK
> > for my target.  I remember needing to do this years ago when
> > one of my boxes also became out of date.  Sadly, this process
> > has not led me to a solution.
> > 
> > First question: is this the proper way to solve this problem?
> > I simply can't update python3 on this box (I've tried), so I
> > need another solution so I can continue to use this workhorse.
> 
> The buildtools-tarball is what you want: you can either build your own
> (bitbake buildtools-tarball) or just download the one we've built for you.
> This looks like the right one:
> 
> http://downloads.yoctoproject.org/releases/yocto/yocto-2.1/buildtools/poky-g
> libc-x86_64-buildtools-tarball-core2-64-buildtools-nativesdk-standalone-2.1.
> sh

Not quite - that's the 2.1 buildtools so it contains Python 2, not 3.

We do have the 2.2 M1 version that does contain Python 3:

  http://downloads.yoctoproject.org/releases/yocto/milestones/yocto-2.2_M1/buildtools/poky-glibc-x86_64-buildtools-tarball-core2-64-buildtools-nativesdk-standalone-2.1%2bsnapshot-20160621.sh

(Still 64-bit though.)

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre