[PATCH] Yocto: Install full set of python modules in Qt SDK toolchain

[PATCH] Yocto: Install full set of python modules in Qt SDK toolchain

[Marek Vasut]

The Qt SDK toolchain pulls in python via packagegroup-cross-canadian-${MACHINE}
and ships it. But the python is missing many modules and is rather incomplete.

The environment-setup-* script configures the PATH variable to point into
it's own sysroot first, it means the python from the SDK is used as well
when the environment is configured. It actually does make sense to use the
python from the SDK, since the SDK should provide the tools needed to build
additional software.

The problem is, the python in the SDK image is stripped to a bare minimum
of modules, which prevents some software from being built with the SDK as
is. Add the same python modules as the buildtools-tarball.bb adds to make
the python shipped with the SDK complete.

Signed-off-by: Marek Vasut <marex@denx.de>
---
 meta/recipes-qt/meta/meta-toolchain-qt.inc | 32 +++++++++++++++++++++++++++++-
 1 file changed, 31 insertions(+), 1 deletion(-)

NOTE: This lets me build U-Boot git HEAD , which now has Kconfig in it
      and that depends on the "subprocess" module. But instead of adding
      just one module, I want to fix this properly and future-proof this.

diff --git a/meta/recipes-qt/meta/meta-toolchain-qt.inc b/meta/recipes-qt/meta/meta-toolchain-qt.inc
index 6b162bd..99d5e64 100644
--- a/meta/recipes-qt/meta/meta-toolchain-qt.inc
+++ b/meta/recipes-qt/meta/meta-toolchain-qt.inc
@@ -1,4 +1,34 @@
-TOOLCHAIN_HOST_TASK = "nativesdk-packagegroup-${QTNAME}-toolchain-host packagegroup-cross-canadian-${MACHINE}"
+TOOLCHAIN_HOST_TASK = " \
+	nativesdk-packagegroup-${QTNAME}-toolchain-host \
+	packagegroup-cross-canadian-${MACHINE} \
+	nativesdk-python-core \
+	nativesdk-python-textutils \
+	nativesdk-python-sqlite3 \
+	nativesdk-python-pickle \
+	nativesdk-python-logging \
+	nativesdk-python-elementtree \
+	nativesdk-python-curses \
+	nativesdk-python-compile \
+	nativesdk-python-compiler \
+	nativesdk-python-fcntl \
+	nativesdk-python-shell \
+	nativesdk-python-misc \
+	nativesdk-python-multiprocessing \
+	nativesdk-python-subprocess \
+	nativesdk-python-xmlrpc \
+	nativesdk-python-netclient \
+	nativesdk-python-netserver \
+	nativesdk-python-distutils \
+	nativesdk-python-unixadmin \
+	nativesdk-python-compression \
+	nativesdk-python-json \
+	nativesdk-python-unittest \
+	nativesdk-python-mmap \
+	nativesdk-python-difflib \
+	nativesdk-python-pprint \
+	nativesdk-python-git \
+	nativesdk-python-pkgutil \
+	"
 TOOLCHAIN_TARGET_TASK = "packagegroup-${QTNAME}-toolchain-target"
 TOOLCHAIN_OUTPUTNAME = "${SDK_NAME}-toolchain-${QTNAME}-${DISTRO_VERSION}"
 
-- 
2.0.1

Re: [PATCH] Yocto: Install full set of python modules in Qt SDK toolchain

[Koen Kooi]

Since 'yocto' is a project like freedesktop.org your commit message doesn't make sense, it should be something like:

meta-toolchain-qt:  Install full set of python modules

Op 7 aug. 2014, om 21:10 heeft Marek Vasut <marex@denx.de> het volgende geschreven:

> The Qt SDK toolchain pulls in python via packagegroup-cross-canadian-${MACHINE}
> and ships it. But the python is missing many modules and is rather incomplete.
> 
> The environment-setup-* script configures the PATH variable to point into
> it's own sysroot first, it means the python from the SDK is used as well
> when the environment is configured. It actually does make sense to use the
> python from the SDK, since the SDK should provide the tools needed to build
> additional software.
> 
> The problem is, the python in the SDK image is stripped to a bare minimum
> of modules, which prevents some software from being built with the SDK as
> is. Add the same python modules as the buildtools-tarball.bb adds to make
> the python shipped with the SDK complete.
> 
> Signed-off-by: Marek Vasut <marex@denx.de>
> ---
> meta/recipes-qt/meta/meta-toolchain-qt.inc | 32 +++++++++++++++++++++++++++++-
> 1 file changed, 31 insertions(+), 1 deletion(-)
> 
> NOTE: This lets me build U-Boot git HEAD , which now has Kconfig in it
>      and that depends on the "subprocess" module. But instead of adding
>      just one module, I want to fix this properly and future-proof this.
> 
> diff --git a/meta/recipes-qt/meta/meta-toolchain-qt.inc b/meta/recipes-qt/meta/meta-toolchain-qt.inc
> index 6b162bd..99d5e64 100644
> --- a/meta/recipes-qt/meta/meta-toolchain-qt.inc
> +++ b/meta/recipes-qt/meta/meta-toolchain-qt.inc
> @@ -1,4 +1,34 @@
> -TOOLCHAIN_HOST_TASK = "nativesdk-packagegroup-${QTNAME}-toolchain-host packagegroup-cross-canadian-${MACHINE}"
> +TOOLCHAIN_HOST_TASK = " \
> +	nativesdk-packagegroup-${QTNAME}-toolchain-host \
> +	packagegroup-cross-canadian-${MACHINE} \
> +	nativesdk-python-core \
> +	nativesdk-python-textutils \
> +	nativesdk-python-sqlite3 \
> +	nativesdk-python-pickle \
> +	nativesdk-python-logging \
> +	nativesdk-python-elementtree \
> +	nativesdk-python-curses \
> +	nativesdk-python-compile \
> +	nativesdk-python-compiler \
> +	nativesdk-python-fcntl \
> +	nativesdk-python-shell \
> +	nativesdk-python-misc \
> +	nativesdk-python-multiprocessing \
> +	nativesdk-python-subprocess \
> +	nativesdk-python-xmlrpc \
> +	nativesdk-python-netclient \
> +	nativesdk-python-netserver \
> +	nativesdk-python-distutils \
> +	nativesdk-python-unixadmin \
> +	nativesdk-python-compression \
> +	nativesdk-python-json \
> +	nativesdk-python-unittest \
> +	nativesdk-python-mmap \
> +	nativesdk-python-difflib \
> +	nativesdk-python-pprint \
> +	nativesdk-python-git \
> +	nativesdk-python-pkgutil \
> +	"
> TOOLCHAIN_TARGET_TASK = "packagegroup-${QTNAME}-toolchain-target"
> TOOLCHAIN_OUTPUTNAME = "${SDK_NAME}-toolchain-${QTNAME}-${DISTRO_VERSION}"
> 
> -- 
> 2.0.1
> 
> -- 
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core
> 

Re: [PATCH] Yocto: Install full set of python modules in Qt SDK toolchain

[Marek Vasut]

On Friday, August 08, 2014 at 07:22:04 AM, Koen Kooi wrote:
> Since 'yocto' is a project like freedesktop.org your commit message doesn't
> make sense, it should be something like:
> 
> meta-toolchain-qt:  Install full set of python modules

Got it, Saul also told me so. I will wait a bit for some more feedback and 
repost with corrections, OK ? Is this approach I took here even correct please ?

Best regards,
Marek Vasut

Re: [PATCH] Yocto: Install full set of python modules in Qt SDK toolchain

[Richard Purdie]

On Fri, 2014-08-08 at 14:22 +0200, Marek Vasut wrote:
> On Friday, August 08, 2014 at 07:22:04 AM, Koen Kooi wrote:
> > Since 'yocto' is a project like freedesktop.org your commit message doesn't
> > make sense, it should be something like:
> > 
> > meta-toolchain-qt:  Install full set of python modules
> 
> Got it, Saul also told me so. I will wait a bit for some more feedback and 
> repost with corrections, OK ? Is this approach I took here even correct please ?

I'd really like to see a packagegroup defined somewhere with this list
of python modules which the various recipes can then reference...

Cheers,

Richard

Re: [PATCH] Yocto: Install full set of python modules in Qt SDK toolchain

[Marek Vasut]

On Monday, August 11, 2014 at 12:26:34 PM, Richard Purdie wrote:
> On Fri, 2014-08-08 at 14:22 +0200, Marek Vasut wrote:
> > On Friday, August 08, 2014 at 07:22:04 AM, Koen Kooi wrote:
> > > Since 'yocto' is a project like freedesktop.org your commit message
> > > doesn't make sense, it should be something like:
> > > 
> > > meta-toolchain-qt:  Install full set of python modules
> > 
> > Got it, Saul also told me so. I will wait a bit for some more feedback
> > and repost with corrections, OK ? Is this approach I took here even
> > correct please ?
> 
> I'd really like to see a packagegroup defined somewhere with this list
> of python modules which the various recipes can then reference...

I agree, that makes sense. Is this approach I took here the correct one please ?

Best regards,
Marek Vasut

Re: [PATCH] Yocto: Install full set of python modules in Qt SDK toolchain

[Laszlo Papp]

Come on, Yocto maintainers, please...

Mark sent this change relatively long ago, and it is still in "new"
state, sadly. The current SDK shipped _breaks_ any third-party
software that uses standard python with regards to the libraries and
all that.

This is is slightly frustrating. We also face the same issue. :(

On Mon, Aug 11, 2014 at 7:26 PM, Marek Vasut <marex@denx.de> wrote:
> On Monday, August 11, 2014 at 12:26:34 PM, Richard Purdie wrote:
>> On Fri, 2014-08-08 at 14:22 +0200, Marek Vasut wrote:
>> > On Friday, August 08, 2014 at 07:22:04 AM, Koen Kooi wrote:
>> > > Since 'yocto' is a project like freedesktop.org your commit message
>> > > doesn't make sense, it should be something like:
>> > >
>> > > meta-toolchain-qt:  Install full set of python modules
>> >
>> > Got it, Saul also told me so. I will wait a bit for some more feedback
>> > and repost with corrections, OK ? Is this approach I took here even
>> > correct please ?
>>
>> I'd really like to see a packagegroup defined somewhere with this list
>> of python modules which the various recipes can then reference...
>
> I agree, that makes sense. Is this approach I took here the correct one please ?
>
> Best regards,
> Marek Vasut
> --
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [PATCH] Yocto: Install full set of python modules in Qt SDK toolchain

[Laszlo Papp]

I meant to write "Marek". I apologize for the spelling issue.

On Thu, Sep 18, 2014 at 10:11 AM, Laszlo Papp <lpapp@kde.org> wrote:
> Come on, Yocto maintainers, please...
>
> Mark sent this change relatively long ago, and it is still in "new"
> state, sadly. The current SDK shipped _breaks_ any third-party
> software that uses standard python with regards to the libraries and
> all that.
>
> This is is slightly frustrating. We also face the same issue. :(
>
> On Mon, Aug 11, 2014 at 7:26 PM, Marek Vasut <marex@denx.de> wrote:
>> On Monday, August 11, 2014 at 12:26:34 PM, Richard Purdie wrote:
>>> On Fri, 2014-08-08 at 14:22 +0200, Marek Vasut wrote:
>>> > On Friday, August 08, 2014 at 07:22:04 AM, Koen Kooi wrote:
>>> > > Since 'yocto' is a project like freedesktop.org your commit message
>>> > > doesn't make sense, it should be something like:
>>> > >
>>> > > meta-toolchain-qt:  Install full set of python modules
>>> >
>>> > Got it, Saul also told me so. I will wait a bit for some more feedback
>>> > and repost with corrections, OK ? Is this approach I took here even
>>> > correct please ?
>>>
>>> I'd really like to see a packagegroup defined somewhere with this list
>>> of python modules which the various recipes can then reference...
>>
>> I agree, that makes sense. Is this approach I took here the correct one please ?
>>
>> Best regards,
>> Marek Vasut
>> --
>> _______________________________________________
>> Openembedded-core mailing list
>> Openembedded-core@lists.openembedded.org
>> http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [PATCH] Yocto: Install full set of python modules in Qt SDK toolchain

[Marek Vasut]

On Thursday, September 18, 2014 at 11:11:45 AM, Laszlo Papp wrote:
> Come on, Yocto maintainers, please...
> 
> Mark sent this change relatively long ago, and it is still in "new"
> state, sadly. The current SDK shipped _breaks_ any third-party
> software that uses standard python with regards to the libraries and
> all that.
> 
> This is is slightly frustrating. We also face the same issue. :(

Thanks for the reminder, new version which should fix the issues with the 
previous one is on the ML now. You're all on CC.

Best regards,
Marek Vasut

Re: [PATCH] Yocto: Install full set of python modules in Qt SDK toolchain

[Laszlo Papp]

On Thu, Sep 18, 2014 at 10:16 AM, Marek Vasut <marex@denx.de> wrote:
> On Thursday, September 18, 2014 at 11:11:45 AM, Laszlo Papp wrote:
>> Come on, Yocto maintainers, please...
>>
>> Mark sent this change relatively long ago, and it is still in "new"
>> state, sadly. The current SDK shipped _breaks_ any third-party
>> software that uses standard python with regards to the libraries and
>> all that.
>>
>> This is is slightly frustrating. We also face the same issue. :(
>
> Thanks for the reminder, new version which should fix the issues with the
> previous one is on the ML now. You're all on CC.

I do not think this is an explicit Qt issue, and hence fixing on that
layer sounds like a weird approach. It seems to be a generic python
issue and so, I think it should be addressed in its core. I opened a
new report for this:

https://bugzilla.yoctoproject.org/show_bug.cgi?id=6735

Re: [PATCH] Yocto: Install full set of python modules in Qt SDK toolchain

[Marek Vasut]

On Thursday, September 18, 2014 at 11:23:13 AM, Laszlo Papp wrote:
> On Thu, Sep 18, 2014 at 10:16 AM, Marek Vasut <marex@denx.de> wrote:
> > On Thursday, September 18, 2014 at 11:11:45 AM, Laszlo Papp wrote:
> >> Come on, Yocto maintainers, please...
> >> 
> >> Mark sent this change relatively long ago, and it is still in "new"
> >> state, sadly. The current SDK shipped _breaks_ any third-party
> >> software that uses standard python with regards to the libraries and
> >> all that.
> >> 
> >> This is is slightly frustrating. We also face the same issue. :(
> > 
> > Thanks for the reminder, new version which should fix the issues with the
> > previous one is on the ML now. You're all on CC.
> 
> I do not think this is an explicit Qt issue, and hence fixing on that
> layer sounds like a weird approach. It seems to be a generic python
> issue and so, I think it should be addressed in its core. I opened a
> new report for this:
> 
> https://bugzilla.yoctoproject.org/show_bug.cgi?id=6735

Then you probably want to add nativesdk-packagegroup-python class into the other 
toolchain recipes as well . Or even better, into the base SDK toolchain class 
(is there one?)

Best regards,
Marek Vasut

Re: [PATCH] Yocto: Install full set of python modules in Qt SDK toolchain

[Laszlo Papp]

On Thu, Sep 18, 2014 at 10:30 AM, Marek Vasut <marex@denx.de> wrote:
> On Thursday, September 18, 2014 at 11:23:13 AM, Laszlo Papp wrote:
>> On Thu, Sep 18, 2014 at 10:16 AM, Marek Vasut <marex@denx.de> wrote:
>> > On Thursday, September 18, 2014 at 11:11:45 AM, Laszlo Papp wrote:
>> >> Come on, Yocto maintainers, please...
>> >>
>> >> Mark sent this change relatively long ago, and it is still in "new"
>> >> state, sadly. The current SDK shipped _breaks_ any third-party
>> >> software that uses standard python with regards to the libraries and
>> >> all that.
>> >>
>> >> This is is slightly frustrating. We also face the same issue. :(
>> >
>> > Thanks for the reminder, new version which should fix the issues with the
>> > previous one is on the ML now. You're all on CC.
>>
>> I do not think this is an explicit Qt issue, and hence fixing on that
>> layer sounds like a weird approach. It seems to be a generic python
>> issue and so, I think it should be addressed in its core. I opened a
>> new report for this:
>>
>> https://bugzilla.yoctoproject.org/show_bug.cgi?id=6735
>
> Then you probably want to add nativesdk-packagegroup-python class into the other
> toolchain recipes as well . Or even better, into the base SDK toolchain class
> (is there one?)

See, I do not understand why this "feature" was integrated the way it
was. IMHO, it lacks any kind of reality. Has the SDK been ever tested
against python based host development systems? My assumption is no. I
do not think there is any sanity in using a _that_ stripped down
version of python on desktop. It just really hurts the python users
for the SDK, since there is no simple workaround and we cannot get the
SDK to let the system python take precedence. This situation is awful
in my opinion. Who is up for fixing this in the core? Let us have a
_standard_ python shipped with the SDK by default or do not ship any
at all.

Re: [PATCH] Yocto: Install full set of python modules in Qt SDK toolchain

[Richard Purdie]

On Thu, 2014-09-18 at 11:29 +0100, Laszlo Papp wrote:
> On Thu, Sep 18, 2014 at 10:30 AM, Marek Vasut <marex@denx.de> wrote:
> > On Thursday, September 18, 2014 at 11:23:13 AM, Laszlo Papp wrote:
> >> On Thu, Sep 18, 2014 at 10:16 AM, Marek Vasut <marex@denx.de> wrote:
> >> > On Thursday, September 18, 2014 at 11:11:45 AM, Laszlo Papp wrote:
> >> >> Come on, Yocto maintainers, please...
> >> >>
> >> >> Mark sent this change relatively long ago, and it is still in "new"
> >> >> state, sadly. The current SDK shipped _breaks_ any third-party
> >> >> software that uses standard python with regards to the libraries and
> >> >> all that.
> >> >>
> >> >> This is is slightly frustrating. We also face the same issue. :(
> >> >
> >> > Thanks for the reminder, new version which should fix the issues with the
> >> > previous one is on the ML now. You're all on CC.
> >>
> >> I do not think this is an explicit Qt issue, and hence fixing on that
> >> layer sounds like a weird approach. It seems to be a generic python
> >> issue and so, I think it should be addressed in its core. I opened a
> >> new report for this:
> >>
> >> https://bugzilla.yoctoproject.org/show_bug.cgi?id=6735
> >
> > Then you probably want to add nativesdk-packagegroup-python class into the other
> > toolchain recipes as well . Or even better, into the base SDK toolchain class
> > (is there one?)
> 
> See, I do not understand why this "feature" was integrated the way it
> was. IMHO, it lacks any kind of reality. Has the SDK been ever tested
> against python based host development systems? My assumption is no. I
> do not think there is any sanity in using a _that_ stripped down
> version of python on desktop. It just really hurts the python users
> for the SDK, since there is no simple workaround and we cannot get the
> SDK to let the system python take precedence. This situation is awful
> in my opinion. Who is up for fixing this in the core? Let us have a
> _standard_ python shipped with the SDK by default or do not ship any
> at all.

If you install nativesdk-python-modules into your SDK, you will get
python *and* all its modules installed. If you just install the python
core, you get a cut down python and need to install the modules you
need.

The python SDK has been tested and used in a number of scenarios. Where
a full set of modules is needed, nativesdk-python-modules gets installed
and everyone is happy. If there is some problem with
nativesdk-python-modules, please let us know.

Cheers,

Richard

Re: [PATCH] Yocto: Install full set of python modules in Qt SDK toolchain

[Marek Vasut]

On Thursday, September 18, 2014 at 03:36:40 PM, Richard Purdie wrote:
> On Thu, 2014-09-18 at 11:29 +0100, Laszlo Papp wrote:
> > On Thu, Sep 18, 2014 at 10:30 AM, Marek Vasut <marex@denx.de> wrote:
> > > On Thursday, September 18, 2014 at 11:23:13 AM, Laszlo Papp wrote:
> > >> On Thu, Sep 18, 2014 at 10:16 AM, Marek Vasut <marex@denx.de> wrote:
> > >> > On Thursday, September 18, 2014 at 11:11:45 AM, Laszlo Papp wrote:
> > >> >> Come on, Yocto maintainers, please...
> > >> >> 
> > >> >> Mark sent this change relatively long ago, and it is still in "new"
> > >> >> state, sadly. The current SDK shipped _breaks_ any third-party
> > >> >> software that uses standard python with regards to the libraries
> > >> >> and all that.
> > >> >> 
> > >> >> This is is slightly frustrating. We also face the same issue. :(
> > >> > 
> > >> > Thanks for the reminder, new version which should fix the issues
> > >> > with the previous one is on the ML now. You're all on CC.
> > >> 
> > >> I do not think this is an explicit Qt issue, and hence fixing on that
> > >> layer sounds like a weird approach. It seems to be a generic python
> > >> issue and so, I think it should be addressed in its core. I opened a
> > >> new report for this:
> > >> 
> > >> https://bugzilla.yoctoproject.org/show_bug.cgi?id=6735
> > > 
> > > Then you probably want to add nativesdk-packagegroup-python class into
> > > the other toolchain recipes as well . Or even better, into the base
> > > SDK toolchain class (is there one?)
> > 
> > See, I do not understand why this "feature" was integrated the way it
> > was. IMHO, it lacks any kind of reality. Has the SDK been ever tested
> > against python based host development systems? My assumption is no. I
> > do not think there is any sanity in using a _that_ stripped down
> > version of python on desktop. It just really hurts the python users
> > for the SDK, since there is no simple workaround and we cannot get the
> > SDK to let the system python take precedence. This situation is awful
> > in my opinion. Who is up for fixing this in the core? Let us have a
> > _standard_ python shipped with the SDK by default or do not ship any
> > at all.
> 
> If you install nativesdk-python-modules into your SDK, you will get
> python *and* all its modules installed. If you just install the python
> core, you get a cut down python and need to install the modules you
> need.
> 
> The python SDK has been tested and used in a number of scenarios. Where
> a full set of modules is needed, nativesdk-python-modules gets installed
> and everyone is happy. If there is some problem with
> nativesdk-python-modules, please let us know.

git grep nativesdk-python-modules doesn't show any matches in 
git://git.yoctoproject.org/poky master . Do you mean the nativesdk-packagegroup-
python I crafted or do you refer to something else please ?

Also, shouldn't full python be installed into all the SDK toolchains ? I am for 
example unable to compile U-Boot 2014.10rc with the Yocto SDK toolchain anymore. 
The SDK is missing python modules and I cannot easily override the usage of 
python from the SDK . So I agree with Laszlo here, the SDK toolchain is somewhat 
unusable as it is.

Best regards,
Marek Vasut

Re: [PATCH] Yocto: Install full set of python modules in Qt SDK toolchain

[Richard Purdie]

On Fri, 2014-09-19 at 09:14 +0200, Marek Vasut wrote:
> On Thursday, September 18, 2014 at 03:36:40 PM, Richard Purdie wrote:
> > On Thu, 2014-09-18 at 11:29 +0100, Laszlo Papp wrote:
> > > On Thu, Sep 18, 2014 at 10:30 AM, Marek Vasut <marex@denx.de> wrote:
> > > > On Thursday, September 18, 2014 at 11:23:13 AM, Laszlo Papp wrote:
> > > >> On Thu, Sep 18, 2014 at 10:16 AM, Marek Vasut <marex@denx.de> wrote:
> > > >> > On Thursday, September 18, 2014 at 11:11:45 AM, Laszlo Papp wrote:
> > > >> >> Come on, Yocto maintainers, please...
> > > >> >> 
> > > >> >> Mark sent this change relatively long ago, and it is still in "new"
> > > >> >> state, sadly. The current SDK shipped _breaks_ any third-party
> > > >> >> software that uses standard python with regards to the libraries
> > > >> >> and all that.
> > > >> >> 
> > > >> >> This is is slightly frustrating. We also face the same issue. :(
> > > >> > 
> > > >> > Thanks for the reminder, new version which should fix the issues
> > > >> > with the previous one is on the ML now. You're all on CC.
> > > >> 
> > > >> I do not think this is an explicit Qt issue, and hence fixing on that
> > > >> layer sounds like a weird approach. It seems to be a generic python
> > > >> issue and so, I think it should be addressed in its core. I opened a
> > > >> new report for this:
> > > >> 
> > > >> https://bugzilla.yoctoproject.org/show_bug.cgi?id=6735
> > > > 
> > > > Then you probably want to add nativesdk-packagegroup-python class into
> > > > the other toolchain recipes as well . Or even better, into the base
> > > > SDK toolchain class (is there one?)
> > > 
> > > See, I do not understand why this "feature" was integrated the way it
> > > was. IMHO, it lacks any kind of reality. Has the SDK been ever tested
> > > against python based host development systems? My assumption is no. I
> > > do not think there is any sanity in using a _that_ stripped down
> > > version of python on desktop. It just really hurts the python users
> > > for the SDK, since there is no simple workaround and we cannot get the
> > > SDK to let the system python take precedence. This situation is awful
> > > in my opinion. Who is up for fixing this in the core? Let us have a
> > > _standard_ python shipped with the SDK by default or do not ship any
> > > at all.
> > 
> > If you install nativesdk-python-modules into your SDK, you will get
> > python *and* all its modules installed. If you just install the python
> > core, you get a cut down python and need to install the modules you
> > need.
> > 
> > The python SDK has been tested and used in a number of scenarios. Where
> > a full set of modules is needed, nativesdk-python-modules gets installed
> > and everyone is happy. If there is some problem with
> > nativesdk-python-modules, please let us know.
> 
> git grep nativesdk-python-modules doesn't show any matches in 
> git://git.yoctoproject.org/poky master . Do you mean the nativesdk-packagegroup-
> python I crafted or do you refer to something else please ?
> 
> Also, shouldn't full python be installed into all the SDK toolchains ? I am for 
> example unable to compile U-Boot 2014.10rc with the Yocto SDK toolchain anymore. 
> The SDK is missing python modules and I cannot easily override the usage of 
> python from the SDK . So I agree with Laszlo here, the SDK toolchain is somewhat 
> unusable as it is.

I refer to the package I mentioned:

$ ls -la tmp/deploy/ipk/x86_64-nativesdk/nativesdk-python-modules*
-rw-r--r-- 2 richard richard 1512 Sep 18 17:09 tmp/deploy/ipk/x86_64-nativesdk/nativesdk-python-modules_2.7.3-r0.3.24_x86_64-nativesdk.ipk

See ${PN}-modules defined in python-2.7-manifest.inc.

I agreed there is a problem, I disagree somewhat about how it should be
fixed since you're just installing a set of modules which is defined as
those needed to run bitbake and I don't think this is what you actually
want. There is also the question of whether nativesdk-python should even
be in there...

Cheers,

Richard

Re: [PATCH] Yocto: Install full set of python modules in Qt SDK toolchain

[Marek Vasut]

On Friday, September 19, 2014 at 09:25:12 AM, Richard Purdie wrote:
[...]
> > git grep nativesdk-python-modules doesn't show any matches in
> > git://git.yoctoproject.org/poky master . Do you mean the
> > nativesdk-packagegroup- python I crafted or do you refer to something
> > else please ?
> > 
> > Also, shouldn't full python be installed into all the SDK toolchains ? I
> > am for example unable to compile U-Boot 2014.10rc with the Yocto SDK
> > toolchain anymore. The SDK is missing python modules and I cannot easily
> > override the usage of python from the SDK . So I agree with Laszlo here,
> > the SDK toolchain is somewhat unusable as it is.
> 
> I refer to the package I mentioned:
> 
> $ ls -la tmp/deploy/ipk/x86_64-nativesdk/nativesdk-python-modules*
> -rw-r--r-- 2 richard richard 1512 Sep 18 17:09
> tmp/deploy/ipk/x86_64-nativesdk/nativesdk-python-modules_2.7.3-r0.3.24_x86
> _64-nativesdk.ipk
> 
> See ${PN}-modules defined in python-2.7-manifest.inc.

Oh I see, sorry. Is there any way to look up packages with bitbake or Yocto 
please ?

> I agreed there is a problem, I disagree somewhat about how it should be
> fixed since you're just installing a set of modules which is defined as
> those needed to run bitbake and I don't think this is what you actually
> want. There is also the question of whether nativesdk-python should even
> be in there...

So would this patch (paste below) work please ? If so, I would submit proper.

diff --git a/meta/classes/populate_sdk_base.bbclass 
b/meta/classes/populate_sdk_base.bbclass
index 10d04ed..7245d53 100644
--- a/meta/classes/populate_sdk_base.bbclass
+++ b/meta/classes/populate_sdk_base.bbclass
@@ -30,7 +30,11 @@ B_task-populate-sdk = "${SDK_DIR}"
 
 SDKTARGETSYSROOT = "${SDKPATH}/sysroots/${REAL_MULTIMACH_TARGET_SYS}"
 
-TOOLCHAIN_HOST_TASK ?= "nativesdk-packagegroup-sdk-host packagegroup-cross-
canadian-${MACHINE}"
+TOOLCHAIN_HOST_TASK ?= "                               \
+       nativesdk-packagegroup-sdk-host                 \
+       packagegroup-cross-canadian-${MACHINE}          \
+       nativesdk-python-modules                        \
+       "
 TOOLCHAIN_HOST_TASK_ATTEMPTONLY ?= ""
 TOOLCHAIN_TARGET_TASK ?= "packagegroup-core-standalone-sdk-target packagegroup-
core-standalone-sdk-target-dbg"
 TOOLCHAIN_TARGET_TASK_ATTEMPTONLY ?= ""
diff --git a/meta/recipes-qt/meta/meta-toolchain-qt.inc b/meta/recipes-
qt/meta/meta-toolchain-qt.inc
index 6b162bd..473a123 100644
--- a/meta/recipes-qt/meta/meta-toolchain-qt.inc
+++ b/meta/recipes-qt/meta/meta-toolchain-qt.inc
@@ -1,4 +1,8 @@
-TOOLCHAIN_HOST_TASK = "nativesdk-packagegroup-${QTNAME}-toolchain-host 
packagegroup-cross-canadian-${MACHINE}"
+TOOLCHAIN_HOST_TASK = "                                                \
+       nativesdk-packagegroup-${QTNAME}-toolchain-host         \
+       packagegroup-cross-canadian-${MACHINE}                  \
+       nativesdk-python-modules                                \
+       "
 TOOLCHAIN_TARGET_TASK = "packagegroup-${QTNAME}-toolchain-target"
 TOOLCHAIN_OUTPUTNAME = "${SDK_NAME}-toolchain-${QTNAME}-${DISTRO_VERSION}"


Best regards,
Marek Vasut

Re: [PATCH] Yocto: Install full set of python modules in Qt SDK toolchain

[Laszlo Papp]

What is the situation on this front?

On Sun, Sep 21, 2014 at 5:02 PM, Marek Vasut <marex@denx.de> wrote:
> On Friday, September 19, 2014 at 09:25:12 AM, Richard Purdie wrote:
> [...]
>> > git grep nativesdk-python-modules doesn't show any matches in
>> > git://git.yoctoproject.org/poky master . Do you mean the
>> > nativesdk-packagegroup- python I crafted or do you refer to something
>> > else please ?
>> >
>> > Also, shouldn't full python be installed into all the SDK toolchains ? I
>> > am for example unable to compile U-Boot 2014.10rc with the Yocto SDK
>> > toolchain anymore. The SDK is missing python modules and I cannot easily
>> > override the usage of python from the SDK . So I agree with Laszlo here,
>> > the SDK toolchain is somewhat unusable as it is.
>>
>> I refer to the package I mentioned:
>>
>> $ ls -la tmp/deploy/ipk/x86_64-nativesdk/nativesdk-python-modules*
>> -rw-r--r-- 2 richard richard 1512 Sep 18 17:09
>> tmp/deploy/ipk/x86_64-nativesdk/nativesdk-python-modules_2.7.3-r0.3.24_x86
>> _64-nativesdk.ipk
>>
>> See ${PN}-modules defined in python-2.7-manifest.inc.
>
> Oh I see, sorry. Is there any way to look up packages with bitbake or Yocto
> please ?
>
>> I agreed there is a problem, I disagree somewhat about how it should be
>> fixed since you're just installing a set of modules which is defined as
>> those needed to run bitbake and I don't think this is what you actually
>> want. There is also the question of whether nativesdk-python should even
>> be in there...
>
> So would this patch (paste below) work please ? If so, I would submit proper.
>
> diff --git a/meta/classes/populate_sdk_base.bbclass
> b/meta/classes/populate_sdk_base.bbclass
> index 10d04ed..7245d53 100644
> --- a/meta/classes/populate_sdk_base.bbclass
> +++ b/meta/classes/populate_sdk_base.bbclass
> @@ -30,7 +30,11 @@ B_task-populate-sdk = "${SDK_DIR}"
>
>  SDKTARGETSYSROOT = "${SDKPATH}/sysroots/${REAL_MULTIMACH_TARGET_SYS}"
>
> -TOOLCHAIN_HOST_TASK ?= "nativesdk-packagegroup-sdk-host packagegroup-cross-
> canadian-${MACHINE}"
> +TOOLCHAIN_HOST_TASK ?= "                               \
> +       nativesdk-packagegroup-sdk-host                 \
> +       packagegroup-cross-canadian-${MACHINE}          \
> +       nativesdk-python-modules                        \
> +       "
>  TOOLCHAIN_HOST_TASK_ATTEMPTONLY ?= ""
>  TOOLCHAIN_TARGET_TASK ?= "packagegroup-core-standalone-sdk-target packagegroup-
> core-standalone-sdk-target-dbg"
>  TOOLCHAIN_TARGET_TASK_ATTEMPTONLY ?= ""
> diff --git a/meta/recipes-qt/meta/meta-toolchain-qt.inc b/meta/recipes-
> qt/meta/meta-toolchain-qt.inc
> index 6b162bd..473a123 100644
> --- a/meta/recipes-qt/meta/meta-toolchain-qt.inc
> +++ b/meta/recipes-qt/meta/meta-toolchain-qt.inc
> @@ -1,4 +1,8 @@
> -TOOLCHAIN_HOST_TASK = "nativesdk-packagegroup-${QTNAME}-toolchain-host
> packagegroup-cross-canadian-${MACHINE}"
> +TOOLCHAIN_HOST_TASK = "                                                \
> +       nativesdk-packagegroup-${QTNAME}-toolchain-host         \
> +       packagegroup-cross-canadian-${MACHINE}                  \
> +       nativesdk-python-modules                                \
> +       "
>  TOOLCHAIN_TARGET_TASK = "packagegroup-${QTNAME}-toolchain-target"
>  TOOLCHAIN_OUTPUTNAME = "${SDK_NAME}-toolchain-${QTNAME}-${DISTRO_VERSION}"
>
>
> Best regards,
> Marek Vasut

Re: [PATCH] Yocto: Install full set of python modules in Qt SDK toolchain

[Burton, Ross]

[-- Attachment #1: Type: text/plain, Size: 1405 bytes --]

On 21 September 2014 at 17:02, Marek Vasut <marex@denx.de> wrote:

> -TOOLCHAIN_HOST_TASK ?= "nativesdk-packagegroup-sdk-host
> packagegroup-cross-
> canadian-${MACHINE}"
> +TOOLCHAIN_HOST_TASK ?= "                               \
> +       nativesdk-packagegroup-sdk-host                 \
> +       packagegroup-cross-canadian-${MACHINE}          \
> +       nativesdk-python-modules
>

Thanks to Laszlo for pinging this.  We fixed a similar problem in the
buildtools tarball by pulling in python-modules but the situation was
different there - the buildtools tarball always contained some of Python so
it's logical to make it pull in all of python.

It's nativesdk-packagegroup-sdk-host that's pulling in parts of Python via
it's dependency on smartpm.  This makes me think we need two changes here:

1) The toolchain should contain the packaging tools for the selected
packaging format of the images, not just smartpm.  So a SDK for a
opkg-based image should be shipping opkg, not smartpm.

2) Toolchains should either ship no Python or all Python, because dropping
a partial Python into $PATH breaks user's expectations (the same argument
that was used for the buildtools).  Not sure how to do this though, maybe
the construction should inspect the installed package list and if anything
Python was installed, ensure python-modules is also installed.

Comments?

Ross

[-- Attachment #2: Type: text/html, Size: 2112 bytes --]

Re: [PATCH] Yocto: Install full set of python modules in Qt SDK toolchain

[Richard Purdie]

On Mon, 2014-12-01 at 15:05 +0000, Burton, Ross wrote:
> On 21 September 2014 at 17:02, Marek Vasut <marex@denx.de> wrote:
>         -TOOLCHAIN_HOST_TASK ?= "nativesdk-packagegroup-sdk-host
>         packagegroup-cross-
>         canadian-${MACHINE}"
>         +TOOLCHAIN_HOST_TASK ?= "                               \
>         +       nativesdk-packagegroup-sdk-host                 \
>         +       packagegroup-cross-canadian-${MACHINE}          \
>         +       nativesdk-python-modules       
> 
> Thanks to Laszlo for pinging this.  We fixed a similar problem in the
> buildtools tarball by pulling in python-modules but the situation was
> different there - the buildtools tarball always contained some of
> Python so it's logical to make it pull in all of python.
>
> It's nativesdk-packagegroup-sdk-host that's pulling in parts of Python
> via it's dependency on smartpm.  This makes me think we need two
> changes here:
>
> 1) The toolchain should contain the packaging tools for the selected
> packaging format of the images, not just smartpm.  So a SDK for a
> opkg-based image should be shipping opkg, not smartpm.

Agreed on smartpm, rpm is a bit of a different story due to where it
gets used in the packaging process. As the SDK and build systems
converge this gets a bit fuzzy :/.
> 
> 2) Toolchains should either ship no Python or all Python, because
> dropping a partial Python into $PATH breaks user's expectations (the
> same argument that was used for the buildtools).  Not sure how to do
> this though, maybe the construction should inspect the installed
> package list and if anything Python was installed, ensure
> python-modules is also installed.
>
> Comments?
> 
Make nativesdk-python-core RRECOMMEND python-modules?

Cheers,

Richard

Re: [PATCH] Yocto: Install full set of python modules in Qt SDK toolchain

[Burton, Ross]

[-- Attachment #1: Type: text/plain, Size: 208 bytes --]

On 1 December 2014 at 15:13, Richard Purdie <
richard.purdie@linuxfoundation.org> wrote:

> Make nativesdk-python-core RRECOMMEND python-modules?
>

Makes sense, and is clean.  Patch incoming.

Ross

[-- Attachment #2: Type: text/html, Size: 627 bytes --]

Re: [PATCH] Yocto: Install full set of python modules in Qt SDK toolchain

[Paul Eggleton]

On Monday 01 December 2014 15:13:30 Richard Purdie wrote:
> On Mon, 2014-12-01 at 15:05 +0000, Burton, Ross wrote:
> > On 21 September 2014 at 17:02, Marek Vasut <marex@denx.de> wrote:
> >         -TOOLCHAIN_HOST_TASK ?= "nativesdk-packagegroup-sdk-host
> >         packagegroup-cross-
> >         canadian-${MACHINE}"
> >         +TOOLCHAIN_HOST_TASK ?= "                               \
> >         +       nativesdk-packagegroup-sdk-host                 \
> >         +       packagegroup-cross-canadian-${MACHINE}          \
> >         +       nativesdk-python-modules
> > 
> > Thanks to Laszlo for pinging this.  We fixed a similar problem in the
> > buildtools tarball by pulling in python-modules but the situation was
> > different there - the buildtools tarball always contained some of
> > Python so it's logical to make it pull in all of python.
> > 
> > It's nativesdk-packagegroup-sdk-host that's pulling in parts of Python
> > via it's dependency on smartpm.  This makes me think we need two
> > changes here:
> > 
> > 1) The toolchain should contain the packaging tools for the selected
> > packaging format of the images, not just smartpm.  So a SDK for a
> > opkg-based image should be shipping opkg, not smartpm.
> 
> Agreed on smartpm, rpm is a bit of a different story due to where it
> gets used in the packaging process. As the SDK and build systems
> converge this gets a bit fuzzy :/.

I'm not entirely sure python-smartpm should have been added by default in the 
first place. I'd say having it there ought to be the choice of the person 
creating the SDK, just as with a lot of other tools one might want in the SDK. 
For most people I suspect it's superfluous.

> > 2) Toolchains should either ship no Python or all Python, because
> > dropping a partial Python into $PATH breaks user's expectations (the
> > same argument that was used for the buildtools).  Not sure how to do
> > this though, maybe the construction should inspect the installed
> > package list and if anything Python was installed, ensure
> > python-modules is also installed.
> > 
> > Comments?
> 
> Make nativesdk-python-core RRECOMMEND python-modules?

This sounds like a reasonable option to me. The split as it is at the moment 
exists mainly for trimming the target; the SDK is perhaps less sensitive to 
size constraints.

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre

Re: [PATCH] Yocto: Install full set of python modules in Qt SDK toolchain

[Burton, Ross]

[-- Attachment #1: Type: text/plain, Size: 356 bytes --]

On 1 December 2014 at 15:21, Paul Eggleton <paul.eggleton@linux.intel.com>
wrote:

> > Make nativesdk-python-core RRECOMMEND python-modules?
>
> This sounds like a reasonable option to me. The split as it is at the
> moment
> exists mainly for trimming the target; the SDK is perhaps less sensitive to
> size constraints.


Patch sent.

Ross

[-- Attachment #2: Type: text/html, Size: 742 bytes --]

Re: [PATCH] Yocto: Install full set of python modules in Qt SDK toolchain

[Laszlo Papp]

On Mon, Dec 1, 2014 at 4:44 PM, Burton, Ross <ross.burton@intel.com> wrote:
>
> On 1 December 2014 at 15:21, Paul Eggleton <paul.eggleton@linux.intel.com>
> wrote:
>>
>> > Make nativesdk-python-core RRECOMMEND python-modules?
>>
>> This sounds like a reasonable option to me. The split as it is at the
>> moment
>> exists mainly for trimming the target; the SDK is perhaps less sensitive
>> to
>> size constraints.
>
>
> Patch sent.

Thanks!

>
> Ross
>
> --
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core
>