OE-Core and Bitbake wrapper changes (min 2.7.3 python version)

OE-Core and Bitbake wrapper changes (min 2.7.3 python version)

[Richard Purdie]

Its not secret that I hate the current bitbake wrapper script and want
to remove it for 101 different reasons.

I now have code which removes the need for the double execution of
bitbake which was the only fundamental reason we had it. The question
therefore remains, what to do with the other pieces of the wrapper,
specifically the tar and git versions checks.

As a reminder for those who don't remember the problem here, the git
version is checked since we use certain parameters in the git fetcher
which need certain versions of git and git is in ASSUME_PROVIDED these
days. Its possible to trigger git operations at part time to resolve
revisions. tar is even more ugly since the wrong version has issues
extracting sstate archives. These issues mean injecting building them
into the dependency chain at the right point is hard.

Personally, I think we carry around a bit too much legacy these days and
its starting to hurt us. I would therefore like to propose that we take
this opportunity to do some spring cleaning and simply error on:

* broken tar versions
* too old versions of git
* python < 2.7.3

The python version check would move to the oe-init-build-env script, the
git/tar versions to sanity.bbclass.

The recommendation for anyone with these older versions would be to
install our standalone tools tarball which would have python 2.7.3 and
working versions of tar/git.

The reason for the python version change is so we can embrace the
unittest improvements in 2.7 and drop all of the workarounds for pre
2.7.3 bugs in bitbake. This starts to move us towards python 3, if this
tarball works well, we'd use the same approach to move to python 3.

Any objections?

Cheers,

Richard

Re: OE-Core and Bitbake wrapper changes (min 2.7.3 python version)

[Otavio Salvador]

[-- Attachment #1: Type: text/plain, Size: 2327 bytes --]

On Fri, Jun 7, 2013 at 7:47 AM, Richard Purdie <
richard.purdie@linuxfoundation.org> wrote:

> Its not secret that I hate the current bitbake wrapper script and want
> to remove it for 101 different reasons.
>
> I now have code which removes the need for the double execution of
> bitbake which was the only fundamental reason we had it. The question
> therefore remains, what to do with the other pieces of the wrapper,
> specifically the tar and git versions checks.
>
> As a reminder for those who don't remember the problem here, the git
> version is checked since we use certain parameters in the git fetcher
> which need certain versions of git and git is in ASSUME_PROVIDED these
> days. Its possible to trigger git operations at part time to resolve
> revisions. tar is even more ugly since the wrong version has issues
> extracting sstate archives. These issues mean injecting building them
> into the dependency chain at the right point is hard.
>
> Personally, I think we carry around a bit too much legacy these days and
> its starting to hurt us. I would therefore like to propose that we take
> this opportunity to do some spring cleaning and simply error on:
>
> * broken tar versions
> * too old versions of git
> * python < 2.7.3
>
> The python version check would move to the oe-init-build-env script, the
> git/tar versions to sanity.bbclass.
>
> The recommendation for anyone with these older versions would be to
> install our standalone tools tarball which would have python 2.7.3 and
> working versions of tar/git.
>
> The reason for the python version change is so we can embrace the
> unittest improvements in 2.7 and drop all of the workarounds for pre
> 2.7.3 bugs in bitbake. This starts to move us towards python 3, if this
> tarball works well, we'd use the same approach to move to python 3.
>
> Any objections?
>

I fully agree in error on these situations and providing a workaround
solution for old host systems seems perfect. This would clean a good amount
of legacy code and I also agree it is the way to go. So count me as a
supporter for it :)

-- 
Otavio Salvador                             O.S. Systems
http://www.ossystems.com.br        http://projetos.ossystems.com.br
Mobile: +55 (53) 9981-7854            Mobile: +1 (347) 903-9750

[-- Attachment #2: Type: text/html, Size: 2902 bytes --]

Re: OE-Core and Bitbake wrapper changes (min 2.7.3 python version)

[Chris Larson]

[-- Attachment #1: Type: text/plain, Size: 2333 bytes --]

On Fri, Jun 7, 2013 at 5:40 AM, Otavio Salvador <otavio@ossystems.com.br>wrote:

>
>
>
> On Fri, Jun 7, 2013 at 7:47 AM, Richard Purdie <
> richard.purdie@linuxfoundation.org> wrote:
>
>> Its not secret that I hate the current bitbake wrapper script and want
>> to remove it for 101 different reasons.
>>
>> I now have code which removes the need for the double execution of
>> bitbake which was the only fundamental reason we had it. The question
>> therefore remains, what to do with the other pieces of the wrapper,
>> specifically the tar and git versions checks.
>>
>> As a reminder for those who don't remember the problem here, the git
>> version is checked since we use certain parameters in the git fetcher
>> which need certain versions of git and git is in ASSUME_PROVIDED these
>> days. Its possible to trigger git operations at part time to resolve
>> revisions. tar is even more ugly since the wrong version has issues
>> extracting sstate archives. These issues mean injecting building them
>> into the dependency chain at the right point is hard.
>>
>> Personally, I think we carry around a bit too much legacy these days and
>> its starting to hurt us. I would therefore like to propose that we take
>> this opportunity to do some spring cleaning and simply error on:
>>
>> * broken tar versions
>> * too old versions of git
>> * python < 2.7.3
>>
>> The python version check would move to the oe-init-build-env script, the
>> git/tar versions to sanity.bbclass.
>>
>> The recommendation for anyone with these older versions would be to
>> install our standalone tools tarball which would have python 2.7.3 and
>> working versions of tar/git.
>>
>> The reason for the python version change is so we can embrace the
>> unittest improvements in 2.7 and drop all of the workarounds for pre
>> 2.7.3 bugs in bitbake. This starts to move us towards python 3, if this
>> tarball works well, we'd use the same approach to move to python 3.
>>
>> Any objections?
>>
>
> I fully agree in error on these situations and providing a workaround
> solution for old host systems seems perfect. This would clean a good amount
> of legacy code and I also agree it is the way to go. So count me as a
> supporter for it :)
>

Agreed, I'm in favor as well.
-- 
Christopher Larson

[-- Attachment #2: Type: text/html, Size: 3074 bytes --]

Re: OE-Core and Bitbake wrapper changes (min 2.7.3 python version)

[Mark Hatle]

On 6/7/13 5:47 AM, Richard Purdie wrote:
> Its not secret that I hate the current bitbake wrapper script and want
> to remove it for 101 different reasons.
>
> I now have code which removes the need for the double execution of
> bitbake which was the only fundamental reason we had it. The question
> therefore remains, what to do with the other pieces of the wrapper,
> specifically the tar and git versions checks.
>
> As a reminder for those who don't remember the problem here, the git
> version is checked since we use certain parameters in the git fetcher
> which need certain versions of git and git is in ASSUME_PROVIDED these
> days. Its possible to trigger git operations at part time to resolve
> revisions. tar is even more ugly since the wrong version has issues
> extracting sstate archives. These issues mean injecting building them
> into the dependency chain at the right point is hard.
>
> Personally, I think we carry around a bit too much legacy these days and
> its starting to hurt us. I would therefore like to propose that we take
> this opportunity to do some spring cleaning and simply error on:
>
> * broken tar versions
> * too old versions of git
> * python < 2.7.3
>
> The python version check would move to the oe-init-build-env script, the
> git/tar versions to sanity.bbclass.

Can we also add the python check to bitbake as well?  My concern is not everyone 
uses the oe-init-build-env script, so ensuring that bitbake stops immediately 
and tells the user what's wrong is important.

> The recommendation for anyone with these older versions would be to
> install our standalone tools tarball which would have python 2.7.3 and
> working versions of tar/git.
>
> The reason for the python version change is so we can embrace the
> unittest improvements in 2.7 and drop all of the workarounds for pre
> 2.7.3 bugs in bitbake. This starts to move us towards python 3, if this
> tarball works well, we'd use the same approach to move to python 3.
>
> Any objections?

No objection, this seems fine.  It would be nice if there was a simple way (for 
the tar and git cases) to be able to build them as a user step and then be able 
to use them, but that "nice experience" can easily be handled by documentation 
as well.

--Mark

> Cheers,
>
> Richard
>
>
>
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core
>

Re: OE-Core and Bitbake wrapper changes (min 2.7.3 python version)

[Richard Purdie]

On Fri, 2013-06-07 at 10:06 -0500, Mark Hatle wrote:
> On 6/7/13 5:47 AM, Richard Purdie wrote:
> > Its not secret that I hate the current bitbake wrapper script and want
> > to remove it for 101 different reasons.
> >
> > I now have code which removes the need for the double execution of
> > bitbake which was the only fundamental reason we had it. The question
> > therefore remains, what to do with the other pieces of the wrapper,
> > specifically the tar and git versions checks.
> >
> > As a reminder for those who don't remember the problem here, the git
> > version is checked since we use certain parameters in the git fetcher
> > which need certain versions of git and git is in ASSUME_PROVIDED these
> > days. Its possible to trigger git operations at part time to resolve
> > revisions. tar is even more ugly since the wrong version has issues
> > extracting sstate archives. These issues mean injecting building them
> > into the dependency chain at the right point is hard.
> >
> > Personally, I think we carry around a bit too much legacy these days and
> > its starting to hurt us. I would therefore like to propose that we take
> > this opportunity to do some spring cleaning and simply error on:
> >
> > * broken tar versions
> > * too old versions of git
> > * python < 2.7.3
> >
> > The python version check would move to the oe-init-build-env script, the
> > git/tar versions to sanity.bbclass.
> 
> Can we also add the python check to bitbake as well?  My concern is not everyone 
> uses the oe-init-build-env script, so ensuring that bitbake stops immediately 
> and tells the user what's wrong is important.

We do have checks in there and these will move to the new versions. The
issue we've had before is that you get a syntax error from python trying
to *parse* bin/bitbake. We obviously try and avoid that but it can slip
in for older python versions.

> > The recommendation for anyone with these older versions would be to
> > install our standalone tools tarball which would have python 2.7.3 and
> > working versions of tar/git.
> >
> > The reason for the python version change is so we can embrace the
> > unittest improvements in 2.7 and drop all of the workarounds for pre
> > 2.7.3 bugs in bitbake. This starts to move us towards python 3, if this
> > tarball works well, we'd use the same approach to move to python 3.
> >
> > Any objections?
> 
> No objection, this seems fine.  It would be nice if there was a simple way (for 
> the tar and git cases) to be able to build them as a user step and then be able 
> to use them, but that "nice experience" can easily be handled by documentation 
> as well.

"bitbake tar-native-replacement git-native-replacement" will still be
available as things stand but you'd have to put something into the
sanity check to look at the recipes being targeted and skip the sanity
tests. Its complexity I'd prefer not to have and will suck from a
usability perspective since the user would have to remember to do this
each time. The question "can't bitbake do this itself" then comes back
but you really do need a wrapper as there is too much version specific
knowledge. So in summary, I don't want to go here for the rapidly
reducing number of users this affects.

Cheers,

Richard

Re: OE-Core and Bitbake wrapper changes (min 2.7.3 python version)

[Mark Hatle]

On 6/7/13 10:12 AM, Richard Purdie wrote:
> On Fri, 2013-06-07 at 10:06 -0500, Mark Hatle wrote:
>> On 6/7/13 5:47 AM, Richard Purdie wrote:
>>> Its not secret that I hate the current bitbake wrapper script and want
>>> to remove it for 101 different reasons.
>>>
>>> I now have code which removes the need for the double execution of
>>> bitbake which was the only fundamental reason we had it. The question
>>> therefore remains, what to do with the other pieces of the wrapper,
>>> specifically the tar and git versions checks.
>>>
>>> As a reminder for those who don't remember the problem here, the git
>>> version is checked since we use certain parameters in the git fetcher
>>> which need certain versions of git and git is in ASSUME_PROVIDED these
>>> days. Its possible to trigger git operations at part time to resolve
>>> revisions. tar is even more ugly since the wrong version has issues
>>> extracting sstate archives. These issues mean injecting building them
>>> into the dependency chain at the right point is hard.
>>>
>>> Personally, I think we carry around a bit too much legacy these days and
>>> its starting to hurt us. I would therefore like to propose that we take
>>> this opportunity to do some spring cleaning and simply error on:
>>>
>>> * broken tar versions
>>> * too old versions of git
>>> * python < 2.7.3
>>>
>>> The python version check would move to the oe-init-build-env script, the
>>> git/tar versions to sanity.bbclass.
>>
>> Can we also add the python check to bitbake as well?  My concern is not everyone
>> uses the oe-init-build-env script, so ensuring that bitbake stops immediately
>> and tells the user what's wrong is important.
>
> We do have checks in there and these will move to the new versions. The
> issue we've had before is that you get a syntax error from python trying
> to *parse* bin/bitbake. We obviously try and avoid that but it can slip
> in for older python versions.
>
>>> The recommendation for anyone with these older versions would be to
>>> install our standalone tools tarball which would have python 2.7.3 and
>>> working versions of tar/git.
>>>
>>> The reason for the python version change is so we can embrace the
>>> unittest improvements in 2.7 and drop all of the workarounds for pre
>>> 2.7.3 bugs in bitbake. This starts to move us towards python 3, if this
>>> tarball works well, we'd use the same approach to move to python 3.
>>>
>>> Any objections?
>>
>> No objection, this seems fine.  It would be nice if there was a simple way (for
>> the tar and git cases) to be able to build them as a user step and then be able
>> to use them, but that "nice experience" can easily be handled by documentation
>> as well.
>
> "bitbake tar-native-replacement git-native-replacement" will still be
> available as things stand but you'd have to put something into the
> sanity check to look at the recipes being targeted and skip the sanity
> tests. Its complexity I'd prefer not to have and will suck from a
> usability perspective since the user would have to remember to do this
> each time. The question "can't bitbake do this itself" then comes back
> but you really do need a wrapper as there is too much version specific
> knowledge. So in summary, I don't want to go here for the rapidly
> reducing number of users this affects.

Sorry, I wasn't clear.  I meant that when it fails, it points the user at 
documentation that explains how they can build git/tar, etc..  Adding the code 
to do the build automatically should not be needed.

> Cheers,
>
> Richard
>
>
>
>
>

Re: OE-Core and Bitbake wrapper changes (min 2.7.3 python version)

[Richard Purdie]

On Fri, 2013-06-07 at 10:26 -0500, Mark Hatle wrote:
> On 6/7/13 10:12 AM, Richard Purdie wrote:
> > On Fri, 2013-06-07 at 10:06 -0500, Mark Hatle wrote:
> >> On 6/7/13 5:47 AM, Richard Purdie wrote:
> >>> Its not secret that I hate the current bitbake wrapper script and want
> >>> to remove it for 101 different reasons.
> >>>
> >>> I now have code which removes the need for the double execution of
> >>> bitbake which was the only fundamental reason we had it. The question
> >>> therefore remains, what to do with the other pieces of the wrapper,
> >>> specifically the tar and git versions checks.
> >>>
> >>> As a reminder for those who don't remember the problem here, the git
> >>> version is checked since we use certain parameters in the git fetcher
> >>> which need certain versions of git and git is in ASSUME_PROVIDED these
> >>> days. Its possible to trigger git operations at part time to resolve
> >>> revisions. tar is even more ugly since the wrong version has issues
> >>> extracting sstate archives. These issues mean injecting building them
> >>> into the dependency chain at the right point is hard.
> >>>
> >>> Personally, I think we carry around a bit too much legacy these days and
> >>> its starting to hurt us. I would therefore like to propose that we take
> >>> this opportunity to do some spring cleaning and simply error on:
> >>>
> >>> * broken tar versions
> >>> * too old versions of git
> >>> * python < 2.7.3
> >>>
> >>> The python version check would move to the oe-init-build-env script, the
> >>> git/tar versions to sanity.bbclass.
> >>
> >> Can we also add the python check to bitbake as well?  My concern is not everyone
> >> uses the oe-init-build-env script, so ensuring that bitbake stops immediately
> >> and tells the user what's wrong is important.
> >
> > We do have checks in there and these will move to the new versions. The
> > issue we've had before is that you get a syntax error from python trying
> > to *parse* bin/bitbake. We obviously try and avoid that but it can slip
> > in for older python versions.
> >
> >>> The recommendation for anyone with these older versions would be to
> >>> install our standalone tools tarball which would have python 2.7.3 and
> >>> working versions of tar/git.
> >>>
> >>> The reason for the python version change is so we can embrace the
> >>> unittest improvements in 2.7 and drop all of the workarounds for pre
> >>> 2.7.3 bugs in bitbake. This starts to move us towards python 3, if this
> >>> tarball works well, we'd use the same approach to move to python 3.
> >>>
> >>> Any objections?
> >>
> >> No objection, this seems fine.  It would be nice if there was a simple way (for
> >> the tar and git cases) to be able to build them as a user step and then be able
> >> to use them, but that "nice experience" can easily be handled by documentation
> >> as well.
> >
> > "bitbake tar-native-replacement git-native-replacement" will still be
> > available as things stand but you'd have to put something into the
> > sanity check to look at the recipes being targeted and skip the sanity
> > tests. Its complexity I'd prefer not to have and will suck from a
> > usability perspective since the user would have to remember to do this
> > each time. The question "can't bitbake do this itself" then comes back
> > but you really do need a wrapper as there is too much version specific
> > knowledge. So in summary, I don't want to go here for the rapidly
> > reducing number of users this affects.
> 
> Sorry, I wasn't clear.  I meant that when it fails, it points the user at 
> documentation that explains how they can build git/tar, etc..  Adding the code 
> to do the build automatically should not be needed.

I was thinking of just pointing them at a link to the manual which in
turn would link to a tarball they could download and install...

Cheers,

Richard