Image history (testlab)

Image history (testlab)

[Paul Eggleton]

Hi Koen / Beth / all,

Following on from my work extending packagehistory (which is still under 
review), I'm looking at tracking history of the content of images so we can 
get warnings about regressions. Thanks to Koen we've had this capability via 
testlab.bbclass in OE for some time now, and from there into meta-oe, but what 
I'd like to do now is bring this into OE-core and extend it so that it 
supports rpm (or even better, all of the package backends we support.) This 
could involve abstracting the package backend functionality out into 
rootfs_*.bbclass.

It also looks like testlab.bbclass in meta-oe has some license dumping 
functionality. Beth, does that mesh with what you've been working on / 
preparing for Yocto 1.2?

Thoughts?

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre

Re: Image history (testlab)

[Koen Kooi]

[-- Attachment #1: Type: text/plain, Size: 2270 bytes --]


Op 25 nov. 2011, om 12:44 heeft Paul Eggleton het volgende geschreven:

> Hi Koen / Beth / all,
> 
> Following on from my work extending packagehistory (which is still under 
> review), I'm looking at tracking history of the content of images so we can 
> get warnings about regressions. Thanks to Koen we've had this capability via 
> testlab.bbclass in OE for some time now, and from there into meta-oe, but what 
> I'd like to do now is bring this into OE-core and extend it so that it 
> supports rpm (or even better, all of the package backends we support.) This 
> could involve abstracting the package backend functionality out into 
> rootfs_*.bbclass.
> 
> It also looks like testlab.bbclass in meta-oe has some license dumping 
> functionality. Beth, does that mesh with what you've been working on / 
> preparing for Yocto 1.2?
> 
> Thoughts?

Don't forget its integration into remote git: http://git.angstrom-distribution.org/cgi-bin/cgit.cgi/testlab/?h=yocto 
I must admit that it completely sucks when you have >1 builder, but that is fixable. There are a number of other things we need to store somewhere (e.g. package history, PR config, etc), so coming up with a plan to store all that in a single repo would be good.

The are a few issues with its current output:

* License info uses the filename instead of the package name (f'oo_1.2.3_arch.ipk' vs 'foo') which leads to too much changes with small version bumps
* Dependencies are not sorted intelligently, so sometimes you get diffs which don't contain any actual change, just ordering junk,
* It only reports oe-core rev, not layer revisions, config and shortlog. Ideally it would output something like:

meta: angstrom-staging:b3ffd40a4a1eacab90dac26df8a4a047486f3b96 pulseaudio: update to 1.1, delete 0.9.x
meta-oe: master:7019fbb50796fedcedef6ff92981a7f26659a251 mplayer: fix build with -Wl,-as-needed 
meta-efl:
meta-gnome:
meta-xfce:
meta-angstrom master:bc72e71ccf7e79cdc8246e0ae0dbe62d11355263 angstrom-2010-preferred-versions.inc : Stop pinning python-cairo

Taking this a step further: it's output should be able to get passed into layer tooling (combo-layer to yocto people, setup-scripts for angstrom people) to reproduce builds.

regards,

Koen

[-- Attachment #2: Message signed with OpenPGP using GPGMail --]
[-- Type: application/pgp-signature, Size: 169 bytes --]

Re: Image history (testlab)

[Richard Purdie]

On Fri, 2011-11-25 at 14:17 +0100, Koen Kooi wrote:
> Op 25 nov. 2011, om 12:44 heeft Paul Eggleton het volgende geschreven:
> > Hi Koen / Beth / all,
> > 
> > Following on from my work extending packagehistory (which is still under 
> > review), I'm looking at tracking history of the content of images so we can 
> > get warnings about regressions. Thanks to Koen we've had this capability via 
> > testlab.bbclass in OE for some time now, and from there into meta-oe, but what 
> > I'd like to do now is bring this into OE-core and extend it so that it 
> > supports rpm (or even better, all of the package backends we support.) This 
> > could involve abstracting the package backend functionality out into 
> > rootfs_*.bbclass.
> > 
> > It also looks like testlab.bbclass in meta-oe has some license dumping 
> > functionality. Beth, does that mesh with what you've been working on / 
> > preparing for Yocto 1.2?
> > 
> > Thoughts?
> 
> Don't forget its integration into remote git: http://git.angstrom-distribution.org/cgi-bin/cgit.cgi/testlab/?h=yocto 
> I must admit that it completely sucks when you have >1 builder, but that is fixable. There are a number of other things we need to store somewhere (e.g. package history, PR config, etc), so coming up with a plan to store all that in a single repo would be good.
> 
> The are a few issues with its current output:
> 
> * License info uses the filename instead of the package name (f'oo_1.2.3_arch.ipk' vs 'foo') which leads to too much changes with small version bumps
> * Dependencies are not sorted intelligently, so sometimes you get diffs which don't contain any actual change, just ordering junk,
> * It only reports oe-core rev, not layer revisions, config and shortlog. Ideally it would output something like:
> 
> meta: angstrom-staging:b3ffd40a4a1eacab90dac26df8a4a047486f3b96 pulseaudio: update to 1.1, delete 0.9.x
> meta-oe: master:7019fbb50796fedcedef6ff92981a7f26659a251 mplayer: fix build with -Wl,-as-needed 
> meta-efl:
> meta-gnome:
> meta-xfce:
> meta-angstrom master:bc72e71ccf7e79cdc8246e0ae0dbe62d11355263 angstrom-2010-preferred-versions.inc : Stop pinning python-cairo
> 
> Taking this a step further: it's output should be able to get passed into layer tooling (combo-layer to yocto people, setup-scripts for angstrom people) to reproduce builds.

So the real question is one of the approach. We have several options and
its related to how compatible we want/need to be.

Do we import testlab from meta-oe as is, then add patches (multiple
package backend support and so on)? How compatible do we need to be? I'm
conscious its something being actively used.

Do we enhance packagehistory reusing code from testlab as appropriate?

Do we start something new as a hybrid of the two? (shouldn't be needed
as I'm prepared to say packagehistory can radically change as needed, I
don't care about compatibility there).

At the end of this I want to have one great "history" tracking mechanism
for finding regressions covering testlab's functionality, package
history and more.

Just to be clear I don't want to reinvent the wheel :).

Cheers,

Richard

Re: Image history (testlab)

[Koen Kooi]

[-- Attachment #1: Type: text/plain, Size: 3708 bytes --]


Op 25 nov. 2011, om 14:52 heeft Richard Purdie het volgende geschreven:

> On Fri, 2011-11-25 at 14:17 +0100, Koen Kooi wrote:
>> Op 25 nov. 2011, om 12:44 heeft Paul Eggleton het volgende geschreven:
>>> Hi Koen / Beth / all,
>>> 
>>> Following on from my work extending packagehistory (which is still under 
>>> review), I'm looking at tracking history of the content of images so we can 
>>> get warnings about regressions. Thanks to Koen we've had this capability via 
>>> testlab.bbclass in OE for some time now, and from there into meta-oe, but what 
>>> I'd like to do now is bring this into OE-core and extend it so that it 
>>> supports rpm (or even better, all of the package backends we support.) This 
>>> could involve abstracting the package backend functionality out into 
>>> rootfs_*.bbclass.
>>> 
>>> It also looks like testlab.bbclass in meta-oe has some license dumping 
>>> functionality. Beth, does that mesh with what you've been working on / 
>>> preparing for Yocto 1.2?
>>> 
>>> Thoughts?
>> 
>> Don't forget its integration into remote git: http://git.angstrom-distribution.org/cgi-bin/cgit.cgi/testlab/?h=yocto 
>> I must admit that it completely sucks when you have >1 builder, but that is fixable. There are a number of other things we need to store somewhere (e.g. package history, PR config, etc), so coming up with a plan to store all that in a single repo would be good.
>> 
>> The are a few issues with its current output:
>> 
>> * License info uses the filename instead of the package name (f'oo_1.2.3_arch.ipk' vs 'foo') which leads to too much changes with small version bumps
>> * Dependencies are not sorted intelligently, so sometimes you get diffs which don't contain any actual change, just ordering junk,
>> * It only reports oe-core rev, not layer revisions, config and shortlog. Ideally it would output something like:
>> 
>> meta: angstrom-staging:b3ffd40a4a1eacab90dac26df8a4a047486f3b96 pulseaudio: update to 1.1, delete 0.9.x
>> meta-oe: master:7019fbb50796fedcedef6ff92981a7f26659a251 mplayer: fix build with -Wl,-as-needed 
>> meta-efl:
>> meta-gnome:
>> meta-xfce:
>> meta-angstrom master:bc72e71ccf7e79cdc8246e0ae0dbe62d11355263 angstrom-2010-preferred-versions.inc : Stop pinning python-cairo
>> 
>> Taking this a step further: it's output should be able to get passed into layer tooling (combo-layer to yocto people, setup-scripts for angstrom people) to reproduce builds.
> 
> So the real question is one of the approach. We have several options and
> its related to how compatible we want/need to be.
> 
> Do we import testlab from meta-oe as is, then add patches (multiple
> package backend support and so on)? How compatible do we need to be? I'm
> conscious its something being actively used.

In this specific case I don't care about backwards compatibility.

> Do we enhance packagehistory reusing code from testlab as appropriate?
> 
> Do we start something new as a hybrid of the two? (shouldn't be needed
> as I'm prepared to say packagehistory can radically change as needed, I
> don't care about compatibility there).
> 
> At the end of this I want to have one great "history" tracking mechanism
> for finding regressions covering testlab's functionality, package
> history and more.
> 
> Just to be clear I don't want to reinvent the wheel :).

For this history tracking both 'testlab' and 'packagehistory' are misnomers, so we can just start a new class that reuses ideas and code if needed.

Hopefully testlab can then fullfill its original goal: fully automated on-target testing. That should be doable with OE-core since that only has qemu machines :)

regards,

Koen

[-- Attachment #2: Message signed with OpenPGP using GPGMail --]
[-- Type: application/pgp-signature, Size: 169 bytes --]