[Announcement] Yocto Project 1.5 Milestone 3 now available.

[Announcement] Yocto Project 1.5 Milestone 3 now available.

[Flanagan, Elizabeth]

Hello,

The latest milestone release of the Yocto Project's upcoming 1.5 release, is now
available for download at:

http://downloads.yoctoproject.org/releases/yocto/milestones/yocto-1.5-m3/poky.tar.bz2

Thanks go out to everyone for all their hard work during this milestone release!

Sincerely,

Elizabeth Flanagan
Yocto Build and Release
<elizabeth.flanagan@intel.com>

--------------------
yocto-1.5_M3 Errata
--------------------
Release Name: poky-1.5_M3
Branch:  master
Tag:  1.5_M3
Hash:  9de0ad47462c13ac4a2d73e12e92be6c1b4e0415
md5: d34596ae2f14056a02fb0fb4b4ebb63c
Download Locations:
http://downloads.yoctoproject.org/releases/yocto/milestones/yocto-1.5-m3/poky.tar.bz2

Release Name: eclipse-poky-juno-1.5_M3
Branch:  master
Tag:  1.5_M3
Hash:  76df60597ae7cce069a5f1fd45089ea04c7394cd
md5: 4059d198768f9f8dc9372dc1c54bc3c3
Download Locations:
http://downloads.yoctoproject.org/releases/yocto/milestones/yocto-1.5-m3/eclipse-poky-juno.tar.bz2

Release Name: eclipse-poky-kepler-1.5_M3
Branch:  master
Tag:  1.5_M3
Hash:  73cb94acdc81fd876c7da1c49334da9e714b11ee
md5: 4059d198768f9f8dc9372dc1c54bc3cr
Download Locations:
http://downloads.yoctoproject.org/releases/yocto/milestones/yocto-1.5-m3/eclipse-poky-kepler.tar.bz2

Release Name: meta-qt3-1.5_M3
Branch:  master
Tag:  1.5_M3
Hash:  b73552fb998fd30a01dbee5a172e432a16078222
md5: bbe92eeccf520f18f067f8f40f891774
Download Locations:
http://downloads.yoctoproject.org/releases/yocto/milestones/yocto-1.5-m3/meta-qt3.tar.bz2

--------------------
Features
--------------------
* Major tool chain upgrade: gcc 4.8, eglibc 2.18 recipes.
* Generic x86 BSP supporting most of the graphic solutions. More
hardware will be supported.
* Bitbake refactoring including task execution structure change to use
worker process (bitbake-worker) and the separation of configuration
from data.
* Eliminated the double bitbake execution of psuedo.
* gcc security flags enabled to improve security level of images/binaries
* meta-security layer to support Bastille Linux and other security tools.
* Eclipse plugin has been upgraded to support Eclipse Kepler .
* Infrastructure for automated QA testing. More testcases will be
automated in the next milestone release.
* Improvement and refactoring of sanity testing the package sanity
test (insane.bbclass, package.bbclass)
* User experience improvement including improved SMART package
management tool error reporting, package date and bitbake error
messages.
* ptest added for more packages including busybox, python, kmod,
libxml2, bzip2, etc. to enable automated testing (from the community)
* Replaced Tinylogin with Busybox as the login manager.
* More improvement and fixes for Wayland and systemd, etc.

--------------------
Known issues:
--------------------
- 3908        Kernel panic on qemux86-64 when using runqemu kvm and cpu host
- 4529        x32 build fails with gmp_5.1.1 do_configure
- 4783        [adt-installer] ARM: target arch change from armv5te to
armv7a-vfp-neon in environment setting script
- 4977        [Build Appliance] An error message when launch hob

Re: [Announcement] Yocto Project 1.5 Milestone 3 now available.

[Robert Berger]

Hi,

On 08/10/2013 03:22 AM, Flanagan, Elizabeth wrote:
> * Infrastructure for automated QA testing. More testcases will be
> automated in the next milestone release.

Are we talking about test automation using autobuilder here?
As far as I can see runtime testing of images using QEMU is
planned/already used.

What I'm after is runtime testing on real hardware. Assuming an
infrastructure where you can remotely turn on/off the power of the board
and where you have access e.g. via a conserver to the serial port I
guess this should be possible as well.

I'm wondering why this seems to be limited to QEMU, am I missing something?

Regards,

Robert



..."The pure and simple truth is rarely pure and never simple." - Oscar
Wilde

My public pgp key is available,at:
http://pgp.mit.edu:11371/pks/lookup?op=get&search=0x90320BF1

Re: [Announcement] Yocto Project 1.5 Milestone 3 now available.

[Paul Eggleton]

Hi Robert,

On Saturday 10 August 2013 10:20:54 Robert Berger wrote:
> On 08/10/2013 03:22 AM, Flanagan, Elizabeth wrote:
> > * Infrastructure for automated QA testing. More testcases will be
> > automated in the next milestone release.
> 
> Are we talking about test automation using autobuilder here?
> As far as I can see runtime testing of images using QEMU is
> planned/already used.

The new framework replaces the old, is written in python and makes writing 
tests much easier. As a result we are going through a process of adding many 
more runtime tests than we had previously.

> What I'm after is runtime testing on real hardware. Assuming an
> infrastructure where you can remotely turn on/off the power of the board
> and where you have access e.g. via a conserver to the serial port I
> guess this should be possible as well.
> 
> I'm wondering why this seems to be limited to QEMU, am I missing something?

We do definitely want automated testing on real hardware; it's just that that 
it presents a number of problems that need solving:

 * How do you deploy the image/kernel/bootloader onto the board in an 
automated manner? To different kinds of boards?
 * How do you manage access to the boards when you have multiple autobuilders 
potentially wanting to make use of them at around the same time?

We'll be working through these but the likelihood is that we won't have this 
part of it for 1.5. Once we do though, connecting the currently QEMU-based 
testing framework to it should be trivial.

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre

Re: [Announcement] Yocto Project 1.5 Milestone 3 now available.

[Robert Berger]

Hi,

On 08/10/2013 01:02 PM, Paul Eggleton wrote:
> 
> The new framework replaces the old, is written in python and makes writing 
> tests much easier. As a result we are going through a process of adding many 
> more runtime tests than we had previously.

I'm confused now. Is it autobuilder + new stuff in python and/or ptest
or something completely different? Are there already some examples in
git I could have a look at?

> 
> 
> We do definitely want automated testing on real hardware; it's just that that 
> it presents a number of problems that need solving:
> 
>  * How do you deploy the image/kernel/bootloader onto the board in an 
> automated manner? 

I use kernel over tftp and rootfs over nfs. The trivial example of a
boot loader upgrade is when you have a running (Linux) system just scp
it over. It's getting non trivial in case you manage to screw up the
boot loader. For the generic case maybe some update strategy with a
fallback solution to a known good boot loader might be needed.

> To different kinds of boards?

kernel over tftp and rootfs over nfs should work with all boards.

>  * How do you manage access to the boards when you have multiple autobuilders 
> potentially wanting to make use of them at around the same time?

Mutual exclusive access to the (serial) console can be handled by
conserver. Mutual exclusive via ssh gets more tricky.

I don't think it makes much sense to access the same board from multiple
autobuilders. This is most likely an error case which you would like to
catch.

> 
> We'll be working through these but the likelihood is that we won't have this 
> part of it for 1.5. Once we do though, connecting the currently QEMU-based 
> testing framework to it should be trivial.

OK understood. Please keep me updated.

> 
> Cheers,
> Paul

Regards,

Robert

...If you think good architecture is expensive, try bad architecture -
Brian Foote and Joseph Yoder

My public pgp key is available,at:
http://pgp.mit.edu:11371/pks/lookup?op=get&search=0x90320BF1

Re: [Announcement] Yocto Project 1.5 Milestone 3 now available.

[Björn Stenberg]

Paul Eggleton wrote:
> We do definitely want automated testing on real hardware; it's just that
> that it presents a number of problems that need solving:
> 
>  * How do you deploy the image/kernel/bootloader onto the board in an 
> automated manner? To different kinds of boards?
>  * How do you manage access to the boards when you have multiple
> autobuilders potentially wanting to make use of them at around the same
> time?

These are all solved problems. We build and auto-test Yocto nightly on a bunch of different hardware using tftp and nfs. I'm sure other teams do something similar. However, since there is no Yocto standard for this yet we all do it slightly differently.

Also, even when/if Intel sets up an automated hardware lab for Yocto, it will necessarily only have a limited selection of boards. Perhaps we could have a system where we pool community test results into one place rather than depend on a single test lab.

I have said it before, and I'll remind/nag about it again: Please don't run off and do this on your own. This is something that touches the daily work of many community members and it should be done with open discussion.

-- 
Björn

Re: [Announcement] Yocto Project 1.5 Milestone 3 now available.

[Paul Eggleton]

Hi Björn,

On Monday 12 August 2013 11:11:45 Björn Stenberg wrote:
> Paul Eggleton wrote:
> > We do definitely want automated testing on real hardware; it's just that
> > that it presents a number of problems that need solving:
> >  * How do you deploy the image/kernel/bootloader onto the board in an
> > automated manner? To different kinds of boards?
> >  * How do you manage access to the boards when you have multiple
> > autobuilders potentially wanting to make use of them at around the same
> > time?
> 
> These are all solved problems. We build and auto-test Yocto nightly on a
> bunch of different hardware using tftp and nfs. I'm sure other teams do
> something similar. However, since there is no Yocto standard for this yet
> we all do it slightly differently.

Right, that's what we would be aiming for. Perhaps you could talk a little bit 
about how you manage your hardware testing and any additional software and 
scripts you make use of there?

> Also, even when/if Intel sets up an automated hardware lab for Yocto, it
> will necessarily only have a limited selection of boards. Perhaps we could
> have a system where we pool community test results into one place rather
> than depend on a single test lab.

I agree. In fact our test lab would probably be limited to the reference 
boards that the Yocto Project officially supports within meta-yocto i.e. what we 
currently test on manually.

> I have said it before, and I'll remind/nag about it again: Please don't run
> off and do this on your own. This is something that touches the daily work
> of many community members and it should be done with open discussion.

Good point, but this is one of the reasons we haven't done anything other than 
some thinking about it on my part :) FWIW we still haven't started anything on 
ptest automation either so it's not the case that work is being done on these 
things outside of community channels. In either case, we can definitely start 
talking about it now if folks are ready but it's likely the implementation 
work won't get done until 1.6.

The automated testing work we have done so far for 1.5 is just about ensuring 
there is a simple way to define runtime tests (completely separate from ptest) 
and make that easy for people to extend to cover their own use cases. This in 
turn makes it easier runtime tests to be written to automate a large portion 
of our existing manual QA process.

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre

Re: [Announcement] Yocto Project 1.5 Milestone 3 now available.

[Paul Eggleton]

On Sunday 11 August 2013 10:03:12 Robert Berger wrote:
> On 08/10/2013 01:02 PM, Paul Eggleton wrote:
> > The new framework replaces the old, is written in python and makes writing
> > tests much easier. As a result we are going through a process of adding
> > many more runtime tests than we had previously.
> 
> I'm confused now. Is it autobuilder + new stuff in python and/or ptest
> or something completely different? Are there already some examples in
> git I could have a look at?

This is largely unrelated to ptest although at some point in the near future 
we'll need to have a discussion about how ptest should be integrated. It's 
really about being able to easily define tests that verify functionality of 
multiple installed packages together, e.g. whether smart can install an rpm 
package from a feed.

There's not much on the autobuilder side apart from ensuring the appropriate 
configuration is set - one of the aims was to avoid too much intrusion into the 
autobuilder code. The main class is testimage.bbclass, so you can just do 
INHERIT += "testimage" in local.conf and then run bitbake -c testimage 
<imagename>. The tests are in meta/lib/oeqa/runtime/ and utility code under 
meta/lib/oeqa/utils. 

There's no documentation for this yet other than what's in the code (and there 
are some comments there), but it's coming soon.

> > We do definitely want automated testing on real hardware; it's just that
> > that it presents a number of problems that need solving:
> >  * How do you deploy the image/kernel/bootloader onto the board in an
> > automated manner?
> 
> I use kernel over tftp and rootfs over nfs. The trivial example of a
> boot loader upgrade is when you have a running (Linux) system just scp
> it over. It's getting non trivial in case you manage to screw up the
> boot loader. For the generic case maybe some update strategy with a
> fallback solution to a known good boot loader might be needed.
> 
> > To different kinds of boards?
> 
> kernel over tftp and rootfs over nfs should work with all boards.

Sounds like a sensible approach then.
 
> >  * How do you manage access to the boards when you have multiple
> >  autobuilders potentially wanting to make use of them at around the same
> >  time?
> 
> Mutual exclusive access to the (serial) console can be handled by
> conserver. Mutual exclusive via ssh gets more tricky.

The point is I think at least for how we run the YP autobuilders you'd want 
there to be a queueing mechanism.
 
> I don't think it makes much sense to access the same board from multiple
> autobuilders. This is most likely an error case which you would like to
> catch.

I agree, but equally I think it would be preferable for an autobuilder task 
that requires access to a board to wait until any other users have finished 
rather than just failing immediately, and if possible to know the difference 
between waiting for access and timing out because the device isn't responding.

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre

Re: [Announcement] Yocto Project 1.5 Milestone 3 now available.

[Björn Stenberg]

Paul Eggleton wrote:
> Right, that's what we would be aiming for. Perhaps you could talk a little
> bit about how you manage your hardware testing and any additional software
> and scripts you make use of there?

We run our tests from buildbot, using an expect script to allocate the right serial port (and wait if it is busy), fetch the built files, unpack the rootfs to an NFS disk, reset the board, tftp the kernel and dtb, boot the kernel and start the test suite.

A line-based timeout is used for for the test suites, meaning that the timeout is reset for each new line of output. This allows us to handle very long-running test cases while still being able detect hung tests rather quickly.

Multiple images and test suites are built and ran in parallell, which is why the script waits for the serial port. One of the test suites we run is ptest-runner.

The output of all tests is parsed and stored in a database, cross-referenced with the buildbot build info. We have written an extension to the buildbot web interface presenting the test results as they relate to each build.

-- 
Björn

Re: [Announcement] Yocto Project 1.5 Milestone 3 now available.

[Robert Berger]

Hi,

On 08/12/2013 02:30 PM, Björn Stenberg wrote:
> Paul Eggleton wrote:
>> Right, that's what we would be aiming for. Perhaps you could talk a little
>> bit about how you manage your hardware testing and any additional software
>> and scripts you make use of there?
> 
> We run our tests from buildbot, using an expect script 

I currently run my tests manually, but the plan was to implement
something in expect as well, since this seems to be made exactly for
this kind of stuff.

... but instead of something proprietary I would rather prefer to have a
generic solution where "the community" solves problems together with me.

> to allocate the right serial port (and wait if it is busy), fetch the built files, unpack the rootfs to an NFS disk, reset the board, tftp the kernel and dtb, boot the kernel and start the test suite.
> 
> A line-based timeout is used for for the test suites, meaning that the timeout is reset for each new line of output. This allows us to handle very long-running test cases while still being able detect hung tests rather quickly.
> 
> Multiple images and test suites are built and ran in parallell, which is why the script waits for the serial port. One of the test suites we run is ptest-runner.
> 
> The output of all tests is parsed and stored in a database, cross-referenced with the buildbot build info. We have written an extension to the buildbot web interface presenting the test results as they relate to each build.
> 

Are your scripts available somewhere to give it a try? Do they integrate
into autobuilder?

Regards,

Robert




..."German programmers tend to take it as a personal insult when a fault
is detected in code that they have written." - Debora Weber-Wulff
weberwu@tfh-berlin.de [comp.risks 16.94]

My public pgp key is available,at:
http://pgp.mit.edu:11371/pks/lookup?op=get&search=0x90320BF1

Re: [Announcement] Yocto Project 1.5 Milestone 3 now available.

[Robert Berger]

Hi,

On 08/12/2013 12:28 PM, Paul Eggleton wrote:
> 
> I agree. In fact our test lab would probably be limited to the reference 
> boards that the Yocto Project officially supports within meta-yocto i.e. what we 
> currently test on manually.

I have the Yocto reference boards as well, but will add boards on
customer request (some are already on the queue).

BTW what should be defined what kind of extra hardware/software is
needed for remote hardware testing. As I said I have some infrastructure
to power on/off boards and monitor the serial over conserver(s).

Is that's it?

Regards,

Robert
..."100% test coverage is insufficient. 35% of the faults are missing
logic paths." - Robert Glass

My public pgp key is available,at:
http://pgp.mit.edu:11371/pks/lookup?op=get&search=0x90320BF1

Re: [Announcement] Yocto Project 1.5 Milestone 3 now available.

[Robert Berger]

Hi,

On 08/12/2013 12:44 PM, Paul Eggleton wrote:
> 
> There's not much on the autobuilder side apart from ensuring the appropriate 
> configuration is set - one of the aims was to avoid too much intrusion into the 
> autobuilder code. The main class is testimage.bbclass, so you can just do 
> INHERIT += "testimage" in local.conf and then run bitbake -c testimage 
> <imagename>. The tests are in meta/lib/oeqa/runtime/ and utility code under 
> meta/lib/oeqa/utils. 
> 
> There's no documentation for this yet other than what's in the code (and there 
> are some comments there), but it's coming soon.

;)

I'll have a look at it.

Regards,

Robert


...A clever person solves a problem.A wise person avoids it.- Einstein

My public pgp key is available,at:
http://pgp.mit.edu:11371/pks/lookup?op=get&search=0x90320BF1

Re: [Announcement] Yocto Project 1.5 Milestone 3 now available.

[Björn Stenberg]

Robert Berger wrote:
> Are your scripts available somewhere to give it a try? Do they integrate
> into autobuilder?

The scripts are hardcoded for our environment and our modified autobuilder and are therefore not published anywhere.

While I could see us publishing these or similar snippets, I think it's better to start with a general discussion. Perhaps there is a slight risk of chicken-and-egg here, with discussions not taking off until code is proposed. But I believe it is the best path for now.

-- 
Björn