Automated testing on real hardware

Automated testing on real hardware

[Paul Eggleton]

Hi folks,

(warning - long email ahead)

As I've talked about recently, one of the things I'm looking into at the 
moment is how to do automated QA for Yocto Project releases on real hardware. 
The Yocto Project QA team does tests on the supported reference platforms for 
each architecture for every release; however this is currently done manually, 
which takes quite a long time as you might imagine. In the recent 1.5 (dora) 
release we've added the ability to define tests to run on the target, but that 
was limited to booting images within QEMU and running the tests there; we now 
want to extend this to run the same tests on real hardware.

What it really boils down to is a small set of key requirements:

* The ability to deploy a recently built kernel + image on the hardware and 
then boot it.

* The ability to obtain a channel through which we can run our existing tests; 
i.e. we don't need a new test harness.

* The image under test must be unmodified. We should be able to test the 
"production" images as they are currently produced by our autobuilder.

I've looked at a couple of existing automated test frameworks, and my findings 
are summarised below.

LAVA
-----

A number of people had suggested I look at LAVA [1]. It is split into different 
components for each function, some of which should be usable independently so 
you don't have to use the entire framework. Many of the components could be 
useful assuming they are independent, but in terms of being able to actually 
run our tests, the one that stands out as being most relevant is lava-
dispatcher. This is the component that deploys images, and boots the hardware 
in order to run the tests.

I've looked at lava-dispatcher a couple of times; firstly at the code, and 
yesterday I looked at actually running it. Initially it looked very promising 
- reasonably licensed, written in Python, has some rudimentary support for OE 
images. However while looking at it I found the following:

* It requires root to run, since it mounts the images temporarily and modifies 
the contents. We've done quite a lot to avoid having to run as root in OE, so 
this is a hard sell.

* It expects images to be created by linaro-media-create, which in turn 
requires an "hwpack" [2]. The hwpack concept seems primarily geared to Ubuntu 
and distributions like it where you'd have a generic base image upon which 
you'd install some board-specific packages in order to make it work on the 
board. OE doesn't work that way - we just build images specific to the board, 
which makes this mechanism completely superfluous for our usage; but the tool 
appears to require it.

* There is a general lack of abstraction and far too much hardcoding. For 
example, even at the most abstract level, the "Target" class (which is the 
base class for defining what to do for different types of targets) has hardcoded 
deploy_linaro / deploy_android functions:

https://git.linaro.org/gitweb?p=lava/lava-dispatcher.git;a=blob;f=lava_dispatcher/device/target.py

* It's not clear to me how well it will work across different host 
distributions that we want to support; the main LAVA installer quits if it 
isn't run on Ubuntu. For convenience I happened to be using an Ubuntu VM, and 
I wasn't running the full installer so I avoided this check altogether. It 
just concerns me that such a check would even be there.

Of course, none of these problems are impossible to overcome, if we're 
prepared to put a significant amount of engineering into resolving them. In 
terms of enabling the Yocto Project QA team to test images on real hardware in 
the 1.6 development cycle however, I don't believe this is going to be 
deliverable.

Autotest
----------

I've also been pointed to Autotest [3]. It initially looked quite promising as 
well - it's simpler than LAVA, and as with LAVA it's reasonably licensed and 
written in Python so it's familiar and easy for us to extend. I found the 
following limitations though:

* Unlike LAVA it's not split into components; the whole thing comes in one 
piece, and most interaction with it is through the web interface. However, it 
does have an RPC interface for starting jobs and a command-line client for 
same.

* It's not really geared to running tests on unmodified images. It normally 
expects to have its client, which requires Python, running on the target 
hardware. We can't have this in all images we build. Server-side control files 
could work in that they will allow you to just run commands over ssh to the 
target without requiring the client to be installed; however, annoyingly 
Autotest still insists on "verifying" and "repairing" the client on the target 
machine for every job even if it's not needed to execute the job, unless you 
set the host as "protected" in the host record for the machine.

* Unfortunately, it has no actual support for deploying the test image on the 
device in the absence of the client; even with the client this is largely left 
to the user to define.

So Autotest might be usable as a frontend, but we'd have to write the 
deployment code for it to be useful. Overall it did seem to me to be both a 
little bit more flexible and easier to understand than LAVA, but then it's 
trying to do a fair bit less.

Other
-------

I've heard from a couple of people who offered to open source their existing 
internal test frameworks. Unfortunately neither of them have done this yet so 
I haven't been able to look at them properly, and time in the 1.6 cycle is 
running out.


Conclusion
------------

So where to from here? We don't seem to have anything we can immediately use 
for 1.6 as far as the deployment goes, and that's the bit we need the most. 
Thus, I propose that we write some basic code to do the deployment, and extend 
our currently QEMU-based testing code to enable the tests to be exported and 
run anywhere. The emphasis will be on keeping things simple, and leaving the 
door open to closer integration with framework(s) in the future.

The approach I'm looking at is to have a bootloader + "master" image pre-
deployed onto the device, with a separate partition for the test image. We 
reset the hardware, boot into the master image, send over the test image to be 
deployed, then reboot and tell the bootloader to boot into the test image. (I 
had considered using root-over-NFS which simplifies deployment, but this 
creates difficulties in other areas e.g. testing networking.)

Breaking it down into individual work items, with time estimates:

1) Abstract out the deployment of the image and running the tests so that it 
can involve targets other than QEMU. This had to be done anyway, so Stefan 
Stanacar has gone ahead with it and has already sent a patch to do the basic 
abstraction and add a backend which can run the tests on a nominated machine 
over SSH, i.e. without any deployment (see bug 5554 [4].)
[0 days - patches in review]

2) Add the ability to export the tests so that they can run independently of 
the build system, as is required if you want to be able to hand the test 
execution off to a scheduler.
[5 days]

3) Add a recipe for a master image, some code to interact with it to deploy 
the test image, and a means of setting up the partition layout. This should be 
straightforward - we just need to send the image over ssh and extract/copy it 
onto the test partition. The partition layout should be able to be defined 
using wic.
[5 days]

4) Write some expect scripts to interact with the bootloaders for the 
reference machines over serial.
[7 days]

5) Add the ability to reset the hardware automatically - this really requires 
hardware assistance in the form of a web/serial controlled power strip, and 
there is no standard protocol for interacting with these although conmux does 
abstract this to a degree. To keep this simple, we would just have a variable 
that points to a command/script that does the hard reset, and then that can be 
defined outside of the build system.
[3 days]

In the absence of another workable solution or other folks stepping up to 
help, this is my current plan for 1.6. I've not had as much response to my 
previous requests for feedback on this area as I would have liked though, so 
if you have any concrete suggestions I'm definitely keen to hear them.

Cheers,
Paul

[1] https://wiki.linaro.org/Platform/LAVA
[2] https://wiki.linaro.org/HardwarePacks
[3] http://autotest.github.io/
[4] https://bugzilla.yoctoproject.org/show_bug.cgi?id=5554
[5] https://github.com/autotest/autotest/wiki/Conmux

(thanks to Darren Hart and Stefan Stanacar for helping me with this.)

-- 

Paul Eggleton
Intel Open Source Technology Centre

Re: Automated testing on real hardware

[Nicolas Dechesne]

[-- Attachment #1: Type: text/plain, Size: 3878 bytes --]

Paul,


On Fri, Nov 29, 2013 at 4:58 PM, Paul Eggleton <
paul.eggleton@linux.intel.com> wrote:

> LAVA
> -----
>
> A number of people had suggested I look at LAVA [1]. It is split into
> different
> components for each function, some of which should be usable independently
> so
> you don't have to use the entire framework. Many of the components could be
> useful assuming they are independent, but in terms of being able to
> actually
> run our tests, the one that stands out as being most relevant is lava-
> dispatcher. This is the component that deploys images, and boots the
> hardware
> in order to run the tests.
>
> I've looked at lava-dispatcher a couple of times; firstly at the code, and
> yesterday I looked at actually running it. Initially it looked very
> promising
> - reasonably licensed, written in Python, has some rudimentary support for
> OE
> images. However while looking at it I found the following:
>
> * It requires root to run, since it mounts the images temporarily and
> modifies
> the contents. We've done quite a lot to avoid having to run as root in OE,
> so
> this is a hard sell.
>

i have forwarded this email to the relevant people in Linaro working on
LAVA. I hope to be able to bring more information about that, as I am not
involved with LAVA, just a 'user' of it. By 'root' here, you mean on the
server side, e.g. the LAVA instance, which will typically not be the
'builder', or do you want to couple build and test on the same 'server'?
LAVA jobs are generally submitted by a CI instance (Jenkins in our case).


> * It expects images to be created by linaro-media-create, which in turn
> requires an "hwpack" [2]. The hwpack concept seems primarily geared to
> Ubuntu
> and distributions like it where you'd have a generic base image upon which
> you'd install some board-specific packages in order to make it work on the
> board. OE doesn't work that way - we just build images specific to the
> board,
> which makes this mechanism completely superfluous for our usage; but the
> tool
> appears to require it.
>

that is no longer the case. we do have support for 'native' OE images, and
we've had that for a while. That is what I am using for my project at
Linaro. None of our projects/jobs deliverable are 'public' at the moment,
so you won't find them, but i can confirm that what we deploy for our jobs
is the output of OE (e.g. what is in the deploy/image folder). hwpack was
something mostly meant to make it simpler to work with Ubuntu and Android
root FS, and is not suited for OE.


>
> * There is a general lack of abstraction and far too much hardcoding. For
> example, even at the most abstract level, the "Target" class (which is the
> base class for defining what to do for different types of targets) has
> hardcoded
> deploy_linaro / deploy_android functions:
>
>
> https://git.linaro.org/gitweb?p=lava/lava-dispatcher.git;a=blob;f=lava_dispatcher/device/target.py
>
> * It's not clear to me how well it will work across different host
> distributions that we want to support; the main LAVA installer quits if it
> isn't run on Ubuntu. For convenience I happened to be using an Ubuntu VM,
> and
> I wasn't running the full installer so I avoided this check altogether. It
> just concerns me that such a check would even be there.
>

I agree that this is a problem and we should fix. Our IT infrastructure is
largely based on Ubuntu servers, but that should not affect what we create
that much.


>
> Of course, none of these problems are impossible to overcome, if we're
> prepared to put a significant amount of engineering into resolving them. In
> terms of enabling the Yocto Project QA team to test images on real
> hardware in
> the 1.6 development cycle however, I don't believe this is going to be
> deliverable.
>

[-- Attachment #2: Type: text/html, Size: 5446 bytes --]

Re: [OE-core] Automated testing on real hardware

[Nicolas Dechesne]

[-- Attachment #1: Type: text/plain, Size: 3878 bytes --]

Paul,


On Fri, Nov 29, 2013 at 4:58 PM, Paul Eggleton <
paul.eggleton@linux.intel.com> wrote:

> LAVA
> -----
>
> A number of people had suggested I look at LAVA [1]. It is split into
> different
> components for each function, some of which should be usable independently
> so
> you don't have to use the entire framework. Many of the components could be
> useful assuming they are independent, but in terms of being able to
> actually
> run our tests, the one that stands out as being most relevant is lava-
> dispatcher. This is the component that deploys images, and boots the
> hardware
> in order to run the tests.
>
> I've looked at lava-dispatcher a couple of times; firstly at the code, and
> yesterday I looked at actually running it. Initially it looked very
> promising
> - reasonably licensed, written in Python, has some rudimentary support for
> OE
> images. However while looking at it I found the following:
>
> * It requires root to run, since it mounts the images temporarily and
> modifies
> the contents. We've done quite a lot to avoid having to run as root in OE,
> so
> this is a hard sell.
>

i have forwarded this email to the relevant people in Linaro working on
LAVA. I hope to be able to bring more information about that, as I am not
involved with LAVA, just a 'user' of it. By 'root' here, you mean on the
server side, e.g. the LAVA instance, which will typically not be the
'builder', or do you want to couple build and test on the same 'server'?
LAVA jobs are generally submitted by a CI instance (Jenkins in our case).


> * It expects images to be created by linaro-media-create, which in turn
> requires an "hwpack" [2]. The hwpack concept seems primarily geared to
> Ubuntu
> and distributions like it where you'd have a generic base image upon which
> you'd install some board-specific packages in order to make it work on the
> board. OE doesn't work that way - we just build images specific to the
> board,
> which makes this mechanism completely superfluous for our usage; but the
> tool
> appears to require it.
>

that is no longer the case. we do have support for 'native' OE images, and
we've had that for a while. That is what I am using for my project at
Linaro. None of our projects/jobs deliverable are 'public' at the moment,
so you won't find them, but i can confirm that what we deploy for our jobs
is the output of OE (e.g. what is in the deploy/image folder). hwpack was
something mostly meant to make it simpler to work with Ubuntu and Android
root FS, and is not suited for OE.


>
> * There is a general lack of abstraction and far too much hardcoding. For
> example, even at the most abstract level, the "Target" class (which is the
> base class for defining what to do for different types of targets) has
> hardcoded
> deploy_linaro / deploy_android functions:
>
>
> https://git.linaro.org/gitweb?p=lava/lava-dispatcher.git;a=blob;f=lava_dispatcher/device/target.py
>
> * It's not clear to me how well it will work across different host
> distributions that we want to support; the main LAVA installer quits if it
> isn't run on Ubuntu. For convenience I happened to be using an Ubuntu VM,
> and
> I wasn't running the full installer so I avoided this check altogether. It
> just concerns me that such a check would even be there.
>

I agree that this is a problem and we should fix. Our IT infrastructure is
largely based on Ubuntu servers, but that should not affect what we create
that much.


>
> Of course, none of these problems are impossible to overcome, if we're
> prepared to put a significant amount of engineering into resolving them. In
> terms of enabling the Yocto Project QA team to test images on real
> hardware in
> the 1.6 development cycle however, I don't believe this is going to be
> deliverable.
>

[-- Attachment #2: Type: text/html, Size: 5446 bytes --]

Re: Automated testing on real hardware

[Paul Eggleton]

Hi Nicolas,

On Friday 29 November 2013 17:31:40 Nicolas Dechesne wrote:
> On Fri, Nov 29, 2013 at 4:58 PM, Paul Eggleton <
> paul.eggleton@linux.intel.com> wrote:
> > * It requires root to run, since it mounts the images temporarily and
> > modifies the contents. We've done quite a lot to avoid having to run as
> > root in OE, so this is a hard sell.
> 
> i have forwarded this email to the relevant people in Linaro working on
> LAVA. I hope to be able to bring more information about that, as I am not
> involved with LAVA, just a 'user' of it. 

OK, thanks. I'm certainly happy to have a wider discussion on this.

> By 'root' here, you mean on the server side, e.g. the LAVA instance, which
> will typically not be the 'builder', or do you want to couple build and test
> on the same 'server'?

That's what I've been looking at - just using lava-dispatcher rather than the 
full LAVA framework. That said, if all it requires root for is mounting the 
image, modifying it and then unmounting it, to test our images I don't think 
we even want this; but there appears to be no way to disable it. I then found 
a launchpad bug where the root requirement was discussed and the accepted 
solution there was for the dispatcher to just check and error out if not 
running as root.

> LAVA jobs are generally submitted by a CI instance (Jenkins in our case).

Our test code currently queries configuration/image information from the build 
system, which means that the build system has to be involved in the dispatch 
process. Our CI solution (the autobuilder) of course runs the build system 
though. One of the work items in my email is to be able to export the tests, 
thus decoupling them from this; the build system would still need to be in 
charge of exporting the tests, but it would allow a test server to manage the 
deployment, execution and results collection.
 
> > * It expects images to be created by linaro-media-create, which in turn
> > requires an "hwpack" [2]. The hwpack concept seems primarily geared to
> > Ubuntu and distributions like it where you'd have a generic base image
> > upon which you'd install some board-specific packages in order to make it
> > work on the board. OE doesn't work that way - we just build images specific
> > to the board, which makes this mechanism completely superfluous for our
> > usage; but the tool appears to require it.
> 
> that is no longer the case. we do have support for 'native' OE images, and
> we've had that for a while. That is what I am using for my project at
> Linaro. None of our projects/jobs deliverable are 'public' at the moment,
> so you won't find them, but i can confirm that what we deploy for our jobs
> is the output of OE (e.g. what is in the deploy/image folder). hwpack was
> something mostly meant to make it simpler to work with Ubuntu and Android
> root FS, and is not suited for OE.

OK, fair enough - I guess the trouble here is it's hard to find any 
documentation talking about this functionality; everything I could find that 
talks about preparing an image refers to needing an hwpack and using linaro-
media-create. Even mailing list posts that I found (admittedly from last year) 
referred to these in the context of booting OE images; there are also hwpacks 
offered for download under "openembedded" directories on releases.linaro.org. 
Given these I hope you can understand my confusion.

Generally, if someone with experience of doing so successfully could document 
how LAVA can be set up to test OE images I think that would be very helpful. I 
did struggle a bit with the documentation - there are still quite a lot of 
broken links to the old documentation on readthedocs.org.

> > * It's not clear to me how well it will work across different host
> > distributions that we want to support; the main LAVA installer quits if it
> > isn't run on Ubuntu. For convenience I happened to be using an Ubuntu VM,
> > and I wasn't running the full installer so I avoided this check
> > altogether. It just concerns me that such a check would even be there.
> 
> I agree that this is a problem and we should fix. Our IT infrastructure is
> largely based on Ubuntu servers, but that should not affect what we create
> that much.

Right. FWIW, as you probably know, the way we handle "unsupported" host 
distros is to allow running on them, but warn the user that they may 
experience problems (well, in our case we provide this host distro check 
functionality, it's on in Poky and off in OE-Core by default, not sure if 
others are using it).

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre

Re: [OE-core] Automated testing on real hardware

[Paul Eggleton]

Hi Nicolas,

On Friday 29 November 2013 17:31:40 Nicolas Dechesne wrote:
> On Fri, Nov 29, 2013 at 4:58 PM, Paul Eggleton <
> paul.eggleton@linux.intel.com> wrote:
> > * It requires root to run, since it mounts the images temporarily and
> > modifies the contents. We've done quite a lot to avoid having to run as
> > root in OE, so this is a hard sell.
> 
> i have forwarded this email to the relevant people in Linaro working on
> LAVA. I hope to be able to bring more information about that, as I am not
> involved with LAVA, just a 'user' of it. 

OK, thanks. I'm certainly happy to have a wider discussion on this.

> By 'root' here, you mean on the server side, e.g. the LAVA instance, which
> will typically not be the 'builder', or do you want to couple build and test
> on the same 'server'?

That's what I've been looking at - just using lava-dispatcher rather than the 
full LAVA framework. That said, if all it requires root for is mounting the 
image, modifying it and then unmounting it, to test our images I don't think 
we even want this; but there appears to be no way to disable it. I then found 
a launchpad bug where the root requirement was discussed and the accepted 
solution there was for the dispatcher to just check and error out if not 
running as root.

> LAVA jobs are generally submitted by a CI instance (Jenkins in our case).

Our test code currently queries configuration/image information from the build 
system, which means that the build system has to be involved in the dispatch 
process. Our CI solution (the autobuilder) of course runs the build system 
though. One of the work items in my email is to be able to export the tests, 
thus decoupling them from this; the build system would still need to be in 
charge of exporting the tests, but it would allow a test server to manage the 
deployment, execution and results collection.
 
> > * It expects images to be created by linaro-media-create, which in turn
> > requires an "hwpack" [2]. The hwpack concept seems primarily geared to
> > Ubuntu and distributions like it where you'd have a generic base image
> > upon which you'd install some board-specific packages in order to make it
> > work on the board. OE doesn't work that way - we just build images specific
> > to the board, which makes this mechanism completely superfluous for our
> > usage; but the tool appears to require it.
> 
> that is no longer the case. we do have support for 'native' OE images, and
> we've had that for a while. That is what I am using for my project at
> Linaro. None of our projects/jobs deliverable are 'public' at the moment,
> so you won't find them, but i can confirm that what we deploy for our jobs
> is the output of OE (e.g. what is in the deploy/image folder). hwpack was
> something mostly meant to make it simpler to work with Ubuntu and Android
> root FS, and is not suited for OE.

OK, fair enough - I guess the trouble here is it's hard to find any 
documentation talking about this functionality; everything I could find that 
talks about preparing an image refers to needing an hwpack and using linaro-
media-create. Even mailing list posts that I found (admittedly from last year) 
referred to these in the context of booting OE images; there are also hwpacks 
offered for download under "openembedded" directories on releases.linaro.org. 
Given these I hope you can understand my confusion.

Generally, if someone with experience of doing so successfully could document 
how LAVA can be set up to test OE images I think that would be very helpful. I 
did struggle a bit with the documentation - there are still quite a lot of 
broken links to the old documentation on readthedocs.org.

> > * It's not clear to me how well it will work across different host
> > distributions that we want to support; the main LAVA installer quits if it
> > isn't run on Ubuntu. For convenience I happened to be using an Ubuntu VM,
> > and I wasn't running the full installer so I avoided this check
> > altogether. It just concerns me that such a check would even be there.
> 
> I agree that this is a problem and we should fix. Our IT infrastructure is
> largely based on Ubuntu servers, but that should not affect what we create
> that much.

Right. FWIW, as you probably know, the way we handle "unsupported" host 
distros is to allow running on them, but warn the user that they may 
experience problems (well, in our case we provide this host distro check 
functionality, it's on in Poky and off in OE-Core by default, not sure if 
others are using it).

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre

Re: [yocto] Automated testing on real hardware

[Philip Balister]

On 11/29/2013 01:03 PM, Paul Eggleton wrote:
> Hi Nicolas,
> 
> On Friday 29 November 2013 17:31:40 Nicolas Dechesne wrote:
>> On Fri, Nov 29, 2013 at 4:58 PM, Paul Eggleton <
>> paul.eggleton@linux.intel.com> wrote:
>>> * It requires root to run, since it mounts the images temporarily and
>>> modifies the contents. We've done quite a lot to avoid having to run as
>>> root in OE, so this is a hard sell.
>>
>> i have forwarded this email to the relevant people in Linaro working on
>> LAVA. I hope to be able to bring more information about that, as I am not
>> involved with LAVA, just a 'user' of it. 
> 
> OK, thanks. I'm certainly happy to have a wider discussion on this.

Part of the discussion should be identify issues with projects like LAVA
and see if the LAVA developers agree with our perceptions and work out
if there is a way we can work together to remove the issues. I certainly
see a lot of promise in LAVA, but the comments Paul has are big problems
for the OpenEmbedded Community.

Part of the point of the Yocto Project is to try and focus people's
efforts on improving the embedded process, either by identifying
promising projects and working together to improve them, or by
developing new technology as needed. But before developing something
new, we nee to make sure we are not creating fragmentation in the
embedded space.

Philip

> 
>> By 'root' here, you mean on the server side, e.g. the LAVA instance, which
>> will typically not be the 'builder', or do you want to couple build and test
>> on the same 'server'?
> 
> That's what I've been looking at - just using lava-dispatcher rather than the 
> full LAVA framework. That said, if all it requires root for is mounting the 
> image, modifying it and then unmounting it, to test our images I don't think 
> we even want this; but there appears to be no way to disable it. I then found 
> a launchpad bug where the root requirement was discussed and the accepted 
> solution there was for the dispatcher to just check and error out if not 
> running as root.
> 
>> LAVA jobs are generally submitted by a CI instance (Jenkins in our case).
> 
> Our test code currently queries configuration/image information from the build 
> system, which means that the build system has to be involved in the dispatch 
> process. Our CI solution (the autobuilder) of course runs the build system 
> though. One of the work items in my email is to be able to export the tests, 
> thus decoupling them from this; the build system would still need to be in 
> charge of exporting the tests, but it would allow a test server to manage the 
> deployment, execution and results collection.
>  
>>> * It expects images to be created by linaro-media-create, which in turn
>>> requires an "hwpack" [2]. The hwpack concept seems primarily geared to
>>> Ubuntu and distributions like it where you'd have a generic base image
>>> upon which you'd install some board-specific packages in order to make it
>>> work on the board. OE doesn't work that way - we just build images specific
>>> to the board, which makes this mechanism completely superfluous for our
>>> usage; but the tool appears to require it.
>>
>> that is no longer the case. we do have support for 'native' OE images, and
>> we've had that for a while. That is what I am using for my project at
>> Linaro. None of our projects/jobs deliverable are 'public' at the moment,
>> so you won't find them, but i can confirm that what we deploy for our jobs
>> is the output of OE (e.g. what is in the deploy/image folder). hwpack was
>> something mostly meant to make it simpler to work with Ubuntu and Android
>> root FS, and is not suited for OE.
> 
> OK, fair enough - I guess the trouble here is it's hard to find any 
> documentation talking about this functionality; everything I could find that 
> talks about preparing an image refers to needing an hwpack and using linaro-
> media-create. Even mailing list posts that I found (admittedly from last year) 
> referred to these in the context of booting OE images; there are also hwpacks 
> offered for download under "openembedded" directories on releases.linaro.org. 
> Given these I hope you can understand my confusion.
> 
> Generally, if someone with experience of doing so successfully could document 
> how LAVA can be set up to test OE images I think that would be very helpful. I 
> did struggle a bit with the documentation - there are still quite a lot of 
> broken links to the old documentation on readthedocs.org.
> 
>>> * It's not clear to me how well it will work across different host
>>> distributions that we want to support; the main LAVA installer quits if it
>>> isn't run on Ubuntu. For convenience I happened to be using an Ubuntu VM,
>>> and I wasn't running the full installer so I avoided this check
>>> altogether. It just concerns me that such a check would even be there.
>>
>> I agree that this is a problem and we should fix. Our IT infrastructure is
>> largely based on Ubuntu servers, but that should not affect what we create
>> that much.
> 
> Right. FWIW, as you probably know, the way we handle "unsupported" host 
> distros is to allow running on them, but warn the user that they may 
> experience problems (well, in our case we provide this host distro check 
> functionality, it's on in Poky and off in OE-Core by default, not sure if 
> others are using it).
> 
> Cheers,
> Paul
> 

Re: [OE-core] Automated testing on real hardware

[Philip Balister]

On 11/29/2013 01:03 PM, Paul Eggleton wrote:
> Hi Nicolas,
> 
> On Friday 29 November 2013 17:31:40 Nicolas Dechesne wrote:
>> On Fri, Nov 29, 2013 at 4:58 PM, Paul Eggleton <
>> paul.eggleton@linux.intel.com> wrote:
>>> * It requires root to run, since it mounts the images temporarily and
>>> modifies the contents. We've done quite a lot to avoid having to run as
>>> root in OE, so this is a hard sell.
>>
>> i have forwarded this email to the relevant people in Linaro working on
>> LAVA. I hope to be able to bring more information about that, as I am not
>> involved with LAVA, just a 'user' of it. 
> 
> OK, thanks. I'm certainly happy to have a wider discussion on this.

Part of the discussion should be identify issues with projects like LAVA
and see if the LAVA developers agree with our perceptions and work out
if there is a way we can work together to remove the issues. I certainly
see a lot of promise in LAVA, but the comments Paul has are big problems
for the OpenEmbedded Community.

Part of the point of the Yocto Project is to try and focus people's
efforts on improving the embedded process, either by identifying
promising projects and working together to improve them, or by
developing new technology as needed. But before developing something
new, we nee to make sure we are not creating fragmentation in the
embedded space.

Philip

> 
>> By 'root' here, you mean on the server side, e.g. the LAVA instance, which
>> will typically not be the 'builder', or do you want to couple build and test
>> on the same 'server'?
> 
> That's what I've been looking at - just using lava-dispatcher rather than the 
> full LAVA framework. That said, if all it requires root for is mounting the 
> image, modifying it and then unmounting it, to test our images I don't think 
> we even want this; but there appears to be no way to disable it. I then found 
> a launchpad bug where the root requirement was discussed and the accepted 
> solution there was for the dispatcher to just check and error out if not 
> running as root.
> 
>> LAVA jobs are generally submitted by a CI instance (Jenkins in our case).
> 
> Our test code currently queries configuration/image information from the build 
> system, which means that the build system has to be involved in the dispatch 
> process. Our CI solution (the autobuilder) of course runs the build system 
> though. One of the work items in my email is to be able to export the tests, 
> thus decoupling them from this; the build system would still need to be in 
> charge of exporting the tests, but it would allow a test server to manage the 
> deployment, execution and results collection.
>  
>>> * It expects images to be created by linaro-media-create, which in turn
>>> requires an "hwpack" [2]. The hwpack concept seems primarily geared to
>>> Ubuntu and distributions like it where you'd have a generic base image
>>> upon which you'd install some board-specific packages in order to make it
>>> work on the board. OE doesn't work that way - we just build images specific
>>> to the board, which makes this mechanism completely superfluous for our
>>> usage; but the tool appears to require it.
>>
>> that is no longer the case. we do have support for 'native' OE images, and
>> we've had that for a while. That is what I am using for my project at
>> Linaro. None of our projects/jobs deliverable are 'public' at the moment,
>> so you won't find them, but i can confirm that what we deploy for our jobs
>> is the output of OE (e.g. what is in the deploy/image folder). hwpack was
>> something mostly meant to make it simpler to work with Ubuntu and Android
>> root FS, and is not suited for OE.
> 
> OK, fair enough - I guess the trouble here is it's hard to find any 
> documentation talking about this functionality; everything I could find that 
> talks about preparing an image refers to needing an hwpack and using linaro-
> media-create. Even mailing list posts that I found (admittedly from last year) 
> referred to these in the context of booting OE images; there are also hwpacks 
> offered for download under "openembedded" directories on releases.linaro.org. 
> Given these I hope you can understand my confusion.
> 
> Generally, if someone with experience of doing so successfully could document 
> how LAVA can be set up to test OE images I think that would be very helpful. I 
> did struggle a bit with the documentation - there are still quite a lot of 
> broken links to the old documentation on readthedocs.org.
> 
>>> * It's not clear to me how well it will work across different host
>>> distributions that we want to support; the main LAVA installer quits if it
>>> isn't run on Ubuntu. For convenience I happened to be using an Ubuntu VM,
>>> and I wasn't running the full installer so I avoided this check
>>> altogether. It just concerns me that such a check would even be there.
>>
>> I agree that this is a problem and we should fix. Our IT infrastructure is
>> largely based on Ubuntu servers, but that should not affect what we create
>> that much.
> 
> Right. FWIW, as you probably know, the way we handle "unsupported" host 
> distros is to allow running on them, but warn the user that they may 
> experience problems (well, in our case we provide this host distro check 
> functionality, it's on in Poky and off in OE-Core by default, not sure if 
> others are using it).
> 
> Cheers,
> Paul
> 

Re: Automated testing on real hardware

[Fathi Boudra]

Hi,

On 29 November 2013 18:31, Nicolas Dechesne <nicolas.dechesne@linaro.org> wrote:
> Paul,
>
> On Fri, Nov 29, 2013 at 4:58 PM, Paul Eggleton
> <paul.eggleton@linux.intel.com> wrote:
>>
>> LAVA
>> -----
>>
>> A number of people had suggested I look at LAVA [1]. It is split into
>> different
>> components for each function, some of which should be usable independently
>> so
>> you don't have to use the entire framework. Many of the components could
>> be
>> useful assuming they are independent, but in terms of being able to
>> actually
>> run our tests, the one that stands out as being most relevant is lava-
>> dispatcher. This is the component that deploys images, and boots the
>> hardware
>> in order to run the tests.
>>
>> I've looked at lava-dispatcher a couple of times; firstly at the code, and
>> yesterday I looked at actually running it. Initially it looked very
>> promising
>> - reasonably licensed, written in Python, has some rudimentary support for
>> OE
>> images. However while looking at it I found the following:
>>
>> * It requires root to run, since it mounts the images temporarily and
>> modifies
>> the contents. We've done quite a lot to avoid having to run as root in OE,
>> so
>> this is a hard sell.
>
>
> i have forwarded this email to the relevant people in Linaro working on
> LAVA. I hope to be able to bring more information about that, as I am not
> involved with LAVA, just a 'user' of it. By 'root' here, you mean on the
> server side, e.g. the LAVA instance, which will typically not be the
> 'builder', or do you want to couple build and test on the same 'server'?
> LAVA jobs are generally submitted by a CI instance (Jenkins in our case).
>
>>
>> * It expects images to be created by linaro-media-create, which in turn
>> requires an "hwpack" [2]. The hwpack concept seems primarily geared to
>> Ubuntu
>> and distributions like it where you'd have a generic base image upon which
>> you'd install some board-specific packages in order to make it work on the
>> board. OE doesn't work that way - we just build images specific to the
>> board,
>> which makes this mechanism completely superfluous for our usage; but the
>> tool
>> appears to require it.
>
> that is no longer the case. we do have support for 'native' OE images, and
> we've had that for a while.

Well, it has never been the case... We had deploy_linaro_image to
deploy images since day 1, supporting hwpack + rootfs image or
pre-built image.
Support for 'native' OE images as you call it, fall into the pre-built
image method.
It has some assumption since we use our own baked images, but I'm
pretty sure we can fix it if more people are using it for OE testing.

Some other deployment methods have been added like
deploy_linaro_kernel, for network boot:
http://validation.linaro.org/static/docs/dispatcher-actions.html

> That is what I am using for my project at
> Linaro. None of our projects/jobs deliverable are 'public' at the moment, so
> you won't find them, but i can confirm that what we deploy for our jobs is
> the output of OE (e.g. what is in the deploy/image folder). hwpack was
> something mostly meant to make it simpler to work with Ubuntu and Android
> root FS, and is not suited for OE.

It isn't suited for OE is a personal opinion. As a matter of fact,
there's use cases where we need only the rootfs and don't want to
rebuild everything just to change the kernel.

>>
>>
>> * There is a general lack of abstraction and far too much hardcoding. For
>> example, even at the most abstract level, the "Target" class (which is the
>> base class for defining what to do for different types of targets) has
>> hardcoded
>> deploy_linaro / deploy_android functions:
>>
>> https://git.linaro.org/gitweb?p=lava/lava-dispatcher.git;a=blob;f=lava_dispatcher/device/target.py
>>
>> * It's not clear to me how well it will work across different host
>> distributions that we want to support; the main LAVA installer quits if it
>> isn't run on Ubuntu. For convenience I happened to be using an Ubuntu VM,
>> and
>> I wasn't running the full installer so I avoided this check altogether. It
>> just concerns me that such a check would even be there.
>
>
> I agree that this is a problem and we should fix. Our IT infrastructure is
> largely based on Ubuntu servers, but that should not affect what we create
> that much.

There's a work in progress on LAVA deployment. It has been
re-organized to make it easier to package for distributions. LAVA does
work on Debian/Ubuntu/Fedora and packages will be available for those
distro.

>>
>> Of course, none of these problems are impossible to overcome, if we're
>> prepared to put a significant amount of engineering into resolving them.
>> In
>> terms of enabling the Yocto Project QA team to test images on real
>> hardware in
>> the 1.6 development cycle however, I don't believe this is going to be
>> deliverable.

afaict, the non-root requirement is the blocker.
The other issues reported will be resolved as part of our roadmap.

Cheers,
Fathi

Re: [OE-core] Automated testing on real hardware

[Fathi Boudra]

Hi,

On 29 November 2013 18:31, Nicolas Dechesne <nicolas.dechesne@linaro.org> wrote:
> Paul,
>
> On Fri, Nov 29, 2013 at 4:58 PM, Paul Eggleton
> <paul.eggleton@linux.intel.com> wrote:
>>
>> LAVA
>> -----
>>
>> A number of people had suggested I look at LAVA [1]. It is split into
>> different
>> components for each function, some of which should be usable independently
>> so
>> you don't have to use the entire framework. Many of the components could
>> be
>> useful assuming they are independent, but in terms of being able to
>> actually
>> run our tests, the one that stands out as being most relevant is lava-
>> dispatcher. This is the component that deploys images, and boots the
>> hardware
>> in order to run the tests.
>>
>> I've looked at lava-dispatcher a couple of times; firstly at the code, and
>> yesterday I looked at actually running it. Initially it looked very
>> promising
>> - reasonably licensed, written in Python, has some rudimentary support for
>> OE
>> images. However while looking at it I found the following:
>>
>> * It requires root to run, since it mounts the images temporarily and
>> modifies
>> the contents. We've done quite a lot to avoid having to run as root in OE,
>> so
>> this is a hard sell.
>
>
> i have forwarded this email to the relevant people in Linaro working on
> LAVA. I hope to be able to bring more information about that, as I am not
> involved with LAVA, just a 'user' of it. By 'root' here, you mean on the
> server side, e.g. the LAVA instance, which will typically not be the
> 'builder', or do you want to couple build and test on the same 'server'?
> LAVA jobs are generally submitted by a CI instance (Jenkins in our case).
>
>>
>> * It expects images to be created by linaro-media-create, which in turn
>> requires an "hwpack" [2]. The hwpack concept seems primarily geared to
>> Ubuntu
>> and distributions like it where you'd have a generic base image upon which
>> you'd install some board-specific packages in order to make it work on the
>> board. OE doesn't work that way - we just build images specific to the
>> board,
>> which makes this mechanism completely superfluous for our usage; but the
>> tool
>> appears to require it.
>
> that is no longer the case. we do have support for 'native' OE images, and
> we've had that for a while.

Well, it has never been the case... We had deploy_linaro_image to
deploy images since day 1, supporting hwpack + rootfs image or
pre-built image.
Support for 'native' OE images as you call it, fall into the pre-built
image method.
It has some assumption since we use our own baked images, but I'm
pretty sure we can fix it if more people are using it for OE testing.

Some other deployment methods have been added like
deploy_linaro_kernel, for network boot:
http://validation.linaro.org/static/docs/dispatcher-actions.html

> That is what I am using for my project at
> Linaro. None of our projects/jobs deliverable are 'public' at the moment, so
> you won't find them, but i can confirm that what we deploy for our jobs is
> the output of OE (e.g. what is in the deploy/image folder). hwpack was
> something mostly meant to make it simpler to work with Ubuntu and Android
> root FS, and is not suited for OE.

It isn't suited for OE is a personal opinion. As a matter of fact,
there's use cases where we need only the rootfs and don't want to
rebuild everything just to change the kernel.

>>
>>
>> * There is a general lack of abstraction and far too much hardcoding. For
>> example, even at the most abstract level, the "Target" class (which is the
>> base class for defining what to do for different types of targets) has
>> hardcoded
>> deploy_linaro / deploy_android functions:
>>
>> https://git.linaro.org/gitweb?p=lava/lava-dispatcher.git;a=blob;f=lava_dispatcher/device/target.py
>>
>> * It's not clear to me how well it will work across different host
>> distributions that we want to support; the main LAVA installer quits if it
>> isn't run on Ubuntu. For convenience I happened to be using an Ubuntu VM,
>> and
>> I wasn't running the full installer so I avoided this check altogether. It
>> just concerns me that such a check would even be there.
>
>
> I agree that this is a problem and we should fix. Our IT infrastructure is
> largely based on Ubuntu servers, but that should not affect what we create
> that much.

There's a work in progress on LAVA deployment. It has been
re-organized to make it easier to package for distributions. LAVA does
work on Debian/Ubuntu/Fedora and packages will be available for those
distro.

>>
>> Of course, none of these problems are impossible to overcome, if we're
>> prepared to put a significant amount of engineering into resolving them.
>> In
>> terms of enabling the Yocto Project QA team to test images on real
>> hardware in
>> the 1.6 development cycle however, I don't believe this is going to be
>> deliverable.

afaict, the non-root requirement is the blocker.
The other issues reported will be resolved as part of our roadmap.

Cheers,
Fathi

Re: Automated testing on real hardware

[Philip Balister]

On 12/02/2013 05:11 AM, Fathi Boudra wrote:
> Hi,
> 
> On 29 November 2013 18:31, Nicolas Dechesne <nicolas.dechesne@linaro.org> wrote:
>> Paul,
>>
>> On Fri, Nov 29, 2013 at 4:58 PM, Paul Eggleton
>> <paul.eggleton@linux.intel.com> wrote:
>>>
>>> LAVA
>>> -----
>>>
>>> A number of people had suggested I look at LAVA [1]. It is split into
>>> different
>>> components for each function, some of which should be usable independently
>>> so
>>> you don't have to use the entire framework. Many of the components could
>>> be
>>> useful assuming they are independent, but in terms of being able to
>>> actually
>>> run our tests, the one that stands out as being most relevant is lava-
>>> dispatcher. This is the component that deploys images, and boots the
>>> hardware
>>> in order to run the tests.
>>>
>>> I've looked at lava-dispatcher a couple of times; firstly at the code, and
>>> yesterday I looked at actually running it. Initially it looked very
>>> promising
>>> - reasonably licensed, written in Python, has some rudimentary support for
>>> OE
>>> images. However while looking at it I found the following:
>>>
>>> * It requires root to run, since it mounts the images temporarily and
>>> modifies
>>> the contents. We've done quite a lot to avoid having to run as root in OE,
>>> so
>>> this is a hard sell.
>>
>>
>> i have forwarded this email to the relevant people in Linaro working on
>> LAVA. I hope to be able to bring more information about that, as I am not
>> involved with LAVA, just a 'user' of it. By 'root' here, you mean on the
>> server side, e.g. the LAVA instance, which will typically not be the
>> 'builder', or do you want to couple build and test on the same 'server'?
>> LAVA jobs are generally submitted by a CI instance (Jenkins in our case).
>>
>>>
>>> * It expects images to be created by linaro-media-create, which in turn
>>> requires an "hwpack" [2]. The hwpack concept seems primarily geared to
>>> Ubuntu
>>> and distributions like it where you'd have a generic base image upon which
>>> you'd install some board-specific packages in order to make it work on the
>>> board. OE doesn't work that way - we just build images specific to the
>>> board,
>>> which makes this mechanism completely superfluous for our usage; but the
>>> tool
>>> appears to require it.
>>
>> that is no longer the case. we do have support for 'native' OE images, and
>> we've had that for a while.
> 
> Well, it has never been the case... We had deploy_linaro_image to
> deploy images since day 1, supporting hwpack + rootfs image or
> pre-built image.
> Support for 'native' OE images as you call it, fall into the pre-built
> image method.
> It has some assumption since we use our own baked images, but I'm
> pretty sure we can fix it if more people are using it for OE testing.
> 
> Some other deployment methods have been added like
> deploy_linaro_kernel, for network boot:
> http://validation.linaro.org/static/docs/dispatcher-actions.html
> 
>> That is what I am using for my project at
>> Linaro. None of our projects/jobs deliverable are 'public' at the moment, so
>> you won't find them, but i can confirm that what we deploy for our jobs is
>> the output of OE (e.g. what is in the deploy/image folder). hwpack was
>> something mostly meant to make it simpler to work with Ubuntu and Android
>> root FS, and is not suited for OE.
> 
> It isn't suited for OE is a personal opinion. As a matter of fact,
> there's use cases where we need only the rootfs and don't want to
> rebuild everything just to change the kernel.

Is there an example of using LAVA to test an OE rootfs with qemu (or
other form of vm). I've installed LAVA on a spare PC, but the install
has no sample test configs :)

It would be really helpful to ahve an example, prefably something OE
based, to try out.

Philip

> 
>>>
>>>
>>> * There is a general lack of abstraction and far too much hardcoding. For
>>> example, even at the most abstract level, the "Target" class (which is the
>>> base class for defining what to do for different types of targets) has
>>> hardcoded
>>> deploy_linaro / deploy_android functions:
>>>
>>> https://git.linaro.org/gitweb?p=lava/lava-dispatcher.git;a=blob;f=lava_dispatcher/device/target.py
>>>
>>> * It's not clear to me how well it will work across different host
>>> distributions that we want to support; the main LAVA installer quits if it
>>> isn't run on Ubuntu. For convenience I happened to be using an Ubuntu VM,
>>> and
>>> I wasn't running the full installer so I avoided this check altogether. It
>>> just concerns me that such a check would even be there.
>>
>>
>> I agree that this is a problem and we should fix. Our IT infrastructure is
>> largely based on Ubuntu servers, but that should not affect what we create
>> that much.
> 
> There's a work in progress on LAVA deployment. It has been
> re-organized to make it easier to package for distributions. LAVA does
> work on Debian/Ubuntu/Fedora and packages will be available for those
> distro.
> 
>>>
>>> Of course, none of these problems are impossible to overcome, if we're
>>> prepared to put a significant amount of engineering into resolving them.
>>> In
>>> terms of enabling the Yocto Project QA team to test images on real
>>> hardware in
>>> the 1.6 development cycle however, I don't believe this is going to be
>>> deliverable.
> 
> afaict, the non-root requirement is the blocker.
> The other issues reported will be resolved as part of our roadmap.
> 
> Cheers,
> Fathi
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core
> 
> 

Re: [OE-core] Automated testing on real hardware

[Philip Balister]

On 12/02/2013 05:11 AM, Fathi Boudra wrote:
> Hi,
> 
> On 29 November 2013 18:31, Nicolas Dechesne <nicolas.dechesne@linaro.org> wrote:
>> Paul,
>>
>> On Fri, Nov 29, 2013 at 4:58 PM, Paul Eggleton
>> <paul.eggleton@linux.intel.com> wrote:
>>>
>>> LAVA
>>> -----
>>>
>>> A number of people had suggested I look at LAVA [1]. It is split into
>>> different
>>> components for each function, some of which should be usable independently
>>> so
>>> you don't have to use the entire framework. Many of the components could
>>> be
>>> useful assuming they are independent, but in terms of being able to
>>> actually
>>> run our tests, the one that stands out as being most relevant is lava-
>>> dispatcher. This is the component that deploys images, and boots the
>>> hardware
>>> in order to run the tests.
>>>
>>> I've looked at lava-dispatcher a couple of times; firstly at the code, and
>>> yesterday I looked at actually running it. Initially it looked very
>>> promising
>>> - reasonably licensed, written in Python, has some rudimentary support for
>>> OE
>>> images. However while looking at it I found the following:
>>>
>>> * It requires root to run, since it mounts the images temporarily and
>>> modifies
>>> the contents. We've done quite a lot to avoid having to run as root in OE,
>>> so
>>> this is a hard sell.
>>
>>
>> i have forwarded this email to the relevant people in Linaro working on
>> LAVA. I hope to be able to bring more information about that, as I am not
>> involved with LAVA, just a 'user' of it. By 'root' here, you mean on the
>> server side, e.g. the LAVA instance, which will typically not be the
>> 'builder', or do you want to couple build and test on the same 'server'?
>> LAVA jobs are generally submitted by a CI instance (Jenkins in our case).
>>
>>>
>>> * It expects images to be created by linaro-media-create, which in turn
>>> requires an "hwpack" [2]. The hwpack concept seems primarily geared to
>>> Ubuntu
>>> and distributions like it where you'd have a generic base image upon which
>>> you'd install some board-specific packages in order to make it work on the
>>> board. OE doesn't work that way - we just build images specific to the
>>> board,
>>> which makes this mechanism completely superfluous for our usage; but the
>>> tool
>>> appears to require it.
>>
>> that is no longer the case. we do have support for 'native' OE images, and
>> we've had that for a while.
> 
> Well, it has never been the case... We had deploy_linaro_image to
> deploy images since day 1, supporting hwpack + rootfs image or
> pre-built image.
> Support for 'native' OE images as you call it, fall into the pre-built
> image method.
> It has some assumption since we use our own baked images, but I'm
> pretty sure we can fix it if more people are using it for OE testing.
> 
> Some other deployment methods have been added like
> deploy_linaro_kernel, for network boot:
> http://validation.linaro.org/static/docs/dispatcher-actions.html
> 
>> That is what I am using for my project at
>> Linaro. None of our projects/jobs deliverable are 'public' at the moment, so
>> you won't find them, but i can confirm that what we deploy for our jobs is
>> the output of OE (e.g. what is in the deploy/image folder). hwpack was
>> something mostly meant to make it simpler to work with Ubuntu and Android
>> root FS, and is not suited for OE.
> 
> It isn't suited for OE is a personal opinion. As a matter of fact,
> there's use cases where we need only the rootfs and don't want to
> rebuild everything just to change the kernel.

Is there an example of using LAVA to test an OE rootfs with qemu (or
other form of vm). I've installed LAVA on a spare PC, but the install
has no sample test configs :)

It would be really helpful to ahve an example, prefably something OE
based, to try out.

Philip

> 
>>>
>>>
>>> * There is a general lack of abstraction and far too much hardcoding. For
>>> example, even at the most abstract level, the "Target" class (which is the
>>> base class for defining what to do for different types of targets) has
>>> hardcoded
>>> deploy_linaro / deploy_android functions:
>>>
>>> https://git.linaro.org/gitweb?p=lava/lava-dispatcher.git;a=blob;f=lava_dispatcher/device/target.py
>>>
>>> * It's not clear to me how well it will work across different host
>>> distributions that we want to support; the main LAVA installer quits if it
>>> isn't run on Ubuntu. For convenience I happened to be using an Ubuntu VM,
>>> and
>>> I wasn't running the full installer so I avoided this check altogether. It
>>> just concerns me that such a check would even be there.
>>
>>
>> I agree that this is a problem and we should fix. Our IT infrastructure is
>> largely based on Ubuntu servers, but that should not affect what we create
>> that much.
> 
> There's a work in progress on LAVA deployment. It has been
> re-organized to make it easier to package for distributions. LAVA does
> work on Debian/Ubuntu/Fedora and packages will be available for those
> distro.
> 
>>>
>>> Of course, none of these problems are impossible to overcome, if we're
>>> prepared to put a significant amount of engineering into resolving them.
>>> In
>>> terms of enabling the Yocto Project QA team to test images on real
>>> hardware in
>>> the 1.6 development cycle however, I don't believe this is going to be
>>> deliverable.
> 
> afaict, the non-root requirement is the blocker.
> The other issues reported will be resolved as part of our roadmap.
> 
> Cheers,
> Fathi
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core
> 
>