osstest - howTo

osstest - howTo

[Mihai Bucicoiu]

Hi,

I'm trying to figure out how can Xen be tested. As far as I could find 
out there is a testing framework that can be used, called osstest. [1]

But this, if I got it right,this requires a reboot of the tested host, 
while we need to test Xen without rebooting it. There seems also to be a 
local version ("OSSTEST_JOB=<job> ./ts-debian-install host=bedbug" from 
README) but it only tests if a VM can be started.

I could also not find a list of what is actually tested. Is there such I 
think?

I know that my questions might be a little bit too general and I thank 
you for your replies.

[1] 
http://blog.xen.org/index.php/2013/02/02/xen-automatic-test-system-osstest/

Regards,
Mihai Bucicoiu

Re: osstest - howTo

[George Dunlap]

On Mon, Jul 15, 2013 at 1:08 PM, Mihai Bucicoiu
<mihai.bucicoiu@trust.cased.de> wrote:
> Hi,
>
> I'm trying to figure out how can Xen be tested. As far as I could find out
> there is a testing framework that can be used, called osstest. [1]

What are you trying to accomplish with the testing?

> But this, if I got it right,this requires a reboot of the tested host, while
> we need to test Xen without rebooting it. There seems also to be a local
> version ("OSSTEST_JOB=<job> ./ts-debian-install host=bedbug" from README)
> but it only tests if a VM can be started.

Why do you not want to reboot the host?

 -George

Re: osstest - howTo

[Mihai Bucicoiu]

On 7/15/2013 4:33 PM, George Dunlap wrote:
> On Mon, Jul 15, 2013 at 1:08 PM, Mihai Bucicoiu
> <mihai.bucicoiu@trust.cased.de> wrote:
>> Hi,
>>
>> I'm trying to figure out how can Xen be tested. As far as I could find out
>> there is a testing framework that can be used, called osstest. [1]
> What are you trying to accomplish with the testing?
This is a very good question that I don't really have an answer to it.
>> But this, if I got it right,this requires a reboot of the tested host, while
>> we need to test Xen without rebooting it. There seems also to be a local
>> version ("OSSTEST_JOB=<job> ./ts-debian-install host=bedbug" from README)
>> but it only tests if a VM can be started.
> Why do you not want to reboot the host?
We are building a live update system for Xen, and after the patch is 
applied we do not want to reboot, we want to test if everything is still 
functional. We tested our system in the sense the the VMs are still 
running and the services (we only tried a simple web server) inside the 
VM still run. New VMs can also be started. But the question is are the 
other services that Xen provide still available (e.g., can dom0 do all 
the hypercalls in a proper way).

- Mihai

Re: osstest - howTo

[Wei Liu]

On Tue, Jul 16, 2013 at 11:08:34AM +0200, Mihai Bucicoiu wrote:
> On 7/15/2013 4:33 PM, George Dunlap wrote:
> >On Mon, Jul 15, 2013 at 1:08 PM, Mihai Bucicoiu
> ><mihai.bucicoiu@trust.cased.de> wrote:
> >>Hi,
> >>
> >>I'm trying to figure out how can Xen be tested. As far as I could find out
> >>there is a testing framework that can be used, called osstest. [1]
> >What are you trying to accomplish with the testing?
> This is a very good question that I don't really have an answer to it.
> >>But this, if I got it right,this requires a reboot of the tested host, while
> >>we need to test Xen without rebooting it. There seems also to be a local
> >>version ("OSSTEST_JOB=<job> ./ts-debian-install host=bedbug" from README)
> >>but it only tests if a VM can be started.
> >Why do you not want to reboot the host?
> We are building a live update system for Xen, and after the patch is

Just courious, are you talking about patching Xen hypervisor on the fly?


Wei.

> applied we do not want to reboot, we want to test if everything is
> still functional. We tested our system in the sense the the VMs are
> still running and the services (we only tried a simple web server)
> inside the VM still run. New VMs can also be started. But the
> question is are the other services that Xen provide still available
> (e.g., can dom0 do all the hypercalls in a proper way).
> 
> - Mihai
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel

Re: osstest - howTo

[Mihai Bucicoiu]

On 7/16/2013 11:14 AM, Wei Liu wrote:
> Just courious, are you talking about patching Xen hypervisor on the fly?
Yes.

Mihai

Re: osstest - howTo

[George Dunlap]

On Tue, Jul 16, 2013 at 10:16 AM, Mihai Bucicoiu
<mihai.bucicoiu@trust.cased.de> wrote:
> On 7/16/2013 11:14 AM, Wei Liu wrote:
>>
>> Just courious, are you talking about patching Xen hypervisor on the fly?
>
> Yes.

That sounds awesome.  When can we expect patches? :-)

I haven't taken a look at osstest yet (on my list of things to do in
the next month or two), but it seems like the easiest way to use it to
accomplish what you want is to add a a line to your local RC scripts
to apply the patch on boot before networking is brought up.  Then all
the tests will be run patched.

 -George

Re: osstest - howTo

[Ian Campbell]

On Mon, 2013-07-15 at 14:08 +0200, Mihai Bucicoiu wrote:
> Hi,
> 
> I'm trying to figure out how can Xen be tested. As far as I could find 
> out there is a testing framework that can be used, called osstest. [1]
> 
> But this, if I got it right,this requires a reboot of the tested host, 
> while we need to test Xen without rebooting it. There seems also to be a 
> local version ("OSSTEST_JOB=<job> ./ts-debian-install host=bedbug" from 
> README) but it only tests if a VM can be started.

There are many ./ts-FOO scripts which you can run by hand as you wish,
e.g.
        OSSTEST_JOB=test-amd64-amd64-pv ./ts-host-install host=h09
        OSSTEST_JOB=test-amd64-amd64-pv ./ts-xen-install host=h09
        OSSTEST_JOB=test-amd64-amd64-pv ./ts-host-reboot host=h09
        OSSTEST_JOB=test-amd64-amd64-pv ./ts-debian-install host=h09
        OSSTEST_JOB=test-amd64-amd64-pv ./ts-debian-fixup host=h09 debian 
        OSSTEST_JOB=test-amd64-amd64-pv ./ts-guest-start host=h09 debian

etc.

> I could also not find a list of what is actually tested. Is there such I 
> think?

I think this is more-or-less defined by the "make-flight" script. It's
probably easiest to just look at the results page from any of the test
reports[0]. Each lines of the grid corresponds to a ./ts-* script, I
think.

Ian.

Re: osstest - howTo

[Ian Jackson]

Ian Campbell writes ("Re: [Xen-devel] osstest - howTo"):
> On Mon, 2013-07-15 at 14:08 +0200, Mihai Bucicoiu wrote:
> > I could also not find a list of what is actually tested. Is there such I 
> > think?
> 
> I think this is more-or-less defined by the "make-flight" script.

Yes.  That defines the tests ("jobs" in osstest-speak), ie the columns
in the html grid.

> It's
> probably easiest to just look at the results page from any of the test
> reports[0].

Or if you run it standalone, the set of tests appears in the ad-hoc
sqlite database it creates in your cwd.

> Each lines of the grid corresponds to a ./ts-* script, I
> think.

Yes.  Strictly speaking, each line corresponds to running one of those
scripts with particular arguments - some of them are run more than
once per test.

Each job is specified by make-flight to use a particular "recipe".
The set of scripts and arguments to use for each recipe is defined in
sg-run-job.  Look after the divider comment "test recipes".

Ian.