Now available: xm-test-0.2.0

Now available: xm-test-0.2.0

[Dan Smith]

I have just posted version 0.2.0 of xm-test here:

  http://static.danplanet.com/xm-test/xm-test-0.2.0.tar.gz

New in this version:

 - Automatic submission of reports
 - New tests:
    - enforce_dom0_cpus
    - concurrent domu tests
    - additional checks for *some* stale-state issues

We would really like to get people running xm-test on a regular basis,
so that we can start to get some ideas about regression trends from
the submitted data.

* For those who are not familiar with xm-test:

Several of us here at IBM have been working on a framework for testing
the xen tools, specifically xm.  Our goal is to provide a way for
developers to _easily_ write tests for new and existing xm commands.
We believe that such a test suite will help reduce breakages in the
user-facing tools when developers modify xm and/or xend.

We would like some feedback from the community on the usefulness of
our framework, in hopes that it might be hosted by xensource so that
everyone can contribute tests to help harden xm and xend.

* Details:

The framework tests (as well as the support libraries) are written in
python, which are executed by the standard automake "make check"
facilities.  We build a standardized ramdisk that can be used for
portable test writing, therefore reducing dependencies on the test
host machine.

The framework library provides several abstractions to make common and
complex tasks easier for the test writer.  For example, we provide a
domain and console abstraction that allows a test writer to start a
domU and execute arbitrary commands, retrieving the status and output
of each.  This allows a decent amount of automation for verifying that
(for example) "xm sysrq mydomain s" actually sent the sysrq.

-- 
Dan Smith
IBM Linux Technology Center
Open Hypervisor Team
email: danms@us.ibm.com

Re: Now available: xm-test-0.2.0

[Ted Kaczmarek]

On Mon, 2005-10-03 at 15:52 -0700, Dan Smith wrote:
> I have just posted version 0.2.0 of xm-test here:
> 
>   http://static.danplanet.com/xm-test/xm-test-0.2.0.tar.gz
> 
> New in this version:
> 
>  - Automatic submission of reports
>  - New tests:
>     - enforce_dom0_cpus
>     - concurrent domu tests
>     - additional checks for *some* stale-state issues

I got up to 

12_create_concurrent_stress_pos.test

saw 

trying to free up IRQ281

screaming by in console, machine was totally unresponsive.

Tyan 2462 SMP , 2 gigs, FC4 dom0

Regards,
Ted

Re: Now available: xm-test-0.2.0

[Hollis Blanchard]

On Monday 03 October 2005 17:52, Dan Smith wrote:
>
> We would like some feedback from the community on the usefulness of
> our framework, in hopes that it might be hosted by xensource so that
> everyone can contribute tests to help harden xm and xend.

Building xm-test takes a very long time, because among other things it takes 
it upon itself to download and build its very own toolchain. That is 
extremely silly; please have it use the existing toolchain instead.

I'm using qemu, so I want to run the tests on a system other than the one I 
built xm-test on (building a toolchain under qemu does not sound fun or 
necessary). How can I do this? Right now I'm getting obnoxious autoconf 
messages like this:
make: *** No rule to make target `../configure.ac', needed by `Makefile.in'.  
Stop.
I have exported the tests/ directory to qemu, and am trying to run "make 
check".

-- 
Hollis Blanchard
IBM Linux Technology Center

Re: Now available: xm-test-0.2.0

[Dan Smith]

HB> Building xm-test takes a very long time, because among other
HB> things it takes it upon itself to download and build its very own
HB> toolchain.

FYI, I've only built xm-test once.  See below.

HB> That is extremely silly; please have it use the existing toolchain
HB> instead.

It builds its own toolchain because it uses uClibc.  This allows us to
compile some utilities and have them all fit nicely inside a
ramdisk-sized image.

HB> I'm using qemu, so I want to run the tests on a system other than
HB> the one I built xm-test on (building a toolchain under qemu does
HB> not sound fun or necessary). How can I do this?

On your fastest machine, allow xm-test to build itself by running
'make' in the toplevel.  The end result will be an initrd.img file in
the ramdisk/ directory.

Unpack a fresh copy of xm-test wherever you want to run it from, and
copy the initrd.img file into the ramdisk/ directory.  Then, you can
run 'make check' (or preferably ./runtest.sh) without the lengthy
build process.

At this point, you never have to build xm-test again.  Just copy
around the initrd.img file to anywhere you run it.

HB> I have exported the tests/ directory to
HB> qemu, and am trying to run "make check".

You can't just use the tests/ directory on its own, you need the whole
thing.  There are other things provided that are not below tests/.

-- 
Dan Smith
IBM Linux Technology Center
Open Hypervisor Team
email: danms@us.ibm.com

Re: Now available: xm-test-0.2.0

[Hollis Blanchard]

On Friday 07 October 2005 12:38, Hollis Blanchard wrote:
> On Monday 03 October 2005 17:52, Dan Smith wrote:
> > We would like some feedback from the community on the usefulness of
> > our framework, in hopes that it might be hosted by xensource so that
> > everyone can contribute tests to help harden xm and xend.
>
> Building xm-test takes a very long time, because among other things it
> takes it upon itself to download and build its very own toolchain. That is
> extremely silly; please have it use the existing toolchain instead.

To summarize IRC conversation:
- xm-test creates an initrd that's used to boot the test DomUs.
- That initrd is created with buildroot[1], which uses uClibc, which requires 
building a whole new toolchain to use.
- If they know how, users can manually copy the initrd from one xm-test 
directory to another, avoiding the need to rebuild it.
- The initrd is plain busybox, which could easily be statically linked with 
the user's GNU libc.
- In the future, people want to install other tools (like e2fsprogs) into the 
ramdisk. These could also be statically linked with GNU libc.
- Other than statically linking, the user's libc.so could be copied to the 
initrd, which doesn't make for a very reliable testing environment.
- Using uClibc makes the initrd smaller, which is of dubious value in this 
environment.
- initrd images are limited in size, while initramfs images are not.

[1] http://buildroot.uclibc.org/

-- 
Hollis Blanchard
IBM Linux Technology Center

Re: Now available: xm-test-0.2.0

[Sean Dague]

[-- Attachment #1.1: Type: text/plain, Size: 4274 bytes --]

On Fri, Oct 07, 2005 at 02:40:27PM -0500, Hollis Blanchard wrote:
> On Friday 07 October 2005 12:38, Hollis Blanchard wrote:
> > On Monday 03 October 2005 17:52, Dan Smith wrote:
> > > We would like some feedback from the community on the usefulness of
> > > our framework, in hopes that it might be hosted by xensource so that
> > > everyone can contribute tests to help harden xm and xend.
> >
> > Building xm-test takes a very long time, because among other things it
> > takes it upon itself to download and build its very own toolchain. That is
> > extremely silly; please have it use the existing toolchain instead.
> 
> To summarize IRC conversation:

Some responses, as I think your summary has a lot of personal commentary,
and I want to make sure another perspective is stated.

> - xm-test creates an initrd that's used to boot the test DomUs.
> - That initrd is created with buildroot[1], which uses uClibc, which requires 
> building a whole new toolchain to use.

Corrent, the initrd uses buildroot, which is a generic tool for building
embedded Linux system images.  It is open source, easy to extend, and
brought to you by the same people that brought you busybox & uClibc.

> - If they know how, users can manually copy the initrd from one xm-test 
> directory to another, avoiding the need to rebuild it.

Correct.  Even if they don't know the build process is 100% automated, it
just takes time.  Given then number of people on xen-devel running gentoo,
not sure why an "ug, it has to compile?" sentiment kept coming up on IRC. ;)

> - The initrd is plain busybox, which could easily be statically linked with 
> the user's GNU libc.

For now, we are only 6 weeks and 80 tests into this, and don't even have
full coverage for xm user commands at this point.

> - In the future, people want to install other tools (like e2fsprogs) into the 
> ramdisk. These could also be statically linked with GNU libc.

Niv talked about hping, and other more advanced network tools.  Saying
everything needs to be statically linked is sort of a bear for adding new
bits.  There is already a very modular way to add things to buildroot.  We
can use that our play a NIH game and reinvent it for static binaries.  I
prefer to stand on the shoulders of those that came before me.

> - Other than statically linking, the user's libc.so could be copied to the 
> initrd, which doesn't make for a very reliable testing environment.

This is a retched idea.  We actually did this in systemimager, and then you
start finding all the joys of the differences in glibc between distros and
versions, and have to go through the joys of mklibs (a debian shared library
dependency tool) to ensure you've copied over all the right bits.  It really
sucks, and breaks quite a bit.

> - Using uClibc makes the initrd smaller, which is of dubious value in this 
> environment.

Unless you use it for scale out testing (i.e. run 100 DomU).  Then the
memory size is important as your root is in memory.  I think Dan was able to
start just under 100 DomU with xm-test on his 2 GB workstation (kinda cool
:))

> - initrd images are limited in size, while initramfs images are not.

Yep, initramfs is a good thing to look into.  We did initrds because we'd
all done initrds before, and they were a well known quantity. :)

While some heathly debate on any subject is always a good thing, it is worth
noting we tried a lot of these other approaches to begin with, and none of
them really panned out in a useful way.  We could spend all our time
creating the *uber static link initrd*, but that's the least interesting
part of xm-test. :)  The actual tests are where we need to focus. 
Optimizing the plumbing can come post 3.0.

	-Sean

-- 
__________________________________________________________________

Sean Dague                                       Mid-Hudson Valley
sean at dague dot net                            Linux Users Group
http://dague.net                                 http://mhvlug.org

There is no silver bullet.  Plus, werewolves make better neighbors
than zombies, and they tend to keep the vampire population down.
__________________________________________________________________

[-- Attachment #1.2: Type: application/pgp-signature, Size: 189 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel