Re: Call to write tests for osstest

[Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>]

On Thu, Aug 08, 2013 at 07:32:21PM +0200, Sander Eikelenboom wrote:
> 
> Thursday, August 8, 2013, 6:01:01 PM, you wrote:
> 
> > As release coordinator, and therefore de-facto
> > tracker-of-bugs-and-regressions, it seems to me that one of the
> > shortcomings in our current development mode is a lack of good
> > regression testing for many of the less common, but still very
> > important features of Xen; things like S3, driver domains, nested
> > virt, and so on.
> 
> > We do have a regression-testing push gate for the xen trees, written
> > by Ian Jackson, called osstest, with a lot of great functionality,
> > including a test scheduler, an automatic Baysean bisector, and so on.
> > Apart from more hardware, it is mainly lacking a more complete set of
> > tests.
> 
> > The Xen Project team here at Citrix agreed with me, and have decided
> > during the 4.4 timeframe to take on the task of adding some important
> > functional tests to osstest; our list is below.  We'd love for you to
> > join us.
> 
> > I will be tracking the implementation of the tests as part of the
> > regular 4.4 release updates.  I encourage people to join us by
> > thinking about a particular feature or bit of functionality that is
> > not yet tested by osstest, which you would like to see implemented in
> > the 4.4 timeframe, and responding to this e-mail, or one of the
> > regular development update e-mails asking it to be added.
> 
> > There is a description of osstest, along with a link to the source code, here:
> 
> > http://blog.xen.org/index.php/2013/02/02/xen-automatic-test-system-osstest/
> 
> > === Testing coverage ===
> 
> > * Network driver domains
> >  @George
> 
> > * new libxl w/ previous versions of xl
> >  @IanJ
> 
> > * Host S3 suspend
> >  @bguthro?
> 
> > * Default [example] XSM policy
> >  @Stefano to ask Daniel D
> 
> > * Xen on ARM
> >  # problem ATM: hardware
> >   @ianc
> >    emulator: @stefano to think about it
> 
> > * Storage driver domains
> >  @roger
> 
> > * HVM pci passthrough
> >  @anthony
> 
> > * Nested virt?
> >  @intel (chased by George)
> 
> > * Fix SRIOV test (chase intel)
> >  @ianj
> 
> > * Fix bisector to e-mail blame-worthy parties
> >  @ianj
> 
> > * Fix xl shutdown
> >   @ianj
> 
> :-)
> 
> > * stub domains
> >   @athony
> 
> Some ideas, perhaps only with a xen version after a push (so Xen shouldn't be directly to blame) :
> 
> * More current kernels:
>    - latest stable ?

No. Too much to chase.

>    - linux-next ? (although perhaps only just before merge window, to prevent spending to much time in breaking on other random kernel stuff)

No. Just tip/tip.git. That is the x86 folks tip. That had caused
headaches from us in the past.

>    - linus's tree ?

Yes.
>    - xen kernel tree's next branch

Yes. That is the xen/tip.git which has the Xen generic, x86 and ARM
subsystems all in one.
> 
> * Some performance testing (network, block, cpu/mem/fork/real apps benchmarks, some other metrics)
>    - perhaps makes separate graphs of these, so one can see performance increase or decrease over a larger time frame, and see at around what commits that occurred.

That would be nice.
>    - only after all basic tests succeeded and a push was done.
> 
> --
> Sander
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel