Re: [PATCH v2 0/4] raisin: introduce tests

[Ian Campbell <ian.campbell@citrix.com>]

On Wed, 2015-05-06 at 15:39 +0100, Stefano Stabellini wrote:
> On Wed, 6 May 2015, George Dunlap wrote:
> > On Tue, May 5, 2015 at 2:55 PM, Stefano Stabellini
> > <stefano.stabellini@eu.citrix.com> wrote:
> > > On Tue, 5 May 2015, Ian Campbell wrote:
> > >> On Fri, 2015-05-01 at 16:48 +0100, Stefano Stabellini wrote:
> > >> > Hi all,
> > >> >
> > >> > this patch series introduces a framework to execute simple unit and
> > >> > functional tests in raisin. It can be used by developers to validate
> > >> > their changes before submitting a patch series to xen-devel. It can also
> > >> > be used by OSSTest to test for regressions on one particular
> > >> > functionality. This patch series only introduces two tests: a PV guest
> > >> > creation test and an HVM guest creation test. They are both based on
> > >> > busybox. More tests will follow.
> > >>
> > >> What is the intended scope of these tests? e.g. when should a test be
> > >> added here rather than to osstest?
> > >
> > > Small functional tests that can be easily run on a single host, without
> > > requiring a specific host or guest operating system. Tests that every
> > > developer should run on their test machine before submitting a patch
> > > series.
> > 
> > So a sort of BVT (build verification test), such that that we could be
> > reasonably annoyed at if an experienced developer submitted a patch
> > that broke said functionality?
> 
> That's right.
> 
> In general any functionality tests, that can be added without
> introducing too much complexity, should be in raisin. osstest will be
> able to run these tests via raisin.

I think if it is intended as a BVT which I'll get shouted out for not
having run then it needs to be time bound as much as functionality
bound.

Ian.

> More complex tests, involving more than one host, or specific guest/host
> OS combinations, should only be in osstest.