Re: [PATCH v2 0/4] raisin: introduce tests

[Ian Campbell <ian.campbell@citrix.com>]

On Wed, 2015-05-06 at 16:11 +0100, George Dunlap wrote:
> On 05/06/2015 03:46 PM, Ian Campbell wrote:
> > On Wed, 2015-05-06 at 15:39 +0100, Stefano Stabellini wrote:
> >> On Wed, 6 May 2015, George Dunlap wrote:
> >>> On Tue, May 5, 2015 at 2:55 PM, Stefano Stabellini
> >>> <stefano.stabellini@eu.citrix.com> wrote:
> >>>> On Tue, 5 May 2015, Ian Campbell wrote:
> >>>>> On Fri, 2015-05-01 at 16:48 +0100, Stefano Stabellini wrote:
> >>>>>> Hi all,
> >>>>>>
> >>>>>> this patch series introduces a framework to execute simple unit and
> >>>>>> functional tests in raisin. It can be used by developers to validate
> >>>>>> their changes before submitting a patch series to xen-devel. It can also
> >>>>>> be used by OSSTest to test for regressions on one particular
> >>>>>> functionality. This patch series only introduces two tests: a PV guest
> >>>>>> creation test and an HVM guest creation test. They are both based on
> >>>>>> busybox. More tests will follow.
> >>>>>
> >>>>> What is the intended scope of these tests? e.g. when should a test be
> >>>>> added here rather than to osstest?
> >>>>
> >>>> Small functional tests that can be easily run on a single host, without
> >>>> requiring a specific host or guest operating system. Tests that every
> >>>> developer should run on their test machine before submitting a patch
> >>>> series.
> >>>
> >>> So a sort of BVT (build verification test), such that that we could be
> >>> reasonably annoyed at if an experienced developer submitted a patch
> >>> that broke said functionality?
> >>
> >> That's right.
> >>
> >> In general any functionality tests, that can be added without
> >> introducing too much complexity, should be in raisin. osstest will be
> >> able to run these tests via raisin.
> > 
> > I think if it is intended as a BVT which I'll get shouted out for not
> > having run then it needs to be time bound as much as functionality
> > bound.
> 
> If we start to use raisin as the repository for a lot of the actual
> basic functionality tests which are now in osstest, then yeah, there
> should be a smallish subset that we might expect people to run.
> 
> OTOH, do we really have a problem with people breaking things
> accidentally at the moment?

I don't think we do really.

>   Does it make sense to impose the cultural
> expectation of running the BVT before every submission, when in the vast
> majority of cases developers are able to decide for themselves what
> testing needs to be done?
> 
> Maybe reviewers / maintainers might decide require assurance of having
> run a certain subset of the tests, based on the particular complexity of
> the patch / code being modified?
> 
>  -George