Re: [PATCH v2 0/4] raisin: introduce tests

[George Dunlap <george.dunlap@eu.citrix.com>]

On 05/06/2015 03:46 PM, Ian Campbell wrote:
> On Wed, 2015-05-06 at 15:39 +0100, Stefano Stabellini wrote:
>> On Wed, 6 May 2015, George Dunlap wrote:
>>> On Tue, May 5, 2015 at 2:55 PM, Stefano Stabellini
>>> <stefano.stabellini@eu.citrix.com> wrote:
>>>> On Tue, 5 May 2015, Ian Campbell wrote:
>>>>> On Fri, 2015-05-01 at 16:48 +0100, Stefano Stabellini wrote:
>>>>>> Hi all,
>>>>>>
>>>>>> this patch series introduces a framework to execute simple unit and
>>>>>> functional tests in raisin. It can be used by developers to validate
>>>>>> their changes before submitting a patch series to xen-devel. It can also
>>>>>> be used by OSSTest to test for regressions on one particular
>>>>>> functionality. This patch series only introduces two tests: a PV guest
>>>>>> creation test and an HVM guest creation test. They are both based on
>>>>>> busybox. More tests will follow.
>>>>>
>>>>> What is the intended scope of these tests? e.g. when should a test be
>>>>> added here rather than to osstest?
>>>>
>>>> Small functional tests that can be easily run on a single host, without
>>>> requiring a specific host or guest operating system. Tests that every
>>>> developer should run on their test machine before submitting a patch
>>>> series.
>>>
>>> So a sort of BVT (build verification test), such that that we could be
>>> reasonably annoyed at if an experienced developer submitted a patch
>>> that broke said functionality?
>>
>> That's right.
>>
>> In general any functionality tests, that can be added without
>> introducing too much complexity, should be in raisin. osstest will be
>> able to run these tests via raisin.
> 
> I think if it is intended as a BVT which I'll get shouted out for not
> having run then it needs to be time bound as much as functionality
> bound.

If we start to use raisin as the repository for a lot of the actual
basic functionality tests which are now in osstest, then yeah, there
should be a smallish subset that we might expect people to run.

OTOH, do we really have a problem with people breaking things
accidentally at the moment?  Does it make sense to impose the cultural
expectation of running the BVT before every submission, when in the vast
majority of cases developers are able to decide for themselves what
testing needs to be done?

Maybe reviewers / maintainers might decide require assurance of having
run a certain subset of the tests, based on the particular complexity of
the patch / code being modified?

 -George