Re: [PATCH 0/2] tests/acceptance: Add boot vmlinux test

["Alex Bennée" <alex.bennee@linaro.org>]

Cleber Rosa <crosa@redhat.com> writes:

> On Fri, Dec 06, 2019 at 09:00:10AM -0500, Wainer dos Santos Moschetta wrote:
>> This series add a new acceptance test: boot an uncompressed
>> Linux kernel built with CONFIG_PVH, so checking the PVH
>> capability introduced in QEMU 4.0 works.
>> 
>> The test implementation depends on [1] which is likely released
>> on next Avocado. So that will need a version 2 of this
>> series to bump Avocado version.
>>
>
> Right, the Avocado bits have been merged, so unless a major reversal
> comes (not expected at all), it will be on Avocado 74.0.
>
>> Also I want to use this as an example of a scenario that test
>> assets could be better managed. As you see on patch 01 the
>> kernel is built at test time on localhost. While Avocado provides
>> an API to easily fetch and build it, the whole process takes
>> reasonable time - besides the fact that localhost must have
>> all build dependencies installed. How could it be done better?
>>
>
> This is clearly not a "kernel build" test, so we should avoid building
> the kernel as part of the "PVH boot" test.  The problem you expose
> here is a very real, and each possible solution has its own problems,
> unfortunately.
>
> The best solution IMO would be to find a "well known" distribution of
> such a kernel.  Maybe something maintained by the Xen project or one
> of its commercial products?
>
> The second best solution is to have a helper script (using the Avocado
> utils API is fine) that will build a kernel and create/populate the
> test's data directory (tests/acceptance/pvh.py.data/) with a vmlinux.
> The test can cancel itself if it doesn't find a kernel there.
>
> The third best solution IMO is for this test to require a parameter
> telling where the CONFIG_PVH enabled vmlinux kernel image is.
>
> Also notice that, with a custom built kernel image (a different one
> for each user), the result of the test as a general indicator to the
> QEMU codebase diminishes quite a bit.

I can see a use case for making it easier for developers to build test
cases otherwise everyone has their own particular variant of the kernel
and buildroot/busybox which they have in their own farm. However as
Cleber has noted they make poor standardised tests given the variation
in kernel builds you can get.

That said I think there are better targets. kvm-unit-tests can be built
against a range of architectures and are fashioned as individual unit
tests for specific functionality. It would be useful to make it easy for
a developer to regenerate the test assets to re-run a test someone else
has found to fail.

>> Nonetheless I argue in favor of merging this as is, and
>> gradually implement corrections to improve the tests assets
>> management. Also if eventually the test is proven to unacceptable
>> slow down the Travis CI then I can add a guard to skip it.
>>
>
> I'm pretty sure that building the kernel will cause a major slow down
> of the Travis CI results for the "acceptance" block (and thus for the
> overall results).  But, it may also time it out (there's a 50 minutes
> deadline).
>
> I hope this brings some ideas, and thanks for coming up with
> interesting problems! :)
>
> - Cleber.
>
>> [1] https://github.com/avocado-framework/avocado/pull/3389
>> 
>> Wainer dos Santos Moschetta (2):
>>   tests/acceptance: Add PVH boot test
>>   .travis.yml: Add kernel build deps for acceptance tests
>> 
>>  .travis.yml             |  2 ++
>>  tests/acceptance/pvh.py | 48 +++++++++++++++++++++++++++++++++++++++++
>>  2 files changed, 50 insertions(+)
>>  create mode 100644 tests/acceptance/pvh.py
>> 
>> -- 
>> 2.21.0
>> 


-- 
Alex Bennée