Re: [PATCH 0/2] tests/acceptance: Add boot vmlinux test

[Cleber Rosa <crosa@redhat.com>]

[-- Attachment #1: Type: text/plain, Size: 2946 bytes --]

On Fri, Dec 06, 2019 at 09:00:10AM -0500, Wainer dos Santos Moschetta wrote:
> This series add a new acceptance test: boot an uncompressed
> Linux kernel built with CONFIG_PVH, so checking the PVH
> capability introduced in QEMU 4.0 works.
> 
> The test implementation depends on [1] which is likely released
> on next Avocado. So that will need a version 2 of this
> series to bump Avocado version.
>

Right, the Avocado bits have been merged, so unless a major reversal
comes (not expected at all), it will be on Avocado 74.0.

> Also I want to use this as an example of a scenario that test
> assets could be better managed. As you see on patch 01 the
> kernel is built at test time on localhost. While Avocado provides
> an API to easily fetch and build it, the whole process takes
> reasonable time - besides the fact that localhost must have
> all build dependencies installed. How could it be done better?
>

This is clearly not a "kernel build" test, so we should avoid building
the kernel as part of the "PVH boot" test.  The problem you expose
here is a very real, and each possible solution has its own problems,
unfortunately.

The best solution IMO would be to find a "well known" distribution of
such a kernel.  Maybe something maintained by the Xen project or one
of its commercial products?

The second best solution is to have a helper script (using the Avocado
utils API is fine) that will build a kernel and create/populate the
test's data directory (tests/acceptance/pvh.py.data/) with a vmlinux.
The test can cancel itself if it doesn't find a kernel there.

The third best solution IMO is for this test to require a parameter
telling where the CONFIG_PVH enabled vmlinux kernel image is.

Also notice that, with a custom built kernel image (a different one
for each user), the result of the test as a general indicator to the
QEMU codebase diminishes quite a bit.

> Nonetheless I argue in favor of merging this as is, and
> gradually implement corrections to improve the tests assets
> management. Also if eventually the test is proven to unacceptable
> slow down the Travis CI then I can add a guard to skip it.
>

I'm pretty sure that building the kernel will cause a major slow down
of the Travis CI results for the "acceptance" block (and thus for the
overall results).  But, it may also time it out (there's a 50 minutes
deadline).

I hope this brings some ideas, and thanks for coming up with
interesting problems! :)

- Cleber.

> [1] https://github.com/avocado-framework/avocado/pull/3389
> 
> Wainer dos Santos Moschetta (2):
>   tests/acceptance: Add PVH boot test
>   .travis.yml: Add kernel build deps for acceptance tests
> 
>  .travis.yml             |  2 ++
>  tests/acceptance/pvh.py | 48 +++++++++++++++++++++++++++++++++++++++++
>  2 files changed, 50 insertions(+)
>  create mode 100644 tests/acceptance/pvh.py
> 
> -- 
> 2.21.0
> 

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]