Re: [Fuego] [Ksummit-discuss] Some ideas on open source,> testing

[Agustin Benito Bethencourt <agustin.benito@codethink.co.uk>]

Hi

it is my first mail here. Apologies for answering based on a digest 
mail. I already disabled the digest mode for the future.

> Message: 2
> Date: Mon, 24 Oct 2016 17:41:16 +0100
> From: Mark Brown <broonie@kernel.org>
> To: Theodore Ts'o <tytso@mit.edu>
> Cc: "ksummit-discuss@lists.linuxfoundation.org"
> 	<ksummit-discuss@lists.linuxfoundation.org>,
> 	"fuego@lists.linuxfoundation.org" <fuego@lists.linuxfoundation.org>
> Subject: Re: [Fuego] [Ksummit-discuss] Some ideas on open source
> 	testing
> Message-ID: <20161024164116.GB17252@sirena.org.uk>
> Content-Type: text/plain; charset="us-ascii"
>
> On Fri, Oct 21, 2016 at 09:19:25PM -0400, Theodore Ts'o wrote:
>
>> And so you might have alarge number of workflows:
>
>> * Developers who want a fast smoke test after applying a patch or
>>    patch set.
>> * Developers who want to dig into a specific test failure, and who
>>    want to be able to very quickly iterate over running a specific test
>>    against a quick succession of test kernels.
>> * Maintainers who want to run a much more comprehensive test suite
>>    that might take hours to run.  * Release engineers who want some
>>    kind of continuous integration test.
>
>> A test runner framework which is good for one is very likely not going
>> to be good for another.
>
> That's true but there's also an awful lot that can be shared - for
> example, the mechanics of booting on a given system are going to be the
> same regardless of how the test was scheduled or how long it will run
> for.  We should be looking for ways to share things as much as we can,
> especially around the bits where the skills needed to implement align
> less well with kernel skills.
>
> [UI for reporting]
>> I'm hoping to have an intern try to create something like that for
>> gce-xfstests that would run on Google App Engine, over the next couple
>> of months.  So maybe by next year I'll have something that we'll be
>> able to show off.  We'll see....
>
> This sort of UI thing seems like a prime example of an area where we
> could really use some sharing - for example we've got some capacity to
> run tests in kernelci but nobody to work on UI for doing anything useful
> with the results.

For UI testing there is an option for systems out there today, openQA, 
used in production by SUSE and more recently by Red Hat (Fedora).

I would look into it before thinking about extending kernelci to UI 
testing, so the complexity of what today is successful, among other 
reasons, for its simplicity.

When it comes to boot testing (acceptance testing) I wonder what is the 
percentage of cases in which testing on real hardware is required vs 
testing on emulated environments.

My experience on this has showed me that the percentage I assumed in 
initially (in theory) was lower than in practice (in favour of 
emulation). Depending on what is that percentage for you, openQA might 
be a good enough answer vs extending kernelci/Fuego.

openQA test visualization: https://openqa.opensuse.org/

Link to the project: http://open.qa/

I also believe Fuego can use openQA as a reference for some decisions 
that need to be taken in the future. There are not that many projects 
out there for system testing that are fully open, developed and used by 
people willing to help others.

Best Regards
-- 
Agustin Benito Bethencourt
Principal Consultant - FOSS at Codethink
agustin.benito@codethink.co.uk