media subsystem test pilot

media subsystem test pilot

[Guillaume Tucker]

Hi,

We've been discussing with Gustavo and Ezequiel how to go further
with KernelCI functional testing, it would be great to start with the
media subsystem as a pilot project.  The objective being to run more
comprehensive tests on the subsystem tree and report results to the
associated mailing list.

The Linux Media Summit is scheduled for 25th October following ELCE
in Edinburgh:

    https://www.spinics.net/lists/linux-media/msg139180.html

The idea would be to set something up on staging.kernelci.org and
show the results at the media summit.  This would give us a good
opportunity to get feedback from the developers about the builds,
tests, reports and maybe regressions and automated bisections.  Each
subsystem has its own particularities but we're hoping that by doing
more extensive media testing we'll learn how to enable other
subsystems in similar ways.


Here's the current plan of action, open to suggestions:

* add media subsystem tree (master branch) with regular boot test

* add config fragment (similar to the kselftest one) with virtual
  video drivers enabled to test v4l2 framework on QEMU and get
  reference test results

* add "streaming" tests to v4l2 test plan (-s option)

* build and use latest version of v4l-utils master branch

* send email report to the media subsystem

* run the test on some real hardware too


Depending on progress made, we may also start looking at:

* detecting and reporting regressions with the v4l2 test plan

* running automated bisections with a fixed v4l-utils build


This would all be done on staging.kernelci.org with WIP branches and
would not disturb the main KernelCI development.  The impact on
available resources (build servers, test farms) should be minimal.
It should then also make it easier to merge things as a follow-up.
Once the initial target for this pilot project has been reached, we
can think about how to do a transfer into production: consider which
parts to merge or refactor, which trees and hardware to run this test
plan on, how to deal with stable trees and older versions of
v4l-utils etc...


Does this sound like a good plan?  If so, it would be good to start
working on it next week if we want to get something done in time for
ELCE and give a chance to developers to provide some feedback on the
test results.

Best wishes,
Guillaume

Re: [kernelci] media subsystem test pilot

[Milosz Wasilewski]

On 12 September 2018 at 09:13, Guillaume Tucker
<guillaume.tucker@gmail.com> wrote:
>
> * add config fragment (similar to the kselftest one) with virtual
>   video drivers enabled to test v4l2 framework on QEMU and get
>   reference test results

Last week I asked Chase (CCed) to try to create a 'test definition'
for v4l for LAVA. I can see the v4l template in the repo now [1] but
it looks like bound to LAVA environment. Do you possibly have some
different implementation that allows for local execution?

[1] https://github.com/kernelci/kernelci-core-staging/blob/master/templates/v4l2/v4l2.jinja2

milosz

Re: [kernelci] media subsystem test pilot

[Mark Brown]

[-- Attachment #1: Type: text/plain, Size: 1110 bytes --]

On Wed, Sep 12, 2018 at 09:13:23AM +0100, Guillaume Tucker wrote:

> * add config fragment (similar to the kselftest one) with virtual
>   video drivers enabled to test v4l2 framework on QEMU and get
>   reference test results

Semi-orthogonal (and definitely not a blocker) but Anders Roxell (CCed)
was looking at proposing a config fragment for inclusion upstream which
would enable all the config options that add ABIs, mainly for use with
testsuites.  The idea was that with such a fragment enabled there'd be a
good chance that any random testsuite would have what it needs from the
kernel without needing to maintain lots of testsuite specific fragments, 
especially if the config options change with kernel versions.  I wonder
if it's also a good idea to have a fragment which enables all test/virtual 
drivers for similar reasons.

> Does this sound like a good plan?  If so, it would be good to start
> working on it next week if we want to get something done in time for
> ELCE and give a chance to developers to provide some feedback on the
> test results.

Makes sense to me.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [kernelci] media subsystem test pilot

[Ezequiel Garcia]

On Wed, 2018-09-12 at 09:51 +0100, Milosz Wasilewski wrote:
> On 12 September 2018 at 09:13, Guillaume Tucker
> <guillaume.tucker@gmail.com> wrote:
> > 
> > * add config fragment (similar to the kselftest one) with virtual
> >   video drivers enabled to test v4l2 framework on QEMU and get
> >   reference test results
> 
> Last week I asked Chase (CCed) to try to create a 'test definition'
> for v4l for LAVA. I can see the v4l template in the repo now [1] but
> it looks like bound to LAVA environment. Do you possibly have some
> different implementation that allows for local execution?
> 
> 

(+Nicolas)

v4l2-compliance is a decent test for simple capture devices
(simple means, those that function thru the V4L API).

However, I am not so sure it's the right tool for codecs,
which go thru the stateful or stateless codec API.

Nicolas and I have been talking about using gstreamer for that,
via gst-validate, which allows very interesting tests.

Regards,
Ezequiel

Re: [kernelci] media subsystem test pilot

[Kevin Hilman]

"Guillaume Tucker" <guillaume.tucker@gmail.com> writes:

[...]

> Does this sound like a good plan?  If so, it would be good to start
> working on it next week if we want to get something done in time for
> ELCE and give a chance to developers to provide some feedback on the
> test results.

Yes!

Kevin