[igt-dev] Mandatory Test Suite for KMS Drivers?

[igt-dev] Mandatory Test Suite for KMS Drivers?

[Maxime Ripard]

[-- Attachment #1: Type: text/plain, Size: 456 bytes --]

Hi,

Back at XDC we floated the idea of creating a test suite for IGT that we
expect any KMS driver to pass, similar to what v4l2-compliance and
cec-compliance provide for v4l2 and CEC respectively.

I was looking at the list of tests, and it's fairly massive, so it's not
clear to me what tests we could start this suite with. I can only assume
all the KMS (but the chamelium ones) and fbdev related ones would be a
good start?

What do you think?
Maxime

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 228 bytes --]

Re: [igt-dev] Mandatory Test Suite for KMS Drivers?

[Rodrigo Vivi]

On Thu, Mar 03, 2022 at 10:05:07AM +0100, Maxime Ripard wrote:
> Hi,
> 
> Back at XDC we floated the idea of creating a test suite for IGT that we
> expect any KMS driver to pass, similar to what v4l2-compliance and
> cec-compliance provide for v4l2 and CEC respectively.
> 
> I was looking at the list of tests, and it's fairly massive, so it's not
> clear to me what tests we could start this suite with. I can only assume
> all the KMS (but the chamelium ones) and fbdev related ones would be a
> good start?
> 
> What do you think?

I believe we should start with the group of the tests that we know that
are reliably passing today on most of the platforms and then increase
the list as the tests and drivers become more reliable.

For instance, many of these would be candidate to be filtered out for now
https://intel-gfx-ci.01.org/tree/drm-next/index.html?testfilter=kms

compared to the whole view of kms tests:
https://intel-gfx-ci.01.org/tree/drm-next/shards-all.html?testfilter=kms

Thanks,
Rodrigo.

> Maxime

Re: [igt-dev] Mandatory Test Suite for KMS Drivers?

[Daniel Stone]

Hi,

On Thu, 3 Mar 2022 at 10:53, Rodrigo Vivi <rodrigo.vivi@intel.com> wrote:
> On Thu, Mar 03, 2022 at 10:05:07AM +0100, Maxime Ripard wrote:
> > Back at XDC we floated the idea of creating a test suite for IGT that we
> > expect any KMS driver to pass, similar to what v4l2-compliance and
> > cec-compliance provide for v4l2 and CEC respectively.
> >
> > I was looking at the list of tests, and it's fairly massive, so it's not
> > clear to me what tests we could start this suite with. I can only assume
> > all the KMS (but the chamelium ones) and fbdev related ones would be a
> > good start?
> >
> > What do you think?
>
> I believe we should start with the group of the tests that we know that
> are reliably passing today on most of the platforms and then increase
> the list as the tests and drivers become more reliable.
>
> For instance, many of these would be candidate to be filtered out for now
> https://intel-gfx-ci.01.org/tree/drm-next/index.html?testfilter=kms
>
> compared to the whole view of kms tests:
> https://intel-gfx-ci.01.org/tree/drm-next/shards-all.html?testfilter=kms

We are running some of IGT on Panfrost + amdgpu + i915 as part of
KernelCI as well: go to https://linux.kernelci.org/test/ and search
for 'igt-gpu'. This gets run for mainline, next, and whatever other
kernel trees push into i915.

There is a Grafana-based dashboard that the KernelCI team have been
working on to visualise test runs, but it's currently having some
backend issues so I can't show you a link for that. I did raise a
suggestion in their design discussion for a proper testing dashboard
for making it easier to see the status, so feel free to pile in there:
https://github.com/kernelci/kernelci-project/discussions/28#discussioncomment-2293696

Cheers,
Daniel

Re: [igt-dev] Mandatory Test Suite for KMS Drivers?

[Maxime Ripard]

[-- Attachment #1: Type: text/plain, Size: 1860 bytes --]

Hi Rodrigo,

On Thu, Mar 03, 2022 at 05:52:57AM -0500, Rodrigo Vivi wrote:
> On Thu, Mar 03, 2022 at 10:05:07AM +0100, Maxime Ripard wrote:
> > Hi,
> > 
> > Back at XDC we floated the idea of creating a test suite for IGT that we
> > expect any KMS driver to pass, similar to what v4l2-compliance and
> > cec-compliance provide for v4l2 and CEC respectively.
> > 
> > I was looking at the list of tests, and it's fairly massive, so it's not
> > clear to me what tests we could start this suite with. I can only assume
> > all the KMS (but the chamelium ones) and fbdev related ones would be a
> > good start?
> > 
> > What do you think?
> 
> I believe we should start with the group of the tests that we know that
> are reliably passing today on most of the platforms and then increase
> the list as the tests and drivers become more reliable.

I can see why that would be an objective too, but I'm not sure it would
cover mine. What I'd like this series to be is something we can ask
upfront to new drivers being submitted to make sure that they are sane.

Whether or not old drivers pass that bar is a bit irrelevant to that
objective (and this would actually create tasks for newcomers that are
looking for something to work on).

So, yeah, I don't mind having failing tests on older drivers, I kind of
even expect them to fail somehow. It would essentially be a bar to show
what any driver should strive for, not the lowest common denominator.

Does that make sense?

> For instance, many of these would be candidate to be filtered out for now
> https://intel-gfx-ci.01.org/tree/drm-next/index.html?testfilter=kms
> 
> compared to the whole view of kms tests:
> https://intel-gfx-ci.01.org/tree/drm-next/shards-all.html?testfilter=kms

So the set of tests that are always run would be the latter, right?

Maxime

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 228 bytes --]

Re: [igt-dev] Mandatory Test Suite for KMS Drivers?

[Maxime Ripard]

[-- Attachment #1: Type: text/plain, Size: 2059 bytes --]

Hi Daniel,

On Fri, Mar 04, 2022 at 08:45:07AM +0000, Daniel Stone wrote:
> On Thu, 3 Mar 2022 at 10:53, Rodrigo Vivi <rodrigo.vivi@intel.com> wrote:
> > On Thu, Mar 03, 2022 at 10:05:07AM +0100, Maxime Ripard wrote:
> > > Back at XDC we floated the idea of creating a test suite for IGT that we
> > > expect any KMS driver to pass, similar to what v4l2-compliance and
> > > cec-compliance provide for v4l2 and CEC respectively.
> > >
> > > I was looking at the list of tests, and it's fairly massive, so it's not
> > > clear to me what tests we could start this suite with. I can only assume
> > > all the KMS (but the chamelium ones) and fbdev related ones would be a
> > > good start?
> > >
> > > What do you think?
> >
> > I believe we should start with the group of the tests that we know that
> > are reliably passing today on most of the platforms and then increase
> > the list as the tests and drivers become more reliable.
> >
> > For instance, many of these would be candidate to be filtered out for now
> > https://intel-gfx-ci.01.org/tree/drm-next/index.html?testfilter=kms
> >
> > compared to the whole view of kms tests:
> > https://intel-gfx-ci.01.org/tree/drm-next/shards-all.html?testfilter=kms
> 
> We are running some of IGT on Panfrost + amdgpu + i915 as part of
> KernelCI as well: go to https://linux.kernelci.org/test/ and search
> for 'igt-gpu'. This gets run for mainline, next, and whatever other
> kernel trees push into i915.

I was mainly interested in KMS, but I saw that there's an igt-kms test
as well, thanks!

> There is a Grafana-based dashboard that the KernelCI team have been
> working on to visualise test runs, but it's currently having some
> backend issues so I can't show you a link for that. I did raise a
> suggestion in their design discussion for a proper testing dashboard
> for making it easier to see the status, so feel free to pile in there:
> https://github.com/kernelci/kernelci-project/discussions/28#discussioncomment-2293696

I'll have a look, thanks
Maxime

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 228 bytes --]