[PATCH i-g-t] scripts: add quick-testlist.py

[PATCH i-g-t] scripts: add quick-testlist.py

[Thomas Wood]

Add a script to take a piglit results file and create a list of tests
that ran in under 60 seconds. This list can be used by the --test-list
option of piglit.

Signed-off-by: Thomas Wood <thomas.wood@intel.com>
---
 scripts/quick-testlist.py | 44 ++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 44 insertions(+)
 create mode 100755 scripts/quick-testlist.py

diff --git a/scripts/quick-testlist.py b/scripts/quick-testlist.py
new file mode 100755
index 0000000..2cd95b4
--- /dev/null
+++ b/scripts/quick-testlist.py
@@ -0,0 +1,44 @@
+#!/usr/bin/env python
+#
+# Copyright 2015 Intel Corporation
+#
+# Permission is hereby granted, free of charge, to any person obtaining a
+# copy of this software and associated documentation files (the "Software"),
+# to deal in the Software without restriction, including without limitation
+# the rights to use, copy, modify, merge, publish, distribute, sublicense,
+# and/or sell copies of the Software, and to permit persons to whom the
+# Software is furnished to do so, subject to the following conditions:
+#
+# The above copyright notice and this permission notice (including the next
+# paragraph) shall be included in all copies or substantial portions of the
+# Software.
+#
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.  IN NO EVENT SHALL
+# THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
+# FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS
+# IN THE SOFTWARE.
+
+from __future__ import print_function
+import json
+import sys
+
+def filter_results(filename):
+    with open(filename) as data:
+        json_data = json.load(data)
+
+    for test_name in json_data["tests"]:
+        if json_data["tests"][test_name]["time"] < 60:
+            print(test_name)
+
+
+if len(sys.argv) < 2:
+    print("Usage: quick-testlist.py [RESULTS]")
+    print("Read piglit results from RESULTS and print the tests that executed"
+          " in under 60 seconds. The list can be used by the --test-list option"
+          " of piglit.")
+    sys.exit(1)
+
+filter_results(sys.argv[1])
-- 
1.9.1

_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

[PATCH i-g-t v2] scripts: add quick-testlist.py

[Thomas Wood]

Add a script to take a piglit results file and create a list of tests
that ran in under 60 seconds. This list can be used by the --test-list
option of piglit.

v2: exclude incomplete tests

Signed-off-by: Thomas Wood <thomas.wood@intel.com>
---
 scripts/quick-testlist.py | 46 ++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 46 insertions(+)
 create mode 100755 scripts/quick-testlist.py

diff --git a/scripts/quick-testlist.py b/scripts/quick-testlist.py
new file mode 100755
index 0000000..0dd4c69
--- /dev/null
+++ b/scripts/quick-testlist.py
@@ -0,0 +1,46 @@
+#!/usr/bin/env python
+#
+# Copyright 2015 Intel Corporation
+#
+# Permission is hereby granted, free of charge, to any person obtaining a
+# copy of this software and associated documentation files (the "Software"),
+# to deal in the Software without restriction, including without limitation
+# the rights to use, copy, modify, merge, publish, distribute, sublicense,
+# and/or sell copies of the Software, and to permit persons to whom the
+# Software is furnished to do so, subject to the following conditions:
+#
+# The above copyright notice and this permission notice (including the next
+# paragraph) shall be included in all copies or substantial portions of the
+# Software.
+#
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.  IN NO EVENT SHALL
+# THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
+# FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS
+# IN THE SOFTWARE.
+
+from __future__ import print_function
+import json
+import sys
+
+def filter_results(filename):
+    with open(filename) as data:
+        json_data = json.load(data)
+
+    for test_name in json_data["tests"]:
+        if json_data["tests"][test_name]["result"] == "incomplete":
+            continue
+        if json_data["tests"][test_name]["time"] < 60:
+            print(test_name)
+
+
+if len(sys.argv) < 2:
+    print("Usage: quick-testlist.py RESULTS")
+    print("Read piglit results from RESULTS and print the tests that executed"
+          " in under 60 seconds, excluding any incomplete tests. The list can"
+          " be used by the --test-list option of piglit.")
+    sys.exit(1)
+
+filter_results(sys.argv[1])
-- 
1.9.1

_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

Re: [PATCH i-g-t v2] scripts: add quick-testlist.py

[Chris Wilson]

On Thu, Sep 17, 2015 at 12:42:44PM +0100, Thomas Wood wrote:
> Add a script to take a piglit results file and create a list of tests
> that ran in under 60 seconds. This list can be used by the --test-list
> option of piglit.

And passed?
-Chris

-- 
Chris Wilson, Intel Open Source Technology Centre
_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

Re: [PATCH i-g-t v2] scripts: add quick-testlist.py

[Thomas Wood]

On 17 September 2015 at 13:09, Chris Wilson <chris@chris-wilson.co.uk> wrote:
> On Thu, Sep 17, 2015 at 12:42:44PM +0100, Thomas Wood wrote:
>> Add a script to take a piglit results file and create a list of tests
>> that ran in under 60 seconds. This list can be used by the --test-list
>> option of piglit.
>
> And passed?

Any status except "incomplete", which normally means a reboot was
required. It might also be worth noting that tests skipped on one
platform may not skip on another, so the output only really applies to
the platform the results were initially produced on.


> -Chris
>
> --
> Chris Wilson, Intel Open Source Technology Centre
> _______________________________________________
> Intel-gfx mailing list
> Intel-gfx@lists.freedesktop.org
> http://lists.freedesktop.org/mailman/listinfo/intel-gfx
_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

Re: [PATCH i-g-t v2] scripts: add quick-testlist.py

[Chris Wilson]

On Thu, Sep 17, 2015 at 01:16:18PM +0100, Thomas Wood wrote:
> On 17 September 2015 at 13:09, Chris Wilson <chris@chris-wilson.co.uk> wrote:
> > On Thu, Sep 17, 2015 at 12:42:44PM +0100, Thomas Wood wrote:
> >> Add a script to take a piglit results file and create a list of tests
> >> that ran in under 60 seconds. This list can be used by the --test-list
> >> option of piglit.
> >
> > And passed?
> 
> Any status except "incomplete", which normally means a reboot was
> required. It might also be worth noting that tests skipped on one
> platform may not skip on another, so the output only really applies to
> the platform the results were initially produced on.

My point is that the timing can only be expected to be consistent for a
pass. A failure may be quick, but success may take a few hours.
-Chris

-- 
Chris Wilson, Intel Open Source Technology Centre
_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

Re: [PATCH i-g-t v2] scripts: add quick-testlist.py

[Feceoru, Gabriel]

This script measures the duration of the subtests. But in case of gem_concurrent_all, 
for instance, there are 3780 subtests for which, although many shorter than 60s, the total
duration will not qualify this for a quick test (~2h).

Thank you,
Gabriel
________________________________________
From: Intel-gfx [intel-gfx-bounces@lists.freedesktop.org] on behalf of Chris Wilson [chris@chris-wilson.co.uk]
Sent: Thursday, September 17, 2015 3:43 PM
To: Wood, Thomas
Cc: Intel Graphics Development
Subject: Re: [Intel-gfx] [PATCH i-g-t v2] scripts: add quick-testlist.py

On Thu, Sep 17, 2015 at 01:16:18PM +0100, Thomas Wood wrote:
> On 17 September 2015 at 13:09, Chris Wilson <chris@chris-wilson.co.uk> wrote:
> > On Thu, Sep 17, 2015 at 12:42:44PM +0100, Thomas Wood wrote:
> >> Add a script to take a piglit results file and create a list of tests
> >> that ran in under 60 seconds. This list can be used by the --test-list
> >> option of piglit.
> >
> > And passed?
>
> Any status except "incomplete", which normally means a reboot was
> required. It might also be worth noting that tests skipped on one
> platform may not skip on another, so the output only really applies to
> the platform the results were initially produced on.

My point is that the timing can only be expected to be consistent for a
pass. A failure may be quick, but success may take a few hours.
-Chris

--
Chris Wilson, Intel Open Source Technology Centre
_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/intel-gfx
_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

Re: [PATCH i-g-t v2] scripts: add quick-testlist.py

[Thomas Wood]

On 17 September 2015 at 15:01, Feceoru, Gabriel
<gabriel.feceoru@intel.com> wrote:
> This script measures the duration of the subtests. But in case of gem_concurrent_all,
> for instance, there are 3780 subtests for which, although many shorter than 60s, the total
> duration will not qualify this for a quick test (~2h).

For this specific case, there has been some discussion about hiding
some of the combinatorial subtests from the default testing list.
kms_frontbuffer_tracking already has an option to show hidden subtests
and pm_rpm has an option to reduce the amount of work in the stress
tests. A common approach to hiding or reducing the number of stress
tests for quick testing would be useful.


>
> Thank you,
> Gabriel
> ________________________________________
> From: Intel-gfx [intel-gfx-bounces@lists.freedesktop.org] on behalf of Chris Wilson [chris@chris-wilson.co.uk]
> Sent: Thursday, September 17, 2015 3:43 PM
> To: Wood, Thomas
> Cc: Intel Graphics Development
> Subject: Re: [Intel-gfx] [PATCH i-g-t v2] scripts: add quick-testlist.py
>
> On Thu, Sep 17, 2015 at 01:16:18PM +0100, Thomas Wood wrote:
>> On 17 September 2015 at 13:09, Chris Wilson <chris@chris-wilson.co.uk> wrote:
>> > On Thu, Sep 17, 2015 at 12:42:44PM +0100, Thomas Wood wrote:
>> >> Add a script to take a piglit results file and create a list of tests
>> >> that ran in under 60 seconds. This list can be used by the --test-list
>> >> option of piglit.
>> >
>> > And passed?
>>
>> Any status except "incomplete", which normally means a reboot was
>> required. It might also be worth noting that tests skipped on one
>> platform may not skip on another, so the output only really applies to
>> the platform the results were initially produced on.
>
> My point is that the timing can only be expected to be consistent for a
> pass. A failure may be quick, but success may take a few hours.
> -Chris
>
> --
> Chris Wilson, Intel Open Source Technology Centre
> _______________________________________________
> Intel-gfx mailing list
> Intel-gfx@lists.freedesktop.org
> http://lists.freedesktop.org/mailman/listinfo/intel-gfx
_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

Re: [PATCH i-g-t v2] scripts: add quick-testlist.py

[Thomas Wood]

On 17 September 2015 at 13:43, Chris Wilson <chris@chris-wilson.co.uk> wrote:
> On Thu, Sep 17, 2015 at 01:16:18PM +0100, Thomas Wood wrote:
>> On 17 September 2015 at 13:09, Chris Wilson <chris@chris-wilson.co.uk> wrote:
>> > On Thu, Sep 17, 2015 at 12:42:44PM +0100, Thomas Wood wrote:
>> >> Add a script to take a piglit results file and create a list of tests
>> >> that ran in under 60 seconds. This list can be used by the --test-list
>> >> option of piglit.
>> >
>> > And passed?
>>
>> Any status except "incomplete", which normally means a reboot was
>> required. It might also be worth noting that tests skipped on one
>> platform may not skip on another, so the output only really applies to
>> the platform the results were initially produced on.
>
> My point is that the timing can only be expected to be consistent for a
> pass. A failure may be quick, but success may take a few hours.

I think including failed tests depends on whether the "quick" run
should be able to detect fixes from previous runs. Perhaps an option
to exclude anything other than successful tests might be useful. There
is also the timeout mechanism in piglit/igt that will prevent any test
from running for more than 10 minutes.


> -Chris
>
> --
> Chris Wilson, Intel Open Source Technology Centre
_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

Re: [PATCH i-g-t v2] scripts: add quick-testlist.py

[Daniel Vetter]

On Thu, Sep 17, 2015 at 04:47:06PM +0100, Thomas Wood wrote:
> On 17 September 2015 at 13:43, Chris Wilson <chris@chris-wilson.co.uk> wrote:
> > On Thu, Sep 17, 2015 at 01:16:18PM +0100, Thomas Wood wrote:
> >> On 17 September 2015 at 13:09, Chris Wilson <chris@chris-wilson.co.uk> wrote:
> >> > On Thu, Sep 17, 2015 at 12:42:44PM +0100, Thomas Wood wrote:
> >> >> Add a script to take a piglit results file and create a list of tests
> >> >> that ran in under 60 seconds. This list can be used by the --test-list
> >> >> option of piglit.
> >> >
> >> > And passed?
> >>
> >> Any status except "incomplete", which normally means a reboot was
> >> required. It might also be worth noting that tests skipped on one
> >> platform may not skip on another, so the output only really applies to
> >> the platform the results were initially produced on.
> >
> > My point is that the timing can only be expected to be consistent for a
> > pass. A failure may be quick, but success may take a few hours.
> 
> I think including failed tests depends on whether the "quick" run
> should be able to detect fixes from previous runs. Perhaps an option
> to exclude anything other than successful tests might be useful. There
> is also the timeout mechanism in piglit/igt that will prevent any test
> from running for more than 10 minutes.

Yeah, for the fast run we also need to reduce the piglit timeout to 60s
(and kill anything that timed out ofc too). Otherwise there's indeed a gap
where a testcase that takes too long could sneak in.
-Daniel
-- 
Daniel Vetter
Software Engineer, Intel Corporation
http://blog.ffwll.ch
_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

Re: [Intel-gfx] [PATCH i-g-t v2] scripts: add quick-testlist.py

[Daniel Vetter]

On Thu, Sep 17, 2015 at 12:42:44PM +0100, Thomas Wood wrote:
> Add a script to take a piglit results file and create a list of tests
> that ran in under 60 seconds. This list can be used by the --test-list
> option of piglit.
> 
> v2: exclude incomplete tests
> 
> Signed-off-by: Thomas Wood <thomas.wood@intel.com>

I think this might be generally useful for piglit uses, maybe in contrib/?

Anyway adding piglit m-l.
-Daniel

> ---
>  scripts/quick-testlist.py | 46 ++++++++++++++++++++++++++++++++++++++++++++++
>  1 file changed, 46 insertions(+)
>  create mode 100755 scripts/quick-testlist.py
> 
> diff --git a/scripts/quick-testlist.py b/scripts/quick-testlist.py
> new file mode 100755
> index 0000000..0dd4c69
> --- /dev/null
> +++ b/scripts/quick-testlist.py
> @@ -0,0 +1,46 @@
> +#!/usr/bin/env python
> +#
> +# Copyright 2015 Intel Corporation
> +#
> +# Permission is hereby granted, free of charge, to any person obtaining a
> +# copy of this software and associated documentation files (the "Software"),
> +# to deal in the Software without restriction, including without limitation
> +# the rights to use, copy, modify, merge, publish, distribute, sublicense,
> +# and/or sell copies of the Software, and to permit persons to whom the
> +# Software is furnished to do so, subject to the following conditions:
> +#
> +# The above copyright notice and this permission notice (including the next
> +# paragraph) shall be included in all copies or substantial portions of the
> +# Software.
> +#
> +# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
> +# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
> +# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.  IN NO EVENT SHALL
> +# THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
> +# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
> +# FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS
> +# IN THE SOFTWARE.
> +
> +from __future__ import print_function
> +import json
> +import sys
> +
> +def filter_results(filename):
> +    with open(filename) as data:
> +        json_data = json.load(data)
> +
> +    for test_name in json_data["tests"]:
> +        if json_data["tests"][test_name]["result"] == "incomplete":
> +            continue
> +        if json_data["tests"][test_name]["time"] < 60:
> +            print(test_name)
> +
> +
> +if len(sys.argv) < 2:
> +    print("Usage: quick-testlist.py RESULTS")
> +    print("Read piglit results from RESULTS and print the tests that executed"
> +          " in under 60 seconds, excluding any incomplete tests. The list can"
> +          " be used by the --test-list option of piglit.")
> +    sys.exit(1)
> +
> +filter_results(sys.argv[1])
> -- 
> 1.9.1
> 
> _______________________________________________
> Intel-gfx mailing list
> Intel-gfx@lists.freedesktop.org
> http://lists.freedesktop.org/mailman/listinfo/intel-gfx

-- 
Daniel Vetter
Software Engineer, Intel Corporation
http://blog.ffwll.ch
_______________________________________________
Piglit mailing list
Piglit@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/piglit