* [PATCH i-g-t] scripts: Add support to retest tests wich changed results
@ 2015-11-27 12:40 Gabriel Feceoru
2015-11-27 14:53 ` Thomas Wood
0 siblings, 1 reply; 3+ messages in thread
From: Gabriel Feceoru @ 2015-11-27 12:40 UTC (permalink / raw)
To: intel-gfx
This script takes multiple test results and prints the diffs, useful
for detecting noise in subsequent executions.
It gives the option to re-run these tests.
Signed-off-by: Gabriel Feceoru <gabriel.feceoru@intel.com>
---
scripts/retest-diff.sh | 72 ++++++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 72 insertions(+)
create mode 100755 scripts/retest-diff.sh
diff --git a/scripts/retest-diff.sh b/scripts/retest-diff.sh
new file mode 100755
index 0000000..aee1d05
--- /dev/null
+++ b/scripts/retest-diff.sh
@@ -0,0 +1,72 @@
+#!/bin/bash
+#
+# Copyright © 2015 Intel Corporation
+#
+# Permission is hereby granted, free of charge, to any person obtaining a
+# copy of this software and associated documentation files (the "Software"),
+# to deal in the Software without restriction, including without limitation
+# the rights to use, copy, modify, merge, publish, distribute, sublicense,
+# and/or sell copies of the Software, and to permit persons to whom the
+# Software is furnished to do so, subject to the following conditions:
+#
+# The above copyright notice and this permission notice (including the next
+# paragraph) shall be included in all copies or substantial portions of the
+# Software.
+#
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
+# THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
+# FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS
+# IN THE SOFTWARE.
+
+ROOT="`dirname $0`"
+ROOT="`readlink -f $ROOT/..`"
+IGT_TEST_ROOT="$ROOT/tests"
+RESULTS="$ROOT/results"
+PIGLIT="$ROOT/piglit/piglit"
+
+if [ ! -d "$IGT_TEST_ROOT" ]; then
+ echo "Error: could not find tests directory."
+ exit 1
+fi
+
+function print_help {
+ echo "Usage: $0 [<Results Paths> ...]"
+ echo ""
+}
+
+
+if [ $# -lt 2 ]; then
+ print_help
+ exit 1
+fi
+
+RESULT_FILES=$@
+
+for result in $RESULT_FILES; do
+ if [ ! -e $result ]; then
+ echo "Wrong result paths"
+ exit 1
+ fi
+done
+
+
+TESTS=`$PIGLIT summary console -d $RESULT_FILES | grep igt | cut -d':' -f1`
+if [[ -z $TESTS ]]; then
+ exit 1
+else
+ RED='\033[0;31m'
+ NC='\033[0m'
+ printf "$RED $TESTS $NC " | sed -e 's/ /\n/g'
+fi
+
+
+read -p "Dow you want to run these tests [y/N]? " -n 2 -r
+if [[ $REPLY =~ ^[Yy]$ ]]; then
+ mkdir -p $RESULTS
+ TEST_LIST=($TESTS)
+ TEST_PARAMS=`printf "%s " "${TEST_LIST[@]/#/-t }"`
+ sudo IGT_TEST_ROOT=$IGT_TEST_ROOT $PIGLIT run igt $RESULTS $TEST_PARAMS
+fi
--
1.9.1
_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/intel-gfx
^ permalink raw reply related [flat|nested] 3+ messages in thread* Re: [PATCH i-g-t] scripts: Add support to retest tests wich changed results
2015-11-27 12:40 [PATCH i-g-t] scripts: Add support to retest tests wich changed results Gabriel Feceoru
@ 2015-11-27 14:53 ` Thomas Wood
2015-11-30 8:34 ` Daniel Vetter
0 siblings, 1 reply; 3+ messages in thread
From: Thomas Wood @ 2015-11-27 14:53 UTC (permalink / raw)
To: Gabriel Feceoru; +Cc: Intel Graphics Development
On 27 November 2015 at 12:40, Gabriel Feceoru <gabriel.feceoru@intel.com> wrote:
> This script takes multiple test results and prints the diffs, useful
> for detecting noise in subsequent executions.
> It gives the option to re-run these tests.
This feature might be useful in Piglit itself. It sounds similar to
the ideas mentioned in this thread:
http://lists.freedesktop.org/archives/piglit/2015-November/018197.html
>
> Signed-off-by: Gabriel Feceoru <gabriel.feceoru@intel.com>
> ---
> scripts/retest-diff.sh | 72 ++++++++++++++++++++++++++++++++++++++++++++++++++
> 1 file changed, 72 insertions(+)
> create mode 100755 scripts/retest-diff.sh
>
> diff --git a/scripts/retest-diff.sh b/scripts/retest-diff.sh
> new file mode 100755
> index 0000000..aee1d05
> --- /dev/null
> +++ b/scripts/retest-diff.sh
> @@ -0,0 +1,72 @@
> +#!/bin/bash
> +#
> +# Copyright © 2015 Intel Corporation
> +#
> +# Permission is hereby granted, free of charge, to any person obtaining a
> +# copy of this software and associated documentation files (the "Software"),
> +# to deal in the Software without restriction, including without limitation
> +# the rights to use, copy, modify, merge, publish, distribute, sublicense,
> +# and/or sell copies of the Software, and to permit persons to whom the
> +# Software is furnished to do so, subject to the following conditions:
> +#
> +# The above copyright notice and this permission notice (including the next
> +# paragraph) shall be included in all copies or substantial portions of the
> +# Software.
> +#
> +# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
> +# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
> +# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
> +# THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
> +# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
> +# FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS
> +# IN THE SOFTWARE.
> +
> +ROOT="`dirname $0`"
> +ROOT="`readlink -f $ROOT/..`"
> +IGT_TEST_ROOT="$ROOT/tests"
> +RESULTS="$ROOT/results"
> +PIGLIT="$ROOT/piglit/piglit"
> +
> +if [ ! -d "$IGT_TEST_ROOT" ]; then
> + echo "Error: could not find tests directory."
> + exit 1
> +fi
> +
> +function print_help {
> + echo "Usage: $0 [<Results Paths> ...]"
> + echo ""
> +}
> +
> +
> +if [ $# -lt 2 ]; then
> + print_help
> + exit 1
> +fi
> +
> +RESULT_FILES=$@
> +
> +for result in $RESULT_FILES; do
> + if [ ! -e $result ]; then
> + echo "Wrong result paths"
> + exit 1
> + fi
> +done
> +
> +
> +TESTS=`$PIGLIT summary console -d $RESULT_FILES | grep igt | cut -d':' -f1`
> +if [[ -z $TESTS ]]; then
> + exit 1
> +else
> + RED='\033[0;31m'
> + NC='\033[0m'
> + printf "$RED $TESTS $NC " | sed -e 's/ /\n/g'
> +fi
> +
> +
> +read -p "Dow you want to run these tests [y/N]? " -n 2 -r
> +if [[ $REPLY =~ ^[Yy]$ ]]; then
> + mkdir -p $RESULTS
> + TEST_LIST=($TESTS)
> + TEST_PARAMS=`printf "%s " "${TEST_LIST[@]/#/-t }"`
> + sudo IGT_TEST_ROOT=$IGT_TEST_ROOT $PIGLIT run igt $RESULTS $TEST_PARAMS
> +fi
> --
> 1.9.1
>
> _______________________________________________
> Intel-gfx mailing list
> Intel-gfx@lists.freedesktop.org
> http://lists.freedesktop.org/mailman/listinfo/intel-gfx
_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/intel-gfx
^ permalink raw reply [flat|nested] 3+ messages in thread* Re: [PATCH i-g-t] scripts: Add support to retest tests wich changed results
2015-11-27 14:53 ` Thomas Wood
@ 2015-11-30 8:34 ` Daniel Vetter
0 siblings, 0 replies; 3+ messages in thread
From: Daniel Vetter @ 2015-11-30 8:34 UTC (permalink / raw)
To: Thomas Wood; +Cc: Intel Graphics Development
On Fri, Nov 27, 2015 at 02:53:36PM +0000, Thomas Wood wrote:
> On 27 November 2015 at 12:40, Gabriel Feceoru <gabriel.feceoru@intel.com> wrote:
> > This script takes multiple test results and prints the diffs, useful
> > for detecting noise in subsequent executions.
> > It gives the option to re-run these tests.
>
> This feature might be useful in Piglit itself. It sounds similar to
> the ideas mentioned in this thread:
>
> http://lists.freedesktop.org/archives/piglit/2015-November/018197.html
Yeah agreed, imo this should be implemented in piglit proper. We could
have different knobs to retest just regressions, fixes, or all changes
even.
-Daniel
>
>
> >
> > Signed-off-by: Gabriel Feceoru <gabriel.feceoru@intel.com>
> > ---
> > scripts/retest-diff.sh | 72 ++++++++++++++++++++++++++++++++++++++++++++++++++
> > 1 file changed, 72 insertions(+)
> > create mode 100755 scripts/retest-diff.sh
> >
> > diff --git a/scripts/retest-diff.sh b/scripts/retest-diff.sh
> > new file mode 100755
> > index 0000000..aee1d05
> > --- /dev/null
> > +++ b/scripts/retest-diff.sh
> > @@ -0,0 +1,72 @@
> > +#!/bin/bash
> > +#
> > +# Copyright © 2015 Intel Corporation
> > +#
> > +# Permission is hereby granted, free of charge, to any person obtaining a
> > +# copy of this software and associated documentation files (the "Software"),
> > +# to deal in the Software without restriction, including without limitation
> > +# the rights to use, copy, modify, merge, publish, distribute, sublicense,
> > +# and/or sell copies of the Software, and to permit persons to whom the
> > +# Software is furnished to do so, subject to the following conditions:
> > +#
> > +# The above copyright notice and this permission notice (including the next
> > +# paragraph) shall be included in all copies or substantial portions of the
> > +# Software.
> > +#
> > +# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
> > +# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
> > +# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
> > +# THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
> > +# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
> > +# FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS
> > +# IN THE SOFTWARE.
> > +
> > +ROOT="`dirname $0`"
> > +ROOT="`readlink -f $ROOT/..`"
> > +IGT_TEST_ROOT="$ROOT/tests"
> > +RESULTS="$ROOT/results"
> > +PIGLIT="$ROOT/piglit/piglit"
> > +
> > +if [ ! -d "$IGT_TEST_ROOT" ]; then
> > + echo "Error: could not find tests directory."
> > + exit 1
> > +fi
> > +
> > +function print_help {
> > + echo "Usage: $0 [<Results Paths> ...]"
> > + echo ""
> > +}
> > +
> > +
> > +if [ $# -lt 2 ]; then
> > + print_help
> > + exit 1
> > +fi
> > +
> > +RESULT_FILES=$@
> > +
> > +for result in $RESULT_FILES; do
> > + if [ ! -e $result ]; then
> > + echo "Wrong result paths"
> > + exit 1
> > + fi
> > +done
> > +
> > +
> > +TESTS=`$PIGLIT summary console -d $RESULT_FILES | grep igt | cut -d':' -f1`
> > +if [[ -z $TESTS ]]; then
> > + exit 1
> > +else
> > + RED='\033[0;31m'
> > + NC='\033[0m'
> > + printf "$RED $TESTS $NC " | sed -e 's/ /\n/g'
> > +fi
> > +
> > +
> > +read -p "Dow you want to run these tests [y/N]? " -n 2 -r
> > +if [[ $REPLY =~ ^[Yy]$ ]]; then
> > + mkdir -p $RESULTS
> > + TEST_LIST=($TESTS)
> > + TEST_PARAMS=`printf "%s " "${TEST_LIST[@]/#/-t }"`
> > + sudo IGT_TEST_ROOT=$IGT_TEST_ROOT $PIGLIT run igt $RESULTS $TEST_PARAMS
> > +fi
> > --
> > 1.9.1
> >
> > _______________________________________________
> > Intel-gfx mailing list
> > Intel-gfx@lists.freedesktop.org
> > http://lists.freedesktop.org/mailman/listinfo/intel-gfx
> _______________________________________________
> Intel-gfx mailing list
> Intel-gfx@lists.freedesktop.org
> http://lists.freedesktop.org/mailman/listinfo/intel-gfx
--
Daniel Vetter
Software Engineer, Intel Corporation
http://blog.ffwll.ch
_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/intel-gfx
^ permalink raw reply [flat|nested] 3+ messages in thread
end of thread, other threads:[~2015-11-30 8:34 UTC | newest]
Thread overview: 3+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2015-11-27 12:40 [PATCH i-g-t] scripts: Add support to retest tests wich changed results Gabriel Feceoru
2015-11-27 14:53 ` Thomas Wood
2015-11-30 8:34 ` Daniel Vetter
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox