* [PATCH 2/8] ktest: Set buildonly=1 for CONFIG_BISECT_TYPE=build
2017-07-17 0:16 [PATCH 1/8] ktest: Clarify config file usage Scott Wood
@ 2017-07-17 0:16 ` Scott Wood
2017-07-17 0:16 ` [PATCH 3/8] ktest: Set do_not_reboot=y " Scott Wood
` (5 subsequent siblings)
6 siblings, 0 replies; 15+ messages in thread
From: Scott Wood @ 2017-07-17 0:16 UTC (permalink / raw)
To: Steven Rostedt; +Cc: linux-kernel, Scott Wood
Rather than adding a third copy of the same logic, rework it to cover
all three buildonly cases at once.
In the future, please consider using the same variable to perform the
same function regardless of context...
Signed-off-by: Scott Wood <swood@redhat.com>
---
tools/testing/ktest/ktest.pl | 18 +++++-------------
1 file changed, 5 insertions(+), 13 deletions(-)
diff --git a/tools/testing/ktest/ktest.pl b/tools/testing/ktest/ktest.pl
index cdf6974bc462..36e1b91259cd 100755
--- a/tools/testing/ktest/ktest.pl
+++ b/tools/testing/ktest/ktest.pl
@@ -718,22 +718,14 @@ sub set_value {
my $prvalue = process_variables($rvalue);
- if ($buildonly && $lvalue =~ /^TEST_TYPE(\[.*\])?$/ && $prvalue ne "build") {
+ if ($lvalue =~ /^(TEST|BISECT|CONFIG_BISECT)_TYPE(\[.*\])?$/ &&
+ $prvalue !~ /^(config_|)bisect$/ &&
+ $prvalue !~ /^build$/ &&
+ $buildonly) {
+
# Note if a test is something other than build, then we
# will need other mandatory options.
if ($prvalue ne "install") {
- # for bisect, we need to check BISECT_TYPE
- if ($prvalue ne "bisect") {
- $buildonly = 0;
- }
- } else {
- # install still limits some mandatory options.
- $buildonly = 2;
- }
- }
-
- if ($buildonly && $lvalue =~ /^BISECT_TYPE(\[.*\])?$/ && $prvalue ne "build") {
- if ($prvalue ne "install") {
$buildonly = 0;
} else {
# install still limits some mandatory options.
--
2.9.4
^ permalink raw reply related [flat|nested] 15+ messages in thread* [PATCH 3/8] ktest: Set do_not_reboot=y for CONFIG_BISECT_TYPE=build
2017-07-17 0:16 [PATCH 1/8] ktest: Clarify config file usage Scott Wood
2017-07-17 0:16 ` [PATCH 2/8] ktest: Set buildonly=1 for CONFIG_BISECT_TYPE=build Scott Wood
@ 2017-07-17 0:16 ` Scott Wood
2017-07-17 0:16 ` [PATCH 4/8] ktest: Separate out config bisect logic Scott Wood
` (4 subsequent siblings)
6 siblings, 0 replies; 15+ messages in thread
From: Scott Wood @ 2017-07-17 0:16 UTC (permalink / raw)
To: Steven Rostedt; +Cc: linux-kernel, Scott Wood
Currently setting do_not_reboot is triggered by simple builds and bisect
builds, but not config bisect builds.
Signed-off-by: Scott Wood <swood@redhat.com>
---
tools/testing/ktest/ktest.pl | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/tools/testing/ktest/ktest.pl b/tools/testing/ktest/ktest.pl
index 36e1b91259cd..7d537bdf462b 100755
--- a/tools/testing/ktest/ktest.pl
+++ b/tools/testing/ktest/ktest.pl
@@ -1397,7 +1397,8 @@ sub do_not_reboot {
return $test_type eq "build" || $no_reboot ||
($test_type eq "patchcheck" && $opt{"PATCHCHECK_TYPE[$i]"} eq "build") ||
- ($test_type eq "bisect" && $opt{"BISECT_TYPE[$i]"} eq "build");
+ ($test_type eq "bisect" && $opt{"BISECT_TYPE[$i]"} eq "build") ||
+ ($test_type eq "config_bisect" && $opt{"CONFIG_BISECT_TYPE[$i]"} eq "build");
}
sub dodie {
--
2.9.4
^ permalink raw reply related [flat|nested] 15+ messages in thread* [PATCH 4/8] ktest: Separate out config bisect logic
2017-07-17 0:16 [PATCH 1/8] ktest: Clarify config file usage Scott Wood
2017-07-17 0:16 ` [PATCH 2/8] ktest: Set buildonly=1 for CONFIG_BISECT_TYPE=build Scott Wood
2017-07-17 0:16 ` [PATCH 3/8] ktest: Set do_not_reboot=y " Scott Wood
@ 2017-07-17 0:16 ` Scott Wood
2017-07-17 0:16 ` [PATCH 5/8] ktest/config_bisect: Simplify " Scott Wood
` (3 subsequent siblings)
6 siblings, 0 replies; 15+ messages in thread
From: Scott Wood @ 2017-07-17 0:16 UTC (permalink / raw)
To: Steven Rostedt; +Cc: linux-kernel, Scott Wood
From: Scott Wood <oss@buserror.net>
This allows config bisecting to be performed when building on the machine
being tested -- all required state is saved in the filesystem rather than
requiring one script instance to run for the duration of the bisecting.
Plus, while ktest appears very featureful, it is also a bit complicated
to set up, which can be a barrier to accessing the config bisect
functionality. Separating it is more in line with traditional Unix
philosophy of small tools working together rather than a big monolith.
Signed-off-by: Scott Wood <swood@redhat.com>
---
tools/testing/ktest/config-bisect.pl | 280 +++++++++++++++++++++++++++++++++++
1 file changed, 280 insertions(+)
create mode 100755 tools/testing/ktest/config-bisect.pl
diff --git a/tools/testing/ktest/config-bisect.pl b/tools/testing/ktest/config-bisect.pl
new file mode 100755
index 000000000000..1768d145ced8
--- /dev/null
+++ b/tools/testing/ktest/config-bisect.pl
@@ -0,0 +1,280 @@
+#!/usr/bin/perl -w
+#
+# Copyright 2010 - Steven Rostedt <srostedt@redhat.com>, Red Hat Inc.
+# Licensed under the terms of the GNU GPL License version 2
+#
+
+use strict;
+
+my $outputdir;
+
+sub make_oldconfig {
+ if (system "\$MAKE O=$outputdir olddefconfig") {
+ # Perhaps olddefconfig doesn't exist in this version of
+ # the kernel; try oldnoconfig
+ print "olddefconfig failed, trying make oldnoconfig\n";
+ if (system "\$MAKE O=$outputdir oldnoconfig") {
+ print "oldnoconfig failed, trying yes '' | make oldconfig\n";
+ # try a yes '' | oldconfig
+ system "yes '' | \$MAKE O=$outputdir oldconfig" == 0 or
+ die "failed make config oldconfig";
+ }
+ }
+}
+
+sub assign_configs {
+ my ($hash, $config) = @_;
+
+ print "Reading configs from $config\n";
+
+ open (IN, $config)
+ or die "Failed to read $config";
+
+ while (<IN>) {
+ chomp;
+ if (/^((CONFIG\S*)=.*)/) {
+ ${$hash}{$2} = $1;
+ } elsif (/^(# (CONFIG\S*) is not set)/) {
+ ${$hash}{$2} = $1;
+ }
+ }
+
+ close(IN);
+}
+
+sub save_config {
+ my ($pc, $file) = @_;
+
+ my %configs = %{$pc};
+
+ print "Saving configs into $file\n";
+
+ open(OUT, ">$file") or die "Can not write to $file";
+
+ foreach my $config (keys %configs) {
+ print OUT "$configs{$config}\n";
+ }
+ close(OUT);
+
+ make_oldconfig;
+}
+
+# compare two config hashes, and return configs with different vals.
+# It returns B's config values, but you can use A to see what A was.
+sub diff_config_vals {
+ my ($pa, $pb) = @_;
+
+ # crappy Perl way to pass in hashes.
+ my %a = %{$pa};
+ my %b = %{$pb};
+
+ my %ret;
+
+ foreach my $item (keys %a) {
+ if (defined($b{$item}) && $b{$item} ne $a{$item}) {
+ $ret{$item} = $b{$item};
+ }
+ }
+
+ return %ret;
+}
+
+# compare two config hashes and return the configs in B but not A
+sub diff_configs {
+ my ($pa, $pb) = @_;
+
+ my %ret;
+
+ # crappy Perl way to pass in hashes.
+ my %a = %{$pa};
+ my %b = %{$pb};
+
+ foreach my $item (keys %b) {
+ if (!defined($a{$item})) {
+ $ret{$item} = $b{$item};
+ }
+ }
+
+ return %ret;
+}
+
+# return if two configs are equal or not
+# 0 is equal +1 b has something a does not
+# +1 if a and b have a different item.
+# -1 if a has something b does not
+sub compare_configs {
+ my ($pa, $pb) = @_;
+
+ my %ret;
+
+ # crappy Perl way to pass in hashes.
+ my %a = %{$pa};
+ my %b = %{$pb};
+
+ foreach my $item (keys %b) {
+ if (!defined($a{$item})) {
+ return 1;
+ }
+ if ($a{$item} ne $b{$item}) {
+ return 1;
+ }
+ }
+
+ foreach my $item (keys %a) {
+ if (!defined($b{$item})) {
+ return -1;
+ }
+ }
+
+ return 0;
+}
+
+
+sub process_new_config {
+ my ($tc, $nc, $gc, $bc, $ofile) = @_;
+
+ my %tmp_config = %{$tc};
+ my %good_configs = %{$gc};
+ my %bad_configs = %{$bc};
+
+ my %new_configs;
+
+ my $runtest = 1;
+ my $ret;
+
+ save_config \%tmp_config, $ofile;
+ assign_configs \%new_configs, $ofile;
+
+ $ret = compare_configs \%new_configs, \%bad_configs;
+ if (!$ret) {
+ print "New config equals bad config, try next test\n";
+ $runtest = 0;
+ }
+
+ if ($runtest) {
+ $ret = compare_configs \%new_configs, \%good_configs;
+ if (!$ret) {
+ print "New config equals good config, try next test\n";
+ $runtest = 0;
+ }
+ }
+
+ %{$nc} = %new_configs;
+
+ return $runtest;
+}
+
+sub run_config_bisect {
+ my ($outfile, $pgood, $pbad) = @_;
+
+ my %good_configs = %{$pgood};
+ my %bad_configs = %{$pbad};
+
+ my %diff_configs = diff_config_vals \%good_configs, \%bad_configs;
+ my %b_configs = diff_configs \%good_configs, \%bad_configs;
+ my %g_configs = diff_configs \%bad_configs, \%good_configs;
+
+ my @diff_arr = keys %diff_configs;
+ my $len_diff = $#diff_arr + 1;
+
+ my @b_arr = keys %b_configs;
+ my $len_b = $#b_arr + 1;
+
+ my @g_arr = keys %g_configs;
+ my $len_g = $#g_arr + 1;
+
+ my $runtest = 1;
+ my %new_configs;
+ my $ret;
+
+ # First, lets get it down to a single subset.
+ # Is the problem with a difference in values?
+ # Is the problem with a missing config?
+ # Is the problem with a config that breaks things?
+
+ # Enable all of one set and see if we get a new bad
+ # or good config.
+
+ # first set the good config to the bad values.
+
+ print "d=$len_diff g=$len_g b=$len_b\n";
+
+ # first lets enable things in bad config that are enabled in good config
+
+ if ($len_diff > 0) {
+ if ($len_b > 0 || $len_g > 0) {
+ my %tmp_config = %bad_configs;
+
+ print "Set tmp config to be bad config with good config values\n";
+ foreach my $item (@diff_arr) {
+ $tmp_config{$item} = $good_configs{$item};
+ }
+
+ $runtest = process_new_config \%tmp_config,
+ \%new_configs, \%good_configs,
+ \%bad_configs, $outfile;
+ }
+ }
+
+ if (!$runtest && $len_diff > 0) {
+
+ if ($len_diff == 1) {
+ process_failed $diff_arr[0];
+ return 1;
+ }
+ my %tmp_config = %bad_configs;
+
+ my $half = int($#diff_arr / 2);
+ my @tophalf = @diff_arr[0 .. $half];
+
+ print "Settings bisect with top half:\n";
+ print "Set tmp config to be bad config with some good config values\n";
+ foreach my $item (@tophalf) {
+ $tmp_config{$item} = $good_configs{$item};
+ }
+
+ $runtest = process_new_config \%tmp_config, \%new_configs,
+ \%good_configs, \%bad_configs, $outfile;
+
+ if (!$runtest) {
+ my %tmp_config = %bad_configs;
+
+ print "Try bottom half\n";
+
+ my @bottomhalf = @diff_arr[$half+1 .. $#diff_arr];
+
+ foreach my $item (@bottomhalf) {
+ $tmp_config{$item} = $good_configs{$item};
+ }
+
+ $runtest = process_new_config \%tmp_config,
+ \%new_configs, \%good_configs,
+ \%bad_configs, $outfile;
+ }
+ }
+
+}
+
+sub cb_by_file {
+ my ($outfile, $goodfile, $badfile) = @_;
+ my (%good, %bad);
+
+ assign_configs \%good, $goodfile;
+ assign_configs \%bad, $badfile;
+
+ run_config_bisect $outfile, \%good, \%bad;
+}
+
+if (!defined($ARGV[2])) {
+ print "Usage: $0 <outputdir> <good> <bad>\n";
+ exit 1;
+}
+
+if (!defined($ENV{MAKE})) {
+ $ENV{MAKE} = "make";
+}
+
+$outputdir = $ARGV[0];
+
+cb_by_file("${outputdir}/.config", $ARGV[1], $ARGV[2]);
+exit 0;
--
2.9.4
^ permalink raw reply related [flat|nested] 15+ messages in thread* [PATCH 5/8] ktest/config_bisect: Simplify bisect logic
2017-07-17 0:16 [PATCH 1/8] ktest: Clarify config file usage Scott Wood
` (2 preceding siblings ...)
2017-07-17 0:16 ` [PATCH 4/8] ktest: Separate out config bisect logic Scott Wood
@ 2017-07-17 0:16 ` Scott Wood
2017-07-17 0:16 ` [PATCH 6/8] ktest/config-bisect: Try harder to find a new config Scott Wood
` (2 subsequent siblings)
6 siblings, 0 replies; 15+ messages in thread
From: Scott Wood @ 2017-07-17 0:16 UTC (permalink / raw)
To: Steven Rostedt; +Cc: linux-kernel, Scott Wood
From: Scott Wood <oss@buserror.net>
There's no need to diff the configs three different ways or to do things
like "first lets enable things in bad config that are enabled in good
config". If "undefined" is treated as just another value that a config
can have, then all the cases reduce to "a problem with a difference in
values".
Plus, the old logic would fail with some inputs, as it confuses a symbol
not appearing in the .config file (and thus in the hash) with a symbol
not being selected (and thus being present in the hash with a value of "#
CONFIG_FOO is not set"). diff_configs checks for the former, but
assuming the input configs have been run through make oldconfig, any
symbols that are not present at all are not selectable and thus are side
effects of some other difference -- not relevant to the bisection.
If the bisection proceeds to the point where there are none of these
side effects left, len_b and len_g will both be zero. If len_diff is
greater than one at this point, a new config to try will not be produced.
Since b_arr and g_arr are irrelevant to the bisection, remove them and
only calculate differences in symbols that are present in both hashes, and
proceed directly to the logic that deals with differences in value.
While we're at it, have diff_config_vals return an array of keys rather
than a hash, since the keys are the only thing ever looked at.
Signed-off-by: Scott Wood <swood@redhat.com>
---
tools/testing/ktest/config-bisect.pl | 108 ++++++++---------------------------
1 file changed, 23 insertions(+), 85 deletions(-)
diff --git a/tools/testing/ktest/config-bisect.pl b/tools/testing/ktest/config-bisect.pl
index 1768d145ced8..63a3a940b5de 100755
--- a/tools/testing/ktest/config-bisect.pl
+++ b/tools/testing/ktest/config-bisect.pl
@@ -60,7 +60,6 @@ sub save_config {
}
# compare two config hashes, and return configs with different vals.
-# It returns B's config values, but you can use A to see what A was.
sub diff_config_vals {
my ($pa, $pb) = @_;
@@ -68,34 +67,15 @@ sub diff_config_vals {
my %a = %{$pa};
my %b = %{$pb};
- my %ret;
+ my @ret;
foreach my $item (keys %a) {
if (defined($b{$item}) && $b{$item} ne $a{$item}) {
- $ret{$item} = $b{$item};
+ push @ret, $item;
}
}
- return %ret;
-}
-
-# compare two config hashes and return the configs in B but not A
-sub diff_configs {
- my ($pa, $pb) = @_;
-
- my %ret;
-
- # crappy Perl way to pass in hashes.
- my %a = %{$pa};
- my %b = %{$pb};
-
- foreach my $item (keys %b) {
- if (!defined($a{$item})) {
- $ret{$item} = $b{$item};
- }
- }
-
- return %ret;
+ return @ret;
}
# return if two configs are equal or not
@@ -170,89 +150,47 @@ sub run_config_bisect {
my %good_configs = %{$pgood};
my %bad_configs = %{$pbad};
- my %diff_configs = diff_config_vals \%good_configs, \%bad_configs;
- my %b_configs = diff_configs \%good_configs, \%bad_configs;
- my %g_configs = diff_configs \%bad_configs, \%good_configs;
-
- my @diff_arr = keys %diff_configs;
+ my @diff_arr = diff_config_vals \%good_configs, \%bad_configs;
my $len_diff = $#diff_arr + 1;
- my @b_arr = keys %b_configs;
- my $len_b = $#b_arr + 1;
-
- my @g_arr = keys %g_configs;
- my $len_g = $#g_arr + 1;
-
my $runtest = 1;
my %new_configs;
my $ret;
- # First, lets get it down to a single subset.
- # Is the problem with a difference in values?
- # Is the problem with a missing config?
- # Is the problem with a config that breaks things?
-
- # Enable all of one set and see if we get a new bad
- # or good config.
-
- # first set the good config to the bad values.
-
- print "d=$len_diff g=$len_g b=$len_b\n";
+ print "d=$len_diff\n";
- # first lets enable things in bad config that are enabled in good config
+ if ($len_diff <= 1) {
+ print "$0: No more bisecting possible\n";
+ exit 2;
+ }
- if ($len_diff > 0) {
- if ($len_b > 0 || $len_g > 0) {
- my %tmp_config = %bad_configs;
+ my %tmp_config = %bad_configs;
- print "Set tmp config to be bad config with good config values\n";
- foreach my $item (@diff_arr) {
- $tmp_config{$item} = $good_configs{$item};
- }
+ my $half = int($#diff_arr / 2);
+ my @tophalf = @diff_arr[0 .. $half];
- $runtest = process_new_config \%tmp_config,
- \%new_configs, \%good_configs,
- \%bad_configs, $outfile;
- }
+ print "Settings bisect with top half:\n";
+ foreach my $item (@tophalf) {
+ $tmp_config{$item} = $good_configs{$item};
}
- if (!$runtest && $len_diff > 0) {
+ $runtest = process_new_config \%tmp_config, \%new_configs,
+ \%good_configs, \%bad_configs, $outfile;
- if ($len_diff == 1) {
- process_failed $diff_arr[0];
- return 1;
- }
+ if (!$runtest) {
my %tmp_config = %bad_configs;
- my $half = int($#diff_arr / 2);
- my @tophalf = @diff_arr[0 .. $half];
+ print "Try bottom half\n";
- print "Settings bisect with top half:\n";
- print "Set tmp config to be bad config with some good config values\n";
- foreach my $item (@tophalf) {
+ my @bottomhalf = @diff_arr[$half+1 .. $#diff_arr];
+
+ foreach my $item (@bottomhalf) {
$tmp_config{$item} = $good_configs{$item};
}
$runtest = process_new_config \%tmp_config, \%new_configs,
- \%good_configs, \%bad_configs, $outfile;
-
- if (!$runtest) {
- my %tmp_config = %bad_configs;
-
- print "Try bottom half\n";
-
- my @bottomhalf = @diff_arr[$half+1 .. $#diff_arr];
-
- foreach my $item (@bottomhalf) {
- $tmp_config{$item} = $good_configs{$item};
- }
-
- $runtest = process_new_config \%tmp_config,
- \%new_configs, \%good_configs,
- \%bad_configs, $outfile;
- }
+ \%good_configs, \%bad_configs, $outfile;
}
-
}
sub cb_by_file {
--
2.9.4
^ permalink raw reply related [flat|nested] 15+ messages in thread* [PATCH 6/8] ktest/config-bisect: Try harder to find a new config
2017-07-17 0:16 [PATCH 1/8] ktest: Clarify config file usage Scott Wood
` (3 preceding siblings ...)
2017-07-17 0:16 ` [PATCH 5/8] ktest/config_bisect: Simplify " Scott Wood
@ 2017-07-17 0:16 ` Scott Wood
2017-07-17 0:16 ` [PATCH 7/8] ktest: Add simple config-bisect frontend Scott Wood
2017-07-17 0:16 ` [PATCH 8/8] ktest: Use config-bisect.pl in ktest.pl Scott Wood
6 siblings, 0 replies; 15+ messages in thread
From: Scott Wood @ 2017-07-17 0:16 UTC (permalink / raw)
To: Steven Rostedt; +Cc: linux-kernel, Scott Wood
From: Scott Wood <oss@buserror.net>
It's possible that the arbitrarily chosen half of the config differences
fails to make any actual difference in the output config, due to options
that are present in .config but not independently changeable (e.g. the
mutually-exclusive symbols of a "choice" kconfig construct, or a symbol
that has been "select"ed by another).
Try harder to generate a new and different config by randomizing the key
order and bisection percentage, and repeating a few times.
Explicitly randomizing also makes it possible to skip a broken generated
config by simply calling the script again, even on older versions of Perl
that didn't automatically randomize hash order.
Signed-off-by: Scott Wood <swood@redhat.com>
---
tools/testing/ktest/config-bisect.pl | 41 +++++++++++++++++++++---------------
1 file changed, 24 insertions(+), 17 deletions(-)
diff --git a/tools/testing/ktest/config-bisect.pl b/tools/testing/ktest/config-bisect.pl
index 63a3a940b5de..54f27b45079f 100755
--- a/tools/testing/ktest/config-bisect.pl
+++ b/tools/testing/ktest/config-bisect.pl
@@ -5,6 +5,7 @@
#
use strict;
+use List::Util qw(shuffle);
my $outputdir;
@@ -153,9 +154,11 @@ sub run_config_bisect {
my @diff_arr = diff_config_vals \%good_configs, \%bad_configs;
my $len_diff = $#diff_arr + 1;
- my $runtest = 1;
+ my $rand_tries = 10;
+ my $runtest = 0;
my %new_configs;
my $ret;
+ my $count = 0;
print "d=$len_diff\n";
@@ -164,33 +167,37 @@ sub run_config_bisect {
exit 2;
}
- my %tmp_config = %bad_configs;
-
- my $half = int($#diff_arr / 2);
- my @tophalf = @diff_arr[0 .. $half];
-
- print "Settings bisect with top half:\n";
- foreach my $item (@tophalf) {
- $tmp_config{$item} = $good_configs{$item};
- }
+ while ($runtest == 0 && ++$count <= $rand_tries) {
+ my %tmp_config = %bad_configs;
- $runtest = process_new_config \%tmp_config, \%new_configs,
- \%good_configs, \%bad_configs, $outfile;
+ # If we fail to generate a new config (due to the top half
+ # configs being unchangeable without the bottom half configs)
+ # then try a few random permutations, and if those fail try
+ # each option one at a time.
- if (!$runtest) {
- my %tmp_config = %bad_configs;
+ if ($count <= $rand_tries) {
+ @diff_arr = shuffle(@diff_arr);
+ }
- print "Try bottom half\n";
+ my $half = int($#diff_arr / 2);
+ if ($count > 2) {
+ $half = int(rand($#diff_arr));
+ }
- my @bottomhalf = @diff_arr[$half+1 .. $#diff_arr];
+ my @tophalf = @diff_arr[0 .. $half];
- foreach my $item (@bottomhalf) {
+ foreach my $item (@tophalf) {
$tmp_config{$item} = $good_configs{$item};
}
$runtest = process_new_config \%tmp_config, \%new_configs,
\%good_configs, \%bad_configs, $outfile;
}
+
+ if ($runtest == 0) {
+ print "$0: No more bisecting possible\n";
+ exit 2;
+ }
}
sub cb_by_file {
--
2.9.4
^ permalink raw reply related [flat|nested] 15+ messages in thread* [PATCH 7/8] ktest: Add simple config-bisect frontend
2017-07-17 0:16 [PATCH 1/8] ktest: Clarify config file usage Scott Wood
` (4 preceding siblings ...)
2017-07-17 0:16 ` [PATCH 6/8] ktest/config-bisect: Try harder to find a new config Scott Wood
@ 2017-07-17 0:16 ` Scott Wood
2017-07-17 0:16 ` [PATCH 8/8] ktest: Use config-bisect.pl in ktest.pl Scott Wood
6 siblings, 0 replies; 15+ messages in thread
From: Scott Wood @ 2017-07-17 0:16 UTC (permalink / raw)
To: Steven Rostedt; +Cc: linux-kernel, Scott Wood
From: Scott Wood <oss@buserror.net>
Add a friendly, git-bisect-like frontend to config-bisect.pl. Unlike
ktest.pl, this frontend requires no configuration beyond specifying
the kernel output directory and (if cross-compiling) $ARCH, and does not
need to run continuously for the entire bisection (and thus can be used
to bisect on the machine testing the kernels).
Signed-off-by: Scott Wood <swood@redhat.com>
---
Is the ktest directory the right place for this, or should it (along with
config-bisect.pl) go elsewhere since it no longer depends on the ktest
infrastructure? If so, where?
tools/testing/ktest/config-bisect.sh | 120 +++++++++++++++++++++++++++++++++++
1 file changed, 120 insertions(+)
create mode 100755 tools/testing/ktest/config-bisect.sh
diff --git a/tools/testing/ktest/config-bisect.sh b/tools/testing/ktest/config-bisect.sh
new file mode 100755
index 000000000000..6032e6736109
--- /dev/null
+++ b/tools/testing/ktest/config-bisect.sh
@@ -0,0 +1,120 @@
+#!/bin/bash
+
+BACKEND=$(dirname $BASH_SOURCE)/config-bisect.pl
+
+usage() {
+ echo Usage:
+ echo "$0 <outputdir> command [args]"
+ echo commands:
+ echo reset
+ echo init
+ echo diff
+ echo bad [conf]
+ echo good [conf]
+ echo skip
+ echo If conf is unspecified, \".config\" is used.
+ echo If cross-compiling, pass the ARCH environment variable.
+}
+
+reset() {
+ rm -rf $STATE
+}
+
+init() {
+ reset
+ mkdir -p $STATE/bad
+ mkdir -p $STATE/good
+ echo 0 > $STATE/good/idx
+ echo 0 > $STATE/bad/idx
+}
+
+next() {
+ if ! [ -d $STATE ]; then
+ echo $0: No config-bisect in progress -- initializing
+ init
+ fi
+
+ IDX=$(cat $STATE/$1/idx)
+ IDX=$(expr $IDX + 1)
+
+ CONF=$2
+ if [ -z "$CONF" ]; then
+ CONF=$O/.config
+ fi
+
+ cp $CONF $STATE/$1/$IDX
+ echo $IDX > $STATE/$1/idx
+}
+
+show_diff() {
+ GI=$(cat $STATE/good/idx)
+ BI=$(cat $STATE/bad/idx)
+
+ if [ $GI != 0 ] && [ $BI != 0 ]; then
+ diff -u $STATE/good/$GI $STATE/bad/$BI
+ else
+ echo $0: cannot diff without at least one good and one bad
+ fi
+}
+
+genconf() {
+ GI=$(cat $STATE/good/idx)
+ BI=$(cat $STATE/bad/idx)
+
+ echo good index $GI, bad index $BI
+
+ if [ $GI != 0 ] && [ $BI != 0 ]; then
+ $BACKEND $O $STATE/good/$GI $STATE/bad/$BI
+
+ case $? in
+ 0)
+ ;;
+ 2)
+ echo Failing config diff:
+ show_diff
+ ;;
+ *)
+ echo $0: error in backend
+ exit 1;
+ ;;
+ esac
+ fi
+}
+
+if [ -z "$1" -o -z "$2" ]; then
+ usage
+ exit 1
+fi
+
+O=$1
+STATE=$O/.config-bisect
+
+case $2 in
+init)
+ init
+ ;;
+reset)
+ reset
+ ;;
+diff)
+ show_diff
+ ;;
+bad)
+ next bad $3
+ genconf
+ ;;
+good)
+ next good $3
+ genconf
+ ;;
+skip)
+ # The options chosen are randomized, so we'll get a different
+ # config just by re-running the config bisect backend with the
+ # same inputs.
+ genconf
+ ;;
+*)
+ usage
+ exit 1
+ ;;
+esac
--
2.9.4
^ permalink raw reply related [flat|nested] 15+ messages in thread* [PATCH 8/8] ktest: Use config-bisect.pl in ktest.pl
2017-07-17 0:16 [PATCH 1/8] ktest: Clarify config file usage Scott Wood
` (5 preceding siblings ...)
2017-07-17 0:16 ` [PATCH 7/8] ktest: Add simple config-bisect frontend Scott Wood
@ 2017-07-17 0:16 ` Scott Wood
2017-09-14 21:41 ` Scott Wood
2017-10-04 19:18 ` Steven Rostedt
6 siblings, 2 replies; 15+ messages in thread
From: Scott Wood @ 2017-07-17 0:16 UTC (permalink / raw)
To: Steven Rostedt; +Cc: linux-kernel, Scott Wood
Reduce code duplication and take advantage of bisection logic
improvements by calling config-bisect.pl.
The output of make oldconfig is now copied directly to the desired file,
rather than doing assign_configs+save_config, in order to preserve the
ordering so that diffing the configs at the end will provide useful
output.
Signed-off-by: Scott Wood <swood@redhat.com>
---
tools/testing/ktest/ktest.pl | 247 +++++--------------------------------------
1 file changed, 29 insertions(+), 218 deletions(-)
diff --git a/tools/testing/ktest/ktest.pl b/tools/testing/ktest/ktest.pl
index 7d537bdf462b..cd1f4b5b76b7 100755
--- a/tools/testing/ktest/ktest.pl
+++ b/tools/testing/ktest/ktest.pl
@@ -3086,76 +3086,6 @@ sub create_config {
make_oldconfig;
}
-# compare two config hashes, and return configs with different vals.
-# It returns B's config values, but you can use A to see what A was.
-sub diff_config_vals {
- my ($pa, $pb) = @_;
-
- # crappy Perl way to pass in hashes.
- my %a = %{$pa};
- my %b = %{$pb};
-
- my %ret;
-
- foreach my $item (keys %a) {
- if (defined($b{$item}) && $b{$item} ne $a{$item}) {
- $ret{$item} = $b{$item};
- }
- }
-
- return %ret;
-}
-
-# compare two config hashes and return the configs in B but not A
-sub diff_configs {
- my ($pa, $pb) = @_;
-
- my %ret;
-
- # crappy Perl way to pass in hashes.
- my %a = %{$pa};
- my %b = %{$pb};
-
- foreach my $item (keys %b) {
- if (!defined($a{$item})) {
- $ret{$item} = $b{$item};
- }
- }
-
- return %ret;
-}
-
-# return if two configs are equal or not
-# 0 is equal +1 b has something a does not
-# +1 if a and b have a different item.
-# -1 if a has something b does not
-sub compare_configs {
- my ($pa, $pb) = @_;
-
- my %ret;
-
- # crappy Perl way to pass in hashes.
- my %a = %{$pa};
- my %b = %{$pb};
-
- foreach my $item (keys %b) {
- if (!defined($a{$item})) {
- return 1;
- }
- if ($a{$item} ne $b{$item}) {
- return 1;
- }
- }
-
- foreach my $item (keys %a) {
- if (!defined($b{$item})) {
- return -1;
- }
- }
-
- return 0;
-}
-
sub run_config_bisect_test {
my ($type) = @_;
@@ -3168,166 +3098,51 @@ sub run_config_bisect_test {
return $ret;
}
-sub process_failed {
- my ($config) = @_;
+sub config_bisect_end {
+ my ($good, $bad) = @_;
doprint "\n\n***************************************\n";
- doprint "Found bad config: $config\n";
+ doprint "No more config bisecting possible.\n";
+ doprint `diff -u $good $bad`;
doprint "***************************************\n\n";
}
-# used for config bisecting
-my $good_config;
-my $bad_config;
-
-sub process_new_config {
- my ($tc, $nc, $gc, $bc) = @_;
-
- my %tmp_config = %{$tc};
- my %good_configs = %{$gc};
- my %bad_configs = %{$bc};
-
- my %new_configs;
-
- my $runtest = 1;
- my $ret;
-
- create_config "tmp_configs", \%tmp_config;
- assign_configs \%new_configs, $output_config;
-
- $ret = compare_configs \%new_configs, \%bad_configs;
- if (!$ret) {
- doprint "New config equals bad config, try next test\n";
- $runtest = 0;
- }
-
- if ($runtest) {
- $ret = compare_configs \%new_configs, \%good_configs;
- if (!$ret) {
- doprint "New config equals good config, try next test\n";
- $runtest = 0;
- }
- }
-
- %{$nc} = %new_configs;
-
- return $runtest;
-}
-
sub run_config_bisect {
- my ($pgood, $pbad) = @_;
-
- my $type = $config_bisect_type;
-
- my %good_configs = %{$pgood};
- my %bad_configs = %{$pbad};
-
- my %diff_configs = diff_config_vals \%good_configs, \%bad_configs;
- my %b_configs = diff_configs \%good_configs, \%bad_configs;
- my %g_configs = diff_configs \%bad_configs, \%good_configs;
-
- my @diff_arr = keys %diff_configs;
- my $len_diff = $#diff_arr + 1;
-
- my @b_arr = keys %b_configs;
- my $len_b = $#b_arr + 1;
-
- my @g_arr = keys %g_configs;
- my $len_g = $#g_arr + 1;
-
- my $runtest = 1;
- my %new_configs;
+ my ($good, $bad) = @_;
+ my $cmd;
my $ret;
- # First, lets get it down to a single subset.
- # Is the problem with a difference in values?
- # Is the problem with a missing config?
- # Is the problem with a config that breaks things?
-
- # Enable all of one set and see if we get a new bad
- # or good config.
+ run_command "$builddir/tools/testing/ktest/config-bisect.pl $outputdir $good $bad";
- # first set the good config to the bad values.
-
- doprint "d=$len_diff g=$len_g b=$len_b\n";
-
- # first lets enable things in bad config that are enabled in good config
-
- if ($len_diff > 0) {
- if ($len_b > 0 || $len_g > 0) {
- my %tmp_config = %bad_configs;
-
- doprint "Set tmp config to be bad config with good config values\n";
- foreach my $item (@diff_arr) {
- $tmp_config{$item} = $good_configs{$item};
- }
-
- $runtest = process_new_config \%tmp_config, \%new_configs,
- \%good_configs, \%bad_configs;
- }
+ if ($run_command_status == 2) {
+ config_bisect_end $good, $bad;
+ return 1;
}
- if (!$runtest && $len_diff > 0) {
-
- if ($len_diff == 1) {
- process_failed $diff_arr[0];
- return 1;
- }
- my %tmp_config = %bad_configs;
-
- my $half = int($#diff_arr / 2);
- my @tophalf = @diff_arr[0 .. $half];
-
- doprint "Settings bisect with top half:\n";
- doprint "Set tmp config to be bad config with some good config values\n";
- foreach my $item (@tophalf) {
- $tmp_config{$item} = $good_configs{$item};
- }
-
- $runtest = process_new_config \%tmp_config, \%new_configs,
- \%good_configs, \%bad_configs;
-
- if (!$runtest) {
- my %tmp_config = %bad_configs;
-
- doprint "Try bottom half\n";
-
- my @bottomhalf = @diff_arr[$half+1 .. $#diff_arr];
-
- foreach my $item (@bottomhalf) {
- $tmp_config{$item} = $good_configs{$item};
- }
-
- $runtest = process_new_config \%tmp_config, \%new_configs,
- \%good_configs, \%bad_configs;
- }
+ if ($run_command_status != 0) {
+ dodie "config-bisect.pl failed";
}
- if ($runtest) {
- $ret = run_config_bisect_test $type;
- if ($ret) {
- doprint "NEW GOOD CONFIG\n";
- %good_configs = %new_configs;
- run_command "mv $good_config ${good_config}.last";
- save_config \%good_configs, $good_config;
- %{$pgood} = %good_configs;
- } else {
- doprint "NEW BAD CONFIG\n";
- %bad_configs = %new_configs;
- run_command "mv $bad_config ${bad_config}.last";
- save_config \%bad_configs, $bad_config;
- %{$pbad} = %bad_configs;
- }
- return 0;
+ $ret = run_config_bisect_test $config_bisect_type;
+ if ($ret) {
+ doprint "NEW GOOD CONFIG\n";
+ $cmd = "mv $good ${good}.last && mv $output_config $good";
+ } else {
+ doprint "NEW BAD CONFIG\n";
+ $cmd = "mv $bad ${bad}.last && mv $output_config $bad";
}
- fail "Hmm, need to do a mix match?\n";
- return -1;
+ system($cmd) == 0 or dodie $cmd;
+
+ return 0;
}
sub config_bisect {
my ($i) = @_;
+ my $good_config;
+ my $bad_config;
+
my $type = $config_bisect_type;
my $ret;
@@ -3358,18 +3173,14 @@ sub config_bisect {
doprint "Run good configs through make oldconfig\n";
assign_configs \%tmp_configs, $good_config;
create_config "$good_config", \%tmp_configs;
- assign_configs \%good_configs, $output_config;
+ $good_config = "$tmpdir/good_config";
+ system("cp $output_config $good_config") == 0 or dodie "cp good config";
doprint "Run bad configs through make oldconfig\n";
assign_configs \%tmp_configs, $bad_config;
create_config "$bad_config", \%tmp_configs;
- assign_configs \%bad_configs, $output_config;
-
- $good_config = "$tmpdir/good_config";
$bad_config = "$tmpdir/bad_config";
-
- save_config \%good_configs, $good_config;
- save_config \%bad_configs, $bad_config;
+ system("cp $output_config $bad_config") == 0 or dodie "cp bad config";
if (defined($config_bisect_check) && $config_bisect_check ne "0") {
if ($config_bisect_check ne "good") {
@@ -3393,7 +3204,7 @@ sub config_bisect {
}
do {
- $ret = run_config_bisect \%good_configs, \%bad_configs;
+ $ret = run_config_bisect $good_config, $bad_config;
print_times;
} while (!$ret);
--
2.9.4
^ permalink raw reply related [flat|nested] 15+ messages in thread* Re: [PATCH 8/8] ktest: Use config-bisect.pl in ktest.pl
2017-07-17 0:16 ` [PATCH 8/8] ktest: Use config-bisect.pl in ktest.pl Scott Wood
@ 2017-09-14 21:41 ` Scott Wood
2017-10-04 19:17 ` Steven Rostedt
2017-10-04 19:18 ` Steven Rostedt
1 sibling, 1 reply; 15+ messages in thread
From: Scott Wood @ 2017-09-14 21:41 UTC (permalink / raw)
To: Steven Rostedt; +Cc: linux-kernel
On Sun, 2017-07-16 at 19:16 -0500, Scott Wood wrote:
> Reduce code duplication and take advantage of bisection logic
> improvements by calling config-bisect.pl.
>
> The output of make oldconfig is now copied directly to the desired
> file,
> rather than doing assign_configs+save_config, in order to preserve the
> ordering so that diffing the configs at the end will provide useful
> output.
>
> Signed-off-by: Scott Wood <swood@redhat.com>
> ---
> tools/testing/ktest/ktest.pl | 247 +++++-----------------------------
Any comments on this patchset?
-Scott
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [PATCH 8/8] ktest: Use config-bisect.pl in ktest.pl
2017-09-14 21:41 ` Scott Wood
@ 2017-10-04 19:17 ` Steven Rostedt
0 siblings, 0 replies; 15+ messages in thread
From: Steven Rostedt @ 2017-10-04 19:17 UTC (permalink / raw)
To: Scott Wood; +Cc: linux-kernel
On Thu, 14 Sep 2017 14:41:12 -0700
Scott Wood <swood@redhat.com> wrote:
> On Sun, 2017-07-16 at 19:16 -0500, Scott Wood wrote:
> > Reduce code duplication and take advantage of bisection logic
> > improvements by calling config-bisect.pl.
> >
> > The output of make oldconfig is now copied directly to the desired
> > file,
> > rather than doing assign_configs+save_config, in order to preserve the
> > ordering so that diffing the configs at the end will provide useful
> > output.
> >
> > Signed-off-by: Scott Wood <swood@redhat.com>
> > ---
> > tools/testing/ktest/ktest.pl | 247 +++++-----------------------------
>
> Any comments on this patchset?
Finally got around to looking at it. Sorry for the late reply.
I didn't see anything to concerning in patches 1-7 and applied them.
I'll start testing them soon too.
-- Steve
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [PATCH 8/8] ktest: Use config-bisect.pl in ktest.pl
2017-07-17 0:16 ` [PATCH 8/8] ktest: Use config-bisect.pl in ktest.pl Scott Wood
2017-09-14 21:41 ` Scott Wood
@ 2017-10-04 19:18 ` Steven Rostedt
2017-10-04 20:24 ` Scott Wood
1 sibling, 1 reply; 15+ messages in thread
From: Steven Rostedt @ 2017-10-04 19:18 UTC (permalink / raw)
To: Scott Wood; +Cc: linux-kernel
On Sun, 16 Jul 2017 19:16:30 -0500
Scott Wood <swood@redhat.com> wrote:
> Reduce code duplication and take advantage of bisection logic
> improvements by calling config-bisect.pl.
>
> The output of make oldconfig is now copied directly to the desired file,
> rather than doing assign_configs+save_config, in order to preserve the
> ordering so that diffing the configs at the end will provide useful
> output.
The reason I never did this, was that I copy ktest.pl all over the
place :-/ I need it to be a stand alone. I don't copy config-bisect
around. Not sure how to deal with this.
-- Steve
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [PATCH 8/8] ktest: Use config-bisect.pl in ktest.pl
2017-10-04 19:18 ` Steven Rostedt
@ 2017-10-04 20:24 ` Scott Wood
2017-10-05 12:50 ` Steven Rostedt
0 siblings, 1 reply; 15+ messages in thread
From: Scott Wood @ 2017-10-04 20:24 UTC (permalink / raw)
To: Steven Rostedt; +Cc: linux-kernel
On Wed, 2017-10-04 at 15:18 -0400, Steven Rostedt wrote:
> On Sun, 16 Jul 2017 19:16:30 -0500
> Scott Wood <swood@redhat.com> wrote:
>
> > Reduce code duplication and take advantage of bisection logic
> > improvements by calling config-bisect.pl.
> >
> > The output of make oldconfig is now copied directly to the desired
> > file,
> > rather than doing assign_configs+save_config, in order to preserve
> > the
> > ordering so that diffing the configs at the end will provide useful
> > output.
>
> The reason I never did this, was that I copy ktest.pl all over the
> place :-/ I need it to be a stand alone. I don't copy config-bisect
> around. Not sure how to deal with this.
By "this" I assume you mean using the external config-bisect code,
rather than the part about copying oldconfig output?
The options I can see are to copy both files as a group (possibly making
that easier by renaming config-bisect.pl to ktest-conf-bisect.pl, so you
can "cp tools/testing/ktest/ktest*.pl <dest>"), or just living with code
duplication. In a previous discussion you suggested you preferred that
latter option, though in that case the config bisect logic changes
should ideally be done before the fork.
It should also be noted that ktest.pl only depends on config-bisect.pl
if a config bisect is being performed, so other ktest.pl functions still
work standalone.
-Scott
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [PATCH 8/8] ktest: Use config-bisect.pl in ktest.pl
2017-10-04 20:24 ` Scott Wood
@ 2017-10-05 12:50 ` Steven Rostedt
2017-10-05 23:18 ` Scott Wood
0 siblings, 1 reply; 15+ messages in thread
From: Steven Rostedt @ 2017-10-05 12:50 UTC (permalink / raw)
To: Scott Wood; +Cc: linux-kernel
On Wed, 04 Oct 2017 15:24:23 -0500
Scott Wood <swood@redhat.com> wrote:
> On Wed, 2017-10-04 at 15:18 -0400, Steven Rostedt wrote:
> > On Sun, 16 Jul 2017 19:16:30 -0500
> > Scott Wood <swood@redhat.com> wrote:
> >
> > > Reduce code duplication and take advantage of bisection logic
> > > improvements by calling config-bisect.pl.
> > >
> > > The output of make oldconfig is now copied directly to the desired
> > > file,
> > > rather than doing assign_configs+save_config, in order to preserve
> > > the
> > > ordering so that diffing the configs at the end will provide useful
> > > output.
> >
> > The reason I never did this, was that I copy ktest.pl all over the
> > place :-/ I need it to be a stand alone. I don't copy config-bisect
> > around. Not sure how to deal with this.
>
> By "this" I assume you mean using the external config-bisect code,
> rather than the part about copying oldconfig output?
>
> The options I can see are to copy both files as a group (possibly making
> that easier by renaming config-bisect.pl to ktest-conf-bisect.pl, so you
> can "cp tools/testing/ktest/ktest*.pl <dest>"), or just living with code
> duplication. In a previous discussion you suggested you preferred that
> latter option, though in that case the config bisect logic changes
> should ideally be done before the fork.
I don't want to rename it. I like the simply "config-bisect.pl" name.
>
> It should also be noted that ktest.pl only depends on config-bisect.pl
> if a config bisect is being performed, so other ktest.pl functions still
> work standalone.
I thought about this too, and may be able to live with that. Perhaps we
should make the config-bisect internals into something that can be a
perl library, and be able to place that someplace that both could use?
-- Steve
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [PATCH 8/8] ktest: Use config-bisect.pl in ktest.pl
2017-10-05 12:50 ` Steven Rostedt
@ 2017-10-05 23:18 ` Scott Wood
2017-10-05 19:43 ` Steven Rostedt
0 siblings, 1 reply; 15+ messages in thread
From: Scott Wood @ 2017-10-05 23:18 UTC (permalink / raw)
To: Steven Rostedt; +Cc: linux-kernel
On Thu, 2017-10-05 at 08:50 -0400, Steven Rostedt wrote:
> On Wed, 04 Oct 2017 15:24:23 -0500
> Scott Wood <swood@redhat.com> wrote:
>
> > It should also be noted that ktest.pl only depends on config-
> > bisect.pl
> > if a config bisect is being performed, so other ktest.pl functions
> > still
> > work standalone.
>
> I thought about this too, and may be able to live with that. Perhaps
> we
> should make the config-bisect internals into something that can be a
> perl library, and be able to place that someplace that both could use?
I'm not familiar with how Perl libraries work -- how would they solve
this problem? If you're talking about having to install the library
into some location outside the kernel tree, that seems contrary to the
desire to have a lightweight tool that can be copied around. If not,
how would it be any better than depending on a callable script?
-Scott
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [PATCH 8/8] ktest: Use config-bisect.pl in ktest.pl
2017-10-05 23:18 ` Scott Wood
@ 2017-10-05 19:43 ` Steven Rostedt
0 siblings, 0 replies; 15+ messages in thread
From: Steven Rostedt @ 2017-10-05 19:43 UTC (permalink / raw)
To: Scott Wood; +Cc: linux-kernel
On Thu, 05 Oct 2017 18:18:54 -0500
Scott Wood <swood@redhat.com> wrote:
> On Thu, 2017-10-05 at 08:50 -0400, Steven Rostedt wrote:
> > On Wed, 04 Oct 2017 15:24:23 -0500
> > Scott Wood <swood@redhat.com> wrote:
> >
> > > It should also be noted that ktest.pl only depends on config-
> > > bisect.pl
> > > if a config bisect is being performed, so other ktest.pl functions
> > > still
> > > work standalone.
> >
> > I thought about this too, and may be able to live with that. Perhaps
> > we
> > should make the config-bisect internals into something that can be a
> > perl library, and be able to place that someplace that both could use?
>
> I'm not familiar with how Perl libraries work -- how would they solve
> this problem? If you're talking about having to install the library
> into some location outside the kernel tree, that seems contrary to the
> desire to have a lightweight tool that can be copied around. If not,
> how would it be any better than depending on a callable script?
>
I'm thinking about installing it in the tree, and then using variables
to find it. I'm not sure how perl works either, but as an interpreted
language, I'm sure there's gotta be a way to add default paths to look
for it. Perhaps there's a way to include other perl scripts.
I'll have to play around with this to come up with better ideas.
-- Steve
^ permalink raw reply [flat|nested] 15+ messages in thread