Kconfig test scripts

Kconfig test scripts

[Sam Ravnborg]

Hi all.

Often when I hack on Kconfig I have missed a set of test cases,
that would allow me to verify that I did not introduce any regressions.

I have not anything fancy in my mind and I spent
a little time on the attached today.
The following is obviously missing a lot a features - but this
would allow me to get started adding simple test cases.

The idea is that each test cases consist of a full
Kconfig file and the resulting output.

Anyone have something better than this,
or maybe some brilliant ideas how to do this much better?

	Sam

diff --git a/scripts/kconfig/tests/run.sh b/scripts/kconfig/tests/run.sh
new file mode 100644
index 0000000..fbd7613
--- /dev/null
+++ b/scripts/kconfig/tests/run.sh
@@ -0,0 +1,20 @@
+# set up test environment
+mkdir -p include/config
+mkdir -p include/generated
+
+tests="single_symbol.sh two_symbols.sh"
+
+
+for t in ${tests}; do
+
+	rm -f dot_config.expect
+
+	sh $t
+
+	if [ -f dot_config.expect ]; then
+		grep -v ^# .config > dot_config.actual
+		if ! cmp -s dot_config.actual dot_config.expect ; then
+			diff -u dot_config.expect dot_config.actual
+		fi
+	fi
+done
diff --git a/scripts/kconfig/tests/single_symbol.sh b/scripts/kconfig/tests/single_symbol.sh
new file mode 100644
index 0000000..6459fcf
--- /dev/null
+++ b/scripts/kconfig/tests/single_symbol.sh
@@ -0,0 +1,11 @@
+cat << EOF > Kconfig.test
+config SINGLE
+	def_bool y
+
+EOF
+
+cat << EOF > dot_config.expect
+CONFIG_SINGLE=y
+EOF
+
+../conf Kconfig.test > /dev/null
diff --git a/scripts/kconfig/tests/two_symbols.sh b/scripts/kconfig/tests/two_symbols.sh
new file mode 100644
index 0000000..c31902d
--- /dev/null
+++ b/scripts/kconfig/tests/two_symbols.sh
@@ -0,0 +1,14 @@
+cat << EOF > Kconfig.test
+config FIRST
+	def_bool y
+
+config SECOND
+	def_bool y
+EOF
+
+cat << EOF > dot_config.expect
+CONFIG_FIRST=y
+CONFIG_SECOND=y
+EOF
+
+../conf Kconfig.test > /dev/null

Re: Kconfig test scripts

[Yann E. MORIN]

Sam, All,

On 2013-07-20 16:19 +0200, Sam Ravnborg spake thusly:
> Often when I hack on Kconfig I have missed a set of test cases,
> that would allow me to verify that I did not introduce any regressions.
> 
> I have not anything fancy in my mind and I spent
> a little time on the attached today.
> The following is obviously missing a lot a features - but this
> would allow me to get started adding simple test cases.
> 
> The idea is that each test cases consist of a full
> Kconfig file and the resulting output.
> 
> Anyone have something better than this,
> or maybe some brilliant ideas how to do this much better?

How would one express a test for 'choice'? Test randconfig?

I'm all for a test-suite. But we need to be able to test two things:
  - the parser
  - the behaviour

For the first, your static tests are doing (can do) fine.
For the second, we'd need something a bit more sophisticated.

For example, we need a way to tell that one of such or such symbol is
expected, but not both at the same time. Eg, for a choice between A and
B:
    CONFIG_A=y | CONFIG_B=y     (inclusive OR, for tristate choice)
or:
    CONFIG_A=y ^ CONFIG_B=y     (exclusive OR, for boolean choice)

Also, for randconfig, we'd need to be able to repeatedly run a test and
check symbols probability. Eg:
    CONFIG_A=y [25] ^ CONFIG_B=y [75]

Of course, allow for a slight distortion of a few percent to account for
bias in the RNG).

No idea (yet) on how to do that...

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

Re: Kconfig test scripts

[Sam Ravnborg]

> 
> How would one express a test for 'choice'? Test randconfig?

I made a minimal test for choice - see below.
Each test is an independent shell script - so we can in each shell script
do whatever we which to do.

This is still WIP - I will let you know when I have something 
that I assume is ready.

I am away for a week - so do not expect prompt replies.

	Sam

diff --git a/scripts/kconfig/tests/rand_two_symbols.sh b/scripts/kconfig/tests/rand_two_symbols.sh
new file mode 100644
index 0000000..ce5c482
--- /dev/null
+++ b/scripts/kconfig/tests/rand_two_symbols.sh
@@ -0,0 +1,30 @@
+#!/bin/sh
+#Randconfig with two symbols
+
+cat << EOF > Kconfig.test
+config A
+	bool "A"
+
+config B
+	bool "A"
+EOF
+
+rm -f dot_config_y.all
+
+for i in $(seq 1 10); do
+	KCONFIG_SEED=$i
+	export KCONFIG_SEED
+	../conf Kconfig.test --randconfig > /dev/null
+	grep "=y" .config >> dot_config_y.all
+done
+
+acount=$(grep A=y dot_config_y.all | wc -l)
+bcount=$(grep B=y dot_config_y.all | wc -l)
+
+if [ "${acount}" != "" -a ${bcount} != "" ]; then
+	if [ ${acount} -gt 0 -a ${bcount} -gt 0 ]; then
+		exit 0
+	fi
+fi
+
+exit 1
diff --git a/scripts/kconfig/tests/run.sh b/scripts/kconfig/tests/run.sh
new file mode 100644
index 0000000..5f3a6ff
--- /dev/null
+++ b/scripts/kconfig/tests/run.sh
@@ -0,0 +1,40 @@
+#!/bin/sh
+# set up test environment
+mkdir -p include/config
+mkdir -p include/generated
+
+tests="single_symbol.sh two_symbols.sh rand_two_symbols.sh"
+failed=0
+passed=0
+
+for t in ${tests}; do
+
+	rm -f dot_config.expect
+
+	echo Testing: $(head -n 1 $t | cut -d '#' -f 2-)
+
+	/bin/sh $t
+
+	fail=$?
+
+	if [ -f dot_config.expect ]; then
+		grep -v ^# .config > dot_config.actual
+		if ! cmp -s dot_config.actual dot_config.expect ; then
+			fail=1
+			echo FAILED:
+			diff -u dot_config.expect dot_config.actual
+		fi
+	fi
+
+	if [ ${fail} -gt 0 ]; then
+		failed=$(expr ${failed} + 1);
+	else
+		passed=$(expr ${passed} + 1);
+	fi
+done
+
+if [ ${failed} -gt 0 ]; then
+	echo Test cases failed: ${failed}
+fi
+
+echo Test cases passed: ${passed} out of $(expr ${passed} + ${failed})
diff --git a/scripts/kconfig/tests/single_symbol.sh b/scripts/kconfig/tests/single_symbol.sh
new file mode 100644
index 0000000..dc03a8c
--- /dev/null
+++ b/scripts/kconfig/tests/single_symbol.sh
@@ -0,0 +1,13 @@
+#Single symbol with default value equal y
+
+cat << EOF > Kconfig.test
+config SINGLE
+	def_bool y
+
+EOF
+
+cat << EOF > dot_config.expect
+CONFIG_SINGLE=y
+EOF
+
+../conf Kconfig.test > /dev/null
diff --git a/scripts/kconfig/tests/two_symbols.sh b/scripts/kconfig/tests/two_symbols.sh
new file mode 100644
index 0000000..7081c1d
--- /dev/null
+++ b/scripts/kconfig/tests/two_symbols.sh
@@ -0,0 +1,16 @@
+# Two symbols with default values equal y
+
+cat << EOF > Kconfig.test
+config FIRST
+	def_bool y
+
+config SECOND
+	def_bool y
+EOF
+
+cat << EOF > dot_config.expect
+CONFIG_FIRST=y
+CONFIG_SECOND=y
+EOF
+
+../conf Kconfig.test > /dev/null