[PATCH 00/14] Add Kconfig unit tests

[PATCH 00/14] Add Kconfig unit tests

[Masahiro Yamada]

I am applying various patches to Kconfig these days.

However, I fear regressions.  I have been thinking of unit-tests.

There are various cryptic parts in Kconfig and corner cases where
it is difficult to notice breakage.  If unit-tests cover those,
I will be able to apply changes more confidently.

So, here is the trial.

After fixing some problems, I will add a basic test framework.
This is based on pytest.  Also, this is written in Python 3.
Python 2 will return in 2020.  So, I believe new python tools should be
written in Python 3.

This is my Python 3 and pytest versions.

$ python3 --version
Python 3.5.2
$ python3 -m pytest --version
This is pytest version 3.4.0, imported from /home/masahiro/.local/lib/python3.5/site-packages/pytest.py

If I use old pytest version, some parts did not work as expected.
If this does not work for you, please consider using newer pytest.

I will brush up the code more and add more test cases to do a better job.
Before proceeding more, I'd like to get consensus for this approach.
If you have an idea for better implementation, comments are appreciated.

How to use?
-----------

Please make sure Python3 and pytest for Python3 is installed on your system.

Then, run "make testconfig"

The result looks like as follows:

masahiro@grover:~/workspace/linux-yamada$ make testconfig
  HOSTCC  scripts/kconfig/conf.o
  HOSTCC  scripts/kconfig/zconf.tab.o
  HOSTLD  scripts/kconfig/conf
python3 -B -m pytest ./scripts/kconfig/tests \
-o cache_dir=/home/masahiro/workspace/linux-yamada/scripts/kconfig/tests/.cache \

========================== test session starts ==========================platform linux -- Python 3.5.2, pytest-3.3.2, py-1.5.2, pluggy-0.6.0 -- /usr/bin/python3
cachedir: scripts/kconfig/tests/.cache
rootdir: /home/masahiro/workspace/linux-yamada/scripts/kconfig/tests, inifile: pytest.ini
collected 12 items

scripts/kconfig/tests/auto_submenu_creation/__init__.py::test PASSED                                                   [  8%]
scripts/kconfig/tests/choice/__init__.py::test_oldask0 PASSED                                                          [ 16%]
scripts/kconfig/tests/choice/__init__.py::test_oldask1 PASSED                                                          [ 25%]
scripts/kconfig/tests/choice/__init__.py::test_allyes PASSED                                                           [ 33%]
scripts/kconfig/tests/choice/__init__.py::test_allmod PASSED                                                           [ 41%]
scripts/kconfig/tests/choice/__init__.py::test_allno PASSED                                                            [ 50%]
scripts/kconfig/tests/choice/__init__.py::test_alldef PASSED                                                           [ 58%]
scripts/kconfig/tests/choice_value_with_m_dep/__init__.py::test PASSED                                                 [ 66%]
scripts/kconfig/tests/err_recursive_inc/__init__.py::test PASSED                                                       [ 75%]
scripts/kconfig/tests/new_choice_with_dep/__init__.py::test PASSED                                                     [ 83%]
scripts/kconfig/tests/no_write_if_dep_unmet/__init__.py::test PASSED                                                   [ 91%]
scripts/kconfig/tests/warn_recursive_dep/__init__.py::test PASSED                                                      [100%]

========================= 12 passed in 0.08 seconds =========================



Masahiro Yamada (14):
  kconfig: send error messages to stderr
  kconfig: do not write choice values when their dependency becomes n
  kconfig: show '?' prompt even if no help text is available
  kconfig: print additional new line for choice for redirection
  kconfig: remove 'config*' pattern from .gitignnore
  kbuild: define PYTHON2 and PYTHON3 variables instead of PYTHON
  kconfig: test: add framework for Kconfig unit-tests
  kconfig: test: add basic 'choice' tests
  kconfig: test: test automatic submenu creation
  kconfig: test: check if new symbols in choice are asked
  kconfig: test: check .config sanity for choice values with unmet dep
  kconfig: test: check visibility of tristate choice values in y choice
  kconfig: test: check if recursive dependencies are detected
  kconfig: test: check if recursive inclusion is detected

 Makefile                                           |   5 +-
 arch/ia64/Makefile                                 |   2 +-
 scripts/kconfig/.gitignore                         |   1 -
 scripts/kconfig/Makefile                           |   8 +
 scripts/kconfig/conf.c                             |  29 +--
 scripts/kconfig/symbol.c                           |   4 +-
 .../kconfig/tests/auto_submenu_creation/Kconfig    |  50 ++++
 .../tests/auto_submenu_creation/__init__.py        |  12 +
 .../tests/auto_submenu_creation/expected_stdout    |  10 +
 scripts/kconfig/tests/choice/Kconfig               |  54 +++++
 scripts/kconfig/tests/choice/__init__.py           |  37 +++
 .../kconfig/tests/choice/alldef_expected_config    |   5 +
 .../kconfig/tests/choice/allmod_expected_config    |   9 +
 scripts/kconfig/tests/choice/allno_expected_config |   5 +
 .../kconfig/tests/choice/allyes_expected_config    |   9 +
 .../kconfig/tests/choice/oldask0_expected_stdout   |  10 +
 scripts/kconfig/tests/choice/oldask1_config        |   2 +
 .../kconfig/tests/choice/oldask1_expected_stdout   |  15 ++
 .../kconfig/tests/choice_value_with_m_dep/Kconfig  |  20 ++
 .../tests/choice_value_with_m_dep/__init__.py      |  15 ++
 .../kconfig/tests/choice_value_with_m_dep/config   |   2 +
 .../tests/choice_value_with_m_dep/expected_config  |   3 +
 .../tests/choice_value_with_m_dep/expected_stdout  |   4 +
 scripts/kconfig/tests/conftest.py                  | 255 +++++++++++++++++++++
 scripts/kconfig/tests/err_recursive_inc/Kconfig    |   1 +
 .../kconfig/tests/err_recursive_inc/Kconfig.inc    |   1 +
 .../kconfig/tests/err_recursive_inc/__init__.py    |  10 +
 .../tests/err_recursive_inc/expected_stderr        |   4 +
 scripts/kconfig/tests/new_choice_with_dep/Kconfig  |  37 +++
 .../kconfig/tests/new_choice_with_dep/__init__.py  |  14 ++
 scripts/kconfig/tests/new_choice_with_dep/config   |   3 +
 .../tests/new_choice_with_dep/expected_stdout      |  10 +
 .../kconfig/tests/no_write_if_dep_unmet/Kconfig    |  14 ++
 .../tests/no_write_if_dep_unmet/__init__.py        |  17 ++
 scripts/kconfig/tests/no_write_if_dep_unmet/config |   1 +
 .../tests/no_write_if_dep_unmet/expected_config    |   5 +
 scripts/kconfig/tests/pytest.ini                   |   6 +
 scripts/kconfig/tests/warn_recursive_dep/Kconfig   |  62 +++++
 .../kconfig/tests/warn_recursive_dep/__init__.py   |  10 +
 .../tests/warn_recursive_dep/expected_stderr       |  33 +++
 scripts/kconfig/zconf.l                            |  27 ++-
 41 files changed, 789 insertions(+), 32 deletions(-)
 create mode 100644 scripts/kconfig/tests/auto_submenu_creation/Kconfig
 create mode 100644 scripts/kconfig/tests/auto_submenu_creation/__init__.py
 create mode 100644 scripts/kconfig/tests/auto_submenu_creation/expected_stdout
 create mode 100644 scripts/kconfig/tests/choice/Kconfig
 create mode 100644 scripts/kconfig/tests/choice/__init__.py
 create mode 100644 scripts/kconfig/tests/choice/alldef_expected_config
 create mode 100644 scripts/kconfig/tests/choice/allmod_expected_config
 create mode 100644 scripts/kconfig/tests/choice/allno_expected_config
 create mode 100644 scripts/kconfig/tests/choice/allyes_expected_config
 create mode 100644 scripts/kconfig/tests/choice/oldask0_expected_stdout
 create mode 100644 scripts/kconfig/tests/choice/oldask1_config
 create mode 100644 scripts/kconfig/tests/choice/oldask1_expected_stdout
 create mode 100644 scripts/kconfig/tests/choice_value_with_m_dep/Kconfig
 create mode 100644 scripts/kconfig/tests/choice_value_with_m_dep/__init__.py
 create mode 100644 scripts/kconfig/tests/choice_value_with_m_dep/config
 create mode 100644 scripts/kconfig/tests/choice_value_with_m_dep/expected_config
 create mode 100644 scripts/kconfig/tests/choice_value_with_m_dep/expected_stdout
 create mode 100644 scripts/kconfig/tests/conftest.py
 create mode 100644 scripts/kconfig/tests/err_recursive_inc/Kconfig
 create mode 100644 scripts/kconfig/tests/err_recursive_inc/Kconfig.inc
 create mode 100644 scripts/kconfig/tests/err_recursive_inc/__init__.py
 create mode 100644 scripts/kconfig/tests/err_recursive_inc/expected_stderr
 create mode 100644 scripts/kconfig/tests/new_choice_with_dep/Kconfig
 create mode 100644 scripts/kconfig/tests/new_choice_with_dep/__init__.py
 create mode 100644 scripts/kconfig/tests/new_choice_with_dep/config
 create mode 100644 scripts/kconfig/tests/new_choice_with_dep/expected_stdout
 create mode 100644 scripts/kconfig/tests/no_write_if_dep_unmet/Kconfig
 create mode 100644 scripts/kconfig/tests/no_write_if_dep_unmet/__init__.py
 create mode 100644 scripts/kconfig/tests/no_write_if_dep_unmet/config
 create mode 100644 scripts/kconfig/tests/no_write_if_dep_unmet/expected_config
 create mode 100644 scripts/kconfig/tests/pytest.ini
 create mode 100644 scripts/kconfig/tests/warn_recursive_dep/Kconfig
 create mode 100644 scripts/kconfig/tests/warn_recursive_dep/__init__.py
 create mode 100644 scripts/kconfig/tests/warn_recursive_dep/expected_stderr

-- 
2.7.4

[PATCH 06/14] kbuild: define PYTHON2 and PYTHON3 variables instead of PYTHON

[Masahiro Yamada]

The variable 'PYTHON' allows users to specify a proper executable
name in case the default 'python' does not work.  However, this does
not address the case where both Python 2 and Python 3 scripts are
used in one system.

PEP 394 (https://www.python.org/dev/peps/pep-0394/) provides a
convention for Python scripts portability.  Here is a quotation:

  In order to tolerate differences across platforms, all new code
  that needs to invoke the Python interpreter should not specify
  'python', but rather should specify either 'python2' or 'python3'.
  This distinction should be made in shebangs, when invoking from a
  shell script, when invoking via the system() call, or when invoking
  in any other context.

arch/ia64/scripts/unwcheck.py is apparently written in Python 2, so
it should be invoked by 'python2'.

It is legitimate to use 'python' for scripts compatible with both
Python 2 and Python 3, but this is rare (at least I do not see the
case in kernel tree).  You do not need to make efforts to write your
scripts in that way.  Anyway, Python 2 will retire in 2020.

This commit is needed for my new scripts written in Python 3.

Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
---

 Makefile           | 5 +++--
 arch/ia64/Makefile | 2 +-
 2 files changed, 4 insertions(+), 3 deletions(-)

diff --git a/Makefile b/Makefile
index 11aff0f..c4e935c 100644
--- a/Makefile
+++ b/Makefile
@@ -384,7 +384,8 @@ GENKSYMS	= scripts/genksyms/genksyms
 INSTALLKERNEL  := installkernel
 DEPMOD		= /sbin/depmod
 PERL		= perl
-PYTHON		= python
+PYTHON2		= python2
+PYTHON3		= python3
 CHECK		= sparse
 
 CHECKFLAGS     := -D__linux__ -Dlinux -D__STDC__ -Dunix -D__unix__ \
@@ -430,7 +431,7 @@ GCC_PLUGINS_CFLAGS : 
 export ARCH SRCARCH CONFIG_SHELL HOSTCC HOSTCFLAGS CROSS_COMPILE AS LD CC
 export CPP AR NM STRIP OBJCOPY OBJDUMP HOSTLDFLAGS HOST_LOADLIBES
-export MAKE LEX YACC AWK GENKSYMS INSTALLKERNEL PERL PYTHON UTS_MACHINE
+export MAKE LEX YACC AWK GENKSYMS INSTALLKERNEL PERL PYTHON2 PYTHON3 UTS_MACHINE
 export HOSTCXX HOSTCXXFLAGS LDFLAGS_MODULE CHECK CHECKFLAGS
 
 export KBUILD_CPPFLAGS NOSTDINC_FLAGS LINUXINCLUDE OBJCOPYFLAGS LDFLAGS
diff --git a/arch/ia64/Makefile b/arch/ia64/Makefile
index 2dd7f51..862a2ba 100644
--- a/arch/ia64/Makefile
+++ b/arch/ia64/Makefile
@@ -75,7 +75,7 @@ vmlinux.gz: vmlinux
 	$(Q)$(MAKE) $(build)=$(boot) $@
 
 unwcheck: vmlinux
-	-$(Q)READELF=$(READELF) $(PYTHON) $(srctree)/arch/ia64/scripts/unwcheck.py $<
+	-$(Q)READELF=$(READELF) $(PYTHON2) $(srctree)/arch/ia64/scripts/unwcheck.py $<
 
 archclean:
 	$(Q)$(MAKE) $(clean)=$(boot)
-- 
2.7.4

Re: [PATCH 06/14] kbuild: define PYTHON2 and PYTHON3 variables instead of PYTHON

[Greg Kroah-Hartman]

On Tue, Feb 06, 2018 at 09:34:46AM +0900, Masahiro Yamada wrote:
> The variable 'PYTHON' allows users to specify a proper executable
> name in case the default 'python' does not work.  However, this does
> not address the case where both Python 2 and Python 3 scripts are
> used in one system.
> 
> PEP 394 (https://www.python.org/dev/peps/pep-0394/) provides a
> convention for Python scripts portability.  Here is a quotation:
> 
>   In order to tolerate differences across platforms, all new code
>   that needs to invoke the Python interpreter should not specify
>   'python', but rather should specify either 'python2' or 'python3'.
>   This distinction should be made in shebangs, when invoking from a
>   shell script, when invoking via the system() call, or when invoking
>   in any other context.
> 
> arch/ia64/scripts/unwcheck.py is apparently written in Python 2, so
> it should be invoked by 'python2'.
> 
> It is legitimate to use 'python' for scripts compatible with both
> Python 2 and Python 3, but this is rare (at least I do not see the
> case in kernel tree).  You do not need to make efforts to write your
> scripts in that way.  Anyway, Python 2 will retire in 2020.
> 
> This commit is needed for my new scripts written in Python 3.
> 
> Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
> ---
> 
>  Makefile           | 5 +++--
>  arch/ia64/Makefile | 2 +-
>  2 files changed, 4 insertions(+), 3 deletions(-)
> 
> diff --git a/Makefile b/Makefile
> index 11aff0f..c4e935c 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -384,7 +384,8 @@ GENKSYMS	= scripts/genksyms/genksyms
>  INSTALLKERNEL  := installkernel
>  DEPMOD		= /sbin/depmod
>  PERL		= perl
> -PYTHON		= python
> +PYTHON2		= python2
> +PYTHON3		= python3

Is this going to break any systems that were previous setting PYTHON?

I like this change, and feel it is the correct thing to do, but having a
"fallback" might be needed here.

Could you do what the perf makefile does and do something like:
	override PYTHON := $(call get-executable-or-default,PYTHON,$(PYTHON2))
or is it really not an issue as only ia64 seems to care about this?

thanks,

greg k-h

Re: [PATCH 00/14] Add Kconfig unit tests

[Greg Kroah-Hartman]

On Tue, Feb 06, 2018 at 09:34:40AM +0900, Masahiro Yamada wrote:
> I am applying various patches to Kconfig these days.
> 
> However, I fear regressions.  I have been thinking of unit-tests.
> 
> There are various cryptic parts in Kconfig and corner cases where
> it is difficult to notice breakage.  If unit-tests cover those,
> I will be able to apply changes more confidently.
> 
> So, here is the trial.
> 
> After fixing some problems, I will add a basic test framework.
> This is based on pytest.  Also, this is written in Python 3.
> Python 2 will return in 2020.  So, I believe new python tools should be
> written in Python 3.
> 
> This is my Python 3 and pytest versions.
> 
> $ python3 --version
> Python 3.5.2
> $ python3 -m pytest --version
> This is pytest version 3.4.0, imported from /home/masahiro/.local/lib/python3.5/site-packages/pytest.py
> 
> If I use old pytest version, some parts did not work as expected.
> If this does not work for you, please consider using newer pytest.
> 
> I will brush up the code more and add more test cases to do a better job.
> Before proceeding more, I'd like to get consensus for this approach.
> If you have an idea for better implementation, comments are appreciated.

Personally I think this is great stuff.  I too have never wanted to
touch Kconfig stuff due to the complexity, and having unit tests like
this is a great idea to help ensure that things do not break.

Your first 5 patches should be queued up for the next merge window, no
problem (see my comments on the 6th).  As for the rest, I don't have any
objection to them, and using python3 over python2 is a good idea.  And
anyone who wants to do Kconfig work can easily install the needed
packages, it's not required by any "normal" kernel developer.

Anyway, nice job, it's great to see this happening, no objection from me
at all!

greg k-h

Re: [PATCH 06/14] kbuild: define PYTHON2 and PYTHON3 variables instead of PYTHON

[Masahiro Yamada]

2018-02-06 18:34 GMT+09:00 Greg Kroah-Hartman <gregkh@linuxfoundation.org>:
> On Tue, Feb 06, 2018 at 09:34:46AM +0900, Masahiro Yamada wrote:
>> The variable 'PYTHON' allows users to specify a proper executable
>> name in case the default 'python' does not work.  However, this does
>> not address the case where both Python 2 and Python 3 scripts are
>> used in one system.
>>
>> PEP 394 (https://www.python.org/dev/peps/pep-0394/) provides a
>> convention for Python scripts portability.  Here is a quotation:
>>
>>   In order to tolerate differences across platforms, all new code
>>   that needs to invoke the Python interpreter should not specify
>>   'python', but rather should specify either 'python2' or 'python3'.
>>   This distinction should be made in shebangs, when invoking from a
>>   shell script, when invoking via the system() call, or when invoking
>>   in any other context.
>>
>> arch/ia64/scripts/unwcheck.py is apparently written in Python 2, so
>> it should be invoked by 'python2'.
>>
>> It is legitimate to use 'python' for scripts compatible with both
>> Python 2 and Python 3, but this is rare (at least I do not see the
>> case in kernel tree).  You do not need to make efforts to write your
>> scripts in that way.  Anyway, Python 2 will retire in 2020.
>>
>> This commit is needed for my new scripts written in Python 3.
>>
>> Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
>> ---
>>
>>  Makefile           | 5 +++--
>>  arch/ia64/Makefile | 2 +-
>>  2 files changed, 4 insertions(+), 3 deletions(-)
>>
>> diff --git a/Makefile b/Makefile
>> index 11aff0f..c4e935c 100644
>> --- a/Makefile
>> +++ b/Makefile
>> @@ -384,7 +384,8 @@ GENKSYMS  = scripts/genksyms/genksyms
>>  INSTALLKERNEL  := installkernel
>>  DEPMOD               = /sbin/depmod
>>  PERL         = perl
>> -PYTHON               = python
>> +PYTHON2              = python2
>> +PYTHON3              = python3
>
> Is this going to break any systems that were previous setting PYTHON?
>
> I like this change, and feel it is the correct thing to do, but having a
> "fallback" might be needed here.
>
> Could you do what the perf makefile does and do something like:
>         override PYTHON := $(call get-executable-or-default,PYTHON,$(PYTHON2))
> or is it really not an issue as only ia64 seems to care about this?
>

As far as I see, ia64 is the only instance that has used this ever.

(the perf Makefile defines PYTHON by itself, so should not be a problem.)


If people expect the backward-compatibility for this, I can do like follows:

# backward compatibility for 'PYTHON'
PYTHON2  := $(if $(PYTHON), $(PYTHON), python2)



Another (unlikely) possible breakage is
'python2' may not be installed on users' system.

I believe this is rare, but if needed, I could do like follows
at the cost of ugliness.


PYTHON2 := $(if $(PYTHON), $(PYTHON), \
        $(shell python2 --version 2>/dev/null && echo python2 || echo python))



-- 
Best Regards
Masahiro Yamada

Re: [PATCH 06/14] kbuild: define PYTHON2 and PYTHON3 variables instead of PYTHON

[Greg Kroah-Hartman]

On Tue, Feb 06, 2018 at 07:44:22PM +0900, Masahiro Yamada wrote:
> 2018-02-06 18:34 GMT+09:00 Greg Kroah-Hartman <gregkh@linuxfoundation.org>:
> > On Tue, Feb 06, 2018 at 09:34:46AM +0900, Masahiro Yamada wrote:
> >> The variable 'PYTHON' allows users to specify a proper executable
> >> name in case the default 'python' does not work.  However, this does
> >> not address the case where both Python 2 and Python 3 scripts are
> >> used in one system.
> >>
> >> PEP 394 (https://www.python.org/dev/peps/pep-0394/) provides a
> >> convention for Python scripts portability.  Here is a quotation:
> >>
> >>   In order to tolerate differences across platforms, all new code
> >>   that needs to invoke the Python interpreter should not specify
> >>   'python', but rather should specify either 'python2' or 'python3'.
> >>   This distinction should be made in shebangs, when invoking from a
> >>   shell script, when invoking via the system() call, or when invoking
> >>   in any other context.
> >>
> >> arch/ia64/scripts/unwcheck.py is apparently written in Python 2, so
> >> it should be invoked by 'python2'.
> >>
> >> It is legitimate to use 'python' for scripts compatible with both
> >> Python 2 and Python 3, but this is rare (at least I do not see the
> >> case in kernel tree).  You do not need to make efforts to write your
> >> scripts in that way.  Anyway, Python 2 will retire in 2020.
> >>
> >> This commit is needed for my new scripts written in Python 3.
> >>
> >> Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
> >> ---
> >>
> >>  Makefile           | 5 +++--
> >>  arch/ia64/Makefile | 2 +-
> >>  2 files changed, 4 insertions(+), 3 deletions(-)
> >>
> >> diff --git a/Makefile b/Makefile
> >> index 11aff0f..c4e935c 100644
> >> --- a/Makefile
> >> +++ b/Makefile
> >> @@ -384,7 +384,8 @@ GENKSYMS  = scripts/genksyms/genksyms
> >>  INSTALLKERNEL  := installkernel
> >>  DEPMOD               = /sbin/depmod
> >>  PERL         = perl
> >> -PYTHON               = python
> >> +PYTHON2              = python2
> >> +PYTHON3              = python3
> >
> > Is this going to break any systems that were previous setting PYTHON?
> >
> > I like this change, and feel it is the correct thing to do, but having a
> > "fallback" might be needed here.
> >
> > Could you do what the perf makefile does and do something like:
> >         override PYTHON := $(call get-executable-or-default,PYTHON,$(PYTHON2))
> > or is it really not an issue as only ia64 seems to care about this?
> >
> 
> As far as I see, ia64 is the only instance that has used this ever.
> 
> (the perf Makefile defines PYTHON by itself, so should not be a problem.)
> 
> 
> If people expect the backward-compatibility for this, I can do like follows:
> 
> # backward compatibility for 'PYTHON'
> PYTHON2  := $(if $(PYTHON), $(PYTHON), python2)

That's true, and who knows if python3 is running on ia64 :)

> Another (unlikely) possible breakage is
> 'python2' may not be installed on users' system.
> 
> I believe this is rare, but if needed, I could do like follows
> at the cost of ugliness.
> 
> 
> PYTHON2 := $(if $(PYTHON), $(PYTHON), \
>         $(shell python2 --version 2>/dev/null && echo python2 || echo python))

Ick, I guess that might work.  Try getting your patch set into the 0-day
system to see how it handles things.  I know it builds for ia64.

But this is a real tiny question overall, and should not derail this
patchset at all :)

thanks,

greg k-h

RE: [PATCH 06/14] kbuild: define PYTHON2 and PYTHON3 variables instead of PYTHON

[Luck, Tony]

> That's true, and who knows if python3 is running on ia64 :)

$ python3
-bash: python3: command not found

So I don't have it installed on my ia64 test/build machine.

-Tony

Re: [PATCH 00/14] Add Kconfig unit tests

[Ulf Magnusson]

On Tue, Feb 6, 2018 at 10:38 AM, Greg Kroah-Hartman
<gregkh@linuxfoundation.org> wrote:
> On Tue, Feb 06, 2018 at 09:34:40AM +0900, Masahiro Yamada wrote:
>> I am applying various patches to Kconfig these days.
>>
>> However, I fear regressions.  I have been thinking of unit-tests.
>>
>> There are various cryptic parts in Kconfig and corner cases where
>> it is difficult to notice breakage.  If unit-tests cover those,
>> I will be able to apply changes more confidently.
>>
>> So, here is the trial.
>>
>> After fixing some problems, I will add a basic test framework.
>> This is based on pytest.  Also, this is written in Python 3.
>> Python 2 will return in 2020.  So, I believe new python tools should be
>> written in Python 3.
>>
>> This is my Python 3 and pytest versions.
>>
>> $ python3 --version
>> Python 3.5.2
>> $ python3 -m pytest --version
>> This is pytest version 3.4.0, imported from /home/masahiro/.local/lib/python3.5/site-packages/pytest.py
>>
>> If I use old pytest version, some parts did not work as expected.
>> If this does not work for you, please consider using newer pytest.
>>
>> I will brush up the code more and add more test cases to do a better job.
>> Before proceeding more, I'd like to get consensus for this approach.
>> If you have an idea for better implementation, comments are appreciated.
>
> Personally I think this is great stuff.  I too have never wanted to
> touch Kconfig stuff due to the complexity, and having unit tests like
> this is a great idea to help ensure that things do not break.
>
> Your first 5 patches should be queued up for the next merge window, no
> problem (see my comments on the 6th).  As for the rest, I don't have any
> objection to them, and using python3 over python2 is a good idea.  And
> anyone who wants to do Kconfig work can easily install the needed
> packages, it's not required by any "normal" kernel developer.
>
> Anyway, nice job, it's great to see this happening, no objection from me
> at all!
>
> greg k-h

Yeah, breaking Kconfig is a sure way to feel the wrath.

The only reason I feel somewhat confident modifying Kconfig is that
the Kconfiglib test suite happens to work as a regression test for the
C implementation as well. It compares the .config files produced by
the two implementations for all defconfig files and for
all{no,yes,def}config, for all ARCHes, meaning any changes to the
output of the C tools get flagged as well (with a diff).

Having some "native" tests is great as well. I'm a big fan of
automatic testing. :)

In case you want to run the Kconfiglib test suite at any point, here's
how to do it (in the kernel root):

        $ git clone git://github.com/ulfalizer/Kconfiglib.git
        $ git am Kconfiglib/makefile.patch
        $ python Kconfiglib/testsuite.py speedy

Cheers,
Ulf

Re: [PATCH 00/14] Add Kconfig unit tests

[Sam Ravnborg]

Hi Masahiro.

On Tue, Feb 06, 2018 at 09:34:40AM +0900, Masahiro Yamada wrote:
> I am applying various patches to Kconfig these days.
> 
> However, I fear regressions.  I have been thinking of unit-tests.
> 
> There are various cryptic parts in Kconfig and corner cases where
> it is difficult to notice breakage.  If unit-tests cover those,
> I will be able to apply changes more confidently.
> 
> So, here is the trial.
...

Great to see this done.
And +1 for basing this on a standard framework rather than inventing your own.

> If you have an idea for better implementation, comments are appreciated.
Nope, please continue with pytest.

	Sam