public inbox for linux-kbuild@vger.kernel.org
 help / color / mirror / Atom feed
* [PATCH 00/14] Add Kconfig unit tests
@ 2018-02-06  0:34 Masahiro Yamada
  2018-02-06  0:34 ` [PATCH 01/14] kconfig: send error messages to stderr Masahiro Yamada
                   ` (15 more replies)
  0 siblings, 16 replies; 41+ messages in thread
From: Masahiro Yamada @ 2018-02-06  0:34 UTC (permalink / raw)
  To: linux-kbuild
  Cc: Greg Kroah-Hartman, Andrew Morton, Nicolas Pitre,
	Luis R . Rodriguez, Randy Dunlap, Ulf Magnusson, Sam Ravnborg,
	Michal Marek, Linus Torvalds, Masahiro Yamada, Luis R. Rodriguez,
	Tony Luck, Borislav Petkov, linux-kernel, Thomas Gleixner,
	Heinrich Schuchardt, Yaakov Selkowitz, linux-ia64, Fenghua Yu,
	Marc Herbert

I am applying various patches to Kconfig these days.

However, I fear regressions.  I have been thinking of unit-tests.

There are various cryptic parts in Kconfig and corner cases where
it is difficult to notice breakage.  If unit-tests cover those,
I will be able to apply changes more confidently.

So, here is the trial.

After fixing some problems, I will add a basic test framework.
This is based on pytest.  Also, this is written in Python 3.
Python 2 will return in 2020.  So, I believe new python tools should be
written in Python 3.

This is my Python 3 and pytest versions.

$ python3 --version
Python 3.5.2
$ python3 -m pytest --version
This is pytest version 3.4.0, imported from /home/masahiro/.local/lib/python3.5/site-packages/pytest.py

If I use old pytest version, some parts did not work as expected.
If this does not work for you, please consider using newer pytest.

I will brush up the code more and add more test cases to do a better job.
Before proceeding more, I'd like to get consensus for this approach.
If you have an idea for better implementation, comments are appreciated.

How to use?
-----------

Please make sure Python3 and pytest for Python3 is installed on your system.

Then, run "make testconfig"

The result looks like as follows:

masahiro@grover:~/workspace/linux-yamada$ make testconfig
  HOSTCC  scripts/kconfig/conf.o
  HOSTCC  scripts/kconfig/zconf.tab.o
  HOSTLD  scripts/kconfig/conf
python3 -B -m pytest ./scripts/kconfig/tests \
-o cache_dir=/home/masahiro/workspace/linux-yamada/scripts/kconfig/tests/.cache \

==================================================== test session starts =====================================================
platform linux -- Python 3.5.2, pytest-3.3.2, py-1.5.2, pluggy-0.6.0 -- /usr/bin/python3
cachedir: scripts/kconfig/tests/.cache
rootdir: /home/masahiro/workspace/linux-yamada/scripts/kconfig/tests, inifile: pytest.ini
collected 12 items

scripts/kconfig/tests/auto_submenu_creation/__init__.py::test PASSED                                                   [  8%]
scripts/kconfig/tests/choice/__init__.py::test_oldask0 PASSED                                                          [ 16%]
scripts/kconfig/tests/choice/__init__.py::test_oldask1 PASSED                                                          [ 25%]
scripts/kconfig/tests/choice/__init__.py::test_allyes PASSED                                                           [ 33%]
scripts/kconfig/tests/choice/__init__.py::test_allmod PASSED                                                           [ 41%]
scripts/kconfig/tests/choice/__init__.py::test_allno PASSED                                                            [ 50%]
scripts/kconfig/tests/choice/__init__.py::test_alldef PASSED                                                           [ 58%]
scripts/kconfig/tests/choice_value_with_m_dep/__init__.py::test PASSED                                                 [ 66%]
scripts/kconfig/tests/err_recursive_inc/__init__.py::test PASSED                                                       [ 75%]
scripts/kconfig/tests/new_choice_with_dep/__init__.py::test PASSED                                                     [ 83%]
scripts/kconfig/tests/no_write_if_dep_unmet/__init__.py::test PASSED                                                   [ 91%]
scripts/kconfig/tests/warn_recursive_dep/__init__.py::test PASSED                                                      [100%]

================================================= 12 passed in 0.08 seconds ==================================================



Masahiro Yamada (14):
  kconfig: send error messages to stderr
  kconfig: do not write choice values when their dependency becomes n
  kconfig: show '?' prompt even if no help text is available
  kconfig: print additional new line for choice for redirection
  kconfig: remove 'config*' pattern from .gitignnore
  kbuild: define PYTHON2 and PYTHON3 variables instead of PYTHON
  kconfig: test: add framework for Kconfig unit-tests
  kconfig: test: add basic 'choice' tests
  kconfig: test: test automatic submenu creation
  kconfig: test: check if new symbols in choice are asked
  kconfig: test: check .config sanity for choice values with unmet dep
  kconfig: test: check visibility of tristate choice values in y choice
  kconfig: test: check if recursive dependencies are detected
  kconfig: test: check if recursive inclusion is detected

 Makefile                                           |   5 +-
 arch/ia64/Makefile                                 |   2 +-
 scripts/kconfig/.gitignore                         |   1 -
 scripts/kconfig/Makefile                           |   8 +
 scripts/kconfig/conf.c                             |  29 +--
 scripts/kconfig/symbol.c                           |   4 +-
 .../kconfig/tests/auto_submenu_creation/Kconfig    |  50 ++++
 .../tests/auto_submenu_creation/__init__.py        |  12 +
 .../tests/auto_submenu_creation/expected_stdout    |  10 +
 scripts/kconfig/tests/choice/Kconfig               |  54 +++++
 scripts/kconfig/tests/choice/__init__.py           |  37 +++
 .../kconfig/tests/choice/alldef_expected_config    |   5 +
 .../kconfig/tests/choice/allmod_expected_config    |   9 +
 scripts/kconfig/tests/choice/allno_expected_config |   5 +
 .../kconfig/tests/choice/allyes_expected_config    |   9 +
 .../kconfig/tests/choice/oldask0_expected_stdout   |  10 +
 scripts/kconfig/tests/choice/oldask1_config        |   2 +
 .../kconfig/tests/choice/oldask1_expected_stdout   |  15 ++
 .../kconfig/tests/choice_value_with_m_dep/Kconfig  |  20 ++
 .../tests/choice_value_with_m_dep/__init__.py      |  15 ++
 .../kconfig/tests/choice_value_with_m_dep/config   |   2 +
 .../tests/choice_value_with_m_dep/expected_config  |   3 +
 .../tests/choice_value_with_m_dep/expected_stdout  |   4 +
 scripts/kconfig/tests/conftest.py                  | 255 +++++++++++++++++++++
 scripts/kconfig/tests/err_recursive_inc/Kconfig    |   1 +
 .../kconfig/tests/err_recursive_inc/Kconfig.inc    |   1 +
 .../kconfig/tests/err_recursive_inc/__init__.py    |  10 +
 .../tests/err_recursive_inc/expected_stderr        |   4 +
 scripts/kconfig/tests/new_choice_with_dep/Kconfig  |  37 +++
 .../kconfig/tests/new_choice_with_dep/__init__.py  |  14 ++
 scripts/kconfig/tests/new_choice_with_dep/config   |   3 +
 .../tests/new_choice_with_dep/expected_stdout      |  10 +
 .../kconfig/tests/no_write_if_dep_unmet/Kconfig    |  14 ++
 .../tests/no_write_if_dep_unmet/__init__.py        |  17 ++
 scripts/kconfig/tests/no_write_if_dep_unmet/config |   1 +
 .../tests/no_write_if_dep_unmet/expected_config    |   5 +
 scripts/kconfig/tests/pytest.ini                   |   6 +
 scripts/kconfig/tests/warn_recursive_dep/Kconfig   |  62 +++++
 .../kconfig/tests/warn_recursive_dep/__init__.py   |  10 +
 .../tests/warn_recursive_dep/expected_stderr       |  33 +++
 scripts/kconfig/zconf.l                            |  27 ++-
 41 files changed, 789 insertions(+), 32 deletions(-)
 create mode 100644 scripts/kconfig/tests/auto_submenu_creation/Kconfig
 create mode 100644 scripts/kconfig/tests/auto_submenu_creation/__init__.py
 create mode 100644 scripts/kconfig/tests/auto_submenu_creation/expected_stdout
 create mode 100644 scripts/kconfig/tests/choice/Kconfig
 create mode 100644 scripts/kconfig/tests/choice/__init__.py
 create mode 100644 scripts/kconfig/tests/choice/alldef_expected_config
 create mode 100644 scripts/kconfig/tests/choice/allmod_expected_config
 create mode 100644 scripts/kconfig/tests/choice/allno_expected_config
 create mode 100644 scripts/kconfig/tests/choice/allyes_expected_config
 create mode 100644 scripts/kconfig/tests/choice/oldask0_expected_stdout
 create mode 100644 scripts/kconfig/tests/choice/oldask1_config
 create mode 100644 scripts/kconfig/tests/choice/oldask1_expected_stdout
 create mode 100644 scripts/kconfig/tests/choice_value_with_m_dep/Kconfig
 create mode 100644 scripts/kconfig/tests/choice_value_with_m_dep/__init__.py
 create mode 100644 scripts/kconfig/tests/choice_value_with_m_dep/config
 create mode 100644 scripts/kconfig/tests/choice_value_with_m_dep/expected_config
 create mode 100644 scripts/kconfig/tests/choice_value_with_m_dep/expected_stdout
 create mode 100644 scripts/kconfig/tests/conftest.py
 create mode 100644 scripts/kconfig/tests/err_recursive_inc/Kconfig
 create mode 100644 scripts/kconfig/tests/err_recursive_inc/Kconfig.inc
 create mode 100644 scripts/kconfig/tests/err_recursive_inc/__init__.py
 create mode 100644 scripts/kconfig/tests/err_recursive_inc/expected_stderr
 create mode 100644 scripts/kconfig/tests/new_choice_with_dep/Kconfig
 create mode 100644 scripts/kconfig/tests/new_choice_with_dep/__init__.py
 create mode 100644 scripts/kconfig/tests/new_choice_with_dep/config
 create mode 100644 scripts/kconfig/tests/new_choice_with_dep/expected_stdout
 create mode 100644 scripts/kconfig/tests/no_write_if_dep_unmet/Kconfig
 create mode 100644 scripts/kconfig/tests/no_write_if_dep_unmet/__init__.py
 create mode 100644 scripts/kconfig/tests/no_write_if_dep_unmet/config
 create mode 100644 scripts/kconfig/tests/no_write_if_dep_unmet/expected_config
 create mode 100644 scripts/kconfig/tests/pytest.ini
 create mode 100644 scripts/kconfig/tests/warn_recursive_dep/Kconfig
 create mode 100644 scripts/kconfig/tests/warn_recursive_dep/__init__.py
 create mode 100644 scripts/kconfig/tests/warn_recursive_dep/expected_stderr

-- 
2.7.4


^ permalink raw reply	[flat|nested] 41+ messages in thread

end of thread, other threads:[~2018-02-18 19:38 UTC | newest]

Thread overview: 41+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2018-02-06  0:34 [PATCH 00/14] Add Kconfig unit tests Masahiro Yamada
2018-02-06  0:34 ` [PATCH 01/14] kconfig: send error messages to stderr Masahiro Yamada
2018-02-07 20:24   ` Ulf Magnusson
2018-02-08  1:49     ` Masahiro Yamada
2018-02-08  2:02       ` Ulf Magnusson
2018-02-06  0:34 ` [PATCH 02/14] kconfig: do not write choice values when their dependency becomes n Masahiro Yamada
2018-02-07 22:55   ` Ulf Magnusson
2018-02-08  2:42     ` Masahiro Yamada
2018-02-08  2:46       ` Ulf Magnusson
2018-02-08 21:21         ` Ulf Magnusson
2018-02-06  0:34 ` [PATCH 03/14] kconfig: show '?' prompt even if no help text is available Masahiro Yamada
2018-02-07 20:28   ` Ulf Magnusson
2018-02-06  0:34 ` [PATCH 04/14] kconfig: print additional new line for choice for redirection Masahiro Yamada
2018-02-07 23:34   ` Ulf Magnusson
2018-02-08  6:00     ` Masahiro Yamada
2018-02-06  0:34 ` [PATCH 05/14] kconfig: remove 'config*' pattern from .gitignnore Masahiro Yamada
2018-02-07 23:43   ` Ulf Magnusson
2018-02-06  0:34 ` [PATCH 06/14] kbuild: define PYTHON2 and PYTHON3 variables instead of PYTHON Masahiro Yamada
2018-02-06  9:34   ` Greg Kroah-Hartman
2018-02-06 10:44     ` Masahiro Yamada
2018-02-06 13:10       ` Greg Kroah-Hartman
2018-02-06 17:07         ` Luck, Tony
2018-02-06  0:34 ` [PATCH 07/14] kconfig: test: add framework for Kconfig unit-tests Masahiro Yamada
2018-02-08  0:35   ` Ulf Magnusson
2018-02-06  0:34 ` [PATCH 08/14] kconfig: test: add basic 'choice' tests Masahiro Yamada
2018-02-07 23:57   ` Ulf Magnusson
2018-02-06  0:34 ` [PATCH 09/14] kconfig: test: test automatic submenu creation Masahiro Yamada
2018-02-07 23:58   ` Ulf Magnusson
2018-02-06  0:34 ` [PATCH 10/14] kconfig: test: check if new symbols in choice are asked Masahiro Yamada
2018-02-08  0:09   ` Ulf Magnusson
2018-02-06  0:34 ` [PATCH 11/14] kconfig: test: check .config sanity for choice values with unmet dep Masahiro Yamada
2018-02-08  0:11   ` Ulf Magnusson
2018-02-06  0:34 ` [PATCH 12/14] kconfig: test: check visibility of tristate choice values in y choice Masahiro Yamada
2018-02-08  0:13   ` Ulf Magnusson
2018-02-06  0:34 ` [PATCH 13/14] kconfig: test: check if recursive dependencies are detected Masahiro Yamada
2018-02-08  0:15   ` Ulf Magnusson
2018-02-06  0:34 ` [PATCH 14/14] kconfig: test: check if recursive inclusion is detected Masahiro Yamada
2018-02-08  0:16   ` Ulf Magnusson
2018-02-06  9:38 ` [PATCH 00/14] Add Kconfig unit tests Greg Kroah-Hartman
2018-02-07 23:19   ` Ulf Magnusson
2018-02-18 19:38 ` Sam Ravnborg

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox