[PATCH 0/3] kbuild,kconfig: generate lexer/parser C files instead of copying _shipped files

[PATCH 0/3] kbuild,kconfig: generate lexer/parser C files instead of copying _shipped files

[Masahiro Yamada]

In Linux build system convention, pre-generated files are version-
controlled with a "_shipped" suffix.  During the kernel building,
they are simply shipped (copied) removing the suffix.

From users' point of view, this approach can reduce external tool
dependency for the kernel build,

From developers point of view, it is tedious to manually regenerate
such artifacts.  In fact, we see several patches to regenerate
_shipped files.  They are noise commits.

When we update a *.y or *.l file, it would be better to update the
corresponding _shipped file in the same commit, but it is painful.
If you use a different version of flex/bison, it will produce lots of
irrelevant diffs.

We could update _shipped files after adding various changes to the
real sources, but it is not very nice for a git-bisect'ability.
In case of a problem, "git bisect" would point to the commit updating
_shipped files, but the root cause would be in another commit that
has changed the corresponding .l or .y files.

Some months ago, I sent RFC patches to run flex, bison, and gperf
during the build.
https://lkml.org/lkml/2017/8/19/49

Basically Linus agreed this, but he found a problem in gperf, then
use of gperf in kernel was removed.

It took some months for me to come back.  This time, I installed various
versions of flex/bison on my machine, and tested them more carefully.

My current motivation is in Kconfig.
There are several Kconfig patches touching *.y and *.l
(and Linus suggested another improvement for Kconfig)
so I want to remove zconf.lex.c_shipped and zconf.tab.c_shipped now.
Kconfig has no problem for this switch.

dtc and genksyms will be taken care of later because
both of them are having shift/reduce conflicts now.
The ambiguous grammar in dtc has been fixed in upstream, but not
reflected to kernel yet.  We can proceed migration in sub-system base.


Masahiro Yamada (3):
  kbuild: add LEX and YACC variables
  kbuild: prepare to remove C files pre-generated by flex and bison
  kconfig: generate lexer and parser during build instead of shipping

 Documentation/process/changes.rst   |   25 +
 Makefile                            |    4 +-
 scripts/Makefile.lib                |   24 +-
 scripts/kconfig/Makefile            |    1 +
 scripts/kconfig/zconf.lex.c_shipped | 2473 -----------------------------------
 scripts/kconfig/zconf.tab.c_shipped | 2471 ----------------------------------
 6 files changed, 48 insertions(+), 4950 deletions(-)
 delete mode 100644 scripts/kconfig/zconf.lex.c_shipped
 delete mode 100644 scripts/kconfig/zconf.tab.c_shipped

-- 
2.7.4

[PATCH 1/3] kbuild: add LEX and YACC variables

[Masahiro Yamada]

Allow users to use their favorite lexer / parser generators.
This is useful for me to test various flex and bison versions.

Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
---

 Makefile             | 4 +++-
 scripts/Makefile.lib | 4 ++--
 2 files changed, 5 insertions(+), 3 deletions(-)

diff --git a/Makefile b/Makefile
index c988e46..83209d6 100644
--- a/Makefile
+++ b/Makefile
@@ -377,6 +377,8 @@ NM		= $(CROSS_COMPILE)nm
 STRIP		= $(CROSS_COMPILE)strip
 OBJCOPY		= $(CROSS_COMPILE)objcopy
 OBJDUMP		= $(CROSS_COMPILE)objdump
+LEX		= flex
+YACC		= bison
 AWK		= awk
 GENKSYMS	= scripts/genksyms/genksyms
 INSTALLKERNEL  := installkernel
@@ -428,7 +430,7 @@ GCC_PLUGINS_CFLAGS :=
 
 export ARCH SRCARCH CONFIG_SHELL HOSTCC HOSTCFLAGS CROSS_COMPILE AS LD CC
 export CPP AR NM STRIP OBJCOPY OBJDUMP HOSTLDFLAGS HOST_LOADLIBES
-export MAKE AWK GENKSYMS INSTALLKERNEL PERL PYTHON UTS_MACHINE
+export MAKE LEX YACC AWK GENKSYMS INSTALLKERNEL PERL PYTHON UTS_MACHINE
 export HOSTCXX HOSTCXXFLAGS LDFLAGS_MODULE CHECK CHECKFLAGS
 
 export KBUILD_CPPFLAGS NOSTDINC_FLAGS LINUXINCLUDE OBJCOPYFLAGS LDFLAGS
diff --git a/scripts/Makefile.lib b/scripts/Makefile.lib
index 1ca4dcd..ee528e3 100644
--- a/scripts/Makefile.lib
+++ b/scripts/Makefile.lib
@@ -193,7 +193,7 @@ ifdef REGENERATE_PARSERS
 LEX_PREFIX = $(if $(LEX_PREFIX_${baseprereq}),$(LEX_PREFIX_${baseprereq}),yy)
 
 quiet_cmd_flex = LEX     $@
-      cmd_flex = flex -o$@ -L -P $(LEX_PREFIX) $<
+      cmd_flex = $(LEX) -o$@ -L -P $(LEX_PREFIX) $<
 
 .PRECIOUS: $(src)/%.lex.c_shipped
 $(src)/%.lex.c_shipped: $(src)/%.l
@@ -204,7 +204,7 @@ $(src)/%.lex.c_shipped: $(src)/%.l
 YACC_PREFIX = $(if $(YACC_PREFIX_${baseprereq}),$(YACC_PREFIX_${baseprereq}),yy)
 
 quiet_cmd_bison = YACC    $@
-      cmd_bison = bison -o$@ -t -l -p $(YACC_PREFIX) $<
+      cmd_bison = $(YACC) -o$@ -t -l -p $(YACC_PREFIX) $<
 
 .PRECIOUS: $(src)/%.tab.c_shipped
 $(src)/%.tab.c_shipped: $(src)/%.y
-- 
2.7.4

[PATCH 2/3] kbuild: prepare to remove C files pre-generated by flex and bison

[Masahiro Yamada]

In Linux build system convention, pre-generated files are version-
controlled with a "_shipped" suffix.  During the kernel building,
they are simply shipped (copied) removing the suffix.

This approach can reduce external tool dependency for the kernel build,
but it is tedious to manually regenerate such artifacts from developers'
point of view.  (We need to do "make REGENERATE_PARSERS=1" every time
we touch real source files such as *.l, *.y)

Some months ago, I sent out RFC patches to run flex, bison, and gperf
during the build.

In the review and test, Linus noticed gperf-3.1 had changed the lookup
function prototype.  Then, the use of gperf in kernel was entirely
removed by commit bb3290d91695 ("Remove gperf usage from toolchain").

I tested several versions of flex and bison, and I was not hit by any
compatibility issue except a flaw in flex-2.6.3; if you generate lexer
for dtc and genksyms with flex-2.6.3, you will see "yywrap redefined"
warning.  This was not intentional, but a bug fixed by flex-2.6.4.
Otherwise, flex and bison look fairly stable for a long time.

This commit prepares some build rules to start removing the _shipped
files.  Also, document minimal requirement for flex and bison.

Rationale for the minimal version:
The -Wmissing-prototypes option of GCC reports "no previous prototype"
warnings for lexers generated by flex-2.5.34, so I chose 2.5.35 as the
required version for flex.  Flex-2.5.35 was released in 2008.  Bison
looked more stable.  I did not see any problem with bison-2.0, released
in 2004.  I did not test bison-1.x, but bison-2.0 should be old enough.

Tested flex versions:
  2.5.35
  2.5.36
  2.5.37
  2.5.39
  2.6.0
  2.6.1
  2.6.2
  2.6.3   (*)
  2.6.4

 (*) flex-2.6.3 causes "yywrap redefined" warning

Tested bison versions:
  2.0
  2.1
  2.2
  2.3
  2.4
  2.4.1
  2.5.1
  2.6
  2.6.1
  2.6.2
  2.6.3
  2.6.4
  2.6.5
  2.7
  2.7.1
  3.0
  3.0.1
  3.0.2
  3.0.3
  3.0.4

Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
---

 Documentation/process/changes.rst | 25 +++++++++++++++++++++++++
 scripts/Makefile.lib              | 20 +++++++++++++++++---
 2 files changed, 42 insertions(+), 3 deletions(-)

diff --git a/Documentation/process/changes.rst b/Documentation/process/changes.rst
index 560beae..fc9c7c3 100644
--- a/Documentation/process/changes.rst
+++ b/Documentation/process/changes.rst
@@ -32,6 +32,8 @@ you probably needn't concern yourself with isdn4k-utils.
 GNU C                  3.2              gcc --version
 GNU make               3.81             make --version
 binutils               2.20             ld -v
+flex                   2.5.35           flex --version
+bison                  2.0              bison --version
 util-linux             2.10o            fdformat --version
 module-init-tools      0.9.10           depmod -V
 e2fsprogs              1.41.4           e2fsck -V
@@ -79,6 +81,19 @@ The build system has, as of 4.13, switched to using thin archives (`ar T`)
 rather than incremental linking (`ld -r`) for built-in.o intermediate steps.
 This requires binutils 2.20 or newer.
 
+Flex
+----
+
+Since Linux 4.16, the build system generates lexical analisers
+during build.  This requires flex 2.5.35 or later.
+
+
+Bison
+-----
+
+Since Linux 4.16, the build system generates parsers
+during build.  This requires bison 2.0 or later.
+
 Perl
 ----
 
@@ -333,6 +348,16 @@ Binutils
 
 - <https://www.kernel.org/pub/linux/devel/binutils/>
 
+Flex
+----
+
+- <https://github.com/westes/flex/releases>
+
+Bison
+-----
+
+- <ftp://ftp.gnu.org/gnu/bison/>
+
 OpenSSL
 -------
 
diff --git a/scripts/Makefile.lib b/scripts/Makefile.lib
index ee528e3..0f9ef3f 100644
--- a/scripts/Makefile.lib
+++ b/scripts/Makefile.lib
@@ -186,8 +186,6 @@ $(foreach m, $(notdir $1), \
 	$(addprefix $(obj)/, $(foreach s, $3, $($(m:%$(strip $2)=%$(s)))))))
 endef
 
-ifdef REGENERATE_PARSERS
-
 # LEX
 # ---------------------------------------------------------------------------
 LEX_PREFIX = $(if $(LEX_PREFIX_${baseprereq}),$(LEX_PREFIX_${baseprereq}),yy)
@@ -195,9 +193,15 @@ LEX_PREFIX = $(if $(LEX_PREFIX_${baseprereq}),$(LEX_PREFIX_${baseprereq}),yy)
 quiet_cmd_flex = LEX     $@
       cmd_flex = $(LEX) -o$@ -L -P $(LEX_PREFIX) $<
 
+ifdef REGENERATE_PARSERS
 .PRECIOUS: $(src)/%.lex.c_shipped
 $(src)/%.lex.c_shipped: $(src)/%.l
 	$(call cmd,flex)
+endif
+
+.PRECIOUS: $(obj)/%.lex.c
+$(filter %.lex.c,$(targets)): $(obj)/%.lex.c: $(src)/%.l FORCE
+	$(call if_changed,flex)
 
 # YACC
 # ---------------------------------------------------------------------------
@@ -206,19 +210,29 @@ YACC_PREFIX = $(if $(YACC_PREFIX_${baseprereq}),$(YACC_PREFIX_${baseprereq}),yy)
 quiet_cmd_bison = YACC    $@
       cmd_bison = $(YACC) -o$@ -t -l -p $(YACC_PREFIX) $<
 
+ifdef REGENERATE_PARSERS
 .PRECIOUS: $(src)/%.tab.c_shipped
 $(src)/%.tab.c_shipped: $(src)/%.y
 	$(call cmd,bison)
+endif
+
+.PRECIOUS: $(obj)/%.tab.c
+$(filter %.tab.c,$(targets)): $(obj)/%.tab.c: $(src)/%.y FORCE
+	$(call if_changed,bison)
 
 quiet_cmd_bison_h = YACC    $@
       cmd_bison_h = bison -o/dev/null --defines=$@ -t -l -p $(YACC_PREFIX) $<
 
+ifdef REGENERATE_PARSERS
 .PRECIOUS: $(src)/%.tab.h_shipped
 $(src)/%.tab.h_shipped: $(src)/%.y
 	$(call cmd,bison_h)
-
 endif
 
+.PRECIOUS: $(obj)/%.tab.h
+$(filter %.tab.h,$(targets)): $(obj)/%.tab.h: $(src)/%.y FORCE
+	$(call if_changed,bison_h)
+
 # Shipped files
 # ===========================================================================
 
-- 
2.7.4

Re: [PATCH 2/3] kbuild: prepare to remove C files pre-generated by flex and bison

[Randy Dunlap]

On 12/09/2017 08:02 AM, Masahiro Yamada wrote:
> 
> Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
> ---
> 
>  Documentation/process/changes.rst | 25 +++++++++++++++++++++++++
>  scripts/Makefile.lib              | 20 +++++++++++++++++---
>  2 files changed, 42 insertions(+), 3 deletions(-)
> 
> diff --git a/Documentation/process/changes.rst b/Documentation/process/changes.rst
> index 560beae..fc9c7c3 100644
> --- a/Documentation/process/changes.rst
> +++ b/Documentation/process/changes.rst
> @@ -32,6 +32,8 @@ you probably needn't concern yourself with isdn4k-utils.
>  GNU C                  3.2              gcc --version
>  GNU make               3.81             make --version
>  binutils               2.20             ld -v
> +flex                   2.5.35           flex --version
> +bison                  2.0              bison --version
>  util-linux             2.10o            fdformat --version
>  module-init-tools      0.9.10           depmod -V
>  e2fsprogs              1.41.4           e2fsck -V
> @@ -79,6 +81,19 @@ The build system has, as of 4.13, switched to using thin archives (`ar T`)
>  rather than incremental linking (`ld -r`) for built-in.o intermediate steps.
>  This requires binutils 2.20 or newer.
>  
> +Flex
> +----
> +
> +Since Linux 4.16, the build system generates lexical analisers

                                                        analyzers
or                                                      analysers

> +during build.  This requires flex 2.5.35 or later.


thanks.
-- 
~Randy

Re: [PATCH 0/3] kbuild,kconfig: generate lexer/parser C files instead of copying _shipped files

[Masahiro Yamada]

2017-12-10 1:02 GMT+09:00 Masahiro Yamada <yamada.masahiro@socionext.com>:
> In Linux build system convention, pre-generated files are version-
> controlled with a "_shipped" suffix.  During the kernel building,
> they are simply shipped (copied) removing the suffix.
>
> From users' point of view, this approach can reduce external tool
> dependency for the kernel build,
>
> From developers point of view, it is tedious to manually regenerate
> such artifacts.  In fact, we see several patches to regenerate
> _shipped files.  They are noise commits.
>
> When we update a *.y or *.l file, it would be better to update the
> corresponding _shipped file in the same commit, but it is painful.
> If you use a different version of flex/bison, it will produce lots of
> irrelevant diffs.
>
> We could update _shipped files after adding various changes to the
> real sources, but it is not very nice for a git-bisect'ability.
> In case of a problem, "git bisect" would point to the commit updating
> _shipped files, but the root cause would be in another commit that
> has changed the corresponding .l or .y files.
>
> Some months ago, I sent RFC patches to run flex, bison, and gperf
> during the build.
> https://lkml.org/lkml/2017/8/19/49
>
> Basically Linus agreed this, but he found a problem in gperf, then
> use of gperf in kernel was removed.
>
> It took some months for me to come back.  This time, I installed various
> versions of flex/bison on my machine, and tested them more carefully.
>
> My current motivation is in Kconfig.
> There are several Kconfig patches touching *.y and *.l
> (and Linus suggested another improvement for Kconfig)
> so I want to remove zconf.lex.c_shipped and zconf.tab.c_shipped now.
> Kconfig has no problem for this switch.
>
> dtc and genksyms will be taken care of later because
> both of them are having shift/reduce conflicts now.
> The ambiguous grammar in dtc has been fixed in upstream, but not
> reflected to kernel yet.  We can proceed migration in sub-system base.
>
>
> Masahiro Yamada (3):
>   kbuild: add LEX and YACC variables
>   kbuild: prepare to remove C files pre-generated by flex and bison
>   kconfig: generate lexer and parser during build instead of shipping
>

I did not get any problem report from the 0-day bot.
I assume flex and bison are installed on their machines.


Fixed a typo pointed out by Randy,
then applied to linux-kbuild/kconfig.



-- 
Best Regards
Masahiro Yamada

Re: [PATCH 0/3] kbuild,kconfig: generate lexer/parser C files instead of copying _shipped files

[Sam Ravnborg]

Hi Masahiro.

> > In Linux build system convention, pre-generated files are version-
> > controlled with a "_shipped" suffix.  During the kernel building,
> > they are simply shipped (copied) removing the suffix.
> >
> > From users' point of view, this approach can reduce external tool
> > dependency for the kernel build,
> >
> > From developers point of view, it is tedious to manually regenerate
> > such artifacts.  In fact, we see several patches to regenerate
> > _shipped files.  They are noise commits.
...

Nice cleanup we should have does years ago.
When we introduced this we did this to minimize the time
it took to configure a clean kernel - as one of the reasons.
Since then the average computer has been significantly faster
so the time to run flex/bison is not an issue anymore.

	Sam