[PATCH] Kbuild: Update config_data.gz only if KCONFIG_CONFIG materially changed

[PATCH] Kbuild: Update config_data.gz only if KCONFIG_CONFIG materially changed

[Elliot Berman]

If you update the timestamp of KCONFIG_CONFIG without actually changing
anything, config_data.gz is re-generated and causes vmlinux to re-link.
When Link Time Optimization is enabled, unnecessary re-linking of
vmlinux is highly desirable since it adds several minutes to build time.

Avoid touching config_data.gz by using filechk to compare the existing
config_data.gz and update only if it changed.

The .config can be touched, for instance, by a build script which
installs the default defconfig and then applies a defconfig fragment on
top.

For a simple example on my x86 machine, I modified x86 default defconfig to set
CONFIG_IKCONFIG=y and run:
  make -j50 defconfig tiny.config vmlinux
  make -j50 defconfig tiny.config vmlinux
With this patch, vmlinux is not re-built as a result of config_data.gz
change.

Signed-off-by: Elliot Berman <eberman@codeaurora.org>
---
 kernel/Makefile      | 2 +-
 scripts/Makefile.lib | 2 ++
 2 files changed, 3 insertions(+), 1 deletion(-)

diff --git a/kernel/Makefile b/kernel/Makefile
index 320f1f3..bd4e558 100644
--- a/kernel/Makefile
+++ b/kernel/Makefile
@@ -140,7 +140,7 @@ $(obj)/configs.o: $(obj)/config_data.gz
 
 targets += config_data.gz
 $(obj)/config_data.gz: $(KCONFIG_CONFIG) FORCE
-	$(call if_changed,gzip)
+	$(call filechk,gzip)
 
 $(obj)/kheaders.o: $(obj)/kheaders_data.tar.xz
 
diff --git a/scripts/Makefile.lib b/scripts/Makefile.lib
index a4fbaf8..81d3ec1 100644
--- a/scripts/Makefile.lib
+++ b/scripts/Makefile.lib
@@ -282,6 +282,8 @@ cmd_objcopy = $(OBJCOPY) $(OBJCOPYFLAGS) $(OBJCOPYFLAGS_$(@F)) $< $@
 quiet_cmd_gzip = GZIP    $@
       cmd_gzip = cat $(real-prereqs) | $(KGZIP) -n -f -9 > $@
 
+filechk_gzip = cat $(real-prereqs) | $(KGZIP) -n -f -9
+
 # DTC
 # ---------------------------------------------------------------------------
 DTC ?= $(objtree)/scripts/dtc/dtc
-- 
The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum,
a Linux Foundation Collaborative Project

Re: [PATCH] Kbuild: Update config_data.gz only if KCONFIG_CONFIG materially changed

[Masahiro Yamada]

On Fri, Apr 2, 2021 at 7:45 AM Elliot Berman <eberman@codeaurora.org> wrote:
>
> If you update the timestamp of KCONFIG_CONFIG without actually changing
> anything, config_data.gz is re-generated and causes vmlinux to re-link.
> When Link Time Optimization is enabled, unnecessary re-linking of
> vmlinux is highly desirable since it adds several minutes to build time.
>
> Avoid touching config_data.gz by using filechk to compare the existing
> config_data.gz and update only if it changed.
>
> The .config can be touched, for instance, by a build script which
> installs the default defconfig and then applies a defconfig fragment on
> top.
>
> For a simple example on my x86 machine, I modified x86 default defconfig to set
> CONFIG_IKCONFIG=y and run:
>   make -j50 defconfig tiny.config vmlinux
>   make -j50 defconfig tiny.config vmlinux
> With this patch, vmlinux is not re-built as a result of config_data.gz
> change.
>
> Signed-off-by: Elliot Berman <eberman@codeaurora.org>
> ---
>  kernel/Makefile      | 2 +-
>  scripts/Makefile.lib | 2 ++
>  2 files changed, 3 insertions(+), 1 deletion(-)
>
> diff --git a/kernel/Makefile b/kernel/Makefile
> index 320f1f3..bd4e558 100644
> --- a/kernel/Makefile
> +++ b/kernel/Makefile
> @@ -140,7 +140,7 @@ $(obj)/configs.o: $(obj)/config_data.gz
>
>  targets += config_data.gz
>  $(obj)/config_data.gz: $(KCONFIG_CONFIG) FORCE
> -       $(call if_changed,gzip)
> +       $(call filechk,gzip)


I do not think this is the right approach
because gzip is executed every time, even
if the time stamp is not changed.







>
>  $(obj)/kheaders.o: $(obj)/kheaders_data.tar.xz
>
> diff --git a/scripts/Makefile.lib b/scripts/Makefile.lib
> index a4fbaf8..81d3ec1 100644
> --- a/scripts/Makefile.lib
> +++ b/scripts/Makefile.lib
> @@ -282,6 +282,8 @@ cmd_objcopy = $(OBJCOPY) $(OBJCOPYFLAGS) $(OBJCOPYFLAGS_$(@F)) $< $@
>  quiet_cmd_gzip = GZIP    $@
>        cmd_gzip = cat $(real-prereqs) | $(KGZIP) -n -f -9 > $@
>
> +filechk_gzip = cat $(real-prereqs) | $(KGZIP) -n -f -9
> +
>  # DTC
>  # ---------------------------------------------------------------------------
>  DTC ?= $(objtree)/scripts/dtc/dtc
> --
> The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum,
> a Linux Foundation Collaborative Project
>


-- 
Best Regards
Masahiro Yamada

Re: [PATCH] Kbuild: Update config_data.gz only if KCONFIG_CONFIG materially changed

[Elliot Berman]

On 4/2/2021 10:11 PM, Masahiro Yamada wrote:
> On Fri, Apr 2, 2021 at 7:45 AM Elliot Berman <eberman@codeaurora.org> wrote:
>>
>> If you update the timestamp of KCONFIG_CONFIG without actually changing
>> anything, config_data.gz is re-generated and causes vmlinux to re-link.
>> When Link Time Optimization is enabled, unnecessary re-linking of
>> vmlinux is highly desirable since it adds several minutes to build time.
>>
>> Avoid touching config_data.gz by using filechk to compare the existing
>> config_data.gz and update only if it changed.
>>
>> The .config can be touched, for instance, by a build script which
>> installs the default defconfig and then applies a defconfig fragment on
>> top.
>>
>> For a simple example on my x86 machine, I modified x86 default defconfig to set
>> CONFIG_IKCONFIG=y and run:
>>    make -j50 defconfig tiny.config vmlinux
>>    make -j50 defconfig tiny.config vmlinux
>> With this patch, vmlinux is not re-built as a result of config_data.gz
>> change.
>>
>> Signed-off-by: Elliot Berman <eberman@codeaurora.org>
>> ---
>>   kernel/Makefile      | 2 +-
>>   scripts/Makefile.lib | 2 ++
>>   2 files changed, 3 insertions(+), 1 deletion(-)
>>
>> diff --git a/kernel/Makefile b/kernel/Makefile
>> index 320f1f3..bd4e558 100644
>> --- a/kernel/Makefile
>> +++ b/kernel/Makefile
>> @@ -140,7 +140,7 @@ $(obj)/configs.o: $(obj)/config_data.gz
>>
>>   targets += config_data.gz
>>   $(obj)/config_data.gz: $(KCONFIG_CONFIG) FORCE
>> -       $(call if_changed,gzip)
>> +       $(call filechk,gzip)
> 
> 
> I do not think this is the right approach
> because gzip is executed every time, even
> if the time stamp is not changed.
> 

Since .config is relatively small, gzip was quickly producing the same 
binary in multiple runs on my host. I thought about following 
gen_ikheaders.sh approach of comparing the md5sum, but I felt it was 
more complex than using filechk and re-compressing every time. I'll send 
a v2 tomorrow which follows gen_ikheaders.sh approach.

> 
>>
>>   $(obj)/kheaders.o: $(obj)/kheaders_data.tar.xz
>>
>> diff --git a/scripts/Makefile.lib b/scripts/Makefile.lib
>> index a4fbaf8..81d3ec1 100644
>> --- a/scripts/Makefile.lib
>> +++ b/scripts/Makefile.lib
>> @@ -282,6 +282,8 @@ cmd_objcopy = $(OBJCOPY) $(OBJCOPYFLAGS) $(OBJCOPYFLAGS_$(@F)) $< $@
>>   quiet_cmd_gzip = GZIP    $@
>>         cmd_gzip = cat $(real-prereqs) | $(KGZIP) -n -f -9 > $@
>>
>> +filechk_gzip = cat $(real-prereqs) | $(KGZIP) -n -f -9
>> +
>>   # DTC
>>   # ---------------------------------------------------------------------------
>>   DTC ?= $(objtree)/scripts/dtc/dtc
>> --
>> The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum,
>> a Linux Foundation Collaborative Project
>>