* [PATCH 1/6] kbuild: specify output names separately for each emission type from rustc
2022-12-31 6:41 [PATCH 0/6] kbuild: fix dep-file processing for rust Masahiro Yamada
@ 2022-12-31 6:41 ` Masahiro Yamada
2022-12-31 20:01 ` Gary Guo
` (2 more replies)
2022-12-31 6:42 ` [PATCH 3/6] kbuild: remove sed commands after rustc rules Masahiro Yamada
` (2 subsequent siblings)
3 siblings, 3 replies; 15+ messages in thread
From: Masahiro Yamada @ 2022-12-31 6:41 UTC (permalink / raw)
To: linux-kbuild
Cc: linux-kernel, Miguel Ojeda, Masahiro Yamada, Alex Gaynor,
Björn Roy Baron, Boqun Feng, Gary Guo, Nathan Chancellor,
Nick Desaulniers, Nicolas Schier, Tom Rix, Wedson Almeida Filho,
llvm, rust-for-linux
In Kbuild, two different rules must not write to the same file, but
it happens when compiling rust source files.
For example, set CONFIG_SAMPLE_RUST_MINIMAL=m and run the following:
$ make -j$(nproc) samples/rust/rust_minimal.o samples/rust/rust_minimal.rsi \
samples/rust/rust_minimal.s samples/rust/rust_minimal.ll
[snip]
RUSTC [M] samples/rust/rust_minimal.o
RUSTC [M] samples/rust/rust_minimal.rsi
RUSTC [M] samples/rust/rust_minimal.s
RUSTC [M] samples/rust/rust_minimal.ll
mv: cannot stat 'samples/rust/rust_minimal.d': No such file or directory
make[3]: *** [scripts/Makefile.build:334: samples/rust/rust_minimal.ll] Error 1
make[3]: *** Waiting for unfinished jobs....
mv: cannot stat 'samples/rust/rust_minimal.d': No such file or directory
make[3]: *** [scripts/Makefile.build:309: samples/rust/rust_minimal.o] Error 1
mv: cannot stat 'samples/rust/rust_minimal.d': No such file or directory
make[3]: *** [scripts/Makefile.build:326: samples/rust/rust_minimal.s] Error 1
make[2]: *** [scripts/Makefile.build:504: samples/rust] Error 2
make[1]: *** [scripts/Makefile.build:504: samples] Error 2
make: *** [Makefile:2008: .] Error 2
The reason for the error is that 4 threads running in parallel creates
and renames the same file path, samples/rust/rust_minimal.d.
This does not happen when compiling C or assembly files because we
explicitly specify the dependency filename by using the preprocessor
option, -Wp,-MMD,$(depfile). $(depfile) is a unique path for each target.
Currently, rustc is only given --out-dir and the list of emitted types.
So, all the rust build rules output the dep-info into the default
<CRATE_NAME>.d, causing the conflict.
Fortunately, the --emit option is able to specify the output path
individually, with the form --emit=<type>=<path>.
Add --emit=dep-info=$(depfile) to the common command part. Also, remove
the redundant --out-dir because we specify the output path for each type.
The code gets much cleaner because we do not need to rename *.d files.
Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>
---
rust/Makefile | 10 ++++------
scripts/Makefile.build | 14 +++++++-------
scripts/Makefile.host | 9 +++------
3 files changed, 14 insertions(+), 19 deletions(-)
diff --git a/rust/Makefile b/rust/Makefile
index ff70c4c916f8..0e2a32f4b3e9 100644
--- a/rust/Makefile
+++ b/rust/Makefile
@@ -331,10 +331,9 @@ $(obj)/exports_kernel_generated.h: $(obj)/kernel.o FORCE
quiet_cmd_rustc_procmacro = $(RUSTC_OR_CLIPPY_QUIET) P $@
cmd_rustc_procmacro = \
$(RUSTC_OR_CLIPPY) $(rust_common_flags) \
- --emit=dep-info,link --extern proc_macro \
- --crate-type proc-macro --out-dir $(objtree)/$(obj) \
+ --emit=dep-info=$(depfile) --emit=link=$@ --extern proc_macro \
+ --crate-type proc-macro \
--crate-name $(patsubst lib%.so,%,$(notdir $@)) $<; \
- mv $(objtree)/$(obj)/$(patsubst lib%.so,%,$(notdir $@)).d $(depfile); \
sed -i '/^\#/d' $(depfile)
# Procedural macros can only be used with the `rustc` that compiled it.
@@ -348,10 +347,9 @@ quiet_cmd_rustc_library = $(if $(skip_clippy),RUSTC,$(RUSTC_OR_CLIPPY_QUIET)) L
OBJTREE=$(abspath $(objtree)) \
$(if $(skip_clippy),$(RUSTC),$(RUSTC_OR_CLIPPY)) \
$(filter-out $(skip_flags),$(rust_flags) $(rustc_target_flags)) \
- --emit=dep-info,obj,metadata --crate-type rlib \
- --out-dir $(objtree)/$(obj) -L$(objtree)/$(obj) \
+ --emit=dep-info=$(depfile) --emit=obj=$@ --emit=metadata=$(dir $@)$(patsubst %.o,lib%.rmeta,$(notdir $@)) \
+ --crate-type rlib -L$(objtree)/$(obj) \
--crate-name $(patsubst %.o,%,$(notdir $@)) $<; \
- mv $(objtree)/$(obj)/$(patsubst %.o,%,$(notdir $@)).d $(depfile); \
sed -i '/^\#/d' $(depfile) \
$(if $(rustc_objcopy),;$(OBJCOPY) $(rustc_objcopy) $@)
diff --git a/scripts/Makefile.build b/scripts/Makefile.build
index a0d5c6cca76d..40de20246e50 100644
--- a/scripts/Makefile.build
+++ b/scripts/Makefile.build
@@ -285,11 +285,11 @@ rust_common_cmd = \
-Zcrate-attr=no_std \
-Zcrate-attr='feature($(rust_allowed_features))' \
--extern alloc --extern kernel \
- --crate-type rlib --out-dir $(obj) -L $(objtree)/rust/ \
- --crate-name $(basename $(notdir $@))
+ --crate-type rlib -L $(objtree)/rust/ \
+ --crate-name $(basename $(notdir $@)) \
+ --emit=dep-info=$(depfile)
rust_handle_depfile = \
- mv $(obj)/$(basename $(notdir $@)).d $(depfile); \
sed -i '/^\#/d' $(depfile)
# `--emit=obj`, `--emit=asm` and `--emit=llvm-ir` imply a single codegen unit
@@ -302,7 +302,7 @@ rust_handle_depfile = \
quiet_cmd_rustc_o_rs = $(RUSTC_OR_CLIPPY_QUIET) $(quiet_modtag) $@
cmd_rustc_o_rs = \
- $(rust_common_cmd) --emit=dep-info,obj $<; \
+ $(rust_common_cmd) --emit=obj=$@ $<; \
$(rust_handle_depfile)
$(obj)/%.o: $(src)/%.rs FORCE
@@ -310,7 +310,7 @@ $(obj)/%.o: $(src)/%.rs FORCE
quiet_cmd_rustc_rsi_rs = $(RUSTC_OR_CLIPPY_QUIET) $(quiet_modtag) $@
cmd_rustc_rsi_rs = \
- $(rust_common_cmd) --emit=dep-info -Zunpretty=expanded $< >$@; \
+ $(rust_common_cmd) -Zunpretty=expanded $< >$@; \
command -v $(RUSTFMT) >/dev/null && $(RUSTFMT) $@; \
$(rust_handle_depfile)
@@ -319,7 +319,7 @@ $(obj)/%.rsi: $(src)/%.rs FORCE
quiet_cmd_rustc_s_rs = $(RUSTC_OR_CLIPPY_QUIET) $(quiet_modtag) $@
cmd_rustc_s_rs = \
- $(rust_common_cmd) --emit=dep-info,asm $<; \
+ $(rust_common_cmd) --emit=asm=$@ $<; \
$(rust_handle_depfile)
$(obj)/%.s: $(src)/%.rs FORCE
@@ -327,7 +327,7 @@ $(obj)/%.s: $(src)/%.rs FORCE
quiet_cmd_rustc_ll_rs = $(RUSTC_OR_CLIPPY_QUIET) $(quiet_modtag) $@
cmd_rustc_ll_rs = \
- $(rust_common_cmd) --emit=dep-info,llvm-ir $<; \
+ $(rust_common_cmd) --emit=llvm-ir=$@ $<; \
$(rust_handle_depfile)
$(obj)/%.ll: $(src)/%.rs FORCE
diff --git a/scripts/Makefile.host b/scripts/Makefile.host
index da133780b751..4434cdbf7b8e 100644
--- a/scripts/Makefile.host
+++ b/scripts/Makefile.host
@@ -84,8 +84,8 @@ _hostc_flags = $(KBUILD_HOSTCFLAGS) $(HOST_EXTRACFLAGS) \
$(HOSTCFLAGS_$(target-stem).o)
_hostcxx_flags = $(KBUILD_HOSTCXXFLAGS) $(HOST_EXTRACXXFLAGS) \
$(HOSTCXXFLAGS_$(target-stem).o)
-_hostrust_flags = $(KBUILD_HOSTRUSTFLAGS) $(HOST_EXTRARUSTFLAGS) \
- $(HOSTRUSTFLAGS_$(target-stem))
+hostrust_flags = $(KBUILD_HOSTRUSTFLAGS) $(HOST_EXTRARUSTFLAGS) \
+ $(HOSTRUSTFLAGS_$(target-stem)) --emit=dep-info=$(depfile)
# $(objtree)/$(obj) for including generated headers from checkin source files
ifeq ($(KBUILD_EXTMOD),)
@@ -97,7 +97,6 @@ endif
hostc_flags = -Wp,-MMD,$(depfile) $(_hostc_flags)
hostcxx_flags = -Wp,-MMD,$(depfile) $(_hostcxx_flags)
-hostrust_flags = $(_hostrust_flags)
#####
# Compile programs on the host
@@ -149,9 +148,7 @@ $(host-cxxobjs): $(obj)/%.o: $(src)/%.cc FORCE
# host-rust -> Executable
quiet_cmd_host-rust = HOSTRUSTC $@
cmd_host-rust = \
- $(HOSTRUSTC) $(hostrust_flags) --emit=dep-info,link \
- --out-dir=$(obj)/ $<; \
- mv $(obj)/$(target-stem).d $(depfile); \
+ $(HOSTRUSTC) $(hostrust_flags) --emit=link=$@ $<; \
sed -i '/^\#/d' $(depfile)
$(host-rust): $(obj)/%: $(src)/%.rs FORCE
$(call if_changed_dep,host-rust)
--
2.34.1
^ permalink raw reply related [flat|nested] 15+ messages in thread* Re: [PATCH 1/6] kbuild: specify output names separately for each emission type from rustc
2022-12-31 6:41 ` [PATCH 1/6] kbuild: specify output names separately for each emission type from rustc Masahiro Yamada
@ 2022-12-31 20:01 ` Gary Guo
2023-01-03 20:44 ` Miguel Ojeda
2023-01-06 9:24 ` Vincenzo
2 siblings, 0 replies; 15+ messages in thread
From: Gary Guo @ 2022-12-31 20:01 UTC (permalink / raw)
To: Masahiro Yamada
Cc: linux-kbuild, linux-kernel, Miguel Ojeda, Alex Gaynor,
Björn Roy Baron, Boqun Feng, Nathan Chancellor,
Nick Desaulniers, Nicolas Schier, Tom Rix, Wedson Almeida Filho,
llvm, rust-for-linux
On Sat, 31 Dec 2022 15:41:58 +0900
Masahiro Yamada <masahiroy@kernel.org> wrote:
> diff --git a/scripts/Makefile.host b/scripts/Makefile.host
> index da133780b751..4434cdbf7b8e 100644
> --- a/scripts/Makefile.host
> +++ b/scripts/Makefile.host
> @@ -84,8 +84,8 @@ _hostc_flags = $(KBUILD_HOSTCFLAGS) $(HOST_EXTRACFLAGS) \
> $(HOSTCFLAGS_$(target-stem).o)
> _hostcxx_flags = $(KBUILD_HOSTCXXFLAGS) $(HOST_EXTRACXXFLAGS) \
> $(HOSTCXXFLAGS_$(target-stem).o)
> -_hostrust_flags = $(KBUILD_HOSTRUSTFLAGS) $(HOST_EXTRARUSTFLAGS) \
> - $(HOSTRUSTFLAGS_$(target-stem))
> +hostrust_flags = $(KBUILD_HOSTRUSTFLAGS) $(HOST_EXTRARUSTFLAGS) \
> + $(HOSTRUSTFLAGS_$(target-stem)) --emit=dep-info=$(depfile)
>
> # $(objtree)/$(obj) for including generated headers from checkin source files
> ifeq ($(KBUILD_EXTMOD),)
> @@ -97,7 +97,6 @@ endif
>
> hostc_flags = -Wp,-MMD,$(depfile) $(_hostc_flags)
> hostcxx_flags = -Wp,-MMD,$(depfile) $(_hostcxx_flags)
> -hostrust_flags = $(_hostrust_flags)
Would it be better to have `--emit=dep-info=$(depfile)` added here
instead so that it mimics c/cxx flags?
>
> #####
> # Compile programs on the host
> @@ -149,9 +148,7 @@ $(host-cxxobjs): $(obj)/%.o: $(src)/%.cc FORCE
> # host-rust -> Executable
> quiet_cmd_host-rust = HOSTRUSTC $@
> cmd_host-rust = \
> - $(HOSTRUSTC) $(hostrust_flags) --emit=dep-info,link \
> - --out-dir=$(obj)/ $<; \
> - mv $(obj)/$(target-stem).d $(depfile); \
> + $(HOSTRUSTC) $(hostrust_flags) --emit=link=$@ $<; \
> sed -i '/^\#/d' $(depfile)
> $(host-rust): $(obj)/%: $(src)/%.rs FORCE
> $(call if_changed_dep,host-rust)
Best,
Gary
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [PATCH 1/6] kbuild: specify output names separately for each emission type from rustc
2022-12-31 6:41 ` [PATCH 1/6] kbuild: specify output names separately for each emission type from rustc Masahiro Yamada
2022-12-31 20:01 ` Gary Guo
@ 2023-01-03 20:44 ` Miguel Ojeda
2023-01-07 9:09 ` Masahiro Yamada
2023-01-06 9:24 ` Vincenzo
2 siblings, 1 reply; 15+ messages in thread
From: Miguel Ojeda @ 2023-01-03 20:44 UTC (permalink / raw)
To: Masahiro Yamada
Cc: linux-kbuild, linux-kernel, Miguel Ojeda, Alex Gaynor,
Björn Roy Baron, Boqun Feng, Gary Guo, Nathan Chancellor,
Nick Desaulniers, Nicolas Schier, Tom Rix, Wedson Almeida Filho,
llvm, rust-for-linux
On Sat, Dec 31, 2022 at 7:42 AM Masahiro Yamada <masahiroy@kernel.org> wrote:
>
> $ make -j$(nproc) samples/rust/rust_minimal.o samples/rust/rust_minimal.rsi \
> samples/rust/rust_minimal.s samples/rust/rust_minimal.ll
Yeah, we were testing the single targets, but not multiple at once, thanks!
> + --emit=dep-info=$(depfile) --emit=obj=$@ --emit=metadata=$(dir $@)$(patsubst %.o,lib%.rmeta,$(notdir $@)) \
Perhaps a newline here to avoid the lengthy line?
> hostc_flags = -Wp,-MMD,$(depfile) $(_hostc_flags)
> hostcxx_flags = -Wp,-MMD,$(depfile) $(_hostcxx_flags)
> -hostrust_flags = $(_hostrust_flags)
This was originally meant to be consistent with C and C++ indeed, but
if you prefer less variables, I guess it is fine, in which case,
should we update the C/C++ side too (in another series)?
Reviewed-by: Miguel Ojeda <ojeda@kernel.org>
Tested-by: Miguel Ojeda <ojeda@kernel.org>
Cheers,
Miguel
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [PATCH 1/6] kbuild: specify output names separately for each emission type from rustc
2023-01-03 20:44 ` Miguel Ojeda
@ 2023-01-07 9:09 ` Masahiro Yamada
0 siblings, 0 replies; 15+ messages in thread
From: Masahiro Yamada @ 2023-01-07 9:09 UTC (permalink / raw)
To: Miguel Ojeda
Cc: linux-kbuild, linux-kernel, Miguel Ojeda, Alex Gaynor,
Björn Roy Baron, Boqun Feng, Gary Guo, Nathan Chancellor,
Nick Desaulniers, Nicolas Schier, Tom Rix, Wedson Almeida Filho,
llvm, rust-for-linux
On Wed, Jan 4, 2023 at 5:45 AM Miguel Ojeda
<miguel.ojeda.sandonis@gmail.com> wrote:
>
> On Sat, Dec 31, 2022 at 7:42 AM Masahiro Yamada <masahiroy@kernel.org> wrote:
> >
> > $ make -j$(nproc) samples/rust/rust_minimal.o samples/rust/rust_minimal.rsi \
> > samples/rust/rust_minimal.s samples/rust/rust_minimal.ll
>
> Yeah, we were testing the single targets, but not multiple at once, thanks!
>
> > + --emit=dep-info=$(depfile) --emit=obj=$@ --emit=metadata=$(dir $@)$(patsubst %.o,lib%.rmeta,$(notdir $@)) \
>
> Perhaps a newline here to avoid the lengthy line?
OK, I will wrap it in v2.
>
> > hostc_flags = -Wp,-MMD,$(depfile) $(_hostc_flags)
> > hostcxx_flags = -Wp,-MMD,$(depfile) $(_hostcxx_flags)
> > -hostrust_flags = $(_hostrust_flags)
>
> This was originally meant to be consistent with C and C++ indeed, but
> if you prefer less variables, I guess it is fine, in which case,
> should we update the C/C++ side too (in another series)?
Yup, we could do this with less variables.
I will send a clean up.
> Reviewed-by: Miguel Ojeda <ojeda@kernel.org>
> Tested-by: Miguel Ojeda <ojeda@kernel.org>
>
> Cheers,
> Miguel
--
Best Regards
Masahiro Yamada
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [PATCH 1/6] kbuild: specify output names separately for each emission type from rustc
2022-12-31 6:41 ` [PATCH 1/6] kbuild: specify output names separately for each emission type from rustc Masahiro Yamada
2022-12-31 20:01 ` Gary Guo
2023-01-03 20:44 ` Miguel Ojeda
@ 2023-01-06 9:24 ` Vincenzo
2 siblings, 0 replies; 15+ messages in thread
From: Vincenzo @ 2023-01-06 9:24 UTC (permalink / raw)
To: Masahiro Yamada, linux-kbuild
Cc: linux-kernel, Miguel Ojeda, Alex Gaynor, Björn Roy Baron,
Boqun Feng, Gary Guo, Nathan Chancellor, Nick Desaulniers,
Nicolas Schier, Tom Rix, Wedson Almeida Filho, llvm,
rust-for-linux
Reviewed-by: Vincenzo Palazzo <vincenzopalazzodev@gmail.com>
On Sat Dec 31, 2022 at 7:41 AM CET, Masahiro Yamada wrote:
> In Kbuild, two different rules must not write to the same file, but
> it happens when compiling rust source files.
>
> For example, set CONFIG_SAMPLE_RUST_MINIMAL=m and run the following:
>
> $ make -j$(nproc) samples/rust/rust_minimal.o samples/rust/rust_minimal.rsi \
> samples/rust/rust_minimal.s samples/rust/rust_minimal.ll
> [snip]
> RUSTC [M] samples/rust/rust_minimal.o
> RUSTC [M] samples/rust/rust_minimal.rsi
> RUSTC [M] samples/rust/rust_minimal.s
> RUSTC [M] samples/rust/rust_minimal.ll
> mv: cannot stat 'samples/rust/rust_minimal.d': No such file or directory
> make[3]: *** [scripts/Makefile.build:334: samples/rust/rust_minimal.ll] Error 1
> make[3]: *** Waiting for unfinished jobs....
> mv: cannot stat 'samples/rust/rust_minimal.d': No such file or directory
> make[3]: *** [scripts/Makefile.build:309: samples/rust/rust_minimal.o] Error 1
> mv: cannot stat 'samples/rust/rust_minimal.d': No such file or directory
> make[3]: *** [scripts/Makefile.build:326: samples/rust/rust_minimal.s] Error 1
> make[2]: *** [scripts/Makefile.build:504: samples/rust] Error 2
> make[1]: *** [scripts/Makefile.build:504: samples] Error 2
> make: *** [Makefile:2008: .] Error 2
>
> The reason for the error is that 4 threads running in parallel creates
> and renames the same file path, samples/rust/rust_minimal.d.
>
> This does not happen when compiling C or assembly files because we
> explicitly specify the dependency filename by using the preprocessor
> option, -Wp,-MMD,$(depfile). $(depfile) is a unique path for each target.
>
> Currently, rustc is only given --out-dir and the list of emitted types.
> So, all the rust build rules output the dep-info into the default
> <CRATE_NAME>.d, causing the conflict.
>
> Fortunately, the --emit option is able to specify the output path
> individually, with the form --emit=<type>=<path>.
>
> Add --emit=dep-info=$(depfile) to the common command part. Also, remove
> the redundant --out-dir because we specify the output path for each type.
>
> The code gets much cleaner because we do not need to rename *.d files.
>
> Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>
> ---
>
> rust/Makefile | 10 ++++------
> scripts/Makefile.build | 14 +++++++-------
> scripts/Makefile.host | 9 +++------
> 3 files changed, 14 insertions(+), 19 deletions(-)
>
> diff --git a/rust/Makefile b/rust/Makefile
> index ff70c4c916f8..0e2a32f4b3e9 100644
> --- a/rust/Makefile
> +++ b/rust/Makefile
> @@ -331,10 +331,9 @@ $(obj)/exports_kernel_generated.h: $(obj)/kernel.o FORCE
> quiet_cmd_rustc_procmacro = $(RUSTC_OR_CLIPPY_QUIET) P $@
> cmd_rustc_procmacro = \
> $(RUSTC_OR_CLIPPY) $(rust_common_flags) \
> - --emit=dep-info,link --extern proc_macro \
> - --crate-type proc-macro --out-dir $(objtree)/$(obj) \
> + --emit=dep-info=$(depfile) --emit=link=$@ --extern proc_macro \
> + --crate-type proc-macro \
> --crate-name $(patsubst lib%.so,%,$(notdir $@)) $<; \
> - mv $(objtree)/$(obj)/$(patsubst lib%.so,%,$(notdir $@)).d $(depfile); \
> sed -i '/^\#/d' $(depfile)
>
> # Procedural macros can only be used with the `rustc` that compiled it.
> @@ -348,10 +347,9 @@ quiet_cmd_rustc_library = $(if $(skip_clippy),RUSTC,$(RUSTC_OR_CLIPPY_QUIET)) L
> OBJTREE=$(abspath $(objtree)) \
> $(if $(skip_clippy),$(RUSTC),$(RUSTC_OR_CLIPPY)) \
> $(filter-out $(skip_flags),$(rust_flags) $(rustc_target_flags)) \
> - --emit=dep-info,obj,metadata --crate-type rlib \
> - --out-dir $(objtree)/$(obj) -L$(objtree)/$(obj) \
> + --emit=dep-info=$(depfile) --emit=obj=$@ --emit=metadata=$(dir $@)$(patsubst %.o,lib%.rmeta,$(notdir $@)) \
> + --crate-type rlib -L$(objtree)/$(obj) \
> --crate-name $(patsubst %.o,%,$(notdir $@)) $<; \
> - mv $(objtree)/$(obj)/$(patsubst %.o,%,$(notdir $@)).d $(depfile); \
> sed -i '/^\#/d' $(depfile) \
> $(if $(rustc_objcopy),;$(OBJCOPY) $(rustc_objcopy) $@)
>
> diff --git a/scripts/Makefile.build b/scripts/Makefile.build
> index a0d5c6cca76d..40de20246e50 100644
> --- a/scripts/Makefile.build
> +++ b/scripts/Makefile.build
> @@ -285,11 +285,11 @@ rust_common_cmd = \
> -Zcrate-attr=no_std \
> -Zcrate-attr='feature($(rust_allowed_features))' \
> --extern alloc --extern kernel \
> - --crate-type rlib --out-dir $(obj) -L $(objtree)/rust/ \
> - --crate-name $(basename $(notdir $@))
> + --crate-type rlib -L $(objtree)/rust/ \
> + --crate-name $(basename $(notdir $@)) \
> + --emit=dep-info=$(depfile)
>
> rust_handle_depfile = \
> - mv $(obj)/$(basename $(notdir $@)).d $(depfile); \
> sed -i '/^\#/d' $(depfile)
>
> # `--emit=obj`, `--emit=asm` and `--emit=llvm-ir` imply a single codegen unit
> @@ -302,7 +302,7 @@ rust_handle_depfile = \
>
> quiet_cmd_rustc_o_rs = $(RUSTC_OR_CLIPPY_QUIET) $(quiet_modtag) $@
> cmd_rustc_o_rs = \
> - $(rust_common_cmd) --emit=dep-info,obj $<; \
> + $(rust_common_cmd) --emit=obj=$@ $<; \
> $(rust_handle_depfile)
>
> $(obj)/%.o: $(src)/%.rs FORCE
> @@ -310,7 +310,7 @@ $(obj)/%.o: $(src)/%.rs FORCE
>
> quiet_cmd_rustc_rsi_rs = $(RUSTC_OR_CLIPPY_QUIET) $(quiet_modtag) $@
> cmd_rustc_rsi_rs = \
> - $(rust_common_cmd) --emit=dep-info -Zunpretty=expanded $< >$@; \
> + $(rust_common_cmd) -Zunpretty=expanded $< >$@; \
> command -v $(RUSTFMT) >/dev/null && $(RUSTFMT) $@; \
> $(rust_handle_depfile)
>
> @@ -319,7 +319,7 @@ $(obj)/%.rsi: $(src)/%.rs FORCE
>
> quiet_cmd_rustc_s_rs = $(RUSTC_OR_CLIPPY_QUIET) $(quiet_modtag) $@
> cmd_rustc_s_rs = \
> - $(rust_common_cmd) --emit=dep-info,asm $<; \
> + $(rust_common_cmd) --emit=asm=$@ $<; \
> $(rust_handle_depfile)
>
> $(obj)/%.s: $(src)/%.rs FORCE
> @@ -327,7 +327,7 @@ $(obj)/%.s: $(src)/%.rs FORCE
>
> quiet_cmd_rustc_ll_rs = $(RUSTC_OR_CLIPPY_QUIET) $(quiet_modtag) $@
> cmd_rustc_ll_rs = \
> - $(rust_common_cmd) --emit=dep-info,llvm-ir $<; \
> + $(rust_common_cmd) --emit=llvm-ir=$@ $<; \
> $(rust_handle_depfile)
>
> $(obj)/%.ll: $(src)/%.rs FORCE
> diff --git a/scripts/Makefile.host b/scripts/Makefile.host
> index da133780b751..4434cdbf7b8e 100644
> --- a/scripts/Makefile.host
> +++ b/scripts/Makefile.host
> @@ -84,8 +84,8 @@ _hostc_flags = $(KBUILD_HOSTCFLAGS) $(HOST_EXTRACFLAGS) \
> $(HOSTCFLAGS_$(target-stem).o)
> _hostcxx_flags = $(KBUILD_HOSTCXXFLAGS) $(HOST_EXTRACXXFLAGS) \
> $(HOSTCXXFLAGS_$(target-stem).o)
> -_hostrust_flags = $(KBUILD_HOSTRUSTFLAGS) $(HOST_EXTRARUSTFLAGS) \
> - $(HOSTRUSTFLAGS_$(target-stem))
> +hostrust_flags = $(KBUILD_HOSTRUSTFLAGS) $(HOST_EXTRARUSTFLAGS) \
> + $(HOSTRUSTFLAGS_$(target-stem)) --emit=dep-info=$(depfile)
>
> # $(objtree)/$(obj) for including generated headers from checkin source files
> ifeq ($(KBUILD_EXTMOD),)
> @@ -97,7 +97,6 @@ endif
>
> hostc_flags = -Wp,-MMD,$(depfile) $(_hostc_flags)
> hostcxx_flags = -Wp,-MMD,$(depfile) $(_hostcxx_flags)
> -hostrust_flags = $(_hostrust_flags)
>
> #####
> # Compile programs on the host
> @@ -149,9 +148,7 @@ $(host-cxxobjs): $(obj)/%.o: $(src)/%.cc FORCE
> # host-rust -> Executable
> quiet_cmd_host-rust = HOSTRUSTC $@
> cmd_host-rust = \
> - $(HOSTRUSTC) $(hostrust_flags) --emit=dep-info,link \
> - --out-dir=$(obj)/ $<; \
> - mv $(obj)/$(target-stem).d $(depfile); \
> + $(HOSTRUSTC) $(hostrust_flags) --emit=link=$@ $<; \
> sed -i '/^\#/d' $(depfile)
> $(host-rust): $(obj)/%: $(src)/%.rs FORCE
> $(call if_changed_dep,host-rust)
> --
> 2.34.1
^ permalink raw reply [flat|nested] 15+ messages in thread
* [PATCH 3/6] kbuild: remove sed commands after rustc rules
2022-12-31 6:41 [PATCH 0/6] kbuild: fix dep-file processing for rust Masahiro Yamada
2022-12-31 6:41 ` [PATCH 1/6] kbuild: specify output names separately for each emission type from rustc Masahiro Yamada
@ 2022-12-31 6:42 ` Masahiro Yamada
2023-01-03 20:45 ` Miguel Ojeda
2023-01-06 9:28 ` Vincenzo
2022-12-31 6:42 ` [PATCH 5/6] fixdep: avoid parsing the same file over again Masahiro Yamada
2022-12-31 13:34 ` [PATCH 0/6] kbuild: fix dep-file processing for rust Miguel Ojeda
3 siblings, 2 replies; 15+ messages in thread
From: Masahiro Yamada @ 2022-12-31 6:42 UTC (permalink / raw)
To: linux-kbuild
Cc: linux-kernel, Miguel Ojeda, Masahiro Yamada, Alex Gaynor,
Björn Roy Baron, Boqun Feng, Gary Guo, Nathan Chancellor,
Nick Desaulniers, Nicolas Schier, Tom Rix, Wedson Almeida Filho,
llvm, rust-for-linux
rustc may put comments in dep-info, so sed is used to drop them before
passing it to fixdep.
Now that fixdep can remove comments, Makefiles do not need to run sed.
Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>
---
rust/Makefile | 6 ++----
scripts/Makefile.build | 18 ++++--------------
scripts/Makefile.host | 3 +--
3 files changed, 7 insertions(+), 20 deletions(-)
diff --git a/rust/Makefile b/rust/Makefile
index 0e2a32f4b3e9..c8941fec6955 100644
--- a/rust/Makefile
+++ b/rust/Makefile
@@ -333,8 +333,7 @@ quiet_cmd_rustc_procmacro = $(RUSTC_OR_CLIPPY_QUIET) P $@
$(RUSTC_OR_CLIPPY) $(rust_common_flags) \
--emit=dep-info=$(depfile) --emit=link=$@ --extern proc_macro \
--crate-type proc-macro \
- --crate-name $(patsubst lib%.so,%,$(notdir $@)) $<; \
- sed -i '/^\#/d' $(depfile)
+ --crate-name $(patsubst lib%.so,%,$(notdir $@)) $<
# Procedural macros can only be used with the `rustc` that compiled it.
# Therefore, to get `libmacros.so` automatically recompiled when the compiler
@@ -349,8 +348,7 @@ quiet_cmd_rustc_library = $(if $(skip_clippy),RUSTC,$(RUSTC_OR_CLIPPY_QUIET)) L
$(filter-out $(skip_flags),$(rust_flags) $(rustc_target_flags)) \
--emit=dep-info=$(depfile) --emit=obj=$@ --emit=metadata=$(dir $@)$(patsubst %.o,lib%.rmeta,$(notdir $@)) \
--crate-type rlib -L$(objtree)/$(obj) \
- --crate-name $(patsubst %.o,%,$(notdir $@)) $<; \
- sed -i '/^\#/d' $(depfile) \
+ --crate-name $(patsubst %.o,%,$(notdir $@)) $< \
$(if $(rustc_objcopy),;$(OBJCOPY) $(rustc_objcopy) $@)
rust-analyzer:
diff --git a/scripts/Makefile.build b/scripts/Makefile.build
index 40de20246e50..76323201232a 100644
--- a/scripts/Makefile.build
+++ b/scripts/Makefile.build
@@ -289,9 +289,6 @@ rust_common_cmd = \
--crate-name $(basename $(notdir $@)) \
--emit=dep-info=$(depfile)
-rust_handle_depfile = \
- sed -i '/^\#/d' $(depfile)
-
# `--emit=obj`, `--emit=asm` and `--emit=llvm-ir` imply a single codegen unit
# will be used. We explicitly request `-Ccodegen-units=1` in any case, and
# the compiler shows a warning if it is not 1. However, if we ever stop
@@ -301,9 +298,7 @@ rust_handle_depfile = \
# would not match each other.
quiet_cmd_rustc_o_rs = $(RUSTC_OR_CLIPPY_QUIET) $(quiet_modtag) $@
- cmd_rustc_o_rs = \
- $(rust_common_cmd) --emit=obj=$@ $<; \
- $(rust_handle_depfile)
+ cmd_rustc_o_rs = $(rust_common_cmd) --emit=obj=$@ $<
$(obj)/%.o: $(src)/%.rs FORCE
$(call if_changed_dep,rustc_o_rs)
@@ -311,24 +306,19 @@ $(obj)/%.o: $(src)/%.rs FORCE
quiet_cmd_rustc_rsi_rs = $(RUSTC_OR_CLIPPY_QUIET) $(quiet_modtag) $@
cmd_rustc_rsi_rs = \
$(rust_common_cmd) -Zunpretty=expanded $< >$@; \
- command -v $(RUSTFMT) >/dev/null && $(RUSTFMT) $@; \
- $(rust_handle_depfile)
+ command -v $(RUSTFMT) >/dev/null && $(RUSTFMT) $@
$(obj)/%.rsi: $(src)/%.rs FORCE
$(call if_changed_dep,rustc_rsi_rs)
quiet_cmd_rustc_s_rs = $(RUSTC_OR_CLIPPY_QUIET) $(quiet_modtag) $@
- cmd_rustc_s_rs = \
- $(rust_common_cmd) --emit=asm=$@ $<; \
- $(rust_handle_depfile)
+ cmd_rustc_s_rs = $(rust_common_cmd) --emit=asm=$@ $<
$(obj)/%.s: $(src)/%.rs FORCE
$(call if_changed_dep,rustc_s_rs)
quiet_cmd_rustc_ll_rs = $(RUSTC_OR_CLIPPY_QUIET) $(quiet_modtag) $@
- cmd_rustc_ll_rs = \
- $(rust_common_cmd) --emit=llvm-ir=$@ $<; \
- $(rust_handle_depfile)
+ cmd_rustc_ll_rs = $(rust_common_cmd) --emit=llvm-ir=$@ $<
$(obj)/%.ll: $(src)/%.rs FORCE
$(call if_changed_dep,rustc_ll_rs)
diff --git a/scripts/Makefile.host b/scripts/Makefile.host
index 4434cdbf7b8e..bc782655d09e 100644
--- a/scripts/Makefile.host
+++ b/scripts/Makefile.host
@@ -148,8 +148,7 @@ $(host-cxxobjs): $(obj)/%.o: $(src)/%.cc FORCE
# host-rust -> Executable
quiet_cmd_host-rust = HOSTRUSTC $@
cmd_host-rust = \
- $(HOSTRUSTC) $(hostrust_flags) --emit=link=$@ $<; \
- sed -i '/^\#/d' $(depfile)
+ $(HOSTRUSTC) $(hostrust_flags) --emit=link=$@ $<
$(host-rust): $(obj)/%: $(src)/%.rs FORCE
$(call if_changed_dep,host-rust)
--
2.34.1
^ permalink raw reply related [flat|nested] 15+ messages in thread* Re: [PATCH 3/6] kbuild: remove sed commands after rustc rules
2022-12-31 6:42 ` [PATCH 3/6] kbuild: remove sed commands after rustc rules Masahiro Yamada
@ 2023-01-03 20:45 ` Miguel Ojeda
2023-01-06 9:28 ` Vincenzo
1 sibling, 0 replies; 15+ messages in thread
From: Miguel Ojeda @ 2023-01-03 20:45 UTC (permalink / raw)
To: Masahiro Yamada
Cc: linux-kbuild, linux-kernel, Miguel Ojeda, Alex Gaynor,
Björn Roy Baron, Boqun Feng, Gary Guo, Nathan Chancellor,
Nick Desaulniers, Nicolas Schier, Tom Rix, Wedson Almeida Filho,
llvm, rust-for-linux
On Sat, Dec 31, 2022 at 7:42 AM Masahiro Yamada <masahiroy@kernel.org> wrote:
>
> rustc may put comments in dep-info, so sed is used to drop them before
> passing it to fixdep.
>
> Now that fixdep can remove comments, Makefiles do not need to run sed.
Finally!
Reviewed-by: Miguel Ojeda <ojeda@kernel.org>
Tested-by: Miguel Ojeda <ojeda@kernel.org>
Cheers,
Miguel
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [PATCH 3/6] kbuild: remove sed commands after rustc rules
2022-12-31 6:42 ` [PATCH 3/6] kbuild: remove sed commands after rustc rules Masahiro Yamada
2023-01-03 20:45 ` Miguel Ojeda
@ 2023-01-06 9:28 ` Vincenzo
1 sibling, 0 replies; 15+ messages in thread
From: Vincenzo @ 2023-01-06 9:28 UTC (permalink / raw)
To: Masahiro Yamada, linux-kbuild
Cc: linux-kernel, Miguel Ojeda, Alex Gaynor, Björn Roy Baron,
Boqun Feng, Gary Guo, Nathan Chancellor, Nick Desaulniers,
Nicolas Schier, Tom Rix, Wedson Almeida Filho, llvm,
rust-for-linux
Reviewed-by: Vincenzo Palazzo <vincenzopalazzodev@gmail.com>
> rustc may put comments in dep-info, so sed is used to drop them before
> passing it to fixdep.
>
> Now that fixdep can remove comments, Makefiles do not need to run sed.
>
> Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>
> ---
>
> rust/Makefile | 6 ++----
> scripts/Makefile.build | 18 ++++--------------
> scripts/Makefile.host | 3 +--
> 3 files changed, 7 insertions(+), 20 deletions(-)
>
> diff --git a/rust/Makefile b/rust/Makefile
> index 0e2a32f4b3e9..c8941fec6955 100644
> --- a/rust/Makefile
> +++ b/rust/Makefile
> @@ -333,8 +333,7 @@ quiet_cmd_rustc_procmacro = $(RUSTC_OR_CLIPPY_QUIET) P $@
> $(RUSTC_OR_CLIPPY) $(rust_common_flags) \
> --emit=dep-info=$(depfile) --emit=link=$@ --extern proc_macro \
> --crate-type proc-macro \
> - --crate-name $(patsubst lib%.so,%,$(notdir $@)) $<; \
> - sed -i '/^\#/d' $(depfile)
> + --crate-name $(patsubst lib%.so,%,$(notdir $@)) $<
>
> # Procedural macros can only be used with the `rustc` that compiled it.
> # Therefore, to get `libmacros.so` automatically recompiled when the compiler
> @@ -349,8 +348,7 @@ quiet_cmd_rustc_library = $(if $(skip_clippy),RUSTC,$(RUSTC_OR_CLIPPY_QUIET)) L
> $(filter-out $(skip_flags),$(rust_flags) $(rustc_target_flags)) \
> --emit=dep-info=$(depfile) --emit=obj=$@ --emit=metadata=$(dir $@)$(patsubst %.o,lib%.rmeta,$(notdir $@)) \
> --crate-type rlib -L$(objtree)/$(obj) \
> - --crate-name $(patsubst %.o,%,$(notdir $@)) $<; \
> - sed -i '/^\#/d' $(depfile) \
> + --crate-name $(patsubst %.o,%,$(notdir $@)) $< \
> $(if $(rustc_objcopy),;$(OBJCOPY) $(rustc_objcopy) $@)
>
> rust-analyzer:
> diff --git a/scripts/Makefile.build b/scripts/Makefile.build
> index 40de20246e50..76323201232a 100644
> --- a/scripts/Makefile.build
> +++ b/scripts/Makefile.build
> @@ -289,9 +289,6 @@ rust_common_cmd = \
> --crate-name $(basename $(notdir $@)) \
> --emit=dep-info=$(depfile)
>
> -rust_handle_depfile = \
> - sed -i '/^\#/d' $(depfile)
> -
> # `--emit=obj`, `--emit=asm` and `--emit=llvm-ir` imply a single codegen unit
> # will be used. We explicitly request `-Ccodegen-units=1` in any case, and
> # the compiler shows a warning if it is not 1. However, if we ever stop
> @@ -301,9 +298,7 @@ rust_handle_depfile = \
> # would not match each other.
>
> quiet_cmd_rustc_o_rs = $(RUSTC_OR_CLIPPY_QUIET) $(quiet_modtag) $@
> - cmd_rustc_o_rs = \
> - $(rust_common_cmd) --emit=obj=$@ $<; \
> - $(rust_handle_depfile)
> + cmd_rustc_o_rs = $(rust_common_cmd) --emit=obj=$@ $<
>
> $(obj)/%.o: $(src)/%.rs FORCE
> $(call if_changed_dep,rustc_o_rs)
> @@ -311,24 +306,19 @@ $(obj)/%.o: $(src)/%.rs FORCE
> quiet_cmd_rustc_rsi_rs = $(RUSTC_OR_CLIPPY_QUIET) $(quiet_modtag) $@
> cmd_rustc_rsi_rs = \
> $(rust_common_cmd) -Zunpretty=expanded $< >$@; \
> - command -v $(RUSTFMT) >/dev/null && $(RUSTFMT) $@; \
> - $(rust_handle_depfile)
> + command -v $(RUSTFMT) >/dev/null && $(RUSTFMT) $@
>
> $(obj)/%.rsi: $(src)/%.rs FORCE
> $(call if_changed_dep,rustc_rsi_rs)
>
> quiet_cmd_rustc_s_rs = $(RUSTC_OR_CLIPPY_QUIET) $(quiet_modtag) $@
> - cmd_rustc_s_rs = \
> - $(rust_common_cmd) --emit=asm=$@ $<; \
> - $(rust_handle_depfile)
> + cmd_rustc_s_rs = $(rust_common_cmd) --emit=asm=$@ $<
>
> $(obj)/%.s: $(src)/%.rs FORCE
> $(call if_changed_dep,rustc_s_rs)
>
> quiet_cmd_rustc_ll_rs = $(RUSTC_OR_CLIPPY_QUIET) $(quiet_modtag) $@
> - cmd_rustc_ll_rs = \
> - $(rust_common_cmd) --emit=llvm-ir=$@ $<; \
> - $(rust_handle_depfile)
> + cmd_rustc_ll_rs = $(rust_common_cmd) --emit=llvm-ir=$@ $<
>
> $(obj)/%.ll: $(src)/%.rs FORCE
> $(call if_changed_dep,rustc_ll_rs)
> diff --git a/scripts/Makefile.host b/scripts/Makefile.host
> index 4434cdbf7b8e..bc782655d09e 100644
> --- a/scripts/Makefile.host
> +++ b/scripts/Makefile.host
> @@ -148,8 +148,7 @@ $(host-cxxobjs): $(obj)/%.o: $(src)/%.cc FORCE
> # host-rust -> Executable
> quiet_cmd_host-rust = HOSTRUSTC $@
> cmd_host-rust = \
> - $(HOSTRUSTC) $(hostrust_flags) --emit=link=$@ $<; \
> - sed -i '/^\#/d' $(depfile)
> + $(HOSTRUSTC) $(hostrust_flags) --emit=link=$@ $<
> $(host-rust): $(obj)/%: $(src)/%.rs FORCE
> $(call if_changed_dep,host-rust)
>
> --
> 2.34.1
^ permalink raw reply [flat|nested] 15+ messages in thread
* [PATCH 5/6] fixdep: avoid parsing the same file over again
2022-12-31 6:41 [PATCH 0/6] kbuild: fix dep-file processing for rust Masahiro Yamada
2022-12-31 6:41 ` [PATCH 1/6] kbuild: specify output names separately for each emission type from rustc Masahiro Yamada
2022-12-31 6:42 ` [PATCH 3/6] kbuild: remove sed commands after rustc rules Masahiro Yamada
@ 2022-12-31 6:42 ` Masahiro Yamada
2023-01-03 20:46 ` Miguel Ojeda
2023-01-06 9:31 ` Vincenzo
2022-12-31 13:34 ` [PATCH 0/6] kbuild: fix dep-file processing for rust Miguel Ojeda
3 siblings, 2 replies; 15+ messages in thread
From: Masahiro Yamada @ 2022-12-31 6:42 UTC (permalink / raw)
To: linux-kbuild
Cc: linux-kernel, Miguel Ojeda, Masahiro Yamada, Alex Gaynor,
Björn Roy Baron, Boqun Feng, Gary Guo, Nathan Chancellor,
Nick Desaulniers, Nicolas Schier, Wedson Almeida Filho,
rust-for-linux
The dep files (*.d files) emitted by C compilers usually contain the
deduplicated list of included files.
There is an exceptional case; if a header is included by the -include
command line option, and also by #include directive, it appears twice
in the *.d file.
For example, the top Makefile specifies the command line option,
-include $(srctree)/include/linux/kconfig.h. You do not need to
add #include <linux/kconfig.h> in every source file.
In fact, include/linux/kconfig.h is listed twice in many .*.cmd files
due to include/linux/xarray.h including <linux/kconfig.h>.
I did not fix that since it is a small redundancy.
However, this is more annoying for rustc. rustc emits the dependency
for each emission type.
For example, cmd_rustc_library emits dep-info, obj, and metadata.
So, the emitted *.d file contains the dependency for those 3 targets,
which makes fixdep parse the same file 3 times.
$ grep rust/alloc/raw_vec.rs rust/.alloc.o.cmd
rust/alloc/raw_vec.rs \
rust/alloc/raw_vec.rs \
rust/alloc/raw_vec.rs \
To skip the second parsing, this commit adds a hash table for parsed
files, just like we did for CONFIG options.
Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>
---
scripts/basic/fixdep.c | 9 +++++++--
1 file changed, 7 insertions(+), 2 deletions(-)
diff --git a/scripts/basic/fixdep.c b/scripts/basic/fixdep.c
index b20777b888d7..cc8f6d34c2ca 100644
--- a/scripts/basic/fixdep.c
+++ b/scripts/basic/fixdep.c
@@ -113,7 +113,7 @@ struct item {
};
#define HASHSZ 256
-static struct item *config_hashtab[HASHSZ];
+static struct item *config_hashtab[HASHSZ], *file_hashtab[HASHSZ];
static unsigned int strhash(const char *str, unsigned int sz)
{
@@ -361,6 +361,10 @@ static void parse_dep_file(char *p, const char *target)
* name, which will be the original one, and ignore any
* other source names, which will be intermediate
* temporary files.
+ *
+ * rustc emits the same dependency list for each
+ * emission type. It is enough to list the source name
+ * just once.
*/
if (!saw_any_target) {
saw_any_target = true;
@@ -368,7 +372,8 @@ static void parse_dep_file(char *p, const char *target)
printf("deps_%s := \\\n", target);
need_parse = true;
}
- } else if (!is_ignored_file(p, q - p)) {
+ } else if (!is_ignored_file(p, q - p) &&
+ !in_hashtable(p, q - p, file_hashtab)) {
printf(" %s \\\n", p);
need_parse = true;
}
--
2.34.1
^ permalink raw reply related [flat|nested] 15+ messages in thread* Re: [PATCH 5/6] fixdep: avoid parsing the same file over again
2022-12-31 6:42 ` [PATCH 5/6] fixdep: avoid parsing the same file over again Masahiro Yamada
@ 2023-01-03 20:46 ` Miguel Ojeda
2023-01-06 9:31 ` Vincenzo
1 sibling, 0 replies; 15+ messages in thread
From: Miguel Ojeda @ 2023-01-03 20:46 UTC (permalink / raw)
To: Masahiro Yamada
Cc: linux-kbuild, linux-kernel, Miguel Ojeda, Alex Gaynor,
Björn Roy Baron, Boqun Feng, Gary Guo, Nathan Chancellor,
Nick Desaulniers, Nicolas Schier, Wedson Almeida Filho,
rust-for-linux
On Sat, Dec 31, 2022 at 7:42 AM Masahiro Yamada <masahiroy@kernel.org> wrote:
>
> To skip the second parsing, this commit adds a hash table for parsed
> files, just like we did for CONFIG options.
Acked-by: Miguel Ojeda <ojeda@kernel.org>
Tested-by: Miguel Ojeda <ojeda@kernel.org>
Cheers,
Miguel
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [PATCH 5/6] fixdep: avoid parsing the same file over again
2022-12-31 6:42 ` [PATCH 5/6] fixdep: avoid parsing the same file over again Masahiro Yamada
2023-01-03 20:46 ` Miguel Ojeda
@ 2023-01-06 9:31 ` Vincenzo
1 sibling, 0 replies; 15+ messages in thread
From: Vincenzo @ 2023-01-06 9:31 UTC (permalink / raw)
To: Masahiro Yamada, linux-kbuild
Cc: linux-kernel, Miguel Ojeda, Alex Gaynor, Björn Roy Baron,
Boqun Feng, Gary Guo, Nathan Chancellor, Nick Desaulniers,
Nicolas Schier, Wedson Almeida Filho, rust-for-linux
Reviewed-by: Vincenzo Palazzo <vincenzopalazzodev@gmail.com>
On Sat Dec 31, 2022 at 7:42 AM CET, Masahiro Yamada wrote:
> The dep files (*.d files) emitted by C compilers usually contain the
> deduplicated list of included files.
>
> There is an exceptional case; if a header is included by the -include
> command line option, and also by #include directive, it appears twice
> in the *.d file.
>
> For example, the top Makefile specifies the command line option,
> -include $(srctree)/include/linux/kconfig.h. You do not need to
> add #include <linux/kconfig.h> in every source file.
>
> In fact, include/linux/kconfig.h is listed twice in many .*.cmd files
> due to include/linux/xarray.h including <linux/kconfig.h>.
> I did not fix that since it is a small redundancy.
>
> However, this is more annoying for rustc. rustc emits the dependency
> for each emission type.
>
> For example, cmd_rustc_library emits dep-info, obj, and metadata.
> So, the emitted *.d file contains the dependency for those 3 targets,
> which makes fixdep parse the same file 3 times.
>
> $ grep rust/alloc/raw_vec.rs rust/.alloc.o.cmd
> rust/alloc/raw_vec.rs \
> rust/alloc/raw_vec.rs \
> rust/alloc/raw_vec.rs \
>
> To skip the second parsing, this commit adds a hash table for parsed
> files, just like we did for CONFIG options.
>
> Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>
> ---
>
> scripts/basic/fixdep.c | 9 +++++++--
> 1 file changed, 7 insertions(+), 2 deletions(-)
>
> diff --git a/scripts/basic/fixdep.c b/scripts/basic/fixdep.c
> index b20777b888d7..cc8f6d34c2ca 100644
> --- a/scripts/basic/fixdep.c
> +++ b/scripts/basic/fixdep.c
> @@ -113,7 +113,7 @@ struct item {
> };
>
> #define HASHSZ 256
> -static struct item *config_hashtab[HASHSZ];
> +static struct item *config_hashtab[HASHSZ], *file_hashtab[HASHSZ];
>
> static unsigned int strhash(const char *str, unsigned int sz)
> {
> @@ -361,6 +361,10 @@ static void parse_dep_file(char *p, const char *target)
> * name, which will be the original one, and ignore any
> * other source names, which will be intermediate
> * temporary files.
> + *
> + * rustc emits the same dependency list for each
> + * emission type. It is enough to list the source name
> + * just once.
> */
> if (!saw_any_target) {
> saw_any_target = true;
> @@ -368,7 +372,8 @@ static void parse_dep_file(char *p, const char *target)
> printf("deps_%s := \\\n", target);
> need_parse = true;
> }
> - } else if (!is_ignored_file(p, q - p)) {
> + } else if (!is_ignored_file(p, q - p) &&
> + !in_hashtable(p, q - p, file_hashtab)) {
> printf(" %s \\\n", p);
> need_parse = true;
> }
> --
> 2.34.1
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [PATCH 0/6] kbuild: fix dep-file processing for rust
2022-12-31 6:41 [PATCH 0/6] kbuild: fix dep-file processing for rust Masahiro Yamada
` (2 preceding siblings ...)
2022-12-31 6:42 ` [PATCH 5/6] fixdep: avoid parsing the same file over again Masahiro Yamada
@ 2022-12-31 13:34 ` Miguel Ojeda
2022-12-31 15:05 ` Masahiro Yamada
3 siblings, 1 reply; 15+ messages in thread
From: Miguel Ojeda @ 2022-12-31 13:34 UTC (permalink / raw)
To: Masahiro Yamada
Cc: linux-kbuild, linux-kernel, Miguel Ojeda, Alex Gaynor,
Björn Roy Baron, Boqun Feng, Gary Guo, Nathan Chancellor,
Nick Desaulniers, Nicolas Schier, Tom Rix, Wedson Almeida Filho,
llvm, rust-for-linux
On Sat, Dec 31, 2022 at 7:42 AM Masahiro Yamada <masahiroy@kernel.org> wrote:
>
> Masahiro Yamada (6):
> kbuild: specify output names separately for each emission type from
> rustc
> fixdep: parse Makefile more correctly to handle comments etc.
> kbuild: remove sed commands after rustc rules
> fixdep: refactor hash table lookup
> fixdep: avoid parsing the same file over again
> fixdep: do not parse *.so, *.rmeta, *.rlib
These cleanups are great, and it is a pleasure to see proper
integration with `fixdep` -- thanks a ton! :)
Will you want to take them through the kbuild tree? (I guess so, given
the bulk of it is on `fixdep`)
Cheers,
Miguel
^ permalink raw reply [flat|nested] 15+ messages in thread* Re: [PATCH 0/6] kbuild: fix dep-file processing for rust
2022-12-31 13:34 ` [PATCH 0/6] kbuild: fix dep-file processing for rust Miguel Ojeda
@ 2022-12-31 15:05 ` Masahiro Yamada
2023-01-03 20:48 ` Miguel Ojeda
0 siblings, 1 reply; 15+ messages in thread
From: Masahiro Yamada @ 2022-12-31 15:05 UTC (permalink / raw)
To: Miguel Ojeda
Cc: linux-kbuild, linux-kernel, Miguel Ojeda, Alex Gaynor,
Björn Roy Baron, Boqun Feng, Gary Guo, Nathan Chancellor,
Nick Desaulniers, Nicolas Schier, Tom Rix, Wedson Almeida Filho,
llvm, rust-for-linux
On Sat, Dec 31, 2022 at 10:34 PM Miguel Ojeda
<miguel.ojeda.sandonis@gmail.com> wrote:
>
> On Sat, Dec 31, 2022 at 7:42 AM Masahiro Yamada <masahiroy@kernel.org> wrote:
> >
> > Masahiro Yamada (6):
> > kbuild: specify output names separately for each emission type from
> > rustc
> > fixdep: parse Makefile more correctly to handle comments etc.
> > kbuild: remove sed commands after rustc rules
> > fixdep: refactor hash table lookup
> > fixdep: avoid parsing the same file over again
> > fixdep: do not parse *.so, *.rmeta, *.rlib
>
> These cleanups are great, and it is a pleasure to see proper
> integration with `fixdep` -- thanks a ton! :)
>
> Will you want to take them through the kbuild tree? (I guess so, given
> the bulk of it is on `fixdep`)
Yes, my plan is to get it in the kbuild tree with your ack.
> Cheers,
> Miguel
--
Best Regards
Masahiro Yamada
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [PATCH 0/6] kbuild: fix dep-file processing for rust
2022-12-31 15:05 ` Masahiro Yamada
@ 2023-01-03 20:48 ` Miguel Ojeda
0 siblings, 0 replies; 15+ messages in thread
From: Miguel Ojeda @ 2023-01-03 20:48 UTC (permalink / raw)
To: Masahiro Yamada
Cc: linux-kbuild, linux-kernel, Miguel Ojeda, Alex Gaynor,
Björn Roy Baron, Boqun Feng, Gary Guo, Nathan Chancellor,
Nick Desaulniers, Nicolas Schier, Tom Rix, Wedson Almeida Filho,
llvm, rust-for-linux
On Sat, Dec 31, 2022 at 4:06 PM Masahiro Yamada <masahiroy@kernel.org> wrote:
>
> Yes, my plan is to get it in the kbuild tree with your ack.
Done, also compile- and boot-tested each (on top of -rc1).
Cheers,
Miguel
^ permalink raw reply [flat|nested] 15+ messages in thread