[PATCH V6 1/2] kbuild: centralize .dts->.dtb rule

[PATCH V6 1/2] kbuild: centralize .dts->.dtb rule

[Stephen Warren]

From: Stephen Warren <swarren-DDmLM1+adcrQT0dZR+AlfA@public.gmane.org>

All architectures that use cmd_dtc do so in the same way. Move the build
rule to a central location to avoid duplication.

Update Documentation/kbuild to remove the explicit call to cmd_dtc from
the example, now that the rule exists in a centralized location, and in
fact replace the example with a couple that still exist.

arch/mips changes courtesy of Ralf Baechle.

Signed-off-by: Ralf Baechle <ralf-6z/3iImG2C8G8FEW9MqTrA@public.gmane.org>
Signed-off-by: Stephen Warren <swarren-DDmLM1+adcrQT0dZR+AlfA@public.gmane.org>
Acked-by: Sam Ravnborg <sam-uyr5N9Q2VtJg9hUCZPvPmw@public.gmane.org>
---
v6: Added arch/{arm64,microblaze,mips} updates.
v5: Updated Documentation/kbuild.
v4: No change.
v3: No change.
v2: New patch.
---
 Documentation/kbuild/makefiles.txt |   21 +++++++++++++++------
 arch/arm/boot/Makefile             |    4 ----
 arch/arm64/boot/Makefile           |    3 ---
 arch/c6x/boot/Makefile             |    3 ---
 arch/microblaze/boot/Makefile      |    3 ---
 arch/mips/cavium-octeon/Makefile   |    3 ---
 arch/mips/lantiq/dts/Makefile      |    3 ---
 arch/mips/netlogic/dts/Makefile    |    3 ---
 arch/openrisc/boot/Makefile        |    3 ---
 arch/powerpc/boot/Makefile         |    4 ----
 scripts/Makefile.lib               |    3 +++
 11 files changed, 18 insertions(+), 35 deletions(-)

diff --git a/Documentation/kbuild/makefiles.txt b/Documentation/kbuild/makefiles.txt
index ec9ae67..a0b0671 100644
--- a/Documentation/kbuild/makefiles.txt
+++ b/Documentation/kbuild/makefiles.txt
@@ -1175,15 +1175,24 @@ When kbuild executes, the following steps are followed (roughly):
 	in an init section in the image. Platform code *must* copy the
 	blob to non-init memory prior to calling unflatten_device_tree().
 
+	To use this command, simply add *.dtb into obj-y or targets, or make
+	some other target depend on %.dtb
+
+	A central rule exists to create $(obj)/%.dtb from $(src)/dts/%.dts;
+	architecture Makefiles do no need to explicitly write out that rule.
+
 	Example:
-		#arch/x86/platform/ce4100/Makefile
-		clean-files := *dtb.S
+		#arch/arm/boot/Makefile
+		targets += $(dtb-y)
+		clean-files := *.dtb
+
+		#arch/powerpc/boot/Makefile
+		DTC_FLAGS       ?= -p 1024
 
-		DTC_FLAGS := -p 1024
-		obj-y += foo.dtb.o
+		$(obj)/dtbImage.%: vmlinux $(wrapperbits) $(obj)/%.dtb
+			$(call if_changed,wrap,$*,,$(obj)/$*.dtb)
 
-		$(obj)/%.dtb: $(src)/%.dts
-			$(call cmd,dtc)
+		clean-files += ... *.dtb
 
 --- 6.8 Custom kbuild commands
 
diff --git a/arch/arm/boot/Makefile b/arch/arm/boot/Makefile
index f2aa09e..208bb4c 100644
--- a/arch/arm/boot/Makefile
+++ b/arch/arm/boot/Makefile
@@ -61,10 +61,6 @@ endif
 
 targets += $(dtb-y)
 
-# Rule to build device tree blobs
-$(obj)/%.dtb: $(src)/dts/%.dts FORCE
-	$(call if_changed_dep,dtc)
-
 $(obj)/dtbs: $(addprefix $(obj)/, $(dtb-y))
 
 clean-files := *.dtb
diff --git a/arch/arm64/boot/Makefile b/arch/arm64/boot/Makefile
index eca209b..20048be 100644
--- a/arch/arm64/boot/Makefile
+++ b/arch/arm64/boot/Makefile
@@ -22,9 +22,6 @@ $(obj)/Image: vmlinux FORCE
 $(obj)/Image.gz: $(obj)/Image FORCE
 	$(call if_changed,gzip)
 
-$(obj)/%.dtb: $(src)/dts/%.dts
-	$(call cmd,dtc)
-
 install: $(obj)/Image
 	$(CONFIG_SHELL) $(srctree)/$(src)/install.sh $(KERNELRELEASE) \
 	$(obj)/Image System.map "$(INSTALL_PATH)"
diff --git a/arch/c6x/boot/Makefile b/arch/c6x/boot/Makefile
index 6891257..ad605fb 100644
--- a/arch/c6x/boot/Makefile
+++ b/arch/c6x/boot/Makefile
@@ -12,9 +12,6 @@ ifneq ($(DTB),)
 obj-y += linked_dtb.o
 endif
 
-$(obj)/%.dtb: $(src)/dts/%.dts FORCE
-	$(call if_changed_dep,dtc)
-
 quiet_cmd_cp = CP      $< $@$2
 	cmd_cp = cat $< >$@$2 || (rm -f $@ && echo false)
 
diff --git a/arch/microblaze/boot/Makefile b/arch/microblaze/boot/Makefile
index fa83ea4..226da18 100644
--- a/arch/microblaze/boot/Makefile
+++ b/arch/microblaze/boot/Makefile
@@ -48,7 +48,4 @@ $(obj)/simpleImage.%: vmlinux FORCE
 # Rule to build device tree blobs
 DTC_FLAGS := -p 1024
 
-$(obj)/%.dtb: $(src)/dts/%.dts FORCE
-	$(call if_changed_dep,dtc)
-
 clean-files += *.dtb simpleImage.*.unstrip linux.bin.ub
diff --git a/arch/mips/cavium-octeon/Makefile b/arch/mips/cavium-octeon/Makefile
index bc96e29..6e927cf 100644
--- a/arch/mips/cavium-octeon/Makefile
+++ b/arch/mips/cavium-octeon/Makefile
@@ -24,9 +24,6 @@ DTB_FILES = $(patsubst %.dts, %.dtb, $(DTS_FILES))
 
 obj-y += $(patsubst %.dts, %.dtb.o, $(DTS_FILES))
 
-$(obj)/%.dtb: $(src)/%.dts FORCE
-	$(call if_changed_dep,dtc)
-
 # Let's keep the .dtb files around in case we want to look at them.
 .SECONDARY:  $(addprefix $(obj)/, $(DTB_FILES))
 
diff --git a/arch/mips/lantiq/dts/Makefile b/arch/mips/lantiq/dts/Makefile
index 674fca4..6fa72dd 100644
--- a/arch/mips/lantiq/dts/Makefile
+++ b/arch/mips/lantiq/dts/Makefile
@@ -1,4 +1 @@
 obj-$(CONFIG_DT_EASY50712) := easy50712.dtb.o
-
-$(obj)/%.dtb: $(obj)/%.dts
-	$(call if_changed,dtc)
diff --git a/arch/mips/netlogic/dts/Makefile b/arch/mips/netlogic/dts/Makefile
index 67ae3fe..d117d46 100644
--- a/arch/mips/netlogic/dts/Makefile
+++ b/arch/mips/netlogic/dts/Makefile
@@ -1,4 +1 @@
 obj-$(CONFIG_DT_XLP_EVP) := xlp_evp.dtb.o
-
-$(obj)/%.dtb: $(obj)/%.dts
-	$(call if_changed,dtc)
diff --git a/arch/openrisc/boot/Makefile b/arch/openrisc/boot/Makefile
index 0995835..fd329bd 100644
--- a/arch/openrisc/boot/Makefile
+++ b/arch/openrisc/boot/Makefile
@@ -10,6 +10,3 @@ obj-y += $(BUILTIN_DTB)
 clean-files := *.dtb.S
 
 #DTC_FLAGS ?= -p 1024
-
-$(obj)/%.dtb: $(src)/dts/%.dts FORCE
-	$(call if_changed_dep,dtc)
diff --git a/arch/powerpc/boot/Makefile b/arch/powerpc/boot/Makefile
index 6a15c96..90206f2 100644
--- a/arch/powerpc/boot/Makefile
+++ b/arch/powerpc/boot/Makefile
@@ -356,10 +356,6 @@ $(obj)/treeImage.initrd.%: vmlinux $(obj)/%.dtb $(wrapperbits)
 $(obj)/treeImage.%: vmlinux $(obj)/%.dtb $(wrapperbits)
 	$(call if_changed,wrap,treeboot-$*,,$(obj)/$*.dtb)
 
-# Rule to build device tree blobs
-$(obj)/%.dtb: $(src)/dts/%.dts FORCE
-	$(call if_changed_dep,dtc)
-
 # If there isn't a platform selected then just strip the vmlinux.
 ifeq (,$(image-y))
 image-y := vmlinux.strip
diff --git a/scripts/Makefile.lib b/scripts/Makefile.lib
index 0be6f11..425578e 100644
--- a/scripts/Makefile.lib
+++ b/scripts/Makefile.lib
@@ -266,6 +266,9 @@ $(obj)/%.dtb.S: $(obj)/%.dtb
 quiet_cmd_dtc = DTC     $@
 cmd_dtc = $(objtree)/scripts/dtc/dtc -O dtb -o $@ -b 0 $(DTC_FLAGS) -d $(depfile) $<
 
+$(obj)/%.dtb: $(src)/dts/%.dts FORCE
+	$(call if_changed_dep,dtc)
+
 # Bzip2
 # ---------------------------------------------------------------------------
 
-- 
1.7.0.4

[PATCH V6 2/2] kbuild: run the pre-processor on *.dts files

[Stephen Warren]

From: Stephen Warren <swarren@nvidia.com>

Modify cmd_dtc to run the C pre-processor on the input .dts file before
passing it to dtc for final compilation. This allows the use of #define
and #include within the .dts file.

Signed-off-by: Stephen Warren <swarren@nvidia.com>
Acked-by: Sam Ravnborg <sam@ravnborg.org>
---
v6: No change.
v5:
* Update Documentation/kbuild for the new command and rule.
v4:
* Use -x assembler-with-cpp so pre-defined macros are set up so that
  #included header files know to only use cpp syntax, not C syntax.
* Define __DTS__ for similar reasons.
* use $(CPP) not $(CC) -E, and use $(cpp_flags).
* Save the pre-processed results so they can be easily inspected when
  debugging build issues.
* The use of -x assembler-with-cpp causes cpp to recognize directives in
  column 1 only. Hence, there's no need to escape property names that
  begin with #. Hence, there's no need for separate skeleton.dtsi and
  skeleton.dtsip. Maintain a separate file extension and build rule so that
  CPP-usage is opt-in. In particular, when using CPP, #include must be used
  rather than /include/ so that dependencies work.
v3: Pass "-x c" not "-xc" to cpp.
v2: Place make %.dtb: %.dtsp rule into Makefile.lib.
---
 Documentation/kbuild/makefiles.txt |   23 +++++++++++++++++++++++
 scripts/Makefile.lib               |    9 +++++++++
 2 files changed, 32 insertions(+), 0 deletions(-)

diff --git a/Documentation/kbuild/makefiles.txt b/Documentation/kbuild/makefiles.txt
index a0b0671..dfe6838 100644
--- a/Documentation/kbuild/makefiles.txt
+++ b/Documentation/kbuild/makefiles.txt
@@ -1194,6 +1194,29 @@ When kbuild executes, the following steps are followed (roughly):
 
 		clean-files += ... *.dtb
 
+    dtc_cpp
+	This is just like dtc as describe above, except that the C pre-
+	processor is invoked upon the .dtsp file before compiling the result
+	with dtc.
+
+	In order for build dependencies to work, all files compiled using
+	dtc_cpp must use the C pre-processor's #include functionality and not
+	dtc's /include/ functionality.
+
+	Using the C pre-processor allows use of #define to create named
+	constants. In turn, the #defines will typically appear in a header
+	file, which may be shared with regular C code. Since the dtc language
+	represents a data structure rather than code in C syntax, similar
+	restrictions are placed on a header file included by a device tree
+	file as for a header file included by an assembly language file.
+	In particular, the C pre-processor is passed -x assembler-with-cpp,
+	which sets macro __ASSEMBLY__. __DTS__ is also set. These allow header
+	files to restrict their content to that compatible with device tree
+	source.
+
+	A central rule exists to create $(obj)/%.dtb from $(src)/dts/%.dtsp;
+	architecture Makefiles do no need to explicitly write out that rule.
+
 --- 6.8 Custom kbuild commands
 
 	When kbuild is executing with KBUILD_VERBOSE=0, then only a shorthand
diff --git a/scripts/Makefile.lib b/scripts/Makefile.lib
index 425578e..33432f4 100644
--- a/scripts/Makefile.lib
+++ b/scripts/Makefile.lib
@@ -269,6 +269,15 @@ cmd_dtc = $(objtree)/scripts/dtc/dtc -O dtb -o $@ -b 0 $(DTC_FLAGS) -d $(depfile
 $(obj)/%.dtb: $(src)/dts/%.dts FORCE
 	$(call if_changed_dep,dtc)
 
+dtc-tmp = $(subst $(comma),_,$(dot-target).dts)
+
+quiet_cmd_dtc_cpp = DTC+CPP $@
+cmd_dtc_cpp = $(CPP) $(cpp_flags) -D__DTS__ -x assembler-with-cpp -o $(dtc-tmp) $< ; \
+	$(objtree)/scripts/dtc/dtc -O dtb -o $@ -b 0 $(DTC_FLAGS) $(dtc-tmp)
+
+$(obj)/%.dtb: $(src)/dts/%.dtsp FORCE
+	$(call if_changed_dep,dtc_cpp)
+
 # Bzip2
 # ---------------------------------------------------------------------------
 
-- 
1.7.0.4

Re: [PATCH V6 2/2] kbuild: run the pre-processor on *.dts files

[Grant Likely]

On Mon, 12 Nov 2012 13:58:27 -0700, Stephen Warren <swarren@wwwdotorg.org> wrote:
> From: Stephen Warren <swarren@nvidia.com>
> 
> Modify cmd_dtc to run the C pre-processor on the input .dts file before
> passing it to dtc for final compilation. This allows the use of #define
> and #include within the .dts file.
> 
> Signed-off-by: Stephen Warren <swarren@nvidia.com>
> Acked-by: Sam Ravnborg <sam@ravnborg.org>
> ---
> v6: No change.
> v5:
> * Update Documentation/kbuild for the new command and rule.
> v4:
> * Use -x assembler-with-cpp so pre-defined macros are set up so that
>   #included header files know to only use cpp syntax, not C syntax.
> * Define __DTS__ for similar reasons.
> * use $(CPP) not $(CC) -E, and use $(cpp_flags).
> * Save the pre-processed results so they can be easily inspected when
>   debugging build issues.
> * The use of -x assembler-with-cpp causes cpp to recognize directives in
>   column 1 only. Hence, there's no need to escape property names that
>   begin with #. Hence, there's no need for separate skeleton.dtsi and
>   skeleton.dtsip. Maintain a separate file extension and build rule so that
>   CPP-usage is opt-in. In particular, when using CPP, #include must be used
>   rather than /include/ so that dependencies work.
> v3: Pass "-x c" not "-xc" to cpp.
> v2: Place make %.dtb: %.dtsp rule into Makefile.lib.

Ugh, I'm definitely nervous about turning this on because of the long
term implications so I'm going to punt on actually making a decision
about it for the momemnt. However, I do have a question...

> ---
>  Documentation/kbuild/makefiles.txt |   23 +++++++++++++++++++++++
>  scripts/Makefile.lib               |    9 +++++++++
>  2 files changed, 32 insertions(+), 0 deletions(-)
> 
> diff --git a/Documentation/kbuild/makefiles.txt b/Documentation/kbuild/makefiles.txt
> index a0b0671..dfe6838 100644
> --- a/Documentation/kbuild/makefiles.txt
> +++ b/Documentation/kbuild/makefiles.txt
> @@ -1194,6 +1194,29 @@ When kbuild executes, the following steps are followed (roughly):
>  
>  		clean-files += ... *.dtb
>  
> +    dtc_cpp
> +	This is just like dtc as describe above, except that the C pre-
> +	processor is invoked upon the .dtsp file before compiling the result
> +	with dtc.
> +
> +	In order for build dependencies to work, all files compiled using
> +	dtc_cpp must use the C pre-processor's #include functionality and not
> +	dtc's /include/ functionality.
> +
> +	Using the C pre-processor allows use of #define to create named
> +	constants. In turn, the #defines will typically appear in a header
> +	file, which may be shared with regular C code. Since the dtc language
> +	represents a data structure rather than code in C syntax, similar
> +	restrictions are placed on a header file included by a device tree
> +	file as for a header file included by an assembly language file.
> +	In particular, the C pre-processor is passed -x assembler-with-cpp,
> +	which sets macro __ASSEMBLY__. __DTS__ is also set. These allow header
> +	files to restrict their content to that compatible with device tree
> +	source.
> +
> +	A central rule exists to create $(obj)/%.dtb from $(src)/dts/%.dtsp;
> +	architecture Makefiles do no need to explicitly write out that rule.
> +
>  --- 6.8 Custom kbuild commands
>  
>  	When kbuild is executing with KBUILD_VERBOSE=0, then only a shorthand
> diff --git a/scripts/Makefile.lib b/scripts/Makefile.lib
> index 425578e..33432f4 100644
> --- a/scripts/Makefile.lib
> +++ b/scripts/Makefile.lib
> @@ -269,6 +269,15 @@ cmd_dtc = $(objtree)/scripts/dtc/dtc -O dtb -o $@ -b 0 $(DTC_FLAGS) -d $(depfile
>  $(obj)/%.dtb: $(src)/dts/%.dts FORCE
>  	$(call if_changed_dep,dtc)
>  
> +dtc-tmp = $(subst $(comma),_,$(dot-target).dts)
> +
> +quiet_cmd_dtc_cpp = DTC+CPP $@
> +cmd_dtc_cpp = $(CPP) $(cpp_flags) -D__DTS__ -x assembler-with-cpp -o $(dtc-tmp) $< ; \
> +	$(objtree)/scripts/dtc/dtc -O dtb -o $@ -b 0 $(DTC_FLAGS) $(dtc-tmp)
> +
> +$(obj)/%.dtb: $(src)/dts/%.dtsp FORCE
> +	$(call if_changed_dep,dtc_cpp)

Why this instead of a "%.dts: %.dtsp" rule? Then the exact same dtc rule
gets used in both cases.

The .dtb rule is probably already rather oddball in that it puts the
output one directory below the input. That probably should be cleaned
up.

g.

Re: [PATCH V6 2/2] kbuild: run the pre-processor on *.dts files

[Stephen Warren]

On 11/14/2012 12:25 PM, Grant Likely wrote:
> On Mon, 12 Nov 2012 13:58:27 -0700, Stephen Warren <swarren@wwwdotorg.org> wrote:
>> From: Stephen Warren <swarren@nvidia.com>
>>
>> Modify cmd_dtc to run the C pre-processor on the input .dts file before
>> passing it to dtc for final compilation. This allows the use of #define
>> and #include within the .dts file.

> Ugh, I'm definitely nervous about turning this on because of the long
> term implications so I'm going to punt on actually making a decision
> about it for the momemnt. However, I do have a question...

Hmmm. So how do we get to a decision on this then?

>> diff --git a/scripts/Makefile.lib b/scripts/Makefile.lib
>> index 425578e..33432f4 100644
>> --- a/scripts/Makefile.lib
>> +++ b/scripts/Makefile.lib
>> @@ -269,6 +269,15 @@ cmd_dtc = $(objtree)/scripts/dtc/dtc -O dtb -o $@ -b 0 $(DTC_FLAGS) -d $(depfile
>>  $(obj)/%.dtb: $(src)/dts/%.dts FORCE
>>  	$(call if_changed_dep,dtc)
>>  
>> +dtc-tmp = $(subst $(comma),_,$(dot-target).dts)
>> +
>> +quiet_cmd_dtc_cpp = DTC+CPP $@
>> +cmd_dtc_cpp = $(CPP) $(cpp_flags) -D__DTS__ -x assembler-with-cpp -o $(dtc-tmp) $< ; \
>> +	$(objtree)/scripts/dtc/dtc -O dtb -o $@ -b 0 $(DTC_FLAGS) $(dtc-tmp)
>> +
>> +$(obj)/%.dtb: $(src)/dts/%.dtsp FORCE
>> +	$(call if_changed_dep,dtc_cpp)
> 
> Why this instead of a "%.dts: %.dtsp" rule? Then the exact same dtc rule
> gets used in both cases.
> 
> The .dtb rule is probably already rather oddball in that it puts the
> output one directory below the input. That probably should be cleaned
> up.

I was a little nervous about potentially having some *.dts be primary
source files, and others be derived from *.dtsp (assuming that we
convert platforms to this new feature as desired, rather than en-mass).
It seems like it'd just cause confusion; people might just end up
accidentally editing a *.dts that was generated without realizing it.

Re: [PATCH V6 2/2] kbuild: run the pre-processor on *.dts files

[Grant Likely]

On Wed, 14 Nov 2012 13:18:24 -0700, Stephen Warren <swarren@wwwdotorg.org> wrote:
> On 11/14/2012 12:25 PM, Grant Likely wrote:
> > On Mon, 12 Nov 2012 13:58:27 -0700, Stephen Warren <swarren@wwwdotorg.org> wrote:
> >> From: Stephen Warren <swarren@nvidia.com>
> >>
> >> Modify cmd_dtc to run the C pre-processor on the input .dts file before
> >> passing it to dtc for final compilation. This allows the use of #define
> >> and #include within the .dts file.
> 
> > Ugh, I'm definitely nervous about turning this on because of the long
> > term implications so I'm going to punt on actually making a decision
> > about it for the momemnt. However, I do have a question...
> 
> Hmmm. So how do we get to a decision on this then?

I'm punting for the evening just because I'm mentally exhausted. I'll
make a decision in the next couple of days.

g.

Re: [PATCH V6 1/2] kbuild: centralize .dts->.dtb rule

[Grant Likely]

On Mon, 12 Nov 2012 13:58:26 -0700, Stephen Warren <swarren@wwwdotorg.org> wrote:
> From: Stephen Warren <swarren@nvidia.com>
> 
> All architectures that use cmd_dtc do so in the same way. Move the build
> rule to a central location to avoid duplication.
> 
> Update Documentation/kbuild to remove the explicit call to cmd_dtc from
> the example, now that the rule exists in a centralized location, and in
> fact replace the example with a couple that still exist.
> 
> arch/mips changes courtesy of Ralf Baechle.
> 
> Signed-off-by: Ralf Baechle <ralf@linux-mips.org>
> Signed-off-by: Stephen Warren <swarren@nvidia.com>
> Acked-by: Sam Ravnborg <sam@ravnborg.org>
> ---
> v6: Added arch/{arm64,microblaze,mips} updates.
> v5: Updated Documentation/kbuild.
> v4: No change.
> v3: No change.
> v2: New patch.

Merged, thanks.

g.