[PATCH v2 0/2] dt-bindings: DTB based validation

devicetree.vger.kernel.org archive mirror
 help / color / mirror / Atom feed

* [PATCH v2 0/2] dt-bindings: DTB based validation
@ 2022-03-10 16:05 Rob Herring
  2022-03-10 16:05 ` [PATCH v2 1/2] dt-bindings: kbuild: Pass DT_SCHEMA_FILES to dt-validate Rob Herring
  2022-03-10 16:05 ` [PATCH v2 2/2] dt-bindings: kbuild: Use DTB files for validation Rob Herring
  0 siblings, 2 replies; 4+ messages in thread
From: Rob Herring @ 2022-03-10 16:05 UTC (permalink / raw)
  To: Masahiro Yamada, Krzysztof Kozlowski
  Cc: devicetree, linux-kernel, Geert Uytterhoeven, Laurent Pinchart,
	Maxime Ripard

This series switches the kernel's DT schema validation from YAML encoded
DT files to using DTB files directly. See patch 2 for the full reasoning
of why. The diffstat also shows this is a nice simplification (at least
from the kernel side). Patch 1 is further reworking of how
DT_SCHEMA_FILES works and builds on [1].

Overall, the build time is about the same (still slow) though we do save
a dtc call for dtbs_check. Extracting the type information is not cached
in any way, so there's some opportunity for a slight optimization there.

Switching to DTB validation found various issues in the bindings and
examples. Patches for those issues have already been posted and applied
over the last month or so.

v2:
 - Drop processed-schema.json dependency when building 'dtbs' 
 - Don't allow override of DT_TMP_SCHEMA

Rob

[1] https://lore.kernel.org/all/20220228201006.1484903-1-robh@kernel.org/

Rob Herring (2):
  dt-bindings: kbuild: Pass DT_SCHEMA_FILES to dt-validate
  dt-bindings: kbuild: Use DTB files for validation

 Documentation/devicetree/bindings/Makefile    | 35 ++++---------------
 .../devicetree/bindings/writing-schema.rst    | 12 -------
 scripts/Makefile.lib                          | 25 +++++--------
 scripts/dtc/Makefile                          | 13 -------
 scripts/dtc/update-dtc-source.sh              |  2 +-
 5 files changed, 16 insertions(+), 71 deletions(-)

-- 
2.32.0

^ permalink raw reply	[flat|nested] 4+ messages in thread

* [PATCH v2 1/2] dt-bindings: kbuild: Pass DT_SCHEMA_FILES to dt-validate
  2022-03-10 16:05 [PATCH v2 0/2] dt-bindings: DTB based validation Rob Herring
@ 2022-03-10 16:05 ` Rob Herring
  2022-03-11 15:30   ` Rob Herring
  2022-03-10 16:05 ` [PATCH v2 2/2] dt-bindings: kbuild: Use DTB files for validation Rob Herring
  1 sibling, 1 reply; 4+ messages in thread
From: Rob Herring @ 2022-03-10 16:05 UTC (permalink / raw)
  To: Masahiro Yamada, Krzysztof Kozlowski
  Cc: devicetree, linux-kernel, Geert Uytterhoeven, Laurent Pinchart,
	Maxime Ripard

In preparation for supporting validation of DTB files, the full
processed schema will always be needed in order to extract type
information from it. Therefore, the processed schema containing only
what DT_SCHEMA_FILES specifies won't work. Instead, dt-validate has
gained an option, -l or --limit, to specify which schema(s) to use for
validation.

As the command line option is new, we the minimum dtschema version must be
updated.

Cc: Masahiro Yamada <masahiroy@kernel.org>
Signed-off-by: Rob Herring <robh@kernel.org>
---
v2:
 - Change DT_TMP_SCHEMA from '?=' to ':='
---
 Documentation/devicetree/bindings/Makefile | 28 +++-------------------
 scripts/Makefile.lib                       |  5 ++--
 2 files changed, 5 insertions(+), 28 deletions(-)

diff --git a/Documentation/devicetree/bindings/Makefile b/Documentation/devicetree/bindings/Makefile
index 29b5dbd0807e..2716f7b9e25e 100644
--- a/Documentation/devicetree/bindings/Makefile
+++ b/Documentation/devicetree/bindings/Makefile
@@ -6,7 +6,7 @@ DT_MK_SCHEMA ?= dt-mk-schema
 DT_SCHEMA_LINT := $(shell which yamllint || \
   echo "warning: python package 'yamllint' not installed, skipping" >&2)
 
-DT_SCHEMA_MIN_VERSION = 2021.2.1
+DT_SCHEMA_MIN_VERSION = 2022.3
 
 PHONY += check_dtschema_version
 check_dtschema_version:
@@ -25,9 +25,6 @@ quiet_cmd_extract_ex = DTEX    $@
 $(obj)/%.example.dts: $(src)/%.yaml check_dtschema_version FORCE
 	$(call if_changed,extract_ex)
 
-# Use full schemas when checking %.example.dts
-DT_TMP_SCHEMA := $(obj)/processed-schema-examples.json
-
 find_all_cmd = find $(srctree)/$(src) \( -name '*.yaml' ! \
 		-name 'processed-schema*' ! \
 		-name '*.example.dt.yaml' \)
@@ -70,29 +67,10 @@ override DTC_FLAGS := \
 # Disable undocumented compatible checks until warning free
 override DT_CHECKER_FLAGS ?=
 
-$(obj)/processed-schema-examples.json: $(DT_DOCS) $(src)/.yamllint check_dtschema_version FORCE
+$(obj)/processed-schema.json: $(DT_DOCS) $(src)/.yamllint check_dtschema_version FORCE
 	$(call if_changed_rule,chkdt)
 
-ifeq ($(DT_SCHEMA_FILES),)
-
-# Unless DT_SCHEMA_FILES is specified, use the full schema for dtbs_check too.
-# Just copy processed-schema-examples.json
-
-$(obj)/processed-schema.json: $(obj)/processed-schema-examples.json FORCE
-	$(call if_changed,copy)
-
-else
-
-# If DT_SCHEMA_FILES is specified, use it for processed-schema.json
-
-$(obj)/processed-schema.json: DT_MK_SCHEMA_FLAGS := -u
-$(obj)/processed-schema.json: $(CHK_DT_DOCS) check_dtschema_version FORCE
-	$(call if_changed,mk_schema)
-
-endif
-
-always-$(CHECK_DT_BINDING) += processed-schema-examples.json
-always-$(CHECK_DTBS)       += processed-schema.json
+always-y += processed-schema.json
 always-$(CHECK_DT_BINDING) += $(patsubst $(srctree)/$(src)/%.yaml,%.example.dts, $(CHK_DT_DOCS))
 always-$(CHECK_DT_BINDING) += $(patsubst $(srctree)/$(src)/%.yaml,%.example.dt.yaml, $(CHK_DT_DOCS))
 
diff --git a/scripts/Makefile.lib b/scripts/Makefile.lib
index 79be57fdd32a..c8c6f1745d03 100644
--- a/scripts/Makefile.lib
+++ b/scripts/Makefile.lib
@@ -361,10 +361,9 @@ $(multi-dtb-y): FORCE
 $(call multi_depend, $(multi-dtb-y), .dtb, -dtbs)
 
 DT_CHECKER ?= dt-validate
-DT_CHECKER_FLAGS ?= $(if $(DT_SCHEMA_FILES),,-m)
+DT_CHECKER_FLAGS ?= $(if $(DT_SCHEMA_FILES),-l $(DT_SCHEMA_FILES),-m)
 DT_BINDING_DIR := Documentation/devicetree/bindings
-# DT_TMP_SCHEMA may be overridden from Documentation/devicetree/bindings/Makefile
-DT_TMP_SCHEMA ?= $(objtree)/$(DT_BINDING_DIR)/processed-schema.json
+DT_TMP_SCHEMA := $(objtree)/$(DT_BINDING_DIR)/processed-schema.json
 
 quiet_cmd_dtb_check =	CHECK   $@
       cmd_dtb_check =	$(DT_CHECKER) $(DT_CHECKER_FLAGS) -u $(srctree)/$(DT_BINDING_DIR) -p $(DT_TMP_SCHEMA) $@
-- 
2.32.0


^ permalink raw reply related	[flat|nested] 4+ messages in thread

* Re: [PATCH v2 1/2] dt-bindings: kbuild: Pass DT_SCHEMA_FILES to dt-validate
  2022-03-10 16:05 ` [PATCH v2 1/2] dt-bindings: kbuild: Pass DT_SCHEMA_FILES to dt-validate Rob Herring
@ 2022-03-11 15:30   ` Rob Herring
  0 siblings, 0 replies; 4+ messages in thread
From: Rob Herring @ 2022-03-11 15:30 UTC (permalink / raw)
  To: Rob Herring
  Cc: devicetree, Krzysztof Kozlowski, Geert Uytterhoeven,
	Laurent Pinchart, Maxime Ripard, Masahiro Yamada, linux-kernel

On Thu, 10 Mar 2022 10:05:12 -0600, Rob Herring wrote:
> In preparation for supporting validation of DTB files, the full
> processed schema will always be needed in order to extract type
> information from it. Therefore, the processed schema containing only
> what DT_SCHEMA_FILES specifies won't work. Instead, dt-validate has
> gained an option, -l or --limit, to specify which schema(s) to use for
> validation.
> 
> As the command line option is new, we the minimum dtschema version must be
> updated.
> 
> Cc: Masahiro Yamada <masahiroy@kernel.org>
> Signed-off-by: Rob Herring <robh@kernel.org>
> ---
> v2:
>  - Change DT_TMP_SCHEMA from '?=' to ':='
> ---
>  Documentation/devicetree/bindings/Makefile | 28 +++-------------------
>  scripts/Makefile.lib                       |  5 ++--
>  2 files changed, 5 insertions(+), 28 deletions(-)
> 

Applied, thanks!

^ permalink raw reply	[flat|nested] 4+ messages in thread

* [PATCH v2 2/2] dt-bindings: kbuild: Use DTB files for validation
  2022-03-10 16:05 [PATCH v2 0/2] dt-bindings: DTB based validation Rob Herring
  2022-03-10 16:05 ` [PATCH v2 1/2] dt-bindings: kbuild: Pass DT_SCHEMA_FILES to dt-validate Rob Herring
@ 2022-03-10 16:05 ` Rob Herring
  1 sibling, 0 replies; 4+ messages in thread
From: Rob Herring @ 2022-03-10 16:05 UTC (permalink / raw)
  To: Masahiro Yamada, Krzysztof Kozlowski
  Cc: devicetree, linux-kernel, Geert Uytterhoeven, Laurent Pinchart,
	Maxime Ripard

Switch the DT validation to use DTB files directly instead of a DTS to
YAML conversion.

The original motivation for supporting validation on DTB files was to
enable running validation on a running system (e.g. 'dt-validate
/sys/firmware/fdt') or other cases where the original source DTS is not
available.

The YAML format was not without issues. Using DTBs with the schema type
information solves some of those problems. The YAML format relies on the
DTS source level information including bracketing of properties, size
directives, and phandle tags all of which are lost in a DTB file. While
standardizing the bracketing is a good thing, it does cause a lot of
extra warnings and churn to fix them.

Another issue has been signed types are not validated correctly as sign
information is not propagated to YAML. Using the schema type information
allows for proper handling of signed types. YAML also can't represent
the full range of 64-bit integers as numbers are stored as floats by
most/all parsers.

The DTB validation works by decoding property values using the type
information in the schemas themselves. The main corner case this does
not work for is matrix types where neither dimension is fixed. For
now, checking the dimensions in these cases are skipped.

Signed-off-by: Rob Herring <robh@kernel.org>
---
v2:
 - Dependency on processed-schema.json needs to be conditional
---
 Documentation/devicetree/bindings/Makefile    |  7 +++----
 .../devicetree/bindings/writing-schema.rst    | 12 -----------
 scripts/Makefile.lib                          | 20 +++++++------------
 scripts/dtc/Makefile                          | 13 ------------
 scripts/dtc/update-dtc-source.sh              |  2 +-
 5 files changed, 11 insertions(+), 43 deletions(-)

diff --git a/Documentation/devicetree/bindings/Makefile b/Documentation/devicetree/bindings/Makefile
index 2716f7b9e25e..e594e5898be0 100644
--- a/Documentation/devicetree/bindings/Makefile
+++ b/Documentation/devicetree/bindings/Makefile
@@ -26,8 +26,7 @@ $(obj)/%.example.dts: $(src)/%.yaml check_dtschema_version FORCE
 	$(call if_changed,extract_ex)
 
 find_all_cmd = find $(srctree)/$(src) \( -name '*.yaml' ! \
-		-name 'processed-schema*' ! \
-		-name '*.example.dt.yaml' \)
+		-name 'processed-schema*' \)
 
 find_cmd = $(find_all_cmd) | grep -F "$(DT_SCHEMA_FILES)"
 CHK_DT_DOCS := $(shell $(find_cmd))
@@ -72,9 +71,9 @@ $(obj)/processed-schema.json: $(DT_DOCS) $(src)/.yamllint check_dtschema_version
 
 always-y += processed-schema.json
 always-$(CHECK_DT_BINDING) += $(patsubst $(srctree)/$(src)/%.yaml,%.example.dts, $(CHK_DT_DOCS))
-always-$(CHECK_DT_BINDING) += $(patsubst $(srctree)/$(src)/%.yaml,%.example.dt.yaml, $(CHK_DT_DOCS))
+always-$(CHECK_DT_BINDING) += $(patsubst $(srctree)/$(src)/%.yaml,%.example.dtb, $(CHK_DT_DOCS))
 
 # Hack: avoid 'Argument list too long' error for 'make clean'. Remove most of
 # build artifacts here before they are processed by scripts/Makefile.clean
 clean-files = $(shell find $(obj) \( -name '*.example.dts' -o \
-			-name '*.example.dt.yaml' \) -delete 2>/dev/null)
+			-name '*.example.dtb' \) -delete 2>/dev/null)
diff --git a/Documentation/devicetree/bindings/writing-schema.rst b/Documentation/devicetree/bindings/writing-schema.rst
index 3b00fe981494..b4258bf81be5 100644
--- a/Documentation/devicetree/bindings/writing-schema.rst
+++ b/Documentation/devicetree/bindings/writing-schema.rst
@@ -123,18 +123,6 @@ project can be installed with pip::
 Several executables (dt-doc-validate, dt-mk-schema, dt-validate) will be
 installed. Ensure they are in your PATH (~/.local/bin by default).
 
-dtc must also be built with YAML output support enabled. This requires that
-libyaml and its headers be installed on the host system. For some distributions
-that involves installing the development package, such as:
-
-Debian::
-
-  apt-get install libyaml-dev
-
-Fedora::
-
-  dnf -y install libyaml-devel
-
 Running checks
 ~~~~~~~~~~~~~~
 
diff --git a/scripts/Makefile.lib b/scripts/Makefile.lib
index c8c6f1745d03..9d5320a47ef8 100644
--- a/scripts/Makefile.lib
+++ b/scripts/Makefile.lib
@@ -87,11 +87,6 @@ base-dtb-y := $(foreach m, $(multi-dtb-y), $(firstword $(call suffix-search, $m,
 
 always-y			+= $(dtb-y)
 
-ifneq ($(CHECK_DTBS),)
-always-y += $(patsubst %.dtb,%.dt.yaml, $(real-dtb-y))
-always-y += $(patsubst %.dtbo,%.dt.yaml, $(real-dtb-y))
-endif
-
 # Add subdir path
 
 extra-y		:= $(addprefix $(obj)/,$(extra-y))
@@ -347,12 +342,6 @@ cmd_dtc = $(HOSTCC) -E $(dtc_cpp_flags) -x assembler-with-cpp -o $(dtc-tmp) $< ;
 		-d $(depfile).dtc.tmp $(dtc-tmp) ; \
 	cat $(depfile).pre.tmp $(depfile).dtc.tmp > $(depfile)
 
-$(obj)/%.dtb: $(src)/%.dts $(DTC) FORCE
-	$(call if_changed_dep,dtc)
-
-$(obj)/%.dtbo: $(src)/%.dts $(DTC) FORCE
-	$(call if_changed_dep,dtc)
-
 quiet_cmd_fdtoverlay = DTOVL   $@
       cmd_fdtoverlay = $(objtree)/scripts/dtc/fdtoverlay -o $@ -i $(real-prereqs)
 
@@ -360,22 +349,27 @@ $(multi-dtb-y): FORCE
 	$(call if_changed,fdtoverlay)
 $(call multi_depend, $(multi-dtb-y), .dtb, -dtbs)
 
+ifneq ($(CHECK_DTBS)$(CHECK_DT_BINDING),)
 DT_CHECKER ?= dt-validate
 DT_CHECKER_FLAGS ?= $(if $(DT_SCHEMA_FILES),-l $(DT_SCHEMA_FILES),-m)
 DT_BINDING_DIR := Documentation/devicetree/bindings
 DT_TMP_SCHEMA := $(objtree)/$(DT_BINDING_DIR)/processed-schema.json
 
 quiet_cmd_dtb_check =	CHECK   $@
-      cmd_dtb_check =	$(DT_CHECKER) $(DT_CHECKER_FLAGS) -u $(srctree)/$(DT_BINDING_DIR) -p $(DT_TMP_SCHEMA) $@
+      cmd_dtb_check =	$(DT_CHECKER) $(DT_CHECKER_FLAGS) -u $(srctree)/$(DT_BINDING_DIR) -p $(DT_TMP_SCHEMA) $@ || true
+endif
 
 define rule_dtc
 	$(call cmd_and_fixdep,dtc)
 	$(call cmd,dtb_check)
 endef
 
-$(obj)/%.dt.yaml: $(src)/%.dts $(DTC) $(DT_TMP_SCHEMA) FORCE
+$(obj)/%.dtb: $(src)/%.dts $(DTC) $(DT_TMP_SCHEMA) FORCE
 	$(call if_changed_rule,dtc)
 
+$(obj)/%.dtbo: $(src)/%.dts $(DTC) FORCE
+	$(call if_changed_dep,dtc)
+
 dtc-tmp = $(subst $(comma),_,$(dot-target).dts.tmp)
 
 # Bzip2
diff --git a/scripts/dtc/Makefile b/scripts/dtc/Makefile
index 1cba78e1dce6..4d32b9497da9 100644
--- a/scripts/dtc/Makefile
+++ b/scripts/dtc/Makefile
@@ -17,20 +17,7 @@ fdtoverlay-objs	:= $(libfdt) fdtoverlay.o util.o
 
 # Source files need to get at the userspace version of libfdt_env.h to compile
 HOST_EXTRACFLAGS += -I $(srctree)/$(src)/libfdt
-
-ifeq ($(shell pkg-config --exists yaml-0.1 2>/dev/null && echo yes),)
-ifneq ($(CHECK_DT_BINDING)$(CHECK_DTBS),)
-$(error dtc needs libyaml for DT schema validation support. \
-	Install the necessary libyaml development package.)
-endif
 HOST_EXTRACFLAGS += -DNO_YAML
-else
-dtc-objs	+= yamltree.o
-# To include <yaml.h> installed in a non-default path
-HOSTCFLAGS_yamltree.o := $(shell pkg-config --cflags yaml-0.1)
-# To link libyaml installed in a non-default path
-HOSTLDLIBS_dtc	:= $(shell pkg-config --libs yaml-0.1)
-endif
 
 # Generated files need one more search path to include headers in source tree
 HOSTCFLAGS_dtc-lexer.lex.o := -I $(srctree)/$(src)
diff --git a/scripts/dtc/update-dtc-source.sh b/scripts/dtc/update-dtc-source.sh
index 32ff17ffd089..94627541533e 100755
--- a/scripts/dtc/update-dtc-source.sh
+++ b/scripts/dtc/update-dtc-source.sh
@@ -32,7 +32,7 @@ DTC_UPSTREAM_PATH=`pwd`/../dtc
 DTC_LINUX_PATH=`pwd`/scripts/dtc
 
 DTC_SOURCE="checks.c data.c dtc.c dtc.h flattree.c fstree.c livetree.c srcpos.c \
-		srcpos.h treesource.c util.c util.h version_gen.h yamltree.c \
+		srcpos.h treesource.c util.c util.h version_gen.h \
 		dtc-lexer.l dtc-parser.y"
 LIBFDT_SOURCE="fdt.c fdt.h fdt_addresses.c fdt_empty_tree.c \
 		fdt_overlay.c fdt_ro.c fdt_rw.c fdt_strerror.c fdt_sw.c \
-- 
2.32.0


^ permalink raw reply related	[flat|nested] 4+ messages in thread

end of thread, other threads:[~2022-03-11 15:30 UTC | newest]

Thread overview: 4+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2022-03-10 16:05 [PATCH v2 0/2] dt-bindings: DTB based validation Rob Herring
2022-03-10 16:05 ` [PATCH v2 1/2] dt-bindings: kbuild: Pass DT_SCHEMA_FILES to dt-validate Rob Herring
2022-03-11 15:30   ` Rob Herring
2022-03-10 16:05 ` [PATCH v2 2/2] dt-bindings: kbuild: Use DTB files for validation Rob Herring

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).