From: David Gibson <david@gibson.dropbear.id.au>
To: Doug Anderson <dianders@chromium.org>
Cc: Devicetree-compiler@vger.kernel.org,
"Rob Herring" <robh@kernel.org>,
"Krzysztof Kozlowski" <krzk+dt@kernel.org>,
"Conor Dooley" <conor+dt@kernel.org>,
"Konrad Dybcio" <konradybcio@kernel.org>,
"Peter Griffin" <peter.griffin@linaro.org>,
"Saravana Kannan" <saravanak@kernel.org>,
Devicetree@vger.kernel.org, "Simon Glass" <sjg@chromium.org>,
"Chen-Yu Tsai" <wenst@chromium.org>,
"Julius Werner" <jwerner@chromium.org>,
"Brian Norris" <briannorris@chromium.org>,
linux-kernel@vger.kernel.org,
"Tudor Ambarus" <tudor.ambarus@linaro.org>,
"André Draszik" <andre.draszik@linaro.org>
Subject: Re: [PATCH v3 12/15] fdtfactor: Add tool to split DTBs into a base DTB and overlays
Date: Mon, 16 Nov 2026 14:35:54 +1100 [thread overview]
Message-ID: <avp6GjL1jqEfEFkR@gractus.seuss> (raw)
In-Reply-To: <CAD=FV=Uy0dxbzoMN6hA7FY3R6hyqcL2j8d2-YeivS=_+dA7NfQ@mail.gmail.com>
[-- Attachment #1: Type: text/plain, Size: 6321 bytes --]
On Thu, Nov 12, 2026 at 09:22:38PM -0800, Doug Anderson wrote:
> Hi David,
>
> On Tue, Nov 10, 2026 at 8:16 PM David Gibson
> <david@gibson.dropbear.id.au> wrote:
> >
> > On Sun, Nov 01, 2026 at 02:01:19PM -0800, Douglas Anderson wrote:
> > > diff --git a/README.md b/README.md
> > > index fcfc2e55d804..61fefc4424fe 100644
> > > --- a/README.md
> > > +++ b/README.md
> > > @@ -112,6 +112,49 @@ $ meson setup -Dpython=disabled build
> > > More work remains to support all of libfdt, including access to numeric
> > > values.
> > >
> > > +## fdtfactor
> >
> > Pre-existing, but it would be nice to have blurbs for dtdiff,
> > fdt{get,put} and the other anciliary tools we already include.
>
> Agreed -- in my draft v4 I've added a preparatory patch adding brief
> `README.md` sections for `dtc` and the ancillary tools (`dtdiff`,
> `fdtget`, `fdtput`, `fdtdump`, `convert-dtsv0`, and `fdtoverlay`).
Terrific, thanks.
> > > + self.filename = filename
> > > + with open(filename, 'rb') as f:
> > > + raw_bytes = f.read()
> > > + self.raw_bytes = raw_bytes
> >
> > AFAICT, self.raw_bytes is never used again (unlike the raw_bytes
> > local).
>
> `_print_dtb_stats()` actually used `len(d.raw_bytes)` further down to
Ok, in a later patch. Better to add the field at that point in the
series, then.
> compute `orig_uncomp` when `--stats` is passed, though nothing ever
> reads the buffer contents again after `__init__()`. In my draft v4 I've
> replaced `self.raw_bytes` with `self.uncompressed_size = len(raw_bytes)`
> (pairing with `self.compressed_size` in the compression patch) so we
> don't keep all the uncompressed DTB buffers in memory just for their
> lengths.
Or that.
> > > + if 'phandle' in node.props:
> > > + node.phandle = node.get_u32('phandle')
> > > + del node.props['phandle']
> > > +
> > > + if node.phandle is not None:
> >
> > Changing this to:
> > if node.phandle is None:
> > continue
> > reduces nesting and IMO, would make this a little more readable.
>
> Done in my draft v4 (and did the same for a couple of other `for` loops
> in `fdtfactor`).
Ok.
> > > + offsets = sorted(set(lf_node.get_u32_list(prop_name)))
> >
> > I don't think there's any reason to convert the list to a set before sorting.
>
> Done in my draft v4 (`offsets = sorted(lf_prop.as_uint32_list())`).
> Neither `dtc` nor `libfdt` emits duplicate offsets in
> `/__local_fixups__`, and dropping `set()` also means any duplicate
> offset in a malformed blob will be caught by the overlapping-offset
> check right below.
Sounds good.
> > > + base_root = FactoredTree()
> > > + next_ph = 1
> > > + for path, _ in dtbs[0].root.walk():
> > > + if path not in common_paths:
> > > + continue
> >
> > I think this makes the loop equivalent to:
> > for path in common_paths:
> > ....
> >
> > > + if path and base_root.get_node(path.rpartition('/')[0]) is None:
> > > + continue
>
> We can't iterate `common_paths` directly here because `common_paths` is
> an unordered `set`:
> 1. The `base_root.get_node(path.rpartition('/')[0]) is None` check right
> below relies on parents always being visited before their children so
> that if an ancestor was excluded, its descendants are skipped too.
> Iterating a `set` can visit `/a/b` before `/a`, causing `/a/b` to be
> wrongly skipped.
> 2. Walking `dtbs[0].root.walk()` preserves `dtbs[0]`'s subnode ordering
> in `base_root` and keeps sequential `next_ph` assignment
> deterministic across `PYTHONHASHSEED` values.
Ah, right.
> I've added a comment above the loop in my draft v4 explaining why it
> iterates `dtbs[0]`'s nodes in pre-order instead of iterating
> `common_paths`.
Thanks.
> > > + has_phandle = any(
> > > + d.root.get_node(path).phandle is not None for d in dtbs
> > > + )
> >
> > I expect this will work in practice, but arguably the condition you
> > really need here is whether the node is referred to by any fixup in
> > any non-common property.
> >
> > > + if has_phandle and path and path not in common_sym_targets:
> > > + continue
> > > + base_node = base_root.create_path(path)
> > > + if has_phandle:
> > > + base_node.phandle = next_ph
> > > + next_ph += 1
>
> For the `common_sym_targets` exclusion (`if has_phandle and path and
> path not in common_sym_targets: continue`), that's true in principle
> with respect to `/__fixups__`: an overlay only needs a symbol in
> `base.dtb`'s `/__symbols__` if a non-common property ends up referencing
> `path`. There are a couple of catches if we tried to relax this, though:
>
> 1. Which properties are common vs. non-common isn't known yet here --
> `_build_base_tree()` decides that next, and its candidate eligibility
> depends on which fixup target nodes are in `base_root` (so resolving
> both together would need an iterative pass).
> 2. Even when no non-common property references `path`, `base_node` still
> needs a `phandle` in `base.dtb` if `path` is referenced by a *common*
> property in `base.dtb`, or if `path` has a symbol in `/__symbols__`
> (from `dtc -@`) so the reconstructed DTB retains its `phandle`
> property and remains valid as a base for any further runtime overlays
> targeting that symbol. Conversely, if one board had a `phandle` on
> `path` (e.g. a board-specific label) and another board didn't,
> putting `path` in `base.dtb` with a `phandle` would add a `phandle`
> property to the second board's reconstructed DTB that wasn't in its
> original DTB.
>
> Since in `dtc -@` inputs almost every node with a `phandle` also has a
> label in `/__symbols__`, checking `has_phandle` up front keeps this
> single-pass and exact. I've expanded the docstring on
> `_find_common_nodes_and_phandles()` in my draft v4 to note this.
Ok, understood.
--
David Gibson (he or they) | I'll have my music baroque, and my code
david AT gibson.dropbear.id.au | minimalist, thank you, not the other way
| around.
http://www.ozlabs.org/~dgibson
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
next prev parent reply other threads:[~2026-11-16 4:34 UTC|newest]
Thread overview: 54+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-11-01 22:01 [PATCH v3 00/15] fdtfactor: Introduce fdtfactor to auto-factor overlays Douglas Anderson
2026-11-01 22:01 ` [PATCH v3 01/15] libfdt: overlay: Fix out-of-bounds write on bad fixup offset Douglas Anderson
2026-11-02 4:29 ` David Gibson
2026-11-01 22:01 ` [PATCH v3 02/15] libfdt: overlay: Drop redundant *sep != ':' checks after memchr() Douglas Anderson
2026-11-02 4:30 ` David Gibson
2026-11-01 22:01 ` [PATCH v3 03/15] pylibfdt: Clarify iterator docstrings and test negative offsets Douglas Anderson
2026-11-02 4:32 ` David Gibson
2026-11-01 22:01 ` [PATCH v3 04/15] pylibfdt: Handle empty properties in as_str() and as_stringlist() Douglas Anderson
2026-11-02 4:33 ` David Gibson
2026-11-01 22:01 ` [PATCH v3 05/15] pylibfdt: Add Fdt.set_boot_cpuid_phys() Douglas Anderson
2026-11-02 4:34 ` David Gibson
2026-11-01 22:01 ` [PATCH v3 06/15] pylibfdt: Support copy.copy() and copy.deepcopy() on Property Douglas Anderson
2026-11-02 4:35 ` David Gibson
2026-11-01 22:01 ` [PATCH v3 07/15] devicetree: Add DeviceTree live tree representation Douglas Anderson
2026-11-02 9:31 ` David Gibson
2026-11-04 4:29 ` Doug Anderson
2026-11-04 6:12 ` David Gibson
2026-11-07 2:43 ` Doug Anderson
2026-11-10 2:10 ` David Gibson
2026-11-04 15:13 ` Simon Glass
2026-11-04 15:14 ` Simon Glass
2026-11-07 2:43 ` Doug Anderson
2026-11-11 2:56 ` David Gibson
2026-11-11 20:15 ` Simon Glass
2026-11-12 1:44 ` David Gibson
2026-11-12 19:28 ` Doug Anderson
2026-11-13 3:26 ` David Gibson
2026-11-13 3:30 ` Doug Anderson
2026-11-01 22:01 ` [PATCH v3 08/15] libfdt: overlay: Factor out fixup and fragment path parsing helpers Douglas Anderson
2026-11-04 6:29 ` David Gibson
2026-11-01 22:01 ` [PATCH v3 09/15] libfdt: overlay: Prepare fragment path helper for reuse Douglas Anderson
2026-11-04 15:14 ` Simon Glass
2026-11-01 22:01 ` [PATCH v3 10/15] libfdt: overlay: Make fdto const in overlay_apply_node() Douglas Anderson
2026-11-04 15:14 ` Simon Glass
2026-11-01 22:01 ` [PATCH v3 11/15] libfdt: overlay: Update base /__local_fixups__ on apply Douglas Anderson
2026-11-04 15:14 ` Simon Glass
2026-11-07 2:43 ` Doug Anderson
2026-11-01 22:01 ` [PATCH v3 12/15] fdtfactor: Add tool to split DTBs into a base DTB and overlays Douglas Anderson
2026-11-11 4:16 ` David Gibson
2026-11-13 5:22 ` Doug Anderson
2026-11-16 3:35 ` David Gibson [this message]
2026-11-01 22:01 ` [PATCH v3 13/15] fdtfactor: Add -S/--strip-local-fixups option Douglas Anderson
2026-11-04 15:14 ` Simon Glass
2026-11-11 4:19 ` David Gibson
2026-11-01 22:01 ` [PATCH v3 14/15] fdtfactor: Support compressed DTB inputs and outputs Douglas Anderson
2026-11-12 2:10 ` David Gibson
2026-11-13 5:23 ` Doug Anderson
2026-11-01 22:01 ` [PATCH v3 15/15] fdtfactor: Add compiled FIT factoring support Douglas Anderson
2026-11-12 2:31 ` David Gibson
2026-11-13 5:23 ` Doug Anderson
2026-11-16 3:44 ` David Gibson
2026-11-02 4:44 ` [PATCH v3 00/15] fdtfactor: Introduce fdtfactor to auto-factor overlays David Gibson
2026-11-04 15:13 ` Simon Glass
2026-11-07 2:42 ` Doug Anderson
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=avp6GjL1jqEfEFkR@gractus.seuss \
--to=david@gibson.dropbear.id.au \
--cc=Devicetree-compiler@vger.kernel.org \
--cc=Devicetree@vger.kernel.org \
--cc=andre.draszik@linaro.org \
--cc=briannorris@chromium.org \
--cc=conor+dt@kernel.org \
--cc=dianders@chromium.org \
--cc=jwerner@chromium.org \
--cc=konradybcio@kernel.org \
--cc=krzk+dt@kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=peter.griffin@linaro.org \
--cc=robh@kernel.org \
--cc=saravanak@kernel.org \
--cc=sjg@chromium.org \
--cc=tudor.ambarus@linaro.org \
--cc=wenst@chromium.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox