Devicetree
 help / color / mirror / Atom feed
From: David Gibson <david@gibson.dropbear.id.au>
To: Doug Anderson <dianders@chromium.org>
Cc: Devicetree-compiler@vger.kernel.org,
	"Rob Herring" <robh@kernel.org>,
	"Krzysztof Kozlowski" <krzk+dt@kernel.org>,
	"Conor Dooley" <conor+dt@kernel.org>,
	"Konrad Dybcio" <konradybcio@kernel.org>,
	"Peter Griffin" <peter.griffin@linaro.org>,
	"Saravana Kannan" <saravanak@kernel.org>,
	Devicetree@vger.kernel.org, "Simon Glass" <sjg@chromium.org>,
	"Chen-Yu Tsai" <wenst@chromium.org>,
	"Julius Werner" <jwerner@chromium.org>,
	"Brian Norris" <briannorris@chromium.org>,
	linux-kernel@vger.kernel.org,
	"Tudor Ambarus" <tudor.ambarus@linaro.org>,
	"André Draszik" <andre.draszik@linaro.org>
Subject: Re: [PATCH v3 12/15] fdtfactor: Add tool to split DTBs into a base DTB and overlays
Date: Mon, 16 Nov 2026 14:35:54 +1100	[thread overview]
Message-ID: <avp6GjL1jqEfEFkR@gractus.seuss> (raw)
In-Reply-To: <CAD=FV=Uy0dxbzoMN6hA7FY3R6hyqcL2j8d2-YeivS=_+dA7NfQ@mail.gmail.com>

[-- Attachment #1: Type: text/plain, Size: 6321 bytes --]

On Thu, Nov 12, 2026 at 09:22:38PM -0800, Doug Anderson wrote:
> Hi David,
> 
> On Tue, Nov 10, 2026 at 8:16 PM David Gibson
> <david@gibson.dropbear.id.au> wrote:
> >
> > On Sun, Nov 01, 2026 at 02:01:19PM -0800, Douglas Anderson wrote:
> > > diff --git a/README.md b/README.md
> > > index fcfc2e55d804..61fefc4424fe 100644
> > > --- a/README.md
> > > +++ b/README.md
> > > @@ -112,6 +112,49 @@ $ meson setup -Dpython=disabled build
> > >  More work remains to support all of libfdt, including access to numeric
> > >  values.
> > >
> > > +## fdtfactor
> >
> > Pre-existing, but it would be nice to have blurbs for dtdiff,
> > fdt{get,put} and the other anciliary tools we already include.
> 
> Agreed -- in my draft v4 I've added a preparatory patch adding brief
> `README.md` sections for `dtc` and the ancillary tools (`dtdiff`,
> `fdtget`, `fdtput`, `fdtdump`, `convert-dtsv0`, and `fdtoverlay`).

Terrific, thanks.

> > > +        self.filename = filename
> > > +        with open(filename, 'rb') as f:
> > > +            raw_bytes = f.read()
> > > +        self.raw_bytes = raw_bytes
> >
> > AFAICT, self.raw_bytes is never used again (unlike the raw_bytes
> > local).
> 
> `_print_dtb_stats()` actually used `len(d.raw_bytes)` further down to

Ok, in a later patch.  Better to add the field at that point in the
series, then.

> compute `orig_uncomp` when `--stats` is passed, though nothing ever
> reads the buffer contents again after `__init__()`. In my draft v4 I've
> replaced `self.raw_bytes` with `self.uncompressed_size = len(raw_bytes)`
> (pairing with `self.compressed_size` in the compression patch) so we
> don't keep all the uncompressed DTB buffers in memory just for their
> lengths.

Or that.

> > > +            if 'phandle' in node.props:
> > > +                node.phandle = node.get_u32('phandle')
> > > +                del node.props['phandle']
> > > +
> > > +            if node.phandle is not None:
> >
> > Changing this to:
> > if node.phandle is None:
> > continue
> > reduces nesting and IMO, would make this a little more readable.
> 
> Done in my draft v4 (and did the same for a couple of other `for` loops
> in `fdtfactor`).

Ok.

> > > +                offsets = sorted(set(lf_node.get_u32_list(prop_name)))
> >
> > I don't think there's any reason to convert the list to a set before sorting.
> 
> Done in my draft v4 (`offsets = sorted(lf_prop.as_uint32_list())`).
> Neither `dtc` nor `libfdt` emits duplicate offsets in
> `/__local_fixups__`, and dropping `set()` also means any duplicate
> offset in a malformed blob will be caught by the overlapping-offset
> check right below.

Sounds good.

> > > +    base_root = FactoredTree()
> > > +    next_ph = 1
> > > +    for path, _ in dtbs[0].root.walk():
> > > +        if path not in common_paths:
> > > +            continue
> >
> > I think this makes the loop equivalent to:
> > for path in common_paths:
> >    ....
> >
> > > +        if path and base_root.get_node(path.rpartition('/')[0]) is None:
> > > +            continue
> 
> We can't iterate `common_paths` directly here because `common_paths` is
> an unordered `set`:

> 1. The `base_root.get_node(path.rpartition('/')[0]) is None` check right
>    below relies on parents always being visited before their children so
>    that if an ancestor was excluded, its descendants are skipped too.
>    Iterating a `set` can visit `/a/b` before `/a`, causing `/a/b` to be
>    wrongly skipped.
> 2. Walking `dtbs[0].root.walk()` preserves `dtbs[0]`'s subnode ordering
>    in `base_root` and keeps sequential `next_ph` assignment
>    deterministic across `PYTHONHASHSEED` values.

Ah, right.

> I've added a comment above the loop in my draft v4 explaining why it
> iterates `dtbs[0]`'s nodes in pre-order instead of iterating
> `common_paths`.

Thanks.

> > > +        has_phandle = any(
> > > +            d.root.get_node(path).phandle is not None for d in dtbs
> > > +        )
> >
> > I expect this will work in practice, but arguably the condition you
> > really need here is whether the node is referred to by any fixup in
> > any non-common property.
> >
> > > +        if has_phandle and path and path not in common_sym_targets:
> > > +            continue
> > > +        base_node = base_root.create_path(path)
> > > +        if has_phandle:
> > > +            base_node.phandle = next_ph
> > > +            next_ph += 1
> 
> For the `common_sym_targets` exclusion (`if has_phandle and path and
> path not in common_sym_targets: continue`), that's true in principle
> with respect to `/__fixups__`: an overlay only needs a symbol in
> `base.dtb`'s `/__symbols__` if a non-common property ends up referencing
> `path`. There are a couple of catches if we tried to relax this, though:
> 
> 1. Which properties are common vs. non-common isn't known yet here --
>    `_build_base_tree()` decides that next, and its candidate eligibility
>    depends on which fixup target nodes are in `base_root` (so resolving
>    both together would need an iterative pass).
> 2. Even when no non-common property references `path`, `base_node` still
>    needs a `phandle` in `base.dtb` if `path` is referenced by a *common*
>    property in `base.dtb`, or if `path` has a symbol in `/__symbols__`
>    (from `dtc -@`) so the reconstructed DTB retains its `phandle`
>    property and remains valid as a base for any further runtime overlays
>    targeting that symbol. Conversely, if one board had a `phandle` on
>    `path` (e.g. a board-specific label) and another board didn't,
>    putting `path` in `base.dtb` with a `phandle` would add a `phandle`
>    property to the second board's reconstructed DTB that wasn't in its
>    original DTB.
> 
> Since in `dtc -@` inputs almost every node with a `phandle` also has a
> label in `/__symbols__`, checking `has_phandle` up front keeps this
> single-pass and exact. I've expanded the docstring on
> `_find_common_nodes_and_phandles()` in my draft v4 to note this.

Ok, understood.

-- 
David Gibson (he or they)	| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you, not the other way
				| around.
http://www.ozlabs.org/~dgibson

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

  reply	other threads:[~2026-11-16  4:34 UTC|newest]

Thread overview: 54+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-11-01 22:01 [PATCH v3 00/15] fdtfactor: Introduce fdtfactor to auto-factor overlays Douglas Anderson
2026-11-01 22:01 ` [PATCH v3 01/15] libfdt: overlay: Fix out-of-bounds write on bad fixup offset Douglas Anderson
2026-11-02  4:29   ` David Gibson
2026-11-01 22:01 ` [PATCH v3 02/15] libfdt: overlay: Drop redundant *sep != ':' checks after memchr() Douglas Anderson
2026-11-02  4:30   ` David Gibson
2026-11-01 22:01 ` [PATCH v3 03/15] pylibfdt: Clarify iterator docstrings and test negative offsets Douglas Anderson
2026-11-02  4:32   ` David Gibson
2026-11-01 22:01 ` [PATCH v3 04/15] pylibfdt: Handle empty properties in as_str() and as_stringlist() Douglas Anderson
2026-11-02  4:33   ` David Gibson
2026-11-01 22:01 ` [PATCH v3 05/15] pylibfdt: Add Fdt.set_boot_cpuid_phys() Douglas Anderson
2026-11-02  4:34   ` David Gibson
2026-11-01 22:01 ` [PATCH v3 06/15] pylibfdt: Support copy.copy() and copy.deepcopy() on Property Douglas Anderson
2026-11-02  4:35   ` David Gibson
2026-11-01 22:01 ` [PATCH v3 07/15] devicetree: Add DeviceTree live tree representation Douglas Anderson
2026-11-02  9:31   ` David Gibson
2026-11-04  4:29     ` Doug Anderson
2026-11-04  6:12       ` David Gibson
2026-11-07  2:43         ` Doug Anderson
2026-11-10  2:10           ` David Gibson
2026-11-04 15:13       ` Simon Glass
2026-11-04 15:14   ` Simon Glass
2026-11-07  2:43     ` Doug Anderson
2026-11-11  2:56       ` David Gibson
2026-11-11 20:15         ` Simon Glass
2026-11-12  1:44           ` David Gibson
2026-11-12 19:28             ` Doug Anderson
2026-11-13  3:26               ` David Gibson
2026-11-13  3:30                 ` Doug Anderson
2026-11-01 22:01 ` [PATCH v3 08/15] libfdt: overlay: Factor out fixup and fragment path parsing helpers Douglas Anderson
2026-11-04  6:29   ` David Gibson
2026-11-01 22:01 ` [PATCH v3 09/15] libfdt: overlay: Prepare fragment path helper for reuse Douglas Anderson
2026-11-04 15:14   ` Simon Glass
2026-11-01 22:01 ` [PATCH v3 10/15] libfdt: overlay: Make fdto const in overlay_apply_node() Douglas Anderson
2026-11-04 15:14   ` Simon Glass
2026-11-01 22:01 ` [PATCH v3 11/15] libfdt: overlay: Update base /__local_fixups__ on apply Douglas Anderson
2026-11-04 15:14   ` Simon Glass
2026-11-07  2:43     ` Doug Anderson
2026-11-01 22:01 ` [PATCH v3 12/15] fdtfactor: Add tool to split DTBs into a base DTB and overlays Douglas Anderson
2026-11-11  4:16   ` David Gibson
2026-11-13  5:22     ` Doug Anderson
2026-11-16  3:35       ` David Gibson [this message]
2026-11-01 22:01 ` [PATCH v3 13/15] fdtfactor: Add -S/--strip-local-fixups option Douglas Anderson
2026-11-04 15:14   ` Simon Glass
2026-11-11  4:19   ` David Gibson
2026-11-01 22:01 ` [PATCH v3 14/15] fdtfactor: Support compressed DTB inputs and outputs Douglas Anderson
2026-11-12  2:10   ` David Gibson
2026-11-13  5:23     ` Doug Anderson
2026-11-01 22:01 ` [PATCH v3 15/15] fdtfactor: Add compiled FIT factoring support Douglas Anderson
2026-11-12  2:31   ` David Gibson
2026-11-13  5:23     ` Doug Anderson
2026-11-16  3:44       ` David Gibson
2026-11-02  4:44 ` [PATCH v3 00/15] fdtfactor: Introduce fdtfactor to auto-factor overlays David Gibson
2026-11-04 15:13 ` Simon Glass
2026-11-07  2:42   ` Doug Anderson

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=avp6GjL1jqEfEFkR@gractus.seuss \
    --to=david@gibson.dropbear.id.au \
    --cc=Devicetree-compiler@vger.kernel.org \
    --cc=Devicetree@vger.kernel.org \
    --cc=andre.draszik@linaro.org \
    --cc=briannorris@chromium.org \
    --cc=conor+dt@kernel.org \
    --cc=dianders@chromium.org \
    --cc=jwerner@chromium.org \
    --cc=konradybcio@kernel.org \
    --cc=krzk+dt@kernel.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=peter.griffin@linaro.org \
    --cc=robh@kernel.org \
    --cc=saravanak@kernel.org \
    --cc=sjg@chromium.org \
    --cc=tudor.ambarus@linaro.org \
    --cc=wenst@chromium.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox