Re: [PATCH v3 0/4] Introduce fdt_overlay_merge() to allow merge of overlay blobs

[David Gibson <david@gibson.dropbear.id.au>]

[-- Attachment #1: Type: text/plain, Size: 6294 bytes --]

On Mon, May 19, 2025 at 02:40:39PM +0530, Wasim Nazir wrote:
> Hello,
> 
> This is follow-up attempt for fdtoverlaymerge tool.
> 
> Currently all the device-tree (DT) code for a given soc is maintained in a
> common kernel repository. For example, this common DT code will have code for
> audio, video, fingerprint, bluetooth etc. Further this, DT code is typically
> split into a base (soc-common) code and board specific code, with the soc code
> being compiled as soc.dtb and board specific code being compiled as respective
> overlay blobs (board1.dtbo, board2.dtbo etc). soc.dtb represents hardware configuration
> of a given SOC while boardX.dtbo represents configuration of a board/platform
> designed using that soc.soc.dtb and boardX.dtbo files are flashed separately on

So.. *build time* separation of the SoC and board pieces makes sense
to me, which is I think how this convention arose.  *Boot time*
separation of the SoC and board seems kind of pointless.  Almost by
definition of what a "board" is, you must know early in boot which
board it is.  Still, I guess the convention is established, even if
it's stupid.

> target (besides improving the overall size of DT blobs flashed on target, Android
> Treble also requires separation of soc and board DT bits). Bootloader will pick
> one of the board overlay blobs and merge it with soc.dtb, before booting kernel
> which is presented a unified DT blob (soc + board overlay).
> 
> For ease of code maintenance and better control over release management, we are
> exploring allowing some of the tech teams (audio/fingerprint sensor etc) to
> maintain their kernel code (including their DT code) outside a common kernel
> repository. In our experience, this simplifies number of branches maintained in
> core kernel repo. New/experimental features in fingerprint sensor driver for
> example that needs to be on a separate branch will not result in unnecessary
> branching in core kenrel repo, affecting all other drivers.
> 
> In addition to compiling DT code outside core kernel tree, we also want to merge
> the blobs back to respective blobs found in kernel build tree at buildtime
> (soc.dtb or boardX.dtbo), as otherwise relying on bootloader to do all the
> overlay impacts boot-time.

It's again unclear to me why you need a boot time separation of these
devices rather than merely boot time.  What does using separate .dtbo
files give you that just /include/ing multiple pieces into a single
.dtbo at build time would not?

> This brings up the need to merge two overlay blobs (fingerprint-overlay.dtbo +
> boardX.dtbo), which currently doesn't seem to be supported and which this patch
> series aims to support.

Merging overlays is a logically sensible operation, but it's not clear
to me why the need for it follows from the premises above.  It's also
unclear why you need to compile to .dtbo *then* merge, rather than
combine .dts files then compile into a single .dtbo.

> fdt_overlay_apply() API currently allows for an overlay DT blob to be merged
> with a base blob. It assumes that all external symbols specified in overlay
> blob's __fixups__ section are found in base blob's __symbols__ section and
> aborts on the first instance where a symbol could not be found in base blob.
> This is mostly fine as the primary use of overlay is on a target for its
> bootloader to merge various overlay blobs based on h/w configuration detected.
> But when the number of overlays increased then bootloader takes lot of time to
> apply the overlays on base DT.
> 
> So we need new API/tool to merge all the overlays into single overlay file
> at host (build machine) side,

Merging into a single overlay at build time makes sense to me.  But at
build time you'd expect to have access to the .dts files.  Why do you
need to merge .dtbo rather than merge the .dts before compiling to
.dtbo?  The latter should be possible already by /include/ing each of
the individual overlays in order then compiling with dtc.

> so that on target side bootloader needs to only
> apply merged-overlay-dt to its base-dt. This saves lot of time due to reduced
> number file reading/loading & minimizing repeatative overlay apply.
> In our test setup we see an improvement of ~60% while applying merged-overlay
> at bootloader and the merged-overlay is product of 7 overlays.
> 
> To serve this overlay-merge feature we have introduce fdtoverlaymerge tool
> which takes input as overlays and gives output to merged-overlay.
> The tool uses fdt_overlay_merge() API introduced in libfdt to do the actual work.
> 
> Additional notes:
>   If snprintf (in libc) may not available in some environments, then we will need
>   to write our own snprintf() in libfdt.
> 
> ---
> Changelog:
> 
> v3:
> - Update copy_node & add copy_fragment_to_base to incorporate two cases i.e
>   - Case1: When target is available and we merge fragments
>   - Case2: When target is not available and we add new fragments
> - Change the logic to update fixups & local_fixups in case of overlay merge.
> - Few patches are squashed, reduced to 4 patches.
> - v2-link: https://lore.kernel.org/all/1599671882-310027-1-git-send-email-gurbaror@codeaurora.org/
> 
> 
> Srivatsa Vaddagiri (4):
>   libfdt: overlay_merge: Introduce fdt_overlay_merge()
>   libfdt: overlay_merge: Rename & copy overlay fragments and their
>     properties
>   libfdt: overlay_merge: Update phandles, symbols, fixups & local_fixups
>   fdtoverlaymerge: A tool that merges overlays
> 
>  .gitignore           |   1 +
>  Makefile             |   4 +
>  Makefile.utils       |   6 +
>  fdtoverlaymerge.c    | 223 +++++++++++
>  libfdt/fdt_overlay.c | 901 ++++++++++++++++++++++++++++++++++++++++++-
>  libfdt/fdt_rw.c      |  14 +-
>  libfdt/libfdt.h      |  18 +
>  libfdt/version.lds   |   1 +
>  meson.build          |   2 +-
>  9 files changed, 1146 insertions(+), 24 deletions(-)
>  create mode 100644 fdtoverlaymerge.c
> 
> 
> base-commit: f4c53f4ebf7809a07666bf728c823005e1f1a612

-- 
David Gibson (he or they)	| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you, not the other way
				| around.
http://www.ozlabs.org/~dgibson

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]