Re: [PATCH 1/2] libfdt: overlay: Allow resolving phandle symbols

[Ayush Singh <ayush@beagleboard.org>]

On 9/18/24 08:06, David Gibson wrote:

> On Mon, Sep 16, 2024 at 03:10:52PM +0530, Ayush Singh wrote:
>> On 9/12/24 09:08, David Gibson wrote:
>>
>>> On Mon, Sep 09, 2024 at 12:54:34PM +0530, Ayush Singh wrote:
>>>> On 9/9/24 10:33, David Gibson wrote:
>>>>
>>>>> On Mon, Sep 02, 2024 at 05:47:55PM +0530, Ayush Singh wrote:
>>>>>> Add ability to resolve symbols pointing to phandles instead of strings.
>>>>>>
>>>>>> Combining this with existing fixups infrastructure allows creating
>>>>>> symbols in overlays that refer to undefined phandles. This is planned to
>>>>>> be used for addon board chaining [1].
>>>>> I don't think this "autodetection" of whether the value is a phandle
>>>>> or path is a good idea.  Yes, it's probably unlikely to get it wrong
>>>>> in practice, but sloppy cases like this have a habit of coming back to
>>>>> bite you later on.  If you want this, I think you need to design a new
>>>>> way of encoding the new options.
>>>> Would something like `__symbols_phandle__` work?
>>> Preferable to the overloaded values in the original version, certainly.
>> I can whip it up if that would be sufficient. But if we are talking about
>> any big rewrite, well, I would like to get the details for that sorted out
>> first.
> Fair enough.
>
>>>> It should be fairly
>>>> straightforward to do. I am not sure how old devicetree compilers will react
>>>> to it though?
>>> Well, the devicetree compiler itself never actually looks at the
>>> overlay encoding stuff.  The relevant thing is libfdt's overlay
>>> application logic... and any other implementations of overlay handling
>>> that are out there.
>>>
>>> At which I should probably emphasize that changes to the overlay
>>> format aren't really mine to approve or not.  It's more or less an
>>> open standard, although not one with a particularly formal definition.
>>> Of course, libfdt's implementation - and therefore I - do have a fair
>>> bit of influence on what's considered the standard.
>> So do I need to start a discussion for this somewhere other than the
>> devicetree-compiler mailing list? Since ZephyrRTOS is also using devicetree,
>> maybe things need to be discussed with them as well?
> <devicetree-spec@vger.kernel.org> and <devicetree@vger.kernel.org> are
> the obvious candidate places.  There will be substantial overlap with
> devicetree-compiler of course, but not total probably.
>
>>>> I really do not think having a different syntax for phandle symbols would be
>>>> good since that would mean we will have 2 ways to represent phandles:
>>>>
>>>> 1. For __symbols__
>>>>
>>>> 2. For every other node.
>>> I'm really not sure what you're getting at here.
>> I just meant that I would like to keep the syntax the same:
>>
>> sym = <&abc>;
> Ok, the syntax for creating them in dts, rather than the encoding
> within the dtb.  Why are you manually creating symbols?
>
> So.. aside from all the rest, I don't really understand why you want
> to encode the symbols as phandles rather than paths.

It's for connector stacking using the approach described here [0].

To go into more detail, let us assume that we have a mikrobus connector 
on the base board. We connect an addon board that exposes a grove 
connector. Now to describe the parent i2c adapter of the grove 
connector, we cannot really specify the full node path. However, having 
support for phandles would make writing the overlay for such an addon 
board possible.

In practice it might look as follows:


mikrobus-connector.dtso


&{/} {

    __symbols__ {

         MIKROBUS_SCL_I2C = "path";

         ...

     };

}


grove-connector-addon.dtso


&{/} {

     __symbols__ {

         GROVE_PIN1_I2C = <&MIKROBUS_SCL_I2C>;

     };

};


grove-addon-board.dtso


&MIKROBUS_SCL_I2C {

     dev {

         ...

     };

};

>>>> I also am not in the favor of doing something bespoke that does not play
>>>> nice with the existing __fixups__ infra since that has already been
>>>> thoroughly tested, and already creates __fixups__ for symbols.
>>> Hmm.  Honestly, the (runtime) overlay format was pretty a hack that's
>>> already trying to do rather more than it was really designed for.  I'm
>>> a bit sceptical of any attempt to extend it further without
>>> redesigning the whole thing with a bit more care and forethought.  I
>>> believe Rob Herring has some thoughts along these lines too.
>> Well, if we are really redesigning stuff, does that mean something like
>> dts-v2, or everything should still be backward compatible?
> Well.. it depends.  There are actually 3 or 4 different layers to consider:
>
> 1) The basic data model of the device tree as consumed by the kernel
>
> This is layer which "properties are just bytestrings" comes from.
>
> There is a case for redesigning this, since it comes from IEEE1275 and
> shows its age.  A modern format would likely be self-describing, so
> type information would be preserved.  A json-derivative would be the
> obvious choice here, except that json can't safely transport 64-bit
> integers, which is kind of a fatal flaw for this application.
>
> This would be a fundamentally incompatible change for all current
> consumers of the device tree, though.  And when I say consumers, I
> don't just mean the kernel base platform code, I mean all the
> individual drivers which actually need the device tree information.
> I'm not suggesting this layer be changed.
>
> 2) The "dtb" format: The linearized encoding of the data model above
>
> Changing this is much more tractable.  The dtb header includes version
> numbers, allowing some degree of backwards compatibility.  There have
> been, IIRC, 5 versions in total (v1, v2, v3, v16 & v17), though we've
> been on v17 for a long time (10+ years) now.
>
> There's not a heap you that can be changed here - at least neatly -
> without requiring an incompatible version bump.  However dealing with
> even an incompatible change at this layer is *much* easier.  As long
> as the base data model remains the same you can mechanically convert
> between versions, at least as long as you're not actively using new
> features.  In particular it's pretty feasible to have a whole set of
> tooling using a newer version that as a last step converts a final
> tree to an old version for consumption by the kernel or whatever.
>
> 3) The "dts" format: the human readable / writable format
>
> Being parsed text, it's relatively easy to extend this in backwards
> compatible ways.  Note that this is more influenced by (1) than the
> details of (2).  To this day, dtc can input and output the
> long-obsolete v1 dtb format, and that doesn't add a lot of complexity
> to it.
>
> Any change to the other layers would likely require extensions here as
> well, but I don't think there would be a need for backwards
> incompatible changes.  Using certain features in the source might
> impose a minimum dtb output version though.
>
> Note that dts isn't in one to one correspondance with either (1) or
> (2): there are multiple ways of specifying identical output, as there
> would be in most languages.  That is a recurring source of confusion,
> but I can't see a reasonable way of changing it short of a complete
> redesign of (1), in which case "dts" would likely simply be obsoleted.
>
> 4) The overlay specific encoding
>
> Overlays first existed purely as a dts construct: a different way of
> arranging things in the source that would compile to the same final
> tree.  Runtime overlays (a.k.a. dtbo) came later.  This is by far the
> most poorly designed layer, IMO.
>
> Basically when dtbos were invented, it was done as a quick and dirty
> encoding of the dts level overlay features into the data model of (1),
> which was then just encoded using (2), thereby blurring the layers a
> bit.  No real thought was given to versioning or backwards
> compatibility.
>
> This is where I think a redesign would make most sense.  However, the
> most sensible (IMO) ways of doing so would also require some redesign
> to (2).  Basically rather than encoding the overlay specific
> information "in band" as specially named and encoded properties, it
> would be carried "out of band" as new special tags in the updated dtb
> format.  You can think of this as a bit like relocation information
> that a C compiler emits alongside the actual instruction stream.
>
>> The problem with guessing type can probably be fixed with a tagged union for
>> the type (so an extra byte for every prop).
> Yeah, it's not so simple.  As things stand this would imply a change
> to (1), which as noted probably isn't really feasible.  Plus, just a
> one-byte type per property wouldn't cut it: some bindings have complex
> encoded values in properties that are a mixture of integers and
> strings and so forth.
>
>> With more and more emphasis on runtime devicetree [0], it would be great to
>> have a design that allows tackling more complex use cases.
>>
>> [0]: https://lpc.events/event/18/contributions/1696/
> Oof.   I don't think overlays as currently designed are a great choice
> for hotplug: overlays essentially allow rewriting any part of the
> whole device tree, which isn't a great model for a hotplug bus.  A
> long time ago some ideas were floated to define "connectors" in the
> device tree that more specifically described a way things could be
> plugged in that could cover several busses / interrupt controllers /
> etc.  That would provide a more structured way of describing plug in
> devices that can actually validate what they can do.
>
> Of course, such a scheme is a lot more work to design and implement,
> which is why the simple hack of overlays have become dominant instead.

Well, a lot of work is still going in this direction [1]. I myself am 
trying to use it for mikroBUS connectors.

The reason for using devicetree overlays for such connectors is because 
of their loose nature (mikroBUS is also open connector). This means that 
as long as the sensor/ device can make do with the pins provided by 
mikroBUS connector, it can be an addon board. And there is no limitation 
of staking the boards. So one can do rpi hat -> mikrbus connectors -> 
grove connector -> grove some addon board.


[0]: 
https://lore.kernel.org/linux-arm-kernel/20240702164403.29067-1-afd@ti.com/

[1]: https://lpc.events/event/18/contributions/1696/


Ayush Singh