[U-Boot] Merging device trees at runtime for module-based systems

[U-Boot] Merging device trees at runtime for module-based systems

[Daniel Mack]

Hi,

a project I'm involved in uses a module/baseboard combo, and components
on either board are described in DT. I'm currently using separate dts
files which build upon each other with include statements, which works
fine for development.

In production though, we will certainly have running changes (and hence
different versions) over the lifetime of the product for both the
baseboard and the module, and the hardware has support for identifying
the versions of both sides at runtime.

So let's say we have n versions of the baseboard and m versions of the
module, we would much like to only prepare n + m files, instead of n * m
by pre-compiling every possible combination (some of which may actually
never occur 'in the wild').

So my question is: is it possible to do that kind of assembly of a
number of files at runtime in U-Boot? I guess all it takes is merging a
number of trees together, right? I browsed through the APIs but couldn't
yet find an clear approach to that kind of problem. If not, what would
it take to add that functionality? I can probably help with the
implementation if someone tells me what would be the right way.

Any pointer greatly appreciated.


Thanks,
Daniel

[U-Boot] Merging device trees at runtime for module-based systems

[Wolfgang Denk]

Dear Daniel,

In message <5087B919.2010006@gmail.com> you wrote:
> 
> So let's say we have n versions of the baseboard and m versions of the
> module, we would much like to only prepare n + m files, instead of n * m
> by pre-compiling every possible combination (some of which may actually
> never occur 'in the wild').

What you are facing is a situation that
1) appears to be pretty common, and
2) has (AFAICT) not been truely satisfactory solved yet.

> So my question is: is it possible to do that kind of assembly of a
> number of files at runtime in U-Boot? I guess all it takes is merging a
> number of trees together, right? I browsed through the APIs but couldn't
> yet find an clear approach to that kind of problem. If not, what would
> it take to add that functionality? I can probably help with the
> implementation if someone tells me what would be the right way.

I think it should be possible to overlay several DT images; ideally
these would be strictly orthogonal, i. e. the newly loaded one would
only add new properties, but I think it should be also no problem to
define some latest-wins policy, i. e. in case of already existing
properties the newly loaded ones would just overwrite the previous
settings.

I definitely can see the benefit of such a feature and would be happy
if you could go forward and implement it.

Thanks in advance.

Best regards,

Wolfgang Denk

-- 
DENX Software Engineering GmbH,     MD: Wolfgang Denk & Detlev Zundel
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd at denx.de
While money can't buy happiness, it certainly lets  you  choose  your
own form of misery.

[U-Boot] Merging device trees at runtime for module-based systems

[Daniel Mack]

Hi Wolfgang,

On 25.10.2012 14:44, Wolfgang Denk wrote:
> In message <5087B919.2010006@gmail.com> you wrote:
>>
>> So let's say we have n versions of the baseboard and m versions of the
>> module, we would much like to only prepare n + m files, instead of n * m
>> by pre-compiling every possible combination (some of which may actually
>> never occur 'in the wild').
> 
> What you are facing is a situation that
> 1) appears to be pretty common, and
> 2) has (AFAICT) not been truely satisfactory solved yet.
> 
>> So my question is: is it possible to do that kind of assembly of a
>> number of files at runtime in U-Boot? I guess all it takes is merging a
>> number of trees together, right? I browsed through the APIs but couldn't
>> yet find an clear approach to that kind of problem. If not, what would
>> it take to add that functionality? I can probably help with the
>> implementation if someone tells me what would be the right way.
> 
> I think it should be possible to overlay several DT images; ideally
> these would be strictly orthogonal, i. e. the newly loaded one would
> only add new properties, but I think it should be also no problem to
> define some latest-wins policy, i. e. in case of already existing
> properties the newly loaded ones would just overwrite the previous
> settings.

Overwrites must be addressed in the first place. The most common example
is that a more generic part (the module tree) registers all details
about a peripheral up-front but then sets its status to 'disabled'. That
way, the more specific part (the base board tree) can overwrite this
property to 'okay' at wish to enable it and not care for the pre-defined
details. This is also how we do things in our device-trees.

> I definitely can see the benefit of such a feature and would be happy
> if you could go forward and implement it.

Ok then. I guess this should be something that can eventually be merged
back into libfdt?


Thanks,
Daniel

[U-Boot] Merging device trees at runtime for module-based systems

[Wolfgang Denk]

Dear Daniel,

In message <50893633.6070408@gmail.com> you wrote:
> 
> Overwrites must be addressed in the first place. The most common example
> is that a more generic part (the module tree) registers all details
> about a peripheral up-front but then sets its status to 'disabled'. That
> way, the more specific part (the base board tree) can overwrite this
> property to 'okay' at wish to enable it and not care for the pre-defined
> details. This is also how we do things in our device-trees.

Agreed.

> > I definitely can see the benefit of such a feature and would be happy
> > if you could go forward and implement it.
> 
> Ok then. I guess this should be something that can eventually be merged
> back into libfdt?

I can't speak for the FDT custodian, but I think this makes a lot of
sense.

Best regards,

Wolfgang Denk

-- 
DENX Software Engineering GmbH,     MD: Wolfgang Denk & Detlev Zundel
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd at denx.de
When the bosses talk about improving  productivity,  they  are  never
talking about themselves.

[U-Boot] Merging device trees at runtime for module-based systems

[David Gibson]

On Thu, Oct 25, 2012 at 10:46:32PM +0200, Wolfgang Denk wrote:
> Dear Daniel,
> 
> In message <50893633.6070408@gmail.com> you wrote:
> > 
> > Overwrites must be addressed in the first place. The most common example
> > is that a more generic part (the module tree) registers all details
> > about a peripheral up-front but then sets its status to 'disabled'. That
> > way, the more specific part (the base board tree) can overwrite this
> > property to 'okay' at wish to enable it and not care for the pre-defined
> > details. This is also how we do things in our device-trees.
> 
> Agreed.
> 
> > > I definitely can see the benefit of such a feature and would be happy
> > > if you could go forward and implement it.
> > 
> > Ok then. I guess this should be something that can eventually be merged
> > back into libfdt?
> 
> I can't speak for the FDT custodian, but I think this makes a lot of
> sense.

As a rule I'm happy to see more functionality for libfdt.  I've only
seen bits and pieces of this thread, though, so I'd need to see a
summary of what exactly is being proposed.

-- 
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson

[U-Boot] Merging device trees at runtime for module-based systems

[Daniel Mack]

On 26.10.2012 02:53, David Gibson wrote:
> On Thu, Oct 25, 2012 at 10:46:32PM +0200, Wolfgang Denk wrote:
>> Dear Daniel,
>>
>> In message <50893633.6070408@gmail.com> you wrote:
>>>
>>> Overwrites must be addressed in the first place. The most common example
>>> is that a more generic part (the module tree) registers all details
>>> about a peripheral up-front but then sets its status to 'disabled'. That
>>> way, the more specific part (the base board tree) can overwrite this
>>> property to 'okay' at wish to enable it and not care for the pre-defined
>>> details. This is also how we do things in our device-trees.
>>
>> Agreed.
>>
>>>> I definitely can see the benefit of such a feature and would be happy
>>>> if you could go forward and implement it.
>>>
>>> Ok then. I guess this should be something that can eventually be merged
>>> back into libfdt?
>>
>> I can't speak for the FDT custodian, but I think this makes a lot of
>> sense.
> 
> As a rule I'm happy to see more functionality for libfdt.  I've only
> seen bits and pieces of this thread, though, so I'd need to see a
> summary of what exactly is being proposed.

That's strange, as I copied you from the very first posting. Anyway,
here's the archive:

  http://lists.denx.de/pipermail/u-boot/2012-October/138227.html

I would especially like to know where such a new functionality should
live, which data types it should operate on and what would be an
appropriate name for it.


Many thanks,
Daniel

[U-Boot] Merging device trees at runtime for module-based systems

[Simon Glass]

Hi Daniel,

On Fri, Oct 26, 2012 at 12:24 AM, Daniel Mack <zonque@gmail.com> wrote:
> On 26.10.2012 02:53, David Gibson wrote:
>> On Thu, Oct 25, 2012 at 10:46:32PM +0200, Wolfgang Denk wrote:
>>> Dear Daniel,
>>>
>>> In message <50893633.6070408@gmail.com> you wrote:
>>>>
>>>> Overwrites must be addressed in the first place. The most common example
>>>> is that a more generic part (the module tree) registers all details
>>>> about a peripheral up-front but then sets its status to 'disabled'. That
>>>> way, the more specific part (the base board tree) can overwrite this
>>>> property to 'okay' at wish to enable it and not care for the pre-defined
>>>> details. This is also how we do things in our device-trees.
>>>
>>> Agreed.
>>>
>>>>> I definitely can see the benefit of such a feature and would be happy
>>>>> if you could go forward and implement it.
>>>>
>>>> Ok then. I guess this should be something that can eventually be merged
>>>> back into libfdt?
>>>
>>> I can't speak for the FDT custodian, but I think this makes a lot of
>>> sense.
>>
>> As a rule I'm happy to see more functionality for libfdt.  I've only
>> seen bits and pieces of this thread, though, so I'd need to see a
>> summary of what exactly is being proposed.
>
> That's strange, as I copied you from the very first posting. Anyway,
> here's the archive:
>
>   http://lists.denx.de/pipermail/u-boot/2012-October/138227.html
>
> I would especially like to know where such a new functionality should
> live, which data types it should operate on and what would be an
> appropriate name for it.
>

It seems like this functionality should live in libfdt with its own
tests there. If you are going to merge then you could perhaps also
'unmerge', i.e. given a base fdt and a derivative, create the 'diff'
device tree.

It should probably deal with all nodes/properties - after all the fdt
doesn't really understand data types.

Regards,
Simon

>
>
> Many thanks,
> Daniel
>
> _______________________________________________
> U-Boot mailing list
> U-Boot at lists.denx.de
> http://lists.denx.de/mailman/listinfo/u-boot

[U-Boot] Merging device trees at runtime for module-based systems

[Stephen Warren]

On 10/24/2012 03:47 AM, Daniel Mack wrote:
> Hi,
> 
> a project I'm involved in uses a module/baseboard combo, and components
> on either board are described in DT. I'm currently using separate dts
> files which build upon each other with include statements, which works
> fine for development.
> 
> In production though, we will certainly have running changes (and hence
> different versions) over the lifetime of the product for both the
> baseboard and the module, and the hardware has support for identifying
> the versions of both sides at runtime.
> 
> So let's say we have n versions of the baseboard and m versions of the
> module, we would much like to only prepare n + m files, instead of n * m
> by pre-compiling every possible combination (some of which may actually
> never occur 'in the wild').
> 
> So my question is: is it possible to do that kind of assembly of a
> number of files at runtime in U-Boot? I guess all it takes is merging a
> number of trees together, right? I browsed through the APIs but couldn't
> yet find an clear approach to that kind of problem. If not, what would
> it take to add that functionality? I can probably help with the
> implementation if someone tells me what would be the right way.

Yes, solving this would be very useful; it's a wide-spread problem.

Some thoughts though:

Simply overlaying two DTBs on top of each-other (in the same fashion
that dtc's /include/ statement would do at compile-time) might not be
fully general enough, although perhaps it would be sufficient for your
immediate needs.

For example, lets say that a GPIO is routed from a device on the main
board to a device on a daughter board, or even from one daughter board
into the main board and back out to a different daughter board. Now,
consider that the different board(s) that are the source of the GPIO
might use completely different SoCs or versions of the SoC, which might
require using a different GPIO specifier to represent the signal. That
means you need to change the .dtb file for the "client" of the GPIO
depending on the HW or .dtb that provides the GPIO. That's certainly not
a simple matter of merging multiple .dtb blobs together.

The same issue could easily apply to I2C or SPI buses, chip selects, etc.

One solution would be to explicitly represent a connector or
connection-point in DT, such that the connector can implement the naming
of all signals that pass through it, and provide a translation point for
hooking the two DT fragments together. This seems within the spirit of DT.

Another solution might be some form of variables/macros/code in the DTB
that can be used to parameterize other DTBs that get merged with it.
This is probably an enormous can of worms.

I wonder if similar yet more subtle issues might arise, such as some
motherboards requiring an active-low IRQ signal yet others requiring an
active-high IRQ signal, thus requiring a daughter-board to program its
IRQ source differently. Similarly, what about different drive strength
requirements for a signal source, depending on what board version
receives the signal?

[U-Boot] Merging device trees at runtime for module-based systems

[Wolfgang Denk]

Dear Stephen Warren,

In message <508AD8F9.8030105@wwwdotorg.org> you wrote:
>
> Simply overlaying two DTBs on top of each-other (in the same fashion
> that dtc's /include/ statement would do at compile-time) might not be
> fully general enough, although perhaps it would be sufficient for your
> immediate needs.

I think it should be sufficient for the overwhelming majority of use
cases.  When designing and implementing this feature, I suggest to
start small with the most common use cases in mind only.

> For example, lets say that a GPIO is routed from a device on the main
> board to a device on a daughter board, or even from one daughter board
> into the main board and back out to a different daughter board. Now,
> consider that the different board(s) that are the source of the GPIO
> might use completely different SoCs or versions of the SoC, which might
> require using a different GPIO specifier to represent the signal. That
> means you need to change the .dtb file for the "client" of the GPIO
> depending on the HW or .dtb that provides the GPIO. That's certainly not
> a simple matter of merging multiple .dtb blobs together.

Yes, one can construct arbitrarily complicated situations.  But I
think it is perfectly reasonable to ignore these, at least for the
initial implementation.

I'm not even convinced that we should try to come up with a solution
that is capable of dealing automtically with any situation of such
complexity.  In reality, we can probably combine a simple overly
mechanism with additional fixup though some shell script running FDT
manipulation commands directly.

> I wonder if similar yet more subtle issues might arise, such as some
> motherboards requiring an active-low IRQ signal yet others requiring an
> active-high IRQ signal, thus requiring a daughter-board to program its
> IRQ source differently. Similarly, what about different drive strength
> requirements for a signal source, depending on what board version
> receives the signal?

I suggest to try to ignore such situations for now, and get started
with a working, simple implementation.  If we actually run into a
situation where handling such situations is needed, we can then
discuss about solutions based on a much beter understanding and
experience with the - then - existing simple code.


In short: let's do a simple, working thing first, and add bells and
whistles later.

Best regards,

Wolfgang Denk

-- 
DENX Software Engineering GmbH,     MD: Wolfgang Denk & Detlev Zundel
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd at denx.de
A dog always bit deepest on the veterinary hand.
                                    - Terry Pratchett, _Wyrd Sisters_

[U-Boot] Merging device trees at runtime for module-based systems

[Daniel Mack]

cc devicetree-discuss. Here's a reference to the full thread:

  http://thread.gmane.org/gmane.comp.boot-loaders.u-boot/145221/

On 26.10.2012 20:39, Stephen Warren wrote:
> On 10/24/2012 03:47 AM, Daniel Mack wrote:
>> Hi,
>>
>> a project I'm involved in uses a module/baseboard combo, and components
>> on either board are described in DT. I'm currently using separate dts
>> files which build upon each other with include statements, which works
>> fine for development.
>>
>> In production though, we will certainly have running changes (and hence
>> different versions) over the lifetime of the product for both the
>> baseboard and the module, and the hardware has support for identifying
>> the versions of both sides at runtime.
>>
>> So let's say we have n versions of the baseboard and m versions of the
>> module, we would much like to only prepare n + m files, instead of n * m
>> by pre-compiling every possible combination (some of which may actually
>> never occur 'in the wild').
>>
>> So my question is: is it possible to do that kind of assembly of a
>> number of files at runtime in U-Boot? I guess all it takes is merging a
>> number of trees together, right? I browsed through the APIs but couldn't
>> yet find an clear approach to that kind of problem. If not, what would
>> it take to add that functionality? I can probably help with the
>> implementation if someone tells me what would be the right way.
> 
> Yes, solving this would be very useful; it's a wide-spread problem.
> 
> Some thoughts though:
> 
> Simply overlaying two DTBs on top of each-other (in the same fashion
> that dtc's /include/ statement would do at compile-time) might not be
> fully general enough, although perhaps it would be sufficient for your
> immediate needs.
> 
> For example, lets say that a GPIO is routed from a device on the main
> board to a device on a daughter board, or even from one daughter board
> into the main board and back out to a different daughter board. Now,
> consider that the different board(s) that are the source of the GPIO
> might use completely different SoCs or versions of the SoC, which might
> require using a different GPIO specifier to represent the signal. That
> means you need to change the .dtb file for the "client" of the GPIO
> depending on the HW or .dtb that provides the GPIO. That's certainly not
> a simple matter of merging multiple .dtb blobs together.

Hmm. After implementing a very simple overlay approach, I can now see
your point :) Yes in fact, that's a real problem.

> The same issue could easily apply to I2C or SPI buses, chip selects, etc.
> 
> One solution would be to explicitly represent a connector or
> connection-point in DT, such that the connector can implement the naming
> of all signals that pass through it, and provide a translation point for
> hooking the two DT fragments together. This seems within the spirit of DT.

Yes, but you still can't handle references that way.

Let me try and conclude this for others. Say the "module" tree "A" looks
something like this:

	/ {
		multi-regulator {
			vcc1v8: regulator at 0 {
				/* ... */
			};
		};
	};

... and the baseboard ("B"), that makes use of (and hence depends on)
the module, has something like this:

	/ {
		consumer {
			main-supply = <&vcc1v8>;
		};
	};

Now, let's say in a subsequent version of the module, we change whatever
provides that supply for 1.8 volts, but the consumer on the baseboard
shouldn't care much of course, thanks to all the abstraction layers that
we have now in the kernel.

However, the problem here is that I can't just compile trees A and B
individually into .dtbs that get merged later, because dtc will bail on
the unresolved reference of &vcc1v8 of course. And cases like this are
the whole reason why I started to think about modularization of trees in
the first place.

So the simple overlay method doesn't help here at all, even though I can
share the code if anyone's interested.

> Another solution might be some form of variables/macros/code in the DTB
> that can be used to parameterize other DTBs that get merged with it.
> This is probably an enormous can of worms.

Yes, exactly, a can of worms and most probably unmaintainble in real
life. I start to believe that the cleanest solution to this would be to
have full DTC functionality in U-Boot and compile the tree from dts, but
then again I have no clue on how to handle the file lookups that arise
from includes. Do you think it would it be worth going that way?

If not, I guess we're down to n*m files eventually, which is really sad
as they might even become a storage problem at some point.


Thanks for your input,
Daniel

[U-Boot] Merging device trees at runtime for module-based systems

[Stephen Warren]

On 10/31/2012 05:00 PM, Daniel Mack wrote:
> cc devicetree-discuss. Here's a reference to the full thread:
> 
>   http://thread.gmane.org/gmane.comp.boot-loaders.u-boot/145221/
> 
> On 26.10.2012 20:39, Stephen Warren wrote:
>> On 10/24/2012 03:47 AM, Daniel Mack wrote:
>>> Hi,
>>>
>>> a project I'm involved in uses a module/baseboard combo, and components
>>> on either board are described in DT. I'm currently using separate dts
>>> files which build upon each other with include statements, which works
>>> fine for development.
>>>
>>> In production though, we will certainly have running changes (and hence
>>> different versions) over the lifetime of the product for both the
>>> baseboard and the module, and the hardware has support for identifying
>>> the versions of both sides at runtime.
>>>
>>> So let's say we have n versions of the baseboard and m versions of the
>>> module, we would much like to only prepare n + m files, instead of n * m
>>> by pre-compiling every possible combination (some of which may actually
>>> never occur 'in the wild').
>>>
>>> So my question is: is it possible to do that kind of assembly of a
>>> number of files at runtime in U-Boot? I guess all it takes is merging a
>>> number of trees together, right? I browsed through the APIs but couldn't
>>> yet find an clear approach to that kind of problem. If not, what would
>>> it take to add that functionality? I can probably help with the
>>> implementation if someone tells me what would be the right way.
>>
>> Yes, solving this would be very useful; it's a wide-spread problem.
>>
>> Some thoughts though:
>>
>> Simply overlaying two DTBs on top of each-other (in the same fashion
>> that dtc's /include/ statement would do at compile-time) might not be
>> fully general enough, although perhaps it would be sufficient for your
>> immediate needs.
>>
>> For example, lets say that a GPIO is routed from a device on the main
>> board to a device on a daughter board, or even from one daughter board
>> into the main board and back out to a different daughter board. Now,
>> consider that the different board(s) that are the source of the GPIO
>> might use completely different SoCs or versions of the SoC, which might
>> require using a different GPIO specifier to represent the signal. That
>> means you need to change the .dtb file for the "client" of the GPIO
>> depending on the HW or .dtb that provides the GPIO. That's certainly not
>> a simple matter of merging multiple .dtb blobs together.
> 
> Hmm. After implementing a very simple overlay approach, I can now see
> your point :) Yes in fact, that's a real problem.
> 
>> The same issue could easily apply to I2C or SPI buses, chip selects, etc.
>>
>> One solution would be to explicitly represent a connector or
>> connection-point in DT, such that the connector can implement the naming
>> of all signals that pass through it, and provide a translation point for
>> hooking the two DT fragments together. This seems within the spirit of DT.
> 
> Yes, but you still can't handle references that way.
> 
> Let me try and conclude this for others. Say the "module" tree "A" looks
> something like this:
> 
> 	/ {
> 		multi-regulator {
> 			vcc1v8: regulator at 0 {
> 				/* ... */
> 			};
> 		};
> 	};
> 
> ... and the baseboard ("B"), that makes use of (and hence depends on)
> the module, has something like this:
> 
> 	/ {
> 		consumer {
> 			main-supply = <&vcc1v8>;
> 		};
> 	};
> 
> Now, let's say in a subsequent version of the module, we change whatever
> provides that supply for 1.8 volts, but the consumer on the baseboard
> shouldn't care much of course, thanks to all the abstraction layers that
> we have now in the kernel.
> 
> However, the problem here is that I can't just compile trees A and B
> individually into .dtbs that get merged later, because dtc will bail on
> the unresolved reference of &vcc1v8 of course. And cases like this are
> the whole reason why I started to think about modularization of trees in
> the first place.
> 
> So the simple overlay method doesn't help here at all, even though I can
> share the code if anyone's interested.

Yes, you've understood me exactly.

The connector-base approach I was thinking about might look (very very)
roughly as follows:

main board:

	/ {
		multi-regulator {
			vcc1v8: regulator at 0 {
				/* ... */
			};
		};
		connector {
			compatible = "vendor,board-socket-a";
			vcc1v8-supply = <&vcc1v8>;
		};
	};

child board:

	/ {
		connector {
			compatible = "vendor,board-plug-a";
			vcc1v8: regulator {
			};
		};
		consumer {
			main-supply = <&vcc1v8>;
		};
	};

... plus some logic so that the "driver"s for the two connector nodes
get linked together, such that the code forwards "requests" for the
regulator that the plug receives on to the node for the socket, which
then lists the actual provider.

Obviously, the above DT is an extremely rough sketch, and we need to
think about:

a) Exactly how the plug/socket get linked together.

b) Can we make a generic driver for the plug/socket, so we don't have to
write a driver for each board's connector design. The driver would have
to forward all kinds of regulator, GPIO, interrupt, ... requests it
receives at one node, on to the node that's listed in the other
connector node. Kinda like "ranges" but for arbitrary resources, not
just memory maps, in a way.

[U-Boot] Merging device trees at runtime for module-based systems

[Daniel Mack]

On 01.11.2012 00:13, Stephen Warren wrote:
> On 10/31/2012 05:00 PM, Daniel Mack wrote:
>> cc devicetree-discuss. Here's a reference to the full thread:
>>
>>   http://thread.gmane.org/gmane.comp.boot-loaders.u-boot/145221/
>>
>> On 26.10.2012 20:39, Stephen Warren wrote:
>>> On 10/24/2012 03:47 AM, Daniel Mack wrote:
>>>> Hi,
>>>>
>>>> a project I'm involved in uses a module/baseboard combo, and components
>>>> on either board are described in DT. I'm currently using separate dts
>>>> files which build upon each other with include statements, which works
>>>> fine for development.
>>>>
>>>> In production though, we will certainly have running changes (and hence
>>>> different versions) over the lifetime of the product for both the
>>>> baseboard and the module, and the hardware has support for identifying
>>>> the versions of both sides at runtime.
>>>>
>>>> So let's say we have n versions of the baseboard and m versions of the
>>>> module, we would much like to only prepare n + m files, instead of n * m
>>>> by pre-compiling every possible combination (some of which may actually
>>>> never occur 'in the wild').
>>>>
>>>> So my question is: is it possible to do that kind of assembly of a
>>>> number of files at runtime in U-Boot? I guess all it takes is merging a
>>>> number of trees together, right? I browsed through the APIs but couldn't
>>>> yet find an clear approach to that kind of problem. If not, what would
>>>> it take to add that functionality? I can probably help with the
>>>> implementation if someone tells me what would be the right way.
>>>
>>> Yes, solving this would be very useful; it's a wide-spread problem.
>>>
>>> Some thoughts though:
>>>
>>> Simply overlaying two DTBs on top of each-other (in the same fashion
>>> that dtc's /include/ statement would do at compile-time) might not be
>>> fully general enough, although perhaps it would be sufficient for your
>>> immediate needs.
>>>
>>> For example, lets say that a GPIO is routed from a device on the main
>>> board to a device on a daughter board, or even from one daughter board
>>> into the main board and back out to a different daughter board. Now,
>>> consider that the different board(s) that are the source of the GPIO
>>> might use completely different SoCs or versions of the SoC, which might
>>> require using a different GPIO specifier to represent the signal. That
>>> means you need to change the .dtb file for the "client" of the GPIO
>>> depending on the HW or .dtb that provides the GPIO. That's certainly not
>>> a simple matter of merging multiple .dtb blobs together.
>>
>> Hmm. After implementing a very simple overlay approach, I can now see
>> your point :) Yes in fact, that's a real problem.
>>
>>> The same issue could easily apply to I2C or SPI buses, chip selects, etc.
>>>
>>> One solution would be to explicitly represent a connector or
>>> connection-point in DT, such that the connector can implement the naming
>>> of all signals that pass through it, and provide a translation point for
>>> hooking the two DT fragments together. This seems within the spirit of DT.
>>
>> Yes, but you still can't handle references that way.
>>
>> Let me try and conclude this for others. Say the "module" tree "A" looks
>> something like this:
>>
>> 	/ {
>> 		multi-regulator {
>> 			vcc1v8: regulator at 0 {
>> 				/* ... */
>> 			};
>> 		};
>> 	};
>>
>> ... and the baseboard ("B"), that makes use of (and hence depends on)
>> the module, has something like this:
>>
>> 	/ {
>> 		consumer {
>> 			main-supply = <&vcc1v8>;
>> 		};
>> 	};
>>
>> Now, let's say in a subsequent version of the module, we change whatever
>> provides that supply for 1.8 volts, but the consumer on the baseboard
>> shouldn't care much of course, thanks to all the abstraction layers that
>> we have now in the kernel.
>>
>> However, the problem here is that I can't just compile trees A and B
>> individually into .dtbs that get merged later, because dtc will bail on
>> the unresolved reference of &vcc1v8 of course. And cases like this are
>> the whole reason why I started to think about modularization of trees in
>> the first place.
>>
>> So the simple overlay method doesn't help here at all, even though I can
>> share the code if anyone's interested.
> 
> Yes, you've understood me exactly.
> 
> The connector-base approach I was thinking about might look (very very)
> roughly as follows:
> 
> main board:
> 
> 	/ {
> 		multi-regulator {
> 			vcc1v8: regulator at 0 {
> 				/* ... */
> 			};
> 		};
> 		connector {
> 			compatible = "vendor,board-socket-a";
> 			vcc1v8-supply = <&vcc1v8>;
> 		};
> 	};
> 
> child board:
> 
> 	/ {
> 		connector {
> 			compatible = "vendor,board-plug-a";
> 			vcc1v8: regulator {
> 			};
> 		};
> 		consumer {
> 			main-supply = <&vcc1v8>;
> 		};
> 	};

... which doesn't eally make the individual bits more readable.

> ... plus some logic so that the "driver"s for the two connector nodes
> get linked together, such that the code forwards "requests" for the
> regulator that the plug receives on to the node for the socket, which
> then lists the actual provider.
> 
> Obviously, the above DT is an extremely rough sketch, and we need to
> think about:
> 
> a) Exactly how the plug/socket get linked together.
> 
> b) Can we make a generic driver for the plug/socket, so we don't have to
> write a driver for each board's connector design. The driver would have
> to forward all kinds of regulator, GPIO, interrupt, ... requests it
> receives at one node, on to the node that's listed in the other
> connector node. Kinda like "ranges" but for arbitrary resources, not
> just memory maps, in a way.

I really wonder if that's not fixing the wrong end after all. I mean, in
dts files and using includes, all this works just beautifully, and what
I really want is the same kind of flexibility in the bootloader.

Any good reason for not doing the dt complilation at runtime? For the
include issue, one could argue that /include/ statements are not
allowed, but the same behaviour can be achieved by simply concatinating
all the dts files right away.


Daniel

[U-Boot] Merging device trees at runtime for module-based systems

[Mitch Bradley]

On 10/31/2012 1:00 PM, Daniel Mack wrote:
> cc devicetree-discuss. Here's a reference to the full thread:
> 
>   http://thread.gmane.org/gmane.comp.boot-loaders.u-boot/145221/
> 
> On 26.10.2012 20:39, Stephen Warren wrote:
>> On 10/24/2012 03:47 AM, Daniel Mack wrote:
>>> Hi,
>>>
>>> a project I'm involved in uses a module/baseboard combo, and components
>>> on either board are described in DT. I'm currently using separate dts
>>> files which build upon each other with include statements, which works
>>> fine for development.
>>>
>>> In production though, we will certainly have running changes (and hence
>>> different versions) over the lifetime of the product for both the
>>> baseboard and the module, and the hardware has support for identifying
>>> the versions of both sides at runtime.
>>>
>>> So let's say we have n versions of the baseboard and m versions of the
>>> module, we would much like to only prepare n + m files, instead of n * m
>>> by pre-compiling every possible combination (some of which may actually
>>> never occur 'in the wild').
>>>
>>> So my question is: is it possible to do that kind of assembly of a
>>> number of files at runtime in U-Boot? I guess all it takes is merging a
>>> number of trees together, right? I browsed through the APIs but couldn't
>>> yet find an clear approach to that kind of problem. If not, what would
>>> it take to add that functionality? I can probably help with the
>>> implementation if someone tells me what would be the right way.
>>
>> Yes, solving this would be very useful; it's a wide-spread problem.
>>
>> Some thoughts though:
>>
>> Simply overlaying two DTBs on top of each-other (in the same fashion
>> that dtc's /include/ statement would do at compile-time) might not be
>> fully general enough, although perhaps it would be sufficient for your
>> immediate needs.
>>
>> For example, lets say that a GPIO is routed from a device on the main
>> board to a device on a daughter board, or even from one daughter board
>> into the main board and back out to a different daughter board. Now,
>> consider that the different board(s) that are the source of the GPIO
>> might use completely different SoCs or versions of the SoC, which might
>> require using a different GPIO specifier to represent the signal. That
>> means you need to change the .dtb file for the "client" of the GPIO
>> depending on the HW or .dtb that provides the GPIO. That's certainly not
>> a simple matter of merging multiple .dtb blobs together.
> 
> Hmm. After implementing a very simple overlay approach, I can now see
> your point :) Yes in fact, that's a real problem.
> 
>> The same issue could easily apply to I2C or SPI buses, chip selects, etc.
>>
>> One solution would be to explicitly represent a connector or
>> connection-point in DT, such that the connector can implement the naming
>> of all signals that pass through it, and provide a translation point for
>> hooking the two DT fragments together. This seems within the spirit of DT.
> 
> Yes, but you still can't handle references that way.
> 
> Let me try and conclude this for others. Say the "module" tree "A" looks
> something like this:
> 
> 	/ {
> 		multi-regulator {
> 			vcc1v8: regulator at 0 {
> 				/* ... */
> 			};
> 		};
> 	};
> 
> ... and the baseboard ("B"), that makes use of (and hence depends on)
> the module, has something like this:
> 
> 	/ {
> 		consumer {
> 			main-supply = <&vcc1v8>;
> 		};
> 	};
> 
> Now, let's say in a subsequent version of the module, we change whatever
> provides that supply for 1.8 volts, but the consumer on the baseboard
> shouldn't care much of course, thanks to all the abstraction layers that
> we have now in the kernel.
> 
> However, the problem here is that I can't just compile trees A and B
> individually into .dtbs that get merged later, because dtc will bail on
> the unresolved reference of &vcc1v8 of course. And cases like this are
> the whole reason why I started to think about modularization of trees in
> the first place.
> 
> So the simple overlay method doesn't help here at all, even though I can
> share the code if anyone's interested.
> 
>> Another solution might be some form of variables/macros/code in the DTB
>> that can be used to parameterize other DTBs that get merged with it.
>> This is probably an enormous can of worms.
> 
> Yes, exactly, a can of worms and most probably unmaintainble in real
> life. I start to believe that the cleanest solution to this would be to
> have full DTC functionality in U-Boot and compile the tree

... which is exactly the way that Open Firmware does it, since the
invention of the device tree.  The model is that the boot firmware,
which needs to know the system configuration to do its job anyway,
exports that configuration via the device tree.

 from dts, but
> then again I have no clue on how to handle the file lookups that arise
> from includes. Do you think it would it be worth going that way?
> 
> If not, I guess we're down to n*m files eventually, which is really sad
> as they might even become a storage problem at some point.
> 
> 
> Thanks for your input,
> Daniel
> 
> 
> _______________________________________________
> devicetree-discuss mailing list
> devicetree-discuss at lists.ozlabs.org
> https://lists.ozlabs.org/listinfo/devicetree-discuss
> 

[U-Boot] Merging device trees at runtime for module-based systems

[David Gibson]

On Fri, Oct 26, 2012 at 09:24:11AM +0200, Daniel Mack wrote:
> On 26.10.2012 02:53, David Gibson wrote:
> > On Thu, Oct 25, 2012 at 10:46:32PM +0200, Wolfgang Denk wrote:
> >> Dear Daniel,
> >>
> >> In message <50893633.6070408@gmail.com> you wrote:
> >>>
> >>> Overwrites must be addressed in the first place. The most common example
> >>> is that a more generic part (the module tree) registers all details
> >>> about a peripheral up-front but then sets its status to 'disabled'. That
> >>> way, the more specific part (the base board tree) can overwrite this
> >>> property to 'okay' at wish to enable it and not care for the pre-defined
> >>> details. This is also how we do things in our device-trees.
> >>
> >> Agreed.
> >>
> >>>> I definitely can see the benefit of such a feature and would be happy
> >>>> if you could go forward and implement it.
> >>>
> >>> Ok then. I guess this should be something that can eventually be merged
> >>> back into libfdt?
> >>
> >> I can't speak for the FDT custodian, but I think this makes a lot of
> >> sense.
> > 
> > As a rule I'm happy to see more functionality for libfdt.  I've only
> > seen bits and pieces of this thread, though, so I'd need to see a
> > summary of what exactly is being proposed.
> 
> That's strange, as I copied you from the very first posting. Anyway,
> here's the archive:

Oh I probably got them somewhere in my mail.  But it's only recently
that I really noticed - I get a fair bit of mail.

>   http://lists.denx.de/pipermail/u-boot/2012-October/138227.html
> 
> I would especially like to know where such a new functionality should
> live, which data types it should operate on and what would be an
> appropriate name for it.

So.. the first thought I have reading the original mail in the thread
is that it's arguable that you really want a more heavyweight firmware
for this setup, that actively maintains a live device tree as OF does,
rather than u-boot which is pretty oriented towards a close-to-static
device setup.  That's just a thought though, I'm not saying that at
least some of this functionality doesn't belong in libfdt.

So, my thought would be that stuff for manipulating big chunks of tree
should go in a new .c file inside the libfdt tree.  We already have
del_node and nop_node of course, which can remove whole subtrees.  I
guess the big extra function you'd want would be something like:

fdt_graft(void *fdt, int offset, void *subtree);

Which would graft the tree blob give by subtree into the "master" tree
(fdt) at node 'offset'.  Actually that might need to take a name for
the top-level of the subtree to take in the new tree too.

Things get trickier when you consider what might need to be tweaked in
the subtree to make it fit into the master tree.  If it requires
widespread alterations through the subtree that's going to get really
ugly and I think you would be better off with a firmware with a fuller
handling of a "live" device tree.  But I think that can probably be
avoided with proper design of the bindings.

To get that to work you'll need to make sure you use some sort of
local addressing within the subtree.  Then it should only be necessary
to insert/alter a "ranges" property at the top level of the subtree
(or possibly its parent) to map that correctly into the global address
space.  Likewise interrupts within the subtree probably shouldn't
address an external interrupt controller but rather the root of the
tree.  You can then insert an "interrupt-map" property which will
wire those into the global interrupt tree.

-- 
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson

[U-Boot] Merging device trees at runtime for module-based systems

[Stephen Warren]

On 10/31/2012 05:56 PM, Mitch Bradley wrote:
> On 10/31/2012 1:00 PM, Daniel Mack wrote:
>> cc devicetree-discuss. Here's a reference to the full thread:
>>
>>   http://thread.gmane.org/gmane.comp.boot-loaders.u-boot/145221/
>>
>> On 26.10.2012 20:39, Stephen Warren wrote:
>>> On 10/24/2012 03:47 AM, Daniel Mack wrote:
>>>> Hi,
>>>>
>>>> a project I'm involved in uses a module/baseboard combo, and components
>>>> on either board are described in DT. I'm currently using separate dts
>>>> files which build upon each other with include statements, which works
>>>> fine for development.
>>>>
>>>> In production though, we will certainly have running changes (and hence
>>>> different versions) over the lifetime of the product for both the
>>>> baseboard and the module, and the hardware has support for identifying
>>>> the versions of both sides at runtime.
...
>> I start to believe that the cleanest solution to this would be to
>> have full DTC functionality in U-Boot and compile the tree
> 
> ... which is exactly the way that Open Firmware does it, since the
> invention of the device tree.  The model is that the boot firmware,
> which needs to know the system configuration to do its job anyway,
> exports that configuration via the device tree.

Doesn't OF generate the DT from internal data structures (although I
don't know where those come from...), whereas what Daniel mentions above
is more like the bootloader having access to a bunch of .dts fragments,
selecting the appropriate subset of those to use, parsing them into an
internal data structure (i.e. running dtc), and then generating a DTB
from it. The overall result is that the bootloader causes a DTB to be
generated at run-time, so at that level it's the same, but the
implementation seems pretty different.

[U-Boot] Merging device trees at runtime for module-based systems

[Mitch Bradley]

On 10/31/2012 6:36 PM, Stephen Warren wrote:
> On 10/31/2012 05:56 PM, Mitch Bradley wrote:
>> On 10/31/2012 1:00 PM, Daniel Mack wrote:
>>> cc devicetree-discuss. Here's a reference to the full thread:
>>>
>>>   http://thread.gmane.org/gmane.comp.boot-loaders.u-boot/145221/
>>>
>>> On 26.10.2012 20:39, Stephen Warren wrote:
>>>> On 10/24/2012 03:47 AM, Daniel Mack wrote:
>>>>> Hi,
>>>>>
>>>>> a project I'm involved in uses a module/baseboard combo, and components
>>>>> on either board are described in DT. I'm currently using separate dts
>>>>> files which build upon each other with include statements, which works
>>>>> fine for development.
>>>>>
>>>>> In production though, we will certainly have running changes (and hence
>>>>> different versions) over the lifetime of the product for both the
>>>>> baseboard and the module, and the hardware has support for identifying
>>>>> the versions of both sides at runtime.
> ...
>>> I start to believe that the cleanest solution to this would be to
>>> have full DTC functionality in U-Boot and compile the tree
>>
>> ... which is exactly the way that Open Firmware does it, since the
>> invention of the device tree.  The model is that the boot firmware,
>> which needs to know the system configuration to do its job anyway,
>> exports that configuration via the device tree.
> 
> Doesn't OF generate the DT from internal data structures (although I
> don't know where those come from...), whereas what Daniel mentions above
> is more like the bootloader having access to a bunch of .dts fragments,
> selecting the appropriate subset of those to use, parsing them into an
> internal data structure (i.e. running dtc), and then generating a DTB
> from it. The overall result is that the bootloader causes a DTB to be
> generated at run-time, so at that level it's the same, but the
> implementation seems pretty different.

Yes, which is why I cut the cited sentence at the place I did (before
the part about dts fragments).

> 

[U-Boot] Merging device trees at runtime for module-based systems

[Daniel Mack]

On 01.11.2012 04:26, David Gibson wrote:
> On Fri, Oct 26, 2012 at 09:24:11AM +0200, Daniel Mack wrote:

>> I would especially like to know where such a new functionality should
>> live, which data types it should operate on and what would be an
>> appropriate name for it.
> 
> So.. the first thought I have reading the original mail in the thread
> is that it's arguable that you really want a more heavyweight firmware
> for this setup, that actively maintains a live device tree as OF does,
> rather than u-boot which is pretty oriented towards a close-to-static
> device setup.  That's just a thought though, I'm not saying that at
> least some of this functionality doesn't belong in libfdt.
> 
> So, my thought would be that stuff for manipulating big chunks of tree
> should go in a new .c file inside the libfdt tree.  We already have
> del_node and nop_node of course, which can remove whole subtrees.  I
> guess the big extra function you'd want would be something like:
> 
> fdt_graft(void *fdt, int offset, void *subtree);
> 
> Which would graft the tree blob give by subtree into the "master" tree
> (fdt) at node 'offset'.  Actually that might need to take a name for
> the top-level of the subtree to take in the new tree too.

I called the function fdt_overlay, but I guess the implementation is
similar to what you thought of. I pushed it here, see the topmost 3 commits:

  https://github.com/zonque/dtc/commits/overlay

> Things get trickier when you consider what might need to be tweaked in
> the subtree to make it fit into the master tree.  If it requires
> widespread alterations through the subtree that's going to get really
> ugly and I think you would be better off with a firmware with a fuller
> handling of a "live" device tree.  But I think that can probably be
> avoided with proper design of the bindings.
> 
> To get that to work you'll need to make sure you use some sort of
> local addressing within the subtree.  Then it should only be necessary
> to insert/alter a "ranges" property at the top level of the subtree
> (or possibly its parent) to map that correctly into the global address
> space.  Likewise interrupts within the subtree probably shouldn't
> address an external interrupt controller but rather the root of the
> tree.  You can then insert an "interrupt-map" property which will
> wire those into the global interrupt tree.

As pointed out on another end of this thread, the use of my simple
implementation is rather limited. I need to think about something more
sophisticated, or abadon the idea alltogether.


Thanks,
Daniel

[U-Boot] Merging device trees at runtime for module-based systems

[David Gibson]

On Wed, Oct 31, 2012 at 10:36:08PM -0600, Stephen Warren wrote:
> On 10/31/2012 05:56 PM, Mitch Bradley wrote:
> > On 10/31/2012 1:00 PM, Daniel Mack wrote:
> >> cc devicetree-discuss. Here's a reference to the full thread:
> >>
> >>   http://thread.gmane.org/gmane.comp.boot-loaders.u-boot/145221/
> >>
> >> On 26.10.2012 20:39, Stephen Warren wrote:
> >>> On 10/24/2012 03:47 AM, Daniel Mack wrote:
> >>>> Hi,
> >>>>
> >>>> a project I'm involved in uses a module/baseboard combo, and components
> >>>> on either board are described in DT. I'm currently using separate dts
> >>>> files which build upon each other with include statements, which works
> >>>> fine for development.
> >>>>
> >>>> In production though, we will certainly have running changes (and hence
> >>>> different versions) over the lifetime of the product for both the
> >>>> baseboard and the module, and the hardware has support for identifying
> >>>> the versions of both sides at runtime.
> ...
> >> I start to believe that the cleanest solution to this would be to
> >> have full DTC functionality in U-Boot and compile the tree
> > 
> > ... which is exactly the way that Open Firmware does it, since the
> > invention of the device tree.  The model is that the boot firmware,
> > which needs to know the system configuration to do its job anyway,
> > exports that configuration via the device tree.
> 
> Doesn't OF generate the DT from internal data structures (although I
> don't know where those come from...),

Well.. in OF the device tree *is* a core live data structure.  It will
be constructed as devices are probed, firmware level device drivers
are attached to it and it might be modified by client interface
operations.  Traditionally there is no tree in the flattened format,
only the live tree which clients will query with OF calls.  Some
recent OF implementations do use the flat tree in some ways - we've
had some systems where a flattened tree is provided by an early boot
to the full OF; it acts as a skeleton that is then extended in OF's
live tree.  We've also had some OF implementations that for
compatiblity with kernels that expect a flattened tree flatten their
internal tree structure into the FDT format as the last thing before
entering the OS.

> whereas what Daniel mentions above
> is more like the bootloader having access to a bunch of .dts fragments,
> selecting the appropriate subset of those to use, parsing them into an
> internal data structure (i.e. running dtc), and then generating a DTB
> from it. The overall result is that the bootloader causes a DTB to be
> generated at run-time, so at that level it's the same, but the
> implementation seems pretty different.

How OF constructs its internal tree can vary a lot with the OF
implementation, and I'm not terribly clear on what you had in mind, so
I don't think either side is really sufficiently well defined to
really say if they're similar or not.

-- 
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson

[U-Boot] Merging device trees at runtime for module-based systems

[David Gibson]

On Thu, Nov 01, 2012 at 10:24:06AM +0100, Daniel Mack wrote:
> On 01.11.2012 04:26, David Gibson wrote:
> > On Fri, Oct 26, 2012 at 09:24:11AM +0200, Daniel Mack wrote:
> 
> >> I would especially like to know where such a new functionality should
> >> live, which data types it should operate on and what would be an
> >> appropriate name for it.
> > 
> > So.. the first thought I have reading the original mail in the thread
> > is that it's arguable that you really want a more heavyweight firmware
> > for this setup, that actively maintains a live device tree as OF does,
> > rather than u-boot which is pretty oriented towards a close-to-static
> > device setup.  That's just a thought though, I'm not saying that at
> > least some of this functionality doesn't belong in libfdt.
> > 
> > So, my thought would be that stuff for manipulating big chunks of tree
> > should go in a new .c file inside the libfdt tree.  We already have
> > del_node and nop_node of course, which can remove whole subtrees.  I
> > guess the big extra function you'd want would be something like:
> > 
> > fdt_graft(void *fdt, int offset, void *subtree);
> > 
> > Which would graft the tree blob give by subtree into the "master" tree
> > (fdt) at node 'offset'.  Actually that might need to take a name for
> > the top-level of the subtree to take in the new tree too.
> 
> I called the function fdt_overlay, but I guess the implementation is
> similar to what you thought of. I pushed it here, see the topmost 3 commits:
> 
>   https://github.com/zonque/dtc/commits/overlay

Interesting.  So, it seems to me that fdt_graft() and fdt_overlay()
are different operations - both could be potentially useful.
fdt_graft() would attach a new subtree somewhere within the master
tree, with the assumption that the root of the subtree would become a
new node in the resulting tree.  Overwriting an existing subtree with
a new one would be an error for a graft.  fdt_overlay, as you've
implemented, can either add new nodes or modify existing ones by
replacing or adding new properties.

So, some notes on the actual implementation:

The in-place modification of the given path (which should really be
const char *) in your fdt_add_subnode_r() is nasty, nasty, nasty.  And
it's unnecessary because you can use the existing
fdt_add_subnode_namelen() to work with subsections of the path without
needing to either have a temporary buffer or do in-place modification.

...except, I don't think you actually need fdt_add_subnoode_r() for
your overlay implementation in any case.

AFAICT in your fdt_overlay() implementation you're only adding nodes
from the second tree if they contain properties (the
fdt_add_subnode_r() call is under the FDT_PROP case).  I'm not sure if
that was a deliberate policy decision - if so I really can't see a
reason for it.

If instead you *always* add subnodes when they exist in the second
tree, you'll be doing your add nodes from the FDT_BEGIN_NODE tag
case.  And you always get BEGIN_NODE tags for parents before subnodes,
so you can naturally add your new subnode path component by component
without having to walk down the path again in fdt_add_subnode_r().  As
an added bonus you no longer need pathbuf and it's arbitrary size
limit.

Hrm.. wait... I guess you need a stack so you can handle FDT_END_NODE
correctly.  I suspect a recursive solution (effectively using the
machine stack) would still take less (machine) stack space than
pathbuf.  Especially if pathbuf was increased up to PATH_MAX, which is
my usual rule of thumb when I can't avoid an arbitrary buffer size.	


On a tangent, note that fdt_graft() as defined above, unlike
fdt_overlay() would allow a considerably optimized implementation.
Instead of doing lots of individual inserts into the tree (and
therefore a lot of memmove()s), you could do one big _fdt_splice(),
copy in the grafted tree's structure block, then run through it
correcting property name offsets.

-- 
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson

[U-Boot] Merging device trees at runtime for module-based systems

[Daniel Mack]

On 03.11.2012 16:25, David Gibson wrote:
> On Thu, Nov 01, 2012 at 10:24:06AM +0100, Daniel Mack wrote:
>> On 01.11.2012 04:26, David Gibson wrote:
>>> On Fri, Oct 26, 2012 at 09:24:11AM +0200, Daniel Mack wrote:
>>
>>>> I would especially like to know where such a new functionality should
>>>> live, which data types it should operate on and what would be an
>>>> appropriate name for it.
>>>
>>> So.. the first thought I have reading the original mail in the thread
>>> is that it's arguable that you really want a more heavyweight firmware
>>> for this setup, that actively maintains a live device tree as OF does,
>>> rather than u-boot which is pretty oriented towards a close-to-static
>>> device setup.  That's just a thought though, I'm not saying that at
>>> least some of this functionality doesn't belong in libfdt.
>>>
>>> So, my thought would be that stuff for manipulating big chunks of tree
>>> should go in a new .c file inside the libfdt tree.  We already have
>>> del_node and nop_node of course, which can remove whole subtrees.  I
>>> guess the big extra function you'd want would be something like:
>>>
>>> fdt_graft(void *fdt, int offset, void *subtree);
>>>
>>> Which would graft the tree blob give by subtree into the "master" tree
>>> (fdt) at node 'offset'.  Actually that might need to take a name for
>>> the top-level of the subtree to take in the new tree too.
>>
>> I called the function fdt_overlay, but I guess the implementation is
>> similar to what you thought of. I pushed it here, see the topmost 3 commits:
>>
>>   https://github.com/zonque/dtc/commits/overlay
> 
> Interesting.  So, it seems to me that fdt_graft() and fdt_overlay()
> are different operations - both could be potentially useful.
> fdt_graft() would attach a new subtree somewhere within the master
> tree, with the assumption that the root of the subtree would become a
> new node in the resulting tree.  Overwriting an existing subtree with
> a new one would be an error for a graft.  fdt_overlay, as you've
> implemented, can either add new nodes or modify existing ones by
> replacing or adding new properties.
> 
> So, some notes on the actual implementation:
> 
> The in-place modification of the given path (which should really be
> const char *) in your fdt_add_subnode_r() is nasty, nasty, nasty.  And
> it's unnecessary because you can use the existing
> fdt_add_subnode_namelen() to work with subsections of the path without
> needing to either have a temporary buffer or do in-place modification.

Ok - thanks for the hint. I was looking for a nicer way as I also was
really unhappy about the implementation, but without access to heap
operations and avoid eating up too much stack, I thought this is a nasty
but working solution.

After all, that whole hacking was just meant as a proof of concept for
the result, which failed ;)

> ...except, I don't think you actually need fdt_add_subnoode_r() for
> your overlay implementation in any case.
> 
> AFAICT in your fdt_overlay() implementation you're only adding nodes
> from the second tree if they contain properties (the
> fdt_add_subnode_r() call is under the FDT_PROP case).  I'm not sure if
> that was a deliberate policy decision - if so I really can't see a
> reason for it.

That was merely to spare some cycles, as fdt_add_subnode_r() turned out
not to be free of costs.

> If instead you *always* add subnodes when they exist in the second
> tree, you'll be doing your add nodes from the FDT_BEGIN_NODE tag
> case.  And you always get BEGIN_NODE tags for parents before subnodes,
> so you can naturally add your new subnode path component by component
> without having to walk down the path again in fdt_add_subnode_r().  As
> an added bonus you no longer need pathbuf and it's arbitrary size
> limit.

Yes, I had something like that in mind, but that requires keeping the
state of both trees around and care for the stacking etc.

> Hrm.. wait... I guess you need a stack so you can handle FDT_END_NODE
> correctly.  I suspect a recursive solution (effectively using the
> machine stack) would still take less (machine) stack space than
> pathbuf.  Especially if pathbuf was increased up to PATH_MAX, which is
> my usual rule of thumb when I can't avoid an arbitrary buffer size.	
> 
> 
> On a tangent, note that fdt_graft() as defined above, unlike
> fdt_overlay() would allow a considerably optimized implementation.
> Instead of doing lots of individual inserts into the tree (and
> therefore a lot of memmove()s), you could do one big _fdt_splice(),
> copy in the grafted tree's structure block, then run through it
> correcting property name offsets.

Given the fact that this implementation turns out to be inappropriate as
solution to my actual problem, I'm uncertain whether it should be merged
at all. Dunno of how much use that would be for others after all.


Many thanks for you input,
Daniel

[U-Boot] Merging device trees at runtime for module-based systems

[Grant Likely]

On Wed, Oct 31, 2012 at 11:00 PM, Daniel Mack <zonque@gmail.com> wrote:
> cc devicetree-discuss. Here's a reference to the full thread:
>
>   http://thread.gmane.org/gmane.comp.boot-loaders.u-boot/145221/

Interesting. I only just was made aware of this thread. There is a
similar discussion going on kicked off by the BeagleBone folks where
they want to insert additional DT data from Linux userspace. Whether
the data is merged at U-Boot time or kernel time, I expect that the
required data format will be very similar.

https://lkml.org/lkml/2012/10/31/502
https://lkml.org/lkml/2012/11/5/615

I'm trying to draft up a document that captures the requirements and
lay out what needs to be done to the tools, U-Boot and the kernel.

g.