Re: [PATCH] devicetree: Enable generation of __symbols__ in all dtb files

[Frank Rowand <frowand.list@gmail.com>]

On 08/16/17 02:42, Pantelis Antoniou wrote:
> Hi Frank,
> 
>> On Aug 16, 2017, at 02:57 , Frank Rowand <frowand.list@gmail.com> wrote:
>>
>> On 08/15/17 15:36, Rob Herring wrote:
>>> On Tue, Aug 15, 2017 at 4:15 PM, Tom Rini <trini@konsulko.com> wrote:
>>>> With support for stacked overlays being part of libfdt it is now
>>>> possible and likely that overlays which require __symbols__ will be
>>>> applied to the dtb files generated by the kernel.  This is done by
>>>> passing -@ to dtc.  This does increase the filesize (and resident memory
>>>> usage) based on the number of __symbol__ entries added to match the
>>>> contents of the dts.
>>>>
>>>> Cc: Rob Herring <robh+dt@kernel.org>
>>>> Cc: Frank Rowand <frowand.list@gmail.com>
>>>> Cc: Masahiro Yamada <yamada.masahiro@socionext.com>
>>>> Cc: Michal Marek <mmarek@suse.com>
>>>> Cc: Pantelis Antoniou <pantelis.antoniou@konsulko.com>
>>>> Cc: devicetree@vger.kernel.org
>>>> Cc: linux-kernel@vger.kernel.org
>>>> CC: linux-kbuild@vger.kernel.org
>>>> Signed-off-by: Tom Rini <trini@konsulko.com>
>>>> ---
>>>> In order for a dtb file to be useful with all types of overlays, it
>>>> needs to be generated with the -@ flag passed to dtc so that __symbols__
>>>> are generated.  This however is not free, and increases the resulting
>>>> dtb file by up to approximately 50% today.  In the current worst case
>>>> this is moving from 88KiB to 133KiB.  In talking with Frank about this,
>>>
>>> Plus some amount for the unflattened tree in memory, too.
>>>
>>>> he outlined 3 possible ways (with the 4th option of something else
>>>> entirely).
>>>>
>>>> 1. Make passing -@ to dtc be dependent upon some CONFIG symbol.
>>>> 2. In the kernel, if the kernel does not have overlay support, discard
>>>> the __symbols__ information that we've been passed.
>>>> 3. Have the bootloader pass in, or not, __symbols__ information.
>>>>
>>>> This patch is an attempt to implement something between the 3rd option
>>>> and a different, 4th option.  Frank was thinking that we might introduce
>>>> a new symbol to control generation of __symbol__ information for option
>>>> 1.  I think this gets the usage backwards and will lead to confusion
>>>> among users and developers.
>>>>
>>>> My proposal is that we do not want __symbols__ existence to be dependent
>>>> on some part of the kernel configuration for a number of reasons.
>>>> First, this is out of step with the rest of how dtbs are created today
>>>> and more importantly, thought about.  Today, all dtb content is
>>>> independent of CONFIG options.  If you build a dtb from a given kernel
>>>> tree, everyone will agree on the result.  This is part of the "contract"
>>>> on passing old kernels and new dtb files even.
>>>
>>> Agree completely. I don't even like that building dtbs depends on the ARCH.
>>>
>>> However, option 2 may still be useful. There's no point exposing what
>>> can't be used. Furthermore, exposing __symbols__ in /proc/device-tree
>>> at all may be a bad idea. We should consider if it should always be
>>> hidden. That would also allow storing the __symbols__ data however we
>>> want internally (i.e. with less memory usage).
>>
>> Yes.  I would prefer to treat the __symbols__ node as an internal
>> representation of information used by the device tree subsystem.
>> It is not hardware description.
>>
>>
> 
> This is correct. This is internal workaround against a serialization format
> omission.
> 
>>> The complication is
>>> always kexec which I haven't thought about too much here.
>>
>> That should not be an issue, because the device tree is exposed to kexec
>> via /sys/firmware/fdt instead of /sys/firmware/devicetree/base (which
>> is what /proc/device-tree links to), according to
>> Documentation/ABI/testing/sysfs-firmware-ofw.  So the __symbols__
>> node will be exposed to kexec.
>>
> 
> Which will have to be recreated if we throw away __symbols__ when converting
> to our internal representation of labels.

Nope.  /sys/firmware/fdt is the fdt that is passed to the kernel.  We are
not proposing any changes to that fdt by the kernel.


>>
>>> Also, perhaps we need finer grain control of __symbols__ generation.
>>> We really don't want userspace to be able to modify anything in the DT
>>> at any point in time. That's a big can of worms and we don't want to
>>> start there. The problem is labels are widely used just for
>>> convenience and weren't part of the ABI. With overlays that changes,
>>> so we either need to restrict labels usage or define another way. It
>>> could be as simple as defining some prefix for label names for labels
>>> to export.
>>
>> Agreed.  We could also restrict labels in connector nodes to be visible.
>>
> 
> I would disagree. This is only being considered because runtime device tree
> consistency checks currently is minimal (i.e. non existent). When we have
> a proper DT syntax and semantic checker (soon I suppose) this restriction 
> will be useless and make things more complex.
> 
> Regards
> 
> — Pantelis
> 
>>
>>>> Second, I think this is out of step with how a lot of overlay usage will
>>>> occur.  My thinking is that with maximally useful overlays being
>>>> available in mainline, lots of use-cases that we have today that result
>>>> in a number of DTS files being included can become just overlays.  This
>>>> is true in terms of not only evaluation kits but also when these systems
>>>> are turned into custom hardware.  This is even more true for SoM based
>>>> systems where a physical widget would be a SoM + carrier overlay +
>>>> custom parts overlay.  These cases are going to be resolved with
>>>> overlays being applied outside of the kernel.
>>>>
>>>> Signed-off-by: Tom Rini <trini@konsulko.com>
>>>> ---
>>>> drivers/of/unittest-data/Makefile | 5 -----
>>>> scripts/Makefile.lib              | 3 +++
>>>> 2 files changed, 3 insertions(+), 5 deletions(-)
>>>>
>>>> diff --git a/drivers/of/unittest-data/Makefile b/drivers/of/unittest-data/Makefile
>>>> index 6e00a9c69e58..70731cfe8900 100644
>>>> --- a/drivers/of/unittest-data/Makefile
>>>> +++ b/drivers/of/unittest-data/Makefile
>>>> @@ -11,8 +11,3 @@ targets += overlay_base.dtb overlay_base.dtb.S
>>>> .PRECIOUS: \
>>>>        $(obj)/%.dtb.S \
>>>>        $(obj)/%.dtb
>>>> -
>>>> -# enable creation of __symbols__ node
>>>> -DTC_FLAGS_overlay := -@
>>>> -DTC_FLAGS_overlay_bad_phandle := -@
>>>> -DTC_FLAGS_overlay_base := -@
>>>> diff --git a/scripts/Makefile.lib b/scripts/Makefile.lib
>>>> index 58c05e5d9870..a1f4a6b29d75 100644
>>>> --- a/scripts/Makefile.lib
>>>> +++ b/scripts/Makefile.lib
>>>> @@ -293,6 +293,9 @@ DTC_FLAGS += -Wnode_name_chars_strict \
>>>>        -Wproperty_name_chars_strict
>>>> endif
>>>>
>>>> +# enable creation of __symbols__ node
>>>> +DTC_FLAGS += -@
>>>> +
>>>> DTC_FLAGS += $(DTC_FLAGS_$(basetarget))
>>>>
>>>> # Generate an assembly file to wrap the output of the device tree compiler
>>>> --
>>>> 1.9.1
>>>>
>>> .
> 
>