Re: [RFC PATCH v3 0/4] dtc: Add a plugin interface

[Andrei Ziureaev <andreiziureaev-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>]

On 11/09/2020 08:06, David Gibson wrote:
> On Sun, Sep 06, 2020 at 02:12:16PM +0100, Andrei Ziureaev wrote:
>> I added a simple live tree API. There's no longer a need to move any
>> definitions. The live tree struct is hidden from plugins, so it's not an ABI.
>> The API doesn't allow plugins to modify the live tree yet, but that will
>> probably change later.
> 
> Hrm.  So.  Before we get into any of the details of implementation, I
> think we need to be clear on what it means to expose the "live tree".
> The live tree in dtc was never designed to be an exposed structure, so
> we need to think about that carefully before we do expose it, because
> we now have to consider stability of it across releases which we never
> did before.

Thanks for the comments.

Unlike in previous versions, in this version I'm passing opaque pointers 
between DTC and the plugins. So,
"struct dt_info" and all the other related structs are not an ABI, but 
the new live tree "library" is. The signatures in "dt.h" need to be 
designed carefully.
> 
> AFAICT the fact the tree is "live" (i.e. a pointer-based random access
> representation) isn't really the relevant point here.  We could easily
> enough serialize/deserialize the tree to get it into plugins, using
> whatever format - and dtb is the obvious one.

Yes, we could use any format, such as dtb, yaml or json, but there 
should be access to line numbers and it should be fast.
> 
> What you're really after here is access to some of the internal
> metadata that dtc maintains about the tree: line numbers at least,
> possibly other things.
> 
> I think we really need to pin down *what* parts of this information
> we need to expose to the plugins.  The data model for the information
> that's going back and forth is more important than exactly how we
> encode the data across the boundary.

The way I came up with this live tree API is I looked at Rob's code that 
converts the tree to Python structures and calls DT Schema
(https://github.com/robherring/dtc/blob/lib-plugin/yamlchecks.c),
and I tried to see what data it needs to access. Turns out, it needs 
read-only access to at least:
- the file name, line number, node name and property name
- the value of the property (strings and ints)
- if the value is an int, its width and whether it's a phandle

So those are the functions I defined in "dt.h". I've added some docs to 
them, so I should probably send a v4 soon.
> 
>>
>> I also removed the EXPORT_FUNCTION macro. Plugins now have to implement
>> certain prototypes, defined in dtc-plugin.h. This allows plugins to be
>> generated by tools.
>> ===========
>>
>> This is one possible implementation of a plugin interface for DTC.
>>
>> A plugin interface would allow us to add checks in languages other than
>> C. Specifically, it would allow DTC and DT Schema (which is written in
>> Python) to be integrated.
>>
>> It would also allow for better debugging of dts files. For example, DT
>> Schema would be able to print dts source line information, without us
>> having to add source line annotations to yaml files.
>>
>> In the future, plugins will be allowed to modify the live tree and pass
>> it back for further processing, but the API doesn't support that yet.
>>
>> There's a question of whether we should relicense some headers to
>> dual BSD to be able call Python code.
>>
>> Any thoughts would be much appreciated.
>>
>> Thanks,
>> Andrei.
>>
>> Changes in v3:
>> - live tree API (dt.h and dt.c)
>> - additional functionality in dtc.h
>> - plugins have to implement prototypes
>> - improved documentation
>>
>> Changes in v2:
>> - improved documentation
>> - plugins must register with the build system
>> - the "validate_fn_t" hook can return a status
>> - define "struct reserve_info" in "dtc-plugin.h"
>> - specify that minor versions are compatible
>> - check if plugin_dir is NULL, just in case
>> - better variable names in register_plugin_info
>>
>> Andrei Ziureaev (4):
>>    dtc: Add marker type functionality to dtc.h
>>    dtc: Add a live tree API
>>    dtc: Add plugin documentation and examples
>>    dtc: Add a plugin interface
>>
>>   Documentation/manual.txt         | 146 ++++++++++++++++++++++++++++
>>   Makefile                         |  29 +++++-
>>   Makefile.dtc                     |   1 +
>>   dt.c                             | 158 +++++++++++++++++++++++++++++++
>>   dt.h                             |  63 ++++++++++++
>>   dtc-plugin.h                     |  76 +++++++++++++++
>>   dtc.c                            | 142 ++++++++++++++++++++++++++-
>>   dtc.h                            |  57 +++++++++++
>>   plugins/example/Makefile.example |  27 ++++++
>>   plugins/example/example.c        |  33 +++++++
>>   treesource.c                     |   8 +-
>>   11 files changed, 734 insertions(+), 6 deletions(-)
>>   create mode 100644 dt.c
>>   create mode 100644 dt.h
>>   create mode 100644 dtc-plugin.h
>>   create mode 100644 plugins/example/Makefile.example
>>   create mode 100644 plugins/example/example.c
>>
>