[RFC] yamldt v0.5, now a DTS compiler too

[RFC] yamldt v0.5, now a DTS compiler too

[Pantelis Antoniou]

Hello again,

Significant progress has been made on yamldt and is now capable of
not only generating yaml from DTS source but also compiling DTS sources
and being almost fully compatible with DTC.

Compiling the kernel's DTBs using yamldt is as simple as using a
DTC=yamldt.

Error reporting is accurate and validation against a YAML based schema
works as well. In a short while I will begin posting patches with
fixes on bindings and DTS files in the kernel.

Please try it on your platform and report if you encounter any problems.

https://github.com/pantoniou/yamldt

I am eagerly awaiting for your comments.

Regards

-- Pantelis

Re: [RFC] yamldt v0.5, now a DTS compiler too

[Rob Herring]

On Thu, Sep 28, 2017 at 2:58 PM, Pantelis Antoniou
<pantelis.antoniou-OWPKS81ov/FWk0Htik3J/w@public.gmane.org> wrote:
> Hello again,
>
> Significant progress has been made on yamldt and is now capable of
> not only generating yaml from DTS source but also compiling DTS sources
> and being almost fully compatible with DTC.

Can you quantify "almost"?

> Compiling the kernel's DTBs using yamldt is as simple as using a
> DTC=yamldt.

Good.

>
> Error reporting is accurate and validation against a YAML based schema
> works as well. In a short while I will begin posting patches with
> fixes on bindings and DTS files in the kernel.

What I would like to see is the schema format posted for review.

I would also like to see the bindings for top-level compatible strings
(aka boards) as an example. That's something that's simple enough that
I'd think we could agree on a format and start moving towards defining
board bindings that way.

Rob

Re: [RFC] yamldt v0.5, now a DTS compiler too

[Pantelis Antoniou]

> On Oct 2, 2017, at 01:00 , Rob Herring <robherring2@gmail.com> wrote:
> 
> On Thu, Sep 28, 2017 at 2:58 PM, Pantelis Antoniou
> <pantelis.antoniou@konsulko.com> wrote:
>> Hello again,
>> 
>> Significant progress has been made on yamldt and is now capable of
>> not only generating yaml from DTS source but also compiling DTS sources
>> and being almost fully compatible with DTC.
> 
> Can you quantify "almost”?

The —sort option and /include-bin/ doesn’t work yet.

It’s a day or so to add them but there are no kernel dtb files that rely on them.

> 

>> Compiling the kernel's DTBs using yamldt is as simple as using a
>> DTC=yamldt.
> 
> Good.
> 
>> 
>> Error reporting is accurate and validation against a YAML based schema
>> works as well. In a short while I will begin posting patches with
>> fixes on bindings and DTS files in the kernel.
> 
> What I would like to see is the schema format posted for review.
> 

It is based on your binding schema you’ve posted but tweaked slightly to
be parseable by the YAML subset that matches the DTS style.

In particular bare sequences do not map to DT at all, so:

- name: foo

is changed to
name: foo

And so on.

> I would also like to see the bindings for top-level compatible strings
> (aka boards) as an example. That's something that's simple enough that
> I'd think we could agree on a format and start moving towards defining
> board bindings that way.
> 

I’ll see what I can do.


> Rob

Regards

— Pantelis

Re: [RFC] yamldt v0.5, now a DTS compiler too

[Pantelis Antoniou]

Hi Rob,

On Sun, 2017-10-01 at 17:00 -0500, Rob Herring wrote:
> On Thu, Sep 28, 2017 at 2:58 PM, Pantelis Antoniou
> <pantelis.antoniou@konsulko.com> wrote:
> > Hello again,
> >
> > Significant progress has been made on yamldt and is now capable of
> > not only generating yaml from DTS source but also compiling DTS sources
> > and being almost fully compatible with DTC.
> 
> Can you quantify "almost"?
> 
> > Compiling the kernel's DTBs using yamldt is as simple as using a
> > DTC=yamldt.
> 
> Good.
> 
> >
> > Error reporting is accurate and validation against a YAML based schema
> > works as well. In a short while I will begin posting patches with
> > fixes on bindings and DTS files in the kernel.
> 
> What I would like to see is the schema format posted for review.
> 

I'm including the skeleton.yaml binding which is the template for
the bindings and a board-example.yaml binding for a top level binding.

> I would also like to see the bindings for top-level compatible strings
> (aka boards) as an example. That's something that's simple enough that
> I'd think we could agree on a format and start moving towards defining
> board bindings that way.
> 

Note there is some line wrapping I'm including a link
to the github repo of the files:


The skeleton.yaml 

https://raw.githubusercontent.com/pantoniou/yamldt/master/validate/bindings/skeleton.yaml

%YAML 1.1
---
# The name of the binding is first
# The anchor is put there for use by others
skeleton: &skeleton

  version: 1

  id: skel-device

  title: >
    Skeleton Device

  maintainer:
    name: Skeleton Person <skel@kernel.org>

  description: >
    The Skeleton Device binding represents the SK11 device produced by
    the Skeleton Corporation. The binding can also support compatible
    clones made by second source vendors.

  # The class is an optional property that declares this
  # binding as part of a larger set
  # Multiple definitions are possible
  class: [ device, spi-device ]

  # This binding inherits property characteristics from the generic
  # spi-slave binding
  # Note that the notation is standard yaml reference
  inherits: *spi-slave

  # virtual bindings do not generate checkers
  virtual: true

  # each property is defined by each name
  properties:

    # The compatible property is a reserved name. The type is always
"string"
    # and should not be repeated device binding.
    compatible:
      category: required        # required property
      type: strseq              # is a sequence of strings

      description: >
        FX11 is a clone of the original SK11 device

      # v is always the name of the value of the property
      # np is passed to the checker and is the current
      # node pointer. We can access properties and call
      # methods that operate on them.
      # There can be multiple constraints, just put them
      # into a sequence.
      # Note that the BASE("skel,sk11") form from the previous
      # binding will have to be reworked.
      constraint: |
        anystreq(v, "skel,sk11") ||
        anystreq(v, "faux,fx11")

    # The reg property is a reserved name. The type is always "int" and
    # should not be repeated in a device binding. Constraints are
defined
    # only in the context of the parent node's address, size, and ranges
    # cells. The description is inherited from the spi-slave binding.
    # Note that if inheriting from a base binding this declaration may
    # be omitted.
    reg:
      category: required        # required property
      type: intseq              # is a sequence of integers

    # spi-max-frequency needs the device-specific constraint to be
supplied
    spi-max-frequency:
      # this constraint is dependent on the compatible property
      # property containing "skel,sk11"
      constraint: |
        v <= anystreq(get_strseq(np, "compatible"), "skel,sk11") ?
10000000 : 1000000

    # This property overrides the generic binding description with
    # a device specific description in order to mention the chip's
    # h/w cfg strapping pins.
    spi-cs-high:
      description: >
        Set if skeleton device configuration pins are set for chip
        select polarity high

    # Device specific properties don't inherit characteristic from a
generic
    # binding so category, type, constraint, and description must be
specified
    # if needed.
    skel,deprecated1:
      # note that the category may be declare more than one option
      category: [ deprecated, optional ]
      type: int
      constraint: |
        v >= 100000 && v <= 200000
      description: >
        First of two deprecated properties.

    # There are no constraints for properties of empty type
    skel,deprecated2:
      category: deprecated
      type: empty
      description: >
        Second of two deprecated properties.

    # This example could be auto-generated rather than explicitly
included
    # in the yaml source.
    # Note that the YAML example must be validated against this binding
    # to be an accepted entry
    example:

      dts: |
        sk11@0 {
            compatible = "skel,sk11";
            reg = <0>;
            spi-max-frequency = <1000000>;
            spi-cs-high;
        };

      yaml: |
        sk11@0:
          compatible: "skel,sk11"
          reg: 0
          sip-max-frequency: 1000000
          spi-cs-high: true
        ---
...

And board-example.yaml

https://raw.githubusercontent.com/pantoniou/yamldt/master/validate/bindings/board-example.yaml

%YAML 1.1
---
board-example: &board-example
  version: 1

  title: A board example using compatible and model properties

  maintainer:
    name: Skeleton Person <skel@kernel.org>

  class: board

  # this binding is selected when the compatible property constraint
matches
  selected: "compatible"

  description: >
    A board binding example. Matches on a top-level compatible string
and model.

  properties:

    compatible:
      category: required
      type: strseq
      description: |
        Compatible strings for the board example.
        The depth of the node must be zero, i.e. root.

      constraint: |
        get_depth(np) == 0 && ( 
        anystreq(v, "example,evm") ||
        anystreq(v, "example,evm2") ||
        anystreq(v, "example,base"))

    model:
      category: required
      type: str
      description: models that this board family supports
      constraint: |
        streq(v, "Example EVM") ||
        streq(v, "Example EVM2")

  example:
    dts: |
      / {
          compatible = "example,evm", "example,base";
          model = "Example EVM";
      };
    yaml: |
      compatible: [ "example,evm", "example,base" ] ;
      model: "Example EVM";

As you see it's almost identical to what you've originally posted.

> Rob

Re: [RFC] yamldt v0.5, now a DTS compiler too

[Geert Uytterhoeven]

Hi Pantelis,

On Mon, Oct 2, 2017 at 9:46 PM, Pantelis Antoniou
<pantelis.antoniou@konsulko.com> wrote:
>     # Note that the YAML example must be validated against this binding
>     # to be an accepted entry

Indeed ;-)

>       yaml: |
>         sk11@0:
>           compatible: "skel,sk11"
>           reg: 0
>           sip-max-frequency: 1000000

s/sip-max-frequency/spi-max-frequency/

Gr{oetje,eeting}s,

                        Geert

--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
                                -- Linus Torvalds

Re: [RFC] yamldt v0.5, now a DTS compiler too

[Pantelis Antoniou]

Hi Geert,

On Tue, 2017-10-03 at 09:17 +0200, Geert Uytterhoeven wrote:
> Hi Pantelis,
> 
> On Mon, Oct 2, 2017 at 9:46 PM, Pantelis Antoniou
> <pantelis.antoniou@konsulko.com> wrote:
> >     # Note that the YAML example must be validated against this binding
> >     # to be an accepted entry
> 
> Indeed ;-)
> 
> >       yaml: |
> >         sk11@0:
> >           compatible: "skel,sk11"
> >           reg: 0
> >           sip-max-frequency: 1000000
> 
> s/sip-max-frequency/spi-max-frequency/
> 

Heh, obviously the auto-check of binding examples is not working yet :)

> Gr{oetje,eeting}s,
> 
>                         Geert
> 
> --
> Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org
> 
> In personal conversations with technical people, I call myself a hacker. But
> when I'm talking to journalists I just say "programmer" or something like that.
>                                 -- Linus Torvalds

Re: [RFC] yamldt v0.5, now a DTS compiler too

[Rob Herring]

On Mon, Oct 2, 2017 at 2:46 PM, Pantelis Antoniou
<pantelis.antoniou@konsulko.com> wrote:
> Hi Rob,
>
> On Sun, 2017-10-01 at 17:00 -0500, Rob Herring wrote:
>> On Thu, Sep 28, 2017 at 2:58 PM, Pantelis Antoniou
>> <pantelis.antoniou@konsulko.com> wrote:
>> > Hello again,
>> >
>> > Significant progress has been made on yamldt and is now capable of
>> > not only generating yaml from DTS source but also compiling DTS sources
>> > and being almost fully compatible with DTC.
>>
>> Can you quantify "almost"?
>>
>> > Compiling the kernel's DTBs using yamldt is as simple as using a
>> > DTC=yamldt.
>>
>> Good.
>>
>> >
>> > Error reporting is accurate and validation against a YAML based schema
>> > works as well. In a short while I will begin posting patches with
>> > fixes on bindings and DTS files in the kernel.
>>
>> What I would like to see is the schema format posted for review.
>>
>
> I'm including the skeleton.yaml binding which is the template for
> the bindings and a board-example.yaml binding for a top level binding.
>
>> I would also like to see the bindings for top-level compatible strings
>> (aka boards) as an example. That's something that's simple enough that
>> I'd think we could agree on a format and start moving towards defining
>> board bindings that way.
>>
>
> Note there is some line wrapping I'm including a link
> to the github repo of the files:
>
>
> The skeleton.yaml
>
> https://raw.githubusercontent.com/pantoniou/yamldt/master/validate/bindings/skeleton.yaml
>
> %YAML 1.1
> ---
> # The name of the binding is first
> # The anchor is put there for use by others
> skeleton: &skeleton

This and "id" seem redundant.

>   version: 1
>
>   id: skel-device
>
>   title: >
>     Skeleton Device
>
>   maintainer:
>     name: Skeleton Person <skel@kernel.org>
>
>   description: >
>     The Skeleton Device binding represents the SK11 device produced by
>     the Skeleton Corporation. The binding can also support compatible
>     clones made by second source vendors.
>
>   # The class is an optional property that declares this
>   # binding as part of a larger set
>   # Multiple definitions are possible
>   class: [ device, spi-device ]
>
>   # This binding inherits property characteristics from the generic
>   # spi-slave binding
>   # Note that the notation is standard yaml reference
>   inherits: *spi-slave
>
>   # virtual bindings do not generate checkers
>   virtual: true

virtual is an overloaded term.

>
>   # each property is defined by each name
>   properties:
>
>     # The compatible property is a reserved name. The type is always
> "string"
>     # and should not be repeated device binding.
>     compatible:
>       category: required        # required property
>       type: strseq              # is a sequence of strings
>
>       description: >
>         FX11 is a clone of the original SK11 device
>
>       # v is always the name of the value of the property
>       # np is passed to the checker and is the current
>       # node pointer. We can access properties and call
>       # methods that operate on them.
>       # There can be multiple constraints, just put them
>       # into a sequence.
>       # Note that the BASE("skel,sk11") form from the previous
>       # binding will have to be reworked.
>       constraint: |
>         anystreq(v, "skel,sk11") ||
>         anystreq(v, "faux,fx11")

Constraints and logic ops were certainly not decided in the last
attempt and I think will be the hardest part to define. I see you are
using eBPF in the checker. Is that where anystreq comes from?

How would you express the ordering requirement (most significant
compatible first)?

>
>     # The reg property is a reserved name. The type is always "int" and
>     # should not be repeated in a device binding. Constraints are
> defined
>     # only in the context of the parent node's address, size, and ranges
>     # cells. The description is inherited from the spi-slave binding.
>     # Note that if inheriting from a base binding this declaration may
>     # be omitted.
>     reg:
>       category: required        # required property
>       type: intseq              # is a sequence of integers
>
>     # spi-max-frequency needs the device-specific constraint to be
> supplied
>     spi-max-frequency:
>       # this constraint is dependent on the compatible property
>       # property containing "skel,sk11"
>       constraint: |
>         v <= anystreq(get_strseq(np, "compatible"), "skel,sk11") ?
> 10000000 : 1000000
>
>     # This property overrides the generic binding description with
>     # a device specific description in order to mention the chip's
>     # h/w cfg strapping pins.
>     spi-cs-high:
>       description: >
>         Set if skeleton device configuration pins are set for chip
>         select polarity high
>
>     # Device specific properties don't inherit characteristic from a
> generic
>     # binding so category, type, constraint, and description must be
> specified
>     # if needed.
>     skel,deprecated1:
>       # note that the category may be declare more than one option
>       category: [ deprecated, optional ]
>       type: int
>       constraint: |
>         v >= 100000 && v <= 200000
>       description: >
>         First of two deprecated properties.
>
>     # There are no constraints for properties of empty type
>     skel,deprecated2:
>       category: deprecated
>       type: empty
>       description: >
>         Second of two deprecated properties.
>
>     # This example could be auto-generated rather than explicitly
> included
>     # in the yaml source.
>     # Note that the YAML example must be validated against this binding
>     # to be an accepted entry
>     example:
>
>       dts: |
>         sk11@0 {
>             compatible = "skel,sk11";
>             reg = <0>;
>             spi-max-frequency = <1000000>;
>             spi-cs-high;
>         };
>
>       yaml: |
>         sk11@0:
>           compatible: "skel,sk11"
>           reg: 0
>           sip-max-frequency: 1000000
>           spi-cs-high: true
>         ---
> ...
>
> And board-example.yaml
>
> https://raw.githubusercontent.com/pantoniou/yamldt/master/validate/bindings/board-example.yaml
>
> %YAML 1.1
> ---
> board-example: &board-example
>   version: 1
>
>   title: A board example using compatible and model properties
>
>   maintainer:
>     name: Skeleton Person <skel@kernel.org>
>
>   class: board
>
>   # this binding is selected when the compatible property constraint
> matches
>   selected: "compatible"

We need some way to express this must be the root node. More
generally, we need to express what bindings must be a child of (think
sub-devices in MFDs). Perhaps that's just a "parent" key with the
value being the id/name.

>
>   description: >
>     A board binding example. Matches on a top-level compatible string
> and model.
>
>   properties:
>
>     compatible:
>       category: required
>       type: strseq
>       description: |
>         Compatible strings for the board example.
>         The depth of the node must be zero, i.e. root.
>
>       constraint: |
>         get_depth(np) == 0 && (

Ahh, I guess this is how you are doing it. I don't think this works
for any value other than 0. An MFD could be at any level.

>         anystreq(v, "example,evm") ||
>         anystreq(v, "example,evm2") ||
>         anystreq(v, "example,base"))
>
>     model:
>       category: required
>       type: str
>       description: models that this board family supports
>       constraint: |
>         streq(v, "Example EVM") ||
>         streq(v, "Example EVM2")
>
>   example:
>     dts: |
>       / {
>           compatible = "example,evm", "example,base";
>           model = "Example EVM";
>       };
>     yaml: |
>       compatible: [ "example,evm", "example,base" ] ;
>       model: "Example EVM";

I really don't want to see 2 examples. For now, it's going to be dts
format. It could be converted by script later if needed.

Overall, I think this format is a bit long for boards. We have
something like ~1000 boards in arch/arm last I checked. I want adding
a board binding to be very short and easy to review. It's often only a
1 line change. The main issue I have is it is just each SoC (or SoC
family) does things their own way.

> As you see it's almost identical to what you've originally posted.

I barely remember and no one ever commented on it.

Rob

Re: [RFC] yamldt v0.5, now a DTS compiler too

[Pantelis Antoniou]

Hi Rob,

On Tue, 2017-10-03 at 08:18 -0500, Rob Herring wrote:
> On Mon, Oct 2, 2017 at 2:46 PM, Pantelis Antoniou
> <pantelis.antoniou-OWPKS81ov/FWk0Htik3J/w@public.gmane.org> wrote:
> > Hi Rob,
> >
> > On Sun, 2017-10-01 at 17:00 -0500, Rob Herring wrote:
> >> On Thu, Sep 28, 2017 at 2:58 PM, Pantelis Antoniou
> >> <pantelis.antoniou-OWPKS81ov/FWk0Htik3J/w@public.gmane.org> wrote:
> >> > Hello again,
> >> >
> >> > Significant progress has been made on yamldt and is now capable of
> >> > not only generating yaml from DTS source but also compiling DTS sources
> >> > and being almost fully compatible with DTC.
> >>
> >> Can you quantify "almost"?
> >>
> >> > Compiling the kernel's DTBs using yamldt is as simple as using a
> >> > DTC=yamldt.
> >>
> >> Good.
> >>
> >> >
> >> > Error reporting is accurate and validation against a YAML based schema
> >> > works as well. In a short while I will begin posting patches with
> >> > fixes on bindings and DTS files in the kernel.
> >>
> >> What I would like to see is the schema format posted for review.
> >>
> >
> > I'm including the skeleton.yaml binding which is the template for
> > the bindings and a board-example.yaml binding for a top level binding.
> >
> >> I would also like to see the bindings for top-level compatible strings
> >> (aka boards) as an example. That's something that's simple enough that
> >> I'd think we could agree on a format and start moving towards defining
> >> board bindings that way.
> >>
> >
> > Note there is some line wrapping I'm including a link
> > to the github repo of the files:
> >
> >
> > The skeleton.yaml
> >
> > https://raw.githubusercontent.com/pantoniou/yamldt/master/validate/bindings/skeleton.yaml
> >
> > %YAML 1.1
> > ---
> > # The name of the binding is first
> > # The anchor is put there for use by others
> > skeleton: &skeleton
> 
> This and "id" seem redundant.
> 

Indeed.

> >   version: 1
> >
> >   id: skel-device
> >
> >   title: >
> >     Skeleton Device
> >
> >   maintainer:
> >     name: Skeleton Person <skel-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org>
> >
> >   description: >
> >     The Skeleton Device binding represents the SK11 device produced by
> >     the Skeleton Corporation. The binding can also support compatible
> >     clones made by second source vendors.
> >
> >   # The class is an optional property that declares this
> >   # binding as part of a larger set
> >   # Multiple definitions are possible
> >   class: [ device, spi-device ]
> >
> >   # This binding inherits property characteristics from the generic
> >   # spi-slave binding
> >   # Note that the notation is standard yaml reference
> >   inherits: *spi-slave
> >
> >   # virtual bindings do not generate checkers
> >   virtual: true
> 
> virtual is an overloaded term.
> 

OK, what term should I use that this binding should not be instantiated
as a checker, only be used by other bindings when inherited?

> >
> >   # each property is defined by each name
> >   properties:
> >
> >     # The compatible property is a reserved name. The type is always
> > "string"
> >     # and should not be repeated device binding.
> >     compatible:
> >       category: required        # required property
> >       type: strseq              # is a sequence of strings
> >
> >       description: >
> >         FX11 is a clone of the original SK11 device
> >
> >       # v is always the name of the value of the property
> >       # np is passed to the checker and is the current
> >       # node pointer. We can access properties and call
> >       # methods that operate on them.
> >       # There can be multiple constraints, just put them
> >       # into a sequence.
> >       # Note that the BASE("skel,sk11") form from the previous
> >       # binding will have to be reworked.
> >       constraint: |
> >         anystreq(v, "skel,sk11") ||
> >         anystreq(v, "faux,fx11")
> 
> Constraints and logic ops were certainly not decided in the last
> attempt and I think will be the hardest part to define. I see you are
> using eBPF in the checker. Is that where anystreq comes from?
> 

Yes. The ebpf environment declares a number of methods that are executed
outside the ebpf sandbox. Check out

https://raw.githubusercontent.com/pantoniou/yamldt/master/validate/schema/codegen.yaml
https://github.com/pantoniou/yamldt/blob/master/ebpf_dt.c

> How would you express the ordering requirement (most significant
> compatible first)?
> 

Err, there might be new bpf API needed there. For the first stab at
the bindings problem I concentrated on getting things working and be as
clear as possible.

You could do something like that:

   orderedstrseq(v, (const char *[]){ "skel,sk11", "faux,fx11", NULL })

Which would check ordering too.

Obviously you'd hide the weird syntax using a macro.

#define ORDEREDSTRSEQ(_v, ...) \
	orderedstrseq(_v, (const char *[]){ __VA_ARGS__ , NULL })


So you'd write the above as:

	ORDEREDSTRSEQ(v, "skel,sk11", "faux,fx11")

> >
> >     # The reg property is a reserved name. The type is always "int" and
> >     # should not be repeated in a device binding. Constraints are
> > defined
> >     # only in the context of the parent node's address, size, and ranges
> >     # cells. The description is inherited from the spi-slave binding.
> >     # Note that if inheriting from a base binding this declaration may
> >     # be omitted.
> >     reg:
> >       category: required        # required property
> >       type: intseq              # is a sequence of integers
> >
> >     # spi-max-frequency needs the device-specific constraint to be
> > supplied
> >     spi-max-frequency:
> >       # this constraint is dependent on the compatible property
> >       # property containing "skel,sk11"
> >       constraint: |
> >         v <= anystreq(get_strseq(np, "compatible"), "skel,sk11") ?
> > 10000000 : 1000000
> >
> >     # This property overrides the generic binding description with
> >     # a device specific description in order to mention the chip's
> >     # h/w cfg strapping pins.
> >     spi-cs-high:
> >       description: >
> >         Set if skeleton device configuration pins are set for chip
> >         select polarity high
> >
> >     # Device specific properties don't inherit characteristic from a
> > generic
> >     # binding so category, type, constraint, and description must be
> > specified
> >     # if needed.
> >     skel,deprecated1:
> >       # note that the category may be declare more than one option
> >       category: [ deprecated, optional ]
> >       type: int
> >       constraint: |
> >         v >= 100000 && v <= 200000
> >       description: >
> >         First of two deprecated properties.
> >
> >     # There are no constraints for properties of empty type
> >     skel,deprecated2:
> >       category: deprecated
> >       type: empty
> >       description: >
> >         Second of two deprecated properties.
> >
> >     # This example could be auto-generated rather than explicitly
> > included
> >     # in the yaml source.
> >     # Note that the YAML example must be validated against this binding
> >     # to be an accepted entry
> >     example:
> >
> >       dts: |
> >         sk11@0 {
> >             compatible = "skel,sk11";
> >             reg = <0>;
> >             spi-max-frequency = <1000000>;
> >             spi-cs-high;
> >         };
> >
> >       yaml: |
> >         sk11@0:
> >           compatible: "skel,sk11"
> >           reg: 0
> >           sip-max-frequency: 1000000
> >           spi-cs-high: true
> >         ---
> > ...
> >
> > And board-example.yaml
> >
> > https://raw.githubusercontent.com/pantoniou/yamldt/master/validate/bindings/board-example.yaml
> >
> > %YAML 1.1
> > ---
> > board-example: &board-example
> >   version: 1
> >
> >   title: A board example using compatible and model properties
> >
> >   maintainer:
> >     name: Skeleton Person <skel-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org>
> >
> >   class: board
> >
> >   # this binding is selected when the compatible property constraint
> > matches
> >   selected: "compatible"
> 
> We need some way to express this must be the root node. More
> generally, we need to express what bindings must be a child of (think
> sub-devices in MFDs). Perhaps that's just a "parent" key with the
> value being the id/name.

> >
> >   description: >
> >     A board binding example. Matches on a top-level compatible string
> > and model.
> >
> >   properties:
> >
> >     compatible:
> >       category: required
> >       type: strseq
> >       description: |
> >         Compatible strings for the board example.
> >         The depth of the node must be zero, i.e. root.
> >
> >       constraint: |
> >         get_depth(np) == 0 && (
> 
> Ahh, I guess this is how you are doing it. I don't think this works
> for any value other than 0. An MFD could be at any level.
> 

Well, I could've used a streq(get_name(get_parent(np)), "/") test but
this is faster. It's up to you what would work best.

> >         anystreq(v, "example,evm") ||
> >         anystreq(v, "example,evm2") ||
> >         anystreq(v, "example,base"))
> >
> >     model:
> >       category: required
> >       type: str
> >       description: models that this board family supports
> >       constraint: |
> >         streq(v, "Example EVM") ||
> >         streq(v, "Example EVM2")
> >
> >   example:
> >     dts: |
> >       / {
> >           compatible = "example,evm", "example,base";
> >           model = "Example EVM";
> >       };
> >     yaml: |
> >       compatible: [ "example,evm", "example,base" ] ;
> >       model: "Example EVM";
> 
> I really don't want to see 2 examples. For now, it's going to be dts
> format. It could be converted by script later if needed.
> 
> Overall, I think this format is a bit long for boards. We have
> something like ~1000 boards in arch/arm last I checked. I want adding
> a board binding to be very short and easy to review. It's often only a
> 1 line change. The main issue I have is it is just each SoC (or SoC
> family) does things their own way.
> 

Well, this is a full featured example; you could declare this a
'virtual' (or what ever you want to call it binding) and use:

board-example-foo:

  inherits: *board-example

  properties:
    compatible: ...

It is not absolutely terse, but it's still YAML. But for what is worth,
those YAML files can be generated using the C preprocessor. You could
define a macro that cuts the churn, albeit you lose the ability to
parse them as normal YAML files.

> > As you see it's almost identical to what you've originally posted.
> 
> I barely remember and no one ever commented on it.
> 

I mentioned that before, I wasn't CCed on it :)

> Rob

Regards

-- Pantelis

Re: [RFC] yamldt v0.5, now a DTS compiler too

[Rob Herring]

On Tue, Oct 3, 2017 at 9:13 AM, Pantelis Antoniou
<pantelis.antoniou@konsulko.com> wrote:
> Hi Rob,
>
> On Tue, 2017-10-03 at 08:18 -0500, Rob Herring wrote:
>> On Mon, Oct 2, 2017 at 2:46 PM, Pantelis Antoniou
>> <pantelis.antoniou@konsulko.com> wrote:
>> > Hi Rob,
>> >
>> > On Sun, 2017-10-01 at 17:00 -0500, Rob Herring wrote:
>> >> On Thu, Sep 28, 2017 at 2:58 PM, Pantelis Antoniou
>> >> <pantelis.antoniou@konsulko.com> wrote:
>> >> > Hello again,
>> >> >
>> >> > Significant progress has been made on yamldt and is now capable of
>> >> > not only generating yaml from DTS source but also compiling DTS sources
>> >> > and being almost fully compatible with DTC.
>> >>
>> >> Can you quantify "almost"?
>> >>
>> >> > Compiling the kernel's DTBs using yamldt is as simple as using a
>> >> > DTC=yamldt.
>> >>
>> >> Good.
>> >>
>> >> >
>> >> > Error reporting is accurate and validation against a YAML based schema
>> >> > works as well. In a short while I will begin posting patches with
>> >> > fixes on bindings and DTS files in the kernel.
>> >>
>> >> What I would like to see is the schema format posted for review.
>> >>
>> >
>> > I'm including the skeleton.yaml binding which is the template for
>> > the bindings and a board-example.yaml binding for a top level binding.
>> >
>> >> I would also like to see the bindings for top-level compatible strings
>> >> (aka boards) as an example. That's something that's simple enough that
>> >> I'd think we could agree on a format and start moving towards defining
>> >> board bindings that way.
>> >>
>> >
>> > Note there is some line wrapping I'm including a link
>> > to the github repo of the files:
>> >
>> >
>> > The skeleton.yaml
>> >
>> > https://raw.githubusercontent.com/pantoniou/yamldt/master/validate/bindings/skeleton.yaml
>> >
>> > %YAML 1.1
>> > ---
>> > # The name of the binding is first
>> > # The anchor is put there for use by others
>> > skeleton: &skeleton
>>
>> This and "id" seem redundant.
>>
>
> Indeed.

One other thing, "skeleton" is a bit weird as a key. It can't be
validated. All the other keys are standard words. I could write
"skeloton" by mistake and I guess I'd only find the mistake when
something inherits it. That's somewhat true with id, but we can at
least check "id" is present and that it's value is unique among all
other id values.

>
>> >   version: 1
>> >
>> >   id: skel-device
>> >
>> >   title: >
>> >     Skeleton Device
>> >
>> >   maintainer:
>> >     name: Skeleton Person <skel@kernel.org>
>> >
>> >   description: >
>> >     The Skeleton Device binding represents the SK11 device produced by
>> >     the Skeleton Corporation. The binding can also support compatible
>> >     clones made by second source vendors.
>> >
>> >   # The class is an optional property that declares this
>> >   # binding as part of a larger set
>> >   # Multiple definitions are possible
>> >   class: [ device, spi-device ]
>> >
>> >   # This binding inherits property characteristics from the generic
>> >   # spi-slave binding
>> >   # Note that the notation is standard yaml reference
>> >   inherits: *spi-slave
>> >
>> >   # virtual bindings do not generate checkers
>> >   virtual: true
>>
>> virtual is an overloaded term.
>>
>
> OK, what term should I use that this binding should not be instantiated
> as a checker, only be used by other bindings when inherited?

checks: true?

I'd really like to avoid having to decide and drop this, but I don't
really get why it is needed.

>
>> >
>> >   # each property is defined by each name
>> >   properties:
>> >
>> >     # The compatible property is a reserved name. The type is always
>> > "string"
>> >     # and should not be repeated device binding.
>> >     compatible:
>> >       category: required        # required property
>> >       type: strseq              # is a sequence of strings
>> >
>> >       description: >
>> >         FX11 is a clone of the original SK11 device
>> >
>> >       # v is always the name of the value of the property
>> >       # np is passed to the checker and is the current
>> >       # node pointer. We can access properties and call
>> >       # methods that operate on them.
>> >       # There can be multiple constraints, just put them
>> >       # into a sequence.
>> >       # Note that the BASE("skel,sk11") form from the previous
>> >       # binding will have to be reworked.
>> >       constraint: |
>> >         anystreq(v, "skel,sk11") ||
>> >         anystreq(v, "faux,fx11")
>>
>> Constraints and logic ops were certainly not decided in the last
>> attempt and I think will be the hardest part to define. I see you are
>> using eBPF in the checker. Is that where anystreq comes from?
>>
>
> Yes. The ebpf environment declares a number of methods that are executed
> outside the ebpf sandbox. Check out
>
> https://raw.githubusercontent.com/pantoniou/yamldt/master/validate/schema/codegen.yaml
> https://github.com/pantoniou/yamldt/blob/master/ebpf_dt.c

I looked at this some a while back. It wasn't clear to me what eBPF
gives us and what you have defined on top of it. What I'm really after
is documentation of the syntax and keywords here.

>> How would you express the ordering requirement (most significant
>> compatible first)?
>>
>
> Err, there might be new bpf API needed there. For the first stab at
> the bindings problem I concentrated on getting things working and be as
> clear as possible.
>
> You could do something like that:
>
>    orderedstrseq(v, (const char *[]){ "skel,sk11", "faux,fx11", NULL })
>
> Which would check ordering too.
>
> Obviously you'd hide the weird syntax using a macro.
>
> #define ORDEREDSTRSEQ(_v, ...) \
>         orderedstrseq(_v, (const char *[]){ __VA_ARGS__ , NULL })
>
>
> So you'd write the above as:
>
>         ORDEREDSTRSEQ(v, "skel,sk11", "faux,fx11")
>
>> >
>> >     # The reg property is a reserved name. The type is always "int" and
>> >     # should not be repeated in a device binding. Constraints are
>> > defined
>> >     # only in the context of the parent node's address, size, and ranges
>> >     # cells. The description is inherited from the spi-slave binding.
>> >     # Note that if inheriting from a base binding this declaration may
>> >     # be omitted.
>> >     reg:
>> >       category: required        # required property
>> >       type: intseq              # is a sequence of integers
>> >
>> >     # spi-max-frequency needs the device-specific constraint to be
>> > supplied
>> >     spi-max-frequency:
>> >       # this constraint is dependent on the compatible property
>> >       # property containing "skel,sk11"
>> >       constraint: |
>> >         v <= anystreq(get_strseq(np, "compatible"), "skel,sk11") ?
>> > 10000000 : 1000000
>> >
>> >     # This property overrides the generic binding description with
>> >     # a device specific description in order to mention the chip's
>> >     # h/w cfg strapping pins.
>> >     spi-cs-high:
>> >       description: >
>> >         Set if skeleton device configuration pins are set for chip
>> >         select polarity high
>> >
>> >     # Device specific properties don't inherit characteristic from a
>> > generic
>> >     # binding so category, type, constraint, and description must be
>> > specified
>> >     # if needed.
>> >     skel,deprecated1:
>> >       # note that the category may be declare more than one option
>> >       category: [ deprecated, optional ]
>> >       type: int
>> >       constraint: |
>> >         v >= 100000 && v <= 200000
>> >       description: >
>> >         First of two deprecated properties.
>> >
>> >     # There are no constraints for properties of empty type
>> >     skel,deprecated2:
>> >       category: deprecated
>> >       type: empty
>> >       description: >
>> >         Second of two deprecated properties.
>> >
>> >     # This example could be auto-generated rather than explicitly
>> > included
>> >     # in the yaml source.
>> >     # Note that the YAML example must be validated against this binding
>> >     # to be an accepted entry
>> >     example:
>> >
>> >       dts: |
>> >         sk11@0 {
>> >             compatible = "skel,sk11";
>> >             reg = <0>;
>> >             spi-max-frequency = <1000000>;
>> >             spi-cs-high;
>> >         };
>> >
>> >       yaml: |
>> >         sk11@0:
>> >           compatible: "skel,sk11"
>> >           reg: 0
>> >           sip-max-frequency: 1000000
>> >           spi-cs-high: true
>> >         ---
>> > ...
>> >
>> > And board-example.yaml
>> >
>> > https://raw.githubusercontent.com/pantoniou/yamldt/master/validate/bindings/board-example.yaml
>> >
>> > %YAML 1.1
>> > ---
>> > board-example: &board-example
>> >   version: 1
>> >
>> >   title: A board example using compatible and model properties
>> >
>> >   maintainer:
>> >     name: Skeleton Person <skel@kernel.org>
>> >
>> >   class: board
>> >
>> >   # this binding is selected when the compatible property constraint
>> > matches
>> >   selected: "compatible"
>>
>> We need some way to express this must be the root node. More
>> generally, we need to express what bindings must be a child of (think
>> sub-devices in MFDs). Perhaps that's just a "parent" key with the
>> value being the id/name.
>
>> >
>> >   description: >
>> >     A board binding example. Matches on a top-level compatible string
>> > and model.
>> >
>> >   properties:
>> >
>> >     compatible:
>> >       category: required
>> >       type: strseq
>> >       description: |
>> >         Compatible strings for the board example.
>> >         The depth of the node must be zero, i.e. root.
>> >
>> >       constraint: |
>> >         get_depth(np) == 0 && (
>>
>> Ahh, I guess this is how you are doing it. I don't think this works
>> for any value other than 0. An MFD could be at any level.
>>
>
> Well, I could've used a streq(get_name(get_parent(np)), "/") test but
> this is faster. It's up to you what would work best.

Why not just a "parent" key?

>> >         anystreq(v, "example,evm") ||
>> >         anystreq(v, "example,evm2") ||
>> >         anystreq(v, "example,base"))
>> >
>> >     model:
>> >       category: required
>> >       type: str
>> >       description: models that this board family supports
>> >       constraint: |
>> >         streq(v, "Example EVM") ||
>> >         streq(v, "Example EVM2")
>> >
>> >   example:
>> >     dts: |
>> >       / {
>> >           compatible = "example,evm", "example,base";
>> >           model = "Example EVM";
>> >       };
>> >     yaml: |
>> >       compatible: [ "example,evm", "example,base" ] ;
>> >       model: "Example EVM";
>>
>> I really don't want to see 2 examples. For now, it's going to be dts
>> format. It could be converted by script later if needed.
>>
>> Overall, I think this format is a bit long for boards. We have
>> something like ~1000 boards in arch/arm last I checked. I want adding
>> a board binding to be very short and easy to review. It's often only a
>> 1 line change. The main issue I have is it is just each SoC (or SoC
>> family) does things their own way.
>>
>
> Well, this is a full featured example; you could declare this a
> 'virtual' (or what ever you want to call it binding) and use:
>
> board-example-foo:
>
>   inherits: *board-example
>
>   properties:
>     compatible: ...
>
> It is not absolutely terse, but it's still YAML. But for what is worth,
> those YAML files can be generated using the C preprocessor. You could
> define a macro that cuts the churn, albeit you lose the ability to
> parse them as normal YAML files.

C preprocessor doesn't sound like a great option to me.

Perhaps it is by SoC compatible and boards are just an addition to the
constraints. That's kind of how things are organized already.

Rob

Re: [RFC] yamldt v0.5, now a DTS compiler too

[Pantelis Antoniou]

Hi Rob,

On Tue, 2017-10-03 at 12:13 -0500, Rob Herring wrote:
> On Tue, Oct 3, 2017 at 9:13 AM, Pantelis Antoniou
> <pantelis.antoniou-OWPKS81ov/FWk0Htik3J/w@public.gmane.org> wrote:
> > Hi Rob,
> >
> > On Tue, 2017-10-03 at 08:18 -0500, Rob Herring wrote:
> >> On Mon, Oct 2, 2017 at 2:46 PM, Pantelis Antoniou
> >> <pantelis.antoniou-OWPKS81ov/FWk0Htik3J/w@public.gmane.org> wrote:
> >> > Hi Rob,
> >> >
> >> > On Sun, 2017-10-01 at 17:00 -0500, Rob Herring wrote:
> >> >> On Thu, Sep 28, 2017 at 2:58 PM, Pantelis Antoniou
> >> >> <pantelis.antoniou-OWPKS81ov/FWk0Htik3J/w@public.gmane.org> wrote:
> >> >> > Hello again,
> >> >> >
> >> >> > Significant progress has been made on yamldt and is now capable of
> >> >> > not only generating yaml from DTS source but also compiling DTS sources
> >> >> > and being almost fully compatible with DTC.
> >> >>
> >> >> Can you quantify "almost"?
> >> >>
> >> >> > Compiling the kernel's DTBs using yamldt is as simple as using a
> >> >> > DTC=yamldt.
> >> >>
> >> >> Good.
> >> >>
> >> >> >
> >> >> > Error reporting is accurate and validation against a YAML based schema
> >> >> > works as well. In a short while I will begin posting patches with
> >> >> > fixes on bindings and DTS files in the kernel.
> >> >>
> >> >> What I would like to see is the schema format posted for review.
> >> >>
> >> >
> >> > I'm including the skeleton.yaml binding which is the template for
> >> > the bindings and a board-example.yaml binding for a top level binding.
> >> >
> >> >> I would also like to see the bindings for top-level compatible strings
> >> >> (aka boards) as an example. That's something that's simple enough that
> >> >> I'd think we could agree on a format and start moving towards defining
> >> >> board bindings that way.
> >> >>
> >> >
> >> > Note there is some line wrapping I'm including a link
> >> > to the github repo of the files:
> >> >
> >> >
> >> > The skeleton.yaml
> >> >
> >> > https://raw.githubusercontent.com/pantoniou/yamldt/master/validate/bindings/skeleton.yaml
> >> >
> >> > %YAML 1.1
> >> > ---
> >> > # The name of the binding is first
> >> > # The anchor is put there for use by others
> >> > skeleton: &skeleton
> >>
> >> This and "id" seem redundant.
> >>
> >
> > Indeed.
> 
> One other thing, "skeleton" is a bit weird as a key. It can't be
> validated. All the other keys are standard words. I could write
> "skeloton" by mistake and I guess I'd only find the mistake when
> something inherits it. That's somewhat true with id, but we can at
> least check "id" is present and that it's value is unique among all
> other id values.
> 

We can keep id and check that it matches the name of the enclosing node.
That way you can be sure that there's no error.

But it's a bit weird cause this is similar to declaring a function name
with a typo. You won't find out until you use it.

> >
> >> >   version: 1
> >> >
> >> >   id: skel-device
> >> >
> >> >   title: >
> >> >     Skeleton Device
> >> >
> >> >   maintainer:
> >> >     name: Skeleton Person <skel-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org>
> >> >
> >> >   description: >
> >> >     The Skeleton Device binding represents the SK11 device produced by
> >> >     the Skeleton Corporation. The binding can also support compatible
> >> >     clones made by second source vendors.
> >> >
> >> >   # The class is an optional property that declares this
> >> >   # binding as part of a larger set
> >> >   # Multiple definitions are possible
> >> >   class: [ device, spi-device ]
> >> >
> >> >   # This binding inherits property characteristics from the generic
> >> >   # spi-slave binding
> >> >   # Note that the notation is standard yaml reference
> >> >   inherits: *spi-slave
> >> >
> >> >   # virtual bindings do not generate checkers
> >> >   virtual: true
> >>
> >> virtual is an overloaded term.
> >>
> >
> > OK, what term should I use that this binding should not be instantiated
> > as a checker, only be used by other bindings when inherited?
> 
> checks: true?
> 
> I'd really like to avoid having to decide and drop this, but I don't
> really get why it is needed.
> 

It is needed because otherwise checker filters will be generated for
the template bindings that while they won't be executed they will be
compiled and take up space in the schema.

> >
> >> >
> >> >   # each property is defined by each name
> >> >   properties:
> >> >
> >> >     # The compatible property is a reserved name. The type is always
> >> > "string"
> >> >     # and should not be repeated device binding.
> >> >     compatible:
> >> >       category: required        # required property
> >> >       type: strseq              # is a sequence of strings
> >> >
> >> >       description: >
> >> >         FX11 is a clone of the original SK11 device
> >> >
> >> >       # v is always the name of the value of the property
> >> >       # np is passed to the checker and is the current
> >> >       # node pointer. We can access properties and call
> >> >       # methods that operate on them.
> >> >       # There can be multiple constraints, just put them
> >> >       # into a sequence.
> >> >       # Note that the BASE("skel,sk11") form from the previous
> >> >       # binding will have to be reworked.
> >> >       constraint: |
> >> >         anystreq(v, "skel,sk11") ||
> >> >         anystreq(v, "faux,fx11")
> >>
> >> Constraints and logic ops were certainly not decided in the last
> >> attempt and I think will be the hardest part to define. I see you are
> >> using eBPF in the checker. Is that where anystreq comes from?
> >>
> >
> > Yes. The ebpf environment declares a number of methods that are executed
> > outside the ebpf sandbox. Check out
> >
> > https://raw.githubusercontent.com/pantoniou/yamldt/master/validate/schema/codegen.yaml
> > https://github.com/pantoniou/yamldt/blob/master/ebpf_dt.c
> 
> I looked at this some a while back. It wasn't clear to me what eBPF
> gives us and what you have defined on top of it. What I'm really after
> is documentation of the syntax and keywords here.
> 

eBPF is portable, can be serialized after compiling in the schema file
and can be executed in the kernel.

By stripping out all documentation related properties and nodes keeping
only the compiled filters you can generate a dtb blob that passed to
kernel can be used for verification of all runtime changes in the
kernel's live tree. eBPF is enforcing an execution model that is 'safe'
so we can be sure that no foul play is possible.

That means that you can a) run it at boot-time, verifying the dtb blob
passed by the bootloader for errors (potentially disabling devices
that their nodes fail) and b) run it when applying overlays to reject
any that result in an invalid tree.

One other use that I'm thinking that might be possible too, like
binding version check between what the driver is compiled against and
what the passed blob is without using an explicit version property. 

The constraint format ofcourse is plain C, you can generate a .so using
the host compiler and execute that if you are looking for performance.

Syntax and keywords of the internals is in a bit of flux right now,
since the binding format is not finalized yet. I'd be happy to
discuss the internal at ELCE. 

> >> How would you express the ordering requirement (most significant
> >> compatible first)?
> >>
> >
> > Err, there might be new bpf API needed there. For the first stab at
> > the bindings problem I concentrated on getting things working and be as
> > clear as possible.
> >
> > You could do something like that:
> >
> >    orderedstrseq(v, (const char *[]){ "skel,sk11", "faux,fx11", NULL })
> >
> > Which would check ordering too.
> >
> > Obviously you'd hide the weird syntax using a macro.
> >
> > #define ORDEREDSTRSEQ(_v, ...) \
> >         orderedstrseq(_v, (const char *[]){ __VA_ARGS__ , NULL })
> >
> >
> > So you'd write the above as:
> >
> >         ORDEREDSTRSEQ(v, "skel,sk11", "faux,fx11")
> >
> >> >
> >> >     # The reg property is a reserved name. The type is always "int" and
> >> >     # should not be repeated in a device binding. Constraints are
> >> > defined
> >> >     # only in the context of the parent node's address, size, and ranges
> >> >     # cells. The description is inherited from the spi-slave binding.
> >> >     # Note that if inheriting from a base binding this declaration may
> >> >     # be omitted.
> >> >     reg:
> >> >       category: required        # required property
> >> >       type: intseq              # is a sequence of integers
> >> >
> >> >     # spi-max-frequency needs the device-specific constraint to be
> >> > supplied
> >> >     spi-max-frequency:
> >> >       # this constraint is dependent on the compatible property
> >> >       # property containing "skel,sk11"
> >> >       constraint: |
> >> >         v <= anystreq(get_strseq(np, "compatible"), "skel,sk11") ?
> >> > 10000000 : 1000000
> >> >
> >> >     # This property overrides the generic binding description with
> >> >     # a device specific description in order to mention the chip's
> >> >     # h/w cfg strapping pins.
> >> >     spi-cs-high:
> >> >       description: >
> >> >         Set if skeleton device configuration pins are set for chip
> >> >         select polarity high
> >> >
> >> >     # Device specific properties don't inherit characteristic from a
> >> > generic
> >> >     # binding so category, type, constraint, and description must be
> >> > specified
> >> >     # if needed.
> >> >     skel,deprecated1:
> >> >       # note that the category may be declare more than one option
> >> >       category: [ deprecated, optional ]
> >> >       type: int
> >> >       constraint: |
> >> >         v >= 100000 && v <= 200000
> >> >       description: >
> >> >         First of two deprecated properties.
> >> >
> >> >     # There are no constraints for properties of empty type
> >> >     skel,deprecated2:
> >> >       category: deprecated
> >> >       type: empty
> >> >       description: >
> >> >         Second of two deprecated properties.
> >> >
> >> >     # This example could be auto-generated rather than explicitly
> >> > included
> >> >     # in the yaml source.
> >> >     # Note that the YAML example must be validated against this binding
> >> >     # to be an accepted entry
> >> >     example:
> >> >
> >> >       dts: |
> >> >         sk11@0 {
> >> >             compatible = "skel,sk11";
> >> >             reg = <0>;
> >> >             spi-max-frequency = <1000000>;
> >> >             spi-cs-high;
> >> >         };
> >> >
> >> >       yaml: |
> >> >         sk11@0:
> >> >           compatible: "skel,sk11"
> >> >           reg: 0
> >> >           sip-max-frequency: 1000000
> >> >           spi-cs-high: true
> >> >         ---
> >> > ...
> >> >
> >> > And board-example.yaml
> >> >
> >> > https://raw.githubusercontent.com/pantoniou/yamldt/master/validate/bindings/board-example.yaml
> >> >
> >> > %YAML 1.1
> >> > ---
> >> > board-example: &board-example
> >> >   version: 1
> >> >
> >> >   title: A board example using compatible and model properties
> >> >
> >> >   maintainer:
> >> >     name: Skeleton Person <skel-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org>
> >> >
> >> >   class: board
> >> >
> >> >   # this binding is selected when the compatible property constraint
> >> > matches
> >> >   selected: "compatible"
> >>
> >> We need some way to express this must be the root node. More
> >> generally, we need to express what bindings must be a child of (think
> >> sub-devices in MFDs). Perhaps that's just a "parent" key with the
> >> value being the id/name.
> >
> >> >
> >> >   description: >
> >> >     A board binding example. Matches on a top-level compatible string
> >> > and model.
> >> >
> >> >   properties:
> >> >
> >> >     compatible:
> >> >       category: required
> >> >       type: strseq
> >> >       description: |
> >> >         Compatible strings for the board example.
> >> >         The depth of the node must be zero, i.e. root.
> >> >
> >> >       constraint: |
> >> >         get_depth(np) == 0 && (
> >>
> >> Ahh, I guess this is how you are doing it. I don't think this works
> >> for any value other than 0. An MFD could be at any level.
> >>
> >
> > Well, I could've used a streq(get_name(get_parent(np)), "/") test but
> > this is faster. It's up to you what would work best.
> 
> Why not just a "parent" key?
> 

Because the property/type prolog code would increase even for properties
that don't need the parent key. While generating the parent key only for
constraints that need it keeps the filter size smaller.

Of course if parent ends up being used for most constraints the
balancing act changes.


> >> >         anystreq(v, "example,evm") ||
> >> >         anystreq(v, "example,evm2") ||
> >> >         anystreq(v, "example,base"))
> >> >
> >> >     model:
> >> >       category: required
> >> >       type: str
> >> >       description: models that this board family supports
> >> >       constraint: |
> >> >         streq(v, "Example EVM") ||
> >> >         streq(v, "Example EVM2")
> >> >
> >> >   example:
> >> >     dts: |
> >> >       / {
> >> >           compatible = "example,evm", "example,base";
> >> >           model = "Example EVM";
> >> >       };
> >> >     yaml: |
> >> >       compatible: [ "example,evm", "example,base" ] ;
> >> >       model: "Example EVM";
> >>
> >> I really don't want to see 2 examples. For now, it's going to be dts
> >> format. It could be converted by script later if needed.
> >>
> >> Overall, I think this format is a bit long for boards. We have
> >> something like ~1000 boards in arch/arm last I checked. I want adding
> >> a board binding to be very short and easy to review. It's often only a
> >> 1 line change. The main issue I have is it is just each SoC (or SoC
> >> family) does things their own way.
> >>
> >
> > Well, this is a full featured example; you could declare this a
> > 'virtual' (or what ever you want to call it binding) and use:
> >
> > board-example-foo:
> >
> >   inherits: *board-example
> >
> >   properties:
> >     compatible: ...
> >
> > It is not absolutely terse, but it's still YAML. But for what is worth,
> > those YAML files can be generated using the C preprocessor. You could
> > define a macro that cuts the churn, albeit you lose the ability to
> > parse them as normal YAML files.
> 
> C preprocessor doesn't sound like a great option to me.
> 
> Perhaps it is by SoC compatible and boards are just an addition to the
> constraints. That's kind of how things are organized already.
> 

The code base supports multiple constraints so we could have multiple
constraint properties. How would you like your SoC example to
work ideally?

> Rob

Regards

-- Pantelis

--
To unsubscribe from this list: send the line "unsubscribe devicetree" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [RFC] yamldt v0.5, now a DTS compiler too

[Rob Herring]

On Tue, Oct 3, 2017 at 12:39 PM, Pantelis Antoniou
<pantelis.antoniou-OWPKS81ov/FWk0Htik3J/w@public.gmane.org> wrote:
> Hi Rob,
>
> On Tue, 2017-10-03 at 12:13 -0500, Rob Herring wrote:
>> On Tue, Oct 3, 2017 at 9:13 AM, Pantelis Antoniou
>> <pantelis.antoniou-OWPKS81ov/FWk0Htik3J/w@public.gmane.org> wrote:
>> > Hi Rob,
>> >
>> > On Tue, 2017-10-03 at 08:18 -0500, Rob Herring wrote:
>> >> On Mon, Oct 2, 2017 at 2:46 PM, Pantelis Antoniou
>> >> <pantelis.antoniou-OWPKS81ov/FWk0Htik3J/w@public.gmane.org> wrote:
>> >> > Hi Rob,
>> >> >
>> >> > On Sun, 2017-10-01 at 17:00 -0500, Rob Herring wrote:
>> >> >> On Thu, Sep 28, 2017 at 2:58 PM, Pantelis Antoniou
>> >> >> <pantelis.antoniou-OWPKS81ov/FWk0Htik3J/w@public.gmane.org> wrote:
>> >> >> > Hello again,
>> >> >> >
>> >> >> > Significant progress has been made on yamldt and is now capable of
>> >> >> > not only generating yaml from DTS source but also compiling DTS sources
>> >> >> > and being almost fully compatible with DTC.
>> >> >>
>> >> >> Can you quantify "almost"?
>> >> >>
>> >> >> > Compiling the kernel's DTBs using yamldt is as simple as using a
>> >> >> > DTC=yamldt.
>> >> >>
>> >> >> Good.
>> >> >>
>> >> >> >
>> >> >> > Error reporting is accurate and validation against a YAML based schema
>> >> >> > works as well. In a short while I will begin posting patches with
>> >> >> > fixes on bindings and DTS files in the kernel.
>> >> >>
>> >> >> What I would like to see is the schema format posted for review.
>> >> >>
>> >> >
>> >> > I'm including the skeleton.yaml binding which is the template for
>> >> > the bindings and a board-example.yaml binding for a top level binding.
>> >> >
>> >> >> I would also like to see the bindings for top-level compatible strings
>> >> >> (aka boards) as an example. That's something that's simple enough that
>> >> >> I'd think we could agree on a format and start moving towards defining
>> >> >> board bindings that way.
>> >> >>
>> >> >
>> >> > Note there is some line wrapping I'm including a link
>> >> > to the github repo of the files:
>> >> >
>> >> >
>> >> > The skeleton.yaml
>> >> >
>> >> > https://raw.githubusercontent.com/pantoniou/yamldt/master/validate/bindings/skeleton.yaml
>> >> >
>> >> > %YAML 1.1
>> >> > ---
>> >> > # The name of the binding is first
>> >> > # The anchor is put there for use by others
>> >> > skeleton: &skeleton
>> >>
>> >> This and "id" seem redundant.
>> >>
>> >
>> > Indeed.
>>
>> One other thing, "skeleton" is a bit weird as a key. It can't be
>> validated. All the other keys are standard words. I could write
>> "skeloton" by mistake and I guess I'd only find the mistake when
>> something inherits it. That's somewhat true with id, but we can at
>> least check "id" is present and that it's value is unique among all
>> other id values.
>>
>
> We can keep id and check that it matches the name of the enclosing node.
> That way you can be sure that there's no error.
>
> But it's a bit weird cause this is similar to declaring a function name
> with a typo. You won't find out until you use it.
>
>> >
>> >> >   version: 1
>> >> >
>> >> >   id: skel-device
>> >> >
>> >> >   title: >
>> >> >     Skeleton Device
>> >> >
>> >> >   maintainer:
>> >> >     name: Skeleton Person <skel-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org>
>> >> >
>> >> >   description: >
>> >> >     The Skeleton Device binding represents the SK11 device produced by
>> >> >     the Skeleton Corporation. The binding can also support compatible
>> >> >     clones made by second source vendors.
>> >> >
>> >> >   # The class is an optional property that declares this
>> >> >   # binding as part of a larger set
>> >> >   # Multiple definitions are possible
>> >> >   class: [ device, spi-device ]
>> >> >
>> >> >   # This binding inherits property characteristics from the generic
>> >> >   # spi-slave binding
>> >> >   # Note that the notation is standard yaml reference
>> >> >   inherits: *spi-slave
>> >> >
>> >> >   # virtual bindings do not generate checkers
>> >> >   virtual: true
>> >>
>> >> virtual is an overloaded term.
>> >>
>> >
>> > OK, what term should I use that this binding should not be instantiated
>> > as a checker, only be used by other bindings when inherited?
>>
>> checks: true?
>>
>> I'd really like to avoid having to decide and drop this, but I don't
>> really get why it is needed.
>>
>
> It is needed because otherwise checker filters will be generated for
> the template bindings that while they won't be executed they will be
> compiled and take up space in the schema.

Size is not a problem I have. That can come later if needed.

>> >> >   # each property is defined by each name
>> >> >   properties:
>> >> >
>> >> >     # The compatible property is a reserved name. The type is always
>> >> > "string"
>> >> >     # and should not be repeated device binding.
>> >> >     compatible:
>> >> >       category: required        # required property
>> >> >       type: strseq              # is a sequence of strings
>> >> >
>> >> >       description: >
>> >> >         FX11 is a clone of the original SK11 device
>> >> >
>> >> >       # v is always the name of the value of the property
>> >> >       # np is passed to the checker and is the current
>> >> >       # node pointer. We can access properties and call
>> >> >       # methods that operate on them.
>> >> >       # There can be multiple constraints, just put them
>> >> >       # into a sequence.
>> >> >       # Note that the BASE("skel,sk11") form from the previous
>> >> >       # binding will have to be reworked.
>> >> >       constraint: |
>> >> >         anystreq(v, "skel,sk11") ||
>> >> >         anystreq(v, "faux,fx11")
>> >>
>> >> Constraints and logic ops were certainly not decided in the last
>> >> attempt and I think will be the hardest part to define. I see you are
>> >> using eBPF in the checker. Is that where anystreq comes from?
>> >>
>> >
>> > Yes. The ebpf environment declares a number of methods that are executed
>> > outside the ebpf sandbox. Check out
>> >
>> > https://raw.githubusercontent.com/pantoniou/yamldt/master/validate/schema/codegen.yaml
>> > https://github.com/pantoniou/yamldt/blob/master/ebpf_dt.c
>>
>> I looked at this some a while back. It wasn't clear to me what eBPF
>> gives us and what you have defined on top of it. What I'm really after
>> is documentation of the syntax and keywords here.
>>
>
> eBPF is portable, can be serialized after compiling in the schema file
> and can be executed in the kernel.

Executing in the kernel is a non-goal for me.

> By stripping out all documentation related properties and nodes keeping
> only the compiled filters you can generate a dtb blob that passed to
> kernel can be used for verification of all runtime changes in the
> kernel's live tree. eBPF is enforcing an execution model that is 'safe'
> so we can be sure that no foul play is possible.

Humm, if you wanted to ensure dtb's are safe, I'd think that we just
sign them like you would for the kernel or modules.

> That means that you can a) run it at boot-time, verifying the dtb blob
> passed by the bootloader for errors (potentially disabling devices
> that their nodes fail) and b) run it when applying overlays to reject
> any that result in an invalid tree.

Let's get verification at build time working first, then we can worry
about something like this.

> One other use that I'm thinking that might be possible too, like
> binding version check between what the driver is compiled against and
> what the passed blob is without using an explicit version property.
>
> The constraint format ofcourse is plain C, you can generate a .so using
> the host compiler and execute that if you are looking for performance.
>
> Syntax and keywords of the internals is in a bit of flux right now,
> since the binding format is not finalized yet. I'd be happy to
> discuss the internal at ELCE.

I won't be there.

I'm just trying to understand what eBPF defines if anything and what's
up for discussion.

[...]

>> >> >       constraint: |
>> >> >         get_depth(np) == 0 && (
>> >>
>> >> Ahh, I guess this is how you are doing it. I don't think this works
>> >> for any value other than 0. An MFD could be at any level.
>> >>
>> >
>> > Well, I could've used a streq(get_name(get_parent(np)), "/") test but
>> > this is faster. It's up to you what would work best.
>>
>> Why not just a "parent" key?
>>
>
> Because the property/type prolog code would increase even for properties
> that don't need the parent key. While generating the parent key only for
> constraints that need it keeps the filter size smaller.

One could argue that category and type are also constraints, but you
don't express them as constraints. I think constraints should be
limited to constraints on the value of the property which are not
generically expressible (like type). I haven't fully thought through
whether there's other cases.

Another problem is this constraint is not really a constraint for
compatible as you have it, but for the whole binding.

[...]

>> >> Overall, I think this format is a bit long for boards. We have
>> >> something like ~1000 boards in arch/arm last I checked. I want adding
>> >> a board binding to be very short and easy to review. It's often only a
>> >> 1 line change. The main issue I have is it is just each SoC (or SoC
>> >> family) does things their own way.
>> >>
>> >
>> > Well, this is a full featured example; you could declare this a
>> > 'virtual' (or what ever you want to call it binding) and use:
>> >
>> > board-example-foo:
>> >
>> >   inherits: *board-example
>> >
>> >   properties:
>> >     compatible: ...
>> >
>> > It is not absolutely terse, but it's still YAML. But for what is worth,
>> > those YAML files can be generated using the C preprocessor. You could
>> > define a macro that cuts the churn, albeit you lose the ability to
>> > parse them as normal YAML files.
>>
>> C preprocessor doesn't sound like a great option to me.
>>
>> Perhaps it is by SoC compatible and boards are just an addition to the
>> constraints. That's kind of how things are organized already.
>>
>
> The code base supports multiple constraints so we could have multiple
> constraint properties. How would you like your SoC example to
> work ideally?

Not sure exactly other than meeting the requirements that I gave.

Rob
--
To unsubscribe from this list: send the line "unsubscribe devicetree" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [RFC] yamldt v0.5, now a DTS compiler too

[Pantelis Antoniou]

Hi Rob,

> On Oct 6, 2017, at 16:55 , Rob Herring <robherring2@gmail.com> wrote:
> 
> On Tue, Oct 3, 2017 at 12:39 PM, Pantelis Antoniou
> <pantelis.antoniou@konsulko.com> wrote:
>> Hi Rob,
>> 
>> On Tue, 2017-10-03 at 12:13 -0500, Rob Herring wrote:
>>> On Tue, Oct 3, 2017 at 9:13 AM, Pantelis Antoniou
>>> <pantelis.antoniou@konsulko.com> wrote:
>>>> Hi Rob,
>>>> 
>>>> On Tue, 2017-10-03 at 08:18 -0500, Rob Herring wrote:
>>>>> On Mon, Oct 2, 2017 at 2:46 PM, Pantelis Antoniou
>>>>> <pantelis.antoniou@konsulko.com> wrote:
>>>>>> Hi Rob,
>>>>>> 
>>>>>> On Sun, 2017-10-01 at 17:00 -0500, Rob Herring wrote:
>>>>>>> On Thu, Sep 28, 2017 at 2:58 PM, Pantelis Antoniou
>>>>>>> <pantelis.antoniou@konsulko.com> wrote:
>>>>>>>> Hello again,
>>>>>>>> 
>>>>>>>> Significant progress has been made on yamldt and is now capable of
>>>>>>>> not only generating yaml from DTS source but also compiling DTS sources
>>>>>>>> and being almost fully compatible with DTC.
>>>>>>> 
>>>>>>> Can you quantify "almost"?
>>>>>>> 
>>>>>>>> Compiling the kernel's DTBs using yamldt is as simple as using a
>>>>>>>> DTC=yamldt.
>>>>>>> 
>>>>>>> Good.
>>>>>>> 
>>>>>>>> 
>>>>>>>> Error reporting is accurate and validation against a YAML based schema
>>>>>>>> works as well. In a short while I will begin posting patches with
>>>>>>>> fixes on bindings and DTS files in the kernel.
>>>>>>> 
>>>>>>> What I would like to see is the schema format posted for review.
>>>>>>> 
>>>>>> 
>>>>>> I'm including the skeleton.yaml binding which is the template for
>>>>>> the bindings and a board-example.yaml binding for a top level binding.
>>>>>> 
>>>>>>> I would also like to see the bindings for top-level compatible strings
>>>>>>> (aka boards) as an example. That's something that's simple enough that
>>>>>>> I'd think we could agree on a format and start moving towards defining
>>>>>>> board bindings that way.
>>>>>>> 
>>>>>> 
>>>>>> Note there is some line wrapping I'm including a link
>>>>>> to the github repo of the files:
>>>>>> 
>>>>>> 
>>>>>> The skeleton.yaml
>>>>>> 
>>>>>> https://raw.githubusercontent.com/pantoniou/yamldt/master/validate/bindings/skeleton.yaml
>>>>>> 
>>>>>> %YAML 1.1
>>>>>> ---
>>>>>> # The name of the binding is first
>>>>>> # The anchor is put there for use by others
>>>>>> skeleton: &skeleton
>>>>> 
>>>>> This and "id" seem redundant.
>>>>> 
>>>> 
>>>> Indeed.
>>> 
>>> One other thing, "skeleton" is a bit weird as a key. It can't be
>>> validated. All the other keys are standard words. I could write
>>> "skeloton" by mistake and I guess I'd only find the mistake when
>>> something inherits it. That's somewhat true with id, but we can at
>>> least check "id" is present and that it's value is unique among all
>>> other id values.
>>> 
>> 
>> We can keep id and check that it matches the name of the enclosing node.
>> That way you can be sure that there's no error.
>> 
>> But it's a bit weird cause this is similar to declaring a function name
>> with a typo. You won't find out until you use it.
>> 
>>>> 
>>>>>>  version: 1
>>>>>> 
>>>>>>  id: skel-device
>>>>>> 
>>>>>>  title: >
>>>>>>    Skeleton Device
>>>>>> 
>>>>>>  maintainer:
>>>>>>    name: Skeleton Person <skel@kernel.org>
>>>>>> 
>>>>>>  description: >
>>>>>>    The Skeleton Device binding represents the SK11 device produced by
>>>>>>    the Skeleton Corporation. The binding can also support compatible
>>>>>>    clones made by second source vendors.
>>>>>> 
>>>>>>  # The class is an optional property that declares this
>>>>>>  # binding as part of a larger set
>>>>>>  # Multiple definitions are possible
>>>>>>  class: [ device, spi-device ]
>>>>>> 
>>>>>>  # This binding inherits property characteristics from the generic
>>>>>>  # spi-slave binding
>>>>>>  # Note that the notation is standard yaml reference
>>>>>>  inherits: *spi-slave
>>>>>> 
>>>>>>  # virtual bindings do not generate checkers
>>>>>>  virtual: true
>>>>> 
>>>>> virtual is an overloaded term.
>>>>> 
>>>> 
>>>> OK, what term should I use that this binding should not be instantiated
>>>> as a checker, only be used by other bindings when inherited?
>>> 
>>> checks: true?
>>> 
>>> I'd really like to avoid having to decide and drop this, but I don't
>>> really get why it is needed.
>>> 
>> 
>> It is needed because otherwise checker filters will be generated for
>> the template bindings that while they won't be executed they will be
>> compiled and take up space in the schema.
> 
> Size is not a problem I have. That can come later if needed.
> 
>>>>>>  # each property is defined by each name
>>>>>>  properties:
>>>>>> 
>>>>>>    # The compatible property is a reserved name. The type is always
>>>>>> "string"
>>>>>>    # and should not be repeated device binding.
>>>>>>    compatible:
>>>>>>      category: required        # required property
>>>>>>      type: strseq              # is a sequence of strings
>>>>>> 
>>>>>>      description: >
>>>>>>        FX11 is a clone of the original SK11 device
>>>>>> 
>>>>>>      # v is always the name of the value of the property
>>>>>>      # np is passed to the checker and is the current
>>>>>>      # node pointer. We can access properties and call
>>>>>>      # methods that operate on them.
>>>>>>      # There can be multiple constraints, just put them
>>>>>>      # into a sequence.
>>>>>>      # Note that the BASE("skel,sk11") form from the previous
>>>>>>      # binding will have to be reworked.
>>>>>>      constraint: |
>>>>>>        anystreq(v, "skel,sk11") ||
>>>>>>        anystreq(v, "faux,fx11")
>>>>> 
>>>>> Constraints and logic ops were certainly not decided in the last
>>>>> attempt and I think will be the hardest part to define. I see you are
>>>>> using eBPF in the checker. Is that where anystreq comes from?
>>>>> 
>>>> 
>>>> Yes. The ebpf environment declares a number of methods that are executed
>>>> outside the ebpf sandbox. Check out
>>>> 
>>>> https://raw.githubusercontent.com/pantoniou/yamldt/master/validate/schema/codegen.yaml
>>>> https://github.com/pantoniou/yamldt/blob/master/ebpf_dt.c
>>> 
>>> I looked at this some a while back. It wasn't clear to me what eBPF
>>> gives us and what you have defined on top of it. What I'm really after
>>> is documentation of the syntax and keywords here.
>>> 
>> 
>> eBPF is portable, can be serialized after compiling in the schema file
>> and can be executed in the kernel.
> 
> Executing in the kernel is a non-goal for me.
> 
>> By stripping out all documentation related properties and nodes keeping
>> only the compiled filters you can generate a dtb blob that passed to
>> kernel can be used for verification of all runtime changes in the
>> kernel's live tree. eBPF is enforcing an execution model that is 'safe'
>> so we can be sure that no foul play is possible.
> 
> Humm, if you wanted to ensure dtb's are safe, I'd think that we just
> sign them like you would for the kernel or modules.
> 

That’s a problem when deploying; the runtime validation is for making sure
developers don’t get bogged down chasing problems when working on their
own platforms/drivers.

We have absolutely zero checks for stopping badly configured DT blobs
hanging the kernel. With runtime validation a bug that might take a few
days to figure out can be cut down to a few minutes.

>> That means that you can a) run it at boot-time, verifying the dtb blob
>> passed by the bootloader for errors (potentially disabling devices
>> that their nodes fail) and b) run it when applying overlays to reject
>> any that result in an invalid tree.
> 
> Let's get verification at build time working first, then we can worry
> about something like this.
> 
>> One other use that I'm thinking that might be possible too, like
>> binding version check between what the driver is compiled against and
>> what the passed blob is without using an explicit version property.
>> 
>> The constraint format ofcourse is plain C, you can generate a .so using
>> the host compiler and execute that if you are looking for performance.
>> 
>> Syntax and keywords of the internals is in a bit of flux right now,
>> since the binding format is not finalized yet. I'd be happy to
>> discuss the internal at ELCE.
> 
> I won't be there.
> 
> I'm just trying to understand what eBPF defines if anything and what's
> up for discussion.

The eBPF defines an API that the constraints use to perform validation.
Anything that can be expressed as a C method can be in the eBPF API.

> 
> [...]
> 
>>>>>>      constraint: |
>>>>>>        get_depth(np) == 0 && (
>>>>> 
>>>>> Ahh, I guess this is how you are doing it. I don't think this works
>>>>> for any value other than 0. An MFD could be at any level.
>>>>> 
>>>> 
>>>> Well, I could've used a streq(get_name(get_parent(np)), "/") test but
>>>> this is faster. It's up to you what would work best.
>>> 
>>> Why not just a "parent" key?
>>> 
>> 
>> Because the property/type prolog code would increase even for properties
>> that don't need the parent key. While generating the parent key only for
>> constraints that need it keeps the filter size smaller.
> 
> One could argue that category and type are also constraints, but you
> don't express them as constraints. I think constraints should be
> limited to constraints on the value of the property which are not
> generically expressible (like type). I haven't fully thought through
> whether there's other cases.
> 

Both type and category constraints work in what I’ve posted, using the
eBPF APIs. Taking for instance the enabled status constraint from
‘device-enabled.yaml’

> %YAML 1.1
> ---
> device-enabled:
>   title: Contraint for enabled devices
> 
>   class: constraint
>   virtual: true
> 
>   properties:
>     status: &device-status-enabled
>       category: optional
>       type: str
>       description: Marks device state as enabled
>       constraint: |
>         !exists || streq(v, "okay") || streq(v, "ok”)
> 

It generates the following code fragment.

>         {
>         uint64_t flags;
>         const char *v = get_str(np, "status", &flags);
>         const bool badtype = !!(flags & BADTYPE);
>         const bool exists = !!(flags & EXISTS);
> 
>         if (badtype)
>             return -BADTYPE_BASE - 100002;
> 
>         if (!exists)
>             goto skip_100002;
> 
>         /* for status from device-compatible rule */
>         if (!(
>             !exists || streq(v, "okay") || streq(v, "ok")
>         ))
>             return -PROPC_BASE - 100002;
>         skip_100002:
>           do { } while(0); /* fix goto that requires a statement */
> 
>         }
> 

Both type checking (BADTYPE) and category (EXISTS) flags are returned
and checked.

> Another problem is this constraint is not really a constraint for
> compatible as you have it, but for the whole binding.
> 

What exactly is the difference in actual use? AFAIKT is our use of bindings and
special meaning of the compatible property.

> [...]
> 
>>>>> Overall, I think this format is a bit long for boards. We have
>>>>> something like ~1000 boards in arch/arm last I checked. I want adding
>>>>> a board binding to be very short and easy to review. It's often only a
>>>>> 1 line change. The main issue I have is it is just each SoC (or SoC
>>>>> family) does things their own way.
>>>>> 
>>>> 
>>>> Well, this is a full featured example; you could declare this a
>>>> 'virtual' (or what ever you want to call it binding) and use:
>>>> 
>>>> board-example-foo:
>>>> 
>>>>  inherits: *board-example
>>>> 
>>>>  properties:
>>>>    compatible: ...
>>>> 
>>>> It is not absolutely terse, but it's still YAML. But for what is worth,
>>>> those YAML files can be generated using the C preprocessor. You could
>>>> define a macro that cuts the churn, albeit you lose the ability to
>>>> parse them as normal YAML files.
>>> 
>>> C preprocessor doesn't sound like a great option to me.
>>> 
>>> Perhaps it is by SoC compatible and boards are just an addition to the
>>> constraints. That's kind of how things are organized already.
>>> 
>> 
>> The code base supports multiple constraints so we could have multiple
>> constraint properties. How would you like your SoC example to
>> work ideally?
> 
> Not sure exactly other than meeting the requirements that I gave.
> 
> Rob

Regards

— Pantelis

Re: [RFC] yamldt v0.5, now a DTS compiler too

[Frank Rowand]

On 10/07/17 03:23, Pantelis Antoniou wrote:
> Hi Rob,
> 
>> On Oct 6, 2017, at 16:55 , Rob Herring <robherring2-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
>>
>> On Tue, Oct 3, 2017 at 12:39 PM, Pantelis Antoniou
>> <pantelis.antoniou-OWPKS81ov/FWk0Htik3J/w@public.gmane.org> wrote:
>>> Hi Rob,

< snip >

>>> eBPF is portable, can be serialized after compiling in the schema file
>>> and can be executed in the kernel.
>>
>> Executing in the kernel is a non-goal for me.

Executing in the kernel is an anti-goal for me.

We are trying to reduce the device tree footprint inside the kernel,
not increase it.

99.99% of the validation should be possible statically, in the compile
phase.


>>> By stripping out all documentation related properties and nodes keeping
>>> only the compiled filters you can generate a dtb blob that passed to
>>> kernel can be used for verification of all runtime changes in the
>>> kernel's live tree. eBPF is enforcing an execution model that is 'safe'
>>> so we can be sure that no foul play is possible.

Run time changes can be assumed correct (short of bugs in the overlay
application code), if the base tree is validated, the overlay is validated,
and the interface between the live tree and the overlay is a connector.
 

>> Humm, if you wanted to ensure dtb's are safe, I'd think that we just
>> sign them like you would for the kernel or modules.
>>
> 
> That’s a problem when deploying; the runtime validation is for making sure
> developers don’t get bogged down chasing problems when working on their
> own platforms/drivers.
> 
> We have absolutely zero checks for stopping badly configured DT blobs
> hanging the kernel. With runtime validation a bug that might take a few
> days to figure out can be cut down to a few minutes.

Same reply as above.


>>> That means that you can a) run it at boot-time, verifying the dtb blob
>>> passed by the bootloader for errors (potentially disabling devices
>>> that their nodes fail) and b) run it when applying overlays to reject
>>> any that result in an invalid tree.
>>
>> Let's get verification at build time working first, then we can worry
>> about something like this.

< snip >

-Frank

Re: [RFC] yamldt v0.5, now a DTS compiler too

[David Gibson]

[-- Attachment #1: Type: text/plain, Size: 2030 bytes --]

On Sun, Oct 08, 2017 at 04:08:03PM -0700, Frank Rowand wrote:
> On 10/07/17 03:23, Pantelis Antoniou wrote:
> > Hi Rob,
> > 
> >> On Oct 6, 2017, at 16:55 , Rob Herring <robherring2-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
> >>
> >> On Tue, Oct 3, 2017 at 12:39 PM, Pantelis Antoniou
> >> <pantelis.antoniou-OWPKS81ov/FWk0Htik3J/w@public.gmane.org> wrote:
> >>> Hi Rob,
> 
> < snip >
> 
> >>> eBPF is portable, can be serialized after compiling in the schema file
> >>> and can be executed in the kernel.
> >>
> >> Executing in the kernel is a non-goal for me.
> 
> Executing in the kernel is an anti-goal for me.
> 
> We are trying to reduce the device tree footprint inside the kernel,
> not increase it.
> 
> 99.99% of the validation should be possible statically, in the compile
> phase.
> 
> 
> >>> By stripping out all documentation related properties and nodes keeping
> >>> only the compiled filters you can generate a dtb blob that passed to
> >>> kernel can be used for verification of all runtime changes in the
> >>> kernel's live tree. eBPF is enforcing an execution model that is 'safe'
> >>> so we can be sure that no foul play is possible.
> 
> Run time changes can be assumed correct (short of bugs in the overlay
> application code), if the base tree is validated, the overlay is validated,
> and the interface between the live tree and the overlay is a
> connector.

In addition, no amount of schema validation can really protect the
kernel from a bad DT.  Even if the schemas can 100% verify that the DT
is "syntactically" correct, which is ambitious, it can't protect
against a DT which is in the right form, but contains information that
is simply wrong for the hardware in question.  That can stuff the
kernel at least as easily as an incorrectly formatted DT.

-- 
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [RFC] yamldt v0.5, now a DTS compiler too

[Pantelis Antoniou]

Hi David,

> On Oct 9, 2017, at 03:00 , David Gibson <david-xT8FGy+AXnRB3Ne2BGzF6laj5H9X9Tb+@public.gmane.org> wrote:
> 
> On Sun, Oct 08, 2017 at 04:08:03PM -0700, Frank Rowand wrote:
>> On 10/07/17 03:23, Pantelis Antoniou wrote:
>>> Hi Rob,
>>> 
>>>> On Oct 6, 2017, at 16:55 , Rob Herring <robherring2-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
>>>> 
>>>> On Tue, Oct 3, 2017 at 12:39 PM, Pantelis Antoniou
>>>> <pantelis.antoniou-OWPKS81ov/FWk0Htik3J/w@public.gmane.org> wrote:
>>>>> Hi Rob,
>> 
>> < snip >
>> 
>>>>> eBPF is portable, can be serialized after compiling in the schema file
>>>>> and can be executed in the kernel.
>>>> 
>>>> Executing in the kernel is a non-goal for me.
>> 
>> Executing in the kernel is an anti-goal for me.
>> 
>> We are trying to reduce the device tree footprint inside the kernel,
>> not increase it.
>> 
>> 99.99% of the validation should be possible statically, in the compile
>> phase.
>> 
>> 
>>>>> By stripping out all documentation related properties and nodes keeping
>>>>> only the compiled filters you can generate a dtb blob that passed to
>>>>> kernel can be used for verification of all runtime changes in the
>>>>> kernel's live tree. eBPF is enforcing an execution model that is 'safe'
>>>>> so we can be sure that no foul play is possible.
>> 
>> Run time changes can be assumed correct (short of bugs in the overlay
>> application code), if the base tree is validated, the overlay is validated,
>> and the interface between the live tree and the overlay is a
>> connector.
> 
> In addition, no amount of schema validation can really protect the
> kernel from a bad DT.  Even if the schemas can 100% verify that the DT
> is "syntactically" correct, which is ambitious, it can't protect
> against a DT which is in the right form, but contains information that
> is simply wrong for the hardware in question.  That can stuff the
> kernel at least as easily as an incorrectly formatted DT.
> 

I disagree.

There are multiple levels of validation. For now we’re only talking about
binding validation. There can be SoC level validation, board level validation,
revision level validation and finally application specific validation.

Binding validation is making sure properties/nodes follow the binding document.
For instance that for a foo device there’s a mandatory interrupt property.

Simplified

interrupt = <X>;

Binding validation would ‘catch’ errors like assigning a string or not having the
interrupt property available.

SoC level validation would list the available interrupt number that a given
SoC would support for that device.

For example that interrupt can only take the values 10 or 99 in a given SoC.

Board level validation would narrow this down even further to a value of 10 for
a given board model.

Similar revision level validation would place further restriction on the allowed
configuration.

Finally application specific validation could place restriction based on the intended
application that piece of hardware is used for. For instance devices that should not
exceed a given power budget would have restrictions on the clock frequency of the processor
or bus frequencies etc.

> -- 
> David Gibson			| I'll have my music baroque, and my code
> david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
> 				| _way_ _around_!
> http://www.ozlabs.org/~dgibson

Regards

— Pantelis

--
To unsubscribe from this list: send the line "unsubscribe devicetree" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [RFC] yamldt v0.5, now a DTS compiler too

[David Gibson]

[-- Attachment #1: Type: text/plain, Size: 4066 bytes --]

On Mon, Oct 09, 2017 at 06:07:28PM +0300, Pantelis Antoniou wrote:
> Hi David,
> 
> > On Oct 9, 2017, at 03:00 , David Gibson <david@gibson.dropbear.id.au> wrote:
> > 
> > On Sun, Oct 08, 2017 at 04:08:03PM -0700, Frank Rowand wrote:
> >> On 10/07/17 03:23, Pantelis Antoniou wrote:
> >>> Hi Rob,
> >>> 
> >>>> On Oct 6, 2017, at 16:55 , Rob Herring <robherring2@gmail.com> wrote:
> >>>> 
> >>>> On Tue, Oct 3, 2017 at 12:39 PM, Pantelis Antoniou
> >>>> <pantelis.antoniou@konsulko.com> wrote:
> >>>>> Hi Rob,
> >> 
> >> < snip >
> >> 
> >>>>> eBPF is portable, can be serialized after compiling in the schema file
> >>>>> and can be executed in the kernel.
> >>>> 
> >>>> Executing in the kernel is a non-goal for me.
> >> 
> >> Executing in the kernel is an anti-goal for me.
> >> 
> >> We are trying to reduce the device tree footprint inside the kernel,
> >> not increase it.
> >> 
> >> 99.99% of the validation should be possible statically, in the compile
> >> phase.
> >> 
> >> 
> >>>>> By stripping out all documentation related properties and nodes keeping
> >>>>> only the compiled filters you can generate a dtb blob that passed to
> >>>>> kernel can be used for verification of all runtime changes in the
> >>>>> kernel's live tree. eBPF is enforcing an execution model that is 'safe'
> >>>>> so we can be sure that no foul play is possible.
> >> 
> >> Run time changes can be assumed correct (short of bugs in the overlay
> >> application code), if the base tree is validated, the overlay is validated,
> >> and the interface between the live tree and the overlay is a
> >> connector.
> > 
> > In addition, no amount of schema validation can really protect the
> > kernel from a bad DT.  Even if the schemas can 100% verify that the DT
> > is "syntactically" correct, which is ambitious, it can't protect
> > against a DT which is in the right form, but contains information that
> > is simply wrong for the hardware in question.  That can stuff the
> > kernel at least as easily as an incorrectly formatted DT.
> > 
> 
> I disagree.
> 
> There are multiple levels of validation. For now we’re only talking about
> binding validation. There can be SoC level validation, board level validation,
> revision level validation and finally application specific validation.
> 
> Binding validation is making sure properties/nodes follow the binding document.
> For instance that for a foo device there’s a mandatory interrupt property.
> 
> Simplified
> 
> interrupt = <X>;
> 
> Binding validation would ‘catch’ errors like assigning a string or not having the
> interrupt property available.
> 
> SoC level validation would list the available interrupt number that a given
> SoC would support for that device.
> 
> For example that interrupt can only take the values 10 or 99 in a given SoC.
> 
> Board level validation would narrow this down even further to a value of 10 for
> a given board model.

> Similar revision level validation would place further restriction on the allowed
> configuration.
> 
> Finally application specific validation could place restriction based on the intended
> application that piece of hardware is used for. For instance devices that should not
> exceed a given power budget would have restrictions on the clock frequency of the processor
> or bus frequencies etc.

This doesn't help.  In order to do this, the validator would need
information that's essentially equivalent to the content of DT, at
which point there's no point to the DT at all - and you're left with
the problem of validating the information that the validator has.

Fundamentally a validator that's useful *cannot* tell the difference
between a correct tree and one which _could_ be correct for some
theoretical hardware, but isn't for this particular hardware.

-- 
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [RFC] yamldt v0.5, now a DTS compiler too

[Pantelis Antoniou]

Hi David,

> On Oct 10, 2017, at 04:50 , David Gibson <david-xT8FGy+AXnRB3Ne2BGzF6laj5H9X9Tb+@public.gmane.org> wrote:
> 
> On Mon, Oct 09, 2017 at 06:07:28PM +0300, Pantelis Antoniou wrote:
>> Hi David,
>> 
>>> On Oct 9, 2017, at 03:00 , David Gibson <david-xT8FGy+AXnRB3Ne2BGzF6laj5H9X9Tb+@public.gmane.org> wrote:
>>> 
>>> On Sun, Oct 08, 2017 at 04:08:03PM -0700, Frank Rowand wrote:
>>>> On 10/07/17 03:23, Pantelis Antoniou wrote:
>>>>> Hi Rob,
>>>>> 
>>>>>> On Oct 6, 2017, at 16:55 , Rob Herring <robherring2-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
>>>>>> 
>>>>>> On Tue, Oct 3, 2017 at 12:39 PM, Pantelis Antoniou
>>>>>> <pantelis.antoniou-OWPKS81ov/FWk0Htik3J/w@public.gmane.org> wrote:
>>>>>>> Hi Rob,
>>>> 
>>>> < snip >
>>>> 
>>>>>>> eBPF is portable, can be serialized after compiling in the schema file
>>>>>>> and can be executed in the kernel.
>>>>>> 
>>>>>> Executing in the kernel is a non-goal for me.
>>>> 
>>>> Executing in the kernel is an anti-goal for me.
>>>> 
>>>> We are trying to reduce the device tree footprint inside the kernel,
>>>> not increase it.
>>>> 
>>>> 99.99% of the validation should be possible statically, in the compile
>>>> phase.
>>>> 
>>>> 
>>>>>>> By stripping out all documentation related properties and nodes keeping
>>>>>>> only the compiled filters you can generate a dtb blob that passed to
>>>>>>> kernel can be used for verification of all runtime changes in the
>>>>>>> kernel's live tree. eBPF is enforcing an execution model that is 'safe'
>>>>>>> so we can be sure that no foul play is possible.
>>>> 
>>>> Run time changes can be assumed correct (short of bugs in the overlay
>>>> application code), if the base tree is validated, the overlay is validated,
>>>> and the interface between the live tree and the overlay is a
>>>> connector.
>>> 
>>> In addition, no amount of schema validation can really protect the
>>> kernel from a bad DT.  Even if the schemas can 100% verify that the DT
>>> is "syntactically" correct, which is ambitious, it can't protect
>>> against a DT which is in the right form, but contains information that
>>> is simply wrong for the hardware in question.  That can stuff the
>>> kernel at least as easily as an incorrectly formatted DT.
>>> 
>> 
>> I disagree.
>> 
>> There are multiple levels of validation. For now we’re only talking about
>> binding validation. There can be SoC level validation, board level validation,
>> revision level validation and finally application specific validation.
>> 
>> Binding validation is making sure properties/nodes follow the binding document.
>> For instance that for a foo device there’s a mandatory interrupt property.
>> 
>> Simplified
>> 
>> interrupt = <X>;
>> 
>> Binding validation would ‘catch’ errors like assigning a string or not having the
>> interrupt property available.
>> 
>> SoC level validation would list the available interrupt number that a given
>> SoC would support for that device.
>> 
>> For example that interrupt can only take the values 10 or 99 in a given SoC.
>> 
>> Board level validation would narrow this down even further to a value of 10 for
>> a given board model.
> 
>> Similar revision level validation would place further restriction on the allowed
>> configuration.
>> 
>> Finally application specific validation could place restriction based on the intended
>> application that piece of hardware is used for. For instance devices that should not
>> exceed a given power budget would have restrictions on the clock frequency of the processor
>> or bus frequencies etc.
> 
> This doesn't help.  In order to do this, the validator would need
> information that's essentially equivalent to the content of DT, at
> which point there's no point to the DT at all - and you're left with
> the problem of validating the information that the validator has.
> 

That would be the case if hardware IP only has a single way to be configured.

The industry standard nowadays is picking reusable IP blocks and integrating them
together in an SoC. The driver and the binding is common for every platform that uses
that block, but the allowed configuration varies according to what the hardware
people uses in a given instance.

> Fundamentally a validator that's useful *cannot* tell the difference
> between a correct tree and one which _could_ be correct for some
> theoretical hardware, but isn't for this particular hardware.
> 

That’s why there’s reason for a nested hierarchy of bindings IMO.

Completeness of validation schemes can be a differentiating factor when
choosing parts for hardware design. They would sure cut down development time.


> -- 
> David Gibson			| I'll have my music baroque, and my code
> david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
> 				| _way_ _around_!
> http://www.ozlabs.org/~dgibson

Regards

— Pantelis

--
To unsubscribe from this list: send the line "unsubscribe devicetree" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [RFC] yamldt v0.5, now a DTS compiler too

[David Gibson]

[-- Attachment #1: Type: text/plain, Size: 5881 bytes --]

On Tue, Oct 10, 2017 at 06:19:03PM +0300, Pantelis Antoniou wrote:
> Hi David,
> 
> > On Oct 10, 2017, at 04:50 , David Gibson <david-xT8FGy+AXnRB3Ne2BGzF6laj5H9X9Tb+@public.gmane.org> wrote:
> > 
> > On Mon, Oct 09, 2017 at 06:07:28PM +0300, Pantelis Antoniou wrote:
> >> Hi David,
> >> 
> >>> On Oct 9, 2017, at 03:00 , David Gibson <david-xT8FGy+AXnRB3Ne2BGzF6laj5H9X9Tb+@public.gmane.org> wrote:
> >>> 
> >>> On Sun, Oct 08, 2017 at 04:08:03PM -0700, Frank Rowand wrote:
> >>>> On 10/07/17 03:23, Pantelis Antoniou wrote:
> >>>>> Hi Rob,
> >>>>> 
> >>>>>> On Oct 6, 2017, at 16:55 , Rob Herring <robherring2-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
> >>>>>> 
> >>>>>> On Tue, Oct 3, 2017 at 12:39 PM, Pantelis Antoniou
> >>>>>> <pantelis.antoniou-OWPKS81ov/FWk0Htik3J/w@public.gmane.org> wrote:
> >>>>>>> Hi Rob,
> >>>> 
> >>>> < snip >
> >>>> 
> >>>>>>> eBPF is portable, can be serialized after compiling in the schema file
> >>>>>>> and can be executed in the kernel.
> >>>>>> 
> >>>>>> Executing in the kernel is a non-goal for me.
> >>>> 
> >>>> Executing in the kernel is an anti-goal for me.
> >>>> 
> >>>> We are trying to reduce the device tree footprint inside the kernel,
> >>>> not increase it.
> >>>> 
> >>>> 99.99% of the validation should be possible statically, in the compile
> >>>> phase.
> >>>> 
> >>>> 
> >>>>>>> By stripping out all documentation related properties and nodes keeping
> >>>>>>> only the compiled filters you can generate a dtb blob that passed to
> >>>>>>> kernel can be used for verification of all runtime changes in the
> >>>>>>> kernel's live tree. eBPF is enforcing an execution model that is 'safe'
> >>>>>>> so we can be sure that no foul play is possible.
> >>>> 
> >>>> Run time changes can be assumed correct (short of bugs in the overlay
> >>>> application code), if the base tree is validated, the overlay is validated,
> >>>> and the interface between the live tree and the overlay is a
> >>>> connector.
> >>> 
> >>> In addition, no amount of schema validation can really protect the
> >>> kernel from a bad DT.  Even if the schemas can 100% verify that the DT
> >>> is "syntactically" correct, which is ambitious, it can't protect
> >>> against a DT which is in the right form, but contains information that
> >>> is simply wrong for the hardware in question.  That can stuff the
> >>> kernel at least as easily as an incorrectly formatted DT.
> >>> 
> >> 
> >> I disagree.
> >> 
> >> There are multiple levels of validation. For now we’re only talking about
> >> binding validation. There can be SoC level validation, board level validation,
> >> revision level validation and finally application specific validation.
> >> 
> >> Binding validation is making sure properties/nodes follow the binding document.
> >> For instance that for a foo device there’s a mandatory interrupt property.
> >> 
> >> Simplified
> >> 
> >> interrupt = <X>;
> >> 
> >> Binding validation would ‘catch’ errors like assigning a string or not having the
> >> interrupt property available.
> >> 
> >> SoC level validation would list the available interrupt number that a given
> >> SoC would support for that device.
> >> 
> >> For example that interrupt can only take the values 10 or 99 in a given SoC.
> >> 
> >> Board level validation would narrow this down even further to a value of 10 for
> >> a given board model.
> > 
> >> Similar revision level validation would place further restriction on the allowed
> >> configuration.
> >> 
> >> Finally application specific validation could place restriction based on the intended
> >> application that piece of hardware is used for. For instance devices that should not
> >> exceed a given power budget would have restrictions on the clock frequency of the processor
> >> or bus frequencies etc.
> > 
> > This doesn't help.  In order to do this, the validator would need
> > information that's essentially equivalent to the content of DT, at
> > which point there's no point to the DT at all - and you're left with
> > the problem of validating the information that the validator has.
> 
> That would be the case if hardware IP only has a single way to be configured.

Right, and if if there's more than one way, then the validator can't
possibly tell whether the DT has the right one.

DTs must always come from a trusted source, because if they don't,
then you don't need the DT in the first place (you could build your
own).

> The industry standard nowadays is picking reusable IP blocks and integrating them
> together in an SoC. The driver and the binding is common for every platform that uses
> that block, but the allowed configuration varies according to what the hardware
> people uses in a given instance.

> > Fundamentally a validator that's useful *cannot* tell the difference
> > between a correct tree and one which _could_ be correct for some
> > theoretical hardware, but isn't for this particular hardware.
> 
> That’s why there’s reason for a nested hierarchy of bindings IMO.

Nothing about how you structure the validation can change the basic
fact that there are only two possibilities.  Either:

a) You know the hardware structure independent of the DT, in which
   case the DT is pointless

or

b) You don't know everything about the hardware without the DT, in
   which case you can't know if the DT is right for this hardware

> Completeness of validation schemes can be a differentiating factor when
> choosing parts for hardware design. They would sure cut down development time.
> 
> 
> 
> Regards
> 
> — Pantelis
> 

-- 
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [RFC] yamldt v0.5, now a DTS compiler too

[Pantelis Antoniou]

Hi Frank,

> On Oct 9, 2017, at 02:08 , Frank Rowand <frowand.list@gmail.com> wrote:
> 
> On 10/07/17 03:23, Pantelis Antoniou wrote:
>> Hi Rob,
>> 
>>> On Oct 6, 2017, at 16:55 , Rob Herring <robherring2@gmail.com> wrote:
>>> 
>>> On Tue, Oct 3, 2017 at 12:39 PM, Pantelis Antoniou
>>> <pantelis.antoniou@konsulko.com> wrote:
>>>> Hi Rob,
> 
> < snip >
> 
>>>> eBPF is portable, can be serialized after compiling in the schema file
>>>> and can be executed in the kernel.
>>> 
>>> Executing in the kernel is a non-goal for me.
> 
> Executing in the kernel is an anti-goal for me.
> 
> We are trying to reduce the device tree footprint inside the kernel,
> not increase it.
> 
> 99.99% of the validation should be possible statically, in the compile
> phase.
> 

That’s not possible when you dynamically alter the tree at runtime.

> 
>>>> By stripping out all documentation related properties and nodes keeping
>>>> only the compiled filters you can generate a dtb blob that passed to
>>>> kernel can be used for verification of all runtime changes in the
>>>> kernel's live tree. eBPF is enforcing an execution model that is 'safe'
>>>> so we can be sure that no foul play is possible.
> 
> Run time changes can be assumed correct (short of bugs in the overlay
> application code), if the base tree is validated, the overlay is validated,
> and the interface between the live tree and the overlay is a connector.
> 

You can validate the base tree statically but not the overlays.

In fact a large percentage of overlay usage simply modifies a status property
to turn on or off a device. There is nothing to validate in such case.

The portable connector is still a long ways off and I haven’t seen anything that
could handle something trickier that a few GPIOs and I2C or SPI busses.

My goal is something that works covering all the cases without any surprising
gotchas.

> 
>>> Humm, if you wanted to ensure dtb's are safe, I'd think that we just
>>> sign them like you would for the kernel or modules.
>>> 
>> 
>> That’s a problem when deploying; the runtime validation is for making sure
>> developers don’t get bogged down chasing problems when working on their
>> own platforms/drivers.
>> 
>> We have absolutely zero checks for stopping badly configured DT blobs
>> hanging the kernel. With runtime validation a bug that might take a few
>> days to figure out can be cut down to a few minutes.
> 
> Same reply as above.
> 
> 
>>>> That means that you can a) run it at boot-time, verifying the dtb blob
>>>> passed by the bootloader for errors (potentially disabling devices
>>>> that their nodes fail) and b) run it when applying overlays to reject
>>>> any that result in an invalid tree.
>>> 
>>> Let's get verification at build time working first, then we can worry
>>> about something like this.
> 
> < snip >
> 

Right, let’s get build verification working first.

> -Frank

Regards

— Pantelis

Re: [RFC] yamldt v0.5, now a DTS compiler too

[Grant Likely]

On Thu, Sep 28, 2017 at 8:58 PM, Pantelis Antoniou
<pantelis.antoniou-OWPKS81ov/FWk0Htik3J/w@public.gmane.org> wrote:
> Hello again,
>
> Significant progress has been made on yamldt and is now capable of
> not only generating yaml from DTS source but also compiling DTS sources
> and being almost fully compatible with DTC.
>
> Compiling the kernel's DTBs using yamldt is as simple as using a
> DTC=yamldt.
>
> Error reporting is accurate and validation against a YAML based schema
> works as well. In a short while I will begin posting patches with
> fixes on bindings and DTS files in the kernel.
>
> Please try it on your platform and report if you encounter any problems.
>
> https://github.com/pantoniou/yamldt
>
> I am eagerly awaiting for your comments.

Hi Pantelis,

This is good work. I've played around with it and I'm looking forward
to chatting next week.

One thing I've done is tried loading the output YAML files into
another YAML interpreter and the current encoding causes problems.
Specifically, in yamldt anchors/aliases are being used as a
replacement for labels/phandles, but that conflicts with the YAML data
model which defines a reference as a way to make a copy of the data
appear in another part of the tree. For example, for the following
snippit:

intc: intc@10000 {
    #interrupt-cells = <1>;
    compatible = "acme,intc";
    reg = <0x10000 0x1000>;
    gpio-controller;
};

serial@20000 {
    compatible = "acme,uart";
    reg = <0x20000 0x1000>;
    interrupt-parent = <&intc>;
    interrupts = <5>;
};

yamldt will encode this as:

intc@10000: &intc
    "#interrupt-cells": 1
    compatible: acme,intc
    reg: [0x10000, 0x1000]
    gpio-controller:

serial@20000:
    compatible: acme,uart
    reg: [0x20000, 0x1000]
    interrupt-parent: *intc
    interrupts: 5

But, the expected behaviour for a YAML parser is expand the alias
'*intc' which results in the following structure:

intc@10000: &intc
    "#interrupt-cells": 1
    compatible: acme,intc
    reg: [0x10000, 0x1000]
    gpio-controller:

serial@20000:
    compatible: acme,uart
    reg: [0x20000, 0x1000]
    interrupt-parent:
        "#interrupt-cells": 1
        compatible: acme,intc
        reg: [0x10000, 0x1000]
        gpio-controller:
    interrupts: 5

See? It results in the entire interrupt controller node to appear as
an instance under the interrupt-parent property, when the intention is
only to create a phandle. yamldt should not redefine the behaviour of
'*' aliases. Instead, it should use a different indicator, either
using an explicit !phandle tag, or by replacing '*' with something
else. I worked around it in my tests by replacing '*' with '$'.

Plus, it would be useful to use normal YAML anchors/aliases for
creating node templates. For example:

serial-template: &acme-uart . # The anchor for the template
    compatible: acme,uart
    interrupt-parent: *intc

root:
    serial@20000:
        <<: *acme-uart   # Alias node merged into serial@20000
        interrupts: 5
        reg: [0x20000, 0x1000]
    serial@30000:
        <<: *acme-uart   # Alias node merged into serial@30000
        interrupts: 5
        reg: [0x30000, 0x1000]

Another problem with anchors/references is YAML seems to require the
anchor to be defined before the reference, or at least that's what
pyyaml and ruamel both expect. Regardless, The chosen YAML encoding
should be readily consumable by existing yaml implementations without
having to do a lot of customization.

I'm slightly concerned about using & anchors for labels because it
seems only one anchor can be defined per node, but DTC allows multiple
labels for a single node. This might not be an issue in practice
though. Another implementation issue related to using & anchors is the
YAML spec defines them as an encoding artifact, and parsers can
discard the anchor names after parsing the YAML structure, which is a
problem if we use something like $name to reference an anchor. The
solution might just be that labels need to go into a special property
so they don't disappear from the data stream.

There appears to be no place to put metadata. The root of the tree is
the top level of the YAML structure. There isn't any provision for
having a top level object to hold things like the memreserve map. We
may need a namespace to use for special properties that aren't nodes
or properties.

The encoding differentiates between nodes and properties implicitly
base on whether the contents are a map, or a scalar/list. This does
mean any parser needs to do a bit more work and it may impact what can
be done with validation (I'm not going to talk about validation in
this email though. We'll talk next week.)

Cheers,
g.
--
To unsubscribe from this list: send the line "unsubscribe devicetree" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [RFC] yamldt v0.5, now a DTS compiler too

[Pantelis Antoniou]

Hi Grant,

> On Oct 20, 2017, at 20:46 , Grant Likely <grant.likely@secretlab.ca> wrote:
> 
> On Thu, Sep 28, 2017 at 8:58 PM, Pantelis Antoniou
> <pantelis.antoniou@konsulko.com> wrote:
>> Hello again,
>> 
>> Significant progress has been made on yamldt and is now capable of
>> not only generating yaml from DTS source but also compiling DTS sources
>> and being almost fully compatible with DTC.
>> 
>> Compiling the kernel's DTBs using yamldt is as simple as using a
>> DTC=yamldt.
>> 
>> Error reporting is accurate and validation against a YAML based schema
>> works as well. In a short while I will begin posting patches with
>> fixes on bindings and DTS files in the kernel.
>> 
>> Please try it on your platform and report if you encounter any problems.
>> 
>> https://github.com/pantoniou/yamldt
>> 
>> I am eagerly awaiting for your comments.
> 
> Hi Pantelis,
> 
> This is good work. I've played around with it and I'm looking forward
> to chatting next week.
> 

Thanks. I’m looking forward to it too.

> One thing I've done is tried loading the output YAML files into
> another YAML interpreter and the current encoding causes problems.
> Specifically, in yamldt anchors/aliases are being used as a
> replacement for labels/phandles, but that conflicts with the YAML data
> model which defines a reference as a way to make a copy of the data
> appear in another part of the tree. For example, for the following
> snippit:
> 
> intc: intc@10000 {
>    #interrupt-cells = <1>;
>    compatible = "acme,intc";
>    reg = <0x10000 0x1000>;
>    gpio-controller;
> };
> 
> serial@20000 {
>    compatible = "acme,uart";
>    reg = <0x20000 0x1000>;
>    interrupt-parent = <&intc>;
>    interrupts = <5>;
> };
> 
> yamldt will encode this as:
> 
> intc@10000: &intc
>    "#interrupt-cells": 1
>    compatible: acme,intc
>    reg: [0x10000, 0x1000]
>    gpio-controller:
> 
> serial@20000:
>    compatible: acme,uart
>    reg: [0x20000, 0x1000]
>    interrupt-parent: *intc
>    interrupts: 5
> 
> But, the expected behaviour for a YAML parser is expand the alias
> '*intc' which results in the following structure:
> 
> intc@10000: &intc
>    "#interrupt-cells": 1
>    compatible: acme,intc
>    reg: [0x10000, 0x1000]
>    gpio-controller:
> 
> serial@20000:
>    compatible: acme,uart
>    reg: [0x20000, 0x1000]
>    interrupt-parent:
>        "#interrupt-cells": 1
>        compatible: acme,intc
>        reg: [0x10000, 0x1000]
>        gpio-controller:
>    interrupts: 5
> 
> See? It results in the entire interrupt controller node to appear as
> an instance under the interrupt-parent property, when the intention is
> only to create a phandle. yamldt should not redefine the behaviour of
> '*' aliases. Instead, it should use a different indicator, either
> using an explicit !phandle tag, or by replacing '*' with something
> else. I worked around it in my tests by replacing '*' with '$’.

Yes. This is expected. I don’t think pure YAML form is a good match for all
the crazy things that are possible (and actually used in practice) with DTS.

For instance there’s no way a normal YAML parser won’t choke with something like

/ {
  foo;
};

/ {
  bar;
};

Which is a common idiom in DTS files.

The decision to use the YAML anchors and references was more borne out of a desire
to sort of match conceptually the labels and references of DTS. It’s not a big
issue to switch to something different.  

If we were to force YAML/DT file to be regular YAML files parseable by everything
we’d have to rethink a lot of conventions that DT files currently use and I’m afraid
a machine translation from DTS to YAML might not be feasible then.

OTOH moving to that model might make it possible to use YAML constructs that are not
mapped at all to DTS. For instance

- foo: true
  bar: “string”

- foo: false
  bar: “another-string”

is not possible to be mapped to a DT node/property structure right now.

> 

> Plus, it would be useful to use normal YAML anchors/aliases for
> creating node templates. For example:
> 
> serial-template: &acme-uart . # The anchor for the template
>    compatible: acme,uart
>    interrupt-parent: *intc
> 
> root:
>    serial@20000:
>        <<: *acme-uart   # Alias node merged into serial@20000
>        interrupts: 5
>        reg: [0x20000, 0x1000]
>    serial@30000:
>        <<: *acme-uart   # Alias node merged into serial@30000
>        interrupts: 5
>        reg: [0x30000, 0x1000]
> 

Yes, I’m aware of this and I planned to talk to you about it :)
It can be a very powerful way to cut down the churn of DT right now.

It’s not going to be a problem for yamldt to support it (perhaps
adding a new editing tags to allow more fine grained control of the
substitions).
 
> Another problem with anchors/references is YAML seems to require the
> anchor to be defined before the reference, or at least that's what
> pyyaml and ruamel both expect. Regardless, The chosen YAML encoding
> should be readily consumable by existing yaml implementations without
> having to do a lot of customization.
> 

I think this is only true for streamable YAML implementations; yamldt
can leave references unresolved until the emit phase (or even leave them
be as object files).

> I'm slightly concerned about using & anchors for labels because it
> seems only one anchor can be defined per node, but DTC allows multiple
> labels for a single node. This might not be an issue in practice
> though. Another implementation issue related to using & anchors is the
> YAML spec defines them as an encoding artifact, and parsers can
> discard the anchor names after parsing the YAML structure, which is a
> problem if we use something like $name to reference an anchor. The
> solution might just be that labels need to go into a special property
> so they don't disappear from the data stream.
> 

I do support multiple labels to the same node in yamldt with the
following trick, that uses an empty tree.

foo: bar: baz { frob; };

bar: &foo
  frob: true

bar: &bar
  ~: ~

Special properties should work, albeit being a little bit cludgy.

> There appears to be no place to put metadata. The root of the tree is
> the top level of the YAML structure. There isn't any provision for
> having a top level object to hold things like the memreserve map. We
> may need a namespace to use for special properties that aren't nodes
> or properties.
> 

I don’t think that’s going to be necessary. Metadata are artifacts that
only have meaning when emitting a serialized form like dtb.

yamldt fully supports /memreserve/ by simply treating a top level
/memreserve/ property as special and constructing the memreserve
entries at the emit phase.

so 

/memreserve/ 1000 10;

is simply:

/memreserve/: [ 1000 10 ]

It’s a matter of definition what properties are encoded as metadata for the
serialized form.

> The encoding differentiates between nodes and properties implicitly
> base on whether the contents are a map, or a scalar/list. This does
> mean any parser needs to do a bit more work and it may impact what can
> be done with validation (I'm not going to talk about validation in
> this email though. We'll talk next week.)

Hmm, I’m not sure I follow this. Is that related to the metadata problem.

BTW, a blog post explaining the rationale behind yamldt is going to come
up soon at our website, I’ll post the link when it does.

> 
> Cheers,
> g.

Regards

— Pantelis

Re: [RFC] yamldt v0.5, now a DTS compiler too

[Grant Likely]

On Fri, Oct 20, 2017 at 8:16 PM, Pantelis Antoniou
<pantelis.antoniou-OWPKS81ov/FWk0Htik3J/w@public.gmane.org> wrote:
> Hi Grant,
>
>> On Oct 20, 2017, at 20:46 , Grant Likely <grant.likely-s3s/WqlpOiPyB63q8FvJNQ@public.gmane.org> wrote:
>>
>> On Thu, Sep 28, 2017 at 8:58 PM, Pantelis Antoniou
>> <pantelis.antoniou-OWPKS81ov/FWk0Htik3J/w@public.gmane.org> wrote:
>>> Hello again,
>>>
>>> Significant progress has been made on yamldt and is now capable of
>>> not only generating yaml from DTS source but also compiling DTS sources
>>> and being almost fully compatible with DTC.
>>>
>>> Compiling the kernel's DTBs using yamldt is as simple as using a
>>> DTC=yamldt.
>>>
>>> Error reporting is accurate and validation against a YAML based schema
>>> works as well. In a short while I will begin posting patches with
>>> fixes on bindings and DTS files in the kernel.
>>>
>>> Please try it on your platform and report if you encounter any problems.
>>>
>>> https://github.com/pantoniou/yamldt
>>>
>>> I am eagerly awaiting for your comments.
>>
>> Hi Pantelis,
>>
>> This is good work. I've played around with it and I'm looking forward
>> to chatting next week.
>>
>
> Thanks. I’m looking forward to it too.
>
>> One thing I've done is tried loading the output YAML files into
>> another YAML interpreter and the current encoding causes problems.
>> Specifically, in yamldt anchors/aliases are being used as a
>> replacement for labels/phandles, but that conflicts with the YAML data
>> model which defines a reference as a way to make a copy of the data
>> appear in another part of the tree. For example, for the following
>> snippit:
>>
>> intc: intc@10000 {
>>    #interrupt-cells = <1>;
>>    compatible = "acme,intc";
>>    reg = <0x10000 0x1000>;
>>    gpio-controller;
>> };
>>
>> serial@20000 {
>>    compatible = "acme,uart";
>>    reg = <0x20000 0x1000>;
>>    interrupt-parent = <&intc>;
>>    interrupts = <5>;
>> };
>>
>> yamldt will encode this as:
>>
>> intc@10000: &intc
>>    "#interrupt-cells": 1
>>    compatible: acme,intc
>>    reg: [0x10000, 0x1000]
>>    gpio-controller:
>>
>> serial@20000:
>>    compatible: acme,uart
>>    reg: [0x20000, 0x1000]
>>    interrupt-parent: *intc
>>    interrupts: 5
>>
>> But, the expected behaviour for a YAML parser is expand the alias
>> '*intc' which results in the following structure:
>>
>> intc@10000: &intc
>>    "#interrupt-cells": 1
>>    compatible: acme,intc
>>    reg: [0x10000, 0x1000]
>>    gpio-controller:
>>
>> serial@20000:
>>    compatible: acme,uart
>>    reg: [0x20000, 0x1000]
>>    interrupt-parent:
>>        "#interrupt-cells": 1
>>        compatible: acme,intc
>>        reg: [0x10000, 0x1000]
>>        gpio-controller:
>>    interrupts: 5
>>
>> See? It results in the entire interrupt controller node to appear as
>> an instance under the interrupt-parent property, when the intention is
>> only to create a phandle. yamldt should not redefine the behaviour of
>> '*' aliases. Instead, it should use a different indicator, either
>> using an explicit !phandle tag, or by replacing '*' with something
>> else. I worked around it in my tests by replacing '*' with '$’.
>
> Yes. This is expected. I don’t think pure YAML form is a good match for all
> the crazy things that are possible (and actually used in practice) with DTS.

I don’t think it is as dire as that. The DTS structure is not complex
and I think can easily be mapped into pure YAML. But, it does require
treating the DTS structure separately from its semantic meaning. For
example, the grammar of nodes, properties and labels easily maps to
pure YAML, but phandles and overlay trees have semantic meaning that
must be resolved by DT specific code. I’ll respond to you’re specific
examples below...

>
> For instance there’s no way a normal YAML parser won’t choke with something like
>
> / {
>   foo;
> };
>
> / {
>   bar;
> };
>
> Which is a common idiom in DTS files.

That’s only true when the top level of nodes is encoded as a map, but
it shouldn’t be. It should be a list instead, for two reasons. First,
order matters for the top level. Subsequent top level trees have to be
applied in order to fully resolve the tree, but a map doesn’t preserve
the ordering. Second, as you rightly point out, the same name can be
used for multiple top level trees. So, the encoding should be a list
with each list entry containing the node path {/path,&ref}, and the
node data. One possible way to do this is with a tuple:

treedata:
- - /
  - foo:
- - /
  - bar:

Another option is to add a special property in the node to contain the name:

treedata:
- $path: /
  foo:
- $path: /
  bar:

Personally, I prefer the special property approach. That would also be
a good way to encode labels

> The decision to use the YAML anchors and references was more borne out of a desire
> to sort of match conceptually the labels and references of DTS. It’s not a big
> issue to switch to something different.

I think it would be a good idea. One reason for defining a YAML
encoding is to be able to use existing tools to work with the data.
Changing the meaning of anchors and aliases breaks those tools
immediately.

> If we were to force YAML/DT file to be regular YAML files parseable by everything
> we’d have to rethink a lot of conventions that DT files currently use and I’m afraid
> a machine translation from DTS to YAML might not be feasible then.
>
> OTOH moving to that model might make it possible to use YAML constructs that are not
> mapped at all to DTS. For instance
>
> - foo: true
>   bar: “string”
>
> - foo: false
>   bar: “another-string”
>
> is not possible to be mapped to a DT node/property structure right now.

I’m not following. Are you saying DT has no way to encode a list of
nodes? What use case are you imagining?

>
>>
>
>> Plus, it would be useful to use normal YAML anchors/aliases for
>> creating node templates. For example:
>>
>> serial-template: &acme-uart . # The anchor for the template
>>    compatible: acme,uart
>>    interrupt-parent: *intc
>>
>> root:
>>    serial@20000:
>>        <<: *acme-uart   # Alias node merged into serial@20000
>>        interrupts: 5
>>        reg: [0x20000, 0x1000]
>>    serial@30000:
>>        <<: *acme-uart   # Alias node merged into serial@30000
>>        interrupts: 5
>>        reg: [0x30000, 0x1000]
>>
>
> Yes, I’m aware of this and I planned to talk to you about it :)
> It can be a very powerful way to cut down the churn of DT right now.
>
> It’s not going to be a problem for yamldt to support it (perhaps
> adding a new editing tags to allow more fine grained control of the
> substitions).
>
>> Another problem with anchors/references is YAML seems to require the
>> anchor to be defined before the reference, or at least that's what
>> pyyaml and ruamel both expect. Regardless, The chosen YAML encoding
>> should be readily consumable by existing yaml implementations without
>> having to do a lot of customization.
>>
>
> I think this is only true for streamable YAML implementations; yamldt
> can leave references unresolved until the emit phase (or even leave them
> be as object files).

it is possible (probable) that I’m just not familiar with how to
change that behaviour in pyyaml and ruamel. However, the point is moot
if we can agree to preserve pure YAML semantics.

>> I'm slightly concerned about using & anchors for labels because it
>> seems only one anchor can be defined per node, but DTC allows multiple
>> labels for a single node. This might not be an issue in practice
>> though. Another implementation issue related to using & anchors is the
>> YAML spec defines them as an encoding artifact, and parsers can
>> discard the anchor names after parsing the YAML structure, which is a
>> problem if we use something like $name to reference an anchor. The
>> solution might just be that labels need to go into a special property
>> so they don't disappear from the data stream.
>>
>
> I do support multiple labels to the same node in yamldt with the
> following trick, that uses an empty tree.
>
> foo: bar: baz { frob; };
>
> bar: &foo
>   frob: true
>
> bar: &bar
>   ~: ~
>
> Special properties should work, albeit being a little bit cludgy.

With special properties, it might look like this:

baz:
  $labels: [foo, bar]
  frob:

>
>> There appears to be no place to put metadata. The root of the tree is
>> the top level of the YAML structure. There isn't any provision for
>> having a top level object to hold things like the memreserve map. We
>> may need a namespace to use for special properties that aren't nodes
>> or properties.
>>
>
> I don’t think that’s going to be necessary. Metadata are artifacts that
> only have meaning when emitting a serialized form like dtb.
>
> yamldt fully supports /memreserve/ by simply treating a top level
> /memreserve/ property as special and constructing the memreserve
> entries at the emit phase.
>
> so
>
> /memreserve/ 1000 10;
>
> is simply:
>
> /memreserve/: [ 1000 10 ]
>
> It’s a matter of definition what properties are encoded as metadata for the
> serialized form.
>
>> The encoding differentiates between nodes and properties implicitly
>> base on whether the contents are a map, or a scalar/list. This does
>> mean any parser needs to do a bit more work and it may impact what can
>> be done with validation (I'm not going to talk about validation in
>> this email though. We'll talk next week.)
>
> Hmm, I’m not sure I follow this. Is that related to the metadata problem.

It has to do with iterating over properties or nodes. Nodes and props
are mixed together, and any code processing them has to explicitly
check the type to differentiate between the two. The alternative would
be to collect all properties under a single key:

parent:
  $props:
    foo: hello
    bar: world
  child1: {}
  child2: {}

>
> BTW, a blog post explaining the rationale behind yamldt is going to come
> up soon at our website, I’ll post the link when it does.

Cool. I see it got posted, but didn’t get a chance to read it before I
got on this airplane. Will try to read this evening.

(BTW, I’m arriving at about 11:00 tonight)

Cheers,
g.
--
To unsubscribe from this list: send the line "unsubscribe devicetree" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [RFC] yamldt v0.5, now a DTS compiler too

[Pantelis Antoniou]

Hi Grant,

> On Oct 22, 2017, at 19:54 , Grant Likely <grant.likely-s3s/WqlpOiPyB63q8FvJNQ@public.gmane.org> wrote:
> 
> On Fri, Oct 20, 2017 at 8:16 PM, Pantelis Antoniou
> <pantelis.antoniou-OWPKS81ov/FWk0Htik3J/w@public.gmane.org> wrote:
>> Hi Grant,
>> 
>>> On Oct 20, 2017, at 20:46 , Grant Likely <grant.likely-s3s/WqlpOiPyB63q8FvJNQ@public.gmane.org> wrote:
>>> 
>>> On Thu, Sep 28, 2017 at 8:58 PM, Pantelis Antoniou
>>> <pantelis.antoniou-OWPKS81ov/FWk0Htik3J/w@public.gmane.org> wrote:
>>>> Hello again,
>>>> 
>>>> Significant progress has been made on yamldt and is now capable of
>>>> not only generating yaml from DTS source but also compiling DTS sources
>>>> and being almost fully compatible with DTC.
>>>> 
>>>> Compiling the kernel's DTBs using yamldt is as simple as using a
>>>> DTC=yamldt.
>>>> 
>>>> Error reporting is accurate and validation against a YAML based schema
>>>> works as well. In a short while I will begin posting patches with
>>>> fixes on bindings and DTS files in the kernel.
>>>> 
>>>> Please try it on your platform and report if you encounter any problems.
>>>> 
>>>> https://github.com/pantoniou/yamldt
>>>> 
>>>> I am eagerly awaiting for your comments.
>>> 
>>> Hi Pantelis,
>>> 
>>> This is good work. I've played around with it and I'm looking forward
>>> to chatting next week.
>>> 
>> 
>> Thanks. I’m looking forward to it too.
>> 
>>> One thing I've done is tried loading the output YAML files into
>>> another YAML interpreter and the current encoding causes problems.
>>> Specifically, in yamldt anchors/aliases are being used as a
>>> replacement for labels/phandles, but that conflicts with the YAML data
>>> model which defines a reference as a way to make a copy of the data
>>> appear in another part of the tree. For example, for the following
>>> snippit:
>>> 
>>> intc: intc@10000 {
>>>   #interrupt-cells = <1>;
>>>   compatible = "acme,intc";
>>>   reg = <0x10000 0x1000>;
>>>   gpio-controller;
>>> };
>>> 
>>> serial@20000 {
>>>   compatible = "acme,uart";
>>>   reg = <0x20000 0x1000>;
>>>   interrupt-parent = <&intc>;
>>>   interrupts = <5>;
>>> };
>>> 
>>> yamldt will encode this as:
>>> 
>>> intc@10000: &intc
>>>   "#interrupt-cells": 1
>>>   compatible: acme,intc
>>>   reg: [0x10000, 0x1000]
>>>   gpio-controller:
>>> 
>>> serial@20000:
>>>   compatible: acme,uart
>>>   reg: [0x20000, 0x1000]
>>>   interrupt-parent: *intc
>>>   interrupts: 5
>>> 
>>> But, the expected behaviour for a YAML parser is expand the alias
>>> '*intc' which results in the following structure:
>>> 
>>> intc@10000: &intc
>>>   "#interrupt-cells": 1
>>>   compatible: acme,intc
>>>   reg: [0x10000, 0x1000]
>>>   gpio-controller:
>>> 
>>> serial@20000:
>>>   compatible: acme,uart
>>>   reg: [0x20000, 0x1000]
>>>   interrupt-parent:
>>>       "#interrupt-cells": 1
>>>       compatible: acme,intc
>>>       reg: [0x10000, 0x1000]
>>>       gpio-controller:
>>>   interrupts: 5
>>> 
>>> See? It results in the entire interrupt controller node to appear as
>>> an instance under the interrupt-parent property, when the intention is
>>> only to create a phandle. yamldt should not redefine the behaviour of
>>> '*' aliases. Instead, it should use a different indicator, either
>>> using an explicit !phandle tag, or by replacing '*' with something
>>> else. I worked around it in my tests by replacing '*' with '$’.
>> 
>> Yes. This is expected. I don’t think pure YAML form is a good match for all
>> the crazy things that are possible (and actually used in practice) with DTS.
> 
> I don’t think it is as dire as that. The DTS structure is not complex
> and I think can easily be mapped into pure YAML. But, it does require
> treating the DTS structure separately from its semantic meaning. For
> example, the grammar of nodes, properties and labels easily maps to
> pure YAML, but phandles and overlay trees have semantic meaning that
> must be resolved by DT specific code. I’ll respond to you’re specific
> examples below…
> 

We are in complete agreement here. Single nodes and properties map to YAML perfectly.
It’s the complex way that we build the complete DTB that stress the YAML structures.

>> 
>> For instance there’s no way a normal YAML parser won’t choke with something like
>> 
>> / {
>>  foo;
>> };
>> 
>> / {
>>  bar;
>> };
>> 
>> Which is a common idiom in DTS files.
> 
> That’s only true when the top level of nodes is encoded as a map, but
> it shouldn’t be. It should be a list instead, for two reasons. First,
> order matters for the top level. Subsequent top level trees have to be
> applied in order to fully resolve the tree, but a map doesn’t preserve
> the ordering. Second, as you rightly point out, the same name can be
> used for multiple top level trees. So, the encoding should be a list
> with each list entry containing the node path {/path,&ref}, and the
> node data. One possible way to do this is with a tuple:
> 
> treedata:
> - - /
>  - foo:
> - - /
>  - bar:
> 
> Another option is to add a special property in the node to contain the name:
> 
> treedata:
> - $path: /
>  foo:
> - $path: /
>  bar:
> 
> Personally, I prefer the special property approach. That would also be
> a good way to encode labels
> 

The YAML bare sequence is problematic; I’ll explain below.
 
>> The decision to use the YAML anchors and references was more borne out of a desire
>> to sort of match conceptually the labels and references of DTS. It’s not a big
>> issue to switch to something different.
> 
> I think it would be a good idea. One reason for defining a YAML
> encoding is to be able to use existing tools to work with the data.
> Changing the meaning of anchors and aliases breaks those tools
> immediately.
> 

Hmm, I see what you mean. Semantically this mapping is more correct but it does
present problems to non-aware tools.

>> If we were to force YAML/DT file to be regular YAML files parseable by everything
>> we’d have to rethink a lot of conventions that DT files currently use and I’m afraid
>> a machine translation from DTS to YAML might not be feasible then.
>> 
>> OTOH moving to that model might make it possible to use YAML constructs that are not
>> mapped at all to DTS. For instance
>> 
>> - foo: true
>>  bar: “string”
>> 
>> - foo: false
>>  bar: “another-string”
>> 
>> is not possible to be mapped to a DT node/property structure right now.
> 
> I’m not following. Are you saying DT has no way to encode a list of
> nodes? What use case are you imagining?
> 

This threw me off too at first.

The problem is when you try to convert this into a live tree structure.

The YAML encoding is

+SEQ
 +MAP
  =VAL :foo
  =VAL :true
  =VAL :bar
  =VAL "string
 -MAP
 +MAP
  =VAL :foo
  =VAL :false
  =VAL :bar
  =VAL "another-string
 -MAP
-SEQ

So it’s a sequence of MAPs.

In a live tree DTS form would be

noname-property = { foo=true; bar = “string”; }, { foo = false; bar=“another-string”; };

We don’t have the capability as right now to express something like this.

Namely properties containing nodes, and the root of the live tree not being a node.

>> 
>>> 
>> 
>>> Plus, it would be useful to use normal YAML anchors/aliases for
>>> creating node templates. For example:
>>> 
>>> serial-template: &acme-uart . # The anchor for the template
>>>   compatible: acme,uart
>>>   interrupt-parent: *intc
>>> 
>>> root:
>>>   serial@20000:
>>>       <<: *acme-uart   # Alias node merged into serial@20000
>>>       interrupts: 5
>>>       reg: [0x20000, 0x1000]
>>>   serial@30000:
>>>       <<: *acme-uart   # Alias node merged into serial@30000
>>>       interrupts: 5
>>>       reg: [0x30000, 0x1000]
>>> 
>> 
>> Yes, I’m aware of this and I planned to talk to you about it :)
>> It can be a very powerful way to cut down the churn of DT right now.
>> 
>> It’s not going to be a problem for yamldt to support it (perhaps
>> adding a new editing tags to allow more fine grained control of the
>> substitions).
>> 
>>> Another problem with anchors/references is YAML seems to require the
>>> anchor to be defined before the reference, or at least that's what
>>> pyyaml and ruamel both expect. Regardless, The chosen YAML encoding
>>> should be readily consumable by existing yaml implementations without
>>> having to do a lot of customization.
>>> 
>> 
>> I think this is only true for streamable YAML implementations; yamldt
>> can leave references unresolved until the emit phase (or even leave them
>> be as object files).
> 
> it is possible (probable) that I’m just not familiar with how to
> change that behaviour in pyyaml and ruamel. However, the point is moot
> if we can agree to preserve pure YAML semantics.
> 

Agreed.

>>> I'm slightly concerned about using & anchors for labels because it
>>> seems only one anchor can be defined per node, but DTC allows multiple
>>> labels for a single node. This might not be an issue in practice
>>> though. Another implementation issue related to using & anchors is the
>>> YAML spec defines them as an encoding artifact, and parsers can
>>> discard the anchor names after parsing the YAML structure, which is a
>>> problem if we use something like $name to reference an anchor. The
>>> solution might just be that labels need to go into a special property
>>> so they don't disappear from the data stream.
>>> 
>> 
>> I do support multiple labels to the same node in yamldt with the
>> following trick, that uses an empty tree.
>> 
>> foo: bar: baz { frob; };
>> 
>> bar: &foo
>>  frob: true
>> 
>> bar: &bar
>>  ~: ~
>> 
>> Special properties should work, albeit being a little bit cludgy.
> 
> With special properties, it might look like this:
> 
> baz:
>  $labels: [foo, bar]
>  frob:
> 

Yeah, that would work.

>> 
>>> There appears to be no place to put metadata. The root of the tree is
>>> the top level of the YAML structure. There isn't any provision for
>>> having a top level object to hold things like the memreserve map. We
>>> may need a namespace to use for special properties that aren't nodes
>>> or properties.
>>> 
>> 
>> I don’t think that’s going to be necessary. Metadata are artifacts that
>> only have meaning when emitting a serialized form like dtb.
>> 
>> yamldt fully supports /memreserve/ by simply treating a top level
>> /memreserve/ property as special and constructing the memreserve
>> entries at the emit phase.
>> 
>> so
>> 
>> /memreserve/ 1000 10;
>> 
>> is simply:
>> 
>> /memreserve/: [ 1000 10 ]
>> 
>> It’s a matter of definition what properties are encoded as metadata for the
>> serialized form.
>> 
>>> The encoding differentiates between nodes and properties implicitly
>>> base on whether the contents are a map, or a scalar/list. This does
>>> mean any parser needs to do a bit more work and it may impact what can
>>> be done with validation (I'm not going to talk about validation in
>>> this email though. We'll talk next week.)
>> 
>> Hmm, I’m not sure I follow this. Is that related to the metadata problem.
> 
> It has to do with iterating over properties or nodes. Nodes and props
> are mixed together, and any code processing them has to explicitly
> check the type to differentiate between the two. The alternative would
> be to collect all properties under a single key:
> 
> parent:
>  $props:
>    foo: hello
>    bar: world
>  child1: {}
>  child2: {}
> 

That’s no problem. It does look weird though. I am using the same parser
for parsing bindings and this might make the files a little bit weird.

Keep in mind that we have two YAML formats to parse; the hardware description
and the bindings describing them.

>> 
>> BTW, a blog post explaining the rationale behind yamldt is going to come
>> up soon at our website, I’ll post the link when it does.
> 
> Cool. I see it got posted, but didn’t get a chance to read it before I
> got on this airplane. Will try to read this evening.
> 
> (BTW, I’m arriving at about 11:00 tonight)
> 

Have a safe flight, you’ll get to beat me up tomorrow :)

> Cheers,
> g.

Regards

— Pantelis

Re: [RFC] yamldt v0.5, now a DTS compiler too

[Grant Likely]

On Sun, Oct 22, 2017 at 7:23 PM, Pantelis Antoniou
<pantelis.antoniou-OWPKS81ov/FWk0Htik3J/w@public.gmane.org> wrote:
> Hi Grant,
>
>> On Oct 22, 2017, at 19:54 , Grant Likely <grant.likely-s3s/WqlpOiPyB63q8FvJNQ@public.gmane.org> wrote:
>>
>> On Fri, Oct 20, 2017 at 8:16 PM, Pantelis Antoniou
>> <pantelis.antoniou-OWPKS81ov/FWk0Htik3J/w@public.gmane.org> wrote:
>>> Hi Grant,
>>>
>>>> On Oct 20, 2017, at 20:46 , Grant Likely <grant.likely-s3s/WqlpOiPyB63q8FvJNQ@public.gmane.org> wrote:
>>>>
>>>> On Thu, Sep 28, 2017 at 8:58 PM, Pantelis Antoniou
>>>> <pantelis.antoniou-OWPKS81ov/FWk0Htik3J/w@public.gmane.org> wrote:
[...]
>>> Yes. This is expected. I don’t think pure YAML form is a good match for all
>>> the crazy things that are possible (and actually used in practice) with DTS.
>>
>> I don’t think it is as dire as that. The DTS structure is not complex
>> and I think can easily be mapped into pure YAML. But, it does require
>> treating the DTS structure separately from its semantic meaning. For
>> example, the grammar of nodes, properties and labels easily maps to
>> pure YAML, but phandles and overlay trees have semantic meaning that
>> must be resolved by DT specific code. I’ll respond to you’re specific
>> examples below…
>>
>
> We are in complete agreement here. Single nodes and properties map to YAML perfectly.
> It’s the complex way that we build the complete DTB that stress the YAML structures.

Okay, good.

>>> If we were to force YAML/DT file to be regular YAML files parseable by everything
>>> we’d have to rethink a lot of conventions that DT files currently use and I’m afraid
>>> a machine translation from DTS to YAML might not be feasible then.
>>>
>>> OTOH moving to that model might make it possible to use YAML constructs that are not
>>> mapped at all to DTS. For instance
>>>
>>> - foo: true
>>>  bar: “string”
>>>
>>> - foo: false
>>>  bar: “another-string”
>>>
>>> is not possible to be mapped to a DT node/property structure right now.
>>
>> I’m not following. Are you saying DT has no way to encode a list of
>> nodes? What use case are you imagining?
>>
>
> This threw me off too at first.
>
> The problem is when you try to convert this into a live tree structure.
>
> The YAML encoding is
>
> +SEQ
>  +MAP
>   =VAL :foo
>   =VAL :true
>   =VAL :bar
>   =VAL "string
>  -MAP
>  +MAP
>   =VAL :foo
>   =VAL :false
>   =VAL :bar
>   =VAL "another-string
>  -MAP
> -SEQ
>
> So it’s a sequence of MAPs.
>
> In a live tree DTS form would be
>
> noname-property = { foo=true; bar = “string”; }, { foo = false; bar=“another-string”; };
>
> We don’t have the capability as right now to express something like this.
>
> Namely properties containing nodes, and the root of the live tree not being a node.

Ah, so you're proposing property data that encodes key/value pairs,
correct? Effectively adding another layer of namespace under a
property, but that namespace is invisible to and generic DT parsing
code in an OS (for how DT is defined today).

I would straight out make that structure illegal in the YAML DT
metaschema until such time as the semantics are defined. For now
properties should only be allowed to contain scalars, or lists of
scalars/lists which can be nested.

[...]

>>>> The encoding differentiates between nodes and properties implicitly
>>>> base on whether the contents are a map, or a scalar/list. This does
>>>> mean any parser needs to do a bit more work and it may impact what can
>>>> be done with validation (I'm not going to talk about validation in
>>>> this email though. We'll talk next week.)
>>>
>>> Hmm, I’m not sure I follow this. Is that related to the metadata problem.
>>
>> It has to do with iterating over properties or nodes. Nodes and props
>> are mixed together, and any code processing them has to explicitly
>> check the type to differentiate between the two. The alternative would
>> be to collect all properties under a single key:
>>
>> parent:
>>  $props:
>>    foo: hello
>>    bar: world
>>  child1: {}
>>  child2: {}
>>
>
> That’s no problem. It does look weird though. I am using the same parser
> for parsing bindings and this might make the files a little bit weird.

That it does. I don’t feel strongly about either approach, but I want
to see how the different encodings affect doing the schema bits.

>
> Keep in mind that we have two YAML formats to parse; the hardware description
> and the bindings describing them.

Three! Hw description, schema & metaschema. :-) But metaschema should
be a normal jsonschema file.

>
>>>
>>> BTW, a blog post explaining the rationale behind yamldt is going to come
>>> up soon at our website, I’ll post the link when it does.
>>
>> Cool. I see it got posted, but didn’t get a chance to read it before I
>> got on this airplane. Will try to read this evening.
>>
>> (BTW, I’m arriving at about 11:00 tonight)
>>
>
> Have a safe flight, you’ll get to beat me up tomorrow :)

See you tomorrow. It will be good to catch up.

g.
--
To unsubscribe from this list: send the line "unsubscribe devicetree" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [RFC] yamldt v0.5, now a DTS compiler too

[Pantelis Antoniou]

Hi all,

I’ve just put up a blog-entry explaining the rationale behind yaml and device tree.

https://www.konsulko.com/yaml-and-device-tree/

Please give it a quick glance and give feedback if you can.

Regards

— Pantelis