dtc symbolic constants - not in the preprocessor?

dtc symbolic constants - not in the preprocessor?

[Stephen Warren]

So, David has been leaning towards a simple text-based preprocessor as
the means to add symbolic constants to dtc. Depending on exactly how
much advanced expression support the dtc language ends up gaining, I'm
not so sure a text-based preprocessor is the right approach.

So, a text-based preprocessor is fine for basic stuff like:

#define FOO 5

/ {
    prop = <(FOO)>;
};

However, if the dtc language itself gains functions, iteration, etc., I
think this breaks down.

Using cpp, we could do something like:

#define TEGRA_USB1_INT_STATUS 0x7000000
#define POS(chip, reg, field) chip##_##reg##_##field##_POS

/ {
    prop1 = <(POS(TEGRA, USB1, INT_STATUS))>;
};

where you can imagine that the POS macro is calculating the name of a
symbolic constant, then reading the value of the name.

However, what if instead of hard-coding USB1 in the above example, we
want to iterate over 1, 2, 3? Expressing this iteration as a set of
functions in the dtc grammar (as in functional programming) has been
floated as an idea. In other words generate a result like:

/ {
    prop1 = <(POS(TEGRA, USB, 1, INT_STATUS))>;
    prop2 = <(POS(TEGRA, USB, 2, INT_STATUS))>;
    prop3 = <(POS(TEGRA, USB, 3, INT_STATUS))>;
};

... but using iteration in the dtc language instead of cut/paste.

If this iteration happens in the dtc language phase rather than the
preprocessing phase, then all the named constants known to the
preprocessing phase have been thrown away, and hence could never be
accessed by any functions/macros/... executing in the dtc language phase.

For this reason, I think that symbolic constants should probably be
something at the dtc language level rather than there being a separate
preprocessing phase, if we envisage dtc ever gaining anything more
complex than simple expression and symbolic constant support.

Do people agree with this?



If so, the simplest symbolic constant syntax might be:

/define/ FOO 5;

/ {
    prop = <(FOO)>;
};

(this is roughly what was in my somewhat recent /define/ patch proposal).



As an equivalent of the cpp ## operator, we could imagine something like:

/define/ TEGRA_USB1_INT_STATUS 0x7000000;
/function/ POS(chip, reg, field)
    readvar(chip + "_" + reg + "_" + field + "_POS");

/ {
    prop = <(POS("TEGRA", "USB1", "INT_STATUS"))>;
};

where readvar() is a built-in function that reads from the symbol table.



Following on from this, I've been envisaging symbolic constants holding
single integer values (perhaps strings too), as in the "FOO" example a
little above.

Simon Glass was thinking (admittedly I believe more in the context of
plain text-expansion) of allowing symbolic constants to be more than
just single integers, perhaps:

/define/ GDB_BASE 0x00e08000;
/define/ CHROME_OS_BOOT_DEVICES "emmc", "spi";
/define/ UART_BAUD_OPTIONS <115200 57600 19200>;

/ {
    prop1 = <GDB_BASE>;
    prop2 = <CHROME_OS_BOOT_DEVICES>;
    prop3 = <UART_BAUD_OPTIONS>;
};

(The syntax re: where <> appear might need some development in this
example.)

The idea here was that (some) variables might contain a sequence of
bytes just would just be dumped directly into the property data without
much interpretation.

This would presumably require symbolic constants to be typed somehow, so
that dtc would know that (GDB_BASE + 0x100) made sense, whereas
(UART_BAUD_OPTIONS + 0x100) didn't.

Re: dtc symbolic constants - not in the preprocessor?

[Simon Glass]

Hi Stephen,

On Tue, Apr 17, 2012 at 7:44 PM, Stephen Warren <swarren-3lzwWm7+Weoh9ZMKESR00Q@public.gmane.org> wrote:
> So, David has been leaning towards a simple text-based preprocessor as
> the means to add symbolic constants to dtc. Depending on exactly how
> much advanced expression support the dtc language ends up gaining, I'm
> not so sure a text-based preprocessor is the right approach.
>
> So, a text-based preprocessor is fine for basic stuff like:
>
> #define FOO 5
>
> / {
>    prop = <(FOO)>;
> };
>
> However, if the dtc language itself gains functions, iteration, etc., I
> think this breaks down.
>
> Using cpp, we could do something like:
>
> #define TEGRA_USB1_INT_STATUS 0x7000000
> #define POS(chip, reg, field) chip##_##reg##_##field##_POS
>
> / {
>    prop1 = <(POS(TEGRA, USB1, INT_STATUS))>;
> };
>
> where you can imagine that the POS macro is calculating the name of a
> symbolic constant, then reading the value of the name.
>
> However, what if instead of hard-coding USB1 in the above example, we
> want to iterate over 1, 2, 3? Expressing this iteration as a set of
> functions in the dtc grammar (as in functional programming) has been
> floated as an idea. In other words generate a result like:
>
> / {
>    prop1 = <(POS(TEGRA, USB, 1, INT_STATUS))>;
>    prop2 = <(POS(TEGRA, USB, 2, INT_STATUS))>;
>    prop3 = <(POS(TEGRA, USB, 3, INT_STATUS))>;
> };
>
> ... but using iteration in the dtc language instead of cut/paste.
>
> If this iteration happens in the dtc language phase rather than the
> preprocessing phase, then all the named constants known to the
> preprocessing phase have been thrown away, and hence could never be
> accessed by any functions/macros/... executing in the dtc language phase.
>
> For this reason, I think that symbolic constants should probably be
> something at the dtc language level rather than there being a separate
> preprocessing phase, if we envisage dtc ever gaining anything more
> complex than simple expression and symbolic constant support.
>
> Do people agree with this?

Yes I strongly agree that symbolic constants should be in the dtc
language. I don't like the brackets though...

>
>
>
> If so, the simplest symbolic constant syntax might be:
>
> /define/ FOO 5;
>
> / {
>    prop = <(FOO)>;
> };
>
> (this is roughly what was in my somewhat recent /define/ patch proposal).
>
>
>
> As an equivalent of the cpp ## operator, we could imagine something like:
>
> /define/ TEGRA_USB1_INT_STATUS 0x7000000;
> /function/ POS(chip, reg, field)
>    readvar(chip + "_" + reg + "_" + field + "_POS");
>
> / {
>    prop = <(POS("TEGRA", "USB1", "INT_STATUS"))>;
> };
>
> where readvar() is a built-in function that reads from the symbol table.
>
>
>
> Following on from this, I've been envisaging symbolic constants holding
> single integer values (perhaps strings too), as in the "FOO" example a
> little above.
>
> Simon Glass was thinking (admittedly I believe more in the context of
> plain text-expansion) of allowing symbolic constants to be more than
> just single integers, perhaps:
>
> /define/ GDB_BASE 0x00e08000;

If it matters I had /define/ GDB_BASE <0x00e08000> for this.

> /define/ CHROME_OS_BOOT_DEVICES "emmc", "spi";
> /define/ UART_BAUD_OPTIONS <115200 57600 19200>;
>
> / {
>    prop1 = <GDB_BASE>;
>    prop2 = <CHROME_OS_BOOT_DEVICES>;
>    prop3 = <UART_BAUD_OPTIONS>;
> };
>
> (The syntax re: where <> appear might need some development in this
> example.)

Yes.

>
> The idea here was that (some) variables might contain a sequence of
> bytes just would just be dumped directly into the property data without
> much interpretation.
>
> This would presumably require symbolic constants to be typed somehow, so
> that dtc would know that (GDB_BASE + 0x100) made sense, whereas
> (UART_BAUD_OPTIONS + 0x100) didn't.

Perhaps, although maybe using /define/ for both is a bad idea.

Actually I was just thinking of dumping the data in. I like your
suggestion in that thread about /defprop/ or /defdata/ instead
actually - so these would just be blobs of data with no value and not
suitable to use in expressions. In other words this feature would be
separate from first class variables / symbols mentioned above which
can be operated on. Perhaps a bit like strings in other languages -
just a sequence of cells / bytes.

The patch is available here, although the title is a misnomer now. It
is just a modification of Stephen's patch mentioned above:

https://gerrit.chromium.org/gerrit/#change,19855

In any case I strongly support efforts to define support for symbolic
constants, one way or another. Looking forward to the outcome!

Regards,
Simon

Re: dtc symbolic constants - not in the preprocessor?

[David Gibson]

On Tue, Apr 17, 2012 at 08:44:23PM -0600, Stephen Warren wrote:
> So, David has been leaning towards a simple text-based preprocessor as
> the means to add symbolic constants to dtc. Depending on exactly how
> much advanced expression support the dtc language ends up gaining, I'm
> not so sure a text-based preprocessor is the right approach.
> 
> So, a text-based preprocessor is fine for basic stuff like:
> 
> #define FOO 5
> 
> / {
>     prop = <(FOO)>;
> };

Note also that this will work nicely:

#define FOO(x)	(/* something */)

	prop = <FOO(1)  FOO(2)  FOO(17)>;

> However, if the dtc language itself gains functions, iteration, etc., I
> think this breaks down.

Well, for starters I think gaining functions and gaining iteration are
quite different cases.  I see using a preprocessor as a way of
avoiding gaining functions in the core language (and particularly
avoiding the necessity of storing expression trees that that would
require).

> Using cpp, we could do something like:
> 
> #define TEGRA_USB1_INT_STATUS 0x7000000
> #define POS(chip, reg, field) chip##_##reg##_##field##_POS
> 
> / {
>     prop1 = <(POS(TEGRA, USB1, INT_STATUS))>;
> };
> 
> where you can imagine that the POS macro is calculating the name of a
> symbolic constant, then reading the value of the name.
> 
> However, what if instead of hard-coding USB1 in the above example, we
> want to iterate over 1, 2, 3? Expressing this iteration as a set of
> functions in the dtc grammar (as in functional programming) has been
> floated as an idea. In other words generate a result like:
> 
> / {
>     prop1 = <(POS(TEGRA, USB, 1, INT_STATUS))>;
>     prop2 = <(POS(TEGRA, USB, 2, INT_STATUS))>;
>     prop3 = <(POS(TEGRA, USB, 3, INT_STATUS))>;
> };
> 
> ... but using iteration in the dtc language instead of cut/paste.

Um.. this seems a contrived example - combining iteration with
construction of property names and symbol names.  The sorts of
iteration I had in mind were more things like generating:
	prop = <0 0x1000 0x2000 0x3000>;
programmatically.  That example could be built using an iterator
built-in "function", which could in turn be generated from a macro.

Do you have a real use case for something more like the example above?

I can also see uses for some sort of foreach construct, which is
conceivable both at the language and preprocessor level, and there are
several possible variants.

> If this iteration happens in the dtc language phase rather than the
> preprocessing phase, then all the named constants known to the
> preprocessing phase have been thrown away, and hence could never be
> accessed by any functions/macros/... executing in the dtc language phase.
> 
> For this reason, I think that symbolic constants should probably be
> something at the dtc language level rather than there being a separate
> preprocessing phase, if we envisage dtc ever gaining anything more
> complex than simple expression and symbolic constant support.

Well.. if we have defined functions at the language level, then we
certainly want constants at that level as well (and it will be
trivial, since that's basically just a 0 argument function).  But as
yet I haven't seen a really convincing argument that we do need such a
thing - I still lean towards preprocessor macros as giving us a good
flexibility to syntactic complexity ration.

And in fact, because of this I really only want to implement either
*both* functions and constants in dtc, or neither - not one without
the other.  So far, I'm still hoping to avoid it.

> Do people agree with this?
> 
> 
> 
> If so, the simplest symbolic constant syntax might be:
> 
> /define/ FOO 5;
> 
> / {
>     prop = <(FOO)>;
> };
> 
> (this is roughly what was in my somewhat recent /define/ patch proposal).
> 
> 
> 
> As an equivalent of the cpp ## operator, we could imagine something like:
> 
> /define/ TEGRA_USB1_INT_STATUS 0x7000000;
> /function/ POS(chip, reg, field)
>     readvar(chip + "_" + reg + "_" + field + "_POS");
> 
> / {
>     prop = <(POS("TEGRA", "USB1", "INT_STATUS"))>;
> };
> 
> where readvar() is a built-in function that reads from the symbol
> table.

Urg.  If you want symbol name construction (and pass by name) it
really seems more like you want macro preprocessing rather than
language constructs.

I could see an argument here for not using cpp but our own macro
language that was similar but with a few extensions (a call-by-name
foreach construct is the most obvious one to me).

> Following on from this, I've been envisaging symbolic constants holding
> single integer values (perhaps strings too), as in the "FOO" example a
> little above.
> 
> Simon Glass was thinking (admittedly I believe more in the context of
> plain text-expansion) of allowing symbolic constants to be more than
> just single integers, perhaps:
> 
> /define/ GDB_BASE 0x00e08000;
> /define/ CHROME_OS_BOOT_DEVICES "emmc", "spi";
> /define/ UART_BAUD_OPTIONS <115200 57600 19200>;

If we have language level constants, I think they should be able to
contain any "type" that we can have an expression for, including
strings and bytestrings at least.  And as I said I don't really want
to do that without having functions as well.

> / {
>     prop1 = <GDB_BASE>;
>     prop2 = <CHROME_OS_BOOT_DEVICES>;
>     prop3 = <UART_BAUD_OPTIONS>;
> };
> 
> (The syntax re: where <> appear might need some development in this
> example.)

> The idea here was that (some) variables might contain a sequence of
> bytes just would just be dumped directly into the property data without
> much interpretation.
> 
> This would presumably require symbolic constants to be typed somehow, so
> that dtc would know that (GDB_BASE + 0x100) made sense, whereas
> (UART_BAUD_OPTIONS + 0x100) didn't.

And this involves a considerable bunch of infrastructure.  Which is
why I'm still hoping we can do what we need with textually expanded
macros rather than adding all the complexity of tracking types and
expression trees.

-- 
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson

Re: dtc symbolic constants - not in the preprocessor?

[David Gibson]

On Tue, Apr 17, 2012 at 09:39:55PM -0700, Simon Glass wrote:
[snip]
> Perhaps, although maybe using /define/ for both is a bad idea.
> 
> Actually I was just thinking of dumping the data in. I like your
> suggestion in that thread about /defprop/ or /defdata/ instead
> actually - so these would just be blobs of data with no value and not
> suitable to use in expressions. In other words this feature would be
> separate from first class variables / symbols mentioned above which
> can be operated on. Perhaps a bit like strings in other languages -
> just a sequence of cells / bytes.

Blech.  Different kinds of define with not just different data types,
but different capabilities?  I hate it.  If we have language level
defines, they should just be an expression with any of the types that
an expression can have.  That should include at least integers,
strings and bytestrings, and might include node bodies as well (the
later types would need suitable operators and builtin functions to be
useful).

> The patch is available here, although the title is a misnomer now. It
> is just a modification of Stephen's patch mentioned above:
> 
> https://gerrit.chromium.org/gerrit/#change,19855
> 
> In any case I strongly support efforts to define support for symbolic
> constants, one way or another. Looking forward to the outcome!
> 
> Regards,
> Simon
> 

-- 
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson

Re: dtc symbolic constants - not in the preprocessor?

[Stephen Warren]

On 04/19/2012 06:11 AM, David Gibson wrote:
> On Tue, Apr 17, 2012 at 08:44:23PM -0600, Stephen Warren wrote:
>> So, David has been leaning towards a simple text-based preprocessor as
>> the means to add symbolic constants to dtc. Depending on exactly how
>> much advanced expression support the dtc language ends up gaining, I'm
>> not so sure a text-based preprocessor is the right approach.
>>
>> So, a text-based preprocessor is fine for basic stuff like:
>>
>> #define FOO 5
>>
>> / {
>>     prop = <(FOO)>;
>> };
> 
> Note also that this will work nicely:
> 
> #define FOO(x)	(/* something */)
> 
> 	prop = <FOO(1)  FOO(2)  FOO(17)>;

I don't think that can be legal; there's no () around the FOO(1), so how
do you know whether it's meant to be two cells; the result of expanding
"FOO" and plain "(1)", or a call of a macro named FOO with parameter
"1"? In other words, shouldn't that be:

    prop = <(FOO(1)) (FOO(2)) (FOO(17))>;

Admittedly, the C pre-processor does appear to know whether FOO was
defined to accept arguments or not, and will differentiate between the
two cases based on that, but that seems pretty nasty to me. And this
relies on the body of FOO having () around it, which seems somewhat
implicit.

>> However, if the dtc language itself gains functions, iteration, etc., I
>> think this breaks down.
> 
> Well, for starters I think gaining functions and gaining iteration are
> quite different cases.  I see using a preprocessor as a way of
> avoiding gaining functions in the core language (and particularly
> avoiding the necessity of storing expression trees that that would
> require).
> 
>> Using cpp, we could do something like:
>>
>> #define TEGRA_USB1_INT_STATUS 0x7000000
>> #define POS(chip, reg, field) chip##_##reg##_##field##_POS
>>
>> / {
>>     prop1 = <(POS(TEGRA, USB1, INT_STATUS))>;
>> };
>>
>> where you can imagine that the POS macro is calculating the name of a
>> symbolic constant, then reading the value of the name.
>>
>> However, what if instead of hard-coding USB1 in the above example, we
>> want to iterate over 1, 2, 3? Expressing this iteration as a set of
>> functions in the dtc grammar (as in functional programming) has been
>> floated as an idea. In other words generate a result like:
>>
>> / {
>>     prop1 = <(POS(TEGRA, USB, 1, INT_STATUS))>;
>>     prop2 = <(POS(TEGRA, USB, 2, INT_STATUS))>;
>>     prop3 = <(POS(TEGRA, USB, 3, INT_STATUS))>;
>> };
>>
>> ... but using iteration in the dtc language instead of cut/paste.
> 
> Um.. this seems a contrived example - combining iteration with
> construction of property names and symbol names.  The sorts of
> iteration I had in mind were more things like generating:
> 	prop = <0 0x1000 0x2000 0x3000>;
> programmatically.  That example could be built using an iterator
> built-in "function", which could in turn be generated from a macro.
> 
> Do you have a real use case for something more like the example above?

I don't immediately foresee the need for any kind of iteration in the
device trees I'm building at the moment. Plain #define of simple
constants (integers or strings) would probably satisfy all the needs I'm
currently aware of.

But Jon's IIRC expressions patch included iteration, and you've posited
iteration functions, and I think in any case where iteration is allowed,
you will want the feature where you can define e.g. the base address of
two HW modules, and have the iteration "body" read from one of those
variables programatically rather than have the list of base addresses
passed in.

> And in fact, because of this I really only want to implement either
> *both* functions and constants in dtc, or neither - not one without
> the other.  So far, I'm still hoping to avoid it.

I don't understand the rationale behind this position. What's the root
of the problem that disallows us supporting named constants without
supporting full macros? I think plain named constants would cover the
majority of current use-cases, and I'm quite happy to stop thinking
about anything more advanced. The only reason I'm trying to drive other
stuff is because of your desire not to support constants without full
macros.

Re: dtc symbolic constants - not in the preprocessor?

[David Gibson]

On Tue, Apr 24, 2012 at 09:22:34PM -0600, Stephen Warren wrote:
> On 04/19/2012 06:11 AM, David Gibson wrote:
> > On Tue, Apr 17, 2012 at 08:44:23PM -0600, Stephen Warren wrote:
> >> So, David has been leaning towards a simple text-based preprocessor as
> >> the means to add symbolic constants to dtc. Depending on exactly how
> >> much advanced expression support the dtc language ends up gaining, I'm
> >> not so sure a text-based preprocessor is the right approach.
> >>
> >> So, a text-based preprocessor is fine for basic stuff like:
> >>
> >> #define FOO 5
> >>
> >> / {
> >>     prop = <(FOO)>;
> >> };
> > 
> > Note also that this will work nicely:
> > 
> > #define FOO(x)	(/* something */)
> > 
> > 	prop = <FOO(1)  FOO(2)  FOO(17)>;
> 
> I don't think that can be legal; there's no () around the FOO(1), so how
> do you know whether it's meant to be two cells; the result of expanding
> "FOO" and plain "(1)", or a call of a macro named FOO with parameter
> "1"? In other words, shouldn't that be:

It's a preprocessor.  By definition it knows nothing about the
surrounding syntax.  FOO(x) will just expand to what it expands to,
which may or may not be legal in context.  I this case it will be
because the definition has parens around it.

>     prop = <(FOO(1)) (FOO(2)) (FOO(17))>;

That might be a wiser way to structure it, but it's no more or less
valid from a preproc point of view.

> Admittedly, the C pre-processor does appear to know whether FOO was
> defined to accept arguments or not, and will differentiate between the
> two cases based on that, but that seems pretty nasty to me. And this
> relies on the body of FOO having () around it, which seems somewhat
> implicit.

Yes.  But that's all stuff you have to deal with for cpp already.
Remember the main target audience for dtc is C programmers, so the
least surprise principle should be interpreted in that context.

> >> However, if the dtc language itself gains functions, iteration, etc., I
> >> think this breaks down.
> > 
> > Well, for starters I think gaining functions and gaining iteration are
> > quite different cases.  I see using a preprocessor as a way of
> > avoiding gaining functions in the core language (and particularly
> > avoiding the necessity of storing expression trees that that would
> > require).
> > 
> >> Using cpp, we could do something like:
> >>
> >> #define TEGRA_USB1_INT_STATUS 0x7000000
> >> #define POS(chip, reg, field) chip##_##reg##_##field##_POS
> >>
> >> / {
> >>     prop1 = <(POS(TEGRA, USB1, INT_STATUS))>;
> >> };
> >>
> >> where you can imagine that the POS macro is calculating the name of a
> >> symbolic constant, then reading the value of the name.
> >>
> >> However, what if instead of hard-coding USB1 in the above example, we
> >> want to iterate over 1, 2, 3? Expressing this iteration as a set of
> >> functions in the dtc grammar (as in functional programming) has been
> >> floated as an idea. In other words generate a result like:
> >>
> >> / {
> >>     prop1 = <(POS(TEGRA, USB, 1, INT_STATUS))>;
> >>     prop2 = <(POS(TEGRA, USB, 2, INT_STATUS))>;
> >>     prop3 = <(POS(TEGRA, USB, 3, INT_STATUS))>;
> >> };
> >>
> >> ... but using iteration in the dtc language instead of cut/paste.
> > 
> > Um.. this seems a contrived example - combining iteration with
> > construction of property names and symbol names.  The sorts of
> > iteration I had in mind were more things like generating:
> > 	prop = <0 0x1000 0x2000 0x3000>;
> > programmatically.  That example could be built using an iterator
> > built-in "function", which could in turn be generated from a macro.
> > 
> > Do you have a real use case for something more like the example above?
> 
> I don't immediately foresee the need for any kind of iteration in the
> device trees I'm building at the moment. Plain #define of simple
> constants (integers or strings) would probably satisfy all the needs I'm
> currently aware of.
> 
> But Jon's IIRC expressions patch included iteration, and you've posited
> iteration functions, and I think in any case where iteration is allowed,
> you will want the feature where you can define e.g. the base address of
> two HW modules, and have the iteration "body" read from one of those
> variables programatically rather than have the list of base addresses
> passed in.

Hrm.  I think I'd need some more concrete examples to make sense of
that.

> > And in fact, because of this I really only want to implement either
> > *both* functions and constants in dtc, or neither - not one without
> > the other.  So far, I'm still hoping to avoid it.
> 
> I don't understand the rationale behind this position. What's the root
> of the problem that disallows us supporting named constants without
> supporting full macros? I think plain named constants would cover the
> majority of current use-cases, and I'm quite happy to stop thinking
> about anything more advanced. The only reason I'm trying to drive other
> stuff is because of your desire not to support constants without full
> macros.

Both functions and macros would - clearly - have constants as an
obvious special case (a zero argument macro/function).  I don't want
to end up with two different syntaxes for constants.

-- 
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson