DTS language enhancements

DTS language enhancements

[David Gibson]

Ok, as promised by thoughts (which have been simmering for a while) on
where we ought to go with DTS expression and whatnot support.

Expressions
===========

With any luck we can agree on what expressions should look like, even
if we're still arguing about when they ought to be evaluated.  My
approach here is to re-interpret some of the existing syntax as
expressions.  In addition to integer expressions, we also have
string/bytestring expressions, and node-content expressions.  That
lets the grammar for property definitions become:
	<property name> = <bytestring expression>;
And for node definitions it becomes:
	<node name> <node-content expression>;

Integer expressions
-------------------

These are the easy ones.  Literals are as in C, and all
side-effect-free C integer operators are supported (including
relationals, logicals and the ?: ternary).

String/Bytestring expressions
-----------------------------

We have two kinds of literal here:
	null-terminated strings:	"hello world\n"
	bytestrings:			[aabbccdd]

The < ... > construct which delimits a cell list becomes a special
operator which takes a list of integer expressions and returns a
bytestring value.

, is a bytestring append operator (note that this is different from a
string append operator, because it doesn't chop a terminating null
from the first argument).

?: would also be supported for bytestrings (first argument is integer,
second and third are bytestring).

We also probably want (but I don't have a specific syntax in mind for yet):
	- string append operator and/or a pythonesque "printf" operator
	- repeat operator (e.g. [aabbcc] * 3 == [aabbccaabbccaabbcc])

Node-content expressions
------------------------

Literals are node definitions in the current format:
	{
		prop = <bytestring expression>;
		somesubnode@whatever <node content expression>;
	}

?: would also be supported here (again, first argument integer, second
and third are node-content).

I'm less sure what other operators we'll need here - probably need to
build these based on actual usage examples.  Likely candidates,
however are:
	- set property
e.g. /setprop/({ }, "reg", < 17 >) == { reg = < 17 >; }
	- remove property
e.g. /delprop({ reg = <17>; }, "reg") == { }
	- add subnode
e.g. /addnode/({ }, "subnode@17", {reg = <17>;}) == 
	{ subnode@17 { reg = <17>; }; }
	- merge
e.g. /merge/({foo = "abc";}, {bar = <17>;}) == {foo = "abc"; bar=<17>;}
(this would recurse down subnodes with identical names)
	- get subnode
e.g. /getnode/({ subnode { foo = "bar"; }; }, "subnode") ==
	{ foo = "bar"; }
	- get property (result of this is a bytestring, not a
node-content expression)
e.g. /getprop/({ foo = "bar"; }, "foo") = "bar"

Expressions in node/property names
----------------------------------

As Jon's examples demonstrate, we need the ability to compute node
names at least with expressions, so that we can fill in appropriate
unit addresses.  I can't think of a use for computed property names
off hand, but if we can do one the other follows trivially so we might
as well allow both.

It's possible to do this just with the /setprop/, /addnode/ operators
described above, but that's awkward and verbose, so allowing
expressions in the same place the property/node names go now seems
better.  jdl's patch series allows this, but I'm not sure what makes
the grammatical distinction between the parser expecting a bare node
name and an expression, which worries me.

What I would suggest here is that expressions for node/property names
must be parenthesized.  ( and ) aren't used in node/property names
either in theory or practice AFAIK and this is consistent with integer
expressions having to be parenthesized within cell lists to avoid
ambiguity.

Expressions in labels
---------------------

Jon's patch also allows expressions instead of literals in labels.
I'm a lot more dubious about this feature: it's very un-C-like, and
removes the current nice lexical distinctness of labels.

But we probably do need some way of making computed labels for canned
devices of various sorts.  Of course, if we use preprocessed macro
expansion rather than runtime expression evaluation we can do that (as
its sometimes done in C code) by using the preprocessor's token
pasting operations.  We could also create a /label/ operator of some
sort.  Need more thought here.

Lexical issues of property / node names
=======================================

Property and node names are lexically troublesome because they can
contain a bunch of characters that would usually have special
meanings.  My intention for dealing with this is that property/node
names will only be lexed in a small number of contexts.  Usually they
will not be recognized, and we can lex C-like identifiers without
trouble.

Contexts which take bare propnodenames are always introduced by { }.
At most one bare propnodename will be recognized per logical-line (';'
delimited), and it must be the first token - we should be able to use
flex's state stack support to track this.

This lexical structure works for node definitions themselves { ... },
and for reference-to-path &{...}.  Any new constructs we introduce
which take bare propnodenames (as opposed to string expressions which
are interpreted as node or property names) will also use { } for
consistency.

Identifiers
===========

Whether we use preprocessing or post-parse evaluation, all identifiers
and parameters should be C style; that is [a-zA-Z_][a-zA-Z0-9_]*.  No
funny delimiters to mark them.  With the handling of propnodenames
described above, this shouldn't be problematic - any context will
either take a propnodename, or an identifier, but not both.

Preprocessing or post-parse evaluation
======================================

I think we need pretty much all the above, particularly the rich
expression support whether or not we have post-parse evaluation of
expressions, or whether we just have a preprocessor to help the user
built up complex constant expressions.

I favour preprocessing, because it's quite a simple extra step on top
of that which I think will give us what we need - at least if we have
the right expression operators available.  Post-parse processing
requires us to carry around expression trees, do more complex handling
of types and other extra work.  I'm not yet convinced (though I could
be) that this is worthwhile.

-- 
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson

Re: DTS language enhancements

[Kumar Gala]

On Oct 2, 2008, at 11:37 PM, David Gibson wrote:

> Node-content expressions
> ------------------------
>
> Literals are node definitions in the current format:
> 	{
> 		prop = <bytestring expression>;
> 		somesubnode@whatever <node content expression>;
> 	}
>
> ?: would also be supported here (again, first argument integer, second
> and third are node-content).
>
> I'm less sure what other operators we'll need here - probably need to
> build these based on actual usage examples.  Likely candidates,
> however are:
> 	- set property
> e.g. /setprop/({ }, "reg", < 17 >) == { reg = < 17 >; }
> 	- remove property
> e.g. /delprop({ reg = <17>; }, "reg") == { }
> 	- add subnode
> e.g. /addnode/({ }, "subnode@17", {reg = <17>;}) ==
> 	{ subnode@17 { reg = <17>; }; }
> 	- merge

should we have /delnode/

> e.g. /merge/({foo = "abc";}, {bar = <17>;}) == {foo = "abc";  
> bar=<17>;}
> (this would recurse down subnodes with identical names)
> 	- get subnode
> e.g. /getnode/({ subnode { foo = "bar"; }; }, "subnode") ==
> 	{ foo = "bar"; }
> 	- get property (result of this is a bytestring, not a
> node-content expression)
> e.g. /getprop/({ foo = "bar"; }, "foo") = "bar"

This is very interesting and seems to address some functionality I've  
talked to Jon about us needing to modify and update nodes.

- k

Re: DTS language enhancements

[David Gibson]

On Fri, Oct 03, 2008 at 12:08:54AM -0500, Kumar Gala wrote:
>
> On Oct 2, 2008, at 11:37 PM, David Gibson wrote:
>
>> Node-content expressions
>> ------------------------
>>
>> Literals are node definitions in the current format:
>> 	{
>> 		prop = <bytestring expression>;
>> 		somesubnode@whatever <node content expression>;
>> 	}
>>
>> ?: would also be supported here (again, first argument integer, second
>> and third are node-content).
>>
>> I'm less sure what other operators we'll need here - probably need to
>> build these based on actual usage examples.  Likely candidates,
>> however are:
>> 	- set property
>> e.g. /setprop/({ }, "reg", < 17 >) == { reg = < 17 >; }
>> 	- remove property
>> e.g. /delprop({ reg = <17>; }, "reg") == { }
>> 	- add subnode
>> e.g. /addnode/({ }, "subnode@17", {reg = <17>;}) ==
>> 	{ subnode@17 { reg = <17>; }; }
>> 	- merge
>
> should we have /delnode/

Ah, yeah, I guess so, if we've got the rest.  Possibly they should be
called /graft/ and /prune/ instead of /addnode/ and /delnode/ to
emphasise the fact that they may be handling whole subtrees.

>> e.g. /merge/({foo = "abc";}, {bar = <17>;}) == {foo = "abc";  
>> bar=<17>;}
>> (this would recurse down subnodes with identical names)
>> 	- get subnode
>> e.g. /getnode/({ subnode { foo = "bar"; }; }, "subnode") ==
>> 	{ foo = "bar"; }
>> 	- get property (result of this is a bytestring, not a
>> node-content expression)
>> e.g. /getprop/({ foo = "bar"; }, "foo") = "bar"
>
> This is very interesting and seems to address some functionality I've  
> talked to Jon about us needing to modify and update nodes.

Yeah, well, this is kind of my point.  I'm pretty sure we'll need this
stuff regardless of whether we use preprocessing or post-evaluation;
and once we have these sorts of operators, I'm not sure that
post-evaluation will actually buy us much.

-- 
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson

Re: DTS language enhancements

[Jon Loeliger]

Kumar Gala wrote:

>> e.g. /merge/({foo = "abc";}, {bar = <17>;}) == {foo = "abc"; bar=<17>;}
>> (this would recurse down subnodes with identical names)
>>     - get subnode
>> e.g. /getnode/({ subnode { foo = "bar"; }; }, "subnode") ==
>>     { foo = "bar"; }
>>     - get property (result of this is a bytestring, not a
>> node-content expression)
>> e.g. /getprop/({ foo = "bar"; }, "foo") = "bar"
> 
> This is very interesting and seems to address some functionality I've 
> talked to Jon about us needing to modify and update nodes.
> 
> - k

And Kumar, to be fair finish our conversation some more:
I *agreed* that we should get to that level of functionality as well,
and even said we'd work towards having it over time.  But, that we
wanted to make a first step towards a framework with something usable.

Feh.

I am *not* the bad-guy here.  For 3 years folks have been saying
"We need _something_...." and for three years no one did jack shit.
In the Open Source development world, the motto right behing
"post early, post often" is most assuredly "Show me the code".

Well, talk is cheap and proposing a vacuous and arguable something
has always been just that -- arguable.  So I am showing you the code
and giving you something to shoot at.

It's more than anyone else has done on this front.

jdl

Re: DTS language enhancements

[David Gibson]

On Fri, Oct 03, 2008 at 09:07:53AM -0500, Jon Loeliger wrote:
> Kumar Gala wrote:
>
>>> e.g. /merge/({foo = "abc";}, {bar = <17>;}) == {foo = "abc"; bar=<17>;}
>>> (this would recurse down subnodes with identical names)
>>>     - get subnode
>>> e.g. /getnode/({ subnode { foo = "bar"; }; }, "subnode") ==
>>>     { foo = "bar"; }
>>>     - get property (result of this is a bytestring, not a
>>> node-content expression)
>>> e.g. /getprop/({ foo = "bar"; }, "foo") = "bar"
>>
>> This is very interesting and seems to address some functionality I've  
>> talked to Jon about us needing to modify and update nodes.
>>
>> - k
>
> And Kumar, to be fair finish our conversation some more:
> I *agreed* that we should get to that level of functionality as well,
> and even said we'd work towards having it over time.  But, that we
> wanted to make a first step towards a framework with something usable.
>
> Feh.
>
> I am *not* the bad-guy here.  For 3 years folks have been saying
> "We need _something_...." and for three years no one did jack shit.

That's not entirely true.  I certainly should have been more public
about what I had, but I'm no guiltier than you.

> In the Open Source development world, the motto right behing
> "post early, post often" is most assuredly "Show me the code".
>
> Well, talk is cheap and proposing a vacuous and arguable something
> has always been just that -- arguable.  So I am showing you the code
> and giving you something to shoot at.
>
> It's more than anyone else has done on this front.

Bear in mind that when I posted my expression support patch, I thought
*that* was more than anyone else had done at the time.  And since I
got no response, not even a "I have something else in my tree I've
been working on" I continued to think so for some weeks until you
posted your series.

-- 
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson

Re: DTS language enhancements

[Scott Wood]

On Fri, Oct 03, 2008 at 02:37:10PM +1000, David Gibson wrote:
> I'm less sure what other operators we'll need here - probably need to
> build these based on actual usage examples.  Likely candidates,
> however are:
> 	- set property
> e.g. /setprop/({ }, "reg", < 17 >) == { reg = < 17 >; }
> 	- remove property
> e.g. /delprop({ reg = <17>; }, "reg") == { }
> 	- add subnode
> e.g. /addnode/({ }, "subnode@17", {reg = <17>;}) == 
> 	{ subnode@17 { reg = <17>; }; }
> 	- merge
> e.g. /merge/({foo = "abc";}, {bar = <17>;}) == {foo = "abc"; bar=<17>;}
> (this would recurse down subnodes with identical names)

Instead of /addnode/, how about an alternate version of (or option to)
/merge/ that merges the second tree with the contents of the first,
rather than treating the trees as sharing a root?  This could also
supersede /setprop/, if conflicts are defined to be resolved in favor of
the second tree.

> It's possible to do this just with the /setprop/, /addnode/ operators
> described above, but that's awkward and verbose, so allowing
> expressions in the same place the property/node names go now seems
> better.  jdl's patch series allows this, but I'm not sure what makes
> the grammatical distinction between the parser expecting a bare node
> name and an expression, which worries me.

I think it's the leading backslash before identifiers that distinguishes
it.

> What I would suggest here is that expressions for node/property names
> must be parenthesized.  ( and ) aren't used in node/property names
> either in theory or practice AFAIK and this is consistent with integer
> expressions having to be parenthesized within cell lists to avoid
> ambiguity.

I'd rather have an identifier prefix than to require parentheses in
otherwise unambiguous contexts (which would basically amount to needing
both a prefix and a suffix).  This applies to cell context as well.

> Expressions in labels
> ---------------------
> 
> Jon's patch also allows expressions instead of literals in labels.
> I'm a lot more dubious about this feature: it's very un-C-like, and
> removes the current nice lexical distinctness of labels.

Why do we need it to be distinguished in the lexer?  We have a parser,
let's use it. :-)

> Lexical issues of property / node names
> =======================================
> 
> Property and node names are lexically troublesome because they can
> contain a bunch of characters that would usually have special
> meanings.  My intention for dealing with this is that property/node
> names will only be lexed in a small number of contexts.  Usually they
> will not be recognized, and we can lex C-like identifiers without
> trouble.

We could simplify the lexing, and eliminate the lexical restricitons on
when we can expect a property or node name, by letting the parser glue
together property/node names when in the appropriate context.  Doing
otherwise seems like a layering violation.

Is there any plan to support expressions in bytestring context? 
Otherwise, there's no way to construct things like MAC addresses that
aren't cell-aligned.

-Scott

Re: DTS language enhancements

[David Gibson]

On Mon, Oct 06, 2008 at 12:06:01PM -0500, Scott Wood wrote:
> On Fri, Oct 03, 2008 at 02:37:10PM +1000, David Gibson wrote:
> > I'm less sure what other operators we'll need here - probably need to
> > build these based on actual usage examples.  Likely candidates,
> > however are:
> > 	- set property
> > e.g. /setprop/({ }, "reg", < 17 >) == { reg = < 17 >; }
> > 	- remove property
> > e.g. /delprop({ reg = <17>; }, "reg") == { }
> > 	- add subnode
> > e.g. /addnode/({ }, "subnode@17", {reg = <17>;}) == 
> > 	{ subnode@17 { reg = <17>; }; }
> > 	- merge
> > e.g. /merge/({foo = "abc";}, {bar = <17>;}) == {foo = "abc"; bar=<17>;}
> > (this would recurse down subnodes with identical names)
> 
> Instead of /addnode/, how about an alternate version of (or option to)
> /merge/ that merges the second tree with the contents of the first,

Um.. I don't entirely see how this variant of /merge/ would differ
from /addnode/.

> rather than treating the trees as sharing a root?  This could also
> supersede /setprop/, if conflicts are defined to be resolved in favor of
> the second tree.

True, and I was assuming /merge/ would resolve conflicts in favour of
one of the trees.  As I said, these suggestions are only an outline -
I'm not entirely sure what we need by way of node expressions.

> > It's possible to do this just with the /setprop/, /addnode/ operators
> > described above, but that's awkward and verbose, so allowing
> > expressions in the same place the property/node names go now seems
> > better.  jdl's patch series allows this, but I'm not sure what makes
> > the grammatical distinction between the parser expecting a bare node
> > name and an expression, which worries me.
> 
> I think it's the leading backslash before identifiers that distinguishes
> it.

Uh.. this doesn't make sense, an expression doesn't have to contain
identifiers (constant expressions).  Even if it does contain
identifiers, they could be arbitrarily far into it, which means we'd
nede to turn glr-parser mode back on, as well as being a bad idea from
a readability point of view.

Actually I suspect it's the presence of quotes which does the trick,
at least in the examples Jon's given.

> > What I would suggest here is that expressions for node/property names
> > must be parenthesized.  ( and ) aren't used in node/property names
> > either in theory or practice AFAIK and this is consistent with integer
> > expressions having to be parenthesized within cell lists to avoid
> > ambiguity.
> 
> I'd rather have an identifier prefix than to require parentheses in
> otherwise unambiguous contexts (which would basically amount to needing
> both a prefix and a suffix).  This applies to cell context as well.

As above, this doesn't work.  It doubly doesn't work for cell context,
because we need to disambiguate <3 (-2)> (2 cells) from <(3-2)> (1 cell).

> > Expressions in labels
> > ---------------------
> > 
> > Jon's patch also allows expressions instead of literals in labels.
> > I'm a lot more dubious about this feature: it's very un-C-like, and
> > removes the current nice lexical distinctness of labels.
> 
> Why do we need it to be distinguished in the lexer?  We have a parser,
> let's use it. :-)

Because they can appear in the same lexical context as propnodenames
which are lexically troublesome already.  Plus I really, really,
seriously want to use nice C like identifiers not these backslash
monstrosities.  That introduces more potential ambiguities if labels
don't have the : as part of the token.

> > Lexical issues of property / node names
> > =======================================
> > 
> > Property and node names are lexically troublesome because they can
> > contain a bunch of characters that would usually have special
> > meanings.  My intention for dealing with this is that property/node
> > names will only be lexed in a small number of contexts.  Usually they
> > will not be recognized, and we can lex C-like identifiers without
> > trouble.
> 
> We could simplify the lexing, and eliminate the lexical restricitons on
> when we can expect a property or node name, by letting the parser glue
> together property/node names when in the appropriate context.  Doing
> otherwise seems like a layering violation.

I've considered this in the past, and experimented.  We really don't
want to go there; attempting to achieve parsing sanity when the
propnodename isn't recognized as a token is just horrible.

I see your point w.r.t. a layering violation, but it's a pretty
standard one in lexer/parser combinations.  It's why it's usually best
to make all the tokens lexically distinct in all contexts.  In our
case I think the readability of bare propnodenames balances the
lex/parsing inconvenience - and in any case we're stuck with it now.

> Is there any plan to support expressions in bytestring context? 
> Otherwise, there's no way to construct things like MAC addresses that
> aren't cell-aligned.

Ah, yes, this is something I had a plan for, way back.  I wanted to
keep the [...] construct as a being a very compact representation of
bytestrings - bytestring literals effectively.  That means bare hex
and no expressions (because expressions and bare hex are a highly
confusing mix as we've discovered).  But, I was intending to extend
the celllist construct to allow the "cells" to be of different sizes -
this would be useful for dealing with 64-bit quantities too.  Not sure
how to do the syntax, though  Possibly:
	<.1 0xab 0xcd > 		(1 byte entries)
	<.8 0xdeadbeef00000000 >	(8 byte entries)
Defaulting to .4, of course.  I'm not over fond of that though.
Better suggestions welcome.

-- 
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson

Re: DTS language enhancements

[David Gibson]

On Tue, Oct 07, 2008 at 12:41:03PM +1100, David Gibson wrote:
> On Mon, Oct 06, 2008 at 12:06:01PM -0500, Scott Wood wrote:
> > On Fri, Oct 03, 2008 at 02:37:10PM +1000, David Gibson wrote:
[snip]
> > We could simplify the lexing, and eliminate the lexical restricitons on
> > when we can expect a property or node name, by letting the parser glue
> > together property/node names when in the appropriate context.  Doing
> > otherwise seems like a layering violation.
> 
> I've considered this in the past, and experimented.  We really don't
> want to go there; attempting to achieve parsing sanity when the
> propnodename isn't recognized as a token is just horrible.

Oh, just remembered the more precise reason why this is the case.  By
the time we get to the parser, whitespace and comments have
disappeared entirely, which means they'd be ignored *in the midst* of
node property names under this proposal, which is just not nice.

It's technically possible to recognize whitespace at the parser level
instead, but it makes the grammar hideous.

-- 
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson

Re: DTS language enhancements

[Scott Wood]

David Gibson wrote:
> Oh, just remembered the more precise reason why this is the case.  By
> the time we get to the parser, whitespace and comments have
> disappeared entirely, which means they'd be ignored *in the midst* of
> node property names under this proposal, which is just not nice.
> 
> It's technically possible to recognize whitespace at the parser level
> instead, but it makes the grammar hideous.

Ah, right, that'd be unpleasant.  I think it could be done with a little 
less ugliness than fully recognizing whitespace in the parser (have the 
lexer set a flag on relevant tokens if the previous character was 
whitespace or a comment, and have the parser rule that glues the 
propnodename together check the flag and raise an error if set), but 
that's still uglier than the propnodename-after-opening-brace approach.

-Scott

Re: DTS language enhancements

[Scott Wood]

On Tue, Oct 07, 2008 at 12:41:03PM +1100, David Gibson wrote:
> > Instead of /addnode/, how about an alternate version of (or option to)
> > /merge/ that merges the second tree with the contents of the first,
> 
> Um.. I don't entirely see how this variant of /merge/ would differ
> from /addnode/.

/addnode/ can only add one node at a time, and has an different syntax
than normal for representing the node to be added (name is split from
body).  /mergeunder/ could add any number of nodes and/or properties,
and would use more normal syntax.

> > rather than treating the trees as sharing a root?  This could also
> > supersede /setprop/, if conflicts are defined to be resolved in favor of
> > the second tree.
> 
> True, and I was assuming /merge/ would resolve conflicts in favour of
> one of the trees.  As I said, these suggestions are only an outline -
> I'm not entirely sure what we need by way of node expressions.

Understood -- I was just trying to help refine the suggestion.  I think
the ideal way forward involves elements of both your proposal and Jon's.

> > > It's possible to do this just with the /setprop/, /addnode/ operators
> > > described above, but that's awkward and verbose, so allowing
> > > expressions in the same place the property/node names go now seems
> > > better.  jdl's patch series allows this, but I'm not sure what makes
> > > the grammatical distinction between the parser expecting a bare node
> > > name and an expression, which worries me.
> > 
> > I think it's the leading backslash before identifiers that distinguishes
> > it.
> 
> Uh.. this doesn't make sense, an expression doesn't have to contain
> identifiers (constant expressions).

It's not nonsensical, just incomplete -- I was assuming that it was
obvious that quote-marks differentiate string constants.  Integer
constants would need parentheses in that context, though.

> Actually I suspect it's the presence of quotes which does the trick,
> at least in the examples Jon's given.
> 
> > > What I would suggest here is that expressions for node/property names
> > > must be parenthesized.  ( and ) aren't used in node/property names
> > > either in theory or practice AFAIK and this is consistent with integer
> > > expressions having to be parenthesized within cell lists to avoid
> > > ambiguity.
> > 
> > I'd rather have an identifier prefix than to require parentheses in
> > otherwise unambiguous contexts (which would basically amount to needing
> > both a prefix and a suffix).  This applies to cell context as well.
> 
> As above, this doesn't work.  It doubly doesn't work for cell context,
> because we need to disambiguate <3 (-2)> (2 cells) from <(3-2)> (1 cell).

It was a bare identifier (or function call) that I had in mind as a
non-ambiguous context, though I can see how allowing that might
complicate the implementation a little -- and allowing things like 3*2
to be unparenthesized would be even more complex.

We probably should have added comma-delimiting when we switched to
decimal-by-default.

> Ah, yes, this is something I had a plan for, way back.  I wanted to
> keep the [...] construct as a being a very compact representation of
> bytestrings - bytestring literals effectively.  That means bare hex
> and no expressions (because expressions and bare hex are a highly
> confusing mix as we've discovered).  But, I was intending to extend
> the celllist construct to allow the "cells" to be of different sizes -
> this would be useful for dealing with 64-bit quantities too.  Not sure
> how to do the syntax, though  Possibly:
> 	<.1 0xab 0xcd > 		(1 byte entries)
> 	<.8 0xdeadbeef00000000 >	(8 byte entries)
> Defaulting to .4, of course.  I'm not over fond of that though.
> Better suggestions welcome.

Seems reasonable.

-Scott

Re: DTS language enhancements

[David Gibson]

On Tue, Oct 07, 2008 at 11:50:38AM -0500, Scott Wood wrote:
> On Tue, Oct 07, 2008 at 12:41:03PM +1100, David Gibson wrote:
> > > Instead of /addnode/, how about an alternate version of (or option to)
> > > /merge/ that merges the second tree with the contents of the first,
> > 
> > Um.. I don't entirely see how this variant of /merge/ would differ
> > from /addnode/.
> 
> /addnode/ can only add one node at a time, and has an different syntax
> than normal for representing the node to be added (name is split from
> body).  /mergeunder/ could add any number of nodes and/or properties,
> and would use more normal syntax.

Um.. in that case I don't see how this differs from the original
/merge/.

> > > rather than treating the trees as sharing a root?  This could also
> > > supersede /setprop/, if conflicts are defined to be resolved in favor of
> > > the second tree.
> > 
> > True, and I was assuming /merge/ would resolve conflicts in favour of
> > one of the trees.  As I said, these suggestions are only an outline -
> > I'm not entirely sure what we need by way of node expressions.
> 
> Understood -- I was just trying to help refine the suggestion.  I think
> the ideal way forward involves elements of both your proposal and Jon's.
> 
> > > > It's possible to do this just with the /setprop/, /addnode/ operators
> > > > described above, but that's awkward and verbose, so allowing
> > > > expressions in the same place the property/node names go now seems
> > > > better.  jdl's patch series allows this, but I'm not sure what makes
> > > > the grammatical distinction between the parser expecting a bare node
> > > > name and an expression, which worries me.
> > > 
> > > I think it's the leading backslash before identifiers that distinguishes
> > > it.
> > 
> > Uh.. this doesn't make sense, an expression doesn't have to contain
> > identifiers (constant expressions).
> 
> It's not nonsensical, just incomplete -- I was assuming that it was
> obvious that quote-marks differentiate string constants.  Integer
> constants would need parentheses in that context, though.

Yeah, and having the grammatically distinguishing difference spread
across the pieces, or (potentially) buried deep inside the expression
is not sensible, either from a parser design or readability point of
view.  Much better to wrap the whole thing in parens.

> > Actually I suspect it's the presence of quotes which does the trick,
> > at least in the examples Jon's given.
> > 
> > > > What I would suggest here is that expressions for node/property names
> > > > must be parenthesized.  ( and ) aren't used in node/property names
> > > > either in theory or practice AFAIK and this is consistent with integer
> > > > expressions having to be parenthesized within cell lists to avoid
> > > > ambiguity.
> > > 
> > > I'd rather have an identifier prefix than to require parentheses in
> > > otherwise unambiguous contexts (which would basically amount to needing
> > > both a prefix and a suffix).  This applies to cell context as well.
> > 
> > As above, this doesn't work.  It doubly doesn't work for cell context,
> > because we need to disambiguate <3 (-2)> (2 cells) from <(3-2)> (1 cell).
> 
> It was a bare identifier (or function call) that I had in mind as a
> non-ambiguous context, though I can see how allowing that might
> complicate the implementation a little -- and allowing things like 3*2
> to be unparenthesized would be even more complex.

Again, I think we should make things non-ambiguous by design, rather
than by allowing any non ambiguous form, even if it requires parsing
deep into the thing to resolve the ambiguity.  Doing otherwise both
complexifies the implementation and threatens readability

> We probably should have added comma-delimiting when we switched to
> decimal-by-default.

Maybe, but its done now.  I really don't think requiring parens aroud
expressions is that big an imposition.

> > Ah, yes, this is something I had a plan for, way back.  I wanted to
> > keep the [...] construct as a being a very compact representation of
> > bytestrings - bytestring literals effectively.  That means bare hex
> > and no expressions (because expressions and bare hex are a highly
> > confusing mix as we've discovered).  But, I was intending to extend
> > the celllist construct to allow the "cells" to be of different sizes -
> > this would be useful for dealing with 64-bit quantities too.  Not sure
> > how to do the syntax, though  Possibly:
> > 	<.1 0xab 0xcd > 		(1 byte entries)
> > 	<.8 0xdeadbeef00000000 >	(8 byte entries)
> > Defaulting to .4, of course.  I'm not over fond of that though.
> > Better suggestions welcome.
> 
> Seems reasonable.
> 
> -Scott
> 

-- 
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson

Re: DTS language enhancements

[Scott Wood]

On Thu, Oct 09, 2008 at 01:27:22PM +1100, David Gibson wrote:
> On Tue, Oct 07, 2008 at 11:50:38AM -0500, Scott Wood wrote:
> > On Tue, Oct 07, 2008 at 12:41:03PM +1100, David Gibson wrote:
> > > > Instead of /addnode/, how about an alternate version of (or option to)
> > > > /merge/ that merges the second tree with the contents of the first,
> > > 
> > > Um.. I don't entirely see how this variant of /merge/ would differ
> > > from /addnode/.
> > 
> > /addnode/ can only add one node at a time, and has an different syntax
> > than normal for representing the node to be added (name is split from
> > body).  /mergeunder/ could add any number of nodes and/or properties,
> > and would use more normal syntax.
> 
> Um.. in that case I don't see how this differs from the original
> /merge/.

I guess the issue is that /merge/ is incompletely specified.  The example
showed it taking anonymous trees as parameters; are the parameters always
wrapped as such, or can named nodes be passed directly?  In the latter
case, /mergeunder/ was intended as a way of dereferencing the former
argument so as to place things inside it rather than create a new
anonymous node containing both, without needing to know the
name/unit-address of the named node.

An alternative would be a function to extract the name of non-anonymous
nodes, which could be used to build a matching subnode in the second
/merge/ argument.

> > We probably should have added comma-delimiting when we switched to
> > decimal-by-default.
> 
> Maybe, but its done now.  I really don't think requiring parens aroud
> expressions is that big an imposition.

Agreed; I may find it as ugly as you find identifier prefixes, but we
don't have much of a choice if we want to retain compatibility.

-Scott