Re: [PATCH v2] lexer: support // comments

Kris Van Hees
2025-07-23T12:57:37Z

On Wed, Jul 23, 2025 at 10:45:54AM +0100, Alan Maguire wrote:
> On 23/07/2025 06:11, Kris Van Hees wrote:
> > Suggested-by: Ruud van der Pas <ruud.vanderpas@oracle.com>
> > Suggested-by: Alan Maguire <alan.maguire@oracle.com>
> > Signed-off-by: Kris Van Hees <kris.van.hees@oracle.com>
>
> Reviewed-by: Alan Maguire <alan.maguire@oracle.com>
> Tested-by: Alan Maguire <alan.maguire@oracle.com>
>
> a few small things below regarding assertions as these are all positive
> tests now I think (allowing /* after // etc), so the assertions need
> updating, but aside from that looks great!

Oops, yes, was a bit hasty changing them.

(The purist in me still feels that the ODX people did the right thing in being
more pedantic on what is reported as an error in //-comments, but I can see
how that can be annoying to people - and we want to be be nice :))

> > ---
> > libdtrace/dt_lex.l | 11 ++++++++++-
> > test/unittest/lexer/tst.boc-in-line-comment.d | 16 ++++++++++++++++
> > test/unittest/lexer/tst.boc-in-line-comment.r | 5 +++++
> > test/unittest/lexer/tst.eoc-in-line-comment.d | 16 ++++++++++++++++
> > test/unittest/lexer/tst.eoc-in-line-comment.r | 5 +++++
> > test/unittest/lexer/tst.eof-in-line-comment.d | 16 ++++++++++++++++
> > test/unittest/lexer/tst.eof-in-line-comment.r | 5 +++++
> > test/unittest/lexer/tst.lc-in-line-comment.d | 16 ++++++++++++++++
> > test/unittest/lexer/tst.lc-in-line-comment.r | 5 +++++
> > test/unittest/lexer/tst.line-comment.d | 16 ++++++++++++++++
> > test/unittest/lexer/tst.line-comment.r | 5 +++++
> > 11 files changed, 115 insertions(+), 1 deletion(-)
> > create mode 100644 test/unittest/lexer/tst.boc-in-line-comment.d
> > create mode 100644 test/unittest/lexer/tst.boc-in-line-comment.r
> > create mode 100644 test/unittest/lexer/tst.eoc-in-line-comment.d
> > create mode 100644 test/unittest/lexer/tst.eoc-in-line-comment.r
> > create mode 100644 test/unittest/lexer/tst.eof-in-line-comment.d
> > create mode 100644 test/unittest/lexer/tst.eof-in-line-comment.r
> > create mode 100644 test/unittest/lexer/tst.lc-in-line-comment.d
> > create mode 100644 test/unittest/lexer/tst.lc-in-line-comment.r
> > create mode 100644 test/unittest/lexer/tst.line-comment.d
> > create mode 100644 test/unittest/lexer/tst.line-comment.r
> >
> > diff --git a/libdtrace/dt_lex.l b/libdtrace/dt_lex.l
> > index 9d502912..e8f3bc8c 100644
> > --- a/libdtrace/dt_lex.l
> > +++ b/libdtrace/dt_lex.l
> > @@ -35,6 +35,7 @@ int yydebug;
> > * S2 - D program outer scope (probe specifiers and declarations)
> > * S3 - D control line parsing (i.e. after ^# is seen but before \n)
> > * S4 - D control line scan (locate control directives only and invoke S3)
> > + * S5 - D line comments (i.e. skip everything until end of line)
> > * SIDENT - identifiers and comments only (after -> and .). (We switch to
> > * SIDENT only from state S0: changing this would require new code
> > * to track the state to switch back to.)
> > @@ -46,7 +47,7 @@ int yydebug;
> > %n 600 /* maximum states */
> > %option yylineno
> >
> > -%s S0 S1 S2 S3 S4 SIDENT
> > +%s S0 S1 S2 S3 S4 S5 SIDENT
> >
> > RGX_AGG "@"[a-zA-Z_][0-9a-zA-Z_]*
> > RGX_PSPEC [-$:a-zA-Z_.?*\\\[\]!][-$:0-9a-zA-Z_.`?*\\\[\]!]*
> > @@ -408,6 +409,11 @@ if (yypcb->pcb_token != 0) {
> > BEGIN(S1);
> > }
> >
> > +<S0,S2,SIDENT>"//" {
> > + yypcb->pcb_cstate = (YYSTATE);
> > + BEGIN(S5);
> > + }
> > +
> > <S0>^{RGX_INTERP} |
> > <S2>^{RGX_INTERP} ; /* discard any #! lines */
> >
> > @@ -548,6 +554,9 @@ if (yypcb->pcb_token != 0) {
> > <S1>.|\n ; /* discard */
> > <S1><<EOF>> yyerror("end-of-file encountered before matching */\n");
> >
> > +<S5>\n BEGIN(yypcb->pcb_cstate);
> > +<S5>. ; /* discard */
> > +
> > <S2>{RGX_PSPEC} {
> > /*
> > * S2 has an ambiguity because RGX_PSPEC includes '*'
> > diff --git a/test/unittest/lexer/tst.boc-in-line-comment.d b/test/unittest/lexer/tst.boc-in-line-comment.d
> > new file mode 100644
> > index 00000000..61f5961f
> > --- /dev/null
> > +++ b/test/unittest/lexer/tst.boc-in-line-comment.d
> > @@ -0,0 +1,16 @@
> > +/*
> > + * Oracle Linux DTrace.
> > + * Copyright (c) 2025, Oracle and/or its affiliates. All rights reserved.
> > + * Licensed under the Universal Permissive License v 1.0 as shown at
> > + * http://oss.oracle.com/licenses/upl.
> > + */
> > +
> > +/*
> > + * ASSERTION: Line comments cannot contain begin-of-comment markers.
> > + */
>
> this needs to be updated I think? i.e. Like C, line comments can contain
> begin-of-comment markers.
>
> > +
> > +// Comment /*
> > +BEGIN
> > +{
> > + exit(0);
> > +}
> > diff --git a/test/unittest/lexer/tst.boc-in-line-comment.r b/test/unittest/lexer/tst.boc-in-line-comment.r
> > new file mode 100644
> > index 00000000..d57f017a
> > --- /dev/null
> > +++ b/test/unittest/lexer/tst.boc-in-line-comment.r
> > @@ -0,0 +1,5 @@
> > + FUNCTION:NAME
> > + :BEGIN
> > +
> > +-- @@stderr --
> > +dtrace: script 'test/unittest/lexer/tst.boc-in-line-comment.d' matched 1 probe
> > diff --git a/test/unittest/lexer/tst.eoc-in-line-comment.d b/test/unittest/lexer/tst.eoc-in-line-comment.d
> > new file mode 100644
> > index 00000000..8fe2ff04
> > --- /dev/null
> > +++ b/test/unittest/lexer/tst.eoc-in-line-comment.d
> > @@ -0,0 +1,16 @@
> > +/*
> > + * Oracle Linux DTrace.
> > + * Copyright (c) 2025, Oracle and/or its affiliates. All rights reserved.
> > + * Licensed under the Universal Permissive License v 1.0 as shown at
> > + * http://oss.oracle.com/licenses/upl.
> > + */
> > +
> > +/*
> > + * ASSERTION: Line comments cannot contain end-of-comment markers.
>
> same here.
>
> > + */
> > +
> > +// Comment */
> > +BEGIN
> > +{
> > + exit(0);
> > +}
> > diff --git a/test/unittest/lexer/tst.eoc-in-line-comment.r b/test/unittest/lexer/tst.eoc-in-line-comment.r
> > new file mode 100644
> > index 00000000..884cd554
> > --- /dev/null
> > +++ b/test/unittest/lexer/tst.eoc-in-line-comment.r
> > @@ -0,0 +1,5 @@
> > + FUNCTION:NAME
> > + :BEGIN
> > +
> > +-- @@stderr --
> > +dtrace: script 'test/unittest/lexer/tst.eoc-in-line-comment.d' matched 1 probe
> > diff --git a/test/unittest/lexer/tst.eof-in-line-comment.d b/test/unittest/lexer/tst.eof-in-line-comment.d
> > new file mode 100644
> > index 00000000..a9207693
> > --- /dev/null
> > +++ b/test/unittest/lexer/tst.eof-in-line-comment.d
> > @@ -0,0 +1,16 @@
> > +/*
> > + * Oracle Linux DTrace.
> > + * Copyright (c) 2025, Oracle and/or its affiliates. All rights reserved.
> > + * Licensed under the Universal Permissive License v 1.0 as shown at
> > + * http://oss.oracle.com/licenses/upl.
> > + */
> > +
> > +/*
> > + * ASSERTION: End-of-file in a line comment is an error.
>
> again this doesn't seem like an error test?
>
> > + */
> > +
> > +BEGIN
> > +{
> > + exit(0);
> > +}
> > +// Comment
> > \ No newline at end of file
> > diff --git a/test/unittest/lexer/tst.eof-in-line-comment.r b/test/unittest/lexer/tst.eof-in-line-comment.r
> > new file mode 100644
> > index 00000000..fdf259e0
> > --- /dev/null
> > +++ b/test/unittest/lexer/tst.eof-in-line-comment.r
> > @@ -0,0 +1,5 @@
> > + FUNCTION:NAME
> > + :BEGIN
> > +
> > +-- @@stderr --
> > +dtrace: script 'test/unittest/lexer/tst.eof-in-line-comment.d' matched 1 probe
> > diff --git a/test/unittest/lexer/tst.lc-in-line-comment.d b/test/unittest/lexer/tst.lc-in-line-comment.d
> > new file mode 100644
> > index 00000000..0332e1ae
> > --- /dev/null
> > +++ b/test/unittest/lexer/tst.lc-in-line-comment.d
> > @@ -0,0 +1,16 @@
> > +/*
> > + * Oracle Linux DTrace.
> > + * Copyright (c) 2025, Oracle and/or its affiliates. All rights reserved.
> > + * Licensed under the Universal Permissive License v 1.0 as shown at
> > + * http://oss.oracle.com/licenses/upl.
> > + */
> > +
> > +/*
> > + * ASSERTION: Line comments cannot contain a line commens marker.
>
> same here
>
> > + */
> > +
> > +// Comment //
> > +BEGIN
> > +{
> > + exit(0);
> > +}
> > diff --git a/test/unittest/lexer/tst.lc-in-line-comment.r b/test/unittest/lexer/tst.lc-in-line-comment.r
> > new file mode 100644
> > index 00000000..3075b987
> > --- /dev/null
> > +++ b/test/unittest/lexer/tst.lc-in-line-comment.r
> > @@ -0,0 +1,5 @@
> > + FUNCTION:NAME
> > + :BEGIN
> > +
> > +-- @@stderr --
> > +dtrace: script 'test/unittest/lexer/tst.lc-in-line-comment.d' matched 1 probe
> > diff --git a/test/unittest/lexer/tst.line-comment.d b/test/unittest/lexer/tst.line-comment.d
> > new file mode 100644
> > index 00000000..4deb66f4
> > --- /dev/null
> > +++ b/test/unittest/lexer/tst.line-comment.d
> > @@ -0,0 +1,16 @@
> > +/*
> > + * Oracle Linux DTrace.
> > + * Copyright (c) 2025, Oracle and/or its affiliates. All rights reserved.
> > + * Licensed under the Universal Permissive License v 1.0 as shown at
> > + * http://oss.oracle.com/licenses/upl.
> > + */
> > +
> > +/*
> > + * ASSERTION: // comments are supported.
> > + */
> > +
> > +// exit(1);
> > +BEGIN // exit(1);
> > +{ // exit(1);
> > + exit(0); // exit(1);
> > +} // exit(1);
> > diff --git a/test/unittest/lexer/tst.line-comment.r b/test/unittest/lexer/tst.line-comment.r
> > new file mode 100644
> > index 00000000..d4e4c325
> > --- /dev/null
> > +++ b/test/unittest/lexer/tst.line-comment.r
> > @@ -0,0 +1,5 @@
> > + FUNCTION:NAME
> > + :BEGIN
> > +
> > +-- @@stderr --
> > +dtrace: script 'test/unittest/lexer/tst.line-comment.d' matched 1 probe