[PATCH] utils: Remove double compile from better_compile

[PATCH] utils: Remove double compile from better_compile

[Richard Purdie]

Poking around the ast to correct linenumbers works well for runtime failures
but not for parsing ones. We can use blank linefeeds to correct the line
numbers instead, with the advantage that we don't need to double compile.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>

diff --git a/bitbake/lib/bb/utils.py b/bitbake/lib/bb/utils.py
index cd5fced..9a3efb2 100644
--- a/bitbake/lib/bb/utils.py
+++ b/bitbake/lib/bb/utils.py
@@ -292,7 +292,7 @@ def _print_trace(body, line):
             error.append('     %.4d:%s' % (i, body[i-1].rstrip()))
     return error
 
-def better_compile(text, file, realfile, mode = "exec", lineno = None):
+def better_compile(text, file, realfile, mode = "exec", lineno = 0):
     """
     A better compile method. This method
     will print the offending lines.
@@ -301,10 +301,9 @@ def better_compile(text, file, realfile, mode = "exec", lineno = None):
         cache = bb.methodpool.compile_cache(text)
         if cache:
             return cache
-        code = compile(text, realfile, mode, ast.PyCF_ONLY_AST)
-        if lineno is not None:
-            ast.increment_lineno(code, lineno)
-        code = compile(code, realfile, mode)
+        # We can't add to the linenumbers for compile, we can pad to the correct number of blank lines though
+        text2 = "\n" * int(lineno) + text
+        code = compile(text2, realfile, mode)
         bb.methodpool.compile_cache_add(text, code)
         return code
     except Exception as e:

Re: [PATCH] utils: Remove double compile from better_compile

[Christopher Larson]

[-- Attachment #1: Type: text/plain, Size: 1997 bytes --]

On Mon, Jan 4, 2016 at 10:34 AM, Richard Purdie <
richard.purdie@linuxfoundation.org> wrote:

> Poking around the ast to correct linenumbers works well for runtime
> failures
> but not for parsing ones. We can use blank linefeeds to correct the line
> numbers instead, with the advantage that we don't need to double compile.
>
> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
>
> diff --git a/bitbake/lib/bb/utils.py b/bitbake/lib/bb/utils.py
> index cd5fced..9a3efb2 100644
> --- a/bitbake/lib/bb/utils.py
> +++ b/bitbake/lib/bb/utils.py
> @@ -292,7 +292,7 @@ def _print_trace(body, line):
>              error.append('     %.4d:%s' % (i, body[i-1].rstrip()))
>      return error
>
> -def better_compile(text, file, realfile, mode = "exec", lineno = None):
> +def better_compile(text, file, realfile, mode = "exec", lineno = 0):
>      """
>      A better compile method. This method
>      will print the offending lines.
> @@ -301,10 +301,9 @@ def better_compile(text, file, realfile, mode =
> "exec", lineno = None):
>          cache = bb.methodpool.compile_cache(text)
>          if cache:
>              return cache
> -        code = compile(text, realfile, mode, ast.PyCF_ONLY_AST)
> -        if lineno is not None:
> -            ast.increment_lineno(code, lineno)
> -        code = compile(code, realfile, mode)
> +        # We can't add to the linenumbers for compile, we can pad to the
> correct number of blank lines though
> +        text2 = "\n" * int(lineno) + text
> +        code = compile(text2, realfile, mode)
>

A SyntaxError can have its line numbers adjusted fairly easily, if that's
your concern. Afaik that's the usual case which ast.increment_lineno isn't
sufficient to handle. See
https://gist.github.com/kergoth/743677#file-compile-py-L67-L86.
-- 
Christopher Larson
clarson at kergoth dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics

[-- Attachment #2: Type: text/html, Size: 2985 bytes --]

Re: [PATCH] utils: Remove double compile from better_compile

[Richard Purdie]

On Mon, 2016-01-04 at 11:06 -0700, Christopher Larson wrote:
> 
> On Mon, Jan 4, 2016 at 10:34 AM, Richard Purdie <
> richard.purdie@linuxfoundation.org> wrote:
> > Poking around the ast to correct linenumbers works well for runtime
> > failures
> > but not for parsing ones. We can use blank linefeeds to correct the
> > line
> > numbers instead, with the advantage that we don't need to double
> > compile.
> > 
> > Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
> > 
> > diff --git a/bitbake/lib/bb/utils.py b/bitbake/lib/bb/utils.py
> > index cd5fced..9a3efb2 100644
> > --- a/bitbake/lib/bb/utils.py
> > +++ b/bitbake/lib/bb/utils.py
> > @@ -292,7 +292,7 @@ def _print_trace(body, line):
> >              error.append('     %.4d:%s' % (i, body[i-1].rstrip()))
> >      return error
> > 
> > -def better_compile(text, file, realfile, mode = "exec", lineno =
> > None):
> > +def better_compile(text, file, realfile, mode = "exec", lineno =
> > 0):
> >      """
> >      A better compile method. This method
> >      will print the offending lines.
> > @@ -301,10 +301,9 @@ def better_compile(text, file, realfile, mode
> > = "exec", lineno = None):
> >          cache = bb.methodpool.compile_cache(text)
> >          if cache:
> >              return cache
> > -        code = compile(text, realfile, mode, ast.PyCF_ONLY_AST)
> > -        if lineno is not None:
> > -            ast.increment_lineno(code, lineno)
> > -        code = compile(code, realfile, mode)
> > +        # We can't add to the linenumbers for compile, we can pad
> > to the correct number of blank lines though
> > +        text2 = "\n" * int(lineno) + text
> > +        code = compile(text2, realfile, mode)
> > 
> A SyntaxError can have its line numbers adjusted fairly easily, if
> that's your concern. Afaik that's the usual case which
> ast.increment_lineno isn't  sufficient to handle. See 
> https://gist.github.com/kergoth/743677#file-compile-py-L67-L86.

I did start to wonder about catching specific exceptions and then
adjusting the line numbers. It seemed unlikely that we'd catch all the
right cases and adjust them correctly though and when there is a much
easier workaround with a few blank lines that likely performs better
than the existing double compile, I concluded that whilst not that
architecturally pleasing, it is simple and effective...

Cheers,

Richard