[PATCH v3 0/1] bitbake: event: Inject empty lines to make code match lineno in filename

[PATCH v3 0/1] bitbake: event: Inject empty lines to make code match lineno in filename

[liezhi.yang]

From: Robert Yang <liezhi.yang@windriver.com>

* V3: Inject empty lines to make code match lineno in filename as suggested by RP.

// Robert

The following changes since commit 6ce42d73a31bf9b2fc362fa4acad11de97ac96ec:

  scripts: Drop shell sstate-cache-management (2023-12-28 10:57:49 +0000)

are available in the Git repository at:

  https://github.com/robertlinux/yocto rbt/event
  https://github.com/robertlinux/yocto/tree/rbt/event

Robert Yang (1):
  bitbake: event: Inject empty lines to make code match lineno in
    filename

 bitbake/lib/bb/event.py | 2 ++
 1 file changed, 2 insertions(+)

-- 
2.42.0

[PATCH v3 1/1] bitbake: event: Inject empty lines to make code match lineno in filename

[liezhi.yang]

From: Robert Yang <liezhi.yang@windriver.com>

So that we can get the correct error messages.

* In python 3.10.9, the error message was:
  ERROR: Unable to register event handler 'defaultbase_eventhandler':
    File "/path/to/poky/meta/classes-global/base.bbclass", line 4
      # SPDX-License-Identifier: MIT
             ^^^^^
  SyntaxError: invalid syntax

  This is hard to debug since the error line number 4 is incorrect, but nothing
  is wrong with the code in line 4.

* Now the error message and lineno is correct:
  ERROR: Unable to register event handler 'defaultbase_eventhandler':
    File "/path/to/poky/meta/classes-global/base.bbclass", line 256
      an error line
         ^^^^^
  SyntaxError: invalid syntax

And no impact on parsing time:
* Before:
$ rm -fr cache tmp; time bitbake -p
real    0m27.254s

* Now:
$ rm -fr cache tmp; time bitbake -p
real    0m27.200s

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
---
 bitbake/lib/bb/event.py | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/bitbake/lib/bb/event.py b/bitbake/lib/bb/event.py
index f8acacd80d1..7722258b7e8 100644
--- a/bitbake/lib/bb/event.py
+++ b/bitbake/lib/bb/event.py
@@ -257,6 +257,8 @@ def register(name, handler, mask=None, filename=None, lineno=None, data=None):
         # handle string containing python code
         if isinstance(handler, str):
             tmp = "def %s(e, d):\n%s" % (name, handler)
+            # Inject empty lines to make code match lineno in filename
+            tmp = "\n" * (lineno-1) + tmp
             try:
                 code = bb.methodpool.compile_cache(tmp)
                 if not code:
-- 
2.42.0

Re: [bitbake-devel] [PATCH v3 1/1] bitbake: event: Inject empty lines to make code match lineno in filename

[Richard Purdie]

On Thu, 2023-12-28 at 22:06 -0800, Robert Yang via
lists.openembedded.org wrote:
> From: Robert Yang <liezhi.yang@windriver.com>
> 
> So that we can get the correct error messages.
> 
> * In python 3.10.9, the error message was:
>   ERROR: Unable to register event handler 'defaultbase_eventhandler':
>     File "/path/to/poky/meta/classes-global/base.bbclass", line 4
>       # SPDX-License-Identifier: MIT
>              ^^^^^
>   SyntaxError: invalid syntax
> 
>   This is hard to debug since the error line number 4 is incorrect, but nothing
>   is wrong with the code in line 4.
> 
> * Now the error message and lineno is correct:
>   ERROR: Unable to register event handler 'defaultbase_eventhandler':
>     File "/path/to/poky/meta/classes-global/base.bbclass", line 256
>       an error line
>          ^^^^^
>   SyntaxError: invalid syntax
> 
> And no impact on parsing time:
> * Before:
> $ rm -fr cache tmp; time bitbake -p
> real    0m27.254s
> 
> * Now:
> $ rm -fr cache tmp; time bitbake -p
> real    0m27.200s
> 
> Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
> ---
>  bitbake/lib/bb/event.py | 2 ++
>  1 file changed, 2 insertions(+)
> 
> diff --git a/bitbake/lib/bb/event.py b/bitbake/lib/bb/event.py
> index f8acacd80d1..7722258b7e8 100644
> --- a/bitbake/lib/bb/event.py
> +++ b/bitbake/lib/bb/event.py
> @@ -257,6 +257,8 @@ def register(name, handler, mask=None, filename=None, lineno=None, data=None):
>          # handle string containing python code
>          if isinstance(handler, str):
>              tmp = "def %s(e, d):\n%s" % (name, handler)
> +            # Inject empty lines to make code match lineno in filename
> +            tmp = "\n" * (lineno-1) + tmp
>              try:
>                  code = bb.methodpool.compile_cache(tmp)
>                  if not code:

Looks good but don't you need remove the ast line number adjustment a
few lines later as well?

Should we put some tests into bitbake-selftest for this?

Cheers,

Richard

Re: [bitbake-devel] [PATCH v3 1/1] bitbake: event: Inject empty lines to make code match lineno in filename

[Robert Yang]

On 12/29/23 7:16 PM, Richard Purdie wrote:
> On Thu, 2023-12-28 at 22:06 -0800, Robert Yang via
> lists.openembedded.org wrote:
>> From: Robert Yang <liezhi.yang@windriver.com>
>>
>> So that we can get the correct error messages.
>>
>> * In python 3.10.9, the error message was:
>>    ERROR: Unable to register event handler 'defaultbase_eventhandler':
>>      File "/path/to/poky/meta/classes-global/base.bbclass", line 4
>>        # SPDX-License-Identifier: MIT
>>               ^^^^^
>>    SyntaxError: invalid syntax
>>
>>    This is hard to debug since the error line number 4 is incorrect, but nothing
>>    is wrong with the code in line 4.
>>
>> * Now the error message and lineno is correct:
>>    ERROR: Unable to register event handler 'defaultbase_eventhandler':
>>      File "/path/to/poky/meta/classes-global/base.bbclass", line 256
>>        an error line
>>           ^^^^^
>>    SyntaxError: invalid syntax
>>
>> And no impact on parsing time:
>> * Before:
>> $ rm -fr cache tmp; time bitbake -p
>> real    0m27.254s
>>
>> * Now:
>> $ rm -fr cache tmp; time bitbake -p
>> real    0m27.200s
>>
>> Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
>> ---
>>   bitbake/lib/bb/event.py | 2 ++
>>   1 file changed, 2 insertions(+)
>>
>> diff --git a/bitbake/lib/bb/event.py b/bitbake/lib/bb/event.py
>> index f8acacd80d1..7722258b7e8 100644
>> --- a/bitbake/lib/bb/event.py
>> +++ b/bitbake/lib/bb/event.py
>> @@ -257,6 +257,8 @@ def register(name, handler, mask=None, filename=None, lineno=None, data=None):
>>           # handle string containing python code
>>           if isinstance(handler, str):
>>               tmp = "def %s(e, d):\n%s" % (name, handler)
>> +            # Inject empty lines to make code match lineno in filename
>> +            tmp = "\n" * (lineno-1) + tmp
>>               try:
>>                   code = bb.methodpool.compile_cache(tmp)
>>                   if not code:
> 
> Looks good but don't you need remove the ast line number adjustment a
> few lines later as well?

Yes, you're right, the code has incremented the lineno by '\n', so the
following line isn't needed:

ast.increment_lineno(code, lineno-1)

Now the patch is: (Check lineno is not None to fix bitbake-selftest error)
--- a/bitbake/lib/bb/event.py
+++ b/bitbake/lib/bb/event.py
@@ -257,14 +257,15 @@ def register(name, handler, mask=None, filename=None, 
lineno=None, data=None):
          # handle string containing python code
          if isinstance(handler, str):
              tmp = "def %s(e, d):\n%s" % (name, handler)
+            # Inject empty lines to make code match lineno in filename
+            if lineno is not None:
+                tmp = "\n" * (lineno-1) + tmp
              try:
                  code = bb.methodpool.compile_cache(tmp)
                  if not code:
                      if filename is None:
                          filename = "%s(e, d)" % name
                      code = compile(tmp, filename, "exec", ast.PyCF_ONLY_AST)
-                    if lineno is not None:
-                        ast.increment_lineno(code, lineno-1)
                      code = compile(code, filename, "exec")
                      bb.methodpool.compile_cache_add(tmp, code)
              except SyntaxError:

I will send a V4 add with a testcase for it.

// Robert

> 
> Should we put some tests into bitbake-selftest for this?
> 
> Cheers,
> 
> Richard

Re: [bitbake-devel] [PATCH v3 1/1] bitbake: event: Inject empty lines to make code match lineno in filename

[Alexandre Belloni]

This causes bitbake-selftest failures:

======================================================================
ERROR: test_register_from_string (bb.tests.event.EventHandlingTest)
Test register method receiving code in string
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/pokybuild/yocto-worker/oe-selftest-debian/build/bitbake/lib/bb/tests/event.py", line 121, in test_register_from_string
    result = bb.event.register("string_handler", "    return True")
  File "/home/pokybuild/yocto-worker/oe-selftest-debian/build/bitbake/lib/bb/event.py", line 261, in register
    tmp = "\n" * (lineno-1) + tmp
TypeError: unsupported operand type(s) for -: 'NoneType' and 'int'

https://autobuilder.yoctoproject.org/typhoon/#/builders/80/builds/6202/steps/11/logs/stdio

On 28/12/2023 22:06:42-0800, Robert Yang via lists.openembedded.org wrote:
> From: Robert Yang <liezhi.yang@windriver.com>
> 
> So that we can get the correct error messages.
> 
> * In python 3.10.9, the error message was:
>   ERROR: Unable to register event handler 'defaultbase_eventhandler':
>     File "/path/to/poky/meta/classes-global/base.bbclass", line 4
>       # SPDX-License-Identifier: MIT
>              ^^^^^
>   SyntaxError: invalid syntax
> 
>   This is hard to debug since the error line number 4 is incorrect, but nothing
>   is wrong with the code in line 4.
> 
> * Now the error message and lineno is correct:
>   ERROR: Unable to register event handler 'defaultbase_eventhandler':
>     File "/path/to/poky/meta/classes-global/base.bbclass", line 256
>       an error line
>          ^^^^^
>   SyntaxError: invalid syntax
> 
> And no impact on parsing time:
> * Before:
> $ rm -fr cache tmp; time bitbake -p
> real    0m27.254s
> 
> * Now:
> $ rm -fr cache tmp; time bitbake -p
> real    0m27.200s
> 
> Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
> ---
>  bitbake/lib/bb/event.py | 2 ++
>  1 file changed, 2 insertions(+)
> 
> diff --git a/bitbake/lib/bb/event.py b/bitbake/lib/bb/event.py
> index f8acacd80d1..7722258b7e8 100644
> --- a/bitbake/lib/bb/event.py
> +++ b/bitbake/lib/bb/event.py
> @@ -257,6 +257,8 @@ def register(name, handler, mask=None, filename=None, lineno=None, data=None):
>          # handle string containing python code
>          if isinstance(handler, str):
>              tmp = "def %s(e, d):\n%s" % (name, handler)
> +            # Inject empty lines to make code match lineno in filename
> +            tmp = "\n" * (lineno-1) + tmp
>              try:
>                  code = bb.methodpool.compile_cache(tmp)
>                  if not code:
> -- 
> 2.42.0
> 

> 
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#15717): https://lists.openembedded.org/g/bitbake-devel/message/15717
> Mute This Topic: https://lists.openembedded.org/mt/103413642/3617179
> Group Owner: bitbake-devel+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/bitbake-devel/unsub [alexandre.belloni@bootlin.com]
> -=-=-=-=-=-=-=-=-=-=-=-
> 


-- 
Alexandre Belloni, co-owner and COO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com