[PATCH v4 0/2] event: Inject empty lines to make code match lineno in filename

[PATCH v4 0/2] event: Inject empty lines to make code match lineno in filename

[liezhi.yang]

From: Robert Yang <liezhi.yang@windriver.com>

* V4:
  - Check lineno is not None
  - Add a testcase for it

// Robert

The following changes since commit 070aa227057e4913cf77c2465393a5a8398d5641:

  bitbake: toaster/tests: Delay driver first action on create new project page (2024-01-07 22:59:55 +0000)

are available in the Git repository at:

  https://github.com/robertlinux/yocto rbt/event
  https://github.com/robertlinux/yocto/tree/rbt/event

Robert Yang (2):
  bitbake: event: Inject empty lines to make code match lineno in
    filename
  bitbake: tests/event: Add test_lineno_in_eventhandler

 bitbake/lib/bb/event.py       |  5 +++--
 bitbake/lib/bb/tests/event.py | 24 ++++++++++++++++++++++++
 2 files changed, 27 insertions(+), 2 deletions(-)

-- 
2.35.5

[PATCH v4 1/2] bitbake: event: Inject empty lines to make code match lineno in filename

[liezhi.yang]

From: Robert Yang <liezhi.yang@windriver.com>

So that we can get the correct error messages.

* In python 3.10.9, the error message was:
  ERROR: Unable to register event handler 'defaultbase_eventhandler':
    File "/path/to/poky/meta/classes-global/base.bbclass", line 4
      # SPDX-License-Identifier: MIT
             ^^^^^
  SyntaxError: invalid syntax

  This is hard to debug since the error line number 4 is incorrect, but nothing
  is wrong with the code in line 4.

* Now the error message and lineno is correct:
  ERROR: Unable to register event handler 'defaultbase_eventhandler':
    File "/path/to/poky/meta/classes-global/base.bbclass", line 256
      an error line
         ^^^^^
  SyntaxError: invalid syntax

And no impact on parsing time:
* Before:
$ rm -fr cache tmp; time bitbake -p
real    0m27.254s

* Now:
$ rm -fr cache tmp; time bitbake -p
real    0m27.200s

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
---
 bitbake/lib/bb/event.py | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/bitbake/lib/bb/event.py b/bitbake/lib/bb/event.py
index f8acacd80d1..4761c868800 100644
--- a/bitbake/lib/bb/event.py
+++ b/bitbake/lib/bb/event.py
@@ -257,14 +257,15 @@ def register(name, handler, mask=None, filename=None, lineno=None, data=None):
         # handle string containing python code
         if isinstance(handler, str):
             tmp = "def %s(e, d):\n%s" % (name, handler)
+            # Inject empty lines to make code match lineno in filename
+            if lineno is not None:
+                tmp = "\n" * (lineno-1) + tmp
             try:
                 code = bb.methodpool.compile_cache(tmp)
                 if not code:
                     if filename is None:
                         filename = "%s(e, d)" % name
                     code = compile(tmp, filename, "exec", ast.PyCF_ONLY_AST)
-                    if lineno is not None:
-                        ast.increment_lineno(code, lineno-1)
                     code = compile(code, filename, "exec")
                     bb.methodpool.compile_cache_add(tmp, code)
             except SyntaxError:
-- 
2.35.5

[PATCH v4 2/2] bitbake: tests/event: Add test_lineno_in_eventhandler

[liezhi.yang]

From: Robert Yang <liezhi.yang@windriver.com>

Add test_lineno_in_eventhandler to test lineno in eventhandler.

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
---
 bitbake/lib/bb/tests/event.py | 24 ++++++++++++++++++++++++
 1 file changed, 24 insertions(+)

diff --git a/bitbake/lib/bb/tests/event.py b/bitbake/lib/bb/tests/event.py
index d959f2d95db..ef61891d302 100644
--- a/bitbake/lib/bb/tests/event.py
+++ b/bitbake/lib/bb/tests/event.py
@@ -13,6 +13,7 @@ import pickle
 import threading
 import time
 import unittest
+import tempfile
 from unittest.mock import Mock
 from unittest.mock import call
 
@@ -468,6 +469,8 @@ class EventClassesTest(unittest.TestCase):
 
     def setUp(self):
         bb.event.worker_pid = EventClassesTest._worker_pid
+        self.d = bb.data.init()
+        bb.parse.siggen = bb.siggen.init(self.d)
 
     def test_Event(self):
         """ Test the Event base class """
@@ -950,3 +953,24 @@ class EventClassesTest(unittest.TestCase):
         event = bb.event.FindSigInfoResult(result)
         self.assertEqual(event.result, result)
         self.assertEqual(event.pid, EventClassesTest._worker_pid)
+
+    def test_lineno_in_eventhandler(self):
+        # The error lineno is 5, not 4 since the first line is '\n'
+        error_line = """
+# Comment line1
+# Comment line2
+python test_lineno_in_eventhandler() {
+    This is an error line
+}
+addhandler test_lineno_in_eventhandler
+test_lineno_in_eventhandler[eventmask] = "bb.event.ConfigParsed"
+"""
+
+        with self.assertLogs() as logs:
+            f = tempfile.NamedTemporaryFile(suffix = '.bb')
+            f.write(bytes(error_line, "utf-8"))
+            f.flush()
+            d = bb.parse.handle(f.name, self.d)['']
+
+        output = "".join(logs.output)
+        self.assertTrue(" line 5\n" in output)
-- 
2.35.5

Re: [bitbake-devel] [PATCH v4 2/2] bitbake: tests/event: Add test_lineno_in_eventhandler

[Christopher Larson]

[-- Attachment #1: Type: text/plain, Size: 3359 bytes --]

This does look like a good idea, though I wonder if injecting newlines is truly the best approach. SyntaxError is a bit of a special case, since it occurs before we have the ast to adjust the lineno, but it can be corrected without string manipulation if you catch and re-raise SyntaxError. Ex:

def _syntaxerror_offset(value, lineoffset):
    """Adjust the line number in a SyntaxError exception"""
    if lineoffset:
        msg, (efname, elineno, eoffset, badline) = value.args
        value.args = (msg, (efname, elineno + lineoffset, eoffset, badline))
        value.lineno = elineno + lineoffset

— 
Christopher Larson
chris_larson@mentor.com, chris.larson@siemens.com, kergoth@gmail.com
Principal Software Engineer, Embedded Linux Solutions, Siemens Digital Industries Software

> On Jan 9, 2024, at 12:45 AM, Robert Yang via lists.openembedded.org <liezhi.yang=windriver.com@lists.openembedded.org> wrote:
> 
> From: Robert Yang <liezhi.yang@windriver.com>
> 
> Add test_lineno_in_eventhandler to test lineno in eventhandler.
> 
> Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
> ---
> bitbake/lib/bb/tests/event.py | 24 ++++++++++++++++++++++++
> 1 file changed, 24 insertions(+)
> 
> diff --git a/bitbake/lib/bb/tests/event.py b/bitbake/lib/bb/tests/event.py
> index d959f2d95db..ef61891d302 100644
> --- a/bitbake/lib/bb/tests/event.py
> +++ b/bitbake/lib/bb/tests/event.py
> @@ -13,6 +13,7 @@ import pickle
> import threading
> import time
> import unittest
> +import tempfile
> from unittest.mock import Mock
> from unittest.mock import call
> 
> @@ -468,6 +469,8 @@ class EventClassesTest(unittest.TestCase):
> 
>     def setUp(self):
>         bb.event.worker_pid = EventClassesTest._worker_pid
> +        self.d = bb.data.init()
> +        bb.parse.siggen = bb.siggen.init(self.d)
> 
>     def test_Event(self):
>         """ Test the Event base class """
> @@ -950,3 +953,24 @@ class EventClassesTest(unittest.TestCase):
>         event = bb.event.FindSigInfoResult(result)
>         self.assertEqual(event.result, result)
>         self.assertEqual(event.pid, EventClassesTest._worker_pid)
> +
> +    def test_lineno_in_eventhandler(self):
> +        # The error lineno is 5, not 4 since the first line is '\n'
> +        error_line = """
> +# Comment line1
> +# Comment line2
> +python test_lineno_in_eventhandler() {
> +    This is an error line
> +}
> +addhandler test_lineno_in_eventhandler
> +test_lineno_in_eventhandler[eventmask] = "bb.event.ConfigParsed"
> +"""
> +
> +        with self.assertLogs() as logs:
> +            f = tempfile.NamedTemporaryFile(suffix = '.bb')
> +            f.write(bytes(error_line, "utf-8"))
> +            f.flush()
> +            d = bb.parse.handle(f.name, self.d)['']
> +
> +        output = "".join(logs.output)
> +        self.assertTrue(" line 5\n" in output)
> -- 
> 2.35.5
> 
> 
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#15756): https://lists.openembedded.org/g/bitbake-devel/message/15756
> Mute This Topic: https://lists.openembedded.org/mt/103616263/3617123
> Group Owner: bitbake-devel+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/bitbake-devel/unsub [kergoth@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
> 


[-- Attachment #2: Type: text/html, Size: 4699 bytes --]

Re: [bitbake-devel] [PATCH v4 2/2] bitbake: tests/event: Add test_lineno_in_eventhandler

[Robert Yang]

Hi Christop,

On 1/10/24 6:08 AM, Christop her Larson wrote:
> This does look like a good idea, though I wonder if injecting newlines is truly 
> the best approach. SyntaxError is a bit of a special case, since it occurs 

Injecting newlines are much simple, and it also can catch more errors such as
IndentationError, now the IndentationError can be reported correctly:


ERROR: Unable to register event handler 'defaultbase_eventhandler':
   File "/path/to/poky/meta/classes-global/base.bbclass", line 259
     d.setVar("ORIGNATIVELSBSTRING", d.getVar("NATIVELSBSTRING", False))
                                                                        ^
IndentationError: unindent does not match any outer indentation level


Such errors were very hard to debug. The _syntaxerror_offset may work, but it is
much more complicated than inject newlines.

// Robert


> before we have the ast to adjust the lineno, but it can be corrected without 
> string manipulation if you catch and re-raise SyntaxError. Ex:
> 
> def _syntaxerror_offset(value, lineoffset):
>      """Adjust the line number in a SyntaxError exception"""
>      if lineoffset:
>          msg, (efname, elineno, eoffset, badline) = value.args
>          value.args = (msg, (efname, elineno + lineoffset, eoffset, badline))
>          value.lineno = elineno + lineoffset
> 
> —
> Christopher Larson
> chris_larson@mentor.com, chris.larson@siemens.com, kergoth@gmail.com
> Principal Software Engineer, Embedded Linux Solutions, Siemens Digital 
> Industries Software
> 
>> On Jan 9, 2024, at 12:45 AM, Robert Yang via lists.openembedded.org 
>> <https://urldefense.com/v3/__http://lists.openembedded.org__;!!AjveYdw8EvQ!YGDlH3GcpHwxON5oeaacPDSvoRd4ElezyHIHZpb7-vl5gV_Ip6p6zmPaMM2MO9LWGJTKQza6t0DNEMRXmNnO$> <liezhi.yang=windriver.com@lists.openembedded.org> wrote:
>>
>> From: Robert Yang <liezhi.yang@windriver.com>
>>
>> Add test_lineno_in_eventhandler to test lineno in eventhandler.
>>
>> Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
>> ---
>> bitbake/lib/bb/tests/event.py 
>> <https://urldefense.com/v3/__http://event.py__;!!AjveYdw8EvQ!YGDlH3GcpHwxON5oeaacPDSvoRd4ElezyHIHZpb7-vl5gV_Ip6p6zmPaMM2MO9LWGJTKQza6t0DNENKyLym7$> | 24 ++++++++++++++++++++++++
>> 1 file changed, 24 insertions(+)
>>
>> diff --git a/bitbake/lib/bb/tests/event.py 
>> <https://urldefense.com/v3/__http://event.py__;!!AjveYdw8EvQ!YGDlH3GcpHwxON5oeaacPDSvoRd4ElezyHIHZpb7-vl5gV_Ip6p6zmPaMM2MO9LWGJTKQza6t0DNENKyLym7$> b/bitbake/lib/bb/tests/event.py <https://urldefense.com/v3/__http://event.py__;!!AjveYdw8EvQ!YGDlH3GcpHwxON5oeaacPDSvoRd4ElezyHIHZpb7-vl5gV_Ip6p6zmPaMM2MO9LWGJTKQza6t0DNENKyLym7$>
>> index d959f2d95db..ef61891d302 100644
>> --- a/bitbake/lib/bb/tests/event.py 
>> <https://urldefense.com/v3/__http://event.py__;!!AjveYdw8EvQ!YGDlH3GcpHwxON5oeaacPDSvoRd4ElezyHIHZpb7-vl5gV_Ip6p6zmPaMM2MO9LWGJTKQza6t0DNENKyLym7$>
>> +++ b/bitbake/lib/bb/tests/event.py 
>> <https://urldefense.com/v3/__http://event.py__;!!AjveYdw8EvQ!YGDlH3GcpHwxON5oeaacPDSvoRd4ElezyHIHZpb7-vl5gV_Ip6p6zmPaMM2MO9LWGJTKQza6t0DNENKyLym7$>
>> @@ -13,6 +13,7 @@ import pickle
>> import threading
>> import time
>> import unittest
>> +import tempfile
>> from unittest.mock import Mock
>> from unittest.mock import call
>>
>> @@ -468,6 +469,8 @@ class EventClassesTest(unittest.TestCase):
>>
>>     def setUp(self):
>>         bb.event.worker_pid = EventClassesTest._worker_pid
>> +        self.d = bb.data.init()
>> +        bb.parse.siggen = bb.siggen.init(self.d)
>>
>>     def test_Event(self):
>>         """ Test the Event base class """
>> @@ -950,3 +953,24 @@ class EventClassesTest(unittest.TestCase):
>>         event = bb.event.FindSigInfoResult(result)
>>         self.assertEqual(event.result, result)
>>         self.assertEqual(event.pid 
>> <https://urldefense.com/v3/__http://event.pid__;!!AjveYdw8EvQ!YGDlH3GcpHwxON5oeaacPDSvoRd4ElezyHIHZpb7-vl5gV_Ip6p6zmPaMM2MO9LWGJTKQza6t0DNENhxad__$>, EventClassesTest._worker_pid)
>> +
>> +    def test_lineno_in_eventhandler(self):
>> +        # The error lineno is 5, not 4 since the first line is '\n'
>> +        error_line = """
>> +# Comment line1
>> +# Comment line2
>> +python test_lineno_in_eventhandler() {
>> +    This is an error line
>> +}
>> +addhandler test_lineno_in_eventhandler
>> +test_lineno_in_eventhandler[eventmask] = "bb.event.ConfigParsed"
>> +"""
>> +
>> +        with self.assertLogs() as logs:
>> +            f = tempfile.NamedTemporaryFile(suffix = '.bb')
>> +            f.write(bytes(error_line, "utf-8"))
>> +            f.flush()
>> +            d = bb.parse.handle(f.name 
>> <https://urldefense.com/v3/__http://f.name__;!!AjveYdw8EvQ!YGDlH3GcpHwxON5oeaacPDSvoRd4ElezyHIHZpb7-vl5gV_Ip6p6zmPaMM2MO9LWGJTKQza6t0DNEDC-ZnzD$>, self.d)['']
>> +
>> +        output = "".join(logs.output)
>> +        self.assertTrue(" line 5\n" in output)
>> -- 
>> 2.35.5
>>
>>
>> -=-=-=-=-=-=-=-=-=-=-=-
>> Links: You receive all messages sent to this group.
>> View/Reply Online (#15756): 
>> https://lists.openembedded.org/g/bitbake-devel/message/15756 
>> <https://urldefense.com/v3/__https://lists.openembedded.org/g/bitbake-devel/message/15756__;!!AjveYdw8EvQ!YGDlH3GcpHwxON5oeaacPDSvoRd4ElezyHIHZpb7-vl5gV_Ip6p6zmPaMM2MO9LWGJTKQza6t0DNEHCVD821$>
>> Mute This Topic: https://lists.openembedded.org/mt/103616263/3617123 
>> <https://urldefense.com/v3/__https://lists.openembedded.org/mt/103616263/3617123__;!!AjveYdw8EvQ!YGDlH3GcpHwxON5oeaacPDSvoRd4ElezyHIHZpb7-vl5gV_Ip6p6zmPaMM2MO9LWGJTKQza6t0DNEIbsb3Sa$>
>> Group Owner: bitbake-devel+owner@lists.openembedded.org
>> Unsubscribe: https://lists.openembedded.org/g/bitbake-devel/unsub 
>> <https://urldefense.com/v3/__https://lists.openembedded.org/g/bitbake-devel/unsub__;!!AjveYdw8EvQ!YGDlH3GcpHwxON5oeaacPDSvoRd4ElezyHIHZpb7-vl5gV_Ip6p6zmPaMM2MO9LWGJTKQza6t0DNEG8cswfl$> [kergoth@gmail.com]
>> -=-=-=-=-=-=-=-=-=-=-=-
>>
>