[PATCH 1/2] Improve types abstraction for gdb python scripts

[PATCH 1/2] Improve types abstraction for gdb python scripts

[buzdelabuz2]

From: Dom Cote <buzdelabuz2+git@gmail.com>

Change the read_u16 function so it accepts both 'str' and 'byte'
as type for the arguments.

When calling read_memory() from gdb API, depending on if
it was built with 2.7 or 3.X, the format used to return the
data will differ ( 'str' for 2.7, and 'byte' for 3.X ).

Add a function read_memoryview() to be able to get a
'memoryview' object back from read_memory() both with
python 2.7 and 3.X .

Tested with python 3.4 and 2.7
Tested with gdb 7.7

Signed-off-by: Dom Cote <buzdelabuz2+git@gmail.com>
---
 scripts/gdb/linux/utils.py | 17 +++++++++++++++--
 1 file changed, 15 insertions(+), 2 deletions(-)

diff --git a/scripts/gdb/linux/utils.py b/scripts/gdb/linux/utils.py
index 0893b326a28b..dba6c88ea7b9 100644
--- a/scripts/gdb/linux/utils.py
+++ b/scripts/gdb/linux/utils.py
@@ -87,11 +87,24 @@ def get_target_endianness():
     return target_endianness
 
 
+def read_memoryview(inf, start, length):
+    return memoryview(inf.read_memory(start, length))
+
+
 def read_u16(buffer):
+    value = [0, 0]
+
+    if type(buffer[0]) is str:
+        value[0] = ord(buffer[0])
+        value[1] = ord(buffer[1])
+    else:
+        value[0] = buffer[0]
+        value[1] = buffer[1]
+
     if get_target_endianness() == LITTLE_ENDIAN:
-        return ord(buffer[0]) + (ord(buffer[1]) << 8)
+        return value[0] + (value[1] << 8)
     else:
-        return ord(buffer[1]) + (ord(buffer[0]) << 8)
+        return value[1] + (value[0] << 8)
 
 
 def read_u32(buffer):
-- 
2.8.2

[PATCH 2/2] Fix issue with dmesg.py and python 3.X

[buzdelabuz2]

From: Dom Cote <buzdelabuz2+git@gmail.com>

Replace the addition (+) of 2 python 'memoryview' objects
with the addition of 2 'bytes' objects, convert the result
back to memoryview.

Tested with python 3.4 and 2.7
Tested with gdb 7.7

Signed-off-by: Dom Cote <buzdelabuz2+git@gmail.com>
---
 scripts/gdb/linux/dmesg.py | 7 ++++---
 1 file changed, 4 insertions(+), 3 deletions(-)

diff --git a/scripts/gdb/linux/dmesg.py b/scripts/gdb/linux/dmesg.py
index 927d0d2a3145..04d6719067f2 100644
--- a/scripts/gdb/linux/dmesg.py
+++ b/scripts/gdb/linux/dmesg.py
@@ -33,11 +33,12 @@ class LxDmesg(gdb.Command):
         if log_first_idx < log_next_idx:
             log_buf_2nd_half = -1
             length = log_next_idx - log_first_idx
-            log_buf = inf.read_memory(start, length)
+            log_buf = utils.read_memoryview(inf, start, length).tobytes()
         else:
             log_buf_2nd_half = log_buf_len - log_first_idx
-            log_buf = inf.read_memory(start, log_buf_2nd_half) + \
-                inf.read_memory(log_buf_addr, log_next_idx)
+            a = utils.read_memoryview(inf, start, log_buf_2nd_half)
+            b = utils.read_memoryview(inf, log_buf_addr, log_next_idx)
+            log_buf = a.tobytes() + b.tobytes()
 
         pos = 0
         while pos < log_buf.__len__():
-- 
2.8.2

Re: [PATCH 2/2] Fix issue with dmesg.py and python 3.X

[Jan Kiszka]

On 2016-05-04 05:12, buzdelabuz2@gmail.com wrote:
> From: Dom Cote <buzdelabuz2+git@gmail.com>
> 
> Replace the addition (+) of 2 python 'memoryview' objects
> with the addition of 2 'bytes' objects, convert the result
> back to memoryview.
> 
> Tested with python 3.4 and 2.7
> Tested with gdb 7.7

A word here on what setup was found broken would have been good. If you
have a sentence at hand, we can fold it in during merge.

> 
> Signed-off-by: Dom Cote <buzdelabuz2+git@gmail.com>
> ---
>  scripts/gdb/linux/dmesg.py | 7 ++++---
>  1 file changed, 4 insertions(+), 3 deletions(-)
> 
> diff --git a/scripts/gdb/linux/dmesg.py b/scripts/gdb/linux/dmesg.py
> index 927d0d2a3145..04d6719067f2 100644
> --- a/scripts/gdb/linux/dmesg.py
> +++ b/scripts/gdb/linux/dmesg.py
> @@ -33,11 +33,12 @@ class LxDmesg(gdb.Command):
>          if log_first_idx < log_next_idx:
>              log_buf_2nd_half = -1
>              length = log_next_idx - log_first_idx
> -            log_buf = inf.read_memory(start, length)
> +            log_buf = utils.read_memoryview(inf, start, length).tobytes()
>          else:
>              log_buf_2nd_half = log_buf_len - log_first_idx
> -            log_buf = inf.read_memory(start, log_buf_2nd_half) + \
> -                inf.read_memory(log_buf_addr, log_next_idx)
> +            a = utils.read_memoryview(inf, start, log_buf_2nd_half)
> +            b = utils.read_memoryview(inf, log_buf_addr, log_next_idx)
> +            log_buf = a.tobytes() + b.tobytes()
>  
>          pos = 0
>          while pos < log_buf.__len__():
> 

Patches look good to me on first glance. Need to try out as well.

Kieran?

Thanks,
Jan

-- 
Siemens AG, Corporate Technology, CT RDA ITP SES-DE
Corporate Competence Center Embedded Linux

Re: [PATCH 2/2] Fix issue with dmesg.py and python 3.X

[Kieran Bingham]

On 04/05/16 04:12, buzdelabuz2@gmail.com wrote:
> From: Dom Cote <buzdelabuz2+git@gmail.com>
> 
> Replace the addition (+) of 2 python 'memoryview' objects
> with the addition of 2 'bytes' objects, convert the result
> back to memoryview.

I'm a little concerned with the text above not quite sounding like the
code below, as the two 'bytes' objects are not converted back to a
memoryview object, rather the memoryview objects are converted to
'bytes' objects (i.e. the result in log_buf) ...

Could I suggest something like the following commit-log-message? (Also
based on Jan's feedback)

---8<---
When built against Python 3, GDB differs in the return type for its
read_memory function, causing the lx-dmesg command to fail.

Now that we have an improved read_16() we can use the new
read_memoryview() abstraction to make lx-dmesg return valid data on both
current Python APIs
--->8---

(Only a suggestion of course, if you feel it should say something else
go with that)

> 
> Tested with python 3.4 and 2.7
> Tested with gdb 7.7

I have also tested with Python 3.4, 2.7 on GDB 11, and I've tested that
things still work on a wrapped log-buffer too.

Looking good.

The only final note is that because of the way Python 3 prints a
byte-stream, we end up with:


[    0.000000] b'x86/fpu: Legacy x87 FPU detected.'
instead of:
[    0.000000] x86/fpu: Legacy x87 FPU detected.

on all of the lines.

I've just done a little digging, and seen we now need to '.decode()' the
line for printing, so I'll post that as a quick patch in reply to this set.


> Signed-off-by: Dom Cote <buzdelabuz2+git@gmail.com>

Tested-by: Kieran Bingham <kieran@bingham.xyz>
Reviewed-by: Kieran Bingham <kieran@bingham.xyz>


> ---
>  scripts/gdb/linux/dmesg.py | 7 ++++---
>  1 file changed, 4 insertions(+), 3 deletions(-)
> 
> diff --git a/scripts/gdb/linux/dmesg.py b/scripts/gdb/linux/dmesg.py
> index 927d0d2a3145..04d6719067f2 100644
> --- a/scripts/gdb/linux/dmesg.py
> +++ b/scripts/gdb/linux/dmesg.py
> @@ -33,11 +33,12 @@ class LxDmesg(gdb.Command):
>          if log_first_idx < log_next_idx:
>              log_buf_2nd_half = -1
>              length = log_next_idx - log_first_idx
> -            log_buf = inf.read_memory(start, length)
> +            log_buf = utils.read_memoryview(inf, start, length).tobytes()
>          else:
>              log_buf_2nd_half = log_buf_len - log_first_idx
> -            log_buf = inf.read_memory(start, log_buf_2nd_half) + \
> -                inf.read_memory(log_buf_addr, log_next_idx)
> +            a = utils.read_memoryview(inf, start, log_buf_2nd_half)
> +            b = utils.read_memoryview(inf, log_buf_addr, log_next_idx)
> +            log_buf = a.tobytes() + b.tobytes()
>  
>          pos = 0
>          while pos < log_buf.__len__():
> 

-- 
Regards

Kieran Bingham

[PATCH] scripts/gdb: decode bytestream on dmesg for Python3

[Kieran Bingham]

The recent fixes to lx-dmesg, now allow the command to print
successfully on Python3, however the python interpreter wraps the bytes
for each line with a b'<text>' marker.

To remove this, we need to decode the line, where .decode() will default
to 'UTF-8'

Signed-off-by: Kieran Bingham <kieran@bingham.xyz>
---
 scripts/gdb/linux/dmesg.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/scripts/gdb/linux/dmesg.py b/scripts/gdb/linux/dmesg.py
index 04d6719067f2..f9b92ece7834 100644
--- a/scripts/gdb/linux/dmesg.py
+++ b/scripts/gdb/linux/dmesg.py
@@ -51,10 +51,10 @@ class LxDmesg(gdb.Command):
                 continue
 
             text_len = utils.read_u16(log_buf[pos + 10:pos + 12])
-            text = log_buf[pos + 16:pos + 16 + text_len]
+            text = log_buf[pos + 16:pos + 16 + text_len].decode()
             time_stamp = utils.read_u64(log_buf[pos:pos + 8])
 
-            for line in memoryview(text).tobytes().splitlines():
+            for line in text.splitlines():
                 gdb.write("[{time:12.6f}] {line}\n".format(
                     time=time_stamp / 1000000000.0,
                     line=line))
-- 
2.5.0

Re: [PATCH] scripts/gdb: decode bytestream on dmesg for Python3

[Dom Cote]

Looks good, b'<text>' marker doesn't appear anymore.
Tested for python 2.7 and 3.4

Acked-by:  Dom Cote <buzdelabuz2@gmail.com>
Tested-by: Dom Cote <buzdelabuz2@gmail.com>

Regards

On Wed, May 4, 2016 at 8:37 AM, Kieran Bingham <kieran@bingham.xyz> wrote:
> The recent fixes to lx-dmesg, now allow the command to print
> successfully on Python3, however the python interpreter wraps the bytes
> for each line with a b'<text>' marker.
>
> To remove this, we need to decode the line, where .decode() will default
> to 'UTF-8'
>
> Signed-off-by: Kieran Bingham <kieran@bingham.xyz>
> ---
>  scripts/gdb/linux/dmesg.py | 4 ++--
>  1 file changed, 2 insertions(+), 2 deletions(-)
>
> diff --git a/scripts/gdb/linux/dmesg.py b/scripts/gdb/linux/dmesg.py
> index 04d6719067f2..f9b92ece7834 100644
> --- a/scripts/gdb/linux/dmesg.py
> +++ b/scripts/gdb/linux/dmesg.py
> @@ -51,10 +51,10 @@ class LxDmesg(gdb.Command):
>                  continue
>
>              text_len = utils.read_u16(log_buf[pos + 10:pos + 12])
> -            text = log_buf[pos + 16:pos + 16 + text_len]
> +            text = log_buf[pos + 16:pos + 16 + text_len].decode()
>              time_stamp = utils.read_u64(log_buf[pos:pos + 8])
>
> -            for line in memoryview(text).tobytes().splitlines():
> +            for line in text.splitlines():
>                  gdb.write("[{time:12.6f}] {line}\n".format(
>                      time=time_stamp / 1000000000.0,
>                      line=line))
> --
> 2.5.0
>

Re: [PATCH 1/2] Improve types abstraction for gdb python scripts

[Kieran Bingham]

On 04/05/16 04:12, buzdelabuz2@gmail.com wrote:
> From: Dom Cote <buzdelabuz2+git@gmail.com>
> 
> Change the read_u16 function so it accepts both 'str' and 'byte'
> as type for the arguments.
> 
> When calling read_memory() from gdb API, depending on if
> it was built with 2.7 or 3.X, the format used to return the
> data will differ ( 'str' for 2.7, and 'byte' for 3.X ).
> 
> Add a function read_memoryview() to be able to get a
> 'memoryview' object back from read_memory() both with
> python 2.7 and 3.X .
> 
> Tested with python 3.4 and 2.7
> Tested with gdb 7.7

I have also tested with Python 3.4, 2.7 on GDB 11

> 
> Signed-off-by: Dom Cote <buzdelabuz2+git@gmail.com>

Acked-by: Kieran Bingham <kieran@bingham.xyz>
Tested-by: Kieran Bingham <kieran@bingham.xyz>

> ---
>  scripts/gdb/linux/utils.py | 17 +++++++++++++++--
>  1 file changed, 15 insertions(+), 2 deletions(-)
> 
> diff --git a/scripts/gdb/linux/utils.py b/scripts/gdb/linux/utils.py
> index 0893b326a28b..dba6c88ea7b9 100644
> --- a/scripts/gdb/linux/utils.py
> +++ b/scripts/gdb/linux/utils.py
> @@ -87,11 +87,24 @@ def get_target_endianness():
>      return target_endianness
>  
>  
> +def read_memoryview(inf, start, length):
> +    return memoryview(inf.read_memory(start, length))
> +
> +
>  def read_u16(buffer):
> +    value = [0, 0]
> +
> +    if type(buffer[0]) is str:
> +        value[0] = ord(buffer[0])
> +        value[1] = ord(buffer[1])
> +    else:
> +        value[0] = buffer[0]
> +        value[1] = buffer[1]
> +
>      if get_target_endianness() == LITTLE_ENDIAN:
> -        return ord(buffer[0]) + (ord(buffer[1]) << 8)
> +        return value[0] + (value[1] << 8)
>      else:
> -        return ord(buffer[1]) + (ord(buffer[0]) << 8)
> +        return value[1] + (value[0] << 8)
>  
>  
>  def read_u32(buffer):
> 

-- 
Regards

Kieran Bingham

[PATCH 1/2] Improve types abstraction for gdb python scripts

[buzdelabuz2]

From: Dom Cote <buzdelabuz2+git@gmail.com>

Change the read_u16 function so it accepts both 'str' and 'byte'
as type for the arguments.

When calling read_memory() from gdb API, depending on if
it was built with 2.7 or 3.X, the format used to return the
data will differ ( 'str' for 2.7, and 'byte' for 3.X ).

Add a function read_memoryview() to be able to get a
'memoryview' object back from read_memory() both with
python 2.7 and 3.X .

Tested with python 3.4 and 2.7
Tested with gdb 7.7

Signed-off-by: Dom Cote <buzdelabuz2+git@gmail.com>
---
 scripts/gdb/linux/utils.py | 17 +++++++++++++++--
 1 file changed, 15 insertions(+), 2 deletions(-)

diff --git a/scripts/gdb/linux/utils.py b/scripts/gdb/linux/utils.py
index 0893b326a28b..dba6c88ea7b9 100644
--- a/scripts/gdb/linux/utils.py
+++ b/scripts/gdb/linux/utils.py
@@ -87,11 +87,24 @@ def get_target_endianness():
     return target_endianness
 
 
+def read_memoryview(inf, start, length):
+    return memoryview(inf.read_memory(start, length))
+
+
 def read_u16(buffer):
+    value = [0, 0]
+
+    if type(buffer[0]) is str:
+        value[0] = ord(buffer[0])
+        value[1] = ord(buffer[1])
+    else:
+        value[0] = buffer[0]
+        value[1] = buffer[1]
+
     if get_target_endianness() == LITTLE_ENDIAN:
-        return ord(buffer[0]) + (ord(buffer[1]) << 8)
+        return value[0] + (value[1] << 8)
     else:
-        return ord(buffer[1]) + (ord(buffer[0]) << 8)
+        return value[1] + (value[0] << 8)
 
 
 def read_u32(buffer):
-- 
2.8.2

[PATCH 2/2] Fix issue with dmesg.py and python 3.X

[buzdelabuz2]

From: Dom Cote <buzdelabuz2+git@gmail.com>

Replace the addition (+) of 2 python 'memoryview' objects
with the addition of 2 'bytes' objects, convert the result
back to memoryview.

Tested with python 3.4 and 2.7
Tested with gdb 7.7

Signed-off-by: Dom Cote <buzdelabuz2+git@gmail.com>
---
 scripts/gdb/linux/dmesg.py | 7 ++++---
 1 file changed, 4 insertions(+), 3 deletions(-)

diff --git a/scripts/gdb/linux/dmesg.py b/scripts/gdb/linux/dmesg.py
index 927d0d2a3145..04d6719067f2 100644
--- a/scripts/gdb/linux/dmesg.py
+++ b/scripts/gdb/linux/dmesg.py
@@ -33,11 +33,12 @@ class LxDmesg(gdb.Command):
         if log_first_idx < log_next_idx:
             log_buf_2nd_half = -1
             length = log_next_idx - log_first_idx
-            log_buf = inf.read_memory(start, length)
+            log_buf = utils.read_memoryview(inf, start, length).tobytes()
         else:
             log_buf_2nd_half = log_buf_len - log_first_idx
-            log_buf = inf.read_memory(start, log_buf_2nd_half) + \
-                inf.read_memory(log_buf_addr, log_next_idx)
+            a = utils.read_memoryview(inf, start, log_buf_2nd_half)
+            b = utils.read_memoryview(inf, log_buf_addr, log_next_idx)
+            log_buf = a.tobytes() + b.tobytes()
 
         pos = 0
         while pos < log_buf.__len__():
-- 
2.8.2