[Qemu-devel] Making QEMU build with Python 3

[Qemu-devel] Making QEMU build with Python 3

[Stefan Hajnoczi]

Ross created a bug to track Python 3 support:

  https://bugs.launchpad.net/qemu/+bug/1708462

Currently most Python code in QEMU is for Python 2.6+ only.  There
have only been a few patches adding Python 3 support to certain
scripts so far.

In this email I want to highlight the most important scripts that need
Python 3 support.  Volunteers are welcome!

Python scripts needed to build QEMU are the highest priority.  They
are invoked by ./configure or make.  I've identified the following:

scripts/signrom.py
scripts/qapi*.py
scripts/modules/module_block.py
scripts/tracetool*

Anyone wishing to tackle a script listed here, please reply to this
email thread to avoid duplicating work.

The fundamentals of adding Python 3 support are:

1. The script must work correctly under both Python 2.6+ and Python 3.
Only use language or standard library features that are available in
both Python versions.
2. Compare Python 2.6 vs Python 3 documentation to find a common
subset.  There is often a Pythonic solution that does not require
writing explicit wrappers.
3. Avoid third-party package dependencies - QEMU currently has none!
That means do not use 'six' or 'python-future'.  Our use of Python
isn't that fancy, but if you feel a third party package is essential
the please justify it.
4. If you decide to do PEP8 cleanups, make them separate patches so
review is easy.

Getting started info (but do not rely on 'python-future'):
http://www.python-future.org/compatible_idioms.html

Once the build scripts are converted the next most important group of
Python scripts are the tests.  This is where the bulk of the work
lies.

I have tracetool on my todo list and hope to add Python 3 support in QEMU 2.11.

Stefan

Re: [Qemu-devel] Making QEMU build with Python 3

[Philippe Mathieu-Daudé]

Hi Stefan,

On 08/09/2017 07:16 AM, Stefan Hajnoczi wrote:
[...]> Python scripts needed to build QEMU are the highest priority.  They
> are invoked by ./configure or make.  I've identified the following:
> 
> scripts/signrom.py
> scripts/qapi*.py
> scripts/modules/module_block.py
> scripts/tracetool*
> 
[...]
> The fundamentals of adding Python 3 support are:
> 
> 1. The script must work correctly under both Python 2.6+ and Python 3.
[...]
> 3. Avoid third-party package dependencies - QEMU currently has none!

This seems true for the "invoked by ./configure or make" set. However:

scripts/qemu-gdb.py:20:import gdb
scripts/qemugdb/aio.py:13:import gdb
scripts/qemugdb/coroutine.py:16:import gdb
scripts/qemugdb/mtree.py:18:import gdb

I just checked gdb8 and can't confirm python3 build works.

Also another 3rd party (python3 compliant):

scripts/analyze-migration.py:20:import numpy as np

Off-topic but we might document how to install dependencies for those 
scripts?

> That means do not use 'six' or 'python-future'.  Our use of Python
> isn't that fancy, but if you feel a third party package is essential
> the please justify it.
[...]

Regards,

Phil.

Re: [Qemu-devel] Making QEMU build with Python 3

[Daniel P. Berrange]

On Wed, Aug 09, 2017 at 11:16:58AM +0100, Stefan Hajnoczi wrote:
> Ross created a bug to track Python 3 support:
> 
>   https://bugs.launchpad.net/qemu/+bug/1708462
> 
> Currently most Python code in QEMU is for Python 2.6+ only.  There
> have only been a few patches adding Python 3 support to certain
> scripts so far.
> 
> In this email I want to highlight the most important scripts that need
> Python 3 support.  Volunteers are welcome!
> 
> Python scripts needed to build QEMU are the highest priority.  They
> are invoked by ./configure or make.  I've identified the following:
> 
> scripts/signrom.py
> scripts/qapi*.py
> scripts/modules/module_block.py
> scripts/tracetool*

> I have tracetool on my todo list and hope to add Python 3 support in QEMU 2.11.

AFAICT tracetool seems to already work - i hacked rules.mak to
run tracetool via python3 and nothing obvious broke

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [Qemu-devel] Making QEMU build with Python 3

[Stefan Hajnoczi]

On Wed, Aug 9, 2017 at 2:10 PM, Daniel P. Berrange <berrange@redhat.com> wrote:
> On Wed, Aug 09, 2017 at 11:16:58AM +0100, Stefan Hajnoczi wrote:
>> Ross created a bug to track Python 3 support:
>>
>>   https://bugs.launchpad.net/qemu/+bug/1708462
>>
>> Currently most Python code in QEMU is for Python 2.6+ only.  There
>> have only been a few patches adding Python 3 support to certain
>> scripts so far.
>>
>> In this email I want to highlight the most important scripts that need
>> Python 3 support.  Volunteers are welcome!
>>
>> Python scripts needed to build QEMU are the highest priority.  They
>> are invoked by ./configure or make.  I've identified the following:
>>
>> scripts/signrom.py
>> scripts/qapi*.py
>> scripts/modules/module_block.py
>> scripts/tracetool*
>
>> I have tracetool on my todo list and hope to add Python 3 support in QEMU 2.11.
>
> AFAICT tracetool seems to already work - i hacked rules.mak to
> run tracetool via python3 and nothing obvious broke

Nice, thanks for checking!  I'll test all the tracetool backends and
audit the code for 2.11.

Stefan

Re: [Qemu-devel] Making QEMU build with Python 3

[Stefan Hajnoczi]

On Wed, Aug 9, 2017 at 1:53 PM, Philippe Mathieu-Daudé <f4bug@amsat.org> wrote:
> On 08/09/2017 07:16 AM, Stefan Hajnoczi wrote:
> [...]> Python scripts needed to build QEMU are the highest priority.  They
>>
>> are invoked by ./configure or make.  I've identified the following:
>>
>> scripts/signrom.py
>> scripts/qapi*.py
>> scripts/modules/module_block.py
>> scripts/tracetool*
>>
> [...]
>>
>> The fundamentals of adding Python 3 support are:
>>
>> 1. The script must work correctly under both Python 2.6+ and Python 3.
>
> [...]
>>
>> 3. Avoid third-party package dependencies - QEMU currently has none!
>
>
> This seems true for the "invoked by ./configure or make" set. However:
>
> scripts/qemu-gdb.py:20:import gdb
> scripts/qemugdb/aio.py:13:import gdb
> scripts/qemugdb/coroutine.py:16:import gdb
> scripts/qemugdb/mtree.py:18:import gdb
>
> I just checked gdb8 and can't confirm python3 build works.

Yes, this is a GDB extension written in Python.  The gdb module is
available in the interpreter environment that GDB sets up.

Even if GDB doesn't support Python 3 yet, they will eventually.  It's
okay if we cannot add Python 3 support to these scripts yet.

> Also another 3rd party (python3 compliant):
>
> scripts/analyze-migration.py:20:import numpy as np

Weird, this script seems to use numpy APIs for unpacking binary data
instead of the standard library struct module.  As far as I can tell
the usual numpy functionality isn't actually used.  It should be easy
to drop the numpy dependency.

> Off-topic but we might document how to install dependencies for those
> scripts?

The analyze-migration.py script doesn't have any documentation, AFAIK.

Stefan