Proposed way forward Re: [PATCH v2 0/7] Python: Drop support for Python 3.6

["Daniel P. Berrangé" <berrange@redhat.com>]

On Thu, Feb 09, 2023 at 07:31:40PM -0500, John Snow wrote:
> Howdy, this series increases our minimum python version to 3.7.

..snip...

So based on the discussion in this thread I see the following as the
key take aways:

 * Use of newer Meson feature

   We need newer meson in order to continue/finish the conversion
   of configure shell script into meson.build file logic. The newer
   meson has python 3.7 as a minimum required.

   The switch to meson was/is a significant strategic goal for
   improving QEMU build system maintainability which is thus
   highly desirable to continue without arbitrary delay.

 * Use of newer python parameter type hinting annotations

   Newer mypy supports a wider range of python type hinting
   annotations. While these ought to only impact the need for
   a new mypy for offline static analysis, in practice they
   also impact the runtime execution of the python code
   breaking compatibility with python 3.6. This unfortunate
   but out of our control.

   The wider use of type hinting in our python code is improving
   its maintainability, because it makes it significantly simpler
   for contributors to understand the behaviour of the code.


 * Use of newer python static analysis tools in general
 
   This applies to thinks like pylint, flak8, mypy (any more?)
   The issues reported by newer versions are not neccessarily
   always a strict superset of issues reports by older versions.
   Thus you can't assume that a passing result from new tools
   implies a passing result from the old tools. As such supporting
   all older versions is a cost/burden with little to no upside.

   With the exception of mypy, these aren't believed to have a
   direct bearing on the minimum python runtime version, but this
   would still technically be a deviation from our current support
   policy.


The first two points feel like pretty decent functional reasons to
justify an increase in the minimum python version. They will both
unlock the use of features that will have a tangible benefit to the
maintainability of QEMU in key areas.

If we do the first two points, then we can justify the 3rd point too
as having negligible additional downside.


The question is how to reconcile this with the platform support policy.
The policy has two goals. First to give us as maintainers a clear rule
as to when it is acceptable to increase minimum versions. The intent
is we don't need to debate the merits of each proposed change, just
validate that it follows our stated support policy. The second goal
is to give consumers of QEMU guidance as to what their platform has
to be able to provide in order to build and deploy QEMU.

The long life enterprise distros are the pain point in unlocking the
new of python features. So the proposal is that, at a minimum, we
augment the current policy with words to the effect that:

  * For long life cycle distributions, QEMU will follow normal
    policy for platform level dependancies / native code. For
    Python modules, QEMU may choose to require a newer versions
    than are available from the distribution package manager.

Potentially we could go further and reserve the right to
mandate newer python versions for any OS distro, because once
we have the facility for doing this with enterprise distros,
that shoudl just work anywhere. Perhaps start with targetted
exception at long life distros though. Either would unblock
our immediate needs wrt droppping python 3.6.

Finally, in terms of putting this into practice

  * If the OS platform doesn't provide the sufficiently new
    python modules, QEMU should default to downloading and
    installing them using pip, into a virtualenv under the
    QEMU build root so we don't impact other stuff the user
    might have in $HOME/.local/lib/python*. IIUC, the use
    of virtualenvs has been desired for a while already.

  * QEMU must provide a mechanism to disable any automatic
    downloading with pip, to allow users to pre-populate the
    addon modules if desired, to cope with scenarios where
    network access isn't available. Should be straightforward.

  * CI should continue to test all features, and be able
    to install the extra pieces with pip, if relevant for
    the distro. This should already be doable with the
    lcitool stuff Paolo has got working, which seemlessly
    uses pip on distros if needed.


So basically I'm suggesting we take a combination of John's series
here, and Paolo's series. John's last patch would change so that our
dockerfiles grab docutils with pip instead of disabling docs build
on centos8. Then look at introducing the use of virtualenvs for auto
installing any python mods the OS doesn't satisfy.

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|