[PATCH v2 0/2] Python: setuptools v60+ workaround

[PATCH v2 0/2] Python: setuptools v60+ workaround

[John Snow]

Long story short: Python 3.7/3.8 on Fedora with setuptools v60.0.0+
together create a bug that ultimately causes pylint to fail. See the
first commit message for more detail.

I sent out a hotfix two weeks ago to fix this behavior on our CI, but
there's a better workaround. This adds the better workaround and reverts
the hotfix.

John Snow (2):
  Python: add setuptools v60.0 workaround
  Revert "python: pin setuptools below v60.0.0"

 python/Makefile                | 2 --
 python/setup.cfg               | 1 -
 python/tests/iotests-pylint.sh | 3 ++-
 python/tests/pylint.sh         | 3 ++-
 4 files changed, 4 insertions(+), 5 deletions(-)

-- 
2.34.1

[PATCH v2 1/2] Python: add setuptools v60.0 workaround

[John Snow]

Setuptools v60 and later include a bundled version of distutils, a
deprecated standard library scheduled for removal in future versions of
Python. Setuptools v60 is only possible to install for Python 3.7 and later.

Python has a distutils.sysconfig.get_python_lib() function that returns
'/usr/lib/pythonX.Y' on posix systems. RPM-based systems actually use
'/usr/lib64/pythonX.Y' instead, so Fedora patches stdlib distutils for
Python 3.7 and Python 3.8 to return the correct value.

Python 3.9 and later introduce a sys.platlibdir property, which returns
the correct value on RPM-based systems.

The change to a distutils package not provided by Fedora on Python 3.7
and 3.8 causes a regression in distutils.sysconfig.get_python_lib() that
ultimately causes false positives to be emitted by pylint, because it
can no longer find the system source libraries.

Many Python tools are fairly aggressive about updating setuptools
packages, and so even though this package is a fair bit newer than
Python 3.7/3.8, it's not entirely unreasonable for a given user to have
such a modern package with a fairly old Python interpreter.

Updates to Python 3.7 and Python 3.8 are being produced for Fedora which
will fix the problem on up-to-date systems. Until then, we can force the
loading of platform-provided distutils when running the pylint
test. This is the least-invasive yet most comprehensive fix.

References:
 https://github.com/pypa/setuptools/pull/2896
 https://github.com/PyCQA/pylint/issues/5704
 https://github.com/pypa/distutils/issues/110

Signed-off-by: John Snow <jsnow@redhat.com>
---
 python/tests/iotests-pylint.sh | 3 ++-
 python/tests/pylint.sh         | 3 ++-
 2 files changed, 4 insertions(+), 2 deletions(-)

diff --git a/python/tests/iotests-pylint.sh b/python/tests/iotests-pylint.sh
index 4cae03424b..33c5ae900a 100755
--- a/python/tests/iotests-pylint.sh
+++ b/python/tests/iotests-pylint.sh
@@ -1,4 +1,5 @@
 #!/bin/sh -e
 
 cd ../tests/qemu-iotests/
-python3 -m linters --pylint
+# See commit message for environment variable explainer.
+SETUPTOOLS_USE_DISTUTILS=stdlib python3 -m linters --pylint
diff --git a/python/tests/pylint.sh b/python/tests/pylint.sh
index 4b10b34db7..03d64705a1 100755
--- a/python/tests/pylint.sh
+++ b/python/tests/pylint.sh
@@ -1,2 +1,3 @@
 #!/bin/sh -e
-python3 -m pylint qemu/
+# See commit message for environment variable explainer.
+SETUPTOOLS_USE_DISTUTILS=stdlib python3 -m pylint qemu/
-- 
2.34.1

[PATCH v2 2/2] Revert "python: pin setuptools below v60.0.0"

[John Snow]

This reverts commit 1e4d8b31be35e54b6429fea54f5ecaa0083f91e7.

Signed-off-by: John Snow <jsnow@redhat.com>
---
 python/Makefile  | 2 --
 python/setup.cfg | 1 -
 2 files changed, 3 deletions(-)

diff --git a/python/Makefile b/python/Makefile
index 949c472624..3334311362 100644
--- a/python/Makefile
+++ b/python/Makefile
@@ -68,8 +68,6 @@ $(QEMU_VENV_DIR) $(QEMU_VENV_DIR)/bin/activate: setup.cfg
 		echo "ACTIVATE $(QEMU_VENV_DIR)";		\
 		. $(QEMU_VENV_DIR)/bin/activate;		\
 		echo "INSTALL qemu[devel] $(QEMU_VENV_DIR)";	\
-		pip install --disable-pip-version-check		\
-			"setuptools<60.0.0" 1>/dev/null;	\
 		make develop 1>/dev/null;			\
 	)
 	@touch $(QEMU_VENV_DIR)
diff --git a/python/setup.cfg b/python/setup.cfg
index 3fb18f845d..168a79c867 100644
--- a/python/setup.cfg
+++ b/python/setup.cfg
@@ -163,7 +163,6 @@ deps =
     .[devel]
     .[fuse]  # Workaround to trigger tox venv rebuild
     .[tui]   # Workaround to trigger tox venv rebuild
-    setuptools < 60  # Workaround, please see commit msg.
 commands =
     make check
 
-- 
2.34.1

Re: [PATCH v2 0/2] Python: setuptools v60+ workaround

[John Snow]

On Fri, Feb 4, 2022 at 5:18 PM John Snow <jsnow@redhat.com> wrote:
>
> Long story short: Python 3.7/3.8 on Fedora with setuptools v60.0.0+
> together create a bug that ultimately causes pylint to fail. See the
> first commit message for more detail.
>
> I sent out a hotfix two weeks ago to fix this behavior on our CI, but
> there's a better workaround. This adds the better workaround and reverts
> the hotfix.
>
> John Snow (2):
>   Python: add setuptools v60.0 workaround
>   Revert "python: pin setuptools below v60.0.0"
>
>  python/Makefile                | 2 --
>  python/setup.cfg               | 1 -
>  python/tests/iotests-pylint.sh | 3 ++-
>  python/tests/pylint.sh         | 3 ++-
>  4 files changed, 4 insertions(+), 5 deletions(-)
>
> --
> 2.34.1
>

Assuming no news is good news. I'm staging this to my Python branch,
and intend to send a PR after I get an ACK on '[PATCH 0/4] iotests:
finalize switch to async QMP' -- so there's time to veto this if you
have concerns.

--js