* [RFC PATCH 0/4] Update CentOS and OpenSUSE CI to Python >=3.7
@ 2023-01-17 9:16 Paolo Bonzini
2023-01-17 9:16 ` [RFC PATCH 1/4] lcitool: update submodule Paolo Bonzini
` (4 more replies)
0 siblings, 5 replies; 9+ messages in thread
From: Paolo Bonzini @ 2023-01-17 9:16 UTC (permalink / raw)
To: qemu-devel; +Cc: alex.bennee, berrange
QEMU build and test scripts would like to use some new features that
were added to Python 3.7. Useful additions include removing the need
for OrderedDict, improved handling of default text encoding, postponed
evaluation of type annotations, and improvements to asyncio.
Using new features that were added to Meson 0.63 is also being blocked
by usage of EOL'd versions of Python; Meson 0.63 does not work with the
3.6 version of Python that is provided by CentOS Stream 8 and OpenSUSE
LEAP 15. Therefore, this series upgrades the CentOS Stream 8 and OpenSUSE
LEAP 15 dockerfiles to use the alternative Python runtimes provided by
the distros.
The main obstacle here is updating the lcitool-generated Dockerfiles;
lcitool is (by design) quite opinionated on the mapping between
"generic" package names and target-specific package names, and it
caters to Libvirt's needs in terms of package versions.
There are two possibilities to change this.
First, lately I have been working on allowing lcitool to be used as
a library. As of this morning, it is possible for QEMU to use
lcitool's databases and formatters directly from Python, and only
replace the search mechanism to allow mappings and target paths to
be overridden. This is the approach that is implemented in this RFC:
the first two patches in this series use libvirt-ci as library, while
the third implements the custom search path.
However, I have also proposed the custom search mechanism directly
upstream (https://gitlab.com/libvirt/libvirt-ci/-/merge_requests/341).
If this is accepted, patch 3 will become much smaller and will not
need to know any lcitool internals; in addition, using lcitool as
a library could be evaluated on its own merits because command-line
operation would be a possibility as well.
With the lcitool changes in place, the final patch shows how to
install custom package mappings and update CentOS and OpenSUSE Python
respectively to 3.8 and 3.9.
Thanks,
Paolo
Paolo Bonzini (4):
lcitool: update submodule
lcitool: use libvirt-ci as library
lcitool: allow overriding package mappings and target facts
ci, docker: update CentOS and OpenSUSE Python to non-EOL versions
.gitlab-ci.d/cirrus/freebsd-12.vars | 8 +-
.gitlab-ci.d/cirrus/freebsd-13.vars | 8 +-
.gitlab-ci.d/cirrus/macos-12.vars | 8 +-
scripts/ci/setup/build-environment.yml | 2 +-
tests/docker/dockerfiles/alpine.docker | 7 +-
tests/docker/dockerfiles/centos8.docker | 29 +++---
.../dockerfiles/debian-amd64-cross.docker | 7 +-
tests/docker/dockerfiles/debian-amd64.docker | 7 +-
.../dockerfiles/debian-arm64-cross.docker | 7 +-
.../dockerfiles/debian-armel-cross.docker | 7 +-
.../dockerfiles/debian-armhf-cross.docker | 7 +-
.../dockerfiles/debian-mips64el-cross.docker | 7 +-
.../dockerfiles/debian-mipsel-cross.docker | 7 +-
.../dockerfiles/debian-ppc64el-cross.docker | 7 +-
.../dockerfiles/debian-s390x-cross.docker | 7 +-
.../dockerfiles/fedora-win32-cross.docker | 7 +-
.../dockerfiles/fedora-win64-cross.docker | 7 +-
tests/docker/dockerfiles/fedora.docker | 7 +-
tests/docker/dockerfiles/opensuse-leap.docker | 29 +++---
tests/docker/dockerfiles/ubuntu2004.docker | 9 +-
tests/lcitool/libvirt-ci | 2 +-
tests/lcitool/mappings.yml | 80 ++++++++++++++++
tests/lcitool/projects/qemu.yml | 1 +
tests/lcitool/refresh | 96 ++++++++++++-------
tests/lcitool/targets/centos-stream-8.yml | 3 +
tests/lcitool/targets/opensuse-leap-153.yml | 3 +
26 files changed, 215 insertions(+), 154 deletions(-)
create mode 100644 tests/lcitool/mappings.yml
create mode 100644 tests/lcitool/targets/centos-stream-8.yml
create mode 100644 tests/lcitool/targets/opensuse-leap-153.yml
--
2.38.1
^ permalink raw reply [flat|nested] 9+ messages in thread
* [RFC PATCH 1/4] lcitool: update submodule
2023-01-17 9:16 [RFC PATCH 0/4] Update CentOS and OpenSUSE CI to Python >=3.7 Paolo Bonzini
@ 2023-01-17 9:16 ` Paolo Bonzini
2023-01-17 9:16 ` [RFC PATCH 2/4] lcitool: use libvirt-ci as library Paolo Bonzini
` (3 subsequent siblings)
4 siblings, 0 replies; 9+ messages in thread
From: Paolo Bonzini @ 2023-01-17 9:16 UTC (permalink / raw)
To: qemu-devel; +Cc: alex.bennee, berrange
Update lcitool to the latest submodule, which has improvements that
allow usage as a library.
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
---
tests/lcitool/libvirt-ci | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/tests/lcitool/libvirt-ci b/tests/lcitool/libvirt-ci
index a29b1c9925ed..d1456dd96183 160000
--- a/tests/lcitool/libvirt-ci
+++ b/tests/lcitool/libvirt-ci
@@ -1 +1 @@
-Subproject commit a29b1c9925ed8549b2a9e83374a8aa5ebe16bc71
+Subproject commit 40589eed1c56f040d0f07fc354c242a0e0d83185
--
2.38.1
^ permalink raw reply related [flat|nested] 9+ messages in thread
* [RFC PATCH 2/4] lcitool: use libvirt-ci as library
2023-01-17 9:16 [RFC PATCH 0/4] Update CentOS and OpenSUSE CI to Python >=3.7 Paolo Bonzini
2023-01-17 9:16 ` [RFC PATCH 1/4] lcitool: update submodule Paolo Bonzini
@ 2023-01-17 9:16 ` Paolo Bonzini
2023-02-08 17:27 ` Daniel P. Berrangé
2023-01-17 9:16 ` [RFC PATCH 3/4] lcitool: allow overriding package mappings and target facts Paolo Bonzini
` (2 subsequent siblings)
4 siblings, 1 reply; 9+ messages in thread
From: Paolo Bonzini @ 2023-01-17 9:16 UTC (permalink / raw)
To: qemu-devel; +Cc: alex.bennee, berrange
Using the lcitool package as a library will make it possible to
customize more of the process, for example by introducing custom
mappings.
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
---
.gitlab-ci.d/cirrus/freebsd-12.vars | 6 +-
.gitlab-ci.d/cirrus/freebsd-13.vars | 6 +-
.gitlab-ci.d/cirrus/macos-12.vars | 6 +-
tests/docker/dockerfiles/alpine.docker | 6 +-
tests/docker/dockerfiles/centos8.docker | 6 +-
.../dockerfiles/debian-amd64-cross.docker | 6 +-
tests/docker/dockerfiles/debian-amd64.docker | 6 +-
.../dockerfiles/debian-arm64-cross.docker | 6 +-
.../dockerfiles/debian-armel-cross.docker | 6 +-
.../dockerfiles/debian-armhf-cross.docker | 6 +-
.../dockerfiles/debian-mips64el-cross.docker | 6 +-
.../dockerfiles/debian-mipsel-cross.docker | 6 +-
.../dockerfiles/debian-ppc64el-cross.docker | 6 +-
.../dockerfiles/debian-s390x-cross.docker | 6 +-
.../dockerfiles/fedora-win32-cross.docker | 6 +-
.../dockerfiles/fedora-win64-cross.docker | 6 +-
tests/docker/dockerfiles/fedora.docker | 6 +-
tests/docker/dockerfiles/opensuse-leap.docker | 6 +-
tests/docker/dockerfiles/ubuntu2004.docker | 6 +-
tests/lcitool/refresh | 83 ++++++++++++-------
20 files changed, 70 insertions(+), 127 deletions(-)
diff --git a/.gitlab-ci.d/cirrus/freebsd-12.vars b/.gitlab-ci.d/cirrus/freebsd-12.vars
index e3fc3235b93e..f395e56fc833 100644
--- a/.gitlab-ci.d/cirrus/freebsd-12.vars
+++ b/.gitlab-ci.d/cirrus/freebsd-12.vars
@@ -1,8 +1,4 @@
-# THIS FILE WAS AUTO-GENERATED
-#
-# $ lcitool variables freebsd-12 qemu
-#
-# https://gitlab.com/libvirt/libvirt-ci
+# THIS FILE WAS AUTO-GENERATED BY tests/lcitool/refresh
CCACHE='/usr/local/bin/ccache'
CPAN_PKGS=''
diff --git a/.gitlab-ci.d/cirrus/freebsd-13.vars b/.gitlab-ci.d/cirrus/freebsd-13.vars
index 9f56babd9ca7..f395e56fc833 100644
--- a/.gitlab-ci.d/cirrus/freebsd-13.vars
+++ b/.gitlab-ci.d/cirrus/freebsd-13.vars
@@ -1,8 +1,4 @@
-# THIS FILE WAS AUTO-GENERATED
-#
-# $ lcitool variables freebsd-13 qemu
-#
-# https://gitlab.com/libvirt/libvirt-ci
+# THIS FILE WAS AUTO-GENERATED BY tests/lcitool/refresh
CCACHE='/usr/local/bin/ccache'
CPAN_PKGS=''
diff --git a/.gitlab-ci.d/cirrus/macos-12.vars b/.gitlab-ci.d/cirrus/macos-12.vars
index ef9e14b373f0..7fb182f93e16 100644
--- a/.gitlab-ci.d/cirrus/macos-12.vars
+++ b/.gitlab-ci.d/cirrus/macos-12.vars
@@ -1,8 +1,4 @@
-# THIS FILE WAS AUTO-GENERATED
-#
-# $ lcitool variables macos-12 qemu
-#
-# https://gitlab.com/libvirt/libvirt-ci
+# THIS FILE WAS AUTO-GENERATED BY tests/lcitool/refresh
CCACHE='/opt/homebrew/bin/ccache'
CPAN_PKGS=''
diff --git a/tests/docker/dockerfiles/alpine.docker b/tests/docker/dockerfiles/alpine.docker
index 094f66f4eb05..10f68f4e1e6d 100644
--- a/tests/docker/dockerfiles/alpine.docker
+++ b/tests/docker/dockerfiles/alpine.docker
@@ -1,8 +1,4 @@
-# THIS FILE WAS AUTO-GENERATED
-#
-# $ lcitool dockerfile --layers all alpine-316 qemu
-#
-# https://gitlab.com/libvirt/libvirt-ci
+# THIS FILE WAS AUTO-GENERATED BY tests/lcitool/refresh
FROM docker.io/library/alpine:3.16
diff --git a/tests/docker/dockerfiles/centos8.docker b/tests/docker/dockerfiles/centos8.docker
index 1f70d41aeb31..2318a95e9255 100644
--- a/tests/docker/dockerfiles/centos8.docker
+++ b/tests/docker/dockerfiles/centos8.docker
@@ -1,8 +1,4 @@
-# THIS FILE WAS AUTO-GENERATED
-#
-# $ lcitool dockerfile --layers all centos-stream-8 qemu
-#
-# https://gitlab.com/libvirt/libvirt-ci
+# THIS FILE WAS AUTO-GENERATED BY tests/lcitool/refresh
FROM quay.io/centos/centos:stream8
diff --git a/tests/docker/dockerfiles/debian-amd64-cross.docker b/tests/docker/dockerfiles/debian-amd64-cross.docker
index 5e57309361b6..bfa88dc7285f 100644
--- a/tests/docker/dockerfiles/debian-amd64-cross.docker
+++ b/tests/docker/dockerfiles/debian-amd64-cross.docker
@@ -1,8 +1,4 @@
-# THIS FILE WAS AUTO-GENERATED
-#
-# $ lcitool dockerfile --layers all --cross x86_64 debian-11 qemu
-#
-# https://gitlab.com/libvirt/libvirt-ci
+# THIS FILE WAS AUTO-GENERATED BY tests/lcitool/refresh
FROM docker.io/library/debian:11-slim
diff --git a/tests/docker/dockerfiles/debian-amd64.docker b/tests/docker/dockerfiles/debian-amd64.docker
index bfeab01ee3e2..e09118c83b2f 100644
--- a/tests/docker/dockerfiles/debian-amd64.docker
+++ b/tests/docker/dockerfiles/debian-amd64.docker
@@ -1,8 +1,4 @@
-# THIS FILE WAS AUTO-GENERATED
-#
-# $ lcitool dockerfile --layers all debian-11 qemu
-#
-# https://gitlab.com/libvirt/libvirt-ci
+# THIS FILE WAS AUTO-GENERATED BY tests/lcitool/refresh
FROM docker.io/library/debian:11-slim
diff --git a/tests/docker/dockerfiles/debian-arm64-cross.docker b/tests/docker/dockerfiles/debian-arm64-cross.docker
index 98885bd0eedf..b20197d5b710 100644
--- a/tests/docker/dockerfiles/debian-arm64-cross.docker
+++ b/tests/docker/dockerfiles/debian-arm64-cross.docker
@@ -1,8 +1,4 @@
-# THIS FILE WAS AUTO-GENERATED
-#
-# $ lcitool dockerfile --layers all --cross aarch64 debian-11 qemu
-#
-# https://gitlab.com/libvirt/libvirt-ci
+# THIS FILE WAS AUTO-GENERATED BY tests/lcitool/refresh
FROM docker.io/library/debian:11-slim
diff --git a/tests/docker/dockerfiles/debian-armel-cross.docker b/tests/docker/dockerfiles/debian-armel-cross.docker
index d5c08714e4de..792e098f273c 100644
--- a/tests/docker/dockerfiles/debian-armel-cross.docker
+++ b/tests/docker/dockerfiles/debian-armel-cross.docker
@@ -1,8 +1,4 @@
-# THIS FILE WAS AUTO-GENERATED
-#
-# $ lcitool dockerfile --layers all --cross armv6l debian-11 qemu
-#
-# https://gitlab.com/libvirt/libvirt-ci
+# THIS FILE WAS AUTO-GENERATED BY tests/lcitool/refresh
FROM docker.io/library/debian:11-slim
diff --git a/tests/docker/dockerfiles/debian-armhf-cross.docker b/tests/docker/dockerfiles/debian-armhf-cross.docker
index 471444fcf489..dbb9cf21e99b 100644
--- a/tests/docker/dockerfiles/debian-armhf-cross.docker
+++ b/tests/docker/dockerfiles/debian-armhf-cross.docker
@@ -1,8 +1,4 @@
-# THIS FILE WAS AUTO-GENERATED
-#
-# $ lcitool dockerfile --layers all --cross armv7l debian-11 qemu
-#
-# https://gitlab.com/libvirt/libvirt-ci
+# THIS FILE WAS AUTO-GENERATED BY tests/lcitool/refresh
FROM docker.io/library/debian:11-slim
diff --git a/tests/docker/dockerfiles/debian-mips64el-cross.docker b/tests/docker/dockerfiles/debian-mips64el-cross.docker
index 15b0224b76bf..712cf98388a1 100644
--- a/tests/docker/dockerfiles/debian-mips64el-cross.docker
+++ b/tests/docker/dockerfiles/debian-mips64el-cross.docker
@@ -1,8 +1,4 @@
-# THIS FILE WAS AUTO-GENERATED
-#
-# $ lcitool dockerfile --layers all --cross mips64el debian-11 qemu
-#
-# https://gitlab.com/libvirt/libvirt-ci
+# THIS FILE WAS AUTO-GENERATED BY tests/lcitool/refresh
FROM docker.io/library/debian:11-slim
diff --git a/tests/docker/dockerfiles/debian-mipsel-cross.docker b/tests/docker/dockerfiles/debian-mipsel-cross.docker
index a5d3ca6e2f7f..c746b7ca04aa 100644
--- a/tests/docker/dockerfiles/debian-mipsel-cross.docker
+++ b/tests/docker/dockerfiles/debian-mipsel-cross.docker
@@ -1,8 +1,4 @@
-# THIS FILE WAS AUTO-GENERATED
-#
-# $ lcitool dockerfile --layers all --cross mipsel debian-11 qemu
-#
-# https://gitlab.com/libvirt/libvirt-ci
+# THIS FILE WAS AUTO-GENERATED BY tests/lcitool/refresh
FROM docker.io/library/debian:11-slim
diff --git a/tests/docker/dockerfiles/debian-ppc64el-cross.docker b/tests/docker/dockerfiles/debian-ppc64el-cross.docker
index d2954e61f678..5591062a8bf1 100644
--- a/tests/docker/dockerfiles/debian-ppc64el-cross.docker
+++ b/tests/docker/dockerfiles/debian-ppc64el-cross.docker
@@ -1,8 +1,4 @@
-# THIS FILE WAS AUTO-GENERATED
-#
-# $ lcitool dockerfile --layers all --cross ppc64le debian-11 qemu
-#
-# https://gitlab.com/libvirt/libvirt-ci
+# THIS FILE WAS AUTO-GENERATED BY tests/lcitool/refresh
FROM docker.io/library/debian:11-slim
diff --git a/tests/docker/dockerfiles/debian-s390x-cross.docker b/tests/docker/dockerfiles/debian-s390x-cross.docker
index d43ce163172e..9c14f3faca40 100644
--- a/tests/docker/dockerfiles/debian-s390x-cross.docker
+++ b/tests/docker/dockerfiles/debian-s390x-cross.docker
@@ -1,8 +1,4 @@
-# THIS FILE WAS AUTO-GENERATED
-#
-# $ lcitool dockerfile --layers all --cross s390x debian-11 qemu
-#
-# https://gitlab.com/libvirt/libvirt-ci
+# THIS FILE WAS AUTO-GENERATED BY tests/lcitool/refresh
FROM docker.io/library/debian:11-slim
diff --git a/tests/docker/dockerfiles/fedora-win32-cross.docker b/tests/docker/dockerfiles/fedora-win32-cross.docker
index aece4f55818d..d691e54d0c15 100644
--- a/tests/docker/dockerfiles/fedora-win32-cross.docker
+++ b/tests/docker/dockerfiles/fedora-win32-cross.docker
@@ -1,8 +1,4 @@
-# THIS FILE WAS AUTO-GENERATED
-#
-# $ lcitool dockerfile --layers all --cross mingw32 fedora-36 qemu
-#
-# https://gitlab.com/libvirt/libvirt-ci
+# THIS FILE WAS AUTO-GENERATED BY tests/lcitool/refresh
FROM registry.fedoraproject.org/fedora:36
diff --git a/tests/docker/dockerfiles/fedora-win64-cross.docker b/tests/docker/dockerfiles/fedora-win64-cross.docker
index 2d573e2505f2..9f5082e02b3b 100644
--- a/tests/docker/dockerfiles/fedora-win64-cross.docker
+++ b/tests/docker/dockerfiles/fedora-win64-cross.docker
@@ -1,8 +1,4 @@
-# THIS FILE WAS AUTO-GENERATED
-#
-# $ lcitool dockerfile --layers all --cross mingw64 fedora-36 qemu
-#
-# https://gitlab.com/libvirt/libvirt-ci
+# THIS FILE WAS AUTO-GENERATED BY tests/lcitool/refresh
FROM registry.fedoraproject.org/fedora:36
diff --git a/tests/docker/dockerfiles/fedora.docker b/tests/docker/dockerfiles/fedora.docker
index 901562bb8e5a..c8e99a6b8951 100644
--- a/tests/docker/dockerfiles/fedora.docker
+++ b/tests/docker/dockerfiles/fedora.docker
@@ -1,8 +1,4 @@
-# THIS FILE WAS AUTO-GENERATED
-#
-# $ lcitool dockerfile --layers all fedora-36 qemu
-#
-# https://gitlab.com/libvirt/libvirt-ci
+# THIS FILE WAS AUTO-GENERATED BY tests/lcitool/refresh
FROM registry.fedoraproject.org/fedora:36
diff --git a/tests/docker/dockerfiles/opensuse-leap.docker b/tests/docker/dockerfiles/opensuse-leap.docker
index 4361b01464af..a98209c15803 100644
--- a/tests/docker/dockerfiles/opensuse-leap.docker
+++ b/tests/docker/dockerfiles/opensuse-leap.docker
@@ -1,8 +1,4 @@
-# THIS FILE WAS AUTO-GENERATED
-#
-# $ lcitool dockerfile --layers all opensuse-leap-153 qemu
-#
-# https://gitlab.com/libvirt/libvirt-ci
+# THIS FILE WAS AUTO-GENERATED BY tests/lcitool/refresh
FROM registry.opensuse.org/opensuse/leap:15.3
diff --git a/tests/docker/dockerfiles/ubuntu2004.docker b/tests/docker/dockerfiles/ubuntu2004.docker
index 9417bca2fa23..077460916ade 100644
--- a/tests/docker/dockerfiles/ubuntu2004.docker
+++ b/tests/docker/dockerfiles/ubuntu2004.docker
@@ -1,8 +1,4 @@
-# THIS FILE WAS AUTO-GENERATED
-#
-# $ lcitool dockerfile --layers all ubuntu-2004 qemu
-#
-# https://gitlab.com/libvirt/libvirt-ci
+# THIS FILE WAS AUTO-GENERATED BY tests/lcitool/refresh
FROM docker.io/library/ubuntu:20.04
diff --git a/tests/lcitool/refresh b/tests/lcitool/refresh
index ee55ee40ee5d..31a34dce7a33 100755
--- a/tests/lcitool/refresh
+++ b/tests/lcitool/refresh
@@ -12,61 +12,80 @@
# or (at your option) any later version. See the COPYING file in
# the top-level directory.
-import sys
-import subprocess
-
+from contextlib import contextmanager
from pathlib import Path
+import sys
+
if len(sys.argv) != 1:
print("syntax: %s" % sys.argv[0], file=sys.stderr)
sys.exit(1)
-self_dir = Path(__file__).parent
-src_dir = self_dir.parent.parent
+script = Path(__file__)
+script_dir = script.parent
+src_dir = script_dir.parent.parent
dockerfiles_dir = Path(src_dir, "tests", "docker", "dockerfiles")
-lcitool_path = Path(self_dir, "libvirt-ci", "bin", "lcitool")
+sys.path.append(str(Path(script_dir, "libvirt-ci")))
-lcitool_cmd = [lcitool_path, "--data-dir", self_dir]
+from lcitool import LcitoolError
+from lcitool.packages import Packages
+from lcitool.projects import Projects
+from lcitool.targets import BuildTarget, Targets
+from lcitool.formatters import DockerfileFormatter, ShellVariablesFormatter
+from lcitool.util import DataDir
+
+PREFIX = ''
+
+DATA_DIR = DataDir(script_dir)
+PROJECTS = Projects(DATA_DIR)
+PACKAGES = Packages()
+TARGETS = Targets()
-def atomic_write(filename, content):
+@contextmanager
+def atomic_write(filename):
tmp = filename.with_suffix(filename.suffix + ".tmp")
try:
with tmp.open("w") as fp:
- print(content, file=fp, end="")
+ yield fp
tmp.rename(filename)
- except Exception as ex:
+ except Exception:
tmp.unlink()
raise
-def generate(filename, cmd, trailer):
- print("Generate %s" % filename)
- lcitool = subprocess.run(cmd, capture_output=True)
-
- if lcitool.returncode != 0:
- raise Exception("Failed to generate %s: %s" % (filename, lcitool.stderr))
-
- content = lcitool.stdout.decode("utf8")
- if trailer is not None:
- content += trailer
- atomic_write(filename, content)
+@contextmanager
+def generate(filename):
+ print("Generating %s" % filename)
+ nonlocal PREFIX
+ try:
+ PREFIX = "Failed to generate %s: " % filename
+ with atomic_write(filename) as fp:
+ print('# THIS FILE WAS AUTO-GENERATED BY',
+ script.relative_to(src_dir), file=fp)
+ print(file=fp)
+ yield fp
+ finally:
+ PREFIX = ''
def generate_dockerfile(host, target, cross=None, trailer=None):
filename = Path(src_dir, "tests", "docker", "dockerfiles", host + ".docker")
- cmd = lcitool_cmd + ["dockerfile"]
- if cross is not None:
- cmd.extend(["--cross", cross])
- cmd.extend([target, "qemu"])
- generate(filename, cmd, trailer)
+ with generate(filename) as fp:
+ dockerfile = DockerfileFormatter(PROJECTS)
+ target = BuildTarget(TARGETS, PACKAGES, target, cross)
+ print(dockerfile.format(target, ["qemu"]), file=fp)
+ if trailer is not None:
+ print(trailer, file=fp, end="")
-def generate_cirrus(target, trailer=None):
+def generate_cirrus(target):
filename = Path(src_dir, ".gitlab-ci.d", "cirrus", target + ".vars")
- cmd = lcitool_cmd + ["variables", target, "qemu"]
- generate(filename, cmd, trailer)
+ with generate(filename) as fp:
+ variables = ShellVariablesFormatter(PROJECTS)
+ target = BuildTarget(TARGETS, PACKAGES, target)
+ print(variables.format(target, ["qemu"]), file=fp)
ubuntu2004_tsanhack = [
@@ -98,11 +117,11 @@ def cross_build(prefix, targets):
targets = "ENV DEF_TARGET_LIST %s\n" % (targets)
return "".join([conf, targets])
+
#
# Update all the various build configurations.
# Please keep each group sorted alphabetically for easy reading.
#
-
try:
#
# Standard native builds
@@ -179,6 +198,6 @@ try:
generate_cirrus("macos-12")
sys.exit(0)
-except Exception as ex:
- print(str(ex), file=sys.stderr)
+except LcitoolError as ex:
+ print(PREFIX + ex.module_prefix + " error: " + str(ex), file=sys.stderr)
sys.exit(1)
--
2.38.1
^ permalink raw reply related [flat|nested] 9+ messages in thread
* [RFC PATCH 3/4] lcitool: allow overriding package mappings and target facts
2023-01-17 9:16 [RFC PATCH 0/4] Update CentOS and OpenSUSE CI to Python >=3.7 Paolo Bonzini
2023-01-17 9:16 ` [RFC PATCH 1/4] lcitool: update submodule Paolo Bonzini
2023-01-17 9:16 ` [RFC PATCH 2/4] lcitool: use libvirt-ci as library Paolo Bonzini
@ 2023-01-17 9:16 ` Paolo Bonzini
2023-02-08 17:31 ` Daniel P. Berrangé
2023-01-17 9:16 ` [RFC PATCH 4/4] ci, docker: update CentOS and OpenSUSE Python to non-EOL versions Paolo Bonzini
2023-02-08 17:22 ` [RFC PATCH 0/4] Update CentOS and OpenSUSE CI to Python >=3.7 Paolo Bonzini
4 siblings, 1 reply; 9+ messages in thread
From: Paolo Bonzini @ 2023-01-17 9:16 UTC (permalink / raw)
To: qemu-devel; +Cc: alex.bennee, berrange
lcitool has generally catered to the Libvirt's needs in terms of
package versions, which are pretty conservative. For example, lcitool
is hardcoding a version of Meson equal to 0.56. QEMU on the other hand
has different needs since some features were added to Meson for the
project's benefit in versions as recent as 0.63.
Until now, QEMU has managed to avoid the problem by shipping its own
version of Meson. However, the next release of QEMU will face a breaking
point in that Meson 0.63 does not work with the 3.6 version of Python
that is provided by CentOS Stream 8 and OpenSUSE LEAP 15.3.
Implement locally the functionality provided by
https://gitlab.com/libvirt/libvirt-ci/-/merge_requests/341, so that
QEMU can override the mappings and target paths that are used for its CI.
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
---
tests/lcitool/refresh | 21 +++++++++++++++++----
1 file changed, 17 insertions(+), 4 deletions(-)
diff --git a/tests/lcitool/refresh b/tests/lcitool/refresh
index 31a34dce7a33..ca36f3536f53 100755
--- a/tests/lcitool/refresh
+++ b/tests/lcitool/refresh
@@ -15,7 +15,9 @@
from contextlib import contextmanager
from pathlib import Path
+import logging
import sys
+import yaml
if len(sys.argv) != 1:
print("syntax: %s" % sys.argv[0], file=sys.stderr)
@@ -33,14 +35,25 @@ from lcitool.packages import Packages
from lcitool.projects import Projects
from lcitool.targets import BuildTarget, Targets
from lcitool.formatters import DockerfileFormatter, ShellVariablesFormatter
-from lcitool.util import DataDir
+from lcitool.util import DataDir, merge_dict
+
+
+class MergeLocalFilesDataDir(DataDir):
+ def load_facts(self, resource_path, name):
+ result = {}
+ for file in self.search(resource_path, name + ".yml"):
+ logging.getLogger().debug(f"Loading facts from '{file}'")
+ with open(file, "r") as infile:
+ merge_dict(yaml.safe_load(infile), result)
+ return result
+
PREFIX = ''
-DATA_DIR = DataDir(script_dir)
+DATA_DIR = MergeLocalFilesDataDir(script_dir)
PROJECTS = Projects(DATA_DIR)
-PACKAGES = Packages()
-TARGETS = Targets()
+PACKAGES = Packages(DATA_DIR)
+TARGETS = Targets(DATA_DIR)
@contextmanager
--
2.38.1
^ permalink raw reply related [flat|nested] 9+ messages in thread
* [RFC PATCH 4/4] ci, docker: update CentOS and OpenSUSE Python to non-EOL versions
2023-01-17 9:16 [RFC PATCH 0/4] Update CentOS and OpenSUSE CI to Python >=3.7 Paolo Bonzini
` (2 preceding siblings ...)
2023-01-17 9:16 ` [RFC PATCH 3/4] lcitool: allow overriding package mappings and target facts Paolo Bonzini
@ 2023-01-17 9:16 ` Paolo Bonzini
2023-02-08 17:45 ` Daniel P. Berrangé
2023-02-08 17:22 ` [RFC PATCH 0/4] Update CentOS and OpenSUSE CI to Python >=3.7 Paolo Bonzini
4 siblings, 1 reply; 9+ messages in thread
From: Paolo Bonzini @ 2023-01-17 9:16 UTC (permalink / raw)
To: qemu-devel; +Cc: alex.bennee, berrange
Python 3.6 is at end-of-life. Update the libvirt-ci module to a
version that supports overrides for targets and package mappings;
this way, QEMU can use the newer versions provided by CentOS 8 (Python
3.8) and OpenSUSE 15.3 (Python 3.9).
One unexpected issue is that sphinx-rtd-theme does not want versions
of docutils starting with 0.18. For distro-installed packages we can
assume that the packager knew this, but when installing from PyPI the
package version needs to be constrained by hand. In order to do this,
add the docutils package explicitly to the qemu project file.
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
---
.gitlab-ci.d/cirrus/freebsd-12.vars | 2 +-
.gitlab-ci.d/cirrus/freebsd-13.vars | 2 +-
.gitlab-ci.d/cirrus/macos-12.vars | 2 +-
scripts/ci/setup/build-environment.yml | 2 +-
tests/docker/dockerfiles/alpine.docker | 1 +
tests/docker/dockerfiles/centos8.docker | 23 +++---
.../dockerfiles/debian-amd64-cross.docker | 1 +
tests/docker/dockerfiles/debian-amd64.docker | 1 +
.../dockerfiles/debian-arm64-cross.docker | 1 +
.../dockerfiles/debian-armel-cross.docker | 1 +
.../dockerfiles/debian-armhf-cross.docker | 1 +
.../dockerfiles/debian-mips64el-cross.docker | 1 +
.../dockerfiles/debian-mipsel-cross.docker | 1 +
.../dockerfiles/debian-ppc64el-cross.docker | 1 +
.../dockerfiles/debian-s390x-cross.docker | 1 +
.../dockerfiles/fedora-win32-cross.docker | 1 +
.../dockerfiles/fedora-win64-cross.docker | 1 +
tests/docker/dockerfiles/fedora.docker | 1 +
tests/docker/dockerfiles/opensuse-leap.docker | 23 +++---
tests/docker/dockerfiles/ubuntu2004.docker | 3 +-
tests/lcitool/mappings.yml | 80 +++++++++++++++++++
tests/lcitool/projects/qemu.yml | 1 +
tests/lcitool/targets/centos-stream-8.yml | 3 +
tests/lcitool/targets/opensuse-leap-153.yml | 3 +
24 files changed, 131 insertions(+), 26 deletions(-)
create mode 100644 tests/lcitool/mappings.yml
create mode 100644 tests/lcitool/targets/centos-stream-8.yml
create mode 100644 tests/lcitool/targets/opensuse-leap-153.yml
diff --git a/.gitlab-ci.d/cirrus/freebsd-12.vars b/.gitlab-ci.d/cirrus/freebsd-12.vars
index f395e56fc833..4f73f5097050 100644
--- a/.gitlab-ci.d/cirrus/freebsd-12.vars
+++ b/.gitlab-ci.d/cirrus/freebsd-12.vars
@@ -7,6 +7,6 @@ MAKE='/usr/local/bin/gmake'
NINJA='/usr/local/bin/ninja'
PACKAGING_COMMAND='pkg'
PIP3='/usr/local/bin/pip-3.8'
-PKGS='alsa-lib bash bison bzip2 ca_root_nss capstone4 ccache cdrkit-genisoimage cmocka ctags curl cyrus-sasl dbus diffutils dtc flex fusefs-libs3 gettext git glib gmake gnutls gsed gtk3 json-c libepoxy libffi libgcrypt libjpeg-turbo libnfs libslirp libspice-server libssh libtasn1 llvm lzo2 meson ncurses nettle ninja opencv perl5 pixman pkgconf png py39-numpy py39-pillow py39-pip py39-sphinx py39-sphinx_rtd_theme py39-yaml python3 rpm2cpio sdl2 sdl2_image snappy sndio spice-protocol tesseract texinfo usbredir virglrenderer vte3 zstd'
+PKGS='alsa-lib bash bison bzip2 ca_root_nss capstone4 ccache cdrkit-genisoimage cmocka ctags curl cyrus-sasl dbus diffutils dtc flex fusefs-libs3 gettext git glib gmake gnutls gsed gtk3 json-c libepoxy libffi libgcrypt libjpeg-turbo libnfs libslirp libspice-server libssh libtasn1 llvm lzo2 meson ncurses nettle ninja opencv perl5 pixman pkgconf png py39-docutils py39-numpy py39-pillow py39-pip py39-sphinx py39-sphinx_rtd_theme py39-yaml python3 rpm2cpio sdl2 sdl2_image snappy sndio spice-protocol tesseract texinfo usbredir virglrenderer vte3 zstd'
PYPI_PKGS=''
PYTHON='/usr/local/bin/python3'
diff --git a/.gitlab-ci.d/cirrus/freebsd-13.vars b/.gitlab-ci.d/cirrus/freebsd-13.vars
index f395e56fc833..4f73f5097050 100644
--- a/.gitlab-ci.d/cirrus/freebsd-13.vars
+++ b/.gitlab-ci.d/cirrus/freebsd-13.vars
@@ -7,6 +7,6 @@ MAKE='/usr/local/bin/gmake'
NINJA='/usr/local/bin/ninja'
PACKAGING_COMMAND='pkg'
PIP3='/usr/local/bin/pip-3.8'
-PKGS='alsa-lib bash bison bzip2 ca_root_nss capstone4 ccache cdrkit-genisoimage cmocka ctags curl cyrus-sasl dbus diffutils dtc flex fusefs-libs3 gettext git glib gmake gnutls gsed gtk3 json-c libepoxy libffi libgcrypt libjpeg-turbo libnfs libslirp libspice-server libssh libtasn1 llvm lzo2 meson ncurses nettle ninja opencv perl5 pixman pkgconf png py39-numpy py39-pillow py39-pip py39-sphinx py39-sphinx_rtd_theme py39-yaml python3 rpm2cpio sdl2 sdl2_image snappy sndio spice-protocol tesseract texinfo usbredir virglrenderer vte3 zstd'
+PKGS='alsa-lib bash bison bzip2 ca_root_nss capstone4 ccache cdrkit-genisoimage cmocka ctags curl cyrus-sasl dbus diffutils dtc flex fusefs-libs3 gettext git glib gmake gnutls gsed gtk3 json-c libepoxy libffi libgcrypt libjpeg-turbo libnfs libslirp libspice-server libssh libtasn1 llvm lzo2 meson ncurses nettle ninja opencv perl5 pixman pkgconf png py39-docutils py39-numpy py39-pillow py39-pip py39-sphinx py39-sphinx_rtd_theme py39-yaml python3 rpm2cpio sdl2 sdl2_image snappy sndio spice-protocol tesseract texinfo usbredir virglrenderer vte3 zstd'
PYPI_PKGS=''
PYTHON='/usr/local/bin/python3'
diff --git a/.gitlab-ci.d/cirrus/macos-12.vars b/.gitlab-ci.d/cirrus/macos-12.vars
index 7fb182f93e16..4ff6919a2140 100644
--- a/.gitlab-ci.d/cirrus/macos-12.vars
+++ b/.gitlab-ci.d/cirrus/macos-12.vars
@@ -7,6 +7,6 @@ MAKE='/opt/homebrew/bin/gmake'
NINJA='/opt/homebrew/bin/ninja'
PACKAGING_COMMAND='brew'
PIP3='/opt/homebrew/bin/pip3'
-PKGS='bash bc bison bzip2 capstone ccache cmocka ctags curl dbus diffutils dtc flex gcovr gettext git glib gnu-sed gnutls gtk+3 jemalloc jpeg-turbo json-c libepoxy libffi libgcrypt libiscsi libnfs libpng libslirp libssh libtasn1 libusb llvm lzo make meson ncurses nettle ninja perl pixman pkg-config python3 rpm2cpio sdl2 sdl2_image snappy sparse spice-protocol tesseract texinfo usbredir vde vte3 zlib zstd'
+PKGS='bash bc bison bzip2 capstone ccache cmocka ctags curl dbus diffutils docutils dtc flex gcovr gettext git glib gnu-sed gnutls gtk+3 jemalloc jpeg-turbo json-c libepoxy libffi libgcrypt libiscsi libnfs libpng libslirp libssh libtasn1 libusb llvm lzo make meson ncurses nettle ninja perl pixman pkg-config python3 rpm2cpio sdl2 sdl2_image snappy sparse spice-protocol tesseract texinfo usbredir vde vte3 zlib zstd'
PYPI_PKGS='PyYAML numpy pillow sphinx sphinx-rtd-theme'
PYTHON='/opt/homebrew/bin/python3'
diff --git a/scripts/ci/setup/build-environment.yml b/scripts/ci/setup/build-environment.yml
index b04c2b7ceee9..fee11d1d0543 100644
--- a/scripts/ci/setup/build-environment.yml
+++ b/scripts/ci/setup/build-environment.yml
@@ -157,7 +157,7 @@
- nmap-ncat
- perl-Test-Harness
- pixman-devel
- - python36
+ - '@python:3.8/python38'
- rdma-core-devel
- spice-glib-devel
- systemtap-sdt-devel
diff --git a/tests/docker/dockerfiles/alpine.docker b/tests/docker/dockerfiles/alpine.docker
index 10f68f4e1e6d..84ead77a76ee 100644
--- a/tests/docker/dockerfiles/alpine.docker
+++ b/tests/docker/dockerfiles/alpine.docker
@@ -77,6 +77,7 @@ RUN apk update && \
pixman-dev \
pkgconf \
pulseaudio-dev \
+ py3-docutils \
py3-numpy \
py3-pillow \
py3-pip \
diff --git a/tests/docker/dockerfiles/centos8.docker b/tests/docker/dockerfiles/centos8.docker
index 2318a95e9255..adc7fd483b29 100644
--- a/tests/docker/dockerfiles/centos8.docker
+++ b/tests/docker/dockerfiles/centos8.docker
@@ -78,7 +78,6 @@ RUN dnf distro-sync -y && \
lzo-devel \
make \
mesa-libgbm-devel \
- meson \
ncurses-devel \
nettle-devel \
ninja-build \
@@ -91,13 +90,12 @@ RUN dnf distro-sync -y && \
pixman-devel \
pkgconfig \
pulseaudio-libs-devel \
- python3 \
- python3-PyYAML \
- python3-numpy \
- python3-pillow \
- python3-pip \
- python3-sphinx \
- python3-sphinx_rtd_theme \
+ python38 \
+ python38-PyYAML \
+ python38-numpy \
+ python38-pip \
+ python38-setuptools \
+ python38-wheel \
rdma-core-devel \
rpm \
sed \
@@ -126,8 +124,15 @@ RUN dnf distro-sync -y && \
ln -s /usr/bin/ccache /usr/libexec/ccache-wrappers/g++ && \
ln -s /usr/bin/ccache /usr/libexec/ccache-wrappers/gcc
+RUN /usr/bin/pip3.8 install \
+ 'docutils<0.18' \
+ meson==0.63.2 \
+ pillow \
+ sphinx \
+ sphinx-rtd-theme
+
ENV CCACHE_WRAPPERSDIR "/usr/libexec/ccache-wrappers"
ENV LANG "en_US.UTF-8"
ENV MAKE "/usr/bin/make"
ENV NINJA "/usr/bin/ninja"
-ENV PYTHON "/usr/bin/python3"
+ENV PYTHON "/usr/bin/python3.8"
diff --git a/tests/docker/dockerfiles/debian-amd64-cross.docker b/tests/docker/dockerfiles/debian-amd64-cross.docker
index bfa88dc7285f..a91f5412ab49 100644
--- a/tests/docker/dockerfiles/debian-amd64-cross.docker
+++ b/tests/docker/dockerfiles/debian-amd64-cross.docker
@@ -39,6 +39,7 @@ RUN export DEBIAN_FRONTEND=noninteractive && \
perl-base \
pkgconf \
python3 \
+ python3-docutils \
python3-numpy \
python3-opencv \
python3-pillow \
diff --git a/tests/docker/dockerfiles/debian-amd64.docker b/tests/docker/dockerfiles/debian-amd64.docker
index e09118c83b2f..14daef8bdaee 100644
--- a/tests/docker/dockerfiles/debian-amd64.docker
+++ b/tests/docker/dockerfiles/debian-amd64.docker
@@ -107,6 +107,7 @@ RUN export DEBIAN_FRONTEND=noninteractive && \
perl-base \
pkgconf \
python3 \
+ python3-docutils \
python3-numpy \
python3-opencv \
python3-pillow \
diff --git a/tests/docker/dockerfiles/debian-arm64-cross.docker b/tests/docker/dockerfiles/debian-arm64-cross.docker
index b20197d5b710..d89b7ca4e568 100644
--- a/tests/docker/dockerfiles/debian-arm64-cross.docker
+++ b/tests/docker/dockerfiles/debian-arm64-cross.docker
@@ -39,6 +39,7 @@ RUN export DEBIAN_FRONTEND=noninteractive && \
perl-base \
pkgconf \
python3 \
+ python3-docutils \
python3-numpy \
python3-opencv \
python3-pillow \
diff --git a/tests/docker/dockerfiles/debian-armel-cross.docker b/tests/docker/dockerfiles/debian-armel-cross.docker
index 792e098f273c..c3eee750db93 100644
--- a/tests/docker/dockerfiles/debian-armel-cross.docker
+++ b/tests/docker/dockerfiles/debian-armel-cross.docker
@@ -39,6 +39,7 @@ RUN export DEBIAN_FRONTEND=noninteractive && \
perl-base \
pkgconf \
python3 \
+ python3-docutils \
python3-numpy \
python3-opencv \
python3-pillow \
diff --git a/tests/docker/dockerfiles/debian-armhf-cross.docker b/tests/docker/dockerfiles/debian-armhf-cross.docker
index dbb9cf21e99b..d88123880680 100644
--- a/tests/docker/dockerfiles/debian-armhf-cross.docker
+++ b/tests/docker/dockerfiles/debian-armhf-cross.docker
@@ -39,6 +39,7 @@ RUN export DEBIAN_FRONTEND=noninteractive && \
perl-base \
pkgconf \
python3 \
+ python3-docutils \
python3-numpy \
python3-opencv \
python3-pillow \
diff --git a/tests/docker/dockerfiles/debian-mips64el-cross.docker b/tests/docker/dockerfiles/debian-mips64el-cross.docker
index 712cf98388a1..33a1f680b663 100644
--- a/tests/docker/dockerfiles/debian-mips64el-cross.docker
+++ b/tests/docker/dockerfiles/debian-mips64el-cross.docker
@@ -39,6 +39,7 @@ RUN export DEBIAN_FRONTEND=noninteractive && \
perl-base \
pkgconf \
python3 \
+ python3-docutils \
python3-numpy \
python3-opencv \
python3-pillow \
diff --git a/tests/docker/dockerfiles/debian-mipsel-cross.docker b/tests/docker/dockerfiles/debian-mipsel-cross.docker
index c746b7ca04aa..e8b1ea928ff7 100644
--- a/tests/docker/dockerfiles/debian-mipsel-cross.docker
+++ b/tests/docker/dockerfiles/debian-mipsel-cross.docker
@@ -39,6 +39,7 @@ RUN export DEBIAN_FRONTEND=noninteractive && \
perl-base \
pkgconf \
python3 \
+ python3-docutils \
python3-numpy \
python3-opencv \
python3-pillow \
diff --git a/tests/docker/dockerfiles/debian-ppc64el-cross.docker b/tests/docker/dockerfiles/debian-ppc64el-cross.docker
index 5591062a8bf1..3e7317250d59 100644
--- a/tests/docker/dockerfiles/debian-ppc64el-cross.docker
+++ b/tests/docker/dockerfiles/debian-ppc64el-cross.docker
@@ -39,6 +39,7 @@ RUN export DEBIAN_FRONTEND=noninteractive && \
perl-base \
pkgconf \
python3 \
+ python3-docutils \
python3-numpy \
python3-opencv \
python3-pillow \
diff --git a/tests/docker/dockerfiles/debian-s390x-cross.docker b/tests/docker/dockerfiles/debian-s390x-cross.docker
index 9c14f3faca40..276f9422f833 100644
--- a/tests/docker/dockerfiles/debian-s390x-cross.docker
+++ b/tests/docker/dockerfiles/debian-s390x-cross.docker
@@ -39,6 +39,7 @@ RUN export DEBIAN_FRONTEND=noninteractive && \
perl-base \
pkgconf \
python3 \
+ python3-docutils \
python3-numpy \
python3-opencv \
python3-pillow \
diff --git a/tests/docker/dockerfiles/fedora-win32-cross.docker b/tests/docker/dockerfiles/fedora-win32-cross.docker
index d691e54d0c15..0938f1901775 100644
--- a/tests/docker/dockerfiles/fedora-win32-cross.docker
+++ b/tests/docker/dockerfiles/fedora-win32-cross.docker
@@ -41,6 +41,7 @@ exec "$@"' > /usr/bin/nosync && \
perl-base \
python3 \
python3-PyYAML \
+ python3-docutils \
python3-numpy \
python3-opencv \
python3-pillow \
diff --git a/tests/docker/dockerfiles/fedora-win64-cross.docker b/tests/docker/dockerfiles/fedora-win64-cross.docker
index 9f5082e02b3b..d36a04d33551 100644
--- a/tests/docker/dockerfiles/fedora-win64-cross.docker
+++ b/tests/docker/dockerfiles/fedora-win64-cross.docker
@@ -41,6 +41,7 @@ exec "$@"' > /usr/bin/nosync && \
perl-base \
python3 \
python3-PyYAML \
+ python3-docutils \
python3-numpy \
python3-opencv \
python3-pillow \
diff --git a/tests/docker/dockerfiles/fedora.docker b/tests/docker/dockerfiles/fedora.docker
index c8e99a6b8951..752f683cae76 100644
--- a/tests/docker/dockerfiles/fedora.docker
+++ b/tests/docker/dockerfiles/fedora.docker
@@ -100,6 +100,7 @@ exec "$@"' > /usr/bin/nosync && \
pulseaudio-libs-devel \
python3 \
python3-PyYAML \
+ python3-docutils \
python3-numpy \
python3-opencv \
python3-pillow \
diff --git a/tests/docker/dockerfiles/opensuse-leap.docker b/tests/docker/dockerfiles/opensuse-leap.docker
index a98209c15803..e51909496d2e 100644
--- a/tests/docker/dockerfiles/opensuse-leap.docker
+++ b/tests/docker/dockerfiles/opensuse-leap.docker
@@ -86,16 +86,9 @@ RUN zypper update -y && \
pcre-devel-static \
perl-base \
pkgconfig \
- python3-Pillow \
- python3-PyYAML \
- python3-Sphinx \
- python3-base \
- python3-numpy \
- python3-opencv \
- python3-pip \
- python3-setuptools \
- python3-sphinx_rtd_theme \
- python3-wheel \
+ python39-base \
+ python39-pip \
+ python39-setuptools \
rdma-core-devel \
rpm \
sed \
@@ -127,10 +120,16 @@ RUN zypper update -y && \
ln -s /usr/bin/ccache /usr/libexec/ccache-wrappers/g++ && \
ln -s /usr/bin/ccache /usr/libexec/ccache-wrappers/gcc
-RUN /usr/bin/pip3 install meson==0.56.0
+RUN /usr/bin/pip3.9 install \
+ PyYAML \
+ 'docutils<0.18' \
+ meson==0.63.2 \
+ pillow \
+ sphinx \
+ sphinx-rtd-theme
ENV CCACHE_WRAPPERSDIR "/usr/libexec/ccache-wrappers"
ENV LANG "en_US.UTF-8"
ENV MAKE "/usr/bin/make"
ENV NINJA "/usr/bin/ninja"
-ENV PYTHON "/usr/bin/python3"
+ENV PYTHON "/usr/bin/python3.9"
diff --git a/tests/docker/dockerfiles/ubuntu2004.docker b/tests/docker/dockerfiles/ubuntu2004.docker
index 077460916ade..13b5c4a766a1 100644
--- a/tests/docker/dockerfiles/ubuntu2004.docker
+++ b/tests/docker/dockerfiles/ubuntu2004.docker
@@ -104,6 +104,7 @@ RUN export DEBIAN_FRONTEND=noninteractive && \
perl-base \
pkgconf \
python3 \
+ python3-docutils \
python3-numpy \
python3-opencv \
python3-pillow \
@@ -136,7 +137,7 @@ RUN export DEBIAN_FRONTEND=noninteractive && \
ln -s /usr/bin/ccache /usr/libexec/ccache-wrappers/g++ && \
ln -s /usr/bin/ccache /usr/libexec/ccache-wrappers/gcc
-RUN /usr/bin/pip3 install meson==0.56.0
+RUN /usr/bin/pip3 install meson==0.63.2
ENV CCACHE_WRAPPERSDIR "/usr/libexec/ccache-wrappers"
ENV LANG "en_US.UTF-8"
diff --git a/tests/lcitool/mappings.yml b/tests/lcitool/mappings.yml
new file mode 100644
index 000000000000..4eef6f425896
--- /dev/null
+++ b/tests/lcitool/mappings.yml
@@ -0,0 +1,80 @@
+mappings:
+ flake8:
+ CentOSStream8:
+ OpenSUSELeap153:
+
+ meson:
+ CentOSStream8:
+ OpenSUSELeap153:
+
+ python3:
+ CentOSStream8: python38
+ OpenSUSELeap153: python39-base
+
+ python3-PyYAML:
+ CentOSStream8: python38-PyYAML
+ OpenSUSELeap153:
+
+ python3-devel:
+ CentOSStream8: python38-devel
+ OpenSUSELeap153: python39-devel
+
+ python3-docutils:
+ CentOSStream8:
+ OpenSUSELeap153:
+
+ python3-numpy:
+ CentOSStream8: python38-numpy
+ OpenSUSELeap153:
+
+ python3-opencv:
+ CentOSStream8:
+ OpenSUSELeap153:
+
+ python3-pillow:
+ CentOSStream8:
+ OpenSUSELeap153:
+
+ python3-pip:
+ CentOSStream8: python38-pip
+ OpenSUSELeap153: python39-pip
+
+ python3-pillow:
+ CentOSStream8:
+ OpenSUSELeap153:
+
+ python3-selinux:
+ CentOSStream8:
+ OpenSUSELeap153:
+
+ python3-setuptools:
+ CentOSStream8: python38-setuptools
+ OpenSUSELeap153: python39-setuptools
+
+ python3-sphinx:
+ CentOSStream8:
+ OpenSUSELeap153:
+
+ python3-sphinx-rtd-theme:
+ CentOSStream8:
+ OpenSUSELeap153:
+
+ python3-venv:
+ CentOSStream8: python38
+ OpenSUSELeap153: python39-base
+
+ python3-wheel:
+ CentOSStream8: python38-wheel
+ OpenSUSELeap153: python39-pip
+
+pypi_mappings:
+ # Request last version
+ meson:
+ default: meson==0.63.2
+
+ python3-docutils:
+ default: docutils<0.18
+
+ # Drop packages that need devel headers
+ python3-numpy:
+ OpenSUSELeap153:
diff --git a/tests/lcitool/projects/qemu.yml b/tests/lcitool/projects/qemu.yml
index c62dbc00f96b..e45e924639ec 100644
--- a/tests/lcitool/projects/qemu.yml
+++ b/tests/lcitool/projects/qemu.yml
@@ -89,6 +89,7 @@ packages:
- pulseaudio
- python3
- python3-PyYAML
+ - python3-docutils
- python3-numpy
- python3-opencv
- python3-pillow
diff --git a/tests/lcitool/targets/centos-stream-8.yml b/tests/lcitool/targets/centos-stream-8.yml
new file mode 100644
index 000000000000..6b11160fd1dc
--- /dev/null
+++ b/tests/lcitool/targets/centos-stream-8.yml
@@ -0,0 +1,3 @@
+paths:
+ pip3: /usr/bin/pip3.8
+ python: /usr/bin/python3.8
diff --git a/tests/lcitool/targets/opensuse-leap-153.yml b/tests/lcitool/targets/opensuse-leap-153.yml
new file mode 100644
index 000000000000..683016e0077a
--- /dev/null
+++ b/tests/lcitool/targets/opensuse-leap-153.yml
@@ -0,0 +1,3 @@
+paths:
+ pip3: /usr/bin/pip3.9
+ python: /usr/bin/python3.9
--
2.38.1
^ permalink raw reply related [flat|nested] 9+ messages in thread
* Re: [RFC PATCH 0/4] Update CentOS and OpenSUSE CI to Python >=3.7
2023-01-17 9:16 [RFC PATCH 0/4] Update CentOS and OpenSUSE CI to Python >=3.7 Paolo Bonzini
` (3 preceding siblings ...)
2023-01-17 9:16 ` [RFC PATCH 4/4] ci, docker: update CentOS and OpenSUSE Python to non-EOL versions Paolo Bonzini
@ 2023-02-08 17:22 ` Paolo Bonzini
4 siblings, 0 replies; 9+ messages in thread
From: Paolo Bonzini @ 2023-02-08 17:22 UTC (permalink / raw)
To: qemu-devel; +Cc: alex.bennee, berrange
Ping.
Paolo
On Tue, Jan 17, 2023 at 10:16 AM Paolo Bonzini <pbonzini@redhat.com> wrote:
>
> QEMU build and test scripts would like to use some new features that
> were added to Python 3.7. Useful additions include removing the need
> for OrderedDict, improved handling of default text encoding, postponed
> evaluation of type annotations, and improvements to asyncio.
>
> Using new features that were added to Meson 0.63 is also being blocked
> by usage of EOL'd versions of Python; Meson 0.63 does not work with the
> 3.6 version of Python that is provided by CentOS Stream 8 and OpenSUSE
> LEAP 15. Therefore, this series upgrades the CentOS Stream 8 and OpenSUSE
> LEAP 15 dockerfiles to use the alternative Python runtimes provided by
> the distros.
>
> The main obstacle here is updating the lcitool-generated Dockerfiles;
> lcitool is (by design) quite opinionated on the mapping between
> "generic" package names and target-specific package names, and it
> caters to Libvirt's needs in terms of package versions.
>
> There are two possibilities to change this.
>
> First, lately I have been working on allowing lcitool to be used as
> a library. As of this morning, it is possible for QEMU to use
> lcitool's databases and formatters directly from Python, and only
> replace the search mechanism to allow mappings and target paths to
> be overridden. This is the approach that is implemented in this RFC:
> the first two patches in this series use libvirt-ci as library, while
> the third implements the custom search path.
>
> However, I have also proposed the custom search mechanism directly
> upstream (https://gitlab.com/libvirt/libvirt-ci/-/merge_requests/341).
> If this is accepted, patch 3 will become much smaller and will not
> need to know any lcitool internals; in addition, using lcitool as
> a library could be evaluated on its own merits because command-line
> operation would be a possibility as well.
>
> With the lcitool changes in place, the final patch shows how to
> install custom package mappings and update CentOS and OpenSUSE Python
> respectively to 3.8 and 3.9.
>
> Thanks,
>
> Paolo
>
> Paolo Bonzini (4):
> lcitool: update submodule
> lcitool: use libvirt-ci as library
> lcitool: allow overriding package mappings and target facts
> ci, docker: update CentOS and OpenSUSE Python to non-EOL versions
>
> .gitlab-ci.d/cirrus/freebsd-12.vars | 8 +-
> .gitlab-ci.d/cirrus/freebsd-13.vars | 8 +-
> .gitlab-ci.d/cirrus/macos-12.vars | 8 +-
> scripts/ci/setup/build-environment.yml | 2 +-
> tests/docker/dockerfiles/alpine.docker | 7 +-
> tests/docker/dockerfiles/centos8.docker | 29 +++---
> .../dockerfiles/debian-amd64-cross.docker | 7 +-
> tests/docker/dockerfiles/debian-amd64.docker | 7 +-
> .../dockerfiles/debian-arm64-cross.docker | 7 +-
> .../dockerfiles/debian-armel-cross.docker | 7 +-
> .../dockerfiles/debian-armhf-cross.docker | 7 +-
> .../dockerfiles/debian-mips64el-cross.docker | 7 +-
> .../dockerfiles/debian-mipsel-cross.docker | 7 +-
> .../dockerfiles/debian-ppc64el-cross.docker | 7 +-
> .../dockerfiles/debian-s390x-cross.docker | 7 +-
> .../dockerfiles/fedora-win32-cross.docker | 7 +-
> .../dockerfiles/fedora-win64-cross.docker | 7 +-
> tests/docker/dockerfiles/fedora.docker | 7 +-
> tests/docker/dockerfiles/opensuse-leap.docker | 29 +++---
> tests/docker/dockerfiles/ubuntu2004.docker | 9 +-
> tests/lcitool/libvirt-ci | 2 +-
> tests/lcitool/mappings.yml | 80 ++++++++++++++++
> tests/lcitool/projects/qemu.yml | 1 +
> tests/lcitool/refresh | 96 ++++++++++++-------
> tests/lcitool/targets/centos-stream-8.yml | 3 +
> tests/lcitool/targets/opensuse-leap-153.yml | 3 +
> 26 files changed, 215 insertions(+), 154 deletions(-)
> create mode 100644 tests/lcitool/mappings.yml
> create mode 100644 tests/lcitool/targets/centos-stream-8.yml
> create mode 100644 tests/lcitool/targets/opensuse-leap-153.yml
>
> --
> 2.38.1
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [RFC PATCH 2/4] lcitool: use libvirt-ci as library
2023-01-17 9:16 ` [RFC PATCH 2/4] lcitool: use libvirt-ci as library Paolo Bonzini
@ 2023-02-08 17:27 ` Daniel P. Berrangé
0 siblings, 0 replies; 9+ messages in thread
From: Daniel P. Berrangé @ 2023-02-08 17:27 UTC (permalink / raw)
To: Paolo Bonzini; +Cc: qemu-devel, alex.bennee
On Tue, Jan 17, 2023 at 10:16:36AM +0100, Paolo Bonzini wrote:
> Using the lcitool package as a library will make it possible to
> customize more of the process, for example by introducing custom
> mappings.
I wouldn't consider the lcitool python code to be a public
library API, which is for example, why its not a published
pakage on pypi. So from that POV I'm really not a fan of
this change to use the internal APIs directly, as it means
that any contributors who want to update the submodule are
liable to get hit by incompatible API changes.
> diff --git a/tests/lcitool/refresh b/tests/lcitool/refresh
> index ee55ee40ee5d..31a34dce7a33 100755
> --- a/tests/lcitool/refresh
> +++ b/tests/lcitool/refresh
> @@ -12,61 +12,80 @@
> # or (at your option) any later version. See the COPYING file in
> # the top-level directory.
>
> -import sys
> -import subprocess
> -
> +from contextlib import contextmanager
> from pathlib import Path
>
> +import sys
> +
> if len(sys.argv) != 1:
> print("syntax: %s" % sys.argv[0], file=sys.stderr)
> sys.exit(1)
>
> -self_dir = Path(__file__).parent
> -src_dir = self_dir.parent.parent
> +script = Path(__file__)
> +script_dir = script.parent
> +src_dir = script_dir.parent.parent
> dockerfiles_dir = Path(src_dir, "tests", "docker", "dockerfiles")
>
> -lcitool_path = Path(self_dir, "libvirt-ci", "bin", "lcitool")
> +sys.path.append(str(Path(script_dir, "libvirt-ci")))
>
> -lcitool_cmd = [lcitool_path, "--data-dir", self_dir]
> +from lcitool import LcitoolError
> +from lcitool.packages import Packages
> +from lcitool.projects import Projects
> +from lcitool.targets import BuildTarget, Targets
> +from lcitool.formatters import DockerfileFormatter, ShellVariablesFormatter
> +from lcitool.util import DataDir
> +
> +PREFIX = ''
> +
> +DATA_DIR = DataDir(script_dir)
> +PROJECTS = Projects(DATA_DIR)
> +PACKAGES = Packages()
> +TARGETS = Targets()
>
>
> -def atomic_write(filename, content):
> +@contextmanager
> +def atomic_write(filename):
> tmp = filename.with_suffix(filename.suffix + ".tmp")
> try:
> with tmp.open("w") as fp:
> - print(content, file=fp, end="")
> + yield fp
> tmp.rename(filename)
> - except Exception as ex:
> + except Exception:
> tmp.unlink()
> raise
>
>
> -def generate(filename, cmd, trailer):
> - print("Generate %s" % filename)
> - lcitool = subprocess.run(cmd, capture_output=True)
> -
> - if lcitool.returncode != 0:
> - raise Exception("Failed to generate %s: %s" % (filename, lcitool.stderr))
> -
> - content = lcitool.stdout.decode("utf8")
> - if trailer is not None:
> - content += trailer
> - atomic_write(filename, content)
> +@contextmanager
> +def generate(filename):
> + print("Generating %s" % filename)
> + nonlocal PREFIX
> + try:
> + PREFIX = "Failed to generate %s: " % filename
> + with atomic_write(filename) as fp:
> + print('# THIS FILE WAS AUTO-GENERATED BY',
> + script.relative_to(src_dir), file=fp)
> + print(file=fp)
> + yield fp
> + finally:
> + PREFIX = ''
>
>
> def generate_dockerfile(host, target, cross=None, trailer=None):
> filename = Path(src_dir, "tests", "docker", "dockerfiles", host + ".docker")
> - cmd = lcitool_cmd + ["dockerfile"]
> - if cross is not None:
> - cmd.extend(["--cross", cross])
> - cmd.extend([target, "qemu"])
> - generate(filename, cmd, trailer)
> + with generate(filename) as fp:
> + dockerfile = DockerfileFormatter(PROJECTS)
> + target = BuildTarget(TARGETS, PACKAGES, target, cross)
> + print(dockerfile.format(target, ["qemu"]), file=fp)
> + if trailer is not None:
> + print(trailer, file=fp, end="")
>
>
> -def generate_cirrus(target, trailer=None):
> +def generate_cirrus(target):
> filename = Path(src_dir, ".gitlab-ci.d", "cirrus", target + ".vars")
> - cmd = lcitool_cmd + ["variables", target, "qemu"]
> - generate(filename, cmd, trailer)
> + with generate(filename) as fp:
> + variables = ShellVariablesFormatter(PROJECTS)
> + target = BuildTarget(TARGETS, PACKAGES, target)
> + print(variables.format(target, ["qemu"]), file=fp)
>
>
> ubuntu2004_tsanhack = [
> @@ -98,11 +117,11 @@ def cross_build(prefix, targets):
> targets = "ENV DEF_TARGET_LIST %s\n" % (targets)
> return "".join([conf, targets])
>
> +
> #
> # Update all the various build configurations.
> # Please keep each group sorted alphabetically for easy reading.
> #
> -
> try:
> #
> # Standard native builds
> @@ -179,6 +198,6 @@ try:
> generate_cirrus("macos-12")
>
> sys.exit(0)
> -except Exception as ex:
> - print(str(ex), file=sys.stderr)
> +except LcitoolError as ex:
> + print(PREFIX + ex.module_prefix + " error: " + str(ex), file=sys.stderr)
> sys.exit(1)
> --
> 2.38.1
>
With regards,
Daniel
--
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o- https://fstop138.berrange.com :|
|: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [RFC PATCH 3/4] lcitool: allow overriding package mappings and target facts
2023-01-17 9:16 ` [RFC PATCH 3/4] lcitool: allow overriding package mappings and target facts Paolo Bonzini
@ 2023-02-08 17:31 ` Daniel P. Berrangé
0 siblings, 0 replies; 9+ messages in thread
From: Daniel P. Berrangé @ 2023-02-08 17:31 UTC (permalink / raw)
To: Paolo Bonzini; +Cc: qemu-devel, alex.bennee
On Tue, Jan 17, 2023 at 10:16:37AM +0100, Paolo Bonzini wrote:
> lcitool has generally catered to the Libvirt's needs in terms of
> package versions, which are pretty conservative. For example, lcitool
> is hardcoding a version of Meson equal to 0.56. QEMU on the other hand
> has different needs since some features were added to Meson for the
> project's benefit in versions as recent as 0.63.
>
> Until now, QEMU has managed to avoid the problem by shipping its own
> version of Meson. However, the next release of QEMU will face a breaking
> point in that Meson 0.63 does not work with the 3.6 version of Python
> that is provided by CentOS Stream 8 and OpenSUSE LEAP 15.3.
>
> Implement locally the functionality provided by
> https://gitlab.com/libvirt/libvirt-ci/-/merge_requests/341, so that
> QEMU can override the mappings and target paths that are used for its CI.
I'd like to just push ahead with getting that merged so we
use the lcitool --data-dir arg to do what QEMU needs, and
stay away from the internal python code.
With regards,
Daniel
--
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o- https://fstop138.berrange.com :|
|: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [RFC PATCH 4/4] ci, docker: update CentOS and OpenSUSE Python to non-EOL versions
2023-01-17 9:16 ` [RFC PATCH 4/4] ci, docker: update CentOS and OpenSUSE Python to non-EOL versions Paolo Bonzini
@ 2023-02-08 17:45 ` Daniel P. Berrangé
0 siblings, 0 replies; 9+ messages in thread
From: Daniel P. Berrangé @ 2023-02-08 17:45 UTC (permalink / raw)
To: Paolo Bonzini; +Cc: qemu-devel, alex.bennee
On Tue, Jan 17, 2023 at 10:16:38AM +0100, Paolo Bonzini wrote:
> Python 3.6 is at end-of-life. Update the libvirt-ci module to a
> version that supports overrides for targets and package mappings;
> this way, QEMU can use the newer versions provided by CentOS 8 (Python
> 3.8) and OpenSUSE 15.3 (Python 3.9).
>
> One unexpected issue is that sphinx-rtd-theme does not want versions
> of docutils starting with 0.18. For distro-installed packages we can
> assume that the packager knew this, but when installing from PyPI the
> package version needs to be constrained by hand. In order to do this,
> add the docutils package explicitly to the qemu project file.
>
> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
> ---
> .gitlab-ci.d/cirrus/freebsd-12.vars | 2 +-
> .gitlab-ci.d/cirrus/freebsd-13.vars | 2 +-
> .gitlab-ci.d/cirrus/macos-12.vars | 2 +-
> scripts/ci/setup/build-environment.yml | 2 +-
> tests/docker/dockerfiles/alpine.docker | 1 +
> tests/docker/dockerfiles/centos8.docker | 23 +++---
> .../dockerfiles/debian-amd64-cross.docker | 1 +
> tests/docker/dockerfiles/debian-amd64.docker | 1 +
> .../dockerfiles/debian-arm64-cross.docker | 1 +
> .../dockerfiles/debian-armel-cross.docker | 1 +
> .../dockerfiles/debian-armhf-cross.docker | 1 +
> .../dockerfiles/debian-mips64el-cross.docker | 1 +
> .../dockerfiles/debian-mipsel-cross.docker | 1 +
> .../dockerfiles/debian-ppc64el-cross.docker | 1 +
> .../dockerfiles/debian-s390x-cross.docker | 1 +
> .../dockerfiles/fedora-win32-cross.docker | 1 +
> .../dockerfiles/fedora-win64-cross.docker | 1 +
> tests/docker/dockerfiles/fedora.docker | 1 +
> tests/docker/dockerfiles/opensuse-leap.docker | 23 +++---
> tests/docker/dockerfiles/ubuntu2004.docker | 3 +-
> tests/lcitool/mappings.yml | 80 +++++++++++++++++++
> tests/lcitool/projects/qemu.yml | 1 +
> tests/lcitool/targets/centos-stream-8.yml | 3 +
> tests/lcitool/targets/opensuse-leap-153.yml | 3 +
> 24 files changed, 131 insertions(+), 26 deletions(-)
> create mode 100644 tests/lcitool/mappings.yml
> create mode 100644 tests/lcitool/targets/centos-stream-8.yml
> create mode 100644 tests/lcitool/targets/opensuse-leap-153.yml
Could you also update docs/devel/testing.rst where it talks
about lcitool, to give guidance about the local mappings/targets
overrides, vs the upstream mappings/targets usage. Most of the
time for new deps, we should continue to change the upstream
one, only using the local override where QEMU needs to diverge,
so that we maximise sharing.
The actual tests/ dir changes look ok to me
With regards,
Daniel
--
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o- https://fstop138.berrange.com :|
|: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
^ permalink raw reply [flat|nested] 9+ messages in thread
end of thread, other threads:[~2023-02-08 17:46 UTC | newest]
Thread overview: 9+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2023-01-17 9:16 [RFC PATCH 0/4] Update CentOS and OpenSUSE CI to Python >=3.7 Paolo Bonzini
2023-01-17 9:16 ` [RFC PATCH 1/4] lcitool: update submodule Paolo Bonzini
2023-01-17 9:16 ` [RFC PATCH 2/4] lcitool: use libvirt-ci as library Paolo Bonzini
2023-02-08 17:27 ` Daniel P. Berrangé
2023-01-17 9:16 ` [RFC PATCH 3/4] lcitool: allow overriding package mappings and target facts Paolo Bonzini
2023-02-08 17:31 ` Daniel P. Berrangé
2023-01-17 9:16 ` [RFC PATCH 4/4] ci, docker: update CentOS and OpenSUSE Python to non-EOL versions Paolo Bonzini
2023-02-08 17:45 ` Daniel P. Berrangé
2023-02-08 17:22 ` [RFC PATCH 0/4] Update CentOS and OpenSUSE CI to Python >=3.7 Paolo Bonzini
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).