[RFC PATCH 0/9] scripts/ci: Consolidate Ansible playbook rules

[RFC PATCH 0/9] scripts/ci: Consolidate Ansible playbook rules

[Philippe Mathieu-Daudé]

Hi,

The CI Ansible playbook scripts failed when trying to provision
a openEuler instance. I was going to do a manual installation,
but Alex strongly suggested to keep using Ansible, implicitly
asking to fix the scripts :) This is what this series does.

Note, when libvirt-ci is upgraded, we neglect to update the
playbook packages list. Not sure how to automate that, so I
updated manually.

Philippe Mathieu-Daudé (9):
  scripts/ci: Do not enforce gitlab-runner path
  scripts/ci: Do not restrict spice package to x86/arm hosts
  scripts/ci: Split EL8 specific packages out of Centos8 list
  scripts/ci: Update Centos8 package list
  scripts/ci: Restrict libpmem-devel package to x86 hosts
  scripts/ci: Install libxdp-devel on x86 hosts
  scripts/ci: Install RH packages on RH derivative distros
  scripts/ci: Update Ubuntu packages list
  scripts/ci: Restrict libpmem-dev and libxen-dev packages to x86 hosts

 scripts/ci/setup/build-environment.yml | 123 +++++++++++++++++++++----
 scripts/ci/setup/gitlab-runner.yml     |   6 +-
 2 files changed, 107 insertions(+), 22 deletions(-)

-- 
2.41.0

[PATCH 1/9] scripts/ci: Do not enforce gitlab-runner path

[Philippe Mathieu-Daudé]

Distributions are not forced to put gitlab-runner binary
in /usr/bin (as long as it is in $PATH).

Signed-off-by: Philippe Mathieu-Daudé <philmd@linaro.org>
---
FWIW this file seems to use legacy options:

TASK [Register the gitlab-runner] *******************************************************************************************************************
task path: /srcqemu/scripts/ci/setup/gitlab-runner.yml:78
changed: [oe2203] => {"changed": true, "cmd": ["gitlab-runner", "register", "--non-interactive", "--url", "https://gitlab.com", "--registration-token", "glrt-FOO_BAR", "--executor", "shell", "--tag-list", "loongarch64,openeuler_22.03", "--description", "openEuler 22.03 loongarch64 (Loongson)"], "delta": "0:00:00.955654", "end": "2024-01-04 02:33:23.607766", "msg": "", "rc": 0, "start": "2024-01-04 02:33:22.652112", "stderr": "Runtime platform                                  \u001b[0;m  arch\u001b[0;m=loong64 os\u001b[0;m=linux pid\u001b[0;m=147134 revision\u001b[0;m=436955cb version\u001b[0;m=16.6.1\nRunning in system-mode.                           \u001b[0;m \n                                                  \u001b[0;m \n\u001b[0;33mWARNING: You have specified an authentication token in the legacy parameter --registration-token. This has triggered the 'legacy-compatible registration process' which has resulted in the following command line parameters being ignored: --locked, --access-level, --run-untagged, --maximum-timeout, --paused, --tag-list, and --maintenance-note. For more information, see https://docs.gitlab.com/ee/ci/runners/new_creation_workflow.html#changes-to-the-gitlab-runner-register-command-syntaxThese parameters and the legacy-compatible registration process will be removed in GitLab Runner 18.0. \u001b[0;m \nVerifying runner... is valid                      \u001b[0;m  runner\u001b[0;m=kDNtto8nr\nRunner registered successfully. Feel free to start it, but if it's running already the config should be automatically reloaded!\n\u001b[0;m \nConfiguration (with the authentication token) was saved in \"/etc/gitlab-runner/config.toml\"\u001b[0;m ", "stderr_lines": ["Runtime platform                                  \u001b[0;m  arch\u001b[0;m=loong64 os\u001b[0;m=linux pid\u001b[0;m=147134 revision\u001b[0;m=436955cb version\u001b[0;m=16.6.1", "Running in system-mode.                           \u001b[0;m ", "                                                  \u001b[0;m ", "\u001b[0;33mWARNING: You have specified an authentication token in the legacy parameter --registration-token. This has triggered the 'legacy-compatible registration process' which has resulted in the following command line parameters being ignored: --locked, --access-level, --run-untagged, --maximum-timeout, --paused, --tag-list, and --maintenance-note. For more information, see https://docs.gitlab.com/ee/ci/runners/new_creation_workflow.html#changes-to-the-gitlab-runner-register-command-syntaxThese parameters and the legacy-compatible registration process will be removed in GitLab Runner 18.0. \u001b[0;m ", "Verifying runner... is valid                      \u001b[0;m  runner\u001b[0;m=kDNtto8nr", "Runner registered successfully. Feel free to start it, but if it's running already the config should be automatically reloaded!", "\u001b[0;m ", "Configuration (with the authentication token) was saved in \"/etc/gitlab-runner/config.toml\"\u001b[0;m "], "stdout": "", "stdout_lines": []}
---
 scripts/ci/setup/gitlab-runner.yml | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/scripts/ci/setup/gitlab-runner.yml b/scripts/ci/setup/gitlab-runner.yml
index 7bdafab511..48a3ecfe72 100644
--- a/scripts/ci/setup/gitlab-runner.yml
+++ b/scripts/ci/setup/gitlab-runner.yml
@@ -74,18 +74,18 @@
         - ansible_facts['distribution'] == 'CentOS'
 
     - name: Register the gitlab-runner
-      command: "/usr/bin/gitlab-runner register --non-interactive --url {{ gitlab_runner_server_url }} --registration-token {{ gitlab_runner_registration_token }} --executor shell --tag-list {{ ansible_facts[\"architecture\"] }},{{ ansible_facts[\"distribution\"]|lower }}_{{ ansible_facts[\"distribution_version\"] }} --description '{{ ansible_facts[\"distribution\"] }} {{ ansible_facts[\"distribution_version\"] }} {{ ansible_facts[\"architecture\"] }} ({{ ansible_facts[\"os_family\"] }})'"
+      command: "gitlab-runner register --non-interactive --url {{ gitlab_runner_server_url }} --registration-token {{ gitlab_runner_registration_token }} --executor shell --tag-list {{ ansible_facts[\"architecture\"] }},{{ ansible_facts[\"distribution\"]|lower }}_{{ ansible_facts[\"distribution_version\"] }} --description '{{ ansible_facts[\"distribution\"] }} {{ ansible_facts[\"distribution_version\"] }} {{ ansible_facts[\"architecture\"] }} ({{ ansible_facts[\"os_family\"] }})'"
 
     # The secondary runner will still run under the single gitlab-runner service
     - name: Register secondary gitlab-runner
-      command: "/usr/bin/gitlab-runner register --non-interactive --url {{ gitlab_runner_server_url }} --registration-token {{ gitlab_runner_registration_token }} --executor shell --tag-list aarch32,{{ ansible_facts[\"distribution\"]|lower }}_{{ ansible_facts[\"distribution_version\"] }} --description '{{ ansible_facts[\"distribution\"] }} {{ ansible_facts[\"distribution_version\"] }} {{ ansible_facts[\"architecture\"] }} ({{ ansible_facts[\"os_family\"] }})'"
+      command: "gitlab-runner register --non-interactive --url {{ gitlab_runner_server_url }} --registration-token {{ gitlab_runner_registration_token }} --executor shell --tag-list aarch32,{{ ansible_facts[\"distribution\"]|lower }}_{{ ansible_facts[\"distribution_version\"] }} --description '{{ ansible_facts[\"distribution\"] }} {{ ansible_facts[\"distribution_version\"] }} {{ ansible_facts[\"architecture\"] }} ({{ ansible_facts[\"os_family\"] }})'"
       when:
         - ansible_facts['distribution'] == 'Ubuntu'
         - ansible_facts['architecture'] == 'aarch64'
         - ansible_facts['distribution_version'] == '22.04'
 
     - name: Install the gitlab-runner service using its own functionality
-      command: "/usr/bin/gitlab-runner install --user gitlab-runner --working-directory /home/gitlab-runner"
+      command: "gitlab-runner install --user gitlab-runner --working-directory /home/gitlab-runner"
       register: gitlab_runner_install_service_result
       failed_when: "gitlab_runner_install_service_result.rc != 0 and \"already exists\" not in gitlab_runner_install_service_result.stderr"
 
-- 
2.41.0

[PATCH 2/9] scripts/ci: Do not restrict spice package to x86/arm hosts

[Philippe Mathieu-Daudé]

Allow installing spice-server-devel package on any non-ppc64le host,
as per commit  556ede028d ("scripts/ci/setup: spice-server only on
x86 aarch64") describes: "Spice server not available in ppc64le".

Signed-off-by: Philippe Mathieu-Daudé <philmd@linaro.org>
---
 scripts/ci/setup/build-environment.yml | 7 +++----
 1 file changed, 3 insertions(+), 4 deletions(-)

diff --git a/scripts/ci/setup/build-environment.yml b/scripts/ci/setup/build-environment.yml
index f344d1a850..32ac0a74f4 100644
--- a/scripts/ci/setup/build-environment.yml
+++ b/scripts/ci/setup/build-environment.yml
@@ -249,7 +249,7 @@
         - ansible_facts['distribution_file_variety'] in ['RedHat', 'CentOS']
         - ansible_facts['distribution_version'] == '8'
 
-    - name: Install packages only available on x86 and aarch64
+    - name: Install Spice packages
       dnf:
         # Spice server not available in ppc64le
         name:
@@ -257,9 +257,8 @@
           - spice-server-devel
         state: present
       when:
-        - ansible_facts['distribution_file_variety'] in ['RedHat', 'CentOS']
-        - ansible_facts['distribution_version'] == '8'
-        - ansible_facts['architecture'] == 'aarch64' or ansible_facts['architecture'] == 'x86_64'
+        - ansible_facts['os_family'] == 'RedHat'
+        - ansible_facts['architecture'] != 'ppc64le'
 
     - name: Check whether the Python runtime version is managed by alternatives
       stat:
-- 
2.41.0

[RFC PATCH 3/9] scripts/ci: Split EL8 specific packages out of Centos8 list

[Philippe Mathieu-Daudé]

Signed-off-by: Philippe Mathieu-Daudé <philmd@linaro.org>
---
 scripts/ci/setup/build-environment.yml | 12 ++++++++++--
 1 file changed, 10 insertions(+), 2 deletions(-)

diff --git a/scripts/ci/setup/build-environment.yml b/scripts/ci/setup/build-environment.yml
index 32ac0a74f4..60b751018f 100644
--- a/scripts/ci/setup/build-environment.yml
+++ b/scripts/ci/setup/build-environment.yml
@@ -234,10 +234,8 @@
           - nmap-ncat
           - numactl-devel
           - pixman-devel
-          - python38
           - python3-sphinx
           - rdma-core-devel
-          - redhat-rpm-config
           - snappy-devel
           - spice-glib-devel
           - systemd-devel
@@ -249,6 +247,16 @@
         - ansible_facts['distribution_file_variety'] in ['RedHat', 'CentOS']
         - ansible_facts['distribution_version'] == '8'
 
+    - name: Install packages only available on EL8
+      dnf:
+        name:
+          - python38
+          - redhat-rpm-config
+        state: present
+      when:
+        - ansible_facts['distribution_file_variety'] in ['RedHat']
+        - ansible_facts['distribution_version'] == '8'
+
     - name: Install Spice packages
       dnf:
         # Spice server not available in ppc64le
-- 
2.41.0

[RFC PATCH 4/9] scripts/ci: Update Centos8 package list

[Philippe Mathieu-Daudé]

Update Centos 8 package list running:

 $ lcitool variables -f json centos-stream-8 qemu \
   | jq -r '.pkgs[]' \
   | xargs -n 1 echo "          -"

On libvirt-ci commit e9e7d3bf ("facts: enable gtk-vnc2-devel
on almalinux 8 / centos stream 8").

Signed-off-by: Philippe Mathieu-Daudé <philmd@linaro.org>
---
 scripts/ci/setup/build-environment.yml | 75 +++++++++++++++++++++++---
 1 file changed, 68 insertions(+), 7 deletions(-)

diff --git a/scripts/ci/setup/build-environment.yml b/scripts/ci/setup/build-environment.yml
index 60b751018f..9518cc55a5 100644
--- a/scripts/ci/setup/build-environment.yml
+++ b/scripts/ci/setup/build-environment.yml
@@ -195,53 +195,114 @@
 
     - name: Install basic packages to build QEMU on EL8
       dnf:
-        # This list of packages start with tests/docker/dockerfiles/centos8.docker
-        # but only include files that are common to all distro variants and present
-        # in the standard repos (no add-ons)
         name:
+          - SDL2-devel
+          - alsa-lib-devel
+          - bash
+          - bc
+          - bison
+          - brlapi-devel
           - bzip2
           - bzip2-devel
+          - ca-certificates
           - capstone-devel
+          - ccache
+          - clang
+          - ctags
+          - cyrus-sasl-devel
+          - daxctl-devel
           - dbus-daemon
           - device-mapper-multipath-devel
           - diffutils
+          - findutils
+          - flex
           - gcc
           - gcc-c++
           - genisoimage
           - gettext
           - git
           - glib2-devel
+          - glib2-static
+          - glibc-langpack-en
+          - glibc-static
           - glusterfs-api-devel
           - gnutls-devel
+          - gtk3-devel
+          - hostname
+          - jemalloc-devel
+          - json-c-devel
           - libaio-devel
+          - libasan
+          - libattr-devel
+          - libbpf-devel
+          - libcacard-devel
           - libcap-ng-devel
+          - libcmocka-devel
           - libcurl-devel
+          - libdrm-devel
           - libepoxy-devel
           - libfdt-devel
+          - libffi-devel
           - libgcrypt-devel
           - libiscsi-devel
+          - libjpeg-devel
+          - libnfs-devel
           - libpmem-devel
-          - librados-devel
+          - libpng-devel
           - librbd-devel
           - libseccomp-devel
+          - libslirp-devel
           - libssh-devel
-          - libxkbcommon-devel
+          - libtasn1-devel
+          - libubsan
+          - liburing-devel
+          - libusbx-devel
+          - libxdp-devel
+          - libxml2-devel
+          - libzstd-devel
+          - llvm
+          - lttng-ust-devel
           - lzo-devel
           - make
-          - mesa-libEGL-devel
+          - mesa-libgbm-devel
+          - meson
+          - ncurses-devel
           - nettle-devel
           - ninja-build
           - nmap-ncat
           - numactl-devel
+          - openssh-clients
+          - pam-devel
+          - pcre-static
+          - pipewire-devel
           - pixman-devel
+          - pkgconfig
+          - pulseaudio-libs-devel
+          - python3
+          - python3-PyYAML
+          - python3-numpy
+          - python3-pillow
+          - python3-pip
           - python3-sphinx
+          - python3-sphinx_rtd_theme
+          - python3-tomli
           - rdma-core-devel
+          - sed
           - snappy-devel
-          - spice-glib-devel
+          - spice-protocol
+          - spice-server-devel
           - systemd-devel
           - systemtap-sdt-devel
           - tar
+          - texinfo
+          - usbredir-devel
+          - util-linux
+          - virglrenderer-devel
+          - vte291-devel
+          - which
+          - xfsprogs-devel
           - zlib-devel
+          - zlib-static
         state: present
       when:
         - ansible_facts['distribution_file_variety'] in ['RedHat', 'CentOS']
-- 
2.41.0

[RFC PATCH 5/9] scripts/ci: Restrict libpmem-devel package to x86 hosts

[Philippe Mathieu-Daudé]

libpmem-devel is not available for all host archs.

Signed-off-by: Philippe Mathieu-Daudé <philmd@linaro.org>
---
 scripts/ci/setup/build-environment.yml | 9 ++++++++-
 1 file changed, 8 insertions(+), 1 deletion(-)

diff --git a/scripts/ci/setup/build-environment.yml b/scripts/ci/setup/build-environment.yml
index 9518cc55a5..280e67593c 100644
--- a/scripts/ci/setup/build-environment.yml
+++ b/scripts/ci/setup/build-environment.yml
@@ -247,7 +247,6 @@
           - libiscsi-devel
           - libjpeg-devel
           - libnfs-devel
-          - libpmem-devel
           - libpng-devel
           - librbd-devel
           - libseccomp-devel
@@ -318,6 +317,14 @@
         - ansible_facts['distribution_file_variety'] in ['RedHat']
         - ansible_facts['distribution_version'] == '8'
 
+    - name: Install arch-specific packages (RH family)
+      dnf:
+        name:
+          - libpmem-devel
+      when:
+        - ansible_facts['os_family'] == 'RedHat'
+        - ansible_facts['architecture'] == 'x86_64'
+
     - name: Install Spice packages
       dnf:
         # Spice server not available in ppc64le
-- 
2.41.0

[RFC PATCH 6/9] scripts/ci: Install libxdp-devel on x86 hosts

[Philippe Mathieu-Daudé]

libxdp is arch-specific, and in use since commit cb039ef3d9
("net: add initial support for AF_XDP network backend").

Signed-off-by: Philippe Mathieu-Daudé <philmd@linaro.org>
---
 scripts/ci/setup/build-environment.yml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/scripts/ci/setup/build-environment.yml b/scripts/ci/setup/build-environment.yml
index 280e67593c..8a6855f472 100644
--- a/scripts/ci/setup/build-environment.yml
+++ b/scripts/ci/setup/build-environment.yml
@@ -256,7 +256,6 @@
           - libubsan
           - liburing-devel
           - libusbx-devel
-          - libxdp-devel
           - libxml2-devel
           - libzstd-devel
           - llvm
@@ -321,6 +320,7 @@
       dnf:
         name:
           - libpmem-devel
+          - libxdp-devel
       when:
         - ansible_facts['os_family'] == 'RedHat'
         - ansible_facts['architecture'] == 'x86_64'
-- 
2.41.0

[RFC PATCH 7/9] scripts/ci: Install RH packages on RH derivative distros

[Philippe Mathieu-Daudé]

Signed-off-by: Philippe Mathieu-Daudé <philmd@linaro.org>
---
 scripts/ci/setup/build-environment.yml | 5 ++---
 1 file changed, 2 insertions(+), 3 deletions(-)

diff --git a/scripts/ci/setup/build-environment.yml b/scripts/ci/setup/build-environment.yml
index 8a6855f472..6baa539b46 100644
--- a/scripts/ci/setup/build-environment.yml
+++ b/scripts/ci/setup/build-environment.yml
@@ -193,7 +193,7 @@
         - ansible_facts['distribution_file_variety'] == 'CentOS'
         - ansible_facts['distribution_major_version'] == '8'
 
-    - name: Install basic packages to build QEMU on EL8
+    - name: Install basic packages to build QEMU on RH family
       dnf:
         name:
           - SDL2-devel
@@ -303,8 +303,7 @@
           - zlib-static
         state: present
       when:
-        - ansible_facts['distribution_file_variety'] in ['RedHat', 'CentOS']
-        - ansible_facts['distribution_version'] == '8'
+        - ansible_facts['os_family'] == 'RedHat'
 
     - name: Install packages only available on EL8
       dnf:
-- 
2.41.0

[RFC PATCH 8/9] scripts/ci: Update Ubuntu packages list

[Philippe Mathieu-Daudé]

Update Ubuntu 22.04 package list running:

 $ lcitool variables -f json ubuntu-2204 qemu \
   | jq -r '.pkgs[]' \
   | xargs -n 1 echo "          -"

On libvirt-ci commit e9e7d3bf ("facts: enable gtk-vnc2-devel
on almalinux 8 / centos stream 8").

Signed-off-by: Philippe Mathieu-Daudé <philmd@linaro.org>
---
 scripts/ci/setup/build-environment.yml | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/scripts/ci/setup/build-environment.yml b/scripts/ci/setup/build-environment.yml
index 6baa539b46..988aef3a07 100644
--- a/scripts/ci/setup/build-environment.yml
+++ b/scripts/ci/setup/build-environment.yml
@@ -58,7 +58,7 @@
           - git
           - hostname
           - libaio-dev
-          - libasan5
+          - libasan6
           - libasound2-dev
           - libattr1-dev
           - libbpf-dev
@@ -94,6 +94,7 @@
           - libnuma-dev
           - libpam0g-dev
           - libpcre2-dev
+          - libpipewire-0.3-dev
           - libpixman-1-dev
           - libpmem-dev
           - libpng-dev
@@ -139,6 +140,7 @@
           - python3-pip
           - python3-sphinx
           - python3-sphinx-rtd-theme
+          - python3-tomli
           - python3-venv
           - python3-yaml
           - rpm2cpio
-- 
2.41.0

[RFC PATCH 9/9] scripts/ci: Restrict libpmem-dev and libxen-dev packages to x86 hosts

[Philippe Mathieu-Daudé]

libpmem-dev / libxen-dev are not available for all host archs.

Signed-off-by: Philippe Mathieu-Daudé <philmd@linaro.org>
---
 scripts/ci/setup/build-environment.yml | 13 +++++++++++--
 1 file changed, 11 insertions(+), 2 deletions(-)

diff --git a/scripts/ci/setup/build-environment.yml b/scripts/ci/setup/build-environment.yml
index 988aef3a07..8b5b8e1dbd 100644
--- a/scripts/ci/setup/build-environment.yml
+++ b/scripts/ci/setup/build-environment.yml
@@ -96,7 +96,6 @@
           - libpcre2-dev
           - libpipewire-0.3-dev
           - libpixman-1-dev
-          - libpmem-dev
           - libpng-dev
           - libpulse-dev
           - librbd-dev
@@ -120,7 +119,6 @@
           - libvdeplug-dev
           - libvirglrenderer-dev
           - libvte-2.91-dev
-          - libxen-dev
           - libxml2-dev
           - libzstd-dev
           - llvm
@@ -158,6 +156,17 @@
         - ansible_facts['distribution'] == 'Ubuntu'
         - ansible_facts['distribution_version'] == '22.04'
 
+    - name: Install arch-specific packages (Ubuntu 22.04)
+      dnf:
+        name:
+          - libpmem-dev
+          - libxen-dev
+      when:
+        - ansible_facts['distribution'] == 'Ubuntu'
+        - ansible_facts['distribution_version'] == '22.04'
+        - ansible_facts['architecture'] == 'x86_64'
+
+
     - name: Install armhf cross-compile packages to build QEMU on AArch64 Ubuntu 22.04
       package:
         name:
-- 
2.41.0