[PATCH v2 0/3] tests/vm/freebsd: Get up-to-date package list from lcitool

[PATCH v2 0/3] tests/vm/freebsd: Get up-to-date package list from lcitool

[Philippe Mathieu-Daudé]

Inspired by this patch from Thomas:
https://lore.kernel.org/qemu-devel/20230531090415.40421-1-thuth@redhat.com/

Instead of updating the package list manually, use lcitool vars file.

Since v1:
- Addressed Erik & Daniel comments (generate in JSON)

Philippe Mathieu-Daudé (3):
  tests/lcitool: Generate distribution packages list in JSON format
  tests/vm: Introduce get_qemu_packages_from_lcitool_json() helper
  tests/vm/freebsd: Get up-to-date package list from lcitool vars file

 tests/lcitool/refresh     | 11 ++++++++++
 tests/vm/Makefile.include |  6 ++++++
 tests/vm/basevm.py        |  9 +++++++++
 tests/vm/freebsd          | 42 ++-------------------------------------
 4 files changed, 28 insertions(+), 40 deletions(-)

-- 
2.38.1

[PATCH v2 1/3] tests/lcitool: Generate distribution packages list in JSON format

[Philippe Mathieu-Daudé]

Add the generate_pkglist() helper to generate a list of packages
required by a distribution to build QEMU.

Generate the FreeBSD JSON file (based on FreeBSD 13).

Suggested-by: Erik Skultety <eskultet@redhat.com>
Signed-off-by: Philippe Mathieu-Daudé <philmd@linaro.org>
---
 tests/lcitool/refresh | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/tests/lcitool/refresh b/tests/lcitool/refresh
index b54566edcc..a5df096074 100755
--- a/tests/lcitool/refresh
+++ b/tests/lcitool/refresh
@@ -84,6 +84,12 @@ def generate_cirrus(target, trailer=None):
     generate(filename, cmd, trailer)
 
 
+def generate_pkglist(vm, target):
+    filename = Path(src_dir, "tests", "vm", vm + ".json")
+    cmd = lcitool_cmd + ["variables", "--format", "json", target, "qemu"]
+    generate(filename, cmd, None)
+
+
 # Netmap still needs to be manually built as it is yet to be packaged
 # into a distro. We also add cscope and gtags which are used in the CI
 # test
@@ -191,6 +197,11 @@ try:
     generate_cirrus("freebsd-13")
     generate_cirrus("macos-12")
 
+    #
+    # VM packages lists
+    #
+    generate_pkglist("freebsd", "freebsd-13")
+
     sys.exit(0)
 except Exception as ex:
     print(str(ex), file=sys.stderr)
-- 
2.38.1

Re: [PATCH v2 1/3] tests/lcitool: Generate distribution packages list in JSON format

[Daniel P. Berrangé]

On Tue, Jul 11, 2023 at 04:01:41PM +0200, Philippe Mathieu-Daudé wrote:
> Add the generate_pkglist() helper to generate a list of packages
> required by a distribution to build QEMU.
> 
> Generate the FreeBSD JSON file (based on FreeBSD 13).
> 
> Suggested-by: Erik Skultety <eskultet@redhat.com>
> Signed-off-by: Philippe Mathieu-Daudé <philmd@linaro.org>
> ---
>  tests/lcitool/refresh | 11 +++++++++++
>  1 file changed, 11 insertions(+)

Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>


With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [PATCH v2 1/3] tests/lcitool: Generate distribution packages list in JSON format

[Philippe Mathieu-Daudé]

On 11/7/23 16:01, Philippe Mathieu-Daudé wrote:
> Add the generate_pkglist() helper to generate a list of packages
> required by a distribution to build QEMU.
> 
> Generate the FreeBSD JSON file (based on FreeBSD 13).
> 
> Suggested-by: Erik Skultety <eskultet@redhat.com>
> Signed-off-by: Philippe Mathieu-Daudé <philmd@linaro.org>
> ---
>   tests/lcitool/refresh | 11 +++++++++++
>   1 file changed, 11 insertions(+)
> 
> diff --git a/tests/lcitool/refresh b/tests/lcitool/refresh
> index b54566edcc..a5df096074 100755
> --- a/tests/lcitool/refresh
> +++ b/tests/lcitool/refresh
> @@ -84,6 +84,12 @@ def generate_cirrus(target, trailer=None):
>       generate(filename, cmd, trailer)
>   
>   
> +def generate_pkglist(vm, target):
> +    filename = Path(src_dir, "tests", "vm", vm + ".json")
> +    cmd = lcitool_cmd + ["variables", "--format", "json", target, "qemu"]
> +    generate(filename, cmd, None)

Note since json files don't have comments, we can't have the
"THIS IS GENERATED BY A SCRIPT" in header.

[PATCH v2 2/3] tests/vm: Introduce get_qemu_packages_from_lcitool_json() helper

[Philippe Mathieu-Daudé]

Add the get_qemu_packages_from_lcitool_json() helper which return
such package list from a lcitool env var file in JSON format.

Suggested-by: Daniel P. Berrangé <berrange@redhat.com>
Signed-off-by: Philippe Mathieu-Daudé <philmd@linaro.org>
---
 tests/vm/Makefile.include | 4 ++++
 tests/vm/basevm.py        | 9 +++++++++
 2 files changed, 13 insertions(+)

diff --git a/tests/vm/Makefile.include b/tests/vm/Makefile.include
index c2a8ca1c17..b021b344b5 100644
--- a/tests/vm/Makefile.include
+++ b/tests/vm/Makefile.include
@@ -106,6 +106,10 @@ $(IMAGES_DIR)/%.img:	$(SRC_PATH)/tests/vm/% \
 		--build-image $@, \
 		"  VM-IMAGE $*")
 
+$(SRC_PATH)/tests/vm/%.json:
+	$(call quiet-command, \
+		make lcitool-refresh)
+
 # Build in VM $(IMAGE)
 vm-build-%: $(IMAGES_DIR)/%.img $(VM_VENV)
 	$(call quiet-command, \
diff --git a/tests/vm/basevm.py b/tests/vm/basevm.py
index 23229e23d1..cc4cc0a973 100644
--- a/tests/vm/basevm.py
+++ b/tests/vm/basevm.py
@@ -27,6 +27,7 @@
 import multiprocessing
 import traceback
 import shlex
+import json
 
 from qemu.machine import QEMUMachine
 from qemu.utils import get_info_usernet_hostfwd_port, kvm_available
@@ -501,6 +502,14 @@ def gen_cloud_init_iso(self):
                               stderr=self._stdout)
         return os.path.join(cidir, "cloud-init.iso")
 
+    def get_qemu_packages_from_lcitool_json(self, json_path=None):
+        """Parse a lcitool variables json file and return the PKGS list."""
+        if json_path is None:
+            json_path = os.path.join(os.path.dirname(__file__), self.name + ".json")
+        with open(json_path, "r") as fh:
+            return json.load(fh)["pkgs"]
+
+
 def get_qemu_path(arch, build_path=None):
     """Fetch the path to the qemu binary."""
     # If QEMU environment variable set, it takes precedence
-- 
2.38.1

Re: [PATCH v2 2/3] tests/vm: Introduce get_qemu_packages_from_lcitool_json() helper

[Daniel P. Berrangé]

On Tue, Jul 11, 2023 at 04:01:42PM +0200, Philippe Mathieu-Daudé wrote:
> Add the get_qemu_packages_from_lcitool_json() helper which return
> such package list from a lcitool env var file in JSON format.
> 
> Suggested-by: Daniel P. Berrangé <berrange@redhat.com>
> Signed-off-by: Philippe Mathieu-Daudé <philmd@linaro.org>
> ---
>  tests/vm/Makefile.include | 4 ++++
>  tests/vm/basevm.py        | 9 +++++++++
>  2 files changed, 13 insertions(+)
> 
> diff --git a/tests/vm/Makefile.include b/tests/vm/Makefile.include
> index c2a8ca1c17..b021b344b5 100644
> --- a/tests/vm/Makefile.include
> +++ b/tests/vm/Makefile.include
> @@ -106,6 +106,10 @@ $(IMAGES_DIR)/%.img:	$(SRC_PATH)/tests/vm/% \
>  		--build-image $@, \
>  		"  VM-IMAGE $*")
>  
> +$(SRC_PATH)/tests/vm/%.json:
> +	$(call quiet-command, \
> +		make lcitool-refresh)

'lcitool-refresh' is something we only run explicitly when we
change something about the lcitool, and also isn't provided
to end users running from tarballs IIRC. So we shouldn't need
to wire it up to build targets, just commit its output JSON
file.


With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [PATCH v2 2/3] tests/vm: Introduce get_qemu_packages_from_lcitool_json() helper

[Philippe Mathieu-Daudé]

On 11/7/23 16:08, Daniel P. Berrangé wrote:
> On Tue, Jul 11, 2023 at 04:01:42PM +0200, Philippe Mathieu-Daudé wrote:
>> Add the get_qemu_packages_from_lcitool_json() helper which return
>> such package list from a lcitool env var file in JSON format.
>>
>> Suggested-by: Daniel P. Berrangé <berrange@redhat.com>
>> Signed-off-by: Philippe Mathieu-Daudé <philmd@linaro.org>
>> ---
>>   tests/vm/Makefile.include | 4 ++++
>>   tests/vm/basevm.py        | 9 +++++++++
>>   2 files changed, 13 insertions(+)
>>
>> diff --git a/tests/vm/Makefile.include b/tests/vm/Makefile.include
>> index c2a8ca1c17..b021b344b5 100644
>> --- a/tests/vm/Makefile.include
>> +++ b/tests/vm/Makefile.include
>> @@ -106,6 +106,10 @@ $(IMAGES_DIR)/%.img:	$(SRC_PATH)/tests/vm/% \
>>   		--build-image $@, \
>>   		"  VM-IMAGE $*")
>>   
>> +$(SRC_PATH)/tests/vm/%.json:
>> +	$(call quiet-command, \
>> +		make lcitool-refresh)
> 
> 'lcitool-refresh' is something we only run explicitly when we
> change something about the lcitool, and also isn't provided
> to end users running from tarballs IIRC. So we shouldn't need
> to wire it up to build targets, just commit its output JSON
> file.

Right, simpler. Thanks!

[PATCH v2 3/3] tests/vm/freebsd: Get up-to-date package list from lcitool vars file

[Philippe Mathieu-Daudé]

Get an up-to-date package list from lcitool, that way we
don't need to manually keep this array in sync.

Inspired-by: Thomas Huth <thuth@redhat.com>
Signed-off-by: Philippe Mathieu-Daudé <philmd@linaro.org>
---
 tests/vm/Makefile.include |  2 ++
 tests/vm/freebsd          | 42 ++-------------------------------------
 2 files changed, 4 insertions(+), 40 deletions(-)

diff --git a/tests/vm/Makefile.include b/tests/vm/Makefile.include
index b021b344b5..84b8ad5222 100644
--- a/tests/vm/Makefile.include
+++ b/tests/vm/Makefile.include
@@ -110,6 +110,8 @@ $(SRC_PATH)/tests/vm/%.json:
 	$(call quiet-command, \
 		make lcitool-refresh)
 
+vm-build-freebsd: $(SRC_PATH)/tests/vm/freebsd.json
+
 # Build in VM $(IMAGE)
 vm-build-%: $(IMAGES_DIR)/%.img $(VM_VENV)
 	$(call quiet-command, \
diff --git a/tests/vm/freebsd b/tests/vm/freebsd
index 11de6473f4..ac51376c82 100755
--- a/tests/vm/freebsd
+++ b/tests/vm/freebsd
@@ -31,45 +31,6 @@ class FreeBSDVM(basevm.BaseVM):
     link = "https://download.freebsd.org/releases/CI-IMAGES/13.2-RELEASE/amd64/Latest/FreeBSD-13.2-RELEASE-amd64-BASIC-CI.raw.xz"
     csum = "a4fb3b6c7b75dd4d58fb0d75e4caf72844bffe0ca00e66459c028b198ffb3c0e"
     size = "20G"
-    pkgs = [
-        # build tools
-        "git",
-        "pkgconf",
-        "bzip2",
-        "python39",
-        "ninja",
-
-        # gnu tools
-        "bash",
-        "gmake",
-        "gsed",
-        "gettext",
-
-        # libs: crypto
-        "gnutls",
-
-        # libs: images
-        "jpeg-turbo",
-        "png",
-
-        # libs: ui
-        "sdl2",
-        "gtk3",
-        "libxkbcommon",
-
-        # libs: opengl
-        "libepoxy",
-        "mesa-libs",
-
-        # libs: migration
-        "zstd",
-
-        # libs: networking
-        "libslirp",
-
-        # libs: sndio
-        "sndio",
-    ]
 
     BUILD_SCRIPT = """
         set -e;
@@ -151,8 +112,9 @@ class FreeBSDVM(basevm.BaseVM):
         self.console_wait(prompt)
         self.console_send("echo 'chmod 666 /dev/vtbd1' >> /etc/rc.local\n")
 
+        pkgs = self.get_qemu_packages_from_lcitool_json()
         self.print_step("Installing packages")
-        self.ssh_root_check("pkg install -y %s\n" % " ".join(self.pkgs))
+        self.ssh_root_check("pkg install -y %s\n" % " ".join(pkgs))
 
         # shutdown
         self.ssh_root(self.poweroff)
-- 
2.38.1

Re: [PATCH v2 3/3] tests/vm/freebsd: Get up-to-date package list from lcitool vars file

[Daniel P. Berrangé]

On Tue, Jul 11, 2023 at 04:01:43PM +0200, Philippe Mathieu-Daudé wrote:
> Get an up-to-date package list from lcitool, that way we
> don't need to manually keep this array in sync.
> 
> Inspired-by: Thomas Huth <thuth@redhat.com>
> Signed-off-by: Philippe Mathieu-Daudé <philmd@linaro.org>
> ---
>  tests/vm/Makefile.include |  2 ++
>  tests/vm/freebsd          | 42 ++-------------------------------------
>  2 files changed, 4 insertions(+), 40 deletions(-)
> 
> diff --git a/tests/vm/Makefile.include b/tests/vm/Makefile.include
> index b021b344b5..84b8ad5222 100644
> --- a/tests/vm/Makefile.include
> +++ b/tests/vm/Makefile.include
> @@ -110,6 +110,8 @@ $(SRC_PATH)/tests/vm/%.json:
>  	$(call quiet-command, \
>  		make lcitool-refresh)
>  
> +vm-build-freebsd: $(SRC_PATH)/tests/vm/freebsd.json

I don't think we need this, as we just commit the result of
lcitool as needed.

> +
>  # Build in VM $(IMAGE)
>  vm-build-%: $(IMAGES_DIR)/%.img $(VM_VENV)
>  	$(call quiet-command, \
> diff --git a/tests/vm/freebsd b/tests/vm/freebsd
> index 11de6473f4..ac51376c82 100755
> --- a/tests/vm/freebsd
> +++ b/tests/vm/freebsd
> @@ -31,45 +31,6 @@ class FreeBSDVM(basevm.BaseVM):
>      link = "https://download.freebsd.org/releases/CI-IMAGES/13.2-RELEASE/amd64/Latest/FreeBSD-13.2-RELEASE-amd64-BASIC-CI.raw.xz"
>      csum = "a4fb3b6c7b75dd4d58fb0d75e4caf72844bffe0ca00e66459c028b198ffb3c0e"
>      size = "20G"
> -    pkgs = [
> -        # build tools
> -        "git",
> -        "pkgconf",
> -        "bzip2",
> -        "python39",
> -        "ninja",
> -
> -        # gnu tools
> -        "bash",
> -        "gmake",
> -        "gsed",
> -        "gettext",
> -
> -        # libs: crypto
> -        "gnutls",
> -
> -        # libs: images
> -        "jpeg-turbo",
> -        "png",
> -
> -        # libs: ui
> -        "sdl2",
> -        "gtk3",
> -        "libxkbcommon",
> -
> -        # libs: opengl
> -        "libepoxy",
> -        "mesa-libs",
> -
> -        # libs: migration
> -        "zstd",
> -
> -        # libs: networking
> -        "libslirp",
> -
> -        # libs: sndio
> -        "sndio",
> -    ]
>  
>      BUILD_SCRIPT = """
>          set -e;
> @@ -151,8 +112,9 @@ class FreeBSDVM(basevm.BaseVM):
>          self.console_wait(prompt)
>          self.console_send("echo 'chmod 666 /dev/vtbd1' >> /etc/rc.local\n")
>  
> +        pkgs = self.get_qemu_packages_from_lcitool_json()
>          self.print_step("Installing packages")
> -        self.ssh_root_check("pkg install -y %s\n" % " ".join(self.pkgs))
> +        self.ssh_root_check("pkg install -y %s\n" % " ".join(pkgs))
>  
>          # shutdown
>          self.ssh_root(self.poweroff)
> -- 
> 2.38.1
> 

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|