[PATCH v9 0/3] Acceptance test: Add "boot_linux" acceptance test

[PATCH v9 0/3] Acceptance test: Add "boot_linux" acceptance test

[Cleber Rosa]

This acceptance test, validates that a full blown Linux guest can
successfully boot in QEMU.  In this specific case, the guest chosen is
Fedora version 31.  It covers the following architectures and
machine types:

 * x86_64, pc-i440fx and pc-q35 machine types, with TCG and KVM as
   accelerators

 * aarch64 and virt machine type, with TCG and KVM as accelerators

 * ppc64 and pseries machine type with TCG as accelerator

 * s390x and s390-ccw-virtio machine type with TCG as accelerator

This has been tested on x86_64, ppc64le and aarch64 hosts and has been
running reliably (in my experience) on Travis CI.

Git:
  - URI: https://github.com/clebergnu/qemu/tree/test_boot_linux_v9
  - Remote: https://github.com/clebergnu/qemu
  - Branch: test_boot_linux_v9

Travis CI:
  - Build: https://travis-ci.org/clebergnu/qemu/builds/652694503

Previous version:
  - v8: https://lists.gnu.org/archive/html/qemu-devel/2019-12/msg04095.html
  - v7: https://lists.gnu.org/archive/html/qemu-devel/2019-11/msg00220.html
  - v6: https://lists.gnu.org/archive/html/qemu-devel/2019-06/msg01202.html
  - v5: https://lists.gnu.org/archive/html/qemu-devel/2019-03/msg04652.html
  - v4: https://lists.gnu.org/archive/html/qemu-devel/2019-02/msg02032.html
  - v3: https://lists.gnu.org/archive/html/qemu-devel/2019-02/msg01677.html
  - v2: https://lists.gnu.org/archive/html/qemu-devel/2018-11/msg04318.html
  - v1: http://lists.nongnu.org/archive/html/qemu-devel/2018-09/msg02530.html

Changes from v8:
================

* Renamed "BLD_DIR" to "BUILD_DIR", "SRC_DIR" to "SOURCE_DIR" and dropped
  "LNK_DIR" variables on tests/acceptance/avocado_qemu/__init__.py

* Changed memory allocation to 1024 MB, so that it puts less pressure on
  the host memory, and should be compatible with 32bit hosts (I've found
  no significant effects to the test times)

* Explicitly enabled TCG and skip tests if it's not available

* Added tags for when accel is TCG ("accel:tcg")

* Added additional tags for "pc" alias, that is, "pc-i440fx"

* Renamed tests to make the machine type and accellerator more explicit:
  - BootLinuxX8664.test_pc => BootLinuxX8664.test_pc_i440fx_tcg
  - BootLinuxX8664.test_pc_kvm => BootLinuxX8664.test_pc_i440fx_kvm
  - BootLinuxX8664.test_q35 => BootLinuxX8664.test_pc_q35_tcg
  - BootLinuxX8664.test_kvm_q35 => BootLinuxX8664.test_pc_q35_kvm
  - BootLinuxAarch64.test_virt => BootLinuxAarch64.test_virt_tcg
  - BootLinuxAarch64.test_kvm_virt => BootLinuxAarch64.test_virt_kvm
  - BootLinuxPPC64.test_pseries => BootLinuxPPC64.test_pseries_tcg
  - BootLinuxS390X.test_s390_ccw_virtio => BootLinuxS390X.test_s390_ccw_virtio_tcg

* Renamed target "get-vmimage" to "get-vm-images", and added a help
  entry under "check-help".

* Bumped pycdlib version to 1.9.0, which contains an endianess bug that
  was seen on s390x hosts.

Changes from v7:
================

This version drops a number of commits that had been already reviewed
and have been merged:

 * Dropped commit "Acceptance tests: use relative location for tests",
   already present in the latest master.

 * Dropped commit "Acceptance tests: use avocado tags for machine type",
   already present in the latest master.

 * Dropped commit: "Acceptance tests: introduce utility method for tags
   unique vals", already present in the latest master.

With regards to the handling of the build directory, and the usage of
a qemu-img binary from the build tree, the following changed:

 * Dropped commit "Acceptance tests: add the build directory to the
   system PATH", because the qemu-img binary to be used is now
   explicitly defined, instead of relying on the modification of the
   PATH environment variable.

 * Dropped commit "Acceptance tests: depend on qemu-img", replaced by
   explicitly setting the qemu-img binary to be used for snapshot
   generation.  Also, the newly added "--enable-tools" configure line
   on Travis CI makes sure that a matching qemu-img binary is
   available on CI.

 * Dropped commit "Acceptance tests: keep a stable reference to the
   QEMU build dir", replaced by a different approach that introduces
   variables tracking the build dir, source dir and link (from build
   to source) dir.

 * New commit "Acceptance tests: introduce BLD_DIR, SRC_DIR and
   LNK_DIR".

 * New commit "Acceptance tests: add make targets to download images",
   that downloads the cloud images, aka vmimages, before the test
   execution itself.

 * New commit "[TO BE REMOVED] Use Avocado master branch + vmimage fix"
   to facilitate the review/test of this version.

Additionally:

  * The check for the availability of kvm now makes use of the
    strengthened qemu.accel.kvm_available() and passes the QEMU binary
    as an argument to make sure KVM support is compiled into that
    binary.

 * The timeout was increased to 900 seconds.  This is just one extra
   step to avoid false negatives on very slow systems.  As a
   comparison, on Travis CI, on a x86_64 host, the slowest test takes
   around 250 seconds (boot_linux.py:BootLinuxAarch64.test_virt).  On
   x86_64 systems with KVM enabled, my experience is that a test will
   take around 15 seconds.

Changes from v6:
================

 * Bumped Fedora to most recently released version (31).

 * Included new architectures (ppc64 and s390x), consolidating all
   tests into the same commit.

 * New commit: "Acceptance tests: use avocado tags for machine type"

 * New commit: "Acceptance tests: introduce utility method for tags
   unique vals"

 * New commit: "Acceptance test x86_cpu_model_versions: use default
   vm", needed to normalize the use of the machine type tags

 * Added a lot of leniency to the test setup (and reliability to the
   test/job), canceling the test if there are any failures while
   downloading/preparing the boot images.

 * Made use of Avocado's data drainer a regular feature (dropped the
   commit with RFC) and squashed it.

 * Bumped pycdlib version to 1.8.0

 * Dropped explicit "--enable-slirp=git" (added on v5) to Travis CI
   configure line, as the default configuration on Travis CI now
   results in user networking capabilities.

Changes from v5:
================

 * Added explicit "--enable-slirp=git" to Travis CI configure line, as
   these tests depend on "-netdev user" like networking.

 * Bumped Fedora to most recently released version (30).

 * Changed "checksum" parameter to 'sha256' and use the same hashes as
   provided by the Fedora project (instead of using Avocado's default
   sha1 and compute and use a different hash value).

 * New commit: Add "boot_linux" test for aarch64 and virt machine type

 * New commit: [RFC]: use Avocado data drainer for console logging

Changes from v4:
================

 * New commit "Acceptance tests: use relative location for tests"

 * New commit "Acceptance tests: keep a stable reference to the QEMU build dir"

 * Pinned the Fedora 29 image by adding a checksum.  The goal is to
   never allow more than one component to change at a time (the one
   allowed to change is QEMU itself).  Updates to the image should be
   manual. (Based on comments from Cornelia)

 * Moved the downloading of the Fedora 29 cloud image to the test
   setUp() method, canceling the test if the image can not be
   downloaded.

 * Removed the ":avocado: enable" tag, given that Avocado versions
   68.0 and later operate on a "recursive by default" manner, that
   is able to correctly identify this as an Avocado test.

Changes from v3:
================

 * New patch "Acceptance tests: depend on qemu-img"

Known Issues on v3 (no longer applicable):
==========================================

 * A recent TCG performance regression[1] affects this test in a
   number of ways:
   - The test execution may timeout by itself
   - The generation of SSH host keys in the guest's first boot is also
     affected (possibly also a timeout)
   - The cloud-init "phone home" feature attempts to read the host keys
     and fails, causing the test to timeout and fail

   These are not observed anymore once the fix[2] is applied.

[1] - https://lists.gnu.org/archive/html/qemu-devel/2019-02/msg00338.html
[2] - https://lists.gnu.org/archive/html/qemu-devel/2019-02/msg01129.html

Changes from v2:
================

 * Updated the tag to include the "arch:" key, in a similar fashion as to
   the tests in the "Acceptance Tests: target architecture support":
   - https://lists.gnu.org/archive/html/qemu-devel/2019-02/msg00369.html

 * Renamed the test method name to test_x86_64_pc, again, similarly to the
   boot_linux_console.py tests in the series mentioned before.

 * Set the machine type explicitly, again similarly to the
   boot_linux_console.py tests in the series mentioned before.

 * Added messages after the launch of the VM, to let test runners know
   the test know waits for a boot confirmation from the the guest (Eduardo).

 * Updated commit message to reflect the fact that this version does
   not allow for parameterization of the guest OS, version, etc.

 * Dropped the RFC prefix on patch "RFC: Acceptance tests: add the
   build directory to the system PATH"

 * Changed the comments on "RFC: Acceptance tests: add the build
   directory to the system PATH" to make it clear the addition of a
   the build directory to the PATH may influence other utility code.

Changes from v1:
================

 * The commit message was adjusted, removing the reference to the
   avocado.utils.vmimage encoding issue on previous Avocado versions
   (<= 64.0) and the fix that would (and was) included in Avocado
   version 65.0.

 * Effectively added pycdlib==1.6.0 to the requirements.txt file,
   added on a56931eef3, and adjusted the commit message was also
   to reflect that.

 * Updated the default version of the guest OS, from Fedora 28 to 29.
   Besides possible improvements in the (virtual) hardware coverage,
   it brings a performance improvement in the order of 20% to the
   test.

 * Removed all direct parameters usage.  Because some parameters and
   its default values implemented in the test would prevent it from
   running on some environments.  Example: the "accel" parameter had a
   default value of "kvm", which would prevent this test, that boots a
   x86_64 OS, from running on a host arch different than x86_64.  I
   recognize that it's desirable to make tests reusable and
   parameterized (that was the reason for the first version doing so),
   but the mechanism to be used to define the architectures that a
   given test should support is still an open issue, and has been
   discussed in other threads.  I'll follow up those discussions with
   a proposal, and until then, removing those aspects from this test
   implementation seemed to be the best option.  A caveat: this test
   currently adds the same tag (x86_64) and follows other assumptions
   made on "boot_linux_console.py", that is, that a x86_64 target
   binary will be used to run it.  If a user is in an environment that
   does not have a x86_64 target binary, it could filter those tests
   out with: "avocado run --filter-by-tags='-x86_64' tests/acceptance".

 * Removed most arguments to the QEMU command line for pretty much the
   same reasons described above, and by following the general
   perception that I could grasp from other discussions that QEMU
   defaults should preferrably be used.  This test, as well as others,
   can and should be extended later to allow for different test
   scenarios by passing well documented parameter values.  That is,
   they should respect well-known parameters such as "accel" mentioned
   above, so that the same test can run with KVM or TCG.

 * Changed the value of the memory argument to 1024, which based on
   my experimentations and observations is the minimum amount of RAM
   for the Fedora 29 cloud image to sucessfully boot on QEMU.  I know
   there's no such thing as a "one size fits all", specially for QEMU,
   but this makes me wonder wether a x86_64 machine type shouldn't
   have its default_ram_size bumped to a number practical enough to
   run modern operating systems.

 * Added a new patch "RFC: Acceptance tests: add the build directory
   to the system PATH", which is supposed to gather feedback on how to
   enable the use of built binaries, such as qemu-img, to code used by
   the test code.  The specific situation here is that the vmimage,
   part of the avocado.utils libraries, makes use of qemu-img to create
   snapshot files.  Even though we could require qemu-img to be installed
   as a dependency of tests, system wide, it actually goes against the
   goal of testing all QEMU things from the source/build tree.  This
   became aparent with tests running on environments such as Travis CI,
   which don't necessarily have qemu-img available elsewhere.

Cleber Rosa (3):
  Acceptance tests: introduce BUILD_DIR and SOURCE_DIR
  Acceptance test: add "boot_linux" tests
  Acceptance tests: add make targets to download images

 .travis.yml                               |   2 +-
 tests/Makefile.include                    |  19 +-
 tests/acceptance/avocado_qemu/__init__.py |  25 ++-
 tests/acceptance/boot_linux.py            | 215 ++++++++++++++++++++++
 tests/requirements.txt                    |   3 +-
 5 files changed, 254 insertions(+), 10 deletions(-)
 create mode 100644 tests/acceptance/boot_linux.py

-- 
2.21.1

[PATCH v9 1/3] Acceptance tests: introduce BUILD_DIR and SOURCE_DIR

[Cleber Rosa]

Some tests may benefit from using resources from a build directory.
This introduces three variables that can help tests find resources in
those directories.

First, a BUILD_DIR is assumed to exist, given that the primary form of
running the acceptance tests is from a build directory (which may or
may not be the same as the source tree, that is, the SOURCE_DIR).

If the directory containing the acceptance tests happens to be a link
to a directory, it's assumed to it points to the source tree
(SOURCE_DIR), which is the behavior defined on the QEMU Makefiles.  If
the directory containing the acceptance tests is not a link, then a
in-tree build is assumed, and the BUILD_DIR and SOURCE_DIR have the
same value.

Signed-off-by: Cleber Rosa <crosa@redhat.com>
---
 tests/acceptance/avocado_qemu/__init__.py | 25 +++++++++++++++++------
 1 file changed, 19 insertions(+), 6 deletions(-)

diff --git a/tests/acceptance/avocado_qemu/__init__.py b/tests/acceptance/avocado_qemu/__init__.py
index d4358eb431..59e7b4f763 100644
--- a/tests/acceptance/avocado_qemu/__init__.py
+++ b/tests/acceptance/avocado_qemu/__init__.py
@@ -16,8 +16,21 @@ import tempfile
 
 import avocado
 
-SRC_ROOT_DIR = os.path.join(os.path.dirname(__file__), '..', '..', '..')
-sys.path.append(os.path.join(SRC_ROOT_DIR, 'python'))
+#: The QEMU build root directory.  It may also be the source directory
+#: if building from the source dir, but it's safer to use BUILD_DIR for
+#: that purpose.  Be aware that if this code is moved outside of a source
+#: and build tree, it will not be accurate.
+BUILD_DIR = os.path.dirname(os.path.dirname(os.path.dirname(os.path.dirname(__file__))))
+
+if os.path.islink(os.path.dirname(os.path.dirname(__file__))):
+    # The link to the acceptance tests dir in the source code directory
+    lnk = os.path.dirname(os.path.dirname(__file__))
+    #: The QEMU root source directory
+    SOURCE_DIR = os.path.dirname(os.path.dirname(os.readlink(lnk)))
+else:
+    SOURCE_DIR = BUILD_DIR
+
+sys.path.append(os.path.join(SOURCE_DIR, 'python'))
 
 from qemu.machine import QEMUMachine
 
@@ -49,10 +62,10 @@ def pick_default_qemu_bin(arch=None):
     if is_readable_executable_file(qemu_bin_relative_path):
         return qemu_bin_relative_path
 
-    qemu_bin_from_src_dir_path = os.path.join(SRC_ROOT_DIR,
+    qemu_bin_from_bld_dir_path = os.path.join(BUILD_DIR,
                                               qemu_bin_relative_path)
-    if is_readable_executable_file(qemu_bin_from_src_dir_path):
-        return qemu_bin_from_src_dir_path
+    if is_readable_executable_file(qemu_bin_from_bld_dir_path):
+        return qemu_bin_from_bld_dir_path
 
 
 def _console_interaction(test, success_message, failure_message,
@@ -153,7 +166,7 @@ class Test(avocado.Test):
         self.qemu_bin = self.params.get('qemu_bin',
                                         default=default_qemu_bin)
         if self.qemu_bin is None:
-            self.cancel("No QEMU binary defined or found in the source tree")
+            self.cancel("No QEMU binary defined or found in the build tree")
 
     def _new_vm(self, *args):
         vm = QEMUMachine(self.qemu_bin, sock_dir=tempfile.mkdtemp())
-- 
2.21.1

Re: [PATCH v9 1/3] Acceptance tests: introduce BUILD_DIR and SOURCE_DIR

[Wainer dos Santos Moschetta]

On 2/19/20 11:06 PM, Cleber Rosa wrote:
> Some tests may benefit from using resources from a build directory.
> This introduces three variables that can help tests find resources in
> those directories.
>
> First, a BUILD_DIR is assumed to exist, given that the primary form of
> running the acceptance tests is from a build directory (which may or
> may not be the same as the source tree, that is, the SOURCE_DIR).
>
> If the directory containing the acceptance tests happens to be a link
> to a directory, it's assumed to it points to the source tree
> (SOURCE_DIR), which is the behavior defined on the QEMU Makefiles.  If
> the directory containing the acceptance tests is not a link, then a
> in-tree build is assumed, and the BUILD_DIR and SOURCE_DIR have the
> same value.
>
> Signed-off-by: Cleber Rosa <crosa@redhat.com>
> ---
>   tests/acceptance/avocado_qemu/__init__.py | 25 +++++++++++++++++------
>   1 file changed, 19 insertions(+), 6 deletions(-)


Reviewed-by: Wainer dos Santos Moschetta <wainersm@redhat.com>

Also tested with QEMU built outside of source dir:

Tested-by: Wainer dos Santos Moschetta <wainersm@redhat.com>

>
> diff --git a/tests/acceptance/avocado_qemu/__init__.py b/tests/acceptance/avocado_qemu/__init__.py
> index d4358eb431..59e7b4f763 100644
> --- a/tests/acceptance/avocado_qemu/__init__.py
> +++ b/tests/acceptance/avocado_qemu/__init__.py
> @@ -16,8 +16,21 @@ import tempfile
>   
>   import avocado
>   
> -SRC_ROOT_DIR = os.path.join(os.path.dirname(__file__), '..', '..', '..')
> -sys.path.append(os.path.join(SRC_ROOT_DIR, 'python'))
> +#: The QEMU build root directory.  It may also be the source directory
> +#: if building from the source dir, but it's safer to use BUILD_DIR for
> +#: that purpose.  Be aware that if this code is moved outside of a source
> +#: and build tree, it will not be accurate.
> +BUILD_DIR = os.path.dirname(os.path.dirname(os.path.dirname(os.path.dirname(__file__))))
> +
> +if os.path.islink(os.path.dirname(os.path.dirname(__file__))):
> +    # The link to the acceptance tests dir in the source code directory
> +    lnk = os.path.dirname(os.path.dirname(__file__))
> +    #: The QEMU root source directory
> +    SOURCE_DIR = os.path.dirname(os.path.dirname(os.readlink(lnk)))
> +else:
> +    SOURCE_DIR = BUILD_DIR
> +
> +sys.path.append(os.path.join(SOURCE_DIR, 'python'))
>   
>   from qemu.machine import QEMUMachine
>   
> @@ -49,10 +62,10 @@ def pick_default_qemu_bin(arch=None):
>       if is_readable_executable_file(qemu_bin_relative_path):
>           return qemu_bin_relative_path
>   
> -    qemu_bin_from_src_dir_path = os.path.join(SRC_ROOT_DIR,
> +    qemu_bin_from_bld_dir_path = os.path.join(BUILD_DIR,
>                                                 qemu_bin_relative_path)
> -    if is_readable_executable_file(qemu_bin_from_src_dir_path):
> -        return qemu_bin_from_src_dir_path
> +    if is_readable_executable_file(qemu_bin_from_bld_dir_path):
> +        return qemu_bin_from_bld_dir_path
>   
>   
>   def _console_interaction(test, success_message, failure_message,
> @@ -153,7 +166,7 @@ class Test(avocado.Test):
>           self.qemu_bin = self.params.get('qemu_bin',
>                                           default=default_qemu_bin)
>           if self.qemu_bin is None:
> -            self.cancel("No QEMU binary defined or found in the source tree")
> +            self.cancel("No QEMU binary defined or found in the build tree")
>   
>       def _new_vm(self, *args):
>           vm = QEMUMachine(self.qemu_bin, sock_dir=tempfile.mkdtemp())

[PATCH v9 2/3] Acceptance test: add "boot_linux" tests

[Cleber Rosa]

This acceptance test, validates that a full blown Linux guest can
successfully boot in QEMU.  In this specific case, the guest chosen is
Fedora version 31.

 * x86_64, pc-i440fx and pc-q35 machine types, with TCG and KVM as
   accelerators

 * aarch64 and virt machine type, with TCG and KVM as accelerators

 * ppc64 and pseries machine type with TCG as accelerator

 * s390x and s390-ccw-virtio machine type with TCG as accelerator

The Avocado vmimage utils library is used to download and cache the
Linux guest images, and from those images a snapshot image is created
and given to QEMU.  If a qemu-img binary is available in the build
directory, it's used to create the snapshot image, so that matching
qemu-system-* and qemu-img are used in the same test run.  If qemu-img
is not available in the build tree, one is attempted to be found
installed system-wide (in the $PATH).  If qemu-img is not found in the
build dir or in the $PATH, the test is canceled.

The method for checking the successful boot is based on "cloudinit"
and its "phone home" feature.  The guest is given an ISO image with
the location of the phone home server, and the information to post
(the instance ID).  Upon receiving the correct information, from the
guest, the test is considered to have PASSed.

This test is currently limited to user mode networking only, and
instructs the guest to connect to the "router" address that is hard
coded in QEMU.

To create the cloudinit ISO image that will be used to configure the
guest, the pycdlib library is also required and has been added as
requirement to the virtual environment created by "check-venv".

The console output is read by a separate thread, by means of the
Avocado datadrainer utility module.

Signed-off-by: Cleber Rosa <crosa@redhat.com>
---
 .travis.yml                    |   2 +-
 tests/acceptance/boot_linux.py | 215 +++++++++++++++++++++++++++++++++
 tests/requirements.txt         |   3 +-
 3 files changed, 218 insertions(+), 2 deletions(-)
 create mode 100644 tests/acceptance/boot_linux.py

diff --git a/.travis.yml b/.travis.yml
index 5887055951..0c54cdf40f 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -313,7 +313,7 @@ matrix:
     # Acceptance (Functional) tests
     - name: "GCC check-acceptance"
       env:
-        - CONFIG="--target-list=aarch64-softmmu,alpha-softmmu,arm-softmmu,m68k-softmmu,microblaze-softmmu,mips-softmmu,mips64el-softmmu,nios2-softmmu,or1k-softmmu,ppc-softmmu,ppc64-softmmu,s390x-softmmu,sparc-softmmu,x86_64-softmmu,xtensa-softmmu"
+        - CONFIG="--enable-tools --target-list=aarch64-softmmu,alpha-softmmu,arm-softmmu,m68k-softmmu,microblaze-softmmu,mips-softmmu,mips64el-softmmu,nios2-softmmu,or1k-softmmu,ppc-softmmu,ppc64-softmmu,s390x-softmmu,sparc-softmmu,x86_64-softmmu,xtensa-softmmu"
         - TEST_CMD="make check-acceptance"
       after_script:
         - python3 -c 'import json; r = json.load(open("tests/results/latest/results.json")); [print(t["logfile"]) for t in r["tests"] if t["status"] not in ("PASS", "SKIP")]' | xargs cat
diff --git a/tests/acceptance/boot_linux.py b/tests/acceptance/boot_linux.py
new file mode 100644
index 0000000000..6787e79aea
--- /dev/null
+++ b/tests/acceptance/boot_linux.py
@@ -0,0 +1,215 @@
+# Functional test that boots a complete Linux system via a cloud image
+#
+# Copyright (c) 2018-2020 Red Hat, Inc.
+#
+# Author:
+#  Cleber Rosa <crosa@redhat.com>
+#
+# This work is licensed under the terms of the GNU GPL, version 2 or
+# later.  See the COPYING file in the top-level directory.
+
+import os
+
+from avocado_qemu import Test, BUILD_DIR
+
+from qemu.accel import kvm_available
+from qemu.accel import tcg_available
+
+from avocado.utils import cloudinit
+from avocado.utils import network
+from avocado.utils import vmimage
+from avocado.utils import datadrainer
+
+ACCEL_NOT_AVAILABLE_FMT = "%s accelerator does not seem to be available"
+KVM_NOT_AVAILABLE = ACCEL_NOT_AVAILABLE_FMT % "KVM"
+TCG_NOT_AVAILABLE = ACCEL_NOT_AVAILABLE_FMT % "TCG"
+
+
+class BootLinux(Test):
+    """
+    Boots a Linux system, checking for a successful initialization
+    """
+
+    timeout = 900
+    chksum = None
+
+    def setUp(self):
+        super(BootLinux, self).setUp()
+        self.prepare_boot()
+        self.vm.add_args('-smp', '2')
+        self.vm.add_args('-m', '1024')
+        self.vm.add_args('-drive', 'file=%s' % self.boot.path)
+        self.prepare_cloudinit()
+
+    def prepare_boot(self):
+        self.log.info('Downloading/preparing boot image')
+        # Fedora 31 only provides ppc64le images
+        image_arch = self.arch
+        if image_arch == 'ppc64':
+            image_arch = 'ppc64le'
+        # If qemu-img has been built, use it, otherwise the system wide one
+        # will be used.  If none is available, the test will cancel.
+        qemu_img = os.path.join(BUILD_DIR, 'qemu-img')
+        if os.path.exists(qemu_img):
+            vmimage.QEMU_IMG = qemu_img
+        try:
+            self.boot = vmimage.get(
+                'fedora', arch=image_arch, version='31',
+                checksum=self.chksum,
+                algorithm='sha256',
+                cache_dir=self.cache_dirs[0],
+                snapshot_dir=self.workdir)
+        except:
+            self.cancel('Failed to download/prepare boot image')
+
+    def prepare_cloudinit(self):
+        self.log.info('Preparing cloudinit image')
+        try:
+            cloudinit_iso = os.path.join(self.workdir, 'cloudinit.iso')
+            self.phone_home_port = network.find_free_port()
+            cloudinit.iso(cloudinit_iso, self.name,
+                          username='root',
+                          password='password',
+                          # QEMU's hard coded usermode router address
+                          phone_home_host='10.0.2.2',
+                          phone_home_port=self.phone_home_port)
+            self.vm.add_args('-drive', 'file=%s,format=raw' % cloudinit_iso)
+        except Exception:
+            self.cancel('Failed to prepared cloudinit image')
+
+    def launch_and_wait(self):
+        self.vm.set_console()
+        self.vm.launch()
+        console_drainer = datadrainer.LineLogger(self.vm.console_socket.fileno(),
+                                                 logger=self.log.getChild('console'))
+        console_drainer.start()
+        self.log.info('VM launched, waiting for boot confirmation from guest')
+        cloudinit.wait_for_phone_home(('0.0.0.0', self.phone_home_port), self.name)
+
+
+class BootLinuxX8664(BootLinux):
+    """
+    :avocado: tags=arch:x86_64
+    """
+
+    chksum = 'e3c1b309d9203604922d6e255c2c5d098a309c2d46215d8fc026954f3c5c27a0'
+
+    def test_pc_i440fx_tcg(self):
+        """
+        :avocado: tags=machine:pc
+        :avocado: tags=machine:pc-i440fx
+        :avocado: tags=accel:tcg
+        """
+        if not tcg_available(self.qemu_bin):
+            self.cancel(TCG_NOT_AVAILABLE)
+        self.vm.add_args("-accel", "tcg")
+        self.launch_and_wait()
+
+    def test_pc_i440fx_kvm(self):
+        """
+        :avocado: tags=machine:pc
+        :avocado: tags=machine:pc-i440fx
+        :avocado: tags=accel:kvm
+        """
+        if not kvm_available(self.arch, self.qemu_bin):
+            self.cancel(KVM_NOT_AVAILABLE)
+        self.vm.add_args("-accel", "kvm")
+        self.launch_and_wait()
+
+    def test_pc_q35_tcg(self):
+        """
+        :avocado: tags=machine:pc
+        :avocado: tags=machine:pc-q35
+        :avocado: tags=accel:tcg
+        """
+        if not tcg_available(self.qemu_bin):
+            self.cancel(TCG_NOT_AVAILABLE)
+        self.vm.add_args("-accel", "tcg")
+        self.launch_and_wait()
+
+    def test_pc_q35_kvm(self):
+        """
+        :avocado: tags=machine:q35
+        :avocado: tags=machine:pc-q35
+        :avocado: tags=accel:kvm
+        """
+        if not kvm_available(self.arch, self.qemu_bin):
+            self.cancel(KVM_NOT_AVAILABLE)
+        self.vm.add_args("-accel", "kvm")
+        self.launch_and_wait()
+
+
+class BootLinuxAarch64(BootLinux):
+    """
+    :avocado: tags=arch:aarch64
+    :avocado: tags=machine:virt
+    """
+
+    chksum = '1e18d9c0cf734940c4b5d5ec592facaed2af0ad0329383d5639c997fdf16fe49'
+
+    def add_common_args(self):
+        self.vm.add_args('-bios',
+                         os.path.join(BUILD_DIR, 'pc-bios',
+                                      'edk2-aarch64-code.fd'))
+        self.vm.add_args('-device', 'virtio-rng-pci,rng=rng0')
+        self.vm.add_args('-object', 'rng-random,id=rng0,filename=/dev/urandom')
+
+    def test_virt_tcg(self):
+        """
+        :avocado: tags=accel:tcg
+        :avocado: tags=cpu:cortex-a53
+        """
+        if not tcg_available(self.qemu_bin):
+            self.cancel(TCG_NOT_AVAILABLE)
+        self.vm.add_args("-accel", "tcg")
+        self.vm.add_args('-cpu', 'cortex-a53')
+        self.add_common_args()
+        self.launch_and_wait()
+
+    def test_virt_kvm(self):
+        """
+        :avocado: tags=accel:kvm
+        :avocado: tags=cpu:host
+        """
+        if not kvm_available(self.arch, self.qemu_bin):
+            self.cancel(KVM_NOT_AVAILABLE)
+        self.vm.add_args("-accel", "kvm")
+        self.vm.add_args("-cpu", "host")
+        self.add_common_args()
+        self.launch_and_wait()
+
+
+class BootLinuxPPC64(BootLinux):
+    """
+    :avocado: tags=arch:ppc64
+    """
+
+    chksum = '7c3528b85a3df4b2306e892199a9e1e43f991c506f2cc390dc4efa2026ad2f58'
+
+    def test_pseries_tcg(self):
+        """
+        :avocado: tags=machine:pseries
+        :avocado: tags=accel:tcg
+        """
+        if not tcg_available(self.qemu_bin):
+            self.cancel(TCG_NOT_AVAILABLE)
+        self.vm.add_args("-accel", "tcg")
+        self.launch_and_wait()
+
+
+class BootLinuxS390X(BootLinux):
+    """
+    :avocado: tags=arch:s390x
+    """
+
+    chksum = '4caaab5a434fd4d1079149a072fdc7891e354f834d355069ca982fdcaf5a122d'
+
+    def test_s390_ccw_virtio_tcg(self):
+        """
+        :avocado: tags=machine:s390-ccw-virtio
+        :avocado: tags=accel:tcg
+        """
+        if not tcg_available(self.qemu_bin):
+            self.cancel(TCG_NOT_AVAILABLE)
+        self.vm.add_args("-accel", "tcg")
+        self.launch_and_wait()
diff --git a/tests/requirements.txt b/tests/requirements.txt
index a2a587223a..a3b5fe4159 100644
--- a/tests/requirements.txt
+++ b/tests/requirements.txt
@@ -1,4 +1,5 @@
 # Add Python module requirements, one per line, to be installed
 # in the tests/venv Python virtual environment. For more info,
 # refer to: https://pip.pypa.io/en/stable/user_guide/#id1
-avocado-framework==72.0
+avocado-framework==74.0
+pycdlib==1.9.0
-- 
2.21.1

Re: [PATCH v9 2/3] Acceptance test: add "boot_linux" tests

[Wainer dos Santos Moschetta]

On 2/19/20 11:06 PM, Cleber Rosa wrote:
> This acceptance test, validates that a full blown Linux guest can
> successfully boot in QEMU.  In this specific case, the guest chosen is
> Fedora version 31.
>
>   * x86_64, pc-i440fx and pc-q35 machine types, with TCG and KVM as
>     accelerators
>
>   * aarch64 and virt machine type, with TCG and KVM as accelerators
>
>   * ppc64 and pseries machine type with TCG as accelerator
>
>   * s390x and s390-ccw-virtio machine type with TCG as accelerator
>
> The Avocado vmimage utils library is used to download and cache the
> Linux guest images, and from those images a snapshot image is created
> and given to QEMU.  If a qemu-img binary is available in the build
> directory, it's used to create the snapshot image, so that matching
> qemu-system-* and qemu-img are used in the same test run.  If qemu-img
> is not available in the build tree, one is attempted to be found
> installed system-wide (in the $PATH).  If qemu-img is not found in the
> build dir or in the $PATH, the test is canceled.
>
> The method for checking the successful boot is based on "cloudinit"
> and its "phone home" feature.  The guest is given an ISO image with
> the location of the phone home server, and the information to post
> (the instance ID).  Upon receiving the correct information, from the
> guest, the test is considered to have PASSed.
>
> This test is currently limited to user mode networking only, and
> instructs the guest to connect to the "router" address that is hard
> coded in QEMU.
>
> To create the cloudinit ISO image that will be used to configure the
> guest, the pycdlib library is also required and has been added as
> requirement to the virtual environment created by "check-venv".
>
> The console output is read by a separate thread, by means of the
> Avocado datadrainer utility module.
>
> Signed-off-by: Cleber Rosa <crosa@redhat.com>
> ---
>   .travis.yml                    |   2 +-
>   tests/acceptance/boot_linux.py | 215 +++++++++++++++++++++++++++++++++
>   tests/requirements.txt         |   3 +-
>   3 files changed, 218 insertions(+), 2 deletions(-)
>   create mode 100644 tests/acceptance/boot_linux.py
>
> diff --git a/.travis.yml b/.travis.yml
> index 5887055951..0c54cdf40f 100644
> --- a/.travis.yml
> +++ b/.travis.yml
> @@ -313,7 +313,7 @@ matrix:
>       # Acceptance (Functional) tests
>       - name: "GCC check-acceptance"
>         env:
> -        - CONFIG="--target-list=aarch64-softmmu,alpha-softmmu,arm-softmmu,m68k-softmmu,microblaze-softmmu,mips-softmmu,mips64el-softmmu,nios2-softmmu,or1k-softmmu,ppc-softmmu,ppc64-softmmu,s390x-softmmu,sparc-softmmu,x86_64-softmmu,xtensa-softmmu"
> +        - CONFIG="--enable-tools --target-list=aarch64-softmmu,alpha-softmmu,arm-softmmu,m68k-softmmu,microblaze-softmmu,mips-softmmu,mips64el-softmmu,nios2-softmmu,or1k-softmmu,ppc-softmmu,ppc64-softmmu,s390x-softmmu,sparc-softmmu,x86_64-softmmu,xtensa-softmmu"
>           - TEST_CMD="make check-acceptance"
>         after_script:
>           - python3 -c 'import json; r = json.load(open("tests/results/latest/results.json")); [print(t["logfile"]) for t in r["tests"] if t["status"] not in ("PASS", "SKIP")]' | xargs cat
> diff --git a/tests/acceptance/boot_linux.py b/tests/acceptance/boot_linux.py
> new file mode 100644
> index 0000000000..6787e79aea
> --- /dev/null
> +++ b/tests/acceptance/boot_linux.py
> @@ -0,0 +1,215 @@
> +# Functional test that boots a complete Linux system via a cloud image
> +#
> +# Copyright (c) 2018-2020 Red Hat, Inc.
> +#
> +# Author:
> +#  Cleber Rosa <crosa@redhat.com>
> +#
> +# This work is licensed under the terms of the GNU GPL, version 2 or
> +# later.  See the COPYING file in the top-level directory.
> +
> +import os
> +
> +from avocado_qemu import Test, BUILD_DIR
> +
> +from qemu.accel import kvm_available
> +from qemu.accel import tcg_available
> +
> +from avocado.utils import cloudinit
> +from avocado.utils import network
> +from avocado.utils import vmimage
> +from avocado.utils import datadrainer
> +
> +ACCEL_NOT_AVAILABLE_FMT = "%s accelerator does not seem to be available"
> +KVM_NOT_AVAILABLE = ACCEL_NOT_AVAILABLE_FMT % "KVM"
> +TCG_NOT_AVAILABLE = ACCEL_NOT_AVAILABLE_FMT % "TCG"
> +
> +
> +class BootLinux(Test):
> +    """
> +    Boots a Linux system, checking for a successful initialization
> +    """
> +
> +    timeout = 900
> +    chksum = None
> +
> +    def setUp(self):
> +        super(BootLinux, self).setUp()
> +        self.prepare_boot()
> +        self.vm.add_args('-smp', '2')
> +        self.vm.add_args('-m', '1024')
> +        self.vm.add_args('-drive', 'file=%s' % self.boot.path)
> +        self.prepare_cloudinit()
> +
> +    def prepare_boot(self):
> +        self.log.info('Downloading/preparing boot image')
> +        # Fedora 31 only provides ppc64le images
> +        image_arch = self.arch
> +        if image_arch == 'ppc64':
> +            image_arch = 'ppc64le'
> +        # If qemu-img has been built, use it, otherwise the system wide one
> +        # will be used.  If none is available, the test will cancel.
> +        qemu_img = os.path.join(BUILD_DIR, 'qemu-img')
> +        if os.path.exists(qemu_img):
> +            vmimage.QEMU_IMG = qemu_img
> +        try:
> +            self.boot = vmimage.get(
> +                'fedora', arch=image_arch, version='31',
> +                checksum=self.chksum,
> +                algorithm='sha256',
> +                cache_dir=self.cache_dirs[0],
> +                snapshot_dir=self.workdir)
> +        except:
> +            self.cancel('Failed to download/prepare boot image')
> +
> +    def prepare_cloudinit(self):
> +        self.log.info('Preparing cloudinit image')
> +        try:
> +            cloudinit_iso = os.path.join(self.workdir, 'cloudinit.iso')
> +            self.phone_home_port = network.find_free_port()
> +            cloudinit.iso(cloudinit_iso, self.name,
> +                          username='root',
> +                          password='password',
> +                          # QEMU's hard coded usermode router address
> +                          phone_home_host='10.0.2.2',
> +                          phone_home_port=self.phone_home_port)
> +            self.vm.add_args('-drive', 'file=%s,format=raw' % cloudinit_iso)
> +        except Exception:
> +            self.cancel('Failed to prepared cloudinit image')
> +
> +    def launch_and_wait(self):
> +        self.vm.set_console()
> +        self.vm.launch()
> +        console_drainer = datadrainer.LineLogger(self.vm.console_socket.fileno(),
> +                                                 logger=self.log.getChild('console'))
> +        console_drainer.start()
> +        self.log.info('VM launched, waiting for boot confirmation from guest')
> +        cloudinit.wait_for_phone_home(('0.0.0.0', self.phone_home_port), self.name)
> +
> +
> +class BootLinuxX8664(BootLinux):
> +    """
> +    :avocado: tags=arch:x86_64
> +    """
> +
> +    chksum = 'e3c1b309d9203604922d6e255c2c5d098a309c2d46215d8fc026954f3c5c27a0'
> +
> +    def test_pc_i440fx_tcg(self):
> +        """
> +        :avocado: tags=machine:pc
> +        :avocado: tags=machine:pc-i440fx
> +        :avocado: tags=accel:tcg
> +        """
> +        if not tcg_available(self.qemu_bin):
> +            self.cancel(TCG_NOT_AVAILABLE)
> +        self.vm.add_args("-accel", "tcg")
> +        self.launch_and_wait()
> +
> +    def test_pc_i440fx_kvm(self):
> +        """
> +        :avocado: tags=machine:pc
> +        :avocado: tags=machine:pc-i440fx
> +        :avocado: tags=accel:kvm
> +        """
> +        if not kvm_available(self.arch, self.qemu_bin):
> +            self.cancel(KVM_NOT_AVAILABLE)
> +        self.vm.add_args("-accel", "kvm")
> +        self.launch_and_wait()
> +
> +    def test_pc_q35_tcg(self):
> +        """
> +        :avocado: tags=machine:pc
> +        :avocado: tags=machine:pc-q35
> +        :avocado: tags=accel:tcg
> +        """
> +        if not tcg_available(self.qemu_bin):
> +            self.cancel(TCG_NOT_AVAILABLE)
> +        self.vm.add_args("-accel", "tcg")
> +        self.launch_and_wait()
> +
> +    def test_pc_q35_kvm(self):
> +        """
> +        :avocado: tags=machine:q35
> +        :avocado: tags=machine:pc-q35
> +        :avocado: tags=accel:kvm
> +        """
> +        if not kvm_available(self.arch, self.qemu_bin):
> +            self.cancel(KVM_NOT_AVAILABLE)
> +        self.vm.add_args("-accel", "kvm")
> +        self.launch_and_wait()
> +
> +
> +class BootLinuxAarch64(BootLinux):
> +    """
> +    :avocado: tags=arch:aarch64
> +    :avocado: tags=machine:virt
> +    """
> +
> +    chksum = '1e18d9c0cf734940c4b5d5ec592facaed2af0ad0329383d5639c997fdf16fe49'
> +
> +    def add_common_args(self):
> +        self.vm.add_args('-bios',
> +                         os.path.join(BUILD_DIR, 'pc-bios',
> +                                      'edk2-aarch64-code.fd'))
> +        self.vm.add_args('-device', 'virtio-rng-pci,rng=rng0')
> +        self.vm.add_args('-object', 'rng-random,id=rng0,filename=/dev/urandom')
> +
> +    def test_virt_tcg(self):
> +        """
> +        :avocado: tags=accel:tcg
> +        :avocado: tags=cpu:cortex-a53
> +        """
> +        if not tcg_available(self.qemu_bin):
> +            self.cancel(TCG_NOT_AVAILABLE)
> +        self.vm.add_args("-accel", "tcg")
> +        self.vm.add_args('-cpu', 'cortex-a53')
> +        self.add_common_args()
> +        self.launch_and_wait()
> +
> +    def test_virt_kvm(self):
> +        """
> +        :avocado: tags=accel:kvm
> +        :avocado: tags=cpu:host
> +        """
> +        if not kvm_available(self.arch, self.qemu_bin):
> +            self.cancel(KVM_NOT_AVAILABLE)
> +        self.vm.add_args("-accel", "kvm")
> +        self.vm.add_args("-cpu", "host")
> +        self.add_common_args()
> +        self.launch_and_wait()


For aarch64 tests it seems '-cpu max' is the best choice. See in 
https://www.mail-archive.com/qemu-devel@nongnu.org/msg672755.html


> +
> +
> +class BootLinuxPPC64(BootLinux):
> +    """
> +    :avocado: tags=arch:ppc64
> +    """
> +
> +    chksum = '7c3528b85a3df4b2306e892199a9e1e43f991c506f2cc390dc4efa2026ad2f58'
> +
> +    def test_pseries_tcg(self):
> +        """
> +        :avocado: tags=machine:pseries
> +        :avocado: tags=accel:tcg
> +        """
> +        if not tcg_available(self.qemu_bin):
> +            self.cancel(TCG_NOT_AVAILABLE)
> +        self.vm.add_args("-accel", "tcg")
> +        self.launch_and_wait()
> +
> +
> +class BootLinuxS390X(BootLinux):
> +    """
> +    :avocado: tags=arch:s390x
> +    """
> +
> +    chksum = '4caaab5a434fd4d1079149a072fdc7891e354f834d355069ca982fdcaf5a122d'
> +
> +    def test_s390_ccw_virtio_tcg(self):
> +        """
> +        :avocado: tags=machine:s390-ccw-virtio
> +        :avocado: tags=accel:tcg
> +        """
> +        if not tcg_available(self.qemu_bin):
> +            self.cancel(TCG_NOT_AVAILABLE)
> +        self.vm.add_args("-accel", "tcg")
> +        self.launch_and_wait()
> diff --git a/tests/requirements.txt b/tests/requirements.txt
> index a2a587223a..a3b5fe4159 100644
> --- a/tests/requirements.txt
> +++ b/tests/requirements.txt
> @@ -1,4 +1,5 @@
>   # Add Python module requirements, one per line, to be installed
>   # in the tests/venv Python virtual environment. For more info,
>   # refer to: https://pip.pypa.io/en/stable/user_guide/#id1
> -avocado-framework==72.0
> +avocado-framework==74.0
> +pycdlib==1.9.0


Tested on x86_64 machine, the tests behave correctly with following 
configurations:

1. ---target-list=x86_64-softmmu --disable-tcg

2. ---target-list=x86_64-softmmu --disable-kvm

3. --target-list=x86_64-softmmu,aarch64-softmmu,ppc64-softmmu,s390x-softmmu

But failed if:

3. ---target-list=x86_64-softmmu --disable-tools.

And the error message is:

(01/32) 
tests/acceptance/boot_linux.py:BootLinuxX8664.test_pc_i440fx_tcg: ERROR: 
Command 'qemu-img' could not be found in any of the PATH dirs: 
['/usr/bin', '/usr/sbin', '/usr/lib64/ccache', '/bin', '/root/bin', 
'/sbin', '/usr/local/sbin', '/usr/local/bin', '/usr/libexec'] (1.58 s)

Thanks!

- Wainer

Re: [PATCH v9 2/3] Acceptance test: add "boot_linux" tests

[Cleber Rosa]

[-- Attachment #1: Type: text/plain, Size: 7897 bytes --]

On Thu, Feb 20, 2020 at 01:49:40PM -0300, Wainer dos Santos Moschetta wrote:
> On 2/19/20 11:06 PM, Cleber Rosa wrote:
> > +
> > +    def test_virt_tcg(self):
> > +        """
> > +        :avocado: tags=accel:tcg
> > +        :avocado: tags=cpu:cortex-a53
> > +        """
> > +        if not tcg_available(self.qemu_bin):
> > +            self.cancel(TCG_NOT_AVAILABLE)
> > +        self.vm.add_args("-accel", "tcg")
> > +        self.vm.add_args('-cpu', 'cortex-a53')
> > +        self.add_common_args()
> > +        self.launch_and_wait()
> > +
> > +    def test_virt_kvm(self):
> > +        """
> > +        :avocado: tags=accel:kvm
> > +        :avocado: tags=cpu:host
> > +        """
> > +        if not kvm_available(self.arch, self.qemu_bin):
> > +            self.cancel(KVM_NOT_AVAILABLE)
> > +        self.vm.add_args("-accel", "kvm")
> > +        self.vm.add_args("-cpu", "host")
> > +        self.add_common_args()
> > +        self.launch_and_wait()
> 
> 
> For aarch64 tests it seems '-cpu max' is the best choice. See in
> https://www.mail-archive.com/qemu-devel@nongnu.org/msg672755.html
> 
>

+drew

Thanks for pointing that out.  There's one thing, though, which I can
not agree on.  And I know that Drew is an expert on the matter, which
makes it harder to disagree on... but, I've got results which clearly
indicate that *not using* the gic-version machine parameter still gets
me KVM:

./tests/venv/bin/avocado run tests/acceptance/boot_linux.py:BootLinuxAarch64.test_virt_kvm
JOB ID     : 21a394b884b474ceee0a045b3e74f98da0aee023
JOB LOG    : /home/cleber/avocado/job-results/job-2020-02-20T14.28-21a394b/job.log
 (1/1) tests/acceptance/boot_linux.py:BootLinuxAarch64.test_virt_kvm: PASS (35.10 s)
RESULTS    : PASS 1 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0
JOB TIME   : 35.87 s

VM launch command:
   aarch64-softmmu/qemu-system-aarch64
   -display none
   -vga none
   -chardev socket,id=mon,path=/var/tmp/tmpntz_r_h7/qemu-18331-monitor.sock
   -mon chardev=mon,mode=control
   -machine virt
   -chardev socket,id=console,path=/var/tmp/tmpntz_r_h7/qemu-18331-console.sock,server,nowait
   -serial chardev:console
   -smp 2
   -m 1024
   -drive file=/var/tmp/avocado_u9jm04di/avocado_job_28oth9kk/1-tests_acceptance_boot_linux.py_BootLinuxAarch64.test_virt_kvm/Fedora-Cloud-Base-31-1.9.aarch64-05265df5.qcow2 -drive file=/var/tmp/avocado_u9jm04di/avocado_job_28oth9kk/1-tests_acceptance_boot_linux.py_BootLinuxAarch64.test_virt_kvm/cloudinit.iso,format=raw
   -accel kvm
   -cpu host
   -bios /home/cleber/build/qemu/pc-bios/edk2-aarch64-code.fd
   -device virtio-rng-pci,rng=rng0
   -object rng-random,id=rng0,filename=/dev/urandom

Guest boot messages shows:
[    1.538955] systemd[1]: Detected virtualization kvm.
[    1.539828] systemd[1]: Detected architecture arm64.

This is in contrast with:

./tests/venv/bin/avocado run tests/acceptance/boot_linux.py:BootLinuxAarch64.test_virt_tcg 
JOB ID     : 90b9412f700e52428b59e97719496c30b4f54435
JOB LOG    : /home/cleber/avocado/job-results/job-2020-02-20T14.32-90b9412/job.log
 (1/1) tests/acceptance/boot_linux.py:BootLinuxAarch64.test_virt_tcg: PASS (581.14 s)
RESULTS    : PASS 1 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0
JOB TIME   : 581.93 s

VM launch command:
   aarch64-softmmu/qemu-system-aarch64
   -display none
   -vga none
   -chardev socket,id=mon,path=/var/tmp/tmpa6i4livg/qemu-18498-monitor.sock
   -mon chardev=mon,mode=control
   -machine virt
   -chardev socket,id=console,path=/var/tmp/tmpa6i4livg/qemu-18498-console.sock,server,nowait
   -serial chardev:console
   -smp 2
   -m 1024
   -drive file=/var/tmp/avocado_slcj2x9e/avocado_job_x5u__309/1-tests_acceptance_boot_linux.py_BootLinuxAarch64.test_virt_tcg/Fedora-Cloud-Base-31-1.9.aarch64-5b006a2f.qcow2 -drive file=/var/tmp/avocado_slcj2x9e/avocado_job_x5u__309/1-tests_acceptance_boot_linux.py_BootLinuxAarch64.test_virt_tcg/cloudinit.iso,format=raw
   -accel tcg
   -cpu cortex-a53
   -bios /home/cleber/build/qemu/pc-bios/edk2-aarch64-code.fd
   -device virtio-rng-pci,rng=rng0
   -object rng-random,id=rng0,filename=/dev/urandom'

Guest boot messages shows:
[   28.606310] systemd[1]: Detected virtualization qemu.
[   28.607861] systemd[1]: Detected architecture arm64.

And with regards to the CPU type, IIRC, "max" will fallback to the
best CPU on TCG mode.  As a general best practice in testing, I'd
rather not have this dynamic aspect where we can avoid it.  Looks like
with TCG we can set it to one CPU and validate that the guests work on
that configuration.

IIUC, by using either "-cpu host" or "-cpu max" for KVM, we may end up
having the same test PASS or FAIL because of the (dynamic) host CPU.
That's not ideal for testing purposes, but given it's outside of our
control, do best we can do is keep track of the host CPU (via Avocado's
sysinfo collection).

Also, I've used the same CPU model that has been used on
boot_linux_console.py:BootLinuxConsole.test_aarch64_virt, which may be
a plus.

> > +
> > +
> > +class BootLinuxPPC64(BootLinux):
> > +    """
> > +    :avocado: tags=arch:ppc64
> > +    """
> > +
> > +    chksum = '7c3528b85a3df4b2306e892199a9e1e43f991c506f2cc390dc4efa2026ad2f58'
> > +
> > +    def test_pseries_tcg(self):
> > +        """
> > +        :avocado: tags=machine:pseries
> > +        :avocado: tags=accel:tcg
> > +        """
> > +        if not tcg_available(self.qemu_bin):
> > +            self.cancel(TCG_NOT_AVAILABLE)
> > +        self.vm.add_args("-accel", "tcg")
> > +        self.launch_and_wait()
> > +
> > +
> > +class BootLinuxS390X(BootLinux):
> > +    """
> > +    :avocado: tags=arch:s390x
> > +    """
> > +
> > +    chksum = '4caaab5a434fd4d1079149a072fdc7891e354f834d355069ca982fdcaf5a122d'
> > +
> > +    def test_s390_ccw_virtio_tcg(self):
> > +        """
> > +        :avocado: tags=machine:s390-ccw-virtio
> > +        :avocado: tags=accel:tcg
> > +        """
> > +        if not tcg_available(self.qemu_bin):
> > +            self.cancel(TCG_NOT_AVAILABLE)
> > +        self.vm.add_args("-accel", "tcg")
> > +        self.launch_and_wait()
> > diff --git a/tests/requirements.txt b/tests/requirements.txt
> > index a2a587223a..a3b5fe4159 100644
> > --- a/tests/requirements.txt
> > +++ b/tests/requirements.txt
> > @@ -1,4 +1,5 @@
> >   # Add Python module requirements, one per line, to be installed
> >   # in the tests/venv Python virtual environment. For more info,
> >   # refer to: https://pip.pypa.io/en/stable/user_guide/#id1
> > -avocado-framework==72.0
> > +avocado-framework==74.0
> > +pycdlib==1.9.0
> 
> 
> Tested on x86_64 machine, the tests behave correctly with following
> configurations:
> 
> 1. ---target-list=x86_64-softmmu --disable-tcg
> 
> 2. ---target-list=x86_64-softmmu --disable-kvm
> 
> 3. --target-list=x86_64-softmmu,aarch64-softmmu,ppc64-softmmu,s390x-softmmu
> 
> But failed if:
> 
> 3. ---target-list=x86_64-softmmu --disable-tools.
> 
> And the error message is:
> 
> (01/32) tests/acceptance/boot_linux.py:BootLinuxX8664.test_pc_i440fx_tcg:
> ERROR: Command 'qemu-img' could not be found in any of the PATH dirs:
> ['/usr/bin', '/usr/sbin', '/usr/lib64/ccache', '/bin', '/root/bin', '/sbin',
> '/usr/local/sbin', '/usr/local/bin', '/usr/libexec'] (1.58 s)

This is what I call comprehensive testing! Thanks!

It looks like I was not paying that much attention to what happens during
the "self.boot.path" attribute access, and had left it "unprotected" in
the QEMU command line arguments assignment.  But there's where a lazy
snapshot image creation is attempted.

I've adjusted the code and will have that on a v10.

> 
> Thanks!
> 
> - Wainer
> 
> 

Thanks a lot for the review!
- Cleber.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH v9 2/3] Acceptance test: add "boot_linux" tests

[Andrew Jones]

On Thu, Feb 20, 2020 at 02:52:45PM -0500, Cleber Rosa wrote:
> On Thu, Feb 20, 2020 at 01:49:40PM -0300, Wainer dos Santos Moschetta wrote:
> > On 2/19/20 11:06 PM, Cleber Rosa wrote:
> > > +
> > > +    def test_virt_tcg(self):
> > > +        """
> > > +        :avocado: tags=accel:tcg
> > > +        :avocado: tags=cpu:cortex-a53
> > > +        """
> > > +        if not tcg_available(self.qemu_bin):
> > > +            self.cancel(TCG_NOT_AVAILABLE)
> > > +        self.vm.add_args("-accel", "tcg")
> > > +        self.vm.add_args('-cpu', 'cortex-a53')

Should use '-cpu max'. gic-version will be '2' by default, which is good
for tcg, but I would probably add an explicit '-machine gic-version=2'
anyway.

> > > +        self.add_common_args()
> > > +        self.launch_and_wait()
> > > +
> > > +    def test_virt_kvm(self):
> > > +        """
> > > +        :avocado: tags=accel:kvm
> > > +        :avocado: tags=cpu:host
> > > +        """
> > > +        if not kvm_available(self.arch, self.qemu_bin):
> > > +            self.cancel(KVM_NOT_AVAILABLE)
> > > +        self.vm.add_args("-accel", "kvm")
> > > +        self.vm.add_args("-cpu", "host")

kvm needs '-machine gic-version=max' and could use '-cpu max' too,
because, for kvm, CPU::max == CPU::host

> > > +        self.add_common_args()
> > > +        self.launch_and_wait()
> > 
> > 
> > For aarch64 tests it seems '-cpu max' is the best choice. See in
> > https://www.mail-archive.com/qemu-devel@nongnu.org/msg672755.html
> > 
> >
> 
> +drew
> 
> Thanks for pointing that out.  There's one thing, though, which I can
> not agree on.  And I know that Drew is an expert on the matter, which
> makes it harder to disagree on... but, I've got results which clearly
> indicate that *not using* the gic-version machine parameter still gets
> me KVM:

The gic-version machine property is completely independent of whether kvm
or tcg is selected. However, if you select kvm and a gic-version that is
incompatible with the host then the guest will not start. If gic-version
is not specified it defaults to '2'. Below the kvm guest started
successfully because it happened to be started on a host with gicv2 or
on a host with gicv3 that supports gicv2-on-v3 (which is an optional
feature that doesn't appear to be getting implemented in modern AArch64
servers). The test below would have failed to start the guest on a host
with only gicv3.

When kvm is in use, one must use gic-version=max in order to automatically
select the latest host-compatible gic version, or the guest will not start
on all hosts.

> 
> ./tests/venv/bin/avocado run tests/acceptance/boot_linux.py:BootLinuxAarch64.test_virt_kvm
> JOB ID     : 21a394b884b474ceee0a045b3e74f98da0aee023
> JOB LOG    : /home/cleber/avocado/job-results/job-2020-02-20T14.28-21a394b/job.log
>  (1/1) tests/acceptance/boot_linux.py:BootLinuxAarch64.test_virt_kvm: PASS (35.10 s)
> RESULTS    : PASS 1 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0
> JOB TIME   : 35.87 s
> 
> VM launch command:
>    aarch64-softmmu/qemu-system-aarch64
>    -display none
>    -vga none
>    -chardev socket,id=mon,path=/var/tmp/tmpntz_r_h7/qemu-18331-monitor.sock
>    -mon chardev=mon,mode=control
>    -machine virt
>    -chardev socket,id=console,path=/var/tmp/tmpntz_r_h7/qemu-18331-console.sock,server,nowait
>    -serial chardev:console
>    -smp 2
>    -m 1024
>    -drive file=/var/tmp/avocado_u9jm04di/avocado_job_28oth9kk/1-tests_acceptance_boot_linux.py_BootLinuxAarch64.test_virt_kvm/Fedora-Cloud-Base-31-1.9.aarch64-05265df5.qcow2 -drive file=/var/tmp/avocado_u9jm04di/avocado_job_28oth9kk/1-tests_acceptance_boot_linux.py_BootLinuxAarch64.test_virt_kvm/cloudinit.iso,format=raw
>    -accel kvm
>    -cpu host
>    -bios /home/cleber/build/qemu/pc-bios/edk2-aarch64-code.fd
>    -device virtio-rng-pci,rng=rng0
>    -object rng-random,id=rng0,filename=/dev/urandom
> 
> Guest boot messages shows:
> [    1.538955] systemd[1]: Detected virtualization kvm.
> [    1.539828] systemd[1]: Detected architecture arm64.
> 
> This is in contrast with:
> 
> ./tests/venv/bin/avocado run tests/acceptance/boot_linux.py:BootLinuxAarch64.test_virt_tcg 
> JOB ID     : 90b9412f700e52428b59e97719496c30b4f54435
> JOB LOG    : /home/cleber/avocado/job-results/job-2020-02-20T14.32-90b9412/job.log
>  (1/1) tests/acceptance/boot_linux.py:BootLinuxAarch64.test_virt_tcg: PASS (581.14 s)
> RESULTS    : PASS 1 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0
> JOB TIME   : 581.93 s
> 
> VM launch command:
>    aarch64-softmmu/qemu-system-aarch64
>    -display none
>    -vga none
>    -chardev socket,id=mon,path=/var/tmp/tmpa6i4livg/qemu-18498-monitor.sock
>    -mon chardev=mon,mode=control
>    -machine virt
>    -chardev socket,id=console,path=/var/tmp/tmpa6i4livg/qemu-18498-console.sock,server,nowait
>    -serial chardev:console
>    -smp 2
>    -m 1024
>    -drive file=/var/tmp/avocado_slcj2x9e/avocado_job_x5u__309/1-tests_acceptance_boot_linux.py_BootLinuxAarch64.test_virt_tcg/Fedora-Cloud-Base-31-1.9.aarch64-5b006a2f.qcow2 -drive file=/var/tmp/avocado_slcj2x9e/avocado_job_x5u__309/1-tests_acceptance_boot_linux.py_BootLinuxAarch64.test_virt_tcg/cloudinit.iso,format=raw
>    -accel tcg
>    -cpu cortex-a53
>    -bios /home/cleber/build/qemu/pc-bios/edk2-aarch64-code.fd
>    -device virtio-rng-pci,rng=rng0
>    -object rng-random,id=rng0,filename=/dev/urandom'
> 
> Guest boot messages shows:
> [   28.606310] systemd[1]: Detected virtualization qemu.
> [   28.607861] systemd[1]: Detected architecture arm64.
> 
> And with regards to the CPU type, IIRC, "max" will fallback to the
> best CPU on TCG mode.

No, it doesn't fallback. It implements the cortex-a57 and enables all
optional CPU features. Why was the cortex-a53 chosen for the tests?

> As a general best practice in testing, I'd
> rather not have this dynamic aspect where we can avoid it.  Looks like
> with TCG we can set it to one CPU and validate that the guests work on
> that configuration.

The only dynamic aspect is that as new CPU features are implemented
they'll get enabled. Personally I'd prefer the tests run with the latest
code enabled in order to find the latest bugs.

> 
> IIUC, by using either "-cpu host" or "-cpu max" for KVM, we may end up
> having the same test PASS or FAIL because of the (dynamic) host CPU.
> That's not ideal for testing purposes, but given it's outside of our
> control, do best we can do is keep track of the host CPU (via Avocado's
> sysinfo collection).

Right. AArch64 KVM must use '-cpu host' (or equivalently '-cpu max'). That
can't be changed. Capturing the host CPU type is a good idea.

> 
> Also, I've used the same CPU model that has been used on
> boot_linux_console.py:BootLinuxConsole.test_aarch64_virt, which may be
> a plus.

Is that the cortex-a53? Was some justification given for its use there?
If not, then maybe it should be changed there too.

Thanks,
drew

[PATCH v9 3/3] Acceptance tests: add make targets to download images

[Cleber Rosa]

The newly introduced "boot linux" tests make use of Linux images that
are larger than usual, and fall into what Avocado calls "vmimages",
and can be referred to by name, version and architecture.

The images can be downloaded automatically during the test. But, to
make for more reliable test results, this introduces a target that
will download the vmimages for the architectures that have been
configured and are available for the currently used distro (Fedora
31).

Signed-off-by: Cleber Rosa <crosa@redhat.com>
---
 tests/Makefile.include | 19 +++++++++++++++++--
 1 file changed, 17 insertions(+), 2 deletions(-)

diff --git a/tests/Makefile.include b/tests/Makefile.include
index 2f1cafed72..3fc6e4f2cc 100644
--- a/tests/Makefile.include
+++ b/tests/Makefile.include
@@ -20,6 +20,8 @@ check-help:
 	@echo " $(MAKE) check-venv           Creates a Python venv for tests"
 	@echo " $(MAKE) check-clean          Clean the tests and related data"
 	@echo
+	@echo " $(MAKE) get-vm-images        Downloads all images used by acceptance tests, according to configured targets (~350 MB each, 1.5 GB max)"
+	@echo
 	@echo
 	@echo "The variable SPEED can be set to control the gtester speed setting."
 	@echo "Default options are -k and (for $(MAKE) V=1) --verbose; they can be"
@@ -886,7 +888,20 @@ $(TESTS_RESULTS_DIR):
 
 check-venv: $(TESTS_VENV_DIR)
 
-check-acceptance: check-venv $(TESTS_RESULTS_DIR)
+FEDORA_31_ARCHES_CANDIDATES=$(patsubst ppc64,ppc64le,$(TARGETS))
+FEDORA_31_ARCHES := x86_64 aarch64 ppc64le s390x
+FEDORA_31_DOWNLOAD=$(filter $(FEDORA_31_ARCHES),$(FEDORA_31_ARCHES_CANDIDATES))
+
+# download one specific Fedora 31 image
+get-vm-image-fedora-31-%: $(check-venv)
+	$(call quiet-command, \
+             $(TESTS_VENV_DIR)/bin/python -m avocado vmimage get \
+             --distro=fedora --distro-version=31 --arch=$*)
+
+# download all vm images, according to defined targets
+get-vm-images: $(check-venv) $(patsubst %,get-vm-image-fedora-31-%, $(FEDORA_31_DOWNLOAD))
+
+check-acceptance: check-venv $(TESTS_RESULTS_DIR) get-vm-images
 	$(call quiet-command, \
             $(TESTS_VENV_DIR)/bin/python -m avocado \
             --show=$(AVOCADO_SHOW) run --job-results-dir=$(TESTS_RESULTS_DIR) \
@@ -897,7 +912,7 @@ check-acceptance: check-venv $(TESTS_RESULTS_DIR)
 
 # Consolidated targets
 
-.PHONY: check-block check-qapi-schema check-qtest check-unit check check-clean
+.PHONY: check-block check-qapi-schema check-qtest check-unit check check-clean get-vm-images
 check-qapi-schema: check-tests/qapi-schema/frontend check-tests/qapi-schema/doc-good.texi
 check-qtest: $(patsubst %,check-qtest-%, $(QTEST_TARGETS))
 ifeq ($(CONFIG_TOOLS),y)
-- 
2.21.1

Re: [PATCH v9 0/3] Acceptance test: Add "boot_linux" acceptance test

[Cleber Rosa]

[-- Attachment #1: Type: text/plain, Size: 325 bytes --]

On Wed, Feb 19, 2020 at 09:06:49PM -0500, Cleber Rosa wrote:
> 
> Changes from v8:
> ================
>

...

> * Bumped pycdlib version to 1.9.0, which contains an endianess bug that
>   was seen on s390x hosts.

I meant, "which contains a bug *fix*". Hopefully not introducing bugs on
purpose! :)

- Cleber.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]