[PATCH v2 0/6] More record/replay acceptance tests

[PATCH v2 0/6] More record/replay acceptance tests

[Pavel Dovgalyuk]

The following series adds new record/replay tests to the acceptance group.

The provided tests perform kernel boot and disk image boot scenarios.
For all of them recording and replaying phases are executed.
Tests were borrowed from existing boot_linux*.py tests.

New tests include kernel boot for s390x, ppc64, alpha, nios2, and openrisc,
and Linux boot with cloudinit image for x86_64.

v2 changes:
 - moved ppc64 test to the right script

---

Pavel Dovgaluk (1):
      tests/acceptance: Linux boot test for record/replay

Pavel Dovgalyuk (5):
      tests/acceptance: add replay kernel test for s390
      tests/acceptance: add replay kernel test for ppc64
      tests/acceptance: add replay kernel test for openrisc
      tests/acceptance: add replay kernel test for nios2
      tests/acceptance: add replay kernel test for alpha


 MAINTAINERS                       |   1 +
 tests/acceptance/replay_kernel.py |  52 ++++++++++++++
 tests/acceptance/replay_linux.py  | 116 ++++++++++++++++++++++++++++++
 3 files changed, 169 insertions(+)
 create mode 100644 tests/acceptance/replay_linux.py

--
Pavel Dovgalyuk

[PATCH v2 1/6] tests/acceptance: add replay kernel test for s390

[Pavel Dovgalyuk]

This patch adds record/replay test which boots Linux
kernel on s390x platform. The test uses kernel binaries
taken from boot_linux_console test.

Signed-off-by: Pavel Dovgalyuk <Pavel.Dovgalyuk@ispras.ru>
Reviewed-by: Willian Rampazzo <willianr@redhat.com>
---
 tests/acceptance/replay_kernel.py |   16 ++++++++++++++++
 1 file changed, 16 insertions(+)

diff --git a/tests/acceptance/replay_kernel.py b/tests/acceptance/replay_kernel.py
index 71facdaa75..cdc22cb6d3 100644
--- a/tests/acceptance/replay_kernel.py
+++ b/tests/acceptance/replay_kernel.py
@@ -208,6 +208,22 @@ def test_arm_cubieboard_initrd(self):
                           '-initrd', initrd_path,
                           '-no-reboot'))
 
+    def test_s390x_s390_ccw_virtio(self):
+        """
+        :avocado: tags=arch:s390x
+        :avocado: tags=machine:s390-ccw-virtio
+        """
+        kernel_url = ('https://archives.fedoraproject.org/pub/archive'
+                      '/fedora-secondary/releases/29/Everything/s390x/os/images'
+                      '/kernel.img')
+        kernel_hash = 'e8e8439103ef8053418ef062644ffd46a7919313'
+        kernel_path = self.fetch_asset(kernel_url, asset_hash=kernel_hash)
+
+        kernel_command_line = self.KERNEL_COMMON_COMMAND_LINE + 'console=sclp0'
+        console_pattern = 'Kernel command line: %s' % kernel_command_line
+        self.run_rr(kernel_path, kernel_command_line, console_pattern, shift=9,
+            args=('-nodefaults', '-smp', '1'))
+
     def test_ppc64_pseries(self):
         """
         :avocado: tags=arch:ppc64

Re: [PATCH v2 1/6] tests/acceptance: add replay kernel test for s390

[Thomas Huth]

On 25/06/2021 07.39, Pavel Dovgalyuk wrote:
> This patch adds record/replay test which boots Linux
> kernel on s390x platform. The test uses kernel binaries
> taken from boot_linux_console test.
> 
> Signed-off-by: Pavel Dovgalyuk <Pavel.Dovgalyuk@ispras.ru>
> Reviewed-by: Willian Rampazzo <willianr@redhat.com>
> ---
>   tests/acceptance/replay_kernel.py |   16 ++++++++++++++++
>   1 file changed, 16 insertions(+)
> 
> diff --git a/tests/acceptance/replay_kernel.py b/tests/acceptance/replay_kernel.py
> index 71facdaa75..cdc22cb6d3 100644
> --- a/tests/acceptance/replay_kernel.py
> +++ b/tests/acceptance/replay_kernel.py
> @@ -208,6 +208,22 @@ def test_arm_cubieboard_initrd(self):
>                             '-initrd', initrd_path,
>                             '-no-reboot'))
>   
> +    def test_s390x_s390_ccw_virtio(self):
> +        """
> +        :avocado: tags=arch:s390x
> +        :avocado: tags=machine:s390-ccw-virtio
> +        """
> +        kernel_url = ('https://archives.fedoraproject.org/pub/archive'
> +                      '/fedora-secondary/releases/29/Everything/s390x/os/images'
> +                      '/kernel.img')
> +        kernel_hash = 'e8e8439103ef8053418ef062644ffd46a7919313'
> +        kernel_path = self.fetch_asset(kernel_url, asset_hash=kernel_hash)
> +
> +        kernel_command_line = self.KERNEL_COMMON_COMMAND_LINE + 'console=sclp0'
> +        console_pattern = 'Kernel command line: %s' % kernel_command_line
> +        self.run_rr(kernel_path, kernel_command_line, console_pattern, shift=9,
> +            args=('-nodefaults', '-smp', '1'))

I guess you could drop the "-smp 1" since one CPU is the default anyway.

Anyway,
Acked-by: Thomas Huth <thuth@redhat.com>

[PATCH v2 2/6] tests/acceptance: add replay kernel test for ppc64

[Pavel Dovgalyuk]

This patch adds record/replay test which boots Linux
kernel on ppc64 platform. The test uses kernel binaries
taken from boot_linux_console test.

Signed-off-by: Pavel Dovgalyuk <Pavel.Dovgalyuk@ispras.ru>
---
 tests/acceptance/replay_kernel.py |   13 +++++++++++++
 1 file changed, 13 insertions(+)

diff --git a/tests/acceptance/replay_kernel.py b/tests/acceptance/replay_kernel.py
index cdc22cb6d3..7e7f4c8ccc 100644
--- a/tests/acceptance/replay_kernel.py
+++ b/tests/acceptance/replay_kernel.py
@@ -367,6 +367,19 @@ def test_xtensa_lx60(self):
         self.do_test_advcal_2018(file_path, 'santas-sleigh-ride.elf',
                                  args=('-cpu', 'dc233c'))
 
+    def test_ppc64_e500(self):
+        """
+        :avocado: tags=arch:ppc64
+        :avocado: tags=machine:ppce500
+        :avocado: tags=cpu:e5500
+        """
+        tar_hash = '6951d86d644b302898da2fd701739c9406527fe1'
+        tar_url = ('https://www.qemu-advent-calendar.org'
+                   '/2018/download/day19.tar.xz')
+        file_path = self.fetch_asset(tar_url, asset_hash=tar_hash)
+        self.do_test_advcal_2018(file_path, 'uImage', ('-cpu', 'e5500'))
+
+
 @skipUnless(os.getenv('AVOCADO_TIMEOUT_EXPECTED'), 'Test might timeout')
 class ReplayKernelSlow(ReplayKernelBase):
     # Override the timeout, because this kernel includes an inner

Re: [PATCH v2 2/6] tests/acceptance: add replay kernel test for ppc64

[Willian Rampazzo]

Already upstream :)

On Fri, Jun 25, 2021 at 2:39 AM Pavel Dovgalyuk
<pavel.dovgalyuk@ispras.ru> wrote:
>
> This patch adds record/replay test which boots Linux
> kernel on ppc64 platform. The test uses kernel binaries
> taken from boot_linux_console test.
>
> Signed-off-by: Pavel Dovgalyuk <Pavel.Dovgalyuk@ispras.ru>
> ---
>  tests/acceptance/replay_kernel.py |   13 +++++++++++++
>  1 file changed, 13 insertions(+)
>
> diff --git a/tests/acceptance/replay_kernel.py b/tests/acceptance/replay_kernel.py
> index cdc22cb6d3..7e7f4c8ccc 100644
> --- a/tests/acceptance/replay_kernel.py
> +++ b/tests/acceptance/replay_kernel.py
> @@ -367,6 +367,19 @@ def test_xtensa_lx60(self):
>          self.do_test_advcal_2018(file_path, 'santas-sleigh-ride.elf',
>                                   args=('-cpu', 'dc233c'))
>
> +    def test_ppc64_e500(self):
> +        """
> +        :avocado: tags=arch:ppc64
> +        :avocado: tags=machine:ppce500
> +        :avocado: tags=cpu:e5500
> +        """
> +        tar_hash = '6951d86d644b302898da2fd701739c9406527fe1'
> +        tar_url = ('https://www.qemu-advent-calendar.org'
> +                   '/2018/download/day19.tar.xz')
> +        file_path = self.fetch_asset(tar_url, asset_hash=tar_hash)
> +        self.do_test_advcal_2018(file_path, 'uImage', ('-cpu', 'e5500'))
> +
> +
>  @skipUnless(os.getenv('AVOCADO_TIMEOUT_EXPECTED'), 'Test might timeout')
>  class ReplayKernelSlow(ReplayKernelBase):
>      # Override the timeout, because this kernel includes an inner
>

Re: [PATCH v2 2/6] tests/acceptance: add replay kernel test for ppc64

[Pavel Dovgalyuk]

On 05.07.2021 20:46, Willian Rampazzo wrote:
> Already upstream :)

Ok, thanks.

> 
> On Fri, Jun 25, 2021 at 2:39 AM Pavel Dovgalyuk
> <pavel.dovgalyuk@ispras.ru> wrote:
>>
>> This patch adds record/replay test which boots Linux
>> kernel on ppc64 platform. The test uses kernel binaries
>> taken from boot_linux_console test.
>>
>> Signed-off-by: Pavel Dovgalyuk <Pavel.Dovgalyuk@ispras.ru>
>> ---
>>   tests/acceptance/replay_kernel.py |   13 +++++++++++++
>>   1 file changed, 13 insertions(+)
>>
>> diff --git a/tests/acceptance/replay_kernel.py b/tests/acceptance/replay_kernel.py
>> index cdc22cb6d3..7e7f4c8ccc 100644
>> --- a/tests/acceptance/replay_kernel.py
>> +++ b/tests/acceptance/replay_kernel.py
>> @@ -367,6 +367,19 @@ def test_xtensa_lx60(self):
>>           self.do_test_advcal_2018(file_path, 'santas-sleigh-ride.elf',
>>                                    args=('-cpu', 'dc233c'))
>>
>> +    def test_ppc64_e500(self):
>> +        """
>> +        :avocado: tags=arch:ppc64
>> +        :avocado: tags=machine:ppce500
>> +        :avocado: tags=cpu:e5500
>> +        """
>> +        tar_hash = '6951d86d644b302898da2fd701739c9406527fe1'
>> +        tar_url = ('https://www.qemu-advent-calendar.org'
>> +                   '/2018/download/day19.tar.xz')
>> +        file_path = self.fetch_asset(tar_url, asset_hash=tar_hash)
>> +        self.do_test_advcal_2018(file_path, 'uImage', ('-cpu', 'e5500'))
>> +
>> +
>>   @skipUnless(os.getenv('AVOCADO_TIMEOUT_EXPECTED'), 'Test might timeout')
>>   class ReplayKernelSlow(ReplayKernelBase):
>>       # Override the timeout, because this kernel includes an inner
>>
> 

[PATCH v2 3/6] tests/acceptance: add replay kernel test for openrisc

[Pavel Dovgalyuk]

This patch adds record/replay test which boots Linux
kernel on openrisc platform. The test uses kernel binaries
taken from boot_linux_console test.

Signed-off-by: Pavel Dovgalyuk <Pavel.Dovgalyuk@ispras.ru>
Reviewed-by: Willian Rampazzo <willianr@redhat.com>
---
 tests/acceptance/replay_kernel.py |   11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/tests/acceptance/replay_kernel.py b/tests/acceptance/replay_kernel.py
index 7e7f4c8ccc..4c682b3f29 100644
--- a/tests/acceptance/replay_kernel.py
+++ b/tests/acceptance/replay_kernel.py
@@ -319,6 +319,17 @@ def test_ppc64_e500(self):
         file_path = self.fetch_asset(tar_url, asset_hash=tar_hash)
         self.do_test_advcal_2018(file_path, 'uImage', ('-cpu', 'e5500'))
 
+    def test_or1k_sim(self):
+        """
+        :avocado: tags=arch:or1k
+        :avocado: tags=machine:or1k-sim
+        """
+        tar_hash = '20334cdaf386108c530ff0badaecc955693027dd'
+        tar_url = ('https://www.qemu-advent-calendar.org'
+                   '/2018/download/day20.tar.xz')
+        file_path = self.fetch_asset(tar_url, asset_hash=tar_hash)
+        self.do_test_advcal_2018(file_path, 'vmlinux')
+
     def test_ppc_g3beige(self):
         """
         :avocado: tags=arch:ppc

[PATCH v2 4/6] tests/acceptance: add replay kernel test for nios2

[Pavel Dovgalyuk]

This patch adds record/replay test which boots Linux
kernel on nios2 platform. The test uses kernel binaries
taken from boot_linux_console test.

Signed-off-by: Pavel Dovgalyuk <Pavel.Dovgalyuk@ispras.ru>
Reviewed-by: Willian Rampazzo <willianr@redhat.com>
---
 tests/acceptance/replay_kernel.py |   11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/tests/acceptance/replay_kernel.py b/tests/acceptance/replay_kernel.py
index 4c682b3f29..ba2314d2ec 100644
--- a/tests/acceptance/replay_kernel.py
+++ b/tests/acceptance/replay_kernel.py
@@ -330,6 +330,17 @@ def test_or1k_sim(self):
         file_path = self.fetch_asset(tar_url, asset_hash=tar_hash)
         self.do_test_advcal_2018(file_path, 'vmlinux')
 
+    def test_nios2_10m50(self):
+        """
+        :avocado: tags=arch:nios2
+        :avocado: tags=machine:10m50-ghrd
+        """
+        tar_hash = 'e4251141726c412ac0407c5a6bceefbbff018918'
+        tar_url = ('https://www.qemu-advent-calendar.org'
+                   '/2018/download/day14.tar.xz')
+        file_path = self.fetch_asset(tar_url, asset_hash=tar_hash)
+        self.do_test_advcal_2018(file_path, 'vmlinux.elf')
+
     def test_ppc_g3beige(self):
         """
         :avocado: tags=arch:ppc

[PATCH v2 5/6] tests/acceptance: add replay kernel test for alpha

[Pavel Dovgalyuk]

This patch adds record/replay test which boots Linux
kernel on alpha platform. The test uses kernel binaries
taken from boot_linux_console test.

Signed-off-by: Pavel Dovgalyuk <Pavel.Dovgalyuk@ispras.ru>
Reviewed-by: Willian Rampazzo <willianr@redhat.com>
---
 tests/acceptance/replay_kernel.py |   17 +++++++++++++++++
 1 file changed, 17 insertions(+)

diff --git a/tests/acceptance/replay_kernel.py b/tests/acceptance/replay_kernel.py
index ba2314d2ec..76af30b28b 100644
--- a/tests/acceptance/replay_kernel.py
+++ b/tests/acceptance/replay_kernel.py
@@ -224,6 +224,23 @@ def test_s390x_s390_ccw_virtio(self):
         self.run_rr(kernel_path, kernel_command_line, console_pattern, shift=9,
             args=('-nodefaults', '-smp', '1'))
 
+    def test_alpha_clipper(self):
+        """
+        :avocado: tags=arch:alpha
+        :avocado: tags=machine:clipper
+        """
+        kernel_url = ('http://archive.debian.org/debian/dists/lenny/main/'
+                      'installer-alpha/20090123lenny10/images/cdrom/vmlinuz')
+        kernel_hash = '3a943149335529e2ed3e74d0d787b85fb5671ba3'
+        kernel_path = self.fetch_asset(kernel_url, asset_hash=kernel_hash)
+
+        uncompressed_kernel = archive.uncompress(kernel_path, self.workdir)
+
+        kernel_command_line = self.KERNEL_COMMON_COMMAND_LINE + 'console=ttyS0'
+        console_pattern = 'Kernel command line: %s' % kernel_command_line
+        self.run_rr(uncompressed_kernel, kernel_command_line, console_pattern, shift=9,
+            args=('-nodefaults', ))
+
     def test_ppc64_pseries(self):
         """
         :avocado: tags=arch:ppc64

[PATCH v2 6/6] tests/acceptance: Linux boot test for record/replay

[Pavel Dovgalyuk]

From: Pavel Dovgalyuk <Pavel.Dovgaluk@gmail.com>

This patch adds a test for record/replay, which boots Linux
image from the disk and interacts with the network.
The idea and code of this test is borrowed from boot_linux.py
This test includes only x86_64 platform. Other platforms and
machines will be added later after testing and improving
record/replay to completely support them.

Each test consists of the following phases:
 - downloading the disk image
 - recording the execution
 - replaying the execution

Replay does not validates the output, but waits until QEMU
finishes the execution. This is reasonable, because
QEMU usually hangs when replay goes wrong.

Signed-off-by: Pavel Dovgalyuk <Pavel.Dovgaluk@ispras.ru>
Reviewed-by: Willian Rampazzo <willianr@redhat.com>
---
 MAINTAINERS                      |    1 
 tests/acceptance/replay_linux.py |  116 ++++++++++++++++++++++++++++++++++++++
 2 files changed, 117 insertions(+)
 create mode 100644 tests/acceptance/replay_linux.py

diff --git a/MAINTAINERS b/MAINTAINERS
index 0ca6b7de94..35ecb2ce2c 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -2864,6 +2864,7 @@ F: include/sysemu/replay.h
 F: docs/replay.txt
 F: stubs/replay.c
 F: tests/acceptance/replay_kernel.py
+F: tests/acceptance/replay_linux.py
 F: tests/acceptance/reverse_debugging.py
 F: qapi/replay.json
 
diff --git a/tests/acceptance/replay_linux.py b/tests/acceptance/replay_linux.py
new file mode 100644
index 0000000000..15953f9e49
--- /dev/null
+++ b/tests/acceptance/replay_linux.py
@@ -0,0 +1,116 @@
+# Record/replay test that boots a complete Linux system via a cloud image
+#
+# Copyright (c) 2020 ISP RAS
+#
+# Author:
+#  Pavel Dovgalyuk <Pavel.Dovgaluk@ispras.ru>
+#
+# This work is licensed under the terms of the GNU GPL, version 2 or
+# later.  See the COPYING file in the top-level directory.
+
+import os
+import logging
+import time
+
+from avocado import skipUnless
+from avocado.utils import cloudinit
+from avocado.utils import network
+from avocado.utils import vmimage
+from avocado.utils import datadrainer
+from avocado.utils.path import find_command
+from avocado_qemu import LinuxTest
+
+class ReplayLinux(LinuxTest):
+    """
+    Boots a Linux system, checking for a successful initialization
+    """
+
+    timeout = 1800
+    chksum = None
+    hdd = 'ide-hd'
+    cd = 'ide-cd'
+    bus = 'ide'
+
+    def setUp(self):
+        super(ReplayLinux, self).setUp()
+        self.boot_path = self.download_boot()
+        self.cloudinit_path = self.prepare_cloudinit()
+
+    def vm_add_disk(self, vm, path, id, device):
+        bus_string = ''
+        if self.bus:
+            bus_string = ',bus=%s.%d' % (self.bus, id,)
+        vm.add_args('-drive', 'file=%s,snapshot,id=disk%s,if=none' % (path, id))
+        vm.add_args('-drive',
+            'driver=blkreplay,id=disk%s-rr,if=none,image=disk%s' % (id, id))
+        vm.add_args('-device',
+            '%s,drive=disk%s-rr%s' % (device, id, bus_string))
+
+    def launch_and_wait(self, record, args, shift):
+        vm = self.get_vm()
+        vm.add_args('-smp', '1')
+        vm.add_args('-m', '1024')
+        vm.add_args('-object', 'filter-replay,id=replay,netdev=hub0port0')
+        if args:
+            vm.add_args(*args)
+        self.vm_add_disk(vm, self.boot_path, 0, self.hdd)
+        self.vm_add_disk(vm, self.cloudinit_path, 1, self.cd)
+        logger = logging.getLogger('replay')
+        if record:
+            logger.info('recording the execution...')
+            mode = 'record'
+        else:
+            logger.info('replaying the execution...')
+            mode = 'replay'
+        replay_path = os.path.join(self.workdir, 'replay.bin')
+        vm.add_args('-icount', 'shift=%s,rr=%s,rrfile=%s' %
+                    (shift, mode, replay_path))
+
+        start_time = time.time()
+
+        vm.set_console()
+        vm.launch()
+        console_drainer = datadrainer.LineLogger(vm.console_socket.fileno(),
+                                    logger=self.log.getChild('console'),
+                                    stop_check=(lambda : not vm.is_running()))
+        console_drainer.start()
+        if record:
+            cloudinit.wait_for_phone_home(('0.0.0.0', self.phone_home_port),
+                                          self.name)
+            vm.shutdown()
+            logger.info('finished the recording with log size %s bytes'
+                % os.path.getsize(replay_path))
+        else:
+            vm.event_wait('SHUTDOWN', self.timeout)
+            vm.shutdown(True)
+            logger.info('successfully fihished the replay')
+        elapsed = time.time() - start_time
+        logger.info('elapsed time %.2f sec' % elapsed)
+        return elapsed
+
+    def run_rr(self, args=None, shift=7):
+        t1 = self.launch_and_wait(True, args, shift)
+        t2 = self.launch_and_wait(False, args, shift)
+        logger = logging.getLogger('replay')
+        logger.info('replay overhead {:.2%}'.format(t2 / t1 - 1))
+
+@skipUnless(os.getenv('AVOCADO_TIMEOUT_EXPECTED'), 'Test might timeout')
+class ReplayLinuxX8664(ReplayLinux):
+    """
+    :avocado: tags=arch:x86_64
+    :avocado: tags=accel:tcg
+    """
+
+    chksum = 'e3c1b309d9203604922d6e255c2c5d098a309c2d46215d8fc026954f3c5c27a0'
+
+    def test_pc_i440fx(self):
+        """
+        :avocado: tags=machine:pc
+        """
+        self.run_rr(shift=1)
+
+    def test_pc_q35(self):
+        """
+        :avocado: tags=machine:q35
+        """
+        self.run_rr(shift=3)