Re: [Qemu-devel] [PATCH] Acceptance test: add coverage tests for -smp option

[Wainer dos Santos Moschetta <wainersm@redhat.com>]

On 11/12/2018 02:31 PM, Eduardo Habkost wrote:
> On Fri, Nov 09, 2018 at 02:58:00PM -0500, Wainer dos Santos Moschetta wrote:
>> This adds tests for SMP option, by passing -smp with
>> various combinations of cpus, cores, threads, and sockets
>> values it checks that invalid topologies are not accepted
>> as well as missing values are correctly calculated.
>>
>> Signed-off-by: Wainer dos Santos Moschetta<wainersm@redhat.com>
> The test code looks nice to me, but: how exactly do you expect
> this to be executed?

'make check-acceptance' executes it with default parameters (i.e. 
cores=2, threads=2, sockets=2, and cpus=8).

It is possible to overwrite the default parameters by using Avocado's -p 
option. For example, 'avocado run -p sockets=1 -p cores=2 -p threads=1 
tests/acceptance/smp_option_coverage.py'.

> Do we have a test runner or multiplexer configuration already
> able to generate the cores/threads/sockets/cpus parameters?

I did not have any variants file until you asked. Then I realized the 
problems (see inline below) that I will need to address on a v2 patch.

Anyway, the variants YAML file may look like this (adapted from an 
example created by Cleber Rosa):

# cat smp_variants.yaml
!mux
min:
   sockets: 1
   cores: 2
   threads: 1

medium:
   sockets: 2
   cores: 2
   threads: 2

max:
   sockets: 2
   cores: 8
   threads: 16

The smp_variants.yaml defines 3 variants (min, medium, and max), each 
with a different SMP topology. You can run the tests as:

# avocado run tests/acceptance/smp_option_coverage.py -m smp_variants.yaml
JOB ID     : 08d9736942e550226de9c3425a9b65c378b6654a
JOB LOG    : /root/avocado/job-results/job-2018-11-19T13.19-08d9736/job.log
  (01/60) 
tests/acceptance/smp_option_coverage.py:SmpOption.test_cpus_eq_maxcpus;min-7e5d: 
PASS (0.04 s)
< output omitted >
  (60/60) 
tests/acceptance/smp_option_coverage.py:SmpOption.test_cpus_gt_cores_threads_sockets;max-a8e5: 
PASS (0.02 s)
RESULTS    : PASS 60 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 
| CANCEL 0
JOB TIME   : 6.25 s

If you wish I can distribute that file with this patch too. I can use 
Avocado's test data file mechanism to run the variants as described in:
https://avocado-framework.readthedocs.io/en/65.0/WritingTests.html#accessing-test-data-files

As an alternative I can document all that on docs/devel/testing.rst. 
Whatever you prefer.

Thanks for the review!

- Wainer

>
>> ---
>>   tests/acceptance/smp_option_coverage.py | 218 ++++++++++++++++++++++++
>>   1 file changed, 218 insertions(+)
>>   create mode 100644 tests/acceptance/smp_option_coverage.py
>>
>> diff --git a/tests/acceptance/smp_option_coverage.py b/tests/acceptance/smp_option_coverage.py
>> new file mode 100644
>> index 0000000000..ab68d1a67d
>> --- /dev/null
>> +++ b/tests/acceptance/smp_option_coverage.py
>> @@ -0,0 +1,218 @@
>> +# QEMU -smp option coverage test.
>> +#
>> +# Copyright (c) 2018 Red Hat, Inc.
>> +#
>> +# Author:
>> +#  Wainer dos Santos Moschetta<wainersm@redhat.com>
>> +#
>> +# This work is licensed under the terms of the GNU GPL, version 2 or
>> +# later.  See the COPYING file in the top-level directory.
>> +
>> +from functools import reduce
>> +from avocado.utils.process import run
>> +
>> +from avocado_qemu import Test
>> +
>> +
>> +class SmpOption(Test):
>> +    """
>> +    Launches QEMU with various cpus, cores, threads, sockets, and maxcpus
>> +    combination through -smp option, to check it does not accept invalid SMP
>> +    topologies as well as it is able to calculate correctly any missing values.
>> +
>> +    :avocado: enable
>> +    :avocado: tags=slow,coverage
>> +    """
>> +    def setUp(self):
>> +        super().setUp()
>> +        self.cores = self.params.get('cores', default=2)
>> +        self.threads = self.params.get('threads', default=2)
>> +        self.sockets = self.params.get('sockets', default=2)
>> +        self.cpus = self.params.get('cpus', default=8)

The self.cpus variable should not be a parameter but rather calculated 
(cores * threads * sockets).
Also needs to type convert the return of self.params.get from string to 
integer.

>> +
>> +    def get_smp_topology(self):
>> +        """
>> +        Returns a dict with the id of cores, threads and sockets.
>> +        """
>> +        res = self.vm.qmp('query-hotpluggable-cpus')['return']
>> +        cpus = [x['props'] for x in res]
>> +
>> +        return reduce(lambda x, y: {'core-id': x['core-id'].union([y['core-id']]),
>> +                                    'thread-id': x['thread-id'].union([y['thread-id']]),
>> +                                    'socket-id': x['socket-id'].union([y['socket-id']])},
>> +                      cpus, {'core-id': set(), 'thread-id': set(), 'socket-id': set()})
>> +
>> +    @staticmethod
>> +    def build_option(**kwargs):
>> +        """
>> +        Builds string for the -smp option.
>> +        Use cpus, cores, threads, sockets, maxcpus keys.
>> +        """
>> +        option_list = []
>> +        if kwargs.get('cpus', None) is not None:
>> +            option_list.append(str(kwargs.get('cpus')))
>> +        for key, val in kwargs.items():
>> +            if key == 'cpus':
>> +                continue
>> +            option_list.append('%s=%s' % (key, val))
>> +
>> +        return ",".join(option_list)
>> +
>> +    def launch_and_check(self, expect_cores=1, expect_threads=1,
>> +                         expect_sockets=1, **kwargs):
>> +        """
>> +        Launches VM and check its SMP topology was correctly set.
>> +        Use cpus, cores, threads, sockets, and maxcpus keys to specify the
>> +        topology.
>> +        """
>> +        option = self.build_option(**{key: val for key, val in kwargs.items()
>> +                                      if key in ['cpus', 'cores', 'threads',
>> +                                                 'sockets', 'maxcpus']})
>> +        self.vm.add_args('-smp', option)
>> +        self.vm.launch()
>> +        smp = self.get_smp_topology()
>> +        self.assertEqual(smp['core-id'], set(range(0, expect_cores)))
>> +        self.assertEqual(smp['thread-id'], set(range(0, expect_threads)))
>> +        self.assertEqual(smp['socket-id'], set(range(0, expect_sockets)))
>> +
>> +    def launch_and_check_fail(self, **kwargs):
>> +        """
>> +        Launches VM and check the process exits with expected error code, for
>> +        cases where the topology is expected not valid.
>> +        """
>> +        option = self.build_option(**kwargs)
>> +        res = run("%s -smp %s" % (self.qemu_bin, option), timeout=10,
>> +                  ignore_status=True)
>> +        self.assertNotEqual(res.exit_status, 0)
>> +
>> +    # Passing cpus and maxcpus only.
>> +    #
>> +
>> +    def test_cpus_eq_maxcpus(self):
>> +        self.launch_and_check(cpus=self.cpus,
>> +                              maxcpus=self.cpus,
>> +                              expect_sockets=self.cpus)
>> +
>> +    def test_cpus_lt_maxcpus(self):
>> +        maxcpus = self.cpus * 2
>> +        self.launch_and_check(cpus=self.cpus,
>> +                              maxcpus=maxcpus,
>> +                              expect_sockets=maxcpus)
>> +
>> +    def test_cpus_gt_maxcpus(self):
>> +        self.launch_and_check_fail(cpus=self.cpus,
>> +                                   maxcpus=self.cpus // 2)
>> +
>> +    # Passing a combination of cores, threads and sockets only.
>> +    #
>> +
>> +    def test_no_cores_no_threads_no_sockets(self):
>> +        self.launch_and_check(cpus=self.cpus,
>> +                              expect_sockets=self.cpus)
>> +
>> +    def test_no_cores_no_threads_sockets(self):
>> +        self.launch_and_check(sockets=self.sockets,
>> +                              expect_sockets=self.sockets)
>> +
>> +    def test_no_cores_threads_no_sockets(self):
>> +        self.launch_and_check(threads=self.threads,
>> +                              expect_threads=self.threads)
>> +
>> +    def test_no_cores_threads_sockets(self):
>> +        self.launch_and_check(threads=self.threads,
>> +                              sockets=self.sockets,
>> +                              expect_threads=self.threads,
>> +                              expect_sockets=self.sockets)
>> +
>> +    def test_cores_no_threads_no_sockets(self):
>> +        self.launch_and_check(cores=self.cores,
>> +                              expect_cores=self.cores)
>> +
>> +    def test_cores_no_threads_sockets(self):
>> +        self.launch_and_check(cores=self.cores,
>> +                              sockets=self.sockets,
>> +                              expect_cores=self.cores,
>> +                              expect_sockets=self.sockets)
>> +
>> +    def test_cores_threads_no_sockets(self):
>> +        self.launch_and_check(cores=self.cores,
>> +                              threads=self.threads,
>> +                              expect_cores=self.cores,
>> +                              expect_threads=self.threads)
>> +
>> +    def test_cores_threads_sockets(self):
>> +        self.launch_and_check(cores=self.cores,
>> +                              threads=self.threads,
>> +                              sockets=self.sockets,
>> +                              expect_cores=self.cores,
>> +                              expect_threads=self.threads,
>> +                              expect_sockets=self.sockets)
>> +
>> +    # Passing cpus always and a combination of cores, threads and sockets.
>> +    #
>> +
>> +    def test_cpus_no_cores_no_threads_sockets(self):
>> +        self.launch_and_check(cpus=self.cpus,
>> +                              sockets=self.sockets,
>> +                              expect_cores=self.cpus // self.sockets,
>> +                              expect_sockets=self.sockets)
>> +
>> +    def test_cpus_no_cores_threads_no_sockets(self):
>> +        self.launch_and_check(cpus=self.cpus,
>> +                              threads=self.threads,
>> +                              expect_threads=self.threads,
>> +                              expect_sockets=self.cpus // self.threads)
>> +
>> +    def test_cpus_no_cores_threads_sockets(self):
>> +        self.launch_and_check(cpus=self.cpus,
>> +                              threads=self.threads,
>> +                              sockets=self.sockets,
>> +                              expect_threads=self.threads,
>> +                              expect_sockets=self.sockets,
>> +                              expect_cores=self.cpus // (self.threads * self.sockets))
>> +
>> +    def test_cpus_cores_no_threads_no_sockets(self):
>> +        self.launch_and_check(cpus=self.cpus,
>> +                              cores=self.cores,
>> +                              expect_cores=self.cores,
>> +                              expect_sockets=self.cpus // self.cores)
>> +
>> +    def test_cpus_cores_no_threads_sockets(self):
>> +        self.launch_and_check(cpus=self.cpus,
>> +                              cores=self.cores,
>> +                              sockets=self.sockets,
>> +                              expect_cores=self.cores,
>> +                              expect_sockets=self.sockets,
>> +                              expect_threads=self.cpus // (self.cores * self.sockets))
>> +
>> +    def test_cpus_cores_threads_no_sockets(self):
>> +        self.launch_and_check(cpus=self.cpus,
>> +                              cores=self.cores,
>> +                              threads=self.threads,
>> +                              expect_cores=self.cores,
>> +                              expect_threads=self.threads,
>> +                              expect_sockets=self.cpus // (self.cores * self.threads))
>> +
>> +    def test_cpus_cores_threads_sockets(self):
>> +        self.launch_and_check(cpus=self.cores * self.threads * self.sockets,
>> +                              cores=self.cores,
>> +                              threads=self.threads,
>> +                              sockets=self.sockets,
>> +                              expect_cores=self.cores,
>> +                              expect_threads=self.threads,
>> +                              expect_sockets=self.sockets)
>> +
>> +    # Passing cpus less (or greater) than cores, threads and sockets.
>> +    #
>> +
>> +    def test_cpus_lt_cores_threads_sockets(self):
>> +        self.launch_and_check_fail(cpus=(self.cores * self.threads * self.sockets) // 2,
>> +                                   cores=self.cores,
>> +                                   threads=self.threads,
>> +                                   sockets=self.sockets,)
>> +
>> +    def test_cpus_gt_cores_threads_sockets(self):
>> +        self.launch_and_check_fail(cpus=self.cores * self.threads * self.sockets * 2,
>> +                                   cores=self.cores,
>> +                                   threads=self.threads,
>> +                                   sockets=self.sockets)
>> -- 
>> 2.17.2
>>