Re: [Qemu-devel] [PATCH 7/7] Acceptance Tests: change the handling of tests for specific archs

[Cleber Rosa <crosa@redhat.com>]

On 10/4/18 11:42 AM, Philippe Mathieu-Daudé wrote:
> On 04/10/2018 17:14, Cleber Rosa wrote:
>> With the introduction of a variants file that can run the same
>> tests on various architectures, it makes sense to make most tests
>> to be reusable on those environments.  The exception should be
>> when a test is really testing a specific architecture feature.
>>
>> With the change proposed here, on a command line such as:
>>
>>   $ avocado run \
>>      --json-variants-load=tests/acceptance/variants/arch.json \
>>      -- tests/acceptance/
>>
>> The boot_linux_console.py tests will appear as "CANCELED: Currently
>> specific to the x86_64 arch", which is as a good thing when compared
>> to being ignored by tags because:
>>
>>  * The architecture specific parts can be addressed
>>  * It will be run on the matching architecture (as opposed to always
>>    being filtered out by the tags mechanism)
>>  * CANCELED tests do no influence negatively the overall job results,
>>    they're not considered an error or failure
>>
>> Signed-off-by: Cleber Rosa <crosa@redhat.com>
>> ---
>>  tests/acceptance/boot_linux_console.py | 3 ++-
>>  1 file changed, 2 insertions(+), 1 deletion(-)
>>
>> diff --git a/tests/acceptance/boot_linux_console.py b/tests/acceptance/boot_linux_console.py
>> index 58032f971c..2fb5da63cb 100644
>> --- a/tests/acceptance/boot_linux_console.py
>> +++ b/tests/acceptance/boot_linux_console.py
>> @@ -19,12 +19,13 @@ class BootLinuxConsole(Test):
>>      and the kernel command line is properly passed from QEMU to the kernel
>>  
>>      :avocado: enable
>> -    :avocado: tags=x86_64
>>      """
>>  
>>      timeout = 60
>>  
>>      def test(self):
>> +        if self.arch != 'x86_64':
>> +            self.cancel('Currently specific to the x86_64 arch')
> 
> Watch out the difference between the HOST arch and the TARGET arch.
> 
> This test is specific to the x86_64 TARGET arch (the one emulated),
> _but_ it can be run on any HOST arch with TCG enabled:
> 

Right, do you think we should be more explicit about the terminology
here and use "Currently specific to a x86_64 target arch" ?

> $ uname -m
> aarch64
> 
> $ avocado \
>       --show=console \
>     run \
>       -p qemu_bin=x86_64-softmmu/qemu-system-x86_64 \
>    tests/acceptance/boot_linux_console.py

The arch will only be based on `uname -m` if not given explicitly.  What
I mean is that on your aarch64 host, you should be able to do:

 $ avocado run -p arch=x86_64 ...

And that test should run the same way.  So yes, arch is about the
target, but it defaults to the same arch as the host.

Again, should we be more explicit about it?

Thanks!
- Cleber.

> JOB ID     : f6e023ac4ee0f999ab01bfb1b196f43fde8538b9
> 
> JOB LOG    :
> /home/phil/avocado/job-results/job-2018-10-04T15.41-f6e023a/job.log
>  (1/1) tests/acceptance/boot_linux_console.py:BootLinuxConsole.test: /
> console: [    0.000000] Linux version 4.16.3-301.fc28.x86_64
> (mockbuild@bkernel02.phx2.fedoraproject.org) (gcc version 8.0.1 20180324
> (Red Hat 8.0.1-0.20) (GCC)) #1 SMP Mon Apr 23 21:59:58 8
> console: [    0.000000] Command line: console=ttyS0
> console: [    0.000000] x86/fpu: x87 FPU will use FXSAVE
> ...
> console: [    0.000000] NX (Execute Disable) protection: active
> console: [    0.000000] random: fast init done
> console: [    0.000000] SMBIOS 2.8 present.
> 
> console: [    0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996),
> BIOS rel-1.11.2-0-gf9626ccb91-prebuilt.qemu-project.org 04/01/2014
> ...
> console: [    0.000000] Booting paravirtualized kernel on bare hardware
> ...
> console: [    0.000000] Kernel command line: console=ttyS0
> PASS (12.89 s)
> RESULTS    : PASS 1 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 |
> CANCEL 0
> JOB TIME   : 13.12 s
> 
>>          kernel_url = ('https://mirrors.kernel.org/fedora/releases/28/'
>>                        'Everything/x86_64/os/images/pxeboot/vmlinuz')
>>          kernel_hash = '238e083e114c48200f80d889f7e32eeb2793e02a'
>>