Re: [Qemu-devel] Who is running QEMU automated tests, and when?

["Lukáš Doktor" <ldoktor@redhat.com>]

[-- Attachment #1: Type: text/plain, Size: 4965 bytes --]

Dne 26.4.2018 v 15:57 Eduardo Habkost napsal(a):
> (Starting a new thread, for more visibility)
> 

Hello guys,

what a nice topic. My Jenkins runs following weekly/daily upstream checks using RHEL.7 as host:

ppc64
=====

frequency: weekly
host:
    - ppc64
    - ppc64le
tests:
    - make    # building all targets
    - SPEED=slow make check
    - kvm-unit-tests:
        - ACCEL=kvm,kvm-type=HV
        - ACCEL=kvm,kvm-type=PR
        - ACCEL=tcg
    - qemu-iotests:
        - ALL_TESTS -qcow2 -file
        - ALL_TESTS -qcow2 -nbd    # quite broken
        - ALL_TESTS -raw -file
        - ALL_TESTS -raw -nbd    # quite broken
        - 059 -vmdk
        - 064 vhdx
        - 070 vhdx
        - 075 cloop
        - 076 parallels
        - 078 -bochs
        - 084 -vdi
        - 088 -vpc
        - 116 -qed
        - 131 -parallels
        - 135 -vpc
        - 146 -vpc
        # I tried -nfs but it seems broken and I'm still waiting for feedback
    - functonal:    # using Avocado-vt
        guest:
            - ppc64
            - ppc64le
        tests:
            - various RHEL.7 install jobs
            - migration between various tagged qemu revisions

s390x
=====

frequency: daily
tests:
    - make     # building all targets
    - SPEED=slow make check
    - kvm-unit-tests
    - functonal:    # using Avocado-vt
        - various RHEL.7 install jobs
        - migration using the latest qemu only

Unfortunately I'm currently changing the setup so for the past 1-2 months it's semi-broken. When I finish the transition I plan to add aarch64 upstream checks as well (as I am already running similar downstream suite there), but now I'm struggling with Jenkins and the new setup (which should have simplified things).

Kind regards,
Lukáš


> (This was: Re: [Qemu-devel] [RFC PATCH] tests/device-introspect: Test
> devices with all machines, not only with "none")
> 
> On Thu, Apr 26, 2018 at 01:54:43PM +0200, Markus Armbruster wrote:
>> Thomas Huth <thuth@redhat.com> writes:
>>> On 17.04.2018 14:12, Markus Armbruster wrote:
>>>> Thomas Huth <thuth@redhat.com> writes:
>>>>
>>>>> Many device introspection crashes only happen if you are using a
>>>>> certain machine, e.g.:
>>>>>
>>>>> $ ppc-softmmu/qemu-system-ppc -S -M ref405ep,accel=qtest -qmp stdio
>>>>> {"QMP": {"version": {"qemu": {"micro": 50, "minor": 11, "major": 2},
>>>>>  "package": "build-all"}, "capabilities": []}}
>>>>> { 'execute': 'qmp_capabilities' }
>>>>> {"return": {}}
>>>>> { 'execute': 'device-list-properties',
>>>>>   'arguments': {'typename': 'macio-newworld'}}
>>>>> Unexpected error in qemu_chr_fe_init() at chardev/char-fe.c:222:
>>>>> Device 'serial0' is in use
>>>>> Aborted (core dumped)
>>>>>
>>>>> To be able to catch these problems, let's extend the device-introspect
>>>>> test to check the devices on all machine types. Since this is a rather
>>>>> slow operation, the test is only run in "SPEED=slow" mode.
>>>>
>>>> If the device works with one machine type, it has a decent chance to
>>>> work with others, too.  Thus, testing each device with every machine
>>>> type is overkill.  I appreciate having overkill as an option :)
>>>>
>>>> What I'd like to see for a quick "make check" is testing each device
>>>> once.  That should flush out most bugs.  
>>>
>>> That's already done with the "none" machine.
>>
>> I was too terse.  We test each device with -machine none for every
>> target.  Fine if that's quick enough.  If not, we might want to reduce
>> redundancy there.
>>
>> Actually, a worse offender in the "waste everybody's time via redunancy"
>> department could be qom-test.
>>
>>> Anyway, do you think my patch here is useful and has a chance of getting
>>> included? I.e. shall I re-spin this as a non-RFC patch? Or shall we
>>> rather wait for Eduardo's python-based tests to get included into the
>>> repository?
>>
>> I don't mind having make check SPEED=slow run more extensive tests.
>> Assuming we actually run them at least once in a while, which seems
>> doubtful.
> 
> We probably don't do that, but we really must be running a more
> extensive (and slower) test set at least once before every
> release.
> 
> Maybe some people are running SPEED=slow tests, or even more
> extensive test suites like avocado-vt once in a while, but we
> need to know who is running them, and when.
> 
> Today, the only test set I know people really run and would
> surely block a release is "make check [SPEED=quick]".
> 
> So, for anybody that runs automated QEMU tests once in a while,
> can we know:
> 
> * What test cases are you running?  Where can we get more
>   information about the tests you run?
> * When do you run them?  What triggers a new test run?
> 
> Peter, do you have additional tests you run before merging a pull
> request?  Additional test sets run before tagging a release?
> 



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]