Re: [PATCH v2 00/18] test/functional: improve functional test debugging & fix tuxrun

["Alex Bennée" <alex.bennee@linaro.org>]

Daniel P. Berrangé <berrange@redhat.com> writes:

> This started out as a series to get rid of the many GBs of temp
> files the functional tests leave behind. Then it expanded into
> improving the functional test debugging by ensuring we preserve
> the QEMU stdout/stderr log file created by the QEMUMachine class.
> In the course of doing that I encountered some other minor points
> worth fixing, and then got side tracked into looking at the tuxrun
> hangs with aarch64be. Investigating the latter exposed some further
> holes in the debugging story prompting yet more patches, as well as
> a final solution for tuxrun. So this series does:
>
>  * Purge all scratch files created by tests
>  * Preserve the stdout/stderr log file
>  * Capture debug log messages on QEMUMachine
>  * Provide a QMP backdoor for debugging stuck QEMUs
>  * Enhance console handling for partial line matches
>  * Fix the tuxrun tests by eliminating sleeps
>
> There's quite alot of code here, but at the same time it feels like
> the kind of stuff that'll be valuable either in the 9.2 release, or
> in the soon to exist 9.2 stable branch.
>
> NB, with this series applied Thomas' tuxrun conversion to functional
> testing survives 200 iterations on my machine, whereas it would
> reliably hang in < 20, and often in < 10, before.

Queued to testing/next, thanks.

I'll combine with plugins/next and some other misc fixes and post a
pre-PR for next week.

>
> Changed in v2:
>
>  - Changed console interaction to forbid 'failure_message'
>    without 'success_message'
>  - Reword console interaction log messages
>  - Avoid stack trace when seeing early failure
>  - Rewrote comment in acpi bits test
>  - Avoid duplicate os.environ access
>
> Daniel P. Berrangé (18):
>   tests/functional: fix mips64el test to honour workdir
>   tests/functional: automatically clean up scratch files after tests
>   tests/functional: remove "AVOCADO" from env variable name
>   tests/functional: remove todo wrt avocado.utils.wait_for
>   tests/functional: remove leftover :avocado: tags
>   tests/functional: remove obsolete reference to avocado bug
>   tests/functional: remove comments talking about avocado
>   tests/functional: honour self.workdir in ACPI bits tests
>   tests/functional: put QEMUMachine logs in testcase log directory
>   tests/functional: honour requested test VM name in QEMUMachine
>   tests/functional: enable debug logging for QEMUMachine
>   tests/functional: logs details of console interaction operations
>   tests/functional: don't try to wait for the empty string
>   tests/functional: require non-NULL success_message for console wait
>   tests/functional: rewrite console handling to be bytewise
>   tests/functional: remove time.sleep usage from tuxrun tests
>   tests/functional: add a QMP backdoor for debugging stalled tests
>   tests/functional: avoid accessing log_filename on earlier failures
>
>  docs/devel/testing/functional.rst        | 16 +++++
>  tests/functional/qemu_test/cmd.py        | 89 +++++++++++++++++++-----
>  tests/functional/qemu_test/testcase.py   | 43 +++++++++---
>  tests/functional/qemu_test/tuxruntest.py | 17 ++---
>  tests/functional/test_acpi_bits.py       | 57 +++++----------
>  tests/functional/test_arm_bpim2u.py      | 20 ------
>  tests/functional/test_arm_orangepi.py    | 27 -------
>  tests/functional/test_m68k_nextcube.py   |  3 +-
>  tests/functional/test_mips64el_malta.py  |  4 +-
>  tests/functional/test_virtio_gpu.py      |  3 +-
>  10 files changed, 150 insertions(+), 129 deletions(-)

-- 
Alex Bennée
Virtualisation Tech Lead @ Linaro