Re: [PATCH v6 00/10] Introduce new acpi/smbios avocado tests using biosbits

["Michael S. Tsirkin" <mst@redhat.com>]

On Thu, Oct 20, 2022 at 06:04:56PM +0530, Ani Sinha wrote:
> Please see the rst doc file in patch 9 for more details.
> Sample runs are as follows:
> 
> Passed tests:
> 
> $ ./tests/venv/bin/avocado run -t acpi tests/avocado
> Fetching asset from tests/avocado/acpi-bits.py:AcpiBitsTest.test_acpi_smbios_bits
> JOB ID     : e95c7413e996bfb59389839e5ca5105464ef098f
> JOB LOG    : /home/anisinha/avocado/job-results/job-2022-10-14T19.15-e95c741/job.log
>  (1/1) tests/avocado/acpi-bits.py:AcpiBitsTest.test_acpi_smbios_bits: PASS (33.08 s)
> RESULTS    : PASS 1 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0
> JOB TIME   : 39.18 s
> 
> 
> $ ./tests/venv/bin/avocado run ./tests/avocado/acpi-bits.py 
> Fetching asset from ./tests/avocado/acpi-bits.py:AcpiBitsTest.test_acpi_smbios_bits
> JOB ID     : c6e588594bceb2c5fdbdf45095559c58e4030f9c
> JOB LOG    : /home/anisinha/avocado/job-results/job-2022-10-14T19.13-c6e5885/job.log
>  (1/1) ./tests/avocado/acpi-bits.py:AcpiBitsTest.test_acpi_smbios_bits: PASS (32.29 s)
> RESULTS    : PASS 1 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0
> JOB TIME   : 39.27 s
> 
> When PSS tests are enabled:
> 
> $ git diff
> diff --git a/tests/avocado/acpi-bits/bits-tests/testacpi.py2 b/tests/avocado/acpi-bits/bits-tests/testacpi.py2
> index f818a9cce6..18dc818d62 100644
> --- a/tests/avocado/acpi-bits/bits-tests/testacpi.py2
> +++ b/tests/avocado/acpi-bits/bits-tests/testacpi.py2
> @@ -40,8 +40,8 @@ import time
>  
>  def register_tests():
>      testsuite.add_test("ACPI _MAT (Multiple APIC Table Entry) under Processor objects", test_mat, submenu="ACPI Tests")
> -#    testsuite.add_test("ACPI _PSS (Pstate) table conformance tests", test_pss, submenu="ACPI Tests")
> -#    testsuite.add_test("ACPI _PSS (Pstate) runtime tests", test_pstates, submenu="ACPI Tests")
> +    testsuite.add_test("ACPI _PSS (Pstate) table conformance tests", test_pss, submenu="ACPI Tests")
> +    testsuite.add_test("ACPI _PSS (Pstate) runtime tests", test_pstates, submenu="ACPI Tests")
>      testsuite.add_test("ACPI DSDT (Differentiated System Description Table)", test_dsdt, submenu="ACPI Tests")
>      testsuite.add_test("ACPI FACP (Fixed ACPI Description Table)", test_facp, submenu="ACPI Tests")
>      testsuite.add_test("ACPI HPET (High Precision Event Timer Table)", test_hpet, submenu="ACPI Tests")
> 
> $ ./tests/venv/bin/avocado run -t acpi tests/avocado
> Fetching asset from tests/avocado/acpi-bits.py:AcpiBitsTest.test_acpi_smbios_bits
> JOB ID     : 16a8b8c7f8dc87c4b84f8a2f95135bb63464a71e
> JOB LOG    : /home/anisinha/avocado/job-results/job-2022-10-14T22.43-16a8b8c/job.log
>  (1/1) tests/avocado/acpi-bits.py:AcpiBitsTest.test_acpi_smbios_bits: FAIL: '1' != '0'\n- 1\n+ 0\n : Some bits tests seems to have failed. Please check the test logs for more info. (33.01 s)
> RESULTS    : PASS 0 | ERROR 0 | FAIL 1 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0
> JOB TIME   : 39.17 s
> 
> 
> Changelog:
> v6:
>   - skip test when dependencies (xorriso for example) are not installed.
>   - skip test when run on a platform other than x86_64.

Hmm why is that btw?

>   - move README as an rst file under docs/devel.
>   - updated tags.
> v5:
>   - fixed bits so that grub-mkrescue is built as a static binary.
>   - updated the test so that it points to the new bits artifact.
>   - V=1 now spits out more meaningful information on failure.
>   - added a check to see if xorriso is installed. If not, the test skips with
>     a meaningful message instructing the user to install it.
>   - fixed README by removing references to stale instructions and adding
>     updated instructions.
>   - tags updated.
> v4:
>   - renamed the bits test files as .py2.
>   - fixed a bug with missing log file parsing. dump bits logs automatically
>     for test failures (no need to re-run with V=1).
>   - fixed issues with README file and enhanced it.
>   - cosmetic comment updates.
> v3:
>   - converted the existing test to avocado tests as per the popular
>     recommendation. Added appropriate tags.
>   - artifact download URL modified based on gitlab-CI URL.
> 
>   For biosbits repo:
>   - moved to a gitlab repo with me being the maintainer.
>   - added .gitlab-ci.yml file to generate the artifacts.
> v2:
>  - a new class of python based tests introduced that is separate from avocado
>    tests or qtests. Can be run by using "make check-pytest".
>  - acpi biosbits tests are the first tests to use pytest environment.
>  - bios bits tests now download the bits binary archives from a remote
>    repository if they are not found locally. The test skips if download
>    fails.
>  - A new environment variable is introduced that can be passed by the tester
>    to specify the location of the bits archives locally. test skips if the
>    bits binaries are not found in that location.
>  - if pip install of python module fails for whatever reaoson, the test skips.
>  - misc code fixes including spell check of the README doc. README has been
>    updated as well.
>  - addition of SPDX license headers to bits test files.
>  - update MAINTAINERS to reflect the new pytest test class.
> 
> For biosbits repo:
>  - added Dockerfile and build script. Made bios bits build on gcc 11.
>    https://github.com/ani-sinha/bits/blob/bits-qemu-logging/Dockerfile
>    https://github.com/ani-sinha/bits/blob/bits-qemu-logging/build-artifacts.sh
>    The build script generates the zip archive and tarball used by the test.
> 
> v1: initial patchset. uses qtest to implement the bios bits tests.
> 
> Cc: Qemu Devel <qemu-devel@nongnu.org>
> Cc: Daniel P. Berrangé <berrange@redhat.com>
> Cc: Paolo Bonzini <pbonzini@redhat.com>
> Cc: Maydell Peter <peter.maydell@linaro.org>
> Cc: John Snow <jsnow@redhat.com>
> Cc: Thomas Huth <thuth@redhat.com>
> Cc: Alex Bennée <alex.bennee@linaro.org>
> Cc: Igor Mammedov <imammedo@redhat.com>
> Cc: Michael Tsirkin <mst@redhat.com>
> 
> Ani Sinha (10):
>   acpi/tests/avocado/bits: initial commit of test scripts that are run
>     by biosbits
>   acpi/tests/avocado/bits: add SPDX license identifiers for bios bits
>     tests
>   acpi/tests/avocado/bits: disable acpi PSS tests that are failing in
>     biosbits
>   acpi/tests/avocado/bits: add smilatency test suite from bits in order
>     to disable it
>   acpi/tests/avocado/bits: add SPDX license identifiers for bios bits
>     smilatency tests
>   acpi/tests/avocado/bits: disable smilatency test since it does not
>     pass everytime
>   acpi/tests/avocado/bits: add biosbits config file for running bios
>     tests
>   acpi/tests/avocado/bits: add acpi and smbios avocado tests that uses
>     biosbits
>   acpi/tests/avocado/bits/doc: add a doc file to describe the acpi bits
>     test
>   MAINTAINERS: add myself as the maintainer for acpi biosbits avocado
>     tests
> 
>  MAINTAINERS                                   |    6 +
>  docs/devel/acpi-bits.rst                      |  148 +
>  tests/avocado/acpi-bits.py                    |  389 +++
>  .../acpi-bits/bits-config/bits-cfg.txt        |   18 +
>  tests/avocado/acpi-bits/bits-tests/smbios.py2 | 2434 +++++++++++++++++
>  .../acpi-bits/bits-tests/smilatency.py2       |  107 +
>  .../avocado/acpi-bits/bits-tests/testacpi.py2 |  287 ++
>  .../acpi-bits/bits-tests/testcpuid.py2        |   87 +
>  8 files changed, 3476 insertions(+)
>  create mode 100644 docs/devel/acpi-bits.rst
>  create mode 100644 tests/avocado/acpi-bits.py
>  create mode 100644 tests/avocado/acpi-bits/bits-config/bits-cfg.txt
>  create mode 100644 tests/avocado/acpi-bits/bits-tests/smbios.py2
>  create mode 100644 tests/avocado/acpi-bits/bits-tests/smilatency.py2
>  create mode 100644 tests/avocado/acpi-bits/bits-tests/testacpi.py2
>  create mode 100644 tests/avocado/acpi-bits/bits-tests/testcpuid.py2
> 
> -- 
> 2.34.1