Re: [PATCH v4 09/10] acpi/tests/avocado/bits: add a README file to describe the test

[Ani Sinha <ani@anisinha.ca>]

On Fri, Oct 14, 2022 at 11:06 PM Ani Sinha <ani@anisinha.ca> wrote:
>
> Add a README file that describes the purpose of the various test files and gives
> guidance to developers on where and how to make changes.
>
> Cc: Daniel P. Berrange" <berrange@redhat.com>
> Cc: Paolo Bonzini <pbonzini@redhat.com>
> Cc: Maydell Peter <peter.maydell@linaro.org>
> Cc: John Snow <jsnow@redhat.com>
> Cc: Thomas Huth <thuth@redhat.com>
> Cc: Alex Bennée <alex.bennee@linaro.org>
> Cc: Igor Mammedov <imammedo@redhat.com>
> Cc: Michael Tsirkin <mst@redhat.com>
> Signed-off-by: Ani Sinha <ani@anisinha.ca>
> ---
>  tests/avocado/acpi-bits/README | 128 +++++++++++++++++++++++++++++++++
>  1 file changed, 128 insertions(+)
>  create mode 100644 tests/avocado/acpi-bits/README
>
> diff --git a/tests/avocado/acpi-bits/README b/tests/avocado/acpi-bits/README
> new file mode 100644
> index 0000000000..6950d8d7db
> --- /dev/null
> +++ b/tests/avocado/acpi-bits/README
> @@ -0,0 +1,128 @@
> +=============================================================================
> +ACPI/SMBIOS AVOCADO TESTS USING BIOSBITS
> +=============================================================================
> +
> +Biosbits is a software written by Josh Triplett that can be downloaded
> +from https://biosbits.org/. The github codebase can be found here:
> +https://github.com/biosbits/bits/tree/master. It is a software that executes
> +the bios components such as acpi and smbios tables directly through acpica
> +bios interpreter (a freely available C based library written by Intel,
> +downloadable from https://acpica.org/ and is included with biosbits) without an
> +operating system getting involved in between.
> +There are several advantages to directly testing the bios in a real physical
> +machine or VM as opposed to indirectly discovering bios issues through the
> +operating system. For one thing, the OSes tend to hide bios problems from the
> +end user. The other is that we have more control of what we wanted to test
> +and how by directly using acpica interpreter on top of the bios on a running
> +system. More details on the inspiration for developing biosbits and its real
> +life uses can be found in (a) and (b).
> +This directory contains tests written in python using avocado framework that
> +exercizes the QEMU bios components using biosbits and reports test failures.
> +For QEMU, we maintain a fork of bios bits in gitlab along with all the
> +dependent submodules:
> +https://gitlab.com/qemu-project/biosbits-bits
> +This fork contains numerous fixes, a newer acpica and changes specific to
> +running this avocado QEMU tests using bits. The author of this document
> +is the sole maintainer of the QEMU fork of bios bits repo.
> +
> +Under the directory tests/avocado/, acpi-bits.py is a QEMU avocado test that
> +drives all this.
> +
> +A brief description of the various test files follows.
> +
> +Under tests/avocado/ as the root we have:
> +
> +├── acpi-bits
> +│ ├── bits-config
> +│ │ └── bits-cfg.txt
> +│ ├── bits-tests
> +│ │ ├── smbios.py2
> +│ │ ├── smilatency.py2
> +│ │ ├── testacpi.py2
> +│ │ └── testcpuid.py2
> +│ └── README
> +├── acpi-bits.py
> +
> +tests/avocado:
> + - acpi-bits.py: This is the main python avocado test script that generates a
> +   biosbits iso. It then spawns a QEMU VM with it, collects the log and reports
> +   test failures. This is the script one would be interested in if they wanted
> +   to add or change some component of the log parsing, add a new command line
> +   to alter how QEMU is spawned etc. Test writers typically would not need to
> +   modify this script unless they wanted to enhance or change the log parsing
> +   for their tests. Following environment variables are used in this test:
> +     - V=1 : This enables verbose mode for the test. It dumps the entire log
> +       from bios bits and also more details in case failure happens. It is
> +       useful for debugging the test failures or tests themselves.
> +
> +   In order to run this test, please perform the following steps:
> +   from the QEMU build directory:
> +
> +   $ make check-venv (needed only the first time to create the venv)
> +   $ ./tests/venv/bin/avocado run -t acpi tests/avocado
> +
> +   The above will run all acpi avocado tests including this one.
> +   In order to run the individual test, perform:
> +
> +   $ ./tests/venv/bin/avocado run tests/avocado/acpi-bits.py --tap -
> +
> +   The above will produce output in tap format. You can omit "--tap -" in the
> +   end and it will produce more output like the following:
> +
> +   $ ./tests/venv/bin/avocado run tests/avocado/acpi-bits.py
> +     Fetching asset from tests/avocado/acpi-bits.py:AcpiBitsTest.test_acpi_smbios_bits
> +     JOB ID     : eab225724da7b64c012c65705dc2fa14ab1defef
> +     JOB LOG    : /home/anisinha/avocado/job-results/job-2022-10-10T17.58-eab2257/job.log
> +     (1/1) tests/avocado/acpi-bits.py:AcpiBitsTest.test_acpi_smbios_bits: PASS (33.09 s)
> +     RESULTS    : PASS 1 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0
> +     JOB TIME   : 39.22 s
> +
> +   You can inspect the log file for more information about the run or in order
> +   to diagnoze issues. If you pass V=1 in the environment, more diagnostic logs
> +   would be found in the test log.
> +
> +tests/avocado/acpi-bits:
> + - README: This text file.
> +
> +tests/avocado/acpi-bits/bits-config:
> +   This location contains biosbits config files that determine how the software
> +   runs the tests.
> + - bits-config.txt: this is the biosbits config file that determines what tests
> +   or actions are performed by bits. The description of the config options are
> +   provided in the file itself.
> +
> +tests/avocado/acpi-bits/bits-tests:
> +   This directory contains biosbits python based tests that are run from within
> +   the biosbits environment in the spawned VM. New additions of test cases can
> +   be made in the appropriate test file. For example, new acpi tests can go
> +   into testacpi.py and one would call testsuite.add_test() to register the new
> +   test so that it gets executed as a part of the ACPI tests.
> +   It might be occasionally necessary to disable some subtests or add a new
> +   test that belongs to a test suite not already present in this directory. To
> +   do this, please extract the bits source from the zip file mentioned above.

I will update this in V5. No zip file reference needed.

> +   Copy the test suite/script that needs modification (addition of new tests
> +   or disabling them) from boot/python location of the extracted directory
> +   into this directory.

This also needs updating. Will do in v5.

> +
> +   step (a): copy unmodified test script to this directory.
> +
> +   step (b): perform modifications to the test.
> +   Commit (a) and (b) in separate commits so that the original test script
> +   and the changes we have made are separated and clear.
> +
> +   The test framework will then use your modified test script to run the test.
> +   No further changes would be needed. Please check the logs to make sure that
> +   appropriate changes have taken effect.
> +
> +   The tests have an extension .py2 in order to indicate that
> +   (a) they are python2.7 based scripts and not python 3 scripts.
> +   (b) they are run from within the bios bits VM and is not subjected to QEMU
> +       build/test python script maintainance and dependency resolutions.
> +   (c) They need not be loaded by avocado framework when running tests.
> +
> +
> +Author: Ani Sinha <ani@anisinha.ca>
> +
> +References:
> +(a) https://blog.linuxplumbersconf.org/2011/ocw/system/presentations/867/original/bits.pdf
> +(b) https://www.youtube.com/watch?v=36QIepyUuhg
> --
> 2.34.1
>