[PATCH v3 00/10] Introduce new acpi/smbios avocado tests using biosbits

[PATCH v3 00/10] Introduce new acpi/smbios avocado tests using biosbits

[Ani Sinha]

Please see the README file added in patch 10 for more details.
Sample runs are as follows:

$ ./tests/venv/bin/avocado run -t acpi tests/avocado --tap -
ERROR:  Missing parentheses in call to 'print'. Did you mean print(...)? (smbios.py, line 92)
ERROR:  Missing parentheses in call to 'print'. Did you mean print(...)? (smilatency.py, line 47)
ERROR:  Missing parentheses in call to 'print'. Did you mean print(...)? (testacpi.py, line 158)
1..1
ok 1 tests/avocado/acpi-bits.py:AcpiBitsTest.test_acpi_smbios_bits

$ ./tests/venv/bin/avocado run -t acpi tests/avocado 
ERROR:  Missing parentheses in call to 'print'. Did you mean print(...)? (smbios.py, line 92)
ERROR:  Missing parentheses in call to 'print'. Did you mean print(...)? (smilatency.py, line 47)
ERROR:  Missing parentheses in call to 'print'. Did you mean print(...)? (testacpi.py, line 158)
Fetching asset from tests/avocado/acpi-bits.py:AcpiBitsTest.test_acpi_smbios_bits
JOB ID     : c06a9feb423bcda5de89bb51353c6c1718719f14
JOB LOG    : /home/anisinha/avocado/job-results/job-2022-10-10T13.12-c06a9fe/job.log
 (1/1) tests/avocado/acpi-bits.py:AcpiBitsTest.test_acpi_smbios_bits: PASS (34.11 s)
RESULTS    : PASS 1 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0
JOB TIME   : 37.40 s

 ./tests/venv/bin/avocado run tests/avocado/acpi-bits.py
Fetching asset from tests/avocado/acpi-bits.py:AcpiBitsTest.test_acpi_smbios_bits
JOB ID     : a2476dd3fafe289c5e6475f447bc8369f77d57ba
JOB LOG    : /home/anisinha/avocado/job-results/job-2022-10-10T13.14-a2476dd/job.log
 (1/1) tests/avocado/acpi-bits.py:AcpiBitsTest.test_acpi_smbios_bits: PASS (34.07 s)
RESULTS    : PASS 1 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0
JOB TIME   : 40.20 s


Changelog:
v3:
  - converted the existing test to avocado tests as per the popular
    recommendation. Added appropriate tags.
  - artifact download URL modified based on gitlab-CI URL.

  For biosbits repo:
  - moved to a gitlab repo with me being the maintainer.
  - added .gitlab-ci.yml file to generate the artifacts.
v2:
 - a new class of python based tests introduced that is separate from avocado
   tests or qtests. Can be run by using "make check-pytest".
 - acpi biosbits tests are the first tests to use pytest environment.
 - bios bits tests now download the bits binary archives from a remote
   repository if they are not found locally. The test skips if download
   fails.
 - A new environment variable is introduced that can be passed by the tester
   to specify the location of the bits archives locally. test skips if the
   bits binaries are not found in that location.
 - if pip install of python module fails for whatever reaoson, the test skips.
 - misc code fixes including spell check of the README doc. README has been
   updated as well.
 - addition of SPDX license headers to bits test files.
 - update MAINTAINERS to reflect the new pytest test class.

For biosbits repo:
 - added Dockerfile and build script. Made bios bits build on gcc 11.
   https://github.com/ani-sinha/bits/blob/bits-qemu-logging/Dockerfile
   https://github.com/ani-sinha/bits/blob/bits-qemu-logging/build-artifacts.sh
   The build script generates the zip archive and tarball used by the test.

v1: initial patchset. uses qtest to implement the bios bits tests.

Cc: Qemu Devel <qemu-devel@nongnu.org>
Cc: Daniel P. Berrangé <berrange@redhat.com>
Cc: Paolo Bonzini <pbonzini@redhat.com>
Cc: Maydell Peter <peter.maydell@linaro.org>
Cc: John Snow <jsnow@redhat.com>
Cc: Thomas Huth <thuth@redhat.com>
Cc: Igor M <imammedo@redhat.com>
Cc: Michael Tsirkin <mst@redhat.com>

Ani Sinha (10):
  acpi/tests/avocado/bits: initial commit of test scripts that are run
    by biosbits
  acpi/tests/avocado/bits: add SPDX license identifiers for bios bits
    tests
  acpi/tests/avocado/bits: disable acpi PSS tests that are failing in
    biosbits
  acpi/tests/avocado/bits: add smilatency test suite from bits in order
    to disable it
  acpi/tests/avocado/bits: add SPDX license identifiers for bios bits
    smilatency tests
  acpi/tests/avocado/bits: disable smilatency test since it does not
    pass everytime
  acpi/tests/avocado/bits: add biosbits config file for running bios
    tests
  acpi/tests/avocado/bits: add acpi and smbios avocado tests that uses
    biosbits
  MAINTAINERS: add myself as the maintainer for acpi biosbits avocado
    tests
  acpi/tests/avocado/bits: add a README file

 MAINTAINERS                                   |    6 +
 tests/avocado/acpi-bits.py                    |  334 +++
 tests/avocado/acpi-bits/README                |   90 +
 .../acpi-bits/bits-config/bits-cfg.txt        |   18 +
 tests/avocado/acpi-bits/bits-tests/smbios.py  | 2434 +++++++++++++++++
 .../acpi-bits/bits-tests/smilatency.py        |  107 +
 .../avocado/acpi-bits/bits-tests/testacpi.py  |  287 ++
 .../avocado/acpi-bits/bits-tests/testcpuid.py |   87 +
 8 files changed, 3363 insertions(+)
 create mode 100644 tests/avocado/acpi-bits.py
 create mode 100644 tests/avocado/acpi-bits/README
 create mode 100644 tests/avocado/acpi-bits/bits-config/bits-cfg.txt
 create mode 100644 tests/avocado/acpi-bits/bits-tests/smbios.py
 create mode 100644 tests/avocado/acpi-bits/bits-tests/smilatency.py
 create mode 100644 tests/avocado/acpi-bits/bits-tests/testacpi.py
 create mode 100644 tests/avocado/acpi-bits/bits-tests/testcpuid.py

-- 
2.34.1

Re: [PATCH v3 00/10] Introduce new acpi/smbios avocado tests using biosbits

[Ani Sinha]

On Mon, Oct 10, 2022 at 1:24 PM Ani Sinha <ani@anisinha.ca> wrote:
>
> Please see the README file added in patch 10 for more details.
> Sample runs are as follows:
>
> $ ./tests/venv/bin/avocado run -t acpi tests/avocado --tap -
> ERROR:  Missing parentheses in call to 'print'. Did you mean print(...)? (smbios.py, line 92)
> ERROR:  Missing parentheses in call to 'print'. Did you mean print(...)? (smilatency.py, line 47)
> ERROR:  Missing parentheses in call to 'print'. Did you mean print(...)? (testacpi.py, line 158)

None of the above files are avocado tests or avocado related python
scripts. They are run from within bits in a python 2.7 environment. I
could not find a mechanism to exclude a directory from avocado tests.
I also do not think making those scripts python 3 compliant is a good
use of my time since upgrading bits to use python 3 would be a major
task unrelated to QEMU testing.

> 1..1
> ok 1 tests/avocado/acpi-bits.py:AcpiBitsTest.test_acpi_smbios_bits
>
> $ ./tests/venv/bin/avocado run -t acpi tests/avocado
> ERROR:  Missing parentheses in call to 'print'. Did you mean print(...)? (smbios.py, line 92)
> ERROR:  Missing parentheses in call to 'print'. Did you mean print(...)? (smilatency.py, line 47)
> ERROR:  Missing parentheses in call to 'print'. Did you mean print(...)? (testacpi.py, line 158)
> Fetching asset from tests/avocado/acpi-bits.py:AcpiBitsTest.test_acpi_smbios_bits
> JOB ID     : c06a9feb423bcda5de89bb51353c6c1718719f14
> JOB LOG    : /home/anisinha/avocado/job-results/job-2022-10-10T13.12-c06a9fe/job.log
>  (1/1) tests/avocado/acpi-bits.py:AcpiBitsTest.test_acpi_smbios_bits: PASS (34.11 s)
> RESULTS    : PASS 1 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0
> JOB TIME   : 37.40 s
>
>  ./tests/venv/bin/avocado run tests/avocado/acpi-bits.py
> Fetching asset from tests/avocado/acpi-bits.py:AcpiBitsTest.test_acpi_smbios_bits
> JOB ID     : a2476dd3fafe289c5e6475f447bc8369f77d57ba
> JOB LOG    : /home/anisinha/avocado/job-results/job-2022-10-10T13.14-a2476dd/job.log
>  (1/1) tests/avocado/acpi-bits.py:AcpiBitsTest.test_acpi_smbios_bits: PASS (34.07 s)
> RESULTS    : PASS 1 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0
> JOB TIME   : 40.20 s
>
>
> Changelog:
> v3:
>   - converted the existing test to avocado tests as per the popular
>     recommendation. Added appropriate tags.
>   - artifact download URL modified based on gitlab-CI URL.
>
>   For biosbits repo:
>   - moved to a gitlab repo with me being the maintainer.
>   - added .gitlab-ci.yml file to generate the artifacts.
> v2:
>  - a new class of python based tests introduced that is separate from avocado
>    tests or qtests. Can be run by using "make check-pytest".
>  - acpi biosbits tests are the first tests to use pytest environment.
>  - bios bits tests now download the bits binary archives from a remote
>    repository if they are not found locally. The test skips if download
>    fails.
>  - A new environment variable is introduced that can be passed by the tester
>    to specify the location of the bits archives locally. test skips if the
>    bits binaries are not found in that location.
>  - if pip install of python module fails for whatever reaoson, the test skips.
>  - misc code fixes including spell check of the README doc. README has been
>    updated as well.
>  - addition of SPDX license headers to bits test files.
>  - update MAINTAINERS to reflect the new pytest test class.
>
> For biosbits repo:
>  - added Dockerfile and build script. Made bios bits build on gcc 11.
>    https://github.com/ani-sinha/bits/blob/bits-qemu-logging/Dockerfile
>    https://github.com/ani-sinha/bits/blob/bits-qemu-logging/build-artifacts.sh
>    The build script generates the zip archive and tarball used by the test.
>
> v1: initial patchset. uses qtest to implement the bios bits tests.
>
> Cc: Qemu Devel <qemu-devel@nongnu.org>
> Cc: Daniel P. Berrangé <berrange@redhat.com>
> Cc: Paolo Bonzini <pbonzini@redhat.com>
> Cc: Maydell Peter <peter.maydell@linaro.org>
> Cc: John Snow <jsnow@redhat.com>
> Cc: Thomas Huth <thuth@redhat.com>
> Cc: Igor M <imammedo@redhat.com>
> Cc: Michael Tsirkin <mst@redhat.com>
>
> Ani Sinha (10):
>   acpi/tests/avocado/bits: initial commit of test scripts that are run
>     by biosbits
>   acpi/tests/avocado/bits: add SPDX license identifiers for bios bits
>     tests
>   acpi/tests/avocado/bits: disable acpi PSS tests that are failing in
>     biosbits
>   acpi/tests/avocado/bits: add smilatency test suite from bits in order
>     to disable it
>   acpi/tests/avocado/bits: add SPDX license identifiers for bios bits
>     smilatency tests
>   acpi/tests/avocado/bits: disable smilatency test since it does not
>     pass everytime
>   acpi/tests/avocado/bits: add biosbits config file for running bios
>     tests
>   acpi/tests/avocado/bits: add acpi and smbios avocado tests that uses
>     biosbits
>   MAINTAINERS: add myself as the maintainer for acpi biosbits avocado
>     tests
>   acpi/tests/avocado/bits: add a README file
>
>  MAINTAINERS                                   |    6 +
>  tests/avocado/acpi-bits.py                    |  334 +++
>  tests/avocado/acpi-bits/README                |   90 +
>  .../acpi-bits/bits-config/bits-cfg.txt        |   18 +
>  tests/avocado/acpi-bits/bits-tests/smbios.py  | 2434 +++++++++++++++++
>  .../acpi-bits/bits-tests/smilatency.py        |  107 +
>  .../avocado/acpi-bits/bits-tests/testacpi.py  |  287 ++
>  .../avocado/acpi-bits/bits-tests/testcpuid.py |   87 +
>  8 files changed, 3363 insertions(+)
>  create mode 100644 tests/avocado/acpi-bits.py
>  create mode 100644 tests/avocado/acpi-bits/README
>  create mode 100644 tests/avocado/acpi-bits/bits-config/bits-cfg.txt
>  create mode 100644 tests/avocado/acpi-bits/bits-tests/smbios.py
>  create mode 100644 tests/avocado/acpi-bits/bits-tests/smilatency.py
>  create mode 100644 tests/avocado/acpi-bits/bits-tests/testacpi.py
>  create mode 100644 tests/avocado/acpi-bits/bits-tests/testcpuid.py
>
> --
> 2.34.1
>

Re: [PATCH v3 00/10] Introduce new acpi/smbios avocado tests using biosbits

[Thomas Huth]

On 10/10/2022 10.13, Ani Sinha wrote:
> On Mon, Oct 10, 2022 at 1:24 PM Ani Sinha <ani@anisinha.ca> wrote:
>>
>> Please see the README file added in patch 10 for more details.
>> Sample runs are as follows:
>>
>> $ ./tests/venv/bin/avocado run -t acpi tests/avocado --tap -
>> ERROR:  Missing parentheses in call to 'print'. Did you mean print(...)? (smbios.py, line 92)
>> ERROR:  Missing parentheses in call to 'print'. Did you mean print(...)? (smilatency.py, line 47)
>> ERROR:  Missing parentheses in call to 'print'. Did you mean print(...)? (testacpi.py, line 158)
> 
> None of the above files are avocado tests or avocado related python
> scripts. They are run from within bits in a python 2.7 environment. I
> could not find a mechanism to exclude a directory from avocado tests.
> I also do not think making those scripts python 3 compliant is a good
> use of my time since upgrading bits to use python 3 would be a major
> task unrelated to QEMU testing.

Maybe you could at least switch those three lines to use the new print() 
syntax to silence at least these ugly errors? ... Python 2.7 should cope 
very well with the new syntax, as far as I know...

Otherwise, it might be better to put the non-avocado python files into 
another directory under tests/ ?

  Thomas

Re: [PATCH v3 00/10] Introduce new acpi/smbios avocado tests using biosbits

[Ani Sinha]

On Mon, Oct 10, 2022 at 2:26 PM Thomas Huth <thuth@redhat.com> wrote:
>
> On 10/10/2022 10.13, Ani Sinha wrote:
> > On Mon, Oct 10, 2022 at 1:24 PM Ani Sinha <ani@anisinha.ca> wrote:
> >>
> >> Please see the README file added in patch 10 for more details.
> >> Sample runs are as follows:
> >>
> >> $ ./tests/venv/bin/avocado run -t acpi tests/avocado --tap -
> >> ERROR:  Missing parentheses in call to 'print'. Did you mean print(...)? (smbios.py, line 92)
> >> ERROR:  Missing parentheses in call to 'print'. Did you mean print(...)? (smilatency.py, line 47)
> >> ERROR:  Missing parentheses in call to 'print'. Did you mean print(...)? (testacpi.py, line 158)
> >
> > None of the above files are avocado tests or avocado related python
> > scripts. They are run from within bits in a python 2.7 environment. I
> > could not find a mechanism to exclude a directory from avocado tests.
> > I also do not think making those scripts python 3 compliant is a good
> > use of my time since upgrading bits to use python 3 would be a major
> > task unrelated to QEMU testing.
>
> Maybe you could at least switch those three lines to use the new print()
> syntax

There are lots of print statements in those three files using old
syntax. It's only complaining about the first one.

to silence at least these ugly errors? ... Python 2.7 should cope
> very well with the new syntax, as far as I know...
>
> Otherwise, it might be better to put the non-avocado python files into
> another directory under tests/ ?
>
>   Thomas
>
>

Re: [PATCH v3 00/10] Introduce new acpi/smbios avocado tests using biosbits

[Thomas Huth]

On 10/10/2022 11.13, Ani Sinha wrote:
> On Mon, Oct 10, 2022 at 2:26 PM Thomas Huth <thuth@redhat.com> wrote:
>>
>> On 10/10/2022 10.13, Ani Sinha wrote:
>>> On Mon, Oct 10, 2022 at 1:24 PM Ani Sinha <ani@anisinha.ca> wrote:
>>>>
>>>> Please see the README file added in patch 10 for more details.
>>>> Sample runs are as follows:
>>>>
>>>> $ ./tests/venv/bin/avocado run -t acpi tests/avocado --tap -
>>>> ERROR:  Missing parentheses in call to 'print'. Did you mean print(...)? (smbios.py, line 92)
>>>> ERROR:  Missing parentheses in call to 'print'. Did you mean print(...)? (smilatency.py, line 47)
>>>> ERROR:  Missing parentheses in call to 'print'. Did you mean print(...)? (testacpi.py, line 158)
>>>
>>> None of the above files are avocado tests or avocado related python
>>> scripts. They are run from within bits in a python 2.7 environment. I
>>> could not find a mechanism to exclude a directory from avocado tests.
>>> I also do not think making those scripts python 3 compliant is a good
>>> use of my time since upgrading bits to use python 3 would be a major
>>> task unrelated to QEMU testing.
>>
>> Maybe you could at least switch those three lines to use the new print()
>> syntax
> 
> There are lots of print statements in those three files using old
> syntax. It's only complaining about the first one.
> 
> to silence at least these ugly errors? ... Python 2.7 should cope

Yeah, but everybody who wants to run the QEMU avocado tests will wonder 
about those ERROR messages! So I don't think that this is acceptable in the 
current shape. Either fix the print lines, or move it to another directory.

  Thomas

Re: [PATCH v3 00/10] Introduce new acpi/smbios avocado tests using biosbits

[Daniel P. Berrangé]

On Mon, Oct 10, 2022 at 01:43:21PM +0530, Ani Sinha wrote:
> On Mon, Oct 10, 2022 at 1:24 PM Ani Sinha <ani@anisinha.ca> wrote:
> >
> > Please see the README file added in patch 10 for more details.
> > Sample runs are as follows:
> >
> > $ ./tests/venv/bin/avocado run -t acpi tests/avocado --tap -
> > ERROR:  Missing parentheses in call to 'print'. Did you mean print(...)? (smbios.py, line 92)
> > ERROR:  Missing parentheses in call to 'print'. Did you mean print(...)? (smilatency.py, line 47)
> > ERROR:  Missing parentheses in call to 'print'. Did you mean print(...)? (testacpi.py, line 158)
> 
> None of the above files are avocado tests or avocado related python
> scripts. They are run from within bits in a python 2.7 environment. I
> could not find a mechanism to exclude a directory from avocado tests.
> I also do not think making those scripts python 3 compliant is a good
> use of my time since upgrading bits to use python 3 would be a major
> task unrelated to QEMU testing.

In one of the later patches copy_test_scripts()  copies the files
into the guest image IIUC.

If you rename them in git, to be .py2  then presumably avocado
wont try to load them, and then in copy_test_scripts you can
give them the normal .py extension for the guest.  The .py2
extension will also make it more obvious to maintainers that
these are different from the rest of the python coded we have.

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [PATCH v3 00/10] Introduce new acpi/smbios avocado tests using biosbits

[Ani Sinha]

On Mon, Oct 10, 2022 at 2:50 PM Daniel P. Berrangé <berrange@redhat.com> wrote:
>
> On Mon, Oct 10, 2022 at 01:43:21PM +0530, Ani Sinha wrote:
> > On Mon, Oct 10, 2022 at 1:24 PM Ani Sinha <ani@anisinha.ca> wrote:
> > >
> > > Please see the README file added in patch 10 for more details.
> > > Sample runs are as follows:
> > >
> > > $ ./tests/venv/bin/avocado run -t acpi tests/avocado --tap -
> > > ERROR:  Missing parentheses in call to 'print'. Did you mean print(...)? (smbios.py, line 92)
> > > ERROR:  Missing parentheses in call to 'print'. Did you mean print(...)? (smilatency.py, line 47)
> > > ERROR:  Missing parentheses in call to 'print'. Did you mean print(...)? (testacpi.py, line 158)
> >
> > None of the above files are avocado tests or avocado related python
> > scripts. They are run from within bits in a python 2.7 environment. I
> > could not find a mechanism to exclude a directory from avocado tests.
> > I also do not think making those scripts python 3 compliant is a good
> > use of my time since upgrading bits to use python 3 would be a major
> > task unrelated to QEMU testing.
>
> In one of the later patches copy_test_scripts()  copies the files
> into the guest image IIUC.
>
> If you rename them in git, to be .py2  then presumably avocado
> wont try to load them, and then in copy_test_scripts you can
> give them the normal .py extension for the guest.  The .py2
> extension will also make it more obvious to maintainers that
> these are different from the rest of the python coded we have.

I did a quick test and this approach seems to work.

 ./tests/venv/bin/avocado run -t acpi tests/avocado --tap -
1..1
ok 1 tests/avocado/acpi-bits.py:AcpiBitsTest.test_acpi_smbios_bits