Re: [PATCH] Revert "check-block: replace -makecheck with TAP output"

[Thomas Huth <thuth@redhat.com>]

On 04/03/2022 17.57, Daniel P. Berrangé wrote:
> On Tue, Feb 08, 2022 at 07:44:32AM -0500, Paolo Bonzini wrote:
>> This reverts commit d316859f4e28c74ab8b618895d2a5e0a865d3cf1.  The
>> TAP output is inferior in that it does not include the diff for
>> failed tests.
>>
>> Reported-by: Peter Maydell <peter.maydell@linaro.org>
>> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
>> ---
>>   tests/check-block.sh             |  6 ++--
>>   tests/qemu-iotests/check         |  6 ++--
>>   tests/qemu-iotests/meson.build   |  1 -
>>   tests/qemu-iotests/testenv.py    | 30 +++++++++----------
>>   tests/qemu-iotests/testrunner.py | 49 +++++++++++++++-----------------
>>   5 files changed, 44 insertions(+), 48 deletions(-)
> 
> Peter mentioned that he's seeing the CentOS 8 CI job fail
> continuously: eg
> 
> https://gitlab.com/qemu-project/qemu/-/jobs/2163919781
> 
> [quote]
> 1/1 qemu:block / qemu-iotests qcow2 ERROR          163.04s   120 subtests passed
> Summary of Failures:
> 1/1 qemu:block / qemu-iotests qcow2 ERROR          163.04s   120 subtests passed
> Ok:                 0
> Expected Fail:      0
> Fail:               1
> Unexpected Pass:    0
> Skipped:            0
> Timeout:            0
> Full log written to /home/gitlab-runner/builds/Jpwtyaz7/0/qemu-project/qemu/build/meson-logs/iotestslog.txt
> make: *** [/home/gitlab-runner/builds/Jpwtyaz7/0/qemu-project/qemu/tests/Makefile.include:156: check-block] Error 1
> 
> [/quote]
> 
> None of the individual tests are showing ERROR though, only the final
> status is error.
> 
> We also don't have access to iotestslog.txt which is very annoying.
> 
> When i run it locally I got the iotests.log and found that it
> contained an error message saying the TAP outout was malformed
> and couldn't be parsed.
> 
> When I ran the iotests manually I confirmed this, it prints:
> 
> ok qcow2 037
> ok qcow2 038
> ok qcow2 039
> ok qcow2 040
> ok qcow2 041
>      [case not run] test_after_a_quorum_snapshot (__main__.TestRepairQuorum): formats ['quorum'] are not whitelisted
>      [case not run] test_cancel (__main__.TestRepairQuorum): formats ['quorum'] are not whitelisted
>      [case not run] test_cancel_after_ready (__main__.TestRepairQuorum): formats ['quorum'] are not whitelisted
>      [case not run] test_complete (__main__.TestRepairQuorum): formats ['quorum'] are not whitelisted
>      [case not run] test_device_not_found (__main__.TestRepairQuorum): formats ['quorum'] are not whitelisted
>      [case not run] test_image_not_found (__main__.TestRepairQuorum): formats ['quorum'] are not whitelisted
>      [case not run] test_medium_not_found (__main__.TestRepairQuorum): formats ['quorum'] are not whitelisted
>      [case not run] test_no_node_name (__main__.TestRepairQuorum): formats ['quorum'] are not whitelisted
>      [case not run] test_nonexistent_replaces (__main__.TestRepairQuorum): formats ['quorum'] are not whitelisted
>      [case not run] test_pause (__main__.TestRepairQuorum): formats ['quorum'] are not whitelisted
>      [case not run] test_with_other_parent (__main__.TestRepairQuorum): formats ['quorum'] are not whitelisted
>      [case not run] test_with_other_parents_after_mirror_start (__main__.TestRepairQuorum): formats ['quorum'] are not whitelisted
>      [case not run] test_wrong_sync_mode (__main__.TestRepairQuorum): formats ['quorum'] are not whitelisted
> 
> ok qcow2 042
> ok qcow2 043
> ok qcow2 046
> 
> 
> All those '[case not run]' lines are invalid tap output.  The
> need to start with a leading '#' to be treated as a diagnostic
> message. I briefly considered sending a patch, until I remember
> this change to stop using TAP.

A patch for fixing the TAP problem is already on the list and has been 
queued by Hanna:

  https://lists.gnu.org/archive/html/qemu-devel/2022-02/msg05074.html

So that should hopefully fix the problem with the CentOS 8 CI job once it 
gets merged.

  Thomas