acpiBitsTest.test_acpi_smbios_bits test intermittently times out

acpiBitsTest.test_acpi_smbios_bits test intermittently times out

[Peter Maydell]

The avocado test acpiBitsTest.test_acpi_smbios_bits seems to be
flaky in CI -- sometimes it appears to time out.

https://gitlab.com/qemu-project/qemu/-/issues/2077
has the details (including links to jobs etc). As far as I can
see, the test is still running when after about a minute it
gets timed out. (Though the python tracebacks in the logs are
not easy for me to interpret, so I might be wrong). This I find
a bit confusing, because tests/avocado/acpi-bits.py sets
"timeout = 200". So maybe that isn't taking effect properly?

Does anybody have time to investigate this? If not, we can disable
the test as flaky until somebody does have the time.

thanks
-- PMM

Re: acpiBitsTest.test_acpi_smbios_bits test intermittently times out

[Ani Sinha]

On Sat, Jan 6, 2024 at 12:11 AM Peter Maydell <peter.maydell@linaro.org> wrote:
>
> The avocado test acpiBitsTest.test_acpi_smbios_bits seems to be
> flaky in CI -- sometimes it appears to time out.
>
> https://gitlab.com/qemu-project/qemu/-/issues/2077
> has the details (including links to jobs etc).

Do you have more data points in terms of the jobs that failed? The one
you point to seems to be stuck in SMI latency tests:

==== SMI latency test ====
16:45:49 DEBUG|
Warning: touching the keyboard can affect the results of this test.
16:45:50 DEBUG|
Starting test. Wait here, I will be back in 15 seconds.

If this is consistently timing out here, we can disable the smi
latency test. This test was a little problematic from the get go.

Meanwhile I have assigned
https://gitlab.com/qemu-project/qemu/-/issues/2077 to myself and will
take a look.

> As far as I can
> see, the test is still running when after about a minute it
> gets timed out. (Though the python tracebacks in the logs are
> not easy for me to interpret, so I might be wrong). This I find
> a bit confusing, because tests/avocado/acpi-bits.py sets
> "timeout = 200". So maybe that isn't taking effect properly?
>
> Does anybody have time to investigate this? If not, we can disable
> the test as flaky until somebody does have the time.
>
> thanks
> -- PMM
>

Re: acpiBitsTest.test_acpi_smbios_bits test intermittently times out

[Ani Sinha]

On Sat, Jan 6, 2024 at 10:05 AM Ani Sinha <anisinha@redhat.com> wrote:
>
> On Sat, Jan 6, 2024 at 12:11 AM Peter Maydell <peter.maydell@linaro.org> wrote:
> >
> > The avocado test acpiBitsTest.test_acpi_smbios_bits seems to be
> > flaky in CI -- sometimes it appears to time out.
> >
> > https://gitlab.com/qemu-project/qemu/-/issues/2077
> > has the details (including links to jobs etc).
>
> Do you have more data points in terms of the jobs that failed?

I just noticed that you attached three examples of failed tests. In
all of them the test seems to be stuck at the ami latency test.

The one
> you point to seems to be stuck in SMI latency tests:
>
> ==== SMI latency test ====
> 16:45:49 DEBUG|
> Warning: touching the keyboard can affect the results of this test.
> 16:45:50 DEBUG|
> Starting test. Wait here, I will be back in 15 seconds.
>
> If this is consistently timing out here, we can disable the smi
> latency test. This test was a little problematic from the get go.
>
> Meanwhile I have assigned
> https://gitlab.com/qemu-project/qemu/-/issues/2077 to myself and will
> take a look.
>
> > As far as I can
> > see, the test is still running when after about a minute it
> > gets timed out. (Though the python tracebacks in the logs are
> > not easy for me to interpret, so I might be wrong). This I find
> > a bit confusing, because tests/avocado/acpi-bits.py sets
> > "timeout = 200". So maybe that isn't taking effect properly?
> >
> > Does anybody have time to investigate this? If not, we can disable
> > the test as flaky until somebody does have the time.
> >
> > thanks
> > -- PMM
> >

Re: acpiBitsTest.test_acpi_smbios_bits test intermittently times out

[Peter Maydell]

On Sat, 6 Jan 2024 at 05:41, Ani Sinha <anisinha@redhat.com> wrote:
>
> On Sat, Jan 6, 2024 at 10:05 AM Ani Sinha <anisinha@redhat.com> wrote:
> >
> > On Sat, Jan 6, 2024 at 12:11 AM Peter Maydell <peter.maydell@linaro.org> wrote:
> > >
> > > The avocado test acpiBitsTest.test_acpi_smbios_bits seems to be
> > > flaky in CI -- sometimes it appears to time out.
> > >
> > > https://gitlab.com/qemu-project/qemu/-/issues/2077
> > > has the details (including links to jobs etc).
> >
> > Do you have more data points in terms of the jobs that failed?
>
> I just noticed that you attached three examples of failed tests. In
> all of them the test seems to be stuck at the ami latency test.

OK, if you think that subtest is suspicious, could you send a
patch that disables just that subpart, and we'll see if it helps?

Thanks for looking into the issue.

-- PMM

Re: acpiBitsTest.test_acpi_smbios_bits test intermittently times out

[Ani Sinha]

On Sat, Jan 6, 2024 at 5:39 PM Peter Maydell <peter.maydell@linaro.org> wrote:
>
> On Sat, 6 Jan 2024 at 05:41, Ani Sinha <anisinha@redhat.com> wrote:
> >
> > On Sat, Jan 6, 2024 at 10:05 AM Ani Sinha <anisinha@redhat.com> wrote:
> > >
> > > On Sat, Jan 6, 2024 at 12:11 AM Peter Maydell <peter.maydell@linaro.org> wrote:
> > > >
> > > > The avocado test acpiBitsTest.test_acpi_smbios_bits seems to be
> > > > flaky in CI -- sometimes it appears to time out.
> > > >
> > > > https://gitlab.com/qemu-project/qemu/-/issues/2077
> > > > has the details (including links to jobs etc).
> > >
> > > Do you have more data points in terms of the jobs that failed?
> >
> > I just noticed that you attached three examples of failed tests. In
> > all of them the test seems to be stuck at the ami latency test.
>
> OK, if you think that subtest is suspicious, could you send a
> patch that disables just that subpart, and we'll see if it helps?

I pushed the patches here:
https://gitlab.com/anisinha/qemu/-/commits/disable-smilatency
and ran the pipeline. It passed
https://gitlab.com/anisinha/qemu/-/jobs/5878585312

and the smilatency test was not run:
https://anisinha.gitlab.io/-/qemu/-/jobs/5878585312/artifacts/build/tests/results/latest/test-results/01-tests_avocado_acpi-bits.py_AcpiBitsTest.test_acpi_smbios_bits/debug.log

Not sure what we can do to have some confidence that disabling the
test got rid of the flakiness. I can send out those two patches in the
list.

Re: acpiBitsTest.test_acpi_smbios_bits test intermittently times out

[Ani Sinha]

> On 07-Jan-2024, at 9:56 PM, Ani Sinha <anisinha@redhat.com> wrote:
> 
> On Sat, Jan 6, 2024 at 5:39 PM Peter Maydell <peter.maydell@linaro.org> wrote:
>> 
>> On Sat, 6 Jan 2024 at 05:41, Ani Sinha <anisinha@redhat.com> wrote:
>>> 
>>> On Sat, Jan 6, 2024 at 10:05 AM Ani Sinha <anisinha@redhat.com> wrote:
>>>> 
>>>> On Sat, Jan 6, 2024 at 12:11 AM Peter Maydell <peter.maydell@linaro.org> wrote:
>>>>> 
>>>>> The avocado test acpiBitsTest.test_acpi_smbios_bits seems to be
>>>>> flaky in CI -- sometimes it appears to time out.
>>>>> 
>>>>> https://gitlab.com/qemu-project/qemu/-/issues/2077
>>>>> has the details (including links to jobs etc).
>>>> 
>>>> Do you have more data points in terms of the jobs that failed?
>>> 
>>> I just noticed that you attached three examples of failed tests. In
>>> all of them the test seems to be stuck at the ami latency test.
>> 
>> OK, if you think that subtest is suspicious, could you send a
>> patch that disables just that subpart, and we'll see if it helps?
> 
> I pushed the patches here:
> https://gitlab.com/anisinha/qemu/-/commits/disable-smilatency
> and ran the pipeline. It passed
> https://gitlab.com/anisinha/qemu/-/jobs/5878585312
> 
> and the smilatency test was not run:
> https://anisinha.gitlab.io/-/qemu/-/jobs/5878585312/artifacts/build/tests/results/latest/test-results/01-tests_avocado_acpi-bits.py_AcpiBitsTest.test_acpi_smbios_bits/debug.log
> 
> Not sure what we can do to have some confidence that disabling the
> test got rid of the flakiness.

I ran the avocado jobs avocado-system-centos and avocado-system-opensuse a few times and all times they passed. So I sent out the patches. Hope this settles it.


> I can send out those two patches in the
> list.