intermittent qtest-aarch64/xlnx-canfd-test test failure

intermittent qtest-aarch64/xlnx-canfd-test test failure

[Michael Tokarev]

Hi!

While doing stable-8.1.1 preparation, I've a CI failure of ubuntu-20.04-s390x-all here:

https://gitlab.com/qemu-project/qemu/-/jobs/5132720046

▶ ERROR:../tests/qtest/xlnx-canfd-test.c:265:match_rx_tx_data: \
   assertion failed ((buf_rx[size] & DLC_FD_BIT_MASK) == (buf_tx[size] & DLC_FD_BIT_MASK)): (2281701376 == 2147483648)

  qemu:qtest+qtest-aarch64 / qtest-aarch64/xlnx-canfd-test  \
            ERROR           0.77s   killed by signal 6 SIGABRT

Re-run of this test went successfully.  However, the assert looks
a bit scary, and the fact that it does not fail in a reliable way
is a bit troubling too.  Is it a flaky test or there's something
else in qemu with concurrent threads?

I don't see this assert in previous test runs.  But ubuntu-20.04-s390x-all
test fails quite often due to other reasons, so it isn't conclusive.

Should something be done with this?

The testing is done for commit f2fc49c302036315db6e8c9f74592decc3be0476,
which is in staging-8.1 branch only temporarily.

Thanks,

/mjt

Re: intermittent qtest-aarch64/xlnx-canfd-test test failure

[Cédric Le Goater]

On 9/21/23 09:51, Michael Tokarev wrote:
> Hi!
> 
> While doing stable-8.1.1 preparation, I've a CI failure of ubuntu-20.04-s390x-all here:
> 
> https://gitlab.com/qemu-project/qemu/-/jobs/5132720046
> 
> ▶ ERROR:../tests/qtest/xlnx-canfd-test.c:265:match_rx_tx_data: \
>    assertion failed ((buf_rx[size] & DLC_FD_BIT_MASK) == (buf_tx[size] & DLC_FD_BIT_MASK)): (2281701376 == 2147483648)
> 
>   qemu:qtest+qtest-aarch64 / qtest-aarch64/xlnx-canfd-test  \
>             ERROR           0.77s   killed by signal 6 SIGABRT
> 
> Re-run of this test went successfully.  However, the assert looks
> a bit scary, and the fact that it does not fail in a reliable way
> is a bit troubling too.  Is it a flaky test or there's something
> else in qemu with concurrent threads?
> 
> I don't see this assert in previous test runs.  But ubuntu-20.04-s390x-all
> test fails quite often due to other reasons, so it isn't conclusive.
> 
> Should something be done with this?
> 
> The testing is done for commit f2fc49c302036315db6e8c9f74592decc3be0476,
> which is in staging-8.1 branch only temporarily.
> 
> Thanks,
> 
> /mjt
> 

I had the same CI error in job https://gitlab.com/legoater/qemu/-/jobs/5416150239
  
▶ 170/258 ERROR:../tests/qtest/xlnx-canfd-test.c:265:match_rx_tx_data: assertion failed ((buf_rx[size] & DLC_FD_BIT_MASK) == (buf_tx[size] & DLC_FD_BIT_MASK)): (2281701376 == 2147483648) ERROR
170/258 qemu:qtest+qtest-aarch64 / qtest-aarch64/xlnx-canfd-test           ERROR           0.49s   killed by signal 6 SIGABRT
>>> QTEST_QEMU_BINARY=./qemu-system-aarch64 PYTHON=/builds/legoater/qemu/build/pyvenv/bin/python3 MALLOC_PERTURB_=60 G_TEST_DBUS_DAEMON=/builds/legoater/qemu/tests/dbus-vmstate-daemon.sh /builds/legoater/qemu/build/tests/qtest/xlnx-canfd-test --tap -k
――――――――――――――――――――――――――――――――――――― ✀  ―――――――――――――――――――――――――――――――――――――
stderr:
**
ERROR:../tests/qtest/xlnx-canfd-test.c:265:match_rx_tx_data: assertion failed ((buf_rx[size] & DLC_FD_BIT_MASK) == (buf_tx[size] & DLC_FD_BIT_MASK)): (2281701376 == 2147483648)
(test program exited with status code -6)

I could reproduce on PPC64 and s390x systems.

C.

Re: intermittent qtest-aarch64/xlnx-canfd-test test failure

[Vikram Garhwal]

Hi Michael & Cedric,
On Mon, Oct 30, 2023 at 05:19:38PM +0100, Cédric Le Goater wrote:
> On 9/21/23 09:51, Michael Tokarev wrote:
> > Hi!
> > 
> > While doing stable-8.1.1 preparation, I've a CI failure of ubuntu-20.04-s390x-all here:
> > 
> > https://gitlab.com/qemu-project/qemu/-/jobs/5132720046
> > 
> > ▶ ERROR:../tests/qtest/xlnx-canfd-test.c:265:match_rx_tx_data: \
> >    assertion failed ((buf_rx[size] & DLC_FD_BIT_MASK) == (buf_tx[size] & DLC_FD_BIT_MASK)): (2281701376 == 2147483648)
> > 
> >   qemu:qtest+qtest-aarch64 / qtest-aarch64/xlnx-canfd-test  \
> >             ERROR           0.77s   killed by signal 6 SIGABRT
> > 
> > Re-run of this test went successfully.  However, the assert looks
> > a bit scary, and the fact that it does not fail in a reliable way
> > is a bit troubling too.  Is it a flaky test or there's something
> > else in qemu with concurrent threads?
> > 
> > I don't see this assert in previous test runs.  But ubuntu-20.04-s390x-all
> > test fails quite often due to other reasons, so it isn't conclusive.
> > 
> > Should something be done with this?
> > 
> > The testing is done for commit f2fc49c302036315db6e8c9f74592decc3be0476,
> > which is in staging-8.1 branch only temporarily.
> > 
I tried with staging-8.1 and didn't see this error. I ran it with registry.gitlab.com/legoater/qemu/qemu/fedora-i386-cross:latest on a x86 host machine. Can you please share more about how you are running
the testes? It will be greatly helpful to me in reproducing the issue and fixing
it.
> > Thanks,
> > 
> > /mjt
> > 
> 
> I had the same CI error in job https://gitlab.com/legoater/qemu/-/jobs/5416150239
> ▶ 170/258 ERROR:../tests/qtest/xlnx-canfd-test.c:265:match_rx_tx_data: assertion failed ((buf_rx[size] & DLC_FD_BIT_MASK) == (buf_tx[size] & DLC_FD_BIT_MASK)): (2281701376 == 2147483648) ERROR
> 170/258 qemu:qtest+qtest-aarch64 / qtest-aarch64/xlnx-canfd-test           ERROR           0.49s   killed by signal 6 SIGABRT
> > > > QTEST_QEMU_BINARY=./qemu-system-aarch64 PYTHON=/builds/legoater/qemu/build/pyvenv/bin/python3 MALLOC_PERTURB_=60 G_TEST_DBUS_DAEMON=/builds/legoater/qemu/tests/dbus-vmstate-daemon.sh /builds/legoater/qemu/build/tests/qtest/xlnx-canfd-test --tap -k
> ――――――――――――――――――――――――――――――――――――― ✀  ―――――――――――――――――――――――――――――――――――――
> stderr:
> **
> ERROR:../tests/qtest/xlnx-canfd-test.c:265:match_rx_tx_data: assertion failed ((buf_rx[size] & DLC_FD_BIT_MASK) == (buf_tx[size] & DLC_FD_BIT_MASK)): (2281701376 == 2147483648)
> (test program exited with status code -6)
> 
> I could reproduce on PPC64 and s390x systems.
> 
> C.
> 

Re: intermittent qtest-aarch64/xlnx-canfd-test test failure

[Cédric Le Goater]

On 10/31/23 21:19, Vikram Garhwal wrote:
> Hi Michael & Cedric,
> On Mon, Oct 30, 2023 at 05:19:38PM +0100, Cédric Le Goater wrote:
>> On 9/21/23 09:51, Michael Tokarev wrote:
>>> Hi!
>>>
>>> While doing stable-8.1.1 preparation, I've a CI failure of ubuntu-20.04-s390x-all here:
>>>
>>> https://gitlab.com/qemu-project/qemu/-/jobs/5132720046
>>>
>>> ▶ ERROR:../tests/qtest/xlnx-canfd-test.c:265:match_rx_tx_data: \
>>>     assertion failed ((buf_rx[size] & DLC_FD_BIT_MASK) == (buf_tx[size] & DLC_FD_BIT_MASK)): (2281701376 == 2147483648)
>>>
>>>    qemu:qtest+qtest-aarch64 / qtest-aarch64/xlnx-canfd-test  \
>>>              ERROR           0.77s   killed by signal 6 SIGABRT
>>>
>>> Re-run of this test went successfully.  However, the assert looks
>>> a bit scary, and the fact that it does not fail in a reliable way
>>> is a bit troubling too.  Is it a flaky test or there's something
>>> else in qemu with concurrent threads?
>>>
>>> I don't see this assert in previous test runs.  But ubuntu-20.04-s390x-all
>>> test fails quite often due to other reasons, so it isn't conclusive.
>>>
>>> Should something be done with this?
>>>
>>> The testing is done for commit f2fc49c302036315db6e8c9f74592decc3be0476,
>>> which is in staging-8.1 branch only temporarily.
>>>
> I tried with staging-8.1 and didn't see this error. I ran it with registry.gitlab.com/legoater/qemu/qemu/fedora-i386-cross:latest on a x86 host machine. Can you please share more about how you are running
> the testes? It will be greatly helpful to me in reproducing the issue and fixing

I simply ran :

while [ 1 ]; do QTEST_QEMU_BINARY=build/aarch64-softmmu/qemu-system-aarch64 QTEST_QEMU_IMG=qemu-img build/tests/qtest/xlnx-canfd-test || break; done
....
# starting QEMU: exec build/aarch64-softmmu/qemu-system-aarch64 -qtest unix:/tmp/qtest-689050.sock -qtest-log /dev/null -chardev socket,path=/tmp/qtest-689050.qmp,id=char0 -mon chardev=char0,mode=control -display none -audio none -machine xlnx-versal-virt -object can-bus,id=canbus -machine canbus0=canbus -machine canbus1=canbus -accel qtest
**
ERROR:../tests/qtest/xlnx-canfd-test.c:265:match_rx_tx_data: assertion failed ((buf_rx[size] & DLC_FD_BIT_MASK) == (buf_tx[size] & DLC_FD_BIT_MASK)): (2281701376 == 2147483648)
Bail out! ERROR:../tests/qtest/xlnx-canfd-test.c:265:match_rx_tx_data: assertion failed ((buf_rx[size] & DLC_FD_BIT_MASK) == (buf_tx[size] & DLC_FD_BIT_MASK)): (2281701376 == 2147483648)
Aborted (core dumped)


It is always the same error : 0x88000000 == 0x80000000

Thanks,

C.

> it.
>>> Thanks,
>>>
>>> /mjt
>>>
>>
>> I had the same CI error in job https://gitlab.com/legoater/qemu/-/jobs/5416150239
>> ▶ 170/258 ERROR:../tests/qtest/xlnx-canfd-test.c:265:match_rx_tx_data: assertion failed ((buf_rx[size] & DLC_FD_BIT_MASK) == (buf_tx[size] & DLC_FD_BIT_MASK)): (2281701376 == 2147483648) ERROR
>> 170/258 qemu:qtest+qtest-aarch64 / qtest-aarch64/xlnx-canfd-test           ERROR           0.49s   killed by signal 6 SIGABRT
>>>>> QTEST_QEMU_BINARY=./qemu-system-aarch64 PYTHON=/builds/legoater/qemu/build/pyvenv/bin/python3 MALLOC_PERTURB_=60 G_TEST_DBUS_DAEMON=/builds/legoater/qemu/tests/dbus-vmstate-daemon.sh /builds/legoater/qemu/build/tests/qtest/xlnx-canfd-test --tap -k
>> ――――――――――――――――――――――――――――――――――――― ✀  ―――――――――――――――――――――――――――――――――――――
>> stderr:
>> **
>> ERROR:../tests/qtest/xlnx-canfd-test.c:265:match_rx_tx_data: assertion failed ((buf_rx[size] & DLC_FD_BIT_MASK) == (buf_tx[size] & DLC_FD_BIT_MASK)): (2281701376 == 2147483648)
>> (test program exited with status code -6)
>>
>> I could reproduce on PPC64 and s390x systems.
>>
>> C.
>>
>